WEBVTT 00:00:00.000 --> 00:00:00.992 align:middle line:90% 00:00:00.992 --> 00:00:02.976 align:middle line:90% [SQUEAKING] 00:00:02.976 --> 00:00:04.960 align:middle line:90% [RUSTLING] 00:00:04.960 --> 00:00:16.425 align:middle line:90% [CLICKING] 00:00:16.425 --> 00:00:17.300 align:middle line:90% PROFESSOR: All right. 00:00:17.300 --> 00:00:19.440 align:middle line:84% So today, we actually come to the last lecture of the class, 00:00:19.440 --> 00:00:21.780 align:middle line:84% because Wednesday is going to be project presentations. 00:00:21.780 --> 00:00:25.700 align:middle line:84% And so I want to talk to you about diffusion models today, 00:00:25.700 --> 00:00:29.520 align:middle line:84% which is an incredibly exciting area, which I don't think 00:00:29.520 --> 00:00:32.360 align:middle line:84% gets the same amount of attention, in some ways, 00:00:32.360 --> 00:00:34.720 align:middle line:84% compared to large language models. 00:00:34.720 --> 00:00:37.480 align:middle line:90% But it's got enormous potential. 00:00:37.480 --> 00:00:39.960 align:middle line:84% So I'm very excited to talk to you about it. 00:00:39.960 --> 00:00:43.040 align:middle line:84% So just for kicks last night, I asked 00:00:43.040 --> 00:00:45.440 align:middle line:84% ChatGPT to create a photorealistic image 00:00:45.440 --> 00:00:48.320 align:middle line:84% of graduate students in class, in a class on deep learning, 00:00:48.320 --> 00:00:51.640 align:middle line:84% and this is what it came back with. 00:00:51.640 --> 00:00:54.816 align:middle line:84% There is a noticeable absence of an instructor. 00:00:54.816 --> 00:00:57.200 align:middle line:90% [LAUGHS] 00:00:57.200 --> 00:01:01.350 align:middle line:84% Plus, various students are facing in various directions. 00:01:01.350 --> 00:01:04.750 align:middle line:84% But apart from that, it's not bad. 00:01:04.750 --> 00:01:08.990 align:middle line:84% And here is an example of a Midjourney 00:01:08.990 --> 00:01:11.990 align:middle line:84% text-to-image diffusion model, which 00:01:11.990 --> 00:01:15.230 align:middle line:84% produces the amazing picture from this prompt. 00:01:15.230 --> 00:01:17.930 align:middle line:84% A quaint Italian seaside village with colorful buildings, 00:01:17.930 --> 00:01:21.430 align:middle line:84% blah blah, blah, blah, blah, rendered in the style 00:01:21.430 --> 00:01:23.930 align:middle line:84% of Claude Monet and so on and so forth. 00:01:23.930 --> 00:01:24.970 align:middle line:90% And that's what you get. 00:01:24.970 --> 00:01:26.790 align:middle line:90% It's pretty unbelievable. 00:01:26.790 --> 00:01:29.290 align:middle line:84% And I'm sure you folks have played around with these things, 00:01:29.290 --> 00:01:31.248 align:middle line:84% and you have your favorite pictures and prompts 00:01:31.248 --> 00:01:33.550 align:middle line:90% and whatnot. 00:01:33.550 --> 00:01:39.990 align:middle line:84% Now, February 15, OpenAI released a text-to-video model 00:01:39.990 --> 00:01:43.790 align:middle line:84% called Sora, which you folks may have seen, 00:01:43.790 --> 00:01:46.530 align:middle line:84% which I find frankly, just stunning what it can do. 00:01:46.530 --> 00:01:50.630 align:middle line:84% It can produce a one minute video from a text prompt. 00:01:50.630 --> 00:01:54.790 align:middle line:84% And so if you actually give it this prompt, 00:01:54.790 --> 00:01:58.220 align:middle line:84% "In an ornate historical hall, a massive tidal wave peaks 00:01:58.220 --> 00:02:01.760 align:middle line:84% and begins to crash, and two surfers, seizing the moment, 00:02:01.760 --> 00:02:04.453 align:middle line:90% skillfully navigate the wave." 00:02:04.453 --> 00:02:06.620 align:middle line:84% I think we can all agree that such a thing has never 00:02:06.620 --> 00:02:08.820 align:middle line:84% happened in history, and therefore, it 00:02:08.820 --> 00:02:10.539 align:middle line:90% was not in the training data. 00:02:10.539 --> 00:02:14.190 align:middle line:84% And then, you get this picture, this video. 00:02:14.190 --> 00:02:26.900 align:middle line:90% 00:02:26.900 --> 00:02:29.100 align:middle line:84% And then, some random person is coming back 00:02:29.100 --> 00:02:31.722 align:middle line:90% in a completely dry hall. 00:02:31.722 --> 00:02:33.180 align:middle line:84% So anyway, but it's pretty amazing. 00:02:33.180 --> 00:02:34.980 align:middle line:90% I think you'd agree. 00:02:34.980 --> 00:02:40.780 align:middle line:84% So if you actually look at the OpenAI's Sora technical report, 00:02:40.780 --> 00:02:44.380 align:middle line:84% you actually find this opening paragraph 00:02:44.380 --> 00:02:46.900 align:middle line:84% where they say that we train text 00:02:46.900 --> 00:02:50.860 align:middle line:84% conditional diffusion models, blah, blah, blah, using 00:02:50.860 --> 00:02:53.620 align:middle line:90% a transformer architecture. 00:02:53.620 --> 00:02:56.550 align:middle line:84% So now, we know what a transformer architecture is. 00:02:56.550 --> 00:02:57.750 align:middle line:90% You've been working with it. 00:02:57.750 --> 00:02:59.670 align:middle line:84% You're quite familiar with it at this point. 00:02:59.670 --> 00:03:03.090 align:middle line:84% So today's class is really about text conditional diffusion 00:03:03.090 --> 00:03:06.090 align:middle line:84% models, the other building block. 00:03:06.090 --> 00:03:08.010 align:middle line:90% OK, so let's get to it. 00:03:08.010 --> 00:03:09.450 align:middle line:84% What I'm going to do is I'm going 00:03:09.450 --> 00:03:11.410 align:middle line:90% to divide this into two parts. 00:03:11.410 --> 00:03:13.410 align:middle line:84% The first part is, I'm just going to talk about, 00:03:13.410 --> 00:03:16.110 align:middle line:84% how do you get a model to just generate an image for you? 00:03:16.110 --> 00:03:17.850 align:middle line:84% If you wanted to generate an image 00:03:17.850 --> 00:03:20.130 align:middle line:84% from a class of potential images, 00:03:20.130 --> 00:03:21.710 align:middle line:84% how can you just generate an image? 00:03:21.710 --> 00:03:24.030 align:middle line:84% And then, next, we talk about, OK, great. 00:03:24.030 --> 00:03:26.770 align:middle line:84% Now, that you can do that, how do you actually control or steer 00:03:26.770 --> 00:03:29.130 align:middle line:84% the model to do an image based on whatever 00:03:29.130 --> 00:03:30.870 align:middle line:90% prompting you give it? 00:03:30.870 --> 00:03:31.870 align:middle line:90% How do you condition it? 00:03:31.870 --> 00:03:32.787 align:middle line:90% How do you control it? 00:03:32.787 --> 00:03:33.787 align:middle line:90% Those are all the words. 00:03:33.787 --> 00:03:34.630 align:middle line:90% How do you steer it? 00:03:34.630 --> 00:03:36.250 align:middle line:84% You'll find all these synonyms being used heavily 00:03:36.250 --> 00:03:37.210 align:middle line:90% in the literature. 00:03:37.210 --> 00:03:38.543 align:middle line:90% That's basically what they mean. 00:03:38.543 --> 00:03:41.410 align:middle line:84% How do you give it a prompt, and then steer what gets produced? 00:03:41.410 --> 00:03:42.210 align:middle line:90% All right. 00:03:42.210 --> 00:03:44.610 align:middle line:84% So let's say you want to build a model that 00:03:44.610 --> 00:03:49.610 align:middle line:84% can be used to generate images of stately college buildings. 00:03:49.610 --> 00:03:51.650 align:middle line:84% Obviously, our very own Killian Court 00:03:51.650 --> 00:03:54.880 align:middle line:84% is the finest example of such a thing. 00:03:54.880 --> 00:03:57.080 align:middle line:84% But let's say you want to do that. 00:03:57.080 --> 00:04:00.220 align:middle line:84% So what you do is you, as we always do with machine learning, 00:04:00.220 --> 00:04:01.620 align:middle line:90% we collect a bunch of data. 00:04:01.620 --> 00:04:04.080 align:middle line:84% In this particular case, we collect a whole bunch of images 00:04:04.080 --> 00:04:06.200 align:middle line:90% of stately college buildings. 00:04:06.200 --> 00:04:07.760 align:middle line:84% And what you see here is literally me 00:04:07.760 --> 00:04:10.440 align:middle line:84% just doing a Google image search with the query stately college 00:04:10.440 --> 00:04:11.280 align:middle line:90% buildings. 00:04:11.280 --> 00:04:13.600 align:middle line:84% So this is the kind of stuff you get. 00:04:13.600 --> 00:04:15.600 align:middle line:84% So you have your training data at your disposal. 00:04:15.600 --> 00:04:17.040 align:middle line:90% It's ready to go. 00:04:17.040 --> 00:04:20.079 align:middle line:84% Now, the question is, if you have such a model, let's say. 00:04:20.079 --> 00:04:23.140 align:middle line:84% And obviously, we talk about how to build such a model very soon. 00:04:23.140 --> 00:04:24.760 align:middle line:84% But let's say you have such a model, 00:04:24.760 --> 00:04:26.980 align:middle line:84% and every time you sample this model, 00:04:26.980 --> 00:04:29.320 align:middle line:84% every time you ask the model, hey, give me an image, 00:04:29.320 --> 00:04:31.320 align:middle line:84% you obviously want it to give a different image, 00:04:31.320 --> 00:04:34.560 align:middle line:90% otherwise it's kind of boring. 00:04:34.560 --> 00:04:36.340 align:middle line:84% Maybe you want the Killian Court, 00:04:36.340 --> 00:04:39.200 align:middle line:84% maybe you want the rotunda from University of Virginia. 00:04:39.200 --> 00:04:42.260 align:middle line:90% Any UVA alums here? 00:04:42.260 --> 00:04:42.760 align:middle line:90% Nobody? 00:04:42.760 --> 00:04:45.220 align:middle line:90% OK. 00:04:45.220 --> 00:04:48.040 align:middle line:84% So the question is, how can we actually get it to randomly give 00:04:48.040 --> 00:04:49.340 align:middle line:90% us different images? 00:04:49.340 --> 00:04:51.990 align:middle line:84% But they all have to be stately college buildings. 00:04:51.990 --> 00:04:54.310 align:middle line:84% It can't be just some random stuff. 00:04:54.310 --> 00:04:55.550 align:middle line:90% So how do you do that? 00:04:55.550 --> 00:04:59.670 align:middle line:84% And the way we do that, and I still find it really astonishing 00:04:59.670 --> 00:05:01.470 align:middle line:84% that this approach actually works. 00:05:01.470 --> 00:05:05.790 align:middle line:84% The way we do that is that we actually give it noise. 00:05:05.790 --> 00:05:07.590 align:middle line:84% And I will define very precisely what 00:05:07.590 --> 00:05:10.850 align:middle line:90% I mean by noise in just a bit. 00:05:10.850 --> 00:05:14.910 align:middle line:84% Basically, assume an image in which all the pixel values 00:05:14.910 --> 00:05:18.310 align:middle line:90% are randomly picked. 00:05:18.310 --> 00:05:20.390 align:middle line:84% So every time you generate a random image 00:05:20.390 --> 00:05:22.110 align:middle line:84% and you give it to the model, it will 00:05:22.110 --> 00:05:25.190 align:middle line:84% use that random starting point and then create 00:05:25.190 --> 00:05:26.130 align:middle line:90% an image for you. 00:05:26.130 --> 00:05:29.590 align:middle line:84% And because, by definition, if you choose noise randomly, 00:05:29.590 --> 00:05:32.370 align:middle line:84% they are obviously going to be different each time. 00:05:32.370 --> 00:05:34.870 align:middle line:84% It's hopefully going to generate a different image. 00:05:34.870 --> 00:05:38.370 align:middle line:84% But if the model is trained on stately college buildings, 00:05:38.370 --> 00:05:40.990 align:middle line:84% it will produce images of stately college buildings. 00:05:40.990 --> 00:05:44.350 align:middle line:84% It's not going to produce a picture of a Labrador Retriever. 00:05:44.350 --> 00:05:47.430 align:middle line:84% So that's basically what we're going to do. 00:05:47.430 --> 00:05:51.680 align:middle line:84% Now, if you look at something like this, the first question, 00:05:51.680 --> 00:05:53.500 align:middle line:84% of course, is that how can we train a model 00:05:53.500 --> 00:05:56.140 align:middle line:84% to generate an image from pure noise? 00:05:56.140 --> 00:05:59.102 align:middle line:90% This just sounds ridiculous. 00:05:59.102 --> 00:06:01.060 align:middle line:84% You basically give it a bunch of random numbers 00:06:01.060 --> 00:06:04.540 align:middle line:90% and say, give me clean code. 00:06:04.540 --> 00:06:06.120 align:middle line:90% It feels really ridiculous. 00:06:06.120 --> 00:06:09.018 align:middle line:84% And at that point, folks can come to a stop 00:06:09.018 --> 00:06:11.060 align:middle line:84% and say, all right, this approach is probably not 00:06:11.060 --> 00:06:12.180 align:middle line:90% going to take me anywhere. 00:06:12.180 --> 00:06:14.060 align:middle line:90% It's a bit of a dead end. 00:06:14.060 --> 00:06:18.340 align:middle line:84% But then, some clever people had this very interesting idea. 00:06:18.340 --> 00:06:24.100 align:middle line:84% They said, it's not clear how to do this. 00:06:24.100 --> 00:06:28.000 align:middle line:84% Just a quick aside, there's this really amazing book, 00:06:28.000 --> 00:06:30.780 align:middle line:84% which was published maybe 50 years ago, maybe earlier 00:06:30.780 --> 00:06:34.540 align:middle line:84% than that, called How to Solve It, by George Polya. 00:06:34.540 --> 00:06:38.020 align:middle line:84% George Polya was an eminent mathematician, 00:06:38.020 --> 00:06:40.640 align:middle line:84% and he wrote this small book called How to Solve It, 00:06:40.640 --> 00:06:43.260 align:middle line:84% and it lists a whole bunch of heuristics 00:06:43.260 --> 00:06:45.920 align:middle line:84% that mathematicians use when they solve problems. 00:06:45.920 --> 00:06:48.460 align:middle line:84% And perhaps the most commonly used heuristic 00:06:48.460 --> 00:06:51.917 align:middle line:90% is just reverse the question. 00:06:51.917 --> 00:06:54.500 align:middle line:84% Just reverse the question and see if anything comes out of it. 00:06:54.500 --> 00:06:56.417 align:middle line:84% Most of the time, nothing will come out of it. 00:06:56.417 --> 00:06:58.980 align:middle line:84% But maybe some other times, something amazing comes out. 00:06:58.980 --> 00:07:01.140 align:middle line:84% This is a great example of that heuristic at work. 00:07:01.140 --> 00:07:02.940 align:middle line:90% We don't know how to do this. 00:07:02.940 --> 00:07:05.640 align:middle line:84% So the question is, can we do the reverse? 00:07:05.640 --> 00:07:07.140 align:middle line:84% If I give you Killian Court, can you 00:07:07.140 --> 00:07:10.380 align:middle line:90% produce noise out of it for me? 00:07:10.380 --> 00:07:15.140 align:middle line:84% And the answer is yeah, of course, we can do that. 00:07:15.140 --> 00:07:18.560 align:middle line:84% Given an image, we can easily create a noisy version of it. 00:07:18.560 --> 00:07:20.780 align:middle line:84% So you can take the original image, 00:07:20.780 --> 00:07:22.817 align:middle line:84% you can add some noise to it to get this. 00:07:22.817 --> 00:07:24.400 align:middle line:84% And you keep on adding a lot of noise. 00:07:24.400 --> 00:07:26.525 align:middle line:84% And finally, you'll get something that's basically, 00:07:26.525 --> 00:07:29.340 align:middle line:84% you can't tell that is Killian Court anymore. 00:07:29.340 --> 00:07:31.100 align:middle line:84% This process, the reverse process, 00:07:31.100 --> 00:07:34.380 align:middle line:90% is actually very easy to do. 00:07:34.380 --> 00:07:36.340 align:middle line:84% By the way, for folks who may not 00:07:36.340 --> 00:07:38.660 align:middle line:84% be very familiar with this notion of adding noise 00:07:38.660 --> 00:07:40.600 align:middle line:84% to an image or making an image noisy, 00:07:40.600 --> 00:07:44.880 align:middle line:84% let me just show you in a Colab, just a minute, how easy it is. 00:07:44.880 --> 00:07:47.450 align:middle line:90% 00:07:47.450 --> 00:07:48.070 align:middle line:90% All right. 00:07:48.070 --> 00:07:52.430 align:middle line:84% So let's say we import a bunch of these things. 00:07:52.430 --> 00:07:53.715 align:middle line:90% As usual, we have NumPy. 00:07:53.715 --> 00:07:55.090 align:middle line:84% And so there is this thing called 00:07:55.090 --> 00:07:57.690 align:middle line:84% the Python Imaging Library, PIL, which 00:07:57.690 --> 00:07:59.930 align:middle line:84% is very handy for image manipulations. 00:07:59.930 --> 00:08:01.270 align:middle line:90% So we import that. 00:08:01.270 --> 00:08:03.490 align:middle line:84% And then, I just literally read this image in. 00:08:03.490 --> 00:08:04.910 align:middle line:90% I uploaded it before class. 00:08:04.910 --> 00:08:06.202 align:middle line:90% Let's just make sure it's here. 00:08:06.202 --> 00:08:06.770 align:middle line:90% OK, good. 00:08:06.770 --> 00:08:07.850 align:middle line:90% killian.png. 00:08:07.850 --> 00:08:09.650 align:middle line:90% So I read this image. 00:08:09.650 --> 00:08:10.890 align:middle line:90% OK. 00:08:10.890 --> 00:08:14.890 align:middle line:84% And then, once I read it, I convert it into a NumPy array. 00:08:14.890 --> 00:08:17.770 align:middle line:84% And then, remember, in any color image, 00:08:17.770 --> 00:08:20.390 align:middle line:84% you have three tables of numbers. 00:08:20.390 --> 00:08:23.870 align:middle line:84% There is a number for each pixel for red, blue, and green. 00:08:23.870 --> 00:08:26.690 align:middle line:84% And then, each number is between 0 and 255. 00:08:26.690 --> 00:08:28.770 align:middle line:84% And so here, what we do is we divide everything 00:08:28.770 --> 00:08:31.790 align:middle line:84% by 255 just to normalize it so it's all between 0 and 1. 00:08:31.790 --> 00:08:33.610 align:middle line:84% And we have done this in the past. 00:08:33.610 --> 00:08:36.350 align:middle line:90% I do that here. 00:08:36.350 --> 00:08:38.912 align:middle line:84% So let me just read this back in, convert it. 00:08:38.912 --> 00:08:40.370 align:middle line:84% And then, if you look at the shape, 00:08:40.370 --> 00:08:44.800 align:middle line:84% it's basically a 411 by 583 times 3. 00:08:44.800 --> 00:08:46.760 align:middle line:84% Three channels, as we have seen before. 00:08:46.760 --> 00:08:49.160 align:middle line:90% And then, let's just show it. 00:08:49.160 --> 00:08:50.840 align:middle line:90% All right, that's the picture. 00:08:50.840 --> 00:08:52.480 align:middle line:84% So now, what we want to do is we want 00:08:52.480 --> 00:08:54.240 align:middle line:90% to add noise to this picture. 00:08:54.240 --> 00:08:58.960 align:middle line:84% All we have to do for each pixel, 00:08:58.960 --> 00:09:02.880 align:middle line:84% we basically, randomly pick a normal variable, 00:09:02.880 --> 00:09:05.760 align:middle line:84% a normal distribution, normally distributed random variable 00:09:05.760 --> 00:09:08.740 align:middle line:84% with a mean of 0 and a small standard deviation. 00:09:08.740 --> 00:09:10.380 align:middle line:90% So it's like a small number. 00:09:10.380 --> 00:09:13.520 align:middle line:84% And then, we just literally add that number to every pixel. 00:09:13.520 --> 00:09:16.262 align:middle line:84% But for every pixel we sample, every pixel we sample, 00:09:16.262 --> 00:09:18.720 align:middle line:84% it's not like we sample once an array of all the pixels, we 00:09:18.720 --> 00:09:20.520 align:middle line:90% sample for every pixel. 00:09:20.520 --> 00:09:23.840 align:middle line:84% And so the way you do that is basically, literally 00:09:23.840 --> 00:09:26.560 align:middle line:90% np.random.normal. 00:09:26.560 --> 00:09:29.540 align:middle line:84% And then, this 0.3 here is a standard deviation. 00:09:29.540 --> 00:09:34.560 align:middle line:84% And we tell it, generate as many of these things as the shape 00:09:34.560 --> 00:09:36.320 align:middle line:90% of the image that I gave you. 00:09:36.320 --> 00:09:39.940 align:middle line:84% And then, add each one of these numbers to the original image. 00:09:39.940 --> 00:09:42.150 align:middle line:90% You get this noisy image. 00:09:42.150 --> 00:09:44.210 align:middle line:84% So if this is the original image, 00:09:44.210 --> 00:09:46.210 align:middle line:84% these are all the values between 0 and 1. 00:09:46.210 --> 00:09:47.893 align:middle line:84% And then, you add do this noisy image. 00:09:47.893 --> 00:09:49.810 align:middle line:84% You can see the numbers have become different. 00:09:49.810 --> 00:09:52.590 align:middle line:84% The 0.23 has become 0.18, 0.15 has become 00:09:52.590 --> 00:09:54.830 align:middle line:90% -0.17, and so on and so forth. 00:09:54.830 --> 00:09:57.610 align:middle line:84% You've just added a small number random to everything. 00:09:57.610 --> 00:10:00.290 align:middle line:84% But as you can see here, now you have some negative numbers. 00:10:00.290 --> 00:10:02.290 align:middle line:84% You may have some numbers that's greater than 1. 00:10:02.290 --> 00:10:04.690 align:middle line:84% And we do want everything to be between 0 and 1. 00:10:04.690 --> 00:10:06.590 align:middle line:84% So all we do is we do this thing called 00:10:06.590 --> 00:10:10.390 align:middle line:84% clip it, where essentially, values smaller than 0 00:10:10.390 --> 00:10:13.010 align:middle line:84% are set to 0, values greater than 1 are set to 1. 00:10:13.010 --> 00:10:15.350 align:middle line:90% And so we'll just do that. 00:10:15.350 --> 00:10:16.010 align:middle line:90% That's it. 00:10:16.010 --> 00:10:17.810 align:middle line:84% Everything over 1, squashed to 1. 00:10:17.810 --> 00:10:19.370 align:middle line:90% Everything under 0, set to 0. 00:10:19.370 --> 00:10:21.030 align:middle line:90% Others, leave it unchanged. 00:10:21.030 --> 00:10:23.370 align:middle line:84% Now, it's again, well-behaved between 0 and 1. 00:10:23.370 --> 00:10:28.350 align:middle line:84% And we can just plot it, and you get this. 00:10:28.350 --> 00:10:28.850 align:middle line:90% That's it. 00:10:28.850 --> 00:10:31.100 align:middle line:84% That's all it takes to actually add noise to an image. 00:10:31.100 --> 00:10:33.578 align:middle line:90% One line of NumPy. 00:10:33.578 --> 00:10:35.870 align:middle line:84% Obviously, you can just put this whole thing in a loop, 00:10:35.870 --> 00:10:37.662 align:middle line:84% and keep increasing that standard deviation 00:10:37.662 --> 00:10:40.210 align:middle line:84% number from 0.3, 0.4, 0.5, so on and so forth. 00:10:40.210 --> 00:10:42.380 align:middle line:84% And when you do that, you get this nice sequence 00:10:42.380 --> 00:10:45.740 align:middle line:84% of clean code, and all the way to some very, very noisy version 00:10:45.740 --> 00:10:46.860 align:middle line:90% of clean code. 00:10:46.860 --> 00:10:47.620 align:middle line:90% That's it. 00:10:47.620 --> 00:10:49.690 align:middle line:84% So that's the basic idea of adding noise. 00:10:49.690 --> 00:10:52.940 align:middle line:90% 00:10:52.940 --> 00:10:57.180 align:middle line:90% Any questions on the mechanics? 00:10:57.180 --> 00:10:58.380 align:middle line:90% OK, good. 00:10:58.380 --> 00:11:01.160 align:middle line:90% So we can add random numbers. 00:11:01.160 --> 00:11:03.100 align:middle line:84% And we can, by increasing the magnitude 00:11:03.100 --> 00:11:06.740 align:middle line:84% of the standard deviation of these normal random variables, 00:11:06.740 --> 00:11:08.820 align:middle line:90% we can make the image noisier. 00:11:08.820 --> 00:11:14.940 align:middle line:84% So that suggests a really interesting idea. 00:11:14.940 --> 00:11:16.080 align:middle line:90% What idea would that be? 00:11:16.080 --> 00:11:19.620 align:middle line:90% 00:11:19.620 --> 00:11:20.140 align:middle line:90% Yeah. 00:11:20.140 --> 00:11:21.320 align:middle line:90% STUDENT: Doing the opposite. 00:11:21.320 --> 00:11:25.108 align:middle line:84% PROFESSOR: Could you please, microphone please. 00:11:25.108 --> 00:11:26.400 align:middle line:90% STUDENT: By doing the opposite. 00:11:26.400 --> 00:11:29.580 align:middle line:84% Like, recreating the image from the noise. 00:11:29.580 --> 00:11:32.980 align:middle line:84% PROFESSOR: So we are trying to create the image from the noise, 00:11:32.980 --> 00:11:36.740 align:middle line:90% but that feels a little hard. 00:11:36.740 --> 00:11:38.010 align:middle line:90% So what exactly can we do? 00:11:38.010 --> 00:11:39.260 align:middle line:90% Be a little bit more specific. 00:11:39.260 --> 00:11:44.490 align:middle line:90% 00:11:44.490 --> 00:11:46.770 align:middle line:84% So here, we have the ability to take any image 00:11:46.770 --> 00:11:49.615 align:middle line:84% and add any amount of noise to it. 00:11:49.615 --> 00:11:52.550 align:middle line:84% That's the data we have that is Killian Court. 00:11:52.550 --> 00:11:54.980 align:middle line:84% And there's various noisy versions of Killian Court, 00:11:54.980 --> 00:11:57.230 align:middle line:84% like that for the rotunda, the University of Virginia, 00:11:57.230 --> 00:11:58.210 align:middle line:90% and so on and so forth. 00:11:58.210 --> 00:11:59.585 align:middle line:84% STUDENT: I would assume you would 00:11:59.585 --> 00:12:02.090 align:middle line:84% do some kind of loss function for the final image 00:12:02.090 --> 00:12:05.170 align:middle line:84% that you get, and compare it with the original image 00:12:05.170 --> 00:12:06.690 align:middle line:90% that you trained it on. 00:12:06.690 --> 00:12:10.610 align:middle line:90% And then, fine tune as you go. 00:12:10.610 --> 00:12:13.210 align:middle line:84% PROFESSOR: OK, you're on the right track. 00:12:13.210 --> 00:12:15.430 align:middle line:90% Any other proposals? 00:12:15.430 --> 00:12:18.570 align:middle line:90% 00:12:18.570 --> 00:12:21.130 align:middle line:84% STUDENT: I think we could try to train the neural network 00:12:21.130 --> 00:12:25.930 align:middle line:84% to reconstruct the image going from the noisy one, that would 00:12:25.930 --> 00:12:30.730 align:middle line:84% have all the set with images, find their noise counterpart, 00:12:30.730 --> 00:12:34.810 align:middle line:84% and train them to do the opposite, the neural network 00:12:34.810 --> 00:12:38.012 align:middle line:90% to do the opposite task. 00:12:38.012 --> 00:12:40.220 align:middle line:84% PROFESSOR: Yup, that's definitely on the right track. 00:12:40.220 --> 00:12:41.762 align:middle line:84% That's definitely on the right track. 00:12:41.762 --> 00:12:42.920 align:middle line:90% Yeah, good ideas. 00:12:42.920 --> 00:12:47.480 align:middle line:84% So what we do more concretely, is 00:12:47.480 --> 00:12:50.480 align:middle line:84% we can take each image in the training data 00:12:50.480 --> 00:12:53.800 align:middle line:84% and create noisy versions of it, as we have seen before. 00:12:53.800 --> 00:12:57.040 align:middle line:84% And then, what we do is that we say 00:12:57.040 --> 00:13:01.200 align:middle line:84% we can create x, y training data pairs, input output 00:13:01.200 --> 00:13:03.920 align:middle line:90% pairs, from all these images. 00:13:03.920 --> 00:13:07.920 align:middle line:84% So specifically, what we do is we 00:13:07.920 --> 00:13:12.360 align:middle line:84% take the slightly noisy version of Killian Court 00:13:12.360 --> 00:13:16.440 align:middle line:84% and call it the input, and we take the nice version of Killian 00:13:16.440 --> 00:13:19.160 align:middle line:90% Court and call it the output. 00:13:19.160 --> 00:13:22.560 align:middle line:90% That's the y1, x1 pair. 00:13:22.560 --> 00:13:28.720 align:middle line:84% And then, we get y2, x2, we get y3, x3 and all the way. 00:13:28.720 --> 00:13:33.452 align:middle line:84% So at any point in time, the relationship between x and y, 00:13:33.452 --> 00:13:35.917 align:middle line:84% what's the relationship between x and y? 00:13:35.917 --> 00:13:38.250 align:middle line:84% If you set it up like this, as the input and the output. 00:13:38.250 --> 00:13:43.110 align:middle line:90% 00:13:43.110 --> 00:13:45.630 align:middle line:84% STUDENT: It's the set of standard deviations 00:13:45.630 --> 00:13:49.470 align:middle line:84% and the values which we obtain for these pixels. 00:13:49.470 --> 00:13:53.070 align:middle line:84% Those are, like, weights, which correspond to another feature. 00:13:53.070 --> 00:13:54.750 align:middle line:84% PROFESSOR: Right, or maybe I was looking 00:13:54.750 --> 00:13:57.390 align:middle line:84% for something simpler, which was that-- that's correct. 00:13:57.390 --> 00:13:58.910 align:middle line:84% What I was looking for is really, 00:13:58.910 --> 00:14:00.550 align:middle line:84% the relationship between x and y. 00:14:00.550 --> 00:14:03.590 align:middle line:84% X is an image, any image, and y happens 00:14:03.590 --> 00:14:07.950 align:middle line:84% to be a slightly less noisy version of the image. 00:14:07.950 --> 00:14:12.350 align:middle line:84% The slightly less noisy is really, really important. 00:14:12.350 --> 00:14:14.610 align:middle line:84% You're not going from Killian Court, 00:14:14.610 --> 00:14:18.510 align:middle line:84% you're not going from the image to full noise. 00:14:18.510 --> 00:14:20.310 align:middle line:90% That's an impossible leap. 00:14:20.310 --> 00:14:23.270 align:middle line:84% You're going from the image to a slightly noisy version 00:14:23.270 --> 00:14:24.670 align:middle line:90% of that image. 00:14:24.670 --> 00:14:30.470 align:middle line:84% It is that slightly that allows all the magic to happen. 00:14:30.470 --> 00:14:33.020 align:middle line:90% So that's what we have. 00:14:33.020 --> 00:14:35.600 align:middle line:84% And so here, what we can do with these x, 00:14:35.600 --> 00:14:37.860 align:middle line:90% y pairs, when you have an-- 00:14:37.860 --> 00:14:38.908 align:middle line:90% so here's the thing. 00:14:38.908 --> 00:14:40.700 align:middle line:84% This is like a larger comment about machine 00:14:40.700 --> 00:14:41.825 align:middle line:90% learning and deep learning. 00:14:41.825 --> 00:14:45.133 align:middle line:90% 00:14:45.133 --> 00:14:47.300 align:middle line:84% Basically, what machine learning, deep learning are, 00:14:47.300 --> 00:14:49.780 align:middle line:84% are really, it's like this black box 00:14:49.780 --> 00:14:53.540 align:middle line:84% where if you can find interesting input output pairs, 00:14:53.540 --> 00:14:56.400 align:middle line:84% you can learn a function to go from the input to the output. 00:14:56.400 --> 00:14:57.700 align:middle line:90% That's it. 00:14:57.700 --> 00:15:01.120 align:middle line:84% But this sounds kind of simple when I describe it like that. 00:15:01.120 --> 00:15:04.100 align:middle line:84% But there are some incredibly non-obvious ways 00:15:04.100 --> 00:15:06.155 align:middle line:90% of applying this idea. 00:15:06.155 --> 00:15:07.780 align:middle line:84% So for example, a few years ago, Google 00:15:07.780 --> 00:15:09.445 align:middle line:84% had this thing, which may actually 00:15:09.445 --> 00:15:10.820 align:middle line:84% be in production in Google Sheets 00:15:10.820 --> 00:15:15.440 align:middle line:84% now, where whenever you choose a bunch of numbers, 00:15:15.440 --> 00:15:17.340 align:middle line:84% a range of numbers in a spreadsheet 00:15:17.340 --> 00:15:19.460 align:middle line:84% and then go into another cell, it will immediately 00:15:19.460 --> 00:15:21.460 align:middle line:90% suggest informative for you. 00:15:21.460 --> 00:15:24.020 align:middle line:90% Where is that coming from? 00:15:24.020 --> 00:15:27.980 align:middle line:84% It's because all the Google Sheets users all over the world, 00:15:27.980 --> 00:15:31.060 align:middle line:84% they have been creating all these numbers with formulas. 00:15:31.060 --> 00:15:33.210 align:middle line:84% So someone says, look, wait a second. 00:15:33.210 --> 00:15:36.470 align:middle line:84% We have all this data on people choosing a range of numbers, 00:15:36.470 --> 00:15:38.210 align:middle line:90% and then entering a formula. 00:15:38.210 --> 00:15:40.330 align:middle line:84% So let's imagine the range as the input 00:15:40.330 --> 00:15:43.130 align:middle line:90% and the formula as the output. 00:15:43.130 --> 00:15:46.170 align:middle line:84% And let's just give a million examples of this pair 00:15:46.170 --> 00:15:47.850 align:middle line:84% and see if anything comes out of it. 00:15:47.850 --> 00:15:50.610 align:middle line:90% And boom, you get that feature. 00:15:50.610 --> 00:15:55.368 align:middle line:84% So similarly here, x is an image, less noisy version 00:15:55.368 --> 00:15:55.910 align:middle line:90% of the image. 00:15:55.910 --> 00:16:01.890 align:middle line:84% What that means is that we can build a denoising network. 00:16:01.890 --> 00:16:04.610 align:middle line:84% We can take an image and we can build 00:16:04.610 --> 00:16:08.430 align:middle line:84% a network using all these x, y pairs to slightly denoise it. 00:16:08.430 --> 00:16:12.426 align:middle line:90% 00:16:12.426 --> 00:16:15.110 align:middle line:90% And so how do we do it? 00:16:15.110 --> 00:16:17.890 align:middle line:84% We just run stochastic gradient descent on the data. 00:16:17.890 --> 00:16:20.530 align:middle line:84% We have a network, which has x and y, 00:16:20.530 --> 00:16:23.350 align:middle line:84% and then y is slightly less noisy version. 00:16:23.350 --> 00:16:26.313 align:middle line:84% And then, boom, boom, boom, boom, boom. 00:16:26.313 --> 00:16:27.230 align:middle line:90% You're just a network. 00:16:27.230 --> 00:16:28.313 align:middle line:90% It has a bunch of weights. 00:16:28.313 --> 00:16:30.080 align:middle line:84% We have the right answer in terms 00:16:30.080 --> 00:16:32.482 align:middle line:90% of what the images need to be. 00:16:32.482 --> 00:16:34.940 align:middle line:84% We can do stochastic gradient descent or Adam or something. 00:16:34.940 --> 00:16:36.660 align:middle line:84% And before you know it, if you have enough data, 00:16:36.660 --> 00:16:39.080 align:middle line:84% you have a network which can denoise anything you give it. 00:16:39.080 --> 00:16:40.160 align:middle line:90% OK. 00:16:40.160 --> 00:16:41.960 align:middle line:90% You had a question. 00:16:41.960 --> 00:16:43.320 align:middle line:90% STUDENT: Why slightly? 00:16:43.320 --> 00:16:44.845 align:middle line:90% PROFESSOR: Why slightly? 00:16:44.845 --> 00:16:46.220 align:middle line:84% We'll come back to that question. 00:16:46.220 --> 00:16:50.000 align:middle line:84% The reason is that in general, you 00:16:50.000 --> 00:16:52.560 align:middle line:84% have to do what you can to help the model. 00:16:52.560 --> 00:16:56.380 align:middle line:84% And this is the proverbial, there is an old adage, 00:16:56.380 --> 00:16:59.840 align:middle line:84% you can't cross a ditch in two jumps. 00:16:59.840 --> 00:17:02.460 align:middle line:84% It's too big, so you can't do it. 00:17:02.460 --> 00:17:05.240 align:middle line:84% So what you do is you create a bridge to go from here to there. 00:17:05.240 --> 00:17:08.560 align:middle line:84% And so what you do is, if you can slightly denoise something 00:17:08.560 --> 00:17:10.560 align:middle line:84% really well, well, I can actually 00:17:10.560 --> 00:17:12.359 align:middle line:84% denoise anything you want really well using 00:17:12.359 --> 00:17:17.055 align:middle line:84% that fundamental capability, as you will see in a second. 00:17:17.055 --> 00:17:18.180 align:middle line:90% STUDENT: Just to follow up. 00:17:18.180 --> 00:17:19.819 align:middle line:84% So if you go back the last slide, 00:17:19.819 --> 00:17:23.960 align:middle line:84% I could have created the same thing as, that is my x1 00:17:23.960 --> 00:17:25.359 align:middle line:90% and that is my y1. 00:17:25.359 --> 00:17:28.250 align:middle line:84% Then, the second one is x2 and still, this is the y. 00:17:28.250 --> 00:17:30.550 align:middle line:84% So there is effectively, there is a learning there 00:17:30.550 --> 00:17:34.970 align:middle line:84% that it could have taken from those pairs and come back with, 00:17:34.970 --> 00:17:36.575 align:middle line:90% OK, this is also a possibility. 00:17:36.575 --> 00:17:37.700 align:middle line:90% This is also a possibility. 00:17:37.700 --> 00:17:38.408 align:middle line:90% PROFESSOR: Right. 00:17:38.408 --> 00:17:40.790 align:middle line:84% STUDENT: And found out that noise matrix 00:17:40.790 --> 00:17:42.175 align:middle line:90% that it can subtract. 00:17:42.175 --> 00:17:43.550 align:middle line:84% PROFESSOR: Yeah, so the thing is, 00:17:43.550 --> 00:17:46.670 align:middle line:84% you want to make sure that each time the amount of learning it 00:17:46.670 --> 00:17:50.350 align:middle line:84% has to do is as bounded and small as possible. 00:17:50.350 --> 00:17:52.710 align:middle line:84% If you give it some starting point and an ending point 00:17:52.710 --> 00:17:54.870 align:middle line:84% and keep moving this ending point, 00:17:54.870 --> 00:17:57.110 align:middle line:84% the gap is still really high for the first several 00:17:57.110 --> 00:17:59.190 align:middle line:90% of those starting points. 00:17:59.190 --> 00:18:01.510 align:middle line:90% That's the problem. 00:18:01.510 --> 00:18:02.256 align:middle line:90% OK. 00:18:02.256 --> 00:18:03.890 align:middle line:90% So to come back to this. 00:18:03.890 --> 00:18:05.950 align:middle line:84% So we can build a denoising model. 00:18:05.950 --> 00:18:07.170 align:middle line:90% We can do this. 00:18:07.170 --> 00:18:09.470 align:middle line:84% And now, once you have built such a thing 00:18:09.470 --> 00:18:11.470 align:middle line:84% you give it some noisy thing, and then it 00:18:11.470 --> 00:18:15.430 align:middle line:84% will give you a slightly less noisy version of it. 00:18:15.430 --> 00:18:18.470 align:middle line:84% The resolution is going to go up slightly if you do that. 00:18:18.470 --> 00:18:20.230 align:middle line:84% This, of course, suggests the obvious way 00:18:20.230 --> 00:18:24.310 align:middle line:84% in which you would use it, which is that once you train it, 00:18:24.310 --> 00:18:27.552 align:middle line:90% we can solve this problem. 00:18:27.552 --> 00:18:29.540 align:middle line:84% And how can we solve this problem? 00:18:29.540 --> 00:18:33.580 align:middle line:84% So what you do is you start with pure noise, 00:18:33.580 --> 00:18:37.060 align:middle line:90% and then repeatedly denoise it. 00:18:37.060 --> 00:18:37.900 align:middle line:90% OK. 00:18:37.900 --> 00:18:39.148 align:middle line:90% You get that, you get that. 00:18:39.148 --> 00:18:40.940 align:middle line:84% And then, before you know it, Killian Court 00:18:40.940 --> 00:18:42.080 align:middle line:90% has emerged from the fog. 00:18:42.080 --> 00:18:44.790 align:middle line:90% 00:18:44.790 --> 00:18:47.630 align:middle line:84% It's pretty insane that it actually works, this idea. 00:18:47.630 --> 00:18:53.780 align:middle line:90% 00:18:53.780 --> 00:18:56.320 align:middle line:84% So the model will generate a sequence of less noisy images, 00:18:56.320 --> 00:18:58.980 align:middle line:84% and the final one you have is the answer. 00:18:58.980 --> 00:19:01.960 align:middle line:84% Now, there's a whole bunch of detail here, 00:19:01.960 --> 00:19:05.860 align:middle line:84% which I'm glossing over, about, OK, how many times 00:19:05.860 --> 00:19:09.400 align:middle line:84% must we run this loop to get to a really good picture? 00:19:09.400 --> 00:19:12.540 align:middle line:84% The short answer is initially, it was like you have to run it 00:19:12.540 --> 00:19:14.480 align:middle line:90% like 1,000 times. 00:19:14.480 --> 00:19:16.723 align:middle line:84% Each denoising step was like a baby step. 00:19:16.723 --> 00:19:18.140 align:middle line:84% You have to do it a thousand times 00:19:18.140 --> 00:19:19.540 align:middle line:90% to get a really good answer. 00:19:19.540 --> 00:19:21.673 align:middle line:84% Again, research has been very active in the area, 00:19:21.673 --> 00:19:22.840 align:middle line:90% continues to be very active. 00:19:22.840 --> 00:19:25.940 align:middle line:84% Now, we can, I think, do it with 50 steps or 100 steps. 00:19:25.940 --> 00:19:28.700 align:middle line:84% But diffusion models, like this, they 00:19:28.700 --> 00:19:32.153 align:middle line:84% tend to take more time than a large language model. 00:19:32.153 --> 00:19:33.820 align:middle line:84% Which is why if you give a prompt to one 00:19:33.820 --> 00:19:36.403 align:middle line:84% of these models like Midjourney, it will take some time for it 00:19:36.403 --> 00:19:37.900 align:middle line:90% to come back with an image. 00:19:37.900 --> 00:19:40.180 align:middle line:84% And the reason for the delay is because it's 00:19:40.180 --> 00:19:43.700 align:middle line:84% going through this incremental denoising loop. 00:19:43.700 --> 00:19:45.365 align:middle line:90% Yeah. 00:19:45.365 --> 00:19:46.740 align:middle line:84% STUDENT: From this, we understand 00:19:46.740 --> 00:19:49.500 align:middle line:84% that each, the final noise output sample 00:19:49.500 --> 00:19:52.460 align:middle line:84% would be very particular to each image in the beginning. 00:19:52.460 --> 00:19:56.380 align:middle line:84% So, I mean, say if you take two images, 00:19:56.380 --> 00:19:58.680 align:middle line:84% finally we are getting a start image. 00:19:58.680 --> 00:20:01.900 align:middle line:84% And after, when we start noising it, and the final output 00:20:01.900 --> 00:20:04.108 align:middle line:84% we get with the noise sample, it will be two distinct 00:20:04.108 --> 00:20:05.108 align:middle line:90% for each of them, right? 00:20:05.108 --> 00:20:05.920 align:middle line:90% PROFESSOR: Correct. 00:20:05.920 --> 00:20:08.100 align:middle line:84% STUDENT: But when we are picking up image 00:20:08.100 --> 00:20:11.620 align:middle line:84% to generate a diffusion model and we work backwards, 00:20:11.620 --> 00:20:14.040 align:middle line:84% we may not have the exact thing available to us, 00:20:14.040 --> 00:20:15.162 align:middle line:90% what was there initially. 00:20:15.162 --> 00:20:16.620 align:middle line:84% PROFESSOR: No, no, the thing is, we 00:20:16.620 --> 00:20:18.580 align:middle line:84% don't want to necessarily regenerate images 00:20:18.580 --> 00:20:20.558 align:middle line:90% that were in the training data. 00:20:20.558 --> 00:20:21.600 align:middle line:90% That's kind of pointless. 00:20:21.600 --> 00:20:24.250 align:middle line:90% We want to generate new images. 00:20:24.250 --> 00:20:29.490 align:middle line:84% And for new images, we just use noise as a starting point. 00:20:29.490 --> 00:20:31.370 align:middle line:84% The fact that Killian Court was here, 00:20:31.370 --> 00:20:33.490 align:middle line:84% and then the fully noised version of Killian Court 00:20:33.490 --> 00:20:35.055 align:middle line:84% is here, that is used for training. 00:20:35.055 --> 00:20:36.430 align:middle line:84% And once you use it for training, 00:20:36.430 --> 00:20:38.410 align:middle line:84% you don't need it anymore because you're not 00:20:38.410 --> 00:20:40.370 align:middle line:84% trying to recreate Killian Court again. 00:20:40.370 --> 00:20:42.450 align:middle line:84% You want to create new images which 00:20:42.450 --> 00:20:44.970 align:middle line:84% belong to the category of stately college buildings. 00:20:44.970 --> 00:20:47.823 align:middle line:84% And for that, you just grab noise, send it in. 00:20:47.823 --> 00:20:49.490 align:middle line:84% It gives you a stately college building. 00:20:49.490 --> 00:20:50.310 align:middle line:90% End of story. 00:20:50.310 --> 00:20:53.730 align:middle line:90% 00:20:53.730 --> 00:20:56.850 align:middle line:84% And because noise by definition is different each time you 00:20:56.850 --> 00:20:59.850 align:middle line:84% pick it, it's going to come up with a different stately college 00:20:59.850 --> 00:21:01.730 align:middle line:90% building. 00:21:01.730 --> 00:21:03.390 align:middle line:84% So the way you think about it is that-- 00:21:03.390 --> 00:21:06.970 align:middle line:90% 00:21:06.970 --> 00:21:07.550 align:middle line:90% all right. 00:21:07.550 --> 00:21:09.270 align:middle line:90% So you can think of it as this. 00:21:09.270 --> 00:21:12.610 align:middle line:90% 00:21:12.610 --> 00:21:17.250 align:middle line:84% So when you sample, think of this as the noise distribution. 00:21:17.250 --> 00:21:22.560 align:middle line:84% Each time you sample, there is a little point you pick from here. 00:21:22.560 --> 00:21:25.680 align:middle line:84% Another time you sample, maybe you get a point here. 00:21:25.680 --> 00:21:27.660 align:middle line:84% Each is just a nice distribution. 00:21:27.660 --> 00:21:28.360 align:middle line:90% That's it. 00:21:28.360 --> 00:21:30.520 align:middle line:84% What actually these things are doing 00:21:30.520 --> 00:21:35.840 align:middle line:84% is they are mapping it to the distribution of stately college 00:21:35.840 --> 00:21:41.120 align:middle line:84% buildings, which might be in a strange, crazy distribution. 00:21:41.120 --> 00:21:43.720 align:middle line:84% So each time you sample, you just go from here, 00:21:43.720 --> 00:21:47.680 align:middle line:90% and you land at a point here. 00:21:47.680 --> 00:21:50.550 align:middle line:84% And when you go from here, you land at a point there. 00:21:50.550 --> 00:21:53.740 align:middle line:90% 00:21:53.740 --> 00:21:56.120 align:middle line:84% So what you have done is when you take the training data, 00:21:56.120 --> 00:21:59.560 align:middle line:84% you basically created points here, and then 00:21:59.560 --> 00:22:02.080 align:middle line:84% found the matching noise here, and then flipped it 00:22:02.080 --> 00:22:03.768 align:middle line:84% for training, as we have seen before. 00:22:03.768 --> 00:22:05.560 align:middle line:84% And once you're done with it, you basically 00:22:05.560 --> 00:22:09.360 align:middle line:84% have a mechanism for transforming any entry 00:22:09.360 --> 00:22:12.480 align:middle line:84% in this distribution of images to an entry 00:22:12.480 --> 00:22:14.340 align:middle line:90% in this distribution of images. 00:22:14.340 --> 00:22:17.140 align:middle line:84% So it's a way to transform one distribution 00:22:17.140 --> 00:22:18.140 align:middle line:90% to another distribution. 00:22:18.140 --> 00:22:20.430 align:middle line:90% That's what's going on. 00:22:20.430 --> 00:22:21.910 align:middle line:90% All right. 00:22:21.910 --> 00:22:26.310 align:middle line:84% So there was a question, yeah, and then we'll go. 00:22:26.310 --> 00:22:29.810 align:middle line:84% STUDENT: I understand going from noise to image and back, how you 00:22:29.810 --> 00:22:31.330 align:middle line:90% call how the training works. 00:22:31.330 --> 00:22:35.210 align:middle line:84% So my question is, in some of these models today, 00:22:35.210 --> 00:22:37.470 align:middle line:84% you have, when you give it the noise 00:22:37.470 --> 00:22:40.970 align:middle line:84% now to generate an image, for example, 00:22:40.970 --> 00:22:45.210 align:middle line:84% it could generate a human with four fingers or stuff like that. 00:22:45.210 --> 00:22:49.270 align:middle line:84% So is it that the model, that the training data is not 00:22:49.270 --> 00:22:52.350 align:middle line:84% just quite enough to or as robust enough 00:22:52.350 --> 00:22:55.850 align:middle line:90% to generate that kind of detail? 00:22:55.850 --> 00:22:58.203 align:middle line:84% Can you talk through what's going on with that? 00:22:58.203 --> 00:22:58.870 align:middle line:90% PROFESSOR: Yeah. 00:22:58.870 --> 00:23:00.990 align:middle line:84% So fundamentally, what it is doing 00:23:00.990 --> 00:23:04.390 align:middle line:84% is it actually does not understand the notion of fingers 00:23:04.390 --> 00:23:08.630 align:middle line:84% and things like that, because we haven't injected any domain 00:23:08.630 --> 00:23:11.110 align:middle line:84% knowledge into this whole process, 00:23:11.110 --> 00:23:15.310 align:middle line:84% by saying that, hey, you need to generate a human body, 00:23:15.310 --> 00:23:17.760 align:middle line:84% and here are the semantics of what the human body is. 00:23:17.760 --> 00:23:21.097 align:middle line:84% It's got five fingers and all the anatomical stuff. 00:23:21.097 --> 00:23:22.180 align:middle line:90% We're not giving anything. 00:23:22.180 --> 00:23:25.360 align:middle line:84% We're literally giving it pixel values, a bunch of pictures. 00:23:25.360 --> 00:23:27.460 align:middle line:84% So everything you're seeing is basically just 00:23:27.460 --> 00:23:29.820 align:middle line:84% coming out of that very blind statistical transformation 00:23:29.820 --> 00:23:32.100 align:middle line:90% process. 00:23:32.100 --> 00:23:35.180 align:middle line:84% So you would expect that macro level details would probably 00:23:35.180 --> 00:23:38.460 align:middle line:84% get it right because there are so many right answers. 00:23:38.460 --> 00:23:42.860 align:middle line:84% So imagine it's actually it's creating the roof of a house. 00:23:42.860 --> 00:23:44.747 align:middle line:84% There could be all kinds of variations 00:23:44.747 --> 00:23:46.580 align:middle line:84% in the roof of the house and you would still 00:23:46.580 --> 00:23:48.900 align:middle line:84% think it's a roof of a house because there are 00:23:48.900 --> 00:23:50.360 align:middle line:90% many possible right answers. 00:23:50.360 --> 00:23:51.800 align:middle line:84% But when it comes to five fingers, 00:23:51.800 --> 00:23:53.160 align:middle line:84% there aren't many possible right answers, 00:23:53.160 --> 00:23:55.260 align:middle line:84% which is why you notice the error very quickly. 00:23:55.260 --> 00:23:57.343 align:middle line:84% As far as the model is concerned, it doesn't know. 00:23:57.343 --> 00:24:00.460 align:middle line:84% It's just producing a statistically plausible sample 00:24:00.460 --> 00:24:01.860 align:middle line:90% from that distribution. 00:24:01.860 --> 00:24:04.932 align:middle line:84% And since we haven't forced it to obey constraints 00:24:04.932 --> 00:24:06.640 align:middle line:84% like five fingers and so on and so forth, 00:24:06.640 --> 00:24:07.840 align:middle line:84% it's not going to do any of that stuff. 00:24:07.840 --> 00:24:09.240 align:middle line:90% It's an unconstrained process. 00:24:09.240 --> 00:24:12.360 align:middle line:84% Now, over time, these things have gotten better and better, 00:24:12.360 --> 00:24:15.310 align:middle line:84% and that's because the data has gotten better, to your point. 00:24:15.310 --> 00:24:17.810 align:middle line:84% But I think our approach to doing these things is also 00:24:17.810 --> 00:24:19.250 align:middle line:90% getting better. 00:24:19.250 --> 00:24:22.070 align:middle line:84% There are lots of ways to now steer it and control it 00:24:22.070 --> 00:24:24.130 align:middle line:84% so it behaves the right way, and that is actually 00:24:24.130 --> 00:24:25.550 align:middle line:84% part of what's happening as well. 00:24:25.550 --> 00:24:28.450 align:middle line:84% So when we talk about, how do you actually give a text prompt 00:24:28.450 --> 00:24:31.270 align:middle line:84% and have it build the image for that particular prompt? 00:24:31.270 --> 00:24:34.190 align:middle line:90% We will revisit this question. 00:24:34.190 --> 00:24:34.690 align:middle line:90% OK. 00:24:34.690 --> 00:24:36.110 align:middle line:90% There were more questions. 00:24:36.110 --> 00:24:38.370 align:middle line:90% Yeah. 00:24:38.370 --> 00:24:41.030 align:middle line:84% STUDENT: Is there some randomness in the model itself? 00:24:41.030 --> 00:24:43.990 align:middle line:84% So if you gave it the same noise image twice, 00:24:43.990 --> 00:24:46.233 align:middle line:84% will it actually produce the same final image or will 00:24:46.233 --> 00:24:47.650 align:middle line:84% it come up with something similar? 00:24:47.650 --> 00:24:49.410 align:middle line:84% PROFESSOR: Yeah, there is randomness, in the process 00:24:49.410 --> 00:24:49.950 align:middle line:90% as well. 00:24:49.950 --> 00:24:50.370 align:middle line:90% STUDENT: In the process as well? 00:24:50.370 --> 00:24:50.990 align:middle line:90% PROFESSOR: In the process. 00:24:50.990 --> 00:24:51.270 align:middle line:90% STUDENT: Yeah. 00:24:51.270 --> 00:24:52.062 align:middle line:90% PROFESSOR: Exactly. 00:24:52.062 --> 00:24:54.770 align:middle line:90% 00:24:54.770 --> 00:24:57.930 align:middle line:84% So to actually, that's a really good point. 00:24:57.930 --> 00:25:00.510 align:middle line:84% But now, I'm afraid to open my laptop or my iPad. 00:25:00.510 --> 00:25:02.890 align:middle line:90% One second. 00:25:02.890 --> 00:25:04.610 align:middle line:90% All right. 00:25:04.610 --> 00:25:10.810 align:middle line:84% So what's going on here is that if you go to this thing, 00:25:10.810 --> 00:25:13.840 align:middle line:84% so I talked about we are transforming 00:25:13.840 --> 00:25:17.240 align:middle line:84% from here, to some crazy distribution here. 00:25:17.240 --> 00:25:19.760 align:middle line:84% So what happens is that let's say that this is the starting 00:25:19.760 --> 00:25:21.800 align:middle line:90% point for the noise input. 00:25:21.800 --> 00:25:23.320 align:middle line:90% This is your noise input. 00:25:23.320 --> 00:25:28.080 align:middle line:84% And then, what you actually do is you go here, 00:25:28.080 --> 00:25:29.620 align:middle line:90% and then you take this point. 00:25:29.620 --> 00:25:32.760 align:middle line:84% And then, you do a small sample next to it. 00:25:32.760 --> 00:25:36.780 align:middle line:84% So you use this as the mean value and then sample around it, 00:25:36.780 --> 00:25:39.780 align:middle line:84% and that's actually what gets published in the user interface. 00:25:39.780 --> 00:25:43.260 align:middle line:84% That's where the randomness comes in. 00:25:43.260 --> 00:25:43.760 align:middle line:90% OK. 00:25:43.760 --> 00:25:48.360 align:middle line:90% 00:25:48.360 --> 00:25:49.220 align:middle line:90% So back to this. 00:25:49.220 --> 00:25:52.360 align:middle line:84% Was there another question somewhere? 00:25:52.360 --> 00:25:53.740 align:middle line:90% Yeah. 00:25:53.740 --> 00:25:54.720 align:middle line:90% STUDENT: Not sure. 00:25:54.720 --> 00:25:56.600 align:middle line:90% PROFESSOR: It's OK. 00:25:56.600 --> 00:26:02.440 align:middle line:84% STUDENT: I was wondering about, when training on a clear picture 00:26:02.440 --> 00:26:05.680 align:middle line:84% to go to a noisy image, to display 00:26:05.680 --> 00:26:08.640 align:middle line:84% where a random sample with a random drop, the sample 00:26:08.640 --> 00:26:10.280 align:middle line:90% probably is pseudorandom. 00:26:10.280 --> 00:26:13.230 align:middle line:84% I'm just wondering if it's like learning relationships that are 00:26:13.230 --> 00:26:14.810 align:middle line:90% dependent on pseudorandomness. 00:26:14.810 --> 00:26:20.285 align:middle line:84% And so when it goes from a noisy image back to a pure image, 00:26:20.285 --> 00:26:22.410 align:middle line:84% is it dependent on that, or if that matters at all? 00:26:22.410 --> 00:26:23.070 align:middle line:90% PROFESSOR: Oh, I see. 00:26:23.070 --> 00:26:24.430 align:middle line:84% So if I understand your question, what you're saying 00:26:24.430 --> 00:26:26.388 align:middle line:84% is that it's pseudorandom, not actually random. 00:26:26.388 --> 00:26:26.973 align:middle line:90% STUDENT: Yeah. 00:26:26.973 --> 00:26:28.390 align:middle line:84% PROFESSOR: And so therefore, there 00:26:28.390 --> 00:26:31.225 align:middle line:84% is some signal in the supposedly random generation. 00:26:31.225 --> 00:26:31.850 align:middle line:90% STUDENT: Right. 00:26:31.850 --> 00:26:33.267 align:middle line:84% PROFESSOR: Is it actually glomming 00:26:33.267 --> 00:26:35.250 align:middle line:84% onto that signal, right, is the question? 00:26:35.250 --> 00:26:35.990 align:middle line:90% STUDENT: Yeah. 00:26:35.990 --> 00:26:38.070 align:middle line:84% PROFESSOR: Theoretically, it's probably possible, 00:26:38.070 --> 00:26:39.430 align:middle line:84% but in practice, it really doesn't matter. 00:26:39.430 --> 00:26:41.190 align:middle line:84% Because we basically say pseudorandom 00:26:41.190 --> 00:26:43.330 align:middle line:90% is good enough for our purposes. 00:26:43.330 --> 00:26:45.770 align:middle line:84% And in fact, in practice, you will see it's not an issue. 00:26:45.770 --> 00:26:50.110 align:middle line:90% 00:26:50.110 --> 00:26:51.408 align:middle line:90% Oh yeah, go ahead. 00:26:51.408 --> 00:26:52.950 align:middle line:84% STUDENT: I just had a quick question. 00:26:52.950 --> 00:26:57.470 align:middle line:84% When you're doing text-to-text, let's 00:26:57.470 --> 00:27:00.330 align:middle line:90% say you're tokenizing the input. 00:27:00.330 --> 00:27:03.910 align:middle line:84% But here, you somehow have to identify that this is Killian 00:27:03.910 --> 00:27:06.670 align:middle line:90% Court and like a stately home. 00:27:06.670 --> 00:27:09.580 align:middle line:84% And this is just going from pixel to image 00:27:09.580 --> 00:27:13.020 align:middle line:90% or decoding a pixel image. 00:27:13.020 --> 00:27:16.340 align:middle line:84% Where does the tag or tokenization 00:27:16.340 --> 00:27:19.695 align:middle line:84% of columns or fingernails, or like-- 00:27:19.695 --> 00:27:20.820 align:middle line:90% PROFESSOR: There's nothing. 00:27:20.820 --> 00:27:22.810 align:middle line:84% It's learning everything from the pixel values. 00:27:22.810 --> 00:27:23.860 align:middle line:84% STUDENT: Everything's from the pixel values? 00:27:23.860 --> 00:27:25.540 align:middle line:84% PROFESSOR: Yeah, and this is what I was, 00:27:25.540 --> 00:27:27.820 align:middle line:84% when Ike asked the question about the four 00:27:27.820 --> 00:27:31.520 align:middle line:84% fingers, five fingers thing, it has no idea of fingers. 00:27:31.520 --> 00:27:33.520 align:middle line:84% It has zero knowledge about any of these things. 00:27:33.520 --> 00:27:36.300 align:middle line:84% All it's seeing is a bunch of photographs. 00:27:36.300 --> 00:27:38.380 align:middle line:84% STUDENT: So when you type in, you say, I 00:27:38.380 --> 00:27:40.365 align:middle line:84% want to see a hand with green fingers-- 00:27:40.365 --> 00:27:41.240 align:middle line:90% PROFESSOR: Oh, I see. 00:27:41.240 --> 00:27:43.800 align:middle line:84% So we haven't yet come to the stage of, OK, 00:27:43.800 --> 00:27:46.840 align:middle line:84% how do you actually steer this image using your text prompt? 00:27:46.840 --> 00:27:47.520 align:middle line:90% It's coming. 00:27:47.520 --> 00:27:48.020 align:middle line:90% STUDENT: OK. 00:27:48.020 --> 00:27:49.520 align:middle line:84% PROFESSOR: Right now, all we are saying is that, 00:27:49.520 --> 00:27:51.980 align:middle line:84% look, I'm going to give you a bunch of photographs 00:27:51.980 --> 00:27:54.587 align:middle line:84% of a particular kind of thing, stately college buildings. 00:27:54.587 --> 00:27:56.920 align:middle line:84% And I want to have a model, which at the end of the day, 00:27:56.920 --> 00:27:57.800 align:middle line:90% I just poke it. 00:27:57.800 --> 00:27:59.967 align:middle line:84% Every time I poke it, it gives me a stately college. 00:27:59.967 --> 00:28:00.500 align:middle line:90% That's it. 00:28:00.500 --> 00:28:03.787 align:middle line:84% Now, I'm going to actually start giving it text and saying, OK, 00:28:03.787 --> 00:28:05.620 align:middle line:84% create the thing I'm just telling you about. 00:28:05.620 --> 00:28:06.440 align:middle line:90% That's coming. 00:28:06.440 --> 00:28:11.030 align:middle line:84% And that's some additional magic is going on to get that done. 00:28:11.030 --> 00:28:11.530 align:middle line:90% OK. 00:28:11.530 --> 00:28:14.090 align:middle line:90% So this is what we have. 00:28:14.090 --> 00:28:16.390 align:middle line:84% And this is called a diffusion model. 00:28:16.390 --> 00:28:19.210 align:middle line:84% And this is the original paper that figured this out. 00:28:19.210 --> 00:28:24.530 align:middle line:84% And the process of actually creating-- 00:28:24.530 --> 00:28:27.490 align:middle line:84% if taking an image and creating noisy versions of it 00:28:27.490 --> 00:28:30.250 align:middle line:84% to create a training data is called the forward process, 00:28:30.250 --> 00:28:33.810 align:middle line:84% and then what we did in reverse is called the reverse process. 00:28:33.810 --> 00:28:34.910 align:middle line:90% Check out the paper. 00:28:34.910 --> 00:28:37.950 align:middle line:84% It's actually really well written and I recommend it. 00:28:37.950 --> 00:28:40.770 align:middle line:84% Now, in practice, some other researchers 00:28:40.770 --> 00:28:44.195 align:middle line:84% came along shortly after this and made a small improvement. 00:28:44.195 --> 00:28:46.570 align:middle line:84% It turns out to be actually a big improvement in practice 00:28:46.570 --> 00:28:48.362 align:middle line:84% in terms of improving the quality of what's 00:28:48.362 --> 00:28:49.270 align:middle line:90% being produced. 00:28:49.270 --> 00:28:52.250 align:middle line:84% And so what they said is, hey, instead of training the model 00:28:52.250 --> 00:28:54.810 align:middle line:84% to predict the less noisy version of the image, 00:28:54.810 --> 00:29:00.090 align:middle line:84% we actually ask it to predict just the noise in the input. 00:29:00.090 --> 00:29:02.450 align:middle line:84% And then, we will just simply subtract the noise 00:29:02.450 --> 00:29:04.410 align:middle line:90% from the input to get the image. 00:29:04.410 --> 00:29:08.917 align:middle line:84% So instead of saying x is an image, y is the noisy image, 00:29:08.917 --> 00:29:10.500 align:middle line:84% we actually tell it, here's the image. 00:29:10.500 --> 00:29:14.120 align:middle line:84% Here is the noise that we added to x to get the noisy version. 00:29:14.120 --> 00:29:15.940 align:middle line:84% And then, just predict the noise for me. 00:29:15.940 --> 00:29:17.960 align:middle line:84% And then, once I get it, I just do x minus noise 00:29:17.960 --> 00:29:21.240 align:middle line:84% and I get the less noisy version of the image. 00:29:21.240 --> 00:29:23.920 align:middle line:84% This feels arithmetically equivalent, 00:29:23.920 --> 00:29:26.720 align:middle line:84% but in practice, it ends up generating much higher quality 00:29:26.720 --> 00:29:27.540 align:middle line:90% images. 00:29:27.540 --> 00:29:28.680 align:middle line:84% And there is some very interesting theory 00:29:28.680 --> 00:29:30.520 align:middle line:84% as to why that works and so on and so forth. 00:29:30.520 --> 00:29:33.200 align:middle line:84% And you can read this paper if you're interested. 00:29:33.200 --> 00:29:34.680 align:middle line:84% So if you actually look at what's 00:29:34.680 --> 00:29:36.960 align:middle line:84% going on in most diffusion models today, 00:29:36.960 --> 00:29:39.080 align:middle line:84% they're basically using an approach like this. 00:29:39.080 --> 00:29:41.788 align:middle line:84% They're actually predicting each time they predict noise and take 00:29:41.788 --> 00:29:43.040 align:middle line:90% it away, subtract it. 00:29:43.040 --> 00:29:47.040 align:middle line:84% So iterative subtracting of predicted noise. 00:29:47.040 --> 00:29:48.720 align:middle line:90% That's what's going on. 00:29:48.720 --> 00:29:51.560 align:middle line:84% So all right, that's what we have. 00:29:51.560 --> 00:29:55.420 align:middle line:84% Now, at this point, you might be wondering, OK, 00:29:55.420 --> 00:29:57.440 align:middle line:84% so far in the semester, we have actually 00:29:57.440 --> 00:29:59.680 align:middle line:84% learned how to take an image and then 00:29:59.680 --> 00:30:02.680 align:middle line:84% classify it into one of 20 things, 10 things whatever. 00:30:02.680 --> 00:30:05.070 align:middle line:84% We've also taken text and figure out what 00:30:05.070 --> 00:30:06.250 align:middle line:90% to do with things with it. 00:30:06.250 --> 00:30:09.390 align:middle line:84% We haven't yet talked about how do you actually take an image, 00:30:09.390 --> 00:30:12.950 align:middle line:84% and how can we get the output also to be another image? 00:30:12.950 --> 00:30:14.910 align:middle line:90% We haven't done that yet. 00:30:14.910 --> 00:30:18.070 align:middle line:84% So we have actually not done image-to-image. 00:30:18.070 --> 00:30:21.070 align:middle line:84% How do you actually build a neural network to do images? 00:30:21.070 --> 00:30:22.968 align:middle line:84% And in the interest of time, you're not 00:30:22.968 --> 00:30:24.510 align:middle line:84% going to get into it massively, but I 00:30:24.510 --> 00:30:27.310 align:middle line:84% want to give you a quick idea of how it works. 00:30:27.310 --> 00:30:33.430 align:middle line:84% So the most, I would say, the dominant architecture 00:30:33.430 --> 00:30:36.510 align:middle line:84% to take an image as an input and produce an image as an output 00:30:36.510 --> 00:30:38.070 align:middle line:90% is called the U-Net. 00:30:38.070 --> 00:30:40.300 align:middle line:84% And that's architecture we see here. 00:30:40.300 --> 00:30:42.910 align:middle line:90% 00:30:42.910 --> 00:30:45.660 align:middle line:84% So fundamentally, if you look at the left half. 00:30:45.660 --> 00:30:47.910 align:middle line:84% So there's a left half to the network and a right half 00:30:47.910 --> 00:30:49.950 align:middle line:84% to the network, hence, the U. If you 00:30:49.950 --> 00:30:51.910 align:middle line:84% look at the left half of the network, 00:30:51.910 --> 00:30:54.790 align:middle line:84% it's a good, old convolutional neural network, 00:30:54.790 --> 00:30:58.670 align:middle line:84% like the kind we know and love and the kind 00:30:58.670 --> 00:31:00.210 align:middle line:90% that we are very familiar with. 00:31:00.210 --> 00:31:02.340 align:middle line:84% So you take an input image, and then you 00:31:02.340 --> 00:31:06.920 align:middle line:84% run it through a bunch of convolutional blocks, 00:31:06.920 --> 00:31:10.560 align:middle line:84% and then we do some max pooling, and then we keep on doing it. 00:31:10.560 --> 00:31:12.780 align:middle line:84% And at some point, it becomes smaller and smaller, 00:31:12.780 --> 00:31:16.280 align:middle line:84% and we get something like this, which we are very familiar with. 00:31:16.280 --> 00:31:18.900 align:middle line:84% The big image with the three channels 00:31:18.900 --> 00:31:20.480 align:middle line:84% gets smaller and smaller and smaller, 00:31:20.480 --> 00:31:22.480 align:middle line:84% but the number of channels gets wider and wider. 00:31:22.480 --> 00:31:24.800 align:middle line:84% It becomes much smaller but much deeper. 00:31:24.800 --> 00:31:27.020 align:middle line:90% It becomes like a 3D volume. 00:31:27.020 --> 00:31:29.140 align:middle line:84% And we have seen that again and again. 00:31:29.140 --> 00:31:31.940 align:middle line:84% The left part is just a good, old convolutional 00:31:31.940 --> 00:31:33.120 align:middle line:90% with pooling layers. 00:31:33.120 --> 00:31:34.660 align:middle line:84% And then, you come to the middle. 00:31:34.660 --> 00:31:37.060 align:middle line:84% And then, from this point on, what we do is 00:31:37.060 --> 00:31:39.920 align:middle line:84% we take whatever this thing here, 00:31:39.920 --> 00:31:42.580 align:middle line:84% and then we essentially reverse the process. 00:31:42.580 --> 00:31:45.260 align:middle line:84% We go from the small things, which are really deep, 00:31:45.260 --> 00:31:48.620 align:middle line:84% to slightly bigger things that are a little less deep, and so 00:31:48.620 --> 00:31:51.580 align:middle line:84% on and so forth, till we get the original size back again. 00:31:51.580 --> 00:31:56.940 align:middle line:84% And we do that using an inverse of the convolution layer 00:31:56.940 --> 00:32:00.060 align:middle line:84% called an up convolution or deconvolution layer. 00:32:00.060 --> 00:32:03.380 align:middle line:84% And you can check out 9.2 in the textbook 00:32:03.380 --> 00:32:05.760 align:middle line:90% to understand how it's done. 00:32:05.760 --> 00:32:09.220 align:middle line:84% It's also called conv 2D transpose. 00:32:09.220 --> 00:32:11.602 align:middle line:90% It's a very similar idea. 00:32:11.602 --> 00:32:13.560 align:middle line:84% And I'm not going to get into the details here, 00:32:13.560 --> 00:32:15.685 align:middle line:84% but you essentially do the inverse of a convolution 00:32:15.685 --> 00:32:18.700 align:middle line:84% operation to get the size to come back to the bigger size. 00:32:18.700 --> 00:32:21.980 align:middle line:84% And you do it gradually, till the output 00:32:21.980 --> 00:32:25.780 align:middle line:84% you have matches the size of the input that came in. 00:32:25.780 --> 00:32:28.760 align:middle line:84% So image gets smaller and smaller into a thing, 00:32:28.760 --> 00:32:31.320 align:middle line:84% and then you just blow it back up again to get an image back. 00:32:31.320 --> 00:32:33.100 align:middle line:90% So that is the U-Net. 00:32:33.100 --> 00:32:35.420 align:middle line:84% Now, there's really one very important thing 00:32:35.420 --> 00:32:42.140 align:middle line:84% that happens in the U-Net, which is you see these connections. 00:32:42.140 --> 00:32:45.820 align:middle line:84% Basically, what they do is at every step, when you're 00:32:45.820 --> 00:32:47.900 align:middle line:84% coming back up in the right half, 00:32:47.900 --> 00:32:51.860 align:middle line:84% you actually attach whatever was in the mirror 00:32:51.860 --> 00:32:55.320 align:middle line:84% image of the original input as we processed on the left side, 00:32:55.320 --> 00:32:57.660 align:middle line:84% we attach it to this side as well. 00:32:57.660 --> 00:33:00.170 align:middle line:84% Remember, I talked about this whole notion of a residual 00:33:00.170 --> 00:33:04.410 align:middle line:84% connection, back many classes ago, where I said, 00:33:04.410 --> 00:33:08.690 align:middle line:84% when an input goes through each layer of a neural network, 00:33:08.690 --> 00:33:10.890 align:middle line:84% at one point, let's say you're in the 10th layer, 00:33:10.890 --> 00:33:13.450 align:middle line:84% you're only seeing what is the ninth layer has produced 00:33:13.450 --> 00:33:13.950 align:middle line:90% for you. 00:33:13.950 --> 00:33:15.430 align:middle line:90% That's all you're working with. 00:33:15.430 --> 00:33:18.530 align:middle line:84% But wouldn't it be nice if the 10th layer actually had access 00:33:18.530 --> 00:33:21.363 align:middle line:84% to the eighth layer, the seventh layer, the sixth layer, 00:33:21.363 --> 00:33:22.030 align:middle line:90% the fifth layer? 00:33:22.030 --> 00:33:23.090 align:middle line:90% Heck, why not the input? 00:33:23.090 --> 00:33:26.490 align:middle line:84% Because the more information it has, the more able 00:33:26.490 --> 00:33:29.450 align:middle line:84% it's probably to do whatever it can with the inputs given. 00:33:29.450 --> 00:33:32.490 align:middle line:84% Why restrict it to only the input of the output 00:33:32.490 --> 00:33:33.430 align:middle line:90% of the previous layer? 00:33:33.430 --> 00:33:35.763 align:middle line:84% Why can't we give it everything that has came before it? 00:33:35.763 --> 00:33:37.390 align:middle line:84% Now, giving everything is too much, 00:33:37.390 --> 00:33:39.950 align:middle line:84% but we can be selective in what we give it. 00:33:39.950 --> 00:33:41.670 align:middle line:84% So what these folks decided, I'm sure, 00:33:41.670 --> 00:33:44.530 align:middle line:84% after much experimentation, is that if they actually 00:33:44.530 --> 00:33:49.290 align:middle line:84% attach whatever was coming out of this layer to this layer 00:33:49.290 --> 00:33:51.950 align:middle line:84% before it goes through the output, it really helped. 00:33:51.950 --> 00:33:53.750 align:middle line:84% Similarly, this thing gets attached, 00:33:53.750 --> 00:33:54.930 align:middle line:90% and so on and so forth. 00:33:54.930 --> 00:33:56.960 align:middle line:90% And it kind of makes sense. 00:33:56.960 --> 00:33:59.560 align:middle line:84% Why force it to figure out everything 00:33:59.560 --> 00:34:02.320 align:middle line:84% it has to figure out just from this thing that came in? 00:34:02.320 --> 00:34:04.240 align:middle line:90% Let's give this that. 00:34:04.240 --> 00:34:06.860 align:middle line:84% Let's also give a little here, a little here. 00:34:06.860 --> 00:34:09.400 align:middle line:84% So these residual connections are a huge building block 00:34:09.400 --> 00:34:11.719 align:middle line:84% for why these things work as well as they do. 00:34:11.719 --> 00:34:15.080 align:middle line:84% And in general, giving a layer as much information 00:34:15.080 --> 00:34:17.420 align:middle line:84% as we can give it is always a good idea, 00:34:17.420 --> 00:34:19.159 align:middle line:84% but you can't go nuts because then you 00:34:19.159 --> 00:34:21.179 align:middle line:84% have much more parameters and all kinds of stuff happens. 00:34:21.179 --> 00:34:23.262 align:middle line:84% So there is a bit of a balance you have to strike, 00:34:23.262 --> 00:34:25.679 align:middle line:84% and this was the balance struck by these researchers. 00:34:25.679 --> 00:34:28.520 align:middle line:84% And so this thing was originally invented 00:34:28.520 --> 00:34:31.300 align:middle line:84% for some medical segmentation use cases, 00:34:31.300 --> 00:34:34.239 align:middle line:84% but it is just heavily used for everything now. 00:34:34.239 --> 00:34:36.840 align:middle line:84% It's a really powerful architecture. 00:34:36.840 --> 00:34:39.760 align:middle line:90% Questions? 00:34:39.760 --> 00:34:42.520 align:middle line:84% STUDENT: Can we have an example of in what scenarios 00:34:42.520 --> 00:34:44.199 align:middle line:84% would you use to spin up a network? 00:34:44.199 --> 00:34:46.560 align:middle line:84% PROFESSOR: Anytime you have an image-to-image. 00:34:46.560 --> 00:34:49.687 align:middle line:84% STUDENT: What kind of conversion do you have [INAUDIBLE] 00:34:49.687 --> 00:34:52.020 align:middle line:84% PROFESSOR: Oh, like, what kind of examples of use cases? 00:34:52.020 --> 00:34:53.159 align:middle line:84% Let's say that, for example, you want 00:34:53.159 --> 00:34:55.429 align:middle line:84% to take an image, like a black and white image, 00:34:55.429 --> 00:34:58.730 align:middle line:84% and you want to colorize it, for instance. 00:34:58.730 --> 00:35:00.055 align:middle line:90% Boom, you U-Net. 00:35:00.055 --> 00:35:02.430 align:middle line:84% You want to take an image and make it a higher resolution 00:35:02.430 --> 00:35:03.630 align:middle line:90% image, U-Net. 00:35:03.630 --> 00:35:06.950 align:middle line:84% You want to take an image and for every pixel in the image, 00:35:06.950 --> 00:35:10.950 align:middle line:84% you want to classify it into one of 10 things. 00:35:10.950 --> 00:35:13.150 align:middle line:84% So anytime when you want the output 00:35:13.150 --> 00:35:16.030 align:middle line:84% shape, the shape of the output, to be basically 00:35:16.030 --> 00:35:18.810 align:middle line:84% the same shape as the input but with other data, 00:35:18.810 --> 00:35:20.770 align:middle line:90% you need to use this. 00:35:20.770 --> 00:35:21.270 align:middle line:90% Yeah. 00:35:21.270 --> 00:35:25.350 align:middle line:90% 00:35:25.350 --> 00:35:28.070 align:middle line:84% STUDENT: This logic of having access 00:35:28.070 --> 00:35:30.687 align:middle line:90% to all the previous iterations-- 00:35:30.687 --> 00:35:31.770 align:middle line:90% PROFESSOR: Not iterations. 00:35:31.770 --> 00:35:33.275 align:middle line:84% STUDENT: All the previous layers. 00:35:33.275 --> 00:35:35.400 align:middle line:84% PROFESSOR: Correct, outputs of the previous layers. 00:35:35.400 --> 00:35:36.870 align:middle line:90% STUDENT: Layers. 00:35:36.870 --> 00:35:41.110 align:middle line:84% But this would also help for clean up 00:35:41.110 --> 00:35:42.870 align:middle line:90% and give better categorization. 00:35:42.870 --> 00:35:45.060 align:middle line:84% Does it always have to be an image-to-image? 00:35:45.060 --> 00:35:45.810 align:middle line:90% PROFESSOR: No, no. 00:35:45.810 --> 00:35:47.957 align:middle line:84% In fact, if you look at ResNet, ResNet 00:35:47.957 --> 00:35:49.790 align:middle line:84% is the one, in fact, that pioneered the idea 00:35:49.790 --> 00:35:51.390 align:middle line:90% of the residual connection. 00:35:51.390 --> 00:35:52.960 align:middle line:90% So we use it for ResNet. 00:35:52.960 --> 00:35:56.120 align:middle line:84% We actually use the transformer stack. 00:35:56.120 --> 00:35:59.120 align:middle line:84% If you remember, it goes through the self-attention layer. 00:35:59.120 --> 00:36:00.440 align:middle line:90% It comes out the other end. 00:36:00.440 --> 00:36:03.400 align:middle line:84% And then, we add the input back to it. 00:36:03.400 --> 00:36:05.900 align:middle line:84% And then, we send it through LayerNorm. 00:36:05.900 --> 00:36:07.780 align:middle line:84% So you will see that this residual connection 00:36:07.780 --> 00:36:10.260 align:middle line:84% is sitting in two different places in a single transformer 00:36:10.260 --> 00:36:11.460 align:middle line:90% block. 00:36:11.460 --> 00:36:13.300 align:middle line:90% So it's extremely heavily used. 00:36:13.300 --> 00:36:16.000 align:middle line:84% There is something called Deep and Wide Network, if I remember, 00:36:16.000 --> 00:36:18.700 align:middle line:84% or DenseNet, which uses the same trick. 00:36:18.700 --> 00:36:23.200 align:middle line:84% In fact, when you're working with structured data, good, old, 00:36:23.200 --> 00:36:25.940 align:middle line:84% say, linear regression, and you've looked at your data 00:36:25.940 --> 00:36:28.780 align:middle line:84% and you've come up with all kinds of very clever features. 00:36:28.780 --> 00:36:30.940 align:middle line:84% I'm going to look at price per square foot. 00:36:30.940 --> 00:36:32.740 align:middle line:84% You do a bunch of feature engineering 00:36:32.740 --> 00:36:34.660 align:middle line:84% and you have a bunch of new features. 00:36:34.660 --> 00:36:38.540 align:middle line:84% Well, you should take your old features and your new features 00:36:38.540 --> 00:36:40.580 align:middle line:90% and send both in. 00:36:40.580 --> 00:36:42.945 align:middle line:84% Why send only the new stuff that you have concocted? 00:36:42.945 --> 00:36:44.320 align:middle line:84% Why can't you send everything in? 00:36:44.320 --> 00:36:44.987 align:middle line:90% That's the idea. 00:36:44.987 --> 00:36:47.420 align:middle line:90% 00:36:47.420 --> 00:36:50.360 align:middle line:84% All right, so let's come back here. 00:36:50.360 --> 00:36:53.090 align:middle line:90% 00:36:53.090 --> 00:36:56.010 align:middle line:84% Now, we have seen how to generate a good image. 00:36:56.010 --> 00:36:58.050 align:middle line:84% Now, let's figure out how to steer it 00:36:58.050 --> 00:37:00.396 align:middle line:84% or condition it with a text prompt because that's, 00:37:00.396 --> 00:37:02.730 align:middle line:90% sort of, the Holy Grail. 00:37:02.730 --> 00:37:07.957 align:middle line:84% So we want to take, so here is some intuition. 00:37:07.957 --> 00:37:09.790 align:middle line:84% We want to take the text prompt into account 00:37:09.790 --> 00:37:11.248 align:middle line:84% and obviously generating the image. 00:37:11.248 --> 00:37:15.530 align:middle line:84% Now, imagine if we had like a rough image that corresponds 00:37:15.530 --> 00:37:16.770 align:middle line:90% to that text prompt. 00:37:16.770 --> 00:37:17.850 align:middle line:90% Just imagine. 00:37:17.850 --> 00:37:20.930 align:middle line:84% So the text prompt is cute Labrador Retriever. 00:37:20.930 --> 00:37:23.610 align:middle line:84% And you have a very noisy image of a Labrador Retriever 00:37:23.610 --> 00:37:25.070 align:middle line:90% that just happens to be handy. 00:37:25.070 --> 00:37:26.170 align:middle line:90% You have it. 00:37:26.170 --> 00:37:29.130 align:middle line:84% Well, now, you're in good shape because you just feed that in, 00:37:29.130 --> 00:37:31.370 align:middle line:84% and your system will denoise it for you. 00:37:31.370 --> 00:37:32.670 align:middle line:90% You can get a better image. 00:37:32.670 --> 00:37:33.970 align:middle line:90% That's pretty easy. 00:37:33.970 --> 00:37:36.303 align:middle line:84% But obviously, in reality, you don't have a rough image. 00:37:36.303 --> 00:37:37.803 align:middle line:84% In fact, you're trying to create one 00:37:37.803 --> 00:37:39.290 align:middle line:84% of those things in the first place. 00:37:39.290 --> 00:37:39.950 align:middle line:90% We don't. 00:37:39.950 --> 00:37:44.770 align:middle line:84% But what if we had an embedding for the prompt that's 00:37:44.770 --> 00:37:47.290 align:middle line:84% close to the embeddings of all the images 00:37:47.290 --> 00:37:48.670 align:middle line:90% that correspond to the prompt? 00:37:48.670 --> 00:37:50.880 align:middle line:84% So let's take a prompt, and let's 00:37:50.880 --> 00:37:54.960 align:middle line:84% imagine all the images in the universe that correspond to that 00:37:54.960 --> 00:37:55.640 align:middle line:90% prompt. 00:37:55.640 --> 00:37:59.858 align:middle line:84% And now, further imagine, because everything is a vector, 00:37:59.858 --> 00:38:01.400 align:middle line:84% everything is embedding in our world, 00:38:01.400 --> 00:38:05.320 align:middle line:90% that the image has an embedding. 00:38:05.320 --> 00:38:07.720 align:middle line:84% Sorry, the text prompt has an embedding. 00:38:07.720 --> 00:38:09.960 align:middle line:90% Every image has an embedding. 00:38:09.960 --> 00:38:13.360 align:middle line:84% And we have somehow calculated these embeddings 00:38:13.360 --> 00:38:17.080 align:middle line:84% so that the text prompt's embedding is smack where 00:38:17.080 --> 00:38:20.040 align:middle line:90% all the image embeddings are. 00:38:20.040 --> 00:38:22.700 align:middle line:84% We will get to how we actually do it in just a moment. 00:38:22.700 --> 00:38:25.700 align:middle line:84% But conceptually, imagine if we had an embedding, 00:38:25.700 --> 00:38:27.200 align:middle line:84% if you could calculate embeddings 00:38:27.200 --> 00:38:29.420 align:middle line:84% for text and embeddings for images, 00:38:29.420 --> 00:38:33.080 align:middle line:84% so they all live in the same space 00:38:33.080 --> 00:38:36.440 align:middle line:84% So if we feed this embedding to a denoising 00:38:36.440 --> 00:38:40.960 align:middle line:84% model, because that text embedding is sitting 00:38:40.960 --> 00:38:44.160 align:middle line:84% in the same space as all the image embeddings that it 00:38:44.160 --> 00:38:48.670 align:middle line:84% corresponds to, maybe our model can just denoise that embedding 00:38:48.670 --> 00:38:52.350 align:middle line:90% and give you what you want. 00:38:52.350 --> 00:38:55.030 align:middle line:84% So since this embedding is already close to the embeddings 00:38:55.030 --> 00:38:56.450 align:middle line:84% of the things we want to generate, 00:38:56.450 --> 00:38:59.230 align:middle line:90% maybe we'll just get it done. 00:38:59.230 --> 00:39:01.190 align:middle line:84% So ultimately, we want to generate an image. 00:39:01.190 --> 00:39:02.950 align:middle line:84% And if we had an embedding for that image, 00:39:02.950 --> 00:39:05.310 align:middle line:84% we could generate the image from the embedding. 00:39:05.310 --> 00:39:07.030 align:middle line:90% And we use the text. 00:39:07.030 --> 00:39:09.295 align:middle line:84% So we go from text to embedding, which 00:39:09.295 --> 00:39:10.670 align:middle line:84% happens to live in the same space 00:39:10.670 --> 00:39:13.050 align:middle line:84% as all the embeddings of the images we care about. 00:39:13.050 --> 00:39:14.550 align:middle line:84% And then, from that image embedding, 00:39:14.550 --> 00:39:15.630 align:middle line:90% we go to the final image. 00:39:15.630 --> 00:39:16.510 align:middle line:90% OK. 00:39:16.510 --> 00:39:18.920 align:middle line:84% This is a bunch of me talking and hand waving. 00:39:18.920 --> 00:39:20.170 align:middle line:90% It will all become very clear. 00:39:20.170 --> 00:39:22.296 align:middle line:90% But that's the rough intuition. 00:39:22.296 --> 00:39:24.110 align:middle line:90% OK. 00:39:24.110 --> 00:39:26.190 align:middle line:84% So what we'll do is we'll describe an approach 00:39:26.190 --> 00:39:29.590 align:middle line:84% to calculate an embedding for any text, any piece of text, 00:39:29.590 --> 00:39:33.430 align:middle line:84% that is close to the embeddings of the images that correspond 00:39:33.430 --> 00:39:35.255 align:middle line:90% to that piece of text. 00:39:35.255 --> 00:39:37.130 align:middle line:84% So this is the problem you're going to solve. 00:39:37.130 --> 00:39:38.510 align:middle line:90% There's a bunch of text. 00:39:38.510 --> 00:39:40.052 align:middle line:84% Conceptually, there are a whole bunch 00:39:40.052 --> 00:39:41.850 align:middle line:84% of images that describe the text. 00:39:41.850 --> 00:39:43.670 align:middle line:84% And we're going to now create embeddings 00:39:43.670 --> 00:39:48.020 align:middle line:84% so that that is close to all the embeddings of those images. 00:39:48.020 --> 00:39:50.620 align:middle line:84% It feels, kind of, almost impossible that you can actually 00:39:50.620 --> 00:39:52.380 align:middle line:90% do something like this. 00:39:52.380 --> 00:39:57.620 align:middle line:84% But there's a very clever idea that OpenAI came up with that 00:39:57.620 --> 00:39:58.740 align:middle line:90% tells you how to do it. 00:39:58.740 --> 00:40:00.300 align:middle line:84% So here's what we're going to do. 00:40:00.300 --> 00:40:03.100 align:middle line:84% So let's say we have an image and a caption. 00:40:03.100 --> 00:40:04.940 align:middle line:90% So here is an image. 00:40:04.940 --> 00:40:06.100 align:middle line:90% Here's a caption. 00:40:06.100 --> 00:40:10.162 align:middle line:84% And we need some way to take that piece of text 00:40:10.162 --> 00:40:12.620 align:middle line:84% and run it through some network and create a nice embedding 00:40:12.620 --> 00:40:13.780 align:middle line:90% from it. 00:40:13.780 --> 00:40:15.800 align:middle line:84% Similarly, we want to take this image, 00:40:15.800 --> 00:40:18.667 align:middle line:84% run it through some network and create an embedding from it. 00:40:18.667 --> 00:40:19.500 align:middle line:90% Now, first question. 00:40:19.500 --> 00:40:21.625 align:middle line:84% How can we compute embeddings from a piece of text? 00:40:21.625 --> 00:40:23.583 align:middle line:84% First question, how can we compute an embedding 00:40:23.583 --> 00:40:24.600 align:middle line:90% from a piece of text? 00:40:24.600 --> 00:40:27.780 align:middle line:90% You know the answer. 00:40:27.780 --> 00:40:29.700 align:middle line:90% Run it through a transformer. 00:40:29.700 --> 00:40:30.360 align:middle line:90% Piece of cake. 00:40:30.360 --> 00:40:31.326 align:middle line:90% We know how to do that. 00:40:31.326 --> 00:40:34.420 align:middle line:90% 00:40:34.420 --> 00:40:36.740 align:middle line:84% In particular, you can do something like BERT. 00:40:36.740 --> 00:40:38.220 align:middle line:84% And for an image encoder, you just 00:40:38.220 --> 00:40:40.700 align:middle line:84% run it through something like ResNet. 00:40:40.700 --> 00:40:43.657 align:middle line:84% The penultimate layer, one of the final layers, the ResNet 00:40:43.657 --> 00:40:45.990 align:middle line:84% is going to be a very good representation of that image. 00:40:45.990 --> 00:40:47.850 align:middle line:90% You get another embedding. 00:40:47.850 --> 00:40:49.970 align:middle line:84% So using the building blocks, we already know, 00:40:49.970 --> 00:40:53.570 align:middle line:84% we can create embeddings very quickly from these things. 00:40:53.570 --> 00:40:56.470 align:middle line:84% But if you just take a piece of text and run it through a BERT, 00:40:56.470 --> 00:40:58.470 align:middle line:84% and you take an image and run it through ResNet, 00:40:58.470 --> 00:40:59.970 align:middle line:84% you're going to get some embeddings. 00:40:59.970 --> 00:41:04.470 align:middle line:84% But why the heck should they be related? 00:41:04.470 --> 00:41:06.970 align:middle line:84% They were not trained together, so there's no basis for them 00:41:06.970 --> 00:41:08.210 align:middle line:90% to be related. 00:41:08.210 --> 00:41:10.190 align:middle line:84% They would just be some two embeddings. 00:41:10.190 --> 00:41:11.678 align:middle line:90% Maybe they are kind of similar. 00:41:11.678 --> 00:41:12.470 align:middle line:90% Maybe they are not. 00:41:12.470 --> 00:41:13.070 align:middle line:90% We don't know. 00:41:13.070 --> 00:41:16.470 align:middle line:84% There's no reason to expect that they're going to be similar. 00:41:16.470 --> 00:41:17.590 align:middle line:90% They just do embeddings. 00:41:17.590 --> 00:41:20.250 align:middle line:90% 00:41:20.250 --> 00:41:23.367 align:middle line:84% Now, what we want to do is, once we have these, 00:41:23.367 --> 00:41:25.450 align:middle line:84% we need to make sure the embeddings that comes out 00:41:25.450 --> 00:41:28.710 align:middle line:84% of these two things satisfy two very important requirements. 00:41:28.710 --> 00:41:31.903 align:middle line:90% 00:41:31.903 --> 00:41:33.570 align:middle line:84% We want to make sure that if you give it 00:41:33.570 --> 00:41:38.660 align:middle line:84% an image and a caption that describes that image. 00:41:38.660 --> 00:41:40.410 align:middle line:84% So you have an image and a subcaption that 00:41:40.410 --> 00:41:42.480 align:middle line:84% describes that image, we want to make sure 00:41:42.480 --> 00:41:44.985 align:middle line:84% that the embeddings that come out of these two boxes, they 00:41:44.985 --> 00:41:46.610 align:middle line:84% are as close to each other as possible. 00:41:46.610 --> 00:41:49.260 align:middle line:90% 00:41:49.260 --> 00:41:51.300 align:middle line:84% Given an image and a caption that describes it, 00:41:51.300 --> 00:41:52.380 align:middle line:90% that's the connection. 00:41:52.380 --> 00:41:53.880 align:middle line:84% They have to be close to each other. 00:41:53.880 --> 00:41:58.040 align:middle line:84% And conversely, if you have an image and a caption, that's 00:41:58.040 --> 00:42:01.960 align:middle line:84% totally irrelevant, a train rounding a bend 00:42:01.960 --> 00:42:05.380 align:middle line:84% to the beautiful fall foliage all around, clearly irrelevant, 00:42:05.380 --> 00:42:07.960 align:middle line:84% those embeddings should be far apart. 00:42:07.960 --> 00:42:11.000 align:middle line:90% That intuitively makes sense. 00:42:11.000 --> 00:42:12.980 align:middle line:84% Pairs of related things should be together. 00:42:12.980 --> 00:42:14.920 align:middle line:84% Irrelevant things should be far apart. 00:42:14.920 --> 00:42:18.140 align:middle line:84% So if you can find embeddings that satisfy these two criteria, 00:42:18.140 --> 00:42:20.880 align:middle line:90% maybe we will be in the game. 00:42:20.880 --> 00:42:24.920 align:middle line:84% So now, this ensures that the text embedding and the image 00:42:24.920 --> 00:42:27.880 align:middle line:84% embedding are referring to the same underlying concept. 00:42:27.880 --> 00:42:31.080 align:middle line:84% These requirements will enforce that. 00:42:31.080 --> 00:42:32.783 align:middle line:84% And so the embedding for any text prompt 00:42:32.783 --> 00:42:34.200 align:middle line:84% is close to the embeddings though, 00:42:34.200 --> 00:42:36.283 align:middle line:84% for all the images that correspond to that prompt. 00:42:36.283 --> 00:42:38.680 align:middle line:90% 00:42:38.680 --> 00:42:41.712 align:middle line:84% So the question is, how do we do this? 00:42:41.712 --> 00:42:44.170 align:middle line:84% First of all, how can we tell how close two embeddings are? 00:42:44.170 --> 00:42:45.337 align:middle line:90% You know the answer to this. 00:42:45.337 --> 00:42:46.105 align:middle line:90% What's the answer? 00:42:46.105 --> 00:42:47.230 align:middle line:90% STUDENT: Cosine similarity. 00:42:47.230 --> 00:42:49.110 align:middle line:84% PROFESSOR: Correct, cosine similarity. 00:42:49.110 --> 00:42:52.390 align:middle line:84% We use the cosine similarity of the embeddings. 00:42:52.390 --> 00:42:53.890 align:middle line:84% So we know how to measure closeness. 00:42:53.890 --> 00:42:56.140 align:middle line:84% So the question is, how can we compute embeddings that 00:42:56.140 --> 00:42:57.470 align:middle line:90% satisfy the two requirements. 00:42:57.470 --> 00:43:01.710 align:middle line:84% And OpenAI built a model called CLIP, which is very famous, 00:43:01.710 --> 00:43:02.810 align:middle line:90% to solve this problem. 00:43:02.810 --> 00:43:07.190 align:middle line:84% It stands for Contrastive Language-Image Pretaining. 00:43:07.190 --> 00:43:09.430 align:middle line:84% And this forms the basis for a whole bunch of models 00:43:09.430 --> 00:43:11.910 align:middle line:84% that have sprung up after this called BLIP and BLIP 2 00:43:11.910 --> 00:43:15.790 align:middle line:84% and so on and so forth, but this is the fundamental idea. 00:43:15.790 --> 00:43:19.910 align:middle line:90% So this is how clip works. 00:43:19.910 --> 00:43:25.470 align:middle line:84% What they did is they took a 12-block, 12-layer, 00:43:25.470 --> 00:43:31.642 align:middle line:84% 8-head Transformer Causal Encoder Stack as a text encoder. 00:43:31.642 --> 00:43:33.150 align:middle line:90% Now, you understand this, right? 00:43:33.150 --> 00:43:35.270 align:middle line:90% That's what it is. 00:43:35.270 --> 00:43:38.820 align:middle line:84% 8-head, 12-layer Transformer Causal Encoder, TCE Stack. 00:43:38.820 --> 00:43:40.520 align:middle line:90% And that's a text encoder. 00:43:40.520 --> 00:43:42.500 align:middle line:84% So we send any piece of text through it. 00:43:42.500 --> 00:43:45.220 align:middle line:84% You get the next word prediction embedding. 00:43:45.220 --> 00:43:47.780 align:middle line:84% And that's embedding you're going to use. 00:43:47.780 --> 00:43:51.120 align:middle line:84% And they took ResNet-50 and made it the image encoder. 00:43:51.120 --> 00:43:54.020 align:middle line:84% They took ResNet-50, chopped off the top. 00:43:54.020 --> 00:43:57.220 align:middle line:84% And whatever was left is the image encoder. 00:43:57.220 --> 00:43:59.060 align:middle line:90% OK. 00:43:59.060 --> 00:44:03.020 align:middle line:84% Then, they initialized with random weights, these things, 00:44:03.020 --> 00:44:06.760 align:middle line:84% and then they grab a batch of image caption pairs. 00:44:06.760 --> 00:44:09.820 align:middle line:84% So in this example, let's say that we have these three images, 00:44:09.820 --> 00:44:13.060 align:middle line:84% and I have captions to go with these images. 00:44:13.060 --> 00:44:14.860 align:middle line:90% We have these three things. 00:44:14.860 --> 00:44:17.380 align:middle line:90% And this is the key step. 00:44:17.380 --> 00:44:20.820 align:middle line:84% They run the images through the image encoder and the captions 00:44:20.820 --> 00:44:23.600 align:middle line:84% through the text encoder and get these embeddings. 00:44:23.600 --> 00:44:24.860 align:middle line:90% It's a forward pass. 00:44:24.860 --> 00:44:29.580 align:middle line:84% You send it through this network, you get to embeddings. 00:44:29.580 --> 00:44:32.002 align:middle line:90% And then, this is what they do. 00:44:32.002 --> 00:44:34.460 align:middle line:84% With these embeddings, they calculate the cosine similarity 00:44:34.460 --> 00:44:36.380 align:middle line:90% for every image caption pair. 00:44:36.380 --> 00:44:37.060 align:middle line:90% OK. 00:44:37.060 --> 00:44:39.340 align:middle line:84% And so imagine something like this. 00:44:39.340 --> 00:44:41.420 align:middle line:84% So you have these three captions. 00:44:41.420 --> 00:44:43.200 align:middle line:90% You have these three images. 00:44:43.200 --> 00:44:45.540 align:middle line:90% And those are the embeddings. 00:44:45.540 --> 00:44:48.140 align:middle line:84% And then, they calculate the cosine similarity 00:44:48.140 --> 00:44:50.870 align:middle line:90% for every one of those things. 00:44:50.870 --> 00:44:52.620 align:middle line:84% It took me like five minutes or 10 minutes 00:44:52.620 --> 00:44:54.340 align:middle line:90% to do this PowerPoint. 00:44:54.340 --> 00:44:55.581 align:middle line:90% You're welcome. 00:44:55.581 --> 00:45:00.660 align:middle line:90% [LAUGHS] 00:45:00.660 --> 00:45:02.580 align:middle line:84% Particularly, trying to get this comma to line 00:45:02.580 --> 00:45:05.060 align:middle line:90% was as a real pain in the neck. 00:45:05.060 --> 00:45:06.180 align:middle line:90% So all right. 00:45:06.180 --> 00:45:09.020 align:middle line:84% So we have this here OK, now what 00:45:09.020 --> 00:45:14.140 align:middle line:84% we want to do is we want these scores to be as high as possible 00:45:14.140 --> 00:45:17.020 align:middle line:84% because the scores in the diagonal 00:45:17.020 --> 00:45:20.050 align:middle line:84% are the ones for the matching picture and caption. 00:45:20.050 --> 00:45:24.282 align:middle line:90% 00:45:24.282 --> 00:45:27.185 align:middle line:84% Those are the scores for the matching pairs of embeddings. 00:45:27.185 --> 00:45:28.810 align:middle line:84% We want them to be as high as possible. 00:45:28.810 --> 00:45:33.420 align:middle line:90% 00:45:33.420 --> 00:45:36.410 align:middle line:84% So we want to maximize the sum of the green cells. 00:45:36.410 --> 00:45:39.490 align:middle line:84% These are the green cells, the diagonal. 00:45:39.490 --> 00:45:42.170 align:middle line:84% So if you want to write it as a loss function, 00:45:42.170 --> 00:45:44.270 align:middle line:84% because the loss function is always minimization, 00:45:44.270 --> 00:45:47.660 align:middle line:84% we basically say minimize the negative sum of the green cells. 00:45:47.660 --> 00:45:50.782 align:middle line:90% 00:45:50.782 --> 00:45:53.790 align:middle line:84% So the question is, will this loss function do the trick? 00:45:53.790 --> 00:45:58.770 align:middle line:90% 00:45:58.770 --> 00:46:00.330 align:middle line:90% It seems reasonable. 00:46:00.330 --> 00:46:02.970 align:middle line:84% You want to make sure that related things are really close 00:46:02.970 --> 00:46:03.630 align:middle line:90% together. 00:46:03.630 --> 00:46:04.870 align:middle line:90% So you want to maximize. 00:46:04.870 --> 00:46:07.197 align:middle line:90% 00:46:07.197 --> 00:46:09.530 align:middle line:84% STUDENT: If that was the only part of the loss function, 00:46:09.530 --> 00:46:11.690 align:middle line:84% wouldn't it just kind of squish everything 00:46:11.690 --> 00:46:13.210 align:middle line:90% to the same spot in space? 00:46:13.210 --> 00:46:14.650 align:middle line:90% PROFESSOR: Correct. 00:46:14.650 --> 00:46:16.730 align:middle line:84% What it's going to do is it's going 00:46:16.730 --> 00:46:20.170 align:middle line:90% to basically ignore the input. 00:46:20.170 --> 00:46:22.230 align:middle line:84% The optimizer can simply ignore the input, 00:46:22.230 --> 00:46:24.450 align:middle line:84% make all the embeddings the same. 00:46:24.450 --> 00:46:28.410 align:middle line:84% For example, it can just make all the embeddings 0. 00:46:28.410 --> 00:46:28.950 align:middle line:90% That's it. 00:46:28.950 --> 00:46:31.690 align:middle line:84% And then, now, we have a perfect cosine similarity 00:46:31.690 --> 00:46:33.160 align:middle line:90% for everything. 00:46:33.160 --> 00:46:35.460 align:middle line:84% For any pair of image and captions, 00:46:35.460 --> 00:46:37.240 align:middle line:84% the cosine similarity is going to be 1. 00:46:37.240 --> 00:46:38.400 align:middle line:90% It's perfect. 00:46:38.400 --> 00:46:40.640 align:middle line:90% So clearly, that's not enough. 00:46:40.640 --> 00:46:43.240 align:middle line:84% This is, by the way, is called model collapse. 00:46:43.240 --> 00:46:45.980 align:middle line:84% So to prevent it from doing that, 00:46:45.980 --> 00:46:48.320 align:middle line:84% we need to do one more thing to the loss function. 00:46:48.320 --> 00:46:48.950 align:middle line:90% Any guesses? 00:46:48.950 --> 00:46:51.560 align:middle line:90% 00:46:51.560 --> 00:46:53.360 align:middle line:90% Yeah. 00:46:53.360 --> 00:46:55.960 align:middle line:84% STUDENT: Make the images that aren't related 00:46:55.960 --> 00:46:58.098 align:middle line:90% not have a cosine similarity. 00:46:58.098 --> 00:46:59.140 align:middle line:90% PROFESSOR: Exactly right. 00:46:59.140 --> 00:47:00.200 align:middle line:90% Exactly right. 00:47:00.200 --> 00:47:04.280 align:middle line:84% So what we want to do is we want the scores of the red stuff 00:47:04.280 --> 00:47:07.240 align:middle line:90% to be as small as possible. 00:47:07.240 --> 00:47:08.840 align:middle line:84% We want the green stuff to be as much 00:47:08.840 --> 00:47:12.560 align:middle line:84% as possible and the red stuff to be as small as possible. 00:47:12.560 --> 00:47:14.420 align:middle line:84% Together, it'll get the job done. 00:47:14.420 --> 00:47:19.478 align:middle line:90% 00:47:19.478 --> 00:47:21.520 align:middle line:84% So we want to maximize the sum of the green cells 00:47:21.520 --> 00:47:23.360 align:middle line:84% and minimize the sum of the red cells, 00:47:23.360 --> 00:47:26.400 align:middle line:84% so the equivalent loss function is minimize 00:47:26.400 --> 00:47:29.440 align:middle line:84% the sum of the red cells and the negative sum of the green cells. 00:47:29.440 --> 00:47:30.830 align:middle line:90% That's it. 00:47:30.830 --> 00:47:35.750 align:middle line:84% So all CLIP does is that it just grabs a batch of image caption 00:47:35.750 --> 00:47:38.470 align:middle line:84% pairs, runs it through the networks, 00:47:38.470 --> 00:47:41.550 align:middle line:84% calculates the embeddings and calculates some of the stuff 00:47:41.550 --> 00:47:42.570 align:middle line:90% here. 00:47:42.570 --> 00:47:43.910 align:middle line:90% And that is your loss. 00:47:43.910 --> 00:47:46.190 align:middle line:84% And then, back propagate through the network. 00:47:46.190 --> 00:47:47.006 align:middle line:90% Boom. 00:47:47.006 --> 00:47:48.230 align:middle line:90% Batch, batch, batch. 00:47:48.230 --> 00:47:49.710 align:middle line:90% Do it a whole bunch of times. 00:47:49.710 --> 00:47:52.310 align:middle line:84% And OpenAI did this with, oh, this 00:47:52.310 --> 00:47:55.390 align:middle line:84% is the official picture from the paper, which 00:47:55.390 --> 00:47:56.870 align:middle line:90% is worth reading, by the way. 00:47:56.870 --> 00:47:59.010 align:middle line:90% It comes in, text encoder. 00:47:59.010 --> 00:48:02.110 align:middle line:84% You get these embedding vectors, image encoder. 00:48:02.110 --> 00:48:05.250 align:middle line:84% And then, boom, the diagonal is maximized 00:48:05.250 --> 00:48:07.790 align:middle line:84% and the off diagonals are minimized. 00:48:07.790 --> 00:48:11.230 align:middle line:84% And they did it with 400 million image caption pairs 00:48:11.230 --> 00:48:14.430 align:middle line:90% scraped from the internet. 00:48:14.430 --> 00:48:16.430 align:middle line:90% 400 million. 00:48:16.430 --> 00:48:18.750 align:middle line:84% By the way, folks who work in this space 00:48:18.750 --> 00:48:21.630 align:middle line:84% may know this really well, but one very 00:48:21.630 --> 00:48:25.190 align:middle line:84% easy way to get a caption for an image. 00:48:25.190 --> 00:48:27.970 align:middle line:84% We see images, but where do you think the captions come from? 00:48:27.970 --> 00:48:29.455 align:middle line:84% Where did they get those captions? 00:48:29.455 --> 00:48:31.580 align:middle line:84% Obviously, they didn't ask people to manually label 00:48:31.580 --> 00:48:32.860 align:middle line:90% each image with the caption. 00:48:32.860 --> 00:48:35.020 align:middle line:84% Where do you think they got it from? 00:48:35.020 --> 00:48:36.460 align:middle line:90% STUDENT: Google Search? 00:48:36.460 --> 00:48:39.340 align:middle line:84% PROFESSOR: Google Search can help, but why does 00:48:39.340 --> 00:48:41.160 align:middle line:84% Google Search actually find the caption? 00:48:41.160 --> 00:48:42.868 align:middle line:84% How does it, because Google Search is not 00:48:42.868 --> 00:48:45.318 align:middle line:90% creating the caption. 00:48:45.318 --> 00:48:47.360 align:middle line:84% STUDENT: Take it from the alt text on the images. 00:48:47.360 --> 00:48:49.460 align:middle line:90% PROFESSOR: Correct, alt text. 00:48:49.460 --> 00:48:51.480 align:middle line:84% So a lot of folks, for accessibility reasons, 00:48:51.480 --> 00:48:54.320 align:middle line:84% they have alt text on the images they create. 00:48:54.320 --> 00:48:56.220 align:middle line:84% A lot of people have alt text in their images 00:48:56.220 --> 00:48:59.220 align:middle line:84% that they have published on the web, and that's what we use. 00:48:59.220 --> 00:49:01.780 align:middle line:84% And the alt text actually, ends up 00:49:01.780 --> 00:49:04.620 align:middle line:84% being a more verbose description of the image 00:49:04.620 --> 00:49:07.420 align:middle line:84% than a typical caption, which tends to be much briefer. 00:49:07.420 --> 00:49:10.660 align:middle line:84% And for us, more verbose, longer the better, 00:49:10.660 --> 00:49:14.540 align:middle line:84% because there's more stuff for the model to learn from. 00:49:14.540 --> 00:49:17.180 align:middle line:90% So that's why they built CLIP. 00:49:17.180 --> 00:49:20.300 align:middle line:84% And so now, what we do is we can use CLIP's text 00:49:20.300 --> 00:49:21.980 align:middle line:90% encoder by itself. 00:49:21.980 --> 00:49:25.100 align:middle line:84% We can send in any text and get an embedding that 00:49:25.100 --> 00:49:27.690 align:middle line:84% is close to the embedding of any image that's 00:49:27.690 --> 00:49:31.010 align:middle line:90% described by the text. 00:49:31.010 --> 00:49:31.890 align:middle line:90% OK. 00:49:31.890 --> 00:49:35.650 align:middle line:84% Now, by the way, CLIP can be used for zero-shot image 00:49:35.650 --> 00:49:37.410 align:middle line:90% classification. 00:49:37.410 --> 00:49:39.830 align:middle line:84% And what I mean by zero-shot image classification, 00:49:39.830 --> 00:49:41.850 align:middle line:84% and I'll walk through the picture in a second, 00:49:41.850 --> 00:49:45.150 align:middle line:84% is that typically when you want to build an image classifier, 00:49:45.150 --> 00:49:47.770 align:middle line:84% you get a whole bunch of training data of images 00:49:47.770 --> 00:49:50.270 align:middle line:84% and their labels, and then we train them. 00:49:50.270 --> 00:49:54.010 align:middle line:84% Maybe you take something like ResNet, chop off the top, 00:49:54.010 --> 00:49:56.320 align:middle line:84% attach our own output head and train, train, train. 00:49:56.320 --> 00:49:56.820 align:middle line:90% Boom. 00:49:56.820 --> 00:49:58.190 align:middle line:90% You have a classifier. 00:49:58.190 --> 00:50:00.890 align:middle line:84% But the only problem with that is, let's say that tomorrow-- 00:50:00.890 --> 00:50:04.268 align:middle line:84% so today, for example, you had five classes in your problem. 00:50:04.268 --> 00:50:06.810 align:middle line:84% And tomorrow, somebody comes along and says, oh, actually, we 00:50:06.810 --> 00:50:09.357 align:middle line:90% have a sixth category. 00:50:09.357 --> 00:50:10.190 align:middle line:90% What do you do then? 00:50:10.190 --> 00:50:11.690 align:middle line:84% Well, you have to go back to the drawing board 00:50:11.690 --> 00:50:13.990 align:middle line:84% and retrain the whole thing with six labels now, 00:50:13.990 --> 00:50:16.450 align:middle line:84% not five, because your problem has changed. 00:50:16.450 --> 00:50:19.330 align:middle line:84% Wouldn't it be great if you had a classifier where you just 00:50:19.330 --> 00:50:21.288 align:middle line:84% come to it and say, here's an image, 00:50:21.288 --> 00:50:23.830 align:middle line:84% and here are the six possible labels I want you to pick from. 00:50:23.830 --> 00:50:25.380 align:middle line:90% Pick one for me. 00:50:25.380 --> 00:50:27.880 align:middle line:84% And you want to be able to give it a different set of labels 00:50:27.880 --> 00:50:29.560 align:middle line:90% each time. 00:50:29.560 --> 00:50:32.360 align:middle line:84% And it will just use the labels you're giving it, and the image, 00:50:32.360 --> 00:50:34.040 align:middle line:84% and figures out which label corresponds 00:50:34.040 --> 00:50:35.880 align:middle line:90% to the image you just fed it. 00:50:35.880 --> 00:50:38.920 align:middle line:84% That would be an insanely flexible image classification 00:50:38.920 --> 00:50:40.440 align:middle line:90% system. 00:50:40.440 --> 00:50:43.560 align:middle line:84% And that's what I mean by zero-shot image classification. 00:50:43.560 --> 00:50:48.523 align:middle line:84% And you can use CLIP to do zero-shot image classification. 00:50:48.523 --> 00:50:50.440 align:middle line:84% Now, how you do it is actually in the picture, 00:50:50.440 --> 00:50:52.280 align:middle line:90% though not very clearly done. 00:50:52.280 --> 00:50:53.460 align:middle line:90% Anyone wants to guess? 00:50:53.460 --> 00:50:58.120 align:middle line:90% 00:50:58.120 --> 00:51:02.320 align:middle line:84% How can you use CLIP to build an infinitely flexible image 00:51:02.320 --> 00:51:03.100 align:middle line:90% classifier? 00:51:03.100 --> 00:51:12.560 align:middle line:90% 00:51:12.560 --> 00:51:15.480 align:middle line:84% STUDENT: I mean, the text input was trained BERT, right? 00:51:15.480 --> 00:51:17.360 align:middle line:84% So in the same way BERT can handle 00:51:17.360 --> 00:51:20.787 align:middle line:84% words it's never seen before, does it essentially do that? 00:51:20.787 --> 00:51:22.870 align:middle line:84% PROFESSOR: Sorry, say that again, the second part. 00:51:22.870 --> 00:51:25.110 align:middle line:84% STUDENT: You're saying it sees a text input with something 00:51:25.110 --> 00:51:26.270 align:middle line:90% it's never seen before, right? 00:51:26.270 --> 00:51:26.550 align:middle line:90% PROFESSOR: Yeah. 00:51:26.550 --> 00:51:27.130 align:middle line:90% STUDENT: OK. 00:51:27.130 --> 00:51:30.230 align:middle line:84% So in the BERT model, which is where it came from, 00:51:30.230 --> 00:51:31.370 align:middle line:90% in the text encoding. 00:51:31.370 --> 00:51:32.870 align:middle line:84% In the BERT model, I think we talked 00:51:32.870 --> 00:51:35.550 align:middle line:84% about when it sees a word it doesn't know, 00:51:35.550 --> 00:51:39.270 align:middle line:84% it's never seen before, it can use the context 00:51:39.270 --> 00:51:40.692 align:middle line:90% words around it to try to-- 00:51:40.692 --> 00:51:41.650 align:middle line:90% PROFESSOR: Right track. 00:51:41.650 --> 00:51:45.470 align:middle line:84% But here, just to be clear, I want 00:51:45.470 --> 00:51:47.990 align:middle line:84% you to use CLIP that we just built. 00:51:47.990 --> 00:51:51.030 align:middle line:84% And assume CLIP knows all the words 00:51:51.030 --> 00:51:52.993 align:middle line:84% because it's been trained on a big vocabulary. 00:51:52.993 --> 00:51:54.410 align:middle line:84% You can give it any text you want. 00:51:54.410 --> 00:51:56.870 align:middle line:84% It will create an embedding from it. 00:51:56.870 --> 00:51:58.290 align:middle line:90% That's the key capability. 00:51:58.290 --> 00:52:02.230 align:middle line:90% 00:52:02.230 --> 00:52:05.843 align:middle line:84% STUDENT: So it creates a text embedding for a text compute. 00:52:05.843 --> 00:52:06.510 align:middle line:90% PROFESSOR: Yeah. 00:52:06.510 --> 00:52:08.468 align:middle line:84% STUDENT: But because there are things, and then 00:52:08.468 --> 00:52:11.190 align:middle line:90% for image [INAUDIBLE]. 00:52:11.190 --> 00:52:14.790 align:middle line:84% So for comparing the similarity scores between the two, 00:52:14.790 --> 00:52:16.850 align:middle line:84% image is complete but the text is not complete. 00:52:16.850 --> 00:52:18.210 align:middle line:90% So there will be missing spaces. 00:52:18.210 --> 00:52:20.970 align:middle line:84% And then, it makes a prediction using this method. 00:52:20.970 --> 00:52:24.060 align:middle line:84% PROFESSOR: But why is there a missing piece in the text? 00:52:24.060 --> 00:52:28.420 align:middle line:84% STUDENT: Because the image, the text, 00:52:28.420 --> 00:52:31.780 align:middle line:84% the text does not contain the class. 00:52:31.780 --> 00:52:35.140 align:middle line:84% And then, well, for the image, the way it was trained, 00:52:35.140 --> 00:52:38.280 align:middle line:84% it was trained with pairs with class included. 00:52:38.280 --> 00:52:41.040 align:middle line:84% PROFESSOR: Right, but we actually know the class now. 00:52:41.040 --> 00:52:44.620 align:middle line:84% So the use case is that I come to you with an image and I say, 00:52:44.620 --> 00:52:48.060 align:middle line:84% here are the seven possible labels for this image. 00:52:48.060 --> 00:52:51.500 align:middle line:84% And each label is a piece of text. 00:52:51.500 --> 00:52:55.180 align:middle line:84% So you actually have seven pieces of text and an image. 00:52:55.180 --> 00:52:58.920 align:middle line:84% And all I want CLIP to do is to tell me, OK, the seventh, 00:52:58.920 --> 00:53:03.310 align:middle line:84% the fourth label is the right one for this image, 00:53:03.310 --> 00:53:04.560 align:middle line:90% but you're on the right track. 00:53:04.560 --> 00:53:07.780 align:middle line:90% 00:53:07.780 --> 00:53:10.280 align:middle line:84% Once you see how it's done, you'll be like, yeah, of course. 00:53:10.280 --> 00:53:13.123 align:middle line:90% 00:53:13.123 --> 00:53:15.040 align:middle line:84% STUDENT: I may not be understanding something, 00:53:15.040 --> 00:53:16.998 align:middle line:84% but wouldn't you just pick the embedding that's 00:53:16.998 --> 00:53:19.772 align:middle line:84% the closest to the, like, the text embedding that's 00:53:19.772 --> 00:53:21.230 align:middle line:84% the closest to the image embedding? 00:53:21.230 --> 00:53:22.170 align:middle line:90% PROFESSOR: Correct. 00:53:22.170 --> 00:53:23.070 align:middle line:90% You're not missing anything. 00:53:23.070 --> 00:53:24.130 align:middle line:90% That's the right answer. 00:53:24.130 --> 00:53:25.250 align:middle line:90% Well done. 00:53:25.250 --> 00:53:26.910 align:middle line:90% [LAUGHS] Come on, people. 00:53:26.910 --> 00:53:29.030 align:middle line:90% Can we applaud our fellow peer? 00:53:29.030 --> 00:53:30.210 align:middle line:90% [APPLAUSE] 00:53:30.210 --> 00:53:32.810 align:middle line:90% You folks are hard to impress. 00:53:32.810 --> 00:53:34.170 align:middle line:90% That's exactly what we do. 00:53:34.170 --> 00:53:40.073 align:middle line:84% So here, the key thing to remember, 00:53:40.073 --> 00:53:41.490 align:middle line:84% the key thing to keep in your head 00:53:41.490 --> 00:53:47.310 align:middle line:84% is that when a label is just text, dog, cat, it's just text. 00:53:47.310 --> 00:53:50.890 align:middle line:84% So you can just imagine taking each label, which in this case 00:53:50.890 --> 00:53:53.770 align:middle line:90% is plane, car, dog, whatever. 00:53:53.770 --> 00:53:55.930 align:middle line:84% For each one of them, you create an embedding. 00:53:55.930 --> 00:53:58.690 align:middle line:84% You get T1 through whatever, if you have n labels. 00:53:58.690 --> 00:54:01.047 align:middle line:84% For the image, you just have one embedding, I. 00:54:01.047 --> 00:54:02.630 align:middle line:84% And then, you just create the cosines, 00:54:02.630 --> 00:54:04.050 align:middle line:90% calculate the cosine similarity. 00:54:04.050 --> 00:54:06.790 align:middle line:84% And whichever is the highest number, you say, OK, it's a dog. 00:54:06.790 --> 00:54:09.250 align:middle line:90% That's it. 00:54:09.250 --> 00:54:12.070 align:middle line:84% It's super, just imagine the level of flexibility here. 00:54:12.070 --> 00:54:15.250 align:middle line:90% 00:54:15.250 --> 00:54:19.313 align:middle line:84% So that's a side use of CLIP unrelated to diffusion models, 00:54:19.313 --> 00:54:20.980 align:middle line:84% but I just thought it was really clever, 00:54:20.980 --> 00:54:22.063 align:middle line:90% so I wanted to share that. 00:54:22.063 --> 00:54:22.903 align:middle line:90% OK, good. 00:54:22.903 --> 00:54:24.320 align:middle line:84% Now, let's see how we can actually 00:54:24.320 --> 00:54:28.120 align:middle line:84% use this entire capability to go to solve the original problem we 00:54:28.120 --> 00:54:31.000 align:middle line:84% set out to solve, which is can we steer the diffusion 00:54:31.000 --> 00:54:33.840 align:middle line:84% model to create an image based on a particular prompt we give 00:54:33.840 --> 00:54:35.240 align:middle line:90% it? 00:54:35.240 --> 00:54:38.520 align:middle line:84% So now, remember, if we go back to how we did it, 00:54:38.520 --> 00:54:40.800 align:middle line:84% we created all these training pairs of x and y 00:54:40.800 --> 00:54:44.240 align:middle line:90% based on denoising the image. 00:54:44.240 --> 00:54:47.320 align:middle line:84% X is the image, y is the less noisy version of the image. 00:54:47.320 --> 00:54:51.640 align:middle line:84% So what we can simply do is we can actually 00:54:51.640 --> 00:54:54.920 align:middle line:84% change the input so it becomes the image, and then 00:54:54.920 --> 00:54:59.287 align:middle line:84% the clip text embedding of the caption for that image. 00:54:59.287 --> 00:55:01.120 align:middle line:84% So you have an image and you have a caption. 00:55:01.120 --> 00:55:03.440 align:middle line:84% You take the caption, run it through CLIP. 00:55:03.440 --> 00:55:04.760 align:middle line:90% You get an embedding. 00:55:04.760 --> 00:55:07.960 align:middle line:84% By definition, that embedding lives 00:55:07.960 --> 00:55:10.160 align:middle line:84% in the same space as all the images that 00:55:10.160 --> 00:55:11.293 align:middle line:90% correspond to that caption. 00:55:11.293 --> 00:55:14.000 align:middle line:90% 00:55:14.000 --> 00:55:17.150 align:middle line:84% So you just attach, you concatenate the embedding 00:55:17.150 --> 00:55:20.890 align:middle line:84% of the CLIP output of a caption along with the image. 00:55:20.890 --> 00:55:23.230 align:middle line:90% You make that the new input now. 00:55:23.230 --> 00:55:25.773 align:middle line:84% Why continues to be the less noisy version of the image? 00:55:25.773 --> 00:55:28.190 align:middle line:84% Or as we saw earlier, it could be just the noise component 00:55:28.190 --> 00:55:29.750 align:middle line:90% of that image. 00:55:29.750 --> 00:55:33.310 align:middle line:84% OK, this is the new x, y pair that we have. 00:55:33.310 --> 00:55:37.310 align:middle line:84% And so now, the model is you send the CLIP text embedding, 00:55:37.310 --> 00:55:38.230 align:middle line:90% the image x. 00:55:38.230 --> 00:55:40.410 align:middle line:84% Send it through this noisy version of the image. 00:55:40.410 --> 00:55:42.550 align:middle line:84% And you keep on training it for a while. 00:55:42.550 --> 00:55:44.550 align:middle line:84% Once your model is trained for when 00:55:44.550 --> 00:55:47.590 align:middle line:84% you want to use it for inference, for a new prompt, 00:55:47.590 --> 00:55:51.890 align:middle line:84% you just give it, Killian Court at MIT during the spring time, 00:55:51.890 --> 00:55:54.270 align:middle line:90% along with a bunch of noise. 00:55:54.270 --> 00:55:56.610 align:middle line:90% Goes in, it starts denoising it. 00:55:56.610 --> 00:55:59.370 align:middle line:84% But because this embedding of this thing, 00:55:59.370 --> 00:56:01.830 align:middle line:84% thanks to CLIP, lives in the same image space 00:56:01.830 --> 00:56:05.120 align:middle line:84% as all Killian Court embeddings, Killian Court images, 00:56:05.120 --> 00:56:06.820 align:middle line:84% you keep on doing it for a while. 00:56:06.820 --> 00:56:08.570 align:middle line:84% At some point, you will get Killian Court. 00:56:08.570 --> 00:56:11.253 align:middle line:90% 00:56:11.253 --> 00:56:12.170 align:middle line:90% That's how they do it. 00:56:12.170 --> 00:56:14.620 align:middle line:90% That's how they steer the image. 00:56:14.620 --> 00:56:15.760 align:middle line:90% It's a two-step process. 00:56:15.760 --> 00:56:18.220 align:middle line:84% You create all these clip embeddings, 00:56:18.220 --> 00:56:20.560 align:middle line:84% which CLIP was a breakthrough, in my opinion, 00:56:20.560 --> 00:56:22.842 align:middle line:84% because it was one of the, maybe the first example. 00:56:22.842 --> 00:56:24.300 align:middle line:84% I don't know if it's the very first 00:56:24.300 --> 00:56:26.580 align:middle line:84% but one of the early examples of saying 00:56:26.580 --> 00:56:29.200 align:middle line:84% we have different kinds of data, we have images, 00:56:29.200 --> 00:56:31.000 align:middle line:90% we have captions, we have text. 00:56:31.000 --> 00:56:33.140 align:middle line:84% How do we create embeddings for every one 00:56:33.140 --> 00:56:35.020 align:middle line:84% of these very different data types 00:56:35.020 --> 00:56:37.820 align:middle line:84% that all happen to live in the same space, the same concept 00:56:37.820 --> 00:56:38.320 align:middle line:90% space? 00:56:38.320 --> 00:56:39.780 align:middle line:90% That was the key idea. 00:56:39.780 --> 00:56:42.420 align:middle line:84% And if you look at the modern multimodal large language 00:56:42.420 --> 00:56:46.260 align:middle line:84% models, they are all based on the same exact idea. 00:56:46.260 --> 00:56:49.400 align:middle line:84% So it's very powerful, this approach. 00:56:49.400 --> 00:56:50.900 align:middle line:84% STUDENT: Yeah, now I understand this 00:56:50.900 --> 00:56:54.700 align:middle line:84% for images, but for video generation models like Sora, 00:56:54.700 --> 00:56:58.420 align:middle line:84% do they have some sort of underlying physics structure, 00:56:58.420 --> 00:57:00.880 align:middle line:84% or do they learn the physical representations? 00:57:00.880 --> 00:57:02.297 align:middle line:84% PROFESSOR: There's a lot of debate 00:57:02.297 --> 00:57:03.963 align:middle line:84% on the internet about this stuff. 00:57:03.963 --> 00:57:05.380 align:middle line:84% They haven't published the results 00:57:05.380 --> 00:57:08.060 align:middle line:84% of the full technical report yet, so we don't know for sure. 00:57:08.060 --> 00:57:10.460 align:middle line:84% But the consensus seems to be no, they 00:57:10.460 --> 00:57:12.100 align:middle line:90% are not using a physics engine. 00:57:12.100 --> 00:57:14.902 align:middle line:84% What they have done, and again, this may be wrong. 00:57:14.902 --> 00:57:16.860 align:middle line:84% Once the report comes out, we'll know for sure. 00:57:16.860 --> 00:57:20.320 align:middle line:84% But what people are saying, computer vision experts, 00:57:20.320 --> 00:57:26.180 align:middle line:84% is that it has been trained on a lot of video game data, 00:57:26.180 --> 00:57:29.060 align:middle line:84% along with actual videos and so on. 00:57:29.060 --> 00:57:31.460 align:middle line:84% And the corpus of training is so massive 00:57:31.460 --> 00:57:34.900 align:middle line:84% that it has basically learned to mimic certain physics aspects 00:57:34.900 --> 00:57:36.600 align:middle line:90% to it just as a side effect. 00:57:36.600 --> 00:57:39.720 align:middle line:84% Much like LLMs, you train them on a large amount of text data, 00:57:39.720 --> 00:57:42.860 align:middle line:84% they begin to start to do things which you didn't anticipate 00:57:42.860 --> 00:57:44.560 align:middle line:90% that they'll do. 00:57:44.560 --> 00:57:46.320 align:middle line:84% So for example, I read this, I thought 00:57:46.320 --> 00:57:49.460 align:middle line:84% it was a really great example of what is surprising 00:57:49.460 --> 00:57:52.900 align:middle line:84% about large language models is not that you train them 00:57:52.900 --> 00:57:54.593 align:middle line:84% on a bunch of high school math problems, 00:57:54.593 --> 00:57:56.760 align:middle line:84% and then you give it a new high school math problem. 00:57:56.760 --> 00:57:57.802 align:middle line:90% It can actually solve it. 00:57:57.802 --> 00:57:58.825 align:middle line:90% That's not surprising. 00:57:58.825 --> 00:58:00.700 align:middle line:84% You give it a whole bunch of high school math 00:58:00.700 --> 00:58:02.900 align:middle line:84% problems in English, then you ask 00:58:02.900 --> 00:58:04.867 align:middle line:84% it to read a bunch of French literature, 00:58:04.867 --> 00:58:06.700 align:middle line:84% and then you give it French high school math 00:58:06.700 --> 00:58:08.040 align:middle line:90% problems to solve it. 00:58:08.040 --> 00:58:10.010 align:middle line:90% That is the new news. 00:58:10.010 --> 00:58:13.090 align:middle line:84% So similarly here, I think the expectation 00:58:13.090 --> 00:58:15.757 align:middle line:84% is that it's not actually using a physics engine under the hood. 00:58:15.757 --> 00:58:17.965 align:middle line:84% It may have used a physics engine to actually come up 00:58:17.965 --> 00:58:19.690 align:middle line:84% with the videos and renderings, but there 00:58:19.690 --> 00:58:22.440 align:middle line:84% are no physics constraints in the model itself. 00:58:22.440 --> 00:58:24.190 align:middle line:84% It just comes out of the training process. 00:58:24.190 --> 00:58:25.690 align:middle line:90% That's the current view. 00:58:25.690 --> 00:58:27.190 align:middle line:84% Once the technical report comes out, 00:58:27.190 --> 00:58:29.230 align:middle line:84% we'll know for sure what they actually did. 00:58:29.230 --> 00:58:33.730 align:middle line:90% 00:58:33.730 --> 00:58:36.030 align:middle line:84% STUDENT: So one quick question about stability. 00:58:36.030 --> 00:58:38.930 align:middle line:84% It's claiming to be a little bit more real time in our image 00:58:38.930 --> 00:58:40.048 align:middle line:90% generation. 00:58:40.048 --> 00:58:41.590 align:middle line:84% PROFESSOR: You mean Stable Diffusion? 00:58:41.590 --> 00:58:41.890 align:middle line:90% STUDENT: Yeah. 00:58:41.890 --> 00:58:42.170 align:middle line:90% PROFESSOR: Yeah, yeah. 00:58:42.170 --> 00:58:42.990 align:middle line:90% STUDENT: For Stable Diffusion. 00:58:42.990 --> 00:58:45.130 align:middle line:84% So are they jumping through the noise more quickly, 00:58:45.130 --> 00:58:47.590 align:middle line:84% or are they kind of preprompting it, and kind of trick-- 00:58:47.590 --> 00:58:49.390 align:middle line:84% PROFESSOR: Very good question, and there's a very key trick. 00:58:49.390 --> 00:58:50.450 align:middle line:90% It's coming. 00:58:50.450 --> 00:58:52.770 align:middle line:90% You had a question. 00:58:52.770 --> 00:58:54.890 align:middle line:84% STUDENT: So here, the example of the noise 00:58:54.890 --> 00:58:55.943 align:middle line:90% is normal distribution. 00:58:55.943 --> 00:58:56.610 align:middle line:90% PROFESSOR: Yeah. 00:58:56.610 --> 00:59:00.090 align:middle line:84% STUDENT: However, if we have changed the noise distribution, 00:59:00.090 --> 00:59:01.215 align:middle line:90% would it change the result? 00:59:01.215 --> 00:59:02.882 align:middle line:84% PROFESSOR: Oh, you mean if you change it 00:59:02.882 --> 00:59:04.770 align:middle line:84% to a Poisson or some other distribution? 00:59:04.770 --> 00:59:06.353 align:middle line:84% It will definitely change the results. 00:59:06.353 --> 00:59:08.440 align:middle line:84% Because if you look at the underlying 00:59:08.440 --> 00:59:10.680 align:middle line:84% math of why this works, it heavily 00:59:10.680 --> 00:59:13.280 align:middle line:84% depends on the Gaussian assumption. 00:59:13.280 --> 00:59:14.800 align:middle line:90% Yeah. 00:59:14.800 --> 00:59:18.260 align:middle line:84% There was another question somewhere here. 00:59:18.260 --> 00:59:19.760 align:middle line:84% STUDENT: You may not know the answer 00:59:19.760 --> 00:59:21.220 align:middle line:84% because the technical report is not out, 00:59:21.220 --> 00:59:23.080 align:middle line:84% but could it be in terms of video generation 00:59:23.080 --> 00:59:26.960 align:middle line:84% that's analogous to going from one noisy image 00:59:26.960 --> 00:59:30.600 align:middle line:84% to another, like you're almost doing a series of still images 00:59:30.600 --> 00:59:31.620 align:middle line:90% and learning how to-- 00:59:31.620 --> 00:59:33.520 align:middle line:84% PROFESSOR: No, that, I think, people 00:59:33.520 --> 00:59:34.900 align:middle line:90% are sure is how it's done. 00:59:34.900 --> 00:59:36.400 align:middle line:84% So basically, you think of the video 00:59:36.400 --> 00:59:40.080 align:middle line:84% as just a series of frames, and each frame is an image. 00:59:40.080 --> 00:59:42.460 align:middle line:84% And there is a sequentiality to it, 00:59:42.460 --> 00:59:44.960 align:middle line:84% which is where the transformer stack will come in because it 00:59:44.960 --> 00:59:46.400 align:middle line:90% handles sequentiality. 00:59:46.400 --> 00:59:49.640 align:middle line:84% So in general, video stuff typically 00:59:49.640 --> 00:59:53.040 align:middle line:84% operates on frame by frame, which is just an image. 00:59:53.040 --> 00:59:54.260 align:middle line:90% So that is definitely there. 00:59:54.260 --> 00:59:56.880 align:middle line:84% What we don't know is if they also 00:59:56.880 --> 00:59:59.080 align:middle line:84% used some understanding of the fact 00:59:59.080 --> 01:00:02.020 align:middle line:84% that, for example, that if an object is dropped, 01:00:02.020 --> 01:00:04.800 align:middle line:84% it has to fall to the Earth in a certain rate. 01:00:04.800 --> 01:00:07.250 align:middle line:84% Or if an object goes behind another object, 01:00:07.250 --> 01:00:08.870 align:middle line:84% you can't see the object anymore. 01:00:08.870 --> 01:00:12.070 align:middle line:84% Things like that, which we take for granted. 01:00:12.070 --> 01:00:13.730 align:middle line:84% The question is, are they using it? 01:00:13.730 --> 01:00:16.368 align:middle line:84% And the consensus seems to be, in the absence 01:00:16.368 --> 01:00:18.910 align:middle line:84% of an actual technical report, that no, they're not doing it. 01:00:18.910 --> 01:00:20.785 align:middle line:84% Because there are lots of examples on Twitter 01:00:20.785 --> 01:00:23.470 align:middle line:84% where people will show a Sora video in which 01:00:23.470 --> 01:00:25.350 align:middle line:84% it's not obeying the laws of physics. 01:00:25.350 --> 01:00:28.370 align:middle line:84% So you take like a beach chair, and then put it in the sand. 01:00:28.370 --> 01:00:31.807 align:middle line:84% You see the sand come through the base of the beach chair. 01:00:31.807 --> 01:00:33.890 align:middle line:84% Or you take an object and put it behind an object, 01:00:33.890 --> 01:00:35.550 align:middle line:84% you can still see the object, even though the original object 01:00:35.550 --> 01:00:36.438 align:middle line:90% is opaque. 01:00:36.438 --> 01:00:38.730 align:middle line:84% So you'll be seeing some evidence that no, no, it's not 01:00:38.730 --> 01:00:39.610 align:middle line:90% obeying the laws of physics. 01:00:39.610 --> 01:00:41.270 align:middle line:84% What you're seeing is just an amazing-- 01:00:41.270 --> 01:00:45.010 align:middle line:84% you just learn to fake the laws of physics in some context. 01:00:45.010 --> 01:00:46.510 align:middle line:84% Much like you can fake five fingers 01:00:46.510 --> 01:00:48.677 align:middle line:84% without knowing that it has to be only five fingers. 01:00:48.677 --> 01:00:51.470 align:middle line:90% 01:00:51.470 --> 01:00:52.590 align:middle line:90% OK. 01:00:52.590 --> 01:00:53.330 align:middle line:90% All right. 01:00:53.330 --> 01:00:55.590 align:middle line:90% So let's keep going. 01:00:55.590 --> 01:00:59.928 align:middle line:84% Now, so there was another paper afterwards, 01:00:59.928 --> 01:01:01.470 align:middle line:84% and this is the original paper, which 01:01:01.470 --> 01:01:04.070 align:middle line:84% took that idea of the diffusion model. 01:01:04.070 --> 01:01:06.400 align:middle line:84% And then, diffusion models are very slow, as Olivia, 01:01:06.400 --> 01:01:07.422 align:middle line:90% you pointed out. 01:01:07.422 --> 01:01:09.380 align:middle line:84% So the question is, can we make it much faster? 01:01:09.380 --> 01:01:11.023 align:middle line:84% So what they did, and I'm not going 01:01:11.023 --> 01:01:12.440 align:middle line:84% to get into this whole thing here. 01:01:12.440 --> 01:01:15.060 align:middle line:84% I just want to highlight a couple of things. 01:01:15.060 --> 01:01:18.720 align:middle line:84% The first one is that, first of all, 01:01:18.720 --> 01:01:21.240 align:middle line:90% notice that you see U-Net here. 01:01:21.240 --> 01:01:25.740 align:middle line:84% So they are using a U-Net, to go from image-to-image. 01:01:25.740 --> 01:01:29.020 align:middle line:84% The second thing is that the CLIP embedding of the text 01:01:29.020 --> 01:01:32.140 align:middle line:84% prompt is basically, is woven, meaning 01:01:32.140 --> 01:01:36.420 align:middle line:84% it's incorporated into the U-Net through an attention 01:01:36.420 --> 01:01:38.440 align:middle line:84% mechanism, a transformer mechanism. 01:01:38.440 --> 01:01:40.820 align:middle line:84% And you can see the QKV business here, 01:01:40.820 --> 01:01:42.700 align:middle line:84% which should be familiar at this point. 01:01:42.700 --> 01:01:44.660 align:middle line:84% So it is integrated into the transformer 01:01:44.660 --> 01:01:47.940 align:middle line:84% stack directly that input the CLIP embedding, the second thing 01:01:47.940 --> 01:01:48.880 align:middle line:90% I want to point out. 01:01:48.880 --> 01:01:52.620 align:middle line:84% And then, thirdly, and this is where the speedup comes. 01:01:52.620 --> 01:01:55.040 align:middle line:84% So what you do is instead of taking the image, 01:01:55.040 --> 01:01:56.540 align:middle line:84% running it through the whole network 01:01:56.540 --> 01:01:58.900 align:middle line:84% and creating a slightly less noisy version of the image, 01:01:58.900 --> 01:02:01.300 align:middle line:84% here, what you do is you take the image, 01:02:01.300 --> 01:02:04.130 align:middle line:84% you run it through an image encoder, you get an embedding. 01:02:04.130 --> 01:02:06.610 align:middle line:84% And now, you only work with the embedding. 01:02:06.610 --> 01:02:09.090 align:middle line:84% You take the embedding and create a slightly less noisy 01:02:09.090 --> 01:02:09.990 align:middle line:90% version embedding. 01:02:09.990 --> 01:02:10.698 align:middle line:90% Keep on doing it. 01:02:10.698 --> 01:02:13.240 align:middle line:84% And these embeddings are much smaller than images, therefore, 01:02:13.240 --> 01:02:14.750 align:middle line:90% they are much faster to process. 01:02:14.750 --> 01:02:17.090 align:middle line:84% And once you've done it 1,000 times, you get a very, 01:02:17.090 --> 01:02:20.150 align:middle line:84% sort of, almost pure noiseless version of the embedding. 01:02:20.150 --> 01:02:24.050 align:middle line:84% Now, you run it through an image decoder to get the image out. 01:02:24.050 --> 01:02:30.623 align:middle line:84% So the idea here is that you operate in the latent space, 01:02:30.623 --> 01:02:31.790 align:middle line:90% meaning the embedding space. 01:02:31.790 --> 01:02:34.413 align:middle line:84% And hence, it's called the latent diffusion model. 01:02:34.413 --> 01:02:35.830 align:middle line:84% So that's where the speedup comes. 01:02:35.830 --> 01:02:38.130 align:middle line:84% But research continues to be very strong, 01:02:38.130 --> 01:02:39.138 align:middle line:90% to make it even faster. 01:02:39.138 --> 01:02:40.930 align:middle line:84% Because for a lot of consumer applications, 01:02:40.930 --> 01:02:42.890 align:middle line:84% people are obviously not going to wait around for, 01:02:42.890 --> 01:02:45.010 align:middle line:84% I mean, who wants to wait for ten seconds, right? 01:02:45.010 --> 01:02:47.860 align:middle line:84% And so there's a lot of pressure to make it even faster. 01:02:47.860 --> 01:02:52.170 align:middle line:90% 01:02:52.170 --> 01:02:52.710 align:middle line:90% All right. 01:02:52.710 --> 01:02:53.790 align:middle line:90% So that's what we have. 01:02:53.790 --> 01:02:57.730 align:middle line:84% Obviously, these models are transforming everything. 01:02:57.730 --> 01:03:00.210 align:middle line:84% And by the way, this site here, lexica.art, 01:03:00.210 --> 01:03:01.480 align:middle line:90% you can go check it out. 01:03:01.480 --> 01:03:03.760 align:middle line:84% It has a whole bunch of very interesting images 01:03:03.760 --> 01:03:05.680 align:middle line:84% and prompts that created the images. 01:03:05.680 --> 01:03:07.380 align:middle line:84% So if you're working in this space, 01:03:07.380 --> 01:03:09.060 align:middle line:84% it gives you a lot of interesting ideas. 01:03:09.060 --> 01:03:12.640 align:middle line:84% But it's not just for consumer fun applications. 01:03:12.640 --> 01:03:16.580 align:middle line:84% These models are being used to actually-- you know, AlphaFold, 01:03:16.580 --> 01:03:19.300 align:middle line:84% if you'll recall, if you give it an amino acid sequence, 01:03:19.300 --> 01:03:21.540 align:middle line:84% it actually create the 3D structure. 01:03:21.540 --> 01:03:25.720 align:middle line:84% So that's an example of, I don't think they use a diffusion 01:03:25.720 --> 01:03:28.560 align:middle line:84% model, but you can imagine using a diffusion model to create 01:03:28.560 --> 01:03:30.800 align:middle line:90% these complicated objects. 01:03:30.800 --> 01:03:34.760 align:middle line:84% Meaning, the objects you create don't have to be images. 01:03:34.760 --> 01:03:37.360 align:middle line:84% They can be arbitrarily complicated things. 01:03:37.360 --> 01:03:39.720 align:middle line:84% As long as you have enough data about such things 01:03:39.720 --> 01:03:43.440 align:middle line:84% to use for training, and the notion of noising the input 01:03:43.440 --> 01:03:45.600 align:middle line:84% is meaningful, you can create some very interesting 01:03:45.600 --> 01:03:46.460 align:middle line:90% structures. 01:03:46.460 --> 01:03:49.623 align:middle line:84% You can create 3D things and protein structures. 01:03:49.623 --> 01:03:52.040 align:middle line:84% And there's a whole bunch of very interesting applications 01:03:52.040 --> 01:03:54.320 align:middle line:90% in biomedical sciences. 01:03:54.320 --> 01:03:56.820 align:middle line:84% So this is really just the tip of the iceberg. 01:03:56.820 --> 01:03:59.390 align:middle line:84% And now, there are these things, there 01:03:59.390 --> 01:04:02.310 align:middle line:84% are ways in which you can use diffusion models 01:04:02.310 --> 01:04:04.510 align:middle line:84% to do large language modeling as well. 01:04:04.510 --> 01:04:07.350 align:middle line:84% So there's a lot of overlap and blending and so on going on 01:04:07.350 --> 01:04:08.430 align:middle line:90% in the space. 01:04:08.430 --> 01:04:10.733 align:middle line:90% So I'm going to do a quick demo. 01:04:10.733 --> 01:04:12.150 align:middle line:84% If you look at Hugging Face, there 01:04:12.150 --> 01:04:14.150 align:middle line:84% is something called the diffusers library, which 01:04:14.150 --> 01:04:16.550 align:middle line:84% is like, as the name suggests, it's 01:04:16.550 --> 01:04:19.950 align:middle line:84% a library for a lot of diffusion models. 01:04:19.950 --> 01:04:22.010 align:middle line:90% And let's take a quick look. 01:04:22.010 --> 01:04:25.790 align:middle line:90% 01:04:25.790 --> 01:04:27.827 align:middle line:84% All right, so the diffusers library 01:04:27.827 --> 01:04:29.410 align:middle line:84% has a whole bunch of diffusion models. 01:04:29.410 --> 01:04:31.577 align:middle line:84% We are going to work with Stable Diffusion, which is 01:04:31.577 --> 01:04:34.270 align:middle line:90% one of the better known models. 01:04:34.270 --> 01:04:36.490 align:middle line:90% So let's install diffusers. 01:04:36.490 --> 01:04:39.670 align:middle line:90% 01:04:39.670 --> 01:04:42.030 align:middle line:84% You will recall, when I did the quick lightning 01:04:42.030 --> 01:04:45.310 align:middle line:84% tour of the Hugging Face ecosystem for language, 01:04:45.310 --> 01:04:49.070 align:middle line:84% Hugging Face has a whole bunch of capabilities built out 01:04:49.070 --> 01:04:50.010 align:middle line:90% of the box. 01:04:50.010 --> 01:04:52.550 align:middle line:84% And you use this thing called the pipeline function to very 01:04:52.550 --> 01:04:54.670 align:middle line:90% quickly use any model you want. 01:04:54.670 --> 01:04:56.890 align:middle line:84% The same exact philosophy applies here. 01:04:56.890 --> 01:04:58.740 align:middle line:90% You still use the pipeline. 01:04:58.740 --> 01:05:00.730 align:middle line:84% So I'm going to import a bunch of stuff. 01:05:00.730 --> 01:05:09.340 align:middle line:90% 01:05:09.340 --> 01:05:10.260 align:middle line:90% All right, so. 01:05:10.260 --> 01:05:11.520 align:middle line:90% Oh, I see, I to do this thing. 01:05:11.520 --> 01:05:12.020 align:middle line:90% OK. 01:05:12.020 --> 01:05:15.980 align:middle line:90% 01:05:15.980 --> 01:05:16.740 align:middle line:90% Great. 01:05:16.740 --> 01:05:17.430 align:middle line:90% Hugging Face. 01:05:17.430 --> 01:05:23.020 align:middle line:90% 01:05:23.020 --> 01:05:24.880 align:middle line:84% All right, now that we have here. 01:05:24.880 --> 01:05:27.760 align:middle line:84% So you remember that when we worked with text, 01:05:27.760 --> 01:05:31.100 align:middle line:84% we would grab a pretrained model and then we actually 01:05:31.100 --> 01:05:32.400 align:middle line:90% run it through a pipeline. 01:05:32.400 --> 01:05:34.560 align:middle line:84% And we can do all the inference we want on it. 01:05:34.560 --> 01:05:38.520 align:middle line:84% The same exact philosophy applies here. 01:05:38.520 --> 01:05:42.500 align:middle line:84% And this is very similar to what we did in lecture 8 for NLP. 01:05:42.500 --> 01:05:44.260 align:middle line:84% So what we're going to do is, we use 01:05:44.260 --> 01:05:46.380 align:middle line:84% this command, the StableDiffusionPipeline 01:05:46.380 --> 01:05:47.440 align:middle line:90% from pretrained. 01:05:47.440 --> 01:05:51.140 align:middle line:84% And we use this version, 1.4 Stable Diffusion model. 01:05:51.140 --> 01:05:52.820 align:middle line:84% So let's just create the pipeline. 01:05:52.820 --> 01:05:56.450 align:middle line:90% 01:05:56.450 --> 01:05:58.950 align:middle line:84% And obviously, we have used TensorFlow, not PyTorch here, 01:05:58.950 --> 01:06:01.370 align:middle line:84% but a lot of these models, unfortunately, 01:06:01.370 --> 01:06:02.750 align:middle line:90% happen to be in PyTorch. 01:06:02.750 --> 01:06:05.170 align:middle line:84% So knowing a little bit of PyTorch 01:06:05.170 --> 01:06:08.450 align:middle line:84% is actually very helpful to be able to work with these things. 01:06:08.450 --> 01:06:12.170 align:middle line:84% And what we're doing here, while it's downloading, 01:06:12.170 --> 01:06:18.710 align:middle line:84% we are using this FP16 storage format for the model weights, 01:06:18.710 --> 01:06:20.970 align:middle line:84% because it's going to be a little smaller than using 01:06:20.970 --> 01:06:22.050 align:middle line:90% 32 bits. 01:06:22.050 --> 01:06:23.617 align:middle line:90% So it will download faster. 01:06:23.617 --> 01:06:24.950 align:middle line:90% So that's what's happening here. 01:06:24.950 --> 01:06:25.533 align:middle line:90% So, all right. 01:06:25.533 --> 01:06:27.390 align:middle line:90% It's downloaded fine. 01:06:27.390 --> 01:06:28.870 align:middle line:84% So now, we just give it a prompt. 01:06:28.870 --> 01:06:31.610 align:middle line:84% And this is actually one of the original famous meme 01:06:31.610 --> 01:06:35.050 align:middle line:84% prompts, a photograph of an astronaut riding a horse. 01:06:35.050 --> 01:06:37.810 align:middle line:84% And so once we have the pipeline set up, 01:06:37.810 --> 01:06:39.550 align:middle line:84% I'll just set a seed for reproducibility. 01:06:39.550 --> 01:06:42.250 align:middle line:84% And then, literally, I do pipe a prompt. 01:06:42.250 --> 01:06:46.410 align:middle line:84% And then, it's actually, you can see here, 50. 01:06:46.410 --> 01:06:49.050 align:middle line:84% So it's going through 50 denoising steps. 01:06:49.050 --> 01:06:50.410 align:middle line:90% OK. 01:06:50.410 --> 01:06:52.865 align:middle line:84% And you come up with an astronaut riding a horse. 01:06:52.865 --> 01:06:54.640 align:middle line:90% So that's that. 01:06:54.640 --> 01:06:56.340 align:middle line:84% You can actually change the seed, 01:06:56.340 --> 01:06:59.000 align:middle line:90% and you can get a different-- 01:06:59.000 --> 01:07:01.600 align:middle line:84% the seed is basically, sets the random starting point 01:07:01.600 --> 01:07:02.460 align:middle line:90% for the image. 01:07:02.460 --> 01:07:04.760 align:middle line:84% So therefore, you would expect a different astronaut. 01:07:04.760 --> 01:07:05.360 align:middle line:90% Yep. 01:07:05.360 --> 01:07:07.400 align:middle line:84% This is an astronaut riding another horse. 01:07:07.400 --> 01:07:10.480 align:middle line:84% So I think people came up with these kinds of fun examples 01:07:10.480 --> 01:07:14.105 align:middle line:84% because it's guaranteed not to be in the training data. 01:07:14.105 --> 01:07:16.280 align:middle line:84% So whatever the model is doing, it's 01:07:16.280 --> 01:07:18.760 align:middle line:84% not regurgitating what it has already seen. 01:07:18.760 --> 01:07:19.572 align:middle line:90% All right. 01:07:19.572 --> 01:07:20.280 align:middle line:90% Give me a prompt. 01:07:20.280 --> 01:07:26.546 align:middle line:90% 01:07:26.546 --> 01:07:27.330 align:middle line:90% Prompts, anyone? 01:07:27.330 --> 01:07:29.840 align:middle line:90% 01:07:29.840 --> 01:07:30.880 align:middle line:90% Wow. 01:07:30.880 --> 01:07:32.680 align:middle line:84% STUDENT: MIT professor riding a horse. 01:07:32.680 --> 01:07:34.490 align:middle line:90% PROFESSOR: [LAUGHS] Ooh. 01:07:34.490 --> 01:07:34.990 align:middle line:90% OK. 01:07:34.990 --> 01:07:38.600 align:middle line:90% 01:07:38.600 --> 01:07:40.306 align:middle line:90% That might be a-- 01:07:40.306 --> 01:07:43.859 align:middle line:90% all right, riding a horse. 01:07:43.859 --> 01:07:48.790 align:middle line:90% [LAUGHS] 01:07:48.790 --> 01:07:49.290 align:middle line:90% All right. 01:07:49.290 --> 01:07:54.750 align:middle line:90% 01:07:54.750 --> 01:07:56.630 align:middle line:90% [LAUGHS] 01:07:56.630 --> 01:07:58.110 align:middle line:90% There are two of them. 01:07:58.110 --> 01:08:01.696 align:middle line:84% And clearly, MIT professors don't have, really-- 01:08:01.696 --> 01:08:03.510 align:middle line:90% [LAUGHS] 01:08:03.510 --> 01:08:05.030 align:middle line:90% Yeah. 01:08:05.030 --> 01:08:07.270 align:middle line:90% Moving on. 01:08:07.270 --> 01:08:10.670 align:middle line:84% So by the way, you should spend some time 01:08:10.670 --> 01:08:11.890 align:middle line:90% with the diffusers library. 01:08:11.890 --> 01:08:14.070 align:middle line:84% They have a bunch of tutorials, which are really interesting. 01:08:14.070 --> 01:08:16.229 align:middle line:84% Because this core capability of giving a prompt 01:08:16.229 --> 01:08:18.189 align:middle line:84% and getting an image out can actually 01:08:18.189 --> 01:08:21.649 align:middle line:84% be manipulated for all sorts of very interesting use cases. 01:08:21.649 --> 01:08:24.670 align:middle line:84% So for example, there is this thing called negative prompting. 01:08:24.670 --> 01:08:26.470 align:middle line:84% And the idea of negative prompting 01:08:26.470 --> 01:08:29.510 align:middle line:84% is that you can give it two prompts 01:08:29.510 --> 01:08:33.069 align:middle line:84% and say, create an image which embodies the first prompt 01:08:33.069 --> 01:08:34.790 align:middle line:90% but not the second prompt. 01:08:34.790 --> 01:08:37.689 align:middle line:84% Essentially, subtract the second prompt from the first one. 01:08:37.689 --> 01:08:39.069 align:middle line:84% That's called negative prompting. 01:08:39.069 --> 01:08:41.470 align:middle line:84% And you might be wondering, what use is that? 01:08:41.470 --> 01:08:43.350 align:middle line:90% There are lots of fun uses. 01:08:43.350 --> 01:08:46.550 align:middle line:84% So here, the prompt is going to be, 01:08:46.550 --> 01:08:48.430 align:middle line:84% Labrador in the style of Vermeer. 01:08:48.430 --> 01:08:51.100 align:middle line:90% That's the first prompt. 01:08:51.100 --> 01:08:53.620 align:middle line:90% 50 steps. 01:08:53.620 --> 01:08:54.500 align:middle line:90% Look at that. 01:08:54.500 --> 01:08:56.819 align:middle line:90% Amazing, right? 01:08:56.819 --> 01:09:00.540 align:middle line:84% But maybe you don't care for the blue scarf. 01:09:00.540 --> 01:09:02.899 align:middle line:84% So you basically give it a negative prompt. 01:09:02.899 --> 01:09:05.500 align:middle line:84% And you basically, the negative prompt is blue, 01:09:05.500 --> 01:09:08.020 align:middle line:84% meaning remove everything that's blue. 01:09:08.020 --> 01:09:09.000 align:middle line:90% I don't like this. 01:09:09.000 --> 01:09:11.140 align:middle line:84% Otherwise, keep the Labrador thing going. 01:09:11.140 --> 01:09:12.479 align:middle line:90% So you run it. 01:09:12.479 --> 01:09:16.420 align:middle line:90% 01:09:16.420 --> 01:09:17.240 align:middle line:90% Look at that. 01:09:17.240 --> 01:09:18.080 align:middle line:90% The blue is gone. 01:09:18.080 --> 01:09:19.460 align:middle line:90% Negative prompting. 01:09:19.460 --> 01:09:19.979 align:middle line:90% OK. 01:09:19.979 --> 01:09:22.380 align:middle line:90% Yeah? 01:09:22.380 --> 01:09:26.859 align:middle line:84% STUDENT: If you change that from 50 to 1,000, 01:09:26.859 --> 01:09:30.140 align:middle line:84% will it become less pixelated or will it eventually just keep 01:09:30.140 --> 01:09:31.380 align:middle line:90% going and iterating? 01:09:31.380 --> 01:09:33.040 align:middle line:84% PROFESSOR: No, typically, if you do more of these things, 01:09:33.040 --> 01:09:33.840 align:middle line:90% it gets better. 01:09:33.840 --> 01:09:35.140 align:middle line:90% The quality is much better. 01:09:35.140 --> 01:09:38.100 align:middle line:84% Because each step will denoise it very slightly, 01:09:38.100 --> 01:09:40.540 align:middle line:84% so errors won't accumulate and things like that. 01:09:40.540 --> 01:09:42.899 align:middle line:84% And the diffuse library gives you lots of controls 01:09:42.899 --> 01:09:46.060 align:middle line:84% for fiddling around with all these things. 01:09:46.060 --> 01:09:48.939 align:middle line:90% OK, so that's what we had. 01:09:48.939 --> 01:09:50.460 align:middle line:90% 9:49. 01:09:50.460 --> 01:09:52.660 align:middle line:84% OK, so check out this tutorial if you're 01:09:52.660 --> 01:09:54.820 align:middle line:84% curious about how this stuff works. 01:09:54.820 --> 01:09:57.540 align:middle line:84% And I'm going to do one other thing, because I 01:09:57.540 --> 01:09:59.980 align:middle line:90% didn't get to do it earlier on. 01:09:59.980 --> 01:10:03.480 align:middle line:84% So we spent some time with the HuggingFace Hub, 01:10:03.480 --> 01:10:05.860 align:middle line:84% and I walked you through a few use cases for text 01:10:05.860 --> 01:10:09.868 align:middle line:84% where you can take a text model and use it for classification, 01:10:09.868 --> 01:10:12.160 align:middle line:84% things like that, summarization and so on and so forth. 01:10:12.160 --> 01:10:14.940 align:middle line:84% You can do the same thing for computer vision models. 01:10:14.940 --> 01:10:17.100 align:middle line:84% So if you have a computer vision problem that 01:10:17.100 --> 01:10:20.040 align:middle line:84% just maps to a standard computer vision task, 01:10:20.040 --> 01:10:22.540 align:middle line:84% you can just use the HuggingFace Hub as well. 01:10:22.540 --> 01:10:26.980 align:middle line:84% So let me just show you very quickly, the same kind of thing 01:10:26.980 --> 01:10:27.830 align:middle line:90% actually works here. 01:10:27.830 --> 01:10:32.460 align:middle line:90% 01:10:32.460 --> 01:10:35.275 align:middle line:90% All right. 01:10:35.275 --> 01:10:37.680 align:middle line:84% So let's say that you want to classify something. 01:10:37.680 --> 01:10:40.580 align:middle line:84% You just import the pipeline as before. 01:10:40.580 --> 01:10:43.580 align:middle line:84% And once you import it, you can just literally 01:10:43.580 --> 01:10:45.680 align:middle line:84% give it the standard task that you care about, 01:10:45.680 --> 01:10:48.690 align:middle line:90% like image classification. 01:10:48.690 --> 01:10:53.390 align:middle line:84% And then, you can start using it right from that point on. 01:10:53.390 --> 01:10:53.890 align:middle line:90% OK. 01:10:53.890 --> 01:10:59.770 align:middle line:90% 01:10:59.770 --> 01:11:00.400 align:middle line:90% All right. 01:11:00.400 --> 01:11:00.900 align:middle line:90% OK. 01:11:00.900 --> 01:11:03.850 align:middle line:84% So now, I'm going to just get this image. 01:11:03.850 --> 01:11:06.090 align:middle line:90% So it's a very famous image. 01:11:06.090 --> 01:11:08.290 align:middle line:84% And we're going to ask it to classify this image. 01:11:08.290 --> 01:11:12.290 align:middle line:84% So we just literally run it through the pipeline. 01:11:12.290 --> 01:11:16.210 align:middle line:84% And it says the most likely label is 94% probability. 01:11:16.210 --> 01:11:18.450 align:middle line:90% It's an Egyptian cat. 01:11:18.450 --> 01:11:19.610 align:middle line:90% Seems reasonable. 01:11:19.610 --> 01:11:21.530 align:middle line:84% I mean, it's a tough picture because there 01:11:21.530 --> 01:11:23.363 align:middle line:84% are lots of things going on in that picture. 01:11:23.363 --> 01:11:25.970 align:middle line:84% It's not like, one image, one object. 01:11:25.970 --> 01:11:28.630 align:middle line:84% So you don't have to use the default model. 01:11:28.630 --> 01:11:30.910 align:middle line:84% You can actually give it your own model that you want. 01:11:30.910 --> 01:11:35.650 align:middle line:84% So for example, you can go, you can 01:11:35.650 --> 01:11:39.730 align:middle line:84% go to the Hub, HuggingFace Hub, and you can go in there 01:11:39.730 --> 01:11:42.290 align:middle line:84% and say, all right I want image classification. 01:11:42.290 --> 01:11:45.180 align:middle line:84% These are all the models, 10,487 models. 01:11:45.180 --> 01:11:48.880 align:middle line:84% They're sorted by, I don't know, most downloads 01:11:48.880 --> 01:11:51.960 align:middle line:90% or maybe most likes. 01:11:51.960 --> 01:11:53.340 align:middle line:90% And you have all these models. 01:11:53.340 --> 01:11:54.300 align:middle line:90% You can pick any one of them. 01:11:54.300 --> 01:11:55.240 align:middle line:84% So for example, let's say you want 01:11:55.240 --> 01:11:57.220 align:middle line:84% to pick Microsoft ResNet as your one. 01:11:57.220 --> 01:11:58.560 align:middle line:90% That's what I tried here. 01:11:58.560 --> 01:11:59.900 align:middle line:90% So I have Microsoft ResNet. 01:11:59.900 --> 01:12:03.520 align:middle line:84% You just do model equals that, run it, 01:12:03.520 --> 01:12:06.200 align:middle line:84% and it takes care of all the tokenization, this, that 01:12:06.200 --> 01:12:06.820 align:middle line:90% and whatnot. 01:12:06.820 --> 01:12:08.280 align:middle line:90% It's really very handy. 01:12:08.280 --> 01:12:10.320 align:middle line:84% And then, you run it through the pipeline again, 01:12:10.320 --> 01:12:11.920 align:middle line:90% and it says tiger cat. 01:12:11.920 --> 01:12:15.440 align:middle line:84% 94% probability, according to ResNet. 01:12:15.440 --> 01:12:17.020 align:middle line:90% So yeah, that's how you do it. 01:12:17.020 --> 01:12:18.600 align:middle line:84% Now, let's actually try a more interesting example 01:12:18.600 --> 01:12:21.183 align:middle line:84% where you want to detect all the objects in the picture, which 01:12:21.183 --> 01:12:23.220 align:middle line:84% we didn't talk about in class, object detection. 01:12:23.220 --> 01:12:27.400 align:middle line:84% So just create an object detection pipeline. 01:12:27.400 --> 01:12:28.880 align:middle line:90% Same thing as before. 01:12:28.880 --> 01:12:30.500 align:middle line:84% When you actually run this command, 01:12:30.500 --> 01:12:32.680 align:middle line:84% an astonishing some amount of complicated stuff 01:12:32.680 --> 01:12:34.700 align:middle line:90% is going on under the hood. 01:12:34.700 --> 01:12:37.560 align:middle line:84% And we are all the beneficiaries of that, so thank you. 01:12:37.560 --> 01:12:38.100 align:middle line:90% So yeah. 01:12:38.100 --> 01:12:42.307 align:middle line:84% So we have this here, and then we run it through the pipeline. 01:12:42.307 --> 01:12:44.390 align:middle line:84% It's looking at all the possible things that might 01:12:44.390 --> 01:12:45.515 align:middle line:90% be sitting in the pipeline. 01:12:45.515 --> 01:12:49.950 align:middle line:84% The results are hard to read, so let's actually visualize them. 01:12:49.950 --> 01:12:51.750 align:middle line:84% And I got some nice code from this site, 01:12:51.750 --> 01:12:52.930 align:middle line:90% for how to visualize them. 01:12:52.930 --> 01:12:54.830 align:middle line:90% Let's just reuse it. 01:12:54.830 --> 01:12:55.530 align:middle line:90% So yeah. 01:12:55.530 --> 01:12:59.410 align:middle line:84% So if you plot the results, look at that. 01:12:59.410 --> 01:13:04.150 align:middle line:90% 01:13:04.150 --> 01:13:07.830 align:middle line:84% So it has picked up the cat, 100% probability, I guess. 01:13:07.830 --> 01:13:12.570 align:middle line:84% The remote, the couch, the other remote, and then the cat. 01:13:12.570 --> 01:13:13.630 align:middle line:90% Pretty good, right? 01:13:13.630 --> 01:13:16.170 align:middle line:90% Off the shelf, ready to go. 01:13:16.170 --> 01:13:17.790 align:middle line:90% No heavy lifting required. 01:13:17.790 --> 01:13:19.550 align:middle line:84% Now, in this case, we are actually 01:13:19.550 --> 01:13:21.350 align:middle line:84% putting these boxes, called bounding boxes, 01:13:21.350 --> 01:13:22.245 align:middle line:90% around each picture. 01:13:22.245 --> 01:13:24.370 align:middle line:84% But what if you actually don't want a bounding box? 01:13:24.370 --> 01:13:26.910 align:middle line:84% What if you want to actually find the exact contour 01:13:26.910 --> 01:13:28.850 align:middle line:90% of that cat or the remote? 01:13:28.850 --> 01:13:29.690 align:middle line:90% No problem. 01:13:29.690 --> 01:13:31.910 align:middle line:84% We do something called image segmentation. 01:13:31.910 --> 01:13:36.510 align:middle line:84% So let's do an image segmentation pipeline. 01:13:36.510 --> 01:13:37.320 align:middle line:90% And run it through. 01:13:37.320 --> 01:13:42.900 align:middle line:90% 01:13:42.900 --> 01:13:45.780 align:middle line:90% It takes some time. 01:13:45.780 --> 01:13:46.660 align:middle line:90% All right. 01:13:46.660 --> 01:13:47.160 align:middle line:90% All right. 01:13:47.160 --> 01:13:49.080 align:middle line:90% Let's visualize it. 01:13:49.080 --> 01:13:51.840 align:middle line:84% So each object it finds, it gives you a mask. 01:13:51.840 --> 01:13:53.940 align:middle line:84% It basically tells you for each object 01:13:53.940 --> 01:13:56.260 align:middle line:84% what object it is, and then which 01:13:56.260 --> 01:13:59.640 align:middle line:84% pixels are on for that object and off for everything else. 01:13:59.640 --> 01:14:00.280 align:middle line:90% It's a mask. 01:14:00.280 --> 01:14:01.680 align:middle line:90% It tells you where it stands. 01:14:01.680 --> 01:14:05.020 align:middle line:84% And you can see here, it has the first object that's found, 01:14:05.020 --> 01:14:06.160 align:middle line:90% is this thing here. 01:14:06.160 --> 01:14:08.300 align:middle line:90% And it's perfectly delineated. 01:14:08.300 --> 01:14:09.420 align:middle line:90% It's pretty amazing. 01:14:09.420 --> 01:14:11.820 align:middle line:84% So we can overlay this on the original image 01:14:11.820 --> 01:14:14.770 align:middle line:90% and see that it's found that. 01:14:14.770 --> 01:14:16.755 align:middle line:90% And let's look at other objects. 01:14:16.755 --> 01:14:17.880 align:middle line:90% Oh, it is found the remote. 01:14:17.880 --> 01:14:20.460 align:middle line:90% That's the second object. 01:14:20.460 --> 01:14:21.970 align:middle line:90% And the third remote. 01:14:21.970 --> 01:14:24.580 align:middle line:90% 01:14:24.580 --> 01:14:26.160 align:middle line:90% And the fourth. 01:14:26.160 --> 01:14:27.988 align:middle line:84% You think any other objects are remaining? 01:14:27.988 --> 01:14:28.780 align:middle line:90% STUDENT: The couch. 01:14:28.780 --> 01:14:30.780 align:middle line:90% PROFESSOR: Couch, good. 01:14:30.780 --> 01:14:33.620 align:middle line:90% All right, let's find the couch. 01:14:33.620 --> 01:14:36.240 align:middle line:84% And look, the couch is pretty good, 01:14:36.240 --> 01:14:39.650 align:middle line:84% except that the middle part has gotten confused. 01:14:39.650 --> 01:14:40.150 align:middle line:90% All right. 01:14:40.150 --> 01:14:41.567 align:middle line:84% But it's still pretty good, right? 01:14:41.567 --> 01:14:44.090 align:middle line:90% So, yeah. 01:14:44.090 --> 01:14:46.050 align:middle line:84% So Hugging Face is all these things, 01:14:46.050 --> 01:14:48.810 align:middle line:84% and so you should definitely check it out if you're not 01:14:48.810 --> 01:14:50.190 align:middle line:90% already very familiar with it. 01:14:50.190 --> 01:14:51.710 align:middle line:90% So we have one minute left. 01:14:51.710 --> 01:14:52.470 align:middle line:90% Any questions? 01:14:52.470 --> 01:14:58.170 align:middle line:90% 01:14:58.170 --> 01:14:59.050 align:middle line:90% No questions? 01:14:59.050 --> 01:14:59.550 align:middle line:90% OK. 01:14:59.550 --> 01:15:00.270 align:middle line:90% All right, folks. 01:15:00.270 --> 01:15:01.145 align:middle line:90% See you on Wednesday. 01:15:01.145 --> 01:15:02.540 align:middle line:90% Thanks. 01:15:02.540 --> 01:15:38.000 align:middle line:90%