WEBVTT 00:00:00.000 --> 00:00:02.982 align:middle line:84% [SQUEAKING] [RUSTLING] [CLICKING] 00:00:02.982 --> 00:00:16.920 align:middle line:90% 00:00:16.920 --> 00:00:20.000 align:middle line:84% RAMA RAMAKRISHNAN: OK, so we'll continue with transformers 00:00:20.000 --> 00:00:21.720 align:middle line:90% today, part two. 00:00:21.720 --> 00:00:23.425 align:middle line:84% We are going to do the second pass. 00:00:23.425 --> 00:00:25.800 align:middle line:84% This is going to be a deeper pass through the transformer 00:00:25.800 --> 00:00:26.960 align:middle line:90% stack. 00:00:26.960 --> 00:00:29.360 align:middle line:84% And I think maybe the next 30 minutes, 00:00:29.360 --> 00:00:31.400 align:middle line:84% it's potentially the most demanding 30 minutes 00:00:31.400 --> 00:00:33.640 align:middle line:90% of the entire course, OK? 00:00:33.640 --> 00:00:36.560 align:middle line:84% With that motivational speech, let's get going. 00:00:36.560 --> 00:00:38.493 align:middle line:90% So a quick review. 00:00:38.493 --> 00:00:39.660 align:middle line:90% Why do we want transformers? 00:00:39.660 --> 00:00:43.880 align:middle line:84% Because we want an architecture that can generate output 00:00:43.880 --> 00:00:47.340 align:middle line:84% that has the same length as the input, same length. 00:00:47.340 --> 00:00:48.472 align:middle line:90% There it is. 00:00:48.472 --> 00:00:50.680 align:middle line:84% Number two, we want to take the context into account, 00:00:50.680 --> 00:00:52.472 align:middle line:84% and we want to take the order into account. 00:00:52.472 --> 00:00:55.080 align:middle line:84% And as you saw last time, the transformer architecture 00:00:55.080 --> 00:00:57.080 align:middle line:84% delivers on those three requirements. 00:00:57.080 --> 00:00:59.060 align:middle line:90% And so just a quick review. 00:00:59.060 --> 00:01:01.660 align:middle line:84% If you have a phrase like, "the train left the station," 00:01:01.660 --> 00:01:04.920 align:middle line:84% we have all these little arrows which stand for the standalone 00:01:04.920 --> 00:01:07.360 align:middle line:90% or uncontextual embeddings. 00:01:07.360 --> 00:01:09.120 align:middle line:84% And then-- sometimes this works, so I'm 00:01:09.120 --> 00:01:10.760 align:middle line:84% going to put it close to me here. 00:01:10.760 --> 00:01:14.440 align:middle line:90% 00:01:14.440 --> 00:01:17.800 align:middle line:84% So if you're here, if we start with either standalone 00:01:17.800 --> 00:01:20.448 align:middle line:84% embeddings, i.e. uncontextual embeddings, which 00:01:20.448 --> 00:01:22.740 align:middle line:84% have been pre-trained or random, doesn't really matter, 00:01:22.740 --> 00:01:26.407 align:middle line:84% if you look at the Colab we did the other day, 00:01:26.407 --> 00:01:28.240 align:middle line:84% we actually just started with random weights 00:01:28.240 --> 00:01:32.240 align:middle line:84% for the embeddings, and then we add positional embeddings 00:01:32.240 --> 00:01:33.000 align:middle line:90% to them. 00:01:33.000 --> 00:01:38.060 align:middle line:84% And so each embedding, each word here, we take its standalone, 00:01:38.060 --> 00:01:40.240 align:middle line:84% we take its positional embedding, we literally just 00:01:40.240 --> 00:01:42.160 align:middle line:90% add them up element by element. 00:01:42.160 --> 00:01:43.960 align:middle line:84% Then we get a total embedding, and that's 00:01:43.960 --> 00:01:47.400 align:middle line:84% called the positional embedding of each word. 00:01:47.400 --> 00:01:50.520 align:middle line:84% And then that's what we have, positional input embeddings. 00:01:50.520 --> 00:01:53.900 align:middle line:84% So this whole thing goes into this transformer encoder stack, 00:01:53.900 --> 00:01:56.980 align:middle line:84% and what pops out the other end is contextual embeddings 00:01:56.980 --> 00:01:58.680 align:middle line:90% So that's the overall flow. 00:01:58.680 --> 00:02:03.480 align:middle line:84% Now we applied this, the transformer 00:02:03.480 --> 00:02:06.580 align:middle line:84% stack, to the word-to-slot classification problem, 00:02:06.580 --> 00:02:09.240 align:middle line:84% where we basically took every incoming natural language 00:02:09.240 --> 00:02:11.800 align:middle line:84% query that comes in, we calculate 00:02:11.800 --> 00:02:13.560 align:middle line:84% its positional embeddings, and then 00:02:13.560 --> 00:02:16.080 align:middle line:84% we run it through the transformer stack, 00:02:16.080 --> 00:02:17.980 align:middle line:84% and then we get contextual embeddings. 00:02:17.980 --> 00:02:21.302 align:middle line:84% And then at this point, since each word that comes out, 00:02:21.302 --> 00:02:22.760 align:middle line:84% each embedding that comes out needs 00:02:22.760 --> 00:02:25.520 align:middle line:84% to be classified into one of 125 possibilities, 00:02:25.520 --> 00:02:28.400 align:middle line:84% we run it through a ReLU, when we attach 00:02:28.400 --> 00:02:30.680 align:middle line:90% a softmax to each embedding. 00:02:30.680 --> 00:02:34.560 align:middle line:84% This is basically what we did last class. 00:02:34.560 --> 00:02:37.840 align:middle line:84% So this is the transformer encoder. 00:02:37.840 --> 00:02:42.180 align:middle line:84% Actually, any questions on this before I continue on? 00:02:42.180 --> 00:02:48.520 align:middle line:90% 00:02:48.520 --> 00:02:50.200 align:middle line:84% AUDIENCE: I was wondering why, when, how 00:02:50.200 --> 00:02:52.520 align:middle line:84% do you decide where to add more self-attention 00:02:52.520 --> 00:02:55.180 align:middle line:84% and where you add the transformer layers? 00:02:55.180 --> 00:02:59.620 align:middle line:84% You mentioned that ChatGPT has 96 of them. 00:02:59.620 --> 00:03:04.220 align:middle line:84% RAMA RAMAKRISHNAN: So GPT3 has 96 transformer blocks. 00:03:04.220 --> 00:03:05.680 align:middle line:90% Each one is a block. 00:03:05.680 --> 00:03:07.840 align:middle line:84% So I think the question goes to, do you 00:03:07.840 --> 00:03:10.560 align:middle line:84% add more attention heads within a single block 00:03:10.560 --> 00:03:12.200 align:middle line:90% or do you add lots of blocks? 00:03:12.200 --> 00:03:14.640 align:middle line:90% And both are good things to do. 00:03:14.640 --> 00:03:17.720 align:middle line:84% What increasing the number of attention heads in a block 00:03:17.720 --> 00:03:20.960 align:middle line:84% does for you, it allows you to pick up more patterns 00:03:20.960 --> 00:03:23.600 align:middle line:90% at that level of abstraction. 00:03:23.600 --> 00:03:27.200 align:middle line:84% But if you add more blocks, much like later convolutional filters 00:03:27.200 --> 00:03:30.360 align:middle line:84% can build on earlier convolutional filters, 00:03:30.360 --> 00:03:32.500 align:middle line:84% you're going up the levels of abstraction. 00:03:32.500 --> 00:03:34.407 align:middle line:84% So to go to vision, for instance, 00:03:34.407 --> 00:03:36.740 align:middle line:84% you have the notion of lines and so on in the beginning, 00:03:36.740 --> 00:03:39.800 align:middle line:84% and then you have a notion of edges, which are two lines. 00:03:39.800 --> 00:03:43.420 align:middle line:84% Then you have nose, eyes, face, and so on and so forth. 00:03:43.420 --> 00:03:45.060 align:middle line:90% So both are worth doing. 00:03:45.060 --> 00:03:46.680 align:middle line:84% So typically that's what you typically 00:03:46.680 --> 00:03:51.920 align:middle line:84% find, that people typically have maybe a dozen heads or five, 00:03:51.920 --> 00:03:53.260 align:middle line:90% six, a dozen heads. 00:03:53.260 --> 00:03:54.840 align:middle line:84% We'll see examples of how many heads 00:03:54.840 --> 00:03:56.760 align:middle line:84% in a couple of architectures later on today. 00:03:56.760 --> 00:04:00.678 align:middle line:84% And the more you go up, the more capable 00:04:00.678 --> 00:04:02.720 align:middle line:84% the model becomes as long as you have enough data 00:04:02.720 --> 00:04:04.480 align:middle line:90% to train it well. 00:04:04.480 --> 00:04:06.960 align:middle line:84% So the perennial question of, do we have enough data 00:04:06.960 --> 00:04:08.700 align:middle line:90% to train this large model-- 00:04:08.700 --> 00:04:10.418 align:middle line:84% because if you don't have enough data, 00:04:10.418 --> 00:04:12.460 align:middle line:84% we might run into overfitting problems and so on. 00:04:12.460 --> 00:04:15.700 align:middle line:90% That's always the tradeoff. 00:04:15.700 --> 00:04:17.880 align:middle line:84% So here, I just want to quickly switch to the Colab 00:04:17.880 --> 00:04:20.140 align:middle line:84% because we didn't have a chance to finish it. 00:04:20.140 --> 00:04:23.320 align:middle line:84% I'm not going to run it because it's going to take some time. 00:04:23.320 --> 00:04:25.390 align:middle line:90% So where we left off last time-- 00:04:25.390 --> 00:04:28.040 align:middle line:90% 00:04:28.040 --> 00:04:31.960 align:middle line:84% so here we basically took this architecture 00:04:31.960 --> 00:04:34.560 align:middle line:84% that we just saw on the slide, and then we essentially 00:04:34.560 --> 00:04:35.860 align:middle line:90% wrote it as a Keras model. 00:04:35.860 --> 00:04:37.340 align:middle line:84% And I went through this model in the last class, 00:04:37.340 --> 00:04:39.440 align:middle line:84% so I'm not going to go through it all over again. 00:04:39.440 --> 00:04:42.640 align:middle line:84% What we did not do last class was to actually run it. 00:04:42.640 --> 00:04:46.560 align:middle line:84% And so if you actually run it, you 00:04:46.560 --> 00:04:49.580 align:middle line:84% can just run it for 10 epochs, just like we normally do. 00:04:49.580 --> 00:04:51.888 align:middle line:84% Give it data, give it a bunch of epochs, 00:04:51.888 --> 00:04:53.180 align:middle line:90% choose a particular batch size. 00:04:53.180 --> 00:04:54.880 align:middle line:90% I just arbitrarily chose 64. 00:04:54.880 --> 00:04:56.320 align:middle line:84% You run it for 10 epochs, and then 00:04:56.320 --> 00:04:58.140 align:middle line:90% you evaluate it on the test set. 00:04:58.140 --> 00:05:01.680 align:middle line:84% You get a 99% accuracy on this problem. 00:05:01.680 --> 00:05:03.960 align:middle line:84% One transformer stack, that's it-- 00:05:03.960 --> 00:05:05.420 align:middle line:90% one block, rather, one block. 00:05:05.420 --> 00:05:06.240 align:middle line:90% That's it. 00:05:06.240 --> 00:05:09.600 align:middle line:84% And of course, here, there is a little trickiness going on here 00:05:09.600 --> 00:05:13.160 align:middle line:84% because a naive model can literally say every word that 00:05:13.160 --> 00:05:17.040 align:middle line:84% comes in is "other," O. And since the Os are the majority 00:05:17.040 --> 00:05:19.397 align:middle line:84% of the words, it's not going to do badly. 00:05:19.397 --> 00:05:21.480 align:middle line:84% It's like having a classification problem in which 00:05:21.480 --> 00:05:23.480 align:middle line:90% one class is very predominant. 00:05:23.480 --> 00:05:25.310 align:middle line:84% So the naive way to actually do well 00:05:25.310 --> 00:05:27.560 align:middle line:84% is to just say every time something comes in, oh, it's 00:05:27.560 --> 00:05:28.982 align:middle line:90% that majority class. 00:05:28.982 --> 00:05:29.940 align:middle line:90% The same thing happens. 00:05:29.940 --> 00:05:31.760 align:middle line:84% But if you don't adjust for that, 00:05:31.760 --> 00:05:34.680 align:middle line:84% it turns out that the accuracy on the non-O slots, 00:05:34.680 --> 00:05:36.180 align:middle line:84% which is really what you care about, 00:05:36.180 --> 00:05:40.400 align:middle line:84% is actually 93%, which is actually pretty good. 00:05:40.400 --> 00:05:44.172 align:middle line:84% And then I had some examples of lots of fun queries you can do, 00:05:44.172 --> 00:05:46.880 align:middle line:84% including queries where I try to break stuff like cheapest flight 00:05:46.880 --> 00:05:50.080 align:middle line:84% to fly from MIT to Mars and see what happens, things like that. 00:05:50.080 --> 00:05:52.810 align:middle line:90% So have fun with it. 00:05:52.810 --> 00:05:54.230 align:middle line:90% All right, back to PowerPoint. 00:05:54.230 --> 00:05:59.450 align:middle line:90% 00:05:59.450 --> 00:06:00.550 align:middle line:90% So this is what we had. 00:06:00.550 --> 00:06:03.370 align:middle line:84% Now what we're going to do in today's class, 00:06:03.370 --> 00:06:06.970 align:middle line:84% we are actually going to take the encoder we built last time 00:06:06.970 --> 00:06:10.195 align:middle line:84% and introduce three new complications into it. 00:06:10.195 --> 00:06:12.570 align:middle line:84% And when we finish introducing these three complications, 00:06:12.570 --> 00:06:15.210 align:middle line:84% we will actually have the actual transformer that 00:06:15.210 --> 00:06:19.770 align:middle line:90% was invented in the 2017 paper. 00:06:19.770 --> 00:06:21.750 align:middle line:84% The first tweak is the hardest tweak, 00:06:21.750 --> 00:06:24.770 align:middle line:84% so we'll slowly work our way to it. 00:06:24.770 --> 00:06:27.350 align:middle line:84% So the thing to remember is, let's review self-attention. 00:06:27.350 --> 00:06:28.590 align:middle line:90% What is self-attention? 00:06:28.590 --> 00:06:31.330 align:middle line:84% You have a bunch of words, and we further said that for any 00:06:31.330 --> 00:06:33.970 align:middle line:84% particular word, like "station," we want to take its positional 00:06:33.970 --> 00:06:36.070 align:middle line:84% embedding and then make it contextual. 00:06:36.070 --> 00:06:39.330 align:middle line:84% And the way we do that is by taking each word's embedding 00:06:39.330 --> 00:06:41.530 align:middle line:84% and then calculating these dot products 00:06:41.530 --> 00:06:42.995 align:middle line:90% between all the other words. 00:06:42.995 --> 00:06:44.370 align:middle line:84% And then since these dot products 00:06:44.370 --> 00:06:48.130 align:middle line:84% can be positive or negative, we want to make them all positive 00:06:48.130 --> 00:06:50.850 align:middle line:84% and normalize them so that they nicely add up to 1. 00:06:50.850 --> 00:06:53.090 align:middle line:84% So we then exponentiate them and then 00:06:53.090 --> 00:06:57.170 align:middle line:84% divide by the total, which is basically softmax. 00:06:57.170 --> 00:07:00.290 align:middle line:84% And when you do that, you have nice fractions that add up to 1. 00:07:00.290 --> 00:07:03.130 align:middle line:84% And then we said, well, the contextual embedding for W6 00:07:03.130 --> 00:07:06.810 align:middle line:84% is just all these weights, S1, S2, 00:07:06.810 --> 00:07:10.150 align:middle line:84% all the way to S6, multiplied by the original Ws, 00:07:10.150 --> 00:07:12.770 align:middle line:84% and then you get the contextual embedding for W6. 00:07:12.770 --> 00:07:15.570 align:middle line:84% So this is the basic logic we covered last time. 00:07:15.570 --> 00:07:20.450 align:middle line:84% Now it is obviously the case that we explained it only 00:07:20.450 --> 00:07:23.215 align:middle line:84% for one word, but we have to do the same exact operation 00:07:23.215 --> 00:07:24.590 align:middle line:84% for every one of the other words, 00:07:24.590 --> 00:07:29.090 align:middle line:84% too, so that we could calculate W5 hat, W4 hat, W3 hat, and so 00:07:29.090 --> 00:07:30.280 align:middle line:90% on and so forth. 00:07:30.280 --> 00:07:32.930 align:middle line:84% So there's a lot of computations that are going on, 00:07:32.930 --> 00:07:35.312 align:middle line:84% and they all look kind of similar, 00:07:35.312 --> 00:07:37.270 align:middle line:84% where you've got to do a bunch of dot products, 00:07:37.270 --> 00:07:40.170 align:middle line:84% you've got to do some softmaxing on it and stuff like that. 00:07:40.170 --> 00:07:41.930 align:middle line:84% So the natural question is, is there 00:07:41.930 --> 00:07:45.210 align:middle line:84% a way to organize it very efficiently? 00:07:45.210 --> 00:07:46.550 align:middle line:90% And the short answer is yes. 00:07:46.550 --> 00:07:47.967 align:middle line:84% In fact, if you could not do that, 00:07:47.967 --> 00:07:50.250 align:middle line:84% there wouldn't be any transformer revolution 00:07:50.250 --> 00:07:52.330 align:middle line:84% because it is that ability to package it up 00:07:52.330 --> 00:07:54.810 align:middle line:84% into a very interesting and efficient operation that 00:07:54.810 --> 00:07:58.830 align:middle line:84% allows you to put the whole thing on GPUs. 00:07:58.830 --> 00:08:03.050 align:middle line:84% So now I'm going to switch to iPad and give you 00:08:03.050 --> 00:08:06.810 align:middle line:84% some iPad scribblings of mine, which were concocted last night 00:08:06.810 --> 00:08:09.770 align:middle line:84% because I was very unhappy with the slides that follow. 00:08:09.770 --> 00:08:12.910 align:middle line:90% So if you're going to do iPad. 00:08:12.910 --> 00:08:16.070 align:middle line:84% So if it works, you folks are lucky. 00:08:16.070 --> 00:08:18.320 align:middle line:84% If it doesn't work, last year's HODL class is luckier. 00:08:18.320 --> 00:08:21.330 align:middle line:90% 00:08:21.330 --> 00:08:22.950 align:middle line:90% So let's shift to that. 00:08:22.950 --> 00:08:25.490 align:middle line:90% 00:08:25.490 --> 00:08:26.840 align:middle line:90% So we're going to go here. 00:08:26.840 --> 00:08:31.210 align:middle line:90% 00:08:31.210 --> 00:08:33.099 align:middle line:84% So let's assume we have a simple thing like-- 00:08:33.099 --> 00:08:38.708 align:middle line:90% 00:08:38.708 --> 00:08:40.250 align:middle line:84% instead of, "Train left the station," 00:08:40.250 --> 00:08:42.667 align:middle line:84% which is a long sentence, let's just say you have a simple 00:08:42.667 --> 00:08:45.210 align:middle line:90% sentence like, "I love HODL." 00:08:45.210 --> 00:08:46.870 align:middle line:84% So "I love HODL" is what you have, 00:08:46.870 --> 00:08:50.530 align:middle line:84% and then you have these standalone embeddings, W1, W2, 00:08:50.530 --> 00:08:52.130 align:middle line:90% W3. 00:08:52.130 --> 00:08:54.930 align:middle line:84% So it comes into the self-attention layer. 00:08:54.930 --> 00:08:57.690 align:middle line:84% And let's assume that these W1s, Ws-2, W3, 00:08:57.690 --> 00:09:00.345 align:middle line:84% they are already positionally encoded. 00:09:00.345 --> 00:09:01.970 align:middle line:84% We have already added up the positional 00:09:01.970 --> 00:09:03.010 align:middle line:90% encoding all that stuff. 00:09:03.010 --> 00:09:03.990 align:middle line:90% It's all behind us. 00:09:03.990 --> 00:09:06.010 align:middle line:84% That all happens outside the transformer. 00:09:06.010 --> 00:09:08.050 align:middle line:90% So you get it here. 00:09:08.050 --> 00:09:13.670 align:middle line:84% Now what you do is you actually make three copies of this thing. 00:09:13.670 --> 00:09:17.170 align:middle line:84% And let's call this whole thing as just x. 00:09:17.170 --> 00:09:18.630 align:middle line:90% I'm just giving it the name x. 00:09:18.630 --> 00:09:21.650 align:middle line:84% It's a matrix of these three vectors. 00:09:21.650 --> 00:09:23.750 align:middle line:84% And so the first copy goes up here. 00:09:23.750 --> 00:09:25.110 align:middle line:90% The second copy goes straight. 00:09:25.110 --> 00:09:26.430 align:middle line:90% The third copy goes down. 00:09:26.430 --> 00:09:28.970 align:middle line:84% And don't worry about the third copy just yet. 00:09:28.970 --> 00:09:31.530 align:middle line:84% So if you look at the first two copies, 00:09:31.530 --> 00:09:35.370 align:middle line:84% here is the key thing to focus on, this whole thing here. 00:09:35.370 --> 00:09:36.850 align:middle line:84% Remember that we want to calculate 00:09:36.850 --> 00:09:39.193 align:middle line:84% dot products between all these vectors, 00:09:39.193 --> 00:09:40.610 align:middle line:84% and basically we want to calculate 00:09:40.610 --> 00:09:44.835 align:middle line:84% the dot product of every pair of vectors, every pair of words. 00:09:44.835 --> 00:09:46.210 align:middle line:84% The whole point of self-attention 00:09:46.210 --> 00:09:47.960 align:middle line:84% is that every pair of words, we figure out 00:09:47.960 --> 00:09:49.388 align:middle line:84% how attracted or related they are, 00:09:49.388 --> 00:09:50.930 align:middle line:84% which means that we have to calculate 00:09:50.930 --> 00:09:52.810 align:middle line:90% all pairs of dot products. 00:09:52.810 --> 00:09:55.890 align:middle line:84% And so what you do is you take this vector right there, 00:09:55.890 --> 00:10:00.230 align:middle line:84% W1, W, W3, you take this other copy that went up, 00:10:00.230 --> 00:10:02.513 align:middle line:90% and then you transpose it. 00:10:02.513 --> 00:10:04.930 align:middle line:84% So when you transpose it, it all becomes nice and vertical 00:10:04.930 --> 00:10:07.210 align:middle line:90% like that. 00:10:07.210 --> 00:10:08.830 align:middle line:84% All the vectors came in like this. 00:10:08.830 --> 00:10:10.690 align:middle line:84% When you transfer it, it becomes vertical. 00:10:10.690 --> 00:10:13.730 align:middle line:84% And now what you do is you take each one. 00:10:13.730 --> 00:10:16.930 align:middle line:84% You take W1 and then you multiply it by W1 here. 00:10:16.930 --> 00:10:21.152 align:middle line:90% You take W1, W2, W1, W3. 00:10:21.152 --> 00:10:23.110 align:middle line:84% You calculate all those dot products like that. 00:10:23.110 --> 00:10:25.810 align:middle line:84% And when you do that, you have these nice cells 00:10:25.810 --> 00:10:29.250 align:middle line:84% where every pair of words, the dot products 00:10:29.250 --> 00:10:31.510 align:middle line:84% have been calculated in this grid. 00:10:31.510 --> 00:10:33.490 align:middle line:90% And the key thing to see here-- 00:10:33.490 --> 00:10:35.690 align:middle line:84% and folks with a matrix algebra background 00:10:35.690 --> 00:10:37.370 align:middle line:90% will see this immediately-- 00:10:37.370 --> 00:10:39.450 align:middle line:84% all we are doing is we are taking this x, which 00:10:39.450 --> 00:10:42.970 align:middle line:84% is the matrix that came in, and then 00:10:42.970 --> 00:10:46.250 align:middle line:84% X transpose, which is the matrix that we sent up and then brought 00:10:46.250 --> 00:10:47.110 align:middle line:90% back down. 00:10:47.110 --> 00:10:49.050 align:middle line:84% We are basically doing a matrix multiplication 00:10:49.050 --> 00:10:50.310 align:middle line:90% of x times x transpose. 00:10:50.310 --> 00:10:51.730 align:middle line:90% That's all we are doing. 00:10:51.730 --> 00:10:57.330 align:middle line:84% And when we do that, we're getting this nice grid 00:10:57.330 --> 00:10:59.650 align:middle line:84% in which every pair of words, their dot products 00:10:59.650 --> 00:11:03.090 align:middle line:84% have been calculated for you with one matrix multiplication. 00:11:03.090 --> 00:11:04.750 align:middle line:90% Boom, done. 00:11:04.750 --> 00:11:07.770 align:middle line:84% So if you have three words, there are 9 multiplications. 00:11:07.770 --> 00:11:11.370 align:middle line:84% So if you have a million words, that's 00:11:11.370 --> 00:11:15.330 align:middle line:84% a lot of multiplications, one trillion multiplications, 00:11:15.330 --> 00:11:17.330 align:middle line:90% on the order of one trillion. 00:11:17.330 --> 00:11:21.010 align:middle line:84% And the reason I say order of is because W1 times W3 00:11:21.010 --> 00:11:24.290 align:middle line:84% is the same as W3 times W1, so there is some duplication here. 00:11:24.290 --> 00:11:28.370 align:middle line:84% So you get this grid in one shot, in one multiplication. 00:11:28.370 --> 00:11:30.215 align:middle line:84% And then because each of these numbers 00:11:30.215 --> 00:11:32.590 align:middle line:84% is just a dot product, which can be negative or positive, 00:11:32.590 --> 00:11:34.572 align:middle line:90% we need to softmax it. 00:11:34.572 --> 00:11:36.530 align:middle line:84% And so what we do is we take all these numbers, 00:11:36.530 --> 00:11:40.190 align:middle line:84% and we put it into a softmax function where for each row, 00:11:40.190 --> 00:11:41.310 align:middle line:90% it calculates the softmax. 00:11:41.310 --> 00:11:42.435 align:middle line:90% And what do I mean by that? 00:11:42.435 --> 00:11:45.732 align:middle line:84% It takes each number here, does e raised to the top, e 00:11:45.732 --> 00:11:48.190 align:middle line:84% raised to the number, it does it for each of these numbers, 00:11:48.190 --> 00:11:51.330 align:middle line:84% and then divides by the sum of those numbers for each row. 00:11:51.330 --> 00:11:53.430 align:middle line:90% And when you do that-- 00:11:53.430 --> 00:11:56.610 align:middle line:84% you can think of this operation as softmax applied to x times 00:11:56.610 --> 00:12:01.010 align:middle line:84% x transpose-- you get this nice little table of numbers. 00:12:01.010 --> 00:12:04.890 align:middle line:84% This table of numbers basically says that for the first word, 00:12:04.890 --> 00:12:09.490 align:middle line:84% W1, for the first word, take 0.1 of the first one, 0.7 00:12:09.490 --> 00:12:13.450 align:middle line:84% of the second, 0.2 of the third, and add them up. 00:12:13.450 --> 00:12:14.890 align:middle line:90% We do a weighted average. 00:12:14.890 --> 00:12:16.930 align:middle line:90% So we have this table here. 00:12:16.930 --> 00:12:21.330 align:middle line:84% We have now the third copy, shows up here. 00:12:21.330 --> 00:12:22.290 align:middle line:90% It's right there. 00:12:22.290 --> 00:12:25.650 align:middle line:84% So we do this times that, which is just a matrix multiplication 00:12:25.650 --> 00:12:26.370 align:middle line:90% again. 00:12:26.370 --> 00:12:30.090 align:middle line:84% And when we do that, we get the final contextual embeddings. 00:12:30.090 --> 00:12:36.730 align:middle line:84% So this, for example, is just 0.1 times W1, 0.2 times W2-- 00:12:36.730 --> 00:12:40.430 align:middle line:84% sorry, 0.7 times W2 and then 0.2 times W3 right there. 00:12:40.430 --> 00:12:43.130 align:middle line:84% And you can see the same logic here as well. 00:12:43.130 --> 00:12:44.910 align:middle line:90% And you can read it later on. 00:12:44.910 --> 00:12:47.050 align:middle line:84% I will post this thing to make sure you understand 00:12:47.050 --> 00:12:48.530 align:middle line:90% exactly how it flowed. 00:12:48.530 --> 00:12:51.250 align:middle line:84% But the larger point I want you to focus on 00:12:51.250 --> 00:12:54.810 align:middle line:84% is that the entire self-attention operation we just 00:12:54.810 --> 00:12:58.330 align:middle line:84% looked at here basically is this beautifully little 00:12:58.330 --> 00:12:59.720 align:middle line:90% compact matrix formula. 00:12:59.720 --> 00:13:02.250 align:middle line:90% 00:13:02.250 --> 00:13:04.987 align:middle line:84% x comes in, you do x transpose, you do a matrix multiplication, 00:13:04.987 --> 00:13:07.570 align:middle line:84% you do a softmax on top of it, and then multiply it by x again 00:13:07.570 --> 00:13:10.250 align:middle line:90% and boom, you're done. 00:13:10.250 --> 00:13:14.170 align:middle line:84% So that is the magic of taking the transformer stack 00:13:14.170 --> 00:13:17.130 align:middle line:84% and representing it using matrix operations because then [POOF], 00:13:17.130 --> 00:13:19.970 align:middle line:90% lightning fast on GPUs. 00:13:19.970 --> 00:13:20.930 align:middle line:90% OK? 00:13:20.930 --> 00:13:24.610 align:middle line:90% All right, that was the warm-up. 00:13:24.610 --> 00:13:27.050 align:middle line:90% Now let's crank it up a notch. 00:13:27.050 --> 00:13:31.010 align:middle line:84% So recall that in the last class, 00:13:31.010 --> 00:13:37.250 align:middle line:84% I talked about the fact the self-attention operation, 00:13:37.250 --> 00:13:39.610 align:middle line:84% the Ws are coming in, and we're doing all this stuff 00:13:39.610 --> 00:13:43.830 align:middle line:84% with the Ws, and then we're getting some W hats out, 00:13:43.830 --> 00:13:46.570 align:middle line:90% but there are no parameters. 00:13:46.570 --> 00:13:49.690 align:middle line:84% There IS nothing to be learned inside the transformer 00:13:49.690 --> 00:13:51.390 align:middle line:90% self-attention layer, right? 00:13:51.390 --> 00:13:52.397 align:middle line:90% There are no weights. 00:13:52.397 --> 00:13:53.230 align:middle line:90% There are no biases. 00:13:53.230 --> 00:13:55.450 align:middle line:90% There are no coefficients. 00:13:55.450 --> 00:13:58.930 align:middle line:84% Well, OK, what are we learning then? 00:13:58.930 --> 00:14:00.730 align:middle line:84% So what we now do is we are going 00:14:00.730 --> 00:14:03.250 align:middle line:84% to make the self-attention layer tunable. 00:14:03.250 --> 00:14:05.450 align:middle line:84% We are going to inject some weights into it 00:14:05.450 --> 00:14:08.210 align:middle line:84% so that when we train it on an actual system, 00:14:08.210 --> 00:14:10.250 align:middle line:84% the weights will keep changing to adapt itself 00:14:10.250 --> 00:14:12.210 align:middle line:84% to the particularities of whatever problem 00:14:12.210 --> 00:14:14.650 align:middle line:90% you're working on. 00:14:14.650 --> 00:14:23.090 align:middle line:84% So that takes us to the tunable self-attention layer, 00:14:23.090 --> 00:14:25.250 align:middle line:90% tunable self-attention layer. 00:14:25.250 --> 00:14:28.105 align:middle line:84% So this is the key thing to keep in mind. 00:14:28.105 --> 00:14:29.730 align:middle line:84% Any questions on this before I continue 00:14:29.730 --> 00:14:30.990 align:middle line:90% with the tunability thing? 00:14:30.990 --> 00:14:34.530 align:middle line:90% 00:14:34.530 --> 00:14:36.860 align:middle line:90% OK. 00:14:36.860 --> 00:14:39.670 align:middle line:84% Is this picture working out, by the way? 00:14:39.670 --> 00:14:40.170 align:middle line:90% OK. 00:14:40.170 --> 00:14:43.970 align:middle line:90% 00:14:43.970 --> 00:14:47.330 align:middle line:84% So what we now do is we have the same exact logic as before, 00:14:47.330 --> 00:14:50.170 align:middle line:84% where we have this thing that comes in. 00:14:50.170 --> 00:14:52.930 align:middle line:84% We have this input that comes in. 00:14:52.930 --> 00:14:56.570 align:middle line:84% We call it x again, this matrix of embeddings. 00:14:56.570 --> 00:14:59.775 align:middle line:84% And then before we just send three copies, 00:14:59.775 --> 00:15:01.650 align:middle line:84% instead of doing that, what we're going to do 00:15:01.650 --> 00:15:04.090 align:middle line:84% is we'll take each copy, x, and then 00:15:04.090 --> 00:15:07.690 align:middle line:84% we will actually multiply it by a matrix. 00:15:07.690 --> 00:15:11.462 align:middle line:84% This matrix is called the key matrix. 00:15:11.462 --> 00:15:15.130 align:middle line:84% And this matrix, this matrix of numbers or weights 00:15:15.130 --> 00:15:18.353 align:middle line:84% that will be learned by backprop. 00:15:18.353 --> 00:15:19.770 align:middle line:84% So basically what we are saying is 00:15:19.770 --> 00:15:22.730 align:middle line:84% that when this thing comes in, let's 00:15:22.730 --> 00:15:24.810 align:middle line:84% see if there is a way to transform 00:15:24.810 --> 00:15:28.130 align:middle line:84% this x into some other set of embeddings which 00:15:28.130 --> 00:15:30.017 align:middle line:90% may be useful for your task. 00:15:30.017 --> 00:15:31.850 align:middle line:84% We don't know if they're going to be useful, 00:15:31.850 --> 00:15:34.130 align:middle line:84% but surely giving it a bit more ability 00:15:34.130 --> 00:15:36.170 align:middle line:84% to have weights which can be learned 00:15:36.170 --> 00:15:39.850 align:middle line:84% means that we are giving it more expressive power, more modeling 00:15:39.850 --> 00:15:40.557 align:middle line:90% capacity. 00:15:40.557 --> 00:15:42.890 align:middle line:84% And whether it actually uses the capacity will depend on 00:15:42.890 --> 00:15:45.410 align:middle line:84% how much data you have and how well you train it. 00:15:45.410 --> 00:15:48.370 align:middle line:84% And maybe if it's not useful, it won't use it. 00:15:48.370 --> 00:15:50.930 align:middle line:84% What I mean is if transforming x actually doesn't really 00:15:50.930 --> 00:15:54.630 align:middle line:84% help at all, then this matrix A is going to be what? 00:15:54.630 --> 00:15:55.610 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:15:55.610 --> 00:15:56.730 align:middle line:84% RAMA RAMAKRISHNAN: It's going to be the identity 00:15:56.730 --> 00:15:58.890 align:middle line:84% matrix because you take basically 1 00:15:58.890 --> 00:16:01.170 align:middle line:84% and multiply it by x, you'll get x again. 00:16:01.170 --> 00:16:02.830 align:middle line:84% So the worst case, maybe it just says, 00:16:02.830 --> 00:16:04.970 align:middle line:84% I have nothing to learn here, but maybe 00:16:04.970 --> 00:16:06.610 align:middle line:84% there is something you can learn. 00:16:06.610 --> 00:16:08.270 align:middle line:90% So that's what we do. 00:16:08.270 --> 00:16:11.010 align:middle line:84% So we multiplied by this matrix AK. 00:16:11.010 --> 00:16:13.610 align:middle line:84% And then we come up with the same-- 00:16:13.610 --> 00:16:15.310 align:middle line:84% some embeddings, transformed embeddings, 00:16:15.310 --> 00:16:18.990 align:middle line:90% and we call these things K. 00:16:18.990 --> 00:16:22.770 align:middle line:84% Now this KQV, as you will see, has its origins 00:16:22.770 --> 00:16:25.330 align:middle line:84% in the field of information retrieval, 00:16:25.330 --> 00:16:28.050 align:middle line:84% but I personally find that interpretation is not 00:16:28.050 --> 00:16:30.132 align:middle line:84% super helpful because transformers 00:16:30.132 --> 00:16:32.340 align:middle line:84% are used for lots of applications outside information 00:16:32.340 --> 00:16:33.780 align:middle line:84% retrieval, so I'm not going to go 00:16:33.780 --> 00:16:35.508 align:middle line:84% with that kind of interpretation. 00:16:35.508 --> 00:16:37.300 align:middle line:84% I'm going to go with the interpretation of, 00:16:37.300 --> 00:16:39.770 align:middle line:84% let's make each of these things tunable. 00:16:39.770 --> 00:16:42.480 align:middle line:84% And tunability means we need to give it weights. 00:16:42.480 --> 00:16:43.820 align:middle line:90% So that's what we have here. 00:16:43.820 --> 00:16:46.147 align:middle line:90% Now the second copy-- 00:16:46.147 --> 00:16:47.480 align:middle line:90% we did this with the first copy. 00:16:47.480 --> 00:16:49.120 align:middle line:84% Well, let's do the same thing with the second copy. 00:16:49.120 --> 00:16:50.870 align:middle line:84% We'll take the second copy and multiply it 00:16:50.870 --> 00:16:53.220 align:middle line:90% by some other matrix called AQ. 00:16:53.220 --> 00:16:56.220 align:middle line:84% And when we are done with that, we get these embeddings, 00:16:56.220 --> 00:17:00.912 align:middle line:84% and we will call these embeddings as Q. 00:17:00.912 --> 00:17:04.540 align:middle line:84% Now just like before, we will take this thing here, 00:17:04.540 --> 00:17:08.380 align:middle line:84% and we will transpose it so it all becomes nice and vertical 00:17:08.380 --> 00:17:09.805 align:middle line:90% like that. 00:17:09.805 --> 00:17:11.680 align:middle line:84% And then we'll do exactly the same as before. 00:17:11.680 --> 00:17:13.638 align:middle line:84% We'll calculate all these pairwise dot products 00:17:13.638 --> 00:17:16.619 align:middle line:84% using one-shot one-matrix multiplication. 00:17:16.619 --> 00:17:19.740 align:middle line:84% And because we are calling this Q 00:17:19.740 --> 00:17:21.780 align:middle line:84% and we are calling this whole thing as K, 00:17:21.780 --> 00:17:23.850 align:middle line:84% this thing just becomes Q times KT. 00:17:23.850 --> 00:17:26.859 align:middle line:90% 00:17:26.859 --> 00:17:30.180 align:middle line:84% At the end of it, you come up with a grid of numbers 00:17:30.180 --> 00:17:32.240 align:middle line:90% just like before. 00:17:32.240 --> 00:17:34.240 align:middle line:84% And these numbers could be negative or positive, 00:17:34.240 --> 00:17:35.900 align:middle line:84% so we need to do the softmax on them 00:17:35.900 --> 00:17:37.942 align:middle line:84% to make sure they are well-behaved fractions that 00:17:37.942 --> 00:17:38.940 align:middle line:90% add up to 1. 00:17:38.940 --> 00:17:44.340 align:middle line:84% So we take this QKT business and then 00:17:44.340 --> 00:17:48.340 align:middle line:84% we put it through a softmax function for each row. 00:17:48.340 --> 00:17:52.740 align:middle line:84% And when we do that, we will get basically a table like the ones 00:17:52.740 --> 00:17:53.600 align:middle line:90% we saw before. 00:17:53.600 --> 00:17:55.308 align:middle line:84% By the way, the numbers here are the same 00:17:55.308 --> 00:17:57.700 align:middle line:84% just because I duplicated it because I'm lazy. 00:17:57.700 --> 00:17:59.140 align:middle line:84% In reality, given that it has gone 00:17:59.140 --> 00:18:00.557 align:middle line:84% through all these transformations, 00:18:00.557 --> 00:18:02.900 align:middle line:84% the numbers are not going to be the same. 00:18:02.900 --> 00:18:06.200 align:middle line:84% You have these numbers and then you take the final copy, 00:18:06.200 --> 00:18:08.437 align:middle line:90% which is X times AV. 00:18:08.437 --> 00:18:10.520 align:middle line:84% Each copy is getting multiplied by its own matrix. 00:18:10.520 --> 00:18:13.100 align:middle line:84% And this copy is being multiplied by AV. 00:18:13.100 --> 00:18:19.420 align:middle line:84% And let's call this XAV, which is here, as just V. 00:18:19.420 --> 00:18:23.570 align:middle line:84% And so what you have here is this softmax QKT times V is 00:18:23.570 --> 00:18:25.820 align:middle line:84% exactly the same kind of dot product as we saw before, 00:18:25.820 --> 00:18:27.460 align:middle line:90% matrix multiplication. 00:18:27.460 --> 00:18:29.340 align:middle line:84% So we have these contextual embeddings, 00:18:29.340 --> 00:18:32.680 align:middle line:84% and that's what's coming out of the transformer block. 00:18:32.680 --> 00:18:36.300 align:middle line:84% So now the whole thing we did here, the whole thing can 00:18:36.300 --> 00:18:40.500 align:middle line:84% be represented as softmax of QKT times V. 00:18:40.500 --> 00:18:44.180 align:middle line:90% So if we zoom in a bit-- 00:18:44.180 --> 00:18:44.800 align:middle line:90% come on. 00:18:44.800 --> 00:18:49.100 align:middle line:90% 00:18:49.100 --> 00:18:54.540 align:middle line:84% So X came in, three tracks went here. 00:18:54.540 --> 00:18:58.940 align:middle line:84% The first track X times AK, X times AQ, X times AV. 00:18:58.940 --> 00:19:00.940 align:middle line:84% And this thing is called K. This thing is called 00:19:00.940 --> 00:19:03.260 align:middle line:84% Q. This thing is called V. And then 00:19:03.260 --> 00:19:05.540 align:middle line:84% we do the same transpose as before. 00:19:05.540 --> 00:19:08.740 align:middle line:84% We do the dot product thing to calculate the pairwise dot 00:19:08.740 --> 00:19:11.500 align:middle line:84% products for everything, which is just QKT. 00:19:11.500 --> 00:19:12.840 align:middle line:90% We run it through a softmax. 00:19:12.840 --> 00:19:14.820 align:middle line:90% We get softmax of QKT. 00:19:14.820 --> 00:19:17.940 align:middle line:84% We multiply it by 1 to do the final weighting, and then boom, 00:19:17.940 --> 00:19:20.040 align:middle line:84% the output comes, and that's this function. 00:19:20.040 --> 00:19:22.020 align:middle line:90% That's it. 00:19:22.020 --> 00:19:24.540 align:middle line:84% So what we have done is we have introduced 00:19:24.540 --> 00:19:28.220 align:middle line:84% three matrices, learnable matrices, 00:19:28.220 --> 00:19:29.680 align:middle line:90% into the self-attention layer. 00:19:29.680 --> 00:19:34.140 align:middle line:90% 00:19:34.140 --> 00:19:35.640 align:middle line:84% Let me just stop there for a second. 00:19:35.640 --> 00:19:37.380 align:middle line:90% Questions? 00:19:37.380 --> 00:19:39.805 align:middle line:90% Yeah. 00:19:39.805 --> 00:19:41.180 align:middle line:84% AUDIENCE: Is there a relationship 00:19:41.180 --> 00:19:43.088 align:middle line:90% between AK, AQ, and AV? 00:19:43.088 --> 00:19:44.380 align:middle line:90% RAMA RAMAKRISHNAN: Independent. 00:19:44.380 --> 00:19:45.255 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:19:45.255 --> 00:19:47.460 align:middle line:84% RAMA RAMAKRISHNAN: Independent matrices. 00:19:47.460 --> 00:19:48.345 align:middle line:90% Yes? 00:19:48.345 --> 00:19:49.220 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:19:49.220 --> 00:19:50.580 align:middle line:84% RAMA RAMAKRISHNAN: Could you use the microphone, please? 00:19:50.580 --> 00:19:52.580 align:middle line:84% AUDIENCE: Here we have three sets of parameters, 00:19:52.580 --> 00:19:54.660 align:middle line:84% K, Q, and V. If there are, let's say, 00:19:54.660 --> 00:19:57.380 align:middle line:84% if there were 100 or the total length 00:19:57.380 --> 00:20:00.480 align:middle line:84% was, let's say, the number of totals, let's say 50, 00:20:00.480 --> 00:20:02.700 align:middle line:84% so you could have 50% of parameters 00:20:02.700 --> 00:20:04.740 align:middle line:90% you have to [INAUDIBLE]. 00:20:04.740 --> 00:20:07.460 align:middle line:84% RAMA RAMAKRISHNAN: So if the dimension 00:20:07.460 --> 00:20:09.620 align:middle line:84% is 50 long, what is coming in, the Ws 00:20:09.620 --> 00:20:13.820 align:middle line:84% are 50 long, then the key, what comes out of it, if you 00:20:13.820 --> 00:20:15.460 align:middle line:90% want it to be 50 as well-- 00:20:15.460 --> 00:20:24.900 align:middle line:84% so this matrix needs to be 50 times 50, 2,500. 00:20:24.900 --> 00:20:27.940 align:middle line:84% AUDIENCE: What are the three different things the three 00:20:27.940 --> 00:20:29.738 align:middle line:90% matrices are trying to learn? 00:20:29.738 --> 00:20:30.780 align:middle line:90% RAMA RAMAKRISHNAN: Sorry? 00:20:30.780 --> 00:20:32.580 align:middle line:84% AUDIENCE: What are the different things that the matrices are 00:20:32.580 --> 00:20:33.620 align:middle line:90% trying to learn? 00:20:33.620 --> 00:20:34.360 align:middle line:84% RAMA RAMAKRISHNAN: We don't know. 00:20:34.360 --> 00:20:36.840 align:middle line:84% All we are saying is that we have a self-attention layer, 00:20:36.840 --> 00:20:38.840 align:middle line:84% which can pay attention to every pair of words, 00:20:38.840 --> 00:20:41.340 align:middle line:84% but we need to give it some ways to transform 00:20:41.340 --> 00:20:45.192 align:middle line:84% what is coming in into potentially useful things. 00:20:45.192 --> 00:20:47.660 align:middle line:84% As to their actual usefulness, we'll 00:20:47.660 --> 00:20:49.792 align:middle line:84% have to figure out if it actually helps or not. 00:20:49.792 --> 00:20:51.500 align:middle line:84% And of course, as you know, the punchline 00:20:51.500 --> 00:20:53.000 align:middle line:84% is that yeah, it helps massively. 00:20:53.000 --> 00:20:53.960 align:middle line:90% That's why we do it. 00:20:53.960 --> 00:20:56.085 align:middle line:84% In general, what you will find in the deep learning 00:20:56.085 --> 00:20:57.620 align:middle line:84% literature is that whenever you want 00:20:57.620 --> 00:20:59.220 align:middle line:84% to increase the capacity, the modeling 00:20:59.220 --> 00:21:02.780 align:middle line:84% capacity of a particular model, you just take a small piece 00:21:02.780 --> 00:21:05.860 align:middle line:84% and inject a little matrix multiplication into it. 00:21:05.860 --> 00:21:08.080 align:middle line:84% You take a vector that's showing up in the middle, 00:21:08.080 --> 00:21:10.860 align:middle line:84% and then you make it run through a matrix to get another vector. 00:21:10.860 --> 00:21:13.560 align:middle line:84% And then further, after you run it through a matrix, 00:21:13.560 --> 00:21:15.480 align:middle line:84% you run it through a little ReLU as well. 00:21:15.480 --> 00:21:16.860 align:middle line:90% Even better. 00:21:16.860 --> 00:21:19.500 align:middle line:84% So that's how you inject modeling capacity 00:21:19.500 --> 00:21:22.140 align:middle line:84% into the middle of these networks, 00:21:22.140 --> 00:21:24.460 align:middle line:84% and that's what these people are doing here. 00:21:24.460 --> 00:21:26.740 align:middle line:90% Yeah? 00:21:26.740 --> 00:21:29.020 align:middle line:84% AUDIENCE: In the last step, you had the matrix 00:21:29.020 --> 00:21:30.837 align:middle line:84% V. So on the previous example, you 00:21:30.837 --> 00:21:33.420 align:middle line:84% had used the original matrix X. Could you just say [INAUDIBLE] 00:21:33.420 --> 00:21:35.000 align:middle line:90% why is it not using X? 00:21:35.000 --> 00:21:36.080 align:middle line:90% And what does that mean? 00:21:36.080 --> 00:21:37.788 align:middle line:84% RAMA RAMAKRISHNAN: So what you are saying 00:21:37.788 --> 00:21:40.280 align:middle line:84% is that in the initial version we had three copies, 00:21:40.280 --> 00:21:42.060 align:middle line:84% and we treated them all identical. 00:21:42.060 --> 00:21:43.762 align:middle line:84% Now we said, well, are there ways 00:21:43.762 --> 00:21:46.220 align:middle line:84% to transform each copy into some other representation which 00:21:46.220 --> 00:21:48.300 align:middle line:84% could be useful, so we may as well 00:21:48.300 --> 00:21:49.980 align:middle line:84% use three different matrices for it. 00:21:49.980 --> 00:21:51.382 align:middle line:90% Why stop with two? 00:21:51.382 --> 00:21:53.840 align:middle line:84% There are three opportunities to make them more expressive. 00:21:53.840 --> 00:21:56.360 align:middle line:90% We'll use all of them. 00:21:56.360 --> 00:21:56.860 align:middle line:90% Yeah? 00:21:56.860 --> 00:21:59.740 align:middle line:90% 00:21:59.740 --> 00:22:02.100 align:middle line:84% AUDIENCE: You mentioned that these are-- 00:22:02.100 --> 00:22:04.468 align:middle line:90% you're fine-tuning it. 00:22:04.468 --> 00:22:05.260 align:middle line:90% Is there any risk-- 00:22:05.260 --> 00:22:07.093 align:middle line:84% RAMA RAMAKRISHNAN: We're not fine-tuning it, 00:22:07.093 --> 00:22:08.980 align:middle line:84% just to be clear on the vocabulary here. 00:22:08.980 --> 00:22:11.520 align:middle line:84% So we have added more weights to make them tunable. 00:22:11.520 --> 00:22:15.952 align:middle line:84% What that means is that when we finally train this entire model, 00:22:15.952 --> 00:22:17.660 align:middle line:84% remember, all the weights are going to be 00:22:17.660 --> 00:22:19.320 align:middle line:90% updated using back propagation. 00:22:19.320 --> 00:22:21.860 align:middle line:84% In particular, these matrices will also 00:22:21.860 --> 00:22:23.862 align:middle line:84% get updated using back propagation. 00:22:23.862 --> 00:22:26.320 align:middle line:84% AUDIENCE: So there's no risk of over-- is there a risk of-- 00:22:26.320 --> 00:22:26.780 align:middle line:84% RAMA RAMAKRISHNAN: There is always 00:22:26.780 --> 00:22:28.740 align:middle line:84% a risk of overfitting when you add more parameters 00:22:28.740 --> 00:22:31.420 align:middle line:84% to a model, which means that you have to look at the validation 00:22:31.420 --> 00:22:34.060 align:middle line:90% set and all that good stuff. 00:22:34.060 --> 00:22:37.340 align:middle line:84% We are basically adding more parameters in a very interesting 00:22:37.340 --> 00:22:40.113 align:middle line:84% way because we want to add more capacity 00:22:40.113 --> 00:22:41.280 align:middle line:90% to the self-attention layer. 00:22:41.280 --> 00:22:42.822 align:middle line:84% We want to give it more of an ability 00:22:42.822 --> 00:22:44.380 align:middle line:90% to learn things from the data. 00:22:44.380 --> 00:22:45.920 align:middle line:84% Before it could not learn anything. 00:22:45.920 --> 00:22:48.020 align:middle line:90% It could only do dot products. 00:22:48.020 --> 00:22:50.765 align:middle line:84% So we want to solve that problem. 00:22:50.765 --> 00:22:52.140 align:middle line:84% All right, I'm going to continue, 00:22:52.140 --> 00:22:53.307 align:middle line:90% and we'll come back to this. 00:22:53.307 --> 00:22:58.820 align:middle line:90% 00:22:58.820 --> 00:23:01.100 align:middle line:84% Just for fun, I'm going to do this. 00:23:01.100 --> 00:23:04.620 align:middle line:84% The original paper is called, "Attention is All You Need." 00:23:04.620 --> 00:23:07.620 align:middle line:90% This is a transformer paper. 00:23:07.620 --> 00:23:11.343 align:middle line:84% You folks should read it at some point. 00:23:11.343 --> 00:23:12.760 align:middle line:84% I just want to show you something. 00:23:12.760 --> 00:23:19.980 align:middle line:90% 00:23:19.980 --> 00:23:21.340 align:middle line:90% You see that? 00:23:21.340 --> 00:23:24.740 align:middle line:84% So that is the famous transformer formula. 00:23:24.740 --> 00:23:28.380 align:middle line:84% And the only thing we ignored is this root 00:23:28.380 --> 00:23:31.220 align:middle line:90% of dk business under it. 00:23:31.220 --> 00:23:32.360 align:middle line:90% I wouldn't worry about it. 00:23:32.360 --> 00:23:35.640 align:middle line:84% The reason they have it is because these softmaxes, when 00:23:35.640 --> 00:23:38.140 align:middle line:84% you have lots of numbers and some numbers are really, really 00:23:38.140 --> 00:23:40.980 align:middle line:84% big, what's going to happen is that all the other numbers 00:23:40.980 --> 00:23:43.340 align:middle line:90% are going to get squashed to 0. 00:23:43.340 --> 00:23:46.240 align:middle line:84% And so to make sure the gradient flows properly, 00:23:46.240 --> 00:23:48.220 align:middle line:84% they just divide it by a particular number 00:23:48.220 --> 00:23:51.060 align:middle line:84% to make sure no number is too big. 00:23:51.060 --> 00:23:52.597 align:middle line:84% It's a small technical-- important, 00:23:52.597 --> 00:23:54.180 align:middle line:84% but a bit of a technical detail, which 00:23:54.180 --> 00:23:56.120 align:middle line:90% is why I ignored it in my iPad. 00:23:56.120 --> 00:23:58.220 align:middle line:84% But the rest of it, you can see this 00:23:58.220 --> 00:24:04.360 align:middle line:84% is exactly the formula we derived, QKT times V softmax. 00:24:04.360 --> 00:24:08.020 align:middle line:84% So this is the famous transformer formula. 00:24:08.020 --> 00:24:11.572 align:middle line:84% And congratulations, now you understand it. 00:24:11.572 --> 00:24:13.030 align:middle line:84% You seem less than fully convinced? 00:24:13.030 --> 00:24:17.180 align:middle line:90% [CHUCKLES] OK. 00:24:17.180 --> 00:24:19.060 align:middle line:90% Yes, iPad. 00:24:19.060 --> 00:24:21.780 align:middle line:84% Now I have a bunch of slides, which I had-- 00:24:21.780 --> 00:24:23.860 align:middle line:84% actually, I'll come back to this. 00:24:23.860 --> 00:24:26.603 align:middle line:84% I had a bunch of other slides-- this is from last year-- 00:24:26.603 --> 00:24:28.020 align:middle line:84% which actually explains what I did 00:24:28.020 --> 00:24:29.478 align:middle line:84% in the iPad in a very different way 00:24:29.478 --> 00:24:31.380 align:middle line:84% without using any matrices and so on. 00:24:31.380 --> 00:24:33.240 align:middle line:84% I was looking at it last evening, 00:24:33.240 --> 00:24:36.420 align:middle line:84% and I was getting very annoyed by these slides for some reason 00:24:36.420 --> 00:24:38.340 align:middle line:84% because I felt that it wasn't really conveying 00:24:38.340 --> 00:24:45.562 align:middle line:84% the ability of using matrix algebra to actually do this 00:24:45.562 --> 00:24:47.020 align:middle line:84% so efficiently and compactly, which 00:24:47.020 --> 00:24:50.320 align:middle line:84% is why I decided to hand-draw this thing on the iPad. 00:24:50.320 --> 00:24:52.060 align:middle line:84% But you should read it afterwards 00:24:52.060 --> 00:24:54.620 align:middle line:84% to make sure that whatever you saw on the iPad 00:24:54.620 --> 00:24:56.340 align:middle line:84% actually matches this because two 00:24:56.340 --> 00:25:00.020 align:middle line:84% different ways of understanding something always helps. 00:25:00.020 --> 00:25:01.800 align:middle line:90% So this is what we have here. 00:25:01.800 --> 00:25:07.063 align:middle line:84% Now to just to recall, by making self-attention tunable, 00:25:07.063 --> 00:25:08.480 align:middle line:84% we get a very interesting benefit, 00:25:08.480 --> 00:25:10.780 align:middle line:84% which is that when you have these different attention 00:25:10.780 --> 00:25:14.600 align:middle line:84% heads, before, you could have two attention heads, 00:25:14.600 --> 00:25:16.740 align:middle line:84% but because there were no parameters inside, 00:25:16.740 --> 00:25:19.260 align:middle line:84% their outputs would have been identical. 00:25:19.260 --> 00:25:21.312 align:middle line:84% Because the inputs are the same for both. 00:25:21.312 --> 00:25:23.020 align:middle line:84% Therefore, the outputs will be identical. 00:25:23.020 --> 00:25:28.220 align:middle line:84% But now since each attention head will have its own AQ AK AV 00:25:28.220 --> 00:25:31.940 align:middle line:84% matrix, the outputs are going to be different. 00:25:31.940 --> 00:25:35.160 align:middle line:84% That's why it makes sense to do the tunability thing 00:25:35.160 --> 00:25:37.660 align:middle line:84% because that's what actually makes multiple attention-- it's 00:25:37.660 --> 00:25:38.327 align:middle line:90% actually useful. 00:25:38.327 --> 00:25:42.987 align:middle line:90% 00:25:42.987 --> 00:25:44.820 align:middle line:84% AUDIENCE: Is there actually any relationship 00:25:44.820 --> 00:25:47.126 align:middle line:84% between AK, AQ, and AV, or is the A 00:25:47.126 --> 00:25:49.245 align:middle line:90% just for a notation standpoint? 00:25:49.245 --> 00:25:50.620 align:middle line:84% RAMA RAMAKRISHNAN: Just notation. 00:25:50.620 --> 00:25:54.340 align:middle line:84% The thing is we want to use QKV for the resulting matrix, 00:25:54.340 --> 00:25:57.000 align:middle line:84% and so I had to find something else to use for the first one. 00:25:57.000 --> 00:25:58.500 align:middle line:90% And I was like, OK, AQ AQ. 00:25:58.500 --> 00:26:01.080 align:middle line:84% And we are at MIT, we do subscripts, superscripts. 00:26:01.080 --> 00:26:03.340 align:middle line:90% Yeah? 00:26:03.340 --> 00:26:05.700 align:middle line:84% AUDIENCE: What is the size of the matrices? 00:26:05.700 --> 00:26:07.540 align:middle line:90% They're like square matrices or? 00:26:07.540 --> 00:26:08.540 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:26:08.540 --> 00:26:11.760 align:middle line:84% So typically what happens is that there is a whole bunch-- 00:26:11.760 --> 00:26:14.380 align:middle line:84% you can think of it as a hyperparameter in some ways. 00:26:14.380 --> 00:26:16.642 align:middle line:84% Typically what people do in most implementations is 00:26:16.642 --> 00:26:18.600 align:middle line:84% that they will actually just preserve the size. 00:26:18.600 --> 00:26:20.375 align:middle line:84% So if the incoming embedding is 10, 00:26:20.375 --> 00:26:22.500 align:middle line:84% they'll make sure the thing coming out of the thing 00:26:22.500 --> 00:26:23.520 align:middle line:90% is also 10. 00:26:23.520 --> 00:26:26.980 align:middle line:84% So you just do a 10 by 10 matrix to transform it. 00:26:26.980 --> 00:26:30.962 align:middle line:84% But the value V AV matrix, on the other hand, 00:26:30.962 --> 00:26:32.420 align:middle line:84% there is a bit more technical stuff 00:26:32.420 --> 00:26:36.020 align:middle line:84% going on where it often tends to be smaller. 00:26:36.020 --> 00:26:38.540 align:middle line:84% So for example, let's say that your incoming is 100. 00:26:38.540 --> 00:26:41.760 align:middle line:84% You do 100 to 100 for the key, 100 to 100 for the query. 00:26:41.760 --> 00:26:46.100 align:middle line:84% But if you have say, 5 attention heads, you may do 100 to 20 00:26:46.100 --> 00:26:48.220 align:middle line:84% for the Vs because ultimately all the Vs 00:26:48.220 --> 00:26:51.340 align:middle line:84% are going to get concatenated into another 100 again. 00:26:51.340 --> 00:26:53.140 align:middle line:90% So I can tell you more offline. 00:26:53.140 --> 00:26:56.080 align:middle line:84% But broadly speaking, these things tend to get transformed. 00:26:56.080 --> 00:26:58.660 align:middle line:84% They preserve the dimension, 10 in, 10 out. 00:26:58.660 --> 00:27:00.260 align:middle line:90% Yeah? 00:27:00.260 --> 00:27:04.700 align:middle line:84% AUDIENCE: So this AQ AV, these numbers 00:27:04.700 --> 00:27:06.995 align:middle line:84% are random you start with it and then [INAUDIBLE]. 00:27:06.995 --> 00:27:08.120 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:27:08.120 --> 00:27:08.620 align:middle line:90% Exactly. 00:27:08.620 --> 00:27:17.620 align:middle line:90% 00:27:17.620 --> 00:27:20.020 align:middle line:84% So the values in these matrices are weights learned 00:27:20.020 --> 00:27:22.460 align:middle line:90% through optimization using SGD. 00:27:22.460 --> 00:27:26.860 align:middle line:84% And then what that means is that each of these attention rates 00:27:26.860 --> 00:27:28.880 align:middle line:84% now has its own copy of these matrices. 00:27:28.880 --> 00:27:30.540 align:middle line:90% It has its own matrices. 00:27:30.540 --> 00:27:32.180 align:middle line:84% And over the course of backpropagation, 00:27:32.180 --> 00:27:35.272 align:middle line:84% these matrices will look very different. 00:27:35.272 --> 00:27:36.700 align:middle line:84% So important, each attention head 00:27:36.700 --> 00:27:39.723 align:middle line:84% will have its own set of three matrices. 00:27:39.723 --> 00:27:41.140 align:middle line:84% So if you have 10 attention heads, 00:27:41.140 --> 00:27:42.307 align:middle line:90% 30 matrices will be learned. 00:27:42.307 --> 00:27:46.300 align:middle line:90% 00:27:46.300 --> 00:27:48.060 align:middle line:84% AUDIENCE: So by the math, it seems 00:27:48.060 --> 00:27:51.380 align:middle line:84% like it's creating essentially a relationship between all 00:27:51.380 --> 00:27:52.960 align:middle line:90% of the content being ingested. 00:27:52.960 --> 00:27:57.360 align:middle line:84% And if you're ingesting all the content for each attention head, 00:27:57.360 --> 00:28:00.140 align:middle line:84% are there different categories of attention head type 00:28:00.140 --> 00:28:01.760 align:middle line:90% that you're trying to go after? 00:28:01.760 --> 00:28:02.060 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:28:02.060 --> 00:28:04.560 align:middle line:84% So basically what we are trying to do is to say a particular 00:28:04.560 --> 00:28:06.860 align:middle line:84% attention head-- so in any particular sentence, 00:28:06.860 --> 00:28:09.980 align:middle line:84% it may turn out to be the case that one pattern could be about 00:28:09.980 --> 00:28:12.110 align:middle line:84% the meanings of these words, like the word "bank" 00:28:12.110 --> 00:28:14.302 align:middle line:84% and what it means, the word "station," "train," 00:28:14.302 --> 00:28:15.010 align:middle line:90% things like that. 00:28:15.010 --> 00:28:16.850 align:middle line:84% That's what really we've been talking about. 00:28:16.850 --> 00:28:18.310 align:middle line:84% But there is a whole other pattern 00:28:18.310 --> 00:28:21.050 align:middle line:84% to do with grammar and tense and things like that. 00:28:21.050 --> 00:28:23.590 align:middle line:84% There could be another one in terms of tone. 00:28:23.590 --> 00:28:25.950 align:middle line:84% All those things are very important, and a priori 00:28:25.950 --> 00:28:27.970 align:middle line:84% we don't know how many such patterns exist. 00:28:27.970 --> 00:28:29.687 align:middle line:84% Much like in a convolutional network, 00:28:29.687 --> 00:28:31.270 align:middle line:84% when we are designing how many filters 00:28:31.270 --> 00:28:33.520 align:middle line:84% to have, we don't know how many kinds of little things 00:28:33.520 --> 00:28:36.697 align:middle line:84% we have to detect, vertical line, horizontal line, 00:28:36.697 --> 00:28:38.530 align:middle line:84% semicircle, quarter circle, stuff like that. 00:28:38.530 --> 00:28:40.535 align:middle line:84% So you just give it a lot of capacity 00:28:40.535 --> 00:28:42.160 align:middle line:84% so that it can learn whatever it wants. 00:28:42.160 --> 00:28:45.530 align:middle line:90% 00:28:45.530 --> 00:28:48.650 align:middle line:84% So that is the transformer encoder. 00:28:48.650 --> 00:28:52.110 align:middle line:84% So we have done the first of the three complications needed 00:28:52.110 --> 00:28:55.772 align:middle line:84% to make it industrial strength and legit. 00:28:55.772 --> 00:28:57.230 align:middle line:84% The second thing we do is something 00:28:57.230 --> 00:28:59.230 align:middle line:90% called the residual connection. 00:28:59.230 --> 00:29:04.350 align:middle line:84% So what we do is that whatever comes out here, 00:29:04.350 --> 00:29:08.750 align:middle line:84% W1 through W6, goes in and comes out as W1 hat, W2, 00:29:08.750 --> 00:29:10.690 align:middle line:90% and so on and so forth. 00:29:10.690 --> 00:29:12.790 align:middle line:90% Actually, sorry. 00:29:12.790 --> 00:29:14.670 align:middle line:90% What comes out here is the hats. 00:29:14.670 --> 00:29:17.582 align:middle line:84% But what comes out here is some intermediate Ws. 00:29:17.582 --> 00:29:19.790 align:middle line:84% That is what the self-attention is going to give you, 00:29:19.790 --> 00:29:21.310 align:middle line:90% some intermediate Ws. 00:29:21.310 --> 00:29:23.750 align:middle line:84% What we do is-- and because what's coming out here, 00:29:23.750 --> 00:29:26.930 align:middle line:84% these vectors are the same length as what goes in, 00:29:26.930 --> 00:29:29.430 align:middle line:84% we can just add them element by element. 00:29:29.430 --> 00:29:31.870 align:middle line:84% So we take the input and we actually 00:29:31.870 --> 00:29:34.950 align:middle line:90% add it to what comes out. 00:29:34.950 --> 00:29:37.350 align:middle line:90% So why would we want to do that? 00:29:37.350 --> 00:29:40.150 align:middle line:84% Why would we want to go to a lot of trouble 00:29:40.150 --> 00:29:43.350 align:middle line:84% to process this thing, and then when it comes out, we literally 00:29:43.350 --> 00:29:45.550 align:middle line:90% add up the original input? 00:29:45.550 --> 00:29:47.210 align:middle line:84% What do you think the intuition is? 00:29:47.210 --> 00:29:52.350 align:middle line:90% 00:29:52.350 --> 00:29:54.950 align:middle line:90% So it turns out-- 00:29:54.950 --> 00:29:56.030 align:middle line:90% think of it this way. 00:29:56.030 --> 00:29:57.330 align:middle line:90% You have a bunch of inputs. 00:29:57.330 --> 00:29:58.990 align:middle line:90% You send it to a neural network. 00:29:58.990 --> 00:30:01.710 align:middle line:84% It transforms it and gives you something else. 00:30:01.710 --> 00:30:05.550 align:middle line:84% At that point, you might be thinking, well, 00:30:05.550 --> 00:30:08.870 align:middle line:84% everything that happens in the network from that point onward 00:30:08.870 --> 00:30:11.390 align:middle line:84% can no longer see your original input. 00:30:11.390 --> 00:30:14.070 align:middle line:84% It can only work with the transformed input, right? 00:30:14.070 --> 00:30:17.590 align:middle line:84% But what if your transformations are not great? 00:30:17.590 --> 00:30:20.470 align:middle line:84% So as an insurance policy, what you can do 00:30:20.470 --> 00:30:23.470 align:middle line:84% is you can take the transformed stuff 00:30:23.470 --> 00:30:28.130 align:middle line:84% and you can take the original stuff and send both in. 00:30:28.130 --> 00:30:29.693 align:middle line:90% And this whole thing-- 00:30:29.693 --> 00:30:30.610 align:middle line:90% and you can Google it. 00:30:30.610 --> 00:30:33.090 align:middle line:84% It's called a wide and deep network and things like that. 00:30:33.090 --> 00:30:35.430 align:middle line:84% But the whole point is that, let's not lose 00:30:35.430 --> 00:30:37.050 align:middle line:90% the original input anywhere. 00:30:37.050 --> 00:30:38.510 align:middle line:90% Let's also send it along. 00:30:38.510 --> 00:30:40.295 align:middle line:84% But if you keep adding the original input 00:30:40.295 --> 00:30:41.670 align:middle line:84% to every intermediate layer, it's 00:30:41.670 --> 00:30:43.450 align:middle line:84% going to get longer and longer and longer and bigger, 00:30:43.450 --> 00:30:46.158 align:middle line:84% which you don't want because you want it all to be the same size. 00:30:46.158 --> 00:30:49.110 align:middle line:84% So the simplest alternative is to just add them up. 00:30:49.110 --> 00:30:51.770 align:middle line:84% You take the transformed stuff and you add the original input, 00:30:51.770 --> 00:30:54.610 align:middle line:90% you get the same thing again. 00:30:54.610 --> 00:30:57.550 align:middle line:84% What came in, W1 was a 100-long vector, 00:30:57.550 --> 00:30:59.890 align:middle line:84% and the transformed version is also 100 long. 00:30:59.890 --> 00:31:02.210 align:middle line:84% So just literally under 100, add them up. 00:31:02.210 --> 00:31:02.910 align:middle line:90% That's it. 00:31:02.910 --> 00:31:04.950 align:middle line:90% You get another 100-long vector. 00:31:04.950 --> 00:31:08.150 align:middle line:84% So that is what's called a residual connection. 00:31:08.150 --> 00:31:12.470 align:middle line:84% And as it turns out, residual connections 00:31:12.470 --> 00:31:16.310 align:middle line:84% improve the gradient flow during backpropagation dramatically, 00:31:16.310 --> 00:31:18.250 align:middle line:84% and that's why they are very heavily used. 00:31:18.250 --> 00:31:22.390 align:middle line:84% And in fact, ResNet, which we looked at for computer vision, 00:31:22.390 --> 00:31:26.150 align:middle line:84% it stands for Residual Net because it was the first network 00:31:26.150 --> 00:31:28.928 align:middle line:90% to actually figure this out. 00:31:28.928 --> 00:31:30.970 align:middle line:84% This is not just a transformer thing, by the way. 00:31:30.970 --> 00:31:34.790 align:middle line:84% It's widely used in lots of deep neural architectures. 00:31:34.790 --> 00:31:36.290 align:middle line:84% The notion of a residual connection, 00:31:36.290 --> 00:31:38.170 align:middle line:90% that's what it means. 00:31:38.170 --> 00:31:41.870 align:middle line:84% So we do a residual connection, and then we 00:31:41.870 --> 00:31:45.590 align:middle line:84% come to the final tweak, which is called layer normalization. 00:31:45.590 --> 00:31:47.350 align:middle line:84% So once we add the residual connection, 00:31:47.350 --> 00:31:50.110 align:middle line:84% we are going to do something else here to these vectors 00:31:50.110 --> 00:31:52.270 align:middle line:90% before they continue flowing. 00:31:52.270 --> 00:31:57.990 align:middle line:84% And what layer normalization does is it basically says that-- 00:31:57.990 --> 00:32:00.330 align:middle line:84% you will recall from the very beginning of the semester, 00:32:00.330 --> 00:32:03.110 align:middle line:84% I've been saying that whatever comes into a neural network, 00:32:03.110 --> 00:32:04.670 align:middle line:84% the inputs, let's just really make 00:32:04.670 --> 00:32:06.128 align:middle line:84% sure that they are all in some sort 00:32:06.128 --> 00:32:07.710 align:middle line:90% of a narrow, well-defined range. 00:32:07.710 --> 00:32:09.950 align:middle line:90% They can't be in a big range. 00:32:09.950 --> 00:32:13.830 align:middle line:84% So for pictures, for images, we divided every number by 255 00:32:13.830 --> 00:32:18.150 align:middle line:84% so that every little pixel value is between 0 and 1. 00:32:18.150 --> 00:32:20.650 align:middle line:84% For continuous things, like the heart disease example, 00:32:20.650 --> 00:32:22.652 align:middle line:84% we standardized it by calculating 00:32:22.652 --> 00:32:24.110 align:middle line:84% the mean and the standard deviation 00:32:24.110 --> 00:32:25.767 align:middle line:84% and subtracting the mean and dividing 00:32:25.767 --> 00:32:26.850 align:middle line:90% by the standard deviation. 00:32:26.850 --> 00:32:28.350 align:middle line:84% So when you do that, all the numbers 00:32:28.350 --> 00:32:31.590 align:middle line:84% are going to roughly be in the minus 1 to plus 1 range. 00:32:31.590 --> 00:32:35.690 align:middle line:84% So in neural networks, for a backprop to work really well, 00:32:35.690 --> 00:32:39.230 align:middle line:84% you have to make sure that no numbers get too big, 00:32:39.230 --> 00:32:42.070 align:middle line:84% that all the numbers are always in some sort of a narrow range. 00:32:42.070 --> 00:32:45.250 align:middle line:84% So what layer normalization does is to say, 00:32:45.250 --> 00:32:47.870 align:middle line:84% you know what, whatever is coming out here, 00:32:47.870 --> 00:32:50.508 align:middle line:84% I want to make sure none of these numbers are too big. 00:32:50.508 --> 00:32:53.050 align:middle line:84% I want to make sure they're all well-behaved in a small range 00:32:53.050 --> 00:32:55.110 align:middle line:84% because if I don't do that, backprop 00:32:55.110 --> 00:32:56.410 align:middle line:90% is not going to work very well. 00:32:56.410 --> 00:32:59.512 align:middle line:90% 00:32:59.512 --> 00:33:01.470 align:middle line:84% AUDIENCE: Is this what we do to ensure we don't 00:33:01.470 --> 00:33:03.428 align:middle line:90% have the problem of vanishing? 00:33:03.428 --> 00:33:04.470 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:33:04.470 --> 00:33:07.317 align:middle line:84% So technically there could be two problems. 00:33:07.317 --> 00:33:09.650 align:middle line:84% There is an exploding gradient and a vanishing gradient. 00:33:09.650 --> 00:33:10.370 align:middle line:90% Both are bad. 00:33:10.370 --> 00:33:11.650 align:middle line:90% This is a way to address it. 00:33:11.650 --> 00:33:14.110 align:middle line:84% So you will find a whole bunch of batch normalization 00:33:14.110 --> 00:33:17.090 align:middle line:84% techniques, layer normalization, batch normalization, 00:33:17.090 --> 00:33:18.190 align:middle line:90% and so on and so forth. 00:33:18.190 --> 00:33:20.310 align:middle line:84% All these are methods to make sure 00:33:20.310 --> 00:33:22.130 align:middle line:84% that these numbers stay in a small range 00:33:22.130 --> 00:33:23.970 align:middle line:84% so it doesn't cause gradient issues later. 00:33:23.970 --> 00:33:27.950 align:middle line:90% 00:33:27.950 --> 00:33:31.750 align:middle line:90% In particular, what we do is-- 00:33:31.750 --> 00:33:34.543 align:middle line:84% or what happens inside this layer normalization-- 00:33:34.543 --> 00:33:36.710 align:middle line:84% is we just calculate the mean and standard deviation 00:33:36.710 --> 00:33:40.230 align:middle line:84% of every one of these embeddings If you have, let's say, 00:33:40.230 --> 00:33:43.090 align:middle line:84% 6 embeddings here, we'll have 6 means and 6 standard deviations 00:33:43.090 --> 00:33:44.750 align:middle line:90% for each one across the rows. 00:33:44.750 --> 00:33:47.595 align:middle line:84% And then we standardize it, meaning subtract the mean, 00:33:47.595 --> 00:33:48.970 align:middle line:84% divide by the standard deviation. 00:33:48.970 --> 00:33:51.030 align:middle line:84% And when you do that, all these things 00:33:51.030 --> 00:33:52.550 align:middle line:90% are going to be nice and small. 00:33:52.550 --> 00:33:54.630 align:middle line:84% And then we do this little other thing 00:33:54.630 --> 00:33:57.790 align:middle line:84% where we have introduced two new parameters 00:33:57.790 --> 00:34:00.790 align:middle line:84% to rescale it and move it around a little bit 00:34:00.790 --> 00:34:03.270 align:middle line:84% just because adding more weights always 00:34:03.270 --> 00:34:06.130 align:middle line:84% helps make these things better, so we add them. 00:34:06.130 --> 00:34:08.550 align:middle line:84% And this gets slightly complicated because of the way 00:34:08.550 --> 00:34:11.469 align:middle line:84% the dimensions work, so I'm not going to spend much time on it. 00:34:11.469 --> 00:34:13.750 align:middle line:84% And then what comes out the other end 00:34:13.750 --> 00:34:15.590 align:middle line:84% is a very well-behaved set of numbers 00:34:15.590 --> 00:34:19.469 align:middle line:84% and a nice and small and narrow range. 00:34:19.469 --> 00:34:21.909 align:middle line:84% So this is called layer normalization. 00:34:21.909 --> 00:34:25.510 align:middle line:84% You can see this link to understand it a bit better. 00:34:25.510 --> 00:34:26.989 align:middle line:90% And we do that as well. 00:34:26.989 --> 00:34:32.185 align:middle line:84% So to put it all together, so this is a transformer encoder 00:34:32.185 --> 00:34:34.310 align:middle line:84% where we have this multi-head attention layer where 00:34:34.310 --> 00:34:36.750 align:middle line:84% each attention head in the inside of it 00:34:36.750 --> 00:34:38.949 align:middle line:84% is tunable with those A matrices, 00:34:38.949 --> 00:34:41.010 align:middle line:84% and then we have a residual connection. 00:34:41.010 --> 00:34:42.830 align:middle line:84% We do that, and then we do layernorm. 00:34:42.830 --> 00:34:45.949 align:middle line:84% And then we do the same thing in the next feedforward layer 00:34:45.949 --> 00:34:47.820 align:middle line:84% as well and then boom, out pops the output. 00:34:47.820 --> 00:34:50.260 align:middle line:90% 00:34:50.260 --> 00:34:52.510 align:middle line:84% AUDIENCE: By that definition, the multi-head attention 00:34:52.510 --> 00:34:54.750 align:middle line:84% layer [INAUDIBLE] and everything, theoretically 00:34:54.750 --> 00:34:59.430 align:middle line:84% I can add even the biases or the hate speech aspects which 00:34:59.430 --> 00:35:01.650 align:middle line:90% come in to take care of it. 00:35:01.650 --> 00:35:05.710 align:middle line:84% So the model can account for the fact that something is biased 00:35:05.710 --> 00:35:06.950 align:middle line:90% or something is not. 00:35:06.950 --> 00:35:08.950 align:middle line:84% RAMA RAMAKRISHNAN: The thing is it's not so much 00:35:08.950 --> 00:35:10.650 align:middle line:90% the model's accounting for it. 00:35:10.650 --> 00:35:12.710 align:middle line:84% It is capturing whatever patterns happen 00:35:12.710 --> 00:35:14.550 align:middle line:90% to be inherent in the data. 00:35:14.550 --> 00:35:16.260 align:middle line:90% It's capturing. 00:35:16.260 --> 00:35:18.337 align:middle line:84% Now what you do with that capture is up to you. 00:35:18.337 --> 00:35:20.670 align:middle line:84% It depends on the actual problem you're trying to solve. 00:35:20.670 --> 00:35:24.070 align:middle line:84% In particular, it is going to capture all the bad stuff, too. 00:35:24.070 --> 00:35:26.730 align:middle line:84% Because if your training data has a lot of biased stuff in it, 00:35:26.730 --> 00:35:29.450 align:middle line:84% toxic things in it, dangerous things in it, 00:35:29.450 --> 00:35:30.830 align:middle line:84% it doesn't have a sense of values 00:35:30.830 --> 00:35:34.990 align:middle line:84% as to what is good or bad, it's just going to pick it up. 00:35:34.990 --> 00:35:35.990 align:middle line:90% Yes? 00:35:35.990 --> 00:35:38.150 align:middle line:84% AUDIENCE: Picking on that, then how do you actually 00:35:38.150 --> 00:35:40.350 align:middle line:84% make it unlearn those or how do you 00:35:40.350 --> 00:35:42.042 align:middle line:90% mitigate the effect of those? 00:35:42.042 --> 00:35:44.250 align:middle line:84% RAMA RAMAKRISHNAN: That's a whole course unto itself, 00:35:44.250 --> 00:35:46.042 align:middle line:84% but I'm happy to give you pointers offline. 00:35:46.042 --> 00:35:48.490 align:middle line:90% 00:35:48.490 --> 00:35:50.270 align:middle line:90% So this is what we have. 00:35:50.270 --> 00:35:52.590 align:middle line:84% And remember what I said, that this is just 00:35:52.590 --> 00:35:54.170 align:middle line:90% a single transformer block. 00:35:54.170 --> 00:35:55.910 align:middle line:84% And since what comes in and what goes out 00:35:55.910 --> 00:35:57.630 align:middle line:84% are the same dimensions, we can just 00:35:57.630 --> 00:35:59.950 align:middle line:90% stack them one after the other. 00:35:59.950 --> 00:36:02.450 align:middle line:90% It's very stackable. 00:36:02.450 --> 00:36:04.670 align:middle line:84% You can stack it vertically as much as you want. 00:36:04.670 --> 00:36:08.390 align:middle line:84% And as I mentioned, I think GPT3 has 96 of these things stacked 00:36:08.390 --> 00:36:09.410 align:middle line:90% one on top of the other. 00:36:09.410 --> 00:36:14.190 align:middle line:90% 00:36:14.190 --> 00:36:17.470 align:middle line:84% That is the transformer encoder, and this exactly maps to that. 00:36:17.470 --> 00:36:19.810 align:middle line:84% So basically the input embeddings come in, 00:36:19.810 --> 00:36:21.790 align:middle line:84% you add positional embeddings, and then 00:36:21.790 --> 00:36:24.550 align:middle line:84% you send it to, say, these mini attention blocks. 00:36:24.550 --> 00:36:26.630 align:middle line:84% And they all get added up, and then it comes out 00:36:26.630 --> 00:36:27.750 align:middle line:90% to the attention block. 00:36:27.750 --> 00:36:30.753 align:middle line:84% You add-- the add and norm here means-- "add" 00:36:30.753 --> 00:36:33.170 align:middle line:84% means residual connection because you're adding the input, 00:36:33.170 --> 00:36:36.350 align:middle line:84% which is why you have this arrow going from the input being added 00:36:36.350 --> 00:36:38.650 align:middle line:84% there, and then you normalize it, send it along, 00:36:38.650 --> 00:36:40.400 align:middle line:84% and do it again, and out comes the output. 00:36:40.400 --> 00:36:45.230 align:middle line:90% 00:36:45.230 --> 00:36:47.070 align:middle line:84% Now just to be very clear on what 00:36:47.070 --> 00:36:48.910 align:middle line:84% is being optimized during backpropagation 00:36:48.910 --> 00:36:54.550 align:middle line:84% in this complex flow, now, clearly the embeddings that you 00:36:54.550 --> 00:36:57.070 align:middle line:84% started out with, both the standalone embeddings as well 00:36:57.070 --> 00:36:59.870 align:middle line:84% as the position embeddings, those things 00:36:59.870 --> 00:37:01.492 align:middle line:90% are going to get optimized. 00:37:01.492 --> 00:37:02.450 align:middle line:90% Those are just weights. 00:37:02.450 --> 00:37:03.910 align:middle line:90% They're going to get optimized. 00:37:03.910 --> 00:37:06.310 align:middle line:84% Clearly everything inside the transformer encoder block 00:37:06.310 --> 00:37:07.890 align:middle line:90% is going to get normalized. 00:37:07.890 --> 00:37:09.130 align:middle line:90% And what are they? 00:37:09.130 --> 00:37:12.170 align:middle line:84% Well, they are the AK, AQ, AV matrices 00:37:12.170 --> 00:37:14.150 align:middle line:90% for each attention head. 00:37:14.150 --> 00:37:17.870 align:middle line:84% Layer norm has parameters as well. 00:37:17.870 --> 00:37:20.990 align:middle line:84% The little feedforward layer has weights as well. 00:37:20.990 --> 00:37:23.350 align:middle line:84% All these things are going to get optimized. 00:37:23.350 --> 00:37:25.380 align:middle line:84% And then it goes through this ReLU which, 00:37:25.380 --> 00:37:26.630 align:middle line:90% again, has a bunch of weights. 00:37:26.630 --> 00:37:27.810 align:middle line:90% That's going to get optimized. 00:37:27.810 --> 00:37:29.893 align:middle line:84% And then the final softmax has a bunch of weights. 00:37:29.893 --> 00:37:31.623 align:middle line:90% That's going to get optimized. 00:37:31.623 --> 00:37:34.040 align:middle line:84% All these things are going to get optimized by a backprop. 00:37:34.040 --> 00:37:36.570 align:middle line:90% 00:37:36.570 --> 00:37:39.083 align:middle line:84% So in that sense, you just step back for a second 00:37:39.083 --> 00:37:40.250 align:middle line:90% and look at the whole thing. 00:37:40.250 --> 00:37:43.370 align:middle line:84% It is just a mathematical model with a lot of parameters. 00:37:43.370 --> 00:37:45.870 align:middle line:84% And we're just going to use a gradient descent or stochastic 00:37:45.870 --> 00:37:46.995 align:middle line:90% gradient descent optimizer. 00:37:46.995 --> 00:37:49.430 align:middle line:90% That's it. 00:37:49.430 --> 00:37:51.290 align:middle line:90% Yeah? 00:37:51.290 --> 00:37:53.990 align:middle line:84% AUDIENCE: [INAUDIBLE] we train the model. 00:37:53.990 --> 00:37:56.550 align:middle line:84% Are we calculating weights for each cell 00:37:56.550 --> 00:37:59.470 align:middle line:84% of every possible matrix based on the number 00:37:59.470 --> 00:38:02.590 align:middle line:84% of inputs, every possible dimension up to the max number 00:38:02.590 --> 00:38:04.523 align:middle line:90% of inputs? 00:38:04.523 --> 00:38:06.190 align:middle line:84% RAMA RAMAKRISHNAN: Actually, the weights 00:38:06.190 --> 00:38:10.253 align:middle line:84% themselves don't depend on how long your input sentence is. 00:38:10.253 --> 00:38:11.670 align:middle line:84% Because remember, what we're doing 00:38:11.670 --> 00:38:14.230 align:middle line:84% is for each sentence that comes in-- let's say the sentence 00:38:14.230 --> 00:38:15.710 align:middle line:90% has, say, three words. 00:38:15.710 --> 00:38:17.670 align:middle line:84% There are three embeddings for that sentence. 00:38:17.670 --> 00:38:22.670 align:middle line:84% Each of those embeddings gets multiplied by, say, AK. 00:38:22.670 --> 00:38:27.070 align:middle line:84% So AK only needs to know how long is each embedding. 00:38:27.070 --> 00:38:31.630 align:middle line:84% It doesn't need to know, how many words do I have. 00:38:31.630 --> 00:38:33.650 align:middle line:84% And I'm glad you raised that question, Ben, 00:38:33.650 --> 00:38:37.550 align:middle line:84% because that's what makes the transformers' number of weights 00:38:37.550 --> 00:38:42.110 align:middle line:84% independent of the number of words in your sentence. 00:38:42.110 --> 00:38:43.502 align:middle line:84% It only depends on the vocabulary 00:38:43.502 --> 00:38:45.710 align:middle line:84% that you're going to work with because the vocabulary 00:38:45.710 --> 00:38:47.810 align:middle line:84% determines how many embeddings you need, 00:38:47.810 --> 00:38:50.030 align:middle line:90% how many embeddings you need. 00:38:50.030 --> 00:38:53.540 align:middle line:84% The length only matters in terms of the positional embedding 00:38:53.540 --> 00:38:55.580 align:middle line:84% because if you have 1,000-long sentence, 00:38:55.580 --> 00:38:58.520 align:middle line:84% you need 1,000-long positional embedding matrix. 00:38:58.520 --> 00:39:02.280 align:middle line:84% But beyond that, it doesn't care. 00:39:02.280 --> 00:39:07.560 align:middle line:84% And that's why, for example, Google Gemini 1.5 Pro, which 00:39:07.560 --> 00:39:09.280 align:middle line:84% can accommodate basically 1 million long, 00:39:09.280 --> 00:39:14.520 align:middle line:84% 1 million token context window, it's still very compute-heavy, 00:39:14.520 --> 00:39:19.080 align:middle line:84% but it does not change the number of parameters. 00:39:19.080 --> 00:39:20.940 align:middle line:90% Yeah? 00:39:20.940 --> 00:39:24.680 align:middle line:84% AUDIENCE: Essentially, which weights are optimized first? 00:39:24.680 --> 00:39:27.305 align:middle line:84% Are they in sequential order or are they 00:39:27.305 --> 00:39:29.680 align:middle line:84% optimizing the weights at the very same time [INAUDIBLE]. 00:39:29.680 --> 00:39:31.097 align:middle line:84% RAMA RAMAKRISHNAN: Simultaneously. 00:39:31.097 --> 00:39:33.100 align:middle line:84% Because if you think of backpropagation, 00:39:33.100 --> 00:39:35.000 align:middle line:84% ultimately, you have a loss function, 00:39:35.000 --> 00:39:37.560 align:middle line:84% and you calculate the gradient of that loss function. 00:39:37.560 --> 00:39:39.940 align:middle line:84% So if you have, say, 1 billion parameters, 00:39:39.940 --> 00:39:43.098 align:middle line:84% that gradient is basically a 1-billion long vector. 00:39:43.098 --> 00:39:44.640 align:middle line:84% And we're going to take the gradient, 00:39:44.640 --> 00:39:49.240 align:middle line:84% and we're going to do W new equals W old minus alpha times 00:39:49.240 --> 00:39:50.400 align:middle line:90% that gradient. 00:39:50.400 --> 00:39:53.520 align:middle line:84% So all the Ws are going to update instantaneously. 00:39:53.520 --> 00:39:55.480 align:middle line:84% Now the way it actually works in computation 00:39:55.480 --> 00:39:56.840 align:middle line:90% is you're going to do it-- 00:39:56.840 --> 00:39:58.540 align:middle line:84% because of the back and backpropagation, 00:39:58.540 --> 00:40:00.915 align:middle line:84% it's going to start at the end and slowly flow backwards, 00:40:00.915 --> 00:40:03.960 align:middle line:84% but when it's done, everything will be updated. 00:40:03.960 --> 00:40:05.933 align:middle line:90% Yeah? 00:40:05.933 --> 00:40:09.800 align:middle line:84% AUDIENCE: [INAUDIBLE] we take two attention heads 00:40:09.800 --> 00:40:14.040 align:middle line:84% and we have the matrices of AK, AQ, and AV. 00:40:14.040 --> 00:40:17.218 align:middle line:84% [INAUDIBLE] the parameters of all three of them, 00:40:17.218 --> 00:40:18.760 align:middle line:84% all the weights of the three matrices 00:40:18.760 --> 00:40:21.460 align:middle line:84% on this side and this side would be different because finally, 00:40:21.460 --> 00:40:24.440 align:middle line:84% the things you are inputting from this side and the output 00:40:24.440 --> 00:40:25.008 align:middle line:90% is same. 00:40:25.008 --> 00:40:26.716 align:middle line:84% So the learning process should be ideally 00:40:26.716 --> 00:40:29.280 align:middle line:84% the same unlike like a CNN, where we had put 00:40:29.280 --> 00:40:30.708 align:middle line:90% filters which were different. 00:40:30.708 --> 00:40:32.000 align:middle line:90% So what different thing we have 00:40:32.000 --> 00:40:34.482 align:middle line:84% RAMA RAMAKRISHNAN: Because the initialization is different. 00:40:34.482 --> 00:40:35.024 align:middle line:90% AUDIENCE: Oh. 00:40:35.024 --> 00:40:35.732 align:middle line:90% What do you mean? 00:40:35.732 --> 00:40:38.066 align:middle line:84% RAMA RAMAKRISHNAN: What I mean is if you have two heads, 00:40:38.066 --> 00:40:39.300 align:middle line:90% each head has three matrices. 00:40:39.300 --> 00:40:42.360 align:middle line:84% The starting values of those six matrices are different. 00:40:42.360 --> 00:40:45.360 align:middle line:84% AUDIENCE: If the starting value of AK, AQ, and AV 00:40:45.360 --> 00:40:46.540 align:middle line:90% is different on both heads. 00:40:46.540 --> 00:40:47.320 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:40:47.320 --> 00:40:49.487 align:middle line:84% Much like for all the weights, typically, the values 00:40:49.487 --> 00:40:51.480 align:middle line:90% are randomly chosen. 00:40:51.480 --> 00:40:53.935 align:middle line:84% If they were all the same thing, you're right, 00:40:53.935 --> 00:40:55.060 align:middle line:90% it won't make a difference. 00:40:55.060 --> 00:40:57.260 align:middle line:84% But they will all change the same way. 00:40:57.260 --> 00:40:57.760 align:middle line:90% Yeah? 00:40:57.760 --> 00:41:00.320 align:middle line:90% 00:41:00.320 --> 00:41:03.360 align:middle line:84% AUDIENCE: Is the input of the transformer the sentence 00:41:03.360 --> 00:41:08.303 align:middle line:84% or the array of the embedding of each word? 00:41:08.303 --> 00:41:09.720 align:middle line:84% RAMA RAMAKRISHNAN: The transformer 00:41:09.720 --> 00:41:12.240 align:middle line:84% itself is expecting embeddings in. 00:41:12.240 --> 00:41:14.940 align:middle line:84% And so what basically happens is that we get some sentence, 00:41:14.940 --> 00:41:17.360 align:middle line:84% we run it through a tokenizer which connects it 00:41:17.360 --> 00:41:20.000 align:middle line:84% to a bunch of tokens, which are just integers, 00:41:20.000 --> 00:41:22.320 align:middle line:84% and then it goes through the embedding layer, which 00:41:22.320 --> 00:41:24.300 align:middle line:84% maps the integers to these embeddings, 00:41:24.300 --> 00:41:26.060 align:middle line:84% and then you feed it to the transformer. 00:41:26.060 --> 00:41:28.000 align:middle line:84% But when you do backpropagation, it 00:41:28.000 --> 00:41:30.560 align:middle line:84% comes all the way back to the starting embedding layer 00:41:30.560 --> 00:41:32.197 align:middle line:90% and updates those weights. 00:41:32.197 --> 00:41:34.280 align:middle line:84% AUDIENCE: So it can be trainable so the embeddings 00:41:34.280 --> 00:41:36.500 align:middle line:84% that we set at the beginning, we see them as input here, 00:41:36.500 --> 00:41:37.420 align:middle line:90% but they can be trained. 00:41:37.420 --> 00:41:39.378 align:middle line:84% RAMA RAMAKRISHNAN: They are trainable, exactly. 00:41:39.378 --> 00:41:40.160 align:middle line:90% Exactly. 00:41:40.160 --> 00:41:41.680 align:middle line:90% Yeah? 00:41:41.680 --> 00:41:44.280 align:middle line:84% AUDIENCE: Are the attention heads solely parallel or can 00:41:44.280 --> 00:41:46.620 align:middle line:84% you have a stack of attention heads? 00:41:46.620 --> 00:41:48.037 align:middle line:84% RAMA RAMAKRISHNAN: Typically, they 00:41:48.037 --> 00:41:50.080 align:middle line:84% are parallelized because you can always 00:41:50.080 --> 00:41:54.940 align:middle line:84% stack the block itself to get more and more power. 00:41:54.940 --> 00:41:59.800 align:middle line:84% So now to apply the transformer, common use cases 00:41:59.800 --> 00:42:02.720 align:middle line:84% are that you have a whole sentence that comes in, 00:42:02.720 --> 00:42:04.600 align:middle line:84% and then you just want to classify it, 00:42:04.600 --> 00:42:08.200 align:middle line:84% the canonical thing being hey, movie sentiment classification, 00:42:08.200 --> 00:42:09.700 align:middle line:84% boom, positive or negative right. 00:42:09.700 --> 00:42:10.800 align:middle line:90% Classification. 00:42:10.800 --> 00:42:13.640 align:middle line:84% Another common one is labeling, where every word gets labeled 00:42:13.640 --> 00:42:16.080 align:middle line:84% as a multiclass label, and that's basically 00:42:16.080 --> 00:42:18.223 align:middle line:84% what we saw with our slot-filling problem. 00:42:18.223 --> 00:42:20.640 align:middle line:84% And then there is another thing called sequence generation 00:42:20.640 --> 00:42:21.848 align:middle line:90% where you give it a sequence. 00:42:21.848 --> 00:42:24.300 align:middle line:84% You want it to continue the sequence, generate more stuff, 00:42:24.300 --> 00:42:26.640 align:middle line:84% i.e. large language models and all that good stuff. 00:42:26.640 --> 00:42:29.800 align:middle line:84% So this we already know how to do because we actually literally 00:42:29.800 --> 00:42:33.040 align:middle line:84% built a Colab with this, with the transformer stack. 00:42:33.040 --> 00:42:35.440 align:middle line:84% Now the question is, how can we do that? 00:42:35.440 --> 00:42:37.080 align:middle line:84% How can we do basic classification 00:42:37.080 --> 00:42:38.160 align:middle line:90% with these things? 00:42:38.160 --> 00:42:41.240 align:middle line:84% So now again, when you send a sentence in, 00:42:41.240 --> 00:42:44.380 align:middle line:84% after all that stuff is done, and when I say, encoder, here, 00:42:44.380 --> 00:42:46.620 align:middle line:84% I'm assuming that you may have one block. 00:42:46.620 --> 00:42:48.505 align:middle line:84% You may have 106 blocks, I don't care. 00:42:48.505 --> 00:42:50.380 align:middle line:84% At the end of the day, you send something in. 00:42:50.380 --> 00:42:52.255 align:middle line:84% You get a bunch of contextual embeddings out. 00:42:52.255 --> 00:42:55.040 align:middle line:90% 00:42:55.040 --> 00:42:58.120 align:middle line:84% So at this point, we need to take these contextual embeddings 00:42:58.120 --> 00:43:00.058 align:middle line:84% and somehow make it work for classification. 00:43:00.058 --> 00:43:02.100 align:middle line:84% We are just classifying something into yes or no, 00:43:02.100 --> 00:43:03.640 align:middle line:90% positive or negative. 00:43:03.640 --> 00:43:06.080 align:middle line:84% So it would be nice if we can actually 00:43:06.080 --> 00:43:08.160 align:middle line:84% take all these embeddings and essentially 00:43:08.160 --> 00:43:12.760 align:middle line:84% summarize them into a single embedding, a single vector. 00:43:12.760 --> 00:43:14.500 align:middle line:84% Because if we have a single vector, 00:43:14.500 --> 00:43:16.180 align:middle line:84% then we can run it through maybe a ReLU, 00:43:16.180 --> 00:43:17.740 align:middle line:84% and then we do a sigmoid, and boom, 00:43:17.740 --> 00:43:19.940 align:middle line:84% we can do a binary classification problem. 00:43:19.940 --> 00:43:21.480 align:middle line:90% Super easy. 00:43:21.480 --> 00:43:22.960 align:middle line:84% So this begs the question, OK, how 00:43:22.960 --> 00:43:25.600 align:middle line:84% are we going to go from all the many blue things to one 00:43:25.600 --> 00:43:26.380 align:middle line:90% green thing? 00:43:26.380 --> 00:43:29.572 align:middle line:90% 00:43:29.572 --> 00:43:35.453 align:middle line:84% Now of course, what we can do is we can simply average them. 00:43:35.453 --> 00:43:36.870 align:middle line:84% We can take each of the embeddings 00:43:36.870 --> 00:43:39.260 align:middle line:84% and just simply average them element by element. 00:43:39.260 --> 00:43:42.700 align:middle line:90% You'll get a nice green thing. 00:43:42.700 --> 00:43:44.740 align:middle line:84% Any shortcomings from doing that? 00:43:44.740 --> 00:43:48.440 align:middle line:90% 00:43:48.440 --> 00:43:51.995 align:middle line:84% AUDIENCE: You would lose the ordering of the words. 00:43:51.995 --> 00:43:53.620 align:middle line:84% RAMA RAMAKRISHNAN: Well, in some sense, 00:43:53.620 --> 00:43:56.000 align:middle line:84% the positional embedding, the positional encoding 00:43:56.000 --> 00:44:00.400 align:middle line:84% you have in the input does have this notion of position. 00:44:00.400 --> 00:44:03.760 align:middle line:84% So you're not necessarily losing the order necessarily, 00:44:03.760 --> 00:44:06.920 align:middle line:84% but you're sort of averaging all this information into something, 00:44:06.920 --> 00:44:08.870 align:middle line:84% and averaging is going to lose some richness. 00:44:08.870 --> 00:44:15.425 align:middle line:90% 00:44:15.425 --> 00:44:17.800 align:middle line:84% AUDIENCE: I think it's going to be skewed to the one that 00:44:17.800 --> 00:44:19.800 align:middle line:90% has the biggest number, so-- 00:44:19.800 --> 00:44:20.800 align:middle line:90% RAMA RAMAKRISHNAN: True. 00:44:20.800 --> 00:44:22.660 align:middle line:84% AUDIENCE: [INAUDIBLE] is influencing your-- 00:44:22.660 --> 00:44:23.320 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, the biggest ones 00:44:23.320 --> 00:44:24.237 align:middle line:90% are going to dominate. 00:44:24.237 --> 00:44:26.160 align:middle line:84% But hopefully we won't have too much of that 00:44:26.160 --> 00:44:28.342 align:middle line:84% because all the layernorm business in the beginning 00:44:28.342 --> 00:44:30.800 align:middle line:84% has hopefully made sure the numbers are all in a reasonably 00:44:30.800 --> 00:44:32.280 align:middle line:90% small and well-behaved range. 00:44:32.280 --> 00:44:33.840 align:middle line:84% But the point really is that you're 00:44:33.840 --> 00:44:35.840 align:middle line:84% going to lose richness in the information 00:44:35.840 --> 00:44:38.360 align:middle line:84% because you're just pssht, mushing it down. 00:44:38.360 --> 00:44:41.160 align:middle line:84% So there's a much better and more elegant 00:44:41.160 --> 00:44:44.680 align:middle line:84% way to do this, which is that what 00:44:44.680 --> 00:44:48.620 align:middle line:84% you do is for every sentence, when you train it, 00:44:48.620 --> 00:44:52.880 align:middle line:84% you add an artificial token called the class token. 00:44:52.880 --> 00:44:54.680 align:middle line:84% Literally, it's an artificial token 00:44:54.680 --> 00:44:58.520 align:middle line:84% and it's designated as CLS in the literature. 00:44:58.520 --> 00:45:04.440 align:middle line:84% And then this token is getting trained with everything else. 00:45:04.440 --> 00:45:08.520 align:middle line:84% And so once you finish training, that token 00:45:08.520 --> 00:45:10.640 align:middle line:90% has its own embedding, too. 00:45:10.640 --> 00:45:13.980 align:middle line:84% And because it has been trained with everything else-- 00:45:13.980 --> 00:45:16.340 align:middle line:84% and this token, remember, is a contextual embedding, 00:45:16.340 --> 00:45:18.840 align:middle line:84% which means that it's very much aware of all the other words 00:45:18.840 --> 00:45:21.080 align:middle line:90% in the sentence. 00:45:21.080 --> 00:45:25.280 align:middle line:84% So in some sense, this CLS token's contextual embedding 00:45:25.280 --> 00:45:29.340 align:middle line:84% captures everything that's going on about that sentence. 00:45:29.340 --> 00:45:31.820 align:middle line:84% And so what we do is once we are done training, 00:45:31.820 --> 00:45:34.040 align:middle line:84% we just grab this thing alone and then 00:45:34.040 --> 00:45:38.920 align:middle line:84% send that through a ReLU and a sigmoid and boom, you're done. 00:45:38.920 --> 00:45:43.425 align:middle line:84% So this is a very clever trick to somehow, instead of averaging 00:45:43.425 --> 00:45:44.800 align:middle line:84% everything at the end, let's just 00:45:44.800 --> 00:45:47.300 align:middle line:84% have something just for the whole thing, the sentence, 00:45:47.300 --> 00:45:50.080 align:middle line:84% and just learn it anyway along with everything else. 00:45:50.080 --> 00:45:51.560 align:middle line:84% A meta principle in deep learning 00:45:51.560 --> 00:45:53.770 align:middle line:84% is that whenever you think you're 00:45:53.770 --> 00:45:55.520 align:middle line:84% making an ad hoc decision about something, 00:45:55.520 --> 00:45:58.220 align:middle line:84% like averaging a bunch of stuff, you should always stop and say, 00:45:58.220 --> 00:45:59.720 align:middle line:84% is there a better way to do it where 00:45:59.720 --> 00:46:01.920 align:middle line:84% it doesn't have to be ad hoc, where the right way is 00:46:01.920 --> 00:46:05.760 align:middle line:84% learnable from the data directly using backpropagation? 00:46:05.760 --> 00:46:06.740 align:middle line:90% There was a hand. 00:46:06.740 --> 00:46:08.400 align:middle line:90% Yeah? 00:46:08.400 --> 00:46:13.900 align:middle line:84% AUDIENCE: Is there a reason that you added the CLS at the start? 00:46:13.900 --> 00:46:14.780 align:middle line:90% Why not earlier? 00:46:14.780 --> 00:46:15.000 align:middle line:90% [INAUDIBLE] 00:46:15.000 --> 00:46:15.400 align:middle line:84% RAMA RAMAKRISHNAN: You can do it that way. 00:46:15.400 --> 00:46:17.317 align:middle line:84% AUDIENCE: At the end, is there any difference? 00:46:17.317 --> 00:46:19.800 align:middle line:84% RAMA RAMAKRISHNAN: The only thing to remember is that-- 00:46:19.800 --> 00:46:20.700 align:middle line:90% it's a good question. 00:46:20.700 --> 00:46:23.283 align:middle line:84% So different sentences are going to be of different length. 00:46:23.283 --> 00:46:24.700 align:middle line:84% So there might be short sentences. 00:46:24.700 --> 00:46:25.950 align:middle line:90% There might be long sentences. 00:46:25.950 --> 00:46:27.440 align:middle line:84% In particular, the short sentences 00:46:27.440 --> 00:46:30.185 align:middle line:90% are going to get padded. 00:46:30.185 --> 00:46:31.560 align:middle line:84% I remember I talked about padding 00:46:31.560 --> 00:46:33.400 align:middle line:90% to make it to fit to one length. 00:46:33.400 --> 00:46:35.200 align:middle line:84% So what internally the transformer will do 00:46:35.200 --> 00:46:37.980 align:middle line:84% is it will ignore all the padded tokens because it doesn't do it. 00:46:37.980 --> 00:46:38.860 align:middle line:90% It's just padding. 00:46:38.860 --> 00:46:40.460 align:middle line:84% It doesn't really matter for anything. 00:46:40.460 --> 00:46:42.360 align:middle line:84% So if you have the CLS at the very end, 00:46:42.360 --> 00:46:44.880 align:middle line:84% we have to have much more administrative bookkeeping 00:46:44.880 --> 00:46:48.200 align:middle line:84% to take everything but the last one, 00:46:48.200 --> 00:46:50.120 align:middle line:84% ignore it, and only do the last one. 00:46:50.120 --> 00:46:52.440 align:middle line:84% It's just much easier to stick it in the beginning. 00:46:52.440 --> 00:46:53.220 align:middle line:90% That's the reason. 00:46:53.220 --> 00:46:54.960 align:middle line:90% Yeah? 00:46:54.960 --> 00:46:57.520 align:middle line:84% AUDIENCE: What could be just a practical application of this? 00:46:57.520 --> 00:46:59.340 align:middle line:84% Would it be something like sentiment analysis-- 00:46:59.340 --> 00:46:59.560 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:46:59.560 --> 00:47:00.520 align:middle line:84% AUDIENCE: --like positive or negative? 00:47:00.520 --> 00:47:00.680 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:47:00.680 --> 00:47:02.640 align:middle line:84% So basically any kind of text comes in 00:47:02.640 --> 00:47:05.160 align:middle line:84% and you want to figure out some labeling problem, 00:47:05.160 --> 00:47:06.720 align:middle line:90% like a classification problem. 00:47:06.720 --> 00:47:09.840 align:middle line:84% The easiest example I could think of was sentiment. 00:47:09.840 --> 00:47:12.000 align:middle line:84% But you can imagine, for example, an email 00:47:12.000 --> 00:47:14.420 align:middle line:84% comes into a call center operation, 00:47:14.420 --> 00:47:16.920 align:middle line:84% and you want to take the email and automatically figure out, 00:47:16.920 --> 00:47:18.462 align:middle line:84% which department should I send it to. 00:47:18.462 --> 00:47:23.040 align:middle line:90% 00:47:23.040 --> 00:47:26.542 align:middle line:84% Now if the input data for a task is natural language text, 00:47:26.542 --> 00:47:28.000 align:middle line:84% we don't have to restrict ourselves 00:47:28.000 --> 00:47:30.990 align:middle line:84% to only the input training data we have. 00:47:30.990 --> 00:47:33.240 align:middle line:84% Wouldn't it be great to learn from all the text that's 00:47:33.240 --> 00:47:34.483 align:middle line:90% out there? 00:47:34.483 --> 00:47:36.400 align:middle line:84% So for example, to go back to that call center 00:47:36.400 --> 00:47:40.800 align:middle line:84% thing I just mentioned, let's say it's coming in English. 00:47:40.800 --> 00:47:43.880 align:middle line:84% The ability to take that English email and route it 00:47:43.880 --> 00:47:46.080 align:middle line:84% to one of 10 things, you shouldn't 00:47:46.080 --> 00:47:48.863 align:middle line:84% have to learn English just for your call center application. 00:47:48.863 --> 00:47:50.280 align:middle line:84% You should learn English generally 00:47:50.280 --> 00:47:51.760 align:middle line:90% and use it for other things. 00:47:51.760 --> 00:47:55.692 align:middle line:84% So why can't we just learn from all the text that's out there? 00:47:55.692 --> 00:47:57.400 align:middle line:84% And so that brings us to something called 00:47:57.400 --> 00:47:59.280 align:middle line:90% self-supervised learning. 00:47:59.280 --> 00:48:01.440 align:middle line:84% And the idea of self-supervised learning is this. 00:48:01.440 --> 00:48:03.680 align:middle line:84% So if you recall the transfer learning example 00:48:03.680 --> 00:48:08.220 align:middle line:84% from lecture four where we had ResNet, and we took ResNet, 00:48:08.220 --> 00:48:11.920 align:middle line:84% we chopped off the final thing, we made it headless, 00:48:11.920 --> 00:48:14.560 align:middle line:84% and then we attached that output of the headless ResNet 00:48:14.560 --> 00:48:17.300 align:middle line:84% to a little hidden layer and output, 00:48:17.300 --> 00:48:18.803 align:middle line:84% and we did the handbags and shoes. 00:48:18.803 --> 00:48:20.720 align:middle line:84% And you will recall that we were able to build 00:48:20.720 --> 00:48:22.595 align:middle line:84% a very good classifier for handbags and shoes 00:48:22.595 --> 00:48:24.280 align:middle line:90% with just 100 examples. 00:48:24.280 --> 00:48:27.440 align:middle line:84% So the question is, why was this so effective? 00:48:27.440 --> 00:48:29.120 align:middle line:90% Why was it so effective? 00:48:29.120 --> 00:48:32.880 align:middle line:84% And turns out the reason why any of this stuff actually works 00:48:32.880 --> 00:48:36.400 align:middle line:84% is because neural networks, they learn representations 00:48:36.400 --> 00:48:38.000 align:middle line:84% automatically when you train them. 00:48:38.000 --> 00:48:40.798 align:middle line:84% So what I mean by that is when you imagine a network, 00:48:40.798 --> 00:48:43.340 align:middle line:84% you feed in a bunch of stuff, it goes through all the layers, 00:48:43.340 --> 00:48:46.320 align:middle line:84% it comes out, you can think of each layer 00:48:46.320 --> 00:48:49.640 align:middle line:84% as transforming the raw input in some different alternate 00:48:49.640 --> 00:48:52.495 align:middle line:90% representation of the input. 00:48:52.495 --> 00:48:54.100 align:middle line:84% And these are called representations. 00:48:54.100 --> 00:48:56.400 align:middle line:84% That's actually a technical term. 00:48:56.400 --> 00:48:59.440 align:middle line:84% And so from this perspective, when you train a neural network, 00:48:59.440 --> 00:49:01.960 align:middle line:84% a deep network with lots of layers, what you're really 00:49:01.960 --> 00:49:07.560 align:middle line:84% learning is you're learning how to represent the input 00:49:07.560 --> 00:49:08.575 align:middle line:90% in many different ways-- 00:49:08.575 --> 00:49:10.200 align:middle line:84% each of these arrows is a different way 00:49:10.200 --> 00:49:11.420 align:middle line:90% of representing things-- 00:49:11.420 --> 00:49:14.113 align:middle line:84% plus you're learning a final regression model, 00:49:14.113 --> 00:49:16.780 align:middle line:84% either a linear regression model or a logistic regression model. 00:49:16.780 --> 00:49:18.363 align:middle line:84% Fundamentally, that's what's going on. 00:49:18.363 --> 00:49:20.340 align:middle line:84% Because the final layers tend to be sigmoid, 00:49:20.340 --> 00:49:22.280 align:middle line:90% softmax or just linear, right. 00:49:22.280 --> 00:49:23.580 align:middle line:90% So the final layer-- 00:49:23.580 --> 00:49:26.512 align:middle line:84% if you just look at this part alone, whatever is coming in 00:49:26.512 --> 00:49:28.720 align:middle line:84% is just going through essentially a linear regression 00:49:28.720 --> 00:49:30.380 align:middle line:84% model or a logistic regression model. 00:49:30.380 --> 00:49:31.450 align:middle line:90% That's it. 00:49:31.450 --> 00:49:33.450 align:middle line:84% So fundamentally you're learning representations 00:49:33.450 --> 00:49:35.598 align:middle line:90% and a final little model. 00:49:35.598 --> 00:49:37.890 align:middle line:84% But the reason why all these things work so much better 00:49:37.890 --> 00:49:40.290 align:middle line:84% than logistic regression is because those representations 00:49:40.290 --> 00:49:43.570 align:middle line:84% have learned all kinds of useful things about the input data. 00:49:43.570 --> 00:49:47.410 align:middle line:84% They have sort of' automatically feature-engineered for you. 00:49:47.410 --> 00:49:52.570 align:middle line:84% So from this perspective, you can imagine that each layer here 00:49:52.570 --> 00:49:53.670 align:middle line:90% is like an encoder. 00:49:53.670 --> 00:49:55.350 align:middle line:90% It encodes the input. 00:49:55.350 --> 00:49:56.510 align:middle line:90% The first layer encodes it. 00:49:56.510 --> 00:49:57.950 align:middle line:84% The first two layers encode something. 00:49:57.950 --> 00:49:59.190 align:middle line:84% The first three layers encode something, 00:49:59.190 --> 00:50:01.010 align:middle line:84% and so on and so forth, so a deep network 00:50:01.010 --> 00:50:03.090 align:middle line:90% contains many encoders. 00:50:03.090 --> 00:50:06.570 align:middle line:84% And so the question is, what do these representations actually 00:50:06.570 --> 00:50:07.370 align:middle line:90% embody? 00:50:07.370 --> 00:50:08.550 align:middle line:90% What do they capture? 00:50:08.550 --> 00:50:11.250 align:middle line:84% Is it specific knowledge about the particular problem 00:50:11.250 --> 00:50:14.210 align:middle line:90% that you train the network on? 00:50:14.210 --> 00:50:16.390 align:middle line:84% Or is it general knowledge about the input data? 00:50:16.390 --> 00:50:18.590 align:middle line:84% Because if it is general knowledge about the input, 00:50:18.590 --> 00:50:20.410 align:middle line:84% we can use it to solve other problems, 00:50:20.410 --> 00:50:23.290 align:middle line:84% unrelated problems, so is it specific knowledge 00:50:23.290 --> 00:50:24.350 align:middle line:90% or general knowledge. 00:50:24.350 --> 00:50:26.210 align:middle line:84% And it turns out they actually capture 00:50:26.210 --> 00:50:28.490 align:middle line:84% a lot of general knowledge about the input. 00:50:28.490 --> 00:50:31.890 align:middle line:84% And that's why you can get reuse out of them. 00:50:31.890 --> 00:50:34.050 align:middle line:84% You can reuse them for other unrelated things 00:50:34.050 --> 00:50:35.830 align:middle line:84% because they have captured general stuff. 00:50:35.830 --> 00:50:36.830 align:middle line:90% So if you look at this-- 00:50:36.830 --> 00:50:38.570 align:middle line:90% I think I've shown you before. 00:50:38.570 --> 00:50:41.410 align:middle line:84% If you look at a network that classifies everyday objects 00:50:41.410 --> 00:50:42.890 align:middle line:84% into a bunch of categories, it can 00:50:42.890 --> 00:50:45.307 align:middle line:84% learn all these little patterns in the beginning and later 00:50:45.307 --> 00:50:46.570 align:middle line:90% on and so on and so forth. 00:50:46.570 --> 00:50:48.230 align:middle line:84% And this is a face-detection network. 00:50:48.230 --> 00:50:51.890 align:middle line:84% It has learned how to identify little circles 00:50:51.890 --> 00:50:54.610 align:middle line:84% and edges and nose-like shapes and finally faces. 00:50:54.610 --> 00:50:57.130 align:middle line:84% So all these things are examples of representation 00:50:57.130 --> 00:50:59.796 align:middle line:84% learning interesting things about the input. 00:50:59.796 --> 00:51:02.410 align:middle line:84% So since these representations are capturing 00:51:02.410 --> 00:51:06.530 align:middle line:84% intrinsic aspects of the data, you can use it for other things. 00:51:06.530 --> 00:51:08.490 align:middle line:84% You can take a face detection neural network 00:51:08.490 --> 00:51:12.970 align:middle line:84% and reuse it for emotion detection, for instance. 00:51:12.970 --> 00:51:14.530 align:middle line:84% So the question is if we can somehow 00:51:14.530 --> 00:51:17.770 align:middle line:84% get an encoder that generates good representations 00:51:17.770 --> 00:51:19.638 align:middle line:84% for your input data, we can simply 00:51:19.638 --> 00:51:21.930 align:middle line:84% build a regression model with those as input and labels 00:51:21.930 --> 00:51:23.030 align:middle line:90% as output and be done. 00:51:23.030 --> 00:51:25.450 align:middle line:84% And this is exactly what we did with ResNet 00:51:25.450 --> 00:51:27.050 align:middle line:90% for handbags and shoes. 00:51:27.050 --> 00:51:28.730 align:middle line:84% We found a thing that had already 00:51:28.730 --> 00:51:32.210 align:middle line:84% been trained on similar everyday objects, everyday images. 00:51:32.210 --> 00:51:34.530 align:middle line:84% And the key insight here is that since we 00:51:34.530 --> 00:51:37.010 align:middle line:84% don't have to spend precious data on learning 00:51:37.010 --> 00:51:41.730 align:middle line:84% these good representations, we won't need as much labeled data 00:51:41.730 --> 00:51:45.690 align:middle line:84% in the first place because the pre-training used a lot of data, 00:51:45.690 --> 00:51:47.730 align:middle line:84% and you're piggybacking on that data. 00:51:47.730 --> 00:51:49.410 align:middle line:84% So in some sense, your training data 00:51:49.410 --> 00:51:51.090 align:middle line:84% is everything that the pre-trained model 00:51:51.090 --> 00:51:53.200 align:middle line:84% was trained on plus your little 200 examples. 00:51:53.200 --> 00:51:55.990 align:middle line:90% 00:51:55.990 --> 00:51:56.950 align:middle line:90% So this is what we did. 00:51:56.950 --> 00:51:59.570 align:middle line:84% We used headless ResNet as an encoder that can take raw input 00:51:59.570 --> 00:52:02.170 align:middle line:84% and transform it into useful representations. 00:52:02.170 --> 00:52:03.750 align:middle line:90% This is what we did. 00:52:03.750 --> 00:52:06.530 align:middle line:84% So the general approach is that you find a deep neural network 00:52:06.530 --> 00:52:09.610 align:middle line:84% built on similar inputs but different outputs, 00:52:09.610 --> 00:52:11.850 align:middle line:84% and then you basically grab maybe 00:52:11.850 --> 00:52:15.410 align:middle line:84% the penultimate representation or the one before that. 00:52:15.410 --> 00:52:17.170 align:middle line:90% Then you chop off the head. 00:52:17.170 --> 00:52:21.730 align:middle line:84% You attach your own output head, train the whole thing-- 00:52:21.730 --> 00:52:24.530 align:middle line:84% just the final layer or train the whole thing if you want. 00:52:24.530 --> 00:52:26.630 align:middle line:84% This is like the playbook we followed for ResNet. 00:52:26.630 --> 00:52:29.290 align:middle line:84% The same thing works for all kinds of other data types 00:52:29.290 --> 00:52:30.250 align:middle line:90% as well. 00:52:30.250 --> 00:52:33.410 align:middle line:84% So now to build such a model, we need labeled data. 00:52:33.410 --> 00:52:36.010 align:middle line:84% We were lucky because ResNet was actually trained on ImageNet 00:52:36.010 --> 00:52:38.070 align:middle line:84% data, which is like 1 million images, 00:52:38.070 --> 00:52:40.330 align:middle line:84% each of which labeled into 1,000 categories, 00:52:40.330 --> 00:52:42.850 align:middle line:90% which is very convenient for us. 00:52:42.850 --> 00:52:47.450 align:middle line:84% But what if you want to build a generally useful model for text 00:52:47.450 --> 00:52:49.592 align:middle line:90% data? 00:52:49.592 --> 00:52:51.550 align:middle line:84% Clearly, we need to collect a lot of text data, 00:52:51.550 --> 00:52:52.810 align:middle line:84% but that's no problem because the internet 00:52:52.810 --> 00:52:53.890 align:middle line:90% is full of text data. 00:52:53.890 --> 00:52:55.390 align:middle line:84% We can easily scrape the internet. 00:52:55.390 --> 00:52:56.830 align:middle line:90% We can just download Wikipedia. 00:52:56.830 --> 00:52:58.350 align:middle line:90% So that's not a problem. 00:52:58.350 --> 00:52:59.850 align:middle line:84% The problem is something else, which 00:52:59.850 --> 00:53:04.930 align:middle line:84% is that, how do we define an input label for a piece of text? 00:53:04.930 --> 00:53:07.590 align:middle line:84% So for an input sentence, what should the output label be? 00:53:07.590 --> 00:53:08.843 align:middle line:90% That's the key question. 00:53:08.843 --> 00:53:10.510 align:middle line:84% Because if you can answer this question, 00:53:10.510 --> 00:53:12.135 align:middle line:84% you can just pre-train all these things 00:53:12.135 --> 00:53:14.214 align:middle line:90% on all kinds of text data. 00:53:14.214 --> 00:53:17.412 align:middle line:84% So a beautiful idea for doing this 00:53:17.412 --> 00:53:18.870 align:middle line:84% is called self-supervised learning. 00:53:18.870 --> 00:53:21.370 align:middle line:84% And the key idea is that you take your input, whatever 00:53:21.370 --> 00:53:25.410 align:middle line:84% the input is, you take a small part of the input 00:53:25.410 --> 00:53:27.970 align:middle line:84% and just remove it, and then ask your network 00:53:27.970 --> 00:53:31.970 align:middle line:84% to fill in the blanks from everything else. 00:53:31.970 --> 00:53:33.737 align:middle line:84% So this is called masking, and it's 00:53:33.737 --> 00:53:36.070 align:middle line:84% just one of many techniques in self-supervised learning, 00:53:36.070 --> 00:53:37.770 align:middle line:90% but this is very commonly used. 00:53:37.770 --> 00:53:40.650 align:middle line:90% So this is the original input. 00:53:40.650 --> 00:53:42.470 align:middle line:84% And then you take it and then you just 00:53:42.470 --> 00:53:43.930 align:middle line:84% take this thing in the middle here 00:53:43.930 --> 00:53:47.330 align:middle line:84% randomly and zero it out or mask it. 00:53:47.330 --> 00:53:51.270 align:middle line:84% And so this incomplete input is your now new input, 00:53:51.270 --> 00:53:56.690 align:middle line:84% and the thing that you took out becomes your fake label. 00:53:56.690 --> 00:54:00.210 align:middle line:84% So you can almost imagine if you're baking donuts, 00:54:00.210 --> 00:54:02.530 align:middle line:84% you make a donut, and then you punch a hole 00:54:02.530 --> 00:54:04.250 align:middle line:90% in the middle of the donut. 00:54:04.250 --> 00:54:07.110 align:middle line:84% The donut with the hole is your new input. 00:54:07.110 --> 00:54:08.900 align:middle line:90% The munchkin is the label. 00:54:08.900 --> 00:54:11.418 align:middle line:90% 00:54:11.418 --> 00:54:13.210 align:middle line:84% Am I making everybody hungry at this point? 00:54:13.210 --> 00:54:15.130 align:middle line:90% [LAUGHTER] 00:54:15.130 --> 00:54:17.670 align:middle line:90% So once you do that, no problem. 00:54:17.670 --> 00:54:18.630 align:middle line:90% You have an input. 00:54:18.630 --> 00:54:19.990 align:middle line:90% You have labels. 00:54:19.990 --> 00:54:25.450 align:middle line:84% You just train a neural network to essentially predict those-- 00:54:25.450 --> 00:54:28.490 align:middle line:90% to basically fill in the blanks. 00:54:28.490 --> 00:54:31.690 align:middle line:84% And so for example, if you take a sentence like, 00:54:31.690 --> 00:54:34.290 align:middle line:84% "The Sloan School's mission," you can just go in there 00:54:34.290 --> 00:54:38.232 align:middle line:84% and just knock out randomly a bunch of words like-- 00:54:38.232 --> 00:54:40.690 align:middle line:84% the ones I'm knocking out, I'm just putting the word "mask" 00:54:40.690 --> 00:54:42.850 align:middle line:84% in it just to show what I'm doing. 00:54:42.850 --> 00:54:45.910 align:middle line:84% And then given this sentence, it will 00:54:45.910 --> 00:54:47.820 align:middle line:84% try to fill in the blanks with actual words. 00:54:47.820 --> 00:54:51.810 align:middle line:90% 00:54:51.810 --> 00:54:53.088 align:middle line:90% So now for the amazing part. 00:54:53.088 --> 00:54:55.130 align:middle line:84% In the process of learning to fill in the blanks, 00:54:55.130 --> 00:54:58.050 align:middle line:84% the network learns a really good representation 00:54:58.050 --> 00:55:00.370 align:middle line:84% of the kind of input data it's seeing. 00:55:00.370 --> 00:55:02.610 align:middle line:84% And it makes sense because if I give you 00:55:02.610 --> 00:55:04.690 align:middle line:84% a sentence with a few missing blanks 00:55:04.690 --> 00:55:07.425 align:middle line:84% and you're able to very successfully fill in the blanks, 00:55:07.425 --> 00:55:09.050 align:middle line:84% you have learned a whole bunch of stuff 00:55:09.050 --> 00:55:11.650 align:middle line:84% about the world to be able to do that. 00:55:11.650 --> 00:55:13.930 align:middle line:84% If I say the capital of France is dash, 00:55:13.930 --> 00:55:16.770 align:middle line:84% and you're like, Paris, OK, how did you know that? 00:55:16.770 --> 00:55:18.133 align:middle line:90% It's sort of like that. 00:55:18.133 --> 00:55:19.550 align:middle line:84% By learning to fill in the blanks, 00:55:19.550 --> 00:55:22.470 align:middle line:84% you really have to learn how all these things work, 00:55:22.470 --> 00:55:24.330 align:middle line:84% all the connections between various words 00:55:24.330 --> 00:55:27.250 align:middle line:90% and so on and so forth. 00:55:27.250 --> 00:55:30.430 align:middle line:84% So what you can do is once we build such a model, 00:55:30.430 --> 00:55:33.270 align:middle line:84% we can just extract an encoder from it, 00:55:33.270 --> 00:55:37.050 align:middle line:84% and then we'll fine-tune like we did for transfer learning. 00:55:37.050 --> 00:55:38.970 align:middle line:84% But this is how you build a generic, 00:55:38.970 --> 00:55:43.530 align:middle line:84% a generic pre-trained model on unlabeled data. 00:55:43.530 --> 00:55:46.090 align:middle line:84% And so we can use a transformer encoder 00:55:46.090 --> 00:55:47.835 align:middle line:84% to build this whole thing in the middle. 00:55:47.835 --> 00:55:49.210 align:middle line:84% Because remember, the transformer 00:55:49.210 --> 00:55:52.170 align:middle line:84% can take any sentence and give you the same-size sentence 00:55:52.170 --> 00:55:54.850 align:middle line:84% back along with predictions for everything, 00:55:54.850 --> 00:55:57.250 align:middle line:84% so we can just have it take this thing in and ask 00:55:57.250 --> 00:55:59.230 align:middle line:84% it to just predict all the missing words here. 00:55:59.230 --> 00:56:03.090 align:middle line:90% 00:56:03.090 --> 00:56:06.210 align:middle line:84% So to put it in other words, masked self-supervised learning 00:56:06.210 --> 00:56:09.410 align:middle line:84% is just a sequence labeling problem. 00:56:09.410 --> 00:56:11.850 align:middle line:84% So basically this is the sequence that comes in, 00:56:11.850 --> 00:56:13.890 align:middle line:84% and then you tell the transformer-- 00:56:13.890 --> 00:56:15.338 align:middle line:84% and you get all these embeddings. 00:56:15.338 --> 00:56:16.630 align:middle line:90% It goes through all that stuff. 00:56:16.630 --> 00:56:18.910 align:middle line:84% You really don't care about these outputs. 00:56:18.910 --> 00:56:22.290 align:middle line:84% But wherever the word "mask" went in in the input, 00:56:22.290 --> 00:56:25.190 align:middle line:84% you basically try to get it to-- the right answer is, 00:56:25.190 --> 00:56:28.060 align:middle line:84% for example, the word "mission," and that is the right answer. 00:56:28.060 --> 00:56:29.310 align:middle line:90% This is the right answer here. 00:56:29.310 --> 00:56:31.893 align:middle line:84% And then you take these right answers, create a loss function, 00:56:31.893 --> 00:56:35.530 align:middle line:84% and do a backprop, and boom, you're done. 00:56:35.530 --> 00:56:37.910 align:middle line:84% Inputs right answers and you're in business. 00:56:37.910 --> 00:56:39.130 align:middle line:90% That's it. 00:56:39.130 --> 00:56:41.770 align:middle line:84% Now if we pre-train a transformer model like this 00:56:41.770 --> 00:56:44.010 align:middle line:84% on massive amounts of English text-- 00:56:44.010 --> 00:56:45.610 align:middle line:90% let's say we did that-- 00:56:45.610 --> 00:56:47.770 align:middle line:90% we get something called BERT. 00:56:47.770 --> 00:56:50.210 align:middle line:84% BERT is a very famous transformer model. 00:56:50.210 --> 00:56:52.390 align:middle line:84% And BERT was the first model, actually, 00:56:52.390 --> 00:56:56.450 align:middle line:84% that Google used to upgrade its search in 2019. 00:56:56.450 --> 00:57:00.090 align:middle line:84% The Brazil visa example, you may recall from earlier lectures, 00:57:00.090 --> 00:57:01.710 align:middle line:90% that uses BERT under the hood. 00:57:01.710 --> 00:57:04.600 align:middle line:90% 00:57:04.600 --> 00:57:06.850 align:middle line:84% Now I just want to show you-- because you can actually 00:57:06.850 --> 00:57:08.433 align:middle line:84% read the BERT paper and it'll actually 00:57:08.433 --> 00:57:10.050 align:middle line:84% make sense to you now based on what 00:57:10.050 --> 00:57:11.510 align:middle line:90% you've learned in this class. 00:57:11.510 --> 00:57:12.530 align:middle line:90% Look at this. 00:57:12.530 --> 00:57:15.290 align:middle line:84% BERT's model architecture is a multi-layer bidirectional 00:57:15.290 --> 00:57:16.250 align:middle line:90% transformer encoder. 00:57:16.250 --> 00:57:17.930 align:middle line:90% OK, transformer encoder. 00:57:17.930 --> 00:57:21.330 align:middle line:84% We denote the number of layers, transformer blocks, as L, 00:57:21.330 --> 00:57:24.250 align:middle line:84% the hidden size as H, and the number of attention heads as A. 00:57:24.250 --> 00:57:26.690 align:middle line:90% And how much is that? 00:57:26.690 --> 00:57:36.530 align:middle line:84% We want H as 768, which means that the embedding sizes are 768 00:57:36.530 --> 00:57:39.570 align:middle line:84% and the hidden feedforward layer is four times as much, 00:57:39.570 --> 00:57:41.590 align:middle line:90% so it's 4,096-- 00:57:41.590 --> 00:57:45.210 align:middle line:84% sorry, 4,096 is the feedforward layer. 00:57:45.210 --> 00:57:47.430 align:middle line:90% The embeddings are 768. 00:57:47.430 --> 00:57:49.570 align:middle line:84% And you can see there are two BERT models here. 00:57:49.570 --> 00:57:52.470 align:middle line:84% This one has 12 transformer blocks. 00:57:52.470 --> 00:57:54.300 align:middle line:84% This one has 24 transformer blocks. 00:57:54.300 --> 00:57:56.930 align:middle line:90% 00:57:56.930 --> 00:57:58.430 align:middle line:84% So you can actually read this paper. 00:57:58.430 --> 00:58:00.193 align:middle line:84% You can actually relate it to exactly what 00:58:00.193 --> 00:58:01.110 align:middle line:90% we discussed in class. 00:58:01.110 --> 00:58:02.423 align:middle line:90% It'll all make sense. 00:58:02.423 --> 00:58:05.090 align:middle line:84% Bidirectional essentially means that the words can pay attention 00:58:05.090 --> 00:58:07.410 align:middle line:84% to every other word in the sentence. 00:58:07.410 --> 00:58:10.170 align:middle line:84% And as we will see on Monday, you 00:58:10.170 --> 00:58:12.928 align:middle line:84% have another transformer thing called a causal transformer, 00:58:12.928 --> 00:58:14.970 align:middle line:84% in which you only pay attention to the words that 00:58:14.970 --> 00:58:17.850 align:middle line:84% came before you, not the ones after you. 00:58:17.850 --> 00:58:20.030 align:middle line:84% So bidirectional means all words are seen. 00:58:20.030 --> 00:58:22.850 align:middle line:90% 00:58:22.850 --> 00:58:25.210 align:middle line:84% So what we do is, remember, we said 00:58:25.210 --> 00:58:27.150 align:middle line:84% to solve sequence classification, 00:58:27.150 --> 00:58:30.650 align:middle line:84% you can add a little token at the beginning and then boom, 00:58:30.650 --> 00:58:31.990 align:middle line:90% use it for classification. 00:58:31.990 --> 00:58:34.690 align:middle line:84% As it turns out, very conveniently 00:58:34.690 --> 00:58:38.030 align:middle line:84% for us, the people who built BERT, when they trained BERT, 00:58:38.030 --> 00:58:41.830 align:middle line:84% they just used the CLS business during training, 00:58:41.830 --> 00:58:44.148 align:middle line:84% so it's actually available for us out of the box. 00:58:44.148 --> 00:58:46.190 align:middle line:84% So when you use BERT for sequence classification, 00:58:46.190 --> 00:58:47.710 align:middle line:84% you don't even have to do any surgery on it. 00:58:47.710 --> 00:58:49.710 align:middle line:84% It just gives you the class token automatically, 00:58:49.710 --> 00:58:51.490 align:middle line:90% which is very convenient. 00:58:51.490 --> 00:58:54.450 align:middle line:84% And you can also use it for sequence labeling as well. 00:58:54.450 --> 00:58:57.410 align:middle line:84% So for sequence classification and sequence labeling, 00:58:57.410 --> 00:58:59.713 align:middle line:84% BERT is actually usually a really good starting point. 00:58:59.713 --> 00:59:01.130 align:middle line:84% And in particular, there have been 00:59:01.130 --> 00:59:02.922 align:middle line:84% lots of improvements and variations of BERT 00:59:02.922 --> 00:59:03.710 align:middle line:90% over the years. 00:59:03.710 --> 00:59:05.230 align:middle line:84% And if you're curious about this, 00:59:05.230 --> 00:59:07.870 align:middle line:84% there's a thing called The Sentence Transformers library, 00:59:07.870 --> 00:59:10.850 align:middle line:84% which has got a whole bunch of BERT-related code and resources 00:59:10.850 --> 00:59:14.290 align:middle line:84% that you can use to do things out of the box. 00:59:14.290 --> 00:59:19.930 align:middle line:84% OK, this is a bit of a word wall. 00:59:19.930 --> 00:59:22.450 align:middle line:84% So to solve any of these problems, classification 00:59:22.450 --> 00:59:24.510 align:middle line:84% or labeling, where the input is natural language, 00:59:24.510 --> 00:59:26.630 align:middle line:84% we can obviously use a model like BERT, 00:59:26.630 --> 00:59:29.130 align:middle line:84% label a few hundred examples, attach the right final layers, 00:59:29.130 --> 00:59:31.490 align:middle line:84% and fine-tune it, like we did for the ResNet. 00:59:31.490 --> 00:59:35.850 align:middle line:84% But if your problem is like a standard NLP problem, 00:59:35.850 --> 00:59:37.957 align:middle line:84% you don't even have to do that because people, 00:59:37.957 --> 00:59:40.290 align:middle line:84% for these standard tasks, they've already pre-trained it 00:59:40.290 --> 00:59:42.393 align:middle line:90% on those standard tasks. 00:59:42.393 --> 00:59:43.810 align:middle line:84% And so you can do all these things 00:59:43.810 --> 00:59:47.370 align:middle line:84% without any fine-tuning at all, literally out of the box. 00:59:47.370 --> 00:59:50.150 align:middle line:84% And so there are many hubs which have these pre-trained models. 00:59:50.150 --> 00:59:53.490 align:middle line:84% But perhaps the biggest one is the Hugging Face Hub. 00:59:53.490 --> 00:59:54.930 align:middle line:90% And I checked last night. 00:59:54.930 --> 00:59:58.930 align:middle line:90% It has 520,000 models available. 00:59:58.930 --> 01:00:01.630 align:middle line:84% I think if I recall, last year when I taught HODL, 01:00:01.630 --> 01:00:04.250 align:middle line:84% I think the number was a lot smaller, maybe 50,000, 01:00:04.250 --> 01:00:08.133 align:middle line:84% so it's growing really, really fast. 01:00:08.133 --> 01:00:09.800 align:middle line:84% Let's just switch to Hugging Face Colab. 01:00:09.800 --> 01:00:15.170 align:middle line:90% 01:00:15.170 --> 01:00:16.850 align:middle line:90% So Hugging Face. 01:00:16.850 --> 01:00:21.690 align:middle line:84% How many of you are familiar with Hugging Face? 01:00:21.690 --> 01:00:23.670 align:middle line:90% OK, that's good. 01:00:23.670 --> 01:00:26.210 align:middle line:84% So for the others, basically, you 01:00:26.210 --> 01:00:27.930 align:middle line:84% have a whole bunch of pre-trained models 01:00:27.930 --> 01:00:28.857 align:middle line:90% on Hugging Face. 01:00:28.857 --> 01:00:30.690 align:middle line:84% You actually have a lot of data sets you can 01:00:30.690 --> 01:00:33.810 align:middle line:90% work with for your own tasks. 01:00:33.810 --> 01:00:36.690 align:middle line:84% There are lots of people demoing what they have built 01:00:36.690 --> 01:00:37.857 align:middle line:90% in this thing called Spaces. 01:00:37.857 --> 01:00:40.232 align:middle line:84% And of course, there is a lot of documentation and so on. 01:00:40.232 --> 01:00:41.610 align:middle line:90% So the thing you can do is-- 01:00:41.610 --> 01:00:44.068 align:middle line:84% what they have done is they have organized all these models 01:00:44.068 --> 01:00:46.025 align:middle line:84% by the kind of task you can use them for. 01:00:46.025 --> 01:00:48.650 align:middle line:84% So you can see here, there are a whole bunch of computer vision 01:00:48.650 --> 01:00:50.330 align:middle line:90% tasks that you can use them for. 01:00:50.330 --> 01:00:52.038 align:middle line:84% There's a whole bunch of natural language 01:00:52.038 --> 01:00:55.770 align:middle line:84% tasks like text classification, feature extraction, this 01:00:55.770 --> 01:00:58.477 align:middle line:84% and that, lots of interesting examples here. 01:00:58.477 --> 01:01:00.810 align:middle line:84% And so what you do is you just literally can go in there 01:01:00.810 --> 01:01:02.810 align:middle line:84% and say, OK, I want to do a text classification. 01:01:02.810 --> 01:01:05.010 align:middle line:84% You hit it, and then it tells you all the models 01:01:05.010 --> 01:01:05.550 align:middle line:90% that are available. 01:01:05.550 --> 01:01:07.090 align:middle line:84% Turns out there are 50,000 models just for text 01:01:07.090 --> 01:01:08.050 align:middle line:90% classification. 01:01:08.050 --> 01:01:10.303 align:middle line:84% And you can look at, OK, which is most downloaded 01:01:10.303 --> 01:01:11.970 align:middle line:84% or which is the most liked, and then you 01:01:11.970 --> 01:01:14.220 align:middle line:84% can just use them as a starting point for whatever you 01:01:14.220 --> 01:01:16.260 align:middle line:90% want to do. 01:01:16.260 --> 01:01:17.360 align:middle line:90% So that is Hugging Face. 01:01:17.360 --> 01:01:20.900 align:middle line:84% And so the way you do Hugging Face is-- 01:01:20.900 --> 01:01:22.000 align:middle line:90% I'm just connecting it. 01:01:22.000 --> 01:01:24.802 align:middle line:90% 01:01:24.802 --> 01:01:27.260 align:middle line:84% If you have a problem which the input is a natural language 01:01:27.260 --> 01:01:28.620 align:middle line:84% text, the first question you have to ask 01:01:28.620 --> 01:01:29.840 align:middle line:90% is, is it standard or not. 01:01:29.840 --> 01:01:31.080 align:middle line:90% Is it a standard task or not. 01:01:31.080 --> 01:01:34.400 align:middle line:84% If it's a standard task, do not reinvent the wheel. 01:01:34.400 --> 01:01:36.680 align:middle line:84% This thing will usually work pretty well. 01:01:36.680 --> 01:01:39.700 align:middle line:84% So here we will use this thing called 01:01:39.700 --> 01:01:41.460 align:middle line:84% the Transformers Library from Hugging 01:01:41.460 --> 01:01:43.740 align:middle line:84% Face, in particular, the pipeline function, 01:01:43.740 --> 01:01:46.032 align:middle line:84% to demonstrate quickly how to do this thing. 01:01:46.032 --> 01:01:47.740 align:middle line:84% Fortunately, this library as of this year 01:01:47.740 --> 01:01:49.960 align:middle line:84% is pre-installed in Colab, so we don't have to install it. 01:01:49.960 --> 01:01:51.543 align:middle line:84% We can just start using it right away. 01:01:51.543 --> 01:01:54.540 align:middle line:84% So we'll take this example where you have a bunch of text which 01:01:54.540 --> 01:01:59.303 align:middle line:84% says, "Dear Amazon, Last week I ordered an Optimus Prime action 01:01:59.303 --> 01:02:00.720 align:middle line:84% figure from your store in Germany. 01:02:00.720 --> 01:02:01.880 align:middle line:84% Unfortunately, when I opened the package, 01:02:01.880 --> 01:02:04.940 align:middle line:84% I discovered to my horror that I had been sent an action figure 01:02:04.940 --> 01:02:06.000 align:middle line:90% of Megatron instead." 01:02:06.000 --> 01:02:09.020 align:middle line:84% Can you imagine that person's sheer distress at this? 01:02:09.020 --> 01:02:11.520 align:middle line:84% "So as a lifelong enemy of the Decepticons, 01:02:11.520 --> 01:02:13.300 align:middle line:84% I hope you can understand my dilemma. 01:02:13.300 --> 01:02:15.740 align:middle line:84% So to resolve the issue, I demand an exchange. 01:02:15.740 --> 01:02:17.040 align:middle line:90% Enclosed are copies. 01:02:17.040 --> 01:02:18.280 align:middle line:90% Expect to hear from you soon. 01:02:18.280 --> 01:02:22.320 align:middle line:90% Sincerely, Bumblebee." 01:02:22.320 --> 01:02:25.625 align:middle line:84% They should come up with a better name for this example. 01:02:25.625 --> 01:02:27.000 align:middle line:84% Cool, so that's the text we have. 01:02:27.000 --> 01:02:28.366 align:middle line:90% So we import the-- 01:02:28.366 --> 01:02:31.020 align:middle line:84% this pipeline function is the one that basically gives you 01:02:31.020 --> 01:02:33.940 align:middle line:84% the ability to out-of-the-box start using it without any 01:02:33.940 --> 01:02:35.700 align:middle line:90% pre-training, nothing like that. 01:02:35.700 --> 01:02:38.740 align:middle line:90% So we download this thing. 01:02:38.740 --> 01:02:39.240 align:middle line:90% Oh, wow. 01:02:39.240 --> 01:02:41.020 align:middle line:90% I got an A100 today. 01:02:41.020 --> 01:02:42.840 align:middle line:90% That happens very rarely. 01:02:42.840 --> 01:02:44.500 align:middle line:90% Sorry. 01:02:44.500 --> 01:02:47.900 align:middle line:84% So here let's say you want to classify that text You just want 01:02:47.900 --> 01:02:49.340 align:middle line:90% to classify it for sentiment. 01:02:49.340 --> 01:02:53.100 align:middle line:84% You literally go in there and say pipeline text 01:02:53.100 --> 01:02:56.300 align:middle line:84% classification-- that's the task you want the pipeline to do 01:02:56.300 --> 01:02:57.300 align:middle line:90% for you-- 01:02:57.300 --> 01:02:59.580 align:middle line:84% and you create a classifier It's going 01:02:59.580 --> 01:03:04.260 align:middle line:84% to download a bunch of stuff and so on and so forth. 01:03:04.260 --> 01:03:06.860 align:middle line:84% The first time it just takes time to download. 01:03:06.860 --> 01:03:08.692 align:middle line:84% And then you literally take the text 01:03:08.692 --> 01:03:10.900 align:middle line:84% you have here, and then run it through the classifier 01:03:10.900 --> 01:03:13.580 align:middle line:84% as if it's just a little function. 01:03:13.580 --> 01:03:16.260 align:middle line:90% You get some outputs. 01:03:16.260 --> 01:03:19.500 align:middle line:84% Actually, let's just do this way. 01:03:19.500 --> 01:03:20.100 align:middle line:90% Negative. 01:03:20.100 --> 01:03:22.380 align:middle line:84% Sentiment is negative with a 90% probability. 01:03:22.380 --> 01:03:25.740 align:middle line:90% Pretty good, right? 01:03:25.740 --> 01:03:27.120 align:middle line:90% Sentiment classification solved. 01:03:27.120 --> 01:03:29.660 align:middle line:84% So we'll try a few different examples. 01:03:29.660 --> 01:03:30.482 align:middle line:90% "I hated the movie. 01:03:30.482 --> 01:03:32.440 align:middle line:84% If I said I loved the movie, I would be lying." 01:03:32.440 --> 01:03:33.620 align:middle line:90% OK, that's a little tricky. 01:03:33.620 --> 01:03:34.800 align:middle line:90% "The movie left me speechless. 01:03:34.800 --> 01:03:35.300 align:middle line:90% Incredible." 01:03:35.300 --> 01:03:38.060 align:middle line:84% And then I had to add this last thing here last night. 01:03:38.060 --> 01:03:41.427 align:middle line:84% "Almost but not quite entirely unlike anything good I've seen." 01:03:41.427 --> 01:03:42.760 align:middle line:90% That's not original, by the way. 01:03:42.760 --> 01:03:44.180 align:middle line:84% People who have read Douglas Adams will 01:03:44.180 --> 01:03:46.940 align:middle line:84% know this famous sentence about somebody drinking some beverage 01:03:46.940 --> 01:03:50.420 align:middle line:84% and saying it's almost but not quite entirely unlike tea. 01:03:50.420 --> 01:03:51.680 align:middle line:90% So I was inspired by that. 01:03:51.680 --> 01:03:53.097 align:middle line:84% So anyway, we'll see what happens. 01:03:53.097 --> 01:03:56.100 align:middle line:90% 01:03:56.100 --> 01:03:58.820 align:middle line:90% All right, put it in there. 01:03:58.820 --> 01:04:00.447 align:middle line:90% So negative. 01:04:00.447 --> 01:04:01.280 align:middle line:90% "I hated the movie." 01:04:01.280 --> 01:04:01.760 align:middle line:90% OK, fine. 01:04:01.760 --> 01:04:02.960 align:middle line:84% "If I said I loved the movie, I'd be lying." 01:04:02.960 --> 01:04:03.460 align:middle line:90% Negative. 01:04:03.460 --> 01:04:04.940 align:middle line:90% "The movie left me speechless." 01:04:04.940 --> 01:04:07.862 align:middle line:84% It says it's negative, but it could go either way, right? 01:04:07.862 --> 01:04:09.820 align:middle line:84% A good classifier would probably have given you 01:04:09.820 --> 01:04:11.780 align:middle line:84% a probability around the 50% mark because it's 01:04:11.780 --> 01:04:13.700 align:middle line:90% right on the fence. 01:04:13.700 --> 01:04:14.260 align:middle line:90% "Incredible." 01:04:14.260 --> 01:04:15.360 align:middle line:90% It's positive. 01:04:15.360 --> 01:04:17.540 align:middle line:84% And then it got fooled by my crazy long sentence, 01:04:17.540 --> 01:04:19.620 align:middle line:90% and it says it's positive. 01:04:19.620 --> 01:04:21.320 align:middle line:90% Now that's classification. 01:04:21.320 --> 01:04:22.760 align:middle line:90% Here is one other quick example. 01:04:22.760 --> 01:04:25.120 align:middle line:84% So you can actually give it a piece of text. 01:04:25.120 --> 01:04:28.260 align:middle line:84% For example, you can take a Reuters news story. 01:04:28.260 --> 01:04:31.983 align:middle line:84% You can feed it and say, extract all the company names from it. 01:04:31.983 --> 01:04:34.400 align:middle line:84% Extract company names, people names, and things like that. 01:04:34.400 --> 01:04:36.460 align:middle line:84% It's called named entity extraction. 01:04:36.460 --> 01:04:40.380 align:middle line:84% And back in the day, people would 01:04:40.380 --> 01:04:43.220 align:middle line:84% hand-build painstakingly all these very complex systems 01:04:43.220 --> 01:04:45.480 align:middle line:90% to do named entity extraction. 01:04:45.480 --> 01:04:46.840 align:middle line:90% Now it's just a pipeline away. 01:04:46.840 --> 01:04:50.500 align:middle line:84% So you can take this thing and you can say, create a pipeline 01:04:50.500 --> 01:04:52.740 align:middle line:90% for ner, named extraction. 01:04:52.740 --> 01:04:54.640 align:middle line:84% For any particular task that you're using, 01:04:54.640 --> 01:04:56.390 align:middle line:84% there might be a few additional parameters 01:04:56.390 --> 01:04:58.940 align:middle line:84% you can set as a part of the configuration. 01:04:58.940 --> 01:05:00.640 align:middle line:90% So we download this pipeline. 01:05:00.640 --> 01:05:09.300 align:middle line:90% 01:05:09.300 --> 01:05:10.120 align:middle line:90% Perfect. 01:05:10.120 --> 01:05:12.460 align:middle line:90% And then we run the output. 01:05:12.460 --> 01:05:14.160 align:middle line:90% So it says, OK, good. 01:05:14.160 --> 01:05:20.740 align:middle line:84% Amazon is an organization and Germany is a location, LOC, 01:05:20.740 --> 01:05:21.480 align:middle line:90% which is nice. 01:05:21.480 --> 01:05:22.860 align:middle line:84% So these things have a standard vocabulary 01:05:22.860 --> 01:05:24.080 align:middle line:84% as to ORG, LOC, things like that, 01:05:24.080 --> 01:05:25.940 align:middle line:84% which you can read up in the documentation. 01:05:25.940 --> 01:05:27.660 align:middle line:90% And then Bumblebee is a person. 01:05:27.660 --> 01:05:31.540 align:middle line:84% And then boy, all the Optimus Prime Transformer stuff 01:05:31.540 --> 01:05:32.980 align:middle line:90% is all-- it got fooled. 01:05:32.980 --> 01:05:35.860 align:middle line:84% It thinks Optimus Prime is miscellaneous, 01:05:35.860 --> 01:05:37.960 align:middle line:84% Decept is miscellaneous, and so on and so forth. 01:05:37.960 --> 01:05:38.600 align:middle line:90% But you get the idea. 01:05:38.600 --> 01:05:40.900 align:middle line:84% You can take standard things like Reuters news stories 01:05:40.900 --> 01:05:41.400 align:middle line:90% and so on. 01:05:41.400 --> 01:05:42.020 align:middle line:90% You can just feed it, boop. 01:05:42.020 --> 01:05:44.680 align:middle line:84% You can get a very good entity extraction right out of the bat. 01:05:44.680 --> 01:05:46.480 align:middle line:84% And once you get these entities extracted, 01:05:46.480 --> 01:05:48.660 align:middle line:84% then you can put them into a nice structured data 01:05:48.660 --> 01:05:50.260 align:middle line:84% table like a database, and then you 01:05:50.260 --> 01:05:54.060 align:middle line:84% can run traditional machine learning on it. 01:05:54.060 --> 01:05:56.660 align:middle line:84% And then I had, I think, a few more examples 01:05:56.660 --> 01:05:58.980 align:middle line:90% of question answering. 01:05:58.980 --> 01:06:00.818 align:middle line:90% Actually, let's just try that. 01:06:00.818 --> 01:06:03.360 align:middle line:84% You can actually give it a thing and ask a question about it, 01:06:03.360 --> 01:06:06.140 align:middle line:84% and it can actually give you the answer, which 01:06:06.140 --> 01:06:08.740 align:middle line:84% gets into the causal transformer thing 01:06:08.740 --> 01:06:10.940 align:middle line:84% that we're going to see on Monday, which builds up 01:06:10.940 --> 01:06:13.485 align:middle line:84% into large language models because you obviously can give 01:06:13.485 --> 01:06:14.860 align:middle line:84% something, you can give a passage 01:06:14.860 --> 01:06:16.340 align:middle line:84% to ChatGPT and ask a question and ask 01:06:16.340 --> 01:06:17.360 align:middle line:90% it to give you an answer. 01:06:17.360 --> 01:06:18.568 align:middle line:90% So it's really in that thing. 01:06:18.568 --> 01:06:21.600 align:middle line:84% But just for fun, let's just do that to see if it's any good. 01:06:21.600 --> 01:06:24.660 align:middle line:90% 01:06:24.660 --> 01:06:26.340 align:middle line:90% What does the customer want? 01:06:26.340 --> 01:06:28.740 align:middle line:84% And the output is an exchange of Megatron. 01:06:28.740 --> 01:06:32.500 align:middle line:84% And it's telling you where it starts in the text 01:06:32.500 --> 01:06:34.540 align:middle line:84% and where it ends the relevant passage. 01:06:34.540 --> 01:06:37.162 align:middle line:90% It's pretty good, right? 01:06:37.162 --> 01:06:38.620 align:middle line:84% Because remember, if you have stuff 01:06:38.620 --> 01:06:41.562 align:middle line:84% like this, then when you ask a large language model a question, 01:06:41.562 --> 01:06:42.520 align:middle line:90% it gives you an answer. 01:06:42.520 --> 01:06:46.020 align:middle line:84% You can actually ask it to give you exactly where in the input 01:06:46.020 --> 01:06:46.853 align:middle line:90% it found the answer. 01:06:46.853 --> 01:06:49.312 align:middle line:84% And because you know these things are going to hallucinate, 01:06:49.312 --> 01:06:50.940 align:middle line:84% you can actually look at the input 01:06:50.940 --> 01:06:53.820 align:middle line:84% that it's claiming to use and look at what it says and see 01:06:53.820 --> 01:06:55.580 align:middle line:90% if they actually match. 01:06:55.580 --> 01:07:00.400 align:middle line:84% It's a way to essentially do QA on LLM output. 01:07:00.400 --> 01:07:01.800 align:middle line:90% So that's what we have here. 01:07:01.800 --> 01:07:03.342 align:middle line:84% And I have another bunch of questions 01:07:03.342 --> 01:07:05.420 align:middle line:84% which I'll ignore for the moment because I want 01:07:05.420 --> 01:07:08.400 align:middle line:90% to go back to the PowerPoint. 01:07:08.400 --> 01:07:11.780 align:middle line:84% So if you have a standard task, you can just use pipelines 01:07:11.780 --> 01:07:13.380 align:middle line:84% and Hugging Face to actually solve 01:07:13.380 --> 01:07:16.300 align:middle line:84% many of them out of the box without any heavy lifting. 01:07:16.300 --> 01:07:18.500 align:middle line:84% So I mentioned earlier on that transformers 01:07:18.500 --> 01:07:21.300 align:middle line:84% have proven to be effective for a whole bunch of domains outside 01:07:21.300 --> 01:07:25.020 align:middle line:84% of natural language processing, like speech 01:07:25.020 --> 01:07:28.380 align:middle line:84% recognition, computer vision, and so on and so forth. 01:07:28.380 --> 01:07:30.940 align:middle line:84% And so I want to give you a couple of quick examples of how 01:07:30.940 --> 01:07:33.060 align:middle line:84% to think about using transformers 01:07:33.060 --> 01:07:34.960 align:middle line:90% for non-text applications. 01:07:34.960 --> 01:07:38.140 align:middle line:90% 01:07:38.140 --> 01:07:40.940 align:middle line:84% The key insight here is that the architecture of the transformer 01:07:40.940 --> 01:07:43.160 align:middle line:84% block that we have looked at, amazingly enough, 01:07:43.160 --> 01:07:45.280 align:middle line:84% can be used as-is with no changes, 01:07:45.280 --> 01:07:47.780 align:middle line:84% no surgery needed, no clever thinking required 01:07:47.780 --> 01:07:49.380 align:middle line:90% for any particular application. 01:07:49.380 --> 01:07:51.920 align:middle line:84% What is needed, where the clever thinking may be required, 01:07:51.920 --> 01:07:55.080 align:middle line:84% is you need to take the inputs that you're working with, 01:07:55.080 --> 01:07:57.860 align:middle line:84% and you need to figure out a way to tokenize and encode them 01:07:57.860 --> 01:08:01.940 align:middle line:84% into embeddings, which can then be sent into the transformer. 01:08:01.940 --> 01:08:05.720 align:middle line:84% So all the action is in taking that input, that non-text input, 01:08:05.720 --> 01:08:07.460 align:middle line:84% and figuring out a way to cast them 01:08:07.460 --> 01:08:09.180 align:middle line:90% in the language of embeddings. 01:08:09.180 --> 01:08:11.070 align:middle line:90% That's the game. 01:08:11.070 --> 01:08:14.240 align:middle line:84% So here is something called the Vision Transformer, 01:08:14.240 --> 01:08:15.820 align:middle line:90% which is very famous, actually. 01:08:15.820 --> 01:08:19.580 align:middle line:84% I think it may be perhaps the first transformer 01:08:19.580 --> 01:08:22.060 align:middle line:84% architecture that was applied to vision problems. 01:08:22.060 --> 01:08:25.580 align:middle line:90% So let's say you have a picture. 01:08:25.580 --> 01:08:28.667 align:middle line:84% So let's say you have this picture. 01:08:28.667 --> 01:08:30.640 align:middle line:90% It is just a picture. 01:08:30.640 --> 01:08:33.020 align:middle line:84% So you have to find a way to create embeddings 01:08:33.020 --> 01:08:36.620 align:middle line:84% from this picture or to tokenize this picture in some way. 01:08:36.620 --> 01:08:39.132 align:middle line:84% With sentences, you know, "I love HODL," well, obviously, 01:08:39.132 --> 01:08:40.840 align:middle line:84% "I," "love," and "HODL" are three tokens. 01:08:40.840 --> 01:08:43.260 align:middle line:84% That's pretty trivial to figure out how to tokenize them. 01:08:43.260 --> 01:08:44.740 align:middle line:84% But with a picture, what do you do? 01:08:44.740 --> 01:08:47.819 align:middle line:84% It's weird to think of tokenizing a picture. 01:08:47.819 --> 01:08:50.385 align:middle line:84% So what these people did is that they say, you know what? 01:08:50.385 --> 01:08:52.260 align:middle line:84% I'm going to take this picture and chop it up 01:08:52.260 --> 01:08:55.340 align:middle line:90% into small squares. 01:08:55.340 --> 01:08:57.939 align:middle line:84% So in this example, they have taken this big picture 01:08:57.939 --> 01:09:01.939 align:middle line:84% and chopped it up into nine little pictures. 01:09:01.939 --> 01:09:05.220 align:middle line:84% Then you can take each of those nine pictures, 01:09:05.220 --> 01:09:06.939 align:middle line:90% each of those nine pictures. 01:09:06.939 --> 01:09:09.720 align:middle line:84% If you look at how it's represented, 01:09:09.720 --> 01:09:14.439 align:middle line:84% it's just three tables of numbers, the RGB values. 01:09:14.439 --> 01:09:16.479 align:middle line:84% So you can take all those numbers, 01:09:16.479 --> 01:09:20.700 align:middle line:84% and you just create a giant long vector from it. 01:09:20.700 --> 01:09:22.620 align:middle line:90% You have a huge long vector. 01:09:22.620 --> 01:09:25.899 align:middle line:84% And then you run it through a dense layer 01:09:25.899 --> 01:09:28.740 align:middle line:84% to come up with a smaller vector. 01:09:28.740 --> 01:09:31.779 align:middle line:84% And that smaller vector is your embedding. 01:09:31.779 --> 01:09:32.660 align:middle line:90% That's it. 01:09:32.660 --> 01:09:35.420 align:middle line:84% But the way you transform the long vector into a small vector 01:09:35.420 --> 01:09:39.100 align:middle line:84% is just a dense layer whose weights can be learned. 01:09:39.100 --> 01:09:41.067 align:middle line:84% So what these people did is they said, 01:09:41.067 --> 01:09:43.359 align:middle line:84% well, I'm going to first chop it up into these patches, 01:09:43.359 --> 01:09:46.740 align:middle line:84% and then I take each patch and do a linear projection. 01:09:46.740 --> 01:09:49.420 align:middle line:84% A flattened patch is nothing more than three tables 01:09:49.420 --> 01:09:51.319 align:middle line:84% of numbers flattened into a long vector. 01:09:51.319 --> 01:09:53.460 align:middle line:84% That's what the word "flatten" here means. 01:09:53.460 --> 01:09:55.100 align:middle line:84% And once you flatten it, I'm just 01:09:55.100 --> 01:09:56.820 align:middle line:84% going to run it through a dense layer. 01:09:56.820 --> 01:09:59.600 align:middle line:84% So by the way, you will see the words "linear projection." 01:09:59.600 --> 01:10:03.420 align:middle line:84% It's a synonym for, run it through a dense layer. 01:10:03.420 --> 01:10:05.520 align:middle line:84% So you run it through a dense layer. 01:10:05.520 --> 01:10:09.540 align:middle line:84% You get these nice vectors, these vectors. 01:10:09.540 --> 01:10:11.113 align:middle line:84% And now you say, well, you know what, 01:10:11.113 --> 01:10:12.780 align:middle line:84% I have to take the order of these things 01:10:12.780 --> 01:10:15.420 align:middle line:84% into account because clearly, this little patch is 01:10:15.420 --> 01:10:18.440 align:middle line:84% in the top left while this patch is somewhere in the middle. 01:10:18.440 --> 01:10:20.475 align:middle line:84% The order matters in the picture, 01:10:20.475 --> 01:10:23.100 align:middle line:84% otherwise, every jumbled version is going to be the same thing. 01:10:23.100 --> 01:10:26.140 align:middle line:84% So you use positional embeddings. 01:10:26.140 --> 01:10:29.780 align:middle line:84% You basically say there are nine positions in any picture. 01:10:29.780 --> 01:10:32.380 align:middle line:90% 0, 1, 2, 3, 4, 5, 6, 7, 8. 01:10:32.380 --> 01:10:35.500 align:middle line:84% There are nine positions, so I'm going to create nine position 01:10:35.500 --> 01:10:36.940 align:middle line:90% embeddings. 01:10:36.940 --> 01:10:38.592 align:middle line:84% And then I'm just going to add them up. 01:10:38.592 --> 01:10:40.800 align:middle line:84% Then I'm just going to add them up to this embedding, 01:10:40.800 --> 01:10:42.726 align:middle line:90% just like we did with words. 01:10:42.726 --> 01:10:44.508 align:middle line:84% With words, each word had an embedding. 01:10:44.508 --> 01:10:45.800 align:middle line:90% Each position had an embedding. 01:10:45.800 --> 01:10:46.508 align:middle line:90% We added them up. 01:10:46.508 --> 01:10:48.500 align:middle line:84% Here each image has an embedding. 01:10:48.500 --> 01:10:50.500 align:middle line:84% The position of that little patch in the picture 01:10:50.500 --> 01:10:51.280 align:middle line:90% has an embedding. 01:10:51.280 --> 01:10:53.180 align:middle line:90% We add them up 01:10:53.180 --> 01:10:54.700 align:middle line:84% And then because we want to use it 01:10:54.700 --> 01:10:56.640 align:middle line:90% for classification, no problem. 01:10:56.640 --> 01:11:00.660 align:middle line:84% We'll have a little CLS token, and then 01:11:00.660 --> 01:11:02.480 align:middle line:84% we just run it through the transformer. 01:11:02.480 --> 01:11:04.300 align:middle line:90% That's it. 01:11:04.300 --> 01:11:06.380 align:middle line:84% And then you get the CLS token, and then 01:11:06.380 --> 01:11:08.900 align:middle line:84% you can attach a softmax to it and say, OK, it's a bird. 01:11:08.900 --> 01:11:09.400 align:middle line:90% It's a ball. 01:11:09.400 --> 01:11:12.020 align:middle line:90% It's a car. 01:11:12.020 --> 01:11:12.840 align:middle line:90% That's it. 01:11:12.840 --> 01:11:20.180 align:middle line:84% This simple approach actually works, amazingly enough. 01:11:20.180 --> 01:11:21.803 align:middle line:84% So that is the vision transformer. 01:11:21.803 --> 01:11:23.220 align:middle line:84% And I'm going through it fast just 01:11:23.220 --> 01:11:25.620 align:middle line:84% to give you a sense for how these things work. 01:11:25.620 --> 01:11:27.300 align:middle line:90% Any questions? 01:11:27.300 --> 01:11:29.060 align:middle line:90% Yeah? 01:11:29.060 --> 01:11:31.740 align:middle line:84% AUDIENCE: My question is in case of text 01:11:31.740 --> 01:11:34.153 align:middle line:84% we had fixed number of tokens that is the amount of words 01:11:34.153 --> 01:11:35.820 align:middle line:84% which could be there in your vocabulary, 01:11:35.820 --> 01:11:37.300 align:middle line:90% any English vocabulary. 01:11:37.300 --> 01:11:39.620 align:middle line:84% But here if you look at images, they will probably 01:11:39.620 --> 01:11:43.560 align:middle line:84% go into trillions and I know we would not upload one image 01:11:43.560 --> 01:11:46.220 align:middle line:84% but we take a total set of product images 01:11:46.220 --> 01:11:47.960 align:middle line:84% and we try to subset each one of them, 01:11:47.960 --> 01:11:53.500 align:middle line:84% each one would have its own weights like [INAUDIBLE]. 01:11:53.500 --> 01:11:55.950 align:middle line:84% RAMA RAMAKRISHNAN: There is no notion of vocabulary here. 01:11:55.950 --> 01:11:58.430 align:middle line:84% All we are saying is that given any image, 01:11:58.430 --> 01:12:02.230 align:middle line:84% we create nine patches, subimages from it. 01:12:02.230 --> 01:12:06.030 align:middle line:84% Each of those patches gets passed through a dense layer, 01:12:06.030 --> 01:12:07.990 align:middle line:90% and out comes an embedding. 01:12:07.990 --> 01:12:09.890 align:middle line:84% So at that point, any image you give me, 01:12:09.890 --> 01:12:12.372 align:middle line:84% I'm going to get you nine embeddings out of it. 01:12:12.372 --> 01:12:13.830 align:middle line:84% And once I get the nine embeddings, 01:12:13.830 --> 01:12:16.110 align:middle line:84% I just throw it into the meat grinder, the transformer meat 01:12:16.110 --> 01:12:16.610 align:middle line:90% grinder. 01:12:16.610 --> 01:12:20.530 align:middle line:90% 01:12:20.530 --> 01:12:23.270 align:middle line:90% So another example. 01:12:23.270 --> 01:12:24.670 align:middle line:84% I think some of you have asked me 01:12:24.670 --> 01:12:27.510 align:middle line:84% outside of class, how good are transformers for 01:12:27.510 --> 01:12:29.780 align:middle line:90% structured data, tabular data. 01:12:29.780 --> 01:12:33.170 align:middle line:84% For tabular data in general, things like XGBoost, 01:12:33.170 --> 01:12:35.050 align:middle line:84% gradient boosting works really, really well, 01:12:35.050 --> 01:12:36.662 align:middle line:84% so it's good to try them, certainly. 01:12:36.662 --> 01:12:38.870 align:middle line:84% I don't think transformers and deep learning networks 01:12:38.870 --> 01:12:43.250 align:middle line:84% have any great edge over XGBoost for structured data problems, 01:12:43.250 --> 01:12:44.850 align:middle line:84% so it's worth trying both of them. 01:12:44.850 --> 01:12:46.910 align:middle line:84% However, you can use transformers for this stuff, 01:12:46.910 --> 01:12:47.590 align:middle line:90% too. 01:12:47.590 --> 01:12:49.610 align:middle line:84% So that's called the tab transformer, 01:12:49.610 --> 01:12:51.430 align:middle line:84% one of the first ones to come out, 01:12:51.430 --> 01:12:53.190 align:middle line:90% a transformer for tabular data. 01:12:53.190 --> 01:12:54.570 align:middle line:90% And again, it's pretty simple. 01:12:54.570 --> 01:12:58.470 align:middle line:84% All you do is in any kind of input that you have, 01:12:58.470 --> 01:13:02.130 align:middle line:84% you will have some categorical variables like blood pressure, 01:13:02.130 --> 01:13:03.360 align:middle line:90% things like that. 01:13:03.360 --> 01:13:04.330 align:middle line:90% Not blood pressure. 01:13:04.330 --> 01:13:05.230 align:middle line:90% Bad example. 01:13:05.230 --> 01:13:08.430 align:middle line:90% Gender and so on and so forth. 01:13:08.430 --> 01:13:11.890 align:middle line:84% And so what you do is you take all the categorical features, 01:13:11.890 --> 01:13:14.110 align:middle line:84% and for each categorical feature, 01:13:14.110 --> 01:13:18.430 align:middle line:84% you create embeddings because a categorical feature is just 01:13:18.430 --> 01:13:20.325 align:middle line:90% text. 01:13:20.325 --> 01:13:22.072 align:middle line:84% A categorical feature is just text, 01:13:22.072 --> 01:13:24.280 align:middle line:84% so you can create text embeddings for it, no problem. 01:13:24.280 --> 01:13:27.830 align:middle line:90% 01:13:27.830 --> 01:13:32.710 align:middle line:84% And you take all the continuous features, cholesterol and blood 01:13:32.710 --> 01:13:35.990 align:middle line:84% pressure and whatnot, to go to the heart disease example, 01:13:35.990 --> 01:13:39.630 align:middle line:84% and then you just correct them all and just create a vector out 01:13:39.630 --> 01:13:41.790 align:middle line:90% of them. 01:13:41.790 --> 01:13:44.170 align:middle line:90% It's just a vector. 01:13:44.170 --> 01:13:48.110 align:middle line:84% Then you run the embeddings for all the categorical variables 01:13:48.110 --> 01:13:50.775 align:middle line:84% through a nice transformer block. 01:13:50.775 --> 01:13:52.150 align:middle line:84% And you can see here it's exactly 01:13:52.150 --> 01:13:53.400 align:middle line:90% the block we have seen before. 01:13:53.400 --> 01:13:54.350 align:middle line:90% No difference. 01:13:54.350 --> 01:13:57.050 align:middle line:84% And then at the very end, when it comes out of the transformer, 01:13:57.050 --> 01:13:58.150 align:middle line:84% you take all the contextual stuff 01:13:58.150 --> 01:14:00.630 align:middle line:84% coming out of the transformer, and then you concatenate it 01:14:00.630 --> 01:14:03.710 align:middle line:90% with the continuous features. 01:14:03.710 --> 01:14:06.470 align:middle line:84% And then you run it through maybe one or more dense layers 01:14:06.470 --> 01:14:09.470 align:middle line:90% and boom, output. 01:14:09.470 --> 01:14:11.610 align:middle line:84% So this is a tabular data transformer. 01:14:11.610 --> 01:14:14.390 align:middle line:84% And there are many refinements, improvements over the years 01:14:14.390 --> 01:14:16.390 align:middle line:90% that have come since then. 01:14:16.390 --> 01:14:19.150 align:middle line:84% But the key thing I want you to remember here 01:14:19.150 --> 01:14:23.950 align:middle line:84% is that categorical variables can be very easily represented 01:14:23.950 --> 01:14:25.230 align:middle line:90% as embeddings. 01:14:25.230 --> 01:14:25.910 align:middle line:90% That's the key. 01:14:25.910 --> 01:14:29.050 align:middle line:90% 01:14:29.050 --> 01:14:30.030 align:middle line:90% So that's that. 01:14:30.030 --> 01:14:31.910 align:middle line:84% Now once the input has been transformed 01:14:31.910 --> 01:14:34.090 align:middle line:84% into this common language of embeddings, 01:14:34.090 --> 01:14:35.590 align:middle line:84% we can process them without changing 01:14:35.590 --> 01:14:37.965 align:middle line:84% the architecture of the block itself because all it wants 01:14:37.965 --> 01:14:38.990 align:middle line:90% is embeddings. 01:14:38.990 --> 01:14:40.530 align:middle line:84% It's like, you give me embeddings, 01:14:40.530 --> 01:14:42.322 align:middle line:84% I give you great contextual embeddings out, 01:14:42.322 --> 01:14:43.697 align:middle line:90% and nobody gets hurt. 01:14:43.697 --> 01:14:45.530 align:middle line:84% That is the deal with the transformer stack. 01:14:45.530 --> 01:14:50.282 align:middle line:90% 01:14:50.282 --> 01:14:53.090 align:middle line:84% Since the transformer is agnostic to the kind of input, 01:14:53.090 --> 01:14:56.230 align:middle line:84% as long as it comes in as the form of an embedding, 01:14:56.230 --> 01:14:58.293 align:middle line:84% you can use it for multi-modal data very easily. 01:14:58.293 --> 01:14:59.710 align:middle line:84% So for example, let's say that you 01:14:59.710 --> 01:15:02.150 align:middle line:84% have a problem in which you have a picture that you have 01:15:02.150 --> 01:15:04.030 align:middle line:84% to be sent in, some text that goes in, 01:15:04.030 --> 01:15:06.550 align:middle line:84% a bunch of tabular data coming in. 01:15:06.550 --> 01:15:09.270 align:middle line:84% Well, you take the text and do language embeddings, 01:15:09.270 --> 01:15:10.330 align:middle line:90% like we know how to do. 01:15:10.330 --> 01:15:11.790 align:middle line:84% You take the image and do image embeddings, 01:15:11.790 --> 01:15:13.610 align:middle line:84% like we just saw with the vision transformer. 01:15:13.610 --> 01:15:15.777 align:middle line:84% You take tabular data under tabular data embeddings, 01:15:15.777 --> 01:15:17.870 align:middle line:84% like we saw with the tab transformer. 01:15:17.870 --> 01:15:21.830 align:middle line:84% Once we do it, it's all a bunch of embeddings. 01:15:21.830 --> 01:15:24.150 align:middle line:84% And then you attach a little class token on top, 01:15:24.150 --> 01:15:26.885 align:middle line:84% send it through a bunch of transformer blocks, 01:15:26.885 --> 01:15:28.510 align:middle line:84% and then out comes the contextual class 01:15:28.510 --> 01:15:30.450 align:middle line:90% token, the contextual version. 01:15:30.450 --> 01:15:33.090 align:middle line:84% Run it through a ReLU, maybe a sigmoid or a softmax, 01:15:33.090 --> 01:15:35.642 align:middle line:90% predict the label, done. 01:15:35.642 --> 01:15:38.310 align:middle line:84% So this is extremely powerful, its ability 01:15:38.310 --> 01:15:42.490 align:middle line:84% to handle multi-modal data And that's why, for example, 01:15:42.490 --> 01:15:47.130 align:middle line:84% if you look at Google Gemini 1.5 Pro, GPT4 Vision and so on, 01:15:47.130 --> 01:15:49.630 align:middle line:84% you can send it images and a question, 01:15:49.630 --> 01:15:51.710 align:middle line:90% and you'll get an answer back. 01:15:51.710 --> 01:15:55.050 align:middle line:84% Because every modality that goes in is cast into embeddings. 01:15:55.050 --> 01:15:59.468 align:middle line:84% And once it's embeddingized, then the transformer 01:15:59.468 --> 01:16:00.010 align:middle line:90% doesn't care. 01:16:00.010 --> 01:16:02.030 align:middle line:90% It'll just do its thing. 01:16:02.030 --> 01:16:05.590 align:middle line:84% It will decide, for example, that this word in your question 01:16:05.590 --> 01:16:10.030 align:middle line:84% actually is highly related to that patch in the picture. 01:16:10.030 --> 01:16:13.388 align:middle line:90% It'll just figure it out. 01:16:13.388 --> 01:16:15.930 align:middle line:84% OK, that's all I had because it's that time to break at 9:55. 01:16:15.930 --> 01:16:16.728 align:middle line:90% Perfect. 01:16:16.728 --> 01:16:17.770 align:middle line:90% All right, folks, thanks. 01:16:17.770 --> 01:16:19.062 align:middle line:90% Have a great rest of your week. 01:16:19.062 --> 01:16:22.560 align:middle line:90% [APPLAUSE] 01:16:22.560 --> 01:16:46.000 align:middle line:90%