WEBVTT 00:00:00.000 --> 00:00:04.419 align:middle line:84% [SQUEAKING] [RUSTLING] [CLICKING] 00:00:04.419 --> 00:00:16.230 align:middle line:90% 00:00:16.230 --> 00:00:18.480 align:middle line:84% RAMA RAMAKRISHNAN: So let's start with a quick review. 00:00:18.480 --> 00:00:22.100 align:middle line:84% Last week, we looked at BERT, how BERT was created. 00:00:22.100 --> 00:00:24.900 align:middle line:84% And we learned about this technique called masking, 00:00:24.900 --> 00:00:27.140 align:middle line:84% which is a kind of self-supervised learning. 00:00:27.140 --> 00:00:29.000 align:middle line:84% And the idea of masking was very simple. 00:00:29.000 --> 00:00:30.880 align:middle line:90% We asked ourselves the question. 00:00:30.880 --> 00:00:32.320 align:middle line:84% We have seen ways in which people 00:00:32.320 --> 00:00:34.680 align:middle line:84% can take images and pre-trained models 00:00:34.680 --> 00:00:38.705 align:middle line:84% like ResNet on vast body of images. 00:00:38.705 --> 00:00:40.080 align:middle line:84% But then for each image, somebody 00:00:40.080 --> 00:00:41.920 align:middle line:90% had to go and label them. 00:00:41.920 --> 00:00:43.860 align:middle line:84% So for text, we asked the question, well, 00:00:43.860 --> 00:00:46.398 align:middle line:84% what does it mean to label a piece of text 00:00:46.398 --> 00:00:48.440 align:middle line:84% when we don't actually have a clearly defined end 00:00:48.440 --> 00:00:51.580 align:middle line:84% goal in mind, except the general goal of pre-training things? 00:00:51.580 --> 00:00:53.120 align:middle line:84% And then we said oh, well, what we 00:00:53.120 --> 00:00:55.120 align:middle line:84% can do is we can actually replace 00:00:55.120 --> 00:00:57.080 align:middle line:84% some of the words in every sentence 00:00:57.080 --> 00:00:59.100 align:middle line:90% with what we call a mask token. 00:00:59.100 --> 00:01:02.520 align:middle line:84% And then we just train the network to recover the blanks, 00:01:02.520 --> 00:01:04.140 align:middle line:90% to fill in the blanks. 00:01:04.140 --> 00:01:06.780 align:middle line:84% And this technique, which is one of many ways of doing 00:01:06.780 --> 00:01:09.580 align:middle line:84% what's called self-supervised learning, is called masking. 00:01:09.580 --> 00:01:12.980 align:middle line:84% And we described how, if you essentially 00:01:12.980 --> 00:01:15.460 align:middle line:84% take all of Wikipedia, and for every sentence, 00:01:15.460 --> 00:01:19.100 align:middle line:84% you mask it like this, and then train a network to recover, 00:01:19.100 --> 00:01:21.660 align:middle line:84% to fill in the blanks, the resulting network 00:01:21.660 --> 00:01:24.400 align:middle line:84% becomes really good at doing all kinds of interesting things. 00:01:24.400 --> 00:01:26.840 align:middle line:84% And that, in fact, the first such network, 00:01:26.840 --> 00:01:29.460 align:middle line:84% or one of the first such networks, was called BERT. 00:01:29.460 --> 00:01:31.047 align:middle line:84% And in fact in your homework, you've 00:01:31.047 --> 00:01:32.880 align:middle line:84% been looking at BERT and so on and so forth. 00:01:32.880 --> 00:01:33.993 align:middle line:90% That's masking. 00:01:33.993 --> 00:01:35.660 align:middle line:84% Now we're going to switch gears and talk 00:01:35.660 --> 00:01:38.180 align:middle line:84% about a different kind of self-supervised learning, which 00:01:38.180 --> 00:01:43.180 align:middle line:84% is different from masking, which turns out to be weirdly more 00:01:43.180 --> 00:01:45.220 align:middle line:90% interesting and powerful. 00:01:45.220 --> 00:01:47.940 align:middle line:84% So we are going to look at another technique. 00:01:47.940 --> 00:01:50.580 align:middle line:84% And this technique is called next word prediction. 00:01:50.580 --> 00:01:53.445 align:middle line:84% So now it is actually, in some sense, 00:01:53.445 --> 00:01:55.820 align:middle line:84% a special case of masking, where you're basically saying, 00:01:55.820 --> 00:01:58.840 align:middle line:84% take a sentence and, instead of randomly picking a word 00:01:58.840 --> 00:02:01.380 align:middle line:84% and making it a blank, you're saying, 00:02:01.380 --> 00:02:04.800 align:middle line:84% I'm just going to take the last word and make it a blank. 00:02:04.800 --> 00:02:07.000 align:middle line:84% And then you send the sentence in. 00:02:07.000 --> 00:02:09.840 align:middle line:84% And then you have the machine just fill in the blank 00:02:09.840 --> 00:02:11.620 align:middle line:84% on the last word, predict the next word. 00:02:11.620 --> 00:02:14.560 align:middle line:84% And you don't have to use full sentences for it. 00:02:14.560 --> 00:02:16.160 align:middle line:84% You can use parts of sentences for it, 00:02:16.160 --> 00:02:17.300 align:middle line:90% sentence fragments as well. 00:02:17.300 --> 00:02:20.000 align:middle line:84% So if you take the same sentence as before, 00:02:20.000 --> 00:02:21.740 align:middle line:84% the mission of the MIT Sloan School, 00:02:21.740 --> 00:02:23.420 align:middle line:84% you can literally divide it into-- well, 00:02:23.420 --> 00:02:26.040 align:middle line:84% you can give the and ask it to predict mission. 00:02:26.040 --> 00:02:29.080 align:middle line:84% If you can give it the mission and ask it to predict of. 00:02:29.080 --> 00:02:31.440 align:middle line:84% You give it the mission, of, ask to predict the. 00:02:31.440 --> 00:02:32.520 align:middle line:90% You get the idea. 00:02:32.520 --> 00:02:35.137 align:middle line:84% So every sentence fragment, you can take, and literally just 00:02:35.137 --> 00:02:36.720 align:middle line:84% give it the first few and then predict 00:02:36.720 --> 00:02:40.226 align:middle line:84% the next one, first few, next one, first few, next one. 00:02:40.226 --> 00:02:42.480 align:middle line:90% So this is next word prediction. 00:02:42.480 --> 00:02:46.240 align:middle line:84% And so what we're going to do now 00:02:46.240 --> 00:02:49.000 align:middle line:84% is we're going to actually take the transformer encoder 00:02:49.000 --> 00:02:52.520 align:middle line:84% architecture that we used to build BERT in the last class. 00:02:52.520 --> 00:02:55.380 align:middle line:84% And we're going to try to use it to solve next word prediction, 00:02:55.380 --> 00:02:58.460 align:middle line:84% to build a model that can do next word prediction. 00:02:58.460 --> 00:03:01.020 align:middle line:90% So this is what we have. 00:03:01.020 --> 00:03:04.060 align:middle line:84% So what we're going to do is, if you take the phrase, 00:03:04.060 --> 00:03:10.820 align:middle line:84% the cat sat on the mat, so the phrase was, let's say, 00:03:10.820 --> 00:03:20.360 align:middle line:84% the cat sat on the mat, so what you might want to do is to say, 00:03:20.360 --> 00:03:21.450 align:middle line:90% OK, this is the input. 00:03:21.450 --> 00:03:25.420 align:middle line:90% 00:03:25.420 --> 00:03:29.980 align:middle line:90% Output, the cat. 00:03:29.980 --> 00:03:33.060 align:middle line:90% Then maybe you have the cat. 00:03:33.060 --> 00:03:36.340 align:middle line:90% And then the output is sat. 00:03:36.340 --> 00:03:39.400 align:middle line:90% The cat sat on, and so on. 00:03:39.400 --> 00:03:40.740 align:middle line:90% You get the idea. 00:03:40.740 --> 00:03:45.180 align:middle line:84% And then, finally, we have the cat sat, whoa, whoa, whoa, whoa, 00:03:45.180 --> 00:03:47.420 align:middle line:90% the mat. 00:03:47.420 --> 00:03:50.120 align:middle line:84% This, basically, what we have, all these inputs and outputs. 00:03:50.120 --> 00:03:52.990 align:middle line:84% But we're going to very compactly express it 00:03:52.990 --> 00:03:55.750 align:middle line:84% as if we're just coming in through as one sort of data 00:03:55.750 --> 00:03:57.550 align:middle line:90% point in one batch. 00:03:57.550 --> 00:03:58.930 align:middle line:84% And that's what we're doing here. 00:03:58.930 --> 00:03:59.910 align:middle line:84% So what we're going to do is we're 00:03:59.910 --> 00:04:01.830 align:middle line:84% going to stack it up like this, where 00:04:01.830 --> 00:04:04.810 align:middle line:84% we have the cat sat on the, on the left, 00:04:04.810 --> 00:04:06.830 align:middle line:84% meaning everything but the last word. 00:04:06.830 --> 00:04:08.910 align:middle line:84% And then we're going to take that same sentence 00:04:08.910 --> 00:04:11.630 align:middle line:84% and just shift it to the left one. 00:04:11.630 --> 00:04:14.890 align:middle line:84% So the cat sat on the mat, we cut off the mat, right? 00:04:14.890 --> 00:04:16.130 align:middle line:90% And that becomes the input. 00:04:16.130 --> 00:04:19.430 align:middle line:84% Then we cut off the first word, and that becomes the output. 00:04:19.430 --> 00:04:22.650 align:middle line:84% So when you look at it that way, you can see here, 00:04:22.650 --> 00:04:27.110 align:middle line:84% the, you will want the to be used to predict cat, 00:04:27.110 --> 00:04:29.630 align:middle line:84% you will want the cat to be used to predict 00:04:29.630 --> 00:04:33.430 align:middle line:90% sat, and so on and so forth. 00:04:33.430 --> 00:04:36.890 align:middle line:84% So this is just a little sort of manipulation 00:04:36.890 --> 00:04:41.030 align:middle line:84% so that we don't have to have dozens of sentences or sentence 00:04:41.030 --> 00:04:44.790 align:middle line:84% examples just for one starting sentence. 00:04:44.790 --> 00:04:47.310 align:middle line:84% So if you have something like this, what you can do 00:04:47.310 --> 00:04:50.070 align:middle line:84% is you can run it through positional input embeddings, 00:04:50.070 --> 00:04:52.730 align:middle line:84% like we have done before with BERT. 00:04:52.730 --> 00:04:56.410 align:middle line:84% Then we can run it through a whole bunch of transformers. 00:04:56.410 --> 00:04:58.290 align:middle line:90% It's like a transformer stack. 00:04:58.290 --> 00:05:00.890 align:middle line:84% Then we get these contextual embeddings. 00:05:00.890 --> 00:05:03.903 align:middle line:84% Then we run them through maybe one or more ReLUs, if you want, 00:05:03.903 --> 00:05:05.570 align:middle line:84% because it's always a good idea to stick 00:05:05.570 --> 00:05:07.770 align:middle line:90% some ReLUs at the very end. 00:05:07.770 --> 00:05:11.330 align:middle line:84% And then we basically attach a softmax 00:05:11.330 --> 00:05:14.310 align:middle line:84% to every one of the things that are coming out. 00:05:14.310 --> 00:05:17.250 align:middle line:84% And then that softmax is actually 00:05:17.250 --> 00:05:24.250 align:middle line:84% going to be a softmax whose range is the entire vocabulary. 00:05:24.250 --> 00:05:27.050 align:middle line:84% For now, let's assume that the vocabulary is just a vocabulary 00:05:27.050 --> 00:05:28.427 align:middle line:90% of words, not tokens. 00:05:28.427 --> 00:05:30.510 align:middle line:84% We'll get into tokens a bit later on in the class. 00:05:30.510 --> 00:05:31.880 align:middle line:90% For now, just assume it's words. 00:05:31.880 --> 00:05:34.130 align:middle line:84% And roughly speaking, let's say there are 50,000 words 00:05:34.130 --> 00:05:35.330 align:middle line:90% in our vocabulary. 00:05:35.330 --> 00:05:37.650 align:middle line:84% So each of these softmaxes-- and this 00:05:37.650 --> 00:05:39.530 align:middle line:84% is exactly what we did for BERT, by the way. 00:05:39.530 --> 00:05:45.210 align:middle line:84% Each of these softmaxes is like a 50,000 way softmax. 00:05:45.210 --> 00:05:47.810 align:middle line:84% But what we're going to do is, here, 00:05:47.810 --> 00:05:51.750 align:middle line:84% when we look at it this way, since we are fundamentally 00:05:51.750 --> 00:05:53.402 align:middle line:84% bothered about next word prediction, 00:05:53.402 --> 00:05:55.110 align:middle line:84% as you will see later on, we are actually 00:05:55.110 --> 00:05:58.070 align:middle line:84% going to ignore all these predictions because who cares? 00:05:58.070 --> 00:06:01.210 align:middle line:84% We are only going to look at the last one to figure out, OK, 00:06:01.210 --> 00:06:03.550 align:middle line:90% what is the last prediction? 00:06:03.550 --> 00:06:04.050 align:middle line:90% What is it? 00:06:04.050 --> 00:06:07.790 align:middle line:84% Because the last prediction is going to be based on everything 00:06:07.790 --> 00:06:09.470 align:middle line:90% that came before it here. 00:06:09.470 --> 00:06:12.450 align:middle line:84% So this is really the next word that's actually being predicted. 00:06:12.450 --> 00:06:16.282 align:middle line:84% All the things before, we don't care so much. 00:06:16.282 --> 00:06:17.990 align:middle line:84% And all this will become slightly clearer 00:06:17.990 --> 00:06:20.050 align:middle line:84% because we're going to make a couple of passes through it. 00:06:20.050 --> 00:06:20.830 align:middle line:90% Yeah? 00:06:20.830 --> 00:06:24.230 align:middle line:84% AUDIENCE: How do we predict the end of a sentence then? 00:06:24.230 --> 00:06:27.230 align:middle line:84% RAMA RAMAKRISHNAN: So the notion of a sentence 00:06:27.230 --> 00:06:28.650 align:middle line:90% has disappeared at this point. 00:06:28.650 --> 00:06:30.067 align:middle line:84% What we're going to do is, when we 00:06:30.067 --> 00:06:33.840 align:middle line:84% look at how we tokenize the input for these kinds of models, 00:06:33.840 --> 00:06:36.090 align:middle line:84% we're actually going to take punctuation into account. 00:06:36.090 --> 00:06:36.430 align:middle line:90% AUDIENCE: Oh. 00:06:36.430 --> 00:06:36.830 align:middle line:84% RAMA RAMAKRISHNAN: So we're going 00:06:36.830 --> 00:06:38.350 align:middle line:84% to take periods into account, exclamation 00:06:38.350 --> 00:06:40.170 align:middle line:84% marks into account, and so on and so forth. 00:06:40.170 --> 00:06:41.630 align:middle line:84% And that will answer your question, 00:06:41.630 --> 00:06:44.470 align:middle line:90% and we'll come back to that. 00:06:44.470 --> 00:06:46.350 align:middle line:90% So this is what we have. 00:06:46.350 --> 00:06:49.150 align:middle line:84% So all right, so just to be clear, 00:06:49.150 --> 00:06:51.610 align:middle line:84% the embedding that's coming out of the final dense layer 00:06:51.610 --> 00:06:53.188 align:middle line:84% is passed through its own softmax 00:06:53.188 --> 00:06:54.730 align:middle line:84% with the number of softmax categories 00:06:54.730 --> 00:06:58.050 align:middle line:90% equal to the vocab size. 00:06:58.050 --> 00:07:03.170 align:middle line:84% All right, so first of all, so let's 00:07:03.170 --> 00:07:05.330 align:middle line:84% say we train models, a model like this, 00:07:05.330 --> 00:07:07.970 align:middle line:90% with lots of inputs and outputs. 00:07:07.970 --> 00:07:09.100 align:middle line:90% This just looks like BERT. 00:07:09.100 --> 00:07:10.850 align:middle line:84% It's not that different, except that there 00:07:10.850 --> 00:07:13.505 align:middle line:90% is no notion of a mask. 00:07:13.505 --> 00:07:15.130 align:middle line:84% Do you notice any problems with the way 00:07:15.130 --> 00:07:16.390 align:middle line:90% this thing has been set up? 00:07:16.390 --> 00:07:19.562 align:middle line:90% 00:07:19.562 --> 00:07:21.020 align:middle line:84% AUDIENCE: For some words, like the, 00:07:21.020 --> 00:07:24.130 align:middle line:84% you're going to have a lot of potential output pairs 00:07:24.130 --> 00:07:25.310 align:middle line:90% that come out of that. 00:07:25.310 --> 00:07:26.810 align:middle line:84% RAMA RAMAKRISHNAN: True, which means 00:07:26.810 --> 00:07:28.958 align:middle line:84% that if you have a word like the, the next word-- 00:07:28.958 --> 00:07:30.250 align:middle line:90% AUDIENCE: Very hard to predict. 00:07:30.250 --> 00:07:31.250 align:middle line:90% RAMA RAMAKRISHNAN: True. 00:07:31.250 --> 00:07:32.890 align:middle line:84% So some words may be hard to predict, 00:07:32.890 --> 00:07:36.550 align:middle line:84% depending on the last word of the sentence that was the input. 00:07:36.550 --> 00:07:38.690 align:middle line:84% That's what you're getting at, yeah. 00:07:38.690 --> 00:07:41.210 align:middle line:90% Other concerns? 00:07:41.210 --> 00:07:43.770 align:middle line:90% So I want you-- yeah? 00:07:43.770 --> 00:07:46.390 align:middle line:84% AUDIENCE: Since you're using contextual on this, 00:07:46.390 --> 00:07:50.110 align:middle line:84% the output of the first word is going to have access 00:07:50.110 --> 00:07:51.290 align:middle line:90% to the second word. 00:07:51.290 --> 00:07:53.790 align:middle line:84% And so it's kind of like cheating. 00:07:53.790 --> 00:07:55.270 align:middle line:90% RAMA RAMAKRISHNAN: Bingo. 00:07:55.270 --> 00:08:00.210 align:middle line:84% So remember, bingo is a technical term in deep learning, 00:08:00.210 --> 00:08:02.030 align:middle line:90% which means great. 00:08:02.030 --> 00:08:07.070 align:middle line:84% So if you go to this, as she points out, 00:08:07.070 --> 00:08:10.330 align:middle line:84% if you look at the self-attention layer, note, 00:08:10.330 --> 00:08:11.870 align:middle line:84% remember, the self-attention layer 00:08:11.870 --> 00:08:15.030 align:middle line:84% is the key building block of the transformer block. 00:08:15.030 --> 00:08:18.470 align:middle line:84% And so in the self-attention layer, every word, 00:08:18.470 --> 00:08:20.590 align:middle line:84% we calculate its contextual embedding 00:08:20.590 --> 00:08:25.110 align:middle line:84% by weighting-- weighted averaging of its relationship 00:08:25.110 --> 00:08:27.390 align:middle line:84% to all other words in the sentence. 00:08:27.390 --> 00:08:29.438 align:middle line:84% So the last word can see the first word, 00:08:29.438 --> 00:08:31.980 align:middle line:84% the first word can see the last word, and so on and so forth. 00:08:31.980 --> 00:08:33.850 align:middle line:84% But when we are doing next word prediction, 00:08:33.850 --> 00:08:36.433 align:middle line:84% this feels problematic because you're peeking into the future. 00:08:36.433 --> 00:08:39.270 align:middle line:90% 00:08:39.270 --> 00:08:43.010 align:middle line:84% So let's say that you want to predict the next word. 00:08:43.010 --> 00:08:45.610 align:middle line:84% If you look at this architecture, what it can simply 00:08:45.610 --> 00:08:48.530 align:middle line:84% do, it can simply copy it from the input 00:08:48.530 --> 00:08:50.570 align:middle line:84% because it can see the whole sentence. 00:08:50.570 --> 00:08:53.530 align:middle line:84% So if I tell you, hey, the cat sat on the mat, 00:08:53.530 --> 00:08:55.688 align:middle line:84% if I just gave you, the cat sat on the, 00:08:55.688 --> 00:08:57.230 align:middle line:84% can you predict the next word for me? 00:08:57.230 --> 00:09:01.127 align:middle line:84% You'd be like, yeah, duh, it's mat. 00:09:01.127 --> 00:09:03.210 align:middle line:84% The whole thing becomes challenging only if I say, 00:09:03.210 --> 00:09:04.350 align:middle line:90% the cat sat on the dash. 00:09:04.350 --> 00:09:07.090 align:middle line:90% Now, predict the dash. 00:09:07.090 --> 00:09:11.330 align:middle line:84% So to put it another way, let's say that you want to predict-- 00:09:11.330 --> 00:09:13.250 align:middle line:84% you're fed in the first two words, 00:09:13.250 --> 00:09:14.510 align:middle line:90% and you want to predict this. 00:09:14.510 --> 00:09:16.890 align:middle line:84% This is the right answer for the prediction. 00:09:16.890 --> 00:09:20.490 align:middle line:84% The network should only use the first two. 00:09:20.490 --> 00:09:24.390 align:middle line:84% However, but because self-attention can see sat, 00:09:24.390 --> 00:09:27.330 align:middle line:84% it can see this next word, it will trivially 00:09:27.330 --> 00:09:31.430 align:middle line:84% learn to predict the next word to be sat. 00:09:31.430 --> 00:09:34.490 align:middle line:90% There is no challenge for it. 00:09:34.490 --> 00:09:36.678 align:middle line:90% So this is the key problem. 00:09:36.678 --> 00:09:38.970 align:middle line:84% This is the key problem with just using the transformer 00:09:38.970 --> 00:09:41.673 align:middle line:90% as is. 00:09:41.673 --> 00:09:43.340 align:middle line:84% AUDIENCE: What's our loss function here? 00:09:43.340 --> 00:09:44.780 align:middle line:84% RAMA RAMAKRISHNAN: The loss function in all these things 00:09:44.780 --> 00:09:46.322 align:middle line:84% is actually the same as before, which 00:09:46.322 --> 00:09:48.492 align:middle line:84% is that, for every output that's coming out-- 00:09:48.492 --> 00:09:50.700 align:middle line:84% so imagine you have just a traditional classification 00:09:50.700 --> 00:09:53.702 align:middle line:84% problem, in which you have one output, 00:09:53.702 --> 00:09:55.660 align:middle line:84% let's say, dividing-- you're classifying things 00:09:55.660 --> 00:09:57.860 align:middle line:84% to 10 categories, like we did with the Fashion 00:09:57.860 --> 00:09:59.380 align:middle line:90% MNIST, 10 digits. 00:09:59.380 --> 00:10:02.820 align:middle line:84% So you have 10 outputs, and that goes through a softmax. 00:10:02.820 --> 00:10:04.680 align:middle line:84% And then you have 10 probabilities. 00:10:04.680 --> 00:10:07.360 align:middle line:90% And there we use cross-entropy. 00:10:07.360 --> 00:10:12.100 align:middle line:84% So here, for every one of these things, we use cross-entropy. 00:10:12.100 --> 00:10:14.420 align:middle line:84% So we take this output, and there's a cross-entropy 00:10:14.420 --> 00:10:16.840 align:middle line:84% just for that, plus a cross-entropy for that, 00:10:16.840 --> 00:10:18.040 align:middle line:90% and so on and so forth. 00:10:18.040 --> 00:10:21.020 align:middle line:84% So we minimize still cross-entropy, but the sum 00:10:21.020 --> 00:10:22.228 align:middle line:90% of all these cross-entropies. 00:10:22.228 --> 00:10:23.770 align:middle line:84% AUDIENCE: And does it get complicated 00:10:23.770 --> 00:10:26.740 align:middle line:84% at all by the fact that we have a large vocabulary size now, So 00:10:26.740 --> 00:10:28.000 align:middle line:90% lots of output? 00:10:28.000 --> 00:10:28.780 align:middle line:84% RAMA RAMAKRISHNAN: It gets complicated 00:10:28.780 --> 00:10:31.197 align:middle line:84% just because there are more things to worry about, compute 00:10:31.197 --> 00:10:32.160 align:middle line:90% and so on and so forth. 00:10:32.160 --> 00:10:33.620 align:middle line:90% But conceptually, no difference. 00:10:33.620 --> 00:10:36.220 align:middle line:84% Whether you have 10,000 or 50,000, it's the same thing. 00:10:36.220 --> 00:10:38.940 align:middle line:84% It's just that, instead of classifying an input into one 00:10:38.940 --> 00:10:41.760 align:middle line:84% of 10 categories, the inputs themselves 00:10:41.760 --> 00:10:44.080 align:middle line:84% are as long as the number of words in your sentence. 00:10:44.080 --> 00:10:46.760 align:middle line:84% So each word that comes into your sentence is being 00:10:46.760 --> 00:10:49.560 align:middle line:84% classified in one of 50,000 ways. 00:10:49.560 --> 00:10:52.800 align:middle line:84% So essentially, you have as many classification problems 00:10:52.800 --> 00:10:54.680 align:middle line:84% as you have number of words in a sentence. 00:10:54.680 --> 00:10:56.160 align:middle line:84% But at the end of the day, the loss function 00:10:56.160 --> 00:10:58.480 align:middle line:84% is the sum of all those things, or to be more precise, 00:10:58.480 --> 00:10:59.813 align:middle line:90% the average of all those things. 00:10:59.813 --> 00:11:02.020 align:middle line:90% 00:11:02.020 --> 00:11:04.020 align:middle line:84% Actually, I think I may have a slide about this, 00:11:04.020 --> 00:11:06.640 align:middle line:84% which I may have hidden because I wasn't sure 00:11:06.640 --> 00:11:09.360 align:middle line:90% if I would have time. 00:11:09.360 --> 00:11:10.540 align:middle line:90% Let's unhide it. 00:11:10.540 --> 00:11:17.360 align:middle line:90% 00:11:17.360 --> 00:11:19.320 align:middle line:84% And I did not agree ahead of time 00:11:19.320 --> 00:11:22.080 align:middle line:84% that we're going to set this up like this. 00:11:22.080 --> 00:11:25.480 align:middle line:84% All right, so, yeah, so we still use a classic-- 00:11:25.480 --> 00:11:27.240 align:middle line:90% cross-entropy loss function. 00:11:27.240 --> 00:11:32.200 align:middle line:84% So each word that comes in, so the cross-entropy is actually 00:11:32.200 --> 00:11:34.840 align:middle line:84% minus log probability of the right answer. 00:11:34.840 --> 00:11:36.940 align:middle line:84% And you may recall this from earlier in the class. 00:11:36.940 --> 00:11:40.700 align:middle line:84% So we just do the same thing for cat sat on the, everything, 00:11:40.700 --> 00:11:42.760 align:middle line:84% and then we just take the average, 1 over 7. 00:11:42.760 --> 00:11:43.960 align:middle line:90% Boom, that's it. 00:11:43.960 --> 00:11:47.340 align:middle line:90% 00:11:47.340 --> 00:11:51.080 align:middle line:84% So to go back to this problem, so this is the issue. 00:11:51.080 --> 00:11:54.420 align:middle line:84% The issue is that we can't allow words to be 00:11:54.420 --> 00:11:56.860 align:middle line:90% predicted knowing the future. 00:11:56.860 --> 00:11:59.300 align:middle line:84% They should only know about the past words. 00:11:59.300 --> 00:12:01.110 align:middle line:90% So what do we do? 00:12:01.110 --> 00:12:03.340 align:middle line:84% We have to make a change to the transformer 00:12:03.340 --> 00:12:05.780 align:middle line:84% to make it work for next word prediction. 00:12:05.780 --> 00:12:08.140 align:middle line:84% So what we're going to do is when we are calculating 00:12:08.140 --> 00:12:10.700 align:middle line:84% the contextual embedding for a word, remember, 00:12:10.700 --> 00:12:12.210 align:middle line:84% the context embedding for a word is 00:12:12.210 --> 00:12:14.460 align:middle line:84% going to be a weighted average of all the other word's 00:12:14.460 --> 00:12:15.580 align:middle line:90% embeddings. 00:12:15.580 --> 00:12:19.980 align:middle line:84% We will simply give zero weight to future words. 00:12:19.980 --> 00:12:22.040 align:middle line:84% If you give zero weight to future words, 00:12:22.040 --> 00:12:23.650 align:middle line:84% it's almost as if they don't exist. 00:12:23.650 --> 00:12:26.180 align:middle line:90% 00:12:26.180 --> 00:12:28.180 align:middle line:84% And this will become clear in a second. 00:12:28.180 --> 00:12:32.720 align:middle line:84% So imagine that this is the thing we are going to calculate. 00:12:32.720 --> 00:12:34.820 align:middle line:84% These are all-- for every word in the sentence, 00:12:34.820 --> 00:12:39.960 align:middle line:84% we are calculating the pairwise attention weight. 00:12:39.960 --> 00:12:43.840 align:middle line:84% And you will remember, I went through this with an iPad thing 00:12:43.840 --> 00:12:44.692 align:middle line:90% last week. 00:12:44.692 --> 00:12:45.900 align:middle line:90% We calculate all the weights. 00:12:45.900 --> 00:12:49.560 align:middle line:90% So, for example, to find the-- 00:12:49.560 --> 00:12:52.960 align:middle line:84% so all these weights in every row will add up to 1. 00:12:52.960 --> 00:12:55.960 align:middle line:84% And so you take the contextual embeddings of the cat 00:12:55.960 --> 00:12:58.857 align:middle line:84% sat on the, multiply them by their respective weights that 00:12:58.857 --> 00:13:00.940 align:middle line:84% add up to 1, which is the first row of this table. 00:13:00.940 --> 00:13:02.732 align:middle line:84% And that gives you the contextual embedding 00:13:02.732 --> 00:13:04.760 align:middle line:84% for the word the, so on and so forth. 00:13:04.760 --> 00:13:07.380 align:middle line:84% And since we can't look at the future words, 00:13:07.380 --> 00:13:09.920 align:middle line:84% all we do is we go take this table 00:13:09.920 --> 00:13:12.020 align:middle line:84% and we just zero everything out in red. 00:13:12.020 --> 00:13:14.920 align:middle line:90% 00:13:14.920 --> 00:13:17.280 align:middle line:84% We just zero everything here out. 00:13:17.280 --> 00:13:20.980 align:middle line:84% And then we renormalize so that the remaining cells, 00:13:20.980 --> 00:13:25.040 align:middle line:84% the non-zero dot cells, will still add up to one in each row. 00:13:25.040 --> 00:13:28.400 align:middle line:84% So what that means is that, if you're actually only looking 00:13:28.400 --> 00:13:30.898 align:middle line:84% at the, only this thing is going to play a role. 00:13:30.898 --> 00:13:32.940 align:middle line:84% For cat, only this thing is going to play a role. 00:13:32.940 --> 00:13:35.220 align:middle line:90% So let's give an example. 00:13:35.220 --> 00:13:44.020 align:middle line:84% So to calculate-- to predict on, you'll only look at the words 00:13:44.020 --> 00:13:47.580 align:middle line:90% for the cat sat. 00:13:47.580 --> 00:13:50.380 align:middle line:84% The rest of it will not be considered at all. 00:13:50.380 --> 00:13:53.380 align:middle line:84% Now, the effect of doing all this is that-- 00:13:53.380 --> 00:13:56.000 align:middle line:84% by the way, this is called causal self-attention. 00:13:56.000 --> 00:13:58.820 align:middle line:84% This tweak is called causal self-attention. 00:13:58.820 --> 00:14:01.160 align:middle line:84% It's also called masked self-attention. 00:14:01.160 --> 00:14:03.860 align:middle line:84% Just different labels for the same thing. 00:14:03.860 --> 00:14:05.980 align:middle line:84% And so what that means is that, when 00:14:05.980 --> 00:14:08.353 align:middle line:84% you're looking at the input for the, 00:14:08.353 --> 00:14:12.660 align:middle line:84% only the is going to be used to predict cat. 00:14:12.660 --> 00:14:15.340 align:middle line:84% When you look, the cat, only these two 00:14:15.340 --> 00:14:18.660 align:middle line:84% are going to be used to predict sat and so on, 00:14:18.660 --> 00:14:20.200 align:middle line:90% and so on, and so forth. 00:14:20.200 --> 00:14:24.780 align:middle line:90% 00:14:24.780 --> 00:14:29.740 align:middle line:84% So this thing here, so all we do is we go into our transformer 00:14:29.740 --> 00:14:31.960 align:middle line:84% and we just change each attention head 00:14:31.960 --> 00:14:33.780 align:middle line:90% to be a causal attention head. 00:14:33.780 --> 00:14:38.325 align:middle line:90% 00:14:38.325 --> 00:14:40.200 align:middle line:84% And the way it's actually done under the hood 00:14:40.200 --> 00:14:43.480 align:middle line:84% is actually very elegant for computational efficiency 00:14:43.480 --> 00:14:45.200 align:middle line:84% purposes, but I won't get into it 00:14:45.200 --> 00:14:48.360 align:middle line:90% because it gets a bit involved. 00:14:48.360 --> 00:14:52.520 align:middle line:84% But the key idea is replace basic plain vanilla attention 00:14:52.520 --> 00:14:57.160 align:middle line:84% with causal attention, a.k.a., masked attention. 00:14:57.160 --> 00:14:59.120 align:middle line:84% When you do that, boom, suddenly, it 00:14:59.120 --> 00:15:01.660 align:middle line:84% starts working for an expert prediction. 00:15:01.660 --> 00:15:04.080 align:middle line:90% It can't cheat anymore. 00:15:04.080 --> 00:15:07.490 align:middle line:84% And when we do that, we get the transformer causal encoder. 00:15:07.490 --> 00:15:11.440 align:middle line:90% 00:15:11.440 --> 00:15:13.720 align:middle line:84% And, by the way, the word causal here, there's 00:15:13.720 --> 00:15:15.140 align:middle line:90% no connection to causality. 00:15:15.140 --> 00:15:19.400 align:middle line:90% So it's just a term. 00:15:19.400 --> 00:15:24.240 align:middle line:84% So if you look at the original transformer paper, 00:15:24.240 --> 00:15:27.200 align:middle line:84% it was created for translation, for machine translation, 00:15:27.200 --> 00:15:30.300 align:middle line:84% English to German, those kinds of use cases. 00:15:30.300 --> 00:15:32.820 align:middle line:84% So it had something called an encoder, which we are very 00:15:32.820 --> 00:15:34.340 align:middle line:90% familiar with from last week. 00:15:34.340 --> 00:15:36.940 align:middle line:84% And then it had something called a decoder. 00:15:36.940 --> 00:15:39.758 align:middle line:84% And it was called the encoder-decoder architecture. 00:15:39.758 --> 00:15:42.300 align:middle line:84% And we're not going to cover the encoder-decoder architecture 00:15:42.300 --> 00:15:43.758 align:middle line:84% because we are not covering machine 00:15:43.758 --> 00:15:45.340 align:middle line:90% translation in this class. 00:15:45.340 --> 00:15:50.740 align:middle line:84% But I'm mentioning this because this part of the architecture 00:15:50.740 --> 00:15:54.460 align:middle line:84% is called a decoder because it uses-- 00:15:54.460 --> 00:15:56.460 align:middle line:84% see, here there is a masked attention business 00:15:56.460 --> 00:15:57.660 align:middle line:90% going on here. 00:15:57.660 --> 00:16:01.380 align:middle line:84% Because it is using this masked attention, it's called decoder. 00:16:01.380 --> 00:16:05.140 align:middle line:84% So the transformer encoder is also 00:16:05.140 --> 00:16:07.860 align:middle line:84% referred to sometimes as the transformer decoder. 00:16:07.860 --> 00:16:11.277 align:middle line:84% But the word decoder has two meanings. 00:16:11.277 --> 00:16:12.860 align:middle line:84% It's a synonym for the causal encoder, 00:16:12.860 --> 00:16:13.960 align:middle line:90% like we have seen today. 00:16:13.960 --> 00:16:17.740 align:middle line:84% It's also used to refer to sequence-to-sequence translation 00:16:17.740 --> 00:16:20.515 align:middle line:84% problems for the second part of its architecture. 00:16:20.515 --> 00:16:23.140 align:middle line:84% So you just have to keep it-- it will become clear from context 00:16:23.140 --> 00:16:24.340 align:middle line:90% what we are talking about. 00:16:24.340 --> 00:16:26.340 align:middle line:84% In this course, of course, there is no confusion 00:16:26.340 --> 00:16:28.630 align:middle line:84% because we're not going to be looking at translation. 00:16:28.630 --> 00:16:30.450 align:middle line:84% We may say decoder, causal encoder. 00:16:30.450 --> 00:16:32.297 align:middle line:90% It's the same thing. 00:16:32.297 --> 00:16:34.630 align:middle line:84% AUDIENCE: So I thought there were some transformers that 00:16:34.630 --> 00:16:36.830 align:middle line:90% use bidirectional things. 00:16:36.830 --> 00:16:39.590 align:middle line:84% Did you-- is it different from [INAUDIBLE]? 00:16:39.590 --> 00:16:43.390 align:middle line:84% RAMA RAMAKRISHNAN: No, all bidirectional means 00:16:43.390 --> 00:16:45.910 align:middle line:90% is that I can see everything. 00:16:45.910 --> 00:16:48.570 align:middle line:84% So the encoder we looked at last week, 00:16:48.570 --> 00:16:51.601 align:middle line:84% the basic self-attention thing is bidirectional. 00:16:51.601 --> 00:16:54.377 align:middle line:90% 00:16:54.377 --> 00:16:56.710 align:middle line:84% Basically, all it means is I can look in both directions 00:16:56.710 --> 00:16:57.990 align:middle line:84% to see, what are the words there? 00:16:57.990 --> 00:17:00.448 align:middle line:84% In causal, you're not using the one in the future, correct? 00:17:00.448 --> 00:17:02.510 align:middle line:90% 00:17:02.510 --> 00:17:07.510 align:middle line:84% All right, so to summarize where we are, 00:17:07.510 --> 00:17:09.410 align:middle line:84% this is what we looked at last week for BERT. 00:17:09.410 --> 00:17:11.109 align:middle line:84% And this is the transformer encoder. 00:17:11.109 --> 00:17:12.910 align:middle line:90% And we take the same thing. 00:17:12.910 --> 00:17:14.849 align:middle line:84% And instead of multi-head attention, 00:17:14.849 --> 00:17:16.710 align:middle line:84% we would do causal multi-head attention. 00:17:16.710 --> 00:17:19.169 align:middle line:84% We get the decoder, a.k.a., causal encoder. 00:17:19.169 --> 00:17:21.770 align:middle line:90% 00:17:21.770 --> 00:17:25.930 align:middle line:84% And we use the left for masked prediction. 00:17:25.930 --> 00:17:29.570 align:middle line:84% We use the right for next word prediction. 00:17:29.570 --> 00:17:32.530 align:middle line:84% All right, so now, instead of having 00:17:32.530 --> 00:17:34.550 align:middle line:84% an encoder, if you have a causal encoder, 00:17:34.550 --> 00:17:37.410 align:middle line:84% a TCE here, now we can train models 00:17:37.410 --> 00:17:42.308 align:middle line:84% for expert prediction using the same exact approach as before. 00:17:42.308 --> 00:17:43.850 align:middle line:84% We set up the inputs and the outputs, 00:17:43.850 --> 00:17:44.892 align:middle line:90% like I described earlier. 00:17:44.892 --> 00:17:46.830 align:middle line:84% We run it through a bunch of stacks, 00:17:46.830 --> 00:17:49.810 align:middle line:84% a stack of causal encoders, dense, ReLU, softmax, 00:17:49.810 --> 00:17:51.130 align:middle line:90% and so on and so forth. 00:17:51.130 --> 00:17:54.050 align:middle line:84% Otherwise, the details don't change, 00:17:54.050 --> 00:17:56.610 align:middle line:84% but all important changes go into the attention layer 00:17:56.610 --> 00:17:58.060 align:middle line:90% and make it masked or causal. 00:17:58.060 --> 00:18:02.490 align:middle line:90% 00:18:02.490 --> 00:18:03.550 align:middle line:90% Any questions so far? 00:18:03.550 --> 00:18:06.370 align:middle line:90% 00:18:06.370 --> 00:18:07.950 align:middle line:90% Yeah? 00:18:07.950 --> 00:18:09.450 align:middle line:84% AUDIENCE: This would only apply when 00:18:09.450 --> 00:18:12.060 align:middle line:84% we're training the model, not when we're validating or testing 00:18:12.060 --> 00:18:13.870 align:middle line:90% the model, right? 00:18:13.870 --> 00:18:15.370 align:middle line:84% RAMA RAMAKRISHNAN: So if you give me 00:18:15.370 --> 00:18:19.250 align:middle line:84% a sentence after training, the final prediction 00:18:19.250 --> 00:18:21.645 align:middle line:84% is the only thing you care about. 00:18:21.645 --> 00:18:23.270 align:middle line:84% And by definition, the final prediction 00:18:23.270 --> 00:18:25.430 align:middle line:84% will use everything that came before it. 00:18:25.430 --> 00:18:27.430 align:middle line:90% So we're OK. 00:18:27.430 --> 00:18:28.730 align:middle line:90% Was that your question? 00:18:28.730 --> 00:18:34.030 align:middle line:84% AUDIENCE: No, I think the fact that we're zeroing out 00:18:34.030 --> 00:18:35.495 align:middle line:84% the weights with feature words, I 00:18:35.495 --> 00:18:37.870 align:middle line:84% thought it would apply more when we're training the model 00:18:37.870 --> 00:18:40.750 align:middle line:84% and we're trying to minimize the loss as opposed to when 00:18:40.750 --> 00:18:44.758 align:middle line:84% we're asking ChatGPT what the next word or sentence should be. 00:18:44.758 --> 00:18:45.800 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:18:45.800 --> 00:18:48.010 align:middle line:84% But the point is, when we actually use them, 00:18:48.010 --> 00:18:48.990 align:middle line:90% what is the objective? 00:18:48.990 --> 00:18:51.290 align:middle line:84% What do we want to do when we actually use them for inference, 00:18:51.290 --> 00:18:52.490 align:middle line:90% once we finish training? 00:18:52.490 --> 00:18:55.230 align:middle line:84% Our objective is given a particular string. 00:18:55.230 --> 00:18:57.170 align:middle line:90% Get me the next word. 00:18:57.170 --> 00:18:59.330 align:middle line:84% And to find the next word, you can, in fact, 00:18:59.330 --> 00:19:01.110 align:middle line:84% use everything that came before it. 00:19:01.110 --> 00:19:03.370 align:middle line:84% And, therefore, without any change to this model, 00:19:03.370 --> 00:19:05.398 align:middle line:84% it'll just work for your intended purpose. 00:19:05.398 --> 00:19:07.190 align:middle line:84% You don't have to go in there and change it 00:19:07.190 --> 00:19:09.887 align:middle line:84% to-- you don't have to unmask it for inference 00:19:09.887 --> 00:19:10.970 align:middle line:90% because you don't need to. 00:19:10.970 --> 00:19:13.510 align:middle line:90% 00:19:13.510 --> 00:19:14.850 align:middle line:90% Yes? 00:19:14.850 --> 00:19:16.350 align:middle line:84% AUDIENCE: I have one question that's 00:19:16.350 --> 00:19:20.410 align:middle line:84% regarding when we do the puzzle transformers, 00:19:20.410 --> 00:19:22.323 align:middle line:84% we are putting certain weights to 0 00:19:22.323 --> 00:19:24.490 align:middle line:84% for the words which are to be predicted, and then we 00:19:24.490 --> 00:19:24.730 align:middle line:90% [INAUDIBLE]. 00:19:24.730 --> 00:19:26.980 align:middle line:84% RAMA RAMAKRISHNAN: No, the words that are in the future. 00:19:26.980 --> 00:19:27.570 align:middle line:90% AUDIENCE: Future. 00:19:27.570 --> 00:19:27.930 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:19:27.930 --> 00:19:29.170 align:middle line:84% AUDIENCE: And then we normalize it. 00:19:29.170 --> 00:19:29.510 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:19:29.510 --> 00:19:32.290 align:middle line:84% AUDIENCE: And if trained a transformer earlier on all 00:19:32.290 --> 00:19:34.310 align:middle line:84% the words, all the words together, 00:19:34.310 --> 00:19:36.490 align:middle line:84% so won't be there be a difference in weights 00:19:36.490 --> 00:19:37.730 align:middle line:90% between both of these? 00:19:37.730 --> 00:19:39.230 align:middle line:84% RAMA RAMAKRISHNAN: Between the two ways of training? 00:19:39.230 --> 00:19:41.022 align:middle line:84% The weights are going to be very different. 00:19:41.022 --> 00:19:43.170 align:middle line:84% And they are two different models. 00:19:43.170 --> 00:19:44.827 align:middle line:90% BERT is used for certain things. 00:19:44.827 --> 00:19:46.910 align:middle line:84% And this kind of model, which is the basis of GPT, 00:19:46.910 --> 00:19:48.030 align:middle line:84% is going to be used for other things. 00:19:48.030 --> 00:19:50.090 align:middle line:84% AUDIENCE: We are training it as well like that, 00:19:50.090 --> 00:19:53.650 align:middle line:84% by putting [INAUDIBLE] by moving some of the weights to 0. 00:19:53.650 --> 00:19:54.910 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:19:54.910 --> 00:19:57.850 align:middle line:84% So what I'm talking about here is 00:19:57.850 --> 00:20:00.090 align:middle line:84% the-- what we are trying to do here is to say, 00:20:00.090 --> 00:20:01.850 align:middle line:84% let's say that we want to do next word 00:20:01.850 --> 00:20:06.170 align:middle line:84% prediction as the task, as a self-supervised learning task. 00:20:06.170 --> 00:20:07.930 align:middle line:84% And we want to train such a model 00:20:07.930 --> 00:20:09.947 align:middle line:90% on a vast amount of text data. 00:20:09.947 --> 00:20:12.530 align:middle line:84% Well, we can't just use what we did last week because it's not 00:20:12.530 --> 00:20:14.830 align:middle line:84% going to work because of the fact you can see the future. 00:20:14.830 --> 00:20:15.955 align:middle line:90% Therefore, we make a tweak. 00:20:15.955 --> 00:20:17.163 align:middle line:90% And then we build this model. 00:20:17.163 --> 00:20:18.580 align:middle line:84% Now the question becomes, OK, what 00:20:18.580 --> 00:20:19.790 align:middle line:90% can we do with such a model? 00:20:19.790 --> 00:20:21.110 align:middle line:84% We have basically trained two different kinds 00:20:21.110 --> 00:20:23.210 align:middle line:84% of models, the one that can see everything, 00:20:23.210 --> 00:20:26.670 align:middle line:84% BERT, and the one that can't see the future, which is actually 00:20:26.670 --> 00:20:27.285 align:middle line:90% GPT. 00:20:27.285 --> 00:20:28.410 align:middle line:90% So what can you do with it? 00:20:28.410 --> 00:20:29.743 align:middle line:90% And we're going to come to that. 00:20:29.743 --> 00:20:33.910 align:middle line:90% 00:20:33.910 --> 00:20:38.470 align:middle line:84% All right, so now once you train such a model, given any input 00:20:38.470 --> 00:20:43.470 align:middle line:84% sentence, let's say that the sentence is, it was a dark and. 00:20:43.470 --> 00:20:45.430 align:middle line:90% It was a dark and. 00:20:45.430 --> 00:20:47.590 align:middle line:84% It goes through all these things. 00:20:47.590 --> 00:20:48.970 align:middle line:84% And remember what I said earlier. 00:20:48.970 --> 00:20:52.790 align:middle line:84% The fact that it's predicting something after just seeing it, 00:20:52.790 --> 00:20:55.870 align:middle line:90% we don't really care. 00:20:55.870 --> 00:20:57.710 align:middle line:84% What we're really curious about is, 00:20:57.710 --> 00:20:59.690 align:middle line:84% what is the next thing it's going to say? 00:20:59.690 --> 00:21:01.190 align:middle line:84% And the next thing it's going to say 00:21:01.190 --> 00:21:02.670 align:middle line:84% is going to be, basically, what's 00:21:02.670 --> 00:21:03.795 align:middle line:90% coming out of this softmax. 00:21:03.795 --> 00:21:06.750 align:middle line:90% 00:21:06.750 --> 00:21:07.895 align:middle line:90% Does it make sense? 00:21:07.895 --> 00:21:09.270 align:middle line:84% We don't care about anything that 00:21:09.270 --> 00:21:12.630 align:middle line:84% went before it because we already 00:21:12.630 --> 00:21:14.835 align:middle line:90% have a half-formed sentence. 00:21:14.835 --> 00:21:16.710 align:middle line:84% And we want to just find the next thing here. 00:21:16.710 --> 00:21:18.830 align:middle line:90% So we only care about this. 00:21:18.830 --> 00:21:22.030 align:middle line:84% These things will come out of the architecture of the model, 00:21:22.030 --> 00:21:22.950 align:middle line:90% but we throw them out. 00:21:22.950 --> 00:21:24.930 align:middle line:84% We don't even pay any attention to them. 00:21:24.930 --> 00:21:27.730 align:middle line:84% We only look at what's coming out in this one here. 00:21:27.730 --> 00:21:31.430 align:middle line:84% And what comes out of that softmax, remember, 00:21:31.430 --> 00:21:34.610 align:middle line:84% is a 50,000 way table of probabilities. 00:21:34.610 --> 00:21:36.730 align:middle line:90% That's what a softmax is. 00:21:36.730 --> 00:21:39.050 align:middle line:84% It's a whole bunch of probabilities that add up to 1. 00:21:39.050 --> 00:21:41.190 align:middle line:84% And so it's going to-- and let's say, for example, 00:21:41.190 --> 00:21:45.330 align:middle line:84% that you have starting with aardvark all the way to zebra. 00:21:45.330 --> 00:21:47.970 align:middle line:90% And these are the probabilities. 00:21:47.970 --> 00:21:52.810 align:middle line:84% So it was a dark and, just for kicks, 00:21:52.810 --> 00:21:55.065 align:middle line:90% I put stormy as the most-- 00:21:55.065 --> 00:21:56.190 align:middle line:90% highest probability number. 00:21:56.190 --> 00:21:58.250 align:middle line:84% But these numbers will add up to 1. 00:21:58.250 --> 00:22:00.050 align:middle line:90% We have this table. 00:22:00.050 --> 00:22:04.090 align:middle line:84% And then what we do is we choose a token from this table. 00:22:04.090 --> 00:22:05.810 align:middle line:90% We get to choose. 00:22:05.810 --> 00:22:07.850 align:middle line:84% There's a whole bunch of numbers in this table, 00:22:07.850 --> 00:22:10.130 align:middle line:90% and we get to choose a token. 00:22:10.130 --> 00:22:11.770 align:middle line:84% The simplest thing one can think of 00:22:11.770 --> 00:22:14.830 align:middle line:84% is just choose the word that is the most likely, 00:22:14.830 --> 00:22:16.920 align:middle line:84% and we choose the word that's most likely here. 00:22:16.920 --> 00:22:19.820 align:middle line:84% And we're going to have a whole section on how to choose 00:22:19.820 --> 00:22:21.732 align:middle line:90% these things coming up. 00:22:21.732 --> 00:22:23.440 align:middle line:84% For now, let's go with the simple option. 00:22:23.440 --> 00:22:26.620 align:middle line:84% We're going to just choose the one that's most likely, 0.6. 00:22:26.620 --> 00:22:30.780 align:middle line:84% And then we attach it to the input. 00:22:30.780 --> 00:22:33.180 align:middle line:84% So now the input has become, it was a dark and stormy. 00:22:33.180 --> 00:22:34.020 align:middle line:90% We run it through. 00:22:34.020 --> 00:22:36.710 align:middle line:84% And, again, we only care about the last one softmax. 00:22:36.710 --> 00:22:40.050 align:middle line:90% 00:22:40.050 --> 00:22:42.380 align:middle line:84% We do that, we get another table. 00:22:42.380 --> 00:22:44.100 align:middle line:84% And this table turns out-- the table 00:22:44.100 --> 00:22:45.700 align:middle line:84% keeps changing because the softmax is 00:22:45.700 --> 00:22:47.450 align:middle line:84% different for each time you run it through 00:22:47.450 --> 00:22:49.020 align:middle line:90% because the input has changed. 00:22:49.020 --> 00:22:50.820 align:middle line:84% So you get a new table, and it turns out 00:22:50.820 --> 00:22:53.300 align:middle line:90% the most likely one is night. 00:22:53.300 --> 00:22:55.460 align:middle line:90% And then we attached-- 00:22:55.460 --> 00:22:58.180 align:middle line:84% so night comes out the other end. 00:22:58.180 --> 00:23:02.432 align:middle line:84% And we attached night here, and we keep on going. 00:23:02.432 --> 00:23:05.500 align:middle line:84% We can keep on going maybe till we basically 00:23:05.500 --> 00:23:09.340 align:middle line:84% tell the model, OK, generate up to 100 tokens and stop. 00:23:09.340 --> 00:23:11.140 align:middle line:90% It might stop after 100. 00:23:11.140 --> 00:23:13.720 align:middle line:84% Or it might decide, the model may decide, in fact, that when 00:23:13.720 --> 00:23:16.920 align:middle line:84% it sees a punctuation, like a period, or an exclamation mark, 00:23:16.920 --> 00:23:18.640 align:middle line:84% or something, that's going to stop. 00:23:18.640 --> 00:23:22.300 align:middle line:84% And we have control over this when it stops and how it stops. 00:23:22.300 --> 00:23:25.378 align:middle line:84% But this is sort of the basic process. 00:23:25.378 --> 00:23:26.920 align:middle line:84% And you folks are all very used to it 00:23:26.920 --> 00:23:29.912 align:middle line:84% because you've all been playing with ChatGPT and the like. 00:23:29.912 --> 00:23:33.060 align:middle line:84% But the basic building block is next word prediction, 00:23:33.060 --> 00:23:35.060 align:middle line:84% feed it back to the input, next word prediction, 00:23:35.060 --> 00:23:36.820 align:middle line:90% keep on doing it. 00:23:36.820 --> 00:23:38.560 align:middle line:84% You keep on doing it, then, suddenly, 00:23:38.560 --> 00:23:41.720 align:middle line:84% it's writing entire novels for you. 00:23:41.720 --> 00:23:43.620 align:middle line:90% Yeah? 00:23:43.620 --> 00:23:46.700 align:middle line:84% AUDIENCE: The longer the initial input is, it's better. 00:23:46.700 --> 00:23:48.700 align:middle line:84% It's going to give you a better prediction? 00:23:48.700 --> 00:23:50.700 align:middle line:84% RAMA RAMAKRISHNAN: It depends on your objective. 00:23:50.700 --> 00:23:54.120 align:middle line:84% So, fundamentally, you have some task 00:23:54.120 --> 00:23:55.950 align:middle line:84% you want the thing to do for you. 00:23:55.950 --> 00:23:59.320 align:middle line:84% And that task may-- and you need to give it all the information 00:23:59.320 --> 00:24:01.020 align:middle line:90% it can possibly find useful. 00:24:01.020 --> 00:24:02.040 align:middle line:90% Yeah, so the long-- 00:24:02.040 --> 00:24:04.060 align:middle line:84% the more helpful the input, the better. 00:24:04.060 --> 00:24:07.360 align:middle line:90% Maybe that's how I would say it. 00:24:07.360 --> 00:24:09.087 align:middle line:90% Yeah? 00:24:09.087 --> 00:24:10.920 align:middle line:84% AUDIENCE: Would this also apply to something 00:24:10.920 --> 00:24:14.060 align:middle line:90% like Google Search or-- 00:24:14.060 --> 00:24:16.640 align:middle line:84% because they also do next letter prediction too, 00:24:16.640 --> 00:24:17.980 align:middle line:90% but with this [INAUDIBLE]. 00:24:17.980 --> 00:24:20.147 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so the Google autocomplete, 00:24:20.147 --> 00:24:21.940 align:middle line:84% for example, I don't know if they actually 00:24:21.940 --> 00:24:24.520 align:middle line:84% use this kind of model under the hood or not. 00:24:24.520 --> 00:24:26.100 align:middle line:90% I just don't know. 00:24:26.100 --> 00:24:28.740 align:middle line:84% These things tend to be kept tightly under wraps. 00:24:28.740 --> 00:24:30.780 align:middle line:84% If they were to-- if they were using it, 00:24:30.780 --> 00:24:34.940 align:middle line:84% my guess is that they-- so I don't know if you folks have 00:24:34.940 --> 00:24:37.340 align:middle line:84% seen, recently, over the last few months, they have-- 00:24:37.340 --> 00:24:39.580 align:middle line:84% there is a generative AI panel that opens up 00:24:39.580 --> 00:24:41.060 align:middle line:90% when you do a Google search. 00:24:41.060 --> 00:24:43.860 align:middle line:84% That panel, I suspect, uses this, 00:24:43.860 --> 00:24:46.460 align:middle line:84% but I don't know if the default Google autocomplete actually 00:24:46.460 --> 00:24:50.060 align:middle line:84% uses it or not because it's very compute heavy. 00:24:50.060 --> 00:24:53.700 align:middle line:90% So I don't know what they do. 00:24:53.700 --> 00:24:55.040 align:middle line:90% So, yeah, this is what you do. 00:24:55.040 --> 00:24:59.940 align:middle line:84% Other questions on this, on the mechanics of it? 00:24:59.940 --> 00:25:01.660 align:middle line:90% Yeah? 00:25:01.660 --> 00:25:03.100 align:middle line:84% AUDIENCE: For our vocabulary list, 00:25:03.100 --> 00:25:05.045 align:middle line:84% I'm assuming it's static [INAUDIBLE]. 00:25:05.045 --> 00:25:06.420 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, correct. 00:25:06.420 --> 00:25:08.960 align:middle line:84% And as you will see here, it's not really a word vocabulary. 00:25:08.960 --> 00:25:10.720 align:middle line:84% It's a token vocabulary, but, yes, it 00:25:10.720 --> 00:25:13.020 align:middle line:90% is static for a given model. 00:25:13.020 --> 00:25:15.080 align:middle line:84% AUDIENCE: And I guess I'm assuming 00:25:15.080 --> 00:25:18.720 align:middle line:84% for Google or any other sort of search engine, that 00:25:18.720 --> 00:25:20.380 align:middle line:90% wouldn't necessarily be static. 00:25:20.380 --> 00:25:24.440 align:middle line:84% And so when it comes to, I guess, base-- 00:25:24.440 --> 00:25:27.200 align:middle line:84% or even [INAUDIBLE] because the model 00:25:27.200 --> 00:25:29.462 align:middle line:90% would be different [INAUDIBLE]. 00:25:29.462 --> 00:25:30.920 align:middle line:84% RAMA RAMAKRISHNAN: Sort of thinking 00:25:30.920 --> 00:25:34.480 align:middle line:84% about what happens to new words and things that are formed? 00:25:34.480 --> 00:25:36.940 align:middle line:84% And how does it handle it if the vocabulary is static? 00:25:36.940 --> 00:25:39.060 align:middle line:84% There's a very elegant solution that's coming up. 00:25:39.060 --> 00:25:45.000 align:middle line:90% 00:25:45.000 --> 00:25:48.100 align:middle line:84% All right, so now, in other words, 00:25:48.100 --> 00:25:51.360 align:middle line:84% we have learned how to do sequence generation. 00:25:51.360 --> 00:25:54.140 align:middle line:84% We already saw that we can do classification with BERT. 00:25:54.140 --> 00:25:56.520 align:middle line:84% We can do labeling with BERT, BERT-like models, which 00:25:56.520 --> 00:25:58.320 align:middle line:90% are trained on mass prediction. 00:25:58.320 --> 00:26:00.928 align:middle line:84% And for generating sequences, now we know how to do it. 00:26:00.928 --> 00:26:02.970 align:middle line:84% We just need to use a transformer causal encoder. 00:26:02.970 --> 00:26:07.700 align:middle line:90% 00:26:07.700 --> 00:26:13.140 align:middle line:84% Now, these kind of models, sequence generation models 00:26:13.140 --> 00:26:15.520 align:middle line:84% trained on text sequences using expert prediction, 00:26:15.520 --> 00:26:17.740 align:middle line:84% are called autoregressive language models 00:26:17.740 --> 00:26:19.980 align:middle line:90% or causal language models. 00:26:19.980 --> 00:26:22.220 align:middle line:84% And, of course, the GPT family is perhaps 00:26:22.220 --> 00:26:26.860 align:middle line:84% the most well-known example of an autoregressive language 00:26:26.860 --> 00:26:29.660 align:middle line:84% model, autoregressive because people 00:26:29.660 --> 00:26:32.140 align:middle line:84% who have done econometrics and some regression 00:26:32.140 --> 00:26:33.740 align:middle line:84% know the notion of autoregression 00:26:33.740 --> 00:26:35.540 align:middle line:84% means that you predict something, 00:26:35.540 --> 00:26:39.420 align:middle line:84% and then you use sort of the past predictions as inputs 00:26:39.420 --> 00:26:41.220 align:middle line:90% into the next time you predict. 00:26:41.220 --> 00:26:44.080 align:middle line:84% So this is the notion of autoregression. 00:26:44.080 --> 00:26:46.440 align:middle line:84% You predict, you feed the prediction back, 00:26:46.440 --> 00:26:48.680 align:middle line:84% get the next prediction, and keep on cycling through. 00:26:48.680 --> 00:26:49.180 align:middle line:90% Yes? 00:26:49.180 --> 00:26:51.900 align:middle line:90% 00:26:51.900 --> 00:26:55.840 align:middle line:84% AUDIENCE: So when you put it in as inputs into GPT, for example, 00:26:55.840 --> 00:26:57.820 align:middle line:90% and it has that-- 00:26:57.820 --> 00:27:00.940 align:middle line:84% it shows you the next words as it's come in. 00:27:00.940 --> 00:27:04.440 align:middle line:84% Is that an indication of it doing this recalculation 00:27:04.440 --> 00:27:05.398 align:middle line:90% and what you described? 00:27:05.398 --> 00:27:06.523 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:27:06.523 --> 00:27:07.840 align:middle line:90% That's exactly what's going on. 00:27:07.840 --> 00:27:09.600 align:middle line:84% In fact, if you use the API, that 00:27:09.600 --> 00:27:11.840 align:middle line:84% is the thing called the streaming API, 00:27:11.840 --> 00:27:14.000 align:middle line:84% where it will actually stream each token that's 00:27:14.000 --> 00:27:16.080 align:middle line:90% coming out through every pass. 00:27:16.080 --> 00:27:18.200 align:middle line:84% And you can actually see everything very clearly. 00:27:18.200 --> 00:27:21.080 align:middle line:84% But when you actually work with the web interface, 00:27:21.080 --> 00:27:23.600 align:middle line:84% and you see the thing almost as if it's typing, 00:27:23.600 --> 00:27:25.412 align:middle line:84% like a human, what I've heard from people-- 00:27:25.412 --> 00:27:26.620 align:middle line:90% I don't know if this is true. 00:27:26.620 --> 00:27:28.880 align:middle line:84% What I've heard from people is that they can actually 00:27:28.880 --> 00:27:30.160 align:middle line:90% do it much faster. 00:27:30.160 --> 00:27:32.000 align:middle line:84% They slow it down intentionally to give you 00:27:32.000 --> 00:27:34.125 align:middle line:84% the feeling that it's actually coming from a human. 00:27:34.125 --> 00:27:36.440 align:middle line:90% 00:27:36.440 --> 00:27:39.583 align:middle line:84% So it's like a UX trick to slow it down, 00:27:39.583 --> 00:27:42.000 align:middle line:84% to make it feel as if someone is actually typing something 00:27:42.000 --> 00:27:43.160 align:middle line:90% on the other end. 00:27:43.160 --> 00:27:45.500 align:middle line:84% So when you're interacting with a chatbot, for example, 00:27:45.500 --> 00:27:47.483 align:middle line:84% sometimes, you will see it actually typing. 00:27:47.483 --> 00:27:49.900 align:middle line:84% Slowly, you can see the bubble and you can see the typing. 00:27:49.900 --> 00:27:51.317 align:middle line:84% It's actually intentionally slowed 00:27:51.317 --> 00:27:54.240 align:middle line:84% down because you know it's a bot otherwise. 00:27:54.240 --> 00:28:00.720 align:middle line:84% So there's a little bit of UX creepiness, maybe, going on. 00:28:00.720 --> 00:28:03.100 align:middle line:84% I don't know to what extent this is 100% true 00:28:03.100 --> 00:28:05.730 align:middle line:84% and how pervasive it is, but folks who work in the field 00:28:05.730 --> 00:28:07.900 align:middle line:84% have told me that this actually is not uncommon. 00:28:07.900 --> 00:28:11.098 align:middle line:90% 00:28:11.098 --> 00:28:12.390 align:middle line:90% So that's what's going on here. 00:28:12.390 --> 00:28:13.630 align:middle line:90% These are language models. 00:28:13.630 --> 00:28:17.790 align:middle line:84% And, of course, GPT-3 is an autoregressive language model. 00:28:17.790 --> 00:28:21.010 align:middle line:84% And the reason why we have an L in front of the LLM, 00:28:21.010 --> 00:28:24.978 align:middle line:84% because it was trained on lots of data with lots of parameters. 00:28:24.978 --> 00:28:27.270 align:middle line:84% At some point, it's not a small language model anymore. 00:28:27.270 --> 00:28:28.437 align:middle line:90% It's a large language model. 00:28:28.437 --> 00:28:30.010 align:middle line:90% Yeah, so it's LLM. 00:28:30.010 --> 00:28:32.490 align:middle line:84% Nothing more momentous than that. 00:28:32.490 --> 00:28:38.890 align:middle line:84% So as it turns out, GPT-3 uses 96 transformer blocks, 00:28:38.890 --> 00:28:40.270 align:middle line:90% 96 blocks,. 00:28:40.270 --> 00:28:44.890 align:middle line:84% And each block has 96 causal attention heads. 00:28:44.890 --> 00:28:47.590 align:middle line:84% And you can see-- you can read the GPT-3 paper. 00:28:47.590 --> 00:28:49.650 align:middle line:84% It gives you all the details of the architecture. 00:28:49.650 --> 00:28:51.608 align:middle line:84% That is interesting because, GPT-4, they didn't 00:28:51.608 --> 00:28:53.330 align:middle line:90% publish the architecture. 00:28:53.330 --> 00:28:56.972 align:middle line:84% From GPT-3-- after GPT-3, everything became closed. 00:28:56.972 --> 00:28:58.930 align:middle line:84% So we actually don't know what the architecture 00:28:58.930 --> 00:29:02.010 align:middle line:84% is, even though there's a lot of speculation on Twitter. 00:29:02.010 --> 00:29:04.070 align:middle line:84% But GPT-3, we know exactly what happened. 00:29:04.070 --> 00:29:05.910 align:middle line:90% It's 96 blocks. 00:29:05.910 --> 00:29:08.870 align:middle line:84% Each has 96 causal attention heads. 00:29:08.870 --> 00:29:10.710 align:middle line:84% And then the data was actually, they 00:29:10.710 --> 00:29:13.950 align:middle line:84% scraped $30 billion sentences from a whole bunch 00:29:13.950 --> 00:29:18.030 align:middle line:84% of sources web text, Wikipedia, a bunch of book databases. 00:29:18.030 --> 00:29:22.550 align:middle line:84% And then they basically just took those 30 billion sentences 00:29:22.550 --> 00:29:25.150 align:middle line:84% and just trained it exactly, next word prediction. 00:29:25.150 --> 00:29:26.888 align:middle line:90% That's it. 00:29:26.888 --> 00:29:28.430 align:middle line:84% Now, when they trained GPT-3, I think 00:29:28.430 --> 00:29:33.567 align:middle line:84% it cost them a lot of money because things were not as-- we 00:29:33.567 --> 00:29:36.150 align:middle line:84% hadn't figured out how to do it as efficiently as we know now, 00:29:36.150 --> 00:29:38.070 align:middle line:90% but it was still pretty amazing. 00:29:38.070 --> 00:29:40.390 align:middle line:84% And I'll talk about what is so special about GPT-3 00:29:40.390 --> 00:29:41.910 align:middle line:90% in just a minute or two. 00:29:41.910 --> 00:29:44.110 align:middle line:90% So this is what we have here. 00:29:44.110 --> 00:29:48.790 align:middle line:84% And as you folks have seen, the notion of generating text 00:29:48.790 --> 00:29:52.330 align:middle line:84% is very powerful because we can obviously generate text. 00:29:52.330 --> 00:29:55.510 align:middle line:84% But you can also generate code because code is just text. 00:29:55.510 --> 00:29:57.250 align:middle line:84% We can generate documentation for code. 00:29:57.250 --> 00:29:58.170 align:middle line:90% We can summarize text. 00:29:58.170 --> 00:29:59.170 align:middle line:90% We can answer questions. 00:29:59.170 --> 00:30:00.030 align:middle line:90% We can do chatbot. 00:30:00.030 --> 00:30:00.830 align:middle line:90% The list goes on. 00:30:00.830 --> 00:30:04.810 align:middle line:84% All the excitement we see around GenAI from the time ChatGPT came 00:30:04.810 --> 00:30:08.310 align:middle line:84% out is precisely because this simple idea of text in, 00:30:08.310 --> 00:30:11.930 align:middle line:90% text out is just so flexible. 00:30:11.930 --> 00:30:12.910 align:middle line:90% It's so versatile. 00:30:12.910 --> 00:30:14.550 align:middle line:84% It can handle all sorts of use cases. 00:30:14.550 --> 00:30:17.810 align:middle line:84% That's why there's so much excitement. 00:30:17.810 --> 00:30:20.595 align:middle line:84% By the way, if you're really curious, 00:30:20.595 --> 00:30:21.970 align:middle line:84% I would actually recommend seeing 00:30:21.970 --> 00:30:26.050 align:middle line:84% this video, where this guy, Andrej Karpathy, 00:30:26.050 --> 00:30:29.290 align:middle line:90% builds GPT from scratch. 00:30:29.290 --> 00:30:31.210 align:middle line:90% It's a fantastic video. 00:30:31.210 --> 00:30:33.880 align:middle line:84% If you have even a little bit of curiosity 00:30:33.880 --> 00:30:35.630 align:middle line:84% about how these things are actually built, 00:30:35.630 --> 00:30:37.810 align:middle line:84% I would strongly recommend checking it out. 00:30:37.810 --> 00:30:40.410 align:middle line:84% And there's also a little blog post where this person-- 00:30:40.410 --> 00:30:42.210 align:middle line:84% basically, if you know NumPy, you 00:30:42.210 --> 00:30:44.810 align:middle line:84% can actually create GPT-3, GPT, using NumPy 00:30:44.810 --> 00:30:47.510 align:middle line:84% without using any frameworks and things like that. 00:30:47.510 --> 00:30:53.010 align:middle line:84% So I found it super interesting and helpful to understand 00:30:53.010 --> 00:30:56.470 align:middle line:84% what exactly is going on, so if you would like to do this. 00:30:56.470 --> 00:30:59.750 align:middle line:84% So now we're going to talk about decoding sampling 00:30:59.750 --> 00:31:03.830 align:middle line:84% strategies, which is, I said that when we produce-- 00:31:03.830 --> 00:31:07.570 align:middle line:84% when we come up with the softmax for that last token, 00:31:07.570 --> 00:31:08.810 align:middle line:90% we have 50,000 choices. 00:31:08.810 --> 00:31:11.270 align:middle line:90% What do we pick? 00:31:11.270 --> 00:31:14.310 align:middle line:84% As it turns out, to actually get really good performance out 00:31:14.310 --> 00:31:17.190 align:middle line:84% of GenAI systems, like ChatGPT, you 00:31:17.190 --> 00:31:19.870 align:middle line:84% need to be quite thoughtful about how to decode, 00:31:19.870 --> 00:31:21.690 align:middle line:84% how to actually sample from that table. 00:31:21.690 --> 00:31:23.710 align:middle line:84% So we'll talk about that for a bit. 00:31:23.710 --> 00:31:28.587 align:middle line:84% So first of all, definition, the process of choosing a token from 00:31:28.587 --> 00:31:30.670 align:middle line:84% the probability distribution from the-- coming out 00:31:30.670 --> 00:31:31.530 align:middle line:90% of the softmax-- 00:31:31.530 --> 00:31:33.950 align:middle line:84% I'm sticking this table right here. 00:31:33.950 --> 00:31:35.650 align:middle line:90% This is the softmax. 00:31:35.650 --> 00:31:38.070 align:middle line:84% This process of choosing it is called decoding. 00:31:38.070 --> 00:31:40.110 align:middle line:84% That's the technical term for it. 00:31:40.110 --> 00:31:42.090 align:middle line:84% We get this table, we have to decode, 00:31:42.090 --> 00:31:44.530 align:middle line:84% meaning we have to pick something from this table. 00:31:44.530 --> 00:31:46.310 align:middle line:90% That's called decoding. 00:31:46.310 --> 00:31:51.630 align:middle line:84% Now, there are two sort of extreme cases of very highly 00:31:51.630 --> 00:31:53.270 align:middle line:90% simple ways to do this. 00:31:53.270 --> 00:31:54.790 align:middle line:84% The first thing, of course, is just 00:31:54.790 --> 00:31:56.330 align:middle line:84% pick the one-- just pick the word 00:31:56.330 --> 00:31:58.770 align:middle line:90% with the highest probability. 00:31:58.770 --> 00:32:00.260 align:middle line:90% This is called greedy decoding. 00:32:00.260 --> 00:32:03.540 align:middle line:90% 00:32:03.540 --> 00:32:05.970 align:middle line:84% So in this case, for example, if stormy 00:32:05.970 --> 00:32:08.330 align:middle line:84% is 0.6, the highest probability in this whole table, 00:32:08.330 --> 00:32:10.450 align:middle line:90% we just pick stormy. 00:32:10.450 --> 00:32:14.170 align:middle line:84% So that is the obvious extreme simple case. 00:32:14.170 --> 00:32:16.810 align:middle line:84% The other thing we can do, which is also super simple, 00:32:16.810 --> 00:32:19.090 align:middle line:84% is that, because we have a probability table 00:32:19.090 --> 00:32:22.450 align:middle line:84% here, we can just reach into the table and sample a word out 00:32:22.450 --> 00:32:26.530 align:middle line:84% of it in proportion to its probability, which 00:32:26.530 --> 00:32:28.250 align:middle line:84% means that, if you have this table 00:32:28.250 --> 00:32:32.450 align:middle line:84% and you're sampling from it, if you sample from it 100 times, 60 00:32:32.450 --> 00:32:36.090 align:middle line:84% times, you probably get stormy because the probability is 0.6. 00:32:36.090 --> 00:32:37.730 align:middle line:84% But some small fraction of the time, 00:32:37.730 --> 00:32:40.330 align:middle line:84% you may get strange things, like aardvark, and zebra, 00:32:40.330 --> 00:32:42.770 align:middle line:90% and so on and so forth. 00:32:42.770 --> 00:32:46.530 align:middle line:84% You're just literally doing random sampling. 00:32:46.530 --> 00:32:48.490 align:middle line:90% That's a fine way to do it too. 00:32:48.490 --> 00:32:48.990 align:middle line:90% Right. 00:32:48.990 --> 00:32:50.350 align:middle line:90% There's nothing wrong with that. 00:32:50.350 --> 00:32:52.410 align:middle line:90% So these are both options. 00:32:52.410 --> 00:32:56.110 align:middle line:84% So the key thing you need to remember is that-- 00:32:56.110 --> 00:32:57.350 align:middle line:90% which one you pick. 00:32:57.350 --> 00:32:58.850 align:middle line:84% And there are some variations on it, 00:32:58.850 --> 00:33:00.230 align:middle line:90% which we'll get to in a moment. 00:33:00.230 --> 00:33:02.150 align:middle line:84% What you pick, which way to decode your pick, 00:33:02.150 --> 00:33:04.510 align:middle line:84% really depends on what your task is, 00:33:04.510 --> 00:33:08.470 align:middle line:84% what you're trying to use the system for, the LLM for. 00:33:08.470 --> 00:33:11.070 align:middle line:84% So the broad thing to remember is 00:33:11.070 --> 00:33:13.830 align:middle line:84% that, if you're working on questions 00:33:13.830 --> 00:33:17.270 align:middle line:84% for which the factual accuracy of the response 00:33:17.270 --> 00:33:21.870 align:middle line:84% is really important and/or you want the output to be 00:33:21.870 --> 00:33:24.390 align:middle line:84% deterministic, meaning every time you ask 00:33:24.390 --> 00:33:25.950 align:middle line:84% it a particular question, you really 00:33:25.950 --> 00:33:28.430 align:middle line:90% want the same answer back. 00:33:28.430 --> 00:33:30.230 align:middle line:84% You can imagine a customer call support 00:33:30.230 --> 00:33:33.930 align:middle line:84% agent, where the two different customers ask the same question, 00:33:33.930 --> 00:33:36.150 align:middle line:90% and they get different answers. 00:33:36.150 --> 00:33:37.390 align:middle line:90% You don't want that. 00:33:37.390 --> 00:33:39.430 align:middle line:84% So you want deterministic outputs. 00:33:39.430 --> 00:33:42.170 align:middle line:84% So in those situations, you should use greedy decoding. 00:33:42.170 --> 00:33:44.910 align:middle line:84% It's a good starting point because you will get-- 00:33:44.910 --> 00:33:49.430 align:middle line:84% you won't get any random stuff because, for any given input 00:33:49.430 --> 00:33:52.450 align:middle line:84% sentence, the softmax that comes out of the table 00:33:52.450 --> 00:33:53.970 align:middle line:90% is not going to change. 00:33:53.970 --> 00:33:55.128 align:middle line:90% It's the same table. 00:33:55.128 --> 00:33:57.670 align:middle line:84% And if you're always picking the highest number in the table, 00:33:57.670 --> 00:33:59.610 align:middle line:84% that's not going to change either. 00:33:59.610 --> 00:34:03.010 align:middle line:90% So guaranteed determinism. 00:34:03.010 --> 00:34:05.770 align:middle line:84% And I've found that for reasoning questions and things 00:34:05.770 --> 00:34:08.770 align:middle line:84% where you're asking questions, math questions, reasoning 00:34:08.770 --> 00:34:11.030 align:middle line:84% questions, logic questions, you should really 00:34:11.030 --> 00:34:15.175 align:middle line:84% sort of keep it as greedy as possible in my experience. 00:34:15.175 --> 00:34:18.570 align:middle line:84% Now, there are other situations where random sampling 00:34:18.570 --> 00:34:20.610 align:middle line:90% is actually a better option. 00:34:20.610 --> 00:34:23.830 align:middle line:84% If you're doing creative things, write a poem, write a haiku, 00:34:23.830 --> 00:34:25.810 align:middle line:84% write a screenplay, things like that, you 00:34:25.810 --> 00:34:28.330 align:middle line:84% do want a lot of creativity, in which case, actually, 00:34:28.330 --> 00:34:30.346 align:middle line:90% randomness is your friend. 00:34:30.346 --> 00:34:32.429 align:middle line:84% You get a lot of different varieties of responses, 00:34:32.429 --> 00:34:33.389 align:middle line:90% diversity of responses. 00:34:33.389 --> 00:34:34.850 align:middle line:90% All that is really good. 00:34:34.850 --> 00:34:38.245 align:middle line:84% The price you pay for it is that, you lose determinism, 00:34:38.245 --> 00:34:39.870 align:middle line:84% the outputs are going to be stochastic. 00:34:39.870 --> 00:34:40.590 align:middle line:90% They're going to be random. 00:34:40.590 --> 00:34:42.190 align:middle line:84% They're going to vary from the same question. 00:34:42.190 --> 00:34:44.023 align:middle line:84% The answer is going to vary again and again. 00:34:44.023 --> 00:34:45.449 align:middle line:84% But in many cases, maybe it's OK. 00:34:45.449 --> 00:34:47.010 align:middle line:90% You don't care. 00:34:47.010 --> 00:34:50.159 align:middle line:84% So that's how, roughly, we think about it. 00:34:50.159 --> 00:34:52.860 align:middle line:84% The other one I want to say is that the diversity of response 00:34:52.860 --> 00:34:57.260 align:middle line:84% is also important because, if you imagine a chatbot, if you 00:34:57.260 --> 00:34:58.980 align:middle line:84% ask questions, if the chatbot always 00:34:58.980 --> 00:35:02.640 align:middle line:84% responds in the same stilted robotic fashion, 00:35:02.640 --> 00:35:04.300 align:middle line:84% it kind of starts to get annoying. 00:35:04.300 --> 00:35:06.460 align:middle line:84% You want some variation in the output 00:35:06.460 --> 00:35:09.780 align:middle line:84% because a human will never give you the same thing back. 00:35:09.780 --> 00:35:12.100 align:middle line:84% Though I must say that when I interact with call center 00:35:12.100 --> 00:35:14.100 align:middle line:84% agents, I think they're just cutting and pasting 00:35:14.100 --> 00:35:17.500 align:middle line:84% from a text library, so it does look kind of robotic. 00:35:17.500 --> 00:35:19.400 align:middle line:84% So maybe we are already kind of used to this. 00:35:19.400 --> 00:35:21.400 align:middle line:84% But anyway, so those are some of the things to keep in mind. 00:35:21.400 --> 00:35:21.900 align:middle line:90% Yeah? 00:35:21.900 --> 00:35:24.420 align:middle line:90% 00:35:24.420 --> 00:35:26.240 align:middle line:84% AUDIENCE: If you're using random sampling, 00:35:26.240 --> 00:35:28.980 align:middle line:84% do you end up with a better estimation of the uncertainty 00:35:28.980 --> 00:35:33.490 align:middle line:84% and probabilities that are more calibrated, 00:35:33.490 --> 00:35:36.300 align:middle line:84% in the sense that the table that you end up at the end 00:35:36.300 --> 00:35:39.220 align:middle line:84% is the real probability that you observe 00:35:39.220 --> 00:35:41.288 align:middle line:90% from the words in your corpus? 00:35:41.288 --> 00:35:43.580 align:middle line:84% RAMA RAMAKRISHNAN: The table doesn't change, regardless 00:35:43.580 --> 00:35:44.760 align:middle line:90% of how you sample it. 00:35:44.760 --> 00:35:47.680 align:middle line:84% The table is a starting point for sampling. 00:35:47.680 --> 00:35:51.360 align:middle line:84% All decoding is about what token from the table 00:35:51.360 --> 00:35:53.020 align:middle line:90% you're going to pull out. 00:35:53.020 --> 00:35:55.640 align:middle line:84% AUDIENCE: Oh, so it doesn't impact the loss function? 00:35:55.640 --> 00:35:56.800 align:middle line:90% RAMA RAMAKRISHNAN: No. 00:35:56.800 --> 00:35:58.858 align:middle line:84% Yeah, all those things are fixed. 00:35:58.858 --> 00:36:00.400 align:middle line:84% You literally get the table, and then 00:36:00.400 --> 00:36:02.680 align:middle line:84% you literally can forget how you got the table. 00:36:02.680 --> 00:36:06.160 align:middle line:90% And now decoding starts. 00:36:06.160 --> 00:36:09.120 align:middle line:84% AUDIENCE: Is that the reason why ChatGPT would generate 00:36:09.120 --> 00:36:11.900 align:middle line:84% a different answer given the same prompt if we run it again 00:36:11.900 --> 00:36:12.400 align:middle line:90% and again? 00:36:12.400 --> 00:36:14.025 align:middle line:84% Because they are using random sampling? 00:36:14.025 --> 00:36:15.150 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:36:15.150 --> 00:36:16.040 align:middle line:90% That's exactly why. 00:36:16.040 --> 00:36:18.000 align:middle line:84% And we'll see-- I'll do a demo of it 00:36:18.000 --> 00:36:22.800 align:middle line:84% very shortly because you can actually manipulate it. 00:36:22.800 --> 00:36:25.160 align:middle line:84% AUDIENCE: If you do the prediction word-by-word, is 00:36:25.160 --> 00:36:28.600 align:middle line:84% there a way to make it resilient to mistakes? 00:36:28.600 --> 00:36:31.260 align:middle line:84% If you say, the night was dark and aardvark, 00:36:31.260 --> 00:36:33.180 align:middle line:84% that can mess up the next word, right? 00:36:33.180 --> 00:36:35.055 align:middle line:84% RAMA RAMAKRISHNAN: It can totally mess it up. 00:36:35.055 --> 00:36:37.432 align:middle line:84% AUDIENCE: So how does it get itself back on track? 00:36:37.432 --> 00:36:38.640 align:middle line:90% RAMA RAMAKRISHNAN: It cannot. 00:36:38.640 --> 00:36:40.120 align:middle line:90% And so great question. 00:36:40.120 --> 00:36:42.120 align:middle line:84% And we'll look at an example of things going off 00:36:42.120 --> 00:36:44.300 align:middle line:90% the rails in just a second. 00:36:44.300 --> 00:36:46.045 align:middle line:90% Yeah? 00:36:46.045 --> 00:36:47.420 align:middle line:84% AUDIENCE: Is this how Bing works, 00:36:47.420 --> 00:36:51.260 align:middle line:84% where you can slide between being more creative and more 00:36:51.260 --> 00:36:51.960 align:middle line:90% accurate? 00:36:51.960 --> 00:36:53.335 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, exactly. 00:36:53.335 --> 00:36:55.900 align:middle line:84% So Bing has this creative, balanced, precise something. 00:36:55.900 --> 00:36:57.620 align:middle line:84% They're basically under the hood. 00:36:57.620 --> 00:36:59.440 align:middle line:84% They're manipulating some of the-- 00:36:59.440 --> 00:37:01.060 align:middle line:84% we're going to look at some of those parameters in just 00:37:01.060 --> 00:37:01.560 align:middle line:90% a moment. 00:37:01.560 --> 00:37:03.340 align:middle line:84% They're just manipulating it for you. 00:37:03.340 --> 00:37:06.150 align:middle line:84% But if you use the API, you can manipulate it directly. 00:37:06.150 --> 00:37:10.980 align:middle line:90% 00:37:10.980 --> 00:37:15.100 align:middle line:84% All right, so here is the basic thing 00:37:15.100 --> 00:37:17.500 align:middle line:84% to remember about random sampling. 00:37:17.500 --> 00:37:21.140 align:middle line:84% So our hope is that, for any given sentence, 00:37:21.140 --> 00:37:23.100 align:middle line:84% we think that there is probably some set 00:37:23.100 --> 00:37:25.620 align:middle line:84% of good answers for the next word 00:37:25.620 --> 00:37:29.180 align:middle line:84% and a whole bunch of bad answers, intuitively. 00:37:29.180 --> 00:37:34.420 align:middle line:84% So we want the probability of the good stuff. 00:37:34.420 --> 00:37:37.697 align:middle line:84% We want-- you can imagine a distribution as going like that. 00:37:37.697 --> 00:37:39.280 align:middle line:84% There is the head of the distribution, 00:37:39.280 --> 00:37:41.655 align:middle line:84% the first few words in the distribution, if you sort them 00:37:41.655 --> 00:37:43.120 align:middle line:90% from high to low probability. 00:37:43.120 --> 00:37:44.920 align:middle line:84% And then there is all the long tail 00:37:44.920 --> 00:37:49.320 align:middle line:84% of kind of inappropriate-- not inappropriate, irrelevant words. 00:37:49.320 --> 00:37:53.520 align:middle line:84% So our hope is that the model is so good that, for any given 00:37:53.520 --> 00:37:56.760 align:middle line:84% input phrase, it basically concentrates the output 00:37:56.760 --> 00:37:59.360 align:middle line:84% probability in the softmax, so just a few good words, 00:37:59.360 --> 00:38:02.400 align:middle line:84% and sort of, kind of zeroes out everything else. 00:38:02.400 --> 00:38:05.780 align:middle line:84% That is the ideal scenario because, in that scenario, 00:38:05.780 --> 00:38:07.720 align:middle line:84% if you do random sampling, by definition, 00:38:07.720 --> 00:38:09.520 align:middle line:84% you'll pick something from the high quality 00:38:09.520 --> 00:38:12.280 align:middle line:84% head of the distribution, and life is good. 00:38:12.280 --> 00:38:16.560 align:middle line:84% Now, we want random sampling to sample from the head 00:38:16.560 --> 00:38:18.185 align:middle line:90% and not from the tail. 00:38:18.185 --> 00:38:19.060 align:middle line:90% That's the key point. 00:38:19.060 --> 00:38:20.560 align:middle line:84% And what do I mean by head and tail? 00:38:20.560 --> 00:38:21.500 align:middle line:90% Let's be very clear. 00:38:21.500 --> 00:38:26.200 align:middle line:90% 00:38:26.200 --> 00:38:30.265 align:middle line:90% So imagine you have-- 00:38:30.265 --> 00:38:31.640 align:middle line:84% take the table that we looked at, 00:38:31.640 --> 00:38:33.098 align:middle line:84% the softmax table, which went from, 00:38:33.098 --> 00:38:34.978 align:middle line:90% what, about aardvark to zebra. 00:38:34.978 --> 00:38:36.520 align:middle line:84% And let's say we sort the table based 00:38:36.520 --> 00:38:38.223 align:middle line:90% on high to low probabilities. 00:38:38.223 --> 00:38:39.640 align:middle line:84% So maybe what's going to happen is 00:38:39.640 --> 00:38:43.940 align:middle line:84% that stormy is going to have a probability of, I don't know, 00:38:43.940 --> 00:38:45.940 align:middle line:90% 0.6. 00:38:45.940 --> 00:38:47.820 align:middle line:84% And I think, if I remember right, 00:38:47.820 --> 00:38:52.020 align:middle line:84% a night had a probability of 0.3. 00:38:52.020 --> 00:38:56.620 align:middle line:84% And then there were a whole bunch of other words, 00:38:56.620 --> 00:39:00.100 align:middle line:84% all the way to the 50,000th word. 00:39:00.100 --> 00:39:03.820 align:middle line:84% From highest low probability, so this is what I'm-- so you can 00:39:03.820 --> 00:39:07.540 align:middle line:84% think of this as like a probability distribution 00:39:07.540 --> 00:39:09.700 align:middle line:84% And so, basically, what we are saying here 00:39:09.700 --> 00:39:13.900 align:middle line:84% is that this is the head of the distribution, 00:39:13.900 --> 00:39:17.060 align:middle line:84% while this long tail is the tail of the distribution. 00:39:17.060 --> 00:39:20.580 align:middle line:84% And we want our system to grab something from the head 00:39:20.580 --> 00:39:22.900 align:middle line:84% and not from the tail because the head is 00:39:22.900 --> 00:39:26.500 align:middle line:84% the stuff that's actually the relevant, useful, good stuff. 00:39:26.500 --> 00:39:28.340 align:middle line:84% That's really what we're trying to do here. 00:39:28.340 --> 00:39:30.140 align:middle line:90% Does it make sense? 00:39:30.140 --> 00:39:33.630 align:middle line:90% So to come back to this-- 00:39:33.630 --> 00:39:37.300 align:middle line:90% 00:39:37.300 --> 00:39:39.420 align:middle line:84% and here is the most important point 00:39:39.420 --> 00:39:41.480 align:middle line:90% to remember about this slide. 00:39:41.480 --> 00:39:44.520 align:middle line:84% While the probability of choosing any individual word 00:39:44.520 --> 00:39:47.300 align:middle line:84% in this long tail is pretty small-- for any one word, 00:39:47.300 --> 00:39:48.480 align:middle line:90% it's pretty small-- 00:39:48.480 --> 00:39:54.000 align:middle line:84% the probability of choosing some word from the tail is high. 00:39:54.000 --> 00:39:55.620 align:middle line:90% Some word from the tail is high. 00:39:55.620 --> 00:39:58.000 align:middle line:84% So to go back to this thing here, 00:39:58.000 --> 00:40:01.840 align:middle line:84% yeah, so in this particular example, 0.6 plus 0.3, 00:40:01.840 --> 00:40:03.560 align:middle line:84% there is a 0.9 probability it's going 00:40:03.560 --> 00:40:05.060 align:middle line:90% to be either stormy or night. 00:40:05.060 --> 00:40:06.640 align:middle line:84% But there is a 10% probability it's 00:40:06.640 --> 00:40:09.360 align:middle line:90% going to be one of these words. 00:40:09.360 --> 00:40:10.860 align:middle line:84% And who knows what that word might-- 00:40:10.860 --> 00:40:11.568 align:middle line:90% it's going to be. 00:40:11.568 --> 00:40:14.300 align:middle line:84% It might be some random nonsense word. 00:40:14.300 --> 00:40:21.400 align:middle line:84% So what that means is then this goes to a point from before. 00:40:21.400 --> 00:40:24.520 align:middle line:84% If the LLM happens to sample a token from the tail, which 00:40:24.520 --> 00:40:27.140 align:middle line:84% is not good, it won't be able to recover from its mistake. 00:40:27.140 --> 00:40:30.160 align:middle line:84% It'll just go off the rails, which 00:40:30.160 --> 00:40:32.080 align:middle line:84% is why every word that gets generated, 00:40:32.080 --> 00:40:33.720 align:middle line:84% it's really important to get it right 00:40:33.720 --> 00:40:37.890 align:middle line:84% because it can't recover very often. 00:40:37.890 --> 00:40:39.410 align:middle line:84% AUDIENCE: Is there a technical way 00:40:39.410 --> 00:40:41.803 align:middle line:84% to define the difference between the head and the tail? 00:40:41.803 --> 00:40:42.720 align:middle line:90% RAMA RAMAKRISHNAN: No. 00:40:42.720 --> 00:40:44.010 align:middle line:90% [CHUCKLING] 00:40:44.010 --> 00:40:46.210 align:middle line:84% It's like this common thing people use. 00:40:46.210 --> 00:40:49.210 align:middle line:84% And the reason why it's not is because it's so problem 00:40:49.210 --> 00:40:52.266 align:middle line:90% dependent as to what the-- 00:40:52.266 --> 00:40:54.790 align:middle line:84% basically, you're saying that for any particular problem, 00:40:54.790 --> 00:40:56.890 align:middle line:84% I think, depending on the question, 00:40:56.890 --> 00:40:59.010 align:middle line:84% the right number of words is probably 20. 00:40:59.010 --> 00:41:01.890 align:middle line:84% For the same-- for a different question, maybe it's 40. 00:41:01.890 --> 00:41:05.250 align:middle line:84% For a totally different model, for the same question, maybe 10. 00:41:05.250 --> 00:41:09.090 align:middle line:84% So because of that variability, we just can't figure it out. 00:41:09.090 --> 00:41:13.770 align:middle line:84% So all right, and I'll show you this, how to do this, 00:41:13.770 --> 00:41:14.810 align:middle line:90% in just a moment. 00:41:14.810 --> 00:41:20.450 align:middle line:84% So just for kicks, I went in to GPT 3.5. 00:41:20.450 --> 00:41:24.410 align:middle line:84% And then I said, students at the MIT Sloan School of Management 00:41:24.410 --> 00:41:25.990 align:middle line:90% are? 00:41:25.990 --> 00:41:27.850 align:middle line:84% And I said, predict the next word. 00:41:27.850 --> 00:41:32.490 align:middle line:84% So it turns out, invited is the most likely next word, 00:41:32.490 --> 00:41:35.530 align:middle line:84% followed by given, expected, required, and able. 00:41:35.530 --> 00:41:39.070 align:middle line:90% These are the top five words. 00:41:39.070 --> 00:41:42.610 align:middle line:84% And the probability is 3%, 2%, pretty small probabilities. 00:41:42.610 --> 00:41:45.350 align:middle line:84% But then the words that are below it, 00:41:45.350 --> 00:41:48.550 align:middle line:84% the remaining whatever, 50,000 odd words, are even lower. 00:41:48.550 --> 00:41:51.750 align:middle line:84% So here, the most likely word is invited. 00:41:51.750 --> 00:41:53.570 align:middle line:84% So what I did is I went in there and said, 00:41:53.570 --> 00:41:56.590 align:middle line:84% OK, let me try again now with, students at the MIT Sloan School 00:41:56.590 --> 00:41:58.790 align:middle line:90% of Management are invited. 00:41:58.790 --> 00:42:00.030 align:middle line:90% And now autocomplete that. 00:42:00.030 --> 00:42:01.390 align:middle line:90% Find me the next thing. 00:42:01.390 --> 00:42:02.632 align:middle line:90% So it comes back with-- 00:42:02.632 --> 00:42:04.340 align:middle line:84% see, now, this is my new prompt, students 00:42:04.340 --> 00:42:05.965 align:middle line:84% at the MIT School of Management invited 00:42:05.965 --> 00:42:08.550 align:middle line:84% to submit their original white papers to the annual MIT 00:42:08.550 --> 00:42:09.350 align:middle line:90% something. 00:42:09.350 --> 00:42:10.790 align:middle line:90% It seems reasonable. 00:42:10.790 --> 00:42:12.690 align:middle line:90% It doesn't seem bad. 00:42:12.690 --> 00:42:14.330 align:middle line:90% It seems reasonable. 00:42:14.330 --> 00:42:16.430 align:middle line:90% Now let's mess it up a bit. 00:42:16.430 --> 00:42:18.790 align:middle line:84% So now I go in there, and I noticed 00:42:18.790 --> 00:42:22.230 align:middle line:84% that the word masters and the word spending 00:42:22.230 --> 00:42:25.690 align:middle line:84% were much lower probability than these top five words. 00:42:25.690 --> 00:42:27.990 align:middle line:84% I just mucked around until I found these things. 00:42:27.990 --> 00:42:29.930 align:middle line:90% So this is only 0.05%. 00:42:29.930 --> 00:42:31.550 align:middle line:90% This is 0.11%. 00:42:31.550 --> 00:42:34.750 align:middle line:84% So these are clearly in the tail. 00:42:34.750 --> 00:42:35.932 align:middle line:90% They're not the most likely. 00:42:35.932 --> 00:42:37.890 align:middle line:84% So I said, what's going to happen if I actually 00:42:37.890 --> 00:42:40.210 align:middle line:90% force it to use masters? 00:42:40.210 --> 00:42:42.850 align:middle line:84% And then I force it to use spending. 00:42:42.850 --> 00:42:45.130 align:middle line:90% This is what you get. 00:42:45.130 --> 00:42:48.180 align:middle line:84% Students at the MIT School of Management are masters of chaos. 00:42:48.180 --> 00:42:49.370 align:middle line:90% [CHUCKLING] 00:42:49.370 --> 00:42:52.690 align:middle line:84% They routinely blow past deadlines, fracture. 00:42:52.690 --> 00:42:54.190 align:middle line:84% And then I couldn't take it anymore. 00:42:54.190 --> 00:42:54.870 align:middle line:90% I stopped it. 00:42:54.870 --> 00:42:58.010 align:middle line:90% [CHUCKLING] 00:42:58.010 --> 00:43:00.402 align:middle line:90% A single word. 00:43:00.402 --> 00:43:01.860 align:middle line:84% And then I said, students, students 00:43:01.860 --> 00:43:03.610 align:middle line:84% at MIT School of Management, are spending, 00:43:03.610 --> 00:43:06.130 align:middle line:84% which is the other unlikely word, the semester learning 00:43:06.130 --> 00:43:08.250 align:middle line:84% life skills-- so far, it looks promising-- 00:43:08.250 --> 00:43:09.220 align:middle line:90% through knitting socks. 00:43:09.220 --> 00:43:12.382 align:middle line:90% [LAUGHTER] 00:43:12.382 --> 00:43:13.262 align:middle line:90% 00:43:13.262 --> 00:43:14.470 align:middle line:90% I'm not making this stuff up. 00:43:14.470 --> 00:43:17.490 align:middle line:90% This is GPT 3.5. 00:43:17.490 --> 00:43:19.030 align:middle line:84% So, yes, it will go off the rails. 00:43:19.030 --> 00:43:21.010 align:middle line:90% You have to be super careful. 00:43:21.010 --> 00:43:25.730 align:middle line:84% And so the way we sort of tame random sampling 00:43:25.730 --> 00:43:27.170 align:middle line:90% to make it work for us-- 00:43:27.170 --> 00:43:29.290 align:middle line:90% yeah? 00:43:29.290 --> 00:43:32.990 align:middle line:84% AUDIENCE: Do you think that these sentences refers-- 00:43:32.990 --> 00:43:36.760 align:middle line:84% the past-- the master of chaos routinely blow past deadlines, 00:43:36.760 --> 00:43:40.350 align:middle line:84% you think is something that it was in the training set? 00:43:40.350 --> 00:43:42.350 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, that is-- the thing is, 00:43:42.350 --> 00:43:45.870 align:middle line:84% it's basically doing some very rough 00:43:45.870 --> 00:43:48.670 align:middle line:84% and approximate pattern matching from all the training data 00:43:48.670 --> 00:43:49.790 align:middle line:90% it was trained on. 00:43:49.790 --> 00:43:54.950 align:middle line:84% So it doesn't mean, for example, that on the mit.edu website, 00:43:54.950 --> 00:43:57.670 align:middle line:84% on the collection of sites, that, actually, there 00:43:57.670 --> 00:44:00.590 align:middle line:84% were text saying that, yeah, MIT Sloan students were 00:44:00.590 --> 00:44:01.950 align:middle line:90% doing all this crazy stuff. 00:44:01.950 --> 00:44:06.230 align:middle line:84% It's probably more like a whole bunch of college, 00:44:06.230 --> 00:44:08.620 align:middle line:84% university websites probably had some content like that. 00:44:08.620 --> 00:44:10.870 align:middle line:84% Maybe there was a bunch of Reddit people posting stuff 00:44:10.870 --> 00:44:11.590 align:middle line:90% like that. 00:44:11.590 --> 00:44:13.567 align:middle line:84% So it's just doing some rough pattern matching. 00:44:13.567 --> 00:44:15.150 align:middle line:84% It's basically looking-- the thing is, 00:44:15.150 --> 00:44:17.733 align:middle line:84% you have to remember, always, with large language models, what 00:44:17.733 --> 00:44:21.150 align:middle line:84% it's trying to give you, it's giving you a response 00:44:21.150 --> 00:44:23.630 align:middle line:90% that is not implausible. 00:44:23.630 --> 00:44:25.330 align:middle line:84% There is no guarantee of correctness. 00:44:25.330 --> 00:44:27.150 align:middle line:84% There is no accuracy, nothing like that. 00:44:27.150 --> 00:44:30.090 align:middle line:84% It's giving you a probabilistically plausible 00:44:30.090 --> 00:44:30.670 align:middle line:90% response. 00:44:30.670 --> 00:44:31.990 align:middle line:90% That's it. 00:44:31.990 --> 00:44:35.490 align:middle line:84% Now, us Sloanies being Sloanies, we look at stuff like this 00:44:35.490 --> 00:44:36.690 align:middle line:90% and we get offended. 00:44:36.690 --> 00:44:40.030 align:middle line:84% So we are imputing our values onto each generation, 00:44:40.030 --> 00:44:43.850 align:middle line:84% but it doesn't know and it doesn't care. 00:44:43.850 --> 00:44:46.570 align:middle line:84% So, in fact, when I typed in something like, 00:44:46.570 --> 00:44:50.450 align:middle line:84% list all the awards that Professor Ramakrishnan has won, 00:44:50.450 --> 00:44:52.490 align:middle line:84% it gave me an amazing list of awards. 00:44:52.490 --> 00:44:54.110 align:middle line:84% Apparently, I won this and I won that. 00:44:54.110 --> 00:44:57.530 align:middle line:84% I won-- now, none of it is true, to which 00:44:57.530 --> 00:44:59.110 align:middle line:90% a student said, not yet. 00:44:59.110 --> 00:45:00.250 align:middle line:90% [LAUGHTER] 00:45:00.250 --> 00:45:02.710 align:middle line:84% So I had the TA make a note of that fine person's name. 00:45:02.710 --> 00:45:05.290 align:middle line:90% [LAUGHTER] 00:45:05.290 --> 00:45:08.970 align:middle line:90% Yeah, so that's what's going on. 00:45:08.970 --> 00:45:11.060 align:middle line:90% Yeah? 00:45:11.060 --> 00:45:12.810 align:middle line:84% AUDIENCE: I get the sense, maybe there's-- 00:45:12.810 --> 00:45:15.497 align:middle line:84% RAMA RAMAKRISHNAN: Could you use the microphone, please? 00:45:15.497 --> 00:45:17.330 align:middle line:84% AUDIENCE: I get the sense that maybe there's 00:45:17.330 --> 00:45:21.010 align:middle line:84% some sort of sliding window, that somehow weighting 00:45:21.010 --> 00:45:24.630 align:middle line:84% later words more strongly than earlier words, 00:45:24.630 --> 00:45:27.990 align:middle line:84% given how far out-- because they do have the context of students 00:45:27.990 --> 00:45:31.050 align:middle line:84% at MIT should have steered it in a certain direction, 00:45:31.050 --> 00:45:33.670 align:middle line:84% even with the presence of the word masters. 00:45:33.670 --> 00:45:35.610 align:middle line:84% So is there something like that happening? 00:45:35.610 --> 00:45:36.590 align:middle line:84% RAMA RAMAKRISHNAN: No, it is just-- 00:45:36.590 --> 00:45:38.548 align:middle line:84% the thing is, think about the training process. 00:45:38.548 --> 00:45:42.190 align:middle line:84% And the training process, we gave it sentence fragments. 00:45:42.190 --> 00:45:43.950 align:middle line:84% And we asked her to predict the next word. 00:45:43.950 --> 00:45:48.430 align:middle line:84% Now, clearly, the more you know about the input that's coming 00:45:48.430 --> 00:45:50.550 align:middle line:84% and the longer the input, the more clues you 00:45:50.550 --> 00:45:52.510 align:middle line:84% have to figure out what the right next word 00:45:52.510 --> 00:45:54.050 align:middle line:90% prediction is going to be. 00:45:54.050 --> 00:45:58.190 align:middle line:84% If I say, the capital of, you'll be like, I don't know. 00:45:58.190 --> 00:46:00.262 align:middle line:84% It's got to be a country, I guess, or a state, 00:46:00.262 --> 00:46:01.970 align:middle line:84% but I don't know anything more than that. 00:46:01.970 --> 00:46:04.130 align:middle line:84% But if I say, the capital of France 00:46:04.130 --> 00:46:08.230 align:middle line:84% is, dramatic narrowing of the cone of uncertainty. 00:46:08.230 --> 00:46:10.230 align:middle line:84% So that's basically what's going on. 00:46:10.230 --> 00:46:12.810 align:middle line:84% And, in fact, there's a very beautiful expression I've heard, 00:46:12.810 --> 00:46:15.830 align:middle line:84% which is that what elements do, they 00:46:15.830 --> 00:46:19.150 align:middle line:90% call it subtractive sculpting. 00:46:19.150 --> 00:46:21.850 align:middle line:84% So what I mean by that is it's sort of like, when you start, 00:46:21.850 --> 00:46:24.030 align:middle line:84% it's like this big block of marble. 00:46:24.030 --> 00:46:26.102 align:middle line:84% And then every word chips away at the marble. 00:46:26.102 --> 00:46:28.310 align:middle line:84% And then when you're done, it's kind of pretty clear. 00:46:28.310 --> 00:46:30.190 align:middle line:90% It's David inside the marble. 00:46:30.190 --> 00:46:34.170 align:middle line:90% That's sort of what's going on. 00:46:34.170 --> 00:46:37.387 align:middle line:84% All right, so to come back to this, what can we do? 00:46:37.387 --> 00:46:38.970 align:middle line:84% We can-- there are three ways in which 00:46:38.970 --> 00:46:41.770 align:middle line:84% you can tune random sampling to make it work for you. 00:46:41.770 --> 00:46:44.810 align:middle line:84% The first way and the idea of all these things 00:46:44.810 --> 00:46:48.050 align:middle line:84% is that you have some probability distribution. 00:46:48.050 --> 00:46:52.655 align:middle line:84% We are now going to sort of manually focus on the head. 00:46:52.655 --> 00:46:54.530 align:middle line:84% And then we're going to kill everything else, 00:46:54.530 --> 00:46:57.410 align:middle line:84% and only focus on the head and sample from that head, which 00:46:57.410 --> 00:46:59.290 align:middle line:84% immediately begs the question, how will you 00:46:59.290 --> 00:47:00.350 align:middle line:90% decide what the head is? 00:47:00.350 --> 00:47:02.508 align:middle line:84% And that was sort of Elena's question from before. 00:47:02.508 --> 00:47:04.050 align:middle line:84% How will you decide what the head is? 00:47:04.050 --> 00:47:07.233 align:middle line:84% So one way we do that is to say, you know what? 00:47:07.233 --> 00:47:09.150 align:middle line:84% I know we have 50,000 words in the vocabulary. 00:47:09.150 --> 00:47:10.530 align:middle line:90% I don't care. 00:47:10.530 --> 00:47:13.130 align:middle line:84% Each time, I'm only going to pick the top K words. 00:47:13.130 --> 00:47:15.460 align:middle line:90% K could be 10, 20, 30, 40, 50. 00:47:15.460 --> 00:47:16.710 align:middle line:90% That's very problem dependent. 00:47:16.710 --> 00:47:18.430 align:middle line:84% I'm going to pick the top 20 words, 00:47:18.430 --> 00:47:20.970 align:middle line:84% and I'm going to ignore everything else and only sample 00:47:20.970 --> 00:47:22.750 align:middle line:90% from the top 10 or the top 20. 00:47:22.750 --> 00:47:23.958 align:middle line:90% That's called top-k sampling. 00:47:23.958 --> 00:47:25.875 align:middle line:84% And so the way it works is that let's say this 00:47:25.875 --> 00:47:27.000 align:middle line:90% is your whole distribution. 00:47:27.000 --> 00:47:29.250 align:middle line:84% And I just stopped at red instead of going all the way 00:47:29.250 --> 00:47:29.800 align:middle line:90% to 50,000. 00:47:29.800 --> 00:47:32.140 align:middle line:84% And then you see and you decide, let's say 00:47:32.140 --> 00:47:34.820 align:middle line:90% that you want k to be 2. 00:47:34.820 --> 00:47:37.820 align:middle line:84% So you just grab the top two words, k equals 2. 00:47:37.820 --> 00:47:41.620 align:middle line:84% And then you renormalize the probability so they add up to 1. 00:47:41.620 --> 00:47:46.780 align:middle line:84% So 0.6 and 0.2, renormalize, it becomes 0.75 and 0.25. 00:47:46.780 --> 00:47:49.340 align:middle line:84% And now, just imagine that this is the new softmax table 00:47:49.340 --> 00:47:50.740 align:middle line:90% that you're sampling from. 00:47:50.740 --> 00:47:52.020 align:middle line:90% And you grab a number from-- 00:47:52.020 --> 00:47:55.940 align:middle line:84% I'm sorry, a word from here and you're done. 00:47:55.940 --> 00:48:00.580 align:middle line:84% That's called top-k sampling, very commonly used. 00:48:00.580 --> 00:48:04.740 align:middle line:84% But it has a small shortcoming, which is that it basically 00:48:04.740 --> 00:48:08.980 align:middle line:84% assumes that this k that you've come up with, let's say 20, 00:48:08.980 --> 00:48:12.020 align:middle line:84% every input sentence, the right number of words in the head 00:48:12.020 --> 00:48:13.980 align:middle line:84% is 20, which seems-- obviously, it's not 00:48:13.980 --> 00:48:16.200 align:middle line:90% a well-supported assumption. 00:48:16.200 --> 00:48:17.620 align:middle line:90% It's just an assumption. 00:48:17.620 --> 00:48:19.820 align:middle line:84% So then the question becomes, can we do better? 00:48:19.820 --> 00:48:22.440 align:middle line:84% Because what you really want is you 00:48:22.440 --> 00:48:24.880 align:middle line:84% want the words that you pick to have 00:48:24.880 --> 00:48:28.983 align:middle line:84% the bulk of the probabilities, as much probability as possible. 00:48:28.983 --> 00:48:31.400 align:middle line:84% You don't really care how many words are inserted, as long 00:48:31.400 --> 00:48:34.520 align:middle line:84% as, together, they have a lot of probability, which brings us 00:48:34.520 --> 00:48:36.840 align:middle line:84% to something called top-p sampling, 00:48:36.840 --> 00:48:38.570 align:middle line:84% also called nucleus sampling, where 00:48:38.570 --> 00:48:40.320 align:middle line:84% instead of deciding on the number of words 00:48:40.320 --> 00:48:42.980 align:middle line:84% we're going to pick every time, we decide, you know what? 00:48:42.980 --> 00:48:46.920 align:middle line:84% We're just going to choose all the words, such 00:48:46.920 --> 00:48:49.400 align:middle line:84% that the probability of such words that we have chosen 00:48:49.400 --> 00:48:51.480 align:middle line:90% is at least p. 00:48:51.480 --> 00:48:52.980 align:middle line:84% Sometimes, it may be just two words. 00:48:52.980 --> 00:48:54.260 align:middle line:90% Sometimes, it may be 20 words. 00:48:54.260 --> 00:48:55.560 align:middle line:90% We don't care. 00:48:55.560 --> 00:48:58.760 align:middle line:90% And then we sample from it. 00:48:58.760 --> 00:49:01.860 align:middle line:90% So here, same thing here. 00:49:01.860 --> 00:49:03.880 align:middle line:84% Let's say you go with p equals 0.9. 00:49:03.880 --> 00:49:07.040 align:middle line:90% So 0.6 plus 0.2, 0.8 plus 0.1. 00:49:07.040 --> 00:49:07.540 align:middle line:90% 0.9. 00:49:07.540 --> 00:49:08.820 align:middle line:90% Boom, we have hit 0.9. 00:49:08.820 --> 00:49:09.920 align:middle line:90% We stop. 00:49:09.920 --> 00:49:11.400 align:middle line:84% And then we grab these three words, 00:49:11.400 --> 00:49:14.660 align:middle line:84% and then we renormalize them to get this thing. 00:49:14.660 --> 00:49:16.408 align:middle line:84% And then, boom, we sample from it. 00:49:16.408 --> 00:49:18.700 align:middle line:84% So this, actually, is even more effective in my opinion 00:49:18.700 --> 00:49:20.560 align:middle line:90% because it sort of fluctuates. 00:49:20.560 --> 00:49:23.900 align:middle line:84% It doesn't hardcode the number of words you think is important. 00:49:23.900 --> 00:49:24.980 align:middle line:90% Was there a question? 00:49:24.980 --> 00:49:25.900 align:middle line:90% Yeah. 00:49:25.900 --> 00:49:29.460 align:middle line:84% AUDIENCE: What if, let's say, 0.9 ended up-- 00:49:29.460 --> 00:49:33.230 align:middle line:84% if foggy was 0.12, would it only be 0.1 from foggy? 00:49:33.230 --> 00:49:34.980 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, what it does is-- 00:49:34.980 --> 00:49:37.180 align:middle line:84% so you give it a point, and what it's going to do 00:49:37.180 --> 00:49:39.340 align:middle line:84% is it's going to keep adding words 00:49:39.340 --> 00:49:43.580 align:middle line:84% till it just crosses that number. 00:49:43.580 --> 00:49:44.673 align:middle line:90% Yeah? 00:49:44.673 --> 00:49:46.340 align:middle line:84% AUDIENCE: I was thinking, can't you just 00:49:46.340 --> 00:49:50.020 align:middle line:84% set a threshold for the words that 00:49:50.020 --> 00:49:53.220 align:middle line:84% don't pick a word below a certain probability? 00:49:53.220 --> 00:49:58.740 align:middle line:84% This stormy, what if stormy was 0.89 and then the other one is 00:49:58.740 --> 00:50:00.660 align:middle line:84% just 0.10 [INAUDIBLE] so you pick two words? 00:50:00.660 --> 00:50:02.368 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, you can do that. 00:50:02.368 --> 00:50:04.760 align:middle line:84% And, in fact, what you can do is you can always say, 00:50:04.760 --> 00:50:07.400 align:middle line:84% I want to pick a word, which is the most likely word. 00:50:07.400 --> 00:50:08.320 align:middle line:90% You can do that. 00:50:08.320 --> 00:50:11.620 align:middle line:90% But if you say, I want a word-- 00:50:11.620 --> 00:50:13.460 align:middle line:84% only consider words whose probabilities 00:50:13.460 --> 00:50:15.940 align:middle line:84% are at least something, then, basically, what you're saying 00:50:15.940 --> 00:50:17.607 align:middle line:84% is that I'm just going to keep on doing. 00:50:17.607 --> 00:50:19.280 align:middle line:90% And then we draw a line here. 00:50:19.280 --> 00:50:22.200 align:middle line:84% But the problem is, you don't know how many words 00:50:22.200 --> 00:50:26.080 align:middle line:90% have crept over your threshold. 00:50:26.080 --> 00:50:28.540 align:middle line:84% You might, for example, find that-- to go to your example, 00:50:28.540 --> 00:50:30.580 align:middle line:84% maybe-- you said 0.9 is the threshold. 00:50:30.580 --> 00:50:33.480 align:middle line:84% Maybe there are a whole bunch of-- there was a word at 0.89 00:50:33.480 --> 00:50:35.340 align:middle line:84% that you just missed because it didn't make the threshold. 00:50:35.340 --> 00:50:37.382 align:middle line:84% You'll be like oh no, I should have made it 0.89. 00:50:37.382 --> 00:50:39.560 align:middle line:84% So there's no right answer, unfortunately. 00:50:39.560 --> 00:50:40.760 align:middle line:90% But these are exactly-- 00:50:40.760 --> 00:50:42.520 align:middle line:84% this is exactly the kind of thinking 00:50:42.520 --> 00:50:46.480 align:middle line:84% that brought us these kinds of ways to tune these things. 00:50:46.480 --> 00:50:49.040 align:middle line:84% Sort of the foundation here is that-- 00:50:49.040 --> 00:50:53.000 align:middle line:84% the realization that we cannot sort of a priori decide what 00:50:53.000 --> 00:50:54.300 align:middle line:90% the right number of words is. 00:50:54.300 --> 00:50:57.145 align:middle line:84% So we have to find heuristics to try to do these things. 00:50:57.145 --> 00:50:59.020 align:middle line:84% So in practice, people try all these methods. 00:50:59.020 --> 00:51:00.260 align:middle line:90% In fact, you can do both. 00:51:00.260 --> 00:51:02.880 align:middle line:84% You can do-- you can set up so that you can do top-p 00:51:02.880 --> 00:51:04.480 align:middle line:90% and top-k at the same time. 00:51:04.480 --> 00:51:07.120 align:middle line:84% Basically, you're saying, grab words 00:51:07.120 --> 00:51:10.800 align:middle line:84% until you cross the probability or you 00:51:10.800 --> 00:51:12.070 align:middle line:90% cross k, whichever is earlier. 00:51:12.070 --> 00:51:15.433 align:middle line:90% 00:51:15.433 --> 00:51:19.140 align:middle line:84% So those are two methods people use heavily. 00:51:19.140 --> 00:51:21.640 align:middle line:84% The third method is called distribution. 00:51:21.640 --> 00:51:22.860 align:middle line:90% I'm sorry, temperature. 00:51:22.860 --> 00:51:26.580 align:middle line:84% And the idea of temperature is that, in top-k and top-p, 00:51:26.580 --> 00:51:30.540 align:middle line:84% it's sort of-- we have to decide on a number up front, k or p. 00:51:30.540 --> 00:51:32.820 align:middle line:84% And then we just draw the line and look at the words 00:51:32.820 --> 00:51:34.460 align:middle line:90% that pass the threshold. 00:51:34.460 --> 00:51:37.000 align:middle line:84% Temperature is like a softer way to do the same thing. 00:51:37.000 --> 00:51:40.740 align:middle line:84% It's a softer way to emphasize the head more than the tail. 00:51:40.740 --> 00:51:44.720 align:middle line:90% So I think iPad, all right. 00:51:44.720 --> 00:51:53.060 align:middle line:90% 00:51:53.060 --> 00:51:57.520 align:middle line:84% So the idea of temperature is, remember, when we have this, 00:51:57.520 --> 00:52:06.620 align:middle line:84% whoops, softmax, so aardvark all the way to zebra, 00:52:06.620 --> 00:52:08.900 align:middle line:84% you have all these probabilities. 00:52:08.900 --> 00:52:11.360 align:middle line:84% Now, remember, where did we get these probabilities? 00:52:11.360 --> 00:52:13.340 align:middle line:84% These probabilities came from a softmax. 00:52:13.340 --> 00:52:15.120 align:middle line:90% So what is a softmax? 00:52:15.120 --> 00:52:18.840 align:middle line:84% We basically had all these nodes, 00:52:18.840 --> 00:52:22.133 align:middle line:84% say 50,000 nodes in some output layer. 00:52:22.133 --> 00:52:23.300 align:middle line:90% And these were just numbers. 00:52:23.300 --> 00:52:27.480 align:middle line:84% Let's just call them a1 through a 50,000. 00:52:27.480 --> 00:52:30.020 align:middle line:84% And then we ran it through a softmax function. 00:52:30.020 --> 00:52:31.040 align:middle line:90% And what did it do? 00:52:31.040 --> 00:52:36.320 align:middle line:84% It basically did e raised to a1, e raised to a2, all the way to e 00:52:36.320 --> 00:52:37.760 align:middle line:90% raised to a, let's call it n. 00:52:37.760 --> 00:52:40.920 align:middle line:84% And then it divided it by the sum of all these things 00:52:40.920 --> 00:52:42.060 align:middle line:90% to get the probabilities. 00:52:42.060 --> 00:52:47.840 align:middle line:84% So this number became e raised to a1 divided by the sum of all 00:52:47.840 --> 00:52:49.100 align:middle line:90% the e raised to a's. 00:52:49.100 --> 00:52:52.197 align:middle line:90% 00:52:52.197 --> 00:52:54.780 align:middle line:84% So e raised to n divided by e raised to a1 plus e raised to a2 00:52:54.780 --> 00:52:55.760 align:middle line:90% and so on and so forth. 00:52:55.760 --> 00:52:57.220 align:middle line:90% So this is how softmax works. 00:52:57.220 --> 00:53:00.600 align:middle line:84% I'm just refreshing your memory from a few weeks ago. 00:53:00.600 --> 00:53:04.360 align:middle line:84% Now, what temperature does is that-- 00:53:04.360 --> 00:53:08.400 align:middle line:84% let me just write it a little easier. 00:53:08.400 --> 00:53:15.890 align:middle line:84% So e raised to a1 plus e raised to a2 plus all the way, n. 00:53:15.890 --> 00:53:18.490 align:middle line:90% 00:53:18.490 --> 00:53:20.810 align:middle line:84% What it does is it introduces a new parameter here 00:53:20.810 --> 00:53:24.170 align:middle line:84% called temperature, which is that we divide everything 00:53:24.170 --> 00:53:43.690 align:middle line:84% here by T. And the effect of adding this little knob 00:53:43.690 --> 00:53:46.530 align:middle line:84% called temperature here is very interesting. 00:53:46.530 --> 00:53:49.530 align:middle line:84% So let's assume for a second that T 00:53:49.530 --> 00:53:52.330 align:middle line:90% is a very, very small number. 00:53:52.330 --> 00:53:54.970 align:middle line:84% Assume that T is pretty close to 0, very small number. 00:53:54.970 --> 00:54:01.930 align:middle line:84% So if T is close to 0, what's going to happen 00:54:01.930 --> 00:54:04.710 align:middle line:84% is that, since it's in the denominator here, 00:54:04.710 --> 00:54:07.490 align:middle line:84% all these numbers, all these numbers 00:54:07.490 --> 00:54:11.590 align:middle line:84% are going to become really big because T is really small. 00:54:11.590 --> 00:54:13.410 align:middle line:84% If a1 happens to be a positive number, 00:54:13.410 --> 00:54:14.330 align:middle line:90% it's going to become really big. 00:54:14.330 --> 00:54:15.913 align:middle line:84% If a1 is a negative number, it's going 00:54:15.913 --> 00:54:18.000 align:middle line:84% to be a really, really small negative number. 00:54:18.000 --> 00:54:19.750 align:middle line:84% Now, in particular, what's going to happen 00:54:19.750 --> 00:54:23.610 align:middle line:84% is the biggest of all the a numbers, it was already big. 00:54:23.610 --> 00:54:26.670 align:middle line:84% Now it's going to get massive, which 00:54:26.670 --> 00:54:29.110 align:middle line:84% means that its probability is going to dominate everything 00:54:29.110 --> 00:54:32.510 align:middle line:84% else because you're taking a really big number and doing 00:54:32.510 --> 00:54:35.070 align:middle line:90% e raised to that number. 00:54:35.070 --> 00:54:37.390 align:middle line:84% So what's going to happen is that-- 00:54:37.390 --> 00:54:38.030 align:middle line:90% wait, what? 00:54:38.030 --> 00:54:38.650 align:middle line:90% What is this? 00:54:38.650 --> 00:54:41.610 align:middle line:90% 00:54:41.610 --> 00:54:48.570 align:middle line:84% So if T is close to 0, the biggest a-- 00:54:48.570 --> 00:54:56.298 align:middle line:90% 00:54:56.298 --> 00:54:59.230 align:middle line:90% hold on one second. 00:54:59.230 --> 00:55:06.670 align:middle line:84% The word corresponding to the biggest a 00:55:06.670 --> 00:55:10.190 align:middle line:84% will have a probability of 1 or close to 1. 00:55:10.190 --> 00:55:12.810 align:middle line:90% 00:55:12.810 --> 00:55:15.150 align:middle line:84% And since all the probabilities have to add up to 0, 00:55:15.150 --> 00:55:17.433 align:middle line:84% which means that everything else is going to be 0. 00:55:17.433 --> 00:55:19.350 align:middle line:84% So the biggest a will have a probability of 1. 00:55:19.350 --> 00:55:20.808 align:middle line:84% Everything else is going to have 0. 00:55:20.808 --> 00:55:22.690 align:middle line:84% So reducing temperature close to 0 00:55:22.690 --> 00:55:24.530 align:middle line:84% means that the probability distribution 00:55:24.530 --> 00:55:26.470 align:middle line:84% is going to peak at the biggest word, 00:55:26.470 --> 00:55:28.130 align:middle line:84% and everything is going to become 0. 00:55:28.130 --> 00:55:29.610 align:middle line:84% So in practice, what that means is 00:55:29.610 --> 00:55:31.890 align:middle line:84% that if you look at something like this, 00:55:31.890 --> 00:55:38.450 align:middle line:84% if you apply temperature here, what's going to happen 00:55:38.450 --> 00:55:43.170 align:middle line:84% is that stormy's thing is going to get something like 0.999 00:55:43.170 --> 00:55:46.790 align:middle line:84% and everything else is going to get wiped out. 00:55:46.790 --> 00:55:49.970 align:middle line:90% 00:55:49.970 --> 00:55:51.530 align:middle line:84% It's going to get really small, it's 00:55:51.530 --> 00:55:54.290 align:middle line:84% going to get even smaller, and so on and so forth. 00:55:54.290 --> 00:55:56.770 align:middle line:84% And so when T is exactly 0, basically, what that means 00:55:56.770 --> 00:55:59.090 align:middle line:84% is that this is going to be exactly 9-- 00:55:59.090 --> 00:56:01.217 align:middle line:84% 1, and everything is going to just get 0. 00:56:01.217 --> 00:56:03.050 align:middle line:84% So when one of them is 1 and everything else 00:56:03.050 --> 00:56:04.430 align:middle line:84% is 0, when you do sampling from it, 00:56:04.430 --> 00:56:05.890 align:middle line:84% you're just picking the big number, 00:56:05.890 --> 00:56:10.470 align:middle line:84% which means that it becomes greedy decoding. 00:56:10.470 --> 00:56:14.070 align:middle line:84% So that is the value of having temperature as a knob. 00:56:14.070 --> 00:56:16.190 align:middle line:84% Conversely, if you take temperature T 00:56:16.190 --> 00:56:19.750 align:middle line:84% and make it bigger and bigger, as opposed to smaller 00:56:19.750 --> 00:56:22.270 align:middle line:84% and smaller, this distribution is 00:56:22.270 --> 00:56:24.510 align:middle line:84% going to become flat, meaning all the words are going 00:56:24.510 --> 00:56:27.590 align:middle line:90% to have the same probability. 00:56:27.590 --> 00:56:30.790 align:middle line:84% So any one of these words becomes equally likely. 00:56:30.790 --> 00:56:34.350 align:middle line:84% So T close to 0, the biggest word gets picked. 00:56:34.350 --> 00:56:35.990 align:middle line:90% T close to, say-- 00:56:35.990 --> 00:56:39.550 align:middle line:84% exceeds 1, goes to 1.52, any word becomes likely. 00:56:39.550 --> 00:56:41.270 align:middle line:90% It becomes truly random. 00:56:41.270 --> 00:56:44.750 align:middle line:84% So that is the effect of temperature. 00:56:44.750 --> 00:56:47.490 align:middle line:84% And this knob, you can actually tune it. 00:56:47.490 --> 00:56:50.470 align:middle line:90% 00:56:50.470 --> 00:56:53.030 align:middle line:90% All right, so this is called-- 00:56:53.030 --> 00:56:57.530 align:middle line:90% I'm at platform.openai.com. 00:56:57.530 --> 00:56:59.243 align:middle line:84% It's called the OpenAI Playground. 00:56:59.243 --> 00:57:00.910 align:middle line:84% And in this playground, you can actually 00:57:00.910 --> 00:57:03.530 align:middle line:84% put in all the sentences you want, you can choose the model, 00:57:03.530 --> 00:57:05.363 align:middle line:84% and then you can-- it will actually tell you 00:57:05.363 --> 00:57:06.890 align:middle line:90% what the softmax output is. 00:57:06.890 --> 00:57:08.410 align:middle line:90% It's very handy. 00:57:08.410 --> 00:57:11.093 align:middle line:90% So this is where I said-- 00:57:11.093 --> 00:57:13.510 align:middle line:84% so here are a few things I want to draw your attention to. 00:57:13.510 --> 00:57:16.090 align:middle line:84% The first one is, you see temperature here? 00:57:16.090 --> 00:57:17.510 align:middle line:90% The default is 1. 00:57:17.510 --> 00:57:20.630 align:middle line:84% If you make it 0, it becomes greedy decoding. 00:57:20.630 --> 00:57:22.720 align:middle line:84% But you can make it more than 1 if you want. 00:57:22.720 --> 00:57:24.470 align:middle line:84% It will give you all kinds of crazy stuff, 00:57:24.470 --> 00:57:26.956 align:middle line:90% as you will see in a second. 00:57:26.956 --> 00:57:30.170 align:middle line:90% And then they don't have top-k. 00:57:30.170 --> 00:57:32.530 align:middle line:84% They don't have support for top-k, OpenAI, 00:57:32.530 --> 00:57:35.130 align:middle line:84% but they do have support for top-p. 00:57:35.130 --> 00:57:37.010 align:middle line:84% You can put p here in this thing, 00:57:37.010 --> 00:57:38.350 align:middle line:90% and I'll ignore these things. 00:57:38.350 --> 00:57:40.910 align:middle line:84% You can read the documentation to understand those things, 00:57:40.910 --> 00:57:43.662 align:middle line:84% but you can actually ask it to show the probabilities. 00:57:43.662 --> 00:57:45.870 align:middle line:84% So I'm going to ask it to show all the probabilities. 00:57:45.870 --> 00:57:48.750 align:middle line:84% And I'm also going to tell it, don't go nuts. 00:57:48.750 --> 00:57:50.890 align:middle line:90% Just give me like a few outputs. 00:57:50.890 --> 00:57:53.290 align:middle line:90% Let's just call it 30. 00:57:53.290 --> 00:57:55.690 align:middle line:84% And now I'm going to enter some sentences for us 00:57:55.690 --> 00:57:56.750 align:middle line:90% to see what's going on. 00:57:56.750 --> 00:57:58.542 align:middle line:84% So let's enter the same sentence as before, 00:57:58.542 --> 00:58:07.990 align:middle line:84% students at the MIT Sloan School of Management. 00:58:07.990 --> 00:58:09.763 align:middle line:90% Or I think that's what we had. 00:58:09.763 --> 00:58:11.100 align:middle line:90% So Submit. 00:58:11.100 --> 00:58:13.990 align:middle line:90% 00:58:13.990 --> 00:58:16.210 align:middle line:90% So this is what's filling out. 00:58:16.210 --> 00:58:20.310 align:middle line:84% Now, you go click on this word, you get all the probabilities. 00:58:20.310 --> 00:58:21.450 align:middle line:90% Pretty cool. 00:58:21.450 --> 00:58:24.410 align:middle line:84% So you can see invited, given, expected. 00:58:24.410 --> 00:58:26.470 align:middle line:84% These are all some of the things we had. 00:58:26.470 --> 00:58:28.790 align:middle line:84% And so what you can do is you can go in there and say, 00:58:28.790 --> 00:58:29.720 align:middle line:90% here, clearly-- 00:58:29.720 --> 00:58:32.350 align:middle line:90% 00:58:32.350 --> 00:58:33.390 align:middle line:90% aking? 00:58:33.390 --> 00:58:35.990 align:middle line:90% What is that? 00:58:35.990 --> 00:58:37.050 align:middle line:90% That's very weird. 00:58:37.050 --> 00:58:39.792 align:middle line:90% So I'm going to nuke it again. 00:58:39.792 --> 00:58:41.750 align:middle line:84% I'm just going to check to make sure that I use 00:58:41.750 --> 00:58:43.210 align:middle line:90% the same sentence as before. 00:58:43.210 --> 00:58:44.750 align:middle line:90% It's very brittle. 00:58:44.750 --> 00:58:48.645 align:middle line:84% Students, MIT School of Management are-- 00:58:48.645 --> 00:58:50.590 align:middle line:90% are. 00:58:50.590 --> 00:58:52.550 align:middle line:90% Oh, I know what it is. 00:58:52.550 --> 00:58:57.870 align:middle line:90% GPT 3.5. 00:58:57.870 --> 00:58:59.170 align:middle line:90% So let's try that again. 00:58:59.170 --> 00:59:04.060 align:middle line:90% 00:59:04.060 --> 00:59:05.288 align:middle line:90% So invited, 3.18. 00:59:05.288 --> 00:59:06.080 align:middle line:90% That's what we had? 00:59:06.080 --> 00:59:08.040 align:middle line:90% Invited 3.19, 3.9. 00:59:08.040 --> 00:59:09.080 align:middle line:90% Close enough. 00:59:09.080 --> 00:59:10.520 align:middle line:90% So this is what we have. 00:59:10.520 --> 00:59:14.240 align:middle line:84% And now, if you wanted to force it to choose invited here, 00:59:14.240 --> 00:59:18.325 align:middle line:84% you just go in there and make the temperature 0. 00:59:18.325 --> 00:59:20.200 align:middle line:84% Temperature 0 means it's always going to pick 00:59:20.200 --> 00:59:21.450 align:middle line:90% the best one, greedy decoding. 00:59:21.450 --> 00:59:24.920 align:middle line:90% So you can hit it again. 00:59:24.920 --> 00:59:26.420 align:middle line:90% And it better give you invited. 00:59:26.420 --> 00:59:26.920 align:middle line:90% See? 00:59:26.920 --> 00:59:29.520 align:middle line:90% It's giving you invited. 00:59:29.520 --> 00:59:32.760 align:middle line:84% So that's how we manipulate it using temperature. 00:59:32.760 --> 00:59:35.300 align:middle line:84% You can also ask it-- you can also manipulate top-p. 00:59:35.300 --> 00:59:37.360 align:middle line:90% You can do all these things. 00:59:37.360 --> 00:59:40.885 align:middle line:84% But so people actually use it very heavily for debugging. 00:59:40.885 --> 00:59:42.760 align:middle line:84% And when they're playing with a bunch of data 00:59:42.760 --> 00:59:44.580 align:middle line:84% with a model for that particular use case, 00:59:44.580 --> 00:59:45.800 align:middle line:84% you just play with it to get a sense 00:59:45.800 --> 00:59:47.967 align:middle line:84% for what kinds of probability distributions you see. 00:59:47.967 --> 00:59:51.840 align:middle line:84% And then you can fine-tune it using that knowledge. 00:59:51.840 --> 00:59:54.400 align:middle line:90% So, yeah, check this out. 00:59:54.400 --> 00:59:59.140 align:middle line:84% Oh, I said that, if the temperature goes above 1 00:59:59.140 --> 01:00:03.000 align:middle line:84% to a higher number, every word in that 50,000 becomes sort 01:00:03.000 --> 01:00:05.765 align:middle line:84% of equally likely, which means it's going to produce garbage. 01:00:05.765 --> 01:00:07.890 align:middle line:84% So let's actually see garbage production in action. 01:00:07.890 --> 01:00:09.100 align:middle line:90% [CHUCKLING] 01:00:09.100 --> 01:00:11.780 align:middle line:84% So all right, let's just nuke this. 01:00:11.780 --> 01:00:14.860 align:middle line:84% And I'm going to take the temperature and max it. 01:00:14.860 --> 01:00:20.100 align:middle line:84% I'm going to call it 2, which means that literally anything is 01:00:20.100 --> 01:00:21.720 align:middle line:90% possible. 01:00:21.720 --> 01:00:22.220 align:middle line:90% Submit. 01:00:22.220 --> 01:00:25.780 align:middle line:90% 01:00:25.780 --> 01:00:28.710 align:middle line:84% Ladies and gentlemen, I present to you a modern large language 01:00:28.710 --> 01:00:29.210 align:middle line:90% model. 01:00:29.210 --> 01:00:30.468 align:middle line:90% [LAUGHTER] 01:00:30.468 --> 01:00:31.456 align:middle line:90% 01:00:31.456 --> 01:00:34.740 align:middle line:90% [SIDE CONVERSATIONS] 01:00:34.740 --> 01:00:35.860 align:middle line:90% 01:00:35.860 --> 01:00:38.020 align:middle line:90% Isn't it shocking? 01:00:38.020 --> 01:00:40.975 align:middle line:84% Because when we work with these language models, we have-- 01:00:40.975 --> 01:00:42.600 align:middle line:84% when we see it doing some smart things, 01:00:42.600 --> 01:00:45.860 align:middle line:84% we always ascribe some level of interesting abilities, 01:00:45.860 --> 01:00:47.300 align:middle line:90% and intelligence, and so on. 01:00:47.300 --> 01:00:49.660 align:middle line:84% And then you realize, all I had to go in, go in there 01:00:49.660 --> 01:00:52.740 align:middle line:84% and change one parameter, and it's garbage. 01:00:52.740 --> 01:00:54.860 align:middle line:84% So you can see the amount of garbage 01:00:54.860 --> 01:00:57.180 align:middle line:84% it's showing, just by twiddling one parameter. 01:00:57.180 --> 01:00:59.505 align:middle line:84% So you have to be in production use cases-- 01:00:59.505 --> 01:01:00.880 align:middle line:84% when you're building applications 01:01:00.880 --> 01:01:02.180 align:middle line:84% on top of these large language models, 01:01:02.180 --> 01:01:04.800 align:middle line:84% you've got to be very, very careful with these parameters. 01:01:04.800 --> 01:01:06.840 align:middle line:90% So pay attention. 01:01:06.840 --> 01:01:09.510 align:middle line:84% All right, so what did I have next? 01:01:09.510 --> 01:01:13.600 align:middle line:90% 01:01:13.600 --> 01:01:18.930 align:middle line:84% So that brings us to sort of the end of the decoding section. 01:01:18.930 --> 01:01:22.162 align:middle line:90% 01:01:22.162 --> 01:01:25.200 align:middle line:84% Oh, see, now I'm going to switch gears and talk 01:01:25.200 --> 01:01:29.360 align:middle line:84% about tokenization, which is that when-- 01:01:29.360 --> 01:01:31.760 align:middle line:84% so far, in all the things we have 01:01:31.760 --> 01:01:33.760 align:middle line:84% done, including the homeworks and so on, we 01:01:33.760 --> 01:01:37.000 align:middle line:84% looked at this tokenization, the standard process 01:01:37.000 --> 01:01:39.580 align:middle line:84% for taking a bunch of text and vectorizing it, 01:01:39.580 --> 01:01:45.140 align:middle line:84% which was the STIE process, standardize, tokenize, index, 01:01:45.140 --> 01:01:46.220 align:middle line:90% and then encode. 01:01:46.220 --> 01:01:47.720 align:middle line:84% And the standardization, I mentioned 01:01:47.720 --> 01:01:51.880 align:middle line:84% earlier, strips out punctuation, lowercases everything, 01:01:51.880 --> 01:01:55.240 align:middle line:84% sometimes removes stop words like a and things like that. 01:01:55.240 --> 01:01:57.045 align:middle line:84% It also does these things called stemming. 01:01:57.045 --> 01:01:58.420 align:middle line:84% But it turns out, if you actually 01:01:58.420 --> 01:02:00.613 align:middle line:84% work with something like GPT, you 01:02:00.613 --> 01:02:04.360 align:middle line:84% know that it hasn't stripped out punctuation. 01:02:04.360 --> 01:02:05.960 align:middle line:90% The punctuation is really good. 01:02:05.960 --> 01:02:08.020 align:middle line:84% It uses case, uppercase and lowercase. 01:02:08.020 --> 01:02:11.140 align:middle line:84% And, in fact, even better, you can actually make up a word 01:02:11.140 --> 01:02:13.820 align:middle line:84% as part of your question, and it'll use that word consistently 01:02:13.820 --> 01:02:14.980 align:middle line:90% in the output. 01:02:14.980 --> 01:02:19.530 align:middle line:84% So just for fun, I made up a word. 01:02:19.530 --> 01:02:21.753 align:middle line:90% 01:02:21.753 --> 01:02:23.420 align:middle line:84% I just did this yesterday or day before. 01:02:23.420 --> 01:02:25.520 align:middle line:84% I said, here's a new word and its definition. 01:02:25.520 --> 01:02:30.020 align:middle line:84% The word is reldoh, hodler backwards. 01:02:30.020 --> 01:02:32.020 align:middle line:84% I said the definition, a student who understands 01:02:32.020 --> 01:02:33.582 align:middle line:90% deep learning backwards. 01:02:33.582 --> 01:02:35.040 align:middle line:84% Please use this word in a sentence. 01:02:35.040 --> 01:02:37.080 align:middle line:84% And here is a sentence it's coming up with. 01:02:37.080 --> 01:02:39.820 align:middle line:90% 01:02:39.820 --> 01:02:41.248 align:middle line:90% I was a little shocked. 01:02:41.248 --> 01:02:43.040 align:middle line:84% During the advanced neural network seminar, 01:02:43.040 --> 01:02:45.820 align:middle line:84% it became evident that Jane was a true reldoh, 01:02:45.820 --> 01:02:47.740 align:middle line:84% effortlessly explaining even the most complex 01:02:47.740 --> 01:02:49.480 align:middle line:84% deep learning concepts in reverse order. 01:02:49.480 --> 01:02:51.160 align:middle line:90% [LAUGHTER] 01:02:51.160 --> 01:02:54.520 align:middle line:84% So it clearly knows how to use anything you make up with. 01:02:54.520 --> 01:02:58.320 align:middle line:84% So it has the ability to compose things from scratch as opposed 01:02:58.320 --> 01:03:00.000 align:middle line:90% to just looking up stuff. 01:03:00.000 --> 01:03:01.580 align:middle line:84% So where is the thing coming from? 01:03:01.580 --> 01:03:02.688 align:middle line:90% That's the question. 01:03:02.688 --> 01:03:04.480 align:middle line:84% And the answer is this very beautiful thing 01:03:04.480 --> 01:03:07.230 align:middle line:84% called byte pair encoding, which we'll look at next. 01:03:07.230 --> 01:03:10.440 align:middle line:90% 01:03:10.440 --> 01:03:15.080 align:middle line:84% So all right, so here, when we look at this process, 01:03:15.080 --> 01:03:16.640 align:middle line:84% the disadvantages are, well, some 01:03:16.640 --> 01:03:17.640 align:middle line:84% of the things we have discussed, which 01:03:17.640 --> 01:03:19.660 align:middle line:84% is that we want to be able to preserve punctuation, 01:03:19.660 --> 01:03:20.820 align:middle line:84% we want to be able to preserve case, 01:03:20.820 --> 01:03:23.445 align:middle line:84% we want to be able to handle new words, and so on and so forth. 01:03:23.445 --> 01:03:28.473 align:middle line:84% So sort of the modern models, like BERT and so on, 01:03:28.473 --> 01:03:30.140 align:middle line:84% they use different tokenization schemes. 01:03:30.140 --> 01:03:32.200 align:middle line:84% They don't actually do the STIE thing. 01:03:32.200 --> 01:03:36.560 align:middle line:84% And the GPT family uses byte pair encoding, BPE. 01:03:36.560 --> 01:03:38.880 align:middle line:84% BERT uses something called wordpiece. 01:03:38.880 --> 01:03:42.280 align:middle line:84% All of these ways of encoding, the fundamental idea 01:03:42.280 --> 01:03:44.437 align:middle line:90% is to say, well, you know what? 01:03:44.437 --> 01:03:46.020 align:middle line:84% Whatever language you're working with, 01:03:46.020 --> 01:03:47.395 align:middle line:84% why don't we start, first of all, 01:03:47.395 --> 01:03:49.297 align:middle line:84% with all the individual characters? 01:03:49.297 --> 01:03:51.880 align:middle line:84% Because if you could actually work with individual characters, 01:03:51.880 --> 01:03:54.740 align:middle line:84% you can clearly compose any word that comes up. 01:03:54.740 --> 01:03:58.137 align:middle line:84% Reldoh is just R-E-L-D-O-H, six tokens, 01:03:58.137 --> 01:04:00.220 align:middle line:84% if you're working with characters at the character 01:04:00.220 --> 01:04:00.820 align:middle line:90% level. 01:04:00.820 --> 01:04:03.500 align:middle line:84% But working only with characters is not great 01:04:03.500 --> 01:04:06.760 align:middle line:84% because that means that the model, 01:04:06.760 --> 01:04:08.760 align:middle line:84% you're giving it no information about the world. 01:04:08.760 --> 01:04:11.680 align:middle line:84% It has to learn every word from scratch, what the word means 01:04:11.680 --> 01:04:12.880 align:middle line:90% and so on and so forth. 01:04:12.880 --> 01:04:16.640 align:middle line:84% So it would be nice if we can actually give it words as well, 01:04:16.640 --> 01:04:19.780 align:middle line:84% but we don't want to give it infrequent words 01:04:19.780 --> 01:04:21.880 align:middle line:84% because infrequent words, by definition, 01:04:21.880 --> 01:04:24.822 align:middle line:84% are not worth adding to your vocabulary. 01:04:24.822 --> 01:04:26.780 align:middle line:84% They're just going to take up another embedding 01:04:26.780 --> 01:04:28.300 align:middle line:90% vector and things like that. 01:04:28.300 --> 01:04:31.193 align:middle line:84% For infrequent words, we'll just compose them. 01:04:31.193 --> 01:04:32.860 align:middle line:84% We'll actually construct them on the fly 01:04:32.860 --> 01:04:35.767 align:middle line:84% because we can always use characters. 01:04:35.767 --> 01:04:37.600 align:middle line:84% So we don't want to put every word in there. 01:04:37.600 --> 01:04:39.500 align:middle line:84% We only want to put frequent words. 01:04:39.500 --> 01:04:42.580 align:middle line:84% But to give this thing the ability to compose new words 01:04:42.580 --> 01:04:44.760 align:middle line:84% and not always have to go to characters, 01:04:44.760 --> 01:04:46.900 align:middle line:90% we will give it parts of words. 01:04:46.900 --> 01:04:48.920 align:middle line:90% These are called subwords. 01:04:48.920 --> 01:04:52.240 align:middle line:84% So the key idea is that, let's come up 01:04:52.240 --> 01:04:55.560 align:middle line:84% with a way to build a vocabulary which has characters, 01:04:55.560 --> 01:04:59.360 align:middle line:84% full words that are frequent enough to be worth adding, 01:04:59.360 --> 01:05:02.160 align:middle line:84% and subwords, or word fragments that occur frequently 01:05:02.160 --> 01:05:03.300 align:middle line:90% enough to be worth adding. 01:05:03.300 --> 01:05:08.620 align:middle line:84% So, for example, the word standardize, i-z-e, normalize, 01:05:08.620 --> 01:05:10.120 align:middle line:84% standardize, and so on and so forth, 01:05:10.120 --> 01:05:11.950 align:middle line:84% ize is going to show up a lot in many places. 01:05:11.950 --> 01:05:14.100 align:middle line:84% So you don't want to have standardized, and normalized, 01:05:14.100 --> 01:05:14.420 align:middle line:90% and so on. 01:05:14.420 --> 01:05:15.380 align:middle line:90% You just want to have ize. 01:05:15.380 --> 01:05:17.213 align:middle line:84% You can just attach it to all kinds of words 01:05:17.213 --> 01:05:18.940 align:middle line:90% and make it all work. 01:05:18.940 --> 01:05:22.180 align:middle line:84% So that's the basic idea of all these tokenization schemes. 01:05:22.180 --> 01:05:24.825 align:middle line:84% And BPE is one such way to figure out 01:05:24.825 --> 01:05:27.200 align:middle line:84% how to actually construct this vocabulary from a training 01:05:27.200 --> 01:05:28.266 align:middle line:90% corpus. 01:05:28.266 --> 01:05:30.660 align:middle line:84% And, by the way, when I say characters, 01:05:30.660 --> 01:05:33.640 align:middle line:84% this will include not just uppercase, lowercase, alphabets, 01:05:33.640 --> 01:05:34.180 align:middle line:90% and numbers. 01:05:34.180 --> 01:05:37.000 align:middle line:84% It will also include punctuation, 01:05:37.000 --> 01:05:40.560 align:middle line:84% so that all these things just become atomic units. 01:05:40.560 --> 01:05:43.280 align:middle line:84% All right, so what we're going-- the way 01:05:43.280 --> 01:05:46.530 align:middle line:84% BPE works is that we're going to start 01:05:46.530 --> 01:05:48.342 align:middle line:90% with each character as a token. 01:05:48.342 --> 01:05:50.050 align:middle line:84% And I'll talk about the rest of the thing 01:05:50.050 --> 01:05:51.150 align:middle line:90% on the page in just a moment. 01:05:51.150 --> 01:05:51.870 align:middle line:90% Don't worry about it. 01:05:51.870 --> 01:05:53.662 align:middle line:84% We'll start with each character as a token. 01:05:53.662 --> 01:05:56.290 align:middle line:84% So let's say that your training corpus is just 01:05:56.290 --> 01:05:58.650 align:middle line:84% a single sentence, the cat sat on the mat. 01:05:58.650 --> 01:06:03.470 align:middle line:84% And even though GPT does not actually do any lowercasing, 01:06:03.470 --> 01:06:04.850 align:middle line:90% it'll just actually use-- 01:06:04.850 --> 01:06:07.770 align:middle line:84% T-H-E uppercase is different than t-h-e lowercase. 01:06:07.770 --> 01:06:10.470 align:middle line:84% Just for simplicity, I'm just going to standardize it here. 01:06:10.470 --> 01:06:12.270 align:middle line:84% So it just becomes the cat sat on the mat. 01:06:12.270 --> 01:06:15.170 align:middle line:84% And then I'm going to write it in this form, where I basically 01:06:15.170 --> 01:06:16.460 align:middle line:90% put a comma after every word. 01:06:16.460 --> 01:06:18.210 align:middle line:84% And then I put a little underscore to show 01:06:18.210 --> 01:06:20.128 align:middle line:90% the space between the words. 01:06:20.128 --> 01:06:21.670 align:middle line:84% I'm going to write it in this format, 01:06:21.670 --> 01:06:24.730 align:middle line:84% and it'll become clear why I'm writing it in just a second. 01:06:24.730 --> 01:06:28.130 align:middle line:84% Now, my starting vocabulary is just all the individual letters 01:06:28.130 --> 01:06:29.930 align:middle line:90% in the training corpus. 01:06:29.930 --> 01:06:32.530 align:middle line:84% So the starting is just whatever, all these letters. 01:06:32.530 --> 01:06:33.730 align:middle line:90% That's it. 01:06:33.730 --> 01:06:35.470 align:middle line:90% And this is a starting point. 01:06:35.470 --> 01:06:38.410 align:middle line:84% And now what we do-- and this is the key step-- 01:06:38.410 --> 01:06:41.810 align:middle line:84% we merge tokens that most frequently 01:06:41.810 --> 01:06:44.350 align:middle line:90% occur right next to each other. 01:06:44.350 --> 01:06:47.070 align:middle line:84% So if two characters or two tokens 01:06:47.070 --> 01:06:49.630 align:middle line:84% are occurring right next to each other a lot, 01:06:49.630 --> 01:06:51.270 align:middle line:84% let's just merge them because they seem 01:06:51.270 --> 01:06:53.070 align:middle line:90% to be occurring a lot together. 01:06:53.070 --> 01:06:54.390 align:middle line:90% May as well merge them. 01:06:54.390 --> 01:06:57.932 align:middle line:84% And so here, for example, I've listed the frequency 01:06:57.932 --> 01:06:58.890 align:middle line:90% of the adjacent tokens. 01:06:58.890 --> 01:07:01.310 align:middle line:84% So, for example, if you look at t, h, t, 01:07:01.310 --> 01:07:03.870 align:middle line:84% h shows up right after each other here. 01:07:03.870 --> 01:07:06.150 align:middle line:90% It also shows up here. 01:07:06.150 --> 01:07:08.550 align:middle line:84% So, therefore, it shows up twice. 01:07:08.550 --> 01:07:11.727 align:middle line:84% Now, h, e, again, is showing up here. 01:07:11.727 --> 01:07:12.810 align:middle line:90% It's also showing up here. 01:07:12.810 --> 01:07:14.550 align:middle line:90% So that also shows up twice. 01:07:14.550 --> 01:07:16.913 align:middle line:84% c, a on the other hand, is only showing up here. 01:07:16.913 --> 01:07:18.330 align:middle line:84% It's not showing up anywhere else. 01:07:18.330 --> 01:07:19.870 align:middle line:90% So it shows up once. 01:07:19.870 --> 01:07:24.452 align:middle line:84% a, t shows up three times, in mat, sat, and in cat, 01:07:24.452 --> 01:07:25.410 align:middle line:90% and so on and so forth. 01:07:25.410 --> 01:07:26.630 align:middle line:90% You get the idea. 01:07:26.630 --> 01:07:30.202 align:middle line:84% So you're just looking at pairwise adjacent tokens. 01:07:30.202 --> 01:07:32.410 align:middle line:84% And you pick the most frequent one that's showing up, 01:07:32.410 --> 01:07:34.230 align:middle line:84% which in this case happens to be a, t. 01:07:34.230 --> 01:07:36.810 align:middle line:84% And then you take a and t and you merge them. 01:07:36.810 --> 01:07:37.910 align:middle line:90% So it becomes at. 01:07:37.910 --> 01:07:40.430 align:middle line:90% 01:07:40.430 --> 01:07:43.047 align:middle line:84% So when you do that, when you merge them 01:07:43.047 --> 01:07:45.130 align:middle line:84% and then you add that new token that you have just 01:07:45.130 --> 01:07:48.130 align:middle line:84% literally created to your vocabulary list, 01:07:48.130 --> 01:07:50.610 align:middle line:84% and then you update the corpus to reflect the merge 01:07:50.610 --> 01:07:54.710 align:middle line:84% you just did, so now the corpus becomes, the cat sat on the mat. 01:07:54.710 --> 01:07:57.250 align:middle line:84% But in this case, there is no a and t separately. 01:07:57.250 --> 01:08:02.410 align:middle line:84% There is just the at combo token here. 01:08:02.410 --> 01:08:06.035 align:middle line:84% Are we good with this step so far? 01:08:06.035 --> 01:08:07.910 align:middle line:84% Take the most frequent things and merge them. 01:08:07.910 --> 01:08:12.570 align:middle line:90% 01:08:12.570 --> 01:08:14.147 align:middle line:90% It's a way to compress the data. 01:08:14.147 --> 01:08:16.689 align:middle line:84% In fact, the algorithm came from someone trying to figure out 01:08:16.689 --> 01:08:17.680 align:middle line:90% a way to compress data. 01:08:17.680 --> 01:08:22.075 align:middle line:90% 01:08:22.075 --> 01:08:22.950 align:middle line:90% Think of it this way. 01:08:22.950 --> 01:08:26.149 align:middle line:84% Suppose I tell you, I want you to compress a message 01:08:26.149 --> 01:08:27.970 align:middle line:90% I'm going to send to you. 01:08:27.970 --> 01:08:30.050 align:middle line:84% And then you look at all the tasks, messages 01:08:30.050 --> 01:08:31.149 align:middle line:90% you've had to deal with. 01:08:31.149 --> 01:08:35.970 align:middle line:84% And it turns out you're finding that certain characters are 01:08:35.970 --> 01:08:38.609 align:middle line:84% occurring next to each other all the time. 01:08:38.609 --> 01:08:41.990 align:middle line:84% Maybe, just for argument, let's say ABC shows up ridiculously 01:08:41.990 --> 01:08:43.430 align:middle line:90% often in the messaging. 01:08:43.430 --> 01:08:45.194 align:middle line:84% And then you'd be like, if it's always 01:08:45.194 --> 01:08:46.569 align:middle line:84% showing up all the time together, 01:08:46.569 --> 01:08:47.777 align:middle line:90% why treat it as three things? 01:08:47.777 --> 01:08:49.189 align:middle line:90% Let me just call it one thing. 01:08:49.189 --> 01:08:50.790 align:middle line:90% ABC, that's it. 01:08:50.790 --> 01:08:52.790 align:middle line:84% You send a single token called ABC every time 01:08:52.790 --> 01:08:57.310 align:middle line:84% you need to send ABC, not A, B, C. That's the basic idea. 01:08:57.310 --> 01:08:59.710 align:middle line:84% So here, if you come here, that's what we have. 01:08:59.710 --> 01:09:02.229 align:middle line:84% And then what we do is now we do, again, 01:09:02.229 --> 01:09:05.649 align:middle line:84% this calculation of adjacency tokens on this updated corpus. 01:09:05.649 --> 01:09:08.229 align:middle line:84% And you can see here, t, h shows up once, 01:09:08.229 --> 01:09:09.689 align:middle line:90% t, h shows up here twice. 01:09:09.689 --> 01:09:11.630 align:middle line:90% So you get two. 01:09:11.630 --> 01:09:13.029 align:middle line:90% H, e shows up twice. 01:09:13.029 --> 01:09:14.990 align:middle line:90% Everything else shows up once. 01:09:14.990 --> 01:09:17.835 align:middle line:84% And, yeah, when many things are showing up with equal frequency, 01:09:17.835 --> 01:09:19.210 align:middle line:84% just pick one randomly from this. 01:09:19.210 --> 01:09:23.070 align:middle line:84% So we pick up t,h and we merge that, which means that we add t, 01:09:23.070 --> 01:09:25.310 align:middle line:90% h to our vocabulary. 01:09:25.310 --> 01:09:27.370 align:middle line:84% And once we do that we update the corpus. 01:09:27.370 --> 01:09:28.883 align:middle line:90% And now we have-- 01:09:28.883 --> 01:09:31.002 align:middle line:84% t, h is now one thing fused together, 01:09:31.002 --> 01:09:32.710 align:middle line:84% along with the previous thing, a, t, that 01:09:32.710 --> 01:09:34.990 align:middle line:90% had been fused together. 01:09:34.990 --> 01:09:36.990 align:middle line:84% That is the corpus after the second merge. 01:09:36.990 --> 01:09:38.521 align:middle line:90% And then we do the same thing. 01:09:38.521 --> 01:09:40.229 align:middle line:84% We find the frequency of adjacent tokens. 01:09:40.229 --> 01:09:42.430 align:middle line:84% So it turns out t, h and e are showing up twice. 01:09:42.430 --> 01:09:43.910 align:middle line:84% Everything else is showing up once. 01:09:43.910 --> 01:09:46.689 align:middle line:84% So we take t, h, e, merge it to get the. 01:09:46.689 --> 01:09:48.250 align:middle line:90% Boom, the. 01:09:48.250 --> 01:09:50.850 align:middle line:84% And now we have the cat sat on the mat. 01:09:50.850 --> 01:09:55.810 align:middle line:84% So this process continues till we reach a predefined limit 01:09:55.810 --> 01:09:57.650 align:middle line:90% for our vocabulary. 01:09:57.650 --> 01:10:01.763 align:middle line:84% Now, as it turns out, when they built GPT-2 and GPT-- 01:10:01.763 --> 01:10:02.430 align:middle line:90% let me just see. 01:10:02.430 --> 01:10:04.630 align:middle line:84% I think I did some digging around on this thing. 01:10:04.630 --> 01:10:08.610 align:middle line:84% Yeah, so GPT-2 and 3, they set the vocabulary size to be 01:10:08.610 --> 01:10:10.450 align:middle line:90% roughly 50,000. 01:10:10.450 --> 01:10:13.570 align:middle line:84% So it basically kept on doing this till it hit our limit 01:10:13.570 --> 01:10:15.770 align:middle line:90% of 50,000, then it stopped. 01:10:15.770 --> 01:10:17.810 align:middle line:84% GPT-4 on the other hand, actually, went-- 01:10:17.810 --> 01:10:23.290 align:middle line:84% goes all the way to 100,000 vocabulary size. 01:10:23.290 --> 01:10:27.150 align:middle line:90% So this is BPE in action. 01:10:27.150 --> 01:10:29.650 align:middle line:84% And so what's going to happen is once you finish this thing, 01:10:29.650 --> 01:10:32.275 align:middle line:84% and you have vocabulary, and you have all these things that you 01:10:32.275 --> 01:10:35.710 align:middle line:84% have merged, when a new piece of text comes in, the merges-- 01:10:35.710 --> 01:10:38.270 align:middle line:84% remember, here, we merged a, t to get at. 01:10:38.270 --> 01:10:40.755 align:middle line:90% This t, h became this and so on. 01:10:40.755 --> 01:10:42.130 align:middle line:84% When a new piece of text arrives, 01:10:42.130 --> 01:10:45.150 align:middle line:84% the tokenization will apply the merges in the exact same order. 01:10:45.150 --> 01:10:47.870 align:middle line:84% So if the new text that comes in is the rat, 01:10:47.870 --> 01:10:49.520 align:middle line:84% it's first going to apply the a, t 01:10:49.520 --> 01:10:53.427 align:middle line:84% to at to become-- fuse this here and then going to fuse t, 01:10:53.427 --> 01:10:54.010 align:middle line:90% h to get this. 01:10:54.010 --> 01:10:56.150 align:middle line:84% And then it's going to fuse t, h and e to get that. 01:10:56.150 --> 01:10:58.870 align:middle line:84% And the final list of tokens that goes in to your model 01:10:58.870 --> 01:11:01.470 align:middle line:84% is going to be the token for the, the token for space, 01:11:01.470 --> 01:11:03.410 align:middle line:84% and the token for r, and the token for at. 01:11:03.410 --> 01:11:06.470 align:middle line:90% 01:11:06.470 --> 01:11:07.840 align:middle line:90% So let's see this in action. 01:11:07.840 --> 01:11:12.950 align:middle line:90% 01:11:12.950 --> 01:11:16.790 align:middle line:84% OpenAI has its own thing, but I found this site 01:11:16.790 --> 01:11:18.010 align:middle line:90% to be really good. 01:11:18.010 --> 01:11:26.030 align:middle line:84% So let's tokenize hands-on deep learning. 01:11:26.030 --> 01:11:30.870 align:middle line:84% So you can see here, look at this. 01:11:30.870 --> 01:11:34.480 align:middle line:84% So H, uppercase H, is its own token. 01:11:34.480 --> 01:11:36.720 align:middle line:90% It's token number 39. 01:11:36.720 --> 01:11:39.560 align:middle line:90% And its own token. 01:11:39.560 --> 01:11:41.160 align:middle line:90% Dash is its own token. 01:11:41.160 --> 01:11:42.140 align:middle line:90% On is its own token. 01:11:42.140 --> 01:11:44.140 align:middle line:84% And then space deep is its token, 01:11:44.140 --> 01:11:46.800 align:middle line:90% and space learning is its token. 01:11:46.800 --> 01:11:50.360 align:middle line:84% Note one thing, suppose you had said-- 01:11:50.360 --> 01:11:53.560 align:middle line:84% let's just say you just had deep, deep learning. 01:11:53.560 --> 01:11:56.880 align:middle line:84% Deep has a different token than space deep. 01:11:56.880 --> 01:11:59.840 align:middle line:90% 01:11:59.840 --> 01:12:02.520 align:middle line:84% What they have realized is that most words are actually 01:12:02.520 --> 01:12:05.760 align:middle line:84% going to show up after the space, after a space, 01:12:05.760 --> 01:12:06.900 align:middle line:90% much more likely. 01:12:06.900 --> 01:12:10.500 align:middle line:84% So having a space attached to the beginning of the word, 01:12:10.500 --> 01:12:13.420 align:middle line:84% it sort of saves you a lot of compute and so on and so forth 01:12:13.420 --> 01:12:15.800 align:middle line:84% because they will, in fact, arrive almost all the time 01:12:15.800 --> 01:12:17.220 align:middle line:90% with the space before it. 01:12:17.220 --> 01:12:18.595 align:middle line:84% That's the way they have attached 01:12:18.595 --> 01:12:20.520 align:middle line:90% the space to the word itself. 01:12:20.520 --> 01:12:25.685 align:middle line:84% And note that deep learning, deep and deep-- 01:12:25.685 --> 01:12:27.060 align:middle line:84% actually, let's call it this way. 01:12:27.060 --> 01:12:30.880 align:middle line:90% 01:12:30.880 --> 01:12:34.860 align:middle line:90% So deep and deep are different. 01:12:34.860 --> 01:12:35.640 align:middle line:90% There is deep. 01:12:35.640 --> 01:12:38.288 align:middle line:84% There is-- so, clearly, it's taking case into account. 01:12:38.288 --> 01:12:39.580 align:middle line:90% Then I put an exclamation here. 01:12:39.580 --> 01:12:41.500 align:middle line:90% Boom, that. 01:12:41.500 --> 01:12:48.180 align:middle line:84% And so, ultimately, what goes in when you have a phrase like, 01:12:48.180 --> 01:12:51.620 align:middle line:90% sat on the mat-- 01:12:51.620 --> 01:12:53.180 align:middle line:90% so the cat sat on the mat. 01:12:53.180 --> 01:12:56.420 align:middle line:84% And you can see here, uppercase The. 01:12:56.420 --> 01:13:01.460 align:middle line:84% And then let's just do another thing here. 01:13:01.460 --> 01:13:06.580 align:middle line:84% So uppercase The with a space is 383. 01:13:06.580 --> 01:13:09.580 align:middle line:90% Lowercase the is 262. 01:13:09.580 --> 01:13:12.038 align:middle line:84% And then that's distinct from just the without any space 01:13:12.038 --> 01:13:13.080 align:middle line:90% that's a different thing. 01:13:13.080 --> 01:13:14.660 align:middle line:90% So these are all the tokens. 01:13:14.660 --> 01:13:18.860 align:middle line:90% Now, let's try something. 01:13:18.860 --> 01:13:21.750 align:middle line:90% Let's try Jane. 01:13:21.750 --> 01:13:24.740 align:middle line:90% 01:13:24.740 --> 01:13:27.260 align:middle line:84% So Jane is one token, which is great. 01:13:27.260 --> 01:13:28.740 align:middle line:90% And is another token. 01:13:28.740 --> 01:13:30.026 align:middle line:90% Let's see, Rama. 01:13:30.026 --> 01:13:31.680 align:middle line:90% Oh, darn. 01:13:31.680 --> 01:13:37.080 align:middle line:84% My name wasn't worthy enough to be its own token. 01:13:37.080 --> 01:13:43.340 align:middle line:84% But strangely enough, I was very surprised by this. 01:13:43.340 --> 01:13:46.310 align:middle line:84% So if I put Rama in lowercase, it's its own token. 01:13:46.310 --> 01:13:48.320 align:middle line:90% [LAUGHTER] 01:13:48.320 --> 01:13:52.680 align:middle line:84% I have no idea what they were scraping, which websites. 01:13:52.680 --> 01:13:59.000 align:middle line:84% And if I put Jane here, now j has become its token with space 01:13:59.000 --> 01:14:01.480 align:middle line:90% and ane has become different. 01:14:01.480 --> 01:14:04.520 align:middle line:84% So the tokenization is a very interesting thing, 01:14:04.520 --> 01:14:06.500 align:middle line:84% and it works in very interesting ways, 01:14:06.500 --> 01:14:09.160 align:middle line:84% but that's the basic idea, what's going on under the hood. 01:14:09.160 --> 01:14:10.920 align:middle line:84% I would encourage you to check out 01:14:10.920 --> 01:14:13.100 align:middle line:84% your names to see if it's actually been tokenized. 01:14:13.100 --> 01:14:14.357 align:middle line:90% So all right, I'm done. 01:14:14.357 --> 01:14:14.940 align:middle line:90% Thanks, folks. 01:14:14.940 --> 01:14:16.023 align:middle line:90% I'll see you on Wednesday. 01:14:16.023 --> 01:14:18.370 align:middle line:90% [APPLAUSE] 01:14:18.370 --> 01:14:29.000 align:middle line:90%