WEBVTT 00:00:00.000 --> 00:00:00.990 align:middle line:90% 00:00:00.990 --> 00:00:02.475 align:middle line:90% [SQUEAKING] 00:00:02.475 --> 00:00:03.960 align:middle line:90% [RUSTLING] 00:00:03.960 --> 00:00:05.940 align:middle line:90% [CLICKING] 00:00:05.940 --> 00:00:16.660 align:middle line:90% 00:00:16.660 --> 00:00:18.200 align:middle line:90% RAMA RAMAKRISHNAN: OK. 00:00:18.200 --> 00:00:21.680 align:middle line:84% So let's continue the journey we started last time. 00:00:21.680 --> 00:00:24.520 align:middle line:84% So what we're going to do, if you remember in the last class, 00:00:24.520 --> 00:00:27.000 align:middle line:84% we showed how we can actually build 00:00:27.000 --> 00:00:29.400 align:middle line:84% an autoregressive large language model-- 00:00:29.400 --> 00:00:30.000 align:middle line:90% a.k.a. 00:00:30.000 --> 00:00:32.439 align:middle line:90% a causal large language model-- 00:00:32.439 --> 00:00:36.260 align:middle line:84% using this idea of causal encoder, 00:00:36.260 --> 00:00:37.735 align:middle line:90% a transformer causal encoder. 00:00:37.735 --> 00:00:39.360 align:middle line:84% And then we showed how you can actually 00:00:39.360 --> 00:00:42.280 align:middle line:84% take a bunch of sentences and use next-word-prediction 00:00:42.280 --> 00:00:43.520 align:middle line:90% and just run it through. 00:00:43.520 --> 00:00:45.840 align:middle line:90% And boom, you get GPT-3. 00:00:45.840 --> 00:00:47.600 align:middle line:90% So that's what we saw last time. 00:00:47.600 --> 00:00:51.720 align:middle line:84% I want to point out an important clarification/correction, which 00:00:51.720 --> 00:00:54.880 align:middle line:84% is that when we work with large language models, 00:00:54.880 --> 00:00:58.080 align:middle line:84% unlike when we work with BERT, for instance, when we work 00:00:58.080 --> 00:01:00.730 align:middle line:84% with these kinds of causal models, actually, 00:01:00.730 --> 00:01:02.670 align:middle line:84% when the contextual embeddings come out, 00:01:02.670 --> 00:01:05.489 align:middle line:84% you don't actually have to use ReLU activations here. 00:01:05.489 --> 00:01:07.370 align:middle line:84% You can literally just run it through just 00:01:07.370 --> 00:01:10.170 align:middle line:84% a single dense layer with linear activations 00:01:10.170 --> 00:01:11.930 align:middle line:90% and then pass it into a softmax. 00:01:11.930 --> 00:01:13.590 align:middle line:90% And boom, you're done. 00:01:13.590 --> 00:01:16.730 align:middle line:84% So that's how GPT-3 and all these models are trained. 00:01:16.730 --> 00:01:19.590 align:middle line:84% And the other thing I want to point out, 00:01:19.590 --> 00:01:20.970 align:middle line:84% which may not have been clear, is 00:01:20.970 --> 00:01:25.210 align:middle line:84% that what is coming out of this dense layer, 00:01:25.210 --> 00:01:29.810 align:middle line:84% this vector is as long as your vocabulary. 00:01:29.810 --> 00:01:32.645 align:middle line:84% Because only then, when it goes into the softmax, 00:01:32.645 --> 00:01:34.770 align:middle line:84% you're going to get probabilities which are as long 00:01:34.770 --> 00:01:38.050 align:middle line:84% as your vocabulary, which means that you get to pick one word 00:01:38.050 --> 00:01:43.722 align:middle line:84% or token out of that entire 50,000-long vocabulary. 00:01:43.722 --> 00:01:47.570 align:middle line:84% So I just want to point that out because I think it's easy for us 00:01:47.570 --> 00:01:50.250 align:middle line:84% to get a little confused because of this little difference 00:01:50.250 --> 00:01:54.530 align:middle line:84% between the way masked language models like BERT work 00:01:54.530 --> 00:01:58.630 align:middle line:84% and causal language models like GPT-3. 00:01:58.630 --> 00:02:02.350 align:middle line:90% So now let's continue with-- 00:02:02.350 --> 00:02:04.550 align:middle line:90% we know how to build GPT-3. 00:02:04.550 --> 00:02:07.150 align:middle line:90% So what about ChatGPT and GPT-2? 00:02:07.150 --> 00:02:08.310 align:middle line:90% What's up with them? 00:02:08.310 --> 00:02:10.830 align:middle line:84% Why is GPT-3 so famous and not GPT-2? 00:02:10.830 --> 00:02:12.933 align:middle line:90% 00:02:12.933 --> 00:02:14.350 align:middle line:84% Well, first of all, you folks know 00:02:14.350 --> 00:02:16.870 align:middle line:84% the GPT stands for generative pretrained transformer. 00:02:16.870 --> 00:02:22.230 align:middle line:84% Now, like GPT-3, GPT-2 and GPT-1 were trained in basically 00:02:22.230 --> 00:02:23.210 align:middle line:90% the same fashion-- 00:02:23.210 --> 00:02:26.590 align:middle line:84% predict the next word, same fashion, the same transformer 00:02:26.590 --> 00:02:31.310 align:middle line:84% stack, except that GPT-3 was trained on much more data 00:02:31.310 --> 00:02:33.950 align:middle line:84% because the underlying transformer stack had many more 00:02:33.950 --> 00:02:35.850 align:middle line:90% layers. 00:02:35.850 --> 00:02:40.410 align:middle line:84% So it is a much bigger stack, meaning lots more parameters, 00:02:40.410 --> 00:02:43.470 align:middle line:84% and therefore you need lots more data to train it well. 00:02:43.470 --> 00:02:44.230 align:middle line:90% OK? 00:02:44.230 --> 00:02:46.910 align:middle line:84% So that was really the only difference. 00:02:46.910 --> 00:02:50.870 align:middle line:84% The difference was literally one of scale, scale of network 00:02:50.870 --> 00:02:52.390 align:middle line:90% and scale of data. 00:02:52.390 --> 00:02:57.087 align:middle line:84% And unlike GPT and GPT-2, GPT-3, even 00:02:57.087 --> 00:02:58.920 align:middle line:84% though it was trained basically the same way 00:02:58.920 --> 00:03:01.280 align:middle line:84% with the same kind of network, it 00:03:01.280 --> 00:03:05.330 align:middle line:84% was one of those situations where more became different. 00:03:05.330 --> 00:03:07.080 align:middle line:84% There was almost like some of phase change 00:03:07.080 --> 00:03:09.320 align:middle line:90% that happened between 2 and 3. 00:03:09.320 --> 00:03:11.800 align:middle line:84% Unlike GPT and GPT-2, GPT-3 could 00:03:11.800 --> 00:03:14.800 align:middle line:84% do amazingly coherent continuations 00:03:14.800 --> 00:03:17.920 align:middle line:90% of any starting prompt. 00:03:17.920 --> 00:03:20.218 align:middle line:84% So for example, if you have this little prompt, 00:03:20.218 --> 00:03:22.760 align:middle line:84% which says "The importance of being on Twitter," by Jerome K. 00:03:22.760 --> 00:03:24.600 align:middle line:84% Jerome, who was a famous humorist, 00:03:24.600 --> 00:03:27.720 align:middle line:84% and then you give it this prompt ending with the word "it," 00:03:27.720 --> 00:03:31.800 align:middle line:84% it produces this continuation, which is really strikingly good. 00:03:31.800 --> 00:03:34.228 align:middle line:84% And if any of you have read Jerome K. Jerome 00:03:34.228 --> 00:03:36.020 align:middle line:84% and if you read this thing, you'll be like, 00:03:36.020 --> 00:03:38.880 align:middle line:84% wow, that actually sounds like Jerome K. Jerome. 00:03:38.880 --> 00:03:41.040 align:middle line:90% So, amazing continuations. 00:03:41.040 --> 00:03:44.260 align:middle line:84% But the interesting thing here is not so much the continuation, 00:03:44.260 --> 00:03:48.220 align:middle line:84% it's the fact that the same prompt, you give it to 2 or GPT, 00:03:48.220 --> 00:03:49.400 align:middle line:90% it won't be very good. 00:03:49.400 --> 00:03:51.860 align:middle line:84% In fact, after the first one, two, or three sentences, 00:03:51.860 --> 00:03:55.210 align:middle line:84% it'll become incoherent and meander and start rambling. 00:03:55.210 --> 00:03:58.970 align:middle line:84% This thing can keep faking it for a longer time. 00:03:58.970 --> 00:04:00.230 align:middle line:90% That's the amazing thing. 00:04:00.230 --> 00:04:02.130 align:middle line:90% That was unexpected. 00:04:02.130 --> 00:04:05.650 align:middle line:90% Researchers did not expect this. 00:04:05.650 --> 00:04:09.010 align:middle line:84% But it wasn't good at following your instructions. 00:04:09.010 --> 00:04:10.610 align:middle line:84% So for instance, if you ask it, help 00:04:10.610 --> 00:04:13.370 align:middle line:84% me write a short note to introduce myself to my neighbor. 00:04:13.370 --> 00:04:14.790 align:middle line:84% This is the kind of thing it will come up with. 00:04:14.790 --> 00:04:16.332 align:middle line:84% And you can actually run it yourself. 00:04:16.332 --> 00:04:18.709 align:middle line:84% You can actually go to GPT-3 on the Playground. 00:04:18.709 --> 00:04:21.690 align:middle line:84% I think GPT-3 is still available in the Playground. 00:04:21.690 --> 00:04:24.870 align:middle line:84% If it is, you can actually start running these prompts. 00:04:24.870 --> 00:04:27.890 align:middle line:84% You will start getting garbage very quickly. 00:04:27.890 --> 00:04:29.542 align:middle line:84% And the reason, So for example here, 00:04:29.542 --> 00:04:30.750 align:middle line:90% "help me write a short note." 00:04:30.750 --> 00:04:33.050 align:middle line:84% It says, "what is a good introduction to a resume." 00:04:33.050 --> 00:04:35.750 align:middle line:84% Resume, for some reason it's glommed on to resume. 00:04:35.750 --> 00:04:37.570 align:middle line:90% I have no idea why. 00:04:37.570 --> 00:04:39.610 align:middle line:84% But the reason it's doing stuff like this 00:04:39.610 --> 00:04:42.930 align:middle line:84% is because a lot of the training data it was trained on 00:04:42.930 --> 00:04:46.050 align:middle line:84% are basically lots of lists of things. 00:04:46.050 --> 00:04:54.442 align:middle line:84% So when you say, for example, "the capital of France 00:04:54.442 --> 00:04:56.650 align:middle line:84% continued," it'll say the capital of France is Paris, 00:04:56.650 --> 00:04:59.370 align:middle line:84% the capital of Hungary is Budapest, and so on. 00:04:59.370 --> 00:05:01.150 align:middle line:84% It just start came up with the list. 00:05:01.150 --> 00:05:03.390 align:middle line:90% So it's very list-driven. 00:05:03.390 --> 00:05:05.970 align:middle line:84% It thinks that you need to complete some sort of list. 00:05:05.970 --> 00:05:07.550 align:middle line:90% That's what's going on here. 00:05:07.550 --> 00:05:08.970 align:middle line:90% And so it's not very good. 00:05:08.970 --> 00:05:11.012 align:middle line:84% So it doesn't realize that you're actually asking 00:05:11.012 --> 00:05:12.950 align:middle line:90% it to do something specific. 00:05:12.950 --> 00:05:16.070 align:middle line:84% So this is the problem when you have an autocomplete thing which 00:05:16.070 --> 00:05:19.030 align:middle line:84% doesn't realize what you're asking it. 00:05:19.030 --> 00:05:22.310 align:middle line:90% It's an autocomplete. 00:05:22.310 --> 00:05:24.387 align:middle line:84% Now, in addition to these unhelpful answers, 00:05:24.387 --> 00:05:25.970 align:middle line:84% it can also produce offensive answers, 00:05:25.970 --> 00:05:28.137 align:middle line:84% factually incorrect answers, and so on and so forth. 00:05:28.137 --> 00:05:31.390 align:middle line:84% The list of bad things it can do is long. 00:05:31.390 --> 00:05:32.370 align:middle line:90% So why does it do that? 00:05:32.370 --> 00:05:34.010 align:middle line:84% Why does it produce unhelpful answers? 00:05:34.010 --> 00:05:36.448 align:middle line:84% Well, as you recall, it was only trained 00:05:36.448 --> 00:05:37.490 align:middle line:90% to predict the next word. 00:05:37.490 --> 00:05:40.770 align:middle line:84% It wasn't explicitly trained to follow instructions. 00:05:40.770 --> 00:05:44.950 align:middle line:84% So it seems reasonable that if it's simply 00:05:44.950 --> 00:05:47.430 align:middle line:84% trying to guess the next word repeatedly, 00:05:47.430 --> 00:05:49.558 align:middle line:84% it can't really do anything more. 00:05:49.558 --> 00:05:51.600 align:middle line:84% How can it figure out that there's an instruction 00:05:51.600 --> 00:05:52.720 align:middle line:90% that it needs to follow? 00:05:52.720 --> 00:05:55.240 align:middle line:84% Unless the training data on the internet 00:05:55.240 --> 00:05:59.800 align:middle line:84% was all instructional, which it clearly is not. 00:05:59.800 --> 00:06:02.440 align:middle line:90% So, light bulb idea-- 00:06:02.440 --> 00:06:06.660 align:middle line:84% let's explicitly train it with instruction data. 00:06:06.660 --> 00:06:08.720 align:middle line:84% Let's just train it with instruction data. 00:06:08.720 --> 00:06:12.040 align:middle line:84% And so OpenAI developed an approach called instruction 00:06:12.040 --> 00:06:14.720 align:middle line:90% tuning to do exactly this. 00:06:14.720 --> 00:06:19.800 align:middle line:84% And this paper is the paper that was the breakthrough. 00:06:19.800 --> 00:06:23.480 align:middle line:84% This is what actually put ChatGPT on the map. 00:06:23.480 --> 00:06:25.020 align:middle line:90% And it's very readable. 00:06:25.020 --> 00:06:28.760 align:middle line:84% So I would encourage you to check it out, if you're curious. 00:06:28.760 --> 00:06:34.440 align:middle line:84% And so we had GPT, GPT-2, GPT-3, just bigger and bigger models 00:06:34.440 --> 00:06:35.480 align:middle line:90% trained the same way. 00:06:35.480 --> 00:06:37.147 align:middle line:84% And then we run into the problem that it 00:06:37.147 --> 00:06:38.260 align:middle line:90% can't handle instructions. 00:06:38.260 --> 00:06:41.240 align:middle line:84% So we do instruction tuning to get to 3.5, also 00:06:41.240 --> 00:06:42.560 align:middle line:90% called InstructGPT. 00:06:42.560 --> 00:06:45.440 align:middle line:84% And then a small tweak after that gets you ChatGPT. 00:06:45.440 --> 00:06:49.260 align:middle line:84% And by the way, this step here, there are really two things 00:06:49.260 --> 00:06:51.240 align:middle line:84% going on in this, as you will soon see, 00:06:51.240 --> 00:06:52.860 align:middle line:84% I'm just calling it instruction tuning 00:06:52.860 --> 00:06:55.100 align:middle line:84% just so that I don't have to say some long thing 00:06:55.100 --> 00:06:56.700 align:middle line:90% every single time. 00:06:56.700 --> 00:06:59.280 align:middle line:84% This is not a consistent piece of terminology, 00:06:59.280 --> 00:07:03.380 align:middle line:84% so just be aware of it, that's all. 00:07:03.380 --> 00:07:06.580 align:middle line:84% All right, first step, they got a bunch of people 00:07:06.580 --> 00:07:09.420 align:middle line:84% to write high-quality answers to questions. 00:07:09.420 --> 00:07:12.960 align:middle line:84% And they created about 12,500 such question-answer pairs. 00:07:12.960 --> 00:07:15.013 align:middle line:84% So for example, let's say this was the question-- 00:07:15.013 --> 00:07:17.180 align:middle line:84% "Explain the moon landing to a six-year-old in a few 00:07:17.180 --> 00:07:18.180 align:middle line:90% sentences." 00:07:18.180 --> 00:07:21.060 align:middle line:84% Believe it or not, GPT-3's answer to that question 00:07:21.060 --> 00:07:23.223 align:middle line:90% was another question. 00:07:23.223 --> 00:07:25.140 align:middle line:84% Because it thinks there is a list of questions 00:07:25.140 --> 00:07:26.936 align:middle line:90% it needs to autocomplete. 00:07:26.936 --> 00:07:29.353 align:middle line:84% So it comes up with, "Explain the theory of gravity to a 6 00:07:29.353 --> 00:07:29.940 align:middle line:90% year old." 00:07:29.940 --> 00:07:30.860 align:middle line:84% It's like one of those people who, 00:07:30.860 --> 00:07:33.480 align:middle line:84% when you ask them a question, they ask you a question back. 00:07:33.480 --> 00:07:36.260 align:middle line:84% So what they did is they said, OK, let's create a nice answer 00:07:36.260 --> 00:07:37.160 align:middle line:90% to this question. 00:07:37.160 --> 00:07:38.705 align:middle line:84% And here's a human-created answer-- 00:07:38.705 --> 00:07:41.080 align:middle line:84% "People went to the moon in a big rocket, walked around," 00:07:41.080 --> 00:07:42.600 align:middle line:90% blah, blah, blah. 00:07:42.600 --> 00:07:44.780 align:middle line:84% Much better answer to that question. 00:07:44.780 --> 00:07:48.710 align:middle line:84% And so once you create these 12,500 question-answer pairs 00:07:48.710 --> 00:07:53.510 align:middle line:84% as training data, we just trained GPT-3 some more using 00:07:53.510 --> 00:07:57.890 align:middle line:84% next word prediction as before, no difference. 00:07:57.890 --> 00:07:58.807 align:middle line:90% So here is the input-- 00:07:58.807 --> 00:08:00.807 align:middle line:84% "Explain the moon landing," blah blah blah blah. 00:08:00.807 --> 00:08:01.870 align:middle line:90% This is the question. 00:08:01.870 --> 00:08:04.150 align:middle line:84% And then we have the answer right there. 00:08:04.150 --> 00:08:08.030 align:middle line:84% And then we take that answer, move it to the right, 00:08:08.030 --> 00:08:13.510 align:middle line:84% and just shift it up so that when it finishes "sentences," 00:08:13.510 --> 00:08:15.413 align:middle line:90% it needs to predict "people." 00:08:15.413 --> 00:08:17.830 align:middle line:84% And then you give it "people," it needs to predict "went," 00:08:17.830 --> 00:08:20.750 align:middle line:84% and so on and so forth, just like we saw before. 00:08:20.750 --> 00:08:24.688 align:middle line:84% "The cat sat on the mat" became "the cat sat on the," "cat sat 00:08:24.688 --> 00:08:25.730 align:middle line:90% on the mat" on the right. 00:08:25.730 --> 00:08:26.890 align:middle line:90% Shifted, right? 00:08:26.890 --> 00:08:30.110 align:middle line:84% That's what makes prediction possible and necessary. 00:08:30.110 --> 00:08:31.230 align:middle line:90% So that's what they did. 00:08:31.230 --> 00:08:35.350 align:middle line:84% This is step one, same as before. 00:08:35.350 --> 00:08:37.590 align:middle line:84% And once you do that, it turns out-- 00:08:37.590 --> 00:08:39.650 align:middle line:84% this step is called supervised fine tuning-- 00:08:39.650 --> 00:08:40.789 align:middle line:90% it really helped. 00:08:40.789 --> 00:08:44.150 align:middle line:84% GPT-3, once you supervise fine tuned it, was much, much better 00:08:44.150 --> 00:08:45.540 align:middle line:90% at following instructions. 00:08:45.540 --> 00:08:47.920 align:middle line:84% But there's a small problem with this approach. 00:08:47.920 --> 00:08:49.720 align:middle line:84% It takes a lot of money and effort 00:08:49.720 --> 00:08:51.800 align:middle line:84% to have humans write high-quality answers 00:08:51.800 --> 00:08:54.400 align:middle line:90% to thousands of questions. 00:08:54.400 --> 00:08:55.800 align:middle line:90% It takes a lot of money. 00:08:55.800 --> 00:08:58.240 align:middle line:84% So the question is, what can we do? 00:08:58.240 --> 00:09:01.470 align:middle line:84% What is easier than writing a good answer to a question? 00:09:01.470 --> 00:09:04.280 align:middle line:90% 00:09:04.280 --> 00:09:07.280 align:middle line:90% OK, all right. 00:09:07.280 --> 00:09:08.780 align:middle line:84% How about somebody from this side? 00:09:08.780 --> 00:09:11.360 align:middle line:90% 00:09:11.360 --> 00:09:13.560 align:middle line:90% Yeah? 00:09:13.560 --> 00:09:15.685 align:middle line:84% AUDIENCE: Perhaps writing a question for an answer. 00:09:15.685 --> 00:09:17.768 align:middle line:84% RAMA RAMAKRISHNAN: Oh, that's actually a good one. 00:09:17.768 --> 00:09:18.960 align:middle line:90% Yeah, yeah, I like that. 00:09:18.960 --> 00:09:21.480 align:middle line:84% So, given an answer, find a question. 00:09:21.480 --> 00:09:24.020 align:middle line:84% And while that is not what I'm going to talk about here, 00:09:24.020 --> 00:09:27.200 align:middle line:84% that technique is actually used very heavily in LLMs. 00:09:27.200 --> 00:09:29.400 align:middle line:84% And so that's great, very Creative. 00:09:29.400 --> 00:09:31.165 align:middle line:90% Mark? 00:09:31.165 --> 00:09:32.540 align:middle line:84% AUDIENCE: Thumbs up, thumbs down. 00:09:32.540 --> 00:09:33.120 align:middle line:90% RAMA RAMAKRISHNAN: Sorry? 00:09:33.120 --> 00:09:34.540 align:middle line:84% AUDIENCE: Thumbs up or thumbs down. 00:09:34.540 --> 00:09:36.748 align:middle line:84% RAMA RAMAKRISHNAN: Thumbs up or thumbs down, exactly. 00:09:36.748 --> 00:09:39.520 align:middle line:84% Because all of us, everyone loves to be a critic. 00:09:39.520 --> 00:09:43.160 align:middle line:84% It's much easier to be a critic than to be a creator. 00:09:43.160 --> 00:09:44.620 align:middle line:90% So what do we do? 00:09:44.620 --> 00:09:46.980 align:middle line:84% We basically say, let's rank answers 00:09:46.980 --> 00:09:50.420 align:middle line:84% written by somebody else, which begs the question, who's 00:09:50.420 --> 00:09:52.388 align:middle line:90% going to write those answers? 00:09:52.388 --> 00:09:54.180 align:middle line:84% And that's where there's a brilliant answer 00:09:54.180 --> 00:09:55.940 align:middle line:90% to that question, which is? 00:09:55.940 --> 00:09:57.112 align:middle line:90% AUDIENCE: Wikipedia. 00:09:57.112 --> 00:09:58.320 align:middle line:90% RAMA RAMAKRISHNAN: Wikipedia. 00:09:58.320 --> 00:09:59.220 align:middle line:90% AUDIENCE: Reddit. 00:09:59.220 --> 00:10:00.830 align:middle line:90% RAMA RAMAKRISHNAN: Reddit. 00:10:00.830 --> 00:10:04.020 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:10:04.020 --> 00:10:08.100 align:middle line:84% RAMA RAMAKRISHNAN: We will just ask GPT-3 to write the answers. 00:10:08.100 --> 00:10:12.860 align:middle line:84% It might be crap, but we don't care because we can rank them. 00:10:12.860 --> 00:10:17.173 align:middle line:84% So we ask GPT-3 to generate several answers to the question. 00:10:17.173 --> 00:10:18.840 align:middle line:84% And how can we generate several answers? 00:10:18.840 --> 00:10:21.180 align:middle line:90% Because we can do sampling. 00:10:21.180 --> 00:10:23.100 align:middle line:90% We can do sampling. 00:10:23.100 --> 00:10:25.780 align:middle line:84% The fact that we had these stochastic outputs because 00:10:25.780 --> 00:10:29.480 align:middle line:84% of sampling is now a feature, not a bug. 00:10:29.480 --> 00:10:32.120 align:middle line:84% We create lots of different answers to the question. 00:10:32.120 --> 00:10:35.022 align:middle line:84% We feed it a question, get three answers out. 00:10:35.022 --> 00:10:36.980 align:middle line:84% Just run it three times, get three answers out, 00:10:36.980 --> 00:10:39.060 align:middle line:84% with a nice temperature of 1 or 1.1 00:10:39.060 --> 00:10:42.270 align:middle line:84% or something so that it's nice and random. 00:10:42.270 --> 00:10:44.110 align:middle line:84% And then we literally have humans 00:10:44.110 --> 00:10:46.310 align:middle line:84% just rank them, do the thumbs up, thumbs down, just 00:10:46.310 --> 00:10:48.990 align:middle line:84% rank them from most useful to least useful. 00:10:48.990 --> 00:10:53.610 align:middle line:84% So this step is a Step 2 of instruction tuning. 00:10:53.610 --> 00:10:57.290 align:middle line:84% So OpenAI collected 33,000 instructions, fed them to GPT-3, 00:10:57.290 --> 00:11:01.990 align:middle line:84% generated answers, and had humans rank them. 00:11:01.990 --> 00:11:04.830 align:middle line:84% Once you do this, you can assemble a beautiful training 00:11:04.830 --> 00:11:06.060 align:middle line:90% dataset. 00:11:06.060 --> 00:11:09.050 align:middle line:84% And so basically, what we have is that we have an instruction. 00:11:09.050 --> 00:11:11.150 align:middle line:84% And let's say we have just two answers, A and B. 00:11:11.150 --> 00:11:13.550 align:middle line:84% And in practice, you can have many, many answers, 00:11:13.550 --> 00:11:14.490 align:middle line:90% which we rank. 00:11:14.490 --> 00:11:17.330 align:middle line:84% But just for simplicity, I'll go with Mark's thumbs up, 00:11:17.330 --> 00:11:19.190 align:middle line:84% thumbs down answer, which is let's 00:11:19.190 --> 00:11:21.630 align:middle line:84% assume we only have two answers to every question. 00:11:21.630 --> 00:11:24.570 align:middle line:84% And the human has said, I prefer this to that. 00:11:24.570 --> 00:11:25.440 align:middle line:90% That's it. 00:11:25.440 --> 00:11:29.790 align:middle line:84% So we have a dataset now where the data point is instruction, 00:11:29.790 --> 00:11:33.310 align:middle line:84% preferred answer is A, the other answer is B. Yeah? 00:11:33.310 --> 00:11:36.350 align:middle line:90% 00:11:36.350 --> 00:11:38.247 align:middle line:84% AUDIENCE: The thumbs up, thumbs down 00:11:38.247 --> 00:11:40.080 align:middle line:84% technique that we were talking about before, 00:11:40.080 --> 00:11:41.900 align:middle line:84% is that why when we're using ChatGPT now, 00:11:41.900 --> 00:11:43.400 align:middle line:84% we also used thumbs up, thumbs down? 00:11:43.400 --> 00:11:45.320 align:middle line:84% Because it's using all the answers to train? 00:11:45.320 --> 00:11:46.260 align:middle line:84% RAMA RAMAKRISHNAN: Exactly right. 00:11:46.260 --> 00:11:46.920 align:middle line:90% AUDIENCE: OK. 00:11:46.920 --> 00:11:48.580 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, all the models have the thumbs up, 00:11:48.580 --> 00:11:50.122 align:middle line:84% thumbs down stuff going on somewhere. 00:11:50.122 --> 00:11:52.472 align:middle line:84% They are all collecting data for this step. 00:11:52.472 --> 00:11:53.180 align:middle line:90% AUDIENCE: Got it. 00:11:53.180 --> 00:11:53.580 align:middle line:90% Thank you. 00:11:53.580 --> 00:11:54.080 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:11:54.080 --> 00:11:55.840 align:middle line:84% It's sort of the old adage, right? 00:11:55.840 --> 00:11:58.360 align:middle line:84% If you're not sure who the product is, you're the product. 00:11:58.360 --> 00:11:59.620 align:middle line:90% So it's one of those things. 00:11:59.620 --> 00:12:00.120 align:middle line:90% Yeah? 00:12:00.120 --> 00:12:07.440 align:middle line:90% 00:12:07.440 --> 00:12:10.120 align:middle line:84% AUDIENCE: So if we understand correctly, when we see thumbs 00:12:10.120 --> 00:12:14.480 align:middle line:84% up, thumbs down, it does mean that ChatGPT is going 00:12:14.480 --> 00:12:16.560 align:middle line:90% to trade on our data, right? 00:12:16.560 --> 00:12:18.720 align:middle line:84% RAMA RAMAKRISHNAN: Unless you opt out, yeah. 00:12:18.720 --> 00:12:20.580 align:middle line:84% So if you actually go to ChatGPT controls, 00:12:20.580 --> 00:12:22.788 align:middle line:84% there is something called data controls or something. 00:12:22.788 --> 00:12:24.440 align:middle line:90% You can toggle it off. 00:12:24.440 --> 00:12:27.120 align:middle line:84% But I think when I last checked, if you toggle it off, 00:12:27.120 --> 00:12:29.425 align:middle line:90% you lose your chat history. 00:12:29.425 --> 00:12:30.800 align:middle line:84% So they have hobbled that feature 00:12:30.800 --> 00:12:34.760 align:middle line:84% to prevent people from setting it to off as much as possible. 00:12:34.760 --> 00:12:37.480 align:middle line:90% Yeah, clever. 00:12:37.480 --> 00:12:38.640 align:middle line:90% But you can opt out. 00:12:38.640 --> 00:12:41.120 align:middle line:84% And if you use the API, as opposed to the web interface, 00:12:41.120 --> 00:12:43.060 align:middle line:90% you're automatically opted out. 00:12:43.060 --> 00:12:44.673 align:middle line:84% So you have to deliberately opt in. 00:12:44.673 --> 00:12:47.340 align:middle line:84% And if you use the versions that are available through Microsoft 00:12:47.340 --> 00:12:48.900 align:middle line:84% Azure and so on and so forth, there 00:12:48.900 --> 00:12:51.240 align:middle line:84% are all kinds of very safe controls and stuff like that. 00:12:51.240 --> 00:12:55.860 align:middle line:84% In fact, I think the Microsoft Copilot license that MIT has, 00:12:55.860 --> 00:12:58.780 align:middle line:84% I think the default is opted out. 00:12:58.780 --> 00:13:01.880 align:middle line:84% So to go here, once you have this data point, 00:13:01.880 --> 00:13:04.620 align:middle line:84% you can build something called a reward model. 00:13:04.620 --> 00:13:07.340 align:middle line:84% And this is a very clever piece of work. 00:13:07.340 --> 00:13:08.980 align:middle line:90% So what you do is-- 00:13:08.980 --> 00:13:11.900 align:middle line:84% you have an instruction, you have a preferred answer, 00:13:11.900 --> 00:13:13.540 align:middle line:90% and you have the other answer-- 00:13:13.540 --> 00:13:17.198 align:middle line:90% you feed it to a network. 00:13:17.198 --> 00:13:18.240 align:middle line:90% You feed it to a network. 00:13:18.240 --> 00:13:20.780 align:middle line:84% This is just a nice language model. 00:13:20.780 --> 00:13:22.700 align:middle line:90% It's just a language model. 00:13:22.700 --> 00:13:25.880 align:middle line:84% And the language model produces a number, 00:13:25.880 --> 00:13:28.260 align:middle line:84% which measures how good this thing is. 00:13:28.260 --> 00:13:31.900 align:middle line:84% How good an answer is this to that particular instruction? 00:13:31.900 --> 00:13:34.240 align:middle line:90% So you get a rating here. 00:13:34.240 --> 00:13:35.620 align:middle line:90% You get a rating here. 00:13:35.620 --> 00:13:38.630 align:middle line:84% And then what you do is you run it 00:13:38.630 --> 00:13:44.350 align:middle line:84% through a little loss function, which essentially encourages 00:13:44.350 --> 00:13:49.990 align:middle line:84% the model to give higher numbers to the better answer. 00:13:49.990 --> 00:13:51.010 align:middle line:90% It's the same model. 00:13:51.010 --> 00:13:53.570 align:middle line:84% You just run the question on the first answer, question 00:13:53.570 --> 00:13:55.810 align:middle line:84% on the second answer, you get these two numbers. 00:13:55.810 --> 00:13:58.510 align:middle line:84% And then initially those numbers are just random. 00:13:58.510 --> 00:14:00.010 align:middle line:84% But then you tell the model, hey, 00:14:00.010 --> 00:14:01.177 align:middle line:90% this is the preferred thing. 00:14:01.177 --> 00:14:05.710 align:middle line:84% Make sure the preferred answer's rating, the R value, 00:14:05.710 --> 00:14:07.470 align:middle line:90% is higher than the other number. 00:14:07.470 --> 00:14:09.700 align:middle line:84% Because more is better, higher is better. 00:14:09.700 --> 00:14:13.190 align:middle line:90% 00:14:13.190 --> 00:14:15.120 align:middle line:84% And this thing is just the sigmoid here. 00:14:15.120 --> 00:14:17.370 align:middle line:84% You basically take the difference of these two things. 00:14:17.370 --> 00:14:18.870 align:middle line:84% Do a sigmoid, take the logarithm. 00:14:18.870 --> 00:14:21.690 align:middle line:84% And you can actually convince yourself afterwards-- 00:14:21.690 --> 00:14:23.310 align:middle line:90% and I encourage you to do that-- 00:14:23.310 --> 00:14:28.350 align:middle line:84% to check for yourself that if we actually 00:14:28.350 --> 00:14:31.530 align:middle line:84% give a higher number to the better answer, 00:14:31.530 --> 00:14:33.790 align:middle line:90% the loss will be lower. 00:14:33.790 --> 00:14:35.580 align:middle line:84% And since we are minimizing loss, 00:14:35.580 --> 00:14:38.360 align:middle line:84% we are essentially training the network 00:14:38.360 --> 00:14:41.680 align:middle line:84% to try to give higher ratings to better answers. 00:14:41.680 --> 00:14:42.380 align:middle line:90% That's it. 00:14:42.380 --> 00:14:43.680 align:middle line:90% So that's the approach. 00:14:43.680 --> 00:14:44.520 align:middle line:90% Did you have a-- 00:14:44.520 --> 00:14:46.660 align:middle line:90% yeah, Ben. 00:14:46.660 --> 00:14:50.520 align:middle line:84% AUDIENCE: So you could imagine training the model on only 00:14:50.520 --> 00:14:52.280 align:middle line:84% the good answers, is the idea of having 00:14:52.280 --> 00:14:54.660 align:middle line:84% both that the model's actually learning what makes good-- 00:14:54.660 --> 00:14:55.980 align:middle line:84% RAMA RAMAKRISHNAN: Correct, exactly. 00:14:55.980 --> 00:14:58.147 align:middle line:84% Much like if you want to build a dog-cat classifier, 00:14:58.147 --> 00:15:01.400 align:middle line:84% you have to show pictures of both. 00:15:01.400 --> 00:15:02.840 align:middle line:90% Yeah? 00:15:02.840 --> 00:15:05.880 align:middle line:84% AUDIENCE: So I understand the feedback mechanism of thumbs up, 00:15:05.880 --> 00:15:06.380 align:middle line:90% thumbs down. 00:15:06.380 --> 00:15:10.560 align:middle line:84% But there are a lot of times when the popular response is not 00:15:10.560 --> 00:15:11.680 align:middle line:90% the accurate one. 00:15:11.680 --> 00:15:14.440 align:middle line:84% So is there a way that they actually 00:15:14.440 --> 00:15:15.562 align:middle line:90% have a layer to correct? 00:15:15.562 --> 00:15:17.520 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, good question, Swathi. 00:15:17.520 --> 00:15:23.400 align:middle line:84% So as it turns out, all these companies like OpenAI, 00:15:23.400 --> 00:15:27.400 align:middle line:84% they have a huge document, 100, 200 pages long, 00:15:27.400 --> 00:15:30.600 align:middle line:84% very, very bulky document, which instructs and teaches 00:15:30.600 --> 00:15:34.400 align:middle line:84% the labelers, the rankers how to rank these things. 00:15:34.400 --> 00:15:36.540 align:middle line:84% So they have to follow these very strict guidelines 00:15:36.540 --> 00:15:39.020 align:middle line:84% to precisely handle strange corner cases 00:15:39.020 --> 00:15:40.120 align:middle line:90% and things like that. 00:15:40.120 --> 00:15:42.660 align:middle line:90% And that document is on the web. 00:15:42.660 --> 00:15:44.010 align:middle line:90% You can dig it up. 00:15:44.010 --> 00:15:46.260 align:middle line:84% And it's actually very instructive to read through it. 00:15:46.260 --> 00:15:47.820 align:middle line:84% I think they put it out on the web 00:15:47.820 --> 00:15:48.900 align:middle line:84% because they wanted to convince people 00:15:48.900 --> 00:15:50.620 align:middle line:84% that they're going to inordinate trouble 00:15:50.620 --> 00:15:52.990 align:middle line:84% to make sure the rankings are actually good. 00:15:52.990 --> 00:15:54.240 align:middle line:90% Do you have a question, Colin? 00:15:54.240 --> 00:15:55.500 align:middle line:90% No. 00:15:55.500 --> 00:15:59.980 align:middle line:90% OK, all right, so back to this. 00:15:59.980 --> 00:16:01.580 align:middle line:90% And how do you train this thing? 00:16:01.580 --> 00:16:03.180 align:middle line:90% SGD. 00:16:03.180 --> 00:16:04.470 align:middle line:90% Because you have a network. 00:16:04.470 --> 00:16:05.720 align:middle line:90% It's coming up with an answer. 00:16:05.720 --> 00:16:08.640 align:middle line:84% You have some way to know if that answer is good or bad. 00:16:08.640 --> 00:16:10.260 align:middle line:90% Better answers have lower loss. 00:16:10.260 --> 00:16:12.260 align:middle line:84% Back propagation through the network, 00:16:12.260 --> 00:16:16.400 align:middle line:84% keep updating the weights, and, boom, you're done. 00:16:16.400 --> 00:16:18.980 align:middle line:84% And once you do that, this reward model 00:16:18.980 --> 00:16:23.407 align:middle line:84% can provide a numerical rating for any instruction-answer pair. 00:16:23.407 --> 00:16:25.740 align:middle line:84% You just give it an instruction, you give it an answer-- 00:16:25.740 --> 00:16:28.115 align:middle line:84% could be a crappy answer, good answer-- it just tells you 00:16:28.115 --> 00:16:32.210 align:middle line:84% how good it is, which means, in this case, for example, 00:16:32.210 --> 00:16:35.470 align:middle line:84% maybe it's going to give you a nice number, 1.5, 00:16:35.470 --> 00:16:38.490 align:middle line:90% which is 1.5 for this answer. 00:16:38.490 --> 00:16:42.590 align:middle line:84% But then a better answer comes along, 3.2. 00:16:42.590 --> 00:16:46.110 align:middle line:84% What we have done by doing this whole thing, this modeling, 00:16:46.110 --> 00:16:48.990 align:middle line:84% is that essentially we have learned 00:16:48.990 --> 00:16:51.750 align:middle line:90% how humans rank responses. 00:16:51.750 --> 00:16:53.630 align:middle line:84% Because we can only have humans rank 00:16:53.630 --> 00:16:56.590 align:middle line:84% responses for some finite number of questions. 00:16:56.590 --> 00:17:01.090 align:middle line:84% What we really want to do is to automate that ranking process 00:17:01.090 --> 00:17:03.070 align:middle line:84% so that we can just do it for tens of thousands 00:17:03.070 --> 00:17:04.990 align:middle line:90% of questions really fast. 00:17:04.990 --> 00:17:06.710 align:middle line:84% So we have essentially built a model 00:17:06.710 --> 00:17:10.641 align:middle line:84% of how humans rank things, which is beautiful. 00:17:10.641 --> 00:17:12.849 align:middle line:84% A lot of the stuff here is all very self-referential, 00:17:12.849 --> 00:17:13.932 align:middle line:90% which I find very elegant. 00:17:13.932 --> 00:17:17.150 align:middle line:84% Anyway, so this can be used to improve GPT-3 even further. 00:17:17.150 --> 00:17:20.530 align:middle line:84% So we take the instruction as before, we feed it, 00:17:20.530 --> 00:17:21.990 align:middle line:90% it gives you some answer. 00:17:21.990 --> 00:17:25.349 align:middle line:84% And then we feed this instruction and the answer 00:17:25.349 --> 00:17:27.630 align:middle line:84% to our newly-minted reward model. 00:17:27.630 --> 00:17:29.770 align:middle line:90% It gives us a numerical rating. 00:17:29.770 --> 00:17:31.550 align:middle line:90% And then this is the key step. 00:17:31.550 --> 00:17:33.030 align:middle line:90% We take this numerical rating. 00:17:33.030 --> 00:17:35.530 align:middle line:84% And then we use this rating to nudge 00:17:35.530 --> 00:17:39.530 align:middle line:84% the internal weights of GPT-3 in the right direction. 00:17:39.530 --> 00:17:44.930 align:middle line:84% This nudging uses a technique called reinforcement learning, 00:17:44.930 --> 00:17:46.510 align:middle line:84% which, just in the interest of time, 00:17:46.510 --> 00:17:49.010 align:middle line:84% we can't get into in this lecture. 00:17:49.010 --> 00:17:52.250 align:middle line:84% But that's a technique you use to nudge these things 00:17:52.250 --> 00:17:54.450 align:middle line:90% in the right direction. 00:17:54.450 --> 00:17:56.220 align:middle line:90% So that's what we do. 00:17:56.220 --> 00:17:57.470 align:middle line:90% That's reinforcement learning. 00:17:57.470 --> 00:18:01.490 align:middle line:84% We nudge it in the right direction. 00:18:01.490 --> 00:18:07.850 align:middle line:84% And OpenAI did this with 31,000 questions. 00:18:07.850 --> 00:18:09.970 align:middle line:84% Nudge, nudge, nudge, nudge, nudge, nudge. 00:18:09.970 --> 00:18:13.690 align:middle line:84% And when you do that, you get GPT-3.5/InstructGPT. 00:18:13.690 --> 00:18:15.530 align:middle line:90% OK? 00:18:15.530 --> 00:18:17.050 align:middle line:90% That's it. 00:18:17.050 --> 00:18:20.865 align:middle line:84% By the way, this step here is called reinforcement 00:18:20.865 --> 00:18:22.490 align:middle line:84% learning with human feedback because we 00:18:22.490 --> 00:18:23.710 align:middle line:90% use reinforcement learning. 00:18:23.710 --> 00:18:25.570 align:middle line:84% And since humans rank the answers which 00:18:25.570 --> 00:18:27.780 align:middle line:84% led to the building of the reward model, 00:18:27.780 --> 00:18:29.157 align:middle line:90% we get human feedback. 00:18:29.157 --> 00:18:31.240 align:middle line:84% That's reinforcement learning with human feedback. 00:18:31.240 --> 00:18:33.900 align:middle line:90% Yeah? 00:18:33.900 --> 00:18:37.180 align:middle line:84% AUDIENCE: I have a question regarding the type of questions 00:18:37.180 --> 00:18:38.420 align:middle line:90% that they are using. 00:18:38.420 --> 00:18:42.380 align:middle line:84% I can imagine maybe they are very simple questions to answer. 00:18:42.380 --> 00:18:46.340 align:middle line:84% Because I'm thinking now you can ask GPT, for example, 00:18:46.340 --> 00:18:49.740 align:middle line:84% respond to this as a pirate or something like that. 00:18:49.740 --> 00:18:53.460 align:middle line:84% It's going to be harder to train if you have a bunch of questions 00:18:53.460 --> 00:18:55.640 align:middle line:84% that are having small instructions 00:18:55.640 --> 00:18:56.932 align:middle line:90% and then there is the question. 00:18:56.932 --> 00:18:58.682 align:middle line:84% RAMA RAMAKRISHNAN: That's a good question. 00:18:58.682 --> 00:19:00.820 align:middle line:84% So the quality of the questions in the dataset 00:19:00.820 --> 00:19:02.480 align:middle line:90% clearly is a big factor. 00:19:02.480 --> 00:19:05.300 align:middle line:84% Because if you have simplistic questions, 00:19:05.300 --> 00:19:08.380 align:middle line:84% it won't be able to handle complex questions later on. 00:19:08.380 --> 00:19:11.160 align:middle line:90% So it's a good question. 00:19:11.160 --> 00:19:13.300 align:middle line:84% So that actually begs the question 00:19:13.300 --> 00:19:16.860 align:middle line:84% of where did they get these questions from? 00:19:16.860 --> 00:19:19.940 align:middle line:84% So they actually got it from their API. 00:19:19.940 --> 00:19:25.540 align:middle line:84% So people were asking GPT-3 on the API before it became 3.5. 00:19:25.540 --> 00:19:27.770 align:middle line:84% The API was already available, fully available, 00:19:27.770 --> 00:19:28.770 align:middle line:90% commercially available. 00:19:28.770 --> 00:19:31.350 align:middle line:84% A lot of people were building products on it already by then. 00:19:31.350 --> 00:19:33.230 align:middle line:84% And so they collected all those questions 00:19:33.230 --> 00:19:34.930 align:middle line:90% and filtered them for quality. 00:19:34.930 --> 00:19:36.970 align:middle line:84% And that was the question set that they used. 00:19:36.970 --> 00:19:38.710 align:middle line:84% And then they judiciously added to it 00:19:38.710 --> 00:19:40.493 align:middle line:90% with human-created questions. 00:19:40.493 --> 00:19:41.910 align:middle line:84% But they couldn't do a lot of that 00:19:41.910 --> 00:19:43.428 align:middle line:84% because it's expensive to do that. 00:19:43.428 --> 00:19:45.470 align:middle line:84% But collecting stuff that somebody else is asking 00:19:45.470 --> 00:19:48.990 align:middle line:90% your API already-- very easy. 00:19:48.990 --> 00:19:50.755 align:middle line:90% Yeah, Tomaso? 00:19:50.755 --> 00:19:53.130 align:middle line:84% AUDIENCE: This might be more of a philosophical question, 00:19:53.130 --> 00:19:56.950 align:middle line:84% but the human bias that's present in the small subset 00:19:56.950 --> 00:19:59.030 align:middle line:84% of human labelers that they've chosen 00:19:59.030 --> 00:20:01.310 align:middle line:84% gets eventually compounded in this model 00:20:01.310 --> 00:20:04.710 align:middle line:84% that we often consider as the source of objective truth. 00:20:04.710 --> 00:20:06.070 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:20:06.070 --> 00:20:07.270 align:middle line:90% Yeah, that's very true. 00:20:07.270 --> 00:20:11.230 align:middle line:84% I think the reward model has probably very faithfully learned 00:20:11.230 --> 00:20:13.630 align:middle line:84% all the biases of the human labelers, which 00:20:13.630 --> 00:20:17.470 align:middle line:84% is why they have these very complex frameworks 00:20:17.470 --> 00:20:20.330 align:middle line:84% and guidelines to try to prevent the bias from happening, 00:20:20.330 --> 00:20:21.250 align:middle line:90% to mitigate it. 00:20:21.250 --> 00:20:23.835 align:middle line:84% So for example, they might give the same question and set 00:20:23.835 --> 00:20:25.210 align:middle line:84% of instead of possible answers to 00:20:25.210 --> 00:20:27.010 align:middle line:90% many, many different labelers. 00:20:27.010 --> 00:20:31.790 align:middle line:84% And only if people pick the same ranking, they might use it. 00:20:31.790 --> 00:20:35.290 align:middle line:84% So that at least inter-labeller bias can be minimized. 00:20:35.290 --> 00:20:38.310 align:middle line:84% But if everybody is biased in the same direction, 00:20:38.310 --> 00:20:40.130 align:middle line:84% it won't protect you against that. 00:20:40.130 --> 00:20:41.930 align:middle line:84% So in general, there's a whole work 00:20:41.930 --> 00:20:44.050 align:middle line:84% that's being done to try to de-bias these things 00:20:44.050 --> 00:20:46.367 align:middle line:84% and build them without too much bias in them. 00:20:46.367 --> 00:20:48.450 align:middle line:84% It's like a whole world unto itself, which we just 00:20:48.450 --> 00:20:49.630 align:middle line:90% don't have time to get into. 00:20:49.630 --> 00:20:54.050 align:middle line:90% 00:20:54.050 --> 00:20:55.810 align:middle line:84% AUDIENCE: Depending on the medium that's 00:20:55.810 --> 00:20:57.602 align:middle line:84% being returned by these models, would there 00:20:57.602 --> 00:20:58.870 align:middle line:90% be more than one reward model? 00:20:58.870 --> 00:20:59.770 align:middle line:90% Because isn't that what Gemini-- 00:20:59.770 --> 00:21:01.810 align:middle line:84% RAMA RAMAKRISHNAN: Sorry, would there be more than one what? 00:21:01.810 --> 00:21:02.768 align:middle line:90% AUDIENCE: Reward model. 00:21:02.768 --> 00:21:05.050 align:middle line:84% Because isn't this what Gemini is running into issues 00:21:05.050 --> 00:21:07.350 align:middle line:84% with right now with their image generation, 00:21:07.350 --> 00:21:09.243 align:middle line:90% is the bias that they hide? 00:21:09.243 --> 00:21:11.910 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so the Gemini business that's going on, 00:21:11.910 --> 00:21:14.530 align:middle line:90% it's unclear what's causing it. 00:21:14.530 --> 00:21:18.090 align:middle line:84% It may be in this step, maybe they were a little overzealous 00:21:18.090 --> 00:21:20.290 align:middle line:84% in preventing certain things from happening. 00:21:20.290 --> 00:21:25.620 align:middle line:84% Some of these systems, they will actually intercept the question 00:21:25.620 --> 00:21:28.020 align:middle line:84% that you ask and then route it differently 00:21:28.020 --> 00:21:30.860 align:middle line:84% based on what they sense is sitting around in the question. 00:21:30.860 --> 00:21:32.280 align:middle line:84% So there could be pre-processing, 00:21:32.280 --> 00:21:34.155 align:middle line:84% post-processing, a lot of stuff that goes on. 00:21:34.155 --> 00:21:36.402 align:middle line:84% So unclear to me where in the pipeline. 00:21:36.402 --> 00:21:37.860 align:middle line:84% And it could be more than one place 00:21:37.860 --> 00:21:39.320 align:middle line:90% these things may be entering. 00:21:39.320 --> 00:21:42.300 align:middle line:84% So yes, here may very well be where it actually enters 00:21:42.300 --> 00:21:45.200 align:middle line:84% a situation where people are told, 00:21:45.200 --> 00:21:47.380 align:middle line:84% if you see any of this kind of answer, 00:21:47.380 --> 00:21:49.847 align:middle line:84% down rank it or don't up-rank it. 00:21:49.847 --> 00:21:52.180 align:middle line:84% And then it learns that ranking very faithfully and then 00:21:52.180 --> 00:21:54.580 align:middle line:84% proceeds to apply it where it should not be applied. 00:21:54.580 --> 00:21:56.060 align:middle line:90% So that does happen. 00:21:56.060 --> 00:21:57.320 align:middle line:90% Jocelyn, you had a question. 00:21:57.320 --> 00:21:59.980 align:middle line:90% 00:21:59.980 --> 00:22:01.860 align:middle line:84% AUDIENCE: I think I still don't totally 00:22:01.860 --> 00:22:05.020 align:middle line:84% understand why when I ask Chat-GPT a question, even 00:22:05.020 --> 00:22:07.420 align:middle line:84% in a lengthy response, it doesn't wander away 00:22:07.420 --> 00:22:09.160 align:middle line:84% from the topic that I'm asking about. 00:22:09.160 --> 00:22:12.400 align:middle line:84% And so understanding that it's predicting each word, 00:22:12.400 --> 00:22:14.540 align:middle line:84% it's taking a random walk from one word 00:22:14.540 --> 00:22:15.800 align:middle line:90% to the next in some sense. 00:22:15.800 --> 00:22:18.020 align:middle line:84% RAMA RAMAKRISHNAN: But each word it utters now 00:22:18.020 --> 00:22:20.563 align:middle line:84% becomes part of the input to the next word it utters. 00:22:20.563 --> 00:22:21.230 align:middle line:90% AUDIENCE: Right. 00:22:21.230 --> 00:22:21.990 align:middle line:84% RAMA RAMAKRISHNAN: So it's not truly 00:22:21.990 --> 00:22:23.490 align:middle line:90% a random walk in that sense. 00:22:23.490 --> 00:22:24.990 align:middle line:84% So the next step is not independent 00:22:24.990 --> 00:22:26.990 align:middle line:90% of the previous step. 00:22:26.990 --> 00:22:28.530 align:middle line:84% It depends on the journey so far. 00:22:28.530 --> 00:22:30.988 align:middle line:84% So it's going to try to be very consistent with the journey 00:22:30.988 --> 00:22:31.688 align:middle line:90% so far. 00:22:31.688 --> 00:22:32.230 align:middle line:90% AUDIENCE: OK. 00:22:32.230 --> 00:22:35.310 align:middle line:90% 00:22:35.310 --> 00:22:39.830 align:middle line:84% Does this part, fine-tuning it on these question 00:22:39.830 --> 00:22:41.710 align:middle line:84% and answer sets, does this play some role 00:22:41.710 --> 00:22:45.670 align:middle line:84% in it being able to constrain itself and not meander away? 00:22:45.670 --> 00:22:47.170 align:middle line:84% RAMA RAMAKRISHNAN: I don't think so. 00:22:47.170 --> 00:22:47.270 align:middle line:90% AUDIENCE: OK. 00:22:47.270 --> 00:22:49.437 align:middle line:84% RAMA RAMAKRISHNAN: I think this is more to make sure 00:22:49.437 --> 00:22:53.270 align:middle line:84% that the weights generally tend to produce the right answer. 00:22:53.270 --> 00:22:55.910 align:middle line:84% Now, one of the things that is possible 00:22:55.910 --> 00:22:58.590 align:middle line:84% is that let's say I'm a ranker and I'm 00:22:58.590 --> 00:23:01.350 align:middle line:84% looking at a few different answers, 00:23:01.350 --> 00:23:03.710 align:middle line:84% I have to figure out if the answer is helpful, 00:23:03.710 --> 00:23:09.190 align:middle line:84% if it is accurate, if it is non-toxic, things like that. 00:23:09.190 --> 00:23:12.310 align:middle line:84% And part of the rubric for evaluating these answers 00:23:12.310 --> 00:23:14.890 align:middle line:90% could be their coherence. 00:23:14.890 --> 00:23:17.670 align:middle line:84% So it could also be that they are saying, 00:23:17.670 --> 00:23:19.290 align:middle line:84% short, coherent answers are better 00:23:19.290 --> 00:23:21.250 align:middle line:90% than long, coherent answers. 00:23:21.250 --> 00:23:23.420 align:middle line:84% But once you adjust for length, maybe coherence 00:23:23.420 --> 00:23:24.170 align:middle line:90% is more important. 00:23:24.170 --> 00:23:25.470 align:middle line:84% There could be any number of these things, 00:23:25.470 --> 00:23:26.807 align:middle line:90% so it could play a role in that. 00:23:26.807 --> 00:23:28.390 align:middle line:84% AUDIENCE: So just one small follow-up. 00:23:28.390 --> 00:23:30.368 align:middle line:84% So in other words, when it's learning 00:23:30.368 --> 00:23:31.910 align:middle line:84% from these question-and-answer pairs, 00:23:31.910 --> 00:23:33.537 align:middle line:84% it's able to look at the whole response 00:23:33.537 --> 00:23:35.370 align:middle line:84% and learn something about the whole response 00:23:35.370 --> 00:23:37.350 align:middle line:84% rather than just one word at a time, right? 00:23:37.350 --> 00:23:38.475 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:23:38.475 --> 00:23:40.530 align:middle line:84% Yeah, the entire question is being ranked. 00:23:40.530 --> 00:23:42.610 align:middle line:90% Correct, correct. 00:23:42.610 --> 00:23:43.810 align:middle line:90% Yeah? 00:23:43.810 --> 00:23:46.010 align:middle line:84% AUDIENCE: On a related note, when 00:23:46.010 --> 00:23:48.230 align:middle line:84% it's generating a new word on topic, 00:23:48.230 --> 00:23:51.490 align:middle line:84% does the attention pertain to that entire prior text? 00:23:51.490 --> 00:23:54.290 align:middle line:84% Or can you have traveling attention? 00:23:54.290 --> 00:23:57.050 align:middle line:90% So last five word attention-- 00:23:57.050 --> 00:24:01.030 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, the short answer is, yeah, you can. 00:24:01.030 --> 00:24:02.890 align:middle line:84% It's called sliding-window attention. 00:24:02.890 --> 00:24:04.210 align:middle line:90% It can be done. 00:24:04.210 --> 00:24:07.010 align:middle line:84% They typically tend to do it not so much because they 00:24:07.010 --> 00:24:09.830 align:middle line:84% want to focus more on the recent words, 00:24:09.830 --> 00:24:13.530 align:middle line:84% but more because it actually makes it very compute efficient. 00:24:13.530 --> 00:24:14.630 align:middle line:90% That's why they do it. 00:24:14.630 --> 00:24:16.297 align:middle line:84% So it's called sliding-window attention. 00:24:16.297 --> 00:24:17.940 align:middle line:90% You can Google it. 00:24:17.940 --> 00:24:19.740 align:middle line:84% AUDIENCE: So normally it's full attention? 00:24:19.740 --> 00:24:23.313 align:middle line:84% RAMA RAMAKRISHNAN: Normally, default is full attention. 00:24:23.313 --> 00:24:26.820 align:middle line:90% So that's what they did. 00:24:26.820 --> 00:24:28.800 align:middle line:84% And by the way, as I think you pointed out, 00:24:28.800 --> 00:24:29.840 align:middle line:90% that's exactly what's going on. 00:24:29.840 --> 00:24:32.048 align:middle line:84% You're training the reward model with these thumbs up 00:24:32.048 --> 00:24:33.420 align:middle line:90% and thumbs down. 00:24:33.420 --> 00:24:34.860 align:middle line:90% Hold on the questions. 00:24:34.860 --> 00:24:39.680 align:middle line:84% And so if you give it the same question to GPT-3.5, 00:24:39.680 --> 00:24:45.620 align:middle line:84% InstructGPT, amazing answer, like night and day difference. 00:24:45.620 --> 00:24:48.206 align:middle line:90% Amazingly good answer. 00:24:48.206 --> 00:24:51.100 align:middle line:84% And then to go from 3.5 to ChatGPT, 00:24:51.100 --> 00:24:53.340 align:middle line:84% they basically followed the exact same playbook, 00:24:53.340 --> 00:24:57.283 align:middle line:84% except that because they wanted to have a chatbot-- 00:24:57.283 --> 00:24:58.700 align:middle line:84% meaning something that could carry 00:24:58.700 --> 00:25:00.260 align:middle line:84% on a question-answer, question-answer pair. 00:25:00.260 --> 00:25:02.060 align:middle line:84% As opposed to just a single question and answer, 00:25:02.060 --> 00:25:04.143 align:middle line:84% they wanted question-answer, question-answer, like 00:25:04.143 --> 00:25:04.820 align:middle line:90% conversation. 00:25:04.820 --> 00:25:06.720 align:middle line:84% They trained it on conversations. 00:25:06.720 --> 00:25:08.260 align:middle line:90% That's it. 00:25:08.260 --> 00:25:11.560 align:middle line:84% Instead of training it on instruction-answer data, 00:25:11.560 --> 00:25:13.660 align:middle line:84% they trained it on instruction-answer, 00:25:13.660 --> 00:25:15.750 align:middle line:84% instruction-answer, instruction-answer, 00:25:15.750 --> 00:25:19.830 align:middle line:84% a sequence of such things, which are strung into a conversation. 00:25:19.830 --> 00:25:20.410 align:middle line:90% That's it. 00:25:20.410 --> 00:25:25.030 align:middle line:84% That is the only difference to go from 3.5 to ChatGPT. 00:25:25.030 --> 00:25:26.990 align:middle line:84% Now ChatGPT, given you do that, it's giving you 00:25:26.990 --> 00:25:28.370 align:middle line:90% a much nicer response. 00:25:28.370 --> 00:25:30.253 align:middle line:84% And then you can ask a follow-on question. 00:25:30.253 --> 00:25:31.420 align:middle line:90% Can you make it more formal? 00:25:31.420 --> 00:25:32.710 align:middle line:84% Boom, it gives you a nice response. 00:25:32.710 --> 00:25:34.377 align:middle line:84% Because now it knows about conversations 00:25:34.377 --> 00:25:36.550 align:middle line:84% because it's been trained on conversational data. 00:25:36.550 --> 00:25:38.230 align:middle line:90% So that's it. 00:25:38.230 --> 00:25:39.750 align:middle line:90% That's how they built ChatGPT. 00:25:39.750 --> 00:25:41.950 align:middle line:84% And all the things we are seeing later on 00:25:41.950 --> 00:25:44.853 align:middle line:84% are all continuations of this sort of approach. 00:25:44.853 --> 00:25:46.770 align:middle line:84% So I'll pause for a couple of quick questions. 00:25:46.770 --> 00:25:47.650 align:middle line:90% Swathi, you had a question. 00:25:47.650 --> 00:25:49.290 align:middle line:84% Then we'll go to you and then to you. 00:25:49.290 --> 00:25:50.510 align:middle line:90% Yeah? 00:25:50.510 --> 00:25:52.790 align:middle line:84% AUDIENCE: So does that make a difference, 00:25:52.790 --> 00:25:57.910 align:middle line:84% if a new question-answer pair or a new training data 00:25:57.910 --> 00:26:00.630 align:middle line:84% comes early in the building of the model 00:26:00.630 --> 00:26:03.810 align:middle line:84% or later in the building of the model, 7 billion parameters. 00:26:03.810 --> 00:26:04.750 align:middle line:90% Does that make a-- 00:26:04.750 --> 00:26:06.170 align:middle line:84% RAMA RAMAKRISHNAN: You mean the order of the questions, 00:26:06.170 --> 00:26:06.910 align:middle line:90% does it matter? 00:26:06.910 --> 00:26:11.090 align:middle line:84% AUDIENCE: So I might have, let's say 5,000 images to start with. 00:26:11.090 --> 00:26:13.830 align:middle line:84% Now, after my model is trained and developed, 00:26:13.830 --> 00:26:17.390 align:middle line:84% now I have a new use case that has come in. 00:26:17.390 --> 00:26:19.610 align:middle line:84% Will that make a difference if I set it in now? 00:26:19.610 --> 00:26:21.402 align:middle line:84% RAMA RAMAKRISHNAN: So if you have a new use 00:26:21.402 --> 00:26:24.570 align:middle line:84% case for which you want to essentially adapt the model, 00:26:24.570 --> 00:26:26.430 align:middle line:84% there's a whole set of techniques you use, 00:26:26.430 --> 00:26:27.480 align:middle line:84% which is going to be the next section. 00:26:27.480 --> 00:26:28.930 align:middle line:84% AUDIENCE: But it's not [INAUDIBLE] 00:26:28.930 --> 00:26:31.472 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, because what you have out of the box 00:26:31.472 --> 00:26:33.130 align:middle line:84% is just a generally good chatbot. 00:26:33.130 --> 00:26:35.463 align:middle line:84% It knows about a lot of stuff because we've been trained 00:26:35.463 --> 00:26:36.870 align:middle line:90% on those 30 billion sentences. 00:26:36.870 --> 00:26:38.662 align:middle line:84% It can answer a lot of questions reasonably 00:26:38.662 --> 00:26:40.770 align:middle line:84% well using common sense and world knowledge. 00:26:40.770 --> 00:26:43.750 align:middle line:84% But any specific use case like medical and so on and so forth, 00:26:43.750 --> 00:26:44.483 align:middle line:90% it may not know. 00:26:44.483 --> 00:26:46.650 align:middle line:84% So you'll need to adapt it to your particular unique 00:26:46.650 --> 00:26:47.510 align:middle line:90% situation. 00:26:47.510 --> 00:26:49.490 align:middle line:90% And that's coming. 00:26:49.490 --> 00:26:52.010 align:middle line:90% All right, yes, 00:26:52.010 --> 00:26:55.730 align:middle line:84% AUDIENCE: What determines if a whole conversation is 00:26:55.730 --> 00:26:58.650 align:middle line:84% ranked positively versus just a specific answer? 00:26:58.650 --> 00:27:01.690 align:middle line:84% proliferating in your question... 00:27:01.690 --> 00:27:03.867 align:middle line:84% If the first answer doesn't get a positive response, 00:27:03.867 --> 00:27:05.950 align:middle line:84% but then after the follow-up, the second one does, 00:27:05.950 --> 00:27:06.820 align:middle line:90% does that indicate-- 00:27:06.820 --> 00:27:07.650 align:middle line:84% RAMA RAMAKRISHNAN: Correct, exactly. 00:27:07.650 --> 00:27:09.108 align:middle line:84% See, if you're a human and you read 00:27:09.108 --> 00:27:11.490 align:middle line:84% the transcript of an exchange between two people 00:27:11.490 --> 00:27:13.845 align:middle line:84% and I'm giving you two exchanges, which all start 00:27:13.845 --> 00:27:15.220 align:middle line:84% with the same question, you'll be 00:27:15.220 --> 00:27:17.520 align:middle line:84% able to assess which one is a better transcript. 00:27:17.520 --> 00:27:19.580 align:middle line:84% That's basically what's going on. 00:27:19.580 --> 00:27:21.340 align:middle line:90% There was something here, right? 00:27:21.340 --> 00:27:22.403 align:middle line:90% Yeah? 00:27:22.403 --> 00:27:23.820 align:middle line:84% AUDIENCE: So I was wondering, when 00:27:23.820 --> 00:27:26.120 align:middle line:84% you ask Chat-GPT a question, very often 00:27:26.120 --> 00:27:27.960 align:middle line:90% it sounds kind of robotic. 00:27:27.960 --> 00:27:30.900 align:middle line:84% You could tell that something was written by ChatGPT, not 00:27:30.900 --> 00:27:32.380 align:middle line:90% by an actual person. 00:27:32.380 --> 00:27:37.140 align:middle line:84% Do you think that comes from the reinforcement learning part? 00:27:37.140 --> 00:27:40.180 align:middle line:84% Or where do you think it comes from in this situation? 00:27:40.180 --> 00:27:41.000 align:middle line:84% RAMA RAMAKRISHNAN: It's a good question. 00:27:41.000 --> 00:27:41.542 align:middle line:90% I don't know. 00:27:41.542 --> 00:27:43.460 align:middle line:84% Because I know that part of the evaluation, 00:27:43.460 --> 00:27:45.420 align:middle line:84% the ranking rubric that are used, 00:27:45.420 --> 00:27:50.340 align:middle line:84% is to favor responses which sound more human-like more 00:27:50.340 --> 00:27:51.540 align:middle line:90% than robot-like. 00:27:51.540 --> 00:27:54.180 align:middle line:84% So if anything, I'm hoping that reinforcement learning would 00:27:54.180 --> 00:27:56.597 align:middle line:84% actually make it sound more human-like because the rankers 00:27:56.597 --> 00:27:57.800 align:middle line:90% would have prioritized that. 00:27:57.800 --> 00:28:01.020 align:middle line:84% So if it still comes up with robotic stuff, 00:28:01.020 --> 00:28:03.920 align:middle line:84% it's something else that's going on. 00:28:03.920 --> 00:28:07.020 align:middle line:84% Maybe there are a lot of text on the internet is not literature. 00:28:07.020 --> 00:28:08.600 align:middle line:84% It's just people writing some crap. 00:28:08.600 --> 00:28:10.720 align:middle line:90% So it could be that. 00:28:10.720 --> 00:28:13.085 align:middle line:90% Yeah? 00:28:13.085 --> 00:28:14.960 align:middle line:84% AUDIENCE: How much of this instruction tuning 00:28:14.960 --> 00:28:17.560 align:middle line:84% or conversation tuning is happening in real-time 00:28:17.560 --> 00:28:18.610 align:middle line:90% within a conversation? 00:28:18.610 --> 00:28:19.860 align:middle line:90% RAMA RAMAKRISHNAN: None of it. 00:28:19.860 --> 00:28:20.735 align:middle line:90% AUDIENCE: None of it. 00:28:20.735 --> 00:28:23.480 align:middle line:84% So as you give feedback to the model, 00:28:23.480 --> 00:28:25.480 align:middle line:84% it's just basically regenerating it. 00:28:25.480 --> 00:28:27.780 align:middle line:84% I don't like that answer, come up with something else. 00:28:27.780 --> 00:28:29.720 align:middle line:84% RAMA RAMAKRISHNAN: No, it's not doing it in real time. 00:28:29.720 --> 00:28:31.760 align:middle line:84% Basically, whatever signals you're giving it 00:28:31.760 --> 00:28:33.135 align:middle line:84% with these thumbs up, thumbs down 00:28:33.135 --> 00:28:35.400 align:middle line:84% business, that gets added to the training logs. 00:28:35.400 --> 00:28:39.280 align:middle line:84% And they periodically will retrain it. 00:28:39.280 --> 00:28:41.360 align:middle line:84% So by the way, this is instruction 00:28:41.360 --> 00:28:42.460 align:middle line:90% tuning in a nutshell. 00:28:42.460 --> 00:28:43.800 align:middle line:84% And I want to point that out-- and you 00:28:43.800 --> 00:28:45.300 align:middle line:84% don't have to read the whole thing-- 00:28:45.300 --> 00:28:46.880 align:middle line:84% but just to quickly point out this 00:28:46.880 --> 00:28:49.777 align:middle line:84% was where we had to have human involvement. 00:28:49.777 --> 00:28:52.360 align:middle line:84% The first step, writing a lot of responses to these questions, 00:28:52.360 --> 00:28:55.200 align:middle line:90% and then ranking the answers. 00:28:55.200 --> 00:28:58.280 align:middle line:84% So these two are still human labor intensive. 00:28:58.280 --> 00:29:01.240 align:middle line:84% Now, it turns out you can actually use helper LLMs 00:29:01.240 --> 00:29:03.363 align:middle line:90% to automate this too. 00:29:03.363 --> 00:29:05.780 align:middle line:84% This is not what OpenAI did in the beginning with ChatGPT. 00:29:05.780 --> 00:29:07.613 align:middle line:84% But now you can do it this way because there 00:29:07.613 --> 00:29:09.730 align:middle line:84% are lots of really good LLMs available for you 00:29:09.730 --> 00:29:11.280 align:middle line:84% to automate many of these things. 00:29:11.280 --> 00:29:13.030 align:middle line:84% We don't have time, but if you're curious, 00:29:13.030 --> 00:29:14.405 align:middle line:84% I had a little blog post on this. 00:29:14.405 --> 00:29:14.950 align:middle line:90% Check it out. 00:29:14.950 --> 00:29:18.782 align:middle line:84% OK, so now we come to the question of, 00:29:18.782 --> 00:29:21.450 align:middle line:84% well, if you want to take a base LLM like GPT-3 00:29:21.450 --> 00:29:24.390 align:middle line:84% and make it useful and respond to instructions, 00:29:24.390 --> 00:29:26.330 align:middle line:84% we have seen that we had to adapt it 00:29:26.330 --> 00:29:28.530 align:middle line:84% with high-quality instruction-answer data 00:29:28.530 --> 00:29:30.690 align:middle line:84% using Supervised Fine-Tuning and Reinforcement 00:29:30.690 --> 00:29:31.910 align:middle line:90% Learning with Human Feedback. 00:29:31.910 --> 00:29:37.370 align:middle line:84% That's what made GPT-3 actually useful and became ChatGPT. 00:29:37.370 --> 00:29:39.900 align:middle line:84% By the same token, this holds true more generally. 00:29:39.900 --> 00:29:41.650 align:middle line:84% If you want to take a large language model 00:29:41.650 --> 00:29:44.010 align:middle line:84% and make it useful for a medical use case, a legal use 00:29:44.010 --> 00:29:46.570 align:middle line:84% case, or some other narrow business use case, 00:29:46.570 --> 00:29:50.650 align:middle line:84% you have to adapt it with business domain-specific data. 00:29:50.650 --> 00:29:53.450 align:middle line:84% And so let's look at techniques for doing so. 00:29:53.450 --> 00:29:56.010 align:middle line:84% All right, so adaptation is the rough name 00:29:56.010 --> 00:29:58.690 align:middle line:84% for the process of taking a base large language model 00:29:58.690 --> 00:30:01.570 align:middle line:84% and tailoring it for your particular use case. 00:30:01.570 --> 00:30:04.170 align:middle line:84% And so there is this ladder of things you can do. 00:30:04.170 --> 00:30:06.150 align:middle line:84% And we're going to look at every one of them. 00:30:06.150 --> 00:30:09.180 align:middle line:84% So you can do this thing called zero-shot prompting, which 00:30:09.180 --> 00:30:13.277 align:middle line:84% is just you literally ask the LLM nicely, clearly, 00:30:13.277 --> 00:30:13.860 align:middle line:90% what you want. 00:30:13.860 --> 00:30:15.723 align:middle line:84% And maybe it'll just give it to you. 00:30:15.723 --> 00:30:17.140 align:middle line:84% And this is the use case we're all 00:30:17.140 --> 00:30:19.300 align:middle line:90% used to in the web interface. 00:30:19.300 --> 00:30:21.660 align:middle line:84% You can also do something called few-shot 00:30:21.660 --> 00:30:23.913 align:middle line:84% prompting where you ask it something, 00:30:23.913 --> 00:30:26.580 align:middle line:84% and you also give a few examples of the kind of things you want. 00:30:26.580 --> 00:30:28.620 align:middle line:90% And that helps it a great deal. 00:30:28.620 --> 00:30:30.660 align:middle line:84% And then there's this thing called 00:30:30.660 --> 00:30:32.840 align:middle line:84% retrieval-augmented generation and fine-tuning. 00:30:32.840 --> 00:30:33.580 align:middle line:90% And we'll look at all of them. 00:30:33.580 --> 00:30:35.622 align:middle line:84% And I'll explain all these things as we go along. 00:30:35.622 --> 00:30:38.540 align:middle line:84% So let's start with zero-shot prompting. 00:30:38.540 --> 00:30:42.460 align:middle line:84% By the way, the word "shot" here is a synonym for "example." 00:30:42.460 --> 00:30:44.983 align:middle line:84% So zero example prompting-- you literally 00:30:44.983 --> 00:30:46.900 align:middle line:84% ask in the prompt what you want without giving 00:30:46.900 --> 00:30:48.540 align:middle line:90% even a single example. 00:30:48.540 --> 00:30:53.405 align:middle line:84% And so let's say we want to look at product reviews and build 00:30:53.405 --> 00:30:55.780 align:middle line:84% a detector to figure out if the product review contains-- 00:30:55.780 --> 00:30:57.820 align:middle line:84% not sentiment, that's kind of boring-- 00:30:57.820 --> 00:31:00.140 align:middle line:84% whether it contains some description 00:31:00.140 --> 00:31:02.940 align:middle line:84% of a potential product defect or not. 00:31:02.940 --> 00:31:06.920 align:middle line:84% And so here is something I actually pulled off Wayfair, 00:31:06.920 --> 00:31:08.560 align:middle line:90% with apologies to Wayfair. 00:31:08.560 --> 00:31:10.560 align:middle line:84% It says here the curve of the back of the chair 00:31:10.560 --> 00:31:13.660 align:middle line:84% does not leave enough room to sit comfortably. 00:31:13.660 --> 00:31:16.620 align:middle line:84% Sounds like a kind of a defectish kind of thing. 00:31:16.620 --> 00:31:19.760 align:middle line:84% So back in the day, you'd have collected all these reviews 00:31:19.760 --> 00:31:22.760 align:middle line:84% and built a special-purpose NLP-based classifier to figure 00:31:22.760 --> 00:31:24.120 align:middle line:90% out defect, yes or no. 00:31:24.120 --> 00:31:27.520 align:middle line:84% Here, you can literally just feed this thing into GPT-3 00:31:27.520 --> 00:31:30.360 align:middle line:84% and ask it, tell me if a product defect is being described 00:31:30.360 --> 00:31:31.540 align:middle line:90% in this product review. 00:31:31.540 --> 00:31:33.140 align:middle line:84% And then, "the curve of the back," and, boom, boom, boom. 00:31:33.140 --> 00:31:35.880 align:middle line:84% And then it comes back and says, yep, that's a product defect. 00:31:35.880 --> 00:31:37.340 align:middle line:90% So this is zero shot. 00:31:37.340 --> 00:31:40.340 align:middle line:84% You just ask a question, you get the answer back. 00:31:40.340 --> 00:31:42.200 align:middle line:84% And it actually works remarkably well. 00:31:42.200 --> 00:31:44.400 align:middle line:84% And the better models, the bigger models 00:31:44.400 --> 00:31:47.080 align:middle line:84% tend to be much better than the smaller, simpler models 00:31:47.080 --> 00:31:48.960 align:middle line:90% for doing zero shot. 00:31:48.960 --> 00:31:52.600 align:middle line:84% All right, now when you adapt an LLM to a specific task, 00:31:52.600 --> 00:31:54.920 align:middle line:84% obviously you need to carefully design the prompt. 00:31:54.920 --> 00:31:57.200 align:middle line:84% As you folks know, this is called prompt engineering. 00:31:57.200 --> 00:31:59.742 align:middle line:84% And we're not going to spend much time on prompt engineering, 00:31:59.742 --> 00:32:01.600 align:middle line:84% except I just want to give a simple example. 00:32:01.600 --> 00:32:04.170 align:middle line:84% So if you actually ask ChatGPT this question, 00:32:04.170 --> 00:32:06.810 align:middle line:84% "What is the fifth word of the sentence?" 00:32:06.810 --> 00:32:09.890 align:middle line:84% Very often it will give the wrong answer. 00:32:09.890 --> 00:32:12.550 align:middle line:84% It's very strange why it can't get this question right. 00:32:12.550 --> 00:32:13.850 align:middle line:90% It's a very simple question. 00:32:13.850 --> 00:32:17.295 align:middle line:84% So, the fifth word of the sentence is "is." 00:32:17.295 --> 00:32:19.920 align:middle line:84% Sometimes it gets it right, but very often, it'll get it wrong. 00:32:19.920 --> 00:32:21.930 align:middle line:84% But now you can do a little prompt engineering 00:32:21.930 --> 00:32:23.230 align:middle line:90% and it'll always get it right. 00:32:23.230 --> 00:32:25.790 align:middle line:84% So for example, you can say, I'll give you a sentence. 00:32:25.790 --> 00:32:27.390 align:middle line:84% First, list all the words that are in the sentence. 00:32:27.390 --> 00:32:28.310 align:middle line:90% Then, tell me the fifth word. 00:32:28.310 --> 00:32:29.352 align:middle line:90% OK, here is the sentence. 00:32:29.352 --> 00:32:31.280 align:middle line:84% Boom, boom, boom, it gets it right. 00:32:31.280 --> 00:32:33.850 align:middle line:84% So it's an example of, you can help it along 00:32:33.850 --> 00:32:36.850 align:middle line:84% by being very, very prescriptive as to what you want it to do 00:32:36.850 --> 00:32:38.270 align:middle line:90% and break down all the steps. 00:32:38.270 --> 00:32:39.590 align:middle line:90% Don't make it guess things. 00:32:39.590 --> 00:32:41.913 align:middle line:90% It does a great job. 00:32:41.913 --> 00:32:43.330 align:middle line:84% And there are lots of other tricks 00:32:43.330 --> 00:32:46.250 align:middle line:84% people have figured out over the last couple of years. 00:32:46.250 --> 00:32:48.750 align:middle line:84% For a long time, this was pretty hot, where you say, 00:32:48.750 --> 00:32:50.290 align:middle line:90% "let's think step by step." 00:32:50.290 --> 00:32:52.910 align:middle line:84% You give it a question and say, let's think step by step, 00:32:52.910 --> 00:32:54.368 align:middle line:84% it actually gives you a better shot 00:32:54.368 --> 00:32:56.810 align:middle line:84% at giving you a good answer back, an accurate answer back. 00:32:56.810 --> 00:32:58.268 align:middle line:84% Now, this kind of thing is actually 00:32:58.268 --> 00:33:00.470 align:middle line:90% already baked into the LLMs. 00:33:00.470 --> 00:33:03.440 align:middle line:84% So when you ask a question to ChatGPT, your question, 00:33:03.440 --> 00:33:07.377 align:middle line:84% your prompt gets appended to what's called the system prompt. 00:33:07.377 --> 00:33:08.960 align:middle line:84% And the whole thing goes into the LLM. 00:33:08.960 --> 00:33:10.293 align:middle line:90% You never see the system prompt. 00:33:10.293 --> 00:33:13.320 align:middle line:84% And the system prompt is telling ChatGPT, think step by step, 00:33:13.320 --> 00:33:17.020 align:middle line:84% take your time, don't blurt out an answer, stuff like that. 00:33:17.020 --> 00:33:18.293 align:middle line:90% And you can just Google it. 00:33:18.293 --> 00:33:19.960 align:middle line:84% The system prompts have been jailbroken. 00:33:19.960 --> 00:33:21.085 align:middle line:90% You can find it on the web. 00:33:21.085 --> 00:33:24.020 align:middle line:90% 00:33:24.020 --> 00:33:25.420 align:middle line:90% And this is funny. 00:33:25.420 --> 00:33:27.080 align:middle line:84% This came out maybe a month or two ago. 00:33:27.080 --> 00:33:28.860 align:middle line:84% It says, apparently, "Take a deep breath and work 00:33:28.860 --> 00:33:31.140 align:middle line:84% on the problem step by step" works better than saying, 00:33:31.140 --> 00:33:33.100 align:middle line:90% "Work on it step by step." 00:33:33.100 --> 00:33:35.180 align:middle line:84% And then more recently, I literally 00:33:35.180 --> 00:33:37.920 align:middle line:84% read this two nights ago, apparently, if you tell it, 00:33:37.920 --> 00:33:39.840 align:middle line:84% if you have a math or a reasoning question, 00:33:39.840 --> 00:33:43.160 align:middle line:84% you tell it you are an officer on the Starship Enterprise, 00:33:43.160 --> 00:33:45.060 align:middle line:84% now solve this problem for me, it's 00:33:45.060 --> 00:33:46.320 align:middle line:90% more likely to get it right. 00:33:46.320 --> 00:33:47.460 align:middle line:90% [LAUGHTER] 00:33:47.460 --> 00:33:48.040 align:middle line:90% Go figure. 00:33:48.040 --> 00:33:48.640 align:middle line:90% Thomas? 00:33:48.640 --> 00:33:50.800 align:middle line:84% AUDIENCE: I read two more that work super funny. 00:33:50.800 --> 00:33:51.100 align:middle line:90% RAMA RAMAKRISHNAN: Yeah? 00:33:51.100 --> 00:33:53.207 align:middle line:84% AUDIENCE: One, "I will tip you if you solve this." 00:33:53.207 --> 00:33:54.332 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:33:54.332 --> 00:33:57.860 align:middle line:84% AUDIENCE: And the other one was an answer was, 00:33:57.860 --> 00:34:01.500 align:middle line:90% "I cannot do that" for output. 00:34:01.500 --> 00:34:06.640 align:middle line:84% And it was, "I tried it on Gemini and it was able to solve 00:34:06.640 --> 00:34:07.140 align:middle line:90% it." 00:34:07.140 --> 00:34:08.140 align:middle line:90% RAMA RAMAKRISHNAN: Nice. 00:34:08.140 --> 00:34:11.120 align:middle line:84% AUDIENCE: And both, like back and forward, "ChatGPT 00:34:11.120 --> 00:34:12.260 align:middle line:90% was able to solve this. 00:34:12.260 --> 00:34:13.343 align:middle line:90% Can you solve this?" 00:34:13.343 --> 00:34:15.260 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, very good, excellent. 00:34:15.260 --> 00:34:17.560 align:middle line:84% One of the things just on that, let's have some fun. 00:34:17.560 --> 00:34:20.100 align:middle line:84% You can say I'm going to tip you $1,000 if you solve this, 00:34:20.100 --> 00:34:20.900 align:middle line:90% it says right. 00:34:20.900 --> 00:34:23.699 align:middle line:84% So this person apparently kept using this tip. 00:34:23.699 --> 00:34:26.420 align:middle line:84% And at one point, it says, you keep promising me tips, 00:34:26.420 --> 00:34:27.902 align:middle line:90% you never give me the tip. 00:34:27.902 --> 00:34:29.860 align:middle line:84% So I'm not going to solve this problem for you. 00:34:29.860 --> 00:34:31.679 align:middle line:90% [LAUGHTER] 00:34:31.679 --> 00:34:33.679 align:middle line:90% Yeah. 00:34:33.679 --> 00:34:35.820 align:middle line:84% And there are many prompt engineering resources. 00:34:35.820 --> 00:34:38.112 align:middle line:84% This one that came out a couple of weeks ago, I thought 00:34:38.112 --> 00:34:40.280 align:middle line:84% was pretty good so I just put a link to it here. 00:34:40.280 --> 00:34:42.120 align:middle line:84% So now let's look at few-shot prompting, 00:34:42.120 --> 00:34:43.800 align:middle line:84% where you give it a few examples. 00:34:43.800 --> 00:34:47.493 align:middle line:84% So here, let's say you want to build a grammar corrector. 00:34:47.493 --> 00:34:50.960 align:middle line:84% So what you can do is, you can actually give it examples 00:34:50.960 --> 00:34:52.719 align:middle line:90% of poor English, good English. 00:34:52.719 --> 00:34:53.440 align:middle line:90% You can see. 00:34:53.440 --> 00:34:55.580 align:middle line:84% Poor English-- "I eated the purple berries." 00:34:55.580 --> 00:34:57.420 align:middle line:84% Good English-- "I ate the purple berries." 00:34:57.420 --> 00:34:59.210 align:middle line:90% And similarly, three examples. 00:34:59.210 --> 00:35:02.730 align:middle line:84% And then you end the prompt with just the poor English input. 00:35:02.730 --> 00:35:06.110 align:middle line:84% And then the response from GPT-3 is the good English output. 00:35:06.110 --> 00:35:08.930 align:middle line:90% And it says, fix the error. 00:35:08.930 --> 00:35:11.610 align:middle line:84% So this is an example of giving a few examples of what you want. 00:35:11.610 --> 00:35:13.370 align:middle line:84% And it just learns on the fly, what 00:35:13.370 --> 00:35:15.730 align:middle line:84% you have in mind, what your intention is. 00:35:15.730 --> 00:35:17.830 align:middle line:90% So that's that. 00:35:17.830 --> 00:35:19.770 align:middle line:84% Now, the ability of LLMs to learn 00:35:19.770 --> 00:35:23.030 align:middle line:84% from just a few examples or even no examples, 00:35:23.030 --> 00:35:25.210 align:middle line:84% and just with a clear instruction, this thing 00:35:25.210 --> 00:35:27.170 align:middle line:90% is called in-context learning. 00:35:27.170 --> 00:35:30.570 align:middle line:84% And that was something that GPT-2 and GPT could not do. 00:35:30.570 --> 00:35:32.970 align:middle line:84% That was new in GPT-3 and what they 00:35:32.970 --> 00:35:34.750 align:middle line:90% call an emergent capability. 00:35:34.750 --> 00:35:39.810 align:middle line:84% It was completely unanticipated by the people who built it. 00:35:39.810 --> 00:35:40.690 align:middle line:90% So that's that. 00:35:40.690 --> 00:35:42.750 align:middle line:84% Now let's look at retrieval augmented generation. 00:35:42.750 --> 00:35:45.370 align:middle line:84% By the way, this thing is also called indexing sometimes. 00:35:45.370 --> 00:35:49.190 align:middle line:84% So it's called RAG, retrieval augmented generation, RAG. 00:35:49.190 --> 00:35:51.290 align:middle line:84% The idea of RAG is actually very simple. 00:35:51.290 --> 00:35:54.590 align:middle line:84% So let's say that we want to ask a question to a chatbot, 00:35:54.590 --> 00:35:58.380 align:middle line:84% but we want the chatbot to leverage proprietary data 00:35:58.380 --> 00:35:59.580 align:middle line:90% that you might have. 00:35:59.580 --> 00:36:02.020 align:middle line:84% Maybe it's a customer call support in a call center 00:36:02.020 --> 00:36:05.500 align:middle line:84% kind of operation and you have this massive FAQ database, 00:36:05.500 --> 00:36:06.600 align:middle line:90% content database. 00:36:06.600 --> 00:36:09.082 align:middle line:84% And you want to give that FAQ to the chatbot 00:36:09.082 --> 00:36:10.540 align:middle line:84% along with your question so that it 00:36:10.540 --> 00:36:13.240 align:middle line:84% can leverage the FAQ to answer the question for you, 00:36:13.240 --> 00:36:15.900 align:middle line:84% as opposed to whatever things it has learned previously 00:36:15.900 --> 00:36:17.280 align:middle line:90% in its general training. 00:36:17.280 --> 00:36:22.100 align:middle line:84% So can't we just include the entire FAQ, the whole dataset, 00:36:22.100 --> 00:36:24.140 align:middle line:90% into a prompt and send it in? 00:36:24.140 --> 00:36:26.140 align:middle line:84% Maybe we just take our question, take everything 00:36:26.140 --> 00:36:28.473 align:middle line:84% we have potentially relevant to the question, everything 00:36:28.473 --> 00:36:31.505 align:middle line:84% we have in the database, just attach it to the question. 00:36:31.505 --> 00:36:32.880 align:middle line:84% The whole thing becomes a prompt. 00:36:32.880 --> 00:36:34.960 align:middle line:84% Feed it in and say, hey, find out for me. 00:36:34.960 --> 00:36:35.960 align:middle line:90% Can't we just do that? 00:36:35.960 --> 00:36:38.580 align:middle line:90% 00:36:38.580 --> 00:36:39.960 align:middle line:90% Theoretically, nothing stops us. 00:36:39.960 --> 00:36:43.180 align:middle line:90% 00:36:43.180 --> 00:36:45.460 align:middle line:84% The reason you can't do it is because this pesky thing 00:36:45.460 --> 00:36:47.620 align:middle line:90% called the context window. 00:36:47.620 --> 00:36:52.760 align:middle line:84% So for any LLM, the prompt plus the output, 00:36:52.760 --> 00:36:54.823 align:middle line:84% the length cannot exceed a predefined limit. 00:36:54.823 --> 00:36:56.240 align:middle line:84% This is called the context window. 00:36:56.240 --> 00:37:01.040 align:middle line:84% Remember the max sequence length we had in our earlier models, 00:37:01.040 --> 00:37:03.730 align:middle line:84% where that was the size of the sentence that could be fed in. 00:37:03.730 --> 00:37:05.480 align:middle line:84% Basically, there is a size of the sentence 00:37:05.480 --> 00:37:06.960 align:middle line:90% for any of these things. 00:37:06.960 --> 00:37:08.360 align:middle line:90% It's called the context window. 00:37:08.360 --> 00:37:10.840 align:middle line:84% There are only so many tokens it can accommodate. 00:37:10.840 --> 00:37:13.760 align:middle line:84% And since what comes in is what comes out, 00:37:13.760 --> 00:37:16.245 align:middle line:84% it is for both the input and the output together. 00:37:16.245 --> 00:37:17.620 align:middle line:84% That's called the context window. 00:37:17.620 --> 00:37:21.065 align:middle line:90% 00:37:21.065 --> 00:37:23.760 align:middle line:84% And furthermore, when you have a conversation 00:37:23.760 --> 00:37:26.560 align:middle line:84% with one of these chatbots, the entire conversation 00:37:26.560 --> 00:37:29.880 align:middle line:90% is fed in every single time. 00:37:29.880 --> 00:37:31.840 align:middle line:84% That's how it actually remembers what's going 00:37:31.840 --> 00:37:33.800 align:middle line:90% on earlier in the conversation. 00:37:33.800 --> 00:37:35.460 align:middle line:84% It doesn't have any memory per se. 00:37:35.460 --> 00:37:38.960 align:middle line:84% Each time you ask a question, the entire thread is fed in. 00:37:38.960 --> 00:37:41.820 align:middle line:84% So initially you say, what's the square root of 17? 00:37:41.820 --> 00:37:42.940 align:middle line:90% It gives you an answer. 00:37:42.940 --> 00:37:45.160 align:middle line:84% Initially, you only send in the red stuff. 00:37:45.160 --> 00:37:47.620 align:middle line:84% Then, the next question you ask is the first question, 00:37:47.620 --> 00:37:50.280 align:middle line:84% the answer, the second question, all of them are fed in. 00:37:50.280 --> 00:37:51.760 align:middle line:90% Then, all these are fed in. 00:37:51.760 --> 00:37:53.930 align:middle line:84% So with the conversation, you're consuming 00:37:53.930 --> 00:37:58.050 align:middle line:84% more and more of the context window as you go along. 00:37:58.050 --> 00:38:01.007 align:middle line:84% So can you imagine taking a whole FAQ, asking a question, 00:38:01.007 --> 00:38:03.590 align:middle line:84% and saying, well, I didn't mean that, I wanted something else. 00:38:03.590 --> 00:38:04.490 align:middle line:84% And before you know it, boom, you've 00:38:04.490 --> 00:38:05.630 align:middle line:90% blown out the context window. 00:38:05.630 --> 00:38:07.547 align:middle line:84% It's going to come back and give you an error. 00:38:07.547 --> 00:38:11.330 align:middle line:84% AUDIENCE: Can you just make it so you can't [INAUDIBLE]. 00:38:11.330 --> 00:38:15.170 align:middle line:84% Or does it take specific windows of it? 00:38:15.170 --> 00:38:17.570 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so there is a whole research 00:38:17.570 --> 00:38:20.050 align:middle line:84% cottage industry around when your thing is 00:38:20.050 --> 00:38:22.650 align:middle line:84% longer than the context window, what do you pick? 00:38:22.650 --> 00:38:25.390 align:middle line:84% So the simplest case is you have a moving window. 00:38:25.390 --> 00:38:27.810 align:middle line:84% If you have 1,000 tokens, you just look at the last 1,000 00:38:27.810 --> 00:38:28.430 align:middle line:90% tokens. 00:38:28.430 --> 00:38:31.130 align:middle line:84% But there are some cleverer schemes where you can actually 00:38:31.130 --> 00:38:34.210 align:middle line:84% take the first stuff that is outside the window that doesn't 00:38:34.210 --> 00:38:37.650 align:middle line:84% fit into the window and use another LLM to summarize it 00:38:37.650 --> 00:38:38.730 align:middle line:90% for you. 00:38:38.730 --> 00:38:41.170 align:middle line:84% And then you attach it to your current prompt. 00:38:41.170 --> 00:38:44.090 align:middle line:90% I know, it gets crazy. 00:38:44.090 --> 00:38:46.170 align:middle line:84% OK, so for all these reasons, we need 00:38:46.170 --> 00:38:48.770 align:middle line:84% to pick and choose what we can send to answer 00:38:48.770 --> 00:38:50.010 align:middle line:90% a particular question. 00:38:50.010 --> 00:38:52.700 align:middle line:84% So what we do is, since we can't include the whole thing, 00:38:52.700 --> 00:38:54.620 align:middle line:84% we first retrieve the relevant content 00:38:54.620 --> 00:38:57.140 align:middle line:84% from the database or the FAQ and then send it 00:38:57.140 --> 00:38:59.420 align:middle line:84% to the LLM along with the question we have. 00:38:59.420 --> 00:39:03.160 align:middle line:84% So retrieval augmented sequence generation, 00:39:03.160 --> 00:39:05.180 align:middle line:90% that's what's going on. 00:39:05.180 --> 00:39:06.380 align:middle line:90% Make sense? 00:39:06.380 --> 00:39:09.940 align:middle line:84% And so pictorially, basically, what we do is, 00:39:09.940 --> 00:39:14.780 align:middle line:84% let's say that this is our external set of documents. 00:39:14.780 --> 00:39:16.260 align:middle line:90% Think of it as an FAQ. 00:39:16.260 --> 00:39:19.260 align:middle line:84% And then we take the FAQ, and imagine for each question 00:39:19.260 --> 00:39:22.140 align:middle line:84% and answer, we take each question and answer in the FAQ, 00:39:22.140 --> 00:39:26.420 align:middle line:84% and then we treat it as its own little unit of text. 00:39:26.420 --> 00:39:29.260 align:middle line:84% And then we actually calculate a contextual embedding 00:39:29.260 --> 00:39:32.100 align:middle line:84% for each of those question-answer pairs. 00:39:32.100 --> 00:39:35.010 align:middle line:84% Remember, we know how to do contextual embeddings. 00:39:35.010 --> 00:39:36.760 align:middle line:84% It's a piece of cake at this point, right? 00:39:36.760 --> 00:39:38.677 align:middle line:84% You folks know how to do contextual embedding. 00:39:38.677 --> 00:39:41.180 align:middle line:84% Run it through something like BERT, you're done. 00:39:41.180 --> 00:39:42.680 align:middle line:90% You get a contextual embedding. 00:39:42.680 --> 00:39:46.660 align:middle line:84% So you get embeddings for all the things that are in your FAQ. 00:39:46.660 --> 00:39:50.600 align:middle line:84% And now when a new question comes in, what you do 00:39:50.600 --> 00:39:52.360 align:middle line:84% is you take that question and you 00:39:52.360 --> 00:39:56.560 align:middle line:84% calculate a contextual embedding for that too. 00:39:56.560 --> 00:39:58.600 align:middle line:84% And then what you do is you then look 00:39:58.600 --> 00:40:03.960 align:middle line:84% to see which of the FAQ elements you have, which of those chunks 00:40:03.960 --> 00:40:05.580 align:middle line:84% are the most similar to your question. 00:40:05.580 --> 00:40:08.160 align:middle line:90% 00:40:08.160 --> 00:40:10.720 align:middle line:84% And then you grab the ones that are the most similar 00:40:10.720 --> 00:40:13.800 align:middle line:84% and then pack it into the prompt and send it in. 00:40:13.800 --> 00:40:17.505 align:middle line:84% Maybe you have 10,000 questions, but you can only accommodate 5 00:40:17.505 --> 00:40:19.880 align:middle line:84% of them in your prompt because the context window is very 00:40:19.880 --> 00:40:20.540 align:middle line:90% small. 00:40:20.540 --> 00:40:22.320 align:middle line:84% So you pick the 5, what you think 00:40:22.320 --> 00:40:25.500 align:middle line:84% is the most relevant content to your particular question, 00:40:25.500 --> 00:40:28.133 align:middle line:90% and then you feed it in. 00:40:28.133 --> 00:40:28.800 align:middle line:90% That's the idea. 00:40:28.800 --> 00:40:30.580 align:middle line:84% That is retrieval augmented generation. 00:40:30.580 --> 00:40:32.240 align:middle line:90% Yep, 00:40:32.240 --> 00:40:34.740 align:middle line:84% AUDIENCE: So does this tie-in, for example, 00:40:34.740 --> 00:40:36.160 align:middle line:84% if I were to prompt and say, help 00:40:36.160 --> 00:40:38.240 align:middle line:84% me work on my startup pitch, but do it 00:40:38.240 --> 00:40:40.160 align:middle line:90% in the voice of Steve Jobs. 00:40:40.160 --> 00:40:45.600 align:middle line:84% Is it then going out there and reducing the subset of data 00:40:45.600 --> 00:40:48.910 align:middle line:84% to things that have been written by Steve Jobs? 00:40:48.910 --> 00:40:51.460 align:middle line:84% And then it's kind of generating its response based-- 00:40:51.460 --> 00:40:53.710 align:middle line:84% RAMA RAMAKRISHNAN: Not as a default, not as a default, 00:40:53.710 --> 00:40:55.950 align:middle line:84% typically, because there's a lot of Steve Jobs stuff on the web. 00:40:55.950 --> 00:40:57.410 align:middle line:84% It was just using that because it's 00:40:57.410 --> 00:40:59.170 align:middle line:84% all part of its pre-training data. 00:40:59.170 --> 00:41:02.318 align:middle line:84% But this tends to be more useful for very targeted applications 00:41:02.318 --> 00:41:04.610 align:middle line:84% where you don't expect it to know the answer because it 00:41:04.610 --> 00:41:07.490 align:middle line:90% is not in the public internet. 00:41:07.490 --> 00:41:09.290 align:middle line:84% It's your proprietary data and you wanted 00:41:09.290 --> 00:41:10.498 align:middle line:90% to use that proprietary data. 00:41:10.498 --> 00:41:12.930 align:middle line:90% And this is how you do it. 00:41:12.930 --> 00:41:13.626 align:middle line:90% Yeah? 00:41:13.626 --> 00:41:16.251 align:middle line:84% AUDIENCE: [INAUDIBLE] ..certain information sure like that will 00:41:16.251 --> 00:41:18.450 align:middle line:90% there be some loss? 00:41:18.450 --> 00:41:19.450 align:middle line:90% RAMA RAMAKRISHNAN: Sure. 00:41:19.450 --> 00:41:21.617 align:middle line:84% AUDIENCE: Do you think that there will be some loss? 00:41:21.617 --> 00:41:23.408 align:middle line:84% RAMA RAMAKRISHNAN: There will be some loss. 00:41:23.408 --> 00:41:25.910 align:middle line:84% Because you have to figure out how to chunk it. 00:41:25.910 --> 00:41:29.850 align:middle line:84% Maybe you have a 300-page PDF, and then maybe you look for each 00:41:29.850 --> 00:41:31.125 align:middle line:90% section and make it a chunk. 00:41:31.125 --> 00:41:33.250 align:middle line:84% Maybe you look for each paragraph, make it a chunk. 00:41:33.250 --> 00:41:36.500 align:middle line:84% Again, there is a whole empirical cottage industry 00:41:36.500 --> 00:41:38.750 align:middle line:84% of techniques for doing these things, better or worse, 00:41:38.750 --> 00:41:40.792 align:middle line:84% depending on the use case and so on and so forth. 00:41:40.792 --> 00:41:43.730 align:middle line:84% But the conceptual idea is chunk and embedding. 00:41:43.730 --> 00:41:45.730 align:middle line:84% AUDIENCE: Chunking in the middle of [INAUDIBLE]. 00:41:45.730 --> 00:41:47.105 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, in fact, 00:41:47.105 --> 00:41:49.430 align:middle line:84% we're going to do it ourselves in the Colab right now. 00:41:49.430 --> 00:41:50.630 align:middle line:90% Yeah? 00:41:50.630 --> 00:41:54.082 align:middle line:84% AUDIENCE: Can you give more weight to a particular chunk? 00:41:54.082 --> 00:41:57.350 align:middle line:84% RAMA RAMAKRISHNAN: [LAUGHS] So in the default implementation, 00:41:57.350 --> 00:41:57.990 align:middle line:90% no. 00:41:57.990 --> 00:42:01.470 align:middle line:84% But in some sense, by picking the 5 most relevant chunks from 00:42:01.470 --> 00:42:06.390 align:middle line:84% 10,000 chunks, you're giving the other 10,000 minus 5 chunks 00:42:06.390 --> 00:42:08.370 align:middle line:84% a weight of 0 and these a weight of 1. 00:42:08.370 --> 00:42:10.710 align:middle line:84% So in some sense, you're weighting it. 00:42:10.710 --> 00:42:11.775 align:middle line:90% Yeah? 00:42:11.775 --> 00:42:13.650 align:middle line:84% AUDIENCE: I'm just curious how much structure 00:42:13.650 --> 00:42:15.010 align:middle line:84% you have to have with an external document? 00:42:15.010 --> 00:42:16.790 align:middle line:84% If you're a hospital or something, 00:42:16.790 --> 00:42:18.550 align:middle line:84% do you have to do a bunch of labelling? 00:42:18.550 --> 00:42:19.967 align:middle line:84% RAMA RAMAKRISHNAN: Pre-processing? 00:42:19.967 --> 00:42:23.070 align:middle line:84% No, you just need to make sure it's relatively clean. 00:42:23.070 --> 00:42:26.430 align:middle line:84% But you will see in the Colab that it can be kind of crappy 00:42:26.430 --> 00:42:28.070 align:middle line:90% and it still works. 00:42:28.070 --> 00:42:28.630 align:middle line:90% Yeah. 00:42:28.630 --> 00:42:30.130 align:middle line:84% Because there is so much crap on the internet 00:42:30.130 --> 00:42:31.338 align:middle line:90% it's been trained on already. 00:42:31.338 --> 00:42:33.590 align:middle line:90% 00:42:33.590 --> 00:42:36.520 align:middle line:84% All right, so let's look at the Colab. 00:42:36.520 --> 00:42:38.270 align:middle line:84% By the way, retrieval augmented generation 00:42:38.270 --> 00:42:41.630 align:middle line:84% is, in my opinion, the most prevalent business 00:42:41.630 --> 00:42:45.760 align:middle line:84% application of LLMs that I've seen up to date. 00:42:45.760 --> 00:42:47.960 align:middle line:84% And there's a huge ecosystem of tools and vendors 00:42:47.960 --> 00:42:51.320 align:middle line:90% and so on and so forth. 00:42:51.320 --> 00:42:53.680 align:middle line:84% I'm going to skip through the verbiage here. 00:42:53.680 --> 00:42:58.760 align:middle line:84% So you have to install the OpenAI library 00:42:58.760 --> 00:43:01.337 align:middle line:84% and this thing called tiktoken, which we'll get to in a bit. 00:43:01.337 --> 00:43:02.920 align:middle line:84% I've already installed it before class 00:43:02.920 --> 00:43:04.560 align:middle line:90% because it takes some time. 00:43:04.560 --> 00:43:07.920 align:middle line:84% So I'll just make sure all these things are already-- 00:43:07.920 --> 00:43:09.040 align:middle line:90% phew, good. 00:43:09.040 --> 00:43:10.640 align:middle line:84% So we don't have to wait for this. 00:43:10.640 --> 00:43:14.295 align:middle line:84% So we imported pandas, as before. 00:43:14.295 --> 00:43:15.920 align:middle line:84% And you could read through these things 00:43:15.920 --> 00:43:21.400 align:middle line:84% because basically I have an OpenAI token that I have to use, 00:43:21.400 --> 00:43:23.348 align:middle line:90% a key, rather, key, API key. 00:43:23.348 --> 00:43:25.140 align:middle line:84% And I'm not showing you the key, obviously. 00:43:25.140 --> 00:43:27.640 align:middle line:84% I have to remember to delete it before I upload the Colab. 00:43:27.640 --> 00:43:30.200 align:middle line:84% You have to get your own key to make it all work, 00:43:30.200 --> 00:43:31.740 align:middle line:90% but the instructions are here. 00:43:31.740 --> 00:43:35.200 align:middle line:84% So we're going to use GPT-3.5-turbo to demonstrate 00:43:35.200 --> 00:43:35.880 align:middle line:90% RAG. 00:43:35.880 --> 00:43:37.440 align:middle line:90% So I give the name of the model. 00:43:37.440 --> 00:43:40.480 align:middle line:84% And then OpenAI also has a whole bunch of different models 00:43:40.480 --> 00:43:43.810 align:middle line:84% which can be used for, you can feed it 00:43:43.810 --> 00:43:45.580 align:middle line:90% a sentence or a chunk of text. 00:43:45.580 --> 00:43:47.330 align:middle line:84% It'll give you a contextual embedding out. 00:43:47.330 --> 00:43:48.470 align:middle line:90% It's like a nice little API. 00:43:48.470 --> 00:43:50.928 align:middle line:84% You don't have to use your own BERT and so on and so forth. 00:43:50.928 --> 00:43:52.835 align:middle line:84% You can just use the OpenAI embeddings. 00:43:52.835 --> 00:43:54.210 align:middle line:84% Obviously, you have to pay OpenAI 00:43:54.210 --> 00:43:56.710 align:middle line:84% every time you make a request, but it's really, really cheap 00:43:56.710 --> 00:43:57.530 align:middle line:90% at this point. 00:43:57.530 --> 00:43:59.885 align:middle line:90% Yeah, 00:43:59.885 --> 00:44:03.314 align:middle line:90% AUDIENCE: [INAUDIBLE] question. 00:44:03.314 --> 00:44:06.410 align:middle line:84% By reading the proprietary data, because a lot of companies 00:44:06.410 --> 00:44:08.210 align:middle line:84% are like, we need to invest in our own LLM 00:44:08.210 --> 00:44:11.410 align:middle line:84% because we don't want our data to be going out. 00:44:11.410 --> 00:44:13.970 align:middle line:84% In this kind of a context, how good 00:44:13.970 --> 00:44:16.547 align:middle line:84% is the cybersecurity or the compliance and legal stuff? 00:44:16.547 --> 00:44:18.130 align:middle line:84% RAMA RAMAKRISHNAN: I think each vendor 00:44:18.130 --> 00:44:21.610 align:middle line:84% has their own set of rules and contractual commitments 00:44:21.610 --> 00:44:22.950 align:middle line:90% they're willing to sign up for. 00:44:22.950 --> 00:44:23.492 align:middle line:90% So you just-- 00:44:23.492 --> 00:44:24.950 align:middle line:84% AUDIENCE: If you use the data here, 00:44:24.950 --> 00:44:26.730 align:middle line:84% does this go into the company [INAUDIBLE]? 00:44:26.730 --> 00:44:27.330 align:middle line:90% No? 00:44:27.330 --> 00:44:29.372 align:middle line:84% RAMA RAMAKRISHNAN: But the vendor gets to see it. 00:44:29.372 --> 00:44:30.070 align:middle line:90% AUDIENCE: OK. 00:44:30.070 --> 00:44:32.570 align:middle line:84% RAMA RAMAKRISHNAN: Meaning the vendor systems get to see it. 00:44:32.570 --> 00:44:35.190 align:middle line:84% But do the vendors' employees get to see it if they need to? 00:44:35.190 --> 00:44:36.090 align:middle line:90% Unclear. 00:44:36.090 --> 00:44:38.530 align:middle line:84% Those are all the legalese, nitty-gritty 00:44:38.530 --> 00:44:39.530 align:middle line:90% you have to worry about. 00:44:39.530 --> 00:44:41.447 align:middle line:84% The other thing you can do is you can actually 00:44:41.447 --> 00:44:43.432 align:middle line:84% just download an open source LLM and do it 00:44:43.432 --> 00:44:44.640 align:middle line:90% all within your own premises. 00:44:44.640 --> 00:44:45.948 align:middle line:90% AUDIENCE: Oh, OK 00:44:45.948 --> 00:44:47.990 align:middle line:84% RAMA RAMAKRISHNAN: That's totally possible to do. 00:44:47.990 --> 00:44:49.995 align:middle line:84% In fact, I probably won't have time today. 00:44:49.995 --> 00:44:51.870 align:middle line:84% I have a whole section on how do you actually 00:44:51.870 --> 00:44:54.710 align:middle line:84% do a fine-tuning with an open source LLM, which I'll do 00:44:54.710 --> 00:44:59.470 align:middle line:90% a video if you don't have time. 00:44:59.470 --> 00:45:01.538 align:middle line:84% And so this model, this embedding-ada-002 00:45:01.538 --> 00:45:03.830 align:middle line:84% is the name of the OpenAI model that actually gives you 00:45:03.830 --> 00:45:04.705 align:middle line:90% contextual embedding. 00:45:04.705 --> 00:45:07.610 align:middle line:90% So we're going to use that. 00:45:07.610 --> 00:45:12.370 align:middle line:84% So the use case here is that we want to ask the LLM, 00:45:12.370 --> 00:45:15.070 align:middle line:84% we want to create a chatbot which can answer questions 00:45:15.070 --> 00:45:17.973 align:middle line:84% about the 2022 Olympics, like random questions 00:45:17.973 --> 00:45:19.390 align:middle line:84% you might have about the Olympics. 00:45:19.390 --> 00:45:24.310 align:middle line:84% So let's first ask it this question. 00:45:24.310 --> 00:45:27.750 align:middle line:84% We'll ask it about the 2020 Summer Olympics. 00:45:27.750 --> 00:45:29.550 align:middle line:90% That's the query. 00:45:29.550 --> 00:45:34.043 align:middle line:84% And then this is the API request we have to make. 00:45:34.043 --> 00:45:35.210 align:middle line:90% And you can read through it. 00:45:35.210 --> 00:45:36.627 align:middle line:84% I have a link to the documentation 00:45:36.627 --> 00:45:37.690 align:middle line:90% here as to how it works. 00:45:37.690 --> 00:45:41.520 align:middle line:84% And then it says that "Barshim of Qatar and Tamberi of Italy 00:45:41.520 --> 00:45:42.460 align:middle line:90% both won the gold." 00:45:42.460 --> 00:45:43.580 align:middle line:84% And you can actually fact check this. 00:45:43.580 --> 00:45:44.538 align:middle line:90% It's actually accurate. 00:45:44.538 --> 00:45:45.680 align:middle line:90% It's correct. 00:45:45.680 --> 00:45:47.560 align:middle line:84% So now let's change the query and ask it 00:45:47.560 --> 00:45:49.870 align:middle line:90% about the 2022 Winter Olympics. 00:45:49.870 --> 00:45:53.960 align:middle line:84% And why '22 versus '20 will become clear in just a moment. 00:45:53.960 --> 00:45:59.520 align:middle line:84% So, "Which athletes won the gold in curling in the '22 Olympics?" 00:45:59.520 --> 00:46:02.520 align:middle line:84% And it says, "The gold medal in curling was won by the Swedish 00:46:02.520 --> 00:46:07.140 align:middle line:84% men's team and the South Korean women's team." 00:46:07.140 --> 00:46:11.440 align:middle line:84% If you fact check this, it turns out, wait for it, 00:46:11.440 --> 00:46:13.308 align:middle line:90% Sweden won the men's gold, yes. 00:46:13.308 --> 00:46:15.600 align:middle line:84% South Korea did participate, but Great Britain actually 00:46:15.600 --> 00:46:17.000 align:middle line:90% won the Women's Gold. 00:46:17.000 --> 00:46:18.160 align:middle line:90% So it got it wrong. 00:46:18.160 --> 00:46:21.000 align:middle line:84% So it sounds like GPT-3.5-turbo could use some help. 00:46:21.000 --> 00:46:23.280 align:middle line:84% And now one of the things we can do 00:46:23.280 --> 00:46:27.160 align:middle line:84% is, the thing is, the reason why GPT-3-turbo didn't know 00:46:27.160 --> 00:46:29.840 align:middle line:84% about this is because its training cutoff 00:46:29.840 --> 00:46:32.440 align:middle line:90% date was September 2021. 00:46:32.440 --> 00:46:34.800 align:middle line:84% So as far as it's concerned, the '22 Olympics 00:46:34.800 --> 00:46:36.560 align:middle line:90% haven't happened yet. 00:46:36.560 --> 00:46:39.570 align:middle line:84% Yet, it confidently gave you the wrong answer, 00:46:39.570 --> 00:46:42.030 align:middle line:90% as it is often prone to do. 00:46:42.030 --> 00:46:44.030 align:middle line:84% And this, by the way, it's called hallucination, 00:46:44.030 --> 00:46:46.210 align:middle line:84% where it gives you a very eloquent, confident, wrong 00:46:46.210 --> 00:46:46.710 align:middle line:90% answer. 00:46:46.710 --> 00:46:50.210 align:middle line:90% 00:46:50.210 --> 00:46:54.210 align:middle line:84% Or as some folks have said about another business school that 00:46:54.210 --> 00:46:57.866 align:middle line:84% shall remain nameless, often in error, but never in doubt. 00:46:57.866 --> 00:46:59.530 align:middle line:90% [LAUGHTER] 00:46:59.530 --> 00:47:01.490 align:middle line:90% All right, back to this. 00:47:01.490 --> 00:47:04.170 align:middle line:84% So one simple thing we can try right off the bat is to tell 00:47:04.170 --> 00:47:07.050 align:middle line:84% 3.5-turbo, you can ask it to say, I don't know, 00:47:07.050 --> 00:47:09.910 align:middle line:84% if it doesn't know, rather than just make stuff up. 00:47:09.910 --> 00:47:11.070 align:middle line:90% And how do you do it? 00:47:11.070 --> 00:47:11.790 align:middle line:90% It's very simple. 00:47:11.790 --> 00:47:14.890 align:middle line:84% You say in your prompt, "Answer the question as truthfully 00:47:14.890 --> 00:47:16.650 align:middle line:90% as possible. 00:47:16.650 --> 00:47:18.590 align:middle line:84% And if you're unsure of the answer, say sorry, 00:47:18.590 --> 00:47:19.290 align:middle line:90% I don't know." 00:47:19.290 --> 00:47:20.750 align:middle line:90% OK, now here is the question. 00:47:20.750 --> 00:47:25.450 align:middle line:84% OK, this is a query, so let's run it through. 00:47:25.450 --> 00:47:27.090 align:middle line:90% "Sorry, I don't know." 00:47:27.090 --> 00:47:28.810 align:middle line:90% Not bad, huh? 00:47:28.810 --> 00:47:30.150 align:middle line:90% So it worked. 00:47:30.150 --> 00:47:33.250 align:middle line:84% It's trying to be humble and honest and self-aware 00:47:33.250 --> 00:47:35.670 align:middle line:90% and things like that. 00:47:35.670 --> 00:47:38.990 align:middle line:84% It's a bit more like a Sloany at this point. 00:47:38.990 --> 00:47:40.870 align:middle line:84% Now, the reason, as I mentioned earlier, you 00:47:40.870 --> 00:47:43.290 align:middle line:84% can check the cutoff date and you can see it's 2021. 00:47:43.290 --> 00:47:44.290 align:middle line:90% Actually, you know what? 00:47:44.290 --> 00:47:45.810 align:middle line:90% Let's just open a new tab. 00:47:45.810 --> 00:47:49.270 align:middle line:90% 00:47:49.270 --> 00:47:51.410 align:middle line:84% So all these cutoff dates are training data. 00:47:51.410 --> 00:47:53.790 align:middle line:84% So 3.5-turbo, this is what we are using-- 00:47:53.790 --> 00:47:55.910 align:middle line:90% cutoff date 2021. 00:47:55.910 --> 00:47:57.690 align:middle line:90% That's why. 00:47:57.690 --> 00:48:00.550 align:middle line:84% So now what we can do is, we can obviously 00:48:00.550 --> 00:48:02.350 align:middle line:84% provide relevant data in the prompt itself. 00:48:02.350 --> 00:48:04.048 align:middle line:90% We can leading up to RAG here. 00:48:04.048 --> 00:48:05.590 align:middle line:84% And by the way, the extra information 00:48:05.590 --> 00:48:07.790 align:middle line:84% we provide in the prompt to help it answer a question is called 00:48:07.790 --> 00:48:08.510 align:middle line:90% context. 00:48:08.510 --> 00:48:10.070 align:middle line:90% That's the lingo for it. 00:48:10.070 --> 00:48:10.770 align:middle line:90% So we can do it. 00:48:10.770 --> 00:48:12.710 align:middle line:90% We'll first do it manually. 00:48:12.710 --> 00:48:15.190 align:middle line:84% So first we'll use the Wikipedia article 00:48:15.190 --> 00:48:17.050 align:middle line:90% for 2022 Winter Olympics. 00:48:17.050 --> 00:48:19.810 align:middle line:84% And we tell it explicitly to make use of this context, 00:48:19.810 --> 00:48:22.670 align:middle line:84% because telling things explicitly always seems to help. 00:48:22.670 --> 00:48:25.070 align:middle line:84% So this is the thing we cut and pasted here, 00:48:25.070 --> 00:48:27.510 align:middle line:90% Wikipedia article on curling. 00:48:27.510 --> 00:48:29.490 align:middle line:90% And it's a pretty long article. 00:48:29.490 --> 00:48:31.470 align:middle line:90% It's got all kinds of stuff. 00:48:31.470 --> 00:48:34.620 align:middle line:84% And it's not even all that cleanly formatted. 00:48:34.620 --> 00:48:36.120 align:middle line:90% It's very strange. 00:48:36.120 --> 00:48:38.280 align:middle line:90% Look at that. 00:48:38.280 --> 00:48:39.780 align:middle line:84% So to answer your question, Spencer, 00:48:39.780 --> 00:48:43.380 align:middle line:84% it can be in pretty bad shape and it still seems to work. 00:48:43.380 --> 00:48:46.363 align:middle line:84% So now, "Use below article on the Olympics 00:48:46.363 --> 00:48:47.780 align:middle line:84% to answer the subsequent question. 00:48:47.780 --> 00:48:49.980 align:middle line:84% If you don't know, say you don't know." 00:48:49.980 --> 00:48:51.440 align:middle line:90% So that's what we have. 00:48:51.440 --> 00:48:52.460 align:middle line:90% There's the query. 00:48:52.460 --> 00:48:54.580 align:middle line:84% And by the way, before I send it into the LLM, 00:48:54.580 --> 00:48:56.240 align:middle line:84% this is the actual query that it's going to be sending. 00:48:56.240 --> 00:48:57.480 align:middle line:90% I'm printing out the query. 00:48:57.480 --> 00:48:59.080 align:middle line:90% Look at how long the query is. 00:48:59.080 --> 00:49:01.080 align:middle line:84% "Use the article below and here is the article," blah, blah, 00:49:01.080 --> 00:49:01.788 align:middle line:90% blah, blah, blah. 00:49:01.788 --> 00:49:03.000 align:middle line:90% Scroll, scroll, scroll. 00:49:03.000 --> 00:49:04.378 align:middle line:90% This is the whole thing. 00:49:04.378 --> 00:49:05.420 align:middle line:90% And it keeps on going on. 00:49:05.420 --> 00:49:08.080 align:middle line:84% And then finally I say, "Which teams won the gold?" 00:49:08.080 --> 00:49:09.540 align:middle line:90% OK, so let's run it. 00:49:09.540 --> 00:49:12.200 align:middle line:90% 00:49:12.200 --> 00:49:15.160 align:middle line:90% Look at that. 00:49:15.160 --> 00:49:17.600 align:middle line:84% Women's curling, Great Britain-- it got it right. 00:49:17.600 --> 00:49:19.280 align:middle line:90% Pretty good, right? 00:49:19.280 --> 00:49:22.760 align:middle line:84% It had to parse all that crap to get and find 00:49:22.760 --> 00:49:25.440 align:middle line:90% the nuggets, so nicely done. 00:49:25.440 --> 00:49:27.940 align:middle line:84% Now, maybe it wasn't super hard because we literally gave it 00:49:27.940 --> 00:49:28.660 align:middle line:90% the answer. 00:49:28.660 --> 00:49:30.080 align:middle line:90% So let's make it a bit harder. 00:49:30.080 --> 00:49:32.790 align:middle line:84% So I noticed that this person, Oskar Eriksson 00:49:32.790 --> 00:49:35.010 align:middle line:84% won two golds in the event, two medals in the event. 00:49:35.010 --> 00:49:39.210 align:middle line:84% So let's ask if any athlete won multiple medals. 00:49:39.210 --> 00:49:41.650 align:middle line:84% That requires a little bit of abstraction. 00:49:41.650 --> 00:49:45.130 align:middle line:84% So, all right, same query, "Did any athlete win multiple medals 00:49:45.130 --> 00:49:46.268 align:middle line:90% in curling?" 00:49:46.268 --> 00:49:47.310 align:middle line:90% The question has changed. 00:49:47.310 --> 00:49:49.010 align:middle line:90% Everything else hasn't changed. 00:49:49.010 --> 00:49:49.710 align:middle line:90% Hit it. 00:49:49.710 --> 00:49:51.810 align:middle line:90% Let's see what happens. 00:49:51.810 --> 00:49:55.193 align:middle line:84% "Yes, Oskar Eriksson won multiple medals in curling. 00:49:55.193 --> 00:49:57.610 align:middle line:84% He won a gold in the men's event and a bronze in the mixed 00:49:57.610 --> 00:49:58.490 align:middle line:90% doubles." 00:49:58.490 --> 00:50:00.250 align:middle line:90% It's pretty cool. 00:50:00.250 --> 00:50:02.557 align:middle line:90% Take that Google. 00:50:02.557 --> 00:50:04.890 align:middle line:84% All right, now we come to retrieval augmented generation 00:50:04.890 --> 00:50:06.290 align:middle line:84% where instead of doing it manually, obviously, 00:50:06.290 --> 00:50:08.767 align:middle line:84% because it doesn't scale, we will do it automatically. 00:50:08.767 --> 00:50:10.350 align:middle line:84% And so the thing you have to remember, 00:50:10.350 --> 00:50:11.933 align:middle line:84% as I mentioned just a few minutes ago, 00:50:11.933 --> 00:50:14.490 align:middle line:84% is that there is a context window for every LLM. 00:50:14.490 --> 00:50:20.450 align:middle line:84% And for GPT-4 and 3.5-turbo, the context window is 16,385 tokens. 00:50:20.450 --> 00:50:23.930 align:middle line:84% That is the length of the input and the output. 00:50:23.930 --> 00:50:25.430 align:middle line:90% So we can't exceed that. 00:50:25.430 --> 00:50:27.870 align:middle line:84% By the way, GPT-4's context window is, I think, 00:50:27.870 --> 00:50:30.150 align:middle line:90% up to 128,000 tokens. 00:50:30.150 --> 00:50:34.022 align:middle line:90% And Google Gemini 1.5 Pro-- 00:50:34.022 --> 00:50:35.730 align:middle line:84% they really need to work on their names-- 00:50:35.730 --> 00:50:38.620 align:middle line:84% Google Gemini 1.5 Pro, the context window 00:50:38.620 --> 00:50:41.210 align:middle line:90% is 1 million tokens. 00:50:41.210 --> 00:50:44.590 align:middle line:84% And in research, they have tested 10 million tokens, 00:50:44.590 --> 00:50:46.145 align:middle line:90% so crazy times. 00:50:46.145 --> 00:50:48.270 align:middle line:84% All that means is that you can upload entire videos 00:50:48.270 --> 00:50:50.673 align:middle line:84% and ask it questions about the video. 00:50:50.673 --> 00:50:52.590 align:middle line:84% All right, to come back to this, what we'll do 00:50:52.590 --> 00:50:56.850 align:middle line:84% is we'll only grab the data from the Wikipedia articles, all 00:50:56.850 --> 00:50:58.350 align:middle line:84% the articles about the Olympics that 00:50:58.350 --> 00:51:01.390 align:middle line:84% are relevant to our question by using pre-trained embeddings. 00:51:01.390 --> 00:51:04.130 align:middle line:84% So again, this is the thing we talked about earlier. 00:51:04.130 --> 00:51:05.870 align:middle line:84% This is the picture we saw in class. 00:51:05.870 --> 00:51:07.750 align:middle line:84% And the only thing I want to point out 00:51:07.750 --> 00:51:10.390 align:middle line:84% is that if you have a particular embedding for a question 00:51:10.390 --> 00:51:12.430 align:middle line:84% and a particular embedding for a chunk of text 00:51:12.430 --> 00:51:14.110 align:middle line:84% that you have in your database, you 00:51:14.110 --> 00:51:17.230 align:middle line:84% have to figure out how similar, how related they are. 00:51:17.230 --> 00:51:20.806 align:middle line:90% And for that, we can use what? 00:51:20.806 --> 00:51:26.110 align:middle line:84% Dot product or something almost as dot 00:51:26.110 --> 00:51:28.440 align:middle line:84% product, which is more easier for us to work with-- 00:51:28.440 --> 00:51:30.630 align:middle line:90% the cosine similarity. 00:51:30.630 --> 00:51:32.380 align:middle line:84% We have done cosine similarity previously. 00:51:32.380 --> 00:51:33.340 align:middle line:90% I've explained it in class. 00:51:33.340 --> 00:51:34.780 align:middle line:84% We're just going to use cosine similarity-- 00:51:34.780 --> 00:51:36.360 align:middle line:90% how similar are these vectors? 00:51:36.360 --> 00:51:39.340 align:middle line:84% So that's what we're going to do. 00:51:39.340 --> 00:51:41.480 align:middle line:84% So the same picture as we saw in class. 00:51:41.480 --> 00:51:43.080 align:middle line:84% So first, what we'll do is we need 00:51:43.080 --> 00:51:44.970 align:middle line:84% to break up the dataset into sections 00:51:44.970 --> 00:51:46.720 align:middle line:84% and then take each section and then run it 00:51:46.720 --> 00:51:47.887 align:middle line:90% through the embedding thing. 00:51:47.887 --> 00:51:50.033 align:middle line:84% But fortunately for us, I have code here 00:51:50.033 --> 00:51:51.700 align:middle line:84% which actually does it for you manually. 00:51:51.700 --> 00:51:53.117 align:middle line:84% You can play around with it later. 00:51:53.117 --> 00:51:56.292 align:middle line:84% But OpenAI has already given us the chunked dataset. 00:51:56.292 --> 00:51:58.500 align:middle line:84% So we'll just use that because it's just easy for us. 00:51:58.500 --> 00:52:00.875 align:middle line:84% And I downloaded it already because it takes five minutes 00:52:00.875 --> 00:52:01.540 align:middle line:90% to download. 00:52:01.540 --> 00:52:03.373 align:middle line:84% I've downloaded this thing and I've stuck it 00:52:03.373 --> 00:52:05.100 align:middle line:90% in a particular dataframe here. 00:52:05.100 --> 00:52:08.280 align:middle line:84% So let's print out five randomly chosen chunks. 00:52:08.280 --> 00:52:11.680 align:middle line:84% So you can see here, this is the first chunk, 00:52:11.680 --> 00:52:13.900 align:middle line:90% somebody else, somebody else. 00:52:13.900 --> 00:52:16.940 align:middle line:84% And look at all this crazy stuff here. 00:52:16.940 --> 00:52:18.180 align:middle line:90% The formatting is off. 00:52:18.180 --> 00:52:20.880 align:middle line:84% But these are all, basically, paragraphs and sections 00:52:20.880 --> 00:52:23.130 align:middle line:84% just grabbed straight from Wikipedia with no cleaning. 00:52:23.130 --> 00:52:25.760 align:middle line:90% 00:52:25.760 --> 00:52:29.780 align:middle line:84% Now, we define a simple function to basically send 00:52:29.780 --> 00:52:32.820 align:middle line:84% in any arbitrary piece of text into the embedding model 00:52:32.820 --> 00:52:35.560 align:middle line:84% and get the contextual embedding vector out. 00:52:35.560 --> 00:52:38.387 align:middle line:84% And there is this little function that does that. 00:52:38.387 --> 00:52:39.720 align:middle line:90% We are using an embedding model. 00:52:39.720 --> 00:52:41.560 align:middle line:84% We send in text, it gives you something. 00:52:41.560 --> 00:52:42.740 align:middle line:90% So let's try it on that. 00:52:42.740 --> 00:52:44.140 align:middle line:90% "HODL is amazing!" 00:52:44.140 --> 00:52:45.440 align:middle line:90% You should get a vector back. 00:52:45.440 --> 00:52:51.220 align:middle line:90% 00:52:51.220 --> 00:52:52.300 align:middle line:90% Oh, come on. 00:52:52.300 --> 00:52:55.092 align:middle line:90% Don't fail me now. 00:52:55.092 --> 00:52:57.060 align:middle line:90% Phew, all right. 00:52:57.060 --> 00:52:58.380 align:middle line:90% How long is it? 00:52:58.380 --> 00:53:00.260 align:middle line:90% 1536. 00:53:00.260 --> 00:53:02.440 align:middle line:84% So how about I say, HODL is incredible, 00:53:02.440 --> 00:53:03.600 align:middle line:90% like HODL is amazing? 00:53:03.600 --> 00:53:05.350 align:middle line:84% Hopefully, the two vectors will be kind of 00:53:05.350 --> 00:53:08.620 align:middle line:90% similar in terms of cosine. 00:53:08.620 --> 00:53:11.060 align:middle line:84% And so to calculate the cosine distance, 00:53:11.060 --> 00:53:13.420 align:middle line:84% I use this particular function from SciPy. 00:53:13.420 --> 00:53:15.740 align:middle line:84% It just calculates the cosine similarity. 00:53:15.740 --> 00:53:18.700 align:middle line:90% And I hit it, so 0.9934. 00:53:18.700 --> 00:53:20.200 align:middle line:90% Maximum is 1. 00:53:20.200 --> 00:53:22.560 align:middle line:84% So 0.934 means that they're very, very similar, 00:53:22.560 --> 00:53:25.150 align:middle line:84% which is comforting because amazing and incredible 00:53:25.150 --> 00:53:27.590 align:middle line:90% are obviously synonyms. 00:53:27.590 --> 00:53:31.022 align:middle line:84% So now given a dataframe with a column of text chunks in it, 00:53:31.022 --> 00:53:33.230 align:middle line:84% we can use this function on every one of these things 00:53:33.230 --> 00:53:34.355 align:middle line:90% to calculate the embedding. 00:53:34.355 --> 00:53:36.970 align:middle line:84% And you have a function here that basically does it for you. 00:53:36.970 --> 00:53:39.610 align:middle line:84% I'm not going to run it because it takes a long time. 00:53:39.610 --> 00:53:41.190 align:middle line:90% But you can run it later on. 00:53:41.190 --> 00:53:41.950 align:middle line:90% Just be prepared. 00:53:41.950 --> 00:53:44.670 align:middle line:84% Go get a cup of coffee and stuff while it does it. 00:53:44.670 --> 00:53:47.110 align:middle line:84% But happily for us, OpenAI has actually 00:53:47.110 --> 00:53:49.497 align:middle line:84% already done this step for us, so we don't have to. 00:53:49.497 --> 00:53:51.330 align:middle line:84% So it's already available in this dataframe. 00:53:51.330 --> 00:53:54.613 align:middle line:84% So if you actually look at this, you can see here 00:53:54.613 --> 00:53:56.030 align:middle line:84% there is a text, and then there is 00:53:56.030 --> 00:53:59.750 align:middle line:84% an embedding that's sitting right there right next to it. 00:53:59.750 --> 00:54:02.430 align:middle line:84% And these embeddings are whatever-- 00:54:02.430 --> 00:54:03.170 align:middle line:90% how long is it? 00:54:03.170 --> 00:54:09.650 align:middle line:84% 1536 long, 1536 long vectors All right, so that's what we have. 00:54:09.650 --> 00:54:14.390 align:middle line:90% 00:54:14.390 --> 00:54:17.830 align:middle line:84% So now that we have this thing, whenever we get a question, 00:54:17.830 --> 00:54:19.890 align:middle line:84% we calculate the question's embedding 00:54:19.890 --> 00:54:22.310 align:middle line:84% and then compare calculated cosine similarity 00:54:22.310 --> 00:54:25.240 align:middle line:84% with all the embeddings sitting in this dataframe. 00:54:25.240 --> 00:54:27.960 align:middle line:84% So to do that we're going to define a couple of helper 00:54:27.960 --> 00:54:29.000 align:middle line:90% functions here. 00:54:29.000 --> 00:54:31.200 align:middle line:84% You can read through the Python later to understand. 00:54:31.200 --> 00:54:35.000 align:middle line:84% It's just basic Python manipulations that are going on. 00:54:35.000 --> 00:54:37.240 align:middle line:84% And so let's just test this function. 00:54:37.240 --> 00:54:39.080 align:middle line:84% So basically, we have a little function 00:54:39.080 --> 00:54:42.080 align:middle line:84% called strings_ranked_by_relatedness 00:54:42.080 --> 00:54:45.660 align:middle line:84% where you give it any input question or text, 00:54:45.660 --> 00:54:48.360 align:middle line:84% and then it's going to give you the top five 00:54:48.360 --> 00:54:52.760 align:middle line:84% most related chunks of text that it had in its dataframe. 00:54:52.760 --> 00:54:55.870 align:middle line:90% So let me just run this thing. 00:54:55.870 --> 00:54:59.820 align:middle line:90% 00:54:59.820 --> 00:55:01.900 align:middle line:84% So curling, the things it pulls back, 00:55:01.900 --> 00:55:04.520 align:middle line:84% it better involve curling and medals and so on. 00:55:04.520 --> 00:55:08.040 align:middle line:84% So this one has a cosine similarity of 0.88. 00:55:08.040 --> 00:55:10.140 align:middle line:84% Curling at the 2012 Olympics, that's good. 00:55:10.140 --> 00:55:13.160 align:middle line:84% Results summary, medal summary, results summary-- 00:55:13.160 --> 00:55:14.080 align:middle line:90% It's all pretty good. 00:55:14.080 --> 00:55:15.960 align:middle line:84% Even the fifth one has a cosine similarity 00:55:15.960 --> 00:55:17.860 align:middle line:90% of 0.867, which is pretty high. 00:55:17.860 --> 00:55:19.180 align:middle line:90% So it's doing the right things. 00:55:19.180 --> 00:55:21.660 align:middle line:84% It's picked up curling gold medal with input text 00:55:21.660 --> 00:55:23.700 align:middle line:84% and it's picked up the right things from it. 00:55:23.700 --> 00:55:29.435 align:middle line:84% Now let's see what we can do with the original question. 00:55:29.435 --> 00:55:31.560 align:middle line:84% So here is a header I'm going to use in the prompt. 00:55:31.560 --> 00:55:33.300 align:middle line:84% I'm going to say, "Use the below articles 00:55:33.300 --> 00:55:34.847 align:middle line:84% to answer the subsequent question. 00:55:34.847 --> 00:55:36.680 align:middle line:84% Answer the questions truthfully as possible. 00:55:36.680 --> 00:55:38.160 align:middle line:84% And if you're unsure of the answer, say I'm sorry, 00:55:38.160 --> 00:55:39.240 align:middle line:90% I don't know," as before. 00:55:39.240 --> 00:55:41.220 align:middle line:90% OK, that's our prompt. 00:55:41.220 --> 00:55:42.360 align:middle line:90% And now here is the thing. 00:55:42.360 --> 00:55:44.500 align:middle line:84% We don't want to exceed the context window. 00:55:44.500 --> 00:55:47.078 align:middle line:84% So we need to count the tokens we are sending in 00:55:47.078 --> 00:55:48.620 align:middle line:84% and the likely number of tokens we're 00:55:48.620 --> 00:55:50.860 align:middle line:84% going to get back so that we don't exceed the budget. 00:55:50.860 --> 00:55:54.740 align:middle line:84% So we use this package called tiktoken package for this. 00:55:54.740 --> 00:55:56.830 align:middle line:84% And then it just helps you count the tokens. 00:55:56.830 --> 00:55:58.080 align:middle line:90% And you can read through this. 00:55:58.080 --> 00:56:00.780 align:middle line:84% It's just again, some basic Python for counting tokens. 00:56:00.780 --> 00:56:04.820 align:middle line:84% And now what we do is, this is where we actually 00:56:04.820 --> 00:56:06.180 align:middle line:90% assemble the prompt. 00:56:06.180 --> 00:56:08.320 align:middle line:90% We start with the header. 00:56:08.320 --> 00:56:11.040 align:middle line:84% We have the header, which says "be truthful" and all that. 00:56:11.040 --> 00:56:15.480 align:middle line:84% Then we say, here is a question that I'm going to ask you. 00:56:15.480 --> 00:56:19.430 align:middle line:84% And then you go in there and keep grabbing Wikipedia articles 00:56:19.430 --> 00:56:22.190 align:middle line:84% till the number of tokens in your prompt 00:56:22.190 --> 00:56:26.105 align:middle line:84% is exceeding your token budget, and then you stop. 00:56:26.105 --> 00:56:27.730 align:middle line:84% When you're about to exceed the budget, 00:56:27.730 --> 00:56:29.910 align:middle line:84% you stop because you can't exceed the budget. 00:56:29.910 --> 00:56:32.190 align:middle line:90% And that's the whole thing. 00:56:32.190 --> 00:56:38.470 align:middle line:84% So here, let's just do tiktoken, run this function. 00:56:38.470 --> 00:56:42.470 align:middle line:84% Now, it turns out, as you saw, we can go up to 1,600 something 00:56:42.470 --> 00:56:43.930 align:middle line:90% tokens in the context window. 00:56:43.930 --> 00:56:47.750 align:middle line:84% I'm just using three 3,700 as my budget, 00:56:47.750 --> 00:56:50.630 align:middle line:84% partly just to show you how to use this thing and also 00:56:50.630 --> 00:56:54.550 align:middle line:84% because it's charging my credit card for every token that 00:56:54.550 --> 00:56:55.630 align:middle line:90% I'm using. 00:56:55.630 --> 00:56:58.070 align:middle line:90% So I'm just being careful. 00:56:58.070 --> 00:56:59.277 align:middle line:90% It charges by the token. 00:56:59.277 --> 00:57:00.610 align:middle line:90% It's a beautiful business model. 00:57:00.610 --> 00:57:02.338 align:middle line:90% Anyway, so back here. 00:57:02.338 --> 00:57:04.630 align:middle line:84% So let's ask the question, "Which athletes won the gold 00:57:04.630 --> 00:57:06.470 align:middle line:84% medal in curling at the 2022 Winter Olympics?" 00:57:06.470 --> 00:57:08.290 align:middle line:84% Here is the dataframe that you should use. 00:57:08.290 --> 00:57:09.490 align:middle line:90% Here is the GPT model. 00:57:09.490 --> 00:57:11.750 align:middle line:90% And don't exceed 3,700 tokens. 00:57:11.750 --> 00:57:14.550 align:middle line:90% That's the query or the prompt. 00:57:14.550 --> 00:57:16.370 align:middle line:84% It's going to compose the prompt now. 00:57:16.370 --> 00:57:18.730 align:middle line:90% And this is the whole prompt. 00:57:18.730 --> 00:57:19.980 align:middle line:90% Let's just go to the very top. 00:57:19.980 --> 00:57:20.900 align:middle line:90% It's really long. 00:57:20.900 --> 00:57:24.527 align:middle line:90% 00:57:24.527 --> 00:57:26.360 align:middle line:84% All right, "Use the below articles to answer 00:57:26.360 --> 00:57:27.985 align:middle line:84% the subsequent question-- as possible." 00:57:27.985 --> 00:57:30.720 align:middle line:84% And boom, boom, boom, boom, boom, it has all these things. 00:57:30.720 --> 00:57:32.800 align:middle line:84% It's added a whole bunch of paragraphs 00:57:32.800 --> 00:57:34.800 align:middle line:90% from the Wikipedia pages. 00:57:34.800 --> 00:57:36.700 align:middle line:84% And then it finally ends with the question, 00:57:36.700 --> 00:57:38.520 align:middle line:90% "which athletes won the gold?" 00:57:38.520 --> 00:57:40.960 align:middle line:84% All right, now let's just ask it the thing. 00:57:40.960 --> 00:57:43.060 align:middle line:84% And this is just a little function 00:57:43.060 --> 00:57:45.200 align:middle line:90% to send stuff into the API. 00:57:45.200 --> 00:57:49.960 align:middle line:84% And now we are finally ready to ask GPT the question. 00:57:49.960 --> 00:57:50.670 align:middle line:90% Fingers crossed. 00:57:50.670 --> 00:57:53.200 align:middle line:90% 00:57:53.200 --> 00:57:57.440 align:middle line:84% All right, curling where, Stefania, curling in the mixed 00:57:57.440 --> 00:57:58.240 align:middle line:90% doubles. 00:57:58.240 --> 00:58:00.840 align:middle line:84% And the team consisting of blah, blah, blah, 00:58:00.840 --> 00:58:03.600 align:middle line:90% in the men's tournament. 00:58:03.600 --> 00:58:04.680 align:middle line:90% Oh, interesting. 00:58:04.680 --> 00:58:07.280 align:middle line:84% It has actually ignored the Great Britain people completely, 00:58:07.280 --> 00:58:07.780 align:middle line:90% I think. 00:58:07.780 --> 00:58:10.464 align:middle line:90% 00:58:10.464 --> 00:58:12.080 align:middle line:90% Last night it didn't. 00:58:12.080 --> 00:58:14.930 align:middle line:90% Welcome to stochasticity. 00:58:14.930 --> 00:58:15.680 align:middle line:90% So you can try it. 00:58:15.680 --> 00:58:18.740 align:middle line:84% When you try it, it will actually give you the thing. 00:58:18.740 --> 00:58:20.740 align:middle line:84% And so let's ask it now a question 00:58:20.740 --> 00:58:23.960 align:middle line:84% about the 2016 Winter Olympics, which, by the way, 00:58:23.960 --> 00:58:25.220 align:middle line:90% didn't happen. 00:58:25.220 --> 00:58:27.300 align:middle line:84% There were no Winter Olympics in 2016. 00:58:27.300 --> 00:58:31.260 align:middle line:90% So if you ask it-- 00:58:31.260 --> 00:58:32.300 align:middle line:90% "Sorry, I don't know." 00:58:32.300 --> 00:58:35.740 align:middle line:84% All right, now let's change the header so 00:58:35.740 --> 00:58:37.780 align:middle line:90% that we don't say be truthful. 00:58:37.780 --> 00:58:39.180 align:middle line:84% So we will remove the need for it 00:58:39.180 --> 00:58:41.010 align:middle line:84% to be truthful and see what happens. 00:58:41.010 --> 00:58:44.700 align:middle line:90% 00:58:44.700 --> 00:58:45.990 align:middle line:90% "Which athletes won the gold?" 00:58:45.990 --> 00:58:51.020 align:middle line:90% 00:58:51.020 --> 00:58:54.220 align:middle line:84% Oh, now it's telling you about the 2022 Olympics. 00:58:54.220 --> 00:58:57.580 align:middle line:84% So it answered an irrelevant question accurately 00:58:57.580 --> 00:59:00.660 align:middle line:84% if you removed the need for it to be truthful. 00:59:00.660 --> 00:59:04.940 align:middle line:84% So I guess the moral of the story is that, first of all, 00:59:04.940 --> 00:59:07.915 align:middle line:84% you can use RAG to grab stuff from vast databases, 00:59:07.915 --> 00:59:10.040 align:middle line:84% and it's very heavily used in industry, number one. 00:59:10.040 --> 00:59:12.700 align:middle line:84% Number two, you have to be careful about these token 00:59:12.700 --> 00:59:14.850 align:middle line:90% budgets and so on and so forth. 00:59:14.850 --> 00:59:17.230 align:middle line:84% And small wording changes in the prompt 00:59:17.230 --> 00:59:19.350 align:middle line:84% can actually dramatically alter behavior, 00:59:19.350 --> 00:59:21.550 align:middle line:84% which makes it very difficult in enterprise settings 00:59:21.550 --> 00:59:23.550 align:middle line:90% to do QA on this stuff. 00:59:23.550 --> 00:59:27.430 align:middle line:84% So a lot of care has to go into it. 00:59:27.430 --> 00:59:29.695 align:middle line:84% And you have seen examples of, for example, Air Canada 00:59:29.695 --> 00:59:31.070 align:middle line:84% had a chatbot which actually gave 00:59:31.070 --> 00:59:32.370 align:middle line:90% the wrong advice to a customer. 00:59:32.370 --> 00:59:33.750 align:middle line:90% The customer sued Air Canada. 00:59:33.750 --> 00:59:36.288 align:middle line:84% And then the court ruled in favor of the passenger. 00:59:36.288 --> 00:59:38.330 align:middle line:84% And then they pulled the chatbot off the website. 00:59:38.330 --> 00:59:39.970 align:middle line:84% So you've got to be very careful. 00:59:39.970 --> 00:59:42.530 align:middle line:84% I think without a human in the loop checking these answers, 00:59:42.530 --> 00:59:45.072 align:middle line:84% it's kind of dangerous, in my opinion, at this current state. 00:59:45.072 --> 00:59:46.990 align:middle line:90% Hopefully, it will get better. 00:59:46.990 --> 00:59:50.170 align:middle line:84% So there's a lot of potential in it, but you have to be careful. 00:59:50.170 --> 00:59:52.270 align:middle line:84% All right, so this is what we have. 00:59:52.270 --> 00:59:55.390 align:middle line:84% And you can actually take this thing here and use it. 00:59:55.390 --> 00:59:58.590 align:middle line:84% You can actually take a 1,000-page PDF that you might 00:59:58.590 --> 01:00:01.530 align:middle line:84% have or something and then chunk it and use this approach. 01:00:01.530 --> 01:00:03.130 align:middle line:84% And I've done it for a whole bunch of different things. 01:00:03.130 --> 01:00:05.010 align:middle line:84% It actually works really well most of the time. 01:00:05.010 --> 01:00:06.468 align:middle line:84% It will make errors here and there. 01:00:06.468 --> 01:00:09.950 align:middle line:84% Most of the time it actually works really well. 01:00:09.950 --> 01:00:11.830 align:middle line:90% Yeah? 01:00:11.830 --> 01:00:14.360 align:middle line:90% AUDIENCE: Just a quick question. 01:00:14.360 --> 01:00:18.000 align:middle line:84% GPT-4 now lets you upload a PDF and it remains locked. 01:00:18.000 --> 01:00:19.000 align:middle line:90% Is it chunking that? 01:00:19.000 --> 01:00:21.220 align:middle line:84% Or is it actually ingesting all the-- 01:00:21.220 --> 01:00:23.262 align:middle line:84% RAMA RAMAKRISHNAN: No, when you upload something, 01:00:23.262 --> 01:00:25.580 align:middle line:84% because GPT-4-Turbo has 128,000 tokens, 01:00:25.580 --> 01:00:28.390 align:middle line:84% which means it can accommodate a whole, long bunch of documents. 01:00:28.390 --> 01:00:30.640 align:middle line:84% So when you upload stuff, it's not doing any chunking. 01:00:30.640 --> 01:00:32.760 align:middle line:84% The chunking we are talking about, you have to do. 01:00:32.760 --> 01:00:34.620 align:middle line:84% The LLM doesn't even know you're doing it. 01:00:34.620 --> 01:00:36.120 align:middle line:84% As far as the LLM is concerned, it's 01:00:36.120 --> 01:00:37.527 align:middle line:90% only seeing the prompt it sees. 01:00:37.527 --> 01:00:39.860 align:middle line:84% And the prompt says, hey, here's a bunch of information, 01:00:39.860 --> 01:00:40.420 align:middle line:90% here's a question. 01:00:40.420 --> 01:00:42.628 align:middle line:84% Answer it for me using this question and be truthful. 01:00:42.628 --> 01:00:44.120 align:middle line:90% That's it. 01:00:44.120 --> 01:00:47.820 align:middle line:84% Now when you ask these things a question, 01:00:47.820 --> 01:00:49.900 align:middle line:84% which is later than its training data, 01:00:49.900 --> 01:00:53.320 align:middle line:84% you will actually see GPT-4 saying, "doing a Bing search" 01:00:53.320 --> 01:00:54.263 align:middle line:90% and things like that. 01:00:54.263 --> 01:00:55.680 align:middle line:84% There, what's actually going on is 01:00:55.680 --> 01:00:59.080 align:middle line:84% there's a pre-processing step and a program which 01:00:59.080 --> 01:01:01.520 align:middle line:84% is doing a Bing search, gathering a bunch of Bing 01:01:01.520 --> 01:01:04.080 align:middle line:84% results, taking the top few results, 01:01:04.080 --> 01:01:07.220 align:middle line:84% chunking, embedding, backing into a prompt, 01:01:07.220 --> 01:01:08.620 align:middle line:90% sending it into GPT-4. 01:01:08.620 --> 01:01:11.220 align:middle line:84% And you don't know all this is going on under the hood. 01:01:11.220 --> 01:01:13.160 align:middle line:84% So when it's actually thinking and saying "Bing search," 01:01:13.160 --> 01:01:14.785 align:middle line:84% this is what's going on under the hood. 01:01:14.785 --> 01:01:19.180 align:middle line:90% 01:01:19.180 --> 01:01:21.640 align:middle line:84% Was there a question somewhere here? 01:01:21.640 --> 01:01:22.340 align:middle line:90% No. 01:01:22.340 --> 01:01:23.040 align:middle line:90% Oh, sorry. 01:01:23.040 --> 01:01:24.275 align:middle line:90% Yeah? 01:01:24.275 --> 01:01:26.150 align:middle line:84% AUDIENCE: I have a question about formatting. 01:01:26.150 --> 01:01:27.380 align:middle line:90% RAMA RAMAKRISHNAN: Yeah? 01:01:27.380 --> 01:01:30.980 align:middle line:84% AUDIENCE: So it seems to be able to understand and ignore 01:01:30.980 --> 01:01:35.060 align:middle line:84% irrelevant formatting, even when there's colloquial tables not 01:01:35.060 --> 01:01:36.480 align:middle line:90% clearly defined tables. 01:01:36.480 --> 01:01:39.120 align:middle line:84% And also when it outputs formats, 01:01:39.120 --> 01:01:42.020 align:middle line:84% it's able to do it really human-ly. 01:01:42.020 --> 01:01:44.820 align:middle line:84% Is that something that it's figuring out 01:01:44.820 --> 01:01:45.945 align:middle line:90% through the neural network? 01:01:45.945 --> 01:01:47.487 align:middle line:84% Or is it something that's going to be 01:01:47.487 --> 01:01:49.820 align:middle line:84% programmed in the header somewhere with the standard-- 01:01:49.820 --> 01:01:51.600 align:middle line:84% RAMA RAMAKRISHNAN: There is no explicit programming going on. 01:01:51.600 --> 01:01:54.380 align:middle line:84% It's typically because a lot of the question-answer pairs that 01:01:54.380 --> 01:01:57.260 align:middle line:84% was used for supervised fine-tuning and reinforcement 01:01:57.260 --> 01:02:01.980 align:middle line:84% learning, the better answers with the same badly-formatted 01:02:01.980 --> 01:02:03.920 align:middle line:84% input, the better answers were just rewarded, 01:02:03.920 --> 01:02:04.840 align:middle line:90% were ranked higher. 01:02:04.840 --> 01:02:06.020 align:middle line:90% That's what's going on. 01:02:06.020 --> 01:02:09.230 align:middle line:84% But on a related note, one thing that's very useful 01:02:09.230 --> 01:02:12.910 align:middle line:84% is that you can actually ask it to give you the answer back 01:02:12.910 --> 01:02:15.750 align:middle line:84% using certain formats, like markdown and JSON and things 01:02:15.750 --> 01:02:16.630 align:middle line:90% like that. 01:02:16.630 --> 01:02:20.072 align:middle line:84% And by forcing it to adhere to certain well-defined formats, 01:02:20.072 --> 01:02:22.030 align:middle line:84% you actually increase the chance of it actually 01:02:22.030 --> 01:02:24.990 align:middle line:84% getting the right answer in the first place. 01:02:24.990 --> 01:02:27.610 align:middle line:84% Again, there's a whole tangent here we can go into. 01:02:27.610 --> 01:02:29.950 align:middle line:84% But those are some of the things that 01:02:29.950 --> 01:02:31.630 align:middle line:90% are part of prompt engineering. 01:02:31.630 --> 01:02:33.690 align:middle line:84% All right, so that's what we have here. 01:02:33.690 --> 01:02:34.930 align:middle line:90% Back to the PowerPoint. 01:02:34.930 --> 01:02:40.630 align:middle line:90% 01:02:40.630 --> 01:02:42.610 align:middle line:84% So that's retrieval augmented generation. 01:02:42.610 --> 01:02:44.710 align:middle line:84% And we finally come to fine-tuning. 01:02:44.710 --> 01:02:48.030 align:middle line:90% So fine-tuning is when-- 01:02:48.030 --> 01:02:50.750 align:middle line:84% up to this point, all the things we have seen 01:02:50.750 --> 01:02:53.318 align:middle line:84% don't alter the internals of the LLM. 01:02:53.318 --> 01:02:55.110 align:middle line:84% You have not messed around with the weights 01:02:55.110 --> 01:02:56.130 align:middle line:90% or changed them at all. 01:02:56.130 --> 01:02:58.576 align:middle line:84% You're just using it as a black box. 01:02:58.576 --> 01:03:01.383 align:middle line:84% With fine-tuning, you actually will train it further, 01:03:01.383 --> 01:03:03.050 align:middle line:84% meaning the weights are going to change. 01:03:03.050 --> 01:03:05.710 align:middle line:90% 01:03:05.710 --> 01:03:09.430 align:middle line:84% Now, remember, we take something like a causal LLM like GPT-- 01:03:09.430 --> 01:03:12.063 align:middle line:90% 01:03:12.063 --> 01:03:13.930 align:middle line:84% and this, I haven't fixed this yet. 01:03:13.930 --> 01:03:17.210 align:middle line:84% There is no ReLU here, as I mentioned earlier. 01:03:17.210 --> 01:03:19.210 align:middle line:90% Just remember that. 01:03:19.210 --> 01:03:22.050 align:middle line:84% And then if you have domain-specific input-output 01:03:22.050 --> 01:03:23.770 align:middle line:84% examples, like input and output, you 01:03:23.770 --> 01:03:26.550 align:middle line:84% can just train it like this, input 01:03:26.550 --> 01:03:28.850 align:middle line:90% and then the shifted output. 01:03:28.850 --> 01:03:32.503 align:middle line:84% And that will update these weights, all these weights. 01:03:32.503 --> 01:03:34.170 align:middle line:84% So this is basically fine-tuning exactly 01:03:34.170 --> 01:03:36.650 align:middle line:90% like we saw with BERT and so on. 01:03:36.650 --> 01:03:39.590 align:middle line:84% And even with ResNet, it's the same sort of thing. 01:03:39.590 --> 01:03:41.290 align:middle line:90% That is fine-tuning. 01:03:41.290 --> 01:03:43.590 align:middle line:84% Now, before we discuss the mechanics of how to do it, 01:03:43.590 --> 01:03:46.370 align:middle line:84% I want to show you a quick example of the usefulness 01:03:46.370 --> 01:03:48.610 align:middle line:90% of fine-tuning. 01:03:48.610 --> 01:03:50.210 align:middle line:84% Imagine for a second that we want 01:03:50.210 --> 01:03:53.690 align:middle line:84% to generate synthetic product reviews from product 01:03:53.690 --> 01:03:55.210 align:middle line:90% descriptions. 01:03:55.210 --> 01:03:58.570 align:middle line:84% So we are building some product which can simulate customer 01:03:58.570 --> 01:04:00.610 align:middle line:90% behavior in e-commerce. 01:04:00.610 --> 01:04:03.770 align:middle line:84% And for that, we need to be able to generate the kinds of reviews 01:04:03.770 --> 01:04:05.740 align:middle line:84% that customers might come up with. 01:04:05.740 --> 01:04:08.162 align:middle line:84% And writing a lot of reviews is very time consuming. 01:04:08.162 --> 01:04:10.620 align:middle line:84% But what you can do is you can get a whole bunch of product 01:04:10.620 --> 01:04:13.240 align:middle line:90% descriptions from the internet. 01:04:13.240 --> 01:04:16.300 align:middle line:84% So let's say you ask an LLM, hey, write a product 01:04:16.300 --> 01:04:19.000 align:middle line:84% review using this information here, product description here. 01:04:19.000 --> 01:04:20.380 align:middle line:90% And it comes up with this-- 01:04:20.380 --> 01:04:20.980 align:middle line:90% "Timeless. 01:04:20.980 --> 01:04:21.540 align:middle line:90% Authentic. 01:04:21.540 --> 01:04:24.100 align:middle line:90% Iconic." 01:04:24.100 --> 01:04:27.460 align:middle line:84% Seriously, do product reviewers actually write stuff like this? 01:04:27.460 --> 01:04:28.140 align:middle line:90% No. 01:04:28.140 --> 01:04:30.700 align:middle line:90% This looks like marketing copy. 01:04:30.700 --> 01:04:33.020 align:middle line:84% This reads like marketing copy because there's 01:04:33.020 --> 01:04:35.380 align:middle line:84% a whole bunch of marketing copy on the internet. 01:04:35.380 --> 01:04:36.430 align:middle line:90% So it's not good. 01:04:36.430 --> 01:04:37.680 align:middle line:90% It doesn't feel like a review. 01:04:37.680 --> 01:04:39.540 align:middle line:90% It's not authentic. 01:04:39.540 --> 01:04:42.160 align:middle line:84% Here's another example for Urban Outfitters. 01:04:42.160 --> 01:04:46.580 align:middle line:84% And it says, "The boxy and cropped silhouette is flattering 01:04:46.580 --> 01:04:47.720 align:middle line:90% on all body types." 01:04:47.720 --> 01:04:48.320 align:middle line:90% Come on! 01:04:48.320 --> 01:04:50.620 align:middle line:90% [LAUGHTER] 01:04:50.620 --> 01:04:52.740 align:middle line:90% So it's not going to work. 01:04:52.740 --> 01:04:55.300 align:middle line:84% So what we do is we fine-tune the LLM. 01:04:55.300 --> 01:04:57.780 align:middle line:84% We can take an LLM and we can fine-tune it 01:04:57.780 --> 01:05:01.060 align:middle line:84% with instruction, product description, and product review 01:05:01.060 --> 01:05:03.420 align:middle line:90% examples. 01:05:03.420 --> 01:05:04.590 align:middle line:90% That's what you can do. 01:05:04.590 --> 01:05:07.790 align:middle line:84% So for instance, we can take something like this. 01:05:07.790 --> 01:05:09.170 align:middle line:90% Let me zoom into this thing. 01:05:09.170 --> 01:05:14.510 align:middle line:90% 01:05:14.510 --> 01:05:17.110 align:middle line:84% So it says here, "Write a positive review 01:05:17.110 --> 01:05:18.410 align:middle line:90% for the following product." 01:05:18.410 --> 01:05:20.870 align:middle line:84% And then this is the description, 01:05:20.870 --> 01:05:22.990 align:middle line:84% is the input and the output is, "The best! 01:05:22.990 --> 01:05:24.770 align:middle line:84% Carhart is my husband's favorite. 01:05:24.770 --> 01:05:26.190 align:middle line:90% They fit well." 01:05:26.190 --> 01:05:27.610 align:middle line:90% They feel like product reviews. 01:05:27.610 --> 01:05:29.270 align:middle line:84% So you just have to get a few hundred 01:05:29.270 --> 01:05:33.150 align:middle line:84% of these product review examples, just a few hundred. 01:05:33.150 --> 01:05:34.730 align:middle line:84% And you may not even need that much. 01:05:34.730 --> 01:05:37.390 align:middle line:90% 01:05:37.390 --> 01:05:41.750 align:middle line:84% Once you do that, you just do fine-tuning, 01:05:41.750 --> 01:05:44.550 align:middle line:90% like I showed earlier-- 01:05:44.550 --> 01:05:45.790 align:middle line:90% instruction, input, output. 01:05:45.790 --> 01:05:47.790 align:middle line:84% And then you take that output and shift it a bit 01:05:47.790 --> 01:05:50.030 align:middle line:84% and make it the actual label, the actual output. 01:05:50.030 --> 01:05:52.530 align:middle line:84% Fine tune, fine tune, fine tune, fine tune a bunch of times. 01:05:52.530 --> 01:05:54.330 align:middle line:84% Gradient descent, weights get updated. 01:05:54.330 --> 01:05:56.470 align:middle line:84% Now you have a new and updated element. 01:05:56.470 --> 01:05:59.730 align:middle line:84% And when you do that, now for the same things, 01:05:59.730 --> 01:06:00.950 align:middle line:90% here's what you get. 01:06:00.950 --> 01:06:01.922 align:middle line:90% "Write a review. 01:06:01.922 --> 01:06:03.630 align:middle line:84% These are the best jeans I've ever owned. 01:06:03.630 --> 01:06:05.203 align:middle line:90% I am--" whatever, some details. 01:06:05.203 --> 01:06:06.870 align:middle line:84% "I've been wearing them for a few weeks. 01:06:06.870 --> 01:06:08.075 align:middle line:90% They still look brand new." 01:06:08.075 --> 01:06:08.950 align:middle line:90% It looks much better. 01:06:08.950 --> 01:06:11.010 align:middle line:90% It doesn't look like marketing. 01:06:11.010 --> 01:06:13.170 align:middle line:84% This is completely fake, by the way. 01:06:13.170 --> 01:06:15.190 align:middle line:84% The LLM came up with it after the fine-tuning. 01:06:15.190 --> 01:06:16.982 align:middle line:84% And then we say, "Write a horrible review," 01:06:16.982 --> 01:06:18.650 align:middle line:90% because we want to be balanced. 01:06:18.650 --> 01:06:20.653 align:middle line:84% "These are the worst jeans I've ever worn. 01:06:20.653 --> 01:06:22.070 align:middle line:84% They are too tight here and there. 01:06:22.070 --> 01:06:25.617 align:middle line:84% I'm going to return them and try a 30, but I'm not optimistic. 01:06:25.617 --> 01:06:26.950 align:middle line:90% I'm going to stick with Levi's." 01:06:26.950 --> 01:06:29.970 align:middle line:90% Phew, OK. 01:06:29.970 --> 01:06:31.510 align:middle line:90% So these read like real reviews. 01:06:31.510 --> 01:06:34.590 align:middle line:84% So just by taking a few hundred examples and fine-tuning it, 01:06:34.590 --> 01:06:36.130 align:middle line:84% it completely changes the behavior 01:06:36.130 --> 01:06:38.230 align:middle line:84% that you want for your particular use case. 01:06:38.230 --> 01:06:39.530 align:middle line:90% That's the key thing. 01:06:39.530 --> 01:06:41.890 align:middle line:84% So for me, the biggest benefit here 01:06:41.890 --> 01:06:45.130 align:middle line:84% is that while it took billions of sentences 01:06:45.130 --> 01:06:47.250 align:middle line:84% for pre-training the original LLM 01:06:47.250 --> 01:06:49.330 align:middle line:84% and then it took tens of thousands of examples 01:06:49.330 --> 01:06:54.850 align:middle line:84% to do supervised fine-tuning and RLHF and so on and so forth, 01:06:54.850 --> 01:06:57.590 align:middle line:84% to make it work for your narrow business use case, 01:06:57.590 --> 01:07:00.430 align:middle line:84% you only had to spend a couple of hundred examples. 01:07:00.430 --> 01:07:01.860 align:middle line:90% That's it. 01:07:01.860 --> 01:07:02.600 align:middle line:90% It's amazing. 01:07:02.600 --> 01:07:05.980 align:middle line:84% Imagine that if you had to collect like 30,000 examples 01:07:05.980 --> 01:07:06.503 align:middle line:90% to make it. 01:07:06.503 --> 01:07:07.920 align:middle line:84% Nobody's going to do these things. 01:07:07.920 --> 01:07:09.540 align:middle line:90% It's too much work. 01:07:09.540 --> 01:07:12.220 align:middle line:84% But a couple of hundred, anybody can do. 01:07:12.220 --> 01:07:14.860 align:middle line:84% That's why it's so powerful to fine tune these things. 01:07:14.860 --> 01:07:16.780 align:middle line:90% Yeah? 01:07:16.780 --> 01:07:19.192 align:middle line:84% AUDIENCE: You talked about being able to, 01:07:19.192 --> 01:07:23.060 align:middle line:84% in industries where you don't want to put some of this stuff 01:07:23.060 --> 01:07:26.380 align:middle line:84% on the internet, downloading the pre-trained model 01:07:26.380 --> 01:07:29.735 align:middle line:84% and being able to do this on your own. 01:07:29.735 --> 01:07:31.860 align:middle line:84% Talking about computer power, some of the computers 01:07:31.860 --> 01:07:35.380 align:middle line:84% we have now, the GPUs, I don't know how powerful they are. 01:07:35.380 --> 01:07:38.340 align:middle line:84% Are you able to do some of these very small use cases 01:07:38.340 --> 01:07:40.460 align:middle line:90% on those types of devices? 01:07:40.460 --> 01:07:42.215 align:middle line:84% RAMA RAMAKRISHNAN: Perfect question, Mike. 01:07:42.215 --> 01:07:43.340 align:middle line:90% We're going to get to that. 01:07:43.340 --> 01:07:45.623 align:middle line:84% Because the short answer, it's hard. 01:07:45.623 --> 01:07:47.040 align:middle line:84% Yeah, just a few hundred examples. 01:07:47.040 --> 01:07:49.082 align:middle line:84% But actually trying to fine-tune these big models 01:07:49.082 --> 01:07:51.860 align:middle line:84% on consumer-grade hardware is actually not easy. 01:07:51.860 --> 01:07:54.520 align:middle line:84% So you have to make certain tricks and simplifications, 01:07:54.520 --> 01:07:56.340 align:middle line:90% which is the next topic. 01:07:56.340 --> 01:07:57.740 align:middle line:90% Yeah? 01:07:57.740 --> 01:08:00.070 align:middle line:84% AUDIENCE: Is fine-tuning always supervised? 01:08:00.070 --> 01:08:01.250 align:middle line:90% You need those pairs? 01:08:01.250 --> 01:08:03.870 align:middle line:84% Or could you do it if the company 01:08:03.870 --> 01:08:05.533 align:middle line:90% has less structured data? 01:08:05.533 --> 01:08:06.450 align:middle line:90% RAMA RAMAKRISHNAN: No. 01:08:06.450 --> 01:08:08.150 align:middle line:84% The thing is, it depends on whether you 01:08:08.150 --> 01:08:11.765 align:middle line:84% want to make it generally smart about the company's business 01:08:11.765 --> 01:08:14.390 align:middle line:84% details, in which case, you can just take a whole bunch of text 01:08:14.390 --> 01:08:16.098 align:middle line:84% and just do a next-word prediction on it. 01:08:16.098 --> 01:08:18.330 align:middle line:84% It's going to get smarter about generally things. 01:08:18.330 --> 01:08:20.538 align:middle line:84% But it doesn't mean it's going to specifically follow 01:08:20.538 --> 01:08:23.109 align:middle line:84% your instructions on your particular business problem. 01:08:23.109 --> 01:08:24.890 align:middle line:84% So if you want it to follow instructions, 01:08:24.890 --> 01:08:27.270 align:middle line:90% you need supervision. 01:08:27.270 --> 01:08:28.510 align:middle line:90% OK. 01:08:28.510 --> 01:08:30.710 align:middle line:84% All right, these are great reviews. 01:08:30.710 --> 01:08:34.590 align:middle line:84% So for small LLMs like GPT-2, fine-tuning isn't difficult, 01:08:34.590 --> 01:08:35.649 align:middle line:90% to go to your question. 01:08:35.649 --> 01:08:37.729 align:middle line:84% You can actually do this with small models. 01:08:37.729 --> 01:08:40.109 align:middle line:84% So for example, Google had this thing called 01:08:40.109 --> 01:08:41.410 align:middle line:90% Gemma, which came out recently. 01:08:41.410 --> 01:08:43.270 align:middle line:84% It's a small model, like 2 billion parameters or something, 01:08:43.270 --> 01:08:43.853 align:middle line:90% if I remember. 01:08:43.853 --> 01:08:44.970 align:middle line:90% It's the smallest one. 01:08:44.970 --> 01:08:49.229 align:middle line:90% 01:08:49.229 --> 01:08:50.630 align:middle line:90% Thank you. 01:08:50.630 --> 01:08:52.950 align:middle line:84% Those things will typically fit into one GPU 01:08:52.950 --> 01:08:54.010 align:middle line:90% and you can fine tune it. 01:08:54.010 --> 01:08:55.967 align:middle line:84% You still need GPUs, just to be clear. 01:08:55.967 --> 01:08:57.550 align:middle line:84% They will actually fit into one thing. 01:08:57.550 --> 01:09:00.250 align:middle line:84% But if you want to use a larger model, it won't fit. 01:09:00.250 --> 01:09:02.790 align:middle line:84% So to make this work, you have to do other things. 01:09:02.790 --> 01:09:04.810 align:middle line:84% And that's what we're going to talk about now. 01:09:04.810 --> 01:09:08.729 align:middle line:84% But there's a family of models called Llama, Llama-2. 01:09:08.729 --> 01:09:11.370 align:middle line:90% These are open source LLMs. 01:09:11.370 --> 01:09:13.729 align:middle line:84% And they are widely used for fine-tuning. 01:09:13.729 --> 01:09:15.729 align:middle line:84% Because you can just download the model and just 01:09:15.729 --> 01:09:16.950 align:middle line:90% do whatever you want with it. 01:09:16.950 --> 01:09:18.950 align:middle line:90% It's open. 01:09:18.950 --> 01:09:20.729 align:middle line:84% It's not strictly open because there 01:09:20.729 --> 01:09:23.670 align:middle line:84% are some footnote considerations you have to worry about. 01:09:23.670 --> 01:09:27.350 align:middle line:84% But for most purposes, it's open enough, in my opinion. 01:09:27.350 --> 01:09:30.890 align:middle line:84% And so let's see how hard it is to build the biggest 01:09:30.890 --> 01:09:33.670 align:middle line:84% model in this family, which is the Llama-2 model with 70 01:09:33.670 --> 01:09:35.310 align:middle line:90% billion parameters, OK? 01:09:35.310 --> 01:09:36.729 align:middle line:90% 70 billion parameters. 01:09:36.729 --> 01:09:39.689 align:middle line:84% So first of all, the model is gigantic. 01:09:39.689 --> 01:09:41.890 align:middle line:84% So 70 billion parameters, each parameter 01:09:41.890 --> 01:09:44.649 align:middle line:84% is, let's say, we store it in 2 bytes per parameter. 01:09:44.649 --> 01:09:49.330 align:middle line:84% And then each of these parameters actually, we 01:09:49.330 --> 01:09:51.250 align:middle line:84% will need a multiplier on each parameter 01:09:51.250 --> 01:09:55.786 align:middle line:84% to store various details about how the optimization is done. 01:09:55.786 --> 01:09:57.520 align:middle line:84% We won't get into the details here. 01:09:57.520 --> 01:10:01.580 align:middle line:84% The one thing I do want to point out is that this 3 to 4 01:10:01.580 --> 01:10:04.740 align:middle line:90% should really be 1 to 6. 01:10:04.740 --> 01:10:08.160 align:middle line:84% So I didn't have a chance to change it this morning. 01:10:08.160 --> 01:10:11.440 align:middle line:84% But the point is that it's going to be a huge model. 01:10:11.440 --> 01:10:12.940 align:middle line:84% So even with this number, it's going 01:10:12.940 --> 01:10:16.140 align:middle line:84% to be like 420 to 560 gigabytes, just to hold the model in memory 01:10:16.140 --> 01:10:17.340 align:middle line:90% and manipulate it. 01:10:17.340 --> 01:10:20.220 align:middle line:84% And so if you use a GPU like an A100 GPU 01:10:20.220 --> 01:10:23.740 align:middle line:84% or an H100 GPU, which are all NVIDIA GPUs, 01:10:23.740 --> 01:10:26.980 align:middle line:84% each of these things typically has 80 gigabytes of RAM memory. 01:10:26.980 --> 01:10:30.260 align:middle line:84% So we need between 6 and 7 to accommodate this thing, 01:10:30.260 --> 01:10:32.600 align:middle line:84% 6 to 7 GPUs just to accommodate this thing. 01:10:32.600 --> 01:10:33.860 align:middle line:90% So that's the first problem. 01:10:33.860 --> 01:10:34.900 align:middle line:90% The model is big. 01:10:34.900 --> 01:10:37.420 align:middle line:84% Just to hold it and work with it, you need lots of GPUs. 01:10:37.420 --> 01:10:38.860 align:middle line:90% The second problem-- 01:10:38.860 --> 01:10:43.300 align:middle line:84% Llama-2 was trained on 2 trillion tokens of text, 01:10:43.300 --> 01:10:45.260 align:middle line:90% 2 trillion tokens of text. 01:10:45.260 --> 01:10:50.580 align:middle line:84% So these GPUs can process about 400 tokens per GPU per second. 01:10:50.580 --> 01:10:54.496 align:middle line:84% By process, I mean the forward pass through the network. 01:10:54.496 --> 01:10:57.610 align:middle line:84% And so if you actually use 7 GPUs with all this thing, 01:10:57.610 --> 01:11:00.597 align:middle line:84% it's going to take you 8,000 days. 01:11:00.597 --> 01:11:02.430 align:middle line:84% Let's say we want to do it in about a month, 01:11:02.430 --> 01:11:09.790 align:middle line:84% you need 2,048 GPUs at this cost of $2.50 per GPU per hour. 01:11:09.790 --> 01:11:12.270 align:middle line:90% This will cost you $4 million. 01:11:12.270 --> 01:11:14.150 align:middle line:84% And we would expect the actual cost 01:11:14.150 --> 01:11:15.970 align:middle line:84% to be a lot higher than this because this is very optimistic. 01:11:15.970 --> 01:11:17.080 align:middle line:84% It assumes you just do one pass-through 01:11:17.080 --> 01:11:17.913 align:middle line:90% and you're all done. 01:11:17.913 --> 01:11:19.837 align:middle line:84% In general, you'll make some mistakes, 01:11:19.837 --> 01:11:22.170 align:middle line:84% you have to do a bunch of times, and so on and so forth. 01:11:22.170 --> 01:11:23.890 align:middle line:84% So this is an overly optimistic estimate, 01:11:23.890 --> 01:11:25.328 align:middle line:90% and that is $4 million. 01:11:25.328 --> 01:11:27.870 align:middle line:84% So you need lots of GPUs and you need to spend a lot of money 01:11:27.870 --> 01:11:28.630 align:middle line:90% for it. 01:11:28.630 --> 01:11:31.910 align:middle line:84% Now, what can we do with fewer resources? 01:11:31.910 --> 01:11:35.072 align:middle line:84% First of all, you need to reduce the size of the dataset. 01:11:35.072 --> 01:11:37.530 align:middle line:84% The second thing is you want to reduce the memory required. 01:11:37.530 --> 01:11:40.510 align:middle line:84% So we can ideally do it in many fewer GPUs, hopefully even one 01:11:40.510 --> 01:11:41.960 align:middle line:90% GPU, literally on Colab. 01:11:41.960 --> 01:11:44.950 align:middle line:90% 01:11:44.950 --> 01:11:46.750 align:middle line:84% Now we have good news on the data front. 01:11:46.750 --> 01:11:49.990 align:middle line:84% Because as I mentioned earlier, while it takes a lot of data 01:11:49.990 --> 01:11:52.290 align:middle line:84% to build these models, to fine-tune them 01:11:52.290 --> 01:11:54.550 align:middle line:84% for your specific data, your use case, 01:11:54.550 --> 01:11:56.990 align:middle line:84% you may just need a few hundred examples. 01:11:56.990 --> 01:11:58.130 align:middle line:90% It's no problem at all. 01:11:58.130 --> 01:12:00.950 align:middle line:84% So the data for fine-tuning is actually not a problem. 01:12:00.950 --> 01:12:04.377 align:middle line:84% Only for building it in the first place, it's a problem. 01:12:04.377 --> 01:12:06.710 align:middle line:84% In fact, there's this famous Alpaca fine-tuning dataset. 01:12:06.710 --> 01:12:08.850 align:middle line:84% It has 50,000 instruction-answer pairs. 01:12:08.850 --> 01:12:13.230 align:middle line:84% And so for that, way less than the 2 trillion tokens. 01:12:13.230 --> 01:12:15.890 align:middle line:84% And that can actually be done in about 20 hours. 01:12:15.890 --> 01:12:19.203 align:middle line:84% You can fine-tune a 50,000 example, fine-tuning dataset, 01:12:19.203 --> 01:12:20.870 align:middle line:84% you can fine tune it with just 20 hours. 01:12:20.870 --> 01:12:21.810 align:middle line:90% OK? 01:12:21.810 --> 01:12:23.282 align:middle line:90% Tomaso? 01:12:23.282 --> 01:12:27.090 align:middle line:84% AUDIENCE: Could Microsoft's 1-bit model drastically reduces 01:12:27.090 --> 01:12:28.823 align:middle line:90% the amount of compute we need? 01:12:28.823 --> 01:12:31.490 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, there's a whole bunch of approximations 01:12:31.490 --> 01:12:35.530 align:middle line:84% and simplifications to make all these things fit into smaller 01:12:35.530 --> 01:12:36.990 align:middle line:90% GPUs and so on and so forth. 01:12:36.990 --> 01:12:39.290 align:middle line:90% And that's one of them. 01:12:39.290 --> 01:12:41.850 align:middle line:84% So the short answer is, yeah, there are many possibilities. 01:12:41.850 --> 01:12:43.642 align:middle line:84% And we have to very carefully look at them. 01:12:43.642 --> 01:12:45.450 align:middle line:84% Because every one of these simplifications, 01:12:45.450 --> 01:12:47.450 align:middle line:84% it will cost you something in terms of accuracy 01:12:47.450 --> 01:12:49.990 align:middle line:84% and the ability of the model to do what it needs to do. 01:12:49.990 --> 01:12:52.540 align:middle line:84% So there's always a trade-off you have to worry about. 01:12:52.540 --> 01:12:54.380 align:middle line:84% So for folks who are interested, there 01:12:54.380 --> 01:12:57.160 align:middle line:84% is this whole field called quantization, LLM quantization. 01:12:57.160 --> 01:12:57.920 align:middle line:90% Google it. 01:12:57.920 --> 01:13:01.833 align:middle line:84% And that's an entry point into that whole area. 01:13:01.833 --> 01:13:04.500 align:middle line:84% Now, how do we reduce the memory required so that we can process 01:13:04.500 --> 01:13:06.160 align:middle line:90% the data using fewer GPUs-- 01:13:06.160 --> 01:13:08.260 align:middle line:90% ideally, just one GPU on Colab? 01:13:08.260 --> 01:13:10.860 align:middle line:84% So if you look at what actually consumes memory, 01:13:10.860 --> 01:13:12.800 align:middle line:84% you have all these model parameters. 01:13:12.800 --> 01:13:15.860 align:middle line:84% Let's say, 70 billion parameters times 2 bytes each, 01:13:15.860 --> 01:13:17.220 align:middle line:90% 140 gigabytes. 01:13:17.220 --> 01:13:20.180 align:middle line:84% Gradient computations is another 140 to hold the gradient. 01:13:20.180 --> 01:13:22.120 align:middle line:84% And then the optimizer state is 2x. 01:13:22.120 --> 01:13:25.820 align:middle line:84% And as I mentioned earlier, it could be between 1 to 6x 01:13:25.820 --> 01:13:27.353 align:middle line:90% as opposed to 3 to 4x. 01:13:27.353 --> 01:13:29.520 align:middle line:84% But we'll just go with these numbers for the moment. 01:13:29.520 --> 01:13:32.320 align:middle line:84% And so the total is 560 gigabytes, 01:13:32.320 --> 01:13:34.420 align:middle line:84% if you just naively want to use it. 01:13:34.420 --> 01:13:36.720 align:middle line:84% So it turns out you can't do anything about that. 01:13:36.720 --> 01:13:38.340 align:middle line:90% It is just for 140. 01:13:38.340 --> 01:13:41.240 align:middle line:84% But by using a trick called gradient checkpointing, 01:13:41.240 --> 01:13:44.740 align:middle line:84% this whole thing can actually be squashed close to 0. 01:13:44.740 --> 01:13:47.600 align:middle line:84% Basically, you say, hey, I don't mind it running longer, 01:13:47.600 --> 01:13:49.320 align:middle line:84% but I don't want to use as much memory. 01:13:49.320 --> 01:13:51.460 align:middle line:84% And that trick is called gradient checkpointing. 01:13:51.460 --> 01:13:53.320 align:middle line:84% We won't go into technical details. 01:13:53.320 --> 01:13:54.620 align:middle line:90% That can go to 0. 01:13:54.620 --> 01:13:56.820 align:middle line:84% But then this thing here, the optimizer state, 01:13:56.820 --> 01:14:00.340 align:middle line:84% turns out even this can be squashed very close to 0. 01:14:00.340 --> 01:14:02.040 align:middle line:84% And that actually was a breakthrough 01:14:02.040 --> 01:14:04.640 align:middle line:90% from maybe a year ago. 01:14:04.640 --> 01:14:07.540 align:middle line:84% And so to do that, what we're going to do is to say, 01:14:07.540 --> 01:14:08.713 align:middle line:90% look you know what? 01:14:08.713 --> 01:14:10.380 align:middle line:84% There are a whole bunch of weights here, 01:14:10.380 --> 01:14:13.440 align:middle line:84% but we're only going to take those matrices 01:14:13.440 --> 01:14:15.142 align:middle line:90% inside each attention layer. 01:14:15.142 --> 01:14:17.100 align:middle line:84% And we're going to only look at those matrices. 01:14:17.100 --> 01:14:19.840 align:middle line:84% We're going to freeze everything else. 01:14:19.840 --> 01:14:23.100 align:middle line:84% So we're going to take only a small set of parameters, 01:14:23.100 --> 01:14:25.058 align:middle line:84% unfreeze them, and update them, and see 01:14:25.058 --> 01:14:26.600 align:middle line:84% if it's any good, if it actually gets 01:14:26.600 --> 01:14:28.280 align:middle line:84% the job done, instead of unfreezing 01:14:28.280 --> 01:14:30.200 align:middle line:90% everything and updating them. 01:14:30.200 --> 01:14:32.060 align:middle line:84% And so if you look at the weight matrix, 01:14:32.060 --> 01:14:35.700 align:middle line:84% let's say the key AK weight matrix in Llama-2, 01:14:35.700 --> 01:14:38.720 align:middle line:84% this is a roughly 8,000 by 8,000 matrix, 01:14:38.720 --> 01:14:41.280 align:middle line:84% which means that there are 64 million parameters inside each 01:14:41.280 --> 01:14:42.900 align:middle line:90% of these matrices, 64 million. 01:14:42.900 --> 01:14:45.920 align:middle line:90% 01:14:45.920 --> 01:14:49.873 align:middle line:84% If you imagine this matrix AK here, and let's say, 01:14:49.873 --> 01:14:52.290 align:middle line:84% thought experiment, you do the fine-tuning and the numbers 01:14:52.290 --> 01:14:55.130 align:middle line:84% have changed as a result of fine-tuning. 01:14:55.130 --> 01:14:57.810 align:middle line:84% Then you can imagine that the resulting matrix is just 01:14:57.810 --> 01:15:02.650 align:middle line:84% the original matrix you had plus just the changes. 01:15:02.650 --> 01:15:03.990 align:middle line:90% The original plus the changes. 01:15:03.990 --> 01:15:06.530 align:middle line:84% And we call the changes delta AK. 01:15:06.530 --> 01:15:09.950 align:middle line:84% And of course, in general, this change is also going to be a 64 01:15:09.950 --> 01:15:12.470 align:middle line:90% million matrix, 8,000 by 8,000. 01:15:12.470 --> 01:15:15.770 align:middle line:84% So the question is, can we make this change matrix smaller? 01:15:15.770 --> 01:15:18.530 align:middle line:84% And to make it smaller, it seems reasonable 01:15:18.530 --> 01:15:21.210 align:middle line:84% because a fine-tune will only make small changes 01:15:21.210 --> 01:15:22.270 align:middle line:90% to just a few weights. 01:15:22.270 --> 01:15:23.810 align:middle line:84% It's not going to change everything. 01:15:23.810 --> 01:15:25.590 align:middle line:84% By definition, a couple of hundred examples, 01:15:25.590 --> 01:15:27.590 align:middle line:84% you do some fine-tuning, hopefully a few weights 01:15:27.590 --> 01:15:28.950 align:middle line:90% are going to change. 01:15:28.950 --> 01:15:30.890 align:middle line:84% And maybe they won't change a whole lot. 01:15:30.890 --> 01:15:33.010 align:middle line:84% So the key insight here is that maybe we 01:15:33.010 --> 01:15:35.810 align:middle line:84% can force this change matrix to be kind of simple 01:15:35.810 --> 01:15:37.890 align:middle line:90% and get the job done. 01:15:37.890 --> 01:15:39.452 align:middle line:90% And it turns out you can. 01:15:39.452 --> 01:15:41.410 align:middle line:84% And what you do is you can think of this matrix 01:15:41.410 --> 01:15:46.800 align:middle line:84% as really coming from two thin, skinny matrices, which, 01:15:46.800 --> 01:15:49.860 align:middle line:84% if you multiply them, gets you the original matrix. 01:15:49.860 --> 01:15:52.360 align:middle line:84% And I'm not going to get into the mathematical details here. 01:15:52.360 --> 01:15:55.140 align:middle line:84% This is called a low-rank approximation. 01:15:55.140 --> 01:15:56.900 align:middle line:84% But the point here is that you can 01:15:56.900 --> 01:15:59.700 align:middle line:90% take two very small matrices. 01:15:59.700 --> 01:16:01.440 align:middle line:84% And if you multiply them the right way, 01:16:01.440 --> 01:16:03.340 align:middle line:84% you actually can recover the original matrix. 01:16:03.340 --> 01:16:05.420 align:middle line:84% You can approximate the original matrix. 01:16:05.420 --> 01:16:07.280 align:middle line:84% And this matrix, as it turns out, 01:16:07.280 --> 01:16:10.700 align:middle line:84% these two matrices are much smaller because each one is just 01:16:10.700 --> 01:16:13.060 align:middle line:90% 8,000 times 2, 16,000. 01:16:13.060 --> 01:16:17.920 align:middle line:84% And so this thing has just 16,192 parameters, 01:16:17.920 --> 01:16:20.600 align:middle line:84% which is 0.02% of the original 64 million. 01:16:20.600 --> 01:16:23.460 align:middle line:90% 01:16:23.460 --> 01:16:26.400 align:middle line:84% So this thing is called low-rank adaptation, or LORA. 01:16:26.400 --> 01:16:29.780 align:middle line:84% And it's incredibly widely used in the industry. 01:16:29.780 --> 01:16:32.780 align:middle line:84% And so what we do is we freeze all the parameters, 01:16:32.780 --> 01:16:35.760 align:middle line:84% we initialize all these change matrices to 0. 01:16:35.760 --> 01:16:39.720 align:middle line:84% And then we update just those two skinny matrices right here, 01:16:39.720 --> 01:16:43.458 align:middle line:84% here, we update only those matrices using gradient descent. 01:16:43.458 --> 01:16:45.000 align:middle line:84% And when you do that, everything will 01:16:45.000 --> 01:16:47.400 align:middle line:84% fit into memory, which means that the whole thing will 01:16:47.400 --> 01:16:47.900 align:middle line:90% fit in. 01:16:47.900 --> 01:16:50.160 align:middle line:84% And you can just use two GPUs and get the job done. 01:16:50.160 --> 01:16:53.440 align:middle line:84% And if you actually use Llama's, the smaller models, 01:16:53.440 --> 01:16:55.920 align:middle line:84% like 7 billion, 13 billion, it can be fine-tuned 01:16:55.920 --> 01:17:00.160 align:middle line:84% comfortably on a single GPU, on a single Colab GPU. 01:17:00.160 --> 01:17:04.560 align:middle line:84% All right, 9:54, time does not permit. 01:17:04.560 --> 01:17:07.320 align:middle line:84% So I have a Colab on how to do the fine-tuning using 01:17:07.320 --> 01:17:08.320 align:middle line:90% this technique. 01:17:08.320 --> 01:17:12.000 align:middle line:84% I will do a video walkthrough tomorrow or day after. 01:17:12.000 --> 01:17:12.960 align:middle line:90% And I'm done. 01:17:12.960 --> 01:17:13.620 align:middle line:90% Thanks, folks. 01:17:13.620 --> 01:17:14.870 align:middle line:90% Have a good rest of your week. 01:17:14.870 --> 01:17:16.160 align:middle line:90% [APPLAUSE] 01:17:16.160 --> 01:17:17.990 align:middle line:90% Thank you. 01:17:17.990 --> 01:17:42.000 align:middle line:90%