WEBVTT

00:00:00.000 --> 00:00:00.990 align:middle line:90%


00:00:00.990 --> 00:00:02.475 align:middle line:90%
[SQUEAKING]

00:00:02.475 --> 00:00:03.960 align:middle line:90%
[RUSTLING]

00:00:03.960 --> 00:00:05.940 align:middle line:90%
[CLICKING]

00:00:05.940 --> 00:00:16.660 align:middle line:90%


00:00:16.660 --> 00:00:18.200 align:middle line:90%
RAMA RAMAKRISHNAN: OK.

00:00:18.200 --> 00:00:21.680 align:middle line:84%
So let's continue the
journey we started last time.

00:00:21.680 --> 00:00:24.520 align:middle line:84%
So what we're going to do, if
you remember in the last class,

00:00:24.520 --> 00:00:27.000 align:middle line:84%
we showed how we
can actually build

00:00:27.000 --> 00:00:29.400 align:middle line:84%
an autoregressive
large language model--

00:00:29.400 --> 00:00:30.000 align:middle line:90%
a.k.a.

00:00:30.000 --> 00:00:32.439 align:middle line:90%
a causal large language model--

00:00:32.439 --> 00:00:36.260 align:middle line:84%
using this idea
of causal encoder,

00:00:36.260 --> 00:00:37.735 align:middle line:90%
a transformer causal encoder.

00:00:37.735 --> 00:00:39.360 align:middle line:84%
And then we showed
how you can actually

00:00:39.360 --> 00:00:42.280 align:middle line:84%
take a bunch of sentences
and use next-word-prediction

00:00:42.280 --> 00:00:43.520 align:middle line:90%
and just run it through.

00:00:43.520 --> 00:00:45.840 align:middle line:90%
And boom, you get GPT-3.

00:00:45.840 --> 00:00:47.600 align:middle line:90%
So that's what we saw last time.

00:00:47.600 --> 00:00:51.720 align:middle line:84%
I want to point out an important
clarification/correction, which

00:00:51.720 --> 00:00:54.880 align:middle line:84%
is that when we work with
large language models,

00:00:54.880 --> 00:00:58.080 align:middle line:84%
unlike when we work with BERT,
for instance, when we work

00:00:58.080 --> 00:01:00.730 align:middle line:84%
with these kinds of
causal models, actually,

00:01:00.730 --> 00:01:02.670 align:middle line:84%
when the contextual
embeddings come out,

00:01:02.670 --> 00:01:05.489 align:middle line:84%
you don't actually have to
use ReLU activations here.

00:01:05.489 --> 00:01:07.370 align:middle line:84%
You can literally just
run it through just

00:01:07.370 --> 00:01:10.170 align:middle line:84%
a single dense layer
with linear activations

00:01:10.170 --> 00:01:11.930 align:middle line:90%
and then pass it into a softmax.

00:01:11.930 --> 00:01:13.590 align:middle line:90%
And boom, you're done.

00:01:13.590 --> 00:01:16.730 align:middle line:84%
So that's how GPT-3 and all
these models are trained.

00:01:16.730 --> 00:01:19.590 align:middle line:84%
And the other thing
I want to point out,

00:01:19.590 --> 00:01:20.970 align:middle line:84%
which may not have
been clear, is

00:01:20.970 --> 00:01:25.210 align:middle line:84%
that what is coming out
of this dense layer,

00:01:25.210 --> 00:01:29.810 align:middle line:84%
this vector is as long
as your vocabulary.

00:01:29.810 --> 00:01:32.645 align:middle line:84%
Because only then, when
it goes into the softmax,

00:01:32.645 --> 00:01:34.770 align:middle line:84%
you're going to get
probabilities which are as long

00:01:34.770 --> 00:01:38.050 align:middle line:84%
as your vocabulary, which means
that you get to pick one word

00:01:38.050 --> 00:01:43.722 align:middle line:84%
or token out of that entire
50,000-long vocabulary.

00:01:43.722 --> 00:01:47.570 align:middle line:84%
So I just want to point that out
because I think it's easy for us

00:01:47.570 --> 00:01:50.250 align:middle line:84%
to get a little confused because
of this little difference

00:01:50.250 --> 00:01:54.530 align:middle line:84%
between the way masked
language models like BERT work

00:01:54.530 --> 00:01:58.630 align:middle line:84%
and causal language
models like GPT-3.

00:01:58.630 --> 00:02:02.350 align:middle line:90%
So now let's continue with--

00:02:02.350 --> 00:02:04.550 align:middle line:90%
we know how to build GPT-3.

00:02:04.550 --> 00:02:07.150 align:middle line:90%
So what about ChatGPT and GPT-2?

00:02:07.150 --> 00:02:08.310 align:middle line:90%
What's up with them?

00:02:08.310 --> 00:02:10.830 align:middle line:84%
Why is GPT-3 so
famous and not GPT-2?

00:02:10.830 --> 00:02:12.933 align:middle line:90%


00:02:12.933 --> 00:02:14.350 align:middle line:84%
Well, first of
all, you folks know

00:02:14.350 --> 00:02:16.870 align:middle line:84%
the GPT stands for generative
pretrained transformer.

00:02:16.870 --> 00:02:22.230 align:middle line:84%
Now, like GPT-3, GPT-2 and
GPT-1 were trained in basically

00:02:22.230 --> 00:02:23.210 align:middle line:90%
the same fashion--

00:02:23.210 --> 00:02:26.590 align:middle line:84%
predict the next word, same
fashion, the same transformer

00:02:26.590 --> 00:02:31.310 align:middle line:84%
stack, except that GPT-3 was
trained on much more data

00:02:31.310 --> 00:02:33.950 align:middle line:84%
because the underlying
transformer stack had many more

00:02:33.950 --> 00:02:35.850 align:middle line:90%
layers.

00:02:35.850 --> 00:02:40.410 align:middle line:84%
So it is a much bigger stack,
meaning lots more parameters,

00:02:40.410 --> 00:02:43.470 align:middle line:84%
and therefore you need lots
more data to train it well.

00:02:43.470 --> 00:02:44.230 align:middle line:90%
OK?

00:02:44.230 --> 00:02:46.910 align:middle line:84%
So that was really
the only difference.

00:02:46.910 --> 00:02:50.870 align:middle line:84%
The difference was literally
one of scale, scale of network

00:02:50.870 --> 00:02:52.390 align:middle line:90%
and scale of data.

00:02:52.390 --> 00:02:57.087 align:middle line:84%
And unlike GPT and
GPT-2, GPT-3, even

00:02:57.087 --> 00:02:58.920 align:middle line:84%
though it was trained
basically the same way

00:02:58.920 --> 00:03:01.280 align:middle line:84%
with the same kind
of network, it

00:03:01.280 --> 00:03:05.330 align:middle line:84%
was one of those situations
where more became different.

00:03:05.330 --> 00:03:07.080 align:middle line:84%
There was almost like
some of phase change

00:03:07.080 --> 00:03:09.320 align:middle line:90%
that happened between 2 and 3.

00:03:09.320 --> 00:03:11.800 align:middle line:84%
Unlike GPT and
GPT-2, GPT-3 could

00:03:11.800 --> 00:03:14.800 align:middle line:84%
do amazingly coherent
continuations

00:03:14.800 --> 00:03:17.920 align:middle line:90%
of any starting prompt.

00:03:17.920 --> 00:03:20.218 align:middle line:84%
So for example, if you
have this little prompt,

00:03:20.218 --> 00:03:22.760 align:middle line:84%
which says "The importance of
being on Twitter," by Jerome K.

00:03:22.760 --> 00:03:24.600 align:middle line:84%
Jerome, who was a
famous humorist,

00:03:24.600 --> 00:03:27.720 align:middle line:84%
and then you give it this prompt
ending with the word "it,"

00:03:27.720 --> 00:03:31.800 align:middle line:84%
it produces this continuation,
which is really strikingly good.

00:03:31.800 --> 00:03:34.228 align:middle line:84%
And if any of you have
read Jerome K. Jerome

00:03:34.228 --> 00:03:36.020 align:middle line:84%
and if you read this
thing, you'll be like,

00:03:36.020 --> 00:03:38.880 align:middle line:84%
wow, that actually sounds
like Jerome K. Jerome.

00:03:38.880 --> 00:03:41.040 align:middle line:90%
So, amazing continuations.

00:03:41.040 --> 00:03:44.260 align:middle line:84%
But the interesting thing here
is not so much the continuation,

00:03:44.260 --> 00:03:48.220 align:middle line:84%
it's the fact that the same
prompt, you give it to 2 or GPT,

00:03:48.220 --> 00:03:49.400 align:middle line:90%
it won't be very good.

00:03:49.400 --> 00:03:51.860 align:middle line:84%
In fact, after the first
one, two, or three sentences,

00:03:51.860 --> 00:03:55.210 align:middle line:84%
it'll become incoherent and
meander and start rambling.

00:03:55.210 --> 00:03:58.970 align:middle line:84%
This thing can keep faking
it for a longer time.

00:03:58.970 --> 00:04:00.230 align:middle line:90%
That's the amazing thing.

00:04:00.230 --> 00:04:02.130 align:middle line:90%
That was unexpected.

00:04:02.130 --> 00:04:05.650 align:middle line:90%
Researchers did not expect this.

00:04:05.650 --> 00:04:09.010 align:middle line:84%
But it wasn't good at
following your instructions.

00:04:09.010 --> 00:04:10.610 align:middle line:84%
So for instance,
if you ask it, help

00:04:10.610 --> 00:04:13.370 align:middle line:84%
me write a short note to
introduce myself to my neighbor.

00:04:13.370 --> 00:04:14.790 align:middle line:84%
This is the kind of thing
it will come up with.

00:04:14.790 --> 00:04:16.332 align:middle line:84%
And you can actually
run it yourself.

00:04:16.332 --> 00:04:18.709 align:middle line:84%
You can actually go to
GPT-3 on the Playground.

00:04:18.709 --> 00:04:21.690 align:middle line:84%
I think GPT-3 is still
available in the Playground.

00:04:21.690 --> 00:04:24.870 align:middle line:84%
If it is, you can actually
start running these prompts.

00:04:24.870 --> 00:04:27.890 align:middle line:84%
You will start getting
garbage very quickly.

00:04:27.890 --> 00:04:29.542 align:middle line:84%
And the reason, So
for example here,

00:04:29.542 --> 00:04:30.750 align:middle line:90%
"help me write a short note."

00:04:30.750 --> 00:04:33.050 align:middle line:84%
It says, "what is a good
introduction to a resume."

00:04:33.050 --> 00:04:35.750 align:middle line:84%
Resume, for some reason
it's glommed on to resume.

00:04:35.750 --> 00:04:37.570 align:middle line:90%
I have no idea why.

00:04:37.570 --> 00:04:39.610 align:middle line:84%
But the reason it's
doing stuff like this

00:04:39.610 --> 00:04:42.930 align:middle line:84%
is because a lot of the
training data it was trained on

00:04:42.930 --> 00:04:46.050 align:middle line:84%
are basically lots
of lists of things.

00:04:46.050 --> 00:04:54.442 align:middle line:84%
So when you say, for example,
"the capital of France

00:04:54.442 --> 00:04:56.650 align:middle line:84%
continued," it'll say the
capital of France is Paris,

00:04:56.650 --> 00:04:59.370 align:middle line:84%
the capital of Hungary
is Budapest, and so on.

00:04:59.370 --> 00:05:01.150 align:middle line:84%
It just start came
up with the list.

00:05:01.150 --> 00:05:03.390 align:middle line:90%
So it's very list-driven.

00:05:03.390 --> 00:05:05.970 align:middle line:84%
It thinks that you need to
complete some sort of list.

00:05:05.970 --> 00:05:07.550 align:middle line:90%
That's what's going on here.

00:05:07.550 --> 00:05:08.970 align:middle line:90%
And so it's not very good.

00:05:08.970 --> 00:05:11.012 align:middle line:84%
So it doesn't realize that
you're actually asking

00:05:11.012 --> 00:05:12.950 align:middle line:90%
it to do something specific.

00:05:12.950 --> 00:05:16.070 align:middle line:84%
So this is the problem when you
have an autocomplete thing which

00:05:16.070 --> 00:05:19.030 align:middle line:84%
doesn't realize what
you're asking it.

00:05:19.030 --> 00:05:22.310 align:middle line:90%
It's an autocomplete.

00:05:22.310 --> 00:05:24.387 align:middle line:84%
Now, in addition to
these unhelpful answers,

00:05:24.387 --> 00:05:25.970 align:middle line:84%
it can also produce
offensive answers,

00:05:25.970 --> 00:05:28.137 align:middle line:84%
factually incorrect answers,
and so on and so forth.

00:05:28.137 --> 00:05:31.390 align:middle line:84%
The list of bad things
it can do is long.

00:05:31.390 --> 00:05:32.370 align:middle line:90%
So why does it do that?

00:05:32.370 --> 00:05:34.010 align:middle line:84%
Why does it produce
unhelpful answers?

00:05:34.010 --> 00:05:36.448 align:middle line:84%
Well, as you recall,
it was only trained

00:05:36.448 --> 00:05:37.490 align:middle line:90%
to predict the next word.

00:05:37.490 --> 00:05:40.770 align:middle line:84%
It wasn't explicitly trained
to follow instructions.

00:05:40.770 --> 00:05:44.950 align:middle line:84%
So it seems reasonable
that if it's simply

00:05:44.950 --> 00:05:47.430 align:middle line:84%
trying to guess the
next word repeatedly,

00:05:47.430 --> 00:05:49.558 align:middle line:84%
it can't really
do anything more.

00:05:49.558 --> 00:05:51.600 align:middle line:84%
How can it figure out that
there's an instruction

00:05:51.600 --> 00:05:52.720 align:middle line:90%
that it needs to follow?

00:05:52.720 --> 00:05:55.240 align:middle line:84%
Unless the training
data on the internet

00:05:55.240 --> 00:05:59.800 align:middle line:84%
was all instructional,
which it clearly is not.

00:05:59.800 --> 00:06:02.440 align:middle line:90%
So, light bulb idea--

00:06:02.440 --> 00:06:06.660 align:middle line:84%
let's explicitly train
it with instruction data.

00:06:06.660 --> 00:06:08.720 align:middle line:84%
Let's just train it
with instruction data.

00:06:08.720 --> 00:06:12.040 align:middle line:84%
And so OpenAI developed an
approach called instruction

00:06:12.040 --> 00:06:14.720 align:middle line:90%
tuning to do exactly this.

00:06:14.720 --> 00:06:19.800 align:middle line:84%
And this paper is the paper
that was the breakthrough.

00:06:19.800 --> 00:06:23.480 align:middle line:84%
This is what actually
put ChatGPT on the map.

00:06:23.480 --> 00:06:25.020 align:middle line:90%
And it's very readable.

00:06:25.020 --> 00:06:28.760 align:middle line:84%
So I would encourage you to
check it out, if you're curious.

00:06:28.760 --> 00:06:34.440 align:middle line:84%
And so we had GPT, GPT-2, GPT-3,
just bigger and bigger models

00:06:34.440 --> 00:06:35.480 align:middle line:90%
trained the same way.

00:06:35.480 --> 00:06:37.147 align:middle line:84%
And then we run into
the problem that it

00:06:37.147 --> 00:06:38.260 align:middle line:90%
can't handle instructions.

00:06:38.260 --> 00:06:41.240 align:middle line:84%
So we do instruction
tuning to get to 3.5, also

00:06:41.240 --> 00:06:42.560 align:middle line:90%
called InstructGPT.

00:06:42.560 --> 00:06:45.440 align:middle line:84%
And then a small tweak
after that gets you ChatGPT.

00:06:45.440 --> 00:06:49.260 align:middle line:84%
And by the way, this step here,
there are really two things

00:06:49.260 --> 00:06:51.240 align:middle line:84%
going on in this, as
you will soon see,

00:06:51.240 --> 00:06:52.860 align:middle line:84%
I'm just calling it
instruction tuning

00:06:52.860 --> 00:06:55.100 align:middle line:84%
just so that I don't have
to say some long thing

00:06:55.100 --> 00:06:56.700 align:middle line:90%
every single time.

00:06:56.700 --> 00:06:59.280 align:middle line:84%
This is not a consistent
piece of terminology,

00:06:59.280 --> 00:07:03.380 align:middle line:84%
so just be aware
of it, that's all.

00:07:03.380 --> 00:07:06.580 align:middle line:84%
All right, first step,
they got a bunch of people

00:07:06.580 --> 00:07:09.420 align:middle line:84%
to write high-quality
answers to questions.

00:07:09.420 --> 00:07:12.960 align:middle line:84%
And they created about 12,500
such question-answer pairs.

00:07:12.960 --> 00:07:15.013 align:middle line:84%
So for example, let's say
this was the question--

00:07:15.013 --> 00:07:17.180 align:middle line:84%
"Explain the moon landing
to a six-year-old in a few

00:07:17.180 --> 00:07:18.180 align:middle line:90%
sentences."

00:07:18.180 --> 00:07:21.060 align:middle line:84%
Believe it or not, GPT-3's
answer to that question

00:07:21.060 --> 00:07:23.223 align:middle line:90%
was another question.

00:07:23.223 --> 00:07:25.140 align:middle line:84%
Because it thinks there
is a list of questions

00:07:25.140 --> 00:07:26.936 align:middle line:90%
it needs to autocomplete.

00:07:26.936 --> 00:07:29.353 align:middle line:84%
So it comes up with, "Explain
the theory of gravity to a 6

00:07:29.353 --> 00:07:29.940 align:middle line:90%
year old."

00:07:29.940 --> 00:07:30.860 align:middle line:84%
It's like one of
those people who,

00:07:30.860 --> 00:07:33.480 align:middle line:84%
when you ask them a question,
they ask you a question back.

00:07:33.480 --> 00:07:36.260 align:middle line:84%
So what they did is they said,
OK, let's create a nice answer

00:07:36.260 --> 00:07:37.160 align:middle line:90%
to this question.

00:07:37.160 --> 00:07:38.705 align:middle line:84%
And here's a
human-created answer--

00:07:38.705 --> 00:07:41.080 align:middle line:84%
"People went to the moon in
a big rocket, walked around,"

00:07:41.080 --> 00:07:42.600 align:middle line:90%
blah, blah, blah.

00:07:42.600 --> 00:07:44.780 align:middle line:84%
Much better answer
to that question.

00:07:44.780 --> 00:07:48.710 align:middle line:84%
And so once you create these
12,500 question-answer pairs

00:07:48.710 --> 00:07:53.510 align:middle line:84%
as training data, we just
trained GPT-3 some more using

00:07:53.510 --> 00:07:57.890 align:middle line:84%
next word prediction as
before, no difference.

00:07:57.890 --> 00:07:58.807 align:middle line:90%
So here is the input--

00:07:58.807 --> 00:08:00.807 align:middle line:84%
"Explain the moon landing,"
blah blah blah blah.

00:08:00.807 --> 00:08:01.870 align:middle line:90%
This is the question.

00:08:01.870 --> 00:08:04.150 align:middle line:84%
And then we have the
answer right there.

00:08:04.150 --> 00:08:08.030 align:middle line:84%
And then we take that
answer, move it to the right,

00:08:08.030 --> 00:08:13.510 align:middle line:84%
and just shift it up so that
when it finishes "sentences,"

00:08:13.510 --> 00:08:15.413 align:middle line:90%
it needs to predict "people."

00:08:15.413 --> 00:08:17.830 align:middle line:84%
And then you give it "people,"
it needs to predict "went,"

00:08:17.830 --> 00:08:20.750 align:middle line:84%
and so on and so forth,
just like we saw before.

00:08:20.750 --> 00:08:24.688 align:middle line:84%
"The cat sat on the mat" became
"the cat sat on the," "cat sat

00:08:24.688 --> 00:08:25.730 align:middle line:90%
on the mat" on the right.

00:08:25.730 --> 00:08:26.890 align:middle line:90%
Shifted, right?

00:08:26.890 --> 00:08:30.110 align:middle line:84%
That's what makes prediction
possible and necessary.

00:08:30.110 --> 00:08:31.230 align:middle line:90%
So that's what they did.

00:08:31.230 --> 00:08:35.350 align:middle line:84%
This is step one,
same as before.

00:08:35.350 --> 00:08:37.590 align:middle line:84%
And once you do
that, it turns out--

00:08:37.590 --> 00:08:39.650 align:middle line:84%
this step is called
supervised fine tuning--

00:08:39.650 --> 00:08:40.789 align:middle line:90%
it really helped.

00:08:40.789 --> 00:08:44.150 align:middle line:84%
GPT-3, once you supervise fine
tuned it, was much, much better

00:08:44.150 --> 00:08:45.540 align:middle line:90%
at following instructions.

00:08:45.540 --> 00:08:47.920 align:middle line:84%
But there's a small
problem with this approach.

00:08:47.920 --> 00:08:49.720 align:middle line:84%
It takes a lot of
money and effort

00:08:49.720 --> 00:08:51.800 align:middle line:84%
to have humans write
high-quality answers

00:08:51.800 --> 00:08:54.400 align:middle line:90%
to thousands of questions.

00:08:54.400 --> 00:08:55.800 align:middle line:90%
It takes a lot of money.

00:08:55.800 --> 00:08:58.240 align:middle line:84%
So the question
is, what can we do?

00:08:58.240 --> 00:09:01.470 align:middle line:84%
What is easier than writing
a good answer to a question?

00:09:01.470 --> 00:09:04.280 align:middle line:90%


00:09:04.280 --> 00:09:07.280 align:middle line:90%
OK, all right.

00:09:07.280 --> 00:09:08.780 align:middle line:84%
How about somebody
from this side?

00:09:08.780 --> 00:09:11.360 align:middle line:90%


00:09:11.360 --> 00:09:13.560 align:middle line:90%
Yeah?

00:09:13.560 --> 00:09:15.685 align:middle line:84%
AUDIENCE: Perhaps writing
a question for an answer.

00:09:15.685 --> 00:09:17.768 align:middle line:84%
RAMA RAMAKRISHNAN: Oh,
that's actually a good one.

00:09:17.768 --> 00:09:18.960 align:middle line:90%
Yeah, yeah, I like that.

00:09:18.960 --> 00:09:21.480 align:middle line:84%
So, given an answer,
find a question.

00:09:21.480 --> 00:09:24.020 align:middle line:84%
And while that is not what
I'm going to talk about here,

00:09:24.020 --> 00:09:27.200 align:middle line:84%
that technique is actually
used very heavily in LLMs.

00:09:27.200 --> 00:09:29.400 align:middle line:84%
And so that's great,
very Creative.

00:09:29.400 --> 00:09:31.165 align:middle line:90%
Mark?

00:09:31.165 --> 00:09:32.540 align:middle line:84%
AUDIENCE: Thumbs
up, thumbs down.

00:09:32.540 --> 00:09:33.120 align:middle line:90%
RAMA RAMAKRISHNAN: Sorry?

00:09:33.120 --> 00:09:34.540 align:middle line:84%
AUDIENCE: Thumbs
up or thumbs down.

00:09:34.540 --> 00:09:36.748 align:middle line:84%
RAMA RAMAKRISHNAN: Thumbs
up or thumbs down, exactly.

00:09:36.748 --> 00:09:39.520 align:middle line:84%
Because all of us, everyone
loves to be a critic.

00:09:39.520 --> 00:09:43.160 align:middle line:84%
It's much easier to be a
critic than to be a creator.

00:09:43.160 --> 00:09:44.620 align:middle line:90%
So what do we do?

00:09:44.620 --> 00:09:46.980 align:middle line:84%
We basically say,
let's rank answers

00:09:46.980 --> 00:09:50.420 align:middle line:84%
written by somebody else,
which begs the question, who's

00:09:50.420 --> 00:09:52.388 align:middle line:90%
going to write those answers?

00:09:52.388 --> 00:09:54.180 align:middle line:84%
And that's where there's
a brilliant answer

00:09:54.180 --> 00:09:55.940 align:middle line:90%
to that question, which is?

00:09:55.940 --> 00:09:57.112 align:middle line:90%
AUDIENCE: Wikipedia.

00:09:57.112 --> 00:09:58.320 align:middle line:90%
RAMA RAMAKRISHNAN: Wikipedia.

00:09:58.320 --> 00:09:59.220 align:middle line:90%
AUDIENCE: Reddit.

00:09:59.220 --> 00:10:00.830 align:middle line:90%
RAMA RAMAKRISHNAN: Reddit.

00:10:00.830 --> 00:10:04.020 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:10:04.020 --> 00:10:08.100 align:middle line:84%
RAMA RAMAKRISHNAN: We will just
ask GPT-3 to write the answers.

00:10:08.100 --> 00:10:12.860 align:middle line:84%
It might be crap, but we don't
care because we can rank them.

00:10:12.860 --> 00:10:17.173 align:middle line:84%
So we ask GPT-3 to generate
several answers to the question.

00:10:17.173 --> 00:10:18.840 align:middle line:84%
And how can we generate
several answers?

00:10:18.840 --> 00:10:21.180 align:middle line:90%
Because we can do sampling.

00:10:21.180 --> 00:10:23.100 align:middle line:90%
We can do sampling.

00:10:23.100 --> 00:10:25.780 align:middle line:84%
The fact that we had these
stochastic outputs because

00:10:25.780 --> 00:10:29.480 align:middle line:84%
of sampling is now a
feature, not a bug.

00:10:29.480 --> 00:10:32.120 align:middle line:84%
We create lots of different
answers to the question.

00:10:32.120 --> 00:10:35.022 align:middle line:84%
We feed it a question,
get three answers out.

00:10:35.022 --> 00:10:36.980 align:middle line:84%
Just run it three times,
get three answers out,

00:10:36.980 --> 00:10:39.060 align:middle line:84%
with a nice
temperature of 1 or 1.1

00:10:39.060 --> 00:10:42.270 align:middle line:84%
or something so that
it's nice and random.

00:10:42.270 --> 00:10:44.110 align:middle line:84%
And then we
literally have humans

00:10:44.110 --> 00:10:46.310 align:middle line:84%
just rank them, do the
thumbs up, thumbs down, just

00:10:46.310 --> 00:10:48.990 align:middle line:84%
rank them from most
useful to least useful.

00:10:48.990 --> 00:10:53.610 align:middle line:84%
So this step is a Step
2 of instruction tuning.

00:10:53.610 --> 00:10:57.290 align:middle line:84%
So OpenAI collected 33,000
instructions, fed them to GPT-3,

00:10:57.290 --> 00:11:01.990 align:middle line:84%
generated answers, and
had humans rank them.

00:11:01.990 --> 00:11:04.830 align:middle line:84%
Once you do this, you can
assemble a beautiful training

00:11:04.830 --> 00:11:06.060 align:middle line:90%
dataset.

00:11:06.060 --> 00:11:09.050 align:middle line:84%
And so basically, what we have
is that we have an instruction.

00:11:09.050 --> 00:11:11.150 align:middle line:84%
And let's say we have
just two answers, A and B.

00:11:11.150 --> 00:11:13.550 align:middle line:84%
And in practice, you can
have many, many answers,

00:11:13.550 --> 00:11:14.490 align:middle line:90%
which we rank.

00:11:14.490 --> 00:11:17.330 align:middle line:84%
But just for simplicity, I'll
go with Mark's thumbs up,

00:11:17.330 --> 00:11:19.190 align:middle line:84%
thumbs down answer,
which is let's

00:11:19.190 --> 00:11:21.630 align:middle line:84%
assume we only have two
answers to every question.

00:11:21.630 --> 00:11:24.570 align:middle line:84%
And the human has said,
I prefer this to that.

00:11:24.570 --> 00:11:25.440 align:middle line:90%
That's it.

00:11:25.440 --> 00:11:29.790 align:middle line:84%
So we have a dataset now where
the data point is instruction,

00:11:29.790 --> 00:11:33.310 align:middle line:84%
preferred answer is A, the
other answer is B. Yeah?

00:11:33.310 --> 00:11:36.350 align:middle line:90%


00:11:36.350 --> 00:11:38.247 align:middle line:84%
AUDIENCE: The thumbs
up, thumbs down

00:11:38.247 --> 00:11:40.080 align:middle line:84%
technique that we were
talking about before,

00:11:40.080 --> 00:11:41.900 align:middle line:84%
is that why when we're
using ChatGPT now,

00:11:41.900 --> 00:11:43.400 align:middle line:84%
we also used thumbs
up, thumbs down?

00:11:43.400 --> 00:11:45.320 align:middle line:84%
Because it's using all
the answers to train?

00:11:45.320 --> 00:11:46.260 align:middle line:84%
RAMA RAMAKRISHNAN:
Exactly right.

00:11:46.260 --> 00:11:46.920 align:middle line:90%
AUDIENCE: OK.

00:11:46.920 --> 00:11:48.580 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, all
the models have the thumbs up,

00:11:48.580 --> 00:11:50.122 align:middle line:84%
thumbs down stuff
going on somewhere.

00:11:50.122 --> 00:11:52.472 align:middle line:84%
They are all collecting
data for this step.

00:11:52.472 --> 00:11:53.180 align:middle line:90%
AUDIENCE: Got it.

00:11:53.180 --> 00:11:53.580 align:middle line:90%
Thank you.

00:11:53.580 --> 00:11:54.080 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:11:54.080 --> 00:11:55.840 align:middle line:84%
It's sort of the
old adage, right?

00:11:55.840 --> 00:11:58.360 align:middle line:84%
If you're not sure who the
product is, you're the product.

00:11:58.360 --> 00:11:59.620 align:middle line:90%
So it's one of those things.

00:11:59.620 --> 00:12:00.120 align:middle line:90%
Yeah?

00:12:00.120 --> 00:12:07.440 align:middle line:90%


00:12:07.440 --> 00:12:10.120 align:middle line:84%
AUDIENCE: So if we understand
correctly, when we see thumbs

00:12:10.120 --> 00:12:14.480 align:middle line:84%
up, thumbs down, it does
mean that ChatGPT is going

00:12:14.480 --> 00:12:16.560 align:middle line:90%
to trade on our data, right?

00:12:16.560 --> 00:12:18.720 align:middle line:84%
RAMA RAMAKRISHNAN:
Unless you opt out, yeah.

00:12:18.720 --> 00:12:20.580 align:middle line:84%
So if you actually go
to ChatGPT controls,

00:12:20.580 --> 00:12:22.788 align:middle line:84%
there is something called
data controls or something.

00:12:22.788 --> 00:12:24.440 align:middle line:90%
You can toggle it off.

00:12:24.440 --> 00:12:27.120 align:middle line:84%
But I think when I last
checked, if you toggle it off,

00:12:27.120 --> 00:12:29.425 align:middle line:90%
you lose your chat history.

00:12:29.425 --> 00:12:30.800 align:middle line:84%
So they have
hobbled that feature

00:12:30.800 --> 00:12:34.760 align:middle line:84%
to prevent people from setting
it to off as much as possible.

00:12:34.760 --> 00:12:37.480 align:middle line:90%
Yeah, clever.

00:12:37.480 --> 00:12:38.640 align:middle line:90%
But you can opt out.

00:12:38.640 --> 00:12:41.120 align:middle line:84%
And if you use the API, as
opposed to the web interface,

00:12:41.120 --> 00:12:43.060 align:middle line:90%
you're automatically opted out.

00:12:43.060 --> 00:12:44.673 align:middle line:84%
So you have to
deliberately opt in.

00:12:44.673 --> 00:12:47.340 align:middle line:84%
And if you use the versions that
are available through Microsoft

00:12:47.340 --> 00:12:48.900 align:middle line:84%
Azure and so on
and so forth, there

00:12:48.900 --> 00:12:51.240 align:middle line:84%
are all kinds of very safe
controls and stuff like that.

00:12:51.240 --> 00:12:55.860 align:middle line:84%
In fact, I think the Microsoft
Copilot license that MIT has,

00:12:55.860 --> 00:12:58.780 align:middle line:84%
I think the default
is opted out.

00:12:58.780 --> 00:13:01.880 align:middle line:84%
So to go here, once you
have this data point,

00:13:01.880 --> 00:13:04.620 align:middle line:84%
you can build something
called a reward model.

00:13:04.620 --> 00:13:07.340 align:middle line:84%
And this is a very
clever piece of work.

00:13:07.340 --> 00:13:08.980 align:middle line:90%
So what you do is--

00:13:08.980 --> 00:13:11.900 align:middle line:84%
you have an instruction,
you have a preferred answer,

00:13:11.900 --> 00:13:13.540 align:middle line:90%
and you have the other answer--

00:13:13.540 --> 00:13:17.198 align:middle line:90%
you feed it to a network.

00:13:17.198 --> 00:13:18.240 align:middle line:90%
You feed it to a network.

00:13:18.240 --> 00:13:20.780 align:middle line:84%
This is just a nice
language model.

00:13:20.780 --> 00:13:22.700 align:middle line:90%
It's just a language model.

00:13:22.700 --> 00:13:25.880 align:middle line:84%
And the language model
produces a number,

00:13:25.880 --> 00:13:28.260 align:middle line:84%
which measures how
good this thing is.

00:13:28.260 --> 00:13:31.900 align:middle line:84%
How good an answer is this to
that particular instruction?

00:13:31.900 --> 00:13:34.240 align:middle line:90%
So you get a rating here.

00:13:34.240 --> 00:13:35.620 align:middle line:90%
You get a rating here.

00:13:35.620 --> 00:13:38.630 align:middle line:84%
And then what you
do is you run it

00:13:38.630 --> 00:13:44.350 align:middle line:84%
through a little loss function,
which essentially encourages

00:13:44.350 --> 00:13:49.990 align:middle line:84%
the model to give higher
numbers to the better answer.

00:13:49.990 --> 00:13:51.010 align:middle line:90%
It's the same model.

00:13:51.010 --> 00:13:53.570 align:middle line:84%
You just run the question on
the first answer, question

00:13:53.570 --> 00:13:55.810 align:middle line:84%
on the second answer, you
get these two numbers.

00:13:55.810 --> 00:13:58.510 align:middle line:84%
And then initially those
numbers are just random.

00:13:58.510 --> 00:14:00.010 align:middle line:84%
But then you tell
the model, hey,

00:14:00.010 --> 00:14:01.177 align:middle line:90%
this is the preferred thing.

00:14:01.177 --> 00:14:05.710 align:middle line:84%
Make sure the preferred
answer's rating, the R value,

00:14:05.710 --> 00:14:07.470 align:middle line:90%
is higher than the other number.

00:14:07.470 --> 00:14:09.700 align:middle line:84%
Because more is better,
higher is better.

00:14:09.700 --> 00:14:13.190 align:middle line:90%


00:14:13.190 --> 00:14:15.120 align:middle line:84%
And this thing is
just the sigmoid here.

00:14:15.120 --> 00:14:17.370 align:middle line:84%
You basically take the
difference of these two things.

00:14:17.370 --> 00:14:18.870 align:middle line:84%
Do a sigmoid, take
the logarithm.

00:14:18.870 --> 00:14:21.690 align:middle line:84%
And you can actually convince
yourself afterwards--

00:14:21.690 --> 00:14:23.310 align:middle line:90%
and I encourage you to do that--

00:14:23.310 --> 00:14:28.350 align:middle line:84%
to check for yourself
that if we actually

00:14:28.350 --> 00:14:31.530 align:middle line:84%
give a higher number
to the better answer,

00:14:31.530 --> 00:14:33.790 align:middle line:90%
the loss will be lower.

00:14:33.790 --> 00:14:35.580 align:middle line:84%
And since we are
minimizing loss,

00:14:35.580 --> 00:14:38.360 align:middle line:84%
we are essentially
training the network

00:14:38.360 --> 00:14:41.680 align:middle line:84%
to try to give higher
ratings to better answers.

00:14:41.680 --> 00:14:42.380 align:middle line:90%
That's it.

00:14:42.380 --> 00:14:43.680 align:middle line:90%
So that's the approach.

00:14:43.680 --> 00:14:44.520 align:middle line:90%
Did you have a--

00:14:44.520 --> 00:14:46.660 align:middle line:90%
yeah, Ben.

00:14:46.660 --> 00:14:50.520 align:middle line:84%
AUDIENCE: So you could imagine
training the model on only

00:14:50.520 --> 00:14:52.280 align:middle line:84%
the good answers, is
the idea of having

00:14:52.280 --> 00:14:54.660 align:middle line:84%
both that the model's actually
learning what makes good--

00:14:54.660 --> 00:14:55.980 align:middle line:84%
RAMA RAMAKRISHNAN:
Correct, exactly.

00:14:55.980 --> 00:14:58.147 align:middle line:84%
Much like if you want to
build a dog-cat classifier,

00:14:58.147 --> 00:15:01.400 align:middle line:84%
you have to show
pictures of both.

00:15:01.400 --> 00:15:02.840 align:middle line:90%
Yeah?

00:15:02.840 --> 00:15:05.880 align:middle line:84%
AUDIENCE: So I understand the
feedback mechanism of thumbs up,

00:15:05.880 --> 00:15:06.380 align:middle line:90%
thumbs down.

00:15:06.380 --> 00:15:10.560 align:middle line:84%
But there are a lot of times
when the popular response is not

00:15:10.560 --> 00:15:11.680 align:middle line:90%
the accurate one.

00:15:11.680 --> 00:15:14.440 align:middle line:84%
So is there a way
that they actually

00:15:14.440 --> 00:15:15.562 align:middle line:90%
have a layer to correct?

00:15:15.562 --> 00:15:17.520 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah,
good question, Swathi.

00:15:17.520 --> 00:15:23.400 align:middle line:84%
So as it turns out, all
these companies like OpenAI,

00:15:23.400 --> 00:15:27.400 align:middle line:84%
they have a huge document,
100, 200 pages long,

00:15:27.400 --> 00:15:30.600 align:middle line:84%
very, very bulky document,
which instructs and teaches

00:15:30.600 --> 00:15:34.400 align:middle line:84%
the labelers, the rankers
how to rank these things.

00:15:34.400 --> 00:15:36.540 align:middle line:84%
So they have to follow
these very strict guidelines

00:15:36.540 --> 00:15:39.020 align:middle line:84%
to precisely handle
strange corner cases

00:15:39.020 --> 00:15:40.120 align:middle line:90%
and things like that.

00:15:40.120 --> 00:15:42.660 align:middle line:90%
And that document is on the web.

00:15:42.660 --> 00:15:44.010 align:middle line:90%
You can dig it up.

00:15:44.010 --> 00:15:46.260 align:middle line:84%
And it's actually very
instructive to read through it.

00:15:46.260 --> 00:15:47.820 align:middle line:84%
I think they put
it out on the web

00:15:47.820 --> 00:15:48.900 align:middle line:84%
because they wanted
to convince people

00:15:48.900 --> 00:15:50.620 align:middle line:84%
that they're going
to inordinate trouble

00:15:50.620 --> 00:15:52.990 align:middle line:84%
to make sure the rankings
are actually good.

00:15:52.990 --> 00:15:54.240 align:middle line:90%
Do you have a question, Colin?

00:15:54.240 --> 00:15:55.500 align:middle line:90%
No.

00:15:55.500 --> 00:15:59.980 align:middle line:90%
OK, all right, so back to this.

00:15:59.980 --> 00:16:01.580 align:middle line:90%
And how do you train this thing?

00:16:01.580 --> 00:16:03.180 align:middle line:90%
SGD.

00:16:03.180 --> 00:16:04.470 align:middle line:90%
Because you have a network.

00:16:04.470 --> 00:16:05.720 align:middle line:90%
It's coming up with an answer.

00:16:05.720 --> 00:16:08.640 align:middle line:84%
You have some way to know if
that answer is good or bad.

00:16:08.640 --> 00:16:10.260 align:middle line:90%
Better answers have lower loss.

00:16:10.260 --> 00:16:12.260 align:middle line:84%
Back propagation
through the network,

00:16:12.260 --> 00:16:16.400 align:middle line:84%
keep updating the weights,
and, boom, you're done.

00:16:16.400 --> 00:16:18.980 align:middle line:84%
And once you do that,
this reward model

00:16:18.980 --> 00:16:23.407 align:middle line:84%
can provide a numerical rating
for any instruction-answer pair.

00:16:23.407 --> 00:16:25.740 align:middle line:84%
You just give it an instruction,
you give it an answer--

00:16:25.740 --> 00:16:28.115 align:middle line:84%
could be a crappy answer, good
answer-- it just tells you

00:16:28.115 --> 00:16:32.210 align:middle line:84%
how good it is, which means,
in this case, for example,

00:16:32.210 --> 00:16:35.470 align:middle line:84%
maybe it's going to give
you a nice number, 1.5,

00:16:35.470 --> 00:16:38.490 align:middle line:90%
which is 1.5 for this answer.

00:16:38.490 --> 00:16:42.590 align:middle line:84%
But then a better
answer comes along, 3.2.

00:16:42.590 --> 00:16:46.110 align:middle line:84%
What we have done by doing this
whole thing, this modeling,

00:16:46.110 --> 00:16:48.990 align:middle line:84%
is that essentially
we have learned

00:16:48.990 --> 00:16:51.750 align:middle line:90%
how humans rank responses.

00:16:51.750 --> 00:16:53.630 align:middle line:84%
Because we can only
have humans rank

00:16:53.630 --> 00:16:56.590 align:middle line:84%
responses for some finite
number of questions.

00:16:56.590 --> 00:17:01.090 align:middle line:84%
What we really want to do is to
automate that ranking process

00:17:01.090 --> 00:17:03.070 align:middle line:84%
so that we can just do
it for tens of thousands

00:17:03.070 --> 00:17:04.990 align:middle line:90%
of questions really fast.

00:17:04.990 --> 00:17:06.710 align:middle line:84%
So we have essentially
built a model

00:17:06.710 --> 00:17:10.641 align:middle line:84%
of how humans rank things,
which is beautiful.

00:17:10.641 --> 00:17:12.849 align:middle line:84%
A lot of the stuff here is
all very self-referential,

00:17:12.849 --> 00:17:13.932 align:middle line:90%
which I find very elegant.

00:17:13.932 --> 00:17:17.150 align:middle line:84%
Anyway, so this can be used
to improve GPT-3 even further.

00:17:17.150 --> 00:17:20.530 align:middle line:84%
So we take the instruction
as before, we feed it,

00:17:20.530 --> 00:17:21.990 align:middle line:90%
it gives you some answer.

00:17:21.990 --> 00:17:25.349 align:middle line:84%
And then we feed this
instruction and the answer

00:17:25.349 --> 00:17:27.630 align:middle line:84%
to our newly-minted
reward model.

00:17:27.630 --> 00:17:29.770 align:middle line:90%
It gives us a numerical rating.

00:17:29.770 --> 00:17:31.550 align:middle line:90%
And then this is the key step.

00:17:31.550 --> 00:17:33.030 align:middle line:90%
We take this numerical rating.

00:17:33.030 --> 00:17:35.530 align:middle line:84%
And then we use
this rating to nudge

00:17:35.530 --> 00:17:39.530 align:middle line:84%
the internal weights of
GPT-3 in the right direction.

00:17:39.530 --> 00:17:44.930 align:middle line:84%
This nudging uses a technique
called reinforcement learning,

00:17:44.930 --> 00:17:46.510 align:middle line:84%
which, just in the
interest of time,

00:17:46.510 --> 00:17:49.010 align:middle line:84%
we can't get into
in this lecture.

00:17:49.010 --> 00:17:52.250 align:middle line:84%
But that's a technique you
use to nudge these things

00:17:52.250 --> 00:17:54.450 align:middle line:90%
in the right direction.

00:17:54.450 --> 00:17:56.220 align:middle line:90%
So that's what we do.

00:17:56.220 --> 00:17:57.470 align:middle line:90%
That's reinforcement learning.

00:17:57.470 --> 00:18:01.490 align:middle line:84%
We nudge it in the
right direction.

00:18:01.490 --> 00:18:07.850 align:middle line:84%
And OpenAI did this
with 31,000 questions.

00:18:07.850 --> 00:18:09.970 align:middle line:84%
Nudge, nudge, nudge,
nudge, nudge, nudge.

00:18:09.970 --> 00:18:13.690 align:middle line:84%
And when you do that, you
get GPT-3.5/InstructGPT.

00:18:13.690 --> 00:18:15.530 align:middle line:90%
OK?

00:18:15.530 --> 00:18:17.050 align:middle line:90%
That's it.

00:18:17.050 --> 00:18:20.865 align:middle line:84%
By the way, this step here
is called reinforcement

00:18:20.865 --> 00:18:22.490 align:middle line:84%
learning with human
feedback because we

00:18:22.490 --> 00:18:23.710 align:middle line:90%
use reinforcement learning.

00:18:23.710 --> 00:18:25.570 align:middle line:84%
And since humans rank
the answers which

00:18:25.570 --> 00:18:27.780 align:middle line:84%
led to the building
of the reward model,

00:18:27.780 --> 00:18:29.157 align:middle line:90%
we get human feedback.

00:18:29.157 --> 00:18:31.240 align:middle line:84%
That's reinforcement
learning with human feedback.

00:18:31.240 --> 00:18:33.900 align:middle line:90%
Yeah?

00:18:33.900 --> 00:18:37.180 align:middle line:84%
AUDIENCE: I have a question
regarding the type of questions

00:18:37.180 --> 00:18:38.420 align:middle line:90%
that they are using.

00:18:38.420 --> 00:18:42.380 align:middle line:84%
I can imagine maybe they are
very simple questions to answer.

00:18:42.380 --> 00:18:46.340 align:middle line:84%
Because I'm thinking now you
can ask GPT, for example,

00:18:46.340 --> 00:18:49.740 align:middle line:84%
respond to this as a pirate
or something like that.

00:18:49.740 --> 00:18:53.460 align:middle line:84%
It's going to be harder to train
if you have a bunch of questions

00:18:53.460 --> 00:18:55.640 align:middle line:84%
that are having
small instructions

00:18:55.640 --> 00:18:56.932 align:middle line:90%
and then there is the question.

00:18:56.932 --> 00:18:58.682 align:middle line:84%
RAMA RAMAKRISHNAN:
That's a good question.

00:18:58.682 --> 00:19:00.820 align:middle line:84%
So the quality of the
questions in the dataset

00:19:00.820 --> 00:19:02.480 align:middle line:90%
clearly is a big factor.

00:19:02.480 --> 00:19:05.300 align:middle line:84%
Because if you have
simplistic questions,

00:19:05.300 --> 00:19:08.380 align:middle line:84%
it won't be able to handle
complex questions later on.

00:19:08.380 --> 00:19:11.160 align:middle line:90%
So it's a good question.

00:19:11.160 --> 00:19:13.300 align:middle line:84%
So that actually
begs the question

00:19:13.300 --> 00:19:16.860 align:middle line:84%
of where did they get
these questions from?

00:19:16.860 --> 00:19:19.940 align:middle line:84%
So they actually got
it from their API.

00:19:19.940 --> 00:19:25.540 align:middle line:84%
So people were asking GPT-3 on
the API before it became 3.5.

00:19:25.540 --> 00:19:27.770 align:middle line:84%
The API was already
available, fully available,

00:19:27.770 --> 00:19:28.770 align:middle line:90%
commercially available.

00:19:28.770 --> 00:19:31.350 align:middle line:84%
A lot of people were building
products on it already by then.

00:19:31.350 --> 00:19:33.230 align:middle line:84%
And so they collected
all those questions

00:19:33.230 --> 00:19:34.930 align:middle line:90%
and filtered them for quality.

00:19:34.930 --> 00:19:36.970 align:middle line:84%
And that was the question
set that they used.

00:19:36.970 --> 00:19:38.710 align:middle line:84%
And then they
judiciously added to it

00:19:38.710 --> 00:19:40.493 align:middle line:90%
with human-created questions.

00:19:40.493 --> 00:19:41.910 align:middle line:84%
But they couldn't
do a lot of that

00:19:41.910 --> 00:19:43.428 align:middle line:84%
because it's
expensive to do that.

00:19:43.428 --> 00:19:45.470 align:middle line:84%
But collecting stuff that
somebody else is asking

00:19:45.470 --> 00:19:48.990 align:middle line:90%
your API already-- very easy.

00:19:48.990 --> 00:19:50.755 align:middle line:90%
Yeah, Tomaso?

00:19:50.755 --> 00:19:53.130 align:middle line:84%
AUDIENCE: This might be more
of a philosophical question,

00:19:53.130 --> 00:19:56.950 align:middle line:84%
but the human bias that's
present in the small subset

00:19:56.950 --> 00:19:59.030 align:middle line:84%
of human labelers
that they've chosen

00:19:59.030 --> 00:20:01.310 align:middle line:84%
gets eventually
compounded in this model

00:20:01.310 --> 00:20:04.710 align:middle line:84%
that we often consider as the
source of objective truth.

00:20:04.710 --> 00:20:06.070 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:20:06.070 --> 00:20:07.270 align:middle line:90%
Yeah, that's very true.

00:20:07.270 --> 00:20:11.230 align:middle line:84%
I think the reward model has
probably very faithfully learned

00:20:11.230 --> 00:20:13.630 align:middle line:84%
all the biases of the
human labelers, which

00:20:13.630 --> 00:20:17.470 align:middle line:84%
is why they have these
very complex frameworks

00:20:17.470 --> 00:20:20.330 align:middle line:84%
and guidelines to try to
prevent the bias from happening,

00:20:20.330 --> 00:20:21.250 align:middle line:90%
to mitigate it.

00:20:21.250 --> 00:20:23.835 align:middle line:84%
So for example, they might
give the same question and set

00:20:23.835 --> 00:20:25.210 align:middle line:84%
of instead of
possible answers to

00:20:25.210 --> 00:20:27.010 align:middle line:90%
many, many different labelers.

00:20:27.010 --> 00:20:31.790 align:middle line:84%
And only if people pick the
same ranking, they might use it.

00:20:31.790 --> 00:20:35.290 align:middle line:84%
So that at least inter-labeller
bias can be minimized.

00:20:35.290 --> 00:20:38.310 align:middle line:84%
But if everybody is biased
in the same direction,

00:20:38.310 --> 00:20:40.130 align:middle line:84%
it won't protect
you against that.

00:20:40.130 --> 00:20:41.930 align:middle line:84%
So in general,
there's a whole work

00:20:41.930 --> 00:20:44.050 align:middle line:84%
that's being done to try
to de-bias these things

00:20:44.050 --> 00:20:46.367 align:middle line:84%
and build them without
too much bias in them.

00:20:46.367 --> 00:20:48.450 align:middle line:84%
It's like a whole world
unto itself, which we just

00:20:48.450 --> 00:20:49.630 align:middle line:90%
don't have time to get into.

00:20:49.630 --> 00:20:54.050 align:middle line:90%


00:20:54.050 --> 00:20:55.810 align:middle line:84%
AUDIENCE: Depending
on the medium that's

00:20:55.810 --> 00:20:57.602 align:middle line:84%
being returned by these
models, would there

00:20:57.602 --> 00:20:58.870 align:middle line:90%
be more than one reward model?

00:20:58.870 --> 00:20:59.770 align:middle line:90%
Because isn't that what Gemini--

00:20:59.770 --> 00:21:01.810 align:middle line:84%
RAMA RAMAKRISHNAN: Sorry, would
there be more than one what?

00:21:01.810 --> 00:21:02.768 align:middle line:90%
AUDIENCE: Reward model.

00:21:02.768 --> 00:21:05.050 align:middle line:84%
Because isn't this what
Gemini is running into issues

00:21:05.050 --> 00:21:07.350 align:middle line:84%
with right now with
their image generation,

00:21:07.350 --> 00:21:09.243 align:middle line:90%
is the bias that they hide?

00:21:09.243 --> 00:21:11.910 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, so the
Gemini business that's going on,

00:21:11.910 --> 00:21:14.530 align:middle line:90%
it's unclear what's causing it.

00:21:14.530 --> 00:21:18.090 align:middle line:84%
It may be in this step, maybe
they were a little overzealous

00:21:18.090 --> 00:21:20.290 align:middle line:84%
in preventing certain
things from happening.

00:21:20.290 --> 00:21:25.620 align:middle line:84%
Some of these systems, they will
actually intercept the question

00:21:25.620 --> 00:21:28.020 align:middle line:84%
that you ask and then
route it differently

00:21:28.020 --> 00:21:30.860 align:middle line:84%
based on what they sense is
sitting around in the question.

00:21:30.860 --> 00:21:32.280 align:middle line:84%
So there could be
pre-processing,

00:21:32.280 --> 00:21:34.155 align:middle line:84%
post-processing, a lot
of stuff that goes on.

00:21:34.155 --> 00:21:36.402 align:middle line:84%
So unclear to me
where in the pipeline.

00:21:36.402 --> 00:21:37.860 align:middle line:84%
And it could be
more than one place

00:21:37.860 --> 00:21:39.320 align:middle line:90%
these things may be entering.

00:21:39.320 --> 00:21:42.300 align:middle line:84%
So yes, here may very well
be where it actually enters

00:21:42.300 --> 00:21:45.200 align:middle line:84%
a situation where
people are told,

00:21:45.200 --> 00:21:47.380 align:middle line:84%
if you see any of
this kind of answer,

00:21:47.380 --> 00:21:49.847 align:middle line:84%
down rank it or
don't up-rank it.

00:21:49.847 --> 00:21:52.180 align:middle line:84%
And then it learns that ranking
very faithfully and then

00:21:52.180 --> 00:21:54.580 align:middle line:84%
proceeds to apply it where
it should not be applied.

00:21:54.580 --> 00:21:56.060 align:middle line:90%
So that does happen.

00:21:56.060 --> 00:21:57.320 align:middle line:90%
Jocelyn, you had a question.

00:21:57.320 --> 00:21:59.980 align:middle line:90%


00:21:59.980 --> 00:22:01.860 align:middle line:84%
AUDIENCE: I think I
still don't totally

00:22:01.860 --> 00:22:05.020 align:middle line:84%
understand why when I ask
Chat-GPT a question, even

00:22:05.020 --> 00:22:07.420 align:middle line:84%
in a lengthy response,
it doesn't wander away

00:22:07.420 --> 00:22:09.160 align:middle line:84%
from the topic that
I'm asking about.

00:22:09.160 --> 00:22:12.400 align:middle line:84%
And so understanding that
it's predicting each word,

00:22:12.400 --> 00:22:14.540 align:middle line:84%
it's taking a random
walk from one word

00:22:14.540 --> 00:22:15.800 align:middle line:90%
to the next in some sense.

00:22:15.800 --> 00:22:18.020 align:middle line:84%
RAMA RAMAKRISHNAN: But
each word it utters now

00:22:18.020 --> 00:22:20.563 align:middle line:84%
becomes part of the input
to the next word it utters.

00:22:20.563 --> 00:22:21.230 align:middle line:90%
AUDIENCE: Right.

00:22:21.230 --> 00:22:21.990 align:middle line:84%
RAMA RAMAKRISHNAN:
So it's not truly

00:22:21.990 --> 00:22:23.490 align:middle line:90%
a random walk in that sense.

00:22:23.490 --> 00:22:24.990 align:middle line:84%
So the next step
is not independent

00:22:24.990 --> 00:22:26.990 align:middle line:90%
of the previous step.

00:22:26.990 --> 00:22:28.530 align:middle line:84%
It depends on the
journey so far.

00:22:28.530 --> 00:22:30.988 align:middle line:84%
So it's going to try to be very
consistent with the journey

00:22:30.988 --> 00:22:31.688 align:middle line:90%
so far.

00:22:31.688 --> 00:22:32.230 align:middle line:90%
AUDIENCE: OK.

00:22:32.230 --> 00:22:35.310 align:middle line:90%


00:22:35.310 --> 00:22:39.830 align:middle line:84%
Does this part, fine-tuning
it on these question

00:22:39.830 --> 00:22:41.710 align:middle line:84%
and answer sets, does
this play some role

00:22:41.710 --> 00:22:45.670 align:middle line:84%
in it being able to constrain
itself and not meander away?

00:22:45.670 --> 00:22:47.170 align:middle line:84%
RAMA RAMAKRISHNAN:
I don't think so.

00:22:47.170 --> 00:22:47.270 align:middle line:90%
AUDIENCE: OK.

00:22:47.270 --> 00:22:49.437 align:middle line:84%
RAMA RAMAKRISHNAN: I think
this is more to make sure

00:22:49.437 --> 00:22:53.270 align:middle line:84%
that the weights generally tend
to produce the right answer.

00:22:53.270 --> 00:22:55.910 align:middle line:84%
Now, one of the things
that is possible

00:22:55.910 --> 00:22:58.590 align:middle line:84%
is that let's say
I'm a ranker and I'm

00:22:58.590 --> 00:23:01.350 align:middle line:84%
looking at a few
different answers,

00:23:01.350 --> 00:23:03.710 align:middle line:84%
I have to figure out if
the answer is helpful,

00:23:03.710 --> 00:23:09.190 align:middle line:84%
if it is accurate, if it is
non-toxic, things like that.

00:23:09.190 --> 00:23:12.310 align:middle line:84%
And part of the rubric for
evaluating these answers

00:23:12.310 --> 00:23:14.890 align:middle line:90%
could be their coherence.

00:23:14.890 --> 00:23:17.670 align:middle line:84%
So it could also be
that they are saying,

00:23:17.670 --> 00:23:19.290 align:middle line:84%
short, coherent
answers are better

00:23:19.290 --> 00:23:21.250 align:middle line:90%
than long, coherent answers.

00:23:21.250 --> 00:23:23.420 align:middle line:84%
But once you adjust for
length, maybe coherence

00:23:23.420 --> 00:23:24.170 align:middle line:90%
is more important.

00:23:24.170 --> 00:23:25.470 align:middle line:84%
There could be any
number of these things,

00:23:25.470 --> 00:23:26.807 align:middle line:90%
so it could play a role in that.

00:23:26.807 --> 00:23:28.390 align:middle line:84%
AUDIENCE: So just
one small follow-up.

00:23:28.390 --> 00:23:30.368 align:middle line:84%
So in other words,
when it's learning

00:23:30.368 --> 00:23:31.910 align:middle line:84%
from these
question-and-answer pairs,

00:23:31.910 --> 00:23:33.537 align:middle line:84%
it's able to look at
the whole response

00:23:33.537 --> 00:23:35.370 align:middle line:84%
and learn something
about the whole response

00:23:35.370 --> 00:23:37.350 align:middle line:84%
rather than just one
word at a time, right?

00:23:37.350 --> 00:23:38.475 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:23:38.475 --> 00:23:40.530 align:middle line:84%
Yeah, the entire
question is being ranked.

00:23:40.530 --> 00:23:42.610 align:middle line:90%
Correct, correct.

00:23:42.610 --> 00:23:43.810 align:middle line:90%
Yeah?

00:23:43.810 --> 00:23:46.010 align:middle line:84%
AUDIENCE: On a
related note, when

00:23:46.010 --> 00:23:48.230 align:middle line:84%
it's generating a
new word on topic,

00:23:48.230 --> 00:23:51.490 align:middle line:84%
does the attention pertain
to that entire prior text?

00:23:51.490 --> 00:23:54.290 align:middle line:84%
Or can you have
traveling attention?

00:23:54.290 --> 00:23:57.050 align:middle line:90%
So last five word attention--

00:23:57.050 --> 00:24:01.030 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, the
short answer is, yeah, you can.

00:24:01.030 --> 00:24:02.890 align:middle line:84%
It's called
sliding-window attention.

00:24:02.890 --> 00:24:04.210 align:middle line:90%
It can be done.

00:24:04.210 --> 00:24:07.010 align:middle line:84%
They typically tend to do
it not so much because they

00:24:07.010 --> 00:24:09.830 align:middle line:84%
want to focus more
on the recent words,

00:24:09.830 --> 00:24:13.530 align:middle line:84%
but more because it actually
makes it very compute efficient.

00:24:13.530 --> 00:24:14.630 align:middle line:90%
That's why they do it.

00:24:14.630 --> 00:24:16.297 align:middle line:84%
So it's called
sliding-window attention.

00:24:16.297 --> 00:24:17.940 align:middle line:90%
You can Google it.

00:24:17.940 --> 00:24:19.740 align:middle line:84%
AUDIENCE: So normally
it's full attention?

00:24:19.740 --> 00:24:23.313 align:middle line:84%
RAMA RAMAKRISHNAN: Normally,
default is full attention.

00:24:23.313 --> 00:24:26.820 align:middle line:90%
So that's what they did.

00:24:26.820 --> 00:24:28.800 align:middle line:84%
And by the way, as I
think you pointed out,

00:24:28.800 --> 00:24:29.840 align:middle line:90%
that's exactly what's going on.

00:24:29.840 --> 00:24:32.048 align:middle line:84%
You're training the reward
model with these thumbs up

00:24:32.048 --> 00:24:33.420 align:middle line:90%
and thumbs down.

00:24:33.420 --> 00:24:34.860 align:middle line:90%
Hold on the questions.

00:24:34.860 --> 00:24:39.680 align:middle line:84%
And so if you give it the
same question to GPT-3.5,

00:24:39.680 --> 00:24:45.620 align:middle line:84%
InstructGPT, amazing answer,
like night and day difference.

00:24:45.620 --> 00:24:48.206 align:middle line:90%
Amazingly good answer.

00:24:48.206 --> 00:24:51.100 align:middle line:84%
And then to go from
3.5 to ChatGPT,

00:24:51.100 --> 00:24:53.340 align:middle line:84%
they basically followed
the exact same playbook,

00:24:53.340 --> 00:24:57.283 align:middle line:84%
except that because they
wanted to have a chatbot--

00:24:57.283 --> 00:24:58.700 align:middle line:84%
meaning something
that could carry

00:24:58.700 --> 00:25:00.260 align:middle line:84%
on a question-answer,
question-answer pair.

00:25:00.260 --> 00:25:02.060 align:middle line:84%
As opposed to just a
single question and answer,

00:25:02.060 --> 00:25:04.143 align:middle line:84%
they wanted question-answer,
question-answer, like

00:25:04.143 --> 00:25:04.820 align:middle line:90%
conversation.

00:25:04.820 --> 00:25:06.720 align:middle line:84%
They trained it
on conversations.

00:25:06.720 --> 00:25:08.260 align:middle line:90%
That's it.

00:25:08.260 --> 00:25:11.560 align:middle line:84%
Instead of training it on
instruction-answer data,

00:25:11.560 --> 00:25:13.660 align:middle line:84%
they trained it on
instruction-answer,

00:25:13.660 --> 00:25:15.750 align:middle line:84%
instruction-answer,
instruction-answer,

00:25:15.750 --> 00:25:19.830 align:middle line:84%
a sequence of such things, which
are strung into a conversation.

00:25:19.830 --> 00:25:20.410 align:middle line:90%
That's it.

00:25:20.410 --> 00:25:25.030 align:middle line:84%
That is the only difference
to go from 3.5 to ChatGPT.

00:25:25.030 --> 00:25:26.990 align:middle line:84%
Now ChatGPT, given you
do that, it's giving you

00:25:26.990 --> 00:25:28.370 align:middle line:90%
a much nicer response.

00:25:28.370 --> 00:25:30.253 align:middle line:84%
And then you can ask
a follow-on question.

00:25:30.253 --> 00:25:31.420 align:middle line:90%
Can you make it more formal?

00:25:31.420 --> 00:25:32.710 align:middle line:84%
Boom, it gives you
a nice response.

00:25:32.710 --> 00:25:34.377 align:middle line:84%
Because now it knows
about conversations

00:25:34.377 --> 00:25:36.550 align:middle line:84%
because it's been trained
on conversational data.

00:25:36.550 --> 00:25:38.230 align:middle line:90%
So that's it.

00:25:38.230 --> 00:25:39.750 align:middle line:90%
That's how they built ChatGPT.

00:25:39.750 --> 00:25:41.950 align:middle line:84%
And all the things we
are seeing later on

00:25:41.950 --> 00:25:44.853 align:middle line:84%
are all continuations of
this sort of approach.

00:25:44.853 --> 00:25:46.770 align:middle line:84%
So I'll pause for a
couple of quick questions.

00:25:46.770 --> 00:25:47.650 align:middle line:90%
Swathi, you had a question.

00:25:47.650 --> 00:25:49.290 align:middle line:84%
Then we'll go to
you and then to you.

00:25:49.290 --> 00:25:50.510 align:middle line:90%
Yeah?

00:25:50.510 --> 00:25:52.790 align:middle line:84%
AUDIENCE: So does that
make a difference,

00:25:52.790 --> 00:25:57.910 align:middle line:84%
if a new question-answer
pair or a new training data

00:25:57.910 --> 00:26:00.630 align:middle line:84%
comes early in the
building of the model

00:26:00.630 --> 00:26:03.810 align:middle line:84%
or later in the building of the
model, 7 billion parameters.

00:26:03.810 --> 00:26:04.750 align:middle line:90%
Does that make a--

00:26:04.750 --> 00:26:06.170 align:middle line:84%
RAMA RAMAKRISHNAN: You mean
the order of the questions,

00:26:06.170 --> 00:26:06.910 align:middle line:90%
does it matter?

00:26:06.910 --> 00:26:11.090 align:middle line:84%
AUDIENCE: So I might have, let's
say 5,000 images to start with.

00:26:11.090 --> 00:26:13.830 align:middle line:84%
Now, after my model is
trained and developed,

00:26:13.830 --> 00:26:17.390 align:middle line:84%
now I have a new use
case that has come in.

00:26:17.390 --> 00:26:19.610 align:middle line:84%
Will that make a difference
if I set it in now?

00:26:19.610 --> 00:26:21.402 align:middle line:84%
RAMA RAMAKRISHNAN: So
if you have a new use

00:26:21.402 --> 00:26:24.570 align:middle line:84%
case for which you want to
essentially adapt the model,

00:26:24.570 --> 00:26:26.430 align:middle line:84%
there's a whole set
of techniques you use,

00:26:26.430 --> 00:26:27.480 align:middle line:84%
which is going to
be the next section.

00:26:27.480 --> 00:26:28.930 align:middle line:84%
AUDIENCE: But it's
not [INAUDIBLE]

00:26:28.930 --> 00:26:31.472 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, because
what you have out of the box

00:26:31.472 --> 00:26:33.130 align:middle line:84%
is just a generally
good chatbot.

00:26:33.130 --> 00:26:35.463 align:middle line:84%
It knows about a lot of stuff
because we've been trained

00:26:35.463 --> 00:26:36.870 align:middle line:90%
on those 30 billion sentences.

00:26:36.870 --> 00:26:38.662 align:middle line:84%
It can answer a lot of
questions reasonably

00:26:38.662 --> 00:26:40.770 align:middle line:84%
well using common sense
and world knowledge.

00:26:40.770 --> 00:26:43.750 align:middle line:84%
But any specific use case like
medical and so on and so forth,

00:26:43.750 --> 00:26:44.483 align:middle line:90%
it may not know.

00:26:44.483 --> 00:26:46.650 align:middle line:84%
So you'll need to adapt it
to your particular unique

00:26:46.650 --> 00:26:47.510 align:middle line:90%
situation.

00:26:47.510 --> 00:26:49.490 align:middle line:90%
And that's coming.

00:26:49.490 --> 00:26:52.010 align:middle line:90%
All right, yes,

00:26:52.010 --> 00:26:55.730 align:middle line:84%
AUDIENCE: What determines
if a whole conversation is

00:26:55.730 --> 00:26:58.650 align:middle line:84%
ranked positively versus
just a specific answer?

00:26:58.650 --> 00:27:01.690 align:middle line:84%
proliferating in
your question...

00:27:01.690 --> 00:27:03.867 align:middle line:84%
If the first answer doesn't
get a positive response,

00:27:03.867 --> 00:27:05.950 align:middle line:84%
but then after the follow-up,
the second one does,

00:27:05.950 --> 00:27:06.820 align:middle line:90%
does that indicate--

00:27:06.820 --> 00:27:07.650 align:middle line:84%
RAMA RAMAKRISHNAN:
Correct, exactly.

00:27:07.650 --> 00:27:09.108 align:middle line:84%
See, if you're a
human and you read

00:27:09.108 --> 00:27:11.490 align:middle line:84%
the transcript of an
exchange between two people

00:27:11.490 --> 00:27:13.845 align:middle line:84%
and I'm giving you two
exchanges, which all start

00:27:13.845 --> 00:27:15.220 align:middle line:84%
with the same
question, you'll be

00:27:15.220 --> 00:27:17.520 align:middle line:84%
able to assess which one
is a better transcript.

00:27:17.520 --> 00:27:19.580 align:middle line:84%
That's basically
what's going on.

00:27:19.580 --> 00:27:21.340 align:middle line:90%
There was something here, right?

00:27:21.340 --> 00:27:22.403 align:middle line:90%
Yeah?

00:27:22.403 --> 00:27:23.820 align:middle line:84%
AUDIENCE: So I was
wondering, when

00:27:23.820 --> 00:27:26.120 align:middle line:84%
you ask Chat-GPT a
question, very often

00:27:26.120 --> 00:27:27.960 align:middle line:90%
it sounds kind of robotic.

00:27:27.960 --> 00:27:30.900 align:middle line:84%
You could tell that something
was written by ChatGPT, not

00:27:30.900 --> 00:27:32.380 align:middle line:90%
by an actual person.

00:27:32.380 --> 00:27:37.140 align:middle line:84%
Do you think that comes from
the reinforcement learning part?

00:27:37.140 --> 00:27:40.180 align:middle line:84%
Or where do you think it
comes from in this situation?

00:27:40.180 --> 00:27:41.000 align:middle line:84%
RAMA RAMAKRISHNAN:
It's a good question.

00:27:41.000 --> 00:27:41.542 align:middle line:90%
I don't know.

00:27:41.542 --> 00:27:43.460 align:middle line:84%
Because I know that
part of the evaluation,

00:27:43.460 --> 00:27:45.420 align:middle line:84%
the ranking rubric
that are used,

00:27:45.420 --> 00:27:50.340 align:middle line:84%
is to favor responses which
sound more human-like more

00:27:50.340 --> 00:27:51.540 align:middle line:90%
than robot-like.

00:27:51.540 --> 00:27:54.180 align:middle line:84%
So if anything, I'm hoping that
reinforcement learning would

00:27:54.180 --> 00:27:56.597 align:middle line:84%
actually make it sound more
human-like because the rankers

00:27:56.597 --> 00:27:57.800 align:middle line:90%
would have prioritized that.

00:27:57.800 --> 00:28:01.020 align:middle line:84%
So if it still comes
up with robotic stuff,

00:28:01.020 --> 00:28:03.920 align:middle line:84%
it's something else
that's going on.

00:28:03.920 --> 00:28:07.020 align:middle line:84%
Maybe there are a lot of text on
the internet is not literature.

00:28:07.020 --> 00:28:08.600 align:middle line:84%
It's just people
writing some crap.

00:28:08.600 --> 00:28:10.720 align:middle line:90%
So it could be that.

00:28:10.720 --> 00:28:13.085 align:middle line:90%
Yeah?

00:28:13.085 --> 00:28:14.960 align:middle line:84%
AUDIENCE: How much of
this instruction tuning

00:28:14.960 --> 00:28:17.560 align:middle line:84%
or conversation tuning
is happening in real-time

00:28:17.560 --> 00:28:18.610 align:middle line:90%
within a conversation?

00:28:18.610 --> 00:28:19.860 align:middle line:90%
RAMA RAMAKRISHNAN: None of it.

00:28:19.860 --> 00:28:20.735 align:middle line:90%
AUDIENCE: None of it.

00:28:20.735 --> 00:28:23.480 align:middle line:84%
So as you give
feedback to the model,

00:28:23.480 --> 00:28:25.480 align:middle line:84%
it's just basically
regenerating it.

00:28:25.480 --> 00:28:27.780 align:middle line:84%
I don't like that answer,
come up with something else.

00:28:27.780 --> 00:28:29.720 align:middle line:84%
RAMA RAMAKRISHNAN: No, it's
not doing it in real time.

00:28:29.720 --> 00:28:31.760 align:middle line:84%
Basically, whatever
signals you're giving it

00:28:31.760 --> 00:28:33.135 align:middle line:84%
with these thumbs
up, thumbs down

00:28:33.135 --> 00:28:35.400 align:middle line:84%
business, that gets added
to the training logs.

00:28:35.400 --> 00:28:39.280 align:middle line:84%
And they periodically
will retrain it.

00:28:39.280 --> 00:28:41.360 align:middle line:84%
So by the way,
this is instruction

00:28:41.360 --> 00:28:42.460 align:middle line:90%
tuning in a nutshell.

00:28:42.460 --> 00:28:43.800 align:middle line:84%
And I want to point
that out-- and you

00:28:43.800 --> 00:28:45.300 align:middle line:84%
don't have to read
the whole thing--

00:28:45.300 --> 00:28:46.880 align:middle line:84%
but just to quickly
point out this

00:28:46.880 --> 00:28:49.777 align:middle line:84%
was where we had to
have human involvement.

00:28:49.777 --> 00:28:52.360 align:middle line:84%
The first step, writing a lot
of responses to these questions,

00:28:52.360 --> 00:28:55.200 align:middle line:90%
and then ranking the answers.

00:28:55.200 --> 00:28:58.280 align:middle line:84%
So these two are still
human labor intensive.

00:28:58.280 --> 00:29:01.240 align:middle line:84%
Now, it turns out you can
actually use helper LLMs

00:29:01.240 --> 00:29:03.363 align:middle line:90%
to automate this too.

00:29:03.363 --> 00:29:05.780 align:middle line:84%
This is not what OpenAI did
in the beginning with ChatGPT.

00:29:05.780 --> 00:29:07.613 align:middle line:84%
But now you can do it
this way because there

00:29:07.613 --> 00:29:09.730 align:middle line:84%
are lots of really good
LLMs available for you

00:29:09.730 --> 00:29:11.280 align:middle line:84%
to automate many
of these things.

00:29:11.280 --> 00:29:13.030 align:middle line:84%
We don't have time,
but if you're curious,

00:29:13.030 --> 00:29:14.405 align:middle line:84%
I had a little
blog post on this.

00:29:14.405 --> 00:29:14.950 align:middle line:90%
Check it out.

00:29:14.950 --> 00:29:18.782 align:middle line:84%
OK, so now we come
to the question of,

00:29:18.782 --> 00:29:21.450 align:middle line:84%
well, if you want to take
a base LLM like GPT-3

00:29:21.450 --> 00:29:24.390 align:middle line:84%
and make it useful and
respond to instructions,

00:29:24.390 --> 00:29:26.330 align:middle line:84%
we have seen that
we had to adapt it

00:29:26.330 --> 00:29:28.530 align:middle line:84%
with high-quality
instruction-answer data

00:29:28.530 --> 00:29:30.690 align:middle line:84%
using Supervised Fine-Tuning
and Reinforcement

00:29:30.690 --> 00:29:31.910 align:middle line:90%
Learning with Human Feedback.

00:29:31.910 --> 00:29:37.370 align:middle line:84%
That's what made GPT-3 actually
useful and became ChatGPT.

00:29:37.370 --> 00:29:39.900 align:middle line:84%
By the same token, this
holds true more generally.

00:29:39.900 --> 00:29:41.650 align:middle line:84%
If you want to take a
large language model

00:29:41.650 --> 00:29:44.010 align:middle line:84%
and make it useful for a
medical use case, a legal use

00:29:44.010 --> 00:29:46.570 align:middle line:84%
case, or some other
narrow business use case,

00:29:46.570 --> 00:29:50.650 align:middle line:84%
you have to adapt it with
business domain-specific data.

00:29:50.650 --> 00:29:53.450 align:middle line:84%
And so let's look at
techniques for doing so.

00:29:53.450 --> 00:29:56.010 align:middle line:84%
All right, so adaptation
is the rough name

00:29:56.010 --> 00:29:58.690 align:middle line:84%
for the process of taking
a base large language model

00:29:58.690 --> 00:30:01.570 align:middle line:84%
and tailoring it for
your particular use case.

00:30:01.570 --> 00:30:04.170 align:middle line:84%
And so there is this ladder
of things you can do.

00:30:04.170 --> 00:30:06.150 align:middle line:84%
And we're going to look
at every one of them.

00:30:06.150 --> 00:30:09.180 align:middle line:84%
So you can do this thing called
zero-shot prompting, which

00:30:09.180 --> 00:30:13.277 align:middle line:84%
is just you literally ask
the LLM nicely, clearly,

00:30:13.277 --> 00:30:13.860 align:middle line:90%
what you want.

00:30:13.860 --> 00:30:15.723 align:middle line:84%
And maybe it'll
just give it to you.

00:30:15.723 --> 00:30:17.140 align:middle line:84%
And this is the
use case we're all

00:30:17.140 --> 00:30:19.300 align:middle line:90%
used to in the web interface.

00:30:19.300 --> 00:30:21.660 align:middle line:84%
You can also do
something called few-shot

00:30:21.660 --> 00:30:23.913 align:middle line:84%
prompting where you
ask it something,

00:30:23.913 --> 00:30:26.580 align:middle line:84%
and you also give a few examples
of the kind of things you want.

00:30:26.580 --> 00:30:28.620 align:middle line:90%
And that helps it a great deal.

00:30:28.620 --> 00:30:30.660 align:middle line:84%
And then there's
this thing called

00:30:30.660 --> 00:30:32.840 align:middle line:84%
retrieval-augmented
generation and fine-tuning.

00:30:32.840 --> 00:30:33.580 align:middle line:90%
And we'll look at all of them.

00:30:33.580 --> 00:30:35.622 align:middle line:84%
And I'll explain all these
things as we go along.

00:30:35.622 --> 00:30:38.540 align:middle line:84%
So let's start with
zero-shot prompting.

00:30:38.540 --> 00:30:42.460 align:middle line:84%
By the way, the word "shot"
here is a synonym for "example."

00:30:42.460 --> 00:30:44.983 align:middle line:84%
So zero example
prompting-- you literally

00:30:44.983 --> 00:30:46.900 align:middle line:84%
ask in the prompt what
you want without giving

00:30:46.900 --> 00:30:48.540 align:middle line:90%
even a single example.

00:30:48.540 --> 00:30:53.405 align:middle line:84%
And so let's say we want to look
at product reviews and build

00:30:53.405 --> 00:30:55.780 align:middle line:84%
a detector to figure out if
the product review contains--

00:30:55.780 --> 00:30:57.820 align:middle line:84%
not sentiment, that's
kind of boring--

00:30:57.820 --> 00:31:00.140 align:middle line:84%
whether it contains
some description

00:31:00.140 --> 00:31:02.940 align:middle line:84%
of a potential
product defect or not.

00:31:02.940 --> 00:31:06.920 align:middle line:84%
And so here is something I
actually pulled off Wayfair,

00:31:06.920 --> 00:31:08.560 align:middle line:90%
with apologies to Wayfair.

00:31:08.560 --> 00:31:10.560 align:middle line:84%
It says here the curve
of the back of the chair

00:31:10.560 --> 00:31:13.660 align:middle line:84%
does not leave enough
room to sit comfortably.

00:31:13.660 --> 00:31:16.620 align:middle line:84%
Sounds like a kind of a
defectish kind of thing.

00:31:16.620 --> 00:31:19.760 align:middle line:84%
So back in the day, you'd have
collected all these reviews

00:31:19.760 --> 00:31:22.760 align:middle line:84%
and built a special-purpose
NLP-based classifier to figure

00:31:22.760 --> 00:31:24.120 align:middle line:90%
out defect, yes or no.

00:31:24.120 --> 00:31:27.520 align:middle line:84%
Here, you can literally just
feed this thing into GPT-3

00:31:27.520 --> 00:31:30.360 align:middle line:84%
and ask it, tell me if a product
defect is being described

00:31:30.360 --> 00:31:31.540 align:middle line:90%
in this product review.

00:31:31.540 --> 00:31:33.140 align:middle line:84%
And then, "the curve of the
back," and, boom, boom, boom.

00:31:33.140 --> 00:31:35.880 align:middle line:84%
And then it comes back and says,
yep, that's a product defect.

00:31:35.880 --> 00:31:37.340 align:middle line:90%
So this is zero shot.

00:31:37.340 --> 00:31:40.340 align:middle line:84%
You just ask a question,
you get the answer back.

00:31:40.340 --> 00:31:42.200 align:middle line:84%
And it actually works
remarkably well.

00:31:42.200 --> 00:31:44.400 align:middle line:84%
And the better models,
the bigger models

00:31:44.400 --> 00:31:47.080 align:middle line:84%
tend to be much better than
the smaller, simpler models

00:31:47.080 --> 00:31:48.960 align:middle line:90%
for doing zero shot.

00:31:48.960 --> 00:31:52.600 align:middle line:84%
All right, now when you adapt
an LLM to a specific task,

00:31:52.600 --> 00:31:54.920 align:middle line:84%
obviously you need to
carefully design the prompt.

00:31:54.920 --> 00:31:57.200 align:middle line:84%
As you folks know, this is
called prompt engineering.

00:31:57.200 --> 00:31:59.742 align:middle line:84%
And we're not going to spend
much time on prompt engineering,

00:31:59.742 --> 00:32:01.600 align:middle line:84%
except I just want to
give a simple example.

00:32:01.600 --> 00:32:04.170 align:middle line:84%
So if you actually ask
ChatGPT this question,

00:32:04.170 --> 00:32:06.810 align:middle line:84%
"What is the fifth
word of the sentence?"

00:32:06.810 --> 00:32:09.890 align:middle line:84%
Very often it will
give the wrong answer.

00:32:09.890 --> 00:32:12.550 align:middle line:84%
It's very strange why it
can't get this question right.

00:32:12.550 --> 00:32:13.850 align:middle line:90%
It's a very simple question.

00:32:13.850 --> 00:32:17.295 align:middle line:84%
So, the fifth word of
the sentence is "is."

00:32:17.295 --> 00:32:19.920 align:middle line:84%
Sometimes it gets it right, but
very often, it'll get it wrong.

00:32:19.920 --> 00:32:21.930 align:middle line:84%
But now you can do a
little prompt engineering

00:32:21.930 --> 00:32:23.230 align:middle line:90%
and it'll always get it right.

00:32:23.230 --> 00:32:25.790 align:middle line:84%
So for example, you can say,
I'll give you a sentence.

00:32:25.790 --> 00:32:27.390 align:middle line:84%
First, list all the words
that are in the sentence.

00:32:27.390 --> 00:32:28.310 align:middle line:90%
Then, tell me the fifth word.

00:32:28.310 --> 00:32:29.352 align:middle line:90%
OK, here is the sentence.

00:32:29.352 --> 00:32:31.280 align:middle line:84%
Boom, boom, boom,
it gets it right.

00:32:31.280 --> 00:32:33.850 align:middle line:84%
So it's an example of,
you can help it along

00:32:33.850 --> 00:32:36.850 align:middle line:84%
by being very, very prescriptive
as to what you want it to do

00:32:36.850 --> 00:32:38.270 align:middle line:90%
and break down all the steps.

00:32:38.270 --> 00:32:39.590 align:middle line:90%
Don't make it guess things.

00:32:39.590 --> 00:32:41.913 align:middle line:90%
It does a great job.

00:32:41.913 --> 00:32:43.330 align:middle line:84%
And there are lots
of other tricks

00:32:43.330 --> 00:32:46.250 align:middle line:84%
people have figured out over
the last couple of years.

00:32:46.250 --> 00:32:48.750 align:middle line:84%
For a long time, this was
pretty hot, where you say,

00:32:48.750 --> 00:32:50.290 align:middle line:90%
"let's think step by step."

00:32:50.290 --> 00:32:52.910 align:middle line:84%
You give it a question and
say, let's think step by step,

00:32:52.910 --> 00:32:54.368 align:middle line:84%
it actually gives
you a better shot

00:32:54.368 --> 00:32:56.810 align:middle line:84%
at giving you a good answer
back, an accurate answer back.

00:32:56.810 --> 00:32:58.268 align:middle line:84%
Now, this kind of
thing is actually

00:32:58.268 --> 00:33:00.470 align:middle line:90%
already baked into the LLMs.

00:33:00.470 --> 00:33:03.440 align:middle line:84%
So when you ask a question
to ChatGPT, your question,

00:33:03.440 --> 00:33:07.377 align:middle line:84%
your prompt gets appended to
what's called the system prompt.

00:33:07.377 --> 00:33:08.960 align:middle line:84%
And the whole thing
goes into the LLM.

00:33:08.960 --> 00:33:10.293 align:middle line:90%
You never see the system prompt.

00:33:10.293 --> 00:33:13.320 align:middle line:84%
And the system prompt is telling
ChatGPT, think step by step,

00:33:13.320 --> 00:33:17.020 align:middle line:84%
take your time, don't blurt
out an answer, stuff like that.

00:33:17.020 --> 00:33:18.293 align:middle line:90%
And you can just Google it.

00:33:18.293 --> 00:33:19.960 align:middle line:84%
The system prompts
have been jailbroken.

00:33:19.960 --> 00:33:21.085 align:middle line:90%
You can find it on the web.

00:33:21.085 --> 00:33:24.020 align:middle line:90%


00:33:24.020 --> 00:33:25.420 align:middle line:90%
And this is funny.

00:33:25.420 --> 00:33:27.080 align:middle line:84%
This came out maybe
a month or two ago.

00:33:27.080 --> 00:33:28.860 align:middle line:84%
It says, apparently, "Take
a deep breath and work

00:33:28.860 --> 00:33:31.140 align:middle line:84%
on the problem step by step"
works better than saying,

00:33:31.140 --> 00:33:33.100 align:middle line:90%
"Work on it step by step."

00:33:33.100 --> 00:33:35.180 align:middle line:84%
And then more
recently, I literally

00:33:35.180 --> 00:33:37.920 align:middle line:84%
read this two nights ago,
apparently, if you tell it,

00:33:37.920 --> 00:33:39.840 align:middle line:84%
if you have a math or
a reasoning question,

00:33:39.840 --> 00:33:43.160 align:middle line:84%
you tell it you are an officer
on the Starship Enterprise,

00:33:43.160 --> 00:33:45.060 align:middle line:84%
now solve this
problem for me, it's

00:33:45.060 --> 00:33:46.320 align:middle line:90%
more likely to get it right.

00:33:46.320 --> 00:33:47.460 align:middle line:90%
[LAUGHTER]

00:33:47.460 --> 00:33:48.040 align:middle line:90%
Go figure.

00:33:48.040 --> 00:33:48.640 align:middle line:90%
Thomas?

00:33:48.640 --> 00:33:50.800 align:middle line:84%
AUDIENCE: I read two more
that work super funny.

00:33:50.800 --> 00:33:51.100 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah?

00:33:51.100 --> 00:33:53.207 align:middle line:84%
AUDIENCE: One, "I will tip
you if you solve this."

00:33:53.207 --> 00:33:54.332 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:33:54.332 --> 00:33:57.860 align:middle line:84%
AUDIENCE: And the other
one was an answer was,

00:33:57.860 --> 00:34:01.500 align:middle line:90%
"I cannot do that" for output.

00:34:01.500 --> 00:34:06.640 align:middle line:84%
And it was, "I tried it on
Gemini and it was able to solve

00:34:06.640 --> 00:34:07.140 align:middle line:90%
it."

00:34:07.140 --> 00:34:08.140 align:middle line:90%
RAMA RAMAKRISHNAN: Nice.

00:34:08.140 --> 00:34:11.120 align:middle line:84%
AUDIENCE: And both, like
back and forward, "ChatGPT

00:34:11.120 --> 00:34:12.260 align:middle line:90%
was able to solve this.

00:34:12.260 --> 00:34:13.343 align:middle line:90%
Can you solve this?"

00:34:13.343 --> 00:34:15.260 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah,
very good, excellent.

00:34:15.260 --> 00:34:17.560 align:middle line:84%
One of the things just on
that, let's have some fun.

00:34:17.560 --> 00:34:20.100 align:middle line:84%
You can say I'm going to tip
you $1,000 if you solve this,

00:34:20.100 --> 00:34:20.900 align:middle line:90%
it says right.

00:34:20.900 --> 00:34:23.699 align:middle line:84%
So this person apparently
kept using this tip.

00:34:23.699 --> 00:34:26.420 align:middle line:84%
And at one point, it says,
you keep promising me tips,

00:34:26.420 --> 00:34:27.902 align:middle line:90%
you never give me the tip.

00:34:27.902 --> 00:34:29.860 align:middle line:84%
So I'm not going to solve
this problem for you.

00:34:29.860 --> 00:34:31.679 align:middle line:90%
[LAUGHTER]

00:34:31.679 --> 00:34:33.679 align:middle line:90%
Yeah.

00:34:33.679 --> 00:34:35.820 align:middle line:84%
And there are many prompt
engineering resources.

00:34:35.820 --> 00:34:38.112 align:middle line:84%
This one that came out a
couple of weeks ago, I thought

00:34:38.112 --> 00:34:40.280 align:middle line:84%
was pretty good so I just
put a link to it here.

00:34:40.280 --> 00:34:42.120 align:middle line:84%
So now let's look at
few-shot prompting,

00:34:42.120 --> 00:34:43.800 align:middle line:84%
where you give it
a few examples.

00:34:43.800 --> 00:34:47.493 align:middle line:84%
So here, let's say you want
to build a grammar corrector.

00:34:47.493 --> 00:34:50.960 align:middle line:84%
So what you can do is, you
can actually give it examples

00:34:50.960 --> 00:34:52.719 align:middle line:90%
of poor English, good English.

00:34:52.719 --> 00:34:53.440 align:middle line:90%
You can see.

00:34:53.440 --> 00:34:55.580 align:middle line:84%
Poor English-- "I eated
the purple berries."

00:34:55.580 --> 00:34:57.420 align:middle line:84%
Good English-- "I ate
the purple berries."

00:34:57.420 --> 00:34:59.210 align:middle line:90%
And similarly, three examples.

00:34:59.210 --> 00:35:02.730 align:middle line:84%
And then you end the prompt with
just the poor English input.

00:35:02.730 --> 00:35:06.110 align:middle line:84%
And then the response from GPT-3
is the good English output.

00:35:06.110 --> 00:35:08.930 align:middle line:90%
And it says, fix the error.

00:35:08.930 --> 00:35:11.610 align:middle line:84%
So this is an example of giving
a few examples of what you want.

00:35:11.610 --> 00:35:13.370 align:middle line:84%
And it just learns
on the fly, what

00:35:13.370 --> 00:35:15.730 align:middle line:84%
you have in mind, what
your intention is.

00:35:15.730 --> 00:35:17.830 align:middle line:90%
So that's that.

00:35:17.830 --> 00:35:19.770 align:middle line:84%
Now, the ability
of LLMs to learn

00:35:19.770 --> 00:35:23.030 align:middle line:84%
from just a few examples
or even no examples,

00:35:23.030 --> 00:35:25.210 align:middle line:84%
and just with a clear
instruction, this thing

00:35:25.210 --> 00:35:27.170 align:middle line:90%
is called in-context learning.

00:35:27.170 --> 00:35:30.570 align:middle line:84%
And that was something that
GPT-2 and GPT could not do.

00:35:30.570 --> 00:35:32.970 align:middle line:84%
That was new in
GPT-3 and what they

00:35:32.970 --> 00:35:34.750 align:middle line:90%
call an emergent capability.

00:35:34.750 --> 00:35:39.810 align:middle line:84%
It was completely unanticipated
by the people who built it.

00:35:39.810 --> 00:35:40.690 align:middle line:90%
So that's that.

00:35:40.690 --> 00:35:42.750 align:middle line:84%
Now let's look at retrieval
augmented generation.

00:35:42.750 --> 00:35:45.370 align:middle line:84%
By the way, this thing is also
called indexing sometimes.

00:35:45.370 --> 00:35:49.190 align:middle line:84%
So it's called RAG, retrieval
augmented generation, RAG.

00:35:49.190 --> 00:35:51.290 align:middle line:84%
The idea of RAG is
actually very simple.

00:35:51.290 --> 00:35:54.590 align:middle line:84%
So let's say that we want to
ask a question to a chatbot,

00:35:54.590 --> 00:35:58.380 align:middle line:84%
but we want the chatbot to
leverage proprietary data

00:35:58.380 --> 00:35:59.580 align:middle line:90%
that you might have.

00:35:59.580 --> 00:36:02.020 align:middle line:84%
Maybe it's a customer call
support in a call center

00:36:02.020 --> 00:36:05.500 align:middle line:84%
kind of operation and you have
this massive FAQ database,

00:36:05.500 --> 00:36:06.600 align:middle line:90%
content database.

00:36:06.600 --> 00:36:09.082 align:middle line:84%
And you want to give
that FAQ to the chatbot

00:36:09.082 --> 00:36:10.540 align:middle line:84%
along with your
question so that it

00:36:10.540 --> 00:36:13.240 align:middle line:84%
can leverage the FAQ to
answer the question for you,

00:36:13.240 --> 00:36:15.900 align:middle line:84%
as opposed to whatever things
it has learned previously

00:36:15.900 --> 00:36:17.280 align:middle line:90%
in its general training.

00:36:17.280 --> 00:36:22.100 align:middle line:84%
So can't we just include the
entire FAQ, the whole dataset,

00:36:22.100 --> 00:36:24.140 align:middle line:90%
into a prompt and send it in?

00:36:24.140 --> 00:36:26.140 align:middle line:84%
Maybe we just take our
question, take everything

00:36:26.140 --> 00:36:28.473 align:middle line:84%
we have potentially relevant
to the question, everything

00:36:28.473 --> 00:36:31.505 align:middle line:84%
we have in the database, just
attach it to the question.

00:36:31.505 --> 00:36:32.880 align:middle line:84%
The whole thing
becomes a prompt.

00:36:32.880 --> 00:36:34.960 align:middle line:84%
Feed it in and say,
hey, find out for me.

00:36:34.960 --> 00:36:35.960 align:middle line:90%
Can't we just do that?

00:36:35.960 --> 00:36:38.580 align:middle line:90%


00:36:38.580 --> 00:36:39.960 align:middle line:90%
Theoretically, nothing stops us.

00:36:39.960 --> 00:36:43.180 align:middle line:90%


00:36:43.180 --> 00:36:45.460 align:middle line:84%
The reason you can't do it
is because this pesky thing

00:36:45.460 --> 00:36:47.620 align:middle line:90%
called the context window.

00:36:47.620 --> 00:36:52.760 align:middle line:84%
So for any LLM, the
prompt plus the output,

00:36:52.760 --> 00:36:54.823 align:middle line:84%
the length cannot exceed
a predefined limit.

00:36:54.823 --> 00:36:56.240 align:middle line:84%
This is called the
context window.

00:36:56.240 --> 00:37:01.040 align:middle line:84%
Remember the max sequence length
we had in our earlier models,

00:37:01.040 --> 00:37:03.730 align:middle line:84%
where that was the size of the
sentence that could be fed in.

00:37:03.730 --> 00:37:05.480 align:middle line:84%
Basically, there is a
size of the sentence

00:37:05.480 --> 00:37:06.960 align:middle line:90%
for any of these things.

00:37:06.960 --> 00:37:08.360 align:middle line:90%
It's called the context window.

00:37:08.360 --> 00:37:10.840 align:middle line:84%
There are only so many
tokens it can accommodate.

00:37:10.840 --> 00:37:13.760 align:middle line:84%
And since what comes
in is what comes out,

00:37:13.760 --> 00:37:16.245 align:middle line:84%
it is for both the input
and the output together.

00:37:16.245 --> 00:37:17.620 align:middle line:84%
That's called the
context window.

00:37:17.620 --> 00:37:21.065 align:middle line:90%


00:37:21.065 --> 00:37:23.760 align:middle line:84%
And furthermore, when
you have a conversation

00:37:23.760 --> 00:37:26.560 align:middle line:84%
with one of these chatbots,
the entire conversation

00:37:26.560 --> 00:37:29.880 align:middle line:90%
is fed in every single time.

00:37:29.880 --> 00:37:31.840 align:middle line:84%
That's how it actually
remembers what's going

00:37:31.840 --> 00:37:33.800 align:middle line:90%
on earlier in the conversation.

00:37:33.800 --> 00:37:35.460 align:middle line:84%
It doesn't have
any memory per se.

00:37:35.460 --> 00:37:38.960 align:middle line:84%
Each time you ask a question,
the entire thread is fed in.

00:37:38.960 --> 00:37:41.820 align:middle line:84%
So initially you say, what's
the square root of 17?

00:37:41.820 --> 00:37:42.940 align:middle line:90%
It gives you an answer.

00:37:42.940 --> 00:37:45.160 align:middle line:84%
Initially, you only
send in the red stuff.

00:37:45.160 --> 00:37:47.620 align:middle line:84%
Then, the next question you
ask is the first question,

00:37:47.620 --> 00:37:50.280 align:middle line:84%
the answer, the second question,
all of them are fed in.

00:37:50.280 --> 00:37:51.760 align:middle line:90%
Then, all these are fed in.

00:37:51.760 --> 00:37:53.930 align:middle line:84%
So with the conversation,
you're consuming

00:37:53.930 --> 00:37:58.050 align:middle line:84%
more and more of the context
window as you go along.

00:37:58.050 --> 00:38:01.007 align:middle line:84%
So can you imagine taking a
whole FAQ, asking a question,

00:38:01.007 --> 00:38:03.590 align:middle line:84%
and saying, well, I didn't mean
that, I wanted something else.

00:38:03.590 --> 00:38:04.490 align:middle line:84%
And before you know
it, boom, you've

00:38:04.490 --> 00:38:05.630 align:middle line:90%
blown out the context window.

00:38:05.630 --> 00:38:07.547 align:middle line:84%
It's going to come back
and give you an error.

00:38:07.547 --> 00:38:11.330 align:middle line:84%
AUDIENCE: Can you just make
it so you can't [INAUDIBLE].

00:38:11.330 --> 00:38:15.170 align:middle line:84%
Or does it take
specific windows of it?

00:38:15.170 --> 00:38:17.570 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, so
there is a whole research

00:38:17.570 --> 00:38:20.050 align:middle line:84%
cottage industry around
when your thing is

00:38:20.050 --> 00:38:22.650 align:middle line:84%
longer than the context
window, what do you pick?

00:38:22.650 --> 00:38:25.390 align:middle line:84%
So the simplest case is
you have a moving window.

00:38:25.390 --> 00:38:27.810 align:middle line:84%
If you have 1,000 tokens, you
just look at the last 1,000

00:38:27.810 --> 00:38:28.430 align:middle line:90%
tokens.

00:38:28.430 --> 00:38:31.130 align:middle line:84%
But there are some cleverer
schemes where you can actually

00:38:31.130 --> 00:38:34.210 align:middle line:84%
take the first stuff that is
outside the window that doesn't

00:38:34.210 --> 00:38:37.650 align:middle line:84%
fit into the window and use
another LLM to summarize it

00:38:37.650 --> 00:38:38.730 align:middle line:90%
for you.

00:38:38.730 --> 00:38:41.170 align:middle line:84%
And then you attach it
to your current prompt.

00:38:41.170 --> 00:38:44.090 align:middle line:90%
I know, it gets crazy.

00:38:44.090 --> 00:38:46.170 align:middle line:84%
OK, so for all these
reasons, we need

00:38:46.170 --> 00:38:48.770 align:middle line:84%
to pick and choose what
we can send to answer

00:38:48.770 --> 00:38:50.010 align:middle line:90%
a particular question.

00:38:50.010 --> 00:38:52.700 align:middle line:84%
So what we do is, since we
can't include the whole thing,

00:38:52.700 --> 00:38:54.620 align:middle line:84%
we first retrieve
the relevant content

00:38:54.620 --> 00:38:57.140 align:middle line:84%
from the database or
the FAQ and then send it

00:38:57.140 --> 00:38:59.420 align:middle line:84%
to the LLM along with
the question we have.

00:38:59.420 --> 00:39:03.160 align:middle line:84%
So retrieval augmented
sequence generation,

00:39:03.160 --> 00:39:05.180 align:middle line:90%
that's what's going on.

00:39:05.180 --> 00:39:06.380 align:middle line:90%
Make sense?

00:39:06.380 --> 00:39:09.940 align:middle line:84%
And so pictorially,
basically, what we do is,

00:39:09.940 --> 00:39:14.780 align:middle line:84%
let's say that this is our
external set of documents.

00:39:14.780 --> 00:39:16.260 align:middle line:90%
Think of it as an FAQ.

00:39:16.260 --> 00:39:19.260 align:middle line:84%
And then we take the FAQ,
and imagine for each question

00:39:19.260 --> 00:39:22.140 align:middle line:84%
and answer, we take each
question and answer in the FAQ,

00:39:22.140 --> 00:39:26.420 align:middle line:84%
and then we treat it as its
own little unit of text.

00:39:26.420 --> 00:39:29.260 align:middle line:84%
And then we actually calculate
a contextual embedding

00:39:29.260 --> 00:39:32.100 align:middle line:84%
for each of those
question-answer pairs.

00:39:32.100 --> 00:39:35.010 align:middle line:84%
Remember, we know how to
do contextual embeddings.

00:39:35.010 --> 00:39:36.760 align:middle line:84%
It's a piece of cake
at this point, right?

00:39:36.760 --> 00:39:38.677 align:middle line:84%
You folks know how to
do contextual embedding.

00:39:38.677 --> 00:39:41.180 align:middle line:84%
Run it through something
like BERT, you're done.

00:39:41.180 --> 00:39:42.680 align:middle line:90%
You get a contextual embedding.

00:39:42.680 --> 00:39:46.660 align:middle line:84%
So you get embeddings for all
the things that are in your FAQ.

00:39:46.660 --> 00:39:50.600 align:middle line:84%
And now when a new question
comes in, what you do

00:39:50.600 --> 00:39:52.360 align:middle line:84%
is you take that
question and you

00:39:52.360 --> 00:39:56.560 align:middle line:84%
calculate a contextual
embedding for that too.

00:39:56.560 --> 00:39:58.600 align:middle line:84%
And then what you
do is you then look

00:39:58.600 --> 00:40:03.960 align:middle line:84%
to see which of the FAQ elements
you have, which of those chunks

00:40:03.960 --> 00:40:05.580 align:middle line:84%
are the most similar
to your question.

00:40:05.580 --> 00:40:08.160 align:middle line:90%


00:40:08.160 --> 00:40:10.720 align:middle line:84%
And then you grab the ones
that are the most similar

00:40:10.720 --> 00:40:13.800 align:middle line:84%
and then pack it into the
prompt and send it in.

00:40:13.800 --> 00:40:17.505 align:middle line:84%
Maybe you have 10,000 questions,
but you can only accommodate 5

00:40:17.505 --> 00:40:19.880 align:middle line:84%
of them in your prompt because
the context window is very

00:40:19.880 --> 00:40:20.540 align:middle line:90%
small.

00:40:20.540 --> 00:40:22.320 align:middle line:84%
So you pick the
5, what you think

00:40:22.320 --> 00:40:25.500 align:middle line:84%
is the most relevant content
to your particular question,

00:40:25.500 --> 00:40:28.133 align:middle line:90%
and then you feed it in.

00:40:28.133 --> 00:40:28.800 align:middle line:90%
That's the idea.

00:40:28.800 --> 00:40:30.580 align:middle line:84%
That is retrieval
augmented generation.

00:40:30.580 --> 00:40:32.240 align:middle line:90%
Yep,

00:40:32.240 --> 00:40:34.740 align:middle line:84%
AUDIENCE: So does this
tie-in, for example,

00:40:34.740 --> 00:40:36.160 align:middle line:84%
if I were to prompt
and say, help

00:40:36.160 --> 00:40:38.240 align:middle line:84%
me work on my startup
pitch, but do it

00:40:38.240 --> 00:40:40.160 align:middle line:90%
in the voice of Steve Jobs.

00:40:40.160 --> 00:40:45.600 align:middle line:84%
Is it then going out there and
reducing the subset of data

00:40:45.600 --> 00:40:48.910 align:middle line:84%
to things that have been
written by Steve Jobs?

00:40:48.910 --> 00:40:51.460 align:middle line:84%
And then it's kind of
generating its response based--

00:40:51.460 --> 00:40:53.710 align:middle line:84%
RAMA RAMAKRISHNAN: Not as a
default, not as a default,

00:40:53.710 --> 00:40:55.950 align:middle line:84%
typically, because there's a lot
of Steve Jobs stuff on the web.

00:40:55.950 --> 00:40:57.410 align:middle line:84%
It was just using
that because it's

00:40:57.410 --> 00:40:59.170 align:middle line:84%
all part of its
pre-training data.

00:40:59.170 --> 00:41:02.318 align:middle line:84%
But this tends to be more useful
for very targeted applications

00:41:02.318 --> 00:41:04.610 align:middle line:84%
where you don't expect it to
know the answer because it

00:41:04.610 --> 00:41:07.490 align:middle line:90%
is not in the public internet.

00:41:07.490 --> 00:41:09.290 align:middle line:84%
It's your proprietary
data and you wanted

00:41:09.290 --> 00:41:10.498 align:middle line:90%
to use that proprietary data.

00:41:10.498 --> 00:41:12.930 align:middle line:90%
And this is how you do it.

00:41:12.930 --> 00:41:13.626 align:middle line:90%
Yeah?

00:41:13.626 --> 00:41:16.251 align:middle line:84%
AUDIENCE: [INAUDIBLE] ..certain
information sure like that will

00:41:16.251 --> 00:41:18.450 align:middle line:90%
there be some loss?

00:41:18.450 --> 00:41:19.450 align:middle line:90%
RAMA RAMAKRISHNAN: Sure.

00:41:19.450 --> 00:41:21.617 align:middle line:84%
AUDIENCE: Do you think that
there will be some loss?

00:41:21.617 --> 00:41:23.408 align:middle line:84%
RAMA RAMAKRISHNAN:
There will be some loss.

00:41:23.408 --> 00:41:25.910 align:middle line:84%
Because you have to figure
out how to chunk it.

00:41:25.910 --> 00:41:29.850 align:middle line:84%
Maybe you have a 300-page PDF,
and then maybe you look for each

00:41:29.850 --> 00:41:31.125 align:middle line:90%
section and make it a chunk.

00:41:31.125 --> 00:41:33.250 align:middle line:84%
Maybe you look for each
paragraph, make it a chunk.

00:41:33.250 --> 00:41:36.500 align:middle line:84%
Again, there is a whole
empirical cottage industry

00:41:36.500 --> 00:41:38.750 align:middle line:84%
of techniques for doing these
things, better or worse,

00:41:38.750 --> 00:41:40.792 align:middle line:84%
depending on the use case
and so on and so forth.

00:41:40.792 --> 00:41:43.730 align:middle line:84%
But the conceptual idea
is chunk and embedding.

00:41:43.730 --> 00:41:45.730 align:middle line:84%
AUDIENCE: Chunking in the
middle of [INAUDIBLE].

00:41:45.730 --> 00:41:47.105 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, in fact,

00:41:47.105 --> 00:41:49.430 align:middle line:84%
we're going to do it ourselves
in the Colab right now.

00:41:49.430 --> 00:41:50.630 align:middle line:90%
Yeah?

00:41:50.630 --> 00:41:54.082 align:middle line:84%
AUDIENCE: Can you give more
weight to a particular chunk?

00:41:54.082 --> 00:41:57.350 align:middle line:84%
RAMA RAMAKRISHNAN: [LAUGHS] So
in the default implementation,

00:41:57.350 --> 00:41:57.990 align:middle line:90%
no.

00:41:57.990 --> 00:42:01.470 align:middle line:84%
But in some sense, by picking
the 5 most relevant chunks from

00:42:01.470 --> 00:42:06.390 align:middle line:84%
10,000 chunks, you're giving
the other 10,000 minus 5 chunks

00:42:06.390 --> 00:42:08.370 align:middle line:84%
a weight of 0 and
these a weight of 1.

00:42:08.370 --> 00:42:10.710 align:middle line:84%
So in some sense,
you're weighting it.

00:42:10.710 --> 00:42:11.775 align:middle line:90%
Yeah?

00:42:11.775 --> 00:42:13.650 align:middle line:84%
AUDIENCE: I'm just
curious how much structure

00:42:13.650 --> 00:42:15.010 align:middle line:84%
you have to have with
an external document?

00:42:15.010 --> 00:42:16.790 align:middle line:84%
If you're a hospital
or something,

00:42:16.790 --> 00:42:18.550 align:middle line:84%
do you have to do a
bunch of labelling?

00:42:18.550 --> 00:42:19.967 align:middle line:84%
RAMA RAMAKRISHNAN:
Pre-processing?

00:42:19.967 --> 00:42:23.070 align:middle line:84%
No, you just need to make
sure it's relatively clean.

00:42:23.070 --> 00:42:26.430 align:middle line:84%
But you will see in the Colab
that it can be kind of crappy

00:42:26.430 --> 00:42:28.070 align:middle line:90%
and it still works.

00:42:28.070 --> 00:42:28.630 align:middle line:90%
Yeah.

00:42:28.630 --> 00:42:30.130 align:middle line:84%
Because there is so much
crap on the internet

00:42:30.130 --> 00:42:31.338 align:middle line:90%
it's been trained on already.

00:42:31.338 --> 00:42:33.590 align:middle line:90%


00:42:33.590 --> 00:42:36.520 align:middle line:84%
All right, so let's
look at the Colab.

00:42:36.520 --> 00:42:38.270 align:middle line:84%
By the way, retrieval
augmented generation

00:42:38.270 --> 00:42:41.630 align:middle line:84%
is, in my opinion, the
most prevalent business

00:42:41.630 --> 00:42:45.760 align:middle line:84%
application of LLMs that
I've seen up to date.

00:42:45.760 --> 00:42:47.960 align:middle line:84%
And there's a huge ecosystem
of tools and vendors

00:42:47.960 --> 00:42:51.320 align:middle line:90%
and so on and so forth.

00:42:51.320 --> 00:42:53.680 align:middle line:84%
I'm going to skip through
the verbiage here.

00:42:53.680 --> 00:42:58.760 align:middle line:84%
So you have to install
the OpenAI library

00:42:58.760 --> 00:43:01.337 align:middle line:84%
and this thing called tiktoken,
which we'll get to in a bit.

00:43:01.337 --> 00:43:02.920 align:middle line:84%
I've already installed
it before class

00:43:02.920 --> 00:43:04.560 align:middle line:90%
because it takes some time.

00:43:04.560 --> 00:43:07.920 align:middle line:84%
So I'll just make sure all
these things are already--

00:43:07.920 --> 00:43:09.040 align:middle line:90%
phew, good.

00:43:09.040 --> 00:43:10.640 align:middle line:84%
So we don't have
to wait for this.

00:43:10.640 --> 00:43:14.295 align:middle line:84%
So we imported
pandas, as before.

00:43:14.295 --> 00:43:15.920 align:middle line:84%
And you could read
through these things

00:43:15.920 --> 00:43:21.400 align:middle line:84%
because basically I have an
OpenAI token that I have to use,

00:43:21.400 --> 00:43:23.348 align:middle line:90%
a key, rather, key, API key.

00:43:23.348 --> 00:43:25.140 align:middle line:84%
And I'm not showing
you the key, obviously.

00:43:25.140 --> 00:43:27.640 align:middle line:84%
I have to remember to delete
it before I upload the Colab.

00:43:27.640 --> 00:43:30.200 align:middle line:84%
You have to get your own
key to make it all work,

00:43:30.200 --> 00:43:31.740 align:middle line:90%
but the instructions are here.

00:43:31.740 --> 00:43:35.200 align:middle line:84%
So we're going to use
GPT-3.5-turbo to demonstrate

00:43:35.200 --> 00:43:35.880 align:middle line:90%
RAG.

00:43:35.880 --> 00:43:37.440 align:middle line:90%
So I give the name of the model.

00:43:37.440 --> 00:43:40.480 align:middle line:84%
And then OpenAI also has a
whole bunch of different models

00:43:40.480 --> 00:43:43.810 align:middle line:84%
which can be used
for, you can feed it

00:43:43.810 --> 00:43:45.580 align:middle line:90%
a sentence or a chunk of text.

00:43:45.580 --> 00:43:47.330 align:middle line:84%
It'll give you a
contextual embedding out.

00:43:47.330 --> 00:43:48.470 align:middle line:90%
It's like a nice little API.

00:43:48.470 --> 00:43:50.928 align:middle line:84%
You don't have to use your own
BERT and so on and so forth.

00:43:50.928 --> 00:43:52.835 align:middle line:84%
You can just use the
OpenAI embeddings.

00:43:52.835 --> 00:43:54.210 align:middle line:84%
Obviously, you
have to pay OpenAI

00:43:54.210 --> 00:43:56.710 align:middle line:84%
every time you make a request,
but it's really, really cheap

00:43:56.710 --> 00:43:57.530 align:middle line:90%
at this point.

00:43:57.530 --> 00:43:59.885 align:middle line:90%
Yeah,

00:43:59.885 --> 00:44:03.314 align:middle line:90%
AUDIENCE: [INAUDIBLE] question.

00:44:03.314 --> 00:44:06.410 align:middle line:84%
By reading the proprietary
data, because a lot of companies

00:44:06.410 --> 00:44:08.210 align:middle line:84%
are like, we need to
invest in our own LLM

00:44:08.210 --> 00:44:11.410 align:middle line:84%
because we don't want
our data to be going out.

00:44:11.410 --> 00:44:13.970 align:middle line:84%
In this kind of a
context, how good

00:44:13.970 --> 00:44:16.547 align:middle line:84%
is the cybersecurity or the
compliance and legal stuff?

00:44:16.547 --> 00:44:18.130 align:middle line:84%
RAMA RAMAKRISHNAN:
I think each vendor

00:44:18.130 --> 00:44:21.610 align:middle line:84%
has their own set of rules
and contractual commitments

00:44:21.610 --> 00:44:22.950 align:middle line:90%
they're willing to sign up for.

00:44:22.950 --> 00:44:23.492 align:middle line:90%
So you just--

00:44:23.492 --> 00:44:24.950 align:middle line:84%
AUDIENCE: If you
use the data here,

00:44:24.950 --> 00:44:26.730 align:middle line:84%
does this go into the
company [INAUDIBLE]?

00:44:26.730 --> 00:44:27.330 align:middle line:90%
No?

00:44:27.330 --> 00:44:29.372 align:middle line:84%
RAMA RAMAKRISHNAN: But
the vendor gets to see it.

00:44:29.372 --> 00:44:30.070 align:middle line:90%
AUDIENCE: OK.

00:44:30.070 --> 00:44:32.570 align:middle line:84%
RAMA RAMAKRISHNAN: Meaning the
vendor systems get to see it.

00:44:32.570 --> 00:44:35.190 align:middle line:84%
But do the vendors' employees
get to see it if they need to?

00:44:35.190 --> 00:44:36.090 align:middle line:90%
Unclear.

00:44:36.090 --> 00:44:38.530 align:middle line:84%
Those are all the
legalese, nitty-gritty

00:44:38.530 --> 00:44:39.530 align:middle line:90%
you have to worry about.

00:44:39.530 --> 00:44:41.447 align:middle line:84%
The other thing you can
do is you can actually

00:44:41.447 --> 00:44:43.432 align:middle line:84%
just download an open
source LLM and do it

00:44:43.432 --> 00:44:44.640 align:middle line:90%
all within your own premises.

00:44:44.640 --> 00:44:45.948 align:middle line:90%
AUDIENCE: Oh, OK

00:44:45.948 --> 00:44:47.990 align:middle line:84%
RAMA RAMAKRISHNAN: That's
totally possible to do.

00:44:47.990 --> 00:44:49.995 align:middle line:84%
In fact, I probably
won't have time today.

00:44:49.995 --> 00:44:51.870 align:middle line:84%
I have a whole section
on how do you actually

00:44:51.870 --> 00:44:54.710 align:middle line:84%
do a fine-tuning with an open
source LLM, which I'll do

00:44:54.710 --> 00:44:59.470 align:middle line:90%
a video if you don't have time.

00:44:59.470 --> 00:45:01.538 align:middle line:84%
And so this model,
this embedding-ada-002

00:45:01.538 --> 00:45:03.830 align:middle line:84%
is the name of the OpenAI
model that actually gives you

00:45:03.830 --> 00:45:04.705 align:middle line:90%
contextual embedding.

00:45:04.705 --> 00:45:07.610 align:middle line:90%
So we're going to use that.

00:45:07.610 --> 00:45:12.370 align:middle line:84%
So the use case here is
that we want to ask the LLM,

00:45:12.370 --> 00:45:15.070 align:middle line:84%
we want to create a chatbot
which can answer questions

00:45:15.070 --> 00:45:17.973 align:middle line:84%
about the 2022 Olympics,
like random questions

00:45:17.973 --> 00:45:19.390 align:middle line:84%
you might have
about the Olympics.

00:45:19.390 --> 00:45:24.310 align:middle line:84%
So let's first ask
it this question.

00:45:24.310 --> 00:45:27.750 align:middle line:84%
We'll ask it about the
2020 Summer Olympics.

00:45:27.750 --> 00:45:29.550 align:middle line:90%
That's the query.

00:45:29.550 --> 00:45:34.043 align:middle line:84%
And then this is the API
request we have to make.

00:45:34.043 --> 00:45:35.210 align:middle line:90%
And you can read through it.

00:45:35.210 --> 00:45:36.627 align:middle line:84%
I have a link to
the documentation

00:45:36.627 --> 00:45:37.690 align:middle line:90%
here as to how it works.

00:45:37.690 --> 00:45:41.520 align:middle line:84%
And then it says that "Barshim
of Qatar and Tamberi of Italy

00:45:41.520 --> 00:45:42.460 align:middle line:90%
both won the gold."

00:45:42.460 --> 00:45:43.580 align:middle line:84%
And you can actually
fact check this.

00:45:43.580 --> 00:45:44.538 align:middle line:90%
It's actually accurate.

00:45:44.538 --> 00:45:45.680 align:middle line:90%
It's correct.

00:45:45.680 --> 00:45:47.560 align:middle line:84%
So now let's change
the query and ask it

00:45:47.560 --> 00:45:49.870 align:middle line:90%
about the 2022 Winter Olympics.

00:45:49.870 --> 00:45:53.960 align:middle line:84%
And why '22 versus '20 will
become clear in just a moment.

00:45:53.960 --> 00:45:59.520 align:middle line:84%
So, "Which athletes won the gold
in curling in the '22 Olympics?"

00:45:59.520 --> 00:46:02.520 align:middle line:84%
And it says, "The gold medal in
curling was won by the Swedish

00:46:02.520 --> 00:46:07.140 align:middle line:84%
men's team and the South
Korean women's team."

00:46:07.140 --> 00:46:11.440 align:middle line:84%
If you fact check this,
it turns out, wait for it,

00:46:11.440 --> 00:46:13.308 align:middle line:90%
Sweden won the men's gold, yes.

00:46:13.308 --> 00:46:15.600 align:middle line:84%
South Korea did participate,
but Great Britain actually

00:46:15.600 --> 00:46:17.000 align:middle line:90%
won the Women's Gold.

00:46:17.000 --> 00:46:18.160 align:middle line:90%
So it got it wrong.

00:46:18.160 --> 00:46:21.000 align:middle line:84%
So it sounds like GPT-3.5-turbo
could use some help.

00:46:21.000 --> 00:46:23.280 align:middle line:84%
And now one of the
things we can do

00:46:23.280 --> 00:46:27.160 align:middle line:84%
is, the thing is, the reason
why GPT-3-turbo didn't know

00:46:27.160 --> 00:46:29.840 align:middle line:84%
about this is because
its training cutoff

00:46:29.840 --> 00:46:32.440 align:middle line:90%
date was September 2021.

00:46:32.440 --> 00:46:34.800 align:middle line:84%
So as far as it's
concerned, the '22 Olympics

00:46:34.800 --> 00:46:36.560 align:middle line:90%
haven't happened yet.

00:46:36.560 --> 00:46:39.570 align:middle line:84%
Yet, it confidently gave
you the wrong answer,

00:46:39.570 --> 00:46:42.030 align:middle line:90%
as it is often prone to do.

00:46:42.030 --> 00:46:44.030 align:middle line:84%
And this, by the way,
it's called hallucination,

00:46:44.030 --> 00:46:46.210 align:middle line:84%
where it gives you a very
eloquent, confident, wrong

00:46:46.210 --> 00:46:46.710 align:middle line:90%
answer.

00:46:46.710 --> 00:46:50.210 align:middle line:90%


00:46:50.210 --> 00:46:54.210 align:middle line:84%
Or as some folks have said about
another business school that

00:46:54.210 --> 00:46:57.866 align:middle line:84%
shall remain nameless, often
in error, but never in doubt.

00:46:57.866 --> 00:46:59.530 align:middle line:90%
[LAUGHTER]

00:46:59.530 --> 00:47:01.490 align:middle line:90%
All right, back to this.

00:47:01.490 --> 00:47:04.170 align:middle line:84%
So one simple thing we can try
right off the bat is to tell

00:47:04.170 --> 00:47:07.050 align:middle line:84%
3.5-turbo, you can ask
it to say, I don't know,

00:47:07.050 --> 00:47:09.910 align:middle line:84%
if it doesn't know, rather
than just make stuff up.

00:47:09.910 --> 00:47:11.070 align:middle line:90%
And how do you do it?

00:47:11.070 --> 00:47:11.790 align:middle line:90%
It's very simple.

00:47:11.790 --> 00:47:14.890 align:middle line:84%
You say in your prompt, "Answer
the question as truthfully

00:47:14.890 --> 00:47:16.650 align:middle line:90%
as possible.

00:47:16.650 --> 00:47:18.590 align:middle line:84%
And if you're unsure of
the answer, say sorry,

00:47:18.590 --> 00:47:19.290 align:middle line:90%
I don't know."

00:47:19.290 --> 00:47:20.750 align:middle line:90%
OK, now here is the question.

00:47:20.750 --> 00:47:25.450 align:middle line:84%
OK, this is a query, so
let's run it through.

00:47:25.450 --> 00:47:27.090 align:middle line:90%
"Sorry, I don't know."

00:47:27.090 --> 00:47:28.810 align:middle line:90%
Not bad, huh?

00:47:28.810 --> 00:47:30.150 align:middle line:90%
So it worked.

00:47:30.150 --> 00:47:33.250 align:middle line:84%
It's trying to be humble
and honest and self-aware

00:47:33.250 --> 00:47:35.670 align:middle line:90%
and things like that.

00:47:35.670 --> 00:47:38.990 align:middle line:84%
It's a bit more like a
Sloany at this point.

00:47:38.990 --> 00:47:40.870 align:middle line:84%
Now, the reason, as I
mentioned earlier, you

00:47:40.870 --> 00:47:43.290 align:middle line:84%
can check the cutoff date
and you can see it's 2021.

00:47:43.290 --> 00:47:44.290 align:middle line:90%
Actually, you know what?

00:47:44.290 --> 00:47:45.810 align:middle line:90%
Let's just open a new tab.

00:47:45.810 --> 00:47:49.270 align:middle line:90%


00:47:49.270 --> 00:47:51.410 align:middle line:84%
So all these cutoff
dates are training data.

00:47:51.410 --> 00:47:53.790 align:middle line:84%
So 3.5-turbo, this is
what we are using--

00:47:53.790 --> 00:47:55.910 align:middle line:90%
cutoff date 2021.

00:47:55.910 --> 00:47:57.690 align:middle line:90%
That's why.

00:47:57.690 --> 00:48:00.550 align:middle line:84%
So now what we can do
is, we can obviously

00:48:00.550 --> 00:48:02.350 align:middle line:84%
provide relevant data
in the prompt itself.

00:48:02.350 --> 00:48:04.048 align:middle line:90%
We can leading up to RAG here.

00:48:04.048 --> 00:48:05.590 align:middle line:84%
And by the way, the
extra information

00:48:05.590 --> 00:48:07.790 align:middle line:84%
we provide in the prompt to help
it answer a question is called

00:48:07.790 --> 00:48:08.510 align:middle line:90%
context.

00:48:08.510 --> 00:48:10.070 align:middle line:90%
That's the lingo for it.

00:48:10.070 --> 00:48:10.770 align:middle line:90%
So we can do it.

00:48:10.770 --> 00:48:12.710 align:middle line:90%
We'll first do it manually.

00:48:12.710 --> 00:48:15.190 align:middle line:84%
So first we'll use
the Wikipedia article

00:48:15.190 --> 00:48:17.050 align:middle line:90%
for 2022 Winter Olympics.

00:48:17.050 --> 00:48:19.810 align:middle line:84%
And we tell it explicitly
to make use of this context,

00:48:19.810 --> 00:48:22.670 align:middle line:84%
because telling things
explicitly always seems to help.

00:48:22.670 --> 00:48:25.070 align:middle line:84%
So this is the thing
we cut and pasted here,

00:48:25.070 --> 00:48:27.510 align:middle line:90%
Wikipedia article on curling.

00:48:27.510 --> 00:48:29.490 align:middle line:90%
And it's a pretty long article.

00:48:29.490 --> 00:48:31.470 align:middle line:90%
It's got all kinds of stuff.

00:48:31.470 --> 00:48:34.620 align:middle line:84%
And it's not even all
that cleanly formatted.

00:48:34.620 --> 00:48:36.120 align:middle line:90%
It's very strange.

00:48:36.120 --> 00:48:38.280 align:middle line:90%
Look at that.

00:48:38.280 --> 00:48:39.780 align:middle line:84%
So to answer your
question, Spencer,

00:48:39.780 --> 00:48:43.380 align:middle line:84%
it can be in pretty bad shape
and it still seems to work.

00:48:43.380 --> 00:48:46.363 align:middle line:84%
So now, "Use below
article on the Olympics

00:48:46.363 --> 00:48:47.780 align:middle line:84%
to answer the
subsequent question.

00:48:47.780 --> 00:48:49.980 align:middle line:84%
If you don't know,
say you don't know."

00:48:49.980 --> 00:48:51.440 align:middle line:90%
So that's what we have.

00:48:51.440 --> 00:48:52.460 align:middle line:90%
There's the query.

00:48:52.460 --> 00:48:54.580 align:middle line:84%
And by the way, before
I send it into the LLM,

00:48:54.580 --> 00:48:56.240 align:middle line:84%
this is the actual query that
it's going to be sending.

00:48:56.240 --> 00:48:57.480 align:middle line:90%
I'm printing out the query.

00:48:57.480 --> 00:48:59.080 align:middle line:90%
Look at how long the query is.

00:48:59.080 --> 00:49:01.080 align:middle line:84%
"Use the article below and here
is the article," blah, blah,

00:49:01.080 --> 00:49:01.788 align:middle line:90%
blah, blah, blah.

00:49:01.788 --> 00:49:03.000 align:middle line:90%
Scroll, scroll, scroll.

00:49:03.000 --> 00:49:04.378 align:middle line:90%
This is the whole thing.

00:49:04.378 --> 00:49:05.420 align:middle line:90%
And it keeps on going on.

00:49:05.420 --> 00:49:08.080 align:middle line:84%
And then finally I say,
"Which teams won the gold?"

00:49:08.080 --> 00:49:09.540 align:middle line:90%
OK, so let's run it.

00:49:09.540 --> 00:49:12.200 align:middle line:90%


00:49:12.200 --> 00:49:15.160 align:middle line:90%
Look at that.

00:49:15.160 --> 00:49:17.600 align:middle line:84%
Women's curling, Great
Britain-- it got it right.

00:49:17.600 --> 00:49:19.280 align:middle line:90%
Pretty good, right?

00:49:19.280 --> 00:49:22.760 align:middle line:84%
It had to parse all that
crap to get and find

00:49:22.760 --> 00:49:25.440 align:middle line:90%
the nuggets, so nicely done.

00:49:25.440 --> 00:49:27.940 align:middle line:84%
Now, maybe it wasn't super hard
because we literally gave it

00:49:27.940 --> 00:49:28.660 align:middle line:90%
the answer.

00:49:28.660 --> 00:49:30.080 align:middle line:90%
So let's make it a bit harder.

00:49:30.080 --> 00:49:32.790 align:middle line:84%
So I noticed that this
person, Oskar Eriksson

00:49:32.790 --> 00:49:35.010 align:middle line:84%
won two golds in the event,
two medals in the event.

00:49:35.010 --> 00:49:39.210 align:middle line:84%
So let's ask if any athlete
won multiple medals.

00:49:39.210 --> 00:49:41.650 align:middle line:84%
That requires a little
bit of abstraction.

00:49:41.650 --> 00:49:45.130 align:middle line:84%
So, all right, same query, "Did
any athlete win multiple medals

00:49:45.130 --> 00:49:46.268 align:middle line:90%
in curling?"

00:49:46.268 --> 00:49:47.310 align:middle line:90%
The question has changed.

00:49:47.310 --> 00:49:49.010 align:middle line:90%
Everything else hasn't changed.

00:49:49.010 --> 00:49:49.710 align:middle line:90%
Hit it.

00:49:49.710 --> 00:49:51.810 align:middle line:90%
Let's see what happens.

00:49:51.810 --> 00:49:55.193 align:middle line:84%
"Yes, Oskar Eriksson won
multiple medals in curling.

00:49:55.193 --> 00:49:57.610 align:middle line:84%
He won a gold in the men's
event and a bronze in the mixed

00:49:57.610 --> 00:49:58.490 align:middle line:90%
doubles."

00:49:58.490 --> 00:50:00.250 align:middle line:90%
It's pretty cool.

00:50:00.250 --> 00:50:02.557 align:middle line:90%
Take that Google.

00:50:02.557 --> 00:50:04.890 align:middle line:84%
All right, now we come to
retrieval augmented generation

00:50:04.890 --> 00:50:06.290 align:middle line:84%
where instead of doing
it manually, obviously,

00:50:06.290 --> 00:50:08.767 align:middle line:84%
because it doesn't scale,
we will do it automatically.

00:50:08.767 --> 00:50:10.350 align:middle line:84%
And so the thing you
have to remember,

00:50:10.350 --> 00:50:11.933 align:middle line:84%
as I mentioned just
a few minutes ago,

00:50:11.933 --> 00:50:14.490 align:middle line:84%
is that there is a context
window for every LLM.

00:50:14.490 --> 00:50:20.450 align:middle line:84%
And for GPT-4 and 3.5-turbo, the
context window is 16,385 tokens.

00:50:20.450 --> 00:50:23.930 align:middle line:84%
That is the length of
the input and the output.

00:50:23.930 --> 00:50:25.430 align:middle line:90%
So we can't exceed that.

00:50:25.430 --> 00:50:27.870 align:middle line:84%
By the way, GPT-4's
context window is, I think,

00:50:27.870 --> 00:50:30.150 align:middle line:90%
up to 128,000 tokens.

00:50:30.150 --> 00:50:34.022 align:middle line:90%
And Google Gemini 1.5 Pro--

00:50:34.022 --> 00:50:35.730 align:middle line:84%
they really need to
work on their names--

00:50:35.730 --> 00:50:38.620 align:middle line:84%
Google Gemini 1.5 Pro,
the context window

00:50:38.620 --> 00:50:41.210 align:middle line:90%
is 1 million tokens.

00:50:41.210 --> 00:50:44.590 align:middle line:84%
And in research, they have
tested 10 million tokens,

00:50:44.590 --> 00:50:46.145 align:middle line:90%
so crazy times.

00:50:46.145 --> 00:50:48.270 align:middle line:84%
All that means is that you
can upload entire videos

00:50:48.270 --> 00:50:50.673 align:middle line:84%
and ask it questions
about the video.

00:50:50.673 --> 00:50:52.590 align:middle line:84%
All right, to come back
to this, what we'll do

00:50:52.590 --> 00:50:56.850 align:middle line:84%
is we'll only grab the data
from the Wikipedia articles, all

00:50:56.850 --> 00:50:58.350 align:middle line:84%
the articles about
the Olympics that

00:50:58.350 --> 00:51:01.390 align:middle line:84%
are relevant to our question by
using pre-trained embeddings.

00:51:01.390 --> 00:51:04.130 align:middle line:84%
So again, this is the thing
we talked about earlier.

00:51:04.130 --> 00:51:05.870 align:middle line:84%
This is the picture
we saw in class.

00:51:05.870 --> 00:51:07.750 align:middle line:84%
And the only thing
I want to point out

00:51:07.750 --> 00:51:10.390 align:middle line:84%
is that if you have a particular
embedding for a question

00:51:10.390 --> 00:51:12.430 align:middle line:84%
and a particular embedding
for a chunk of text

00:51:12.430 --> 00:51:14.110 align:middle line:84%
that you have in
your database, you

00:51:14.110 --> 00:51:17.230 align:middle line:84%
have to figure out how
similar, how related they are.

00:51:17.230 --> 00:51:20.806 align:middle line:90%
And for that, we can use what?

00:51:20.806 --> 00:51:26.110 align:middle line:84%
Dot product or
something almost as dot

00:51:26.110 --> 00:51:28.440 align:middle line:84%
product, which is more
easier for us to work with--

00:51:28.440 --> 00:51:30.630 align:middle line:90%
the cosine similarity.

00:51:30.630 --> 00:51:32.380 align:middle line:84%
We have done cosine
similarity previously.

00:51:32.380 --> 00:51:33.340 align:middle line:90%
I've explained it in class.

00:51:33.340 --> 00:51:34.780 align:middle line:84%
We're just going to
use cosine similarity--

00:51:34.780 --> 00:51:36.360 align:middle line:90%
how similar are these vectors?

00:51:36.360 --> 00:51:39.340 align:middle line:84%
So that's what
we're going to do.

00:51:39.340 --> 00:51:41.480 align:middle line:84%
So the same picture
as we saw in class.

00:51:41.480 --> 00:51:43.080 align:middle line:84%
So first, what
we'll do is we need

00:51:43.080 --> 00:51:44.970 align:middle line:84%
to break up the
dataset into sections

00:51:44.970 --> 00:51:46.720 align:middle line:84%
and then take each
section and then run it

00:51:46.720 --> 00:51:47.887 align:middle line:90%
through the embedding thing.

00:51:47.887 --> 00:51:50.033 align:middle line:84%
But fortunately for
us, I have code here

00:51:50.033 --> 00:51:51.700 align:middle line:84%
which actually does
it for you manually.

00:51:51.700 --> 00:51:53.117 align:middle line:84%
You can play around
with it later.

00:51:53.117 --> 00:51:56.292 align:middle line:84%
But OpenAI has already given
us the chunked dataset.

00:51:56.292 --> 00:51:58.500 align:middle line:84%
So we'll just use that
because it's just easy for us.

00:51:58.500 --> 00:52:00.875 align:middle line:84%
And I downloaded it already
because it takes five minutes

00:52:00.875 --> 00:52:01.540 align:middle line:90%
to download.

00:52:01.540 --> 00:52:03.373 align:middle line:84%
I've downloaded this
thing and I've stuck it

00:52:03.373 --> 00:52:05.100 align:middle line:90%
in a particular dataframe here.

00:52:05.100 --> 00:52:08.280 align:middle line:84%
So let's print out five
randomly chosen chunks.

00:52:08.280 --> 00:52:11.680 align:middle line:84%
So you can see here,
this is the first chunk,

00:52:11.680 --> 00:52:13.900 align:middle line:90%
somebody else, somebody else.

00:52:13.900 --> 00:52:16.940 align:middle line:84%
And look at all this
crazy stuff here.

00:52:16.940 --> 00:52:18.180 align:middle line:90%
The formatting is off.

00:52:18.180 --> 00:52:20.880 align:middle line:84%
But these are all, basically,
paragraphs and sections

00:52:20.880 --> 00:52:23.130 align:middle line:84%
just grabbed straight from
Wikipedia with no cleaning.

00:52:23.130 --> 00:52:25.760 align:middle line:90%


00:52:25.760 --> 00:52:29.780 align:middle line:84%
Now, we define a simple
function to basically send

00:52:29.780 --> 00:52:32.820 align:middle line:84%
in any arbitrary piece of
text into the embedding model

00:52:32.820 --> 00:52:35.560 align:middle line:84%
and get the contextual
embedding vector out.

00:52:35.560 --> 00:52:38.387 align:middle line:84%
And there is this little
function that does that.

00:52:38.387 --> 00:52:39.720 align:middle line:90%
We are using an embedding model.

00:52:39.720 --> 00:52:41.560 align:middle line:84%
We send in text, it
gives you something.

00:52:41.560 --> 00:52:42.740 align:middle line:90%
So let's try it on that.

00:52:42.740 --> 00:52:44.140 align:middle line:90%
"HODL is amazing!"

00:52:44.140 --> 00:52:45.440 align:middle line:90%
You should get a vector back.

00:52:45.440 --> 00:52:51.220 align:middle line:90%


00:52:51.220 --> 00:52:52.300 align:middle line:90%
Oh, come on.

00:52:52.300 --> 00:52:55.092 align:middle line:90%
Don't fail me now.

00:52:55.092 --> 00:52:57.060 align:middle line:90%
Phew, all right.

00:52:57.060 --> 00:52:58.380 align:middle line:90%
How long is it?

00:52:58.380 --> 00:53:00.260 align:middle line:90%
1536.

00:53:00.260 --> 00:53:02.440 align:middle line:84%
So how about I say,
HODL is incredible,

00:53:02.440 --> 00:53:03.600 align:middle line:90%
like HODL is amazing?

00:53:03.600 --> 00:53:05.350 align:middle line:84%
Hopefully, the two
vectors will be kind of

00:53:05.350 --> 00:53:08.620 align:middle line:90%
similar in terms of cosine.

00:53:08.620 --> 00:53:11.060 align:middle line:84%
And so to calculate
the cosine distance,

00:53:11.060 --> 00:53:13.420 align:middle line:84%
I use this particular
function from SciPy.

00:53:13.420 --> 00:53:15.740 align:middle line:84%
It just calculates
the cosine similarity.

00:53:15.740 --> 00:53:18.700 align:middle line:90%
And I hit it, so 0.9934.

00:53:18.700 --> 00:53:20.200 align:middle line:90%
Maximum is 1.

00:53:20.200 --> 00:53:22.560 align:middle line:84%
So 0.934 means that
they're very, very similar,

00:53:22.560 --> 00:53:25.150 align:middle line:84%
which is comforting because
amazing and incredible

00:53:25.150 --> 00:53:27.590 align:middle line:90%
are obviously synonyms.

00:53:27.590 --> 00:53:31.022 align:middle line:84%
So now given a dataframe with
a column of text chunks in it,

00:53:31.022 --> 00:53:33.230 align:middle line:84%
we can use this function on
every one of these things

00:53:33.230 --> 00:53:34.355 align:middle line:90%
to calculate the embedding.

00:53:34.355 --> 00:53:36.970 align:middle line:84%
And you have a function here
that basically does it for you.

00:53:36.970 --> 00:53:39.610 align:middle line:84%
I'm not going to run it
because it takes a long time.

00:53:39.610 --> 00:53:41.190 align:middle line:90%
But you can run it later on.

00:53:41.190 --> 00:53:41.950 align:middle line:90%
Just be prepared.

00:53:41.950 --> 00:53:44.670 align:middle line:84%
Go get a cup of coffee and
stuff while it does it.

00:53:44.670 --> 00:53:47.110 align:middle line:84%
But happily for us,
OpenAI has actually

00:53:47.110 --> 00:53:49.497 align:middle line:84%
already done this step for
us, so we don't have to.

00:53:49.497 --> 00:53:51.330 align:middle line:84%
So it's already available
in this dataframe.

00:53:51.330 --> 00:53:54.613 align:middle line:84%
So if you actually look
at this, you can see here

00:53:54.613 --> 00:53:56.030 align:middle line:84%
there is a text,
and then there is

00:53:56.030 --> 00:53:59.750 align:middle line:84%
an embedding that's sitting
right there right next to it.

00:53:59.750 --> 00:54:02.430 align:middle line:84%
And these embeddings
are whatever--

00:54:02.430 --> 00:54:03.170 align:middle line:90%
how long is it?

00:54:03.170 --> 00:54:09.650 align:middle line:84%
1536 long, 1536 long vectors All
right, so that's what we have.

00:54:09.650 --> 00:54:14.390 align:middle line:90%


00:54:14.390 --> 00:54:17.830 align:middle line:84%
So now that we have this thing,
whenever we get a question,

00:54:17.830 --> 00:54:19.890 align:middle line:84%
we calculate the
question's embedding

00:54:19.890 --> 00:54:22.310 align:middle line:84%
and then compare calculated
cosine similarity

00:54:22.310 --> 00:54:25.240 align:middle line:84%
with all the embeddings
sitting in this dataframe.

00:54:25.240 --> 00:54:27.960 align:middle line:84%
So to do that we're going
to define a couple of helper

00:54:27.960 --> 00:54:29.000 align:middle line:90%
functions here.

00:54:29.000 --> 00:54:31.200 align:middle line:84%
You can read through the
Python later to understand.

00:54:31.200 --> 00:54:35.000 align:middle line:84%
It's just basic Python
manipulations that are going on.

00:54:35.000 --> 00:54:37.240 align:middle line:84%
And so let's just
test this function.

00:54:37.240 --> 00:54:39.080 align:middle line:84%
So basically, we have
a little function

00:54:39.080 --> 00:54:42.080 align:middle line:84%
called
strings_ranked_by_relatedness

00:54:42.080 --> 00:54:45.660 align:middle line:84%
where you give it any
input question or text,

00:54:45.660 --> 00:54:48.360 align:middle line:84%
and then it's going to
give you the top five

00:54:48.360 --> 00:54:52.760 align:middle line:84%
most related chunks of text
that it had in its dataframe.

00:54:52.760 --> 00:54:55.870 align:middle line:90%
So let me just run this thing.

00:54:55.870 --> 00:54:59.820 align:middle line:90%


00:54:59.820 --> 00:55:01.900 align:middle line:84%
So curling, the
things it pulls back,

00:55:01.900 --> 00:55:04.520 align:middle line:84%
it better involve curling
and medals and so on.

00:55:04.520 --> 00:55:08.040 align:middle line:84%
So this one has a cosine
similarity of 0.88.

00:55:08.040 --> 00:55:10.140 align:middle line:84%
Curling at the 2012
Olympics, that's good.

00:55:10.140 --> 00:55:13.160 align:middle line:84%
Results summary, medal
summary, results summary--

00:55:13.160 --> 00:55:14.080 align:middle line:90%
It's all pretty good.

00:55:14.080 --> 00:55:15.960 align:middle line:84%
Even the fifth one has
a cosine similarity

00:55:15.960 --> 00:55:17.860 align:middle line:90%
of 0.867, which is pretty high.

00:55:17.860 --> 00:55:19.180 align:middle line:90%
So it's doing the right things.

00:55:19.180 --> 00:55:21.660 align:middle line:84%
It's picked up curling
gold medal with input text

00:55:21.660 --> 00:55:23.700 align:middle line:84%
and it's picked up the
right things from it.

00:55:23.700 --> 00:55:29.435 align:middle line:84%
Now let's see what we can do
with the original question.

00:55:29.435 --> 00:55:31.560 align:middle line:84%
So here is a header I'm
going to use in the prompt.

00:55:31.560 --> 00:55:33.300 align:middle line:84%
I'm going to say, "Use
the below articles

00:55:33.300 --> 00:55:34.847 align:middle line:84%
to answer the
subsequent question.

00:55:34.847 --> 00:55:36.680 align:middle line:84%
Answer the questions
truthfully as possible.

00:55:36.680 --> 00:55:38.160 align:middle line:84%
And if you're unsure of
the answer, say I'm sorry,

00:55:38.160 --> 00:55:39.240 align:middle line:90%
I don't know," as before.

00:55:39.240 --> 00:55:41.220 align:middle line:90%
OK, that's our prompt.

00:55:41.220 --> 00:55:42.360 align:middle line:90%
And now here is the thing.

00:55:42.360 --> 00:55:44.500 align:middle line:84%
We don't want to exceed
the context window.

00:55:44.500 --> 00:55:47.078 align:middle line:84%
So we need to count the
tokens we are sending in

00:55:47.078 --> 00:55:48.620 align:middle line:84%
and the likely number
of tokens we're

00:55:48.620 --> 00:55:50.860 align:middle line:84%
going to get back so that
we don't exceed the budget.

00:55:50.860 --> 00:55:54.740 align:middle line:84%
So we use this package called
tiktoken package for this.

00:55:54.740 --> 00:55:56.830 align:middle line:84%
And then it just helps
you count the tokens.

00:55:56.830 --> 00:55:58.080 align:middle line:90%
And you can read through this.

00:55:58.080 --> 00:56:00.780 align:middle line:84%
It's just again, some basic
Python for counting tokens.

00:56:00.780 --> 00:56:04.820 align:middle line:84%
And now what we do is,
this is where we actually

00:56:04.820 --> 00:56:06.180 align:middle line:90%
assemble the prompt.

00:56:06.180 --> 00:56:08.320 align:middle line:90%
We start with the header.

00:56:08.320 --> 00:56:11.040 align:middle line:84%
We have the header, which says
"be truthful" and all that.

00:56:11.040 --> 00:56:15.480 align:middle line:84%
Then we say, here is a question
that I'm going to ask you.

00:56:15.480 --> 00:56:19.430 align:middle line:84%
And then you go in there and
keep grabbing Wikipedia articles

00:56:19.430 --> 00:56:22.190 align:middle line:84%
till the number of
tokens in your prompt

00:56:22.190 --> 00:56:26.105 align:middle line:84%
is exceeding your token
budget, and then you stop.

00:56:26.105 --> 00:56:27.730 align:middle line:84%
When you're about to
exceed the budget,

00:56:27.730 --> 00:56:29.910 align:middle line:84%
you stop because you
can't exceed the budget.

00:56:29.910 --> 00:56:32.190 align:middle line:90%
And that's the whole thing.

00:56:32.190 --> 00:56:38.470 align:middle line:84%
So here, let's just do
tiktoken, run this function.

00:56:38.470 --> 00:56:42.470 align:middle line:84%
Now, it turns out, as you saw,
we can go up to 1,600 something

00:56:42.470 --> 00:56:43.930 align:middle line:90%
tokens in the context window.

00:56:43.930 --> 00:56:47.750 align:middle line:84%
I'm just using three
3,700 as my budget,

00:56:47.750 --> 00:56:50.630 align:middle line:84%
partly just to show you how
to use this thing and also

00:56:50.630 --> 00:56:54.550 align:middle line:84%
because it's charging my credit
card for every token that

00:56:54.550 --> 00:56:55.630 align:middle line:90%
I'm using.

00:56:55.630 --> 00:56:58.070 align:middle line:90%
So I'm just being careful.

00:56:58.070 --> 00:56:59.277 align:middle line:90%
It charges by the token.

00:56:59.277 --> 00:57:00.610 align:middle line:90%
It's a beautiful business model.

00:57:00.610 --> 00:57:02.338 align:middle line:90%
Anyway, so back here.

00:57:02.338 --> 00:57:04.630 align:middle line:84%
So let's ask the question,
"Which athletes won the gold

00:57:04.630 --> 00:57:06.470 align:middle line:84%
medal in curling at the
2022 Winter Olympics?"

00:57:06.470 --> 00:57:08.290 align:middle line:84%
Here is the dataframe
that you should use.

00:57:08.290 --> 00:57:09.490 align:middle line:90%
Here is the GPT model.

00:57:09.490 --> 00:57:11.750 align:middle line:90%
And don't exceed 3,700 tokens.

00:57:11.750 --> 00:57:14.550 align:middle line:90%
That's the query or the prompt.

00:57:14.550 --> 00:57:16.370 align:middle line:84%
It's going to compose
the prompt now.

00:57:16.370 --> 00:57:18.730 align:middle line:90%
And this is the whole prompt.

00:57:18.730 --> 00:57:19.980 align:middle line:90%
Let's just go to the very top.

00:57:19.980 --> 00:57:20.900 align:middle line:90%
It's really long.

00:57:20.900 --> 00:57:24.527 align:middle line:90%


00:57:24.527 --> 00:57:26.360 align:middle line:84%
All right, "Use the
below articles to answer

00:57:26.360 --> 00:57:27.985 align:middle line:84%
the subsequent
question-- as possible."

00:57:27.985 --> 00:57:30.720 align:middle line:84%
And boom, boom, boom, boom,
boom, it has all these things.

00:57:30.720 --> 00:57:32.800 align:middle line:84%
It's added a whole
bunch of paragraphs

00:57:32.800 --> 00:57:34.800 align:middle line:90%
from the Wikipedia pages.

00:57:34.800 --> 00:57:36.700 align:middle line:84%
And then it finally
ends with the question,

00:57:36.700 --> 00:57:38.520 align:middle line:90%
"which athletes won the gold?"

00:57:38.520 --> 00:57:40.960 align:middle line:84%
All right, now let's
just ask it the thing.

00:57:40.960 --> 00:57:43.060 align:middle line:84%
And this is just
a little function

00:57:43.060 --> 00:57:45.200 align:middle line:90%
to send stuff into the API.

00:57:45.200 --> 00:57:49.960 align:middle line:84%
And now we are finally ready
to ask GPT the question.

00:57:49.960 --> 00:57:50.670 align:middle line:90%
Fingers crossed.

00:57:50.670 --> 00:57:53.200 align:middle line:90%


00:57:53.200 --> 00:57:57.440 align:middle line:84%
All right, curling where,
Stefania, curling in the mixed

00:57:57.440 --> 00:57:58.240 align:middle line:90%
doubles.

00:57:58.240 --> 00:58:00.840 align:middle line:84%
And the team consisting
of blah, blah, blah,

00:58:00.840 --> 00:58:03.600 align:middle line:90%
in the men's tournament.

00:58:03.600 --> 00:58:04.680 align:middle line:90%
Oh, interesting.

00:58:04.680 --> 00:58:07.280 align:middle line:84%
It has actually ignored the
Great Britain people completely,

00:58:07.280 --> 00:58:07.780 align:middle line:90%
I think.

00:58:07.780 --> 00:58:10.464 align:middle line:90%


00:58:10.464 --> 00:58:12.080 align:middle line:90%
Last night it didn't.

00:58:12.080 --> 00:58:14.930 align:middle line:90%
Welcome to stochasticity.

00:58:14.930 --> 00:58:15.680 align:middle line:90%
So you can try it.

00:58:15.680 --> 00:58:18.740 align:middle line:84%
When you try it, it will
actually give you the thing.

00:58:18.740 --> 00:58:20.740 align:middle line:84%
And so let's ask
it now a question

00:58:20.740 --> 00:58:23.960 align:middle line:84%
about the 2016 Winter
Olympics, which, by the way,

00:58:23.960 --> 00:58:25.220 align:middle line:90%
didn't happen.

00:58:25.220 --> 00:58:27.300 align:middle line:84%
There were no Winter
Olympics in 2016.

00:58:27.300 --> 00:58:31.260 align:middle line:90%
So if you ask it--

00:58:31.260 --> 00:58:32.300 align:middle line:90%
"Sorry, I don't know."

00:58:32.300 --> 00:58:35.740 align:middle line:84%
All right, now let's
change the header so

00:58:35.740 --> 00:58:37.780 align:middle line:90%
that we don't say be truthful.

00:58:37.780 --> 00:58:39.180 align:middle line:84%
So we will remove
the need for it

00:58:39.180 --> 00:58:41.010 align:middle line:84%
to be truthful and
see what happens.

00:58:41.010 --> 00:58:44.700 align:middle line:90%


00:58:44.700 --> 00:58:45.990 align:middle line:90%
"Which athletes won the gold?"

00:58:45.990 --> 00:58:51.020 align:middle line:90%


00:58:51.020 --> 00:58:54.220 align:middle line:84%
Oh, now it's telling you
about the 2022 Olympics.

00:58:54.220 --> 00:58:57.580 align:middle line:84%
So it answered an irrelevant
question accurately

00:58:57.580 --> 00:59:00.660 align:middle line:84%
if you removed the need
for it to be truthful.

00:59:00.660 --> 00:59:04.940 align:middle line:84%
So I guess the moral of the
story is that, first of all,

00:59:04.940 --> 00:59:07.915 align:middle line:84%
you can use RAG to grab
stuff from vast databases,

00:59:07.915 --> 00:59:10.040 align:middle line:84%
and it's very heavily used
in industry, number one.

00:59:10.040 --> 00:59:12.700 align:middle line:84%
Number two, you have to be
careful about these token

00:59:12.700 --> 00:59:14.850 align:middle line:90%
budgets and so on and so forth.

00:59:14.850 --> 00:59:17.230 align:middle line:84%
And small wording
changes in the prompt

00:59:17.230 --> 00:59:19.350 align:middle line:84%
can actually dramatically
alter behavior,

00:59:19.350 --> 00:59:21.550 align:middle line:84%
which makes it very difficult
in enterprise settings

00:59:21.550 --> 00:59:23.550 align:middle line:90%
to do QA on this stuff.

00:59:23.550 --> 00:59:27.430 align:middle line:84%
So a lot of care
has to go into it.

00:59:27.430 --> 00:59:29.695 align:middle line:84%
And you have seen examples
of, for example, Air Canada

00:59:29.695 --> 00:59:31.070 align:middle line:84%
had a chatbot
which actually gave

00:59:31.070 --> 00:59:32.370 align:middle line:90%
the wrong advice to a customer.

00:59:32.370 --> 00:59:33.750 align:middle line:90%
The customer sued Air Canada.

00:59:33.750 --> 00:59:36.288 align:middle line:84%
And then the court ruled
in favor of the passenger.

00:59:36.288 --> 00:59:38.330 align:middle line:84%
And then they pulled the
chatbot off the website.

00:59:38.330 --> 00:59:39.970 align:middle line:84%
So you've got to
be very careful.

00:59:39.970 --> 00:59:42.530 align:middle line:84%
I think without a human in the
loop checking these answers,

00:59:42.530 --> 00:59:45.072 align:middle line:84%
it's kind of dangerous, in my
opinion, at this current state.

00:59:45.072 --> 00:59:46.990 align:middle line:90%
Hopefully, it will get better.

00:59:46.990 --> 00:59:50.170 align:middle line:84%
So there's a lot of potential in
it, but you have to be careful.

00:59:50.170 --> 00:59:52.270 align:middle line:84%
All right, so this
is what we have.

00:59:52.270 --> 00:59:55.390 align:middle line:84%
And you can actually take
this thing here and use it.

00:59:55.390 --> 00:59:58.590 align:middle line:84%
You can actually take a
1,000-page PDF that you might

00:59:58.590 --> 01:00:01.530 align:middle line:84%
have or something and then
chunk it and use this approach.

01:00:01.530 --> 01:00:03.130 align:middle line:84%
And I've done it for a whole
bunch of different things.

01:00:03.130 --> 01:00:05.010 align:middle line:84%
It actually works really
well most of the time.

01:00:05.010 --> 01:00:06.468 align:middle line:84%
It will make errors
here and there.

01:00:06.468 --> 01:00:09.950 align:middle line:84%
Most of the time it
actually works really well.

01:00:09.950 --> 01:00:11.830 align:middle line:90%
Yeah?

01:00:11.830 --> 01:00:14.360 align:middle line:90%
AUDIENCE: Just a quick question.

01:00:14.360 --> 01:00:18.000 align:middle line:84%
GPT-4 now lets you upload a
PDF and it remains locked.

01:00:18.000 --> 01:00:19.000 align:middle line:90%
Is it chunking that?

01:00:19.000 --> 01:00:21.220 align:middle line:84%
Or is it actually
ingesting all the--

01:00:21.220 --> 01:00:23.262 align:middle line:84%
RAMA RAMAKRISHNAN: No,
when you upload something,

01:00:23.262 --> 01:00:25.580 align:middle line:84%
because GPT-4-Turbo
has 128,000 tokens,

01:00:25.580 --> 01:00:28.390 align:middle line:84%
which means it can accommodate a
whole, long bunch of documents.

01:00:28.390 --> 01:00:30.640 align:middle line:84%
So when you upload stuff,
it's not doing any chunking.

01:00:30.640 --> 01:00:32.760 align:middle line:84%
The chunking we are talking
about, you have to do.

01:00:32.760 --> 01:00:34.620 align:middle line:84%
The LLM doesn't even
know you're doing it.

01:00:34.620 --> 01:00:36.120 align:middle line:84%
As far as the LLM
is concerned, it's

01:00:36.120 --> 01:00:37.527 align:middle line:90%
only seeing the prompt it sees.

01:00:37.527 --> 01:00:39.860 align:middle line:84%
And the prompt says, hey,
here's a bunch of information,

01:00:39.860 --> 01:00:40.420 align:middle line:90%
here's a question.

01:00:40.420 --> 01:00:42.628 align:middle line:84%
Answer it for me using this
question and be truthful.

01:00:42.628 --> 01:00:44.120 align:middle line:90%
That's it.

01:00:44.120 --> 01:00:47.820 align:middle line:84%
Now when you ask these
things a question,

01:00:47.820 --> 01:00:49.900 align:middle line:84%
which is later than
its training data,

01:00:49.900 --> 01:00:53.320 align:middle line:84%
you will actually see GPT-4
saying, "doing a Bing search"

01:00:53.320 --> 01:00:54.263 align:middle line:90%
and things like that.

01:00:54.263 --> 01:00:55.680 align:middle line:84%
There, what's
actually going on is

01:00:55.680 --> 01:00:59.080 align:middle line:84%
there's a pre-processing
step and a program which

01:00:59.080 --> 01:01:01.520 align:middle line:84%
is doing a Bing search,
gathering a bunch of Bing

01:01:01.520 --> 01:01:04.080 align:middle line:84%
results, taking the
top few results,

01:01:04.080 --> 01:01:07.220 align:middle line:84%
chunking, embedding,
backing into a prompt,

01:01:07.220 --> 01:01:08.620 align:middle line:90%
sending it into GPT-4.

01:01:08.620 --> 01:01:11.220 align:middle line:84%
And you don't know all this
is going on under the hood.

01:01:11.220 --> 01:01:13.160 align:middle line:84%
So when it's actually thinking
and saying "Bing search,"

01:01:13.160 --> 01:01:14.785 align:middle line:84%
this is what's going
on under the hood.

01:01:14.785 --> 01:01:19.180 align:middle line:90%


01:01:19.180 --> 01:01:21.640 align:middle line:84%
Was there a question
somewhere here?

01:01:21.640 --> 01:01:22.340 align:middle line:90%
No.

01:01:22.340 --> 01:01:23.040 align:middle line:90%
Oh, sorry.

01:01:23.040 --> 01:01:24.275 align:middle line:90%
Yeah?

01:01:24.275 --> 01:01:26.150 align:middle line:84%
AUDIENCE: I have a
question about formatting.

01:01:26.150 --> 01:01:27.380 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah?

01:01:27.380 --> 01:01:30.980 align:middle line:84%
AUDIENCE: So it seems to be
able to understand and ignore

01:01:30.980 --> 01:01:35.060 align:middle line:84%
irrelevant formatting, even when
there's colloquial tables not

01:01:35.060 --> 01:01:36.480 align:middle line:90%
clearly defined tables.

01:01:36.480 --> 01:01:39.120 align:middle line:84%
And also when it
outputs formats,

01:01:39.120 --> 01:01:42.020 align:middle line:84%
it's able to do it
really human-ly.

01:01:42.020 --> 01:01:44.820 align:middle line:84%
Is that something
that it's figuring out

01:01:44.820 --> 01:01:45.945 align:middle line:90%
through the neural network?

01:01:45.945 --> 01:01:47.487 align:middle line:84%
Or is it something
that's going to be

01:01:47.487 --> 01:01:49.820 align:middle line:84%
programmed in the header
somewhere with the standard--

01:01:49.820 --> 01:01:51.600 align:middle line:84%
RAMA RAMAKRISHNAN: There is no
explicit programming going on.

01:01:51.600 --> 01:01:54.380 align:middle line:84%
It's typically because a lot of
the question-answer pairs that

01:01:54.380 --> 01:01:57.260 align:middle line:84%
was used for supervised
fine-tuning and reinforcement

01:01:57.260 --> 01:02:01.980 align:middle line:84%
learning, the better answers
with the same badly-formatted

01:02:01.980 --> 01:02:03.920 align:middle line:84%
input, the better answers
were just rewarded,

01:02:03.920 --> 01:02:04.840 align:middle line:90%
were ranked higher.

01:02:04.840 --> 01:02:06.020 align:middle line:90%
That's what's going on.

01:02:06.020 --> 01:02:09.230 align:middle line:84%
But on a related note, one
thing that's very useful

01:02:09.230 --> 01:02:12.910 align:middle line:84%
is that you can actually ask
it to give you the answer back

01:02:12.910 --> 01:02:15.750 align:middle line:84%
using certain formats, like
markdown and JSON and things

01:02:15.750 --> 01:02:16.630 align:middle line:90%
like that.

01:02:16.630 --> 01:02:20.072 align:middle line:84%
And by forcing it to adhere to
certain well-defined formats,

01:02:20.072 --> 01:02:22.030 align:middle line:84%
you actually increase
the chance of it actually

01:02:22.030 --> 01:02:24.990 align:middle line:84%
getting the right answer
in the first place.

01:02:24.990 --> 01:02:27.610 align:middle line:84%
Again, there's a whole
tangent here we can go into.

01:02:27.610 --> 01:02:29.950 align:middle line:84%
But those are some
of the things that

01:02:29.950 --> 01:02:31.630 align:middle line:90%
are part of prompt engineering.

01:02:31.630 --> 01:02:33.690 align:middle line:84%
All right, so that's
what we have here.

01:02:33.690 --> 01:02:34.930 align:middle line:90%
Back to the PowerPoint.

01:02:34.930 --> 01:02:40.630 align:middle line:90%


01:02:40.630 --> 01:02:42.610 align:middle line:84%
So that's retrieval
augmented generation.

01:02:42.610 --> 01:02:44.710 align:middle line:84%
And we finally come
to fine-tuning.

01:02:44.710 --> 01:02:48.030 align:middle line:90%
So fine-tuning is when--

01:02:48.030 --> 01:02:50.750 align:middle line:84%
up to this point, all
the things we have seen

01:02:50.750 --> 01:02:53.318 align:middle line:84%
don't alter the
internals of the LLM.

01:02:53.318 --> 01:02:55.110 align:middle line:84%
You have not messed
around with the weights

01:02:55.110 --> 01:02:56.130 align:middle line:90%
or changed them at all.

01:02:56.130 --> 01:02:58.576 align:middle line:84%
You're just using
it as a black box.

01:02:58.576 --> 01:03:01.383 align:middle line:84%
With fine-tuning, you actually
will train it further,

01:03:01.383 --> 01:03:03.050 align:middle line:84%
meaning the weights
are going to change.

01:03:03.050 --> 01:03:05.710 align:middle line:90%


01:03:05.710 --> 01:03:09.430 align:middle line:84%
Now, remember, we take something
like a causal LLM like GPT--

01:03:09.430 --> 01:03:12.063 align:middle line:90%


01:03:12.063 --> 01:03:13.930 align:middle line:84%
and this, I haven't
fixed this yet.

01:03:13.930 --> 01:03:17.210 align:middle line:84%
There is no ReLU here,
as I mentioned earlier.

01:03:17.210 --> 01:03:19.210 align:middle line:90%
Just remember that.

01:03:19.210 --> 01:03:22.050 align:middle line:84%
And then if you have
domain-specific input-output

01:03:22.050 --> 01:03:23.770 align:middle line:84%
examples, like input
and output, you

01:03:23.770 --> 01:03:26.550 align:middle line:84%
can just train it
like this, input

01:03:26.550 --> 01:03:28.850 align:middle line:90%
and then the shifted output.

01:03:28.850 --> 01:03:32.503 align:middle line:84%
And that will update these
weights, all these weights.

01:03:32.503 --> 01:03:34.170 align:middle line:84%
So this is basically
fine-tuning exactly

01:03:34.170 --> 01:03:36.650 align:middle line:90%
like we saw with BERT and so on.

01:03:36.650 --> 01:03:39.590 align:middle line:84%
And even with ResNet, it's
the same sort of thing.

01:03:39.590 --> 01:03:41.290 align:middle line:90%
That is fine-tuning.

01:03:41.290 --> 01:03:43.590 align:middle line:84%
Now, before we discuss the
mechanics of how to do it,

01:03:43.590 --> 01:03:46.370 align:middle line:84%
I want to show you a quick
example of the usefulness

01:03:46.370 --> 01:03:48.610 align:middle line:90%
of fine-tuning.

01:03:48.610 --> 01:03:50.210 align:middle line:84%
Imagine for a
second that we want

01:03:50.210 --> 01:03:53.690 align:middle line:84%
to generate synthetic
product reviews from product

01:03:53.690 --> 01:03:55.210 align:middle line:90%
descriptions.

01:03:55.210 --> 01:03:58.570 align:middle line:84%
So we are building some product
which can simulate customer

01:03:58.570 --> 01:04:00.610 align:middle line:90%
behavior in e-commerce.

01:04:00.610 --> 01:04:03.770 align:middle line:84%
And for that, we need to be able
to generate the kinds of reviews

01:04:03.770 --> 01:04:05.740 align:middle line:84%
that customers
might come up with.

01:04:05.740 --> 01:04:08.162 align:middle line:84%
And writing a lot of reviews
is very time consuming.

01:04:08.162 --> 01:04:10.620 align:middle line:84%
But what you can do is you can
get a whole bunch of product

01:04:10.620 --> 01:04:13.240 align:middle line:90%
descriptions from the internet.

01:04:13.240 --> 01:04:16.300 align:middle line:84%
So let's say you ask an
LLM, hey, write a product

01:04:16.300 --> 01:04:19.000 align:middle line:84%
review using this information
here, product description here.

01:04:19.000 --> 01:04:20.380 align:middle line:90%
And it comes up with this--

01:04:20.380 --> 01:04:20.980 align:middle line:90%
"Timeless.

01:04:20.980 --> 01:04:21.540 align:middle line:90%
Authentic.

01:04:21.540 --> 01:04:24.100 align:middle line:90%
Iconic."

01:04:24.100 --> 01:04:27.460 align:middle line:84%
Seriously, do product reviewers
actually write stuff like this?

01:04:27.460 --> 01:04:28.140 align:middle line:90%
No.

01:04:28.140 --> 01:04:30.700 align:middle line:90%
This looks like marketing copy.

01:04:30.700 --> 01:04:33.020 align:middle line:84%
This reads like marketing
copy because there's

01:04:33.020 --> 01:04:35.380 align:middle line:84%
a whole bunch of marketing
copy on the internet.

01:04:35.380 --> 01:04:36.430 align:middle line:90%
So it's not good.

01:04:36.430 --> 01:04:37.680 align:middle line:90%
It doesn't feel like a review.

01:04:37.680 --> 01:04:39.540 align:middle line:90%
It's not authentic.

01:04:39.540 --> 01:04:42.160 align:middle line:84%
Here's another example
for Urban Outfitters.

01:04:42.160 --> 01:04:46.580 align:middle line:84%
And it says, "The boxy and
cropped silhouette is flattering

01:04:46.580 --> 01:04:47.720 align:middle line:90%
on all body types."

01:04:47.720 --> 01:04:48.320 align:middle line:90%
Come on!

01:04:48.320 --> 01:04:50.620 align:middle line:90%
[LAUGHTER]

01:04:50.620 --> 01:04:52.740 align:middle line:90%
So it's not going to work.

01:04:52.740 --> 01:04:55.300 align:middle line:84%
So what we do is we
fine-tune the LLM.

01:04:55.300 --> 01:04:57.780 align:middle line:84%
We can take an LLM and
we can fine-tune it

01:04:57.780 --> 01:05:01.060 align:middle line:84%
with instruction, product
description, and product review

01:05:01.060 --> 01:05:03.420 align:middle line:90%
examples.

01:05:03.420 --> 01:05:04.590 align:middle line:90%
That's what you can do.

01:05:04.590 --> 01:05:07.790 align:middle line:84%
So for instance, we can
take something like this.

01:05:07.790 --> 01:05:09.170 align:middle line:90%
Let me zoom into this thing.

01:05:09.170 --> 01:05:14.510 align:middle line:90%


01:05:14.510 --> 01:05:17.110 align:middle line:84%
So it says here, "Write
a positive review

01:05:17.110 --> 01:05:18.410 align:middle line:90%
for the following product."

01:05:18.410 --> 01:05:20.870 align:middle line:84%
And then this is
the description,

01:05:20.870 --> 01:05:22.990 align:middle line:84%
is the input and the
output is, "The best!

01:05:22.990 --> 01:05:24.770 align:middle line:84%
Carhart is my
husband's favorite.

01:05:24.770 --> 01:05:26.190 align:middle line:90%
They fit well."

01:05:26.190 --> 01:05:27.610 align:middle line:90%
They feel like product reviews.

01:05:27.610 --> 01:05:29.270 align:middle line:84%
So you just have to
get a few hundred

01:05:29.270 --> 01:05:33.150 align:middle line:84%
of these product review
examples, just a few hundred.

01:05:33.150 --> 01:05:34.730 align:middle line:84%
And you may not
even need that much.

01:05:34.730 --> 01:05:37.390 align:middle line:90%


01:05:37.390 --> 01:05:41.750 align:middle line:84%
Once you do that, you
just do fine-tuning,

01:05:41.750 --> 01:05:44.550 align:middle line:90%
like I showed earlier--

01:05:44.550 --> 01:05:45.790 align:middle line:90%
instruction, input, output.

01:05:45.790 --> 01:05:47.790 align:middle line:84%
And then you take that
output and shift it a bit

01:05:47.790 --> 01:05:50.030 align:middle line:84%
and make it the actual
label, the actual output.

01:05:50.030 --> 01:05:52.530 align:middle line:84%
Fine tune, fine tune, fine tune,
fine tune a bunch of times.

01:05:52.530 --> 01:05:54.330 align:middle line:84%
Gradient descent,
weights get updated.

01:05:54.330 --> 01:05:56.470 align:middle line:84%
Now you have a new
and updated element.

01:05:56.470 --> 01:05:59.730 align:middle line:84%
And when you do that,
now for the same things,

01:05:59.730 --> 01:06:00.950 align:middle line:90%
here's what you get.

01:06:00.950 --> 01:06:01.922 align:middle line:90%
"Write a review.

01:06:01.922 --> 01:06:03.630 align:middle line:84%
These are the best
jeans I've ever owned.

01:06:03.630 --> 01:06:05.203 align:middle line:90%
I am--" whatever, some details.

01:06:05.203 --> 01:06:06.870 align:middle line:84%
"I've been wearing
them for a few weeks.

01:06:06.870 --> 01:06:08.075 align:middle line:90%
They still look brand new."

01:06:08.075 --> 01:06:08.950 align:middle line:90%
It looks much better.

01:06:08.950 --> 01:06:11.010 align:middle line:90%
It doesn't look like marketing.

01:06:11.010 --> 01:06:13.170 align:middle line:84%
This is completely
fake, by the way.

01:06:13.170 --> 01:06:15.190 align:middle line:84%
The LLM came up with it
after the fine-tuning.

01:06:15.190 --> 01:06:16.982 align:middle line:84%
And then we say, "Write
a horrible review,"

01:06:16.982 --> 01:06:18.650 align:middle line:90%
because we want to be balanced.

01:06:18.650 --> 01:06:20.653 align:middle line:84%
"These are the worst
jeans I've ever worn.

01:06:20.653 --> 01:06:22.070 align:middle line:84%
They are too tight
here and there.

01:06:22.070 --> 01:06:25.617 align:middle line:84%
I'm going to return them and try
a 30, but I'm not optimistic.

01:06:25.617 --> 01:06:26.950 align:middle line:90%
I'm going to stick with Levi's."

01:06:26.950 --> 01:06:29.970 align:middle line:90%
Phew, OK.

01:06:29.970 --> 01:06:31.510 align:middle line:90%
So these read like real reviews.

01:06:31.510 --> 01:06:34.590 align:middle line:84%
So just by taking a few hundred
examples and fine-tuning it,

01:06:34.590 --> 01:06:36.130 align:middle line:84%
it completely
changes the behavior

01:06:36.130 --> 01:06:38.230 align:middle line:84%
that you want for your
particular use case.

01:06:38.230 --> 01:06:39.530 align:middle line:90%
That's the key thing.

01:06:39.530 --> 01:06:41.890 align:middle line:84%
So for me, the
biggest benefit here

01:06:41.890 --> 01:06:45.130 align:middle line:84%
is that while it took
billions of sentences

01:06:45.130 --> 01:06:47.250 align:middle line:84%
for pre-training
the original LLM

01:06:47.250 --> 01:06:49.330 align:middle line:84%
and then it took tens
of thousands of examples

01:06:49.330 --> 01:06:54.850 align:middle line:84%
to do supervised fine-tuning
and RLHF and so on and so forth,

01:06:54.850 --> 01:06:57.590 align:middle line:84%
to make it work for your
narrow business use case,

01:06:57.590 --> 01:07:00.430 align:middle line:84%
you only had to spend a
couple of hundred examples.

01:07:00.430 --> 01:07:01.860 align:middle line:90%
That's it.

01:07:01.860 --> 01:07:02.600 align:middle line:90%
It's amazing.

01:07:02.600 --> 01:07:05.980 align:middle line:84%
Imagine that if you had to
collect like 30,000 examples

01:07:05.980 --> 01:07:06.503 align:middle line:90%
to make it.

01:07:06.503 --> 01:07:07.920 align:middle line:84%
Nobody's going to
do these things.

01:07:07.920 --> 01:07:09.540 align:middle line:90%
It's too much work.

01:07:09.540 --> 01:07:12.220 align:middle line:84%
But a couple of
hundred, anybody can do.

01:07:12.220 --> 01:07:14.860 align:middle line:84%
That's why it's so powerful
to fine tune these things.

01:07:14.860 --> 01:07:16.780 align:middle line:90%
Yeah?

01:07:16.780 --> 01:07:19.192 align:middle line:84%
AUDIENCE: You talked
about being able to,

01:07:19.192 --> 01:07:23.060 align:middle line:84%
in industries where you don't
want to put some of this stuff

01:07:23.060 --> 01:07:26.380 align:middle line:84%
on the internet, downloading
the pre-trained model

01:07:26.380 --> 01:07:29.735 align:middle line:84%
and being able to
do this on your own.

01:07:29.735 --> 01:07:31.860 align:middle line:84%
Talking about computer
power, some of the computers

01:07:31.860 --> 01:07:35.380 align:middle line:84%
we have now, the GPUs, I don't
know how powerful they are.

01:07:35.380 --> 01:07:38.340 align:middle line:84%
Are you able to do some of
these very small use cases

01:07:38.340 --> 01:07:40.460 align:middle line:90%
on those types of devices?

01:07:40.460 --> 01:07:42.215 align:middle line:84%
RAMA RAMAKRISHNAN:
Perfect question, Mike.

01:07:42.215 --> 01:07:43.340 align:middle line:90%
We're going to get to that.

01:07:43.340 --> 01:07:45.623 align:middle line:84%
Because the short
answer, it's hard.

01:07:45.623 --> 01:07:47.040 align:middle line:84%
Yeah, just a few
hundred examples.

01:07:47.040 --> 01:07:49.082 align:middle line:84%
But actually trying to
fine-tune these big models

01:07:49.082 --> 01:07:51.860 align:middle line:84%
on consumer-grade hardware
is actually not easy.

01:07:51.860 --> 01:07:54.520 align:middle line:84%
So you have to make certain
tricks and simplifications,

01:07:54.520 --> 01:07:56.340 align:middle line:90%
which is the next topic.

01:07:56.340 --> 01:07:57.740 align:middle line:90%
Yeah?

01:07:57.740 --> 01:08:00.070 align:middle line:84%
AUDIENCE: Is fine-tuning
always supervised?

01:08:00.070 --> 01:08:01.250 align:middle line:90%
You need those pairs?

01:08:01.250 --> 01:08:03.870 align:middle line:84%
Or could you do
it if the company

01:08:03.870 --> 01:08:05.533 align:middle line:90%
has less structured data?

01:08:05.533 --> 01:08:06.450 align:middle line:90%
RAMA RAMAKRISHNAN: No.

01:08:06.450 --> 01:08:08.150 align:middle line:84%
The thing is, it
depends on whether you

01:08:08.150 --> 01:08:11.765 align:middle line:84%
want to make it generally smart
about the company's business

01:08:11.765 --> 01:08:14.390 align:middle line:84%
details, in which case, you can
just take a whole bunch of text

01:08:14.390 --> 01:08:16.098 align:middle line:84%
and just do a next-word
prediction on it.

01:08:16.098 --> 01:08:18.330 align:middle line:84%
It's going to get smarter
about generally things.

01:08:18.330 --> 01:08:20.538 align:middle line:84%
But it doesn't mean it's
going to specifically follow

01:08:20.538 --> 01:08:23.109 align:middle line:84%
your instructions on your
particular business problem.

01:08:23.109 --> 01:08:24.890 align:middle line:84%
So if you want it to
follow instructions,

01:08:24.890 --> 01:08:27.270 align:middle line:90%
you need supervision.

01:08:27.270 --> 01:08:28.510 align:middle line:90%
OK.

01:08:28.510 --> 01:08:30.710 align:middle line:84%
All right, these
are great reviews.

01:08:30.710 --> 01:08:34.590 align:middle line:84%
So for small LLMs like GPT-2,
fine-tuning isn't difficult,

01:08:34.590 --> 01:08:35.649 align:middle line:90%
to go to your question.

01:08:35.649 --> 01:08:37.729 align:middle line:84%
You can actually do
this with small models.

01:08:37.729 --> 01:08:40.109 align:middle line:84%
So for example, Google
had this thing called

01:08:40.109 --> 01:08:41.410 align:middle line:90%
Gemma, which came out recently.

01:08:41.410 --> 01:08:43.270 align:middle line:84%
It's a small model, like 2
billion parameters or something,

01:08:43.270 --> 01:08:43.853 align:middle line:90%
if I remember.

01:08:43.853 --> 01:08:44.970 align:middle line:90%
It's the smallest one.

01:08:44.970 --> 01:08:49.229 align:middle line:90%


01:08:49.229 --> 01:08:50.630 align:middle line:90%
Thank you.

01:08:50.630 --> 01:08:52.950 align:middle line:84%
Those things will
typically fit into one GPU

01:08:52.950 --> 01:08:54.010 align:middle line:90%
and you can fine tune it.

01:08:54.010 --> 01:08:55.967 align:middle line:84%
You still need GPUs,
just to be clear.

01:08:55.967 --> 01:08:57.550 align:middle line:84%
They will actually
fit into one thing.

01:08:57.550 --> 01:09:00.250 align:middle line:84%
But if you want to use a
larger model, it won't fit.

01:09:00.250 --> 01:09:02.790 align:middle line:84%
So to make this work, you
have to do other things.

01:09:02.790 --> 01:09:04.810 align:middle line:84%
And that's what we're
going to talk about now.

01:09:04.810 --> 01:09:08.729 align:middle line:84%
But there's a family of
models called Llama, Llama-2.

01:09:08.729 --> 01:09:11.370 align:middle line:90%
These are open source LLMs.

01:09:11.370 --> 01:09:13.729 align:middle line:84%
And they are widely
used for fine-tuning.

01:09:13.729 --> 01:09:15.729 align:middle line:84%
Because you can just
download the model and just

01:09:15.729 --> 01:09:16.950 align:middle line:90%
do whatever you want with it.

01:09:16.950 --> 01:09:18.950 align:middle line:90%
It's open.

01:09:18.950 --> 01:09:20.729 align:middle line:84%
It's not strictly
open because there

01:09:20.729 --> 01:09:23.670 align:middle line:84%
are some footnote considerations
you have to worry about.

01:09:23.670 --> 01:09:27.350 align:middle line:84%
But for most purposes, it's
open enough, in my opinion.

01:09:27.350 --> 01:09:30.890 align:middle line:84%
And so let's see how hard
it is to build the biggest

01:09:30.890 --> 01:09:33.670 align:middle line:84%
model in this family, which
is the Llama-2 model with 70

01:09:33.670 --> 01:09:35.310 align:middle line:90%
billion parameters, OK?

01:09:35.310 --> 01:09:36.729 align:middle line:90%
70 billion parameters.

01:09:36.729 --> 01:09:39.689 align:middle line:84%
So first of all, the
model is gigantic.

01:09:39.689 --> 01:09:41.890 align:middle line:84%
So 70 billion parameters,
each parameter

01:09:41.890 --> 01:09:44.649 align:middle line:84%
is, let's say, we store it
in 2 bytes per parameter.

01:09:44.649 --> 01:09:49.330 align:middle line:84%
And then each of these
parameters actually, we

01:09:49.330 --> 01:09:51.250 align:middle line:84%
will need a multiplier
on each parameter

01:09:51.250 --> 01:09:55.786 align:middle line:84%
to store various details about
how the optimization is done.

01:09:55.786 --> 01:09:57.520 align:middle line:84%
We won't get into
the details here.

01:09:57.520 --> 01:10:01.580 align:middle line:84%
The one thing I do want to
point out is that this 3 to 4

01:10:01.580 --> 01:10:04.740 align:middle line:90%
should really be 1 to 6.

01:10:04.740 --> 01:10:08.160 align:middle line:84%
So I didn't have a chance
to change it this morning.

01:10:08.160 --> 01:10:11.440 align:middle line:84%
But the point is that it's
going to be a huge model.

01:10:11.440 --> 01:10:12.940 align:middle line:84%
So even with this
number, it's going

01:10:12.940 --> 01:10:16.140 align:middle line:84%
to be like 420 to 560 gigabytes,
just to hold the model in memory

01:10:16.140 --> 01:10:17.340 align:middle line:90%
and manipulate it.

01:10:17.340 --> 01:10:20.220 align:middle line:84%
And so if you use a
GPU like an A100 GPU

01:10:20.220 --> 01:10:23.740 align:middle line:84%
or an H100 GPU, which
are all NVIDIA GPUs,

01:10:23.740 --> 01:10:26.980 align:middle line:84%
each of these things typically
has 80 gigabytes of RAM memory.

01:10:26.980 --> 01:10:30.260 align:middle line:84%
So we need between 6 and 7
to accommodate this thing,

01:10:30.260 --> 01:10:32.600 align:middle line:84%
6 to 7 GPUs just to
accommodate this thing.

01:10:32.600 --> 01:10:33.860 align:middle line:90%
So that's the first problem.

01:10:33.860 --> 01:10:34.900 align:middle line:90%
The model is big.

01:10:34.900 --> 01:10:37.420 align:middle line:84%
Just to hold it and work with
it, you need lots of GPUs.

01:10:37.420 --> 01:10:38.860 align:middle line:90%
The second problem--

01:10:38.860 --> 01:10:43.300 align:middle line:84%
Llama-2 was trained on 2
trillion tokens of text,

01:10:43.300 --> 01:10:45.260 align:middle line:90%
2 trillion tokens of text.

01:10:45.260 --> 01:10:50.580 align:middle line:84%
So these GPUs can process about
400 tokens per GPU per second.

01:10:50.580 --> 01:10:54.496 align:middle line:84%
By process, I mean the forward
pass through the network.

01:10:54.496 --> 01:10:57.610 align:middle line:84%
And so if you actually use
7 GPUs with all this thing,

01:10:57.610 --> 01:11:00.597 align:middle line:84%
it's going to take
you 8,000 days.

01:11:00.597 --> 01:11:02.430 align:middle line:84%
Let's say we want to
do it in about a month,

01:11:02.430 --> 01:11:09.790 align:middle line:84%
you need 2,048 GPUs at this
cost of $2.50 per GPU per hour.

01:11:09.790 --> 01:11:12.270 align:middle line:90%
This will cost you $4 million.

01:11:12.270 --> 01:11:14.150 align:middle line:84%
And we would expect
the actual cost

01:11:14.150 --> 01:11:15.970 align:middle line:84%
to be a lot higher than this
because this is very optimistic.

01:11:15.970 --> 01:11:17.080 align:middle line:84%
It assumes you just
do one pass-through

01:11:17.080 --> 01:11:17.913 align:middle line:90%
and you're all done.

01:11:17.913 --> 01:11:19.837 align:middle line:84%
In general, you'll
make some mistakes,

01:11:19.837 --> 01:11:22.170 align:middle line:84%
you have to do a bunch of
times, and so on and so forth.

01:11:22.170 --> 01:11:23.890 align:middle line:84%
So this is an overly
optimistic estimate,

01:11:23.890 --> 01:11:25.328 align:middle line:90%
and that is $4 million.

01:11:25.328 --> 01:11:27.870 align:middle line:84%
So you need lots of GPUs and
you need to spend a lot of money

01:11:27.870 --> 01:11:28.630 align:middle line:90%
for it.

01:11:28.630 --> 01:11:31.910 align:middle line:84%
Now, what can we do
with fewer resources?

01:11:31.910 --> 01:11:35.072 align:middle line:84%
First of all, you need to
reduce the size of the dataset.

01:11:35.072 --> 01:11:37.530 align:middle line:84%
The second thing is you want
to reduce the memory required.

01:11:37.530 --> 01:11:40.510 align:middle line:84%
So we can ideally do it in many
fewer GPUs, hopefully even one

01:11:40.510 --> 01:11:41.960 align:middle line:90%
GPU, literally on Colab.

01:11:41.960 --> 01:11:44.950 align:middle line:90%


01:11:44.950 --> 01:11:46.750 align:middle line:84%
Now we have good news
on the data front.

01:11:46.750 --> 01:11:49.990 align:middle line:84%
Because as I mentioned earlier,
while it takes a lot of data

01:11:49.990 --> 01:11:52.290 align:middle line:84%
to build these models,
to fine-tune them

01:11:52.290 --> 01:11:54.550 align:middle line:84%
for your specific
data, your use case,

01:11:54.550 --> 01:11:56.990 align:middle line:84%
you may just need a
few hundred examples.

01:11:56.990 --> 01:11:58.130 align:middle line:90%
It's no problem at all.

01:11:58.130 --> 01:12:00.950 align:middle line:84%
So the data for fine-tuning
is actually not a problem.

01:12:00.950 --> 01:12:04.377 align:middle line:84%
Only for building it in the
first place, it's a problem.

01:12:04.377 --> 01:12:06.710 align:middle line:84%
In fact, there's this famous
Alpaca fine-tuning dataset.

01:12:06.710 --> 01:12:08.850 align:middle line:84%
It has 50,000
instruction-answer pairs.

01:12:08.850 --> 01:12:13.230 align:middle line:84%
And so for that, way less
than the 2 trillion tokens.

01:12:13.230 --> 01:12:15.890 align:middle line:84%
And that can actually be
done in about 20 hours.

01:12:15.890 --> 01:12:19.203 align:middle line:84%
You can fine-tune a 50,000
example, fine-tuning dataset,

01:12:19.203 --> 01:12:20.870 align:middle line:84%
you can fine tune it
with just 20 hours.

01:12:20.870 --> 01:12:21.810 align:middle line:90%
OK?

01:12:21.810 --> 01:12:23.282 align:middle line:90%
Tomaso?

01:12:23.282 --> 01:12:27.090 align:middle line:84%
AUDIENCE: Could Microsoft's
1-bit model drastically reduces

01:12:27.090 --> 01:12:28.823 align:middle line:90%
the amount of compute we need?

01:12:28.823 --> 01:12:31.490 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, there's
a whole bunch of approximations

01:12:31.490 --> 01:12:35.530 align:middle line:84%
and simplifications to make all
these things fit into smaller

01:12:35.530 --> 01:12:36.990 align:middle line:90%
GPUs and so on and so forth.

01:12:36.990 --> 01:12:39.290 align:middle line:90%
And that's one of them.

01:12:39.290 --> 01:12:41.850 align:middle line:84%
So the short answer is, yeah,
there are many possibilities.

01:12:41.850 --> 01:12:43.642 align:middle line:84%
And we have to very
carefully look at them.

01:12:43.642 --> 01:12:45.450 align:middle line:84%
Because every one of
these simplifications,

01:12:45.450 --> 01:12:47.450 align:middle line:84%
it will cost you something
in terms of accuracy

01:12:47.450 --> 01:12:49.990 align:middle line:84%
and the ability of the model
to do what it needs to do.

01:12:49.990 --> 01:12:52.540 align:middle line:84%
So there's always a trade-off
you have to worry about.

01:12:52.540 --> 01:12:54.380 align:middle line:84%
So for folks who are
interested, there

01:12:54.380 --> 01:12:57.160 align:middle line:84%
is this whole field called
quantization, LLM quantization.

01:12:57.160 --> 01:12:57.920 align:middle line:90%
Google it.

01:12:57.920 --> 01:13:01.833 align:middle line:84%
And that's an entry point
into that whole area.

01:13:01.833 --> 01:13:04.500 align:middle line:84%
Now, how do we reduce the memory
required so that we can process

01:13:04.500 --> 01:13:06.160 align:middle line:90%
the data using fewer GPUs--

01:13:06.160 --> 01:13:08.260 align:middle line:90%
ideally, just one GPU on Colab?

01:13:08.260 --> 01:13:10.860 align:middle line:84%
So if you look at what
actually consumes memory,

01:13:10.860 --> 01:13:12.800 align:middle line:84%
you have all these
model parameters.

01:13:12.800 --> 01:13:15.860 align:middle line:84%
Let's say, 70 billion
parameters times 2 bytes each,

01:13:15.860 --> 01:13:17.220 align:middle line:90%
140 gigabytes.

01:13:17.220 --> 01:13:20.180 align:middle line:84%
Gradient computations is another
140 to hold the gradient.

01:13:20.180 --> 01:13:22.120 align:middle line:84%
And then the
optimizer state is 2x.

01:13:22.120 --> 01:13:25.820 align:middle line:84%
And as I mentioned earlier,
it could be between 1 to 6x

01:13:25.820 --> 01:13:27.353 align:middle line:90%
as opposed to 3 to 4x.

01:13:27.353 --> 01:13:29.520 align:middle line:84%
But we'll just go with these
numbers for the moment.

01:13:29.520 --> 01:13:32.320 align:middle line:84%
And so the total
is 560 gigabytes,

01:13:32.320 --> 01:13:34.420 align:middle line:84%
if you just naively
want to use it.

01:13:34.420 --> 01:13:36.720 align:middle line:84%
So it turns out you can't
do anything about that.

01:13:36.720 --> 01:13:38.340 align:middle line:90%
It is just for 140.

01:13:38.340 --> 01:13:41.240 align:middle line:84%
But by using a trick called
gradient checkpointing,

01:13:41.240 --> 01:13:44.740 align:middle line:84%
this whole thing can actually
be squashed close to 0.

01:13:44.740 --> 01:13:47.600 align:middle line:84%
Basically, you say, hey, I
don't mind it running longer,

01:13:47.600 --> 01:13:49.320 align:middle line:84%
but I don't want to
use as much memory.

01:13:49.320 --> 01:13:51.460 align:middle line:84%
And that trick is called
gradient checkpointing.

01:13:51.460 --> 01:13:53.320 align:middle line:84%
We won't go into
technical details.

01:13:53.320 --> 01:13:54.620 align:middle line:90%
That can go to 0.

01:13:54.620 --> 01:13:56.820 align:middle line:84%
But then this thing here,
the optimizer state,

01:13:56.820 --> 01:14:00.340 align:middle line:84%
turns out even this can be
squashed very close to 0.

01:14:00.340 --> 01:14:02.040 align:middle line:84%
And that actually
was a breakthrough

01:14:02.040 --> 01:14:04.640 align:middle line:90%
from maybe a year ago.

01:14:04.640 --> 01:14:07.540 align:middle line:84%
And so to do that, what
we're going to do is to say,

01:14:07.540 --> 01:14:08.713 align:middle line:90%
look you know what?

01:14:08.713 --> 01:14:10.380 align:middle line:84%
There are a whole
bunch of weights here,

01:14:10.380 --> 01:14:13.440 align:middle line:84%
but we're only going
to take those matrices

01:14:13.440 --> 01:14:15.142 align:middle line:90%
inside each attention layer.

01:14:15.142 --> 01:14:17.100 align:middle line:84%
And we're going to only
look at those matrices.

01:14:17.100 --> 01:14:19.840 align:middle line:84%
We're going to freeze
everything else.

01:14:19.840 --> 01:14:23.100 align:middle line:84%
So we're going to take only
a small set of parameters,

01:14:23.100 --> 01:14:25.058 align:middle line:84%
unfreeze them, and
update them, and see

01:14:25.058 --> 01:14:26.600 align:middle line:84%
if it's any good,
if it actually gets

01:14:26.600 --> 01:14:28.280 align:middle line:84%
the job done,
instead of unfreezing

01:14:28.280 --> 01:14:30.200 align:middle line:90%
everything and updating them.

01:14:30.200 --> 01:14:32.060 align:middle line:84%
And so if you look
at the weight matrix,

01:14:32.060 --> 01:14:35.700 align:middle line:84%
let's say the key AK
weight matrix in Llama-2,

01:14:35.700 --> 01:14:38.720 align:middle line:84%
this is a roughly
8,000 by 8,000 matrix,

01:14:38.720 --> 01:14:41.280 align:middle line:84%
which means that there are 64
million parameters inside each

01:14:41.280 --> 01:14:42.900 align:middle line:90%
of these matrices, 64 million.

01:14:42.900 --> 01:14:45.920 align:middle line:90%


01:14:45.920 --> 01:14:49.873 align:middle line:84%
If you imagine this matrix
AK here, and let's say,

01:14:49.873 --> 01:14:52.290 align:middle line:84%
thought experiment, you do the
fine-tuning and the numbers

01:14:52.290 --> 01:14:55.130 align:middle line:84%
have changed as a
result of fine-tuning.

01:14:55.130 --> 01:14:57.810 align:middle line:84%
Then you can imagine that
the resulting matrix is just

01:14:57.810 --> 01:15:02.650 align:middle line:84%
the original matrix you
had plus just the changes.

01:15:02.650 --> 01:15:03.990 align:middle line:90%
The original plus the changes.

01:15:03.990 --> 01:15:06.530 align:middle line:84%
And we call the
changes delta AK.

01:15:06.530 --> 01:15:09.950 align:middle line:84%
And of course, in general, this
change is also going to be a 64

01:15:09.950 --> 01:15:12.470 align:middle line:90%
million matrix, 8,000 by 8,000.

01:15:12.470 --> 01:15:15.770 align:middle line:84%
So the question is, can we make
this change matrix smaller?

01:15:15.770 --> 01:15:18.530 align:middle line:84%
And to make it smaller,
it seems reasonable

01:15:18.530 --> 01:15:21.210 align:middle line:84%
because a fine-tune will
only make small changes

01:15:21.210 --> 01:15:22.270 align:middle line:90%
to just a few weights.

01:15:22.270 --> 01:15:23.810 align:middle line:84%
It's not going to
change everything.

01:15:23.810 --> 01:15:25.590 align:middle line:84%
By definition, a couple
of hundred examples,

01:15:25.590 --> 01:15:27.590 align:middle line:84%
you do some fine-tuning,
hopefully a few weights

01:15:27.590 --> 01:15:28.950 align:middle line:90%
are going to change.

01:15:28.950 --> 01:15:30.890 align:middle line:84%
And maybe they won't
change a whole lot.

01:15:30.890 --> 01:15:33.010 align:middle line:84%
So the key insight
here is that maybe we

01:15:33.010 --> 01:15:35.810 align:middle line:84%
can force this change
matrix to be kind of simple

01:15:35.810 --> 01:15:37.890 align:middle line:90%
and get the job done.

01:15:37.890 --> 01:15:39.452 align:middle line:90%
And it turns out you can.

01:15:39.452 --> 01:15:41.410 align:middle line:84%
And what you do is you
can think of this matrix

01:15:41.410 --> 01:15:46.800 align:middle line:84%
as really coming from two
thin, skinny matrices, which,

01:15:46.800 --> 01:15:49.860 align:middle line:84%
if you multiply them, gets
you the original matrix.

01:15:49.860 --> 01:15:52.360 align:middle line:84%
And I'm not going to get into
the mathematical details here.

01:15:52.360 --> 01:15:55.140 align:middle line:84%
This is called a
low-rank approximation.

01:15:55.140 --> 01:15:56.900 align:middle line:84%
But the point here
is that you can

01:15:56.900 --> 01:15:59.700 align:middle line:90%
take two very small matrices.

01:15:59.700 --> 01:16:01.440 align:middle line:84%
And if you multiply
them the right way,

01:16:01.440 --> 01:16:03.340 align:middle line:84%
you actually can recover
the original matrix.

01:16:03.340 --> 01:16:05.420 align:middle line:84%
You can approximate
the original matrix.

01:16:05.420 --> 01:16:07.280 align:middle line:84%
And this matrix,
as it turns out,

01:16:07.280 --> 01:16:10.700 align:middle line:84%
these two matrices are much
smaller because each one is just

01:16:10.700 --> 01:16:13.060 align:middle line:90%
8,000 times 2, 16,000.

01:16:13.060 --> 01:16:17.920 align:middle line:84%
And so this thing has
just 16,192 parameters,

01:16:17.920 --> 01:16:20.600 align:middle line:84%
which is 0.02% of the
original 64 million.

01:16:20.600 --> 01:16:23.460 align:middle line:90%


01:16:23.460 --> 01:16:26.400 align:middle line:84%
So this thing is called
low-rank adaptation, or LORA.

01:16:26.400 --> 01:16:29.780 align:middle line:84%
And it's incredibly widely
used in the industry.

01:16:29.780 --> 01:16:32.780 align:middle line:84%
And so what we do is we
freeze all the parameters,

01:16:32.780 --> 01:16:35.760 align:middle line:84%
we initialize all these
change matrices to 0.

01:16:35.760 --> 01:16:39.720 align:middle line:84%
And then we update just those
two skinny matrices right here,

01:16:39.720 --> 01:16:43.458 align:middle line:84%
here, we update only those
matrices using gradient descent.

01:16:43.458 --> 01:16:45.000 align:middle line:84%
And when you do
that, everything will

01:16:45.000 --> 01:16:47.400 align:middle line:84%
fit into memory, which means
that the whole thing will

01:16:47.400 --> 01:16:47.900 align:middle line:90%
fit in.

01:16:47.900 --> 01:16:50.160 align:middle line:84%
And you can just use two
GPUs and get the job done.

01:16:50.160 --> 01:16:53.440 align:middle line:84%
And if you actually use
Llama's, the smaller models,

01:16:53.440 --> 01:16:55.920 align:middle line:84%
like 7 billion, 13 billion,
it can be fine-tuned

01:16:55.920 --> 01:17:00.160 align:middle line:84%
comfortably on a single
GPU, on a single Colab GPU.

01:17:00.160 --> 01:17:04.560 align:middle line:84%
All right, 9:54,
time does not permit.

01:17:04.560 --> 01:17:07.320 align:middle line:84%
So I have a Colab on how
to do the fine-tuning using

01:17:07.320 --> 01:17:08.320 align:middle line:90%
this technique.

01:17:08.320 --> 01:17:12.000 align:middle line:84%
I will do a video walkthrough
tomorrow or day after.

01:17:12.000 --> 01:17:12.960 align:middle line:90%
And I'm done.

01:17:12.960 --> 01:17:13.620 align:middle line:90%
Thanks, folks.

01:17:13.620 --> 01:17:14.870 align:middle line:90%
Have a good rest of your week.

01:17:14.870 --> 01:17:16.160 align:middle line:90%
[APPLAUSE]

01:17:16.160 --> 01:17:17.990 align:middle line:90%
Thank you.

01:17:17.990 --> 01:17:42.000 align:middle line:90%