WEBVTT

00:00:00.000 --> 00:00:02.982 align:middle line:84%
[SQUEAKING]
[RUSTLING] [CLICKING]

00:00:02.982 --> 00:00:16.920 align:middle line:90%


00:00:16.920 --> 00:00:20.000 align:middle line:84%
RAMA RAMAKRISHNAN: OK, so we'll
continue with transformers

00:00:20.000 --> 00:00:21.720 align:middle line:90%
today, part two.

00:00:21.720 --> 00:00:23.425 align:middle line:84%
We are going to do
the second pass.

00:00:23.425 --> 00:00:25.800 align:middle line:84%
This is going to be a deeper
pass through the transformer

00:00:25.800 --> 00:00:26.960 align:middle line:90%
stack.

00:00:26.960 --> 00:00:29.360 align:middle line:84%
And I think maybe
the next 30 minutes,

00:00:29.360 --> 00:00:31.400 align:middle line:84%
it's potentially the
most demanding 30 minutes

00:00:31.400 --> 00:00:33.640 align:middle line:90%
of the entire course, OK?

00:00:33.640 --> 00:00:36.560 align:middle line:84%
With that motivational
speech, let's get going.

00:00:36.560 --> 00:00:38.493 align:middle line:90%
So a quick review.

00:00:38.493 --> 00:00:39.660 align:middle line:90%
Why do we want transformers?

00:00:39.660 --> 00:00:43.880 align:middle line:84%
Because we want an architecture
that can generate output

00:00:43.880 --> 00:00:47.340 align:middle line:84%
that has the same length
as the input, same length.

00:00:47.340 --> 00:00:48.472 align:middle line:90%
There it is.

00:00:48.472 --> 00:00:50.680 align:middle line:84%
Number two, we want to take
the context into account,

00:00:50.680 --> 00:00:52.472 align:middle line:84%
and we want to take
the order into account.

00:00:52.472 --> 00:00:55.080 align:middle line:84%
And as you saw last time,
the transformer architecture

00:00:55.080 --> 00:00:57.080 align:middle line:84%
delivers on those
three requirements.

00:00:57.080 --> 00:00:59.060 align:middle line:90%
And so just a quick review.

00:00:59.060 --> 00:01:01.660 align:middle line:84%
If you have a phrase like,
"the train left the station,"

00:01:01.660 --> 00:01:04.920 align:middle line:84%
we have all these little arrows
which stand for the standalone

00:01:04.920 --> 00:01:07.360 align:middle line:90%
or uncontextual embeddings.

00:01:07.360 --> 00:01:09.120 align:middle line:84%
And then-- sometimes
this works, so I'm

00:01:09.120 --> 00:01:10.760 align:middle line:84%
going to put it
close to me here.

00:01:10.760 --> 00:01:14.440 align:middle line:90%


00:01:14.440 --> 00:01:17.800 align:middle line:84%
So if you're here, if we
start with either standalone

00:01:17.800 --> 00:01:20.448 align:middle line:84%
embeddings, i.e. uncontextual
embeddings, which

00:01:20.448 --> 00:01:22.740 align:middle line:84%
have been pre-trained or
random, doesn't really matter,

00:01:22.740 --> 00:01:26.407 align:middle line:84%
if you look at the Colab
we did the other day,

00:01:26.407 --> 00:01:28.240 align:middle line:84%
we actually just started
with random weights

00:01:28.240 --> 00:01:32.240 align:middle line:84%
for the embeddings, and then
we add positional embeddings

00:01:32.240 --> 00:01:33.000 align:middle line:90%
to them.

00:01:33.000 --> 00:01:38.060 align:middle line:84%
And so each embedding, each word
here, we take its standalone,

00:01:38.060 --> 00:01:40.240 align:middle line:84%
we take its positional
embedding, we literally just

00:01:40.240 --> 00:01:42.160 align:middle line:90%
add them up element by element.

00:01:42.160 --> 00:01:43.960 align:middle line:84%
Then we get a total
embedding, and that's

00:01:43.960 --> 00:01:47.400 align:middle line:84%
called the positional
embedding of each word.

00:01:47.400 --> 00:01:50.520 align:middle line:84%
And then that's what we have,
positional input embeddings.

00:01:50.520 --> 00:01:53.900 align:middle line:84%
So this whole thing goes into
this transformer encoder stack,

00:01:53.900 --> 00:01:56.980 align:middle line:84%
and what pops out the other
end is contextual embeddings

00:01:56.980 --> 00:01:58.680 align:middle line:90%
So that's the overall flow.

00:01:58.680 --> 00:02:03.480 align:middle line:84%
Now we applied this,
the transformer

00:02:03.480 --> 00:02:06.580 align:middle line:84%
stack, to the word-to-slot
classification problem,

00:02:06.580 --> 00:02:09.240 align:middle line:84%
where we basically took every
incoming natural language

00:02:09.240 --> 00:02:11.800 align:middle line:84%
query that comes
in, we calculate

00:02:11.800 --> 00:02:13.560 align:middle line:84%
its positional
embeddings, and then

00:02:13.560 --> 00:02:16.080 align:middle line:84%
we run it through the
transformer stack,

00:02:16.080 --> 00:02:17.980 align:middle line:84%
and then we get
contextual embeddings.

00:02:17.980 --> 00:02:21.302 align:middle line:84%
And then at this point, since
each word that comes out,

00:02:21.302 --> 00:02:22.760 align:middle line:84%
each embedding that
comes out needs

00:02:22.760 --> 00:02:25.520 align:middle line:84%
to be classified into
one of 125 possibilities,

00:02:25.520 --> 00:02:28.400 align:middle line:84%
we run it through a
ReLU, when we attach

00:02:28.400 --> 00:02:30.680 align:middle line:90%
a softmax to each embedding.

00:02:30.680 --> 00:02:34.560 align:middle line:84%
This is basically what
we did last class.

00:02:34.560 --> 00:02:37.840 align:middle line:84%
So this is the
transformer encoder.

00:02:37.840 --> 00:02:42.180 align:middle line:84%
Actually, any questions on
this before I continue on?

00:02:42.180 --> 00:02:48.520 align:middle line:90%


00:02:48.520 --> 00:02:50.200 align:middle line:84%
AUDIENCE: I was
wondering why, when, how

00:02:50.200 --> 00:02:52.520 align:middle line:84%
do you decide where to
add more self-attention

00:02:52.520 --> 00:02:55.180 align:middle line:84%
and where you add the
transformer layers?

00:02:55.180 --> 00:02:59.620 align:middle line:84%
You mentioned that
ChatGPT has 96 of them.

00:02:59.620 --> 00:03:04.220 align:middle line:84%
RAMA RAMAKRISHNAN: So GPT3
has 96 transformer blocks.

00:03:04.220 --> 00:03:05.680 align:middle line:90%
Each one is a block.

00:03:05.680 --> 00:03:07.840 align:middle line:84%
So I think the question
goes to, do you

00:03:07.840 --> 00:03:10.560 align:middle line:84%
add more attention heads
within a single block

00:03:10.560 --> 00:03:12.200 align:middle line:90%
or do you add lots of blocks?

00:03:12.200 --> 00:03:14.640 align:middle line:90%
And both are good things to do.

00:03:14.640 --> 00:03:17.720 align:middle line:84%
What increasing the number
of attention heads in a block

00:03:17.720 --> 00:03:20.960 align:middle line:84%
does for you, it allows you
to pick up more patterns

00:03:20.960 --> 00:03:23.600 align:middle line:90%
at that level of abstraction.

00:03:23.600 --> 00:03:27.200 align:middle line:84%
But if you add more blocks, much
like later convolutional filters

00:03:27.200 --> 00:03:30.360 align:middle line:84%
can build on earlier
convolutional filters,

00:03:30.360 --> 00:03:32.500 align:middle line:84%
you're going up the
levels of abstraction.

00:03:32.500 --> 00:03:34.407 align:middle line:84%
So to go to vision,
for instance,

00:03:34.407 --> 00:03:36.740 align:middle line:84%
you have the notion of lines
and so on in the beginning,

00:03:36.740 --> 00:03:39.800 align:middle line:84%
and then you have a notion of
edges, which are two lines.

00:03:39.800 --> 00:03:43.420 align:middle line:84%
Then you have nose, eyes,
face, and so on and so forth.

00:03:43.420 --> 00:03:45.060 align:middle line:90%
So both are worth doing.

00:03:45.060 --> 00:03:46.680 align:middle line:84%
So typically that's
what you typically

00:03:46.680 --> 00:03:51.920 align:middle line:84%
find, that people typically have
maybe a dozen heads or five,

00:03:51.920 --> 00:03:53.260 align:middle line:90%
six, a dozen heads.

00:03:53.260 --> 00:03:54.840 align:middle line:84%
We'll see examples
of how many heads

00:03:54.840 --> 00:03:56.760 align:middle line:84%
in a couple of architectures
later on today.

00:03:56.760 --> 00:04:00.678 align:middle line:84%
And the more you go
up, the more capable

00:04:00.678 --> 00:04:02.720 align:middle line:84%
the model becomes as long
as you have enough data

00:04:02.720 --> 00:04:04.480 align:middle line:90%
to train it well.

00:04:04.480 --> 00:04:06.960 align:middle line:84%
So the perennial question
of, do we have enough data

00:04:06.960 --> 00:04:08.700 align:middle line:90%
to train this large model--

00:04:08.700 --> 00:04:10.418 align:middle line:84%
because if you don't
have enough data,

00:04:10.418 --> 00:04:12.460 align:middle line:84%
we might run into overfitting
problems and so on.

00:04:12.460 --> 00:04:15.700 align:middle line:90%
That's always the tradeoff.

00:04:15.700 --> 00:04:17.880 align:middle line:84%
So here, I just want to
quickly switch to the Colab

00:04:17.880 --> 00:04:20.140 align:middle line:84%
because we didn't have
a chance to finish it.

00:04:20.140 --> 00:04:23.320 align:middle line:84%
I'm not going to run it because
it's going to take some time.

00:04:23.320 --> 00:04:25.390 align:middle line:90%
So where we left off last time--

00:04:25.390 --> 00:04:28.040 align:middle line:90%


00:04:28.040 --> 00:04:31.960 align:middle line:84%
so here we basically
took this architecture

00:04:31.960 --> 00:04:34.560 align:middle line:84%
that we just saw on the
slide, and then we essentially

00:04:34.560 --> 00:04:35.860 align:middle line:90%
wrote it as a Keras model.

00:04:35.860 --> 00:04:37.340 align:middle line:84%
And I went through this
model in the last class,

00:04:37.340 --> 00:04:39.440 align:middle line:84%
so I'm not going to go
through it all over again.

00:04:39.440 --> 00:04:42.640 align:middle line:84%
What we did not do last
class was to actually run it.

00:04:42.640 --> 00:04:46.560 align:middle line:84%
And so if you
actually run it, you

00:04:46.560 --> 00:04:49.580 align:middle line:84%
can just run it for 10 epochs,
just like we normally do.

00:04:49.580 --> 00:04:51.888 align:middle line:84%
Give it data, give
it a bunch of epochs,

00:04:51.888 --> 00:04:53.180 align:middle line:90%
choose a particular batch size.

00:04:53.180 --> 00:04:54.880 align:middle line:90%
I just arbitrarily chose 64.

00:04:54.880 --> 00:04:56.320 align:middle line:84%
You run it for 10
epochs, and then

00:04:56.320 --> 00:04:58.140 align:middle line:90%
you evaluate it on the test set.

00:04:58.140 --> 00:05:01.680 align:middle line:84%
You get a 99% accuracy
on this problem.

00:05:01.680 --> 00:05:03.960 align:middle line:84%
One transformer
stack, that's it--

00:05:03.960 --> 00:05:05.420 align:middle line:90%
one block, rather, one block.

00:05:05.420 --> 00:05:06.240 align:middle line:90%
That's it.

00:05:06.240 --> 00:05:09.600 align:middle line:84%
And of course, here, there is a
little trickiness going on here

00:05:09.600 --> 00:05:13.160 align:middle line:84%
because a naive model can
literally say every word that

00:05:13.160 --> 00:05:17.040 align:middle line:84%
comes in is "other," O. And
since the Os are the majority

00:05:17.040 --> 00:05:19.397 align:middle line:84%
of the words, it's
not going to do badly.

00:05:19.397 --> 00:05:21.480 align:middle line:84%
It's like having a
classification problem in which

00:05:21.480 --> 00:05:23.480 align:middle line:90%
one class is very predominant.

00:05:23.480 --> 00:05:25.310 align:middle line:84%
So the naive way
to actually do well

00:05:25.310 --> 00:05:27.560 align:middle line:84%
is to just say every time
something comes in, oh, it's

00:05:27.560 --> 00:05:28.982 align:middle line:90%
that majority class.

00:05:28.982 --> 00:05:29.940 align:middle line:90%
The same thing happens.

00:05:29.940 --> 00:05:31.760 align:middle line:84%
But if you don't
adjust for that,

00:05:31.760 --> 00:05:34.680 align:middle line:84%
it turns out that the
accuracy on the non-O slots,

00:05:34.680 --> 00:05:36.180 align:middle line:84%
which is really
what you care about,

00:05:36.180 --> 00:05:40.400 align:middle line:84%
is actually 93%, which
is actually pretty good.

00:05:40.400 --> 00:05:44.172 align:middle line:84%
And then I had some examples of
lots of fun queries you can do,

00:05:44.172 --> 00:05:46.880 align:middle line:84%
including queries where I try to
break stuff like cheapest flight

00:05:46.880 --> 00:05:50.080 align:middle line:84%
to fly from MIT to Mars and see
what happens, things like that.

00:05:50.080 --> 00:05:52.810 align:middle line:90%
So have fun with it.

00:05:52.810 --> 00:05:54.230 align:middle line:90%
All right, back to PowerPoint.

00:05:54.230 --> 00:05:59.450 align:middle line:90%


00:05:59.450 --> 00:06:00.550 align:middle line:90%
So this is what we had.

00:06:00.550 --> 00:06:03.370 align:middle line:84%
Now what we're going
to do in today's class,

00:06:03.370 --> 00:06:06.970 align:middle line:84%
we are actually going to take
the encoder we built last time

00:06:06.970 --> 00:06:10.195 align:middle line:84%
and introduce three new
complications into it.

00:06:10.195 --> 00:06:12.570 align:middle line:84%
And when we finish introducing
these three complications,

00:06:12.570 --> 00:06:15.210 align:middle line:84%
we will actually have the
actual transformer that

00:06:15.210 --> 00:06:19.770 align:middle line:90%
was invented in the 2017 paper.

00:06:19.770 --> 00:06:21.750 align:middle line:84%
The first tweak is
the hardest tweak,

00:06:21.750 --> 00:06:24.770 align:middle line:84%
so we'll slowly
work our way to it.

00:06:24.770 --> 00:06:27.350 align:middle line:84%
So the thing to remember is,
let's review self-attention.

00:06:27.350 --> 00:06:28.590 align:middle line:90%
What is self-attention?

00:06:28.590 --> 00:06:31.330 align:middle line:84%
You have a bunch of words, and
we further said that for any

00:06:31.330 --> 00:06:33.970 align:middle line:84%
particular word, like "station,"
we want to take its positional

00:06:33.970 --> 00:06:36.070 align:middle line:84%
embedding and then
make it contextual.

00:06:36.070 --> 00:06:39.330 align:middle line:84%
And the way we do that is by
taking each word's embedding

00:06:39.330 --> 00:06:41.530 align:middle line:84%
and then calculating
these dot products

00:06:41.530 --> 00:06:42.995 align:middle line:90%
between all the other words.

00:06:42.995 --> 00:06:44.370 align:middle line:84%
And then since
these dot products

00:06:44.370 --> 00:06:48.130 align:middle line:84%
can be positive or negative, we
want to make them all positive

00:06:48.130 --> 00:06:50.850 align:middle line:84%
and normalize them so that
they nicely add up to 1.

00:06:50.850 --> 00:06:53.090 align:middle line:84%
So we then exponentiate
them and then

00:06:53.090 --> 00:06:57.170 align:middle line:84%
divide by the total, which
is basically softmax.

00:06:57.170 --> 00:07:00.290 align:middle line:84%
And when you do that, you have
nice fractions that add up to 1.

00:07:00.290 --> 00:07:03.130 align:middle line:84%
And then we said, well, the
contextual embedding for W6

00:07:03.130 --> 00:07:06.810 align:middle line:84%
is just all these
weights, S1, S2,

00:07:06.810 --> 00:07:10.150 align:middle line:84%
all the way to S6, multiplied
by the original Ws,

00:07:10.150 --> 00:07:12.770 align:middle line:84%
and then you get the
contextual embedding for W6.

00:07:12.770 --> 00:07:15.570 align:middle line:84%
So this is the basic logic
we covered last time.

00:07:15.570 --> 00:07:20.450 align:middle line:84%
Now it is obviously the case
that we explained it only

00:07:20.450 --> 00:07:23.215 align:middle line:84%
for one word, but we have to
do the same exact operation

00:07:23.215 --> 00:07:24.590 align:middle line:84%
for every one of
the other words,

00:07:24.590 --> 00:07:29.090 align:middle line:84%
too, so that we could calculate
W5 hat, W4 hat, W3 hat, and so

00:07:29.090 --> 00:07:30.280 align:middle line:90%
on and so forth.

00:07:30.280 --> 00:07:32.930 align:middle line:84%
So there's a lot of
computations that are going on,

00:07:32.930 --> 00:07:35.312 align:middle line:84%
and they all look
kind of similar,

00:07:35.312 --> 00:07:37.270 align:middle line:84%
where you've got to do
a bunch of dot products,

00:07:37.270 --> 00:07:40.170 align:middle line:84%
you've got to do some softmaxing
on it and stuff like that.

00:07:40.170 --> 00:07:41.930 align:middle line:84%
So the natural
question is, is there

00:07:41.930 --> 00:07:45.210 align:middle line:84%
a way to organize
it very efficiently?

00:07:45.210 --> 00:07:46.550 align:middle line:90%
And the short answer is yes.

00:07:46.550 --> 00:07:47.967 align:middle line:84%
In fact, if you
could not do that,

00:07:47.967 --> 00:07:50.250 align:middle line:84%
there wouldn't be any
transformer revolution

00:07:50.250 --> 00:07:52.330 align:middle line:84%
because it is that
ability to package it up

00:07:52.330 --> 00:07:54.810 align:middle line:84%
into a very interesting and
efficient operation that

00:07:54.810 --> 00:07:58.830 align:middle line:84%
allows you to put the
whole thing on GPUs.

00:07:58.830 --> 00:08:03.050 align:middle line:84%
So now I'm going to switch
to iPad and give you

00:08:03.050 --> 00:08:06.810 align:middle line:84%
some iPad scribblings of mine,
which were concocted last night

00:08:06.810 --> 00:08:09.770 align:middle line:84%
because I was very unhappy
with the slides that follow.

00:08:09.770 --> 00:08:12.910 align:middle line:90%
So if you're going to do iPad.

00:08:12.910 --> 00:08:16.070 align:middle line:84%
So if it works, you
folks are lucky.

00:08:16.070 --> 00:08:18.320 align:middle line:84%
If it doesn't work, last
year's HODL class is luckier.

00:08:18.320 --> 00:08:21.330 align:middle line:90%


00:08:21.330 --> 00:08:22.950 align:middle line:90%
So let's shift to that.

00:08:22.950 --> 00:08:25.490 align:middle line:90%


00:08:25.490 --> 00:08:26.840 align:middle line:90%
So we're going to go here.

00:08:26.840 --> 00:08:31.210 align:middle line:90%


00:08:31.210 --> 00:08:33.099 align:middle line:84%
So let's assume we have
a simple thing like--

00:08:33.099 --> 00:08:38.708 align:middle line:90%


00:08:38.708 --> 00:08:40.250 align:middle line:84%
instead of, "Train
left the station,"

00:08:40.250 --> 00:08:42.667 align:middle line:84%
which is a long sentence, let's
just say you have a simple

00:08:42.667 --> 00:08:45.210 align:middle line:90%
sentence like, "I love HODL."

00:08:45.210 --> 00:08:46.870 align:middle line:84%
So "I love HODL"
is what you have,

00:08:46.870 --> 00:08:50.530 align:middle line:84%
and then you have these
standalone embeddings, W1, W2,

00:08:50.530 --> 00:08:52.130 align:middle line:90%
W3.

00:08:52.130 --> 00:08:54.930 align:middle line:84%
So it comes into the
self-attention layer.

00:08:54.930 --> 00:08:57.690 align:middle line:84%
And let's assume that
these W1s, Ws-2, W3,

00:08:57.690 --> 00:09:00.345 align:middle line:84%
they are already
positionally encoded.

00:09:00.345 --> 00:09:01.970 align:middle line:84%
We have already added
up the positional

00:09:01.970 --> 00:09:03.010 align:middle line:90%
encoding all that stuff.

00:09:03.010 --> 00:09:03.990 align:middle line:90%
It's all behind us.

00:09:03.990 --> 00:09:06.010 align:middle line:84%
That all happens
outside the transformer.

00:09:06.010 --> 00:09:08.050 align:middle line:90%
So you get it here.

00:09:08.050 --> 00:09:13.670 align:middle line:84%
Now what you do is you actually
make three copies of this thing.

00:09:13.670 --> 00:09:17.170 align:middle line:84%
And let's call this
whole thing as just x.

00:09:17.170 --> 00:09:18.630 align:middle line:90%
I'm just giving it the name x.

00:09:18.630 --> 00:09:21.650 align:middle line:84%
It's a matrix of
these three vectors.

00:09:21.650 --> 00:09:23.750 align:middle line:84%
And so the first
copy goes up here.

00:09:23.750 --> 00:09:25.110 align:middle line:90%
The second copy goes straight.

00:09:25.110 --> 00:09:26.430 align:middle line:90%
The third copy goes down.

00:09:26.430 --> 00:09:28.970 align:middle line:84%
And don't worry about
the third copy just yet.

00:09:28.970 --> 00:09:31.530 align:middle line:84%
So if you look at
the first two copies,

00:09:31.530 --> 00:09:35.370 align:middle line:84%
here is the key thing to focus
on, this whole thing here.

00:09:35.370 --> 00:09:36.850 align:middle line:84%
Remember that we
want to calculate

00:09:36.850 --> 00:09:39.193 align:middle line:84%
dot products between
all these vectors,

00:09:39.193 --> 00:09:40.610 align:middle line:84%
and basically we
want to calculate

00:09:40.610 --> 00:09:44.835 align:middle line:84%
the dot product of every pair
of vectors, every pair of words.

00:09:44.835 --> 00:09:46.210 align:middle line:84%
The whole point
of self-attention

00:09:46.210 --> 00:09:47.960 align:middle line:84%
is that every pair of
words, we figure out

00:09:47.960 --> 00:09:49.388 align:middle line:84%
how attracted or
related they are,

00:09:49.388 --> 00:09:50.930 align:middle line:84%
which means that we
have to calculate

00:09:50.930 --> 00:09:52.810 align:middle line:90%
all pairs of dot products.

00:09:52.810 --> 00:09:55.890 align:middle line:84%
And so what you do is you
take this vector right there,

00:09:55.890 --> 00:10:00.230 align:middle line:84%
W1, W, W3, you take this
other copy that went up,

00:10:00.230 --> 00:10:02.513 align:middle line:90%
and then you transpose it.

00:10:02.513 --> 00:10:04.930 align:middle line:84%
So when you transpose it, it
all becomes nice and vertical

00:10:04.930 --> 00:10:07.210 align:middle line:90%
like that.

00:10:07.210 --> 00:10:08.830 align:middle line:84%
All the vectors
came in like this.

00:10:08.830 --> 00:10:10.690 align:middle line:84%
When you transfer it,
it becomes vertical.

00:10:10.690 --> 00:10:13.730 align:middle line:84%
And now what you do
is you take each one.

00:10:13.730 --> 00:10:16.930 align:middle line:84%
You take W1 and then you
multiply it by W1 here.

00:10:16.930 --> 00:10:21.152 align:middle line:90%
You take W1, W2, W1, W3.

00:10:21.152 --> 00:10:23.110 align:middle line:84%
You calculate all those
dot products like that.

00:10:23.110 --> 00:10:25.810 align:middle line:84%
And when you do that,
you have these nice cells

00:10:25.810 --> 00:10:29.250 align:middle line:84%
where every pair of
words, the dot products

00:10:29.250 --> 00:10:31.510 align:middle line:84%
have been calculated
in this grid.

00:10:31.510 --> 00:10:33.490 align:middle line:90%
And the key thing to see here--

00:10:33.490 --> 00:10:35.690 align:middle line:84%
and folks with a matrix
algebra background

00:10:35.690 --> 00:10:37.370 align:middle line:90%
will see this immediately--

00:10:37.370 --> 00:10:39.450 align:middle line:84%
all we are doing is we
are taking this x, which

00:10:39.450 --> 00:10:42.970 align:middle line:84%
is the matrix that
came in, and then

00:10:42.970 --> 00:10:46.250 align:middle line:84%
X transpose, which is the matrix
that we sent up and then brought

00:10:46.250 --> 00:10:47.110 align:middle line:90%
back down.

00:10:47.110 --> 00:10:49.050 align:middle line:84%
We are basically doing
a matrix multiplication

00:10:49.050 --> 00:10:50.310 align:middle line:90%
of x times x transpose.

00:10:50.310 --> 00:10:51.730 align:middle line:90%
That's all we are doing.

00:10:51.730 --> 00:10:57.330 align:middle line:84%
And when we do that, we're
getting this nice grid

00:10:57.330 --> 00:10:59.650 align:middle line:84%
in which every pair of
words, their dot products

00:10:59.650 --> 00:11:03.090 align:middle line:84%
have been calculated for you
with one matrix multiplication.

00:11:03.090 --> 00:11:04.750 align:middle line:90%
Boom, done.

00:11:04.750 --> 00:11:07.770 align:middle line:84%
So if you have three words,
there are 9 multiplications.

00:11:07.770 --> 00:11:11.370 align:middle line:84%
So if you have a
million words, that's

00:11:11.370 --> 00:11:15.330 align:middle line:84%
a lot of multiplications,
one trillion multiplications,

00:11:15.330 --> 00:11:17.330 align:middle line:90%
on the order of one trillion.

00:11:17.330 --> 00:11:21.010 align:middle line:84%
And the reason I say order
of is because W1 times W3

00:11:21.010 --> 00:11:24.290 align:middle line:84%
is the same as W3 times W1, so
there is some duplication here.

00:11:24.290 --> 00:11:28.370 align:middle line:84%
So you get this grid in one
shot, in one multiplication.

00:11:28.370 --> 00:11:30.215 align:middle line:84%
And then because
each of these numbers

00:11:30.215 --> 00:11:32.590 align:middle line:84%
is just a dot product, which
can be negative or positive,

00:11:32.590 --> 00:11:34.572 align:middle line:90%
we need to softmax it.

00:11:34.572 --> 00:11:36.530 align:middle line:84%
And so what we do is we
take all these numbers,

00:11:36.530 --> 00:11:40.190 align:middle line:84%
and we put it into a softmax
function where for each row,

00:11:40.190 --> 00:11:41.310 align:middle line:90%
it calculates the softmax.

00:11:41.310 --> 00:11:42.435 align:middle line:90%
And what do I mean by that?

00:11:42.435 --> 00:11:45.732 align:middle line:84%
It takes each number here,
does e raised to the top, e

00:11:45.732 --> 00:11:48.190 align:middle line:84%
raised to the number, it does
it for each of these numbers,

00:11:48.190 --> 00:11:51.330 align:middle line:84%
and then divides by the sum
of those numbers for each row.

00:11:51.330 --> 00:11:53.430 align:middle line:90%
And when you do that--

00:11:53.430 --> 00:11:56.610 align:middle line:84%
you can think of this operation
as softmax applied to x times

00:11:56.610 --> 00:12:01.010 align:middle line:84%
x transpose-- you get this
nice little table of numbers.

00:12:01.010 --> 00:12:04.890 align:middle line:84%
This table of numbers basically
says that for the first word,

00:12:04.890 --> 00:12:09.490 align:middle line:84%
W1, for the first word, take
0.1 of the first one, 0.7

00:12:09.490 --> 00:12:13.450 align:middle line:84%
of the second, 0.2 of the
third, and add them up.

00:12:13.450 --> 00:12:14.890 align:middle line:90%
We do a weighted average.

00:12:14.890 --> 00:12:16.930 align:middle line:90%
So we have this table here.

00:12:16.930 --> 00:12:21.330 align:middle line:84%
We have now the third
copy, shows up here.

00:12:21.330 --> 00:12:22.290 align:middle line:90%
It's right there.

00:12:22.290 --> 00:12:25.650 align:middle line:84%
So we do this times that, which
is just a matrix multiplication

00:12:25.650 --> 00:12:26.370 align:middle line:90%
again.

00:12:26.370 --> 00:12:30.090 align:middle line:84%
And when we do that, we get the
final contextual embeddings.

00:12:30.090 --> 00:12:36.730 align:middle line:84%
So this, for example, is just
0.1 times W1, 0.2 times W2--

00:12:36.730 --> 00:12:40.430 align:middle line:84%
sorry, 0.7 times W2 and then
0.2 times W3 right there.

00:12:40.430 --> 00:12:43.130 align:middle line:84%
And you can see the
same logic here as well.

00:12:43.130 --> 00:12:44.910 align:middle line:90%
And you can read it later on.

00:12:44.910 --> 00:12:47.050 align:middle line:84%
I will post this thing to
make sure you understand

00:12:47.050 --> 00:12:48.530 align:middle line:90%
exactly how it flowed.

00:12:48.530 --> 00:12:51.250 align:middle line:84%
But the larger point
I want you to focus on

00:12:51.250 --> 00:12:54.810 align:middle line:84%
is that the entire
self-attention operation we just

00:12:54.810 --> 00:12:58.330 align:middle line:84%
looked at here basically
is this beautifully little

00:12:58.330 --> 00:12:59.720 align:middle line:90%
compact matrix formula.

00:12:59.720 --> 00:13:02.250 align:middle line:90%


00:13:02.250 --> 00:13:04.987 align:middle line:84%
x comes in, you do x transpose,
you do a matrix multiplication,

00:13:04.987 --> 00:13:07.570 align:middle line:84%
you do a softmax on top of it,
and then multiply it by x again

00:13:07.570 --> 00:13:10.250 align:middle line:90%
and boom, you're done.

00:13:10.250 --> 00:13:14.170 align:middle line:84%
So that is the magic of
taking the transformer stack

00:13:14.170 --> 00:13:17.130 align:middle line:84%
and representing it using matrix
operations because then [POOF],

00:13:17.130 --> 00:13:19.970 align:middle line:90%
lightning fast on GPUs.

00:13:19.970 --> 00:13:20.930 align:middle line:90%
OK?

00:13:20.930 --> 00:13:24.610 align:middle line:90%
All right, that was the warm-up.

00:13:24.610 --> 00:13:27.050 align:middle line:90%
Now let's crank it up a notch.

00:13:27.050 --> 00:13:31.010 align:middle line:84%
So recall that in
the last class,

00:13:31.010 --> 00:13:37.250 align:middle line:84%
I talked about the fact the
self-attention operation,

00:13:37.250 --> 00:13:39.610 align:middle line:84%
the Ws are coming in, and
we're doing all this stuff

00:13:39.610 --> 00:13:43.830 align:middle line:84%
with the Ws, and then we're
getting some W hats out,

00:13:43.830 --> 00:13:46.570 align:middle line:90%
but there are no parameters.

00:13:46.570 --> 00:13:49.690 align:middle line:84%
There IS nothing to be
learned inside the transformer

00:13:49.690 --> 00:13:51.390 align:middle line:90%
self-attention layer, right?

00:13:51.390 --> 00:13:52.397 align:middle line:90%
There are no weights.

00:13:52.397 --> 00:13:53.230 align:middle line:90%
There are no biases.

00:13:53.230 --> 00:13:55.450 align:middle line:90%
There are no coefficients.

00:13:55.450 --> 00:13:58.930 align:middle line:84%
Well, OK, what are
we learning then?

00:13:58.930 --> 00:14:00.730 align:middle line:84%
So what we now do
is we are going

00:14:00.730 --> 00:14:03.250 align:middle line:84%
to make the self-attention
layer tunable.

00:14:03.250 --> 00:14:05.450 align:middle line:84%
We are going to inject
some weights into it

00:14:05.450 --> 00:14:08.210 align:middle line:84%
so that when we train
it on an actual system,

00:14:08.210 --> 00:14:10.250 align:middle line:84%
the weights will keep
changing to adapt itself

00:14:10.250 --> 00:14:12.210 align:middle line:84%
to the particularities
of whatever problem

00:14:12.210 --> 00:14:14.650 align:middle line:90%
you're working on.

00:14:14.650 --> 00:14:23.090 align:middle line:84%
So that takes us to the
tunable self-attention layer,

00:14:23.090 --> 00:14:25.250 align:middle line:90%
tunable self-attention layer.

00:14:25.250 --> 00:14:28.105 align:middle line:84%
So this is the key
thing to keep in mind.

00:14:28.105 --> 00:14:29.730 align:middle line:84%
Any questions on this
before I continue

00:14:29.730 --> 00:14:30.990 align:middle line:90%
with the tunability thing?

00:14:30.990 --> 00:14:34.530 align:middle line:90%


00:14:34.530 --> 00:14:36.860 align:middle line:90%
OK.

00:14:36.860 --> 00:14:39.670 align:middle line:84%
Is this picture working
out, by the way?

00:14:39.670 --> 00:14:40.170 align:middle line:90%
OK.

00:14:40.170 --> 00:14:43.970 align:middle line:90%


00:14:43.970 --> 00:14:47.330 align:middle line:84%
So what we now do is we have
the same exact logic as before,

00:14:47.330 --> 00:14:50.170 align:middle line:84%
where we have this
thing that comes in.

00:14:50.170 --> 00:14:52.930 align:middle line:84%
We have this input
that comes in.

00:14:52.930 --> 00:14:56.570 align:middle line:84%
We call it x again, this
matrix of embeddings.

00:14:56.570 --> 00:14:59.775 align:middle line:84%
And then before we
just send three copies,

00:14:59.775 --> 00:15:01.650 align:middle line:84%
instead of doing that,
what we're going to do

00:15:01.650 --> 00:15:04.090 align:middle line:84%
is we'll take each
copy, x, and then

00:15:04.090 --> 00:15:07.690 align:middle line:84%
we will actually
multiply it by a matrix.

00:15:07.690 --> 00:15:11.462 align:middle line:84%
This matrix is called
the key matrix.

00:15:11.462 --> 00:15:15.130 align:middle line:84%
And this matrix, this
matrix of numbers or weights

00:15:15.130 --> 00:15:18.353 align:middle line:84%
that will be
learned by backprop.

00:15:18.353 --> 00:15:19.770 align:middle line:84%
So basically what
we are saying is

00:15:19.770 --> 00:15:22.730 align:middle line:84%
that when this thing
comes in, let's

00:15:22.730 --> 00:15:24.810 align:middle line:84%
see if there is a
way to transform

00:15:24.810 --> 00:15:28.130 align:middle line:84%
this x into some other
set of embeddings which

00:15:28.130 --> 00:15:30.017 align:middle line:90%
may be useful for your task.

00:15:30.017 --> 00:15:31.850 align:middle line:84%
We don't know if they're
going to be useful,

00:15:31.850 --> 00:15:34.130 align:middle line:84%
but surely giving it
a bit more ability

00:15:34.130 --> 00:15:36.170 align:middle line:84%
to have weights
which can be learned

00:15:36.170 --> 00:15:39.850 align:middle line:84%
means that we are giving it more
expressive power, more modeling

00:15:39.850 --> 00:15:40.557 align:middle line:90%
capacity.

00:15:40.557 --> 00:15:42.890 align:middle line:84%
And whether it actually uses
the capacity will depend on

00:15:42.890 --> 00:15:45.410 align:middle line:84%
how much data you have
and how well you train it.

00:15:45.410 --> 00:15:48.370 align:middle line:84%
And maybe if it's not
useful, it won't use it.

00:15:48.370 --> 00:15:50.930 align:middle line:84%
What I mean is if transforming
x actually doesn't really

00:15:50.930 --> 00:15:54.630 align:middle line:84%
help at all, then this
matrix A is going to be what?

00:15:54.630 --> 00:15:55.610 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:15:55.610 --> 00:15:56.730 align:middle line:84%
RAMA RAMAKRISHNAN: It's
going to be the identity

00:15:56.730 --> 00:15:58.890 align:middle line:84%
matrix because you
take basically 1

00:15:58.890 --> 00:16:01.170 align:middle line:84%
and multiply it by x,
you'll get x again.

00:16:01.170 --> 00:16:02.830 align:middle line:84%
So the worst case,
maybe it just says,

00:16:02.830 --> 00:16:04.970 align:middle line:84%
I have nothing to
learn here, but maybe

00:16:04.970 --> 00:16:06.610 align:middle line:84%
there is something
you can learn.

00:16:06.610 --> 00:16:08.270 align:middle line:90%
So that's what we do.

00:16:08.270 --> 00:16:11.010 align:middle line:84%
So we multiplied
by this matrix AK.

00:16:11.010 --> 00:16:13.610 align:middle line:84%
And then we come
up with the same--

00:16:13.610 --> 00:16:15.310 align:middle line:84%
some embeddings,
transformed embeddings,

00:16:15.310 --> 00:16:18.990 align:middle line:90%
and we call these things K.

00:16:18.990 --> 00:16:22.770 align:middle line:84%
Now this KQV, as you
will see, has its origins

00:16:22.770 --> 00:16:25.330 align:middle line:84%
in the field of
information retrieval,

00:16:25.330 --> 00:16:28.050 align:middle line:84%
but I personally find
that interpretation is not

00:16:28.050 --> 00:16:30.132 align:middle line:84%
super helpful
because transformers

00:16:30.132 --> 00:16:32.340 align:middle line:84%
are used for lots of
applications outside information

00:16:32.340 --> 00:16:33.780 align:middle line:84%
retrieval, so I'm
not going to go

00:16:33.780 --> 00:16:35.508 align:middle line:84%
with that kind of
interpretation.

00:16:35.508 --> 00:16:37.300 align:middle line:84%
I'm going to go with
the interpretation of,

00:16:37.300 --> 00:16:39.770 align:middle line:84%
let's make each of
these things tunable.

00:16:39.770 --> 00:16:42.480 align:middle line:84%
And tunability means we
need to give it weights.

00:16:42.480 --> 00:16:43.820 align:middle line:90%
So that's what we have here.

00:16:43.820 --> 00:16:46.147 align:middle line:90%
Now the second copy--

00:16:46.147 --> 00:16:47.480 align:middle line:90%
we did this with the first copy.

00:16:47.480 --> 00:16:49.120 align:middle line:84%
Well, let's do the same
thing with the second copy.

00:16:49.120 --> 00:16:50.870 align:middle line:84%
We'll take the second
copy and multiply it

00:16:50.870 --> 00:16:53.220 align:middle line:90%
by some other matrix called AQ.

00:16:53.220 --> 00:16:56.220 align:middle line:84%
And when we are done with
that, we get these embeddings,

00:16:56.220 --> 00:17:00.912 align:middle line:84%
and we will call
these embeddings as Q.

00:17:00.912 --> 00:17:04.540 align:middle line:84%
Now just like before, we
will take this thing here,

00:17:04.540 --> 00:17:08.380 align:middle line:84%
and we will transpose it so it
all becomes nice and vertical

00:17:08.380 --> 00:17:09.805 align:middle line:90%
like that.

00:17:09.805 --> 00:17:11.680 align:middle line:84%
And then we'll do exactly
the same as before.

00:17:11.680 --> 00:17:13.638 align:middle line:84%
We'll calculate all these
pairwise dot products

00:17:13.638 --> 00:17:16.619 align:middle line:84%
using one-shot one-matrix
multiplication.

00:17:16.619 --> 00:17:19.740 align:middle line:84%
And because we
are calling this Q

00:17:19.740 --> 00:17:21.780 align:middle line:84%
and we are calling
this whole thing as K,

00:17:21.780 --> 00:17:23.850 align:middle line:84%
this thing just
becomes Q times KT.

00:17:23.850 --> 00:17:26.859 align:middle line:90%


00:17:26.859 --> 00:17:30.180 align:middle line:84%
At the end of it, you come
up with a grid of numbers

00:17:30.180 --> 00:17:32.240 align:middle line:90%
just like before.

00:17:32.240 --> 00:17:34.240 align:middle line:84%
And these numbers could
be negative or positive,

00:17:34.240 --> 00:17:35.900 align:middle line:84%
so we need to do
the softmax on them

00:17:35.900 --> 00:17:37.942 align:middle line:84%
to make sure they are
well-behaved fractions that

00:17:37.942 --> 00:17:38.940 align:middle line:90%
add up to 1.

00:17:38.940 --> 00:17:44.340 align:middle line:84%
So we take this QKT
business and then

00:17:44.340 --> 00:17:48.340 align:middle line:84%
we put it through a softmax
function for each row.

00:17:48.340 --> 00:17:52.740 align:middle line:84%
And when we do that, we will get
basically a table like the ones

00:17:52.740 --> 00:17:53.600 align:middle line:90%
we saw before.

00:17:53.600 --> 00:17:55.308 align:middle line:84%
By the way, the numbers
here are the same

00:17:55.308 --> 00:17:57.700 align:middle line:84%
just because I duplicated
it because I'm lazy.

00:17:57.700 --> 00:17:59.140 align:middle line:84%
In reality, given
that it has gone

00:17:59.140 --> 00:18:00.557 align:middle line:84%
through all these
transformations,

00:18:00.557 --> 00:18:02.900 align:middle line:84%
the numbers are not
going to be the same.

00:18:02.900 --> 00:18:06.200 align:middle line:84%
You have these numbers and
then you take the final copy,

00:18:06.200 --> 00:18:08.437 align:middle line:90%
which is X times AV.

00:18:08.437 --> 00:18:10.520 align:middle line:84%
Each copy is getting
multiplied by its own matrix.

00:18:10.520 --> 00:18:13.100 align:middle line:84%
And this copy is being
multiplied by AV.

00:18:13.100 --> 00:18:19.420 align:middle line:84%
And let's call this XAV,
which is here, as just V.

00:18:19.420 --> 00:18:23.570 align:middle line:84%
And so what you have here is
this softmax QKT times V is

00:18:23.570 --> 00:18:25.820 align:middle line:84%
exactly the same kind of dot
product as we saw before,

00:18:25.820 --> 00:18:27.460 align:middle line:90%
matrix multiplication.

00:18:27.460 --> 00:18:29.340 align:middle line:84%
So we have these
contextual embeddings,

00:18:29.340 --> 00:18:32.680 align:middle line:84%
and that's what's coming out
of the transformer block.

00:18:32.680 --> 00:18:36.300 align:middle line:84%
So now the whole thing we
did here, the whole thing can

00:18:36.300 --> 00:18:40.500 align:middle line:84%
be represented as
softmax of QKT times V.

00:18:40.500 --> 00:18:44.180 align:middle line:90%
So if we zoom in a bit--

00:18:44.180 --> 00:18:44.800 align:middle line:90%
come on.

00:18:44.800 --> 00:18:49.100 align:middle line:90%


00:18:49.100 --> 00:18:54.540 align:middle line:84%
So X came in, three
tracks went here.

00:18:54.540 --> 00:18:58.940 align:middle line:84%
The first track X times
AK, X times AQ, X times AV.

00:18:58.940 --> 00:19:00.940 align:middle line:84%
And this thing is called
K. This thing is called

00:19:00.940 --> 00:19:03.260 align:middle line:84%
Q. This thing is
called V. And then

00:19:03.260 --> 00:19:05.540 align:middle line:84%
we do the same
transpose as before.

00:19:05.540 --> 00:19:08.740 align:middle line:84%
We do the dot product thing
to calculate the pairwise dot

00:19:08.740 --> 00:19:11.500 align:middle line:84%
products for everything,
which is just QKT.

00:19:11.500 --> 00:19:12.840 align:middle line:90%
We run it through a softmax.

00:19:12.840 --> 00:19:14.820 align:middle line:90%
We get softmax of QKT.

00:19:14.820 --> 00:19:17.940 align:middle line:84%
We multiply it by 1 to do the
final weighting, and then boom,

00:19:17.940 --> 00:19:20.040 align:middle line:84%
the output comes, and
that's this function.

00:19:20.040 --> 00:19:22.020 align:middle line:90%
That's it.

00:19:22.020 --> 00:19:24.540 align:middle line:84%
So what we have done
is we have introduced

00:19:24.540 --> 00:19:28.220 align:middle line:84%
three matrices,
learnable matrices,

00:19:28.220 --> 00:19:29.680 align:middle line:90%
into the self-attention layer.

00:19:29.680 --> 00:19:34.140 align:middle line:90%


00:19:34.140 --> 00:19:35.640 align:middle line:84%
Let me just stop
there for a second.

00:19:35.640 --> 00:19:37.380 align:middle line:90%
Questions?

00:19:37.380 --> 00:19:39.805 align:middle line:90%
Yeah.

00:19:39.805 --> 00:19:41.180 align:middle line:84%
AUDIENCE: Is there
a relationship

00:19:41.180 --> 00:19:43.088 align:middle line:90%
between AK, AQ, and AV?

00:19:43.088 --> 00:19:44.380 align:middle line:90%
RAMA RAMAKRISHNAN: Independent.

00:19:44.380 --> 00:19:45.255 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:19:45.255 --> 00:19:47.460 align:middle line:84%
RAMA RAMAKRISHNAN:
Independent matrices.

00:19:47.460 --> 00:19:48.345 align:middle line:90%
Yes?

00:19:48.345 --> 00:19:49.220 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:19:49.220 --> 00:19:50.580 align:middle line:84%
RAMA RAMAKRISHNAN: Could you
use the microphone, please?

00:19:50.580 --> 00:19:52.580 align:middle line:84%
AUDIENCE: Here we have
three sets of parameters,

00:19:52.580 --> 00:19:54.660 align:middle line:84%
K, Q, and V. If
there are, let's say,

00:19:54.660 --> 00:19:57.380 align:middle line:84%
if there were 100
or the total length

00:19:57.380 --> 00:20:00.480 align:middle line:84%
was, let's say, the number
of totals, let's say 50,

00:20:00.480 --> 00:20:02.700 align:middle line:84%
so you could have
50% of parameters

00:20:02.700 --> 00:20:04.740 align:middle line:90%
you have to [INAUDIBLE].

00:20:04.740 --> 00:20:07.460 align:middle line:84%
RAMA RAMAKRISHNAN:
So if the dimension

00:20:07.460 --> 00:20:09.620 align:middle line:84%
is 50 long, what is
coming in, the Ws

00:20:09.620 --> 00:20:13.820 align:middle line:84%
are 50 long, then the key,
what comes out of it, if you

00:20:13.820 --> 00:20:15.460 align:middle line:90%
want it to be 50 as well--

00:20:15.460 --> 00:20:24.900 align:middle line:84%
so this matrix needs to
be 50 times 50, 2,500.

00:20:24.900 --> 00:20:27.940 align:middle line:84%
AUDIENCE: What are the three
different things the three

00:20:27.940 --> 00:20:29.738 align:middle line:90%
matrices are trying to learn?

00:20:29.738 --> 00:20:30.780 align:middle line:90%
RAMA RAMAKRISHNAN: Sorry?

00:20:30.780 --> 00:20:32.580 align:middle line:84%
AUDIENCE: What are the different
things that the matrices are

00:20:32.580 --> 00:20:33.620 align:middle line:90%
trying to learn?

00:20:33.620 --> 00:20:34.360 align:middle line:84%
RAMA RAMAKRISHNAN:
We don't know.

00:20:34.360 --> 00:20:36.840 align:middle line:84%
All we are saying is that we
have a self-attention layer,

00:20:36.840 --> 00:20:38.840 align:middle line:84%
which can pay attention
to every pair of words,

00:20:38.840 --> 00:20:41.340 align:middle line:84%
but we need to give it
some ways to transform

00:20:41.340 --> 00:20:45.192 align:middle line:84%
what is coming in into
potentially useful things.

00:20:45.192 --> 00:20:47.660 align:middle line:84%
As to their actual
usefulness, we'll

00:20:47.660 --> 00:20:49.792 align:middle line:84%
have to figure out if it
actually helps or not.

00:20:49.792 --> 00:20:51.500 align:middle line:84%
And of course, as you
know, the punchline

00:20:51.500 --> 00:20:53.000 align:middle line:84%
is that yeah, it
helps massively.

00:20:53.000 --> 00:20:53.960 align:middle line:90%
That's why we do it.

00:20:53.960 --> 00:20:56.085 align:middle line:84%
In general, what you will
find in the deep learning

00:20:56.085 --> 00:20:57.620 align:middle line:84%
literature is that
whenever you want

00:20:57.620 --> 00:20:59.220 align:middle line:84%
to increase the
capacity, the modeling

00:20:59.220 --> 00:21:02.780 align:middle line:84%
capacity of a particular model,
you just take a small piece

00:21:02.780 --> 00:21:05.860 align:middle line:84%
and inject a little matrix
multiplication into it.

00:21:05.860 --> 00:21:08.080 align:middle line:84%
You take a vector that's
showing up in the middle,

00:21:08.080 --> 00:21:10.860 align:middle line:84%
and then you make it run through
a matrix to get another vector.

00:21:10.860 --> 00:21:13.560 align:middle line:84%
And then further, after you
run it through a matrix,

00:21:13.560 --> 00:21:15.480 align:middle line:84%
you run it through a
little ReLU as well.

00:21:15.480 --> 00:21:16.860 align:middle line:90%
Even better.

00:21:16.860 --> 00:21:19.500 align:middle line:84%
So that's how you
inject modeling capacity

00:21:19.500 --> 00:21:22.140 align:middle line:84%
into the middle
of these networks,

00:21:22.140 --> 00:21:24.460 align:middle line:84%
and that's what these
people are doing here.

00:21:24.460 --> 00:21:26.740 align:middle line:90%
Yeah?

00:21:26.740 --> 00:21:29.020 align:middle line:84%
AUDIENCE: In the last
step, you had the matrix

00:21:29.020 --> 00:21:30.837 align:middle line:84%
V. So on the
previous example, you

00:21:30.837 --> 00:21:33.420 align:middle line:84%
had used the original matrix X.
Could you just say [INAUDIBLE]

00:21:33.420 --> 00:21:35.000 align:middle line:90%
why is it not using X?

00:21:35.000 --> 00:21:36.080 align:middle line:90%
And what does that mean?

00:21:36.080 --> 00:21:37.788 align:middle line:84%
RAMA RAMAKRISHNAN: So
what you are saying

00:21:37.788 --> 00:21:40.280 align:middle line:84%
is that in the initial
version we had three copies,

00:21:40.280 --> 00:21:42.060 align:middle line:84%
and we treated
them all identical.

00:21:42.060 --> 00:21:43.762 align:middle line:84%
Now we said, well,
are there ways

00:21:43.762 --> 00:21:46.220 align:middle line:84%
to transform each copy into
some other representation which

00:21:46.220 --> 00:21:48.300 align:middle line:84%
could be useful,
so we may as well

00:21:48.300 --> 00:21:49.980 align:middle line:84%
use three different
matrices for it.

00:21:49.980 --> 00:21:51.382 align:middle line:90%
Why stop with two?

00:21:51.382 --> 00:21:53.840 align:middle line:84%
There are three opportunities
to make them more expressive.

00:21:53.840 --> 00:21:56.360 align:middle line:90%
We'll use all of them.

00:21:56.360 --> 00:21:56.860 align:middle line:90%
Yeah?

00:21:56.860 --> 00:21:59.740 align:middle line:90%


00:21:59.740 --> 00:22:02.100 align:middle line:84%
AUDIENCE: You mentioned
that these are--

00:22:02.100 --> 00:22:04.468 align:middle line:90%
you're fine-tuning it.

00:22:04.468 --> 00:22:05.260 align:middle line:90%
Is there any risk--

00:22:05.260 --> 00:22:07.093 align:middle line:84%
RAMA RAMAKRISHNAN: We're
not fine-tuning it,

00:22:07.093 --> 00:22:08.980 align:middle line:84%
just to be clear on
the vocabulary here.

00:22:08.980 --> 00:22:11.520 align:middle line:84%
So we have added more
weights to make them tunable.

00:22:11.520 --> 00:22:15.952 align:middle line:84%
What that means is that when we
finally train this entire model,

00:22:15.952 --> 00:22:17.660 align:middle line:84%
remember, all the
weights are going to be

00:22:17.660 --> 00:22:19.320 align:middle line:90%
updated using back propagation.

00:22:19.320 --> 00:22:21.860 align:middle line:84%
In particular, these
matrices will also

00:22:21.860 --> 00:22:23.862 align:middle line:84%
get updated using
back propagation.

00:22:23.862 --> 00:22:26.320 align:middle line:84%
AUDIENCE: So there's no risk
of over-- is there a risk of--

00:22:26.320 --> 00:22:26.780 align:middle line:84%
RAMA RAMAKRISHNAN:
There is always

00:22:26.780 --> 00:22:28.740 align:middle line:84%
a risk of overfitting when
you add more parameters

00:22:28.740 --> 00:22:31.420 align:middle line:84%
to a model, which means that you
have to look at the validation

00:22:31.420 --> 00:22:34.060 align:middle line:90%
set and all that good stuff.

00:22:34.060 --> 00:22:37.340 align:middle line:84%
We are basically adding more
parameters in a very interesting

00:22:37.340 --> 00:22:40.113 align:middle line:84%
way because we want
to add more capacity

00:22:40.113 --> 00:22:41.280 align:middle line:90%
to the self-attention layer.

00:22:41.280 --> 00:22:42.822 align:middle line:84%
We want to give it
more of an ability

00:22:42.822 --> 00:22:44.380 align:middle line:90%
to learn things from the data.

00:22:44.380 --> 00:22:45.920 align:middle line:84%
Before it could
not learn anything.

00:22:45.920 --> 00:22:48.020 align:middle line:90%
It could only do dot products.

00:22:48.020 --> 00:22:50.765 align:middle line:84%
So we want to
solve that problem.

00:22:50.765 --> 00:22:52.140 align:middle line:84%
All right, I'm
going to continue,

00:22:52.140 --> 00:22:53.307 align:middle line:90%
and we'll come back to this.

00:22:53.307 --> 00:22:58.820 align:middle line:90%


00:22:58.820 --> 00:23:01.100 align:middle line:84%
Just for fun, I'm
going to do this.

00:23:01.100 --> 00:23:04.620 align:middle line:84%
The original paper is called,
"Attention is All You Need."

00:23:04.620 --> 00:23:07.620 align:middle line:90%
This is a transformer paper.

00:23:07.620 --> 00:23:11.343 align:middle line:84%
You folks should read
it at some point.

00:23:11.343 --> 00:23:12.760 align:middle line:84%
I just want to
show you something.

00:23:12.760 --> 00:23:19.980 align:middle line:90%


00:23:19.980 --> 00:23:21.340 align:middle line:90%
You see that?

00:23:21.340 --> 00:23:24.740 align:middle line:84%
So that is the famous
transformer formula.

00:23:24.740 --> 00:23:28.380 align:middle line:84%
And the only thing we
ignored is this root

00:23:28.380 --> 00:23:31.220 align:middle line:90%
of dk business under it.

00:23:31.220 --> 00:23:32.360 align:middle line:90%
I wouldn't worry about it.

00:23:32.360 --> 00:23:35.640 align:middle line:84%
The reason they have it is
because these softmaxes, when

00:23:35.640 --> 00:23:38.140 align:middle line:84%
you have lots of numbers and
some numbers are really, really

00:23:38.140 --> 00:23:40.980 align:middle line:84%
big, what's going to happen
is that all the other numbers

00:23:40.980 --> 00:23:43.340 align:middle line:90%
are going to get squashed to 0.

00:23:43.340 --> 00:23:46.240 align:middle line:84%
And so to make sure the
gradient flows properly,

00:23:46.240 --> 00:23:48.220 align:middle line:84%
they just divide it
by a particular number

00:23:48.220 --> 00:23:51.060 align:middle line:84%
to make sure no
number is too big.

00:23:51.060 --> 00:23:52.597 align:middle line:84%
It's a small
technical-- important,

00:23:52.597 --> 00:23:54.180 align:middle line:84%
but a bit of a
technical detail, which

00:23:54.180 --> 00:23:56.120 align:middle line:90%
is why I ignored it in my iPad.

00:23:56.120 --> 00:23:58.220 align:middle line:84%
But the rest of it,
you can see this

00:23:58.220 --> 00:24:04.360 align:middle line:84%
is exactly the formula we
derived, QKT times V softmax.

00:24:04.360 --> 00:24:08.020 align:middle line:84%
So this is the famous
transformer formula.

00:24:08.020 --> 00:24:11.572 align:middle line:84%
And congratulations,
now you understand it.

00:24:11.572 --> 00:24:13.030 align:middle line:84%
You seem less than
fully convinced?

00:24:13.030 --> 00:24:17.180 align:middle line:90%
[CHUCKLES] OK.

00:24:17.180 --> 00:24:19.060 align:middle line:90%
Yes, iPad.

00:24:19.060 --> 00:24:21.780 align:middle line:84%
Now I have a bunch of
slides, which I had--

00:24:21.780 --> 00:24:23.860 align:middle line:84%
actually, I'll
come back to this.

00:24:23.860 --> 00:24:26.603 align:middle line:84%
I had a bunch of other slides--
this is from last year--

00:24:26.603 --> 00:24:28.020 align:middle line:84%
which actually
explains what I did

00:24:28.020 --> 00:24:29.478 align:middle line:84%
in the iPad in a
very different way

00:24:29.478 --> 00:24:31.380 align:middle line:84%
without using any
matrices and so on.

00:24:31.380 --> 00:24:33.240 align:middle line:84%
I was looking at
it last evening,

00:24:33.240 --> 00:24:36.420 align:middle line:84%
and I was getting very annoyed
by these slides for some reason

00:24:36.420 --> 00:24:38.340 align:middle line:84%
because I felt that it
wasn't really conveying

00:24:38.340 --> 00:24:45.562 align:middle line:84%
the ability of using matrix
algebra to actually do this

00:24:45.562 --> 00:24:47.020 align:middle line:84%
so efficiently and
compactly, which

00:24:47.020 --> 00:24:50.320 align:middle line:84%
is why I decided to hand-draw
this thing on the iPad.

00:24:50.320 --> 00:24:52.060 align:middle line:84%
But you should
read it afterwards

00:24:52.060 --> 00:24:54.620 align:middle line:84%
to make sure that whatever
you saw on the iPad

00:24:54.620 --> 00:24:56.340 align:middle line:84%
actually matches
this because two

00:24:56.340 --> 00:25:00.020 align:middle line:84%
different ways of understanding
something always helps.

00:25:00.020 --> 00:25:01.800 align:middle line:90%
So this is what we have here.

00:25:01.800 --> 00:25:07.063 align:middle line:84%
Now to just to recall, by
making self-attention tunable,

00:25:07.063 --> 00:25:08.480 align:middle line:84%
we get a very
interesting benefit,

00:25:08.480 --> 00:25:10.780 align:middle line:84%
which is that when you have
these different attention

00:25:10.780 --> 00:25:14.600 align:middle line:84%
heads, before, you could
have two attention heads,

00:25:14.600 --> 00:25:16.740 align:middle line:84%
but because there were
no parameters inside,

00:25:16.740 --> 00:25:19.260 align:middle line:84%
their outputs would
have been identical.

00:25:19.260 --> 00:25:21.312 align:middle line:84%
Because the inputs
are the same for both.

00:25:21.312 --> 00:25:23.020 align:middle line:84%
Therefore, the outputs
will be identical.

00:25:23.020 --> 00:25:28.220 align:middle line:84%
But now since each attention
head will have its own AQ AK AV

00:25:28.220 --> 00:25:31.940 align:middle line:84%
matrix, the outputs are
going to be different.

00:25:31.940 --> 00:25:35.160 align:middle line:84%
That's why it makes sense
to do the tunability thing

00:25:35.160 --> 00:25:37.660 align:middle line:84%
because that's what actually
makes multiple attention-- it's

00:25:37.660 --> 00:25:38.327 align:middle line:90%
actually useful.

00:25:38.327 --> 00:25:42.987 align:middle line:90%


00:25:42.987 --> 00:25:44.820 align:middle line:84%
AUDIENCE: Is there
actually any relationship

00:25:44.820 --> 00:25:47.126 align:middle line:84%
between AK, AQ,
and AV, or is the A

00:25:47.126 --> 00:25:49.245 align:middle line:90%
just for a notation standpoint?

00:25:49.245 --> 00:25:50.620 align:middle line:84%
RAMA RAMAKRISHNAN:
Just notation.

00:25:50.620 --> 00:25:54.340 align:middle line:84%
The thing is we want to use
QKV for the resulting matrix,

00:25:54.340 --> 00:25:57.000 align:middle line:84%
and so I had to find something
else to use for the first one.

00:25:57.000 --> 00:25:58.500 align:middle line:90%
And I was like, OK, AQ AQ.

00:25:58.500 --> 00:26:01.080 align:middle line:84%
And we are at MIT, we do
subscripts, superscripts.

00:26:01.080 --> 00:26:03.340 align:middle line:90%
Yeah?

00:26:03.340 --> 00:26:05.700 align:middle line:84%
AUDIENCE: What is the
size of the matrices?

00:26:05.700 --> 00:26:07.540 align:middle line:90%
They're like square matrices or?

00:26:07.540 --> 00:26:08.540 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:26:08.540 --> 00:26:11.760 align:middle line:84%
So typically what happens is
that there is a whole bunch--

00:26:11.760 --> 00:26:14.380 align:middle line:84%
you can think of it as a
hyperparameter in some ways.

00:26:14.380 --> 00:26:16.642 align:middle line:84%
Typically what people do
in most implementations is

00:26:16.642 --> 00:26:18.600 align:middle line:84%
that they will actually
just preserve the size.

00:26:18.600 --> 00:26:20.375 align:middle line:84%
So if the incoming
embedding is 10,

00:26:20.375 --> 00:26:22.500 align:middle line:84%
they'll make sure the thing
coming out of the thing

00:26:22.500 --> 00:26:23.520 align:middle line:90%
is also 10.

00:26:23.520 --> 00:26:26.980 align:middle line:84%
So you just do a 10 by 10
matrix to transform it.

00:26:26.980 --> 00:26:30.962 align:middle line:84%
But the value V AV
matrix, on the other hand,

00:26:30.962 --> 00:26:32.420 align:middle line:84%
there is a bit more
technical stuff

00:26:32.420 --> 00:26:36.020 align:middle line:84%
going on where it often
tends to be smaller.

00:26:36.020 --> 00:26:38.540 align:middle line:84%
So for example, let's say
that your incoming is 100.

00:26:38.540 --> 00:26:41.760 align:middle line:84%
You do 100 to 100 for the
key, 100 to 100 for the query.

00:26:41.760 --> 00:26:46.100 align:middle line:84%
But if you have say, 5 attention
heads, you may do 100 to 20

00:26:46.100 --> 00:26:48.220 align:middle line:84%
for the Vs because
ultimately all the Vs

00:26:48.220 --> 00:26:51.340 align:middle line:84%
are going to get concatenated
into another 100 again.

00:26:51.340 --> 00:26:53.140 align:middle line:90%
So I can tell you more offline.

00:26:53.140 --> 00:26:56.080 align:middle line:84%
But broadly speaking, these
things tend to get transformed.

00:26:56.080 --> 00:26:58.660 align:middle line:84%
They preserve the
dimension, 10 in, 10 out.

00:26:58.660 --> 00:27:00.260 align:middle line:90%
Yeah?

00:27:00.260 --> 00:27:04.700 align:middle line:84%
AUDIENCE: So this
AQ AV, these numbers

00:27:04.700 --> 00:27:06.995 align:middle line:84%
are random you start with
it and then [INAUDIBLE].

00:27:06.995 --> 00:27:08.120 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:27:08.120 --> 00:27:08.620 align:middle line:90%
Exactly.

00:27:08.620 --> 00:27:17.620 align:middle line:90%


00:27:17.620 --> 00:27:20.020 align:middle line:84%
So the values in these
matrices are weights learned

00:27:20.020 --> 00:27:22.460 align:middle line:90%
through optimization using SGD.

00:27:22.460 --> 00:27:26.860 align:middle line:84%
And then what that means is that
each of these attention rates

00:27:26.860 --> 00:27:28.880 align:middle line:84%
now has its own copy
of these matrices.

00:27:28.880 --> 00:27:30.540 align:middle line:90%
It has its own matrices.

00:27:30.540 --> 00:27:32.180 align:middle line:84%
And over the course
of backpropagation,

00:27:32.180 --> 00:27:35.272 align:middle line:84%
these matrices will
look very different.

00:27:35.272 --> 00:27:36.700 align:middle line:84%
So important, each
attention head

00:27:36.700 --> 00:27:39.723 align:middle line:84%
will have its own set
of three matrices.

00:27:39.723 --> 00:27:41.140 align:middle line:84%
So if you have 10
attention heads,

00:27:41.140 --> 00:27:42.307 align:middle line:90%
30 matrices will be learned.

00:27:42.307 --> 00:27:46.300 align:middle line:90%


00:27:46.300 --> 00:27:48.060 align:middle line:84%
AUDIENCE: So by
the math, it seems

00:27:48.060 --> 00:27:51.380 align:middle line:84%
like it's creating essentially
a relationship between all

00:27:51.380 --> 00:27:52.960 align:middle line:90%
of the content being ingested.

00:27:52.960 --> 00:27:57.360 align:middle line:84%
And if you're ingesting all the
content for each attention head,

00:27:57.360 --> 00:28:00.140 align:middle line:84%
are there different categories
of attention head type

00:28:00.140 --> 00:28:01.760 align:middle line:90%
that you're trying to go after?

00:28:01.760 --> 00:28:02.060 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:28:02.060 --> 00:28:04.560 align:middle line:84%
So basically what we are trying
to do is to say a particular

00:28:04.560 --> 00:28:06.860 align:middle line:84%
attention head-- so in
any particular sentence,

00:28:06.860 --> 00:28:09.980 align:middle line:84%
it may turn out to be the case
that one pattern could be about

00:28:09.980 --> 00:28:12.110 align:middle line:84%
the meanings of these
words, like the word "bank"

00:28:12.110 --> 00:28:14.302 align:middle line:84%
and what it means, the
word "station," "train,"

00:28:14.302 --> 00:28:15.010 align:middle line:90%
things like that.

00:28:15.010 --> 00:28:16.850 align:middle line:84%
That's what really we've
been talking about.

00:28:16.850 --> 00:28:18.310 align:middle line:84%
But there is a
whole other pattern

00:28:18.310 --> 00:28:21.050 align:middle line:84%
to do with grammar and
tense and things like that.

00:28:21.050 --> 00:28:23.590 align:middle line:84%
There could be another
one in terms of tone.

00:28:23.590 --> 00:28:25.950 align:middle line:84%
All those things are very
important, and a priori

00:28:25.950 --> 00:28:27.970 align:middle line:84%
we don't know how many
such patterns exist.

00:28:27.970 --> 00:28:29.687 align:middle line:84%
Much like in a
convolutional network,

00:28:29.687 --> 00:28:31.270 align:middle line:84%
when we are designing
how many filters

00:28:31.270 --> 00:28:33.520 align:middle line:84%
to have, we don't know how
many kinds of little things

00:28:33.520 --> 00:28:36.697 align:middle line:84%
we have to detect, vertical
line, horizontal line,

00:28:36.697 --> 00:28:38.530 align:middle line:84%
semicircle, quarter
circle, stuff like that.

00:28:38.530 --> 00:28:40.535 align:middle line:84%
So you just give it
a lot of capacity

00:28:40.535 --> 00:28:42.160 align:middle line:84%
so that it can learn
whatever it wants.

00:28:42.160 --> 00:28:45.530 align:middle line:90%


00:28:45.530 --> 00:28:48.650 align:middle line:84%
So that is the
transformer encoder.

00:28:48.650 --> 00:28:52.110 align:middle line:84%
So we have done the first of
the three complications needed

00:28:52.110 --> 00:28:55.772 align:middle line:84%
to make it industrial
strength and legit.

00:28:55.772 --> 00:28:57.230 align:middle line:84%
The second thing
we do is something

00:28:57.230 --> 00:28:59.230 align:middle line:90%
called the residual connection.

00:28:59.230 --> 00:29:04.350 align:middle line:84%
So what we do is that
whatever comes out here,

00:29:04.350 --> 00:29:08.750 align:middle line:84%
W1 through W6, goes in and
comes out as W1 hat, W2,

00:29:08.750 --> 00:29:10.690 align:middle line:90%
and so on and so forth.

00:29:10.690 --> 00:29:12.790 align:middle line:90%
Actually, sorry.

00:29:12.790 --> 00:29:14.670 align:middle line:90%
What comes out here is the hats.

00:29:14.670 --> 00:29:17.582 align:middle line:84%
But what comes out here
is some intermediate Ws.

00:29:17.582 --> 00:29:19.790 align:middle line:84%
That is what the self-attention
is going to give you,

00:29:19.790 --> 00:29:21.310 align:middle line:90%
some intermediate Ws.

00:29:21.310 --> 00:29:23.750 align:middle line:84%
What we do is-- and because
what's coming out here,

00:29:23.750 --> 00:29:26.930 align:middle line:84%
these vectors are the same
length as what goes in,

00:29:26.930 --> 00:29:29.430 align:middle line:84%
we can just add them
element by element.

00:29:29.430 --> 00:29:31.870 align:middle line:84%
So we take the input
and we actually

00:29:31.870 --> 00:29:34.950 align:middle line:90%
add it to what comes out.

00:29:34.950 --> 00:29:37.350 align:middle line:90%
So why would we want to do that?

00:29:37.350 --> 00:29:40.150 align:middle line:84%
Why would we want to
go to a lot of trouble

00:29:40.150 --> 00:29:43.350 align:middle line:84%
to process this thing, and then
when it comes out, we literally

00:29:43.350 --> 00:29:45.550 align:middle line:90%
add up the original input?

00:29:45.550 --> 00:29:47.210 align:middle line:84%
What do you think
the intuition is?

00:29:47.210 --> 00:29:52.350 align:middle line:90%


00:29:52.350 --> 00:29:54.950 align:middle line:90%
So it turns out--

00:29:54.950 --> 00:29:56.030 align:middle line:90%
think of it this way.

00:29:56.030 --> 00:29:57.330 align:middle line:90%
You have a bunch of inputs.

00:29:57.330 --> 00:29:58.990 align:middle line:90%
You send it to a neural network.

00:29:58.990 --> 00:30:01.710 align:middle line:84%
It transforms it and
gives you something else.

00:30:01.710 --> 00:30:05.550 align:middle line:84%
At that point, you
might be thinking, well,

00:30:05.550 --> 00:30:08.870 align:middle line:84%
everything that happens in the
network from that point onward

00:30:08.870 --> 00:30:11.390 align:middle line:84%
can no longer see
your original input.

00:30:11.390 --> 00:30:14.070 align:middle line:84%
It can only work with the
transformed input, right?

00:30:14.070 --> 00:30:17.590 align:middle line:84%
But what if your
transformations are not great?

00:30:17.590 --> 00:30:20.470 align:middle line:84%
So as an insurance
policy, what you can do

00:30:20.470 --> 00:30:23.470 align:middle line:84%
is you can take the
transformed stuff

00:30:23.470 --> 00:30:28.130 align:middle line:84%
and you can take the original
stuff and send both in.

00:30:28.130 --> 00:30:29.693 align:middle line:90%
And this whole thing--

00:30:29.693 --> 00:30:30.610 align:middle line:90%
and you can Google it.

00:30:30.610 --> 00:30:33.090 align:middle line:84%
It's called a wide and deep
network and things like that.

00:30:33.090 --> 00:30:35.430 align:middle line:84%
But the whole point is
that, let's not lose

00:30:35.430 --> 00:30:37.050 align:middle line:90%
the original input anywhere.

00:30:37.050 --> 00:30:38.510 align:middle line:90%
Let's also send it along.

00:30:38.510 --> 00:30:40.295 align:middle line:84%
But if you keep adding
the original input

00:30:40.295 --> 00:30:41.670 align:middle line:84%
to every intermediate
layer, it's

00:30:41.670 --> 00:30:43.450 align:middle line:84%
going to get longer and
longer and longer and bigger,

00:30:43.450 --> 00:30:46.158 align:middle line:84%
which you don't want because you
want it all to be the same size.

00:30:46.158 --> 00:30:49.110 align:middle line:84%
So the simplest alternative
is to just add them up.

00:30:49.110 --> 00:30:51.770 align:middle line:84%
You take the transformed stuff
and you add the original input,

00:30:51.770 --> 00:30:54.610 align:middle line:90%
you get the same thing again.

00:30:54.610 --> 00:30:57.550 align:middle line:84%
What came in, W1 was
a 100-long vector,

00:30:57.550 --> 00:30:59.890 align:middle line:84%
and the transformed
version is also 100 long.

00:30:59.890 --> 00:31:02.210 align:middle line:84%
So just literally
under 100, add them up.

00:31:02.210 --> 00:31:02.910 align:middle line:90%
That's it.

00:31:02.910 --> 00:31:04.950 align:middle line:90%
You get another 100-long vector.

00:31:04.950 --> 00:31:08.150 align:middle line:84%
So that is what's called
a residual connection.

00:31:08.150 --> 00:31:12.470 align:middle line:84%
And as it turns out,
residual connections

00:31:12.470 --> 00:31:16.310 align:middle line:84%
improve the gradient flow during
backpropagation dramatically,

00:31:16.310 --> 00:31:18.250 align:middle line:84%
and that's why they
are very heavily used.

00:31:18.250 --> 00:31:22.390 align:middle line:84%
And in fact, ResNet, which we
looked at for computer vision,

00:31:22.390 --> 00:31:26.150 align:middle line:84%
it stands for Residual Net
because it was the first network

00:31:26.150 --> 00:31:28.928 align:middle line:90%
to actually figure this out.

00:31:28.928 --> 00:31:30.970 align:middle line:84%
This is not just a
transformer thing, by the way.

00:31:30.970 --> 00:31:34.790 align:middle line:84%
It's widely used in lots of
deep neural architectures.

00:31:34.790 --> 00:31:36.290 align:middle line:84%
The notion of a
residual connection,

00:31:36.290 --> 00:31:38.170 align:middle line:90%
that's what it means.

00:31:38.170 --> 00:31:41.870 align:middle line:84%
So we do a residual
connection, and then we

00:31:41.870 --> 00:31:45.590 align:middle line:84%
come to the final tweak, which
is called layer normalization.

00:31:45.590 --> 00:31:47.350 align:middle line:84%
So once we add the
residual connection,

00:31:47.350 --> 00:31:50.110 align:middle line:84%
we are going to do something
else here to these vectors

00:31:50.110 --> 00:31:52.270 align:middle line:90%
before they continue flowing.

00:31:52.270 --> 00:31:57.990 align:middle line:84%
And what layer normalization
does is it basically says that--

00:31:57.990 --> 00:32:00.330 align:middle line:84%
you will recall from the very
beginning of the semester,

00:32:00.330 --> 00:32:03.110 align:middle line:84%
I've been saying that whatever
comes into a neural network,

00:32:03.110 --> 00:32:04.670 align:middle line:84%
the inputs, let's
just really make

00:32:04.670 --> 00:32:06.128 align:middle line:84%
sure that they are
all in some sort

00:32:06.128 --> 00:32:07.710 align:middle line:90%
of a narrow, well-defined range.

00:32:07.710 --> 00:32:09.950 align:middle line:90%
They can't be in a big range.

00:32:09.950 --> 00:32:13.830 align:middle line:84%
So for pictures, for images,
we divided every number by 255

00:32:13.830 --> 00:32:18.150 align:middle line:84%
so that every little pixel
value is between 0 and 1.

00:32:18.150 --> 00:32:20.650 align:middle line:84%
For continuous things, like
the heart disease example,

00:32:20.650 --> 00:32:22.652 align:middle line:84%
we standardized
it by calculating

00:32:22.652 --> 00:32:24.110 align:middle line:84%
the mean and the
standard deviation

00:32:24.110 --> 00:32:25.767 align:middle line:84%
and subtracting the
mean and dividing

00:32:25.767 --> 00:32:26.850 align:middle line:90%
by the standard deviation.

00:32:26.850 --> 00:32:28.350 align:middle line:84%
So when you do that,
all the numbers

00:32:28.350 --> 00:32:31.590 align:middle line:84%
are going to roughly be in
the minus 1 to plus 1 range.

00:32:31.590 --> 00:32:35.690 align:middle line:84%
So in neural networks, for a
backprop to work really well,

00:32:35.690 --> 00:32:39.230 align:middle line:84%
you have to make sure that
no numbers get too big,

00:32:39.230 --> 00:32:42.070 align:middle line:84%
that all the numbers are always
in some sort of a narrow range.

00:32:42.070 --> 00:32:45.250 align:middle line:84%
So what layer normalization
does is to say,

00:32:45.250 --> 00:32:47.870 align:middle line:84%
you know what, whatever
is coming out here,

00:32:47.870 --> 00:32:50.508 align:middle line:84%
I want to make sure none of
these numbers are too big.

00:32:50.508 --> 00:32:53.050 align:middle line:84%
I want to make sure they're all
well-behaved in a small range

00:32:53.050 --> 00:32:55.110 align:middle line:84%
because if I don't
do that, backprop

00:32:55.110 --> 00:32:56.410 align:middle line:90%
is not going to work very well.

00:32:56.410 --> 00:32:59.512 align:middle line:90%


00:32:59.512 --> 00:33:01.470 align:middle line:84%
AUDIENCE: Is this what
we do to ensure we don't

00:33:01.470 --> 00:33:03.428 align:middle line:90%
have the problem of vanishing?

00:33:03.428 --> 00:33:04.470 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:33:04.470 --> 00:33:07.317 align:middle line:84%
So technically there
could be two problems.

00:33:07.317 --> 00:33:09.650 align:middle line:84%
There is an exploding gradient
and a vanishing gradient.

00:33:09.650 --> 00:33:10.370 align:middle line:90%
Both are bad.

00:33:10.370 --> 00:33:11.650 align:middle line:90%
This is a way to address it.

00:33:11.650 --> 00:33:14.110 align:middle line:84%
So you will find a whole
bunch of batch normalization

00:33:14.110 --> 00:33:17.090 align:middle line:84%
techniques, layer normalization,
batch normalization,

00:33:17.090 --> 00:33:18.190 align:middle line:90%
and so on and so forth.

00:33:18.190 --> 00:33:20.310 align:middle line:84%
All these are
methods to make sure

00:33:20.310 --> 00:33:22.130 align:middle line:84%
that these numbers
stay in a small range

00:33:22.130 --> 00:33:23.970 align:middle line:84%
so it doesn't cause
gradient issues later.

00:33:23.970 --> 00:33:27.950 align:middle line:90%


00:33:27.950 --> 00:33:31.750 align:middle line:90%
In particular, what we do is--

00:33:31.750 --> 00:33:34.543 align:middle line:84%
or what happens inside
this layer normalization--

00:33:34.543 --> 00:33:36.710 align:middle line:84%
is we just calculate the
mean and standard deviation

00:33:36.710 --> 00:33:40.230 align:middle line:84%
of every one of these embeddings
If you have, let's say,

00:33:40.230 --> 00:33:43.090 align:middle line:84%
6 embeddings here, we'll have 6
means and 6 standard deviations

00:33:43.090 --> 00:33:44.750 align:middle line:90%
for each one across the rows.

00:33:44.750 --> 00:33:47.595 align:middle line:84%
And then we standardize it,
meaning subtract the mean,

00:33:47.595 --> 00:33:48.970 align:middle line:84%
divide by the
standard deviation.

00:33:48.970 --> 00:33:51.030 align:middle line:84%
And when you do that,
all these things

00:33:51.030 --> 00:33:52.550 align:middle line:90%
are going to be nice and small.

00:33:52.550 --> 00:33:54.630 align:middle line:84%
And then we do this
little other thing

00:33:54.630 --> 00:33:57.790 align:middle line:84%
where we have introduced
two new parameters

00:33:57.790 --> 00:34:00.790 align:middle line:84%
to rescale it and move
it around a little bit

00:34:00.790 --> 00:34:03.270 align:middle line:84%
just because adding
more weights always

00:34:03.270 --> 00:34:06.130 align:middle line:84%
helps make these things
better, so we add them.

00:34:06.130 --> 00:34:08.550 align:middle line:84%
And this gets slightly
complicated because of the way

00:34:08.550 --> 00:34:11.469 align:middle line:84%
the dimensions work, so I'm not
going to spend much time on it.

00:34:11.469 --> 00:34:13.750 align:middle line:84%
And then what comes
out the other end

00:34:13.750 --> 00:34:15.590 align:middle line:84%
is a very well-behaved
set of numbers

00:34:15.590 --> 00:34:19.469 align:middle line:84%
and a nice and small
and narrow range.

00:34:19.469 --> 00:34:21.909 align:middle line:84%
So this is called
layer normalization.

00:34:21.909 --> 00:34:25.510 align:middle line:84%
You can see this link to
understand it a bit better.

00:34:25.510 --> 00:34:26.989 align:middle line:90%
And we do that as well.

00:34:26.989 --> 00:34:32.185 align:middle line:84%
So to put it all together, so
this is a transformer encoder

00:34:32.185 --> 00:34:34.310 align:middle line:84%
where we have this multi-head
attention layer where

00:34:34.310 --> 00:34:36.750 align:middle line:84%
each attention head
in the inside of it

00:34:36.750 --> 00:34:38.949 align:middle line:84%
is tunable with
those A matrices,

00:34:38.949 --> 00:34:41.010 align:middle line:84%
and then we have a
residual connection.

00:34:41.010 --> 00:34:42.830 align:middle line:84%
We do that, and then
we do layernorm.

00:34:42.830 --> 00:34:45.949 align:middle line:84%
And then we do the same thing
in the next feedforward layer

00:34:45.949 --> 00:34:47.820 align:middle line:84%
as well and then boom,
out pops the output.

00:34:47.820 --> 00:34:50.260 align:middle line:90%


00:34:50.260 --> 00:34:52.510 align:middle line:84%
AUDIENCE: By that definition,
the multi-head attention

00:34:52.510 --> 00:34:54.750 align:middle line:84%
layer [INAUDIBLE] and
everything, theoretically

00:34:54.750 --> 00:34:59.430 align:middle line:84%
I can add even the biases or
the hate speech aspects which

00:34:59.430 --> 00:35:01.650 align:middle line:90%
come in to take care of it.

00:35:01.650 --> 00:35:05.710 align:middle line:84%
So the model can account for the
fact that something is biased

00:35:05.710 --> 00:35:06.950 align:middle line:90%
or something is not.

00:35:06.950 --> 00:35:08.950 align:middle line:84%
RAMA RAMAKRISHNAN: The
thing is it's not so much

00:35:08.950 --> 00:35:10.650 align:middle line:90%
the model's accounting for it.

00:35:10.650 --> 00:35:12.710 align:middle line:84%
It is capturing
whatever patterns happen

00:35:12.710 --> 00:35:14.550 align:middle line:90%
to be inherent in the data.

00:35:14.550 --> 00:35:16.260 align:middle line:90%
It's capturing.

00:35:16.260 --> 00:35:18.337 align:middle line:84%
Now what you do with that
capture is up to you.

00:35:18.337 --> 00:35:20.670 align:middle line:84%
It depends on the actual
problem you're trying to solve.

00:35:20.670 --> 00:35:24.070 align:middle line:84%
In particular, it is going to
capture all the bad stuff, too.

00:35:24.070 --> 00:35:26.730 align:middle line:84%
Because if your training data
has a lot of biased stuff in it,

00:35:26.730 --> 00:35:29.450 align:middle line:84%
toxic things in it,
dangerous things in it,

00:35:29.450 --> 00:35:30.830 align:middle line:84%
it doesn't have
a sense of values

00:35:30.830 --> 00:35:34.990 align:middle line:84%
as to what is good or bad,
it's just going to pick it up.

00:35:34.990 --> 00:35:35.990 align:middle line:90%
Yes?

00:35:35.990 --> 00:35:38.150 align:middle line:84%
AUDIENCE: Picking on that,
then how do you actually

00:35:38.150 --> 00:35:40.350 align:middle line:84%
make it unlearn
those or how do you

00:35:40.350 --> 00:35:42.042 align:middle line:90%
mitigate the effect of those?

00:35:42.042 --> 00:35:44.250 align:middle line:84%
RAMA RAMAKRISHNAN: That's
a whole course unto itself,

00:35:44.250 --> 00:35:46.042 align:middle line:84%
but I'm happy to give
you pointers offline.

00:35:46.042 --> 00:35:48.490 align:middle line:90%


00:35:48.490 --> 00:35:50.270 align:middle line:90%
So this is what we have.

00:35:50.270 --> 00:35:52.590 align:middle line:84%
And remember what I
said, that this is just

00:35:52.590 --> 00:35:54.170 align:middle line:90%
a single transformer block.

00:35:54.170 --> 00:35:55.910 align:middle line:84%
And since what comes
in and what goes out

00:35:55.910 --> 00:35:57.630 align:middle line:84%
are the same
dimensions, we can just

00:35:57.630 --> 00:35:59.950 align:middle line:90%
stack them one after the other.

00:35:59.950 --> 00:36:02.450 align:middle line:90%
It's very stackable.

00:36:02.450 --> 00:36:04.670 align:middle line:84%
You can stack it vertically
as much as you want.

00:36:04.670 --> 00:36:08.390 align:middle line:84%
And as I mentioned, I think GPT3
has 96 of these things stacked

00:36:08.390 --> 00:36:09.410 align:middle line:90%
one on top of the other.

00:36:09.410 --> 00:36:14.190 align:middle line:90%


00:36:14.190 --> 00:36:17.470 align:middle line:84%
That is the transformer encoder,
and this exactly maps to that.

00:36:17.470 --> 00:36:19.810 align:middle line:84%
So basically the input
embeddings come in,

00:36:19.810 --> 00:36:21.790 align:middle line:84%
you add positional
embeddings, and then

00:36:21.790 --> 00:36:24.550 align:middle line:84%
you send it to, say, these
mini attention blocks.

00:36:24.550 --> 00:36:26.630 align:middle line:84%
And they all get added
up, and then it comes out

00:36:26.630 --> 00:36:27.750 align:middle line:90%
to the attention block.

00:36:27.750 --> 00:36:30.753 align:middle line:84%
You add-- the add and
norm here means-- "add"

00:36:30.753 --> 00:36:33.170 align:middle line:84%
means residual connection
because you're adding the input,

00:36:33.170 --> 00:36:36.350 align:middle line:84%
which is why you have this arrow
going from the input being added

00:36:36.350 --> 00:36:38.650 align:middle line:84%
there, and then you
normalize it, send it along,

00:36:38.650 --> 00:36:40.400 align:middle line:84%
and do it again, and
out comes the output.

00:36:40.400 --> 00:36:45.230 align:middle line:90%


00:36:45.230 --> 00:36:47.070 align:middle line:84%
Now just to be
very clear on what

00:36:47.070 --> 00:36:48.910 align:middle line:84%
is being optimized
during backpropagation

00:36:48.910 --> 00:36:54.550 align:middle line:84%
in this complex flow, now,
clearly the embeddings that you

00:36:54.550 --> 00:36:57.070 align:middle line:84%
started out with, both the
standalone embeddings as well

00:36:57.070 --> 00:36:59.870 align:middle line:84%
as the position
embeddings, those things

00:36:59.870 --> 00:37:01.492 align:middle line:90%
are going to get optimized.

00:37:01.492 --> 00:37:02.450 align:middle line:90%
Those are just weights.

00:37:02.450 --> 00:37:03.910 align:middle line:90%
They're going to get optimized.

00:37:03.910 --> 00:37:06.310 align:middle line:84%
Clearly everything inside
the transformer encoder block

00:37:06.310 --> 00:37:07.890 align:middle line:90%
is going to get normalized.

00:37:07.890 --> 00:37:09.130 align:middle line:90%
And what are they?

00:37:09.130 --> 00:37:12.170 align:middle line:84%
Well, they are the
AK, AQ, AV matrices

00:37:12.170 --> 00:37:14.150 align:middle line:90%
for each attention head.

00:37:14.150 --> 00:37:17.870 align:middle line:84%
Layer norm has
parameters as well.

00:37:17.870 --> 00:37:20.990 align:middle line:84%
The little feedforward
layer has weights as well.

00:37:20.990 --> 00:37:23.350 align:middle line:84%
All these things are
going to get optimized.

00:37:23.350 --> 00:37:25.380 align:middle line:84%
And then it goes
through this ReLU which,

00:37:25.380 --> 00:37:26.630 align:middle line:90%
again, has a bunch of weights.

00:37:26.630 --> 00:37:27.810 align:middle line:90%
That's going to get optimized.

00:37:27.810 --> 00:37:29.893 align:middle line:84%
And then the final softmax
has a bunch of weights.

00:37:29.893 --> 00:37:31.623 align:middle line:90%
That's going to get optimized.

00:37:31.623 --> 00:37:34.040 align:middle line:84%
All these things are going to
get optimized by a backprop.

00:37:34.040 --> 00:37:36.570 align:middle line:90%


00:37:36.570 --> 00:37:39.083 align:middle line:84%
So in that sense, you just
step back for a second

00:37:39.083 --> 00:37:40.250 align:middle line:90%
and look at the whole thing.

00:37:40.250 --> 00:37:43.370 align:middle line:84%
It is just a mathematical
model with a lot of parameters.

00:37:43.370 --> 00:37:45.870 align:middle line:84%
And we're just going to use a
gradient descent or stochastic

00:37:45.870 --> 00:37:46.995 align:middle line:90%
gradient descent optimizer.

00:37:46.995 --> 00:37:49.430 align:middle line:90%
That's it.

00:37:49.430 --> 00:37:51.290 align:middle line:90%
Yeah?

00:37:51.290 --> 00:37:53.990 align:middle line:84%
AUDIENCE: [INAUDIBLE]
we train the model.

00:37:53.990 --> 00:37:56.550 align:middle line:84%
Are we calculating
weights for each cell

00:37:56.550 --> 00:37:59.470 align:middle line:84%
of every possible matrix
based on the number

00:37:59.470 --> 00:38:02.590 align:middle line:84%
of inputs, every possible
dimension up to the max number

00:38:02.590 --> 00:38:04.523 align:middle line:90%
of inputs?

00:38:04.523 --> 00:38:06.190 align:middle line:84%
RAMA RAMAKRISHNAN:
Actually, the weights

00:38:06.190 --> 00:38:10.253 align:middle line:84%
themselves don't depend on how
long your input sentence is.

00:38:10.253 --> 00:38:11.670 align:middle line:84%
Because remember,
what we're doing

00:38:11.670 --> 00:38:14.230 align:middle line:84%
is for each sentence that comes
in-- let's say the sentence

00:38:14.230 --> 00:38:15.710 align:middle line:90%
has, say, three words.

00:38:15.710 --> 00:38:17.670 align:middle line:84%
There are three embeddings
for that sentence.

00:38:17.670 --> 00:38:22.670 align:middle line:84%
Each of those embeddings
gets multiplied by, say, AK.

00:38:22.670 --> 00:38:27.070 align:middle line:84%
So AK only needs to know
how long is each embedding.

00:38:27.070 --> 00:38:31.630 align:middle line:84%
It doesn't need to know,
how many words do I have.

00:38:31.630 --> 00:38:33.650 align:middle line:84%
And I'm glad you raised
that question, Ben,

00:38:33.650 --> 00:38:37.550 align:middle line:84%
because that's what makes the
transformers' number of weights

00:38:37.550 --> 00:38:42.110 align:middle line:84%
independent of the number
of words in your sentence.

00:38:42.110 --> 00:38:43.502 align:middle line:84%
It only depends
on the vocabulary

00:38:43.502 --> 00:38:45.710 align:middle line:84%
that you're going to work
with because the vocabulary

00:38:45.710 --> 00:38:47.810 align:middle line:84%
determines how many
embeddings you need,

00:38:47.810 --> 00:38:50.030 align:middle line:90%
how many embeddings you need.

00:38:50.030 --> 00:38:53.540 align:middle line:84%
The length only matters in terms
of the positional embedding

00:38:53.540 --> 00:38:55.580 align:middle line:84%
because if you have
1,000-long sentence,

00:38:55.580 --> 00:38:58.520 align:middle line:84%
you need 1,000-long
positional embedding matrix.

00:38:58.520 --> 00:39:02.280 align:middle line:84%
But beyond that,
it doesn't care.

00:39:02.280 --> 00:39:07.560 align:middle line:84%
And that's why, for example,
Google Gemini 1.5 Pro, which

00:39:07.560 --> 00:39:09.280 align:middle line:84%
can accommodate
basically 1 million long,

00:39:09.280 --> 00:39:14.520 align:middle line:84%
1 million token context window,
it's still very compute-heavy,

00:39:14.520 --> 00:39:19.080 align:middle line:84%
but it does not change
the number of parameters.

00:39:19.080 --> 00:39:20.940 align:middle line:90%
Yeah?

00:39:20.940 --> 00:39:24.680 align:middle line:84%
AUDIENCE: Essentially, which
weights are optimized first?

00:39:24.680 --> 00:39:27.305 align:middle line:84%
Are they in sequential
order or are they

00:39:27.305 --> 00:39:29.680 align:middle line:84%
optimizing the weights at the
very same time [INAUDIBLE].

00:39:29.680 --> 00:39:31.097 align:middle line:84%
RAMA RAMAKRISHNAN:
Simultaneously.

00:39:31.097 --> 00:39:33.100 align:middle line:84%
Because if you think
of backpropagation,

00:39:33.100 --> 00:39:35.000 align:middle line:84%
ultimately, you have
a loss function,

00:39:35.000 --> 00:39:37.560 align:middle line:84%
and you calculate the gradient
of that loss function.

00:39:37.560 --> 00:39:39.940 align:middle line:84%
So if you have, say,
1 billion parameters,

00:39:39.940 --> 00:39:43.098 align:middle line:84%
that gradient is basically
a 1-billion long vector.

00:39:43.098 --> 00:39:44.640 align:middle line:84%
And we're going to
take the gradient,

00:39:44.640 --> 00:39:49.240 align:middle line:84%
and we're going to do W new
equals W old minus alpha times

00:39:49.240 --> 00:39:50.400 align:middle line:90%
that gradient.

00:39:50.400 --> 00:39:53.520 align:middle line:84%
So all the Ws are going
to update instantaneously.

00:39:53.520 --> 00:39:55.480 align:middle line:84%
Now the way it actually
works in computation

00:39:55.480 --> 00:39:56.840 align:middle line:90%
is you're going to do it--

00:39:56.840 --> 00:39:58.540 align:middle line:84%
because of the back
and backpropagation,

00:39:58.540 --> 00:40:00.915 align:middle line:84%
it's going to start at the
end and slowly flow backwards,

00:40:00.915 --> 00:40:03.960 align:middle line:84%
but when it's done,
everything will be updated.

00:40:03.960 --> 00:40:05.933 align:middle line:90%
Yeah?

00:40:05.933 --> 00:40:09.800 align:middle line:84%
AUDIENCE: [INAUDIBLE] we
take two attention heads

00:40:09.800 --> 00:40:14.040 align:middle line:84%
and we have the matrices
of AK, AQ, and AV.

00:40:14.040 --> 00:40:17.218 align:middle line:84%
[INAUDIBLE] the parameters
of all three of them,

00:40:17.218 --> 00:40:18.760 align:middle line:84%
all the weights of
the three matrices

00:40:18.760 --> 00:40:21.460 align:middle line:84%
on this side and this side would
be different because finally,

00:40:21.460 --> 00:40:24.440 align:middle line:84%
the things you are inputting
from this side and the output

00:40:24.440 --> 00:40:25.008 align:middle line:90%
is same.

00:40:25.008 --> 00:40:26.716 align:middle line:84%
So the learning process
should be ideally

00:40:26.716 --> 00:40:29.280 align:middle line:84%
the same unlike like a
CNN, where we had put

00:40:29.280 --> 00:40:30.708 align:middle line:90%
filters which were different.

00:40:30.708 --> 00:40:32.000 align:middle line:90%
So what different thing we have

00:40:32.000 --> 00:40:34.482 align:middle line:84%
RAMA RAMAKRISHNAN: Because the
initialization is different.

00:40:34.482 --> 00:40:35.024 align:middle line:90%
AUDIENCE: Oh.

00:40:35.024 --> 00:40:35.732 align:middle line:90%
What do you mean?

00:40:35.732 --> 00:40:38.066 align:middle line:84%
RAMA RAMAKRISHNAN: What I
mean is if you have two heads,

00:40:38.066 --> 00:40:39.300 align:middle line:90%
each head has three matrices.

00:40:39.300 --> 00:40:42.360 align:middle line:84%
The starting values of those
six matrices are different.

00:40:42.360 --> 00:40:45.360 align:middle line:84%
AUDIENCE: If the starting
value of AK, AQ, and AV

00:40:45.360 --> 00:40:46.540 align:middle line:90%
is different on both heads.

00:40:46.540 --> 00:40:47.320 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:40:47.320 --> 00:40:49.487 align:middle line:84%
Much like for all the
weights, typically, the values

00:40:49.487 --> 00:40:51.480 align:middle line:90%
are randomly chosen.

00:40:51.480 --> 00:40:53.935 align:middle line:84%
If they were all the
same thing, you're right,

00:40:53.935 --> 00:40:55.060 align:middle line:90%
it won't make a difference.

00:40:55.060 --> 00:40:57.260 align:middle line:84%
But they will all
change the same way.

00:40:57.260 --> 00:40:57.760 align:middle line:90%
Yeah?

00:40:57.760 --> 00:41:00.320 align:middle line:90%


00:41:00.320 --> 00:41:03.360 align:middle line:84%
AUDIENCE: Is the input of
the transformer the sentence

00:41:03.360 --> 00:41:08.303 align:middle line:84%
or the array of the
embedding of each word?

00:41:08.303 --> 00:41:09.720 align:middle line:84%
RAMA RAMAKRISHNAN:
The transformer

00:41:09.720 --> 00:41:12.240 align:middle line:84%
itself is expecting
embeddings in.

00:41:12.240 --> 00:41:14.940 align:middle line:84%
And so what basically happens
is that we get some sentence,

00:41:14.940 --> 00:41:17.360 align:middle line:84%
we run it through a
tokenizer which connects it

00:41:17.360 --> 00:41:20.000 align:middle line:84%
to a bunch of tokens,
which are just integers,

00:41:20.000 --> 00:41:22.320 align:middle line:84%
and then it goes through
the embedding layer, which

00:41:22.320 --> 00:41:24.300 align:middle line:84%
maps the integers
to these embeddings,

00:41:24.300 --> 00:41:26.060 align:middle line:84%
and then you feed it
to the transformer.

00:41:26.060 --> 00:41:28.000 align:middle line:84%
But when you do
backpropagation, it

00:41:28.000 --> 00:41:30.560 align:middle line:84%
comes all the way back to
the starting embedding layer

00:41:30.560 --> 00:41:32.197 align:middle line:90%
and updates those weights.

00:41:32.197 --> 00:41:34.280 align:middle line:84%
AUDIENCE: So it can be
trainable so the embeddings

00:41:34.280 --> 00:41:36.500 align:middle line:84%
that we set at the beginning,
we see them as input here,

00:41:36.500 --> 00:41:37.420 align:middle line:90%
but they can be trained.

00:41:37.420 --> 00:41:39.378 align:middle line:84%
RAMA RAMAKRISHNAN: They
are trainable, exactly.

00:41:39.378 --> 00:41:40.160 align:middle line:90%
Exactly.

00:41:40.160 --> 00:41:41.680 align:middle line:90%
Yeah?

00:41:41.680 --> 00:41:44.280 align:middle line:84%
AUDIENCE: Are the attention
heads solely parallel or can

00:41:44.280 --> 00:41:46.620 align:middle line:84%
you have a stack
of attention heads?

00:41:46.620 --> 00:41:48.037 align:middle line:84%
RAMA RAMAKRISHNAN:
Typically, they

00:41:48.037 --> 00:41:50.080 align:middle line:84%
are parallelized
because you can always

00:41:50.080 --> 00:41:54.940 align:middle line:84%
stack the block itself to
get more and more power.

00:41:54.940 --> 00:41:59.800 align:middle line:84%
So now to apply the
transformer, common use cases

00:41:59.800 --> 00:42:02.720 align:middle line:84%
are that you have a whole
sentence that comes in,

00:42:02.720 --> 00:42:04.600 align:middle line:84%
and then you just
want to classify it,

00:42:04.600 --> 00:42:08.200 align:middle line:84%
the canonical thing being hey,
movie sentiment classification,

00:42:08.200 --> 00:42:09.700 align:middle line:84%
boom, positive or
negative right.

00:42:09.700 --> 00:42:10.800 align:middle line:90%
Classification.

00:42:10.800 --> 00:42:13.640 align:middle line:84%
Another common one is labeling,
where every word gets labeled

00:42:13.640 --> 00:42:16.080 align:middle line:84%
as a multiclass label,
and that's basically

00:42:16.080 --> 00:42:18.223 align:middle line:84%
what we saw with our
slot-filling problem.

00:42:18.223 --> 00:42:20.640 align:middle line:84%
And then there is another thing
called sequence generation

00:42:20.640 --> 00:42:21.848 align:middle line:90%
where you give it a sequence.

00:42:21.848 --> 00:42:24.300 align:middle line:84%
You want it to continue the
sequence, generate more stuff,

00:42:24.300 --> 00:42:26.640 align:middle line:84%
i.e. large language models
and all that good stuff.

00:42:26.640 --> 00:42:29.800 align:middle line:84%
So this we already know how to
do because we actually literally

00:42:29.800 --> 00:42:33.040 align:middle line:84%
built a Colab with this,
with the transformer stack.

00:42:33.040 --> 00:42:35.440 align:middle line:84%
Now the question is,
how can we do that?

00:42:35.440 --> 00:42:37.080 align:middle line:84%
How can we do basic
classification

00:42:37.080 --> 00:42:38.160 align:middle line:90%
with these things?

00:42:38.160 --> 00:42:41.240 align:middle line:84%
So now again, when you
send a sentence in,

00:42:41.240 --> 00:42:44.380 align:middle line:84%
after all that stuff is done,
and when I say, encoder, here,

00:42:44.380 --> 00:42:46.620 align:middle line:84%
I'm assuming that you
may have one block.

00:42:46.620 --> 00:42:48.505 align:middle line:84%
You may have 106
blocks, I don't care.

00:42:48.505 --> 00:42:50.380 align:middle line:84%
At the end of the day,
you send something in.

00:42:50.380 --> 00:42:52.255 align:middle line:84%
You get a bunch of
contextual embeddings out.

00:42:52.255 --> 00:42:55.040 align:middle line:90%


00:42:55.040 --> 00:42:58.120 align:middle line:84%
So at this point, we need to
take these contextual embeddings

00:42:58.120 --> 00:43:00.058 align:middle line:84%
and somehow make it
work for classification.

00:43:00.058 --> 00:43:02.100 align:middle line:84%
We are just classifying
something into yes or no,

00:43:02.100 --> 00:43:03.640 align:middle line:90%
positive or negative.

00:43:03.640 --> 00:43:06.080 align:middle line:84%
So it would be nice
if we can actually

00:43:06.080 --> 00:43:08.160 align:middle line:84%
take all these embeddings
and essentially

00:43:08.160 --> 00:43:12.760 align:middle line:84%
summarize them into a single
embedding, a single vector.

00:43:12.760 --> 00:43:14.500 align:middle line:84%
Because if we have
a single vector,

00:43:14.500 --> 00:43:16.180 align:middle line:84%
then we can run it
through maybe a ReLU,

00:43:16.180 --> 00:43:17.740 align:middle line:84%
and then we do a
sigmoid, and boom,

00:43:17.740 --> 00:43:19.940 align:middle line:84%
we can do a binary
classification problem.

00:43:19.940 --> 00:43:21.480 align:middle line:90%
Super easy.

00:43:21.480 --> 00:43:22.960 align:middle line:84%
So this begs the
question, OK, how

00:43:22.960 --> 00:43:25.600 align:middle line:84%
are we going to go from all
the many blue things to one

00:43:25.600 --> 00:43:26.380 align:middle line:90%
green thing?

00:43:26.380 --> 00:43:29.572 align:middle line:90%


00:43:29.572 --> 00:43:35.453 align:middle line:84%
Now of course, what we can do
is we can simply average them.

00:43:35.453 --> 00:43:36.870 align:middle line:84%
We can take each
of the embeddings

00:43:36.870 --> 00:43:39.260 align:middle line:84%
and just simply average
them element by element.

00:43:39.260 --> 00:43:42.700 align:middle line:90%
You'll get a nice green thing.

00:43:42.700 --> 00:43:44.740 align:middle line:84%
Any shortcomings
from doing that?

00:43:44.740 --> 00:43:48.440 align:middle line:90%


00:43:48.440 --> 00:43:51.995 align:middle line:84%
AUDIENCE: You would lose
the ordering of the words.

00:43:51.995 --> 00:43:53.620 align:middle line:84%
RAMA RAMAKRISHNAN:
Well, in some sense,

00:43:53.620 --> 00:43:56.000 align:middle line:84%
the positional embedding,
the positional encoding

00:43:56.000 --> 00:44:00.400 align:middle line:84%
you have in the input does
have this notion of position.

00:44:00.400 --> 00:44:03.760 align:middle line:84%
So you're not necessarily
losing the order necessarily,

00:44:03.760 --> 00:44:06.920 align:middle line:84%
but you're sort of averaging all
this information into something,

00:44:06.920 --> 00:44:08.870 align:middle line:84%
and averaging is going
to lose some richness.

00:44:08.870 --> 00:44:15.425 align:middle line:90%


00:44:15.425 --> 00:44:17.800 align:middle line:84%
AUDIENCE: I think it's going
to be skewed to the one that

00:44:17.800 --> 00:44:19.800 align:middle line:90%
has the biggest number, so--

00:44:19.800 --> 00:44:20.800 align:middle line:90%
RAMA RAMAKRISHNAN: True.

00:44:20.800 --> 00:44:22.660 align:middle line:84%
AUDIENCE: [INAUDIBLE]
is influencing your--

00:44:22.660 --> 00:44:23.320 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, the biggest ones

00:44:23.320 --> 00:44:24.237 align:middle line:90%
are going to dominate.

00:44:24.237 --> 00:44:26.160 align:middle line:84%
But hopefully we won't
have too much of that

00:44:26.160 --> 00:44:28.342 align:middle line:84%
because all the layernorm
business in the beginning

00:44:28.342 --> 00:44:30.800 align:middle line:84%
has hopefully made sure the
numbers are all in a reasonably

00:44:30.800 --> 00:44:32.280 align:middle line:90%
small and well-behaved range.

00:44:32.280 --> 00:44:33.840 align:middle line:84%
But the point really
is that you're

00:44:33.840 --> 00:44:35.840 align:middle line:84%
going to lose richness
in the information

00:44:35.840 --> 00:44:38.360 align:middle line:84%
because you're just
pssht, mushing it down.

00:44:38.360 --> 00:44:41.160 align:middle line:84%
So there's a much
better and more elegant

00:44:41.160 --> 00:44:44.680 align:middle line:84%
way to do this,
which is that what

00:44:44.680 --> 00:44:48.620 align:middle line:84%
you do is for every
sentence, when you train it,

00:44:48.620 --> 00:44:52.880 align:middle line:84%
you add an artificial token
called the class token.

00:44:52.880 --> 00:44:54.680 align:middle line:84%
Literally, it's an
artificial token

00:44:54.680 --> 00:44:58.520 align:middle line:84%
and it's designated as
CLS in the literature.

00:44:58.520 --> 00:45:04.440 align:middle line:84%
And then this token is getting
trained with everything else.

00:45:04.440 --> 00:45:08.520 align:middle line:84%
And so once you finish
training, that token

00:45:08.520 --> 00:45:10.640 align:middle line:90%
has its own embedding, too.

00:45:10.640 --> 00:45:13.980 align:middle line:84%
And because it has been
trained with everything else--

00:45:13.980 --> 00:45:16.340 align:middle line:84%
and this token, remember,
is a contextual embedding,

00:45:16.340 --> 00:45:18.840 align:middle line:84%
which means that it's very much
aware of all the other words

00:45:18.840 --> 00:45:21.080 align:middle line:90%
in the sentence.

00:45:21.080 --> 00:45:25.280 align:middle line:84%
So in some sense, this CLS
token's contextual embedding

00:45:25.280 --> 00:45:29.340 align:middle line:84%
captures everything that's
going on about that sentence.

00:45:29.340 --> 00:45:31.820 align:middle line:84%
And so what we do is once
we are done training,

00:45:31.820 --> 00:45:34.040 align:middle line:84%
we just grab this
thing alone and then

00:45:34.040 --> 00:45:38.920 align:middle line:84%
send that through a ReLU and a
sigmoid and boom, you're done.

00:45:38.920 --> 00:45:43.425 align:middle line:84%
So this is a very clever trick
to somehow, instead of averaging

00:45:43.425 --> 00:45:44.800 align:middle line:84%
everything at the
end, let's just

00:45:44.800 --> 00:45:47.300 align:middle line:84%
have something just for the
whole thing, the sentence,

00:45:47.300 --> 00:45:50.080 align:middle line:84%
and just learn it anyway
along with everything else.

00:45:50.080 --> 00:45:51.560 align:middle line:84%
A meta principle
in deep learning

00:45:51.560 --> 00:45:53.770 align:middle line:84%
is that whenever
you think you're

00:45:53.770 --> 00:45:55.520 align:middle line:84%
making an ad hoc
decision about something,

00:45:55.520 --> 00:45:58.220 align:middle line:84%
like averaging a bunch of stuff,
you should always stop and say,

00:45:58.220 --> 00:45:59.720 align:middle line:84%
is there a better
way to do it where

00:45:59.720 --> 00:46:01.920 align:middle line:84%
it doesn't have to be ad
hoc, where the right way is

00:46:01.920 --> 00:46:05.760 align:middle line:84%
learnable from the data
directly using backpropagation?

00:46:05.760 --> 00:46:06.740 align:middle line:90%
There was a hand.

00:46:06.740 --> 00:46:08.400 align:middle line:90%
Yeah?

00:46:08.400 --> 00:46:13.900 align:middle line:84%
AUDIENCE: Is there a reason that
you added the CLS at the start?

00:46:13.900 --> 00:46:14.780 align:middle line:90%
Why not earlier?

00:46:14.780 --> 00:46:15.000 align:middle line:90%
[INAUDIBLE]

00:46:15.000 --> 00:46:15.400 align:middle line:84%
RAMA RAMAKRISHNAN: You
can do it that way.

00:46:15.400 --> 00:46:17.317 align:middle line:84%
AUDIENCE: At the end,
is there any difference?

00:46:17.317 --> 00:46:19.800 align:middle line:84%
RAMA RAMAKRISHNAN: The only
thing to remember is that--

00:46:19.800 --> 00:46:20.700 align:middle line:90%
it's a good question.

00:46:20.700 --> 00:46:23.283 align:middle line:84%
So different sentences are
going to be of different length.

00:46:23.283 --> 00:46:24.700 align:middle line:84%
So there might be
short sentences.

00:46:24.700 --> 00:46:25.950 align:middle line:90%
There might be long sentences.

00:46:25.950 --> 00:46:27.440 align:middle line:84%
In particular, the
short sentences

00:46:27.440 --> 00:46:30.185 align:middle line:90%
are going to get padded.

00:46:30.185 --> 00:46:31.560 align:middle line:84%
I remember I talked
about padding

00:46:31.560 --> 00:46:33.400 align:middle line:90%
to make it to fit to one length.

00:46:33.400 --> 00:46:35.200 align:middle line:84%
So what internally the
transformer will do

00:46:35.200 --> 00:46:37.980 align:middle line:84%
is it will ignore all the padded
tokens because it doesn't do it.

00:46:37.980 --> 00:46:38.860 align:middle line:90%
It's just padding.

00:46:38.860 --> 00:46:40.460 align:middle line:84%
It doesn't really
matter for anything.

00:46:40.460 --> 00:46:42.360 align:middle line:84%
So if you have the
CLS at the very end,

00:46:42.360 --> 00:46:44.880 align:middle line:84%
we have to have much more
administrative bookkeeping

00:46:44.880 --> 00:46:48.200 align:middle line:84%
to take everything
but the last one,

00:46:48.200 --> 00:46:50.120 align:middle line:84%
ignore it, and only
do the last one.

00:46:50.120 --> 00:46:52.440 align:middle line:84%
It's just much easier to
stick it in the beginning.

00:46:52.440 --> 00:46:53.220 align:middle line:90%
That's the reason.

00:46:53.220 --> 00:46:54.960 align:middle line:90%
Yeah?

00:46:54.960 --> 00:46:57.520 align:middle line:84%
AUDIENCE: What could be just a
practical application of this?

00:46:57.520 --> 00:46:59.340 align:middle line:84%
Would it be something
like sentiment analysis--

00:46:59.340 --> 00:46:59.560 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:46:59.560 --> 00:47:00.520 align:middle line:84%
AUDIENCE: --like
positive or negative?

00:47:00.520 --> 00:47:00.680 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:47:00.680 --> 00:47:02.640 align:middle line:84%
So basically any
kind of text comes in

00:47:02.640 --> 00:47:05.160 align:middle line:84%
and you want to figure
out some labeling problem,

00:47:05.160 --> 00:47:06.720 align:middle line:90%
like a classification problem.

00:47:06.720 --> 00:47:09.840 align:middle line:84%
The easiest example I could
think of was sentiment.

00:47:09.840 --> 00:47:12.000 align:middle line:84%
But you can imagine,
for example, an email

00:47:12.000 --> 00:47:14.420 align:middle line:84%
comes into a call
center operation,

00:47:14.420 --> 00:47:16.920 align:middle line:84%
and you want to take the email
and automatically figure out,

00:47:16.920 --> 00:47:18.462 align:middle line:84%
which department
should I send it to.

00:47:18.462 --> 00:47:23.040 align:middle line:90%


00:47:23.040 --> 00:47:26.542 align:middle line:84%
Now if the input data for a
task is natural language text,

00:47:26.542 --> 00:47:28.000 align:middle line:84%
we don't have to
restrict ourselves

00:47:28.000 --> 00:47:30.990 align:middle line:84%
to only the input
training data we have.

00:47:30.990 --> 00:47:33.240 align:middle line:84%
Wouldn't it be great to learn
from all the text that's

00:47:33.240 --> 00:47:34.483 align:middle line:90%
out there?

00:47:34.483 --> 00:47:36.400 align:middle line:84%
So for example, to go
back to that call center

00:47:36.400 --> 00:47:40.800 align:middle line:84%
thing I just mentioned, let's
say it's coming in English.

00:47:40.800 --> 00:47:43.880 align:middle line:84%
The ability to take that
English email and route it

00:47:43.880 --> 00:47:46.080 align:middle line:84%
to one of 10 things,
you shouldn't

00:47:46.080 --> 00:47:48.863 align:middle line:84%
have to learn English just for
your call center application.

00:47:48.863 --> 00:47:50.280 align:middle line:84%
You should learn
English generally

00:47:50.280 --> 00:47:51.760 align:middle line:90%
and use it for other things.

00:47:51.760 --> 00:47:55.692 align:middle line:84%
So why can't we just learn from
all the text that's out there?

00:47:55.692 --> 00:47:57.400 align:middle line:84%
And so that brings us
to something called

00:47:57.400 --> 00:47:59.280 align:middle line:90%
self-supervised learning.

00:47:59.280 --> 00:48:01.440 align:middle line:84%
And the idea of self-supervised
learning is this.

00:48:01.440 --> 00:48:03.680 align:middle line:84%
So if you recall the
transfer learning example

00:48:03.680 --> 00:48:08.220 align:middle line:84%
from lecture four where we had
ResNet, and we took ResNet,

00:48:08.220 --> 00:48:11.920 align:middle line:84%
we chopped off the final
thing, we made it headless,

00:48:11.920 --> 00:48:14.560 align:middle line:84%
and then we attached that
output of the headless ResNet

00:48:14.560 --> 00:48:17.300 align:middle line:84%
to a little hidden
layer and output,

00:48:17.300 --> 00:48:18.803 align:middle line:84%
and we did the
handbags and shoes.

00:48:18.803 --> 00:48:20.720 align:middle line:84%
And you will recall that
we were able to build

00:48:20.720 --> 00:48:22.595 align:middle line:84%
a very good classifier
for handbags and shoes

00:48:22.595 --> 00:48:24.280 align:middle line:90%
with just 100 examples.

00:48:24.280 --> 00:48:27.440 align:middle line:84%
So the question is, why
was this so effective?

00:48:27.440 --> 00:48:29.120 align:middle line:90%
Why was it so effective?

00:48:29.120 --> 00:48:32.880 align:middle line:84%
And turns out the reason why
any of this stuff actually works

00:48:32.880 --> 00:48:36.400 align:middle line:84%
is because neural networks,
they learn representations

00:48:36.400 --> 00:48:38.000 align:middle line:84%
automatically when
you train them.

00:48:38.000 --> 00:48:40.798 align:middle line:84%
So what I mean by that is
when you imagine a network,

00:48:40.798 --> 00:48:43.340 align:middle line:84%
you feed in a bunch of stuff,
it goes through all the layers,

00:48:43.340 --> 00:48:46.320 align:middle line:84%
it comes out, you can
think of each layer

00:48:46.320 --> 00:48:49.640 align:middle line:84%
as transforming the raw input
in some different alternate

00:48:49.640 --> 00:48:52.495 align:middle line:90%
representation of the input.

00:48:52.495 --> 00:48:54.100 align:middle line:84%
And these are called
representations.

00:48:54.100 --> 00:48:56.400 align:middle line:84%
That's actually
a technical term.

00:48:56.400 --> 00:48:59.440 align:middle line:84%
And so from this perspective,
when you train a neural network,

00:48:59.440 --> 00:49:01.960 align:middle line:84%
a deep network with lots of
layers, what you're really

00:49:01.960 --> 00:49:07.560 align:middle line:84%
learning is you're learning
how to represent the input

00:49:07.560 --> 00:49:08.575 align:middle line:90%
in many different ways--

00:49:08.575 --> 00:49:10.200 align:middle line:84%
each of these arrows
is a different way

00:49:10.200 --> 00:49:11.420 align:middle line:90%
of representing things--

00:49:11.420 --> 00:49:14.113 align:middle line:84%
plus you're learning a
final regression model,

00:49:14.113 --> 00:49:16.780 align:middle line:84%
either a linear regression model
or a logistic regression model.

00:49:16.780 --> 00:49:18.363 align:middle line:84%
Fundamentally, that's
what's going on.

00:49:18.363 --> 00:49:20.340 align:middle line:84%
Because the final layers
tend to be sigmoid,

00:49:20.340 --> 00:49:22.280 align:middle line:90%
softmax or just linear, right.

00:49:22.280 --> 00:49:23.580 align:middle line:90%
So the final layer--

00:49:23.580 --> 00:49:26.512 align:middle line:84%
if you just look at this part
alone, whatever is coming in

00:49:26.512 --> 00:49:28.720 align:middle line:84%
is just going through
essentially a linear regression

00:49:28.720 --> 00:49:30.380 align:middle line:84%
model or a logistic
regression model.

00:49:30.380 --> 00:49:31.450 align:middle line:90%
That's it.

00:49:31.450 --> 00:49:33.450 align:middle line:84%
So fundamentally you're
learning representations

00:49:33.450 --> 00:49:35.598 align:middle line:90%
and a final little model.

00:49:35.598 --> 00:49:37.890 align:middle line:84%
But the reason why all these
things work so much better

00:49:37.890 --> 00:49:40.290 align:middle line:84%
than logistic regression is
because those representations

00:49:40.290 --> 00:49:43.570 align:middle line:84%
have learned all kinds of useful
things about the input data.

00:49:43.570 --> 00:49:47.410 align:middle line:84%
They have sort of' automatically
feature-engineered for you.

00:49:47.410 --> 00:49:52.570 align:middle line:84%
So from this perspective, you
can imagine that each layer here

00:49:52.570 --> 00:49:53.670 align:middle line:90%
is like an encoder.

00:49:53.670 --> 00:49:55.350 align:middle line:90%
It encodes the input.

00:49:55.350 --> 00:49:56.510 align:middle line:90%
The first layer encodes it.

00:49:56.510 --> 00:49:57.950 align:middle line:84%
The first two layers
encode something.

00:49:57.950 --> 00:49:59.190 align:middle line:84%
The first three layers
encode something,

00:49:59.190 --> 00:50:01.010 align:middle line:84%
and so on and so forth,
so a deep network

00:50:01.010 --> 00:50:03.090 align:middle line:90%
contains many encoders.

00:50:03.090 --> 00:50:06.570 align:middle line:84%
And so the question is, what do
these representations actually

00:50:06.570 --> 00:50:07.370 align:middle line:90%
embody?

00:50:07.370 --> 00:50:08.550 align:middle line:90%
What do they capture?

00:50:08.550 --> 00:50:11.250 align:middle line:84%
Is it specific knowledge
about the particular problem

00:50:11.250 --> 00:50:14.210 align:middle line:90%
that you train the network on?

00:50:14.210 --> 00:50:16.390 align:middle line:84%
Or is it general knowledge
about the input data?

00:50:16.390 --> 00:50:18.590 align:middle line:84%
Because if it is general
knowledge about the input,

00:50:18.590 --> 00:50:20.410 align:middle line:84%
we can use it to
solve other problems,

00:50:20.410 --> 00:50:23.290 align:middle line:84%
unrelated problems, so
is it specific knowledge

00:50:23.290 --> 00:50:24.350 align:middle line:90%
or general knowledge.

00:50:24.350 --> 00:50:26.210 align:middle line:84%
And it turns out
they actually capture

00:50:26.210 --> 00:50:28.490 align:middle line:84%
a lot of general
knowledge about the input.

00:50:28.490 --> 00:50:31.890 align:middle line:84%
And that's why you can
get reuse out of them.

00:50:31.890 --> 00:50:34.050 align:middle line:84%
You can reuse them for
other unrelated things

00:50:34.050 --> 00:50:35.830 align:middle line:84%
because they have
captured general stuff.

00:50:35.830 --> 00:50:36.830 align:middle line:90%
So if you look at this--

00:50:36.830 --> 00:50:38.570 align:middle line:90%
I think I've shown you before.

00:50:38.570 --> 00:50:41.410 align:middle line:84%
If you look at a network that
classifies everyday objects

00:50:41.410 --> 00:50:42.890 align:middle line:84%
into a bunch of
categories, it can

00:50:42.890 --> 00:50:45.307 align:middle line:84%
learn all these little patterns
in the beginning and later

00:50:45.307 --> 00:50:46.570 align:middle line:90%
on and so on and so forth.

00:50:46.570 --> 00:50:48.230 align:middle line:84%
And this is a
face-detection network.

00:50:48.230 --> 00:50:51.890 align:middle line:84%
It has learned how to
identify little circles

00:50:51.890 --> 00:50:54.610 align:middle line:84%
and edges and nose-like
shapes and finally faces.

00:50:54.610 --> 00:50:57.130 align:middle line:84%
So all these things are
examples of representation

00:50:57.130 --> 00:50:59.796 align:middle line:84%
learning interesting
things about the input.

00:50:59.796 --> 00:51:02.410 align:middle line:84%
So since these
representations are capturing

00:51:02.410 --> 00:51:06.530 align:middle line:84%
intrinsic aspects of the data,
you can use it for other things.

00:51:06.530 --> 00:51:08.490 align:middle line:84%
You can take a face
detection neural network

00:51:08.490 --> 00:51:12.970 align:middle line:84%
and reuse it for emotion
detection, for instance.

00:51:12.970 --> 00:51:14.530 align:middle line:84%
So the question is
if we can somehow

00:51:14.530 --> 00:51:17.770 align:middle line:84%
get an encoder that generates
good representations

00:51:17.770 --> 00:51:19.638 align:middle line:84%
for your input
data, we can simply

00:51:19.638 --> 00:51:21.930 align:middle line:84%
build a regression model with
those as input and labels

00:51:21.930 --> 00:51:23.030 align:middle line:90%
as output and be done.

00:51:23.030 --> 00:51:25.450 align:middle line:84%
And this is exactly
what we did with ResNet

00:51:25.450 --> 00:51:27.050 align:middle line:90%
for handbags and shoes.

00:51:27.050 --> 00:51:28.730 align:middle line:84%
We found a thing
that had already

00:51:28.730 --> 00:51:32.210 align:middle line:84%
been trained on similar everyday
objects, everyday images.

00:51:32.210 --> 00:51:34.530 align:middle line:84%
And the key insight
here is that since we

00:51:34.530 --> 00:51:37.010 align:middle line:84%
don't have to spend
precious data on learning

00:51:37.010 --> 00:51:41.730 align:middle line:84%
these good representations, we
won't need as much labeled data

00:51:41.730 --> 00:51:45.690 align:middle line:84%
in the first place because the
pre-training used a lot of data,

00:51:45.690 --> 00:51:47.730 align:middle line:84%
and you're piggybacking
on that data.

00:51:47.730 --> 00:51:49.410 align:middle line:84%
So in some sense,
your training data

00:51:49.410 --> 00:51:51.090 align:middle line:84%
is everything that
the pre-trained model

00:51:51.090 --> 00:51:53.200 align:middle line:84%
was trained on plus your
little 200 examples.

00:51:53.200 --> 00:51:55.990 align:middle line:90%


00:51:55.990 --> 00:51:56.950 align:middle line:90%
So this is what we did.

00:51:56.950 --> 00:51:59.570 align:middle line:84%
We used headless ResNet as an
encoder that can take raw input

00:51:59.570 --> 00:52:02.170 align:middle line:84%
and transform it into
useful representations.

00:52:02.170 --> 00:52:03.750 align:middle line:90%
This is what we did.

00:52:03.750 --> 00:52:06.530 align:middle line:84%
So the general approach is that
you find a deep neural network

00:52:06.530 --> 00:52:09.610 align:middle line:84%
built on similar inputs
but different outputs,

00:52:09.610 --> 00:52:11.850 align:middle line:84%
and then you
basically grab maybe

00:52:11.850 --> 00:52:15.410 align:middle line:84%
the penultimate representation
or the one before that.

00:52:15.410 --> 00:52:17.170 align:middle line:90%
Then you chop off the head.

00:52:17.170 --> 00:52:21.730 align:middle line:84%
You attach your own output
head, train the whole thing--

00:52:21.730 --> 00:52:24.530 align:middle line:84%
just the final layer or train
the whole thing if you want.

00:52:24.530 --> 00:52:26.630 align:middle line:84%
This is like the playbook
we followed for ResNet.

00:52:26.630 --> 00:52:29.290 align:middle line:84%
The same thing works for all
kinds of other data types

00:52:29.290 --> 00:52:30.250 align:middle line:90%
as well.

00:52:30.250 --> 00:52:33.410 align:middle line:84%
So now to build such a
model, we need labeled data.

00:52:33.410 --> 00:52:36.010 align:middle line:84%
We were lucky because ResNet
was actually trained on ImageNet

00:52:36.010 --> 00:52:38.070 align:middle line:84%
data, which is like
1 million images,

00:52:38.070 --> 00:52:40.330 align:middle line:84%
each of which labeled
into 1,000 categories,

00:52:40.330 --> 00:52:42.850 align:middle line:90%
which is very convenient for us.

00:52:42.850 --> 00:52:47.450 align:middle line:84%
But what if you want to build a
generally useful model for text

00:52:47.450 --> 00:52:49.592 align:middle line:90%
data?

00:52:49.592 --> 00:52:51.550 align:middle line:84%
Clearly, we need to
collect a lot of text data,

00:52:51.550 --> 00:52:52.810 align:middle line:84%
but that's no problem
because the internet

00:52:52.810 --> 00:52:53.890 align:middle line:90%
is full of text data.

00:52:53.890 --> 00:52:55.390 align:middle line:84%
We can easily
scrape the internet.

00:52:55.390 --> 00:52:56.830 align:middle line:90%
We can just download Wikipedia.

00:52:56.830 --> 00:52:58.350 align:middle line:90%
So that's not a problem.

00:52:58.350 --> 00:52:59.850 align:middle line:84%
The problem is
something else, which

00:52:59.850 --> 00:53:04.930 align:middle line:84%
is that, how do we define an
input label for a piece of text?

00:53:04.930 --> 00:53:07.590 align:middle line:84%
So for an input sentence, what
should the output label be?

00:53:07.590 --> 00:53:08.843 align:middle line:90%
That's the key question.

00:53:08.843 --> 00:53:10.510 align:middle line:84%
Because if you can
answer this question,

00:53:10.510 --> 00:53:12.135 align:middle line:84%
you can just pre-train
all these things

00:53:12.135 --> 00:53:14.214 align:middle line:90%
on all kinds of text data.

00:53:14.214 --> 00:53:17.412 align:middle line:84%
So a beautiful
idea for doing this

00:53:17.412 --> 00:53:18.870 align:middle line:84%
is called
self-supervised learning.

00:53:18.870 --> 00:53:21.370 align:middle line:84%
And the key idea is that you
take your input, whatever

00:53:21.370 --> 00:53:25.410 align:middle line:84%
the input is, you take a
small part of the input

00:53:25.410 --> 00:53:27.970 align:middle line:84%
and just remove it, and
then ask your network

00:53:27.970 --> 00:53:31.970 align:middle line:84%
to fill in the blanks
from everything else.

00:53:31.970 --> 00:53:33.737 align:middle line:84%
So this is called
masking, and it's

00:53:33.737 --> 00:53:36.070 align:middle line:84%
just one of many techniques
in self-supervised learning,

00:53:36.070 --> 00:53:37.770 align:middle line:90%
but this is very commonly used.

00:53:37.770 --> 00:53:40.650 align:middle line:90%
So this is the original input.

00:53:40.650 --> 00:53:42.470 align:middle line:84%
And then you take
it and then you just

00:53:42.470 --> 00:53:43.930 align:middle line:84%
take this thing
in the middle here

00:53:43.930 --> 00:53:47.330 align:middle line:84%
randomly and zero
it out or mask it.

00:53:47.330 --> 00:53:51.270 align:middle line:84%
And so this incomplete
input is your now new input,

00:53:51.270 --> 00:53:56.690 align:middle line:84%
and the thing that you took
out becomes your fake label.

00:53:56.690 --> 00:54:00.210 align:middle line:84%
So you can almost imagine
if you're baking donuts,

00:54:00.210 --> 00:54:02.530 align:middle line:84%
you make a donut, and
then you punch a hole

00:54:02.530 --> 00:54:04.250 align:middle line:90%
in the middle of the donut.

00:54:04.250 --> 00:54:07.110 align:middle line:84%
The donut with the
hole is your new input.

00:54:07.110 --> 00:54:08.900 align:middle line:90%
The munchkin is the label.

00:54:08.900 --> 00:54:11.418 align:middle line:90%


00:54:11.418 --> 00:54:13.210 align:middle line:84%
Am I making everybody
hungry at this point?

00:54:13.210 --> 00:54:15.130 align:middle line:90%
[LAUGHTER]

00:54:15.130 --> 00:54:17.670 align:middle line:90%
So once you do that, no problem.

00:54:17.670 --> 00:54:18.630 align:middle line:90%
You have an input.

00:54:18.630 --> 00:54:19.990 align:middle line:90%
You have labels.

00:54:19.990 --> 00:54:25.450 align:middle line:84%
You just train a neural network
to essentially predict those--

00:54:25.450 --> 00:54:28.490 align:middle line:90%
to basically fill in the blanks.

00:54:28.490 --> 00:54:31.690 align:middle line:84%
And so for example, if
you take a sentence like,

00:54:31.690 --> 00:54:34.290 align:middle line:84%
"The Sloan School's mission,"
you can just go in there

00:54:34.290 --> 00:54:38.232 align:middle line:84%
and just knock out randomly
a bunch of words like--

00:54:38.232 --> 00:54:40.690 align:middle line:84%
the ones I'm knocking out, I'm
just putting the word "mask"

00:54:40.690 --> 00:54:42.850 align:middle line:84%
in it just to show
what I'm doing.

00:54:42.850 --> 00:54:45.910 align:middle line:84%
And then given this
sentence, it will

00:54:45.910 --> 00:54:47.820 align:middle line:84%
try to fill in the
blanks with actual words.

00:54:47.820 --> 00:54:51.810 align:middle line:90%


00:54:51.810 --> 00:54:53.088 align:middle line:90%
So now for the amazing part.

00:54:53.088 --> 00:54:55.130 align:middle line:84%
In the process of learning
to fill in the blanks,

00:54:55.130 --> 00:54:58.050 align:middle line:84%
the network learns a
really good representation

00:54:58.050 --> 00:55:00.370 align:middle line:84%
of the kind of input
data it's seeing.

00:55:00.370 --> 00:55:02.610 align:middle line:84%
And it makes sense
because if I give you

00:55:02.610 --> 00:55:04.690 align:middle line:84%
a sentence with a
few missing blanks

00:55:04.690 --> 00:55:07.425 align:middle line:84%
and you're able to very
successfully fill in the blanks,

00:55:07.425 --> 00:55:09.050 align:middle line:84%
you have learned a
whole bunch of stuff

00:55:09.050 --> 00:55:11.650 align:middle line:84%
about the world to
be able to do that.

00:55:11.650 --> 00:55:13.930 align:middle line:84%
If I say the capital
of France is dash,

00:55:13.930 --> 00:55:16.770 align:middle line:84%
and you're like, Paris,
OK, how did you know that?

00:55:16.770 --> 00:55:18.133 align:middle line:90%
It's sort of like that.

00:55:18.133 --> 00:55:19.550 align:middle line:84%
By learning to
fill in the blanks,

00:55:19.550 --> 00:55:22.470 align:middle line:84%
you really have to learn
how all these things work,

00:55:22.470 --> 00:55:24.330 align:middle line:84%
all the connections
between various words

00:55:24.330 --> 00:55:27.250 align:middle line:90%
and so on and so forth.

00:55:27.250 --> 00:55:30.430 align:middle line:84%
So what you can do is once
we build such a model,

00:55:30.430 --> 00:55:33.270 align:middle line:84%
we can just extract
an encoder from it,

00:55:33.270 --> 00:55:37.050 align:middle line:84%
and then we'll fine-tune like
we did for transfer learning.

00:55:37.050 --> 00:55:38.970 align:middle line:84%
But this is how you
build a generic,

00:55:38.970 --> 00:55:43.530 align:middle line:84%
a generic pre-trained
model on unlabeled data.

00:55:43.530 --> 00:55:46.090 align:middle line:84%
And so we can use a
transformer encoder

00:55:46.090 --> 00:55:47.835 align:middle line:84%
to build this whole
thing in the middle.

00:55:47.835 --> 00:55:49.210 align:middle line:84%
Because remember,
the transformer

00:55:49.210 --> 00:55:52.170 align:middle line:84%
can take any sentence and give
you the same-size sentence

00:55:52.170 --> 00:55:54.850 align:middle line:84%
back along with
predictions for everything,

00:55:54.850 --> 00:55:57.250 align:middle line:84%
so we can just have it
take this thing in and ask

00:55:57.250 --> 00:55:59.230 align:middle line:84%
it to just predict all
the missing words here.

00:55:59.230 --> 00:56:03.090 align:middle line:90%


00:56:03.090 --> 00:56:06.210 align:middle line:84%
So to put it in other words,
masked self-supervised learning

00:56:06.210 --> 00:56:09.410 align:middle line:84%
is just a sequence
labeling problem.

00:56:09.410 --> 00:56:11.850 align:middle line:84%
So basically this is the
sequence that comes in,

00:56:11.850 --> 00:56:13.890 align:middle line:84%
and then you tell
the transformer--

00:56:13.890 --> 00:56:15.338 align:middle line:84%
and you get all
these embeddings.

00:56:15.338 --> 00:56:16.630 align:middle line:90%
It goes through all that stuff.

00:56:16.630 --> 00:56:18.910 align:middle line:84%
You really don't care
about these outputs.

00:56:18.910 --> 00:56:22.290 align:middle line:84%
But wherever the word
"mask" went in in the input,

00:56:22.290 --> 00:56:25.190 align:middle line:84%
you basically try to get it
to-- the right answer is,

00:56:25.190 --> 00:56:28.060 align:middle line:84%
for example, the word "mission,"
and that is the right answer.

00:56:28.060 --> 00:56:29.310 align:middle line:90%
This is the right answer here.

00:56:29.310 --> 00:56:31.893 align:middle line:84%
And then you take these right
answers, create a loss function,

00:56:31.893 --> 00:56:35.530 align:middle line:84%
and do a backprop,
and boom, you're done.

00:56:35.530 --> 00:56:37.910 align:middle line:84%
Inputs right answers
and you're in business.

00:56:37.910 --> 00:56:39.130 align:middle line:90%
That's it.

00:56:39.130 --> 00:56:41.770 align:middle line:84%
Now if we pre-train a
transformer model like this

00:56:41.770 --> 00:56:44.010 align:middle line:84%
on massive amounts
of English text--

00:56:44.010 --> 00:56:45.610 align:middle line:90%
let's say we did that--

00:56:45.610 --> 00:56:47.770 align:middle line:90%
we get something called BERT.

00:56:47.770 --> 00:56:50.210 align:middle line:84%
BERT is a very famous
transformer model.

00:56:50.210 --> 00:56:52.390 align:middle line:84%
And BERT was the
first model, actually,

00:56:52.390 --> 00:56:56.450 align:middle line:84%
that Google used to
upgrade its search in 2019.

00:56:56.450 --> 00:57:00.090 align:middle line:84%
The Brazil visa example, you may
recall from earlier lectures,

00:57:00.090 --> 00:57:01.710 align:middle line:90%
that uses BERT under the hood.

00:57:01.710 --> 00:57:04.600 align:middle line:90%


00:57:04.600 --> 00:57:06.850 align:middle line:84%
Now I just want to show you--
because you can actually

00:57:06.850 --> 00:57:08.433 align:middle line:84%
read the BERT paper
and it'll actually

00:57:08.433 --> 00:57:10.050 align:middle line:84%
make sense to you
now based on what

00:57:10.050 --> 00:57:11.510 align:middle line:90%
you've learned in this class.

00:57:11.510 --> 00:57:12.530 align:middle line:90%
Look at this.

00:57:12.530 --> 00:57:15.290 align:middle line:84%
BERT's model architecture is
a multi-layer bidirectional

00:57:15.290 --> 00:57:16.250 align:middle line:90%
transformer encoder.

00:57:16.250 --> 00:57:17.930 align:middle line:90%
OK, transformer encoder.

00:57:17.930 --> 00:57:21.330 align:middle line:84%
We denote the number of layers,
transformer blocks, as L,

00:57:21.330 --> 00:57:24.250 align:middle line:84%
the hidden size as H, and the
number of attention heads as A.

00:57:24.250 --> 00:57:26.690 align:middle line:90%
And how much is that?

00:57:26.690 --> 00:57:36.530 align:middle line:84%
We want H as 768, which means
that the embedding sizes are 768

00:57:36.530 --> 00:57:39.570 align:middle line:84%
and the hidden feedforward
layer is four times as much,

00:57:39.570 --> 00:57:41.590 align:middle line:90%
so it's 4,096--

00:57:41.590 --> 00:57:45.210 align:middle line:84%
sorry, 4,096 is the
feedforward layer.

00:57:45.210 --> 00:57:47.430 align:middle line:90%
The embeddings are 768.

00:57:47.430 --> 00:57:49.570 align:middle line:84%
And you can see there
are two BERT models here.

00:57:49.570 --> 00:57:52.470 align:middle line:84%
This one has 12
transformer blocks.

00:57:52.470 --> 00:57:54.300 align:middle line:84%
This one has 24
transformer blocks.

00:57:54.300 --> 00:57:56.930 align:middle line:90%


00:57:56.930 --> 00:57:58.430 align:middle line:84%
So you can actually
read this paper.

00:57:58.430 --> 00:58:00.193 align:middle line:84%
You can actually relate
it to exactly what

00:58:00.193 --> 00:58:01.110 align:middle line:90%
we discussed in class.

00:58:01.110 --> 00:58:02.423 align:middle line:90%
It'll all make sense.

00:58:02.423 --> 00:58:05.090 align:middle line:84%
Bidirectional essentially means
that the words can pay attention

00:58:05.090 --> 00:58:07.410 align:middle line:84%
to every other word
in the sentence.

00:58:07.410 --> 00:58:10.170 align:middle line:84%
And as we will
see on Monday, you

00:58:10.170 --> 00:58:12.928 align:middle line:84%
have another transformer thing
called a causal transformer,

00:58:12.928 --> 00:58:14.970 align:middle line:84%
in which you only pay
attention to the words that

00:58:14.970 --> 00:58:17.850 align:middle line:84%
came before you, not
the ones after you.

00:58:17.850 --> 00:58:20.030 align:middle line:84%
So bidirectional means
all words are seen.

00:58:20.030 --> 00:58:22.850 align:middle line:90%


00:58:22.850 --> 00:58:25.210 align:middle line:84%
So what we do is,
remember, we said

00:58:25.210 --> 00:58:27.150 align:middle line:84%
to solve sequence
classification,

00:58:27.150 --> 00:58:30.650 align:middle line:84%
you can add a little token at
the beginning and then boom,

00:58:30.650 --> 00:58:31.990 align:middle line:90%
use it for classification.

00:58:31.990 --> 00:58:34.690 align:middle line:84%
As it turns out,
very conveniently

00:58:34.690 --> 00:58:38.030 align:middle line:84%
for us, the people who built
BERT, when they trained BERT,

00:58:38.030 --> 00:58:41.830 align:middle line:84%
they just used the CLS
business during training,

00:58:41.830 --> 00:58:44.148 align:middle line:84%
so it's actually available
for us out of the box.

00:58:44.148 --> 00:58:46.190 align:middle line:84%
So when you use BERT for
sequence classification,

00:58:46.190 --> 00:58:47.710 align:middle line:84%
you don't even have to
do any surgery on it.

00:58:47.710 --> 00:58:49.710 align:middle line:84%
It just gives you the
class token automatically,

00:58:49.710 --> 00:58:51.490 align:middle line:90%
which is very convenient.

00:58:51.490 --> 00:58:54.450 align:middle line:84%
And you can also use it for
sequence labeling as well.

00:58:54.450 --> 00:58:57.410 align:middle line:84%
So for sequence classification
and sequence labeling,

00:58:57.410 --> 00:58:59.713 align:middle line:84%
BERT is actually usually a
really good starting point.

00:58:59.713 --> 00:59:01.130 align:middle line:84%
And in particular,
there have been

00:59:01.130 --> 00:59:02.922 align:middle line:84%
lots of improvements
and variations of BERT

00:59:02.922 --> 00:59:03.710 align:middle line:90%
over the years.

00:59:03.710 --> 00:59:05.230 align:middle line:84%
And if you're
curious about this,

00:59:05.230 --> 00:59:07.870 align:middle line:84%
there's a thing called The
Sentence Transformers library,

00:59:07.870 --> 00:59:10.850 align:middle line:84%
which has got a whole bunch of
BERT-related code and resources

00:59:10.850 --> 00:59:14.290 align:middle line:84%
that you can use to do
things out of the box.

00:59:14.290 --> 00:59:19.930 align:middle line:84%
OK, this is a bit
of a word wall.

00:59:19.930 --> 00:59:22.450 align:middle line:84%
So to solve any of these
problems, classification

00:59:22.450 --> 00:59:24.510 align:middle line:84%
or labeling, where the
input is natural language,

00:59:24.510 --> 00:59:26.630 align:middle line:84%
we can obviously use
a model like BERT,

00:59:26.630 --> 00:59:29.130 align:middle line:84%
label a few hundred examples,
attach the right final layers,

00:59:29.130 --> 00:59:31.490 align:middle line:84%
and fine-tune it, like
we did for the ResNet.

00:59:31.490 --> 00:59:35.850 align:middle line:84%
But if your problem is like
a standard NLP problem,

00:59:35.850 --> 00:59:37.957 align:middle line:84%
you don't even have to
do that because people,

00:59:37.957 --> 00:59:40.290 align:middle line:84%
for these standard tasks,
they've already pre-trained it

00:59:40.290 --> 00:59:42.393 align:middle line:90%
on those standard tasks.

00:59:42.393 --> 00:59:43.810 align:middle line:84%
And so you can do
all these things

00:59:43.810 --> 00:59:47.370 align:middle line:84%
without any fine-tuning at
all, literally out of the box.

00:59:47.370 --> 00:59:50.150 align:middle line:84%
And so there are many hubs which
have these pre-trained models.

00:59:50.150 --> 00:59:53.490 align:middle line:84%
But perhaps the biggest one
is the Hugging Face Hub.

00:59:53.490 --> 00:59:54.930 align:middle line:90%
And I checked last night.

00:59:54.930 --> 00:59:58.930 align:middle line:90%
It has 520,000 models available.

00:59:58.930 --> 01:00:01.630 align:middle line:84%
I think if I recall, last
year when I taught HODL,

01:00:01.630 --> 01:00:04.250 align:middle line:84%
I think the number was a
lot smaller, maybe 50,000,

01:00:04.250 --> 01:00:08.133 align:middle line:84%
so it's growing
really, really fast.

01:00:08.133 --> 01:00:09.800 align:middle line:84%
Let's just switch to
Hugging Face Colab.

01:00:09.800 --> 01:00:15.170 align:middle line:90%


01:00:15.170 --> 01:00:16.850 align:middle line:90%
So Hugging Face.

01:00:16.850 --> 01:00:21.690 align:middle line:84%
How many of you are
familiar with Hugging Face?

01:00:21.690 --> 01:00:23.670 align:middle line:90%
OK, that's good.

01:00:23.670 --> 01:00:26.210 align:middle line:84%
So for the others,
basically, you

01:00:26.210 --> 01:00:27.930 align:middle line:84%
have a whole bunch
of pre-trained models

01:00:27.930 --> 01:00:28.857 align:middle line:90%
on Hugging Face.

01:00:28.857 --> 01:00:30.690 align:middle line:84%
You actually have a lot
of data sets you can

01:00:30.690 --> 01:00:33.810 align:middle line:90%
work with for your own tasks.

01:00:33.810 --> 01:00:36.690 align:middle line:84%
There are lots of people
demoing what they have built

01:00:36.690 --> 01:00:37.857 align:middle line:90%
in this thing called Spaces.

01:00:37.857 --> 01:00:40.232 align:middle line:84%
And of course, there is a lot
of documentation and so on.

01:00:40.232 --> 01:00:41.610 align:middle line:90%
So the thing you can do is--

01:00:41.610 --> 01:00:44.068 align:middle line:84%
what they have done is they
have organized all these models

01:00:44.068 --> 01:00:46.025 align:middle line:84%
by the kind of task
you can use them for.

01:00:46.025 --> 01:00:48.650 align:middle line:84%
So you can see here, there are
a whole bunch of computer vision

01:00:48.650 --> 01:00:50.330 align:middle line:90%
tasks that you can use them for.

01:00:50.330 --> 01:00:52.038 align:middle line:84%
There's a whole bunch
of natural language

01:00:52.038 --> 01:00:55.770 align:middle line:84%
tasks like text classification,
feature extraction, this

01:00:55.770 --> 01:00:58.477 align:middle line:84%
and that, lots of
interesting examples here.

01:00:58.477 --> 01:01:00.810 align:middle line:84%
And so what you do is you
just literally can go in there

01:01:00.810 --> 01:01:02.810 align:middle line:84%
and say, OK, I want to
do a text classification.

01:01:02.810 --> 01:01:05.010 align:middle line:84%
You hit it, and then it
tells you all the models

01:01:05.010 --> 01:01:05.550 align:middle line:90%
that are available.

01:01:05.550 --> 01:01:07.090 align:middle line:84%
Turns out there are 50,000
models just for text

01:01:07.090 --> 01:01:08.050 align:middle line:90%
classification.

01:01:08.050 --> 01:01:10.303 align:middle line:84%
And you can look at, OK,
which is most downloaded

01:01:10.303 --> 01:01:11.970 align:middle line:84%
or which is the most
liked, and then you

01:01:11.970 --> 01:01:14.220 align:middle line:84%
can just use them as a
starting point for whatever you

01:01:14.220 --> 01:01:16.260 align:middle line:90%
want to do.

01:01:16.260 --> 01:01:17.360 align:middle line:90%
So that is Hugging Face.

01:01:17.360 --> 01:01:20.900 align:middle line:84%
And so the way you
do Hugging Face is--

01:01:20.900 --> 01:01:22.000 align:middle line:90%
I'm just connecting it.

01:01:22.000 --> 01:01:24.802 align:middle line:90%


01:01:24.802 --> 01:01:27.260 align:middle line:84%
If you have a problem which
the input is a natural language

01:01:27.260 --> 01:01:28.620 align:middle line:84%
text, the first
question you have to ask

01:01:28.620 --> 01:01:29.840 align:middle line:90%
is, is it standard or not.

01:01:29.840 --> 01:01:31.080 align:middle line:90%
Is it a standard task or not.

01:01:31.080 --> 01:01:34.400 align:middle line:84%
If it's a standard task,
do not reinvent the wheel.

01:01:34.400 --> 01:01:36.680 align:middle line:84%
This thing will usually
work pretty well.

01:01:36.680 --> 01:01:39.700 align:middle line:84%
So here we will use
this thing called

01:01:39.700 --> 01:01:41.460 align:middle line:84%
the Transformers
Library from Hugging

01:01:41.460 --> 01:01:43.740 align:middle line:84%
Face, in particular,
the pipeline function,

01:01:43.740 --> 01:01:46.032 align:middle line:84%
to demonstrate quickly
how to do this thing.

01:01:46.032 --> 01:01:47.740 align:middle line:84%
Fortunately, this
library as of this year

01:01:47.740 --> 01:01:49.960 align:middle line:84%
is pre-installed in Colab, so
we don't have to install it.

01:01:49.960 --> 01:01:51.543 align:middle line:84%
We can just start
using it right away.

01:01:51.543 --> 01:01:54.540 align:middle line:84%
So we'll take this example where
you have a bunch of text which

01:01:54.540 --> 01:01:59.303 align:middle line:84%
says, "Dear Amazon, Last week I
ordered an Optimus Prime action

01:01:59.303 --> 01:02:00.720 align:middle line:84%
figure from your
store in Germany.

01:02:00.720 --> 01:02:01.880 align:middle line:84%
Unfortunately, when
I opened the package,

01:02:01.880 --> 01:02:04.940 align:middle line:84%
I discovered to my horror that
I had been sent an action figure

01:02:04.940 --> 01:02:06.000 align:middle line:90%
of Megatron instead."

01:02:06.000 --> 01:02:09.020 align:middle line:84%
Can you imagine that person's
sheer distress at this?

01:02:09.020 --> 01:02:11.520 align:middle line:84%
"So as a lifelong enemy
of the Decepticons,

01:02:11.520 --> 01:02:13.300 align:middle line:84%
I hope you can
understand my dilemma.

01:02:13.300 --> 01:02:15.740 align:middle line:84%
So to resolve the issue,
I demand an exchange.

01:02:15.740 --> 01:02:17.040 align:middle line:90%
Enclosed are copies.

01:02:17.040 --> 01:02:18.280 align:middle line:90%
Expect to hear from you soon.

01:02:18.280 --> 01:02:22.320 align:middle line:90%
Sincerely, Bumblebee."

01:02:22.320 --> 01:02:25.625 align:middle line:84%
They should come up with a
better name for this example.

01:02:25.625 --> 01:02:27.000 align:middle line:84%
Cool, so that's
the text we have.

01:02:27.000 --> 01:02:28.366 align:middle line:90%
So we import the--

01:02:28.366 --> 01:02:31.020 align:middle line:84%
this pipeline function is the
one that basically gives you

01:02:31.020 --> 01:02:33.940 align:middle line:84%
the ability to out-of-the-box
start using it without any

01:02:33.940 --> 01:02:35.700 align:middle line:90%
pre-training, nothing like that.

01:02:35.700 --> 01:02:38.740 align:middle line:90%
So we download this thing.

01:02:38.740 --> 01:02:39.240 align:middle line:90%
Oh, wow.

01:02:39.240 --> 01:02:41.020 align:middle line:90%
I got an A100 today.

01:02:41.020 --> 01:02:42.840 align:middle line:90%
That happens very rarely.

01:02:42.840 --> 01:02:44.500 align:middle line:90%
Sorry.

01:02:44.500 --> 01:02:47.900 align:middle line:84%
So here let's say you want to
classify that text You just want

01:02:47.900 --> 01:02:49.340 align:middle line:90%
to classify it for sentiment.

01:02:49.340 --> 01:02:53.100 align:middle line:84%
You literally go in there
and say pipeline text

01:02:53.100 --> 01:02:56.300 align:middle line:84%
classification-- that's the
task you want the pipeline to do

01:02:56.300 --> 01:02:57.300 align:middle line:90%
for you--

01:02:57.300 --> 01:02:59.580 align:middle line:84%
and you create a
classifier It's going

01:02:59.580 --> 01:03:04.260 align:middle line:84%
to download a bunch of stuff
and so on and so forth.

01:03:04.260 --> 01:03:06.860 align:middle line:84%
The first time it just
takes time to download.

01:03:06.860 --> 01:03:08.692 align:middle line:84%
And then you literally
take the text

01:03:08.692 --> 01:03:10.900 align:middle line:84%
you have here, and then run
it through the classifier

01:03:10.900 --> 01:03:13.580 align:middle line:84%
as if it's just a
little function.

01:03:13.580 --> 01:03:16.260 align:middle line:90%
You get some outputs.

01:03:16.260 --> 01:03:19.500 align:middle line:84%
Actually, let's
just do this way.

01:03:19.500 --> 01:03:20.100 align:middle line:90%
Negative.

01:03:20.100 --> 01:03:22.380 align:middle line:84%
Sentiment is negative
with a 90% probability.

01:03:22.380 --> 01:03:25.740 align:middle line:90%
Pretty good, right?

01:03:25.740 --> 01:03:27.120 align:middle line:90%
Sentiment classification solved.

01:03:27.120 --> 01:03:29.660 align:middle line:84%
So we'll try a few
different examples.

01:03:29.660 --> 01:03:30.482 align:middle line:90%
"I hated the movie.

01:03:30.482 --> 01:03:32.440 align:middle line:84%
If I said I loved the
movie, I would be lying."

01:03:32.440 --> 01:03:33.620 align:middle line:90%
OK, that's a little tricky.

01:03:33.620 --> 01:03:34.800 align:middle line:90%
"The movie left me speechless.

01:03:34.800 --> 01:03:35.300 align:middle line:90%
Incredible."

01:03:35.300 --> 01:03:38.060 align:middle line:84%
And then I had to add this
last thing here last night.

01:03:38.060 --> 01:03:41.427 align:middle line:84%
"Almost but not quite entirely
unlike anything good I've seen."

01:03:41.427 --> 01:03:42.760 align:middle line:90%
That's not original, by the way.

01:03:42.760 --> 01:03:44.180 align:middle line:84%
People who have read
Douglas Adams will

01:03:44.180 --> 01:03:46.940 align:middle line:84%
know this famous sentence about
somebody drinking some beverage

01:03:46.940 --> 01:03:50.420 align:middle line:84%
and saying it's almost but
not quite entirely unlike tea.

01:03:50.420 --> 01:03:51.680 align:middle line:90%
So I was inspired by that.

01:03:51.680 --> 01:03:53.097 align:middle line:84%
So anyway, we'll
see what happens.

01:03:53.097 --> 01:03:56.100 align:middle line:90%


01:03:56.100 --> 01:03:58.820 align:middle line:90%
All right, put it in there.

01:03:58.820 --> 01:04:00.447 align:middle line:90%
So negative.

01:04:00.447 --> 01:04:01.280 align:middle line:90%
"I hated the movie."

01:04:01.280 --> 01:04:01.760 align:middle line:90%
OK, fine.

01:04:01.760 --> 01:04:02.960 align:middle line:84%
"If I said I loved the
movie, I'd be lying."

01:04:02.960 --> 01:04:03.460 align:middle line:90%
Negative.

01:04:03.460 --> 01:04:04.940 align:middle line:90%
"The movie left me speechless."

01:04:04.940 --> 01:04:07.862 align:middle line:84%
It says it's negative, but it
could go either way, right?

01:04:07.862 --> 01:04:09.820 align:middle line:84%
A good classifier would
probably have given you

01:04:09.820 --> 01:04:11.780 align:middle line:84%
a probability around the
50% mark because it's

01:04:11.780 --> 01:04:13.700 align:middle line:90%
right on the fence.

01:04:13.700 --> 01:04:14.260 align:middle line:90%
"Incredible."

01:04:14.260 --> 01:04:15.360 align:middle line:90%
It's positive.

01:04:15.360 --> 01:04:17.540 align:middle line:84%
And then it got fooled by
my crazy long sentence,

01:04:17.540 --> 01:04:19.620 align:middle line:90%
and it says it's positive.

01:04:19.620 --> 01:04:21.320 align:middle line:90%
Now that's classification.

01:04:21.320 --> 01:04:22.760 align:middle line:90%
Here is one other quick example.

01:04:22.760 --> 01:04:25.120 align:middle line:84%
So you can actually
give it a piece of text.

01:04:25.120 --> 01:04:28.260 align:middle line:84%
For example, you can take
a Reuters news story.

01:04:28.260 --> 01:04:31.983 align:middle line:84%
You can feed it and say, extract
all the company names from it.

01:04:31.983 --> 01:04:34.400 align:middle line:84%
Extract company names, people
names, and things like that.

01:04:34.400 --> 01:04:36.460 align:middle line:84%
It's called named
entity extraction.

01:04:36.460 --> 01:04:40.380 align:middle line:84%
And back in the
day, people would

01:04:40.380 --> 01:04:43.220 align:middle line:84%
hand-build painstakingly all
these very complex systems

01:04:43.220 --> 01:04:45.480 align:middle line:90%
to do named entity extraction.

01:04:45.480 --> 01:04:46.840 align:middle line:90%
Now it's just a pipeline away.

01:04:46.840 --> 01:04:50.500 align:middle line:84%
So you can take this thing and
you can say, create a pipeline

01:04:50.500 --> 01:04:52.740 align:middle line:90%
for ner, named extraction.

01:04:52.740 --> 01:04:54.640 align:middle line:84%
For any particular
task that you're using,

01:04:54.640 --> 01:04:56.390 align:middle line:84%
there might be a few
additional parameters

01:04:56.390 --> 01:04:58.940 align:middle line:84%
you can set as a part
of the configuration.

01:04:58.940 --> 01:05:00.640 align:middle line:90%
So we download this pipeline.

01:05:00.640 --> 01:05:09.300 align:middle line:90%


01:05:09.300 --> 01:05:10.120 align:middle line:90%
Perfect.

01:05:10.120 --> 01:05:12.460 align:middle line:90%
And then we run the output.

01:05:12.460 --> 01:05:14.160 align:middle line:90%
So it says, OK, good.

01:05:14.160 --> 01:05:20.740 align:middle line:84%
Amazon is an organization and
Germany is a location, LOC,

01:05:20.740 --> 01:05:21.480 align:middle line:90%
which is nice.

01:05:21.480 --> 01:05:22.860 align:middle line:84%
So these things have
a standard vocabulary

01:05:22.860 --> 01:05:24.080 align:middle line:84%
as to ORG, LOC,
things like that,

01:05:24.080 --> 01:05:25.940 align:middle line:84%
which you can read up
in the documentation.

01:05:25.940 --> 01:05:27.660 align:middle line:90%
And then Bumblebee is a person.

01:05:27.660 --> 01:05:31.540 align:middle line:84%
And then boy, all the Optimus
Prime Transformer stuff

01:05:31.540 --> 01:05:32.980 align:middle line:90%
is all-- it got fooled.

01:05:32.980 --> 01:05:35.860 align:middle line:84%
It thinks Optimus
Prime is miscellaneous,

01:05:35.860 --> 01:05:37.960 align:middle line:84%
Decept is miscellaneous,
and so on and so forth.

01:05:37.960 --> 01:05:38.600 align:middle line:90%
But you get the idea.

01:05:38.600 --> 01:05:40.900 align:middle line:84%
You can take standard things
like Reuters news stories

01:05:40.900 --> 01:05:41.400 align:middle line:90%
and so on.

01:05:41.400 --> 01:05:42.020 align:middle line:90%
You can just feed it, boop.

01:05:42.020 --> 01:05:44.680 align:middle line:84%
You can get a very good entity
extraction right out of the bat.

01:05:44.680 --> 01:05:46.480 align:middle line:84%
And once you get these
entities extracted,

01:05:46.480 --> 01:05:48.660 align:middle line:84%
then you can put them into
a nice structured data

01:05:48.660 --> 01:05:50.260 align:middle line:84%
table like a
database, and then you

01:05:50.260 --> 01:05:54.060 align:middle line:84%
can run traditional
machine learning on it.

01:05:54.060 --> 01:05:56.660 align:middle line:84%
And then I had, I think,
a few more examples

01:05:56.660 --> 01:05:58.980 align:middle line:90%
of question answering.

01:05:58.980 --> 01:06:00.818 align:middle line:90%
Actually, let's just try that.

01:06:00.818 --> 01:06:03.360 align:middle line:84%
You can actually give it a thing
and ask a question about it,

01:06:03.360 --> 01:06:06.140 align:middle line:84%
and it can actually give
you the answer, which

01:06:06.140 --> 01:06:08.740 align:middle line:84%
gets into the causal
transformer thing

01:06:08.740 --> 01:06:10.940 align:middle line:84%
that we're going to see
on Monday, which builds up

01:06:10.940 --> 01:06:13.485 align:middle line:84%
into large language models
because you obviously can give

01:06:13.485 --> 01:06:14.860 align:middle line:84%
something, you
can give a passage

01:06:14.860 --> 01:06:16.340 align:middle line:84%
to ChatGPT and ask
a question and ask

01:06:16.340 --> 01:06:17.360 align:middle line:90%
it to give you an answer.

01:06:17.360 --> 01:06:18.568 align:middle line:90%
So it's really in that thing.

01:06:18.568 --> 01:06:21.600 align:middle line:84%
But just for fun, let's just do
that to see if it's any good.

01:06:21.600 --> 01:06:24.660 align:middle line:90%


01:06:24.660 --> 01:06:26.340 align:middle line:90%
What does the customer want?

01:06:26.340 --> 01:06:28.740 align:middle line:84%
And the output is an
exchange of Megatron.

01:06:28.740 --> 01:06:32.500 align:middle line:84%
And it's telling you where
it starts in the text

01:06:32.500 --> 01:06:34.540 align:middle line:84%
and where it ends
the relevant passage.

01:06:34.540 --> 01:06:37.162 align:middle line:90%
It's pretty good, right?

01:06:37.162 --> 01:06:38.620 align:middle line:84%
Because remember,
if you have stuff

01:06:38.620 --> 01:06:41.562 align:middle line:84%
like this, then when you ask a
large language model a question,

01:06:41.562 --> 01:06:42.520 align:middle line:90%
it gives you an answer.

01:06:42.520 --> 01:06:46.020 align:middle line:84%
You can actually ask it to give
you exactly where in the input

01:06:46.020 --> 01:06:46.853 align:middle line:90%
it found the answer.

01:06:46.853 --> 01:06:49.312 align:middle line:84%
And because you know these
things are going to hallucinate,

01:06:49.312 --> 01:06:50.940 align:middle line:84%
you can actually
look at the input

01:06:50.940 --> 01:06:53.820 align:middle line:84%
that it's claiming to use and
look at what it says and see

01:06:53.820 --> 01:06:55.580 align:middle line:90%
if they actually match.

01:06:55.580 --> 01:07:00.400 align:middle line:84%
It's a way to essentially
do QA on LLM output.

01:07:00.400 --> 01:07:01.800 align:middle line:90%
So that's what we have here.

01:07:01.800 --> 01:07:03.342 align:middle line:84%
And I have another
bunch of questions

01:07:03.342 --> 01:07:05.420 align:middle line:84%
which I'll ignore for
the moment because I want

01:07:05.420 --> 01:07:08.400 align:middle line:90%
to go back to the PowerPoint.

01:07:08.400 --> 01:07:11.780 align:middle line:84%
So if you have a standard task,
you can just use pipelines

01:07:11.780 --> 01:07:13.380 align:middle line:84%
and Hugging Face
to actually solve

01:07:13.380 --> 01:07:16.300 align:middle line:84%
many of them out of the box
without any heavy lifting.

01:07:16.300 --> 01:07:18.500 align:middle line:84%
So I mentioned earlier
on that transformers

01:07:18.500 --> 01:07:21.300 align:middle line:84%
have proven to be effective for
a whole bunch of domains outside

01:07:21.300 --> 01:07:25.020 align:middle line:84%
of natural language
processing, like speech

01:07:25.020 --> 01:07:28.380 align:middle line:84%
recognition, computer vision,
and so on and so forth.

01:07:28.380 --> 01:07:30.940 align:middle line:84%
And so I want to give you a
couple of quick examples of how

01:07:30.940 --> 01:07:33.060 align:middle line:84%
to think about
using transformers

01:07:33.060 --> 01:07:34.960 align:middle line:90%
for non-text applications.

01:07:34.960 --> 01:07:38.140 align:middle line:90%


01:07:38.140 --> 01:07:40.940 align:middle line:84%
The key insight here is that the
architecture of the transformer

01:07:40.940 --> 01:07:43.160 align:middle line:84%
block that we have looked
at, amazingly enough,

01:07:43.160 --> 01:07:45.280 align:middle line:84%
can be used as-is
with no changes,

01:07:45.280 --> 01:07:47.780 align:middle line:84%
no surgery needed, no
clever thinking required

01:07:47.780 --> 01:07:49.380 align:middle line:90%
for any particular application.

01:07:49.380 --> 01:07:51.920 align:middle line:84%
What is needed, where the
clever thinking may be required,

01:07:51.920 --> 01:07:55.080 align:middle line:84%
is you need to take the inputs
that you're working with,

01:07:55.080 --> 01:07:57.860 align:middle line:84%
and you need to figure out a
way to tokenize and encode them

01:07:57.860 --> 01:08:01.940 align:middle line:84%
into embeddings, which can then
be sent into the transformer.

01:08:01.940 --> 01:08:05.720 align:middle line:84%
So all the action is in taking
that input, that non-text input,

01:08:05.720 --> 01:08:07.460 align:middle line:84%
and figuring out
a way to cast them

01:08:07.460 --> 01:08:09.180 align:middle line:90%
in the language of embeddings.

01:08:09.180 --> 01:08:11.070 align:middle line:90%
That's the game.

01:08:11.070 --> 01:08:14.240 align:middle line:84%
So here is something called
the Vision Transformer,

01:08:14.240 --> 01:08:15.820 align:middle line:90%
which is very famous, actually.

01:08:15.820 --> 01:08:19.580 align:middle line:84%
I think it may be perhaps
the first transformer

01:08:19.580 --> 01:08:22.060 align:middle line:84%
architecture that was
applied to vision problems.

01:08:22.060 --> 01:08:25.580 align:middle line:90%
So let's say you have a picture.

01:08:25.580 --> 01:08:28.667 align:middle line:84%
So let's say you
have this picture.

01:08:28.667 --> 01:08:30.640 align:middle line:90%
It is just a picture.

01:08:30.640 --> 01:08:33.020 align:middle line:84%
So you have to find a
way to create embeddings

01:08:33.020 --> 01:08:36.620 align:middle line:84%
from this picture or to tokenize
this picture in some way.

01:08:36.620 --> 01:08:39.132 align:middle line:84%
With sentences, you know, "I
love HODL," well, obviously,

01:08:39.132 --> 01:08:40.840 align:middle line:84%
"I," "love," and "HODL"
are three tokens.

01:08:40.840 --> 01:08:43.260 align:middle line:84%
That's pretty trivial to figure
out how to tokenize them.

01:08:43.260 --> 01:08:44.740 align:middle line:84%
But with a picture,
what do you do?

01:08:44.740 --> 01:08:47.819 align:middle line:84%
It's weird to think of
tokenizing a picture.

01:08:47.819 --> 01:08:50.385 align:middle line:84%
So what these people did is
that they say, you know what?

01:08:50.385 --> 01:08:52.260 align:middle line:84%
I'm going to take this
picture and chop it up

01:08:52.260 --> 01:08:55.340 align:middle line:90%
into small squares.

01:08:55.340 --> 01:08:57.939 align:middle line:84%
So in this example, they
have taken this big picture

01:08:57.939 --> 01:09:01.939 align:middle line:84%
and chopped it up into
nine little pictures.

01:09:01.939 --> 01:09:05.220 align:middle line:84%
Then you can take each
of those nine pictures,

01:09:05.220 --> 01:09:06.939 align:middle line:90%
each of those nine pictures.

01:09:06.939 --> 01:09:09.720 align:middle line:84%
If you look at how
it's represented,

01:09:09.720 --> 01:09:14.439 align:middle line:84%
it's just three tables of
numbers, the RGB values.

01:09:14.439 --> 01:09:16.479 align:middle line:84%
So you can take
all those numbers,

01:09:16.479 --> 01:09:20.700 align:middle line:84%
and you just create a
giant long vector from it.

01:09:20.700 --> 01:09:22.620 align:middle line:90%
You have a huge long vector.

01:09:22.620 --> 01:09:25.899 align:middle line:84%
And then you run it
through a dense layer

01:09:25.899 --> 01:09:28.740 align:middle line:84%
to come up with
a smaller vector.

01:09:28.740 --> 01:09:31.779 align:middle line:84%
And that smaller vector
is your embedding.

01:09:31.779 --> 01:09:32.660 align:middle line:90%
That's it.

01:09:32.660 --> 01:09:35.420 align:middle line:84%
But the way you transform the
long vector into a small vector

01:09:35.420 --> 01:09:39.100 align:middle line:84%
is just a dense layer whose
weights can be learned.

01:09:39.100 --> 01:09:41.067 align:middle line:84%
So what these people
did is they said,

01:09:41.067 --> 01:09:43.359 align:middle line:84%
well, I'm going to first chop
it up into these patches,

01:09:43.359 --> 01:09:46.740 align:middle line:84%
and then I take each patch
and do a linear projection.

01:09:46.740 --> 01:09:49.420 align:middle line:84%
A flattened patch is nothing
more than three tables

01:09:49.420 --> 01:09:51.319 align:middle line:84%
of numbers flattened
into a long vector.

01:09:51.319 --> 01:09:53.460 align:middle line:84%
That's what the word
"flatten" here means.

01:09:53.460 --> 01:09:55.100 align:middle line:84%
And once you
flatten it, I'm just

01:09:55.100 --> 01:09:56.820 align:middle line:84%
going to run it
through a dense layer.

01:09:56.820 --> 01:09:59.600 align:middle line:84%
So by the way, you will see
the words "linear projection."

01:09:59.600 --> 01:10:03.420 align:middle line:84%
It's a synonym for, run
it through a dense layer.

01:10:03.420 --> 01:10:05.520 align:middle line:84%
So you run it through
a dense layer.

01:10:05.520 --> 01:10:09.540 align:middle line:84%
You get these nice
vectors, these vectors.

01:10:09.540 --> 01:10:11.113 align:middle line:84%
And now you say,
well, you know what,

01:10:11.113 --> 01:10:12.780 align:middle line:84%
I have to take the
order of these things

01:10:12.780 --> 01:10:15.420 align:middle line:84%
into account because
clearly, this little patch is

01:10:15.420 --> 01:10:18.440 align:middle line:84%
in the top left while this patch
is somewhere in the middle.

01:10:18.440 --> 01:10:20.475 align:middle line:84%
The order matters
in the picture,

01:10:20.475 --> 01:10:23.100 align:middle line:84%
otherwise, every jumbled version
is going to be the same thing.

01:10:23.100 --> 01:10:26.140 align:middle line:84%
So you use positional
embeddings.

01:10:26.140 --> 01:10:29.780 align:middle line:84%
You basically say there are
nine positions in any picture.

01:10:29.780 --> 01:10:32.380 align:middle line:90%
0, 1, 2, 3, 4, 5, 6, 7, 8.

01:10:32.380 --> 01:10:35.500 align:middle line:84%
There are nine positions, so I'm
going to create nine position

01:10:35.500 --> 01:10:36.940 align:middle line:90%
embeddings.

01:10:36.940 --> 01:10:38.592 align:middle line:84%
And then I'm just
going to add them up.

01:10:38.592 --> 01:10:40.800 align:middle line:84%
Then I'm just going to add
them up to this embedding,

01:10:40.800 --> 01:10:42.726 align:middle line:90%
just like we did with words.

01:10:42.726 --> 01:10:44.508 align:middle line:84%
With words, each word
had an embedding.

01:10:44.508 --> 01:10:45.800 align:middle line:90%
Each position had an embedding.

01:10:45.800 --> 01:10:46.508 align:middle line:90%
We added them up.

01:10:46.508 --> 01:10:48.500 align:middle line:84%
Here each image
has an embedding.

01:10:48.500 --> 01:10:50.500 align:middle line:84%
The position of that
little patch in the picture

01:10:50.500 --> 01:10:51.280 align:middle line:90%
has an embedding.

01:10:51.280 --> 01:10:53.180 align:middle line:90%
We add them up

01:10:53.180 --> 01:10:54.700 align:middle line:84%
And then because
we want to use it

01:10:54.700 --> 01:10:56.640 align:middle line:90%
for classification, no problem.

01:10:56.640 --> 01:11:00.660 align:middle line:84%
We'll have a little
CLS token, and then

01:11:00.660 --> 01:11:02.480 align:middle line:84%
we just run it through
the transformer.

01:11:02.480 --> 01:11:04.300 align:middle line:90%
That's it.

01:11:04.300 --> 01:11:06.380 align:middle line:84%
And then you get the
CLS token, and then

01:11:06.380 --> 01:11:08.900 align:middle line:84%
you can attach a softmax to
it and say, OK, it's a bird.

01:11:08.900 --> 01:11:09.400 align:middle line:90%
It's a ball.

01:11:09.400 --> 01:11:12.020 align:middle line:90%
It's a car.

01:11:12.020 --> 01:11:12.840 align:middle line:90%
That's it.

01:11:12.840 --> 01:11:20.180 align:middle line:84%
This simple approach actually
works, amazingly enough.

01:11:20.180 --> 01:11:21.803 align:middle line:84%
So that is the
vision transformer.

01:11:21.803 --> 01:11:23.220 align:middle line:84%
And I'm going
through it fast just

01:11:23.220 --> 01:11:25.620 align:middle line:84%
to give you a sense for
how these things work.

01:11:25.620 --> 01:11:27.300 align:middle line:90%
Any questions?

01:11:27.300 --> 01:11:29.060 align:middle line:90%
Yeah?

01:11:29.060 --> 01:11:31.740 align:middle line:84%
AUDIENCE: My question
is in case of text

01:11:31.740 --> 01:11:34.153 align:middle line:84%
we had fixed number of tokens
that is the amount of words

01:11:34.153 --> 01:11:35.820 align:middle line:84%
which could be there
in your vocabulary,

01:11:35.820 --> 01:11:37.300 align:middle line:90%
any English vocabulary.

01:11:37.300 --> 01:11:39.620 align:middle line:84%
But here if you look at
images, they will probably

01:11:39.620 --> 01:11:43.560 align:middle line:84%
go into trillions and I know
we would not upload one image

01:11:43.560 --> 01:11:46.220 align:middle line:84%
but we take a total
set of product images

01:11:46.220 --> 01:11:47.960 align:middle line:84%
and we try to subset
each one of them,

01:11:47.960 --> 01:11:53.500 align:middle line:84%
each one would have its own
weights like [INAUDIBLE].

01:11:53.500 --> 01:11:55.950 align:middle line:84%
RAMA RAMAKRISHNAN: There is
no notion of vocabulary here.

01:11:55.950 --> 01:11:58.430 align:middle line:84%
All we are saying is
that given any image,

01:11:58.430 --> 01:12:02.230 align:middle line:84%
we create nine patches,
subimages from it.

01:12:02.230 --> 01:12:06.030 align:middle line:84%
Each of those patches gets
passed through a dense layer,

01:12:06.030 --> 01:12:07.990 align:middle line:90%
and out comes an embedding.

01:12:07.990 --> 01:12:09.890 align:middle line:84%
So at that point, any
image you give me,

01:12:09.890 --> 01:12:12.372 align:middle line:84%
I'm going to get you nine
embeddings out of it.

01:12:12.372 --> 01:12:13.830 align:middle line:84%
And once I get the
nine embeddings,

01:12:13.830 --> 01:12:16.110 align:middle line:84%
I just throw it into the meat
grinder, the transformer meat

01:12:16.110 --> 01:12:16.610 align:middle line:90%
grinder.

01:12:16.610 --> 01:12:20.530 align:middle line:90%


01:12:20.530 --> 01:12:23.270 align:middle line:90%
So another example.

01:12:23.270 --> 01:12:24.670 align:middle line:84%
I think some of
you have asked me

01:12:24.670 --> 01:12:27.510 align:middle line:84%
outside of class, how
good are transformers for

01:12:27.510 --> 01:12:29.780 align:middle line:90%
structured data, tabular data.

01:12:29.780 --> 01:12:33.170 align:middle line:84%
For tabular data in general,
things like XGBoost,

01:12:33.170 --> 01:12:35.050 align:middle line:84%
gradient boosting works
really, really well,

01:12:35.050 --> 01:12:36.662 align:middle line:84%
so it's good to try
them, certainly.

01:12:36.662 --> 01:12:38.870 align:middle line:84%
I don't think transformers
and deep learning networks

01:12:38.870 --> 01:12:43.250 align:middle line:84%
have any great edge over XGBoost
for structured data problems,

01:12:43.250 --> 01:12:44.850 align:middle line:84%
so it's worth
trying both of them.

01:12:44.850 --> 01:12:46.910 align:middle line:84%
However, you can use
transformers for this stuff,

01:12:46.910 --> 01:12:47.590 align:middle line:90%
too.

01:12:47.590 --> 01:12:49.610 align:middle line:84%
So that's called
the tab transformer,

01:12:49.610 --> 01:12:51.430 align:middle line:84%
one of the first
ones to come out,

01:12:51.430 --> 01:12:53.190 align:middle line:90%
a transformer for tabular data.

01:12:53.190 --> 01:12:54.570 align:middle line:90%
And again, it's pretty simple.

01:12:54.570 --> 01:12:58.470 align:middle line:84%
All you do is in any kind
of input that you have,

01:12:58.470 --> 01:13:02.130 align:middle line:84%
you will have some categorical
variables like blood pressure,

01:13:02.130 --> 01:13:03.360 align:middle line:90%
things like that.

01:13:03.360 --> 01:13:04.330 align:middle line:90%
Not blood pressure.

01:13:04.330 --> 01:13:05.230 align:middle line:90%
Bad example.

01:13:05.230 --> 01:13:08.430 align:middle line:90%
Gender and so on and so forth.

01:13:08.430 --> 01:13:11.890 align:middle line:84%
And so what you do is you take
all the categorical features,

01:13:11.890 --> 01:13:14.110 align:middle line:84%
and for each
categorical feature,

01:13:14.110 --> 01:13:18.430 align:middle line:84%
you create embeddings because
a categorical feature is just

01:13:18.430 --> 01:13:20.325 align:middle line:90%
text.

01:13:20.325 --> 01:13:22.072 align:middle line:84%
A categorical
feature is just text,

01:13:22.072 --> 01:13:24.280 align:middle line:84%
so you can create text
embeddings for it, no problem.

01:13:24.280 --> 01:13:27.830 align:middle line:90%


01:13:27.830 --> 01:13:32.710 align:middle line:84%
And you take all the continuous
features, cholesterol and blood

01:13:32.710 --> 01:13:35.990 align:middle line:84%
pressure and whatnot, to go
to the heart disease example,

01:13:35.990 --> 01:13:39.630 align:middle line:84%
and then you just correct them
all and just create a vector out

01:13:39.630 --> 01:13:41.790 align:middle line:90%
of them.

01:13:41.790 --> 01:13:44.170 align:middle line:90%
It's just a vector.

01:13:44.170 --> 01:13:48.110 align:middle line:84%
Then you run the embeddings for
all the categorical variables

01:13:48.110 --> 01:13:50.775 align:middle line:84%
through a nice
transformer block.

01:13:50.775 --> 01:13:52.150 align:middle line:84%
And you can see
here it's exactly

01:13:52.150 --> 01:13:53.400 align:middle line:90%
the block we have seen before.

01:13:53.400 --> 01:13:54.350 align:middle line:90%
No difference.

01:13:54.350 --> 01:13:57.050 align:middle line:84%
And then at the very end, when
it comes out of the transformer,

01:13:57.050 --> 01:13:58.150 align:middle line:84%
you take all the
contextual stuff

01:13:58.150 --> 01:14:00.630 align:middle line:84%
coming out of the transformer,
and then you concatenate it

01:14:00.630 --> 01:14:03.710 align:middle line:90%
with the continuous features.

01:14:03.710 --> 01:14:06.470 align:middle line:84%
And then you run it through
maybe one or more dense layers

01:14:06.470 --> 01:14:09.470 align:middle line:90%
and boom, output.

01:14:09.470 --> 01:14:11.610 align:middle line:84%
So this is a tabular
data transformer.

01:14:11.610 --> 01:14:14.390 align:middle line:84%
And there are many refinements,
improvements over the years

01:14:14.390 --> 01:14:16.390 align:middle line:90%
that have come since then.

01:14:16.390 --> 01:14:19.150 align:middle line:84%
But the key thing I want
you to remember here

01:14:19.150 --> 01:14:23.950 align:middle line:84%
is that categorical variables
can be very easily represented

01:14:23.950 --> 01:14:25.230 align:middle line:90%
as embeddings.

01:14:25.230 --> 01:14:25.910 align:middle line:90%
That's the key.

01:14:25.910 --> 01:14:29.050 align:middle line:90%


01:14:29.050 --> 01:14:30.030 align:middle line:90%
So that's that.

01:14:30.030 --> 01:14:31.910 align:middle line:84%
Now once the input
has been transformed

01:14:31.910 --> 01:14:34.090 align:middle line:84%
into this common
language of embeddings,

01:14:34.090 --> 01:14:35.590 align:middle line:84%
we can process them
without changing

01:14:35.590 --> 01:14:37.965 align:middle line:84%
the architecture of the block
itself because all it wants

01:14:37.965 --> 01:14:38.990 align:middle line:90%
is embeddings.

01:14:38.990 --> 01:14:40.530 align:middle line:84%
It's like, you
give me embeddings,

01:14:40.530 --> 01:14:42.322 align:middle line:84%
I give you great
contextual embeddings out,

01:14:42.322 --> 01:14:43.697 align:middle line:90%
and nobody gets hurt.

01:14:43.697 --> 01:14:45.530 align:middle line:84%
That is the deal with
the transformer stack.

01:14:45.530 --> 01:14:50.282 align:middle line:90%


01:14:50.282 --> 01:14:53.090 align:middle line:84%
Since the transformer is
agnostic to the kind of input,

01:14:53.090 --> 01:14:56.230 align:middle line:84%
as long as it comes in as
the form of an embedding,

01:14:56.230 --> 01:14:58.293 align:middle line:84%
you can use it for
multi-modal data very easily.

01:14:58.293 --> 01:14:59.710 align:middle line:84%
So for example,
let's say that you

01:14:59.710 --> 01:15:02.150 align:middle line:84%
have a problem in which you
have a picture that you have

01:15:02.150 --> 01:15:04.030 align:middle line:84%
to be sent in, some
text that goes in,

01:15:04.030 --> 01:15:06.550 align:middle line:84%
a bunch of tabular
data coming in.

01:15:06.550 --> 01:15:09.270 align:middle line:84%
Well, you take the text
and do language embeddings,

01:15:09.270 --> 01:15:10.330 align:middle line:90%
like we know how to do.

01:15:10.330 --> 01:15:11.790 align:middle line:84%
You take the image and
do image embeddings,

01:15:11.790 --> 01:15:13.610 align:middle line:84%
like we just saw with
the vision transformer.

01:15:13.610 --> 01:15:15.777 align:middle line:84%
You take tabular data under
tabular data embeddings,

01:15:15.777 --> 01:15:17.870 align:middle line:84%
like we saw with
the tab transformer.

01:15:17.870 --> 01:15:21.830 align:middle line:84%
Once we do it, it's all
a bunch of embeddings.

01:15:21.830 --> 01:15:24.150 align:middle line:84%
And then you attach a
little class token on top,

01:15:24.150 --> 01:15:26.885 align:middle line:84%
send it through a bunch
of transformer blocks,

01:15:26.885 --> 01:15:28.510 align:middle line:84%
and then out comes
the contextual class

01:15:28.510 --> 01:15:30.450 align:middle line:90%
token, the contextual version.

01:15:30.450 --> 01:15:33.090 align:middle line:84%
Run it through a ReLU, maybe
a sigmoid or a softmax,

01:15:33.090 --> 01:15:35.642 align:middle line:90%
predict the label, done.

01:15:35.642 --> 01:15:38.310 align:middle line:84%
So this is extremely
powerful, its ability

01:15:38.310 --> 01:15:42.490 align:middle line:84%
to handle multi-modal data
And that's why, for example,

01:15:42.490 --> 01:15:47.130 align:middle line:84%
if you look at Google Gemini
1.5 Pro, GPT4 Vision and so on,

01:15:47.130 --> 01:15:49.630 align:middle line:84%
you can send it
images and a question,

01:15:49.630 --> 01:15:51.710 align:middle line:90%
and you'll get an answer back.

01:15:51.710 --> 01:15:55.050 align:middle line:84%
Because every modality that
goes in is cast into embeddings.

01:15:55.050 --> 01:15:59.468 align:middle line:84%
And once it's embeddingized,
then the transformer

01:15:59.468 --> 01:16:00.010 align:middle line:90%
doesn't care.

01:16:00.010 --> 01:16:02.030 align:middle line:90%
It'll just do its thing.

01:16:02.030 --> 01:16:05.590 align:middle line:84%
It will decide, for example,
that this word in your question

01:16:05.590 --> 01:16:10.030 align:middle line:84%
actually is highly related
to that patch in the picture.

01:16:10.030 --> 01:16:13.388 align:middle line:90%
It'll just figure it out.

01:16:13.388 --> 01:16:15.930 align:middle line:84%
OK, that's all I had because
it's that time to break at 9:55.

01:16:15.930 --> 01:16:16.728 align:middle line:90%
Perfect.

01:16:16.728 --> 01:16:17.770 align:middle line:90%
All right, folks, thanks.

01:16:17.770 --> 01:16:19.062 align:middle line:90%
Have a great rest of your week.

01:16:19.062 --> 01:16:22.560 align:middle line:90%
[APPLAUSE]

01:16:22.560 --> 01:16:46.000 align:middle line:90%