WEBVTT

00:00:00.000 --> 00:00:02.495 align:middle line:90%
[SQUEAKING]

00:00:02.495 --> 00:00:04.491 align:middle line:90%
[RUSTLING]

00:00:04.491 --> 00:00:07.485 align:middle line:90%
[CLICKING]

00:00:07.485 --> 00:00:16.452 align:middle line:90%


00:00:16.452 --> 00:00:17.660 align:middle line:90%
RAMA RAMAKRISHNAN: All right.

00:00:17.660 --> 00:00:19.800 align:middle line:84%
So transformers, even
though they were originally

00:00:19.800 --> 00:00:22.080 align:middle line:84%
invented for
machine translation,

00:00:22.080 --> 00:00:24.320 align:middle line:84%
going from English to
German and German to French

00:00:24.320 --> 00:00:26.320 align:middle line:84%
and so on and so
forth, they have

00:00:26.320 --> 00:00:29.600 align:middle line:84%
turned out to be an incredibly
effective deep neural network

00:00:29.600 --> 00:00:33.482 align:middle line:84%
architecture for just, really,
a vast array of domains.

00:00:33.482 --> 00:00:35.440 align:middle line:84%
It has reached a point
where if you're actually

00:00:35.440 --> 00:00:38.640 align:middle line:84%
working on a particular problem,
you almost reflexively will

00:00:38.640 --> 00:00:40.760 align:middle line:84%
try a transformer first,
because it's probably

00:00:40.760 --> 00:00:42.920 align:middle line:90%
going to be pretty darn good.

00:00:42.920 --> 00:00:43.680 align:middle line:90%
OK?

00:00:43.680 --> 00:00:46.480 align:middle line:84%
So they have just
taken over everything.

00:00:46.480 --> 00:00:49.780 align:middle line:84%
And obviously they have
transformed translation,

00:00:49.780 --> 00:00:52.080 align:middle line:90%
which is the original target--

00:00:52.080 --> 00:00:54.220 align:middle line:84%
Google search,
information retrieval,

00:00:54.220 --> 00:00:56.260 align:middle line:84%
completely transformed
speech recognition,

00:00:56.260 --> 00:00:58.517 align:middle line:84%
text to speech, even
computer vision.

00:00:58.517 --> 00:01:00.850 align:middle line:84%
Even the stuff that we learned
with convolutional neural

00:01:00.850 --> 00:01:03.730 align:middle line:84%
networks, now, there are
transformers for computer vision

00:01:03.730 --> 00:01:07.292 align:middle line:84%
problems that are
actually quite good.

00:01:07.292 --> 00:01:09.250 align:middle line:84%
Which is kind of shocking
because they were not

00:01:09.250 --> 00:01:11.290 align:middle line:90%
even designed for that.

00:01:11.290 --> 00:01:13.118 align:middle line:90%
And then reinforcement learning.

00:01:13.118 --> 00:01:15.410 align:middle line:84%
And of course, all the crazy
stuff that's going on with

00:01:15.410 --> 00:01:18.290 align:middle line:84%
generative AI-- large language
models, multimodal models--

00:01:18.290 --> 00:01:21.450 align:middle line:84%
everything-- everything--
runs on a transformer.

00:01:21.450 --> 00:01:25.570 align:middle line:84%
And then there are numerous
special purpose systems.

00:01:25.570 --> 00:01:27.950 align:middle line:84%
And I find these to be
even more interesting.

00:01:27.950 --> 00:01:31.210 align:middle line:84%
Like AlphaFold, the
protein-folding AI--

00:01:31.210 --> 00:01:33.690 align:middle line:90%
runs on a transformer stack.

00:01:33.690 --> 00:01:36.650 align:middle line:84%
And I could just list the
examples one after the other.

00:01:36.650 --> 00:01:37.630 align:middle line:90%
So it's just amazing.

00:01:37.630 --> 00:01:40.690 align:middle line:84%
It's incredibly a
flexible architecture.

00:01:40.690 --> 00:01:43.650 align:middle line:84%
And I think we are lucky
to be alive during a time

00:01:43.650 --> 00:01:47.130 align:middle line:90%
when such a thing was invented.

00:01:47.130 --> 00:01:50.330 align:middle line:84%
And I'm not getting paid to
tell you any of this stuff.

00:01:50.330 --> 00:01:50.830 align:middle line:90%
All right.

00:01:50.830 --> 00:01:51.630 align:middle line:90%
It's just amazing.

00:01:51.630 --> 00:01:53.110 align:middle line:90%
So let's get going.

00:01:53.110 --> 00:01:55.090 align:middle line:90%
We will use search--

00:01:55.090 --> 00:01:57.130 align:middle line:84%
or, more broadly,
information retrieval,

00:01:57.130 --> 00:01:58.717 align:middle line:90%
as the motivating use case.

00:01:58.717 --> 00:02:00.300 align:middle line:84%
So these are all
examples where people

00:02:00.300 --> 00:02:01.883 align:middle line:84%
are typing in natural
language queries

00:02:01.883 --> 00:02:04.040 align:middle line:84%
or uttering natural language
queries into a phone,

00:02:04.040 --> 00:02:06.400 align:middle line:84%
and we need to make
sense of what they want.

00:02:06.400 --> 00:02:09.340 align:middle line:84%
And it's not, like, write me a
limerick about deep learning,

00:02:09.340 --> 00:02:11.800 align:middle line:84%
where there could be many
possible right answers.

00:02:11.800 --> 00:02:14.300 align:middle line:84%
It's more like, OK, tell me all
the flights that are leaving

00:02:14.300 --> 00:02:16.383 align:middle line:84%
from Boston to going to
LaGuardia tomorrow morning

00:02:16.383 --> 00:02:17.400 align:middle line:90%
between 8:00 and 9:00.

00:02:17.400 --> 00:02:18.880 align:middle line:90%
Well, you better get it right.

00:02:18.880 --> 00:02:19.380 align:middle line:90%
OK.

00:02:19.380 --> 00:02:21.140 align:middle line:90%
Accuracy is a high bar.

00:02:21.140 --> 00:02:24.440 align:middle line:84%
Or how many customers
abandon their shopping cart?

00:02:24.440 --> 00:02:27.020 align:middle line:84%
Find all contracts that are
up for renewal next month.

00:02:27.020 --> 00:02:31.140 align:middle line:84%
Tell me all the customers who
ended the phone call to the call

00:02:31.140 --> 00:02:34.260 align:middle line:84%
center yesterday not entirely
pleased with the transaction.

00:02:34.260 --> 00:02:36.140 align:middle line:90%
The list goes on and on.

00:02:36.140 --> 00:02:38.220 align:middle line:84%
And so in particular,
we will focus

00:02:38.220 --> 00:02:40.980 align:middle line:84%
on this travel-related
example today.

00:02:40.980 --> 00:02:44.060 align:middle line:84%
Find me all flights from Boston
to LaGuardia tomorrow morning.

00:02:44.060 --> 00:02:46.020 align:middle line:90%
That kind of query.

00:02:46.020 --> 00:02:48.580 align:middle line:84%
And so in these
sorts of use cases,

00:02:48.580 --> 00:02:51.580 align:middle line:84%
a very common approach
historically has been,

00:02:51.580 --> 00:02:55.630 align:middle line:84%
well, we will take this
natural language query

00:02:55.630 --> 00:02:59.210 align:middle line:84%
and then we will convert
it into a structured query.

00:02:59.210 --> 00:03:01.830 align:middle line:84%
By that, I mean we
will parse the query

00:03:01.830 --> 00:03:05.430 align:middle line:84%
and we'll extract out
key things in that query.

00:03:05.430 --> 00:03:07.330 align:middle line:84%
Once we extract out
those key things,

00:03:07.330 --> 00:03:09.830 align:middle line:84%
we will reassemble it
into a structured query,

00:03:09.830 --> 00:03:12.310 align:middle line:90%
like a SQL query.

00:03:12.310 --> 00:03:15.170 align:middle line:84%
SQL is just one example of
a possible structured query.

00:03:15.170 --> 00:03:17.150 align:middle line:84%
There are many, many ways
to structure queries.

00:03:17.150 --> 00:03:20.250 align:middle line:84%
But SQL is familiar to lots
of people, so I'm using that.

00:03:20.250 --> 00:03:21.810 align:middle line:90%
So you take the SQL.

00:03:21.810 --> 00:03:23.390 align:middle line:84%
Once you have the
SQL query, you're

00:03:23.390 --> 00:03:26.390 align:middle line:84%
in a very comfortable
structured land, in which case

00:03:26.390 --> 00:03:29.370 align:middle line:84%
you just run the query through
some database that you have,

00:03:29.370 --> 00:03:33.050 align:middle line:84%
get the results back, format it
nicely, and show it to the user.

00:03:33.050 --> 00:03:34.710 align:middle line:90%
That's the flow.

00:03:34.710 --> 00:03:39.150 align:middle line:84%
So the question becomes,
how do we automatically

00:03:39.150 --> 00:03:43.910 align:middle line:84%
extract all the travel-related
entities from this query?

00:03:43.910 --> 00:03:48.870 align:middle line:84%
We want to be able to extract
BOS, LGA, tomorrow morning.

00:03:48.870 --> 00:03:49.650 align:middle line:90%
Flights.

00:03:49.650 --> 00:03:50.150 align:middle line:90%
So on.

00:03:50.150 --> 00:03:52.858 align:middle line:84%
These are all the travel-related
entities we want to extract out.

00:03:52.858 --> 00:03:54.840 align:middle line:90%
That's the problem.

00:03:54.840 --> 00:03:59.040 align:middle line:84%
And so we will use a really
cool data set called the Airline

00:03:59.040 --> 00:04:00.740 align:middle line:84%
Travel Information
Systems Data Set--

00:04:00.740 --> 00:04:02.660 align:middle line:84%
and I'll explain the
data set in just a bit.

00:04:02.660 --> 00:04:05.160 align:middle line:84%
We'll use this as the
basis for this example.

00:04:05.160 --> 00:04:07.240 align:middle line:84%
And so the way we
think about it is

00:04:07.240 --> 00:04:12.120 align:middle line:84%
that we have a whole bunch
of queries in this data set.

00:04:12.120 --> 00:04:14.400 align:middle line:84%
And fortunately for
us, the researchers

00:04:14.400 --> 00:04:17.839 align:middle line:84%
who compiled this data set,
they went through every one

00:04:17.839 --> 00:04:19.360 align:middle line:90%
of these queries.

00:04:19.360 --> 00:04:21.920 align:middle line:84%
And we have several
thousands of them.

00:04:21.920 --> 00:04:24.000 align:middle line:84%
They went through every
one of those queries,

00:04:24.000 --> 00:04:26.640 align:middle line:84%
and they manually
tagged each word

00:04:26.640 --> 00:04:31.520 align:middle line:84%
in the query with what kind
of travel entity it is--

00:04:31.520 --> 00:04:32.620 align:middle line:90%
or none of them.

00:04:32.620 --> 00:04:36.100 align:middle line:84%
So for instance, so
they call them slots.

00:04:36.100 --> 00:04:38.120 align:middle line:84%
So they will take
each word in the query

00:04:38.120 --> 00:04:41.200 align:middle line:84%
and assign it to a slot,
a particular kind of slot.

00:04:41.200 --> 00:04:43.460 align:middle line:84%
And I'll explain what "slot"
means in just a second.

00:04:43.460 --> 00:04:44.740 align:middle line:90%
That's the basic idea.

00:04:44.740 --> 00:04:47.720 align:middle line:84%
So for example, if you
have something like,

00:04:47.720 --> 00:04:49.800 align:middle line:90%
I want to fly from--

00:04:49.800 --> 00:04:50.480 align:middle line:90%
OK?

00:04:50.480 --> 00:04:52.690 align:middle line:84%
And this is a
flights database, so

00:04:52.690 --> 00:04:55.530 align:middle line:84%
you, can assume that everything
is related to a flight flying.

00:04:55.530 --> 00:04:58.610 align:middle line:84%
So if you have all these
words-- 'I want to fly from"--

00:04:58.610 --> 00:05:00.770 align:middle line:84%
each of these words,
these five words,

00:05:00.770 --> 00:05:04.610 align:middle line:84%
gets mapped to something called
the O-- which means other.

00:05:04.610 --> 00:05:06.050 align:middle line:90%
It's the other slot.

00:05:06.050 --> 00:05:07.310 align:middle line:90%
We don't really care about it.

00:05:07.310 --> 00:05:09.250 align:middle line:90%
It's the other slot.

00:05:09.250 --> 00:05:11.610 align:middle line:90%
And then we come to Boston.

00:05:11.610 --> 00:05:13.450 align:middle line:84%
Oh, Boston is very
special, right?

00:05:13.450 --> 00:05:16.810 align:middle line:84%
Because it's clearly
a departure city.

00:05:16.810 --> 00:05:18.270 align:middle line:90%
So we actually tag it.

00:05:18.270 --> 00:05:19.585 align:middle line:90%
We assign it this label.

00:05:19.585 --> 00:05:21.710 align:middle line:84%
Think of it as just like
a classification problem--

00:05:21.710 --> 00:05:23.810 align:middle line:84%
a multi-class
classification problem.

00:05:23.810 --> 00:05:29.250 align:middle line:84%
So we assign it to
B-fromloc.city_name.

00:05:29.250 --> 00:05:31.210 align:middle line:90%
That is the label you assign it.

00:05:31.210 --> 00:05:34.290 align:middle line:90%
And then you go to at.

00:05:34.290 --> 00:05:35.290 align:middle line:90%
You don't care about at.

00:05:35.290 --> 00:05:36.770 align:middle line:90%
It's O-- other.

00:05:36.770 --> 00:05:38.650 align:middle line:90%
You come to 7:00 AM.

00:05:38.650 --> 00:05:40.910 align:middle line:84%
And then, OK, that
is depart time.

00:05:40.910 --> 00:05:42.330 align:middle line:90%
So depart time.

00:05:42.330 --> 00:05:43.670 align:middle line:90%
And then another depart time.

00:05:43.670 --> 00:05:48.130 align:middle line:84%
And here you see there is a B
and then there is an I, right?

00:05:48.130 --> 00:05:51.860 align:middle line:84%
So what we are saying here is
that there could be entities who

00:05:51.860 --> 00:05:55.940 align:middle line:84%
are described using more
than one word, like 7:00 AM--

00:05:55.940 --> 00:05:57.315 align:middle line:90%
two tokens.

00:05:57.315 --> 00:05:59.440 align:middle line:84%
And for that, we need to
be able to figure out, OK.

00:05:59.440 --> 00:06:01.780 align:middle line:90%
The second token is really--

00:06:01.780 --> 00:06:03.280 align:middle line:90%
is part of the first token.

00:06:03.280 --> 00:06:06.100 align:middle line:84%
Together, they define the
notion of a departure time.

00:06:06.100 --> 00:06:09.960 align:middle line:84%
So what the B means that
is that this is the word,

00:06:09.960 --> 00:06:11.860 align:middle line:84%
this is the token in
which we are beginning

00:06:11.860 --> 00:06:13.700 align:middle line:90%
the idea of a departure time.

00:06:13.700 --> 00:06:17.940 align:middle line:84%
And then "I" means we are in
the middle of this description.

00:06:17.940 --> 00:06:19.980 align:middle line:90%
B is for Beginning.

00:06:19.980 --> 00:06:23.260 align:middle line:84%
So you can see here,
so there is a B here.

00:06:23.260 --> 00:06:24.220 align:middle line:90%
And there is an I--

00:06:24.220 --> 00:06:28.100 align:middle line:84%
B for Beginning, I for
Intermediate, or in the middle.

00:06:28.100 --> 00:06:30.580 align:middle line:90%
And then "at," we don't care.

00:06:30.580 --> 00:06:33.960 align:middle line:90%
11, B-arrive_time.

00:06:33.960 --> 00:06:38.260 align:middle line:90%
Morning, arrive_time.period.

00:06:38.260 --> 00:06:41.940 align:middle line:84%
So this is an example of
how you can take a sentence

00:06:41.940 --> 00:06:44.118 align:middle line:84%
and then manually
label every word

00:06:44.118 --> 00:06:45.660 align:middle line:84%
in the sentence with
something that's

00:06:45.660 --> 00:06:47.160 align:middle line:84%
relevant to your
particular problem.

00:06:47.160 --> 00:06:50.390 align:middle line:90%


00:06:50.390 --> 00:06:55.150 align:middle line:84%
And it turns out, these
people, every word

00:06:55.150 --> 00:06:59.510 align:middle line:84%
is classified into one
of 123 possibilities.

00:06:59.510 --> 00:07:00.750 align:middle line:90%
OK?

00:07:00.750 --> 00:07:04.290 align:middle line:84%
So aircraft code, airline
code, airline name,

00:07:04.290 --> 00:07:07.570 align:middle line:84%
airport code. airport name,
arrival date, relative name.

00:07:07.570 --> 00:07:08.950 align:middle line:90%
Now, you get the idea.

00:07:08.950 --> 00:07:11.590 align:middle line:84%
They want a round
trip versus a one way.

00:07:11.590 --> 00:07:12.708 align:middle line:90%
The relative to today--

00:07:12.708 --> 00:07:15.250 align:middle line:84%
Because some would say tomorrow
morning is relative to today.

00:07:15.250 --> 00:07:17.430 align:middle line:84%
So you need absolute
time, and you

00:07:17.430 --> 00:07:19.550 align:middle line:90%
need a notion of relative time.

00:07:19.550 --> 00:07:21.950 align:middle line:84%
So they basically thought
of every possibility

00:07:21.950 --> 00:07:23.350 align:middle line:90%
with these researchers.

00:07:23.350 --> 00:07:26.230 align:middle line:84%
And so every word in
every one of these queries

00:07:26.230 --> 00:07:28.260 align:middle line:84%
is assigned one of
these 123 labels.

00:07:28.260 --> 00:07:32.270 align:middle line:90%


00:07:32.270 --> 00:07:33.400 align:middle line:90%
Any questions on the setup?

00:07:33.400 --> 00:07:40.230 align:middle line:90%


00:07:40.230 --> 00:07:42.030 align:middle line:84%
AUDIENCE: Did they
have to contextualize

00:07:42.030 --> 00:07:44.590 align:middle line:84%
what comes before
relative Boston?

00:07:44.590 --> 00:07:47.880 align:middle line:84%
So if someone says from
Boston, so that there should be

00:07:47.880 --> 00:07:50.160 align:middle line:84%
contextualization with
the "from Boston."

00:07:50.160 --> 00:07:51.480 align:middle line:84%
RAMA RAMAKRISHNAN: So
because they did it manually,

00:07:51.480 --> 00:07:53.230 align:middle line:84%
they could just read
it and figure it out,

00:07:53.230 --> 00:07:54.160 align:middle line:90%
that's what they mean.

00:07:54.160 --> 00:07:54.700 align:middle line:90%
Right?

00:07:54.700 --> 00:07:57.180 align:middle line:84%
Boston is the departure city
and not the arrival city.

00:07:57.180 --> 00:07:59.097 align:middle line:84%
AUDIENCE: So do they
have to attach to Boston,

00:07:59.097 --> 00:08:02.680 align:middle line:84%
which is some departure city
as well as arrival city with

00:08:02.680 --> 00:08:04.027 align:middle line:90%
the word "Boston"?

00:08:04.027 --> 00:08:05.860 align:middle line:84%
RAMA RAMAKRISHNAN: In
the particular phrase,

00:08:05.860 --> 00:08:08.480 align:middle line:84%
it's clear from that particular
case in the context of it,

00:08:08.480 --> 00:08:11.220 align:middle line:84%
as a human reading it, that
Boston is a departure city.

00:08:11.220 --> 00:08:14.360 align:middle line:84%
So it just only gets that
tag in that sentence.

00:08:14.360 --> 00:08:16.860 align:middle line:84%
In some other sentence, where
people are coming into Boston,

00:08:16.860 --> 00:08:18.068 align:middle line:90%
it will have a different tag.

00:08:18.068 --> 00:08:21.200 align:middle line:90%


00:08:21.200 --> 00:08:23.780 align:middle line:84%
AUDIENCE: I was wondering if
my query, like the others,

00:08:23.780 --> 00:08:28.080 align:middle line:84%
basically there is, for example,
if my query was "give me flights

00:08:28.080 --> 00:08:32.686 align:middle line:84%
from Boston at 7:00 AM," and
flights from Denver at 11:00 AM.

00:08:32.686 --> 00:08:34.769 align:middle line:84%
RAMA RAMAKRISHNAN: You
mean like a compound query?

00:08:34.769 --> 00:08:35.520 align:middle line:90%
AUDIENCE: Yeah?

00:08:35.520 --> 00:08:37.228 align:middle line:84%
RAMA RAMAKRISHNAN: So
this one only takes

00:08:37.228 --> 00:08:40.159 align:middle line:84%
single queries into account,
because most people are like,

00:08:40.159 --> 00:08:41.659 align:middle line:84%
give me a flight
from here to there,

00:08:41.659 --> 00:08:43.701 align:middle line:84%
or what is the cheapest
thing from here to there?

00:08:43.701 --> 00:08:46.070 align:middle line:84%
And we'll see examples
of queries later on.

00:08:46.070 --> 00:08:50.010 align:middle line:90%


00:08:50.010 --> 00:08:51.530 align:middle line:90%
OK.

00:08:51.530 --> 00:08:52.030 align:middle line:90%
All right.

00:08:52.030 --> 00:08:53.610 align:middle line:90%
So that's the deal.

00:08:53.610 --> 00:08:59.290 align:middle line:84%
So basically, what we have,
this problem that we have here

00:08:59.290 --> 00:09:02.170 align:middle line:90%
is really a word to slot--

00:09:02.170 --> 00:09:06.450 align:middle line:84%
word to slot-- multi-class
classification problem.

00:09:06.450 --> 00:09:09.447 align:middle line:84%
OK because if you
look at that input,

00:09:09.447 --> 00:09:11.030 align:middle line:84%
we want to be able
to take that input.

00:09:11.030 --> 00:09:12.890 align:middle line:84%
And a really good
model will then

00:09:12.890 --> 00:09:17.730 align:middle line:84%
give you this as the
output, because this is

00:09:17.730 --> 00:09:20.250 align:middle line:90%
what a human would have done.

00:09:20.250 --> 00:09:23.518 align:middle line:90%
So that is our problem.

00:09:23.518 --> 00:09:26.382 align:middle line:90%
So the question is--

00:09:26.382 --> 00:09:28.090 align:middle line:84%
the key thing here is
that each of the 18

00:09:28.090 --> 00:09:31.010 align:middle line:84%
words in this particular
example must be assigned

00:09:31.010 --> 00:09:33.030 align:middle line:90%
to one of 123 slot types--

00:09:33.030 --> 00:09:34.230 align:middle line:90%
each word.

00:09:34.230 --> 00:09:36.250 align:middle line:84%
It's not like we
take the entire query

00:09:36.250 --> 00:09:39.670 align:middle line:84%
and classify the entire query
into one of 123 possibilities.

00:09:39.670 --> 00:09:42.490 align:middle line:84%
Every word in the query
has to be classified.

00:09:42.490 --> 00:09:45.460 align:middle line:90%
That is the wrinkle.

00:09:45.460 --> 00:09:47.020 align:middle line:90%
OK?

00:09:47.020 --> 00:09:49.260 align:middle line:84%
So now, if we
could run the query

00:09:49.260 --> 00:09:54.140 align:middle line:84%
through a deep neural network,
and generate 18 output nodes,

00:09:54.140 --> 00:09:56.760 align:middle line:84%
it goes through some
unspecified deep neural network.

00:09:56.760 --> 00:09:58.260 align:middle line:84%
And when it comes
out the other end,

00:09:58.260 --> 00:10:00.420 align:middle line:90%
the output layer has 18 nodes.

00:10:00.420 --> 00:10:01.380 align:middle line:90%
OK?

00:10:01.380 --> 00:10:05.840 align:middle line:84%
Because that is the dimension of
the output that we care about.

00:10:05.840 --> 00:10:07.300 align:middle line:90%
18 in, 18 out.

00:10:07.300 --> 00:10:08.690 align:middle line:90%
18 in, 18 out.

00:10:08.690 --> 00:10:11.760 align:middle line:84%
And then, for each
one of those 18 nodes,

00:10:11.760 --> 00:10:16.610 align:middle line:84%
maybe we could attach a 123-way
softmax to each of those 18

00:10:16.610 --> 00:10:17.110 align:middle line:90%
outputs.

00:10:17.110 --> 00:10:20.180 align:middle line:90%


00:10:20.180 --> 00:10:22.900 align:middle line:84%
By the way, isn't it cool that
we can just casually talk about

00:10:22.900 --> 00:10:27.140 align:middle line:84%
sticking a 123-way softmax
onto each one of 18 nodes?

00:10:27.140 --> 00:10:28.120 align:middle line:90%
Folks, wake up.

00:10:28.120 --> 00:10:31.340 align:middle line:90%


00:10:31.340 --> 00:10:32.640 align:middle line:90%
You're not easily impressed.

00:10:32.640 --> 00:10:34.820 align:middle line:90%
I'm impressed by that.

00:10:34.820 --> 00:10:37.940 align:middle line:90%
So, OK.

00:10:37.940 --> 00:10:40.400 align:middle line:90%
So here's the key thing, right.

00:10:40.400 --> 00:10:42.350 align:middle line:84%
We want to generate
an output that has

00:10:42.350 --> 00:10:45.610 align:middle line:90%
the same length as the input.

00:10:45.610 --> 00:10:48.110 align:middle line:84%
But the problem is, the inputs
could be of different lengths

00:10:48.110 --> 00:10:49.050 align:middle line:90%
as they come in.

00:10:49.050 --> 00:10:51.090 align:middle line:84%
There could be short
sentences, large sentences.

00:10:51.090 --> 00:10:52.830 align:middle line:90%
We don't know.

00:10:52.830 --> 00:10:56.270 align:middle line:84%
Yet we need to accommodate
this range, this variable size

00:10:56.270 --> 00:10:57.852 align:middle line:90%
of input that's coming in.

00:10:57.852 --> 00:10:59.310 align:middle line:84%
But the key thing
is the output has

00:10:59.310 --> 00:11:00.768 align:middle line:84%
to be the same
thing as the input--

00:11:00.768 --> 00:11:02.490 align:middle line:84%
the same cardinality
as the input.

00:11:02.490 --> 00:11:02.990 align:middle line:90%
OK?

00:11:02.990 --> 00:11:05.510 align:middle line:90%
That's a one big requirement.

00:11:05.510 --> 00:11:08.270 align:middle line:84%
In addition, we want to
take the surrounding context

00:11:08.270 --> 00:11:09.390 align:middle line:90%
of each word into account.

00:11:09.390 --> 00:11:12.652 align:middle line:84%
To go to [INAUDIBLE] when
you see the word "Boston,"

00:11:12.652 --> 00:11:15.110 align:middle line:84%
you can't conclude whether it's
a departure city or arrival

00:11:15.110 --> 00:11:15.870 align:middle line:90%
city.

00:11:15.870 --> 00:11:18.190 align:middle line:84%
You have to look at what
else is going on around it.

00:11:18.190 --> 00:11:19.962 align:middle line:84%
Is there a "from,"
is there a "to."

00:11:19.962 --> 00:11:20.670 align:middle line:90%
Things like that.

00:11:20.670 --> 00:11:22.250 align:middle line:90%
To figure out how to tag it.

00:11:22.250 --> 00:11:24.258 align:middle line:90%
So clearly, the context matters.

00:11:24.258 --> 00:11:26.550 align:middle line:84%
And then we clearly have to
take the order of the words

00:11:26.550 --> 00:11:27.270 align:middle line:90%
into account.

00:11:27.270 --> 00:11:29.430 align:middle line:84%
Going from Boston
to LaGuardia is

00:11:29.430 --> 00:11:31.670 align:middle line:84%
very different than going
from LaGuardia to Boston.

00:11:31.670 --> 00:11:34.110 align:middle line:90%
So clearly the order matters.

00:11:34.110 --> 00:11:36.530 align:middle line:84%
So the context matters
and the order matters.

00:11:36.530 --> 00:11:38.780 align:middle line:84%
And the output has to be the
same length as the input.

00:11:38.780 --> 00:11:42.560 align:middle line:90%


00:11:42.560 --> 00:11:43.880 align:middle line:90%
So context matters.

00:11:43.880 --> 00:11:45.720 align:middle line:90%
Just a few fun examples.

00:11:45.720 --> 00:11:48.280 align:middle line:84%
Remember, from the last week,
that the meaning of a word

00:11:48.280 --> 00:11:50.700 align:middle line:84%
can change dramatically
depending on the context.

00:11:50.700 --> 00:11:54.200 align:middle line:84%
And we also saw that the
standalone or uncontextual

00:11:54.200 --> 00:11:57.080 align:middle line:84%
embeddings that we saw
last week, like "glove,"

00:11:57.080 --> 00:12:01.000 align:middle line:84%
they don't take context into
account because they give

00:12:01.000 --> 00:12:04.000 align:middle line:84%
a single unique embedding
vector to every word.

00:12:04.000 --> 00:12:06.980 align:middle line:84%
And if a word ends up having
lots of different meanings,

00:12:06.980 --> 00:12:11.680 align:middle line:84%
that vector is some mushy
average of all those meanings.

00:12:11.680 --> 00:12:15.238 align:middle line:84%
So the word "see," "I
will see you soon."

00:12:15.238 --> 00:12:16.780 align:middle line:84%
"I will see this
project to its end."

00:12:16.780 --> 00:12:17.860 align:middle line:90%
"I see what you mean."

00:12:17.860 --> 00:12:20.200 align:middle line:84%
Very different meanings
of the word "see."

00:12:20.200 --> 00:12:22.192 align:middle line:90%
This is my favorite-- bank.

00:12:22.192 --> 00:12:23.900 align:middle line:84%
"I went to the bank
to apply for a loan."

00:12:23.900 --> 00:12:24.942 align:middle line:90%
"I'm banking on the job."

00:12:24.942 --> 00:12:27.400 align:middle line:84%
I'm standing on the
left bank," and so on.

00:12:27.400 --> 00:12:29.840 align:middle line:84%
It, the animal or--
this actually very--

00:12:29.840 --> 00:12:30.710 align:middle line:90%
it's a good one.

00:12:30.710 --> 00:12:32.960 align:middle line:84%
"The animal didn't cross the
street because it was too

00:12:32.960 --> 00:12:33.660 align:middle line:90%
tired."

00:12:33.660 --> 00:12:37.250 align:middle line:84%
"The animal didn't cross the
street because it was too wide."

00:12:37.250 --> 00:12:40.825 align:middle line:84%
Can you imagine a deep neural
network, looking at this word,

00:12:40.825 --> 00:12:43.450 align:middle line:84%
"it," and trying to figure out
what the heck does the word "it"

00:12:43.450 --> 00:12:44.370 align:middle line:90%
mean?

00:12:44.370 --> 00:12:46.130 align:middle line:90%
What is it referring to?

00:12:46.130 --> 00:12:49.130 align:middle line:90%
Tricky, right?

00:12:49.130 --> 00:12:51.250 align:middle line:84%
And then, if you take
the word "station"--

00:12:51.250 --> 00:12:53.250 align:middle line:84%
and I have this station example
here because we're going to use

00:12:53.250 --> 00:12:55.210 align:middle line:84%
it a bit more in the
rest of the lecture--

00:12:55.210 --> 00:12:59.170 align:middle line:84%
the station could be a radio
station, a train station,

00:12:59.170 --> 00:13:01.790 align:middle line:84%
being stationed somewhere, the
International Space Station.

00:13:01.790 --> 00:13:03.010 align:middle line:90%
The list goes on.

00:13:03.010 --> 00:13:04.410 align:middle line:90%
So clearly, order matters--

00:13:04.410 --> 00:13:06.010 align:middle line:90%
context matters.

00:13:06.010 --> 00:13:10.045 align:middle line:90%
And clearly order matters.

00:13:10.045 --> 00:13:11.670 align:middle line:84%
You can come up with
your own examples.

00:13:11.670 --> 00:13:13.330 align:middle line:90%
Let's keep moving.

00:13:13.330 --> 00:13:15.170 align:middle line:90%
OK?

00:13:15.170 --> 00:13:22.130 align:middle line:84%
So the transformer architecture
is a very elegant architecture

00:13:22.130 --> 00:13:26.396 align:middle line:84%
which checks these
three boxes beautifully.

00:13:26.396 --> 00:13:28.830 align:middle line:84%
It takes the context into
account, order into account.

00:13:28.830 --> 00:13:32.010 align:middle line:84%
And then whatever is
produced out there

00:13:32.010 --> 00:13:34.370 align:middle line:84%
is the same length as
whatever is coming in.

00:13:34.370 --> 00:13:36.940 align:middle line:84%
And the reason it's
called the transformer,

00:13:36.940 --> 00:13:41.000 align:middle line:84%
it is because if 10 things
come in, 10 things go out.

00:13:41.000 --> 00:13:43.700 align:middle line:84%
But the 10 things that go
out are a transformed version

00:13:43.700 --> 00:13:45.980 align:middle line:90%
of the 10 things that came in.

00:13:45.980 --> 00:13:48.020 align:middle line:84%
That's why it's called
the transformer.

00:13:48.020 --> 00:13:51.680 align:middle line:84%
If 10 things came in, and
like one thing goes out,

00:13:51.680 --> 00:13:53.280 align:middle line:84%
well, sure, it's
been transformed.

00:13:53.280 --> 00:13:54.040 align:middle line:90%
But what is it?

00:13:54.040 --> 00:13:55.320 align:middle line:90%
It's some weird thing.

00:13:55.320 --> 00:13:58.640 align:middle line:84%
But when 10 comes in and 10
goes out, the 10 is preserved.

00:13:58.640 --> 00:14:00.980 align:middle line:84%
Each one is getting transformed
in an interesting way.

00:14:00.980 --> 00:14:04.500 align:middle line:84%
That's why it's
called a transformer.

00:14:04.500 --> 00:14:08.420 align:middle line:84%
So developed in 2017,
this dramatic impact.

00:14:08.420 --> 00:14:11.100 align:middle line:84%
So by the way, the
effect of transformer--

00:14:11.100 --> 00:14:13.300 align:middle line:84%
Google had spent
a lot of research

00:14:13.300 --> 00:14:16.380 align:middle line:84%
on machine translation
and, obviously, search.

00:14:16.380 --> 00:14:19.020 align:middle line:84%
And then, when the
transformer is invented,

00:14:19.020 --> 00:14:21.940 align:middle line:84%
they took a model called
Bert, which we will

00:14:21.940 --> 00:14:23.620 align:middle line:90%
see on Wednesday in detail.

00:14:23.620 --> 00:14:27.720 align:middle line:84%
And then they introduced
Bert into their search.

00:14:27.720 --> 00:14:29.620 align:middle line:90%
And the results were dramatic.

00:14:29.620 --> 00:14:31.900 align:middle line:84%
And from what I've
read, apparently

00:14:31.900 --> 00:14:34.310 align:middle line:90%
the impact of doing that was--

00:14:34.310 --> 00:14:36.450 align:middle line:84%
Typically, when you make
an improvement to search,

00:14:36.450 --> 00:14:38.075 align:middle line:84%
the improvement is
very, very marginal,

00:14:38.075 --> 00:14:40.690 align:middle line:84%
because it's already a very
heavily optimized system.

00:14:40.690 --> 00:14:42.690 align:middle line:84%
And then, when the
transformer thing came along,

00:14:42.690 --> 00:14:45.330 align:middle line:84%
there was actually a significant
jump in search quality.

00:14:45.330 --> 00:14:47.110 align:middle line:84%
So for example-- and
you can actually read

00:14:47.110 --> 00:14:50.430 align:middle line:84%
this blog post which came
out when they introduced Bert

00:14:50.430 --> 00:14:51.290 align:middle line:90%
into search.

00:14:51.290 --> 00:14:52.830 align:middle line:90%
It gives you a bit more detail.

00:14:52.830 --> 00:14:58.550 align:middle line:84%
But here, so if you queried
something like "Brazil traveler

00:14:58.550 --> 00:15:02.098 align:middle line:84%
to USA needs a visa," you would
think that it should give you

00:15:02.098 --> 00:15:04.390 align:middle line:84%
information about how to get
a visa if you're Brazilian

00:15:04.390 --> 00:15:05.910 align:middle line:90%
and want to come to the US.

00:15:05.910 --> 00:15:07.950 align:middle line:84%
But it turns out the
first result was,

00:15:07.950 --> 00:15:13.270 align:middle line:84%
how US citizens going to
Brazil can get a visa.

00:15:13.270 --> 00:15:16.990 align:middle line:84%
So, clearly, it's not taking
the order into account.

00:15:16.990 --> 00:15:19.630 align:middle line:84%
But once they introduced
it, boom, the first thing

00:15:19.630 --> 00:15:24.150 align:middle line:84%
was the US Embassy in Brazil,
and a page on how to get a visa.

00:15:24.150 --> 00:15:26.830 align:middle line:90%
So the effect was dramatic.

00:15:26.830 --> 00:15:30.150 align:middle line:90%
And so this is a seminal paper.

00:15:30.150 --> 00:15:32.280 align:middle line:84%
And it's actually worth
reading the paper.

00:15:32.280 --> 00:15:35.340 align:middle line:84%
And it's worth-- and
this is the picture.

00:15:35.340 --> 00:15:37.810 align:middle line:84%
This is, like, an iconic
picture at this point,

00:15:37.810 --> 00:15:39.560 align:middle line:90%
in the deep learning community.

00:15:39.560 --> 00:15:41.640 align:middle line:84%
And we will actually
understand this picture

00:15:41.640 --> 00:15:43.872 align:middle line:90%
by the end of Wednesday.

00:15:43.872 --> 00:15:46.580 align:middle line:84%
But the funny thing is that when
the researchers came up with it,

00:15:46.580 --> 00:15:48.560 align:middle line:84%
they didn't realize,
in some sense,

00:15:48.560 --> 00:15:51.120 align:middle line:84%
like, what they had stumbled
on, because they were really

00:15:51.120 --> 00:15:52.927 align:middle line:90%
focused on machine translation.

00:15:52.927 --> 00:15:54.760 align:middle line:84%
It's only the rest of
the research community

00:15:54.760 --> 00:15:56.927 align:middle line:84%
that took it and started
applying to everything else

00:15:56.927 --> 00:15:59.640 align:middle line:84%
and found it to be
really, really effective.

00:15:59.640 --> 00:16:01.722 align:middle line:84%
So we're going to take
each one of these things

00:16:01.722 --> 00:16:03.680 align:middle line:84%
and figure out how to
address them, and thereby

00:16:03.680 --> 00:16:05.440 align:middle line:90%
build up the architecture.

00:16:05.440 --> 00:16:07.660 align:middle line:90%
Any questions before I continue?

00:16:07.660 --> 00:16:08.160 align:middle line:90%
Yeah.

00:16:08.160 --> 00:16:11.200 align:middle line:90%


00:16:11.200 --> 00:16:15.520 align:middle line:84%
AUDIENCE: Is there any
benefits to discarding

00:16:15.520 --> 00:16:18.900 align:middle line:84%
some of those unclassified
O's before it goes out?

00:16:18.900 --> 00:16:21.822 align:middle line:84%
So, rather than going like
you have 18-word input.

00:16:21.822 --> 00:16:24.280 align:middle line:84%
Discarding all the ones that
don't actually matter and just

00:16:24.280 --> 00:16:26.213 align:middle line:84%
doing, like, an 8-word
output, for example?

00:16:26.213 --> 00:16:28.880 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, yeah, I
think that's a totally fine way

00:16:28.880 --> 00:16:29.690 align:middle line:90%
to think about it.

00:16:29.690 --> 00:16:31.050 align:middle line:84%
Basically what you're
saying is that can we

00:16:31.050 --> 00:16:32.350 align:middle line:90%
have a two-stage model.

00:16:32.350 --> 00:16:35.458 align:middle line:84%
The first stage model is
like a O, non-O classifier,

00:16:35.458 --> 00:16:37.750 align:middle line:84%
and the second stage model
only goes after the non-O's.

00:16:37.750 --> 00:16:39.330 align:middle line:84%
That's a totally
fine way to do it.

00:16:39.330 --> 00:16:40.250 align:middle line:90%
Yeah.

00:16:40.250 --> 00:16:42.090 align:middle line:84%
But as you can
see, even if you go

00:16:42.090 --> 00:16:44.870 align:middle line:84%
with just a simple one-stage
model, if you use a transformer,

00:16:44.870 --> 00:16:47.130 align:middle line:90%
you get fantastic accuracy.

00:16:47.130 --> 00:16:50.690 align:middle line:90%
And we'll do the Colab in a bit.

00:16:50.690 --> 00:16:51.390 align:middle line:90%
All right.

00:16:51.390 --> 00:16:52.670 align:middle line:90%
So let's take the first thing.

00:16:52.670 --> 00:16:55.290 align:middle line:84%
How do you take the context
of everything around the word

00:16:55.290 --> 00:16:57.050 align:middle line:90%
into account?

00:16:57.050 --> 00:17:02.450 align:middle line:84%
So let's say that this is the
sentence we have "The train

00:17:02.450 --> 00:17:05.010 align:middle line:90%
slowly left the station."

00:17:05.010 --> 00:17:08.290 align:middle line:84%
For each of these
words, we can calculate

00:17:08.290 --> 00:17:09.890 align:middle line:90%
a standalone embedding--

00:17:09.890 --> 00:17:11.329 align:middle line:90%
say, something like "left."

00:17:11.329 --> 00:17:14.530 align:middle line:84%
So I'm just depicting
these standalone embeddings

00:17:14.530 --> 00:17:17.980 align:middle line:90%
using these thingies here.

00:17:17.980 --> 00:17:19.730 align:middle line:84%
Please appreciate them,
because it took me

00:17:19.730 --> 00:17:22.550 align:middle line:84%
a while to get them
to do in PowerPoint.

00:17:22.550 --> 00:17:24.589 align:middle line:90%
So these are w1 through w6.

00:17:24.589 --> 00:17:28.339 align:middle line:84%
These are the vectors
standing up Now let's

00:17:28.339 --> 00:17:30.420 align:middle line:84%
say that-- so we
can easily do that.

00:17:30.420 --> 00:17:33.220 align:middle line:84%
Now what we want to figure out
is we want to focus on the word

00:17:33.220 --> 00:17:34.533 align:middle line:90%
"station."

00:17:34.533 --> 00:17:36.700 align:middle line:84%
And since "station" could
mean very different things

00:17:36.700 --> 00:17:38.740 align:middle line:84%
in different contexts,
we want to figure out,

00:17:38.740 --> 00:17:42.140 align:middle line:84%
how do we actually take
"station's" embedding

00:17:42.140 --> 00:17:45.460 align:middle line:84%
and contextualize it using all
the other words that are going

00:17:45.460 --> 00:17:47.400 align:middle line:90%
on in that sentence.

00:17:47.400 --> 00:17:49.780 align:middle line:90%
Clearly, it's a train station.

00:17:49.780 --> 00:17:52.460 align:middle line:84%
So we need to take the fact
that there is a train involved

00:17:52.460 --> 00:17:55.000 align:middle line:84%
to alter the embedding
of the word "station."

00:17:55.000 --> 00:17:58.780 align:middle line:84%
That's what taking context
into account actually means.

00:17:58.780 --> 00:18:03.380 align:middle line:84%
So how can we modify
"station's" embedding so that it

00:18:03.380 --> 00:18:05.017 align:middle line:84%
incorporates all
the other words?

00:18:05.017 --> 00:18:05.850 align:middle line:90%
That's the question.

00:18:05.850 --> 00:18:08.860 align:middle line:90%


00:18:08.860 --> 00:18:13.980 align:middle line:84%
So when you look at it this way,
imagine, just for for a moment--

00:18:13.980 --> 00:18:15.700 align:middle line:90%
just for a moment--

00:18:15.700 --> 00:18:17.660 align:middle line:90%
that we--

00:18:17.660 --> 00:18:19.880 align:middle line:84%
Now some of the other words
sentence don't matter.

00:18:19.880 --> 00:18:22.220 align:middle line:84%
The word "the" probably
doesn't matter.

00:18:22.220 --> 00:18:25.140 align:middle line:84%
But some of the other words
like "train," "slowly," "left"

00:18:25.140 --> 00:18:26.710 align:middle line:90%
probably does matter.

00:18:26.710 --> 00:18:29.230 align:middle line:84%
And suppose, just
magically, we have

00:18:29.230 --> 00:18:32.550 align:middle line:84%
been told all the other
words in the sentence,

00:18:32.550 --> 00:18:35.310 align:middle line:84%
this is how much weight
you have to give to them.

00:18:35.310 --> 00:18:36.730 align:middle line:90%
These don't give it any weight.

00:18:36.730 --> 00:18:37.990 align:middle line:90%
Those give it a lot of weight.

00:18:37.990 --> 00:18:39.813 align:middle line:90%
Suppose we are told that.

00:18:39.813 --> 00:18:41.230 align:middle line:84%
Or to put it another
way, and this

00:18:41.230 --> 00:18:44.230 align:middle line:84%
is the word that's heavily
used in the literature--

00:18:44.230 --> 00:18:46.070 align:middle line:84%
someone tells you
how much attention

00:18:46.070 --> 00:18:47.815 align:middle line:84%
to pay to the other
words, whether you

00:18:47.815 --> 00:18:49.190 align:middle line:84%
have to pay it a
lot of attention

00:18:49.190 --> 00:18:51.310 align:middle line:90%
or very little attention.

00:18:51.310 --> 00:18:53.910 align:middle line:84%
And this, how much
attention to pay,

00:18:53.910 --> 00:18:56.010 align:middle line:84%
is given in the form of a
weight that you can use.

00:18:56.010 --> 00:19:01.150 align:middle line:84%
So if you look at it that way,
from this notion of which word

00:19:01.150 --> 00:19:04.070 align:middle line:84%
should I give a lot of weight
to and very little weight to,

00:19:04.070 --> 00:19:05.790 align:middle line:84%
in this example,
intuitively, which words

00:19:05.790 --> 00:19:07.130 align:middle line:84%
do you think should
get the most weight,

00:19:07.130 --> 00:19:09.790 align:middle line:84%
and which words do you think
should get the least weight?

00:19:09.790 --> 00:19:10.590 align:middle line:90%
Yeah.

00:19:10.590 --> 00:19:11.390 align:middle line:90%
AUDIENCE: "Train."

00:19:11.390 --> 00:19:11.990 align:middle line:90%
RAMA RAMAKRISHNAN: "Train."

00:19:11.990 --> 00:19:12.950 align:middle line:90%
Correct.

00:19:12.950 --> 00:19:14.397 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:19:14.397 --> 00:19:16.230 align:middle line:84%
RAMA RAMAKRISHNAN: You
can do one at a time.

00:19:16.230 --> 00:19:16.670 align:middle line:90%
"Train."

00:19:16.670 --> 00:19:17.170 align:middle line:90%
OK.

00:19:17.170 --> 00:19:18.790 align:middle line:90%
Thank you.

00:19:18.790 --> 00:19:20.342 align:middle line:90%
OK, others?

00:19:20.342 --> 00:19:21.752 align:middle line:90%
AUDIENCE: "Slowly."

00:19:21.752 --> 00:19:23.210 align:middle line:84%
RAMA RAMAKRISHNAN:
"Slowly," right.

00:19:23.210 --> 00:19:25.300 align:middle line:84%
So that also seems to
have some bearing to it.

00:19:25.300 --> 00:19:28.640 align:middle line:84%
What about words that
don't really we don't think

00:19:28.640 --> 00:19:30.936 align:middle line:90%
are going to help at all.

00:19:30.936 --> 00:19:31.840 align:middle line:90%
AUDIENCE: "The."

00:19:31.840 --> 00:19:32.200 align:middle line:90%
RAMA RAMAKRISHNAN: "The."

00:19:32.200 --> 00:19:32.700 align:middle line:90%
Exactly.

00:19:32.700 --> 00:19:34.400 align:middle line:84%
It probably doesn't
do much here.

00:19:34.400 --> 00:19:36.820 align:middle line:84%
Some context it actually
might make a difference,

00:19:36.820 --> 00:19:38.720 align:middle line:90%
but in this sentence maybe not.

00:19:38.720 --> 00:19:40.800 align:middle line:90%
Intuitively.

00:19:40.800 --> 00:19:43.700 align:middle line:84%
So we should probably give
a lot of weight to "train,"

00:19:43.700 --> 00:19:45.320 align:middle line:84%
maybe a little to
"slowly" and "left,"

00:19:45.320 --> 00:19:49.130 align:middle line:90%
and hardly anything to "the."

00:19:49.130 --> 00:19:52.480 align:middle line:84%
And so this intuition
that we have

00:19:52.480 --> 00:19:56.600 align:middle line:84%
can be written
numerically as maybe

00:19:56.600 --> 00:19:59.020 align:middle line:84%
we have a bunch of
weights that add up to 1.

00:19:59.020 --> 00:20:02.377 align:middle line:90%


00:20:02.377 --> 00:20:03.460 align:middle line:90%
Maybe something like this.

00:20:03.460 --> 00:20:05.360 align:middle line:90%
So we are saying, the "train"--

00:20:05.360 --> 00:20:07.080 align:middle line:90%
30% weightage.

00:20:07.080 --> 00:20:10.040 align:middle line:90%
Maybe 8% weightage to "left."

00:20:10.040 --> 00:20:13.600 align:middle line:90%
Maybe 12% weightage to "slowly."

00:20:13.600 --> 00:20:17.720 align:middle line:84%
And then, as you will see here,
"station's" own embedding also

00:20:17.720 --> 00:20:21.120 align:middle line:84%
plays a role, because we want
to take its own standalone

00:20:21.120 --> 00:20:23.078 align:middle line:84%
embedding and just
move it slightly.

00:20:23.078 --> 00:20:23.870 align:middle line:90%
Change it slightly.

00:20:23.870 --> 00:20:26.810 align:middle line:84%
Which means that has to
be the starting point.

00:20:26.810 --> 00:20:28.270 align:middle line:90%
So it will get a lot of weight.

00:20:28.270 --> 00:20:30.810 align:middle line:84%
You can't ignore
itself, in other words.

00:20:30.810 --> 00:20:33.630 align:middle line:90%
So we give it maybe 40% weight.

00:20:33.630 --> 00:20:37.010 align:middle line:84%
By the way, these numbers,
I just made them up.

00:20:37.010 --> 00:20:38.477 align:middle line:90%
Yeah.

00:20:38.477 --> 00:20:39.310 align:middle line:90%
AUDIENCE: I'm sorry.

00:20:39.310 --> 00:20:40.227 align:middle line:90%
Just a quick question.

00:20:40.227 --> 00:20:44.320 align:middle line:90%
So the weights are--

00:20:44.320 --> 00:20:46.970 align:middle line:84%
are they are they standalone
to understand the context

00:20:46.970 --> 00:20:50.268 align:middle line:84%
the entire sentence, or are
they related to "stations"

00:20:50.268 --> 00:20:51.810 align:middle line:84%
that we started off
with, "stations."

00:20:51.810 --> 00:20:54.998 align:middle line:84%
RAMA RAMAKRISHNAN: These six
numbers are only pertinent

00:20:54.998 --> 00:20:55.540 align:middle line:90%
to "station."

00:20:55.540 --> 00:20:55.816 align:middle line:90%
AUDIENCE: Got it.

00:20:55.816 --> 00:20:56.410 align:middle line:90%
Right.

00:20:56.410 --> 00:20:57.910 align:middle line:84%
RAMA RAMAKRISHNAN:
And for each word

00:20:57.910 --> 00:20:59.610 align:middle line:84%
we're going to do
something similar.

00:20:59.610 --> 00:21:00.443 align:middle line:90%
AUDIENCE: Thank you.

00:21:00.443 --> 00:21:01.650 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:21:01.650 --> 00:21:04.700 align:middle line:84%
AUDIENCE: At this point, does
the model understand order?

00:21:04.700 --> 00:21:06.450 align:middle line:84%
Because like, I'm just
thinking of "left,"

00:21:06.450 --> 00:21:09.610 align:middle line:90%
because it gave it a very low--

00:21:09.610 --> 00:21:10.710 align:middle line:90%
a very low weight.

00:21:10.710 --> 00:21:14.270 align:middle line:84%
But let's say "left" comes
"slowly we left station."

00:21:14.270 --> 00:21:16.070 align:middle line:84%
The station only
happened to be higher.

00:21:16.070 --> 00:21:17.445 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, correct.

00:21:17.445 --> 00:21:19.450 align:middle line:84%
So at this point, we not
worrying about order.

00:21:19.450 --> 00:21:22.620 align:middle line:84%
We are only really
worrying about context.

00:21:22.620 --> 00:21:24.592 align:middle line:84%
Later, we will take
order into account.

00:21:24.592 --> 00:21:26.425 align:middle line:84%
AUDIENCE: How did the
model know that "left"

00:21:26.425 --> 00:21:29.380 align:middle line:84%
here is of lesser importance
because it's a verb rather than

00:21:29.380 --> 00:21:31.100 align:middle line:90%
an adjective.

00:21:31.100 --> 00:21:33.840 align:middle line:84%
RAMA RAMAKRISHNAN: It
has to figure it out.

00:21:33.840 --> 00:21:36.540 align:middle line:84%
We are just giving it a
whole bunch of capabilities.

00:21:36.540 --> 00:21:38.748 align:middle line:84%
How it manifests
those capabilities

00:21:38.748 --> 00:21:40.290 align:middle line:84%
is all going to
emerge from training.

00:21:40.290 --> 00:21:42.900 align:middle line:90%


00:21:42.900 --> 00:21:44.592 align:middle line:90%
So, all right.

00:21:44.592 --> 00:21:46.300 align:middle line:84%
So let's say we have
something like this.

00:21:46.300 --> 00:21:48.140 align:middle line:90%
So what we can do--

00:21:48.140 --> 00:21:49.980 align:middle line:84%
and we'll get to the
all-important question

00:21:49.980 --> 00:21:52.272 align:middle line:84%
of where do we get these
numbers from in just a moment.

00:21:52.272 --> 00:21:54.660 align:middle line:90%
But suppose you had the numbers.

00:21:54.660 --> 00:21:58.540 align:middle line:84%
How can we use these
numbers to contextualize w6?

00:21:58.540 --> 00:22:00.420 align:middle line:90%
What can we do?

00:22:00.420 --> 00:22:02.160 align:middle line:84%
What is the simplest
thing you can do?

00:22:02.160 --> 00:22:05.380 align:middle line:90%


00:22:05.380 --> 00:22:06.480 align:middle line:90%
You have w6.

00:22:06.480 --> 00:22:09.680 align:middle line:84%
You want to make it a new
w6, which is now contextual.

00:22:09.680 --> 00:22:11.360 align:middle line:84%
Is aware of what
else is going on.

00:22:11.360 --> 00:22:15.216 align:middle line:90%


00:22:15.216 --> 00:22:17.037 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:22:17.037 --> 00:22:18.912 align:middle line:84%
RAMA RAMAKRISHNAN: It's
working now, I think.

00:22:18.912 --> 00:22:19.324 align:middle line:84%
AUDIENCE: Take a
weighted average.

00:22:19.324 --> 00:22:21.530 align:middle line:84%
RAMA RAMAKRISHNAN: We can
take a weighted average.

00:22:21.530 --> 00:22:22.493 align:middle line:90%
Exactly, exactly.

00:22:22.493 --> 00:22:23.910 align:middle line:84%
So when you have
a bunch of things

00:22:23.910 --> 00:22:27.470 align:middle line:84%
and you have a bunch of weights,
and we have to somehow modify

00:22:27.470 --> 00:22:29.130 align:middle line:84%
one of those things
with those weights,

00:22:29.130 --> 00:22:32.110 align:middle line:84%
the simplest thing you can do
is to take a weighted average.

00:22:32.110 --> 00:22:34.310 align:middle line:84%
So that's exactly what
we're going to do.

00:22:34.310 --> 00:22:37.470 align:middle line:84%
So we're going to take all
these weights and just,

00:22:37.470 --> 00:22:40.670 align:middle line:90%
like, move them up.

00:22:40.670 --> 00:22:42.443 align:middle line:90%
Move them up.

00:22:42.443 --> 00:22:44.110 align:middle line:84%
Don't even get me
started on how long it

00:22:44.110 --> 00:22:46.182 align:middle line:90%
took me to get this arrow run.

00:22:46.182 --> 00:22:47.390 align:middle line:90%
I don't know about you folks.

00:22:47.390 --> 00:22:49.470 align:middle line:84%
It is extremely painful
to get the U-turn

00:22:49.470 --> 00:22:51.230 align:middle line:90%
arrows to work in PowerPoint.

00:22:51.230 --> 00:22:53.850 align:middle line:90%
Anyway, back to work.

00:22:53.850 --> 00:22:57.030 align:middle line:90%
So we just move these up here.

00:22:57.030 --> 00:23:00.430 align:middle line:84%
So now we can do 0.05
times this vector

00:23:00.430 --> 00:23:03.710 align:middle line:84%
plus 0.3 times that vector,
and so on and so forth.

00:23:03.710 --> 00:23:08.710 align:middle line:84%
And the result is just
another vector, right?

00:23:08.710 --> 00:23:13.350 align:middle line:84%
And that vector, folks, is the
contextual embedding vector

00:23:13.350 --> 00:23:15.950 align:middle line:90%
of "station."

00:23:15.950 --> 00:23:17.840 align:middle line:90%
That was a standalone embedding.

00:23:17.840 --> 00:23:20.920 align:middle line:84%
And now we did the-- we
multiplied this by that,

00:23:20.920 --> 00:23:21.780 align:middle line:90%
added them all up.

00:23:21.780 --> 00:23:24.840 align:middle line:90%
And then you get a new vector.

00:23:24.840 --> 00:23:29.600 align:middle line:84%
And contextual embeddings have
this bluish kind of color.

00:23:29.600 --> 00:23:33.600 align:middle line:84%
And I'll maintain that
color scheme as we go along.

00:23:33.600 --> 00:23:36.360 align:middle line:90%
So that's it.

00:23:36.360 --> 00:23:37.060 align:middle line:90%
That's it.

00:23:37.060 --> 00:23:38.480 align:middle line:90%
That's the idea.

00:23:38.480 --> 00:23:41.760 align:middle line:90%
Any questions?

00:23:41.760 --> 00:23:43.157 align:middle line:90%
Yeah.

00:23:43.157 --> 00:23:45.740 align:middle line:84%
AUDIENCE: How did you come up
with the original weights again?

00:23:45.740 --> 00:23:46.880 align:middle line:90%
You just kind of guess?

00:23:46.880 --> 00:23:48.422 align:middle line:84%
RAMA RAMAKRISHNAN:
No, these weights,

00:23:48.422 --> 00:23:51.280 align:middle line:84%
I just hand type them
in manually, just

00:23:51.280 --> 00:23:52.075 align:middle line:90%
to make the point.

00:23:52.075 --> 00:23:54.200 align:middle line:84%
And now I'm going to talk
about how we are actually

00:23:54.200 --> 00:23:57.640 align:middle line:90%
going to calculate them.

00:23:57.640 --> 00:23:59.000 align:middle line:90%
OK.

00:23:59.000 --> 00:23:59.500 align:middle line:90%
All right.

00:23:59.500 --> 00:24:00.000 align:middle line:90%
Cool.

00:24:00.000 --> 00:24:01.520 align:middle line:90%
So now I'm going to--

00:24:01.520 --> 00:24:02.780 align:middle line:90%
OK, enough pictures.

00:24:02.780 --> 00:24:04.200 align:middle line:90%
Let's switch to some math.

00:24:04.200 --> 00:24:08.760 align:middle line:84%
So basically what I'm-- so let's
write it a bit more formally.

00:24:08.760 --> 00:24:11.280 align:middle line:84%
So we have these
w1 through w6 which

00:24:11.280 --> 00:24:12.920 align:middle line:90%
are the standalone embeddings.

00:24:12.920 --> 00:24:16.330 align:middle line:84%
And then, for "station," we want
to calculate w6 with a little

00:24:16.330 --> 00:24:18.370 align:middle line:84%
hat on it, which is the
contextual embedding.

00:24:18.370 --> 00:24:22.010 align:middle line:84%
And the way we do it is to
say we calculate some weights

00:24:22.010 --> 00:24:23.390 align:middle line:90%
for each of these words.

00:24:23.390 --> 00:24:28.730 align:middle line:84%
So this weight, s1,6, means that
the weight of the first word

00:24:28.730 --> 00:24:32.130 align:middle line:84%
on the sixth word, which
happens to be station.

00:24:32.130 --> 00:24:34.590 align:middle line:84%
The weight of the second
word on the sixth word,

00:24:34.590 --> 00:24:35.790 align:middle line:90%
and so on and so forth.

00:24:35.790 --> 00:24:38.970 align:middle line:84%
And so what we are
saying is that w6 is just

00:24:38.970 --> 00:24:41.930 align:middle line:84%
this weight times
w1, this time w--

00:24:41.930 --> 00:24:42.550 align:middle line:90%
that's it.

00:24:42.550 --> 00:24:45.813 align:middle line:90%


00:24:45.813 --> 00:24:47.730 align:middle line:84%
I have to inflict all
these subscripts and all

00:24:47.730 --> 00:24:51.250 align:middle line:90%
that because we need it.

00:24:51.250 --> 00:24:51.750 align:middle line:90%
All right.

00:24:51.750 --> 00:24:53.930 align:middle line:90%
So that's it.

00:24:53.930 --> 00:24:56.610 align:middle line:90%
That's what we have.

00:24:56.610 --> 00:25:00.250 align:middle line:84%
Now let's talk about-- any
questions on the mechanics of it

00:25:00.250 --> 00:25:01.230 align:middle line:90%
before I get to--

00:25:01.230 --> 00:25:02.930 align:middle line:84%
OK, where do these
weights come from?

00:25:02.930 --> 00:25:04.260 align:middle line:90%
Yeah.

00:25:04.260 --> 00:25:08.910 align:middle line:84%
AUDIENCE: [INAUDIBLE] utilizing
something like Google,

00:25:08.910 --> 00:25:09.410 align:middle line:90%
for example.

00:25:09.410 --> 00:25:14.140 align:middle line:84%
How does it understand
the context of new words

00:25:14.140 --> 00:25:15.060 align:middle line:90%
that comes to play?

00:25:15.060 --> 00:25:18.780 align:middle line:84%
Is the process immediately
through the training data

00:25:18.780 --> 00:25:20.630 align:middle line:84%
the users put in,
or what makes it--

00:25:20.630 --> 00:25:22.380 align:middle line:84%
RAMA RAMAKRISHNAN:
Like a totally new word

00:25:22.380 --> 00:25:23.638 align:middle line:90%
that didn't exist before?

00:25:23.638 --> 00:25:25.180 align:middle line:84%
AUDIENCE: A new word
or a new context

00:25:25.180 --> 00:25:26.962 align:middle line:90%
to a word that already exists?

00:25:26.962 --> 00:25:29.420 align:middle line:84%
RAMA RAMAKRISHNAN: No, I think
that the context is supplied

00:25:29.420 --> 00:25:31.820 align:middle line:84%
because the query coming
into something like Google

00:25:31.820 --> 00:25:34.660 align:middle line:84%
is a full sentence, and
we only take that sentence

00:25:34.660 --> 00:25:37.980 align:middle line:84%
and take only the sentence into
account as the context for us.

00:25:37.980 --> 00:25:41.620 align:middle line:84%
So the context is always present
to us when we get the input.

00:25:41.620 --> 00:25:43.922 align:middle line:84%
But the other question
you had of, OK,

00:25:43.922 --> 00:25:46.380 align:middle line:84%
what if there is a brand-new
word you've never seen before,

00:25:46.380 --> 00:25:48.505 align:middle line:84%
for which there is not even
a standalone embedding?

00:25:48.505 --> 00:25:49.780 align:middle line:90%
What do you do then?

00:25:49.780 --> 00:25:51.940 align:middle line:84%
So let's punt on
that till Wednesday,

00:25:51.940 --> 00:25:53.660 align:middle line:84%
because I have to
talk about something

00:25:53.660 --> 00:25:56.020 align:middle line:84%
called byte-pair encoding
and stuff like that

00:25:56.020 --> 00:25:57.780 align:middle line:90%
before I can answer that.

00:25:57.780 --> 00:25:59.780 align:middle line:84%
AUDIENCE: Really quickly--
does that immediately

00:25:59.780 --> 00:26:03.620 align:middle line:84%
translate to their
predictive search queries?

00:26:03.620 --> 00:26:08.660 align:middle line:84%
Utilizing, like, for-- we
have a new word, for example.

00:26:08.660 --> 00:26:11.480 align:middle line:84%
Does that automatically get
applied to the predictive search

00:26:11.480 --> 00:26:13.230 align:middle line:84%
queries, like what we
were saying, how to,

00:26:13.230 --> 00:26:14.070 align:middle line:90%
and then just automatic.

00:26:14.070 --> 00:26:16.320 align:middle line:84%
RAMA RAMAKRISHNAN: Oh, you
mean like the autocomplete?

00:26:16.320 --> 00:26:19.630 align:middle line:84%
Autocomplete uses a slightly
different mechanism.

00:26:19.630 --> 00:26:22.470 align:middle line:84%
They had a very complicated
non-transformer thing

00:26:22.470 --> 00:26:23.235 align:middle line:90%
for a long time.

00:26:23.235 --> 00:26:25.110 align:middle line:84%
I'm sure they have a
transformer version now.

00:26:25.110 --> 00:26:27.670 align:middle line:84%
But I'm not privy to how
exactly they've done it.

00:26:27.670 --> 00:26:29.275 align:middle line:84%
So I don't quite
know how they do it.

00:26:29.275 --> 00:26:31.150 align:middle line:84%
But what you're proposing
is a reasonable way

00:26:31.150 --> 00:26:33.350 align:middle line:90%
to think about it.

00:26:33.350 --> 00:26:34.333 align:middle line:90%
Yeah.

00:26:34.333 --> 00:26:35.750 align:middle line:84%
AUDIENCE: My
question is, you take

00:26:35.750 --> 00:26:40.710 align:middle line:84%
the path, a particular number
of parameters, as in, let's

00:26:40.710 --> 00:26:41.910 align:middle line:90%
say, 10 of them.

00:26:41.910 --> 00:26:45.430 align:middle line:84%
And then we have calculated
the contextual version of w6.

00:26:45.430 --> 00:26:48.123 align:middle line:84%
So this gets added as
H2 or it is [INAUDIBLE].

00:26:48.123 --> 00:26:49.290 align:middle line:90%
RAMA RAMAKRISHNAN: Replaces.

00:26:49.290 --> 00:26:50.630 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:26:50.630 --> 00:26:55.350 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah w6 becomes w6 hat.

00:26:55.350 --> 00:26:57.790 align:middle line:84%
AUDIENCE: And now we
are extending that.

00:26:57.790 --> 00:26:59.030 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:26:59.030 --> 00:27:01.590 align:middle line:90%
AUDIENCE: [INAUDIBLE]

00:27:01.590 --> 00:27:09.643 align:middle line:90%


00:27:09.643 --> 00:27:11.060 align:middle line:84%
RAMA RAMAKRISHNAN:
No, we lose it.

00:27:11.060 --> 00:27:13.660 align:middle line:84%
And as you will see here, as it
flows through the transformer,

00:27:13.660 --> 00:27:16.760 align:middle line:84%
it's getting more and more
and more contextualized.

00:27:16.760 --> 00:27:20.040 align:middle line:90%
So it's a left to right flow.

00:27:20.040 --> 00:27:20.892 align:middle line:90%
All right.

00:27:20.892 --> 00:27:21.600 align:middle line:90%
All right, great.

00:27:21.600 --> 00:27:24.180 align:middle line:84%
So by the way, this thing
that we did for "station,"

00:27:24.180 --> 00:27:28.040 align:middle line:84%
we will do it for each
word in the sentence--

00:27:28.040 --> 00:27:29.300 align:middle line:90%
the same exact logic.

00:27:29.300 --> 00:27:32.480 align:middle line:84%
Obviously, the weights
are going to change.

00:27:32.480 --> 00:27:34.840 align:middle line:84%
But what will happen
is that w1 through W6

00:27:34.840 --> 00:27:39.480 align:middle line:84%
will become w1 hat
through w6 hat.

00:27:39.480 --> 00:27:41.880 align:middle line:84%
The same exact logic
is going to hold.

00:27:41.880 --> 00:27:42.800 align:middle line:90%
OK.

00:27:42.800 --> 00:27:45.677 align:middle line:84%
I just don't have the slides for
it because it's a waste of time.

00:27:45.677 --> 00:27:47.260 align:middle line:84%
The same exact logic
is going to hold.

00:27:47.260 --> 00:27:47.760 align:middle line:90%
All right.

00:27:47.760 --> 00:27:50.927 align:middle line:84%
Now switch gears and answer
the all-important question

00:27:50.927 --> 00:27:52.760 align:middle line:84%
of where are the weights
going to come from.

00:27:52.760 --> 00:27:56.920 align:middle line:84%
So the intuition here is really,
really interesting and elegant.

00:27:56.920 --> 00:28:01.040 align:middle line:84%
So, clearly, the weight of a
word should be proportional

00:28:01.040 --> 00:28:03.620 align:middle line:84%
to how related it is
to the word "station."

00:28:03.620 --> 00:28:06.150 align:middle line:90%


00:28:06.150 --> 00:28:09.928 align:middle line:84%
The word "train" clearly is very
related to the word "station."

00:28:09.928 --> 00:28:11.970 align:middle line:84%
The word "the," it's not
clear how related it is.

00:28:11.970 --> 00:28:13.270 align:middle line:90%
Probably not all that related.

00:28:13.270 --> 00:28:16.190 align:middle line:84%
So the relatedness
matters to the weight.

00:28:16.190 --> 00:28:17.750 align:middle line:90%
More related, higher the weight.

00:28:17.750 --> 00:28:19.370 align:middle line:90%
Just intuitively.

00:28:19.370 --> 00:28:22.530 align:middle line:84%
So one way to quantify
how related two words are

00:28:22.530 --> 00:28:24.610 align:middle line:84%
is to take their
standalone embeddings

00:28:24.610 --> 00:28:25.960 align:middle line:90%
and calculate the dot product.

00:28:25.960 --> 00:28:28.836 align:middle line:90%


00:28:28.836 --> 00:28:34.680 align:middle line:84%
So in case folks have forgotten
about the dot product--

00:28:34.680 --> 00:28:39.690 align:middle line:90%


00:28:39.690 --> 00:28:41.370 align:middle line:90%
oops, that's not what I want.

00:28:41.370 --> 00:28:42.570 align:middle line:90%
Here we go.

00:28:42.570 --> 00:28:45.180 align:middle line:90%
So let's say you have a vector.

00:28:45.180 --> 00:28:51.050 align:middle line:90%


00:28:51.050 --> 00:28:52.960 align:middle line:84%
Let's say this is the
vector for "train."

00:28:52.960 --> 00:28:55.770 align:middle line:90%


00:28:55.770 --> 00:28:59.410 align:middle line:84%
This is the vector
for "station."

00:28:59.410 --> 00:29:02.420 align:middle line:84%
So the dot product of
these two vectors--

00:29:02.420 --> 00:29:05.610 align:middle line:90%


00:29:05.610 --> 00:29:15.340 align:middle line:84%
and I'll write it as train,
station equals, basically,

00:29:15.340 --> 00:29:21.540 align:middle line:84%
the length of the
vector for "train"

00:29:21.540 --> 00:29:28.580 align:middle line:84%
times the length of the
vector for "station"

00:29:28.580 --> 00:29:35.440 align:middle line:84%
times the cosine of
the angle between them.

00:29:35.440 --> 00:29:42.460 align:middle line:90%


00:29:42.460 --> 00:29:45.467 align:middle line:84%
So how long is each
vector, the product

00:29:45.467 --> 00:29:47.300 align:middle line:84%
of the two, and then
the angle between them.

00:29:47.300 --> 00:29:50.380 align:middle line:84%
Now, let's assume for
simplicity that these links

00:29:50.380 --> 00:29:52.620 align:middle line:90%
are roughly the same.

00:29:52.620 --> 00:29:54.340 align:middle line:90%
They are just one unit length.

00:29:54.340 --> 00:29:55.660 align:middle line:90%
Just roughly.

00:29:55.660 --> 00:30:02.680 align:middle line:84%
So if you assume that, this
thing, let's say, becomes 1,

00:30:02.680 --> 00:30:05.390 align:middle line:90%
let's say.

00:30:05.390 --> 00:30:07.230 align:middle line:90%
This thing becomes one.

00:30:07.230 --> 00:30:09.900 align:middle line:90%
So all the action is here.

00:30:09.900 --> 00:30:12.482 align:middle line:90%


00:30:12.482 --> 00:30:14.307 align:middle line:90%
So all the action is here.

00:30:14.307 --> 00:30:16.390 align:middle line:84%
So basically, the dot
product of these two vectors

00:30:16.390 --> 00:30:20.150 align:middle line:84%
is really the cosine of
the angle between them.

00:30:20.150 --> 00:30:22.670 align:middle line:84%
So now the question is,
if you have something

00:30:22.670 --> 00:30:29.790 align:middle line:84%
like this, which are very
close to each other, the cosine

00:30:29.790 --> 00:30:31.510 align:middle line:90%
of a very small angle--

00:30:31.510 --> 00:30:34.550 align:middle line:84%
actually, the
cosine of 0 is what?

00:30:34.550 --> 00:30:35.590 align:middle line:90%
1.

00:30:35.590 --> 00:30:38.210 align:middle line:84%
So if the angle is
really, really small,

00:30:38.210 --> 00:30:41.190 align:middle line:84%
the cosine is going to be very
close to 1, because 0 is 1.

00:30:41.190 --> 00:30:42.830 align:middle line:90%
The cosine of 0 is 1.

00:30:42.830 --> 00:30:46.710 align:middle line:84%
So this thing is going
to be pretty close to 1.

00:30:46.710 --> 00:30:49.070 align:middle line:84%
If you have a cosine
of two vectors that

00:30:49.070 --> 00:30:52.870 align:middle line:84%
are like this, 90 degrees
apart, what is the cosine?

00:30:52.870 --> 00:30:53.590 align:middle line:90%
0.

00:30:53.590 --> 00:30:56.350 align:middle line:84%
They are orthogonal, right,
which maps to the English,

00:30:56.350 --> 00:30:58.190 align:middle line:90%
orthogonal.

00:30:58.190 --> 00:31:00.790 align:middle line:90%
So the cosine of that is 0.

00:31:00.790 --> 00:31:04.000 align:middle line:84%
And then, if you have something
like this where they're

00:31:04.000 --> 00:31:07.680 align:middle line:84%
literally pointing in
opposite direction,

00:31:07.680 --> 00:31:10.000 align:middle line:90%
what is the cosine of that 180?

00:31:10.000 --> 00:31:11.320 align:middle line:90%
Minus 1.

00:31:11.320 --> 00:31:12.100 align:middle line:90%
So that's it.

00:31:12.100 --> 00:31:16.160 align:middle line:84%
So if these two vectors are
very close to each other,

00:31:16.160 --> 00:31:18.000 align:middle line:84%
the cosine of the
angle between them

00:31:18.000 --> 00:31:19.640 align:middle line:90%
is going to be very close to 1.

00:31:19.640 --> 00:31:22.180 align:middle line:84%
If they are really kind of
unrelated, it's going to be 0.

00:31:22.180 --> 00:31:24.320 align:middle line:84%
If they're anti-related
it's going to be minus 1.

00:31:24.320 --> 00:31:28.880 align:middle line:84%
So that's how dot products
capture this notion of closeness

00:31:28.880 --> 00:31:29.900 align:middle line:90%
or relatedness.

00:31:29.900 --> 00:31:32.640 align:middle line:90%


00:31:32.640 --> 00:31:34.400 align:middle line:90%
All right.

00:31:34.400 --> 00:31:36.400 align:middle line:90%
Input.

00:31:36.400 --> 00:31:38.480 align:middle line:84%
So we can use the dot
product of these embeddings

00:31:38.480 --> 00:31:40.560 align:middle line:90%
to capture relatedness.

00:31:40.560 --> 00:31:43.600 align:middle line:90%
And so-- OK, iPad done.

00:31:43.600 --> 00:31:47.040 align:middle line:84%
So now what we do is, now that
we know that dot products can

00:31:47.040 --> 00:31:49.247 align:middle line:84%
be used, we can't
use them as is,

00:31:49.247 --> 00:31:51.080 align:middle line:84%
because we need to do
one more thing to make

00:31:51.080 --> 00:31:52.720 align:middle line:90%
them proper weights.

00:31:52.720 --> 00:31:54.240 align:middle line:84%
And what I mean
by proper weights

00:31:54.240 --> 00:31:57.900 align:middle line:84%
is that we want the weights to
be, first of all, non-negative.

00:31:57.900 --> 00:31:59.660 align:middle line:90%
And we want them to add up to 1.

00:31:59.660 --> 00:32:02.370 align:middle line:84%
That's what a weighted average
actually is going to mean.

00:32:02.370 --> 00:32:06.110 align:middle line:84%
But these cosines could
be negative, right?

00:32:06.110 --> 00:32:08.497 align:middle line:84%
And so we need to now adjust
them to make them proper so

00:32:08.497 --> 00:32:10.830 align:middle line:84%
that every one of them is
guaranteed to be non-negative,

00:32:10.830 --> 00:32:12.330 align:middle line:90%
and they will add up to 1.

00:32:12.330 --> 00:32:15.570 align:middle line:84%
When was the last time you had
to take a bunch of numbers which

00:32:15.570 --> 00:32:18.330 align:middle line:84%
could be anything, and
then somehow make sure

00:32:18.330 --> 00:32:21.210 align:middle line:84%
that they are going to be
positive, non-negative,

00:32:21.210 --> 00:32:22.150 align:middle line:90%
and they add up to 1?

00:32:22.150 --> 00:32:23.110 align:middle line:90%
When was the last time?

00:32:23.110 --> 00:32:23.970 align:middle line:90%
AUDIENCE: Softmax.

00:32:23.970 --> 00:32:24.990 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, softmax.

00:32:24.990 --> 00:32:25.490 align:middle line:90%
Exactly.

00:32:25.490 --> 00:32:27.650 align:middle line:90%
So we'll do the same trick.

00:32:27.650 --> 00:32:32.290 align:middle line:84%
So what we'll simply do is
we'll just exponentiate them.

00:32:32.290 --> 00:32:35.558 align:middle line:84%
So this w1, w6, this
angle bracket thing

00:32:35.558 --> 00:32:36.350 align:middle line:90%
is the dot product.

00:32:36.350 --> 00:32:37.970 align:middle line:90%
That's the notation I'm using.

00:32:37.970 --> 00:32:39.970 align:middle line:84%
exp of that is just
you exponentiate

00:32:39.970 --> 00:32:41.330 align:middle line:90%
them, e raised to that.

00:32:41.330 --> 00:32:44.020 align:middle line:84%
And once you exponentiate them,
they all become non-negative.

00:32:44.020 --> 00:32:46.270 align:middle line:84%
And then we just divide each
by the sum of everything,

00:32:46.270 --> 00:32:48.562 align:middle line:84%
so that the whole thing will
become like a probability.

00:32:48.562 --> 00:32:50.210 align:middle line:90%
It'll just add up to 1.

00:32:50.210 --> 00:32:51.265 align:middle line:90%
Makes sense?

00:32:51.265 --> 00:32:52.890 align:middle line:84%
So that's how we take
arbitrary numbers

00:32:52.890 --> 00:32:54.098 align:middle line:90%
and make them proper weights.

00:32:54.098 --> 00:32:56.770 align:middle line:90%


00:32:56.770 --> 00:32:57.390 align:middle line:90%
All right.

00:32:57.390 --> 00:32:59.980 align:middle line:90%


00:32:59.980 --> 00:33:04.860 align:middle line:84%
So to summarize, from embeddings
to contextual embeddings, that's

00:33:04.860 --> 00:33:05.820 align:middle line:90%
what we do.

00:33:05.820 --> 00:33:08.200 align:middle line:84%
We take all the
standalone embeddings.

00:33:08.200 --> 00:33:10.640 align:middle line:84%
We calculate these weights
using this formula.

00:33:10.640 --> 00:33:12.400 align:middle line:84%
And then we just do
the weighted average,

00:33:12.400 --> 00:33:14.817 align:middle line:84%
and we arrive at the contextual
embedding, and boom, done.

00:33:14.817 --> 00:33:17.810 align:middle line:90%


00:33:17.810 --> 00:33:20.580 align:middle line:84%
And so by choosing
weights in this manner,

00:33:20.580 --> 00:33:22.940 align:middle line:84%
the embedding of a
word gets dragged

00:33:22.940 --> 00:33:26.220 align:middle line:84%
closer to the embeddings of
the other words in proportion

00:33:26.220 --> 00:33:27.400 align:middle line:90%
to how related they are.

00:33:27.400 --> 00:33:30.140 align:middle line:84%
So just imagine for a second,
in this case, "station"

00:33:30.140 --> 00:33:31.480 align:middle line:90%
obviously has many contexts.

00:33:31.480 --> 00:33:33.355 align:middle line:84%
But let's assume for a
second that it'll only

00:33:33.355 --> 00:33:35.940 align:middle line:84%
have the train context and
the radio-station context.

00:33:35.940 --> 00:33:40.100 align:middle line:84%
In the current context, "train"
is closely tied to "station,"

00:33:40.100 --> 00:33:43.940 align:middle line:84%
and therefore exerts
a strong pull on it.

00:33:43.940 --> 00:33:45.800 align:middle line:84%
Now "radio" is also
related to "station,"

00:33:45.800 --> 00:33:48.540 align:middle line:84%
but it doesn't appear in
the word, in the sentence.

00:33:48.540 --> 00:33:50.210 align:middle line:84%
So effectively it
has a weight of 0.

00:33:50.210 --> 00:33:52.940 align:middle line:90%


00:33:52.940 --> 00:33:54.878 align:middle line:90%
And that's the beauty of it.

00:33:54.878 --> 00:33:56.420 align:middle line:84%
And please do not
ask me things like,

00:33:56.420 --> 00:33:59.270 align:middle line:84%
I was listening to a great
song on the radio station,

00:33:59.270 --> 00:34:02.190 align:middle line:84%
and the train pulled
out of the station.

00:34:02.190 --> 00:34:04.330 align:middle line:84%
Transformers can deal
with stuff like that.

00:34:04.330 --> 00:34:07.630 align:middle line:84%
But yeah, but you get
the idea-- the main idea.

00:34:07.630 --> 00:34:11.550 align:middle line:84%
So by moving "station"
closer to "train"

00:34:11.550 --> 00:34:13.409 align:middle line:84%
by paying more
attention to "train,"

00:34:13.409 --> 00:34:15.730 align:middle line:84%
we are contextualizing
the "station," the word,

00:34:15.730 --> 00:34:19.810 align:middle line:84%
the embedding, to the context of
trains, platforms, departures,

00:34:19.810 --> 00:34:20.969 align:middle line:90%
tickets, and so on.

00:34:20.969 --> 00:34:25.590 align:middle line:84%
It's like this portal into
the whole train world.

00:34:25.590 --> 00:34:26.610 align:middle line:90%
It's beautiful.

00:34:26.610 --> 00:34:28.210 align:middle line:84%
The simple idea
will get you there.

00:34:28.210 --> 00:34:31.416 align:middle line:90%


00:34:31.416 --> 00:34:36.750 align:middle line:84%
So this, folks, is
called self-attention.

00:34:36.750 --> 00:34:39.710 align:middle line:84%
What we just described
is called self-attention.

00:34:39.710 --> 00:34:43.656 align:middle line:84%
And it's the key building
block of transformers.

00:34:43.656 --> 00:34:47.170 align:middle line:84%
And so to summarize,
standalone embeddings come in,

00:34:47.170 --> 00:34:50.830 align:middle line:90%
contextual embeddings go out.

00:34:50.830 --> 00:34:52.670 align:middle line:90%
Any questions?

00:34:52.670 --> 00:34:54.608 align:middle line:90%
Yeah.

00:34:54.608 --> 00:34:56.400 align:middle line:84%
AUDIENCE: I'm still
struggling a little bit

00:34:56.400 --> 00:34:58.740 align:middle line:90%
with the intuition of the word.

00:34:58.740 --> 00:35:00.840 align:middle line:84%
It's the weight
of the word itself

00:35:00.840 --> 00:35:02.580 align:middle line:90%
in its own contextual embedding.

00:35:02.580 --> 00:35:05.260 align:middle line:84%
So the weight of "station"
and the "station" embedding,

00:35:05.260 --> 00:35:06.940 align:middle line:90%
how should I think about that.

00:35:06.940 --> 00:35:07.320 align:middle line:90%
RAMA RAMAKRISHNAN: Think of it--

00:35:07.320 --> 00:35:08.040 align:middle line:84%
AUDIENCE: Seem
intuitively like it

00:35:08.040 --> 00:35:10.420 align:middle line:84%
would be high for all
contextual embeddings,

00:35:10.420 --> 00:35:12.160 align:middle line:84%
but I assume that's
not the case.

00:35:12.160 --> 00:35:12.900 align:middle line:84%
RAMA RAMAKRISHNAN:
It'll be high.

00:35:12.900 --> 00:35:14.600 align:middle line:84%
It'll be typically
be a high number,

00:35:14.600 --> 00:35:16.657 align:middle line:84%
because the cosine of
the vector to itself

00:35:16.657 --> 00:35:18.740 align:middle line:84%
is going to be very-- the
cosine is going to be 1.

00:35:18.740 --> 00:35:20.400 align:middle line:90%
So it's going to be pretty high.

00:35:20.400 --> 00:35:23.260 align:middle line:84%
But there's no guarantee it's
going to be the highest, right?

00:35:23.260 --> 00:35:26.540 align:middle line:84%
Because they are not actually--
the length doesn't have to be 1.

00:35:26.540 --> 00:35:28.880 align:middle line:84%
They could be-- we try
to keep them smallish,

00:35:28.880 --> 00:35:30.585 align:middle line:90%
but they don't have to be.

00:35:30.585 --> 00:35:31.960 align:middle line:84%
So the way I would
think of it is

00:35:31.960 --> 00:35:34.840 align:middle line:84%
imagine that you take an average
of everything else first,

00:35:34.840 --> 00:35:38.123 align:middle line:84%
and then you average it
with the old embedding.

00:35:38.123 --> 00:35:40.040 align:middle line:84%
Effectively, it's the
same as just calculating

00:35:40.040 --> 00:35:42.165 align:middle line:84%
the different weights and
averaging the whole thing

00:35:42.165 --> 00:35:42.760 align:middle line:90%
together.

00:35:42.760 --> 00:35:43.400 align:middle line:90%
Sure.

00:35:43.400 --> 00:35:44.304 align:middle line:90%
Yeah.

00:35:44.304 --> 00:35:46.520 align:middle line:84%
AUDIENCE: A little
confused [INAUDIBLE]

00:35:46.520 --> 00:35:49.860 align:middle line:84%
you need the same number
of inputs and outputs.

00:35:49.860 --> 00:35:51.520 align:middle line:84%
But is this the reason
why, because you

00:35:51.520 --> 00:35:56.730 align:middle line:84%
need the contextual embedding,
even if it's like a other word,

00:35:56.730 --> 00:35:58.925 align:middle line:84%
and it's related
definitely [INAUDIBLE].

00:35:58.925 --> 00:36:00.050 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:36:00.050 --> 00:36:00.590 align:middle line:90%
Correct.

00:36:00.590 --> 00:36:01.090 align:middle line:90%
Exactly.

00:36:01.090 --> 00:36:01.750 align:middle line:90%
Exactly.

00:36:01.750 --> 00:36:04.850 align:middle line:84%
And the other thing to
remember is that by getting--

00:36:04.850 --> 00:36:09.050 align:middle line:84%
by keeping the inputs the size
of the input cardinality intact

00:36:09.050 --> 00:36:11.123 align:middle line:84%
as you move through
the transformer stack,

00:36:11.123 --> 00:36:12.790 align:middle line:84%
when you finally come
out the other end,

00:36:12.790 --> 00:36:14.890 align:middle line:90%
there is no loss of information.

00:36:14.890 --> 00:36:16.997 align:middle line:84%
And at the very end, you
can choose to aggregate,

00:36:16.997 --> 00:36:18.830 align:middle line:84%
simplify, summarize,
and so on and so forth.

00:36:18.830 --> 00:36:20.913 align:middle line:84%
It preserves your optionality
as long as possible.

00:36:20.913 --> 00:36:23.560 align:middle line:90%


00:36:23.560 --> 00:36:32.483 align:middle line:84%
AUDIENCE: Do you know how long
[INAUDIBLE] going to take?

00:36:32.483 --> 00:36:35.150 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, so what
we do is the sentence comes in.

00:36:35.150 --> 00:36:36.890 align:middle line:84%
There is a whole notion of
something called a context

00:36:36.890 --> 00:36:37.550 align:middle line:90%
window--

00:36:37.550 --> 00:36:40.430 align:middle line:84%
or what is the maximum length
of the sentence we'll handle.

00:36:40.430 --> 00:36:42.330 align:middle line:84%
And that's a
parameter you can set.

00:36:42.330 --> 00:36:46.690 align:middle line:84%
And we'll come to that when
you actually look at the Colab.

00:36:46.690 --> 00:36:48.250 align:middle line:84%
Was there a question
in the middle?

00:36:48.250 --> 00:36:48.750 align:middle line:90%
No.

00:36:48.750 --> 00:36:49.650 align:middle line:90%
OK.

00:36:49.650 --> 00:36:50.150 align:middle line:90%
All right.

00:36:50.150 --> 00:36:51.350 align:middle line:90%
So that is self-attention.

00:36:51.350 --> 00:36:54.180 align:middle line:90%


00:36:54.180 --> 00:36:58.100 align:middle line:84%
And now, because
that felt too easy,

00:36:58.100 --> 00:37:02.140 align:middle line:84%
we're going to do a little tweak
called multi-head attention.

00:37:02.140 --> 00:37:06.060 align:middle line:84%
So this is the
self-attention we just saw.

00:37:06.060 --> 00:37:09.060 align:middle line:84%
What we can do is we can
be like, you know what?

00:37:09.060 --> 00:37:10.640 align:middle line:84%
Why can't we have
more than this?

00:37:10.640 --> 00:37:12.420 align:middle line:84%
Why can't we have more
than one of these?

00:37:12.420 --> 00:37:15.140 align:middle line:84%
So this is called an attention
head-- self-attention head.

00:37:15.140 --> 00:37:19.060 align:middle line:84%
We'll have multiple
self-attention heads.

00:37:19.060 --> 00:37:21.740 align:middle line:84%
And I'll come back to the
top thing in a second.

00:37:21.740 --> 00:37:23.460 align:middle line:84%
But the question is,
why should we have

00:37:23.460 --> 00:37:25.442 align:middle line:90%
multiple self-attention heads?

00:37:25.442 --> 00:37:26.900 align:middle line:84%
Because a particular
attention head

00:37:26.900 --> 00:37:28.317 align:middle line:84%
is going to pick
up some patterns.

00:37:28.317 --> 00:37:31.140 align:middle line:84%
The reason is
because it will help

00:37:31.140 --> 00:37:33.540 align:middle line:84%
us attend to the multiple
patterns that may be

00:37:33.540 --> 00:37:35.740 align:middle line:90%
present in a single sentence.

00:37:35.740 --> 00:37:37.277 align:middle line:84%
So far, when I've
been explaining,

00:37:37.277 --> 00:37:39.860 align:middle line:84%
I've basically been looking at
what the meaning of these words

00:37:39.860 --> 00:37:41.940 align:middle line:84%
are, just the meaning
of these words.

00:37:41.940 --> 00:37:45.320 align:middle line:84%
But in any complicated sentence,
you have to worry about grammar.

00:37:45.320 --> 00:37:46.760 align:middle line:90%
You have to worry about tense.

00:37:46.760 --> 00:37:48.420 align:middle line:90%
You have to worry about tone.

00:37:48.420 --> 00:37:51.788 align:middle line:84%
You have to worry about
facts versus opinions.

00:37:51.788 --> 00:37:53.830 align:middle line:84%
There could be any number
of complicated patterns

00:37:53.830 --> 00:37:55.590 align:middle line:84%
that are sitting in
a simple sentence.

00:37:55.590 --> 00:37:59.250 align:middle line:84%
Which means, well, there is just
not one way to pay attention.

00:37:59.250 --> 00:38:02.270 align:middle line:84%
There could be many ways
of paying attention.

00:38:02.270 --> 00:38:04.870 align:middle line:84%
There could be many
needs to pay attention.

00:38:04.870 --> 00:38:09.350 align:middle line:84%
Which means that let's have
many of these attention heads,

00:38:09.350 --> 00:38:11.370 align:middle line:84%
and each one could be
learning something else.

00:38:11.370 --> 00:38:13.270 align:middle line:84%
It's exactly like
having lots of filters

00:38:13.270 --> 00:38:15.275 align:middle line:90%
in a convolutional network.

00:38:15.275 --> 00:38:17.150 align:middle line:84%
One filter might learn
a line, another filter

00:38:17.150 --> 00:38:18.890 align:middle line:84%
might learn a curve,
and so on and so forth.

00:38:18.890 --> 00:38:20.510 align:middle line:84%
And we don't want
to decide a priori,

00:38:20.510 --> 00:38:22.290 align:middle line:84%
oh, you're going to
learn a line, right?

00:38:22.290 --> 00:38:23.810 align:middle line:84%
Similarly, here, we're not
telling any of these things

00:38:23.810 --> 00:38:24.893 align:middle line:90%
on what you have to learn.

00:38:24.893 --> 00:38:27.430 align:middle line:84%
They just have to learn based
on the training process.

00:38:27.430 --> 00:38:29.030 align:middle line:90%
So what we do is--

00:38:29.030 --> 00:38:30.990 align:middle line:84%
so actually, this is
an example where this,

00:38:30.990 --> 00:38:32.750 align:middle line:84%
from the original
transformer paper,

00:38:32.750 --> 00:38:36.010 align:middle line:84%
where this sentence is
"The lawyer will--" sorry.

00:38:36.010 --> 00:38:39.230 align:middle line:84%
"The law will never be
perfect, but its application

00:38:39.230 --> 00:38:40.710 align:middle line:90%
should be just.

00:38:40.710 --> 00:38:44.470 align:middle line:84%
This is what we are
missing, in my opinion."

00:38:44.470 --> 00:38:45.900 align:middle line:90%
A complicated sentence, right?

00:38:45.900 --> 00:38:48.400 align:middle line:84%
So the first one,
attention head--

00:38:48.400 --> 00:38:50.680 align:middle line:84%
actually, this is the
pattern of things.

00:38:50.680 --> 00:38:53.280 align:middle line:84%
So for example, the
word "perfect" here.

00:38:53.280 --> 00:38:57.520 align:middle line:84%
the contextual embedding
of the word "perfect"

00:38:57.520 --> 00:39:02.732 align:middle line:84%
draws upon heavily from the
word "law" in this example.

00:39:02.732 --> 00:39:04.602 align:middle line:84%
If you look at another
attention head,

00:39:04.602 --> 00:39:06.560 align:middle line:84%
the contextual embedding
for the word "perfect"

00:39:06.560 --> 00:39:11.040 align:middle line:84%
is actually drawing heavily from
just perfect and nothing else.

00:39:11.040 --> 00:39:13.440 align:middle line:84%
And if you look at other
words, the patterns

00:39:13.440 --> 00:39:17.160 align:middle line:84%
are subtly different of what
it's paying attention to.

00:39:17.160 --> 00:39:19.160 align:middle line:84%
So these are two
different attention heads.

00:39:19.160 --> 00:39:22.120 align:middle line:84%
And they are learning
different kinds of attentions.

00:39:22.120 --> 00:39:26.320 align:middle line:84%
In reality, trying to make
sense of why they pay attention

00:39:26.320 --> 00:39:29.148 align:middle line:84%
to the way they do, it's
usually quite difficult

00:39:29.148 --> 00:39:29.940 align:middle line:90%
to figure that out.

00:39:29.940 --> 00:39:31.420 align:middle line:90%
You can't actually interpret it.

00:39:31.420 --> 00:39:33.420 align:middle line:84%
But when you have lots
of attention heads,

00:39:33.420 --> 00:39:35.400 align:middle line:84%
the performance on the
task that you care about

00:39:35.400 --> 00:39:38.000 align:middle line:90%
gets really much better.

00:39:38.000 --> 00:39:40.000 align:middle line:84%
And then you say,
OK, I can use that.

00:39:40.000 --> 00:39:40.920 align:middle line:90%
Yeah.

00:39:40.920 --> 00:39:42.860 align:middle line:84%
AUDIENCE: Not to
oversimplify anything.

00:39:42.860 --> 00:39:45.880 align:middle line:84%
Is this -- is the idea
behind this [INAUDIBLE]?

00:39:45.880 --> 00:39:50.525 align:middle line:90%


00:39:50.525 --> 00:39:51.650 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:39:51.650 --> 00:39:52.550 align:middle line:90%
Same logic.

00:39:52.550 --> 00:39:53.660 align:middle line:90%
Same logic.

00:39:53.660 --> 00:39:54.343 align:middle line:90%
Yeah.

00:39:54.343 --> 00:39:56.590 align:middle line:84%
AUDIENCE: [INAUDIBLE],
with that filter,

00:39:56.590 --> 00:39:59.850 align:middle line:84%
we had designed a particular
filter for a particular task

00:39:59.850 --> 00:40:02.130 align:middle line:84%
in terms of 1's and
0's on that line.

00:40:02.130 --> 00:40:04.010 align:middle line:84%
But here, like the
attention head,

00:40:04.010 --> 00:40:06.210 align:middle line:84%
how do we differentiate--
how do we create

00:40:06.210 --> 00:40:08.890 align:middle line:84%
an instance for each
different kind of scenario

00:40:08.890 --> 00:40:10.370 align:middle line:84%
or different kind
of parameters we

00:40:10.370 --> 00:40:13.490 align:middle line:84%
want to all kind of different
kind of pattern [INAUDIBLE].

00:40:13.490 --> 00:40:15.782 align:middle line:84%
RAMA RAMAKRISHNAN: Actually,
in the convolutional case,

00:40:15.782 --> 00:40:17.850 align:middle line:84%
the 1's and 0's I had
were just example numbers

00:40:17.850 --> 00:40:20.090 align:middle line:84%
to show that that
particular filter could

00:40:20.090 --> 00:40:22.130 align:middle line:84%
detect a vertical line
or a horizontal line.

00:40:22.130 --> 00:40:23.783 align:middle line:84%
You will recall that
when we actually

00:40:23.783 --> 00:40:25.450 align:middle line:84%
train a convolution
network, we actually

00:40:25.450 --> 00:40:26.950 align:middle line:90%
don't specify the numbers.

00:40:26.950 --> 00:40:29.010 align:middle line:84%
We start with the random
initialized weights,

00:40:29.010 --> 00:40:32.170 align:middle line:84%
and then we let
backpropagation figure it out.

00:40:32.170 --> 00:40:34.750 align:middle line:84%
Similarly, here, we don't
decide any of these things.

00:40:34.750 --> 00:40:37.850 align:middle line:84%
We just let backprop
figure it out.

00:40:37.850 --> 00:40:39.907 align:middle line:84%
And now the question
of what are the weights

00:40:39.907 --> 00:40:41.490 align:middle line:84%
that are actually
going to be learned,

00:40:41.490 --> 00:40:44.220 align:middle line:90%
we'll come to that in a bit.

00:40:44.220 --> 00:40:44.720 align:middle line:90%
Yeah.

00:40:44.720 --> 00:40:47.940 align:middle line:90%


00:40:47.940 --> 00:40:52.020 align:middle line:84%
AUDIENCE: I was wondering, how
come we have different attention

00:40:52.020 --> 00:40:55.980 align:middle line:84%
heads, even though it seems like
they're only function of the dot

00:40:55.980 --> 00:40:57.780 align:middle line:84%
product, and we have
the same dot product

00:40:57.780 --> 00:40:58.903 align:middle line:90%
for the same embeddings.

00:40:58.903 --> 00:41:00.320 align:middle line:84%
RAMA RAMAKRISHNAN:
Great question.

00:41:00.320 --> 00:41:01.120 align:middle line:90%
Great question.

00:41:01.120 --> 00:41:03.580 align:middle line:84%
And I literally have a
note in my slide saying

00:41:03.580 --> 00:41:05.940 align:middle line:84%
if a student asks this
good question, tell

00:41:05.940 --> 00:41:08.500 align:middle line:90%
them to wait till Wednesday.

00:41:08.500 --> 00:41:09.840 align:middle line:90%
So, great question.

00:41:09.840 --> 00:41:11.580 align:middle line:84%
And we'll come back
to that on Wednesday

00:41:11.580 --> 00:41:13.450 align:middle line:84%
and spend a fair
amount of time on it.

00:41:13.450 --> 00:41:17.860 align:middle line:84%
So the point that's
being made here is that--

00:41:17.860 --> 00:41:19.780 align:middle line:90%
oops.

00:41:19.780 --> 00:41:24.220 align:middle line:84%
When we look at self-attention,
the embeddings came in,

00:41:24.220 --> 00:41:25.860 align:middle line:84%
and we did all
these dot products,

00:41:25.860 --> 00:41:28.300 align:middle line:84%
and the contextual things
popped out the other end.

00:41:28.300 --> 00:41:30.300 align:middle line:84%
Note that inside the
self-attention box

00:41:30.300 --> 00:41:32.700 align:middle line:90%
there are no parameters.

00:41:32.700 --> 00:41:34.220 align:middle line:90%
There are no parameters.

00:41:34.220 --> 00:41:36.780 align:middle line:84%
So the question that
is being raised here

00:41:36.780 --> 00:41:39.980 align:middle line:84%
is that, so what are we
learning, really, if there is

00:41:39.980 --> 00:41:41.657 align:middle line:90%
nothing inside to be learned?

00:41:41.657 --> 00:41:43.490 align:middle line:84%
If there are no parameters,
no coefficients,

00:41:43.490 --> 00:41:44.130 align:middle line:90%
what are we learning?

00:41:44.130 --> 00:41:45.090 align:middle line:90%
That's the question.

00:41:45.090 --> 00:41:48.435 align:middle line:84%
And by extension, if we have two
of these, and neither of them

00:41:48.435 --> 00:41:50.060 align:middle line:84%
is learning anything,
what's the point?

00:41:50.060 --> 00:41:52.870 align:middle line:90%


00:41:52.870 --> 00:41:56.210 align:middle line:84%
Sadly, you have to
wait till Wednesday.

00:41:56.210 --> 00:41:58.010 align:middle line:84%
But we have a great
answer to the question,

00:41:58.010 --> 00:41:59.730 align:middle line:90%
so it'll be worth it.

00:41:59.730 --> 00:42:03.310 align:middle line:84%
And if you can't stand the
suspense, read the book.

00:42:03.310 --> 00:42:06.790 align:middle line:84%
All right, so that's why
we need multiple heads.

00:42:06.790 --> 00:42:09.590 align:middle line:84%
And now to come back to
this, so what we do is,

00:42:09.590 --> 00:42:12.410 align:middle line:84%
it goes through this head,
and you get these w1's.

00:42:12.410 --> 00:42:15.830 align:middle line:84%
And it goes through here, and
we get another set of w's.

00:42:15.830 --> 00:42:19.670 align:middle line:84%
Then, what we do at the very
end is we concatenate them.

00:42:19.670 --> 00:42:20.170 align:middle line:90%
OK.

00:42:20.170 --> 00:42:21.470 align:middle line:90%
We concatenate them.

00:42:21.470 --> 00:42:22.717 align:middle line:90%
And we do a projection.

00:42:22.717 --> 00:42:24.050 align:middle line:90%
And this is what I mean by that.

00:42:24.050 --> 00:42:29.310 align:middle line:90%


00:42:29.310 --> 00:42:33.310 align:middle line:84%
So we have-- this is one
self-attention head--

00:42:33.310 --> 00:42:35.950 align:middle line:90%
self-attention 1.

00:42:35.950 --> 00:42:38.950 align:middle line:90%
This is self-attention 2.

00:42:38.950 --> 00:42:41.880 align:middle line:90%
And let's say that.

00:42:41.880 --> 00:42:44.680 align:middle line:90%
w1 hat comes out.

00:42:44.680 --> 00:42:48.140 align:middle line:84%
And I'm just going to call
it Z1 for the same thing

00:42:48.140 --> 00:42:49.960 align:middle line:90%
so that there's no name clash.

00:42:49.960 --> 00:42:53.040 align:middle line:84%
And w2, w6, all of
them are coming, right?

00:42:53.040 --> 00:42:55.360 align:middle line:90%
Let's focus on w1 and z1.

00:42:55.360 --> 00:42:59.360 align:middle line:84%
w1 and z1 are both contextual
embeddings for the same word,

00:42:59.360 --> 00:43:01.560 align:middle line:90%
for the first word, word 1.

00:43:01.560 --> 00:43:05.360 align:middle line:84%
And so what we do is,
let's say this w1,

00:43:05.360 --> 00:43:07.960 align:middle line:84%
let's say this
vector is like this.

00:43:07.960 --> 00:43:12.400 align:middle line:84%
And let's say that this
vector is like this.

00:43:12.400 --> 00:43:14.400 align:middle line:84%
What I mean when
I say concatenated

00:43:14.400 --> 00:43:19.820 align:middle line:84%
here is we literally take this
word here, this embedding here.

00:43:19.820 --> 00:43:23.120 align:middle line:90%
Then we take this thing here.

00:43:23.120 --> 00:43:25.140 align:middle line:84%
And we just make
it a long vector.

00:43:25.140 --> 00:43:26.340 align:middle line:90%
We concatenate it.

00:43:26.340 --> 00:43:31.020 align:middle line:84%
But now, this vector has
become twice as long, right?

00:43:31.020 --> 00:43:33.680 align:middle line:84%
But remember, we
always want to preserve

00:43:33.680 --> 00:43:36.800 align:middle line:84%
the number of inputs we have
and the lengths of these vectors

00:43:36.800 --> 00:43:38.330 align:middle line:90%
everywhere as we go along.

00:43:38.330 --> 00:43:41.170 align:middle line:84%
So what we do is,
at this point, we

00:43:41.170 --> 00:43:44.810 align:middle line:84%
run it through a
single dense layer,

00:43:44.810 --> 00:43:47.290 align:middle line:84%
which will take this
thing and make it back

00:43:47.290 --> 00:43:50.010 align:middle line:84%
into the same small
shape as before.

00:43:50.010 --> 00:43:51.080 align:middle line:90%
So this is a dense layer.

00:43:51.080 --> 00:43:54.330 align:middle line:90%


00:43:54.330 --> 00:43:55.030 align:middle line:90%
That's it.

00:43:55.030 --> 00:43:58.570 align:middle line:84%
So this vector comes in,
and it gets compressed back

00:43:58.570 --> 00:44:01.210 align:middle line:84%
to the original shape
that came out of here.

00:44:01.210 --> 00:44:04.930 align:middle line:84%
So you could have 20 of
these attention heads,

00:44:04.930 --> 00:44:06.910 align:middle line:84%
and the concatenating
will be 20 times longer.

00:44:06.910 --> 00:44:08.243 align:middle line:90%
And then you just project, boom.

00:44:08.243 --> 00:44:12.890 align:middle line:84%
One dense layer comes back
to the original shape.

00:44:12.890 --> 00:44:16.290 align:middle line:90%
So that is the projection step.

00:44:16.290 --> 00:44:20.450 align:middle line:84%
And that's what I mean here when
I say concatenate and project.

00:44:20.450 --> 00:44:22.710 align:middle line:84%
So at this point, what we
have is things come in.

00:44:22.710 --> 00:44:25.230 align:middle line:84%
We contextualize them using
these different attention heads.

00:44:25.230 --> 00:44:27.610 align:middle line:84%
And when they come out
of the attention heads,

00:44:27.610 --> 00:44:28.630 align:middle line:90%
we take them all.

00:44:28.630 --> 00:44:31.250 align:middle line:84%
We just concatenate them
and then compress them back

00:44:31.250 --> 00:44:33.370 align:middle line:84%
to the same original
starting shape.

00:44:33.370 --> 00:44:37.240 align:middle line:84%
If these vectors are 100 units
long or 100 dimension long,

00:44:37.240 --> 00:44:39.660 align:middle line:90%
whatever comes out is 100 still.

00:44:39.660 --> 00:44:42.835 align:middle line:84%
And to preserving this
size as we go along

00:44:42.835 --> 00:44:44.460 align:middle line:84%
is very important
for reasons that will

00:44:44.460 --> 00:44:47.460 align:middle line:90%
become apparent a bit later.

00:44:47.460 --> 00:44:50.340 align:middle line:84%
So that is the
multi-attention thing.

00:44:50.340 --> 00:44:54.300 align:middle line:84%
Now, a final tweak
for today is that we

00:44:54.300 --> 00:44:58.680 align:middle line:84%
will inject some non-linearity
with some dense layers,

00:44:58.680 --> 00:45:00.160 align:middle line:84%
dense ReLU layers
at the very end.

00:45:00.160 --> 00:45:02.680 align:middle line:84%
So we went through a
bunch of attention heads.

00:45:02.680 --> 00:45:05.220 align:middle line:84%
We came up with a bunch of
contextual embeddings now.

00:45:05.220 --> 00:45:07.940 align:middle line:84%
So at this point
so far, there are

00:45:07.940 --> 00:45:11.860 align:middle line:84%
no-- since there are no
parameters inside these boxes,

00:45:11.860 --> 00:45:13.940 align:middle line:84%
and there are some
parameters here,

00:45:13.940 --> 00:45:15.408 align:middle line:84%
we need to do some
non-linearity.

00:45:15.408 --> 00:45:17.700 align:middle line:84%
So far, there has been nothing
that's nonlinear so far.

00:45:17.700 --> 00:45:21.520 align:middle line:84%
So here we actually send it
through one or more ReLUs.

00:45:21.520 --> 00:45:24.312 align:middle line:84%
Typically, they
just use one ReLU.

00:45:24.312 --> 00:45:25.800 align:middle line:90%
And what I mean by that--

00:45:25.800 --> 00:45:34.140 align:middle line:90%


00:45:34.140 --> 00:45:34.880 align:middle line:90%
oops, sorry.

00:45:34.880 --> 00:45:37.950 align:middle line:90%


00:45:37.950 --> 00:45:39.850 align:middle line:90%
So this is what we had here.

00:45:39.850 --> 00:45:47.440 align:middle line:84%
And then we take it in
and then run it through--

00:45:47.440 --> 00:45:50.190 align:middle line:90%


00:45:50.190 --> 00:45:58.670 align:middle line:84%
actually, we typically
run it through a ReLU.

00:45:58.670 --> 00:46:01.470 align:middle line:90%
This is a nice ReLU.

00:46:01.470 --> 00:46:03.900 align:middle line:84%
And then the rule of
thumb, as you will see,

00:46:03.900 --> 00:46:06.710 align:middle line:84%
if, let's say, this vector
is, say, 100 dimensions long,

00:46:06.710 --> 00:46:10.762 align:middle line:84%
they typically will choose a
ReLU which is about 400 wide.

00:46:10.762 --> 00:46:16.710 align:middle line:84%
And then it just gets projected
out again back to 100.

00:46:16.710 --> 00:46:20.230 align:middle line:90%
So this is just a simple--

00:46:20.230 --> 00:46:23.110 align:middle line:84%
the input comes in, goes through
a single hidden layer with four

00:46:23.110 --> 00:46:24.770 align:middle line:90%
times as many as here.

00:46:24.770 --> 00:46:29.830 align:middle line:84%
And then it project another
dense layer to 100 again.

00:46:29.830 --> 00:46:32.710 align:middle line:84%
And since there
are ReLUs here, we

00:46:32.710 --> 00:46:35.880 align:middle line:84%
have injected some non-linearity
into the processing.

00:46:35.880 --> 00:46:39.280 align:middle line:84%
Now, a lot of this
stuff, when it came out,

00:46:39.280 --> 00:46:41.980 align:middle line:90%
felt very ad hoc, right?

00:46:41.980 --> 00:46:45.600 align:middle line:84%
It didn't come from some
deep theoretical motivations,

00:46:45.600 --> 00:46:47.640 align:middle line:84%
but people had strong
intuitions as to why

00:46:47.640 --> 00:46:49.040 align:middle line:90%
these things were helpful.

00:46:49.040 --> 00:46:51.900 align:middle line:84%
And as it turns out, since
the transformer came out,

00:46:51.900 --> 00:46:55.417 align:middle line:84%
people have tried to optimize
every aspect of this thing.

00:46:55.417 --> 00:46:57.000 align:middle line:84%
It's actually pretty
difficult to beat

00:46:57.000 --> 00:46:58.923 align:middle line:90%
the starting architecture.

00:46:58.923 --> 00:47:00.840 align:middle line:84%
Improvements have been
made, but it's actually

00:47:00.840 --> 00:47:02.760 align:middle line:90%
a very robust architecture.

00:47:02.760 --> 00:47:05.360 align:middle line:90%
So that's what's going on here.

00:47:05.360 --> 00:47:08.960 align:middle line:84%
And then, when we come
out of this thing,

00:47:08.960 --> 00:47:10.920 align:middle line:84%
this is what we have--
the story so far.

00:47:10.920 --> 00:47:13.540 align:middle line:84%
We start with random
standalone embeddings.

00:47:13.540 --> 00:47:15.255 align:middle line:90%
These could be glove embeddings.

00:47:15.255 --> 00:47:16.380 align:middle line:90%
It could be random weights.

00:47:16.380 --> 00:47:17.320 align:middle line:90%
It doesn't matter.

00:47:17.320 --> 00:47:19.860 align:middle line:84%
It goes through a bunch
of self-attention heads.

00:47:19.860 --> 00:47:23.840 align:middle line:84%
We concatenate it when it
comes out the other end,

00:47:23.840 --> 00:47:25.960 align:middle line:84%
concatenate it when it
comes out the other end.

00:47:25.960 --> 00:47:28.700 align:middle line:84%
And then we project it back
to the same size as before.

00:47:28.700 --> 00:47:32.870 align:middle line:84%
Then we run it through a ReLU
followed by a linear layer.

00:47:32.870 --> 00:47:34.150 align:middle line:90%
And we get these things again.

00:47:34.150 --> 00:47:37.110 align:middle line:84%
So in this whole process,
if six things came in,

00:47:37.110 --> 00:47:38.410 align:middle line:90%
six things will come out.

00:47:38.410 --> 00:47:41.970 align:middle line:84%
And if those six
things that came in

00:47:41.970 --> 00:47:43.730 align:middle line:84%
were standalone
embedding vectors

00:47:43.730 --> 00:47:46.802 align:middle line:84%
of 100 dimensions, what comes
out is also 100 dimensions.

00:47:46.802 --> 00:47:49.010 align:middle line:84%
So in that sense, you could
think of this whole thing

00:47:49.010 --> 00:47:51.830 align:middle line:84%
as a black box in which
whatever you send in,

00:47:51.830 --> 00:47:54.327 align:middle line:84%
the same number of things will
come out of the same length.

00:47:54.327 --> 00:47:56.410 align:middle line:84%
The numbers will be
different, because they've now

00:47:56.410 --> 00:47:58.530 align:middle line:90%
been heavily contextualized.

00:47:58.530 --> 00:48:00.560 align:middle line:84%
The numbers are much
smarter, in other words.

00:48:00.560 --> 00:48:03.405 align:middle line:90%


00:48:03.405 --> 00:48:05.530 align:middle line:84%
So far, what we have seen
is that we have satisfied

00:48:05.530 --> 00:48:06.890 align:middle line:90%
two of the three requirements.

00:48:06.890 --> 00:48:08.610 align:middle line:84%
We have taken the
context of each word

00:48:08.610 --> 00:48:11.010 align:middle line:84%
into account by using
these dot products

00:48:11.010 --> 00:48:12.490 align:middle line:84%
in the self-attention
layer, and we

00:48:12.490 --> 00:48:14.990 align:middle line:84%
can generate an output that is
the same length as the input.

00:48:14.990 --> 00:48:16.810 align:middle line:84%
But we have ignored
the fact that-- we have

00:48:16.810 --> 00:48:20.310 align:middle line:90%
ignored word order completely.

00:48:20.310 --> 00:48:23.410 align:middle line:84%
Because whether I had said "the
train slowly left the station,"

00:48:23.410 --> 00:48:26.810 align:middle line:84%
or I had said that, "The
station slowly left the train,"

00:48:26.810 --> 00:48:30.780 align:middle line:84%
this thing won't
know the difference,

00:48:30.780 --> 00:48:34.560 align:middle line:84%
because dot products function
on sets, not on sequences.

00:48:34.560 --> 00:48:36.620 align:middle line:90%
They function on sets.

00:48:36.620 --> 00:48:38.740 align:middle line:84%
You should convince
yourself of this.

00:48:38.740 --> 00:48:40.620 align:middle line:84%
Regardless of the
order, the dot reduction

00:48:40.620 --> 00:48:43.210 align:middle line:84%
doesn't change anything,
because we are doing every pair.

00:48:43.210 --> 00:48:46.405 align:middle line:90%


00:48:46.405 --> 00:48:48.780 align:middle line:84%
So the question is, how do we
take the order of the words

00:48:48.780 --> 00:48:50.710 align:middle line:90%
into account?

00:48:50.710 --> 00:48:52.460 align:middle line:84%
As I was saying, we
can scramble the order

00:48:52.460 --> 00:48:53.660 align:middle line:84%
of the words in a
sentence, and we'll

00:48:53.660 --> 00:48:55.827 align:middle line:84%
get the exact same contextual
embeddings at the end.

00:48:55.827 --> 00:48:58.580 align:middle line:84%
So by the way, if you're working
on a problem in which the order

00:48:58.580 --> 00:49:01.620 align:middle line:84%
doesn't matter, then
you could stop right now

00:49:01.620 --> 00:49:03.625 align:middle line:90%
and use the transformer.

00:49:03.625 --> 00:49:05.500 align:middle line:84%
And there are many
problems that are actually

00:49:05.500 --> 00:49:07.980 align:middle line:84%
in that category, where
the order doesn't matter.

00:49:07.980 --> 00:49:11.300 align:middle line:84%
So if you take traditional
structured data, or tabular

00:49:11.300 --> 00:49:13.140 align:middle line:90%
data--

00:49:13.140 --> 00:49:15.270 align:middle line:84%
blood pressure, cholesterol
level, boom, boom boom.

00:49:15.270 --> 00:49:16.520 align:middle line:90%
Does it predict heart disease?

00:49:16.520 --> 00:49:18.240 align:middle line:84%
Well, there is no
order in that thing.

00:49:18.240 --> 00:49:22.100 align:middle line:84%
You can use a transformer as
is, without doing anything more.

00:49:22.100 --> 00:49:25.380 align:middle line:84%
So transformers work for
both sets and sequences

00:49:25.380 --> 00:49:27.180 align:middle line:90%
where order matters.

00:49:27.180 --> 00:49:27.870 align:middle line:90%
OK.

00:49:27.870 --> 00:49:33.150 align:middle line:84%
So the fix for this is something
called the positional encoding.

00:49:33.150 --> 00:49:34.455 align:middle line:90%
So what we do is very simple.

00:49:34.455 --> 00:49:35.830 align:middle line:84%
By the way, there
are many things

00:49:35.830 --> 00:49:40.830 align:middle line:84%
that have been invented
to tell transformers--

00:49:40.830 --> 00:49:42.750 align:middle line:84%
to give transformers
some information

00:49:42.750 --> 00:49:45.710 align:middle line:84%
about the order of each of
the things that are coming in.

00:49:45.710 --> 00:49:47.928 align:middle line:84%
I'm going to go with
something called the--

00:49:47.928 --> 00:49:49.470 align:middle line:84%
simplest possible
way, which actually

00:49:49.470 --> 00:49:51.030 align:middle line:90%
works pretty well in practice.

00:49:51.030 --> 00:49:54.990 align:middle line:84%
So what we do is,
for each position,

00:49:54.990 --> 00:49:56.637 align:middle line:84%
each possible
position in the input,

00:49:56.637 --> 00:49:58.470 align:middle line:84%
starting from the first
position all the way

00:49:58.470 --> 00:50:00.910 align:middle line:84%
through the last
position, we imagine

00:50:00.910 --> 00:50:05.590 align:middle line:84%
that that position itself
is a categorical variable.

00:50:05.590 --> 00:50:08.330 align:middle line:84%
If a sentence can
only be 30 words long,

00:50:08.330 --> 00:50:11.270 align:middle line:84%
let's say, we say that, hey,
the position of each word

00:50:11.270 --> 00:50:14.550 align:middle line:90%
is a number between 0 and 29.

00:50:14.550 --> 00:50:18.030 align:middle line:84%
And so we can just think of
it as a categorical variable.

00:50:18.030 --> 00:50:19.970 align:middle line:84%
And because a
categorical variable,

00:50:19.970 --> 00:50:22.150 align:middle line:84%
we could just imagine
an embedding for that,

00:50:22.150 --> 00:50:23.443 align:middle line:90%
for each potential value.

00:50:23.443 --> 00:50:25.110 align:middle line:84%
So it will become
clear in just a moment

00:50:25.110 --> 00:50:26.960 align:middle line:84%
because I have a
numerical example.

00:50:26.960 --> 00:50:28.760 align:middle line:84%
And so what we do
is we will just

00:50:28.760 --> 00:50:30.920 align:middle line:84%
take that standalone
embedding, and then we'll

00:50:30.920 --> 00:50:33.040 align:middle line:84%
take this position
embedding, which

00:50:33.040 --> 00:50:35.207 align:middle line:84%
represents the position of
the word in the sentence.

00:50:35.207 --> 00:50:37.440 align:middle line:90%
And we'll just add them up.

00:50:37.440 --> 00:50:39.680 align:middle line:90%
Yeah.

00:50:39.680 --> 00:50:42.360 align:middle line:84%
AUDIENCE: So if, in the
initial sentence, it says,

00:50:42.360 --> 00:50:43.820 align:middle line:90%
I have made a mistake.

00:50:43.820 --> 00:50:48.040 align:middle line:84%
So I just write it as, "The
train slowly a station."

00:50:48.040 --> 00:50:50.780 align:middle line:84%
So which means my output is
actually going to be wrong.

00:50:50.780 --> 00:50:52.180 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:50:52.180 --> 00:50:55.600 align:middle line:84%
Now the transformers are, since
they're trained on lots of data,

00:50:55.600 --> 00:50:58.160 align:middle line:84%
they will be quite
robust to these things.

00:50:58.160 --> 00:51:00.680 align:middle line:84%
But strictly, arithmetically
speaking, correct.

00:51:00.680 --> 00:51:01.180 align:middle line:90%
Yes.

00:51:01.180 --> 00:51:03.840 align:middle line:90%


00:51:03.840 --> 00:51:06.680 align:middle line:90%
So let's look at an example.

00:51:06.680 --> 00:51:11.260 align:middle line:84%
Let's assume that your
standalone embeddings-- right?

00:51:11.260 --> 00:51:13.385 align:middle line:90%
This is your vocabulary.

00:51:13.385 --> 00:51:15.760 align:middle line:84%
"unknown," "cat," "mat," "I,"
"sit," "love, "the," "you,"

00:51:15.760 --> 00:51:16.460 align:middle line:90%
"on."

00:51:16.460 --> 00:51:17.080 align:middle line:90%
That's it.

00:51:17.080 --> 00:51:18.920 align:middle line:90%
That's our vocabulary.

00:51:18.920 --> 00:51:22.148 align:middle line:84%
And for this vocabulary, we have
these standalone embeddings.

00:51:22.148 --> 00:51:24.690 align:middle line:84%
And just for argument, let's
assume these embeddings are only

00:51:24.690 --> 00:51:26.410 align:middle line:90%
2 long.

00:51:26.410 --> 00:51:28.570 align:middle line:84%
The dimension of
these embeddings is 2.

00:51:28.570 --> 00:51:31.030 align:middle line:84%
If you recall, the glove
embeddings we used last week,

00:51:31.030 --> 00:51:32.703 align:middle line:84%
I think they were
about 100 long.

00:51:32.703 --> 00:51:34.370 align:middle line:84%
And the ones we're
using in the homework

00:51:34.370 --> 00:51:35.952 align:middle line:90%
are even longer than that.

00:51:35.952 --> 00:51:37.910 align:middle line:84%
But here, we are assuming
they are only 2 long.

00:51:37.910 --> 00:51:42.770 align:middle line:84%
So the embedding for
"cat" is 0.5, 7.1

00:51:42.770 --> 00:51:43.270 align:middle line:90%
All right.

00:51:43.270 --> 00:51:46.250 align:middle line:84%
Now let's assume
that we can have,

00:51:46.250 --> 00:51:49.050 align:middle line:84%
at most, 10 words in any
sentence that's coming in.

00:51:49.050 --> 00:51:51.930 align:middle line:84%
And obviously, a particular
word could be in position 0

00:51:51.930 --> 00:51:53.650 align:middle line:90%
all the way through position 9.

00:51:53.650 --> 00:51:57.150 align:middle line:84%
And we will learn embeddings
for each of these positions.

00:51:57.150 --> 00:51:59.730 align:middle line:84%
And these embeddings
are also 2 long--

00:51:59.730 --> 00:52:01.510 align:middle line:90%
2 units long, dimension 2.

00:52:01.510 --> 00:52:04.650 align:middle line:90%


00:52:04.650 --> 00:52:07.732 align:middle line:84%
Now where will these
embeddings come from?

00:52:07.732 --> 00:52:09.190 align:middle line:84%
What's the answer
to that question?

00:52:09.190 --> 00:52:10.890 align:middle line:84%
What is the answer to the
general question of where

00:52:10.890 --> 00:52:12.098 align:middle line:90%
will these weights come from?

00:52:12.098 --> 00:52:14.428 align:middle line:90%


00:52:14.428 --> 00:52:15.720 align:middle line:90%
We will learn it with backprop.

00:52:15.720 --> 00:52:20.026 align:middle line:90%


00:52:20.026 --> 00:52:22.050 align:middle line:84%
We will start initially
with random numbers,

00:52:22.050 --> 00:52:24.540 align:middle line:84%
and then we'll get them,
make them better and better

00:52:24.540 --> 00:52:26.580 align:middle line:90%
as over the course of training.

00:52:26.580 --> 00:52:31.100 align:middle line:84%
So what we do is we have these
two tables of embeddings--

00:52:31.100 --> 00:52:34.020 align:middle line:84%
the standalone embedding for the
word and the position embedding.

00:52:34.020 --> 00:52:36.940 align:middle line:84%
And then, we
literally add them up.

00:52:36.940 --> 00:52:39.140 align:middle line:84%
So for example,
let's say the word,

00:52:39.140 --> 00:52:41.620 align:middle line:84%
the sentence that came
in is "Cat sat mat."

00:52:41.620 --> 00:52:42.680 align:middle line:90%
That's a sentence.

00:52:42.680 --> 00:52:44.780 align:middle line:84%
It's got three words--
"cat," "sat," "mat."

00:52:44.780 --> 00:52:48.540 align:middle line:84%
So what we do is we say,
well, the embedding for "cat"

00:52:48.540 --> 00:52:51.140 align:middle line:90%
is this thing here, 0.571.

00:52:51.140 --> 00:52:53.380 align:middle line:90%
So I write it here, 0.5, 7.1.

00:52:53.380 --> 00:52:56.180 align:middle line:84%
"Cat" happens to be the
zeroth position of the word.

00:52:56.180 --> 00:52:59.800 align:middle line:84%
So I grab the embedding
for 0, which is 1.33.9.

00:52:59.800 --> 00:53:00.780 align:middle line:90%
I stick it there.

00:53:00.780 --> 00:53:05.680 align:middle line:84%
And then I literally add them
up, 0.5 plus 1.3, 1.8, 11.0.

00:53:05.680 --> 00:53:07.220 align:middle line:90%
That's it.

00:53:07.220 --> 00:53:12.140 align:middle line:84%
So now the positional encoded
embedding for the word "cat"

00:53:12.140 --> 00:53:14.300 align:middle line:90%
is 1.8, 11.0--

00:53:14.300 --> 00:53:18.420 align:middle line:90%
not 0.5, 7.1.

00:53:18.420 --> 00:53:20.990 align:middle line:84%
So if "cat" happens to
show up in another part

00:53:20.990 --> 00:53:23.690 align:middle line:84%
of the sentence-- let's say
instead of "Cat sat mat,"

00:53:23.690 --> 00:53:28.150 align:middle line:90%
we had "Mat sat cat."

00:53:28.150 --> 00:53:31.990 align:middle line:84%
Now "cat" is now the third
position, which is 0, 1, and 2,

00:53:31.990 --> 00:53:34.490 align:middle line:84%
which means its
embedding doesn't change.

00:53:34.490 --> 00:53:35.770 align:middle line:90%
It's just embedding for "cat."

00:53:35.770 --> 00:53:38.290 align:middle line:84%
But now, instead of picking
0, we'll pick this one--

00:53:38.290 --> 00:53:41.080 align:middle line:84%
0.681-- and put that here
and add them up instead.

00:53:41.080 --> 00:53:43.710 align:middle line:90%


00:53:43.710 --> 00:53:46.870 align:middle line:84%
So this is the idea of
the positional encoding.

00:53:46.870 --> 00:53:49.970 align:middle line:84%
This is how we inject position
knowledge into the transformer.

00:53:49.970 --> 00:53:52.950 align:middle line:90%


00:53:52.950 --> 00:53:54.727 align:middle line:90%
Yes.

00:53:54.727 --> 00:53:56.310 align:middle line:84%
AUDIENCE: The position
embedding would

00:53:56.310 --> 00:53:57.670 align:middle line:90%
be different for each sentence?

00:53:57.670 --> 00:53:58.730 align:middle line:90%
Like, how do you--

00:53:58.730 --> 00:54:00.030 align:middle line:84%
RAMA RAMAKRISHNAN: No, this
is just one table, which

00:54:00.030 --> 00:54:01.750 align:middle line:90%
tells you what the position is.

00:54:01.750 --> 00:54:04.190 align:middle line:84%
So it says for a
word that appears

00:54:04.190 --> 00:54:06.950 align:middle line:84%
in the seventh position
in any input sentence

00:54:06.950 --> 00:54:09.190 align:middle line:84%
that you're feeding in,
this is the embedding

00:54:09.190 --> 00:54:11.980 align:middle line:84%
that you need to use
for that position.

00:54:11.980 --> 00:54:16.870 align:middle line:90%


00:54:16.870 --> 00:54:20.540 align:middle line:84%
AUDIENCE: If a word appears
twice in the same sentence,

00:54:20.540 --> 00:54:21.600 align:middle line:90%
how do we [INAUDIBLE]?

00:54:21.600 --> 00:54:23.017 align:middle line:84%
RAMA RAMAKRISHNAN:
Great question.

00:54:23.017 --> 00:54:26.540 align:middle line:84%
So let's say, just for argument,
let's say the sentence was "Cat

00:54:26.540 --> 00:54:27.560 align:middle line:90%
cat cat."

00:54:27.560 --> 00:54:33.160 align:middle line:84%
So for each one of those
cat-- for "Cat cat cat,"

00:54:33.160 --> 00:54:34.720 align:middle line:84%
this embedding
will be the same--

00:54:34.720 --> 00:54:37.120 align:middle line:84%
0.5, 7.1-- because that happens
to be just the embedding

00:54:37.120 --> 00:54:39.480 align:middle line:84%
for "cat" regardless
of position.

00:54:39.480 --> 00:54:42.940 align:middle line:84%
But then the first "cat,"
for the first "cat,"

00:54:42.940 --> 00:54:46.660 align:middle line:84%
we will use 1.3,
3.9 as the addition.

00:54:46.660 --> 00:54:50.040 align:middle line:84%
For the second "cat," we'll use
6.3, 3.7, and the third "cat,"

00:54:50.040 --> 00:54:51.640 align:middle line:90%
we'll use 0.6, 8.1.

00:54:51.640 --> 00:54:53.380 align:middle line:84%
So only the things
that are adding,

00:54:53.380 --> 00:54:54.880 align:middle line:84%
the positional
encoding will change,

00:54:54.880 --> 00:54:56.125 align:middle line:90%
the positional embedding.

00:54:56.125 --> 00:54:58.000 align:middle line:84%
So the resulting sum is
going to be different

00:54:58.000 --> 00:55:00.120 align:middle line:84%
for each of these three words,
even though they're exactly

00:55:00.120 --> 00:55:00.980 align:middle line:90%
the same word.

00:55:00.980 --> 00:55:05.880 align:middle line:90%


00:55:05.880 --> 00:55:08.080 align:middle line:84%
AUDIENCE: Is that
position embedding table

00:55:08.080 --> 00:55:10.360 align:middle line:84%
specific to that
standalone embedding table?

00:55:10.360 --> 00:55:14.200 align:middle line:84%
If you were to add or remove
some words from the standalone?

00:55:14.200 --> 00:55:15.920 align:middle line:84%
RAMA RAMAKRISHNAN:
It's independent.

00:55:15.920 --> 00:55:16.480 align:middle line:90%
Independent.

00:55:16.480 --> 00:55:19.450 align:middle line:84%
It only depends on your
assumption about how long

00:55:19.450 --> 00:55:20.970 align:middle line:90%
the sentences can be.

00:55:20.970 --> 00:55:21.850 align:middle line:90%
That's it.

00:55:21.850 --> 00:55:24.430 align:middle line:84%
It doesn't really care about
what words are coming in.

00:55:24.430 --> 00:55:25.952 align:middle line:90%
That's a whole different thing.

00:55:25.952 --> 00:55:27.410 align:middle line:84%
So these are two
independent tables

00:55:27.410 --> 00:55:29.368 align:middle line:84%
that have just learned
as part of this process.

00:55:29.368 --> 00:55:31.650 align:middle line:90%


00:55:31.650 --> 00:55:35.730 align:middle line:84%
So yeah, I have the same thing
for "sat" and "mat," "sat"

00:55:35.730 --> 00:55:36.390 align:middle line:90%
and "mat."

00:55:36.390 --> 00:55:39.450 align:middle line:90%
That's what we have.

00:55:39.450 --> 00:55:42.450 align:middle line:84%
So just make sure you understand
these two slides to really make

00:55:42.450 --> 00:55:44.010 align:middle line:90%
sure the mechanics are clear.

00:55:44.010 --> 00:55:46.330 align:middle line:90%
Yeah.

00:55:46.330 --> 00:55:48.870 align:middle line:84%
AUDIENCE: How do you filter
out for filler words?

00:55:48.870 --> 00:55:53.650 align:middle line:84%
For example, if you're taking
NLP output for transcription,

00:55:53.650 --> 00:55:55.750 align:middle line:84%
and you were trying to
run a transformer on it,

00:55:55.750 --> 00:55:58.170 align:middle line:84%
and you have a lot
of "ums" and "likes"

00:55:58.170 --> 00:56:00.210 align:middle line:84%
that are
disproportionately large,

00:56:00.210 --> 00:56:03.610 align:middle line:84%
and have these random
assignments or really deep

00:56:03.610 --> 00:56:04.330 align:middle line:90%
embeddings.

00:56:04.330 --> 00:56:06.970 align:middle line:84%
Are there ways to filter
out a certain noise?

00:56:06.970 --> 00:56:09.428 align:middle line:84%
RAMA RAMAKRISHNAN: Typically,
what they do is, as we will--

00:56:09.428 --> 00:56:10.845 align:middle line:84%
we'll talk about
this thing called

00:56:10.845 --> 00:56:12.610 align:middle line:84%
byte-pair encoding,
in which we take

00:56:12.610 --> 00:56:15.940 align:middle line:84%
individual characters,
fragments of words,

00:56:15.940 --> 00:56:18.180 align:middle line:84%
and words into
account as tokens.

00:56:18.180 --> 00:56:19.980 align:middle line:84%
So when you hear
stuff like "um," "uh,"

00:56:19.980 --> 00:56:22.912 align:middle line:84%
and so on, it gets mapped
to these small tokens.

00:56:22.912 --> 00:56:24.870 align:middle line:84%
And then we treat them
as just any other token.

00:56:24.870 --> 00:56:29.377 align:middle line:90%


00:56:29.377 --> 00:56:31.460 align:middle line:84%
AUDIENCE: Yeah, is
aggregation, like, a simple sum

00:56:31.460 --> 00:56:34.380 align:middle line:84%
or [INAUDIBLE], the
actual semantic meaning

00:56:34.380 --> 00:56:36.140 align:middle line:84%
of the word in the
standalone embedding

00:56:36.140 --> 00:56:38.580 align:middle line:84%
might be more important
than its relative position

00:56:38.580 --> 00:56:39.828 align:middle line:90%
in the sentence?

00:56:39.828 --> 00:56:41.120 align:middle line:90%
RAMA RAMAKRISHNAN: It could be.

00:56:41.120 --> 00:56:42.787 align:middle line:84%
We just don't know a
priori whether it's

00:56:42.787 --> 00:56:45.380 align:middle line:84%
going to be important or not
for any particular sentence.

00:56:45.380 --> 00:56:48.985 align:middle line:84%
When we train the transformer
with a lot of textual data,

00:56:48.985 --> 00:56:51.360 align:middle line:84%
it will just figure out the
right values for these things

00:56:51.360 --> 00:56:55.220 align:middle line:84%
so that on average, the
accuracy is as high as possible.

00:56:55.220 --> 00:56:56.787 align:middle line:84%
So in many of these
things, there's

00:56:56.787 --> 00:56:58.620 align:middle line:84%
always a tension between
our human intuition

00:56:58.620 --> 00:57:00.990 align:middle line:84%
as to how it should work,
and whether you should just

00:57:00.990 --> 00:57:02.740 align:middle line:84%
throw it into the meat
grinder of backprop

00:57:02.740 --> 00:57:03.965 align:middle line:90%
and see what happens.

00:57:03.965 --> 00:57:05.340 align:middle line:84%
And so here, as
it turns out, you

00:57:05.340 --> 00:57:06.673 align:middle line:90%
can just throw it into backprop.

00:57:06.673 --> 00:57:09.060 align:middle line:84%
It'll actually do
a pretty good job.

00:57:09.060 --> 00:57:10.900 align:middle line:90%
Yeah.

00:57:10.900 --> 00:57:12.780 align:middle line:84%
AUDIENCE: For the
positional encoding,

00:57:12.780 --> 00:57:16.010 align:middle line:84%
we would just be
using the sum vector.

00:57:16.010 --> 00:57:18.270 align:middle line:84%
We wouldn't be using
this 2 by 3 matrix

00:57:18.270 --> 00:57:20.243 align:middle line:84%
that you have on the
far right, right?

00:57:20.243 --> 00:57:21.410 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, yeah.

00:57:21.410 --> 00:57:22.890 align:middle line:90%
This is just for demonstration.

00:57:22.890 --> 00:57:24.310 align:middle line:84%
Basically, this is the
thing that will actually

00:57:24.310 --> 00:57:25.410 align:middle line:90%
go into the transformer.

00:57:25.410 --> 00:57:26.350 align:middle line:90%
Correct.

00:57:26.350 --> 00:57:28.510 align:middle line:90%
Yeah.

00:57:28.510 --> 00:57:31.830 align:middle line:84%
That was just me being
overly verbose in the slides.

00:57:31.830 --> 00:57:33.310 align:middle line:90%
Yeah.

00:57:33.310 --> 00:57:35.890 align:middle line:84%
AUDIENCE: [INAUDIBLE] sentence
the input at this point,

00:57:35.890 --> 00:57:37.670 align:middle line:84%
are we still parsing
out punctuation,

00:57:37.670 --> 00:57:39.610 align:middle line:84%
or do we have a
multi-sentence input?

00:57:39.610 --> 00:57:41.350 align:middle line:84%
Is there positional
embedding vector

00:57:41.350 --> 00:57:42.810 align:middle line:90%
for each of the sentences?

00:57:42.810 --> 00:57:43.810 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:57:43.810 --> 00:57:47.495 align:middle line:84%
So here, basically, the
starting point is tokens, right?

00:57:47.495 --> 00:57:48.870 align:middle line:84%
And in our example,
because we're

00:57:48.870 --> 00:57:51.550 align:middle line:84%
working with the idea of simple
standardization and stripping

00:57:51.550 --> 00:57:52.467 align:middle line:90%
and things like that--

00:57:52.467 --> 00:57:53.950 align:middle line:90%
I'm just showing actual words--

00:57:53.950 --> 00:57:55.790 align:middle line:84%
if you go to
something like GPT-4,

00:57:55.790 --> 00:57:58.470 align:middle line:84%
since it uses a different
tokenization scheme,

00:57:58.470 --> 00:58:01.310 align:middle line:84%
each token might
be part of a word.

00:58:01.310 --> 00:58:02.810 align:middle line:84%
It might be an
individual character.

00:58:02.810 --> 00:58:04.530 align:middle line:90%
It might be a punctuation mark.

00:58:04.530 --> 00:58:08.390 align:middle line:84%
In fact, the GPT family
doesn't support punctuation,

00:58:08.390 --> 00:58:10.410 align:middle line:84%
which is why when
you ask a question,

00:58:10.410 --> 00:58:14.120 align:middle line:84%
it comes back with intact
punctuation in its response.

00:58:14.120 --> 00:58:17.520 align:middle line:84%
And so we'll revisit this when
we look at BP byte-pair encoding

00:58:17.520 --> 00:58:19.605 align:middle line:90%
later on.

00:58:19.605 --> 00:58:21.480 align:middle line:84%
But the key thing to
remember is that all the

00:58:21.480 --> 00:58:22.855 align:middle line:84%
stuff you're
talking about starts

00:58:22.855 --> 00:58:24.680 align:middle line:90%
from the notion of a token.

00:58:24.680 --> 00:58:27.460 align:middle line:84%
As to how you define a
token, given a bunch of text,

00:58:27.460 --> 00:58:29.440 align:middle line:90%
that's the tokenizer's job.

00:58:29.440 --> 00:58:34.480 align:middle line:84%
And we just assumed a simple
tokenizer for the time being.

00:58:34.480 --> 00:58:39.175 align:middle line:84%
So at this point, folks, we have
satisfied all the requirements.

00:58:39.175 --> 00:58:41.300 align:middle line:84%
We have taken the surrounding
context of each word,

00:58:41.300 --> 00:58:43.460 align:middle line:84%
we taken the order,
and so on and so forth,

00:58:43.460 --> 00:58:47.020 align:middle line:84%
because what's coming in here
is the positional embeddings.

00:58:47.020 --> 00:58:49.640 align:middle line:84%
And it runs through the
whole transformer stack.

00:58:49.640 --> 00:58:55.920 align:middle line:84%
So this is called a
transformer encoder.

00:58:55.920 --> 00:58:57.597 align:middle line:90%
This is the transformer encoder.

00:58:57.597 --> 00:58:59.680 align:middle line:84%
And you can see here, this
is the original picture

00:58:59.680 --> 00:59:01.040 align:middle line:90%
from the paper.

00:59:01.040 --> 00:59:03.240 align:middle line:84%
It's an iconic
picture at this point.

00:59:03.240 --> 00:59:05.220 align:middle line:84%
So it says here,
these are the inputs.

00:59:05.220 --> 00:59:08.320 align:middle line:84%
This is like "The cat sat on
the Mat " It comes in here,

00:59:08.320 --> 00:59:09.610 align:middle line:90%
gets transferred to--

00:59:09.610 --> 00:59:12.650 align:middle line:84%
transformed into embeddings,
standalone embeddings.

00:59:12.650 --> 00:59:16.250 align:middle line:84%
And then, based on the
position of each word, we add--

00:59:16.250 --> 00:59:18.290 align:middle line:84%
that is why you see
a plus sign here.

00:59:18.290 --> 00:59:22.090 align:middle line:84%
We add the positional
embedding to that,

00:59:22.090 --> 00:59:25.570 align:middle line:84%
and the resulting thing goes
into this transformer block.

00:59:25.570 --> 00:59:30.770 align:middle line:84%
And here, we go through
multi-head attention.

00:59:30.770 --> 00:59:32.803 align:middle line:84%
And things come
out the other end.

00:59:32.803 --> 00:59:34.970 align:middle line:84%
Then, there is this thing
called add and norm, which

00:59:34.970 --> 00:59:37.410 align:middle line:90%
we'll revisit on Wednesday.

00:59:37.410 --> 00:59:40.170 align:middle line:84%
And then it goes through a feed
forward network, another add

00:59:40.170 --> 00:59:42.250 align:middle line:84%
and norm, which we'll
revisit on Wednesday.

00:59:42.250 --> 00:59:43.750 align:middle line:84%
And then it comes
out the other end.

00:59:43.750 --> 00:59:44.230 align:middle line:90%
That's it.

00:59:44.230 --> 00:59:45.522 align:middle line:90%
That's the transformer encoder.

00:59:45.522 --> 00:59:48.162 align:middle line:90%


00:59:48.162 --> 00:59:53.830 align:middle line:84%
And so if you look at this, just
to point out a couple of things,

00:59:53.830 --> 00:59:55.630 align:middle line:84%
the input embeddings
can be random weights,

00:59:55.630 --> 00:59:58.463 align:middle line:84%
or it could be
pretrained embeddings.

00:59:58.463 --> 01:00:00.130 align:middle line:84%
We add in a
position-dependent embedding

01:00:00.130 --> 01:00:01.670 align:middle line:84%
to represent the position of
each word in the sentence.

01:00:01.670 --> 01:00:02.573 align:middle line:90%
That's the plus.

01:00:02.573 --> 01:00:04.490 align:middle line:84%
Then we pass it through
multi-headed attention

01:00:04.490 --> 01:00:07.210 align:middle line:84%
to get a contextual
representation.

01:00:07.210 --> 01:00:10.760 align:middle line:84%
Then, finally, we pass all
this through a simple--

01:00:10.760 --> 01:00:12.317 align:middle line:84%
typically, it's a
two-layer network,

01:00:12.317 --> 01:00:13.900 align:middle line:84%
a one hidden layer
with ReLUs and then

01:00:13.900 --> 01:00:15.900 align:middle line:84%
a linear layer after
that, and boom.

01:00:15.900 --> 01:00:17.240 align:middle line:90%
And then we do it.

01:00:17.240 --> 01:00:18.200 align:middle line:90%
This is the encoder.

01:00:18.200 --> 01:00:23.780 align:middle line:84%
And here is perhaps the most
important point to keep in mind.

01:00:23.780 --> 01:00:25.540 align:middle line:84%
Because we have
taken inordinate care

01:00:25.540 --> 01:00:27.498 align:middle line:84%
to make sure that the
things that are coming in

01:00:27.498 --> 01:00:30.180 align:middle line:84%
and the things that are
going out have the same size,

01:00:30.180 --> 01:00:32.060 align:middle line:84%
both in terms of
the number of tokens

01:00:32.060 --> 01:00:34.180 align:middle line:84%
as well as the length
of each vector,

01:00:34.180 --> 01:00:37.260 align:middle line:84%
we can then stack
them up like pancakes.

01:00:37.260 --> 01:00:39.260 align:middle line:84%
We can have lots of
transformer stacks, one

01:00:39.260 --> 01:00:41.690 align:middle line:90%
on top of each other.

01:00:41.690 --> 01:00:43.660 align:middle line:90%
Because it's the perfect API.

01:00:43.660 --> 01:00:45.380 align:middle line:90%
It's the simplest possible API.

01:00:45.380 --> 01:00:46.838 align:middle line:84%
The same thing
comes in, same thing

01:00:46.838 --> 01:00:49.038 align:middle line:90%
goes out, in terms of size.

01:00:49.038 --> 01:00:51.580 align:middle line:84%
So you can have a transformer
encoder, another one top, boom,

01:00:51.580 --> 01:00:53.960 align:middle line:84%
boom, boom, boom, boom,
one after the other.

01:00:53.960 --> 01:00:56.210 align:middle line:90%
GPT-3 has 96 transformer stacks.

01:00:56.210 --> 01:00:58.740 align:middle line:90%


01:00:58.740 --> 01:01:01.660 align:middle line:84%
And like in all things
deep-learning related, the more

01:01:01.660 --> 01:01:03.940 align:middle line:84%
layers you have, the
more complicated things

01:01:03.940 --> 01:01:05.950 align:middle line:84%
we can do with
it, as long as you

01:01:05.950 --> 01:01:07.670 align:middle line:84%
have enough data to
keep the model happy

01:01:07.670 --> 01:01:08.587 align:middle line:90%
so it doesn't overfit.

01:01:08.587 --> 01:01:11.790 align:middle line:90%


01:01:11.790 --> 01:01:13.430 align:middle line:90%
OK?

01:01:13.430 --> 01:01:14.170 align:middle line:90%
All right.

01:01:14.170 --> 01:01:18.190 align:middle line:84%
So what we haven't covered,
which we'll cover on Wednesday,

01:01:18.190 --> 01:01:22.667 align:middle line:84%
is the question that he
had posed, about how,

01:01:22.667 --> 01:01:25.250 align:middle line:84%
since there are no parameters
inside the self-attention block,

01:01:25.250 --> 01:01:26.513 align:middle line:90%
what are we actually learning?

01:01:26.513 --> 01:01:28.930 align:middle line:84%
And then there is these things
called residual connections

01:01:28.930 --> 01:01:30.132 align:middle line:90%
and layer normalization.

01:01:30.132 --> 01:01:32.090 align:middle line:84%
We'll talk about all
those things on Wednesday.

01:01:32.090 --> 01:01:35.230 align:middle line:84%
Those are all
refinements to the idea.

01:01:35.230 --> 01:01:36.530 align:middle line:90%
So all right.

01:01:36.530 --> 01:01:37.910 align:middle line:90%
9:39.

01:01:37.910 --> 01:01:40.950 align:middle line:84%
Let's apply the transformer
encoder to an actual problem.

01:01:40.950 --> 01:01:43.630 align:middle line:90%
Any questions?

01:01:43.630 --> 01:01:44.975 align:middle line:90%
Yeah.

01:01:44.975 --> 01:01:46.350 align:middle line:84%
AUDIENCE: My
question, regarding,

01:01:46.350 --> 01:01:49.070 align:middle line:84%
like, you said you could
have multiple transformers.

01:01:49.070 --> 01:01:50.510 align:middle line:84%
What is the
difference with having

01:01:50.510 --> 01:01:53.503 align:middle line:84%
multiple self-attention
heads, and rather than

01:01:53.503 --> 01:01:54.670 align:middle line:90%
having multiple [INAUDIBLE]?

01:01:54.670 --> 01:01:56.753 align:middle line:84%
RAMA RAMAKRISHNAN: When I
say a transformer block,

01:01:56.753 --> 01:01:58.810 align:middle line:84%
within the block, there
could be multiple heads.

01:01:58.810 --> 01:02:00.930 align:middle line:84%
AUDIENCE: So if you
increase accuracy,

01:02:00.930 --> 01:02:04.295 align:middle line:84%
as you say, by
[INAUDIBLE] this this way

01:02:04.295 --> 01:02:06.920 align:middle line:84%
RAMA RAMAKRISHNAN: Yeah, you can
have a lot of attention heads.

01:02:06.920 --> 01:02:08.100 align:middle line:90%
And that's totally fine.

01:02:08.100 --> 01:02:10.807 align:middle line:84%
And typically, I forget
how many GPT-3 and 4 have.

01:02:10.807 --> 01:02:12.140 align:middle line:90%
They have a whole bunch of them.

01:02:12.140 --> 01:02:15.360 align:middle line:84%
But you can-- so you can go
wide and you can go deep.

01:02:15.360 --> 01:02:18.240 align:middle line:90%
Both are done in practice.

01:02:18.240 --> 01:02:20.520 align:middle line:84%
But the thing is, the one
thing you have to remember

01:02:20.520 --> 01:02:23.440 align:middle line:84%
is that if you go wide, you
have a lot of attention heads,

01:02:23.440 --> 01:02:27.020 align:middle line:84%
then, given the particular input
that's coming into that block,

01:02:27.020 --> 01:02:29.360 align:middle line:84%
it will learn different
patterns from it.

01:02:29.360 --> 01:02:31.080 align:middle line:84%
Well, if you stack
them all up, it's

01:02:31.080 --> 01:02:33.010 align:middle line:84%
going to learn different
ways to contextualize

01:02:33.010 --> 01:02:34.260 align:middle line:90%
the things that are coming in.

01:02:34.260 --> 01:02:36.093 align:middle line:84%
It operates at higher
levels of abstraction.

01:02:36.093 --> 01:02:38.160 align:middle line:84%
So the analogy would be
that, the seventh layer

01:02:38.160 --> 01:02:41.760 align:middle line:84%
of a convolutional net, take
the sixth layer's output

01:02:41.760 --> 01:02:44.420 align:middle line:84%
and say, oh, I'm seeing
a lot of edges here.

01:02:44.420 --> 01:02:46.920 align:middle line:84%
I'm going to take an edge like
this, two circles like that,

01:02:46.920 --> 01:02:48.480 align:middle line:90%
and call it a face.

01:02:48.480 --> 01:02:50.688 align:middle line:84%
So it will operate at a
high level of abstraction.

01:02:50.688 --> 01:02:58.320 align:middle line:90%


01:02:58.320 --> 01:02:59.020 align:middle line:90%
All right.

01:02:59.020 --> 01:03:01.473 align:middle line:90%
Let's go to the Colab.

01:03:01.473 --> 01:03:02.890 align:middle line:84%
So what we're going
to do is we're

01:03:02.890 --> 01:03:05.265 align:middle line:84%
going to take the transformer
that we just learned about,

01:03:05.265 --> 01:03:08.270 align:middle line:84%
and we're going to apply it to
solve the travel slot problem.

01:03:08.270 --> 01:03:09.730 align:middle line:90%
OK?

01:03:09.730 --> 01:03:10.430 align:middle line:90%
All right.

01:03:10.430 --> 01:03:12.670 align:middle line:90%
So OK.

01:03:12.670 --> 01:03:16.210 align:middle line:84%
So we'll start with the
usual preliminaries.

01:03:16.210 --> 01:03:18.050 align:middle line:84%
And then we have
taken the eighth test

01:03:18.050 --> 01:03:20.210 align:middle line:84%
data set I talked about,
and we have stuck them

01:03:20.210 --> 01:03:23.676 align:middle line:90%
in raw box for easy consumption.

01:03:23.676 --> 01:03:24.440 align:middle line:90%
It's here.

01:03:24.440 --> 01:03:30.636 align:middle line:90%


01:03:30.636 --> 01:03:34.310 align:middle line:84%
So if you look at to the
top view, you can see here,

01:03:34.310 --> 01:03:37.330 align:middle line:84%
for example, I want to
fly from Boston, 8:38 AM.

01:03:37.330 --> 01:03:38.560 align:middle line:90%
And then this is the output.

01:03:38.560 --> 01:03:41.090 align:middle line:84%
And the slot filling
is the output.

01:03:41.090 --> 01:03:44.430 align:middle line:84%
And so, as it turns
out, here there is--

01:03:44.430 --> 01:03:46.048 align:middle line:90%
these people also gave it--

01:03:46.048 --> 01:03:48.590 align:middle line:84%
they took the whole query and
gave it an intent as to is it--

01:03:48.590 --> 01:03:49.587 align:middle line:90%
it's a flight query.

01:03:49.587 --> 01:03:51.170 align:middle line:84%
It's a something
else query and so on,

01:03:51.170 --> 01:03:52.330 align:middle line:90%
which we're not going to--

01:03:52.330 --> 01:03:54.490 align:middle line:90%
are you kidding me?

01:03:54.490 --> 01:03:57.250 align:middle line:84%
I want to fly from Boston at
8:30 AM and arrive in Denver

01:03:57.250 --> 01:03:58.290 align:middle line:90%
at 11:00 in the morning.

01:03:58.290 --> 01:03:59.850 align:middle line:84%
What kind of ground
transportation

01:03:59.850 --> 01:04:01.300 align:middle line:90%
is available in Denver?

01:04:01.300 --> 01:04:04.340 align:middle line:90%
What's the airport at Orlando?

01:04:04.340 --> 01:04:08.180 align:middle line:84%
How much does the limo service
cost within Pittsburgh?

01:04:08.180 --> 01:04:09.360 align:middle line:90%
And so on and so forth.

01:04:09.360 --> 01:04:10.320 align:middle line:90%
So you get the idea.

01:04:10.320 --> 01:04:12.100 align:middle line:84%
It's a very wide
range of queries

01:04:12.100 --> 01:04:14.140 align:middle line:90%
that are in this data set.

01:04:14.140 --> 01:04:17.820 align:middle line:84%
So let's just ignore
that for a sec.

01:04:17.820 --> 01:04:19.740 align:middle line:84%
So what we're now
going to do is we're

01:04:19.740 --> 01:04:24.980 align:middle line:84%
going to take only this column,
right, the query column.

01:04:24.980 --> 01:04:27.820 align:middle line:84%
That's going to
be our input text.

01:04:27.820 --> 01:04:29.620 align:middle line:84%
And then the
slot-filling column is

01:04:29.620 --> 01:04:32.620 align:middle line:84%
going to be our dependent
variable, the output.

01:04:32.620 --> 01:04:37.460 align:middle line:84%
So we'll just gather
them all up here.

01:04:37.460 --> 01:04:38.040 align:middle line:90%
Let it run.

01:04:38.040 --> 01:04:40.580 align:middle line:84%
We'll do it for the training
data and the test data.

01:04:40.580 --> 01:04:42.460 align:middle line:84%
And so what we have
done is that we

01:04:42.460 --> 01:04:45.700 align:middle line:84%
have taken the
transformer-related code

01:04:45.700 --> 01:04:48.580 align:middle line:84%
in Keras, and we have
packaged it into a little HODL

01:04:48.580 --> 01:04:50.980 align:middle line:90%
library for easy consumption.

01:04:50.980 --> 01:04:53.160 align:middle line:90%
And so that thing is here.

01:04:53.160 --> 01:04:55.260 align:middle line:90%
You can download it.

01:04:55.260 --> 01:04:56.920 align:middle line:84%
Calling it a library
is overstating it.

01:04:56.920 --> 01:04:58.628 align:middle line:84%
We literally just
created a bunch of code

01:04:58.628 --> 01:05:00.750 align:middle line:90%
and stuck it in a file.

01:05:00.750 --> 01:05:02.630 align:middle line:84%
And so what we'll
do is from HODL,

01:05:02.630 --> 01:05:05.030 align:middle line:84%
we'll import the
transformer encoder.

01:05:05.030 --> 01:05:07.693 align:middle line:84%
And we will import this
positional embedding layer,

01:05:07.693 --> 01:05:09.110 align:middle line:84%
because what we
are going to do is

01:05:09.110 --> 01:05:11.910 align:middle line:84%
we are going to take the input,
do the positional encoding

01:05:11.910 --> 01:05:14.270 align:middle line:84%
business, and then send
it into the transformer.

01:05:14.270 --> 01:05:16.656 align:middle line:90%
OK?

01:05:16.656 --> 01:05:19.430 align:middle line:84%
But first, let's
vectorize the input

01:05:19.430 --> 01:05:21.230 align:middle line:90%
queries that are coming in.

01:05:21.230 --> 01:05:25.310 align:middle line:90%
So we'll define a thing here.

01:05:25.310 --> 01:05:28.410 align:middle line:84%
This is a-- max query
length is not defined.

01:05:28.410 --> 01:05:32.110 align:middle line:84%
That's what happens when
you don't run everything.

01:05:32.110 --> 01:05:32.730 align:middle line:90%
All right.

01:05:32.730 --> 01:05:38.670 align:middle line:90%


01:05:38.670 --> 01:05:41.050 align:middle line:90%
So now we have this thing here.

01:05:41.050 --> 01:05:45.010 align:middle line:84%
So turns out that
there are 888 tokens.

01:05:45.010 --> 01:05:49.150 align:middle line:84%
888 words in the input queries
that we have in the data.

01:05:49.150 --> 01:05:52.350 align:middle line:90%
So take a look at the first few.

01:05:52.350 --> 01:05:55.190 align:middle line:84%
And you can see
here, there is UNK.

01:05:55.190 --> 01:05:57.040 align:middle line:84%
And because the
output mode here is--

01:05:57.040 --> 01:05:58.805 align:middle line:84%
you just want integers
to come out, not

01:05:58.805 --> 01:06:00.680 align:middle line:84%
multi-hot encoding or
anything, because we're

01:06:00.680 --> 01:06:04.720 align:middle line:84%
going to take those integers and
then do embeddings from them.

01:06:04.720 --> 01:06:08.760 align:middle line:84%
It will reserve this empty
string as the pad token.

01:06:08.760 --> 01:06:11.120 align:middle line:84%
This should be familiar
from last week.

01:06:11.120 --> 01:06:12.785 align:middle line:84%
And then the UNK,
for unknown tokens,

01:06:12.785 --> 01:06:15.160 align:middle line:84%
and then 'to' 'from' 'flights'
These are some of the most

01:06:15.160 --> 01:06:16.662 align:middle line:90%
frequent.

01:06:16.662 --> 01:06:18.620 align:middle line:84%
Turns out Boston is
actually the most frequent.

01:06:18.620 --> 01:06:21.160 align:middle line:84%
I don't know what's up with
that, but it is what it is.

01:06:21.160 --> 01:06:23.200 align:middle line:84%
Then we'll do the
same vectorization

01:06:23.200 --> 01:06:25.360 align:middle line:90%
to the train and test data sets.

01:06:25.360 --> 01:06:29.000 align:middle line:84%
Now we need to do STIE for the
output side of the problem,

01:06:29.000 --> 01:06:32.280 align:middle line:84%
because the slots, the dependent
variable here, remember,

01:06:32.280 --> 01:06:35.040 align:middle line:84%
are all sentences
as well, with the B,

01:06:35.040 --> 01:06:36.320 align:middle line:90%
O, things like that, right?

01:06:36.320 --> 01:06:40.980 align:middle line:84%
So we need to vectorize those,
or we need to do STIE on them.

01:06:40.980 --> 01:06:43.240 align:middle line:84%
So let's take a look
at some of these slots.

01:06:43.240 --> 01:06:45.880 align:middle line:84%
And you can see here, all
this stuff is going on.

01:06:45.880 --> 01:06:47.897 align:middle line:84%
Note-- so now,
here is an example

01:06:47.897 --> 01:06:49.480 align:middle line:84%
where you have to
be very careful when

01:06:49.480 --> 01:06:51.400 align:middle line:90%
you do the standardization.

01:06:51.400 --> 01:06:54.480 align:middle line:84%
Typically, standardization, you
will remove punctuation, and do

01:06:54.480 --> 01:06:55.750 align:middle line:90%
things like that in lowercase.

01:06:55.750 --> 01:07:00.110 align:middle line:84%
But here, these things
have a specific meaning.

01:07:00.110 --> 01:07:01.610 align:middle line:84%
We can't just go
in there and remove

01:07:01.610 --> 01:07:03.478 align:middle line:84%
the period and the
underscore and then make

01:07:03.478 --> 01:07:05.270 align:middle line:84%
the B into lowercase
B and stuff like that.

01:07:05.270 --> 01:07:06.710 align:middle line:90%
That will just harm it, right?

01:07:06.710 --> 01:07:09.890 align:middle line:84%
We need to be able to preserve
the nomenclature of the output

01:07:09.890 --> 01:07:11.570 align:middle line:90%
in terms of all those tags.

01:07:11.570 --> 01:07:13.637 align:middle line:84%
So we don't want
the standardization

01:07:13.637 --> 01:07:14.470 align:middle line:90%
to do all those out.

01:07:14.470 --> 01:07:16.970 align:middle line:84%
So what we do is we say
standardization, none.

01:07:16.970 --> 01:07:18.370 align:middle line:90%
Look at that.

01:07:18.370 --> 01:07:20.190 align:middle line:84%
We tell Keras, do
not standardize this.

01:07:20.190 --> 01:07:21.330 align:middle line:90%
Do not do your usual thing.

01:07:21.330 --> 01:07:24.010 align:middle line:90%


01:07:24.010 --> 01:07:28.450 align:middle line:84%
So we do that for
the output side.

01:07:28.450 --> 01:07:31.010 align:middle line:84%
And then let's look
at the vocabulary.

01:07:31.010 --> 01:07:33.370 align:middle line:84%
Yeah, so this
sounds pretty good.

01:07:33.370 --> 01:07:35.850 align:middle line:84%
These are all the things
that we would expect to see.

01:07:35.850 --> 01:07:40.290 align:middle line:84%
These are the distinct
tokens in the output strings.

01:07:40.290 --> 01:07:43.210 align:middle line:90%
All right.

01:07:43.210 --> 01:07:43.710 align:middle line:90%
OK.

01:07:43.710 --> 01:07:45.890 align:middle line:90%
We get it.

01:07:45.890 --> 01:07:47.890 align:middle line:90%
So we have 125 of them.

01:07:47.890 --> 01:07:50.450 align:middle line:84%
In the lecture, I said
there are 123 slots.

01:07:50.450 --> 01:07:51.230 align:middle line:90%
Possible slots.

01:07:51.230 --> 01:07:53.167 align:middle line:90%
Why is it 125 here?

01:07:53.167 --> 01:07:54.300 align:middle line:90%
AUDIENCE: [INAUDIBLE]

01:07:54.300 --> 01:07:55.758 align:middle line:84%
RAMA RAMAKRISHNAN:
Yes, uncompared.

01:07:55.758 --> 01:07:57.620 align:middle line:90%
Correct.

01:07:57.620 --> 01:08:01.685 align:middle line:84%
OK, now we'll set up
a transformer encoder.

01:08:01.685 --> 01:08:02.560 align:middle line:90%
Oh, wait, wait, wait.

01:08:02.560 --> 01:08:04.980 align:middle line:90%
I forgot about doing this.

01:08:04.980 --> 01:08:07.300 align:middle line:90%
My bad.

01:08:07.300 --> 01:08:07.840 align:middle line:90%
All right.

01:08:07.840 --> 01:08:11.157 align:middle line:90%


01:08:11.157 --> 01:08:12.740 align:middle line:84%
I just thought, when
I saw this slide,

01:08:12.740 --> 01:08:16.300 align:middle line:84%
that we should go to the
Colab without giving you

01:08:16.300 --> 01:08:17.319 align:middle line:90%
a bit more background.

01:08:17.319 --> 01:08:18.939 align:middle line:90%
No problem.

01:08:18.939 --> 01:08:21.132 align:middle line:84%
So the way we are going
to model this problem

01:08:21.132 --> 01:08:23.340 align:middle line:84%
is that we're going to have
something like this-- fly

01:08:23.340 --> 01:08:24.823 align:middle line:90%
from Boston to Denver.

01:08:24.823 --> 01:08:26.240 align:middle line:84%
That's the input
that's coming in.

01:08:26.240 --> 01:08:28.660 align:middle line:90%
And that is the correct answer.

01:08:28.660 --> 01:08:30.899 align:middle line:84%
0, 0, some B, something
or the other--

01:08:30.899 --> 01:08:32.359 align:middle line:84%
I mean, O, and then
something else.

01:08:32.359 --> 01:08:33.859 align:middle line:90%
That's the correct answer.

01:08:33.859 --> 01:08:36.420 align:middle line:84%
That's the input, and
that is the right answer.

01:08:36.420 --> 01:08:39.939 align:middle line:84%
So what we'll do is we will
create these positional input

01:08:39.939 --> 01:08:42.620 align:middle line:84%
embeddings like we
have discussed before.

01:08:42.620 --> 01:08:45.340 align:middle line:84%
We will run it
through a transformer.

01:08:45.340 --> 01:08:47.660 align:middle line:84%
It gives us
contextual embeddings.

01:08:47.660 --> 01:08:50.040 align:middle line:84%
So if we send five in, it's
going to send us five out,

01:08:50.040 --> 01:08:52.029 align:middle line:90%
except the color is now blue.

01:08:52.029 --> 01:08:57.290 align:middle line:84%
And then what we do is we
will run it through a ReLU.

01:08:57.290 --> 01:08:57.790 align:middle line:90%
OK?

01:08:57.790 --> 01:08:59.390 align:middle line:90%
We'll run it through a ReLU.

01:08:59.390 --> 01:09:02.510 align:middle line:84%
We will still have
five vectors here.

01:09:02.510 --> 01:09:04.029 align:middle line:90%
Five vectors will come in.

01:09:04.029 --> 01:09:07.069 align:middle line:84%
And then, for each of the things
that comes in, we will stick

01:09:07.069 --> 01:09:08.539 align:middle line:90%
a 123-way softmax.

01:09:08.539 --> 01:09:12.109 align:middle line:90%


01:09:12.109 --> 01:09:15.111 align:middle line:84%
For each thing that comes out,
we'll have a 123-way softmax.

01:09:15.111 --> 01:09:17.569 align:middle line:84%
And that's the classification
problem we're going to solve.

01:09:17.569 --> 01:09:21.308 align:middle line:90%


01:09:21.308 --> 01:09:25.470 align:middle line:84%
So the weights in
all these layers

01:09:25.470 --> 01:09:28.310 align:middle line:90%
will get optimized by backprop.

01:09:28.310 --> 01:09:30.990 align:middle line:84%
All these weights are
going to get optimized.

01:09:30.990 --> 01:09:31.670 align:middle line:90%
Yeah.

01:09:31.670 --> 01:09:34.109 align:middle line:90%
AUDIENCE: [INAUDIBLE]

01:09:34.109 --> 01:09:36.298 align:middle line:90%
RAMA RAMAKRISHNAN: Sorry?

01:09:36.298 --> 01:09:38.590 align:middle line:84%
AUDIENCE: What about the
positional embeddings imported

01:09:38.590 --> 01:09:41.149 align:middle line:90%
in the Colab [INAUDIBLE]?

01:09:41.149 --> 01:09:43.189 align:middle line:84%
RAMA RAMAKRISHNAN:
No, that's the layer.

01:09:43.189 --> 01:09:46.870 align:middle line:84%
The weights in the layer will
still need to be learned.

01:09:46.870 --> 01:09:48.569 align:middle line:84%
It's like the text
vectorization layer.

01:09:48.569 --> 01:09:49.779 align:middle line:90%
It's a bunch of code.

01:09:49.779 --> 01:09:52.000 align:middle line:84%
And then you actually run
it on a particular corpus

01:09:52.000 --> 01:09:54.480 align:middle line:84%
to adapt it and fill a
vocabulary out of it.

01:09:54.480 --> 01:09:57.647 align:middle line:84%
So it's like an empty shell
that needs to get populated.

01:09:57.647 --> 01:09:59.480 align:middle line:84%
OK, so with the weights
in, all these things

01:09:59.480 --> 01:10:04.080 align:middle line:84%
are going to get updated when
we train the model by backprop.

01:10:04.080 --> 01:10:04.820 align:middle line:90%
And that's it.

01:10:04.820 --> 01:10:06.360 align:middle line:90%
That's the setup.

01:10:06.360 --> 01:10:09.600 align:middle line:84%
Does this make sense, before
I switch back to the Colab?

01:10:09.600 --> 01:10:12.580 align:middle line:84%
In particular, does this
make sense, this part of it?

01:10:12.580 --> 01:10:15.920 align:middle line:90%


01:10:15.920 --> 01:10:16.980 align:middle line:90%
Bunch of things come out.

01:10:16.980 --> 01:10:18.120 align:middle line:84%
And then, for each
one of those things,

01:10:18.120 --> 01:10:19.745 align:middle line:84%
we need to figure
out a classification,

01:10:19.745 --> 01:10:21.320 align:middle line:90%
a 123-way classification.

01:10:21.320 --> 01:10:23.320 align:middle line:84%
And that's where we stick
a softmax in every one

01:10:23.320 --> 01:10:25.400 align:middle line:90%
of those output nodes.

01:10:25.400 --> 01:10:26.297 align:middle line:90%
Yeah.

01:10:26.297 --> 01:10:28.380 align:middle line:84%
AUDIENCE: [INAUDIBLE] for
the yellow vector layer,

01:10:28.380 --> 01:10:32.305 align:middle line:84%
is there a reason that
they're, like, 3-dimensional?

01:10:32.305 --> 01:10:33.680 align:middle line:84%
RAMA RAMAKRISHNAN:
Oh, oh, I see.

01:10:33.680 --> 01:10:34.513 align:middle line:90%
AUDIENCE: Not three.

01:10:34.513 --> 01:10:36.583 align:middle line:84%
But why it's changed
from three to four?

01:10:36.583 --> 01:10:38.000 align:middle line:84%
RAMA RAMAKRISHNAN:
That's a good--

01:10:38.000 --> 01:10:40.200 align:middle line:84%
AUDIENCE: Or is it just kind
of random, like [INAUDIBLE].?

01:10:40.200 --> 01:10:41.180 align:middle line:84%
RAMA RAMAKRISHNAN:
It could be whatever.

01:10:41.180 --> 01:10:43.760 align:middle line:84%
Or to put it another way, it
is your choice as the user,

01:10:43.760 --> 01:10:44.800 align:middle line:90%
as the modeler.

01:10:44.800 --> 01:10:45.360 align:middle line:90%
Correct.

01:10:45.360 --> 01:10:47.990 align:middle line:84%
The thing is, at this
point, with the blue stuff,

01:10:47.990 --> 01:10:51.370 align:middle line:84%
the transformers basically
say, my job is done.

01:10:51.370 --> 01:10:53.970 align:middle line:84%
It has given you these
valuable contextual embeddings

01:10:53.970 --> 01:10:55.738 align:middle line:84%
at some high level
of abstraction.

01:10:55.738 --> 01:10:58.030 align:middle line:84%
What you do with it depends
on your particular problem.

01:10:58.030 --> 01:11:00.750 align:middle line:84%
And so the best practice
would be to take it.

01:11:00.750 --> 01:11:03.530 align:middle line:84%
And then maybe if these
embeddings are really long,

01:11:03.530 --> 01:11:06.665 align:middle line:84%
maybe make them a little
smaller using a ReLU.

01:11:06.665 --> 01:11:08.290 align:middle line:84%
And using a ReLU is
always a good idea,

01:11:08.290 --> 01:11:11.650 align:middle line:84%
because when in doubt, throw
in a bit of non-linearity.

01:11:11.650 --> 01:11:14.690 align:middle line:84%
And then, once you are done
with that, well, at this point

01:11:14.690 --> 01:11:16.670 align:middle line:84%
you need to actually
classify it.

01:11:16.670 --> 01:11:20.530 align:middle line:84%
So you stick an
output softmax on it.

01:11:20.530 --> 01:11:22.390 align:middle line:90%
So that's what we have.

01:11:22.390 --> 01:11:27.650 align:middle line:90%


01:11:27.650 --> 01:11:29.690 align:middle line:90%
All right, back to this picture.

01:11:29.690 --> 01:11:32.570 align:middle line:84%
So what we're going
to do is we also

01:11:32.570 --> 01:11:35.190 align:middle line:84%
get to decide how long are
these embedding vectors.

01:11:35.190 --> 01:11:35.690 align:middle line:90%
How long.

01:11:35.690 --> 01:11:37.940 align:middle line:84%
Because here, we're not going
to use glove embeddings.

01:11:37.940 --> 01:11:40.135 align:middle line:84%
We're just going to learn
everything from scratch.

01:11:40.135 --> 01:11:42.010 align:middle line:84%
We're going to learn
everything from scratch.

01:11:42.010 --> 01:11:44.490 align:middle line:84%
And we can decide how long
these embedding vectors are.

01:11:44.490 --> 01:11:47.700 align:middle line:84%
So these embedding vectors,
I'm going to decide.

01:11:47.700 --> 01:11:51.100 align:middle line:84%
I have decided that I
want them to be 512 long.

01:11:51.100 --> 01:11:53.953 align:middle line:84%
I want these actually
to be 512 long.

01:11:53.953 --> 01:11:55.120 align:middle line:90%
So that's what I have here--

01:11:55.120 --> 01:11:56.980 align:middle line:90%
512.

01:11:56.980 --> 01:11:58.960 align:middle line:84%
And then, inside
the transformer,

01:11:58.960 --> 01:12:02.078 align:middle line:84%
remember, when we
concatenate everything,

01:12:02.078 --> 01:12:03.620 align:middle line:84%
and then we have
something, we run it

01:12:03.620 --> 01:12:05.900 align:middle line:90%
through a final ReLU layer.

01:12:05.900 --> 01:12:08.980 align:middle line:90%
How big should that layer be?

01:12:08.980 --> 01:12:11.640 align:middle line:84%
That is what, here, what
I mean by dense_dim.

01:12:11.640 --> 01:12:13.320 align:middle line:90%
I want it to be 64.

01:12:13.320 --> 01:12:17.540 align:middle line:84%
And then I, for fun, I'm going
to use five attention heads,

01:12:17.540 --> 01:12:20.460 align:middle line:90%
because why not?

01:12:20.460 --> 01:12:26.180 align:middle line:84%
And then, in the final thing
here, to go to Alina's question

01:12:26.180 --> 01:12:29.980 align:middle line:84%
here, these things are all 512
long, as I mentioned earlier.

01:12:29.980 --> 01:12:32.060 align:middle line:90%
These are all 512.

01:12:32.060 --> 01:12:36.820 align:middle line:84%
But this thing here, I'm
going to make it just 128.

01:12:36.820 --> 01:12:38.780 align:middle line:84%
That's what I mean
by units here.

01:12:38.780 --> 01:12:41.260 align:middle line:84%
And so if you look
at the actual model,

01:12:41.260 --> 01:12:44.410 align:middle line:84%
OK, whatever comes in has a max
query length of, I think, 30,

01:12:44.410 --> 01:12:46.025 align:middle line:90%
if I recall.

01:12:46.025 --> 01:12:47.650 align:middle line:84%
Actually, let's just
make sure of that.

01:12:47.650 --> 01:12:48.400 align:middle line:90%
What did I assume?

01:12:48.400 --> 01:12:51.330 align:middle line:90%


01:12:51.330 --> 01:12:51.830 align:middle line:90%
30.

01:12:51.830 --> 01:12:52.210 align:middle line:90%
Correct.

01:12:52.210 --> 01:12:53.090 align:middle line:90%
Max query length 30.

01:12:53.090 --> 01:12:55.090 align:middle line:90%
So each sentence is 30.

01:12:55.090 --> 01:12:58.475 align:middle line:84%
So if a sentence has 35 words
in it, what's going to happen?

01:12:58.475 --> 01:12:59.753 align:middle line:90%
AUDIENCE: [INAUDIBLE]

01:12:59.753 --> 01:13:02.170 align:middle line:84%
RAMA RAMAKRISHNAN: The last 5
will get chopped, truncated.

01:13:02.170 --> 01:13:04.030 align:middle line:84%
If it comes in at
22, we're going

01:13:04.030 --> 01:13:06.530 align:middle line:84%
to pad it with eight more
tokens, with a pad token.

01:13:06.530 --> 01:13:09.550 align:middle line:84%
That's how we make sure
everything gets to 30.

01:13:09.550 --> 01:13:10.170 align:middle line:90%
All right.

01:13:10.170 --> 01:13:12.230 align:middle line:90%
So we come back here.

01:13:12.230 --> 01:13:15.150 align:middle line:84%
So the input is still
sentences which are 30 long,

01:13:15.150 --> 01:13:16.750 align:middle line:90%
tokens which are 30 long.

01:13:16.750 --> 01:13:20.550 align:middle line:84%
And then we run it through a
positional embedding layer.

01:13:20.550 --> 01:13:21.190 align:middle line:90%
OK?

01:13:21.190 --> 01:13:24.870 align:middle line:84%
This positional embedding
layer has the actual embedding

01:13:24.870 --> 01:13:28.750 align:middle line:84%
for each word in that table,
and it has the positional table,

01:13:28.750 --> 01:13:30.070 align:middle line:90%
the positional embedding table.

01:13:30.070 --> 01:13:32.870 align:middle line:84%
So just to be clear, this
positional embedding layer

01:13:32.870 --> 01:13:34.150 align:middle line:90%
is basically--

01:13:34.150 --> 01:13:37.150 align:middle line:90%
it's basically this.

01:13:37.150 --> 01:13:40.190 align:middle line:84%
So this table and
this table together

01:13:40.190 --> 01:13:43.720 align:middle line:84%
are packaged up into the
positional encoding layer.

01:13:43.720 --> 01:13:45.300 align:middle line:84%
But they are totally
distinct tables.

01:13:45.300 --> 01:13:47.520 align:middle line:84%
They just happen
to be packaged up.

01:13:47.520 --> 01:13:51.120 align:middle line:90%
So this is what we have here.

01:13:51.120 --> 01:13:53.940 align:middle line:84%
And then we get a nice
positional embedding out.

01:13:53.940 --> 01:13:56.240 align:middle line:84%
And then, boom, we run it
through the transformer.

01:13:56.240 --> 01:14:00.120 align:middle line:84%
And this transformer encoder
object, we have to tell it,

01:14:00.120 --> 01:14:02.280 align:middle line:84%
obviously, hey, this is
the embedding dimension

01:14:02.280 --> 01:14:03.343 align:middle line:90%
that's going to come out.

01:14:03.343 --> 01:14:04.760 align:middle line:84%
This is the dense
dimension you're

01:14:04.760 --> 01:14:07.240 align:middle line:84%
going to use in that
final feed forward layer

01:14:07.240 --> 01:14:08.492 align:middle line:90%
inside each attention block.

01:14:08.492 --> 01:14:10.200 align:middle line:84%
And this is the number
of attention heads

01:14:10.200 --> 01:14:10.950 align:middle line:90%
I want you to use.

01:14:10.950 --> 01:14:11.600 align:middle line:90%
That's it.

01:14:11.600 --> 01:14:12.900 align:middle line:90%
It's very-- right?

01:14:12.900 --> 01:14:14.840 align:middle line:84%
Only three things
have to be specified.

01:14:14.840 --> 01:14:17.560 align:middle line:84%
And then, whatever comes out
of the transformer encoder

01:14:17.560 --> 01:14:19.160 align:middle line:90%
are these blue vectors.

01:14:19.160 --> 01:14:23.000 align:middle line:84%
And then we are back into good
old traditional DNN stuff, where

01:14:23.000 --> 01:14:27.380 align:middle line:84%
we take this thing, run it
through a ReLU with 128 units.

01:14:27.380 --> 01:14:29.360 align:middle line:90%
We add a dropout.

01:14:29.360 --> 01:14:31.920 align:middle line:84%
And then we run it
through a dense layer,

01:14:31.920 --> 01:14:34.580 align:middle line:84%
which the vocab
size here is 125,

01:14:34.580 --> 01:14:38.080 align:middle line:90%
which is the 125-way softmax.

01:14:38.080 --> 01:14:39.887 align:middle line:90%
Activation softmax.

01:14:39.887 --> 01:14:41.970 align:middle line:84%
Connect up everything into
model input and output.

01:14:41.970 --> 01:14:44.530 align:middle line:84%
And boom, that's
the whole model.

01:14:44.530 --> 01:14:48.645 align:middle line:90%
So that's what we have here.

01:14:48.645 --> 01:14:56.010 align:middle line:84%
Now, this, after Wednesday's
class, for extra credit

01:14:56.010 --> 01:14:59.330 align:middle line:84%
and for your
personal edification,

01:14:59.330 --> 01:15:02.970 align:middle line:84%
try to work through this thing
to come up with this number--

01:15:02.970 --> 01:15:07.010 align:middle line:90%
53 million-- sorry, 5.3 million.

01:15:07.010 --> 01:15:12.570 align:middle line:84%
And see if it matches
this number here.

01:15:12.570 --> 01:15:13.945 align:middle line:90%
It should match.

01:15:13.945 --> 01:15:15.570 align:middle line:84%
Hand calculate the
number of parameters

01:15:15.570 --> 01:15:17.490 align:middle line:90%
inside the transformer.

01:15:17.490 --> 01:15:18.550 align:middle line:90%
For fame and fortune.

01:15:18.550 --> 01:15:19.770 align:middle line:90%
That's an optional thing.

01:15:19.770 --> 01:15:23.382 align:middle line:84%
So do it after Wednesday's
class, not right now.

01:15:23.382 --> 01:15:25.090 align:middle line:84%
And I have actually
listed the exact math

01:15:25.090 --> 01:15:26.730 align:middle line:90%
that goes into it here.

01:15:26.730 --> 01:15:27.310 align:middle line:90%
All right.

01:15:27.310 --> 01:15:29.625 align:middle line:84%
So, by the way, you can peek
into any layer's weights

01:15:29.625 --> 01:15:30.750 align:middle line:90%
using its weight attribute.

01:15:30.750 --> 01:15:33.130 align:middle line:84%
This is the embedding,
the positional embedding

01:15:33.130 --> 01:15:33.830 align:middle line:90%
thing we had.

01:15:33.830 --> 01:15:35.510 align:middle line:90%
So we can click it.

01:15:35.510 --> 01:15:38.220 align:middle line:84%
And you can see here,
it has two tables.

01:15:38.220 --> 01:15:40.880 align:middle line:84%
There is the first table, which
is just the embedding table,

01:15:40.880 --> 01:15:44.680 align:middle line:84%
which says there are 888
tokens in my vocabulary,

01:15:44.680 --> 01:15:46.680 align:middle line:84%
and each of those tokens
is an embedding vector,

01:15:46.680 --> 01:15:47.860 align:middle line:90%
which is 512 long.

01:15:47.860 --> 01:15:49.580 align:middle line:90%
That is the first table here.

01:15:49.580 --> 01:15:51.300 align:middle line:84%
And then it has the
second object, which

01:15:51.300 --> 01:15:53.140 align:middle line:90%
is the positional embedding.

01:15:53.140 --> 01:15:56.400 align:middle line:84%
And it says here, well, my
sentences can be 30 long.

01:15:56.400 --> 01:15:59.300 align:middle line:84%
And for each position
of the 30-long sentence,

01:15:59.300 --> 01:16:02.180 align:middle line:90%
I will have a 512 embedding.

01:16:02.180 --> 01:16:04.200 align:middle line:84%
Both these tables, as
I mentioned earlier,

01:16:04.200 --> 01:16:05.180 align:middle line:90%
are packaged up inside.

01:16:05.180 --> 01:16:06.888 align:middle line:84%
And you can actually
see what the weights

01:16:06.888 --> 01:16:09.590 align:middle line:90%
are before you do any training.

01:16:09.590 --> 01:16:10.580 align:middle line:90%
So all right.

01:16:10.580 --> 01:16:12.653 align:middle line:84%
So I'm going to stop
here, because the model,

01:16:12.653 --> 01:16:14.320 align:middle line:84%
it's going to take a
few minutes to run.

01:16:14.320 --> 01:16:16.260 align:middle line:90%
And we are already at 9:45.

01:16:16.260 --> 01:16:18.228 align:middle line:84%
So we will continue the
journey on Wednesday.

01:16:18.228 --> 01:16:20.520 align:middle line:84%
If some of it is not super
clear, don't worry about it.

01:16:20.520 --> 01:16:21.920 align:middle line:84%
It'll become much
clearer on Wednesday.

01:16:21.920 --> 01:16:22.320 align:middle line:90%
All right.

01:16:22.320 --> 01:16:24.195 align:middle line:84%
All right, folks, have
a good couple of days.

01:16:24.195 --> 01:16:25.590 align:middle line:90%
I'll see you on Wednesday.

01:16:25.590 --> 01:16:37.000 align:middle line:90%