WEBVTT

00:00:00.000 --> 00:00:04.464 align:middle line:84%
[SQUEAKING]
[RUSTLING] [CLICKING]

00:00:04.464 --> 00:00:20.405 align:middle line:90%


00:00:20.405 --> 00:00:22.280 align:middle line:84%
RAMA RAMAKRISHNAN: We'll
continue our journey

00:00:22.280 --> 00:00:25.080 align:middle line:84%
with natural
language processing.

00:00:25.080 --> 00:00:27.620 align:middle line:84%
We looked at the bag of words
model, one-hot embeddings,

00:00:27.620 --> 00:00:28.760 align:middle line:90%
and so on and so forth.

00:00:28.760 --> 00:00:31.240 align:middle line:84%
And today, we will
talk about embeddings,

00:00:31.240 --> 00:00:34.170 align:middle line:84%
or to be more precise,
standalone embeddings.

00:00:34.170 --> 00:00:35.920 align:middle line:84%
And then that will tee
us up for something

00:00:35.920 --> 00:00:38.000 align:middle line:84%
called contextual
embeddings, which

00:00:38.000 --> 00:00:41.680 align:middle line:84%
is where the transformer
really comes into play.

00:00:41.680 --> 00:00:42.180 align:middle line:90%
All right.

00:00:42.180 --> 00:00:43.240 align:middle line:90%
So let's get going.

00:00:43.240 --> 00:00:49.060 align:middle line:84%
So FR, we have encoded input
text with one-hot vector.

00:00:49.060 --> 00:00:52.480 align:middle line:84%
So to just to refresh
your memories from Monday,

00:00:52.480 --> 00:00:55.260 align:middle line:84%
if this is the phrase that's
coming into the system,

00:00:55.260 --> 00:00:57.960 align:middle line:84%
we run it through
the STIE process.

00:00:57.960 --> 00:00:59.950 align:middle line:84%
And when we do
that, what happens

00:00:59.950 --> 00:01:03.330 align:middle line:84%
is that we first of
all, we standardize,

00:01:03.330 --> 00:01:06.750 align:middle line:84%
then we split on whitespace
to get individual words.

00:01:06.750 --> 00:01:08.950 align:middle line:84%
Then we assign
words to integers,

00:01:08.950 --> 00:01:11.550 align:middle line:84%
and then we take each integer
and essentially create

00:01:11.550 --> 00:01:13.990 align:middle line:84%
a one-hot version
of that integer.

00:01:13.990 --> 00:01:19.210 align:middle line:84%
And when we do that, basically,
we have a vocabulary, right?

00:01:19.210 --> 00:01:21.410 align:middle line:84%
And in this example,
we just have 100 words.

00:01:21.410 --> 00:01:25.230 align:middle line:84%
And you will note that this
vocabulary which you arrive

00:01:25.230 --> 00:01:28.282 align:middle line:84%
at once you standardize
and tokenize

00:01:28.282 --> 00:01:30.390 align:middle line:84%
has words like the
because we decided

00:01:30.390 --> 00:01:34.790 align:middle line:84%
not to remove stop words
like a, and, the, and so on.

00:01:34.790 --> 00:01:38.150 align:middle line:84%
So just to be clear,
standardization--

00:01:38.150 --> 00:01:42.150 align:middle line:84%
here, standardization-- while it
has historically been all about

00:01:42.150 --> 00:01:45.050 align:middle line:84%
stripping punctuation,
lowercasing everything,

00:01:45.050 --> 00:01:47.630 align:middle line:84%
removing stopwords,
and stemming--

00:01:47.630 --> 00:01:49.330 align:middle line:84%
while that has been
true historically,

00:01:49.330 --> 00:01:52.710 align:middle line:84%
if you look at modern
practice, people essentially

00:01:52.710 --> 00:01:55.350 align:middle line:84%
strip punctuation, maybe,
and then lowercase,

00:01:55.350 --> 00:01:58.580 align:middle line:84%
and they often don't even
bother to do stemming and things

00:01:58.580 --> 00:02:00.986 align:middle line:84%
like that, or to
remove stop words.

00:02:00.986 --> 00:02:05.140 align:middle line:84%
And that's why in Keras, the
default standardization is only

00:02:05.140 --> 00:02:05.780 align:middle line:90%
lowercasing.

00:02:05.780 --> 00:02:09.220 align:middle line:90%
And punctuation stripping.

00:02:09.220 --> 00:02:12.620 align:middle line:84%
This detail may actually be
handy for homework too, perhaps.

00:02:12.620 --> 00:02:14.420 align:middle line:90%
That's why I'm pointing it out.

00:02:14.420 --> 00:02:16.480 align:middle line:90%
OK, so that's what we have.

00:02:16.480 --> 00:02:18.500 align:middle line:84%
And so for each word
that's coming in,

00:02:18.500 --> 00:02:21.580 align:middle line:90%
we have a one-hot vector.

00:02:21.580 --> 00:02:24.820 align:middle line:84%
The one-hot vector is
just onto the vocabulary.

00:02:24.820 --> 00:02:30.660 align:middle line:84%
And we can either add them up
and get a multi-hot encoding,

00:02:30.660 --> 00:02:32.900 align:middle line:90%
or we can--

00:02:32.900 --> 00:02:35.802 align:middle line:84%
sorry-- get a count encoding,
or we can just do or.

00:02:35.802 --> 00:02:37.260 align:middle line:84%
Look for just any
ones in a column,

00:02:37.260 --> 00:02:39.300 align:middle line:90%
and get multi-hot encoding.

00:02:39.300 --> 00:02:41.860 align:middle line:84%
So that's what we
saw last class.

00:02:41.860 --> 00:02:43.980 align:middle line:84%
But this scheme,
while it's quite

00:02:43.980 --> 00:02:48.100 align:middle line:84%
effective for
simple problems, it

00:02:48.100 --> 00:02:49.800 align:middle line:84%
has some very
serious shortcomings.

00:02:49.800 --> 00:02:52.420 align:middle line:84%
And so we will delve
into those shortcomings

00:02:52.420 --> 00:02:54.300 align:middle line:84%
and then step back
and say, all right,

00:02:54.300 --> 00:02:56.872 align:middle line:84%
is there a solution
to fix these things?

00:02:56.872 --> 00:02:57.372 align:middle line:90%
OK.

00:02:57.372 --> 00:03:00.180 align:middle line:84%
Explain the problem
with one-hot vectors.

00:03:00.180 --> 00:03:01.520 align:middle line:90%
There are lots of problems.

00:03:01.520 --> 00:03:02.480 align:middle line:90%
Any volunteers?

00:03:02.480 --> 00:03:07.980 align:middle line:90%


00:03:07.980 --> 00:03:10.880 align:middle line:84%
AUDIENCE: So similar words
are understood differently.

00:03:10.880 --> 00:03:15.060 align:middle line:84%
So for example, if you
have this house is great

00:03:15.060 --> 00:03:17.420 align:middle line:84%
and this house is
awesome, great and awesome

00:03:17.420 --> 00:03:18.580 align:middle line:90%
is encoded differently.

00:03:18.580 --> 00:03:21.610 align:middle line:84%
And obviously, differently,
even though this [INAUDIBLE].

00:03:21.610 --> 00:03:22.860 align:middle line:90%
RAMA RAMAKRISHNAN: Absolutely.

00:03:22.860 --> 00:03:26.300 align:middle line:84%
So what he's pointing out is
that if you have two words which

00:03:26.300 --> 00:03:29.210 align:middle line:84%
are synonyms-- let's
say great and awesome--

00:03:29.210 --> 00:03:32.100 align:middle line:84%
we would hope that the way
we represent them using

00:03:32.100 --> 00:03:34.500 align:middle line:84%
these vectors would have
some connection to what

00:03:34.500 --> 00:03:35.940 align:middle line:90%
the words actually mean.

00:03:35.940 --> 00:03:37.780 align:middle line:84%
In particular, we
would hope that if they

00:03:37.780 --> 00:03:40.022 align:middle line:84%
mean similar things,
that they are close by.

00:03:40.022 --> 00:03:41.480 align:middle line:84%
If they mean very
different things,

00:03:41.480 --> 00:03:43.230 align:middle line:84%
we would hope that
they are very far away.

00:03:43.230 --> 00:03:45.700 align:middle line:84%
Things like that, common
sensical expectations of what

00:03:45.700 --> 00:03:47.580 align:middle line:90%
you want the vectors to have.

00:03:47.580 --> 00:03:50.460 align:middle line:84%
So it clearly won't have
that, and we'll look into it

00:03:50.460 --> 00:03:51.880 align:middle line:90%
in more detail in a bit.

00:03:51.880 --> 00:03:54.943 align:middle line:84%
But before we do that, there
is also a computational issue,

00:03:54.943 --> 00:03:56.110 align:middle line:90%
which we covered last class.

00:03:56.110 --> 00:03:59.170 align:middle line:84%
Which is that if the
vocabulary is really long,

00:03:59.170 --> 00:04:01.970 align:middle line:84%
then each token, each
word that's coming in here

00:04:01.970 --> 00:04:03.570 align:middle line:84%
will have a one-hot
vector that's

00:04:03.570 --> 00:04:05.730 align:middle line:84%
as long as the size
of the vocabulary.

00:04:05.730 --> 00:04:07.750 align:middle line:84%
If you have 500,000
words in your vocabulary,

00:04:07.750 --> 00:04:10.490 align:middle line:84%
every little word that comes in
has a vector which is 500,000

00:04:10.490 --> 00:04:13.760 align:middle line:84%
long, which feels like
a gross waste of stuff.

00:04:13.760 --> 00:04:16.647 align:middle line:90%


00:04:16.647 --> 00:04:18.730 align:middle line:84%
Now, you can mitigate it
somewhat by choosing only

00:04:18.730 --> 00:04:20.019 align:middle line:90%
the most frequent words.

00:04:20.019 --> 00:04:21.769 align:middle line:84%
But it does increase
the number of weights

00:04:21.769 --> 00:04:23.330 align:middle line:84%
the model has to
learn, and increase

00:04:23.330 --> 00:04:27.290 align:middle line:84%
the need for compute, and
data, and so on and so forth.

00:04:27.290 --> 00:04:29.490 align:middle line:84%
Now, let's say that we
have created a vocabulary

00:04:29.490 --> 00:04:31.210 align:middle line:90%
from a training corpus.

00:04:31.210 --> 00:04:34.690 align:middle line:84%
We have a bunch of
text that's coming in.

00:04:34.690 --> 00:04:36.023 align:middle line:90%
We have done the SD.

00:04:36.023 --> 00:04:37.690 align:middle line:84%
The standardization,
tokenization, we've

00:04:37.690 --> 00:04:39.290 align:middle line:90%
created vocabulary from it.

00:04:39.290 --> 00:04:42.250 align:middle line:84%
And let's say we get the
words movie and film.

00:04:42.250 --> 00:04:43.370 align:middle line:90%
So the question is--

00:04:43.370 --> 00:04:47.043 align:middle line:84%
and O's observation gets
to this immediately.

00:04:47.043 --> 00:04:48.710 align:middle line:84%
If you look at the
words movie and film,

00:04:48.710 --> 00:04:51.722 align:middle line:84%
are these two vectors
close to each other or not?

00:04:51.722 --> 00:04:56.920 align:middle line:84%
So if you have two vectors,
how would we measure closeness?

00:04:56.920 --> 00:04:59.020 align:middle line:84%
What's the simplest way
to think about closeness?

00:04:59.020 --> 00:05:02.560 align:middle line:90%


00:05:02.560 --> 00:05:05.180 align:middle line:90%
It's not a trick question.

00:05:05.180 --> 00:05:05.680 align:middle line:90%
Distance.

00:05:05.680 --> 00:05:06.380 align:middle line:90%
Yeah, exactly.

00:05:06.380 --> 00:05:08.480 align:middle line:84%
So if they're really
close, distance-wise,

00:05:08.480 --> 00:05:13.320 align:middle line:84%
you would hope the similar
words should be close by.

00:05:13.320 --> 00:05:20.040 align:middle line:84%
So here, let's just imagine
that the vector for movie--

00:05:20.040 --> 00:05:27.960 align:middle line:84%
let's say your vocabulary is,
I don't know, 100,000 long.

00:05:27.960 --> 00:05:33.280 align:middle line:84%
So your vector is 100,000
long, and the word for movie is

00:05:33.280 --> 00:05:33.940 align:middle line:90%
the position.

00:05:33.940 --> 00:05:38.520 align:middle line:84%
So this has a 1,
everything else is 0.

00:05:38.520 --> 00:05:39.520 align:middle line:90%
Right?

00:05:39.520 --> 00:05:42.520 align:middle line:90%


00:05:42.520 --> 00:05:44.040 align:middle line:84%
Sorry, this is the
vector for film,

00:05:44.040 --> 00:05:47.320 align:middle line:84%
and maybe this is the
position for film.

00:05:47.320 --> 00:05:48.720 align:middle line:90%
So that has a 1.

00:05:48.720 --> 00:05:51.150 align:middle line:90%
Everything else here--

00:05:51.150 --> 00:05:52.310 align:middle line:90%
0.

00:05:52.310 --> 00:05:55.670 align:middle line:84%
What is the distance
between these two vectors?

00:05:55.670 --> 00:05:57.870 align:middle line:84%
You just use the
Euclidean distance.

00:05:57.870 --> 00:05:59.510 align:middle line:84%
So the Euclidean
distance, you will

00:05:59.510 --> 00:06:04.390 align:middle line:84%
recall, you literally just take
the difference of these values,

00:06:04.390 --> 00:06:07.190 align:middle line:84%
square them, add them
up, take the square root.

00:06:07.190 --> 00:06:11.150 align:middle line:84%
So which means that all the
zeros will obviously give you 0.

00:06:11.150 --> 00:06:14.110 align:middle line:90%
This 1 is going to give you a 1.

00:06:14.110 --> 00:06:16.070 align:middle line:84%
This comparison is going
to give you another 1.

00:06:16.070 --> 00:06:17.190 align:middle line:90%
1 plus 1-- 2.

00:06:17.190 --> 00:06:17.910 align:middle line:90%
Root 2.

00:06:17.910 --> 00:06:20.090 align:middle line:90%
That's the answer.

00:06:20.090 --> 00:06:22.090 align:middle line:84%
So this is between these
two vectors, is root 2.

00:06:22.090 --> 00:06:25.190 align:middle line:90%


00:06:25.190 --> 00:06:30.590 align:middle line:84%
Now, so the distance
between them is root 2.

00:06:30.590 --> 00:06:34.070 align:middle line:84%
What about the one-hot encoded
vectors for good and bad?

00:06:34.070 --> 00:06:36.710 align:middle line:84%
Clearly, good and bad
mean opposite things.

00:06:36.710 --> 00:06:39.250 align:middle line:84%
What is the distance between
the good and bad 0, 1 vectors?

00:06:39.250 --> 00:06:42.670 align:middle line:90%


00:06:42.670 --> 00:06:45.510 align:middle line:90%
Still root 2.

00:06:45.510 --> 00:06:47.930 align:middle line:84%
Because the zeros
don't mean anything,

00:06:47.930 --> 00:06:49.895 align:middle line:84%
the ones are not
in the same place.

00:06:49.895 --> 00:06:52.520 align:middle line:84%
So when you subtract the 1 and
the 0, you'll get ones and ones.

00:06:52.520 --> 00:06:54.980 align:middle line:90%
Add them up to root 2.

00:06:54.980 --> 00:06:57.160 align:middle line:84%
In fact, you take any two
words in your vocabulary.

00:06:57.160 --> 00:06:59.000 align:middle line:84%
What's the distance between
the two one-hot vectors

00:06:59.000 --> 00:06:59.700 align:middle line:90%
for those words?

00:06:59.700 --> 00:07:01.660 align:middle line:90%
It's root 2.

00:07:01.660 --> 00:07:04.980 align:middle line:84%
So if any two words
have the same distance,

00:07:04.980 --> 00:07:08.780 align:middle line:84%
does this even have
a notion of distance?

00:07:08.780 --> 00:07:11.020 align:middle line:90%
It doesn't.

00:07:11.020 --> 00:07:13.980 align:middle line:84%
There's no notion of distance
from one-hot vectors.

00:07:13.980 --> 00:07:17.900 align:middle line:84%
It has no connection to the
actual meanings of these words.

00:07:17.900 --> 00:07:22.192 align:middle line:84%
It's just a way of
representing them.

00:07:22.192 --> 00:07:26.340 align:middle line:84%
So that is the big problem
with one-hot vectors.

00:07:26.340 --> 00:07:28.400 align:middle line:84%
So the distance between
them is the same,

00:07:28.400 --> 00:07:29.400 align:middle line:90%
regardless of the words.

00:07:29.400 --> 00:07:32.100 align:middle line:84%
It's got nothing to do with
the meaning of the words.

00:07:32.100 --> 00:07:35.900 align:middle line:84%
And this is a huge problem,
which we'll have to solve.

00:07:35.900 --> 00:07:38.598 align:middle line:84%
So to summarize where we are,
if the vocabulary is very long,

00:07:38.598 --> 00:07:40.140 align:middle line:84%
each token will have
a one-hot vector

00:07:40.140 --> 00:07:41.560 align:middle line:90%
that's long as vocabulary.

00:07:41.560 --> 00:07:45.660 align:middle line:84%
That's a computational
and training problem.

00:07:45.660 --> 00:07:47.260 align:middle line:84%
And then this is
a deeper problem

00:07:47.260 --> 00:07:49.218 align:middle line:84%
where there's no connection
between the meaning

00:07:49.218 --> 00:07:51.340 align:middle line:90%
of a word and its vector.

00:07:51.340 --> 00:07:56.180 align:middle line:84%
So wouldn't it be nice
if vectors that represent

00:07:56.180 --> 00:07:57.700 align:middle line:90%
synonyms--

00:07:57.700 --> 00:07:59.860 align:middle line:90%
movie and film, apple-banana--

00:07:59.860 --> 00:08:01.980 align:middle line:84%
hopefully, they are
close to each other.

00:08:01.980 --> 00:08:03.460 align:middle line:84%
Wouldn't it be
nice if the vectors

00:08:03.460 --> 00:08:05.210 align:middle line:84%
for things that mean
very different things

00:08:05.210 --> 00:08:07.020 align:middle line:90%
were far from each other?

00:08:07.020 --> 00:08:09.280 align:middle line:84%
So let's take a look at
a particular example.

00:08:09.280 --> 00:08:09.780 align:middle line:90%
OK?

00:08:09.780 --> 00:08:13.300 align:middle line:84%
Let's assume that we
have been magically given

00:08:13.300 --> 00:08:15.340 align:middle line:84%
these vectors so
that they actually

00:08:15.340 --> 00:08:17.340 align:middle line:90%
have some notion of meaning.

00:08:17.340 --> 00:08:21.980 align:middle line:84%
And for convenience, let's say
that we take just the first two

00:08:21.980 --> 00:08:23.438 align:middle line:90%
dimensions of these vectors--

00:08:23.438 --> 00:08:24.980 align:middle line:84%
the first two
dimensions-- so that we

00:08:24.980 --> 00:08:28.140 align:middle line:90%
can do a scatter plot on them.

00:08:28.140 --> 00:08:30.640 align:middle line:84%
So we plot the first
dimension of these vectors,

00:08:30.640 --> 00:08:31.780 align:middle line:90%
the second dimension.

00:08:31.780 --> 00:08:34.179 align:middle line:84%
And what we have in
this little cartoon is

00:08:34.179 --> 00:08:41.520 align:middle line:84%
we have plotted the word for
factory, for home, for building,

00:08:41.520 --> 00:08:43.980 align:middle line:84%
and they all happen
to be clustered here.

00:08:43.980 --> 00:08:46.730 align:middle line:84%
Clearly, this representation
is capturing some notion

00:08:46.730 --> 00:08:52.890 align:middle line:84%
of what the thing is, some sort
of building And here, we have,

00:08:52.890 --> 00:08:55.090 align:middle line:90%
a bicycle, truck, and car.

00:08:55.090 --> 00:08:59.370 align:middle line:84%
Clearly, this is the automobile
cluster, transportation cluster.

00:08:59.370 --> 00:09:01.450 align:middle line:84%
And here, we have
a fruit cluster.

00:09:01.450 --> 00:09:04.330 align:middle line:84%
And here, we have
sports balls cluster.

00:09:04.330 --> 00:09:06.060 align:middle line:90%
OK?

00:09:06.060 --> 00:09:07.650 align:middle line:84%
Because it's a
cartoon, things are

00:09:07.650 --> 00:09:10.116 align:middle line:90%
all nice and cleanly separated.

00:09:10.116 --> 00:09:12.190 align:middle line:84%
So now, if you take
the word apple,

00:09:12.190 --> 00:09:14.450 align:middle line:84%
where do you think
it's going to go?

00:09:14.450 --> 00:09:18.620 align:middle line:84%
It's going to go
into A, C, D, or B?

00:09:18.620 --> 00:09:19.510 align:middle line:90%
C, right?

00:09:19.510 --> 00:09:24.210 align:middle line:84%
It makes eminent sense it's
going to go to C. Good.

00:09:24.210 --> 00:09:29.070 align:middle line:84%
Now wouldn't it be nice
if, more generally,

00:09:29.070 --> 00:09:32.810 align:middle line:84%
if the geometric relationship
between word vectors

00:09:32.810 --> 00:09:35.130 align:middle line:84%
represents a
semantic relationship

00:09:35.130 --> 00:09:39.250 align:middle line:84%
between the underlying objects
that the words represent?

00:09:39.250 --> 00:09:41.690 align:middle line:84%
And I say relationship
and not distance

00:09:41.690 --> 00:09:43.350 align:middle line:90%
because it's not just distance.

00:09:43.350 --> 00:09:46.360 align:middle line:90%
It's actually more than that.

00:09:46.360 --> 00:09:48.320 align:middle line:90%
So let's look another one.

00:09:48.320 --> 00:09:53.000 align:middle line:84%
Here, we have-- this is the
vector plotted for puppy

00:09:53.000 --> 00:09:57.300 align:middle line:90%
and dog, and this is calf.

00:09:57.300 --> 00:09:58.740 align:middle line:84%
We have plotted
the word for calf.

00:09:58.740 --> 00:10:00.440 align:middle line:84%
And let's say that
we need to figure out

00:10:00.440 --> 00:10:04.920 align:middle line:84%
where would the embedding--
the word vector for cow appear?

00:10:04.920 --> 00:10:06.060 align:middle line:90%
Where is the most logical?

00:10:06.060 --> 00:10:07.920 align:middle line:90%
Should it be A?

00:10:07.920 --> 00:10:08.900 align:middle line:90%
Should it be C?

00:10:08.900 --> 00:10:09.540 align:middle line:90%
Should it be B?

00:10:09.540 --> 00:10:11.760 align:middle line:90%
Where should it be?

00:10:11.760 --> 00:10:13.744 align:middle line:90%
AUDIENCE: This is C.

00:10:13.744 --> 00:10:14.720 align:middle line:90%
RAMA RAMAKRISHNAN: C?

00:10:14.720 --> 00:10:16.360 align:middle line:90%
OK, what's the logic?

00:10:16.360 --> 00:10:17.460 align:middle line:90%
Any volunteers?

00:10:17.460 --> 00:10:20.020 align:middle line:90%
Just put your hand up.

00:10:20.020 --> 00:10:20.520 align:middle line:90%
Yes?

00:10:20.520 --> 00:10:23.920 align:middle line:90%


00:10:23.920 --> 00:10:25.560 align:middle line:84%
AUDIENCE: A calf
is a baby animal,

00:10:25.560 --> 00:10:27.680 align:middle line:84%
whereas the cow is
probably the adult.

00:10:27.680 --> 00:10:29.520 align:middle line:84%
So it'd be closer
to dog, which is B--

00:10:29.520 --> 00:10:30.728 align:middle line:90%
the adult version of the dog.

00:10:30.728 --> 00:10:31.812 align:middle line:90%
RAMA RAMAKRISHNAN: Got it.

00:10:31.812 --> 00:10:34.080 align:middle line:84%
So you're basically saying,
go from the puppy version

00:10:34.080 --> 00:10:35.600 align:middle line:90%
to the grown up version.

00:10:35.600 --> 00:10:37.360 align:middle line:84%
That's what you're
getting at, right?

00:10:37.360 --> 00:10:39.600 align:middle line:84%
And that's a totally valid
way to think about it.

00:10:39.600 --> 00:10:42.058 align:middle line:84%
But there are a couple of ways
to think about this-- which,

00:10:42.058 --> 00:10:43.588 align:middle line:90%
this is one of those two ways.

00:10:43.588 --> 00:10:45.630 align:middle line:84%
So what you can do is you
can actually look at it

00:10:45.630 --> 00:10:47.410 align:middle line:90%
and say, well, OK.

00:10:47.410 --> 00:10:50.710 align:middle line:84%
If this is bringing you bad
memories of GMAT and GRE

00:10:50.710 --> 00:10:52.540 align:middle line:84%
and stuff like
that, I apologize.

00:10:52.540 --> 00:10:55.110 align:middle line:90%


00:10:55.110 --> 00:10:58.750 align:middle line:84%
So a puppy is to a dog like
a calf is to a cow, right?

00:10:58.750 --> 00:11:01.170 align:middle line:84%
Which means that there's exactly
what Jay is pointing out.

00:11:01.170 --> 00:11:03.910 align:middle line:84%
You can go from the baby version
to the full grown version

00:11:03.910 --> 00:11:06.590 align:middle line:84%
if you go in the
horizontal direction.

00:11:06.590 --> 00:11:10.130 align:middle line:84%
But maybe, if you go in
the vertical direction,

00:11:10.130 --> 00:11:12.110 align:middle line:84%
you're essentially
going up and down

00:11:12.110 --> 00:11:16.590 align:middle line:90%
the young entities of animals.

00:11:16.590 --> 00:11:18.430 align:middle line:90%
So here, you're growing with--

00:11:18.430 --> 00:11:21.250 align:middle line:84%
you're still across the
same dimension of animals.

00:11:21.250 --> 00:11:24.030 align:middle line:84%
You're just going
from the age level.

00:11:24.030 --> 00:11:25.470 align:middle line:90%
That is the band here.

00:11:25.470 --> 00:11:27.470 align:middle line:84%
So this is the grown up
version of a whole bunch

00:11:27.470 --> 00:11:30.010 align:middle line:84%
of animals, the puppy version
of a whole bunch of animals.

00:11:30.010 --> 00:11:32.430 align:middle line:84%
So the vertical dimension
measures some variation

00:11:32.430 --> 00:11:38.530 align:middle line:84%
across animal species of the
same, roughly, maturity stage.

00:11:38.530 --> 00:11:41.060 align:middle line:90%
So these directions also matter.

00:11:41.060 --> 00:11:43.780 align:middle line:90%
It's not just the distance.

00:11:43.780 --> 00:11:46.740 align:middle line:84%
That's what I mean when I
say semantic relationship

00:11:46.740 --> 00:11:48.900 align:middle line:90%
and geometric relationship.

00:11:48.900 --> 00:11:51.580 align:middle line:84%
The relationship is
distance and direction.

00:11:51.580 --> 00:11:53.220 align:middle line:90%
Both have to be involved.

00:11:53.220 --> 00:11:58.900 align:middle line:84%
So now, word embeddings--
as we will learn soon--

00:11:58.900 --> 00:12:01.860 align:middle line:84%
are word vectors designed
to achieve exactly

00:12:01.860 --> 00:12:04.120 align:middle line:90%
these requirements.

00:12:04.120 --> 00:12:06.660 align:middle line:84%
They will achieve
these requirements,

00:12:06.660 --> 00:12:09.680 align:middle line:84%
and they will fix both these
problems very elegantly.

00:12:09.680 --> 00:12:12.983 align:middle line:90%


00:12:12.983 --> 00:12:14.900 align:middle line:84%
So let's say that we
have word embeddings that

00:12:14.900 --> 00:12:16.260 align:middle line:90%
achieve both these problems.

00:12:16.260 --> 00:12:17.660 align:middle line:90%
Are we basically done?

00:12:17.660 --> 00:12:19.860 align:middle line:90%
Can we declare victory?

00:12:19.860 --> 00:12:23.660 align:middle line:84%
Or is there anything
that even words

00:12:23.660 --> 00:12:28.420 align:middle line:84%
which actually capture the
meaning of the underlying thing

00:12:28.420 --> 00:12:29.600 align:middle line:90%
don't fully address?

00:12:29.600 --> 00:12:32.160 align:middle line:84%
Is there any remaining problem
we have to worry about?

00:12:32.160 --> 00:12:33.460 align:middle line:90%
Yes?

00:12:33.460 --> 00:12:34.280 align:middle line:90%
AUDIENCE: Context.

00:12:34.280 --> 00:12:35.405 align:middle line:90%
RAMA RAMAKRISHNAN: Context?

00:12:35.405 --> 00:12:36.860 align:middle line:90%
Yes.

00:12:36.860 --> 00:12:38.380 align:middle line:90%
Context, right?

00:12:38.380 --> 00:12:41.420 align:middle line:90%
The fact is a word's meaning--

00:12:41.420 --> 00:12:43.160 align:middle line:90%
sure, every word has a meaning.

00:12:43.160 --> 00:12:46.540 align:middle line:84%
But we know that some words
have multiple meanings,

00:12:46.540 --> 00:12:50.400 align:middle line:84%
and that meaning is
really inferrential

00:12:50.400 --> 00:12:53.233 align:middle line:84%
or you can make sense of it--
only if you know the surrounding

00:12:53.233 --> 00:12:55.100 align:middle line:90%
context.

00:12:55.100 --> 00:12:59.140 align:middle line:84%
If you see the word
bank, B-A-N-K--

00:12:59.140 --> 00:13:01.220 align:middle line:84%
sure, it could be a
financial institution.

00:13:01.220 --> 00:13:03.020 align:middle line:90%
It could be the side of a river.

00:13:03.020 --> 00:13:07.260 align:middle line:84%
It could be the act of a plane
turning in one direction.

00:13:07.260 --> 00:13:11.820 align:middle line:84%
It could be someone hoping for
something, banking on something.

00:13:11.820 --> 00:13:15.140 align:middle line:84%
The list of possible meanings
of the word bank is basically

00:13:15.140 --> 00:13:18.740 align:middle line:84%
enormous, and you cannot figure
out what it means unless you

00:13:18.740 --> 00:13:20.980 align:middle line:84%
know what else is going
on around that word.

00:13:20.980 --> 00:13:23.360 align:middle line:84%
So context is super,
super important.

00:13:23.360 --> 00:13:25.900 align:middle line:84%
And these word
embeddings just tell you

00:13:25.900 --> 00:13:27.620 align:middle line:90%
what the meaning of the word is.

00:13:27.620 --> 00:13:29.340 align:middle line:84%
And basically, what's
going to happen

00:13:29.340 --> 00:13:32.340 align:middle line:84%
when you have a word which could
mean many different things,

00:13:32.340 --> 00:13:36.660 align:middle line:84%
it's going to give you some
average version of that meaning.

00:13:36.660 --> 00:13:39.412 align:middle line:84%
The average version is
not going to be very good.

00:13:39.412 --> 00:13:41.370 align:middle line:84%
There are some words
which only mean one thing,

00:13:41.370 --> 00:13:42.850 align:middle line:90%
and you'll be OK there.

00:13:42.850 --> 00:13:47.650 align:middle line:84%
But for the rest of it,
it's going to be tough.

00:13:47.650 --> 00:13:50.840 align:middle line:90%
So what we need is some way--

00:13:50.840 --> 00:13:53.410 align:middle line:90%


00:13:53.410 --> 00:13:56.683 align:middle line:84%
you need to find a way to make
word embeddings contextual.

00:13:56.683 --> 00:13:58.850 align:middle line:84%
Meaning, we need to somehow
consider the other words

00:13:58.850 --> 00:14:01.010 align:middle line:90%
in the sentence.

00:14:01.010 --> 00:14:05.210 align:middle line:84%
So if we can do that, then
we will be in great shape

00:14:05.210 --> 00:14:08.370 align:middle line:84%
to solve all sorts
of NLP problems.

00:14:08.370 --> 00:14:13.250 align:middle line:84%
So as it turns out, contextual
word embeddings or word vectors

00:14:13.250 --> 00:14:17.050 align:middle line:84%
or word embeddings will achieve
both these requirements.

00:14:17.050 --> 00:14:19.970 align:middle line:84%
They capture the semantic
geometric relationship

00:14:19.970 --> 00:14:23.970 align:middle line:84%
thing I talked about,
and they are contextual.

00:14:23.970 --> 00:14:26.370 align:middle line:90%
They're really fantastic.

00:14:26.370 --> 00:14:29.890 align:middle line:84%
And the key to calculating
contextual word embeddings

00:14:29.890 --> 00:14:33.450 align:middle line:90%
is the transformer.

00:14:33.450 --> 00:14:35.920 align:middle line:84%
That is why transformers
are justifiably famous.

00:14:35.920 --> 00:14:39.600 align:middle line:90%


00:14:39.600 --> 00:14:41.520 align:middle line:90%
That's the lay of the land here.

00:14:41.520 --> 00:14:43.240 align:middle line:84%
So today, we're
going to look at how

00:14:43.240 --> 00:14:48.520 align:middle line:84%
to calculate standalone or
uncontextual word embeddings.

00:14:48.520 --> 00:14:50.480 align:middle line:84%
And then starting
Monday, we will

00:14:50.480 --> 00:14:52.800 align:middle line:84%
take these
un-standalone embeddings

00:14:52.800 --> 00:14:55.720 align:middle line:84%
and make them contextual
using transformers.

00:14:55.720 --> 00:14:57.240 align:middle line:90%
That is the plan.

00:14:57.240 --> 00:14:58.840 align:middle line:90%
Any questions so far?

00:14:58.840 --> 00:15:00.840 align:middle line:84%
So now, let's think
about how we can

00:15:00.840 --> 00:15:03.773 align:middle line:84%
learn these standalone
embeddings from data.

00:15:03.773 --> 00:15:06.320 align:middle line:84%
Now, the naive way
to think about it

00:15:06.320 --> 00:15:08.480 align:middle line:84%
would be, hey, why don't
we manually collect

00:15:08.480 --> 00:15:11.110 align:middle line:84%
a whole bunch of synonyms,
antonyms, related words, et

00:15:11.110 --> 00:15:14.240 align:middle line:84%
cetera and try to assign
embedding vectors to them that

00:15:14.240 --> 00:15:17.698 align:middle line:90%
satisfy our requirements?

00:15:17.698 --> 00:15:19.240 align:middle line:84%
Now as you can
imagine, there's going

00:15:19.240 --> 00:15:22.540 align:middle line:84%
to be a long, painful, and
never quite complete exercise.

00:15:22.540 --> 00:15:26.040 align:middle line:90%


00:15:26.040 --> 00:15:29.310 align:middle line:84%
And given that we are
machine learning people,

00:15:29.310 --> 00:15:31.060 align:middle line:84%
question is, can we
do it in a better way?

00:15:31.060 --> 00:15:32.477 align:middle line:84%
Can we just learn
it from the data

00:15:32.477 --> 00:15:34.160 align:middle line:84%
without doing any of
this manual stuff?

00:15:34.160 --> 00:15:39.710 align:middle line:84%
And the key insight
that makes it all happen

00:15:39.710 --> 00:15:44.350 align:middle line:84%
is this humble looking line on
the screen by John Firth, who

00:15:44.350 --> 00:15:45.950 align:middle line:90%
was a linguist.

00:15:45.950 --> 00:15:49.250 align:middle line:84%
"You shall know a word
by the company it keeps."

00:15:49.250 --> 00:15:52.590 align:middle line:84%
I wish I could deliver
this in a British accent.

00:15:52.590 --> 00:15:55.590 align:middle line:84%
"You shall know a word
by the company it keeps."

00:15:55.590 --> 00:15:58.450 align:middle line:90%
It's a very profound statement.

00:15:58.450 --> 00:16:02.630 align:middle line:84%
And here is the key
intuition behind this.

00:16:02.630 --> 00:16:05.590 align:middle line:84%
It says, let's say that
you have a sentence like,

00:16:05.590 --> 00:16:09.630 align:middle line:84%
"the acting in the
dash was superb."

00:16:09.630 --> 00:16:11.150 align:middle line:84%
What are some words
that you folks

00:16:11.150 --> 00:16:12.942 align:middle line:84%
think are likely to
appear in the sentence?

00:16:12.942 --> 00:16:15.270 align:middle line:90%


00:16:15.270 --> 00:16:16.870 align:middle line:90%
Shout it out.

00:16:16.870 --> 00:16:22.070 align:middle line:90%
Play, movie, show, musical.

00:16:22.070 --> 00:16:24.613 align:middle line:84%
Those are all some
great candidates.

00:16:24.613 --> 00:16:26.030 align:middle line:84%
The acting in the
movie, the film,

00:16:26.030 --> 00:16:27.450 align:middle line:90%
musical, and so on and so forth.

00:16:27.450 --> 00:16:29.052 align:middle line:84%
Now, let's say that
I ask you, what

00:16:29.052 --> 00:16:31.510 align:middle line:84%
are some words that are unlikely
to appear in the sentence?

00:16:31.510 --> 00:16:35.820 align:middle line:84%
And I think we could all be
here for days listing them out.

00:16:35.820 --> 00:16:38.125 align:middle line:90%
I just listed these out.

00:16:38.125 --> 00:16:39.500 align:middle line:84%
I love the word
tensor, so I have

00:16:39.500 --> 00:16:41.900 align:middle line:84%
to find a way to
use it somewhere.

00:16:41.900 --> 00:16:42.400 align:middle line:90%
All right.

00:16:42.400 --> 00:16:44.180 align:middle line:84%
So the acting of the
banana was superb.

00:16:44.180 --> 00:16:46.100 align:middle line:90%
Clearly nonsensical.

00:16:46.100 --> 00:16:52.380 align:middle line:84%
So what we are seeing here
is that if certain words are

00:16:52.380 --> 00:16:55.500 align:middle line:90%
interchangeable in a sentence--

00:16:55.500 --> 00:17:00.060 align:middle line:84%
meaning, you change them, and
the sentence still makes sense--

00:17:00.060 --> 00:17:02.020 align:middle line:84%
if they appear in
the same context

00:17:02.020 --> 00:17:04.359 align:middle line:84%
very often, i.e. if
they're interchangeable,

00:17:04.359 --> 00:17:06.998 align:middle line:90%
they are probably related.

00:17:06.998 --> 00:17:10.660 align:middle line:84%
Sort of like, we don't even
have to know what the word is.

00:17:10.660 --> 00:17:13.285 align:middle line:84%
All we have to know is that
this word and this word,

00:17:13.285 --> 00:17:15.160 align:middle line:84%
you can drop them into
a particular sentence.

00:17:15.160 --> 00:17:17.220 align:middle line:84%
You can fill in the blank of
that sentence with that word,

00:17:17.220 --> 00:17:18.395 align:middle line:90%
and it actually makes sense.

00:17:18.395 --> 00:17:19.520 align:middle line:90%
Then you're like-- oh, wow.

00:17:19.520 --> 00:17:20.020 align:middle line:90%
OK.

00:17:20.020 --> 00:17:21.980 align:middle line:90%
These words are related then.

00:17:21.980 --> 00:17:24.660 align:middle line:84%
You're inferring
their relatedness,

00:17:24.660 --> 00:17:30.060 align:middle line:84%
not by looking at them directly,
but by seeing where they live.

00:17:30.060 --> 00:17:35.900 align:middle line:84%
It's a very, very clever idea,
and it'll slowly sink into you.

00:17:35.900 --> 00:17:37.283 align:middle line:90%
So that's the first observation.

00:17:37.283 --> 00:17:39.200 align:middle line:84%
If they appear in the
same context very often,

00:17:39.200 --> 00:17:41.780 align:middle line:90%
they are likely to be related.

00:17:41.780 --> 00:17:45.190 align:middle line:84%
Generally, related words
appear in related contexts.

00:17:45.190 --> 00:17:47.700 align:middle line:90%


00:17:47.700 --> 00:17:51.340 align:middle line:84%
So all we have to do
is to figure out a way

00:17:51.340 --> 00:17:57.300 align:middle line:84%
to calculate context, and
then use that to understand

00:17:57.300 --> 00:18:00.472 align:middle line:84%
what the words are that happen
to live in these contexts.

00:18:00.472 --> 00:18:02.680 align:middle line:84%
And there are some beautiful
ways to do these things,

00:18:02.680 --> 00:18:06.260 align:middle line:84%
and we'll and we'll really dive
deep into one such way to do it.

00:18:06.260 --> 00:18:10.940 align:middle line:84%
So what we're going to
do in this approach is

00:18:10.940 --> 00:18:15.900 align:middle line:84%
that since words that
appear in related contexts

00:18:15.900 --> 00:18:19.180 align:middle line:84%
mean same similar things, first
of all, I have to define what

00:18:19.180 --> 00:18:21.297 align:middle line:90%
do you mean by context?

00:18:21.297 --> 00:18:22.880 align:middle line:84%
There are many ways
to define context.

00:18:22.880 --> 00:18:24.280 align:middle line:84%
We're going to go with a
very simple explanation,

00:18:24.280 --> 00:18:25.113 align:middle line:90%
a simple definition.

00:18:25.113 --> 00:18:27.020 align:middle line:84%
Which is that if
words happen to appear

00:18:27.020 --> 00:18:31.350 align:middle line:84%
in the same sentence a
lot, let me think that they

00:18:31.350 --> 00:18:32.350 align:middle line:90%
are in the same context.

00:18:32.350 --> 00:18:34.650 align:middle line:90%
So context here means sentence.

00:18:34.650 --> 00:18:38.290 align:middle line:84%
So what we can do
is we can actually

00:18:38.290 --> 00:18:39.870 align:middle line:90%
take a whole bunch of text--

00:18:39.870 --> 00:18:42.010 align:middle line:90%
maybe all of Wikipedia--

00:18:42.010 --> 00:18:43.630 align:middle line:84%
and then break it
up into sentences.

00:18:43.630 --> 00:18:45.890 align:middle line:84%
We'll have billions
of sentences.

00:18:45.890 --> 00:18:47.830 align:middle line:84%
And then for all these
billion sentences,

00:18:47.830 --> 00:18:50.770 align:middle line:84%
we can literally go and
count for every pair of words

00:18:50.770 --> 00:18:52.770 align:middle line:84%
how many times are both
these words showing up

00:18:52.770 --> 00:18:53.645 align:middle line:90%
in the same sentence.

00:18:53.645 --> 00:18:55.970 align:middle line:90%


00:18:55.970 --> 00:18:58.130 align:middle line:90%
And we call this co-occurrence.

00:18:58.130 --> 00:19:00.730 align:middle line:84%
The words are co-occurring
in the sentence.

00:19:00.730 --> 00:19:03.210 align:middle line:84%
And it doesn't have to
be next to each other.

00:19:03.210 --> 00:19:05.770 align:middle line:84%
We know that in
complicated words,

00:19:05.770 --> 00:19:07.810 align:middle line:84%
a word at the very
end of the sentence

00:19:07.810 --> 00:19:10.690 align:middle line:84%
could actually alter the meaning
Its meaning could be altered

00:19:10.690 --> 00:19:12.590 align:middle line:84%
by a word that happened in the
very beginning of the sentence,

00:19:12.590 --> 00:19:14.290 align:middle line:84%
and it could be a
really long sentence.

00:19:14.290 --> 00:19:16.330 align:middle line:84%
So we take the whole
sentence and say,

00:19:16.330 --> 00:19:18.550 align:middle line:84%
are two words co-occurring
in the sentence?

00:19:18.550 --> 00:19:19.150 align:middle line:90%
Yes or no?

00:19:19.150 --> 00:19:20.730 align:middle line:90%
And we just count them up.

00:19:20.730 --> 00:19:27.800 align:middle line:84%
And when we do that, we will
get something like this.

00:19:27.800 --> 00:19:31.620 align:middle line:84%
So this just captures what
I've been talking about.

00:19:31.620 --> 00:19:34.120 align:middle line:84%
Identify all the words that
occur, let's say, in Wikipedia.

00:19:34.120 --> 00:19:36.480 align:middle line:84%
And then for every sentence,
you look at every word pair

00:19:36.480 --> 00:19:37.897 align:middle line:84%
and count the
number of times they

00:19:37.897 --> 00:19:41.640 align:middle line:84%
appear in the same sentence
across all those sentences.

00:19:41.640 --> 00:19:44.760 align:middle line:84%
This is a word-word
co-occurrence matrix.

00:19:44.760 --> 00:19:47.673 align:middle line:84%
So for example, let's assume
that you took all of Wikipedia,

00:19:47.673 --> 00:19:49.340 align:middle line:84%
looked at all the
words, distinct words,

00:19:49.340 --> 00:19:52.420 align:middle line:84%
and you found there
were 500,000 words.

00:19:52.420 --> 00:19:56.840 align:middle line:84%
So there are 500,000
words here in the columns.

00:19:56.840 --> 00:20:00.055 align:middle line:90%
500,000 words on the rows.

00:20:00.055 --> 00:20:01.560 align:middle line:90%
So columns and rows.

00:20:01.560 --> 00:20:05.000 align:middle line:84%
And then you go, and
each cell of this table

00:20:05.000 --> 00:20:07.760 align:middle line:84%
basically has a number
that you calculate

00:20:07.760 --> 00:20:10.960 align:middle line:84%
which is the number of times
the word in the row and the word

00:20:10.960 --> 00:20:13.420 align:middle line:84%
in the column happened to
show up in the same sentence.

00:20:13.420 --> 00:20:14.320 align:middle line:90%
That's it.

00:20:14.320 --> 00:20:18.220 align:middle line:84%
So for instance, if you
look at deep and learning--

00:20:18.220 --> 00:20:20.400 align:middle line:84%
the word deep and
the word learning--

00:20:20.400 --> 00:20:25.180 align:middle line:84%
maybe those two words
occurred in the same sentence,

00:20:25.180 --> 00:20:28.430 align:middle line:90%
maybe 3,025 times.

00:20:28.430 --> 00:20:31.430 align:middle line:84%
3,025 sentences across
all of Wikipedia.

00:20:31.430 --> 00:20:36.030 align:middle line:84%
You put 3,025
right in that cell.

00:20:36.030 --> 00:20:38.950 align:middle line:84%
Many words are unlikely to
appear in the same sentence.

00:20:38.950 --> 00:20:41.010 align:middle line:84%
So much of this matrix
is going to be 0.

00:20:41.010 --> 00:20:44.280 align:middle line:90%


00:20:44.280 --> 00:20:50.030 align:middle line:84%
But we fundamentally form
this co-occurrence matrix.

00:20:50.030 --> 00:20:55.030 align:middle line:84%
This matrix essentially embodies
all the context information

00:20:55.030 --> 00:21:00.810 align:middle line:84%
that we can work with in a very
compact, beautiful, elegant way.

00:21:00.810 --> 00:21:03.253 align:middle line:90%


00:21:03.253 --> 00:21:04.670 align:middle line:84%
And using this,
we're going to try

00:21:04.670 --> 00:21:07.390 align:middle line:84%
to figure out what the
word embeddings actually

00:21:07.390 --> 00:21:08.170 align:middle line:90%
are going to be.

00:21:08.170 --> 00:21:11.710 align:middle line:90%


00:21:11.710 --> 00:21:14.310 align:middle line:84%
So by the way, the approach
I'm describing here

00:21:14.310 --> 00:21:17.100 align:middle line:84%
to calculate standalone
embeddings is called GloVe.

00:21:17.100 --> 00:21:20.590 align:middle line:90%


00:21:20.590 --> 00:21:21.710 align:middle line:90%
It's called GloVe.

00:21:21.710 --> 00:21:26.820 align:middle line:84%
And standalone embeddings first
came on to the NLP deep learning

00:21:26.820 --> 00:21:27.620 align:middle line:90%
scene--

00:21:27.620 --> 00:21:29.840 align:middle line:90%
there were two ways of doing it.

00:21:29.840 --> 00:21:31.360 align:middle line:90%
One was called word2vec.

00:21:31.360 --> 00:21:32.620 align:middle line:90%
Word2vec.

00:21:32.620 --> 00:21:34.380 align:middle line:90%
The other one is GloVe.

00:21:34.380 --> 00:21:35.820 align:middle line:90%
And they are both comparable.

00:21:35.820 --> 00:21:38.620 align:middle line:84%
They use slightly different
mechanisms of doing this.

00:21:38.620 --> 00:21:40.620 align:middle line:84%
We went with word
for this lecture

00:21:40.620 --> 00:21:43.780 align:middle line:84%
because I think it's actually
a little easier to understand,

00:21:43.780 --> 00:21:47.580 align:middle line:90%
and equally effective.

00:21:47.580 --> 00:21:49.140 align:middle line:90%
So this is what we have.

00:21:49.140 --> 00:21:51.260 align:middle line:84%
And so what we want
to do is we want

00:21:51.260 --> 00:21:53.060 align:middle line:84%
to learn these
embedding vectors that

00:21:53.060 --> 00:21:56.580 align:middle line:84%
can be used to essentially
approximate this matrix.

00:21:56.580 --> 00:22:00.100 align:middle line:84%
If we can find vectors that
can actually approximate

00:22:00.100 --> 00:22:03.300 align:middle line:84%
this matrix, then hopefully
those vectors do, in fact,

00:22:03.300 --> 00:22:05.840 align:middle line:84%
capture some notion of what
the words actually mean.

00:22:05.840 --> 00:22:07.460 align:middle line:90%
So let me put it differently.

00:22:07.460 --> 00:22:10.100 align:middle line:90%
You come to me with this matrix.

00:22:10.100 --> 00:22:12.060 align:middle line:90%
And you say, OK.

00:22:12.060 --> 00:22:14.208 align:middle line:84%
Rama, do you have
embeddings for me.

00:22:14.208 --> 00:22:15.000 align:middle line:90%
And I'm like, yeah.

00:22:15.000 --> 00:22:16.740 align:middle line:84%
I reach into my bag,
and I'm like, OK.

00:22:16.740 --> 00:22:19.793 align:middle line:84%
Every one of those 500,000
words, I have an embedding.

00:22:19.793 --> 00:22:21.460 align:middle line:84%
Let's ignore for a
moment how I actually

00:22:21.460 --> 00:22:22.660 align:middle line:90%
calculated the embeddings.

00:22:22.660 --> 00:22:24.060 align:middle line:90%
I have the embeddings.

00:22:24.060 --> 00:22:28.820 align:middle line:84%
How will you know if my
embeddings are any good?

00:22:28.820 --> 00:22:30.300 align:middle line:90%
How will you know?

00:22:30.300 --> 00:22:34.313 align:middle line:84%
How can you actually assess if
the embeddings are any good?

00:22:34.313 --> 00:22:35.980 align:middle line:84%
Well, you can certainly
say, OK, give me

00:22:35.980 --> 00:22:37.340 align:middle line:84%
the embeddings for
movie and film,

00:22:37.340 --> 00:22:39.180 align:middle line:84%
and you can see if
they're really close by.

00:22:39.180 --> 00:22:42.020 align:middle line:84%
If you look at the embedding
for movie and tensor--

00:22:42.020 --> 00:22:43.900 align:middle line:90%
and hopefully they'll far away.

00:22:43.900 --> 00:22:47.380 align:middle line:84%
But you'll never
get done, right?

00:22:47.380 --> 00:22:51.300 align:middle line:84%
How can you systematically
evaluate this?

00:22:51.300 --> 00:22:54.700 align:middle line:84%
Well, what if you could
actually-- what if I come to you

00:22:54.700 --> 00:22:57.420 align:middle line:84%
and say, not only am I going
to give you an embedding,

00:22:57.420 --> 00:23:00.500 align:middle line:84%
here is a procedure which you
can use with these embeddings

00:23:00.500 --> 00:23:02.120 align:middle line:90%
to validate how good they are.

00:23:02.120 --> 00:23:03.320 align:middle line:90%
And here is the procedure.

00:23:03.320 --> 00:23:05.940 align:middle line:84%
What you can do is you
can use the embedding

00:23:05.940 --> 00:23:10.020 align:middle line:84%
to recreate the
co-occurrence matrix.

00:23:10.020 --> 00:23:12.460 align:middle line:84%
And if the recreated
co-occurrence matrix actually

00:23:12.460 --> 00:23:15.100 align:middle line:84%
matches the real matrix,
well, these embeddings

00:23:15.100 --> 00:23:17.223 align:middle line:90%
probably are pretty good.

00:23:17.223 --> 00:23:19.140 align:middle line:84%
Remember, the whole point
of the co-occurrence

00:23:19.140 --> 00:23:20.960 align:middle line:84%
is to handle this
context information.

00:23:20.960 --> 00:23:23.150 align:middle line:84%
So if the embeddings can
actually recreate them,

00:23:23.150 --> 00:23:24.998 align:middle line:84%
reconstruct them
pretty close, right?

00:23:24.998 --> 00:23:26.040 align:middle line:90%
It will never be perfect.

00:23:26.040 --> 00:23:28.190 align:middle line:84%
But if it comes pretty close,
they'll be like, wow, OK.

00:23:28.190 --> 00:23:29.730 align:middle line:84%
These embeddings
do mean something.

00:23:29.730 --> 00:23:31.110 align:middle line:84%
So if it turns
out, for instance,

00:23:31.110 --> 00:23:35.610 align:middle line:84%
that the matrix has 3,000
possible value of 3,000 for deep

00:23:35.610 --> 00:23:43.530 align:middle line:84%
end learning and values of,
say, 50 for extreme learning--

00:23:43.530 --> 00:23:47.450 align:middle line:84%
under embedding comes in and
says 3,002 for the first one

00:23:47.450 --> 00:23:49.250 align:middle line:84%
and 48 for the second
one, we'll be like,

00:23:49.250 --> 00:23:51.545 align:middle line:90%
we'll be pretty impressed.

00:23:51.545 --> 00:23:54.050 align:middle line:84%
[INAUDIBLE] needed to be that
close unless it was actually

00:23:54.050 --> 00:23:55.930 align:middle line:90%
capturing something.

00:23:55.930 --> 00:23:57.530 align:middle line:84%
So that's what
we're going to do.

00:23:57.530 --> 00:24:00.250 align:middle line:84%
And so we're going to
take this logic of saying

00:24:00.250 --> 00:24:04.050 align:middle line:84%
find embeddings that can
approximate what we actually

00:24:04.050 --> 00:24:06.410 align:middle line:90%
see in Wikipedia.

00:24:06.410 --> 00:24:08.970 align:middle line:84%
And we're going to use that
idea to actually build a model

00:24:08.970 --> 00:24:12.370 align:middle line:84%
and learn from it using
nothing more than basically

00:24:12.370 --> 00:24:13.310 align:middle line:90%
linear regression.

00:24:13.310 --> 00:24:16.170 align:middle line:90%


00:24:16.170 --> 00:24:18.170 align:middle line:84%
And here, you are thinking
the linear regression

00:24:18.170 --> 00:24:20.100 align:middle line:84%
is useless now that you've
graduated to machine learning,

00:24:20.100 --> 00:24:20.600 align:middle line:90%
right?

00:24:20.600 --> 00:24:23.280 align:middle line:90%


00:24:23.280 --> 00:24:25.160 align:middle line:84%
So we can think of
the embedding vectors

00:24:25.160 --> 00:24:29.000 align:middle line:84%
that we want to figure out as
just the weights in a model

00:24:29.000 --> 00:24:31.360 align:middle line:90%
in a little regression.

00:24:31.360 --> 00:24:33.200 align:middle line:84%
We can think of the
co-occurrence matrix

00:24:33.200 --> 00:24:36.120 align:middle line:84%
as just the data we're
going to use in this model

00:24:36.120 --> 00:24:37.760 align:middle line:90%
to estimate these weights.

00:24:37.760 --> 00:24:42.240 align:middle line:84%
And the model we're going to
use is something like this.

00:24:42.240 --> 00:24:45.015 align:middle line:84%
So first, I have to inflict
some notation on you.

00:24:45.015 --> 00:24:46.640 align:middle line:84%
We will denote the
co-occurrence matrix

00:24:46.640 --> 00:24:50.000 align:middle line:90%
of, say, words I and J as Xij.

00:24:50.000 --> 00:24:51.720 align:middle line:90%
Xij is just data.

00:24:51.720 --> 00:24:52.900 align:middle line:90%
It's just data, OK?

00:24:52.900 --> 00:24:55.680 align:middle line:90%
It's not a variable, it's data.

00:24:55.680 --> 00:24:58.715 align:middle line:84%
And then we'll denote an
embedding vector for each word.

00:24:58.715 --> 00:25:00.840 align:middle line:84%
So we're going to calculate
a vector for each word.

00:25:00.840 --> 00:25:03.200 align:middle line:90%
So we'll call it wi.

00:25:03.200 --> 00:25:06.280 align:middle line:84%
wi is the embedding
vector for each word.

00:25:06.280 --> 00:25:09.920 align:middle line:84%
And we will also assume
that some words are just

00:25:09.920 --> 00:25:11.100 align:middle line:90%
inherently very popular.

00:25:11.100 --> 00:25:13.960 align:middle line:84%
They're going to show up all
the time, like the word the.

00:25:13.960 --> 00:25:17.160 align:middle line:84%
So we'll assume that every
word has some natural frequency

00:25:17.160 --> 00:25:21.830 align:middle line:84%
of occurring, like movie versus
flick, the versus tensor.

00:25:21.830 --> 00:25:23.910 align:middle line:84%
So we want the
vectors to capture

00:25:23.910 --> 00:25:27.310 align:middle line:84%
the co-occurrence patterns
independent of how naturally

00:25:27.310 --> 00:25:29.390 align:middle line:90%
frequent the words are.

00:25:29.390 --> 00:25:31.770 align:middle line:84%
And so to capture this
natural frequency,

00:25:31.770 --> 00:25:34.190 align:middle line:84%
we will assign a bias,
or bi, to each word

00:25:34.190 --> 00:25:35.530 align:middle line:90%
that we're going to calculate.

00:25:35.530 --> 00:25:38.222 align:middle line:84%
And all this will become
clear in just a moment.

00:25:38.222 --> 00:25:41.448 align:middle line:84%
So with this setup,
basically what we're saying

00:25:41.448 --> 00:25:42.490 align:middle line:90%
is something very simple.

00:25:42.490 --> 00:25:45.070 align:middle line:84%
We're saying, look, this
co-occurrence matrix that we

00:25:45.070 --> 00:25:47.510 align:middle line:84%
have that we're
able to compute--

00:25:47.510 --> 00:25:52.430 align:middle line:84%
it came about because in
truth, in reality, in nature,

00:25:52.430 --> 00:25:55.590 align:middle line:84%
there are these embedding
vectors for every word.

00:25:55.590 --> 00:25:58.110 align:middle line:84%
There are these biases,
bi, by for every word.

00:25:58.110 --> 00:26:01.150 align:middle line:84%
And every co-occurrence
number that you see just

00:26:01.150 --> 00:26:03.730 align:middle line:84%
came about because
under the hood,

00:26:03.730 --> 00:26:07.110 align:middle line:84%
mother nature grabbed the
bias number for the word

00:26:07.110 --> 00:26:09.350 align:middle line:84%
i, the bias number
for the word j,

00:26:09.350 --> 00:26:12.230 align:middle line:84%
took the two embedding vectors
which only mother nature knows

00:26:12.230 --> 00:26:15.050 align:middle line:84%
at this point, did the dot
product of them, added them,

00:26:15.050 --> 00:26:16.762 align:middle line:84%
and that's how we
get this number.

00:26:16.762 --> 00:26:18.220 align:middle line:84%
So it basically
says the number you

00:26:18.220 --> 00:26:22.440 align:middle line:84%
see is the sum of the inherent
popularity of the first word,

00:26:22.440 --> 00:26:24.540 align:middle line:84%
plus the inherent popularity
of the second word,

00:26:24.540 --> 00:26:29.060 align:middle line:84%
plus the way in which these two
words connect to each other.

00:26:29.060 --> 00:26:30.020 align:middle line:90%
That's it.

00:26:30.020 --> 00:26:32.940 align:middle line:84%
And you will agree with
me that it literally

00:26:32.940 --> 00:26:34.860 align:middle line:90%
can't get simpler than this.

00:26:34.860 --> 00:26:36.565 align:middle line:84%
If I tell you, hey,
here are two things.

00:26:36.565 --> 00:26:38.440 align:middle line:84%
I want you to tell me
how connected they are.

00:26:38.440 --> 00:26:40.760 align:middle line:84%
You'll be like, well,
let's take the first one.

00:26:40.760 --> 00:26:42.740 align:middle line:84%
Figure out how
inherently popular

00:26:42.740 --> 00:26:44.160 align:middle line:90%
it is, inherent popularity.

00:26:44.160 --> 00:26:46.452 align:middle line:84%
And then, of course, you to
worry about the connection,

00:26:46.452 --> 00:26:47.620 align:middle line:90%
so do a dot product.

00:26:47.620 --> 00:26:48.120 align:middle line:90%
That's it.

00:26:48.120 --> 00:26:50.380 align:middle line:90%
Those three things.

00:26:50.380 --> 00:26:51.700 align:middle line:90%
So this is what we have.

00:26:51.700 --> 00:26:58.060 align:middle line:84%
Now you may have seen from your
good old linear regression that

00:26:58.060 --> 00:27:03.020 align:middle line:84%
whenever your dependent variable
happens to be positive--

00:27:03.020 --> 00:27:05.140 align:middle line:90%
guaranteed to be positive--

00:27:05.140 --> 00:27:08.380 align:middle line:84%
and it ends up
having a big range,

00:27:08.380 --> 00:27:10.980 align:middle line:84%
we always advise
your folks to take

00:27:10.980 --> 00:27:14.580 align:middle line:84%
the logarithmic transformation
to squash it into a narrow range

00:27:14.580 --> 00:27:18.460 align:middle line:84%
because that will make these
models much more well-behaved.

00:27:18.460 --> 00:27:21.220 align:middle line:84%
Regression if the y value
is like a huge range.

00:27:21.220 --> 00:27:22.820 align:middle line:84%
The canonical example
is that if you

00:27:22.820 --> 00:27:26.520 align:middle line:84%
are trying to model the
net worth of people,

00:27:26.520 --> 00:27:29.060 align:middle line:84%
it's going to have a long right
tail with people like Elon

00:27:29.060 --> 00:27:31.500 align:middle line:84%
and Jeff and so on the right
side, and the rest of us

00:27:31.500 --> 00:27:32.280 align:middle line:90%
on the left.

00:27:32.280 --> 00:27:35.135 align:middle line:84%
So to model this big,
long tail distribution,

00:27:35.135 --> 00:27:37.260 align:middle line:84%
you just take the logarithm,
just squash everything

00:27:37.260 --> 00:27:38.480 align:middle line:90%
to a very narrow range.

00:27:38.480 --> 00:27:41.540 align:middle line:84%
And that will make regression
much better behaved.

00:27:41.540 --> 00:27:45.420 align:middle line:84%
Here, most of the counts
are going to be 0,

00:27:45.420 --> 00:27:49.220 align:middle line:84%
but some of the counts
could be very high.

00:27:49.220 --> 00:27:52.107 align:middle line:84%
And therefore, we wanted to--
if you take the logarithm,

00:27:52.107 --> 00:27:53.440 align:middle line:90%
it makes it much better behaved.

00:27:53.440 --> 00:27:54.860 align:middle line:90%
So take the logarithm here.

00:27:54.860 --> 00:27:56.400 align:middle line:90%
So this is actually our model.

00:27:56.400 --> 00:27:57.460 align:middle line:90%
That's it.

00:27:57.460 --> 00:27:59.780 align:middle line:84%
And I know that many of the
numbers are 0, and log of 0

00:27:59.780 --> 00:28:02.500 align:middle line:84%
is not defined, so we
can just add the number 1

00:28:02.500 --> 00:28:07.460 align:middle line:84%
to all the numbers to avoid that
kind of technical arithmetic

00:28:07.460 --> 00:28:07.968 align:middle line:90%
problems.

00:28:07.968 --> 00:28:09.760 align:middle line:84%
But this, conceptually,
is what's going on.

00:28:09.760 --> 00:28:11.500 align:middle line:84%
This is the model we
want to calculate.

00:28:11.500 --> 00:28:14.170 align:middle line:84%
So given that we
have essentially

00:28:14.170 --> 00:28:17.250 align:middle line:84%
postulated this
model and we have

00:28:17.250 --> 00:28:19.190 align:middle line:84%
this data, this
co-occurrence matrix,

00:28:19.190 --> 00:28:21.030 align:middle line:84%
how can we actually
find the weights?

00:28:21.030 --> 00:28:24.130 align:middle line:84%
How can we actually find
the b's and the w's?

00:28:24.130 --> 00:28:25.730 align:middle line:90%
What will what should we do?

00:28:25.730 --> 00:28:27.930 align:middle line:84%
Go back to the
fundamentals of regression.

00:28:27.930 --> 00:28:29.330 align:middle line:90%
Think about it conceptually.

00:28:29.330 --> 00:28:31.640 align:middle line:84%
You have some model,
which has some weights.

00:28:31.640 --> 00:28:35.530 align:middle line:84%
There's some data you can
use to train the model.

00:28:35.530 --> 00:28:37.510 align:middle line:84%
And you need to find
the best set of weights.

00:28:37.510 --> 00:28:40.330 align:middle line:90%
What does the best mean here?

00:28:40.330 --> 00:28:41.855 align:middle line:84%
AUDIENCE: Probably
the lowest error.

00:28:41.855 --> 00:28:43.730 align:middle line:84%
RAMA RAMAKRISHNAN: The
lowest error, exactly.

00:28:43.730 --> 00:28:46.323 align:middle line:84%
There are many ways
to measure error.

00:28:46.323 --> 00:28:47.990 align:middle line:84%
What is the simplest
thing we could use?

00:28:47.990 --> 00:28:49.740 align:middle line:84%
So what you would do
is you would actually

00:28:49.740 --> 00:28:52.050 align:middle line:84%
do mean squared error, which
is what you're getting at.

00:28:52.050 --> 00:28:53.250 align:middle line:84%
You could take the
actual thing, you

00:28:53.250 --> 00:28:55.375 align:middle line:84%
could take the predicted
thing, take the difference

00:28:55.375 --> 00:28:57.490 align:middle line:84%
and square it, and
minimize the sum of it.

00:28:57.490 --> 00:29:00.010 align:middle line:84%
If your model exactly
nails every number

00:29:00.010 --> 00:29:03.542 align:middle line:84%
in the co-occurrence matrix,
the rest is going to be 0.

00:29:03.542 --> 00:29:07.810 align:middle line:84%
So what we do is we
literally just do that.

00:29:07.810 --> 00:29:09.250 align:middle line:90%
This is the data.

00:29:09.250 --> 00:29:10.770 align:middle line:84%
This is the actual
predicted value.

00:29:10.770 --> 00:29:14.500 align:middle line:84%
Predicted value, actual
value, difference squared,

00:29:14.500 --> 00:29:17.920 align:middle line:90%
add them all up, minimize.

00:29:17.920 --> 00:29:19.400 align:middle line:90%
OK?

00:29:19.400 --> 00:29:21.280 align:middle line:90%
Yes?

00:29:21.280 --> 00:29:22.960 align:middle line:84%
AUDIENCE: I'm kind
of lost where how

00:29:22.960 --> 00:29:24.560 align:middle line:90%
is this capturing the context?

00:29:24.560 --> 00:29:28.760 align:middle line:84%
Because unless my input
data is having that context,

00:29:28.760 --> 00:29:32.200 align:middle line:84%
how will this actually
differentiate based on where

00:29:32.200 --> 00:29:34.320 align:middle line:90%
the particular word is used?

00:29:34.320 --> 00:29:37.080 align:middle line:84%
RAMA RAMAKRISHNAN:
The way the word is--

00:29:37.080 --> 00:29:39.400 align:middle line:84%
so let's take two words,
like deep and learning.

00:29:39.400 --> 00:29:41.200 align:middle line:84%
AUDIENCE: No,
let's take one word

00:29:41.200 --> 00:29:43.080 align:middle line:84%
and change it
according to the final.

00:29:43.080 --> 00:29:44.840 align:middle line:90%
RAMA RAMAKRISHNAN: OK.

00:29:44.840 --> 00:29:45.740 align:middle line:90%
Sorry, go ahead.

00:29:45.740 --> 00:29:48.120 align:middle line:84%
AUDIENCE: Yeah, so basically
let's say I have a banana.

00:29:48.120 --> 00:29:49.980 align:middle line:90%
So it's a fruit in some context.

00:29:49.980 --> 00:29:52.200 align:middle line:84%
And I could be saying,
he's going bananas.

00:29:52.200 --> 00:29:54.230 align:middle line:90%
So whatever, right?

00:29:54.230 --> 00:29:57.220 align:middle line:84%
So now these are two different
contexts in my understanding.

00:29:57.220 --> 00:29:59.560 align:middle line:84%
And my same model needs
to be able to tell me

00:29:59.560 --> 00:30:01.860 align:middle line:84%
that banana is the right
word in this context,

00:30:01.860 --> 00:30:05.400 align:middle line:84%
but wrong word in this context,
or correct in both contexts.

00:30:05.400 --> 00:30:06.400 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:30:06.400 --> 00:30:07.340 align:middle line:90%
Very good question.

00:30:07.340 --> 00:30:09.260 align:middle line:84%
So let's actually
spend a minute on that.

00:30:09.260 --> 00:30:10.110 align:middle line:90%
Good question.

00:30:10.110 --> 00:30:13.350 align:middle line:90%
I'm going to swap to my iPad.

00:30:13.350 --> 00:30:18.490 align:middle line:84%
So let's assume that this
is our co-occurrence matrix.

00:30:18.490 --> 00:30:20.830 align:middle line:84%
And then we have words
going from all the way

00:30:20.830 --> 00:30:23.150 align:middle line:90%
to-- let's say zebra.

00:30:23.150 --> 00:30:25.750 align:middle line:84%
These are all the words
in the vocabulary,

00:30:25.750 --> 00:30:29.430 align:middle line:84%
and we have a
through zebra here.

00:30:29.430 --> 00:30:37.000 align:middle line:84%
And now what we have is
we have apple and banana.

00:30:37.000 --> 00:30:40.336 align:middle line:90%


00:30:40.336 --> 00:30:42.790 align:middle line:84%
So basically what's
going on at this point

00:30:42.790 --> 00:30:49.070 align:middle line:84%
is that here, every number
here measures for every word

00:30:49.070 --> 00:30:51.630 align:middle line:84%
here, how many times that
word and apple show up

00:30:51.630 --> 00:30:53.260 align:middle line:90%
in the same sentence.

00:30:53.260 --> 00:30:57.390 align:middle line:84%
It is not measuring, to your
point, how many times apple

00:30:57.390 --> 00:30:58.570 align:middle line:90%
and banana are showing up.

00:30:58.570 --> 00:31:00.630 align:middle line:84%
It's measuring how
many times apple

00:31:00.630 --> 00:31:02.103 align:middle line:90%
is showing up in each sentence.

00:31:02.103 --> 00:31:06.430 align:middle line:84%
Now, if apple and banana
are interchangeable,

00:31:06.430 --> 00:31:11.760 align:middle line:84%
what do we expect these two
rows of numbers to look like?

00:31:11.760 --> 00:31:14.580 align:middle line:84%
Let's assume that apple and
banana are perfect synonyms,

00:31:14.580 --> 00:31:15.460 align:middle line:90%
just for argument.

00:31:15.460 --> 00:31:17.220 align:middle line:84%
Let's say it's a
perfect synonyms.

00:31:17.220 --> 00:31:24.020 align:middle line:84%
What do we expect these
two numbers to look like?

00:31:24.020 --> 00:31:25.540 align:middle line:90%
Very similar.

00:31:25.540 --> 00:31:30.120 align:middle line:84%
So if two words are related,
their entry row vectors,

00:31:30.120 --> 00:31:32.620 align:middle line:84%
the co-occurrence matrix are
going to be very, very similar.

00:31:32.620 --> 00:31:36.020 align:middle line:84%
So that is how the context comes
into the co-occurrence matrix.

00:31:36.020 --> 00:31:39.460 align:middle line:84%
So what we want to do is we
want to find if embeddings can

00:31:39.460 --> 00:31:44.180 align:middle line:84%
recreate the same pattern of
numbers in these two rows,

00:31:44.180 --> 00:31:47.900 align:middle line:84%
it's actually capturing
the underlying context.

00:31:47.900 --> 00:31:51.180 align:middle line:84%
So words which are similar
will zig and zag together

00:31:51.180 --> 00:31:53.340 align:middle line:84%
the same way through the
co-occurrence matrix,

00:31:53.340 --> 00:31:54.560 align:middle line:90%
and that's where it comes in.

00:31:54.560 --> 00:31:57.460 align:middle line:90%


00:31:57.460 --> 00:31:58.900 align:middle line:90%
yeah

00:31:58.900 --> 00:32:01.760 align:middle line:84%
AUDIENCE: What's on the diagonal
of the co-occurrence matrix?

00:32:01.760 --> 00:32:03.892 align:middle line:90%
Like, if apple shows up twice?

00:32:03.892 --> 00:32:05.100 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, I see.

00:32:05.100 --> 00:32:08.820 align:middle line:84%
So here, you can just ignore
the diagonal, typically,

00:32:08.820 --> 00:32:11.490 align:middle line:84%
because all the action
is off diagonal entries.

00:32:11.490 --> 00:32:15.340 align:middle line:90%


00:32:15.340 --> 00:32:17.820 align:middle line:90%
So that's basically the idea.

00:32:17.820 --> 00:32:20.180 align:middle line:84%
And if words which
are very similar

00:32:20.180 --> 00:32:23.020 align:middle line:84%
will have a very similar
pattern of numbers, and then

00:32:23.020 --> 00:32:25.780 align:middle line:84%
any embeddings that
can actually recreate

00:32:25.780 --> 00:32:28.500 align:middle line:84%
the same pattern of numbers is
capturing the underlying reality

00:32:28.500 --> 00:32:29.700 align:middle line:90%
of what's going on.

00:32:29.700 --> 00:32:34.000 align:middle line:84%
If words are unrelated,
those two vectors--

00:32:34.000 --> 00:32:36.080 align:middle line:84%
let's say that the
word you have is--

00:32:36.080 --> 00:32:40.420 align:middle line:90%


00:32:40.420 --> 00:32:42.628 align:middle line:90%
let's assume the word is--

00:32:42.628 --> 00:32:43.920 align:middle line:90%
you know what I'm going to say.

00:32:43.920 --> 00:32:46.300 align:middle line:90%
Tensor.

00:32:46.300 --> 00:32:50.440 align:middle line:84%
These two vectors won't have
any connection to each other.

00:32:50.440 --> 00:32:51.940 align:middle line:84%
Which means if you
look at something

00:32:51.940 --> 00:32:53.690 align:middle line:84%
like the correlation
of those two vectors,

00:32:53.690 --> 00:32:56.680 align:middle line:90%
it's going to be around zero.

00:32:56.680 --> 00:32:59.460 align:middle line:84%
But words which
are interchangeable

00:32:59.460 --> 00:33:01.620 align:middle line:84%
will have a very
high correlation.

00:33:01.620 --> 00:33:04.420 align:middle line:84%
Words which are antonyms and
never show up in the same place

00:33:04.420 --> 00:33:07.050 align:middle line:84%
together may have a highly
negative correlation,

00:33:07.050 --> 00:33:08.690 align:middle line:90%
close to minus 1, for instance.

00:33:08.690 --> 00:33:10.410 align:middle line:84%
So that's the
intuition behind what's

00:33:10.410 --> 00:33:12.890 align:middle line:90%
going on on these row vectors.

00:33:12.890 --> 00:33:15.530 align:middle line:84%
And so the point is given
this co-occurrence matrix is

00:33:15.530 --> 00:33:19.890 align:middle line:84%
capturing all these word-word
correlational structure,

00:33:19.890 --> 00:33:21.770 align:middle line:84%
any embedding that
can recreate it

00:33:21.770 --> 00:33:26.130 align:middle line:84%
must have captured the structure
as well because you can't

00:33:26.130 --> 00:33:28.410 align:middle line:84%
recreate something like
this with great fidelity

00:33:28.410 --> 00:33:31.850 align:middle line:84%
unless you have some notion of
what's going on under the hood.

00:33:31.850 --> 00:33:33.690 align:middle line:90%
That's the basic idea.

00:33:33.690 --> 00:33:34.435 align:middle line:90%
Yeah?

00:33:34.435 --> 00:33:35.810 align:middle line:84%
AUDIENCE: So just
connecting just

00:33:35.810 --> 00:33:39.210 align:middle line:84%
from this question-- so
in that example then,

00:33:39.210 --> 00:33:41.390 align:middle line:84%
banana is a fruit and
apple is a fruit as well.

00:33:41.390 --> 00:33:42.852 align:middle line:90%
Banana and apple are synonyms.

00:33:42.852 --> 00:33:44.810 align:middle line:84%
And you are going mad or
you are going bananas,

00:33:44.810 --> 00:33:45.830 align:middle line:90%
and that comes together?

00:33:45.830 --> 00:33:47.190 align:middle line:90%
Is that how we read it?

00:33:47.190 --> 00:33:47.490 align:middle line:90%
RAMA RAMAKRISHNAN: Oh, I see.

00:33:47.490 --> 00:33:48.030 align:middle line:90%
You're going mad.

00:33:48.030 --> 00:33:48.905 align:middle line:90%
You're going bananas.

00:33:48.905 --> 00:33:52.370 align:middle line:84%
Yeah, so those will also have
some correlation structure

00:33:52.370 --> 00:33:54.890 align:middle line:84%
to it, which the embeddings
will hopefully catch.

00:33:54.890 --> 00:33:59.530 align:middle line:84%
But words like banana,
which are very--

00:33:59.530 --> 00:34:01.050 align:middle line:84%
the thing is it's
called polysemy,

00:34:01.050 --> 00:34:03.110 align:middle line:90%
where the word looks one way.

00:34:03.110 --> 00:34:03.860 align:middle line:90%
It looks the same.

00:34:03.860 --> 00:34:05.220 align:middle line:90%
It's like the word bank, right?

00:34:05.220 --> 00:34:06.240 align:middle line:84%
It can mean very
different things

00:34:06.240 --> 00:34:07.365 align:middle line:90%
in many different contexts.

00:34:07.365 --> 00:34:10.040 align:middle line:84%
So the embedding is going to
be some average representation

00:34:10.040 --> 00:34:11.380 align:middle line:90%
of it.

00:34:11.380 --> 00:34:12.980 align:middle line:84%
But we're not happy
with that average,

00:34:12.980 --> 00:34:14.920 align:middle line:84%
and we will get
around that average

00:34:14.920 --> 00:34:18.239 align:middle line:84%
next week when we
do contextual stuff.

00:34:18.239 --> 00:34:20.320 align:middle line:90%
All right.

00:34:20.320 --> 00:34:21.920 align:middle line:90%
So that's what we have here.

00:34:21.920 --> 00:34:23.429 align:middle line:90%
So to go back to this thing--

00:34:23.429 --> 00:34:26.719 align:middle line:90%


00:34:26.719 --> 00:34:28.738 align:middle line:90%
so what we can do is-- yeah?

00:34:28.738 --> 00:34:30.280 align:middle line:84%
AUDIENCE: After
getting that initial,

00:34:30.280 --> 00:34:33.179 align:middle line:84%
how did we get the mean
square error in this?

00:34:33.179 --> 00:34:36.820 align:middle line:84%
Because we split the embedding
you put in the data and then we

00:34:36.820 --> 00:34:37.320 align:middle line:90%
get...

00:34:37.320 --> 00:34:39.653 align:middle line:84%
RAMA RAMAKRISHNAN: We haven't
calculated the embeddings.

00:34:39.653 --> 00:34:41.139 align:middle line:90%
We are trying to calculate them.

00:34:41.139 --> 00:34:43.199 align:middle line:84%
Those are just-- it's
like in regression, you

00:34:43.199 --> 00:34:47.960 align:middle line:84%
have beta 1 times x1 plus
beta 2 times x2 kind of thing.

00:34:47.960 --> 00:34:51.040 align:middle line:84%
The betas are what the
regression produces for us.

00:34:51.040 --> 00:34:52.699 align:middle line:90%
The embeddings are exactly that.

00:34:52.699 --> 00:34:55.280 align:middle line:84%
They're just coefficients that
we're trying to figure out.

00:34:55.280 --> 00:34:59.520 align:middle line:84%
The data is only
the x's, the Xij.

00:34:59.520 --> 00:35:02.670 align:middle line:84%
And so this is what we're
trying to calculate.

00:35:02.670 --> 00:35:04.710 align:middle line:84%
And so what you can
do is you can actually

00:35:04.710 --> 00:35:08.310 align:middle line:84%
start with some random
values for these things

00:35:08.310 --> 00:35:11.390 align:middle line:84%
and then keep on
trying to improve

00:35:11.390 --> 00:35:16.310 align:middle line:84%
to minimize the error starting
from these random values.

00:35:16.310 --> 00:35:18.470 align:middle line:84%
Are you aware of
any algorithm that

00:35:18.470 --> 00:35:20.390 align:middle line:84%
allows us to take random
values starting point

00:35:20.390 --> 00:35:22.090 align:middle line:84%
and then minimize
some notion of error?

00:35:22.090 --> 00:35:26.194 align:middle line:90%


00:35:26.194 --> 00:35:27.750 align:middle line:90%
AUDIENCE: Who takes the random?

00:35:27.750 --> 00:35:32.297 align:middle line:84%
How can we order it random
if it has to be random?

00:35:32.297 --> 00:35:34.630 align:middle line:84%
RAMA RAMAKRISHNAN: How do you
know it's actually random?

00:35:34.630 --> 00:35:35.590 align:middle line:90%
Oh.

00:35:35.590 --> 00:35:39.030 align:middle line:84%
So that's actually a
very deep question.

00:35:39.030 --> 00:35:43.430 align:middle line:84%
And so it's actually a tough
question because ultimately,

00:35:43.430 --> 00:35:46.030 align:middle line:84%
the random number is
coming from a computer.

00:35:46.030 --> 00:35:47.830 align:middle line:84%
And we know how
the computer runs.

00:35:47.830 --> 00:35:50.030 align:middle line:84%
It's deterministic,
at the end of the day.

00:35:50.030 --> 00:35:53.990 align:middle line:84%
So we actually use something
called pseudorandom numbers.

00:35:53.990 --> 00:35:56.930 align:middle line:84%
And there's a whole
specialized field of math

00:35:56.930 --> 00:35:59.430 align:middle line:84%
which essentially
says, look, how can I

00:35:59.430 --> 00:36:03.020 align:middle line:84%
get random numbers that are
sufficiently random, even

00:36:03.020 --> 00:36:04.900 align:middle line:84%
though they come from
a non-random computer

00:36:04.900 --> 00:36:06.300 align:middle line:90%
deterministic process?

00:36:06.300 --> 00:36:08.620 align:middle line:90%
So we can talk offline about it.

00:36:08.620 --> 00:36:10.500 align:middle line:84%
But fundamentally,
all these systems

00:36:10.500 --> 00:36:12.540 align:middle line:84%
have some random number
generators built in.

00:36:12.540 --> 00:36:14.940 align:middle line:84%
We just cross our fingers
and hope for the best

00:36:14.940 --> 00:36:17.420 align:middle line:90%
and just use them.

00:36:17.420 --> 00:36:19.640 align:middle line:90%
So come back to this.

00:36:19.640 --> 00:36:22.220 align:middle line:84%
We can start with random
values for these weights,

00:36:22.220 --> 00:36:24.600 align:middle line:84%
and then we can try to
minimize the squared error.

00:36:24.600 --> 00:36:26.100 align:middle line:84%
Are you folks aware
of any algorithm

00:36:26.100 --> 00:36:28.420 align:middle line:90%
that can help us do that?

00:36:28.420 --> 00:36:30.075 align:middle line:90%
Yes?

00:36:30.075 --> 00:36:31.200 align:middle line:90%
AUDIENCE: Gradient descent.

00:36:31.200 --> 00:36:33.060 align:middle line:84%
RAMA RAMAKRISHNAN:
Yes, gradient descent.

00:36:33.060 --> 00:36:34.980 align:middle line:90%
Again, comes to the rescue.

00:36:34.980 --> 00:36:39.460 align:middle line:84%
And since we are cool, we'll
do stochastic gradient descent.

00:36:39.460 --> 00:36:40.920 align:middle line:90%
So that's it.

00:36:40.920 --> 00:36:42.420 align:middle line:84%
So gradient descent
actually doesn't

00:36:42.420 --> 00:36:43.900 align:middle line:84%
care what the function
is, as long as you can

00:36:43.900 --> 00:36:45.240 align:middle line:90%
calculate a derivative from it.

00:36:45.240 --> 00:36:47.660 align:middle line:84%
As long as you calculate
the gradient, you're good.

00:36:47.660 --> 00:36:50.926 align:middle line:84%
So we can just run gradient
descent on this thing.

00:36:50.926 --> 00:36:54.420 align:middle line:84%
One key point here is that
stochastic gradient descent

00:36:54.420 --> 00:36:57.140 align:middle line:90%
work for any models.

00:36:57.140 --> 00:37:00.500 align:middle line:84%
As long as you can calculate
good gradients from them,

00:37:00.500 --> 00:37:04.220 align:middle line:84%
it doesn't have to
be a neural network.

00:37:04.220 --> 00:37:05.660 align:middle line:84%
Any mathematical
function, as long

00:37:05.660 --> 00:37:09.420 align:middle line:84%
as it's differentiable and
gives you a good gradient.

00:37:09.420 --> 00:37:11.540 align:middle line:84%
So here, this is not a
neural network, per se,

00:37:11.540 --> 00:37:14.180 align:middle line:84%
but we can use gradient
descent for it.

00:37:14.180 --> 00:37:19.720 align:middle line:84%
So we do that, and
when we are done,

00:37:19.720 --> 00:37:23.540 align:middle line:84%
we would have calculated
some nice embeddings.

00:37:23.540 --> 00:37:25.240 align:middle line:84%
We can also calculate
all these biases.

00:37:25.240 --> 00:37:26.360 align:middle line:84%
But we don't need
the biases anymore.

00:37:26.360 --> 00:37:28.360 align:middle line:84%
We can just throw out the
biases because we only

00:37:28.360 --> 00:37:31.700 align:middle line:84%
care about the embeddings and
how they connect to each other.

00:37:31.700 --> 00:37:33.420 align:middle line:90%
Yeah?

00:37:33.420 --> 00:37:35.340 align:middle line:84%
AUDIENCE: So when you're
doing the regression,

00:37:35.340 --> 00:37:37.217 align:middle line:84%
are you predicting the
co-occurrence matrix?

00:37:37.217 --> 00:37:38.300 align:middle line:90%
RAMA RAMAKRISHNAN: Mm-hmm.

00:37:38.300 --> 00:37:39.500 align:middle line:90%
AUDIENCE: OK.

00:37:39.500 --> 00:37:40.625 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:37:40.625 --> 00:37:42.380 align:middle line:90%


00:37:42.380 --> 00:37:47.460 align:middle line:84%
So actually, let me just show
a very quick numerical example

00:37:47.460 --> 00:37:48.180 align:middle line:90%
here.

00:37:48.180 --> 00:37:49.920 align:middle line:84%
So let's say, for
example, that--

00:37:49.920 --> 00:37:53.580 align:middle line:90%


00:37:53.580 --> 00:37:54.900 align:middle line:90%
you know what?

00:37:54.900 --> 00:37:57.170 align:middle line:90%
Deep learning.

00:37:57.170 --> 00:38:00.450 align:middle line:84%
So this is, say,
w1 and this is w2.

00:38:00.450 --> 00:38:01.830 align:middle line:90%
And this is the vector.

00:38:01.830 --> 00:38:06.410 align:middle line:84%
And let's assume for a moment
that it has two dimensions.

00:38:06.410 --> 00:38:07.930 align:middle line:90%
Two dimensions.

00:38:07.930 --> 00:38:10.750 align:middle line:84%
And we also need to calculate b1
and b2, which is just a number.

00:38:10.750 --> 00:38:14.710 align:middle line:90%


00:38:14.710 --> 00:38:17.290 align:middle line:84%
And let's say the
number for deep learning

00:38:17.290 --> 00:38:19.410 align:middle line:84%
in the co-occurrence
matrix, let's say

00:38:19.410 --> 00:38:21.730 align:middle line:90%
it has occurred 104 times.

00:38:21.730 --> 00:38:27.170 align:middle line:84%
So all we are doing
is to say log of 104--

00:38:27.170 --> 00:38:28.730 align:middle line:90%
that is the actual value--

00:38:28.730 --> 00:38:34.890 align:middle line:84%
minus b1, which we don't know,
plus B2, which we don't know.

00:38:34.890 --> 00:38:36.970 align:middle line:84%
And then this thing
here, let's just

00:38:36.970 --> 00:38:45.240 align:middle line:84%
call it w1 one, w1
two, w2 one, w2 two.

00:38:45.240 --> 00:38:46.990 align:middle line:84%
And then we're just
doing the dot product,

00:38:46.990 --> 00:38:55.840 align:middle line:84%
which is w1 one times w1
two plus w2 one, w2 two.

00:38:55.840 --> 00:38:58.645 align:middle line:90%
So this is our prediction.

00:38:58.645 --> 00:39:00.020 align:middle line:84%
Where is that cool
laser pointer?

00:39:00.020 --> 00:39:03.560 align:middle line:90%
Yeah, so this is our prediction.

00:39:03.560 --> 00:39:05.280 align:middle line:90%
This is the actual.

00:39:05.280 --> 00:39:08.320 align:middle line:84%
So all we do is to
say, OK, this thing--

00:39:08.320 --> 00:39:11.000 align:middle line:84%
the difference, we're
going to square it.

00:39:11.000 --> 00:39:13.040 align:middle line:84%
And then we're going to
do the same exact thing

00:39:13.040 --> 00:39:16.700 align:middle line:90%
for every other word pair.

00:39:16.700 --> 00:39:18.880 align:middle line:84%
And when we are done
with all of that thing,

00:39:18.880 --> 00:39:21.880 align:middle line:84%
we just take this whole thing
and say gradient descent,

00:39:21.880 --> 00:39:23.960 align:middle line:90%
minimize.

00:39:23.960 --> 00:39:27.120 align:middle line:84%
So then it has to find the
b's and the w's and everything

00:39:27.120 --> 00:39:29.360 align:middle line:90%
for every pair, every word.

00:39:29.360 --> 00:39:31.890 align:middle line:84%
So that's actually
what's going on.

00:39:31.890 --> 00:39:32.390 align:middle line:90%
Make sense?

00:39:32.390 --> 00:39:37.080 align:middle line:90%


00:39:37.080 --> 00:39:37.840 align:middle line:90%
All right.

00:39:37.840 --> 00:39:45.120 align:middle line:84%
So by the way, here,
I said let's assume

00:39:45.120 --> 00:39:51.200 align:middle line:84%
that the embeddings are just
vectors which are dimension two.

00:39:51.200 --> 00:39:53.590 align:middle line:84%
Well, that's an
arbitrary decision

00:39:53.590 --> 00:39:55.690 align:middle line:84%
that I made just to
show you how it works

00:39:55.690 --> 00:39:57.550 align:middle line:90%
because I was doing it by hand.

00:39:57.550 --> 00:39:59.350 align:middle line:84%
But more generally,
we get to choose

00:39:59.350 --> 00:40:02.070 align:middle line:90%
how long these vectors are.

00:40:02.070 --> 00:40:05.205 align:middle line:84%
And the longer the vector,
the more interesting ways

00:40:05.205 --> 00:40:07.330 align:middle line:84%
it can actually reproduce
the co-occurrence matrix.

00:40:07.330 --> 00:40:08.790 align:middle line:90%
It has more flexibility.

00:40:08.790 --> 00:40:13.350 align:middle line:84%
But the longer the vector,
what is the risk that you run?

00:40:13.350 --> 00:40:14.595 align:middle line:90%
Overfitting.

00:40:14.595 --> 00:40:15.970 align:middle line:84%
Because these are
all parameters,

00:40:15.970 --> 00:40:16.887 align:middle line:90%
at the end of the day.

00:40:16.887 --> 00:40:19.750 align:middle line:84%
More parameters you have,
the more risk of overfitting.

00:40:19.750 --> 00:40:22.710 align:middle line:84%
So you get to choose how
big these things can be.

00:40:22.710 --> 00:40:24.990 align:middle line:90%
Yes?

00:40:24.990 --> 00:40:26.710 align:middle line:84%
AUDIENCE: Don't you
find it surprising

00:40:26.710 --> 00:40:29.670 align:middle line:84%
that we're able to fit a
model where we have a lot

00:40:29.670 --> 00:40:31.170 align:middle line:84%
more parameters
than we have data?

00:40:31.170 --> 00:40:33.253 align:middle line:84%
Because usually when it
comes to machine learning,

00:40:33.253 --> 00:40:33.970 align:middle line:90%
we like sparsity.

00:40:33.970 --> 00:40:36.170 align:middle line:84%
We like to not have
a lot of parameters.

00:40:36.170 --> 00:40:38.530 align:middle line:84%
But here, we're going
to have, as you said,

00:40:38.530 --> 00:40:41.350 align:middle line:84%
the number of dimensions
times more parameters

00:40:41.350 --> 00:40:43.088 align:middle line:90%
than we have of data points.

00:40:43.088 --> 00:40:45.130 align:middle line:84%
RAMA RAMAKRISHNAN: Well,
in this particular case,

00:40:45.130 --> 00:40:46.870 align:middle line:90%
as it turns out--

00:40:46.870 --> 00:40:49.510 align:middle line:84%
let's assume that you
only have 10 words.

00:40:49.510 --> 00:40:53.020 align:middle line:84%
And for each word, let's
assume that you have--

00:40:53.020 --> 00:40:54.580 align:middle line:84%
just to keep the
math simple, you

00:40:54.580 --> 00:40:56.300 align:middle line:90%
have a two dimensional vector.

00:40:56.300 --> 00:40:58.540 align:middle line:90%
So 10 words times 2, that's 20.

00:40:58.540 --> 00:41:01.820 align:middle line:84%
Plus you have 10 biases for the
words, so there's another 10.

00:41:01.820 --> 00:41:02.860 align:middle line:90%
That's 30.

00:41:02.860 --> 00:41:06.020 align:middle line:84%
But 10 by 10, the
matrix has 100 entries.

00:41:06.020 --> 00:41:09.500 align:middle line:84%
So because of the matrix
being order n squared matrix,

00:41:09.500 --> 00:41:12.980 align:middle line:84%
you'll have a lot more
numbers than parameters.

00:41:12.980 --> 00:41:17.220 align:middle line:84%
In this particular case, you
have more data than parameters.

00:41:17.220 --> 00:41:20.020 align:middle line:84%
So that particular problem
doesn't apply in this case,

00:41:20.020 --> 00:41:22.040 align:middle line:84%
but that does show
up in other cases.

00:41:22.040 --> 00:41:24.420 align:middle line:84%
And there is some very
interesting research

00:41:24.420 --> 00:41:26.620 align:middle line:84%
in neural networks which
suggests that oftentimes,

00:41:26.620 --> 00:41:30.620 align:middle line:84%
the traditional assumptions of
data and overfitting and all

00:41:30.620 --> 00:41:33.780 align:middle line:84%
can all be called into
question under some situations.

00:41:33.780 --> 00:41:36.440 align:middle line:84%
I'm happy to tell you more
offline, but if you're curious,

00:41:36.440 --> 00:41:39.260 align:middle line:84%
just Google something
called double descent

00:41:39.260 --> 00:41:42.500 align:middle line:90%
to know what I mean.

00:41:42.500 --> 00:41:46.340 align:middle line:84%
But in this case,
it's not a problem.

00:41:46.340 --> 00:41:48.220 align:middle line:90%
OK.

00:41:48.220 --> 00:41:49.940 align:middle line:84%
So what that means
is that we can choose

00:41:49.940 --> 00:41:51.300 align:middle line:90%
how big these things are.

00:41:51.300 --> 00:41:56.362 align:middle line:84%
So if you look at one-hot
vectors, where there's a 1

00:41:56.362 --> 00:41:57.820 align:middle line:84%
and everything else
is 0, depending

00:41:57.820 --> 00:41:59.300 align:middle line:84%
on the position of
the word-- these

00:41:59.300 --> 00:42:03.580 align:middle line:84%
are vectors as long as the
vocabulary, as we saw earlier.

00:42:03.580 --> 00:42:07.420 align:middle line:84%
Word embeddings, on the other
hand, they can be very dense.

00:42:07.420 --> 00:42:09.722 align:middle line:84%
The numbers that make
up these embeddings,

00:42:09.722 --> 00:42:12.180 align:middle line:84%
we actually want to figure out
from the data what they are.

00:42:12.180 --> 00:42:14.120 align:middle line:90%
So it can be anything.

00:42:14.120 --> 00:42:17.180 align:middle line:84%
So the first dimension may
stand for some combination

00:42:17.180 --> 00:42:23.000 align:middle line:84%
of brightness plus speed
plus animalness or something.

00:42:23.000 --> 00:42:24.780 align:middle line:90%
We have no idea what it means.

00:42:24.780 --> 00:42:26.900 align:middle line:84%
All we know is that
it's able to reproduce

00:42:26.900 --> 00:42:29.260 align:middle line:84%
the co-occurrence matrix
really well, so it probably

00:42:29.260 --> 00:42:31.300 align:middle line:90%
has figured something out.

00:42:31.300 --> 00:42:32.960 align:middle line:84%
And so we can keep
it really short.

00:42:32.960 --> 00:42:36.160 align:middle line:84%
So the word embeddings
tend to be very dense,

00:42:36.160 --> 00:42:38.962 align:middle line:84%
meaning not zeros and ones,
but some arbitrary numbers.

00:42:38.962 --> 00:42:40.420 align:middle line:84%
It's very low
dimensional, and it's

00:42:40.420 --> 00:42:42.436 align:middle line:90%
of course learned from data.

00:42:42.436 --> 00:42:47.260 align:middle line:84%
So once you do this, once you
actually run GloVe on this data

00:42:47.260 --> 00:42:49.850 align:middle line:84%
and do gradient descent
and so on and so forth,

00:42:49.850 --> 00:42:51.410 align:middle line:84%
you actually come
up with embeddings.

00:42:51.410 --> 00:42:53.510 align:middle line:84%
And then you can actually
plot the embeddings.

00:42:53.510 --> 00:42:57.370 align:middle line:84%
You can take these embeddings
and just plot them.

00:42:57.370 --> 00:43:00.090 align:middle line:84%
Here, they are not literally
plotting the first two

00:43:00.090 --> 00:43:01.010 align:middle line:90%
dimensions.

00:43:01.010 --> 00:43:03.470 align:middle line:84%
They're using a particular
technique called t-SNE,

00:43:03.470 --> 00:43:06.410 align:middle line:84%
which is a way to take long
vectors and project them to 2D

00:43:06.410 --> 00:43:09.170 align:middle line:84%
space for
visualization purposes.

00:43:09.170 --> 00:43:12.325 align:middle line:84%
And you can see here, some
very interesting things

00:43:12.325 --> 00:43:12.950 align:middle line:90%
are showing up.

00:43:12.950 --> 00:43:15.530 align:middle line:84%
So they're basically,
they plotted the embedding

00:43:15.530 --> 00:43:17.090 align:middle line:90%
for brother--

00:43:17.090 --> 00:43:20.750 align:middle line:84%
nephew, uncle, sister, niece,
aunt and so on and so forth.

00:43:20.750 --> 00:43:22.210 align:middle line:90%
It's all showing up here.

00:43:22.210 --> 00:43:24.290 align:middle line:84%
This is the embedding
for man, embedding

00:43:24.290 --> 00:43:31.290 align:middle line:84%
for woman, sir madam, heiress,
heir, duke, emperor, king.

00:43:31.290 --> 00:43:33.090 align:middle line:90%
You get the idea.

00:43:33.090 --> 00:43:35.810 align:middle line:84%
So clearly, there are
patterns here where

00:43:35.810 --> 00:43:37.975 align:middle line:84%
things which are
similar in their nature

00:43:37.975 --> 00:43:39.850 align:middle line:84%
are all hanging out
together in the same part

00:43:39.850 --> 00:43:42.390 align:middle line:84%
of the space, which
is comforting,

00:43:42.390 --> 00:43:45.430 align:middle line:90%
which is good to know.

00:43:45.430 --> 00:43:47.920 align:middle line:84%
But as I mentioned earlier,
it's not just about the fact

00:43:47.920 --> 00:43:50.840 align:middle line:84%
that similar things happen
to be near each other.

00:43:50.840 --> 00:43:53.240 align:middle line:84%
The direction also
actually matters.

00:43:53.240 --> 00:43:55.920 align:middle line:84%
And beautiful things happen
when you look at directions.

00:43:55.920 --> 00:44:00.240 align:middle line:84%
So for instance, let's
say that you have man,

00:44:00.240 --> 00:44:03.600 align:middle line:84%
and you want to go
from man to brother.

00:44:03.600 --> 00:44:06.760 align:middle line:84%
So to go from man to brother,
you have to start with man

00:44:06.760 --> 00:44:11.440 align:middle line:84%
and then travel along this
arrow to get to brother.

00:44:11.440 --> 00:44:16.770 align:middle line:84%
So this arrow has some notion
of a person becoming a sibling.

00:44:16.770 --> 00:44:19.400 align:middle line:90%


00:44:19.400 --> 00:44:22.360 align:middle line:84%
So you would hope that if
you take that same arrow

00:44:22.360 --> 00:44:26.080 align:middle line:84%
and then start here with
that arrow, hopefully

00:44:26.080 --> 00:44:29.480 align:middle line:90%
the woman will become a sister.

00:44:29.480 --> 00:44:32.720 align:middle line:90%
Sure enough, it is.

00:44:32.720 --> 00:44:35.680 align:middle line:84%
So this is called
word vector algebra.

00:44:35.680 --> 00:44:36.800 align:middle line:90%
Embedding algebra.

00:44:36.800 --> 00:44:40.000 align:middle line:84%
And these relationships are
actually showing up in the data.

00:44:40.000 --> 00:44:42.000 align:middle line:84%
We didn't tell it
any of these things.

00:44:42.000 --> 00:44:44.920 align:middle line:84%
We just literally gave it
the co-occurrence matrix

00:44:44.920 --> 00:44:47.890 align:middle line:90%
and asked her to reproduce it.

00:44:47.890 --> 00:44:50.510 align:middle line:84%
So I find it pretty shocking
that these things are actually

00:44:50.510 --> 00:44:52.510 align:middle line:90%
true.

00:44:52.510 --> 00:44:55.150 align:middle line:84%
And it gives us
evidence and comfort

00:44:55.150 --> 00:44:57.110 align:middle line:84%
that whatever has
been learned does

00:44:57.110 --> 00:44:59.750 align:middle line:84%
have some deep connection
to describing the underlying

00:44:59.750 --> 00:45:02.030 align:middle line:90%
nature of what's going on.

00:45:02.030 --> 00:45:05.830 align:middle line:84%
It's not some statistically
fluky artifact.

00:45:05.830 --> 00:45:07.070 align:middle line:90%
Yeah?

00:45:07.070 --> 00:45:08.750 align:middle line:84%
AUDIENCE: So we
can find synonyms

00:45:08.750 --> 00:45:11.550 align:middle line:84%
by context or by
adjacency to other words,

00:45:11.550 --> 00:45:14.760 align:middle line:84%
and not by replacing
the same word, right?

00:45:14.760 --> 00:45:17.830 align:middle line:84%
Because they won't appear
in the same sentence

00:45:17.830 --> 00:45:19.488 align:middle line:90%
if you have two words--

00:45:19.488 --> 00:45:20.530 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:45:20.530 --> 00:45:22.155 align:middle line:84%
They won't appear in
the same sentence,

00:45:22.155 --> 00:45:25.270 align:middle line:84%
but the pattern of co-occurrence
will be the same for them,

00:45:25.270 --> 00:45:28.030 align:middle line:84%
which is what we've been able to
reproduce with these embeddings.

00:45:28.030 --> 00:45:28.988 align:middle line:90%
So that's the key idea.

00:45:28.988 --> 00:45:34.830 align:middle line:90%


00:45:34.830 --> 00:45:37.990 align:middle line:84%
AUDIENCE: So my question
is around how are we able

00:45:37.990 --> 00:45:42.470 align:middle line:84%
to capture all these directions
and in 2D matrix versus

00:45:42.470 --> 00:45:44.340 align:middle line:90%
a multi-dimensional matrix?

00:45:44.340 --> 00:45:46.780 align:middle line:84%
Because I feel like,
OK, so this relationship

00:45:46.780 --> 00:45:49.300 align:middle line:90%
is kind of concerned.

00:45:49.300 --> 00:45:53.218 align:middle line:84%
Yeah, I'm going to a family
or blood relationship

00:45:53.218 --> 00:45:54.260 align:middle line:90%
or something of the sort.

00:45:54.260 --> 00:45:57.743 align:middle line:84%
But how does it not mess up
the other side of that matrix?

00:45:57.743 --> 00:46:00.160 align:middle line:84%
RAMA RAMAKRISHNAN: No, this
is just a visualization thing.

00:46:00.160 --> 00:46:02.340 align:middle line:90%
So we're basically taking this--

00:46:02.340 --> 00:46:04.220 align:middle line:84%
as you will see,
GloVe embeddings

00:46:04.220 --> 00:46:06.020 align:middle line:90%
come in lots of different sizes.

00:46:06.020 --> 00:46:09.180 align:middle line:84%
And this, I think, uses the 100
dimension embedding and just

00:46:09.180 --> 00:46:12.180 align:middle line:84%
projects it to 2D space
using a particular technique,

00:46:12.180 --> 00:46:16.420 align:middle line:84%
and then looks to
see what's going on.

00:46:16.420 --> 00:46:18.123 align:middle line:90%
Yeah?

00:46:18.123 --> 00:46:20.540 align:middle line:84%
AUDIENCE: If the input data
being the co-occurrence matrix

00:46:20.540 --> 00:46:22.608 align:middle line:84%
is biased, are we
amplifying that bias?

00:46:22.608 --> 00:46:23.900 align:middle line:90%
RAMA RAMAKRISHNAN: Yes, we are.

00:46:23.900 --> 00:46:24.400 align:middle line:90%
Yes.

00:46:24.400 --> 00:46:26.100 align:middle line:90%
No, it's a great observation.

00:46:26.100 --> 00:46:28.660 align:middle line:84%
Any data you scrape from
the internet and use

00:46:28.660 --> 00:46:32.260 align:middle line:84%
for this modeling exercise will
be subject to all the biases

00:46:32.260 --> 00:46:34.540 align:middle line:84%
that produce the data
in the first place,

00:46:34.540 --> 00:46:37.540 align:middle line:84%
and the model will faithfully
learn those biases.

00:46:37.540 --> 00:46:40.380 align:middle line:84%
And if you're not careful,
it will perpetuate them.

00:46:40.380 --> 00:46:42.300 align:middle line:84%
And that's a whole
very important topic

00:46:42.300 --> 00:46:44.740 align:middle line:84%
that unfortunately we won't
cover in this course because

00:46:44.740 --> 00:46:45.720 align:middle line:90%
of time constraints.

00:46:45.720 --> 00:46:47.137 align:middle line:84%
But it's something
you always have

00:46:47.137 --> 00:46:50.217 align:middle line:84%
to worry about when you're
building these models.

00:46:50.217 --> 00:46:52.300 align:middle line:84%
AUDIENCE: How do we think
about the dimensionality

00:46:52.300 --> 00:46:55.340 align:middle line:84%
of the embeddings, not the 2D
representation of the actual

00:46:55.340 --> 00:46:55.840 align:middle line:90%
embedding?

00:46:55.840 --> 00:46:58.423 align:middle line:84%
RAMA RAMAKRISHNAN: The one that
we choose that's in our hands?

00:46:58.423 --> 00:47:00.580 align:middle line:84%
So you should think of
them as a hyperparameter.

00:47:00.580 --> 00:47:03.300 align:middle line:84%
So much like the
number of hidden units

00:47:03.300 --> 00:47:07.260 align:middle line:84%
to use in a particular hidden
layer, it's a hyperparameter.

00:47:07.260 --> 00:47:09.100 align:middle line:90%
I would, again, start small.

00:47:09.100 --> 00:47:10.900 align:middle line:84%
And if it solves the
problem that you're

00:47:10.900 --> 00:47:13.280 align:middle line:84%
trying to solve with
these embeddings, great.

00:47:13.280 --> 00:47:14.860 align:middle line:90%
If not, keep increasing them.

00:47:14.860 --> 00:47:16.660 align:middle line:84%
And at some point,
there might be

00:47:16.660 --> 00:47:19.530 align:middle line:84%
a flattening out and
an overfitting dynamic,

00:47:19.530 --> 00:47:20.280 align:middle line:90%
and then you stop.

00:47:20.280 --> 00:47:22.740 align:middle line:84%
So just think of them
as a hyperparameter.

00:47:22.740 --> 00:47:24.340 align:middle line:90%
Yeah?

00:47:24.340 --> 00:47:26.220 align:middle line:84%
AUDIENCE: Do you see
any benefit in practice

00:47:26.220 --> 00:47:30.420 align:middle line:84%
of using penalized regression
to do this instead of making

00:47:30.420 --> 00:47:33.500 align:middle line:84%
the embeddings more
sparse or just lowering

00:47:33.500 --> 00:47:34.700 align:middle line:90%
the magnitude of them?

00:47:34.700 --> 00:47:35.700 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:47:35.700 --> 00:47:36.900 align:middle line:90%
AUDIENCE: [INAUDIBLE]?

00:47:36.900 --> 00:47:37.858 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:47:37.858 --> 00:47:41.610 align:middle line:84%
So there are lots of techniques
to apply regularization

00:47:41.610 --> 00:47:44.930 align:middle line:84%
in the estimation itself
of all these numbers.

00:47:44.930 --> 00:47:46.570 align:middle line:90%
I'm happy to give you pointers.

00:47:46.570 --> 00:47:49.610 align:middle line:84%
I'm just going with the
simplest version possible.

00:47:49.610 --> 00:47:50.850 align:middle line:90%
Yeah?

00:47:50.850 --> 00:47:53.530 align:middle line:84%
AUDIENCE: I'm not
understanding why overfitting

00:47:53.530 --> 00:47:56.010 align:middle line:84%
is a problem in this case
because we're not doing

00:47:56.010 --> 00:47:59.050 align:middle line:90%
any out-of-sample predictions.

00:47:59.050 --> 00:48:03.210 align:middle line:84%
Wouldn't you want the embeddings
to be high dimension so you

00:48:03.210 --> 00:48:05.277 align:middle line:90%
capture unique relationships?

00:48:05.277 --> 00:48:07.110 align:middle line:84%
RAMA RAMAKRISHNAN: Ah,
interesting question.

00:48:07.110 --> 00:48:08.970 align:middle line:84%
So the question is,
given that there's

00:48:08.970 --> 00:48:12.650 align:middle line:84%
no notion of a out-of-sample
test set that we're

00:48:12.650 --> 00:48:14.270 align:middle line:84%
going to evaluate
these things on,

00:48:14.270 --> 00:48:16.130 align:middle line:84%
why do we really care
about overfitting?

00:48:16.130 --> 00:48:18.970 align:middle line:84%
Shouldn't we do the best we
can to capture everything

00:48:18.970 --> 00:48:20.602 align:middle line:90%
in the data?

00:48:20.602 --> 00:48:23.690 align:middle line:84%
Well, the thing is
even when you're not

00:48:23.690 --> 00:48:25.970 align:middle line:84%
trying to use it for
out-of-sample prediction,

00:48:25.970 --> 00:48:29.570 align:middle line:84%
you do want to make sure
that your model only

00:48:29.570 --> 00:48:32.770 align:middle line:84%
captures the true patterns
and not the noise.

00:48:32.770 --> 00:48:35.530 align:middle line:84%
In every data set,
there is always noise,

00:48:35.530 --> 00:48:38.880 align:middle line:84%
and you want it to capture
the signal but not the noise

00:48:38.880 --> 00:48:40.640 align:middle line:84%
regardless of what
you use it for.

00:48:40.640 --> 00:48:43.400 align:middle line:84%
Because if it captures the
noise, then the insights

00:48:43.400 --> 00:48:45.960 align:middle line:84%
you draw from the word
embeddings may be flawed.

00:48:45.960 --> 00:48:48.880 align:middle line:90%
That's the reason.

00:48:48.880 --> 00:48:50.040 align:middle line:90%
OK.

00:48:50.040 --> 00:48:51.340 align:middle line:90%
All right, so let's keep going.

00:48:51.340 --> 00:48:55.400 align:middle line:84%
So here, the algebra is brother
minus man plus woman is sister.

00:48:55.400 --> 00:48:55.940 align:middle line:90%
That's it.

00:48:55.940 --> 00:48:59.160 align:middle line:84%
Human biology reduced
to a single sentence.

00:48:59.160 --> 00:48:59.700 align:middle line:90%
All right.

00:48:59.700 --> 00:49:01.367 align:middle line:84%
So now the pros and
cons of these things

00:49:01.367 --> 00:49:04.280 align:middle line:84%
are you should use something
like a GloVe embedding

00:49:04.280 --> 00:49:09.960 align:middle line:84%
if you don't have enough data to
learn a task specific embedding

00:49:09.960 --> 00:49:11.160 align:middle line:90%
for your own vocabulary.

00:49:11.160 --> 00:49:12.720 align:middle line:84%
As I'll show you
in the Colab, you

00:49:12.720 --> 00:49:14.920 align:middle line:84%
can actually learn these things
just for your own data set,

00:49:14.920 --> 00:49:15.460 align:middle line:90%
if you want.

00:49:15.460 --> 00:49:17.840 align:middle line:84%
You don't have to use
these GloVe embeddings.

00:49:17.840 --> 00:49:20.000 align:middle line:84%
But the reason to use these
pre-trained embeddings

00:49:20.000 --> 00:49:22.360 align:middle line:84%
is that if you're working
with natural language,

00:49:22.360 --> 00:49:24.180 align:middle line:90%
the word is the word.

00:49:24.180 --> 00:49:25.560 align:middle line:90%
It means something.

00:49:25.560 --> 00:49:29.900 align:middle line:84%
And so there's no
reason for your model,

00:49:29.900 --> 00:49:33.200 align:middle line:84%
for your little use case for you
to actually somehow learn all

00:49:33.200 --> 00:49:35.030 align:middle line:90%
the fundamentals of English.

00:49:35.030 --> 00:49:37.530 align:middle line:84%
The fundamentals of English are
the fundamentals of English.

00:49:37.530 --> 00:49:40.510 align:middle line:84%
May as well learn it once
and then piggyback on it.

00:49:40.510 --> 00:49:43.990 align:middle line:84%
So that's the whole idea of
using pre-trained embeddings

00:49:43.990 --> 00:49:46.530 align:middle line:84%
because these things are all
common aspects of language.

00:49:46.530 --> 00:49:49.250 align:middle line:84%
May as well learn them using all
the data you can throw at it,

00:49:49.250 --> 00:49:51.510 align:middle line:84%
and then you can fine
tune and tweak and adapt

00:49:51.510 --> 00:49:54.890 align:middle line:90%
to your particular use case.

00:49:54.890 --> 00:49:56.390 align:middle line:84%
And this is
particularly useful when

00:49:56.390 --> 00:49:59.510 align:middle line:84%
you don't have a lot of data
in your particular use case.

00:49:59.510 --> 00:50:01.230 align:middle line:90%
That's one big advantage.

00:50:01.230 --> 00:50:03.910 align:middle line:84%
Now, it does have the drawback
that this embedding will not

00:50:03.910 --> 00:50:05.527 align:middle line:90%
be customized to your data.

00:50:05.527 --> 00:50:07.110 align:middle line:84%
For example, if
you're trying to build

00:50:07.110 --> 00:50:09.850 align:middle line:84%
an application for a
medical or legal use,

00:50:09.850 --> 00:50:12.313 align:middle line:84%
it's going to have
a lot of jargon.

00:50:12.313 --> 00:50:14.730 align:middle line:84%
And this pre-trained embedding
trained on all of Wikipedia

00:50:14.730 --> 00:50:16.630 align:middle line:84%
may not capture
enough of the jargon

00:50:16.630 --> 00:50:18.530 align:middle line:84%
and know its meaning
really accurately.

00:50:18.530 --> 00:50:19.590 align:middle line:84%
So what you want
to do is you want

00:50:19.590 --> 00:50:21.490 align:middle line:84%
to take this thing-- you may
still want to take this thing,

00:50:21.490 --> 00:50:22.990 align:middle line:84%
and then you can
adapt and fine tune

00:50:22.990 --> 00:50:29.050 align:middle line:84%
it using your jargon packed,
heavy, domain-specific data set.

00:50:29.050 --> 00:50:32.148 align:middle line:84%
Those are some of the
things to keep in mind.

00:50:32.148 --> 00:50:34.690 align:middle line:84%
And of course, we can also learn
it from scratch if you want.

00:50:34.690 --> 00:50:38.100 align:middle line:84%
And the Colab, I demonstrate
all these options.

00:50:38.100 --> 00:50:40.820 align:middle line:84%
So when you're working
with embeddings in Keras,

00:50:40.820 --> 00:50:44.700 align:middle line:84%
so what we do is
remember STIE, where

00:50:44.700 --> 00:50:46.260 align:middle line:84%
after we standardize
and tokenize

00:50:46.260 --> 00:50:50.480 align:middle line:84%
and index, at this point, we
go from integers to vectors.

00:50:50.480 --> 00:50:53.420 align:middle line:84%
And so far, we have been using
integers to one-hot vectors.

00:50:53.420 --> 00:50:55.180 align:middle line:84%
Here, we're going to
use embedding vectors

00:50:55.180 --> 00:50:56.805 align:middle line:84%
that we're going to
learn or that we're

00:50:56.805 --> 00:50:58.860 align:middle line:90%
going to use from GloVe.

00:50:58.860 --> 00:51:02.860 align:middle line:84%
And so what we do is we tell
Keras's text vectorization

00:51:02.860 --> 00:51:06.500 align:middle line:84%
layer to do only S,
T, and I. And then

00:51:06.500 --> 00:51:08.820 align:middle line:84%
we will use a new layer
called the embedding layer

00:51:08.820 --> 00:51:10.660 align:middle line:90%
to do the encoding.

00:51:10.660 --> 00:51:14.980 align:middle line:84%
That's how we're
going to divide it up.

00:51:14.980 --> 00:51:17.060 align:middle line:84%
So we'll take a
look at this first

00:51:17.060 --> 00:51:18.620 align:middle line:90%
before we switch to the Colab.

00:51:18.620 --> 00:51:22.820 align:middle line:84%
So before, we told
Keras in this layer,

00:51:22.820 --> 00:51:26.300 align:middle line:84%
output mode should be
multi-hot or whatever.

00:51:26.300 --> 00:51:28.020 align:middle line:84%
Here, we don't want
it to actually encode

00:51:28.020 --> 00:51:28.937 align:middle line:90%
anything in multi-hot.

00:51:28.937 --> 00:51:30.600 align:middle line:84%
We just want to give
it integers back.

00:51:30.600 --> 00:51:33.800 align:middle line:90%
So we tell it, give me int.

00:51:33.800 --> 00:51:35.180 align:middle line:90%
That's the first change.

00:51:35.180 --> 00:51:36.600 align:middle line:90%
We tell it, give us int.

00:51:36.600 --> 00:51:39.580 align:middle line:84%
If you say give us int,
it'll stop with STI

00:51:39.580 --> 00:51:41.580 align:middle line:90%
and just give you the integers.

00:51:41.580 --> 00:51:45.060 align:middle line:84%
And then what you do is that all
the incoming centers are going

00:51:45.060 --> 00:51:46.418 align:middle line:90%
to have different lengths.

00:51:46.418 --> 00:51:48.460 align:middle line:84%
So what we want to do is
we want to actually take

00:51:48.460 --> 00:51:50.440 align:middle line:84%
all these sentences
and normalize them

00:51:50.440 --> 00:51:55.780 align:middle line:84%
so they are of the
same length And the way

00:51:55.780 --> 00:51:59.220 align:middle line:84%
we do that very quickly
is that we choose

00:51:59.220 --> 00:52:02.480 align:middle line:84%
a maximum length
for the sentences,

00:52:02.480 --> 00:52:06.040 align:middle line:84%
and then if something is exactly
fits that length-- perfect.

00:52:06.040 --> 00:52:08.220 align:middle line:84%
Let's say in this case,
we want a max length of 5.

00:52:08.220 --> 00:52:09.760 align:middle line:84%
Cat sat on the mat
is exactly five.

00:52:09.760 --> 00:52:10.260 align:middle line:90%
Boom.

00:52:10.260 --> 00:52:11.072 align:middle line:90%
Fits perfectly.

00:52:11.072 --> 00:52:12.280 align:middle line:90%
But if something is smaller--

00:52:12.280 --> 00:52:14.030 align:middle line:84%
I love you is only
three of these things--

00:52:14.030 --> 00:52:17.866 align:middle line:84%
we actually pad it with
something called the pad token.

00:52:17.866 --> 00:52:20.620 align:middle line:84%
Also the unk token-- pad
token is a special token

00:52:20.620 --> 00:52:23.380 align:middle line:90%
which we use for padding.

00:52:23.380 --> 00:52:25.700 align:middle line:84%
And so in Keras, you
will see, will use zeros

00:52:25.700 --> 00:52:28.820 align:middle line:84%
for these paddings so that it
fills it up and gets all the way

00:52:28.820 --> 00:52:29.640 align:middle line:90%
to the end.

00:52:29.640 --> 00:52:32.395 align:middle line:84%
And if you have something
which is much longer than five,

00:52:32.395 --> 00:52:33.770 align:middle line:84%
you just truncate
everything else

00:52:33.770 --> 00:52:36.250 align:middle line:90%
and just use the first five.

00:52:36.250 --> 00:52:38.928 align:middle line:84%
So this is what we do to
get all the sentences to be

00:52:38.928 --> 00:52:39.720 align:middle line:90%
of the same length.

00:52:39.720 --> 00:52:43.250 align:middle line:90%


00:52:43.250 --> 00:52:47.250 align:middle line:84%
And once we do that, we then
go to the embedding layer.

00:52:47.250 --> 00:52:50.370 align:middle line:84%
And the embedding layer
is actually very simple.

00:52:50.370 --> 00:52:51.590 align:middle line:90%
What is an embedding?

00:52:51.590 --> 00:52:52.430 align:middle line:90%
It's just a vector.

00:52:52.430 --> 00:52:54.433 align:middle line:84%
And we need a vector
for every token.

00:52:54.433 --> 00:52:56.350 align:middle line:84%
Of course, we're going
to learn these vectors.

00:52:56.350 --> 00:52:57.750 align:middle line:90%
We need one for every token.

00:52:57.750 --> 00:52:59.730 align:middle line:84%
So in this case, for
example, let's say

00:52:59.730 --> 00:53:03.210 align:middle line:84%
that these are all the tokens
we have in our vocabulary

00:53:03.210 --> 00:53:05.030 align:middle line:90%
after the STI process.

00:53:05.030 --> 00:53:07.830 align:middle line:84%
Maybe in this case,
we have 5,000 tokens.

00:53:07.830 --> 00:53:10.077 align:middle line:84%
Each token, we have
this embedding vector,

00:53:10.077 --> 00:53:12.410 align:middle line:84%
and we choose what the dimension
of the embedding vector

00:53:12.410 --> 00:53:13.390 align:middle line:90%
is right.

00:53:13.390 --> 00:53:17.170 align:middle line:84%
And so we can set it up by
saying keras.layers.Embedding.

00:53:17.170 --> 00:53:19.270 align:middle line:84%
And we tell it max
tokens which means

00:53:19.270 --> 00:53:21.542 align:middle line:90%
how many rows do we have here?

00:53:21.542 --> 00:53:23.750 align:middle line:84%
What is the vocabulary size
that we are working with?

00:53:23.750 --> 00:53:26.490 align:middle line:84%
And then we tell it, OK,
this is how long I want

00:53:26.490 --> 00:53:28.550 align:middle line:90%
each embedding vector to be.

00:53:28.550 --> 00:53:31.740 align:middle line:84%
So rows, the size
of the columns,

00:53:31.740 --> 00:53:33.060 align:middle line:90%
and that's the embedding layer.

00:53:33.060 --> 00:53:34.100 align:middle line:90%
And we will use it in a second.

00:53:34.100 --> 00:53:35.517 align:middle line:84%
I just want to
show it to you here

00:53:35.517 --> 00:53:37.080 align:middle line:90%
because it's slightly clearer.

00:53:37.080 --> 00:53:38.640 align:middle line:84%
So when an input
sentence arrives,

00:53:38.640 --> 00:53:40.893 align:middle line:84%
the text vectorization
layer will run STI on it.

00:53:40.893 --> 00:53:43.060 align:middle line:84%
It will truncate and pad
it to max length as needed.

00:53:43.060 --> 00:53:45.120 align:middle line:84%
So let's say this
phrase comes in.

00:53:45.120 --> 00:53:48.240 align:middle line:84%
STI will give you the
same tokens plus pad pad

00:53:48.240 --> 00:53:50.100 align:middle line:84%
because let's say
the max length is 5,

00:53:50.100 --> 00:53:52.640 align:middle line:84%
and then these are the
corresponding integers.

00:53:52.640 --> 00:53:55.257 align:middle line:84%
And then the embedding
layer will just look up

00:53:55.257 --> 00:53:56.340 align:middle line:90%
the corresponding vectors.

00:53:56.340 --> 00:53:59.120 align:middle line:84%
So for example here,
the vectors are--

00:53:59.120 --> 00:54:02.360 align:middle line:84%
we need to look up the vectors
for 23, 9, 5, 0, and 0.

00:54:02.360 --> 00:54:06.640 align:middle line:84%
So we just go here and
look up 23, 5, 9, and 0.

00:54:06.640 --> 00:54:08.840 align:middle line:84%
And then once we
have that, boom,

00:54:08.840 --> 00:54:10.480 align:middle line:90%
this is the resulting output.

00:54:10.480 --> 00:54:12.240 align:middle line:84%
So whatever input
sentence comes in,

00:54:12.240 --> 00:54:15.120 align:middle line:84%
we have now five embedding
vectors that have been looked up

00:54:15.120 --> 00:54:17.520 align:middle line:90%
from the embedding layer.

00:54:17.520 --> 00:54:20.120 align:middle line:90%
And once we do that--

00:54:20.120 --> 00:54:21.300 align:middle line:90%
this is a table.

00:54:21.300 --> 00:54:22.420 align:middle line:90%
So I love you comes in.

00:54:22.420 --> 00:54:23.760 align:middle line:90%
It becomes this table.

00:54:23.760 --> 00:54:26.480 align:middle line:84%
As we have seen
before, neural networks

00:54:26.480 --> 00:54:29.410 align:middle line:84%
can only accommodate
vectors as inputs.

00:54:29.410 --> 00:54:31.670 align:middle line:84%
We need to make
this into a vector.

00:54:31.670 --> 00:54:34.630 align:middle line:84%
And as we have done before, we
can either take all these things

00:54:34.630 --> 00:54:37.010 align:middle line:84%
and concatenate them,
make a one long vector,

00:54:37.010 --> 00:54:39.350 align:middle line:84%
or we can find a way to
average them or sum them

00:54:39.350 --> 00:54:41.610 align:middle line:84%
and things like that,
as we have seen before.

00:54:41.610 --> 00:54:43.190 align:middle line:90%
And we will use the same--

00:54:43.190 --> 00:54:45.350 align:middle line:84%
the simplest thing is
probably just average them.

00:54:45.350 --> 00:54:49.830 align:middle line:84%
So these are some options,
but we will average them here.

00:54:49.830 --> 00:54:54.350 align:middle line:84%
And this is called the
GlobalAveragePooling layer, 1D.

00:54:54.350 --> 00:54:56.110 align:middle line:84%
And all it does is
whatever you give it--

00:54:56.110 --> 00:54:57.670 align:middle line:84%
a table you give
it, it just takes

00:54:57.670 --> 00:54:59.600 align:middle line:90%
each dimension and averages it.

00:54:59.600 --> 00:55:01.850 align:middle line:84%
The first dimension average,
second dimension average,

00:55:01.850 --> 00:55:02.930 align:middle line:90%
and so on and so forth.

00:55:02.930 --> 00:55:05.510 align:middle line:84%
And once that's done,
that's the whole flow.

00:55:05.510 --> 00:55:09.510 align:middle line:84%
So the phrase comes
in, STI gives you

00:55:09.510 --> 00:55:13.065 align:middle line:84%
these things, padding as
needed or truncating as needed.

00:55:13.065 --> 00:55:15.190 align:middle line:84%
We look up the embeddings
from the embedding layer,

00:55:15.190 --> 00:55:17.270 align:middle line:90%
and then we get all this thing.

00:55:17.270 --> 00:55:20.510 align:middle line:84%
We do global pooling
on it, and it's done.

00:55:20.510 --> 00:55:22.230 align:middle line:84%
The resulting thing
is a vector that

00:55:22.230 --> 00:55:24.970 align:middle line:84%
can then be passed into hidden
layers just like we normally do.

00:55:24.970 --> 00:55:27.580 align:middle line:90%


00:55:27.580 --> 00:55:29.240 align:middle line:84%
I'm going go with
this a little fast,

00:55:29.240 --> 00:55:31.220 align:middle line:84%
but make sure you
look at it afterwards

00:55:31.220 --> 00:55:33.140 align:middle line:84%
and understand every
step, and the Colab

00:55:33.140 --> 00:55:36.220 align:middle line:90%
will mirror this perfectly.

00:55:36.220 --> 00:55:39.620 align:middle line:84%
All right, so let's
switch to the Colab.

00:55:39.620 --> 00:55:40.340 align:middle line:90%
OK.

00:55:40.340 --> 00:55:41.500 align:middle line:90%
All right.

00:55:41.500 --> 00:55:43.780 align:middle line:90%
Can folks see this OK?

00:55:43.780 --> 00:55:44.280 align:middle line:90%
All right.

00:55:44.280 --> 00:55:45.197 align:middle line:90%
So we'll do the usual.

00:55:45.197 --> 00:55:47.740 align:middle line:90%


00:55:47.740 --> 00:55:49.300 align:middle line:90%
Import all the stuff you need.

00:55:49.300 --> 00:55:52.420 align:middle line:84%
And then because I want to plot
some of these loss and accuracy

00:55:52.420 --> 00:55:55.180 align:middle line:84%
curves just to see
what's going on,

00:55:55.180 --> 00:56:00.620 align:middle line:84%
I'll just bring in the functions
from the previous Colabs here.

00:56:00.620 --> 00:56:02.680 align:middle line:84%
And I think I already
have downloaded this.

00:56:02.680 --> 00:56:04.280 align:middle line:90%
Let me just make sure I have it.

00:56:04.280 --> 00:56:08.460 align:middle line:90%


00:56:08.460 --> 00:56:10.012 align:middle line:90%
It's not there.

00:56:10.012 --> 00:56:13.500 align:middle line:90%
So let's do it again.

00:56:13.500 --> 00:56:16.000 align:middle line:84%
This is the same songs data set
that we looked at on Monday.

00:56:16.000 --> 00:56:18.806 align:middle line:90%


00:56:18.806 --> 00:56:22.140 align:middle line:84%
So roughly 49,000
examples, as we saw before.

00:56:22.140 --> 00:56:25.580 align:middle line:90%
We'll one-hot encode them.

00:56:25.580 --> 00:56:26.080 align:middle line:90%
All right.

00:56:26.080 --> 00:56:28.747 align:middle line:84%
So there's a bunch of stuff that
we've already covered in class.

00:56:28.747 --> 00:56:30.620 align:middle line:90%
So this is the thing.

00:56:30.620 --> 00:56:34.880 align:middle line:84%
This URL has all the GloVe
vectors available for download.

00:56:34.880 --> 00:56:38.340 align:middle line:84%
I downloaded it before class
because it takes a few minutes.

00:56:38.340 --> 00:56:41.780 align:middle line:90%
And I've also-- did I unzip it?

00:56:41.780 --> 00:56:42.680 align:middle line:90%
Yes, I did.

00:56:42.680 --> 00:56:45.940 align:middle line:84%
So let's just look
at the first few.

00:56:45.940 --> 00:56:46.440 align:middle line:90%
All right.

00:56:46.440 --> 00:56:47.732 align:middle line:90%
So these are all the first few.

00:56:47.732 --> 00:56:50.770 align:middle line:84%
We'll create an easier to view
version of these GloVe vectors.

00:56:50.770 --> 00:56:54.860 align:middle line:90%


00:56:54.860 --> 00:56:57.880 align:middle line:84%
So I'm going to use the
vectors which are hundred long,

00:56:57.880 --> 00:56:59.940 align:middle line:84%
but it comes in many
different shapes.

00:56:59.940 --> 00:57:02.300 align:middle line:90%
So we have 400,000 vectors.

00:57:02.300 --> 00:57:06.020 align:middle line:84%
400,000 word vectors
each as 100 dimension.

00:57:06.020 --> 00:57:08.420 align:middle line:84%
And these all have been
calculated from Wikipedia

00:57:08.420 --> 00:57:11.820 align:middle line:84%
using the model we described
using gradient descent.

00:57:11.820 --> 00:57:13.140 align:middle line:90%
OK?

00:57:13.140 --> 00:57:13.640 align:middle line:90%
All right.

00:57:13.640 --> 00:57:18.153 align:middle line:84%
So this is the vector
for the word for movie.

00:57:18.153 --> 00:57:19.820 align:middle line:84%
I don't know what
these dimensions mean,

00:57:19.820 --> 00:57:21.840 align:middle line:90%
but there's something going on.

00:57:21.840 --> 00:57:23.970 align:middle line:90%
It has figured stuff out.

00:57:23.970 --> 00:57:25.550 align:middle line:90%
But the proof is in the pudding.

00:57:25.550 --> 00:57:28.170 align:middle line:84%
So all right, now we will first
set up the text vectorization

00:57:28.170 --> 00:57:30.930 align:middle line:84%
and embedding layers
like we saw before.

00:57:30.930 --> 00:57:36.950 align:middle line:84%
And so I'm going to use a max
length of 300 for the songs

00:57:36.950 --> 00:57:40.330 align:middle line:84%
because all the sentences
have to be the same length.

00:57:40.330 --> 00:57:42.890 align:middle line:84%
And you might be wondering, OK,
why did you pick 300 or not,

00:57:42.890 --> 00:57:44.210 align:middle line:90%
say, 400 or 200?

00:57:44.210 --> 00:57:47.370 align:middle line:84%
So typically, what you do is
you actually look at the length

00:57:47.370 --> 00:57:50.650 align:middle line:84%
distribution of the songs
you have, and you will find--

00:57:50.650 --> 00:57:53.550 align:middle line:84%
you're looking for an 80-20
or one of those things.

00:57:53.550 --> 00:57:56.330 align:middle line:84%
And in this case, it turns out
90% of the songs have less than

00:57:56.330 --> 00:57:58.910 align:middle line:84%
or equal to 300 words
in our data set.

00:57:58.910 --> 00:58:00.670 align:middle line:84%
So I'm just going
to go with 300.

00:58:00.670 --> 00:58:02.370 align:middle line:90%
It's pretty good.

00:58:02.370 --> 00:58:03.910 align:middle line:84%
The problem is if
you actually say--

00:58:03.910 --> 00:58:06.850 align:middle line:84%
if you look at the song
which has the maximum length,

00:58:06.850 --> 00:58:10.370 align:middle line:84%
that might be like 3,000 words,
and there might be hardly any

00:58:10.370 --> 00:58:11.350 align:middle line:90%
songs of 3,000 long.

00:58:11.350 --> 00:58:13.630 align:middle line:84%
You're just wasting a lot
of capacity by doing that.

00:58:13.630 --> 00:58:16.610 align:middle line:84%
So you're just being a
little pragmatic here.

00:58:16.610 --> 00:58:19.870 align:middle line:84%
OK, and then as before,
for the vocabulary itself,

00:58:19.870 --> 00:58:23.120 align:middle line:84%
we tell Keras use the most
frequent 5,000 words when

00:58:23.120 --> 00:58:25.840 align:middle line:90%
you're doing the STI.

00:58:25.840 --> 00:58:29.000 align:middle line:84%
So we do that, and we tell
it the output mode is int,

00:58:29.000 --> 00:58:32.520 align:middle line:90%
like we saw before.

00:58:32.520 --> 00:58:35.280 align:middle line:90%
We have there.

00:58:35.280 --> 00:58:36.880 align:middle line:90%
OK, perfect.

00:58:36.880 --> 00:58:39.560 align:middle line:84%
OK, this is a very
dangerous thing,

00:58:39.560 --> 00:58:41.960 align:middle line:84%
where somebody is
remotely changing it

00:58:41.960 --> 00:58:44.640 align:middle line:90%
in another tab somewhere.

00:58:44.640 --> 00:58:48.400 align:middle line:90%
Fingers crossed, OK?

00:58:48.400 --> 00:58:50.240 align:middle line:90%
Take this one.

00:58:50.240 --> 00:58:50.760 align:middle line:90%
OK.

00:58:50.760 --> 00:58:54.200 align:middle line:84%
So we have this, and this is
what we did with all this stuff,

00:58:54.200 --> 00:58:55.300 align:middle line:90%
as I've covered.

00:58:55.300 --> 00:58:59.560 align:middle line:84%
So now we will adapt this
layer, as we have seen before,

00:58:59.560 --> 00:59:00.890 align:middle line:90%
using all the lyrics we have.

00:59:00.890 --> 00:59:04.440 align:middle line:90%


00:59:04.440 --> 00:59:08.360 align:middle line:84%
Then once we do that, we'll
take a look at the first few.

00:59:08.360 --> 00:59:10.180 align:middle line:84%
And so here's a very
important thing.

00:59:10.180 --> 00:59:12.760 align:middle line:84%
Before-- when we asked it
to do multi-hot encoding

00:59:12.760 --> 00:59:14.840 align:middle line:90%
and so on, on Monday--

00:59:14.840 --> 00:59:18.020 align:middle line:84%
the zero, the first
position was unk.

00:59:18.020 --> 00:59:19.000 align:middle line:90%
Unk had zero.

00:59:19.000 --> 00:59:21.910 align:middle line:90%
But here, unk actually has one.

00:59:21.910 --> 00:59:24.590 align:middle line:84%
And the reason is that
the zeroth position

00:59:24.590 --> 00:59:28.230 align:middle line:84%
is going to be used for
essentially-- you can think

00:59:28.230 --> 00:59:29.690 align:middle line:90%
of this as the empty string.

00:59:29.690 --> 00:59:32.870 align:middle line:84%
That's how Keras
will print out pad.

00:59:32.870 --> 00:59:36.170 align:middle line:84%
So the zeroth position is
the padding, the pad token.

00:59:36.170 --> 00:59:39.030 align:middle line:84%
The first position
is the unk token.

00:59:39.030 --> 00:59:41.470 align:middle line:90%
So it's an important thing here.

00:59:41.470 --> 00:59:46.910 align:middle line:84%
So let's say that we do
HODL, you're the best.

00:59:46.910 --> 00:59:49.590 align:middle line:90%
We vectorize it.

00:59:49.590 --> 00:59:53.030 align:middle line:84%
Do you think HODL is going to
be part of those 400,000 word

00:59:53.030 --> 00:59:55.070 align:middle line:90%
vectors [INAUDIBLE]?

00:59:55.070 --> 00:59:57.790 align:middle line:90%
Not yet.

00:59:57.790 --> 00:59:59.650 align:middle line:90%
All right, so let's try that.

00:59:59.650 --> 01:00:03.510 align:middle line:90%


01:00:03.510 --> 01:00:07.990 align:middle line:84%
And as you can tell,
HODL is an unknown word.

01:00:07.990 --> 01:00:12.790 align:middle line:90%
That's why it's showing up here.

01:00:12.790 --> 01:00:14.690 align:middle line:90%
So 1 is unknown.

01:00:14.690 --> 01:00:16.430 align:middle line:90%
The index value 1 is unknown.

01:00:16.430 --> 01:00:18.110 align:middle line:90%
0 is pad.

01:00:18.110 --> 01:00:21.980 align:middle line:84%
But then this is
unknown HODL, I--

01:00:21.980 --> 01:00:24.713 align:middle line:90%
sorry, you are the best.

01:00:24.713 --> 01:00:26.380 align:middle line:84%
And then everything
else from that point

01:00:26.380 --> 01:00:30.600 align:middle line:84%
on is a 0 because we are
padding all the way to 300.

01:00:30.600 --> 01:00:32.580 align:middle line:84%
So that's why you see
all these zeros here.

01:00:32.580 --> 01:00:35.180 align:middle line:84%
All right, now let's
just run everything

01:00:35.180 --> 01:00:38.860 align:middle line:84%
through the vectorization
layer, and then we'll

01:00:38.860 --> 01:00:40.160 align:middle line:90%
get to the embedding layer.

01:00:40.160 --> 01:00:44.340 align:middle line:90%


01:00:44.340 --> 01:00:44.840 align:middle line:90%
OK.

01:00:44.840 --> 01:00:48.580 align:middle line:90%
Now we will first--

01:00:48.580 --> 01:00:52.300 align:middle line:84%
just a bit of
Python housekeeping

01:00:52.300 --> 01:00:55.308 align:middle line:84%
to create a nice, easy
to look at matrix.

01:00:55.308 --> 01:00:57.100 align:middle line:84%
So what we're going to
do is we're actually

01:00:57.100 --> 01:01:01.420 align:middle line:84%
going to create a nice matrix
that shows us all the GloVe

01:01:01.420 --> 01:01:04.100 align:middle line:90%
embeddings.

01:01:04.100 --> 01:01:07.140 align:middle line:84%
And so this is the
embedding matrix,

01:01:07.140 --> 01:01:11.700 align:middle line:84%
and this matrix has only 5,000
words, and each is 100 long.

01:01:11.700 --> 01:01:13.700 align:middle line:84%
Why is this embedding
matrix only 5,000,

01:01:13.700 --> 01:01:21.940 align:middle line:84%
even though we downloaded
400,000 vectors?

01:01:21.940 --> 01:01:24.640 align:middle line:84%
So clearly, the 5,000 we used,
there has some bearing to this.

01:01:24.640 --> 01:01:30.660 align:middle line:90%
But what is the 5,000?

01:01:30.660 --> 01:01:33.660 align:middle line:84%
We told Keras to take the
most frequent 5,000 words

01:01:33.660 --> 01:01:37.040 align:middle line:84%
in our corpus so we'll only
have 5,000 in vocabulary.

01:01:37.040 --> 01:01:38.360 align:middle line:90%
That's why there's 5,000.

01:01:38.360 --> 01:01:41.820 align:middle line:84%
So we grab just the GloVe
vectors for those 5,000 that

01:01:41.820 --> 01:01:43.660 align:middle line:84%
Keras has chosen to
be in the vocabulary,

01:01:43.660 --> 01:01:45.580 align:middle line:90%
and that's our embedding matrix.

01:01:45.580 --> 01:01:48.580 align:middle line:84%
And then if you look
at the first few rows,

01:01:48.580 --> 01:01:50.780 align:middle line:84%
the first two rows
should be all zeros

01:01:50.780 --> 01:01:53.968 align:middle line:84%
because it's pad and
unk, which clearly, GloVe

01:01:53.968 --> 01:01:54.760 align:middle line:90%
doesn't know about.

01:01:54.760 --> 01:01:56.180 align:middle line:84%
So it's all going
to be all zeros.

01:01:56.180 --> 01:01:59.000 align:middle line:84%
And so you can see
all these zeros here.

01:01:59.000 --> 01:02:01.700 align:middle line:84%
And then from the third onwards,
you start getting some numbers.

01:02:01.700 --> 01:02:02.620 align:middle line:90%
OK?

01:02:02.620 --> 01:02:03.460 align:middle line:90%
All right.

01:02:03.460 --> 01:02:06.203 align:middle line:84%
Next, we'll set up
the embedding layer.

01:02:06.203 --> 01:02:07.620 align:middle line:84%
So basically,
what's going on here

01:02:07.620 --> 01:02:10.600 align:middle line:84%
is we tell the embedding
layer how many rows,

01:02:10.600 --> 01:02:12.900 align:middle line:84%
which is just the
vocab size max tokens.

01:02:12.900 --> 01:02:14.473 align:middle line:90%
What is the embedding dimension?

01:02:14.473 --> 01:02:16.890 align:middle line:84%
Well, that's going to be 100
because the GloVe vectors are

01:02:16.890 --> 01:02:17.590 align:middle line:90%
hundred.

01:02:17.590 --> 01:02:18.770 align:middle line:90%
And then here's the thing.

01:02:18.770 --> 01:02:21.690 align:middle line:84%
You can tell it in
this embedding layer,

01:02:21.690 --> 01:02:24.597 align:middle line:84%
just use this matrix I'm giving
you as the embedding layer

01:02:24.597 --> 01:02:26.430 align:middle line:84%
because we already know
what embeddings are.

01:02:26.430 --> 01:02:28.260 align:middle line:84%
We downloaded it
from whatever GloVe.

01:02:28.260 --> 01:02:31.030 align:middle line:84%
So we will tell it to
use GloVe as the weights

01:02:31.030 --> 01:02:32.670 align:middle line:84%
for here, as the
embeddings here.

01:02:32.670 --> 01:02:36.120 align:middle line:84%
So we initialize it using
that embedding matrix.

01:02:36.120 --> 01:02:40.675 align:middle line:84%
And then we tell it, when we
do back propagation later on,

01:02:40.675 --> 01:02:42.050 align:middle line:84%
don't change any
of these weights

01:02:42.050 --> 01:02:44.170 align:middle line:84%
because somebody
spent a lot of money

01:02:44.170 --> 01:02:45.850 align:middle line:90%
to create this weights for us--

01:02:45.850 --> 01:02:46.730 align:middle line:90%
Stanford.

01:02:46.730 --> 01:02:49.050 align:middle line:84%
So we don't want to further
change them, just freeze

01:02:49.050 --> 01:02:51.042 align:middle line:90%
them and use them as they are.

01:02:51.042 --> 01:02:53.230 align:middle line:84%
And this mask zero business,
I'll come back later.

01:02:53.230 --> 01:02:55.190 align:middle line:84%
Don't worry about
it for the moment.

01:02:55.190 --> 01:02:55.950 align:middle line:90%
All right.

01:02:55.950 --> 01:02:59.910 align:middle line:84%
So once we do that, we are
ready to set up our model.

01:02:59.910 --> 01:03:01.850 align:middle line:90%
So this model is pretty simple.

01:03:01.850 --> 01:03:02.870 align:middle line:90%
Keras input.

01:03:02.870 --> 01:03:04.370 align:middle line:84%
The length, of
course, is the length

01:03:04.370 --> 01:03:07.090 align:middle line:84%
of the sentence,
which is 300 long.

01:03:07.090 --> 01:03:10.830 align:middle line:84%
And then the input runs through
an embedding layer right there,

01:03:10.830 --> 01:03:13.170 align:middle line:84%
and out comes a
300 by 100 table.

01:03:13.170 --> 01:03:17.860 align:middle line:84%
And then we average pool it, and
that becomes 100 element vector.

01:03:17.860 --> 01:03:19.940 align:middle line:84%
And then we are back
in familiar ground.

01:03:19.940 --> 01:03:23.240 align:middle line:84%
And we run it through a dense
layer with eight ReLU neurons.

01:03:23.240 --> 01:03:24.888 align:middle line:90%
Eight ReLU neurons.

01:03:24.888 --> 01:03:26.680 align:middle line:84%
And then we run it
through the final output

01:03:26.680 --> 01:03:28.240 align:middle line:84%
layer, which is a
three way softmax

01:03:28.240 --> 01:03:29.960 align:middle line:90%
as before-- hip hop, rock, pop.

01:03:29.960 --> 01:03:32.040 align:middle line:84%
And then we tell Keras
that's the model,

01:03:32.040 --> 01:03:34.760 align:middle line:90%
and then we summarize it.

01:03:34.760 --> 01:03:36.520 align:middle line:90%
OK, so that's what we have.

01:03:36.520 --> 01:03:41.000 align:middle line:84%
And you can see here, the
total parameters are 500,835,

01:03:41.000 --> 01:03:44.560 align:middle line:84%
but the trainable
parameters are only 835.

01:03:44.560 --> 01:03:46.800 align:middle line:84%
It's because the total
parameters are all the GloVe

01:03:46.800 --> 01:03:50.080 align:middle line:84%
embeddings plus
the things we added

01:03:50.080 --> 01:03:52.840 align:middle line:84%
to the GloVe embeddings like
the hidden layer and so on.

01:03:52.840 --> 01:03:54.560 align:middle line:90%
But the GloVe embeddings are--

01:03:54.560 --> 01:03:56.060 align:middle line:90%
we have told Keras, freeze it.

01:03:56.060 --> 01:03:57.565 align:middle line:90%
Do not train it.

01:03:57.565 --> 01:03:59.940 align:middle line:84%
Which means only the rest of
it is going to be trainable.

01:03:59.940 --> 01:04:01.140 align:middle line:90%
That's the 835.

01:04:01.140 --> 01:04:03.495 align:middle line:90%
Yeah?

01:04:03.495 --> 01:04:05.620 align:middle line:84%
AUDIENCE: So when we do
the global average pooling,

01:04:05.620 --> 01:04:08.200 align:middle line:84%
don't we lose any
sense of meaning

01:04:08.200 --> 01:04:12.430 align:middle line:84%
that we gain from the
embedding, as we average very

01:04:12.430 --> 01:04:13.742 align:middle line:90%
different embeddings together?

01:04:13.742 --> 01:04:15.450 align:middle line:84%
RAMA RAMAKRISHNAN:
Sorry, say that again.

01:04:15.450 --> 01:04:16.450 align:middle line:90%
I missed the first part.

01:04:16.450 --> 01:04:19.330 align:middle line:84%
AUDIENCE: If we average the
embedding of apple and learning,

01:04:19.330 --> 01:04:21.550 align:middle line:84%
for instance, you have
very different words that

01:04:21.550 --> 01:04:23.130 align:middle line:90%
are used in different meanings.

01:04:23.130 --> 01:04:25.650 align:middle line:84%
So we have different
embeddings, but we average it

01:04:25.650 --> 01:04:27.410 align:middle line:90%
so we can't lose this--

01:04:27.410 --> 01:04:28.570 align:middle line:84%
RAMA RAMAKRISHNAN: We will
lose a bunch of stuff.

01:04:28.570 --> 01:04:29.278 align:middle line:90%
Yeah, yeah, yeah.

01:04:29.278 --> 01:04:30.950 align:middle line:84%
So yeah, anytime you
average anything,

01:04:30.950 --> 01:04:33.410 align:middle line:84%
you're going to lose
some nuance and so on.

01:04:33.410 --> 01:04:36.610 align:middle line:84%
So the real question is
despite that averaging,

01:04:36.610 --> 01:04:37.790 align:middle line:90%
is it good enough for you?

01:04:37.790 --> 01:04:39.230 align:middle line:90%
And sometimes it's good enough.

01:04:39.230 --> 01:04:41.357 align:middle line:84%
Very often, it's good
enough, as it turns out.

01:04:41.357 --> 01:04:43.690 align:middle line:84%
But as you will see when we
go to contextual embeddings,

01:04:43.690 --> 01:04:45.610 align:middle line:84%
there's just a
better way to do it,

01:04:45.610 --> 01:04:47.230 align:middle line:84%
when you have
contextual embeddings.

01:04:47.230 --> 01:04:49.610 align:middle line:84%
But it requires bigger
models, more powerful stuff,

01:04:49.610 --> 01:04:50.610 align:middle line:90%
and so on and so forth.

01:04:50.610 --> 01:04:52.693 align:middle line:84%
And that's where you're
going from the foundations

01:04:52.693 --> 01:04:53.950 align:middle line:90%
to the advanced stuff.

01:04:53.950 --> 01:04:56.230 align:middle line:90%
Yeah?

01:04:56.230 --> 01:04:58.270 align:middle line:84%
AUDIENCE: When we're
doing optimization-- like,

01:04:58.270 --> 01:05:00.310 align:middle line:84%
let's say we are
having a word problem--

01:05:00.310 --> 01:05:02.990 align:middle line:84%
it's often best to
optimize everything

01:05:02.990 --> 01:05:05.490 align:middle line:84%
together than to optimize
one part of the system

01:05:05.490 --> 01:05:07.190 align:middle line:84%
and then optimize
the response system.

01:05:07.190 --> 01:05:09.870 align:middle line:84%
So in that case,
why wouldn't we want

01:05:09.870 --> 01:05:12.960 align:middle line:90%
to also change the embeddings?

01:05:12.960 --> 01:05:16.820 align:middle line:84%
I understand why you would like
to start with those weights

01:05:16.820 --> 01:05:19.960 align:middle line:84%
that some people have spent a
lot of money trying to find,

01:05:19.960 --> 01:05:24.180 align:middle line:84%
but will we be able to find
more specific embeddings related

01:05:24.180 --> 01:05:27.330 align:middle line:84%
to our problem if we let
everything be trainable?

01:05:27.330 --> 01:05:29.080 align:middle line:84%
RAMA RAMAKRISHNAN:
Absolutely, absolutely.

01:05:29.080 --> 01:05:33.040 align:middle line:84%
And in fact, you will see in the
Colab that we will do that next.

01:05:33.040 --> 01:05:35.508 align:middle line:84%
I just want to show people
you don't have to do it.

01:05:35.508 --> 01:05:37.300 align:middle line:84%
You start with not
training it because it's

01:05:37.300 --> 01:05:39.550 align:middle line:84%
going to be much faster, and
then you train everything

01:05:39.550 --> 01:05:40.640 align:middle line:90%
and see if it gets better.

01:05:40.640 --> 01:05:41.900 align:middle line:84%
And sometimes it will
get better, in which case

01:05:41.900 --> 01:05:42.600 align:middle line:90%
it's great.

01:05:42.600 --> 01:05:43.900 align:middle line:90%
Sometimes it won't get better.

01:05:43.900 --> 01:05:45.620 align:middle line:84%
And I will also show
you-- and I probably

01:05:45.620 --> 01:05:47.677 align:middle line:84%
will run out of time,
which I'll do it on Monday.

01:05:47.677 --> 01:05:49.260 align:middle line:84%
I will also show
you, hey, what if you

01:05:49.260 --> 01:05:51.635 align:middle line:84%
want to do your own embeddings
from scratch without using

01:05:51.635 --> 01:05:52.620 align:middle line:90%
GloVe?

01:05:52.620 --> 01:05:55.840 align:middle line:84%
So all possibilities
will be covered.

01:05:55.840 --> 01:05:56.340 align:middle line:90%
Yeah.

01:05:56.340 --> 01:05:59.340 align:middle line:84%
So to come back to this--
this is the model we have.

01:05:59.340 --> 01:06:01.460 align:middle line:90%
And then-- all right.

01:06:01.460 --> 01:06:04.540 align:middle line:84%
So if we take a look at the
first few embedding vectors--

01:06:04.540 --> 01:06:07.700 align:middle line:84%
by the way, this
model.layers will give you

01:06:07.700 --> 01:06:09.490 align:middle line:84%
every layer as a list
of all the layers,

01:06:09.490 --> 01:06:11.740 align:middle line:84%
and then you can just grab
any layer you want and look

01:06:11.740 --> 01:06:12.820 align:middle line:90%
at its weights.

01:06:12.820 --> 01:06:13.785 align:middle line:90%
It's very handy.

01:06:13.785 --> 01:06:15.160 align:middle line:84%
So we are looking
at the weights,

01:06:15.160 --> 01:06:18.340 align:middle line:84%
and you can see
here the first two

01:06:18.340 --> 01:06:21.440 align:middle line:84%
vectors are all zeros because
that stands for unk and pad,

01:06:21.440 --> 01:06:22.880 align:middle line:84%
and then we have
everything else.

01:06:22.880 --> 01:06:24.580 align:middle line:90%
So everything looks fine so far.

01:06:24.580 --> 01:06:26.600 align:middle line:84%
And now we just
compile and fit it.

01:06:26.600 --> 01:06:30.740 align:middle line:84%
So as usual, Adam cross-entropy
accuracy, and then

01:06:30.740 --> 01:06:33.420 align:middle line:90%
we will just fit the model.

01:06:33.420 --> 01:06:34.820 align:middle line:90%
All right.

01:06:34.820 --> 01:06:39.020 align:middle line:84%
It's going to take
a few minutes.

01:06:39.020 --> 01:06:42.340 align:middle line:84%
And while it's running, so
what you will see in this Colab

01:06:42.340 --> 01:06:45.820 align:middle line:84%
is that in this particular
case, the embeddings actually

01:06:45.820 --> 01:06:47.500 align:middle line:90%
don't help a whole lot.

01:06:47.500 --> 01:06:48.680 align:middle line:90%
Why do you think that is?

01:06:48.680 --> 01:06:51.822 align:middle line:90%


01:06:51.822 --> 01:06:54.280 align:middle line:84%
Part of it could be because
we're averaging a lot of stuff.

01:06:54.280 --> 01:06:57.180 align:middle line:90%
Maybe that's hurting us.

01:06:57.180 --> 01:06:58.090 align:middle line:90%
Yeah?

01:06:58.090 --> 01:07:00.340 align:middle line:84%
AUDIENCE: I mean, I think
that embeddings are probably

01:07:00.340 --> 01:07:02.820 align:middle line:84%
trained on some corpus,
right-- like Wikipedia,

01:07:02.820 --> 01:07:05.460 align:middle line:84%
or something like that--
that is a little bit

01:07:05.460 --> 01:07:08.110 align:middle line:84%
different from the language
we tend to use in song lyrics.

01:07:08.110 --> 01:07:10.250 align:middle line:84%
And so maybe it's
not its ability

01:07:10.250 --> 01:07:16.770 align:middle line:84%
to extract the meaning of
candy from a song lyric,

01:07:16.770 --> 01:07:19.810 align:middle line:84%
maybe it's limited because it's
thinking of all the other ways

01:07:19.810 --> 01:07:20.950 align:middle line:90%
that could be represented.

01:07:20.950 --> 01:07:21.330 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, so there

01:07:21.330 --> 01:07:23.530 align:middle line:84%
could be a mismatch
between the corpus on which

01:07:23.530 --> 01:07:26.692 align:middle line:84%
the pre-trained stuff was
trained on versus the corpus

01:07:26.692 --> 01:07:28.150 align:middle line:84%
that you're working
with right now.

01:07:28.150 --> 01:07:29.370 align:middle line:90%
That's one big reason.

01:07:29.370 --> 01:07:32.150 align:middle line:84%
The other reason is that
we actually may have--

01:07:32.150 --> 01:07:34.690 align:middle line:84%
we have 50,000
examples, basically.

01:07:34.690 --> 01:07:36.090 align:middle line:90%
It's a lot of data.

01:07:36.090 --> 01:07:37.930 align:middle line:84%
So when you have
a lot of data, you

01:07:37.930 --> 01:07:39.817 align:middle line:84%
may not need any
of these things.

01:07:39.817 --> 01:07:41.650 align:middle line:84%
These things tend to
do really well when you

01:07:41.650 --> 01:07:43.730 align:middle line:84%
don't have a lot of
data, which means

01:07:43.730 --> 01:07:47.530 align:middle line:84%
you get to piggyback on what
these embeddings have learned

01:07:47.530 --> 01:07:49.970 align:middle line:90%
from all of Wikipedia.

01:07:49.970 --> 01:07:54.330 align:middle line:84%
So when you have a smallish data
set, basically the rule of thumb

01:07:54.330 --> 01:07:56.390 align:middle line:84%
here is that when your
data is really small,

01:07:56.390 --> 01:07:58.385 align:middle line:90%
try to use a pre-trained model.

01:07:58.385 --> 01:08:01.010 align:middle line:84%
And that's what you saw with the
handbags and shoes classifier.

01:08:01.010 --> 01:08:03.170 align:middle line:84%
We had 100 examples
of handbags and shoes,

01:08:03.170 --> 01:08:06.600 align:middle line:84%
and we used ResNet to
basically get to 100% accuracy.

01:08:06.600 --> 01:08:08.320 align:middle line:90%
The same of logic applies here.

01:08:08.320 --> 01:08:08.820 align:middle line:90%
All right.

01:08:08.820 --> 01:08:11.560 align:middle line:84%
So here, let's see
what's happening.

01:08:11.560 --> 01:08:12.840 align:middle line:90%
OK, it's done.

01:08:12.840 --> 01:08:13.470 align:middle line:90%
So we'll plot.

01:08:13.470 --> 01:08:16.080 align:middle line:90%


01:08:16.080 --> 01:08:16.920 align:middle line:90%
Right.

01:08:16.920 --> 01:08:17.420 align:middle line:90%
OK.

01:08:17.420 --> 01:08:17.920 align:middle line:90%
Let's look.

01:08:17.920 --> 01:08:20.250 align:middle line:84%
A very well-behaved
loss function curve.

01:08:20.250 --> 01:08:25.680 align:middle line:90%


01:08:25.680 --> 01:08:26.480 align:middle line:90%
OK.

01:08:26.480 --> 01:08:29.700 align:middle line:84%
So there doesn't seem to be any
massive overfitting going on.

01:08:29.700 --> 01:08:31.800 align:middle line:84%
They're moving really
nicely in lockstep.

01:08:31.800 --> 01:08:33.500 align:middle line:90%
Let's see what the thing is.

01:08:33.500 --> 01:08:36.359 align:middle line:90%


01:08:36.359 --> 01:08:39.640 align:middle line:90%
OK, 63%, which is not great.

01:08:39.640 --> 01:08:42.498 align:middle line:84%
It's not as good as what we saw
before when we used all 50,000

01:08:42.498 --> 01:08:44.540 align:middle line:84%
examples and just trained
something from scratch.

01:08:44.540 --> 01:08:47.180 align:middle line:84%
And that's just because in this
case, we have lots of examples.

01:08:47.180 --> 01:08:49.120 align:middle line:84%
These pre-trained
embeddings aren't as helpful

01:08:49.120 --> 01:08:50.319 align:middle line:90%
as they could be.

01:08:50.319 --> 01:08:53.540 align:middle line:84%
But if you have a small data
set, they could be very helpful.

01:08:53.540 --> 01:08:57.640 align:middle line:84%
And now we go to
what he pointed out.

01:08:57.640 --> 01:08:59.779 align:middle line:84%
Why can't we just optimize
these embeddings too?

01:08:59.779 --> 01:09:01.589 align:middle line:84%
Why do we have to
treat them as sacred?

01:09:01.589 --> 01:09:04.490 align:middle line:90%


01:09:04.490 --> 01:09:07.950 align:middle line:84%
Let's just unleash back prop
on it and see what happens.

01:09:07.950 --> 01:09:11.109 align:middle line:90%
So we'll do that.

01:09:11.109 --> 01:09:13.430 align:middle line:84%
So here, what we do
is we retrain it.

01:09:13.430 --> 01:09:15.069 align:middle line:84%
But here, we set
trainable equals

01:09:15.069 --> 01:09:16.490 align:middle line:90%
true for the embedding layer.

01:09:16.490 --> 01:09:16.990 align:middle line:90%
OK?

01:09:16.990 --> 01:09:17.930 align:middle line:90%
This is the key step.

01:09:17.930 --> 01:09:19.189 align:middle line:90%
Trainable equals true.

01:09:19.189 --> 01:09:21.350 align:middle line:90%
Otherwise, it's unchanged.

01:09:21.350 --> 01:09:24.250 align:middle line:90%
And then skip that.

01:09:24.250 --> 01:09:27.189 align:middle line:90%


01:09:27.189 --> 01:09:28.970 align:middle line:84%
We'll run it and
see what happens.

01:09:28.970 --> 01:09:32.290 align:middle line:84%
So before, it was whatever,
63% accuracy or something.

01:09:32.290 --> 01:09:35.430 align:middle line:84%
We'll see if it gets better
if you train the whole thing.

01:09:35.430 --> 01:09:38.229 align:middle line:84%
And the thing is you
can never be sure

01:09:38.229 --> 01:09:40.236 align:middle line:84%
because it may start
to overfit, which

01:09:40.236 --> 01:09:42.569 align:middle line:84%
is why you just have to
empirically see what's going on.

01:09:42.569 --> 01:09:43.689 align:middle line:90%
There are no guarantees.

01:09:43.689 --> 01:09:47.790 align:middle line:90%


01:09:47.790 --> 01:09:48.290 align:middle line:90%
All right.

01:09:48.290 --> 01:09:50.149 align:middle line:84%
Any questions while
it's training?

01:09:50.149 --> 01:09:52.069 align:middle line:90%
Yeah?

01:09:52.069 --> 01:09:54.149 align:middle line:84%
AUDIENCE: In the
first graph, when

01:09:54.149 --> 01:09:56.690 align:middle line:84%
you had the training
accuracy still increasing,

01:09:56.690 --> 01:09:59.530 align:middle line:84%
that might suggest that you
could use even more [INAUDIBLE].

01:09:59.530 --> 01:09:59.990 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

01:09:59.990 --> 01:10:00.970 align:middle line:90%
Exactly, exactly.

01:10:00.970 --> 01:10:03.380 align:middle line:84%
So in that curve, we
saw that the training

01:10:03.380 --> 01:10:04.525 align:middle line:90%
was continuing to increase.

01:10:04.525 --> 01:10:05.900 align:middle line:84%
Typically, what's
going to happen

01:10:05.900 --> 01:10:07.442 align:middle line:84%
is that training
will continue to get

01:10:07.442 --> 01:10:08.720 align:middle line:90%
better the more you train it.

01:10:08.720 --> 01:10:11.460 align:middle line:84%
The key thing is the
validation also improving.

01:10:11.460 --> 01:10:13.160 align:middle line:84%
If the validation
continues to improve,

01:10:13.160 --> 01:10:15.160 align:middle line:84%
there is a little bit
more gas left in the tank.

01:10:15.160 --> 01:10:16.368 align:middle line:90%
You can keep increasing more.

01:10:16.368 --> 01:10:17.533 align:middle line:90%
If it starts to flatten--

01:10:17.533 --> 01:10:19.200 align:middle line:84%
and even worse, if
it starts to go down,

01:10:19.200 --> 01:10:22.020 align:middle line:90%
then you want to pull back.

01:10:22.020 --> 01:10:23.580 align:middle line:90%
Yeah?

01:10:23.580 --> 01:10:26.060 align:middle line:84%
AUDIENCE: So you had used
the max tokens to limit

01:10:26.060 --> 01:10:28.760 align:middle line:84%
the vocabulary to the
most common 5,000,

01:10:28.760 --> 01:10:30.640 align:middle line:84%
and then the width
of that was 100.

01:10:30.640 --> 01:10:31.403 align:middle line:90%
What is the 100?

01:10:31.403 --> 01:10:32.820 align:middle line:84%
RAMA RAMAKRISHNAN:
The 100 is just

01:10:32.820 --> 01:10:34.712 align:middle line:90%
the length of the GloVe vector.

01:10:34.712 --> 01:10:36.420 align:middle line:84%
AUDIENCE: Does that
mean that it can only

01:10:36.420 --> 01:10:39.817 align:middle line:84%
capture how that word is
related to 100 other words?

01:10:39.817 --> 01:10:40.900 align:middle line:90%
RAMA RAMAKRISHNAN: No, no.

01:10:40.900 --> 01:10:43.080 align:middle line:84%
Basically, we are
saying that every word,

01:10:43.080 --> 01:10:45.620 align:middle line:84%
its intrinsic meaning
can be captured using

01:10:45.620 --> 01:10:48.133 align:middle line:90%
a vector of 100 dimensions.

01:10:48.133 --> 01:10:49.800 align:middle line:84%
Each of those dimensions
mean something.

01:10:49.800 --> 01:10:50.940 align:middle line:90%
We don't know what it is.

01:10:50.940 --> 01:10:53.380 align:middle line:84%
The first dimension
could mean color.

01:10:53.380 --> 01:10:55.320 align:middle line:84%
The second could mean
some sort of location.

01:10:55.320 --> 01:10:58.080 align:middle line:84%
The third could mean some
sort of time of the year.

01:10:58.080 --> 01:11:00.918 align:middle line:90%
We just have no idea.

01:11:00.918 --> 01:11:01.460 align:middle line:90%
AUDIENCE: OK.

01:11:01.460 --> 01:11:03.335 align:middle line:84%
And the pre-trained
model-- because we're not

01:11:03.335 --> 01:11:05.127 align:middle line:84%
allowing it to learn
the pre-trained model,

01:11:05.127 --> 01:11:06.280 align:middle line:90%
it has those already.

01:11:06.280 --> 01:11:08.580 align:middle line:84%
We don't know what they
are, but it has some--

01:11:08.580 --> 01:11:09.020 align:middle line:84%
RAMA RAMAKRISHNAN:
The people who

01:11:09.020 --> 01:11:10.600 align:middle line:84%
created it don't know
what they are either.

01:11:10.600 --> 01:11:11.020 align:middle line:90%
AUDIENCE: Yeah.

01:11:11.020 --> 01:11:13.312 align:middle line:84%
RAMA RAMAKRISHNAN: All they
know is that for each word,

01:11:13.312 --> 01:11:15.860 align:middle line:90%
they learned 100 long vector.

01:11:15.860 --> 01:11:19.700 align:middle line:84%
And that 100 long vector
was able to recreate

01:11:19.700 --> 01:11:21.540 align:middle line:90%
the co-occurrence matrix.

01:11:21.540 --> 01:11:24.180 align:middle line:84%
And then they probed it
using that visualization

01:11:24.180 --> 01:11:26.620 align:middle line:84%
of man, woman, sister,
brother, all that stuff, and it

01:11:26.620 --> 01:11:29.940 align:middle line:84%
seems to fit with
what we would expect.

01:11:29.940 --> 01:11:31.980 align:middle line:84%
AUDIENCE: Can you think
of it as analogous to--

01:11:31.980 --> 01:11:33.560 align:middle line:84%
when we did the
convolutional ones,

01:11:33.560 --> 01:11:35.100 align:middle line:90%
you have the number of kernels.

01:11:35.100 --> 01:11:37.700 align:middle line:84%
So in this case, so
if you had 32 kernels,

01:11:37.700 --> 01:11:40.062 align:middle line:84%
it's like 32 things
it can learn.

01:11:40.062 --> 01:11:42.520 align:middle line:84%
RAMA RAMAKRISHNAN: I think
that's actually a great analogy.

01:11:42.520 --> 01:11:43.138 align:middle line:90%
I love it.

01:11:43.138 --> 01:11:44.680 align:middle line:84%
That's a great way
to think about it.

01:11:44.680 --> 01:11:45.500 align:middle line:90%
Yes.

01:11:45.500 --> 01:11:48.438 align:middle line:84%
Much like we got to decide
how many filters to have,

01:11:48.438 --> 01:11:50.980 align:middle line:84%
here, we get to decide how long
the embedding dimension needs

01:11:50.980 --> 01:11:51.480 align:middle line:90%
to be.

01:11:51.480 --> 01:11:53.208 align:middle line:84%
And our hope is
that the more things

01:11:53.208 --> 01:11:55.500 align:middle line:84%
we are able to accommodate,
the more complicated things

01:11:55.500 --> 01:11:56.942 align:middle line:90%
it will pick up.

01:11:56.942 --> 01:11:59.650 align:middle line:84%
At the same time, you don't want
to have too many of these things

01:11:59.650 --> 01:12:01.930 align:middle line:84%
because it's going to
start picking up noise,

01:12:01.930 --> 01:12:05.890 align:middle line:90%
and that's never a good thing.

01:12:05.890 --> 01:12:07.990 align:middle line:90%
OK.

01:12:07.990 --> 01:12:09.930 align:middle line:84%
Was that a question
on this side?

01:12:09.930 --> 01:12:10.890 align:middle line:90%
Yeah?

01:12:10.890 --> 01:12:11.975 align:middle line:90%
Go ahead.

01:12:11.975 --> 01:12:14.410 align:middle line:90%
AUDIENCE: [INAUDIBLE].

01:12:14.410 --> 01:12:18.050 align:middle line:84%
Why do we use embeddings and
not the actual correlation

01:12:18.050 --> 01:12:22.690 align:middle line:90%
matrix rows to represent words?

01:12:22.690 --> 01:12:23.857 align:middle line:90%
Why do we need to abstract--

01:12:23.857 --> 01:12:25.357 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, yeah, yeah.

01:12:25.357 --> 01:12:26.850 align:middle line:90%
That's actually a good--

01:12:26.850 --> 01:12:28.650 align:middle line:90%
that's a good question.

01:12:28.650 --> 01:12:31.750 align:middle line:84%
One immediate reason is that
row is 500,000 vectors long.

01:12:31.750 --> 01:12:33.890 align:middle line:90%
500,000 long.

01:12:33.890 --> 01:12:37.290 align:middle line:84%
So you want that compact,
dense representation.

01:12:37.290 --> 01:12:39.090 align:middle line:84%
The second thing
is that thing is

01:12:39.090 --> 01:12:41.750 align:middle line:84%
subject to all the counts
of the Wikipedia corpus.

01:12:41.750 --> 01:12:43.810 align:middle line:90%
It's not normalized.

01:12:43.810 --> 01:12:46.530 align:middle line:84%
So you need to normalize it so
that if you take any two rows

01:12:46.530 --> 01:12:49.170 align:middle line:84%
and do dot product, you
will get some number which

01:12:49.170 --> 01:12:50.550 align:middle line:90%
is in a narrow range.

01:12:50.550 --> 01:12:53.890 align:middle line:84%
Otherwise, things don't
become comparable.

01:12:53.890 --> 01:12:56.270 align:middle line:84%
Now, both these
objections can be handled.

01:12:56.270 --> 01:12:57.197 align:middle line:90%
You can normalize.

01:12:57.197 --> 01:12:59.780 align:middle line:84%
You can reduce the size of the
corpus, and so on and so forth.

01:12:59.780 --> 01:13:01.697 align:middle line:84%
And in fact, that used
to be a very common way

01:13:01.697 --> 01:13:03.412 align:middle line:90%
people used to do it before.

01:13:03.412 --> 01:13:06.120 align:middle line:84%
But what they have discovered is
that the way we learn embeddings

01:13:06.120 --> 01:13:08.120 align:middle line:84%
now tends to be much more
effective in practice.

01:13:08.120 --> 01:13:11.080 align:middle line:90%


01:13:11.080 --> 01:13:14.040 align:middle line:84%
AUDIENCE: So what
we've done is--

01:13:14.040 --> 01:13:16.560 align:middle line:84%
what this process
does is it creates

01:13:16.560 --> 01:13:19.720 align:middle line:84%
this n dimensional
incomprehensible matrix

01:13:19.720 --> 01:13:23.600 align:middle line:84%
that captures, in essence,
a summarized version

01:13:23.600 --> 01:13:24.652 align:middle line:90%
of these relationships?

01:13:24.652 --> 01:13:25.760 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

01:13:25.760 --> 01:13:26.260 align:middle line:90%
Correct.

01:13:26.260 --> 01:13:29.460 align:middle line:84%
A compact representation
of relationships,

01:13:29.460 --> 01:13:33.200 align:middle line:84%
which is not subject to the
size of your vocabulary.

01:13:33.200 --> 01:13:35.140 align:middle line:84%
So you find that
1,000 words today.

01:13:35.140 --> 01:13:37.500 align:middle line:84%
Tomorrow, somebody comes up
with a word called selfie,

01:13:37.500 --> 01:13:39.000 align:middle line:84%
which didn't exist
five years ago,

01:13:39.000 --> 01:13:41.583 align:middle line:84%
and now your corpus has
gotten a little bit more.

01:13:41.583 --> 01:13:43.000 align:middle line:84%
So here, it's very
compact, and it

01:13:43.000 --> 01:13:45.020 align:middle line:84%
tends to have a much
longer shelf life.

01:13:45.020 --> 01:13:48.120 align:middle line:90%


01:13:48.120 --> 01:13:49.438 align:middle line:90%
Yeah.

01:13:49.438 --> 01:13:50.980 align:middle line:84%
All right, so let's
see where we are.

01:13:50.980 --> 01:13:54.400 align:middle line:90%


01:13:54.400 --> 01:13:56.290 align:middle line:90%
OK, so evaluate.

01:13:56.290 --> 01:13:59.110 align:middle line:90%


01:13:59.110 --> 01:14:00.770 align:middle line:90%
68-- 69%, almost.

01:14:00.770 --> 01:14:01.310 align:middle line:90%
It was 63.

01:14:01.310 --> 01:14:02.210 align:middle line:90%
We're going to 69.

01:14:02.210 --> 01:14:04.870 align:middle line:84%
So clearly, here, training the
whole thing-- including GloVe--

01:14:04.870 --> 01:14:06.110 align:middle line:90%
actually helps.

01:14:06.110 --> 01:14:11.410 align:middle line:84%
And so that begs the question--
well, if training GloVe helps,

01:14:11.410 --> 01:14:15.030 align:middle line:84%
maybe we should actually train
the whole thing from scratch.

01:14:15.030 --> 01:14:16.750 align:middle line:90%
Like, why the hell not?

01:14:16.750 --> 01:14:17.730 align:middle line:90%
Why the heck not?

01:14:17.730 --> 01:14:19.470 align:middle line:90%
I apologize.

01:14:19.470 --> 01:14:21.310 align:middle line:84%
So what we'll do
is we'll actually

01:14:21.310 --> 01:14:23.288 align:middle line:84%
create our own embeddings
and just train them.

01:14:23.288 --> 01:14:25.830 align:middle line:84%
And here, we don't have to worry
about co-occurrence matrices

01:14:25.830 --> 01:14:27.230 align:middle line:84%
and so on and so
forth because we

01:14:27.230 --> 01:14:28.530 align:middle line:90%
have a very specific objective.

01:14:28.530 --> 01:14:30.630 align:middle line:84%
We want to be very accurate
in predicting genre

01:14:30.630 --> 01:14:32.310 align:middle line:90%
for these songs.

01:14:32.310 --> 01:14:35.198 align:middle line:84%
The people who had
worked on GloVe,

01:14:35.198 --> 01:14:36.490 align:middle line:90%
they didn't have any objective.

01:14:36.490 --> 01:14:38.032 align:middle line:84%
They just wanted to
create embeddings

01:14:38.032 --> 01:14:40.210 align:middle line:90%
that were generally useful.

01:14:40.210 --> 01:14:43.550 align:middle line:84%
Here, we want to be specifically
useful for genre prediction.

01:14:43.550 --> 01:14:46.150 align:middle line:84%
And so what we can do
is we can actually train

01:14:46.150 --> 01:14:48.670 align:middle line:90%
the whole thing ourselves.

01:14:48.670 --> 01:14:50.350 align:middle line:90%
We can actually give it--

01:14:50.350 --> 01:14:52.350 align:middle line:84%
we can actually put an
embedding layer here.

01:14:52.350 --> 01:14:54.740 align:middle line:84%
We just arbitrarily
decided to choose 64

01:14:54.740 --> 01:14:57.400 align:middle line:84%
as the dimension,
as opposed to 100.

01:14:57.400 --> 01:14:58.772 align:middle line:90%
It'll run faster.

01:14:58.772 --> 01:14:59.980 align:middle line:90%
And then it's the same thing.

01:14:59.980 --> 01:15:02.188 align:middle line:84%
Global average pooling,
activation, blah, blah, blah,

01:15:02.188 --> 01:15:03.540 align:middle line:90%
blah, blah.

01:15:03.540 --> 01:15:05.320 align:middle line:90%
And then you run it.

01:15:05.320 --> 01:15:08.077 align:middle line:90%


01:15:08.077 --> 01:15:09.910 align:middle line:84%
Let's see if it finishes
in the next minute.

01:15:09.910 --> 01:15:12.780 align:middle line:90%


01:15:12.780 --> 01:15:14.558 align:middle line:84%
And we'll see if it
actually does better

01:15:14.558 --> 01:15:17.100 align:middle line:84%
than the pre-trained embeddings
or the pre-trained embeddings

01:15:17.100 --> 01:15:18.820 align:middle line:84%
that have been
further fine tuned.

01:15:18.820 --> 01:15:22.300 align:middle line:84%
And I don't remember what I
saw when I ran it yesterday.

01:15:22.300 --> 01:15:25.020 align:middle line:84%
And while it's running,
other questions?

01:15:25.020 --> 01:15:26.198 align:middle line:90%
Yeah?

01:15:26.198 --> 01:15:28.240 align:middle line:84%
AUDIENCE: So my question
is regarding embeddings.

01:15:28.240 --> 01:15:30.980 align:middle line:84%
When you call embedding
for a particular word,

01:15:30.980 --> 01:15:33.560 align:middle line:84%
we indicate that we have
certain amount of parameters.

01:15:33.560 --> 01:15:36.240 align:middle line:84%
Let's say in this case, we have
defined the predefined 100,

01:15:36.240 --> 01:15:37.990 align:middle line:84%
so there'll be 100
parameters, and they'll

01:15:37.990 --> 01:15:40.180 align:middle line:84%
be coefficient weights
for each of them.

01:15:40.180 --> 01:15:44.550 align:middle line:84%
So when we take a pre-trained
model like the one we took from

01:15:44.550 --> 01:15:47.260 align:middle line:84%
GloVe, so for each word, there
would already be those number

01:15:47.260 --> 01:15:49.240 align:middle line:90%
of parameters in terms of--

01:15:49.240 --> 01:15:49.540 align:middle line:90%
RAMA RAMAKRISHNAN: Right, yeah.

01:15:49.540 --> 01:15:51.957 align:middle line:84%
AUDIENCE: But then how do we
redefine them as that we want

01:15:51.957 --> 01:15:54.357 align:middle line:84%
only 100 or we want
only 10 parameters...?

01:15:54.357 --> 01:15:55.440 align:middle line:90%
RAMA RAMAKRISHNAN: No, no.

01:15:55.440 --> 01:15:57.660 align:middle line:84%
The GloVe thing
actually gives you

01:15:57.660 --> 01:16:00.500 align:middle line:84%
packaged-- it's
prepackaged to be 100 long.

01:16:00.500 --> 01:16:03.000 align:middle line:84%
I think they have 200 and
300 as well, if I recall.

01:16:03.000 --> 01:16:05.620 align:middle line:84%
We just happen to
use the one with 100.

01:16:05.620 --> 01:16:07.780 align:middle line:84%
AUDIENCE: The one that
is available [INAUDIBLE]?

01:16:07.780 --> 01:16:08.380 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah, yeah.

01:16:08.380 --> 01:16:10.922 align:middle line:84%
And there are many available,
we just get to pick and choose,

01:16:10.922 --> 01:16:13.980 align:middle line:90%
and I happen to pick 100.

01:16:13.980 --> 01:16:14.860 align:middle line:90%
Oh, OK.

01:16:14.860 --> 01:16:18.700 align:middle line:84%
It's a bit slow, but it's
actually looking promising.

01:16:18.700 --> 01:16:19.300 align:middle line:90%
9:55.

01:16:19.300 --> 01:16:21.180 align:middle line:90%
Yeah?

01:16:21.180 --> 01:16:22.660 align:middle line:84%
AUDIENCE: So during
the CNN models

01:16:22.660 --> 01:16:25.580 align:middle line:84%
or even during our
assignments, changing

01:16:25.580 --> 01:16:29.860 align:middle line:84%
the filters gave us more depth
and improvement in performance.

01:16:29.860 --> 01:16:32.898 align:middle line:84%
So here, would I be right in
concluding that it's actually

01:16:32.898 --> 01:16:34.940 align:middle line:84%
training the embeddings,
which is giving us more,

01:16:34.940 --> 01:16:36.980 align:middle line:84%
assuming that epoch and
batch changes are not

01:16:36.980 --> 01:16:38.600 align:middle line:90%
going to change as much?

01:16:38.600 --> 01:16:41.160 align:middle line:84%
So if I really want a genuine
change in performance,

01:16:41.160 --> 01:16:44.600 align:middle line:84%
we go to the level of
retraining the embeddings?

01:16:44.600 --> 01:16:45.100 align:middle line:90%
But--

01:16:45.100 --> 01:16:45.260 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

01:16:45.260 --> 01:16:47.820 align:middle line:84%
So what we saw was that
using GloVe assist was OK.

01:16:47.820 --> 01:16:49.820 align:middle line:84%
Using GloVe and then
training them helped a lot.

01:16:49.820 --> 01:16:51.403 align:middle line:84%
And now we are
basically saying, well,

01:16:51.403 --> 01:16:54.130 align:middle line:84%
what if we just abandon GloVe
and train our own embeddings

01:16:54.130 --> 01:16:55.790 align:middle line:90%
for our particular problem?

01:16:55.790 --> 01:16:57.503 align:middle line:84%
See, GloVe is a
general purpose tool,

01:16:57.503 --> 01:16:59.170 align:middle line:84%
so a general purpose
tool is really good

01:16:59.170 --> 01:17:00.610 align:middle line:90%
if you don't have a lot of data.

01:17:00.610 --> 01:17:01.630 align:middle line:90%
That's a good starting point.

01:17:01.630 --> 01:17:02.850 align:middle line:84%
But when you have
a lot of data, you

01:17:02.850 --> 01:17:04.433 align:middle line:84%
should always try
to do your own thing

01:17:04.433 --> 01:17:05.730 align:middle line:90%
and see if it's any better.

01:17:05.730 --> 01:17:07.450 align:middle line:90%
And in this case, I--

01:17:07.450 --> 01:17:08.430 align:middle line:90%
well, OK.

01:17:08.430 --> 01:17:10.610 align:middle line:90%
I think it's--

01:17:10.610 --> 01:17:11.110 align:middle line:90%
Come on.

01:17:11.110 --> 01:17:13.910 align:middle line:90%
It's 9:55.

01:17:13.910 --> 01:17:16.110 align:middle line:84%
I thought I was going
to enter any moment now.

01:17:16.110 --> 01:17:22.870 align:middle line:84%
So-- right, let's
look at the thing.

01:17:22.870 --> 01:17:25.970 align:middle line:90%


01:17:25.970 --> 01:17:26.750 align:middle line:90%
OK, folks.

01:17:26.750 --> 01:17:28.310 align:middle line:90%
So 72%.

01:17:28.310 --> 01:17:30.810 align:middle line:84%
So you can actually retrain
your own thing because there are

01:17:30.810 --> 01:17:33.110 align:middle line:84%
50,000 examples, and it gives
you an even better thing.

01:17:33.110 --> 01:17:33.670 align:middle line:90%
Thanks a lot.

01:17:33.670 --> 01:17:35.880 align:middle line:90%
Have a good rest of the week.

01:17:35.880 --> 01:17:50.000 align:middle line:90%