WEBVTT 00:00:00.000 --> 00:00:04.464 align:middle line:84% [SQUEAKING] [RUSTLING] [CLICKING] 00:00:04.464 --> 00:00:20.405 align:middle line:90% 00:00:20.405 --> 00:00:22.280 align:middle line:84% RAMA RAMAKRISHNAN: We'll continue our journey 00:00:22.280 --> 00:00:25.080 align:middle line:84% with natural language processing. 00:00:25.080 --> 00:00:27.620 align:middle line:84% We looked at the bag of words model, one-hot embeddings, 00:00:27.620 --> 00:00:28.760 align:middle line:90% and so on and so forth. 00:00:28.760 --> 00:00:31.240 align:middle line:84% And today, we will talk about embeddings, 00:00:31.240 --> 00:00:34.170 align:middle line:84% or to be more precise, standalone embeddings. 00:00:34.170 --> 00:00:35.920 align:middle line:84% And then that will tee us up for something 00:00:35.920 --> 00:00:38.000 align:middle line:84% called contextual embeddings, which 00:00:38.000 --> 00:00:41.680 align:middle line:84% is where the transformer really comes into play. 00:00:41.680 --> 00:00:42.180 align:middle line:90% All right. 00:00:42.180 --> 00:00:43.240 align:middle line:90% So let's get going. 00:00:43.240 --> 00:00:49.060 align:middle line:84% So FR, we have encoded input text with one-hot vector. 00:00:49.060 --> 00:00:52.480 align:middle line:84% So to just to refresh your memories from Monday, 00:00:52.480 --> 00:00:55.260 align:middle line:84% if this is the phrase that's coming into the system, 00:00:55.260 --> 00:00:57.960 align:middle line:84% we run it through the STIE process. 00:00:57.960 --> 00:00:59.950 align:middle line:84% And when we do that, what happens 00:00:59.950 --> 00:01:03.330 align:middle line:84% is that we first of all, we standardize, 00:01:03.330 --> 00:01:06.750 align:middle line:84% then we split on whitespace to get individual words. 00:01:06.750 --> 00:01:08.950 align:middle line:84% Then we assign words to integers, 00:01:08.950 --> 00:01:11.550 align:middle line:84% and then we take each integer and essentially create 00:01:11.550 --> 00:01:13.990 align:middle line:84% a one-hot version of that integer. 00:01:13.990 --> 00:01:19.210 align:middle line:84% And when we do that, basically, we have a vocabulary, right? 00:01:19.210 --> 00:01:21.410 align:middle line:84% And in this example, we just have 100 words. 00:01:21.410 --> 00:01:25.230 align:middle line:84% And you will note that this vocabulary which you arrive 00:01:25.230 --> 00:01:28.282 align:middle line:84% at once you standardize and tokenize 00:01:28.282 --> 00:01:30.390 align:middle line:84% has words like the because we decided 00:01:30.390 --> 00:01:34.790 align:middle line:84% not to remove stop words like a, and, the, and so on. 00:01:34.790 --> 00:01:38.150 align:middle line:84% So just to be clear, standardization-- 00:01:38.150 --> 00:01:42.150 align:middle line:84% here, standardization-- while it has historically been all about 00:01:42.150 --> 00:01:45.050 align:middle line:84% stripping punctuation, lowercasing everything, 00:01:45.050 --> 00:01:47.630 align:middle line:84% removing stopwords, and stemming-- 00:01:47.630 --> 00:01:49.330 align:middle line:84% while that has been true historically, 00:01:49.330 --> 00:01:52.710 align:middle line:84% if you look at modern practice, people essentially 00:01:52.710 --> 00:01:55.350 align:middle line:84% strip punctuation, maybe, and then lowercase, 00:01:55.350 --> 00:01:58.580 align:middle line:84% and they often don't even bother to do stemming and things 00:01:58.580 --> 00:02:00.986 align:middle line:84% like that, or to remove stop words. 00:02:00.986 --> 00:02:05.140 align:middle line:84% And that's why in Keras, the default standardization is only 00:02:05.140 --> 00:02:05.780 align:middle line:90% lowercasing. 00:02:05.780 --> 00:02:09.220 align:middle line:90% And punctuation stripping. 00:02:09.220 --> 00:02:12.620 align:middle line:84% This detail may actually be handy for homework too, perhaps. 00:02:12.620 --> 00:02:14.420 align:middle line:90% That's why I'm pointing it out. 00:02:14.420 --> 00:02:16.480 align:middle line:90% OK, so that's what we have. 00:02:16.480 --> 00:02:18.500 align:middle line:84% And so for each word that's coming in, 00:02:18.500 --> 00:02:21.580 align:middle line:90% we have a one-hot vector. 00:02:21.580 --> 00:02:24.820 align:middle line:84% The one-hot vector is just onto the vocabulary. 00:02:24.820 --> 00:02:30.660 align:middle line:84% And we can either add them up and get a multi-hot encoding, 00:02:30.660 --> 00:02:32.900 align:middle line:90% or we can-- 00:02:32.900 --> 00:02:35.802 align:middle line:84% sorry-- get a count encoding, or we can just do or. 00:02:35.802 --> 00:02:37.260 align:middle line:84% Look for just any ones in a column, 00:02:37.260 --> 00:02:39.300 align:middle line:90% and get multi-hot encoding. 00:02:39.300 --> 00:02:41.860 align:middle line:84% So that's what we saw last class. 00:02:41.860 --> 00:02:43.980 align:middle line:84% But this scheme, while it's quite 00:02:43.980 --> 00:02:48.100 align:middle line:84% effective for simple problems, it 00:02:48.100 --> 00:02:49.800 align:middle line:84% has some very serious shortcomings. 00:02:49.800 --> 00:02:52.420 align:middle line:84% And so we will delve into those shortcomings 00:02:52.420 --> 00:02:54.300 align:middle line:84% and then step back and say, all right, 00:02:54.300 --> 00:02:56.872 align:middle line:84% is there a solution to fix these things? 00:02:56.872 --> 00:02:57.372 align:middle line:90% OK. 00:02:57.372 --> 00:03:00.180 align:middle line:84% Explain the problem with one-hot vectors. 00:03:00.180 --> 00:03:01.520 align:middle line:90% There are lots of problems. 00:03:01.520 --> 00:03:02.480 align:middle line:90% Any volunteers? 00:03:02.480 --> 00:03:07.980 align:middle line:90% 00:03:07.980 --> 00:03:10.880 align:middle line:84% AUDIENCE: So similar words are understood differently. 00:03:10.880 --> 00:03:15.060 align:middle line:84% So for example, if you have this house is great 00:03:15.060 --> 00:03:17.420 align:middle line:84% and this house is awesome, great and awesome 00:03:17.420 --> 00:03:18.580 align:middle line:90% is encoded differently. 00:03:18.580 --> 00:03:21.610 align:middle line:84% And obviously, differently, even though this [INAUDIBLE]. 00:03:21.610 --> 00:03:22.860 align:middle line:90% RAMA RAMAKRISHNAN: Absolutely. 00:03:22.860 --> 00:03:26.300 align:middle line:84% So what he's pointing out is that if you have two words which 00:03:26.300 --> 00:03:29.210 align:middle line:84% are synonyms-- let's say great and awesome-- 00:03:29.210 --> 00:03:32.100 align:middle line:84% we would hope that the way we represent them using 00:03:32.100 --> 00:03:34.500 align:middle line:84% these vectors would have some connection to what 00:03:34.500 --> 00:03:35.940 align:middle line:90% the words actually mean. 00:03:35.940 --> 00:03:37.780 align:middle line:84% In particular, we would hope that if they 00:03:37.780 --> 00:03:40.022 align:middle line:84% mean similar things, that they are close by. 00:03:40.022 --> 00:03:41.480 align:middle line:84% If they mean very different things, 00:03:41.480 --> 00:03:43.230 align:middle line:84% we would hope that they are very far away. 00:03:43.230 --> 00:03:45.700 align:middle line:84% Things like that, common sensical expectations of what 00:03:45.700 --> 00:03:47.580 align:middle line:90% you want the vectors to have. 00:03:47.580 --> 00:03:50.460 align:middle line:84% So it clearly won't have that, and we'll look into it 00:03:50.460 --> 00:03:51.880 align:middle line:90% in more detail in a bit. 00:03:51.880 --> 00:03:54.943 align:middle line:84% But before we do that, there is also a computational issue, 00:03:54.943 --> 00:03:56.110 align:middle line:90% which we covered last class. 00:03:56.110 --> 00:03:59.170 align:middle line:84% Which is that if the vocabulary is really long, 00:03:59.170 --> 00:04:01.970 align:middle line:84% then each token, each word that's coming in here 00:04:01.970 --> 00:04:03.570 align:middle line:84% will have a one-hot vector that's 00:04:03.570 --> 00:04:05.730 align:middle line:84% as long as the size of the vocabulary. 00:04:05.730 --> 00:04:07.750 align:middle line:84% If you have 500,000 words in your vocabulary, 00:04:07.750 --> 00:04:10.490 align:middle line:84% every little word that comes in has a vector which is 500,000 00:04:10.490 --> 00:04:13.760 align:middle line:84% long, which feels like a gross waste of stuff. 00:04:13.760 --> 00:04:16.647 align:middle line:90% 00:04:16.647 --> 00:04:18.730 align:middle line:84% Now, you can mitigate it somewhat by choosing only 00:04:18.730 --> 00:04:20.019 align:middle line:90% the most frequent words. 00:04:20.019 --> 00:04:21.769 align:middle line:84% But it does increase the number of weights 00:04:21.769 --> 00:04:23.330 align:middle line:84% the model has to learn, and increase 00:04:23.330 --> 00:04:27.290 align:middle line:84% the need for compute, and data, and so on and so forth. 00:04:27.290 --> 00:04:29.490 align:middle line:84% Now, let's say that we have created a vocabulary 00:04:29.490 --> 00:04:31.210 align:middle line:90% from a training corpus. 00:04:31.210 --> 00:04:34.690 align:middle line:84% We have a bunch of text that's coming in. 00:04:34.690 --> 00:04:36.023 align:middle line:90% We have done the SD. 00:04:36.023 --> 00:04:37.690 align:middle line:84% The standardization, tokenization, we've 00:04:37.690 --> 00:04:39.290 align:middle line:90% created vocabulary from it. 00:04:39.290 --> 00:04:42.250 align:middle line:84% And let's say we get the words movie and film. 00:04:42.250 --> 00:04:43.370 align:middle line:90% So the question is-- 00:04:43.370 --> 00:04:47.043 align:middle line:84% and O's observation gets to this immediately. 00:04:47.043 --> 00:04:48.710 align:middle line:84% If you look at the words movie and film, 00:04:48.710 --> 00:04:51.722 align:middle line:84% are these two vectors close to each other or not? 00:04:51.722 --> 00:04:56.920 align:middle line:84% So if you have two vectors, how would we measure closeness? 00:04:56.920 --> 00:04:59.020 align:middle line:84% What's the simplest way to think about closeness? 00:04:59.020 --> 00:05:02.560 align:middle line:90% 00:05:02.560 --> 00:05:05.180 align:middle line:90% It's not a trick question. 00:05:05.180 --> 00:05:05.680 align:middle line:90% Distance. 00:05:05.680 --> 00:05:06.380 align:middle line:90% Yeah, exactly. 00:05:06.380 --> 00:05:08.480 align:middle line:84% So if they're really close, distance-wise, 00:05:08.480 --> 00:05:13.320 align:middle line:84% you would hope the similar words should be close by. 00:05:13.320 --> 00:05:20.040 align:middle line:84% So here, let's just imagine that the vector for movie-- 00:05:20.040 --> 00:05:27.960 align:middle line:84% let's say your vocabulary is, I don't know, 100,000 long. 00:05:27.960 --> 00:05:33.280 align:middle line:84% So your vector is 100,000 long, and the word for movie is 00:05:33.280 --> 00:05:33.940 align:middle line:90% the position. 00:05:33.940 --> 00:05:38.520 align:middle line:84% So this has a 1, everything else is 0. 00:05:38.520 --> 00:05:39.520 align:middle line:90% Right? 00:05:39.520 --> 00:05:42.520 align:middle line:90% 00:05:42.520 --> 00:05:44.040 align:middle line:84% Sorry, this is the vector for film, 00:05:44.040 --> 00:05:47.320 align:middle line:84% and maybe this is the position for film. 00:05:47.320 --> 00:05:48.720 align:middle line:90% So that has a 1. 00:05:48.720 --> 00:05:51.150 align:middle line:90% Everything else here-- 00:05:51.150 --> 00:05:52.310 align:middle line:90% 0. 00:05:52.310 --> 00:05:55.670 align:middle line:84% What is the distance between these two vectors? 00:05:55.670 --> 00:05:57.870 align:middle line:84% You just use the Euclidean distance. 00:05:57.870 --> 00:05:59.510 align:middle line:84% So the Euclidean distance, you will 00:05:59.510 --> 00:06:04.390 align:middle line:84% recall, you literally just take the difference of these values, 00:06:04.390 --> 00:06:07.190 align:middle line:84% square them, add them up, take the square root. 00:06:07.190 --> 00:06:11.150 align:middle line:84% So which means that all the zeros will obviously give you 0. 00:06:11.150 --> 00:06:14.110 align:middle line:90% This 1 is going to give you a 1. 00:06:14.110 --> 00:06:16.070 align:middle line:84% This comparison is going to give you another 1. 00:06:16.070 --> 00:06:17.190 align:middle line:90% 1 plus 1-- 2. 00:06:17.190 --> 00:06:17.910 align:middle line:90% Root 2. 00:06:17.910 --> 00:06:20.090 align:middle line:90% That's the answer. 00:06:20.090 --> 00:06:22.090 align:middle line:84% So this is between these two vectors, is root 2. 00:06:22.090 --> 00:06:25.190 align:middle line:90% 00:06:25.190 --> 00:06:30.590 align:middle line:84% Now, so the distance between them is root 2. 00:06:30.590 --> 00:06:34.070 align:middle line:84% What about the one-hot encoded vectors for good and bad? 00:06:34.070 --> 00:06:36.710 align:middle line:84% Clearly, good and bad mean opposite things. 00:06:36.710 --> 00:06:39.250 align:middle line:84% What is the distance between the good and bad 0, 1 vectors? 00:06:39.250 --> 00:06:42.670 align:middle line:90% 00:06:42.670 --> 00:06:45.510 align:middle line:90% Still root 2. 00:06:45.510 --> 00:06:47.930 align:middle line:84% Because the zeros don't mean anything, 00:06:47.930 --> 00:06:49.895 align:middle line:84% the ones are not in the same place. 00:06:49.895 --> 00:06:52.520 align:middle line:84% So when you subtract the 1 and the 0, you'll get ones and ones. 00:06:52.520 --> 00:06:54.980 align:middle line:90% Add them up to root 2. 00:06:54.980 --> 00:06:57.160 align:middle line:84% In fact, you take any two words in your vocabulary. 00:06:57.160 --> 00:06:59.000 align:middle line:84% What's the distance between the two one-hot vectors 00:06:59.000 --> 00:06:59.700 align:middle line:90% for those words? 00:06:59.700 --> 00:07:01.660 align:middle line:90% It's root 2. 00:07:01.660 --> 00:07:04.980 align:middle line:84% So if any two words have the same distance, 00:07:04.980 --> 00:07:08.780 align:middle line:84% does this even have a notion of distance? 00:07:08.780 --> 00:07:11.020 align:middle line:90% It doesn't. 00:07:11.020 --> 00:07:13.980 align:middle line:84% There's no notion of distance from one-hot vectors. 00:07:13.980 --> 00:07:17.900 align:middle line:84% It has no connection to the actual meanings of these words. 00:07:17.900 --> 00:07:22.192 align:middle line:84% It's just a way of representing them. 00:07:22.192 --> 00:07:26.340 align:middle line:84% So that is the big problem with one-hot vectors. 00:07:26.340 --> 00:07:28.400 align:middle line:84% So the distance between them is the same, 00:07:28.400 --> 00:07:29.400 align:middle line:90% regardless of the words. 00:07:29.400 --> 00:07:32.100 align:middle line:84% It's got nothing to do with the meaning of the words. 00:07:32.100 --> 00:07:35.900 align:middle line:84% And this is a huge problem, which we'll have to solve. 00:07:35.900 --> 00:07:38.598 align:middle line:84% So to summarize where we are, if the vocabulary is very long, 00:07:38.598 --> 00:07:40.140 align:middle line:84% each token will have a one-hot vector 00:07:40.140 --> 00:07:41.560 align:middle line:90% that's long as vocabulary. 00:07:41.560 --> 00:07:45.660 align:middle line:84% That's a computational and training problem. 00:07:45.660 --> 00:07:47.260 align:middle line:84% And then this is a deeper problem 00:07:47.260 --> 00:07:49.218 align:middle line:84% where there's no connection between the meaning 00:07:49.218 --> 00:07:51.340 align:middle line:90% of a word and its vector. 00:07:51.340 --> 00:07:56.180 align:middle line:84% So wouldn't it be nice if vectors that represent 00:07:56.180 --> 00:07:57.700 align:middle line:90% synonyms-- 00:07:57.700 --> 00:07:59.860 align:middle line:90% movie and film, apple-banana-- 00:07:59.860 --> 00:08:01.980 align:middle line:84% hopefully, they are close to each other. 00:08:01.980 --> 00:08:03.460 align:middle line:84% Wouldn't it be nice if the vectors 00:08:03.460 --> 00:08:05.210 align:middle line:84% for things that mean very different things 00:08:05.210 --> 00:08:07.020 align:middle line:90% were far from each other? 00:08:07.020 --> 00:08:09.280 align:middle line:84% So let's take a look at a particular example. 00:08:09.280 --> 00:08:09.780 align:middle line:90% OK? 00:08:09.780 --> 00:08:13.300 align:middle line:84% Let's assume that we have been magically given 00:08:13.300 --> 00:08:15.340 align:middle line:84% these vectors so that they actually 00:08:15.340 --> 00:08:17.340 align:middle line:90% have some notion of meaning. 00:08:17.340 --> 00:08:21.980 align:middle line:84% And for convenience, let's say that we take just the first two 00:08:21.980 --> 00:08:23.438 align:middle line:90% dimensions of these vectors-- 00:08:23.438 --> 00:08:24.980 align:middle line:84% the first two dimensions-- so that we 00:08:24.980 --> 00:08:28.140 align:middle line:90% can do a scatter plot on them. 00:08:28.140 --> 00:08:30.640 align:middle line:84% So we plot the first dimension of these vectors, 00:08:30.640 --> 00:08:31.780 align:middle line:90% the second dimension. 00:08:31.780 --> 00:08:34.179 align:middle line:84% And what we have in this little cartoon is 00:08:34.179 --> 00:08:41.520 align:middle line:84% we have plotted the word for factory, for home, for building, 00:08:41.520 --> 00:08:43.980 align:middle line:84% and they all happen to be clustered here. 00:08:43.980 --> 00:08:46.730 align:middle line:84% Clearly, this representation is capturing some notion 00:08:46.730 --> 00:08:52.890 align:middle line:84% of what the thing is, some sort of building And here, we have, 00:08:52.890 --> 00:08:55.090 align:middle line:90% a bicycle, truck, and car. 00:08:55.090 --> 00:08:59.370 align:middle line:84% Clearly, this is the automobile cluster, transportation cluster. 00:08:59.370 --> 00:09:01.450 align:middle line:84% And here, we have a fruit cluster. 00:09:01.450 --> 00:09:04.330 align:middle line:84% And here, we have sports balls cluster. 00:09:04.330 --> 00:09:06.060 align:middle line:90% OK? 00:09:06.060 --> 00:09:07.650 align:middle line:84% Because it's a cartoon, things are 00:09:07.650 --> 00:09:10.116 align:middle line:90% all nice and cleanly separated. 00:09:10.116 --> 00:09:12.190 align:middle line:84% So now, if you take the word apple, 00:09:12.190 --> 00:09:14.450 align:middle line:84% where do you think it's going to go? 00:09:14.450 --> 00:09:18.620 align:middle line:84% It's going to go into A, C, D, or B? 00:09:18.620 --> 00:09:19.510 align:middle line:90% C, right? 00:09:19.510 --> 00:09:24.210 align:middle line:84% It makes eminent sense it's going to go to C. Good. 00:09:24.210 --> 00:09:29.070 align:middle line:84% Now wouldn't it be nice if, more generally, 00:09:29.070 --> 00:09:32.810 align:middle line:84% if the geometric relationship between word vectors 00:09:32.810 --> 00:09:35.130 align:middle line:84% represents a semantic relationship 00:09:35.130 --> 00:09:39.250 align:middle line:84% between the underlying objects that the words represent? 00:09:39.250 --> 00:09:41.690 align:middle line:84% And I say relationship and not distance 00:09:41.690 --> 00:09:43.350 align:middle line:90% because it's not just distance. 00:09:43.350 --> 00:09:46.360 align:middle line:90% It's actually more than that. 00:09:46.360 --> 00:09:48.320 align:middle line:90% So let's look another one. 00:09:48.320 --> 00:09:53.000 align:middle line:84% Here, we have-- this is the vector plotted for puppy 00:09:53.000 --> 00:09:57.300 align:middle line:90% and dog, and this is calf. 00:09:57.300 --> 00:09:58.740 align:middle line:84% We have plotted the word for calf. 00:09:58.740 --> 00:10:00.440 align:middle line:84% And let's say that we need to figure out 00:10:00.440 --> 00:10:04.920 align:middle line:84% where would the embedding-- the word vector for cow appear? 00:10:04.920 --> 00:10:06.060 align:middle line:90% Where is the most logical? 00:10:06.060 --> 00:10:07.920 align:middle line:90% Should it be A? 00:10:07.920 --> 00:10:08.900 align:middle line:90% Should it be C? 00:10:08.900 --> 00:10:09.540 align:middle line:90% Should it be B? 00:10:09.540 --> 00:10:11.760 align:middle line:90% Where should it be? 00:10:11.760 --> 00:10:13.744 align:middle line:90% AUDIENCE: This is C. 00:10:13.744 --> 00:10:14.720 align:middle line:90% RAMA RAMAKRISHNAN: C? 00:10:14.720 --> 00:10:16.360 align:middle line:90% OK, what's the logic? 00:10:16.360 --> 00:10:17.460 align:middle line:90% Any volunteers? 00:10:17.460 --> 00:10:20.020 align:middle line:90% Just put your hand up. 00:10:20.020 --> 00:10:20.520 align:middle line:90% Yes? 00:10:20.520 --> 00:10:23.920 align:middle line:90% 00:10:23.920 --> 00:10:25.560 align:middle line:84% AUDIENCE: A calf is a baby animal, 00:10:25.560 --> 00:10:27.680 align:middle line:84% whereas the cow is probably the adult. 00:10:27.680 --> 00:10:29.520 align:middle line:84% So it'd be closer to dog, which is B-- 00:10:29.520 --> 00:10:30.728 align:middle line:90% the adult version of the dog. 00:10:30.728 --> 00:10:31.812 align:middle line:90% RAMA RAMAKRISHNAN: Got it. 00:10:31.812 --> 00:10:34.080 align:middle line:84% So you're basically saying, go from the puppy version 00:10:34.080 --> 00:10:35.600 align:middle line:90% to the grown up version. 00:10:35.600 --> 00:10:37.360 align:middle line:84% That's what you're getting at, right? 00:10:37.360 --> 00:10:39.600 align:middle line:84% And that's a totally valid way to think about it. 00:10:39.600 --> 00:10:42.058 align:middle line:84% But there are a couple of ways to think about this-- which, 00:10:42.058 --> 00:10:43.588 align:middle line:90% this is one of those two ways. 00:10:43.588 --> 00:10:45.630 align:middle line:84% So what you can do is you can actually look at it 00:10:45.630 --> 00:10:47.410 align:middle line:90% and say, well, OK. 00:10:47.410 --> 00:10:50.710 align:middle line:84% If this is bringing you bad memories of GMAT and GRE 00:10:50.710 --> 00:10:52.540 align:middle line:84% and stuff like that, I apologize. 00:10:52.540 --> 00:10:55.110 align:middle line:90% 00:10:55.110 --> 00:10:58.750 align:middle line:84% So a puppy is to a dog like a calf is to a cow, right? 00:10:58.750 --> 00:11:01.170 align:middle line:84% Which means that there's exactly what Jay is pointing out. 00:11:01.170 --> 00:11:03.910 align:middle line:84% You can go from the baby version to the full grown version 00:11:03.910 --> 00:11:06.590 align:middle line:84% if you go in the horizontal direction. 00:11:06.590 --> 00:11:10.130 align:middle line:84% But maybe, if you go in the vertical direction, 00:11:10.130 --> 00:11:12.110 align:middle line:84% you're essentially going up and down 00:11:12.110 --> 00:11:16.590 align:middle line:90% the young entities of animals. 00:11:16.590 --> 00:11:18.430 align:middle line:90% So here, you're growing with-- 00:11:18.430 --> 00:11:21.250 align:middle line:84% you're still across the same dimension of animals. 00:11:21.250 --> 00:11:24.030 align:middle line:84% You're just going from the age level. 00:11:24.030 --> 00:11:25.470 align:middle line:90% That is the band here. 00:11:25.470 --> 00:11:27.470 align:middle line:84% So this is the grown up version of a whole bunch 00:11:27.470 --> 00:11:30.010 align:middle line:84% of animals, the puppy version of a whole bunch of animals. 00:11:30.010 --> 00:11:32.430 align:middle line:84% So the vertical dimension measures some variation 00:11:32.430 --> 00:11:38.530 align:middle line:84% across animal species of the same, roughly, maturity stage. 00:11:38.530 --> 00:11:41.060 align:middle line:90% So these directions also matter. 00:11:41.060 --> 00:11:43.780 align:middle line:90% It's not just the distance. 00:11:43.780 --> 00:11:46.740 align:middle line:84% That's what I mean when I say semantic relationship 00:11:46.740 --> 00:11:48.900 align:middle line:90% and geometric relationship. 00:11:48.900 --> 00:11:51.580 align:middle line:84% The relationship is distance and direction. 00:11:51.580 --> 00:11:53.220 align:middle line:90% Both have to be involved. 00:11:53.220 --> 00:11:58.900 align:middle line:84% So now, word embeddings-- as we will learn soon-- 00:11:58.900 --> 00:12:01.860 align:middle line:84% are word vectors designed to achieve exactly 00:12:01.860 --> 00:12:04.120 align:middle line:90% these requirements. 00:12:04.120 --> 00:12:06.660 align:middle line:84% They will achieve these requirements, 00:12:06.660 --> 00:12:09.680 align:middle line:84% and they will fix both these problems very elegantly. 00:12:09.680 --> 00:12:12.983 align:middle line:90% 00:12:12.983 --> 00:12:14.900 align:middle line:84% So let's say that we have word embeddings that 00:12:14.900 --> 00:12:16.260 align:middle line:90% achieve both these problems. 00:12:16.260 --> 00:12:17.660 align:middle line:90% Are we basically done? 00:12:17.660 --> 00:12:19.860 align:middle line:90% Can we declare victory? 00:12:19.860 --> 00:12:23.660 align:middle line:84% Or is there anything that even words 00:12:23.660 --> 00:12:28.420 align:middle line:84% which actually capture the meaning of the underlying thing 00:12:28.420 --> 00:12:29.600 align:middle line:90% don't fully address? 00:12:29.600 --> 00:12:32.160 align:middle line:84% Is there any remaining problem we have to worry about? 00:12:32.160 --> 00:12:33.460 align:middle line:90% Yes? 00:12:33.460 --> 00:12:34.280 align:middle line:90% AUDIENCE: Context. 00:12:34.280 --> 00:12:35.405 align:middle line:90% RAMA RAMAKRISHNAN: Context? 00:12:35.405 --> 00:12:36.860 align:middle line:90% Yes. 00:12:36.860 --> 00:12:38.380 align:middle line:90% Context, right? 00:12:38.380 --> 00:12:41.420 align:middle line:90% The fact is a word's meaning-- 00:12:41.420 --> 00:12:43.160 align:middle line:90% sure, every word has a meaning. 00:12:43.160 --> 00:12:46.540 align:middle line:84% But we know that some words have multiple meanings, 00:12:46.540 --> 00:12:50.400 align:middle line:84% and that meaning is really inferrential 00:12:50.400 --> 00:12:53.233 align:middle line:84% or you can make sense of it-- only if you know the surrounding 00:12:53.233 --> 00:12:55.100 align:middle line:90% context. 00:12:55.100 --> 00:12:59.140 align:middle line:84% If you see the word bank, B-A-N-K-- 00:12:59.140 --> 00:13:01.220 align:middle line:84% sure, it could be a financial institution. 00:13:01.220 --> 00:13:03.020 align:middle line:90% It could be the side of a river. 00:13:03.020 --> 00:13:07.260 align:middle line:84% It could be the act of a plane turning in one direction. 00:13:07.260 --> 00:13:11.820 align:middle line:84% It could be someone hoping for something, banking on something. 00:13:11.820 --> 00:13:15.140 align:middle line:84% The list of possible meanings of the word bank is basically 00:13:15.140 --> 00:13:18.740 align:middle line:84% enormous, and you cannot figure out what it means unless you 00:13:18.740 --> 00:13:20.980 align:middle line:84% know what else is going on around that word. 00:13:20.980 --> 00:13:23.360 align:middle line:84% So context is super, super important. 00:13:23.360 --> 00:13:25.900 align:middle line:84% And these word embeddings just tell you 00:13:25.900 --> 00:13:27.620 align:middle line:90% what the meaning of the word is. 00:13:27.620 --> 00:13:29.340 align:middle line:84% And basically, what's going to happen 00:13:29.340 --> 00:13:32.340 align:middle line:84% when you have a word which could mean many different things, 00:13:32.340 --> 00:13:36.660 align:middle line:84% it's going to give you some average version of that meaning. 00:13:36.660 --> 00:13:39.412 align:middle line:84% The average version is not going to be very good. 00:13:39.412 --> 00:13:41.370 align:middle line:84% There are some words which only mean one thing, 00:13:41.370 --> 00:13:42.850 align:middle line:90% and you'll be OK there. 00:13:42.850 --> 00:13:47.650 align:middle line:84% But for the rest of it, it's going to be tough. 00:13:47.650 --> 00:13:50.840 align:middle line:90% So what we need is some way-- 00:13:50.840 --> 00:13:53.410 align:middle line:90% 00:13:53.410 --> 00:13:56.683 align:middle line:84% you need to find a way to make word embeddings contextual. 00:13:56.683 --> 00:13:58.850 align:middle line:84% Meaning, we need to somehow consider the other words 00:13:58.850 --> 00:14:01.010 align:middle line:90% in the sentence. 00:14:01.010 --> 00:14:05.210 align:middle line:84% So if we can do that, then we will be in great shape 00:14:05.210 --> 00:14:08.370 align:middle line:84% to solve all sorts of NLP problems. 00:14:08.370 --> 00:14:13.250 align:middle line:84% So as it turns out, contextual word embeddings or word vectors 00:14:13.250 --> 00:14:17.050 align:middle line:84% or word embeddings will achieve both these requirements. 00:14:17.050 --> 00:14:19.970 align:middle line:84% They capture the semantic geometric relationship 00:14:19.970 --> 00:14:23.970 align:middle line:84% thing I talked about, and they are contextual. 00:14:23.970 --> 00:14:26.370 align:middle line:90% They're really fantastic. 00:14:26.370 --> 00:14:29.890 align:middle line:84% And the key to calculating contextual word embeddings 00:14:29.890 --> 00:14:33.450 align:middle line:90% is the transformer. 00:14:33.450 --> 00:14:35.920 align:middle line:84% That is why transformers are justifiably famous. 00:14:35.920 --> 00:14:39.600 align:middle line:90% 00:14:39.600 --> 00:14:41.520 align:middle line:90% That's the lay of the land here. 00:14:41.520 --> 00:14:43.240 align:middle line:84% So today, we're going to look at how 00:14:43.240 --> 00:14:48.520 align:middle line:84% to calculate standalone or uncontextual word embeddings. 00:14:48.520 --> 00:14:50.480 align:middle line:84% And then starting Monday, we will 00:14:50.480 --> 00:14:52.800 align:middle line:84% take these un-standalone embeddings 00:14:52.800 --> 00:14:55.720 align:middle line:84% and make them contextual using transformers. 00:14:55.720 --> 00:14:57.240 align:middle line:90% That is the plan. 00:14:57.240 --> 00:14:58.840 align:middle line:90% Any questions so far? 00:14:58.840 --> 00:15:00.840 align:middle line:84% So now, let's think about how we can 00:15:00.840 --> 00:15:03.773 align:middle line:84% learn these standalone embeddings from data. 00:15:03.773 --> 00:15:06.320 align:middle line:84% Now, the naive way to think about it 00:15:06.320 --> 00:15:08.480 align:middle line:84% would be, hey, why don't we manually collect 00:15:08.480 --> 00:15:11.110 align:middle line:84% a whole bunch of synonyms, antonyms, related words, et 00:15:11.110 --> 00:15:14.240 align:middle line:84% cetera and try to assign embedding vectors to them that 00:15:14.240 --> 00:15:17.698 align:middle line:90% satisfy our requirements? 00:15:17.698 --> 00:15:19.240 align:middle line:84% Now as you can imagine, there's going 00:15:19.240 --> 00:15:22.540 align:middle line:84% to be a long, painful, and never quite complete exercise. 00:15:22.540 --> 00:15:26.040 align:middle line:90% 00:15:26.040 --> 00:15:29.310 align:middle line:84% And given that we are machine learning people, 00:15:29.310 --> 00:15:31.060 align:middle line:84% question is, can we do it in a better way? 00:15:31.060 --> 00:15:32.477 align:middle line:84% Can we just learn it from the data 00:15:32.477 --> 00:15:34.160 align:middle line:84% without doing any of this manual stuff? 00:15:34.160 --> 00:15:39.710 align:middle line:84% And the key insight that makes it all happen 00:15:39.710 --> 00:15:44.350 align:middle line:84% is this humble looking line on the screen by John Firth, who 00:15:44.350 --> 00:15:45.950 align:middle line:90% was a linguist. 00:15:45.950 --> 00:15:49.250 align:middle line:84% "You shall know a word by the company it keeps." 00:15:49.250 --> 00:15:52.590 align:middle line:84% I wish I could deliver this in a British accent. 00:15:52.590 --> 00:15:55.590 align:middle line:84% "You shall know a word by the company it keeps." 00:15:55.590 --> 00:15:58.450 align:middle line:90% It's a very profound statement. 00:15:58.450 --> 00:16:02.630 align:middle line:84% And here is the key intuition behind this. 00:16:02.630 --> 00:16:05.590 align:middle line:84% It says, let's say that you have a sentence like, 00:16:05.590 --> 00:16:09.630 align:middle line:84% "the acting in the dash was superb." 00:16:09.630 --> 00:16:11.150 align:middle line:84% What are some words that you folks 00:16:11.150 --> 00:16:12.942 align:middle line:84% think are likely to appear in the sentence? 00:16:12.942 --> 00:16:15.270 align:middle line:90% 00:16:15.270 --> 00:16:16.870 align:middle line:90% Shout it out. 00:16:16.870 --> 00:16:22.070 align:middle line:90% Play, movie, show, musical. 00:16:22.070 --> 00:16:24.613 align:middle line:84% Those are all some great candidates. 00:16:24.613 --> 00:16:26.030 align:middle line:84% The acting in the movie, the film, 00:16:26.030 --> 00:16:27.450 align:middle line:90% musical, and so on and so forth. 00:16:27.450 --> 00:16:29.052 align:middle line:84% Now, let's say that I ask you, what 00:16:29.052 --> 00:16:31.510 align:middle line:84% are some words that are unlikely to appear in the sentence? 00:16:31.510 --> 00:16:35.820 align:middle line:84% And I think we could all be here for days listing them out. 00:16:35.820 --> 00:16:38.125 align:middle line:90% I just listed these out. 00:16:38.125 --> 00:16:39.500 align:middle line:84% I love the word tensor, so I have 00:16:39.500 --> 00:16:41.900 align:middle line:84% to find a way to use it somewhere. 00:16:41.900 --> 00:16:42.400 align:middle line:90% All right. 00:16:42.400 --> 00:16:44.180 align:middle line:84% So the acting of the banana was superb. 00:16:44.180 --> 00:16:46.100 align:middle line:90% Clearly nonsensical. 00:16:46.100 --> 00:16:52.380 align:middle line:84% So what we are seeing here is that if certain words are 00:16:52.380 --> 00:16:55.500 align:middle line:90% interchangeable in a sentence-- 00:16:55.500 --> 00:17:00.060 align:middle line:84% meaning, you change them, and the sentence still makes sense-- 00:17:00.060 --> 00:17:02.020 align:middle line:84% if they appear in the same context 00:17:02.020 --> 00:17:04.359 align:middle line:84% very often, i.e. if they're interchangeable, 00:17:04.359 --> 00:17:06.998 align:middle line:90% they are probably related. 00:17:06.998 --> 00:17:10.660 align:middle line:84% Sort of like, we don't even have to know what the word is. 00:17:10.660 --> 00:17:13.285 align:middle line:84% All we have to know is that this word and this word, 00:17:13.285 --> 00:17:15.160 align:middle line:84% you can drop them into a particular sentence. 00:17:15.160 --> 00:17:17.220 align:middle line:84% You can fill in the blank of that sentence with that word, 00:17:17.220 --> 00:17:18.395 align:middle line:90% and it actually makes sense. 00:17:18.395 --> 00:17:19.520 align:middle line:90% Then you're like-- oh, wow. 00:17:19.520 --> 00:17:20.020 align:middle line:90% OK. 00:17:20.020 --> 00:17:21.980 align:middle line:90% These words are related then. 00:17:21.980 --> 00:17:24.660 align:middle line:84% You're inferring their relatedness, 00:17:24.660 --> 00:17:30.060 align:middle line:84% not by looking at them directly, but by seeing where they live. 00:17:30.060 --> 00:17:35.900 align:middle line:84% It's a very, very clever idea, and it'll slowly sink into you. 00:17:35.900 --> 00:17:37.283 align:middle line:90% So that's the first observation. 00:17:37.283 --> 00:17:39.200 align:middle line:84% If they appear in the same context very often, 00:17:39.200 --> 00:17:41.780 align:middle line:90% they are likely to be related. 00:17:41.780 --> 00:17:45.190 align:middle line:84% Generally, related words appear in related contexts. 00:17:45.190 --> 00:17:47.700 align:middle line:90% 00:17:47.700 --> 00:17:51.340 align:middle line:84% So all we have to do is to figure out a way 00:17:51.340 --> 00:17:57.300 align:middle line:84% to calculate context, and then use that to understand 00:17:57.300 --> 00:18:00.472 align:middle line:84% what the words are that happen to live in these contexts. 00:18:00.472 --> 00:18:02.680 align:middle line:84% And there are some beautiful ways to do these things, 00:18:02.680 --> 00:18:06.260 align:middle line:84% and we'll and we'll really dive deep into one such way to do it. 00:18:06.260 --> 00:18:10.940 align:middle line:84% So what we're going to do in this approach is 00:18:10.940 --> 00:18:15.900 align:middle line:84% that since words that appear in related contexts 00:18:15.900 --> 00:18:19.180 align:middle line:84% mean same similar things, first of all, I have to define what 00:18:19.180 --> 00:18:21.297 align:middle line:90% do you mean by context? 00:18:21.297 --> 00:18:22.880 align:middle line:84% There are many ways to define context. 00:18:22.880 --> 00:18:24.280 align:middle line:84% We're going to go with a very simple explanation, 00:18:24.280 --> 00:18:25.113 align:middle line:90% a simple definition. 00:18:25.113 --> 00:18:27.020 align:middle line:84% Which is that if words happen to appear 00:18:27.020 --> 00:18:31.350 align:middle line:84% in the same sentence a lot, let me think that they 00:18:31.350 --> 00:18:32.350 align:middle line:90% are in the same context. 00:18:32.350 --> 00:18:34.650 align:middle line:90% So context here means sentence. 00:18:34.650 --> 00:18:38.290 align:middle line:84% So what we can do is we can actually 00:18:38.290 --> 00:18:39.870 align:middle line:90% take a whole bunch of text-- 00:18:39.870 --> 00:18:42.010 align:middle line:90% maybe all of Wikipedia-- 00:18:42.010 --> 00:18:43.630 align:middle line:84% and then break it up into sentences. 00:18:43.630 --> 00:18:45.890 align:middle line:84% We'll have billions of sentences. 00:18:45.890 --> 00:18:47.830 align:middle line:84% And then for all these billion sentences, 00:18:47.830 --> 00:18:50.770 align:middle line:84% we can literally go and count for every pair of words 00:18:50.770 --> 00:18:52.770 align:middle line:84% how many times are both these words showing up 00:18:52.770 --> 00:18:53.645 align:middle line:90% in the same sentence. 00:18:53.645 --> 00:18:55.970 align:middle line:90% 00:18:55.970 --> 00:18:58.130 align:middle line:90% And we call this co-occurrence. 00:18:58.130 --> 00:19:00.730 align:middle line:84% The words are co-occurring in the sentence. 00:19:00.730 --> 00:19:03.210 align:middle line:84% And it doesn't have to be next to each other. 00:19:03.210 --> 00:19:05.770 align:middle line:84% We know that in complicated words, 00:19:05.770 --> 00:19:07.810 align:middle line:84% a word at the very end of the sentence 00:19:07.810 --> 00:19:10.690 align:middle line:84% could actually alter the meaning Its meaning could be altered 00:19:10.690 --> 00:19:12.590 align:middle line:84% by a word that happened in the very beginning of the sentence, 00:19:12.590 --> 00:19:14.290 align:middle line:84% and it could be a really long sentence. 00:19:14.290 --> 00:19:16.330 align:middle line:84% So we take the whole sentence and say, 00:19:16.330 --> 00:19:18.550 align:middle line:84% are two words co-occurring in the sentence? 00:19:18.550 --> 00:19:19.150 align:middle line:90% Yes or no? 00:19:19.150 --> 00:19:20.730 align:middle line:90% And we just count them up. 00:19:20.730 --> 00:19:27.800 align:middle line:84% And when we do that, we will get something like this. 00:19:27.800 --> 00:19:31.620 align:middle line:84% So this just captures what I've been talking about. 00:19:31.620 --> 00:19:34.120 align:middle line:84% Identify all the words that occur, let's say, in Wikipedia. 00:19:34.120 --> 00:19:36.480 align:middle line:84% And then for every sentence, you look at every word pair 00:19:36.480 --> 00:19:37.897 align:middle line:84% and count the number of times they 00:19:37.897 --> 00:19:41.640 align:middle line:84% appear in the same sentence across all those sentences. 00:19:41.640 --> 00:19:44.760 align:middle line:84% This is a word-word co-occurrence matrix. 00:19:44.760 --> 00:19:47.673 align:middle line:84% So for example, let's assume that you took all of Wikipedia, 00:19:47.673 --> 00:19:49.340 align:middle line:84% looked at all the words, distinct words, 00:19:49.340 --> 00:19:52.420 align:middle line:84% and you found there were 500,000 words. 00:19:52.420 --> 00:19:56.840 align:middle line:84% So there are 500,000 words here in the columns. 00:19:56.840 --> 00:20:00.055 align:middle line:90% 500,000 words on the rows. 00:20:00.055 --> 00:20:01.560 align:middle line:90% So columns and rows. 00:20:01.560 --> 00:20:05.000 align:middle line:84% And then you go, and each cell of this table 00:20:05.000 --> 00:20:07.760 align:middle line:84% basically has a number that you calculate 00:20:07.760 --> 00:20:10.960 align:middle line:84% which is the number of times the word in the row and the word 00:20:10.960 --> 00:20:13.420 align:middle line:84% in the column happened to show up in the same sentence. 00:20:13.420 --> 00:20:14.320 align:middle line:90% That's it. 00:20:14.320 --> 00:20:18.220 align:middle line:84% So for instance, if you look at deep and learning-- 00:20:18.220 --> 00:20:20.400 align:middle line:84% the word deep and the word learning-- 00:20:20.400 --> 00:20:25.180 align:middle line:84% maybe those two words occurred in the same sentence, 00:20:25.180 --> 00:20:28.430 align:middle line:90% maybe 3,025 times. 00:20:28.430 --> 00:20:31.430 align:middle line:84% 3,025 sentences across all of Wikipedia. 00:20:31.430 --> 00:20:36.030 align:middle line:84% You put 3,025 right in that cell. 00:20:36.030 --> 00:20:38.950 align:middle line:84% Many words are unlikely to appear in the same sentence. 00:20:38.950 --> 00:20:41.010 align:middle line:84% So much of this matrix is going to be 0. 00:20:41.010 --> 00:20:44.280 align:middle line:90% 00:20:44.280 --> 00:20:50.030 align:middle line:84% But we fundamentally form this co-occurrence matrix. 00:20:50.030 --> 00:20:55.030 align:middle line:84% This matrix essentially embodies all the context information 00:20:55.030 --> 00:21:00.810 align:middle line:84% that we can work with in a very compact, beautiful, elegant way. 00:21:00.810 --> 00:21:03.253 align:middle line:90% 00:21:03.253 --> 00:21:04.670 align:middle line:84% And using this, we're going to try 00:21:04.670 --> 00:21:07.390 align:middle line:84% to figure out what the word embeddings actually 00:21:07.390 --> 00:21:08.170 align:middle line:90% are going to be. 00:21:08.170 --> 00:21:11.710 align:middle line:90% 00:21:11.710 --> 00:21:14.310 align:middle line:84% So by the way, the approach I'm describing here 00:21:14.310 --> 00:21:17.100 align:middle line:84% to calculate standalone embeddings is called GloVe. 00:21:17.100 --> 00:21:20.590 align:middle line:90% 00:21:20.590 --> 00:21:21.710 align:middle line:90% It's called GloVe. 00:21:21.710 --> 00:21:26.820 align:middle line:84% And standalone embeddings first came on to the NLP deep learning 00:21:26.820 --> 00:21:27.620 align:middle line:90% scene-- 00:21:27.620 --> 00:21:29.840 align:middle line:90% there were two ways of doing it. 00:21:29.840 --> 00:21:31.360 align:middle line:90% One was called word2vec. 00:21:31.360 --> 00:21:32.620 align:middle line:90% Word2vec. 00:21:32.620 --> 00:21:34.380 align:middle line:90% The other one is GloVe. 00:21:34.380 --> 00:21:35.820 align:middle line:90% And they are both comparable. 00:21:35.820 --> 00:21:38.620 align:middle line:84% They use slightly different mechanisms of doing this. 00:21:38.620 --> 00:21:40.620 align:middle line:84% We went with word for this lecture 00:21:40.620 --> 00:21:43.780 align:middle line:84% because I think it's actually a little easier to understand, 00:21:43.780 --> 00:21:47.580 align:middle line:90% and equally effective. 00:21:47.580 --> 00:21:49.140 align:middle line:90% So this is what we have. 00:21:49.140 --> 00:21:51.260 align:middle line:84% And so what we want to do is we want 00:21:51.260 --> 00:21:53.060 align:middle line:84% to learn these embedding vectors that 00:21:53.060 --> 00:21:56.580 align:middle line:84% can be used to essentially approximate this matrix. 00:21:56.580 --> 00:22:00.100 align:middle line:84% If we can find vectors that can actually approximate 00:22:00.100 --> 00:22:03.300 align:middle line:84% this matrix, then hopefully those vectors do, in fact, 00:22:03.300 --> 00:22:05.840 align:middle line:84% capture some notion of what the words actually mean. 00:22:05.840 --> 00:22:07.460 align:middle line:90% So let me put it differently. 00:22:07.460 --> 00:22:10.100 align:middle line:90% You come to me with this matrix. 00:22:10.100 --> 00:22:12.060 align:middle line:90% And you say, OK. 00:22:12.060 --> 00:22:14.208 align:middle line:84% Rama, do you have embeddings for me. 00:22:14.208 --> 00:22:15.000 align:middle line:90% And I'm like, yeah. 00:22:15.000 --> 00:22:16.740 align:middle line:84% I reach into my bag, and I'm like, OK. 00:22:16.740 --> 00:22:19.793 align:middle line:84% Every one of those 500,000 words, I have an embedding. 00:22:19.793 --> 00:22:21.460 align:middle line:84% Let's ignore for a moment how I actually 00:22:21.460 --> 00:22:22.660 align:middle line:90% calculated the embeddings. 00:22:22.660 --> 00:22:24.060 align:middle line:90% I have the embeddings. 00:22:24.060 --> 00:22:28.820 align:middle line:84% How will you know if my embeddings are any good? 00:22:28.820 --> 00:22:30.300 align:middle line:90% How will you know? 00:22:30.300 --> 00:22:34.313 align:middle line:84% How can you actually assess if the embeddings are any good? 00:22:34.313 --> 00:22:35.980 align:middle line:84% Well, you can certainly say, OK, give me 00:22:35.980 --> 00:22:37.340 align:middle line:84% the embeddings for movie and film, 00:22:37.340 --> 00:22:39.180 align:middle line:84% and you can see if they're really close by. 00:22:39.180 --> 00:22:42.020 align:middle line:84% If you look at the embedding for movie and tensor-- 00:22:42.020 --> 00:22:43.900 align:middle line:90% and hopefully they'll far away. 00:22:43.900 --> 00:22:47.380 align:middle line:84% But you'll never get done, right? 00:22:47.380 --> 00:22:51.300 align:middle line:84% How can you systematically evaluate this? 00:22:51.300 --> 00:22:54.700 align:middle line:84% Well, what if you could actually-- what if I come to you 00:22:54.700 --> 00:22:57.420 align:middle line:84% and say, not only am I going to give you an embedding, 00:22:57.420 --> 00:23:00.500 align:middle line:84% here is a procedure which you can use with these embeddings 00:23:00.500 --> 00:23:02.120 align:middle line:90% to validate how good they are. 00:23:02.120 --> 00:23:03.320 align:middle line:90% And here is the procedure. 00:23:03.320 --> 00:23:05.940 align:middle line:84% What you can do is you can use the embedding 00:23:05.940 --> 00:23:10.020 align:middle line:84% to recreate the co-occurrence matrix. 00:23:10.020 --> 00:23:12.460 align:middle line:84% And if the recreated co-occurrence matrix actually 00:23:12.460 --> 00:23:15.100 align:middle line:84% matches the real matrix, well, these embeddings 00:23:15.100 --> 00:23:17.223 align:middle line:90% probably are pretty good. 00:23:17.223 --> 00:23:19.140 align:middle line:84% Remember, the whole point of the co-occurrence 00:23:19.140 --> 00:23:20.960 align:middle line:84% is to handle this context information. 00:23:20.960 --> 00:23:23.150 align:middle line:84% So if the embeddings can actually recreate them, 00:23:23.150 --> 00:23:24.998 align:middle line:84% reconstruct them pretty close, right? 00:23:24.998 --> 00:23:26.040 align:middle line:90% It will never be perfect. 00:23:26.040 --> 00:23:28.190 align:middle line:84% But if it comes pretty close, they'll be like, wow, OK. 00:23:28.190 --> 00:23:29.730 align:middle line:84% These embeddings do mean something. 00:23:29.730 --> 00:23:31.110 align:middle line:84% So if it turns out, for instance, 00:23:31.110 --> 00:23:35.610 align:middle line:84% that the matrix has 3,000 possible value of 3,000 for deep 00:23:35.610 --> 00:23:43.530 align:middle line:84% end learning and values of, say, 50 for extreme learning-- 00:23:43.530 --> 00:23:47.450 align:middle line:84% under embedding comes in and says 3,002 for the first one 00:23:47.450 --> 00:23:49.250 align:middle line:84% and 48 for the second one, we'll be like, 00:23:49.250 --> 00:23:51.545 align:middle line:90% we'll be pretty impressed. 00:23:51.545 --> 00:23:54.050 align:middle line:84% [INAUDIBLE] needed to be that close unless it was actually 00:23:54.050 --> 00:23:55.930 align:middle line:90% capturing something. 00:23:55.930 --> 00:23:57.530 align:middle line:84% So that's what we're going to do. 00:23:57.530 --> 00:24:00.250 align:middle line:84% And so we're going to take this logic of saying 00:24:00.250 --> 00:24:04.050 align:middle line:84% find embeddings that can approximate what we actually 00:24:04.050 --> 00:24:06.410 align:middle line:90% see in Wikipedia. 00:24:06.410 --> 00:24:08.970 align:middle line:84% And we're going to use that idea to actually build a model 00:24:08.970 --> 00:24:12.370 align:middle line:84% and learn from it using nothing more than basically 00:24:12.370 --> 00:24:13.310 align:middle line:90% linear regression. 00:24:13.310 --> 00:24:16.170 align:middle line:90% 00:24:16.170 --> 00:24:18.170 align:middle line:84% And here, you are thinking the linear regression 00:24:18.170 --> 00:24:20.100 align:middle line:84% is useless now that you've graduated to machine learning, 00:24:20.100 --> 00:24:20.600 align:middle line:90% right? 00:24:20.600 --> 00:24:23.280 align:middle line:90% 00:24:23.280 --> 00:24:25.160 align:middle line:84% So we can think of the embedding vectors 00:24:25.160 --> 00:24:29.000 align:middle line:84% that we want to figure out as just the weights in a model 00:24:29.000 --> 00:24:31.360 align:middle line:90% in a little regression. 00:24:31.360 --> 00:24:33.200 align:middle line:84% We can think of the co-occurrence matrix 00:24:33.200 --> 00:24:36.120 align:middle line:84% as just the data we're going to use in this model 00:24:36.120 --> 00:24:37.760 align:middle line:90% to estimate these weights. 00:24:37.760 --> 00:24:42.240 align:middle line:84% And the model we're going to use is something like this. 00:24:42.240 --> 00:24:45.015 align:middle line:84% So first, I have to inflict some notation on you. 00:24:45.015 --> 00:24:46.640 align:middle line:84% We will denote the co-occurrence matrix 00:24:46.640 --> 00:24:50.000 align:middle line:90% of, say, words I and J as Xij. 00:24:50.000 --> 00:24:51.720 align:middle line:90% Xij is just data. 00:24:51.720 --> 00:24:52.900 align:middle line:90% It's just data, OK? 00:24:52.900 --> 00:24:55.680 align:middle line:90% It's not a variable, it's data. 00:24:55.680 --> 00:24:58.715 align:middle line:84% And then we'll denote an embedding vector for each word. 00:24:58.715 --> 00:25:00.840 align:middle line:84% So we're going to calculate a vector for each word. 00:25:00.840 --> 00:25:03.200 align:middle line:90% So we'll call it wi. 00:25:03.200 --> 00:25:06.280 align:middle line:84% wi is the embedding vector for each word. 00:25:06.280 --> 00:25:09.920 align:middle line:84% And we will also assume that some words are just 00:25:09.920 --> 00:25:11.100 align:middle line:90% inherently very popular. 00:25:11.100 --> 00:25:13.960 align:middle line:84% They're going to show up all the time, like the word the. 00:25:13.960 --> 00:25:17.160 align:middle line:84% So we'll assume that every word has some natural frequency 00:25:17.160 --> 00:25:21.830 align:middle line:84% of occurring, like movie versus flick, the versus tensor. 00:25:21.830 --> 00:25:23.910 align:middle line:84% So we want the vectors to capture 00:25:23.910 --> 00:25:27.310 align:middle line:84% the co-occurrence patterns independent of how naturally 00:25:27.310 --> 00:25:29.390 align:middle line:90% frequent the words are. 00:25:29.390 --> 00:25:31.770 align:middle line:84% And so to capture this natural frequency, 00:25:31.770 --> 00:25:34.190 align:middle line:84% we will assign a bias, or bi, to each word 00:25:34.190 --> 00:25:35.530 align:middle line:90% that we're going to calculate. 00:25:35.530 --> 00:25:38.222 align:middle line:84% And all this will become clear in just a moment. 00:25:38.222 --> 00:25:41.448 align:middle line:84% So with this setup, basically what we're saying 00:25:41.448 --> 00:25:42.490 align:middle line:90% is something very simple. 00:25:42.490 --> 00:25:45.070 align:middle line:84% We're saying, look, this co-occurrence matrix that we 00:25:45.070 --> 00:25:47.510 align:middle line:84% have that we're able to compute-- 00:25:47.510 --> 00:25:52.430 align:middle line:84% it came about because in truth, in reality, in nature, 00:25:52.430 --> 00:25:55.590 align:middle line:84% there are these embedding vectors for every word. 00:25:55.590 --> 00:25:58.110 align:middle line:84% There are these biases, bi, by for every word. 00:25:58.110 --> 00:26:01.150 align:middle line:84% And every co-occurrence number that you see just 00:26:01.150 --> 00:26:03.730 align:middle line:84% came about because under the hood, 00:26:03.730 --> 00:26:07.110 align:middle line:84% mother nature grabbed the bias number for the word 00:26:07.110 --> 00:26:09.350 align:middle line:84% i, the bias number for the word j, 00:26:09.350 --> 00:26:12.230 align:middle line:84% took the two embedding vectors which only mother nature knows 00:26:12.230 --> 00:26:15.050 align:middle line:84% at this point, did the dot product of them, added them, 00:26:15.050 --> 00:26:16.762 align:middle line:84% and that's how we get this number. 00:26:16.762 --> 00:26:18.220 align:middle line:84% So it basically says the number you 00:26:18.220 --> 00:26:22.440 align:middle line:84% see is the sum of the inherent popularity of the first word, 00:26:22.440 --> 00:26:24.540 align:middle line:84% plus the inherent popularity of the second word, 00:26:24.540 --> 00:26:29.060 align:middle line:84% plus the way in which these two words connect to each other. 00:26:29.060 --> 00:26:30.020 align:middle line:90% That's it. 00:26:30.020 --> 00:26:32.940 align:middle line:84% And you will agree with me that it literally 00:26:32.940 --> 00:26:34.860 align:middle line:90% can't get simpler than this. 00:26:34.860 --> 00:26:36.565 align:middle line:84% If I tell you, hey, here are two things. 00:26:36.565 --> 00:26:38.440 align:middle line:84% I want you to tell me how connected they are. 00:26:38.440 --> 00:26:40.760 align:middle line:84% You'll be like, well, let's take the first one. 00:26:40.760 --> 00:26:42.740 align:middle line:84% Figure out how inherently popular 00:26:42.740 --> 00:26:44.160 align:middle line:90% it is, inherent popularity. 00:26:44.160 --> 00:26:46.452 align:middle line:84% And then, of course, you to worry about the connection, 00:26:46.452 --> 00:26:47.620 align:middle line:90% so do a dot product. 00:26:47.620 --> 00:26:48.120 align:middle line:90% That's it. 00:26:48.120 --> 00:26:50.380 align:middle line:90% Those three things. 00:26:50.380 --> 00:26:51.700 align:middle line:90% So this is what we have. 00:26:51.700 --> 00:26:58.060 align:middle line:84% Now you may have seen from your good old linear regression that 00:26:58.060 --> 00:27:03.020 align:middle line:84% whenever your dependent variable happens to be positive-- 00:27:03.020 --> 00:27:05.140 align:middle line:90% guaranteed to be positive-- 00:27:05.140 --> 00:27:08.380 align:middle line:84% and it ends up having a big range, 00:27:08.380 --> 00:27:10.980 align:middle line:84% we always advise your folks to take 00:27:10.980 --> 00:27:14.580 align:middle line:84% the logarithmic transformation to squash it into a narrow range 00:27:14.580 --> 00:27:18.460 align:middle line:84% because that will make these models much more well-behaved. 00:27:18.460 --> 00:27:21.220 align:middle line:84% Regression if the y value is like a huge range. 00:27:21.220 --> 00:27:22.820 align:middle line:84% The canonical example is that if you 00:27:22.820 --> 00:27:26.520 align:middle line:84% are trying to model the net worth of people, 00:27:26.520 --> 00:27:29.060 align:middle line:84% it's going to have a long right tail with people like Elon 00:27:29.060 --> 00:27:31.500 align:middle line:84% and Jeff and so on the right side, and the rest of us 00:27:31.500 --> 00:27:32.280 align:middle line:90% on the left. 00:27:32.280 --> 00:27:35.135 align:middle line:84% So to model this big, long tail distribution, 00:27:35.135 --> 00:27:37.260 align:middle line:84% you just take the logarithm, just squash everything 00:27:37.260 --> 00:27:38.480 align:middle line:90% to a very narrow range. 00:27:38.480 --> 00:27:41.540 align:middle line:84% And that will make regression much better behaved. 00:27:41.540 --> 00:27:45.420 align:middle line:84% Here, most of the counts are going to be 0, 00:27:45.420 --> 00:27:49.220 align:middle line:84% but some of the counts could be very high. 00:27:49.220 --> 00:27:52.107 align:middle line:84% And therefore, we wanted to-- if you take the logarithm, 00:27:52.107 --> 00:27:53.440 align:middle line:90% it makes it much better behaved. 00:27:53.440 --> 00:27:54.860 align:middle line:90% So take the logarithm here. 00:27:54.860 --> 00:27:56.400 align:middle line:90% So this is actually our model. 00:27:56.400 --> 00:27:57.460 align:middle line:90% That's it. 00:27:57.460 --> 00:27:59.780 align:middle line:84% And I know that many of the numbers are 0, and log of 0 00:27:59.780 --> 00:28:02.500 align:middle line:84% is not defined, so we can just add the number 1 00:28:02.500 --> 00:28:07.460 align:middle line:84% to all the numbers to avoid that kind of technical arithmetic 00:28:07.460 --> 00:28:07.968 align:middle line:90% problems. 00:28:07.968 --> 00:28:09.760 align:middle line:84% But this, conceptually, is what's going on. 00:28:09.760 --> 00:28:11.500 align:middle line:84% This is the model we want to calculate. 00:28:11.500 --> 00:28:14.170 align:middle line:84% So given that we have essentially 00:28:14.170 --> 00:28:17.250 align:middle line:84% postulated this model and we have 00:28:17.250 --> 00:28:19.190 align:middle line:84% this data, this co-occurrence matrix, 00:28:19.190 --> 00:28:21.030 align:middle line:84% how can we actually find the weights? 00:28:21.030 --> 00:28:24.130 align:middle line:84% How can we actually find the b's and the w's? 00:28:24.130 --> 00:28:25.730 align:middle line:90% What will what should we do? 00:28:25.730 --> 00:28:27.930 align:middle line:84% Go back to the fundamentals of regression. 00:28:27.930 --> 00:28:29.330 align:middle line:90% Think about it conceptually. 00:28:29.330 --> 00:28:31.640 align:middle line:84% You have some model, which has some weights. 00:28:31.640 --> 00:28:35.530 align:middle line:84% There's some data you can use to train the model. 00:28:35.530 --> 00:28:37.510 align:middle line:84% And you need to find the best set of weights. 00:28:37.510 --> 00:28:40.330 align:middle line:90% What does the best mean here? 00:28:40.330 --> 00:28:41.855 align:middle line:84% AUDIENCE: Probably the lowest error. 00:28:41.855 --> 00:28:43.730 align:middle line:84% RAMA RAMAKRISHNAN: The lowest error, exactly. 00:28:43.730 --> 00:28:46.323 align:middle line:84% There are many ways to measure error. 00:28:46.323 --> 00:28:47.990 align:middle line:84% What is the simplest thing we could use? 00:28:47.990 --> 00:28:49.740 align:middle line:84% So what you would do is you would actually 00:28:49.740 --> 00:28:52.050 align:middle line:84% do mean squared error, which is what you're getting at. 00:28:52.050 --> 00:28:53.250 align:middle line:84% You could take the actual thing, you 00:28:53.250 --> 00:28:55.375 align:middle line:84% could take the predicted thing, take the difference 00:28:55.375 --> 00:28:57.490 align:middle line:84% and square it, and minimize the sum of it. 00:28:57.490 --> 00:29:00.010 align:middle line:84% If your model exactly nails every number 00:29:00.010 --> 00:29:03.542 align:middle line:84% in the co-occurrence matrix, the rest is going to be 0. 00:29:03.542 --> 00:29:07.810 align:middle line:84% So what we do is we literally just do that. 00:29:07.810 --> 00:29:09.250 align:middle line:90% This is the data. 00:29:09.250 --> 00:29:10.770 align:middle line:84% This is the actual predicted value. 00:29:10.770 --> 00:29:14.500 align:middle line:84% Predicted value, actual value, difference squared, 00:29:14.500 --> 00:29:17.920 align:middle line:90% add them all up, minimize. 00:29:17.920 --> 00:29:19.400 align:middle line:90% OK? 00:29:19.400 --> 00:29:21.280 align:middle line:90% Yes? 00:29:21.280 --> 00:29:22.960 align:middle line:84% AUDIENCE: I'm kind of lost where how 00:29:22.960 --> 00:29:24.560 align:middle line:90% is this capturing the context? 00:29:24.560 --> 00:29:28.760 align:middle line:84% Because unless my input data is having that context, 00:29:28.760 --> 00:29:32.200 align:middle line:84% how will this actually differentiate based on where 00:29:32.200 --> 00:29:34.320 align:middle line:90% the particular word is used? 00:29:34.320 --> 00:29:37.080 align:middle line:84% RAMA RAMAKRISHNAN: The way the word is-- 00:29:37.080 --> 00:29:39.400 align:middle line:84% so let's take two words, like deep and learning. 00:29:39.400 --> 00:29:41.200 align:middle line:84% AUDIENCE: No, let's take one word 00:29:41.200 --> 00:29:43.080 align:middle line:84% and change it according to the final. 00:29:43.080 --> 00:29:44.840 align:middle line:90% RAMA RAMAKRISHNAN: OK. 00:29:44.840 --> 00:29:45.740 align:middle line:90% Sorry, go ahead. 00:29:45.740 --> 00:29:48.120 align:middle line:84% AUDIENCE: Yeah, so basically let's say I have a banana. 00:29:48.120 --> 00:29:49.980 align:middle line:90% So it's a fruit in some context. 00:29:49.980 --> 00:29:52.200 align:middle line:84% And I could be saying, he's going bananas. 00:29:52.200 --> 00:29:54.230 align:middle line:90% So whatever, right? 00:29:54.230 --> 00:29:57.220 align:middle line:84% So now these are two different contexts in my understanding. 00:29:57.220 --> 00:29:59.560 align:middle line:84% And my same model needs to be able to tell me 00:29:59.560 --> 00:30:01.860 align:middle line:84% that banana is the right word in this context, 00:30:01.860 --> 00:30:05.400 align:middle line:84% but wrong word in this context, or correct in both contexts. 00:30:05.400 --> 00:30:06.400 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:30:06.400 --> 00:30:07.340 align:middle line:90% Very good question. 00:30:07.340 --> 00:30:09.260 align:middle line:84% So let's actually spend a minute on that. 00:30:09.260 --> 00:30:10.110 align:middle line:90% Good question. 00:30:10.110 --> 00:30:13.350 align:middle line:90% I'm going to swap to my iPad. 00:30:13.350 --> 00:30:18.490 align:middle line:84% So let's assume that this is our co-occurrence matrix. 00:30:18.490 --> 00:30:20.830 align:middle line:84% And then we have words going from all the way 00:30:20.830 --> 00:30:23.150 align:middle line:90% to-- let's say zebra. 00:30:23.150 --> 00:30:25.750 align:middle line:84% These are all the words in the vocabulary, 00:30:25.750 --> 00:30:29.430 align:middle line:84% and we have a through zebra here. 00:30:29.430 --> 00:30:37.000 align:middle line:84% And now what we have is we have apple and banana. 00:30:37.000 --> 00:30:40.336 align:middle line:90% 00:30:40.336 --> 00:30:42.790 align:middle line:84% So basically what's going on at this point 00:30:42.790 --> 00:30:49.070 align:middle line:84% is that here, every number here measures for every word 00:30:49.070 --> 00:30:51.630 align:middle line:84% here, how many times that word and apple show up 00:30:51.630 --> 00:30:53.260 align:middle line:90% in the same sentence. 00:30:53.260 --> 00:30:57.390 align:middle line:84% It is not measuring, to your point, how many times apple 00:30:57.390 --> 00:30:58.570 align:middle line:90% and banana are showing up. 00:30:58.570 --> 00:31:00.630 align:middle line:84% It's measuring how many times apple 00:31:00.630 --> 00:31:02.103 align:middle line:90% is showing up in each sentence. 00:31:02.103 --> 00:31:06.430 align:middle line:84% Now, if apple and banana are interchangeable, 00:31:06.430 --> 00:31:11.760 align:middle line:84% what do we expect these two rows of numbers to look like? 00:31:11.760 --> 00:31:14.580 align:middle line:84% Let's assume that apple and banana are perfect synonyms, 00:31:14.580 --> 00:31:15.460 align:middle line:90% just for argument. 00:31:15.460 --> 00:31:17.220 align:middle line:84% Let's say it's a perfect synonyms. 00:31:17.220 --> 00:31:24.020 align:middle line:84% What do we expect these two numbers to look like? 00:31:24.020 --> 00:31:25.540 align:middle line:90% Very similar. 00:31:25.540 --> 00:31:30.120 align:middle line:84% So if two words are related, their entry row vectors, 00:31:30.120 --> 00:31:32.620 align:middle line:84% the co-occurrence matrix are going to be very, very similar. 00:31:32.620 --> 00:31:36.020 align:middle line:84% So that is how the context comes into the co-occurrence matrix. 00:31:36.020 --> 00:31:39.460 align:middle line:84% So what we want to do is we want to find if embeddings can 00:31:39.460 --> 00:31:44.180 align:middle line:84% recreate the same pattern of numbers in these two rows, 00:31:44.180 --> 00:31:47.900 align:middle line:84% it's actually capturing the underlying context. 00:31:47.900 --> 00:31:51.180 align:middle line:84% So words which are similar will zig and zag together 00:31:51.180 --> 00:31:53.340 align:middle line:84% the same way through the co-occurrence matrix, 00:31:53.340 --> 00:31:54.560 align:middle line:90% and that's where it comes in. 00:31:54.560 --> 00:31:57.460 align:middle line:90% 00:31:57.460 --> 00:31:58.900 align:middle line:90% yeah 00:31:58.900 --> 00:32:01.760 align:middle line:84% AUDIENCE: What's on the diagonal of the co-occurrence matrix? 00:32:01.760 --> 00:32:03.892 align:middle line:90% Like, if apple shows up twice? 00:32:03.892 --> 00:32:05.100 align:middle line:90% RAMA RAMAKRISHNAN: Oh, I see. 00:32:05.100 --> 00:32:08.820 align:middle line:84% So here, you can just ignore the diagonal, typically, 00:32:08.820 --> 00:32:11.490 align:middle line:84% because all the action is off diagonal entries. 00:32:11.490 --> 00:32:15.340 align:middle line:90% 00:32:15.340 --> 00:32:17.820 align:middle line:90% So that's basically the idea. 00:32:17.820 --> 00:32:20.180 align:middle line:84% And if words which are very similar 00:32:20.180 --> 00:32:23.020 align:middle line:84% will have a very similar pattern of numbers, and then 00:32:23.020 --> 00:32:25.780 align:middle line:84% any embeddings that can actually recreate 00:32:25.780 --> 00:32:28.500 align:middle line:84% the same pattern of numbers is capturing the underlying reality 00:32:28.500 --> 00:32:29.700 align:middle line:90% of what's going on. 00:32:29.700 --> 00:32:34.000 align:middle line:84% If words are unrelated, those two vectors-- 00:32:34.000 --> 00:32:36.080 align:middle line:84% let's say that the word you have is-- 00:32:36.080 --> 00:32:40.420 align:middle line:90% 00:32:40.420 --> 00:32:42.628 align:middle line:90% let's assume the word is-- 00:32:42.628 --> 00:32:43.920 align:middle line:90% you know what I'm going to say. 00:32:43.920 --> 00:32:46.300 align:middle line:90% Tensor. 00:32:46.300 --> 00:32:50.440 align:middle line:84% These two vectors won't have any connection to each other. 00:32:50.440 --> 00:32:51.940 align:middle line:84% Which means if you look at something 00:32:51.940 --> 00:32:53.690 align:middle line:84% like the correlation of those two vectors, 00:32:53.690 --> 00:32:56.680 align:middle line:90% it's going to be around zero. 00:32:56.680 --> 00:32:59.460 align:middle line:84% But words which are interchangeable 00:32:59.460 --> 00:33:01.620 align:middle line:84% will have a very high correlation. 00:33:01.620 --> 00:33:04.420 align:middle line:84% Words which are antonyms and never show up in the same place 00:33:04.420 --> 00:33:07.050 align:middle line:84% together may have a highly negative correlation, 00:33:07.050 --> 00:33:08.690 align:middle line:90% close to minus 1, for instance. 00:33:08.690 --> 00:33:10.410 align:middle line:84% So that's the intuition behind what's 00:33:10.410 --> 00:33:12.890 align:middle line:90% going on on these row vectors. 00:33:12.890 --> 00:33:15.530 align:middle line:84% And so the point is given this co-occurrence matrix is 00:33:15.530 --> 00:33:19.890 align:middle line:84% capturing all these word-word correlational structure, 00:33:19.890 --> 00:33:21.770 align:middle line:84% any embedding that can recreate it 00:33:21.770 --> 00:33:26.130 align:middle line:84% must have captured the structure as well because you can't 00:33:26.130 --> 00:33:28.410 align:middle line:84% recreate something like this with great fidelity 00:33:28.410 --> 00:33:31.850 align:middle line:84% unless you have some notion of what's going on under the hood. 00:33:31.850 --> 00:33:33.690 align:middle line:90% That's the basic idea. 00:33:33.690 --> 00:33:34.435 align:middle line:90% Yeah? 00:33:34.435 --> 00:33:35.810 align:middle line:84% AUDIENCE: So just connecting just 00:33:35.810 --> 00:33:39.210 align:middle line:84% from this question-- so in that example then, 00:33:39.210 --> 00:33:41.390 align:middle line:84% banana is a fruit and apple is a fruit as well. 00:33:41.390 --> 00:33:42.852 align:middle line:90% Banana and apple are synonyms. 00:33:42.852 --> 00:33:44.810 align:middle line:84% And you are going mad or you are going bananas, 00:33:44.810 --> 00:33:45.830 align:middle line:90% and that comes together? 00:33:45.830 --> 00:33:47.190 align:middle line:90% Is that how we read it? 00:33:47.190 --> 00:33:47.490 align:middle line:90% RAMA RAMAKRISHNAN: Oh, I see. 00:33:47.490 --> 00:33:48.030 align:middle line:90% You're going mad. 00:33:48.030 --> 00:33:48.905 align:middle line:90% You're going bananas. 00:33:48.905 --> 00:33:52.370 align:middle line:84% Yeah, so those will also have some correlation structure 00:33:52.370 --> 00:33:54.890 align:middle line:84% to it, which the embeddings will hopefully catch. 00:33:54.890 --> 00:33:59.530 align:middle line:84% But words like banana, which are very-- 00:33:59.530 --> 00:34:01.050 align:middle line:84% the thing is it's called polysemy, 00:34:01.050 --> 00:34:03.110 align:middle line:90% where the word looks one way. 00:34:03.110 --> 00:34:03.860 align:middle line:90% It looks the same. 00:34:03.860 --> 00:34:05.220 align:middle line:90% It's like the word bank, right? 00:34:05.220 --> 00:34:06.240 align:middle line:84% It can mean very different things 00:34:06.240 --> 00:34:07.365 align:middle line:90% in many different contexts. 00:34:07.365 --> 00:34:10.040 align:middle line:84% So the embedding is going to be some average representation 00:34:10.040 --> 00:34:11.380 align:middle line:90% of it. 00:34:11.380 --> 00:34:12.980 align:middle line:84% But we're not happy with that average, 00:34:12.980 --> 00:34:14.920 align:middle line:84% and we will get around that average 00:34:14.920 --> 00:34:18.239 align:middle line:84% next week when we do contextual stuff. 00:34:18.239 --> 00:34:20.320 align:middle line:90% All right. 00:34:20.320 --> 00:34:21.920 align:middle line:90% So that's what we have here. 00:34:21.920 --> 00:34:23.429 align:middle line:90% So to go back to this thing-- 00:34:23.429 --> 00:34:26.719 align:middle line:90% 00:34:26.719 --> 00:34:28.738 align:middle line:90% so what we can do is-- yeah? 00:34:28.738 --> 00:34:30.280 align:middle line:84% AUDIENCE: After getting that initial, 00:34:30.280 --> 00:34:33.179 align:middle line:84% how did we get the mean square error in this? 00:34:33.179 --> 00:34:36.820 align:middle line:84% Because we split the embedding you put in the data and then we 00:34:36.820 --> 00:34:37.320 align:middle line:90% get... 00:34:37.320 --> 00:34:39.653 align:middle line:84% RAMA RAMAKRISHNAN: We haven't calculated the embeddings. 00:34:39.653 --> 00:34:41.139 align:middle line:90% We are trying to calculate them. 00:34:41.139 --> 00:34:43.199 align:middle line:84% Those are just-- it's like in regression, you 00:34:43.199 --> 00:34:47.960 align:middle line:84% have beta 1 times x1 plus beta 2 times x2 kind of thing. 00:34:47.960 --> 00:34:51.040 align:middle line:84% The betas are what the regression produces for us. 00:34:51.040 --> 00:34:52.699 align:middle line:90% The embeddings are exactly that. 00:34:52.699 --> 00:34:55.280 align:middle line:84% They're just coefficients that we're trying to figure out. 00:34:55.280 --> 00:34:59.520 align:middle line:84% The data is only the x's, the Xij. 00:34:59.520 --> 00:35:02.670 align:middle line:84% And so this is what we're trying to calculate. 00:35:02.670 --> 00:35:04.710 align:middle line:84% And so what you can do is you can actually 00:35:04.710 --> 00:35:08.310 align:middle line:84% start with some random values for these things 00:35:08.310 --> 00:35:11.390 align:middle line:84% and then keep on trying to improve 00:35:11.390 --> 00:35:16.310 align:middle line:84% to minimize the error starting from these random values. 00:35:16.310 --> 00:35:18.470 align:middle line:84% Are you aware of any algorithm that 00:35:18.470 --> 00:35:20.390 align:middle line:84% allows us to take random values starting point 00:35:20.390 --> 00:35:22.090 align:middle line:84% and then minimize some notion of error? 00:35:22.090 --> 00:35:26.194 align:middle line:90% 00:35:26.194 --> 00:35:27.750 align:middle line:90% AUDIENCE: Who takes the random? 00:35:27.750 --> 00:35:32.297 align:middle line:84% How can we order it random if it has to be random? 00:35:32.297 --> 00:35:34.630 align:middle line:84% RAMA RAMAKRISHNAN: How do you know it's actually random? 00:35:34.630 --> 00:35:35.590 align:middle line:90% Oh. 00:35:35.590 --> 00:35:39.030 align:middle line:84% So that's actually a very deep question. 00:35:39.030 --> 00:35:43.430 align:middle line:84% And so it's actually a tough question because ultimately, 00:35:43.430 --> 00:35:46.030 align:middle line:84% the random number is coming from a computer. 00:35:46.030 --> 00:35:47.830 align:middle line:84% And we know how the computer runs. 00:35:47.830 --> 00:35:50.030 align:middle line:84% It's deterministic, at the end of the day. 00:35:50.030 --> 00:35:53.990 align:middle line:84% So we actually use something called pseudorandom numbers. 00:35:53.990 --> 00:35:56.930 align:middle line:84% And there's a whole specialized field of math 00:35:56.930 --> 00:35:59.430 align:middle line:84% which essentially says, look, how can I 00:35:59.430 --> 00:36:03.020 align:middle line:84% get random numbers that are sufficiently random, even 00:36:03.020 --> 00:36:04.900 align:middle line:84% though they come from a non-random computer 00:36:04.900 --> 00:36:06.300 align:middle line:90% deterministic process? 00:36:06.300 --> 00:36:08.620 align:middle line:90% So we can talk offline about it. 00:36:08.620 --> 00:36:10.500 align:middle line:84% But fundamentally, all these systems 00:36:10.500 --> 00:36:12.540 align:middle line:84% have some random number generators built in. 00:36:12.540 --> 00:36:14.940 align:middle line:84% We just cross our fingers and hope for the best 00:36:14.940 --> 00:36:17.420 align:middle line:90% and just use them. 00:36:17.420 --> 00:36:19.640 align:middle line:90% So come back to this. 00:36:19.640 --> 00:36:22.220 align:middle line:84% We can start with random values for these weights, 00:36:22.220 --> 00:36:24.600 align:middle line:84% and then we can try to minimize the squared error. 00:36:24.600 --> 00:36:26.100 align:middle line:84% Are you folks aware of any algorithm 00:36:26.100 --> 00:36:28.420 align:middle line:90% that can help us do that? 00:36:28.420 --> 00:36:30.075 align:middle line:90% Yes? 00:36:30.075 --> 00:36:31.200 align:middle line:90% AUDIENCE: Gradient descent. 00:36:31.200 --> 00:36:33.060 align:middle line:84% RAMA RAMAKRISHNAN: Yes, gradient descent. 00:36:33.060 --> 00:36:34.980 align:middle line:90% Again, comes to the rescue. 00:36:34.980 --> 00:36:39.460 align:middle line:84% And since we are cool, we'll do stochastic gradient descent. 00:36:39.460 --> 00:36:40.920 align:middle line:90% So that's it. 00:36:40.920 --> 00:36:42.420 align:middle line:84% So gradient descent actually doesn't 00:36:42.420 --> 00:36:43.900 align:middle line:84% care what the function is, as long as you can 00:36:43.900 --> 00:36:45.240 align:middle line:90% calculate a derivative from it. 00:36:45.240 --> 00:36:47.660 align:middle line:84% As long as you calculate the gradient, you're good. 00:36:47.660 --> 00:36:50.926 align:middle line:84% So we can just run gradient descent on this thing. 00:36:50.926 --> 00:36:54.420 align:middle line:84% One key point here is that stochastic gradient descent 00:36:54.420 --> 00:36:57.140 align:middle line:90% work for any models. 00:36:57.140 --> 00:37:00.500 align:middle line:84% As long as you can calculate good gradients from them, 00:37:00.500 --> 00:37:04.220 align:middle line:84% it doesn't have to be a neural network. 00:37:04.220 --> 00:37:05.660 align:middle line:84% Any mathematical function, as long 00:37:05.660 --> 00:37:09.420 align:middle line:84% as it's differentiable and gives you a good gradient. 00:37:09.420 --> 00:37:11.540 align:middle line:84% So here, this is not a neural network, per se, 00:37:11.540 --> 00:37:14.180 align:middle line:84% but we can use gradient descent for it. 00:37:14.180 --> 00:37:19.720 align:middle line:84% So we do that, and when we are done, 00:37:19.720 --> 00:37:23.540 align:middle line:84% we would have calculated some nice embeddings. 00:37:23.540 --> 00:37:25.240 align:middle line:84% We can also calculate all these biases. 00:37:25.240 --> 00:37:26.360 align:middle line:84% But we don't need the biases anymore. 00:37:26.360 --> 00:37:28.360 align:middle line:84% We can just throw out the biases because we only 00:37:28.360 --> 00:37:31.700 align:middle line:84% care about the embeddings and how they connect to each other. 00:37:31.700 --> 00:37:33.420 align:middle line:90% Yeah? 00:37:33.420 --> 00:37:35.340 align:middle line:84% AUDIENCE: So when you're doing the regression, 00:37:35.340 --> 00:37:37.217 align:middle line:84% are you predicting the co-occurrence matrix? 00:37:37.217 --> 00:37:38.300 align:middle line:90% RAMA RAMAKRISHNAN: Mm-hmm. 00:37:38.300 --> 00:37:39.500 align:middle line:90% AUDIENCE: OK. 00:37:39.500 --> 00:37:40.625 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:37:40.625 --> 00:37:42.380 align:middle line:90% 00:37:42.380 --> 00:37:47.460 align:middle line:84% So actually, let me just show a very quick numerical example 00:37:47.460 --> 00:37:48.180 align:middle line:90% here. 00:37:48.180 --> 00:37:49.920 align:middle line:84% So let's say, for example, that-- 00:37:49.920 --> 00:37:53.580 align:middle line:90% 00:37:53.580 --> 00:37:54.900 align:middle line:90% you know what? 00:37:54.900 --> 00:37:57.170 align:middle line:90% Deep learning. 00:37:57.170 --> 00:38:00.450 align:middle line:84% So this is, say, w1 and this is w2. 00:38:00.450 --> 00:38:01.830 align:middle line:90% And this is the vector. 00:38:01.830 --> 00:38:06.410 align:middle line:84% And let's assume for a moment that it has two dimensions. 00:38:06.410 --> 00:38:07.930 align:middle line:90% Two dimensions. 00:38:07.930 --> 00:38:10.750 align:middle line:84% And we also need to calculate b1 and b2, which is just a number. 00:38:10.750 --> 00:38:14.710 align:middle line:90% 00:38:14.710 --> 00:38:17.290 align:middle line:84% And let's say the number for deep learning 00:38:17.290 --> 00:38:19.410 align:middle line:84% in the co-occurrence matrix, let's say 00:38:19.410 --> 00:38:21.730 align:middle line:90% it has occurred 104 times. 00:38:21.730 --> 00:38:27.170 align:middle line:84% So all we are doing is to say log of 104-- 00:38:27.170 --> 00:38:28.730 align:middle line:90% that is the actual value-- 00:38:28.730 --> 00:38:34.890 align:middle line:84% minus b1, which we don't know, plus B2, which we don't know. 00:38:34.890 --> 00:38:36.970 align:middle line:84% And then this thing here, let's just 00:38:36.970 --> 00:38:45.240 align:middle line:84% call it w1 one, w1 two, w2 one, w2 two. 00:38:45.240 --> 00:38:46.990 align:middle line:84% And then we're just doing the dot product, 00:38:46.990 --> 00:38:55.840 align:middle line:84% which is w1 one times w1 two plus w2 one, w2 two. 00:38:55.840 --> 00:38:58.645 align:middle line:90% So this is our prediction. 00:38:58.645 --> 00:39:00.020 align:middle line:84% Where is that cool laser pointer? 00:39:00.020 --> 00:39:03.560 align:middle line:90% Yeah, so this is our prediction. 00:39:03.560 --> 00:39:05.280 align:middle line:90% This is the actual. 00:39:05.280 --> 00:39:08.320 align:middle line:84% So all we do is to say, OK, this thing-- 00:39:08.320 --> 00:39:11.000 align:middle line:84% the difference, we're going to square it. 00:39:11.000 --> 00:39:13.040 align:middle line:84% And then we're going to do the same exact thing 00:39:13.040 --> 00:39:16.700 align:middle line:90% for every other word pair. 00:39:16.700 --> 00:39:18.880 align:middle line:84% And when we are done with all of that thing, 00:39:18.880 --> 00:39:21.880 align:middle line:84% we just take this whole thing and say gradient descent, 00:39:21.880 --> 00:39:23.960 align:middle line:90% minimize. 00:39:23.960 --> 00:39:27.120 align:middle line:84% So then it has to find the b's and the w's and everything 00:39:27.120 --> 00:39:29.360 align:middle line:90% for every pair, every word. 00:39:29.360 --> 00:39:31.890 align:middle line:84% So that's actually what's going on. 00:39:31.890 --> 00:39:32.390 align:middle line:90% Make sense? 00:39:32.390 --> 00:39:37.080 align:middle line:90% 00:39:37.080 --> 00:39:37.840 align:middle line:90% All right. 00:39:37.840 --> 00:39:45.120 align:middle line:84% So by the way, here, I said let's assume 00:39:45.120 --> 00:39:51.200 align:middle line:84% that the embeddings are just vectors which are dimension two. 00:39:51.200 --> 00:39:53.590 align:middle line:84% Well, that's an arbitrary decision 00:39:53.590 --> 00:39:55.690 align:middle line:84% that I made just to show you how it works 00:39:55.690 --> 00:39:57.550 align:middle line:90% because I was doing it by hand. 00:39:57.550 --> 00:39:59.350 align:middle line:84% But more generally, we get to choose 00:39:59.350 --> 00:40:02.070 align:middle line:90% how long these vectors are. 00:40:02.070 --> 00:40:05.205 align:middle line:84% And the longer the vector, the more interesting ways 00:40:05.205 --> 00:40:07.330 align:middle line:84% it can actually reproduce the co-occurrence matrix. 00:40:07.330 --> 00:40:08.790 align:middle line:90% It has more flexibility. 00:40:08.790 --> 00:40:13.350 align:middle line:84% But the longer the vector, what is the risk that you run? 00:40:13.350 --> 00:40:14.595 align:middle line:90% Overfitting. 00:40:14.595 --> 00:40:15.970 align:middle line:84% Because these are all parameters, 00:40:15.970 --> 00:40:16.887 align:middle line:90% at the end of the day. 00:40:16.887 --> 00:40:19.750 align:middle line:84% More parameters you have, the more risk of overfitting. 00:40:19.750 --> 00:40:22.710 align:middle line:84% So you get to choose how big these things can be. 00:40:22.710 --> 00:40:24.990 align:middle line:90% Yes? 00:40:24.990 --> 00:40:26.710 align:middle line:84% AUDIENCE: Don't you find it surprising 00:40:26.710 --> 00:40:29.670 align:middle line:84% that we're able to fit a model where we have a lot 00:40:29.670 --> 00:40:31.170 align:middle line:84% more parameters than we have data? 00:40:31.170 --> 00:40:33.253 align:middle line:84% Because usually when it comes to machine learning, 00:40:33.253 --> 00:40:33.970 align:middle line:90% we like sparsity. 00:40:33.970 --> 00:40:36.170 align:middle line:84% We like to not have a lot of parameters. 00:40:36.170 --> 00:40:38.530 align:middle line:84% But here, we're going to have, as you said, 00:40:38.530 --> 00:40:41.350 align:middle line:84% the number of dimensions times more parameters 00:40:41.350 --> 00:40:43.088 align:middle line:90% than we have of data points. 00:40:43.088 --> 00:40:45.130 align:middle line:84% RAMA RAMAKRISHNAN: Well, in this particular case, 00:40:45.130 --> 00:40:46.870 align:middle line:90% as it turns out-- 00:40:46.870 --> 00:40:49.510 align:middle line:84% let's assume that you only have 10 words. 00:40:49.510 --> 00:40:53.020 align:middle line:84% And for each word, let's assume that you have-- 00:40:53.020 --> 00:40:54.580 align:middle line:84% just to keep the math simple, you 00:40:54.580 --> 00:40:56.300 align:middle line:90% have a two dimensional vector. 00:40:56.300 --> 00:40:58.540 align:middle line:90% So 10 words times 2, that's 20. 00:40:58.540 --> 00:41:01.820 align:middle line:84% Plus you have 10 biases for the words, so there's another 10. 00:41:01.820 --> 00:41:02.860 align:middle line:90% That's 30. 00:41:02.860 --> 00:41:06.020 align:middle line:84% But 10 by 10, the matrix has 100 entries. 00:41:06.020 --> 00:41:09.500 align:middle line:84% So because of the matrix being order n squared matrix, 00:41:09.500 --> 00:41:12.980 align:middle line:84% you'll have a lot more numbers than parameters. 00:41:12.980 --> 00:41:17.220 align:middle line:84% In this particular case, you have more data than parameters. 00:41:17.220 --> 00:41:20.020 align:middle line:84% So that particular problem doesn't apply in this case, 00:41:20.020 --> 00:41:22.040 align:middle line:84% but that does show up in other cases. 00:41:22.040 --> 00:41:24.420 align:middle line:84% And there is some very interesting research 00:41:24.420 --> 00:41:26.620 align:middle line:84% in neural networks which suggests that oftentimes, 00:41:26.620 --> 00:41:30.620 align:middle line:84% the traditional assumptions of data and overfitting and all 00:41:30.620 --> 00:41:33.780 align:middle line:84% can all be called into question under some situations. 00:41:33.780 --> 00:41:36.440 align:middle line:84% I'm happy to tell you more offline, but if you're curious, 00:41:36.440 --> 00:41:39.260 align:middle line:84% just Google something called double descent 00:41:39.260 --> 00:41:42.500 align:middle line:90% to know what I mean. 00:41:42.500 --> 00:41:46.340 align:middle line:84% But in this case, it's not a problem. 00:41:46.340 --> 00:41:48.220 align:middle line:90% OK. 00:41:48.220 --> 00:41:49.940 align:middle line:84% So what that means is that we can choose 00:41:49.940 --> 00:41:51.300 align:middle line:90% how big these things are. 00:41:51.300 --> 00:41:56.362 align:middle line:84% So if you look at one-hot vectors, where there's a 1 00:41:56.362 --> 00:41:57.820 align:middle line:84% and everything else is 0, depending 00:41:57.820 --> 00:41:59.300 align:middle line:84% on the position of the word-- these 00:41:59.300 --> 00:42:03.580 align:middle line:84% are vectors as long as the vocabulary, as we saw earlier. 00:42:03.580 --> 00:42:07.420 align:middle line:84% Word embeddings, on the other hand, they can be very dense. 00:42:07.420 --> 00:42:09.722 align:middle line:84% The numbers that make up these embeddings, 00:42:09.722 --> 00:42:12.180 align:middle line:84% we actually want to figure out from the data what they are. 00:42:12.180 --> 00:42:14.120 align:middle line:90% So it can be anything. 00:42:14.120 --> 00:42:17.180 align:middle line:84% So the first dimension may stand for some combination 00:42:17.180 --> 00:42:23.000 align:middle line:84% of brightness plus speed plus animalness or something. 00:42:23.000 --> 00:42:24.780 align:middle line:90% We have no idea what it means. 00:42:24.780 --> 00:42:26.900 align:middle line:84% All we know is that it's able to reproduce 00:42:26.900 --> 00:42:29.260 align:middle line:84% the co-occurrence matrix really well, so it probably 00:42:29.260 --> 00:42:31.300 align:middle line:90% has figured something out. 00:42:31.300 --> 00:42:32.960 align:middle line:84% And so we can keep it really short. 00:42:32.960 --> 00:42:36.160 align:middle line:84% So the word embeddings tend to be very dense, 00:42:36.160 --> 00:42:38.962 align:middle line:84% meaning not zeros and ones, but some arbitrary numbers. 00:42:38.962 --> 00:42:40.420 align:middle line:84% It's very low dimensional, and it's 00:42:40.420 --> 00:42:42.436 align:middle line:90% of course learned from data. 00:42:42.436 --> 00:42:47.260 align:middle line:84% So once you do this, once you actually run GloVe on this data 00:42:47.260 --> 00:42:49.850 align:middle line:84% and do gradient descent and so on and so forth, 00:42:49.850 --> 00:42:51.410 align:middle line:84% you actually come up with embeddings. 00:42:51.410 --> 00:42:53.510 align:middle line:84% And then you can actually plot the embeddings. 00:42:53.510 --> 00:42:57.370 align:middle line:84% You can take these embeddings and just plot them. 00:42:57.370 --> 00:43:00.090 align:middle line:84% Here, they are not literally plotting the first two 00:43:00.090 --> 00:43:01.010 align:middle line:90% dimensions. 00:43:01.010 --> 00:43:03.470 align:middle line:84% They're using a particular technique called t-SNE, 00:43:03.470 --> 00:43:06.410 align:middle line:84% which is a way to take long vectors and project them to 2D 00:43:06.410 --> 00:43:09.170 align:middle line:84% space for visualization purposes. 00:43:09.170 --> 00:43:12.325 align:middle line:84% And you can see here, some very interesting things 00:43:12.325 --> 00:43:12.950 align:middle line:90% are showing up. 00:43:12.950 --> 00:43:15.530 align:middle line:84% So they're basically, they plotted the embedding 00:43:15.530 --> 00:43:17.090 align:middle line:90% for brother-- 00:43:17.090 --> 00:43:20.750 align:middle line:84% nephew, uncle, sister, niece, aunt and so on and so forth. 00:43:20.750 --> 00:43:22.210 align:middle line:90% It's all showing up here. 00:43:22.210 --> 00:43:24.290 align:middle line:84% This is the embedding for man, embedding 00:43:24.290 --> 00:43:31.290 align:middle line:84% for woman, sir madam, heiress, heir, duke, emperor, king. 00:43:31.290 --> 00:43:33.090 align:middle line:90% You get the idea. 00:43:33.090 --> 00:43:35.810 align:middle line:84% So clearly, there are patterns here where 00:43:35.810 --> 00:43:37.975 align:middle line:84% things which are similar in their nature 00:43:37.975 --> 00:43:39.850 align:middle line:84% are all hanging out together in the same part 00:43:39.850 --> 00:43:42.390 align:middle line:84% of the space, which is comforting, 00:43:42.390 --> 00:43:45.430 align:middle line:90% which is good to know. 00:43:45.430 --> 00:43:47.920 align:middle line:84% But as I mentioned earlier, it's not just about the fact 00:43:47.920 --> 00:43:50.840 align:middle line:84% that similar things happen to be near each other. 00:43:50.840 --> 00:43:53.240 align:middle line:84% The direction also actually matters. 00:43:53.240 --> 00:43:55.920 align:middle line:84% And beautiful things happen when you look at directions. 00:43:55.920 --> 00:44:00.240 align:middle line:84% So for instance, let's say that you have man, 00:44:00.240 --> 00:44:03.600 align:middle line:84% and you want to go from man to brother. 00:44:03.600 --> 00:44:06.760 align:middle line:84% So to go from man to brother, you have to start with man 00:44:06.760 --> 00:44:11.440 align:middle line:84% and then travel along this arrow to get to brother. 00:44:11.440 --> 00:44:16.770 align:middle line:84% So this arrow has some notion of a person becoming a sibling. 00:44:16.770 --> 00:44:19.400 align:middle line:90% 00:44:19.400 --> 00:44:22.360 align:middle line:84% So you would hope that if you take that same arrow 00:44:22.360 --> 00:44:26.080 align:middle line:84% and then start here with that arrow, hopefully 00:44:26.080 --> 00:44:29.480 align:middle line:90% the woman will become a sister. 00:44:29.480 --> 00:44:32.720 align:middle line:90% Sure enough, it is. 00:44:32.720 --> 00:44:35.680 align:middle line:84% So this is called word vector algebra. 00:44:35.680 --> 00:44:36.800 align:middle line:90% Embedding algebra. 00:44:36.800 --> 00:44:40.000 align:middle line:84% And these relationships are actually showing up in the data. 00:44:40.000 --> 00:44:42.000 align:middle line:84% We didn't tell it any of these things. 00:44:42.000 --> 00:44:44.920 align:middle line:84% We just literally gave it the co-occurrence matrix 00:44:44.920 --> 00:44:47.890 align:middle line:90% and asked her to reproduce it. 00:44:47.890 --> 00:44:50.510 align:middle line:84% So I find it pretty shocking that these things are actually 00:44:50.510 --> 00:44:52.510 align:middle line:90% true. 00:44:52.510 --> 00:44:55.150 align:middle line:84% And it gives us evidence and comfort 00:44:55.150 --> 00:44:57.110 align:middle line:84% that whatever has been learned does 00:44:57.110 --> 00:44:59.750 align:middle line:84% have some deep connection to describing the underlying 00:44:59.750 --> 00:45:02.030 align:middle line:90% nature of what's going on. 00:45:02.030 --> 00:45:05.830 align:middle line:84% It's not some statistically fluky artifact. 00:45:05.830 --> 00:45:07.070 align:middle line:90% Yeah? 00:45:07.070 --> 00:45:08.750 align:middle line:84% AUDIENCE: So we can find synonyms 00:45:08.750 --> 00:45:11.550 align:middle line:84% by context or by adjacency to other words, 00:45:11.550 --> 00:45:14.760 align:middle line:84% and not by replacing the same word, right? 00:45:14.760 --> 00:45:17.830 align:middle line:84% Because they won't appear in the same sentence 00:45:17.830 --> 00:45:19.488 align:middle line:90% if you have two words-- 00:45:19.488 --> 00:45:20.530 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:45:20.530 --> 00:45:22.155 align:middle line:84% They won't appear in the same sentence, 00:45:22.155 --> 00:45:25.270 align:middle line:84% but the pattern of co-occurrence will be the same for them, 00:45:25.270 --> 00:45:28.030 align:middle line:84% which is what we've been able to reproduce with these embeddings. 00:45:28.030 --> 00:45:28.988 align:middle line:90% So that's the key idea. 00:45:28.988 --> 00:45:34.830 align:middle line:90% 00:45:34.830 --> 00:45:37.990 align:middle line:84% AUDIENCE: So my question is around how are we able 00:45:37.990 --> 00:45:42.470 align:middle line:84% to capture all these directions and in 2D matrix versus 00:45:42.470 --> 00:45:44.340 align:middle line:90% a multi-dimensional matrix? 00:45:44.340 --> 00:45:46.780 align:middle line:84% Because I feel like, OK, so this relationship 00:45:46.780 --> 00:45:49.300 align:middle line:90% is kind of concerned. 00:45:49.300 --> 00:45:53.218 align:middle line:84% Yeah, I'm going to a family or blood relationship 00:45:53.218 --> 00:45:54.260 align:middle line:90% or something of the sort. 00:45:54.260 --> 00:45:57.743 align:middle line:84% But how does it not mess up the other side of that matrix? 00:45:57.743 --> 00:46:00.160 align:middle line:84% RAMA RAMAKRISHNAN: No, this is just a visualization thing. 00:46:00.160 --> 00:46:02.340 align:middle line:90% So we're basically taking this-- 00:46:02.340 --> 00:46:04.220 align:middle line:84% as you will see, GloVe embeddings 00:46:04.220 --> 00:46:06.020 align:middle line:90% come in lots of different sizes. 00:46:06.020 --> 00:46:09.180 align:middle line:84% And this, I think, uses the 100 dimension embedding and just 00:46:09.180 --> 00:46:12.180 align:middle line:84% projects it to 2D space using a particular technique, 00:46:12.180 --> 00:46:16.420 align:middle line:84% and then looks to see what's going on. 00:46:16.420 --> 00:46:18.123 align:middle line:90% Yeah? 00:46:18.123 --> 00:46:20.540 align:middle line:84% AUDIENCE: If the input data being the co-occurrence matrix 00:46:20.540 --> 00:46:22.608 align:middle line:84% is biased, are we amplifying that bias? 00:46:22.608 --> 00:46:23.900 align:middle line:90% RAMA RAMAKRISHNAN: Yes, we are. 00:46:23.900 --> 00:46:24.400 align:middle line:90% Yes. 00:46:24.400 --> 00:46:26.100 align:middle line:90% No, it's a great observation. 00:46:26.100 --> 00:46:28.660 align:middle line:84% Any data you scrape from the internet and use 00:46:28.660 --> 00:46:32.260 align:middle line:84% for this modeling exercise will be subject to all the biases 00:46:32.260 --> 00:46:34.540 align:middle line:84% that produce the data in the first place, 00:46:34.540 --> 00:46:37.540 align:middle line:84% and the model will faithfully learn those biases. 00:46:37.540 --> 00:46:40.380 align:middle line:84% And if you're not careful, it will perpetuate them. 00:46:40.380 --> 00:46:42.300 align:middle line:84% And that's a whole very important topic 00:46:42.300 --> 00:46:44.740 align:middle line:84% that unfortunately we won't cover in this course because 00:46:44.740 --> 00:46:45.720 align:middle line:90% of time constraints. 00:46:45.720 --> 00:46:47.137 align:middle line:84% But it's something you always have 00:46:47.137 --> 00:46:50.217 align:middle line:84% to worry about when you're building these models. 00:46:50.217 --> 00:46:52.300 align:middle line:84% AUDIENCE: How do we think about the dimensionality 00:46:52.300 --> 00:46:55.340 align:middle line:84% of the embeddings, not the 2D representation of the actual 00:46:55.340 --> 00:46:55.840 align:middle line:90% embedding? 00:46:55.840 --> 00:46:58.423 align:middle line:84% RAMA RAMAKRISHNAN: The one that we choose that's in our hands? 00:46:58.423 --> 00:47:00.580 align:middle line:84% So you should think of them as a hyperparameter. 00:47:00.580 --> 00:47:03.300 align:middle line:84% So much like the number of hidden units 00:47:03.300 --> 00:47:07.260 align:middle line:84% to use in a particular hidden layer, it's a hyperparameter. 00:47:07.260 --> 00:47:09.100 align:middle line:90% I would, again, start small. 00:47:09.100 --> 00:47:10.900 align:middle line:84% And if it solves the problem that you're 00:47:10.900 --> 00:47:13.280 align:middle line:84% trying to solve with these embeddings, great. 00:47:13.280 --> 00:47:14.860 align:middle line:90% If not, keep increasing them. 00:47:14.860 --> 00:47:16.660 align:middle line:84% And at some point, there might be 00:47:16.660 --> 00:47:19.530 align:middle line:84% a flattening out and an overfitting dynamic, 00:47:19.530 --> 00:47:20.280 align:middle line:90% and then you stop. 00:47:20.280 --> 00:47:22.740 align:middle line:84% So just think of them as a hyperparameter. 00:47:22.740 --> 00:47:24.340 align:middle line:90% Yeah? 00:47:24.340 --> 00:47:26.220 align:middle line:84% AUDIENCE: Do you see any benefit in practice 00:47:26.220 --> 00:47:30.420 align:middle line:84% of using penalized regression to do this instead of making 00:47:30.420 --> 00:47:33.500 align:middle line:84% the embeddings more sparse or just lowering 00:47:33.500 --> 00:47:34.700 align:middle line:90% the magnitude of them? 00:47:34.700 --> 00:47:35.700 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:47:35.700 --> 00:47:36.900 align:middle line:90% AUDIENCE: [INAUDIBLE]? 00:47:36.900 --> 00:47:37.858 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:47:37.858 --> 00:47:41.610 align:middle line:84% So there are lots of techniques to apply regularization 00:47:41.610 --> 00:47:44.930 align:middle line:84% in the estimation itself of all these numbers. 00:47:44.930 --> 00:47:46.570 align:middle line:90% I'm happy to give you pointers. 00:47:46.570 --> 00:47:49.610 align:middle line:84% I'm just going with the simplest version possible. 00:47:49.610 --> 00:47:50.850 align:middle line:90% Yeah? 00:47:50.850 --> 00:47:53.530 align:middle line:84% AUDIENCE: I'm not understanding why overfitting 00:47:53.530 --> 00:47:56.010 align:middle line:84% is a problem in this case because we're not doing 00:47:56.010 --> 00:47:59.050 align:middle line:90% any out-of-sample predictions. 00:47:59.050 --> 00:48:03.210 align:middle line:84% Wouldn't you want the embeddings to be high dimension so you 00:48:03.210 --> 00:48:05.277 align:middle line:90% capture unique relationships? 00:48:05.277 --> 00:48:07.110 align:middle line:84% RAMA RAMAKRISHNAN: Ah, interesting question. 00:48:07.110 --> 00:48:08.970 align:middle line:84% So the question is, given that there's 00:48:08.970 --> 00:48:12.650 align:middle line:84% no notion of a out-of-sample test set that we're 00:48:12.650 --> 00:48:14.270 align:middle line:84% going to evaluate these things on, 00:48:14.270 --> 00:48:16.130 align:middle line:84% why do we really care about overfitting? 00:48:16.130 --> 00:48:18.970 align:middle line:84% Shouldn't we do the best we can to capture everything 00:48:18.970 --> 00:48:20.602 align:middle line:90% in the data? 00:48:20.602 --> 00:48:23.690 align:middle line:84% Well, the thing is even when you're not 00:48:23.690 --> 00:48:25.970 align:middle line:84% trying to use it for out-of-sample prediction, 00:48:25.970 --> 00:48:29.570 align:middle line:84% you do want to make sure that your model only 00:48:29.570 --> 00:48:32.770 align:middle line:84% captures the true patterns and not the noise. 00:48:32.770 --> 00:48:35.530 align:middle line:84% In every data set, there is always noise, 00:48:35.530 --> 00:48:38.880 align:middle line:84% and you want it to capture the signal but not the noise 00:48:38.880 --> 00:48:40.640 align:middle line:84% regardless of what you use it for. 00:48:40.640 --> 00:48:43.400 align:middle line:84% Because if it captures the noise, then the insights 00:48:43.400 --> 00:48:45.960 align:middle line:84% you draw from the word embeddings may be flawed. 00:48:45.960 --> 00:48:48.880 align:middle line:90% That's the reason. 00:48:48.880 --> 00:48:50.040 align:middle line:90% OK. 00:48:50.040 --> 00:48:51.340 align:middle line:90% All right, so let's keep going. 00:48:51.340 --> 00:48:55.400 align:middle line:84% So here, the algebra is brother minus man plus woman is sister. 00:48:55.400 --> 00:48:55.940 align:middle line:90% That's it. 00:48:55.940 --> 00:48:59.160 align:middle line:84% Human biology reduced to a single sentence. 00:48:59.160 --> 00:48:59.700 align:middle line:90% All right. 00:48:59.700 --> 00:49:01.367 align:middle line:84% So now the pros and cons of these things 00:49:01.367 --> 00:49:04.280 align:middle line:84% are you should use something like a GloVe embedding 00:49:04.280 --> 00:49:09.960 align:middle line:84% if you don't have enough data to learn a task specific embedding 00:49:09.960 --> 00:49:11.160 align:middle line:90% for your own vocabulary. 00:49:11.160 --> 00:49:12.720 align:middle line:84% As I'll show you in the Colab, you 00:49:12.720 --> 00:49:14.920 align:middle line:84% can actually learn these things just for your own data set, 00:49:14.920 --> 00:49:15.460 align:middle line:90% if you want. 00:49:15.460 --> 00:49:17.840 align:middle line:84% You don't have to use these GloVe embeddings. 00:49:17.840 --> 00:49:20.000 align:middle line:84% But the reason to use these pre-trained embeddings 00:49:20.000 --> 00:49:22.360 align:middle line:84% is that if you're working with natural language, 00:49:22.360 --> 00:49:24.180 align:middle line:90% the word is the word. 00:49:24.180 --> 00:49:25.560 align:middle line:90% It means something. 00:49:25.560 --> 00:49:29.900 align:middle line:84% And so there's no reason for your model, 00:49:29.900 --> 00:49:33.200 align:middle line:84% for your little use case for you to actually somehow learn all 00:49:33.200 --> 00:49:35.030 align:middle line:90% the fundamentals of English. 00:49:35.030 --> 00:49:37.530 align:middle line:84% The fundamentals of English are the fundamentals of English. 00:49:37.530 --> 00:49:40.510 align:middle line:84% May as well learn it once and then piggyback on it. 00:49:40.510 --> 00:49:43.990 align:middle line:84% So that's the whole idea of using pre-trained embeddings 00:49:43.990 --> 00:49:46.530 align:middle line:84% because these things are all common aspects of language. 00:49:46.530 --> 00:49:49.250 align:middle line:84% May as well learn them using all the data you can throw at it, 00:49:49.250 --> 00:49:51.510 align:middle line:84% and then you can fine tune and tweak and adapt 00:49:51.510 --> 00:49:54.890 align:middle line:90% to your particular use case. 00:49:54.890 --> 00:49:56.390 align:middle line:84% And this is particularly useful when 00:49:56.390 --> 00:49:59.510 align:middle line:84% you don't have a lot of data in your particular use case. 00:49:59.510 --> 00:50:01.230 align:middle line:90% That's one big advantage. 00:50:01.230 --> 00:50:03.910 align:middle line:84% Now, it does have the drawback that this embedding will not 00:50:03.910 --> 00:50:05.527 align:middle line:90% be customized to your data. 00:50:05.527 --> 00:50:07.110 align:middle line:84% For example, if you're trying to build 00:50:07.110 --> 00:50:09.850 align:middle line:84% an application for a medical or legal use, 00:50:09.850 --> 00:50:12.313 align:middle line:84% it's going to have a lot of jargon. 00:50:12.313 --> 00:50:14.730 align:middle line:84% And this pre-trained embedding trained on all of Wikipedia 00:50:14.730 --> 00:50:16.630 align:middle line:84% may not capture enough of the jargon 00:50:16.630 --> 00:50:18.530 align:middle line:84% and know its meaning really accurately. 00:50:18.530 --> 00:50:19.590 align:middle line:84% So what you want to do is you want 00:50:19.590 --> 00:50:21.490 align:middle line:84% to take this thing-- you may still want to take this thing, 00:50:21.490 --> 00:50:22.990 align:middle line:84% and then you can adapt and fine tune 00:50:22.990 --> 00:50:29.050 align:middle line:84% it using your jargon packed, heavy, domain-specific data set. 00:50:29.050 --> 00:50:32.148 align:middle line:84% Those are some of the things to keep in mind. 00:50:32.148 --> 00:50:34.690 align:middle line:84% And of course, we can also learn it from scratch if you want. 00:50:34.690 --> 00:50:38.100 align:middle line:84% And the Colab, I demonstrate all these options. 00:50:38.100 --> 00:50:40.820 align:middle line:84% So when you're working with embeddings in Keras, 00:50:40.820 --> 00:50:44.700 align:middle line:84% so what we do is remember STIE, where 00:50:44.700 --> 00:50:46.260 align:middle line:84% after we standardize and tokenize 00:50:46.260 --> 00:50:50.480 align:middle line:84% and index, at this point, we go from integers to vectors. 00:50:50.480 --> 00:50:53.420 align:middle line:84% And so far, we have been using integers to one-hot vectors. 00:50:53.420 --> 00:50:55.180 align:middle line:84% Here, we're going to use embedding vectors 00:50:55.180 --> 00:50:56.805 align:middle line:84% that we're going to learn or that we're 00:50:56.805 --> 00:50:58.860 align:middle line:90% going to use from GloVe. 00:50:58.860 --> 00:51:02.860 align:middle line:84% And so what we do is we tell Keras's text vectorization 00:51:02.860 --> 00:51:06.500 align:middle line:84% layer to do only S, T, and I. And then 00:51:06.500 --> 00:51:08.820 align:middle line:84% we will use a new layer called the embedding layer 00:51:08.820 --> 00:51:10.660 align:middle line:90% to do the encoding. 00:51:10.660 --> 00:51:14.980 align:middle line:84% That's how we're going to divide it up. 00:51:14.980 --> 00:51:17.060 align:middle line:84% So we'll take a look at this first 00:51:17.060 --> 00:51:18.620 align:middle line:90% before we switch to the Colab. 00:51:18.620 --> 00:51:22.820 align:middle line:84% So before, we told Keras in this layer, 00:51:22.820 --> 00:51:26.300 align:middle line:84% output mode should be multi-hot or whatever. 00:51:26.300 --> 00:51:28.020 align:middle line:84% Here, we don't want it to actually encode 00:51:28.020 --> 00:51:28.937 align:middle line:90% anything in multi-hot. 00:51:28.937 --> 00:51:30.600 align:middle line:84% We just want to give it integers back. 00:51:30.600 --> 00:51:33.800 align:middle line:90% So we tell it, give me int. 00:51:33.800 --> 00:51:35.180 align:middle line:90% That's the first change. 00:51:35.180 --> 00:51:36.600 align:middle line:90% We tell it, give us int. 00:51:36.600 --> 00:51:39.580 align:middle line:84% If you say give us int, it'll stop with STI 00:51:39.580 --> 00:51:41.580 align:middle line:90% and just give you the integers. 00:51:41.580 --> 00:51:45.060 align:middle line:84% And then what you do is that all the incoming centers are going 00:51:45.060 --> 00:51:46.418 align:middle line:90% to have different lengths. 00:51:46.418 --> 00:51:48.460 align:middle line:84% So what we want to do is we want to actually take 00:51:48.460 --> 00:51:50.440 align:middle line:84% all these sentences and normalize them 00:51:50.440 --> 00:51:55.780 align:middle line:84% so they are of the same length And the way 00:51:55.780 --> 00:51:59.220 align:middle line:84% we do that very quickly is that we choose 00:51:59.220 --> 00:52:02.480 align:middle line:84% a maximum length for the sentences, 00:52:02.480 --> 00:52:06.040 align:middle line:84% and then if something is exactly fits that length-- perfect. 00:52:06.040 --> 00:52:08.220 align:middle line:84% Let's say in this case, we want a max length of 5. 00:52:08.220 --> 00:52:09.760 align:middle line:84% Cat sat on the mat is exactly five. 00:52:09.760 --> 00:52:10.260 align:middle line:90% Boom. 00:52:10.260 --> 00:52:11.072 align:middle line:90% Fits perfectly. 00:52:11.072 --> 00:52:12.280 align:middle line:90% But if something is smaller-- 00:52:12.280 --> 00:52:14.030 align:middle line:84% I love you is only three of these things-- 00:52:14.030 --> 00:52:17.866 align:middle line:84% we actually pad it with something called the pad token. 00:52:17.866 --> 00:52:20.620 align:middle line:84% Also the unk token-- pad token is a special token 00:52:20.620 --> 00:52:23.380 align:middle line:90% which we use for padding. 00:52:23.380 --> 00:52:25.700 align:middle line:84% And so in Keras, you will see, will use zeros 00:52:25.700 --> 00:52:28.820 align:middle line:84% for these paddings so that it fills it up and gets all the way 00:52:28.820 --> 00:52:29.640 align:middle line:90% to the end. 00:52:29.640 --> 00:52:32.395 align:middle line:84% And if you have something which is much longer than five, 00:52:32.395 --> 00:52:33.770 align:middle line:84% you just truncate everything else 00:52:33.770 --> 00:52:36.250 align:middle line:90% and just use the first five. 00:52:36.250 --> 00:52:38.928 align:middle line:84% So this is what we do to get all the sentences to be 00:52:38.928 --> 00:52:39.720 align:middle line:90% of the same length. 00:52:39.720 --> 00:52:43.250 align:middle line:90% 00:52:43.250 --> 00:52:47.250 align:middle line:84% And once we do that, we then go to the embedding layer. 00:52:47.250 --> 00:52:50.370 align:middle line:84% And the embedding layer is actually very simple. 00:52:50.370 --> 00:52:51.590 align:middle line:90% What is an embedding? 00:52:51.590 --> 00:52:52.430 align:middle line:90% It's just a vector. 00:52:52.430 --> 00:52:54.433 align:middle line:84% And we need a vector for every token. 00:52:54.433 --> 00:52:56.350 align:middle line:84% Of course, we're going to learn these vectors. 00:52:56.350 --> 00:52:57.750 align:middle line:90% We need one for every token. 00:52:57.750 --> 00:52:59.730 align:middle line:84% So in this case, for example, let's say 00:52:59.730 --> 00:53:03.210 align:middle line:84% that these are all the tokens we have in our vocabulary 00:53:03.210 --> 00:53:05.030 align:middle line:90% after the STI process. 00:53:05.030 --> 00:53:07.830 align:middle line:84% Maybe in this case, we have 5,000 tokens. 00:53:07.830 --> 00:53:10.077 align:middle line:84% Each token, we have this embedding vector, 00:53:10.077 --> 00:53:12.410 align:middle line:84% and we choose what the dimension of the embedding vector 00:53:12.410 --> 00:53:13.390 align:middle line:90% is right. 00:53:13.390 --> 00:53:17.170 align:middle line:84% And so we can set it up by saying keras.layers.Embedding. 00:53:17.170 --> 00:53:19.270 align:middle line:84% And we tell it max tokens which means 00:53:19.270 --> 00:53:21.542 align:middle line:90% how many rows do we have here? 00:53:21.542 --> 00:53:23.750 align:middle line:84% What is the vocabulary size that we are working with? 00:53:23.750 --> 00:53:26.490 align:middle line:84% And then we tell it, OK, this is how long I want 00:53:26.490 --> 00:53:28.550 align:middle line:90% each embedding vector to be. 00:53:28.550 --> 00:53:31.740 align:middle line:84% So rows, the size of the columns, 00:53:31.740 --> 00:53:33.060 align:middle line:90% and that's the embedding layer. 00:53:33.060 --> 00:53:34.100 align:middle line:90% And we will use it in a second. 00:53:34.100 --> 00:53:35.517 align:middle line:84% I just want to show it to you here 00:53:35.517 --> 00:53:37.080 align:middle line:90% because it's slightly clearer. 00:53:37.080 --> 00:53:38.640 align:middle line:84% So when an input sentence arrives, 00:53:38.640 --> 00:53:40.893 align:middle line:84% the text vectorization layer will run STI on it. 00:53:40.893 --> 00:53:43.060 align:middle line:84% It will truncate and pad it to max length as needed. 00:53:43.060 --> 00:53:45.120 align:middle line:84% So let's say this phrase comes in. 00:53:45.120 --> 00:53:48.240 align:middle line:84% STI will give you the same tokens plus pad pad 00:53:48.240 --> 00:53:50.100 align:middle line:84% because let's say the max length is 5, 00:53:50.100 --> 00:53:52.640 align:middle line:84% and then these are the corresponding integers. 00:53:52.640 --> 00:53:55.257 align:middle line:84% And then the embedding layer will just look up 00:53:55.257 --> 00:53:56.340 align:middle line:90% the corresponding vectors. 00:53:56.340 --> 00:53:59.120 align:middle line:84% So for example here, the vectors are-- 00:53:59.120 --> 00:54:02.360 align:middle line:84% we need to look up the vectors for 23, 9, 5, 0, and 0. 00:54:02.360 --> 00:54:06.640 align:middle line:84% So we just go here and look up 23, 5, 9, and 0. 00:54:06.640 --> 00:54:08.840 align:middle line:84% And then once we have that, boom, 00:54:08.840 --> 00:54:10.480 align:middle line:90% this is the resulting output. 00:54:10.480 --> 00:54:12.240 align:middle line:84% So whatever input sentence comes in, 00:54:12.240 --> 00:54:15.120 align:middle line:84% we have now five embedding vectors that have been looked up 00:54:15.120 --> 00:54:17.520 align:middle line:90% from the embedding layer. 00:54:17.520 --> 00:54:20.120 align:middle line:90% And once we do that-- 00:54:20.120 --> 00:54:21.300 align:middle line:90% this is a table. 00:54:21.300 --> 00:54:22.420 align:middle line:90% So I love you comes in. 00:54:22.420 --> 00:54:23.760 align:middle line:90% It becomes this table. 00:54:23.760 --> 00:54:26.480 align:middle line:84% As we have seen before, neural networks 00:54:26.480 --> 00:54:29.410 align:middle line:84% can only accommodate vectors as inputs. 00:54:29.410 --> 00:54:31.670 align:middle line:84% We need to make this into a vector. 00:54:31.670 --> 00:54:34.630 align:middle line:84% And as we have done before, we can either take all these things 00:54:34.630 --> 00:54:37.010 align:middle line:84% and concatenate them, make a one long vector, 00:54:37.010 --> 00:54:39.350 align:middle line:84% or we can find a way to average them or sum them 00:54:39.350 --> 00:54:41.610 align:middle line:84% and things like that, as we have seen before. 00:54:41.610 --> 00:54:43.190 align:middle line:90% And we will use the same-- 00:54:43.190 --> 00:54:45.350 align:middle line:84% the simplest thing is probably just average them. 00:54:45.350 --> 00:54:49.830 align:middle line:84% So these are some options, but we will average them here. 00:54:49.830 --> 00:54:54.350 align:middle line:84% And this is called the GlobalAveragePooling layer, 1D. 00:54:54.350 --> 00:54:56.110 align:middle line:84% And all it does is whatever you give it-- 00:54:56.110 --> 00:54:57.670 align:middle line:84% a table you give it, it just takes 00:54:57.670 --> 00:54:59.600 align:middle line:90% each dimension and averages it. 00:54:59.600 --> 00:55:01.850 align:middle line:84% The first dimension average, second dimension average, 00:55:01.850 --> 00:55:02.930 align:middle line:90% and so on and so forth. 00:55:02.930 --> 00:55:05.510 align:middle line:84% And once that's done, that's the whole flow. 00:55:05.510 --> 00:55:09.510 align:middle line:84% So the phrase comes in, STI gives you 00:55:09.510 --> 00:55:13.065 align:middle line:84% these things, padding as needed or truncating as needed. 00:55:13.065 --> 00:55:15.190 align:middle line:84% We look up the embeddings from the embedding layer, 00:55:15.190 --> 00:55:17.270 align:middle line:90% and then we get all this thing. 00:55:17.270 --> 00:55:20.510 align:middle line:84% We do global pooling on it, and it's done. 00:55:20.510 --> 00:55:22.230 align:middle line:84% The resulting thing is a vector that 00:55:22.230 --> 00:55:24.970 align:middle line:84% can then be passed into hidden layers just like we normally do. 00:55:24.970 --> 00:55:27.580 align:middle line:90% 00:55:27.580 --> 00:55:29.240 align:middle line:84% I'm going go with this a little fast, 00:55:29.240 --> 00:55:31.220 align:middle line:84% but make sure you look at it afterwards 00:55:31.220 --> 00:55:33.140 align:middle line:84% and understand every step, and the Colab 00:55:33.140 --> 00:55:36.220 align:middle line:90% will mirror this perfectly. 00:55:36.220 --> 00:55:39.620 align:middle line:84% All right, so let's switch to the Colab. 00:55:39.620 --> 00:55:40.340 align:middle line:90% OK. 00:55:40.340 --> 00:55:41.500 align:middle line:90% All right. 00:55:41.500 --> 00:55:43.780 align:middle line:90% Can folks see this OK? 00:55:43.780 --> 00:55:44.280 align:middle line:90% All right. 00:55:44.280 --> 00:55:45.197 align:middle line:90% So we'll do the usual. 00:55:45.197 --> 00:55:47.740 align:middle line:90% 00:55:47.740 --> 00:55:49.300 align:middle line:90% Import all the stuff you need. 00:55:49.300 --> 00:55:52.420 align:middle line:84% And then because I want to plot some of these loss and accuracy 00:55:52.420 --> 00:55:55.180 align:middle line:84% curves just to see what's going on, 00:55:55.180 --> 00:56:00.620 align:middle line:84% I'll just bring in the functions from the previous Colabs here. 00:56:00.620 --> 00:56:02.680 align:middle line:84% And I think I already have downloaded this. 00:56:02.680 --> 00:56:04.280 align:middle line:90% Let me just make sure I have it. 00:56:04.280 --> 00:56:08.460 align:middle line:90% 00:56:08.460 --> 00:56:10.012 align:middle line:90% It's not there. 00:56:10.012 --> 00:56:13.500 align:middle line:90% So let's do it again. 00:56:13.500 --> 00:56:16.000 align:middle line:84% This is the same songs data set that we looked at on Monday. 00:56:16.000 --> 00:56:18.806 align:middle line:90% 00:56:18.806 --> 00:56:22.140 align:middle line:84% So roughly 49,000 examples, as we saw before. 00:56:22.140 --> 00:56:25.580 align:middle line:90% We'll one-hot encode them. 00:56:25.580 --> 00:56:26.080 align:middle line:90% All right. 00:56:26.080 --> 00:56:28.747 align:middle line:84% So there's a bunch of stuff that we've already covered in class. 00:56:28.747 --> 00:56:30.620 align:middle line:90% So this is the thing. 00:56:30.620 --> 00:56:34.880 align:middle line:84% This URL has all the GloVe vectors available for download. 00:56:34.880 --> 00:56:38.340 align:middle line:84% I downloaded it before class because it takes a few minutes. 00:56:38.340 --> 00:56:41.780 align:middle line:90% And I've also-- did I unzip it? 00:56:41.780 --> 00:56:42.680 align:middle line:90% Yes, I did. 00:56:42.680 --> 00:56:45.940 align:middle line:84% So let's just look at the first few. 00:56:45.940 --> 00:56:46.440 align:middle line:90% All right. 00:56:46.440 --> 00:56:47.732 align:middle line:90% So these are all the first few. 00:56:47.732 --> 00:56:50.770 align:middle line:84% We'll create an easier to view version of these GloVe vectors. 00:56:50.770 --> 00:56:54.860 align:middle line:90% 00:56:54.860 --> 00:56:57.880 align:middle line:84% So I'm going to use the vectors which are hundred long, 00:56:57.880 --> 00:56:59.940 align:middle line:84% but it comes in many different shapes. 00:56:59.940 --> 00:57:02.300 align:middle line:90% So we have 400,000 vectors. 00:57:02.300 --> 00:57:06.020 align:middle line:84% 400,000 word vectors each as 100 dimension. 00:57:06.020 --> 00:57:08.420 align:middle line:84% And these all have been calculated from Wikipedia 00:57:08.420 --> 00:57:11.820 align:middle line:84% using the model we described using gradient descent. 00:57:11.820 --> 00:57:13.140 align:middle line:90% OK? 00:57:13.140 --> 00:57:13.640 align:middle line:90% All right. 00:57:13.640 --> 00:57:18.153 align:middle line:84% So this is the vector for the word for movie. 00:57:18.153 --> 00:57:19.820 align:middle line:84% I don't know what these dimensions mean, 00:57:19.820 --> 00:57:21.840 align:middle line:90% but there's something going on. 00:57:21.840 --> 00:57:23.970 align:middle line:90% It has figured stuff out. 00:57:23.970 --> 00:57:25.550 align:middle line:90% But the proof is in the pudding. 00:57:25.550 --> 00:57:28.170 align:middle line:84% So all right, now we will first set up the text vectorization 00:57:28.170 --> 00:57:30.930 align:middle line:84% and embedding layers like we saw before. 00:57:30.930 --> 00:57:36.950 align:middle line:84% And so I'm going to use a max length of 300 for the songs 00:57:36.950 --> 00:57:40.330 align:middle line:84% because all the sentences have to be the same length. 00:57:40.330 --> 00:57:42.890 align:middle line:84% And you might be wondering, OK, why did you pick 300 or not, 00:57:42.890 --> 00:57:44.210 align:middle line:90% say, 400 or 200? 00:57:44.210 --> 00:57:47.370 align:middle line:84% So typically, what you do is you actually look at the length 00:57:47.370 --> 00:57:50.650 align:middle line:84% distribution of the songs you have, and you will find-- 00:57:50.650 --> 00:57:53.550 align:middle line:84% you're looking for an 80-20 or one of those things. 00:57:53.550 --> 00:57:56.330 align:middle line:84% And in this case, it turns out 90% of the songs have less than 00:57:56.330 --> 00:57:58.910 align:middle line:84% or equal to 300 words in our data set. 00:57:58.910 --> 00:58:00.670 align:middle line:84% So I'm just going to go with 300. 00:58:00.670 --> 00:58:02.370 align:middle line:90% It's pretty good. 00:58:02.370 --> 00:58:03.910 align:middle line:84% The problem is if you actually say-- 00:58:03.910 --> 00:58:06.850 align:middle line:84% if you look at the song which has the maximum length, 00:58:06.850 --> 00:58:10.370 align:middle line:84% that might be like 3,000 words, and there might be hardly any 00:58:10.370 --> 00:58:11.350 align:middle line:90% songs of 3,000 long. 00:58:11.350 --> 00:58:13.630 align:middle line:84% You're just wasting a lot of capacity by doing that. 00:58:13.630 --> 00:58:16.610 align:middle line:84% So you're just being a little pragmatic here. 00:58:16.610 --> 00:58:19.870 align:middle line:84% OK, and then as before, for the vocabulary itself, 00:58:19.870 --> 00:58:23.120 align:middle line:84% we tell Keras use the most frequent 5,000 words when 00:58:23.120 --> 00:58:25.840 align:middle line:90% you're doing the STI. 00:58:25.840 --> 00:58:29.000 align:middle line:84% So we do that, and we tell it the output mode is int, 00:58:29.000 --> 00:58:32.520 align:middle line:90% like we saw before. 00:58:32.520 --> 00:58:35.280 align:middle line:90% We have there. 00:58:35.280 --> 00:58:36.880 align:middle line:90% OK, perfect. 00:58:36.880 --> 00:58:39.560 align:middle line:84% OK, this is a very dangerous thing, 00:58:39.560 --> 00:58:41.960 align:middle line:84% where somebody is remotely changing it 00:58:41.960 --> 00:58:44.640 align:middle line:90% in another tab somewhere. 00:58:44.640 --> 00:58:48.400 align:middle line:90% Fingers crossed, OK? 00:58:48.400 --> 00:58:50.240 align:middle line:90% Take this one. 00:58:50.240 --> 00:58:50.760 align:middle line:90% OK. 00:58:50.760 --> 00:58:54.200 align:middle line:84% So we have this, and this is what we did with all this stuff, 00:58:54.200 --> 00:58:55.300 align:middle line:90% as I've covered. 00:58:55.300 --> 00:58:59.560 align:middle line:84% So now we will adapt this layer, as we have seen before, 00:58:59.560 --> 00:59:00.890 align:middle line:90% using all the lyrics we have. 00:59:00.890 --> 00:59:04.440 align:middle line:90% 00:59:04.440 --> 00:59:08.360 align:middle line:84% Then once we do that, we'll take a look at the first few. 00:59:08.360 --> 00:59:10.180 align:middle line:84% And so here's a very important thing. 00:59:10.180 --> 00:59:12.760 align:middle line:84% Before-- when we asked it to do multi-hot encoding 00:59:12.760 --> 00:59:14.840 align:middle line:90% and so on, on Monday-- 00:59:14.840 --> 00:59:18.020 align:middle line:84% the zero, the first position was unk. 00:59:18.020 --> 00:59:19.000 align:middle line:90% Unk had zero. 00:59:19.000 --> 00:59:21.910 align:middle line:90% But here, unk actually has one. 00:59:21.910 --> 00:59:24.590 align:middle line:84% And the reason is that the zeroth position 00:59:24.590 --> 00:59:28.230 align:middle line:84% is going to be used for essentially-- you can think 00:59:28.230 --> 00:59:29.690 align:middle line:90% of this as the empty string. 00:59:29.690 --> 00:59:32.870 align:middle line:84% That's how Keras will print out pad. 00:59:32.870 --> 00:59:36.170 align:middle line:84% So the zeroth position is the padding, the pad token. 00:59:36.170 --> 00:59:39.030 align:middle line:84% The first position is the unk token. 00:59:39.030 --> 00:59:41.470 align:middle line:90% So it's an important thing here. 00:59:41.470 --> 00:59:46.910 align:middle line:84% So let's say that we do HODL, you're the best. 00:59:46.910 --> 00:59:49.590 align:middle line:90% We vectorize it. 00:59:49.590 --> 00:59:53.030 align:middle line:84% Do you think HODL is going to be part of those 400,000 word 00:59:53.030 --> 00:59:55.070 align:middle line:90% vectors [INAUDIBLE]? 00:59:55.070 --> 00:59:57.790 align:middle line:90% Not yet. 00:59:57.790 --> 00:59:59.650 align:middle line:90% All right, so let's try that. 00:59:59.650 --> 01:00:03.510 align:middle line:90% 01:00:03.510 --> 01:00:07.990 align:middle line:84% And as you can tell, HODL is an unknown word. 01:00:07.990 --> 01:00:12.790 align:middle line:90% That's why it's showing up here. 01:00:12.790 --> 01:00:14.690 align:middle line:90% So 1 is unknown. 01:00:14.690 --> 01:00:16.430 align:middle line:90% The index value 1 is unknown. 01:00:16.430 --> 01:00:18.110 align:middle line:90% 0 is pad. 01:00:18.110 --> 01:00:21.980 align:middle line:84% But then this is unknown HODL, I-- 01:00:21.980 --> 01:00:24.713 align:middle line:90% sorry, you are the best. 01:00:24.713 --> 01:00:26.380 align:middle line:84% And then everything else from that point 01:00:26.380 --> 01:00:30.600 align:middle line:84% on is a 0 because we are padding all the way to 300. 01:00:30.600 --> 01:00:32.580 align:middle line:84% So that's why you see all these zeros here. 01:00:32.580 --> 01:00:35.180 align:middle line:84% All right, now let's just run everything 01:00:35.180 --> 01:00:38.860 align:middle line:84% through the vectorization layer, and then we'll 01:00:38.860 --> 01:00:40.160 align:middle line:90% get to the embedding layer. 01:00:40.160 --> 01:00:44.340 align:middle line:90% 01:00:44.340 --> 01:00:44.840 align:middle line:90% OK. 01:00:44.840 --> 01:00:48.580 align:middle line:90% Now we will first-- 01:00:48.580 --> 01:00:52.300 align:middle line:84% just a bit of Python housekeeping 01:00:52.300 --> 01:00:55.308 align:middle line:84% to create a nice, easy to look at matrix. 01:00:55.308 --> 01:00:57.100 align:middle line:84% So what we're going to do is we're actually 01:00:57.100 --> 01:01:01.420 align:middle line:84% going to create a nice matrix that shows us all the GloVe 01:01:01.420 --> 01:01:04.100 align:middle line:90% embeddings. 01:01:04.100 --> 01:01:07.140 align:middle line:84% And so this is the embedding matrix, 01:01:07.140 --> 01:01:11.700 align:middle line:84% and this matrix has only 5,000 words, and each is 100 long. 01:01:11.700 --> 01:01:13.700 align:middle line:84% Why is this embedding matrix only 5,000, 01:01:13.700 --> 01:01:21.940 align:middle line:84% even though we downloaded 400,000 vectors? 01:01:21.940 --> 01:01:24.640 align:middle line:84% So clearly, the 5,000 we used, there has some bearing to this. 01:01:24.640 --> 01:01:30.660 align:middle line:90% But what is the 5,000? 01:01:30.660 --> 01:01:33.660 align:middle line:84% We told Keras to take the most frequent 5,000 words 01:01:33.660 --> 01:01:37.040 align:middle line:84% in our corpus so we'll only have 5,000 in vocabulary. 01:01:37.040 --> 01:01:38.360 align:middle line:90% That's why there's 5,000. 01:01:38.360 --> 01:01:41.820 align:middle line:84% So we grab just the GloVe vectors for those 5,000 that 01:01:41.820 --> 01:01:43.660 align:middle line:84% Keras has chosen to be in the vocabulary, 01:01:43.660 --> 01:01:45.580 align:middle line:90% and that's our embedding matrix. 01:01:45.580 --> 01:01:48.580 align:middle line:84% And then if you look at the first few rows, 01:01:48.580 --> 01:01:50.780 align:middle line:84% the first two rows should be all zeros 01:01:50.780 --> 01:01:53.968 align:middle line:84% because it's pad and unk, which clearly, GloVe 01:01:53.968 --> 01:01:54.760 align:middle line:90% doesn't know about. 01:01:54.760 --> 01:01:56.180 align:middle line:84% So it's all going to be all zeros. 01:01:56.180 --> 01:01:59.000 align:middle line:84% And so you can see all these zeros here. 01:01:59.000 --> 01:02:01.700 align:middle line:84% And then from the third onwards, you start getting some numbers. 01:02:01.700 --> 01:02:02.620 align:middle line:90% OK? 01:02:02.620 --> 01:02:03.460 align:middle line:90% All right. 01:02:03.460 --> 01:02:06.203 align:middle line:84% Next, we'll set up the embedding layer. 01:02:06.203 --> 01:02:07.620 align:middle line:84% So basically, what's going on here 01:02:07.620 --> 01:02:10.600 align:middle line:84% is we tell the embedding layer how many rows, 01:02:10.600 --> 01:02:12.900 align:middle line:84% which is just the vocab size max tokens. 01:02:12.900 --> 01:02:14.473 align:middle line:90% What is the embedding dimension? 01:02:14.473 --> 01:02:16.890 align:middle line:84% Well, that's going to be 100 because the GloVe vectors are 01:02:16.890 --> 01:02:17.590 align:middle line:90% hundred. 01:02:17.590 --> 01:02:18.770 align:middle line:90% And then here's the thing. 01:02:18.770 --> 01:02:21.690 align:middle line:84% You can tell it in this embedding layer, 01:02:21.690 --> 01:02:24.597 align:middle line:84% just use this matrix I'm giving you as the embedding layer 01:02:24.597 --> 01:02:26.430 align:middle line:84% because we already know what embeddings are. 01:02:26.430 --> 01:02:28.260 align:middle line:84% We downloaded it from whatever GloVe. 01:02:28.260 --> 01:02:31.030 align:middle line:84% So we will tell it to use GloVe as the weights 01:02:31.030 --> 01:02:32.670 align:middle line:84% for here, as the embeddings here. 01:02:32.670 --> 01:02:36.120 align:middle line:84% So we initialize it using that embedding matrix. 01:02:36.120 --> 01:02:40.675 align:middle line:84% And then we tell it, when we do back propagation later on, 01:02:40.675 --> 01:02:42.050 align:middle line:84% don't change any of these weights 01:02:42.050 --> 01:02:44.170 align:middle line:84% because somebody spent a lot of money 01:02:44.170 --> 01:02:45.850 align:middle line:90% to create this weights for us-- 01:02:45.850 --> 01:02:46.730 align:middle line:90% Stanford. 01:02:46.730 --> 01:02:49.050 align:middle line:84% So we don't want to further change them, just freeze 01:02:49.050 --> 01:02:51.042 align:middle line:90% them and use them as they are. 01:02:51.042 --> 01:02:53.230 align:middle line:84% And this mask zero business, I'll come back later. 01:02:53.230 --> 01:02:55.190 align:middle line:84% Don't worry about it for the moment. 01:02:55.190 --> 01:02:55.950 align:middle line:90% All right. 01:02:55.950 --> 01:02:59.910 align:middle line:84% So once we do that, we are ready to set up our model. 01:02:59.910 --> 01:03:01.850 align:middle line:90% So this model is pretty simple. 01:03:01.850 --> 01:03:02.870 align:middle line:90% Keras input. 01:03:02.870 --> 01:03:04.370 align:middle line:84% The length, of course, is the length 01:03:04.370 --> 01:03:07.090 align:middle line:84% of the sentence, which is 300 long. 01:03:07.090 --> 01:03:10.830 align:middle line:84% And then the input runs through an embedding layer right there, 01:03:10.830 --> 01:03:13.170 align:middle line:84% and out comes a 300 by 100 table. 01:03:13.170 --> 01:03:17.860 align:middle line:84% And then we average pool it, and that becomes 100 element vector. 01:03:17.860 --> 01:03:19.940 align:middle line:84% And then we are back in familiar ground. 01:03:19.940 --> 01:03:23.240 align:middle line:84% And we run it through a dense layer with eight ReLU neurons. 01:03:23.240 --> 01:03:24.888 align:middle line:90% Eight ReLU neurons. 01:03:24.888 --> 01:03:26.680 align:middle line:84% And then we run it through the final output 01:03:26.680 --> 01:03:28.240 align:middle line:84% layer, which is a three way softmax 01:03:28.240 --> 01:03:29.960 align:middle line:90% as before-- hip hop, rock, pop. 01:03:29.960 --> 01:03:32.040 align:middle line:84% And then we tell Keras that's the model, 01:03:32.040 --> 01:03:34.760 align:middle line:90% and then we summarize it. 01:03:34.760 --> 01:03:36.520 align:middle line:90% OK, so that's what we have. 01:03:36.520 --> 01:03:41.000 align:middle line:84% And you can see here, the total parameters are 500,835, 01:03:41.000 --> 01:03:44.560 align:middle line:84% but the trainable parameters are only 835. 01:03:44.560 --> 01:03:46.800 align:middle line:84% It's because the total parameters are all the GloVe 01:03:46.800 --> 01:03:50.080 align:middle line:84% embeddings plus the things we added 01:03:50.080 --> 01:03:52.840 align:middle line:84% to the GloVe embeddings like the hidden layer and so on. 01:03:52.840 --> 01:03:54.560 align:middle line:90% But the GloVe embeddings are-- 01:03:54.560 --> 01:03:56.060 align:middle line:90% we have told Keras, freeze it. 01:03:56.060 --> 01:03:57.565 align:middle line:90% Do not train it. 01:03:57.565 --> 01:03:59.940 align:middle line:84% Which means only the rest of it is going to be trainable. 01:03:59.940 --> 01:04:01.140 align:middle line:90% That's the 835. 01:04:01.140 --> 01:04:03.495 align:middle line:90% Yeah? 01:04:03.495 --> 01:04:05.620 align:middle line:84% AUDIENCE: So when we do the global average pooling, 01:04:05.620 --> 01:04:08.200 align:middle line:84% don't we lose any sense of meaning 01:04:08.200 --> 01:04:12.430 align:middle line:84% that we gain from the embedding, as we average very 01:04:12.430 --> 01:04:13.742 align:middle line:90% different embeddings together? 01:04:13.742 --> 01:04:15.450 align:middle line:84% RAMA RAMAKRISHNAN: Sorry, say that again. 01:04:15.450 --> 01:04:16.450 align:middle line:90% I missed the first part. 01:04:16.450 --> 01:04:19.330 align:middle line:84% AUDIENCE: If we average the embedding of apple and learning, 01:04:19.330 --> 01:04:21.550 align:middle line:84% for instance, you have very different words that 01:04:21.550 --> 01:04:23.130 align:middle line:90% are used in different meanings. 01:04:23.130 --> 01:04:25.650 align:middle line:84% So we have different embeddings, but we average it 01:04:25.650 --> 01:04:27.410 align:middle line:90% so we can't lose this-- 01:04:27.410 --> 01:04:28.570 align:middle line:84% RAMA RAMAKRISHNAN: We will lose a bunch of stuff. 01:04:28.570 --> 01:04:29.278 align:middle line:90% Yeah, yeah, yeah. 01:04:29.278 --> 01:04:30.950 align:middle line:84% So yeah, anytime you average anything, 01:04:30.950 --> 01:04:33.410 align:middle line:84% you're going to lose some nuance and so on. 01:04:33.410 --> 01:04:36.610 align:middle line:84% So the real question is despite that averaging, 01:04:36.610 --> 01:04:37.790 align:middle line:90% is it good enough for you? 01:04:37.790 --> 01:04:39.230 align:middle line:90% And sometimes it's good enough. 01:04:39.230 --> 01:04:41.357 align:middle line:84% Very often, it's good enough, as it turns out. 01:04:41.357 --> 01:04:43.690 align:middle line:84% But as you will see when we go to contextual embeddings, 01:04:43.690 --> 01:04:45.610 align:middle line:84% there's just a better way to do it, 01:04:45.610 --> 01:04:47.230 align:middle line:84% when you have contextual embeddings. 01:04:47.230 --> 01:04:49.610 align:middle line:84% But it requires bigger models, more powerful stuff, 01:04:49.610 --> 01:04:50.610 align:middle line:90% and so on and so forth. 01:04:50.610 --> 01:04:52.693 align:middle line:84% And that's where you're going from the foundations 01:04:52.693 --> 01:04:53.950 align:middle line:90% to the advanced stuff. 01:04:53.950 --> 01:04:56.230 align:middle line:90% Yeah? 01:04:56.230 --> 01:04:58.270 align:middle line:84% AUDIENCE: When we're doing optimization-- like, 01:04:58.270 --> 01:05:00.310 align:middle line:84% let's say we are having a word problem-- 01:05:00.310 --> 01:05:02.990 align:middle line:84% it's often best to optimize everything 01:05:02.990 --> 01:05:05.490 align:middle line:84% together than to optimize one part of the system 01:05:05.490 --> 01:05:07.190 align:middle line:84% and then optimize the response system. 01:05:07.190 --> 01:05:09.870 align:middle line:84% So in that case, why wouldn't we want 01:05:09.870 --> 01:05:12.960 align:middle line:90% to also change the embeddings? 01:05:12.960 --> 01:05:16.820 align:middle line:84% I understand why you would like to start with those weights 01:05:16.820 --> 01:05:19.960 align:middle line:84% that some people have spent a lot of money trying to find, 01:05:19.960 --> 01:05:24.180 align:middle line:84% but will we be able to find more specific embeddings related 01:05:24.180 --> 01:05:27.330 align:middle line:84% to our problem if we let everything be trainable? 01:05:27.330 --> 01:05:29.080 align:middle line:84% RAMA RAMAKRISHNAN: Absolutely, absolutely. 01:05:29.080 --> 01:05:33.040 align:middle line:84% And in fact, you will see in the Colab that we will do that next. 01:05:33.040 --> 01:05:35.508 align:middle line:84% I just want to show people you don't have to do it. 01:05:35.508 --> 01:05:37.300 align:middle line:84% You start with not training it because it's 01:05:37.300 --> 01:05:39.550 align:middle line:84% going to be much faster, and then you train everything 01:05:39.550 --> 01:05:40.640 align:middle line:90% and see if it gets better. 01:05:40.640 --> 01:05:41.900 align:middle line:84% And sometimes it will get better, in which case 01:05:41.900 --> 01:05:42.600 align:middle line:90% it's great. 01:05:42.600 --> 01:05:43.900 align:middle line:90% Sometimes it won't get better. 01:05:43.900 --> 01:05:45.620 align:middle line:84% And I will also show you-- and I probably 01:05:45.620 --> 01:05:47.677 align:middle line:84% will run out of time, which I'll do it on Monday. 01:05:47.677 --> 01:05:49.260 align:middle line:84% I will also show you, hey, what if you 01:05:49.260 --> 01:05:51.635 align:middle line:84% want to do your own embeddings from scratch without using 01:05:51.635 --> 01:05:52.620 align:middle line:90% GloVe? 01:05:52.620 --> 01:05:55.840 align:middle line:84% So all possibilities will be covered. 01:05:55.840 --> 01:05:56.340 align:middle line:90% Yeah. 01:05:56.340 --> 01:05:59.340 align:middle line:84% So to come back to this-- this is the model we have. 01:05:59.340 --> 01:06:01.460 align:middle line:90% And then-- all right. 01:06:01.460 --> 01:06:04.540 align:middle line:84% So if we take a look at the first few embedding vectors-- 01:06:04.540 --> 01:06:07.700 align:middle line:84% by the way, this model.layers will give you 01:06:07.700 --> 01:06:09.490 align:middle line:84% every layer as a list of all the layers, 01:06:09.490 --> 01:06:11.740 align:middle line:84% and then you can just grab any layer you want and look 01:06:11.740 --> 01:06:12.820 align:middle line:90% at its weights. 01:06:12.820 --> 01:06:13.785 align:middle line:90% It's very handy. 01:06:13.785 --> 01:06:15.160 align:middle line:84% So we are looking at the weights, 01:06:15.160 --> 01:06:18.340 align:middle line:84% and you can see here the first two 01:06:18.340 --> 01:06:21.440 align:middle line:84% vectors are all zeros because that stands for unk and pad, 01:06:21.440 --> 01:06:22.880 align:middle line:84% and then we have everything else. 01:06:22.880 --> 01:06:24.580 align:middle line:90% So everything looks fine so far. 01:06:24.580 --> 01:06:26.600 align:middle line:84% And now we just compile and fit it. 01:06:26.600 --> 01:06:30.740 align:middle line:84% So as usual, Adam cross-entropy accuracy, and then 01:06:30.740 --> 01:06:33.420 align:middle line:90% we will just fit the model. 01:06:33.420 --> 01:06:34.820 align:middle line:90% All right. 01:06:34.820 --> 01:06:39.020 align:middle line:84% It's going to take a few minutes. 01:06:39.020 --> 01:06:42.340 align:middle line:84% And while it's running, so what you will see in this Colab 01:06:42.340 --> 01:06:45.820 align:middle line:84% is that in this particular case, the embeddings actually 01:06:45.820 --> 01:06:47.500 align:middle line:90% don't help a whole lot. 01:06:47.500 --> 01:06:48.680 align:middle line:90% Why do you think that is? 01:06:48.680 --> 01:06:51.822 align:middle line:90% 01:06:51.822 --> 01:06:54.280 align:middle line:84% Part of it could be because we're averaging a lot of stuff. 01:06:54.280 --> 01:06:57.180 align:middle line:90% Maybe that's hurting us. 01:06:57.180 --> 01:06:58.090 align:middle line:90% Yeah? 01:06:58.090 --> 01:07:00.340 align:middle line:84% AUDIENCE: I mean, I think that embeddings are probably 01:07:00.340 --> 01:07:02.820 align:middle line:84% trained on some corpus, right-- like Wikipedia, 01:07:02.820 --> 01:07:05.460 align:middle line:84% or something like that-- that is a little bit 01:07:05.460 --> 01:07:08.110 align:middle line:84% different from the language we tend to use in song lyrics. 01:07:08.110 --> 01:07:10.250 align:middle line:84% And so maybe it's not its ability 01:07:10.250 --> 01:07:16.770 align:middle line:84% to extract the meaning of candy from a song lyric, 01:07:16.770 --> 01:07:19.810 align:middle line:84% maybe it's limited because it's thinking of all the other ways 01:07:19.810 --> 01:07:20.950 align:middle line:90% that could be represented. 01:07:20.950 --> 01:07:21.330 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so there 01:07:21.330 --> 01:07:23.530 align:middle line:84% could be a mismatch between the corpus on which 01:07:23.530 --> 01:07:26.692 align:middle line:84% the pre-trained stuff was trained on versus the corpus 01:07:26.692 --> 01:07:28.150 align:middle line:84% that you're working with right now. 01:07:28.150 --> 01:07:29.370 align:middle line:90% That's one big reason. 01:07:29.370 --> 01:07:32.150 align:middle line:84% The other reason is that we actually may have-- 01:07:32.150 --> 01:07:34.690 align:middle line:84% we have 50,000 examples, basically. 01:07:34.690 --> 01:07:36.090 align:middle line:90% It's a lot of data. 01:07:36.090 --> 01:07:37.930 align:middle line:84% So when you have a lot of data, you 01:07:37.930 --> 01:07:39.817 align:middle line:84% may not need any of these things. 01:07:39.817 --> 01:07:41.650 align:middle line:84% These things tend to do really well when you 01:07:41.650 --> 01:07:43.730 align:middle line:84% don't have a lot of data, which means 01:07:43.730 --> 01:07:47.530 align:middle line:84% you get to piggyback on what these embeddings have learned 01:07:47.530 --> 01:07:49.970 align:middle line:90% from all of Wikipedia. 01:07:49.970 --> 01:07:54.330 align:middle line:84% So when you have a smallish data set, basically the rule of thumb 01:07:54.330 --> 01:07:56.390 align:middle line:84% here is that when your data is really small, 01:07:56.390 --> 01:07:58.385 align:middle line:90% try to use a pre-trained model. 01:07:58.385 --> 01:08:01.010 align:middle line:84% And that's what you saw with the handbags and shoes classifier. 01:08:01.010 --> 01:08:03.170 align:middle line:84% We had 100 examples of handbags and shoes, 01:08:03.170 --> 01:08:06.600 align:middle line:84% and we used ResNet to basically get to 100% accuracy. 01:08:06.600 --> 01:08:08.320 align:middle line:90% The same of logic applies here. 01:08:08.320 --> 01:08:08.820 align:middle line:90% All right. 01:08:08.820 --> 01:08:11.560 align:middle line:84% So here, let's see what's happening. 01:08:11.560 --> 01:08:12.840 align:middle line:90% OK, it's done. 01:08:12.840 --> 01:08:13.470 align:middle line:90% So we'll plot. 01:08:13.470 --> 01:08:16.080 align:middle line:90% 01:08:16.080 --> 01:08:16.920 align:middle line:90% Right. 01:08:16.920 --> 01:08:17.420 align:middle line:90% OK. 01:08:17.420 --> 01:08:17.920 align:middle line:90% Let's look. 01:08:17.920 --> 01:08:20.250 align:middle line:84% A very well-behaved loss function curve. 01:08:20.250 --> 01:08:25.680 align:middle line:90% 01:08:25.680 --> 01:08:26.480 align:middle line:90% OK. 01:08:26.480 --> 01:08:29.700 align:middle line:84% So there doesn't seem to be any massive overfitting going on. 01:08:29.700 --> 01:08:31.800 align:middle line:84% They're moving really nicely in lockstep. 01:08:31.800 --> 01:08:33.500 align:middle line:90% Let's see what the thing is. 01:08:33.500 --> 01:08:36.359 align:middle line:90% 01:08:36.359 --> 01:08:39.640 align:middle line:90% OK, 63%, which is not great. 01:08:39.640 --> 01:08:42.498 align:middle line:84% It's not as good as what we saw before when we used all 50,000 01:08:42.498 --> 01:08:44.540 align:middle line:84% examples and just trained something from scratch. 01:08:44.540 --> 01:08:47.180 align:middle line:84% And that's just because in this case, we have lots of examples. 01:08:47.180 --> 01:08:49.120 align:middle line:84% These pre-trained embeddings aren't as helpful 01:08:49.120 --> 01:08:50.319 align:middle line:90% as they could be. 01:08:50.319 --> 01:08:53.540 align:middle line:84% But if you have a small data set, they could be very helpful. 01:08:53.540 --> 01:08:57.640 align:middle line:84% And now we go to what he pointed out. 01:08:57.640 --> 01:08:59.779 align:middle line:84% Why can't we just optimize these embeddings too? 01:08:59.779 --> 01:09:01.589 align:middle line:84% Why do we have to treat them as sacred? 01:09:01.589 --> 01:09:04.490 align:middle line:90% 01:09:04.490 --> 01:09:07.950 align:middle line:84% Let's just unleash back prop on it and see what happens. 01:09:07.950 --> 01:09:11.109 align:middle line:90% So we'll do that. 01:09:11.109 --> 01:09:13.430 align:middle line:84% So here, what we do is we retrain it. 01:09:13.430 --> 01:09:15.069 align:middle line:84% But here, we set trainable equals 01:09:15.069 --> 01:09:16.490 align:middle line:90% true for the embedding layer. 01:09:16.490 --> 01:09:16.990 align:middle line:90% OK? 01:09:16.990 --> 01:09:17.930 align:middle line:90% This is the key step. 01:09:17.930 --> 01:09:19.189 align:middle line:90% Trainable equals true. 01:09:19.189 --> 01:09:21.350 align:middle line:90% Otherwise, it's unchanged. 01:09:21.350 --> 01:09:24.250 align:middle line:90% And then skip that. 01:09:24.250 --> 01:09:27.189 align:middle line:90% 01:09:27.189 --> 01:09:28.970 align:middle line:84% We'll run it and see what happens. 01:09:28.970 --> 01:09:32.290 align:middle line:84% So before, it was whatever, 63% accuracy or something. 01:09:32.290 --> 01:09:35.430 align:middle line:84% We'll see if it gets better if you train the whole thing. 01:09:35.430 --> 01:09:38.229 align:middle line:84% And the thing is you can never be sure 01:09:38.229 --> 01:09:40.236 align:middle line:84% because it may start to overfit, which 01:09:40.236 --> 01:09:42.569 align:middle line:84% is why you just have to empirically see what's going on. 01:09:42.569 --> 01:09:43.689 align:middle line:90% There are no guarantees. 01:09:43.689 --> 01:09:47.790 align:middle line:90% 01:09:47.790 --> 01:09:48.290 align:middle line:90% All right. 01:09:48.290 --> 01:09:50.149 align:middle line:84% Any questions while it's training? 01:09:50.149 --> 01:09:52.069 align:middle line:90% Yeah? 01:09:52.069 --> 01:09:54.149 align:middle line:84% AUDIENCE: In the first graph, when 01:09:54.149 --> 01:09:56.690 align:middle line:84% you had the training accuracy still increasing, 01:09:56.690 --> 01:09:59.530 align:middle line:84% that might suggest that you could use even more [INAUDIBLE]. 01:09:59.530 --> 01:09:59.990 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 01:09:59.990 --> 01:10:00.970 align:middle line:90% Exactly, exactly. 01:10:00.970 --> 01:10:03.380 align:middle line:84% So in that curve, we saw that the training 01:10:03.380 --> 01:10:04.525 align:middle line:90% was continuing to increase. 01:10:04.525 --> 01:10:05.900 align:middle line:84% Typically, what's going to happen 01:10:05.900 --> 01:10:07.442 align:middle line:84% is that training will continue to get 01:10:07.442 --> 01:10:08.720 align:middle line:90% better the more you train it. 01:10:08.720 --> 01:10:11.460 align:middle line:84% The key thing is the validation also improving. 01:10:11.460 --> 01:10:13.160 align:middle line:84% If the validation continues to improve, 01:10:13.160 --> 01:10:15.160 align:middle line:84% there is a little bit more gas left in the tank. 01:10:15.160 --> 01:10:16.368 align:middle line:90% You can keep increasing more. 01:10:16.368 --> 01:10:17.533 align:middle line:90% If it starts to flatten-- 01:10:17.533 --> 01:10:19.200 align:middle line:84% and even worse, if it starts to go down, 01:10:19.200 --> 01:10:22.020 align:middle line:90% then you want to pull back. 01:10:22.020 --> 01:10:23.580 align:middle line:90% Yeah? 01:10:23.580 --> 01:10:26.060 align:middle line:84% AUDIENCE: So you had used the max tokens to limit 01:10:26.060 --> 01:10:28.760 align:middle line:84% the vocabulary to the most common 5,000, 01:10:28.760 --> 01:10:30.640 align:middle line:84% and then the width of that was 100. 01:10:30.640 --> 01:10:31.403 align:middle line:90% What is the 100? 01:10:31.403 --> 01:10:32.820 align:middle line:84% RAMA RAMAKRISHNAN: The 100 is just 01:10:32.820 --> 01:10:34.712 align:middle line:90% the length of the GloVe vector. 01:10:34.712 --> 01:10:36.420 align:middle line:84% AUDIENCE: Does that mean that it can only 01:10:36.420 --> 01:10:39.817 align:middle line:84% capture how that word is related to 100 other words? 01:10:39.817 --> 01:10:40.900 align:middle line:90% RAMA RAMAKRISHNAN: No, no. 01:10:40.900 --> 01:10:43.080 align:middle line:84% Basically, we are saying that every word, 01:10:43.080 --> 01:10:45.620 align:middle line:84% its intrinsic meaning can be captured using 01:10:45.620 --> 01:10:48.133 align:middle line:90% a vector of 100 dimensions. 01:10:48.133 --> 01:10:49.800 align:middle line:84% Each of those dimensions mean something. 01:10:49.800 --> 01:10:50.940 align:middle line:90% We don't know what it is. 01:10:50.940 --> 01:10:53.380 align:middle line:84% The first dimension could mean color. 01:10:53.380 --> 01:10:55.320 align:middle line:84% The second could mean some sort of location. 01:10:55.320 --> 01:10:58.080 align:middle line:84% The third could mean some sort of time of the year. 01:10:58.080 --> 01:11:00.918 align:middle line:90% We just have no idea. 01:11:00.918 --> 01:11:01.460 align:middle line:90% AUDIENCE: OK. 01:11:01.460 --> 01:11:03.335 align:middle line:84% And the pre-trained model-- because we're not 01:11:03.335 --> 01:11:05.127 align:middle line:84% allowing it to learn the pre-trained model, 01:11:05.127 --> 01:11:06.280 align:middle line:90% it has those already. 01:11:06.280 --> 01:11:08.580 align:middle line:84% We don't know what they are, but it has some-- 01:11:08.580 --> 01:11:09.020 align:middle line:84% RAMA RAMAKRISHNAN: The people who 01:11:09.020 --> 01:11:10.600 align:middle line:84% created it don't know what they are either. 01:11:10.600 --> 01:11:11.020 align:middle line:90% AUDIENCE: Yeah. 01:11:11.020 --> 01:11:13.312 align:middle line:84% RAMA RAMAKRISHNAN: All they know is that for each word, 01:11:13.312 --> 01:11:15.860 align:middle line:90% they learned 100 long vector. 01:11:15.860 --> 01:11:19.700 align:middle line:84% And that 100 long vector was able to recreate 01:11:19.700 --> 01:11:21.540 align:middle line:90% the co-occurrence matrix. 01:11:21.540 --> 01:11:24.180 align:middle line:84% And then they probed it using that visualization 01:11:24.180 --> 01:11:26.620 align:middle line:84% of man, woman, sister, brother, all that stuff, and it 01:11:26.620 --> 01:11:29.940 align:middle line:84% seems to fit with what we would expect. 01:11:29.940 --> 01:11:31.980 align:middle line:84% AUDIENCE: Can you think of it as analogous to-- 01:11:31.980 --> 01:11:33.560 align:middle line:84% when we did the convolutional ones, 01:11:33.560 --> 01:11:35.100 align:middle line:90% you have the number of kernels. 01:11:35.100 --> 01:11:37.700 align:middle line:84% So in this case, so if you had 32 kernels, 01:11:37.700 --> 01:11:40.062 align:middle line:84% it's like 32 things it can learn. 01:11:40.062 --> 01:11:42.520 align:middle line:84% RAMA RAMAKRISHNAN: I think that's actually a great analogy. 01:11:42.520 --> 01:11:43.138 align:middle line:90% I love it. 01:11:43.138 --> 01:11:44.680 align:middle line:84% That's a great way to think about it. 01:11:44.680 --> 01:11:45.500 align:middle line:90% Yes. 01:11:45.500 --> 01:11:48.438 align:middle line:84% Much like we got to decide how many filters to have, 01:11:48.438 --> 01:11:50.980 align:middle line:84% here, we get to decide how long the embedding dimension needs 01:11:50.980 --> 01:11:51.480 align:middle line:90% to be. 01:11:51.480 --> 01:11:53.208 align:middle line:84% And our hope is that the more things 01:11:53.208 --> 01:11:55.500 align:middle line:84% we are able to accommodate, the more complicated things 01:11:55.500 --> 01:11:56.942 align:middle line:90% it will pick up. 01:11:56.942 --> 01:11:59.650 align:middle line:84% At the same time, you don't want to have too many of these things 01:11:59.650 --> 01:12:01.930 align:middle line:84% because it's going to start picking up noise, 01:12:01.930 --> 01:12:05.890 align:middle line:90% and that's never a good thing. 01:12:05.890 --> 01:12:07.990 align:middle line:90% OK. 01:12:07.990 --> 01:12:09.930 align:middle line:84% Was that a question on this side? 01:12:09.930 --> 01:12:10.890 align:middle line:90% Yeah? 01:12:10.890 --> 01:12:11.975 align:middle line:90% Go ahead. 01:12:11.975 --> 01:12:14.410 align:middle line:90% AUDIENCE: [INAUDIBLE]. 01:12:14.410 --> 01:12:18.050 align:middle line:84% Why do we use embeddings and not the actual correlation 01:12:18.050 --> 01:12:22.690 align:middle line:90% matrix rows to represent words? 01:12:22.690 --> 01:12:23.857 align:middle line:90% Why do we need to abstract-- 01:12:23.857 --> 01:12:25.357 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, yeah, yeah. 01:12:25.357 --> 01:12:26.850 align:middle line:90% That's actually a good-- 01:12:26.850 --> 01:12:28.650 align:middle line:90% that's a good question. 01:12:28.650 --> 01:12:31.750 align:middle line:84% One immediate reason is that row is 500,000 vectors long. 01:12:31.750 --> 01:12:33.890 align:middle line:90% 500,000 long. 01:12:33.890 --> 01:12:37.290 align:middle line:84% So you want that compact, dense representation. 01:12:37.290 --> 01:12:39.090 align:middle line:84% The second thing is that thing is 01:12:39.090 --> 01:12:41.750 align:middle line:84% subject to all the counts of the Wikipedia corpus. 01:12:41.750 --> 01:12:43.810 align:middle line:90% It's not normalized. 01:12:43.810 --> 01:12:46.530 align:middle line:84% So you need to normalize it so that if you take any two rows 01:12:46.530 --> 01:12:49.170 align:middle line:84% and do dot product, you will get some number which 01:12:49.170 --> 01:12:50.550 align:middle line:90% is in a narrow range. 01:12:50.550 --> 01:12:53.890 align:middle line:84% Otherwise, things don't become comparable. 01:12:53.890 --> 01:12:56.270 align:middle line:84% Now, both these objections can be handled. 01:12:56.270 --> 01:12:57.197 align:middle line:90% You can normalize. 01:12:57.197 --> 01:12:59.780 align:middle line:84% You can reduce the size of the corpus, and so on and so forth. 01:12:59.780 --> 01:13:01.697 align:middle line:84% And in fact, that used to be a very common way 01:13:01.697 --> 01:13:03.412 align:middle line:90% people used to do it before. 01:13:03.412 --> 01:13:06.120 align:middle line:84% But what they have discovered is that the way we learn embeddings 01:13:06.120 --> 01:13:08.120 align:middle line:84% now tends to be much more effective in practice. 01:13:08.120 --> 01:13:11.080 align:middle line:90% 01:13:11.080 --> 01:13:14.040 align:middle line:84% AUDIENCE: So what we've done is-- 01:13:14.040 --> 01:13:16.560 align:middle line:84% what this process does is it creates 01:13:16.560 --> 01:13:19.720 align:middle line:84% this n dimensional incomprehensible matrix 01:13:19.720 --> 01:13:23.600 align:middle line:84% that captures, in essence, a summarized version 01:13:23.600 --> 01:13:24.652 align:middle line:90% of these relationships? 01:13:24.652 --> 01:13:25.760 align:middle line:90% RAMA RAMAKRISHNAN: Right. 01:13:25.760 --> 01:13:26.260 align:middle line:90% Correct. 01:13:26.260 --> 01:13:29.460 align:middle line:84% A compact representation of relationships, 01:13:29.460 --> 01:13:33.200 align:middle line:84% which is not subject to the size of your vocabulary. 01:13:33.200 --> 01:13:35.140 align:middle line:84% So you find that 1,000 words today. 01:13:35.140 --> 01:13:37.500 align:middle line:84% Tomorrow, somebody comes up with a word called selfie, 01:13:37.500 --> 01:13:39.000 align:middle line:84% which didn't exist five years ago, 01:13:39.000 --> 01:13:41.583 align:middle line:84% and now your corpus has gotten a little bit more. 01:13:41.583 --> 01:13:43.000 align:middle line:84% So here, it's very compact, and it 01:13:43.000 --> 01:13:45.020 align:middle line:84% tends to have a much longer shelf life. 01:13:45.020 --> 01:13:48.120 align:middle line:90% 01:13:48.120 --> 01:13:49.438 align:middle line:90% Yeah. 01:13:49.438 --> 01:13:50.980 align:middle line:84% All right, so let's see where we are. 01:13:50.980 --> 01:13:54.400 align:middle line:90% 01:13:54.400 --> 01:13:56.290 align:middle line:90% OK, so evaluate. 01:13:56.290 --> 01:13:59.110 align:middle line:90% 01:13:59.110 --> 01:14:00.770 align:middle line:90% 68-- 69%, almost. 01:14:00.770 --> 01:14:01.310 align:middle line:90% It was 63. 01:14:01.310 --> 01:14:02.210 align:middle line:90% We're going to 69. 01:14:02.210 --> 01:14:04.870 align:middle line:84% So clearly, here, training the whole thing-- including GloVe-- 01:14:04.870 --> 01:14:06.110 align:middle line:90% actually helps. 01:14:06.110 --> 01:14:11.410 align:middle line:84% And so that begs the question-- well, if training GloVe helps, 01:14:11.410 --> 01:14:15.030 align:middle line:84% maybe we should actually train the whole thing from scratch. 01:14:15.030 --> 01:14:16.750 align:middle line:90% Like, why the hell not? 01:14:16.750 --> 01:14:17.730 align:middle line:90% Why the heck not? 01:14:17.730 --> 01:14:19.470 align:middle line:90% I apologize. 01:14:19.470 --> 01:14:21.310 align:middle line:84% So what we'll do is we'll actually 01:14:21.310 --> 01:14:23.288 align:middle line:84% create our own embeddings and just train them. 01:14:23.288 --> 01:14:25.830 align:middle line:84% And here, we don't have to worry about co-occurrence matrices 01:14:25.830 --> 01:14:27.230 align:middle line:84% and so on and so forth because we 01:14:27.230 --> 01:14:28.530 align:middle line:90% have a very specific objective. 01:14:28.530 --> 01:14:30.630 align:middle line:84% We want to be very accurate in predicting genre 01:14:30.630 --> 01:14:32.310 align:middle line:90% for these songs. 01:14:32.310 --> 01:14:35.198 align:middle line:84% The people who had worked on GloVe, 01:14:35.198 --> 01:14:36.490 align:middle line:90% they didn't have any objective. 01:14:36.490 --> 01:14:38.032 align:middle line:84% They just wanted to create embeddings 01:14:38.032 --> 01:14:40.210 align:middle line:90% that were generally useful. 01:14:40.210 --> 01:14:43.550 align:middle line:84% Here, we want to be specifically useful for genre prediction. 01:14:43.550 --> 01:14:46.150 align:middle line:84% And so what we can do is we can actually train 01:14:46.150 --> 01:14:48.670 align:middle line:90% the whole thing ourselves. 01:14:48.670 --> 01:14:50.350 align:middle line:90% We can actually give it-- 01:14:50.350 --> 01:14:52.350 align:middle line:84% we can actually put an embedding layer here. 01:14:52.350 --> 01:14:54.740 align:middle line:84% We just arbitrarily decided to choose 64 01:14:54.740 --> 01:14:57.400 align:middle line:84% as the dimension, as opposed to 100. 01:14:57.400 --> 01:14:58.772 align:middle line:90% It'll run faster. 01:14:58.772 --> 01:14:59.980 align:middle line:90% And then it's the same thing. 01:14:59.980 --> 01:15:02.188 align:middle line:84% Global average pooling, activation, blah, blah, blah, 01:15:02.188 --> 01:15:03.540 align:middle line:90% blah, blah. 01:15:03.540 --> 01:15:05.320 align:middle line:90% And then you run it. 01:15:05.320 --> 01:15:08.077 align:middle line:90% 01:15:08.077 --> 01:15:09.910 align:middle line:84% Let's see if it finishes in the next minute. 01:15:09.910 --> 01:15:12.780 align:middle line:90% 01:15:12.780 --> 01:15:14.558 align:middle line:84% And we'll see if it actually does better 01:15:14.558 --> 01:15:17.100 align:middle line:84% than the pre-trained embeddings or the pre-trained embeddings 01:15:17.100 --> 01:15:18.820 align:middle line:84% that have been further fine tuned. 01:15:18.820 --> 01:15:22.300 align:middle line:84% And I don't remember what I saw when I ran it yesterday. 01:15:22.300 --> 01:15:25.020 align:middle line:84% And while it's running, other questions? 01:15:25.020 --> 01:15:26.198 align:middle line:90% Yeah? 01:15:26.198 --> 01:15:28.240 align:middle line:84% AUDIENCE: So my question is regarding embeddings. 01:15:28.240 --> 01:15:30.980 align:middle line:84% When you call embedding for a particular word, 01:15:30.980 --> 01:15:33.560 align:middle line:84% we indicate that we have certain amount of parameters. 01:15:33.560 --> 01:15:36.240 align:middle line:84% Let's say in this case, we have defined the predefined 100, 01:15:36.240 --> 01:15:37.990 align:middle line:84% so there'll be 100 parameters, and they'll 01:15:37.990 --> 01:15:40.180 align:middle line:84% be coefficient weights for each of them. 01:15:40.180 --> 01:15:44.550 align:middle line:84% So when we take a pre-trained model like the one we took from 01:15:44.550 --> 01:15:47.260 align:middle line:84% GloVe, so for each word, there would already be those number 01:15:47.260 --> 01:15:49.240 align:middle line:90% of parameters in terms of-- 01:15:49.240 --> 01:15:49.540 align:middle line:90% RAMA RAMAKRISHNAN: Right, yeah. 01:15:49.540 --> 01:15:51.957 align:middle line:84% AUDIENCE: But then how do we redefine them as that we want 01:15:51.957 --> 01:15:54.357 align:middle line:84% only 100 or we want only 10 parameters...? 01:15:54.357 --> 01:15:55.440 align:middle line:90% RAMA RAMAKRISHNAN: No, no. 01:15:55.440 --> 01:15:57.660 align:middle line:84% The GloVe thing actually gives you 01:15:57.660 --> 01:16:00.500 align:middle line:84% packaged-- it's prepackaged to be 100 long. 01:16:00.500 --> 01:16:03.000 align:middle line:84% I think they have 200 and 300 as well, if I recall. 01:16:03.000 --> 01:16:05.620 align:middle line:84% We just happen to use the one with 100. 01:16:05.620 --> 01:16:07.780 align:middle line:84% AUDIENCE: The one that is available [INAUDIBLE]? 01:16:07.780 --> 01:16:08.380 align:middle line:90% RAMA RAMAKRISHNAN: Yeah, yeah. 01:16:08.380 --> 01:16:10.922 align:middle line:84% And there are many available, we just get to pick and choose, 01:16:10.922 --> 01:16:13.980 align:middle line:90% and I happen to pick 100. 01:16:13.980 --> 01:16:14.860 align:middle line:90% Oh, OK. 01:16:14.860 --> 01:16:18.700 align:middle line:84% It's a bit slow, but it's actually looking promising. 01:16:18.700 --> 01:16:19.300 align:middle line:90% 9:55. 01:16:19.300 --> 01:16:21.180 align:middle line:90% Yeah? 01:16:21.180 --> 01:16:22.660 align:middle line:84% AUDIENCE: So during the CNN models 01:16:22.660 --> 01:16:25.580 align:middle line:84% or even during our assignments, changing 01:16:25.580 --> 01:16:29.860 align:middle line:84% the filters gave us more depth and improvement in performance. 01:16:29.860 --> 01:16:32.898 align:middle line:84% So here, would I be right in concluding that it's actually 01:16:32.898 --> 01:16:34.940 align:middle line:84% training the embeddings, which is giving us more, 01:16:34.940 --> 01:16:36.980 align:middle line:84% assuming that epoch and batch changes are not 01:16:36.980 --> 01:16:38.600 align:middle line:90% going to change as much? 01:16:38.600 --> 01:16:41.160 align:middle line:84% So if I really want a genuine change in performance, 01:16:41.160 --> 01:16:44.600 align:middle line:84% we go to the level of retraining the embeddings? 01:16:44.600 --> 01:16:45.100 align:middle line:90% But-- 01:16:45.100 --> 01:16:45.260 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 01:16:45.260 --> 01:16:47.820 align:middle line:84% So what we saw was that using GloVe assist was OK. 01:16:47.820 --> 01:16:49.820 align:middle line:84% Using GloVe and then training them helped a lot. 01:16:49.820 --> 01:16:51.403 align:middle line:84% And now we are basically saying, well, 01:16:51.403 --> 01:16:54.130 align:middle line:84% what if we just abandon GloVe and train our own embeddings 01:16:54.130 --> 01:16:55.790 align:middle line:90% for our particular problem? 01:16:55.790 --> 01:16:57.503 align:middle line:84% See, GloVe is a general purpose tool, 01:16:57.503 --> 01:16:59.170 align:middle line:84% so a general purpose tool is really good 01:16:59.170 --> 01:17:00.610 align:middle line:90% if you don't have a lot of data. 01:17:00.610 --> 01:17:01.630 align:middle line:90% That's a good starting point. 01:17:01.630 --> 01:17:02.850 align:middle line:84% But when you have a lot of data, you 01:17:02.850 --> 01:17:04.433 align:middle line:84% should always try to do your own thing 01:17:04.433 --> 01:17:05.730 align:middle line:90% and see if it's any better. 01:17:05.730 --> 01:17:07.450 align:middle line:90% And in this case, I-- 01:17:07.450 --> 01:17:08.430 align:middle line:90% well, OK. 01:17:08.430 --> 01:17:10.610 align:middle line:90% I think it's-- 01:17:10.610 --> 01:17:11.110 align:middle line:90% Come on. 01:17:11.110 --> 01:17:13.910 align:middle line:90% It's 9:55. 01:17:13.910 --> 01:17:16.110 align:middle line:84% I thought I was going to enter any moment now. 01:17:16.110 --> 01:17:22.870 align:middle line:84% So-- right, let's look at the thing. 01:17:22.870 --> 01:17:25.970 align:middle line:90% 01:17:25.970 --> 01:17:26.750 align:middle line:90% OK, folks. 01:17:26.750 --> 01:17:28.310 align:middle line:90% So 72%. 01:17:28.310 --> 01:17:30.810 align:middle line:84% So you can actually retrain your own thing because there are 01:17:30.810 --> 01:17:33.110 align:middle line:84% 50,000 examples, and it gives you an even better thing. 01:17:33.110 --> 01:17:33.670 align:middle line:90% Thanks a lot. 01:17:33.670 --> 01:17:35.880 align:middle line:90% Have a good rest of the week. 01:17:35.880 --> 01:17:50.000 align:middle line:90%