WEBVTT 00:00:00.000 --> 00:00:02.495 align:middle line:90% [SQUEAKING] 00:00:02.495 --> 00:00:04.491 align:middle line:90% [RUSTLING] 00:00:04.491 --> 00:00:07.485 align:middle line:90% [CLICKING] 00:00:07.485 --> 00:00:16.452 align:middle line:90% 00:00:16.452 --> 00:00:17.660 align:middle line:90% RAMA RAMAKRISHNAN: All right. 00:00:17.660 --> 00:00:19.800 align:middle line:84% So transformers, even though they were originally 00:00:19.800 --> 00:00:22.080 align:middle line:84% invented for machine translation, 00:00:22.080 --> 00:00:24.320 align:middle line:84% going from English to German and German to French 00:00:24.320 --> 00:00:26.320 align:middle line:84% and so on and so forth, they have 00:00:26.320 --> 00:00:29.600 align:middle line:84% turned out to be an incredibly effective deep neural network 00:00:29.600 --> 00:00:33.482 align:middle line:84% architecture for just, really, a vast array of domains. 00:00:33.482 --> 00:00:35.440 align:middle line:84% It has reached a point where if you're actually 00:00:35.440 --> 00:00:38.640 align:middle line:84% working on a particular problem, you almost reflexively will 00:00:38.640 --> 00:00:40.760 align:middle line:84% try a transformer first, because it's probably 00:00:40.760 --> 00:00:42.920 align:middle line:90% going to be pretty darn good. 00:00:42.920 --> 00:00:43.680 align:middle line:90% OK? 00:00:43.680 --> 00:00:46.480 align:middle line:84% So they have just taken over everything. 00:00:46.480 --> 00:00:49.780 align:middle line:84% And obviously they have transformed translation, 00:00:49.780 --> 00:00:52.080 align:middle line:90% which is the original target-- 00:00:52.080 --> 00:00:54.220 align:middle line:84% Google search, information retrieval, 00:00:54.220 --> 00:00:56.260 align:middle line:84% completely transformed speech recognition, 00:00:56.260 --> 00:00:58.517 align:middle line:84% text to speech, even computer vision. 00:00:58.517 --> 00:01:00.850 align:middle line:84% Even the stuff that we learned with convolutional neural 00:01:00.850 --> 00:01:03.730 align:middle line:84% networks, now, there are transformers for computer vision 00:01:03.730 --> 00:01:07.292 align:middle line:84% problems that are actually quite good. 00:01:07.292 --> 00:01:09.250 align:middle line:84% Which is kind of shocking because they were not 00:01:09.250 --> 00:01:11.290 align:middle line:90% even designed for that. 00:01:11.290 --> 00:01:13.118 align:middle line:90% And then reinforcement learning. 00:01:13.118 --> 00:01:15.410 align:middle line:84% And of course, all the crazy stuff that's going on with 00:01:15.410 --> 00:01:18.290 align:middle line:84% generative AI-- large language models, multimodal models-- 00:01:18.290 --> 00:01:21.450 align:middle line:84% everything-- everything-- runs on a transformer. 00:01:21.450 --> 00:01:25.570 align:middle line:84% And then there are numerous special purpose systems. 00:01:25.570 --> 00:01:27.950 align:middle line:84% And I find these to be even more interesting. 00:01:27.950 --> 00:01:31.210 align:middle line:84% Like AlphaFold, the protein-folding AI-- 00:01:31.210 --> 00:01:33.690 align:middle line:90% runs on a transformer stack. 00:01:33.690 --> 00:01:36.650 align:middle line:84% And I could just list the examples one after the other. 00:01:36.650 --> 00:01:37.630 align:middle line:90% So it's just amazing. 00:01:37.630 --> 00:01:40.690 align:middle line:84% It's incredibly a flexible architecture. 00:01:40.690 --> 00:01:43.650 align:middle line:84% And I think we are lucky to be alive during a time 00:01:43.650 --> 00:01:47.130 align:middle line:90% when such a thing was invented. 00:01:47.130 --> 00:01:50.330 align:middle line:84% And I'm not getting paid to tell you any of this stuff. 00:01:50.330 --> 00:01:50.830 align:middle line:90% All right. 00:01:50.830 --> 00:01:51.630 align:middle line:90% It's just amazing. 00:01:51.630 --> 00:01:53.110 align:middle line:90% So let's get going. 00:01:53.110 --> 00:01:55.090 align:middle line:90% We will use search-- 00:01:55.090 --> 00:01:57.130 align:middle line:84% or, more broadly, information retrieval, 00:01:57.130 --> 00:01:58.717 align:middle line:90% as the motivating use case. 00:01:58.717 --> 00:02:00.300 align:middle line:84% So these are all examples where people 00:02:00.300 --> 00:02:01.883 align:middle line:84% are typing in natural language queries 00:02:01.883 --> 00:02:04.040 align:middle line:84% or uttering natural language queries into a phone, 00:02:04.040 --> 00:02:06.400 align:middle line:84% and we need to make sense of what they want. 00:02:06.400 --> 00:02:09.340 align:middle line:84% And it's not, like, write me a limerick about deep learning, 00:02:09.340 --> 00:02:11.800 align:middle line:84% where there could be many possible right answers. 00:02:11.800 --> 00:02:14.300 align:middle line:84% It's more like, OK, tell me all the flights that are leaving 00:02:14.300 --> 00:02:16.383 align:middle line:84% from Boston to going to LaGuardia tomorrow morning 00:02:16.383 --> 00:02:17.400 align:middle line:90% between 8:00 and 9:00. 00:02:17.400 --> 00:02:18.880 align:middle line:90% Well, you better get it right. 00:02:18.880 --> 00:02:19.380 align:middle line:90% OK. 00:02:19.380 --> 00:02:21.140 align:middle line:90% Accuracy is a high bar. 00:02:21.140 --> 00:02:24.440 align:middle line:84% Or how many customers abandon their shopping cart? 00:02:24.440 --> 00:02:27.020 align:middle line:84% Find all contracts that are up for renewal next month. 00:02:27.020 --> 00:02:31.140 align:middle line:84% Tell me all the customers who ended the phone call to the call 00:02:31.140 --> 00:02:34.260 align:middle line:84% center yesterday not entirely pleased with the transaction. 00:02:34.260 --> 00:02:36.140 align:middle line:90% The list goes on and on. 00:02:36.140 --> 00:02:38.220 align:middle line:84% And so in particular, we will focus 00:02:38.220 --> 00:02:40.980 align:middle line:84% on this travel-related example today. 00:02:40.980 --> 00:02:44.060 align:middle line:84% Find me all flights from Boston to LaGuardia tomorrow morning. 00:02:44.060 --> 00:02:46.020 align:middle line:90% That kind of query. 00:02:46.020 --> 00:02:48.580 align:middle line:84% And so in these sorts of use cases, 00:02:48.580 --> 00:02:51.580 align:middle line:84% a very common approach historically has been, 00:02:51.580 --> 00:02:55.630 align:middle line:84% well, we will take this natural language query 00:02:55.630 --> 00:02:59.210 align:middle line:84% and then we will convert it into a structured query. 00:02:59.210 --> 00:03:01.830 align:middle line:84% By that, I mean we will parse the query 00:03:01.830 --> 00:03:05.430 align:middle line:84% and we'll extract out key things in that query. 00:03:05.430 --> 00:03:07.330 align:middle line:84% Once we extract out those key things, 00:03:07.330 --> 00:03:09.830 align:middle line:84% we will reassemble it into a structured query, 00:03:09.830 --> 00:03:12.310 align:middle line:90% like a SQL query. 00:03:12.310 --> 00:03:15.170 align:middle line:84% SQL is just one example of a possible structured query. 00:03:15.170 --> 00:03:17.150 align:middle line:84% There are many, many ways to structure queries. 00:03:17.150 --> 00:03:20.250 align:middle line:84% But SQL is familiar to lots of people, so I'm using that. 00:03:20.250 --> 00:03:21.810 align:middle line:90% So you take the SQL. 00:03:21.810 --> 00:03:23.390 align:middle line:84% Once you have the SQL query, you're 00:03:23.390 --> 00:03:26.390 align:middle line:84% in a very comfortable structured land, in which case 00:03:26.390 --> 00:03:29.370 align:middle line:84% you just run the query through some database that you have, 00:03:29.370 --> 00:03:33.050 align:middle line:84% get the results back, format it nicely, and show it to the user. 00:03:33.050 --> 00:03:34.710 align:middle line:90% That's the flow. 00:03:34.710 --> 00:03:39.150 align:middle line:84% So the question becomes, how do we automatically 00:03:39.150 --> 00:03:43.910 align:middle line:84% extract all the travel-related entities from this query? 00:03:43.910 --> 00:03:48.870 align:middle line:84% We want to be able to extract BOS, LGA, tomorrow morning. 00:03:48.870 --> 00:03:49.650 align:middle line:90% Flights. 00:03:49.650 --> 00:03:50.150 align:middle line:90% So on. 00:03:50.150 --> 00:03:52.858 align:middle line:84% These are all the travel-related entities we want to extract out. 00:03:52.858 --> 00:03:54.840 align:middle line:90% That's the problem. 00:03:54.840 --> 00:03:59.040 align:middle line:84% And so we will use a really cool data set called the Airline 00:03:59.040 --> 00:04:00.740 align:middle line:84% Travel Information Systems Data Set-- 00:04:00.740 --> 00:04:02.660 align:middle line:84% and I'll explain the data set in just a bit. 00:04:02.660 --> 00:04:05.160 align:middle line:84% We'll use this as the basis for this example. 00:04:05.160 --> 00:04:07.240 align:middle line:84% And so the way we think about it is 00:04:07.240 --> 00:04:12.120 align:middle line:84% that we have a whole bunch of queries in this data set. 00:04:12.120 --> 00:04:14.400 align:middle line:84% And fortunately for us, the researchers 00:04:14.400 --> 00:04:17.839 align:middle line:84% who compiled this data set, they went through every one 00:04:17.839 --> 00:04:19.360 align:middle line:90% of these queries. 00:04:19.360 --> 00:04:21.920 align:middle line:84% And we have several thousands of them. 00:04:21.920 --> 00:04:24.000 align:middle line:84% They went through every one of those queries, 00:04:24.000 --> 00:04:26.640 align:middle line:84% and they manually tagged each word 00:04:26.640 --> 00:04:31.520 align:middle line:84% in the query with what kind of travel entity it is-- 00:04:31.520 --> 00:04:32.620 align:middle line:90% or none of them. 00:04:32.620 --> 00:04:36.100 align:middle line:84% So for instance, so they call them slots. 00:04:36.100 --> 00:04:38.120 align:middle line:84% So they will take each word in the query 00:04:38.120 --> 00:04:41.200 align:middle line:84% and assign it to a slot, a particular kind of slot. 00:04:41.200 --> 00:04:43.460 align:middle line:84% And I'll explain what "slot" means in just a second. 00:04:43.460 --> 00:04:44.740 align:middle line:90% That's the basic idea. 00:04:44.740 --> 00:04:47.720 align:middle line:84% So for example, if you have something like, 00:04:47.720 --> 00:04:49.800 align:middle line:90% I want to fly from-- 00:04:49.800 --> 00:04:50.480 align:middle line:90% OK? 00:04:50.480 --> 00:04:52.690 align:middle line:84% And this is a flights database, so 00:04:52.690 --> 00:04:55.530 align:middle line:84% you, can assume that everything is related to a flight flying. 00:04:55.530 --> 00:04:58.610 align:middle line:84% So if you have all these words-- 'I want to fly from"-- 00:04:58.610 --> 00:05:00.770 align:middle line:84% each of these words, these five words, 00:05:00.770 --> 00:05:04.610 align:middle line:84% gets mapped to something called the O-- which means other. 00:05:04.610 --> 00:05:06.050 align:middle line:90% It's the other slot. 00:05:06.050 --> 00:05:07.310 align:middle line:90% We don't really care about it. 00:05:07.310 --> 00:05:09.250 align:middle line:90% It's the other slot. 00:05:09.250 --> 00:05:11.610 align:middle line:90% And then we come to Boston. 00:05:11.610 --> 00:05:13.450 align:middle line:84% Oh, Boston is very special, right? 00:05:13.450 --> 00:05:16.810 align:middle line:84% Because it's clearly a departure city. 00:05:16.810 --> 00:05:18.270 align:middle line:90% So we actually tag it. 00:05:18.270 --> 00:05:19.585 align:middle line:90% We assign it this label. 00:05:19.585 --> 00:05:21.710 align:middle line:84% Think of it as just like a classification problem-- 00:05:21.710 --> 00:05:23.810 align:middle line:84% a multi-class classification problem. 00:05:23.810 --> 00:05:29.250 align:middle line:84% So we assign it to B-fromloc.city_name. 00:05:29.250 --> 00:05:31.210 align:middle line:90% That is the label you assign it. 00:05:31.210 --> 00:05:34.290 align:middle line:90% And then you go to at. 00:05:34.290 --> 00:05:35.290 align:middle line:90% You don't care about at. 00:05:35.290 --> 00:05:36.770 align:middle line:90% It's O-- other. 00:05:36.770 --> 00:05:38.650 align:middle line:90% You come to 7:00 AM. 00:05:38.650 --> 00:05:40.910 align:middle line:84% And then, OK, that is depart time. 00:05:40.910 --> 00:05:42.330 align:middle line:90% So depart time. 00:05:42.330 --> 00:05:43.670 align:middle line:90% And then another depart time. 00:05:43.670 --> 00:05:48.130 align:middle line:84% And here you see there is a B and then there is an I, right? 00:05:48.130 --> 00:05:51.860 align:middle line:84% So what we are saying here is that there could be entities who 00:05:51.860 --> 00:05:55.940 align:middle line:84% are described using more than one word, like 7:00 AM-- 00:05:55.940 --> 00:05:57.315 align:middle line:90% two tokens. 00:05:57.315 --> 00:05:59.440 align:middle line:84% And for that, we need to be able to figure out, OK. 00:05:59.440 --> 00:06:01.780 align:middle line:90% The second token is really-- 00:06:01.780 --> 00:06:03.280 align:middle line:90% is part of the first token. 00:06:03.280 --> 00:06:06.100 align:middle line:84% Together, they define the notion of a departure time. 00:06:06.100 --> 00:06:09.960 align:middle line:84% So what the B means that is that this is the word, 00:06:09.960 --> 00:06:11.860 align:middle line:84% this is the token in which we are beginning 00:06:11.860 --> 00:06:13.700 align:middle line:90% the idea of a departure time. 00:06:13.700 --> 00:06:17.940 align:middle line:84% And then "I" means we are in the middle of this description. 00:06:17.940 --> 00:06:19.980 align:middle line:90% B is for Beginning. 00:06:19.980 --> 00:06:23.260 align:middle line:84% So you can see here, so there is a B here. 00:06:23.260 --> 00:06:24.220 align:middle line:90% And there is an I-- 00:06:24.220 --> 00:06:28.100 align:middle line:84% B for Beginning, I for Intermediate, or in the middle. 00:06:28.100 --> 00:06:30.580 align:middle line:90% And then "at," we don't care. 00:06:30.580 --> 00:06:33.960 align:middle line:90% 11, B-arrive_time. 00:06:33.960 --> 00:06:38.260 align:middle line:90% Morning, arrive_time.period. 00:06:38.260 --> 00:06:41.940 align:middle line:84% So this is an example of how you can take a sentence 00:06:41.940 --> 00:06:44.118 align:middle line:84% and then manually label every word 00:06:44.118 --> 00:06:45.660 align:middle line:84% in the sentence with something that's 00:06:45.660 --> 00:06:47.160 align:middle line:84% relevant to your particular problem. 00:06:47.160 --> 00:06:50.390 align:middle line:90% 00:06:50.390 --> 00:06:55.150 align:middle line:84% And it turns out, these people, every word 00:06:55.150 --> 00:06:59.510 align:middle line:84% is classified into one of 123 possibilities. 00:06:59.510 --> 00:07:00.750 align:middle line:90% OK? 00:07:00.750 --> 00:07:04.290 align:middle line:84% So aircraft code, airline code, airline name, 00:07:04.290 --> 00:07:07.570 align:middle line:84% airport code. airport name, arrival date, relative name. 00:07:07.570 --> 00:07:08.950 align:middle line:90% Now, you get the idea. 00:07:08.950 --> 00:07:11.590 align:middle line:84% They want a round trip versus a one way. 00:07:11.590 --> 00:07:12.708 align:middle line:90% The relative to today-- 00:07:12.708 --> 00:07:15.250 align:middle line:84% Because some would say tomorrow morning is relative to today. 00:07:15.250 --> 00:07:17.430 align:middle line:84% So you need absolute time, and you 00:07:17.430 --> 00:07:19.550 align:middle line:90% need a notion of relative time. 00:07:19.550 --> 00:07:21.950 align:middle line:84% So they basically thought of every possibility 00:07:21.950 --> 00:07:23.350 align:middle line:90% with these researchers. 00:07:23.350 --> 00:07:26.230 align:middle line:84% And so every word in every one of these queries 00:07:26.230 --> 00:07:28.260 align:middle line:84% is assigned one of these 123 labels. 00:07:28.260 --> 00:07:32.270 align:middle line:90% 00:07:32.270 --> 00:07:33.400 align:middle line:90% Any questions on the setup? 00:07:33.400 --> 00:07:40.230 align:middle line:90% 00:07:40.230 --> 00:07:42.030 align:middle line:84% AUDIENCE: Did they have to contextualize 00:07:42.030 --> 00:07:44.590 align:middle line:84% what comes before relative Boston? 00:07:44.590 --> 00:07:47.880 align:middle line:84% So if someone says from Boston, so that there should be 00:07:47.880 --> 00:07:50.160 align:middle line:84% contextualization with the "from Boston." 00:07:50.160 --> 00:07:51.480 align:middle line:84% RAMA RAMAKRISHNAN: So because they did it manually, 00:07:51.480 --> 00:07:53.230 align:middle line:84% they could just read it and figure it out, 00:07:53.230 --> 00:07:54.160 align:middle line:90% that's what they mean. 00:07:54.160 --> 00:07:54.700 align:middle line:90% Right? 00:07:54.700 --> 00:07:57.180 align:middle line:84% Boston is the departure city and not the arrival city. 00:07:57.180 --> 00:07:59.097 align:middle line:84% AUDIENCE: So do they have to attach to Boston, 00:07:59.097 --> 00:08:02.680 align:middle line:84% which is some departure city as well as arrival city with 00:08:02.680 --> 00:08:04.027 align:middle line:90% the word "Boston"? 00:08:04.027 --> 00:08:05.860 align:middle line:84% RAMA RAMAKRISHNAN: In the particular phrase, 00:08:05.860 --> 00:08:08.480 align:middle line:84% it's clear from that particular case in the context of it, 00:08:08.480 --> 00:08:11.220 align:middle line:84% as a human reading it, that Boston is a departure city. 00:08:11.220 --> 00:08:14.360 align:middle line:84% So it just only gets that tag in that sentence. 00:08:14.360 --> 00:08:16.860 align:middle line:84% In some other sentence, where people are coming into Boston, 00:08:16.860 --> 00:08:18.068 align:middle line:90% it will have a different tag. 00:08:18.068 --> 00:08:21.200 align:middle line:90% 00:08:21.200 --> 00:08:23.780 align:middle line:84% AUDIENCE: I was wondering if my query, like the others, 00:08:23.780 --> 00:08:28.080 align:middle line:84% basically there is, for example, if my query was "give me flights 00:08:28.080 --> 00:08:32.686 align:middle line:84% from Boston at 7:00 AM," and flights from Denver at 11:00 AM. 00:08:32.686 --> 00:08:34.769 align:middle line:84% RAMA RAMAKRISHNAN: You mean like a compound query? 00:08:34.769 --> 00:08:35.520 align:middle line:90% AUDIENCE: Yeah? 00:08:35.520 --> 00:08:37.228 align:middle line:84% RAMA RAMAKRISHNAN: So this one only takes 00:08:37.228 --> 00:08:40.159 align:middle line:84% single queries into account, because most people are like, 00:08:40.159 --> 00:08:41.659 align:middle line:84% give me a flight from here to there, 00:08:41.659 --> 00:08:43.701 align:middle line:84% or what is the cheapest thing from here to there? 00:08:43.701 --> 00:08:46.070 align:middle line:84% And we'll see examples of queries later on. 00:08:46.070 --> 00:08:50.010 align:middle line:90% 00:08:50.010 --> 00:08:51.530 align:middle line:90% OK. 00:08:51.530 --> 00:08:52.030 align:middle line:90% All right. 00:08:52.030 --> 00:08:53.610 align:middle line:90% So that's the deal. 00:08:53.610 --> 00:08:59.290 align:middle line:84% So basically, what we have, this problem that we have here 00:08:59.290 --> 00:09:02.170 align:middle line:90% is really a word to slot-- 00:09:02.170 --> 00:09:06.450 align:middle line:84% word to slot-- multi-class classification problem. 00:09:06.450 --> 00:09:09.447 align:middle line:84% OK because if you look at that input, 00:09:09.447 --> 00:09:11.030 align:middle line:84% we want to be able to take that input. 00:09:11.030 --> 00:09:12.890 align:middle line:84% And a really good model will then 00:09:12.890 --> 00:09:17.730 align:middle line:84% give you this as the output, because this is 00:09:17.730 --> 00:09:20.250 align:middle line:90% what a human would have done. 00:09:20.250 --> 00:09:23.518 align:middle line:90% So that is our problem. 00:09:23.518 --> 00:09:26.382 align:middle line:90% So the question is-- 00:09:26.382 --> 00:09:28.090 align:middle line:84% the key thing here is that each of the 18 00:09:28.090 --> 00:09:31.010 align:middle line:84% words in this particular example must be assigned 00:09:31.010 --> 00:09:33.030 align:middle line:90% to one of 123 slot types-- 00:09:33.030 --> 00:09:34.230 align:middle line:90% each word. 00:09:34.230 --> 00:09:36.250 align:middle line:84% It's not like we take the entire query 00:09:36.250 --> 00:09:39.670 align:middle line:84% and classify the entire query into one of 123 possibilities. 00:09:39.670 --> 00:09:42.490 align:middle line:84% Every word in the query has to be classified. 00:09:42.490 --> 00:09:45.460 align:middle line:90% That is the wrinkle. 00:09:45.460 --> 00:09:47.020 align:middle line:90% OK? 00:09:47.020 --> 00:09:49.260 align:middle line:84% So now, if we could run the query 00:09:49.260 --> 00:09:54.140 align:middle line:84% through a deep neural network, and generate 18 output nodes, 00:09:54.140 --> 00:09:56.760 align:middle line:84% it goes through some unspecified deep neural network. 00:09:56.760 --> 00:09:58.260 align:middle line:84% And when it comes out the other end, 00:09:58.260 --> 00:10:00.420 align:middle line:90% the output layer has 18 nodes. 00:10:00.420 --> 00:10:01.380 align:middle line:90% OK? 00:10:01.380 --> 00:10:05.840 align:middle line:84% Because that is the dimension of the output that we care about. 00:10:05.840 --> 00:10:07.300 align:middle line:90% 18 in, 18 out. 00:10:07.300 --> 00:10:08.690 align:middle line:90% 18 in, 18 out. 00:10:08.690 --> 00:10:11.760 align:middle line:84% And then, for each one of those 18 nodes, 00:10:11.760 --> 00:10:16.610 align:middle line:84% maybe we could attach a 123-way softmax to each of those 18 00:10:16.610 --> 00:10:17.110 align:middle line:90% outputs. 00:10:17.110 --> 00:10:20.180 align:middle line:90% 00:10:20.180 --> 00:10:22.900 align:middle line:84% By the way, isn't it cool that we can just casually talk about 00:10:22.900 --> 00:10:27.140 align:middle line:84% sticking a 123-way softmax onto each one of 18 nodes? 00:10:27.140 --> 00:10:28.120 align:middle line:90% Folks, wake up. 00:10:28.120 --> 00:10:31.340 align:middle line:90% 00:10:31.340 --> 00:10:32.640 align:middle line:90% You're not easily impressed. 00:10:32.640 --> 00:10:34.820 align:middle line:90% I'm impressed by that. 00:10:34.820 --> 00:10:37.940 align:middle line:90% So, OK. 00:10:37.940 --> 00:10:40.400 align:middle line:90% So here's the key thing, right. 00:10:40.400 --> 00:10:42.350 align:middle line:84% We want to generate an output that has 00:10:42.350 --> 00:10:45.610 align:middle line:90% the same length as the input. 00:10:45.610 --> 00:10:48.110 align:middle line:84% But the problem is, the inputs could be of different lengths 00:10:48.110 --> 00:10:49.050 align:middle line:90% as they come in. 00:10:49.050 --> 00:10:51.090 align:middle line:84% There could be short sentences, large sentences. 00:10:51.090 --> 00:10:52.830 align:middle line:90% We don't know. 00:10:52.830 --> 00:10:56.270 align:middle line:84% Yet we need to accommodate this range, this variable size 00:10:56.270 --> 00:10:57.852 align:middle line:90% of input that's coming in. 00:10:57.852 --> 00:10:59.310 align:middle line:84% But the key thing is the output has 00:10:59.310 --> 00:11:00.768 align:middle line:84% to be the same thing as the input-- 00:11:00.768 --> 00:11:02.490 align:middle line:84% the same cardinality as the input. 00:11:02.490 --> 00:11:02.990 align:middle line:90% OK? 00:11:02.990 --> 00:11:05.510 align:middle line:90% That's a one big requirement. 00:11:05.510 --> 00:11:08.270 align:middle line:84% In addition, we want to take the surrounding context 00:11:08.270 --> 00:11:09.390 align:middle line:90% of each word into account. 00:11:09.390 --> 00:11:12.652 align:middle line:84% To go to [INAUDIBLE] when you see the word "Boston," 00:11:12.652 --> 00:11:15.110 align:middle line:84% you can't conclude whether it's a departure city or arrival 00:11:15.110 --> 00:11:15.870 align:middle line:90% city. 00:11:15.870 --> 00:11:18.190 align:middle line:84% You have to look at what else is going on around it. 00:11:18.190 --> 00:11:19.962 align:middle line:84% Is there a "from," is there a "to." 00:11:19.962 --> 00:11:20.670 align:middle line:90% Things like that. 00:11:20.670 --> 00:11:22.250 align:middle line:90% To figure out how to tag it. 00:11:22.250 --> 00:11:24.258 align:middle line:90% So clearly, the context matters. 00:11:24.258 --> 00:11:26.550 align:middle line:84% And then we clearly have to take the order of the words 00:11:26.550 --> 00:11:27.270 align:middle line:90% into account. 00:11:27.270 --> 00:11:29.430 align:middle line:84% Going from Boston to LaGuardia is 00:11:29.430 --> 00:11:31.670 align:middle line:84% very different than going from LaGuardia to Boston. 00:11:31.670 --> 00:11:34.110 align:middle line:90% So clearly the order matters. 00:11:34.110 --> 00:11:36.530 align:middle line:84% So the context matters and the order matters. 00:11:36.530 --> 00:11:38.780 align:middle line:84% And the output has to be the same length as the input. 00:11:38.780 --> 00:11:42.560 align:middle line:90% 00:11:42.560 --> 00:11:43.880 align:middle line:90% So context matters. 00:11:43.880 --> 00:11:45.720 align:middle line:90% Just a few fun examples. 00:11:45.720 --> 00:11:48.280 align:middle line:84% Remember, from the last week, that the meaning of a word 00:11:48.280 --> 00:11:50.700 align:middle line:84% can change dramatically depending on the context. 00:11:50.700 --> 00:11:54.200 align:middle line:84% And we also saw that the standalone or uncontextual 00:11:54.200 --> 00:11:57.080 align:middle line:84% embeddings that we saw last week, like "glove," 00:11:57.080 --> 00:12:01.000 align:middle line:84% they don't take context into account because they give 00:12:01.000 --> 00:12:04.000 align:middle line:84% a single unique embedding vector to every word. 00:12:04.000 --> 00:12:06.980 align:middle line:84% And if a word ends up having lots of different meanings, 00:12:06.980 --> 00:12:11.680 align:middle line:84% that vector is some mushy average of all those meanings. 00:12:11.680 --> 00:12:15.238 align:middle line:84% So the word "see," "I will see you soon." 00:12:15.238 --> 00:12:16.780 align:middle line:84% "I will see this project to its end." 00:12:16.780 --> 00:12:17.860 align:middle line:90% "I see what you mean." 00:12:17.860 --> 00:12:20.200 align:middle line:84% Very different meanings of the word "see." 00:12:20.200 --> 00:12:22.192 align:middle line:90% This is my favorite-- bank. 00:12:22.192 --> 00:12:23.900 align:middle line:84% "I went to the bank to apply for a loan." 00:12:23.900 --> 00:12:24.942 align:middle line:90% "I'm banking on the job." 00:12:24.942 --> 00:12:27.400 align:middle line:84% I'm standing on the left bank," and so on. 00:12:27.400 --> 00:12:29.840 align:middle line:84% It, the animal or-- this actually very-- 00:12:29.840 --> 00:12:30.710 align:middle line:90% it's a good one. 00:12:30.710 --> 00:12:32.960 align:middle line:84% "The animal didn't cross the street because it was too 00:12:32.960 --> 00:12:33.660 align:middle line:90% tired." 00:12:33.660 --> 00:12:37.250 align:middle line:84% "The animal didn't cross the street because it was too wide." 00:12:37.250 --> 00:12:40.825 align:middle line:84% Can you imagine a deep neural network, looking at this word, 00:12:40.825 --> 00:12:43.450 align:middle line:84% "it," and trying to figure out what the heck does the word "it" 00:12:43.450 --> 00:12:44.370 align:middle line:90% mean? 00:12:44.370 --> 00:12:46.130 align:middle line:90% What is it referring to? 00:12:46.130 --> 00:12:49.130 align:middle line:90% Tricky, right? 00:12:49.130 --> 00:12:51.250 align:middle line:84% And then, if you take the word "station"-- 00:12:51.250 --> 00:12:53.250 align:middle line:84% and I have this station example here because we're going to use 00:12:53.250 --> 00:12:55.210 align:middle line:84% it a bit more in the rest of the lecture-- 00:12:55.210 --> 00:12:59.170 align:middle line:84% the station could be a radio station, a train station, 00:12:59.170 --> 00:13:01.790 align:middle line:84% being stationed somewhere, the International Space Station. 00:13:01.790 --> 00:13:03.010 align:middle line:90% The list goes on. 00:13:03.010 --> 00:13:04.410 align:middle line:90% So clearly, order matters-- 00:13:04.410 --> 00:13:06.010 align:middle line:90% context matters. 00:13:06.010 --> 00:13:10.045 align:middle line:90% And clearly order matters. 00:13:10.045 --> 00:13:11.670 align:middle line:84% You can come up with your own examples. 00:13:11.670 --> 00:13:13.330 align:middle line:90% Let's keep moving. 00:13:13.330 --> 00:13:15.170 align:middle line:90% OK? 00:13:15.170 --> 00:13:22.130 align:middle line:84% So the transformer architecture is a very elegant architecture 00:13:22.130 --> 00:13:26.396 align:middle line:84% which checks these three boxes beautifully. 00:13:26.396 --> 00:13:28.830 align:middle line:84% It takes the context into account, order into account. 00:13:28.830 --> 00:13:32.010 align:middle line:84% And then whatever is produced out there 00:13:32.010 --> 00:13:34.370 align:middle line:84% is the same length as whatever is coming in. 00:13:34.370 --> 00:13:36.940 align:middle line:84% And the reason it's called the transformer, 00:13:36.940 --> 00:13:41.000 align:middle line:84% it is because if 10 things come in, 10 things go out. 00:13:41.000 --> 00:13:43.700 align:middle line:84% But the 10 things that go out are a transformed version 00:13:43.700 --> 00:13:45.980 align:middle line:90% of the 10 things that came in. 00:13:45.980 --> 00:13:48.020 align:middle line:84% That's why it's called the transformer. 00:13:48.020 --> 00:13:51.680 align:middle line:84% If 10 things came in, and like one thing goes out, 00:13:51.680 --> 00:13:53.280 align:middle line:84% well, sure, it's been transformed. 00:13:53.280 --> 00:13:54.040 align:middle line:90% But what is it? 00:13:54.040 --> 00:13:55.320 align:middle line:90% It's some weird thing. 00:13:55.320 --> 00:13:58.640 align:middle line:84% But when 10 comes in and 10 goes out, the 10 is preserved. 00:13:58.640 --> 00:14:00.980 align:middle line:84% Each one is getting transformed in an interesting way. 00:14:00.980 --> 00:14:04.500 align:middle line:84% That's why it's called a transformer. 00:14:04.500 --> 00:14:08.420 align:middle line:84% So developed in 2017, this dramatic impact. 00:14:08.420 --> 00:14:11.100 align:middle line:84% So by the way, the effect of transformer-- 00:14:11.100 --> 00:14:13.300 align:middle line:84% Google had spent a lot of research 00:14:13.300 --> 00:14:16.380 align:middle line:84% on machine translation and, obviously, search. 00:14:16.380 --> 00:14:19.020 align:middle line:84% And then, when the transformer is invented, 00:14:19.020 --> 00:14:21.940 align:middle line:84% they took a model called Bert, which we will 00:14:21.940 --> 00:14:23.620 align:middle line:90% see on Wednesday in detail. 00:14:23.620 --> 00:14:27.720 align:middle line:84% And then they introduced Bert into their search. 00:14:27.720 --> 00:14:29.620 align:middle line:90% And the results were dramatic. 00:14:29.620 --> 00:14:31.900 align:middle line:84% And from what I've read, apparently 00:14:31.900 --> 00:14:34.310 align:middle line:90% the impact of doing that was-- 00:14:34.310 --> 00:14:36.450 align:middle line:84% Typically, when you make an improvement to search, 00:14:36.450 --> 00:14:38.075 align:middle line:84% the improvement is very, very marginal, 00:14:38.075 --> 00:14:40.690 align:middle line:84% because it's already a very heavily optimized system. 00:14:40.690 --> 00:14:42.690 align:middle line:84% And then, when the transformer thing came along, 00:14:42.690 --> 00:14:45.330 align:middle line:84% there was actually a significant jump in search quality. 00:14:45.330 --> 00:14:47.110 align:middle line:84% So for example-- and you can actually read 00:14:47.110 --> 00:14:50.430 align:middle line:84% this blog post which came out when they introduced Bert 00:14:50.430 --> 00:14:51.290 align:middle line:90% into search. 00:14:51.290 --> 00:14:52.830 align:middle line:90% It gives you a bit more detail. 00:14:52.830 --> 00:14:58.550 align:middle line:84% But here, so if you queried something like "Brazil traveler 00:14:58.550 --> 00:15:02.098 align:middle line:84% to USA needs a visa," you would think that it should give you 00:15:02.098 --> 00:15:04.390 align:middle line:84% information about how to get a visa if you're Brazilian 00:15:04.390 --> 00:15:05.910 align:middle line:90% and want to come to the US. 00:15:05.910 --> 00:15:07.950 align:middle line:84% But it turns out the first result was, 00:15:07.950 --> 00:15:13.270 align:middle line:84% how US citizens going to Brazil can get a visa. 00:15:13.270 --> 00:15:16.990 align:middle line:84% So, clearly, it's not taking the order into account. 00:15:16.990 --> 00:15:19.630 align:middle line:84% But once they introduced it, boom, the first thing 00:15:19.630 --> 00:15:24.150 align:middle line:84% was the US Embassy in Brazil, and a page on how to get a visa. 00:15:24.150 --> 00:15:26.830 align:middle line:90% So the effect was dramatic. 00:15:26.830 --> 00:15:30.150 align:middle line:90% And so this is a seminal paper. 00:15:30.150 --> 00:15:32.280 align:middle line:84% And it's actually worth reading the paper. 00:15:32.280 --> 00:15:35.340 align:middle line:84% And it's worth-- and this is the picture. 00:15:35.340 --> 00:15:37.810 align:middle line:84% This is, like, an iconic picture at this point, 00:15:37.810 --> 00:15:39.560 align:middle line:90% in the deep learning community. 00:15:39.560 --> 00:15:41.640 align:middle line:84% And we will actually understand this picture 00:15:41.640 --> 00:15:43.872 align:middle line:90% by the end of Wednesday. 00:15:43.872 --> 00:15:46.580 align:middle line:84% But the funny thing is that when the researchers came up with it, 00:15:46.580 --> 00:15:48.560 align:middle line:84% they didn't realize, in some sense, 00:15:48.560 --> 00:15:51.120 align:middle line:84% like, what they had stumbled on, because they were really 00:15:51.120 --> 00:15:52.927 align:middle line:90% focused on machine translation. 00:15:52.927 --> 00:15:54.760 align:middle line:84% It's only the rest of the research community 00:15:54.760 --> 00:15:56.927 align:middle line:84% that took it and started applying to everything else 00:15:56.927 --> 00:15:59.640 align:middle line:84% and found it to be really, really effective. 00:15:59.640 --> 00:16:01.722 align:middle line:84% So we're going to take each one of these things 00:16:01.722 --> 00:16:03.680 align:middle line:84% and figure out how to address them, and thereby 00:16:03.680 --> 00:16:05.440 align:middle line:90% build up the architecture. 00:16:05.440 --> 00:16:07.660 align:middle line:90% Any questions before I continue? 00:16:07.660 --> 00:16:08.160 align:middle line:90% Yeah. 00:16:08.160 --> 00:16:11.200 align:middle line:90% 00:16:11.200 --> 00:16:15.520 align:middle line:84% AUDIENCE: Is there any benefits to discarding 00:16:15.520 --> 00:16:18.900 align:middle line:84% some of those unclassified O's before it goes out? 00:16:18.900 --> 00:16:21.822 align:middle line:84% So, rather than going like you have 18-word input. 00:16:21.822 --> 00:16:24.280 align:middle line:84% Discarding all the ones that don't actually matter and just 00:16:24.280 --> 00:16:26.213 align:middle line:84% doing, like, an 8-word output, for example? 00:16:26.213 --> 00:16:28.880 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, yeah, I think that's a totally fine way 00:16:28.880 --> 00:16:29.690 align:middle line:90% to think about it. 00:16:29.690 --> 00:16:31.050 align:middle line:84% Basically what you're saying is that can we 00:16:31.050 --> 00:16:32.350 align:middle line:90% have a two-stage model. 00:16:32.350 --> 00:16:35.458 align:middle line:84% The first stage model is like a O, non-O classifier, 00:16:35.458 --> 00:16:37.750 align:middle line:84% and the second stage model only goes after the non-O's. 00:16:37.750 --> 00:16:39.330 align:middle line:84% That's a totally fine way to do it. 00:16:39.330 --> 00:16:40.250 align:middle line:90% Yeah. 00:16:40.250 --> 00:16:42.090 align:middle line:84% But as you can see, even if you go 00:16:42.090 --> 00:16:44.870 align:middle line:84% with just a simple one-stage model, if you use a transformer, 00:16:44.870 --> 00:16:47.130 align:middle line:90% you get fantastic accuracy. 00:16:47.130 --> 00:16:50.690 align:middle line:90% And we'll do the Colab in a bit. 00:16:50.690 --> 00:16:51.390 align:middle line:90% All right. 00:16:51.390 --> 00:16:52.670 align:middle line:90% So let's take the first thing. 00:16:52.670 --> 00:16:55.290 align:middle line:84% How do you take the context of everything around the word 00:16:55.290 --> 00:16:57.050 align:middle line:90% into account? 00:16:57.050 --> 00:17:02.450 align:middle line:84% So let's say that this is the sentence we have "The train 00:17:02.450 --> 00:17:05.010 align:middle line:90% slowly left the station." 00:17:05.010 --> 00:17:08.290 align:middle line:84% For each of these words, we can calculate 00:17:08.290 --> 00:17:09.890 align:middle line:90% a standalone embedding-- 00:17:09.890 --> 00:17:11.329 align:middle line:90% say, something like "left." 00:17:11.329 --> 00:17:14.530 align:middle line:84% So I'm just depicting these standalone embeddings 00:17:14.530 --> 00:17:17.980 align:middle line:90% using these thingies here. 00:17:17.980 --> 00:17:19.730 align:middle line:84% Please appreciate them, because it took me 00:17:19.730 --> 00:17:22.550 align:middle line:84% a while to get them to do in PowerPoint. 00:17:22.550 --> 00:17:24.589 align:middle line:90% So these are w1 through w6. 00:17:24.589 --> 00:17:28.339 align:middle line:84% These are the vectors standing up Now let's 00:17:28.339 --> 00:17:30.420 align:middle line:84% say that-- so we can easily do that. 00:17:30.420 --> 00:17:33.220 align:middle line:84% Now what we want to figure out is we want to focus on the word 00:17:33.220 --> 00:17:34.533 align:middle line:90% "station." 00:17:34.533 --> 00:17:36.700 align:middle line:84% And since "station" could mean very different things 00:17:36.700 --> 00:17:38.740 align:middle line:84% in different contexts, we want to figure out, 00:17:38.740 --> 00:17:42.140 align:middle line:84% how do we actually take "station's" embedding 00:17:42.140 --> 00:17:45.460 align:middle line:84% and contextualize it using all the other words that are going 00:17:45.460 --> 00:17:47.400 align:middle line:90% on in that sentence. 00:17:47.400 --> 00:17:49.780 align:middle line:90% Clearly, it's a train station. 00:17:49.780 --> 00:17:52.460 align:middle line:84% So we need to take the fact that there is a train involved 00:17:52.460 --> 00:17:55.000 align:middle line:84% to alter the embedding of the word "station." 00:17:55.000 --> 00:17:58.780 align:middle line:84% That's what taking context into account actually means. 00:17:58.780 --> 00:18:03.380 align:middle line:84% So how can we modify "station's" embedding so that it 00:18:03.380 --> 00:18:05.017 align:middle line:84% incorporates all the other words? 00:18:05.017 --> 00:18:05.850 align:middle line:90% That's the question. 00:18:05.850 --> 00:18:08.860 align:middle line:90% 00:18:08.860 --> 00:18:13.980 align:middle line:84% So when you look at it this way, imagine, just for for a moment-- 00:18:13.980 --> 00:18:15.700 align:middle line:90% just for a moment-- 00:18:15.700 --> 00:18:17.660 align:middle line:90% that we-- 00:18:17.660 --> 00:18:19.880 align:middle line:84% Now some of the other words sentence don't matter. 00:18:19.880 --> 00:18:22.220 align:middle line:84% The word "the" probably doesn't matter. 00:18:22.220 --> 00:18:25.140 align:middle line:84% But some of the other words like "train," "slowly," "left" 00:18:25.140 --> 00:18:26.710 align:middle line:90% probably does matter. 00:18:26.710 --> 00:18:29.230 align:middle line:84% And suppose, just magically, we have 00:18:29.230 --> 00:18:32.550 align:middle line:84% been told all the other words in the sentence, 00:18:32.550 --> 00:18:35.310 align:middle line:84% this is how much weight you have to give to them. 00:18:35.310 --> 00:18:36.730 align:middle line:90% These don't give it any weight. 00:18:36.730 --> 00:18:37.990 align:middle line:90% Those give it a lot of weight. 00:18:37.990 --> 00:18:39.813 align:middle line:90% Suppose we are told that. 00:18:39.813 --> 00:18:41.230 align:middle line:84% Or to put it another way, and this 00:18:41.230 --> 00:18:44.230 align:middle line:84% is the word that's heavily used in the literature-- 00:18:44.230 --> 00:18:46.070 align:middle line:84% someone tells you how much attention 00:18:46.070 --> 00:18:47.815 align:middle line:84% to pay to the other words, whether you 00:18:47.815 --> 00:18:49.190 align:middle line:84% have to pay it a lot of attention 00:18:49.190 --> 00:18:51.310 align:middle line:90% or very little attention. 00:18:51.310 --> 00:18:53.910 align:middle line:84% And this, how much attention to pay, 00:18:53.910 --> 00:18:56.010 align:middle line:84% is given in the form of a weight that you can use. 00:18:56.010 --> 00:19:01.150 align:middle line:84% So if you look at it that way, from this notion of which word 00:19:01.150 --> 00:19:04.070 align:middle line:84% should I give a lot of weight to and very little weight to, 00:19:04.070 --> 00:19:05.790 align:middle line:84% in this example, intuitively, which words 00:19:05.790 --> 00:19:07.130 align:middle line:84% do you think should get the most weight, 00:19:07.130 --> 00:19:09.790 align:middle line:84% and which words do you think should get the least weight? 00:19:09.790 --> 00:19:10.590 align:middle line:90% Yeah. 00:19:10.590 --> 00:19:11.390 align:middle line:90% AUDIENCE: "Train." 00:19:11.390 --> 00:19:11.990 align:middle line:90% RAMA RAMAKRISHNAN: "Train." 00:19:11.990 --> 00:19:12.950 align:middle line:90% Correct. 00:19:12.950 --> 00:19:14.397 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:19:14.397 --> 00:19:16.230 align:middle line:84% RAMA RAMAKRISHNAN: You can do one at a time. 00:19:16.230 --> 00:19:16.670 align:middle line:90% "Train." 00:19:16.670 --> 00:19:17.170 align:middle line:90% OK. 00:19:17.170 --> 00:19:18.790 align:middle line:90% Thank you. 00:19:18.790 --> 00:19:20.342 align:middle line:90% OK, others? 00:19:20.342 --> 00:19:21.752 align:middle line:90% AUDIENCE: "Slowly." 00:19:21.752 --> 00:19:23.210 align:middle line:84% RAMA RAMAKRISHNAN: "Slowly," right. 00:19:23.210 --> 00:19:25.300 align:middle line:84% So that also seems to have some bearing to it. 00:19:25.300 --> 00:19:28.640 align:middle line:84% What about words that don't really we don't think 00:19:28.640 --> 00:19:30.936 align:middle line:90% are going to help at all. 00:19:30.936 --> 00:19:31.840 align:middle line:90% AUDIENCE: "The." 00:19:31.840 --> 00:19:32.200 align:middle line:90% RAMA RAMAKRISHNAN: "The." 00:19:32.200 --> 00:19:32.700 align:middle line:90% Exactly. 00:19:32.700 --> 00:19:34.400 align:middle line:84% It probably doesn't do much here. 00:19:34.400 --> 00:19:36.820 align:middle line:84% Some context it actually might make a difference, 00:19:36.820 --> 00:19:38.720 align:middle line:90% but in this sentence maybe not. 00:19:38.720 --> 00:19:40.800 align:middle line:90% Intuitively. 00:19:40.800 --> 00:19:43.700 align:middle line:84% So we should probably give a lot of weight to "train," 00:19:43.700 --> 00:19:45.320 align:middle line:84% maybe a little to "slowly" and "left," 00:19:45.320 --> 00:19:49.130 align:middle line:90% and hardly anything to "the." 00:19:49.130 --> 00:19:52.480 align:middle line:84% And so this intuition that we have 00:19:52.480 --> 00:19:56.600 align:middle line:84% can be written numerically as maybe 00:19:56.600 --> 00:19:59.020 align:middle line:84% we have a bunch of weights that add up to 1. 00:19:59.020 --> 00:20:02.377 align:middle line:90% 00:20:02.377 --> 00:20:03.460 align:middle line:90% Maybe something like this. 00:20:03.460 --> 00:20:05.360 align:middle line:90% So we are saying, the "train"-- 00:20:05.360 --> 00:20:07.080 align:middle line:90% 30% weightage. 00:20:07.080 --> 00:20:10.040 align:middle line:90% Maybe 8% weightage to "left." 00:20:10.040 --> 00:20:13.600 align:middle line:90% Maybe 12% weightage to "slowly." 00:20:13.600 --> 00:20:17.720 align:middle line:84% And then, as you will see here, "station's" own embedding also 00:20:17.720 --> 00:20:21.120 align:middle line:84% plays a role, because we want to take its own standalone 00:20:21.120 --> 00:20:23.078 align:middle line:84% embedding and just move it slightly. 00:20:23.078 --> 00:20:23.870 align:middle line:90% Change it slightly. 00:20:23.870 --> 00:20:26.810 align:middle line:84% Which means that has to be the starting point. 00:20:26.810 --> 00:20:28.270 align:middle line:90% So it will get a lot of weight. 00:20:28.270 --> 00:20:30.810 align:middle line:84% You can't ignore itself, in other words. 00:20:30.810 --> 00:20:33.630 align:middle line:90% So we give it maybe 40% weight. 00:20:33.630 --> 00:20:37.010 align:middle line:84% By the way, these numbers, I just made them up. 00:20:37.010 --> 00:20:38.477 align:middle line:90% Yeah. 00:20:38.477 --> 00:20:39.310 align:middle line:90% AUDIENCE: I'm sorry. 00:20:39.310 --> 00:20:40.227 align:middle line:90% Just a quick question. 00:20:40.227 --> 00:20:44.320 align:middle line:90% So the weights are-- 00:20:44.320 --> 00:20:46.970 align:middle line:84% are they are they standalone to understand the context 00:20:46.970 --> 00:20:50.268 align:middle line:84% the entire sentence, or are they related to "stations" 00:20:50.268 --> 00:20:51.810 align:middle line:84% that we started off with, "stations." 00:20:51.810 --> 00:20:54.998 align:middle line:84% RAMA RAMAKRISHNAN: These six numbers are only pertinent 00:20:54.998 --> 00:20:55.540 align:middle line:90% to "station." 00:20:55.540 --> 00:20:55.816 align:middle line:90% AUDIENCE: Got it. 00:20:55.816 --> 00:20:56.410 align:middle line:90% Right. 00:20:56.410 --> 00:20:57.910 align:middle line:84% RAMA RAMAKRISHNAN: And for each word 00:20:57.910 --> 00:20:59.610 align:middle line:84% we're going to do something similar. 00:20:59.610 --> 00:21:00.443 align:middle line:90% AUDIENCE: Thank you. 00:21:00.443 --> 00:21:01.650 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:21:01.650 --> 00:21:04.700 align:middle line:84% AUDIENCE: At this point, does the model understand order? 00:21:04.700 --> 00:21:06.450 align:middle line:84% Because like, I'm just thinking of "left," 00:21:06.450 --> 00:21:09.610 align:middle line:90% because it gave it a very low-- 00:21:09.610 --> 00:21:10.710 align:middle line:90% a very low weight. 00:21:10.710 --> 00:21:14.270 align:middle line:84% But let's say "left" comes "slowly we left station." 00:21:14.270 --> 00:21:16.070 align:middle line:84% The station only happened to be higher. 00:21:16.070 --> 00:21:17.445 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, correct. 00:21:17.445 --> 00:21:19.450 align:middle line:84% So at this point, we not worrying about order. 00:21:19.450 --> 00:21:22.620 align:middle line:84% We are only really worrying about context. 00:21:22.620 --> 00:21:24.592 align:middle line:84% Later, we will take order into account. 00:21:24.592 --> 00:21:26.425 align:middle line:84% AUDIENCE: How did the model know that "left" 00:21:26.425 --> 00:21:29.380 align:middle line:84% here is of lesser importance because it's a verb rather than 00:21:29.380 --> 00:21:31.100 align:middle line:90% an adjective. 00:21:31.100 --> 00:21:33.840 align:middle line:84% RAMA RAMAKRISHNAN: It has to figure it out. 00:21:33.840 --> 00:21:36.540 align:middle line:84% We are just giving it a whole bunch of capabilities. 00:21:36.540 --> 00:21:38.748 align:middle line:84% How it manifests those capabilities 00:21:38.748 --> 00:21:40.290 align:middle line:84% is all going to emerge from training. 00:21:40.290 --> 00:21:42.900 align:middle line:90% 00:21:42.900 --> 00:21:44.592 align:middle line:90% So, all right. 00:21:44.592 --> 00:21:46.300 align:middle line:84% So let's say we have something like this. 00:21:46.300 --> 00:21:48.140 align:middle line:90% So what we can do-- 00:21:48.140 --> 00:21:49.980 align:middle line:84% and we'll get to the all-important question 00:21:49.980 --> 00:21:52.272 align:middle line:84% of where do we get these numbers from in just a moment. 00:21:52.272 --> 00:21:54.660 align:middle line:90% But suppose you had the numbers. 00:21:54.660 --> 00:21:58.540 align:middle line:84% How can we use these numbers to contextualize w6? 00:21:58.540 --> 00:22:00.420 align:middle line:90% What can we do? 00:22:00.420 --> 00:22:02.160 align:middle line:84% What is the simplest thing you can do? 00:22:02.160 --> 00:22:05.380 align:middle line:90% 00:22:05.380 --> 00:22:06.480 align:middle line:90% You have w6. 00:22:06.480 --> 00:22:09.680 align:middle line:84% You want to make it a new w6, which is now contextual. 00:22:09.680 --> 00:22:11.360 align:middle line:84% Is aware of what else is going on. 00:22:11.360 --> 00:22:15.216 align:middle line:90% 00:22:15.216 --> 00:22:17.037 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:22:17.037 --> 00:22:18.912 align:middle line:84% RAMA RAMAKRISHNAN: It's working now, I think. 00:22:18.912 --> 00:22:19.324 align:middle line:84% AUDIENCE: Take a weighted average. 00:22:19.324 --> 00:22:21.530 align:middle line:84% RAMA RAMAKRISHNAN: We can take a weighted average. 00:22:21.530 --> 00:22:22.493 align:middle line:90% Exactly, exactly. 00:22:22.493 --> 00:22:23.910 align:middle line:84% So when you have a bunch of things 00:22:23.910 --> 00:22:27.470 align:middle line:84% and you have a bunch of weights, and we have to somehow modify 00:22:27.470 --> 00:22:29.130 align:middle line:84% one of those things with those weights, 00:22:29.130 --> 00:22:32.110 align:middle line:84% the simplest thing you can do is to take a weighted average. 00:22:32.110 --> 00:22:34.310 align:middle line:84% So that's exactly what we're going to do. 00:22:34.310 --> 00:22:37.470 align:middle line:84% So we're going to take all these weights and just, 00:22:37.470 --> 00:22:40.670 align:middle line:90% like, move them up. 00:22:40.670 --> 00:22:42.443 align:middle line:90% Move them up. 00:22:42.443 --> 00:22:44.110 align:middle line:84% Don't even get me started on how long it 00:22:44.110 --> 00:22:46.182 align:middle line:90% took me to get this arrow run. 00:22:46.182 --> 00:22:47.390 align:middle line:90% I don't know about you folks. 00:22:47.390 --> 00:22:49.470 align:middle line:84% It is extremely painful to get the U-turn 00:22:49.470 --> 00:22:51.230 align:middle line:90% arrows to work in PowerPoint. 00:22:51.230 --> 00:22:53.850 align:middle line:90% Anyway, back to work. 00:22:53.850 --> 00:22:57.030 align:middle line:90% So we just move these up here. 00:22:57.030 --> 00:23:00.430 align:middle line:84% So now we can do 0.05 times this vector 00:23:00.430 --> 00:23:03.710 align:middle line:84% plus 0.3 times that vector, and so on and so forth. 00:23:03.710 --> 00:23:08.710 align:middle line:84% And the result is just another vector, right? 00:23:08.710 --> 00:23:13.350 align:middle line:84% And that vector, folks, is the contextual embedding vector 00:23:13.350 --> 00:23:15.950 align:middle line:90% of "station." 00:23:15.950 --> 00:23:17.840 align:middle line:90% That was a standalone embedding. 00:23:17.840 --> 00:23:20.920 align:middle line:84% And now we did the-- we multiplied this by that, 00:23:20.920 --> 00:23:21.780 align:middle line:90% added them all up. 00:23:21.780 --> 00:23:24.840 align:middle line:90% And then you get a new vector. 00:23:24.840 --> 00:23:29.600 align:middle line:84% And contextual embeddings have this bluish kind of color. 00:23:29.600 --> 00:23:33.600 align:middle line:84% And I'll maintain that color scheme as we go along. 00:23:33.600 --> 00:23:36.360 align:middle line:90% So that's it. 00:23:36.360 --> 00:23:37.060 align:middle line:90% That's it. 00:23:37.060 --> 00:23:38.480 align:middle line:90% That's the idea. 00:23:38.480 --> 00:23:41.760 align:middle line:90% Any questions? 00:23:41.760 --> 00:23:43.157 align:middle line:90% Yeah. 00:23:43.157 --> 00:23:45.740 align:middle line:84% AUDIENCE: How did you come up with the original weights again? 00:23:45.740 --> 00:23:46.880 align:middle line:90% You just kind of guess? 00:23:46.880 --> 00:23:48.422 align:middle line:84% RAMA RAMAKRISHNAN: No, these weights, 00:23:48.422 --> 00:23:51.280 align:middle line:84% I just hand type them in manually, just 00:23:51.280 --> 00:23:52.075 align:middle line:90% to make the point. 00:23:52.075 --> 00:23:54.200 align:middle line:84% And now I'm going to talk about how we are actually 00:23:54.200 --> 00:23:57.640 align:middle line:90% going to calculate them. 00:23:57.640 --> 00:23:59.000 align:middle line:90% OK. 00:23:59.000 --> 00:23:59.500 align:middle line:90% All right. 00:23:59.500 --> 00:24:00.000 align:middle line:90% Cool. 00:24:00.000 --> 00:24:01.520 align:middle line:90% So now I'm going to-- 00:24:01.520 --> 00:24:02.780 align:middle line:90% OK, enough pictures. 00:24:02.780 --> 00:24:04.200 align:middle line:90% Let's switch to some math. 00:24:04.200 --> 00:24:08.760 align:middle line:84% So basically what I'm-- so let's write it a bit more formally. 00:24:08.760 --> 00:24:11.280 align:middle line:84% So we have these w1 through w6 which 00:24:11.280 --> 00:24:12.920 align:middle line:90% are the standalone embeddings. 00:24:12.920 --> 00:24:16.330 align:middle line:84% And then, for "station," we want to calculate w6 with a little 00:24:16.330 --> 00:24:18.370 align:middle line:84% hat on it, which is the contextual embedding. 00:24:18.370 --> 00:24:22.010 align:middle line:84% And the way we do it is to say we calculate some weights 00:24:22.010 --> 00:24:23.390 align:middle line:90% for each of these words. 00:24:23.390 --> 00:24:28.730 align:middle line:84% So this weight, s1,6, means that the weight of the first word 00:24:28.730 --> 00:24:32.130 align:middle line:84% on the sixth word, which happens to be station. 00:24:32.130 --> 00:24:34.590 align:middle line:84% The weight of the second word on the sixth word, 00:24:34.590 --> 00:24:35.790 align:middle line:90% and so on and so forth. 00:24:35.790 --> 00:24:38.970 align:middle line:84% And so what we are saying is that w6 is just 00:24:38.970 --> 00:24:41.930 align:middle line:84% this weight times w1, this time w-- 00:24:41.930 --> 00:24:42.550 align:middle line:90% that's it. 00:24:42.550 --> 00:24:45.813 align:middle line:90% 00:24:45.813 --> 00:24:47.730 align:middle line:84% I have to inflict all these subscripts and all 00:24:47.730 --> 00:24:51.250 align:middle line:90% that because we need it. 00:24:51.250 --> 00:24:51.750 align:middle line:90% All right. 00:24:51.750 --> 00:24:53.930 align:middle line:90% So that's it. 00:24:53.930 --> 00:24:56.610 align:middle line:90% That's what we have. 00:24:56.610 --> 00:25:00.250 align:middle line:84% Now let's talk about-- any questions on the mechanics of it 00:25:00.250 --> 00:25:01.230 align:middle line:90% before I get to-- 00:25:01.230 --> 00:25:02.930 align:middle line:84% OK, where do these weights come from? 00:25:02.930 --> 00:25:04.260 align:middle line:90% Yeah. 00:25:04.260 --> 00:25:08.910 align:middle line:84% AUDIENCE: [INAUDIBLE] utilizing something like Google, 00:25:08.910 --> 00:25:09.410 align:middle line:90% for example. 00:25:09.410 --> 00:25:14.140 align:middle line:84% How does it understand the context of new words 00:25:14.140 --> 00:25:15.060 align:middle line:90% that comes to play? 00:25:15.060 --> 00:25:18.780 align:middle line:84% Is the process immediately through the training data 00:25:18.780 --> 00:25:20.630 align:middle line:84% the users put in, or what makes it-- 00:25:20.630 --> 00:25:22.380 align:middle line:84% RAMA RAMAKRISHNAN: Like a totally new word 00:25:22.380 --> 00:25:23.638 align:middle line:90% that didn't exist before? 00:25:23.638 --> 00:25:25.180 align:middle line:84% AUDIENCE: A new word or a new context 00:25:25.180 --> 00:25:26.962 align:middle line:90% to a word that already exists? 00:25:26.962 --> 00:25:29.420 align:middle line:84% RAMA RAMAKRISHNAN: No, I think that the context is supplied 00:25:29.420 --> 00:25:31.820 align:middle line:84% because the query coming into something like Google 00:25:31.820 --> 00:25:34.660 align:middle line:84% is a full sentence, and we only take that sentence 00:25:34.660 --> 00:25:37.980 align:middle line:84% and take only the sentence into account as the context for us. 00:25:37.980 --> 00:25:41.620 align:middle line:84% So the context is always present to us when we get the input. 00:25:41.620 --> 00:25:43.922 align:middle line:84% But the other question you had of, OK, 00:25:43.922 --> 00:25:46.380 align:middle line:84% what if there is a brand-new word you've never seen before, 00:25:46.380 --> 00:25:48.505 align:middle line:84% for which there is not even a standalone embedding? 00:25:48.505 --> 00:25:49.780 align:middle line:90% What do you do then? 00:25:49.780 --> 00:25:51.940 align:middle line:84% So let's punt on that till Wednesday, 00:25:51.940 --> 00:25:53.660 align:middle line:84% because I have to talk about something 00:25:53.660 --> 00:25:56.020 align:middle line:84% called byte-pair encoding and stuff like that 00:25:56.020 --> 00:25:57.780 align:middle line:90% before I can answer that. 00:25:57.780 --> 00:25:59.780 align:middle line:84% AUDIENCE: Really quickly-- does that immediately 00:25:59.780 --> 00:26:03.620 align:middle line:84% translate to their predictive search queries? 00:26:03.620 --> 00:26:08.660 align:middle line:84% Utilizing, like, for-- we have a new word, for example. 00:26:08.660 --> 00:26:11.480 align:middle line:84% Does that automatically get applied to the predictive search 00:26:11.480 --> 00:26:13.230 align:middle line:84% queries, like what we were saying, how to, 00:26:13.230 --> 00:26:14.070 align:middle line:90% and then just automatic. 00:26:14.070 --> 00:26:16.320 align:middle line:84% RAMA RAMAKRISHNAN: Oh, you mean like the autocomplete? 00:26:16.320 --> 00:26:19.630 align:middle line:84% Autocomplete uses a slightly different mechanism. 00:26:19.630 --> 00:26:22.470 align:middle line:84% They had a very complicated non-transformer thing 00:26:22.470 --> 00:26:23.235 align:middle line:90% for a long time. 00:26:23.235 --> 00:26:25.110 align:middle line:84% I'm sure they have a transformer version now. 00:26:25.110 --> 00:26:27.670 align:middle line:84% But I'm not privy to how exactly they've done it. 00:26:27.670 --> 00:26:29.275 align:middle line:84% So I don't quite know how they do it. 00:26:29.275 --> 00:26:31.150 align:middle line:84% But what you're proposing is a reasonable way 00:26:31.150 --> 00:26:33.350 align:middle line:90% to think about it. 00:26:33.350 --> 00:26:34.333 align:middle line:90% Yeah. 00:26:34.333 --> 00:26:35.750 align:middle line:84% AUDIENCE: My question is, you take 00:26:35.750 --> 00:26:40.710 align:middle line:84% the path, a particular number of parameters, as in, let's 00:26:40.710 --> 00:26:41.910 align:middle line:90% say, 10 of them. 00:26:41.910 --> 00:26:45.430 align:middle line:84% And then we have calculated the contextual version of w6. 00:26:45.430 --> 00:26:48.123 align:middle line:84% So this gets added as H2 or it is [INAUDIBLE]. 00:26:48.123 --> 00:26:49.290 align:middle line:90% RAMA RAMAKRISHNAN: Replaces. 00:26:49.290 --> 00:26:50.630 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:26:50.630 --> 00:26:55.350 align:middle line:84% RAMA RAMAKRISHNAN: Yeah w6 becomes w6 hat. 00:26:55.350 --> 00:26:57.790 align:middle line:84% AUDIENCE: And now we are extending that. 00:26:57.790 --> 00:26:59.030 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:26:59.030 --> 00:27:01.590 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:27:01.590 --> 00:27:09.643 align:middle line:90% 00:27:09.643 --> 00:27:11.060 align:middle line:84% RAMA RAMAKRISHNAN: No, we lose it. 00:27:11.060 --> 00:27:13.660 align:middle line:84% And as you will see here, as it flows through the transformer, 00:27:13.660 --> 00:27:16.760 align:middle line:84% it's getting more and more and more contextualized. 00:27:16.760 --> 00:27:20.040 align:middle line:90% So it's a left to right flow. 00:27:20.040 --> 00:27:20.892 align:middle line:90% All right. 00:27:20.892 --> 00:27:21.600 align:middle line:90% All right, great. 00:27:21.600 --> 00:27:24.180 align:middle line:84% So by the way, this thing that we did for "station," 00:27:24.180 --> 00:27:28.040 align:middle line:84% we will do it for each word in the sentence-- 00:27:28.040 --> 00:27:29.300 align:middle line:90% the same exact logic. 00:27:29.300 --> 00:27:32.480 align:middle line:84% Obviously, the weights are going to change. 00:27:32.480 --> 00:27:34.840 align:middle line:84% But what will happen is that w1 through W6 00:27:34.840 --> 00:27:39.480 align:middle line:84% will become w1 hat through w6 hat. 00:27:39.480 --> 00:27:41.880 align:middle line:84% The same exact logic is going to hold. 00:27:41.880 --> 00:27:42.800 align:middle line:90% OK. 00:27:42.800 --> 00:27:45.677 align:middle line:84% I just don't have the slides for it because it's a waste of time. 00:27:45.677 --> 00:27:47.260 align:middle line:84% The same exact logic is going to hold. 00:27:47.260 --> 00:27:47.760 align:middle line:90% All right. 00:27:47.760 --> 00:27:50.927 align:middle line:84% Now switch gears and answer the all-important question 00:27:50.927 --> 00:27:52.760 align:middle line:84% of where are the weights going to come from. 00:27:52.760 --> 00:27:56.920 align:middle line:84% So the intuition here is really, really interesting and elegant. 00:27:56.920 --> 00:28:01.040 align:middle line:84% So, clearly, the weight of a word should be proportional 00:28:01.040 --> 00:28:03.620 align:middle line:84% to how related it is to the word "station." 00:28:03.620 --> 00:28:06.150 align:middle line:90% 00:28:06.150 --> 00:28:09.928 align:middle line:84% The word "train" clearly is very related to the word "station." 00:28:09.928 --> 00:28:11.970 align:middle line:84% The word "the," it's not clear how related it is. 00:28:11.970 --> 00:28:13.270 align:middle line:90% Probably not all that related. 00:28:13.270 --> 00:28:16.190 align:middle line:84% So the relatedness matters to the weight. 00:28:16.190 --> 00:28:17.750 align:middle line:90% More related, higher the weight. 00:28:17.750 --> 00:28:19.370 align:middle line:90% Just intuitively. 00:28:19.370 --> 00:28:22.530 align:middle line:84% So one way to quantify how related two words are 00:28:22.530 --> 00:28:24.610 align:middle line:84% is to take their standalone embeddings 00:28:24.610 --> 00:28:25.960 align:middle line:90% and calculate the dot product. 00:28:25.960 --> 00:28:28.836 align:middle line:90% 00:28:28.836 --> 00:28:34.680 align:middle line:84% So in case folks have forgotten about the dot product-- 00:28:34.680 --> 00:28:39.690 align:middle line:90% 00:28:39.690 --> 00:28:41.370 align:middle line:90% oops, that's not what I want. 00:28:41.370 --> 00:28:42.570 align:middle line:90% Here we go. 00:28:42.570 --> 00:28:45.180 align:middle line:90% So let's say you have a vector. 00:28:45.180 --> 00:28:51.050 align:middle line:90% 00:28:51.050 --> 00:28:52.960 align:middle line:84% Let's say this is the vector for "train." 00:28:52.960 --> 00:28:55.770 align:middle line:90% 00:28:55.770 --> 00:28:59.410 align:middle line:84% This is the vector for "station." 00:28:59.410 --> 00:29:02.420 align:middle line:84% So the dot product of these two vectors-- 00:29:02.420 --> 00:29:05.610 align:middle line:90% 00:29:05.610 --> 00:29:15.340 align:middle line:84% and I'll write it as train, station equals, basically, 00:29:15.340 --> 00:29:21.540 align:middle line:84% the length of the vector for "train" 00:29:21.540 --> 00:29:28.580 align:middle line:84% times the length of the vector for "station" 00:29:28.580 --> 00:29:35.440 align:middle line:84% times the cosine of the angle between them. 00:29:35.440 --> 00:29:42.460 align:middle line:90% 00:29:42.460 --> 00:29:45.467 align:middle line:84% So how long is each vector, the product 00:29:45.467 --> 00:29:47.300 align:middle line:84% of the two, and then the angle between them. 00:29:47.300 --> 00:29:50.380 align:middle line:84% Now, let's assume for simplicity that these links 00:29:50.380 --> 00:29:52.620 align:middle line:90% are roughly the same. 00:29:52.620 --> 00:29:54.340 align:middle line:90% They are just one unit length. 00:29:54.340 --> 00:29:55.660 align:middle line:90% Just roughly. 00:29:55.660 --> 00:30:02.680 align:middle line:84% So if you assume that, this thing, let's say, becomes 1, 00:30:02.680 --> 00:30:05.390 align:middle line:90% let's say. 00:30:05.390 --> 00:30:07.230 align:middle line:90% This thing becomes one. 00:30:07.230 --> 00:30:09.900 align:middle line:90% So all the action is here. 00:30:09.900 --> 00:30:12.482 align:middle line:90% 00:30:12.482 --> 00:30:14.307 align:middle line:90% So all the action is here. 00:30:14.307 --> 00:30:16.390 align:middle line:84% So basically, the dot product of these two vectors 00:30:16.390 --> 00:30:20.150 align:middle line:84% is really the cosine of the angle between them. 00:30:20.150 --> 00:30:22.670 align:middle line:84% So now the question is, if you have something 00:30:22.670 --> 00:30:29.790 align:middle line:84% like this, which are very close to each other, the cosine 00:30:29.790 --> 00:30:31.510 align:middle line:90% of a very small angle-- 00:30:31.510 --> 00:30:34.550 align:middle line:84% actually, the cosine of 0 is what? 00:30:34.550 --> 00:30:35.590 align:middle line:90% 1. 00:30:35.590 --> 00:30:38.210 align:middle line:84% So if the angle is really, really small, 00:30:38.210 --> 00:30:41.190 align:middle line:84% the cosine is going to be very close to 1, because 0 is 1. 00:30:41.190 --> 00:30:42.830 align:middle line:90% The cosine of 0 is 1. 00:30:42.830 --> 00:30:46.710 align:middle line:84% So this thing is going to be pretty close to 1. 00:30:46.710 --> 00:30:49.070 align:middle line:84% If you have a cosine of two vectors that 00:30:49.070 --> 00:30:52.870 align:middle line:84% are like this, 90 degrees apart, what is the cosine? 00:30:52.870 --> 00:30:53.590 align:middle line:90% 0. 00:30:53.590 --> 00:30:56.350 align:middle line:84% They are orthogonal, right, which maps to the English, 00:30:56.350 --> 00:30:58.190 align:middle line:90% orthogonal. 00:30:58.190 --> 00:31:00.790 align:middle line:90% So the cosine of that is 0. 00:31:00.790 --> 00:31:04.000 align:middle line:84% And then, if you have something like this where they're 00:31:04.000 --> 00:31:07.680 align:middle line:84% literally pointing in opposite direction, 00:31:07.680 --> 00:31:10.000 align:middle line:90% what is the cosine of that 180? 00:31:10.000 --> 00:31:11.320 align:middle line:90% Minus 1. 00:31:11.320 --> 00:31:12.100 align:middle line:90% So that's it. 00:31:12.100 --> 00:31:16.160 align:middle line:84% So if these two vectors are very close to each other, 00:31:16.160 --> 00:31:18.000 align:middle line:84% the cosine of the angle between them 00:31:18.000 --> 00:31:19.640 align:middle line:90% is going to be very close to 1. 00:31:19.640 --> 00:31:22.180 align:middle line:84% If they are really kind of unrelated, it's going to be 0. 00:31:22.180 --> 00:31:24.320 align:middle line:84% If they're anti-related it's going to be minus 1. 00:31:24.320 --> 00:31:28.880 align:middle line:84% So that's how dot products capture this notion of closeness 00:31:28.880 --> 00:31:29.900 align:middle line:90% or relatedness. 00:31:29.900 --> 00:31:32.640 align:middle line:90% 00:31:32.640 --> 00:31:34.400 align:middle line:90% All right. 00:31:34.400 --> 00:31:36.400 align:middle line:90% Input. 00:31:36.400 --> 00:31:38.480 align:middle line:84% So we can use the dot product of these embeddings 00:31:38.480 --> 00:31:40.560 align:middle line:90% to capture relatedness. 00:31:40.560 --> 00:31:43.600 align:middle line:90% And so-- OK, iPad done. 00:31:43.600 --> 00:31:47.040 align:middle line:84% So now what we do is, now that we know that dot products can 00:31:47.040 --> 00:31:49.247 align:middle line:84% be used, we can't use them as is, 00:31:49.247 --> 00:31:51.080 align:middle line:84% because we need to do one more thing to make 00:31:51.080 --> 00:31:52.720 align:middle line:90% them proper weights. 00:31:52.720 --> 00:31:54.240 align:middle line:84% And what I mean by proper weights 00:31:54.240 --> 00:31:57.900 align:middle line:84% is that we want the weights to be, first of all, non-negative. 00:31:57.900 --> 00:31:59.660 align:middle line:90% And we want them to add up to 1. 00:31:59.660 --> 00:32:02.370 align:middle line:84% That's what a weighted average actually is going to mean. 00:32:02.370 --> 00:32:06.110 align:middle line:84% But these cosines could be negative, right? 00:32:06.110 --> 00:32:08.497 align:middle line:84% And so we need to now adjust them to make them proper so 00:32:08.497 --> 00:32:10.830 align:middle line:84% that every one of them is guaranteed to be non-negative, 00:32:10.830 --> 00:32:12.330 align:middle line:90% and they will add up to 1. 00:32:12.330 --> 00:32:15.570 align:middle line:84% When was the last time you had to take a bunch of numbers which 00:32:15.570 --> 00:32:18.330 align:middle line:84% could be anything, and then somehow make sure 00:32:18.330 --> 00:32:21.210 align:middle line:84% that they are going to be positive, non-negative, 00:32:21.210 --> 00:32:22.150 align:middle line:90% and they add up to 1? 00:32:22.150 --> 00:32:23.110 align:middle line:90% When was the last time? 00:32:23.110 --> 00:32:23.970 align:middle line:90% AUDIENCE: Softmax. 00:32:23.970 --> 00:32:24.990 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, softmax. 00:32:24.990 --> 00:32:25.490 align:middle line:90% Exactly. 00:32:25.490 --> 00:32:27.650 align:middle line:90% So we'll do the same trick. 00:32:27.650 --> 00:32:32.290 align:middle line:84% So what we'll simply do is we'll just exponentiate them. 00:32:32.290 --> 00:32:35.558 align:middle line:84% So this w1, w6, this angle bracket thing 00:32:35.558 --> 00:32:36.350 align:middle line:90% is the dot product. 00:32:36.350 --> 00:32:37.970 align:middle line:90% That's the notation I'm using. 00:32:37.970 --> 00:32:39.970 align:middle line:84% exp of that is just you exponentiate 00:32:39.970 --> 00:32:41.330 align:middle line:90% them, e raised to that. 00:32:41.330 --> 00:32:44.020 align:middle line:84% And once you exponentiate them, they all become non-negative. 00:32:44.020 --> 00:32:46.270 align:middle line:84% And then we just divide each by the sum of everything, 00:32:46.270 --> 00:32:48.562 align:middle line:84% so that the whole thing will become like a probability. 00:32:48.562 --> 00:32:50.210 align:middle line:90% It'll just add up to 1. 00:32:50.210 --> 00:32:51.265 align:middle line:90% Makes sense? 00:32:51.265 --> 00:32:52.890 align:middle line:84% So that's how we take arbitrary numbers 00:32:52.890 --> 00:32:54.098 align:middle line:90% and make them proper weights. 00:32:54.098 --> 00:32:56.770 align:middle line:90% 00:32:56.770 --> 00:32:57.390 align:middle line:90% All right. 00:32:57.390 --> 00:32:59.980 align:middle line:90% 00:32:59.980 --> 00:33:04.860 align:middle line:84% So to summarize, from embeddings to contextual embeddings, that's 00:33:04.860 --> 00:33:05.820 align:middle line:90% what we do. 00:33:05.820 --> 00:33:08.200 align:middle line:84% We take all the standalone embeddings. 00:33:08.200 --> 00:33:10.640 align:middle line:84% We calculate these weights using this formula. 00:33:10.640 --> 00:33:12.400 align:middle line:84% And then we just do the weighted average, 00:33:12.400 --> 00:33:14.817 align:middle line:84% and we arrive at the contextual embedding, and boom, done. 00:33:14.817 --> 00:33:17.810 align:middle line:90% 00:33:17.810 --> 00:33:20.580 align:middle line:84% And so by choosing weights in this manner, 00:33:20.580 --> 00:33:22.940 align:middle line:84% the embedding of a word gets dragged 00:33:22.940 --> 00:33:26.220 align:middle line:84% closer to the embeddings of the other words in proportion 00:33:26.220 --> 00:33:27.400 align:middle line:90% to how related they are. 00:33:27.400 --> 00:33:30.140 align:middle line:84% So just imagine for a second, in this case, "station" 00:33:30.140 --> 00:33:31.480 align:middle line:90% obviously has many contexts. 00:33:31.480 --> 00:33:33.355 align:middle line:84% But let's assume for a second that it'll only 00:33:33.355 --> 00:33:35.940 align:middle line:84% have the train context and the radio-station context. 00:33:35.940 --> 00:33:40.100 align:middle line:84% In the current context, "train" is closely tied to "station," 00:33:40.100 --> 00:33:43.940 align:middle line:84% and therefore exerts a strong pull on it. 00:33:43.940 --> 00:33:45.800 align:middle line:84% Now "radio" is also related to "station," 00:33:45.800 --> 00:33:48.540 align:middle line:84% but it doesn't appear in the word, in the sentence. 00:33:48.540 --> 00:33:50.210 align:middle line:84% So effectively it has a weight of 0. 00:33:50.210 --> 00:33:52.940 align:middle line:90% 00:33:52.940 --> 00:33:54.878 align:middle line:90% And that's the beauty of it. 00:33:54.878 --> 00:33:56.420 align:middle line:84% And please do not ask me things like, 00:33:56.420 --> 00:33:59.270 align:middle line:84% I was listening to a great song on the radio station, 00:33:59.270 --> 00:34:02.190 align:middle line:84% and the train pulled out of the station. 00:34:02.190 --> 00:34:04.330 align:middle line:84% Transformers can deal with stuff like that. 00:34:04.330 --> 00:34:07.630 align:middle line:84% But yeah, but you get the idea-- the main idea. 00:34:07.630 --> 00:34:11.550 align:middle line:84% So by moving "station" closer to "train" 00:34:11.550 --> 00:34:13.409 align:middle line:84% by paying more attention to "train," 00:34:13.409 --> 00:34:15.730 align:middle line:84% we are contextualizing the "station," the word, 00:34:15.730 --> 00:34:19.810 align:middle line:84% the embedding, to the context of trains, platforms, departures, 00:34:19.810 --> 00:34:20.969 align:middle line:90% tickets, and so on. 00:34:20.969 --> 00:34:25.590 align:middle line:84% It's like this portal into the whole train world. 00:34:25.590 --> 00:34:26.610 align:middle line:90% It's beautiful. 00:34:26.610 --> 00:34:28.210 align:middle line:84% The simple idea will get you there. 00:34:28.210 --> 00:34:31.416 align:middle line:90% 00:34:31.416 --> 00:34:36.750 align:middle line:84% So this, folks, is called self-attention. 00:34:36.750 --> 00:34:39.710 align:middle line:84% What we just described is called self-attention. 00:34:39.710 --> 00:34:43.656 align:middle line:84% And it's the key building block of transformers. 00:34:43.656 --> 00:34:47.170 align:middle line:84% And so to summarize, standalone embeddings come in, 00:34:47.170 --> 00:34:50.830 align:middle line:90% contextual embeddings go out. 00:34:50.830 --> 00:34:52.670 align:middle line:90% Any questions? 00:34:52.670 --> 00:34:54.608 align:middle line:90% Yeah. 00:34:54.608 --> 00:34:56.400 align:middle line:84% AUDIENCE: I'm still struggling a little bit 00:34:56.400 --> 00:34:58.740 align:middle line:90% with the intuition of the word. 00:34:58.740 --> 00:35:00.840 align:middle line:84% It's the weight of the word itself 00:35:00.840 --> 00:35:02.580 align:middle line:90% in its own contextual embedding. 00:35:02.580 --> 00:35:05.260 align:middle line:84% So the weight of "station" and the "station" embedding, 00:35:05.260 --> 00:35:06.940 align:middle line:90% how should I think about that. 00:35:06.940 --> 00:35:07.320 align:middle line:90% RAMA RAMAKRISHNAN: Think of it-- 00:35:07.320 --> 00:35:08.040 align:middle line:84% AUDIENCE: Seem intuitively like it 00:35:08.040 --> 00:35:10.420 align:middle line:84% would be high for all contextual embeddings, 00:35:10.420 --> 00:35:12.160 align:middle line:84% but I assume that's not the case. 00:35:12.160 --> 00:35:12.900 align:middle line:84% RAMA RAMAKRISHNAN: It'll be high. 00:35:12.900 --> 00:35:14.600 align:middle line:84% It'll be typically be a high number, 00:35:14.600 --> 00:35:16.657 align:middle line:84% because the cosine of the vector to itself 00:35:16.657 --> 00:35:18.740 align:middle line:84% is going to be very-- the cosine is going to be 1. 00:35:18.740 --> 00:35:20.400 align:middle line:90% So it's going to be pretty high. 00:35:20.400 --> 00:35:23.260 align:middle line:84% But there's no guarantee it's going to be the highest, right? 00:35:23.260 --> 00:35:26.540 align:middle line:84% Because they are not actually-- the length doesn't have to be 1. 00:35:26.540 --> 00:35:28.880 align:middle line:84% They could be-- we try to keep them smallish, 00:35:28.880 --> 00:35:30.585 align:middle line:90% but they don't have to be. 00:35:30.585 --> 00:35:31.960 align:middle line:84% So the way I would think of it is 00:35:31.960 --> 00:35:34.840 align:middle line:84% imagine that you take an average of everything else first, 00:35:34.840 --> 00:35:38.123 align:middle line:84% and then you average it with the old embedding. 00:35:38.123 --> 00:35:40.040 align:middle line:84% Effectively, it's the same as just calculating 00:35:40.040 --> 00:35:42.165 align:middle line:84% the different weights and averaging the whole thing 00:35:42.165 --> 00:35:42.760 align:middle line:90% together. 00:35:42.760 --> 00:35:43.400 align:middle line:90% Sure. 00:35:43.400 --> 00:35:44.304 align:middle line:90% Yeah. 00:35:44.304 --> 00:35:46.520 align:middle line:84% AUDIENCE: A little confused [INAUDIBLE] 00:35:46.520 --> 00:35:49.860 align:middle line:84% you need the same number of inputs and outputs. 00:35:49.860 --> 00:35:51.520 align:middle line:84% But is this the reason why, because you 00:35:51.520 --> 00:35:56.730 align:middle line:84% need the contextual embedding, even if it's like a other word, 00:35:56.730 --> 00:35:58.925 align:middle line:84% and it's related definitely [INAUDIBLE]. 00:35:58.925 --> 00:36:00.050 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:36:00.050 --> 00:36:00.590 align:middle line:90% Correct. 00:36:00.590 --> 00:36:01.090 align:middle line:90% Exactly. 00:36:01.090 --> 00:36:01.750 align:middle line:90% Exactly. 00:36:01.750 --> 00:36:04.850 align:middle line:84% And the other thing to remember is that by getting-- 00:36:04.850 --> 00:36:09.050 align:middle line:84% by keeping the inputs the size of the input cardinality intact 00:36:09.050 --> 00:36:11.123 align:middle line:84% as you move through the transformer stack, 00:36:11.123 --> 00:36:12.790 align:middle line:84% when you finally come out the other end, 00:36:12.790 --> 00:36:14.890 align:middle line:90% there is no loss of information. 00:36:14.890 --> 00:36:16.997 align:middle line:84% And at the very end, you can choose to aggregate, 00:36:16.997 --> 00:36:18.830 align:middle line:84% simplify, summarize, and so on and so forth. 00:36:18.830 --> 00:36:20.913 align:middle line:84% It preserves your optionality as long as possible. 00:36:20.913 --> 00:36:23.560 align:middle line:90% 00:36:23.560 --> 00:36:32.483 align:middle line:84% AUDIENCE: Do you know how long [INAUDIBLE] going to take? 00:36:32.483 --> 00:36:35.150 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so what we do is the sentence comes in. 00:36:35.150 --> 00:36:36.890 align:middle line:84% There is a whole notion of something called a context 00:36:36.890 --> 00:36:37.550 align:middle line:90% window-- 00:36:37.550 --> 00:36:40.430 align:middle line:84% or what is the maximum length of the sentence we'll handle. 00:36:40.430 --> 00:36:42.330 align:middle line:84% And that's a parameter you can set. 00:36:42.330 --> 00:36:46.690 align:middle line:84% And we'll come to that when you actually look at the Colab. 00:36:46.690 --> 00:36:48.250 align:middle line:84% Was there a question in the middle? 00:36:48.250 --> 00:36:48.750 align:middle line:90% No. 00:36:48.750 --> 00:36:49.650 align:middle line:90% OK. 00:36:49.650 --> 00:36:50.150 align:middle line:90% All right. 00:36:50.150 --> 00:36:51.350 align:middle line:90% So that is self-attention. 00:36:51.350 --> 00:36:54.180 align:middle line:90% 00:36:54.180 --> 00:36:58.100 align:middle line:84% And now, because that felt too easy, 00:36:58.100 --> 00:37:02.140 align:middle line:84% we're going to do a little tweak called multi-head attention. 00:37:02.140 --> 00:37:06.060 align:middle line:84% So this is the self-attention we just saw. 00:37:06.060 --> 00:37:09.060 align:middle line:84% What we can do is we can be like, you know what? 00:37:09.060 --> 00:37:10.640 align:middle line:84% Why can't we have more than this? 00:37:10.640 --> 00:37:12.420 align:middle line:84% Why can't we have more than one of these? 00:37:12.420 --> 00:37:15.140 align:middle line:84% So this is called an attention head-- self-attention head. 00:37:15.140 --> 00:37:19.060 align:middle line:84% We'll have multiple self-attention heads. 00:37:19.060 --> 00:37:21.740 align:middle line:84% And I'll come back to the top thing in a second. 00:37:21.740 --> 00:37:23.460 align:middle line:84% But the question is, why should we have 00:37:23.460 --> 00:37:25.442 align:middle line:90% multiple self-attention heads? 00:37:25.442 --> 00:37:26.900 align:middle line:84% Because a particular attention head 00:37:26.900 --> 00:37:28.317 align:middle line:84% is going to pick up some patterns. 00:37:28.317 --> 00:37:31.140 align:middle line:84% The reason is because it will help 00:37:31.140 --> 00:37:33.540 align:middle line:84% us attend to the multiple patterns that may be 00:37:33.540 --> 00:37:35.740 align:middle line:90% present in a single sentence. 00:37:35.740 --> 00:37:37.277 align:middle line:84% So far, when I've been explaining, 00:37:37.277 --> 00:37:39.860 align:middle line:84% I've basically been looking at what the meaning of these words 00:37:39.860 --> 00:37:41.940 align:middle line:84% are, just the meaning of these words. 00:37:41.940 --> 00:37:45.320 align:middle line:84% But in any complicated sentence, you have to worry about grammar. 00:37:45.320 --> 00:37:46.760 align:middle line:90% You have to worry about tense. 00:37:46.760 --> 00:37:48.420 align:middle line:90% You have to worry about tone. 00:37:48.420 --> 00:37:51.788 align:middle line:84% You have to worry about facts versus opinions. 00:37:51.788 --> 00:37:53.830 align:middle line:84% There could be any number of complicated patterns 00:37:53.830 --> 00:37:55.590 align:middle line:84% that are sitting in a simple sentence. 00:37:55.590 --> 00:37:59.250 align:middle line:84% Which means, well, there is just not one way to pay attention. 00:37:59.250 --> 00:38:02.270 align:middle line:84% There could be many ways of paying attention. 00:38:02.270 --> 00:38:04.870 align:middle line:84% There could be many needs to pay attention. 00:38:04.870 --> 00:38:09.350 align:middle line:84% Which means that let's have many of these attention heads, 00:38:09.350 --> 00:38:11.370 align:middle line:84% and each one could be learning something else. 00:38:11.370 --> 00:38:13.270 align:middle line:84% It's exactly like having lots of filters 00:38:13.270 --> 00:38:15.275 align:middle line:90% in a convolutional network. 00:38:15.275 --> 00:38:17.150 align:middle line:84% One filter might learn a line, another filter 00:38:17.150 --> 00:38:18.890 align:middle line:84% might learn a curve, and so on and so forth. 00:38:18.890 --> 00:38:20.510 align:middle line:84% And we don't want to decide a priori, 00:38:20.510 --> 00:38:22.290 align:middle line:84% oh, you're going to learn a line, right? 00:38:22.290 --> 00:38:23.810 align:middle line:84% Similarly, here, we're not telling any of these things 00:38:23.810 --> 00:38:24.893 align:middle line:90% on what you have to learn. 00:38:24.893 --> 00:38:27.430 align:middle line:84% They just have to learn based on the training process. 00:38:27.430 --> 00:38:29.030 align:middle line:90% So what we do is-- 00:38:29.030 --> 00:38:30.990 align:middle line:84% so actually, this is an example where this, 00:38:30.990 --> 00:38:32.750 align:middle line:84% from the original transformer paper, 00:38:32.750 --> 00:38:36.010 align:middle line:84% where this sentence is "The lawyer will--" sorry. 00:38:36.010 --> 00:38:39.230 align:middle line:84% "The law will never be perfect, but its application 00:38:39.230 --> 00:38:40.710 align:middle line:90% should be just. 00:38:40.710 --> 00:38:44.470 align:middle line:84% This is what we are missing, in my opinion." 00:38:44.470 --> 00:38:45.900 align:middle line:90% A complicated sentence, right? 00:38:45.900 --> 00:38:48.400 align:middle line:84% So the first one, attention head-- 00:38:48.400 --> 00:38:50.680 align:middle line:84% actually, this is the pattern of things. 00:38:50.680 --> 00:38:53.280 align:middle line:84% So for example, the word "perfect" here. 00:38:53.280 --> 00:38:57.520 align:middle line:84% the contextual embedding of the word "perfect" 00:38:57.520 --> 00:39:02.732 align:middle line:84% draws upon heavily from the word "law" in this example. 00:39:02.732 --> 00:39:04.602 align:middle line:84% If you look at another attention head, 00:39:04.602 --> 00:39:06.560 align:middle line:84% the contextual embedding for the word "perfect" 00:39:06.560 --> 00:39:11.040 align:middle line:84% is actually drawing heavily from just perfect and nothing else. 00:39:11.040 --> 00:39:13.440 align:middle line:84% And if you look at other words, the patterns 00:39:13.440 --> 00:39:17.160 align:middle line:84% are subtly different of what it's paying attention to. 00:39:17.160 --> 00:39:19.160 align:middle line:84% So these are two different attention heads. 00:39:19.160 --> 00:39:22.120 align:middle line:84% And they are learning different kinds of attentions. 00:39:22.120 --> 00:39:26.320 align:middle line:84% In reality, trying to make sense of why they pay attention 00:39:26.320 --> 00:39:29.148 align:middle line:84% to the way they do, it's usually quite difficult 00:39:29.148 --> 00:39:29.940 align:middle line:90% to figure that out. 00:39:29.940 --> 00:39:31.420 align:middle line:90% You can't actually interpret it. 00:39:31.420 --> 00:39:33.420 align:middle line:84% But when you have lots of attention heads, 00:39:33.420 --> 00:39:35.400 align:middle line:84% the performance on the task that you care about 00:39:35.400 --> 00:39:38.000 align:middle line:90% gets really much better. 00:39:38.000 --> 00:39:40.000 align:middle line:84% And then you say, OK, I can use that. 00:39:40.000 --> 00:39:40.920 align:middle line:90% Yeah. 00:39:40.920 --> 00:39:42.860 align:middle line:84% AUDIENCE: Not to oversimplify anything. 00:39:42.860 --> 00:39:45.880 align:middle line:84% Is this -- is the idea behind this [INAUDIBLE]? 00:39:45.880 --> 00:39:50.525 align:middle line:90% 00:39:50.525 --> 00:39:51.650 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:39:51.650 --> 00:39:52.550 align:middle line:90% Same logic. 00:39:52.550 --> 00:39:53.660 align:middle line:90% Same logic. 00:39:53.660 --> 00:39:54.343 align:middle line:90% Yeah. 00:39:54.343 --> 00:39:56.590 align:middle line:84% AUDIENCE: [INAUDIBLE], with that filter, 00:39:56.590 --> 00:39:59.850 align:middle line:84% we had designed a particular filter for a particular task 00:39:59.850 --> 00:40:02.130 align:middle line:84% in terms of 1's and 0's on that line. 00:40:02.130 --> 00:40:04.010 align:middle line:84% But here, like the attention head, 00:40:04.010 --> 00:40:06.210 align:middle line:84% how do we differentiate-- how do we create 00:40:06.210 --> 00:40:08.890 align:middle line:84% an instance for each different kind of scenario 00:40:08.890 --> 00:40:10.370 align:middle line:84% or different kind of parameters we 00:40:10.370 --> 00:40:13.490 align:middle line:84% want to all kind of different kind of pattern [INAUDIBLE]. 00:40:13.490 --> 00:40:15.782 align:middle line:84% RAMA RAMAKRISHNAN: Actually, in the convolutional case, 00:40:15.782 --> 00:40:17.850 align:middle line:84% the 1's and 0's I had were just example numbers 00:40:17.850 --> 00:40:20.090 align:middle line:84% to show that that particular filter could 00:40:20.090 --> 00:40:22.130 align:middle line:84% detect a vertical line or a horizontal line. 00:40:22.130 --> 00:40:23.783 align:middle line:84% You will recall that when we actually 00:40:23.783 --> 00:40:25.450 align:middle line:84% train a convolution network, we actually 00:40:25.450 --> 00:40:26.950 align:middle line:90% don't specify the numbers. 00:40:26.950 --> 00:40:29.010 align:middle line:84% We start with the random initialized weights, 00:40:29.010 --> 00:40:32.170 align:middle line:84% and then we let backpropagation figure it out. 00:40:32.170 --> 00:40:34.750 align:middle line:84% Similarly, here, we don't decide any of these things. 00:40:34.750 --> 00:40:37.850 align:middle line:84% We just let backprop figure it out. 00:40:37.850 --> 00:40:39.907 align:middle line:84% And now the question of what are the weights 00:40:39.907 --> 00:40:41.490 align:middle line:84% that are actually going to be learned, 00:40:41.490 --> 00:40:44.220 align:middle line:90% we'll come to that in a bit. 00:40:44.220 --> 00:40:44.720 align:middle line:90% Yeah. 00:40:44.720 --> 00:40:47.940 align:middle line:90% 00:40:47.940 --> 00:40:52.020 align:middle line:84% AUDIENCE: I was wondering, how come we have different attention 00:40:52.020 --> 00:40:55.980 align:middle line:84% heads, even though it seems like they're only function of the dot 00:40:55.980 --> 00:40:57.780 align:middle line:84% product, and we have the same dot product 00:40:57.780 --> 00:40:58.903 align:middle line:90% for the same embeddings. 00:40:58.903 --> 00:41:00.320 align:middle line:84% RAMA RAMAKRISHNAN: Great question. 00:41:00.320 --> 00:41:01.120 align:middle line:90% Great question. 00:41:01.120 --> 00:41:03.580 align:middle line:84% And I literally have a note in my slide saying 00:41:03.580 --> 00:41:05.940 align:middle line:84% if a student asks this good question, tell 00:41:05.940 --> 00:41:08.500 align:middle line:90% them to wait till Wednesday. 00:41:08.500 --> 00:41:09.840 align:middle line:90% So, great question. 00:41:09.840 --> 00:41:11.580 align:middle line:84% And we'll come back to that on Wednesday 00:41:11.580 --> 00:41:13.450 align:middle line:84% and spend a fair amount of time on it. 00:41:13.450 --> 00:41:17.860 align:middle line:84% So the point that's being made here is that-- 00:41:17.860 --> 00:41:19.780 align:middle line:90% oops. 00:41:19.780 --> 00:41:24.220 align:middle line:84% When we look at self-attention, the embeddings came in, 00:41:24.220 --> 00:41:25.860 align:middle line:84% and we did all these dot products, 00:41:25.860 --> 00:41:28.300 align:middle line:84% and the contextual things popped out the other end. 00:41:28.300 --> 00:41:30.300 align:middle line:84% Note that inside the self-attention box 00:41:30.300 --> 00:41:32.700 align:middle line:90% there are no parameters. 00:41:32.700 --> 00:41:34.220 align:middle line:90% There are no parameters. 00:41:34.220 --> 00:41:36.780 align:middle line:84% So the question that is being raised here 00:41:36.780 --> 00:41:39.980 align:middle line:84% is that, so what are we learning, really, if there is 00:41:39.980 --> 00:41:41.657 align:middle line:90% nothing inside to be learned? 00:41:41.657 --> 00:41:43.490 align:middle line:84% If there are no parameters, no coefficients, 00:41:43.490 --> 00:41:44.130 align:middle line:90% what are we learning? 00:41:44.130 --> 00:41:45.090 align:middle line:90% That's the question. 00:41:45.090 --> 00:41:48.435 align:middle line:84% And by extension, if we have two of these, and neither of them 00:41:48.435 --> 00:41:50.060 align:middle line:84% is learning anything, what's the point? 00:41:50.060 --> 00:41:52.870 align:middle line:90% 00:41:52.870 --> 00:41:56.210 align:middle line:84% Sadly, you have to wait till Wednesday. 00:41:56.210 --> 00:41:58.010 align:middle line:84% But we have a great answer to the question, 00:41:58.010 --> 00:41:59.730 align:middle line:90% so it'll be worth it. 00:41:59.730 --> 00:42:03.310 align:middle line:84% And if you can't stand the suspense, read the book. 00:42:03.310 --> 00:42:06.790 align:middle line:84% All right, so that's why we need multiple heads. 00:42:06.790 --> 00:42:09.590 align:middle line:84% And now to come back to this, so what we do is, 00:42:09.590 --> 00:42:12.410 align:middle line:84% it goes through this head, and you get these w1's. 00:42:12.410 --> 00:42:15.830 align:middle line:84% And it goes through here, and we get another set of w's. 00:42:15.830 --> 00:42:19.670 align:middle line:84% Then, what we do at the very end is we concatenate them. 00:42:19.670 --> 00:42:20.170 align:middle line:90% OK. 00:42:20.170 --> 00:42:21.470 align:middle line:90% We concatenate them. 00:42:21.470 --> 00:42:22.717 align:middle line:90% And we do a projection. 00:42:22.717 --> 00:42:24.050 align:middle line:90% And this is what I mean by that. 00:42:24.050 --> 00:42:29.310 align:middle line:90% 00:42:29.310 --> 00:42:33.310 align:middle line:84% So we have-- this is one self-attention head-- 00:42:33.310 --> 00:42:35.950 align:middle line:90% self-attention 1. 00:42:35.950 --> 00:42:38.950 align:middle line:90% This is self-attention 2. 00:42:38.950 --> 00:42:41.880 align:middle line:90% And let's say that. 00:42:41.880 --> 00:42:44.680 align:middle line:90% w1 hat comes out. 00:42:44.680 --> 00:42:48.140 align:middle line:84% And I'm just going to call it Z1 for the same thing 00:42:48.140 --> 00:42:49.960 align:middle line:90% so that there's no name clash. 00:42:49.960 --> 00:42:53.040 align:middle line:84% And w2, w6, all of them are coming, right? 00:42:53.040 --> 00:42:55.360 align:middle line:90% Let's focus on w1 and z1. 00:42:55.360 --> 00:42:59.360 align:middle line:84% w1 and z1 are both contextual embeddings for the same word, 00:42:59.360 --> 00:43:01.560 align:middle line:90% for the first word, word 1. 00:43:01.560 --> 00:43:05.360 align:middle line:84% And so what we do is, let's say this w1, 00:43:05.360 --> 00:43:07.960 align:middle line:84% let's say this vector is like this. 00:43:07.960 --> 00:43:12.400 align:middle line:84% And let's say that this vector is like this. 00:43:12.400 --> 00:43:14.400 align:middle line:84% What I mean when I say concatenated 00:43:14.400 --> 00:43:19.820 align:middle line:84% here is we literally take this word here, this embedding here. 00:43:19.820 --> 00:43:23.120 align:middle line:90% Then we take this thing here. 00:43:23.120 --> 00:43:25.140 align:middle line:84% And we just make it a long vector. 00:43:25.140 --> 00:43:26.340 align:middle line:90% We concatenate it. 00:43:26.340 --> 00:43:31.020 align:middle line:84% But now, this vector has become twice as long, right? 00:43:31.020 --> 00:43:33.680 align:middle line:84% But remember, we always want to preserve 00:43:33.680 --> 00:43:36.800 align:middle line:84% the number of inputs we have and the lengths of these vectors 00:43:36.800 --> 00:43:38.330 align:middle line:90% everywhere as we go along. 00:43:38.330 --> 00:43:41.170 align:middle line:84% So what we do is, at this point, we 00:43:41.170 --> 00:43:44.810 align:middle line:84% run it through a single dense layer, 00:43:44.810 --> 00:43:47.290 align:middle line:84% which will take this thing and make it back 00:43:47.290 --> 00:43:50.010 align:middle line:84% into the same small shape as before. 00:43:50.010 --> 00:43:51.080 align:middle line:90% So this is a dense layer. 00:43:51.080 --> 00:43:54.330 align:middle line:90% 00:43:54.330 --> 00:43:55.030 align:middle line:90% That's it. 00:43:55.030 --> 00:43:58.570 align:middle line:84% So this vector comes in, and it gets compressed back 00:43:58.570 --> 00:44:01.210 align:middle line:84% to the original shape that came out of here. 00:44:01.210 --> 00:44:04.930 align:middle line:84% So you could have 20 of these attention heads, 00:44:04.930 --> 00:44:06.910 align:middle line:84% and the concatenating will be 20 times longer. 00:44:06.910 --> 00:44:08.243 align:middle line:90% And then you just project, boom. 00:44:08.243 --> 00:44:12.890 align:middle line:84% One dense layer comes back to the original shape. 00:44:12.890 --> 00:44:16.290 align:middle line:90% So that is the projection step. 00:44:16.290 --> 00:44:20.450 align:middle line:84% And that's what I mean here when I say concatenate and project. 00:44:20.450 --> 00:44:22.710 align:middle line:84% So at this point, what we have is things come in. 00:44:22.710 --> 00:44:25.230 align:middle line:84% We contextualize them using these different attention heads. 00:44:25.230 --> 00:44:27.610 align:middle line:84% And when they come out of the attention heads, 00:44:27.610 --> 00:44:28.630 align:middle line:90% we take them all. 00:44:28.630 --> 00:44:31.250 align:middle line:84% We just concatenate them and then compress them back 00:44:31.250 --> 00:44:33.370 align:middle line:84% to the same original starting shape. 00:44:33.370 --> 00:44:37.240 align:middle line:84% If these vectors are 100 units long or 100 dimension long, 00:44:37.240 --> 00:44:39.660 align:middle line:90% whatever comes out is 100 still. 00:44:39.660 --> 00:44:42.835 align:middle line:84% And to preserving this size as we go along 00:44:42.835 --> 00:44:44.460 align:middle line:84% is very important for reasons that will 00:44:44.460 --> 00:44:47.460 align:middle line:90% become apparent a bit later. 00:44:47.460 --> 00:44:50.340 align:middle line:84% So that is the multi-attention thing. 00:44:50.340 --> 00:44:54.300 align:middle line:84% Now, a final tweak for today is that we 00:44:54.300 --> 00:44:58.680 align:middle line:84% will inject some non-linearity with some dense layers, 00:44:58.680 --> 00:45:00.160 align:middle line:84% dense ReLU layers at the very end. 00:45:00.160 --> 00:45:02.680 align:middle line:84% So we went through a bunch of attention heads. 00:45:02.680 --> 00:45:05.220 align:middle line:84% We came up with a bunch of contextual embeddings now. 00:45:05.220 --> 00:45:07.940 align:middle line:84% So at this point so far, there are 00:45:07.940 --> 00:45:11.860 align:middle line:84% no-- since there are no parameters inside these boxes, 00:45:11.860 --> 00:45:13.940 align:middle line:84% and there are some parameters here, 00:45:13.940 --> 00:45:15.408 align:middle line:84% we need to do some non-linearity. 00:45:15.408 --> 00:45:17.700 align:middle line:84% So far, there has been nothing that's nonlinear so far. 00:45:17.700 --> 00:45:21.520 align:middle line:84% So here we actually send it through one or more ReLUs. 00:45:21.520 --> 00:45:24.312 align:middle line:84% Typically, they just use one ReLU. 00:45:24.312 --> 00:45:25.800 align:middle line:90% And what I mean by that-- 00:45:25.800 --> 00:45:34.140 align:middle line:90% 00:45:34.140 --> 00:45:34.880 align:middle line:90% oops, sorry. 00:45:34.880 --> 00:45:37.950 align:middle line:90% 00:45:37.950 --> 00:45:39.850 align:middle line:90% So this is what we had here. 00:45:39.850 --> 00:45:47.440 align:middle line:84% And then we take it in and then run it through-- 00:45:47.440 --> 00:45:50.190 align:middle line:90% 00:45:50.190 --> 00:45:58.670 align:middle line:84% actually, we typically run it through a ReLU. 00:45:58.670 --> 00:46:01.470 align:middle line:90% This is a nice ReLU. 00:46:01.470 --> 00:46:03.900 align:middle line:84% And then the rule of thumb, as you will see, 00:46:03.900 --> 00:46:06.710 align:middle line:84% if, let's say, this vector is, say, 100 dimensions long, 00:46:06.710 --> 00:46:10.762 align:middle line:84% they typically will choose a ReLU which is about 400 wide. 00:46:10.762 --> 00:46:16.710 align:middle line:84% And then it just gets projected out again back to 100. 00:46:16.710 --> 00:46:20.230 align:middle line:90% So this is just a simple-- 00:46:20.230 --> 00:46:23.110 align:middle line:84% the input comes in, goes through a single hidden layer with four 00:46:23.110 --> 00:46:24.770 align:middle line:90% times as many as here. 00:46:24.770 --> 00:46:29.830 align:middle line:84% And then it project another dense layer to 100 again. 00:46:29.830 --> 00:46:32.710 align:middle line:84% And since there are ReLUs here, we 00:46:32.710 --> 00:46:35.880 align:middle line:84% have injected some non-linearity into the processing. 00:46:35.880 --> 00:46:39.280 align:middle line:84% Now, a lot of this stuff, when it came out, 00:46:39.280 --> 00:46:41.980 align:middle line:90% felt very ad hoc, right? 00:46:41.980 --> 00:46:45.600 align:middle line:84% It didn't come from some deep theoretical motivations, 00:46:45.600 --> 00:46:47.640 align:middle line:84% but people had strong intuitions as to why 00:46:47.640 --> 00:46:49.040 align:middle line:90% these things were helpful. 00:46:49.040 --> 00:46:51.900 align:middle line:84% And as it turns out, since the transformer came out, 00:46:51.900 --> 00:46:55.417 align:middle line:84% people have tried to optimize every aspect of this thing. 00:46:55.417 --> 00:46:57.000 align:middle line:84% It's actually pretty difficult to beat 00:46:57.000 --> 00:46:58.923 align:middle line:90% the starting architecture. 00:46:58.923 --> 00:47:00.840 align:middle line:84% Improvements have been made, but it's actually 00:47:00.840 --> 00:47:02.760 align:middle line:90% a very robust architecture. 00:47:02.760 --> 00:47:05.360 align:middle line:90% So that's what's going on here. 00:47:05.360 --> 00:47:08.960 align:middle line:84% And then, when we come out of this thing, 00:47:08.960 --> 00:47:10.920 align:middle line:84% this is what we have-- the story so far. 00:47:10.920 --> 00:47:13.540 align:middle line:84% We start with random standalone embeddings. 00:47:13.540 --> 00:47:15.255 align:middle line:90% These could be glove embeddings. 00:47:15.255 --> 00:47:16.380 align:middle line:90% It could be random weights. 00:47:16.380 --> 00:47:17.320 align:middle line:90% It doesn't matter. 00:47:17.320 --> 00:47:19.860 align:middle line:84% It goes through a bunch of self-attention heads. 00:47:19.860 --> 00:47:23.840 align:middle line:84% We concatenate it when it comes out the other end, 00:47:23.840 --> 00:47:25.960 align:middle line:84% concatenate it when it comes out the other end. 00:47:25.960 --> 00:47:28.700 align:middle line:84% And then we project it back to the same size as before. 00:47:28.700 --> 00:47:32.870 align:middle line:84% Then we run it through a ReLU followed by a linear layer. 00:47:32.870 --> 00:47:34.150 align:middle line:90% And we get these things again. 00:47:34.150 --> 00:47:37.110 align:middle line:84% So in this whole process, if six things came in, 00:47:37.110 --> 00:47:38.410 align:middle line:90% six things will come out. 00:47:38.410 --> 00:47:41.970 align:middle line:84% And if those six things that came in 00:47:41.970 --> 00:47:43.730 align:middle line:84% were standalone embedding vectors 00:47:43.730 --> 00:47:46.802 align:middle line:84% of 100 dimensions, what comes out is also 100 dimensions. 00:47:46.802 --> 00:47:49.010 align:middle line:84% So in that sense, you could think of this whole thing 00:47:49.010 --> 00:47:51.830 align:middle line:84% as a black box in which whatever you send in, 00:47:51.830 --> 00:47:54.327 align:middle line:84% the same number of things will come out of the same length. 00:47:54.327 --> 00:47:56.410 align:middle line:84% The numbers will be different, because they've now 00:47:56.410 --> 00:47:58.530 align:middle line:90% been heavily contextualized. 00:47:58.530 --> 00:48:00.560 align:middle line:84% The numbers are much smarter, in other words. 00:48:00.560 --> 00:48:03.405 align:middle line:90% 00:48:03.405 --> 00:48:05.530 align:middle line:84% So far, what we have seen is that we have satisfied 00:48:05.530 --> 00:48:06.890 align:middle line:90% two of the three requirements. 00:48:06.890 --> 00:48:08.610 align:middle line:84% We have taken the context of each word 00:48:08.610 --> 00:48:11.010 align:middle line:84% into account by using these dot products 00:48:11.010 --> 00:48:12.490 align:middle line:84% in the self-attention layer, and we 00:48:12.490 --> 00:48:14.990 align:middle line:84% can generate an output that is the same length as the input. 00:48:14.990 --> 00:48:16.810 align:middle line:84% But we have ignored the fact that-- we have 00:48:16.810 --> 00:48:20.310 align:middle line:90% ignored word order completely. 00:48:20.310 --> 00:48:23.410 align:middle line:84% Because whether I had said "the train slowly left the station," 00:48:23.410 --> 00:48:26.810 align:middle line:84% or I had said that, "The station slowly left the train," 00:48:26.810 --> 00:48:30.780 align:middle line:84% this thing won't know the difference, 00:48:30.780 --> 00:48:34.560 align:middle line:84% because dot products function on sets, not on sequences. 00:48:34.560 --> 00:48:36.620 align:middle line:90% They function on sets. 00:48:36.620 --> 00:48:38.740 align:middle line:84% You should convince yourself of this. 00:48:38.740 --> 00:48:40.620 align:middle line:84% Regardless of the order, the dot reduction 00:48:40.620 --> 00:48:43.210 align:middle line:84% doesn't change anything, because we are doing every pair. 00:48:43.210 --> 00:48:46.405 align:middle line:90% 00:48:46.405 --> 00:48:48.780 align:middle line:84% So the question is, how do we take the order of the words 00:48:48.780 --> 00:48:50.710 align:middle line:90% into account? 00:48:50.710 --> 00:48:52.460 align:middle line:84% As I was saying, we can scramble the order 00:48:52.460 --> 00:48:53.660 align:middle line:84% of the words in a sentence, and we'll 00:48:53.660 --> 00:48:55.827 align:middle line:84% get the exact same contextual embeddings at the end. 00:48:55.827 --> 00:48:58.580 align:middle line:84% So by the way, if you're working on a problem in which the order 00:48:58.580 --> 00:49:01.620 align:middle line:84% doesn't matter, then you could stop right now 00:49:01.620 --> 00:49:03.625 align:middle line:90% and use the transformer. 00:49:03.625 --> 00:49:05.500 align:middle line:84% And there are many problems that are actually 00:49:05.500 --> 00:49:07.980 align:middle line:84% in that category, where the order doesn't matter. 00:49:07.980 --> 00:49:11.300 align:middle line:84% So if you take traditional structured data, or tabular 00:49:11.300 --> 00:49:13.140 align:middle line:90% data-- 00:49:13.140 --> 00:49:15.270 align:middle line:84% blood pressure, cholesterol level, boom, boom boom. 00:49:15.270 --> 00:49:16.520 align:middle line:90% Does it predict heart disease? 00:49:16.520 --> 00:49:18.240 align:middle line:84% Well, there is no order in that thing. 00:49:18.240 --> 00:49:22.100 align:middle line:84% You can use a transformer as is, without doing anything more. 00:49:22.100 --> 00:49:25.380 align:middle line:84% So transformers work for both sets and sequences 00:49:25.380 --> 00:49:27.180 align:middle line:90% where order matters. 00:49:27.180 --> 00:49:27.870 align:middle line:90% OK. 00:49:27.870 --> 00:49:33.150 align:middle line:84% So the fix for this is something called the positional encoding. 00:49:33.150 --> 00:49:34.455 align:middle line:90% So what we do is very simple. 00:49:34.455 --> 00:49:35.830 align:middle line:84% By the way, there are many things 00:49:35.830 --> 00:49:40.830 align:middle line:84% that have been invented to tell transformers-- 00:49:40.830 --> 00:49:42.750 align:middle line:84% to give transformers some information 00:49:42.750 --> 00:49:45.710 align:middle line:84% about the order of each of the things that are coming in. 00:49:45.710 --> 00:49:47.928 align:middle line:84% I'm going to go with something called the-- 00:49:47.928 --> 00:49:49.470 align:middle line:84% simplest possible way, which actually 00:49:49.470 --> 00:49:51.030 align:middle line:90% works pretty well in practice. 00:49:51.030 --> 00:49:54.990 align:middle line:84% So what we do is, for each position, 00:49:54.990 --> 00:49:56.637 align:middle line:84% each possible position in the input, 00:49:56.637 --> 00:49:58.470 align:middle line:84% starting from the first position all the way 00:49:58.470 --> 00:50:00.910 align:middle line:84% through the last position, we imagine 00:50:00.910 --> 00:50:05.590 align:middle line:84% that that position itself is a categorical variable. 00:50:05.590 --> 00:50:08.330 align:middle line:84% If a sentence can only be 30 words long, 00:50:08.330 --> 00:50:11.270 align:middle line:84% let's say, we say that, hey, the position of each word 00:50:11.270 --> 00:50:14.550 align:middle line:90% is a number between 0 and 29. 00:50:14.550 --> 00:50:18.030 align:middle line:84% And so we can just think of it as a categorical variable. 00:50:18.030 --> 00:50:19.970 align:middle line:84% And because a categorical variable, 00:50:19.970 --> 00:50:22.150 align:middle line:84% we could just imagine an embedding for that, 00:50:22.150 --> 00:50:23.443 align:middle line:90% for each potential value. 00:50:23.443 --> 00:50:25.110 align:middle line:84% So it will become clear in just a moment 00:50:25.110 --> 00:50:26.960 align:middle line:84% because I have a numerical example. 00:50:26.960 --> 00:50:28.760 align:middle line:84% And so what we do is we will just 00:50:28.760 --> 00:50:30.920 align:middle line:84% take that standalone embedding, and then we'll 00:50:30.920 --> 00:50:33.040 align:middle line:84% take this position embedding, which 00:50:33.040 --> 00:50:35.207 align:middle line:84% represents the position of the word in the sentence. 00:50:35.207 --> 00:50:37.440 align:middle line:90% And we'll just add them up. 00:50:37.440 --> 00:50:39.680 align:middle line:90% Yeah. 00:50:39.680 --> 00:50:42.360 align:middle line:84% AUDIENCE: So if, in the initial sentence, it says, 00:50:42.360 --> 00:50:43.820 align:middle line:90% I have made a mistake. 00:50:43.820 --> 00:50:48.040 align:middle line:84% So I just write it as, "The train slowly a station." 00:50:48.040 --> 00:50:50.780 align:middle line:84% So which means my output is actually going to be wrong. 00:50:50.780 --> 00:50:52.180 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:50:52.180 --> 00:50:55.600 align:middle line:84% Now the transformers are, since they're trained on lots of data, 00:50:55.600 --> 00:50:58.160 align:middle line:84% they will be quite robust to these things. 00:50:58.160 --> 00:51:00.680 align:middle line:84% But strictly, arithmetically speaking, correct. 00:51:00.680 --> 00:51:01.180 align:middle line:90% Yes. 00:51:01.180 --> 00:51:03.840 align:middle line:90% 00:51:03.840 --> 00:51:06.680 align:middle line:90% So let's look at an example. 00:51:06.680 --> 00:51:11.260 align:middle line:84% Let's assume that your standalone embeddings-- right? 00:51:11.260 --> 00:51:13.385 align:middle line:90% This is your vocabulary. 00:51:13.385 --> 00:51:15.760 align:middle line:84% "unknown," "cat," "mat," "I," "sit," "love, "the," "you," 00:51:15.760 --> 00:51:16.460 align:middle line:90% "on." 00:51:16.460 --> 00:51:17.080 align:middle line:90% That's it. 00:51:17.080 --> 00:51:18.920 align:middle line:90% That's our vocabulary. 00:51:18.920 --> 00:51:22.148 align:middle line:84% And for this vocabulary, we have these standalone embeddings. 00:51:22.148 --> 00:51:24.690 align:middle line:84% And just for argument, let's assume these embeddings are only 00:51:24.690 --> 00:51:26.410 align:middle line:90% 2 long. 00:51:26.410 --> 00:51:28.570 align:middle line:84% The dimension of these embeddings is 2. 00:51:28.570 --> 00:51:31.030 align:middle line:84% If you recall, the glove embeddings we used last week, 00:51:31.030 --> 00:51:32.703 align:middle line:84% I think they were about 100 long. 00:51:32.703 --> 00:51:34.370 align:middle line:84% And the ones we're using in the homework 00:51:34.370 --> 00:51:35.952 align:middle line:90% are even longer than that. 00:51:35.952 --> 00:51:37.910 align:middle line:84% But here, we are assuming they are only 2 long. 00:51:37.910 --> 00:51:42.770 align:middle line:84% So the embedding for "cat" is 0.5, 7.1 00:51:42.770 --> 00:51:43.270 align:middle line:90% All right. 00:51:43.270 --> 00:51:46.250 align:middle line:84% Now let's assume that we can have, 00:51:46.250 --> 00:51:49.050 align:middle line:84% at most, 10 words in any sentence that's coming in. 00:51:49.050 --> 00:51:51.930 align:middle line:84% And obviously, a particular word could be in position 0 00:51:51.930 --> 00:51:53.650 align:middle line:90% all the way through position 9. 00:51:53.650 --> 00:51:57.150 align:middle line:84% And we will learn embeddings for each of these positions. 00:51:57.150 --> 00:51:59.730 align:middle line:84% And these embeddings are also 2 long-- 00:51:59.730 --> 00:52:01.510 align:middle line:90% 2 units long, dimension 2. 00:52:01.510 --> 00:52:04.650 align:middle line:90% 00:52:04.650 --> 00:52:07.732 align:middle line:84% Now where will these embeddings come from? 00:52:07.732 --> 00:52:09.190 align:middle line:84% What's the answer to that question? 00:52:09.190 --> 00:52:10.890 align:middle line:84% What is the answer to the general question of where 00:52:10.890 --> 00:52:12.098 align:middle line:90% will these weights come from? 00:52:12.098 --> 00:52:14.428 align:middle line:90% 00:52:14.428 --> 00:52:15.720 align:middle line:90% We will learn it with backprop. 00:52:15.720 --> 00:52:20.026 align:middle line:90% 00:52:20.026 --> 00:52:22.050 align:middle line:84% We will start initially with random numbers, 00:52:22.050 --> 00:52:24.540 align:middle line:84% and then we'll get them, make them better and better 00:52:24.540 --> 00:52:26.580 align:middle line:90% as over the course of training. 00:52:26.580 --> 00:52:31.100 align:middle line:84% So what we do is we have these two tables of embeddings-- 00:52:31.100 --> 00:52:34.020 align:middle line:84% the standalone embedding for the word and the position embedding. 00:52:34.020 --> 00:52:36.940 align:middle line:84% And then, we literally add them up. 00:52:36.940 --> 00:52:39.140 align:middle line:84% So for example, let's say the word, 00:52:39.140 --> 00:52:41.620 align:middle line:84% the sentence that came in is "Cat sat mat." 00:52:41.620 --> 00:52:42.680 align:middle line:90% That's a sentence. 00:52:42.680 --> 00:52:44.780 align:middle line:84% It's got three words-- "cat," "sat," "mat." 00:52:44.780 --> 00:52:48.540 align:middle line:84% So what we do is we say, well, the embedding for "cat" 00:52:48.540 --> 00:52:51.140 align:middle line:90% is this thing here, 0.571. 00:52:51.140 --> 00:52:53.380 align:middle line:90% So I write it here, 0.5, 7.1. 00:52:53.380 --> 00:52:56.180 align:middle line:84% "Cat" happens to be the zeroth position of the word. 00:52:56.180 --> 00:52:59.800 align:middle line:84% So I grab the embedding for 0, which is 1.33.9. 00:52:59.800 --> 00:53:00.780 align:middle line:90% I stick it there. 00:53:00.780 --> 00:53:05.680 align:middle line:84% And then I literally add them up, 0.5 plus 1.3, 1.8, 11.0. 00:53:05.680 --> 00:53:07.220 align:middle line:90% That's it. 00:53:07.220 --> 00:53:12.140 align:middle line:84% So now the positional encoded embedding for the word "cat" 00:53:12.140 --> 00:53:14.300 align:middle line:90% is 1.8, 11.0-- 00:53:14.300 --> 00:53:18.420 align:middle line:90% not 0.5, 7.1. 00:53:18.420 --> 00:53:20.990 align:middle line:84% So if "cat" happens to show up in another part 00:53:20.990 --> 00:53:23.690 align:middle line:84% of the sentence-- let's say instead of "Cat sat mat," 00:53:23.690 --> 00:53:28.150 align:middle line:90% we had "Mat sat cat." 00:53:28.150 --> 00:53:31.990 align:middle line:84% Now "cat" is now the third position, which is 0, 1, and 2, 00:53:31.990 --> 00:53:34.490 align:middle line:84% which means its embedding doesn't change. 00:53:34.490 --> 00:53:35.770 align:middle line:90% It's just embedding for "cat." 00:53:35.770 --> 00:53:38.290 align:middle line:84% But now, instead of picking 0, we'll pick this one-- 00:53:38.290 --> 00:53:41.080 align:middle line:84% 0.681-- and put that here and add them up instead. 00:53:41.080 --> 00:53:43.710 align:middle line:90% 00:53:43.710 --> 00:53:46.870 align:middle line:84% So this is the idea of the positional encoding. 00:53:46.870 --> 00:53:49.970 align:middle line:84% This is how we inject position knowledge into the transformer. 00:53:49.970 --> 00:53:52.950 align:middle line:90% 00:53:52.950 --> 00:53:54.727 align:middle line:90% Yes. 00:53:54.727 --> 00:53:56.310 align:middle line:84% AUDIENCE: The position embedding would 00:53:56.310 --> 00:53:57.670 align:middle line:90% be different for each sentence? 00:53:57.670 --> 00:53:58.730 align:middle line:90% Like, how do you-- 00:53:58.730 --> 00:54:00.030 align:middle line:84% RAMA RAMAKRISHNAN: No, this is just one table, which 00:54:00.030 --> 00:54:01.750 align:middle line:90% tells you what the position is. 00:54:01.750 --> 00:54:04.190 align:middle line:84% So it says for a word that appears 00:54:04.190 --> 00:54:06.950 align:middle line:84% in the seventh position in any input sentence 00:54:06.950 --> 00:54:09.190 align:middle line:84% that you're feeding in, this is the embedding 00:54:09.190 --> 00:54:11.980 align:middle line:84% that you need to use for that position. 00:54:11.980 --> 00:54:16.870 align:middle line:90% 00:54:16.870 --> 00:54:20.540 align:middle line:84% AUDIENCE: If a word appears twice in the same sentence, 00:54:20.540 --> 00:54:21.600 align:middle line:90% how do we [INAUDIBLE]? 00:54:21.600 --> 00:54:23.017 align:middle line:84% RAMA RAMAKRISHNAN: Great question. 00:54:23.017 --> 00:54:26.540 align:middle line:84% So let's say, just for argument, let's say the sentence was "Cat 00:54:26.540 --> 00:54:27.560 align:middle line:90% cat cat." 00:54:27.560 --> 00:54:33.160 align:middle line:84% So for each one of those cat-- for "Cat cat cat," 00:54:33.160 --> 00:54:34.720 align:middle line:84% this embedding will be the same-- 00:54:34.720 --> 00:54:37.120 align:middle line:84% 0.5, 7.1-- because that happens to be just the embedding 00:54:37.120 --> 00:54:39.480 align:middle line:84% for "cat" regardless of position. 00:54:39.480 --> 00:54:42.940 align:middle line:84% But then the first "cat," for the first "cat," 00:54:42.940 --> 00:54:46.660 align:middle line:84% we will use 1.3, 3.9 as the addition. 00:54:46.660 --> 00:54:50.040 align:middle line:84% For the second "cat," we'll use 6.3, 3.7, and the third "cat," 00:54:50.040 --> 00:54:51.640 align:middle line:90% we'll use 0.6, 8.1. 00:54:51.640 --> 00:54:53.380 align:middle line:84% So only the things that are adding, 00:54:53.380 --> 00:54:54.880 align:middle line:84% the positional encoding will change, 00:54:54.880 --> 00:54:56.125 align:middle line:90% the positional embedding. 00:54:56.125 --> 00:54:58.000 align:middle line:84% So the resulting sum is going to be different 00:54:58.000 --> 00:55:00.120 align:middle line:84% for each of these three words, even though they're exactly 00:55:00.120 --> 00:55:00.980 align:middle line:90% the same word. 00:55:00.980 --> 00:55:05.880 align:middle line:90% 00:55:05.880 --> 00:55:08.080 align:middle line:84% AUDIENCE: Is that position embedding table 00:55:08.080 --> 00:55:10.360 align:middle line:84% specific to that standalone embedding table? 00:55:10.360 --> 00:55:14.200 align:middle line:84% If you were to add or remove some words from the standalone? 00:55:14.200 --> 00:55:15.920 align:middle line:84% RAMA RAMAKRISHNAN: It's independent. 00:55:15.920 --> 00:55:16.480 align:middle line:90% Independent. 00:55:16.480 --> 00:55:19.450 align:middle line:84% It only depends on your assumption about how long 00:55:19.450 --> 00:55:20.970 align:middle line:90% the sentences can be. 00:55:20.970 --> 00:55:21.850 align:middle line:90% That's it. 00:55:21.850 --> 00:55:24.430 align:middle line:84% It doesn't really care about what words are coming in. 00:55:24.430 --> 00:55:25.952 align:middle line:90% That's a whole different thing. 00:55:25.952 --> 00:55:27.410 align:middle line:84% So these are two independent tables 00:55:27.410 --> 00:55:29.368 align:middle line:84% that have just learned as part of this process. 00:55:29.368 --> 00:55:31.650 align:middle line:90% 00:55:31.650 --> 00:55:35.730 align:middle line:84% So yeah, I have the same thing for "sat" and "mat," "sat" 00:55:35.730 --> 00:55:36.390 align:middle line:90% and "mat." 00:55:36.390 --> 00:55:39.450 align:middle line:90% That's what we have. 00:55:39.450 --> 00:55:42.450 align:middle line:84% So just make sure you understand these two slides to really make 00:55:42.450 --> 00:55:44.010 align:middle line:90% sure the mechanics are clear. 00:55:44.010 --> 00:55:46.330 align:middle line:90% Yeah. 00:55:46.330 --> 00:55:48.870 align:middle line:84% AUDIENCE: How do you filter out for filler words? 00:55:48.870 --> 00:55:53.650 align:middle line:84% For example, if you're taking NLP output for transcription, 00:55:53.650 --> 00:55:55.750 align:middle line:84% and you were trying to run a transformer on it, 00:55:55.750 --> 00:55:58.170 align:middle line:84% and you have a lot of "ums" and "likes" 00:55:58.170 --> 00:56:00.210 align:middle line:84% that are disproportionately large, 00:56:00.210 --> 00:56:03.610 align:middle line:84% and have these random assignments or really deep 00:56:03.610 --> 00:56:04.330 align:middle line:90% embeddings. 00:56:04.330 --> 00:56:06.970 align:middle line:84% Are there ways to filter out a certain noise? 00:56:06.970 --> 00:56:09.428 align:middle line:84% RAMA RAMAKRISHNAN: Typically, what they do is, as we will-- 00:56:09.428 --> 00:56:10.845 align:middle line:84% we'll talk about this thing called 00:56:10.845 --> 00:56:12.610 align:middle line:84% byte-pair encoding, in which we take 00:56:12.610 --> 00:56:15.940 align:middle line:84% individual characters, fragments of words, 00:56:15.940 --> 00:56:18.180 align:middle line:84% and words into account as tokens. 00:56:18.180 --> 00:56:19.980 align:middle line:84% So when you hear stuff like "um," "uh," 00:56:19.980 --> 00:56:22.912 align:middle line:84% and so on, it gets mapped to these small tokens. 00:56:22.912 --> 00:56:24.870 align:middle line:84% And then we treat them as just any other token. 00:56:24.870 --> 00:56:29.377 align:middle line:90% 00:56:29.377 --> 00:56:31.460 align:middle line:84% AUDIENCE: Yeah, is aggregation, like, a simple sum 00:56:31.460 --> 00:56:34.380 align:middle line:84% or [INAUDIBLE], the actual semantic meaning 00:56:34.380 --> 00:56:36.140 align:middle line:84% of the word in the standalone embedding 00:56:36.140 --> 00:56:38.580 align:middle line:84% might be more important than its relative position 00:56:38.580 --> 00:56:39.828 align:middle line:90% in the sentence? 00:56:39.828 --> 00:56:41.120 align:middle line:90% RAMA RAMAKRISHNAN: It could be. 00:56:41.120 --> 00:56:42.787 align:middle line:84% We just don't know a priori whether it's 00:56:42.787 --> 00:56:45.380 align:middle line:84% going to be important or not for any particular sentence. 00:56:45.380 --> 00:56:48.985 align:middle line:84% When we train the transformer with a lot of textual data, 00:56:48.985 --> 00:56:51.360 align:middle line:84% it will just figure out the right values for these things 00:56:51.360 --> 00:56:55.220 align:middle line:84% so that on average, the accuracy is as high as possible. 00:56:55.220 --> 00:56:56.787 align:middle line:84% So in many of these things, there's 00:56:56.787 --> 00:56:58.620 align:middle line:84% always a tension between our human intuition 00:56:58.620 --> 00:57:00.990 align:middle line:84% as to how it should work, and whether you should just 00:57:00.990 --> 00:57:02.740 align:middle line:84% throw it into the meat grinder of backprop 00:57:02.740 --> 00:57:03.965 align:middle line:90% and see what happens. 00:57:03.965 --> 00:57:05.340 align:middle line:84% And so here, as it turns out, you 00:57:05.340 --> 00:57:06.673 align:middle line:90% can just throw it into backprop. 00:57:06.673 --> 00:57:09.060 align:middle line:84% It'll actually do a pretty good job. 00:57:09.060 --> 00:57:10.900 align:middle line:90% Yeah. 00:57:10.900 --> 00:57:12.780 align:middle line:84% AUDIENCE: For the positional encoding, 00:57:12.780 --> 00:57:16.010 align:middle line:84% we would just be using the sum vector. 00:57:16.010 --> 00:57:18.270 align:middle line:84% We wouldn't be using this 2 by 3 matrix 00:57:18.270 --> 00:57:20.243 align:middle line:84% that you have on the far right, right? 00:57:20.243 --> 00:57:21.410 align:middle line:90% RAMA RAMAKRISHNAN: Oh, yeah. 00:57:21.410 --> 00:57:22.890 align:middle line:90% This is just for demonstration. 00:57:22.890 --> 00:57:24.310 align:middle line:84% Basically, this is the thing that will actually 00:57:24.310 --> 00:57:25.410 align:middle line:90% go into the transformer. 00:57:25.410 --> 00:57:26.350 align:middle line:90% Correct. 00:57:26.350 --> 00:57:28.510 align:middle line:90% Yeah. 00:57:28.510 --> 00:57:31.830 align:middle line:84% That was just me being overly verbose in the slides. 00:57:31.830 --> 00:57:33.310 align:middle line:90% Yeah. 00:57:33.310 --> 00:57:35.890 align:middle line:84% AUDIENCE: [INAUDIBLE] sentence the input at this point, 00:57:35.890 --> 00:57:37.670 align:middle line:84% are we still parsing out punctuation, 00:57:37.670 --> 00:57:39.610 align:middle line:84% or do we have a multi-sentence input? 00:57:39.610 --> 00:57:41.350 align:middle line:84% Is there positional embedding vector 00:57:41.350 --> 00:57:42.810 align:middle line:90% for each of the sentences? 00:57:42.810 --> 00:57:43.810 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:57:43.810 --> 00:57:47.495 align:middle line:84% So here, basically, the starting point is tokens, right? 00:57:47.495 --> 00:57:48.870 align:middle line:84% And in our example, because we're 00:57:48.870 --> 00:57:51.550 align:middle line:84% working with the idea of simple standardization and stripping 00:57:51.550 --> 00:57:52.467 align:middle line:90% and things like that-- 00:57:52.467 --> 00:57:53.950 align:middle line:90% I'm just showing actual words-- 00:57:53.950 --> 00:57:55.790 align:middle line:84% if you go to something like GPT-4, 00:57:55.790 --> 00:57:58.470 align:middle line:84% since it uses a different tokenization scheme, 00:57:58.470 --> 00:58:01.310 align:middle line:84% each token might be part of a word. 00:58:01.310 --> 00:58:02.810 align:middle line:84% It might be an individual character. 00:58:02.810 --> 00:58:04.530 align:middle line:90% It might be a punctuation mark. 00:58:04.530 --> 00:58:08.390 align:middle line:84% In fact, the GPT family doesn't support punctuation, 00:58:08.390 --> 00:58:10.410 align:middle line:84% which is why when you ask a question, 00:58:10.410 --> 00:58:14.120 align:middle line:84% it comes back with intact punctuation in its response. 00:58:14.120 --> 00:58:17.520 align:middle line:84% And so we'll revisit this when we look at BP byte-pair encoding 00:58:17.520 --> 00:58:19.605 align:middle line:90% later on. 00:58:19.605 --> 00:58:21.480 align:middle line:84% But the key thing to remember is that all the 00:58:21.480 --> 00:58:22.855 align:middle line:84% stuff you're talking about starts 00:58:22.855 --> 00:58:24.680 align:middle line:90% from the notion of a token. 00:58:24.680 --> 00:58:27.460 align:middle line:84% As to how you define a token, given a bunch of text, 00:58:27.460 --> 00:58:29.440 align:middle line:90% that's the tokenizer's job. 00:58:29.440 --> 00:58:34.480 align:middle line:84% And we just assumed a simple tokenizer for the time being. 00:58:34.480 --> 00:58:39.175 align:middle line:84% So at this point, folks, we have satisfied all the requirements. 00:58:39.175 --> 00:58:41.300 align:middle line:84% We have taken the surrounding context of each word, 00:58:41.300 --> 00:58:43.460 align:middle line:84% we taken the order, and so on and so forth, 00:58:43.460 --> 00:58:47.020 align:middle line:84% because what's coming in here is the positional embeddings. 00:58:47.020 --> 00:58:49.640 align:middle line:84% And it runs through the whole transformer stack. 00:58:49.640 --> 00:58:55.920 align:middle line:84% So this is called a transformer encoder. 00:58:55.920 --> 00:58:57.597 align:middle line:90% This is the transformer encoder. 00:58:57.597 --> 00:58:59.680 align:middle line:84% And you can see here, this is the original picture 00:58:59.680 --> 00:59:01.040 align:middle line:90% from the paper. 00:59:01.040 --> 00:59:03.240 align:middle line:84% It's an iconic picture at this point. 00:59:03.240 --> 00:59:05.220 align:middle line:84% So it says here, these are the inputs. 00:59:05.220 --> 00:59:08.320 align:middle line:84% This is like "The cat sat on the Mat " It comes in here, 00:59:08.320 --> 00:59:09.610 align:middle line:90% gets transferred to-- 00:59:09.610 --> 00:59:12.650 align:middle line:84% transformed into embeddings, standalone embeddings. 00:59:12.650 --> 00:59:16.250 align:middle line:84% And then, based on the position of each word, we add-- 00:59:16.250 --> 00:59:18.290 align:middle line:84% that is why you see a plus sign here. 00:59:18.290 --> 00:59:22.090 align:middle line:84% We add the positional embedding to that, 00:59:22.090 --> 00:59:25.570 align:middle line:84% and the resulting thing goes into this transformer block. 00:59:25.570 --> 00:59:30.770 align:middle line:84% And here, we go through multi-head attention. 00:59:30.770 --> 00:59:32.803 align:middle line:84% And things come out the other end. 00:59:32.803 --> 00:59:34.970 align:middle line:84% Then, there is this thing called add and norm, which 00:59:34.970 --> 00:59:37.410 align:middle line:90% we'll revisit on Wednesday. 00:59:37.410 --> 00:59:40.170 align:middle line:84% And then it goes through a feed forward network, another add 00:59:40.170 --> 00:59:42.250 align:middle line:84% and norm, which we'll revisit on Wednesday. 00:59:42.250 --> 00:59:43.750 align:middle line:84% And then it comes out the other end. 00:59:43.750 --> 00:59:44.230 align:middle line:90% That's it. 00:59:44.230 --> 00:59:45.522 align:middle line:90% That's the transformer encoder. 00:59:45.522 --> 00:59:48.162 align:middle line:90% 00:59:48.162 --> 00:59:53.830 align:middle line:84% And so if you look at this, just to point out a couple of things, 00:59:53.830 --> 00:59:55.630 align:middle line:84% the input embeddings can be random weights, 00:59:55.630 --> 00:59:58.463 align:middle line:84% or it could be pretrained embeddings. 00:59:58.463 --> 01:00:00.130 align:middle line:84% We add in a position-dependent embedding 01:00:00.130 --> 01:00:01.670 align:middle line:84% to represent the position of each word in the sentence. 01:00:01.670 --> 01:00:02.573 align:middle line:90% That's the plus. 01:00:02.573 --> 01:00:04.490 align:middle line:84% Then we pass it through multi-headed attention 01:00:04.490 --> 01:00:07.210 align:middle line:84% to get a contextual representation. 01:00:07.210 --> 01:00:10.760 align:middle line:84% Then, finally, we pass all this through a simple-- 01:00:10.760 --> 01:00:12.317 align:middle line:84% typically, it's a two-layer network, 01:00:12.317 --> 01:00:13.900 align:middle line:84% a one hidden layer with ReLUs and then 01:00:13.900 --> 01:00:15.900 align:middle line:84% a linear layer after that, and boom. 01:00:15.900 --> 01:00:17.240 align:middle line:90% And then we do it. 01:00:17.240 --> 01:00:18.200 align:middle line:90% This is the encoder. 01:00:18.200 --> 01:00:23.780 align:middle line:84% And here is perhaps the most important point to keep in mind. 01:00:23.780 --> 01:00:25.540 align:middle line:84% Because we have taken inordinate care 01:00:25.540 --> 01:00:27.498 align:middle line:84% to make sure that the things that are coming in 01:00:27.498 --> 01:00:30.180 align:middle line:84% and the things that are going out have the same size, 01:00:30.180 --> 01:00:32.060 align:middle line:84% both in terms of the number of tokens 01:00:32.060 --> 01:00:34.180 align:middle line:84% as well as the length of each vector, 01:00:34.180 --> 01:00:37.260 align:middle line:84% we can then stack them up like pancakes. 01:00:37.260 --> 01:00:39.260 align:middle line:84% We can have lots of transformer stacks, one 01:00:39.260 --> 01:00:41.690 align:middle line:90% on top of each other. 01:00:41.690 --> 01:00:43.660 align:middle line:90% Because it's the perfect API. 01:00:43.660 --> 01:00:45.380 align:middle line:90% It's the simplest possible API. 01:00:45.380 --> 01:00:46.838 align:middle line:84% The same thing comes in, same thing 01:00:46.838 --> 01:00:49.038 align:middle line:90% goes out, in terms of size. 01:00:49.038 --> 01:00:51.580 align:middle line:84% So you can have a transformer encoder, another one top, boom, 01:00:51.580 --> 01:00:53.960 align:middle line:84% boom, boom, boom, boom, one after the other. 01:00:53.960 --> 01:00:56.210 align:middle line:90% GPT-3 has 96 transformer stacks. 01:00:56.210 --> 01:00:58.740 align:middle line:90% 01:00:58.740 --> 01:01:01.660 align:middle line:84% And like in all things deep-learning related, the more 01:01:01.660 --> 01:01:03.940 align:middle line:84% layers you have, the more complicated things 01:01:03.940 --> 01:01:05.950 align:middle line:84% we can do with it, as long as you 01:01:05.950 --> 01:01:07.670 align:middle line:84% have enough data to keep the model happy 01:01:07.670 --> 01:01:08.587 align:middle line:90% so it doesn't overfit. 01:01:08.587 --> 01:01:11.790 align:middle line:90% 01:01:11.790 --> 01:01:13.430 align:middle line:90% OK? 01:01:13.430 --> 01:01:14.170 align:middle line:90% All right. 01:01:14.170 --> 01:01:18.190 align:middle line:84% So what we haven't covered, which we'll cover on Wednesday, 01:01:18.190 --> 01:01:22.667 align:middle line:84% is the question that he had posed, about how, 01:01:22.667 --> 01:01:25.250 align:middle line:84% since there are no parameters inside the self-attention block, 01:01:25.250 --> 01:01:26.513 align:middle line:90% what are we actually learning? 01:01:26.513 --> 01:01:28.930 align:middle line:84% And then there is these things called residual connections 01:01:28.930 --> 01:01:30.132 align:middle line:90% and layer normalization. 01:01:30.132 --> 01:01:32.090 align:middle line:84% We'll talk about all those things on Wednesday. 01:01:32.090 --> 01:01:35.230 align:middle line:84% Those are all refinements to the idea. 01:01:35.230 --> 01:01:36.530 align:middle line:90% So all right. 01:01:36.530 --> 01:01:37.910 align:middle line:90% 9:39. 01:01:37.910 --> 01:01:40.950 align:middle line:84% Let's apply the transformer encoder to an actual problem. 01:01:40.950 --> 01:01:43.630 align:middle line:90% Any questions? 01:01:43.630 --> 01:01:44.975 align:middle line:90% Yeah. 01:01:44.975 --> 01:01:46.350 align:middle line:84% AUDIENCE: My question, regarding, 01:01:46.350 --> 01:01:49.070 align:middle line:84% like, you said you could have multiple transformers. 01:01:49.070 --> 01:01:50.510 align:middle line:84% What is the difference with having 01:01:50.510 --> 01:01:53.503 align:middle line:84% multiple self-attention heads, and rather than 01:01:53.503 --> 01:01:54.670 align:middle line:90% having multiple [INAUDIBLE]? 01:01:54.670 --> 01:01:56.753 align:middle line:84% RAMA RAMAKRISHNAN: When I say a transformer block, 01:01:56.753 --> 01:01:58.810 align:middle line:84% within the block, there could be multiple heads. 01:01:58.810 --> 01:02:00.930 align:middle line:84% AUDIENCE: So if you increase accuracy, 01:02:00.930 --> 01:02:04.295 align:middle line:84% as you say, by [INAUDIBLE] this this way 01:02:04.295 --> 01:02:06.920 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, you can have a lot of attention heads. 01:02:06.920 --> 01:02:08.100 align:middle line:90% And that's totally fine. 01:02:08.100 --> 01:02:10.807 align:middle line:84% And typically, I forget how many GPT-3 and 4 have. 01:02:10.807 --> 01:02:12.140 align:middle line:90% They have a whole bunch of them. 01:02:12.140 --> 01:02:15.360 align:middle line:84% But you can-- so you can go wide and you can go deep. 01:02:15.360 --> 01:02:18.240 align:middle line:90% Both are done in practice. 01:02:18.240 --> 01:02:20.520 align:middle line:84% But the thing is, the one thing you have to remember 01:02:20.520 --> 01:02:23.440 align:middle line:84% is that if you go wide, you have a lot of attention heads, 01:02:23.440 --> 01:02:27.020 align:middle line:84% then, given the particular input that's coming into that block, 01:02:27.020 --> 01:02:29.360 align:middle line:84% it will learn different patterns from it. 01:02:29.360 --> 01:02:31.080 align:middle line:84% Well, if you stack them all up, it's 01:02:31.080 --> 01:02:33.010 align:middle line:84% going to learn different ways to contextualize 01:02:33.010 --> 01:02:34.260 align:middle line:90% the things that are coming in. 01:02:34.260 --> 01:02:36.093 align:middle line:84% It operates at higher levels of abstraction. 01:02:36.093 --> 01:02:38.160 align:middle line:84% So the analogy would be that, the seventh layer 01:02:38.160 --> 01:02:41.760 align:middle line:84% of a convolutional net, take the sixth layer's output 01:02:41.760 --> 01:02:44.420 align:middle line:84% and say, oh, I'm seeing a lot of edges here. 01:02:44.420 --> 01:02:46.920 align:middle line:84% I'm going to take an edge like this, two circles like that, 01:02:46.920 --> 01:02:48.480 align:middle line:90% and call it a face. 01:02:48.480 --> 01:02:50.688 align:middle line:84% So it will operate at a high level of abstraction. 01:02:50.688 --> 01:02:58.320 align:middle line:90% 01:02:58.320 --> 01:02:59.020 align:middle line:90% All right. 01:02:59.020 --> 01:03:01.473 align:middle line:90% Let's go to the Colab. 01:03:01.473 --> 01:03:02.890 align:middle line:84% So what we're going to do is we're 01:03:02.890 --> 01:03:05.265 align:middle line:84% going to take the transformer that we just learned about, 01:03:05.265 --> 01:03:08.270 align:middle line:84% and we're going to apply it to solve the travel slot problem. 01:03:08.270 --> 01:03:09.730 align:middle line:90% OK? 01:03:09.730 --> 01:03:10.430 align:middle line:90% All right. 01:03:10.430 --> 01:03:12.670 align:middle line:90% So OK. 01:03:12.670 --> 01:03:16.210 align:middle line:84% So we'll start with the usual preliminaries. 01:03:16.210 --> 01:03:18.050 align:middle line:84% And then we have taken the eighth test 01:03:18.050 --> 01:03:20.210 align:middle line:84% data set I talked about, and we have stuck them 01:03:20.210 --> 01:03:23.676 align:middle line:90% in raw box for easy consumption. 01:03:23.676 --> 01:03:24.440 align:middle line:90% It's here. 01:03:24.440 --> 01:03:30.636 align:middle line:90% 01:03:30.636 --> 01:03:34.310 align:middle line:84% So if you look at to the top view, you can see here, 01:03:34.310 --> 01:03:37.330 align:middle line:84% for example, I want to fly from Boston, 8:38 AM. 01:03:37.330 --> 01:03:38.560 align:middle line:90% And then this is the output. 01:03:38.560 --> 01:03:41.090 align:middle line:84% And the slot filling is the output. 01:03:41.090 --> 01:03:44.430 align:middle line:84% And so, as it turns out, here there is-- 01:03:44.430 --> 01:03:46.048 align:middle line:90% these people also gave it-- 01:03:46.048 --> 01:03:48.590 align:middle line:84% they took the whole query and gave it an intent as to is it-- 01:03:48.590 --> 01:03:49.587 align:middle line:90% it's a flight query. 01:03:49.587 --> 01:03:51.170 align:middle line:84% It's a something else query and so on, 01:03:51.170 --> 01:03:52.330 align:middle line:90% which we're not going to-- 01:03:52.330 --> 01:03:54.490 align:middle line:90% are you kidding me? 01:03:54.490 --> 01:03:57.250 align:middle line:84% I want to fly from Boston at 8:30 AM and arrive in Denver 01:03:57.250 --> 01:03:58.290 align:middle line:90% at 11:00 in the morning. 01:03:58.290 --> 01:03:59.850 align:middle line:84% What kind of ground transportation 01:03:59.850 --> 01:04:01.300 align:middle line:90% is available in Denver? 01:04:01.300 --> 01:04:04.340 align:middle line:90% What's the airport at Orlando? 01:04:04.340 --> 01:04:08.180 align:middle line:84% How much does the limo service cost within Pittsburgh? 01:04:08.180 --> 01:04:09.360 align:middle line:90% And so on and so forth. 01:04:09.360 --> 01:04:10.320 align:middle line:90% So you get the idea. 01:04:10.320 --> 01:04:12.100 align:middle line:84% It's a very wide range of queries 01:04:12.100 --> 01:04:14.140 align:middle line:90% that are in this data set. 01:04:14.140 --> 01:04:17.820 align:middle line:84% So let's just ignore that for a sec. 01:04:17.820 --> 01:04:19.740 align:middle line:84% So what we're now going to do is we're 01:04:19.740 --> 01:04:24.980 align:middle line:84% going to take only this column, right, the query column. 01:04:24.980 --> 01:04:27.820 align:middle line:84% That's going to be our input text. 01:04:27.820 --> 01:04:29.620 align:middle line:84% And then the slot-filling column is 01:04:29.620 --> 01:04:32.620 align:middle line:84% going to be our dependent variable, the output. 01:04:32.620 --> 01:04:37.460 align:middle line:84% So we'll just gather them all up here. 01:04:37.460 --> 01:04:38.040 align:middle line:90% Let it run. 01:04:38.040 --> 01:04:40.580 align:middle line:84% We'll do it for the training data and the test data. 01:04:40.580 --> 01:04:42.460 align:middle line:84% And so what we have done is that we 01:04:42.460 --> 01:04:45.700 align:middle line:84% have taken the transformer-related code 01:04:45.700 --> 01:04:48.580 align:middle line:84% in Keras, and we have packaged it into a little HODL 01:04:48.580 --> 01:04:50.980 align:middle line:90% library for easy consumption. 01:04:50.980 --> 01:04:53.160 align:middle line:90% And so that thing is here. 01:04:53.160 --> 01:04:55.260 align:middle line:90% You can download it. 01:04:55.260 --> 01:04:56.920 align:middle line:84% Calling it a library is overstating it. 01:04:56.920 --> 01:04:58.628 align:middle line:84% We literally just created a bunch of code 01:04:58.628 --> 01:05:00.750 align:middle line:90% and stuck it in a file. 01:05:00.750 --> 01:05:02.630 align:middle line:84% And so what we'll do is from HODL, 01:05:02.630 --> 01:05:05.030 align:middle line:84% we'll import the transformer encoder. 01:05:05.030 --> 01:05:07.693 align:middle line:84% And we will import this positional embedding layer, 01:05:07.693 --> 01:05:09.110 align:middle line:84% because what we are going to do is 01:05:09.110 --> 01:05:11.910 align:middle line:84% we are going to take the input, do the positional encoding 01:05:11.910 --> 01:05:14.270 align:middle line:84% business, and then send it into the transformer. 01:05:14.270 --> 01:05:16.656 align:middle line:90% OK? 01:05:16.656 --> 01:05:19.430 align:middle line:84% But first, let's vectorize the input 01:05:19.430 --> 01:05:21.230 align:middle line:90% queries that are coming in. 01:05:21.230 --> 01:05:25.310 align:middle line:90% So we'll define a thing here. 01:05:25.310 --> 01:05:28.410 align:middle line:84% This is a-- max query length is not defined. 01:05:28.410 --> 01:05:32.110 align:middle line:84% That's what happens when you don't run everything. 01:05:32.110 --> 01:05:32.730 align:middle line:90% All right. 01:05:32.730 --> 01:05:38.670 align:middle line:90% 01:05:38.670 --> 01:05:41.050 align:middle line:90% So now we have this thing here. 01:05:41.050 --> 01:05:45.010 align:middle line:84% So turns out that there are 888 tokens. 01:05:45.010 --> 01:05:49.150 align:middle line:84% 888 words in the input queries that we have in the data. 01:05:49.150 --> 01:05:52.350 align:middle line:90% So take a look at the first few. 01:05:52.350 --> 01:05:55.190 align:middle line:84% And you can see here, there is UNK. 01:05:55.190 --> 01:05:57.040 align:middle line:84% And because the output mode here is-- 01:05:57.040 --> 01:05:58.805 align:middle line:84% you just want integers to come out, not 01:05:58.805 --> 01:06:00.680 align:middle line:84% multi-hot encoding or anything, because we're 01:06:00.680 --> 01:06:04.720 align:middle line:84% going to take those integers and then do embeddings from them. 01:06:04.720 --> 01:06:08.760 align:middle line:84% It will reserve this empty string as the pad token. 01:06:08.760 --> 01:06:11.120 align:middle line:84% This should be familiar from last week. 01:06:11.120 --> 01:06:12.785 align:middle line:84% And then the UNK, for unknown tokens, 01:06:12.785 --> 01:06:15.160 align:middle line:84% and then 'to' 'from' 'flights' These are some of the most 01:06:15.160 --> 01:06:16.662 align:middle line:90% frequent. 01:06:16.662 --> 01:06:18.620 align:middle line:84% Turns out Boston is actually the most frequent. 01:06:18.620 --> 01:06:21.160 align:middle line:84% I don't know what's up with that, but it is what it is. 01:06:21.160 --> 01:06:23.200 align:middle line:84% Then we'll do the same vectorization 01:06:23.200 --> 01:06:25.360 align:middle line:90% to the train and test data sets. 01:06:25.360 --> 01:06:29.000 align:middle line:84% Now we need to do STIE for the output side of the problem, 01:06:29.000 --> 01:06:32.280 align:middle line:84% because the slots, the dependent variable here, remember, 01:06:32.280 --> 01:06:35.040 align:middle line:84% are all sentences as well, with the B, 01:06:35.040 --> 01:06:36.320 align:middle line:90% O, things like that, right? 01:06:36.320 --> 01:06:40.980 align:middle line:84% So we need to vectorize those, or we need to do STIE on them. 01:06:40.980 --> 01:06:43.240 align:middle line:84% So let's take a look at some of these slots. 01:06:43.240 --> 01:06:45.880 align:middle line:84% And you can see here, all this stuff is going on. 01:06:45.880 --> 01:06:47.897 align:middle line:84% Note-- so now, here is an example 01:06:47.897 --> 01:06:49.480 align:middle line:84% where you have to be very careful when 01:06:49.480 --> 01:06:51.400 align:middle line:90% you do the standardization. 01:06:51.400 --> 01:06:54.480 align:middle line:84% Typically, standardization, you will remove punctuation, and do 01:06:54.480 --> 01:06:55.750 align:middle line:90% things like that in lowercase. 01:06:55.750 --> 01:07:00.110 align:middle line:84% But here, these things have a specific meaning. 01:07:00.110 --> 01:07:01.610 align:middle line:84% We can't just go in there and remove 01:07:01.610 --> 01:07:03.478 align:middle line:84% the period and the underscore and then make 01:07:03.478 --> 01:07:05.270 align:middle line:84% the B into lowercase B and stuff like that. 01:07:05.270 --> 01:07:06.710 align:middle line:90% That will just harm it, right? 01:07:06.710 --> 01:07:09.890 align:middle line:84% We need to be able to preserve the nomenclature of the output 01:07:09.890 --> 01:07:11.570 align:middle line:90% in terms of all those tags. 01:07:11.570 --> 01:07:13.637 align:middle line:84% So we don't want the standardization 01:07:13.637 --> 01:07:14.470 align:middle line:90% to do all those out. 01:07:14.470 --> 01:07:16.970 align:middle line:84% So what we do is we say standardization, none. 01:07:16.970 --> 01:07:18.370 align:middle line:90% Look at that. 01:07:18.370 --> 01:07:20.190 align:middle line:84% We tell Keras, do not standardize this. 01:07:20.190 --> 01:07:21.330 align:middle line:90% Do not do your usual thing. 01:07:21.330 --> 01:07:24.010 align:middle line:90% 01:07:24.010 --> 01:07:28.450 align:middle line:84% So we do that for the output side. 01:07:28.450 --> 01:07:31.010 align:middle line:84% And then let's look at the vocabulary. 01:07:31.010 --> 01:07:33.370 align:middle line:84% Yeah, so this sounds pretty good. 01:07:33.370 --> 01:07:35.850 align:middle line:84% These are all the things that we would expect to see. 01:07:35.850 --> 01:07:40.290 align:middle line:84% These are the distinct tokens in the output strings. 01:07:40.290 --> 01:07:43.210 align:middle line:90% All right. 01:07:43.210 --> 01:07:43.710 align:middle line:90% OK. 01:07:43.710 --> 01:07:45.890 align:middle line:90% We get it. 01:07:45.890 --> 01:07:47.890 align:middle line:90% So we have 125 of them. 01:07:47.890 --> 01:07:50.450 align:middle line:84% In the lecture, I said there are 123 slots. 01:07:50.450 --> 01:07:51.230 align:middle line:90% Possible slots. 01:07:51.230 --> 01:07:53.167 align:middle line:90% Why is it 125 here? 01:07:53.167 --> 01:07:54.300 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:07:54.300 --> 01:07:55.758 align:middle line:84% RAMA RAMAKRISHNAN: Yes, uncompared. 01:07:55.758 --> 01:07:57.620 align:middle line:90% Correct. 01:07:57.620 --> 01:08:01.685 align:middle line:84% OK, now we'll set up a transformer encoder. 01:08:01.685 --> 01:08:02.560 align:middle line:90% Oh, wait, wait, wait. 01:08:02.560 --> 01:08:04.980 align:middle line:90% I forgot about doing this. 01:08:04.980 --> 01:08:07.300 align:middle line:90% My bad. 01:08:07.300 --> 01:08:07.840 align:middle line:90% All right. 01:08:07.840 --> 01:08:11.157 align:middle line:90% 01:08:11.157 --> 01:08:12.740 align:middle line:84% I just thought, when I saw this slide, 01:08:12.740 --> 01:08:16.300 align:middle line:84% that we should go to the Colab without giving you 01:08:16.300 --> 01:08:17.319 align:middle line:90% a bit more background. 01:08:17.319 --> 01:08:18.939 align:middle line:90% No problem. 01:08:18.939 --> 01:08:21.132 align:middle line:84% So the way we are going to model this problem 01:08:21.132 --> 01:08:23.340 align:middle line:84% is that we're going to have something like this-- fly 01:08:23.340 --> 01:08:24.823 align:middle line:90% from Boston to Denver. 01:08:24.823 --> 01:08:26.240 align:middle line:84% That's the input that's coming in. 01:08:26.240 --> 01:08:28.660 align:middle line:90% And that is the correct answer. 01:08:28.660 --> 01:08:30.899 align:middle line:84% 0, 0, some B, something or the other-- 01:08:30.899 --> 01:08:32.359 align:middle line:84% I mean, O, and then something else. 01:08:32.359 --> 01:08:33.859 align:middle line:90% That's the correct answer. 01:08:33.859 --> 01:08:36.420 align:middle line:84% That's the input, and that is the right answer. 01:08:36.420 --> 01:08:39.939 align:middle line:84% So what we'll do is we will create these positional input 01:08:39.939 --> 01:08:42.620 align:middle line:84% embeddings like we have discussed before. 01:08:42.620 --> 01:08:45.340 align:middle line:84% We will run it through a transformer. 01:08:45.340 --> 01:08:47.660 align:middle line:84% It gives us contextual embeddings. 01:08:47.660 --> 01:08:50.040 align:middle line:84% So if we send five in, it's going to send us five out, 01:08:50.040 --> 01:08:52.029 align:middle line:90% except the color is now blue. 01:08:52.029 --> 01:08:57.290 align:middle line:84% And then what we do is we will run it through a ReLU. 01:08:57.290 --> 01:08:57.790 align:middle line:90% OK? 01:08:57.790 --> 01:08:59.390 align:middle line:90% We'll run it through a ReLU. 01:08:59.390 --> 01:09:02.510 align:middle line:84% We will still have five vectors here. 01:09:02.510 --> 01:09:04.029 align:middle line:90% Five vectors will come in. 01:09:04.029 --> 01:09:07.069 align:middle line:84% And then, for each of the things that comes in, we will stick 01:09:07.069 --> 01:09:08.539 align:middle line:90% a 123-way softmax. 01:09:08.539 --> 01:09:12.109 align:middle line:90% 01:09:12.109 --> 01:09:15.111 align:middle line:84% For each thing that comes out, we'll have a 123-way softmax. 01:09:15.111 --> 01:09:17.569 align:middle line:84% And that's the classification problem we're going to solve. 01:09:17.569 --> 01:09:21.308 align:middle line:90% 01:09:21.308 --> 01:09:25.470 align:middle line:84% So the weights in all these layers 01:09:25.470 --> 01:09:28.310 align:middle line:90% will get optimized by backprop. 01:09:28.310 --> 01:09:30.990 align:middle line:84% All these weights are going to get optimized. 01:09:30.990 --> 01:09:31.670 align:middle line:90% Yeah. 01:09:31.670 --> 01:09:34.109 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:09:34.109 --> 01:09:36.298 align:middle line:90% RAMA RAMAKRISHNAN: Sorry? 01:09:36.298 --> 01:09:38.590 align:middle line:84% AUDIENCE: What about the positional embeddings imported 01:09:38.590 --> 01:09:41.149 align:middle line:90% in the Colab [INAUDIBLE]? 01:09:41.149 --> 01:09:43.189 align:middle line:84% RAMA RAMAKRISHNAN: No, that's the layer. 01:09:43.189 --> 01:09:46.870 align:middle line:84% The weights in the layer will still need to be learned. 01:09:46.870 --> 01:09:48.569 align:middle line:84% It's like the text vectorization layer. 01:09:48.569 --> 01:09:49.779 align:middle line:90% It's a bunch of code. 01:09:49.779 --> 01:09:52.000 align:middle line:84% And then you actually run it on a particular corpus 01:09:52.000 --> 01:09:54.480 align:middle line:84% to adapt it and fill a vocabulary out of it. 01:09:54.480 --> 01:09:57.647 align:middle line:84% So it's like an empty shell that needs to get populated. 01:09:57.647 --> 01:09:59.480 align:middle line:84% OK, so with the weights in, all these things 01:09:59.480 --> 01:10:04.080 align:middle line:84% are going to get updated when we train the model by backprop. 01:10:04.080 --> 01:10:04.820 align:middle line:90% And that's it. 01:10:04.820 --> 01:10:06.360 align:middle line:90% That's the setup. 01:10:06.360 --> 01:10:09.600 align:middle line:84% Does this make sense, before I switch back to the Colab? 01:10:09.600 --> 01:10:12.580 align:middle line:84% In particular, does this make sense, this part of it? 01:10:12.580 --> 01:10:15.920 align:middle line:90% 01:10:15.920 --> 01:10:16.980 align:middle line:90% Bunch of things come out. 01:10:16.980 --> 01:10:18.120 align:middle line:84% And then, for each one of those things, 01:10:18.120 --> 01:10:19.745 align:middle line:84% we need to figure out a classification, 01:10:19.745 --> 01:10:21.320 align:middle line:90% a 123-way classification. 01:10:21.320 --> 01:10:23.320 align:middle line:84% And that's where we stick a softmax in every one 01:10:23.320 --> 01:10:25.400 align:middle line:90% of those output nodes. 01:10:25.400 --> 01:10:26.297 align:middle line:90% Yeah. 01:10:26.297 --> 01:10:28.380 align:middle line:84% AUDIENCE: [INAUDIBLE] for the yellow vector layer, 01:10:28.380 --> 01:10:32.305 align:middle line:84% is there a reason that they're, like, 3-dimensional? 01:10:32.305 --> 01:10:33.680 align:middle line:84% RAMA RAMAKRISHNAN: Oh, oh, I see. 01:10:33.680 --> 01:10:34.513 align:middle line:90% AUDIENCE: Not three. 01:10:34.513 --> 01:10:36.583 align:middle line:84% But why it's changed from three to four? 01:10:36.583 --> 01:10:38.000 align:middle line:84% RAMA RAMAKRISHNAN: That's a good-- 01:10:38.000 --> 01:10:40.200 align:middle line:84% AUDIENCE: Or is it just kind of random, like [INAUDIBLE].? 01:10:40.200 --> 01:10:41.180 align:middle line:84% RAMA RAMAKRISHNAN: It could be whatever. 01:10:41.180 --> 01:10:43.760 align:middle line:84% Or to put it another way, it is your choice as the user, 01:10:43.760 --> 01:10:44.800 align:middle line:90% as the modeler. 01:10:44.800 --> 01:10:45.360 align:middle line:90% Correct. 01:10:45.360 --> 01:10:47.990 align:middle line:84% The thing is, at this point, with the blue stuff, 01:10:47.990 --> 01:10:51.370 align:middle line:84% the transformers basically say, my job is done. 01:10:51.370 --> 01:10:53.970 align:middle line:84% It has given you these valuable contextual embeddings 01:10:53.970 --> 01:10:55.738 align:middle line:84% at some high level of abstraction. 01:10:55.738 --> 01:10:58.030 align:middle line:84% What you do with it depends on your particular problem. 01:10:58.030 --> 01:11:00.750 align:middle line:84% And so the best practice would be to take it. 01:11:00.750 --> 01:11:03.530 align:middle line:84% And then maybe if these embeddings are really long, 01:11:03.530 --> 01:11:06.665 align:middle line:84% maybe make them a little smaller using a ReLU. 01:11:06.665 --> 01:11:08.290 align:middle line:84% And using a ReLU is always a good idea, 01:11:08.290 --> 01:11:11.650 align:middle line:84% because when in doubt, throw in a bit of non-linearity. 01:11:11.650 --> 01:11:14.690 align:middle line:84% And then, once you are done with that, well, at this point 01:11:14.690 --> 01:11:16.670 align:middle line:84% you need to actually classify it. 01:11:16.670 --> 01:11:20.530 align:middle line:84% So you stick an output softmax on it. 01:11:20.530 --> 01:11:22.390 align:middle line:90% So that's what we have. 01:11:22.390 --> 01:11:27.650 align:middle line:90% 01:11:27.650 --> 01:11:29.690 align:middle line:90% All right, back to this picture. 01:11:29.690 --> 01:11:32.570 align:middle line:84% So what we're going to do is we also 01:11:32.570 --> 01:11:35.190 align:middle line:84% get to decide how long are these embedding vectors. 01:11:35.190 --> 01:11:35.690 align:middle line:90% How long. 01:11:35.690 --> 01:11:37.940 align:middle line:84% Because here, we're not going to use glove embeddings. 01:11:37.940 --> 01:11:40.135 align:middle line:84% We're just going to learn everything from scratch. 01:11:40.135 --> 01:11:42.010 align:middle line:84% We're going to learn everything from scratch. 01:11:42.010 --> 01:11:44.490 align:middle line:84% And we can decide how long these embedding vectors are. 01:11:44.490 --> 01:11:47.700 align:middle line:84% So these embedding vectors, I'm going to decide. 01:11:47.700 --> 01:11:51.100 align:middle line:84% I have decided that I want them to be 512 long. 01:11:51.100 --> 01:11:53.953 align:middle line:84% I want these actually to be 512 long. 01:11:53.953 --> 01:11:55.120 align:middle line:90% So that's what I have here-- 01:11:55.120 --> 01:11:56.980 align:middle line:90% 512. 01:11:56.980 --> 01:11:58.960 align:middle line:84% And then, inside the transformer, 01:11:58.960 --> 01:12:02.078 align:middle line:84% remember, when we concatenate everything, 01:12:02.078 --> 01:12:03.620 align:middle line:84% and then we have something, we run it 01:12:03.620 --> 01:12:05.900 align:middle line:90% through a final ReLU layer. 01:12:05.900 --> 01:12:08.980 align:middle line:90% How big should that layer be? 01:12:08.980 --> 01:12:11.640 align:middle line:84% That is what, here, what I mean by dense_dim. 01:12:11.640 --> 01:12:13.320 align:middle line:90% I want it to be 64. 01:12:13.320 --> 01:12:17.540 align:middle line:84% And then I, for fun, I'm going to use five attention heads, 01:12:17.540 --> 01:12:20.460 align:middle line:90% because why not? 01:12:20.460 --> 01:12:26.180 align:middle line:84% And then, in the final thing here, to go to Alina's question 01:12:26.180 --> 01:12:29.980 align:middle line:84% here, these things are all 512 long, as I mentioned earlier. 01:12:29.980 --> 01:12:32.060 align:middle line:90% These are all 512. 01:12:32.060 --> 01:12:36.820 align:middle line:84% But this thing here, I'm going to make it just 128. 01:12:36.820 --> 01:12:38.780 align:middle line:84% That's what I mean by units here. 01:12:38.780 --> 01:12:41.260 align:middle line:84% And so if you look at the actual model, 01:12:41.260 --> 01:12:44.410 align:middle line:84% OK, whatever comes in has a max query length of, I think, 30, 01:12:44.410 --> 01:12:46.025 align:middle line:90% if I recall. 01:12:46.025 --> 01:12:47.650 align:middle line:84% Actually, let's just make sure of that. 01:12:47.650 --> 01:12:48.400 align:middle line:90% What did I assume? 01:12:48.400 --> 01:12:51.330 align:middle line:90% 01:12:51.330 --> 01:12:51.830 align:middle line:90% 30. 01:12:51.830 --> 01:12:52.210 align:middle line:90% Correct. 01:12:52.210 --> 01:12:53.090 align:middle line:90% Max query length 30. 01:12:53.090 --> 01:12:55.090 align:middle line:90% So each sentence is 30. 01:12:55.090 --> 01:12:58.475 align:middle line:84% So if a sentence has 35 words in it, what's going to happen? 01:12:58.475 --> 01:12:59.753 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:12:59.753 --> 01:13:02.170 align:middle line:84% RAMA RAMAKRISHNAN: The last 5 will get chopped, truncated. 01:13:02.170 --> 01:13:04.030 align:middle line:84% If it comes in at 22, we're going 01:13:04.030 --> 01:13:06.530 align:middle line:84% to pad it with eight more tokens, with a pad token. 01:13:06.530 --> 01:13:09.550 align:middle line:84% That's how we make sure everything gets to 30. 01:13:09.550 --> 01:13:10.170 align:middle line:90% All right. 01:13:10.170 --> 01:13:12.230 align:middle line:90% So we come back here. 01:13:12.230 --> 01:13:15.150 align:middle line:84% So the input is still sentences which are 30 long, 01:13:15.150 --> 01:13:16.750 align:middle line:90% tokens which are 30 long. 01:13:16.750 --> 01:13:20.550 align:middle line:84% And then we run it through a positional embedding layer. 01:13:20.550 --> 01:13:21.190 align:middle line:90% OK? 01:13:21.190 --> 01:13:24.870 align:middle line:84% This positional embedding layer has the actual embedding 01:13:24.870 --> 01:13:28.750 align:middle line:84% for each word in that table, and it has the positional table, 01:13:28.750 --> 01:13:30.070 align:middle line:90% the positional embedding table. 01:13:30.070 --> 01:13:32.870 align:middle line:84% So just to be clear, this positional embedding layer 01:13:32.870 --> 01:13:34.150 align:middle line:90% is basically-- 01:13:34.150 --> 01:13:37.150 align:middle line:90% it's basically this. 01:13:37.150 --> 01:13:40.190 align:middle line:84% So this table and this table together 01:13:40.190 --> 01:13:43.720 align:middle line:84% are packaged up into the positional encoding layer. 01:13:43.720 --> 01:13:45.300 align:middle line:84% But they are totally distinct tables. 01:13:45.300 --> 01:13:47.520 align:middle line:84% They just happen to be packaged up. 01:13:47.520 --> 01:13:51.120 align:middle line:90% So this is what we have here. 01:13:51.120 --> 01:13:53.940 align:middle line:84% And then we get a nice positional embedding out. 01:13:53.940 --> 01:13:56.240 align:middle line:84% And then, boom, we run it through the transformer. 01:13:56.240 --> 01:14:00.120 align:middle line:84% And this transformer encoder object, we have to tell it, 01:14:00.120 --> 01:14:02.280 align:middle line:84% obviously, hey, this is the embedding dimension 01:14:02.280 --> 01:14:03.343 align:middle line:90% that's going to come out. 01:14:03.343 --> 01:14:04.760 align:middle line:84% This is the dense dimension you're 01:14:04.760 --> 01:14:07.240 align:middle line:84% going to use in that final feed forward layer 01:14:07.240 --> 01:14:08.492 align:middle line:90% inside each attention block. 01:14:08.492 --> 01:14:10.200 align:middle line:84% And this is the number of attention heads 01:14:10.200 --> 01:14:10.950 align:middle line:90% I want you to use. 01:14:10.950 --> 01:14:11.600 align:middle line:90% That's it. 01:14:11.600 --> 01:14:12.900 align:middle line:90% It's very-- right? 01:14:12.900 --> 01:14:14.840 align:middle line:84% Only three things have to be specified. 01:14:14.840 --> 01:14:17.560 align:middle line:84% And then, whatever comes out of the transformer encoder 01:14:17.560 --> 01:14:19.160 align:middle line:90% are these blue vectors. 01:14:19.160 --> 01:14:23.000 align:middle line:84% And then we are back into good old traditional DNN stuff, where 01:14:23.000 --> 01:14:27.380 align:middle line:84% we take this thing, run it through a ReLU with 128 units. 01:14:27.380 --> 01:14:29.360 align:middle line:90% We add a dropout. 01:14:29.360 --> 01:14:31.920 align:middle line:84% And then we run it through a dense layer, 01:14:31.920 --> 01:14:34.580 align:middle line:84% which the vocab size here is 125, 01:14:34.580 --> 01:14:38.080 align:middle line:90% which is the 125-way softmax. 01:14:38.080 --> 01:14:39.887 align:middle line:90% Activation softmax. 01:14:39.887 --> 01:14:41.970 align:middle line:84% Connect up everything into model input and output. 01:14:41.970 --> 01:14:44.530 align:middle line:84% And boom, that's the whole model. 01:14:44.530 --> 01:14:48.645 align:middle line:90% So that's what we have here. 01:14:48.645 --> 01:14:56.010 align:middle line:84% Now, this, after Wednesday's class, for extra credit 01:14:56.010 --> 01:14:59.330 align:middle line:84% and for your personal edification, 01:14:59.330 --> 01:15:02.970 align:middle line:84% try to work through this thing to come up with this number-- 01:15:02.970 --> 01:15:07.010 align:middle line:90% 53 million-- sorry, 5.3 million. 01:15:07.010 --> 01:15:12.570 align:middle line:84% And see if it matches this number here. 01:15:12.570 --> 01:15:13.945 align:middle line:90% It should match. 01:15:13.945 --> 01:15:15.570 align:middle line:84% Hand calculate the number of parameters 01:15:15.570 --> 01:15:17.490 align:middle line:90% inside the transformer. 01:15:17.490 --> 01:15:18.550 align:middle line:90% For fame and fortune. 01:15:18.550 --> 01:15:19.770 align:middle line:90% That's an optional thing. 01:15:19.770 --> 01:15:23.382 align:middle line:84% So do it after Wednesday's class, not right now. 01:15:23.382 --> 01:15:25.090 align:middle line:84% And I have actually listed the exact math 01:15:25.090 --> 01:15:26.730 align:middle line:90% that goes into it here. 01:15:26.730 --> 01:15:27.310 align:middle line:90% All right. 01:15:27.310 --> 01:15:29.625 align:middle line:84% So, by the way, you can peek into any layer's weights 01:15:29.625 --> 01:15:30.750 align:middle line:90% using its weight attribute. 01:15:30.750 --> 01:15:33.130 align:middle line:84% This is the embedding, the positional embedding 01:15:33.130 --> 01:15:33.830 align:middle line:90% thing we had. 01:15:33.830 --> 01:15:35.510 align:middle line:90% So we can click it. 01:15:35.510 --> 01:15:38.220 align:middle line:84% And you can see here, it has two tables. 01:15:38.220 --> 01:15:40.880 align:middle line:84% There is the first table, which is just the embedding table, 01:15:40.880 --> 01:15:44.680 align:middle line:84% which says there are 888 tokens in my vocabulary, 01:15:44.680 --> 01:15:46.680 align:middle line:84% and each of those tokens is an embedding vector, 01:15:46.680 --> 01:15:47.860 align:middle line:90% which is 512 long. 01:15:47.860 --> 01:15:49.580 align:middle line:90% That is the first table here. 01:15:49.580 --> 01:15:51.300 align:middle line:84% And then it has the second object, which 01:15:51.300 --> 01:15:53.140 align:middle line:90% is the positional embedding. 01:15:53.140 --> 01:15:56.400 align:middle line:84% And it says here, well, my sentences can be 30 long. 01:15:56.400 --> 01:15:59.300 align:middle line:84% And for each position of the 30-long sentence, 01:15:59.300 --> 01:16:02.180 align:middle line:90% I will have a 512 embedding. 01:16:02.180 --> 01:16:04.200 align:middle line:84% Both these tables, as I mentioned earlier, 01:16:04.200 --> 01:16:05.180 align:middle line:90% are packaged up inside. 01:16:05.180 --> 01:16:06.888 align:middle line:84% And you can actually see what the weights 01:16:06.888 --> 01:16:09.590 align:middle line:90% are before you do any training. 01:16:09.590 --> 01:16:10.580 align:middle line:90% So all right. 01:16:10.580 --> 01:16:12.653 align:middle line:84% So I'm going to stop here, because the model, 01:16:12.653 --> 01:16:14.320 align:middle line:84% it's going to take a few minutes to run. 01:16:14.320 --> 01:16:16.260 align:middle line:90% And we are already at 9:45. 01:16:16.260 --> 01:16:18.228 align:middle line:84% So we will continue the journey on Wednesday. 01:16:18.228 --> 01:16:20.520 align:middle line:84% If some of it is not super clear, don't worry about it. 01:16:20.520 --> 01:16:21.920 align:middle line:84% It'll become much clearer on Wednesday. 01:16:21.920 --> 01:16:22.320 align:middle line:90% All right. 01:16:22.320 --> 01:16:24.195 align:middle line:84% All right, folks, have a good couple of days. 01:16:24.195 --> 01:16:25.590 align:middle line:90% I'll see you on Wednesday. 01:16:25.590 --> 01:16:37.000 align:middle line:90%