WEBVTT 00:00:00.000 --> 00:00:01.968 align:middle line:90% [RUSTLING] 00:00:01.968 --> 00:00:03.936 align:middle line:90% [SQUEAKING] 00:00:03.936 --> 00:00:04.920 align:middle line:90% [CLICKING] 00:00:04.920 --> 00:00:17.060 align:middle line:90% 00:00:17.060 --> 00:00:20.680 align:middle line:84% RAMA RAMAKRISHNAN: OK, so today we start the natural language 00:00:20.680 --> 00:00:22.200 align:middle line:90% processing sequence. 00:00:22.200 --> 00:00:24.260 align:middle line:84% And so just to give you a quick idea, 00:00:24.260 --> 00:00:28.440 align:middle line:84% we're going to start with what's called vectorization, and then 00:00:28.440 --> 00:00:29.700 align:middle line:90% the bag of words model. 00:00:29.700 --> 00:00:32.159 align:middle line:84% And then we'll spend a fair amount of time on a Colab. 00:00:32.159 --> 00:00:34.560 align:middle line:84% And then on Wednesday, we talk about these things 00:00:34.560 --> 00:00:37.000 align:middle line:84% called embeddings, which you will come to appreciate over 00:00:37.000 --> 00:00:41.160 align:middle line:84% the next couple of weeks, form the core atomic unit 00:00:41.160 --> 00:00:44.120 align:middle line:84% of all modern natural language processing, 00:00:44.120 --> 00:00:46.960 align:middle line:84% and for that matter, vision processing as well. 00:00:46.960 --> 00:00:49.000 align:middle line:84% And then the following week, we'll 00:00:49.000 --> 00:00:51.092 align:middle line:84% do transformers, two lectures on transformers. 00:00:51.092 --> 00:00:52.800 align:middle line:84% We'll get into the theory, and then we'll 00:00:52.800 --> 00:00:54.400 align:middle line:84% get into a bunch of applications. 00:00:54.400 --> 00:00:58.860 align:middle line:84% And then lectures 9 and 10 will be all about LLMs. 00:00:58.860 --> 00:01:00.980 align:middle line:84% So it's going to be a lot of fun. 00:01:00.980 --> 00:01:03.973 align:middle line:84% This is one of my favorite segments of the class. 00:01:03.973 --> 00:01:06.140 align:middle line:84% Of course, truth be told, every segment of the class 00:01:06.140 --> 00:01:08.540 align:middle line:84% is my favorite, so don't judge me. 00:01:08.540 --> 00:01:10.580 align:middle line:90% All right, so let's get going. 00:01:10.580 --> 00:01:15.100 align:middle line:84% So why natural language processing? 00:01:15.100 --> 00:01:18.080 align:middle line:84% In some sense, the things I have on the slide here are obvious, 00:01:18.080 --> 00:01:20.980 align:middle line:84% but I think it's actually worth reminding ourselves 00:01:20.980 --> 00:01:24.780 align:middle line:84% of how important text is for everything we do. 00:01:24.780 --> 00:01:27.980 align:middle line:84% Obviously, human knowledge is mostly encoded as text. 00:01:27.980 --> 00:01:29.800 align:middle line:90% The internet is mostly text. 00:01:29.800 --> 00:01:33.180 align:middle line:84% At least this was true until the advent of TikTok and YouTube. 00:01:33.180 --> 00:01:35.500 align:middle line:84% And human communication is mostly text. 00:01:35.500 --> 00:01:39.820 align:middle line:84% And cultural production-- movies, books, arts and so on, 00:01:39.820 --> 00:01:41.860 align:middle line:90% so much of it is so text heavy. 00:01:41.860 --> 00:01:45.740 align:middle line:84% And so, in some sense, text forms not just 00:01:45.740 --> 00:01:48.640 align:middle line:84% a big chunk of all the media that's out there, 00:01:48.640 --> 00:01:50.565 align:middle line:84% but it also happens to be the way in which we 00:01:50.565 --> 00:01:52.440 align:middle line:84% think and communicate and so on and so forth. 00:01:52.440 --> 00:01:55.760 align:middle line:84% So its primacy is, in my opinion, 00:01:55.760 --> 00:01:59.030 align:middle line:84% unparalleled in how we think about the world. 00:01:59.030 --> 00:02:01.870 align:middle line:84% And so the tantalizing possibility is that, 00:02:01.870 --> 00:02:05.470 align:middle line:84% imagine if we had an AI system which could just read and quote 00:02:05.470 --> 00:02:08.630 align:middle line:84% unquote, "understand" all this text. 00:02:08.630 --> 00:02:12.510 align:middle line:84% And so you can imagine such a system reading all of PubMed, 00:02:12.510 --> 00:02:14.250 align:middle line:84% reading all the medical literature, 00:02:14.250 --> 00:02:18.230 align:middle line:84% and then coming back and saying, for this particular disease, 00:02:18.230 --> 00:02:20.190 align:middle line:84% this particular protein is actually 00:02:20.190 --> 00:02:21.570 align:middle line:90% the malfunctioning protein. 00:02:21.570 --> 00:02:22.950 align:middle line:84% And for that, that small molecule 00:02:22.950 --> 00:02:25.750 align:middle line:84% is going to dock into the protein and cure the disease. 00:02:25.750 --> 00:02:28.167 align:middle line:84% And you didn't know this-- it came back and told you that. 00:02:28.167 --> 00:02:29.430 align:middle line:90% Wouldn't it be unbelievable? 00:02:29.430 --> 00:02:32.670 align:middle line:84% So my feeling is that such things are going to happen. 00:02:32.670 --> 00:02:34.350 align:middle line:84% It's just that it's not going to happen 00:02:34.350 --> 00:02:36.050 align:middle line:90% soon enough for my lifetime. 00:02:36.050 --> 00:02:37.910 align:middle line:84% But perhaps it'll happen in yours. 00:02:37.910 --> 00:02:39.970 align:middle line:90% All right, so let's continue. 00:02:39.970 --> 00:02:42.390 align:middle line:84% So NLP is in action all around us. 00:02:42.390 --> 00:02:45.950 align:middle line:84% According to Google, apparently Google autocomplete, 00:02:45.950 --> 00:02:48.750 align:middle line:84% which uses a fair bit of NLP, saves 00:02:48.750 --> 00:02:51.970 align:middle line:84% 200 years of typing time, apparently, every day. 00:02:51.970 --> 00:02:54.342 align:middle line:90% 00:02:54.342 --> 00:02:56.050 align:middle line:84% I wasn't very impressed with this number, 00:02:56.050 --> 00:02:57.930 align:middle line:84% frankly, because billions of searches 00:02:57.930 --> 00:03:00.370 align:middle line:84% are being done every day, and I'm like, only 200 years? 00:03:00.370 --> 00:03:03.370 align:middle line:84% So anyway, but I think the more important point is 00:03:03.370 --> 00:03:05.530 align:middle line:90% that it made mobile possible. 00:03:05.530 --> 00:03:07.930 align:middle line:84% If you didn't have autocomplete, people 00:03:07.930 --> 00:03:10.830 align:middle line:84% would not be typing and pecking on their keyboards. 00:03:10.830 --> 00:03:12.350 align:middle line:90% It's going to be much worse. 00:03:12.350 --> 00:03:14.990 align:middle line:84% You would have had a hugely dampening effect on e-commerce, 00:03:14.990 --> 00:03:15.810 align:middle line:90% for instance. 00:03:15.810 --> 00:03:18.090 align:middle line:84% So this humble little autocomplete 00:03:18.090 --> 00:03:21.235 align:middle line:84% has incredible, incredible impact on the world economy. 00:03:21.235 --> 00:03:22.610 align:middle line:84% And the other thing which I heard 00:03:22.610 --> 00:03:24.350 align:middle line:84% about, I'm not sure if it's 100% true, 00:03:24.350 --> 00:03:27.690 align:middle line:84% but it's an interesting example, apparently the very first iPhone 00:03:27.690 --> 00:03:31.890 align:middle line:84% keyboard that came out, the soft keyboard, not the hard keyboard, 00:03:31.890 --> 00:03:36.090 align:middle line:84% they had some very basic word continuation prediction 00:03:36.090 --> 00:03:36.910 align:middle line:90% going on. 00:03:36.910 --> 00:03:40.850 align:middle line:84% And so when you start typing t and h, 00:03:40.850 --> 00:03:44.430 align:middle line:84% obviously it's going to guess the e is going to come next. 00:03:44.430 --> 00:03:46.430 align:middle line:90% So that part is old, old news. 00:03:46.430 --> 00:03:47.230 align:middle line:90% Nothing new there. 00:03:47.230 --> 00:03:50.210 align:middle line:84% But apparently, the e letter in the keyboard 00:03:50.210 --> 00:03:52.570 align:middle line:90% will become slightly bigger. 00:03:52.570 --> 00:03:54.070 align:middle line:84% So when your finger goes towards it, 00:03:54.070 --> 00:03:56.260 align:middle line:84% it has a better shot of actually connecting with it. 00:03:56.260 --> 00:03:58.940 align:middle line:84% So these kinds of things are used 00:03:58.940 --> 00:04:01.760 align:middle line:84% to change the UI in real time in a whole bunch of applications, 00:04:01.760 --> 00:04:04.540 align:middle line:84% and you just don't even realize it. 00:04:04.540 --> 00:04:08.800 align:middle line:84% And of course, we all know about LLMs at this point. 00:04:08.800 --> 00:04:10.300 align:middle line:84% So I asked you to write a limerick 00:04:10.300 --> 00:04:13.027 align:middle line:84% about the beauty and power of deep learning yesterday. 00:04:13.027 --> 00:04:15.360 align:middle line:84% And it says, "In a world where data flows like a stream, 00:04:15.360 --> 00:04:16.839 align:middle line:84% deep learning is more than a dream. 00:04:16.839 --> 00:04:18.881 align:middle line:84% It sifts through the noise with an elegant poise, 00:04:18.881 --> 00:04:22.380 align:middle line:90% unveiling insights that gleam. 00:04:22.380 --> 00:04:23.660 align:middle line:90% Cool, right? 00:04:23.660 --> 00:04:25.980 align:middle line:84% All right, so let's get back to work. 00:04:25.980 --> 00:04:27.900 align:middle line:84% So NLP has extraordinary potential 00:04:27.900 --> 00:04:31.540 align:middle line:84% for making products and services much, much smarter. 00:04:31.540 --> 00:04:33.540 align:middle line:84% And what I want to point out here 00:04:33.540 --> 00:04:36.660 align:middle line:84% is that even if you focus on this very, very 00:04:36.660 --> 00:04:40.700 align:middle line:84% simple formalism, a bunch of text comes in, a bunch of text 00:04:40.700 --> 00:04:41.240 align:middle line:90% goes out. 00:04:41.240 --> 00:04:41.940 align:middle line:90% That's it. 00:04:41.940 --> 00:04:45.300 align:middle line:84% If you take that very simple text in, text out formalism, 00:04:45.300 --> 00:04:49.060 align:middle line:84% this little humble little thing has just an enormous, enormous 00:04:49.060 --> 00:04:50.780 align:middle line:90% range of applicability. 00:04:50.780 --> 00:04:53.920 align:middle line:84% So, obviously, you can send a bunch of text 00:04:53.920 --> 00:04:57.400 align:middle line:84% in and ask it to classify it, sentiment 00:04:57.400 --> 00:04:59.127 align:middle line:90% routed for customer support. 00:04:59.127 --> 00:05:00.960 align:middle line:84% You can try to figure out the intent of what 00:05:00.960 --> 00:05:02.580 align:middle line:90% the person is asking in search. 00:05:02.580 --> 00:05:03.330 align:middle line:90% You can filter it. 00:05:03.330 --> 00:05:05.122 align:middle line:84% You can content filter to make sure there's 00:05:05.122 --> 00:05:06.540 align:middle line:84% no toxic, abusive stuff going on. 00:05:06.540 --> 00:05:09.200 align:middle line:84% I mean, the possibilities for just text classification 00:05:09.200 --> 00:05:10.740 align:middle line:90% are numerous. 00:05:10.740 --> 00:05:14.060 align:middle line:84% But that's a use case we're all kind of familiar with. 00:05:14.060 --> 00:05:15.780 align:middle line:90% So no surprise there. 00:05:15.780 --> 00:05:18.660 align:middle line:84% Now, text extraction we may be less familiar with here. 00:05:18.660 --> 00:05:20.360 align:middle line:84% And the idea is that you can actually 00:05:20.360 --> 00:05:23.400 align:middle line:84% look at a lot of unstructured textual data 00:05:23.400 --> 00:05:27.540 align:middle line:84% and extract all sorts of interesting entities from it. 00:05:27.540 --> 00:05:29.100 align:middle line:90% Hedge funds use it very heavily. 00:05:29.100 --> 00:05:31.183 align:middle line:84% They will extract all sorts of company information 00:05:31.183 --> 00:05:32.360 align:middle line:90% from news articles. 00:05:32.360 --> 00:05:34.255 align:middle line:84% And then obviously doctor's notes. 00:05:34.255 --> 00:05:35.880 align:middle line:84% There are a whole bunch of NLP startups 00:05:35.880 --> 00:05:38.900 align:middle line:84% that will take the doctor-patient conversation, 00:05:38.900 --> 00:05:42.040 align:middle line:84% transcribe it, and then extract disease codes, diagnosis 00:05:42.040 --> 00:05:44.520 align:middle line:84% codes, medication codes, and things like that. 00:05:44.520 --> 00:05:46.960 align:middle line:84% So the possibilities for this are enormous. 00:05:46.960 --> 00:05:48.640 align:middle line:84% Of course, text summarization, and we 00:05:48.640 --> 00:05:50.720 align:middle line:84% all have been doing it thanks to ChatGPT. 00:05:50.720 --> 00:05:53.330 align:middle line:84% Take text in, and any kind of summary 00:05:53.330 --> 00:05:55.290 align:middle line:84% that comes out of the text, it's just text out. 00:05:55.290 --> 00:05:58.030 align:middle line:84% And then text generation, of course. 00:05:58.030 --> 00:06:00.430 align:middle line:84% We can take text and do marketing, copy sales, emails, 00:06:00.430 --> 00:06:01.730 align:middle line:84% market summaries, so on and so forth, 00:06:01.730 --> 00:06:03.050 align:middle line:84% including, troublingly for educators, 00:06:03.050 --> 00:06:04.175 align:middle line:90% college application essays. 00:06:04.175 --> 00:06:06.850 align:middle line:90% 00:06:06.850 --> 00:06:11.650 align:middle line:84% Code generation is a more subtle example of text 00:06:11.650 --> 00:06:14.810 align:middle line:90% out because code is just text. 00:06:14.810 --> 00:06:20.090 align:middle line:84% So text in, text out also covers text in, code out 00:06:20.090 --> 00:06:21.430 align:middle line:90% and question answering. 00:06:21.430 --> 00:06:25.090 align:middle line:84% So you can take a bunch of text, you can take a whole bunch 00:06:25.090 --> 00:06:27.437 align:middle line:84% of documents, you can add a bit of text to it, 00:06:27.437 --> 00:06:29.770 align:middle line:84% which is your question-- and this whole thing at the end 00:06:29.770 --> 00:06:31.070 align:middle line:90% of the day is just text-- 00:06:31.070 --> 00:06:33.315 align:middle line:84% send it in, and then you can use it 00:06:33.315 --> 00:06:34.690 align:middle line:84% to answer questions and therefore 00:06:34.690 --> 00:06:37.270 align:middle line:84% create chatbots for all sorts of interesting applications. 00:06:37.270 --> 00:06:39.810 align:middle line:90% 00:06:39.810 --> 00:06:43.370 align:middle line:84% And if you look at this example, call centers, 00:06:43.370 --> 00:06:46.410 align:middle line:84% that is where a lot of money is being spent right now to build 00:06:46.410 --> 00:06:48.050 align:middle line:84% these call center chatbots for text in, 00:06:48.050 --> 00:06:49.490 align:middle line:90% text out question answering. 00:06:49.490 --> 00:06:51.910 align:middle line:84% And so just if you drill into this, 00:06:51.910 --> 00:06:55.190 align:middle line:84% if you imagine taking all the call center transcripts 00:06:55.190 --> 00:06:57.750 align:middle line:84% and their internal product documentation, service 00:06:57.750 --> 00:07:00.950 align:middle line:84% documentation, FAQs, et cetera, stick it in, 00:07:00.950 --> 00:07:03.130 align:middle line:84% you can start to answer these kinds of questions. 00:07:03.130 --> 00:07:05.030 align:middle line:84% OK, yesterday, what are the top reasons why 00:07:05.030 --> 00:07:06.870 align:middle line:90% customers were upset with us? 00:07:06.870 --> 00:07:09.570 align:middle line:84% What interventions made by the agent actually worked? 00:07:09.570 --> 00:07:11.390 align:middle line:90% What did not work? 00:07:11.390 --> 00:07:13.938 align:middle line:84% What characterizes the best agents from the rest? 00:07:13.938 --> 00:07:16.230 align:middle line:84% How should we grade this particular agent's interaction 00:07:16.230 --> 00:07:17.750 align:middle line:90% with this particular customer? 00:07:17.750 --> 00:07:20.090 align:middle line:84% How should we change the call center script? 00:07:20.090 --> 00:07:22.910 align:middle line:84% How should we coach the agent in real time? 00:07:22.910 --> 00:07:25.590 align:middle line:84% Every one of these applications is amenable to this very humble 00:07:25.590 --> 00:07:29.340 align:middle line:90% text in, text out model. 00:07:29.340 --> 00:07:32.550 align:middle line:84% And of course, everybody knows this potential 00:07:32.550 --> 00:07:35.590 align:middle line:84% because of the advent of large language models. 00:07:35.590 --> 00:07:39.550 align:middle line:84% By the way, Google released something called Google Gemini 00:07:39.550 --> 00:07:46.870 align:middle line:84% 1.5 Pro a couple of days ago, and it's incredible. 00:07:46.870 --> 00:07:47.650 align:middle line:90% It's incredible. 00:07:47.650 --> 00:07:50.260 align:middle line:84% And anyway, we'll get back to that later. 00:07:50.260 --> 00:07:52.480 align:middle line:84% But the point is that the kind of potential we have 00:07:52.480 --> 00:07:54.800 align:middle line:84% is just amazing, even for text in, text out. 00:07:54.800 --> 00:07:56.660 align:middle line:90% And as you would imagine-- 00:07:56.660 --> 00:08:00.472 align:middle line:90% 00:08:00.472 --> 00:08:02.680 align:middle line:84% AUDIENCE: This is all, like, though we are calling it 00:08:02.680 --> 00:08:05.707 align:middle line:84% language, this is all primarily English. 00:08:05.707 --> 00:08:08.040 align:middle line:84% RAMA RAMAKRISHNAN: There are lots of multilingual models 00:08:08.040 --> 00:08:08.660 align:middle line:90% as well. 00:08:08.660 --> 00:08:10.400 align:middle line:90% There are multilingual models. 00:08:10.400 --> 00:08:13.425 align:middle line:84% By that, I mean models which are specialized to other languages, 00:08:13.425 --> 00:08:14.800 align:middle line:84% non-English languages, and models 00:08:14.800 --> 00:08:18.180 align:middle line:84% which are truly multilingual, like polyglot models as well. 00:08:18.180 --> 00:08:20.660 align:middle line:84% And both of them are available right now. 00:08:20.660 --> 00:08:24.720 align:middle line:84% And many, many modern LLMs are actually trained from the get-go 00:08:24.720 --> 00:08:27.240 align:middle line:84% to be multilingual in a bunch of what 00:08:27.240 --> 00:08:29.440 align:middle line:84% are called high-resource languages, languages which 00:08:29.440 --> 00:08:30.909 align:middle line:90% are spoken by lots of people. 00:08:30.909 --> 00:08:33.159 align:middle line:84% But actually, it's funny you should ask that question, 00:08:33.159 --> 00:08:35.240 align:middle line:84% because this Google Gemini model that I just 00:08:35.240 --> 00:08:37.440 align:middle line:90% described, they actually-- 00:08:37.440 --> 00:08:39.960 align:middle line:84% so there is a language called Kalimpong, 00:08:39.960 --> 00:08:43.080 align:middle line:84% which is spoken by 200 people in the world. 00:08:43.080 --> 00:08:47.220 align:middle line:84% And so a researcher had created one book, 00:08:47.220 --> 00:08:49.860 align:middle line:84% which is sort of like a grammar manual for Kalimpong 00:08:49.860 --> 00:08:53.142 align:middle line:84% because there are no other written works in that language. 00:08:53.142 --> 00:08:55.100 align:middle line:84% And so what they did is they took a whole bunch 00:08:55.100 --> 00:08:58.540 align:middle line:84% of English dialogue and this book, 00:08:58.540 --> 00:09:03.420 align:middle line:84% fed it into a Google Gemini Pro 1.5, 00:09:03.420 --> 00:09:05.300 align:middle line:84% and it translated it into Kalimpong 00:09:05.300 --> 00:09:07.660 align:middle line:90% at human level proficiency. 00:09:07.660 --> 00:09:09.820 align:middle line:90% It had never seen it before. 00:09:09.820 --> 00:09:15.620 align:middle line:90% So that's an example of this. 00:09:15.620 --> 00:09:16.540 align:middle line:90% Yes. 00:09:16.540 --> 00:09:19.420 align:middle line:84% So the question text here is all the things 00:09:19.420 --> 00:09:21.460 align:middle line:84% you want to translate from English to Kalimpong. 00:09:21.460 --> 00:09:24.020 align:middle line:84% The documents here is just one document, singular, 00:09:24.020 --> 00:09:25.700 align:middle line:90% the grammar book, the manual. 00:09:25.700 --> 00:09:28.340 align:middle line:84% And then what comes out is a translation. 00:09:28.340 --> 00:09:30.380 align:middle line:84% So these models, even when they're not 00:09:30.380 --> 00:09:32.940 align:middle line:84% explicitly trained on a different language, if you give 00:09:32.940 --> 00:09:35.920 align:middle line:84% them enough of grammar manuals and stuff like that, 00:09:35.920 --> 00:09:38.220 align:middle line:84% they may do a pretty decent job from the get-go 00:09:38.220 --> 00:09:40.460 align:middle line:90% with no training. 00:09:40.460 --> 00:09:41.820 align:middle line:90% It's kind of a shocker. 00:09:41.820 --> 00:09:44.820 align:middle line:84% Two years ago, people would be like, that's impossible. 00:09:44.820 --> 00:09:50.570 align:middle line:90% All right, so back to this. 00:09:50.570 --> 00:09:52.230 align:middle line:84% And as you folks may already know, 00:09:52.230 --> 00:09:54.647 align:middle line:84% and maybe you're, in fact, participating in this gold rush 00:09:54.647 --> 00:09:57.710 align:middle line:84% already, lots of people are creating 00:09:57.710 --> 00:10:00.272 align:middle line:84% lots of really cool companies to take some of these ideas 00:10:00.272 --> 00:10:02.230 align:middle line:84% and actually create really interesting products 00:10:02.230 --> 00:10:04.110 align:middle line:90% and services out of them. 00:10:04.110 --> 00:10:06.190 align:middle line:84% So if you're not doing it, and if you've 00:10:06.190 --> 00:10:08.510 align:middle line:84% been thinking about entrepreneurial stuff, 00:10:08.510 --> 00:10:09.553 align:middle line:90% here's a word of advice-- 00:10:09.553 --> 00:10:10.220 align:middle line:90% take the plunge. 00:10:10.220 --> 00:10:15.110 align:middle line:90% 00:10:15.110 --> 00:10:15.630 align:middle line:90% Dismissed. 00:10:15.630 --> 00:10:17.490 align:middle line:90% I'm just kidding. 00:10:17.490 --> 00:10:20.390 align:middle line:84% And as you can imagine, enterprise vendors 00:10:20.390 --> 00:10:23.130 align:middle line:84% are rushing to add NLP to all their products. 00:10:23.130 --> 00:10:25.750 align:middle line:84% Salesforce-- Einstein now has Einstein GPT, 00:10:25.750 --> 00:10:27.730 align:middle line:90% Microsoft has Copilot. 00:10:27.730 --> 00:10:29.250 align:middle line:90% I mean, the list goes on. 00:10:29.250 --> 00:10:31.150 align:middle line:84% Everybody's scrambling and really 00:10:31.150 --> 00:10:33.110 align:middle line:84% trying hard to infuse some GPT magic 00:10:33.110 --> 00:10:34.510 align:middle line:90% into whatever they're doing. 00:10:34.510 --> 00:10:37.590 align:middle line:84% Some of it is real, a lot of it is not. 00:10:37.590 --> 00:10:40.890 align:middle line:84% OK, so let's go to the arc of NLP progress. 00:10:40.890 --> 00:10:44.810 align:middle line:84% How did we get to this kind of crazy times that we live in? 00:10:44.810 --> 00:10:48.000 align:middle line:84% So if you look at natural language processing, basically 00:10:48.000 --> 00:10:51.570 align:middle line:84% the efforts to take language and try to analyze language and do 00:10:51.570 --> 00:10:56.210 align:middle line:84% predictions with language and so on and so forth, 00:10:56.210 --> 00:10:58.890 align:middle line:84% the first phase of it was just hand-crafted rules 00:10:58.890 --> 00:11:00.210 align:middle line:90% based on linguistics. 00:11:00.210 --> 00:11:02.130 align:middle line:84% So these were all linguists who would really 00:11:02.130 --> 00:11:03.792 align:middle line:84% understand the grammar of a language, 00:11:03.792 --> 00:11:06.250 align:middle line:84% and then they would use their deep knowledge of linguistics 00:11:06.250 --> 00:11:08.330 align:middle line:84% to figure out all these rules by which you 00:11:08.330 --> 00:11:11.610 align:middle line:84% can process and analyze natural language text. 00:11:11.610 --> 00:11:13.717 align:middle line:84% And then this other thing came along, 00:11:13.717 --> 00:11:16.050 align:middle line:84% which was a statistical machine learning approach, which 00:11:16.050 --> 00:11:19.250 align:middle line:84% basically said, never mind all that complicated knowledge 00:11:19.250 --> 00:11:20.750 align:middle line:90% of linguistics and grammar. 00:11:20.750 --> 00:11:23.217 align:middle line:84% Why don't we simply count things? 00:11:23.217 --> 00:11:25.550 align:middle line:84% Let's count the number of times these two will co-occur. 00:11:25.550 --> 00:11:27.350 align:middle line:84% Now, let's count that, let's count this. 00:11:27.350 --> 00:11:28.730 align:middle line:90% Basically, just count a lot. 00:11:28.730 --> 00:11:32.490 align:middle line:84% And let's see if it does, if it does for predicting things, 00:11:32.490 --> 00:11:34.450 align:middle line:84% say for classifying text and so on. 00:11:34.450 --> 00:11:38.137 align:middle line:84% And shockingly, those methods ended up being really good. 00:11:38.137 --> 00:11:39.470 align:middle line:90% They ended up being really good. 00:11:39.470 --> 00:11:43.150 align:middle line:84% And in fact, they actually were better than the lovingly 00:11:43.150 --> 00:11:46.710 align:middle line:84% hand-curated, linguistically-driven rules, 00:11:46.710 --> 00:11:50.290 align:middle line:84% so much so this is a famous quote which says, 00:11:50.290 --> 00:11:52.790 align:middle line:84% "Every time I fire a linguist, the performance of the speech 00:11:52.790 --> 00:11:55.510 align:middle line:90% recognizer goes up." 00:11:55.510 --> 00:11:57.190 align:middle line:84% Obviously made in jest, but there 00:11:57.190 --> 00:11:59.710 align:middle line:90% is a kernel of truth to it. 00:11:59.710 --> 00:12:02.410 align:middle line:90% So that's where we were. 00:12:02.410 --> 00:12:07.052 align:middle line:84% And then deep learning happened in 2012, roughly. 00:12:07.052 --> 00:12:08.510 align:middle line:84% And then we had these things called 00:12:08.510 --> 00:12:10.010 align:middle line:84% recurrent neural networks, which are 00:12:10.010 --> 00:12:13.190 align:middle line:84% based on deep learning, which actually moved the ball forward. 00:12:13.190 --> 00:12:16.830 align:middle line:84% And then in 2017, something called the transformer 00:12:16.830 --> 00:12:20.150 align:middle line:90% was invented, 2017. 00:12:20.150 --> 00:12:23.390 align:middle line:84% And the transformer replaced everything else 00:12:23.390 --> 00:12:24.750 align:middle line:90% across the board. 00:12:24.750 --> 00:12:27.310 align:middle line:84% So we're just going to leapfrog directly 00:12:27.310 --> 00:12:28.477 align:middle line:90% to transformers in HODL. 00:12:28.477 --> 00:12:30.810 align:middle line:84% We will not spend any time on recurrent neural networks. 00:12:30.810 --> 00:12:34.510 align:middle line:84% And that is not to say that they're dead. 00:12:34.510 --> 00:12:36.430 align:middle line:84% There's a very interesting work which actually 00:12:36.430 --> 00:12:38.590 align:middle line:84% is trying to now revive recurrent neural networks 00:12:38.590 --> 00:12:42.180 align:middle line:84% to make it work for these kinds of modern LLM kinds of tasks, 00:12:42.180 --> 00:12:43.960 align:middle line:90% but it's still very early days. 00:12:43.960 --> 00:12:46.560 align:middle line:84% So for now, we'll just focus on transformers. 00:12:46.560 --> 00:12:50.080 align:middle line:84% So the very high-level view of the problem 00:12:50.080 --> 00:12:53.020 align:middle line:84% here is that, like most things in deep learning, 00:12:53.020 --> 00:12:55.560 align:middle line:90% it's basically fancy regression. 00:12:55.560 --> 00:12:57.660 align:middle line:84% There is some variable, x, that comes in. 00:12:57.660 --> 00:13:01.320 align:middle line:84% It goes through this very complicated function 00:13:01.320 --> 00:13:03.640 align:middle line:84% along with this w, which is the weights, 00:13:03.640 --> 00:13:05.300 align:middle line:90% and then out pops an output. 00:13:05.300 --> 00:13:07.600 align:middle line:84% That's just the view that you've always had. 00:13:07.600 --> 00:13:10.940 align:middle line:84% And so in this case, x happens to be text. 00:13:10.940 --> 00:13:11.868 align:middle line:90% y-hat can be text. 00:13:11.868 --> 00:13:12.660 align:middle line:90% It could be labels. 00:13:12.660 --> 00:13:13.260 align:middle line:90% It could be numbers. 00:13:13.260 --> 00:13:14.220 align:middle line:90% It could be anything else. 00:13:14.220 --> 00:13:15.803 align:middle line:84% The w is the weights, and the function 00:13:15.803 --> 00:13:18.348 align:middle line:90% is a deep neural network. 00:13:18.348 --> 00:13:20.140 align:middle line:84% At this point, when you look at this slide, 00:13:20.140 --> 00:13:23.360 align:middle line:90% it should be blindingly obvious. 00:13:23.360 --> 00:13:26.240 align:middle line:84% So now, the key question here is, how do you actually 00:13:26.240 --> 00:13:27.920 align:middle line:90% represent x? 00:13:27.920 --> 00:13:29.520 align:middle line:90% That's the key question. 00:13:29.520 --> 00:13:32.400 align:middle line:84% For pictures, for images, we saw that we just 00:13:32.400 --> 00:13:35.280 align:middle line:84% took the pixel values, which were light intensity 00:13:35.280 --> 00:13:37.200 align:middle line:84% numbers between 0 and 255, and you could just 00:13:37.200 --> 00:13:38.340 align:middle line:90% use that directly. 00:13:38.340 --> 00:13:41.680 align:middle line:84% But when a sentence comes in, like, "I love deep learning", 00:13:41.680 --> 00:13:42.785 align:middle line:90% what do you do? 00:13:42.785 --> 00:13:44.160 align:middle line:84% How do you actually represent it? 00:13:44.160 --> 00:13:46.202 align:middle line:84% Because remember, we have to numerical everything 00:13:46.202 --> 00:13:47.700 align:middle line:90% that's coming in. 00:13:47.700 --> 00:13:48.620 align:middle line:90% So that's the key. 00:13:48.620 --> 00:13:51.420 align:middle line:84% And this actually is a very subtle question, very important 00:13:51.420 --> 00:13:54.260 align:middle line:84% question, and we'll focus on that today. 00:13:54.260 --> 00:13:57.980 align:middle line:84% And then next week, when we look at transformers, 00:13:57.980 --> 00:14:00.580 align:middle line:84% we will look at what neural network architecture is best 00:14:00.580 --> 00:14:04.000 align:middle line:84% suited to process these text inputs that are coming in. 00:14:04.000 --> 00:14:07.260 align:middle line:84% Those are the two big questions we're going to look at. 00:14:07.260 --> 00:14:11.048 align:middle line:90% All right, so processing basics. 00:14:11.048 --> 00:14:13.090 align:middle line:84% We're going to follow this very standard process. 00:14:13.090 --> 00:14:15.860 align:middle line:90% 00:14:15.860 --> 00:14:20.380 align:middle line:84% This is the process by which we take any text that comes in, 00:14:20.380 --> 00:14:22.680 align:middle line:84% and we do run it through these four steps. 00:14:22.680 --> 00:14:25.107 align:middle line:84% And this process is called text vectorization. 00:14:25.107 --> 00:14:26.940 align:middle line:84% And as the name suggests, we are essentially 00:14:26.940 --> 00:14:29.820 align:middle line:84% taking text and creating vectors of numbers out of them. 00:14:29.820 --> 00:14:31.420 align:middle line:90% Text vectorization. 00:14:31.420 --> 00:14:33.420 align:middle line:84% And we'll go through each of these processes one 00:14:33.420 --> 00:14:34.500 align:middle line:90% after the other. 00:14:34.500 --> 00:14:38.510 align:middle line:84% So I just find it very useful to just have this acronym S-T-I-E 00:14:38.510 --> 00:14:41.210 align:middle line:84% in my head, like "stie", S-T-I-E. Just keep that in mind. 00:14:41.210 --> 00:14:42.830 align:middle line:90% It may be helpful. 00:14:42.830 --> 00:14:46.110 align:middle line:84% All right, so what we do is, the setup 00:14:46.110 --> 00:14:49.650 align:middle line:84% here is that we have a whole bunch of documents. 00:14:49.650 --> 00:14:51.290 align:middle line:90% We call it the training corpus. 00:14:51.290 --> 00:14:54.132 align:middle line:84% We have a whole bunch of text documents, text data. 00:14:54.132 --> 00:14:55.590 align:middle line:84% And as far as we are concerned, you 00:14:55.590 --> 00:14:59.770 align:middle line:84% can just imagine it as just lists of long passages. 00:14:59.770 --> 00:15:00.970 align:middle line:90% What is a novel? 00:15:00.970 --> 00:15:03.090 align:middle line:84% It's just a long passage of text. 00:15:03.090 --> 00:15:06.810 align:middle line:84% So whether it's a novel or a sentence doesn't really matter. 00:15:06.810 --> 00:15:08.270 align:middle line:84% We just think of them as a big list 00:15:08.270 --> 00:15:10.250 align:middle line:90% of strings, a big list of text. 00:15:10.250 --> 00:15:12.110 align:middle line:90% That's our training corpus. 00:15:12.110 --> 00:15:14.590 align:middle line:84% And what we do is we take this training corpus 00:15:14.590 --> 00:15:16.630 align:middle line:84% and we run it through, and we apply 00:15:16.630 --> 00:15:18.330 align:middle line:84% standardization and tokenization, 00:15:18.330 --> 00:15:22.390 align:middle line:84% which I will describe, to this entire training corpus upfront. 00:15:22.390 --> 00:15:25.630 align:middle line:90% So we first do this. 00:15:25.630 --> 00:15:29.750 align:middle line:84% And standardization is basically-- 00:15:29.750 --> 00:15:32.190 align:middle line:84% the default for most applications 00:15:32.190 --> 00:15:35.230 align:middle line:84% tends to be this, which is we first strip capitalization 00:15:35.230 --> 00:15:38.610 align:middle line:84% and make everything lowercase, and then 00:15:38.610 --> 00:15:41.750 align:middle line:84% we remove punctuation and accents and so on and so forth. 00:15:41.750 --> 00:15:44.050 align:middle line:90% That's the first thing we do. 00:15:44.050 --> 00:15:46.470 align:middle line:84% I'll talk about why we do it in just a moment. 00:15:46.470 --> 00:15:49.410 align:middle line:84% But the mechanics of it are, we do this first, 00:15:49.410 --> 00:15:53.270 align:middle line:84% then we look at words like a, the, it, and so on and so forth, 00:15:53.270 --> 00:15:57.570 align:middle line:84% basically filler words, which we need to actually make 00:15:57.570 --> 00:15:59.570 align:middle line:84% complete sentences, but they may not have 00:15:59.570 --> 00:16:01.850 align:middle line:90% any value in predicting things. 00:16:01.850 --> 00:16:04.350 align:middle line:84% So we remove them, and they are called stop words. 00:16:04.350 --> 00:16:07.510 align:middle line:84% And then finally, we take words which are very similar, 00:16:07.510 --> 00:16:10.070 align:middle line:84% which have sort of the same kind of stem or root, 00:16:10.070 --> 00:16:13.410 align:middle line:84% and then we just map it to a common representation, 00:16:13.410 --> 00:16:16.210 align:middle line:84% like ate, eaten, eating, eaten, all these things will just 00:16:16.210 --> 00:16:18.210 align:middle line:90% become, let's say, eats. 00:16:18.210 --> 00:16:19.510 align:middle line:90% And we do that sometimes. 00:16:19.510 --> 00:16:21.210 align:middle line:90% So this, we almost always do. 00:16:21.210 --> 00:16:22.230 align:middle line:90% This, we often do. 00:16:22.230 --> 00:16:23.510 align:middle line:90% And this, we do it sometimes. 00:16:23.510 --> 00:16:25.950 align:middle line:84% Now, why do we do any of these things? 00:16:25.950 --> 00:16:34.490 align:middle line:90% 00:16:34.490 --> 00:16:37.780 align:middle line:84% AUDIENCE: I think we want to try to recognize the essential thing 00:16:37.780 --> 00:16:40.240 align:middle line:84% with the word, whether it's eaten or eat, 00:16:40.240 --> 00:16:42.580 align:middle line:84% but the essential thing is the eat part. 00:16:42.580 --> 00:16:46.220 align:middle line:84% So we want to try to abstract from it the 00:16:46.220 --> 00:16:47.167 align:middle line:90% more essential thing. 00:16:47.167 --> 00:16:49.500 align:middle line:84% RAMA RAMAKRISHNAN: Right, so why do we need to abstract? 00:16:49.500 --> 00:16:50.600 align:middle line:90% You're absolutely correct. 00:16:50.600 --> 00:16:51.700 align:middle line:90% We're trying to abstract. 00:16:51.700 --> 00:16:54.200 align:middle line:84% Why is there a benefit to doing this abstraction? 00:16:54.200 --> 00:16:57.723 align:middle line:90% 00:16:57.723 --> 00:16:59.640 align:middle line:84% How about somebody from this side of the room? 00:16:59.640 --> 00:17:02.020 align:middle line:90% Oh, yes. 00:17:02.020 --> 00:17:06.593 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:17:06.593 --> 00:17:08.260 align:middle line:84% RAMA RAMAKRISHNAN: Why is it a good idea 00:17:08.260 --> 00:17:10.700 align:middle line:84% to reduce the size of the library? 00:17:10.700 --> 00:17:12.220 align:middle line:84% AUDIENCE: To reduce the computation. 00:17:12.220 --> 00:17:13.887 align:middle line:84% RAMA RAMAKRISHNAN: Because of the amount 00:17:13.887 --> 00:17:15.500 align:middle line:90% of computation needed. 00:17:15.500 --> 00:17:17.460 align:middle line:90% So, that is part of the answer. 00:17:17.460 --> 00:17:21.099 align:middle line:84% There's another part to the answer, which is? 00:17:21.099 --> 00:17:22.599 align:middle line:84% All right, let's swing to the right. 00:17:22.599 --> 00:17:26.708 align:middle line:90% 00:17:26.708 --> 00:17:28.500 align:middle line:84% AUDIENCE: Because it facilitates comparison 00:17:28.500 --> 00:17:31.840 align:middle line:84% between different sets if you are able to standardize it. 00:17:31.840 --> 00:17:34.440 align:middle line:90% 00:17:34.440 --> 00:17:36.680 align:middle line:84% RAMA RAMAKRISHNAN: OK, so I will go with that. 00:17:36.680 --> 00:17:40.560 align:middle line:84% But I think the key thing to realize here 00:17:40.560 --> 00:17:43.100 align:middle line:90% is that you want the model-- 00:17:43.100 --> 00:17:45.820 align:middle line:84% much like when we talk about computer vision, we said, 00:17:45.820 --> 00:17:48.240 align:middle line:84% look, if it's a vertical line, I want 00:17:48.240 --> 00:17:50.720 align:middle line:84% to be able to detect it wherever it happens. 00:17:50.720 --> 00:17:53.080 align:middle line:84% I don't want the model to think that the vertical line 00:17:53.080 --> 00:17:54.800 align:middle line:84% on the left side is different from the vertical line 00:17:54.800 --> 00:17:56.800 align:middle line:84% on the right side and then later realize 00:17:56.800 --> 00:17:59.200 align:middle line:84% they are the same thing, because you would have wasted 00:17:59.200 --> 00:18:01.800 align:middle line:84% valuable capacity learning things which actually happen 00:18:01.800 --> 00:18:04.600 align:middle line:84% to be the same because you didn't know it was the same. 00:18:04.600 --> 00:18:08.000 align:middle line:84% So here, if you, for example, take a word and lowercase 00:18:08.000 --> 00:18:10.440 align:middle line:84% it, clearly the case of it, whether it's 00:18:10.440 --> 00:18:12.283 align:middle line:84% uppercase or lowercase, most of the time, 00:18:12.283 --> 00:18:14.700 align:middle line:84% it's not going to matter for anything you want to predict. 00:18:14.700 --> 00:18:16.760 align:middle line:84% So you're essentially telling the model, 00:18:16.760 --> 00:18:18.780 align:middle line:84% the lowercase version, the uppercase version, 00:18:18.780 --> 00:18:19.738 align:middle line:90% they are not different. 00:18:19.738 --> 00:18:20.948 align:middle line:90% They're actually the same. 00:18:20.948 --> 00:18:23.240 align:middle line:84% And the easiest way to tell the model they are the same 00:18:23.240 --> 00:18:25.600 align:middle line:84% is just make everything lowercase. 00:18:25.600 --> 00:18:28.132 align:middle line:90% That is the key idea. 00:18:28.132 --> 00:18:29.968 align:middle line:84% And similarly, if you look at stop words, 00:18:29.968 --> 00:18:31.760 align:middle line:84% the reason is that these stop words may not 00:18:31.760 --> 00:18:32.850 align:middle line:90% help you predict anything. 00:18:32.850 --> 00:18:36.130 align:middle line:84% Whether a word a, and, the showed up in a movie review 00:18:36.130 --> 00:18:39.105 align:middle line:84% probably does not affect the sentiment of the review. 00:18:39.105 --> 00:18:40.230 align:middle line:90% Therefore, let's remove it. 00:18:40.230 --> 00:18:42.170 align:middle line:84% So that's a slightly different reason. 00:18:42.170 --> 00:18:44.390 align:middle line:84% Stemming is the same reason as the first one, 00:18:44.390 --> 00:18:47.470 align:middle line:84% which is that all these words kind of mean the same thing. 00:18:47.470 --> 00:18:49.550 align:middle line:84% We don't have to be super precise about it, 00:18:49.550 --> 00:18:51.810 align:middle line:84% and so let's just collapse them onto the same thing. 00:18:51.810 --> 00:18:55.130 align:middle line:84% Now, these are all the standard things we do. 00:18:55.130 --> 00:18:58.950 align:middle line:84% There are important exceptions to all these things. 00:18:58.950 --> 00:19:01.550 align:middle line:84% And we'll come back to the exceptions a bit later. 00:19:01.550 --> 00:19:03.730 align:middle line:84% But that is the standard thing we do. 00:19:03.730 --> 00:19:05.170 align:middle line:90% Make sense? 00:19:05.170 --> 00:19:05.750 align:middle line:90% All right. 00:19:05.750 --> 00:19:08.650 align:middle line:90% 00:19:08.650 --> 00:19:13.574 align:middle line:84% So if you look at something like this sentence here, "Hola! 00:19:13.574 --> 00:19:16.116 align:middle line:84% What do you picture when you think of travel to Mexico", bah, 00:19:16.116 --> 00:19:17.810 align:middle line:84% bah, boom, and then you can see here, 00:19:17.810 --> 00:19:19.890 align:middle line:84% this is the standardized version, 00:19:19.890 --> 00:19:21.570 align:middle line:90% everything has become lowercase. 00:19:21.570 --> 00:19:23.170 align:middle line:90% The H has become small h. 00:19:23.170 --> 00:19:24.830 align:middle line:90% The punctuation has disappeared. 00:19:24.830 --> 00:19:26.370 align:middle line:90% That's part of standardization. 00:19:26.370 --> 00:19:29.170 align:middle line:90% 00:19:29.170 --> 00:19:33.430 align:middle line:84% And you can see here that Mexico m has become small, 00:19:33.430 --> 00:19:36.990 align:middle line:84% sipping has become sips, think has become 00:19:36.990 --> 00:19:38.990 align:middle line:90% thinks, and so on and so forth. 00:19:38.990 --> 00:19:41.690 align:middle line:84% So that's an example of standardization at work. 00:19:41.690 --> 00:19:49.000 align:middle line:90% 00:19:49.000 --> 00:19:51.510 align:middle line:84% The next thing we do is something very important, 00:19:51.510 --> 00:19:53.798 align:middle line:90% and it's called tokenization. 00:19:53.798 --> 00:19:55.590 align:middle line:84% So what we do typically is that, OK, now we 00:19:55.590 --> 00:19:56.798 align:middle line:90% have standardized everything. 00:19:56.798 --> 00:19:58.810 align:middle line:90% We have a bunch of words. 00:19:58.810 --> 00:20:01.870 align:middle line:84% We need to now split them into what are called tokens. 00:20:01.870 --> 00:20:07.110 align:middle line:84% So the most common default is to just think of a word as a token. 00:20:07.110 --> 00:20:09.290 align:middle line:90% We just split on the whitespace. 00:20:09.290 --> 00:20:11.950 align:middle line:84% You take each string, and wherever there is whitespace, 00:20:11.950 --> 00:20:15.430 align:middle line:84% meaning actual spaces or carriage returns and things 00:20:15.430 --> 00:20:17.270 align:middle line:84% like that, boom, you just split on them, 00:20:17.270 --> 00:20:19.110 align:middle line:84% and you just create words out of it. 00:20:19.110 --> 00:20:23.770 align:middle line:84% So, for instance, if you have this standardized sentence here, 00:20:23.770 --> 00:20:27.890 align:middle line:84% you just split it after every word, and you get this thing. 00:20:27.890 --> 00:20:29.770 align:middle line:90% So each of these is now a token. 00:20:29.770 --> 00:20:32.840 align:middle line:90% 00:20:32.840 --> 00:20:36.160 align:middle line:84% Now, this has some disadvantages. 00:20:36.160 --> 00:20:38.760 align:middle line:84% What are some disadvantages of just splitting 00:20:38.760 --> 00:20:41.080 align:middle line:90% on the space between words? 00:20:41.080 --> 00:20:43.920 align:middle line:90% Yeah. 00:20:43.920 --> 00:20:46.240 align:middle line:84% AUDIENCE: I think we lose any context because we 00:20:46.240 --> 00:20:49.000 align:middle line:90% look at each word separately. 00:20:49.000 --> 00:20:52.318 align:middle line:84% So we don't have any passwords or what's happened [INAUDIBLE.] 00:20:52.318 --> 00:20:53.360 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:20:53.360 --> 00:20:55.760 align:middle line:84% So, for example, "the cat sat on the mat" 00:20:55.760 --> 00:21:00.040 align:middle line:84% and "the mat sat on the cat" will have the same set. 00:21:00.040 --> 00:21:01.722 align:middle line:90% Yeah, so you lose the order. 00:21:01.722 --> 00:21:03.180 align:middle line:84% What are some other issues with it? 00:21:03.180 --> 00:21:05.190 align:middle line:90% 00:21:05.190 --> 00:21:07.440 align:middle line:84% AUDIENCE: For words that should have chained together, 00:21:07.440 --> 00:21:09.400 align:middle line:84% like Puerto Vallarta, you lose the fact 00:21:09.400 --> 00:21:11.700 align:middle line:84% that that's one name because you separated it. 00:21:11.700 --> 00:21:13.200 align:middle line:84% RAMA RAMAKRISHNAN: Correct, exactly. 00:21:13.200 --> 00:21:16.300 align:middle line:84% So there are compound words, like father-in-law, 00:21:16.300 --> 00:21:17.320 align:middle line:90% for instance. 00:21:17.320 --> 00:21:18.340 align:middle line:90% That's one problem. 00:21:18.340 --> 00:21:21.240 align:middle line:84% Another problem is that lots of non-English languages, 00:21:21.240 --> 00:21:24.900 align:middle line:84% they actually don't have this notion of a space between words. 00:21:24.900 --> 00:21:27.740 align:middle line:84% It actually runs one after the other, 00:21:27.740 --> 00:21:29.860 align:middle line:84% and the native speakers know from context 00:21:29.860 --> 00:21:32.060 align:middle line:90% how to chunk it and break it. 00:21:32.060 --> 00:21:34.700 align:middle line:90% So, well, what do we do then? 00:21:34.700 --> 00:21:37.860 align:middle line:84% Because you basically will have one word for the whole passage, 00:21:37.860 --> 00:21:38.980 align:middle line:90% one token. 00:21:38.980 --> 00:21:41.740 align:middle line:84% The other problem is that there are languages-- 00:21:41.740 --> 00:21:43.940 align:middle line:84% German is perhaps the most notable one-- in which 00:21:43.940 --> 00:21:45.170 align:middle line:90% you have very long words. 00:21:45.170 --> 00:21:48.060 align:middle line:90% 00:21:48.060 --> 00:21:50.940 align:middle line:84% I saw a word, which I think I might have it 00:21:50.940 --> 00:21:53.220 align:middle line:84% in a slide somewhere, that's, like, this long, which 00:21:53.220 --> 00:21:59.760 align:middle line:84% means you realize that something amazing is happening, 00:21:59.760 --> 00:22:02.280 align:middle line:84% but the rest of the world hasn't woken up to it yet. 00:22:02.280 --> 00:22:04.460 align:middle line:90% It's that feeling. 00:22:04.460 --> 00:22:06.140 align:middle line:90% There's a word for that. 00:22:06.140 --> 00:22:07.900 align:middle line:90% Amazing, right? 00:22:07.900 --> 00:22:11.040 align:middle line:84% Anyway, so yeah, some words, in Japanese, for example, 00:22:11.040 --> 00:22:12.380 align:middle line:90% there's a word called komorebi. 00:22:12.380 --> 00:22:16.540 align:middle line:84% Do people know the meaning of the word komorebi? 00:22:16.540 --> 00:22:21.140 align:middle line:84% It means the transient beauty of sunlight 00:22:21.140 --> 00:22:24.700 align:middle line:90% going through fall foliage. 00:22:24.700 --> 00:22:26.060 align:middle line:90% There's a word for that. 00:22:26.060 --> 00:22:27.560 align:middle line:90% How cool is that? 00:22:27.560 --> 00:22:28.340 align:middle line:90% Anyway, sorry. 00:22:28.340 --> 00:22:29.220 align:middle line:90% I love that word. 00:22:29.220 --> 00:22:30.640 align:middle line:90% So, back to this. 00:22:30.640 --> 00:22:32.280 align:middle line:90% So we have this thing here. 00:22:32.280 --> 00:22:34.120 align:middle line:84% So there are reasons for which splitting 00:22:34.120 --> 00:22:36.220 align:middle line:84% on the space between words is not going to work. 00:22:36.220 --> 00:22:38.860 align:middle line:90% 00:22:38.860 --> 00:22:43.920 align:middle line:84% So what happens is that modern large language models-- 00:22:43.920 --> 00:22:47.060 align:middle line:84% so what we have described so far, despite its shortcomings, 00:22:47.060 --> 00:22:50.280 align:middle line:84% is actually really good for lots of NLP use cases. 00:22:50.280 --> 00:22:53.360 align:middle line:84% If you want to classify text, it's good enough, for instance. 00:22:53.360 --> 00:22:57.000 align:middle line:84% But if you want to generate text, like LLMs do, 00:22:57.000 --> 00:22:58.240 align:middle line:90% it's not going to work. 00:22:58.240 --> 00:22:59.380 align:middle line:90% It's not going to work. 00:22:59.380 --> 00:23:01.620 align:middle line:84% Because you know when you ask ChatGPT a question, 00:23:01.620 --> 00:23:03.760 align:middle line:84% it comes back with perfect punctuation. 00:23:03.760 --> 00:23:05.780 align:middle line:84% Clearly, punctuation was not stripped. 00:23:05.780 --> 00:23:07.988 align:middle line:84% It comes back with a particular upper and lower case. 00:23:07.988 --> 00:23:09.520 align:middle line:90% Clearly, that wasn't stripped. 00:23:09.520 --> 00:23:11.120 align:middle line:84% You can actually make up new words 00:23:11.120 --> 00:23:12.840 align:middle line:90% and ask it to use the new word. 00:23:12.840 --> 00:23:13.860 align:middle line:90% It'll use it. 00:23:13.860 --> 00:23:17.240 align:middle line:84% Therefore, it's not like it can only recognize a finite set. 00:23:17.240 --> 00:23:19.560 align:middle line:84% So there's a very clever scheme called 00:23:19.560 --> 00:23:24.500 align:middle line:84% byte-pair encoding, which is invented to do all those things. 00:23:24.500 --> 00:23:26.710 align:middle line:84% And I have slides at the end, and if we have time, 00:23:26.710 --> 00:23:28.710 align:middle line:90% we'll talk about it. 00:23:28.710 --> 00:23:30.770 align:middle line:84% For now, let's continue this thing. 00:23:30.770 --> 00:23:35.010 align:middle line:84% So when this is done for every sentence or every passage 00:23:35.010 --> 00:23:39.813 align:middle line:84% in a training data set, we now have a list of distinct tokens. 00:23:39.813 --> 00:23:41.230 align:middle line:84% We have a list of distinct tokens. 00:23:41.230 --> 00:23:43.730 align:middle line:84% In this simple case, it happens to be all the distinct words 00:23:43.730 --> 00:23:45.130 align:middle line:90% that we have seen. 00:23:45.130 --> 00:23:47.330 align:middle line:90% That's called the vocabulary. 00:23:47.330 --> 00:23:49.810 align:middle line:90% That's called the vocabulary. 00:23:49.810 --> 00:23:51.930 align:middle line:84% So now, we move to the third and fourth stages. 00:23:51.930 --> 00:23:54.830 align:middle line:84% In these stages, the indexing and encoding stage, 00:23:54.830 --> 00:23:58.010 align:middle line:84% we only work with the vocabulary. 00:23:58.010 --> 00:24:01.170 align:middle line:84% And so what we do is, the first thing, the indexing, 00:24:01.170 --> 00:24:04.070 align:middle line:84% we assign a unique integer to each distinct token 00:24:04.070 --> 00:24:05.330 align:middle line:90% in the vocabulary. 00:24:05.330 --> 00:24:07.730 align:middle line:84% So, for instance, let's say that you 00:24:07.730 --> 00:24:10.770 align:middle line:84% took a whole bunch of English literature 00:24:10.770 --> 00:24:13.670 align:middle line:84% as your training corpus, and you ran it through. 00:24:13.670 --> 00:24:16.050 align:middle line:84% Basically, you come up with English dictionary. 00:24:16.050 --> 00:24:18.330 align:middle line:84% So it will have maybe starting with a all the way 00:24:18.330 --> 00:24:22.010 align:middle line:84% to zebra, a whole bunch of words. 00:24:22.010 --> 00:24:25.790 align:middle line:84% And so I'm just putting 50,000 here because it turns out 00:24:25.790 --> 00:24:28.290 align:middle line:84% the GPT family uses something called 50,000 tokens, 00:24:28.290 --> 00:24:29.410 align:middle line:90% so I'm just using 50,000. 00:24:29.410 --> 00:24:31.910 align:middle line:84% It's not the actual number of words in the English language. 00:24:31.910 --> 00:24:32.970 align:middle line:90% It's much more than that. 00:24:32.970 --> 00:24:35.970 align:middle line:84% So let's say that we give a number 1 through 50,000, 00:24:35.970 --> 00:24:39.030 align:middle line:84% and then we actually also introduce a special token called 00:24:39.030 --> 00:24:41.590 align:middle line:90% UNK, which stands for unknown. 00:24:41.590 --> 00:24:43.110 align:middle line:84% And we'll come back to this later. 00:24:43.110 --> 00:24:47.370 align:middle line:84% And we give unknown the integer 0. 00:24:47.370 --> 00:24:49.550 align:middle line:84% So this is what we mean by indexing. 00:24:49.550 --> 00:24:52.250 align:middle line:84% Take the word, the tokens you have identified. 00:24:52.250 --> 00:24:55.470 align:middle line:90% Just map it to an integer. 00:24:55.470 --> 00:24:57.190 align:middle line:90% That's the indexing step. 00:24:57.190 --> 00:25:00.070 align:middle line:84% Then, what we do is we assign a vector 00:25:00.070 --> 00:25:03.570 align:middle line:90% to every one of these integers. 00:25:03.570 --> 00:25:05.410 align:middle line:90% And that is the encoding step. 00:25:05.410 --> 00:25:08.910 align:middle line:84% We assign a vector to each integer. 00:25:08.910 --> 00:25:10.670 align:middle line:84% So you have a bunch of distinct words. 00:25:10.670 --> 00:25:12.530 align:middle line:84% Under each word, we put an integer on it. 00:25:12.530 --> 00:25:14.738 align:middle line:84% And then we take that integer and map it to a vector. 00:25:14.738 --> 00:25:16.950 align:middle line:90% Yeah. 00:25:16.950 --> 00:25:18.550 align:middle line:84% Can you use the microphone, please? 00:25:18.550 --> 00:25:20.870 align:middle line:84% AUDIENCE: Can you please explain what unknown means? 00:25:20.870 --> 00:25:21.870 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:25:21.870 --> 00:25:23.440 align:middle line:90% So I'll come back to that. 00:25:23.440 --> 00:25:26.100 align:middle line:84% For now, just assume that we have a token called unknown. 00:25:26.100 --> 00:25:27.517 align:middle line:84% And the way we are going to use it 00:25:27.517 --> 00:25:29.487 align:middle line:84% will become apparent in a few minutes. 00:25:29.487 --> 00:25:31.320 align:middle line:84% AUDIENCE: Does it mean there's a base to it, 00:25:31.320 --> 00:25:32.980 align:middle line:90% there's a letter or something? 00:25:32.980 --> 00:25:35.540 align:middle line:84% RAMA RAMAKRISHNAN: It's a placeholder for something else, 00:25:35.540 --> 00:25:38.600 align:middle line:90% which I'll describe shortly. 00:25:38.600 --> 00:25:42.600 align:middle line:90% So that's what we have. 00:25:42.600 --> 00:25:45.640 align:middle line:84% So let's say that we want to assign a vector to each integer 00:25:45.640 --> 00:25:46.920 align:middle line:90% in our vocabulary. 00:25:46.920 --> 00:25:50.480 align:middle line:90% And let's assume that we have-- 00:25:50.480 --> 00:25:53.440 align:middle line:84% OK, let's say we have 50,000 possible integers because we 00:25:53.440 --> 00:25:56.340 align:middle line:84% have 50,000 possible words, and we want to assign a vector 00:25:56.340 --> 00:25:59.220 align:middle line:84% so that if you take the vector two different words, 00:25:59.220 --> 00:26:01.068 align:middle line:90% they should look different. 00:26:01.068 --> 00:26:03.360 align:middle line:84% Clearly, that's the whole point of mapping from integer 00:26:03.360 --> 00:26:04.320 align:middle line:90% to vector. 00:26:04.320 --> 00:26:05.840 align:middle line:90% They better be different. 00:26:05.840 --> 00:26:07.320 align:middle line:84% What is the simplest way to come up 00:26:07.320 --> 00:26:10.270 align:middle line:84% with a vector for each of these tokens? 00:26:10.270 --> 00:26:20.120 align:middle line:90% 00:26:20.120 --> 00:26:21.820 align:middle line:84% AUDIENCE: It's the same as the index. 00:26:21.820 --> 00:26:22.170 align:middle line:90% RAMA RAMAKRISHNAN: Sorry? 00:26:22.170 --> 00:26:23.820 align:middle line:84% AUDIENCE: It's the same as the index. 00:26:23.820 --> 00:26:26.900 align:middle line:84% It's just a vector one by one by one [INAUDIBLE] 00:26:26.900 --> 00:26:31.860 align:middle line:84% RAMA RAMAKRISHNAN: So a vector of zeros and ones, or? 00:26:31.860 --> 00:26:34.780 align:middle line:84% AUDIENCE: Just a vector with one dimension. 00:26:34.780 --> 00:26:36.220 align:middle line:90% RAMA RAMAKRISHNAN: Oh, I see. 00:26:36.220 --> 00:26:36.760 align:middle line:90% Got it. 00:26:36.760 --> 00:26:39.775 align:middle line:84% Well, it's creative, but it's a little cheating, right? 00:26:39.775 --> 00:26:41.900 align:middle line:84% Because you're essentially putting a square bracket 00:26:41.900 --> 00:26:44.420 align:middle line:84% around the number and saying it's a vector. 00:26:44.420 --> 00:26:46.492 align:middle line:90% Good try. 00:26:46.492 --> 00:26:48.200 align:middle line:84% AUDIENCE: You could try one-hot encoding. 00:26:48.200 --> 00:26:49.242 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:26:49.242 --> 00:26:51.420 align:middle line:90% You can try one-hot encoding. 00:26:51.420 --> 00:26:54.080 align:middle line:84% So, remember, the list of distinct tokens you have, 00:26:54.080 --> 00:26:56.260 align:middle line:84% you can just think of them as the distinct levels 00:26:56.260 --> 00:26:59.740 align:middle line:90% of a categorical variable. 00:26:59.740 --> 00:27:04.540 align:middle line:84% And you could just use one-hot encoding for it. 00:27:04.540 --> 00:27:09.220 align:middle line:84% So what you can do is the simple thing is to do one-hot encoding. 00:27:09.220 --> 00:27:13.660 align:middle line:84% And the way it is going to work is that if you have, let's say, 00:27:13.660 --> 00:27:16.700 align:middle line:84% 50,000 possible values, the vector is going to be 50,000 00:27:16.700 --> 00:27:17.200 align:middle line:90% long. 00:27:17.200 --> 00:27:19.110 align:middle line:84% It's going to have 0's everywhere 00:27:19.110 --> 00:27:22.730 align:middle line:84% except in the index value of whatever the token is. 00:27:22.730 --> 00:27:25.110 align:middle line:84% So, for instance, since we said UNK 00:27:25.110 --> 00:27:30.390 align:middle line:84% is going to be the first number, 0, it has a 1 here. 00:27:30.390 --> 00:27:33.010 align:middle line:84% The zeroth index position has a 1. 00:27:33.010 --> 00:27:34.190 align:middle line:90% Everything is 0. 00:27:34.190 --> 00:27:36.310 align:middle line:84% a happens to be the second one, so it 00:27:36.310 --> 00:27:38.430 align:middle line:84% happens to be 1 in the second position, zero. 00:27:38.430 --> 00:27:40.942 align:middle line:90% You get the idea. 00:27:40.942 --> 00:27:42.150 align:middle line:90% So this is zero-hot encoding. 00:27:42.150 --> 00:27:44.910 align:middle line:84% We can do this zero-hot encoding, one-hot encoding. 00:27:44.910 --> 00:27:49.230 align:middle line:84% And so the dimension of this encoding vector, how long it is, 00:27:49.230 --> 00:27:51.150 align:middle line:84% it's basically the number of distinct tokens 00:27:51.150 --> 00:27:54.830 align:middle line:84% that you have seen in the training corpus plus one 00:27:54.830 --> 00:27:56.570 align:middle line:84% for this UNK thing, which we'll get to. 00:27:56.570 --> 00:28:00.610 align:middle line:90% 00:28:00.610 --> 00:28:03.370 align:middle line:84% So that is the dimensional-encoding vector, 00:28:03.370 --> 00:28:05.500 align:middle line:84% which this is called the vocabulary size. 00:28:05.500 --> 00:28:09.162 align:middle line:90% 00:28:09.162 --> 00:28:10.620 align:middle line:84% This is called the vocabulary size. 00:28:10.620 --> 00:28:14.110 align:middle line:90% 00:28:14.110 --> 00:28:17.650 align:middle line:84% So, at this point, we have created a vocabulary 00:28:17.650 --> 00:28:19.830 align:middle line:84% from the training data, training corpus. 00:28:19.830 --> 00:28:21.490 align:middle line:84% Every distinct token vocabulary has 00:28:21.490 --> 00:28:23.530 align:middle line:84% been assigned a one-hot vector, and we are 00:28:23.530 --> 00:28:27.370 align:middle line:90% done with basic preprocessing. 00:28:27.370 --> 00:28:29.250 align:middle line:84% So all the text that has come in, 00:28:29.250 --> 00:28:32.570 align:middle line:84% every token has been mapped to one potentially very 00:28:32.570 --> 00:28:35.930 align:middle line:90% long one-hot vector. 00:28:35.930 --> 00:28:38.550 align:middle line:84% Any questions on the mechanics of this before we continue on? 00:28:38.550 --> 00:28:45.050 align:middle line:90% 00:28:45.050 --> 00:28:48.810 align:middle line:84% Now, let's see if, when you get a new input sentence in, 00:28:48.810 --> 00:28:51.130 align:middle line:84% a new sentence freshly arriving, and we 00:28:51.130 --> 00:28:52.955 align:middle line:84% want to feed it into a deep neural network, 00:28:52.955 --> 00:28:55.330 align:middle line:84% how will this process actually apply to that new sentence 00:28:55.330 --> 00:28:56.922 align:middle line:90% that's coming in? 00:28:56.922 --> 00:29:01.010 align:middle line:84% So let's assume that we have completed our S-T-I-E 00:29:01.010 --> 00:29:04.890 align:middle line:84% on the training corpus, and it turns out we found only 99 00:29:04.890 --> 00:29:07.870 align:middle line:84% distinct tokens, 99 distinct words. 00:29:07.870 --> 00:29:11.850 align:middle line:84% And then we add this UNK thing to it, so we got 100. 00:29:11.850 --> 00:29:13.590 align:middle line:90% So this is our vocabulary. 00:29:13.590 --> 00:29:16.840 align:middle line:84% It starts with UNK, a, and then goes all the way to zebra. 00:29:16.840 --> 00:29:19.780 align:middle line:84% But there are only 100 of them in total. 00:29:19.780 --> 00:29:22.540 align:middle line:84% And just to be very clear, we didn't bother to do things like 00:29:22.540 --> 00:29:25.000 align:middle line:84% stemming and stop-word removal and stuff like that, 00:29:25.000 --> 00:29:28.170 align:middle line:84% which is why you have words like "the" showing up in this list. 00:29:28.170 --> 00:29:32.420 align:middle line:90% 00:29:32.420 --> 00:29:35.720 align:middle line:84% So let's say this input string arrives, 00:29:35.720 --> 00:29:39.180 align:middle line:84% "the cat sat on the mat", and then we run it through S-T-I-E. 00:29:39.180 --> 00:29:42.040 align:middle line:84% So "the cat sat on the mat" goes through this thing, boom, boom, 00:29:42.040 --> 00:29:43.500 align:middle line:90% boom, boom, boom. 00:29:43.500 --> 00:29:48.620 align:middle line:84% Then, the output is going to be a table with a bunch of rows 00:29:48.620 --> 00:29:50.460 align:middle line:90% and a bunch of columns. 00:29:50.460 --> 00:29:54.050 align:middle line:84% Any guesses how many rows and how many columns? 00:29:54.050 --> 00:30:01.878 align:middle line:90% 00:30:01.878 --> 00:30:02.920 align:middle line:90% Please, raise your hands. 00:30:02.920 --> 00:30:03.640 align:middle line:90% I'll call on you. 00:30:03.640 --> 00:30:13.107 align:middle line:90% 00:30:13.107 --> 00:30:14.440 align:middle line:90% Yeah, please use the microphone. 00:30:14.440 --> 00:30:15.080 align:middle line:90% Go for it. 00:30:15.080 --> 00:30:19.332 align:middle line:84% AUDIENCE: I would guess 100 rows and 6 columns. 00:30:19.332 --> 00:30:20.540 align:middle line:90% RAMA RAMAKRISHNAN: All right. 00:30:20.540 --> 00:30:23.080 align:middle line:90% We'll take a look. 00:30:23.080 --> 00:30:25.640 align:middle line:84% 100 and 6 as well as 6 and 100 are both correct. 00:30:25.640 --> 00:30:29.500 align:middle line:84% So the way I've done it, it is 6 and 100, 00:30:29.500 --> 00:30:31.720 align:middle line:90% and that's exactly right. 00:30:31.720 --> 00:30:34.520 align:middle line:84% So the idea is that this is your vocabulary. 00:30:34.520 --> 00:30:37.420 align:middle line:84% So the word "the cat sat on the mat", 00:30:37.420 --> 00:30:41.600 align:middle line:84% once you change the case of it, it becomes like this. 00:30:41.600 --> 00:30:46.120 align:middle line:84% So the "the" happens to be a one-hot vector with a 1 where 00:30:46.120 --> 00:30:48.133 align:middle line:84% there is a "the" and 0 everywhere else. 00:30:48.133 --> 00:30:49.800 align:middle line:84% I'm not showing all the zeros because it 00:30:49.800 --> 00:30:52.040 align:middle line:90% will get too cluttered. 00:30:52.040 --> 00:30:57.120 align:middle line:84% Similarly, cat has a 1 where the cat position is and 0 everywhere 00:30:57.120 --> 00:30:59.400 align:middle line:90% else, and so on and so forth. 00:30:59.400 --> 00:31:00.600 align:middle line:90% Does that make sense? 00:31:00.600 --> 00:31:03.400 align:middle line:84% So the phrase, "the cat sat on the mat" came in as just, 00:31:03.400 --> 00:31:07.450 align:middle line:84% whatever, six words, and then it became this 600-entry table. 00:31:07.450 --> 00:31:12.200 align:middle line:90% 00:31:12.200 --> 00:31:16.370 align:middle line:84% Now, what is the best way to feed this table 00:31:16.370 --> 00:31:18.010 align:middle line:90% to a deep neural network? 00:31:18.010 --> 00:31:18.790 align:middle line:90% What can we do? 00:31:18.790 --> 00:31:23.610 align:middle line:90% 00:31:23.610 --> 00:31:26.088 align:middle line:90% It's not a vector, it's a table. 00:31:26.088 --> 00:31:27.630 align:middle line:84% If it's a vector, we know what to do. 00:31:27.630 --> 00:31:28.422 align:middle line:90% We just feed it in. 00:31:28.422 --> 00:31:30.810 align:middle line:84% We'll just maybe send it to some hidden layer 00:31:30.810 --> 00:31:34.410 align:middle line:84% and declare victory at that point. 00:31:34.410 --> 00:31:35.050 align:middle line:90% Yeah. 00:31:35.050 --> 00:31:36.425 align:middle line:84% AUDIENCE: You need to flatten it. 00:31:36.425 --> 00:31:38.425 align:middle line:84% RAMA RAMAKRISHNAN: You would like to flatten it? 00:31:38.425 --> 00:31:40.150 align:middle line:90% And how might you do it? 00:31:40.150 --> 00:31:43.330 align:middle line:90% 00:31:43.330 --> 00:31:46.720 align:middle line:84% Flattening is a reasonable answer, by the way. 00:31:46.720 --> 00:31:52.490 align:middle line:84% AUDIENCE: I think you just have to take each column, 00:31:52.490 --> 00:31:55.302 align:middle line:84% take the first one, go into each word and just kinda, like-- 00:31:55.302 --> 00:31:57.010 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so basically you 00:31:57.010 --> 00:31:59.810 align:middle line:84% can take all the first columns and then take the second column 00:31:59.810 --> 00:32:02.630 align:middle line:84% and attach it under the first column and so on and so forth. 00:32:02.630 --> 00:32:04.510 align:middle line:90% So we can certainly do that. 00:32:04.510 --> 00:32:06.830 align:middle line:84% And that's very akin to how we work with images. 00:32:06.830 --> 00:32:10.003 align:middle line:84% But there is one downside to that. 00:32:10.003 --> 00:32:10.920 align:middle line:90% What is that downside? 00:32:10.920 --> 00:32:15.870 align:middle line:90% 00:32:15.870 --> 00:32:18.510 align:middle line:90% Yes. 00:32:18.510 --> 00:32:19.830 align:middle line:90% AUDIENCE: It's pretty long. 00:32:19.830 --> 00:32:24.090 align:middle line:84% I wonder if instead you could for the first word, it's 1, 00:32:24.090 --> 00:32:27.130 align:middle line:84% for the second word, it's 2, and then you maintain the order, 00:32:27.130 --> 00:32:30.470 align:middle line:84% but you still keep it just as one row. 00:32:30.470 --> 00:32:32.530 align:middle line:90% RAMA RAMAKRISHNAN: One row. 00:32:32.530 --> 00:32:34.530 align:middle line:84% So we'll come back to what we do about this. 00:32:34.530 --> 00:32:37.410 align:middle line:84% But what you're pointing out is it could be very long. 00:32:37.410 --> 00:32:41.550 align:middle line:84% Because if each word is a 50,000-long one-hot vector, 00:32:41.550 --> 00:32:45.817 align:middle line:84% with just six words, it becomes a 300,000-long vector. 00:32:45.817 --> 00:32:48.150 align:middle line:84% Can you imagine taking a 300,000-long vector and sending 00:32:48.150 --> 00:32:51.270 align:middle line:90% it into a 100-unit hidden layer? 00:32:51.270 --> 00:32:53.590 align:middle line:90% 300,000 times 100 parameters? 00:32:53.590 --> 00:32:54.130 align:middle line:90% Too much. 00:32:54.130 --> 00:32:56.070 align:middle line:90% Can't learn anything. 00:32:56.070 --> 00:32:57.670 align:middle line:90% So that's one issue. 00:32:57.670 --> 00:33:00.750 align:middle line:84% The other issue is that different length 00:33:00.750 --> 00:33:04.470 align:middle line:84% texts that are coming in will have different-sized inputs. 00:33:04.470 --> 00:33:07.650 align:middle line:84% So here, "the cats are on the mat" has 6 times 50,000. 00:33:07.650 --> 00:33:10.970 align:middle line:84% But maybe the cat sat on the mat and the rat ran over to the cat. 00:33:10.970 --> 00:33:12.470 align:middle line:90% It becomes even longer. 00:33:12.470 --> 00:33:15.410 align:middle line:84% We can't handle variable-sized inputs. 00:33:15.410 --> 00:33:19.610 align:middle line:84% The inputs all have to be mapped to the same length. 00:33:19.610 --> 00:33:22.130 align:middle line:90% That's another problem. 00:33:22.130 --> 00:33:25.390 align:middle line:84% AUDIENCE: So maybe you can sum the columns, basically, 00:33:25.390 --> 00:33:27.850 align:middle line:84% and count how many times each word appears since you're 00:33:27.850 --> 00:33:30.090 align:middle line:90% using the spatial relationship. 00:33:30.090 --> 00:33:31.610 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:33:31.610 --> 00:33:33.810 align:middle line:84% So both you and [MUTED] are on the same sort 00:33:33.810 --> 00:33:37.410 align:middle line:84% of trajectory, which is that we need to somehow take this table 00:33:37.410 --> 00:33:38.930 align:middle line:90% and make it into a vector. 00:33:38.930 --> 00:33:41.470 align:middle line:84% And there are many ways, like what you folks are describing, 00:33:41.470 --> 00:33:42.770 align:middle line:90% to make it into a vector. 00:33:42.770 --> 00:33:43.760 align:middle line:90% And it turns out-- 00:33:43.760 --> 00:33:45.695 align:middle line:90% 00:33:45.695 --> 00:33:48.070 align:middle line:84% this is all the things that we've been discussing so far, 00:33:48.070 --> 00:33:50.250 align:middle line:84% the varying length issue and so on. 00:33:50.250 --> 00:33:54.250 align:middle line:84% So, what we can do is we can aggregate all these things. 00:33:54.250 --> 00:33:56.585 align:middle line:90% If you just add them up-- 00:33:56.585 --> 00:33:58.210 align:middle line:84% this is what you described, I believe-- 00:33:58.210 --> 00:34:00.690 align:middle line:90% it's called sum encoding. 00:34:00.690 --> 00:34:02.370 align:middle line:84% And if, instead of adding, you just 00:34:02.370 --> 00:34:04.350 align:middle line:84% order them, meaning if you look at the column and say, 00:34:04.350 --> 00:34:05.600 align:middle line:90% is there any 1 in this column? 00:34:05.600 --> 00:34:07.430 align:middle line:84% If there's any 1, I'll put a stick a 1. 00:34:07.430 --> 00:34:08.780 align:middle line:90% Otherwise it's 0. 00:34:08.780 --> 00:34:11.380 align:middle line:90% It's called multi-hot encoding. 00:34:11.380 --> 00:34:14.540 align:middle line:84% So if you look at this thing, if you literally just go column 00:34:14.540 --> 00:34:18.179 align:middle line:84% by column and count everything, OK, there's a 1 here, 1 here. 00:34:18.179 --> 00:34:20.951 align:middle line:84% Oh, wait, there are two 2s here, so you put a 2. 00:34:20.951 --> 00:34:22.159 align:middle line:90% That's called count encoding. 00:34:22.159 --> 00:34:26.100 align:middle line:84% Multi-hot encoding just looks for any 1s and it puts 1. 00:34:26.100 --> 00:34:27.367 align:middle line:90% Makes sense? 00:34:27.367 --> 00:34:29.659 align:middle line:84% So by the way, there are many ways to take these tables 00:34:29.659 --> 00:34:30.840 align:middle line:90% and make them into vectors. 00:34:30.840 --> 00:34:33.020 align:middle line:84% These two happen to be very commonly used, 00:34:33.020 --> 00:34:34.690 align:middle line:84% and they kind of make common sense. 00:34:34.690 --> 00:34:41.520 align:middle line:90% 00:34:41.520 --> 00:34:43.659 align:middle line:84% So this aggregation approach that we just described 00:34:43.659 --> 00:34:46.739 align:middle line:84% is called the Bag of Words model. 00:34:46.739 --> 00:34:48.179 align:middle line:90% The Bag of Words model. 00:34:48.179 --> 00:34:51.420 align:middle line:84% And the reason is that, first of all, this bag that we 00:34:51.420 --> 00:34:52.860 align:middle line:90% have has words. 00:34:52.860 --> 00:34:55.179 align:middle line:84% Either it counts whether a word exists or not, 00:34:55.179 --> 00:35:00.380 align:middle line:84% or it counts how many times the word has appeared, 00:35:00.380 --> 00:35:03.620 align:middle line:84% multi-hot versus sum encoding, count encoding. 00:35:03.620 --> 00:35:07.720 align:middle line:84% But more importantly, and this goes back to your observation, 00:35:07.720 --> 00:35:11.560 align:middle line:84% is that we have lost the order of the words now. 00:35:11.560 --> 00:35:14.800 align:middle line:84% Whether the phrase came in was "the cat sat on the mat" 00:35:14.800 --> 00:35:19.160 align:middle line:84% or "the mat sat on the cat", the count encoding and the multi-hot 00:35:19.160 --> 00:35:20.960 align:middle line:90% encoding are exactly the same. 00:35:20.960 --> 00:35:22.960 align:middle line:84% There's no difference because we're just 00:35:22.960 --> 00:35:25.140 align:middle line:84% looking for the presence or absence of words. 00:35:25.140 --> 00:35:26.000 align:middle line:90% That's it. 00:35:26.000 --> 00:35:29.160 align:middle line:84% We don't care in which order they appear. 00:35:29.160 --> 00:35:30.780 align:middle line:90% That's a huge limitation. 00:35:30.780 --> 00:35:33.940 align:middle line:84% But shockingly, for many applications, it doesn't matter. 00:35:33.940 --> 00:35:35.800 align:middle line:90% It's good enough. 00:35:35.800 --> 00:35:39.242 align:middle line:84% So it's called the Bag of Words model. 00:35:39.242 --> 00:35:40.950 align:middle line:84% So this is called the Bag of Words model. 00:35:40.950 --> 00:35:43.640 align:middle line:90% 00:35:43.640 --> 00:35:45.925 align:middle line:84% Now, does it have any shortcomings? 00:35:45.925 --> 00:35:47.800 align:middle line:84% I already talked about the first shortcoming, 00:35:47.800 --> 00:35:50.580 align:middle line:84% which is that it loses sequentiality, the order. 00:35:50.580 --> 00:35:53.666 align:middle line:90% We lost this order information. 00:35:53.666 --> 00:35:56.200 align:middle line:84% We lose the meaning inherent in the order of the words. 00:35:56.200 --> 00:35:57.660 align:middle line:84% What are some other issues with it? 00:35:57.660 --> 00:36:01.111 align:middle line:90% 00:36:01.111 --> 00:36:04.080 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:36:04.080 --> 00:36:07.090 align:middle line:84% RAMA RAMAKRISHNAN: What do you mean by that? 00:36:07.090 --> 00:36:11.954 align:middle line:84% AUDIENCE: For example, you have six different words [INAUDIBLE]. 00:36:11.954 --> 00:36:14.390 align:middle line:84% RAMA RAMAKRISHNAN: Right, so there are lots of zeros, 00:36:14.390 --> 00:36:15.430 align:middle line:90% not that many ones. 00:36:15.430 --> 00:36:17.722 align:middle line:84% So it's a very sparse amount of information, 00:36:17.722 --> 00:36:19.930 align:middle line:84% but maybe you're carrying around a lot of information 00:36:19.930 --> 00:36:21.190 align:middle line:90% to make it all work. 00:36:21.190 --> 00:36:23.730 align:middle line:84% Now, there are some tricks, CS, Computer Science, 00:36:23.730 --> 00:36:26.310 align:middle line:84% tricks to handle sparsity in some clever ways, 00:36:26.310 --> 00:36:28.425 align:middle line:90% but it is certainly an issue. 00:36:28.425 --> 00:36:30.050 align:middle line:84% Now, the other issue is that, let's say 00:36:30.050 --> 00:36:32.570 align:middle line:90% the vocabulary is very long. 00:36:32.570 --> 00:36:35.530 align:middle line:84% Each input sentence, whether it's the collected works 00:36:35.530 --> 00:36:39.130 align:middle line:84% of William Shakespeare or the phrase "I love you", 00:36:39.130 --> 00:36:42.690 align:middle line:90% will have the same length input. 00:36:42.690 --> 00:36:45.490 align:middle line:84% We'll have the same length input. 00:36:45.490 --> 00:36:48.090 align:middle line:84% Because ultimately, every incoming thing 00:36:48.090 --> 00:36:51.090 align:middle line:90% gets mapped into one vector. 00:36:51.090 --> 00:36:54.300 align:middle line:90% That feels a little suboptimal. 00:36:54.300 --> 00:36:56.050 align:middle line:84% Clearly, the collected works of literature 00:36:56.050 --> 00:37:00.010 align:middle line:84% have a lot more stuff going on in them. 00:37:00.010 --> 00:37:02.410 align:middle line:90% So that's the problem. 00:37:02.410 --> 00:37:04.890 align:middle line:84% In particular, very, very small things that come in, 00:37:04.890 --> 00:37:07.910 align:middle line:84% you'll be spending a lot of compute on those long vectors 00:37:07.910 --> 00:37:09.950 align:middle line:90% and processing them. 00:37:09.950 --> 00:37:12.110 align:middle line:84% Now, you can mitigate some of this 00:37:12.110 --> 00:37:15.272 align:middle line:84% by choosing only the most frequent words. 00:37:15.272 --> 00:37:17.230 align:middle line:84% I think the English language, I read somewhere, 00:37:17.230 --> 00:37:20.070 align:middle line:90% has roughly 500,000 words or so. 00:37:20.070 --> 00:37:22.873 align:middle line:84% But turns out the top 50,000 most frequent words were 00:37:22.873 --> 00:37:25.290 align:middle line:84% responsible for just about everything you're going to see, 00:37:25.290 --> 00:37:26.390 align:middle line:90% ever. 00:37:26.390 --> 00:37:29.430 align:middle line:84% And the other 50,000 are what's called the long-tail. 00:37:29.430 --> 00:37:31.110 align:middle line:90% They almost never happen. 00:37:31.110 --> 00:37:32.310 align:middle line:90% You never see them. 00:37:32.310 --> 00:37:34.010 align:middle line:84% So you can be very pragmatic and say, 00:37:34.010 --> 00:37:35.870 align:middle line:84% I'm not going to take every little word 00:37:35.870 --> 00:37:37.490 align:middle line:90% that I see in my vocabulary. 00:37:37.490 --> 00:37:39.450 align:middle line:84% I'm going to only take the most frequent words. 00:37:39.450 --> 00:37:41.993 align:middle line:84% I'm just going to ignore the rest. 00:37:41.993 --> 00:37:43.410 align:middle line:84% I'm just going to ignore the rest. 00:37:43.410 --> 00:37:46.660 align:middle line:90% 00:37:46.660 --> 00:37:49.310 align:middle line:90% But if you ignore the rest-- 00:37:49.310 --> 00:37:51.750 align:middle line:90% let's say there is one word-- 00:37:51.750 --> 00:37:54.830 align:middle line:84% let's take some Shakespeare word, Hamlet. 00:37:54.830 --> 00:37:57.830 align:middle line:84% Let's assume that you ignore the word Hamlet from your training 00:37:57.830 --> 00:37:58.330 align:middle line:90% corpus. 00:37:58.330 --> 00:37:59.905 align:middle line:84% You just delete it because it's not 00:37:59.905 --> 00:38:02.030 align:middle line:84% one of the top, most frequent things you have seen. 00:38:02.030 --> 00:38:05.560 align:middle line:84% And then somebody sends you a text saying, 00:38:05.560 --> 00:38:08.240 align:middle line:90% "Hamlet was a bad prince. 00:38:08.240 --> 00:38:10.820 align:middle line:84% Analyze the sentiment of the sentence", well, 00:38:10.820 --> 00:38:13.832 align:middle line:84% when you see Hamlet, what is your system going to do? 00:38:13.832 --> 00:38:15.540 align:middle line:84% It's going to look at the Hamlet and say, 00:38:15.540 --> 00:38:18.003 align:middle line:84% I can't see it in my vocabulary anywhere. 00:38:18.003 --> 00:38:19.920 align:middle line:84% And if it can't see it in the vocabulary, what 00:38:19.920 --> 00:38:22.400 align:middle line:90% is the only thing it can do? 00:38:22.400 --> 00:38:25.040 align:middle line:90% Replace it with UNK. 00:38:25.040 --> 00:38:28.440 align:middle line:84% So that's where UNK comes into the picture. 00:38:28.440 --> 00:38:29.920 align:middle line:84% So whenever it can't see something 00:38:29.920 --> 00:38:31.660 align:middle line:84% in the vocabulary in a new input, 00:38:31.660 --> 00:38:33.400 align:middle line:84% it just replaces it with UNK, which 00:38:33.400 --> 00:38:38.880 align:middle line:84% means that if you had ignored Romeo and Juliet and Hamlet 00:38:38.880 --> 00:38:41.240 align:middle line:84% in the training corpus, all of them 00:38:41.240 --> 00:38:43.360 align:middle line:84% are going to be replaced by the same UNK, which 00:38:43.360 --> 00:38:46.760 align:middle line:84% means that we can't distinguish between them anymore. 00:38:46.760 --> 00:38:48.880 align:middle line:84% AUDIENCE: So is this where hallucination 00:38:48.880 --> 00:38:54.147 align:middle line:84% comes into play here, where it doesn't recognize [INAUDIBLE]. 00:38:54.147 --> 00:38:55.980 align:middle line:84% RAMA RAMAKRISHNAN: Ah, interesting question. 00:38:55.980 --> 00:38:57.740 align:middle line:84% Is this where hallucination comes up? 00:38:57.740 --> 00:38:59.760 align:middle line:90% Actually, as it turns out, no. 00:38:59.760 --> 00:39:02.900 align:middle line:84% As we will see when we talk about LLMs later, 00:39:02.900 --> 00:39:05.890 align:middle line:84% LLMs actually will not have this UNK problem 00:39:05.890 --> 00:39:08.140 align:middle line:84% because they use a different tokenization scheme which 00:39:08.140 --> 00:39:10.848 align:middle line:84% can handle anything you throw at it, including new stuff you just 00:39:10.848 --> 00:39:12.460 align:middle line:90% made up. 00:39:12.460 --> 00:39:15.940 align:middle line:90% So we'll come back to that. 00:39:15.940 --> 00:39:17.820 align:middle line:90% So that's what we have. 00:39:17.820 --> 00:39:20.560 align:middle line:84% And so what we're going to do is, despite its shortcomings, 00:39:20.560 --> 00:39:23.940 align:middle line:84% Bag of Words is actually a really good default for many NLP 00:39:23.940 --> 00:39:25.100 align:middle line:90% tasks. 00:39:25.100 --> 00:39:27.460 align:middle line:84% And in the spirit of, "do the simple stuff first and do 00:39:27.460 --> 00:39:29.400 align:middle line:84% complicated things only if the simple stuff doesn't work", 00:39:29.400 --> 00:39:31.192 align:middle line:84% we'll use the Bag of Words model right now. 00:39:31.192 --> 00:39:36.340 align:middle line:84% So we'll switch to a Colab and see how it's done. 00:39:36.340 --> 00:39:39.860 align:middle line:84% So here, the application we're going to work with 00:39:39.860 --> 00:39:42.260 align:middle line:90% is kind of a fun application. 00:39:42.260 --> 00:39:45.980 align:middle line:84% We're going to try to predict the genre of songs. 00:39:45.980 --> 00:39:48.700 align:middle line:84% It's a nice classification use case. 00:39:48.700 --> 00:39:51.660 align:middle line:84% So we want to take some arbitrary song 00:39:51.660 --> 00:39:56.300 align:middle line:84% and then classify it into either hip hop, rock, or pop. 00:39:56.300 --> 00:40:00.510 align:middle line:84% And so, for instance, this is the kind 00:40:00.510 --> 00:40:02.590 align:middle line:90% of lyrics you're going to see. 00:40:02.590 --> 00:40:04.970 align:middle line:84% And as you will see in this data set, 00:40:04.970 --> 00:40:07.950 align:middle line:84% just a quick word of caution the data set does 00:40:07.950 --> 00:40:13.350 align:middle line:84% have lyrics which may not be safe for work, as it were. 00:40:13.350 --> 00:40:16.550 align:middle line:84% So I'm not going to be exploring the lyrics in the Colab, 00:40:16.550 --> 00:40:19.650 align:middle line:84% but I just want you to be aware of it. 00:40:19.650 --> 00:40:22.470 align:middle line:84% But it's just some data set that we downloaded from somewhere. 00:40:22.470 --> 00:40:23.770 align:middle line:90% It's got all these lyrics. 00:40:23.770 --> 00:40:26.630 align:middle line:84% So we're going to try to classify each verse that we 00:40:26.630 --> 00:40:29.010 align:middle line:84% see into one of three things, hip hop, rock or pop. 00:40:29.010 --> 00:40:32.330 align:middle line:84% It's a multi-class classification problem. 00:40:32.330 --> 00:40:34.990 align:middle line:84% Actually, what is the simplest neural network-based classifier 00:40:34.990 --> 00:40:38.460 align:middle line:90% we can build for this problem? 00:40:38.460 --> 00:40:41.770 align:middle line:90% 00:40:41.770 --> 00:40:43.390 align:middle line:84% So what is the simplest neural network 00:40:43.390 --> 00:40:46.310 align:middle line:90% we can build for this problem? 00:40:46.310 --> 00:40:47.610 align:middle line:90% So remember, what is the input? 00:40:47.610 --> 00:40:49.930 align:middle line:84% The input is going to be a bunch of song lyrics. 00:40:49.930 --> 00:40:52.430 align:middle line:84% It's going to be a really long song for all you know. 00:40:52.430 --> 00:40:54.950 align:middle line:84% And we're going to use the Bag of Words model. 00:40:54.950 --> 00:40:56.510 align:middle line:84% And let's assume for a moment that we 00:40:56.510 --> 00:40:59.050 align:middle line:90% will use multi-hot encoding. 00:40:59.050 --> 00:41:02.710 align:middle line:84% We'll create a vocabulary from the song, 00:41:02.710 --> 00:41:04.710 align:middle line:84% we'll take all the songs, we'll process them, 00:41:04.710 --> 00:41:07.670 align:middle line:84% run it through S-T-I-E, we'll do multi-hot encoding, 00:41:07.670 --> 00:41:11.550 align:middle line:84% which means that every song that comes in will be a vector 00:41:11.550 --> 00:41:14.170 align:middle line:90% that's how long? 00:41:14.170 --> 00:41:17.290 align:middle line:90% It will be as long as the? 00:41:17.290 --> 00:41:20.690 align:middle line:90% Correct, as the vocabulary size. 00:41:20.690 --> 00:41:24.430 align:middle line:84% So maybe what comes in is this phrase, 00:41:24.430 --> 00:41:26.930 align:middle line:84% since it's supposed to be songs, I'll say something which is 00:41:26.930 --> 00:41:32.020 align:middle line:84% probably common to 90% of songs, "I love you". 00:41:32.020 --> 00:41:38.410 align:middle line:84% That goes in, it goes into our S-T-I-E process. 00:41:38.410 --> 00:41:45.106 align:middle line:84% And then this S-T-I-E process gives us a vector, which is x1, 00:41:45.106 --> 00:41:50.090 align:middle line:84% x2, all the way to xv, where v stands for the size 00:41:50.090 --> 00:41:51.290 align:middle line:90% of the vocabulary. 00:41:51.290 --> 00:41:56.220 align:middle line:84% So that's our input layer all the way. 00:41:56.220 --> 00:41:59.270 align:middle line:84% So, knowing what we know now about deep learning, what can we 00:41:59.270 --> 00:41:59.770 align:middle line:90% do next? 00:41:59.770 --> 00:42:03.663 align:middle line:90% 00:42:03.663 --> 00:42:05.080 align:middle line:84% AUDIENCE: Maybe I'm getting ahead, 00:42:05.080 --> 00:42:07.740 align:middle line:84% but wouldn't the classifier just be-- 00:42:07.740 --> 00:42:09.740 align:middle line:84% like, the baseline would be classify it 00:42:09.740 --> 00:42:11.878 align:middle line:90% as the most common genre? 00:42:11.878 --> 00:42:13.920 align:middle line:84% RAMA RAMAKRISHNAN: That is the baseline, correct. 00:42:13.920 --> 00:42:16.060 align:middle line:84% And we'll come to the baseline a bit later. 00:42:16.060 --> 00:42:19.780 align:middle line:84% But here, I'm saying suppose you wanted to build a neural network 00:42:19.780 --> 00:42:21.060 align:middle line:90% model for this. 00:42:21.060 --> 00:42:23.340 align:middle line:90% How would you set it up? 00:42:23.340 --> 00:42:25.660 align:middle line:84% AUDIENCE: Do you think about the layers that you want? 00:42:25.660 --> 00:42:26.020 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:42:26.020 --> 00:42:27.460 align:middle line:84% And what is the simplest thing you 00:42:27.460 --> 00:42:28.860 align:middle line:90% can do with a neural network? 00:42:28.860 --> 00:42:30.260 align:middle line:90% How many layers? 00:42:30.260 --> 00:42:31.553 align:middle line:90% AUDIENCE: No additional layers. 00:42:31.553 --> 00:42:33.220 align:middle line:84% RAMA RAMAKRISHNAN: Well, then it becomes 00:42:33.220 --> 00:42:35.340 align:middle line:84% problematic with a singlular neural network 00:42:35.340 --> 00:42:37.515 align:middle line:84% because it could just be logistical regression. 00:42:37.515 --> 00:42:38.640 align:middle line:90% AUDIENCE: One hidden layer. 00:42:38.640 --> 00:42:40.640 align:middle line:84% RAMA RAMAKRISHNAN: Yes, thank you. 00:42:40.640 --> 00:42:43.140 align:middle line:84% I'm being a little squishy about this because there are some 00:42:43.140 --> 00:42:45.515 align:middle line:84% people who would be like, well, even if there's no hidden 00:42:45.515 --> 00:42:48.055 align:middle line:84% layers, if you're using ReLUs and this and that and sigma, 00:42:48.055 --> 00:42:50.680 align:middle line:84% maybe it's a neural network, and I don't want to get into that, 00:42:50.680 --> 00:42:53.560 align:middle line:84% "how many angels in the tip of a pin" argument. 00:42:53.560 --> 00:42:55.267 align:middle line:84% So yeah, we need one hidden layer. 00:42:55.267 --> 00:42:57.600 align:middle line:84% In this course, we need at least one hidden layer for it 00:42:57.600 --> 00:42:59.040 align:middle line:90% to qualify as a neural network. 00:42:59.040 --> 00:43:01.060 align:middle line:84% OK, so let's have a hidden layer. 00:43:01.060 --> 00:43:02.810 align:middle line:84% And we'll have a bunch of ReLUs, as usual. 00:43:02.810 --> 00:43:05.360 align:middle line:90% 00:43:05.360 --> 00:43:06.920 align:middle line:90% Bunch of ReLUs. 00:43:06.920 --> 00:43:08.860 align:middle line:84% And I'll ignore all the arrows between them. 00:43:08.860 --> 00:43:10.280 align:middle line:90% It's kind of a pain. 00:43:10.280 --> 00:43:12.180 align:middle line:84% And then we come to the output layer. 00:43:12.180 --> 00:43:15.360 align:middle line:84% And what should the output layer be? 00:43:15.360 --> 00:43:18.400 align:middle line:84% How many nodes do we need in the output layer? 00:43:18.400 --> 00:43:21.560 align:middle line:90% Three, hip hop, rock, or pop. 00:43:21.560 --> 00:43:23.220 align:middle line:84% And then that layer is called what? 00:43:23.220 --> 00:43:25.400 align:middle line:90% What activation function? 00:43:25.400 --> 00:43:26.560 align:middle line:90% Softmax, perfect. 00:43:26.560 --> 00:43:27.060 align:middle line:90% Love it. 00:43:27.060 --> 00:43:28.520 align:middle line:90% Love this class. 00:43:28.520 --> 00:43:35.840 align:middle line:84% All right, three things, rock, hip hop, and pop. 00:43:35.840 --> 00:43:39.200 align:middle line:84% And this is a softmax right there. 00:43:39.200 --> 00:43:43.160 align:middle line:84% And then it's going to give us three probabilities that add 00:43:43.160 --> 00:43:45.420 align:middle line:90% up to 1 because of the softmax. 00:43:45.420 --> 00:43:47.460 align:middle line:90% So that's our basic network. 00:43:47.460 --> 00:43:47.960 align:middle line:90% Perfect. 00:43:47.960 --> 00:43:49.107 align:middle line:90% Yeah. 00:43:49.107 --> 00:43:51.440 align:middle line:84% AUDIENCE: Why do you need those probabilities at the end 00:43:51.440 --> 00:43:54.510 align:middle line:84% if you just want to identify the most likely genre? 00:43:54.510 --> 00:43:58.147 align:middle line:84% The softmax just gives you a way to add them all up at once. 00:43:58.147 --> 00:43:59.230 align:middle line:90% Why do you need a softmax? 00:43:59.230 --> 00:44:01.563 align:middle line:84% Why don't you just take the max value and say it's that? 00:44:01.563 --> 00:44:03.397 align:middle line:84% RAMA RAMAKRISHNAN: Oh, interesting question. 00:44:03.397 --> 00:44:05.170 align:middle line:84% Why can't we just produce three numbers 00:44:05.170 --> 00:44:06.850 align:middle line:90% and grab the maximum number? 00:44:06.850 --> 00:44:09.890 align:middle line:84% So, it turns out finding the maximum 00:44:09.890 --> 00:44:14.330 align:middle line:84% of a bunch of numbers, that function is not very 00:44:14.330 --> 00:44:16.930 align:middle line:90% friendly for differentiation. 00:44:16.930 --> 00:44:19.170 align:middle line:84% And ultimately, you want to take this output 00:44:19.170 --> 00:44:22.290 align:middle line:84% and run it through a loss function like cross-entropy 00:44:22.290 --> 00:44:24.930 align:middle line:84% and then be able to run backprop on it. 00:44:24.930 --> 00:44:27.250 align:middle line:84% And so, fundamentally, backpropagation 00:44:27.250 --> 00:44:30.490 align:middle line:84% is just differentiation, and it requires everything inside of it 00:44:30.490 --> 00:44:33.250 align:middle line:90% to have well-behaved gradients. 00:44:33.250 --> 00:44:37.250 align:middle line:84% And so this little max function is actually not well-behaved, 00:44:37.250 --> 00:44:41.130 align:middle line:84% which is why we have a soft version of it, softmax, which 00:44:41.130 --> 00:44:43.250 align:middle line:90% makes it easy to differentiate. 00:44:43.250 --> 00:44:45.150 align:middle line:84% So I can tell you more about it offline, 00:44:45.150 --> 00:44:46.400 align:middle line:90% but that's the quick synopsis. 00:44:46.400 --> 00:44:49.003 align:middle line:90% 00:44:49.003 --> 00:44:51.670 align:middle line:84% So a lot of tricks you will see in the neural network literature 00:44:51.670 --> 00:44:55.670 align:middle line:84% are ways to avoid the problem of having certain-- 00:44:55.670 --> 00:44:57.370 align:middle line:84% like, the obvious choice of function 00:44:57.370 --> 00:44:59.328 align:middle line:84% will not be well-behaved for differentiation. 00:44:59.328 --> 00:45:00.870 align:middle line:84% That's why you need to go through all 00:45:00.870 --> 00:45:02.910 align:middle line:90% these other mechanisms. 00:45:02.910 --> 00:45:04.663 align:middle line:84% Much like we couldn't just say accuracy, 00:45:04.663 --> 00:45:06.830 align:middle line:84% why don't we just maximize accuracy instead of doing 00:45:06.830 --> 00:45:08.370 align:middle line:90% this cross-entropy business? 00:45:08.370 --> 00:45:10.790 align:middle line:90% Same reason. 00:45:10.790 --> 00:45:11.590 align:middle line:90% All right. 00:45:11.590 --> 00:45:13.050 align:middle line:90% So let's come back here. 00:45:13.050 --> 00:45:20.590 align:middle line:90% 00:45:20.590 --> 00:45:25.350 align:middle line:84% So that's what we created on the thing, "cat sat on the mat" 00:45:25.350 --> 00:45:28.050 align:middle line:90% vocabulary thing and so on. 00:45:28.050 --> 00:45:30.930 align:middle line:84% And I was playing around with it earlier, 00:45:30.930 --> 00:45:34.150 align:middle line:84% and so I found that eight ReLU neurons were pretty 00:45:34.150 --> 00:45:35.210 align:middle line:90% good to get the job done. 00:45:35.210 --> 00:45:38.070 align:middle line:84% So I'm just going to go with eight ReLU neurons 00:45:38.070 --> 00:45:39.870 align:middle line:90% in the hidden layer. 00:45:39.870 --> 00:45:44.030 align:middle line:84% So I think that brings us to the Colab. 00:45:44.030 --> 00:45:46.950 align:middle line:90% Yeah, so let's switch to Colab. 00:45:46.950 --> 00:45:47.710 align:middle line:90% All right. 00:45:47.710 --> 00:45:49.970 align:middle line:90% So that's what we have here. 00:45:49.970 --> 00:45:51.693 align:middle line:84% There's a little bit of verbiage here, 00:45:51.693 --> 00:45:53.610 align:middle line:84% which just describes what I just talked about. 00:45:53.610 --> 00:45:56.490 align:middle line:84% So we'll do the usual things and upload everything, 00:45:56.490 --> 00:46:00.010 align:middle line:84% import everything we want, TensorFlow and Keras, 00:46:00.010 --> 00:46:03.610 align:middle line:84% and the Holy Trinity of NumPy, pandas, and Matplotlib, 00:46:03.610 --> 00:46:07.010 align:middle line:84% set the random seed, as usual, at 42. 00:46:07.010 --> 00:46:09.250 align:middle line:84% There's our S-T-I-E framework here. 00:46:09.250 --> 00:46:13.290 align:middle line:84% And the nice thing is that all four of these things, S-T-I-E, 00:46:13.290 --> 00:46:15.730 align:middle line:84% are beautifully implemented in Keras as a single, 00:46:15.730 --> 00:46:19.210 align:middle line:84% simple layer called the text vectorization layer, 00:46:19.210 --> 00:46:20.650 align:middle line:90% which is nice. 00:46:20.650 --> 00:46:23.610 align:middle line:84% So we have the text vectorization array right here. 00:46:23.610 --> 00:46:25.290 align:middle line:84% And so in our first example, what 00:46:25.290 --> 00:46:28.370 align:middle line:84% we'll do is we will use a default standardization, which 00:46:28.370 --> 00:46:30.950 align:middle line:84% will just remove punctuation, convert to lowercase. 00:46:30.950 --> 00:46:32.670 align:middle line:84% We'll use the default tokenization, 00:46:32.670 --> 00:46:35.450 align:middle line:84% which just means split on the space between words. 00:46:35.450 --> 00:46:38.490 align:middle line:84% And then we will set the output to multi-hot. 00:46:38.490 --> 00:46:40.810 align:middle line:84% All the things we talked about, Keras will just 00:46:40.810 --> 00:46:42.050 align:middle line:90% do it for you automatically. 00:46:42.050 --> 00:46:44.770 align:middle line:84% And so output mode, multi-hot, standardize 00:46:44.770 --> 00:46:46.730 align:middle line:84% this, split whitespace, and boom, 00:46:46.730 --> 00:46:48.390 align:middle line:84% you run the text vectorization thing. 00:46:48.390 --> 00:46:51.300 align:middle line:84% And once you do it, Keras creates this textualization 00:46:51.300 --> 00:46:53.380 align:middle line:84% layer with these settings, and it's now 00:46:53.380 --> 00:46:55.380 align:middle line:90% ready to swing into action. 00:46:55.380 --> 00:46:57.820 align:middle line:84% So what does swing into action actually mean? 00:46:57.820 --> 00:47:00.360 align:middle line:84% Well, now you need to actually feed it a training corpus so 00:47:00.360 --> 00:47:02.860 align:middle line:84% that it can do all the things it's supposed to do and create 00:47:02.860 --> 00:47:04.000 align:middle line:90% the vocabulary for you. 00:47:04.000 --> 00:47:06.780 align:middle line:90% 00:47:06.780 --> 00:47:08.720 align:middle line:84% And that thing is called the adapt method. 00:47:08.720 --> 00:47:11.920 align:middle line:84% So we create a tiny training corpus for us. 00:47:11.920 --> 00:47:14.720 align:middle line:90% This is our data set. 00:47:14.720 --> 00:47:16.860 align:middle line:84% This is a bunch of words from some of these lyrics. 00:47:16.860 --> 00:47:19.420 align:middle line:84% And then what we'll do is we'll take this layer that we just 00:47:19.420 --> 00:47:21.480 align:middle line:84% defined here that we have set up here, 00:47:21.480 --> 00:47:24.420 align:middle line:84% and then we will ask this layer to actually create 00:47:24.420 --> 00:47:28.180 align:middle line:84% the vocabulary using this adapt command. 00:47:28.180 --> 00:47:30.360 align:middle line:84% It will index the vocabulary, and it's done. 00:47:30.360 --> 00:47:32.220 align:middle line:84% And once it does it, you can actually 00:47:32.220 --> 00:47:34.720 align:middle line:90% ask it for the vocabulary-- 00:47:34.720 --> 00:47:37.840 align:middle line:84% this is vocabulary-- using the get_vocabulary command. 00:47:37.840 --> 00:47:39.840 align:middle line:84% And so, first of all, how long is the vocab? 00:47:39.840 --> 00:47:40.740 align:middle line:90% 17. 00:47:40.740 --> 00:47:43.140 align:middle line:90% 17 words, 17 tokens. 00:47:43.140 --> 00:47:44.980 align:middle line:90% What are they? 00:47:44.980 --> 00:47:45.952 align:middle line:90% You can see here. 00:47:45.952 --> 00:47:47.660 align:middle line:84% And you can see, these are all the words. 00:47:47.660 --> 00:47:50.760 align:middle line:84% And you can see it has stuck an UNK in the very beginning. 00:47:50.760 --> 00:47:52.040 align:middle line:90% It's sort of the default. 00:47:52.040 --> 00:47:54.857 align:middle line:84% By the way, just a little programming tip, 00:47:54.857 --> 00:47:56.940 align:middle line:84% if you don't have a ton of programming experience, 00:47:56.940 --> 00:47:59.120 align:middle line:84% if you want to print these Python objects, 00:47:59.120 --> 00:48:02.440 align:middle line:84% like lists and so on, in a pretty way, one trick that 00:48:02.440 --> 00:48:05.800 align:middle line:84% often works is just stick it into a data frame 00:48:05.800 --> 00:48:07.265 align:middle line:90% and then print it. 00:48:07.265 --> 00:48:09.140 align:middle line:84% Usually, it'll print it in a much better way. 00:48:09.140 --> 00:48:11.640 align:middle line:90% So you can see it like that. 00:48:11.640 --> 00:48:14.120 align:middle line:84% So you can see here, UNK, arrays, blah, blah, blah, blah, 00:48:14.120 --> 00:48:15.000 align:middle line:90% blah. 00:48:15.000 --> 00:48:17.600 align:middle line:84% And you can see integer zeros assigned the UNK token. 00:48:17.600 --> 00:48:20.640 align:middle line:84% By the way, how come it picked the word "arrays" 00:48:20.640 --> 00:48:22.060 align:middle line:90% as the second entry? 00:48:22.060 --> 00:48:28.540 align:middle line:84% Why not something like "an" or why not "a"? 00:48:28.540 --> 00:48:31.340 align:middle line:84% How come "a" is not chosen as a second entry? 00:48:31.340 --> 00:48:33.340 align:middle line:84% Why did it pick "arrays", do you think? 00:48:33.340 --> 00:48:40.880 align:middle line:90% 00:48:40.880 --> 00:48:43.160 align:middle line:84% AUDIENCE: Maybe it tried the words 00:48:43.160 --> 00:48:46.490 align:middle line:84% that are most influential on the meaning of the sentence 00:48:46.490 --> 00:48:49.368 align:middle line:84% to be on the top and less influential. 00:48:49.368 --> 00:48:50.910 align:middle line:84% RAMA RAMAKRISHNAN: But at this point, 00:48:50.910 --> 00:48:54.170 align:middle line:84% it doesn't know what we're going to use it for, 00:48:54.170 --> 00:48:56.210 align:middle line:84% so it has no way to know what word is useful 00:48:56.210 --> 00:48:59.450 align:middle line:84% because we haven't told it how we're going to use it. 00:48:59.450 --> 00:49:02.110 align:middle line:84% But you're kind of on the right track. 00:49:02.110 --> 00:49:06.470 align:middle line:84% So what Keras does is it will find all these tokens, 00:49:06.470 --> 00:49:09.730 align:middle line:84% and then it'll actually just sort them by frequency. 00:49:09.730 --> 00:49:11.952 align:middle line:84% So the most frequent, as it turns out, 00:49:11.952 --> 00:49:13.410 align:middle line:84% in those four sentences we gave it, 00:49:13.410 --> 00:49:15.210 align:middle line:84% happened to be the word "arrays". 00:49:15.210 --> 00:49:17.235 align:middle line:84% That's why "arrays" is showing up on top. 00:49:17.235 --> 00:49:18.610 align:middle line:84% And you can actually confirm this 00:49:18.610 --> 00:49:20.510 align:middle line:90% by going to the little data set. 00:49:20.510 --> 00:49:23.690 align:middle line:84% And you can see here "arrays" shows up here and once up here 00:49:23.690 --> 00:49:24.270 align:middle line:90% twice. 00:49:24.270 --> 00:49:27.790 align:middle line:84% And that's why it came up on top. 00:49:27.790 --> 00:49:29.770 align:middle line:90% So that's what we have. 00:49:29.770 --> 00:49:32.790 align:middle line:84% And now that we have populated this, 00:49:32.790 --> 00:49:34.910 align:middle line:84% we can run any sentence through it easily. 00:49:34.910 --> 00:49:36.370 align:middle line:90% Yeah. 00:49:36.370 --> 00:49:38.808 align:middle line:84% AUDIENCE: Does it matter that it's on the top? 00:49:38.808 --> 00:49:40.350 align:middle line:84% RAMA RAMAKRISHNAN: It doesn't matter. 00:49:40.350 --> 00:49:41.230 align:middle line:90% It doesn't matter. 00:49:41.230 --> 00:49:43.250 align:middle line:84% The reason why it's helpful later on 00:49:43.250 --> 00:49:45.790 align:middle line:84% is because suppose you tell Keras, hey, don't take 00:49:45.790 --> 00:49:49.390 align:middle line:84% every word you see here, give me only the most frequent 00:49:49.390 --> 00:49:50.277 align:middle line:90% 100 words. 00:49:50.277 --> 00:49:51.610 align:middle line:90% I don't want any more than that. 00:49:51.610 --> 00:49:52.690 align:middle line:90% It can easily do that. 00:49:52.690 --> 00:49:53.570 align:middle line:90% That's the reason. 00:49:53.570 --> 00:49:54.230 align:middle line:90% Yeah. 00:49:54.230 --> 00:49:56.470 align:middle line:84% AUDIENCE: We could do it after [INAUDIBLE]. 00:49:56.470 --> 00:50:00.712 align:middle line:90% 00:50:00.712 --> 00:50:02.670 align:middle line:84% RAMA RAMAKRISHNAN: This is just the vocabulary. 00:50:02.670 --> 00:50:05.443 align:middle line:84% So basically, you give it all these phrases-- 00:50:05.443 --> 00:50:07.610 align:middle line:84% it happened to be just four phrases in our example-- 00:50:07.610 --> 00:50:09.318 align:middle line:84% and then it finds all the distinct words, 00:50:09.318 --> 00:50:10.490 align:middle line:90% and it does all that stuff. 00:50:10.490 --> 00:50:12.230 align:middle line:84% And then it has created a vocabulary. 00:50:12.230 --> 00:50:17.070 align:middle line:84% At this point, the training corpus you fed it is forgotten. 00:50:17.070 --> 00:50:19.190 align:middle line:84% And the only thing that survived this processing 00:50:19.190 --> 00:50:20.370 align:middle line:90% is just the vocabulary. 00:50:20.370 --> 00:50:21.055 align:middle line:90% That's it. 00:50:21.055 --> 00:50:23.430 align:middle line:84% And now, we have to start applying it to any kind of text 00:50:23.430 --> 00:50:25.790 align:middle line:90% we want to use it for. 00:50:25.790 --> 00:50:29.410 align:middle line:84% So here, when you come back here, this is what we have. 00:50:29.410 --> 00:50:31.870 align:middle line:84% And so what you can do is you can take any sentence 00:50:31.870 --> 00:50:33.670 align:middle line:84% and you can just run it through the layer 00:50:33.670 --> 00:50:36.295 align:middle line:84% to make sure that it actually is doing the right thing for you. 00:50:36.295 --> 00:50:37.530 align:middle line:90% So we'll take this sentence. 00:50:37.530 --> 00:50:40.070 align:middle line:84% We will then run it through the text vectorization layer 00:50:40.070 --> 00:50:41.730 align:middle line:84% by just passing that sentence into it, 00:50:41.730 --> 00:50:43.060 align:middle line:90% and then we can just print it. 00:50:43.060 --> 00:50:46.000 align:middle line:90% 00:50:46.000 --> 00:50:47.740 align:middle line:84% So now, it's giving you a tensor. 00:50:47.740 --> 00:50:50.000 align:middle line:84% This is a multi-hot encoded tensor 00:50:50.000 --> 00:50:52.000 align:middle line:90% with all these ones and zeros. 00:50:52.000 --> 00:50:55.960 align:middle line:84% So note that this tensor is 17 units long, 00:50:55.960 --> 00:50:58.380 align:middle line:84% which is a good check because our vocabulary is 17 long, 00:50:58.380 --> 00:51:00.520 align:middle line:90% so it better match that. 00:51:00.520 --> 00:51:04.100 align:middle line:84% Now, recall that the UNK token is at the first location. 00:51:04.100 --> 00:51:05.440 align:middle line:90% It's at index 0. 00:51:05.440 --> 00:51:10.440 align:middle line:84% And it says that this encoded sentence does have an UNK word. 00:51:10.440 --> 00:51:14.640 align:middle line:90% So why is that? 00:51:14.640 --> 00:51:15.820 align:middle line:90% What is this UNK word? 00:51:15.820 --> 00:51:19.040 align:middle line:90% Anyone can guess? 00:51:19.040 --> 00:51:23.080 align:middle line:84% Well, it turns out to be the word still, I think. 00:51:23.080 --> 00:51:24.900 align:middle line:84% Yeah, still is not in our vocabulary, 00:51:24.900 --> 00:51:27.640 align:middle line:84% because if the four sentences, which is our training corpus 00:51:27.640 --> 00:51:29.680 align:middle line:84% used to build the vocabulary, they 00:51:29.680 --> 00:51:31.320 align:middle line:84% had a lot write and rewrite, but there 00:51:31.320 --> 00:51:32.880 align:middle line:90% was no still in it anywhere. 00:51:32.880 --> 00:51:34.960 align:middle line:84% That's why there is an UNK for it. 00:51:34.960 --> 00:51:38.620 align:middle line:84% We can just double-check that by asking Python, is it vocabulary? 00:51:38.620 --> 00:51:39.260 align:middle line:90% No, it is not. 00:51:39.260 --> 00:51:41.680 align:middle line:84% OK, now, in the spirit of making small changes to the code 00:51:41.680 --> 00:51:43.180 align:middle line:84% to understand what's going on, which 00:51:43.180 --> 00:51:45.620 align:middle line:84% is a very useful tip for folks who 00:51:45.620 --> 00:51:47.860 align:middle line:84% don't have a ton of programming knowledge, 00:51:47.860 --> 00:51:52.608 align:middle line:84% let's say that you send the phrase Sloan, HODL, and DMD. 00:51:52.608 --> 00:51:54.900 align:middle line:84% I think you will agree with me that none of these words 00:51:54.900 --> 00:51:57.100 align:middle line:90% is in the training corpus. 00:51:57.100 --> 00:52:01.220 align:middle line:84% So what is the multi-hot encoded vector 00:52:01.220 --> 00:52:03.490 align:middle line:84% for this phrase, Sloan, HODL, DMD? 00:52:03.490 --> 00:52:07.320 align:middle line:90% 00:52:07.320 --> 00:52:07.820 align:middle line:90% Three? 00:52:07.820 --> 00:52:11.420 align:middle line:90% 00:52:11.420 --> 00:52:12.560 align:middle line:90% It's not count encoding. 00:52:12.560 --> 00:52:14.780 align:middle line:90% It's multi-hot encoding. 00:52:14.780 --> 00:52:16.900 align:middle line:90% It's going to be 1, 0, 0. 00:52:16.900 --> 00:52:18.540 align:middle line:84% So you can see here, in this case, 00:52:18.540 --> 00:52:21.920 align:middle line:90% remember the vocabulary is 17. 00:52:21.920 --> 00:52:24.100 align:middle line:84% So each of these words is going to be 00:52:24.100 --> 00:52:27.420 align:middle line:90% a one followed by 16 zeros. 00:52:27.420 --> 00:52:29.140 align:middle line:84% And then it's going to multi-hot encode 00:52:29.140 --> 00:52:32.980 align:middle line:84% them, which means the three ones in the column just become a one. 00:52:32.980 --> 00:52:36.940 align:middle line:90% So you still have only this one. 00:52:36.940 --> 00:52:37.860 align:middle line:90% All right, good. 00:52:37.860 --> 00:52:42.310 align:middle line:84% So now now, let's actually get to the data set we have, 00:52:42.310 --> 00:52:45.230 align:middle line:90% this 90,000 songs. 00:52:45.230 --> 00:52:47.390 align:middle line:84% And it's in this little thing here. 00:52:47.390 --> 00:52:49.367 align:middle line:84% We have grabbed the data and cleaned it up, 00:52:49.367 --> 00:52:50.950 align:middle line:84% cleaned it up meaning formatting-wise, 00:52:50.950 --> 00:52:52.790 align:middle line:90% not content-wise. 00:52:52.790 --> 00:52:54.630 align:middle line:84% And then we stuck it in this data frame. 00:52:54.630 --> 00:52:57.630 align:middle line:84% And we already have divided it into train, test, and validation 00:52:57.630 --> 00:53:00.430 align:middle line:84% for your benefit, so you don't have to worry about it. 00:53:00.430 --> 00:53:04.550 align:middle line:84% Turns out we have almost 49,000 songs in the training set, 00:53:04.550 --> 00:53:06.450 align:middle line:84% 16,000 songs in the validation set, 00:53:06.450 --> 00:53:09.630 align:middle line:84% and roughly 22,000 in the test set. 00:53:09.630 --> 00:53:10.590 align:middle line:90% It's a lot of songs. 00:53:10.590 --> 00:53:11.090 align:middle line:90% It's a lot. 00:53:11.090 --> 00:53:12.750 align:middle line:90% It's a big data set. 00:53:12.750 --> 00:53:15.790 align:middle line:84% So let's just look at the first few. 00:53:15.790 --> 00:53:17.570 align:middle line:84% So, "Oh, girl, I can't get ready", 00:53:17.570 --> 00:53:21.990 align:middle line:84% "we met on rainy evening", "paralysis through analysis". 00:53:21.990 --> 00:53:24.840 align:middle line:84% OK, that, I can relate to as a data science person, but anyway. 00:53:24.840 --> 00:53:27.390 align:middle line:90% 00:53:27.390 --> 00:53:29.150 align:middle line:84% By the way, these things are very 00:53:29.150 --> 00:53:33.150 align:middle line:84% useful for exploration of any data frames that you might have. 00:53:33.150 --> 00:53:36.510 align:middle line:84% It's a Colab feature, just check it out. 00:53:36.510 --> 00:53:38.907 align:middle line:84% So anyway, that's the first few rows. 00:53:38.907 --> 00:53:40.240 align:middle line:90% Let's look at the last few rows. 00:53:40.240 --> 00:53:48.810 align:middle line:90% 00:53:48.810 --> 00:53:51.050 align:middle line:90% "You never listen to me" is pop. 00:53:51.050 --> 00:53:52.850 align:middle line:90% "Bimmer, Benz" is hip hop. 00:53:52.850 --> 00:53:54.664 align:middle line:90% Yeah, of course. 00:53:54.664 --> 00:54:00.930 align:middle line:84% [INAUDIBLE] OK, now to go back to the question of, OK, what 00:54:00.930 --> 00:54:02.230 align:middle line:90% could be a good baseline model? 00:54:02.230 --> 00:54:04.490 align:middle line:84% We need to understand the proportion of these three 00:54:04.490 --> 00:54:06.050 align:middle line:90% classes of songs. 00:54:06.050 --> 00:54:07.730 align:middle line:90% So we will do a quick check. 00:54:07.730 --> 00:54:10.170 align:middle line:90% It turns out rock is 55%. 00:54:10.170 --> 00:54:12.910 align:middle line:84% So if you had to just guess something just naively, 00:54:12.910 --> 00:54:14.702 align:middle line:84% you would just guess everything to be rock, 00:54:14.702 --> 00:54:16.770 align:middle line:84% and you'd be right 55% of the time. 00:54:16.770 --> 00:54:20.090 align:middle line:84% So now, by the way, the target variable, which tells you 00:54:20.090 --> 00:54:21.978 align:middle line:84% whether which of these three genres it is, 00:54:21.978 --> 00:54:25.030 align:middle line:90% is actually a dummy variable. 00:54:25.030 --> 00:54:28.130 align:middle line:84% So need to, one, hot-encode that. 00:54:28.130 --> 00:54:31.330 align:middle line:84% So we'll just turn that this way using the pandas 00:54:31.330 --> 00:54:32.710 align:middle line:90% get_dummies function. 00:54:32.710 --> 00:54:34.727 align:middle line:84% And when we do that, this is y_train, 00:54:34.727 --> 00:54:36.310 align:middle line:84% which contains the dependent variable. 00:54:36.310 --> 00:54:38.820 align:middle line:84% And you can see that is one hot-encoded now. 00:54:38.820 --> 00:54:41.660 align:middle line:84% 010, 010, 001, and so on and so forth. 00:54:41.660 --> 00:54:42.160 align:middle line:90% That's it. 00:54:42.160 --> 00:54:44.620 align:middle line:84% So I think the first, I forget it, 00:54:44.620 --> 00:54:47.600 align:middle line:84% rock, hip hop, pop or whatever, it's in some order. 00:54:47.600 --> 00:54:48.680 align:middle line:90% We'll get to that later. 00:54:48.680 --> 00:54:50.660 align:middle line:90% So it's one hot-encoded as well. 00:54:50.660 --> 00:54:52.860 align:middle line:84% So that is as far as the data downloading 00:54:52.860 --> 00:54:53.920 align:middle line:90% and setup is concerned. 00:54:53.920 --> 00:54:55.620 align:middle line:90% Any questions? 00:54:55.620 --> 00:54:57.460 align:middle line:90% Yeah. 00:54:57.460 --> 00:54:59.460 align:middle line:84% AUDIENCE: This kind of goes back to the transfer 00:54:59.460 --> 00:55:01.780 align:middle line:84% learning concept, but do you always 00:55:01.780 --> 00:55:04.300 align:middle line:84% want to build your corpus based off 00:55:04.300 --> 00:55:06.780 align:middle line:84% of the vocabulary of your training data? 00:55:06.780 --> 00:55:09.500 align:middle line:84% Or could you have a pre-compiled, like, 00:55:09.500 --> 00:55:13.020 align:middle line:84% somebody already made like a list of the 50,000 most common 00:55:13.020 --> 00:55:13.600 align:middle line:90% words? 00:55:13.600 --> 00:55:15.140 align:middle line:84% RAMA RAMAKRISHNAN: That's a really good question. 00:55:15.140 --> 00:55:17.390 align:middle line:84% Unfortunately, I'm going to punt on it for the moment, 00:55:17.390 --> 00:55:21.520 align:middle line:84% because with modern large language models, 00:55:21.520 --> 00:55:25.620 align:middle line:84% a number of these NLP tasks for which you had to roll your own 00:55:25.620 --> 00:55:27.420 align:middle line:84% and build your own thing, can now 00:55:27.420 --> 00:55:31.300 align:middle line:84% be very easily done using large language models without even 00:55:31.300 --> 00:55:32.672 align:middle line:90% any further training. 00:55:32.672 --> 00:55:34.380 align:middle line:84% Now, the price you pay for it is that you 00:55:34.380 --> 00:55:35.880 align:middle line:84% have to use a large language model, which means you 00:55:35.880 --> 00:55:38.260 align:middle line:84% have to pay somebody an API call and things like that. 00:55:38.260 --> 00:55:40.080 align:middle line:84% And there are other issues with it. 00:55:40.080 --> 00:55:45.338 align:middle line:84% But we will talk a lot about transfer learning for text when 00:55:45.338 --> 00:55:47.380 align:middle line:84% we come to it a little later in the NLP sequence. 00:55:47.380 --> 00:55:49.620 align:middle line:84% So if I forget, please bring it up again. 00:55:49.620 --> 00:55:53.320 align:middle line:90% 00:55:53.320 --> 00:55:55.840 align:middle line:90% Yeah. 00:55:55.840 --> 00:55:59.000 align:middle line:84% AUDIENCE: Quick clarification on the encoding vector. 00:55:59.000 --> 00:56:00.860 align:middle line:84% I noticed it was floats, not ints. 00:56:00.860 --> 00:56:03.440 align:middle line:84% If it gets incredibly long, wouldn't that 00:56:03.440 --> 00:56:05.100 align:middle line:90% eat into compute time? 00:56:05.100 --> 00:56:06.540 align:middle line:84% Is there a reason why it's floats? 00:56:06.540 --> 00:56:08.080 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so question 00:56:08.080 --> 00:56:12.080 align:middle line:84% is that when I showed you that tensor, 00:56:12.080 --> 00:56:15.140 align:middle line:84% it's written as a continuous number, a floating point number. 00:56:15.140 --> 00:56:16.820 align:middle line:84% But we know these are zeros and ones. 00:56:16.820 --> 00:56:19.400 align:middle line:84% So why do we have to waste compute capacity 00:56:19.400 --> 00:56:22.160 align:middle line:84% by telling the computer that it's all big continuous numbers 00:56:22.160 --> 00:56:23.435 align:middle line:90% when it's just a 0 or 1? 00:56:23.435 --> 00:56:25.560 align:middle line:84% There are ways to optimize that, but these problems 00:56:25.560 --> 00:56:27.947 align:middle line:84% are so small we just don't worry about it. 00:56:27.947 --> 00:56:30.280 align:middle line:84% But when we come to something called parameter-efficient 00:56:30.280 --> 00:56:33.680 align:middle line:84% fine-tuning, lecture maybe 10-ish, 00:56:33.680 --> 00:56:36.180 align:middle line:84% we will actually exploit that particular fact to make things 00:56:36.180 --> 00:56:38.873 align:middle line:90% faster. 00:56:38.873 --> 00:56:40.900 align:middle line:90% So that's what we have. 00:56:40.900 --> 00:56:44.292 align:middle line:84% So we will do the Bag of Words model. 00:56:44.292 --> 00:56:46.250 align:middle line:84% By the way, there's a whole bunch of stuff here 00:56:46.250 --> 00:56:48.750 align:middle line:84% that just repeats what I've been telling you in the lecture. 00:56:48.750 --> 00:56:50.460 align:middle line:84% So feel free to read it again, but we 00:56:50.460 --> 00:56:52.420 align:middle line:90% can ignore it for the moment. 00:56:52.420 --> 00:56:55.300 align:middle line:84% And now, there is a new thing we are doing here. 00:56:55.300 --> 00:56:57.760 align:middle line:84% So we are basically saying, look, 00:56:57.760 --> 00:57:01.860 align:middle line:84% instead of taking every word you see in these 49,000 songs 00:57:01.860 --> 00:57:05.360 align:middle line:84% in the training corpus, it's going to be too many words, 00:57:05.360 --> 00:57:09.140 align:middle line:84% just pick the 5,000 most frequent words. 00:57:09.140 --> 00:57:13.020 align:middle line:84% And that's what this max_tokens stands for. 00:57:13.020 --> 00:57:16.380 align:middle line:84% And so we tell it, all right, do this thing, 00:57:16.380 --> 00:57:21.360 align:middle line:84% max_tokens 5,000 and still do multi-hot. 00:57:21.360 --> 00:57:23.620 align:middle line:84% And we are not explicitly saying the standardization 00:57:23.620 --> 00:57:25.260 align:middle line:84% and all that stuff because the defaults 00:57:25.260 --> 00:57:27.060 align:middle line:90% are what we're going with. 00:57:27.060 --> 00:57:29.140 align:middle line:90% Yeah. 00:57:29.140 --> 00:57:31.220 align:middle line:84% AUDIENCE: This is for making it more efficient? 00:57:31.220 --> 00:57:32.870 align:middle line:84% This is, like, don't waste your time 00:57:32.870 --> 00:57:35.630 align:middle line:90% on these thousands of words. 00:57:35.630 --> 00:57:37.370 align:middle line:90% Use them more. 00:57:37.370 --> 00:57:39.553 align:middle line:84% Just focus on that to make it more efficient. 00:57:39.553 --> 00:57:41.470 align:middle line:84% RAMA RAMAKRISHNAN: It makes it more efficient, 00:57:41.470 --> 00:57:43.750 align:middle line:84% but there is a related and important point, which 00:57:43.750 --> 00:57:46.310 align:middle line:84% is that, fundamentally, the number of tokens 00:57:46.310 --> 00:57:49.590 align:middle line:84% you allow this layer to have dictates 00:57:49.590 --> 00:57:51.190 align:middle line:90% the size of your vocabulary. 00:57:51.190 --> 00:57:54.310 align:middle line:84% And the size of your vocabulary dictates the size of the vector 00:57:54.310 --> 00:57:55.167 align:middle line:90% that you feed in. 00:57:55.167 --> 00:57:57.250 align:middle line:84% So shorter vectors are better than longer vectors. 00:57:57.250 --> 00:57:58.570 align:middle line:90% That's the efficiency point. 00:57:58.570 --> 00:58:01.670 align:middle line:84% The other point is that the longer the input vector, 00:58:01.670 --> 00:58:03.330 align:middle line:84% the more the number of parameters 00:58:03.330 --> 00:58:06.110 align:middle line:84% the network has to learn because the first layer itself 00:58:06.110 --> 00:58:08.390 align:middle line:84% is the size of the input roughly times 00:58:08.390 --> 00:58:09.610 align:middle line:90% the size of the hidden layer. 00:58:09.610 --> 00:58:11.010 align:middle line:84% So this thing becomes 10 times as long, 00:58:11.010 --> 00:58:12.927 align:middle line:84% you have 10 times as many parameters to learn. 00:58:12.927 --> 00:58:15.390 align:middle line:84% And given a finite amount of data, 00:58:15.390 --> 00:58:17.110 align:middle line:84% the more parameters you have, the worse 00:58:17.110 --> 00:58:18.350 align:middle line:84% it's going to do when you actually start 00:58:18.350 --> 00:58:19.475 align:middle line:90% using it in the real world. 00:58:19.475 --> 00:58:20.730 align:middle line:90% It's going to overfit heavily. 00:58:20.730 --> 00:58:22.355 align:middle line:84% That's why you need to be very careful. 00:58:22.355 --> 00:58:25.590 align:middle line:90% 00:58:25.590 --> 00:58:27.433 align:middle line:90% Yeah. 00:58:27.433 --> 00:58:29.350 align:middle line:84% AUDIENCE: So have you downloaded the data set? 00:58:29.350 --> 00:58:32.110 align:middle line:84% Are you still using the vocabulary of the 17 words, 00:58:32.110 --> 00:58:33.348 align:middle line:90% or did you broaden it? 00:58:33.348 --> 00:58:35.390 align:middle line:84% RAMA RAMAKRISHNAN: No, no, that was just for fun. 00:58:35.390 --> 00:58:37.370 align:middle line:84% I'm going to actually build the vocabulary now. 00:58:37.370 --> 00:58:38.070 align:middle line:90% It's coming. 00:58:38.070 --> 00:58:40.130 align:middle line:90% Yeah, good question. 00:58:40.130 --> 00:58:41.770 align:middle line:90% So, let's do that. 00:58:41.770 --> 00:58:44.850 align:middle line:90% So first, I define this layer. 00:58:44.850 --> 00:58:46.510 align:middle line:90% OK, I just defined it. 00:58:46.510 --> 00:58:49.130 align:middle line:84% Now, we actually build the vocabulary by essentially 00:58:49.130 --> 00:58:55.010 align:middle line:84% telling it to adapt the layer using essentially the full all 00:58:55.010 --> 00:58:57.890 align:middle line:84% basically 49,000 songs in the training data set. 00:58:57.890 --> 00:59:00.210 align:middle line:90% That's a long list of songs. 00:59:00.210 --> 00:59:02.090 align:middle line:84% As far as Keras is concerned, you're just 00:59:02.090 --> 00:59:03.543 align:middle line:90% looking for a list of strings. 00:59:03.543 --> 00:59:05.210 align:middle line:84% So you just give it the list of strings. 00:59:05.210 --> 00:59:07.270 align:middle line:84% Instead of four, we're giving it 49,000. 00:59:07.270 --> 00:59:11.450 align:middle line:84% The same philosophy applies, so we run it. 00:59:11.450 --> 00:59:13.890 align:middle line:84% It's obviously going to take a few seconds to do that 00:59:13.890 --> 00:59:16.730 align:middle line:90% because it's 49,000 songs. 00:59:16.730 --> 00:59:18.410 align:middle line:90% Oh, it's about five seconds. 00:59:18.410 --> 00:59:21.730 align:middle line:84% All right, let's look at the most common 20. 00:59:21.730 --> 00:59:24.730 align:middle line:84% We get the vocabulary from our layer. 00:59:24.730 --> 00:59:27.230 align:middle line:84% See, once you adapt the layer and it's built a vocabulary, 00:59:27.230 --> 00:59:28.730 align:middle line:84% the layer has sort of been populated 00:59:28.730 --> 00:59:31.060 align:middle line:84% with all this information, so you can query it. 00:59:31.060 --> 00:59:32.320 align:middle line:90% So you can get the vocab. 00:59:32.320 --> 00:59:36.380 align:middle line:84% Top 20 words-- UNK; "the", most frequent word, no surprise; 00:59:36.380 --> 00:59:38.340 align:middle line:90% "you", "I", blah, blah, blah. 00:59:38.340 --> 00:59:40.940 align:middle line:90% Let's look at the last few. 00:59:40.940 --> 00:59:46.780 align:middle line:84% "Dagger", "cheddar", "terrified". 00:59:46.780 --> 00:59:51.340 align:middle line:84% Moving on, and then once we have done that, 00:59:51.340 --> 00:59:53.500 align:middle line:84% now we actually can vectorize all the data sets 00:59:53.500 --> 00:59:54.947 align:middle line:90% we have using this. 00:59:54.947 --> 00:59:56.780 align:middle line:84% And by vectorize, you mean take every string 00:59:56.780 --> 00:59:59.660 align:middle line:84% and create the multi-hot encoded vector from it. 00:59:59.660 --> 01:00:00.435 align:middle line:90% Yeah. 01:00:00.435 --> 01:00:03.060 align:middle line:84% AUDIENCE: Are we doing S-T-I-E because we're keeping stuff like 01:00:03.060 --> 01:00:04.103 align:middle line:90% D-E-A, et cetera? 01:00:04.103 --> 01:00:06.520 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, we're not strictly doing S-T-I-E, 01:00:06.520 --> 01:00:09.580 align:middle line:84% or to put it differently, typically S has 01:00:09.580 --> 01:00:12.740 align:middle line:84% lowercase/uppercase, strip punctuation, stemming, 01:00:12.740 --> 01:00:14.000 align:middle line:90% stop word removal. 01:00:14.000 --> 01:00:17.400 align:middle line:84% Here, the default in Keras happens to not do stemming, 01:00:17.400 --> 01:00:18.560 align:middle line:90% not do stop word removal. 01:00:18.560 --> 01:00:20.102 align:middle line:84% So we're just going with the default. 01:00:20.102 --> 01:00:21.918 align:middle line:90% Thanks for the clarification. 01:00:21.918 --> 01:00:23.460 align:middle line:84% And in fact, in practice, what I find 01:00:23.460 --> 01:00:25.508 align:middle line:84% these days is that don't even bother to stem. 01:00:25.508 --> 01:00:27.300 align:middle line:84% Don't even bother to remove the stop words. 01:00:27.300 --> 01:00:28.740 align:middle line:90% It's going to work well enough. 01:00:28.740 --> 01:00:33.360 align:middle line:90% 01:00:33.360 --> 01:00:35.420 align:middle line:84% OK, so now, each phrase is a vector. 01:00:35.420 --> 01:00:37.192 align:middle line:90% How long is this vector? 01:00:37.192 --> 01:00:38.460 align:middle line:90% Each song is now a vector. 01:00:38.460 --> 01:00:41.040 align:middle line:90% How long is that vector? 01:00:41.040 --> 01:00:43.860 align:middle line:84% 5,000, correct, because that is the size of the vocabulary. 01:00:43.860 --> 01:00:44.360 align:middle line:90% Correct. 01:00:44.360 --> 01:00:47.200 align:middle line:90% 01:00:47.200 --> 01:00:49.500 align:middle line:84% It's max tokens long, which is 5,000. 01:00:49.500 --> 01:00:51.025 align:middle line:90% So if you actually look at-- 01:00:51.025 --> 01:00:52.400 align:middle line:84% oh, wait, wait, wait, wait, wait. 01:00:52.400 --> 01:00:53.780 align:middle line:90% I haven't run this thing yet. 01:00:53.780 --> 01:00:57.960 align:middle line:90% 01:00:57.960 --> 01:00:59.260 align:middle line:90% It's going through 49,000. 01:00:59.260 --> 01:01:01.420 align:middle line:84% It's going through another 23,000. 01:01:01.420 --> 01:01:02.080 align:middle line:90% Fine. 01:01:02.080 --> 01:01:04.400 align:middle line:90% So let's run it. 01:01:04.400 --> 01:01:07.520 align:middle line:84% OK, now we can see x-train, which is all the training data 01:01:07.520 --> 01:01:12.600 align:middle line:84% you have, is a tensor, is a table, with 48,991 rows. 01:01:12.600 --> 01:01:18.040 align:middle line:84% And each row is a 5,000-long vector. 01:01:18.040 --> 01:01:19.660 align:middle line:90% All right, good. 01:01:19.660 --> 01:01:21.880 align:middle line:84% Now, we will try this simple neural network 01:01:21.880 --> 01:01:24.800 align:middle line:90% that we wrote up in class. 01:01:24.800 --> 01:01:29.170 align:middle line:84% And now, at this point, this code 01:01:29.170 --> 01:01:31.610 align:middle line:90% should be sort of second-nature. 01:01:31.610 --> 01:01:33.130 align:middle line:90% Isn't that cool? 01:01:33.130 --> 01:01:35.610 align:middle line:90% It's so easy to write the thing. 01:01:35.610 --> 01:01:37.110 align:middle line:90% The power of abstraction. 01:01:37.110 --> 01:01:39.250 align:middle line:90% So we take Keras' write input. 01:01:39.250 --> 01:01:40.210 align:middle line:90% As usual, input layer. 01:01:40.210 --> 01:01:42.710 align:middle line:84% We tell it, what is the size of each thing that's coming in? 01:01:42.710 --> 01:01:45.790 align:middle line:84% Well, the size of each thing is a 50 max_tokens long vector. 01:01:45.790 --> 01:01:47.730 align:middle line:84% So we tell it the shape is max_tokens, 01:01:47.730 --> 01:01:51.050 align:middle line:84% and then we run it through a dense layer with eight ReLUs. 01:01:51.050 --> 01:01:53.676 align:middle line:90% OK, I'm hurrying. 01:01:53.676 --> 01:01:55.230 align:middle line:90% So we get the outputs. 01:01:55.230 --> 01:01:57.817 align:middle line:84% Then, we string the inputs and the outputs into a model, 01:01:57.817 --> 01:01:59.150 align:middle line:90% and then we summarize the model. 01:01:59.150 --> 01:01:59.650 align:middle line:90% That's it. 01:01:59.650 --> 01:02:00.970 align:middle line:90% So we go here. 01:02:00.970 --> 01:02:02.790 align:middle line:90% And this has 40,000 parameters. 01:02:02.790 --> 01:02:05.990 align:middle line:84% And you can see here, when you go from the input, 01:02:05.990 --> 01:02:10.650 align:middle line:84% the 5,000 times 8, that gives you 40,000 plus the 8 neurons 01:02:10.650 --> 01:02:11.630 align:middle line:90% have a bias coming in. 01:02:11.630 --> 01:02:12.505 align:middle line:90% That's another eight. 01:02:12.505 --> 01:02:15.010 align:middle line:90% So you get 40,008. 01:02:15.010 --> 01:02:16.430 align:middle line:90% And we compile it as usual. 01:02:16.430 --> 01:02:17.910 align:middle line:90% We use Adam as usual. 01:02:17.910 --> 01:02:22.530 align:middle line:84% And because now the output y variable, the y-train variable, 01:02:22.530 --> 01:02:27.790 align:middle line:84% is now itself actually one-hot encoded, 010, 01:02:27.790 --> 01:02:30.710 align:middle line:84% 001 depending on pop, rock, and so on and so forth. 01:02:30.710 --> 01:02:33.290 align:middle line:84% We don't use sparse categorical cross-entropy. 01:02:33.290 --> 01:02:36.710 align:middle line:84% We just use plain, old categorical cross-entropy here. 01:02:36.710 --> 01:02:40.010 align:middle line:84% And this was explained in lecture last week, 01:02:40.010 --> 01:02:43.050 align:middle line:84% so you can revisit it if it's not familiar. 01:02:43.050 --> 01:02:44.330 align:middle line:90% We again report accuracy. 01:02:44.330 --> 01:02:45.830 align:middle line:90% So let's compile it. 01:02:45.830 --> 01:02:47.590 align:middle line:90% And we get a model. 01:02:47.590 --> 01:02:50.590 align:middle line:84% So we'll just run it for 10 epochs with a batch size of 32. 01:02:50.590 --> 01:02:53.065 align:middle line:84% And because we have validation data already supplied to us, 01:02:53.065 --> 01:02:55.190 align:middle line:84% we don't have to tell Keras, take the training data 01:02:55.190 --> 01:02:57.690 align:middle line:84% and keep 20% of it aside for validation. 01:02:57.690 --> 01:02:59.690 align:middle line:84% We can literally tell it what validation to use. 01:02:59.690 --> 01:03:00.898 align:middle line:90% That's what we're doing here. 01:03:00.898 --> 01:03:03.910 align:middle line:90% 01:03:03.910 --> 01:03:05.500 align:middle line:90% All right, so it's running. 01:03:05.500 --> 01:03:09.092 align:middle line:90% 01:03:09.092 --> 01:03:09.800 align:middle line:90% It's pretty fast. 01:03:09.800 --> 01:03:16.310 align:middle line:90% 01:03:16.310 --> 01:03:18.150 align:middle line:90% Any questions so far? 01:03:18.150 --> 01:03:20.070 align:middle line:90% Yes. 01:03:20.070 --> 01:03:23.380 align:middle line:90% Use the microphone, please. 01:03:23.380 --> 01:03:25.380 align:middle line:84% AUDIENCE: How do we decide the max total length? 01:03:25.380 --> 01:03:27.120 align:middle line:84% We defined the number of 5,000 here, 01:03:27.120 --> 01:03:29.120 align:middle line:84% but we do not know how many words would be there 01:03:29.120 --> 01:03:29.940 align:middle line:90% in the entire text. 01:03:29.940 --> 01:03:31.020 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, so it's a good question. 01:03:31.020 --> 01:03:33.160 align:middle line:84% How do you decide on this maximum vocabulary? 01:03:33.160 --> 01:03:34.660 align:middle line:84% What do you typically do in practice 01:03:34.660 --> 01:03:37.740 align:middle line:84% is that you actually do it without the max tokens, 01:03:37.740 --> 01:03:39.580 align:middle line:84% and then you see how long the vocabulary is, 01:03:39.580 --> 01:03:41.247 align:middle line:84% and then you can actually get statistics 01:03:41.247 --> 01:03:44.000 align:middle line:84% on how frequently the very infrequent words actually 01:03:44.000 --> 01:03:44.540 align:middle line:90% show up. 01:03:44.540 --> 01:03:48.300 align:middle line:84% And then you'll typically see a dramatic fall off at some point. 01:03:48.300 --> 01:03:49.965 align:middle line:84% And you pick that fall-off point, 01:03:49.965 --> 01:03:51.590 align:middle line:84% and then set that to be the max tokens. 01:03:51.590 --> 01:03:57.160 align:middle line:90% 01:03:57.160 --> 01:03:57.700 align:middle line:90% Perfect. 01:03:57.700 --> 01:03:58.960 align:middle line:90% Let's test it. 01:03:58.960 --> 01:04:03.560 align:middle line:84% Accuracy is pretty good, 87% on the training and 73 01:04:03.560 --> 01:04:05.000 align:middle line:90% on the validation. 01:04:05.000 --> 01:04:07.440 align:middle line:90% We'll do it on the test set. 01:04:07.440 --> 01:04:09.240 align:middle line:90% All right, 72%. 01:04:09.240 --> 01:04:12.160 align:middle line:84% So we saw earlier the largest class of the three-way 01:04:12.160 --> 01:04:14.020 align:middle line:90% is rock, with around 50%. 01:04:14.020 --> 01:04:16.485 align:middle line:84% So the naive model is going to get 50% accuracy, 01:04:16.485 --> 01:04:18.360 align:middle line:84% and this little neural network model gets you 01:04:18.360 --> 01:04:20.040 align:middle line:90% 72%, which is pretty nice. 01:04:20.040 --> 01:04:22.860 align:middle line:84% So now, let's actually kick it up a notch 01:04:22.860 --> 01:04:25.220 align:middle line:84% and make it slightly more capable. 01:04:25.220 --> 01:04:29.140 align:middle line:84% So the key thing here is that, as has been observed 01:04:29.140 --> 01:04:31.720 align:middle line:84% in class already, when you go with a Bag of Words model, 01:04:31.720 --> 01:04:33.980 align:middle line:90% we lose all notion of order. 01:04:33.980 --> 01:04:36.580 align:middle line:84% The word order clearly matters, and we're kind of ignoring it. 01:04:36.580 --> 01:04:40.100 align:middle line:84% So what we do to get around it is-- 01:04:40.100 --> 01:04:43.140 align:middle line:84% this is actually really interesting sentence here. 01:04:43.140 --> 01:04:44.660 align:middle line:84% Let's say this is a movie review. 01:04:44.660 --> 01:04:48.100 align:middle line:84% "Kate Winslet's performance as a detective trying to solve 01:04:48.100 --> 01:04:50.940 align:middle line:84% a terrible crime in a small Pennsylvania town is anything 01:04:50.940 --> 01:04:52.820 align:middle line:90% but disappointing". 01:04:52.820 --> 01:04:55.153 align:middle line:84% Tricky, tricky thing, because if you look at the words 01:04:55.153 --> 01:04:57.320 align:middle line:84% separately, the word "terrible" and "disappointing", 01:04:57.320 --> 01:04:59.940 align:middle line:84% you're going to be, like, negative sentiment. 01:04:59.940 --> 01:05:04.900 align:middle line:84% But then if you actually know that the word terrible refers 01:05:04.900 --> 01:05:08.060 align:middle line:84% to the crime, not to the movie or anything but disappointing 01:05:08.060 --> 01:05:10.560 align:middle line:84% changes the meaning of the word disappointing, you will see, 01:05:10.560 --> 01:05:12.300 align:middle line:84% obviously, it's a positive review. 01:05:12.300 --> 01:05:15.540 align:middle line:84% So clearly, the words around the word 01:05:15.540 --> 01:05:19.100 align:middle line:84% provide valuable clues as to how to interpret that word. 01:05:19.100 --> 01:05:23.830 align:middle line:84% And so what we do is, how can we make our little model a bit more 01:05:23.830 --> 01:05:27.130 align:middle line:84% capable of recognizing the context around every word? 01:05:27.130 --> 01:05:29.590 align:middle line:84% And the way we do it is something called bigrams. 01:05:29.590 --> 01:05:33.110 align:middle line:84% And for bigrams, what we basically 01:05:33.110 --> 01:05:37.470 align:middle line:84% do is instead of just taking each word, we take each word 01:05:37.470 --> 01:05:42.310 align:middle line:84% and we further take every pair of adjacent words, 01:05:42.310 --> 01:05:43.970 align:middle line:90% and those become our tokens. 01:05:43.970 --> 01:05:48.250 align:middle line:84% And because we take two adjacent words, they are called bigrams. 01:05:48.250 --> 01:05:50.650 align:middle line:84% You can take three adjacent words, trigrams. 01:05:50.650 --> 01:05:52.610 align:middle line:90% You get the idea, n-grams. 01:05:52.610 --> 01:05:54.390 align:middle line:90% So that's the idea of bigrams. 01:05:54.390 --> 01:05:59.870 align:middle line:84% And so, for example, if had "the cat sat on the mat", 01:05:59.870 --> 01:06:03.330 align:middle line:84% you will have "the", "the cat", "cat", "cat sat", "sat". 01:06:03.330 --> 01:06:05.070 align:middle line:90% You get the idea. 01:06:05.070 --> 01:06:06.090 align:middle line:90% That's what we have. 01:06:06.090 --> 01:06:07.562 align:middle line:90% So let's do a little example. 01:06:07.562 --> 01:06:08.770 align:middle line:90% And Keras makes it very easy. 01:06:08.770 --> 01:06:13.270 align:middle line:84% You literally tell it n-grams equals 2, bigrams. 01:06:13.270 --> 01:06:15.470 align:middle line:84% And now, from this, you immediately 01:06:15.470 --> 01:06:19.050 align:middle line:84% should know that n-grams equals 1 is the default. 01:06:19.050 --> 01:06:21.236 align:middle line:84% That's why we didn't have to specify it. 01:06:21.236 --> 01:06:26.170 align:middle line:84% So, you run it, and then you do "cat sat on the mat" 01:06:26.170 --> 01:06:27.457 align:middle line:90% as your training corpus. 01:06:27.457 --> 01:06:28.790 align:middle line:90% And then you get the vocabulary. 01:06:28.790 --> 01:06:32.170 align:middle line:84% And you can see here, it has created all these nice bigrams 01:06:32.170 --> 01:06:33.770 align:middle line:90% for you. 01:06:33.770 --> 01:06:34.710 align:middle line:90% And so that's it. 01:06:34.710 --> 01:06:37.210 align:middle line:84% All right, now, what we do is we go back to the songs, 01:06:37.210 --> 01:06:40.550 align:middle line:84% and we actually tell Keras to not just take each word, 01:06:40.550 --> 01:06:42.350 align:middle line:84% but take all the bigrams as well. 01:06:42.350 --> 01:06:45.010 align:middle line:84% And hopefully, it'll do a better job of figuring out 01:06:45.010 --> 01:06:46.230 align:middle line:90% what the sentiment is. 01:06:46.230 --> 01:06:51.390 align:middle line:84% And now, because when you say, OK, take the top 5,000 words, 01:06:51.390 --> 01:06:54.450 align:middle line:84% that's great for single unigrams, as they are called, 01:06:54.450 --> 01:06:57.250 align:middle line:84% but when you have bigrams, you have 5,000 possibilities 01:06:57.250 --> 01:06:59.890 align:middle line:84% of the first word, maybe 5,000 for the second word. 01:06:59.890 --> 01:07:01.672 align:middle line:84% That's a lot of possibilities, 25 million. 01:07:01.672 --> 01:07:04.130 align:middle line:84% Now, most of the 25 million possibilities are going to show 01:07:04.130 --> 01:07:06.547 align:middle line:84% up in the data, so you don't need to actually make it much 01:07:06.547 --> 01:07:08.970 align:middle line:84% larger, but you should make the vocabulary a bit more than 01:07:08.970 --> 01:07:09.910 align:middle line:90% 5,000. 01:07:09.910 --> 01:07:12.800 align:middle line:84% So here, we go with, say, 20,000. 01:07:12.800 --> 01:07:14.550 align:middle line:84% Otherwise, it's the same, still multi-hot. 01:07:14.550 --> 01:07:15.850 align:middle line:90% So let's run it. 01:07:15.850 --> 01:07:19.133 align:middle line:84% And now, we will run this now that the layer has been set up 01:07:19.133 --> 01:07:20.550 align:middle line:84% with all the right settings, we'll 01:07:20.550 --> 01:07:23.530 align:middle line:84% ask it to create the vocabulary, again, 01:07:23.530 --> 01:07:25.310 align:middle line:84% by doing exactly what we did before. 01:07:25.310 --> 01:07:26.450 align:middle line:90% Create the vocabulary. 01:07:26.450 --> 01:07:29.467 align:middle line:90% 01:07:29.467 --> 01:07:30.300 align:middle line:90% [INAUDIBLE] seconds. 01:07:30.300 --> 01:07:41.870 align:middle line:90% 01:07:41.870 --> 01:07:45.510 align:middle line:84% Bigrams, trigrams, all of them will get much more 01:07:45.510 --> 01:07:46.410 align:middle line:90% compute-intensive. 01:07:46.410 --> 01:07:47.710 align:middle line:90% That's why you're seeing this. 01:07:47.710 --> 01:07:50.670 align:middle line:84% So, all right, let's look at the first 10 words. 01:07:50.670 --> 01:07:53.010 align:middle line:84% The first 10 words are all just single words, 01:07:53.010 --> 01:07:54.790 align:middle line:84% and that's not surprising because the single words are 01:07:54.790 --> 01:07:56.040 align:middle line:90% going to be the more frequent. 01:07:56.040 --> 01:07:59.182 align:middle line:90% 01:07:59.182 --> 01:08:02.470 align:middle line:84% And then let's look at the last few. 01:08:02.470 --> 01:08:05.510 align:middle line:84% "Your mom", "your God", "you shot", "you hell". 01:08:05.510 --> 01:08:09.870 align:middle line:90% 01:08:09.870 --> 01:08:15.760 align:middle line:84% All right, let's just index all the data we have, 01:08:15.760 --> 01:08:18.300 align:middle line:84% the training validation test sets, using this vocabulary. 01:08:18.300 --> 01:08:23.180 align:middle line:90% 01:08:23.180 --> 01:08:23.680 align:middle line:90% Perfect. 01:08:23.680 --> 01:08:25.055 align:middle line:84% Now, we come to our second model, 01:08:25.055 --> 01:08:28.300 align:middle line:84% where we say the incoming shape is now 20,000 long, 01:08:28.300 --> 01:08:31.160 align:middle line:84% because we increased max tokens from 5,000 to 20,000. 01:08:31.160 --> 01:08:33.439 align:middle line:84% So each thing is a 20,000-long vector. 01:08:33.439 --> 01:08:34.760 align:middle line:90% Otherwise, it's the same. 01:08:34.760 --> 01:08:37.160 align:middle line:84% And now, we will use this thing called dropout 01:08:37.160 --> 01:08:40.760 align:middle line:84% for the first time, which is a regularization thing that I have 01:08:40.760 --> 01:08:43.520 align:middle line:84% referred to earlier that I never really described, 01:08:43.520 --> 01:08:45.790 align:middle line:84% and I will describe today if we have time. 01:08:45.790 --> 01:08:47.540 align:middle line:84% But I'll first run through the whole demo. 01:08:47.540 --> 01:08:51.359 align:middle line:84% So you can just think of dropout as just another layer 01:08:51.359 --> 01:08:52.340 align:middle line:90% you can insert. 01:08:52.340 --> 01:08:54.840 align:middle line:84% And it's essentially a great way to prevent overfitting. 01:08:54.840 --> 01:08:57.279 align:middle line:90% So I just routinely will use it. 01:08:57.279 --> 01:08:58.620 align:middle line:90% And I'll talk more about it. 01:08:58.620 --> 01:09:01.180 align:middle line:84% So for now, we have this dropout layer in the middle. 01:09:01.180 --> 01:09:03.157 align:middle line:84% It receives the input from the dense layer 01:09:03.157 --> 01:09:04.740 align:middle line:84% and then sends it to the output layer. 01:09:04.740 --> 01:09:05.990 align:middle line:90% The output layer is unchanged. 01:09:05.990 --> 01:09:09.279 align:middle line:84% It's a three-way softmax, same model as before. 01:09:09.279 --> 01:09:11.840 align:middle line:84% And now, all right, I'll come back to dropout. 01:09:11.840 --> 01:09:13.680 align:middle line:84% So we'll compile it the same way as before. 01:09:13.680 --> 01:09:17.260 align:middle line:84% And then I will just fit it for three epochs. 01:09:17.260 --> 01:09:19.100 align:middle line:84% If you're interested after class later on, 01:09:19.100 --> 01:09:20.725 align:middle line:84% you can actually try it for more epochs 01:09:20.725 --> 01:09:22.024 align:middle line:90% and see if it does better. 01:09:22.024 --> 01:09:23.399 align:middle line:84% For now, in the interest of time, 01:09:23.399 --> 01:09:24.524 align:middle line:90% we'll just do it for three. 01:09:24.524 --> 01:09:36.219 align:middle line:90% 01:09:36.219 --> 01:09:43.314 align:middle line:84% I think 72% was the single-word unigram thing we had. 01:09:43.314 --> 01:09:44.939 align:middle line:84% AUDIENCE: If you're rerunning this code 01:09:44.939 --> 01:09:46.740 align:middle line:84% with the same number of epochs, do you ever 01:09:46.740 --> 01:09:49.357 align:middle line:90% expect the accuracy to change? 01:09:49.357 --> 01:09:50.899 align:middle line:84% RAMA RAMAKRISHNAN: If you were to run 01:09:50.899 --> 01:09:53.020 align:middle line:84% this code on your machine, you would 01:09:53.020 --> 01:09:54.472 align:middle line:84% expect it to be roughly the same, 01:09:54.472 --> 01:09:56.180 align:middle line:84% but there are some minute differences due 01:09:56.180 --> 01:09:58.193 align:middle line:84% to hardware and device drivers and things like. 01:09:58.193 --> 01:10:00.360 align:middle line:84% AUDIENCE: If you reran it on your own machine twice, 01:10:00.360 --> 01:10:02.900 align:middle line:84% would you ever expect the accuracy to change? 01:10:02.900 --> 01:10:05.580 align:middle line:84% RAMA RAMAKRISHNAN: That's actually a very tricky question, 01:10:05.580 --> 01:10:07.340 align:middle line:84% because it depends on what else I have 01:10:07.340 --> 01:10:09.620 align:middle line:90% been doing in that notebook. 01:10:09.620 --> 01:10:12.430 align:middle line:84% If I start fresh and do nothing but that, 01:10:12.430 --> 01:10:14.287 align:middle line:84% typically I get the same numbers, typically. 01:10:14.287 --> 01:10:16.370 align:middle line:84% But for some reason, I don't get it exactly right. 01:10:16.370 --> 01:10:19.270 align:middle line:90% 01:10:19.270 --> 01:10:21.090 align:middle line:90% OK, so we come to this. 01:10:21.090 --> 01:10:22.860 align:middle line:90% Let's evaluate our little model. 01:10:22.860 --> 01:10:25.830 align:middle line:90% 01:10:25.830 --> 01:10:27.330 align:middle line:90% OK, 75%. 01:10:27.330 --> 01:10:30.270 align:middle line:84% So it went from 72 to 75, which is actually a meaningful jump, 01:10:30.270 --> 01:10:31.790 align:middle line:90% just by using bigrams. 01:10:31.790 --> 01:10:33.950 align:middle line:84% And I ran it only for three epochs. 01:10:33.950 --> 01:10:36.870 align:middle line:84% If you run it for 10, maybe it's going to do even better. 01:10:36.870 --> 01:10:39.210 align:middle line:84% So that is the beauty of this thing. 01:10:39.210 --> 01:10:41.950 align:middle line:84% Now, let's just actually do a little demo. 01:10:41.950 --> 01:10:44.190 align:middle line:84% We'll try to predict some lyrics. 01:10:44.190 --> 01:10:46.730 align:middle line:84% I'll try "another one bites the dust". 01:10:46.730 --> 01:10:49.262 align:middle line:90% 01:10:49.262 --> 01:10:49.970 align:middle line:90% It's a rock song. 01:10:49.970 --> 01:10:52.110 align:middle line:90% I think that's correct, yes? 01:10:52.110 --> 01:10:56.110 align:middle line:90% OK, folks, your turn now. 01:10:56.110 --> 01:10:57.770 align:middle line:84% Somebody, tell me your favorite song. 01:10:57.770 --> 01:11:00.550 align:middle line:90% 01:11:00.550 --> 01:11:02.598 align:middle line:84% AUDIENCE: "Dancing Queen" from ABBA. 01:11:02.598 --> 01:11:03.890 align:middle line:90% RAMA RAMAKRISHNAN: I love ABBA. 01:11:03.890 --> 01:11:04.690 align:middle line:90% That's awesome. 01:11:04.690 --> 01:11:11.420 align:middle line:84% All right, OK, dancing queen lyrics. 01:11:11.420 --> 01:11:17.048 align:middle line:90% 01:11:17.048 --> 01:11:18.590 align:middle line:84% Verse one, intro-- I don't like that. 01:11:18.590 --> 01:11:21.390 align:middle line:84% Let's just go to something without all this metadata. 01:11:21.390 --> 01:11:27.353 align:middle line:90% 01:11:27.353 --> 01:11:28.770 align:middle line:84% I'll just take the first page, OK? 01:11:28.770 --> 01:11:38.050 align:middle line:90% 01:11:38.050 --> 01:11:40.490 align:middle line:90% Here. 01:11:40.490 --> 01:11:42.450 align:middle line:90% Are we good? 01:11:42.450 --> 01:11:46.210 align:middle line:84% All right, don't let me down, model. 01:11:46.210 --> 01:11:46.820 align:middle line:90% Let's predict. 01:11:46.820 --> 01:11:50.190 align:middle line:90% 01:11:50.190 --> 01:11:50.690 align:middle line:90% Pop. 01:11:50.690 --> 01:11:52.110 align:middle line:90% Just about. 01:11:52.110 --> 01:11:52.610 align:middle line:90% Yay! 01:11:52.610 --> 01:11:55.610 align:middle line:90% [APPLAUSE] 01:11:55.610 --> 01:11:56.570 align:middle line:90% All right. 01:11:56.570 --> 01:11:59.488 align:middle line:84% So, yeah, that's basically the model. 01:11:59.488 --> 01:12:00.530 align:middle line:90% But we have five minutes. 01:12:00.530 --> 01:12:02.655 align:middle line:84% I want to get back to-- you can play around and put 01:12:02.655 --> 01:12:03.890 align:middle line:90% your own lyrics in. 01:12:03.890 --> 01:12:06.210 align:middle line:84% Typically, what happens is that the last two years 01:12:06.210 --> 01:12:08.590 align:middle line:84% that I've been doing this particular lecture, 01:12:08.590 --> 01:12:10.700 align:middle line:84% I've noticed that the songs are always 01:12:10.700 --> 01:12:12.840 align:middle line:90% rock songs for some reason. 01:12:12.840 --> 01:12:15.340 align:middle line:84% It's the first time getting a pop song from a group 01:12:15.340 --> 01:12:18.500 align:middle line:84% that I actually like, so thank you. 01:12:18.500 --> 01:12:21.980 align:middle line:84% All right, let's go back to dropout. 01:12:21.980 --> 01:12:25.020 align:middle line:84% So the idea here in dropout is that you 01:12:25.020 --> 01:12:26.840 align:middle line:84% have all these-- the input comes in, 01:12:26.840 --> 01:12:30.480 align:middle line:84% it goes through a hidden layer, and so on and so forth. 01:12:30.480 --> 01:12:32.785 align:middle line:90% So dropout is a layer. 01:12:32.785 --> 01:12:35.160 align:middle line:84% And you put this layer just like you use any other layer. 01:12:35.160 --> 01:12:38.140 align:middle line:84% And what dropout does is that it takes all the things that 01:12:38.140 --> 01:12:40.300 align:middle line:84% are coming into it from the previous layer 01:12:40.300 --> 01:12:46.100 align:middle line:84% and randomly decides to replace that number with a zero. 01:12:46.100 --> 01:12:46.720 align:middle line:90% That's it. 01:12:46.720 --> 01:12:49.580 align:middle line:84% It drops that number and replaces it with zero. 01:12:49.580 --> 01:12:51.160 align:middle line:90% But it does it randomly. 01:12:51.160 --> 01:12:52.640 align:middle line:90% It basically will toss a coin. 01:12:52.640 --> 01:12:54.360 align:middle line:84% If the coin comes up heads, zero. 01:12:54.360 --> 01:12:57.460 align:middle line:84% If it comes up tails, let it through, pass it through. 01:12:57.460 --> 01:13:00.300 align:middle line:84% And the reason why this is very effective 01:13:00.300 --> 01:13:04.380 align:middle line:84% is because you can imagine all the neurons 01:13:04.380 --> 01:13:05.940 align:middle line:90% in a particular layer. 01:13:05.940 --> 01:13:09.040 align:middle line:84% When they overfit to a particular data set, 01:13:09.040 --> 01:13:11.360 align:middle line:84% the overfitting happens because the neurons essentially 01:13:11.360 --> 01:13:13.880 align:middle line:90% collude with each other. 01:13:13.880 --> 01:13:16.680 align:middle line:84% They collude with each other to actually overfit and predict 01:13:16.680 --> 01:13:19.040 align:middle line:90% things in a very accurate way. 01:13:19.040 --> 01:13:23.320 align:middle line:84% So you want to break any sort of collusion between the neurons. 01:13:23.320 --> 01:13:27.140 align:middle line:84% I'm obviously using a game theoretic way of describing it. 01:13:27.140 --> 01:13:30.440 align:middle line:84% But the idea is that any kind of spurious correlations 01:13:30.440 --> 01:13:33.160 align:middle line:84% in your data, neurons can pick it up 01:13:33.160 --> 01:13:36.080 align:middle line:90% by being correlated themselves. 01:13:36.080 --> 01:13:39.200 align:middle line:84% And so the way you avoid the spurious correlation is 01:13:39.200 --> 01:13:41.203 align:middle line:90% by dropping neurons randomly. 01:13:41.203 --> 01:13:42.620 align:middle line:84% You just kill the neuron randomly, 01:13:42.620 --> 01:13:45.400 align:middle line:84% which means that no neuron can depend on another neuron being 01:13:45.400 --> 01:13:47.720 align:middle line:90% available. 01:13:47.720 --> 01:13:50.120 align:middle line:84% I know, it's a bit grim, but that's 01:13:50.120 --> 01:13:51.720 align:middle line:90% the basic idea of dropout. 01:13:51.720 --> 01:13:54.840 align:middle line:84% And apparently, the story goes that the team that 01:13:54.840 --> 01:13:58.100 align:middle line:84% invented it, Jeff Hinton, who won the Turing for this stuff, 01:13:58.100 --> 01:14:00.745 align:middle line:84% not for the dropout, just for deep learning, 01:14:00.745 --> 01:14:03.120 align:middle line:84% I don't know if it's true, but he said that apparently he 01:14:03.120 --> 01:14:05.120 align:middle line:84% got the idea when he went to a bank 01:14:05.120 --> 01:14:08.890 align:middle line:84% and realized that, very often, the folks who are working 01:14:08.890 --> 01:14:13.130 align:middle line:84% in that bank branch that he used to go to kept changing. 01:14:13.130 --> 01:14:14.380 align:middle line:90% They were never the same. 01:14:14.380 --> 01:14:16.630 align:middle line:84% The people would be transferring in, transferring out. 01:14:16.630 --> 01:14:18.430 align:middle line:84% And he was like, why can't they just leave these people alone? 01:14:18.430 --> 01:14:19.513 align:middle line:90% Why does it keep changing? 01:14:19.513 --> 01:14:22.090 align:middle line:84% And then he got the insight that maybe a lot of fraud 01:14:22.090 --> 01:14:25.210 align:middle line:84% happens because the person working in the branch 01:14:25.210 --> 01:14:27.050 align:middle line:90% colludes with the customer. 01:14:27.050 --> 01:14:28.950 align:middle line:84% But by changing the staff constantly, 01:14:28.950 --> 01:14:31.430 align:middle line:84% you break the risk of fraud happening. 01:14:31.430 --> 01:14:34.450 align:middle line:84% And that apparently was the genesis for this idea. 01:14:34.450 --> 01:14:36.310 align:middle line:84% True, apocryphal, I have no idea, 01:14:36.310 --> 01:14:38.050 align:middle line:90% but it's sort of a fun story. 01:14:38.050 --> 01:14:40.530 align:middle line:90% Yes. 01:14:40.530 --> 01:14:42.050 align:middle line:84% AUDIENCE: Instead of random, if we 01:14:42.050 --> 01:14:44.450 align:middle line:84% go to the way historical models are built, 01:14:44.450 --> 01:14:46.010 align:middle line:84% the concepts of multicollinearity 01:14:46.010 --> 01:14:50.150 align:middle line:84% and all of that, would that make it sharper as compared to this? 01:14:50.150 --> 01:14:53.210 align:middle line:84% RAMA RAMAKRISHNAN: The problem is that these networks 01:14:53.210 --> 01:14:54.690 align:middle line:90% are massive. 01:14:54.690 --> 01:14:57.370 align:middle line:84% And for you to take each layer and look 01:14:57.370 --> 01:14:59.150 align:middle line:84% at its correlation with some other layer 01:14:59.150 --> 01:15:01.570 align:middle line:84% and so on and so forth, first of all, investigating 01:15:01.570 --> 01:15:04.170 align:middle line:90% multicollinearity is a problem. 01:15:04.170 --> 01:15:07.030 align:middle line:84% The second thing is, OK, what do you do then next? 01:15:07.030 --> 01:15:09.910 align:middle line:84% In linear regression, you can do things like principal components 01:15:09.910 --> 01:15:11.050 align:middle line:90% analysis to get around it. 01:15:11.050 --> 01:15:12.490 align:middle line:90% Here, everything is non-linear. 01:15:12.490 --> 01:15:14.290 align:middle line:84% There is no easy way to solve the problem. 01:15:14.290 --> 01:15:16.123 align:middle line:84% So we are like, we'll just solve the problem 01:15:16.123 --> 01:15:17.550 align:middle line:90% in one shot using dropout. 01:15:17.550 --> 01:15:18.790 align:middle line:90% That's the idea. 01:15:18.790 --> 01:15:24.070 align:middle line:84% All right, so I had some material on something 01:15:24.070 --> 01:15:29.092 align:middle line:84% called byte-pair encoding, which I will do when we get to LLMs. 01:15:29.092 --> 01:15:30.550 align:middle line:84% And I stuck it in the end because I 01:15:30.550 --> 01:15:32.508 align:middle line:84% knew that we probably wouldn't have enough time 01:15:32.508 --> 01:15:33.530 align:middle line:90% to cover this anyway. 01:15:33.530 --> 01:15:36.690 align:middle line:84% And that is a very clever tokenization scheme used by, 01:15:36.690 --> 01:15:38.950 align:middle line:84% for example, the GPT family, that 01:15:38.950 --> 01:15:41.010 align:middle line:84% allows them to do beautiful punctuation, 01:15:41.010 --> 01:15:44.150 align:middle line:84% keep the case intact, and then use words that you just made up 01:15:44.150 --> 01:15:45.050 align:middle line:90% and things like that. 01:15:45.050 --> 01:15:47.130 align:middle line:90% So we have one more minute. 01:15:47.130 --> 01:15:50.662 align:middle line:84% I'm happy to answer any questions you might have. 01:15:50.662 --> 01:15:52.870 align:middle line:84% AUDIENCE: So initially when we are picking the hidden 01:15:52.870 --> 01:15:55.910 align:middle line:84% layer, the number of neurons, and [INAUDIBLE] 01:15:55.910 --> 01:15:59.293 align:middle line:84% to so far all the materials this has been given to us. 01:15:59.293 --> 01:16:00.710 align:middle line:84% But initially, how do you pick it? 01:16:00.710 --> 01:16:03.597 align:middle line:84% Is it more of a trial and error type of thing? 01:16:03.597 --> 01:16:05.680 align:middle line:84% RAMA RAMAKRISHNAN: It tends to be trial and error. 01:16:05.680 --> 01:16:09.820 align:middle line:84% That's, in fact, what I did when I created the Colabs. 01:16:09.820 --> 01:16:12.800 align:middle line:84% And you can actually make it a bit more systematic by trying 01:16:12.800 --> 01:16:14.180 align:middle line:90% lots of different values. 01:16:14.180 --> 01:16:18.320 align:middle line:84% And there is a particular Python package called KerasTuner. 01:16:18.320 --> 01:16:21.423 align:middle line:84% So just Google KerasTuner, and it comes with very nice Colabs. 01:16:21.423 --> 01:16:22.840 align:middle line:84% And if I have a chance, maybe I'll 01:16:22.840 --> 01:16:25.100 align:middle line:84% just record a screen walkthrough of doing that. 01:16:25.100 --> 01:16:27.540 align:middle line:84% But that's the very efficient way to do these things. 01:16:27.540 --> 01:16:29.082 align:middle line:84% And it comes under the broad category 01:16:29.082 --> 01:16:31.200 align:middle line:84% of something called hyperparameter optimization, 01:16:31.200 --> 01:16:34.120 align:middle line:84% where the number of neurons, the activation you use, the learning 01:16:34.120 --> 01:16:36.220 align:middle line:84% rate, all those things can all be tried. 01:16:36.220 --> 01:16:37.740 align:middle line:90% You can try lots of variations. 01:16:37.740 --> 01:16:41.920 align:middle line:84% And Keras is a great way to do it in the context of Keras. 01:16:41.920 --> 01:16:42.820 align:middle line:90% Other questions? 01:16:42.820 --> 01:16:45.420 align:middle line:90% 01:16:45.420 --> 01:16:45.920 align:middle line:90% No? 01:16:45.920 --> 01:16:47.260 align:middle line:84% All right, I'll give you 30 seconds back. 01:16:47.260 --> 01:16:48.040 align:middle line:90% Thank you. 01:16:48.040 --> 01:16:49.830 align:middle line:90% See you tomorrow. 01:16:49.830 --> 01:17:03.000 align:middle line:90%