WEBVTT 00:00:00.000 --> 00:00:02.445 align:middle line:90% [SQUEAKING] 00:00:02.445 --> 00:00:03.912 align:middle line:90% [RUSTLING] 00:00:03.912 --> 00:00:06.846 align:middle line:90% [CLICKING] 00:00:06.846 --> 00:00:21.820 align:middle line:90% 00:00:21.820 --> 00:00:24.100 align:middle line:84% RAMA RAMAKRISHNAN: OK, so let's get going. 00:00:24.100 --> 00:00:26.880 align:middle line:84% Today, we're going to talk about how do you actually 00:00:26.880 --> 00:00:30.400 align:middle line:84% train a neural network, because that is the heart of the game 00:00:30.400 --> 00:00:31.560 align:middle line:90% here. 00:00:31.560 --> 00:00:35.000 align:middle line:84% So just to recap, we looked, last class, 00:00:35.000 --> 00:00:37.260 align:middle line:84% at what it takes to design a neural network, 00:00:37.260 --> 00:00:40.600 align:middle line:84% and we made this very important distinction between the things 00:00:40.600 --> 00:00:43.200 align:middle line:84% that you are handed by your problem and the things 00:00:43.200 --> 00:00:45.960 align:middle line:84% that you have agency over, that you have control over. 00:00:45.960 --> 00:00:48.800 align:middle line:84% And we noticed that the input layer 00:00:48.800 --> 00:00:51.400 align:middle line:84% for your problem, the input is the input, 00:00:51.400 --> 00:00:52.442 align:middle line:90% the output is the output. 00:00:52.442 --> 00:00:54.650 align:middle line:84% You've got to do something with the output, something 00:00:54.650 --> 00:00:55.620 align:middle line:90% that's expected. 00:00:55.620 --> 00:00:57.520 align:middle line:84% But everything that happens in the middle 00:00:57.520 --> 00:00:59.140 align:middle line:90% is actually in your hands. 00:00:59.140 --> 00:01:01.660 align:middle line:84% And in particular, we noticed that we 00:01:01.660 --> 00:01:05.019 align:middle line:84% have to decide how many hidden layers we want; 00:01:05.019 --> 00:01:08.580 align:middle line:84% we have to decide, in each layer, how many neurons to have; 00:01:08.580 --> 00:01:12.475 align:middle line:84% and then we have to decide what activation to use, 00:01:12.475 --> 00:01:15.100 align:middle line:84% even though I'm kind of cheating when I say that because I told 00:01:15.100 --> 00:01:17.820 align:middle line:84% you very clearly on Monday that for the hidden layer 00:01:17.820 --> 00:01:20.640 align:middle line:84% activations, just go with the ReLU activation function. 00:01:20.640 --> 00:01:23.230 align:middle line:84% You don't have to think deep thoughts about this. 00:01:23.230 --> 00:01:25.480 align:middle line:84% But the other things are all choices you have to make, 00:01:25.480 --> 00:01:28.100 align:middle line:84% and we will talk a bit later about how do you actually 00:01:28.100 --> 00:01:29.900 align:middle line:90% make those choices? 00:01:29.900 --> 00:01:30.400 align:middle line:90% OK. 00:01:30.400 --> 00:01:33.580 align:middle line:84% Now The rule of thumb, the rule of thumb 00:01:33.580 --> 00:01:35.820 align:middle line:84% always is to start with the simplest 00:01:35.820 --> 00:01:39.560 align:middle line:84% network you can think of, and if it gets the job done, 00:01:39.560 --> 00:01:41.500 align:middle line:90% stop working on it. 00:01:41.500 --> 00:01:45.180 align:middle line:84% If it's not good enough, make it slightly more complicated. 00:01:45.180 --> 00:01:48.340 align:middle line:84% So that's the meta thing you have to remember always when 00:01:48.340 --> 00:01:49.940 align:middle line:90% you're designing these things 00:01:49.940 --> 00:01:54.120 align:middle line:84% OK, so that's what it takes to design a deep neural network. 00:01:54.120 --> 00:01:56.100 align:middle line:84% So what we will do in this class is 00:01:56.100 --> 00:01:59.060 align:middle line:84% we'll actually take a real example with real data, 00:01:59.060 --> 00:02:01.760 align:middle line:84% and then we'll think through how we would design a network 00:02:01.760 --> 00:02:03.320 align:middle line:90% to solve this problem. 00:02:03.320 --> 00:02:05.880 align:middle line:84% And while doing so, we will cover a whole bunch 00:02:05.880 --> 00:02:09.132 align:middle line:84% of conceptual foundations such as optimization, 00:02:09.132 --> 00:02:10.840 align:middle line:84% loss functions, gradient descent, and all 00:02:10.840 --> 00:02:12.960 align:middle line:90% that good stuff. 00:02:12.960 --> 00:02:13.620 align:middle line:90% All right. 00:02:13.620 --> 00:02:17.640 align:middle line:84% So the case study, or the scenario here, 00:02:17.640 --> 00:02:20.720 align:middle line:84% is we have a data set of patients made available 00:02:20.720 --> 00:02:23.680 align:middle line:84% by the Cleveland Clinic, and essentially, we 00:02:23.680 --> 00:02:25.180 align:middle line:90% have a bunch of patients. 00:02:25.180 --> 00:02:26.920 align:middle line:84% And for all these patients, the setting 00:02:26.920 --> 00:02:29.280 align:middle line:84% is that they have come into the Cleveland Clinic, 00:02:29.280 --> 00:02:31.517 align:middle line:84% and they have not come in with a heart problem, 00:02:31.517 --> 00:02:33.100 align:middle line:84% they have come in with something else. 00:02:33.100 --> 00:02:34.840 align:middle line:84% Maybe they just came in for a physical. 00:02:34.840 --> 00:02:37.240 align:middle line:84% And we measured a whole bunch of things about them. 00:02:37.240 --> 00:02:39.560 align:middle line:84% And the kinds of things we measured 00:02:39.560 --> 00:02:43.878 align:middle line:84% are demographic information, like what's their age, gender, 00:02:43.878 --> 00:02:45.920 align:middle line:84% whether they have any chest pain at all when they 00:02:45.920 --> 00:02:49.420 align:middle line:84% came in, blood pressure, cholesterol, sugar, 00:02:49.420 --> 00:02:50.520 align:middle line:90% so on and so forth. 00:02:50.520 --> 00:02:51.780 align:middle line:90% You get the idea. 00:02:51.780 --> 00:02:55.140 align:middle line:84% Demographic information and a bunch of biomarker information. 00:02:55.140 --> 00:02:59.580 align:middle line:84% And then, what the Cleveland Clinic did was they actually 00:02:59.580 --> 00:03:04.100 align:middle line:84% tracked these people and figured out, in the next year, 00:03:04.100 --> 00:03:07.780 align:middle line:84% did they get diagnosed with heart disease or not? 00:03:07.780 --> 00:03:09.020 align:middle line:90% In the next year. 00:03:09.020 --> 00:03:11.330 align:middle line:84% Which means that maybe you can build a model, 00:03:11.330 --> 00:03:13.580 align:middle line:84% when someone comes in, even though they didn't come in 00:03:13.580 --> 00:03:15.313 align:middle line:84% for a chest problem, maybe you can 00:03:15.313 --> 00:03:16.980 align:middle line:84% predict that something's going to happen 00:03:16.980 --> 00:03:18.040 align:middle line:90% to them in the next year. 00:03:18.040 --> 00:03:23.020 align:middle line:84% It's a nice classic machine learning setup. 00:03:23.020 --> 00:03:23.620 align:middle line:90% All right. 00:03:23.620 --> 00:03:26.500 align:middle line:84% So this is the thing-- so what we want to do is we can totally 00:03:26.500 --> 00:03:28.880 align:middle line:84% solve this problem using decision trees neural net-- 00:03:28.880 --> 00:03:31.220 align:middle line:84% I mean, sorry, random forest and gradient boosting 00:03:31.220 --> 00:03:33.740 align:middle line:84% and all that good stuff you folks may have already learned 00:03:33.740 --> 00:03:35.460 align:middle line:84% from machine learning, but we will 00:03:35.460 --> 00:03:37.240 align:middle line:84% try to solve it using neural networks. 00:03:37.240 --> 00:03:39.522 align:middle line:84% This is an example, of course, of what's 00:03:39.522 --> 00:03:40.980 align:middle line:84% called structured data because this 00:03:40.980 --> 00:03:43.940 align:middle line:84% is all data sitting in the columns of a spreadsheet. 00:03:43.940 --> 00:03:46.140 align:middle line:84% So working with structured data is 00:03:46.140 --> 00:03:49.160 align:middle line:84% the way we warm up our knowledge of neural networks. 00:03:49.160 --> 00:03:51.900 align:middle line:84% And then we will do things like working with unstructured data, 00:03:51.900 --> 00:03:53.800 align:middle line:84% starting next week, with images, and then 00:03:53.800 --> 00:03:55.840 align:middle line:84% later on with text and so on and so forth. 00:03:55.840 --> 00:03:57.340 align:middle line:90% OK, any questions on this? 00:03:57.340 --> 00:04:00.560 align:middle line:90% 00:04:00.560 --> 00:04:01.320 align:middle line:90% OK. 00:04:01.320 --> 00:04:02.438 align:middle line:90% Yes? 00:04:02.438 --> 00:04:04.480 align:middle line:84% STUDENT: Just connected even to last time's class 00:04:04.480 --> 00:04:07.880 align:middle line:84% where we took the same example, and first it was a logistic, 00:04:07.880 --> 00:04:09.540 align:middle line:84% and then we did a neural network. 00:04:09.540 --> 00:04:13.380 align:middle line:84% So the probability in case of one was 0.85, then was 0.22, 00:04:13.380 --> 00:04:14.560 align:middle line:90% and here as well. 00:04:14.560 --> 00:04:18.240 align:middle line:84% How do you know when to use what? 00:04:18.240 --> 00:04:20.200 align:middle line:84% Usually in textbooks, when to use logistic 00:04:20.200 --> 00:04:25.600 align:middle line:84% or when to use something else, but in this case, 00:04:25.600 --> 00:04:28.440 align:middle line:84% when do I complicate it to neural networks vis-a-vis, 00:04:28.440 --> 00:04:30.220 align:middle line:84% in this case, maybe just doing a random? 00:04:30.220 --> 00:04:31.320 align:middle line:84% RAMA RAMAKRISHNAN: It's a great question. 00:04:31.320 --> 00:04:32.540 align:middle line:90% When do you use what? 00:04:32.540 --> 00:04:34.332 align:middle line:84% So I think there are two broad dimensions 00:04:34.332 --> 00:04:35.540 align:middle line:90% that you have to think about. 00:04:35.540 --> 00:04:38.560 align:middle line:84% One broad dimension is, how important 00:04:38.560 --> 00:04:41.120 align:middle line:84% is it that you need to explain or interpret 00:04:41.120 --> 00:04:43.760 align:middle line:84% what's going on inside the model to perhaps 00:04:43.760 --> 00:04:46.320 align:middle line:90% a non-technical consumer? 00:04:46.320 --> 00:04:48.160 align:middle line:84% The other dimension is, how important 00:04:48.160 --> 00:04:50.800 align:middle line:90% is sheer predictive accuracy? 00:04:50.800 --> 00:04:53.860 align:middle line:84% In some situations, predictive accuracy trumps everything else, 00:04:53.860 --> 00:04:55.458 align:middle line:90% in which case, just go with it. 00:04:55.458 --> 00:04:57.500 align:middle line:84% In other cases, explainability becomes a big deal 00:04:57.500 --> 00:05:00.273 align:middle line:84% because if they can't understand it, they won't use it. 00:05:00.273 --> 00:05:01.940 align:middle line:84% And in those cases, it's probably better 00:05:01.940 --> 00:05:05.320 align:middle line:84% to go with simpler models, such as decision trees and neural-- 00:05:05.320 --> 00:05:08.720 align:middle line:84% I mean, not-- decision trees, maybe even random forests, 00:05:08.720 --> 00:05:10.220 align:middle line:84% certainly logistic regression, those 00:05:10.220 --> 00:05:12.340 align:middle line:90% are all a little more amenable. 00:05:12.340 --> 00:05:15.500 align:middle line:84% But that said, even complex black box 00:05:15.500 --> 00:05:17.380 align:middle line:84% methods like neural networks, there 00:05:17.380 --> 00:05:20.820 align:middle line:84% is a whole field called mechanistic interpretability, 00:05:20.820 --> 00:05:23.820 align:middle line:84% which seeks to try to get insight into what's going on 00:05:23.820 --> 00:05:25.700 align:middle line:90% inside these big black boxes. 00:05:25.700 --> 00:05:30.280 align:middle line:84% So the story isn't over, but that's just the first-cut way 00:05:30.280 --> 00:05:33.170 align:middle line:90% you analyze the problem. 00:05:33.170 --> 00:05:33.670 align:middle line:90% OK. 00:05:33.670 --> 00:05:36.820 align:middle line:90% So, let's get going. 00:05:36.820 --> 00:05:39.740 align:middle line:84% So if you want to design a network-- 00:05:39.740 --> 00:05:40.280 align:middle line:90% all right. 00:05:40.280 --> 00:05:41.940 align:middle line:90% So we design the network. 00:05:41.940 --> 00:05:43.940 align:middle line:84% So we have to choose the number of hidden layers 00:05:43.940 --> 00:05:45.787 align:middle line:84% and the number of neurons in each layer. 00:05:45.787 --> 00:05:47.620 align:middle line:84% Then we have to pick the right output layer. 00:05:47.620 --> 00:05:50.307 align:middle line:90% So here, what I did is-- 00:05:50.307 --> 00:05:52.140 align:middle line:84% the simplest thing you can do is, of course, 00:05:52.140 --> 00:05:53.760 align:middle line:90% just have no hidden layer. 00:05:53.760 --> 00:05:58.520 align:middle line:84% So if you have no hidden layers, what is that model called? 00:05:58.520 --> 00:06:00.945 align:middle line:90% Yes, logistic regression. 00:06:00.945 --> 00:06:02.820 align:middle line:84% So, of course, we want to do neural networks, 00:06:02.820 --> 00:06:04.720 align:middle line:84% so I'm going to have one hidden layer because that's 00:06:04.720 --> 00:06:06.120 align:middle line:90% the simplest thing I can do. 00:06:06.120 --> 00:06:09.800 align:middle line:84% And then, I'll confess, I tried a few different numbers 00:06:09.800 --> 00:06:12.320 align:middle line:84% of neurons in this thing, and when I had 16 neurons, 00:06:12.320 --> 00:06:14.185 align:middle line:90% it actually did pretty well. 00:06:14.185 --> 00:06:15.560 align:middle line:84% So there was some trial and error 00:06:15.560 --> 00:06:18.535 align:middle line:84% that went on before I landed on the number 16. 00:06:18.535 --> 00:06:20.660 align:middle line:84% And for some reason, people always use powers of 2, 00:06:20.660 --> 00:06:22.640 align:middle line:90% so may as well do that. 00:06:22.640 --> 00:06:25.520 align:middle line:84% So I tried 4, 8, 16, and 16 was really good. 00:06:25.520 --> 00:06:28.440 align:middle line:84% And it turns out, when I went above 16, 00:06:28.440 --> 00:06:30.322 align:middle line:90% it started to do badly. 00:06:30.322 --> 00:06:32.280 align:middle line:84% And it started to do badly because of something 00:06:32.280 --> 00:06:35.520 align:middle line:84% called overfitting, which we're going to talk about later. 00:06:35.520 --> 00:06:37.680 align:middle line:90% So, yeah, 16. 00:06:37.680 --> 00:06:40.280 align:middle line:84% And then by default, I use ReLUs. 00:06:40.280 --> 00:06:41.860 align:middle line:90% So 16 ReLU neurons. 00:06:41.860 --> 00:06:45.980 align:middle line:84% And then here, the output is a categorical output. 00:06:45.980 --> 00:06:47.280 align:middle line:90% Heart disease, yes or no? 00:06:47.280 --> 00:06:48.320 align:middle line:90% 1 or 0. 00:06:48.320 --> 00:06:49.780 align:middle line:90% Classification problem. 00:06:49.780 --> 00:06:52.760 align:middle line:84% Which means that we want to emit a probability at the very end, 00:06:52.760 --> 00:06:54.940 align:middle line:84% therefore, we will use a sigmoid. 00:06:54.940 --> 00:06:57.160 align:middle line:90% So, so far, so good, right? 00:06:57.160 --> 00:06:59.180 align:middle line:90% Any questions? 00:06:59.180 --> 00:07:00.420 align:middle line:90% All right. 00:07:00.420 --> 00:07:04.640 align:middle line:84% So, we're going to lay out this network visually. 00:07:04.640 --> 00:07:06.180 align:middle line:90% So we have an input. 00:07:06.180 --> 00:07:08.580 align:middle line:90% And so I just have an input. 00:07:08.580 --> 00:07:11.760 align:middle line:84% And as you will see here, x1 through x29, 00:07:11.760 --> 00:07:13.100 align:middle line:90% that's our input layer. 00:07:13.100 --> 00:07:15.120 align:middle line:90% And you may be wondering, 29? 00:07:15.120 --> 00:07:17.380 align:middle line:90% Where did he get that from? 00:07:17.380 --> 00:07:20.460 align:middle line:84% Because that doesn't seem to be 29 rows here 00:07:20.460 --> 00:07:21.900 align:middle line:90% of independent variables. 00:07:21.900 --> 00:07:26.420 align:middle line:84% So it turns out, there are only 13 input variables here, 00:07:26.420 --> 00:07:29.260 align:middle line:84% but some of them are categorical. 00:07:29.260 --> 00:07:32.380 align:middle line:84% So what I ended up doing is to take each categorical variable 00:07:32.380 --> 00:07:34.100 align:middle line:90% and one-hot-encode it. 00:07:34.100 --> 00:07:37.340 align:middle line:84% And when you do that, you get to 39-- 00:07:37.340 --> 00:07:39.300 align:middle line:90% I'm sorry, 29. 00:07:39.300 --> 00:07:39.880 align:middle line:90% All right. 00:07:39.880 --> 00:07:42.300 align:middle line:84% And when we actually do the Colab later on, 00:07:42.300 --> 00:07:45.380 align:middle line:84% I'll show you exactly how I one-hot-encoded it, 00:07:45.380 --> 00:07:46.920 align:middle line:90% but that's what I'm doing here. 00:07:46.920 --> 00:07:49.280 align:middle line:90% That's why you have 29, not 13. 00:07:49.280 --> 00:07:50.040 align:middle line:90% OK. 00:07:50.040 --> 00:07:54.160 align:middle line:84% Now, obviously we have decided on these hidden units, 16 units, 00:07:54.160 --> 00:07:56.240 align:middle line:90% with nice ReLUs here. 00:07:56.240 --> 00:07:59.520 align:middle line:84% And then we have an output layer with a sigmoid. 00:07:59.520 --> 00:08:01.065 align:middle line:84% And I got bored of trying to draw 00:08:01.065 --> 00:08:02.940 align:middle line:84% all these arrows, so I just gave up and said, 00:08:02.940 --> 00:08:07.880 align:middle line:84% assume there are arrows between all these things. 00:08:07.880 --> 00:08:09.880 align:middle line:90% Good? 00:08:09.880 --> 00:08:10.628 align:middle line:90% Yeah? 00:08:10.628 --> 00:08:12.920 align:middle line:84% STUDENT: Yeah, I'm sorry, I think you already mentioned 00:08:12.920 --> 00:08:15.200 align:middle line:90% this, but why 16 units? 00:08:15.200 --> 00:08:16.480 align:middle line:90% RAMA RAMAKRISHNAN: Why 16? 00:08:16.480 --> 00:08:19.360 align:middle line:84% I tried a bunch of different numbers of units, 00:08:19.360 --> 00:08:22.060 align:middle line:84% and at 16, the resulting model did well, 00:08:22.060 --> 00:08:23.280 align:middle line:90% so I just went with that. 00:08:23.280 --> 00:08:25.443 align:middle line:84% STUDENT: And the logic of why it's a ReLU? 00:08:25.443 --> 00:08:26.860 align:middle line:84% RAMA RAMAKRISHNAN: Oh, why a ReLU? 00:08:26.860 --> 00:08:27.360 align:middle line:90% Yeah. 00:08:27.360 --> 00:08:29.960 align:middle line:84% So there is just a mountain of empirical evidence 00:08:29.960 --> 00:08:33.080 align:middle line:84% that suggests that ReLU is a really good default 00:08:33.080 --> 00:08:36.880 align:middle line:84% option for using as activations in hidden layers. 00:08:36.880 --> 00:08:40.220 align:middle line:84% There is also a really great set of theoretical results, 00:08:40.220 --> 00:08:42.240 align:middle line:84% and I'll allude to some of them when we actually 00:08:42.240 --> 00:08:45.270 align:middle line:90% talk about gradient descent. 00:08:45.270 --> 00:08:47.682 align:middle line:90% Yeah? 00:08:47.682 --> 00:08:48.890 align:middle line:90% STUDENT: So a quick question. 00:08:48.890 --> 00:08:51.250 align:middle line:84% You mentioned, in the input layer, 00:08:51.250 --> 00:08:54.310 align:middle line:84% how did you get to 29 again when you had 13 variables? 00:08:54.310 --> 00:08:56.310 align:middle line:84% RAMA RAMAKRISHNAN: So some of those 13 variables 00:08:56.310 --> 00:09:00.230 align:middle line:84% are categorical variables, like cholesterol, low, medium, high. 00:09:00.230 --> 00:09:02.790 align:middle line:84% And so I took them and one-hot-encoded them. 00:09:02.790 --> 00:09:08.950 align:middle line:84% So if it had five levels, I would get five columns now. 00:09:08.950 --> 00:09:09.870 align:middle line:90% Yeah? 00:09:09.870 --> 00:09:13.950 align:middle line:84% And by the way, folks, just like-- 00:09:13.950 --> 00:09:16.402 align:middle line:84% yeah, just like-- please use a microphone so 00:09:16.402 --> 00:09:18.610 align:middle line:84% that people on the livestream can hear your question. 00:09:18.610 --> 00:09:19.673 align:middle line:90% Yeah, go ahead. 00:09:19.673 --> 00:09:21.090 align:middle line:84% STUDENT: Sorry, just one question. 00:09:21.090 --> 00:09:23.290 align:middle line:84% So the vectors, since we didn't represent them, 00:09:23.290 --> 00:09:25.875 align:middle line:84% are we assuming every x is connected to all the units? 00:09:25.875 --> 00:09:27.000 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:09:27.000 --> 00:09:28.542 align:middle line:84% STUDENT: And this is also a parameter 00:09:28.542 --> 00:09:30.187 align:middle line:90% that we have to decide or-- 00:09:30.187 --> 00:09:32.270 align:middle line:84% RAMA RAMAKRISHNAN: That ends up being the default. 00:09:32.270 --> 00:09:35.590 align:middle line:84% And we will see deviations from that assumption 00:09:35.590 --> 00:09:38.537 align:middle line:84% when we go to Image processing and language processing and so 00:09:38.537 --> 00:09:40.870 align:middle line:84% on, but when you're working with structured data like we 00:09:40.870 --> 00:09:44.070 align:middle line:84% are doing now, that's the default. OK, 00:09:44.070 --> 00:09:46.130 align:middle line:90% so let's keep going. 00:09:46.130 --> 00:09:47.850 align:middle line:90% So this is what we have. 00:09:47.850 --> 00:09:49.930 align:middle line:84% So remember what I told you in the last class. 00:09:49.930 --> 00:09:51.763 align:middle line:84% Whenever you're working with these networks, 00:09:51.763 --> 00:09:54.170 align:middle line:84% get into the habit of very quickly calculating 00:09:54.170 --> 00:09:55.970 align:middle line:90% the number of parameters. 00:09:55.970 --> 00:09:58.410 align:middle line:84% Just do it a few times the first few times so 00:09:58.410 --> 00:10:01.290 align:middle line:84% that you really know cold exactly what's going on. 00:10:01.290 --> 00:10:04.570 align:middle line:84% So, yeah, how many parameters do we have here? 00:10:04.570 --> 00:10:05.990 align:middle line:90% How many weights and biases? 00:10:05.990 --> 00:10:07.358 align:middle line:90% You can work through it. 00:10:07.358 --> 00:10:09.150 align:middle line:84% You don't have to tell me the final number. 00:10:09.150 --> 00:10:14.410 align:middle line:84% You can say x times y plus z, stuff like that. 00:10:14.410 --> 00:10:15.810 align:middle line:90% Yeah? 00:10:15.810 --> 00:10:16.870 align:middle line:90% STUDENT: 65. 00:10:16.870 --> 00:10:19.495 align:middle line:84% You have 48 weights and 17 biases. 00:10:19.495 --> 00:10:21.870 align:middle line:84% RAMA RAMAKRISHNAN: OK, and how did you come up with that? 00:10:21.870 --> 00:10:24.370 align:middle line:84% STUDENT: So for the weights, you have-- for the first layer, 00:10:24.370 --> 00:10:25.390 align:middle line:90% it's 2 times 16. 00:10:25.390 --> 00:10:28.210 align:middle line:84% Then for the second connection, it's 1 times 16. 00:10:28.210 --> 00:10:32.250 align:middle line:84% And then the biases, it's 16 hidden plus the outputs. 00:10:32.250 --> 00:10:33.770 align:middle line:90% RAMA RAMAKRISHNAN: OK. 00:10:33.770 --> 00:10:36.090 align:middle line:90% Any other views on this? 00:10:36.090 --> 00:10:38.670 align:middle line:84% STUDENT: I think it's 29 into 16. 00:10:38.670 --> 00:10:39.690 align:middle line:90% RAMA RAMAKRISHNAN: 29-- 00:10:39.690 --> 00:10:41.550 align:middle line:90% OK, 29 into 16. 00:10:41.550 --> 00:10:44.110 align:middle line:90% STUDENT: And then 16 into plus-- 00:10:44.110 --> 00:10:45.290 align:middle line:90% I mean 16 there. 00:10:45.290 --> 00:10:46.290 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:10:46.290 --> 00:10:48.838 align:middle line:84% STUDENT: And then biases, 16 biases and one bias. 00:10:48.838 --> 00:10:49.880 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:10:49.880 --> 00:10:53.670 align:middle line:84% So, the way it's going to work is we have 29 things 00:10:53.670 --> 00:10:55.370 align:middle line:90% here, 16 in the middle. 00:10:55.370 --> 00:10:58.390 align:middle line:90% So 29 into 16 arrows. 00:10:58.390 --> 00:11:00.270 align:middle line:84% And then for each of these fellows, 00:11:00.270 --> 00:11:04.990 align:middle line:84% there is a bias coming in, so that's another 16. 00:11:04.990 --> 00:11:09.590 align:middle line:84% Plus, you have 16 times 1, which is here, 00:11:09.590 --> 00:11:12.070 align:middle line:84% plus there is one bias for this one. 00:11:12.070 --> 00:11:16.750 align:middle line:90% So the total is 497. 00:11:16.750 --> 00:11:19.830 align:middle line:84% So, you can see here, there is something very interesting 00:11:19.830 --> 00:11:22.350 align:middle line:84% going on, which is that when you go from one layer 00:11:22.350 --> 00:11:25.390 align:middle line:84% to another layer, the number of weights 00:11:25.390 --> 00:11:29.610 align:middle line:84% is roughly on the order of A times B. The number of units. 00:11:29.610 --> 00:11:31.270 align:middle line:84% And so that's a dramatic explosion 00:11:31.270 --> 00:11:33.470 align:middle line:90% in the number of parameters. 00:11:33.470 --> 00:11:35.750 align:middle line:84% And that's something we have to watch for later on 00:11:35.750 --> 00:11:38.090 align:middle line:90% to prevent overfitting. 00:11:38.090 --> 00:11:40.250 align:middle line:84% OK, that's where the explosion of parameters 00:11:40.250 --> 00:11:42.730 align:middle line:84% comes from the fact that each layer is fully connected 00:11:42.730 --> 00:11:44.590 align:middle line:90% to the next layer. 00:11:44.590 --> 00:11:46.210 align:middle line:90% But we'll revisit this later on. 00:11:46.210 --> 00:11:47.142 align:middle line:90% OK. 00:11:47.142 --> 00:11:49.890 align:middle line:84% So what I'm going to do now is I'm 00:11:49.890 --> 00:11:52.930 align:middle line:84% going to actually translate this network, the one that we have 00:11:52.930 --> 00:11:56.010 align:middle line:84% laid out graphically, into Keras code 00:11:56.010 --> 00:11:58.810 align:middle line:90% to demonstrate how easy it is. 00:11:58.810 --> 00:12:02.890 align:middle line:84% So I will give a fuller intro to Keras and TensorFlow later on, 00:12:02.890 --> 00:12:06.250 align:middle line:84% but for now, just suspend your disbelief, 00:12:06.250 --> 00:12:09.850 align:middle line:84% we will just try to do it in Keras as if we know Keras. 00:12:09.850 --> 00:12:10.730 align:middle line:90% So let's try that. 00:12:10.730 --> 00:12:12.610 align:middle line:84% Later on, we'll get into all the gory details 00:12:12.610 --> 00:12:15.130 align:middle line:84% and train it in Colab and so on and so forth. 00:12:15.130 --> 00:12:16.210 align:middle line:90% OK, all right. 00:12:16.210 --> 00:12:19.610 align:middle line:84% So, the way we typically do it is 00:12:19.610 --> 00:12:21.950 align:middle line:84% that once we have a network like this, 00:12:21.950 --> 00:12:25.290 align:middle line:84% we typically start from the left and start defining each layer 00:12:25.290 --> 00:12:26.710 align:middle line:90% in Keras, one after the other. 00:12:26.710 --> 00:12:28.210 align:middle line:90% So we flow left to right. 00:12:28.210 --> 00:12:30.930 align:middle line:90% So let's take the input layer. 00:12:30.930 --> 00:12:34.730 align:middle line:84% The way you define an input layer in Keras is really easy. 00:12:34.730 --> 00:12:38.470 align:middle line:90% You literally say keras.Input. 00:12:38.470 --> 00:12:42.510 align:middle line:84% And then you tell Keras how many nodes you have in the input 00:12:42.510 --> 00:12:43.350 align:middle line:90% coming in. 00:12:43.350 --> 00:12:45.950 align:middle line:84% In this case, it happens to be 29, so you tell it the shape, 00:12:45.950 --> 00:12:47.210 align:middle line:90% shape equals 29. 00:12:47.210 --> 00:12:49.870 align:middle line:84% And the reason why we say shape as opposed to length 00:12:49.870 --> 00:12:52.110 align:middle line:84% is because, as you will see later on, 00:12:52.110 --> 00:12:54.190 align:middle line:84% we don't have to just send vectors in. 00:12:54.190 --> 00:12:57.310 align:middle line:84% We can send complicated things into Keras. 00:12:57.310 --> 00:13:00.010 align:middle line:84% And those complicated objects could be matrices, 00:13:00.010 --> 00:13:02.590 align:middle line:84% it could be 3D cubes, it could be 4D tensors, 00:13:02.590 --> 00:13:06.497 align:middle line:84% and so on and so forth, so it's expecting a shape. 00:13:06.497 --> 00:13:08.830 align:middle line:84% What is the shape of this thing you're going to send me? 00:13:08.830 --> 00:13:10.288 align:middle line:84% In this particular case, it happens 00:13:10.288 --> 00:13:12.570 align:middle line:84% to be a nice list or a vector, so it's 29. 00:13:12.570 --> 00:13:13.590 align:middle line:90% OK, that's it. 00:13:13.590 --> 00:13:15.650 align:middle line:90% So we write this down. 00:13:15.650 --> 00:13:19.150 align:middle line:84% This creates the input layer, and we give it a name. 00:13:19.150 --> 00:13:22.910 align:middle line:84% And the name here means this layer, whatever comes out 00:13:22.910 --> 00:13:24.370 align:middle line:90% of this layer has a name input. 00:13:24.370 --> 00:13:27.790 align:middle line:90% 00:13:27.790 --> 00:13:28.570 align:middle line:90% Good. 00:13:28.570 --> 00:13:30.012 align:middle line:90% Next. 00:13:30.012 --> 00:13:31.470 align:middle line:84% You specify the shape of the input, 00:13:31.470 --> 00:13:34.330 align:middle line:90% as I mentioned, right there. 00:13:34.330 --> 00:13:35.730 align:middle line:90% Then we go to the next one. 00:13:35.730 --> 00:13:37.790 align:middle line:84% And here-- and we will unpack this, 00:13:37.790 --> 00:13:41.370 align:middle line:84% the way you define a layer is typically-- 00:13:41.370 --> 00:13:43.490 align:middle line:84% a hidden layer, keras.layers.Dense 00:13:43.490 --> 00:13:44.650 align:middle line:90% and all this stuff. 00:13:44.650 --> 00:13:47.270 align:middle line:90% So what this is is-- 00:13:47.270 --> 00:13:50.070 align:middle line:84% first of all, it says, I want a dense layer. 00:13:50.070 --> 00:13:51.930 align:middle line:84% By dense layer, I mean a layer that's 00:13:51.930 --> 00:13:55.090 align:middle line:84% going to fully connect to the prior and the later layers. 00:13:55.090 --> 00:13:55.750 align:middle line:90% Fully connect. 00:13:55.750 --> 00:13:58.130 align:middle line:84% That's what the word "dense" means. 00:13:58.130 --> 00:14:03.410 align:middle line:84% Number two, I want 16 nodes here in this layer. 00:14:03.410 --> 00:14:06.890 align:middle line:90% Finally, I want to use a ReLU. 00:14:06.890 --> 00:14:09.977 align:middle line:84% See how compact and parsimonious it is. 00:14:09.977 --> 00:14:11.310 align:middle line:90% And that is the appeal of Keras. 00:14:11.310 --> 00:14:13.620 align:middle line:90% It's very easy to get going. 00:14:13.620 --> 00:14:15.370 align:middle line:84% So the moment you do that, you've actually 00:14:15.370 --> 00:14:18.570 align:middle line:90% defined this layer. 00:14:18.570 --> 00:14:21.770 align:middle line:84% But what you have not done is you have not 00:14:21.770 --> 00:14:25.370 align:middle line:84% told this layer what input is going to get. 00:14:25.370 --> 00:14:27.350 align:middle line:84% Because as far as this layer is concerned, 00:14:27.350 --> 00:14:30.330 align:middle line:84% it doesn't know that this other layer exists. 00:14:30.330 --> 00:14:31.630 align:middle line:90% So you need to connect them. 00:14:31.630 --> 00:14:33.670 align:middle line:90% Yes? 00:14:33.670 --> 00:14:35.910 align:middle line:84% STUDENT: Do we need to define for the ReLU 00:14:35.910 --> 00:14:37.510 align:middle line:90% where the bends are? 00:14:37.510 --> 00:14:38.975 align:middle line:90% Like, where you take the max? 00:14:38.975 --> 00:14:41.350 align:middle line:84% RAMA RAMAKRISHNAN: No, the ReLU, the bend is always at 0. 00:14:41.350 --> 00:14:41.850 align:middle line:90% STUDENT: OK. 00:14:41.850 --> 00:14:42.410 align:middle line:90% Thank you. 00:14:42.410 --> 00:14:45.630 align:middle line:90% 00:14:45.630 --> 00:14:47.350 align:middle line:90% RAMA RAMAKRISHNAN: OK. 00:14:47.350 --> 00:14:48.790 align:middle line:90% All right. 00:14:48.790 --> 00:14:51.230 align:middle line:90% So that's what we have here. 00:14:51.230 --> 00:14:54.470 align:middle line:84% And then, what we do is we have to tell it-- 00:14:54.470 --> 00:14:57.670 align:middle line:84% you want to feed this layer, the output of the previous layer. 00:14:57.670 --> 00:15:00.270 align:middle line:84% So you feed it by taking whatever is coming out 00:15:00.270 --> 00:15:02.090 align:middle line:84% of this thing, which is called input, 00:15:02.090 --> 00:15:05.470 align:middle line:84% and you basically stick it in here. 00:15:05.470 --> 00:15:07.790 align:middle line:84% So the moment you do that, boom, it's 00:15:07.790 --> 00:15:10.470 align:middle line:84% going to receive the input from the previous layer. 00:15:10.470 --> 00:15:14.470 align:middle line:84% And because this one's output needs to go to the final layer, 00:15:14.470 --> 00:15:16.510 align:middle line:84% you need to give a name to that output. 00:15:16.510 --> 00:15:17.490 align:middle line:90% So you give it a name. 00:15:17.490 --> 00:15:18.990 align:middle line:84% I'm just calling it h4 because it's 00:15:18.990 --> 00:15:20.550 align:middle line:90% coming out of the hidden layer. 00:15:20.550 --> 00:15:22.883 align:middle line:84% It's just a variable, you can call it anything you want. 00:15:22.883 --> 00:15:24.990 align:middle line:90% 00:15:24.990 --> 00:15:28.950 align:middle line:84% Now what we do, we go to the final output layer, 00:15:28.950 --> 00:15:30.070 align:middle line:90% and this is what we use. 00:15:30.070 --> 00:15:32.810 align:middle line:84% The output layer is just another dense layer. 00:15:32.810 --> 00:15:34.410 align:middle line:90% That's why I use the word dense. 00:15:34.410 --> 00:15:36.810 align:middle line:84% But we say, hey, give me just one thing because I just 00:15:36.810 --> 00:15:39.410 align:middle line:84% literally just need one unit here because I need 00:15:39.410 --> 00:15:41.930 align:middle line:90% to emit just one probability. 00:15:41.930 --> 00:15:46.950 align:middle line:84% And the activation I want to use is a sigmoid. 00:15:46.950 --> 00:15:47.450 align:middle line:90% Done. 00:15:47.450 --> 00:15:50.410 align:middle line:90% 00:15:50.410 --> 00:15:53.490 align:middle line:84% And once you do that, you have to feed 00:15:53.490 --> 00:15:57.810 align:middle line:84% it the input from the second layer, so you stick an h here. 00:15:57.810 --> 00:16:01.370 align:middle line:84% Now we have connected the third and the second layers. 00:16:01.370 --> 00:16:04.050 align:middle line:84% And after you do that, you give a name to the output coming out 00:16:04.050 --> 00:16:05.790 align:middle line:84% of that, we'll just call it output. 00:16:05.790 --> 00:16:07.540 align:middle line:84% You can call it y, you can call it output, 00:16:07.540 --> 00:16:09.730 align:middle line:84% you can call it whatever you want. 00:16:09.730 --> 00:16:12.010 align:middle line:84% So at this point, what we have done 00:16:12.010 --> 00:16:16.210 align:middle line:84% is we have mapped that picture into those three lines. 00:16:16.210 --> 00:16:16.810 align:middle line:90% That's it. 00:16:16.810 --> 00:16:19.370 align:middle line:90% 00:16:19.370 --> 00:16:20.710 align:middle line:90% But we aren't quite done yet. 00:16:20.710 --> 00:16:22.490 align:middle line:84% There's one little thing we have to do. 00:16:22.490 --> 00:16:26.990 align:middle line:84% So what we have to do is we have to formally define a model 00:16:26.990 --> 00:16:30.130 align:middle line:84% so that Keras can just work with this model object-- 00:16:30.130 --> 00:16:31.850 align:middle line:84% it can train it, it can evaluate it, 00:16:31.850 --> 00:16:34.017 align:middle line:84% it can use it for prediction and so on and so forth. 00:16:34.017 --> 00:16:36.810 align:middle line:84% So we tell Keras, hey, create a model for me, 00:16:36.810 --> 00:16:40.770 align:middle line:84% keras.Model, and basically where the input is this thing here 00:16:40.770 --> 00:16:42.470 align:middle line:84% and the output is that thing there. 00:16:42.470 --> 00:16:45.950 align:middle line:84% And then the whole thing, we'll just call it model. 00:16:45.950 --> 00:16:48.550 align:middle line:90% So that's it. 00:16:48.550 --> 00:16:49.250 align:middle line:90% We are done. 00:16:49.250 --> 00:16:50.403 align:middle line:90% That is the whole model. 00:16:50.403 --> 00:16:52.070 align:middle line:84% That is-- it sounds really fancy, right? 00:16:52.070 --> 00:16:54.650 align:middle line:84% A neural model for heart disease prediction. 00:16:54.650 --> 00:16:56.670 align:middle line:90% That's pretty cool. 00:16:56.670 --> 00:16:58.630 align:middle line:90% Four lines. 00:16:58.630 --> 00:17:01.690 align:middle line:84% And we will show how to train this model with real data, 00:17:01.690 --> 00:17:05.270 align:middle line:84% and so on and so forth, and use it for prediction after we 00:17:05.270 --> 00:17:08.609 align:middle line:84% switch gears and really get into some conceptual building blocks. 00:17:08.609 --> 00:17:09.359 align:middle line:90% We had a question. 00:17:09.359 --> 00:17:13.869 align:middle line:90% 00:17:13.869 --> 00:17:16.430 align:middle line:84% STUDENT: Can you define a custom activation 00:17:16.430 --> 00:17:20.151 align:middle line:84% function that is not in the list of characters library? 00:17:20.151 --> 00:17:21.109 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:17:21.109 --> 00:17:21.369 align:middle line:90% Yeah. 00:17:21.369 --> 00:17:22.490 align:middle line:84% You can define-- the question was, 00:17:22.490 --> 00:17:24.323 align:middle line:84% can you define a custom activation function? 00:17:24.323 --> 00:17:25.650 align:middle line:90% You totally can. 00:17:25.650 --> 00:17:28.490 align:middle line:84% And in fact, I mean, the kind of flexibility you have 00:17:28.490 --> 00:17:30.330 align:middle line:90% here is incredible. 00:17:30.330 --> 00:17:33.010 align:middle line:84% And these innocent four lines, unfortunately, 00:17:33.010 --> 00:17:36.310 align:middle line:84% hide the potential that's possible here, 00:17:36.310 --> 00:17:38.490 align:middle line:84% but I guarantee you, in two to three weeks, 00:17:38.490 --> 00:17:41.410 align:middle line:84% you folks will be thinking in building blocks like LEGOs. 00:17:41.410 --> 00:17:44.422 align:middle line:90% So I'm so happy when it happens. 00:17:44.422 --> 00:17:46.130 align:middle line:84% Students come to my office hours and say, 00:17:46.130 --> 00:17:48.850 align:middle line:84% I want to create a network where I have a little network going up 00:17:48.850 --> 00:17:51.470 align:middle line:84% on top, one going in the bottom, then they meet in the middle, 00:17:51.470 --> 00:17:52.845 align:middle line:84% then they fork again, they split. 00:17:52.845 --> 00:17:54.090 align:middle line:90% I'm like, unbelievable. 00:17:54.090 --> 00:17:54.950 align:middle line:90% It's fantastic. 00:17:54.950 --> 00:17:56.908 align:middle line:84% And you're going to be doing this in two weeks, 00:17:56.908 --> 00:17:58.513 align:middle line:90% I guarantee you. 00:17:58.513 --> 00:18:00.930 align:middle line:84% STUDENT: Yeah, in the case of a multi-class classification 00:18:00.930 --> 00:18:03.710 align:middle line:84% problem, are the output nodes equal to the number of classes? 00:18:03.710 --> 00:18:05.330 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:18:05.330 --> 00:18:08.543 align:middle line:84% So we will come to-- so this is binary classification. 00:18:08.543 --> 00:18:10.710 align:middle line:84% And the question is, for multi-class classification, 00:18:10.710 --> 00:18:13.370 align:middle line:84% let's say you're trying to classify some input into one 00:18:13.370 --> 00:18:14.830 align:middle line:90% of 10 possibilities. 00:18:14.830 --> 00:18:16.690 align:middle line:90% We will have 10 outputs. 00:18:16.690 --> 00:18:19.490 align:middle line:84% But the way we define it is going to be using something 00:18:19.490 --> 00:18:24.070 align:middle line:84% called a softmax function, which we're going to cover on Monday. 00:18:24.070 --> 00:18:29.270 align:middle line:84% So for now, we just live with binary classification. 00:18:29.270 --> 00:18:32.230 align:middle line:84% STUDENT: Is there a default activation method in Keras 00:18:32.230 --> 00:18:33.870 align:middle line:90% or you have to put something? 00:18:33.870 --> 00:18:35.620 align:middle line:84% RAMA RAMAKRISHNAN: That's a good question. 00:18:35.620 --> 00:18:37.930 align:middle line:84% I believe the default might be ReLUs for hidden layers, 00:18:37.930 --> 00:18:39.170 align:middle line:90% but I'm not 100% sure. 00:18:39.170 --> 00:18:40.510 align:middle line:90% Let's double-check that, yeah. 00:18:40.510 --> 00:18:43.147 align:middle line:90% 00:18:43.147 --> 00:18:45.230 align:middle line:84% STUDENT: Just to get a clearer understanding, when 00:18:45.230 --> 00:18:47.390 align:middle line:84% you said that beyond 16, when you 00:18:47.390 --> 00:18:52.140 align:middle line:84% tried working on those neurons, the performance worsened. 00:18:52.140 --> 00:18:54.390 align:middle line:84% That is where you are playing around with initially 2, 00:18:54.390 --> 00:18:56.170 align:middle line:90% and then maybe 4 and 6 and 8? 00:18:56.170 --> 00:18:58.442 align:middle line:84% RAMA RAMAKRISHNAN: Exactly, right. 00:18:58.442 --> 00:18:59.650 align:middle line:90% Could you use the microphone? 00:18:59.650 --> 00:19:02.140 align:middle line:90% Yeah. 00:19:02.140 --> 00:19:04.390 align:middle line:84% STUDENT: Do we need to define each of the hidden layer 00:19:04.390 --> 00:19:06.110 align:middle line:84% when the model gets more complex, when 00:19:06.110 --> 00:19:07.768 align:middle line:90% we have more than one layer? 00:19:07.768 --> 00:19:08.310 align:middle line:90% Or is there-- 00:19:08.310 --> 00:19:10.047 align:middle line:84% RAMA RAMAKRISHNAN: If you have 25 layers? 00:19:10.047 --> 00:19:10.630 align:middle line:90% STUDENT: Yeah. 00:19:10.630 --> 00:19:11.130 align:middle line:84% RAMA RAMAKRISHNAN: Yeah, yeah, yeah. 00:19:11.130 --> 00:19:12.672 align:middle line:84% So what we typically-- good question. 00:19:12.672 --> 00:19:14.790 align:middle line:84% If you have, let's say, 100 layers. 00:19:14.790 --> 00:19:17.690 align:middle line:84% Do you actually have to type in each by hand and cut and paste? 00:19:17.690 --> 00:19:18.190 align:middle line:90% No. 00:19:18.190 --> 00:19:20.773 align:middle line:84% You can actually write a little loop, which just automatically 00:19:20.773 --> 00:19:22.810 align:middle line:90% create them for you. 00:19:22.810 --> 00:19:25.570 align:middle line:84% And so, basically what's going on is that this little output 00:19:25.570 --> 00:19:27.650 align:middle line:84% thing you see here, this variable, 00:19:27.650 --> 00:19:32.213 align:middle line:84% this output could be the result of a 1,000-layer network with 00:19:32.213 --> 00:19:34.130 align:middle line:84% all sorts of complicated transformations going 00:19:34.130 --> 00:19:36.630 align:middle line:84% on, and then finally, it pops up as this little thing called 00:19:36.630 --> 00:19:37.490 align:middle line:90% the output. 00:19:37.490 --> 00:19:39.390 align:middle line:84% And what Keras will do is it will be like, 00:19:39.390 --> 00:19:42.390 align:middle line:84% OK, this model has this input and has this output, 00:19:42.390 --> 00:19:44.890 align:middle line:84% but boy, this output came from incredible transformations 00:19:44.890 --> 00:19:46.448 align:middle line:84% applied to the input, and Keras will 00:19:46.448 --> 00:19:47.990 align:middle line:84% process all that very easily for you, 00:19:47.990 --> 00:19:50.070 align:middle line:84% you don't have to worry about it. 00:19:50.070 --> 00:19:53.330 align:middle line:84% It's really a beautiful example of the power of abstraction. 00:19:53.330 --> 00:19:55.210 align:middle line:84% And you will see that as we go along. 00:19:55.210 --> 00:19:58.510 align:middle line:84% OK, so, now let's switch gears and say, 00:19:58.510 --> 00:20:00.550 align:middle line:84% once you've written a model like that in Keras, 00:20:00.550 --> 00:20:01.890 align:middle line:90% how do you actually train it? 00:20:01.890 --> 00:20:05.570 align:middle line:84% Now, training is something you've been doing a lot. 00:20:05.570 --> 00:20:08.290 align:middle line:84% So for example, when you have something like linear regression 00:20:08.290 --> 00:20:10.207 align:middle line:84% where you have all these coefficients you need 00:20:10.207 --> 00:20:11.870 align:middle line:84% to estimate, you have this model, 00:20:11.870 --> 00:20:13.370 align:middle line:90% then you have a bunch of data. 00:20:13.370 --> 00:20:16.650 align:middle line:84% Then you run it through something like LLM if you use R. 00:20:16.650 --> 00:20:19.630 align:middle line:84% And what it gives you is actual values for the coefficients. 00:20:19.630 --> 00:20:21.630 align:middle line:84% 2.8, 2.9, and so on and so forth. 00:20:21.630 --> 00:20:25.570 align:middle line:84% So the role of the data is to give you the coefficients. 00:20:25.570 --> 00:20:27.070 align:middle line:84% Or you can think of the coefficients 00:20:27.070 --> 00:20:30.350 align:middle line:84% as really a compressed version of the data. 00:20:30.350 --> 00:20:32.630 align:middle line:84% Similarly, if you do logistic regression, 00:20:32.630 --> 00:20:34.850 align:middle line:84% you have a model like that, you add some data, 00:20:34.850 --> 00:20:36.600 align:middle line:84% you run it through some estimation routine 00:20:36.600 --> 00:20:40.190 align:middle line:84% like GNN or scikit-learn or statsmodels, 00:20:40.190 --> 00:20:42.030 align:middle line:84% pick your favorite tool, then you'll come up 00:20:42.030 --> 00:20:43.110 align:middle line:90% with something like that. 00:20:43.110 --> 00:20:44.750 align:middle line:84% So basically what's going on here 00:20:44.750 --> 00:20:48.790 align:middle line:84% is training simply means find the values of the coefficients 00:20:48.790 --> 00:20:50.750 align:middle line:84% so that the model's predictions are 00:20:50.750 --> 00:20:52.970 align:middle line:84% as close to the actual values as possible. 00:20:52.970 --> 00:20:55.160 align:middle line:90% That's it. 00:20:55.160 --> 00:20:58.310 align:middle line:84% And to find the one that is as close to the actual value 00:20:58.310 --> 00:21:00.830 align:middle line:84% as possible, a whole bunch of optimization involved. 00:21:00.830 --> 00:21:02.830 align:middle line:84% You didn't have to worry about optimization when 00:21:02.830 --> 00:21:04.990 align:middle line:84% you did regression, linear or logistic, 00:21:04.990 --> 00:21:07.050 align:middle line:84% because it's all done under the hood for you, 00:21:07.050 --> 00:21:08.670 align:middle line:84% but for neural networks, we actually 00:21:08.670 --> 00:21:12.910 align:middle line:84% get to know how it's done because it's important. 00:21:12.910 --> 00:21:16.770 align:middle line:84% OK, so training a neural network, a deep neural network, 00:21:16.770 --> 00:21:19.450 align:middle line:84% even GPT-4, it's basically the same process 00:21:19.450 --> 00:21:21.610 align:middle line:90% as what you do for regression. 00:21:21.610 --> 00:21:24.210 align:middle line:84% You basically-- you're just a very complicated function 00:21:24.210 --> 00:21:26.355 align:middle line:84% with lots of parameters, but ultimately, you 00:21:26.355 --> 00:21:28.230 align:middle line:84% have a network with all these question marks, 00:21:28.230 --> 00:21:30.350 align:middle line:84% you add some data, you do some training, and boom, 00:21:30.350 --> 00:21:31.225 align:middle line:90% you get some numbers. 00:21:31.225 --> 00:21:36.290 align:middle line:90% 00:21:36.290 --> 00:21:38.330 align:middle line:84% STUDENT: You may get into this, but are we 00:21:38.330 --> 00:21:43.472 align:middle line:84% determining the architecture of the network before we train it? 00:21:43.472 --> 00:21:44.430 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:21:44.430 --> 00:21:47.130 align:middle line:84% Because if you don't define the architecture, 00:21:47.130 --> 00:21:49.730 align:middle line:84% Keras doesn't know how to actually calculate 00:21:49.730 --> 00:21:52.270 align:middle line:90% the output given an input. 00:21:52.270 --> 00:21:55.130 align:middle line:84% And unless it knows input/output pairs, 00:21:55.130 --> 00:21:58.130 align:middle line:84% it can't do anything more with it. 00:21:58.130 --> 00:22:00.410 align:middle line:90% OK. 00:22:00.410 --> 00:22:02.850 align:middle line:84% So the essence of training is to find the best values 00:22:02.850 --> 00:22:05.610 align:middle line:90% for the weights and biases. 00:22:05.610 --> 00:22:07.610 align:middle line:84% And the way we think of the best values 00:22:07.610 --> 00:22:10.490 align:middle line:84% is that we basically set up a little function, 00:22:10.490 --> 00:22:14.170 align:middle line:84% and this function measures the discrepancy between the actual 00:22:14.170 --> 00:22:16.370 align:middle line:90% and the predicted values. 00:22:16.370 --> 00:22:19.590 align:middle line:84% And I use the word discrepancy because the way 00:22:19.590 --> 00:22:21.710 align:middle line:84% you define discrepancy, there's incredible amounts 00:22:21.710 --> 00:22:23.270 align:middle line:90% of creativity in the field. 00:22:23.270 --> 00:22:25.750 align:middle line:84% In fact, a lot of breakthroughs in deep learning 00:22:25.750 --> 00:22:28.150 align:middle line:84% come because people define a very clever measure 00:22:28.150 --> 00:22:30.037 align:middle line:84% of discrepancy, and then it turns out, 00:22:30.037 --> 00:22:32.370 align:middle line:84% it actually gives you all sorts of interesting behavior. 00:22:32.370 --> 00:22:34.590 align:middle line:84% That's why I use the word discrepancy as opposed 00:22:34.590 --> 00:22:36.690 align:middle line:84% to the word error because when I say error, 00:22:36.690 --> 00:22:38.315 align:middle line:84% you might be just thinking of something 00:22:38.315 --> 00:22:40.070 align:middle line:90% like predicted minus actual. 00:22:40.070 --> 00:22:42.023 align:middle line:90% That's still limiting. 00:22:42.023 --> 00:22:43.690 align:middle line:84% Prediction minus actual is too limiting, 00:22:43.690 --> 00:22:45.357 align:middle line:84% that's why you use the word discrepancy. 00:22:45.357 --> 00:22:48.110 align:middle line:84% So we basically define a function 00:22:48.110 --> 00:22:50.190 align:middle line:84% that captures the discrepancy between the actual 00:22:50.190 --> 00:22:52.550 align:middle line:84% and the predicted values, and these functions 00:22:52.550 --> 00:22:55.830 align:middle line:84% are called loss functions in the deep learning world. 00:22:55.830 --> 00:22:57.950 align:middle line:84% And every paper that you read, you 00:22:57.950 --> 00:23:00.170 align:middle line:84% will find interesting loss functions. 00:23:00.170 --> 00:23:01.890 align:middle line:84% There are hundreds of loss functions. 00:23:01.890 --> 00:23:04.350 align:middle line:84% Enormous research creativity goes into defining these loss 00:23:04.350 --> 00:23:05.910 align:middle line:90% functions. 00:23:05.910 --> 00:23:06.550 align:middle line:90% All right. 00:23:06.550 --> 00:23:08.590 align:middle line:90% So, these are loss functions. 00:23:08.590 --> 00:23:10.470 align:middle line:84% And so a loss function is a function that 00:23:10.470 --> 00:23:12.110 align:middle line:90% quantifies the discrepancy. 00:23:12.110 --> 00:23:13.930 align:middle line:84% So let's say the predictions are really 00:23:13.930 --> 00:23:15.690 align:middle line:90% close to the actual values. 00:23:15.690 --> 00:23:19.042 align:middle line:90% The loss would be what? 00:23:19.042 --> 00:23:20.440 align:middle line:90% STUDENT: Close to 0. 00:23:20.440 --> 00:23:21.690 align:middle line:90% RAMA RAMAKRISHNAN: Close to 0. 00:23:21.690 --> 00:23:23.370 align:middle line:90% Very small. 00:23:23.370 --> 00:23:26.970 align:middle line:84% And if you have a perfect model, perfect crystal 00:23:26.970 --> 00:23:28.850 align:middle line:90% ball, what would the loss be? 00:23:28.850 --> 00:23:30.090 align:middle line:90% Exactly 0. 00:23:30.090 --> 00:23:31.710 align:middle line:90% Exactly 0. 00:23:31.710 --> 00:23:34.890 align:middle line:84% So, in linear regression, the loss 00:23:34.890 --> 00:23:37.445 align:middle line:84% function we use is called sum of squared errors. 00:23:37.445 --> 00:23:39.570 align:middle line:84% We didn't call it loss function because we were not 00:23:39.570 --> 00:23:41.670 align:middle line:84% doing deep learning, just linear regression, 00:23:41.670 --> 00:23:43.810 align:middle line:84% but that's basically the loss function. 00:23:43.810 --> 00:23:48.450 align:middle line:84% So the loss function we use must be very matched very properly 00:23:48.450 --> 00:23:51.290 align:middle line:90% with the kind of output we have. 00:23:51.290 --> 00:23:54.395 align:middle line:84% So if your output is a number like 23, 00:23:54.395 --> 00:23:55.770 align:middle line:84% you're trying to predict demand-- 00:23:55.770 --> 00:23:59.330 align:middle line:84% like a product demand for next week for a particular product, 00:23:59.330 --> 00:24:01.630 align:middle line:90% and the predicted value is 23. 00:24:01.630 --> 00:24:05.490 align:middle line:84% the actual value is 21, it's OK to do 23 minus 21, 00:24:05.490 --> 00:24:07.270 align:middle line:90% too, as a discrepancy. 00:24:07.270 --> 00:24:08.650 align:middle line:90% The error. 00:24:08.650 --> 00:24:12.320 align:middle line:84% But for other kinds of outputs, it's not so obvious 00:24:12.320 --> 00:24:14.060 align:middle line:84% what the correct loss function is, 00:24:14.060 --> 00:24:16.160 align:middle line:84% what the correct measure of discrepancy is. 00:24:16.160 --> 00:24:22.680 align:middle line:84% And so here, for the simple case of regression, the yi-- 00:24:22.680 --> 00:24:25.040 align:middle line:84% the i here, by the way, is a superscript which 00:24:25.040 --> 00:24:27.360 align:middle line:90% stands for the i-th data point. 00:24:27.360 --> 00:24:28.700 align:middle line:90% The i-th data point. 00:24:28.700 --> 00:24:31.520 align:middle line:84% So what I'm saying is that for the i-th data point, 00:24:31.520 --> 00:24:33.880 align:middle line:84% this is the actual value y, and this 00:24:33.880 --> 00:24:36.760 align:middle line:90% is what the model predicted. 00:24:36.760 --> 00:24:39.100 align:middle line:84% I take the difference, square it, 00:24:39.100 --> 00:24:41.440 align:middle line:84% and I square it for each point I just 00:24:41.440 --> 00:24:43.840 align:middle line:84% average all these numbers to get an average 00:24:43.840 --> 00:24:47.120 align:middle line:84% squared error-- i.e., Mean Squared Error, MSE. 00:24:47.120 --> 00:24:52.280 align:middle line:84% So this is the easiest loss function 00:24:52.280 --> 00:24:55.040 align:middle line:90% Now, let's crank it up a notch. 00:24:55.040 --> 00:24:57.800 align:middle line:84% In the heart disease example, the heart disease-- 00:24:57.800 --> 00:25:00.520 align:middle line:84% the neural prediction model, the prediction 00:25:00.520 --> 00:25:03.240 align:middle line:84% is a number between 0 and 1 because it's 00:25:03.240 --> 00:25:04.760 align:middle line:90% coming out of the sigmoid. 00:25:04.760 --> 00:25:06.000 align:middle line:90% It's a fraction. 00:25:06.000 --> 00:25:09.020 align:middle line:90% The actual output is a 0 or 1. 00:25:09.020 --> 00:25:11.180 align:middle line:90% One of the two, it's binary. 00:25:11.180 --> 00:25:13.560 align:middle line:84% So how would we compare the discrepancy? 00:25:13.560 --> 00:25:16.420 align:middle line:84% How would we measure the discrepancy between a fraction 00:25:16.420 --> 00:25:18.180 align:middle line:90% and the numbers 0 and 1? 00:25:18.180 --> 00:25:21.100 align:middle line:84% What is a good loss function in this situation 00:25:21.100 --> 00:25:22.640 align:middle line:90% is the key question. 00:25:22.640 --> 00:25:26.060 align:middle line:84% So let's build some intuition around this. 00:25:26.060 --> 00:25:30.852 align:middle line:84% And let's see if my little daisy chain iPad thing works. 00:25:30.852 --> 00:25:33.060 align:middle line:84% I'm doing it on iPad so that people on the livestream 00:25:33.060 --> 00:25:34.620 align:middle line:84% can see it, otherwise the blackboard 00:25:34.620 --> 00:25:37.100 align:middle line:90% is a little tough for them. 00:25:37.100 --> 00:25:41.106 align:middle line:84% OK, so let's have a situation here-- 00:25:41.106 --> 00:25:41.606 align:middle line:90% OK. 00:25:41.606 --> 00:25:44.740 align:middle line:84% So let's say that you have a patient who comes in, 00:25:44.740 --> 00:25:47.000 align:middle line:84% and let's say they have heart disease. 00:25:47.000 --> 00:25:50.520 align:middle line:90% So for that patient, y equals 1. 00:25:50.520 --> 00:25:53.020 align:middle line:84% The true value is 1 for that patient. 00:25:53.020 --> 00:25:56.560 align:middle line:90% And now, you have this model. 00:25:56.560 --> 00:26:01.410 align:middle line:84% And this is the predicted probability from this model. 00:26:01.410 --> 00:26:04.540 align:middle line:90% 00:26:04.540 --> 00:26:07.560 align:middle line:84% Can people see my handwriting OK? 00:26:07.560 --> 00:26:08.320 align:middle line:90% Good. 00:26:08.320 --> 00:26:11.320 align:middle line:84% I could have never been a doctor, right? 00:26:11.320 --> 00:26:13.100 align:middle line:90% So 0, 1. 00:26:13.100 --> 00:26:15.480 align:middle line:84% It's going to be between 0 and 1 because of probability. 00:26:15.480 --> 00:26:17.900 align:middle line:84% And then this is the loss we want to have. 00:26:17.900 --> 00:26:19.160 align:middle line:90% This is the loss. 00:26:19.160 --> 00:26:23.403 align:middle line:84% So, this patient actually had heart disease, y equals 1. 00:26:23.403 --> 00:26:25.320 align:middle line:84% So let's say that the predicted probability is 00:26:25.320 --> 00:26:27.440 align:middle line:90% pretty close to 1. 00:26:27.440 --> 00:26:29.960 align:middle line:84% What do you think the loss should be? 00:26:29.960 --> 00:26:31.320 align:middle line:90% STUDENT: Small. 00:26:31.320 --> 00:26:32.920 align:middle line:90% STUDENT: Small. 00:26:32.920 --> 00:26:34.240 align:middle line:90% RAMA RAMAKRISHNAN: Sorry? 00:26:34.240 --> 00:26:35.160 align:middle line:90% Close to 0, exactly. 00:26:35.160 --> 00:26:37.700 align:middle line:84% So here, if the prediction comes here, 00:26:37.700 --> 00:26:39.400 align:middle line:84% you want the loss to be-- you want 00:26:39.400 --> 00:26:42.320 align:middle line:90% the loss to be somewhere here. 00:26:42.320 --> 00:26:44.747 align:middle line:84% But if the predicted probability is pretty close to 0, 00:26:44.747 --> 00:26:47.080 align:middle line:84% even though the patient actually has heart disease, what 00:26:47.080 --> 00:26:49.480 align:middle line:90% do you want the loss to be? 00:26:49.480 --> 00:26:52.640 align:middle line:84% Really high because it's screwing up badly. 00:26:52.640 --> 00:26:55.360 align:middle line:84% So you want the loss to be somewhere here. 00:26:55.360 --> 00:26:57.200 align:middle line:84% So basically, you want a function 00:26:57.200 --> 00:26:58.310 align:middle line:90% that's kind of like that. 00:26:58.310 --> 00:27:00.840 align:middle line:90% 00:27:00.840 --> 00:27:04.165 align:middle line:84% You want the loss function shape to be like that. 00:27:04.165 --> 00:27:05.540 align:middle line:84% High values of probability should 00:27:05.540 --> 00:27:06.880 align:middle line:90% have low losses, low values. 00:27:06.880 --> 00:27:08.380 align:middle line:84% Probability should have high losses. 00:27:08.380 --> 00:27:08.980 align:middle line:90% Yeah? 00:27:08.980 --> 00:27:11.810 align:middle line:84% STUDENT: I understand why it has to be increasing or decreasing, 00:27:11.810 --> 00:27:13.268 align:middle line:84% but then your statement has to be-- 00:27:13.268 --> 00:27:14.518 align:middle line:90% RAMA RAMAKRISHNAN: Yeah, yeah. 00:27:14.518 --> 00:27:15.600 align:middle line:90% So it can be linear. 00:27:15.600 --> 00:27:17.160 align:middle line:90% It can certainly be linear. 00:27:17.160 --> 00:27:18.540 align:middle line:84% But basically what you want to do 00:27:18.540 --> 00:27:22.540 align:middle line:84% is, the more it makes a mistake, the more harshly you 00:27:22.540 --> 00:27:24.560 align:middle line:90% want to penalize it. 00:27:24.560 --> 00:27:26.740 align:middle line:84% So basically what you really want 00:27:26.740 --> 00:27:28.340 align:middle line:84% is something where if it basically 00:27:28.340 --> 00:27:29.960 align:middle line:84% says this person's probability is, 00:27:29.960 --> 00:27:32.780 align:middle line:84% say, the probability-- the predicted probability, 00:27:32.780 --> 00:27:35.640 align:middle line:84% say, 1 over a million, basically to 0, 00:27:35.640 --> 00:27:38.860 align:middle line:84% you want the loss to be super high so that the model is 00:27:38.860 --> 00:27:41.320 align:middle line:84% like-- it's like a huge rap on the knuckles for the model. 00:27:41.320 --> 00:27:42.383 align:middle line:90% Don't do that! 00:27:42.383 --> 00:27:43.800 align:middle line:84% That's basically what we're doing. 00:27:43.800 --> 00:27:45.540 align:middle line:84% And I'm demonstrating that dynamic 00:27:45.540 --> 00:27:49.565 align:middle line:84% by using a very curved and steep loss function. 00:27:49.565 --> 00:27:51.440 align:middle line:84% But you can absolutely use a linear function. 00:27:51.440 --> 00:27:52.340 align:middle line:90% It's totally fine. 00:27:52.340 --> 00:27:54.540 align:middle line:84% It won't be as effective for gradient descent 00:27:54.540 --> 00:27:57.820 align:middle line:84% later on, but a bunch of technical details. 00:27:57.820 --> 00:27:59.380 align:middle line:90% Are we good with this? 00:27:59.380 --> 00:28:00.060 align:middle line:90% All right. 00:28:00.060 --> 00:28:03.720 align:middle line:84% So, now let's look at the case where a patient does not 00:28:03.720 --> 00:28:06.840 align:middle line:90% have heart disease, y equals 0. 00:28:06.840 --> 00:28:09.080 align:middle line:90% Same setup. 00:28:09.080 --> 00:28:15.320 align:middle line:84% Predicted probability, 0, 1, loss. 00:28:15.320 --> 00:28:20.640 align:middle line:84% So for this patient, they don't have whatever-- 00:28:20.640 --> 00:28:22.000 align:middle line:90% they don't have heart disease. 00:28:22.000 --> 00:28:26.280 align:middle line:84% If the probability is close to 0, what should the loss be? 00:28:26.280 --> 00:28:26.860 align:middle line:90% Close to 0. 00:28:26.860 --> 00:28:28.027 align:middle line:90% It should be somewhere here. 00:28:28.027 --> 00:28:30.978 align:middle line:84% And the more and more the probability 00:28:30.978 --> 00:28:32.520 align:middle line:84% gets closer and closer to 1, you want 00:28:32.520 --> 00:28:34.160 align:middle line:84% to penalize it very heavily, which 00:28:34.160 --> 00:28:36.120 align:middle line:84% means you want the loss to be somewhere here. 00:28:36.120 --> 00:28:37.703 align:middle line:84% So you basically want a loss, ideally, 00:28:37.703 --> 00:28:42.200 align:middle line:84% that's going up like that and climbing higher and higher. 00:28:42.200 --> 00:28:43.240 align:middle line:90% Are we good? 00:28:43.240 --> 00:28:44.640 align:middle line:90% OK, perfect. 00:28:44.640 --> 00:28:49.000 align:middle line:84% Because we have a perfect loss function for that. 00:28:49.000 --> 00:28:53.080 align:middle line:84% So, just to recap, this is what we want. 00:28:53.080 --> 00:28:56.260 align:middle line:84% People with for points with y equals 1, lower prediction, 00:28:56.260 --> 00:28:57.880 align:middle line:84% predictions should have higher loss. 00:28:57.880 --> 00:28:59.260 align:middle line:90% We want something like that. 00:28:59.260 --> 00:29:04.740 align:middle line:84% And then turns out, there's a very little simple loss function 00:29:04.740 --> 00:29:06.873 align:middle line:84% which just literally just uses the logarithm, 00:29:06.873 --> 00:29:08.040 align:middle line:90% which will get the job done. 00:29:08.040 --> 00:29:11.220 align:middle line:84% So what you do is you literally do minus log 00:29:11.220 --> 00:29:13.180 align:middle line:90% of that predicted probability. 00:29:13.180 --> 00:29:14.060 align:middle line:90% That's it. 00:29:14.060 --> 00:29:17.120 align:middle line:84% And that thing has exactly that shape. 00:29:17.120 --> 00:29:18.960 align:middle line:84% And in fact, you can see it numerically. 00:29:18.960 --> 00:29:20.680 align:middle line:90% So if the loss is 1, it's 0. 00:29:20.680 --> 00:29:22.760 align:middle line:90% If it's half, it's 1.0. 00:29:22.760 --> 00:29:25.340 align:middle line:84% And if it's 1 over 1,000, it's almost 10. 00:29:25.340 --> 00:29:28.480 align:middle line:84% If it's 1 over 10,000, it's going to be much higher, 00:29:28.480 --> 00:29:30.760 align:middle line:90% very high losses. 00:29:30.760 --> 00:29:34.540 align:middle line:84% So minus log probability, boom, done. 00:29:34.540 --> 00:29:37.140 align:middle line:84% Similarly, this is what we want for patients 00:29:37.140 --> 00:29:38.560 align:middle line:90% for whom y equals 0. 00:29:38.560 --> 00:29:42.780 align:middle line:84% And it turns out, if you do minus log 1 minus 00:29:42.780 --> 00:29:45.240 align:middle line:84% predicted probability, it does the same thing. 00:29:45.240 --> 00:29:47.800 align:middle line:90% 00:29:47.800 --> 00:29:48.300 align:middle line:90% OK. 00:29:48.300 --> 00:29:50.840 align:middle line:90% 00:29:50.840 --> 00:29:54.700 align:middle line:84% Mathematicians, once again, saved by the logarithm. 00:29:54.700 --> 00:29:58.892 align:middle line:84% So in summary, this is what we have. 00:29:58.892 --> 00:30:01.600 align:middle line:84% For data points with y equals 1, we have this; data points with y 00:30:01.600 --> 00:30:02.980 align:middle line:90% equals 0, we have this. 00:30:02.980 --> 00:30:07.240 align:middle line:84% But it feels a little inelegant to say, well if it's y equals 1, 00:30:07.240 --> 00:30:09.560 align:middle line:84% I want to use this; if y equals 0, I want to use that. 00:30:09.560 --> 00:30:12.205 align:middle line:84% There's an if-then thing going on here, 00:30:12.205 --> 00:30:13.580 align:middle line:84% and I don't know about you folks, 00:30:13.580 --> 00:30:16.440 align:middle line:84% but if-then really irks me mathematically 00:30:16.440 --> 00:30:20.280 align:middle line:84% because you can't do derivatives and so on very easily. 00:30:20.280 --> 00:30:21.600 align:middle line:90% But no worries. 00:30:21.600 --> 00:30:22.520 align:middle line:90% This is MIT. 00:30:22.520 --> 00:30:26.680 align:middle line:84% We know-- we have a bag of math tricks, so what we do is, 00:30:26.680 --> 00:30:30.880 align:middle line:84% we can actually combine them both into a single expression 00:30:30.880 --> 00:30:32.600 align:middle line:90% like this. 00:30:32.600 --> 00:30:35.740 align:middle line:84% And here, the y i, again, is the i-th data point. 00:30:35.740 --> 00:30:38.360 align:middle line:84% Remember, y i is either 1 or 0 always. 00:30:38.360 --> 00:30:42.160 align:middle line:84% And this model of x i is the predicted probability. 00:30:42.160 --> 00:30:46.060 align:middle line:84% And I've just taken the minus log, the minus, 00:30:46.060 --> 00:30:48.860 align:middle line:90% and I've just moved it here. 00:30:48.860 --> 00:30:52.740 align:middle line:84% And I've taken the minus that was here and just moved it here. 00:30:52.740 --> 00:30:54.600 align:middle line:90% That's why you see it like this. 00:30:54.600 --> 00:30:58.420 align:middle line:84% So this one is basically, you can convince yourself 00:30:58.420 --> 00:30:59.240 align:middle line:90% what happens. 00:30:59.240 --> 00:31:01.073 align:middle line:84% The single expression will get the job done. 00:31:01.073 --> 00:31:04.020 align:middle line:84% So let's say there is a patient for whom y equals 1. 00:31:04.020 --> 00:31:07.080 align:middle line:84% What's going to happen is that when you plug in y equals 1, 00:31:07.080 --> 00:31:10.540 align:middle line:84% this becomes 0, the whole thing will collapse to 0. 00:31:10.540 --> 00:31:12.460 align:middle line:84% While here, y equals 1 just means 00:31:12.460 --> 00:31:14.260 align:middle line:84% it becomes minus log probability, which 00:31:14.260 --> 00:31:15.120 align:middle line:90% is what we want. 00:31:15.120 --> 00:31:17.660 align:middle line:90% 00:31:17.660 --> 00:31:20.340 align:middle line:84% Conversely, if y equals 0, this whole thing 00:31:20.340 --> 00:31:22.060 align:middle line:90% is going to disappear. 00:31:22.060 --> 00:31:24.680 align:middle line:84% And this thing becomes 1 minus 0, which is just 1, 00:31:24.680 --> 00:31:27.180 align:middle line:84% and so it becomes minus log 1 minus probability, which 00:31:27.180 --> 00:31:29.700 align:middle line:90% is, again, what we want. 00:31:29.700 --> 00:31:31.980 align:middle line:90% Simple and neat, right? 00:31:31.980 --> 00:31:35.880 align:middle line:84% So in one expression, we have defined the perfect loss. 00:31:35.880 --> 00:31:39.500 align:middle line:90% No if-thens, none of that crap. 00:31:39.500 --> 00:31:40.220 align:middle line:90% Good. 00:31:40.220 --> 00:31:44.020 align:middle line:84% So now what we do is, that was true for every data point, 00:31:44.020 --> 00:31:45.800 align:middle line:84% but we obviously have lots of data points, 00:31:45.800 --> 00:31:50.460 align:middle line:84% so we just add them all up and take the average. 00:31:50.460 --> 00:31:50.960 align:middle line:90% That's it. 00:31:50.960 --> 00:31:52.877 align:middle line:84% We average across all the data points we have. 00:31:52.877 --> 00:31:55.520 align:middle line:90% So that we get an average loss. 00:31:55.520 --> 00:31:57.352 align:middle line:90% OK. 00:31:57.352 --> 00:31:59.310 align:middle line:84% This is the binary cross entropy loss function. 00:31:59.310 --> 00:32:06.680 align:middle line:90% 00:32:06.680 --> 00:32:09.620 align:middle line:84% STUDENT: Is there a way you can edit the loss function 00:32:09.620 --> 00:32:12.602 align:middle line:84% so that you penalize false negatives more strongly than-- 00:32:12.602 --> 00:32:13.560 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:32:13.560 --> 00:32:14.518 align:middle line:90% You can do all of them. 00:32:14.518 --> 00:32:15.680 align:middle line:90% Great question. 00:32:15.680 --> 00:32:20.160 align:middle line:84% I'm just looking at the basic case where it's symmetric loss. 00:32:20.160 --> 00:32:23.040 align:middle line:84% You can actually penalize overestimates much more 00:32:23.040 --> 00:32:25.382 align:middle line:84% than underestimates and things like that. 00:32:25.382 --> 00:32:26.840 align:middle line:84% And if you're curious, you can just 00:32:26.840 --> 00:32:28.570 align:middle line:84% Google something called the pinball loss. 00:32:28.570 --> 00:32:31.480 align:middle line:90% 00:32:31.480 --> 00:32:32.700 align:middle line:90% OK. 00:32:32.700 --> 00:32:34.440 align:middle line:90% Any other questions on this? 00:32:34.440 --> 00:32:37.240 align:middle line:84% So when you see this massive deep neural network built 00:32:37.240 --> 00:32:39.340 align:middle line:84% by Google for doing something or the other, 00:32:39.340 --> 00:32:41.380 align:middle line:84% if it's a binary classification problem, 00:32:41.380 --> 00:32:45.080 align:middle line:84% chances are they're using this thing. 00:32:45.080 --> 00:32:45.960 align:middle line:90% All right. 00:32:45.960 --> 00:32:49.380 align:middle line:84% So, now let's figure out how to minimize these loss functions 00:32:49.380 --> 00:32:51.880 align:middle line:84% because the name of the game is to find a way to minimize 00:32:51.880 --> 00:32:53.160 align:middle line:90% these loss functions. 00:32:53.160 --> 00:32:56.820 align:middle line:84% So now loss functions are just a particular kind of function. 00:32:56.820 --> 00:33:00.020 align:middle line:84% So we'll first consider the general problem of minimizing 00:33:00.020 --> 00:33:02.060 align:middle line:90% some arbitrary function. 00:33:02.060 --> 00:33:04.580 align:middle line:84% And once we develop a little bit of intuition about that, 00:33:04.580 --> 00:33:06.380 align:middle line:84% we'll return to the specific task 00:33:06.380 --> 00:33:08.240 align:middle line:90% of minimizing loss functions. 00:33:08.240 --> 00:33:12.180 align:middle line:90% 00:33:12.180 --> 00:33:15.260 align:middle line:90% How's everyone doing? 00:33:15.260 --> 00:33:18.860 align:middle line:90% Yes, no, good, bad? 00:33:18.860 --> 00:33:23.220 align:middle line:84% You have a bit of a tough-to-interpret head shake. 00:33:23.220 --> 00:33:25.038 align:middle line:84% STUDENT: It's more like I kind of lost you 00:33:25.038 --> 00:33:26.580 align:middle line:84% where you said that the loss function 00:33:26.580 --> 00:33:29.132 align:middle line:84% and the predicted probability, how were they 00:33:29.132 --> 00:33:31.340 align:middle line:84% inversely because the understanding was that the loss 00:33:31.340 --> 00:33:33.820 align:middle line:84% function is supposed to be the sum of errors, 00:33:33.820 --> 00:33:35.000 align:middle line:90% we are averaging the errors. 00:33:35.000 --> 00:33:35.660 align:middle line:90% And when you said that-- 00:33:35.660 --> 00:33:36.100 align:middle line:90% RAMA RAMAKRISHNAN: No, no, no. 00:33:36.100 --> 00:33:36.400 align:middle line:90% No, no. 00:33:36.400 --> 00:33:36.942 align:middle line:90% Sorry, sorry. 00:33:36.942 --> 00:33:38.900 align:middle line:90% Let us stop there for a second. 00:33:38.900 --> 00:33:41.142 align:middle line:84% For each point, you define the loss. 00:33:41.142 --> 00:33:42.600 align:middle line:84% That's the whole point of the game. 00:33:42.600 --> 00:33:44.980 align:middle line:84% And once you define it, you calculate it for every point 00:33:44.980 --> 00:33:45.760 align:middle line:90% and average it. 00:33:45.760 --> 00:33:49.460 align:middle line:84% So just focus on a single data point. 00:33:49.460 --> 00:33:50.980 align:middle line:90% And so now continue. 00:33:50.980 --> 00:33:53.020 align:middle line:84% STUDENT: So now when the heart patient has, 00:33:53.020 --> 00:33:56.140 align:middle line:84% there is more probability that they know. 00:33:56.140 --> 00:33:59.440 align:middle line:84% So when there is a person who has the heart disease, 00:33:59.440 --> 00:34:02.600 align:middle line:84% you said that you want the loss function to be high. 00:34:02.600 --> 00:34:04.100 align:middle line:84% I think I'm going back to the graph. 00:34:04.100 --> 00:34:05.975 align:middle line:84% RAMA RAMAKRISHNAN: You want the loss function 00:34:05.975 --> 00:34:08.040 align:middle line:84% to be high if I'm predicting that they basically 00:34:08.040 --> 00:34:09.920 align:middle line:90% don't have heart disease. 00:34:09.920 --> 00:34:12.840 align:middle line:84% If the prediction is close to 0, the predicted probability 00:34:12.840 --> 00:34:18.040 align:middle line:84% is close to 0, then I'm badly wrong because in reality, they 00:34:18.040 --> 00:34:19.958 align:middle line:90% do have heart disease. 00:34:19.958 --> 00:34:22.000 align:middle line:84% And that's why I want the loss to be really high. 00:34:22.000 --> 00:34:23.375 align:middle line:84% STUDENT: OK, so effectively, loss 00:34:23.375 --> 00:34:25.594 align:middle line:84% is my way of finding out how good my model is. 00:34:25.594 --> 00:34:26.719 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:34:26.719 --> 00:34:28.440 align:middle line:90% STUDENT: Instead of saying, OK-- 00:34:28.440 --> 00:34:30.656 align:middle line:84% RAMA RAMAKRISHNAN: Or rather, how bad your model is. 00:34:30.656 --> 00:34:31.239 align:middle line:90% STUDENT: Yeah. 00:34:31.239 --> 00:34:31.739 align:middle line:90% RAMA RAMAKRISHNAN: Right? 00:34:31.739 --> 00:34:32.552 align:middle line:90% How bad is it? 00:34:32.552 --> 00:34:34.219 align:middle line:84% That's really what the loss function is. 00:34:34.219 --> 00:34:34.886 align:middle line:90% STUDENT: Got it. 00:34:34.886 --> 00:34:37.360 align:middle line:84% RAMA RAMAKRISHNAN: And you want to minimize badness. 00:34:37.360 --> 00:34:40.120 align:middle line:84% That's the whole point of optimization. 00:34:40.120 --> 00:34:41.588 align:middle line:90% STUDENT: OK. 00:34:41.588 --> 00:34:43.880 align:middle line:84% RAMA RAMAKRISHNAN: I guess I don't have a fully-- like, 00:34:43.880 --> 00:34:45.880 align:middle line:84% similar to the point raised before, I don't have 00:34:45.880 --> 00:34:48.440 align:middle line:84% a fully clear intuition of why exactly a log function 00:34:48.440 --> 00:34:52.659 align:middle line:84% rather than something that, say, flatter for small and then 00:34:52.659 --> 00:34:53.556 align:middle line:90% really steep later. 00:34:53.556 --> 00:34:55.639 align:middle line:84% RAMA RAMAKRISHNAN: Those are all fantastic things, 00:34:55.639 --> 00:34:57.020 align:middle line:90% you can totally do it. 00:34:57.020 --> 00:35:00.540 align:middle line:84% The reason we pick the loss-- this function because, 00:35:00.540 --> 00:35:03.440 align:middle line:84% A, is easy to work with, it has good gradients, 00:35:03.440 --> 00:35:05.220 align:middle line:84% it's well-behaved mathematically, 00:35:05.220 --> 00:35:07.160 align:middle line:84% but there are many alternatives to it. 00:35:07.160 --> 00:35:10.100 align:middle line:84% I don't want you to think that this is the only game in town 00:35:10.100 --> 00:35:11.720 align:middle line:90% or it's the only choice for us. 00:35:11.720 --> 00:35:13.000 align:middle line:90% We have many choices. 00:35:13.000 --> 00:35:15.980 align:middle line:84% This is really-- this happens to be a very easy choice, which 00:35:15.980 --> 00:35:18.900 align:middle line:84% also happens to be empirically very effective. 00:35:18.900 --> 00:35:20.380 align:middle line:84% And I'm happy to give you pointers 00:35:20.380 --> 00:35:23.300 align:middle line:84% to other crazy loss functions which can actually 00:35:23.300 --> 00:35:24.400 align:middle line:90% do all these things, too. 00:35:24.400 --> 00:35:26.720 align:middle line:90% 00:35:26.720 --> 00:35:27.220 align:middle line:90% OK. 00:35:27.220 --> 00:35:30.340 align:middle line:90% 00:35:30.340 --> 00:35:33.280 align:middle line:84% All right, so, minimizing a single variable function, 00:35:33.280 --> 00:35:37.040 align:middle line:84% we will warm up by looking at this little function here, 00:35:37.040 --> 00:35:38.277 align:middle line:90% which is a-- 00:35:38.277 --> 00:35:39.610 align:middle line:90% what do you call a fourth power? 00:35:39.610 --> 00:35:42.180 align:middle line:90% 00:35:42.180 --> 00:35:43.093 align:middle line:90% Quartic, right? 00:35:43.093 --> 00:35:43.760 align:middle line:90% Yeah, Thank you. 00:35:43.760 --> 00:35:44.540 align:middle line:90% Quartic. 00:35:44.540 --> 00:35:47.640 align:middle line:84% So yeah, it's a quartic function. 00:35:47.640 --> 00:35:50.000 align:middle line:90% And this is how it looks like. 00:35:50.000 --> 00:35:52.480 align:middle line:84% And you can see, there is a minimum somewhere here, 00:35:52.480 --> 00:35:54.560 align:middle line:90% between minus 1 and minus 2. 00:35:54.560 --> 00:35:56.000 align:middle line:90% Maybe minus 1.5. 00:35:56.000 --> 00:35:58.560 align:middle line:84% So we want to minimize this function. 00:35:58.560 --> 00:36:00.760 align:middle line:84% It's obviously a toy function, a little function 00:36:00.760 --> 00:36:03.400 align:middle line:84% with one variable, but the intuition we use here 00:36:03.400 --> 00:36:08.240 align:middle line:84% is going to be exactly what we use for GPT-4, so pay attention. 00:36:08.240 --> 00:36:10.960 align:middle line:84% So, how can we go about minimizing this function? 00:36:10.960 --> 00:36:11.820 align:middle line:90% What will we do? 00:36:11.820 --> 00:36:15.080 align:middle line:90% 00:36:15.080 --> 00:36:16.755 align:middle line:90% Yeah? 00:36:16.755 --> 00:36:18.880 align:middle line:84% STUDENT: Take the derivative and set it equal to 0? 00:36:18.880 --> 00:36:19.860 align:middle line:84% RAMA RAMAKRISHNAN: You take the derivative. 00:36:19.860 --> 00:36:20.360 align:middle line:90% Exactly. 00:36:20.360 --> 00:36:22.320 align:middle line:90% So you take the derivative. 00:36:22.320 --> 00:36:26.640 align:middle line:84% So let's look at what the derivative does for us, but then 00:36:26.640 --> 00:36:29.900 align:middle line:90% the second part of what I said-- 00:36:29.900 --> 00:36:30.400 align:middle line:90% yeah. 00:36:30.400 --> 00:36:33.760 align:middle line:84% Second part of what I said was set it to 0, setting it to 0 00:36:33.760 --> 00:36:37.860 align:middle line:84% becomes problematic when you have very complicated functions. 00:36:37.860 --> 00:36:40.140 align:middle line:84% It's not clear at all what's going to make them 0, 00:36:40.140 --> 00:36:41.000 align:middle line:90% unfortunately. 00:36:41.000 --> 00:36:43.880 align:middle line:84% But the idea of derivative is, in fact, the right idea. 00:36:43.880 --> 00:36:46.123 align:middle line:84% So we can go about this, we can calculate derivative, 00:36:46.123 --> 00:36:48.040 align:middle line:84% and that actually happens with the derivative, 00:36:48.040 --> 00:36:49.460 align:middle line:90% you can convince yourself. 00:36:49.460 --> 00:36:53.460 align:middle line:84% And if you plot the derivative, it looks like that. 00:36:53.460 --> 00:36:56.240 align:middle line:84% And, as you would hope, wherever the minimum is, in fact, 00:36:56.240 --> 00:36:58.700 align:middle line:90% the derivative is crossing-- 00:36:58.700 --> 00:37:01.300 align:middle line:84% the derivative at 0 here is crossing the x-axis, 00:37:01.300 --> 00:37:03.740 align:middle line:84% in this case we can actually do that. 00:37:03.740 --> 00:37:06.580 align:middle line:84% So let's say you have the derivative, how can you use it? 00:37:06.580 --> 00:37:08.380 align:middle line:84% What is the value of a derivative? 00:37:08.380 --> 00:37:09.780 align:middle line:90% What does it tell you? 00:37:09.780 --> 00:37:11.300 align:middle line:90% Yeah? 00:37:11.300 --> 00:37:13.682 align:middle line:84% STUDENT: You use a gradient descent algorithm? 00:37:13.682 --> 00:37:16.140 align:middle line:84% RAMA RAMAKRISHNAN: You are 10 steps ahead of me, my friend. 00:37:16.140 --> 00:37:18.300 align:middle line:90% I just want a basic answer. 00:37:18.300 --> 00:37:20.233 align:middle line:90% Like, what good is a derivative? 00:37:20.233 --> 00:37:21.400 align:middle line:90% Like, what does it tell you? 00:37:21.400 --> 00:37:23.020 align:middle line:84% When you calculate the derivative of something 00:37:23.020 --> 00:37:23.890 align:middle line:90% at a particular point-- 00:37:23.890 --> 00:37:25.980 align:middle line:84% STUDENT: It tells you the rate of change of the function 00:37:25.980 --> 00:37:26.720 align:middle line:90% at the place you are. 00:37:26.720 --> 00:37:28.470 align:middle line:84% RAMA RAMAKRISHNAN: Correct, exactly right. 00:37:28.470 --> 00:37:30.260 align:middle line:84% So here, what the derivative tells 00:37:30.260 --> 00:37:33.780 align:middle line:84% us is that the slope tells us a change in the function 00:37:33.780 --> 00:37:36.060 align:middle line:90% for a very small increase in w. 00:37:36.060 --> 00:37:38.420 align:middle line:84% And this just high school calculus, I'm 00:37:38.420 --> 00:37:41.940 align:middle line:90% just doing a quick refresher. 00:37:41.940 --> 00:37:47.680 align:middle line:84% So, what that means is that if the derivative is positive, 00:37:47.680 --> 00:37:50.200 align:middle line:84% what that means is that increasing w slightly 00:37:50.200 --> 00:37:52.080 align:middle line:90% will increase the function. 00:37:52.080 --> 00:37:55.035 align:middle line:84% So if you're here, you calculate the slope is positive, 00:37:55.035 --> 00:37:57.160 align:middle line:84% it means that if you go slightly in this direction, 00:37:57.160 --> 00:38:00.200 align:middle line:84% the function is going to get higher. 00:38:00.200 --> 00:38:02.720 align:middle line:84% Similarly, if it's negative, let's 00:38:02.720 --> 00:38:05.340 align:middle line:84% say here, you calculate the slope is like this, 00:38:05.340 --> 00:38:08.265 align:middle line:84% it's negative, which means that if you increase w, 00:38:08.265 --> 00:38:09.640 align:middle line:84% if you go in this direction, it's 00:38:09.640 --> 00:38:10.932 align:middle line:90% going to decrease the function. 00:38:10.932 --> 00:38:13.720 align:middle line:90% 00:38:13.720 --> 00:38:15.000 align:middle line:90% All right. 00:38:15.000 --> 00:38:17.960 align:middle line:84% And if it's kind of close to 0, it 00:38:17.960 --> 00:38:22.320 align:middle line:84% means that changing w slightly won't change anything. 00:38:22.320 --> 00:38:26.880 align:middle line:84% So if you're here, changing it slightly won't change anything. 00:38:26.880 --> 00:38:28.080 align:middle line:90% That's it. 00:38:28.080 --> 00:38:31.680 align:middle line:84% So what we do is this immediately 00:38:31.680 --> 00:38:33.760 align:middle line:84% suggests an algorithm for minimizing 00:38:33.760 --> 00:38:38.360 align:middle line:84% g w, which is let's start with some random point w, 00:38:38.360 --> 00:38:41.400 align:middle line:84% and then let's calculate the derivative at that point. 00:38:41.400 --> 00:38:44.980 align:middle line:84% And once we do that, there are three possibilities. 00:38:44.980 --> 00:38:48.300 align:middle line:84% It could be positive, negative, or close to 0. 00:38:48.300 --> 00:38:50.540 align:middle line:84% And if it's positive, we know that increasing w, 00:38:50.540 --> 00:38:52.820 align:middle line:84% will increase the function, but we 00:38:52.820 --> 00:38:55.440 align:middle line:84% want to decrease the function, we want to minimize it, 00:38:55.440 --> 00:38:57.460 align:middle line:84% which means that we should not be increasing w. 00:38:57.460 --> 00:38:59.420 align:middle line:90% We should be doing what here? 00:38:59.420 --> 00:39:01.340 align:middle line:90% STUDENTS: Decrease. 00:39:01.340 --> 00:39:02.300 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:39:02.300 --> 00:39:04.790 align:middle line:84% And similarly, if it's negative, what should we do here? 00:39:04.790 --> 00:39:07.315 align:middle line:90% STUDENT: Increase. 00:39:07.315 --> 00:39:08.440 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:39:08.440 --> 00:39:10.660 align:middle line:84% So in the first case, you reduce w slightly. 00:39:10.660 --> 00:39:12.633 align:middle line:84% In the second case, you increase w slightly. 00:39:12.633 --> 00:39:14.300 align:middle line:84% And if the thing is close to 0, you just 00:39:14.300 --> 00:39:16.175 align:middle line:84% stop because there's nothing else you can do. 00:39:16.175 --> 00:39:21.380 align:middle line:90% 00:39:21.380 --> 00:39:26.700 align:middle line:84% This is the basic intuition behind how GPT-4 was built, 00:39:26.700 --> 00:39:29.220 align:middle line:84% which is kind of shocking if you think about it. 00:39:29.220 --> 00:39:32.780 align:middle line:84% Which means that all the heavy-duty optimization stuff 00:39:32.780 --> 00:39:35.060 align:middle line:84% that people have figured out over the decades 00:39:35.060 --> 00:39:38.020 align:middle line:90% is kind of not used. 00:39:38.020 --> 00:39:39.850 align:middle line:84% This algorithm is what's being used 00:39:39.850 --> 00:39:42.170 align:middle line:90% with some flavors on top of it. 00:39:42.170 --> 00:39:44.490 align:middle line:90% So, yeah, so back to this. 00:39:44.490 --> 00:39:46.070 align:middle line:90% And you do that. 00:39:46.070 --> 00:39:49.890 align:middle line:84% And then, if you have run out of time or compute, 00:39:49.890 --> 00:39:53.950 align:middle line:84% or if you're run out of time and so on, just stop. 00:39:53.950 --> 00:39:55.950 align:middle line:84% Otherwise, just go back to step 2 and try again. 00:39:55.950 --> 00:40:00.090 align:middle line:84% Of course, if it's close to 0, you've got to stop anyway. 00:40:00.090 --> 00:40:02.330 align:middle line:90% Yeah? 00:40:02.330 --> 00:40:06.250 align:middle line:84% STUDENT: Is there the concern of a potentially local minimum 00:40:06.250 --> 00:40:06.750 align:middle line:90% there? 00:40:06.750 --> 00:40:08.042 align:middle line:90% RAMA RAMAKRISHNAN: It's coming. 00:40:08.042 --> 00:40:10.010 align:middle line:90% 00:40:10.010 --> 00:40:11.230 align:middle line:90% So that's a function. 00:40:11.230 --> 00:40:13.570 align:middle line:84% It's going to find you some point where 00:40:13.570 --> 00:40:15.090 align:middle line:90% the data is kind of close to 0. 00:40:15.090 --> 00:40:19.950 align:middle line:84% So this is called gradient descent. 00:40:19.950 --> 00:40:24.810 align:middle line:84% This is gradient descent, this little algorithm. 00:40:24.810 --> 00:40:29.090 align:middle line:84% And this very PowerPointy MBA table 00:40:29.090 --> 00:40:32.650 align:middle line:84% can be collapsed into this little expression. 00:40:32.650 --> 00:40:35.930 align:middle line:84% Basically says, calculate the derivative multiplied 00:40:35.930 --> 00:40:38.410 align:middle line:84% by a small number, which we'll get to in a second, 00:40:38.410 --> 00:40:40.990 align:middle line:90% and then change the old w to-- 00:40:40.990 --> 00:40:43.870 align:middle line:84% the new w is the old w minus the little number 00:40:43.870 --> 00:40:45.670 align:middle line:90% times the gradient. 00:40:45.670 --> 00:40:48.790 align:middle line:84% So this little one-line formula is basically gradient descent. 00:40:48.790 --> 00:40:51.550 align:middle line:90% 00:40:51.550 --> 00:40:54.750 align:middle line:84% And what you should do, just to build your intuition, 00:40:54.750 --> 00:40:58.270 align:middle line:84% is to make sure that these three possibilities here 00:40:58.270 --> 00:40:59.105 align:middle line:90% map nicely to this. 00:40:59.105 --> 00:41:01.230 align:middle line:84% Like, this thing will actually captures these three 00:41:01.230 --> 00:41:02.595 align:middle line:90% possibilities. 00:41:02.595 --> 00:41:05.020 align:middle line:84% Can you guess when gradient descent was invented? 00:41:05.020 --> 00:41:07.237 align:middle line:90% 00:41:07.237 --> 00:41:09.070 align:middle line:84% You need to have some historical fun, right? 00:41:09.070 --> 00:41:12.910 align:middle line:90% 00:41:12.910 --> 00:41:14.990 align:middle line:90% STUDENT: The 19th century? 00:41:14.990 --> 00:41:16.350 align:middle line:90% RAMA RAMAKRISHNAN: 19th century. 00:41:16.350 --> 00:41:17.110 align:middle line:90% Yeah, OK. 00:41:17.110 --> 00:41:17.610 align:middle line:90% Good. 00:41:17.610 --> 00:41:18.210 align:middle line:90% Very good. 00:41:18.210 --> 00:41:20.350 align:middle line:90% Excellent guess. 00:41:20.350 --> 00:41:22.710 align:middle line:90% 1847. 00:41:22.710 --> 00:41:27.190 align:middle line:84% It was invented in 1847 by Cauchy, the great mathematician. 00:41:27.190 --> 00:41:31.410 align:middle line:84% And in fact, if you're curious, you can check out the paper. 00:41:31.410 --> 00:41:35.650 align:middle line:84% I gave you give you the paper here for handy reference. 00:41:35.650 --> 00:41:36.610 align:middle line:90% OK. 00:41:36.610 --> 00:41:38.170 align:middle line:90% So 1847. 00:41:38.170 --> 00:41:44.250 align:middle line:84% So GPT-4 is built using an algorithm invented in 1847. 00:41:44.250 --> 00:41:47.890 align:middle line:84% Which I find astonishing, frankly. 00:41:47.890 --> 00:41:51.610 align:middle line:90% This little thing is so capable. 00:41:51.610 --> 00:41:52.970 align:middle line:90% OK. 00:41:52.970 --> 00:41:54.470 align:middle line:90% So that's gradient descent. 00:41:54.470 --> 00:41:58.130 align:middle line:84% And this little number alpha is called the learning rate, 00:41:58.130 --> 00:42:00.610 align:middle line:84% and it's our way of essentially quantifying 00:42:00.610 --> 00:42:04.790 align:middle line:84% the idea of, let's not increase or decrease w massively, 00:42:04.790 --> 00:42:07.370 align:middle line:84% let's do it slightly because the gradient is 00:42:07.370 --> 00:42:11.270 align:middle line:84% only valid for small movements around your point. 00:42:11.270 --> 00:42:14.810 align:middle line:84% If you take a big step, all bets are off. 00:42:14.810 --> 00:42:20.850 align:middle line:84% So this alpha tells you how small a step should you take. 00:42:20.850 --> 00:42:25.712 align:middle line:84% And typically, it's set to very small values, like 0.1, 0.01, 00:42:25.712 --> 00:42:26.670 align:middle line:90% and so on and so forth. 00:42:26.670 --> 00:42:28.045 align:middle line:84% And in fact, if you read any deep 00:42:28.045 --> 00:42:30.610 align:middle line:84% learning academic papers where they have trained 00:42:30.610 --> 00:42:33.990 align:middle line:84% like a big model to do something, a lot of researchers 00:42:33.990 --> 00:42:35.693 align:middle line:84% will very quickly go to the appendix 00:42:35.693 --> 00:42:38.110 align:middle line:84% where they have described exactly what learning rates were 00:42:38.110 --> 00:42:41.030 align:middle line:84% used because the learning rate is 00:42:41.030 --> 00:42:43.813 align:middle line:84% part of the IP for how it's built. 00:42:43.813 --> 00:42:45.230 align:middle line:84% It's a lot of trial and error that 00:42:45.230 --> 00:42:47.510 align:middle line:90% goes into these learning rates. 00:42:47.510 --> 00:42:51.230 align:middle line:90% OK, so that is gradient descent. 00:42:51.230 --> 00:42:55.590 align:middle line:84% So if we apply this algorithm to g w, our original function, we 00:42:55.590 --> 00:42:59.910 align:middle line:84% just keep on doing this thing a few times, what you will find 00:42:59.910 --> 00:43:03.110 align:middle line:84% is that if-- let's say we start with two points-- the point 00:43:03.110 --> 00:43:05.390 align:middle line:90% we randomly pick is 2.5. 00:43:05.390 --> 00:43:07.250 align:middle line:90% We set the alpha to 1. 00:43:07.250 --> 00:43:09.667 align:middle line:84% We run this algorithm, it starts here, then it goes there, 00:43:09.667 --> 00:43:11.500 align:middle line:84% it goes there, boom, boom, boom, boom, boom, 00:43:11.500 --> 00:43:12.910 align:middle line:84% and then it finally ends up here. 00:43:12.910 --> 00:43:14.327 align:middle line:84% In, like, four or five iterations, 00:43:14.327 --> 00:43:16.430 align:middle line:90% it finds the minimum. 00:43:16.430 --> 00:43:18.910 align:middle line:84% This is obviously a very simple, well-behaved, nice 00:43:18.910 --> 00:43:22.670 align:middle line:84% little function, so you can easily optimize it. 00:43:22.670 --> 00:43:24.490 align:middle line:84% If you want, you can just go to this thing. 00:43:24.490 --> 00:43:27.810 align:middle line:84% There's a nice animation of this thing as well. 00:43:27.810 --> 00:43:28.310 align:middle line:90% OK. 00:43:28.310 --> 00:43:31.150 align:middle line:84% So now-- all right, before we actually 00:43:31.150 --> 00:43:32.868 align:middle line:84% go to the multivariable function, 00:43:32.868 --> 00:43:34.410 align:middle line:84% I want to go to the question that you 00:43:34.410 --> 00:43:36.180 align:middle line:90% posed about local minima. 00:43:36.180 --> 00:43:37.310 align:middle line:90% Actually, you know what? 00:43:37.310 --> 00:43:38.852 align:middle line:84% I think I may have some slides on it, 00:43:38.852 --> 00:43:40.290 align:middle line:84% so, sorry, I'll come back to this. 00:43:40.290 --> 00:43:41.930 align:middle line:90% So let's actually see what-- 00:43:41.930 --> 00:43:44.850 align:middle line:84% we looked at a toy example where there was only one variable. 00:43:44.850 --> 00:43:46.570 align:middle line:90% What if you have-- 00:43:46.570 --> 00:43:47.990 align:middle line:90% what if it was GPT-3? 00:43:47.990 --> 00:43:51.570 align:middle line:84% GPT-3 has 175 billion parameters. 00:43:51.570 --> 00:43:52.910 align:middle line:90% 175 billion. 00:43:52.910 --> 00:43:55.310 align:middle line:84% And GPT-4, they haven't published it, so we don't know. 00:43:55.310 --> 00:43:57.730 align:middle line:84% It's supposed to be 8 times as much. 00:43:57.730 --> 00:44:01.150 align:middle line:84% So, I mean, the number of parameters is massive. 00:44:01.150 --> 00:44:06.850 align:middle line:84% So basically, our loss function has billions of variables, 00:44:06.850 --> 00:44:10.970 align:middle line:84% billions of w's that we need to optimize over, minimize over. 00:44:10.970 --> 00:44:13.830 align:middle line:84% So we need to use this notion of a partial derivative. 00:44:13.830 --> 00:44:16.090 align:middle line:84% So let's take baby steps and say, OK, what if you 00:44:16.090 --> 00:44:18.270 align:middle line:90% have a two-variable function? 00:44:18.270 --> 00:44:19.970 align:middle line:84% Something like this, very simple. 00:44:19.970 --> 00:44:22.610 align:middle line:84% So what we can do is, we can calculate the partial derivative 00:44:22.610 --> 00:44:26.570 align:middle line:84% of g with respect to each of these w's, and 00:44:26.570 --> 00:44:29.830 align:middle line:84% the partial derivative, just to quickly refresh your memories, 00:44:29.830 --> 00:44:32.590 align:middle line:84% is you take a function, you pretend 00:44:32.590 --> 00:44:36.510 align:middle line:84% that everything other than w is a constant. 00:44:36.510 --> 00:44:38.790 align:middle line:84% Then the function becomes-- a function 00:44:38.790 --> 00:44:40.245 align:middle line:90% of just one variable w, w1. 00:44:40.245 --> 00:44:42.870 align:middle line:84% And then you just differentiate it like you do everything else, 00:44:42.870 --> 00:44:48.550 align:middle line:84% and you get something, and that is this thing here. 00:44:48.550 --> 00:44:51.230 align:middle line:84% And then you do the same thing for w2, you get this thing here. 00:44:51.230 --> 00:44:55.110 align:middle line:84% And then you just stack them up in a nice list. 00:44:55.110 --> 00:44:58.017 align:middle line:84% This is the vector of partial derivatives. 00:44:58.017 --> 00:44:59.350 align:middle line:90% So how should we interpret this? 00:44:59.350 --> 00:45:00.370 align:middle line:90% The same way as before. 00:45:00.370 --> 00:45:03.350 align:middle line:84% Basically, for a small change in w1, 00:45:03.350 --> 00:45:05.550 align:middle line:84% keeping w2 and everything else fixed, 00:45:05.550 --> 00:45:09.210 align:middle line:84% how does the function change if we change just w1 slightly? 00:45:09.210 --> 00:45:14.070 align:middle line:84% And similarly for w2 and all the way to w1, 75 billion. 00:45:14.070 --> 00:45:15.750 align:middle line:90% It's the same thing. 00:45:15.750 --> 00:45:20.690 align:middle line:84% So, now, when you have these functions with many variables, 00:45:20.690 --> 00:45:23.910 align:middle line:84% many w's, since we have a gradient 00:45:23.910 --> 00:45:26.390 align:middle line:84% for each one of those w's, we stack them 00:45:26.390 --> 00:45:29.470 align:middle line:84% up into a nice vector of derivatives, 00:45:29.470 --> 00:45:32.250 align:middle line:84% and this vector is called the gradient. 00:45:32.250 --> 00:45:36.410 align:middle line:90% And it's denoted using this-- 00:45:36.410 --> 00:45:38.834 align:middle line:84% anyone know what the symbol is called? 00:45:38.834 --> 00:45:39.798 align:middle line:90% STUDENT: Laplacian? 00:45:39.798 --> 00:45:41.730 align:middle line:90% RAMA RAMAKRISHNAN: Yeah? 00:45:41.730 --> 00:45:43.348 align:middle line:90% STUDENT: Laplacian? 00:45:43.348 --> 00:45:44.390 align:middle line:90% RAMA RAMAKRISHNAN: Maybe. 00:45:44.390 --> 00:45:48.930 align:middle line:84% Maybe it's a synonym, but the one I'm familiar with is nabla. 00:45:48.930 --> 00:45:51.122 align:middle line:84% Delta is the one that's upside-down triangle, 00:45:51.122 --> 00:45:53.330 align:middle line:84% but I think the upside-down triangle is called nabla, 00:45:53.330 --> 00:45:54.810 align:middle line:90% if I recall. 00:45:54.810 --> 00:45:55.930 align:middle line:90% Am I right? 00:45:55.930 --> 00:45:58.147 align:middle line:90% Thank you. 00:45:58.147 --> 00:45:58.730 align:middle line:90% He's my go-to. 00:45:58.730 --> 00:46:02.530 align:middle line:90% 00:46:02.530 --> 00:46:03.030 align:middle line:90% So yeah. 00:46:03.030 --> 00:46:05.990 align:middle line:84% So the gradient, we just call it the gradient, 00:46:05.990 --> 00:46:08.490 align:middle line:90% and it's written as this. 00:46:08.490 --> 00:46:10.970 align:middle line:84% All right, so what we do is we simply 00:46:10.970 --> 00:46:13.570 align:middle line:84% do gradient descent on every one of the w's 00:46:13.570 --> 00:46:16.990 align:middle line:90% using its partial derivative. 00:46:16.990 --> 00:46:21.010 align:middle line:84% So in a gradient step, we update w1 using this formula, 00:46:21.010 --> 00:46:23.010 align:middle line:90% w2 using this formula. 00:46:23.010 --> 00:46:25.485 align:middle line:90% Finished. 00:46:25.485 --> 00:46:27.110 align:middle line:84% We've just generalized gradient descent 00:46:27.110 --> 00:46:30.870 align:middle line:84% to an arbitrary number of variables. 00:46:30.870 --> 00:46:33.710 align:middle line:84% And of course, as before, this can be summarized compactly 00:46:33.710 --> 00:46:37.030 align:middle line:90% as this vector formula. 00:46:37.030 --> 00:46:38.260 align:middle line:90% We'll just do this. 00:46:38.260 --> 00:46:42.950 align:middle line:90% 00:46:42.950 --> 00:46:46.830 align:middle line:84% So what's going on here is that we 00:46:46.830 --> 00:46:59.310 align:middle line:84% have w1 or w1 minus alpha times the function g of w1 and w2, 00:46:59.310 --> 00:47:03.578 align:middle line:90% w2 minus alpha g by w2. 00:47:03.578 --> 00:47:05.870 align:middle line:84% And then all we're doing is we're just stacking them up 00:47:05.870 --> 00:47:08.990 align:middle line:90% into a vector like that. 00:47:08.990 --> 00:47:14.970 align:middle line:90% 00:47:14.970 --> 00:47:16.630 align:middle line:90% Minus alpha. 00:47:16.630 --> 00:47:22.100 align:middle line:90% And this vector is like that. 00:47:22.100 --> 00:47:27.690 align:middle line:90% 00:47:27.690 --> 00:47:31.760 align:middle line:84% So this can be written as just this vector w, the new vector, 00:47:31.760 --> 00:47:37.610 align:middle line:84% old vector minus alpha, and the gradient is finished. 00:47:37.610 --> 00:47:40.410 align:middle line:90% And you can see, if it's GPT-3. 00:47:40.410 --> 00:47:45.210 align:middle line:84% This vector is going to be 175 billion long. 00:47:45.210 --> 00:47:47.390 align:middle line:84% Whether it's 2 or 175 billion, who cares? 00:47:47.390 --> 00:47:50.370 align:middle line:90% It's the same thing, right? 00:47:50.370 --> 00:47:51.810 align:middle line:90% OK. 00:47:51.810 --> 00:47:53.755 align:middle line:84% So yeah, so that's what we have here. 00:47:53.755 --> 00:47:56.130 align:middle line:84% I'm really thrilled, by the way, this whole iPad business 00:47:56.130 --> 00:47:56.990 align:middle line:90% is working out. 00:47:56.990 --> 00:47:59.350 align:middle line:84% So I was a little worried about it. 00:47:59.350 --> 00:48:00.180 align:middle line:90% OK. 00:48:00.180 --> 00:48:03.218 align:middle line:84% So, if you look at two dimensions, this function-- 00:48:03.218 --> 00:48:05.510 align:middle line:84% and if you actually look at-- if you plot the function, 00:48:05.510 --> 00:48:08.292 align:middle line:84% this is w, the first w, the second w, and then 00:48:08.292 --> 00:48:09.750 align:middle line:84% this is actually the loss function, 00:48:09.750 --> 00:48:12.250 align:middle line:84% match that function, g w, and so you 00:48:12.250 --> 00:48:14.290 align:middle line:84% want to find the minimum here, and so this 00:48:14.290 --> 00:48:16.950 align:middle line:84% is how gradient descent will progress if you're 00:48:16.950 --> 00:48:18.410 align:middle line:90% starting from this point. 00:48:18.410 --> 00:48:21.887 align:middle line:84% Or you can also look at it from up top-down into the function. 00:48:21.887 --> 00:48:24.470 align:middle line:84% And that's what this picture is, and it shows gradient descent 00:48:24.470 --> 00:48:28.310 align:middle line:84% starting from there, and working its way down from here all 00:48:28.310 --> 00:48:30.630 align:middle line:90% the way to the center. 00:48:30.630 --> 00:48:34.590 align:middle line:90% OK, so all right, local minima. 00:48:34.590 --> 00:48:37.070 align:middle line:84% So now gradient descent will just 00:48:37.070 --> 00:48:41.930 align:middle line:90% stop near hopefully a minimum. 00:48:41.930 --> 00:48:44.690 align:middle line:84% But the problem is, it may not be a global minimum. 00:48:44.690 --> 00:48:47.390 align:middle line:90% It may not even be a minimum. 00:48:47.390 --> 00:48:51.070 align:middle line:84% So let's see what I'm talking about here. 00:48:51.070 --> 00:48:53.870 align:middle line:90% Here are some possibilities. 00:48:53.870 --> 00:48:57.590 align:middle line:90% So let's take a simple function. 00:48:57.590 --> 00:49:00.990 align:middle line:84% Let's say this is g w, this is w. 00:49:00.990 --> 00:49:03.900 align:middle line:84% And turns out, this function actually looks like this. 00:49:03.900 --> 00:49:12.230 align:middle line:90% 00:49:12.230 --> 00:49:14.620 align:middle line:90% So you can see here-- 00:49:14.620 --> 00:49:17.790 align:middle line:90% 00:49:17.790 --> 00:49:27.130 align:middle line:84% well, this point, this point here is a local minimum. 00:49:27.130 --> 00:49:28.810 align:middle line:90% This is a local minimum. 00:49:28.810 --> 00:49:30.890 align:middle line:90% This is a local minimum. 00:49:30.890 --> 00:49:35.130 align:middle line:84% These are all lots of local minima here. 00:49:35.130 --> 00:49:39.370 align:middle line:84% And, yeah, there's a lot of local minima here, too. 00:49:39.370 --> 00:49:43.930 align:middle line:84% So these are all places in which the derivative is going to be 0. 00:49:43.930 --> 00:49:46.250 align:middle line:84% So if you run gradient descent and it 00:49:46.250 --> 00:49:49.290 align:middle line:84% stops because the gradient is 0, you 00:49:49.290 --> 00:49:52.630 align:middle line:90% could be in any of these places. 00:49:52.630 --> 00:49:55.610 align:middle line:84% So there is no guarantee, so this in this picture 00:49:55.610 --> 00:49:59.450 align:middle line:84% happens to be maybe the global minimum because it's 00:49:59.450 --> 00:50:02.450 align:middle line:84% the lowest of the lot, But there's 00:50:02.450 --> 00:50:04.950 align:middle line:84% no guarantee you're actually going to get there. 00:50:04.950 --> 00:50:06.450 align:middle line:84% There's not even a guarantee you're 00:50:06.450 --> 00:50:08.070 align:middle line:84% going to be in any of these places 00:50:08.070 --> 00:50:10.010 align:middle line:84% because you could literally be in this thing 00:50:10.010 --> 00:50:14.650 align:middle line:84% here where it's taking a break and then continuing on down. 00:50:14.650 --> 00:50:16.750 align:middle line:84% That, by the way, is called a saddle point. 00:50:16.750 --> 00:50:19.630 align:middle line:84% I drew it badly, but the source of coming in-- taking a break 00:50:19.630 --> 00:50:22.150 align:middle line:84% and going down again, it's called a saddle point. 00:50:22.150 --> 00:50:25.010 align:middle line:84% So gradient descent can stop at a saddle point, 00:50:25.010 --> 00:50:26.155 align:middle line:90% it can stop at some minima. 00:50:26.155 --> 00:50:28.030 align:middle line:84% There's no guarantee it's going to be global. 00:50:28.030 --> 00:50:33.030 align:middle line:90% 00:50:33.030 --> 00:50:37.310 align:middle line:84% But it turns out, it has not mattered. 00:50:37.310 --> 00:50:38.810 align:middle line:90% So it has not mattered. 00:50:38.810 --> 00:50:41.987 align:middle line:84% And there are a whole bunch of reasons why it has not mattered. 00:50:41.987 --> 00:50:44.570 align:middle line:84% Because when you have these very complicated neural networks-- 00:50:44.570 --> 00:50:49.042 align:middle line:84% they're very complex functions, even finding a decent solution 00:50:49.042 --> 00:50:50.750 align:middle line:84% to these complicated networks is actually 00:50:50.750 --> 00:50:52.990 align:middle line:84% really good for solving the problem. 00:50:52.990 --> 00:50:56.010 align:middle line:84% You don't have to go to the best, best possible solution. 00:50:56.010 --> 00:50:58.690 align:middle line:84% And in fact, if you go to the best possible solution, 00:50:58.690 --> 00:51:02.110 align:middle line:84% you actually run the risk of overfitting. 00:51:02.110 --> 00:51:03.330 align:middle line:90% So that's one reason. 00:51:03.330 --> 00:51:05.190 align:middle line:90% The other interesting reason-- 00:51:05.190 --> 00:51:07.950 align:middle line:84% and by the way, this is a very hot area of research, 00:51:07.950 --> 00:51:09.610 align:middle line:90% to figure out exactly-- 00:51:09.610 --> 00:51:10.505 align:middle line:90% so it's like this. 00:51:10.505 --> 00:51:12.630 align:middle line:84% Empirically, what we have seen is that not worrying 00:51:12.630 --> 00:51:14.090 align:middle line:84% about local minima, global minima, 00:51:14.090 --> 00:51:16.930 align:middle line:84% all that stuff has not hurt us because these things 00:51:16.930 --> 00:51:18.670 align:middle line:90% are amazing. 00:51:18.670 --> 00:51:20.573 align:middle line:84% GPT-4, probably they just stopped somewhere. 00:51:20.573 --> 00:51:22.490 align:middle line:84% They probably-- it wasn't even a local minima. 00:51:22.490 --> 00:51:24.698 align:middle line:84% They're like, oh God, it's been running for six days, 00:51:24.698 --> 00:51:27.090 align:middle line:84% it's spent $2 million, let's stop. 00:51:27.090 --> 00:51:29.050 align:middle line:84% Because these are very expensive. 00:51:29.050 --> 00:51:31.777 align:middle line:90% But that's still so magical. 00:51:31.777 --> 00:51:34.110 align:middle line:84% You don't need to get to anywhere close to local minima. 00:51:34.110 --> 00:51:37.650 align:middle line:84% But there's another interesting point, which I read about. 00:51:37.650 --> 00:51:41.650 align:middle line:84% People basically hypothesize that for you 00:51:41.650 --> 00:51:44.350 align:middle line:84% to be at a local minimum, just think about what it means. 00:51:44.350 --> 00:51:47.050 align:middle line:84% It means that you're standing at a particular point. 00:51:47.050 --> 00:51:49.450 align:middle line:84% In every direction that you look, 00:51:49.450 --> 00:51:52.850 align:middle line:90% things are just sloping upward. 00:51:52.850 --> 00:51:54.650 align:middle line:90% Everything is sloping upward. 00:51:54.650 --> 00:51:56.870 align:middle line:84% If everything is sloping upward all around you, 00:51:56.870 --> 00:51:59.690 align:middle line:84% could you be at a local minima, by definition? 00:51:59.690 --> 00:52:03.698 align:middle line:84% But if you have a billion dimensions, what are the odds 00:52:03.698 --> 00:52:05.490 align:middle line:84% that you're going to be standing at a point 00:52:05.490 --> 00:52:08.730 align:middle line:84% where every one of those billion is going upward? 00:52:08.730 --> 00:52:10.145 align:middle line:90% The odds are really low. 00:52:10.145 --> 00:52:12.270 align:middle line:84% Chances are, some of them are going to be going up, 00:52:12.270 --> 00:52:14.395 align:middle line:84% some of them are going down, others are coming down 00:52:14.395 --> 00:52:16.870 align:middle line:84% and going the other way, it's going to be crazy. 00:52:16.870 --> 00:52:18.310 align:middle line:84% So in some sense, the best you can 00:52:18.310 --> 00:52:20.550 align:middle line:84% hope for in these very high-dimensional situations 00:52:20.550 --> 00:52:23.150 align:middle line:90% is probably a saddle point. 00:52:23.150 --> 00:52:25.870 align:middle line:84% And it turns out, it's good enough. 00:52:25.870 --> 00:52:29.843 align:middle line:84% So for those reasons, we are content with just running 00:52:29.843 --> 00:52:31.510 align:middle line:84% gradient descent with some tweaks, which 00:52:31.510 --> 00:52:35.910 align:middle line:84% I'll get to in a second, and it just performs really admirably. 00:52:35.910 --> 00:52:37.470 align:middle line:90% Yeah? 00:52:37.470 --> 00:52:41.290 align:middle line:84% STUDENT: How does alpha depend on how much compute you have? 00:52:41.290 --> 00:52:42.990 align:middle line:84% Like, would you set the learning rate 00:52:42.990 --> 00:52:45.430 align:middle line:90% based on that, or not really? 00:52:45.430 --> 00:52:48.030 align:middle line:84% RAMA RAMAKRISHNAN: No, the learning rate is really-- 00:52:48.030 --> 00:52:50.710 align:middle line:84% is a measure of-- it's like this. 00:52:50.710 --> 00:52:53.750 align:middle line:84% When you're at a point where you think that the gradient is 00:52:53.750 --> 00:52:54.870 align:middle line:90% looking nice and-- 00:52:54.870 --> 00:52:56.370 align:middle line:84% if you take a step in the direction, 00:52:56.370 --> 00:52:57.410 align:middle line:90% it's going to go down. 00:52:57.410 --> 00:52:59.790 align:middle line:84% And if you further believe that it's 00:52:59.790 --> 00:53:02.550 align:middle line:84% going to keep going down in the direction for a while, 00:53:02.550 --> 00:53:05.975 align:middle line:84% then you're very confident about taking a big step. 00:53:05.975 --> 00:53:07.350 align:middle line:84% But if you're like, I don't know, 00:53:07.350 --> 00:53:09.250 align:middle line:84% because maybe I take a little step, 00:53:09.250 --> 00:53:10.455 align:middle line:90% maybe I have to go this way. 00:53:10.455 --> 00:53:11.830 align:middle line:84% I can't go straight anymore, then 00:53:11.830 --> 00:53:13.913 align:middle line:84% you don't want to take a big step because then you 00:53:13.913 --> 00:53:14.730 align:middle line:90% have to backtrack. 00:53:14.730 --> 00:53:17.930 align:middle line:84% So those kinds of considerations go into the learning rate. 00:53:17.930 --> 00:53:21.630 align:middle line:84% And so that's of the rough answer to your question. 00:53:21.630 --> 00:53:24.850 align:middle line:84% It's not so much determined by compute and bandwidth and things 00:53:24.850 --> 00:53:25.850 align:middle line:90% like that. 00:53:25.850 --> 00:53:29.250 align:middle line:84% But, again, it's a complicated thing because sometimes 00:53:29.250 --> 00:53:31.370 align:middle line:84% with a given amount of compute, if you 00:53:31.370 --> 00:53:33.130 align:middle line:84% have a particular kind of data, you 00:53:33.130 --> 00:53:35.290 align:middle line:84% can have very aggressive learning rates. 00:53:35.290 --> 00:53:39.230 align:middle line:84% So it tends to be a bit jumbled up, complicated, 00:53:39.230 --> 00:53:41.930 align:middle line:84% but that's the quick surface-level idea of what's 00:53:41.930 --> 00:53:43.830 align:middle line:90% going on. 00:53:43.830 --> 00:53:44.330 align:middle line:90% OK. 00:53:44.330 --> 00:53:47.010 align:middle line:90% 00:53:47.010 --> 00:53:47.510 align:middle line:90% 9:31. 00:53:47.510 --> 00:53:50.938 align:middle line:90% 00:53:50.938 --> 00:53:52.730 align:middle line:84% By the way, folks, this lecture is probably 00:53:52.730 --> 00:53:54.570 align:middle line:84% one of the driest in the semester 00:53:54.570 --> 00:53:57.410 align:middle line:84% because I have to go through all the concepts. 00:53:57.410 --> 00:54:00.930 align:middle line:84% Once we start doing Colabs, things get a lot more lively. 00:54:00.930 --> 00:54:01.890 align:middle line:90% OK. 00:54:01.890 --> 00:54:02.510 align:middle line:90% All right. 00:54:02.510 --> 00:54:05.650 align:middle line:84% So, now let's talk about minimizing a loss function 00:54:05.650 --> 00:54:06.650 align:middle line:90% gradient descent. 00:54:06.650 --> 00:54:09.170 align:middle line:84% So here's our little binary cross entropy loss function 00:54:09.170 --> 00:54:11.230 align:middle line:90% that we saw from before. 00:54:11.230 --> 00:54:12.750 align:middle line:84% This is what we want to minimize. 00:54:12.750 --> 00:54:15.190 align:middle line:84% So if you look at this thing, where 00:54:15.190 --> 00:54:17.150 align:middle line:84% are the variables we need to change 00:54:17.150 --> 00:54:19.310 align:middle line:90% to minimize this function? 00:54:19.310 --> 00:54:21.310 align:middle line:84% Folks, don't look at your phones. 00:54:21.310 --> 00:54:24.540 align:middle line:84% Even with laptop and iPad use, don't look at your phones. 00:54:24.540 --> 00:54:27.670 align:middle line:90% 00:54:27.670 --> 00:54:31.970 align:middle line:84% STUDENT: Sorry, we've kind of abstracted the variable w, 00:54:31.970 --> 00:54:34.630 align:middle line:84% but just to bring it back, those are actually the weights 00:54:34.630 --> 00:54:36.073 align:middle line:90% in the neural network, right? 00:54:36.073 --> 00:54:37.990 align:middle line:84% RAMA RAMAKRISHNAN: The weights and the biases. 00:54:37.990 --> 00:54:39.450 align:middle line:84% I'm just calling them as weights. 00:54:39.450 --> 00:54:44.870 align:middle line:84% STUDENT: So the output of these minimization functions 00:54:44.870 --> 00:54:47.505 align:middle line:84% are going to be the actual weights in our model? 00:54:47.505 --> 00:54:48.630 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:54:48.630 --> 00:54:50.030 align:middle line:90% Exactly right. 00:54:50.030 --> 00:54:52.315 align:middle line:84% The whole name of the game is to find the weights. 00:54:52.315 --> 00:54:54.190 align:middle line:84% And so for example, when you see in the press 00:54:54.190 --> 00:54:58.950 align:middle line:84% that Meta has essentially made the weights of Llama 00:54:58.950 --> 00:55:03.030 align:middle line:84% 2 or something available, that's basically what they have done. 00:55:03.030 --> 00:55:04.830 align:middle line:90% Basically published the weights. 00:55:04.830 --> 00:55:06.030 align:middle line:84% STUDENT: The reason that's so valuable is-- 00:55:06.030 --> 00:55:06.990 align:middle line:84% RAMA RAMAKRISHNAN: Microphone, please. 00:55:06.990 --> 00:55:07.490 align:middle line:90% STUDENT: Oh. 00:55:07.490 --> 00:55:09.560 align:middle line:84% Because if you have a billion parameters, 00:55:09.560 --> 00:55:12.720 align:middle line:84% the compute time on that is horrendous and expensive. 00:55:12.720 --> 00:55:14.720 align:middle line:84% That's why those weights are so valuable? 00:55:14.720 --> 00:55:15.140 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:55:15.140 --> 00:55:17.140 align:middle line:84% The weights are the crown jewel because they 00:55:17.140 --> 00:55:19.140 align:middle line:84% are the result of a lot of money and time 00:55:19.140 --> 00:55:21.762 align:middle line:90% and smartness being spent. 00:55:21.762 --> 00:55:23.220 align:middle line:84% There is a separate question of why 00:55:23.220 --> 00:55:26.020 align:middle line:90% are they making it open source? 00:55:26.020 --> 00:55:28.740 align:middle line:84% I'm happy to chat about it offline. 00:55:28.740 --> 00:55:29.240 align:middle line:90% All right. 00:55:29.240 --> 00:55:29.540 align:middle line:90% Cool. 00:55:29.540 --> 00:55:31.550 align:middle line:84% So what are the variables we need to minimize? 00:55:31.550 --> 00:55:33.300 align:middle line:84% It's basically the parameters, and they're 00:55:33.300 --> 00:55:36.460 align:middle line:90% hiding inside the model term. 00:55:36.460 --> 00:55:38.258 align:middle line:90% Because what is the model? 00:55:38.258 --> 00:55:39.800 align:middle line:84% The model is some function like that. 00:55:39.800 --> 00:55:43.220 align:middle line:84% If you look at the simple GPA and experience 00:55:43.220 --> 00:55:45.820 align:middle line:84% thing we looked at on Monday, we finally 00:55:45.820 --> 00:55:48.660 align:middle line:84% figured out that the actual thing that comes out here 00:55:48.660 --> 00:55:51.260 align:middle line:84% is going to be this complicated function of all the x's 00:55:51.260 --> 00:55:52.840 align:middle line:84% and the w's and so on and so forth, 00:55:52.840 --> 00:55:56.440 align:middle line:84% and that complicated thing is showing up inside this thing. 00:55:56.440 --> 00:55:59.364 align:middle line:90% 00:55:59.364 --> 00:56:02.860 align:middle line:84% And the w's here are the variables we need to change 00:56:02.860 --> 00:56:04.580 align:middle line:90% to minimize the loss function. 00:56:04.580 --> 00:56:07.840 align:middle line:84% And it's important for you to note and understand 00:56:07.840 --> 00:56:13.240 align:middle line:84% that the values of x and y and so on are just data. 00:56:13.240 --> 00:56:15.800 align:middle line:84% You're not optimizing anything, they're just data. 00:56:15.800 --> 00:56:17.910 align:middle line:84% What you're optimizing is the w's. 00:56:17.910 --> 00:56:18.410 align:middle line:90% The weights. 00:56:18.410 --> 00:56:22.480 align:middle line:90% 00:56:22.480 --> 00:56:23.320 align:middle line:90% OK. 00:56:23.320 --> 00:56:26.258 align:middle line:84% So you imagine replacing the model 00:56:26.258 --> 00:56:27.800 align:middle line:84% here with the mathematical expression 00:56:27.800 --> 00:56:30.480 align:middle line:84% above, whenever this appears the loss function. 00:56:30.480 --> 00:56:33.480 align:middle line:84% And once you do that, your loss function is just a good old 00:56:33.480 --> 00:56:35.880 align:middle line:90% function of the w's. 00:56:35.880 --> 00:56:39.200 align:middle line:84% The fact that it's a loss function is kind of irrelevant. 00:56:39.200 --> 00:56:40.788 align:middle line:90% It's just a function. 00:56:40.788 --> 00:56:43.080 align:middle line:84% And since it's just a good old function of the w's, you 00:56:43.080 --> 00:56:45.960 align:middle line:84% can apply gradient descent to it as we normally would. 00:56:45.960 --> 00:56:46.820 align:middle line:90% It's no big deal. 00:56:46.820 --> 00:56:49.480 align:middle line:90% 00:56:49.480 --> 00:56:52.140 align:middle line:84% Which brings us to something called backpropagation. 00:56:52.140 --> 00:56:56.375 align:middle line:90% 00:56:56.375 --> 00:56:58.500 align:middle line:84% If you remember nothing else about backpropagation, 00:56:58.500 --> 00:56:59.960 align:middle line:90% just remember this-- 00:56:59.960 --> 00:57:01.860 align:middle line:84% never used the word backpropagation again. 00:57:01.860 --> 00:57:05.140 align:middle line:90% Only use the word backprop. 00:57:05.140 --> 00:57:07.900 align:middle line:84% You're hip and cool to the deep learning community. 00:57:07.900 --> 00:57:09.880 align:middle line:90% Backprop. 00:57:09.880 --> 00:57:12.340 align:middle line:90% All right, so what is backprop? 00:57:12.340 --> 00:57:15.660 align:middle line:84% Backprop is a very efficient way to compute the gradient 00:57:15.660 --> 00:57:17.540 align:middle line:90% of the loss function. 00:57:17.540 --> 00:57:19.620 align:middle line:84% So when you have this loss function, and let's 00:57:19.620 --> 00:57:22.260 align:middle line:84% say you have a billion w's, and you 00:57:22.260 --> 00:57:24.440 align:middle line:90% have 10 million data points-- 00:57:24.440 --> 00:57:27.620 align:middle line:84% so, the little n we saw was 10 million. 00:57:27.620 --> 00:57:30.580 align:middle line:90% That is a lot of computation. 00:57:30.580 --> 00:57:34.340 align:middle line:84% And that is just for one step of gradient descent. 00:57:34.340 --> 00:57:36.940 align:middle line:90% So backprop is a way-- 00:57:36.940 --> 00:57:38.980 align:middle line:84% is a very efficient and clever way 00:57:38.980 --> 00:57:41.700 align:middle line:84% to compute the gradient of the loss function, which 00:57:41.700 --> 00:57:45.580 align:middle line:84% takes advantage of the fact that what we have here is not 00:57:45.580 --> 00:57:47.980 align:middle line:84% some arbitrary model, it's a model that 00:57:47.980 --> 00:57:51.962 align:middle line:84% came from a particular kind of neural network, which has layers 00:57:51.962 --> 00:57:53.420 align:middle line:84% one after the other, and then there 00:57:53.420 --> 00:57:55.220 align:middle line:90% was an output at the very end. 00:57:55.220 --> 00:57:58.153 align:middle line:84% So what backprop does is it organizes 00:57:58.153 --> 00:57:59.820 align:middle line:84% the computation in the form of something 00:57:59.820 --> 00:58:01.740 align:middle line:84% called a computational graph, and the book has 00:58:01.740 --> 00:58:03.100 align:middle line:90% a good discussion about it. 00:58:03.100 --> 00:58:06.040 align:middle line:84% And so what we do is we start at the very end, 00:58:06.040 --> 00:58:10.160 align:middle line:84% we calculate the gradient of the loss with respect to the output. 00:58:10.160 --> 00:58:12.692 align:middle line:84% Then we move left, we calculate the gradient 00:58:12.692 --> 00:58:14.400 align:middle line:84% of that output with respect to the output 00:58:14.400 --> 00:58:17.200 align:middle line:90% of just the prior hidden layer. 00:58:17.200 --> 00:58:19.962 align:middle line:84% Step to the left, calculate the gradient of the current 00:58:19.962 --> 00:58:21.420 align:middle line:84% with respect to the previous layer. 00:58:21.420 --> 00:58:22.760 align:middle line:90% You get the idea. 00:58:22.760 --> 00:58:25.520 align:middle line:84% It's iterative and it moves backwards, and by 00:58:25.520 --> 00:58:28.400 align:middle line:84% doing so, you never repeat the same computation twice 00:58:28.400 --> 00:58:30.920 align:middle line:90% wastefully. 00:58:30.920 --> 00:58:32.320 align:middle line:90% That's the big advantage. 00:58:32.320 --> 00:58:35.720 align:middle line:84% You calculate once and reuse it many, many, many, many times. 00:58:35.720 --> 00:58:38.880 align:middle line:84% The second advantage is that if you organize it this way, 00:58:38.880 --> 00:58:43.000 align:middle line:84% it just becomes a sequence of matrix multiplications. 00:58:43.000 --> 00:58:46.320 align:middle line:84% And [AUDIO OUT] sequence of matrix 00:58:46.320 --> 00:58:49.480 align:middle line:84% multiplications and eliminates redundant calculations. 00:58:49.480 --> 00:58:52.760 align:middle line:84% And, best of all, there are these things called 00:58:52.760 --> 00:58:55.160 align:middle line:84% GPUs, Graphics Processing Units, originally invented 00:58:55.160 --> 00:58:57.400 align:middle line:84% to accelerate video game rendering, 00:58:57.400 --> 00:58:59.880 align:middle line:84% and as it turns out, to accelerate video game rendering, 00:58:59.880 --> 00:59:01.980 align:middle line:84% the core math operation you do is basically 00:59:01.980 --> 00:59:03.580 align:middle line:90% matrix multiplication. 00:59:03.580 --> 00:59:06.660 align:middle line:90% Some linear algebra operations. 00:59:06.660 --> 00:59:08.460 align:middle line:84% And so someone really, at some point, 00:59:08.460 --> 00:59:11.940 align:middle line:84% had the bright idea for deep learning, calculating gradients, 00:59:11.940 --> 00:59:14.480 align:middle line:84% and so on, we need to do matrix multiplications, 00:59:14.480 --> 00:59:17.660 align:middle line:84% and here are some specialized hardware that does really-- 00:59:17.660 --> 00:59:20.160 align:middle line:84% that does a fast job of matrix multiplications. 00:59:20.160 --> 00:59:22.340 align:middle line:90% Can we use this for that? 00:59:22.340 --> 00:59:26.260 align:middle line:84% And they did it, and all hell broke loose. 00:59:26.260 --> 00:59:28.060 align:middle line:90% That's literally what happened. 00:59:28.060 --> 00:59:30.900 align:middle line:84% And that's why NVIDIA is valued at about $1.5 trillion 00:59:30.900 --> 00:59:32.420 align:middle line:90% or something? 00:59:32.420 --> 00:59:33.460 align:middle line:90% So, yeah. 00:59:33.460 --> 00:59:35.340 align:middle line:90% So they are really good. 00:59:35.340 --> 00:59:40.980 align:middle line:84% And so backprop, the way you do backprop, plus using it on GPUs, 00:59:40.980 --> 00:59:44.420 align:middle line:84% leads to fast calculation of loss function gradients. 00:59:44.420 --> 00:59:47.140 align:middle line:84% If this thing were not true, this class 00:59:47.140 --> 00:59:50.300 align:middle line:84% would not exist because there wouldn't be any deep learning 00:59:50.300 --> 00:59:52.860 align:middle line:90% revolution. 00:59:52.860 --> 00:59:54.850 align:middle line:84% This is a fundamental, seminal reason. 00:59:54.850 --> 00:59:57.860 align:middle line:90% 00:59:57.860 --> 01:00:01.920 align:middle line:84% All right, so the book has a bunch of detail. 01:00:01.920 --> 01:00:05.080 align:middle line:90% And I actually did-- 01:00:05.080 --> 01:00:08.600 align:middle line:84% I hand-worked-out an example of calculating 01:00:08.600 --> 01:00:10.880 align:middle line:84% a gradient the old-fashioned way, 01:00:10.880 --> 01:00:13.400 align:middle line:84% and calculating it using backprop. 01:00:13.400 --> 01:00:15.680 align:middle line:84% So take a look at it, I'll post it on Canvas, 01:00:15.680 --> 01:00:18.420 align:middle line:84% and you will understand exactly where the savings come from, 01:00:18.420 --> 01:00:21.617 align:middle line:84% where the efficiency gains come from. 01:00:21.617 --> 01:00:23.700 align:middle line:84% Because of time, I'm not going to get into it now. 01:00:23.700 --> 01:00:26.440 align:middle line:90% 01:00:26.440 --> 01:00:27.020 align:middle line:90% All right. 01:00:27.020 --> 01:00:28.880 align:middle line:90% Any questions so far? 01:00:28.880 --> 01:00:30.175 align:middle line:90% Yep? 01:00:30.175 --> 01:00:30.800 align:middle line:90% STUDENT: Sorry. 01:00:30.800 --> 01:00:31.980 align:middle line:90% I followed up to-- 01:00:31.980 --> 01:00:34.380 align:middle line:84% and so we've done gradient descent, 01:00:34.380 --> 01:00:36.038 align:middle line:84% which is different than calculation 01:00:36.038 --> 01:00:37.580 align:middle line:84% of the gradient of the loss function. 01:00:37.580 --> 01:00:39.830 align:middle line:84% What is the purpose of the calculation of the gradient 01:00:39.830 --> 01:00:41.020 align:middle line:90% of the loss function? 01:00:41.020 --> 01:00:42.895 align:middle line:84% RAMA RAMAKRISHNAN: You calculate the gradient 01:00:42.895 --> 01:00:45.120 align:middle line:84% because the fundamental operation of gradient descent 01:00:45.120 --> 01:00:50.060 align:middle line:84% is to take your current value of w and modify it slightly, 01:00:50.060 --> 01:00:52.960 align:middle line:84% and the modification is old value minus learning 01:00:52.960 --> 01:00:53.830 align:middle line:90% rate times gradient. 01:00:53.830 --> 01:01:02.964 align:middle line:90% 01:01:02.964 --> 01:01:06.220 align:middle line:84% It would be cool if I say go back five slides to this thing 01:01:06.220 --> 01:01:07.340 align:middle line:90% and it just goes back. 01:01:07.340 --> 01:01:08.040 align:middle line:90% Product idea. 01:01:08.040 --> 01:01:08.540 align:middle line:90% Anyone? 01:01:08.540 --> 01:01:09.040 align:middle line:90% Startups? 01:01:09.040 --> 01:01:11.460 align:middle line:90% 01:01:11.460 --> 01:01:14.300 align:middle line:90% So this one. 01:01:14.300 --> 01:01:16.900 align:middle line:84% So this is the fundamental step of gradient descent. 01:01:16.900 --> 01:01:19.260 align:middle line:84% So this is the current value of w. 01:01:19.260 --> 01:01:22.420 align:middle line:84% You calculate the gradient at that current value multiplied 01:01:22.420 --> 01:01:26.140 align:middle line:84% by alpha, do this thing, and you get the new value. 01:01:26.140 --> 01:01:27.460 align:middle line:90% And you keep repeating. 01:01:27.460 --> 01:01:31.572 align:middle line:84% STUDENT: Right, but g w, that's not the loss function? 01:01:31.572 --> 01:01:33.280 align:middle line:84% RAMA RAMAKRISHNAN: It is a loss function. 01:01:33.280 --> 01:01:34.240 align:middle line:84% STUDENT: That is the loss function? 01:01:34.240 --> 01:01:35.240 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 01:01:35.240 --> 01:01:37.860 align:middle line:84% Here, I'm just using g as an arbitrary function 01:01:37.860 --> 01:01:39.720 align:middle line:84% to just to demonstrate the point, 01:01:39.720 --> 01:01:41.900 align:middle line:84% but when you're optimizing-- when you're training 01:01:41.900 --> 01:01:44.260 align:middle line:84% a neural network, what you're actually doing 01:01:44.260 --> 01:01:47.697 align:middle line:84% is minimizing a loss function, loss of w. 01:01:47.697 --> 01:01:49.280 align:middle line:84% STUDENT: Sorry, I got things mixed up. 01:01:49.280 --> 01:01:51.420 align:middle line:90% RAMA RAMAKRISHNAN: No worries. 01:01:51.420 --> 01:01:52.885 align:middle line:90% Yeah? 01:01:52.885 --> 01:01:54.760 align:middle line:84% STUDENT: How do we define the initial weights 01:01:54.760 --> 01:01:56.947 align:middle line:90% for the neural network? 01:01:56.947 --> 01:01:59.030 align:middle line:84% RAMA RAMAKRISHNAN: So, yeah, the initial weights-- 01:01:59.030 --> 01:02:02.240 align:middle line:90% 01:02:02.240 --> 01:02:04.500 align:middle line:84% so there are many ways to-- so first of all, 01:02:04.500 --> 01:02:07.560 align:middle line:84% they are initialized randomly, but randomly 01:02:07.560 --> 01:02:09.720 align:middle line:84% doesn't mean you can just pick any random weight. 01:02:09.720 --> 01:02:11.120 align:middle line:84% There are actually some good ways 01:02:11.120 --> 01:02:13.360 align:middle line:84% to randomly pick the weights, and those are 01:02:13.360 --> 01:02:15.760 align:middle line:90% called initialization schemes. 01:02:15.760 --> 01:02:18.480 align:middle line:84% And there are a bunch of very effective initialization schemes 01:02:18.480 --> 01:02:20.140 align:middle line:84% people have figured out over the years, 01:02:20.140 --> 01:02:22.880 align:middle line:84% and those things are baked into Keras as the default. 01:02:22.880 --> 01:02:24.920 align:middle line:84% So Keras, I believe, uses something 01:02:24.920 --> 01:02:29.160 align:middle line:84% called the HE initialization, H-E initialization, 01:02:29.160 --> 01:02:31.795 align:middle line:84% or the Xavier Glorot initialization. 01:02:31.795 --> 01:02:33.920 align:middle line:84% I wouldn't worry about it, just go with the default 01:02:33.920 --> 01:02:35.825 align:middle line:90% initialization. 01:02:35.825 --> 01:02:38.200 align:middle line:84% And the reason why they have to be very careful about how 01:02:38.200 --> 01:02:40.640 align:middle line:84% these weights are initialized is because if you 01:02:40.640 --> 01:02:44.200 align:middle line:84% have a very big network and if you initialize badly, 01:02:44.200 --> 01:02:49.040 align:middle line:84% then the gradient will just explode as you calculate it. 01:02:49.040 --> 01:02:52.040 align:middle line:84% The earlier layers, the weights will have massive gradients 01:02:52.040 --> 01:02:54.253 align:middle line:84% or gradients will vanish, so therefore, 01:02:54.253 --> 01:02:56.420 align:middle line:84% call the exploding gradient problem or the vanishing 01:02:56.420 --> 01:02:57.420 align:middle line:90% gradient problem. 01:02:57.420 --> 01:02:59.020 align:middle line:84% To avoid all those things, researchers 01:02:59.020 --> 01:03:01.270 align:middle line:84% figured out some clever way to initialize so that it's 01:03:01.270 --> 01:03:03.740 align:middle line:90% well-behaved throughout. 01:03:03.740 --> 01:03:05.380 align:middle line:90% Yeah? 01:03:05.380 --> 01:03:09.240 align:middle line:84% STUDENT: If using backprops and GPUs was so critical, 01:03:09.240 --> 01:03:11.620 align:middle line:84% I'm just curious, who first did it and when? 01:03:11.620 --> 01:03:13.560 align:middle line:90% Was this a couple years ago? 01:03:13.560 --> 01:03:14.620 align:middle line:90% Was it a company? 01:03:14.620 --> 01:03:15.120 align:middle line:90% Was it-- 01:03:15.120 --> 01:03:16.120 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 01:03:16.120 --> 01:03:20.140 align:middle line:84% Well, GPUs have been used for deep learning, I want to say-- 01:03:20.140 --> 01:03:24.745 align:middle line:84% I think the first case may have been in the mid-2005, 2006 sort 01:03:24.745 --> 01:03:26.220 align:middle line:90% of thing. 01:03:26.220 --> 01:03:29.660 align:middle line:84% But I would say that it burst out onto the world stage 01:03:29.660 --> 01:03:33.140 align:middle line:84% and made everyone take notice when a deep learning 01:03:33.140 --> 01:03:39.180 align:middle line:84% model called AlexNet in 2012 won a very famous computer vision 01:03:39.180 --> 01:03:40.700 align:middle line:90% competition. 01:03:40.700 --> 01:03:42.380 align:middle line:90% And it beat the-- 01:03:42.380 --> 01:03:45.302 align:middle line:84% and it set a world record for how good it was, 01:03:45.302 --> 01:03:47.760 align:middle line:84% and that's when everyone was like, hey, what is this thing? 01:03:47.760 --> 01:03:49.720 align:middle line:84% And that's really when it burst onto the world stage. 01:03:49.720 --> 01:03:51.137 align:middle line:84% I'll talk a bit more about it when 01:03:51.137 --> 01:03:54.098 align:middle line:84% I get into the computer vision segment of the class. 01:03:54.098 --> 01:03:55.640 align:middle line:84% But you can Google AlexNet and you'll 01:03:55.640 --> 01:03:59.412 align:middle line:84% find a whole bunch of history around it. 01:03:59.412 --> 01:04:01.120 align:middle line:84% STUDENT: Hypothetically, then, is it true 01:04:01.120 --> 01:04:04.960 align:middle line:84% that if we could get to a global minima, 01:04:04.960 --> 01:04:07.340 align:middle line:84% that would mean there would be no hallucinations? 01:04:07.340 --> 01:04:09.840 align:middle line:84% RAMA RAMAKRISHNAN: Aha, good question. 01:04:09.840 --> 01:04:13.120 align:middle line:84% So if it is perfect, if you get to a global minimum-- first 01:04:13.120 --> 01:04:15.580 align:middle line:84% of all, global minima doesn't mean the model is perfect. 01:04:15.580 --> 01:04:18.440 align:middle line:90% It may still have some loss. 01:04:18.440 --> 01:04:24.320 align:middle line:84% But global minima is going to be on the training data. 01:04:24.320 --> 01:04:26.960 align:middle line:84% You can imagine that the test data, future data 01:04:26.960 --> 01:04:29.520 align:middle line:90% has its own loss function. 01:04:29.520 --> 01:04:31.818 align:middle line:84% So what is minimum here may not be minimum there. 01:04:31.818 --> 01:04:32.610 align:middle line:90% That's the problem. 01:04:32.610 --> 01:04:36.440 align:middle line:90% 01:04:36.440 --> 01:04:37.460 align:middle line:90% Is it a comment? 01:04:37.460 --> 01:04:37.960 align:middle line:90% No? 01:04:37.960 --> 01:04:38.880 align:middle line:90% OK. 01:04:38.880 --> 01:04:42.080 align:middle line:84% STUDENT: Just saying that that would mean that also you 01:04:42.080 --> 01:04:42.965 align:middle line:90% can be overfitting-- 01:04:42.965 --> 01:04:44.840 align:middle line:84% RAMA RAMAKRISHNAN: Correct, exactly, exactly. 01:04:44.840 --> 01:04:47.600 align:middle line:84% So if you overdo, if you find the best thing in the training 01:04:47.600 --> 01:04:50.180 align:middle line:84% function, chances are, it doesn't match the best 01:04:50.180 --> 01:04:52.180 align:middle line:90% thing of the test data. 01:04:52.180 --> 01:04:56.420 align:middle line:84% So on the test data, you're actually doing badly. 01:04:56.420 --> 01:04:57.060 align:middle line:90% OK. 01:04:57.060 --> 01:04:59.120 align:middle line:90% So, come back to this. 01:04:59.120 --> 01:05:03.820 align:middle line:90% 01:05:03.820 --> 01:05:04.580 align:middle line:90% OK. 01:05:04.580 --> 01:05:07.620 align:middle line:84% Now the final twist to the tale here, 01:05:07.620 --> 01:05:09.780 align:middle line:84% we're going to go from something, gradient descent, 01:05:09.780 --> 01:05:11.800 align:middle line:84% to something called stochastic gradient descent. 01:05:11.800 --> 01:05:14.460 align:middle line:84% And stochastic gradient descent, or SGD, 01:05:14.460 --> 01:05:17.500 align:middle line:84% is the workhorse for all deep learning. 01:05:17.500 --> 01:05:21.140 align:middle line:84% And funnily enough, SGD is simpler than GD. 01:05:21.140 --> 01:05:23.820 align:middle line:84% Just when you thought it couldn't get simpler. 01:05:23.820 --> 01:05:25.300 align:middle line:90% OK. 01:05:25.300 --> 01:05:27.340 align:middle line:84% So for large data sets, computing 01:05:27.340 --> 01:05:30.300 align:middle line:84% the gradient of the loss function can be very expensive, 01:05:30.300 --> 01:05:33.040 align:middle line:84% needless to say, because it has to be done at every step, 01:05:33.040 --> 01:05:35.860 align:middle line:84% and the cardinality of the data set is really big, 01:05:35.860 --> 01:05:38.080 align:middle line:84% and you may have billions of parameters, 01:05:38.080 --> 01:05:43.180 align:middle line:84% it's just very, very tough to compute it even with backprop. 01:05:43.180 --> 01:05:45.700 align:middle line:84% So the solution is, at each iteration-- 01:05:45.700 --> 01:05:48.000 align:middle line:84% when I say iteration, I'm talking about this step 01:05:48.000 --> 01:05:50.160 align:middle line:90% of gradient descent. 01:05:50.160 --> 01:05:54.280 align:middle line:84% Instead of using all the data, instead of calculating the loss 01:05:54.280 --> 01:05:58.145 align:middle line:84% function by averaging the loss across all n data points, 01:05:58.145 --> 01:05:59.520 align:middle line:84% and then calculating the gradient 01:05:59.520 --> 01:06:01.920 align:middle line:84% of that thing, what you do is you just 01:06:01.920 --> 01:06:04.360 align:middle line:90% choose a small sample randomly. 01:06:04.360 --> 01:06:06.520 align:middle line:84% You choose just a few of the n observations, 01:06:06.520 --> 01:06:08.400 align:middle line:90% and we call it a batch. 01:06:08.400 --> 01:06:10.200 align:middle line:84% So for example, the number of data 01:06:10.200 --> 01:06:12.560 align:middle line:84% points-- you may have 10 billion data points, 01:06:12.560 --> 01:06:14.360 align:middle line:84% but in every iteration, you may literally 01:06:14.360 --> 01:06:18.300 align:middle line:84% grab just 32 or 64, something really small. 01:06:18.300 --> 01:06:20.280 align:middle line:90% Like absurdly small. 01:06:20.280 --> 01:06:24.240 align:middle line:84% And then you pretend that, OK, that's all the data I have, 01:06:24.240 --> 01:06:26.700 align:middle line:84% you calculate the loss, find the gradient, 01:06:26.700 --> 01:06:28.390 align:middle line:90% and just use that here instead. 01:06:28.390 --> 01:06:31.200 align:middle line:90% 01:06:31.200 --> 01:06:34.320 align:middle line:84% So this is called stochastic gradient descent. 01:06:34.320 --> 01:06:38.520 align:middle line:84% So strictly speaking, theoretically, SGD uses just 01:06:38.520 --> 01:06:41.480 align:middle line:84% one data point, but in practice, we 01:06:41.480 --> 01:06:42.880 align:middle line:90% use what's called a minibatch. 01:06:42.880 --> 01:06:45.060 align:middle line:90% 32, 64, whatever. 01:06:45.060 --> 01:06:47.780 align:middle line:84% And so minibatch gradient descent is just loosely 01:06:47.780 --> 01:06:49.570 align:middle line:84% called stochastic gradient descent, SGD. 01:06:49.570 --> 01:06:52.740 align:middle line:90% 01:06:52.740 --> 01:06:56.140 align:middle line:84% So an SGD, as it turns out, you can see, 01:06:56.140 --> 01:06:59.180 align:middle line:84% it's clearly very efficient because it's just 01:06:59.180 --> 01:07:01.180 align:middle line:90% processing a few at a time. 01:07:01.180 --> 01:07:03.540 align:middle line:84% And in fact, if you have a lot of data, 01:07:03.540 --> 01:07:06.320 align:middle line:84% and you calculate the gradient of the loss function, 01:07:06.320 --> 01:07:09.600 align:middle line:90% it may not even fit into memory. 01:07:09.600 --> 01:07:11.020 align:middle line:90% It's really problematic. 01:07:11.020 --> 01:07:12.620 align:middle line:84% But with SGD, it says, I don't care 01:07:12.620 --> 01:07:14.995 align:middle line:84% whether you have a billion data points or a trillion data 01:07:14.995 --> 01:07:17.680 align:middle line:84% points, just give me 32 at a time. 01:07:17.680 --> 01:07:20.660 align:middle line:90% And you just keep on doing it. 01:07:20.660 --> 01:07:22.740 align:middle line:84% It turns out, because not all the points are 01:07:22.740 --> 01:07:24.780 align:middle line:84% used in the calculation, this only 01:07:24.780 --> 01:07:26.595 align:middle line:90% approximates the true gradient. 01:07:26.595 --> 01:07:27.720 align:middle line:90% It's only an approximation. 01:07:27.720 --> 01:07:29.887 align:middle line:84% It's not the real thing, it's only an approximation, 01:07:29.887 --> 01:07:32.120 align:middle line:84% but it works extremely well in practice. 01:07:32.120 --> 01:07:33.438 align:middle line:90% Extremely well in practice. 01:07:33.438 --> 01:07:34.980 align:middle line:84% And there's a whole bunch of research 01:07:34.980 --> 01:07:37.340 align:middle line:84% that goes into why is it so effective. 01:07:37.340 --> 01:07:40.580 align:middle line:84% And people are discovering interesting things about SGD, 01:07:40.580 --> 01:07:42.560 align:middle line:84% but we don't have a definitive theory 01:07:42.560 --> 01:07:44.360 align:middle line:90% as to why it's so good yet. 01:07:44.360 --> 01:07:46.400 align:middle line:84% We have some interesting research threads 01:07:46.400 --> 01:07:47.840 align:middle line:90% that have happened. 01:07:47.840 --> 01:07:51.840 align:middle line:84% And, very tantalizingly, very tantalizingly, 01:07:51.840 --> 01:07:53.840 align:middle line:84% because it's only an approximation 01:07:53.840 --> 01:07:59.520 align:middle line:84% of the true gradient, SGD can actually escape local minima. 01:07:59.520 --> 01:08:04.200 align:middle line:84% So in the true loss function, you're at a local minimum, 01:08:04.200 --> 01:08:07.580 align:middle line:84% but in SGD loss function, when you're doing SGD, 01:08:07.580 --> 01:08:10.400 align:middle line:84% you're reaching the minimum of the SGD loss 01:08:10.400 --> 01:08:14.240 align:middle line:84% function, which actually may not be the actual loss function. 01:08:14.240 --> 01:08:15.942 align:middle line:84% So as you're moving around, you're 01:08:15.942 --> 01:08:17.359 align:middle line:84% actually jumping from local minima 01:08:17.359 --> 01:08:20.399 align:middle line:84% to local minima of the actual loss function. 01:08:20.399 --> 01:08:23.080 align:middle line:84% I know, that's a mouthful, I'm happy to tell you more. 01:08:23.080 --> 01:08:26.516 align:middle line:84% It's just a side thing that I just want you to be aware of. 01:08:26.516 --> 01:08:28.819 align:middle line:84% One of the reasons why SGD is actually effective, 01:08:28.819 --> 01:08:34.097 align:middle line:84% it's almost like you work less and you do better. 01:08:34.097 --> 01:08:35.680 align:middle line:84% How many times does it happen in life? 01:08:35.680 --> 01:08:36.580 align:middle line:90% This is one of them. 01:08:36.580 --> 01:08:40.030 align:middle line:90% 01:08:40.030 --> 01:08:43.145 align:middle line:90% Now SGD comes in many flavors. 01:08:43.145 --> 01:08:45.520 align:middle line:84% Many siblings, it's got a lot of siblings and variations. 01:08:45.520 --> 01:08:46.819 align:middle line:90% It's a big family. 01:08:46.819 --> 01:08:49.899 align:middle line:84% And we're going to use a particular flavor called Adam 01:08:49.899 --> 01:08:52.140 align:middle line:84% as our default in this course, and I'll 01:08:52.140 --> 01:08:55.979 align:middle line:84% get back to it when you get into the Colabs and things like that. 01:08:55.979 --> 01:08:58.140 align:middle line:90% All right. 01:08:58.140 --> 01:09:01.053 align:middle line:84% By the way you know how all these 01:09:01.053 --> 01:09:03.720 align:middle line:84% pictures I've been showing you a nice little function like that, 01:09:03.720 --> 01:09:05.420 align:middle line:90% a little bowl and so on? 01:09:05.420 --> 01:09:09.100 align:middle line:84% This is a visualization of an actual neural network loss 01:09:09.100 --> 01:09:11.380 align:middle line:90% function. 01:09:11.380 --> 01:09:13.439 align:middle line:84% You can see the hills and valleys and the cracks 01:09:13.439 --> 01:09:14.939 align:middle line:90% and so on and so forth. 01:09:14.939 --> 01:09:17.859 align:middle line:84% And you can check out that paper to get more insight 01:09:17.859 --> 01:09:20.439 align:middle line:84% into how they actually came up with this visualization. 01:09:20.439 --> 01:09:21.660 align:middle line:90% It's crazy. 01:09:21.660 --> 01:09:24.340 align:middle line:90% It's complicated. 01:09:24.340 --> 01:09:25.580 align:middle line:90% Yeah? 01:09:25.580 --> 01:09:29.580 align:middle line:84% STUDENT: So for SGD, do you perform the iterations until you 01:09:29.580 --> 01:09:32.340 align:middle line:84% minimize the loss function for each minibatch 01:09:32.340 --> 01:09:34.200 align:middle line:84% and then move to another minibatch? 01:09:34.200 --> 01:09:34.300 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 01:09:34.300 --> 01:09:36.500 align:middle line:84% So what you do is you take each minibatch, 01:09:36.500 --> 01:09:40.100 align:middle line:84% and then you calculate the loss for that minibatch, 01:09:40.100 --> 01:09:41.560 align:middle line:90% you find the gradient. 01:09:41.560 --> 01:09:43.740 align:middle line:84% And use the gradient and update the w. 01:09:43.740 --> 01:09:45.279 align:middle line:84% Then you pick up the next minibatch. 01:09:45.279 --> 01:09:47.200 align:middle line:84% STUDENT: So you don't pick a minibatch 01:09:47.200 --> 01:09:49.660 align:middle line:84% and try to perform the iterations on that minibatch 01:09:49.660 --> 01:09:50.160 align:middle line:90% until you-- 01:09:50.160 --> 01:09:51.399 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 01:09:51.399 --> 01:09:52.779 align:middle line:90% Each minibatch, one iteration. 01:09:52.779 --> 01:09:54.200 align:middle line:90% Each minibatch, one iteration. 01:09:54.200 --> 01:09:55.840 align:middle line:84% Because if you do a lot of iterations 01:09:55.840 --> 01:09:57.790 align:middle line:84% on one minibatch, first of all, you never 01:09:57.790 --> 01:10:00.040 align:middle line:84% be sure that you're going to find any optimal solution 01:10:00.040 --> 01:10:02.260 align:middle line:84% because you're not guaranteed of any global minima. 01:10:02.260 --> 01:10:03.843 align:middle line:84% And secondly, it's much better for you 01:10:03.843 --> 01:10:05.420 align:middle line:84% to get new information constantly 01:10:05.420 --> 01:10:08.440 align:middle line:84% because what you can do is you can revisit the minibatch later 01:10:08.440 --> 01:10:09.640 align:middle line:90% on. 01:10:09.640 --> 01:10:12.400 align:middle line:84% And that gets us to these things called epochs and batch size 01:10:12.400 --> 01:10:14.920 align:middle line:84% and so on, which we'll get into a lot of gory detail 01:10:14.920 --> 01:10:16.320 align:middle line:90% when we do the Colab. 01:10:16.320 --> 01:10:17.860 align:middle line:90% So let's revisit that question. 01:10:17.860 --> 01:10:20.440 align:middle line:90% It's a good question. 01:10:20.440 --> 01:10:22.640 align:middle line:90% Yeah? 01:10:22.640 --> 01:10:25.220 align:middle line:84% STUDENT: When you do the backprop process, it's-- 01:10:25.220 --> 01:10:25.680 align:middle line:90% RAMA RAMAKRISHNAN: Very good. 01:10:25.680 --> 01:10:26.940 align:middle line:90% Backprop, no backpropagation. 01:10:26.940 --> 01:10:27.280 align:middle line:90% Nice. 01:10:27.280 --> 01:10:28.155 align:middle line:90% STUDENT: I made sure. 01:10:28.155 --> 01:10:29.240 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 01:10:29.240 --> 01:10:32.040 align:middle line:84% It sounded like you started from the layers that 01:10:32.040 --> 01:10:34.100 align:middle line:84% were closest to the output and you went backward. 01:10:34.100 --> 01:10:38.030 align:middle line:84% And my question is, are you doing that once or is it 01:10:38.030 --> 01:10:39.090 align:middle line:90% looping multiple times? 01:10:39.090 --> 01:10:39.590 align:middle line:90% And then-- 01:10:39.590 --> 01:10:40.350 align:middle line:84% RAMA RAMAKRISHNAN: You just do it once. 01:10:40.350 --> 01:10:41.110 align:middle line:90% STUDENT: Just once? 01:10:41.110 --> 01:10:42.110 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 01:10:42.110 --> 01:10:44.588 align:middle line:84% So for each gradient calculation, you do it once. 01:10:44.588 --> 01:10:46.630 align:middle line:84% STUDENT: Why does it want to start from the layer 01:10:46.630 --> 01:10:47.410 align:middle line:90% that's closest-- 01:10:47.410 --> 01:10:48.830 align:middle line:84% or why do you want to start it from the layer that's 01:10:48.830 --> 01:10:49.890 align:middle line:90% closest to the output? 01:10:49.890 --> 01:10:50.030 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 01:10:50.030 --> 01:10:52.113 align:middle line:84% So basically what happens is, let's say that, just 01:10:52.113 --> 01:10:54.830 align:middle line:84% for argument, that you go in the reverse direction, 01:10:54.830 --> 01:10:57.390 align:middle line:84% you will discover that a lot of paths to go from the left 01:10:57.390 --> 01:10:59.070 align:middle line:84% to the right will end up calculating 01:10:59.070 --> 01:11:01.390 align:middle line:84% certain intermediate quantities, including 01:11:01.390 --> 01:11:05.977 align:middle line:84% the very final gradient item, again and again and again. 01:11:05.977 --> 01:11:08.310 align:middle line:84% The same thing is going to be calculated again and again 01:11:08.310 --> 01:11:08.990 align:middle line:90% and again. 01:11:08.990 --> 01:11:11.170 align:middle line:84% So by starting from the end and working backwards, 01:11:11.170 --> 01:11:14.270 align:middle line:84% you just reuse stuff you've already calculated. 01:11:14.270 --> 01:11:15.850 align:middle line:90% So that is the rough idea. 01:11:15.850 --> 01:11:18.648 align:middle line:84% But if you see my PDF, I've actually worked out the example, 01:11:18.648 --> 01:11:20.690 align:middle line:84% and that will demonstrate what I'm talking about. 01:11:20.690 --> 01:11:23.350 align:middle line:90% 01:11:23.350 --> 01:11:26.120 align:middle line:84% By the way, the backprop is just a-- 01:11:26.120 --> 01:11:28.950 align:middle line:90% 01:11:28.950 --> 01:11:31.640 align:middle line:84% in calculus, we have something called the chain rule. 01:11:31.640 --> 01:11:33.890 align:middle line:84% To calculate the derivative of a complicated function, 01:11:33.890 --> 01:11:36.550 align:middle line:84% you calculate the derivative of the outer function, 01:11:36.550 --> 01:11:38.610 align:middle line:84% then the inner function, and so on and so forth. 01:11:38.610 --> 01:11:39.985 align:middle line:84% The backdrop is essentially a way 01:11:39.985 --> 01:11:41.730 align:middle line:84% to organize the chain rule to work 01:11:41.730 --> 01:11:44.070 align:middle line:84% with the neural network, layer-by-layer architecture, 01:11:44.070 --> 01:11:44.590 align:middle line:90% that's all. 01:11:44.590 --> 01:11:48.250 align:middle line:90% 01:11:48.250 --> 01:11:49.530 align:middle line:90% Yeah? 01:11:49.530 --> 01:11:52.610 align:middle line:84% STUDENT: So is it fair to say that once we are finding 01:11:52.610 --> 01:11:55.490 align:middle line:84% the local minimum, we are not optimizing to all 01:11:55.490 --> 01:11:57.853 align:middle line:84% the g w's because this local minimum 01:11:57.853 --> 01:12:00.270 align:middle line:84% is coming like from different curves from different lines, 01:12:00.270 --> 01:12:01.437 align:middle line:90% so is that fair to say-- 01:12:01.437 --> 01:12:03.770 align:middle line:84% RAMA RAMAKRISHNAN: When we are using stochastic gradient 01:12:03.770 --> 01:12:04.530 align:middle line:90% descent, yes. 01:12:04.530 --> 01:12:06.430 align:middle line:84% So in stochastic gradient descent, 01:12:06.430 --> 01:12:09.290 align:middle line:84% when you take, say, 32 data points from a million 01:12:09.290 --> 01:12:12.230 align:middle line:84% and you're calculating the loss for that 32 data points, 01:12:12.230 --> 01:12:15.010 align:middle line:84% you're basically trying to do a gradient step. 01:12:15.010 --> 01:12:18.130 align:middle line:84% The w equals w minus alpha gradient thing. 01:12:18.130 --> 01:12:21.490 align:middle line:84% You're doing it for that 32 points 01:12:21.490 --> 01:12:25.770 align:middle line:84% loss function, which is not the 1 million points loss function. 01:12:25.770 --> 01:12:27.610 align:middle line:90% That's why it's approximate. 01:12:27.610 --> 01:12:30.010 align:middle line:84% But the approximation, instead of hurting you, 01:12:30.010 --> 01:12:32.170 align:middle line:84% actually helps you because it helps 01:12:32.170 --> 01:12:35.710 align:middle line:84% you escape the local minima of the global loss function. 01:12:35.710 --> 01:12:38.002 align:middle line:84% So it's an interesting and somewhat technically subtle 01:12:38.002 --> 01:12:40.210 align:middle line:84% point, which is why I'm not getting into it too much, 01:12:40.210 --> 01:12:42.830 align:middle line:84% but I'm happy to give pointers if people are interested. 01:12:42.830 --> 01:12:44.288 align:middle line:90% Yeah? 01:12:44.288 --> 01:12:46.330 align:middle line:84% STUDENT: When you say you initialize the weights, 01:12:46.330 --> 01:12:49.750 align:middle line:84% you initialize for the whole network or just the end layer 01:12:49.750 --> 01:12:51.110 align:middle line:90% and then go backwards-- 01:12:51.110 --> 01:12:53.830 align:middle line:84% RAMA RAMAKRISHNAN: So usually it's everything in one shot. 01:12:53.830 --> 01:12:55.110 align:middle line:84% Because if you don't initialize everything 01:12:55.110 --> 01:12:56.568 align:middle line:84% in one shot, what's going to happen 01:12:56.568 --> 01:12:58.590 align:middle line:84% is that you can't do the forward computation 01:12:58.590 --> 01:13:01.110 align:middle line:90% to find the prediction. 01:13:01.110 --> 01:13:02.850 align:middle line:84% And so they are done independently, 01:13:02.850 --> 01:13:05.730 align:middle line:84% and the initialization schemes will take into account, 01:13:05.730 --> 01:13:08.310 align:middle line:84% OK, I'm initializing the weights between a layer which 01:13:08.310 --> 01:13:11.870 align:middle line:84% has 10 nodes on one side and 32 on the other side, and the 10 01:13:11.870 --> 01:13:15.950 align:middle line:84% and the 32 actually play a role in how you initialize. 01:13:15.950 --> 01:13:16.830 align:middle line:90% OK. 01:13:16.830 --> 01:13:19.910 align:middle line:84% So, the summary of the overall training flow 01:13:19.910 --> 01:13:22.830 align:middle line:90% is that you have an input. 01:13:22.830 --> 01:13:24.370 align:middle line:84% It goes through a bunch of layers. 01:13:24.370 --> 01:13:25.650 align:middle line:90% You come up with a prediction. 01:13:25.650 --> 01:13:27.590 align:middle line:84% You compare it to the true values. 01:13:27.590 --> 01:13:30.250 align:middle line:84% And these two things go into the loss function calculation, 01:13:30.250 --> 01:13:32.170 align:middle line:90% you get a loss number. 01:13:32.170 --> 01:13:34.250 align:middle line:84% And you do it for, say, 10 points or 32 points 01:13:34.250 --> 01:13:35.390 align:middle line:90% or a million points. 01:13:35.390 --> 01:13:38.110 align:middle line:84% And this loss thing goes into the optimizer, 01:13:38.110 --> 01:13:39.650 align:middle line:90% which calculates the gradient. 01:13:39.650 --> 01:13:41.190 align:middle line:84% And once it calculates the gradient, 01:13:41.190 --> 01:13:44.210 align:middle line:84% it updates the weights of every layer using the w 01:13:44.210 --> 01:13:46.470 align:middle line:84% equals w minus alpha times gradient formula, 01:13:46.470 --> 01:13:47.710 align:middle line:90% gradient descent formula. 01:13:47.710 --> 01:13:50.370 align:middle line:84% And then you keep doing this again and again and again. 01:13:50.370 --> 01:13:52.955 align:middle line:90% This is the overall flow. 01:13:52.955 --> 01:13:54.330 align:middle line:84% This is how our little network is 01:13:54.330 --> 01:13:57.070 align:middle line:84% going to get built for heart disease prediction, 01:13:57.070 --> 01:14:02.690 align:middle line:84% this is how GPT was built. And this is how AlphaFold was built. 01:14:02.690 --> 01:14:07.330 align:middle line:84% And AlphaGo was built. You get the idea. 01:14:07.330 --> 01:14:09.010 align:middle line:84% I mean, it's astonishing, frankly. 01:14:09.010 --> 01:14:11.130 align:middle line:84% If you're not getting goosebumps at the thought 01:14:11.130 --> 01:14:13.808 align:middle line:84% that this simple thing can do all these complicated things, 01:14:13.808 --> 01:14:15.100 align:middle line:90% we really need to talk offline. 01:14:15.100 --> 01:14:17.487 align:middle line:90% 01:14:17.487 --> 01:14:19.070 align:middle line:84% STUDENT: There was a hand raised here. 01:14:19.070 --> 01:14:20.410 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 01:14:20.410 --> 01:14:21.290 align:middle line:90% STUDENT: Sorry. 01:14:21.290 --> 01:14:24.836 align:middle line:84% Just quickly, this is for each minibatch, right? 01:14:24.836 --> 01:14:27.850 align:middle line:84% so my question is, if you came with different weights 01:14:27.850 --> 01:14:31.460 align:middle line:84% for each minibatch, how do you add it up? 01:14:31.460 --> 01:14:34.710 align:middle line:84% Like, OK, this weight is the perfect combination 01:14:34.710 --> 01:14:38.230 align:middle line:84% for this minibatch, but you have way different weight 01:14:38.230 --> 01:14:40.670 align:middle line:84% for another minibatch, how do you combine those two? 01:14:40.670 --> 01:14:41.590 align:middle line:90% RAMA RAMAKRISHNAN: No. 01:14:41.590 --> 01:14:44.570 align:middle line:84% At each point, what you do is you find you find-- 01:14:44.570 --> 01:14:47.835 align:middle line:84% you start with a weight, you run it through for a minibatch, 01:14:47.835 --> 01:14:49.210 align:middle line:84% you come up with a loss function, 01:14:49.210 --> 01:14:50.565 align:middle line:90% you calculate the gradient. 01:14:50.565 --> 01:14:52.690 align:middle line:84% And now using the gradient, you updated the weight, 01:14:52.690 --> 01:14:53.870 align:middle line:84% now you have a new set of weights, 01:14:53.870 --> 01:14:55.078 align:middle line:90% which is the updated weights. 01:14:55.078 --> 01:14:57.550 align:middle line:90% Call it w2 instead of w1. 01:14:57.550 --> 01:14:59.470 align:middle line:84% Now w2 is using your network, and when 01:14:59.470 --> 01:15:00.845 align:middle line:84% you take the next minibatch, it's 01:15:00.845 --> 01:15:03.550 align:middle line:84% going to use w2 to calculate the prediction. 01:15:03.550 --> 01:15:06.150 align:middle line:84% And this whole flow will become a lot clearer 01:15:06.150 --> 01:15:08.790 align:middle line:90% when we do the Colabs. 01:15:08.790 --> 01:15:11.790 align:middle line:90% OK, so we have three minutes. 01:15:11.790 --> 01:15:14.070 align:middle line:84% I don't want to go into regularization 01:15:14.070 --> 01:15:19.710 align:middle line:84% overfitting in three minutes, so let's have some more questions. 01:15:19.710 --> 01:15:20.670 align:middle line:90% Yeah? 01:15:20.670 --> 01:15:22.550 align:middle line:84% STUDENT: Can you use any activation function 01:15:22.550 --> 01:15:26.110 align:middle line:84% as long as it gives positive values for x squared 01:15:26.110 --> 01:15:28.570 align:middle line:90% or mod x or something? 01:15:28.570 --> 01:15:34.170 align:middle line:84% RAMA RAMAKRISHNAN: You can use a variety of activation functions, 01:15:34.170 --> 01:15:35.130 align:middle line:90% but-- 01:15:35.130 --> 01:15:37.610 align:middle line:84% there's a whole literature on the pros and cons 01:15:37.610 --> 01:15:39.810 align:middle line:84% of various activation functions that you could use, 01:15:39.810 --> 01:15:43.430 align:middle line:84% but in general, you have to make sure of a couple of things. 01:15:43.430 --> 01:15:46.683 align:middle line:84% One is that when you do backprop, 01:15:46.683 --> 01:15:48.850 align:middle line:84% the gradient is going to flow through the activation 01:15:48.850 --> 01:15:50.610 align:middle line:84% function in the reverse direction, 01:15:50.610 --> 01:15:52.610 align:middle line:84% and the activation function should actually 01:15:52.610 --> 01:15:55.410 align:middle line:84% make sure the gradient doesn't get squished. 01:15:55.410 --> 01:15:58.570 align:middle line:84% It shouldn't get squished, it shouldn't get exploded. 01:15:58.570 --> 01:15:59.987 align:middle line:84% So those are some considerations-- 01:15:59.987 --> 01:16:01.445 align:middle line:84% these are technical considerations, 01:16:01.445 --> 01:16:03.790 align:middle line:84% but all those conditions have to be taken into account. 01:16:03.790 --> 01:16:07.050 align:middle line:84% If you can take those into account, then you're OK. 01:16:07.050 --> 01:16:09.010 align:middle line:84% That's the key thing to keep in mind. 01:16:09.010 --> 01:16:11.890 align:middle line:84% And that's, in fact, why the ReLU is actually very popular, 01:16:11.890 --> 01:16:13.990 align:middle line:84% because as long as the value is positive, 01:16:13.990 --> 01:16:17.130 align:middle line:84% the gradient of the ReLU is just 1 01:16:17.130 --> 01:16:24.330 align:middle line:84% because if you look at something-- 01:16:24.330 --> 01:16:24.830 align:middle line:90% oops. 01:16:24.830 --> 01:16:28.750 align:middle line:90% 01:16:28.750 --> 01:16:30.430 align:middle line:90% Because it's frozen. 01:16:30.430 --> 01:16:31.950 align:middle line:90% I jinxed it. 01:16:31.950 --> 01:16:34.870 align:middle line:90% So, sorry, livestream. 01:16:34.870 --> 01:16:39.870 align:middle line:84% If you have something like this, the ReLU is like that. 01:16:39.870 --> 01:16:42.190 align:middle line:84% So the gradient here is always going 01:16:42.190 --> 01:16:45.330 align:middle line:84% to be 1, which means that as long as the value is positive, 01:16:45.330 --> 01:16:46.890 align:middle line:84% whatever gradient comes in like this, 01:16:46.890 --> 01:16:49.650 align:middle line:84% it just gets multiplied by 1 and gets pushed out the other side. 01:16:49.650 --> 01:16:53.870 align:middle line:84% So it doesn't get harmed or squished or anything like that. 01:16:53.870 --> 01:16:57.190 align:middle line:84% So that's one reason why the ReLU is very popular, because it 01:16:57.190 --> 01:16:59.270 align:middle line:84% preserves the gradient while injecting 01:16:59.270 --> 01:17:01.590 align:middle line:84% almost the minimum amount of non-linearity 01:17:01.590 --> 01:17:02.770 align:middle line:90% to do interesting things. 01:17:02.770 --> 01:17:05.350 align:middle line:90% 01:17:05.350 --> 01:17:07.630 align:middle line:90% Yeah? 01:17:07.630 --> 01:17:10.450 align:middle line:84% STUDENT: If you have a high number of dimensions, 01:17:10.450 --> 01:17:14.190 align:middle line:84% can you do minibatching on features, dimensions 01:17:14.190 --> 01:17:15.930 align:middle line:90% instead of just observations? 01:17:15.930 --> 01:17:18.090 align:middle line:84% Keep the same number of observations, 01:17:18.090 --> 01:17:21.630 align:middle line:84% but just take a small sample of the number of features 01:17:21.630 --> 01:17:22.930 align:middle line:90% that you're actually using? 01:17:22.930 --> 01:17:24.430 align:middle line:84% RAMA RAMAKRISHNAN: Oh, I see, I see. 01:17:24.430 --> 01:17:27.090 align:middle line:84% So you're saying-- let's say you have 10 features. 01:17:27.090 --> 01:17:29.830 align:middle line:84% Instead of taking all data points or 10 features, 01:17:29.830 --> 01:17:33.330 align:middle line:84% what if you choose five features and just use them 01:17:33.330 --> 01:17:34.930 align:middle line:90% and do this thing? 01:17:34.930 --> 01:17:38.190 align:middle line:84% As long as you can actually compute the prediction. 01:17:38.190 --> 01:17:41.930 align:middle line:84% To compute the prediction, you may need all 10 features. 01:17:41.930 --> 01:17:45.210 align:middle line:84% Or you need to have some defaults for those features. 01:17:45.210 --> 01:17:48.230 align:middle line:84% But if you define defaults for those other five features, 01:17:48.230 --> 01:17:50.310 align:middle line:84% you're basically using all features. 01:17:50.310 --> 01:17:51.310 align:middle line:90% So that's the key thing. 01:17:51.310 --> 01:17:54.390 align:middle line:84% Can you actually calculate the prediction by manipulating? 01:17:54.390 --> 01:17:57.810 align:middle line:90% And typically you can't. 01:17:57.810 --> 01:17:58.950 align:middle line:90% All right. 01:17:58.950 --> 01:17:59.450 align:middle line:90% OK, folks. 01:17:59.450 --> 01:18:00.790 align:middle line:90% 9:55, I'm done. 01:18:00.790 --> 01:18:04.080 align:middle line:84% Have a great rest of your week, I'll see you on Monday. 01:18:04.080 --> 01:18:22.000 align:middle line:90%