WEBVTT 00:00:00.000 --> 00:00:01.972 align:middle line:90% [SQUEAKING] 00:00:01.972 --> 00:00:03.451 align:middle line:90% [RUSTLING] 00:00:03.451 --> 00:00:04.930 align:middle line:90% [CLICKING] 00:00:04.930 --> 00:00:16.280 align:middle line:90% 00:00:16.280 --> 00:00:17.400 align:middle line:90% RAMA RAMAKRISHNAN: OK. 00:00:17.400 --> 00:00:17.900 align:middle line:90% All right. 00:00:17.900 --> 00:00:20.520 align:middle line:90% Let's get going. 00:00:20.520 --> 00:00:22.280 align:middle line:90% Today is going to be packed. 00:00:22.280 --> 00:00:24.880 align:middle line:84% I'm going to spend the first roughly half of the lecture 00:00:24.880 --> 00:00:27.920 align:middle line:90% on actually building a model-- 00:00:27.920 --> 00:00:29.840 align:middle line:90% a Keras model in Colab-- 00:00:29.840 --> 00:00:32.360 align:middle line:84% to solve the heart disease problem we saw earlier, 00:00:32.360 --> 00:00:34.880 align:middle line:84% and then switch gears halfway, and then 00:00:34.880 --> 00:00:37.660 align:middle line:84% talk about how to solve image classification. 00:00:37.660 --> 00:00:40.850 align:middle line:84% So we're going to do two Colabs today. 00:00:40.850 --> 00:00:42.100 align:middle line:90% I've been talking about Colab. 00:00:42.100 --> 00:00:43.740 align:middle line:90% Colab, I've been teasing you. 00:00:43.740 --> 00:00:45.300 align:middle line:90% We'll actually do Colabs today. 00:00:45.300 --> 00:00:46.020 align:middle line:90% All right. 00:00:46.020 --> 00:00:47.722 align:middle line:90% So summary of where we were. 00:00:47.722 --> 00:00:49.680 align:middle line:84% By the way, I've shut off the lights in the top 00:00:49.680 --> 00:00:51.480 align:middle line:84% because, when I switch to Colab, it's 00:00:51.480 --> 00:00:53.360 align:middle line:84% going to be much better for you folks, particularly the folks 00:00:53.360 --> 00:00:54.950 align:middle line:84% in the back, to be able to see it. 00:00:54.950 --> 00:00:57.840 align:middle line:84% But I hope you can see the slide right now. 00:00:57.840 --> 00:00:59.890 align:middle line:90% Yes? 00:00:59.890 --> 00:01:00.390 align:middle line:90% OK. 00:01:00.390 --> 00:01:01.010 align:middle line:90% Great. 00:01:01.010 --> 00:01:04.836 align:middle line:84% So this is just a quick recap of what we did last class. 00:01:04.836 --> 00:01:07.283 align:middle line:84% Broadly speaking, training a neural network 00:01:07.283 --> 00:01:09.950 align:middle line:84% essentially is no different than training other kinds of models. 00:01:09.950 --> 00:01:12.870 align:middle line:84% We have a bunch of parameters, i.e., weights and biases, 00:01:12.870 --> 00:01:15.170 align:middle line:84% and we need to use the data to find 00:01:15.170 --> 00:01:16.990 align:middle line:90% good values of those weights. 00:01:16.990 --> 00:01:18.190 align:middle line:90% And what does good mean? 00:01:18.190 --> 00:01:20.130 align:middle line:84% Typically, it means that we define 00:01:20.130 --> 00:01:23.570 align:middle line:84% some measure of discrepancy between what the model predicts 00:01:23.570 --> 00:01:26.330 align:middle line:84% for a given set of weights and what the right answer is, what 00:01:26.330 --> 00:01:27.910 align:middle line:90% the ground truth answer is. 00:01:27.910 --> 00:01:30.910 align:middle line:84% And then we try to find weights that minimize this discrepancy. 00:01:30.910 --> 00:01:31.430 align:middle line:90% That's it. 00:01:31.430 --> 00:01:33.970 align:middle line:84% And this notion of discrepancy is called a loss function. 00:01:33.970 --> 00:01:37.250 align:middle line:84% So broadly speaking, the overall training flow 00:01:37.250 --> 00:01:38.830 align:middle line:90% is that you define some network. 00:01:38.830 --> 00:01:39.533 align:middle line:90% It has an input. 00:01:39.533 --> 00:01:40.950 align:middle line:84% It goes through a bunch of layers. 00:01:40.950 --> 00:01:42.230 align:middle line:84% You come up with some predictions. 00:01:42.230 --> 00:01:43.272 align:middle line:90% You take the predictions. 00:01:43.272 --> 00:01:44.590 align:middle line:90% You take the true values. 00:01:44.590 --> 00:01:46.910 align:middle line:84% And then those two go into the loss function, 00:01:46.910 --> 00:01:48.365 align:middle line:90% i.e., the discrepancy function. 00:01:48.365 --> 00:01:49.990 align:middle line:84% And then you come up with a loss score. 00:01:49.990 --> 00:01:52.110 align:middle line:84% And then you send it to the optimizer, 00:01:52.110 --> 00:01:55.290 align:middle line:84% which then proceeds to calculate the gradient of this loss 00:01:55.290 --> 00:01:57.190 align:middle line:84% function with respect to all the parameters. 00:01:57.190 --> 00:01:59.900 align:middle line:84% And then it updates all the weights using that gradient, 00:01:59.900 --> 00:02:01.940 align:middle line:90% and then the process repeats. 00:02:01.940 --> 00:02:02.640 align:middle line:90% That's it. 00:02:02.640 --> 00:02:04.420 align:middle line:90% So that is the training flow. 00:02:04.420 --> 00:02:05.500 align:middle line:90% Quick recap. 00:02:05.500 --> 00:02:08.820 align:middle line:84% Now, we also talked about the optimization algorithm 00:02:08.820 --> 00:02:12.620 align:middle line:84% we're going to use, which is called gradient descent. 00:02:12.620 --> 00:02:16.360 align:middle line:84% And gradient descent, as you noticed, in each iteration, 00:02:16.360 --> 00:02:20.100 align:middle line:84% every data point is being used to make predictions 00:02:20.100 --> 00:02:21.700 align:middle line:84% and therefore to calculate the loss, 00:02:21.700 --> 00:02:23.900 align:middle line:84% and then to calculate the gradient. 00:02:23.900 --> 00:02:27.300 align:middle line:84% And then we pointed out that gradient descent is actually not 00:02:27.300 --> 00:02:30.420 align:middle line:84% as good as something called stochastic gradient descent-- 00:02:30.420 --> 00:02:32.660 align:middle line:84% stochastic gradient descent-- where, 00:02:32.660 --> 00:02:35.320 align:middle line:84% instead of taking all the points, 00:02:35.320 --> 00:02:38.460 align:middle line:84% we just randomly choose a small number of points, 00:02:38.460 --> 00:02:41.900 align:middle line:84% pretend for a moment as if those are the only points we have, 00:02:41.900 --> 00:02:45.240 align:middle line:84% make predictions, calculate loss, calculate gradient, 00:02:45.240 --> 00:02:46.420 align:middle line:90% and go on. 00:02:46.420 --> 00:02:49.300 align:middle line:84% So that was the basic idea behind stochastic gradient 00:02:49.300 --> 00:02:50.860 align:middle line:90% descent. 00:02:50.860 --> 00:02:52.120 align:middle line:90% Two different kinds of things. 00:02:52.120 --> 00:02:54.860 align:middle line:84% Now, what it means is that when we actually 00:02:54.860 --> 00:02:58.300 align:middle line:84% start training the model, as we will in a few minutes, 00:02:58.300 --> 00:03:00.740 align:middle line:84% because we only take a few points at a time, 00:03:00.740 --> 00:03:03.380 align:middle line:84% we have to be a bit careful in what's going on. 00:03:03.380 --> 00:03:04.840 align:middle line:84% And I want to make sure you clearly 00:03:04.840 --> 00:03:07.200 align:middle line:84% understand what the differences are before we actually 00:03:07.200 --> 00:03:08.300 align:middle line:90% get to the Colab. 00:03:08.300 --> 00:03:10.560 align:middle line:90% OK? 00:03:10.560 --> 00:03:11.140 align:middle line:90% All right. 00:03:11.140 --> 00:03:14.800 align:middle line:84% So there is the notion of an epoch. 00:03:14.800 --> 00:03:17.760 align:middle line:84% An epoch essentially just means that we make one 00:03:17.760 --> 00:03:19.880 align:middle line:90% pass through the training data. 00:03:19.880 --> 00:03:20.960 align:middle line:90% All the training data. 00:03:20.960 --> 00:03:22.660 align:middle line:90% We make one pass through it. 00:03:22.660 --> 00:03:27.280 align:middle line:84% And so what is one pass is that if you have something 00:03:27.280 --> 00:03:31.080 align:middle line:84% like gradient descent, one pass means every data point 00:03:31.080 --> 00:03:32.460 align:middle line:90% is sent through the network. 00:03:32.460 --> 00:03:34.820 align:middle line:84% We calculate its predictions, calculate the loss, 00:03:34.820 --> 00:03:36.740 align:middle line:90% calculate the gradient. 00:03:36.740 --> 00:03:39.020 align:middle line:84% We run every training sample through it. 00:03:39.020 --> 00:03:42.300 align:middle line:84% We calculate the gradient, which is just this thing here. 00:03:42.300 --> 00:03:46.400 align:middle line:84% I mean, I will sometimes say d of loss, dW. 00:03:46.400 --> 00:03:48.260 align:middle line:84% Derivative of loss with respect to W. 00:03:48.260 --> 00:03:50.680 align:middle line:84% Sometimes I might use this nabla symbol. 00:03:50.680 --> 00:03:52.640 align:middle line:90% These are all interchangeable. 00:03:52.640 --> 00:03:54.720 align:middle line:84% So we'll calculate the gradient, and then 00:03:54.720 --> 00:03:57.330 align:middle line:84% we update using some version of this. 00:03:57.330 --> 00:04:01.130 align:middle line:84% But we just do it once at the end of the epoch. 00:04:01.130 --> 00:04:03.930 align:middle line:84% Because if you have 10 billion data points, every one of them 00:04:03.930 --> 00:04:06.330 align:middle line:84% flows through, you get 10 billion outputs. 00:04:06.330 --> 00:04:08.080 align:middle line:84% And then we calculate the epoch just once. 00:04:08.080 --> 00:04:10.205 align:middle line:84% At the end of this thing, we calculate the gradient 00:04:10.205 --> 00:04:10.890 align:middle line:90% and update once. 00:04:10.890 --> 00:04:13.330 align:middle line:90% One update per epoch. 00:04:13.330 --> 00:04:14.970 align:middle line:90% Yes? 00:04:14.970 --> 00:04:18.610 align:middle line:84% Now, in stochastic gradient descent, what we do 00:04:18.610 --> 00:04:21.930 align:middle line:84% is that we process the data in batches. 00:04:21.930 --> 00:04:24.890 align:middle line:84% Small numbers of points at a time. 00:04:24.890 --> 00:04:27.150 align:middle line:84% And these are called-- technically speaking, 00:04:27.150 --> 00:04:28.872 align:middle line:90% they're called minibatches. 00:04:28.872 --> 00:04:29.830 align:middle line:90% I don't know about you. 00:04:29.830 --> 00:04:31.455 align:middle line:84% I just get tired of saying minibatches, 00:04:31.455 --> 00:04:34.090 align:middle line:84% so I'm just going to say batches from this point on. 00:04:34.090 --> 00:04:34.690 align:middle line:90% OK? 00:04:34.690 --> 00:04:37.650 align:middle line:84% And in fact, that is widely done in the literature. 00:04:37.650 --> 00:04:40.670 align:middle line:84% So we'll have to process it in batches. 00:04:40.670 --> 00:04:42.190 align:middle line:90% So we take the training data. 00:04:42.190 --> 00:04:44.690 align:middle line:84% And then we divide it up into batches-- 00:04:44.690 --> 00:04:47.810 align:middle line:84% batch 1, batch 2, all the way till the final batch. 00:04:47.810 --> 00:04:52.010 align:middle line:84% And so what we do is, for each batch, 00:04:52.010 --> 00:04:54.380 align:middle line:84% we basically do gradient descent. 00:04:54.380 --> 00:04:56.980 align:middle line:84% For each batch, we take batch 1, and then we 00:04:56.980 --> 00:04:59.815 align:middle line:84% run just the training samples in that batch 00:04:59.815 --> 00:05:01.440 align:middle line:84% through the network to get predictions. 00:05:01.440 --> 00:05:03.800 align:middle line:84% We calculate the gradient, we update the parameters, 00:05:03.800 --> 00:05:05.040 align:middle line:90% and then we go to batch 2. 00:05:05.040 --> 00:05:07.320 align:middle line:84% Then we go to batch 3 and so on and so forth. 00:05:07.320 --> 00:05:11.280 align:middle line:84% So pictorially, this is how it's going to look like. 00:05:11.280 --> 00:05:14.220 align:middle line:84% Let's say the first batch is, say, 32 points. 00:05:14.220 --> 00:05:15.672 align:middle line:90% We take those 32 points. 00:05:15.672 --> 00:05:17.880 align:middle line:84% We run it through the network, get all the stuff out. 00:05:17.880 --> 00:05:20.500 align:middle line:84% We calculate the gradient, update the weights. 00:05:20.500 --> 00:05:25.300 align:middle line:84% So when we now get to batch 2, the weights have changed. 00:05:25.300 --> 00:05:26.773 align:middle line:90% They have been updated. 00:05:26.773 --> 00:05:28.940 align:middle line:84% And then we do the same thing with batch 2, batch 3, 00:05:28.940 --> 00:05:31.258 align:middle line:84% and all the way till we get to the end of the thing. 00:05:31.258 --> 00:05:32.800 align:middle line:84% And when we are done with this thing, 00:05:32.800 --> 00:05:36.300 align:middle line:84% this whole thing is called a what? 00:05:36.300 --> 00:05:38.100 align:middle line:90% An epoch. 00:05:38.100 --> 00:05:40.140 align:middle line:90% This whole thing is an epoch. 00:05:40.140 --> 00:05:42.780 align:middle line:90% OK? 00:05:42.780 --> 00:05:43.710 align:middle line:90% All right. 00:05:43.710 --> 00:05:45.460 align:middle line:84% So the question, of course, is that if you 00:05:45.460 --> 00:05:47.085 align:middle line:84% have a bunch of data points, and you're 00:05:47.085 --> 00:05:49.180 align:middle line:84% going to run stochastic gradient descent on it, 00:05:49.180 --> 00:05:54.550 align:middle line:84% in a particular epoch, how many batches are going to be there? 00:05:54.550 --> 00:05:56.410 align:middle line:84% How many batches are going to be there? 00:05:56.410 --> 00:05:58.410 align:middle line:84% Now, Keras is going to calculate all this stuff. 00:05:58.410 --> 00:05:59.290 align:middle line:84% You don't have to worry about it. 00:05:59.290 --> 00:06:01.590 align:middle line:84% But you just need to understand exactly what happens. 00:06:01.590 --> 00:06:04.070 align:middle line:84% So my philosophy, by the way, is that you 00:06:04.070 --> 00:06:06.088 align:middle line:84% have to know the details of what's going on. 00:06:06.088 --> 00:06:08.630 align:middle line:84% If you don't know the details, and if you haven't figured out 00:06:08.630 --> 00:06:10.840 align:middle line:84% at least once, you will not actually 00:06:10.840 --> 00:06:12.590 align:middle line:84% be able to think new and creative thoughts 00:06:12.590 --> 00:06:14.510 align:middle line:90% for a new problem. 00:06:14.510 --> 00:06:17.630 align:middle line:84% It's because the concepts are not manipulable in your head 00:06:17.630 --> 00:06:18.570 align:middle line:90% yet. 00:06:18.570 --> 00:06:19.070 align:middle line:90% OK. 00:06:19.070 --> 00:06:23.930 align:middle line:90% 00:06:23.930 --> 00:06:27.350 align:middle line:90% Please use the microphone. 00:06:27.350 --> 00:06:31.430 align:middle line:84% AUDIENCE: So when we talk about SDG and we are talking about-- 00:06:31.430 --> 00:06:32.890 align:middle line:84% we are only taking some part of it, 00:06:32.890 --> 00:06:35.910 align:middle line:84% is it what we are saying is that we only take some variables, 00:06:35.910 --> 00:06:37.930 align:middle line:84% or we're only taking some part of the data? 00:06:37.930 --> 00:06:40.283 align:middle line:84% RAMA RAMAKRISHNAN: We are taking some rows. 00:06:40.283 --> 00:06:41.950 align:middle line:84% AUDIENCE: We're taking only-- oh, right. 00:06:41.950 --> 00:06:42.310 align:middle line:90% So that-- 00:06:42.310 --> 00:06:43.310 align:middle line:90% RAMA RAMAKRISHNAN: Data points. 00:06:43.310 --> 00:06:44.570 align:middle line:84% AUDIENCE: That means the batch itself. 00:06:44.570 --> 00:06:45.695 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:06:45.695 --> 00:06:48.130 align:middle line:84% So for example, let's say you have 1,000 data points, 00:06:48.130 --> 00:06:51.250 align:middle line:84% 1,000 rows of observations, 1,000 patients in the heart 00:06:51.250 --> 00:06:54.650 align:middle line:84% disease example, or 1,000 images that you're trying to classify. 00:06:54.650 --> 00:06:58.470 align:middle line:84% You take, let's say, 32 of those images, 32 of those patients, 00:06:58.470 --> 00:06:59.610 align:middle line:90% and that's a batch. 00:06:59.610 --> 00:07:01.562 align:middle line:84% Then you go to the next 32, then the next 32, 00:07:01.562 --> 00:07:03.770 align:middle line:84% and so on and so forth, until you run out of patients 00:07:03.770 --> 00:07:05.250 align:middle line:90% or run out of images. 00:07:05.250 --> 00:07:06.912 align:middle line:84% AUDIENCE: And each iterative time you 00:07:06.912 --> 00:07:09.370 align:middle line:84% are updating with the weights, the weights that you've got? 00:07:09.370 --> 00:07:09.650 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:07:09.650 --> 00:07:12.210 align:middle line:84% AUDIENCE: And it means you keep correcting it 00:07:12.210 --> 00:07:13.410 align:middle line:90% or keep moving towards-- 00:07:13.410 --> 00:07:14.440 align:middle line:84% RAMA RAMAKRISHNAN: You're basically updating the weights 00:07:14.440 --> 00:07:15.107 align:middle line:90% as you go along. 00:07:15.107 --> 00:07:17.138 align:middle line:90% AUDIENCE: Updating the weights. 00:07:17.138 --> 00:07:18.930 align:middle line:84% AUDIENCE: And what we are calling the epoch 00:07:18.930 --> 00:07:20.730 align:middle line:84% is ultimately the equation of loss function 00:07:20.730 --> 00:07:21.990 align:middle line:90% that we are trying to get. 00:07:21.990 --> 00:07:23.630 align:middle line:84% RAMA RAMAKRISHNAN: No, an epoch-- see, 00:07:23.630 --> 00:07:28.050 align:middle line:84% the thing to remember is that here, this whole thing is called 00:07:28.050 --> 00:07:30.850 align:middle line:84% an epoch, because we have to do one full pass 00:07:30.850 --> 00:07:32.210 align:middle line:90% through the training data. 00:07:32.210 --> 00:07:37.250 align:middle line:84% But within that epoch, we update the weights many times. 00:07:37.250 --> 00:07:39.690 align:middle line:84% Basically, we update the weights as many times 00:07:39.690 --> 00:07:40.580 align:middle line:90% as we have batches. 00:07:40.580 --> 00:07:44.010 align:middle line:90% 00:07:44.010 --> 00:07:46.410 align:middle line:90% All right. 00:07:46.410 --> 00:07:48.630 align:middle line:84% So to go here-- let's say, for example-- 00:07:48.630 --> 00:07:50.880 align:middle line:84% basically, the idea is that you take the training set, 00:07:50.880 --> 00:07:54.917 align:middle line:84% you divide it by the batch size, and you choose the batch size. 00:07:54.917 --> 00:07:56.000 align:middle line:90% You choose the batch size. 00:07:56.000 --> 00:07:58.500 align:middle line:84% And we'll talk about, well, How do you choose that? later on. 00:07:58.500 --> 00:07:59.583 align:middle line:90% You choose the batch size. 00:07:59.583 --> 00:08:02.580 align:middle line:84% And once you choose the size, just divide it and round it up. 00:08:02.580 --> 00:08:04.960 align:middle line:84% So for example, as you will see in the Colab, 00:08:04.960 --> 00:08:08.180 align:middle line:84% a training set is going to be 194 patients. 00:08:08.180 --> 00:08:10.980 align:middle line:84% And then we're going to choose a batch size of 32. 00:08:10.980 --> 00:08:14.660 align:middle line:84% And we typically tend to choose batch sizes of 32, 64, 00:08:14.660 --> 00:08:16.420 align:middle line:84% and things like that because it actually 00:08:16.420 --> 00:08:18.960 align:middle line:84% aligns very well with the nature of the parallel hardware 00:08:18.960 --> 00:08:20.740 align:middle line:90% we're going to use. 00:08:20.740 --> 00:08:24.280 align:middle line:90% And so here, 32 and so on. 00:08:24.280 --> 00:08:26.140 align:middle line:90% So divide 194 by 32. 00:08:26.140 --> 00:08:27.440 align:middle line:90% You get 6-point-something. 00:08:27.440 --> 00:08:30.120 align:middle line:90% You round it up to 7. 00:08:30.120 --> 00:08:32.700 align:middle line:84% And so what that means is that the first six batches will 00:08:32.700 --> 00:08:34.700 align:middle line:90% have 32 samples each. 00:08:34.700 --> 00:08:37.799 align:middle line:84% And then the final batch has only two samples left. 00:08:37.799 --> 00:08:38.679 align:middle line:90% And that's OK. 00:08:38.679 --> 00:08:41.622 align:middle line:84% It can be a nice, little, small batch at the end. 00:08:41.622 --> 00:08:43.539 align:middle line:84% There's nothing that says that every batch has 00:08:43.539 --> 00:08:44.372 align:middle line:90% to be the same size. 00:08:44.372 --> 00:08:46.980 align:middle line:90% 00:08:46.980 --> 00:08:47.700 align:middle line:90% That's it. 00:08:47.700 --> 00:08:48.320 align:middle line:90% Epochs. 00:08:48.320 --> 00:08:48.970 align:middle line:90% Batches. 00:08:48.970 --> 00:08:53.110 align:middle line:90% [AUDIO OUT] 00:08:53.110 --> 00:08:54.390 align:middle line:90% AUDIENCE: And are you-- 00:08:54.390 --> 00:08:57.890 align:middle line:84% for each batch, you run through the whole network, 00:08:57.890 --> 00:08:58.630 align:middle line:90% all the layers? 00:08:58.630 --> 00:09:00.670 align:middle line:90% Or each layer is one batch? 00:09:00.670 --> 00:09:01.630 align:middle line:90% RAMA RAMAKRISHNAN: No. 00:09:01.630 --> 00:09:03.970 align:middle line:84% For a batch, you run it through the entire network. 00:09:03.970 --> 00:09:06.630 align:middle line:84% So the way to think about it is that you take a batch. 00:09:06.630 --> 00:09:10.910 align:middle line:84% Just momentarily, you assume that's all the data you have. 00:09:10.910 --> 00:09:13.510 align:middle line:84% Just run it through the network, because unless you run it 00:09:13.510 --> 00:09:15.130 align:middle line:84% through every layer of the network, 00:09:15.130 --> 00:09:17.230 align:middle line:90% you can't get a prediction. 00:09:17.230 --> 00:09:19.670 align:middle line:84% Unless you get a prediction, you can't calculate the loss. 00:09:19.670 --> 00:09:20.730 align:middle line:84% And unless you calculate the loss, 00:09:20.730 --> 00:09:22.105 align:middle line:84% you can't calculate the gradient. 00:09:22.105 --> 00:09:24.808 align:middle line:84% Unless you calculate gradient, you can't update the weights. 00:09:24.808 --> 00:09:25.850 align:middle line:90% AUDIENCE: The last thing. 00:09:25.850 --> 00:09:27.690 align:middle line:84% But if you're using all the data, 00:09:27.690 --> 00:09:29.910 align:middle line:84% just doing the gradient descent, then you just 00:09:29.910 --> 00:09:31.885 align:middle line:84% go through the network once, right? 00:09:31.885 --> 00:09:33.010 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:09:33.010 --> 00:09:34.630 align:middle line:84% So in gradient descent, one epoch 00:09:34.630 --> 00:09:37.710 align:middle line:84% is one pass and one weight update. 00:09:37.710 --> 00:09:40.055 align:middle line:84% And in stochastic gradient descent, 00:09:40.055 --> 00:09:41.430 align:middle line:84% the number of updates you make is 00:09:41.430 --> 00:09:43.150 align:middle line:84% equal to the number of batches you have, 00:09:43.150 --> 00:09:46.670 align:middle line:84% which ends up being the training set divided 00:09:46.670 --> 00:09:50.285 align:middle line:90% by the batch size rounded up. 00:09:50.285 --> 00:09:51.660 align:middle line:84% AUDIENCE: Sorry, just to confirm. 00:09:51.660 --> 00:09:53.440 align:middle line:84% So initially, when you introduced 00:09:53.440 --> 00:09:55.440 align:middle line:84% the concept of batches, the whole purpose 00:09:55.440 --> 00:09:57.160 align:middle line:84% was not to run through all the data 00:09:57.160 --> 00:10:00.140 align:middle line:84% and be able to do some prediction from a subset. 00:10:00.140 --> 00:10:02.640 align:middle line:84% So now, the advantage is that after batch 1, 00:10:02.640 --> 00:10:05.560 align:middle line:84% we are using more accurate coefficient to run 00:10:05.560 --> 00:10:06.980 align:middle line:90% through batch 2 and so on? 00:10:06.980 --> 00:10:08.560 align:middle line:84% That's really the advantage of it? 00:10:08.560 --> 00:10:08.920 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:10:08.920 --> 00:10:10.200 align:middle line:84% AUDIENCE: Or is there something else here? 00:10:10.200 --> 00:10:11.220 align:middle line:84% RAMA RAMAKRISHNAN: Perfectly said. 00:10:11.220 --> 00:10:12.620 align:middle line:90% That's exactly the advantage. 00:10:12.620 --> 00:10:14.360 align:middle line:84% So we take a small amount of data, 00:10:14.360 --> 00:10:17.060 align:middle line:84% and we say, hey, we know this is not all the data, 00:10:17.060 --> 00:10:18.617 align:middle line:84% it's just a small subset of the data. 00:10:18.617 --> 00:10:20.700 align:middle line:84% So therefore, it's not going to be super accurate. 00:10:20.700 --> 00:10:21.908 align:middle line:90% It's going to be approximate. 00:10:21.908 --> 00:10:23.280 align:middle line:90% But it's OK. 00:10:23.280 --> 00:10:26.800 align:middle line:84% So we still tend to move in the right direction. 00:10:26.800 --> 00:10:29.057 align:middle line:84% So instead of waiting for the whole thing to get done 00:10:29.057 --> 00:10:30.640 align:middle line:84% and then updating it, we're just going 00:10:30.640 --> 00:10:33.600 align:middle line:90% to update it as we go along. 00:10:33.600 --> 00:10:34.400 align:middle line:90% All right. 00:10:34.400 --> 00:10:35.740 align:middle line:90% Yes. 00:10:35.740 --> 00:10:37.060 align:middle line:90% AUDIENCE: Another question. 00:10:37.060 --> 00:10:40.800 align:middle line:84% Is it that doing this process, SGD, will 00:10:40.800 --> 00:10:42.680 align:middle line:84% render us a more better solution, 00:10:42.680 --> 00:10:44.940 align:middle line:84% or it requires less compute power? 00:10:44.940 --> 00:10:45.940 align:middle line:90% RAMA RAMAKRISHNAN: Both. 00:10:45.940 --> 00:10:46.420 align:middle line:90% AUDIENCE: Both. 00:10:46.420 --> 00:10:47.420 align:middle line:90% RAMA RAMAKRISHNAN: Both. 00:10:47.420 --> 00:10:49.780 align:middle line:84% And the reasons for both are in the previous lecture. 00:10:49.780 --> 00:10:50.347 align:middle line:90% Yeah. 00:10:50.347 --> 00:10:52.180 align:middle line:84% And I'm saying that instead of repeating it, 00:10:52.180 --> 00:10:54.280 align:middle line:84% just because I'm very pressed for time today. 00:10:54.280 --> 00:10:55.540 align:middle line:90% That's why. 00:10:55.540 --> 00:10:56.040 align:middle line:90% All right. 00:10:56.040 --> 00:10:56.620 align:middle line:90% Cool. 00:10:56.620 --> 00:10:58.740 align:middle line:90% So that's what we have. 00:10:58.740 --> 00:11:01.146 align:middle line:90% Are we good? 00:11:01.146 --> 00:11:01.646 align:middle line:90% OK. 00:11:01.646 --> 00:11:04.580 align:middle line:84% So now we come to the last step before we actually 00:11:04.580 --> 00:11:07.740 align:middle line:84% fire up the Colab, which is overfitting and regularization. 00:11:07.740 --> 00:11:11.260 align:middle line:84% So if you remember from your machine-learning background, 00:11:11.260 --> 00:11:13.390 align:middle line:84% when your model gets more and more complex-- 00:11:13.390 --> 00:11:17.992 align:middle line:90% 00:11:17.992 --> 00:11:19.940 align:middle line:84% use a simple model, then use a more complex 00:11:19.940 --> 00:11:21.540 align:middle line:90% model, and so on and so forth-- 00:11:21.540 --> 00:11:25.002 align:middle line:84% what happens to the error on the training data, typically? 00:11:25.002 --> 00:11:26.960 align:middle line:84% What happens to the error on the training data? 00:11:26.960 --> 00:11:28.480 align:middle line:84% So let's say you have a simple regression model. 00:11:28.480 --> 00:11:29.440 align:middle line:90% You get some error. 00:11:29.440 --> 00:11:30.500 align:middle line:84% And then you have a regression model 00:11:30.500 --> 00:11:32.560 align:middle line:84% in which you use all kinds of interaction terms, 00:11:32.560 --> 00:11:33.980 align:middle line:84% you use logarithms, and this and that, 00:11:33.980 --> 00:11:35.240 align:middle line:90% and make it super complicated. 00:11:35.240 --> 00:11:36.100 align:middle line:84% What do you think is going to happen 00:11:36.100 --> 00:11:37.640 align:middle line:84% to the error on the training data? 00:11:37.640 --> 00:11:38.377 align:middle line:90% AUDIENCE: Reduce. 00:11:38.377 --> 00:11:39.460 align:middle line:90% RAMA RAMAKRISHNAN: Reduce. 00:11:39.460 --> 00:11:42.810 align:middle line:84% Basically, it's going to go down as the model gets more complex. 00:11:42.810 --> 00:11:43.992 align:middle line:90% Correct? 00:11:43.992 --> 00:11:45.530 align:middle line:84% Now, of course, comes the punchline, 00:11:45.530 --> 00:11:47.030 align:middle line:84% which is, what do you think is going 00:11:47.030 --> 00:11:49.270 align:middle line:90% to happen to the training data? 00:11:49.270 --> 00:11:50.370 align:middle line:90% I showed you the answer. 00:11:50.370 --> 00:11:52.933 align:middle line:90% 00:11:52.933 --> 00:11:54.350 align:middle line:84% Basically, what's going to happen, 00:11:54.350 --> 00:11:56.010 align:middle line:84% typically, at least conceptually, 00:11:56.010 --> 00:11:57.270 align:middle line:84% is that it's going to get better and better. 00:11:57.270 --> 00:11:59.030 align:middle line:84% At some point, it's going to bottom-out, 00:11:59.030 --> 00:12:00.850 align:middle line:84% and it's going to start climbing again. 00:12:00.850 --> 00:12:03.790 align:middle line:84% And so we typically refer to this phenomenon 00:12:03.790 --> 00:12:05.430 align:middle line:84% here, when it starts to climb again, 00:12:05.430 --> 00:12:08.190 align:middle line:84% as overfitting because the model is essentially 00:12:08.190 --> 00:12:11.190 align:middle line:84% fitting to the idiosyncrasies of the training data 00:12:11.190 --> 00:12:13.790 align:middle line:84% as opposed to generalizing patterns. 00:12:13.790 --> 00:12:15.710 align:middle line:84% And then, in this thing, we call it 00:12:15.710 --> 00:12:17.360 align:middle line:84% underfitting because still there's 00:12:17.360 --> 00:12:18.610 align:middle line:90% a lot of potential to improve. 00:12:18.610 --> 00:12:22.390 align:middle line:84% And we really are hoping to find the sweet spot in the middle. 00:12:22.390 --> 00:12:25.470 align:middle line:84% That's the basic idea of overfitting, underfitting. 00:12:25.470 --> 00:12:28.910 align:middle line:84% And the way we-- and to relate this to neural networks, 00:12:28.910 --> 00:12:31.550 align:middle line:84% as you see, as you've learned so far, 00:12:31.550 --> 00:12:33.990 align:middle line:84% you have to learn smart representations of the input 00:12:33.990 --> 00:12:34.690 align:middle line:90% data. 00:12:34.690 --> 00:12:37.550 align:middle line:84% And to do that, I have argued that you need to have 00:12:37.550 --> 00:12:39.130 align:middle line:90% lots of layers in your network. 00:12:39.130 --> 00:12:41.550 align:middle line:84% The more layers you have, the better things get. 00:12:41.550 --> 00:12:45.500 align:middle line:84% GPT-3, for example, has 96 layers, if I recall. 00:12:45.500 --> 00:12:46.700 align:middle line:90% More layers the better. 00:12:46.700 --> 00:12:49.160 align:middle line:84% But more layers means more parameters. 00:12:49.160 --> 00:12:51.760 align:middle line:84% More parameters means more complexity to the model, 00:12:51.760 --> 00:12:55.240 align:middle line:84% and therefore more chance of overfitting. 00:12:55.240 --> 00:12:58.400 align:middle line:84% So it's really important in neural networks 00:12:58.400 --> 00:13:00.520 align:middle line:84% that we think about regularization. 00:13:00.520 --> 00:13:02.200 align:middle line:84% And regularization, you will recall 00:13:02.200 --> 00:13:04.020 align:middle line:84% from your machine-learning background, 00:13:04.020 --> 00:13:06.200 align:middle line:84% is the way we handle the risk of overfitting 00:13:06.200 --> 00:13:09.240 align:middle line:84% and try to find models that fit just right. 00:13:09.240 --> 00:13:12.240 align:middle line:84% And so several regularization methods 00:13:12.240 --> 00:13:13.960 align:middle line:84% have been developed over the years. 00:13:13.960 --> 00:13:16.100 align:middle line:84% And we are going to use only two of them. 00:13:16.100 --> 00:13:18.680 align:middle line:84% The first one is called early stopping. 00:13:18.680 --> 00:13:21.480 align:middle line:84% And this has been famously referred to 00:13:21.480 --> 00:13:23.960 align:middle line:84% by Geoff Hinton, who is one of the pioneers, 00:13:23.960 --> 00:13:25.340 align:middle line:84% or as he's more colorfully known, 00:13:25.340 --> 00:13:28.360 align:middle line:84% one of the godfathers of deep learning-- 00:13:28.360 --> 00:13:30.160 align:middle line:84% he also won the Turing a few years ago-- 00:13:30.160 --> 00:13:32.940 align:middle line:90% as a "beautiful free lunch." 00:13:32.940 --> 00:13:34.220 align:middle line:90% That's what he calls it. 00:13:34.220 --> 00:13:35.720 align:middle line:90% So the idea is very simple. 00:13:35.720 --> 00:13:37.337 align:middle line:90% We take a validation set. 00:13:37.337 --> 00:13:38.920 align:middle line:84% We take the training data, we split it 00:13:38.920 --> 00:13:40.740 align:middle line:84% into a training and a validation set, 00:13:40.740 --> 00:13:43.260 align:middle line:84% and then we just keep doing gradient descent. 00:13:43.260 --> 00:13:44.607 align:middle line:90% Bup, bup, bup, bup, bup, bup. 00:13:44.607 --> 00:13:47.190 align:middle line:84% The training will hopefully keep on getting better and better. 00:13:47.190 --> 00:13:49.130 align:middle line:90% Lower and lower error. 00:13:49.130 --> 00:13:50.730 align:middle line:84% And then we just keep track of what's 00:13:50.730 --> 00:13:52.428 align:middle line:90% going on in the validation set. 00:13:52.428 --> 00:13:53.970 align:middle line:84% And then, at some point, if it starts 00:13:53.970 --> 00:13:56.510 align:middle line:84% to flatten out and start to climb, 00:13:56.510 --> 00:13:59.833 align:middle line:84% we just say, OK, that's when we stop training. 00:13:59.833 --> 00:14:02.250 align:middle line:84% And what we're going to do in the Colab is actually run it 00:14:02.250 --> 00:14:03.950 align:middle line:84% through the whole thing, see where it flattens out, 00:14:03.950 --> 00:14:05.650 align:middle line:84% and then we say, OK, that's where we should stop. 00:14:05.650 --> 00:14:07.530 align:middle line:84% But of course, you don't want to go all the way to the end 00:14:07.530 --> 00:14:10.410 align:middle line:84% and then go back and say, well, I want to stop at the 10th 00:14:10.410 --> 00:14:11.370 align:middle line:90% epoch. 00:14:11.370 --> 00:14:13.290 align:middle line:84% And there are ways you can use Keras to be 00:14:13.290 --> 00:14:14.670 align:middle line:90% very efficient about this. 00:14:14.670 --> 00:14:17.030 align:middle line:84% But the fundamental idea is you take the training data, 00:14:17.030 --> 00:14:19.230 align:middle line:84% split it into training and validation, 00:14:19.230 --> 00:14:21.530 align:middle line:84% and just track what's going on in the validation set 00:14:21.530 --> 00:14:23.890 align:middle line:84% to see whether this kind of bottoming out happens. 00:14:23.890 --> 00:14:26.650 align:middle line:84% So this is called early stopping. 00:14:26.650 --> 00:14:28.850 align:middle line:84% And the other way we're going to do-- 00:14:28.850 --> 00:14:30.850 align:middle line:90% this is called early stopping. 00:14:30.850 --> 00:14:32.530 align:middle line:90% We're looking for this part. 00:14:32.530 --> 00:14:34.650 align:middle line:84% The other thing is called dropout. 00:14:34.650 --> 00:14:39.963 align:middle line:84% And I'm going to come back to dropout on Wednesday's lecture 00:14:39.963 --> 00:14:42.130 align:middle line:84% because that's the first time we're going to use it. 00:14:42.130 --> 00:14:43.750 align:middle line:84% And so I'll come back to dropout and tell you 00:14:43.750 --> 00:14:44.650 align:middle line:90% exactly how it works. 00:14:44.650 --> 00:14:46.410 align:middle line:84% It's a very, very clever strategy, 00:14:46.410 --> 00:14:47.770 align:middle line:90% but we will not use it today. 00:14:47.770 --> 00:14:48.990 align:middle line:90% We'll use it on Wednesday. 00:14:48.990 --> 00:14:52.090 align:middle line:90% So in summary, what do we do? 00:14:52.090 --> 00:14:53.450 align:middle line:90% We get the data ready. 00:14:53.450 --> 00:14:54.898 align:middle line:90% We design the network-- 00:14:54.898 --> 00:14:56.690 align:middle line:84% number of hidden layers, number of neurons, 00:14:56.690 --> 00:14:57.770 align:middle line:90% and so on and so forth. 00:14:57.770 --> 00:14:59.410 align:middle line:90% We pick the right output layer. 00:14:59.410 --> 00:15:01.590 align:middle line:90% We pick the right loss function. 00:15:01.590 --> 00:15:03.730 align:middle line:90% We choose an optimizer. 00:15:03.730 --> 00:15:05.310 align:middle line:84% As I mentioned earlier, SGD comes 00:15:05.310 --> 00:15:08.390 align:middle line:84% in lots of flavors, lots of variations on the theme. 00:15:08.390 --> 00:15:12.650 align:middle line:84% And empirically, much like for hidden-layer neurons 00:15:12.650 --> 00:15:15.950 align:middle line:84% we tend to use ReLU as the activation function, 00:15:15.950 --> 00:15:17.710 align:middle line:84% for optimization, we tend to use a flavor 00:15:17.710 --> 00:15:20.830 align:middle line:84% of SGD called Adam as the default 00:15:20.830 --> 00:15:22.050 align:middle line:90% because it's really good. 00:15:22.050 --> 00:15:24.270 align:middle line:84% So we'll use Adam, as you will see. 00:15:24.270 --> 00:15:27.810 align:middle line:84% We typically use either early stopping or dropout. 00:15:27.810 --> 00:15:30.830 align:middle line:84% And then we just fire it up and start training Keras 00:15:30.830 --> 00:15:31.830 align:middle line:90% in TensorFlow. 00:15:31.830 --> 00:15:32.330 align:middle line:90% All right. 00:15:32.330 --> 00:15:33.810 align:middle line:90% So that is the training loop. 00:15:33.810 --> 00:15:35.560 align:middle line:84% Now I'm going to switch gears and give you 00:15:35.560 --> 00:15:37.390 align:middle line:84% a quick intro to Teras and Kensor-- 00:15:37.390 --> 00:15:39.480 align:middle line:90% Teras and KensorFlow. 00:15:39.480 --> 00:15:42.020 align:middle line:84% Keras and Tensor-- no, TensorFlow and Keras. 00:15:42.020 --> 00:15:43.080 align:middle line:90% Thank you. 00:15:43.080 --> 00:15:44.955 align:middle line:84% And then we'll actually fire up the Colab. 00:15:44.955 --> 00:15:46.330 align:middle line:84% So first of all, what's a tensor? 00:15:46.330 --> 00:15:49.815 align:middle line:90% 00:15:49.815 --> 00:15:50.440 align:middle line:90% AUDIENCE: Yeah. 00:15:50.440 --> 00:15:53.800 align:middle line:84% Just a quick question on the previous thing. 00:15:53.800 --> 00:15:55.600 align:middle line:84% If you're looking at the validation 00:15:55.600 --> 00:15:58.360 align:middle line:84% sets to avoid overfitting, but aren't you 00:15:58.360 --> 00:16:01.840 align:middle line:84% actually overfitting because you're 00:16:01.840 --> 00:16:04.880 align:middle line:84% kind of using the validation set as a training set? 00:16:04.880 --> 00:16:05.530 align:middle line:90% Or not. 00:16:05.530 --> 00:16:06.780 align:middle line:90% RAMA RAMAKRISHNAN: No, no, no. 00:16:06.780 --> 00:16:10.200 align:middle line:84% The validation set is never used to calculate any gradients. 00:16:10.200 --> 00:16:14.360 align:middle line:84% It's only used to calculate accuracy and loss. 00:16:14.360 --> 00:16:14.860 align:middle line:90% Yeah. 00:16:14.860 --> 00:16:17.680 align:middle line:84% It's kept aside and only used for evaluation, not 00:16:17.680 --> 00:16:19.320 align:middle line:90% for training. 00:16:19.320 --> 00:16:22.360 align:middle line:90% That's what keeps you honest. 00:16:22.360 --> 00:16:23.160 align:middle line:90% All right. 00:16:23.160 --> 00:16:25.180 align:middle line:84% And this will become clear when we actually go into the Colab. 00:16:25.180 --> 00:16:25.972 align:middle line:90% So what's a tensor? 00:16:25.972 --> 00:16:28.600 align:middle line:90% 00:16:28.600 --> 00:16:30.560 align:middle line:90% All right. 00:16:30.560 --> 00:16:33.240 align:middle line:90% OK. 00:16:33.240 --> 00:16:35.013 align:middle line:84% AUDIENCE: Tensor is the input data, which 00:16:35.013 --> 00:16:36.180 align:middle line:90% you're giving to the system. 00:16:36.180 --> 00:16:39.190 align:middle line:84% It could be in various formats like, for instance, an image. 00:16:39.190 --> 00:16:41.870 align:middle line:84% It could be-- like we call it a 4D tensor. 00:16:41.870 --> 00:16:44.850 align:middle line:84% If it's time series data, it's 3D. 00:16:44.850 --> 00:16:47.310 align:middle line:84% And typically, if we just send numbers in, 00:16:47.310 --> 00:16:51.610 align:middle line:84% it becomes a vector, which would coincide. 00:16:51.610 --> 00:16:56.050 align:middle line:84% It gives the value of the variable 00:16:56.050 --> 00:16:58.710 align:middle line:84% as well as-- the values of the variables associated to it, 00:16:58.710 --> 00:17:06.102 align:middle line:84% as well as the information you want [INAUDIBLE]. 00:17:06.102 --> 00:17:08.310 align:middle line:84% RAMA RAMAKRISHNAN: You're kind of on the right track, 00:17:08.310 --> 00:17:09.410 align:middle line:90% but not entirely. 00:17:09.410 --> 00:17:11.490 align:middle line:84% It's actually a simpler concept than that. 00:17:11.490 --> 00:17:13.268 align:middle line:90% So-- 00:17:13.268 --> 00:17:14.810 align:middle line:84% AUDIENCE: A matrix, but generalizable 00:17:14.810 --> 00:17:16.165 align:middle line:90% with higher dimensions. 00:17:16.165 --> 00:17:17.290 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:17:17.290 --> 00:17:19.970 align:middle line:84% That's also actually correct, but incomplete. 00:17:19.970 --> 00:17:23.609 align:middle line:84% The reason is because it can be simpler than a matrix. 00:17:23.609 --> 00:17:24.982 align:middle line:90% It's not matrix or higher. 00:17:24.982 --> 00:17:26.190 align:middle line:90% It actually could be simpler. 00:17:26.190 --> 00:17:30.090 align:middle line:84% In fact, you take a number, it's actually a tensor. 00:17:30.090 --> 00:17:30.630 align:middle line:90% All right. 00:17:30.630 --> 00:17:33.130 align:middle line:84% The simplest case of a tensor is a number. 00:17:33.130 --> 00:17:36.340 align:middle line:84% The next simpler case is a vector, which is a list. 00:17:36.340 --> 00:17:40.780 align:middle line:90% The next higher case is a table. 00:17:40.780 --> 00:17:42.880 align:middle line:90% So these are all tensors. 00:17:42.880 --> 00:17:45.700 align:middle line:84% So tensors, basically, are a generalization 00:17:45.700 --> 00:17:49.940 align:middle line:84% of the notion of both a number, a vector, and a table 00:17:49.940 --> 00:17:53.060 align:middle line:90% to higher dimensions. 00:17:53.060 --> 00:17:58.620 align:middle line:84% So you can think of a tensor as having what are called-- 00:17:58.620 --> 00:18:01.700 align:middle line:84% every tensor has something called a rank. 00:18:01.700 --> 00:18:04.240 align:middle line:90% So a number is just a number. 00:18:04.240 --> 00:18:06.160 align:middle line:84% It doesn't have a dimensionality to it. 00:18:06.160 --> 00:18:08.140 align:middle line:90% So it has got rank 0. 00:18:08.140 --> 00:18:08.940 align:middle line:90% OK? 00:18:08.940 --> 00:18:11.840 align:middle line:84% While a vector, it's a list of numbers. 00:18:11.840 --> 00:18:16.060 align:middle line:84% You can write it down, top to bottom, and it's one dimension. 00:18:16.060 --> 00:18:18.920 align:middle line:84% So that dimension, that one dimension, is called a rank. 00:18:18.920 --> 00:18:20.540 align:middle line:90% So it's called rank 1. 00:18:20.540 --> 00:18:26.500 align:middle line:84% A table is 2D, two dimensional, so it's called rank 2. 00:18:26.500 --> 00:18:32.100 align:middle line:84% And you can have a rank 3, which is just a bunch of tables. 00:18:32.100 --> 00:18:34.440 align:middle line:84% A bunch of tables is a rank 3 tensor. 00:18:34.440 --> 00:18:37.160 align:middle line:90% We also think of it as a cube. 00:18:37.160 --> 00:18:41.600 align:middle line:84% So these things are very useful because, obviously, we 00:18:41.600 --> 00:18:44.240 align:middle line:90% are all familiar with vectors. 00:18:44.240 --> 00:18:46.080 align:middle line:84% As you will see very shortly later 00:18:46.080 --> 00:18:49.040 align:middle line:84% in this class, black and white, grayscale images, 00:18:49.040 --> 00:18:52.320 align:middle line:84% are usually represented using tables of numbers, like this. 00:18:52.320 --> 00:18:54.830 align:middle line:84% Color images are represented using three tables. 00:18:54.830 --> 00:18:57.360 align:middle line:90% 00:18:57.360 --> 00:19:00.360 align:middle line:84% Can you think of what might be representable 00:19:00.360 --> 00:19:05.720 align:middle line:84% as a tensor of rank 4, meaning, every element 00:19:05.720 --> 00:19:08.890 align:middle line:84% of a tensor of rank 4 is actually a color picture? 00:19:08.890 --> 00:19:11.880 align:middle line:90% 00:19:11.880 --> 00:19:12.620 align:middle line:90% Shout it out. 00:19:12.620 --> 00:19:13.440 align:middle line:90% AUDIENCE: Video. 00:19:13.440 --> 00:19:13.800 align:middle line:90% RAMA RAMAKRISHNAN: Video. 00:19:13.800 --> 00:19:14.480 align:middle line:90% Exactly. 00:19:14.480 --> 00:19:15.340 align:middle line:90% What is the video? 00:19:15.340 --> 00:19:19.100 align:middle line:84% A video is basically a stream of color images. 00:19:19.100 --> 00:19:20.560 align:middle line:90% A color video. 00:19:20.560 --> 00:19:25.240 align:middle line:84% So each element of that stream, the first dimension 00:19:25.240 --> 00:19:28.360 align:middle line:84% of the tensor is which frame it is, 00:19:28.360 --> 00:19:31.200 align:middle line:84% and then everything else is the actual frame. 00:19:31.200 --> 00:19:36.758 align:middle line:84% So the way I think about these tensors always is-- 00:19:36.758 --> 00:19:39.930 align:middle line:84% if you take a tensor, you can just think of it as a-- 00:19:39.930 --> 00:19:44.130 align:middle line:84% you can think of a tensor as being this array, which has 00:19:44.130 --> 00:19:46.690 align:middle line:90% all these axes or dimensions. 00:19:46.690 --> 00:19:49.210 align:middle line:84% This is the first one, this is the second one, 00:19:49.210 --> 00:19:51.210 align:middle line:84% this is the third one, and so on. 00:19:51.210 --> 00:19:54.650 align:middle line:90% This is a tensor of rank 4. 00:19:54.650 --> 00:19:57.450 align:middle line:90% 1, 2, 3, 4. 00:19:57.450 --> 00:20:00.350 align:middle line:90% And so if you have a vector-- 00:20:00.350 --> 00:20:02.730 align:middle line:84% so you can imagine, if it's just a vector, 00:20:02.730 --> 00:20:06.012 align:middle line:84% you can imagine the vector actually living like this. 00:20:06.012 --> 00:20:06.970 align:middle line:90% Just a list of numbers. 00:20:06.970 --> 00:20:10.210 align:middle line:90% 00:20:10.210 --> 00:20:11.290 align:middle line:90% But if it's just-- 00:20:11.290 --> 00:20:16.270 align:middle line:84% if it is just a 2D, a rank 2 tensor, 00:20:16.270 --> 00:20:18.010 align:middle line:90% which is just like that-- 00:20:18.010 --> 00:20:21.250 align:middle line:90% which is just like that. 00:20:21.250 --> 00:20:24.970 align:middle line:84% So this thing becomes like that, and that thing becomes 00:20:24.970 --> 00:20:26.090 align:middle line:90% like that. 00:20:26.090 --> 00:20:28.690 align:middle line:84% So for example, if this is a 7 comma 3, 00:20:28.690 --> 00:20:35.167 align:middle line:84% that means that there are 7 rows and 3 columns. 00:20:35.167 --> 00:20:36.000 align:middle line:90% So you get the idea. 00:20:36.000 --> 00:20:37.417 align:middle line:84% So the way you think about tensors 00:20:37.417 --> 00:20:39.460 align:middle line:84% is always as if there's open square bracket, 00:20:39.460 --> 00:20:41.460 align:middle line:84% a bunch of things, a closed square bracket. 00:20:41.460 --> 00:20:43.860 align:middle line:84% And that's really what a tensor object is. 00:20:43.860 --> 00:20:48.412 align:middle line:84% So what that means is that anytime you have a tensor-- 00:20:48.412 --> 00:20:50.620 align:middle line:84% anytime you have a tensor, however complicated it is, 00:20:50.620 --> 00:20:54.460 align:middle line:84% you can always create a more complicated tensor by-- if you 00:20:54.460 --> 00:20:56.120 align:middle line:84% want to take a list of those tensors-- 00:20:56.120 --> 00:20:59.300 align:middle line:84% let's say that you have a list of videos. 00:20:59.300 --> 00:21:03.580 align:middle line:84% Each video is a rank 4 tensor, which means a list of videos 00:21:03.580 --> 00:21:04.700 align:middle line:90% is what rank? 00:21:04.700 --> 00:21:05.495 align:middle line:90% AUDIENCE: 5. 00:21:05.495 --> 00:21:06.620 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:21:06.620 --> 00:21:15.220 align:middle line:84% So a tensor of rank, say, 10, is just a list of rank 9 tensors. 00:21:15.220 --> 00:21:17.620 align:middle line:84% So that is the most important thing 00:21:17.620 --> 00:21:19.620 align:middle line:84% you need to understand about tensors. 00:21:19.620 --> 00:21:21.980 align:middle line:84% So at any point in time, if I give you a tensor, 00:21:21.980 --> 00:21:24.800 align:middle line:84% you can just iterate through the first dimension of it, 00:21:24.800 --> 00:21:26.880 align:middle line:90% the first aspect of it. 00:21:26.880 --> 00:21:28.880 align:middle line:84% And as you go through each one of these values-- 00:21:28.880 --> 00:21:35.550 align:middle line:84% so for example, here, yeah, that can do it. 00:21:35.550 --> 00:21:42.710 align:middle line:84% So if you have this tensor here, and if you 00:21:42.710 --> 00:21:46.910 align:middle line:84% want to create a more complicated tensor, no problem. 00:21:46.910 --> 00:21:48.690 align:middle line:84% So you add another dimension here. 00:21:48.690 --> 00:21:52.038 align:middle line:90% 00:21:52.038 --> 00:21:54.590 align:middle line:84% And now it just becomes-- this dimension, let's say, 00:21:54.590 --> 00:21:55.360 align:middle line:90% has 9 values. 00:21:55.360 --> 00:21:58.110 align:middle line:90% 00:21:58.110 --> 00:21:59.630 align:middle line:90% 0, 1, all the way to 9. 00:21:59.630 --> 00:22:01.008 align:middle line:90% So you put 0 here. 00:22:01.008 --> 00:22:02.050 align:middle line:90% And then what do you get? 00:22:02.050 --> 00:22:04.310 align:middle line:84% This whole tensor is a rank 4 tensor. 00:22:04.310 --> 00:22:05.930 align:middle line:90% And you put a 1 here. 00:22:05.930 --> 00:22:07.210 align:middle line:90% It's another rank 4 tensor. 00:22:07.210 --> 00:22:08.030 align:middle line:90% You put a 2 here. 00:22:08.030 --> 00:22:09.510 align:middle line:90% Another rank 4 tensor. 00:22:09.510 --> 00:22:12.950 align:middle line:84% So every tensor, you take the first element. 00:22:12.950 --> 00:22:14.790 align:middle line:84% It's just a list, but it's a list 00:22:14.790 --> 00:22:17.910 align:middle line:90% of the next down-rank tensor. 00:22:17.910 --> 00:22:20.630 align:middle line:84% Now, this tensor concept is actually 00:22:20.630 --> 00:22:22.830 align:middle line:90% something Einstein came up with. 00:22:22.830 --> 00:22:27.990 align:middle line:84% And so it's simultaneously kind of easy to understand 00:22:27.990 --> 00:22:29.355 align:middle line:90% and also slippery. 00:22:29.355 --> 00:22:30.730 align:middle line:84% So I would actually encourage you 00:22:30.730 --> 00:22:32.897 align:middle line:84% to read the book, which has a really good discussion 00:22:32.897 --> 00:22:33.510 align:middle line:90% of tensors. 00:22:33.510 --> 00:22:36.650 align:middle line:84% And the more you practice with it, the easier it'll get. 00:22:36.650 --> 00:22:39.090 align:middle line:84% So if you feel you kind of understood but not quite, 00:22:39.090 --> 00:22:40.850 align:middle line:90% you're not alone. 00:22:40.850 --> 00:22:42.010 align:middle line:90% It happens to all of us. 00:22:42.010 --> 00:22:44.690 align:middle line:84% You have to pay the price and go through the crucible. 00:22:44.690 --> 00:22:46.070 align:middle line:90% OK? 00:22:46.070 --> 00:22:48.610 align:middle line:90% All right. 00:22:48.610 --> 00:22:52.240 align:middle line:84% So to come back to this, that's what we have. 00:22:52.240 --> 00:22:54.777 align:middle line:90% 00:22:54.777 --> 00:22:56.610 align:middle line:84% And we already talked about a rank 4 tensor. 00:22:56.610 --> 00:22:57.770 align:middle line:90% It's a video. 00:22:57.770 --> 00:23:00.470 align:middle line:84% So 2.2, the text has a lot more detail. 00:23:00.470 --> 00:23:02.690 align:middle line:90% You should definitely read it. 00:23:02.690 --> 00:23:06.930 align:middle line:84% So here, TensorFlow is a library. 00:23:06.930 --> 00:23:09.370 align:middle line:84% And as you can imagine, neural networks, tensors 00:23:09.370 --> 00:23:11.130 align:middle line:84% come in and go through the network 00:23:11.130 --> 00:23:12.410 align:middle line:90% and go out the other end. 00:23:12.410 --> 00:23:16.330 align:middle line:84% And since tensors capture everything-- numbers, lists, 00:23:16.330 --> 00:23:17.790 align:middle line:90% tables, and so on and so forth-- 00:23:17.790 --> 00:23:19.870 align:middle line:84% it's just tensors flowing from input to output. 00:23:19.870 --> 00:23:21.250 align:middle line:90% Hence, it's called TensorFlow. 00:23:21.250 --> 00:23:22.732 align:middle line:84% And it gives you a couple of things 00:23:22.732 --> 00:23:24.190 align:middle line:84% which are really, really important, 00:23:24.190 --> 00:23:25.190 align:middle line:90% which is why we use it. 00:23:25.190 --> 00:23:27.130 align:middle line:84% The first one is that it will automatically 00:23:27.130 --> 00:23:31.300 align:middle line:84% calculate gradients for you of arbitrarily complicated loss 00:23:31.300 --> 00:23:32.200 align:middle line:90% functions. 00:23:32.200 --> 00:23:34.700 align:middle line:84% You don't have to calculate the gradient because calculating 00:23:34.700 --> 00:23:36.740 align:middle line:90% the gradient is very painful. 00:23:36.740 --> 00:23:39.165 align:middle line:84% It will automatically calculate the gradients for you. 00:23:39.165 --> 00:23:40.040 align:middle line:90% That's the best part. 00:23:40.040 --> 00:23:41.080 align:middle line:84% You don't have to use the chain rule. 00:23:41.080 --> 00:23:42.400 align:middle line:90% You don't have to do anything. 00:23:42.400 --> 00:23:45.280 align:middle line:84% The second thing it will do, it gives you all these optimizers, 00:23:45.280 --> 00:23:47.245 align:middle line:84% including SGD and all its variations, 00:23:47.245 --> 00:23:49.620 align:middle line:84% so you don't have to worry about the optimization itself. 00:23:49.620 --> 00:23:51.660 align:middle line:84% You can just pick and choose what you want. 00:23:51.660 --> 00:23:54.347 align:middle line:84% Third, if you have a lot of servers, 00:23:54.347 --> 00:23:56.180 align:middle line:84% it will actually take the computational load 00:23:56.180 --> 00:23:57.972 align:middle line:84% and distribute it across all those servers. 00:23:57.972 --> 00:24:00.100 align:middle line:84% People here with a CS background know 00:24:00.100 --> 00:24:02.620 align:middle line:84% that parallelizing computation is actually 00:24:02.620 --> 00:24:04.583 align:middle line:90% a very difficult problem. 00:24:04.583 --> 00:24:07.000 align:middle line:84% There are things which are called embarrassingly parallel. 00:24:07.000 --> 00:24:08.460 align:middle line:90% Many things are not. 00:24:08.460 --> 00:24:10.480 align:middle line:84% And it's actually quite tricky to figure it out. 00:24:10.480 --> 00:24:11.813 align:middle line:90% You don't have to figure it out. 00:24:11.813 --> 00:24:13.560 align:middle line:90% TensorFlow will figure it out. 00:24:13.560 --> 00:24:15.460 align:middle line:84% And then finally, I talked about the fact 00:24:15.460 --> 00:24:18.060 align:middle line:84% that there are these things called GPUs, Graphics Processing 00:24:18.060 --> 00:24:20.420 align:middle line:84% Units, which are parallel hardware. 00:24:20.420 --> 00:24:24.460 align:middle line:84% And so, even if you have just one computer but it has GPUs, 00:24:24.460 --> 00:24:25.860 align:middle line:84% there's a particular way in which 00:24:25.860 --> 00:24:27.910 align:middle line:84% you have to take your computation 00:24:27.910 --> 00:24:29.750 align:middle line:84% and organize it to really exploit 00:24:29.750 --> 00:24:31.630 align:middle line:90% the fact that you have a GPU. 00:24:31.630 --> 00:24:35.307 align:middle line:84% And so TensorFlow will actually do it for you, out of the box, 00:24:35.307 --> 00:24:35.890 align:middle line:90% automatically. 00:24:35.890 --> 00:24:37.470 align:middle line:84% You don't have to worry about any of that stuff. 00:24:37.470 --> 00:24:39.650 align:middle line:84% So those are all the advantages of this thing. 00:24:39.650 --> 00:24:42.530 align:middle line:84% By the way TPU is called a Tensor Processing Unit. 00:24:42.530 --> 00:24:46.170 align:middle line:84% It's something that-- you can think of it as Google's GPU. 00:24:46.170 --> 00:24:48.630 align:middle line:84% They came up with their own variation on the theme. 00:24:48.630 --> 00:24:52.910 align:middle line:84% Now, Keras sits on top of TensorFlow. 00:24:52.910 --> 00:24:55.230 align:middle line:90% This is the hardware you have. 00:24:55.230 --> 00:24:57.350 align:middle line:84% TensorFlow sits on top of the hardware. 00:24:57.350 --> 00:25:00.830 align:middle line:84% Keras sits on top of TensorFlow, and it basically 00:25:00.830 --> 00:25:02.890 align:middle line:84% gives you a whole bunch of convenience features. 00:25:02.890 --> 00:25:05.870 align:middle line:84% So for example, it gives you the notion of a layer. 00:25:05.870 --> 00:25:09.890 align:middle line:84% We already saw, keras.dense is a dense layer. 00:25:09.890 --> 00:25:11.370 align:middle line:84% It gives you the notion of a layer, 00:25:11.370 --> 00:25:13.370 align:middle line:84% it gives you the notion of activation functions, 00:25:13.370 --> 00:25:14.330 align:middle line:90% and so on and so forth. 00:25:14.330 --> 00:25:17.190 align:middle line:84% It gives you easy ways to preprocess the data, easy ways 00:25:17.190 --> 00:25:19.470 align:middle line:84% to train the model, report on metrics, 00:25:19.470 --> 00:25:22.110 align:middle line:84% calculate validation loss, validation accuracy, training 00:25:22.110 --> 00:25:24.130 align:middle line:84% loss, all the metrics we care about. 00:25:24.130 --> 00:25:25.880 align:middle line:84% And then it also gives you a whole library 00:25:25.880 --> 00:25:28.240 align:middle line:84% of pre-trained models that you can just use and adapt 00:25:28.240 --> 00:25:30.000 align:middle line:90% for your particular problem. 00:25:30.000 --> 00:25:32.320 align:middle line:84% So it gives you a whole bunch of conveniences. 00:25:32.320 --> 00:25:33.780 align:middle line:84% And that's why it's very popular. 00:25:33.780 --> 00:25:35.440 align:middle line:84% And by the way, many of you might also 00:25:35.440 --> 00:25:38.160 align:middle line:84% be familiar with PyTorch, which is a fantastic framework 00:25:38.160 --> 00:25:39.420 align:middle line:90% as well for deep learning. 00:25:39.420 --> 00:25:42.520 align:middle line:84% And the reason we chose to go with TensorFlow for this course, 00:25:42.520 --> 00:25:44.600 align:middle line:84% rather than PyTorch, is because we 00:25:44.600 --> 00:25:48.320 align:middle line:84% wanted to make the course accessible to folks 00:25:48.320 --> 00:25:50.480 align:middle line:84% who don't have a ton of programming background 00:25:50.480 --> 00:25:53.840 align:middle line:84% before coming to the class, and PyTorch is a bit more demanding 00:25:53.840 --> 00:25:55.260 align:middle line:90% from a CS perspective. 00:25:55.260 --> 00:25:57.807 align:middle line:84% It requires more knowledge of object-oriented programming, 00:25:57.807 --> 00:26:00.140 align:middle line:84% which is why we decided to go with TensorFlow and Keras, 00:26:00.140 --> 00:26:03.980 align:middle line:84% because I think it's actually as powerful, in many ways, 00:26:03.980 --> 00:26:07.060 align:middle line:84% and it's a little easier to get going. 00:26:07.060 --> 00:26:08.680 align:middle line:90% So that's what we have here. 00:26:08.680 --> 00:26:10.120 align:middle line:84% And one other thing I will mention 00:26:10.120 --> 00:26:12.680 align:middle line:84% is that there are three ways in which you can use Keras. 00:26:12.680 --> 00:26:14.400 align:middle line:90% There are three kinds of APIs-- 00:26:14.400 --> 00:26:16.300 align:middle line:84% sequential, functional, subclassing. 00:26:16.300 --> 00:26:20.180 align:middle line:84% And we'll almost exclusively use the functional API. 00:26:20.180 --> 00:26:22.680 align:middle line:84% And in fact, the model we built for heart disease prediction 00:26:22.680 --> 00:26:24.130 align:middle line:90% uses the functional API. 00:26:24.130 --> 00:26:27.410 align:middle line:84% And so just read 7.2.2 of the textbook to understand in detail 00:26:27.410 --> 00:26:28.630 align:middle line:90% how the API works. 00:26:28.630 --> 00:26:30.650 align:middle line:84% I find in my own work, the functional API 00:26:30.650 --> 00:26:31.790 align:middle line:90% is basically all I need. 00:26:31.790 --> 00:26:34.290 align:middle line:84% I don't need to do anything more complicated than that. 00:26:34.290 --> 00:26:37.010 align:middle line:84% And as you will see as you work on the homeworks 00:26:37.010 --> 00:26:40.410 align:middle line:84% and on your project, it's sort of a beautifully designed 00:26:40.410 --> 00:26:43.150 align:middle line:84% LEGO-block environment for doing these things. 00:26:43.150 --> 00:26:47.462 align:middle line:84% And you can create very complicated models very easily. 00:26:47.462 --> 00:26:50.110 align:middle line:84% There is a whole bunch of stuff here on these websites, 00:26:50.110 --> 00:26:51.450 align:middle line:90% so check them out. 00:26:51.450 --> 00:26:54.870 align:middle line:90% Lots of Colabs are available. 00:26:54.870 --> 00:26:57.370 align:middle line:84% So now, if you go back to the neural model for heart disease 00:26:57.370 --> 00:27:00.290 align:middle line:84% prediction, this is what we came up with in the last class. 00:27:00.290 --> 00:27:03.910 align:middle line:84% We had an input layer, one dense layer with 16 neurons, 00:27:03.910 --> 00:27:06.270 align:middle line:84% ReLU neurons, an output layer with a sigmoid, 00:27:06.270 --> 00:27:07.930 align:middle line:84% and then, boom, that was a model. 00:27:07.930 --> 00:27:10.250 align:middle line:90% So let's train this model. 00:27:10.250 --> 00:27:13.690 align:middle line:84% And so the training checklist is that we have already done this. 00:27:13.690 --> 00:27:16.170 align:middle line:90% Hidden layer of 16 neurons. 00:27:16.170 --> 00:27:16.848 align:middle line:90% Sigmoid. 00:27:16.848 --> 00:27:18.890 align:middle line:84% We need to use an appropriate loss function based 00:27:18.890 --> 00:27:19.973 align:middle line:90% on the type of the output. 00:27:19.973 --> 00:27:22.950 align:middle line:84% What loss function should we use? 00:27:22.950 --> 00:27:26.510 align:middle line:90% What is the output here? 00:27:26.510 --> 00:27:28.670 align:middle line:84% It's a binary classification problem. 00:27:28.670 --> 00:27:30.450 align:middle line:84% So what should the loss function be? 00:27:30.450 --> 00:27:33.310 align:middle line:90% 00:27:33.310 --> 00:27:34.990 align:middle line:90% I kind of heard it somewhere. 00:27:34.990 --> 00:27:35.930 align:middle line:90% You can shout it out. 00:27:35.930 --> 00:27:37.550 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:27:37.550 --> 00:27:39.590 align:middle line:84% RAMA RAMAKRISHNAN: The output is a sigmoid. 00:27:39.590 --> 00:27:40.730 align:middle line:90% The loss function. 00:27:40.730 --> 00:27:42.750 align:middle line:90% AUDIENCE: Binary cross-entropy. 00:27:42.750 --> 00:27:45.250 align:middle line:84% RAMA RAMAKRISHNAN: Binary cross-entropy. 00:27:45.250 --> 00:27:48.290 align:middle line:84% Remember, if you're predicting a number, an arbitrary number, 00:27:48.290 --> 00:27:50.522 align:middle line:84% you can use something like mean squared error. 00:27:50.522 --> 00:27:52.230 align:middle line:84% If you're predicting a probability, which 00:27:52.230 --> 00:27:55.030 align:middle line:84% has to be compared to a 0, 1 output, which 00:27:55.030 --> 00:27:57.090 align:middle line:84% is what binary classification is all about, 00:27:57.090 --> 00:27:58.990 align:middle line:90% we use binary cross-entropy. 00:27:58.990 --> 00:28:00.890 align:middle line:90% So that's what we do here. 00:28:00.890 --> 00:28:03.750 align:middle line:90% So we do binary cross-entropy. 00:28:03.750 --> 00:28:06.170 align:middle line:90% And then we will go with Adam. 00:28:06.170 --> 00:28:07.910 align:middle line:84% And then we will use early stopping 00:28:07.910 --> 00:28:10.670 align:middle line:90% to make sure we don't overfit. 00:28:10.670 --> 00:28:13.230 align:middle line:84% I know this-- OK, I promise, this is literally the last slide 00:28:13.230 --> 00:28:15.190 align:middle line:90% before I go to the Colab. 00:28:15.190 --> 00:28:17.350 align:middle line:84% I feel like one of those used car salesmen. 00:28:17.350 --> 00:28:19.470 align:middle line:90% Wait, there is more! 00:28:19.470 --> 00:28:20.920 align:middle line:90% So anyway. 00:28:20.920 --> 00:28:24.640 align:middle line:84% So don't worry if you don't understand every detail of what 00:28:24.640 --> 00:28:25.760 align:middle line:90% I'm going to go through. 00:28:25.760 --> 00:28:28.672 align:middle line:84% I'm going to link to the Colab as soon as the class is over. 00:28:28.672 --> 00:28:30.380 align:middle line:84% But once you get your hands on the Colab, 00:28:30.380 --> 00:28:32.920 align:middle line:84% make sure you actually go through every line in Colab. 00:28:32.920 --> 00:28:35.337 align:middle line:84% What I typically do when I'm trying to learn something new 00:28:35.337 --> 00:28:38.680 align:middle line:90% is I'll actually cut and paste. 00:28:38.680 --> 00:28:39.483 align:middle line:90% I won't do that. 00:28:39.483 --> 00:28:41.900 align:middle line:84% I won't actually cut and paste the code and run it myself. 00:28:41.900 --> 00:28:43.805 align:middle line:90% I will retype the code. 00:28:43.805 --> 00:28:46.180 align:middle line:84% If you retype the code as opposed to cutting and pasting, 00:28:46.180 --> 00:28:48.120 align:middle line:84% trust me, you will learn a lot more. 00:28:48.120 --> 00:28:52.320 align:middle line:84% So I strongly encourage you to do it that way. 00:28:52.320 --> 00:28:55.392 align:middle line:84% And so all the Colabs you're going to publish in the class, 00:28:55.392 --> 00:28:57.600 align:middle line:84% the first thing you should do is you should just make 00:28:57.600 --> 00:29:00.120 align:middle line:90% your own copy of the notebook. 00:29:00.120 --> 00:29:01.740 align:middle line:90% Copy to Drive. 00:29:01.740 --> 00:29:03.880 align:middle line:84% And then, if you're using anything other 00:29:03.880 --> 00:29:07.160 align:middle line:84% than today's Colab, anything involving natural language 00:29:07.160 --> 00:29:09.520 align:middle line:84% processing or vision, you probably should use a GPU. 00:29:09.520 --> 00:29:11.040 align:middle line:90% So just go into-- 00:29:11.040 --> 00:29:14.840 align:middle line:84% you go in here, choose the runtime to be GPU, 00:29:14.840 --> 00:29:17.020 align:middle line:84% and then you start your notebook, and you're done. 00:29:17.020 --> 00:29:18.728 align:middle line:84% And the second time onwards, you can just 00:29:18.728 --> 00:29:20.032 align:middle line:90% go directly to this step. 00:29:20.032 --> 00:29:21.490 align:middle line:84% You don't have to do all this stuff 00:29:21.490 --> 00:29:23.322 align:middle line:90% for that particular notebook. 00:29:23.322 --> 00:29:25.530 align:middle line:84% There are numerous tutorials, like five-minute videos 00:29:25.530 --> 00:29:27.128 align:middle line:90% and so on, on how to use Colab. 00:29:27.128 --> 00:29:27.670 align:middle line:90% Just do that. 00:29:27.670 --> 00:29:30.210 align:middle line:84% I'm not going to spend time on it here. 00:29:30.210 --> 00:29:31.490 align:middle line:90% All right. 00:29:31.490 --> 00:29:31.990 align:middle line:90% OK. 00:29:31.990 --> 00:29:35.010 align:middle line:84% So I just ran it a few hours ago. 00:29:35.010 --> 00:29:36.510 align:middle line:84% I'm not going to run every cell now 00:29:36.510 --> 00:29:37.510 align:middle line:84% because this is going to take some time. 00:29:37.510 --> 00:29:39.468 align:middle line:84% It's going to get in the way of the class time. 00:29:39.468 --> 00:29:41.970 align:middle line:84% But I'm going to just go through it slowly and explain 00:29:41.970 --> 00:29:42.970 align:middle line:90% what's going on. 00:29:42.970 --> 00:29:45.750 align:middle line:84% So here, this is just an introduction to the data set. 00:29:45.750 --> 00:29:48.730 align:middle line:84% We already saw this introduction last week. 00:29:48.730 --> 00:29:51.370 align:middle line:90% We have whatever-- 303 patients. 00:29:51.370 --> 00:29:52.430 align:middle line:90% Heart patients. 00:29:52.430 --> 00:29:55.850 align:middle line:84% We have a whole bunch of variables here-- 00:29:55.850 --> 00:29:59.870 align:middle line:84% age, demographics, and a whole bunch of biomarker information. 00:29:59.870 --> 00:30:02.850 align:middle line:90% And this is our target variable. 00:30:02.850 --> 00:30:03.770 align:middle line:90% 0 or 1. 00:30:03.770 --> 00:30:05.530 align:middle line:90% Heart disease, yes or no. 00:30:05.530 --> 00:30:09.770 align:middle line:84% And so, by the way, just some technical preliminaries here, 00:30:09.770 --> 00:30:11.637 align:middle line:84% basically every time we load these things, 00:30:11.637 --> 00:30:13.470 align:middle line:84% we're actually going to load these packages. 00:30:13.470 --> 00:30:15.512 align:middle line:84% So you can see here, these are the two key things 00:30:15.512 --> 00:30:16.110 align:middle line:90% we need to do. 00:30:16.110 --> 00:30:18.140 align:middle line:90% We import TensorFlow first. 00:30:18.140 --> 00:30:20.280 align:middle line:84% And then, from within TensorFlow, we import Keras. 00:30:20.280 --> 00:30:24.085 align:middle line:84% That's what these two lines do here. 00:30:24.085 --> 00:30:26.460 align:middle line:84% And folks who have done data science and machine learning 00:30:26.460 --> 00:30:28.553 align:middle line:90% a bit before, you'll know this. 00:30:28.553 --> 00:30:31.380 align:middle line:84% We will actually load the three packages 00:30:31.380 --> 00:30:33.780 align:middle line:84% that were just most commonly used right, which is 00:30:33.780 --> 00:30:36.660 align:middle line:90% NumPy, Pandas, and Matplotlib. 00:30:36.660 --> 00:30:40.220 align:middle line:84% NumPy, because it's very easy for manipulating matrices 00:30:40.220 --> 00:30:42.020 align:middle line:90% and arrays and tensors. 00:30:42.020 --> 00:30:45.800 align:middle line:84% Pandas because, oftentimes, you get some data from somewhere, 00:30:45.800 --> 00:30:48.130 align:middle line:84% you need to massage it and wrangle it to a point 00:30:48.130 --> 00:30:49.880 align:middle line:84% where you can actually feed it into Keras. 00:30:49.880 --> 00:30:51.120 align:middle line:90% So you need pandas for that. 00:30:51.120 --> 00:30:55.173 align:middle line:84% And Matplotlib because you just want to plot these loss curves 00:30:55.173 --> 00:30:57.340 align:middle line:84% and accuracy curves to see whether early stopping is 00:30:57.340 --> 00:30:57.960 align:middle line:90% needed. 00:30:57.960 --> 00:31:00.100 align:middle line:90% So that's why we use it. 00:31:00.100 --> 00:31:01.350 align:middle line:90% So we import all these things. 00:31:01.350 --> 00:31:03.600 align:middle line:84% And then, I guess the other thing you have to remember 00:31:03.600 --> 00:31:06.140 align:middle line:84% is that when we are training these deep-learning models, 00:31:06.140 --> 00:31:08.580 align:middle line:84% there is randomness in the process, which enters 00:31:08.580 --> 00:31:10.140 align:middle line:90% in a few different places. 00:31:10.140 --> 00:31:13.340 align:middle line:84% So clearly, the starting values for these weights 00:31:13.340 --> 00:31:14.180 align:middle line:90% are going to be-- 00:31:14.180 --> 00:31:16.600 align:middle line:84% the weights are going to be randomly initialized. 00:31:16.600 --> 00:31:20.560 align:middle line:84% And therefore, that's obviously a source of randomness. 00:31:20.560 --> 00:31:22.183 align:middle line:84% Now, we talked about how you take-- 00:31:22.183 --> 00:31:24.100 align:middle line:84% when you're doing stochastic gradient descent, 00:31:24.100 --> 00:31:27.440 align:middle line:84% you take all the data, and then you randomly choose batches 00:31:27.440 --> 00:31:30.500 align:middle line:84% right from this data till we finish a whole pass through it. 00:31:30.500 --> 00:31:33.100 align:middle line:84% Well, that immediately raised the question, well, 00:31:33.100 --> 00:31:34.780 align:middle line:84% what do you mean by randomly choose? 00:31:34.780 --> 00:31:37.080 align:middle line:84% So typically, what we do in practice is that-- 00:31:37.080 --> 00:31:39.200 align:middle line:84% and Keras will take care of all this for you-- 00:31:39.200 --> 00:31:42.340 align:middle line:84% you basically take the data and just shuffle it once, randomly. 00:31:42.340 --> 00:31:45.020 align:middle line:84% And then you just go first 32, next 32, next 32, next 32, 00:31:45.020 --> 00:31:46.090 align:middle line:90% like that. 00:31:46.090 --> 00:31:47.960 align:middle line:84% But it is a source of randomness. 00:31:47.960 --> 00:31:51.340 align:middle line:84% And then when we split the data into train, validation, testing, 00:31:51.340 --> 00:31:53.120 align:middle line:84% and so on, particularly if you want 00:31:53.120 --> 00:31:55.480 align:middle line:84% to look for early stopping and overfitting, 00:31:55.480 --> 00:31:57.540 align:middle line:84% we need to again split the data randomly. 00:31:57.540 --> 00:31:59.300 align:middle line:84% And that's another source of randomness. 00:31:59.300 --> 00:32:01.240 align:middle line:84% And then when we do dropout, which 00:32:01.240 --> 00:32:03.260 align:middle line:84% we will talk about on Wednesday-- again, 00:32:03.260 --> 00:32:06.260 align:middle line:84% dropout has a little bit of a random element to it, 00:32:06.260 --> 00:32:09.100 align:middle line:84% and so that's another source of randomness. 00:32:09.100 --> 00:32:12.140 align:middle line:84% So all this means is that if you're working with these models 00:32:12.140 --> 00:32:14.640 align:middle line:84% and if you want to build a model and you want to hand it off 00:32:14.640 --> 00:32:17.450 align:middle line:84% to someone so that they can reproduce your results, 00:32:17.450 --> 00:32:20.890 align:middle line:84% well, you better make sure that you make it easy for them 00:32:20.890 --> 00:32:22.110 align:middle line:90% to replicate what you have. 00:32:22.110 --> 00:32:24.970 align:middle line:84% And the way you do it is by sending a random seed for all 00:32:24.970 --> 00:32:25.830 align:middle line:90% these things. 00:32:25.830 --> 00:32:27.650 align:middle line:84% And the way you do it is by having 00:32:27.650 --> 00:32:30.730 align:middle line:84% this little handy function here, set_random_seed. 00:32:30.730 --> 00:32:34.810 align:middle line:84% And of course, I use 42, just like everybody should. 00:32:34.810 --> 00:32:36.890 align:middle line:90% So that's that. 00:32:36.890 --> 00:32:39.090 align:middle line:84% By the way, that's just a pop culture reference 00:32:39.090 --> 00:32:40.840 align:middle line:84% to this book called The Hitchhiker's Guide 00:32:40.840 --> 00:32:42.057 align:middle line:90% to the Galaxy. 00:32:42.057 --> 00:32:43.890 align:middle line:84% So when I say something about the number 42, 00:32:43.890 --> 00:32:45.370 align:middle line:90% and you'll know what I mean. 00:32:45.370 --> 00:32:49.310 align:middle line:84% So by the way, the question inevitably comes at this point, 00:32:49.310 --> 00:32:51.250 align:middle line:84% OK, if we do exactly this, will we actually 00:32:51.250 --> 00:32:52.708 align:middle line:84% get the exact same numbers that you 00:32:52.708 --> 00:32:55.150 align:middle line:84% have in your version of the notebook? 00:32:55.150 --> 00:32:57.510 align:middle line:84% And the answer is, hopefully, most of the time, 00:32:57.510 --> 00:32:58.790 align:middle line:90% but it's not guaranteed. 00:32:58.790 --> 00:33:01.070 align:middle line:84% So this is called bitwise reproducibility. 00:33:01.070 --> 00:33:03.770 align:middle line:84% It's not guaranteed due to certain hardware things 00:33:03.770 --> 00:33:05.950 align:middle line:84% and device drivers and stuff like that, 00:33:05.950 --> 00:33:08.090 align:middle line:84% so we won't get into all that stuff, which 00:33:08.090 --> 00:33:10.330 align:middle line:84% is why, as you see here, I have a bit 00:33:10.330 --> 00:33:13.050 align:middle line:90% of a fingers-crossed thing. 00:33:13.050 --> 00:33:13.550 align:middle line:90% All right. 00:33:13.550 --> 00:33:14.060 align:middle line:90% Cool. 00:33:14.060 --> 00:33:15.740 align:middle line:90% So that's what we have. 00:33:15.740 --> 00:33:18.757 align:middle line:84% So as it turns out, Francois Chollet, who wrote the book, 00:33:18.757 --> 00:33:21.340 align:middle line:84% the textbook, he actually made this data available in a Pandas 00:33:21.340 --> 00:33:22.180 align:middle line:90% dataframe. 00:33:22.180 --> 00:33:26.020 align:middle line:84% So we read the CSV file into this dataframe right there. 00:33:26.020 --> 00:33:30.420 align:middle line:84% And then it's 303 rows, 14 columns. 00:33:30.420 --> 00:33:33.860 align:middle line:84% And you can see here, we'll take a look at the first few rows. 00:33:33.860 --> 00:33:37.020 align:middle line:84% And these are all the rows-- age, gender, cholesterol, blah, 00:33:37.020 --> 00:33:38.140 align:middle line:90% blah, blah, blah, blah. 00:33:38.140 --> 00:33:40.980 align:middle line:84% And then this is the target variable right there. 00:33:40.980 --> 00:33:42.607 align:middle line:84% And one of the first things I always 00:33:42.607 --> 00:33:44.940 align:middle line:84% do when I'm working with a binary classification problem 00:33:44.940 --> 00:33:47.580 align:middle line:84% is to quickly check whether the positive and negative classes 00:33:47.580 --> 00:33:48.983 align:middle line:90% are balanced or not. 00:33:48.983 --> 00:33:50.900 align:middle line:84% And so what you can do is you can just quickly 00:33:50.900 --> 00:33:53.880 align:middle line:84% check to see what percent of the data points is 0 versus 1. 00:33:53.880 --> 00:33:58.318 align:middle line:84% And you can see here, 72.6% of the patients 00:33:58.318 --> 00:33:59.360 align:middle line:90% don't have heart disease. 00:33:59.360 --> 00:34:01.100 align:middle line:90% That's a good thing, of course. 00:34:01.100 --> 00:34:03.760 align:middle line:84% And then 27.4 have heart disease. 00:34:03.760 --> 00:34:06.040 align:middle line:84% So it's not 50/50 or roughly 50/50. 00:34:06.040 --> 00:34:07.260 align:middle line:90% It's a little thin. 00:34:07.260 --> 00:34:09.520 align:middle line:90% So by the way, quick question. 00:34:09.520 --> 00:34:12.915 align:middle line:84% What is a good baseline model for this problem? 00:34:12.915 --> 00:34:15.290 align:middle line:84% Suppose you couldn't use anything, any complicated thing. 00:34:15.290 --> 00:34:18.126 align:middle line:90% What's a good baseline model? 00:34:18.126 --> 00:34:21.886 align:middle line:90% Let's go to this other side. 00:34:21.886 --> 00:34:22.469 align:middle line:90% AUDIENCE: Yes. 00:34:22.469 --> 00:34:23.922 align:middle line:90% Just predict zero every time. 00:34:23.922 --> 00:34:25.630 align:middle line:84% RAMA RAMAKRISHNAN: Why would you do that? 00:34:25.630 --> 00:34:28.185 align:middle line:84% AUDIENCE: It would give you a 72.6% accuracy. 00:34:28.185 --> 00:34:29.310 align:middle line:90% RAMA RAMAKRISHNAN: Exactly. 00:34:29.310 --> 00:34:32.810 align:middle line:84% Because 72.6% is the higher class, 00:34:32.810 --> 00:34:35.370 align:middle line:84% higher class of the higher percentage, you just predict it. 00:34:35.370 --> 00:34:37.373 align:middle line:84% You'll be right on those 72.6% of the cases, 00:34:37.373 --> 00:34:38.790 align:middle line:84% you'll be wrong on the rest, which 00:34:38.790 --> 00:34:43.750 align:middle line:84% means that your accuracy of this model is going to be 72.6%. 00:34:43.750 --> 00:34:48.330 align:middle line:84% And so any fancy model we build, it's got to do better than this. 00:34:48.330 --> 00:34:51.250 align:middle line:84% Otherwise, it's not worth its weight in layers. 00:34:51.250 --> 00:34:51.750 align:middle line:90% All right. 00:34:51.750 --> 00:34:53.389 align:middle line:84% So we'll come back to this later. 00:34:53.389 --> 00:34:55.931 align:middle line:84% So the first thing we want to do is we want to preprocess it, 00:34:55.931 --> 00:34:58.190 align:middle line:84% because this data set has both categorical variables 00:34:58.190 --> 00:35:00.030 align:middle line:90% and numeric variables. 00:35:00.030 --> 00:35:02.750 align:middle line:84% And so it's usually convenient just to group them 00:35:02.750 --> 00:35:04.250 align:middle line:90% into two different groups. 00:35:04.250 --> 00:35:06.430 align:middle line:84% So I have listed all the categorical variables here 00:35:06.430 --> 00:35:08.190 align:middle line:90% and the numerics here. 00:35:08.190 --> 00:35:10.500 align:middle line:84% And then we have the preprocessing here. 00:35:10.500 --> 00:35:12.250 align:middle line:84% We have to take the categorical variables, 00:35:12.250 --> 00:35:14.410 align:middle line:84% and we have to one-hot encode them. 00:35:14.410 --> 00:35:17.750 align:middle line:84% And the reason is that unlike, say, a decision tree model, 00:35:17.750 --> 00:35:22.170 align:middle line:84% a neural network cannot handle categorical inputs directly. 00:35:22.170 --> 00:35:24.190 align:middle line:84% It can only handle numeric inputs, 00:35:24.190 --> 00:35:26.050 align:middle line:84% which means that we have to numericalize 00:35:26.050 --> 00:35:27.910 align:middle line:84% every categorical thing that comes in. 00:35:27.910 --> 00:35:29.430 align:middle line:84% And there are many ways to do it, 00:35:29.430 --> 00:35:32.570 align:middle line:84% but the standard way to do it is one-hot encoding. 00:35:32.570 --> 00:35:35.230 align:middle line:84% And for the numeric variables, we need to normalize them. 00:35:35.230 --> 00:35:36.730 align:middle line:84% And I'll come to that in a second. 00:35:36.730 --> 00:35:40.008 align:middle line:84% So Pandas has this get_dummies function here. 00:35:40.008 --> 00:35:42.050 align:middle line:84% And you can just run this thing, and it will just 00:35:42.050 --> 00:35:43.550 align:middle line:90% one-hot encode the whole thing. 00:35:43.550 --> 00:35:45.510 align:middle line:84% So once you do that, this is what you have. 00:35:45.510 --> 00:35:50.650 align:middle line:84% So you can see here, previously, let's say, thal 00:35:50.650 --> 00:35:52.650 align:middle line:84% had three values-- fixed, normal, reversible, 00:35:52.650 --> 00:35:53.510 align:middle line:90% or something. 00:35:53.510 --> 00:35:56.490 align:middle line:84% And then you go to the one-hot encoded version. 00:35:56.490 --> 00:35:59.770 align:middle line:84% And now we can see here, thal_fixed, thal_normal, 00:35:59.770 --> 00:36:00.590 align:middle line:90% thal_reversible. 00:36:00.590 --> 00:36:02.130 align:middle line:90% That's three columns. 00:36:02.130 --> 00:36:04.610 align:middle line:84% That's the one-hot encoding in action. 00:36:04.610 --> 00:36:08.690 align:middle line:84% Now, the other thing to remember is that neural networks 00:36:08.690 --> 00:36:10.700 align:middle line:84% work best when the numeric inputs 00:36:10.700 --> 00:36:13.560 align:middle line:84% you send them are all in a relatively small range. 00:36:13.560 --> 00:36:16.300 align:middle line:84% They shouldn't have a wide range of variation. 00:36:16.300 --> 00:36:19.160 align:middle line:84% And so the standard practice is to standardize 00:36:19.160 --> 00:36:20.160 align:middle line:90% the numerical variables. 00:36:20.160 --> 00:36:22.660 align:middle line:84% By standardize, I mean typically subtract the mean, divide 00:36:22.660 --> 00:36:24.580 align:middle line:90% by the standard deviation. 00:36:24.580 --> 00:36:25.800 align:middle line:90% We should do that. 00:36:25.800 --> 00:36:28.140 align:middle line:84% But before we do so, we should split the data 00:36:28.140 --> 00:36:30.740 align:middle line:84% into a training set and a test set. 00:36:30.740 --> 00:36:32.800 align:middle line:84% And why do we want to split into a test set? 00:36:32.800 --> 00:36:34.420 align:middle line:84% Because at the very end, once we have built the model 00:36:34.420 --> 00:36:36.337 align:middle line:84% and done all the things we want to do with it, 00:36:36.337 --> 00:36:38.380 align:middle line:84% we finally want to take out the test set 00:36:38.380 --> 00:36:43.140 align:middle line:84% and evaluate it once so that we get this true measure of how 00:36:43.140 --> 00:36:45.980 align:middle line:84% it's going to perform in the wild after you deploy it. 00:36:45.980 --> 00:36:50.620 align:middle line:84% So you want to divide it, let's say, 80% training and 20% test 00:36:50.620 --> 00:36:51.200 align:middle line:90% set. 00:36:51.200 --> 00:36:53.860 align:middle line:84% So the question is, why should we do the splitting now 00:36:53.860 --> 00:36:55.900 align:middle line:90% before we do the normalization? 00:36:55.900 --> 00:36:57.860 align:middle line:84% Why can't we just do the normalization 00:36:57.860 --> 00:36:59.040 align:middle line:90% and then do splitting? 00:36:59.040 --> 00:37:04.020 align:middle line:90% 00:37:04.020 --> 00:37:06.340 align:middle line:90% All right. 00:37:06.340 --> 00:37:09.430 align:middle line:84% AUDIENCE: Because then your validation set 00:37:09.430 --> 00:37:11.470 align:middle line:84% is also somewhat dependent on your test 00:37:11.470 --> 00:37:13.770 align:middle line:84% set results, as well as the mean of the test set. 00:37:13.770 --> 00:37:14.895 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:37:14.895 --> 00:37:17.790 align:middle line:84% Because the test set has now essentially 00:37:17.790 --> 00:37:21.710 align:middle line:84% been influenced by the training set. 00:37:21.710 --> 00:37:24.090 align:middle line:84% The modeling process-- part of the modeling process 00:37:24.090 --> 00:37:24.830 align:middle line:90% is the splitting. 00:37:24.830 --> 00:37:26.390 align:middle line:90% And the splitting also-- 00:37:26.390 --> 00:37:30.510 align:middle line:84% the standardization-- if the standardization, which 00:37:30.510 --> 00:37:33.730 align:middle line:84% is part of the process, uses information about the test set, 00:37:33.730 --> 00:37:37.510 align:middle line:84% well, the test set is not really kept away from anything, is it? 00:37:37.510 --> 00:37:39.790 align:middle line:84% That's why we want to split it, lock away 00:37:39.790 --> 00:37:42.250 align:middle line:84% the test set somewhere, and then proceed with the modeling. 00:37:42.250 --> 00:37:43.750 align:middle line:84% Again, this is like machine learning 00:37:43.750 --> 00:37:46.950 align:middle line:84% 101, which is why I'm going through it pretty fast. 00:37:46.950 --> 00:37:52.550 align:middle line:84% So we do this sampling function, take 20% of the data, 00:37:52.550 --> 00:37:53.890 align:middle line:90% and make it the test set. 00:37:53.890 --> 00:37:56.090 align:middle line:84% And the remaining is going to be the training set. 00:37:56.090 --> 00:37:58.750 align:middle line:84% And when we do that, you can see the training set 00:37:58.750 --> 00:38:05.110 align:middle line:84% is now 242 rows, while the test is 61 rows. 00:38:05.110 --> 00:38:08.640 align:middle line:84% And any of these data frames, you'll know the shape attribute 00:38:08.640 --> 00:38:11.540 align:middle line:84% gives you the dimensions of the number of rows and the columns. 00:38:11.540 --> 00:38:12.780 align:middle line:90% That's what we're doing here. 00:38:12.780 --> 00:38:14.580 align:middle line:84% And now that we have done that, we have done the split, 00:38:14.580 --> 00:38:17.220 align:middle line:84% we can calculate the mean and the standard deviation. 00:38:17.220 --> 00:38:18.440 align:middle line:90% So I calculate the mean here. 00:38:18.440 --> 00:38:20.060 align:middle line:84% I calculate the standard deviation. 00:38:20.060 --> 00:38:21.220 align:middle line:90% These are all the means. 00:38:21.220 --> 00:38:25.420 align:middle line:84% And once I do that, I just do each column minus the mean, 00:38:25.420 --> 00:38:26.900 align:middle line:90% divide the standard deviation. 00:38:26.900 --> 00:38:28.920 align:middle line:84% And then once I do that, I save them 00:38:28.920 --> 00:38:31.300 align:middle line:84% in the train and the test dataframes. 00:38:31.300 --> 00:38:34.920 align:middle line:84% And you can see here now all the numbers are all very smallish-- 00:38:34.920 --> 00:38:37.340 align:middle line:84% 0, 1, minus 1, around that range. 00:38:37.340 --> 00:38:39.840 align:middle line:84% And that's kind of ideal for neural network training. 00:38:39.840 --> 00:38:40.800 align:middle line:90% OK? 00:38:40.800 --> 00:38:41.300 align:middle line:90% All right. 00:38:41.300 --> 00:38:43.520 align:middle line:84% So at this point, the data is entirely numeric. 00:38:43.520 --> 00:38:46.640 align:middle line:84% And then we are almost ready to feed it into Keras. 00:38:46.640 --> 00:38:49.760 align:middle line:84% And the way you do it is you take a NumPy array-- 00:38:49.760 --> 00:38:51.960 align:middle line:84% you take a Pandas dataframe, and then you 00:38:51.960 --> 00:38:53.620 align:middle line:90% convert it into a NumPy array. 00:38:53.620 --> 00:38:56.340 align:middle line:84% And then Keras is happy to take it, happy to receive it. 00:38:56.340 --> 00:39:00.320 align:middle line:84% So we use this thing called to_numpy, which I think 00:39:00.320 --> 00:39:02.920 align:middle line:84% is as descriptive as it gets in programming. 00:39:02.920 --> 00:39:04.780 align:middle line:84% And then you save it as train and test. 00:39:04.780 --> 00:39:07.180 align:middle line:84% Now, train and test are two NumPy arrays 00:39:07.180 --> 00:39:08.880 align:middle line:84% with exactly the same information, 00:39:08.880 --> 00:39:10.780 align:middle line:84% and now we can feed it into Keras. 00:39:10.780 --> 00:39:11.280 align:middle line:90% All right. 00:39:11.280 --> 00:39:13.363 align:middle line:84% Now I guess there's one other thing we need to do, 00:39:13.363 --> 00:39:17.300 align:middle line:84% which is that in this dataframe, train and test are 00:39:17.300 --> 00:39:18.385 align:middle line:90% independent variables. 00:39:18.385 --> 00:39:20.760 align:middle line:84% All the features, as well as the target, the 0, 1 target, 00:39:20.760 --> 00:39:23.620 align:middle line:90% they're all in this. 00:39:23.620 --> 00:39:25.420 align:middle line:84% And we need to now take it and just 00:39:25.420 --> 00:39:29.220 align:middle line:84% take the dependent variable, the 0, 1 column, and split it out, 00:39:29.220 --> 00:39:31.540 align:middle line:84% and keep the x and the y separately. 00:39:31.540 --> 00:39:32.800 align:middle line:90% That's the whole point of it. 00:39:32.800 --> 00:39:35.260 align:middle line:84% Because you need to feed the x, do the prediction, 00:39:35.260 --> 00:39:36.980 align:middle line:84% and then compare it to the actual y, 00:39:36.980 --> 00:39:39.180 align:middle line:84% and calculate the loss, and so on and so forth. 00:39:39.180 --> 00:39:42.900 align:middle line:84% So the target column is our y variable. 00:39:42.900 --> 00:39:44.800 align:middle line:84% And it's column number 6 from the left. 00:39:44.800 --> 00:39:46.400 align:middle line:90% If you count it, you can see it. 00:39:46.400 --> 00:39:51.900 align:middle line:84% So we delete it from the train and test. 00:39:51.900 --> 00:39:56.720 align:middle line:84% And now we have 242 rows and 29 columns, 29 features. 00:39:56.720 --> 00:39:59.160 align:middle line:84% You will recall from the network that we made way back, 00:39:59.160 --> 00:40:03.040 align:middle line:84% it had 29 inputs, 29 nodes in the input layer. 00:40:03.040 --> 00:40:05.110 align:middle line:84% And that's where the 29 is coming from. 00:40:05.110 --> 00:40:07.647 align:middle line:84% And so now we just select the sixth column, 00:40:07.647 --> 00:40:09.230 align:middle line:84% which is the target, and make it the y 00:40:09.230 --> 00:40:11.670 align:middle line:90% variable, train_y and test_y. 00:40:11.670 --> 00:40:13.170 align:middle line:84% And that is, of course, a vector, 00:40:13.170 --> 00:40:15.590 align:middle line:84% which is 242 long in the training set and 61 long 00:40:15.590 --> 00:40:16.510 align:middle line:90% in the thing. 00:40:16.510 --> 00:40:18.510 align:middle line:84% So at this point, all we have done 00:40:18.510 --> 00:40:21.495 align:middle line:84% is, to be honest, boring preprocessing. 00:40:21.495 --> 00:40:23.370 align:middle line:84% We haven't actually gotten to the action yet. 00:40:23.370 --> 00:40:26.170 align:middle line:90% Finally, let's do something. 00:40:26.170 --> 00:40:28.818 align:middle line:84% And we start with a single hidden layer. 00:40:28.818 --> 00:40:30.610 align:middle line:84% Since it's a binary classification problem, 00:40:30.610 --> 00:40:32.530 align:middle line:84% we'll use sigmoids, as we saw earlier. 00:40:32.530 --> 00:40:36.490 align:middle line:84% And this is the model we created in class, last class. 00:40:36.490 --> 00:40:39.150 align:middle line:90% This is the model we created. 00:40:39.150 --> 00:40:41.390 align:middle line:84% The only difference between that model and this model 00:40:41.390 --> 00:40:44.710 align:middle line:84% is that I've actually given names to these layers. 00:40:44.710 --> 00:40:46.743 align:middle line:84% And this name thing is totally optional. 00:40:46.743 --> 00:40:48.410 align:middle line:84% If you want to give a name, give a name. 00:40:48.410 --> 00:40:50.410 align:middle line:84% It's just a little easier to interpret later on. 00:40:50.410 --> 00:40:52.910 align:middle line:90% It's just cosmetic. 00:40:52.910 --> 00:40:54.990 align:middle line:90% But I've just put it here. 00:40:54.990 --> 00:40:57.550 align:middle line:84% And once you build the model, you 00:40:57.550 --> 00:41:00.470 align:middle line:84% should immediately run the model.summary command 00:41:00.470 --> 00:41:03.705 align:middle line:84% because it gives you a nice overview of the model. 00:41:03.705 --> 00:41:05.080 align:middle line:84% For each layer, it tells you what 00:41:05.080 --> 00:41:08.020 align:middle line:84% the layer is, it tells you what's coming into the layer, 00:41:08.020 --> 00:41:10.200 align:middle line:84% meaning the shape of the tensor that's coming in 00:41:10.200 --> 00:41:13.460 align:middle line:84% and what's going out, and how many parameters the layer has. 00:41:13.460 --> 00:41:15.880 align:middle line:84% And it turns out, this layer has-- 00:41:15.880 --> 00:41:20.640 align:middle line:84% sorry, this network has 497 parameters. 00:41:20.640 --> 00:41:22.887 align:middle line:84% And I have told you repeatedly, the first few times, 00:41:22.887 --> 00:41:24.720 align:middle line:84% just hand-calculate the number of parameters 00:41:24.720 --> 00:41:26.140 align:middle line:90% to make sure it verifies. 00:41:26.140 --> 00:41:29.340 align:middle line:84% So we should just make sure that it is, in fact, 497. 00:41:29.340 --> 00:41:30.900 align:middle line:90% So let's hand-calculate it. 00:41:30.900 --> 00:41:34.000 align:middle line:84% And you do basically-- it's basically what's going on here. 00:41:34.000 --> 00:41:36.300 align:middle line:90% 29 inputs times 16. 00:41:36.300 --> 00:41:37.240 align:middle line:90% All the arrows. 00:41:37.240 --> 00:41:39.080 align:middle line:90% 29 times 16 arrows. 00:41:39.080 --> 00:41:41.300 align:middle line:84% And then you have a bias of another 16. 00:41:41.300 --> 00:41:42.880 align:middle line:84% That's why you have this expression. 00:41:42.880 --> 00:41:44.560 align:middle line:84% And then the next one is 16 times 00:41:44.560 --> 00:41:46.740 align:middle line:84% 1 plus 1 bias for the output sigmoid. 00:41:46.740 --> 00:41:48.480 align:middle line:90% And you get to 497. 00:41:48.480 --> 00:41:48.980 align:middle line:90% OK. 00:41:48.980 --> 00:41:50.840 align:middle line:84% Just make sure you follow this later on when 00:41:50.840 --> 00:41:52.160 align:middle line:90% you work with the Colab. 00:41:52.160 --> 00:41:54.392 align:middle line:90% We did this in class last week. 00:41:54.392 --> 00:41:56.600 align:middle line:84% And you can visualize the network graphically as well 00:41:56.600 --> 00:41:58.600 align:middle line:84% by using the plot_model function. 00:41:58.600 --> 00:42:00.908 align:middle line:90% So we do that here. 00:42:00.908 --> 00:42:02.950 align:middle line:84% And let's say, it gives you the same information, 00:42:02.950 --> 00:42:05.390 align:middle line:84% but in a slightly easier form to consume. 00:42:05.390 --> 00:42:08.302 align:middle line:84% And when we work with larger networks, starting on Wednesday, 00:42:08.302 --> 00:42:10.010 align:middle line:84% you will see that being able to visualize 00:42:10.010 --> 00:42:13.010 align:middle line:84% the topology of the network is actually quite handy. 00:42:13.010 --> 00:42:13.650 align:middle line:90% OK. 00:42:13.650 --> 00:42:17.410 align:middle line:84% We finally come to actually trying to train this thing. 00:42:17.410 --> 00:42:20.050 align:middle line:84% And so what loss function should we use? 00:42:20.050 --> 00:42:23.090 align:middle line:84% We need to use binary_crossentropy, 00:42:23.090 --> 00:42:24.090 align:middle line:90% right there. 00:42:24.090 --> 00:42:25.590 align:middle line:90% What optimizer to use? 00:42:25.590 --> 00:42:29.930 align:middle line:84% Well, as I mentioned earlier, we will use Adam. 00:42:29.930 --> 00:42:32.490 align:middle line:90% Adam. 00:42:32.490 --> 00:42:33.030 align:middle line:90% All right. 00:42:33.030 --> 00:42:35.370 align:middle line:90% Adam. 00:42:35.370 --> 00:42:37.210 align:middle line:84% And then the final thing is, you can 00:42:37.210 --> 00:42:40.230 align:middle line:84% ask Keras to report out whatever metrics you care about. 00:42:40.230 --> 00:42:42.730 align:middle line:84% These metrics are not going to be used in any optimization. 00:42:42.730 --> 00:42:44.330 align:middle line:90% It's just reporting it to you. 00:42:44.330 --> 00:42:46.130 align:middle line:84% And the most common thing people report 00:42:46.130 --> 00:42:49.070 align:middle line:84% out for binary classification is accuracy. 00:42:49.070 --> 00:42:51.130 align:middle line:84% So we'll just go with that metric. 00:42:51.130 --> 00:42:55.050 align:middle line:84% And so what we do is we tell Keras, take the model we just 00:42:55.050 --> 00:42:58.510 align:middle line:84% built and compile it with this choice of optimizer, 00:42:58.510 --> 00:43:00.710 align:middle line:84% this choice of loss function, and these metrics. 00:43:00.710 --> 00:43:02.470 align:middle line:84% And this compilation step, what it does 00:43:02.470 --> 00:43:05.630 align:middle line:84% is, essentially, Keras will take this information 00:43:05.630 --> 00:43:08.350 align:middle line:84% and take the model you've built, and it will reorganize 00:43:08.350 --> 00:43:12.750 align:middle line:84% the model in such a way that the parallel computing, 00:43:12.750 --> 00:43:14.710 align:middle line:84% distribution of computing across many servers, 00:43:14.710 --> 00:43:18.350 align:middle line:84% and so on, that's what's happening in the compile step. 00:43:18.350 --> 00:43:20.390 align:middle line:84% Organizing it so that-- reorganizing the model 00:43:20.390 --> 00:43:22.670 align:middle line:84% so that it becomes amenable to parallelization 00:43:22.670 --> 00:43:24.490 align:middle line:84% and distribution, that's what's going on. 00:43:24.490 --> 00:43:26.323 align:middle line:84% That's why you actually have to do something 00:43:26.323 --> 00:43:27.460 align:middle line:90% called a compile step. 00:43:27.460 --> 00:43:30.190 align:middle line:84% And once we do that, we are finally 00:43:30.190 --> 00:43:31.790 align:middle line:90% ready to train the model. 00:43:31.790 --> 00:43:35.370 align:middle line:84% And to do that, we have to decide what the batch size is 00:43:35.370 --> 00:43:36.370 align:middle line:90% that we're going to use. 00:43:36.370 --> 00:43:37.390 align:middle line:84% Remember, we're using some flavor 00:43:37.390 --> 00:43:40.270 align:middle line:84% of SGD, which means we have to choose what is the batch size. 00:43:40.270 --> 00:43:43.470 align:middle line:84% And typically what people do is that 32 is 00:43:43.470 --> 00:43:45.430 align:middle line:90% a good default for batch size. 00:43:45.430 --> 00:43:48.390 align:middle line:84% If you're just getting started with something, just use 32. 00:43:48.390 --> 00:43:50.172 align:middle line:84% And there's a whole bunch of literature 00:43:50.172 --> 00:43:51.630 align:middle line:84% on what the right batch size should 00:43:51.630 --> 00:43:53.930 align:middle line:84% be for the number of data points you have, 00:43:53.930 --> 00:43:56.050 align:middle line:84% the size of the network, and so on and so forth. 00:43:56.050 --> 00:43:58.410 align:middle line:90% My philosophy is start with 32. 00:43:58.410 --> 00:44:01.020 align:middle line:84% And you can always try 32, 64, 128. 00:44:01.020 --> 00:44:03.480 align:middle line:84% It's kind of like-- oftentimes what people 00:44:03.480 --> 00:44:06.400 align:middle line:84% tell me, researchers tell me, is that just use the biggest batch 00:44:06.400 --> 00:44:09.782 align:middle line:84% size that doesn't make your machine die. 00:44:09.782 --> 00:44:11.740 align:middle line:84% If you can fit into memory, it's probably good. 00:44:11.740 --> 00:44:12.900 align:middle line:90% Just try the bigger sizes. 00:44:12.900 --> 00:44:13.700 align:middle line:90% We'll just start with 32. 00:44:13.700 --> 00:44:14.533 align:middle line:90% It's a tiny problem. 00:44:14.533 --> 00:44:15.400 align:middle line:90% It's not a big deal. 00:44:15.400 --> 00:44:18.740 align:middle line:84% And then we also have to decide, how many epochs through the data 00:44:18.740 --> 00:44:20.440 align:middle line:90% do we want to go through? 00:44:20.440 --> 00:44:22.180 align:middle line:90% How many epochs? 00:44:22.180 --> 00:44:26.365 align:middle line:84% And usually, 20 to 30 epochs is a good starting point. 00:44:26.365 --> 00:44:28.740 align:middle line:84% And then, because this is a tiny problem, just for kicks, 00:44:28.740 --> 00:44:31.080 align:middle line:84% I just had to run it for 300 epochs, just 00:44:31.080 --> 00:44:33.498 align:middle line:84% to see if any overfitting is going to happen. 00:44:33.498 --> 00:44:35.540 align:middle line:84% And then whether we want to use a validation set. 00:44:35.540 --> 00:44:38.220 align:middle line:84% Of course, we want to use a validation set. 00:44:38.220 --> 00:44:40.158 align:middle line:84% So we will use 20% of the data points 00:44:40.158 --> 00:44:42.450 align:middle line:84% as a validation set so that we can look for overfitting 00:44:42.450 --> 00:44:44.320 align:middle line:90% and underfitting. 00:44:44.320 --> 00:44:44.980 align:middle line:90% All right. 00:44:44.980 --> 00:44:47.840 align:middle line:84% So with these decisions made, we finally-- 00:44:47.840 --> 00:44:50.060 align:middle line:90% we use the model.fit command. 00:44:50.060 --> 00:44:54.320 align:middle line:84% model.fit is what actually trains the neural network. 00:44:54.320 --> 00:44:59.330 align:middle line:84% And you have to tell it what the x tensor is. 00:44:59.330 --> 00:45:02.970 align:middle line:84% You have to tell it what the dependent variable y tensor is. 00:45:02.970 --> 00:45:05.270 align:middle line:84% We need to tell it how many epochs to do this, 00:45:05.270 --> 00:45:06.690 align:middle line:90% what the batch size to use. 00:45:06.690 --> 00:45:10.770 align:middle line:84% verbose=1 just means just put a lot of descriptive output as you 00:45:10.770 --> 00:45:11.670 align:middle line:90% do this thing. 00:45:11.670 --> 00:45:16.370 align:middle line:84% And then validation_split means, take 20% of the training data 00:45:16.370 --> 00:45:18.285 align:middle line:84% and set it aside as your validation data set. 00:45:18.285 --> 00:45:20.410 align:middle line:84% Don't use it for training because I want to measure 00:45:20.410 --> 00:45:22.210 align:middle line:90% overfitting using that. 00:45:22.210 --> 00:45:22.910 align:middle line:90% So that's it. 00:45:22.910 --> 00:45:24.150 align:middle line:90% So you do that thing. 00:45:24.150 --> 00:45:25.750 align:middle line:90% It will run for 300 epochs. 00:45:25.750 --> 00:45:28.650 align:middle line:84% And this is the reason why I decided to just not actually 00:45:28.650 --> 00:45:31.050 align:middle line:90% run it in class. 00:45:31.050 --> 00:45:32.730 align:middle line:90% And so you keep on doing it. 00:45:32.730 --> 00:45:34.310 align:middle line:90% It gives you a lot of output. 00:45:34.310 --> 00:45:37.540 align:middle line:90% And finally, we reach the end. 00:45:37.540 --> 00:45:41.550 align:middle line:90% 00:45:41.550 --> 00:45:42.050 align:middle line:90% OK. 00:45:42.050 --> 00:45:43.610 align:middle line:84% Now, let's take a moment to understand 00:45:43.610 --> 00:45:44.527 align:middle line:90% what's being reported. 00:45:44.527 --> 00:45:46.350 align:middle line:84% So I'll just take this one line here. 00:45:46.350 --> 00:45:48.890 align:middle line:90% So these two-- 00:45:48.890 --> 00:45:51.010 align:middle line:84% There is a pair of lines for each epoch. 00:45:51.010 --> 00:45:53.834 align:middle line:90% And then here it's telling you-- 00:45:53.834 --> 00:46:00.020 align:middle line:84% it actually uses-- in this 300th epoch, it used 7 batches. 00:46:00.020 --> 00:46:01.980 align:middle line:90% 7 out of 7 batches. 00:46:01.980 --> 00:46:03.520 align:middle line:90% So it used 7 batches. 00:46:03.520 --> 00:46:06.378 align:middle line:84% And you will recall, from the math we did in the class, 00:46:06.378 --> 00:46:08.420 align:middle line:84% that it's actually 7 batches, where the first six 00:46:08.420 --> 00:46:10.300 align:middle line:84% batches are 32, and the last batch 00:46:10.300 --> 00:46:11.720 align:middle line:90% is just a couple of examples. 00:46:11.720 --> 00:46:13.680 align:middle line:90% But we have 7 batches. 00:46:13.680 --> 00:46:18.100 align:middle line:84% This is the 193 by 32 rounded up. 00:46:18.100 --> 00:46:19.820 align:middle line:90% So that's why you have 7 here. 00:46:19.820 --> 00:46:22.000 align:middle line:84% And then it tells you how long it took it for that. 00:46:22.000 --> 00:46:23.920 align:middle line:90% And then this is the loss value. 00:46:23.920 --> 00:46:26.180 align:middle line:84% This is the binary cross-entropy loss value 00:46:26.180 --> 00:46:30.300 align:middle line:84% on the training set on that particular batch 00:46:30.300 --> 00:46:32.060 align:middle line:90% that it calculated. 00:46:32.060 --> 00:46:35.040 align:middle line:84% This is the accuracy that you asked it to report out. 00:46:35.040 --> 00:46:37.700 align:middle line:90% 98.5% accuracy on that batch. 00:46:37.700 --> 00:46:41.260 align:middle line:84% And then, at the end of this epoch, 00:46:41.260 --> 00:46:43.765 align:middle line:84% using whatever weights were available in that network, 00:46:43.765 --> 00:46:45.140 align:middle line:84% you'd actually calculate the loss 00:46:45.140 --> 00:46:47.820 align:middle line:84% on the validation set, which is the 20% the data 00:46:47.820 --> 00:46:49.100 align:middle line:90% we have set aside. 00:46:49.100 --> 00:46:52.620 align:middle line:84% And then this is the accuracy on the validation set. 00:46:52.620 --> 00:46:55.480 align:middle line:84% So that's what each of these numbers mean. 00:46:55.480 --> 00:46:58.380 align:middle line:84% Now, looking at this wall of numbers is kind of painful. 00:46:58.380 --> 00:47:00.940 align:middle line:90% So usually you just plot it. 00:47:00.940 --> 00:47:03.883 align:middle line:84% And the way you do that is if you notice here-- 00:47:03.883 --> 00:47:05.300 align:middle line:84% OK, I'm not going to go back here. 00:47:05.300 --> 00:47:08.320 align:middle line:84% So I said history equals model.fit, blah, blah, blah, 00:47:08.320 --> 00:47:08.940 align:middle line:90% blah, blah. 00:47:08.940 --> 00:47:11.080 align:middle line:84% And that history object has a lot of information 00:47:11.080 --> 00:47:14.400 align:middle line:84% that we can use for plotting and diagnostics and so on. 00:47:14.400 --> 00:47:17.520 align:middle line:84% And that history thing, history object, 00:47:17.520 --> 00:47:20.620 align:middle line:84% has another object called history, history.history, 00:47:20.620 --> 00:47:22.580 align:middle line:84% which is a dictionary with all these values. 00:47:22.580 --> 00:47:23.820 align:middle line:84% And that's what we're going to plot. 00:47:23.820 --> 00:47:24.903 align:middle line:90% Was there a question here? 00:47:24.903 --> 00:47:25.840 align:middle line:90% Yeah. 00:47:25.840 --> 00:47:29.640 align:middle line:84% AUDIENCE: So you prompted it to keep 20% aside for validation. 00:47:29.640 --> 00:47:32.780 align:middle line:84% But didn't we already do a test set? 00:47:32.780 --> 00:47:34.680 align:middle line:84% So that's going to be a secondary validation. 00:47:34.680 --> 00:47:35.722 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:47:35.722 --> 00:47:38.040 align:middle line:84% So basically, we have a training and then 00:47:38.040 --> 00:47:39.700 align:middle line:90% a validation and a test. 00:47:39.700 --> 00:47:41.880 align:middle line:84% The role of the validation set is to figure out 00:47:41.880 --> 00:47:43.267 align:middle line:90% things like early stopping. 00:47:43.267 --> 00:47:44.100 align:middle line:90% Should we stop here? 00:47:44.100 --> 00:47:45.060 align:middle line:90% Should we go back? 00:47:45.060 --> 00:47:48.200 align:middle line:84% And as you will see later on, if we use hyperparameters, 00:47:48.200 --> 00:47:50.325 align:middle line:84% we will try different values of the hyperparameters 00:47:50.325 --> 00:47:52.533 align:middle line:84% and figure out-- use the validation set to figure out 00:47:52.533 --> 00:47:53.650 align:middle line:90% which one is the best one. 00:47:53.650 --> 00:47:55.270 align:middle line:84% But once we are done with all that, 00:47:55.270 --> 00:47:57.090 align:middle line:90% we will finally have a model. 00:47:57.090 --> 00:48:00.290 align:middle line:84% At that point, we open the safe, take out the test set, 00:48:00.290 --> 00:48:03.240 align:middle line:84% and use it just once with your final, final model. 00:48:03.240 --> 00:48:04.990 align:middle line:84% Not because you want to improve the model, 00:48:04.990 --> 00:48:07.157 align:middle line:84% but because you want to have a realistic idea of how 00:48:07.157 --> 00:48:11.770 align:middle line:84% it will do when you actually deploy it out in the real world. 00:48:11.770 --> 00:48:14.370 align:middle line:90% Yeah. 00:48:14.370 --> 00:48:16.410 align:middle line:90% AUDIENCE: Can we use-- 00:48:16.410 --> 00:48:19.730 align:middle line:84% instead of accuracy, can be used other metrics 00:48:19.730 --> 00:48:21.090 align:middle line:90% to evaluate whether to-- 00:48:21.090 --> 00:48:21.370 align:middle line:90% RAMA RAMAKRISHNAN: Absolutely. 00:48:21.370 --> 00:48:23.745 align:middle line:84% AUDIENCE: --actually [INAUDIBLE] on the confusion matrix, 00:48:23.745 --> 00:48:24.250 align:middle line:90% let's say? 00:48:24.250 --> 00:48:24.850 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:48:24.850 --> 00:48:26.058 align:middle line:90% You can do whatever you want. 00:48:26.058 --> 00:48:28.450 align:middle line:84% You can use-- like I said, it's not used for training. 00:48:28.450 --> 00:48:31.470 align:middle line:84% So there is no mathematical implication of what you choose. 00:48:31.470 --> 00:48:34.302 align:middle line:84% You can choose error rates, accuracy, F1 F beta. 00:48:34.302 --> 00:48:35.510 align:middle line:90% You can do whatever you want. 00:48:35.510 --> 00:48:38.570 align:middle line:84% And Keras, as you will see, has this dizzying list 00:48:38.570 --> 00:48:40.975 align:middle line:84% of possible metrics you can use for reporting. 00:48:40.975 --> 00:48:42.850 align:middle line:84% The key thing to remember is that you're just 00:48:42.850 --> 00:48:43.870 align:middle line:90% reporting these metrics. 00:48:43.870 --> 00:48:47.490 align:middle line:84% You're not actually using them for any training. 00:48:47.490 --> 00:48:49.103 align:middle line:90% Yeah. 00:48:49.103 --> 00:48:51.270 align:middle line:84% AUDIENCE: My question is with respect to validation. 00:48:51.270 --> 00:48:53.740 align:middle line:84% Like, we've got a training data set. 00:48:53.740 --> 00:48:58.660 align:middle line:84% So when we take out 20% of data from validation, 00:48:58.660 --> 00:49:00.820 align:middle line:84% are we taking out from the training set or-- 00:49:00.820 --> 00:49:01.140 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:49:01.140 --> 00:49:02.515 align:middle line:84% AUDIENCE: From there, that level? 00:49:02.515 --> 00:49:04.980 align:middle line:84% Or we go to each batch and take out 20% from each? 00:49:04.980 --> 00:49:06.360 align:middle line:84% RAMA RAMAKRISHNAN: We're taking out from the training set. 00:49:06.360 --> 00:49:07.943 align:middle line:84% AUDIENCE: So it means the batch size-- 00:49:07.943 --> 00:49:10.733 align:middle line:84% the number of data would be available for calculating 00:49:10.733 --> 00:49:11.900 align:middle line:90% the batch size would reduce? 00:49:11.900 --> 00:49:13.025 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:49:13.025 --> 00:49:15.140 align:middle line:84% And in fact, once we take out the validation set, 00:49:15.140 --> 00:49:17.100 align:middle line:90% whatever remaining is 193. 00:49:17.100 --> 00:49:19.240 align:middle line:84% AUDIENCE: And then we divide that into batches. 00:49:19.240 --> 00:49:22.700 align:middle line:84% And then every epoch, the validation and the data 00:49:22.700 --> 00:49:23.703 align:middle line:90% gets different. 00:49:23.703 --> 00:49:24.620 align:middle line:90% RAMA RAMAKRISHNAN: No. 00:49:24.620 --> 00:49:27.260 align:middle line:84% Once you take out the validation set at the very beginning, 00:49:27.260 --> 00:49:29.340 align:middle line:84% you keep it aside, and then you only 00:49:29.340 --> 00:49:32.580 align:middle line:84% evaluate at the end of each epoch 00:49:32.580 --> 00:49:35.865 align:middle line:84% what your loss and accuracy is on that validation set. 00:49:35.865 --> 00:49:37.740 align:middle line:84% AUDIENCE: So you would have cross-validation. 00:49:37.740 --> 00:49:39.400 align:middle line:84% RAMA RAMAKRISHNAN: No, no, we're not doing any of that stuff. 00:49:39.400 --> 00:49:41.733 align:middle line:84% We're just taking it out once, and we're just evaluating 00:49:41.733 --> 00:49:43.460 align:middle line:90% the end of every epoch. 00:49:43.460 --> 00:49:44.340 align:middle line:90% OK. 00:49:44.340 --> 00:49:47.700 align:middle line:90% So yeah. 00:49:47.700 --> 00:49:51.910 align:middle line:84% AUDIENCE: So I know we both asked similar questions- but-- 00:49:51.910 --> 00:49:53.910 align:middle line:84% RAMA RAMAKRISHNAN: Use the microphone, please. 00:49:53.910 --> 00:49:55.370 align:middle line:84% AUDIENCE: I know both of us asked similar questions, 00:49:55.370 --> 00:49:56.610 align:middle line:90% but just to reconfirm. 00:49:56.610 --> 00:49:59.350 align:middle line:84% So here, my training model is giving me, say, 00:49:59.350 --> 00:50:01.710 align:middle line:90% a loss of 0.0860. 00:50:01.710 --> 00:50:04.750 align:middle line:84% My validation model is giving me 0.660. 00:50:04.750 --> 00:50:07.590 align:middle line:84% That means I have already crossed the int. 00:50:07.590 --> 00:50:11.950 align:middle line:84% So when I have to actually test the model, that 00:50:11.950 --> 00:50:13.630 align:middle line:84% is the midpoint which I did, that 00:50:13.630 --> 00:50:16.705 align:middle line:84% will be the model which will get deployed in production. 00:50:16.705 --> 00:50:17.830 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 00:50:17.830 --> 00:50:19.930 align:middle line:84% And as to, OK, what do we do to get that model-- 00:50:19.930 --> 00:50:22.430 align:middle line:84% do we actually have to go back to the beginning and run it 00:50:22.430 --> 00:50:25.570 align:middle line:84% for a few epochs, or can we do something smarter than that?-- 00:50:25.570 --> 00:50:26.870 align:middle line:90% we'll get to that. 00:50:26.870 --> 00:50:27.870 align:middle line:90% Yeah. 00:50:27.870 --> 00:50:30.850 align:middle line:84% AUDIENCE: Isn't that validation set different for each epoch, 00:50:30.850 --> 00:50:31.600 align:middle line:90% or is it the same? 00:50:31.600 --> 00:50:32.975 align:middle line:84% RAMA RAMAKRISHNAN: It's the same. 00:50:32.975 --> 00:50:34.950 align:middle line:84% So what you do is you have a training set. 00:50:34.950 --> 00:50:36.950 align:middle line:84% Before you do any training, you take out 20%. 00:50:36.950 --> 00:50:38.130 align:middle line:90% Keep it aside. 00:50:38.130 --> 00:50:39.930 align:middle line:90% You take whatever is left over. 00:50:39.930 --> 00:50:42.430 align:middle line:84% You divide that into minibatches and then start 00:50:42.430 --> 00:50:43.930 align:middle line:90% running it through each epochs. 00:50:43.930 --> 00:50:46.030 align:middle line:84% But at the end of each epoch, you just 00:50:46.030 --> 00:50:47.870 align:middle line:84% evaluate the quality of that resulting 00:50:47.870 --> 00:50:49.810 align:middle line:90% model using the validation set. 00:50:49.810 --> 00:50:51.772 align:middle line:84% AUDIENCE: What's different between each epoch? 00:50:51.772 --> 00:50:52.730 align:middle line:90% Is it just the weight-- 00:50:52.730 --> 00:50:54.563 align:middle line:84% RAMA RAMAKRISHNAN: The weights have changed. 00:50:54.563 --> 00:50:57.753 align:middle line:84% AUDIENCE: It's the division into the different-- 00:50:57.753 --> 00:50:58.670 align:middle line:90% RAMA RAMAKRISHNAN: No. 00:50:58.670 --> 00:51:02.090 align:middle line:84% So the difference in each epoch is the weights have changed. 00:51:02.090 --> 00:51:04.230 align:middle line:84% So after every minibatch, the weights have changed. 00:51:04.230 --> 00:51:05.730 align:middle line:84% At the end of one epoch, you've gone 00:51:05.730 --> 00:51:09.138 align:middle line:84% through all the data points you ever had in the training set. 00:51:09.138 --> 00:51:11.680 align:middle line:84% And then you come back to the beginning, and you do it again. 00:51:11.680 --> 00:51:17.690 align:middle line:90% 00:51:17.690 --> 00:51:20.770 align:middle line:84% AUDIENCE: How do you identify the sweet spot? 00:51:20.770 --> 00:51:22.490 align:middle line:90% RAMA RAMAKRISHNAN: It's coming. 00:51:22.490 --> 00:51:23.050 align:middle line:90% Yeah. 00:51:23.050 --> 00:51:23.550 align:middle line:90% All right. 00:51:23.550 --> 00:51:24.970 align:middle line:90% So I'm going to keep going. 00:51:24.970 --> 00:51:26.530 align:middle line:90% So we have this here. 00:51:26.530 --> 00:51:27.930 align:middle line:90% And so you just-- 00:51:27.930 --> 00:51:30.310 align:middle line:84% I mean, there's a little bit of Matplotlib code. 00:51:30.310 --> 00:51:32.410 align:middle line:84% So what we do is we just plot the training 00:51:32.410 --> 00:51:34.890 align:middle line:84% loss and the validation loss as a function 00:51:34.890 --> 00:51:36.770 align:middle line:90% of the number of epochs. 00:51:36.770 --> 00:51:39.450 align:middle line:84% And as you can see here, the training loss 00:51:39.450 --> 00:51:40.730 align:middle line:90% is these things here. 00:51:40.730 --> 00:51:44.530 align:middle line:84% And it's steadily going down, as you would expect. 00:51:44.530 --> 00:51:46.910 align:middle line:84% The validation loss goes down here. 00:51:46.910 --> 00:51:49.280 align:middle line:84% And then, at some point, it kind of flattens out, 00:51:49.280 --> 00:51:51.460 align:middle line:84% and then maybe gently starts to rise. 00:51:51.460 --> 00:51:56.140 align:middle line:84% So do you think there's overfitting? 00:51:56.140 --> 00:51:58.320 align:middle line:84% There seems to be some level of overfitting here. 00:51:58.320 --> 00:52:00.180 align:middle line:84% But the thing you have to always remember 00:52:00.180 --> 00:52:04.980 align:middle line:84% is that the binary cross-entropy loss is a loss function that 00:52:04.980 --> 00:52:07.420 align:middle line:84% is convenient for you because it sort of captures 00:52:07.420 --> 00:52:09.980 align:middle line:84% the thing you want to capture, the discrepancy, 00:52:09.980 --> 00:52:13.140 align:middle line:84% but also because it's mathematically convenient. 00:52:13.140 --> 00:52:15.540 align:middle line:84% But what you may actually care about in practice 00:52:15.540 --> 00:52:17.755 align:middle line:90% is something like accuracy. 00:52:17.755 --> 00:52:20.380 align:middle line:84% So I always-- that's why you're reporting out the accuracy when 00:52:20.380 --> 00:52:21.320 align:middle line:90% we do these things. 00:52:21.320 --> 00:52:24.180 align:middle line:84% So you should also plot the accuracy to see what's going on. 00:52:24.180 --> 00:52:26.100 align:middle line:84% And really, you should look at the accuracy 00:52:26.100 --> 00:52:28.725 align:middle line:84% and figure out overfitting and underfitting and all that stuff. 00:52:28.725 --> 00:52:29.740 align:middle line:90% So let's just do that. 00:52:29.740 --> 00:52:34.300 align:middle line:90% So I have here overfitting. 00:52:34.300 --> 00:52:36.060 align:middle line:84% So this is how it looks like for accuracy. 00:52:36.060 --> 00:52:37.727 align:middle line:84% Accuracy, of course, as the model gets-- 00:52:37.727 --> 00:52:39.560 align:middle line:84% as you do more and more epochs, hopefully it 00:52:39.560 --> 00:52:41.100 align:middle line:84% gets better and better for training. 00:52:41.100 --> 00:52:44.380 align:middle line:84% So you can see here accuracy actually climbs all the way up 00:52:44.380 --> 00:52:47.270 align:middle line:90% to the mid 90s right there. 00:52:47.270 --> 00:52:48.630 align:middle line:90% The low 90s here. 00:52:48.630 --> 00:52:51.190 align:middle line:84% The validation gets to this point 00:52:51.190 --> 00:52:53.730 align:middle line:84% after, I don't know, 50 epochs maybe. 00:52:53.730 --> 00:52:55.190 align:middle line:84% And then it kind of flattens out. 00:52:55.190 --> 00:52:57.960 align:middle line:84% And then, strangely, it climbs up again a bit later. 00:52:57.960 --> 00:53:00.470 align:middle line:90% 00:53:00.470 --> 00:53:04.030 align:middle line:84% Now, the fact that the accuracy actually got better 00:53:04.030 --> 00:53:07.150 align:middle line:84% at the very end suggests that maybe we 00:53:07.150 --> 00:53:09.630 align:middle line:90% can live with this overfitting. 00:53:09.630 --> 00:53:10.670 align:middle line:90% It's OK. 00:53:10.670 --> 00:53:11.970 align:middle line:90% It's not the end of the world. 00:53:11.970 --> 00:53:16.332 align:middle line:84% So you can certainly-- what you can do is you can go back 00:53:16.332 --> 00:53:17.290 align:middle line:90% and say, you know what? 00:53:17.290 --> 00:53:19.210 align:middle line:84% No, I'm going to be a purist about this. 00:53:19.210 --> 00:53:20.770 align:middle line:90% Around 50 epochs or so. 00:53:20.770 --> 00:53:23.410 align:middle line:84% I think that's when it actually flattened out for loss. 00:53:23.410 --> 00:53:25.790 align:middle line:84% So you can just go back and just restart the model 00:53:25.790 --> 00:53:28.303 align:middle line:84% and run it only for 50 epochs, not 300. 00:53:28.303 --> 00:53:30.470 align:middle line:84% And then stop and just use that model for everything 00:53:30.470 --> 00:53:31.520 align:middle line:90% from that point on. 00:53:31.520 --> 00:53:32.770 align:middle line:90% Or you can say, you know what? 00:53:32.770 --> 00:53:33.270 align:middle line:90% It's OK. 00:53:33.270 --> 00:53:34.500 align:middle line:90% I can live with this thing. 00:53:34.500 --> 00:53:36.250 align:middle line:84% And so that's what we're going to do here. 00:53:36.250 --> 00:53:37.500 align:middle line:90% Let me just stop for a second. 00:53:37.500 --> 00:53:39.790 align:middle line:90% There was the question. 00:53:39.790 --> 00:53:40.370 align:middle line:90% Yeah. 00:53:40.370 --> 00:53:42.370 align:middle line:84% AUDIENCE: Originally, when we were starting out, 00:53:42.370 --> 00:53:45.520 align:middle line:84% we were saying 20 to 30 epochs, but we're going to be doing 300. 00:53:45.520 --> 00:53:47.580 align:middle line:90% 50 is over 20 to 30. 00:53:47.580 --> 00:53:50.480 align:middle line:84% So when it comes to validation of if you've run enough epochs, 00:53:50.480 --> 00:53:52.620 align:middle line:84% are you doing a derivative calculation? 00:53:52.620 --> 00:53:53.180 align:middle line:90% RAMA RAMAKRISHNAN: Oh, I see. 00:53:53.180 --> 00:53:54.440 align:middle line:90% No, that's a great question. 00:53:54.440 --> 00:53:57.128 align:middle line:84% So the question is, I said start with 20 and 30 epochs 00:53:57.128 --> 00:53:57.920 align:middle line:90% as a rule of thumb. 00:53:57.920 --> 00:53:59.345 align:middle line:90% Here, I'm just going with 300. 00:53:59.345 --> 00:54:00.720 align:middle line:84% And because I'm going with 300, I 00:54:00.720 --> 00:54:03.100 align:middle line:84% can actually see some potential evidence of overfitting. 00:54:03.100 --> 00:54:04.812 align:middle line:84% But if I had done only 20 to 30, maybe I 00:54:04.812 --> 00:54:06.020 align:middle line:90% wouldn't have even seen that. 00:54:06.020 --> 00:54:06.860 align:middle line:90% What happens next? 00:54:06.860 --> 00:54:07.740 align:middle line:90% Is that the question? 00:54:07.740 --> 00:54:08.700 align:middle line:90% Great question. 00:54:08.700 --> 00:54:11.440 align:middle line:84% So what you should do is when you look at these curves, 00:54:11.440 --> 00:54:14.600 align:middle line:84% if, at the end of 30 epochs, you find that the validation 00:54:14.600 --> 00:54:17.960 align:middle line:84% loss continues to drop, then you know maybe there 00:54:17.960 --> 00:54:19.540 align:middle line:90% is more room for it to drop. 00:54:19.540 --> 00:54:21.420 align:middle line:84% So you continue from that point on. 00:54:21.420 --> 00:54:23.560 align:middle line:84% The thing about Keras is that you can actually 00:54:23.560 --> 00:54:27.260 align:middle line:84% run the fit command at that point, 00:54:27.260 --> 00:54:29.020 align:middle line:84% and it will continue where it left off. 00:54:29.020 --> 00:54:31.680 align:middle line:84% It won't go to the beginning again. 00:54:31.680 --> 00:54:32.960 align:middle line:90% So you can run 10. 00:54:32.960 --> 00:54:33.460 align:middle line:90% OK. 00:54:33.460 --> 00:54:35.220 align:middle line:84% The validation is still getting better and better. 00:54:35.220 --> 00:54:35.400 align:middle line:90% OK. 00:54:35.400 --> 00:54:36.340 align:middle line:90% Run for another 10. 00:54:36.340 --> 00:54:37.800 align:middle line:90% It's getting better and better. 00:54:37.800 --> 00:54:38.620 align:middle line:90% Run for another 10. 00:54:38.620 --> 00:54:39.380 align:middle line:90% Getting better and better. 00:54:39.380 --> 00:54:40.140 align:middle line:90% Run for another 10. 00:54:40.140 --> 00:54:41.100 align:middle line:90% Oh, it starts to climb up again. 00:54:41.100 --> 00:54:42.350 align:middle line:90% OK, now I'm going to back off. 00:54:42.350 --> 00:54:44.730 align:middle line:90% That's what you do. 00:54:44.730 --> 00:54:45.470 align:middle line:90% All right. 00:54:45.470 --> 00:54:48.610 align:middle line:84% Now, all this manual stuff, I'm going through it just because-- 00:54:48.610 --> 00:54:49.985 align:middle line:90% to build intuition. 00:54:49.985 --> 00:54:51.610 align:middle line:84% There are these things called callbacks 00:54:51.610 --> 00:54:54.170 align:middle line:84% in Keras, which we'll get to later on, in which you can 00:54:54.170 --> 00:54:59.310 align:middle line:84% actually tell it, hey, when the validation loss stops improving, 00:54:59.310 --> 00:55:01.990 align:middle line:84% stop everything, or when it stops improving, 00:55:01.990 --> 00:55:04.365 align:middle line:84% save that model for me somewhere. 00:55:04.365 --> 00:55:06.490 align:middle line:84% So they don't have to go back and rerun everything. 00:55:06.490 --> 00:55:08.448 align:middle line:84% It will have saved it for you, and you can just 00:55:08.448 --> 00:55:09.530 align:middle line:90% pick it up and use it. 00:55:09.530 --> 00:55:12.290 align:middle line:90% Yeah. 00:55:12.290 --> 00:55:16.490 align:middle line:84% AUDIENCE: What's the intuition behind the accuracy continuing 00:55:16.490 --> 00:55:19.135 align:middle line:84% to improve when the loss is getting higher? 00:55:19.135 --> 00:55:21.510 align:middle line:84% RAMA RAMAKRISHNAN: Because accuracy and loss are related, 00:55:21.510 --> 00:55:24.310 align:middle line:90% but they're not the same thing. 00:55:24.310 --> 00:55:26.030 align:middle line:90% So it's a really good question. 00:55:26.030 --> 00:55:27.690 align:middle line:84% It's also kind of a profound question 00:55:27.690 --> 00:55:29.953 align:middle line:84% because accuracy is a very discrete measure. 00:55:29.953 --> 00:55:32.370 align:middle line:84% So if a particular point, we're predicting its probability 00:55:32.370 --> 00:55:35.270 align:middle line:84% to be, say, 0.49, we're going to say, OK, 00:55:35.270 --> 00:55:37.610 align:middle line:90% that's a 0, no heart disease. 00:55:37.610 --> 00:55:39.290 align:middle line:84% But if it goes to 0.51, we're going 00:55:39.290 --> 00:55:40.930 align:middle line:84% to be like, oh, that's heart disease. 00:55:40.930 --> 00:55:43.810 align:middle line:84% So when you go from 0.49 to 0.51, 00:55:43.810 --> 00:55:46.950 align:middle line:84% the binary cross-entropy loss will change very, very slightly. 00:55:46.950 --> 00:55:49.270 align:middle line:84% Well, the accuracy will go from 0 to 1. 00:55:49.270 --> 00:55:50.290 align:middle line:90% Dramatic jump. 00:55:50.290 --> 00:55:52.230 align:middle line:90% So it's very jumpy and discrete. 00:55:52.230 --> 00:55:54.310 align:middle line:84% And that's why it tends to be a proxy 00:55:54.310 --> 00:55:57.082 align:middle line:90% but a crude proxy for loss. 00:55:57.082 --> 00:55:58.290 align:middle line:90% So that's part of the reason. 00:55:58.290 --> 00:56:01.183 align:middle line:90% And I can talk more offline. 00:56:01.183 --> 00:56:01.683 align:middle line:90% OK. 00:56:01.683 --> 00:56:04.550 align:middle line:90% So-- yeah. 00:56:04.550 --> 00:56:06.610 align:middle line:84% AUDIENCE: You mentioned that if you are curious, 00:56:06.610 --> 00:56:10.350 align:middle line:84% you could stop at epoch 50 in this case 00:56:10.350 --> 00:56:11.930 align:middle line:90% and run it and stop it there. 00:56:11.930 --> 00:56:14.730 align:middle line:84% I was wondering, if you could see the history of the model, 00:56:14.730 --> 00:56:17.750 align:middle line:84% take the weights at epoch 50, and include it in your model, 00:56:17.750 --> 00:56:20.510 align:middle line:84% if it would be roughly the same, or it 00:56:20.510 --> 00:56:22.130 align:middle line:90% would be certain differences? 00:56:22.130 --> 00:56:23.630 align:middle line:84% RAMA RAMAKRISHNAN: You could try it. 00:56:23.630 --> 00:56:24.610 align:middle line:90% Yeah, you should just try it. 00:56:24.610 --> 00:56:27.110 align:middle line:84% Because what happens is that, ultimately, what we care about 00:56:27.110 --> 00:56:29.213 align:middle line:84% is how it performs on the validation set. 00:56:29.213 --> 00:56:30.630 align:middle line:84% Here, it appears to perform better 00:56:30.630 --> 00:56:33.488 align:middle line:84% on the validation set if you stop at 50, but only 00:56:33.488 --> 00:56:34.030 align:middle line:90% for the loss. 00:56:34.030 --> 00:56:36.280 align:middle line:84% For accuracy, actually, if you wait till the very end, 00:56:36.280 --> 00:56:37.790 align:middle line:90% it gets better. 00:56:37.790 --> 00:56:40.310 align:middle line:84% So my thrust tends to be, what is 00:56:40.310 --> 00:56:44.080 align:middle line:84% the measure that's closest to the real-world deployment? 00:56:44.080 --> 00:56:44.940 align:middle line:90% It's accuracy. 00:56:44.940 --> 00:56:48.640 align:middle line:90% So I tend to go with accuracy. 00:56:48.640 --> 00:56:51.440 align:middle line:84% Binary cross-entropy is a beautiful proxy, 00:56:51.440 --> 00:56:53.442 align:middle line:84% but an imperfect proxy for the thing 00:56:53.442 --> 00:56:55.400 align:middle line:84% we actually care about in the real world, which 00:56:55.400 --> 00:56:57.440 align:middle line:90% is error rate and accuracy. 00:56:57.440 --> 00:56:58.993 align:middle line:90% That's why I tend to want both. 00:56:58.993 --> 00:57:00.660 align:middle line:84% And if accuracy is telling me one thing, 00:57:00.660 --> 00:57:03.040 align:middle line:90% I kind of tend to believe that. 00:57:03.040 --> 00:57:03.540 align:middle line:90% All right. 00:57:03.540 --> 00:57:08.000 align:middle line:90% So here is what we have. 00:57:08.000 --> 00:57:09.740 align:middle line:84% So once we do all this we have a model. 00:57:09.740 --> 00:57:11.572 align:middle line:84% And we know we want to evaluate to see, OK, 00:57:11.572 --> 00:57:13.780 align:middle line:84% if we actually deploy it, how good is it going to be? 00:57:13.780 --> 00:57:16.600 align:middle line:84% So you use this thing called the model.evaluate function. 00:57:16.600 --> 00:57:18.540 align:middle line:84% So you take the model.evaluate function. 00:57:18.540 --> 00:57:21.000 align:middle line:84% Now you use the test and the test_x 00:57:21.000 --> 00:57:22.505 align:middle line:84% and the test_y data set, which we 00:57:22.505 --> 00:57:23.880 align:middle line:84% split at the very, very beginning 00:57:23.880 --> 00:57:25.840 align:middle line:84% and never used from that point on. 00:57:25.840 --> 00:57:26.460 align:middle line:90% We run it. 00:57:26.460 --> 00:57:29.240 align:middle line:84% And when I ran it last night, it came up 00:57:29.240 --> 00:57:32.680 align:middle line:84% with an 83.6% accuracy for the model. 00:57:32.680 --> 00:57:34.740 align:middle line:84% And remember, our baseline model, 00:57:34.740 --> 00:57:36.380 align:middle line:84% which just predicts everybody's a 0, 00:57:36.380 --> 00:57:38.663 align:middle line:84% is going to have a 72.6% accuracy. 00:57:38.663 --> 00:57:40.330 align:middle line:84% And this little neural network gives you 00:57:40.330 --> 00:57:43.270 align:middle line:90% 83.6%, which is pretty good. 00:57:43.270 --> 00:57:46.290 align:middle line:84% So it's actually-- phew, it's beating 00:57:46.290 --> 00:57:48.847 align:middle line:84% the model, the baseline model, which is nice. 00:57:48.847 --> 00:57:50.930 align:middle line:84% And I guess there is something here about the fact 00:57:50.930 --> 00:57:53.493 align:middle line:84% that we did a bunch of preprocessing outside Keras, 00:57:53.493 --> 00:57:54.910 align:middle line:84% and then we send stuff into Keras. 00:57:54.910 --> 00:57:57.202 align:middle line:84% You can actually do all this preprocessing inside Keras 00:57:57.202 --> 00:57:57.790 align:middle line:90% automatically. 00:57:57.790 --> 00:57:59.040 align:middle line:90% And there are layers for that. 00:57:59.040 --> 00:58:00.850 align:middle line:84% And I have linked to a bunch of stuff here. 00:58:00.850 --> 00:58:03.127 align:middle line:84% So that's it as far as this model is concerned. 00:58:03.127 --> 00:58:04.710 align:middle line:84% I know we went through it really fast, 00:58:04.710 --> 00:58:06.570 align:middle line:84% but please go through it afterwards 00:58:06.570 --> 00:58:08.590 align:middle line:84% and make sure you understand every single line. 00:58:08.590 --> 00:58:10.585 align:middle line:84% Change each of these lines, rerun it, 00:58:10.585 --> 00:58:11.710 align:middle line:90% see how the output changes. 00:58:11.710 --> 00:58:13.410 align:middle line:84% That's how you build some intuition. 00:58:13.410 --> 00:58:14.690 align:middle line:90% OK. 00:58:14.690 --> 00:58:15.190 align:middle line:90% All right. 00:58:15.190 --> 00:58:17.970 align:middle line:90% Computer vision. 00:58:17.970 --> 00:58:20.170 align:middle line:90% As I do. 00:58:20.170 --> 00:58:22.010 align:middle line:90% AUDIENCE: Just one question. 00:58:22.010 --> 00:58:23.730 align:middle line:84% Is there a way to build the model just 00:58:23.730 --> 00:58:26.490 align:middle line:84% to have less false positives or less false negative? 00:58:26.490 --> 00:58:27.670 align:middle line:90% Or we don't know that? 00:58:27.670 --> 00:58:29.545 align:middle line:84% RAMA RAMAKRISHNAN: Oh, yeah, you can do that. 00:58:29.545 --> 00:58:31.770 align:middle line:84% But there are-- so you can report on all those things 00:58:31.770 --> 00:58:32.710 align:middle line:90% very easily. 00:58:32.710 --> 00:58:35.010 align:middle line:84% But there are more complex loss functions 00:58:35.010 --> 00:58:38.860 align:middle line:84% which will take the asymmetry between the false positive/false 00:58:38.860 --> 00:58:41.320 align:middle line:90% negative into account. 00:58:41.320 --> 00:58:41.820 align:middle line:90% Yeah. 00:58:41.820 --> 00:58:43.540 align:middle line:84% So the short answer-- it's possible. 00:58:43.540 --> 00:58:45.140 align:middle line:90% Yeah. 00:58:45.140 --> 00:58:45.680 align:middle line:90% All right. 00:58:45.680 --> 00:58:47.180 align:middle line:84% So first, let's just talk about, how 00:58:47.180 --> 00:58:50.540 align:middle line:84% do you represent an image digitally? 00:58:50.540 --> 00:58:54.100 align:middle line:84% And so these are our grayscale images represented, 00:58:54.100 --> 00:58:55.280 align:middle line:90% black and white images. 00:58:55.280 --> 00:58:57.040 align:middle line:84% So the basic idea is very simple. 00:58:57.040 --> 00:59:00.020 align:middle line:84% Every picture you have, every location in that picture 00:59:00.020 --> 00:59:01.000 align:middle line:90% is a pixel. 00:59:01.000 --> 00:59:04.020 align:middle line:84% And a pixel basically has a light intensity-- the amount 00:59:04.020 --> 00:59:05.780 align:middle line:90% of light at that location. 00:59:05.780 --> 00:59:11.300 align:middle line:84% And that light level is measured from 0, no light, to blinding 00:59:11.300 --> 00:59:13.100 align:middle line:90% white light, which is 255. 00:59:13.100 --> 00:59:15.780 align:middle line:90% And so all the numbers here-- 00:59:15.780 --> 00:59:20.100 align:middle line:84% if you take this 5, for example, you can see a lot of no light. 00:59:20.100 --> 00:59:23.323 align:middle line:84% All the black regions, those are all zeros. 00:59:23.323 --> 00:59:25.740 align:middle line:84% And then wherever there is white light, there is a number. 00:59:25.740 --> 00:59:29.197 align:middle line:84% And more the amount of light, the closer it gets to 255. 00:59:29.197 --> 00:59:31.280 align:middle line:84% In fact, if you just step back and squint at this, 00:59:31.280 --> 00:59:34.258 align:middle line:90% you can actually see the 5. 00:59:34.258 --> 00:59:34.800 align:middle line:90% So that's it. 00:59:34.800 --> 00:59:37.460 align:middle line:84% That's how black and white images are represented. 00:59:37.460 --> 00:59:39.013 align:middle line:90% Very simple. 00:59:39.013 --> 00:59:40.680 align:middle line:90% Now-- yeah. 00:59:40.680 --> 00:59:42.240 align:middle line:84% AUDIENCE: Can you see [INAUDIBLE]-- 00:59:42.240 --> 00:59:43.880 align:middle line:84% RAMA RAMAKRISHNAN: Microphone, please. 00:59:43.880 --> 00:59:45.880 align:middle line:84% AUDIENCE: Just when you say amount of light, what's 00:59:45.880 --> 00:59:47.172 align:middle line:90% the unit that's being measured? 00:59:47.172 --> 00:59:48.260 align:middle line:90% What do you mean? 00:59:48.260 --> 00:59:50.135 align:middle line:84% RAMA RAMAKRISHNAN: So here, basically what we 00:59:50.135 --> 00:59:54.120 align:middle line:84% have is the computer takes whatever-- 00:59:54.120 --> 00:59:56.867 align:middle line:84% so when you send an analog-- you take an analog picture. 00:59:56.867 --> 00:59:59.200 align:middle line:84% There is a process by which you take that analog picture 00:59:59.200 --> 01:00:01.920 align:middle line:84% and read it in, and it gets mapped to a scale 01:00:01.920 --> 01:00:03.220 align:middle line:90% between 0 and 255. 01:00:03.220 --> 01:00:04.080 align:middle line:90% That's it. 01:00:04.080 --> 01:00:04.580 align:middle line:90% That's all. 01:00:04.580 --> 01:00:06.580 align:middle line:84% So you can think of it as like a relative scale, 01:00:06.580 --> 01:00:08.600 align:middle line:84% a normalized scale, between 0 and 255. 01:00:08.600 --> 01:00:11.840 align:middle line:84% And so it just roughly maps to the amount of light 01:00:11.840 --> 01:00:12.900 align:middle line:90% in that location. 01:00:12.900 --> 01:00:15.413 align:middle line:84% The exact lumens to the number mapping, 01:00:15.413 --> 01:00:16.580 align:middle line:90% I don't know how they do it. 01:00:16.580 --> 01:00:19.480 align:middle line:84% My guess is there are a dizzying number of variations on that. 01:00:19.480 --> 01:00:21.680 align:middle line:84% But for our purposes, just think of it 01:00:21.680 --> 01:00:26.200 align:middle line:84% as, it's a normalized scale which runs from 0 to 255. 01:00:26.200 --> 01:00:26.700 align:middle line:90% All right. 01:00:26.700 --> 01:00:28.780 align:middle line:90% So if you look at-- 01:00:28.780 --> 01:00:29.960 align:middle line:90% so that's what's happening. 01:00:29.960 --> 01:00:32.160 align:middle line:84% Every is a number between 0 to 255. 01:00:32.160 --> 01:00:33.240 align:middle line:90% Boom, boom, boom. 01:00:33.240 --> 01:00:37.810 align:middle line:84% So if you have a color image, each pixel of a color image 01:00:37.810 --> 01:00:41.730 align:middle line:84% is represented by three numbers, and these numbers 01:00:41.730 --> 01:00:44.270 align:middle line:84% measure the intensity of red light, blue light, 01:00:44.270 --> 01:00:45.055 align:middle line:90% and green light. 01:00:45.055 --> 01:00:46.930 align:middle line:84% Because red, blue, and green, if you mix them 01:00:46.930 --> 01:00:50.530 align:middle line:84% in the right proportion, you can get whatever you want. 01:00:50.530 --> 01:00:52.410 align:middle line:84% And so each light density is still 01:00:52.410 --> 01:00:55.188 align:middle line:84% a number between 0 and 255, and that's what you have, 01:00:55.188 --> 01:00:57.730 align:middle line:84% which means that now you have three tables of numbers instead 01:00:57.730 --> 01:00:59.130 align:middle line:90% of one table of numbers. 01:00:59.130 --> 01:01:01.370 align:middle line:84% And by the way, just some lingo here, 01:01:01.370 --> 01:01:05.310 align:middle line:84% in the deep-learning world, these colors, RGB, Red, Blue, 01:01:05.310 --> 01:01:09.010 align:middle line:84% Green, are sometimes referred to as channels. 01:01:09.010 --> 01:01:09.630 align:middle line:90% All right. 01:01:09.630 --> 01:01:10.870 align:middle line:90% So this is what we have here. 01:01:10.870 --> 01:01:12.530 align:middle line:84% This is a picture of Killian Court. 01:01:12.530 --> 01:01:14.113 align:middle line:84% And then if you take that little thing 01:01:14.113 --> 01:01:18.250 align:middle line:84% here, the red table, the green table, and the blue table. 01:01:18.250 --> 01:01:23.730 align:middle line:84% So for this picture, these three tables is a tensor of rank what? 01:01:23.730 --> 01:01:26.090 align:middle line:90% Good. 01:01:26.090 --> 01:01:26.790 align:middle line:90% All right. 01:01:26.790 --> 01:01:27.870 align:middle line:90% Any questions on this? 01:01:27.870 --> 01:01:33.512 align:middle line:90% 01:01:33.512 --> 01:01:35.740 align:middle line:84% So the key task in computer vision-- 01:01:35.740 --> 01:01:39.700 align:middle line:84% obviously, the important thing is image classification. 01:01:39.700 --> 01:01:41.613 align:middle line:84% The most basic task, if you will, 01:01:41.613 --> 01:01:43.780 align:middle line:84% when you're working with images is you have an image 01:01:43.780 --> 01:01:45.740 align:middle line:84% and you want to take whatever-- you take the image 01:01:45.740 --> 01:01:47.780 align:middle line:84% and figure out, OK, you have a list of possible objects 01:01:47.780 --> 01:01:48.960 align:middle line:90% the image could contain. 01:01:48.960 --> 01:01:51.460 align:middle line:84% And you're figuring out, OK, which of these possible objects 01:01:51.460 --> 01:01:52.460 align:middle line:90% exists in that image? 01:01:52.460 --> 01:01:55.460 align:middle line:84% The Dog/Cat classification is the canonical example 01:01:55.460 --> 01:01:57.820 align:middle line:90% that we all know and love. 01:01:57.820 --> 01:02:01.040 align:middle line:84% And that's what we will solve later today and on Wednesday. 01:02:01.040 --> 01:02:03.620 align:middle line:84% But there are many other tasks that you need to be aware of. 01:02:03.620 --> 01:02:06.640 align:middle line:84% So then you actually not just classify an image, 01:02:06.640 --> 01:02:10.583 align:middle line:84% but you also localize where in the image is it. 01:02:10.583 --> 01:02:12.000 align:middle line:84% It's not just enough to say sheep. 01:02:12.000 --> 01:02:14.060 align:middle line:84% You want to figure out, where is the sheep? 01:02:14.060 --> 01:02:15.437 align:middle line:90% And that's called localization. 01:02:15.437 --> 01:02:17.020 align:middle line:84% And the way you do localization is you 01:02:17.020 --> 01:02:19.240 align:middle line:90% put this little box around it. 01:02:19.240 --> 01:02:23.740 align:middle line:84% And then you output not just whether it's a sheep, yes or no, 01:02:23.740 --> 01:02:25.800 align:middle line:90% but the coordinates of this box. 01:02:25.800 --> 01:02:28.147 align:middle line:84% The top left and the bottom right, for example. 01:02:28.147 --> 01:02:29.980 align:middle line:84% If you put the coordinates, you can actually 01:02:29.980 --> 01:02:31.280 align:middle line:90% draw a box around it. 01:02:31.280 --> 01:02:34.030 align:middle line:84% So you output the numbers, the coordinates, 01:02:34.030 --> 01:02:37.510 align:middle line:84% of where this box is in the picture. 01:02:37.510 --> 01:02:39.670 align:middle line:90% This is called localization. 01:02:39.670 --> 01:02:42.470 align:middle line:84% Now, this is object detection where 01:02:42.470 --> 01:02:44.890 align:middle line:84% you may have lots of objects going on, 01:02:44.890 --> 01:02:47.070 align:middle line:84% and you want to pick up every one of them, 01:02:47.070 --> 01:02:49.928 align:middle line:90% and you want to localize it. 01:02:49.928 --> 01:02:50.970 align:middle line:90% This is object detection. 01:02:50.970 --> 01:02:52.850 align:middle line:84% So here, we have gone in there and said, 01:02:52.850 --> 01:02:55.470 align:middle line:90% OK, sheep 1, sheep 2, sheep 3. 01:02:55.470 --> 01:02:59.510 align:middle line:84% And each of these sheep has a little box around it. 01:02:59.510 --> 01:03:03.692 align:middle line:84% By the way, self-driving cars, the camera vision system 01:03:03.692 --> 01:03:06.150 align:middle line:84% is constantly scanning what's coming in through the cameras 01:03:06.150 --> 01:03:09.070 align:middle line:84% and doing object detection constantly, many times a second. 01:03:09.070 --> 01:03:13.390 align:middle line:84% Pedestrian box, zebra crossing box, doggy box, stroller box, 01:03:13.390 --> 01:03:16.310 align:middle line:90% and so on and so forth. 01:03:16.310 --> 01:03:19.070 align:middle line:84% And then we have this thing called semantic segmentation, 01:03:19.070 --> 01:03:22.110 align:middle line:84% where we take every pixel in the picture 01:03:22.110 --> 01:03:23.623 align:middle line:90% and classify every pixel. 01:03:23.623 --> 01:03:25.290 align:middle line:84% We're not classifying the whole picture. 01:03:25.290 --> 01:03:26.790 align:middle line:90% We're classifying every pixel. 01:03:26.790 --> 01:03:31.050 align:middle line:84% So we are saying, OK, all these gray pixels, road. 01:03:31.050 --> 01:03:34.730 align:middle line:84% All these pixels are sheep, and all these pixels are grass. 01:03:34.730 --> 01:03:37.730 align:middle line:90% Every pixel is being classified. 01:03:37.730 --> 01:03:39.690 align:middle line:84% So we are taking an image and, instead 01:03:39.690 --> 01:03:42.550 align:middle line:84% of giving one classification, for every pixel, 01:03:42.550 --> 01:03:44.740 align:middle line:84% we are solving a multi-class classification problem. 01:03:44.740 --> 01:03:48.630 align:middle line:90% 01:03:48.630 --> 01:03:49.750 align:middle line:90% Every pixel is classified. 01:03:49.750 --> 01:03:53.250 align:middle line:84% And just when you think it can't get more complicated than this, 01:03:53.250 --> 01:03:56.010 align:middle line:84% we have something called instance segmentation, where 01:03:56.010 --> 01:03:58.130 align:middle line:84% not only are we classifying every pixel, 01:03:58.130 --> 01:04:01.890 align:middle line:84% we are distinguishing between the different sheep. 01:04:01.890 --> 01:04:05.130 align:middle line:84% So every pixel is classified, and different instances 01:04:05.130 --> 01:04:07.550 align:middle line:84% of the same category need to be identified. 01:04:07.550 --> 01:04:10.490 align:middle line:90% 01:04:10.490 --> 01:04:13.250 align:middle line:84% So these are all some of the most-- 01:04:13.250 --> 01:04:16.230 align:middle line:84% I won't say popular, most prevalently and useful-- 01:04:16.230 --> 01:04:18.290 align:middle line:84% most prevalent and useful categories 01:04:18.290 --> 01:04:19.850 align:middle line:84% of image-processing problems that 01:04:19.850 --> 01:04:23.810 align:middle line:84% are amenable to a deep-learning system. 01:04:23.810 --> 01:04:24.590 align:middle line:90% All right. 01:04:24.590 --> 01:04:26.450 align:middle line:84% So let's go to image classification. 01:04:26.450 --> 01:04:28.617 align:middle line:84% And we're going to work with this application called 01:04:28.617 --> 01:04:29.230 align:middle line:90% fashion-mnist. 01:04:29.230 --> 01:04:33.060 align:middle line:90% 01:04:33.060 --> 01:04:37.660 align:middle line:84% So the idea here is that you have 70,000 images of clothing 01:04:37.660 --> 01:04:42.592 align:middle line:84% items across 10 categories, like boots and sweaters and T-shirts. 01:04:42.592 --> 01:04:43.300 align:middle line:90% You get the idea. 01:04:43.300 --> 01:04:45.260 align:middle line:90% 10 categories of clothing. 01:04:45.260 --> 01:04:48.220 align:middle line:90% We have 70,000 images like this. 01:04:48.220 --> 01:04:50.260 align:middle line:84% And then we'll build a network from scratch 01:04:50.260 --> 01:04:53.720 align:middle line:84% to classify all these things, with pretty high accuracy. 01:04:53.720 --> 01:04:55.580 align:middle line:84% So these classes-- by the way, this 01:04:55.580 --> 01:04:57.160 align:middle line:90% is a very balanced data set. 01:04:57.160 --> 01:04:59.740 align:middle line:84% So 10% of the data is sweaters, 10% is boots, 01:04:59.740 --> 01:05:00.960 align:middle line:90% and so on and so forth. 01:05:00.960 --> 01:05:04.200 align:middle line:84% So a naive baseline model would give you what accuracy? 01:05:04.200 --> 01:05:07.660 align:middle line:90% 01:05:07.660 --> 01:05:08.580 align:middle line:90% 10%. 01:05:08.580 --> 01:05:09.512 align:middle line:90% Exactly. 01:05:09.512 --> 01:05:11.720 align:middle line:84% So we need to build something that's better than 10%. 01:05:11.720 --> 01:05:14.012 align:middle line:84% And I'm glad to report that a simple neural network can 01:05:14.012 --> 01:05:18.460 align:middle line:84% actually get you close to 90% on this [INAUDIBLE]. 01:05:18.460 --> 01:05:19.000 align:middle line:90% All right. 01:05:19.000 --> 01:05:23.380 align:middle line:84% So this is the simple network that we have. 01:05:23.380 --> 01:05:28.141 align:middle line:84% The input in this case is a 28 by 28 picture. 01:05:28.141 --> 01:05:31.310 align:middle line:90% It's a 28 by 28 picture. 01:05:31.310 --> 01:05:37.690 align:middle line:84% And so far, we have been feeding vectors into our neural network. 01:05:37.690 --> 01:05:39.810 align:middle line:84% Now we have a picture, which is 28 by 28. 01:05:39.810 --> 01:05:42.430 align:middle line:90% It's a tensor of rank 2. 01:05:42.430 --> 01:05:44.707 align:middle line:90% It's a table of numbers. 01:05:44.707 --> 01:05:45.290 align:middle line:90% What do we do? 01:05:45.290 --> 01:05:46.530 align:middle line:90% How do we feed that in? 01:05:46.530 --> 01:05:51.230 align:middle line:90% 01:05:51.230 --> 01:05:52.350 align:middle line:90% It's a-- no. 01:05:52.350 --> 01:05:54.170 align:middle line:84% Each image is a table of numbers. 01:05:54.170 --> 01:05:57.590 align:middle line:90% Let's just take a single image. 01:05:57.590 --> 01:05:58.970 align:middle line:90% What do we do? 01:05:58.970 --> 01:06:01.710 align:middle line:90% What do we do with this table? 01:06:01.710 --> 01:06:03.290 align:middle line:90% Convert it into a vector. 01:06:03.290 --> 01:06:03.790 align:middle line:90% Exactly. 01:06:03.790 --> 01:06:05.775 align:middle line:90% And that's called flattening. 01:06:05.775 --> 01:06:07.150 align:middle line:84% So we take this table of numbers, 01:06:07.150 --> 01:06:09.230 align:middle line:90% and we flatten it into a vector. 01:06:09.230 --> 01:06:13.510 align:middle line:84% And so what we do is-- let me just-- 01:06:13.510 --> 01:06:14.550 align:middle line:90% OK. 01:06:14.550 --> 01:06:20.230 align:middle line:90% So we have 28 by 28. 01:06:20.230 --> 01:06:24.310 align:middle line:84% So what we can do is we can take each row. 01:06:24.310 --> 01:06:27.960 align:middle line:84% We can take this row and then write it like that. 01:06:27.960 --> 01:06:29.340 align:middle line:90% We take the second row. 01:06:29.340 --> 01:06:29.840 align:middle line:90% Oops. 01:06:29.840 --> 01:06:32.917 align:middle line:90% 01:06:32.917 --> 01:06:34.250 align:middle line:90% I'm going to write it like that. 01:06:34.250 --> 01:06:38.120 align:middle line:90% 01:06:38.120 --> 01:06:40.600 align:middle line:90% Third row is here. 01:06:40.600 --> 01:06:42.180 align:middle line:90% I'm going to write it like that. 01:06:42.180 --> 01:06:43.160 align:middle line:90% You get the idea. 01:06:43.160 --> 01:06:45.620 align:middle line:84% So you take each row, just rotate it, and stack it all up. 01:06:45.620 --> 01:06:46.860 align:middle line:90% And string them up. 01:06:46.860 --> 01:06:48.500 align:middle line:90% It becomes one long vector. 01:06:48.500 --> 01:06:50.120 align:middle line:90% So this is called flattening. 01:06:50.120 --> 01:06:51.815 align:middle line:84% So that's how you take this thing 01:06:51.815 --> 01:06:53.190 align:middle line:84% and make it into one long vector. 01:06:53.190 --> 01:06:56.400 align:middle line:90% 01:06:56.400 --> 01:06:59.740 align:middle line:84% So when you do that, 28 by 28 is? 01:06:59.740 --> 01:07:00.240 align:middle line:90% What is it? 01:07:00.240 --> 01:07:03.440 align:middle line:90% 01:07:03.440 --> 01:07:05.000 align:middle line:90% 784. 01:07:05.000 --> 01:07:06.060 align:middle line:90% So we get 7-- 01:07:06.060 --> 01:07:07.400 align:middle line:90% so we get a vector. 01:07:07.400 --> 01:07:08.900 align:middle line:90% This is the flattened input. 01:07:08.900 --> 01:07:11.480 align:middle line:90% And you get 784. 01:07:11.480 --> 01:07:16.790 align:middle line:84% It's a vector that's 784 long after the flattening. 01:07:16.790 --> 01:07:18.540 align:middle line:84% We have not done anything complicated yet. 01:07:18.540 --> 01:07:19.915 align:middle line:84% We've literally taken the numbers 01:07:19.915 --> 01:07:22.320 align:middle line:84% and just reorganized them in a different way. 01:07:22.320 --> 01:07:24.560 align:middle line:84% And once we do that, now we are back 01:07:24.560 --> 01:07:27.140 align:middle line:84% in our familiar neural network territory. 01:07:27.140 --> 01:07:28.740 align:middle line:84% We know how to work with vectors. 01:07:28.740 --> 01:07:31.980 align:middle line:84% So we just need to pass it through a hidden layer. 01:07:31.980 --> 01:07:36.380 align:middle line:84% And this hidden layer, we're going to use ReLU neurons. 01:07:36.380 --> 01:07:37.880 align:middle line:84% And I tried a few different values. 01:07:37.880 --> 01:07:42.340 align:middle line:84% And it turns out that 256 neurons does a really good job. 01:07:42.340 --> 01:07:45.100 align:middle line:84% And so I'm going to use 256 neurons here. 01:07:45.100 --> 01:07:46.940 align:middle line:84% And then we need to now think about what 01:07:46.940 --> 01:07:49.940 align:middle line:90% the output layer should be. 01:07:49.940 --> 01:07:53.348 align:middle line:84% Now we run into a problem because the output layer-- 01:07:53.348 --> 01:07:55.140 align:middle line:84% before we saw for the heart disease example 01:07:55.140 --> 01:07:56.440 align:middle line:90% it's just 0 or 1. 01:07:56.440 --> 01:07:59.620 align:middle line:84% Here, there are 10 possible outputs. 01:07:59.620 --> 01:08:02.940 align:middle line:84% It could be a boot, a sweater, a shirt, and so on and so forth. 01:08:02.940 --> 01:08:04.340 align:middle line:90% 10 possible categories. 01:08:04.340 --> 01:08:06.300 align:middle line:84% So we need some way to handle something 01:08:06.300 --> 01:08:11.660 align:middle line:84% with more than one binary output, many possible outputs. 01:08:11.660 --> 01:08:15.380 align:middle line:90% So the way we do that-- 01:08:15.380 --> 01:08:18.060 align:middle line:84% by the way, pay attention to this, because this is actually 01:08:18.060 --> 01:08:19.979 align:middle line:90% how GPT-4 works. 01:08:19.979 --> 01:08:24.670 align:middle line:84% So what we do is, here's what we have. 01:08:24.670 --> 01:08:27.310 align:middle line:84% We know how to output 10 numbers. 01:08:27.310 --> 01:08:29.670 align:middle line:84% If you want to output 10 numbers, no problem. 01:08:29.670 --> 01:08:31.430 align:middle line:84% We can easily output 10 numbers by just 01:08:31.430 --> 01:08:32.870 align:middle line:90% using a linear activation. 01:08:32.870 --> 01:08:36.630 align:middle line:84% We also know how to output 10 probabilities. 01:08:36.630 --> 01:08:38.490 align:middle line:84% Each one just needs to be a sigmoid. 01:08:38.490 --> 01:08:41.609 align:middle line:84% But here, we can't use 10 sigmoids as the output. 01:08:41.609 --> 01:08:44.031 align:middle line:90% Why is that? 01:08:44.031 --> 01:08:45.239 align:middle line:90% Why can't we use 10 sigmoids? 01:08:45.239 --> 01:08:47.640 align:middle line:90% 01:08:47.640 --> 01:08:49.890 align:middle line:84% AUDIENCE: Because the probabilities wouldn't sum to 1. 01:08:49.890 --> 01:08:51.015 align:middle line:90% RAMA RAMAKRISHNAN: Correct. 01:08:51.015 --> 01:08:53.330 align:middle line:84% So here, when the output comes, we need to figure out, 01:08:53.330 --> 01:08:56.210 align:middle line:84% OK, is it a boot, a sweater, a shirt, and so on and so forth? 01:08:56.210 --> 01:08:58.832 align:middle line:90% There is only one right answer. 01:08:58.832 --> 01:09:00.790 align:middle line:84% Which means that we need to actually figure out 01:09:00.790 --> 01:09:03.415 align:middle line:84% which of these 10 is the right answer, which means that we need 01:09:03.415 --> 01:09:05.149 align:middle line:84% to produce probabilities, but they 01:09:05.149 --> 01:09:09.350 align:middle line:84% have to add up to 1 because only one of them can be true. 01:09:09.350 --> 01:09:10.350 align:middle line:90% So that's the key thing. 01:09:10.350 --> 01:09:11.290 align:middle line:90% They have to add up to 1. 01:09:11.290 --> 01:09:12.082 align:middle line:90% That's the wrinkle. 01:09:12.082 --> 01:09:14.750 align:middle line:84% If not for that, we can just use 10 sigmoids. 01:09:14.750 --> 01:09:17.189 align:middle line:84% And the way we do that is using something 01:09:17.189 --> 01:09:20.029 align:middle line:84% called the softmax function or the softmax layer. 01:09:20.029 --> 01:09:22.029 align:middle line:84% And the idea is actually very simple. 01:09:22.029 --> 01:09:25.600 align:middle line:84% We have these 10 outputs in the very final layer, which 01:09:25.600 --> 01:09:27.720 align:middle line:90% is just linear activations. 01:09:27.720 --> 01:09:30.439 align:middle line:84% And then we take each one of these numbers, 01:09:30.439 --> 01:09:33.880 align:middle line:84% and then run it through the exponential function, 01:09:33.880 --> 01:09:36.160 align:middle line:90% and then divide by the total. 01:09:36.160 --> 01:09:38.165 align:middle line:84% So when you do that, two things happen. 01:09:38.165 --> 01:09:40.540 align:middle line:84% The first one is when you take these numbers and run it-- 01:09:40.540 --> 01:09:43.340 align:middle line:84% say you take a1 and do e raised to a1, 01:09:43.340 --> 01:09:45.583 align:middle line:90% you now get a positive number. 01:09:45.583 --> 01:09:47.000 align:middle line:84% And now you have a positive number 01:09:47.000 --> 01:09:49.100 align:middle line:84% divided by the sum of a bunch of positive numbers. 01:09:49.100 --> 01:09:50.220 align:middle line:84% And they're all-- you can see here, 01:09:50.220 --> 01:09:52.240 align:middle line:84% you can confirm visually that they will add up 01:09:52.240 --> 01:09:54.720 align:middle line:84% to 1 because you're literally taking each number, 01:09:54.720 --> 01:09:55.698 align:middle line:90% dividing by the total. 01:09:55.698 --> 01:09:56.740 align:middle line:90% So they will add up to 1. 01:09:56.740 --> 01:09:58.880 align:middle line:90% There's no other option. 01:09:58.880 --> 01:10:00.540 align:middle line:84% So this is called the softmax function, 01:10:00.540 --> 01:10:02.680 align:middle line:84% which means that you can take any set of 10 numbers that's 01:10:02.680 --> 01:10:05.222 align:middle line:84% coming out of the network and convert them into probabilities 01:10:05.222 --> 01:10:07.160 align:middle line:90% that add up to 1. 01:10:07.160 --> 01:10:09.840 align:middle line:84% And so, by the way, the GPT-4 reference, 01:10:09.840 --> 01:10:13.440 align:middle line:84% when you actually put a prompt into GPT-4 and it starts giving 01:10:13.440 --> 01:10:17.480 align:middle line:84% you the output, every word it's emitting-- 01:10:17.480 --> 01:10:19.780 align:middle line:84% it's actually a token, but we'll get to that later. 01:10:19.780 --> 01:10:20.900 align:middle line:90% Imagine it's a word. 01:10:20.900 --> 01:10:27.650 align:middle line:84% Every word it's emitting, it's doing a 52,000-way softmax. 01:10:27.650 --> 01:10:31.390 align:middle line:84% Think of it as every word in the language is a possible output. 01:10:31.390 --> 01:10:33.730 align:middle line:84% So it's a vector, which is 52,000 long, 01:10:33.730 --> 01:10:35.570 align:middle line:90% but it's actually a softmax. 01:10:35.570 --> 01:10:39.690 align:middle line:84% And it just picks the most probable word and emits that. 01:10:39.690 --> 01:10:43.770 align:middle line:84% So this notion of a softmax is actually very powerful. 01:10:43.770 --> 01:10:45.890 align:middle line:84% But we'll come back to that later. 01:10:45.890 --> 01:10:49.810 align:middle line:84% So to summarize, if you have a single number, 01:10:49.810 --> 01:10:53.330 align:middle line:84% you can use a simple output layer, a single probability, 01:10:53.330 --> 01:10:54.268 align:middle line:90% a sigmoid. 01:10:54.268 --> 01:10:55.310 align:middle line:90% You have lots of numbers. 01:10:55.310 --> 01:10:56.753 align:middle line:84% Just have a stack of these things. 01:10:56.753 --> 01:10:58.170 align:middle line:84% And when you have a lot of numbers 01:10:58.170 --> 01:11:00.430 align:middle line:84% that have to add up to 1 that have to be probabilities, 01:11:00.430 --> 01:11:00.940 align:middle line:90% use softmax. 01:11:00.940 --> 01:11:03.610 align:middle line:90% 01:11:03.610 --> 01:11:04.450 align:middle line:90% OK. 01:11:04.450 --> 01:11:05.090 align:middle line:90% So. 01:11:05.090 --> 01:11:06.423 align:middle line:90% Yeah? 01:11:06.423 --> 01:11:08.090 align:middle line:84% AUDIENCE: Why do we choose probabilities 01:11:08.090 --> 01:11:11.488 align:middle line:84% instead of just number when we know it's going to be only 1? 01:11:11.488 --> 01:11:12.530 align:middle line:90% RAMA RAMAKRISHNAN: Sorry? 01:11:12.530 --> 01:11:14.090 align:middle line:84% AUDIENCE: When we know it's only going to be 1? 01:11:14.090 --> 01:11:15.810 align:middle line:84% RAMA RAMAKRISHNAN: Because you can't force the network to give 01:11:15.810 --> 01:11:16.660 align:middle line:90% you 1's or 0's. 01:11:16.660 --> 01:11:19.530 align:middle line:90% 01:11:19.530 --> 01:11:22.670 align:middle line:84% It's going to produce what it's going to produce. 01:11:22.670 --> 01:11:26.230 align:middle line:84% You can't force it to be exactly 1 or 0. 01:11:26.230 --> 01:11:27.530 align:middle line:90% It will give you some number. 01:11:27.530 --> 01:11:29.697 align:middle line:84% The best you can do is to tame that number 01:11:29.697 --> 01:11:31.530 align:middle line:84% so that it comes into a range that you like, 01:11:31.530 --> 01:11:34.670 align:middle line:90% like between 0 and 1. 01:11:34.670 --> 01:11:39.990 align:middle line:84% So here, very quickly, we have a binary classification example 01:11:39.990 --> 01:11:43.010 align:middle line:84% like yes or no, this is the one-hot-encoded version, 1 or 0. 01:11:43.010 --> 01:11:45.390 align:middle line:84% This is what we saw in the heart disease example. 01:11:45.390 --> 01:11:47.765 align:middle line:84% When you have something like this example, fashion-mnist, 01:11:47.765 --> 01:11:50.030 align:middle line:84% where you have all these different possibilities, 01:11:50.030 --> 01:11:52.290 align:middle line:84% then you can encode it in one of two ways. 01:11:52.290 --> 01:11:55.450 align:middle line:84% You can encode it just using integers, like 0 to 9. 01:11:55.450 --> 01:11:58.270 align:middle line:84% This is called the sparse encoded version. 01:11:58.270 --> 01:12:01.510 align:middle line:84% Or you can do a one-hot-encoded version of the output. 01:12:01.510 --> 01:12:04.630 align:middle line:84% You can have a one-hot-encoded version of the output. 01:12:04.630 --> 01:12:08.330 align:middle line:84% And depending on how your data comes in to you, 01:12:08.330 --> 01:12:11.850 align:middle line:84% comes into your Colab, just pay attention to this. 01:12:11.850 --> 01:12:14.910 align:middle line:84% And depending on what it is, you have to pick the right Keras 01:12:14.910 --> 01:12:15.950 align:middle line:90% loss function. 01:12:15.950 --> 01:12:19.387 align:middle line:84% So if our data comes like a 1 and 0 01:12:19.387 --> 01:12:21.720 align:middle line:84% thing, which is exactly what we had in the heart disease 01:12:21.720 --> 01:12:23.880 align:middle line:84% example, we use binary_crossentropy. 01:12:23.880 --> 01:12:27.500 align:middle line:84% If your data comes in this form where it's sparse encoded, 01:12:27.500 --> 01:12:30.600 align:middle line:84% you use sparse_categorical_crossentropy. 01:12:30.600 --> 01:12:32.240 align:middle line:84% And then if it comes in this form, 01:12:32.240 --> 01:12:34.727 align:middle line:84% you use categorical_crossentropy. 01:12:34.727 --> 01:12:36.060 align:middle line:90% These are all equivalent things. 01:12:36.060 --> 01:12:37.780 align:middle line:84% It just depends on the data that you get, 01:12:37.780 --> 01:12:41.092 align:middle line:84% how it happens to be encoded by the people who sent it to you. 01:12:41.092 --> 01:12:43.300 align:middle line:84% And if they send it this way, use this loss function. 01:12:43.300 --> 01:12:46.060 align:middle line:84% If they send it that way, use that loss function. 01:12:46.060 --> 01:12:47.560 align:middle line:84% Now, as it turns out, in our example 01:12:47.560 --> 01:12:49.623 align:middle line:84% here, the data is actually coming in this form. 01:12:49.623 --> 01:12:51.040 align:middle line:84% So we'll use this thing called the 01:12:51.040 --> 01:12:53.360 align:middle line:90% sparse_categorical_crossentropy. 01:12:53.360 --> 01:12:55.640 align:middle line:84% And categorical_crossentropy is a generalization 01:12:55.640 --> 01:12:57.348 align:middle line:84% of binary_crossentropy, which I'm not 01:12:57.348 --> 01:12:59.140 align:middle line:84% going to get into the mathematical details, 01:12:59.140 --> 01:13:01.660 align:middle line:84% but the intuition is basically roughly the same. 01:13:01.660 --> 01:13:04.280 align:middle line:90% 01:13:04.280 --> 01:13:07.080 align:middle line:90% So this is what we have. 01:13:07.080 --> 01:13:09.680 align:middle line:84% If this is your output layer, use mean squared error. 01:13:09.680 --> 01:13:13.182 align:middle line:84% If this is your output layer, use binary cross-entropy. 01:13:13.182 --> 01:13:15.140 align:middle line:84% And if you still have a stack of these numbers, 01:13:15.140 --> 01:13:16.682 align:middle line:84% you can still use mean squared error. 01:13:16.682 --> 01:13:18.850 align:middle line:84% And if your output is a softmax, use 01:13:18.850 --> 01:13:21.970 align:middle line:84% categorical cross-entropy or sparse categorical 01:13:21.970 --> 01:13:24.450 align:middle line:90% cross-entropy. 01:13:24.450 --> 01:13:25.090 align:middle line:90% OK. 01:13:25.090 --> 01:13:26.740 align:middle line:84% So let's actually run this in Colab. 01:13:26.740 --> 01:13:31.970 align:middle line:90% 01:13:31.970 --> 01:13:33.550 align:middle line:90% So this is what we have. 01:13:33.550 --> 01:13:35.370 align:middle line:90% Can folks see this OK? 01:13:35.370 --> 01:13:36.650 align:middle line:90% All right. 01:13:36.650 --> 01:13:40.730 align:middle line:84% So this is the data set we saw earlier. 01:13:40.730 --> 01:13:41.650 align:middle line:90% Here. 01:13:41.650 --> 01:13:45.470 align:middle line:84% As usual, we load TensorFlow and Keras. 01:13:45.470 --> 01:13:48.330 align:middle line:84% We load our usual three packages. 01:13:48.330 --> 01:13:50.830 align:middle line:84% And then we set the random_seed for reproducibility. 01:13:50.830 --> 01:13:53.370 align:middle line:84% And it turns out that the fashion-mnist data is actually 01:13:53.370 --> 01:13:54.370 align:middle line:90% available in Keras. 01:13:54.370 --> 01:13:56.310 align:middle line:84% You don't have to go find it somewhere and bring it in. 01:13:56.310 --> 01:13:57.390 align:middle line:84% It's actually available in Keras. 01:13:57.390 --> 01:13:59.050 align:middle line:84% It's one of the standard data sets. 01:13:59.050 --> 01:13:59.790 align:middle line:90% We luck out. 01:13:59.790 --> 01:14:02.130 align:middle line:84% So we just actually load the data 01:14:02.130 --> 01:14:04.190 align:middle line:90% using this load_data command. 01:14:04.190 --> 01:14:06.690 align:middle line:84% And then, if you do that, and conveniently for us, 01:14:06.690 --> 01:14:08.950 align:middle line:84% Keras has not only made the data available, 01:14:08.950 --> 01:14:11.568 align:middle line:84% it has already split it into a training and test set. 01:14:11.568 --> 01:14:13.110 align:middle line:84% So we don't have to do the splitting. 01:14:13.110 --> 01:14:15.530 align:middle line:84% And the reason they do that-- why would they do that? 01:14:15.530 --> 01:14:18.660 align:middle line:90% 01:14:18.660 --> 01:14:20.247 align:middle line:84% They do that so that different people 01:14:20.247 --> 01:14:22.580 align:middle line:84% who are building algorithms for that particular data set 01:14:22.580 --> 01:14:26.572 align:middle line:84% can all be evaluated using the same test set. 01:14:26.572 --> 01:14:28.280 align:middle line:84% Otherwise, if I split it one way and say, 01:14:28.280 --> 01:14:30.363 align:middle line:84% hey, look how well I did that, well, I don't know. 01:14:30.363 --> 01:14:31.380 align:middle line:90% How did you split it? 01:14:31.380 --> 01:14:34.000 align:middle line:90% That's the reason. 01:14:34.000 --> 01:14:38.220 align:middle line:84% And you can see here that we have-- 01:14:38.220 --> 01:14:45.860 align:middle line:84% the input data is a tensor of rank 3. 01:14:45.860 --> 01:14:48.580 align:middle line:84% And basically, another way to think about a tensor of rank 3 01:14:48.580 --> 01:14:50.970 align:middle line:84% is just a list of rank 2 tensors. 01:14:50.970 --> 01:14:54.060 align:middle line:90% So here you have 60,000 images. 01:14:54.060 --> 01:14:55.580 align:middle line:90% 60,000 images. 01:14:55.580 --> 01:15:01.620 align:middle line:84% And each image is a 28 by 28 square of numbers. 01:15:01.620 --> 01:15:04.060 align:middle line:90% Each image is a 28 by 28 table. 01:15:04.060 --> 01:15:06.940 align:middle line:84% And then, of course, the output is just 01:15:06.940 --> 01:15:10.000 align:middle line:84% what category it is, a number between 0 and 9. 01:15:10.000 --> 01:15:11.400 align:middle line:90% So you just have 60,000 numbers. 01:15:11.400 --> 01:15:13.160 align:middle line:84% It's just a vector of 60,000 numbers. 01:15:13.160 --> 01:15:16.220 align:middle line:84% So there are 60,000 in the training set. 01:15:16.220 --> 01:15:17.280 align:middle line:90% Oops. 01:15:17.280 --> 01:15:20.020 align:middle line:84% And then there are 10,000 in the test set. 01:15:20.020 --> 01:15:22.280 align:middle line:90% Same structure, 28 by 28. 01:15:22.280 --> 01:15:23.180 align:middle line:90% That's what we have. 01:15:23.180 --> 01:15:24.720 align:middle line:84% So if you look at the first 10 rows 01:15:24.720 --> 01:15:28.280 align:middle line:84% of the dependent variable y, you get these numbers, 9, 0, 3, 3, 01:15:28.280 --> 01:15:28.780 align:middle line:90% like that. 01:15:28.780 --> 01:15:30.400 align:middle line:90% There are numbers from 0 to 9. 01:15:30.400 --> 01:15:33.140 align:middle line:84% So if you look at the fashion-mnist GitHub site, 01:15:33.140 --> 01:15:34.320 align:middle line:90% this is what it refers to. 01:15:34.320 --> 01:15:38.480 align:middle line:84% 0 is a T-shirt, 1 is a trouser, and so on and so forth. 01:15:38.480 --> 01:15:41.520 align:middle line:90% And 9 is an ankle boot. 01:15:41.520 --> 01:15:42.520 align:middle line:90% All right. 01:15:42.520 --> 01:15:45.200 align:middle line:84% So whenever I'm working with multi-class classification 01:15:45.200 --> 01:15:47.840 align:middle line:84% problems, I always do a little thing 01:15:47.840 --> 01:15:50.902 align:middle line:84% here to help me figure out that 9 corresponds to an ankle boot 01:15:50.902 --> 01:15:51.860 align:middle line:90% and so on and so forth. 01:15:51.860 --> 01:15:54.235 align:middle line:84% It just makes it a little easier to work with this stuff. 01:15:54.235 --> 01:15:56.560 align:middle line:90% So I create this little list. 01:15:56.560 --> 01:15:58.930 align:middle line:90% And then it turns out, if you-- 01:15:58.930 --> 01:16:00.760 align:middle line:84% OK, what is the very first data point? 01:16:00.760 --> 01:16:01.260 align:middle line:90% What is it? 01:16:01.260 --> 01:16:02.540 align:middle line:90% What is its y value? 01:16:02.540 --> 01:16:04.400 align:middle line:84% It turns out to be an ankle boot. 01:16:04.400 --> 01:16:07.840 align:middle line:84% So you can actually look at the raw data for that image, which 01:16:07.840 --> 01:16:09.780 align:middle line:90% is just a 28 by 28 thing. 01:16:09.780 --> 01:16:12.701 align:middle line:84% And these are the numbers you have. 01:16:12.701 --> 01:16:13.710 align:middle line:90% You can see all these. 01:16:13.710 --> 01:16:18.142 align:middle line:84% 250, 233, lots of zeros, and so on and so forth. 01:16:18.142 --> 01:16:20.350 align:middle line:84% And so you can actually look at the first-- visualize 01:16:20.350 --> 01:16:21.210 align:middle line:90% the first 25 images. 01:16:21.210 --> 01:16:23.610 align:middle line:84% I have a little bit of code here which visualizes that. 01:16:23.610 --> 01:16:24.722 align:middle line:90% Just Matplotlib code. 01:16:24.722 --> 01:16:26.430 align:middle line:84% And you can see these are all the images. 01:16:26.430 --> 01:16:27.530 align:middle line:90% They're kind of smallish. 01:16:27.530 --> 01:16:30.000 align:middle line:84% This, my friends, is an ankle boot. 01:16:30.000 --> 01:16:32.870 align:middle line:90% 01:16:32.870 --> 01:16:35.210 align:middle line:84% It's like, OK, can the network really make any sense out 01:16:35.210 --> 01:16:35.793 align:middle line:90% of this thing? 01:16:35.793 --> 01:16:37.590 align:middle line:84% It looks very blurry and, I don't know. 01:16:37.590 --> 01:16:41.586 align:middle line:90% 01:16:41.586 --> 01:16:43.630 align:middle line:84% Oh, this is actually a better ankle boot. 01:16:43.630 --> 01:16:44.230 align:middle line:90% Look at that. 01:16:44.230 --> 01:16:44.650 align:middle line:90% OK. 01:16:44.650 --> 01:16:44.990 align:middle line:90% Sorry. 01:16:44.990 --> 01:16:45.950 align:middle line:90% I'm getting distracted. 01:16:45.950 --> 01:16:49.810 align:middle line:90% So this is what we have here. 01:16:49.810 --> 01:16:51.450 align:middle line:90% OK, we're at 9:55. 01:16:51.450 --> 01:16:53.410 align:middle line:84% I'm going to stop so you folks are not 01:16:53.410 --> 01:16:54.470 align:middle line:90% late for your next class. 01:16:54.470 --> 01:16:56.270 align:middle line:84% So we'll continue this journey on Wednesday, 01:16:56.270 --> 01:16:58.410 align:middle line:84% and then we'll go on to color images in the next class 01:16:58.410 --> 01:16:58.910 align:middle line:90% as well. 01:16:58.910 --> 01:16:59.618 align:middle line:90% Thank you, folks. 01:16:59.618 --> 01:17:01.440 align:middle line:90% Have a good one. 01:17:01.440 --> 01:17:12.000 align:middle line:90%