WEBVTT 00:00:00.000 --> 00:00:01.964 align:middle line:90% [SQUEAKING] 00:00:01.964 --> 00:00:03.928 align:middle line:90% [RUSTLING] 00:00:03.928 --> 00:00:07.856 align:middle line:90% [CLICKING] 00:00:07.856 --> 00:00:17.020 align:middle line:90% 00:00:17.020 --> 00:00:19.920 align:middle line:84% RAMA RAMAKRISHNAN: All right, folks, good morning. 00:00:19.920 --> 00:00:20.740 align:middle line:90% Welcome back. 00:00:20.740 --> 00:00:23.240 align:middle line:84% I hope you all had a nice weekend 00:00:23.240 --> 00:00:26.440 align:middle line:84% and I hope you had a chance to watch the video walkthrough I 00:00:26.440 --> 00:00:28.320 align:middle line:90% posted yesterday. 00:00:28.320 --> 00:00:31.000 align:middle line:84% It's going to save us some time today, so let's get right in. 00:00:31.000 --> 00:00:33.360 align:middle line:84% Today is going to be super packed. 00:00:33.360 --> 00:00:35.440 align:middle line:84% You're going to go from not knowing anything 00:00:35.440 --> 00:00:37.353 align:middle line:84% about convolutions perhaps for some of you, 00:00:37.353 --> 00:00:39.520 align:middle line:84% to actually knowing how convolutional networks work, 00:00:39.520 --> 00:00:42.200 align:middle line:84% and actually to build one, and demo it in class. 00:00:42.200 --> 00:00:44.600 align:middle line:84% And this demo has actually worked pretty well 00:00:44.600 --> 00:00:46.860 align:middle line:84% for the last few years that I've taught the class. 00:00:46.860 --> 00:00:48.380 align:middle line:84% But you never know because it's a live demo. 00:00:48.380 --> 00:00:49.100 align:middle line:90% It may not work. 00:00:49.100 --> 00:00:51.880 align:middle line:90% We'll see. 00:00:51.880 --> 00:00:54.720 align:middle line:84% Valentine's Day gods, may they be with us. 00:00:54.720 --> 00:00:56.260 align:middle line:90% OK, so let's get going. 00:00:56.260 --> 00:01:00.070 align:middle line:84% So Fashion MNIST, we saw previously, i.e. 00:01:00.070 --> 00:01:02.950 align:middle line:84% as in, in the walkthrough, the video walkthrough, 00:01:02.950 --> 00:01:07.110 align:middle line:84% that a neural network with a single hidden layer can get us 00:01:07.110 --> 00:01:10.670 align:middle line:90% to an accuracy in the high 80s. 00:01:10.670 --> 00:01:13.590 align:middle line:84% And that thing, that network actually 00:01:13.590 --> 00:01:16.270 align:middle line:84% didn't know what was coming in was an image. 00:01:16.270 --> 00:01:19.413 align:middle line:84% It literally took this table of numbers, and just took each row, 00:01:19.413 --> 00:01:20.830 align:middle line:84% and then concatenated all the rows 00:01:20.830 --> 00:01:23.530 align:middle line:84% into one giant, long vector, and then sent it in. 00:01:23.530 --> 00:01:26.230 align:middle line:84% So the neural network did exploit the fact that the input 00:01:26.230 --> 00:01:29.950 align:middle line:84% data was known to be of a certain type, which 00:01:29.950 --> 00:01:32.750 align:middle line:84% is the clue for, how can we do better? 00:01:32.750 --> 00:01:37.150 align:middle line:84% So let's just spend a few minutes on, 00:01:37.150 --> 00:01:41.070 align:middle line:84% what is it about images that we have to really pay attention to, 00:01:41.070 --> 00:01:43.350 align:middle line:84% as opposed to any arbitrary vector of numbers that's 00:01:43.350 --> 00:01:44.390 align:middle line:90% coming in? 00:01:44.390 --> 00:01:48.590 align:middle line:84% So when we flatten the image into a long vector 00:01:48.590 --> 00:01:52.230 align:middle line:84% and feed it into a dense layer, several undesirable things 00:01:52.230 --> 00:01:55.133 align:middle line:90% can actually happen. 00:01:55.133 --> 00:01:56.050 align:middle line:90% What are some of them. 00:01:56.050 --> 00:01:56.760 align:middle line:90% Any guesses? 00:01:56.760 --> 00:02:00.630 align:middle line:90% 00:02:00.630 --> 00:02:02.710 align:middle line:90% Yeah? 00:02:02.710 --> 00:02:04.750 align:middle line:84% AUDIENCE: Did you lose the proximity of one 00:02:04.750 --> 00:02:06.583 align:middle line:84% pixel to other ones that would be around it? 00:02:06.583 --> 00:02:07.625 align:middle line:90% RAMA RAMAKRISHNAN: Right. 00:02:07.625 --> 00:02:09.490 align:middle line:84% So if you take a particular pixel, 00:02:09.490 --> 00:02:12.790 align:middle line:84% then let's say that the picture shows a t-shirt. 00:02:12.790 --> 00:02:15.810 align:middle line:84% If there's a little pixel in the center of the t-shirt, 00:02:15.810 --> 00:02:17.790 align:middle line:84% knowing that the surrounding pixels are related 00:02:17.790 --> 00:02:19.470 align:middle line:84% to the pixel in a way, because they 00:02:19.470 --> 00:02:22.190 align:middle line:84% are all part of this concept called a t-shirt, 00:02:22.190 --> 00:02:24.950 align:middle line:90% would certainly be helpful. 00:02:24.950 --> 00:02:28.750 align:middle line:84% So to put it more technically, spatial adjacency information 00:02:28.750 --> 00:02:30.150 align:middle line:90% is very important. 00:02:30.150 --> 00:02:33.710 align:middle line:84% And we need to somehow take that into account. 00:02:33.710 --> 00:02:34.210 align:middle line:90% All right. 00:02:34.210 --> 00:02:34.690 align:middle line:90% What else? 00:02:34.690 --> 00:02:36.065 align:middle line:84% What else might be going on here? 00:02:36.065 --> 00:02:40.955 align:middle line:90% 00:02:40.955 --> 00:02:42.830 align:middle line:84% AUDIENCE: Yeah, doing some metadata about it, 00:02:42.830 --> 00:02:44.830 align:middle line:84% like read the dimensions or the resolution 00:02:44.830 --> 00:02:45.872 align:middle line:90% can help you [INAUDIBLE]. 00:02:45.872 --> 00:02:47.080 align:middle line:90% RAMA RAMAKRISHNAN: Oh, I see. 00:02:47.080 --> 00:02:49.590 align:middle line:84% So if you actually had structured data about the image, 00:02:49.590 --> 00:02:52.650 align:middle line:84% such as various characters about, that might be helpful. 00:02:52.650 --> 00:02:53.665 align:middle line:90% True. 00:02:53.665 --> 00:02:55.790 align:middle line:84% But let's just focus on the case where we only have 00:02:55.790 --> 00:02:57.860 align:middle line:90% the raw image and nothing else. 00:02:57.860 --> 00:03:02.500 align:middle line:84% And under that constraint, what else might go wrong 00:03:02.500 --> 00:03:04.320 align:middle line:84% or what else might be suboptimal? 00:03:04.320 --> 00:03:08.700 align:middle line:90% 00:03:08.700 --> 00:03:10.860 align:middle line:84% Well, the first thing that might happen 00:03:10.860 --> 00:03:14.880 align:middle line:84% is that we may have too many parameters. 00:03:14.880 --> 00:03:17.500 align:middle line:90% 00:03:17.500 --> 00:03:20.620 align:middle line:84% So these numbers are from my older iPhone. 00:03:20.620 --> 00:03:23.720 align:middle line:84% I noticed that when I take a color picture with my phone, 00:03:23.720 --> 00:03:28.800 align:middle line:84% it's a 3,000-times-3,000, roughly, grid. 00:03:28.800 --> 00:03:32.960 align:middle line:84% So the picture is actually 3,024 pixels on this axis, 00:03:32.960 --> 00:03:34.940 align:middle line:90% 3,024 on that axis. 00:03:34.940 --> 00:03:39.540 align:middle line:84% So that gets us to roughly 9 million pixels. 00:03:39.540 --> 00:03:41.280 align:middle line:84% But remember, there's a color picture, 00:03:41.280 --> 00:03:43.660 align:middle line:84% which means there are three channels, which 00:03:43.660 --> 00:03:47.380 align:middle line:84% means there are 27 million numbers, each of which 00:03:47.380 --> 00:03:50.580 align:middle line:84% is between 0 and 255 from that little picture. 00:03:50.580 --> 00:03:55.650 align:middle line:84% And now let's say we connect it to a single 100-neuron dense 00:03:55.650 --> 00:03:58.875 align:middle line:84% layer, a single 100-neuron dense layer. 00:03:58.875 --> 00:04:01.250 align:middle line:84% How many parameters are we going to have just in that one 00:04:01.250 --> 00:04:02.375 align:middle line:90% little part of the network? 00:04:02.375 --> 00:04:07.010 align:middle line:90% 00:04:07.010 --> 00:04:10.290 align:middle line:90% Could the mumbling be louder? 00:04:10.290 --> 00:04:12.590 align:middle line:90% Yes, roughly 2.7 billion. 00:04:12.590 --> 00:04:17.070 align:middle line:84% Because 27 million parameters times 100, roughly, of course. 00:04:17.070 --> 00:04:18.829 align:middle line:84% Forget about the biases for a moment. 00:04:18.829 --> 00:04:20.970 align:middle line:90% It's 2.7 billion. 00:04:20.970 --> 00:04:23.490 align:middle line:90% 2.7 billion parameters. 00:04:23.490 --> 00:04:27.210 align:middle line:84% Do you think we can actually get 2.7 billion images 00:04:27.210 --> 00:04:32.530 align:middle line:84% to train any of these things so that we don't overfit? 00:04:32.530 --> 00:04:34.130 align:middle line:90% Too many parameters. 00:04:34.130 --> 00:04:35.510 align:middle line:84% We have to be smarter about this. 00:04:35.510 --> 00:04:36.770 align:middle line:90% It's not going to work. 00:04:36.770 --> 00:04:39.810 align:middle line:90% That's the first problem. 00:04:39.810 --> 00:04:41.870 align:middle line:84% So this clearly is computationally demanding, 00:04:41.870 --> 00:04:46.370 align:middle line:84% very data hungry, and increase the risk of overfitting. 00:04:46.370 --> 00:04:51.610 align:middle line:90% Next, we lose spatial adjacency. 00:04:51.610 --> 00:04:55.450 align:middle line:84% We literally are ignoring what's nearby. 00:04:55.450 --> 00:04:57.330 align:middle line:90% So that's a huge, huge factor. 00:04:57.330 --> 00:05:01.010 align:middle line:84% There is a third factor that we have to worry about, 00:05:01.010 --> 00:05:06.170 align:middle line:84% which is that, let's say that the picture has a vertical line 00:05:06.170 --> 00:05:07.790 align:middle line:90% on the top left side. 00:05:07.790 --> 00:05:12.250 align:middle line:84% And it has some other vertical line on the bottom right side. 00:05:12.250 --> 00:05:15.210 align:middle line:84% What this sort of dumb approach is going to do, 00:05:15.210 --> 00:05:17.730 align:middle line:84% it's going to learn to detect that vertical line 00:05:17.730 --> 00:05:18.770 align:middle line:90% on the top left. 00:05:18.770 --> 00:05:20.610 align:middle line:84% And independent of that, it's going 00:05:20.610 --> 00:05:24.690 align:middle line:84% to learn to detect the vertical line on the bottom right, which 00:05:24.690 --> 00:05:26.570 align:middle line:90% doesn't make any sense. 00:05:26.570 --> 00:05:28.290 align:middle line:84% A vertical line is a vertical line. 00:05:28.290 --> 00:05:31.370 align:middle line:84% So you want to be able to detect it wherever it happens. 00:05:31.370 --> 00:05:33.890 align:middle line:90% Detect once, reuse everywhere. 00:05:33.890 --> 00:05:35.490 align:middle line:90% That's what you need to do. 00:05:35.490 --> 00:05:38.650 align:middle line:84% So this, by the way, is called translation invariance. 00:05:38.650 --> 00:05:41.970 align:middle line:84% Translation is math speak for move stuff around. 00:05:41.970 --> 00:05:43.960 align:middle line:84% You take a line and it moves around. 00:05:43.960 --> 00:05:44.710 align:middle line:90% It doesn't matter. 00:05:44.710 --> 00:05:45.460 align:middle line:90% It's still a line. 00:05:45.460 --> 00:05:47.210 align:middle line:90% Let's figure it out. 00:05:47.210 --> 00:05:50.410 align:middle line:84% So these are the three things we need to worry about. 00:05:50.410 --> 00:05:53.280 align:middle line:84% So we want to learn once and use all over the place. 00:05:53.280 --> 00:05:55.000 align:middle line:84% We want to take spatial adjacency 00:05:55.000 --> 00:05:56.185 align:middle line:90% into account, number two. 00:05:56.185 --> 00:05:57.560 align:middle line:84% And number three, let's just find 00:05:57.560 --> 00:06:00.060 align:middle line:84% a way to make sure that we don't have billions of parameters 00:06:00.060 --> 00:06:02.320 align:middle line:90% for simple toy problems. 00:06:02.320 --> 00:06:05.520 align:middle line:90% Any questions? 00:06:05.520 --> 00:06:07.800 align:middle line:90% Yeah? 00:06:07.800 --> 00:06:10.840 align:middle line:84% AUDIENCE: Is this the problem of just because we 00:06:10.840 --> 00:06:12.840 align:middle line:84% are compressing the image, or that 00:06:12.840 --> 00:06:14.120 align:middle line:90% would've happened nonetheless? 00:06:14.120 --> 00:06:14.940 align:middle line:84% RAMA RAMAKRISHNAN: It would have happened. 00:06:14.940 --> 00:06:16.440 align:middle line:84% So the question was, is it a problem 00:06:16.440 --> 00:06:18.480 align:middle line:84% because we're compressing the image or would 00:06:18.480 --> 00:06:19.800 align:middle line:90% it have happened anyway? 00:06:19.800 --> 00:06:20.880 align:middle line:84% The answer is it would've happened anyway. 00:06:20.880 --> 00:06:22.922 align:middle line:84% You can take any picture, that's going to happen. 00:06:22.922 --> 00:06:25.440 align:middle line:84% Because I'm not making any assumptions 00:06:25.440 --> 00:06:28.240 align:middle line:84% about how the image is coming in to me, whether it's compressed 00:06:28.240 --> 00:06:29.660 align:middle line:90% or not, and so on, and so forth. 00:06:29.660 --> 00:06:32.320 align:middle line:90% 00:06:32.320 --> 00:06:33.600 align:middle line:90% All right. 00:06:33.600 --> 00:06:38.640 align:middle line:84% So convolutional layers were developed to precisely address 00:06:38.640 --> 00:06:41.113 align:middle line:84% these shortcomings, and they're an amazing solution, 00:06:41.113 --> 00:06:41.780 align:middle line:90% as you will see. 00:06:41.780 --> 00:06:42.322 align:middle line:90% Very elegant. 00:06:42.322 --> 00:06:45.040 align:middle line:90% 00:06:45.040 --> 00:06:45.840 align:middle line:90% All right. 00:06:45.840 --> 00:06:49.120 align:middle line:84% So the next, I don't know, half an hour 00:06:49.120 --> 00:06:52.400 align:middle line:84% is going to be me defining a whole bunch of stuff 00:06:52.400 --> 00:06:54.120 align:middle line:84% before we actually get to the fun Colabs, 00:06:54.120 --> 00:06:55.885 align:middle line:90% and so on, and so forth. 00:06:55.885 --> 00:06:57.260 align:middle line:84% So just to put it in perspective, 00:06:57.260 --> 00:07:02.480 align:middle line:84% I have a PowerPoint, two Colabs, and an Excel spreadsheet, 00:07:02.480 --> 00:07:06.080 align:middle line:84% and maybe even a Notability file to cover today. 00:07:06.080 --> 00:07:08.480 align:middle line:84% But hang on for the next 30 minutes 00:07:08.480 --> 00:07:10.880 align:middle line:84% because it's going to be a little concept heavy before we 00:07:10.880 --> 00:07:12.120 align:middle line:90% get to the fun stuff. 00:07:12.120 --> 00:07:12.740 align:middle line:90% Stop me. 00:07:12.740 --> 00:07:15.120 align:middle line:84% Ask me questions because we do have time. 00:07:15.120 --> 00:07:15.740 align:middle line:90% All right. 00:07:15.740 --> 00:07:18.040 align:middle line:84% A convolutional layer is made up of something 00:07:18.040 --> 00:07:20.000 align:middle line:90% called a convolutional filter. 00:07:20.000 --> 00:07:22.200 align:middle line:84% That's the atomic building block. 00:07:22.200 --> 00:07:26.240 align:middle line:84% A convolutional filter is nothing but a small matrix 00:07:26.240 --> 00:07:28.160 align:middle line:90% of numbers, like this. 00:07:28.160 --> 00:07:30.220 align:middle line:84% It's just a small square matrix of numbers. 00:07:30.220 --> 00:07:32.560 align:middle line:90% That's the convolutional filter. 00:07:32.560 --> 00:07:38.160 align:middle line:84% Now a layer is just composed of one or more of these filters. 00:07:38.160 --> 00:07:41.440 align:middle line:90% All right, filters and layers. 00:07:41.440 --> 00:07:44.840 align:middle line:84% Now the thing about the convolutional filter that 00:07:44.840 --> 00:07:47.840 align:middle line:84% makes it really magical is that if you choose 00:07:47.840 --> 00:07:51.230 align:middle line:84% the numbers in a filter carefully and then 00:07:51.230 --> 00:07:52.930 align:middle line:84% you apply the filter to an image, 00:07:52.930 --> 00:07:55.932 align:middle line:84% and I'll get to what I mean by applying the filter. 00:07:55.932 --> 00:07:57.390 align:middle line:84% If you choose the numbers carefully 00:07:57.390 --> 00:08:01.150 align:middle line:84% and you apply it to that image, this little, humble thing 00:08:01.150 --> 00:08:04.750 align:middle line:84% has the ability to detect features in your image. 00:08:04.750 --> 00:08:08.230 align:middle line:84% It can detect lines, curves, gradations in color, 00:08:08.230 --> 00:08:09.910 align:middle line:90% circles, things like that. 00:08:09.910 --> 00:08:11.350 align:middle line:90% It's pretty cool. 00:08:11.350 --> 00:08:13.910 align:middle line:84% And so I'm going to claim and I'm 00:08:13.910 --> 00:08:16.230 align:middle line:84% going to prove shortly that this little humble filter 00:08:16.230 --> 00:08:19.030 align:middle line:84% with the 1's and 0's, it can detect horizontal lines 00:08:19.030 --> 00:08:22.110 align:middle line:90% in any picture you give it. 00:08:22.110 --> 00:08:27.630 align:middle line:84% This thing here has the ability to detect vertical lines. 00:08:27.630 --> 00:08:29.910 align:middle line:84% So I will demonstrate how this thing actually 00:08:29.910 --> 00:08:31.350 align:middle line:90% detects all these things. 00:08:31.350 --> 00:08:33.669 align:middle line:84% And then we will ask the big question that's 00:08:33.669 --> 00:08:35.230 align:middle line:90% probably in your minds already. 00:08:35.230 --> 00:08:37.787 align:middle line:84% Where are we going to get these numbers from? 00:08:37.787 --> 00:08:38.870 align:middle line:90% It all sounds great, Rama. 00:08:38.870 --> 00:08:40.662 align:middle line:84% Where are we going to get the numbers from? 00:08:40.662 --> 00:08:43.870 align:middle line:84% And we have a beautiful answer to that question. 00:08:43.870 --> 00:08:46.110 align:middle line:90% All right, so let's go. 00:08:46.110 --> 00:08:47.710 align:middle line:84% Now I'm going to first explain to you 00:08:47.710 --> 00:08:50.980 align:middle line:84% what I mean by applying a filter to an image. 00:08:50.980 --> 00:08:52.678 align:middle line:84% And then I'm going to give you examples 00:08:52.678 --> 00:08:54.220 align:middle line:84% of how the filter works for detecting 00:08:54.220 --> 00:08:56.680 align:middle line:90% vertical and horizontal lines. 00:08:56.680 --> 00:09:00.900 align:middle line:84% So let's say that this is the image we have. 00:09:00.900 --> 00:09:03.580 align:middle line:84% Again, an image, I assume it's a grayscale image. 00:09:03.580 --> 00:09:06.680 align:middle line:84% So you just have a bunch of numbers between 0 and 255. 00:09:06.680 --> 00:09:08.160 align:middle line:90% So this is the image we have. 00:09:08.160 --> 00:09:09.617 align:middle line:90% It's a little, tiny image. 00:09:09.617 --> 00:09:11.700 align:middle line:84% And this is the filter that's been magically given 00:09:11.700 --> 00:09:13.180 align:middle line:90% to us by somebody. 00:09:13.180 --> 00:09:15.220 align:middle line:84% And what we're trying to do now is to apply it. 00:09:15.220 --> 00:09:17.660 align:middle line:84% So what we do is that we literally 00:09:17.660 --> 00:09:21.780 align:middle line:84% take this filter, the little one, and then 00:09:21.780 --> 00:09:24.780 align:middle line:84% we superimpose it on the top left part of the image. 00:09:24.780 --> 00:09:26.580 align:middle line:90% So you have the image here. 00:09:26.580 --> 00:09:29.180 align:middle line:84% You take this little filter and then you move it to the top 00:09:29.180 --> 00:09:32.100 align:middle line:84% left so that they're right on top of each other. 00:09:32.100 --> 00:09:35.380 align:middle line:84% Once you have it right on top of each other, 00:09:35.380 --> 00:09:37.440 align:middle line:90% you have these matching numbers. 00:09:37.440 --> 00:09:39.560 align:middle line:84% You have three numbers in the image. 00:09:39.560 --> 00:09:41.112 align:middle line:84% There are three numbers in the filter 00:09:41.112 --> 00:09:42.820 align:middle line:84% and they're all matching each other right 00:09:42.820 --> 00:09:43.695 align:middle line:90% on top of each other. 00:09:43.695 --> 00:09:46.420 align:middle line:84% So you have nine pairs of numbers. 00:09:46.420 --> 00:09:48.980 align:middle line:84% And then what we do once we overlay it, 00:09:48.980 --> 00:09:51.660 align:middle line:84% we literally just multiply all the matching numbers 00:09:51.660 --> 00:09:54.193 align:middle line:90% and add them up. 00:09:54.193 --> 00:09:56.360 align:middle line:84% You just multiply all the numbers and match them up. 00:09:56.360 --> 00:09:59.380 align:middle line:84% And you can confirm later on that the arithmetic I'm doing 00:09:59.380 --> 00:10:00.540 align:middle line:90% is actually accurate. 00:10:00.540 --> 00:10:05.540 align:middle line:84% And once you do that, you'll get some number. 00:10:05.540 --> 00:10:09.700 align:middle line:84% And once you get that number, what we do 00:10:09.700 --> 00:10:12.300 align:middle line:84% is we go to our good, old friend, the ReLU, 00:10:12.300 --> 00:10:15.163 align:middle line:84% and then we just run it through ReLU. 00:10:15.163 --> 00:10:16.580 align:middle line:84% Now, in this case, all that effort 00:10:16.580 --> 00:10:18.180 align:middle line:90% came to nothing because it's 0. 00:10:18.180 --> 00:10:19.540 align:middle line:90% That's OK. 00:10:19.540 --> 00:10:23.140 align:middle line:84% So 0, and this number becomes the top left cell 00:10:23.140 --> 00:10:23.870 align:middle line:90% of your output. 00:10:23.870 --> 00:10:26.620 align:middle line:90% 00:10:26.620 --> 00:10:29.798 align:middle line:84% So this is called the convolution operation. 00:10:29.798 --> 00:10:32.340 align:middle line:84% And we won't get into why it's called that, and so on, and so 00:10:32.340 --> 00:10:32.840 align:middle line:90% forth. 00:10:32.840 --> 00:10:35.780 align:middle line:84% There's a long, and rich, and storied history of these things. 00:10:35.780 --> 00:10:38.180 align:middle line:84% But this is the convolution operation. 00:10:38.180 --> 00:10:42.060 align:middle line:84% And once we do that, you can now predict what's going to happen. 00:10:42.060 --> 00:10:44.380 align:middle line:84% We take the same exact operation and we just 00:10:44.380 --> 00:10:46.890 align:middle line:90% move it to the right. 00:10:46.890 --> 00:10:49.410 align:middle line:84% We move this little 3-by-3 thing to the right and repeat 00:10:49.410 --> 00:10:51.450 align:middle line:90% the exact same process. 00:10:51.450 --> 00:10:55.750 align:middle line:84% Matching numbers, multiply all the matching numbers together. 00:10:55.750 --> 00:10:59.370 align:middle line:84% Add them up, run them through a ReLU. 00:10:59.370 --> 00:11:03.730 align:middle line:84% And then boom, you get the second number here. 00:11:03.730 --> 00:11:06.670 align:middle line:84% And you keep doing that until you reach the very end. 00:11:06.670 --> 00:11:08.210 align:middle line:84% You fill up all these numbers, then 00:11:08.210 --> 00:11:12.610 align:middle line:84% you come to the top of the second row. 00:11:12.610 --> 00:11:16.410 align:middle line:84% And you keep on doing that till you reach the very bottom. 00:11:16.410 --> 00:11:22.130 align:middle line:84% So this is what I mean when I say apply a filter to an image. 00:11:22.130 --> 00:11:22.990 align:middle line:90% Any questions? 00:11:22.990 --> 00:11:29.490 align:middle line:90% 00:11:29.490 --> 00:11:31.050 align:middle line:90% Microphone, please. 00:11:31.050 --> 00:11:32.170 align:middle line:90% Microphone. 00:11:32.170 --> 00:11:35.050 align:middle line:90% 00:11:35.050 --> 00:11:39.410 align:middle line:84% AUDIENCE: What happens when they aren't particularly [INAUDIBLE] 00:11:39.410 --> 00:11:43.690 align:middle line:84% the remaining but the filter doesn't perfectly match? 00:11:43.690 --> 00:11:44.690 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:11:44.690 --> 00:11:47.100 align:middle line:84% So you start from the left and then you keep on going. 00:11:47.100 --> 00:11:49.200 align:middle line:84% At some point, the right edge of the filter 00:11:49.200 --> 00:11:52.800 align:middle line:84% is going to match the right edge of the image, and then you stop. 00:11:52.800 --> 00:11:55.360 align:middle line:90% Now there are some nuances here. 00:11:55.360 --> 00:11:59.880 align:middle line:84% So for example, you can actually pad the whole image 00:11:59.880 --> 00:12:03.080 align:middle line:84% on its borders so that you can actually go outside the image 00:12:03.080 --> 00:12:05.560 align:middle line:90% and it'll still work. 00:12:05.560 --> 00:12:06.520 align:middle line:90% Number one. 00:12:06.520 --> 00:12:08.480 align:middle line:90% Number two, nuance. 00:12:08.480 --> 00:12:10.960 align:middle line:84% Instead of just moving one step to the right every time 00:12:10.960 --> 00:12:13.880 align:middle line:84% you finish, you can move two steps to the right. 00:12:13.880 --> 00:12:16.520 align:middle line:84% And that's something called a stride. 00:12:16.520 --> 00:12:19.360 align:middle line:84% So there are a bunch of pesky details here, 00:12:19.360 --> 00:12:22.600 align:middle line:84% but I'm just ignoring them because this basic default 00:12:22.600 --> 00:12:25.860 align:middle line:84% approach works well, amazingly well, almost all the time. 00:12:25.860 --> 00:12:29.140 align:middle line:90% 00:12:29.140 --> 00:12:33.243 align:middle line:84% So that's the mechanics of how this operation works. 00:12:33.243 --> 00:12:35.160 align:middle line:84% Now I'm going to switch to a spreadsheet which 00:12:35.160 --> 00:12:41.340 align:middle line:84% shows this really beautifully, courtesy of the fast.ai people. 00:12:41.340 --> 00:12:43.240 align:middle line:90% So what I'm going to do here-- 00:12:43.240 --> 00:12:44.360 align:middle line:90% this is a big spreadsheet. 00:12:44.360 --> 00:12:46.777 align:middle line:84% I'll upload the spreadsheet after class so you can see it. 00:12:46.777 --> 00:12:48.480 align:middle line:84% So all I have done here, or rather, 00:12:48.480 --> 00:12:51.067 align:middle line:84% all they have done here, thanks to them, 00:12:51.067 --> 00:12:53.400 align:middle line:84% is that they have essentially created a table of numbers 00:12:53.400 --> 00:12:55.360 align:middle line:90% in Excel, as you can tell. 00:12:55.360 --> 00:12:57.500 align:middle line:84% And they have just put some numbers. 00:12:57.500 --> 00:13:00.840 align:middle line:84% Most of the numbers are 0, but some of these numbers 00:13:00.840 --> 00:13:02.060 align:middle line:90% are all more than 0. 00:13:02.060 --> 00:13:03.333 align:middle line:90% They are 0.8, 0.9, and so on. 00:13:03.333 --> 00:13:05.000 align:middle line:84% Basically all they have done is, instead 00:13:05.000 --> 00:13:06.858 align:middle line:84% of working with numbers between 0 and 255, 00:13:06.858 --> 00:13:08.400 align:middle line:84% they're just dividing all the numbers 00:13:08.400 --> 00:13:09.920 align:middle line:90% by 255 so you get fractions. 00:13:09.920 --> 00:13:11.940 align:middle line:84% And they just put the fractions in the table. 00:13:11.940 --> 00:13:16.880 align:middle line:84% And then they have used Excel's very cool conditional formatting 00:13:16.880 --> 00:13:21.620 align:middle line:84% to essentially mark in red all the values that are high. 00:13:21.620 --> 00:13:24.680 align:middle line:84% If the number is closer to 1, the more reddish it gets. 00:13:24.680 --> 00:13:28.560 align:middle line:84% And when you do that, the 3 obviously pops out. 00:13:28.560 --> 00:13:30.160 align:middle line:90% So there is a 3 in the image. 00:13:30.160 --> 00:13:31.040 align:middle line:90% Yes? 00:13:31.040 --> 00:13:32.000 align:middle line:90% OK, good. 00:13:32.000 --> 00:13:35.080 align:middle line:84% So now what we're going to do is we're 00:13:35.080 --> 00:13:37.880 align:middle line:84% going to move to our little filter here. 00:13:37.880 --> 00:13:40.020 align:middle line:90% You can see the filter. 00:13:40.020 --> 00:13:42.080 align:middle line:84% And I'm claiming this detects horizontal lines. 00:13:42.080 --> 00:13:47.130 align:middle line:90% 00:13:47.130 --> 00:13:47.630 align:middle line:90% Sorry. 00:13:47.630 --> 00:13:51.350 align:middle line:90% 00:13:51.350 --> 00:13:54.710 align:middle line:84% This table here is the result of applying that filter 00:13:54.710 --> 00:13:57.790 align:middle line:84% to that tree And you can see here, 00:13:57.790 --> 00:13:59.400 align:middle line:84% I'm looking at the top left cell here. 00:13:59.400 --> 00:14:03.790 align:middle line:90% 00:14:03.790 --> 00:14:05.010 align:middle line:90% Look at this top left cell. 00:14:05.010 --> 00:14:08.670 align:middle line:84% The formula is nothing more than multiply all those things 00:14:08.670 --> 00:14:09.630 align:middle line:90% and add them up. 00:14:09.630 --> 00:14:13.010 align:middle line:84% And then once you add it up, run it through a max of 0 comma 00:14:13.010 --> 00:14:15.990 align:middle line:90% that, which is just the ReLU. 00:14:15.990 --> 00:14:18.110 align:middle line:90% Basic arithmetic. 00:14:18.110 --> 00:14:19.470 align:middle line:90% So we do that. 00:14:19.470 --> 00:14:21.030 align:middle line:90% And this is the output. 00:14:21.030 --> 00:14:23.830 align:middle line:84% And the output is also conditionally formatted 00:14:23.830 --> 00:14:26.670 align:middle line:84% to show you where things are lighting up. 00:14:26.670 --> 00:14:31.710 align:middle line:84% And you can see, only the horizontal lines of the 3 00:14:31.710 --> 00:14:34.430 align:middle line:90% are lighting up. 00:14:34.430 --> 00:14:36.750 align:middle line:90% Everyone see that? 00:14:36.750 --> 00:14:38.730 align:middle line:90% So now you understand. 00:14:38.730 --> 00:14:43.270 align:middle line:84% The filter, in fact, is living up to the claim I made for it. 00:14:43.270 --> 00:14:46.250 align:middle line:84% Similarly, if you look at what's going on here, 00:14:46.250 --> 00:14:48.350 align:middle line:84% this is a vertical filter, the same thing. 00:14:48.350 --> 00:14:53.510 align:middle line:84% You apply it, only the vertical line is lighting up. 00:14:53.510 --> 00:14:58.470 align:middle line:84% Now what you can do is, I would encourage you to do this 00:14:58.470 --> 00:15:01.110 align:middle line:84% after class, is you can look at all these numbers 00:15:01.110 --> 00:15:02.490 align:middle line:90% here, for example. 00:15:02.490 --> 00:15:06.510 align:middle line:84% And then ask yourself, OK, why is that lighting up? 00:15:06.510 --> 00:15:09.830 align:middle line:84% And you will discover that what's actually going on 00:15:09.830 --> 00:15:12.470 align:middle line:90% is that it's looking for edges. 00:15:12.470 --> 00:15:16.150 align:middle line:84% It's looking for rows in the table 00:15:16.150 --> 00:15:18.990 align:middle line:84% where there is some non-zero thing in the first row 00:15:18.990 --> 00:15:21.750 align:middle line:90% and zeros in the second row. 00:15:21.750 --> 00:15:23.650 align:middle line:84% And by choosing the numbers carefully, 00:15:23.650 --> 00:15:25.930 align:middle line:84% you multiply the ones with positive numbers 00:15:25.930 --> 00:15:28.072 align:middle line:84% and you multiply the zeros with zeros. 00:15:28.072 --> 00:15:30.030 align:middle line:84% And then you'll come up with a positive number, 00:15:30.030 --> 00:15:32.910 align:middle line:90% and thereby you detect an edge. 00:15:32.910 --> 00:15:37.340 align:middle line:84% So what I would encourage you to do is use this Excel thing here. 00:15:37.340 --> 00:15:40.050 align:middle line:90% 00:15:40.050 --> 00:15:42.140 align:middle line:90% So here is a cell we have. 00:15:42.140 --> 00:15:51.380 align:middle line:90% So let's trace its incidence. 00:15:51.380 --> 00:15:57.340 align:middle line:84% So you can see here, these numbers, this 00:15:57.340 --> 00:15:59.440 align:middle line:90% is what it's processing. 00:15:59.440 --> 00:16:01.540 align:middle line:84% That is a grid that's being processing to come up 00:16:01.540 --> 00:16:02.840 align:middle line:90% with that big number. 00:16:02.840 --> 00:16:06.980 align:middle line:84% And you can see here in this grid, these numbers are here. 00:16:06.980 --> 00:16:10.620 align:middle line:84% And then these numbers are a lot lower than these numbers 00:16:10.620 --> 00:16:13.513 align:middle line:90% because there is an edge. 00:16:13.513 --> 00:16:14.680 align:middle line:90% The numbers are a lot lower. 00:16:14.680 --> 00:16:17.820 align:middle line:84% That's why you can see the horizontal part of the 3. 00:16:17.820 --> 00:16:21.180 align:middle line:84% And so what this filter is doing, it's basically saying, 00:16:21.180 --> 00:16:24.980 align:middle line:84% well, the row that I'm catching here has the 1's. 00:16:24.980 --> 00:16:25.960 align:middle line:90% The middle has 0's. 00:16:25.960 --> 00:16:27.740 align:middle line:90% The rest are all minus 1's. 00:16:27.740 --> 00:16:31.460 align:middle line:84% So the small values are going to get very small. 00:16:31.460 --> 00:16:33.240 align:middle line:84% The big values are going to get very big, 00:16:33.240 --> 00:16:35.580 align:middle line:84% and the overall thing is going to be emphasized. 00:16:35.580 --> 00:16:38.380 align:middle line:84% So that's the basic idea of edge detection. 00:16:38.380 --> 00:16:40.110 align:middle line:84% Spend some time with it, with Excel, 00:16:40.110 --> 00:16:43.130 align:middle line:84% and it will become clear to you what I'm talking about here. 00:16:43.130 --> 00:16:44.210 align:middle line:90% All right, cool. 00:16:44.210 --> 00:16:46.710 align:middle line:90% So that's that. 00:16:46.710 --> 00:16:48.450 align:middle line:90% By the way, I also have-- 00:16:48.450 --> 00:16:51.850 align:middle line:84% there is a very cool site here in which you can actually 00:16:51.850 --> 00:16:53.490 align:middle line:84% go in, and punch in your own numbers, 00:16:53.490 --> 00:16:55.432 align:middle line:90% and see what it detects. 00:16:55.432 --> 00:16:57.390 align:middle line:84% A lot of edges, and curves, and this, and that. 00:16:57.390 --> 00:17:00.090 align:middle line:84% It's very cool so I encourage you to try it out. 00:17:00.090 --> 00:17:02.330 align:middle line:84% So the key thing here I want to say 00:17:02.330 --> 00:17:08.210 align:middle line:84% is by choosing the numbers in a filter 00:17:08.210 --> 00:17:10.310 align:middle line:84% carefully and applying this operation, 00:17:10.310 --> 00:17:13.730 align:middle line:84% different features can be detected. 00:17:13.730 --> 00:17:17.130 align:middle line:84% Now I mentioned earlier that a convolution layer is composed 00:17:17.130 --> 00:17:21.490 align:middle line:84% of one or more of these filters, so one or more of these filters. 00:17:21.490 --> 00:17:25.569 align:middle line:84% And so you can think of each filter as a specialist 00:17:25.569 --> 00:17:28.650 align:middle line:90% for a particular feature. 00:17:28.650 --> 00:17:29.750 align:middle line:90% So it's a specialist. 00:17:29.750 --> 00:17:31.410 align:middle line:84% Maybe it specializes in detecting 00:17:31.410 --> 00:17:34.863 align:middle line:84% vertical lines, horizontal lines, semicircles, 00:17:34.863 --> 00:17:35.530 align:middle line:90% quarter circles. 00:17:35.530 --> 00:17:36.790 align:middle line:90% You don't know. 00:17:36.790 --> 00:17:39.850 align:middle line:84% You can imagine each of them as being specialists. 00:17:39.850 --> 00:17:43.110 align:middle line:84% And given that modern images could be very complicated, 00:17:43.110 --> 00:17:45.310 align:middle line:84% they may have lots of interesting features going on, 00:17:45.310 --> 00:17:49.050 align:middle line:84% you probably want to have lots of these filters. 00:17:49.050 --> 00:17:54.050 align:middle line:84% But the key is that you don't have to decide upfront. 00:17:54.050 --> 00:17:56.250 align:middle line:84% Hey, you, filter, you better specialize 00:17:56.250 --> 00:17:57.590 align:middle line:90% in detecting vertical lines. 00:17:57.590 --> 00:17:59.150 align:middle line:84% And you, on the other hand, do not. 00:17:59.150 --> 00:18:00.030 align:middle line:90% Stay in your lane. 00:18:00.030 --> 00:18:01.030 align:middle line:90% Do vertical lines. 00:18:01.030 --> 00:18:02.530 align:middle line:90% You're not going to do that. 00:18:02.530 --> 00:18:07.030 align:middle line:84% You will let the system figure out what it wants to figure out. 00:18:07.030 --> 00:18:10.170 align:middle line:84% So there is no human bottleneck in doing this. 00:18:10.170 --> 00:18:11.810 align:middle line:84% And I mention this because there used 00:18:11.810 --> 00:18:15.000 align:middle line:84% to be a human bottleneck before deep learning happened. 00:18:15.000 --> 00:18:17.570 align:middle line:90% 00:18:17.570 --> 00:18:19.298 align:middle line:84% Now let's just make sure we understand 00:18:19.298 --> 00:18:21.090 align:middle line:84% the mechanics of what happens when you have 00:18:21.090 --> 00:18:22.850 align:middle line:90% two of these filters, not one. 00:18:22.850 --> 00:18:25.170 align:middle line:84% So this is the input image as before. 00:18:25.170 --> 00:18:26.890 align:middle line:84% This is the filter we saw earlier. 00:18:26.890 --> 00:18:29.238 align:middle line:84% And this is another filter we have. 00:18:29.238 --> 00:18:31.030 align:middle line:84% The thing is, we just run them in parallel. 00:18:31.030 --> 00:18:32.570 align:middle line:84% We take each filter, do the operation, 00:18:32.570 --> 00:18:33.430 align:middle line:90% come up with an output. 00:18:33.430 --> 00:18:35.388 align:middle line:84% Take the other filter, do the operation come up 00:18:35.388 --> 00:18:36.130 align:middle line:90% with its output. 00:18:36.130 --> 00:18:38.463 align:middle line:84% And then when you do that, the first one gives you that, 00:18:38.463 --> 00:18:39.880 align:middle line:90% the second one gives you that. 00:18:39.880 --> 00:18:42.460 align:middle line:84% And this output is a table of some-- 00:18:42.460 --> 00:18:44.820 align:middle line:90% it's actually not a table. 00:18:44.820 --> 00:18:45.500 align:middle line:90% What is it? 00:18:45.500 --> 00:18:49.320 align:middle line:90% 00:18:49.320 --> 00:18:51.400 align:middle line:90% Louder, please. 00:18:51.400 --> 00:18:52.300 align:middle line:90% It's a tensor. 00:18:52.300 --> 00:18:52.860 align:middle line:90% Thank you. 00:18:52.860 --> 00:18:54.040 align:middle line:90% It's a tensor. 00:18:54.040 --> 00:18:57.320 align:middle line:84% And so these two 5-by-5 matrices can be represented as a tensor 00:18:57.320 --> 00:18:57.910 align:middle line:90% of what shape? 00:18:57.910 --> 00:19:01.907 align:middle line:90% 00:19:01.907 --> 00:19:03.240 align:middle line:90% And there are two right answers. 00:19:03.240 --> 00:19:04.710 align:middle line:90% AUDIENCE: 5 by 5. 00:19:04.710 --> 00:19:05.880 align:middle line:90% RAMA RAMAKRISHNAN: 5 by 5? 00:19:05.880 --> 00:19:06.440 align:middle line:90% AUDIENCE: Into 2. 00:19:06.440 --> 00:19:07.523 align:middle line:90% RAMA RAMAKRISHNAN: Into 2. 00:19:07.523 --> 00:19:08.120 align:middle line:90% Correct. 00:19:08.120 --> 00:19:11.720 align:middle line:84% So you can either think of it as 5 by 5 times 2, or 2 times 5 00:19:11.720 --> 00:19:12.480 align:middle line:90% by 5. 00:19:12.480 --> 00:19:14.377 align:middle line:90% They're both fine. 00:19:14.377 --> 00:19:15.960 align:middle line:84% Which one you go with actually ends up 00:19:15.960 --> 00:19:18.720 align:middle line:90% being a matter of convention. 00:19:18.720 --> 00:19:22.720 align:middle line:84% So now you begin to see why we care about tensors. 00:19:22.720 --> 00:19:27.960 align:middle line:84% Imagine if, instead of having 2 filters, we have 103 filters. 00:19:27.960 --> 00:19:33.560 align:middle line:84% The resulting tensor is going to be 5 by 5 by 103. 00:19:33.560 --> 00:19:36.530 align:middle line:90% OK, good. 00:19:36.530 --> 00:19:41.110 align:middle line:84% Now let's look at the slightly more complex situation, 00:19:41.110 --> 00:19:44.010 align:middle line:84% where you have not a black and white image, 00:19:44.010 --> 00:19:46.170 align:middle line:84% a grayscale image with just a little table, 00:19:46.170 --> 00:19:49.670 align:middle line:90% but an actual color image. 00:19:49.670 --> 00:19:54.030 align:middle line:84% So we know how to apply a filter to a 2D tensor, like this, 00:19:54.030 --> 00:19:55.230 align:middle line:90% and to get that. 00:19:55.230 --> 00:19:58.890 align:middle line:84% But let's say we have something like this, where it has three. 00:19:58.890 --> 00:20:01.650 align:middle line:84% It's got three channels, red, blue, green, RGB. 00:20:01.650 --> 00:20:03.510 align:middle line:84% It's got three tables of numbers. 00:20:03.510 --> 00:20:07.470 align:middle line:84% So this is a tensor of shape 6 times 6 times 3, let's say. 00:20:07.470 --> 00:20:10.830 align:middle line:84% And you want to apply this 3-by-3 filter, just like before, 00:20:10.830 --> 00:20:11.490 align:middle line:90% to this thing. 00:20:11.490 --> 00:20:13.550 align:middle line:84% You want to apply the convolution operation. 00:20:13.550 --> 00:20:14.810 align:middle line:90% How is that going to work? 00:20:14.810 --> 00:20:18.430 align:middle line:90% 00:20:18.430 --> 00:20:21.750 align:middle line:90% Do we just apply this to each? 00:20:21.750 --> 00:20:25.210 align:middle line:84% We first apply it to the red, then we apply it to the green, 00:20:25.210 --> 00:20:26.790 align:middle line:90% then we apply it to the blue. 00:20:26.790 --> 00:20:33.080 align:middle line:84% Should we do that, or is there a problem with that approach? 00:20:33.080 --> 00:20:36.070 align:middle line:90% 00:20:36.070 --> 00:20:37.492 align:middle line:90% Yeah? 00:20:37.492 --> 00:20:40.030 align:middle line:90% AUDIENCE: The problem-- 00:20:40.030 --> 00:20:42.505 align:middle line:84% RAMA RAMAKRISHNAN: Could you use the microphone, please? 00:20:42.505 --> 00:20:44.130 align:middle line:84% AUDIENCE: The problem with the approach 00:20:44.130 --> 00:20:46.130 align:middle line:84% would be the same as what you said earlier, 00:20:46.130 --> 00:20:47.250 align:middle line:90% that it wouldn't learn. 00:20:47.250 --> 00:20:50.430 align:middle line:84% The lines are probably the same in each channel. 00:20:50.430 --> 00:20:53.352 align:middle line:84% The location of the lines are probably the same each channel. 00:20:53.352 --> 00:20:54.310 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 00:20:54.310 --> 00:20:55.685 align:middle line:84% The location of the line is going 00:20:55.685 --> 00:20:58.470 align:middle line:84% to be the same thing because that line, if you will, 00:20:58.470 --> 00:21:01.110 align:middle line:84% is the aggregation of information from the three 00:21:01.110 --> 00:21:02.090 align:middle line:90% different channels. 00:21:02.090 --> 00:21:02.590 align:middle line:90% Right. 00:21:02.590 --> 00:21:07.190 align:middle line:84% But the problem here is slightly different, 00:21:07.190 --> 00:21:12.030 align:middle line:84% which is that if you do them independently, 00:21:12.030 --> 00:21:15.030 align:middle line:84% the network has not been informed 00:21:15.030 --> 00:21:19.737 align:middle line:84% that these things are all part of the same underlying concept. 00:21:19.737 --> 00:21:21.570 align:middle line:84% As far as concerned, it's just three things. 00:21:21.570 --> 00:21:23.487 align:middle line:84% It's just going to process them independently. 00:21:23.487 --> 00:21:26.750 align:middle line:84% So we need to somehow change the filter so that it understands, 00:21:26.750 --> 00:21:29.950 align:middle line:84% what is at this pixel location, the three numbers under it? 00:21:29.950 --> 00:21:30.770 align:middle line:90% RGB. 00:21:30.770 --> 00:21:35.140 align:middle line:84% They are actually part of the same thing, underlying thing. 00:21:35.140 --> 00:21:38.020 align:middle line:84% So what we do is actually very simple. 00:21:38.020 --> 00:21:42.580 align:middle line:84% We just take this filter and make it 3D. 00:21:42.580 --> 00:21:43.900 align:middle line:90% So we take this filter. 00:21:43.900 --> 00:21:45.440 align:middle line:84% Instead of having just one of them, 00:21:45.440 --> 00:21:49.220 align:middle line:84% we just make it a cube, like that, three times. 00:21:49.220 --> 00:21:53.260 align:middle line:84% And once we do that, you can imagine taking this thing here 00:21:53.260 --> 00:21:56.380 align:middle line:90% and essentially doing that. 00:21:56.380 --> 00:22:00.460 align:middle line:84% Now, instead of having 9 numbers in the image and 9 numbers 00:22:00.460 --> 00:22:03.060 align:middle line:84% in the filter, you have 27 numbers 00:22:03.060 --> 00:22:05.620 align:middle line:84% in the image, 27 numbers in the filter. 00:22:05.620 --> 00:22:08.268 align:middle line:84% But you still match them up, multiply them, 00:22:08.268 --> 00:22:09.810 align:middle line:84% add them up, run them through a ReLU. 00:22:09.810 --> 00:22:14.780 align:middle line:90% 00:22:14.780 --> 00:22:16.540 align:middle line:84% By the way, I tried to get ChatGPT 00:22:16.540 --> 00:22:19.092 align:middle line:90% to give me a picture like that. 00:22:19.092 --> 00:22:21.565 align:middle line:90% It just completely bombed. 00:22:21.565 --> 00:22:23.440 align:middle line:84% I tried three, four, five different variants. 00:22:23.440 --> 00:22:24.200 align:middle line:90% It just gave up. 00:22:24.200 --> 00:22:26.700 align:middle line:84% Then I found this nice picture in DeepLearning.ai 00:22:26.700 --> 00:22:28.623 align:middle line:90% and I used it. 00:22:28.623 --> 00:22:30.540 align:middle line:84% AUDIENCE: So then if you put different numbers 00:22:30.540 --> 00:22:33.120 align:middle line:84% in each of the layers of that color processing, 00:22:33.120 --> 00:22:35.245 align:middle line:84% would you get a different thing to green and blue? 00:22:35.245 --> 00:22:37.120 align:middle line:84% RAMA RAMAKRISHNAN: I'm sorry, say that again. 00:22:37.120 --> 00:22:38.740 align:middle line:84% AUDIENCE: If you put different numbers 00:22:38.740 --> 00:22:40.820 align:middle line:90% in each of the layers of your-- 00:22:40.820 --> 00:22:43.420 align:middle line:84% not layers, in each of the different depth dimensions 00:22:43.420 --> 00:22:45.260 align:middle line:84% of your convolution filter, would that 00:22:45.260 --> 00:22:47.360 align:middle line:90% be like color processing? 00:22:47.360 --> 00:22:48.360 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:22:48.360 --> 00:22:49.300 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:22:49.300 --> 00:22:49.540 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:22:49.540 --> 00:22:50.832 align:middle line:90% You will put different numbers. 00:22:50.832 --> 00:22:52.383 align:middle line:84% In fact, you have 27 numbers now. 00:22:52.383 --> 00:22:54.300 align:middle line:84% But we haven't gotten to the question of where 00:22:54.300 --> 00:22:55.640 align:middle line:90% these numbers are coming from. 00:22:55.640 --> 00:22:58.780 align:middle line:84% So just hold the thought till we get there. 00:22:58.780 --> 00:23:03.340 align:middle line:90% So any questions on this? 00:23:03.340 --> 00:23:05.740 align:middle line:84% You literally take the 2D thing and make it 3D. 00:23:05.740 --> 00:23:08.740 align:middle line:84% You basically give it depth, and the depth just 00:23:08.740 --> 00:23:11.300 align:middle line:90% matches the depth of the input. 00:23:11.300 --> 00:23:15.560 align:middle line:84% So if the input is 10 deep, your filter is going to get 10 deep. 00:23:15.560 --> 00:23:18.260 align:middle line:90% 00:23:18.260 --> 00:23:20.100 align:middle line:90% OK? 00:23:20.100 --> 00:23:22.740 align:middle line:90% Yes? 00:23:22.740 --> 00:23:26.140 align:middle line:84% AUDIENCE: Rather than increasing the rank order of the tensor 00:23:26.140 --> 00:23:27.620 align:middle line:84% by one, is there any instance where 00:23:27.620 --> 00:23:29.828 align:middle line:84% you would create an abstraction layer where you would 00:23:29.828 --> 00:23:31.930 align:middle line:84% run an operation across the different layers 00:23:31.930 --> 00:23:34.770 align:middle line:84% to come up with a intermediary layer 00:23:34.770 --> 00:23:38.210 align:middle line:84% that you would run a lower rank tensor or filter upper? 00:23:38.210 --> 00:23:39.210 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:23:39.210 --> 00:23:41.970 align:middle line:84% So there is a lot of stuff in the research literature 00:23:41.970 --> 00:23:43.970 align:middle line:84% which tries to do things like that. 00:23:43.970 --> 00:23:49.090 align:middle line:84% I'm just describing the most basic approach to doing this. 00:23:49.090 --> 00:23:53.450 align:middle line:84% And as it turns out, this basic approach is extremely powerful. 00:23:53.450 --> 00:23:56.890 align:middle line:84% And of course, researchers try to go from the 95th percent 00:23:56.890 --> 00:23:59.510 align:middle line:90% thing to the 95.1%. 00:23:59.510 --> 00:24:02.250 align:middle line:84% So they invent all sorts of crazy, complicated stuff, which 00:24:02.250 --> 00:24:03.958 align:middle line:90% is all good for us, humanity. 00:24:03.958 --> 00:24:05.750 align:middle line:84% But for practical use, this is good enough. 00:24:05.750 --> 00:24:08.330 align:middle line:90% 00:24:08.330 --> 00:24:11.310 align:middle line:84% AUDIENCE: How do you convert the 3-by-3 layer into a single 00:24:11.310 --> 00:24:12.050 align:middle line:90% 4-by-4 matrix? 00:24:12.050 --> 00:24:15.170 align:middle line:84% [INAUDIBLE], what with the three layers part of it? 00:24:15.170 --> 00:24:16.170 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:24:16.170 --> 00:24:17.430 align:middle line:90% So we are coming to that. 00:24:17.430 --> 00:24:18.678 align:middle line:90% I think we have a slide here. 00:24:18.678 --> 00:24:19.470 align:middle line:90% Actually, we don't. 00:24:19.470 --> 00:24:19.970 align:middle line:90% Never mind. 00:24:19.970 --> 00:24:21.010 align:middle line:90% I'll answer that. 00:24:21.010 --> 00:24:23.670 align:middle line:90% So here you have one filter. 00:24:23.670 --> 00:24:27.450 align:middle line:84% You have one 3-by-3-by-3 filter, which plugs into this thing 00:24:27.450 --> 00:24:28.030 align:middle line:90% here. 00:24:28.030 --> 00:24:31.000 align:middle line:84% And then it gives you the 4 by 4 at the end. 00:24:31.000 --> 00:24:37.160 align:middle line:84% So for one filter, we know that by doing this operation, 00:24:37.160 --> 00:24:38.720 align:middle line:90% we get this 4 by 4. 00:24:38.720 --> 00:24:41.840 align:middle line:84% Let's say that you have another filter which is also 3D. 00:24:41.840 --> 00:24:45.080 align:middle line:84% You do that thing, you'll get another 4 by 4. 00:24:45.080 --> 00:24:47.700 align:middle line:84% And if you have 10 filters, you'll get 10 of these 4 by 4s, 00:24:47.700 --> 00:24:50.550 align:middle line:84% which then gets packaged up into a 4-by-4-by-10 tensor. 00:24:50.550 --> 00:24:54.560 align:middle line:90% 00:24:54.560 --> 00:24:57.880 align:middle line:84% Remember, whether it's 2D, 3D, 10D, 00:24:57.880 --> 00:25:02.080 align:middle line:90% what is coming out is always 2D. 00:25:02.080 --> 00:25:04.360 align:middle line:84% Because ultimately when you apply all this operation 00:25:04.360 --> 00:25:06.640 align:middle line:84% at each position, you just have one number. 00:25:06.640 --> 00:25:08.440 align:middle line:84% And then ultimately you just do all those things, you just 00:25:08.440 --> 00:25:10.065 align:middle line:84% come up with a table of numbers always. 00:25:10.065 --> 00:25:14.240 align:middle line:84% So what's coming out is always a 2D number table like that. 00:25:14.240 --> 00:25:16.340 align:middle line:84% But when you have lots of filters, 00:25:16.340 --> 00:25:18.780 align:middle line:84% you have lots of these 2D tables, one after the other, 00:25:18.780 --> 00:25:21.290 align:middle line:84% and therefore, they get packaged up into a tensor. 00:25:21.290 --> 00:25:27.040 align:middle line:90% 00:25:27.040 --> 00:25:30.080 align:middle line:84% So textbook Chapter 8.1 has a lot 00:25:30.080 --> 00:25:32.820 align:middle line:84% of detail on intuition, which I think is really good. 00:25:32.820 --> 00:25:35.120 align:middle line:90% So please try it out. 00:25:35.120 --> 00:25:39.040 align:middle line:84% And folks, by the way, this convolutional stuff, 00:25:39.040 --> 00:25:41.080 align:middle line:90% it grows in the telling. 00:25:41.080 --> 00:25:42.780 align:middle line:84% So I would encourage you to revisit it, 00:25:42.780 --> 00:25:45.600 align:middle line:84% revisit it a few times, and then it slowly 00:25:45.600 --> 00:25:48.023 align:middle line:84% becomes part of your muscle memory. 00:25:48.023 --> 00:25:49.440 align:middle line:84% So don't expect to just understand 00:25:49.440 --> 00:25:51.600 align:middle line:90% all the nuances like one shot. 00:25:51.600 --> 00:25:53.160 align:middle line:90% Do it a few times. 00:25:53.160 --> 00:25:55.700 align:middle line:84% It'll become wired into your head. 00:25:55.700 --> 00:25:58.240 align:middle line:90% 00:25:58.240 --> 00:26:00.615 align:middle line:84% The big question-- these seem excellent, but how are we 00:26:00.615 --> 00:26:02.240 align:middle line:84% supposed to come up with these numbers? 00:26:02.240 --> 00:26:05.240 align:middle line:84% In fact, traditionally these filters actually 00:26:05.240 --> 00:26:07.185 align:middle line:90% used to be designed by hand. 00:26:07.185 --> 00:26:08.560 align:middle line:84% Computer vision researchers would 00:26:08.560 --> 00:26:10.360 align:middle line:84% invest prodigious amounts of time, 00:26:10.360 --> 00:26:15.440 align:middle line:84% and effort, and talent to figure out the right kinds of filters 00:26:15.440 --> 00:26:17.860 align:middle line:84% to use for various specific applications. 00:26:17.860 --> 00:26:19.840 align:middle line:84% So if you wanted to build an application which 00:26:19.840 --> 00:26:22.700 align:middle line:84% would look at, say, MRI images and figure out, OK, 00:26:22.700 --> 00:26:25.960 align:middle line:84% what kind of features should I extract from this MRI thing 00:26:25.960 --> 00:26:29.430 align:middle line:84% to be able to predict the evidence for a stroke? 00:26:29.430 --> 00:26:32.483 align:middle line:84% They would actually hand design the filter, 00:26:32.483 --> 00:26:34.650 align:middle line:84% try lots of different values, and then come up with, 00:26:34.650 --> 00:26:36.850 align:middle line:84% I got the perfect filter for this thing here. 00:26:36.850 --> 00:26:39.010 align:middle line:84% So that's the way it used to be done. 00:26:39.010 --> 00:26:41.733 align:middle line:90% 00:26:41.733 --> 00:26:44.150 align:middle line:84% But as we figured out how to train deep networks with lots 00:26:44.150 --> 00:26:46.390 align:middle line:84% of parameters, we figured out things 00:26:46.390 --> 00:26:50.030 align:middle line:84% like ReLU activation, stochastic gradient descent, GPUs, 00:26:50.030 --> 00:26:54.255 align:middle line:84% backprop, things like that, this big idea emerged. 00:26:54.255 --> 00:26:55.630 align:middle line:84% Why don't we think of the numbers 00:26:55.630 --> 00:26:57.725 align:middle line:90% in the filter as just weights? 00:26:57.725 --> 00:26:59.350 align:middle line:84% And why don't we just simply learn them 00:26:59.350 --> 00:27:02.363 align:middle line:84% from the data using backprop, just like we 00:27:02.363 --> 00:27:03.530 align:middle line:90% learn all the other weights? 00:27:03.530 --> 00:27:06.550 align:middle line:90% What's the big deal? 00:27:06.550 --> 00:27:09.470 align:middle line:84% This simple idea, and it feels a bit, I don't know, 00:27:09.470 --> 00:27:12.070 align:middle line:90% blindingly obvious in hindsight. 00:27:12.070 --> 00:27:14.990 align:middle line:84% I'm sure it was not obvious in foresight. 00:27:14.990 --> 00:27:16.550 align:middle line:90% This was the breakthrough. 00:27:16.550 --> 00:27:18.190 align:middle line:90% This was the key breakthrough. 00:27:18.190 --> 00:27:20.390 align:middle line:84% And now it's actually possible to do this 00:27:20.390 --> 00:27:23.590 align:middle line:84% because a convolutional filter that we've seen 00:27:23.590 --> 00:27:26.100 align:middle line:90% is actually just a neuron. 00:27:26.100 --> 00:27:29.540 align:middle line:84% The underlying arithmetic of it is just a neuronal arithmetic. 00:27:29.540 --> 00:27:33.560 align:middle line:84% And so it just happens to be a slightly special one. 00:27:33.560 --> 00:27:36.260 align:middle line:84% It's actually even simpler than a regular neuron. 00:27:36.260 --> 00:27:38.637 align:middle line:84% And in the interest of time, I have one or two 00:27:38.637 --> 00:27:40.220 align:middle line:84% slides in the appendix which tells you 00:27:40.220 --> 00:27:42.740 align:middle line:84% exactly why it's a neuron, so check it out. 00:27:42.740 --> 00:27:44.120 align:middle line:90% But just take my word for it. 00:27:44.120 --> 00:27:45.940 align:middle line:84% It's just a particular kind of neuron. 00:27:45.940 --> 00:27:48.060 align:middle line:84% And because it's a particular kind of neuron, 00:27:48.060 --> 00:27:50.660 align:middle line:84% and we know how to work with neurons, 00:27:50.660 --> 00:27:52.220 align:middle line:84% we know how to work with neurons, 00:27:52.220 --> 00:27:56.180 align:middle line:84% which means that our entire machinery, layers, 00:27:56.180 --> 00:27:58.920 align:middle line:84% loss functions, gradient descent, SGD, blah, blah, 00:27:58.920 --> 00:28:01.300 align:middle line:84% everything is immediately applicable. 00:28:01.300 --> 00:28:04.210 align:middle line:84% We don't have to invent any new stuff to make it work. 00:28:04.210 --> 00:28:09.760 align:middle line:90% 00:28:09.760 --> 00:28:11.260 align:middle line:84% AUDIENCE: Do you initialize the data 00:28:11.260 --> 00:28:14.220 align:middle line:84% basically between applications or just 00:28:14.220 --> 00:28:16.280 align:middle line:84% because the network has different sizes? 00:28:16.280 --> 00:28:19.280 align:middle line:84% Does computer vision versus [INAUDIBLE] do it differently, 00:28:19.280 --> 00:28:21.060 align:middle line:84% or it's just because the network has 00:28:21.060 --> 00:28:22.420 align:middle line:90% different numbers of neurons? 00:28:22.420 --> 00:28:23.420 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:28:23.420 --> 00:28:27.237 align:middle line:84% So the initialization, it's a good question. 00:28:27.237 --> 00:28:29.820 align:middle line:84% Let's come back to it when we get to something called transfer 00:28:29.820 --> 00:28:32.450 align:middle line:84% learning, which I'm going to get to by about 9:30. 00:28:32.450 --> 00:28:35.400 align:middle line:90% 00:28:35.400 --> 00:28:36.558 align:middle line:90% So that's it. 00:28:36.558 --> 00:28:39.100 align:middle line:84% So this turned out to be a huge turning point in the computer 00:28:39.100 --> 00:28:39.940 align:middle line:90% vision field. 00:28:39.940 --> 00:28:43.100 align:middle line:84% And this was the massive unlock in the year 2012. 00:28:43.100 --> 00:28:45.780 align:middle line:84% This computer vision system that used this technology called 00:28:45.780 --> 00:28:48.140 align:middle line:84% AlexNet burst out onto the world stage 00:28:48.140 --> 00:28:51.620 align:middle line:84% because it crushed the competition in a competition 00:28:51.620 --> 00:28:52.740 align:middle line:90% called ImageNet. 00:28:52.740 --> 00:28:56.900 align:middle line:84% And the previous best score was 26% error rate. 00:28:56.900 --> 00:29:00.008 align:middle line:84% And this thing came in and had 16% error rate. 00:29:00.008 --> 00:29:01.800 align:middle line:84% It's the kind of thing where if you see it, 00:29:01.800 --> 00:29:03.467 align:middle line:84% you'll be like, oh, this must be a typo. 00:29:03.467 --> 00:29:05.900 align:middle line:84% Because every year the improvements 00:29:05.900 --> 00:29:07.960 align:middle line:90% are very little, 0.5%, 1%. 00:29:07.960 --> 00:29:09.642 align:middle line:84% And then this year was 10%, and that 00:29:09.642 --> 00:29:10.850 align:middle line:90% was because of this approach. 00:29:10.850 --> 00:29:14.040 align:middle line:90% 00:29:14.040 --> 00:29:15.740 align:middle line:84% Now one other thing I want to talk about 00:29:15.740 --> 00:29:20.260 align:middle line:84% is that with every succeeding convolutional layer, 00:29:20.260 --> 00:29:23.290 align:middle line:84% this particular convolution, any particular convolutional filter 00:29:23.290 --> 00:29:26.530 align:middle line:84% is basically implicitly seeing much more of the input image 00:29:26.530 --> 00:29:30.310 align:middle line:84% as we go along, which means that if, in the very beginning, 00:29:30.310 --> 00:29:33.070 align:middle line:84% if this is the input, this little convolutional filter, 00:29:33.070 --> 00:29:36.410 align:middle line:84% this number here in the first layer, let's say only 00:29:36.410 --> 00:29:40.130 align:middle line:84% sees the top of the chimney or whatever of this house. 00:29:40.130 --> 00:29:43.890 align:middle line:84% But then the next layer, remember, the next layer's input 00:29:43.890 --> 00:29:45.930 align:middle line:90% is this particular layer. 00:29:45.930 --> 00:29:48.410 align:middle line:84% And so this particular little thing 00:29:48.410 --> 00:29:52.210 align:middle line:84% here is getting information from this whole square here. 00:29:52.210 --> 00:29:54.330 align:middle line:84% And every one of the points in that square 00:29:54.330 --> 00:29:57.330 align:middle line:84% is actually something big in the original picture. 00:29:57.330 --> 00:30:00.250 align:middle line:84% So with every additional layer, you're seeing more, and more, 00:30:00.250 --> 00:30:03.490 align:middle line:90% and more of the image. 00:30:03.490 --> 00:30:05.670 align:middle line:84% And this is a key part of why these things work, 00:30:05.670 --> 00:30:08.410 align:middle line:84% because you're essentially hierarchically building 00:30:08.410 --> 00:30:10.650 align:middle line:84% a better and better understanding of the image. 00:30:10.650 --> 00:30:12.990 align:middle line:84% It is the hierarchical understanding, 00:30:12.990 --> 00:30:17.810 align:middle line:84% the hierarchical learning that's a very key part of the unlock. 00:30:17.810 --> 00:30:20.970 align:middle line:84% And so if you look at networks and what they're visualizing, 00:30:20.970 --> 00:30:23.850 align:middle line:84% this actually is a face detection deep network 00:30:23.850 --> 00:30:25.518 align:middle line:84% visualized as to what it's learning. 00:30:25.518 --> 00:30:27.810 align:middle line:84% You'll see that the first layer is just learning lines, 00:30:27.810 --> 00:30:29.890 align:middle line:90% and edges, and so on, lines. 00:30:29.890 --> 00:30:32.118 align:middle line:84% And the second layer is actually learning edges. 00:30:32.118 --> 00:30:32.910 align:middle line:90% Look at this thing. 00:30:32.910 --> 00:30:37.050 align:middle line:84% It's learning to put these lines together 00:30:37.050 --> 00:30:39.630 align:middle line:84% to get some sort of an edge here, another edge here. 00:30:39.630 --> 00:30:43.890 align:middle line:84% This looks like 3/4 of somebody's ears. 00:30:43.890 --> 00:30:46.250 align:middle line:84% And then these things are now being assembled 00:30:46.250 --> 00:30:49.087 align:middle line:90% to get whole faces out. 00:30:49.087 --> 00:30:51.170 align:middle line:84% Can you imagine the researchers who did this work? 00:30:51.170 --> 00:30:52.030 align:middle line:90% They built the network. 00:30:52.030 --> 00:30:53.790 align:middle line:84% It's doing really well on detecting faces. 00:30:53.790 --> 00:30:56.248 align:middle line:84% And they're saying, OK, let's see what it's actually doing. 00:30:56.248 --> 00:30:58.090 align:middle line:90% And then this picture pops up. 00:30:58.090 --> 00:31:00.490 align:middle line:90% I mean, the goosebumps. 00:31:00.490 --> 00:31:03.090 align:middle line:90% So pooling layers, the next one. 00:31:03.090 --> 00:31:06.030 align:middle line:84% So far you talked about convolutional layers. 00:31:06.030 --> 00:31:08.110 align:middle line:84% This is the second thing, second building block, 00:31:08.110 --> 00:31:10.670 align:middle line:90% and then we'll go to the Colabs. 00:31:10.670 --> 00:31:12.770 align:middle line:84% So pooling layers are also called subsampling 00:31:12.770 --> 00:31:15.010 align:middle line:90% or down-sampling layers. 00:31:15.010 --> 00:31:17.130 align:middle line:84% And so the idea is that every time a tensor 00:31:17.130 --> 00:31:20.680 align:middle line:84% is coming out of these convolutional layers, 00:31:20.680 --> 00:31:23.480 align:middle line:84% we'll try to make it slightly smaller. 00:31:23.480 --> 00:31:25.440 align:middle line:84% Because the act of making it smaller 00:31:25.440 --> 00:31:28.960 align:middle line:84% will force the network to try to summarize and learn what's 00:31:28.960 --> 00:31:32.040 align:middle line:84% going on in this complicated thing that's coming into it. 00:31:32.040 --> 00:31:35.160 align:middle line:84% So I will describe the mechanics first. 00:31:35.160 --> 00:31:40.720 align:middle line:84% So let's say that this is the output of a convolutional layer. 00:31:40.720 --> 00:31:42.840 align:middle line:90% This is four of them, a 4 by 4. 00:31:42.840 --> 00:31:46.760 align:middle line:84% So what we do is that there are two kinds of pooling, max 00:31:46.760 --> 00:31:48.180 align:middle line:90% pooling and average pooling. 00:31:48.180 --> 00:31:49.540 align:middle line:90% This is called max pooling. 00:31:49.540 --> 00:31:50.865 align:middle line:90% And the idea is really simple. 00:31:50.865 --> 00:31:53.240 align:middle line:84% In this max pooling layer there are no weights parameters 00:31:53.240 --> 00:31:53.823 align:middle line:90% to be learned. 00:31:53.823 --> 00:31:56.080 align:middle line:84% It's just a simple arithmetic operation. 00:31:56.080 --> 00:31:58.720 align:middle line:90% We basically take this. 00:31:58.720 --> 00:32:03.400 align:middle line:84% We basically superimpose a 2-by-2 empty grid on the top 00:32:03.400 --> 00:32:04.200 align:middle line:90% left. 00:32:04.200 --> 00:32:06.880 align:middle line:84% And then we say, hey, what's the biggest number among these four 00:32:06.880 --> 00:32:07.468 align:middle line:90% numbers? 00:32:07.468 --> 00:32:08.760 align:middle line:90% Well, the biggest number is 43. 00:32:08.760 --> 00:32:09.260 align:middle line:90% Boom. 00:32:09.260 --> 00:32:11.240 align:middle line:84% OK, I'm going to stick a 43 here. 00:32:11.240 --> 00:32:13.600 align:middle line:84% Then, I move my 2 by 2 to the right 00:32:13.600 --> 00:32:16.182 align:middle line:84% so that it overlaps with these numbers in blue. 00:32:16.182 --> 00:32:18.140 align:middle line:84% And I say, hey, what's the biggest number here? 00:32:18.140 --> 00:32:20.238 align:middle line:84% OK, that's 109, and I move it down. 00:32:20.238 --> 00:32:21.530 align:middle line:90% What's the biggest number here? 00:32:21.530 --> 00:32:23.190 align:middle line:90% 105, stick it in here. 00:32:23.190 --> 00:32:24.050 align:middle line:90% Biggest number here? 00:32:24.050 --> 00:32:25.450 align:middle line:90% 35, and I stick it in there. 00:32:25.450 --> 00:32:26.110 align:middle line:90% That's it. 00:32:26.110 --> 00:32:28.932 align:middle line:90% This is max pooling. 00:32:28.932 --> 00:32:31.390 align:middle line:84% Similarly, there's this thing called average pooling, where 00:32:31.390 --> 00:32:33.557 align:middle line:84% instead of taking the maximum of these four numbers, 00:32:33.557 --> 00:32:35.430 align:middle line:84% we just average the four numbers. 00:32:35.430 --> 00:32:38.470 align:middle line:84% The average of these four things in yellow. 00:32:38.470 --> 00:32:41.082 align:middle line:90% Am I done? 00:32:41.082 --> 00:32:43.610 align:middle line:84% The average of these four numbers is 22.2. 00:32:43.610 --> 00:32:45.610 align:middle line:84% The average of these blue numbers is 45.5. 00:32:45.610 --> 00:32:46.830 align:middle line:90% You get the idea. 00:32:46.830 --> 00:32:48.990 align:middle line:84% That's it, max pooling and average pooling. 00:32:48.990 --> 00:32:52.590 align:middle line:84% Now as you can see, when you apply pooling, 00:32:52.590 --> 00:32:55.910 align:middle line:84% the number of entries drops significantly. 00:32:55.910 --> 00:32:58.150 align:middle line:84% The number of entries drops significantly. 00:32:58.150 --> 00:32:59.870 align:middle line:84% And the output from this layer is just 00:32:59.870 --> 00:33:03.110 align:middle line:90% fit to the next layer as usual. 00:33:03.110 --> 00:33:05.670 align:middle line:90% There's nothing crazy going on. 00:33:05.670 --> 00:33:07.430 align:middle line:84% So it's a way to shrink the output 00:33:07.430 --> 00:33:09.230 align:middle line:90% from one convolutional layer. 00:33:09.230 --> 00:33:11.450 align:middle line:84% Before it passes on to the next convolutional layer, 00:33:11.450 --> 00:33:13.790 align:middle line:84% you interject with a pooling layer. 00:33:13.790 --> 00:33:17.690 align:middle line:84% Now I have actually, even if I may say so myself, 00:33:17.690 --> 00:33:20.870 align:middle line:84% a very nice handwritten explanation of what pooling 00:33:20.870 --> 00:33:23.270 align:middle line:90% does, the effect of pooling. 00:33:23.270 --> 00:33:25.190 align:middle line:84% And unfortunately, I can't get my iPad 00:33:25.190 --> 00:33:27.927 align:middle line:84% to actually show up on my laptop so I'm not 00:33:27.927 --> 00:33:29.010 align:middle line:90% going to be able to do it. 00:33:29.010 --> 00:33:32.450 align:middle line:84% But I will record a walkthrough and I'll post it. 00:33:32.450 --> 00:33:33.050 align:middle line:90% Check it out. 00:33:33.050 --> 00:33:37.310 align:middle line:84% But the intuition that I try to convey with that thing is that-- 00:33:37.310 --> 00:33:38.290 align:middle line:90% oh, sorry. 00:33:38.290 --> 00:33:39.310 align:middle line:90% I'll come back to this. 00:33:39.310 --> 00:33:42.030 align:middle line:84% So max pooling acts like an OR condition. 00:33:42.030 --> 00:33:44.870 align:middle line:84% It basically says, I have this big picture. 00:33:44.870 --> 00:33:47.230 align:middle line:84% So in the four things that I'm looking at, 00:33:47.230 --> 00:33:50.270 align:middle line:84% if there is any number which is really high, 00:33:50.270 --> 00:33:54.067 align:middle line:84% that means that some feature is being detected. 00:33:54.067 --> 00:33:56.650 align:middle line:84% The number is really high coming out of a convolutional layer. 00:33:56.650 --> 00:34:00.510 align:middle line:84% That means that something somewhere fired up, lit up. 00:34:00.510 --> 00:34:03.770 align:middle line:84% And so I'm just looking to see if anything lit up in that part. 00:34:03.770 --> 00:34:06.781 align:middle line:84% If it did, I'm going to say, yeah, something lit up. 00:34:06.781 --> 00:34:08.489 align:middle line:84% If nothing lit up, then I'm going to say, 00:34:08.489 --> 00:34:09.590 align:middle line:90% oh, nothing lit up. 00:34:09.590 --> 00:34:12.949 align:middle line:84% So in that sense, you can imagine it's 00:34:12.949 --> 00:34:15.350 align:middle line:90% acting like an OR condition. 00:34:15.350 --> 00:34:16.159 align:middle line:90% Anything fired up? 00:34:16.159 --> 00:34:16.719 align:middle line:90% Anything fired up? 00:34:16.719 --> 00:34:17.239 align:middle line:90% Anything fired up? 00:34:17.239 --> 00:34:17.620 align:middle line:90% Anything? 00:34:17.620 --> 00:34:17.730 align:middle line:90% Yes? 00:34:17.730 --> 00:34:18.230 align:middle line:90% OK. 00:34:18.230 --> 00:34:19.000 align:middle line:90% Otherwise, no. 00:34:19.000 --> 00:34:22.340 align:middle line:90% 00:34:22.340 --> 00:34:24.620 align:middle line:84% Sadly, I can't switch to Notability. 00:34:24.620 --> 00:34:26.639 align:middle line:84% So it acts like a feature detector. 00:34:26.639 --> 00:34:28.300 align:middle line:84% So if you have lots of things going on 00:34:28.300 --> 00:34:29.900 align:middle line:84% in a particular picture, you want 00:34:29.900 --> 00:34:32.340 align:middle line:84% to be able to summarize and aggregate all the things that 00:34:32.340 --> 00:34:34.242 align:middle line:84% are going on so that you can say, 00:34:34.242 --> 00:34:36.659 align:middle line:84% you may have a big picture with lots of things lighting up 00:34:36.659 --> 00:34:37.400 align:middle line:90% here and there. 00:34:37.400 --> 00:34:39.442 align:middle line:84% But you want to step back and say, you know what? 00:34:39.442 --> 00:34:42.460 align:middle line:84% In this picture, the top left, nothing lit up. 00:34:42.460 --> 00:34:44.800 align:middle line:90% The top right, something lit up. 00:34:44.800 --> 00:34:46.199 align:middle line:90% Bottom left, something lit up. 00:34:46.199 --> 00:34:48.300 align:middle line:84% And the bottom right, nothing lit up. 00:34:48.300 --> 00:34:51.739 align:middle line:84% So you're operating at a higher level of abstraction. 00:34:51.739 --> 00:34:55.179 align:middle line:90% That's the effect of pooling. 00:34:55.179 --> 00:35:00.220 align:middle line:84% AUDIENCE: But once you use spatial elements, [INAUDIBLE]? 00:35:00.220 --> 00:35:03.060 align:middle line:84% RAMA RAMAKRISHNAN: You don't, because what you're actually 00:35:03.060 --> 00:35:06.220 align:middle line:84% saying is, the top left has this thing. 00:35:06.220 --> 00:35:08.580 align:middle line:84% You already know it is in the top left. 00:35:08.580 --> 00:35:12.140 align:middle line:84% And you've already moved up to that level of abstraction. 00:35:12.140 --> 00:35:15.890 align:middle line:84% So the fact, for example, in the top left there is a human eye 00:35:15.890 --> 00:35:18.258 align:middle line:90% and there is a circle detector. 00:35:18.258 --> 00:35:20.550 align:middle line:84% It's going to fire up and saying, hey, in the top left, 00:35:20.550 --> 00:35:22.677 align:middle line:90% there is an eye lit up. 00:35:22.677 --> 00:35:24.510 align:middle line:84% So you're not looking at the pixels anymore. 00:35:24.510 --> 00:35:26.590 align:middle line:84% You're already operating at a higher level of abstraction, 00:35:26.590 --> 00:35:27.923 align:middle line:90% and that's how we get around it. 00:35:27.923 --> 00:35:30.330 align:middle line:84% But this proceeds slowly and incrementally, which 00:35:30.330 --> 00:35:31.920 align:middle line:84% is why you have these big networks. 00:35:31.920 --> 00:35:35.690 align:middle line:90% 00:35:35.690 --> 00:35:39.010 align:middle line:84% So now, as we saw, some successive convolutional layers 00:35:39.010 --> 00:35:41.610 align:middle line:84% can see more and more of the original image. 00:35:41.610 --> 00:35:43.210 align:middle line:84% The max pooling layers that follow 00:35:43.210 --> 00:35:45.010 align:middle line:84% them can detect if a feature exists 00:35:45.010 --> 00:35:47.610 align:middle line:84% in more and more of the original input as well. 00:35:47.610 --> 00:35:49.710 align:middle line:84% So by the time you get to the seventh, and eighth, 00:35:49.710 --> 00:35:52.750 align:middle line:84% ninth layers, and so on, this thing is actually really smart. 00:35:52.750 --> 00:35:56.145 align:middle line:84% It's operating at a very high level of abstraction. 00:35:56.145 --> 00:35:57.770 align:middle line:84% You can think of it as it has basically 00:35:57.770 --> 00:36:01.390 align:middle line:84% tagged all the features of that image at various resolutions, 00:36:01.390 --> 00:36:04.557 align:middle line:90% and it can work with it. 00:36:04.557 --> 00:36:06.390 align:middle line:84% AUDIENCE: Is there a trade off between doing 00:36:06.390 --> 00:36:10.570 align:middle line:84% the pre-processing as opposed to adding additional convolution 00:36:10.570 --> 00:36:11.070 align:middle line:90% layers? 00:36:11.070 --> 00:36:12.890 align:middle line:84% I'm thinking if you had a video, turning it 00:36:12.890 --> 00:36:16.890 align:middle line:84% into black-and-white static images in a sequence, as opposed 00:36:16.890 --> 00:36:20.690 align:middle line:84% to shoving in a color video with a time, 00:36:20.690 --> 00:36:22.270 align:middle line:90% the parameters have to expand. 00:36:22.270 --> 00:36:23.590 align:middle line:90% Is there a trade-off element? 00:36:23.590 --> 00:36:25.257 align:middle line:84% RAMA RAMAKRISHNAN: There is a trade off. 00:36:25.257 --> 00:36:28.050 align:middle line:84% If your particular data set and input 00:36:28.050 --> 00:36:30.570 align:middle line:84% has some-- there is some very important domain 00:36:30.570 --> 00:36:33.890 align:middle line:84% knowledge that you want to encode when you set up 00:36:33.890 --> 00:36:36.570 align:middle line:84% the network, so that the network doesn't waste 00:36:36.570 --> 00:36:39.730 align:middle line:84% its capacity learning things that have to be true, 00:36:39.730 --> 00:36:41.650 align:middle line:90% then, yeah, modify the input. 00:36:41.650 --> 00:36:45.530 align:middle line:84% But if you're not sure, then you want just the network, 00:36:45.530 --> 00:36:47.290 align:middle line:84% learn whatever it can, as long as it's 00:36:47.290 --> 00:36:50.750 align:middle line:84% focused on predicting accuracy as well as possible, 00:36:50.750 --> 00:36:51.710 align:middle line:90% then just let it be. 00:36:51.710 --> 00:36:56.750 align:middle line:90% 00:36:56.750 --> 00:36:57.950 align:middle line:90% So that's the basic idea. 00:36:57.950 --> 00:37:01.490 align:middle line:84% And again, I'm sorry this is Notability thing is not working. 00:37:01.490 --> 00:37:04.090 align:middle line:84% But take a look to really understand 00:37:04.090 --> 00:37:06.850 align:middle line:84% how this max pooling thing business works. 00:37:06.850 --> 00:37:09.290 align:middle line:90% Oh, I think I skipped over this. 00:37:09.290 --> 00:37:12.940 align:middle line:84% So when you have something like this, so this, let's say, 00:37:12.940 --> 00:37:15.400 align:middle line:84% is a tensor coming out of some convolutional layer 00:37:15.400 --> 00:37:18.240 align:middle line:84% and its size is 224 by 224 by 64. 00:37:18.240 --> 00:37:20.102 align:middle line:84% Then you apply something like a pooling. 00:37:20.102 --> 00:37:22.560 align:middle line:84% The thing I want to point out is that the pooling will work 00:37:22.560 --> 00:37:25.840 align:middle line:90% with every slice of the tensor. 00:37:25.840 --> 00:37:28.880 align:middle line:84% So if the tensor is 224 by 224 by 64, 00:37:28.880 --> 00:37:31.160 align:middle line:84% it has a depth of 64, which is basically 00:37:31.160 --> 00:37:34.780 align:middle line:84% like saying it's got 64 tables of 224 by 224. 00:37:34.780 --> 00:37:38.440 align:middle line:84% And the pooling will work on every one of those tables, which 00:37:38.440 --> 00:37:43.340 align:middle line:84% means that you'll still have 64 things at the very end. 00:37:43.340 --> 00:37:46.840 align:middle line:84% It's just that every one of the things of the 64, the 224 by 224 00:37:46.840 --> 00:37:49.800 align:middle line:90% will shrink to 112 by 112. 00:37:49.800 --> 00:37:52.382 align:middle line:84% So each table shrinks due to pooling, 00:37:52.382 --> 00:37:54.090 align:middle line:84% but the number of tables does not change. 00:37:54.090 --> 00:37:59.960 align:middle line:90% 00:37:59.960 --> 00:38:04.512 align:middle line:84% By the way, this link here is a beautiful explanation 00:38:04.512 --> 00:38:06.720 align:middle line:84% of all these things with a little bit more complexity 00:38:06.720 --> 00:38:09.390 align:middle line:84% as well from a course taught at Stanford in 2018, 00:38:09.390 --> 00:38:11.360 align:middle line:90% or 2019, or something, I forget. 00:38:11.360 --> 00:38:13.860 align:middle line:84% So just check it out if you're curious about this stuff. 00:38:13.860 --> 00:38:14.580 align:middle line:90% It's really good. 00:38:14.580 --> 00:38:19.060 align:middle line:90% 00:38:19.060 --> 00:38:22.400 align:middle line:84% So that brings us to the architecture of a basic CNN. 00:38:22.400 --> 00:38:25.780 align:middle line:84% And so what we do is we have an input. 00:38:25.780 --> 00:38:26.740 align:middle line:90% We take that input. 00:38:26.740 --> 00:38:29.600 align:middle line:84% We run it through a bunch of convolutional and pooling 00:38:29.600 --> 00:38:30.440 align:middle line:90% layers. 00:38:30.440 --> 00:38:32.320 align:middle line:84% So this is a convolutional layer. 00:38:32.320 --> 00:38:37.400 align:middle line:84% And then we pool it, which is why it has shrunk in size. 00:38:37.400 --> 00:38:39.620 align:middle line:84% And then it goes through another convolutional layer. 00:38:39.620 --> 00:38:42.280 align:middle line:84% Then, we pool it, which is shrunk again, 00:38:42.280 --> 00:38:43.820 align:middle line:90% and then it keeps on doing it. 00:38:43.820 --> 00:38:45.480 align:middle line:90% So we have a series of these. 00:38:45.480 --> 00:38:47.560 align:middle line:84% These are called convolutional blocks. 00:38:47.560 --> 00:38:49.680 align:middle line:84% So a convolutional block is typically 00:38:49.680 --> 00:38:52.960 align:middle line:84% one to two convolutional layers followed by a pooling layer. 00:38:52.960 --> 00:38:57.580 align:middle line:84% So you have a series of convolutional blocks. 00:38:57.580 --> 00:39:01.360 align:middle line:84% And the thing to notice is that, as you go further and further 00:39:01.360 --> 00:39:04.760 align:middle line:84% in the network, the blocks will actually 00:39:04.760 --> 00:39:08.120 align:middle line:84% get smaller and smaller because of max pooling. 00:39:08.120 --> 00:39:12.260 align:middle line:84% It gets smaller and smaller, but they'll get deeper and deeper. 00:39:12.260 --> 00:39:14.790 align:middle line:90% 00:39:14.790 --> 00:39:16.390 align:middle line:84% And we have empirically figured out 00:39:16.390 --> 00:39:20.430 align:middle line:84% that actually that model of reducing the size, the height 00:39:20.430 --> 00:39:22.350 align:middle line:84% and the width, but then making it deeper, 00:39:22.350 --> 00:39:25.510 align:middle line:84% tends to work really well in practice. 00:39:25.510 --> 00:39:30.630 align:middle line:84% And so, in fact, apologies to the live stream that I can't use 00:39:30.630 --> 00:39:31.130 align:middle line:90% iPad. 00:39:31.130 --> 00:39:32.463 align:middle line:90% I'm going to do it on the board. 00:39:32.463 --> 00:39:35.990 align:middle line:90% 00:39:35.990 --> 00:39:38.590 align:middle line:84% So let's say that you have a picture, which 00:39:38.590 --> 00:39:44.830 align:middle line:90% is coming in as 224, 224. 00:39:44.830 --> 00:39:47.950 align:middle line:84% And then you have, say, three of them 00:39:47.950 --> 00:39:49.530 align:middle line:90% because it's a color picture. 00:39:49.530 --> 00:39:52.350 align:middle line:90% So you have three of them. 00:39:52.350 --> 00:39:54.350 align:middle line:90% Can you folks see this OK? 00:39:54.350 --> 00:39:55.110 align:middle line:90% All right. 00:39:55.110 --> 00:39:57.670 align:middle line:84% So let's say this is the input coming in. 00:39:57.670 --> 00:40:00.790 align:middle line:84% And ResNet, which is a very famous network that 00:40:00.790 --> 00:40:03.750 align:middle line:84% we're actually going to work with in a few minutes, 00:40:03.750 --> 00:40:06.790 align:middle line:84% when it actually gets done with all this convolution pooling 00:40:06.790 --> 00:40:13.980 align:middle line:84% business, the final tensor that it has is actually of shape 7 00:40:13.980 --> 00:40:23.920 align:middle line:90% by 7, but it is 2,048 long. 00:40:23.920 --> 00:40:26.700 align:middle line:84% It has processed something which is 224, 224 times 3, 00:40:26.700 --> 00:40:29.160 align:middle line:84% to a much smaller height and width, just 7 by 7. 00:40:29.160 --> 00:40:32.940 align:middle line:84% But it has gotten much deeper 2,048 layers. 00:40:32.940 --> 00:40:35.060 align:middle line:84% This is a numerical example of what 00:40:35.060 --> 00:40:37.820 align:middle line:84% I am talking about there in terms of, as you go along, 00:40:37.820 --> 00:40:39.670 align:middle line:90% things get smaller but deeper. 00:40:39.670 --> 00:40:44.460 align:middle line:90% 00:40:44.460 --> 00:40:46.020 align:middle line:90% Yes? 00:40:46.020 --> 00:40:49.740 align:middle line:84% AUDIENCE: Is the reason that it gets deeper because each-- 00:40:49.740 --> 00:40:52.660 align:middle line:84% it gets deeper because each layer has a single feature that 00:40:52.660 --> 00:40:55.160 align:middle line:84% is picked up, and then it gets stacked on top of each other? 00:40:55.160 --> 00:40:55.860 align:middle line:84% RAMA RAMAKRISHNAN: It's not so much 00:40:55.860 --> 00:40:58.200 align:middle line:84% that each layer has picking up a single feature. 00:40:58.200 --> 00:41:02.220 align:middle line:84% It's more that, basically, the way I 00:41:02.220 --> 00:41:06.540 align:middle line:84% think about it is that the number of atomic features 00:41:06.540 --> 00:41:09.460 align:middle line:84% that you may want to detect are probably not that many. 00:41:09.460 --> 00:41:12.760 align:middle line:84% Lines, curves, gradations, and color, and things like that. 00:41:12.760 --> 00:41:14.300 align:middle line:84% But the way in which you can combine 00:41:14.300 --> 00:41:18.540 align:middle line:84% these atomic features to depict real world things 00:41:18.540 --> 00:41:20.140 align:middle line:90% is combinatorial. 00:41:20.140 --> 00:41:23.000 align:middle line:84% It's sort of like, I have 10 kinds of atoms. 00:41:23.000 --> 00:41:25.020 align:middle line:84% How many molecules can I make from it? 00:41:25.020 --> 00:41:27.740 align:middle line:84% You can make a lot of molecules from those 10 atoms, which 00:41:27.740 --> 00:41:30.660 align:middle line:84% means that you better give the network the ability 00:41:30.660 --> 00:41:32.900 align:middle line:84% to capture more and more of these possible things 00:41:32.900 --> 00:41:35.380 align:middle line:84% that the real world can come up with. 00:41:35.380 --> 00:41:39.120 align:middle line:84% And so as the depth increases, you have more filters. 00:41:39.120 --> 00:41:41.540 align:middle line:84% And every filter now has the ability 00:41:41.540 --> 00:41:44.480 align:middle line:84% to pick up some combinatorial combination of what's coming in. 00:41:44.480 --> 00:41:49.757 align:middle line:90% 00:41:49.757 --> 00:41:51.840 align:middle line:84% AUDIENCE: Sorry, a quick question related to this. 00:41:51.840 --> 00:41:53.940 align:middle line:84% So right now our model is being trained 00:41:53.940 --> 00:41:56.780 align:middle line:84% to detect specific features, like a line, eye color, 00:41:56.780 --> 00:41:57.940 align:middle line:90% or something of this sort. 00:41:57.940 --> 00:42:00.900 align:middle line:84% But still, it doesn't have meaning to this. 00:42:00.900 --> 00:42:06.230 align:middle line:84% Still, they don't know if that arc is a sun or is an eye. 00:42:06.230 --> 00:42:07.230 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:42:07.230 --> 00:42:09.270 align:middle line:84% So we don't tell it what to learn. 00:42:09.270 --> 00:42:10.690 align:middle line:90% It just learns. 00:42:10.690 --> 00:42:12.450 align:middle line:84% All we tell it is, make sure that you 00:42:12.450 --> 00:42:14.010 align:middle line:90% minimize the loss function. 00:42:14.010 --> 00:42:15.830 align:middle line:84% Now, once it has finished learning, 00:42:15.830 --> 00:42:18.330 align:middle line:84% if it's a good network, it has good accuracy. 00:42:18.330 --> 00:42:20.170 align:middle line:90% Then, we can introspect. 00:42:20.170 --> 00:42:23.170 align:middle line:84% We can peek into the internals and try to understand, 00:42:23.170 --> 00:42:24.530 align:middle line:90% what is it learning? 00:42:24.530 --> 00:42:27.180 align:middle line:84% And sometimes, like you saw in the face detection example, 00:42:27.180 --> 00:42:28.930 align:middle line:84% it's actually learning interesting things, 00:42:28.930 --> 00:42:32.290 align:middle line:84% like basic lines, and edges, and then slowly, more complicated 00:42:32.290 --> 00:42:32.790 align:middle line:90% shapes. 00:42:32.790 --> 00:42:35.550 align:middle line:84% And then finally, entire human faces. 00:42:35.550 --> 00:42:37.268 align:middle line:84% Sometimes it may not be understandable. 00:42:37.268 --> 00:42:38.810 align:middle line:84% AUDIENCE: And the way it's doing this 00:42:38.810 --> 00:42:41.960 align:middle line:84% is by constructing features, not by learning. 00:42:41.960 --> 00:42:44.460 align:middle line:84% RAMA RAMAKRISHNAN: How do you figure out what it's learning? 00:42:44.460 --> 00:42:45.050 align:middle line:90% AUDIENCE: Yes. 00:42:45.050 --> 00:42:46.258 align:middle line:90% RAMA RAMAKRISHNAN: Oh, I see. 00:42:46.258 --> 00:42:49.090 align:middle line:84% So I'm going to give a reference in just a few minutes. 00:42:49.090 --> 00:42:50.030 align:middle line:90% Read the paper. 00:42:50.030 --> 00:42:52.197 align:middle line:84% That was one of the first ones to actually visualize 00:42:52.197 --> 00:42:53.630 align:middle line:90% what these things are learning. 00:42:53.630 --> 00:42:55.750 align:middle line:84% And that will give you an idea of how it actually works. 00:42:55.750 --> 00:42:57.583 align:middle line:84% And I'm also happy to talk about it offline. 00:42:57.583 --> 00:43:00.310 align:middle line:84% It's a bit of a tangent, but it's a really rich tangent. 00:43:00.310 --> 00:43:02.095 align:middle line:84% So if I keep talking about it, I'll 00:43:02.095 --> 00:43:03.470 align:middle line:84% end up spending 10 minutes on it. 00:43:03.470 --> 00:43:04.512 align:middle line:90% So I'm going to back off. 00:43:04.512 --> 00:43:09.720 align:middle line:90% 00:43:09.720 --> 00:43:13.520 align:middle line:84% So now, once we do that, now we are back 00:43:13.520 --> 00:43:16.240 align:middle line:84% in familiar territory, where we take whatever tensor 00:43:16.240 --> 00:43:18.880 align:middle line:84% is coming out from these convolutional operations 00:43:18.880 --> 00:43:22.600 align:middle line:84% and pooling operations, and then we just flatten them, only now 00:43:22.600 --> 00:43:23.960 align:middle line:90% into a long vector. 00:43:23.960 --> 00:43:26.200 align:middle line:84% And once we flatten them, we can connect them 00:43:26.200 --> 00:43:30.260 align:middle line:84% to some good, old dense layers, like we know how to do. 00:43:30.260 --> 00:43:33.120 align:middle line:84% And then we finally connect them with whatever 00:43:33.120 --> 00:43:34.800 align:middle line:90% output layer you want. 00:43:34.800 --> 00:43:36.600 align:middle line:84% In this case, this example is using 00:43:36.600 --> 00:43:40.440 align:middle line:84% some multi-class classification of classifying images 00:43:40.440 --> 00:43:42.660 align:middle line:84% to what kind of automobile or whatever it is. 00:43:42.660 --> 00:43:43.980 align:middle line:90% So it's like a softmax. 00:43:43.980 --> 00:43:45.460 align:middle line:84% So this is the general framework. 00:43:45.460 --> 00:43:48.680 align:middle line:90% 00:43:48.680 --> 00:43:50.900 align:middle line:90% Any questions? 00:43:50.900 --> 00:43:54.600 align:middle line:90% 00:43:54.600 --> 00:43:55.720 align:middle line:90% Yeah? 00:43:55.720 --> 00:43:59.110 align:middle line:84% AUDIENCE: Can you explain again how the depth increases exactly? 00:43:59.110 --> 00:44:01.360 align:middle line:84% RAMA RAMAKRISHNAN: Oh, the depth increases because you 00:44:01.360 --> 00:44:02.580 align:middle line:90% decide what the depth is. 00:44:02.580 --> 00:44:05.960 align:middle line:84% So when you add a convolutional layer, 00:44:05.960 --> 00:44:07.680 align:middle line:84% you decide how many filters it has. 00:44:07.680 --> 00:44:11.120 align:middle line:84% So you just keep adding more and more filters the later 00:44:11.120 --> 00:44:13.280 align:middle line:90% on you go in the network. 00:44:13.280 --> 00:44:14.300 align:middle line:90% So it's in your hands. 00:44:14.300 --> 00:44:16.640 align:middle line:84% So remember, the number of neurons in a hidden layer 00:44:16.640 --> 00:44:18.120 align:middle line:90% is in your control. 00:44:18.120 --> 00:44:20.300 align:middle line:84% Similarly, the number of filters is in your control. 00:44:20.300 --> 00:44:22.560 align:middle line:90% It's a design choice. 00:44:22.560 --> 00:44:24.760 align:middle line:84% And we design it so that the later we go, 00:44:24.760 --> 00:44:25.940 align:middle line:90% the more depth we have. 00:44:25.940 --> 00:44:30.200 align:middle line:84% AUDIENCE: So you stack layers with-- 00:44:30.200 --> 00:44:32.880 align:middle line:84% each of those layers has a different filter 00:44:32.880 --> 00:44:34.600 align:middle line:90% applied to the end output? 00:44:34.600 --> 00:44:35.600 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:44:35.600 --> 00:44:37.333 align:middle line:90% A layer is made up of filters. 00:44:37.333 --> 00:44:40.000 align:middle line:84% And so the depth just comes from having lots, and lots, and lots 00:44:40.000 --> 00:44:40.520 align:middle line:90% of filters. 00:44:40.520 --> 00:44:42.020 align:middle line:84% And you get to choose what they are. 00:44:42.020 --> 00:44:45.080 align:middle line:90% 00:44:45.080 --> 00:44:48.920 align:middle line:84% So now let's go to the Fashion MNIST Colab 00:44:48.920 --> 00:44:52.240 align:middle line:84% that I did the video walkthrough on, and then actually solve 00:44:52.240 --> 00:44:55.880 align:middle line:84% it using a convolutional network. 00:44:55.880 --> 00:44:56.380 align:middle line:90% All right. 00:44:56.380 --> 00:44:56.880 align:middle line:90% Cool. 00:44:56.880 --> 00:44:59.560 align:middle line:84% So at this point I'm going to zip through some of this stuff 00:44:59.560 --> 00:45:01.950 align:middle line:84% because the preliminaries have to be done. 00:45:01.950 --> 00:45:03.610 align:middle line:90% Import all these packages. 00:45:03.610 --> 00:45:05.570 align:middle line:90% Set the random seed here. 00:45:05.570 --> 00:45:06.070 align:middle line:90% Great. 00:45:06.070 --> 00:45:08.370 align:middle line:84% And then we will load the MNIST data set, 00:45:08.370 --> 00:45:10.910 align:middle line:84% just like I did in the Colab yesterday. 00:45:10.910 --> 00:45:13.790 align:middle line:84% We create these little labels, and then 00:45:13.790 --> 00:45:15.430 align:middle line:84% we just have these standard functions 00:45:15.430 --> 00:45:18.870 align:middle line:84% to plot accuracy and loss that we've been using so far. 00:45:18.870 --> 00:45:21.030 align:middle line:84% All right, now we come to the convolutional thing. 00:45:21.030 --> 00:45:25.190 align:middle line:84% And so as before, we're going to divide it 00:45:25.190 --> 00:45:28.310 align:middle line:84% by 255 to normalize everything to a 0 to 1 range. 00:45:28.310 --> 00:45:30.710 align:middle line:84% Let's confirm to make sure that the data, nothing has 00:45:30.710 --> 00:45:32.570 align:middle line:90% gotten tampered with. 00:45:32.570 --> 00:45:34.410 align:middle line:90% Yes, we have 60,000 images. 00:45:34.410 --> 00:45:36.710 align:middle line:84% Each one is 28 by 28 in the training set. 00:45:36.710 --> 00:45:41.510 align:middle line:84% Now convolutional networks, they expect the input 00:45:41.510 --> 00:45:44.510 align:middle line:84% to have three channels, or it expects 00:45:44.510 --> 00:45:48.390 align:middle line:84% to have an additional thing, which is like a channel. 00:45:48.390 --> 00:45:50.750 align:middle line:84% The color images have three channels. 00:45:50.750 --> 00:45:52.930 align:middle line:84% But black and white images have only one channel, 00:45:52.930 --> 00:45:54.450 align:middle line:90% one table of numbers. 00:45:54.450 --> 00:45:58.350 align:middle line:84% So instead of saying 28 by 28, we tell the convolutional layer, 00:45:58.350 --> 00:46:01.660 align:middle line:90% expect 28 by 28 by 1. 00:46:01.660 --> 00:46:03.040 align:middle line:84% It's the same thing conceptually, 00:46:03.040 --> 00:46:05.700 align:middle line:84% but that's the format that it expects. 00:46:05.700 --> 00:46:08.200 align:middle line:90% And so we go here. 00:46:08.200 --> 00:46:10.180 align:middle line:84% And then we say, all right, there's 00:46:10.180 --> 00:46:11.920 align:middle line:90% a thing called expand_dimension. 00:46:11.920 --> 00:46:13.840 align:middle line:84% I'm just telling it to expand its dimension. 00:46:13.840 --> 00:46:17.000 align:middle line:84% And once I do that, you can see here, it's still 60,000. 00:46:17.000 --> 00:46:20.800 align:middle line:84% But instead of 28 by 28, it has become 28 by 28 by 1. 00:46:20.800 --> 00:46:21.820 align:middle line:90% Same thing. 00:46:21.820 --> 00:46:24.450 align:middle line:84% Now, let's define our very first CNN. 00:46:24.450 --> 00:46:27.700 align:middle line:90% 00:46:27.700 --> 00:46:31.940 align:middle line:84% As before, the input is just keras.Input, as before. 00:46:31.940 --> 00:46:33.158 align:middle line:90% No difference here. 00:46:33.158 --> 00:46:34.200 align:middle line:90% And we tell it the shape. 00:46:34.200 --> 00:46:37.340 align:middle line:84% And the shape is, of course, just 28 by 28 by 1. 00:46:37.340 --> 00:46:39.220 align:middle line:90% That's what I have here. 00:46:39.220 --> 00:46:43.860 align:middle line:84% And then we come to the first convolutional block. 00:46:43.860 --> 00:46:45.820 align:middle line:90% And this is the key thing. 00:46:45.820 --> 00:46:49.740 align:middle line:84% If you want to tell Keras to use a convolutional layer, 00:46:49.740 --> 00:46:53.223 align:middle line:84% you use this keyword, layers.Conv2D. 00:46:53.223 --> 00:46:55.140 align:middle line:84% And from this you can probably also figure out 00:46:55.140 --> 00:46:57.460 align:middle line:84% that there's a Conv1D, and there's a Conv3D, 00:46:57.460 --> 00:47:00.680 align:middle line:84% and so on, and so forth, which, explore. 00:47:00.680 --> 00:47:01.940 align:middle line:90% It's really good stuff. 00:47:01.940 --> 00:47:05.180 align:middle line:84% But for image processing, Conv2D is all you need. 00:47:05.180 --> 00:47:08.200 align:middle line:84% And now we tell it how many filters you want. 00:47:08.200 --> 00:47:10.720 align:middle line:84% So we decide on the number of filters. 00:47:10.720 --> 00:47:13.060 align:middle line:84% So I've decided to have 32 filters. 00:47:13.060 --> 00:47:16.820 align:middle line:84% And then we also have to decide the size of the filter. 00:47:16.820 --> 00:47:19.180 align:middle line:90% The simplest size is 2 by 2. 00:47:19.180 --> 00:47:22.587 align:middle line:84% So I'm just going to go with that, kernel_size is 2 by 2. 00:47:22.587 --> 00:47:24.420 align:middle line:84% And then the activation is, of course, ReLU. 00:47:24.420 --> 00:47:27.580 align:middle line:84% I give it a name, convolution 1, and then I feed it the input. 00:47:27.580 --> 00:47:30.780 align:middle line:84% And then once I do that, I followed up with a pooling layer 00:47:30.780 --> 00:47:33.620 align:middle line:90% where I use MaxPool2D. 00:47:33.620 --> 00:47:36.000 align:middle line:84% And MaxPool2D, you just literally pass the input. 00:47:36.000 --> 00:47:37.940 align:middle line:84% You get the output back, it just shrinks 00:47:37.940 --> 00:47:39.420 align:middle line:90% everything using pooling. 00:47:39.420 --> 00:47:41.620 align:middle line:84% So that is the first convolutional block. 00:47:41.620 --> 00:47:43.820 align:middle line:90% And you know what? 00:47:43.820 --> 00:47:45.240 align:middle line:90% I know how to cut and paste. 00:47:45.240 --> 00:47:48.635 align:middle line:84% Cut and paste, I get the second convolutional block. 00:47:48.635 --> 00:47:50.260 align:middle line:84% Here is the second convolutional block. 00:47:50.260 --> 00:47:52.780 align:middle line:84% And I know in just the lecture I mentioned 00:47:52.780 --> 00:47:55.880 align:middle line:84% that, as you go deeper, you get more depth to it. 00:47:55.880 --> 00:47:57.488 align:middle line:84% But this is just a starting point. 00:47:57.488 --> 00:47:59.030 align:middle line:84% I'm just going to use the same depth. 00:47:59.030 --> 00:47:59.630 align:middle line:90% It's not a big deal. 00:47:59.630 --> 00:48:01.047 align:middle line:84% It's a simple problem, so which is 00:48:01.047 --> 00:48:04.490 align:middle line:84% why in the second convolutional block, I'm still using only 32. 00:48:04.490 --> 00:48:06.510 align:middle line:84% But you can totally go to 64, for instance, 00:48:06.510 --> 00:48:07.730 align:middle line:90% to make it much deeper. 00:48:07.730 --> 00:48:11.970 align:middle line:84% And once I do that, I finally come to the point 00:48:11.970 --> 00:48:14.290 align:middle line:84% where I flatten everything into a long vector. 00:48:14.290 --> 00:48:19.450 align:middle line:84% Then, I connect it to one dense layer of 256 neurons. 00:48:19.450 --> 00:48:22.130 align:middle line:84% And then finally, I come to the softmax 00:48:22.130 --> 00:48:25.950 align:middle line:84% where I have 10 outputs, 10 categories of clothing, softmax. 00:48:25.950 --> 00:48:28.750 align:middle line:84% And then I tell Keras, OK, take this input and the output. 00:48:28.750 --> 00:48:30.130 align:middle line:90% String them up together. 00:48:30.130 --> 00:48:32.290 align:middle line:90% Define a model for me. 00:48:32.290 --> 00:48:32.910 align:middle line:90% That's it. 00:48:32.910 --> 00:48:34.350 align:middle line:84% That's the convolutional network. 00:48:34.350 --> 00:48:36.250 align:middle line:84% The new concepts we are seeing here 00:48:36.250 --> 00:48:39.010 align:middle line:84% are Conv2D for the convolutional layer, 00:48:39.010 --> 00:48:42.810 align:middle line:84% and then MaxPool2D for the max pooling layer. 00:48:42.810 --> 00:48:44.250 align:middle line:90% That's it. 00:48:44.250 --> 00:48:45.010 align:middle line:90% Coming. 00:48:45.010 --> 00:48:47.730 align:middle line:84% So let me just run this thing, make sure it runs. 00:48:47.730 --> 00:48:48.230 align:middle line:90% OK. 00:48:48.230 --> 00:48:48.770 align:middle line:90% Good. 00:48:48.770 --> 00:48:50.290 align:middle line:90% Yeah? 00:48:50.290 --> 00:48:52.650 align:middle line:84% AUDIENCE: How do you decide when to flatten? 00:48:52.650 --> 00:48:54.450 align:middle line:84% And would there ever be a situation 00:48:54.450 --> 00:48:57.810 align:middle line:84% in which we just use the method that we used before 00:48:57.810 --> 00:48:59.017 align:middle line:90% and not using CNN? 00:48:59.017 --> 00:49:01.350 align:middle line:84% RAMA RAMAKRISHNAN: Well, we already tried it with MNIST. 00:49:01.350 --> 00:49:02.190 align:middle line:90% We didn't use the CNN. 00:49:02.190 --> 00:49:03.190 align:middle line:90% We just flattened it right away. 00:49:03.190 --> 00:49:03.870 align:middle line:90% AUDIENCE: Yeah, it did work. 00:49:03.870 --> 00:49:04.990 align:middle line:90% RAMA RAMAKRISHNAN: It's not bad. 00:49:04.990 --> 00:49:07.063 align:middle line:84% But we are like, can we do better than 85, or 88, 00:49:07.063 --> 00:49:08.230 align:middle line:90% or whatever the percent was? 00:49:08.230 --> 00:49:10.650 align:middle line:90% But we are working with images. 00:49:10.650 --> 00:49:12.970 align:middle line:84% It's typically a good idea to just start with the CNN 00:49:12.970 --> 00:49:15.095 align:middle line:84% straight off the bat because you're not losing any. 00:49:15.095 --> 00:49:16.770 align:middle line:90% You're not giving up anything. 00:49:16.770 --> 00:49:20.090 align:middle line:84% So in terms of how many layers you should have, 00:49:20.090 --> 00:49:21.790 align:middle line:90% my philosophy is start simple. 00:49:21.790 --> 00:49:23.750 align:middle line:84% And if it works, stop working on it. 00:49:23.750 --> 00:49:27.290 align:middle line:90% If it doesn't, add more layers. 00:49:27.290 --> 00:49:27.935 align:middle line:90% Yeah? 00:49:27.935 --> 00:49:29.810 align:middle line:84% AUDIENCE: Yeah, just to build on that, is it, 00:49:29.810 --> 00:49:32.150 align:middle line:84% the architecture design, so the number of filters, 00:49:32.150 --> 00:49:33.950 align:middle line:84% the kernel size, the number of layers, 00:49:33.950 --> 00:49:35.570 align:middle line:84% convolution, pooling, is that just 00:49:35.570 --> 00:49:37.430 align:middle line:84% all based on trial and error to see what works best? 00:49:37.430 --> 00:49:37.490 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:49:37.490 --> 00:49:39.282 align:middle line:84% So typically it's based on trial and error, 00:49:39.282 --> 00:49:40.758 align:middle line:90% to answer your question. 00:49:40.758 --> 00:49:43.050 align:middle line:84% But as you will see in the transfer learning discussion 00:49:43.050 --> 00:49:45.850 align:middle line:84% we're going to have soon, you can actually, 00:49:45.850 --> 00:49:47.990 align:middle line:84% instead of doing anything from scratch, 00:49:47.990 --> 00:49:50.490 align:middle line:84% it's much better to just download a pre-trained model 00:49:50.490 --> 00:49:52.630 align:middle line:84% and just adapt it for your particular problem. 00:49:52.630 --> 00:49:55.270 align:middle line:84% That is actually the norm by which people do these things. 00:49:55.270 --> 00:49:56.770 align:middle line:84% The reason I'm doing it from scratch 00:49:56.770 --> 00:50:00.560 align:middle line:84% is because you should know how it was done. 00:50:00.560 --> 00:50:02.420 align:middle line:84% It should not be a black box to you. 00:50:02.420 --> 00:50:03.920 align:middle line:90% That's my goal. 00:50:03.920 --> 00:50:05.160 align:middle line:90% Yeah? 00:50:05.160 --> 00:50:07.060 align:middle line:84% AUDIENCE: Just from a notation perspective, 00:50:07.060 --> 00:50:09.718 align:middle line:84% I noticed that you made all of these layers x. 00:50:09.718 --> 00:50:12.260 align:middle line:84% Is that a habit we should get into, making them all the same, 00:50:12.260 --> 00:50:13.100 align:middle line:90% or is that just-- 00:50:13.100 --> 00:50:16.320 align:middle line:84% RAMA RAMAKRISHNAN: Actually, I'm not naming the layers as x. 00:50:16.320 --> 00:50:19.720 align:middle line:84% What's going on here is I'm feeding it x. 00:50:19.720 --> 00:50:22.680 align:middle line:84% And whatever is coming out of it, I'm just calling it x. 00:50:22.680 --> 00:50:23.180 align:middle line:90% That's all. 00:50:23.180 --> 00:50:25.190 align:middle line:84% It's just a notational convenience for me. 00:50:25.190 --> 00:50:26.940 align:middle line:84% I'm just calling the input and the output, 00:50:26.940 --> 00:50:29.240 align:middle line:84% and Keras, under the hood, will track everything and make 00:50:29.240 --> 00:50:30.500 align:middle line:90% sure the right thing happens. 00:50:30.500 --> 00:50:32.802 align:middle line:84% Otherwise, I'd have to be, x1, x2, x3, x4. 00:50:32.802 --> 00:50:34.760 align:middle line:84% And then if I want to add a new layer somewhere 00:50:34.760 --> 00:50:37.980 align:middle line:84% in the middle between x3 and x4, I'll have to call that x4. 00:50:37.980 --> 00:50:40.060 align:middle line:84% And then I'll change everything to 5, 6, 7. 00:50:40.060 --> 00:50:41.200 align:middle line:90% Complete pain in the neck. 00:50:41.200 --> 00:50:43.600 align:middle line:90% That's why I do this. 00:50:43.600 --> 00:50:50.920 align:middle line:84% So model.summary, it has got 302,000 parameters. 00:50:50.920 --> 00:50:53.240 align:middle line:90% I'll just plot it. 00:50:53.240 --> 00:50:54.030 align:middle line:90% Great. 00:50:54.030 --> 00:50:58.030 align:middle line:84% And I encourage you to hand calculate it later on 00:50:58.030 --> 00:51:00.790 align:middle line:90% and make sure the numbers tally. 00:51:00.790 --> 00:51:02.430 align:middle line:90% For now, let's just go. 00:51:02.430 --> 00:51:06.370 align:middle line:84% So as before, we'll just use the same compilation. 00:51:06.370 --> 00:51:10.890 align:middle line:84% We'll use adam, and then we'll train it for just 10 epochs. 00:51:10.890 --> 00:51:13.970 align:middle line:84% We'll use a validation split, again, as usual of 20%. 00:51:13.970 --> 00:51:15.790 align:middle line:90% So let's just run it. 00:51:15.790 --> 00:51:17.010 align:middle line:90% It's actually going to run. 00:51:17.010 --> 00:51:19.042 align:middle line:84% And as you will see, convolution networks, 00:51:19.042 --> 00:51:20.750 align:middle line:84% there's a lot more going on so it's going 00:51:20.750 --> 00:51:22.610 align:middle line:90% to be a bit slower to run. 00:51:22.610 --> 00:51:25.350 align:middle line:90% Hopefully not too much slower. 00:51:25.350 --> 00:51:26.890 align:middle line:84% While it's doing, other questions? 00:51:26.890 --> 00:51:30.685 align:middle line:90% 00:51:30.685 --> 00:51:33.310 align:middle line:84% AUDIENCE: So if we have a task other than image classification, 00:51:33.310 --> 00:51:36.870 align:middle line:84% do we still flatten the model like for semantic segmentation? 00:51:36.870 --> 00:51:37.870 align:middle line:90% RAMA RAMAKRISHNAN: Yeah. 00:51:37.870 --> 00:51:39.790 align:middle line:84% So this is for image classification. 00:51:39.790 --> 00:51:43.312 align:middle line:84% For other kinds of applications, typically you 00:51:43.312 --> 00:51:45.270 align:middle line:84% run it through a bunch of convolutional layers, 00:51:45.270 --> 00:51:46.670 align:middle line:90% and so on, and so forth. 00:51:46.670 --> 00:51:48.630 align:middle line:84% But the output side of the equation 00:51:48.630 --> 00:51:50.710 align:middle line:90% gets much more complicated. 00:51:50.710 --> 00:51:54.790 align:middle line:84% Because instead of classifying just the whole picture 00:51:54.790 --> 00:51:58.790 align:middle line:84% into dog or cat, if you have to take every pixel 00:51:58.790 --> 00:52:01.630 align:middle line:84% and classify it, then, well, you better 00:52:01.630 --> 00:52:04.950 align:middle line:84% have an output shape that is the same dimensions as the input 00:52:04.950 --> 00:52:06.350 align:middle line:90% shape. 00:52:06.350 --> 00:52:08.210 align:middle line:84% So for that we use a different architecture. 00:52:08.210 --> 00:52:11.030 align:middle line:84% It's called U-Net, and so on, which unfortunately I 00:52:11.030 --> 00:52:12.270 align:middle line:90% won't be able to get into. 00:52:12.270 --> 00:52:15.790 align:middle line:84% But I am planning to post another video walkthrough where 00:52:15.790 --> 00:52:18.510 align:middle line:84% I show you how to use the Hugging Face 00:52:18.510 --> 00:52:23.170 align:middle line:84% Hub to very quickly build models for the other applications, 00:52:23.170 --> 00:52:24.530 align:middle line:90% like segmentation, and so on. 00:52:24.530 --> 00:52:26.030 align:middle line:84% I'm hoping to post that tomorrow. 00:52:26.030 --> 00:52:27.322 align:middle line:90% It's an optional viewing thing. 00:52:27.322 --> 00:52:30.110 align:middle line:90% That might help with that. 00:52:30.110 --> 00:52:31.870 align:middle line:90% So is it done? 00:52:31.870 --> 00:52:32.370 align:middle line:90% OK, good. 00:52:32.370 --> 00:52:33.670 align:middle line:90% It's done. 00:52:33.670 --> 00:52:36.522 align:middle line:84% All right, let's plot the thing here. 00:52:36.522 --> 00:52:38.230 align:middle line:84% All right, so it seems like this training 00:52:38.230 --> 00:52:39.950 align:middle line:90% is going down nice and nicely. 00:52:39.950 --> 00:52:43.910 align:middle line:84% Validation is flattening out somewhere here around the 8th 00:52:43.910 --> 00:52:44.670 align:middle line:90% epoch. 00:52:44.670 --> 00:52:47.390 align:middle line:90% Let's look at the accuracy. 00:52:47.390 --> 00:52:48.370 align:middle line:90% Same situation here. 00:52:48.370 --> 00:52:50.212 align:middle line:90% The accuracy is in the 90s. 00:52:50.212 --> 00:52:51.920 align:middle line:84% Of course, the final question, of course, 00:52:51.920 --> 00:52:55.900 align:middle line:84% is how well it does on the thing. 00:52:55.900 --> 00:52:58.300 align:middle line:90% Whoa, 90.5%. 00:52:58.300 --> 00:52:59.162 align:middle line:90% It's pretty good. 00:52:59.162 --> 00:53:00.620 align:middle line:84% By the way, if you're not impressed 00:53:00.620 --> 00:53:05.780 align:middle line:84% that we went from 88 to 90, these applications 00:53:05.780 --> 00:53:08.480 align:middle line:84% are the proverbial diminishing returns problems. 00:53:08.480 --> 00:53:10.940 align:middle line:84% So what you should always think of is, 00:53:10.940 --> 00:53:13.960 align:middle line:84% look at the amount of error that's left, and ask yourself, 00:53:13.960 --> 00:53:17.500 align:middle line:84% how much of that error am I able to reduce? 00:53:17.500 --> 00:53:20.380 align:middle line:84% We had 12%, roughly, of error left when we 00:53:20.380 --> 00:53:22.140 align:middle line:90% did the simple Colab yesterday. 00:53:22.140 --> 00:53:24.900 align:middle line:84% From that 12%, we have knocked off 2% of the 12% 00:53:24.900 --> 00:53:27.260 align:middle line:84% to get to over 90%, which is amazing. 00:53:27.260 --> 00:53:34.420 align:middle line:84% And in fact, I think the state of the art on this is 97%. 00:53:34.420 --> 00:53:38.045 align:middle line:84% So I invite you to take this thing, 00:53:38.045 --> 00:53:40.420 align:middle line:84% and try different filters, and so on, and so forth to see 00:53:40.420 --> 00:53:42.820 align:middle line:90% if you can get to the mid 90s. 00:53:42.820 --> 00:53:44.700 align:middle line:90% It's not easy, but try it. 00:53:44.700 --> 00:53:46.045 align:middle line:90% Yeah? 00:53:46.045 --> 00:53:47.420 align:middle line:84% AUDIENCE: Is the number of epochs 00:53:47.420 --> 00:53:50.330 align:middle line:84% has to do with it's related to the number of batches? 00:53:50.330 --> 00:53:52.253 align:middle line:84% Because you put 64 batches and then epoch. 00:53:52.253 --> 00:53:53.170 align:middle line:90% RAMA RAMAKRISHNAN: No. 00:53:53.170 --> 00:53:58.370 align:middle line:84% The epochs is just the number of passes through the whole data. 00:53:58.370 --> 00:54:01.330 align:middle line:84% But within each pass, within each epoch, 00:54:01.330 --> 00:54:03.690 align:middle line:84% the batch size tells you how many batches 00:54:03.690 --> 00:54:05.050 align:middle line:90% you're going to process. 00:54:05.050 --> 00:54:06.970 align:middle line:84% So it is basically the number of examples 00:54:06.970 --> 00:54:10.010 align:middle line:84% you have in your training data divided by the batch size 00:54:10.010 --> 00:54:11.770 align:middle line:90% that you have chosen. 00:54:11.770 --> 00:54:14.250 align:middle line:84% That number rounded up is the number 00:54:14.250 --> 00:54:16.865 align:middle line:90% of batches within each epoch. 00:54:16.865 --> 00:54:18.490 align:middle line:84% And here I'm just choosing 10 because-- 00:54:18.490 --> 00:54:19.930 align:middle line:90% SIRI: I found this on the web. 00:54:19.930 --> 00:54:23.090 align:middle line:84% RAMA RAMAKRISHNAN: Siri found something on the web. 00:54:23.090 --> 00:54:24.890 align:middle line:84% I chose 10 because it's going to be fast 00:54:24.890 --> 00:54:26.070 align:middle line:90% for me to do it in class. 00:54:26.070 --> 00:54:27.528 align:middle line:84% And 10 is actually more than enough 00:54:27.528 --> 00:54:31.050 align:middle line:84% because you can see it's already beginning to overfit. 00:54:31.050 --> 00:54:33.730 align:middle line:90% Yeah? 00:54:33.730 --> 00:54:36.250 align:middle line:84% AUDIENCE: This is more of a conceptual question, but is it 00:54:36.250 --> 00:54:38.330 align:middle line:84% always the case that a neural network will 00:54:38.330 --> 00:54:42.510 align:middle line:84% have better accuracy than its machine learning algorithm? 00:54:42.510 --> 00:54:45.563 align:middle line:84% And I'm asking more on the case of the heart disease problem. 00:54:45.563 --> 00:54:46.730 align:middle line:90% RAMA RAMAKRISHNAN: Oh, yeah. 00:54:46.730 --> 00:54:47.590 align:middle line:90% Great question. 00:54:47.590 --> 00:54:50.415 align:middle line:84% So neural networks are really good for unstructured data, 00:54:50.415 --> 00:54:51.790 align:middle line:84% like the ones we are having here. 00:54:51.790 --> 00:54:54.370 align:middle line:84% But if you have structured data like the heart disease problem, 00:54:54.370 --> 00:54:56.037 align:middle line:84% sometimes it actually works really well. 00:54:56.037 --> 00:54:59.810 align:middle line:84% Sometimes things like gradient boosting, XGBoost, 00:54:59.810 --> 00:55:00.970 align:middle line:90% work really well. 00:55:00.970 --> 00:55:03.550 align:middle line:84% So if I am actually working on a structured data problem, 00:55:03.550 --> 00:55:04.570 align:middle line:90% I'll try both. 00:55:04.570 --> 00:55:06.130 align:middle line:84% I'm not going to axiomatically assume 00:55:06.130 --> 00:55:07.922 align:middle line:84% that the DNN is going to be the best thing. 00:55:07.922 --> 00:55:10.590 align:middle line:84% But if you have structured data, it's the best game in town. 00:55:10.590 --> 00:55:13.865 align:middle line:90% 00:55:13.865 --> 00:55:15.490 align:middle line:84% By the way, I have a whole section here 00:55:15.490 --> 00:55:17.907 align:middle line:84% on once you build a model, how do you actually improve it? 00:55:17.907 --> 00:55:18.870 align:middle line:90% Check it out. 00:55:18.870 --> 00:55:20.450 align:middle line:90% It's an optional thing. 00:55:20.450 --> 00:55:23.070 align:middle line:84% All right, I'm going to stop this here. 00:55:23.070 --> 00:55:25.570 align:middle line:84% So the next thing I want to do is, 00:55:25.570 --> 00:55:28.650 align:middle line:84% so we went from 88% to 90-plus percent using convolutional 00:55:28.650 --> 00:55:29.970 align:middle line:90% networks. 00:55:29.970 --> 00:55:31.590 align:middle line:84% Now let's work with color images. 00:55:31.590 --> 00:55:33.050 align:middle line:90% Let's kick it up a notch. 00:55:33.050 --> 00:55:40.010 align:middle line:84% So I actually web scraped all these pictures for you folks 00:55:40.010 --> 00:55:41.170 align:middle line:90% for your enjoyment. 00:55:41.170 --> 00:55:45.490 align:middle line:84% I web scraped about 100 color images of handbags and shoes, 00:55:45.490 --> 00:55:47.350 align:middle line:90% roughly 100 handbags and shoes. 00:55:47.350 --> 00:55:51.200 align:middle line:84% So the question is, with these essentially 200 images, 00:55:51.200 --> 00:55:53.040 align:middle line:84% can we build a really good neural network 00:55:53.040 --> 00:55:54.780 align:middle line:90% to classify handbags and shoes? 00:55:54.780 --> 00:55:57.860 align:middle line:84% It seems kind of absurd because 200 examples, 00:55:57.860 --> 00:55:59.020 align:middle line:90% it's not that much. 00:55:59.020 --> 00:56:00.200 align:middle line:90% It doesn't feel like a lot. 00:56:00.200 --> 00:56:05.520 align:middle line:84% The MNIST data, Fashion MNIST, has 60,000 images. 00:56:05.520 --> 00:56:08.220 align:middle line:84% Even with that, we are overfitting in 5, 6, 7, 00:56:08.220 --> 00:56:09.200 align:middle line:90% 8 epochs. 00:56:09.200 --> 00:56:11.420 align:middle line:84% With 200 images, maybe, is there any hope? 00:56:11.420 --> 00:56:12.560 align:middle line:90% Obviously, there is hope. 00:56:12.560 --> 00:56:14.260 align:middle line:84% Otherwise it wouldn't be in the lecture. 00:56:14.260 --> 00:56:15.760 align:middle line:84% So we're going to take this data set 00:56:15.760 --> 00:56:17.220 align:middle line:84% and let's see what we can do with it. 00:56:17.220 --> 00:56:19.400 align:middle line:84% So we will first actually build a convolutional network 00:56:19.400 --> 00:56:20.858 align:middle line:84% from scratch to solve this problem. 00:56:20.858 --> 00:56:24.370 align:middle line:90% 00:56:24.370 --> 00:56:26.120 align:middle line:84% I'm actually going to run through the code 00:56:26.120 --> 00:56:28.540 align:middle line:84% because at the end of it, we'll have a live demo. 00:56:28.540 --> 00:56:30.800 align:middle line:84% So I would like one volunteer to give me 00:56:30.800 --> 00:56:32.800 align:middle line:84% a handbag and one volunteer to give me 00:56:32.800 --> 00:56:35.750 align:middle line:84% their footwear [INAUDIBLE] in class. 00:56:35.750 --> 00:56:39.720 align:middle line:90% 00:56:39.720 --> 00:56:41.640 align:middle line:84% Unlike the previous data set, this one 00:56:41.640 --> 00:56:44.980 align:middle line:90% actually, I just web scraped it. 00:56:44.980 --> 00:56:47.440 align:middle line:84% I've stuck it in this Dropbox folder. 00:56:47.440 --> 00:56:49.140 align:middle line:84% Let's just download it and unzip it. 00:56:49.140 --> 00:56:52.520 align:middle line:84% And once we do that, we have to now organize it 00:56:52.520 --> 00:56:53.900 align:middle line:90% with these 200 images. 00:56:53.900 --> 00:57:00.520 align:middle line:84% So I have to do some boring-ish Python stuff here. 00:57:00.520 --> 00:57:03.540 align:middle line:84% So here what we're doing is that we have 100 handbags, 00:57:03.540 --> 00:57:04.700 align:middle line:90% roughly 100 shoes. 00:57:04.700 --> 00:57:06.800 align:middle line:84% And what this code is doing is it's actually 00:57:06.800 --> 00:57:09.040 align:middle line:84% creating a directory of saying, it's 00:57:09.040 --> 00:57:11.580 align:middle line:84% splitting stuff into train, and validation, and test. 00:57:11.580 --> 00:57:13.200 align:middle line:84% And then for each of the splits, it's 00:57:13.200 --> 00:57:15.020 align:middle line:84% doing the handbags and the shoes folder. 00:57:15.020 --> 00:57:19.320 align:middle line:84% So once we do that, basically this directory structure 00:57:19.320 --> 00:57:21.320 align:middle line:90% is created. 00:57:21.320 --> 00:57:24.200 align:middle line:84% Training, validation folder, test folder, handbags, 00:57:24.200 --> 00:57:24.860 align:middle line:90% and shoes. 00:57:24.860 --> 00:57:28.040 align:middle line:84% In fact, actually, I think you can see it here, see here, 00:57:28.040 --> 00:57:29.360 align:middle line:90% handbags and shoes. 00:57:29.360 --> 00:57:32.180 align:middle line:84% And within that, there is train, test, validation. 00:57:32.180 --> 00:57:34.440 align:middle line:84% And within each of these there's handbags and shoes. 00:57:34.440 --> 00:57:37.040 align:middle line:84% So the idea is that when you're working with images, what 00:57:37.040 --> 00:57:38.920 align:middle line:84% you can do is you can just create folders 00:57:38.920 --> 00:57:40.400 align:middle line:90% for each kind of image. 00:57:40.400 --> 00:57:43.820 align:middle line:84% Let's say dogs, cats, two folders with cat images and dog 00:57:43.820 --> 00:57:44.320 align:middle line:90% images. 00:57:44.320 --> 00:57:46.410 align:middle line:90% And then just point Keras at it. 00:57:46.410 --> 00:57:47.910 align:middle line:84% It'll automatically figure out those 00:57:47.910 --> 00:57:50.370 align:middle line:84% are the labels that it needs to use. 00:57:50.370 --> 00:57:52.870 align:middle line:84% So it's very convenient when you're working with images. 00:57:52.870 --> 00:57:56.210 align:middle line:84% And the book explains this thing in great detail. 00:57:56.210 --> 00:57:57.967 align:middle line:84% So when working with these color images 00:57:57.967 --> 00:57:59.050 align:middle line:90% we'll follow this process. 00:57:59.050 --> 00:58:00.370 align:middle line:90% We'll read in the JPEGs. 00:58:00.370 --> 00:58:01.652 align:middle line:90% We'll convert them to tensors. 00:58:01.652 --> 00:58:03.110 align:middle line:84% And then since I'm web scraping it, 00:58:03.110 --> 00:58:04.943 align:middle line:84% they all come in different shapes and sizes. 00:58:04.943 --> 00:58:07.470 align:middle line:84% So I need to bring it all to the same size. 00:58:07.470 --> 00:58:10.370 align:middle line:84% I resize it, and then I'm going to batch it into whatever. 00:58:10.370 --> 00:58:13.390 align:middle line:84% I'm going to batch it using a batch size of 32 here. 00:58:13.390 --> 00:58:17.670 align:middle line:84% And this utility from Keras will do all that for you 00:58:17.670 --> 00:58:19.572 align:middle line:90% very quickly. 00:58:19.572 --> 00:58:21.030 align:middle line:84% So basically what it says is that I 00:58:21.030 --> 00:58:25.610 align:middle line:84% found 98 images in the training data belonging to two classes, 00:58:25.610 --> 00:58:28.210 align:middle line:84% 49 in the validation, and 38 in the test. 00:58:28.210 --> 00:58:30.170 align:middle line:84% So less than 100 examples in the training set. 00:58:30.170 --> 00:58:32.263 align:middle line:90% That's what we have here. 00:58:32.263 --> 00:58:32.930 align:middle line:90% What's the time? 00:58:32.930 --> 00:58:34.950 align:middle line:90% 9:38. 00:58:34.950 --> 00:58:37.610 align:middle line:84% Now let's just check the dimensions to make sure. 00:58:37.610 --> 00:58:38.110 align:middle line:90% Good. 00:58:38.110 --> 00:58:39.650 align:middle line:90% So 224, 224 by 3. 00:58:39.650 --> 00:58:41.853 align:middle line:84% And the reason why did I pick 224 to 24? 00:58:41.853 --> 00:58:43.270 align:middle line:84% As you will see later, we're going 00:58:43.270 --> 00:58:45.100 align:middle line:90% to use something called ResNet. 00:58:45.100 --> 00:58:48.320 align:middle line:84% And the ResNet expects it to be 224 by 224 by 3. 00:58:48.320 --> 00:58:50.740 align:middle line:84% That's why I resize it to 224, 224. 00:58:50.740 --> 00:58:53.820 align:middle line:84% Let's look at a few examples of my wonderful web scraping 00:58:53.820 --> 00:58:56.160 align:middle line:90% in action. 00:58:56.160 --> 00:58:56.660 align:middle line:90% Oh. 00:58:56.660 --> 00:59:01.140 align:middle line:90% 00:59:01.140 --> 00:59:04.060 align:middle line:90% It's pretty wild, right? 00:59:04.060 --> 00:59:07.020 align:middle line:84% Now let's do a simple convolutional network. 00:59:07.020 --> 00:59:10.140 align:middle line:84% And before, we would take all the x values in Fashion MNIST 00:59:10.140 --> 00:59:13.407 align:middle line:84% and divide them manually by 255 to normalize it to 0, 1. 00:59:13.407 --> 00:59:14.240 align:middle line:90% Well, you know what? 00:59:14.240 --> 00:59:16.573 align:middle line:84% We are actually graduating to the higher levels of Keras 00:59:16.573 --> 00:59:17.780 align:middle line:90% now so let's not do that. 00:59:17.780 --> 00:59:18.960 align:middle line:90% Manual stuff is bad. 00:59:18.960 --> 00:59:21.193 align:middle line:84% So we'll do it within Keras by using something called 00:59:21.193 --> 00:59:22.860 align:middle line:84% a Rescaling layer, where we just tell it 00:59:22.860 --> 00:59:25.540 align:middle line:84% how much to rescale, and boom, it will do it for you. 00:59:25.540 --> 00:59:28.580 align:middle line:84% The first convolutional block, just like the Fashion MNIST, 32. 00:59:28.580 --> 00:59:30.740 align:middle line:90% Second block, again, 32. 00:59:30.740 --> 00:59:32.240 align:middle line:90% Maxpool, flatten. 00:59:32.240 --> 00:59:34.520 align:middle line:84% And then here we only have handbags versus shoes. 00:59:34.520 --> 00:59:35.940 align:middle line:90% Just a sigmoid is enough. 00:59:35.940 --> 00:59:37.760 align:middle line:84% It's just a binary classification problem. 00:59:37.760 --> 00:59:40.480 align:middle line:84% So I'm just using one output layer with a sigmoid 00:59:40.480 --> 00:59:41.660 align:middle line:90% and that's our model. 00:59:41.660 --> 00:59:44.520 align:middle line:90% So let's do the model. 00:59:44.520 --> 00:59:45.520 align:middle line:90% model.summary. 00:59:45.520 --> 00:59:49.620 align:middle line:90% 00:59:49.620 --> 00:59:52.980 align:middle line:84% 101,000 parameters in this little model. 00:59:52.980 --> 00:59:55.260 align:middle line:90% OK, let's compile it and run it. 00:59:55.260 --> 00:59:58.580 align:middle line:84% And note here, because it's a binary classification problem 00:59:58.580 --> 01:00:01.660 align:middle line:90% I'm using binary cross entropy. 01:00:01.660 --> 01:00:05.060 align:middle line:84% And this is the same Adam and accuracy compile. 01:00:05.060 --> 01:00:06.800 align:middle line:90% And then, boom, let's run it. 01:00:06.800 --> 01:00:08.540 align:middle line:90% We'll do it for 20 epochs. 01:00:08.540 --> 01:00:12.482 align:middle line:84% Hopefully it won't take too long. 01:00:12.482 --> 01:00:13.940 align:middle line:84% While it's doing this business, I'm 01:00:13.940 --> 01:00:17.780 align:middle line:84% going to shift to the PowerPoint. 01:00:17.780 --> 01:00:19.800 align:middle line:84% So we'll go back to see how well it did. 01:00:19.800 --> 01:00:23.040 align:middle line:84% But the question is, whatever it did, we built it from scratch. 01:00:23.040 --> 01:00:25.460 align:middle line:84% So the question is, can we do better than that? 01:00:25.460 --> 01:00:27.640 align:middle line:84% Because we only have 100 examples of each class, 01:00:27.640 --> 01:00:30.140 align:middle line:84% and which brings us to something very cool and very powerful 01:00:30.140 --> 01:00:32.030 align:middle line:90% called transfer learning. 01:00:32.030 --> 01:00:33.780 align:middle line:84% So the key thing is there are two research 01:00:33.780 --> 01:00:35.905 align:middle line:84% trends that are going on that we take advantage of. 01:00:35.905 --> 01:00:37.340 align:middle line:84% The first one is that researchers 01:00:37.340 --> 01:00:41.810 align:middle line:84% have designed architectures which exploit the kind of input 01:00:41.810 --> 01:00:42.310 align:middle line:90% you have. 01:00:42.310 --> 01:00:43.810 align:middle line:84% So Olivia asked the question, if you 01:00:43.810 --> 01:00:45.485 align:middle line:84% have a particular kind of input images, 01:00:45.485 --> 01:00:47.610 align:middle line:84% do you actually change the input or do you actually 01:00:47.610 --> 01:00:48.670 align:middle line:90% change the network? 01:00:48.670 --> 01:00:50.830 align:middle line:84% As it turns out here, for example, if it's images, 01:00:50.830 --> 01:00:52.290 align:middle line:84% we know that we should use convolutional layers 01:00:52.290 --> 01:00:54.890 align:middle line:84% because convolutional layers were designed to exploit 01:00:54.890 --> 01:00:57.770 align:middle line:90% the image nest of the input. 01:00:57.770 --> 01:01:01.290 align:middle line:84% Similarly, if you have sequences of information, obviously 01:01:01.290 --> 01:01:03.588 align:middle line:84% natural language, audio, video, gene sequence, 01:01:03.588 --> 01:01:05.130 align:middle line:84% and so on, and so forth, these things 01:01:05.130 --> 01:01:07.942 align:middle line:84% called transformers were invented to exploit them. 01:01:07.942 --> 01:01:10.650 align:middle line:84% And we'll spend a lot of time on transformers starting next week. 01:01:10.650 --> 01:01:12.110 align:middle line:90% So that's the first trend. 01:01:12.110 --> 01:01:14.050 align:middle line:84% The second trend is that researchers 01:01:14.050 --> 01:01:18.730 align:middle line:84% have used these innovations to actually create and train models 01:01:18.730 --> 01:01:20.450 align:middle line:90% on vast data sets. 01:01:20.450 --> 01:01:23.410 align:middle line:84% And thankfully, they have made them publicly available 01:01:23.410 --> 01:01:24.610 align:middle line:90% for us to use. 01:01:24.610 --> 01:01:27.210 align:middle line:84% So transfer learning is the idea that if you 01:01:27.210 --> 01:01:28.850 align:middle line:84% have a particular problem, let's just 01:01:28.850 --> 01:01:30.570 align:middle line:84% take a pre-trained network somebody may 01:01:30.570 --> 01:01:32.310 align:middle line:90% have already created. 01:01:32.310 --> 01:01:34.650 align:middle line:84% And then let's just customize it to our problem 01:01:34.650 --> 01:01:37.090 align:middle line:84% rather than actually build anything from scratch. 01:01:37.090 --> 01:01:38.610 align:middle line:90% That's the basic idea. 01:01:38.610 --> 01:01:41.245 align:middle line:90% So here, we have this. 01:01:41.245 --> 01:01:43.120 align:middle line:84% Basically we have to build a classifier which 01:01:43.120 --> 01:01:45.000 align:middle line:84% takes in an arbitrary image and figures out 01:01:45.000 --> 01:01:46.220 align:middle line:90% if it's a handbag or a shoe. 01:01:46.220 --> 01:01:48.640 align:middle line:90% That's our goal. 01:01:48.640 --> 01:01:50.740 align:middle line:84% Now handbags and shoes are everyday objects. 01:01:50.740 --> 01:01:53.520 align:middle line:84% And so what you can do is you can look around and see 01:01:53.520 --> 01:01:55.520 align:middle line:84% if there are any networks that have been trained 01:01:55.520 --> 01:01:58.560 align:middle line:84% by other people which actually have been trained 01:01:58.560 --> 01:02:02.620 align:middle line:84% on everyday images, as opposed to MRI, or x-rays, 01:02:02.620 --> 01:02:04.120 align:middle line:84% specialized images, everyday images. 01:02:04.120 --> 01:02:06.162 align:middle line:84% Of course, the first thing you should probably do 01:02:06.162 --> 01:02:09.060 align:middle line:84% is to see if anybody has built the specific thing you want, 01:02:09.060 --> 01:02:10.800 align:middle line:84% handbag, shoes classifier on GitHub. 01:02:10.800 --> 01:02:13.920 align:middle line:84% Assuming it's not, then you do transfer learning. 01:02:13.920 --> 01:02:17.840 align:middle line:84% So now it turns out that there is this thing called ImageNet 01:02:17.840 --> 01:02:21.400 align:middle line:84% which is a database of millions of images of everyday objects 01:02:21.400 --> 01:02:25.420 align:middle line:84% in 1,000 different categories, furniture, animals, automobiles. 01:02:25.420 --> 01:02:26.340 align:middle line:90% You get the idea. 01:02:26.340 --> 01:02:29.280 align:middle line:84% And so we can look for networks that 01:02:29.280 --> 01:02:31.858 align:middle line:90% have been trained on ImageNet. 01:02:31.858 --> 01:02:33.400 align:middle line:84% Let me just go back to the Colab just 01:02:33.400 --> 01:02:34.775 align:middle line:84% to make sure it doesn't time out. 01:02:34.775 --> 01:02:38.180 align:middle line:90% 01:02:38.180 --> 01:02:40.680 align:middle line:90% So it has finished doing it. 01:02:40.680 --> 01:02:41.890 align:middle line:90% Let's just plot these things. 01:02:41.890 --> 01:02:48.680 align:middle line:90% 01:02:48.680 --> 01:02:52.920 align:middle line:84% So there is some overfitting that happens around here 01:02:52.920 --> 01:02:55.000 align:middle line:84% on the training and the 10th epoch. 01:02:55.000 --> 01:02:58.640 align:middle line:90% Let's look at the history. 01:02:58.640 --> 01:02:59.460 align:middle line:90% OK, interesting. 01:02:59.460 --> 01:03:03.440 align:middle line:84% So the training accuracy is actually getting almost to 100%. 01:03:03.440 --> 01:03:06.100 align:middle line:84% But we're not impressed by training accuracy. 01:03:06.100 --> 01:03:08.660 align:middle line:84% We care about validation and test accuracy. 01:03:08.660 --> 01:03:11.840 align:middle line:84% And that seems to be hovering around in the 80s. 01:03:11.840 --> 01:03:15.240 align:middle line:84% So let's just evaluate it anyway to see what happens. 01:03:15.240 --> 01:03:21.040 align:middle line:84% So it gets to 87% accuracy on this data set. 01:03:21.040 --> 01:03:24.040 align:middle line:84% It's actually pretty good, given that we only have 100 examples. 01:03:24.040 --> 01:03:26.980 align:middle line:84% So 87% accuracy, but we pre-trained the whole thing. 01:03:26.980 --> 01:03:28.770 align:middle line:84% Sorry, we did everything from scratch. 01:03:28.770 --> 01:03:31.800 align:middle line:90% 01:03:31.800 --> 01:03:34.420 align:middle line:84% There's this whole section about data augmentation, 01:03:34.420 --> 01:03:35.560 align:middle line:90% which, you know what? 01:03:35.560 --> 01:03:36.700 align:middle line:90% Do we have time? 01:03:36.700 --> 01:03:37.200 align:middle line:90% Yeah. 01:03:37.200 --> 01:03:40.110 align:middle line:90% 01:03:40.110 --> 01:03:44.350 align:middle line:84% So the idea of augmentation is that when you have an image, 01:03:44.350 --> 01:03:47.350 align:middle line:84% let's say you take this image and you just rotate it slightly 01:03:47.350 --> 01:03:49.230 align:middle line:90% by 10 degrees. 01:03:49.230 --> 01:03:52.150 align:middle line:84% If it's a handbag before you rotated it, it sure as hell 01:03:52.150 --> 01:03:55.110 align:middle line:84% is a handbag after you rotated it. 01:03:55.110 --> 01:03:55.910 align:middle line:90% It doesn't change. 01:03:55.910 --> 01:03:58.577 align:middle line:84% The meaning of the image doesn't change just because you rotated 01:03:58.577 --> 01:04:00.577 align:middle line:84% it slightly, or maybe you zoom in slightly, 01:04:00.577 --> 01:04:02.410 align:middle line:84% you zoom out slightly, you crop it slightly. 01:04:02.410 --> 01:04:03.670 align:middle line:90% Nothing happens. 01:04:03.670 --> 01:04:05.150 align:middle line:84% So what you can do is you can take 01:04:05.150 --> 01:04:10.090 align:middle line:84% any image you have and you just perturb it slightly, like that. 01:04:10.090 --> 01:04:14.156 align:middle line:84% And then add it as a new example to your training data. 01:04:14.156 --> 01:04:16.990 align:middle line:84% It's an unbelievable free lunch, frankly. 01:04:16.990 --> 01:04:19.630 align:middle line:84% And the same thing actually, the same kinds of techniques 01:04:19.630 --> 01:04:22.590 align:middle line:84% actually work for text also, which we'll cover later on. 01:04:22.590 --> 01:04:26.217 align:middle line:84% This broad area is called data augmentation. 01:04:26.217 --> 01:04:28.550 align:middle line:84% And it's a great way, when you don't have a lot of data, 01:04:28.550 --> 01:04:31.450 align:middle line:84% to artificially bolster the amount of data you have. 01:04:31.450 --> 01:04:34.030 align:middle line:90% 01:04:34.030 --> 01:04:35.550 align:middle line:84% Of course, Keras makes it very easy 01:04:35.550 --> 01:04:37.050 align:middle line:90% for you to do all these things. 01:04:37.050 --> 01:04:40.670 align:middle line:84% It has already predefined a whole bunch of data augmentation 01:04:40.670 --> 01:04:41.670 align:middle line:90% layers for you. 01:04:41.670 --> 01:04:44.910 align:middle line:84% So here's a little example where I basically take a picture 01:04:44.910 --> 01:04:46.610 align:middle line:90% and then I randomly flip it. 01:04:46.610 --> 01:04:50.430 align:middle line:84% So if it looks like this, I flip it this way horizontal. 01:04:50.430 --> 01:04:52.770 align:middle line:84% And then I randomly rotate it by 0.1. 01:04:52.770 --> 01:04:54.730 align:middle line:84% I forget if it's 0.1 degrees or radians. 01:04:54.730 --> 01:04:56.190 align:middle line:84% You can look up the documentation. 01:04:56.190 --> 01:05:00.110 align:middle line:84% And then random zoom, zoom in and out a little bit. 01:05:00.110 --> 01:05:03.030 align:middle line:84% But it won't do this for every picture. 01:05:03.030 --> 01:05:04.723 align:middle line:90% You only do it randomly. 01:05:04.723 --> 01:05:07.390 align:middle line:84% So there are only some pictures will get perturbed in some ways. 01:05:07.390 --> 01:05:08.390 align:middle line:84% And that's how you make sure there's 01:05:08.390 --> 01:05:10.630 align:middle line:84% enough diversity of pictures that you have. 01:05:10.630 --> 01:05:13.630 align:middle line:84% So once you do that, you can actually take a picture 01:05:13.630 --> 01:05:15.825 align:middle line:90% and see what it does. 01:05:15.825 --> 01:05:18.450 align:middle line:84% I just randomly grab a picture so it keeps changing every time. 01:05:18.450 --> 01:05:21.050 align:middle line:90% 01:05:21.050 --> 01:05:21.550 align:middle line:90% Yay. 01:05:21.550 --> 01:05:22.910 align:middle line:90% Look at this handbag. 01:05:22.910 --> 01:05:26.510 align:middle line:84% Handbag slightly rotated this way, rotated that way, 01:05:26.510 --> 01:05:29.690 align:middle line:84% some more, maybe a little bit of zooming going on, and so on. 01:05:29.690 --> 01:05:30.683 align:middle line:90% You get the idea. 01:05:30.683 --> 01:05:32.850 align:middle line:84% And there's a whole list of these things you can do. 01:05:32.850 --> 01:05:36.837 align:middle line:84% But when you do those things, make sure that what you're doing 01:05:36.837 --> 01:05:38.420 align:middle line:84% doesn't actually change the underlying 01:05:38.420 --> 01:05:39.740 align:middle line:90% meaning of the picture. 01:05:39.740 --> 01:05:42.220 align:middle line:90% It's really important. 01:05:42.220 --> 01:05:45.732 align:middle line:84% So for example, if you're working with satellite data, 01:05:45.732 --> 01:05:49.162 align:middle line:84% you must be very careful not to do flips of crazy flips, 01:05:49.162 --> 01:05:51.120 align:middle line:84% or even if you're working with everyday images, 01:05:51.120 --> 01:05:52.520 align:middle line:90% horizontal flips are OK. 01:05:52.520 --> 01:05:54.728 align:middle line:90% Don't do vertical flips. 01:05:54.728 --> 01:05:57.020 align:middle line:84% How many times will you have an upside down dog picture 01:05:57.020 --> 01:05:59.010 align:middle line:90% that you need to classify? 01:05:59.010 --> 01:06:01.050 align:middle line:84% Make sure your augmentation doesn't go nuts. 01:06:01.050 --> 01:06:05.715 align:middle line:90% 01:06:05.715 --> 01:06:07.340 align:middle line:84% Once you do that, you can actually just 01:06:07.340 --> 01:06:09.740 align:middle line:84% insert the data augmentation layers in your model 01:06:09.740 --> 01:06:11.800 align:middle line:84% right there, right after the input. 01:06:11.800 --> 01:06:14.300 align:middle line:84% The rest of it can stay unchanged. 01:06:14.300 --> 01:06:15.740 align:middle line:84% So this is a great way to increase 01:06:15.740 --> 01:06:17.200 align:middle line:90% the size of your training data. 01:06:17.200 --> 01:06:18.360 align:middle line:90% And here is a model. 01:06:18.360 --> 01:06:20.900 align:middle line:84% And then I invite you to actually just play with it 01:06:20.900 --> 01:06:21.818 align:middle line:90% and train it. 01:06:21.818 --> 01:06:24.360 align:middle line:84% In the interest of time, you won't actually train this model. 01:06:24.360 --> 01:06:24.960 align:middle line:90% But it's in the Colab. 01:06:24.960 --> 01:06:25.793 align:middle line:90% You can just try it. 01:06:25.793 --> 01:06:28.080 align:middle line:84% It also figures prominently in homework 1, 01:06:28.080 --> 01:06:29.400 align:middle line:90% by the way, data augmentation. 01:06:29.400 --> 01:06:31.620 align:middle line:84% So you'll get more experience with this. 01:06:31.620 --> 01:06:34.890 align:middle line:90% So back to the PPT. 01:06:34.890 --> 01:06:36.690 align:middle line:90% So this is what we have. 01:06:36.690 --> 01:06:40.370 align:middle line:84% And so any network that has been trained on this ImageNet thing, 01:06:40.370 --> 01:06:42.970 align:middle line:84% turns out, learns all kinds of interesting features 01:06:42.970 --> 01:06:44.350 align:middle line:90% in every one of its layers. 01:06:44.350 --> 01:06:45.910 align:middle line:90% So here this is the first layer. 01:06:45.910 --> 01:06:47.930 align:middle line:84% And you can see it's picking up gradations 01:06:47.930 --> 01:06:50.810 align:middle line:84% of color, line-ish kind of behavior. 01:06:50.810 --> 01:06:53.650 align:middle line:84% Layer 2 is actually picking up, hey, look, 01:06:53.650 --> 01:06:55.170 align:middle line:90% it's picking up an edge. 01:06:55.170 --> 01:06:59.610 align:middle line:84% Can you see that edge, like that? 01:06:59.610 --> 01:07:02.810 align:middle line:84% And then layer 3 is picking up these interesting honeycomb 01:07:02.810 --> 01:07:04.830 align:middle line:90% shapes, and so on. 01:07:04.830 --> 01:07:07.850 align:middle line:84% Oh, it's actually, this thing is already picking up 01:07:07.850 --> 01:07:09.080 align:middle line:90% the shape of a human torso. 01:07:09.080 --> 01:07:12.202 align:middle line:90% 01:07:12.202 --> 01:07:13.970 align:middle line:84% Yeah, this here is actually picking up 01:07:13.970 --> 01:07:17.290 align:middle line:84% what looks like a Labrador retriever. 01:07:17.290 --> 01:07:19.410 align:middle line:90% Isn't that cute? 01:07:19.410 --> 01:07:23.150 align:middle line:84% Come on, even if you're not a dog person. 01:07:23.150 --> 01:07:27.130 align:middle line:84% So this visualization I was referring to earlier, 01:07:27.130 --> 01:07:30.130 align:middle line:84% to figure out what are these networks actually learning, 01:07:30.130 --> 01:07:31.690 align:middle line:84% this paper was one of the first ones 01:07:31.690 --> 01:07:33.710 align:middle line:84% to actually visualize what's going on inside. 01:07:33.710 --> 01:07:36.168 align:middle line:84% So if you folks are curious how these pictures are actually 01:07:36.168 --> 01:07:39.275 align:middle line:84% produced, I would encourage you to check this out. 01:07:39.275 --> 01:07:39.775 align:middle line:90% Yeah? 01:07:39.775 --> 01:07:44.027 align:middle line:90% 01:07:44.027 --> 01:07:45.610 align:middle line:84% AUDIENCE: Sorry, we spoke about images 01:07:45.610 --> 01:07:49.310 align:middle line:84% and you referred to classes on text next week on transformers. 01:07:49.310 --> 01:07:52.050 align:middle line:84% But what about, say, an email which 01:07:52.050 --> 01:07:53.770 align:middle line:84% has both text and image, and there'll 01:07:53.770 --> 01:07:56.510 align:middle line:84% be whitespace depending on who has written it? 01:07:56.510 --> 01:08:00.918 align:middle line:84% How does that get put in as an input for an image? 01:08:00.918 --> 01:08:03.210 align:middle line:84% RAMA RAMAKRISHNAN: So we'll revisit this great question 01:08:03.210 --> 01:08:04.890 align:middle line:90% a bit later on in the course. 01:08:04.890 --> 01:08:06.470 align:middle line:84% So the answer is a bit complicated 01:08:06.470 --> 01:08:07.950 align:middle line:90% so I want to do it justice. 01:08:07.950 --> 01:08:09.830 align:middle line:90% So we'll come back to it. 01:08:09.830 --> 01:08:12.170 align:middle line:84% So it turns out this thing called 01:08:12.170 --> 01:08:14.690 align:middle line:84% ResNet is a family of networks which are trained 01:08:14.690 --> 01:08:16.069 align:middle line:90% on this ImageNet data set. 01:08:16.069 --> 01:08:18.569 align:middle line:84% And they did really well in this competition 01:08:18.569 --> 01:08:21.290 align:middle line:84% that's associated with dimension data set called ImageNet. 01:08:21.290 --> 01:08:23.350 align:middle line:84% And so this is an example of such a network. 01:08:23.350 --> 01:08:27.470 align:middle line:84% So you would expect the weights and the parameters of ResNet, 01:08:27.470 --> 01:08:29.770 align:middle line:84% given that it's been trained on ImageNet, to have 01:08:29.770 --> 01:08:32.680 align:middle line:84% some knowledge about lines, and shapes, and curves, 01:08:32.680 --> 01:08:33.779 align:middle line:90% and things like that. 01:08:33.779 --> 01:08:37.700 align:middle line:90% So maybe we can just use that. 01:08:37.700 --> 01:08:40.399 align:middle line:84% But the thing is, we can't use ResNet as is because remember, 01:08:40.399 --> 01:08:42.720 align:middle line:84% it was trained to classify an incoming image in 1,000 01:08:42.720 --> 01:08:43.923 align:middle line:90% possibilities. 01:08:43.923 --> 01:08:45.840 align:middle line:84% Here, we only have two possibilities, handbags 01:08:45.840 --> 01:08:46.600 align:middle line:90% and shoes. 01:08:46.600 --> 01:08:49.319 align:middle line:84% So what we do is very simple and elegant. 01:08:49.319 --> 01:08:51.880 align:middle line:84% We do just a little bit of surgery. 01:08:51.880 --> 01:08:55.520 align:middle line:84% We take ResNet and stop just before the final layer. 01:08:55.520 --> 01:08:57.560 align:middle line:90% So take my word for it. 01:08:57.560 --> 01:08:59.399 align:middle line:84% This thing here, what it says is, 01:08:59.399 --> 01:09:03.060 align:middle line:84% fully connected 1,000 because it's got 1,000 way, 01:09:03.060 --> 01:09:03.979 align:middle line:90% 1,000 objects. 01:09:03.979 --> 01:09:06.200 align:middle line:84% So what we do is we just take everything except, 01:09:06.200 --> 01:09:08.279 align:middle line:84% and we stop just before that last layer. 01:09:08.279 --> 01:09:10.279 align:middle line:84% And then what comes out of that layer 01:09:10.279 --> 01:09:12.960 align:middle line:84% hopefully will be a very smart representation of the images 01:09:12.960 --> 01:09:14.520 align:middle line:90% that has been trained on. 01:09:14.520 --> 01:09:17.760 align:middle line:84% And so what we do is we can think of headless 01:09:17.760 --> 01:09:19.850 align:middle line:90% ResNet as our model. 01:09:19.850 --> 01:09:22.700 align:middle line:90% 01:09:22.700 --> 01:09:24.359 align:middle line:84% We can take all our data and run it 01:09:24.359 --> 01:09:28.740 align:middle line:84% through ResNet, up to but not including the last layer. 01:09:28.740 --> 01:09:30.069 align:middle line:90% You'll get some tensor. 01:09:30.069 --> 01:09:32.888 align:middle line:84% And that tensor probably has a very rich understanding 01:09:32.888 --> 01:09:34.430 align:middle line:84% of what's going on in that image, all 01:09:34.430 --> 01:09:36.569 align:middle line:84% the objects, and features, and things like that. 01:09:36.569 --> 01:09:39.553 align:middle line:84% And then we can just simply connect that. 01:09:39.553 --> 01:09:41.970 align:middle line:84% We can think of it as a smart representation of our input. 01:09:41.970 --> 01:09:44.370 align:middle line:84% We can connect it to just a little hidden layer. 01:09:44.370 --> 01:09:45.830 align:middle line:84% And then we have a little sigmoid 01:09:45.830 --> 01:09:47.609 align:middle line:90% which tells you handbag or shoe. 01:09:47.609 --> 01:09:50.350 align:middle line:90% We can just run this network. 01:09:50.350 --> 01:09:53.590 align:middle line:84% And so since the outputs to the hidden layer 01:09:53.590 --> 01:09:57.110 align:middle line:84% now are not raw images anymore, but this much higher level 01:09:57.110 --> 01:09:59.050 align:middle line:84% of abstraction that ResNet has learned, 01:09:59.050 --> 01:10:02.510 align:middle line:84% hopefully it can get the job done with hardly any examples. 01:10:02.510 --> 01:10:04.563 align:middle line:90% And now you can get fancier. 01:10:04.563 --> 01:10:06.730 align:middle line:84% That's the basic idea, but you can get much fancier. 01:10:06.730 --> 01:10:09.138 align:middle line:84% You can connect up headless ResNet directly 01:10:09.138 --> 01:10:10.930 align:middle line:84% with our little network with a hidden layer 01:10:10.930 --> 01:10:11.763 align:middle line:90% and the final thing. 01:10:11.763 --> 01:10:15.948 align:middle line:84% And the whole thing can be trained end to end. 01:10:15.948 --> 01:10:17.990 align:middle line:84% But when you do that, you must start the training 01:10:17.990 --> 01:10:20.123 align:middle line:84% with the weights that you downloaded with ResNet 01:10:20.123 --> 01:10:22.290 align:middle line:84% because that is the crown jewel that's been learned. 01:10:22.290 --> 01:10:24.070 align:middle line:84% So you'll want to start from there. 01:10:24.070 --> 01:10:27.570 align:middle line:84% And you will do this in homework 1. 01:10:27.570 --> 01:10:29.070 align:middle line:84% By the way, these pre-trained models 01:10:29.070 --> 01:10:30.570 align:middle line:84% are available all over the internet. 01:10:30.570 --> 01:10:32.570 align:middle line:84% There is the TensorFlow Hub, the PyTorch Hub, 01:10:32.570 --> 01:10:34.195 align:middle line:84% and then there is the Hugging Face Hub. 01:10:34.195 --> 01:10:36.110 align:middle line:84% When I checked it on the 13th, yesterday, 01:10:36.110 --> 01:10:39.770 align:middle line:84% it had over half a million models available for download, 01:10:39.770 --> 01:10:41.270 align:middle line:90% half a million. 01:10:41.270 --> 01:10:44.750 align:middle line:84% I think last year it was 50,000 when I taught the course. 01:10:44.750 --> 01:10:46.990 align:middle line:90% Yes? 01:10:46.990 --> 01:10:49.070 align:middle line:84% AUDIENCE: I was just wondering, doesn't this 01:10:49.070 --> 01:10:51.510 align:middle line:84% make you prone to adversarial attacks 01:10:51.510 --> 01:10:54.290 align:middle line:84% because the weights are known by everybody? 01:10:54.290 --> 01:10:55.390 align:middle line:90% RAMA RAMAKRISHNAN: Yes. 01:10:55.390 --> 01:10:56.890 align:middle line:90% There is some adversarial risk. 01:10:56.890 --> 01:10:59.230 align:middle line:84% I'm happy to talk about it offline. 01:10:59.230 --> 01:10:59.810 align:middle line:90% All right. 01:10:59.810 --> 01:11:02.790 align:middle line:84% So that's what we have, so back to Colab. 01:11:02.790 --> 01:11:04.290 align:middle line:90% So that's what we have. 01:11:04.290 --> 01:11:06.390 align:middle line:90% This is ResNet. 01:11:06.390 --> 01:11:07.650 align:middle line:90% And ResNet is all packaged up. 01:11:07.650 --> 01:11:10.560 align:middle line:84% It's available for download, so we download it here. 01:11:10.560 --> 01:11:13.710 align:middle line:90% 01:11:13.710 --> 01:11:19.430 align:middle line:84% And you see here that I'm saying, include_top=False. 01:11:19.430 --> 01:11:21.990 align:middle line:84% So basically, you are telling Keras, 01:11:21.990 --> 01:11:25.010 align:middle line:84% the top, the very final layer of the thing, don't give it to me. 01:11:25.010 --> 01:11:27.380 align:middle line:84% Just give me everything up to but not including that. 01:11:27.380 --> 01:11:29.660 align:middle line:84% And of course, I think of it as left to right. 01:11:29.660 --> 01:11:31.500 align:middle line:84% People think of it as bottom to top. 01:11:31.500 --> 01:11:33.833 align:middle line:84% So the very, very top layer, don't give it to me. 01:11:33.833 --> 01:11:35.500 align:middle line:84% You're telling it so that you don't have 01:11:35.500 --> 01:11:37.380 align:middle line:90% to manually go and remove it. 01:11:37.380 --> 01:11:39.697 align:middle line:84% And then I'm not going to summarize. 01:11:39.697 --> 01:11:42.280 align:middle line:84% Well, I'll just summarize the model to show you how big it is. 01:11:42.280 --> 01:11:45.860 align:middle line:90% 01:11:45.860 --> 01:11:49.820 align:middle line:90% 23 million parameters, ResNet. 01:11:49.820 --> 01:11:51.500 align:middle line:84% And I won't plot it because then I'll 01:11:51.500 --> 01:11:53.540 align:middle line:90% be scrolling for five minutes. 01:11:53.540 --> 01:11:54.960 align:middle line:90% So let's just do this now. 01:11:54.960 --> 01:11:56.100 align:middle line:84% So what we're now going to do is we're 01:11:56.100 --> 01:11:57.920 align:middle line:84% going to run all the data through this thing. 01:11:57.920 --> 01:12:00.000 align:middle line:84% And whatever comes out in that penultimate thing, 01:12:00.000 --> 01:12:01.900 align:middle line:84% I'm going to just grab it and store it. 01:12:01.900 --> 01:12:05.500 align:middle line:90% So that's what this thing does. 01:12:05.500 --> 01:12:06.820 align:middle line:90% And now we create-- 01:12:06.820 --> 01:12:09.580 align:middle line:84% this is a little, handy function to do all these things. 01:12:09.580 --> 01:12:11.900 align:middle line:84% And once I do that, every image has 01:12:11.900 --> 01:12:15.137 align:middle line:84% been sent through ResNet, up to but not the final layer. 01:12:15.137 --> 01:12:17.720 align:middle line:84% And then whatever comes to the final layer, we are storing it. 01:12:17.720 --> 01:12:19.637 align:middle line:84% And then we're going to create a network where 01:12:19.637 --> 01:12:23.300 align:middle line:84% we'll only feed that layer, that information to a simple network. 01:12:23.300 --> 01:12:26.120 align:middle line:90% So what is coming out of ResNet? 01:12:26.120 --> 01:12:29.360 align:middle line:84% You can see here 98 examples in the training data. 01:12:29.360 --> 01:12:33.608 align:middle line:84% And each example is now a 7-by-7-by-2,048 tensor. 01:12:33.608 --> 01:12:34.900 align:middle line:90% That's what came out of ResNet. 01:12:34.900 --> 01:12:38.200 align:middle line:84% And you saw that's what I did there. 01:12:38.200 --> 01:12:39.500 align:middle line:90% So that's what it looks like. 01:12:39.500 --> 01:12:42.440 align:middle line:84% Now let's just create our actual model now. 01:12:42.440 --> 01:12:46.540 align:middle line:84% We have our input, which is just a 7 by 7 by 224, 2,048. 01:12:46.540 --> 01:12:49.380 align:middle line:84% We flatten it immediately, then we run it 01:12:49.380 --> 01:12:51.960 align:middle line:84% through a dense layer with 256 ReLU neurons. 01:12:51.960 --> 01:12:54.600 align:middle line:84% And then we use Dropout, which I haven't talked about yet, 01:12:54.600 --> 01:12:57.557 align:middle line:84% which I will talk about early next week. 01:12:57.557 --> 01:12:58.640 align:middle line:90% And we'll come back to it. 01:12:58.640 --> 01:13:00.740 align:middle line:84% Don't worry about this detail for the moment. 01:13:00.740 --> 01:13:06.100 align:middle line:84% And then we just run through a sigmoid and that's our model. 01:13:06.100 --> 01:13:07.820 align:middle line:90% Finished, plot the model. 01:13:07.820 --> 01:13:09.720 align:middle line:90% This is what we have. 01:13:09.720 --> 01:13:14.360 align:middle line:84% model.summary, so on, and so forth. 01:13:14.360 --> 01:13:15.200 align:middle line:90% All right, good. 01:13:15.200 --> 01:13:18.780 align:middle line:84% Now let's actually train this thing. 01:13:18.780 --> 01:13:20.620 align:middle line:84% I'm just going to run it for 10 epochs 01:13:20.620 --> 01:13:22.800 align:middle line:84% because I tried running it previously 01:13:22.800 --> 01:13:25.510 align:middle line:84% and it seems to do a fine job with just 10 epochs. 01:13:25.510 --> 01:13:26.590 align:middle line:90% OK, it's already done. 01:13:26.590 --> 01:13:29.610 align:middle line:84% It's so fast because we ran everything through this monster 01:13:29.610 --> 01:13:33.488 align:middle line:84% ResNet thing, and basically took all the output values and used 01:13:33.488 --> 01:13:34.530 align:middle line:90% them as a starting point. 01:13:34.530 --> 01:13:36.490 align:middle line:84% We don't have to run it every single time. 01:13:36.490 --> 01:13:42.212 align:middle line:84% So you can see here, the accuracy is quite high. 01:13:42.212 --> 01:13:44.510 align:middle line:90% Let me just plot it. 01:13:44.510 --> 01:13:45.010 align:middle line:90% Interesting. 01:13:45.010 --> 01:13:48.570 align:middle line:84% So the 10th epoch, something bad happened. 01:13:48.570 --> 01:13:50.670 align:middle line:84% So maybe I should have stopped at the 9th epoch. 01:13:50.670 --> 01:13:52.628 align:middle line:84% I didn't see this yesterday when I was running. 01:13:52.628 --> 01:13:55.210 align:middle line:84% So much for random reproducibility. 01:13:55.210 --> 01:13:56.410 align:middle line:90% So let's just run this. 01:13:56.410 --> 01:13:56.910 align:middle line:90% Oh, wow. 01:13:56.910 --> 01:14:01.610 align:middle line:84% Look, on the test set, it's achieving 100% accuracy. 01:14:01.610 --> 01:14:03.690 align:middle line:90% It's unbelievable. 01:14:03.690 --> 01:14:06.410 align:middle line:84% OK, folks, now for the moment of truth. 01:14:06.410 --> 01:14:06.910 align:middle line:90% All right. 01:14:06.910 --> 01:14:08.285 align:middle line:84% I have a little code snippet here 01:14:08.285 --> 01:14:10.050 align:middle line:84% to capture stuff from the webcam. 01:14:10.050 --> 01:14:12.270 align:middle line:84% Because that last epoch it went down, 01:14:12.270 --> 01:14:14.970 align:middle line:84% I'm a little worried that the demo is going to flunk. 01:14:14.970 --> 01:14:15.950 align:middle line:90% But you know what? 01:14:15.950 --> 01:14:19.690 align:middle line:84% We all have to live dangerously, so here's a little function 01:14:19.690 --> 01:14:21.690 align:middle line:84% to predict what's going to happen. 01:14:21.690 --> 01:14:25.320 align:middle line:84% OK, now I tried it at home yesterday, by the way, 01:14:25.320 --> 01:14:28.040 align:middle line:84% and it's like, yeah, it's a handbag. 01:14:28.040 --> 01:14:30.800 align:middle line:84% OK, now let's just do something else. 01:14:30.800 --> 01:14:32.600 align:middle line:90% OK, any volunteers? 01:14:32.600 --> 01:14:35.570 align:middle line:84% I want a piece of footwear or a handbag. 01:14:35.570 --> 01:14:36.820 align:middle line:90% AUDIENCE: I'll give a handbag. 01:14:36.820 --> 01:14:39.195 align:middle line:84% RAMA RAMAKRISHNAN: That's kind of like a backpack, right? 01:14:39.195 --> 01:14:39.838 align:middle line:90% [LAUGHTER] 01:14:39.838 --> 01:14:40.380 align:middle line:90% I don't know. 01:14:40.380 --> 01:14:42.005 align:middle line:84% This feels like an adversarial example, 01:14:42.005 --> 01:14:43.560 align:middle line:90% but yeah, let's just try it. 01:14:43.560 --> 01:14:45.160 align:middle line:90% AUDIENCE: Someone else? 01:14:45.160 --> 01:14:46.660 align:middle line:84% RAMA RAMAKRISHNAN: No disrespect. 01:14:46.660 --> 01:14:48.440 align:middle line:90% Let me go with the shoe first. 01:14:48.440 --> 01:14:51.080 align:middle line:84% I have a better chance of it working. 01:14:51.080 --> 01:14:52.680 align:middle line:90% So it's a pretty big shoe. 01:14:52.680 --> 01:14:56.040 align:middle line:84% If it can't get this shoe, I'm worried about this model. 01:14:56.040 --> 01:14:56.660 align:middle line:90% All right. 01:14:56.660 --> 01:15:05.200 align:middle line:90% 01:15:05.200 --> 01:15:05.700 align:middle line:90% OK. 01:15:05.700 --> 01:15:08.000 align:middle line:90% Hold on, hold on, hold on. 01:15:08.000 --> 01:15:10.800 align:middle line:90% All right. 01:15:10.800 --> 01:15:14.640 align:middle line:84% Please don't get distracted by my hand. 01:15:14.640 --> 01:15:17.040 align:middle line:90% Capture. 01:15:17.040 --> 01:15:18.040 align:middle line:90% It's a shoe! 01:15:18.040 --> 01:15:18.900 align:middle line:90% Look at that! 01:15:18.900 --> 01:15:21.840 align:middle line:90% [APPLAUSE] 01:15:21.840 --> 01:15:22.560 align:middle line:90% Phew. 01:15:22.560 --> 01:15:25.040 align:middle line:90% All right, thanks. 01:15:25.040 --> 01:15:26.340 align:middle line:90% OK, now let's try that. 01:15:26.340 --> 01:15:27.608 align:middle line:90% I'm feeling brave now. 01:15:27.608 --> 01:15:28.400 align:middle line:90% AUDIENCE: OK, yeah. 01:15:28.400 --> 01:15:29.608 align:middle line:90% RAMA RAMAKRISHNAN: Thank you. 01:15:29.608 --> 01:15:32.880 align:middle line:90% All right, let's do this. 01:15:32.880 --> 01:15:35.080 align:middle line:90% All right. 01:15:35.080 --> 01:15:35.810 align:middle line:90% Camera Capture. 01:15:35.810 --> 01:15:40.473 align:middle line:90% 01:15:40.473 --> 01:15:40.973 align:middle line:90% OK. 01:15:40.973 --> 01:15:43.691 align:middle line:90% 01:15:43.691 --> 01:15:45.235 align:middle line:90% I'll put its better side. 01:15:45.235 --> 01:15:45.860 align:middle line:90% AUDIENCE: Yeah. 01:15:45.860 --> 01:15:54.883 align:middle line:90% 01:15:54.883 --> 01:15:56.300 align:middle line:84% RAMA RAMAKRISHNAN: It's a handbag! 01:15:56.300 --> 01:15:57.411 align:middle line:90% Look at that! 01:15:57.411 --> 01:15:58.704 align:middle line:90% [APPLAUSE] 01:15:58.704 --> 01:16:00.000 align:middle line:90% Phew. 01:16:00.000 --> 01:16:03.455 align:middle line:84% I swear, every time I do the demo, I age a few years. 01:16:03.455 --> 01:16:04.580 align:middle line:90% All right, folks, I'm done. 01:16:04.580 --> 01:16:06.350 align:middle line:90% Thank you. 01:16:06.350 --> 01:16:22.000 align:middle line:90%