WEBVTT 00:00:00.040 --> 00:00:02.410 The following content is provided under a Creative 00:00:02.410 --> 00:00:03.790 Commons license. 00:00:03.790 --> 00:00:06.030 Your support will help MIT OpenCourseWare 00:00:06.030 --> 00:00:10.100 continue to offer high-quality educational resources for free. 00:00:10.100 --> 00:00:12.680 To make a donation or to view additional materials 00:00:12.680 --> 00:00:16.496 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.496 --> 00:00:17.120 at ocw.mit.edu. 00:00:30.885 --> 00:00:33.855 [MUSIC PLAYING] 00:00:37.780 --> 00:00:40.030 PATRICK H. WINSTON: Well, what we're going to do today 00:00:40.030 --> 00:00:42.110 is climb a pretty big mountain because we're 00:00:42.110 --> 00:00:43.920 going to go from a neural net with two 00:00:43.920 --> 00:00:47.940 parameters to discussing the kind of neural nets 00:00:47.940 --> 00:00:56.590 in which people end up dealing with 60 million parameters. 00:00:56.590 --> 00:00:59.176 So it's going to be a pretty big jump. 00:00:59.176 --> 00:01:00.550 Along the way are a couple things 00:01:00.550 --> 00:01:05.750 I wanted to underscore from our previous discussion. 00:01:05.750 --> 00:01:08.340 Last time, I tried to develop some intuition 00:01:08.340 --> 00:01:11.600 for the kinds of formulas that you use to actually do 00:01:11.600 --> 00:01:14.230 the calculations in a small neural net about how 00:01:14.230 --> 00:01:16.160 the weights are going to change. 00:01:16.160 --> 00:01:18.230 And the main thing I tried to emphasize 00:01:18.230 --> 00:01:30.420 is that when you have a neural net like this one, 00:01:30.420 --> 00:01:36.610 everything is sort of divided in each column. 00:01:36.610 --> 00:01:42.360 You can't have the performance based on this output 00:01:42.360 --> 00:01:44.940 affect some weight change back here 00:01:44.940 --> 00:01:49.150 without going through this finite number of output 00:01:49.150 --> 00:01:51.036 variables, the y1s. 00:01:51.036 --> 00:01:57.310 And by the way, there's no y2 and y4-- there's no y2 and y3. 00:01:57.310 --> 00:02:00.510 Dealing with this is really a notational nightmare, 00:02:00.510 --> 00:02:03.890 and I spent a lot of time yesterday 00:02:03.890 --> 00:02:06.149 trying to clean it up a little bit. 00:02:06.149 --> 00:02:07.690 But basically, what I'm trying to say 00:02:07.690 --> 00:02:09.959 has nothing to do with the notation I have used 00:02:09.959 --> 00:02:11.500 but rather with the fact that there's 00:02:11.500 --> 00:02:15.124 a limited number of ways in which that can influence this, 00:02:15.124 --> 00:02:17.290 even though the number of paths through this network 00:02:17.290 --> 00:02:19.930 can be growing exponential. 00:02:19.930 --> 00:02:22.840 So those equations underneath are 00:02:22.840 --> 00:02:26.420 equations that derive from trying to figure out 00:02:26.420 --> 00:02:31.690 how the output performance depends on some 00:02:31.690 --> 00:02:33.620 of these weights back here. 00:02:33.620 --> 00:02:36.180 And what I've calculated is I've calculated 00:02:36.180 --> 00:02:41.000 the dependence of the performance on w1 00:02:41.000 --> 00:02:44.260 going that way, and I've also calculated 00:02:44.260 --> 00:02:52.420 the dependence of performance on w1 going that way. 00:02:52.420 --> 00:02:55.910 So that's one of the equations I've got down there. 00:02:55.910 --> 00:02:58.830 And another one deals with w3, and it 00:02:58.830 --> 00:03:07.060 involves going both this way and this way. 00:03:07.060 --> 00:03:10.110 And all I've done in both cases, in all four cases, 00:03:10.110 --> 00:03:13.350 is just take the partial derivative of performance 00:03:13.350 --> 00:03:15.750 with respect to those weights and use the chain rule 00:03:15.750 --> 00:03:17.270 to expand it. 00:03:17.270 --> 00:03:25.732 And when I do that, this is the stuff I get. 00:03:25.732 --> 00:03:27.940 And that's just a whole bunch of partial derivatives. 00:03:27.940 --> 00:03:30.510 But if you look at it and let it sing a little bit to you, 00:03:30.510 --> 00:03:32.509 what you see is that there's a lot of redundancy 00:03:32.509 --> 00:03:34.270 in the computation. 00:03:34.270 --> 00:03:38.990 So for example, this guy here, partial 00:03:38.990 --> 00:03:41.800 of performance with respect to w1, 00:03:41.800 --> 00:03:45.350 depends on both paths, of course. 00:03:45.350 --> 00:03:51.150 But look at the first elements here, these guys right here. 00:03:51.150 --> 00:03:53.850 And look at the first elements in the expression 00:03:53.850 --> 00:03:56.210 for calculating the partial derivative of performance 00:03:56.210 --> 00:04:00.004 with respect to w3, these guys. 00:04:04.612 --> 00:04:05.320 They're the same. 00:04:07.920 --> 00:04:12.310 And not only that, if you look inside these expressions 00:04:12.310 --> 00:04:16.399 and look at this particular piece here, 00:04:16.399 --> 00:04:18.810 you see that that is an expression that 00:04:18.810 --> 00:04:21.779 was needed in order to calculate one 00:04:21.779 --> 00:04:24.710 of the downstream weights, the changes in one 00:04:24.710 --> 00:04:27.370 of the downstream weights. 00:04:27.370 --> 00:04:30.070 But it happens to be the same thing as you see over here. 00:04:32.780 --> 00:04:41.532 And likewise, this piece is the same thing you see over here. 00:04:44.630 --> 00:04:47.180 So each time you move further and further back 00:04:47.180 --> 00:04:49.290 from the outputs toward the inputs, 00:04:49.290 --> 00:04:51.270 you're reusing a lot of computation 00:04:51.270 --> 00:04:54.180 that you've already done. 00:04:54.180 --> 00:04:57.250 So I'm trying to find a way to sloganize this, 00:04:57.250 --> 00:05:02.280 and what I've come up with is what's done is done and cannot 00:05:02.280 --> 00:05:03.260 be-- no, no. 00:05:03.260 --> 00:05:05.080 That's not quite right, is it? 00:05:05.080 --> 00:05:10.090 It's what's computed is computed and need not be recomputed. 00:05:10.090 --> 00:05:10.719 OK? 00:05:10.719 --> 00:05:12.010 So that's what's going on here. 00:05:12.010 --> 00:05:16.180 And that's why this is a calculation that's 00:05:16.180 --> 00:05:22.075 linear in the depths of the neural net, not exponential. 00:05:22.075 --> 00:05:24.450 There's another thing I wanted to point out in connection 00:05:24.450 --> 00:05:28.900 with these neural nets. 00:05:28.900 --> 00:05:30.400 And that has to do with what happens 00:05:30.400 --> 00:05:34.870 when we look at a single neuron and note that what we've got 00:05:34.870 --> 00:05:37.920 is we've got a bunch of weights that you multiply times 00:05:37.920 --> 00:05:39.415 a bunch of inputs like so. 00:05:46.960 --> 00:05:51.390 And then those are all summed up in a summing box 00:05:51.390 --> 00:05:57.920 before they enter some kind of non-linearity, in our case 00:05:57.920 --> 00:06:00.480 a sigmoid function. 00:06:00.480 --> 00:06:05.890 But if I ask you to write down the expression for the value 00:06:05.890 --> 00:06:07.150 we've got there, what is it? 00:06:07.150 --> 00:06:13.072 Well, it's just the sum of the w's times the x's. 00:06:16.570 --> 00:06:17.080 What's that? 00:06:20.590 --> 00:06:22.847 That's the dot product. 00:06:22.847 --> 00:06:24.930 Remember a few lectures ago I said that some of us 00:06:24.930 --> 00:06:28.690 believe that the dot product is a fundamental calculation that 00:06:28.690 --> 00:06:30.760 takes place in our heads? 00:06:30.760 --> 00:06:33.790 So this is why we think so. 00:06:33.790 --> 00:06:36.880 If neural nets are doing anything like this, 00:06:36.880 --> 00:06:39.200 then there's a dot product between some weights 00:06:39.200 --> 00:06:41.250 and some input values. 00:06:41.250 --> 00:06:43.900 Now, it's a funny kind of dot product 00:06:43.900 --> 00:06:47.410 because in the models that we've been using, 00:06:47.410 --> 00:06:50.890 these input variables are all or none, or 0 or 1. 00:06:50.890 --> 00:06:52.500 But that's OK. 00:06:52.500 --> 00:06:54.110 I have it on good authority that there 00:06:54.110 --> 00:06:58.090 are neurons in our head for which the values that 00:06:58.090 --> 00:07:01.600 are produced are not exactly all or none 00:07:01.600 --> 00:07:03.950 but rather have a kind of proportionality to them. 00:07:03.950 --> 00:07:07.850 So you get a real dot product type of operation out of that. 00:07:07.850 --> 00:07:09.510 So that's by way of a couple of asides 00:07:09.510 --> 00:07:11.400 that I wanted to underscore before we 00:07:11.400 --> 00:07:16.360 get into the center of today's discussion, 00:07:16.360 --> 00:07:20.830 which will be to talk about the so-called deep nets. 00:07:20.830 --> 00:07:23.890 Now, let's see, what's a deep net do? 00:07:23.890 --> 00:07:29.820 Well, from last time, you know that a deep net does 00:07:29.820 --> 00:07:34.040 that sort of thing, and it's interesting to look 00:07:34.040 --> 00:07:36.470 at some of the offerings here. 00:07:36.470 --> 00:07:40.370 By the way, how good was this performance in 2012? 00:07:40.370 --> 00:07:44.630 Well, it turned out that the fraction 00:07:44.630 --> 00:07:48.880 of the time that the system had the right answer 00:07:48.880 --> 00:07:52.690 in its top five choices was about 15%. 00:07:52.690 --> 00:07:55.390 And the fraction of the time that it got exactly the right 00:07:55.390 --> 00:07:59.900 answer as its top pick was about 37%-- error, 00:07:59.900 --> 00:08:05.740 15% error if you count it as an error if it's-- what am I 00:08:05.740 --> 00:08:06.960 saying? 00:08:06.960 --> 00:08:09.440 You got it right if you got it in the top five. 00:08:09.440 --> 00:08:12.660 An error rate on that calculation, about 15%. 00:08:12.660 --> 00:08:16.110 If you say you only get it right if it was your top choice, then 00:08:16.110 --> 00:08:18.830 the error rate was about 37%. 00:08:18.830 --> 00:08:21.530 So pretty good, especially since some of these things 00:08:21.530 --> 00:08:24.602 are highly ambiguous even to us. 00:08:24.602 --> 00:08:26.060 And what kind of a system did that? 00:08:26.060 --> 00:08:30.610 Well, it wasn't one that looked exactly 00:08:30.610 --> 00:08:33.820 like that, although that is the essence of it. 00:08:33.820 --> 00:08:36.447 The system actually looked like that. 00:08:36.447 --> 00:08:38.030 There's quite a lot of stuff in there. 00:08:38.030 --> 00:08:40.530 And what I'm going to talk about is not exactly this system, 00:08:40.530 --> 00:08:44.530 but I'm going to talk about the stuff of which such systems are 00:08:44.530 --> 00:08:46.880 made because there's nothing particularly 00:08:46.880 --> 00:08:47.920 special about this. 00:08:47.920 --> 00:08:50.990 It just happens to be a particular assembly 00:08:50.990 --> 00:08:54.240 of components that tend to reappear when anyone does 00:08:54.240 --> 00:08:56.910 this sort of neural net stuff. 00:08:56.910 --> 00:08:59.030 So let me explain that this way. 00:08:59.030 --> 00:09:05.110 First thing I need to talk about is the concept of-- well, 00:09:05.110 --> 00:09:06.090 I don't like the term. 00:09:06.090 --> 00:09:07.780 It's called convolution. 00:09:07.780 --> 00:09:11.290 I don't like the term because in the second-best course 00:09:11.290 --> 00:09:12.979 at the Institute, Signals and Systems, 00:09:12.979 --> 00:09:15.020 you learn about impulse responses and convolution 00:09:15.020 --> 00:09:16.810 integrals and stuff like that. 00:09:16.810 --> 00:09:19.500 And this hints at that, but it's not the same thing 00:09:19.500 --> 00:09:22.930 because there's no memory involved in what's going on 00:09:22.930 --> 00:09:24.560 as these signals are processed. 00:09:24.560 --> 00:09:27.474 But they call it convolutional neural nets anyway. 00:09:27.474 --> 00:09:28.140 So here you are. 00:09:28.140 --> 00:09:29.733 You got some kind of image. 00:09:32.290 --> 00:09:37.620 And even with lots of computing power and GPUs and all 00:09:37.620 --> 00:09:39.970 that sort of stuff, we're not talking about images 00:09:39.970 --> 00:09:42.630 with 4 million pixels. 00:09:42.630 --> 00:09:46.090 We're talking about images that might be 256 on a side. 00:09:53.284 --> 00:09:54.950 As I say, we're not talking about images 00:09:54.950 --> 00:09:58.250 that are 1,000 by 1,000 or 4,000 by 4,000 or anything like that. 00:09:58.250 --> 00:10:00.940 They tend to be kind of compressed 00:10:00.940 --> 00:10:04.540 into a 256-by-256 image. 00:10:04.540 --> 00:10:09.180 And now what we do is we run over this 00:10:09.180 --> 00:10:12.270 with a neuron that is looking only 00:10:12.270 --> 00:10:22.980 at a 10-by-10 square like so, and that produces an output. 00:10:22.980 --> 00:10:27.090 And next, we went over that again having 00:10:27.090 --> 00:10:32.270 shifted this neuron a little bit like so. 00:10:32.270 --> 00:10:36.950 And then the next thing we do is we shift it again, so we 00:10:36.950 --> 00:10:39.870 get that output right there. 00:10:39.870 --> 00:10:46.190 So each of those deployments of a neuron produces an output, 00:10:46.190 --> 00:10:48.830 and that output is associated with a particular place 00:10:48.830 --> 00:10:50.530 in the image. 00:10:50.530 --> 00:10:58.060 This is the process that is called convolution 00:10:58.060 --> 00:10:59.570 as a term of art. 00:10:59.570 --> 00:11:04.160 Now, this guy, or this convolution operation, 00:11:04.160 --> 00:11:06.040 results in a bunch of points over here. 00:11:12.384 --> 00:11:16.730 And the next thing that we do with those points 00:11:16.730 --> 00:11:19.210 is we look in local neighborhoods 00:11:19.210 --> 00:11:22.340 and see what the maximum value is. 00:11:22.340 --> 00:11:24.650 And then we take that maximum value 00:11:24.650 --> 00:11:28.000 and construct yet another mapping of the image 00:11:28.000 --> 00:11:31.030 over here using that maximum value. 00:11:31.030 --> 00:11:36.470 Then we slide that over like so, and we produce another value. 00:11:36.470 --> 00:11:40.870 And then we slide that over one more 00:11:40.870 --> 00:11:44.400 time with a different color, and now we've 00:11:44.400 --> 00:11:46.630 got yet another value. 00:11:46.630 --> 00:11:48.232 So this process is called pooling. 00:11:54.969 --> 00:11:56.510 And because we're taking the maximum, 00:11:56.510 --> 00:12:00.960 this particular kind of pooling is called max pooling. 00:12:00.960 --> 00:12:03.200 So now let's see what's next. 00:12:03.200 --> 00:12:05.560 This is taking a particular neuron 00:12:05.560 --> 00:12:08.870 and running it across the image. 00:12:08.870 --> 00:12:12.970 We call that a kernel, again sucking some terminology out 00:12:12.970 --> 00:12:14.136 of Signals and Systems. 00:12:14.136 --> 00:12:15.510 But now what we're going to do is 00:12:15.510 --> 00:12:19.140 we're going to say we could use a whole bunch of kernels. 00:12:19.140 --> 00:12:21.990 So the thing that I produce with one kernel 00:12:21.990 --> 00:12:27.430 can now be repeated many times like so. 00:12:27.430 --> 00:12:30.870 In fact, a typical number is 100 times. 00:12:30.870 --> 00:12:34.440 So now what we've got is we've got a 256-by-256 image. 00:12:34.440 --> 00:12:38.240 We've gone over it with a 10-by-10 kernel. 00:12:38.240 --> 00:12:41.760 We have taken the maximum values that 00:12:41.760 --> 00:12:43.750 are in the vicinity of each other, 00:12:43.750 --> 00:12:48.270 and then we repeated that 100 times. 00:12:48.270 --> 00:12:53.320 So now we can take that, and we can feed all those results 00:12:53.320 --> 00:12:55.540 into some kind of neural net. 00:12:55.540 --> 00:12:59.210 And then we can, through perhaps a fully-connected job 00:12:59.210 --> 00:13:04.010 on the final layers of this, and then in the ultimate output we 00:13:04.010 --> 00:13:07.770 get some sort of indication of how likely it 00:13:07.770 --> 00:13:11.660 is that the thing that's being seen is, say, a mite. 00:13:15.300 --> 00:13:20.850 So that's roughly how these things work. 00:13:20.850 --> 00:13:22.530 So what have we talked about so far? 00:13:22.530 --> 00:13:27.780 We've talked about pooling, and we've talked about convolution. 00:13:27.780 --> 00:13:31.450 And now we can talk about some of the good stuff. 00:13:31.450 --> 00:13:35.175 But before I get into that, this is what we can do now, 00:13:35.175 --> 00:13:38.008 and you can compare this with what was done in the old days. 00:13:42.630 --> 00:13:45.040 It was done in the old days before massive amounts 00:13:45.040 --> 00:13:56.070 of computing became available is a kind of neural net activity 00:13:56.070 --> 00:13:58.350 that's a little easier to see. 00:13:58.350 --> 00:14:03.090 You might, in the old days, only have enough computing power 00:14:03.090 --> 00:14:05.740 to deal with a small grid of picture elements, 00:14:05.740 --> 00:14:07.480 or so-called pixels. 00:14:07.480 --> 00:14:12.890 And then each of these might be a value that is fed as an input 00:14:12.890 --> 00:14:15.000 into some kind of neuron. 00:14:15.000 --> 00:14:19.110 And so you might have a column of neurons that are looking 00:14:19.110 --> 00:14:23.860 at these pixels in your image. 00:14:23.860 --> 00:14:26.410 And then there might be a small number of columns 00:14:26.410 --> 00:14:27.830 that follow from that. 00:14:27.830 --> 00:14:30.780 And finally, something that says this neuron 00:14:30.780 --> 00:14:35.820 is looking for things that are a number 1, that is to say, 00:14:35.820 --> 00:14:43.330 something that looks like a number 1 in the image. 00:14:43.330 --> 00:14:46.430 So this stuff up here is what you 00:14:46.430 --> 00:14:48.597 can do when you have a massive amount of computation 00:14:48.597 --> 00:14:49.971 relative to the kind of thing you 00:14:49.971 --> 00:14:51.240 used to see in the old days. 00:14:55.400 --> 00:14:58.006 So what's different? 00:14:58.006 --> 00:14:59.380 Well, what's different is instead 00:14:59.380 --> 00:15:02.730 of a few hundred parameters, we've got a lot more. 00:15:02.730 --> 00:15:07.000 Instead of 10 digits, we have 1,000 classes. 00:15:07.000 --> 00:15:09.130 Instead of a few hundred samples, 00:15:09.130 --> 00:15:13.570 we have maybe 1,000 examples of each class. 00:15:13.570 --> 00:15:16.270 So that makes a million samples. 00:15:16.270 --> 00:15:20.030 And we got 60 million parameters to play with. 00:15:20.030 --> 00:15:23.060 And the surprising thing is that the net result 00:15:23.060 --> 00:15:26.800 is we've got a function approximator that 00:15:26.800 --> 00:15:28.380 astonishes everybody. 00:15:28.380 --> 00:15:30.070 And no one quite knows why it works, 00:15:30.070 --> 00:15:34.230 except that when you throw an immense amount of computation 00:15:34.230 --> 00:15:38.350 into this kind of arrangement, it's 00:15:38.350 --> 00:15:42.770 possible to get a performance that no one expected would 00:15:42.770 --> 00:15:45.540 be possible. 00:15:45.540 --> 00:15:47.170 So that's sort of the bottom line. 00:15:47.170 --> 00:15:51.240 But now there are a couple of ideas beyond that that I think 00:15:51.240 --> 00:15:56.030 are especially interesting, and I want to talk about those. 00:15:56.030 --> 00:15:58.690 First idea that's especially interesting 00:15:58.690 --> 00:16:01.730 is the idea of autocoding, and here's 00:16:01.730 --> 00:16:03.530 how the idea of autocoding works. 00:16:09.190 --> 00:16:10.690 I'm going to run out of board space, 00:16:10.690 --> 00:16:14.230 so I think I'll do it right here. 00:16:14.230 --> 00:16:16.320 You have some input values. 00:16:24.860 --> 00:16:29.420 They go into a layer of neurons, the input layer. 00:16:29.420 --> 00:16:33.181 Then there is a so-called hidden layer that's much smaller. 00:16:36.010 --> 00:16:39.830 So maybe in the example, there will be 10 neurons here 00:16:39.830 --> 00:16:41.810 and just a couple here. 00:16:41.810 --> 00:16:47.750 And then these expand to an output layer like so. 00:16:47.750 --> 00:16:53.180 Now we can take the output layer, z1 through zn, 00:16:53.180 --> 00:16:59.030 and compare it with the desired values, d1 through dn. 00:17:01.920 --> 00:17:03.770 You following me so far? 00:17:03.770 --> 00:17:09.020 Now, the trick is to say, well, what are the desired values? 00:17:09.020 --> 00:17:13.605 Let's let the desired values be the input values. 00:17:17.254 --> 00:17:18.670 So what we're going to do is we're 00:17:18.670 --> 00:17:20.839 going to train this net up so that the output's 00:17:20.839 --> 00:17:23.600 the same as the input. 00:17:23.600 --> 00:17:24.599 What's the good of that? 00:17:24.599 --> 00:17:27.030 Well, we're going to force it down through this 00:17:27.030 --> 00:17:30.270 [? neck-down ?] piece of network. 00:17:30.270 --> 00:17:33.640 So if this network is going to succeed 00:17:33.640 --> 00:17:37.160 in taking all the possibilities here and cramming them 00:17:37.160 --> 00:17:42.720 into this smaller inner layer, the so-called hidden layer, 00:17:42.720 --> 00:17:45.900 such that it can reproduce the input [? at ?] the output, 00:17:45.900 --> 00:17:48.300 it must be doing some kind of generalization 00:17:48.300 --> 00:17:52.560 of the kinds of things it sees on its input. 00:17:52.560 --> 00:17:56.830 And that's a very clever idea, and it's seen in various forms 00:17:56.830 --> 00:18:00.200 in a large fraction of the papers that 00:18:00.200 --> 00:18:03.340 appear on deep neural nets. 00:18:03.340 --> 00:18:05.384 But now I want to talk about an example 00:18:05.384 --> 00:18:06.800 so I can show you a demonstration. 00:18:06.800 --> 00:18:07.860 OK? 00:18:07.860 --> 00:18:11.410 So we don't have GPUs, and we don't have three days 00:18:11.410 --> 00:18:12.670 to do this. 00:18:12.670 --> 00:18:17.520 So I'm going to make up a very simple example that's 00:18:17.520 --> 00:18:21.010 reminiscent of what goes on here but involves 00:18:21.010 --> 00:18:22.974 hardly any computation. 00:18:22.974 --> 00:18:24.390 What I'm going to imagine is we're 00:18:24.390 --> 00:18:31.260 trying to recognize animals from how tall they 00:18:31.260 --> 00:18:35.920 are from the shadows that they cast. 00:18:35.920 --> 00:18:43.940 So we're going to recognize three animals, a cheetah, 00:18:43.940 --> 00:18:51.060 a zebra, and a giraffe, and they will each cast a shadow 00:18:51.060 --> 00:18:55.800 on the blackboard like me. 00:18:55.800 --> 00:18:57.012 No vampire involved here. 00:18:57.012 --> 00:18:58.470 And what we're going to do is we're 00:18:58.470 --> 00:19:03.290 going to use the shadow as an input to a neural net. 00:19:03.290 --> 00:19:03.976 All right? 00:19:03.976 --> 00:19:05.350 So let's see how that would work. 00:19:19.350 --> 00:19:24.180 So there is our network. 00:19:24.180 --> 00:19:26.480 And if I just clicked into one of these test samples, 00:19:26.480 --> 00:19:32.010 that's the height of the shadow that a cheetah casts on a wall. 00:19:32.010 --> 00:19:34.900 And there are 10 input neurons corresponding 00:19:34.900 --> 00:19:37.500 to each level of the shadow. 00:19:37.500 --> 00:19:41.350 They're rammed through three inner layer neurons, 00:19:41.350 --> 00:19:46.167 and from that it spreads out and becomes the outer layer values. 00:19:46.167 --> 00:19:48.000 And we're going to compare those outer layer 00:19:48.000 --> 00:19:50.520 values to the desired values, but the desired values 00:19:50.520 --> 00:19:52.640 are the same as the input values. 00:19:52.640 --> 00:19:54.840 So this column is a column of input values. 00:19:54.840 --> 00:19:58.420 On the far right, we have our column of desired values. 00:19:58.420 --> 00:20:00.650 And we haven't trained this neural net yet. 00:20:00.650 --> 00:20:02.900 All we've got is random values in there. 00:20:02.900 --> 00:20:08.140 So if we run the test samples through, we get that and that. 00:20:08.140 --> 00:20:11.400 Yeah, cheetahs are short, zebras are medium height, 00:20:11.400 --> 00:20:13.140 and giraffes are tall. 00:20:13.140 --> 00:20:19.630 But our output is just pretty much 0.5 for all of them, 00:20:19.630 --> 00:20:21.470 for all of those shadow heights, all right, 00:20:21.470 --> 00:20:24.110 [? with ?] no training so far. 00:20:24.110 --> 00:20:25.270 So let's run this thing. 00:20:25.270 --> 00:20:27.810 We're just using simple [? backdrop, ?] just like on 00:20:27.810 --> 00:20:30.570 our world's simplest neural net. 00:20:30.570 --> 00:20:36.700 And it's interesting to see what happens. 00:20:36.700 --> 00:20:38.310 You see all those values changing? 00:20:38.310 --> 00:20:42.010 Now, I need to mention that when you see a green connection, 00:20:42.010 --> 00:20:44.390 that means it's a positive weight, 00:20:44.390 --> 00:20:49.870 and the density of the green indicates how positive it is. 00:20:49.870 --> 00:20:51.860 And the red ones are negative weights, 00:20:51.860 --> 00:20:55.000 and the intensity of the red indicates how red it is. 00:20:55.000 --> 00:20:56.590 So here you can see that we still 00:20:56.590 --> 00:20:59.630 have from our random inputs a variety 00:20:59.630 --> 00:21:00.950 of red and green values. 00:21:00.950 --> 00:21:02.680 We haven't really done much training, 00:21:02.680 --> 00:21:06.840 so everything correctly looks pretty much random. 00:21:06.840 --> 00:21:10.160 So let's run this thing. 00:21:10.160 --> 00:21:17.244 And after only 1,000 iterations going through these examples 00:21:17.244 --> 00:21:19.410 and trying to make the output the same as the input, 00:21:19.410 --> 00:21:22.174 we reached a point where the error rate has dropped. 00:21:22.174 --> 00:21:23.590 In fact, it's dropped so much it's 00:21:23.590 --> 00:21:26.910 interesting to relook at the test cases. 00:21:26.910 --> 00:21:29.150 So here's a test case where we have a cheetah. 00:21:29.150 --> 00:21:30.980 And now the output value is, in fact, 00:21:30.980 --> 00:21:37.740 very close to the desired value in all the output neurons. 00:21:37.740 --> 00:21:40.680 So if we look at another one, once again, 00:21:40.680 --> 00:21:43.780 there's a correspondence in the right two columns. 00:21:43.780 --> 00:21:45.945 And if we look at the final one, yeah, there's 00:21:45.945 --> 00:21:47.695 a correspondence in the right two columns. 00:21:50.980 --> 00:21:52.930 Now, you back up from this and say, well, 00:21:52.930 --> 00:21:55.570 what's going on here? 00:21:55.570 --> 00:22:00.320 It turns out that you're not training this thing 00:22:00.320 --> 00:22:02.490 to classify animals. 00:22:02.490 --> 00:22:05.590 You're training it to understand the nature of the things 00:22:05.590 --> 00:22:09.447 that it sees in the environment because all it sees 00:22:09.447 --> 00:22:10.530 is the height of a shadow. 00:22:10.530 --> 00:22:12.613 It doesn't know anything about the classifications 00:22:12.613 --> 00:22:14.610 you're going to try to get out of that. 00:22:14.610 --> 00:22:17.470 All it sees is that there's a kind of consistency 00:22:17.470 --> 00:22:21.630 in the kind of data that it sees on the input values. 00:22:21.630 --> 00:22:23.170 Right? 00:22:23.170 --> 00:22:24.860 Now, you might say, OK, oh, that's cool, 00:22:24.860 --> 00:22:26.610 because what must be happening is 00:22:26.610 --> 00:22:29.730 that that hidden layer, because everything is forced 00:22:29.730 --> 00:22:32.120 through that narrow pipe, must be doing 00:22:32.120 --> 00:22:35.229 some kind of generalization. 00:22:35.229 --> 00:22:37.020 So it ought to be the case that if we click 00:22:37.020 --> 00:22:38.436 on each of those neurons, we ought 00:22:38.436 --> 00:22:40.820 to see it specialize to a particular height, 00:22:40.820 --> 00:22:46.290 because that's the sort of stuff that's presented on the input. 00:22:46.290 --> 00:22:49.170 Well, let's go see what, in fact, is 00:22:49.170 --> 00:22:52.980 the maximum stimulation to be seen 00:22:52.980 --> 00:22:56.540 on the neurons in that hidden layer. 00:22:56.540 --> 00:22:59.860 So when I click on these guys, what we're going to see 00:22:59.860 --> 00:23:02.990 is the input values that maximally 00:23:02.990 --> 00:23:05.174 stimulate that neuron. 00:23:05.174 --> 00:23:06.590 And by the way, I have no idea how 00:23:06.590 --> 00:23:09.480 this is going to turn out because the initialization's 00:23:09.480 --> 00:23:11.580 all random. 00:23:11.580 --> 00:23:12.360 Well, that's good. 00:23:12.360 --> 00:23:13.860 That one looks like it's generalized 00:23:13.860 --> 00:23:16.920 the notion of short. 00:23:16.920 --> 00:23:20.690 Ugh, that doesn't look like medium. 00:23:20.690 --> 00:23:24.670 And in fact, the maximum stimulation 00:23:24.670 --> 00:23:28.570 doesn't involve any stimulation from that lower neuron. 00:23:28.570 --> 00:23:31.170 Here, look at this one. 00:23:31.170 --> 00:23:32.640 That doesn't look like tall. 00:23:32.640 --> 00:23:34.910 So we got one that looks like short and two that 00:23:34.910 --> 00:23:37.505 just look completely random. 00:23:40.320 --> 00:23:42.510 So in fact, maybe we better back off the idea 00:23:42.510 --> 00:23:44.730 that what's going on in that hidden layer 00:23:44.730 --> 00:23:48.850 is generalization and say that what 00:23:48.850 --> 00:23:51.660 is going on in there is maybe the encoding 00:23:51.660 --> 00:23:53.840 of a generalization. 00:23:53.840 --> 00:23:56.190 It doesn't look like an encoding we can see, 00:23:56.190 --> 00:24:01.060 but there is a generalization that's-- let me start that 00:24:01.060 --> 00:24:01.820 over. 00:24:01.820 --> 00:24:08.312 We don't see the generalization in the stimulating values. 00:24:08.312 --> 00:24:10.020 What we have instead is we have some kind 00:24:10.020 --> 00:24:12.567 of encoded generalization. 00:24:12.567 --> 00:24:14.150 And because we got this stuff encoded, 00:24:14.150 --> 00:24:16.570 it's what makes these neural nets so extraordinarily 00:24:16.570 --> 00:24:17.970 difficult to understand. 00:24:17.970 --> 00:24:20.610 We don't understand what they're doing. 00:24:20.610 --> 00:24:23.050 We don't understand why they can recognize a cheetah. 00:24:23.050 --> 00:24:25.400 We don't understand why it can recognize a school 00:24:25.400 --> 00:24:27.070 bus in some cases, but not in others, 00:24:27.070 --> 00:24:29.780 because we don't really understand 00:24:29.780 --> 00:24:32.530 what these neurons are responding to. 00:24:32.530 --> 00:24:34.110 Well, that's not quite true. 00:24:34.110 --> 00:24:36.300 There's been a lot of work recently 00:24:36.300 --> 00:24:38.690 on trying to sort that out, but it's still 00:24:38.690 --> 00:24:41.870 a lot of mystery in this world. 00:24:41.870 --> 00:24:44.700 In any event, that's the autocoding idea. 00:24:44.700 --> 00:24:45.945 It comes in various guises. 00:24:45.945 --> 00:24:48.320 Sometimes people talk about Boltzmann machines and things 00:24:48.320 --> 00:24:48.880 of that sort. 00:24:48.880 --> 00:24:51.230 But it's basically all the same sort of idea. 00:24:51.230 --> 00:24:53.300 And so what you can do is layer by layer. 00:24:53.300 --> 00:24:55.290 Once you've trained the input layer, 00:24:55.290 --> 00:24:57.810 then you can use that layer to train the next layer, 00:24:57.810 --> 00:25:00.110 and then that can train the next layer after that. 00:25:00.110 --> 00:25:04.360 And it's only at the very, very end that you say to yourself, 00:25:04.360 --> 00:25:06.720 well, now I've accumulated a lot of knowledge 00:25:06.720 --> 00:25:10.220 about the environment and what can be seen in the environment. 00:25:10.220 --> 00:25:12.781 Maybe it's time to get around to using 00:25:12.781 --> 00:25:17.770 some samples of particular classes and train on classes. 00:25:17.770 --> 00:25:19.320 So that's the story on autocoding. 00:25:22.780 --> 00:25:26.730 Now, the next thing to talk about is that final layer. 00:25:29.660 --> 00:25:32.384 So let's see what the final layer might look like. 00:25:35.110 --> 00:25:39.940 Let's see, it might look like this. 00:25:39.940 --> 00:25:44.937 There's a [? summer. ?] There's a minus 1 up here. 00:25:44.937 --> 00:25:45.436 No. 00:25:45.436 --> 00:25:47.926 Let's see, there's a minus 1 up-- [INAUDIBLE]. 00:25:50.710 --> 00:25:53.120 There's a minus 1 up there. 00:25:53.120 --> 00:25:55.420 There's a multiplier here. 00:25:55.420 --> 00:25:58.091 And there's a threshold value there. 00:25:58.091 --> 00:26:00.950 Now, likewise, there's some other input values here. 00:26:00.950 --> 00:26:07.690 Let me call this one x, and it gets multiplied by some weight. 00:26:07.690 --> 00:26:10.500 And then that goes into the [? summer ?] as well. 00:26:10.500 --> 00:26:19.540 And that, in turn, goes into a sigmoid that looks like so. 00:26:19.540 --> 00:26:25.180 And finally, you get an output, which we'll z. 00:26:25.180 --> 00:26:29.400 So it's clear that if you just write out the value of z 00:26:29.400 --> 00:26:36.020 as it depends on those inputs using the formula that we 00:26:36.020 --> 00:26:38.770 worked with last time, then what you 00:26:38.770 --> 00:26:48.280 see is that z is equal to 1 over 1 00:26:48.280 --> 00:27:01.218 plus e to the minus w times x minus T-- plus T, I guess. 00:27:01.218 --> 00:27:01.717 Right? 00:27:04.990 --> 00:27:08.190 So that's a sigmoid function that 00:27:08.190 --> 00:27:11.390 depends on the value of that weight 00:27:11.390 --> 00:27:13.970 and on the value of that threshold. 00:27:13.970 --> 00:27:20.510 So let's look at how those values might change things. 00:27:20.510 --> 00:27:23.803 So here we have an ordinary sigmoid. 00:27:26.550 --> 00:27:31.390 And what happens if we shift it with a threshold value? 00:27:31.390 --> 00:27:34.520 If we change that threshold value, 00:27:34.520 --> 00:27:36.390 then it's going to shift the place 00:27:36.390 --> 00:27:41.948 where that sigmoid comes down. 00:27:45.600 --> 00:27:47.860 So a change in T could cause this thing 00:27:47.860 --> 00:27:50.770 to shift over that way. 00:27:50.770 --> 00:27:53.010 And if we change the value of w, that 00:27:53.010 --> 00:27:54.870 could change how steep this guy is. 00:27:58.460 --> 00:28:02.700 So we might think that the performance, since it depends 00:28:02.700 --> 00:28:08.820 on w and T, should be adjusted in such a way 00:28:08.820 --> 00:28:14.470 as to make the classification do the right thing. 00:28:14.470 --> 00:28:17.020 But what's the right thing? 00:28:17.020 --> 00:28:19.397 Well, that depends on the samples that we've seen. 00:28:24.490 --> 00:28:28.603 Suppose, for example, that this is our sigmoid function. 00:28:31.330 --> 00:28:37.380 And we see some examples of a class, some positive examples 00:28:37.380 --> 00:28:40.780 of a class, that have values that 00:28:40.780 --> 00:28:46.800 lie at that point and that point and that point. 00:28:46.800 --> 00:28:54.180 And we have some values that correspond to situations where 00:28:54.180 --> 00:28:57.040 the class is not one of the things that are associated 00:28:57.040 --> 00:28:58.580 with this neuron. 00:28:58.580 --> 00:29:01.930 And in that case, what we see is examples that 00:29:01.930 --> 00:29:03.370 are over in this vicinity here. 00:29:06.370 --> 00:29:10.710 So the probability that we would see this particular guy 00:29:10.710 --> 00:29:15.300 in this world is associated with the value on the sigmoid curve. 00:29:15.300 --> 00:29:17.420 So you could think of this as the probability 00:29:17.420 --> 00:29:19.380 of that positive example, and this 00:29:19.380 --> 00:29:21.840 is the probability of that positive example, 00:29:21.840 --> 00:29:25.020 and this is the probability of that positive example. 00:29:25.020 --> 00:29:28.020 What's the probability of this negative example? 00:29:28.020 --> 00:29:32.480 Well, it's 1 minus the value on that curve. 00:29:32.480 --> 00:29:36.830 And this one's 1 minus the value on that curve. 00:29:36.830 --> 00:29:39.510 So we could go through the calculations. 00:29:39.510 --> 00:29:43.230 And what we would determine is that to maximize 00:29:43.230 --> 00:29:46.790 the probability of seeing this data, this particular stuff 00:29:46.790 --> 00:29:50.140 in a set of experiments, to maximize that probability, 00:29:50.140 --> 00:29:55.860 we would have to adjust T and w so as to get this curve doing 00:29:55.860 --> 00:29:57.770 the optimal thing. 00:29:57.770 --> 00:29:59.634 And there's nothing mysterious about it. 00:29:59.634 --> 00:30:01.050 It's just more partial derivatives 00:30:01.050 --> 00:30:03.150 and that sort of thing. 00:30:03.150 --> 00:30:09.720 But the bottom line is that the probability of seeing this data 00:30:09.720 --> 00:30:12.570 is dependent on the shape of this curve, 00:30:12.570 --> 00:30:16.540 and the shape of this curve is dependent on those parameters. 00:30:16.540 --> 00:30:19.330 And if we wanted to maximize the probability that we've 00:30:19.330 --> 00:30:23.058 seen this data, then we have to adjust those parameters 00:30:23.058 --> 00:30:23.558 accordingly. 00:30:25.559 --> 00:30:27.100 Let's have a look at a demonstration. 00:30:39.770 --> 00:30:40.270 OK. 00:30:40.270 --> 00:30:43.730 So there's an ordinary sigmoid curve. 00:30:43.730 --> 00:30:46.850 Here are a couple of positive examples. 00:30:46.850 --> 00:30:49.995 Here's a negative example. 00:30:53.500 --> 00:30:58.170 Let's put in some more positive examples over here. 00:30:58.170 --> 00:31:04.670 And now let's run the good, old gradient ascent algorithm 00:31:04.670 --> 00:31:06.870 on that. 00:31:06.870 --> 00:31:08.920 And this is what happens. 00:31:08.920 --> 00:31:11.640 You've seen how the probability, as we adjust 00:31:11.640 --> 00:31:14.370 the shape of the curve, the probability of seeing 00:31:14.370 --> 00:31:18.060 those examples of the class goes up, 00:31:18.060 --> 00:31:22.950 and the probability of seeing the non-example goes down. 00:31:22.950 --> 00:31:26.110 So what if we put some more examples in? 00:31:26.110 --> 00:31:27.640 If we put a negative example there, 00:31:27.640 --> 00:31:30.030 not much is going to happen. 00:31:30.030 --> 00:31:33.762 What would happen if we put a positive example right there? 00:31:33.762 --> 00:31:35.970 Then we're going to start seeing some dramatic shifts 00:31:35.970 --> 00:31:37.053 in the shape of the curve. 00:31:48.000 --> 00:31:50.450 So that's probably a noise point. 00:31:50.450 --> 00:31:54.750 But we can put some more negative examples in there 00:31:54.750 --> 00:31:56.262 and see how that adjusts the curve. 00:31:59.510 --> 00:32:00.010 All right. 00:32:00.010 --> 00:32:01.135 So that's what we're doing. 00:32:01.135 --> 00:32:03.430 We're viewing this output value as something 00:32:03.430 --> 00:32:07.250 that's related to the probability of seeing a class. 00:32:07.250 --> 00:32:09.900 And we're adjusting the parameters on that output layer 00:32:09.900 --> 00:32:12.620 so as to maximize the probability of the sample data 00:32:12.620 --> 00:32:14.525 that we've got at hand. 00:32:14.525 --> 00:32:15.025 Right? 00:32:17.930 --> 00:32:20.124 Now, there's one more thing. 00:32:20.124 --> 00:32:21.540 Because see what we've got here is 00:32:21.540 --> 00:32:24.880 we've got the basic idea of back propagation, which 00:32:24.880 --> 00:32:30.200 has layers and layers of additional-- 00:32:30.200 --> 00:32:33.390 let me be flattering and call them ideas layered on top. 00:32:33.390 --> 00:32:38.820 So here's the next idea that's layered on top. 00:32:38.820 --> 00:32:43.110 So we've got an output value here. 00:32:45.740 --> 00:32:50.270 And it's a function after all, and it's got a value. 00:32:50.270 --> 00:32:54.120 And if we have 1,000 classes, we're 00:32:54.120 --> 00:32:56.180 going to have 1,000 output neurons, 00:32:56.180 --> 00:32:59.020 and each is going to be producing some kind of value. 00:32:59.020 --> 00:33:02.326 And we can think of that value as a probability. 00:33:04.705 --> 00:33:06.580 But I didn't want to write a probability yet. 00:33:06.580 --> 00:33:07.996 I just want to say that what we've 00:33:07.996 --> 00:33:13.186 got for this output neuron is a function of class 1. 00:33:13.186 --> 00:33:15.060 And then there will be another output neuron, 00:33:15.060 --> 00:33:18.200 which is a function of class 2. 00:33:18.200 --> 00:33:21.040 And these values will be presumably higher-- 00:33:21.040 --> 00:33:24.550 this will be higher if we are, in fact, looking at class 1. 00:33:24.550 --> 00:33:27.210 And this one down here will be, in fact, higher 00:33:27.210 --> 00:33:28.658 if we're looking at class m. 00:33:31.820 --> 00:33:35.020 So what we would like to do is we'd like to not just pick 00:33:35.020 --> 00:33:37.950 one of these outputs and say, well, you've 00:33:37.950 --> 00:33:40.550 got the highest value, so you win. 00:33:40.550 --> 00:33:42.710 What we want to do instead is we want 00:33:42.710 --> 00:33:45.130 to associate some kind of probability 00:33:45.130 --> 00:33:47.074 with each of the classes. 00:33:47.074 --> 00:33:48.740 Because, after all, we want to do things 00:33:48.740 --> 00:33:52.602 like find the most probable five. 00:33:52.602 --> 00:33:54.060 So what we do is we say, all right, 00:33:54.060 --> 00:33:59.950 so the actual probability of class 1 00:33:59.950 --> 00:34:07.990 is equal to the output of that sigmoid function divided 00:34:07.990 --> 00:34:11.357 by the sum over all functions. 00:34:14.909 --> 00:34:17.239 So that takes all of that entire output vector 00:34:17.239 --> 00:34:20.920 and converts each output value into a probability. 00:34:20.920 --> 00:34:24.475 So when we used that sigmoid function, 00:34:24.475 --> 00:34:26.100 we did it with the view toward thinking 00:34:26.100 --> 00:34:27.266 about that as a probability. 00:34:27.266 --> 00:34:30.000 And in fact, we assumed it was a probability when 00:34:30.000 --> 00:34:32.429 we made this argument. 00:34:32.429 --> 00:34:35.170 But in the end, there's an output 00:34:35.170 --> 00:34:36.280 for each of those classes. 00:34:36.280 --> 00:34:39.000 And so what we get is, in the end, not exactly a probability 00:34:39.000 --> 00:34:43.219 until we divide by a normalizing factor. 00:34:43.219 --> 00:34:49.500 So this, by the way, is called-- not on my list of things, 00:34:49.500 --> 00:34:50.526 but it soon will be. 00:34:54.580 --> 00:34:59.640 Since we're not talking about taking the maximum 00:34:59.640 --> 00:35:02.580 and using that to classify the picture, what we're going to do 00:35:02.580 --> 00:35:05.290 is we're going to use what's called softmax. 00:35:09.140 --> 00:35:11.730 So we're going to give a range of classifications, 00:35:11.730 --> 00:35:14.680 and we're going to associate a probability with each. 00:35:14.680 --> 00:35:18.610 And that's what you saw in all of those samples. 00:35:18.610 --> 00:35:20.360 You saw, yes, this is [? containership, ?] 00:35:20.360 --> 00:35:24.070 but maybe it's also this, that, or a third, or fourth, 00:35:24.070 --> 00:35:26.760 and fifth thing. 00:35:26.760 --> 00:35:31.924 So that is a pretty good summary of the kinds 00:35:31.924 --> 00:35:33.090 of things that are involved. 00:35:33.090 --> 00:35:36.890 But now we've got one more step, because what we can do now 00:35:36.890 --> 00:35:41.090 is we can take this output layer idea, this softmax idea, 00:35:41.090 --> 00:35:43.470 and we can put them together with the autocoding idea. 00:35:47.770 --> 00:35:50.820 So we've trained just a layer up. 00:35:50.820 --> 00:35:53.700 And now we're going to detach it from the output layer 00:35:53.700 --> 00:35:55.800 but retain those weights that connect 00:35:55.800 --> 00:35:58.170 the input to the hidden layer. 00:35:58.170 --> 00:36:00.560 And when we do that, what we're going to see 00:36:00.560 --> 00:36:03.430 is something that looks like this. 00:36:03.430 --> 00:36:05.410 And now we've got a trained first layer 00:36:05.410 --> 00:36:07.850 but an untrained output layer. 00:36:07.850 --> 00:36:10.280 We're going to freeze the input layer 00:36:10.280 --> 00:36:16.590 and train the output layer using the sigmoid curve 00:36:16.590 --> 00:36:18.140 and see what happens when we do that. 00:36:18.140 --> 00:36:21.725 Oh, by the way, let's run our test samples through. 00:36:21.725 --> 00:36:23.225 You can see it's not doing anything, 00:36:23.225 --> 00:36:27.180 and the output is half for each of the categories 00:36:27.180 --> 00:36:29.390 even though we've got a trained middle layer. 00:36:29.390 --> 00:36:30.890 So we have to train the outer layer. 00:36:30.890 --> 00:36:32.826 Let's see how long it takes. 00:36:32.826 --> 00:36:33.950 Whoa, that was pretty fast. 00:36:36.880 --> 00:36:40.210 Now there's an extraordinarily good match between the outputs 00:36:40.210 --> 00:36:41.809 and the desired outputs. 00:36:41.809 --> 00:36:43.600 So that's the combination of the autocoding 00:36:43.600 --> 00:36:45.541 idea and the softmax idea. 00:36:50.150 --> 00:36:55.540 [? There's ?] just one more idea that's worthy of mention, 00:36:55.540 --> 00:36:57.020 and that's the idea of dropout. 00:37:00.320 --> 00:37:02.880 The plague of any neural net is that it gets stuck 00:37:02.880 --> 00:37:06.040 in some kind of local maximum. 00:37:06.040 --> 00:37:08.740 So it was discovered that these things train 00:37:08.740 --> 00:37:16.500 better if, on every iteration, you 00:37:16.500 --> 00:37:19.620 flip a coin for each neuron. 00:37:19.620 --> 00:37:22.260 And if the coin ends up tails, you 00:37:22.260 --> 00:37:26.920 assume it's just died and has no influence on the output. 00:37:26.920 --> 00:37:29.930 It's called dropping out those neurons. 00:37:29.930 --> 00:37:33.950 And in our next iteration, you drop out a different set. 00:37:33.950 --> 00:37:35.660 So what this seems to do is it seems 00:37:35.660 --> 00:37:41.021 to prevent this thing from going into a frozen local maximum 00:37:41.021 --> 00:37:41.520 state. 00:37:44.840 --> 00:37:46.230 So that's deep nets. 00:37:46.230 --> 00:37:50.000 They should be called, by the way, wide nets because they 00:37:50.000 --> 00:37:53.020 tend to be enormously wide but rarely 00:37:53.020 --> 00:38:01.250 more than 10 columns deep. 00:38:01.250 --> 00:38:04.050 Now, let's see, where to go from here? 00:38:04.050 --> 00:38:10.900 Maybe what we should do is talk about the awesome curiosity 00:38:10.900 --> 00:38:13.820 in the current state of the art. 00:38:13.820 --> 00:38:17.910 And that is that all of [? this ?] 00:38:17.910 --> 00:38:21.750 sophistication with output layers that are probabilities 00:38:21.750 --> 00:38:28.580 and training using autocoding or Boltzmann machines, 00:38:28.580 --> 00:38:33.190 it doesn't seem to help much relative to plain, old back 00:38:33.190 --> 00:38:35.640 propagation. 00:38:35.640 --> 00:38:38.090 So back propagation with a convolutional net 00:38:38.090 --> 00:38:41.630 seems to do just about as good as anything. 00:38:41.630 --> 00:38:46.530 And while we're on the subject of an ordinary deep net, 00:38:46.530 --> 00:38:50.070 I'd like to examine a situation here 00:38:50.070 --> 00:38:59.175 where we have a deep net-- well, it's a classroom deep net. 00:38:59.175 --> 00:39:02.670 And we'll will put five layers in there, 00:39:02.670 --> 00:39:04.930 and its job is still to do the same thing. 00:39:04.930 --> 00:39:09.630 It's to classify an animal as a cheetah, a zebra, or a giraffe 00:39:09.630 --> 00:39:13.630 based on the height of the shadow it casts. 00:39:13.630 --> 00:39:16.610 And as before, if it's green, that means positive. 00:39:16.610 --> 00:39:19.300 If it's red, that means negative. 00:39:19.300 --> 00:39:22.660 And right at the moment, we have no training. 00:39:22.660 --> 00:39:24.420 So if we run our test samples through, 00:39:24.420 --> 00:39:29.120 the output is always a 1/2 no matter what the animal is. 00:39:29.120 --> 00:39:29.975 All right? 00:39:29.975 --> 00:39:31.350 So what we're going to do is just 00:39:31.350 --> 00:39:34.920 going to use ordinary back prop on this, same thing 00:39:34.920 --> 00:39:41.970 as in that sample that's underneath the blackboard. 00:39:41.970 --> 00:39:43.890 Only now we've got a lot more parameters. 00:39:43.890 --> 00:39:46.950 We've got five columns, and each one of them 00:39:46.950 --> 00:39:50.320 has 9 or 10 neurons in it. 00:39:50.320 --> 00:39:52.270 So let's let this one run. 00:39:56.160 --> 00:39:57.740 Now, look at that stuff on the right. 00:39:57.740 --> 00:39:59.310 It's all turned red. 00:39:59.310 --> 00:40:03.270 At first I thought this was a bug in my program. 00:40:03.270 --> 00:40:04.699 But that makes absolute sense. 00:40:04.699 --> 00:40:06.990 If you don't know what the actual animal is going to be 00:40:06.990 --> 00:40:08.865 and there are a whole bunch of possibilities, 00:40:08.865 --> 00:40:10.970 you better just say no for everybody. 00:40:10.970 --> 00:40:13.550 It's like when a biologist says, we don't know. 00:40:13.550 --> 00:40:16.380 It's the most probable answer. 00:40:16.380 --> 00:40:20.400 Well, but eventually, after about 160,000 iterations, 00:40:20.400 --> 00:40:21.400 it seems to have got it. 00:40:21.400 --> 00:40:22.858 Let's run the test samples through. 00:40:27.044 --> 00:40:29.310 Now it's doing great. 00:40:29.310 --> 00:40:31.387 Let's do it again just to see if this is a fluke. 00:40:37.131 --> 00:40:52.590 And all red on the right side, and finally, you 00:40:52.590 --> 00:40:56.930 start seeing some changes go in the final layers there. 00:40:56.930 --> 00:40:59.600 And if you look at the error rate down at the bottom, 00:40:59.600 --> 00:41:02.700 you'll see that it kind of falls off a cliff. 00:41:02.700 --> 00:41:04.806 So nothing happens for a real long time, 00:41:04.806 --> 00:41:06.056 and then it falls off a cliff. 00:41:09.560 --> 00:41:13.620 Now, what would happen if this neural net were not 00:41:13.620 --> 00:41:15.860 quite so wide? 00:41:15.860 --> 00:41:16.760 Good question. 00:41:16.760 --> 00:41:19.093 But before we get to that question, what I'm going to do 00:41:19.093 --> 00:41:21.880 is I'm going to do a funny kind of variation 00:41:21.880 --> 00:41:23.676 on the theme of dropout. 00:41:23.676 --> 00:41:25.050 What I'm going to do is I'm going 00:41:25.050 --> 00:41:28.070 to kill off one neuron in each column, 00:41:28.070 --> 00:41:30.590 and then see if I can retrain the network 00:41:30.590 --> 00:41:33.750 to do the right thing. 00:41:33.750 --> 00:41:37.210 So I'm going to reassign those to some other purpose. 00:41:37.210 --> 00:41:40.100 So now there's one fewer neuron in the network. 00:41:40.100 --> 00:41:44.622 If we rerun that, we see that it trains itself up very fast. 00:41:44.622 --> 00:41:46.080 So we seem to be still close enough 00:41:46.080 --> 00:41:50.470 to a solution we can do without one 00:41:50.470 --> 00:41:52.110 of the neurons in each column. 00:41:52.110 --> 00:41:52.920 Let's do it again. 00:41:55.269 --> 00:41:57.060 Now it goes up a little bit, but it quickly 00:41:57.060 --> 00:41:59.530 falls down to a solution. 00:41:59.530 --> 00:42:02.060 Try again. 00:42:02.060 --> 00:42:05.620 Quickly falls down to a solution. 00:42:05.620 --> 00:42:08.560 Oh, my god, how much of this am I going to do? 00:42:08.560 --> 00:42:10.970 Each time I knock something out and retrain, 00:42:10.970 --> 00:42:13.010 it finds its solution very fast. 00:42:30.480 --> 00:42:34.170 Whoa, I got it all the way down to two neurons in each column, 00:42:34.170 --> 00:42:37.290 and it still has a solution. 00:42:37.290 --> 00:42:40.440 It's interesting, don't you think? 00:42:40.440 --> 00:42:43.120 But let's repeat the experiment, but this time we're 00:42:43.120 --> 00:42:45.181 going to do it a little differently. 00:42:45.181 --> 00:42:46.715 We're going to take our five layers, 00:42:46.715 --> 00:42:49.610 and before we do any training I'm 00:42:49.610 --> 00:42:57.390 going to knock out all but two neurons in each column. 00:42:57.390 --> 00:42:59.760 Now, I know that with two neurons in each column, 00:42:59.760 --> 00:43:01.520 I've got a solution. 00:43:01.520 --> 00:43:02.370 I just showed it. 00:43:02.370 --> 00:43:03.345 I just showed one. 00:43:03.345 --> 00:43:06.050 But let's run it this way. 00:43:19.060 --> 00:43:23.790 It looks like increasingly bad news. 00:43:23.790 --> 00:43:25.810 What's happened is that this sucker's got itself 00:43:25.810 --> 00:43:28.440 into a local maximum. 00:43:28.440 --> 00:43:31.550 So now you can see why there's been 00:43:31.550 --> 00:43:35.600 a breakthrough in this neural net learning stuff. 00:43:35.600 --> 00:43:39.300 And it's because when you widen the net, 00:43:39.300 --> 00:43:43.750 you turn local maxima into saddle points. 00:43:43.750 --> 00:43:45.560 So now it's got a way of crawling its way 00:43:45.560 --> 00:43:48.790 through this vast space without getting 00:43:48.790 --> 00:43:53.650 stuck on a local maximum, as suggested by this. 00:43:53.650 --> 00:43:54.150 All right. 00:43:54.150 --> 00:43:57.880 So those are some, I think, interesting things 00:43:57.880 --> 00:44:01.810 to look at by way of these demonstrations. 00:44:01.810 --> 00:44:04.510 But now I'd like to go back to my slide set 00:44:04.510 --> 00:44:06.860 and show you some examples that will address 00:44:06.860 --> 00:44:09.670 the question of whether these things are seeing like we see. 00:44:20.610 --> 00:44:22.380 So you can try these examples online. 00:44:22.380 --> 00:44:24.370 There are a variety of websites that allow 00:44:24.370 --> 00:44:27.950 you to put in your own picture. 00:44:27.950 --> 00:44:33.510 And there's a cottage industry of producing papers in journals 00:44:33.510 --> 00:44:35.840 that fool neural nets. 00:44:35.840 --> 00:44:38.600 So in this case, a very small number of pixels 00:44:38.600 --> 00:44:39.420 have been changed. 00:44:39.420 --> 00:44:41.640 You don't see the difference, but it's 00:44:41.640 --> 00:44:44.290 enough to take this particular neural net 00:44:44.290 --> 00:44:47.850 from a high confidence that it's looking at a school bus 00:44:47.850 --> 00:44:51.777 to thinking that it's not a school bus. 00:44:51.777 --> 00:44:54.026 Those are some things that it thinks are a school bus. 00:44:56.780 --> 00:44:58.490 So it appears to be the case that what 00:44:58.490 --> 00:45:01.320 is triggering this school bus result 00:45:01.320 --> 00:45:04.340 is that it's seeing enough local evidence that this is not 00:45:04.340 --> 00:45:10.080 one of the other 999 classes and enough positive evidence 00:45:10.080 --> 00:45:12.310 from these local looks to conclude 00:45:12.310 --> 00:45:13.313 that it's a school bus. 00:45:18.020 --> 00:45:20.330 So do you see any of those things? 00:45:20.330 --> 00:45:20.870 I don't. 00:45:24.494 --> 00:45:28.290 And here you can say, OK, well, look at that baseball one. 00:45:28.290 --> 00:45:31.500 Yeah, that looks like it's got a little bit of baseball texture 00:45:31.500 --> 00:45:32.020 in it. 00:45:32.020 --> 00:45:33.978 So maybe what it's doing is looking at texture. 00:45:39.130 --> 00:45:43.380 These are some examples from a recent and very famous 00:45:43.380 --> 00:45:47.380 paper by Google using essentially the same ideas 00:45:47.380 --> 00:45:51.290 to put captions on pictures. 00:45:51.290 --> 00:45:53.790 So this, by the way, is what has stimulated 00:45:53.790 --> 00:45:56.620 all this enormous concern about artificial intelligence. 00:45:56.620 --> 00:45:58.870 Because a naive viewer looks at that picture and says, 00:45:58.870 --> 00:46:00.245 oh, my god, this thing knows what 00:46:00.245 --> 00:46:06.260 it's like to play, or be young, or move, or what a Frisbee is. 00:46:06.260 --> 00:46:08.070 And of course, it knows none of that. 00:46:08.070 --> 00:46:10.950 It just knows how to label this picture. 00:46:10.950 --> 00:46:14.080 And to the credit of the people who wrote this paper, 00:46:14.080 --> 00:46:17.540 they show examples that don't do so well. 00:46:17.540 --> 00:46:21.000 So yeah, it's a cat, but it's not lying. 00:46:21.000 --> 00:46:24.620 Oh, it's a little girl, but she's not blowing bubbles. 00:46:24.620 --> 00:46:25.884 What about this one? 00:46:25.884 --> 00:46:28.848 [LAUGHTER] 00:46:31.820 --> 00:46:34.770 So we've been doing our own work in my laboratory 00:46:34.770 --> 00:46:36.390 on some of this. 00:46:36.390 --> 00:46:39.900 And the way the following set of pictures was produced was this. 00:46:39.900 --> 00:46:41.910 You take an image, and you separate it 00:46:41.910 --> 00:46:44.310 into a bunch of slices, each representing 00:46:44.310 --> 00:46:46.760 a particular frequency band. 00:46:46.760 --> 00:46:49.300 And then you go into one of those frequency bands 00:46:49.300 --> 00:46:51.680 and you knock out a rectangle from the picture, 00:46:51.680 --> 00:46:54.730 and then you reassemble the thing. 00:46:54.730 --> 00:46:56.876 And if you hadn't knocked that piece out, 00:46:56.876 --> 00:46:58.750 when you reassemble it, it would look exactly 00:46:58.750 --> 00:47:00.760 like it did when you started. 00:47:00.760 --> 00:47:03.370 So what we're doing is we knock out as much as we can 00:47:03.370 --> 00:47:05.827 and still retain the neural net's impression 00:47:05.827 --> 00:47:08.160 that it's the thing that it started out thinking it was. 00:47:08.160 --> 00:47:09.575 So what do you think this is? 00:47:13.640 --> 00:47:17.310 It's identified by a neural net as a railroad car 00:47:17.310 --> 00:47:21.589 because this is the image that it started with. 00:47:21.589 --> 00:47:22.380 How about this one? 00:47:22.380 --> 00:47:23.370 That's easy, right? 00:47:23.370 --> 00:47:25.100 That's a guitar. 00:47:25.100 --> 00:47:28.090 We weren't able to mutilate that one very much and still retain 00:47:28.090 --> 00:47:30.830 the guitar-ness of it. 00:47:30.830 --> 00:47:32.320 How about this one? 00:47:32.320 --> 00:47:33.029 AUDIENCE: A lamp? 00:47:33.029 --> 00:47:34.361 PATRICK H. WINSTON: What's that? 00:47:34.361 --> 00:47:35.020 AUDIENCE: Lamp. 00:47:35.020 --> 00:47:35.250 PATRICK H. WINSTON: What? 00:47:35.250 --> 00:47:36.190 AUDIENCE: Lamp. 00:47:36.190 --> 00:47:37.330 PATRICK H. WINSTON: A lamp. 00:47:37.330 --> 00:47:38.067 Any other ideas? 00:47:38.067 --> 00:47:38.983 AUDIENCE: [INAUDIBLE]. 00:47:38.983 --> 00:47:40.280 AUDIENCE: [INAUDIBLE]. 00:47:40.280 --> 00:47:42.321 PATRICK H. WINSTON: Ken, what do you think it is? 00:47:42.321 --> 00:47:43.157 AUDIENCE: A toilet. 00:47:43.157 --> 00:47:45.490 PATRICK H. WINSTON: See, he's an expert on this subject. 00:47:45.490 --> 00:47:46.880 [LAUGHTER] 00:47:46.880 --> 00:47:50.480 It was identified as a barbell. 00:47:50.480 --> 00:47:51.290 What's that? 00:47:51.290 --> 00:47:52.206 AUDIENCE: [INAUDIBLE]. 00:47:52.206 --> 00:47:53.450 PATRICK H. WINSTON: A what? 00:47:53.450 --> 00:47:54.340 AUDIENCE: Cello. 00:47:54.340 --> 00:47:55.730 PATRICK H. WINSTON: Cello. 00:47:55.730 --> 00:47:59.361 You didn't see the little girl or the instructor. 00:47:59.361 --> 00:48:00.152 How about this one? 00:48:00.152 --> 00:48:01.330 AUDIENCE: [INAUDIBLE]. 00:48:01.330 --> 00:48:01.830 PATRICK H. WINSTON: What? 00:48:01.830 --> 00:48:02.830 AUDIENCE: [INAUDIBLE]. 00:48:02.830 --> 00:48:03.788 PATRICK H. WINSTON: No. 00:48:07.205 --> 00:48:08.630 AUDIENCE: [INAUDIBLE]. 00:48:08.630 --> 00:48:10.680 PATRICK H. WINSTON: It's a grasshopper. 00:48:10.680 --> 00:48:11.390 What's this? 00:48:11.390 --> 00:48:12.330 AUDIENCE: A wolf. 00:48:12.330 --> 00:48:13.871 PATRICK H. WINSTON: Wow, you're good. 00:48:15.870 --> 00:48:17.693 It's actually not a two-headed wolf. 00:48:17.693 --> 00:48:20.000 [LAUGHTER] 00:48:20.000 --> 00:48:23.438 It's two wolves that are close together. 00:48:23.438 --> 00:48:24.694 AUDIENCE: [INAUDIBLE]. 00:48:24.694 --> 00:48:26.402 PATRICK H. WINSTON: That's a bird, right? 00:48:26.402 --> 00:48:27.775 AUDIENCE: [INAUDIBLE]. 00:48:27.775 --> 00:48:29.150 PATRICK H. WINSTON: Good for you. 00:48:29.150 --> 00:48:29.837 It's a rabbit. 00:48:29.837 --> 00:48:32.194 [LAUGHTER] 00:48:32.194 --> 00:48:32.819 How about that? 00:48:32.819 --> 00:48:33.819 [? AUDIENCE: Giraffe. ?] 00:48:36.040 --> 00:48:38.362 PATRICK H. WINSTON: Russian wolfhound. 00:48:38.362 --> 00:48:39.278 AUDIENCE: [INAUDIBLE]. 00:48:46.415 --> 00:48:48.290 PATRICK H. WINSTON: If you've been to Venice, 00:48:48.290 --> 00:48:49.314 you recognize this. 00:48:49.314 --> 00:48:51.920 AUDIENCE: [INAUDIBLE]. 00:48:51.920 --> 00:48:54.230 PATRICK H. WINSTON: So bottom line 00:48:54.230 --> 00:48:55.960 is that these things are an engineering 00:48:55.960 --> 00:49:00.536 marvel and do great things, but they don't see like we see.