WEBVTT 00:00:00.000 --> 00:00:00.490 align:middle line:90% 00:00:00.490 --> 00:00:00.990 align:middle line:90% [SQUEAKING] 00:00:00.990 --> 00:00:01.970 align:middle line:90% [CLICKING] 00:00:01.970 --> 00:00:02.950 align:middle line:90% [RUSTLING] 00:00:02.950 --> 00:00:03.920 align:middle line:90% [CLICKING] 00:00:03.920 --> 00:00:04.420 align:middle line:90% [TAPPING] 00:00:04.420 --> 00:00:12.447 align:middle line:90% 00:00:12.447 --> 00:00:13.280 align:middle line:90% PHILLIP ISOLA: Good. 00:00:13.280 --> 00:00:15.840 align:middle line:90% So welcome to today's lecture. 00:00:15.840 --> 00:00:20.560 align:middle line:84% And today, we're going to talk about representation learning. 00:00:20.560 --> 00:00:24.020 align:middle line:84% So we're moving into the next third of the class. 00:00:24.020 --> 00:00:27.240 align:middle line:84% So we covered basic approximation and generalization 00:00:27.240 --> 00:00:29.600 align:middle line:90% properties of MLPs. 00:00:29.600 --> 00:00:33.440 align:middle line:84% And then we talked about different architectures, 00:00:33.440 --> 00:00:36.223 align:middle line:84% transformers, and RNNs, and so forth. 00:00:36.223 --> 00:00:38.640 align:middle line:84% And now we're going to come to a kind of a new perspective 00:00:38.640 --> 00:00:42.120 align:middle line:84% on deep learning, which is the idea of representation learning 00:00:42.120 --> 00:00:44.640 align:middle line:90% and generative modeling. 00:00:44.640 --> 00:00:48.040 align:middle line:84% And this is a perspective, which is associated mostly 00:00:48.040 --> 00:00:52.360 align:middle line:84% with deep neural networks, but actually could be construed 00:00:52.360 --> 00:00:53.620 align:middle line:90% to be broader than this. 00:00:53.620 --> 00:00:55.360 align:middle line:84% You could do representation learning and generative 00:00:55.360 --> 00:00:56.840 align:middle line:84% modeling with other architectures 00:00:56.840 --> 00:00:59.930 align:middle line:84% that are maybe not neural networks, 00:00:59.930 --> 00:01:02.305 align:middle line:84% but most of the techniques have been developed intimately 00:01:02.305 --> 00:01:03.805 align:middle line:84% in conjunction with neural networks. 00:01:03.805 --> 00:01:05.530 align:middle line:84% And this field of representation learning 00:01:05.530 --> 00:01:08.650 align:middle line:84% usually goes hand in hand with deep learning. 00:01:08.650 --> 00:01:13.410 align:middle line:84% So we'll talk about the idea that deep nets 00:01:13.410 --> 00:01:14.430 align:middle line:90% learn representations. 00:01:14.430 --> 00:01:16.010 align:middle line:90% What does that even mean? 00:01:16.010 --> 00:01:18.770 align:middle line:84% We'll ask, why should we learn representations? 00:01:18.770 --> 00:01:22.730 align:middle line:84% And then we'll look at one kind of representation learner called 00:01:22.730 --> 00:01:24.610 align:middle line:90% the autoencoder. 00:01:24.610 --> 00:01:26.810 align:middle line:84% We'll look at the relationship to clustering 00:01:26.810 --> 00:01:29.080 align:middle line:84% and what's called vector quantization. 00:01:29.080 --> 00:01:31.330 align:middle line:84% And we'll look at the idea of learning representations 00:01:31.330 --> 00:01:35.730 align:middle line:84% via prediction and what's called self-supervised learning. 00:01:35.730 --> 00:01:37.308 align:middle line:84% And then, in the next lecture, we'll 00:01:37.308 --> 00:01:39.850 align:middle line:84% talk about a different type of representation learning, which 00:01:39.850 --> 00:01:42.450 align:middle line:84% is called metric learning or contrastive 00:01:42.450 --> 00:01:45.850 align:middle line:84% learning, where you learn about similarities and distances. 00:01:45.850 --> 00:01:50.370 align:middle line:84% So this is going to be the new picture for deep learning 00:01:50.370 --> 00:01:52.650 align:middle line:90% for the next few weeks. 00:01:52.650 --> 00:01:55.770 align:middle line:84% We have data that comes into a system on the bottom. 00:01:55.770 --> 00:01:57.757 align:middle line:84% And then layer by layer, we change 00:01:57.757 --> 00:01:59.590 align:middle line:84% the changed the representation of that data, 00:01:59.590 --> 00:02:01.132 align:middle line:84% until we get to an output at the top. 00:02:01.132 --> 00:02:04.310 align:middle line:84% And we'll call the output an embedding or representation. 00:02:04.310 --> 00:02:07.710 align:middle line:84% But actually, each layer is a representation of the data. 00:02:07.710 --> 00:02:11.070 align:middle line:84% So layer by layer, the deep net takes your raw data 00:02:11.070 --> 00:02:15.630 align:middle line:84% x and turns it into f of x, and then f of f of x, and so forth. 00:02:15.630 --> 00:02:18.377 align:middle line:84% And each layer is a different representation, 00:02:18.377 --> 00:02:20.210 align:middle line:84% so the data will be differently distributed. 00:02:20.210 --> 00:02:23.030 align:middle line:90% It'll be transformed somehow. 00:02:23.030 --> 00:02:25.150 align:middle line:84% And the forward direction through a neural network 00:02:25.150 --> 00:02:27.650 align:middle line:84% will be called the encoding direction. 00:02:27.650 --> 00:02:31.110 align:middle line:84% So it's going from data to maybe simpler, better, more abstract 00:02:31.110 --> 00:02:32.655 align:middle line:90% representations. 00:02:32.655 --> 00:02:35.030 align:middle line:84% And that's the direction in which representation learning 00:02:35.030 --> 00:02:35.530 align:middle line:90% operates. 00:02:35.530 --> 00:02:38.470 align:middle line:84% It tries to find a good mapping from data to embeddings. 00:02:38.470 --> 00:02:40.890 align:middle line:84% And the backwards direction is not going to be backprop. 00:02:40.890 --> 00:02:42.570 align:middle line:84% Now backprop tries to find the gradient. 00:02:42.570 --> 00:02:44.570 align:middle line:84% The backwards direction is more like the inverse 00:02:44.570 --> 00:02:46.350 align:middle line:84% of the forward direction in this context. 00:02:46.350 --> 00:02:47.933 align:middle line:84% And that's called generative modeling. 00:02:47.933 --> 00:02:50.590 align:middle line:84% It's going from abstract representations and latent 00:02:50.590 --> 00:02:52.610 align:middle line:90% variables to data. 00:02:52.610 --> 00:02:56.400 align:middle line:84% And we'll talk about generative modeling in a few weeks. 00:02:56.400 --> 00:02:57.890 align:middle line:90% OK. 00:02:57.890 --> 00:02:59.640 align:middle line:84% So these two things can roughly be thought 00:02:59.640 --> 00:03:00.848 align:middle line:90% of as inverses of each other. 00:03:00.848 --> 00:03:03.538 align:middle line:84% And we'll clarify that connection as we go. 00:03:03.538 --> 00:03:06.080 align:middle line:84% But today, we're just going to talk about the first direction 00:03:06.080 --> 00:03:08.120 align:middle line:84% going forward through a network and view that 00:03:08.120 --> 00:03:11.480 align:middle line:90% as learning representations. 00:03:11.480 --> 00:03:18.680 align:middle line:84% And one label for this idea is data to vec, so x2vec. 00:03:18.680 --> 00:03:19.920 align:middle line:90% X is your data. 00:03:19.920 --> 00:03:23.740 align:middle line:84% And people have developed x2vec for all different modalities. 00:03:23.740 --> 00:03:27.560 align:middle line:84% So the most famous early version of x2vec was called word2vec. 00:03:27.560 --> 00:03:30.640 align:middle line:84% You want to go from words in English or some other language 00:03:30.640 --> 00:03:34.008 align:middle line:84% to representations that are vectors. 00:03:34.008 --> 00:03:35.300 align:middle line:90% But you can do this for images. 00:03:35.300 --> 00:03:36.425 align:middle line:90% You can do this for sounds. 00:03:36.425 --> 00:03:38.920 align:middle line:84% You can do this for molecules, whatever you want. 00:03:38.920 --> 00:03:40.417 align:middle line:84% And here's an example of an image. 00:03:40.417 --> 00:03:41.500 align:middle line:90% We take our data as input. 00:03:41.500 --> 00:03:43.100 align:middle line:84% And then, now, we're thinking, layer by layer, 00:03:43.100 --> 00:03:45.295 align:middle line:84% we get a vector representation or maybe a tensor 00:03:45.295 --> 00:03:46.420 align:middle line:90% representation of the data. 00:03:46.420 --> 00:03:48.503 align:middle line:84% And let's investigate what that representation is, 00:03:48.503 --> 00:03:52.840 align:middle line:84% what each of these is indicating a neuron or mathematically just 00:03:52.840 --> 00:03:55.980 align:middle line:90% a scalar dimension of a vector. 00:03:55.980 --> 00:03:58.940 align:middle line:84% And we can investigate what those actually represent. 00:03:58.940 --> 00:04:01.980 align:middle line:84% So we'll talk about the layer 3 representation of this image 00:04:01.980 --> 00:04:04.620 align:middle line:84% is if I go through three layers of neural processing, three 00:04:04.620 --> 00:04:06.460 align:middle line:84% convolutional layers, for example, 00:04:06.460 --> 00:04:08.400 align:middle line:84% what is the tensor of activations I get out? 00:04:08.400 --> 00:04:10.740 align:middle line:84% That's the representation at layer 3. 00:04:10.740 --> 00:04:13.660 align:middle line:84% And then we can talk about other layers, too. 00:04:13.660 --> 00:04:16.140 align:middle line:84% So the representations will be a vector or tensor 00:04:16.140 --> 00:04:17.560 align:middle line:90% of neural activations. 00:04:17.560 --> 00:04:19.980 align:middle line:84% And in this example, they might be representing 00:04:19.980 --> 00:04:23.740 align:middle line:84% that there's a car, or there's a ground, or so forth, a building, 00:04:23.740 --> 00:04:24.970 align:middle line:90% a road. 00:04:24.970 --> 00:04:27.220 align:middle line:84% So we're going to try to see how representations might 00:04:27.220 --> 00:04:29.900 align:middle line:84% map onto these semantic representations 00:04:29.900 --> 00:04:31.500 align:middle line:90% that we think about in our head. 00:04:31.500 --> 00:04:32.340 align:middle line:90% OK, question? 00:04:32.340 --> 00:04:36.580 align:middle line:84% AUDIENCE: With this family of two vectors, 00:04:36.580 --> 00:04:38.940 align:middle line:90% are they feature extractions? 00:04:38.940 --> 00:04:40.980 align:middle line:84% PHILLIP ISOLA: Feature extraction is the same-- 00:04:40.980 --> 00:04:43.420 align:middle line:84% is another name for the same thing, yeah. 00:04:43.420 --> 00:04:49.500 align:middle line:84% So x2vec can also be called extracting features 00:04:49.500 --> 00:04:53.580 align:middle line:84% from input data x, and the features in the form of vectors 00:04:53.580 --> 00:04:56.267 align:middle line:90% or tensors in this case. 00:04:56.267 --> 00:04:57.350 align:middle line:90% We have one more question. 00:04:57.350 --> 00:04:58.808 align:middle line:84% AUDIENCE: Was x2vec, traditionally, 00:04:58.808 --> 00:05:01.870 align:middle line:84% only getting from bigger data sets to smaller representations? 00:05:01.870 --> 00:05:03.070 align:middle line:90% [INAUDIBLE] 00:05:03.070 --> 00:05:05.205 align:middle line:84% PHILLIP ISOLA: So the question is, is x2vec always 00:05:05.205 --> 00:05:06.830 align:middle line:84% going to smaller representations, lower 00:05:06.830 --> 00:05:08.430 align:middle line:90% dimensional representations? 00:05:08.430 --> 00:05:11.813 align:middle line:84% Well, that is one of the classic ways of doing it. 00:05:11.813 --> 00:05:13.230 align:middle line:84% Often, the representation you want 00:05:13.230 --> 00:05:15.928 align:middle line:84% to be a smaller object, a lower dimensional 00:05:15.928 --> 00:05:17.470 align:middle line:84% vector than the input, but that's not 00:05:17.470 --> 00:05:18.512 align:middle line:90% the only way of doing it. 00:05:18.512 --> 00:05:20.750 align:middle line:84% You don't have to do it that way. 00:05:20.750 --> 00:05:22.870 align:middle line:90% OK, the intuition here is-- 00:05:22.870 --> 00:05:25.370 align:middle line:84% I'll use these cartoons a few times over the next few weeks, 00:05:25.370 --> 00:05:27.310 align:middle line:84% or we'll use them in a few lectures. 00:05:27.310 --> 00:05:28.970 align:middle line:84% So the data will be this blue blob. 00:05:28.970 --> 00:05:33.430 align:middle line:84% It's some complicated object, high-dimensional, weird 00:05:33.430 --> 00:05:34.390 align:middle line:90% topology. 00:05:34.390 --> 00:05:36.850 align:middle line:90% That's like the space of images. 00:05:36.850 --> 00:05:39.830 align:middle line:90% It's this weird manifold. 00:05:39.830 --> 00:05:45.590 align:middle line:84% And the representation will be this orange or reddish circle. 00:05:45.590 --> 00:05:47.830 align:middle line:84% And it's a circle because we're going 00:05:47.830 --> 00:05:50.550 align:middle line:84% to make the point that the representation is, oftentimes, 00:05:50.550 --> 00:05:52.230 align:middle line:90% a simpler space. 00:05:52.230 --> 00:05:54.850 align:middle line:84% So the distribution over embeddings 00:05:54.850 --> 00:05:56.830 align:middle line:84% is often going to be a very simple object. 00:05:56.830 --> 00:05:59.750 align:middle line:84% In fact, it will typically be a Gaussian distribution. 00:05:59.750 --> 00:06:02.770 align:middle line:84% So Gaussian is a circle in high dimensions, 00:06:02.770 --> 00:06:05.490 align:middle line:84% or Gaussian in high dimensions looks like a circle 00:06:05.490 --> 00:06:06.850 align:middle line:90% OK. 00:06:06.850 --> 00:06:09.290 align:middle line:84% So this is the picture where somehow learning embeddings 00:06:09.290 --> 00:06:10.070 align:middle line:90% are simpler. 00:06:10.070 --> 00:06:12.570 align:middle line:84% They have a simpler distribution than the data distribution. 00:06:12.570 --> 00:06:15.270 align:middle line:84% The input blob is like the data distribution. 00:06:15.270 --> 00:06:16.770 align:middle line:84% And we call the mapping from data 00:06:16.770 --> 00:06:20.570 align:middle line:90% to representation an encoder. 00:06:20.570 --> 00:06:24.845 align:middle line:84% So I'm going to now show some visuals. 00:06:24.845 --> 00:06:26.970 align:middle line:84% And I want to introduce a different way of thinking 00:06:26.970 --> 00:06:28.930 align:middle line:84% about functions than what you might have 00:06:28.930 --> 00:06:31.770 align:middle line:90% learned about in grade school. 00:06:31.770 --> 00:06:36.230 align:middle line:84% So when you first encountered functions, 00:06:36.230 --> 00:06:38.150 align:middle line:84% we learned this way of representing them. 00:06:38.150 --> 00:06:41.410 align:middle line:84% You have on the x-axis the input and on the y-axis the output 00:06:41.410 --> 00:06:42.552 align:middle line:90% to your function. 00:06:42.552 --> 00:06:44.010 align:middle line:84% But that's not really the only way. 00:06:44.010 --> 00:06:47.010 align:middle line:84% That's basically telling you, what is the mapping from Xin 00:06:47.010 --> 00:06:48.990 align:middle line:84% to Xout, or from x to y, traditionally. 00:06:48.990 --> 00:06:51.380 align:middle line:84% But in this class, we usually talk about Xin 00:06:51.380 --> 00:06:53.900 align:middle line:90% to Xout for some layer. 00:06:53.900 --> 00:06:56.100 align:middle line:84% So this is just telling me that if I'm 00:06:56.100 --> 00:06:59.040 align:middle line:84% at some point on the x-axis, where do I map to on the y-axis? 00:06:59.040 --> 00:07:01.100 align:middle line:84% It's telling me something about the mapping. 00:07:01.100 --> 00:07:05.000 align:middle line:84% So we're just going to rotate the y-axis to be at the top. 00:07:05.000 --> 00:07:06.620 align:middle line:84% And now we'll look at this mapping 00:07:06.620 --> 00:07:11.100 align:middle line:84% as mapping a set of points on the input dimension 00:07:11.100 --> 00:07:13.060 align:middle line:84% to the set of points on the output dimension. 00:07:13.060 --> 00:07:15.393 align:middle line:84% And the identity mapping now looks really simple, right? 00:07:15.393 --> 00:07:16.440 align:middle line:90% It's just straight lines. 00:07:16.440 --> 00:07:17.820 align:middle line:90% Nothing changed. 00:07:17.820 --> 00:07:20.017 align:middle line:84% So I don't know why we decided to go 00:07:20.017 --> 00:07:22.100 align:middle line:84% with this way of representing functions as opposed 00:07:22.100 --> 00:07:24.007 align:middle line:84% to this way, but I think both of them 00:07:24.007 --> 00:07:26.340 align:middle line:84% have some interesting properties and advantages in terms 00:07:26.340 --> 00:07:28.860 align:middle line:90% of clarifying what's going on. 00:07:28.860 --> 00:07:32.660 align:middle line:84% But the second way will help us to visualize 00:07:32.660 --> 00:07:34.513 align:middle line:84% some of the mappings in neural networks. 00:07:34.513 --> 00:07:36.180 align:middle line:84% Actually, in some of the earlier slides, 00:07:36.180 --> 00:07:39.140 align:middle line:84% we already showed a few examples of this. 00:07:39.140 --> 00:07:42.300 align:middle line:84% So here are a few different neural layers for a scalar 00:07:42.300 --> 00:07:43.480 align:middle line:90% input to a scalar output. 00:07:43.480 --> 00:07:46.780 align:middle line:84% So this is the width of the network is 1, in this case. 00:07:46.780 --> 00:07:50.990 align:middle line:90% And what does a linear layer do? 00:07:50.990 --> 00:07:55.110 align:middle line:84% A linear layer can-- sorry, if the linear layer has weight 2, 00:07:55.110 --> 00:07:56.450 align:middle line:90% it will just expand. 00:07:56.450 --> 00:07:59.910 align:middle line:84% It'll take your input data, and it will make it scaled up. 00:07:59.910 --> 00:08:02.990 align:middle line:90% So it'll just spread it out. 00:08:02.990 --> 00:08:04.090 align:middle line:90% What does a ReLU do? 00:08:04.090 --> 00:08:05.670 align:middle line:90% OK, ReLU is kind of interesting. 00:08:05.670 --> 00:08:08.590 align:middle line:84% A ReLU takes all of your data in the negative half space 00:08:08.590 --> 00:08:10.230 align:middle line:90% and maps it to 0. 00:08:10.230 --> 00:08:14.230 align:middle line:84% So this ReLU has this property that the density of data 00:08:14.230 --> 00:08:17.110 align:middle line:90% that will get mapped to 0. 00:08:17.110 --> 00:08:18.710 align:middle line:84% You'll usually have a spike of density 00:08:18.710 --> 00:08:21.590 align:middle line:84% at 0, which we'll see on the next slide. 00:08:21.590 --> 00:08:25.370 align:middle line:84% And the sigmoid will map most of your inputs to either 0 or 1, 00:08:25.370 --> 00:08:28.750 align:middle line:90% but in a soft way. 00:08:28.750 --> 00:08:31.790 align:middle line:84% So I like thinking about functions this way 00:08:31.790 --> 00:08:34.470 align:middle line:84% because I think it helps to understand representations 00:08:34.470 --> 00:08:36.669 align:middle line:84% and distributional transformations. 00:08:36.669 --> 00:08:38.318 align:middle line:84% So think of your functions as taking 00:08:38.318 --> 00:08:40.110 align:middle line:84% a set, a distribution of input data points, 00:08:40.110 --> 00:08:43.309 align:middle line:84% and doing these geometric transformations, 00:08:43.309 --> 00:08:47.220 align:middle line:84% squishing them, and skewing them, and remapping them. 00:08:47.220 --> 00:08:49.470 align:middle line:84% And so here are some of the functions that we've seen, 00:08:49.470 --> 00:08:51.730 align:middle line:84% some of the basic neural networks 00:08:51.730 --> 00:08:54.050 align:middle line:84% layers that we've seen in this class. 00:08:54.050 --> 00:08:58.170 align:middle line:84% So we have the linear, the ReLU, the normalization layers, 00:08:58.170 --> 00:09:00.130 align:middle line:90% softmax layers. 00:09:00.130 --> 00:09:03.997 align:middle line:84% And activations are going to be colored in this red font. 00:09:03.997 --> 00:09:05.830 align:middle line:84% And parameters are colored in the blue font, 00:09:05.830 --> 00:09:07.832 align:middle line:84% just to clarify what is actually being learned. 00:09:07.832 --> 00:09:10.290 align:middle line:84% So the only learnable parameters are the weights and biases 00:09:10.290 --> 00:09:12.650 align:middle line:90% in these layers. 00:09:12.650 --> 00:09:15.158 align:middle line:90% So here's the wiring diagrams. 00:09:15.158 --> 00:09:17.450 align:middle line:84% So let's look at the mapping diagrams for these layers, 00:09:17.450 --> 00:09:20.610 align:middle line:90% just to build up this intuition. 00:09:20.610 --> 00:09:23.250 align:middle line:84% So we're always mapping from Xin to Xout. 00:09:23.250 --> 00:09:28.170 align:middle line:84% Now one nice property of this other way of plotting functions 00:09:28.170 --> 00:09:30.750 align:middle line:84% is I don't have to only plot scalar to scalar functions. 00:09:30.750 --> 00:09:33.430 align:middle line:90% I can plot 2D to 2D functions. 00:09:33.430 --> 00:09:34.680 align:middle line:90% I don't have to plot 3D to 3D. 00:09:34.680 --> 00:09:35.847 align:middle line:90% That's where it breaks down. 00:09:35.847 --> 00:09:37.270 align:middle line:84% But at least I can go to 2D to 2D, 00:09:37.270 --> 00:09:40.470 align:middle line:84% which you can't do with traditional plotting. 00:09:40.470 --> 00:09:41.890 align:middle line:90% So here's the mapping. 00:09:41.890 --> 00:09:43.690 align:middle line:84% I'm showing in two different ways. 00:09:43.690 --> 00:09:47.580 align:middle line:84% On the left, I'm showing how each point on a grid of input 00:09:47.580 --> 00:09:51.140 align:middle line:90% values maps to the output. 00:09:51.140 --> 00:09:54.180 align:middle line:84% And on the right, I'm showing how each point in a Gaussian 00:09:54.180 --> 00:09:57.560 align:middle line:84% cloud over the input space maps to the output. 00:09:57.560 --> 00:09:59.120 align:middle line:90% So 2D data goes in. 00:09:59.120 --> 00:10:00.680 align:middle line:84% And what does the linear layer do? 00:10:00.680 --> 00:10:02.980 align:middle line:84% It's going to be an affine transformation. 00:10:02.980 --> 00:10:05.820 align:middle line:84% And geometrically, it will just look like rotating, squishing, 00:10:05.820 --> 00:10:08.075 align:middle line:90% scaling, anything like this. 00:10:08.075 --> 00:10:09.700 align:middle line:84% So it's like a geometric transformation 00:10:09.700 --> 00:10:11.660 align:middle line:90% of the input data. 00:10:11.660 --> 00:10:13.660 align:middle line:90% That's all it's doing. 00:10:13.660 --> 00:10:15.580 align:middle line:90% What does a ReLU do? 00:10:15.580 --> 00:10:18.920 align:middle line:84% So a ReLU looks like this funny shape. 00:10:18.920 --> 00:10:21.580 align:middle line:84% So there's two important properties to notice. 00:10:21.580 --> 00:10:25.900 align:middle line:84% So one is the ReLU maps all data to the positive orthant, 00:10:25.900 --> 00:10:26.760 align:middle line:90% in this case. 00:10:26.760 --> 00:10:28.343 align:middle line:84% So in high dimensions, it's just going 00:10:28.343 --> 00:10:32.860 align:middle line:84% to be the subspace where all the dimensions are 00:10:32.860 --> 00:10:36.100 align:middle line:84% positive because anything that's negative gets mapped to 0. 00:10:36.100 --> 00:10:38.900 align:middle line:84% And two is you can see how most of the points 00:10:38.900 --> 00:10:40.320 align:middle line:90% get mapped to these axes. 00:10:40.320 --> 00:10:43.240 align:middle line:84% You get this what's called a sparse representation. 00:10:43.240 --> 00:10:45.760 align:middle line:84% So ReLU has this property of making things sparser 00:10:45.760 --> 00:10:49.620 align:middle line:84% because most of the inputs or in each dimension, 00:10:49.620 --> 00:10:51.483 align:middle line:84% half of the values will tend to be negative 00:10:51.483 --> 00:10:52.900 align:middle line:84% and half will tend to be positive. 00:10:52.900 --> 00:10:58.880 align:middle line:84% And the negative values will get mapped to the axes here. 00:10:58.880 --> 00:11:02.253 align:middle line:84% And there'll be a lot of density right at the origin 00:11:02.253 --> 00:11:03.920 align:middle line:84% because that will correspond to anything 00:11:03.920 --> 00:11:05.900 align:middle line:84% in this strictly negative orthant. 00:11:05.900 --> 00:11:08.680 align:middle line:84% And in high dimensions, I think this effect becomes even more 00:11:08.680 --> 00:11:11.800 align:middle line:90% extreme. 00:11:11.800 --> 00:11:13.985 align:middle line:90% So what does L2 norm look like? 00:11:13.985 --> 00:11:15.360 align:middle line:84% And this is what I think I showed 00:11:15.360 --> 00:11:18.000 align:middle line:84% in one of the last lectures, but I didn't really 00:11:18.000 --> 00:11:19.960 align:middle line:90% fully explain it. 00:11:19.960 --> 00:11:23.680 align:middle line:84% What L2 norm will look like-- and same with RMSE norm, 00:11:23.680 --> 00:11:27.240 align:middle line:84% and same with LayerNorm, which is a variation on this-- 00:11:27.240 --> 00:11:29.020 align:middle line:84% is it will take all of your data, 00:11:29.020 --> 00:11:31.260 align:middle line:84% and it will map it to unitary vectors. 00:11:31.260 --> 00:11:33.680 align:middle line:84% It will map it to vectors that have norm 1. 00:11:33.680 --> 00:11:35.960 align:middle line:84% And so in this particular case, with Euclidean norm, 00:11:35.960 --> 00:11:39.080 align:middle line:84% it will map it to points on the circle. 00:11:39.080 --> 00:11:42.280 align:middle line:84% And high dimensions will map it to points on the hypersphere. 00:11:42.280 --> 00:11:45.090 align:middle line:90% So that's what these layers do. 00:11:45.090 --> 00:11:49.090 align:middle line:84% And that's nice for various reasons, but one of them 00:11:49.090 --> 00:11:53.090 align:middle line:84% is that we know that the numerics are 00:11:53.090 --> 00:11:57.290 align:middle line:90% going to be bounded somehow. 00:11:57.290 --> 00:11:59.870 align:middle line:84% The vectors will not go to infinity or not go to 0. 00:11:59.870 --> 00:12:02.510 align:middle line:84% They'll all get into a unitary range. 00:12:02.510 --> 00:12:05.885 align:middle line:90% 00:12:05.885 --> 00:12:08.268 align:middle line:90% So what is softmax? 00:12:08.268 --> 00:12:09.810 align:middle line:84% Now I'm going to ask you, can anybody 00:12:09.810 --> 00:12:11.550 align:middle line:84% explain, why does softmax look like that? 00:12:11.550 --> 00:12:14.958 align:middle line:84% What is that line that's showing up in the softmax graph? 00:12:14.958 --> 00:12:16.750 align:middle line:84% So I have input data, two-dimensional data, 00:12:16.750 --> 00:12:19.050 align:middle line:84% and I'm doing softmax of those two dimensions. 00:12:19.050 --> 00:12:21.770 align:middle line:84% Let's let me see if somebody else-- yeah? 00:12:21.770 --> 00:12:25.643 align:middle line:84% AUDIENCE: x1 plus x2 equals [INAUDIBLE] 00:12:25.643 --> 00:12:26.810 align:middle line:90% PHILLIP ISOLA: Yes, exactly. 00:12:26.810 --> 00:12:35.237 align:middle line:84% So the answer was that's a line that x1 plus x2 equals 1, 00:12:35.237 --> 00:12:37.570 align:middle line:84% or x plus y equals 1, if you think of one dimension as y 00:12:37.570 --> 00:12:39.330 align:middle line:90% and one dimension as x. 00:12:39.330 --> 00:12:42.428 align:middle line:90% And that's the simplex. 00:12:42.428 --> 00:12:44.470 align:middle line:84% Is that the one dimensional simplex or the zero-- 00:12:44.470 --> 00:12:46.262 align:middle line:84% I think that's the one-dimensional simplex. 00:12:46.262 --> 00:12:48.010 align:middle line:84% I can't remember how to define simplex. 00:12:48.010 --> 00:12:53.030 align:middle line:84% But anyway, the simplex is the set of points in Rd, 00:12:53.030 --> 00:12:55.270 align:middle line:90% where the dimensions sum to 1. 00:12:55.270 --> 00:12:57.870 align:middle line:84% And the output of a softmax is going 00:12:57.870 --> 00:13:00.110 align:middle line:90% to be a point in the simplex. 00:13:00.110 --> 00:13:01.850 align:middle line:90% So that's what that looks like. 00:13:01.850 --> 00:13:05.870 align:middle line:84% This is, basically, going to be your probability is 00:13:05.870 --> 00:13:08.110 align:middle line:84% high of one class, if you're at that point, 00:13:08.110 --> 00:13:12.550 align:middle line:84% and low for that class if you're at the other point. 00:13:12.550 --> 00:13:15.930 align:middle line:84% So this is what neural networks are doing. 00:13:15.930 --> 00:13:18.128 align:middle line:90% Those are our basic layers. 00:13:18.128 --> 00:13:20.670 align:middle line:84% It's a little more complicated with attention and other kinds 00:13:20.670 --> 00:13:22.790 align:middle line:84% of layers, but these are the basic ones. 00:13:22.790 --> 00:13:28.230 align:middle line:84% So let's now look at what an MLP looks like plotted this way. 00:13:28.230 --> 00:13:28.730 align:middle line:90% Let's see. 00:13:28.730 --> 00:13:30.438 align:middle line:84% This is a three-layer MLP because there's 00:13:30.438 --> 00:13:33.110 align:middle line:84% three linear layers, so our convention is three layers. 00:13:33.110 --> 00:13:35.750 align:middle line:84% And the input data is going to be 00:13:35.750 --> 00:13:38.630 align:middle line:84% a cloud of points at the origin with label red 00:13:38.630 --> 00:13:42.880 align:middle line:84% and a hemisphere of points around with label blue. 00:13:42.880 --> 00:13:46.440 align:middle line:84% So we're going to try to do a binary classification. 00:13:46.440 --> 00:13:48.680 align:middle line:84% So this will be training with cross-entropy loss 00:13:48.680 --> 00:13:51.040 align:middle line:90% doing softmax regression. 00:13:51.040 --> 00:13:53.603 align:middle line:84% And we're going to look at what training actually looks like. 00:13:53.603 --> 00:13:55.520 align:middle line:84% So this is a non-linear classification problem 00:13:55.520 --> 00:13:57.280 align:middle line:84% because there's no linear hyperplane that 00:13:57.280 --> 00:14:00.760 align:middle line:84% separates the red cloud from the blue cloud. 00:14:00.760 --> 00:14:03.440 align:middle line:84% So layer by layer, the representation 00:14:03.440 --> 00:14:06.022 align:middle line:84% of the distribution of data gets transformed. 00:14:06.022 --> 00:14:07.480 align:middle line:84% And what you're going to see happen 00:14:07.480 --> 00:14:10.300 align:middle line:84% is that the red points will move away from the blue points. 00:14:10.300 --> 00:14:12.100 align:middle line:90% And this is what it looks like. 00:14:12.100 --> 00:14:15.080 align:middle line:84% So after a few steps of gradient descent, 00:14:15.080 --> 00:14:16.960 align:middle line:84% they start to spread out like this. 00:14:16.960 --> 00:14:22.160 align:middle line:84% And at this layer we're shifting things over. 00:14:22.160 --> 00:14:24.340 align:middle line:84% And then the ReLU snaps it back onto the axes. 00:14:24.340 --> 00:14:26.860 align:middle line:84% And then we skew things in a different direction. 00:14:26.860 --> 00:14:29.580 align:middle line:84% And eventually, we get things to spread out on a line. 00:14:29.580 --> 00:14:33.760 align:middle line:84% And then the softmax clamps that back down to this simplex. 00:14:33.760 --> 00:14:37.213 align:middle line:84% And so the red points are all heading toward the 0.10. 00:14:37.213 --> 00:14:39.380 align:middle line:84% And the blue points are all heading toward the 0.01. 00:14:39.380 --> 00:14:41.580 align:middle line:84% And those are the one-hot labels for your data. 00:14:41.580 --> 00:14:43.342 align:middle line:84% So this optimization is actually working 00:14:43.342 --> 00:14:45.300 align:middle line:84% at segregating the blue points to be classified 00:14:45.300 --> 00:14:49.340 align:middle line:84% as class II and the red points to be classified as class 1. 00:14:49.340 --> 00:14:51.390 align:middle line:90% So let's look at that animated. 00:14:51.390 --> 00:14:52.640 align:middle line:90% So this is what it looks like. 00:14:52.640 --> 00:14:56.820 align:middle line:84% This is every single layer has width 2. 00:14:56.820 --> 00:14:58.202 align:middle line:90% So we can plot everything here. 00:14:58.202 --> 00:14:59.160 align:middle line:90% There's nothing hidden. 00:14:59.160 --> 00:15:00.980 align:middle line:84% It's not like I'm doing dimensionality reduction 00:15:00.980 --> 00:15:01.760 align:middle line:90% or visualizing. 00:15:01.760 --> 00:15:06.620 align:middle line:84% It's just the raw activation values at each of these layers. 00:15:06.620 --> 00:15:10.460 align:middle line:84% And each of the layers now can be understood 00:15:10.460 --> 00:15:13.020 align:middle line:84% as a different representation of the data distribution 00:15:13.020 --> 00:15:15.620 align:middle line:84% and a better and better representation 00:15:15.620 --> 00:15:17.300 align:middle line:84% for solving the class of segregating 00:15:17.300 --> 00:15:19.050 align:middle line:84% the red points from the blue points, which 00:15:19.050 --> 00:15:21.500 align:middle line:90% is the classification task. 00:15:21.500 --> 00:15:23.780 align:middle line:90% OK. 00:15:23.780 --> 00:15:30.370 align:middle line:84% Oh, yeah, just for fun, I didn't really have a deep insight 00:15:30.370 --> 00:15:31.120 align:middle line:90% to take from this. 00:15:31.120 --> 00:15:34.780 align:middle line:84% But just for fun, you can compare the optimization 00:15:34.780 --> 00:15:38.992 align:middle line:84% using SGD, gradient descent, to the optimization using 00:15:38.992 --> 00:15:40.950 align:middle line:84% the spectral descent method that you worked out 00:15:40.950 --> 00:15:42.670 align:middle line:90% in your problem set 2. 00:15:42.670 --> 00:15:45.190 align:middle line:84% So in the spectral descent method, 00:15:45.190 --> 00:15:48.870 align:middle line:84% you are normalizing the updates by their spectral norm. 00:15:48.870 --> 00:15:51.967 align:middle line:84% And you can see there's some interesting different dynamics. 00:15:51.967 --> 00:15:54.050 align:middle line:84% It's a little bit hard to read too much into this. 00:15:54.050 --> 00:15:56.150 align:middle line:84% Well, the spectral descent is descending faster, 00:15:56.150 --> 00:15:58.775 align:middle line:84% but that depends on the learning rate, so there's some caveats. 00:15:58.775 --> 00:16:00.210 align:middle line:84% But it is getting to a lower loss. 00:16:00.210 --> 00:16:01.918 align:middle line:84% The loss is tiny, little font up here, 00:16:01.918 --> 00:16:04.210 align:middle line:84% is actually segregating the red and blue points better. 00:16:04.210 --> 00:16:07.150 align:middle line:84% So you can understand that different optimization schemes 00:16:07.150 --> 00:16:09.048 align:middle line:90% actually do perform differently. 00:16:09.048 --> 00:16:11.590 align:middle line:84% And there was some reason for working out what you worked out 00:16:11.590 --> 00:16:12.550 align:middle line:90% in your problem set. 00:16:12.550 --> 00:16:14.550 align:middle line:84% I think something that might be interesting here 00:16:14.550 --> 00:16:17.750 align:middle line:84% is it looks like this first linear layer is 00:16:17.750 --> 00:16:21.730 align:middle line:84% almost like a rotation, which is a orthogonal transformation. 00:16:21.730 --> 00:16:24.510 align:middle line:84% And if your updates are orthogonalized, 00:16:24.510 --> 00:16:28.638 align:middle line:84% then maybe that somehow relates to this first layer finding 00:16:28.638 --> 00:16:29.930 align:middle line:90% this orthogonal transformation. 00:16:29.930 --> 00:16:33.220 align:middle line:84% It's just a rotation, but it's also scaling. 00:16:33.220 --> 00:16:35.470 align:middle line:84% I think it might be fun to see if you can get insights 00:16:35.470 --> 00:16:38.060 align:middle line:84% from this, but it's a little complicated. 00:16:38.060 --> 00:16:40.680 align:middle line:90% 00:16:40.680 --> 00:16:45.080 align:middle line:84% So we can also try visualizing more advanced nets in this way. 00:16:45.080 --> 00:16:48.080 align:middle line:84% And this is where the power of the ideas of representation 00:16:48.080 --> 00:16:49.120 align:middle line:90% learning really come in. 00:16:49.120 --> 00:16:52.640 align:middle line:84% Not when I'm just having an MLP over two-dimensional embeddings, 00:16:52.640 --> 00:16:58.720 align:middle line:84% but when I am having a very high-dimensional feature space. 00:16:58.720 --> 00:17:00.620 align:middle line:84% So this is the CLIP neural network. 00:17:00.620 --> 00:17:04.119 align:middle line:84% It's a computer vision system that 00:17:04.119 --> 00:17:07.960 align:middle line:84% is quite good at extracting good features from images. 00:17:07.960 --> 00:17:11.079 align:middle line:84% We'll talk about that a little bit later in the course. 00:17:11.079 --> 00:17:14.440 align:middle line:84% And now the embeddings are not two-dimensional, 00:17:14.440 --> 00:17:18.339 align:middle line:84% but I am doing a dimensionality reduction technique, PCA, 00:17:18.339 --> 00:17:19.948 align:middle line:90% to show what they look like. 00:17:19.948 --> 00:17:22.240 align:middle line:84% So it's a bit of a cartoon rather than just showing you 00:17:22.240 --> 00:17:22.859 align:middle line:90% the real data. 00:17:22.859 --> 00:17:24.859 align:middle line:84% But the point is that layer by layer-- and these 00:17:24.859 --> 00:17:27.240 align:middle line:84% are vision transformer layers or a block 00:17:27.240 --> 00:17:30.880 align:middle line:84% of three-vision transformer blocks. 00:17:30.880 --> 00:17:33.440 align:middle line:84% In the transforming, you have attention, and then 00:17:33.440 --> 00:17:36.560 align:middle line:84% LayerNorm, and then softmax, and then this and that. 00:17:36.560 --> 00:17:40.222 align:middle line:84% So this is an entire block of sequence of layers. 00:17:40.222 --> 00:17:42.180 align:middle line:84% But what happens is as you go deeper and deeper 00:17:42.180 --> 00:17:44.842 align:middle line:84% into the network, you end up segregating 00:17:44.842 --> 00:17:45.800 align:middle line:90% these different colors. 00:17:45.800 --> 00:17:49.100 align:middle line:84% And the colors are corresponding to the class of images 00:17:49.100 --> 00:17:50.600 align:middle line:90% in a computer vision data set. 00:17:50.600 --> 00:17:51.360 align:middle line:90% So is it a cat? 00:17:51.360 --> 00:17:52.260 align:middle line:90% Is it a dog? 00:17:52.260 --> 00:17:53.600 align:middle line:84% And you see that at the top of the network, 00:17:53.600 --> 00:17:55.000 align:middle line:84% they're all very separated from each other. 00:17:55.000 --> 00:17:57.880 align:middle line:84% And that's because this network was trained for classification. 00:17:57.880 --> 00:18:01.020 align:middle line:84% So it's trained to find a way to disentangle 00:18:01.020 --> 00:18:03.300 align:middle line:84% all of these different classes and have 00:18:03.300 --> 00:18:05.620 align:middle line:84% their representation in some high-dimensional space 00:18:05.620 --> 00:18:07.960 align:middle line:84% get all separated so that then, at the very end, 00:18:07.960 --> 00:18:10.420 align:middle line:84% I can just take a softmax of that. 00:18:10.420 --> 00:18:14.700 align:middle line:84% And they'll be cleanly separated to the vertices 00:18:14.700 --> 00:18:17.140 align:middle line:84% of the simplex, which is the one-hot labels 00:18:17.140 --> 00:18:18.780 align:middle line:90% for these classes. 00:18:18.780 --> 00:18:22.552 align:middle line:84% So it's a little bit hard to see because, again, it's doing PCA, 00:18:22.552 --> 00:18:24.260 align:middle line:84% so it's not actually showing the simplex. 00:18:24.260 --> 00:18:27.380 align:middle line:84% But it's something kind of like that. 00:18:27.380 --> 00:18:30.260 align:middle line:84% So the whole point is just to build up 00:18:30.260 --> 00:18:33.910 align:middle line:84% the intuition of how we go from entangled, complicated data 00:18:33.910 --> 00:18:36.570 align:middle line:84% at the input to a neural network to, layer by layer, 00:18:36.570 --> 00:18:38.110 align:middle line:84% gradually morphing, disentangling 00:18:38.110 --> 00:18:40.030 align:middle line:84% it through these geometric transformations, 00:18:40.030 --> 00:18:44.030 align:middle line:84% until you get clean separation of your semantics of interest 00:18:44.030 --> 00:18:46.030 align:middle line:84% that can be read off into a prediction, 00:18:46.030 --> 00:18:47.643 align:middle line:90% for example, classification. 00:18:47.643 --> 00:18:49.310 align:middle line:84% And the output space is a simpler object 00:18:49.310 --> 00:18:50.250 align:middle line:90% than the input space. 00:18:50.250 --> 00:18:53.630 align:middle line:84% That's the point I'm trying to make. 00:18:53.630 --> 00:18:55.550 align:middle line:90% OK. 00:18:55.550 --> 00:18:56.050 align:middle line:90% Cool. 00:18:56.050 --> 00:18:58.825 align:middle line:84% So any questions about the general idea of representation 00:18:58.825 --> 00:19:00.950 align:middle line:84% learning, and what these neural networks are doing, 00:19:00.950 --> 00:19:02.730 align:middle line:84% and what these visualizations represent? 00:19:02.730 --> 00:19:05.330 align:middle line:90% 00:19:05.330 --> 00:19:05.830 align:middle line:90% OK. 00:19:05.830 --> 00:19:06.710 align:middle line:90% Cool. 00:19:06.710 --> 00:19:12.570 align:middle line:84% So one of the initial inspirations for deep learning, 00:19:12.570 --> 00:19:15.350 align:middle line:84% of course, was to understand the human brain. 00:19:15.350 --> 00:19:21.270 align:middle line:84% And in neuroscience, people have studied how representations 00:19:21.270 --> 00:19:23.630 align:middle line:90% are learned in the brain. 00:19:23.630 --> 00:19:27.350 align:middle line:84% So I wanted to make this connection. 00:19:27.350 --> 00:19:30.010 align:middle line:84% So this is a map of the visual cortex, 00:19:30.010 --> 00:19:33.530 align:middle line:84% a cartoon of the visual cortex of the human brain. 00:19:33.530 --> 00:19:35.470 align:middle line:84% And it goes through a sequence of layers. 00:19:35.470 --> 00:19:37.970 align:middle line:84% The layers are just like layers in our neural networks. 00:19:37.970 --> 00:19:41.310 align:middle line:84% And they are linear followed by pointwise nonlinear operations, 00:19:41.310 --> 00:19:44.010 align:middle line:84% plus some extra complexities that biologists debate over 00:19:44.010 --> 00:19:46.490 align:middle line:84% whether or not these extra complexities are important 00:19:46.490 --> 00:19:46.990 align:middle line:90% or not. 00:19:46.990 --> 00:19:51.050 align:middle line:84% But for us, it's just linear ReLU, linear ReLU. 00:19:51.050 --> 00:19:54.050 align:middle line:84% And in visual cortex, there's about six of these layers 00:19:54.050 --> 00:19:55.422 align:middle line:90% or five or six of these layers. 00:19:55.422 --> 00:19:57.130 align:middle line:84% And then there's a final layer called IT, 00:19:57.130 --> 00:20:00.350 align:middle line:84% where that's thought to be where the semantics are segregated, 00:20:00.350 --> 00:20:03.250 align:middle line:84% where object recognition has occurred. 00:20:03.250 --> 00:20:06.570 align:middle line:84% And so we have first layers extract, 00:20:06.570 --> 00:20:07.902 align:middle line:90% do filtering to find edges. 00:20:07.902 --> 00:20:09.610 align:middle line:84% And the next layers look for conjunctions 00:20:09.610 --> 00:20:13.170 align:middle line:84% of edges, and disjunctions, and all kinds of combinations 00:20:13.170 --> 00:20:15.050 align:middle line:84% until you get pattern recognizers. 00:20:15.050 --> 00:20:17.570 align:middle line:84% And if you look at the neurons, and you poke into the brain, 00:20:17.570 --> 00:20:18.930 align:middle line:84% and you measure what they're responsive to, 00:20:18.930 --> 00:20:21.230 align:middle line:84% you'll see that as you go deeper into the brain, 00:20:21.230 --> 00:20:23.450 align:middle line:84% they become responsive selectively for more 00:20:23.450 --> 00:20:25.650 align:middle line:90% and more complex patterns. 00:20:25.650 --> 00:20:28.290 align:middle line:84% So let's see, do deep nets artificial deep nets do 00:20:28.290 --> 00:20:30.850 align:middle line:90% the same thing? 00:20:30.850 --> 00:20:33.060 align:middle line:90% What do they learn internally? 00:20:33.060 --> 00:20:34.820 align:middle line:84% OK, so let's say I train a deep net 00:20:34.820 --> 00:20:36.820 align:middle line:90% to do classification of images. 00:20:36.820 --> 00:20:38.652 align:middle line:84% Now I'm going to try to probe and interpret 00:20:38.652 --> 00:20:40.860 align:middle line:84% the representation that's learned in order to perform 00:20:40.860 --> 00:20:42.520 align:middle line:90% that classification task. 00:20:42.520 --> 00:20:44.980 align:middle line:84% And we'll do it just like the neuroscientists do. 00:20:44.980 --> 00:20:49.253 align:middle line:84% So we will stick artificial probes into the neurons, 00:20:49.253 --> 00:20:51.420 align:middle line:84% and measure their value as a function of the inputs, 00:20:51.420 --> 00:20:55.300 align:middle line:84% and see what inputs these neurons are responsive to. 00:20:55.300 --> 00:20:58.500 align:middle line:84% So in the brain, we would stick an electrode in 00:20:58.500 --> 00:20:59.880 align:middle line:90% and see when the neuron fires. 00:20:59.880 --> 00:21:01.560 align:middle line:90% That's when the ReLU is above 0. 00:21:01.560 --> 00:21:03.520 align:middle line:84% But it might not be exactly ReLU in the brain. 00:21:03.520 --> 00:21:06.260 align:middle line:84% It might be some other threshold function. 00:21:06.260 --> 00:21:08.180 align:middle line:84% And we're going to do the same thing. 00:21:08.180 --> 00:21:10.060 align:middle line:84% So for which inputs does the neuron turn on? 00:21:10.060 --> 00:21:13.660 align:middle line:84% And for which inputs does it turn off? 00:21:13.660 --> 00:21:17.585 align:middle line:84% So people have been doing this for a while. 00:21:17.585 --> 00:21:19.460 align:middle line:84% One of the modern names for this type of work 00:21:19.460 --> 00:21:22.180 align:middle line:84% is interpretability or mechanistic interpretability 00:21:22.180 --> 00:21:23.140 align:middle line:90% sometimes. 00:21:23.140 --> 00:21:24.540 align:middle line:84% So it's a really interesting area 00:21:24.540 --> 00:21:28.200 align:middle line:84% if you've come across those terms, but it's an old problem. 00:21:28.200 --> 00:21:32.840 align:middle line:84% And I'll show you some results from this Zeiler and Fergus 2014 00:21:32.840 --> 00:21:36.320 align:middle line:84% paper on trying to interpret the units within a CNN. 00:21:36.320 --> 00:21:37.360 align:middle line:90% OK. 00:21:37.360 --> 00:21:43.260 align:middle line:84% So the first layer in the CNN that they were working with, 00:21:43.260 --> 00:21:47.000 align:middle line:84% this was just a vanilla convolutional network 00:21:47.000 --> 00:21:50.040 align:middle line:90% with five or six layers. 00:21:50.040 --> 00:21:53.200 align:middle line:84% Well, the first layer has a set of filters. 00:21:53.200 --> 00:21:56.420 align:middle line:84% So the filters will all output a feature map. 00:21:56.420 --> 00:21:59.360 align:middle line:84% So each filter will output some spatially varying 00:21:59.360 --> 00:22:00.680 align:middle line:90% set of activations. 00:22:00.680 --> 00:22:05.960 align:middle line:84% And we can look at what input patches in the image 00:22:05.960 --> 00:22:11.380 align:middle line:84% are most strongly activating one of the filters to fire. 00:22:11.380 --> 00:22:14.000 align:middle line:84% So what is the filter looking for? 00:22:14.000 --> 00:22:20.720 align:middle line:84% So here are nine filters in this montage of nine subgrids. 00:22:20.720 --> 00:22:23.440 align:middle line:84% And on each of these sub areas, we 00:22:23.440 --> 00:22:28.490 align:middle line:84% have the top nine input patches that most activated 00:22:28.490 --> 00:22:29.510 align:middle line:90% that particular filter. 00:22:29.510 --> 00:22:33.255 align:middle line:90% So this is channel 1 filter. 00:22:33.255 --> 00:22:34.630 align:middle line:84% And what do you think it's doing? 00:22:34.630 --> 00:22:39.906 align:middle line:84% So what is this a filter for, this top left one? 00:22:39.906 --> 00:22:41.330 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:22:41.330 --> 00:22:42.510 align:middle line:84% PHILLIP ISOLA: Yeah, it's like an edge detector. 00:22:42.510 --> 00:22:44.050 align:middle line:84% It's looking for an oriented line, 00:22:44.050 --> 00:22:47.302 align:middle line:84% a bright followed by a dark transition, so, right, filter. 00:22:47.302 --> 00:22:48.510 align:middle line:90% What is the idea of a filter? 00:22:48.510 --> 00:22:52.250 align:middle line:84% A filter is trying to filter out something from the data, 00:22:52.250 --> 00:22:55.070 align:middle line:84% find the thing it's looking for and throw away everything else. 00:22:55.070 --> 00:22:58.547 align:middle line:84% That's what the word filter means, the English word filter. 00:22:58.547 --> 00:22:59.630 align:middle line:90% So that's what it's doing. 00:22:59.630 --> 00:23:01.330 align:middle line:84% It's just it's responding to some things 00:23:01.330 --> 00:23:02.330 align:middle line:90% and not to other things. 00:23:02.330 --> 00:23:04.530 align:middle line:84% And what it responds to is edges. 00:23:04.530 --> 00:23:05.170 align:middle line:90% OK. 00:23:05.170 --> 00:23:07.650 align:middle line:84% And then we have other-- this is a different orientation 00:23:07.650 --> 00:23:08.230 align:middle line:90% of the edge. 00:23:08.230 --> 00:23:10.430 align:middle line:84% And this is like a greenness detector. 00:23:10.430 --> 00:23:12.850 align:middle line:84% It fires selectively when it sees green. 00:23:12.850 --> 00:23:13.350 align:middle line:90% OK. 00:23:13.350 --> 00:23:14.808 align:middle line:84% So it looks like at the first layer 00:23:14.808 --> 00:23:17.170 align:middle line:84% of processing in a deep neural net, 00:23:17.170 --> 00:23:21.870 align:middle line:84% you get these simple feature detectors 00:23:21.870 --> 00:23:24.630 align:middle line:84% So the next layer, what is that going to look like? 00:23:24.630 --> 00:23:26.670 align:middle line:84% So it turns out not so different than what 00:23:26.670 --> 00:23:28.420 align:middle line:84% the neuroscientists came up with when they 00:23:28.420 --> 00:23:30.100 align:middle line:90% were studying animal brains. 00:23:30.100 --> 00:23:33.020 align:middle line:84% So the second layer looks like it's 00:23:33.020 --> 00:23:35.340 align:middle line:84% having neurons filters that are selective 00:23:35.340 --> 00:23:37.580 align:middle line:84% not for edges, but for some conjunction of edges, 00:23:37.580 --> 00:23:38.960 align:middle line:90% higher-order patterns. 00:23:38.960 --> 00:23:42.060 align:middle line:84% So if I have two channels that detect 00:23:42.060 --> 00:23:43.940 align:middle line:84% two different orientations of edges, 00:23:43.940 --> 00:23:45.860 align:middle line:84% the next layer of my network could 00:23:45.860 --> 00:23:49.300 align:middle line:84% look for a conjunction of those two orientations, like a cross. 00:23:49.300 --> 00:23:53.300 align:middle line:84% And that turns out to map to maybe even more complicated 00:23:53.300 --> 00:23:56.460 align:middle line:84% things, like gradients, or circles, and so forth. 00:23:56.460 --> 00:23:58.180 align:middle line:84% So then as we go layer by layer, we're 00:23:58.180 --> 00:24:00.780 align:middle line:84% going to get more and more complex pattern detectors. 00:24:00.780 --> 00:24:03.860 align:middle line:84% So here are the image patches that most strongly 00:24:03.860 --> 00:24:07.820 align:middle line:84% activate nine different filters on layer 3 of this network. 00:24:07.820 --> 00:24:10.240 align:middle line:84% And already you can start to see, just layer 3, 00:24:10.240 --> 00:24:13.740 align:middle line:84% we get selectivity for faces right. 00:24:13.740 --> 00:24:17.800 align:middle line:84% And it's not because it's really recognizing humans necessarily, 00:24:17.800 --> 00:24:20.380 align:middle line:84% but maybe it's because it's recognizing 00:24:20.380 --> 00:24:23.100 align:middle line:84% little circle of darkness around a bright background 00:24:23.100 --> 00:24:26.580 align:middle line:84% with a broader torso thing underneath it, 00:24:26.580 --> 00:24:28.600 align:middle line:90% so some blobbiness detector. 00:24:28.600 --> 00:24:32.200 align:middle line:84% But it happens to be an OK template for a human. 00:24:32.200 --> 00:24:35.000 align:middle line:84% And then as we go deeper, we get much more selectivity. 00:24:35.000 --> 00:24:38.280 align:middle line:84% And eventually, we start actually detecting, 00:24:38.280 --> 00:24:41.100 align:middle line:84% selectively segregating dog faces from human faces, 00:24:41.100 --> 00:24:42.160 align:middle line:90% for example. 00:24:42.160 --> 00:24:43.677 align:middle line:90% And again, this is 2014. 00:24:43.677 --> 00:24:45.760 align:middle line:84% This is the fifth layer of a convolutional network 00:24:45.760 --> 00:24:47.120 align:middle line:90% from that era. 00:24:47.120 --> 00:24:49.320 align:middle line:84% If you run the same types of methods 00:24:49.320 --> 00:24:53.160 align:middle line:84% on modern networks, which are dozens or hundreds of layers, 00:24:53.160 --> 00:24:57.088 align:middle line:84% you will get very precise detectors 00:24:57.088 --> 00:24:59.380 align:middle line:84% deep in the network that code for very specific things. 00:24:59.380 --> 00:25:01.640 align:middle line:84% And that's this area of mechanistic interpretability, 00:25:01.640 --> 00:25:03.440 align:middle line:84% where people are trying to understand, 00:25:03.440 --> 00:25:06.400 align:middle line:84% what are the emergent detectors in these systems? 00:25:06.400 --> 00:25:09.400 align:middle line:84% And you can do it for images, or language models, or other things 00:25:09.400 --> 00:25:09.940 align:middle line:90% as well. 00:25:09.940 --> 00:25:12.800 align:middle line:84% We can say, is there a neuron within a language model 00:25:12.800 --> 00:25:15.350 align:middle line:90% that codes for the sentiment? 00:25:15.350 --> 00:25:17.600 align:middle line:84% There was a famous paper called "The Sentiment Neuron" 00:25:17.600 --> 00:25:19.475 align:middle line:84% that found a neuron that codes for sentiment. 00:25:19.475 --> 00:25:21.680 align:middle line:84% But now this is well known, that this just occurs 00:25:21.680 --> 00:25:24.410 align:middle line:90% in all of our modern deep nets. 00:25:24.410 --> 00:25:26.570 align:middle line:90% OK. 00:25:26.570 --> 00:25:32.290 align:middle line:84% So the rough story is that deep nets and the neuroscience models 00:25:32.290 --> 00:25:35.810 align:middle line:84% of the brain are quite in alignment here. 00:25:35.810 --> 00:25:39.210 align:middle line:84% The first layer of the human visual system or monkey 00:25:39.210 --> 00:25:41.370 align:middle line:84% visual system, to be more precise, 00:25:41.370 --> 00:25:43.210 align:middle line:90% will have these edge detectors. 00:25:43.210 --> 00:25:46.190 align:middle line:84% And the next layer will have shape and pattern detectors. 00:25:46.190 --> 00:25:48.650 align:middle line:84% And then, finally, at IT at the sixth layer or so, 00:25:48.650 --> 00:25:53.373 align:middle line:90% we'll have dog face recognition. 00:25:53.373 --> 00:25:55.290 align:middle line:84% One difference-- so there are some differences 00:25:55.290 --> 00:25:57.530 align:middle line:90% between brains and machines. 00:25:57.530 --> 00:25:59.930 align:middle line:84% One interesting difference is that the brain seems 00:25:59.930 --> 00:26:02.423 align:middle line:90% to only have about seven-- 00:26:02.423 --> 00:26:04.090 align:middle line:84% it depends how you count, but maybe like 00:26:04.090 --> 00:26:06.930 align:middle line:84% seven layers of convolutional filters. 00:26:06.930 --> 00:26:08.770 align:middle line:84% And our modern architectures have 00:26:08.770 --> 00:26:14.250 align:middle line:84% hundreds of layers of filters and other operations. 00:26:14.250 --> 00:26:17.210 align:middle line:84% But the brain has recurrence, so it uses those seven layers over 00:26:17.210 --> 00:26:18.030 align:middle line:90% and over again. 00:26:18.030 --> 00:26:21.390 align:middle line:90% 00:26:21.390 --> 00:26:25.070 align:middle line:84% So let's now get a little bit more formal 00:26:25.070 --> 00:26:26.910 align:middle line:84% and define what a representation is 00:26:26.910 --> 00:26:29.910 align:middle line:90% for the context of this class. 00:26:29.910 --> 00:26:31.950 align:middle line:84% So we're going to, basically, restrict ourselves 00:26:31.950 --> 00:26:35.870 align:middle line:84% to representation as a vector embedding, is a vector. 00:26:35.870 --> 00:26:38.013 align:middle line:84% Or it could be a more general tensor object. 00:26:38.013 --> 00:26:39.430 align:middle line:84% It could have multiple dimensions, 00:26:39.430 --> 00:26:43.210 align:middle line:84% but most canonically, it will be a vector. 00:26:43.210 --> 00:26:45.190 align:middle line:84% And there's two different ways we're going 00:26:45.190 --> 00:26:47.852 align:middle line:90% to use the word representation. 00:26:47.852 --> 00:26:50.310 align:middle line:84% So the first way is we'll say a representation of some data 00:26:50.310 --> 00:26:54.110 align:middle line:84% domain x or maybe a data distribution p of x, probability 00:26:54.110 --> 00:26:57.870 align:middle line:84% of x, is a function that maps from data in that domain 00:26:57.870 --> 00:27:01.750 align:middle line:84% to embeddings in Rd, vector embeddings. 00:27:01.750 --> 00:27:04.532 align:middle line:84% That function f is called the encoder. 00:27:04.532 --> 00:27:05.990 align:middle line:84% And it will assign a feature vector 00:27:05.990 --> 00:27:08.110 align:middle line:84% to every element in the input domain. 00:27:08.110 --> 00:27:10.310 align:middle line:84% And we'll say that function f, that's 00:27:10.310 --> 00:27:13.090 align:middle line:84% the representation of that data domain. 00:27:13.090 --> 00:27:16.050 align:middle line:84% So what is actually parameterizing f? 00:27:16.050 --> 00:27:17.467 align:middle line:90% It's the weights and biases. 00:27:17.467 --> 00:27:19.550 align:middle line:84% So if I have a neural network, the representation, 00:27:19.550 --> 00:27:22.880 align:middle line:84% by this definition, is its weights and biases. 00:27:22.880 --> 00:27:26.057 align:middle line:84% That's a learned representation of the data distribution. 00:27:26.057 --> 00:27:27.640 align:middle line:84% Another use of the word representation 00:27:27.640 --> 00:27:29.680 align:middle line:84% will be the representation not of the domain 00:27:29.680 --> 00:27:31.900 align:middle line:84% or the distribution, but of a single point. 00:27:31.900 --> 00:27:34.480 align:middle line:84% And that will just be a vector z in Rd, 00:27:34.480 --> 00:27:37.460 align:middle line:84% where z is equal to the encoder applied to f. 00:27:37.460 --> 00:27:40.428 align:middle line:84% So people use both of these definitions of representation. 00:27:40.428 --> 00:27:42.720 align:middle line:84% If you want to be clear, you can say the representation 00:27:42.720 --> 00:27:46.090 align:middle line:84% of a data point versus the representation of a data domain. 00:27:46.090 --> 00:27:47.840 align:middle line:84% But I wanted to point out that distinction 00:27:47.840 --> 00:27:50.568 align:middle line:84% because we'll talk about the learned representation. 00:27:50.568 --> 00:27:52.360 align:middle line:84% And when we say the learned representation, 00:27:52.360 --> 00:27:54.740 align:middle line:90% we usually mean f, OK? 00:27:54.740 --> 00:27:56.240 align:middle line:84% So that's the learned representation 00:27:56.240 --> 00:27:59.680 align:middle line:84% is the weights and biases that fit the data. 00:27:59.680 --> 00:28:03.240 align:middle line:90% 00:28:03.240 --> 00:28:04.680 align:middle line:84% So that's the formalism we'll work 00:28:04.680 --> 00:28:06.420 align:middle line:90% with for the next few lectures. 00:28:06.420 --> 00:28:10.800 align:middle line:90% So why learn representations? 00:28:10.800 --> 00:28:14.680 align:middle line:84% So there's a lot of answers you could come up with. 00:28:14.680 --> 00:28:18.940 align:middle line:84% But maybe the simplest is to do more learning, right? 00:28:18.940 --> 00:28:20.980 align:middle line:90% So we learn to learn. 00:28:20.980 --> 00:28:22.140 align:middle line:90% It's interesting. 00:28:22.140 --> 00:28:26.100 align:middle line:84% It seems a little bit circular, but this is usually 00:28:26.100 --> 00:28:29.240 align:middle line:84% the context in which we're going to use representations. 00:28:29.240 --> 00:28:32.372 align:middle line:84% We're going to use them to accelerate future learning. 00:28:32.372 --> 00:28:34.580 align:middle line:84% We'll have a few lectures on transfer learning, where 00:28:34.580 --> 00:28:35.560 align:middle line:90% we get into more details. 00:28:35.560 --> 00:28:37.393 align:middle line:84% But I have to introduce a little bit of this 00:28:37.393 --> 00:28:41.260 align:middle line:84% now because the main use of trained representations 00:28:41.260 --> 00:28:43.900 align:middle line:84% is to accelerate future learning. 00:28:43.900 --> 00:28:47.020 align:middle line:84% So quote in Goodfellow, "Generally speaking, a good"-- 00:28:47.020 --> 00:28:49.780 align:middle line:84% or the Deep Learning book-- "a good representation is one that 00:28:49.780 --> 00:28:52.900 align:middle line:84% makes subsequent learning easier." 00:28:52.900 --> 00:28:59.612 align:middle line:84% And I think this was a strange misconception in the early days 00:28:59.612 --> 00:29:00.320 align:middle line:90% of deep learning. 00:29:00.320 --> 00:29:05.140 align:middle line:84% It was that we will make deep networks that 00:29:05.140 --> 00:29:06.440 align:middle line:90% act as blank slates. 00:29:06.440 --> 00:29:11.380 align:middle line:84% They come at new problems with no pretrained prior information. 00:29:11.380 --> 00:29:15.820 align:middle line:84% And people would complain about deep learning as being very data 00:29:15.820 --> 00:29:16.540 align:middle line:90% hungry. 00:29:16.540 --> 00:29:20.310 align:middle line:84% It would take massive data to apply deep net to your problem. 00:29:20.310 --> 00:29:23.970 align:middle line:84% And they would say, well, humans are much more sample efficient. 00:29:23.970 --> 00:29:28.030 align:middle line:84% We only take a little bit of data to solve a new problem 00:29:28.030 --> 00:29:31.510 align:middle line:90% But it wasn't apples to apples. 00:29:31.510 --> 00:29:33.470 align:middle line:90% So humans are born. 00:29:33.470 --> 00:29:37.470 align:middle line:84% And you in this class are coming to learn new information not as 00:29:37.470 --> 00:29:38.090 align:middle line:90% a blank slate. 00:29:38.090 --> 00:29:41.070 align:middle line:84% You've gone through billions of years of evolution, 00:29:41.070 --> 00:29:43.430 align:middle line:84% tens of billions of years of the physical evolution 00:29:43.430 --> 00:29:45.270 align:middle line:84% of the universe before that, before even 00:29:45.270 --> 00:29:46.790 align:middle line:90% biological evolution. 00:29:46.790 --> 00:29:50.950 align:middle line:84% And you're also having 20 plus years of pre-training 00:29:50.950 --> 00:29:51.730 align:middle line:90% in your lifetime. 00:29:51.730 --> 00:29:53.670 align:middle line:84% You've learned very good internal skills, 00:29:53.670 --> 00:29:55.670 align:middle line:84% and knowledge, and representations, so you're 00:29:55.670 --> 00:29:56.870 align:middle line:90% not blank slates. 00:29:56.870 --> 00:29:58.990 align:middle line:84% And deep nets also should generally not 00:29:58.990 --> 00:30:00.920 align:middle line:90% be used as blank slates. 00:30:00.920 --> 00:30:03.170 align:middle line:84% This is something that's changed over the last decade. 00:30:03.170 --> 00:30:06.190 align:middle line:84% It used to be we thought deep nets require a lot of data 00:30:06.190 --> 00:30:07.690 align:middle line:84% because we'll start from scratch. 00:30:07.690 --> 00:30:10.190 align:middle line:84% We'll initialize them randomly, and they'll be blank slates. 00:30:10.190 --> 00:30:12.310 align:middle line:84% But these days, this is not really what we do. 00:30:12.310 --> 00:30:14.590 align:middle line:84% We start with pretrained representations. 00:30:14.590 --> 00:30:20.840 align:middle line:90% So let's look at that idea. 00:30:20.840 --> 00:30:24.170 align:middle line:84% So very often, we will have-- let's imagine 00:30:24.170 --> 00:30:25.920 align:middle line:84% that you're working for some music company 00:30:25.920 --> 00:30:27.680 align:middle line:90% at Spotify or something. 00:30:27.680 --> 00:30:29.960 align:middle line:84% And you will train your neural network 00:30:29.960 --> 00:30:33.240 align:middle line:84% to classify the genre of the music. 00:30:33.240 --> 00:30:38.080 align:middle line:84% So in two parts of your network, you'll have some encoder f, 00:30:38.080 --> 00:30:42.040 align:middle line:84% and then you might have some extra readout operation 00:30:42.040 --> 00:30:44.760 align:middle line:84% at the end to do your classification. 00:30:44.760 --> 00:30:47.000 align:middle line:84% We'll think of w as just being like a linear layer 00:30:47.000 --> 00:30:48.800 align:middle line:90% on top of a representation z. 00:30:48.800 --> 00:30:51.520 align:middle line:84% But we can actually select any layer in the network 00:30:51.520 --> 00:30:53.800 align:middle line:84% to be called the representation, any prior layers 00:30:53.800 --> 00:30:55.720 align:middle line:84% are the encoder, and any subsequent layers 00:30:55.720 --> 00:30:59.540 align:middle line:90% are like the readout, OK? 00:30:59.540 --> 00:31:01.720 align:middle line:84% But often, what we're going to be tested on 00:31:01.720 --> 00:31:04.540 align:middle line:84% is not going to be the same as what we've trained on. 00:31:04.540 --> 00:31:10.760 align:middle line:84% Maybe we invested $1 billion to train the Spotify sound 00:31:10.760 --> 00:31:12.620 align:middle line:90% recognition system. 00:31:12.620 --> 00:31:16.740 align:middle line:84% We don't want to use $1 billion to solve 00:31:16.740 --> 00:31:18.780 align:middle line:84% a new task because maybe the next year, 00:31:18.780 --> 00:31:20.300 align:middle line:84% we realize, actually, we also need 00:31:20.300 --> 00:31:22.880 align:middle line:84% to predict if users the music or don't like the music, 00:31:22.880 --> 00:31:25.380 align:middle line:84% so we can give them recommendations. 00:31:25.380 --> 00:31:30.660 align:middle line:84% So the trick is that we will just use our same encoder up 00:31:30.660 --> 00:31:34.420 align:middle line:84% to some layer in the network, and we will only 00:31:34.420 --> 00:31:36.763 align:middle line:90% retrain the final readout. 00:31:36.763 --> 00:31:39.180 align:middle line:84% And that might, in this case, be a linear operation, which 00:31:39.180 --> 00:31:41.580 align:middle line:84% would be called a linear probe, or it could 00:31:41.580 --> 00:31:44.420 align:middle line:90% be MLP, or something else. 00:31:44.420 --> 00:31:48.760 align:middle line:84% So we're not going to retrain from scratch. 00:31:48.760 --> 00:31:50.820 align:middle line:84% We're going to initialize our network 00:31:50.820 --> 00:31:52.397 align:middle line:84% with the pre-trained representation. 00:31:52.397 --> 00:31:53.980 align:middle line:84% And in this case, we might just freeze 00:31:53.980 --> 00:31:56.188 align:middle line:84% that pre-trained representation, no longer update it, 00:31:56.188 --> 00:32:00.420 align:middle line:84% and just learn a linear mapping onto our new prediction problem. 00:32:00.420 --> 00:32:03.440 align:middle line:90% So these are the two phases. 00:32:03.440 --> 00:32:06.340 align:middle line:84% Now they have new names for these, pre-training 00:32:06.340 --> 00:32:07.620 align:middle line:90% and post-training. 00:32:07.620 --> 00:32:08.890 align:middle line:90% But we used to call it-- 00:32:08.890 --> 00:32:10.640 align:middle line:84% there's a lot of different names for this, 00:32:10.640 --> 00:32:13.260 align:middle line:84% but you pre-train and then you post-train. 00:32:13.260 --> 00:32:17.270 align:middle line:84% So another thing you can do is not just 00:32:17.270 --> 00:32:19.590 align:middle line:90% train this read-off head. 00:32:19.590 --> 00:32:21.790 align:middle line:84% You can actually update the parameters 00:32:21.790 --> 00:32:26.330 align:middle line:84% of your encoder, as well, but initialize from the prior f. 00:32:26.330 --> 00:32:29.390 align:middle line:84% So you just continue to do backprop 00:32:29.390 --> 00:32:33.470 align:middle line:84% to update to find some fine-tuned perturbation 00:32:33.470 --> 00:32:36.390 align:middle line:84% of your parameters that will solve the new problem. 00:32:36.390 --> 00:32:37.730 align:middle line:90% So this is called fine-tuning. 00:32:37.730 --> 00:32:41.150 align:middle line:84% I'll talk about that on the next slide as well. 00:32:41.150 --> 00:32:44.810 align:middle line:84% So we initialize F prime not with random weights, 00:32:44.810 --> 00:32:46.810 align:middle line:84% but we initialize it with the same weights as f, 00:32:46.810 --> 00:32:47.893 align:middle line:90% and then we keep training. 00:32:47.893 --> 00:32:51.150 align:middle line:84% That's the whole idea of fine-tuning. 00:32:51.150 --> 00:32:53.490 align:middle line:84% And here's the transfer learning paradigm. 00:32:53.490 --> 00:32:57.135 align:middle line:84% This is what is typically done for real-world problems. 00:32:57.135 --> 00:32:59.010 align:middle line:84% For toy problems, you can train from scratch. 00:32:59.010 --> 00:33:01.230 align:middle line:84% But for real problems, you generally can't. 00:33:01.230 --> 00:33:03.590 align:middle line:90% So you have three phases. 00:33:03.590 --> 00:33:06.170 align:middle line:84% You have now not just train and test. 00:33:06.170 --> 00:33:09.710 align:middle line:84% You have pre-train, which can be thought of as learning 00:33:09.710 --> 00:33:10.490 align:middle line:90% a representation. 00:33:10.490 --> 00:33:13.070 align:middle line:90% You do that on a lot of data. 00:33:13.070 --> 00:33:15.070 align:middle line:84% And typically, you can do that on a lot of data. 00:33:15.070 --> 00:33:16.820 align:middle line:84% You can afford to do that on a lot of data 00:33:16.820 --> 00:33:19.623 align:middle line:84% because the pre-training phase, the representation learning 00:33:19.623 --> 00:33:21.290 align:middle line:84% phase, doesn't require as much knowledge 00:33:21.290 --> 00:33:22.790 align:middle line:84% about what the final task is going to be. 00:33:22.790 --> 00:33:25.050 align:middle line:84% We'll see that autoencoders and contrastive learning 00:33:25.050 --> 00:33:27.650 align:middle line:84% are these pre-training methods that don't require knowing what 00:33:27.650 --> 00:33:29.370 align:middle line:90% the final task is going to be. 00:33:29.370 --> 00:33:33.090 align:middle line:84% But then you adapt to your actual task of interest. 00:33:33.090 --> 00:33:35.360 align:middle line:84% And you can do that on just a little data 00:33:35.360 --> 00:33:37.610 align:middle line:84% because you already have learned a good representation 00:33:37.610 --> 00:33:40.490 align:middle line:84% of the domain f, of the data domain x 00:33:40.490 --> 00:33:41.490 align:middle line:90% through this function f. 00:33:41.490 --> 00:33:44.210 align:middle line:90% F is the representation. 00:33:44.210 --> 00:33:45.390 align:middle line:90% And then you test right. 00:33:45.390 --> 00:33:46.470 align:middle line:90% That's just regular. 00:33:46.470 --> 00:33:50.210 align:middle line:84% You deploy your system out in the real world. 00:33:50.210 --> 00:33:54.130 align:middle line:84% So I think a lot of you have probably done this because this 00:33:54.130 --> 00:33:55.270 align:middle line:90% is just a practical thing. 00:33:55.270 --> 00:33:57.348 align:middle line:84% If you're ever going to play with deep nets, 00:33:57.348 --> 00:33:58.890 align:middle line:84% you'll download a pre-trained system, 00:33:58.890 --> 00:34:00.230 align:middle line:90% and you'll fine-tune it, right? 00:34:00.230 --> 00:34:02.410 align:middle line:84% So a lot of you I'm sure, have done this, 00:34:02.410 --> 00:34:05.470 align:middle line:84% but it's worth understanding why and how it works. 00:34:05.470 --> 00:34:07.330 align:middle line:84% So fine-tuning-- extremely simple recipe. 00:34:07.330 --> 00:34:10.370 align:middle line:84% Pre-train a network on task A resulting in parameters 00:34:10.370 --> 00:34:12.620 align:middle line:90% w and b, weights and biases. 00:34:12.620 --> 00:34:15.219 align:middle line:84% Those weights and biases are your pre-trained representation 00:34:15.219 --> 00:34:16.820 align:middle line:90% of the data domain. 00:34:16.820 --> 00:34:18.659 align:middle line:84% Then you initialize a second network 00:34:18.659 --> 00:34:21.159 align:middle line:84% using some or all of the weights and biases. 00:34:21.159 --> 00:34:23.100 align:middle line:84% You could initialize some weights 00:34:23.100 --> 00:34:26.000 align:middle line:84% from scratch, some from your pre-trained weights. 00:34:26.000 --> 00:34:28.219 align:middle line:84% You can do all kinds of interesting network surgery 00:34:28.219 --> 00:34:29.139 align:middle line:90% on that. 00:34:29.139 --> 00:34:31.420 align:middle line:84% And then you'll train the second network 00:34:31.420 --> 00:34:34.659 align:middle line:84% to resolve some parameters, W prime and beta prime. 00:34:34.659 --> 00:34:36.159 align:middle line:84% And this is often called fine-tuning 00:34:36.159 --> 00:34:40.699 align:middle line:84% because we assume the W prime is just a minor modification of W, 00:34:40.699 --> 00:34:44.820 align:middle line:84% because we only fine-tune it on a little bit of data. 00:34:44.820 --> 00:34:50.540 align:middle line:84% So the point here is a lot of people 00:34:50.540 --> 00:34:52.300 align:middle line:84% think of deep learning as the thing you 00:34:52.300 --> 00:34:54.179 align:middle line:90% do when you have a ton of data. 00:34:54.179 --> 00:34:56.500 align:middle line:84% But I think the real point of deep learning 00:34:56.500 --> 00:34:58.220 align:middle line:84% is it's the thing you do that enables 00:34:58.220 --> 00:35:00.140 align:middle line:90% learning from little data. 00:35:00.140 --> 00:35:01.900 align:middle line:84% So deep nets and deep learning, to me, 00:35:01.900 --> 00:35:04.460 align:middle line:84% is really all about how to learn from little data. 00:35:04.460 --> 00:35:06.340 align:middle line:84% And the way you learn from little data 00:35:06.340 --> 00:35:08.280 align:middle line:84% is by pre-training on massive data. 00:35:08.280 --> 00:35:10.340 align:middle line:84% So, yes, you have to pre-train on massive data, 00:35:10.340 --> 00:35:13.080 align:middle line:84% but once you do, you get a very sample efficient method, 00:35:13.080 --> 00:35:15.560 align:middle line:84% which is just update the representation a little bit 00:35:15.560 --> 00:35:17.000 align:middle line:90% to solve the new task. 00:35:17.000 --> 00:35:21.840 align:middle line:84% And this is the best known way right now for a lot of problems 00:35:21.840 --> 00:35:23.765 align:middle line:90% to do learning from little data. 00:35:23.765 --> 00:35:25.140 align:middle line:84% To do, learning from little data, 00:35:25.140 --> 00:35:27.882 align:middle line:84% you have to pre-train on massive data. 00:35:27.882 --> 00:35:29.840 align:middle line:84% That's a little trick that happened that people 00:35:29.840 --> 00:35:31.262 align:middle line:90% didn't quite expect. 00:35:31.262 --> 00:35:32.720 align:middle line:84% I think for a while, people thought 00:35:32.720 --> 00:35:35.348 align:middle line:84% there might be some algorithm that's just better at learning. 00:35:35.348 --> 00:35:37.140 align:middle line:84% You don't have to pre-train that algorithm, 00:35:37.140 --> 00:35:38.480 align:middle line:90% but this works better. 00:35:38.480 --> 00:35:39.720 align:middle line:90% Yeah, question in the back? 00:35:39.720 --> 00:35:42.560 align:middle line:84% AUDIENCE: When you talk about massive data versus little data, 00:35:42.560 --> 00:35:47.160 align:middle line:84% roughly, what's the scale or magnitude of the amount? 00:35:47.160 --> 00:35:49.440 align:middle line:84% PHILLIP ISOLA: Yeah, so what's the scale for massive, 00:35:49.440 --> 00:35:52.240 align:middle line:90% and what's the scale for little? 00:35:52.240 --> 00:35:58.920 align:middle line:84% So in practice, the ratio is always becoming more extreme, 00:35:58.920 --> 00:35:59.720 align:middle line:90% I would say. 00:35:59.720 --> 00:36:03.020 align:middle line:84% But there's not going to be a precise answer. 00:36:03.020 --> 00:36:05.600 align:middle line:84% But I would say, in practice, each year, 00:36:05.600 --> 00:36:08.740 align:middle line:84% the pre-trained representations get 00:36:08.740 --> 00:36:09.990 align:middle line:90% trained on more and more data. 00:36:09.990 --> 00:36:14.730 align:middle line:84% So that will be a language model trained on 10 trillion tokens 00:36:14.730 --> 00:36:16.890 align:middle line:90% or words on the internet. 00:36:16.890 --> 00:36:19.050 align:middle line:84% And the fine-tuning is typically something 00:36:19.050 --> 00:36:22.570 align:middle line:84% that you can do on CoLab yourself in a few minutes 00:36:22.570 --> 00:36:24.330 align:middle line:90% or hours. 00:36:24.330 --> 00:36:27.910 align:middle line:84% Now it depends what kind of fine-tuning you're doing. 00:36:27.910 --> 00:36:30.490 align:middle line:84% So we'll talk about linear probes and low-rank fine-tuning 00:36:30.490 --> 00:36:33.770 align:middle line:84% methods, but that will come in a future lecture. 00:36:33.770 --> 00:36:36.530 align:middle line:84% But it would be a trillion pre-training tokens 00:36:36.530 --> 00:36:41.030 align:middle line:84% and a million or less fine-tuning tokens, 00:36:41.030 --> 00:36:45.330 align:middle line:84% so the ratio is many orders of magnitude. 00:36:45.330 --> 00:36:47.130 align:middle line:84% There's a question in the way back. 00:36:47.130 --> 00:36:49.990 align:middle line:84% AUDIENCE: Yeah, for a lot of applications, 00:36:49.990 --> 00:36:52.850 align:middle line:84% there are, obviously, pre-trained models 00:36:52.850 --> 00:36:56.190 align:middle line:84% on specific types of data, like language and images. 00:36:56.190 --> 00:36:59.130 align:middle line:84% But let's say you're in some scientific field 00:36:59.130 --> 00:37:06.980 align:middle line:84% and there are not like [INAUDIBLE] models that are not 00:37:06.980 --> 00:37:12.480 align:middle line:84% trained [INAUDIBLE] use of, for example, [INAUDIBLE] 00:37:12.480 --> 00:37:14.580 align:middle line:84% PHILLIP ISOLA: Yeah, very good question. 00:37:14.580 --> 00:37:17.240 align:middle line:84% So if I can rephrase the question, 00:37:17.240 --> 00:37:18.740 align:middle line:84% what if you're in a domain where you 00:37:18.740 --> 00:37:21.240 align:middle line:84% don't have a lot of pre-training data for that domain? 00:37:21.240 --> 00:37:24.220 align:middle line:84% What if you are in a scientific domain 00:37:24.220 --> 00:37:26.580 align:middle line:84% where it's very specialized measurements? 00:37:26.580 --> 00:37:28.780 align:middle line:84% Are you allowed to use a language models 00:37:28.780 --> 00:37:33.140 align:middle line:84% pre-training that's trained on internet talking and text? 00:37:33.140 --> 00:37:36.720 align:middle line:84% And the empirical answer is, yes. 00:37:36.720 --> 00:37:38.520 align:middle line:84% That often actually works decently well. 00:37:38.520 --> 00:37:41.700 align:middle line:84% So you don't have to pre-train on the same type of data or even 00:37:41.700 --> 00:37:46.043 align:middle line:84% the same data distribution as you're going to fine-tune on. 00:37:46.043 --> 00:37:48.460 align:middle line:84% There's often a lot of shared information between one data 00:37:48.460 --> 00:37:51.220 align:middle line:84% modality and one distribution and other ones that 00:37:51.220 --> 00:37:57.460 align:middle line:84% might look very disparate, but your mileage may vary. 00:37:57.460 --> 00:38:00.580 align:middle line:84% If you have a really, really big gap between the type of data you 00:38:00.580 --> 00:38:02.480 align:middle line:84% pre-train on and the type you fine-tune on, 00:38:02.480 --> 00:38:04.340 align:middle line:90% then it might not work as well. 00:38:04.340 --> 00:38:07.000 align:middle line:84% But I would say, in practice, the surprise 00:38:07.000 --> 00:38:09.360 align:middle line:84% has been if you fine-tune a language 00:38:09.360 --> 00:38:12.280 align:middle line:84% model trained on the internet on almost anything, it will help. 00:38:12.280 --> 00:38:14.985 align:middle line:84% That's kind of been the current finding. 00:38:14.985 --> 00:38:16.860 align:middle line:84% It might not be the best way of doing things, 00:38:16.860 --> 00:38:19.000 align:middle line:84% but it will be better than initializing 00:38:19.000 --> 00:38:21.440 align:middle line:90% the network from scratch. 00:38:21.440 --> 00:38:22.600 align:middle line:90% Yeah, question? 00:38:22.600 --> 00:38:24.160 align:middle line:84% AUDIENCE: Yeah, I want to ask, what's 00:38:24.160 --> 00:38:27.580 align:middle line:84% the theory behind this fine-tune framework? 00:38:27.580 --> 00:38:32.480 align:middle line:84% There is no guarantee for similarity between task A 00:38:32.480 --> 00:38:36.520 align:middle line:84% or task B. And what's the guarantee 00:38:36.520 --> 00:38:37.603 align:middle line:90% for this kind of paradigm? 00:38:37.603 --> 00:38:39.020 align:middle line:84% PHILLIP ISOLA: So the question is, 00:38:39.020 --> 00:38:42.460 align:middle line:84% I think, what is the theoretical guarantees about fine-tuning? 00:38:42.460 --> 00:38:45.720 align:middle line:84% So the big question in this area and transfer learning-- 00:38:45.720 --> 00:38:48.200 align:middle line:84% we'll have more on this later-- but the big question is, 00:38:48.200 --> 00:38:51.880 align:middle line:84% when and why does training on task A on date A 00:38:51.880 --> 00:38:54.520 align:middle line:90% help on task B and data B? 00:38:54.520 --> 00:38:57.600 align:middle line:84% What is the property of A and B that has to be alike for this 00:38:57.600 --> 00:38:58.540 align:middle line:90% to actually work? 00:38:58.540 --> 00:39:01.000 align:middle line:84% So empirically, it just generally works pretty well, 00:39:01.000 --> 00:39:02.760 align:middle line:90% more than people maybe thought. 00:39:02.760 --> 00:39:05.930 align:middle line:84% Theoretically, I would say it's very much an open question. 00:39:05.930 --> 00:39:08.730 align:middle line:84% Maybe I'll point you to Sanjeev Arora 00:39:08.730 --> 00:39:11.090 align:middle line:84% has some papers and some talks that phrase it 00:39:11.090 --> 00:39:12.490 align:middle line:90% the way I just gave it. 00:39:12.490 --> 00:39:15.050 align:middle line:84% When does training on task A help on task B? 00:39:15.050 --> 00:39:17.870 align:middle line:84% And he tries to develop some theory out of this, 00:39:17.870 --> 00:39:21.250 align:middle line:84% but I would say it's in its early days. 00:39:21.250 --> 00:39:21.750 align:middle line:90% OK. 00:39:21.750 --> 00:39:22.470 align:middle line:90% Let me move on. 00:39:22.470 --> 00:39:24.970 align:middle line:84% We'll come back to question more questions later. 00:39:24.970 --> 00:39:27.930 align:middle line:84% So that's really just all motivation 00:39:27.930 --> 00:39:30.390 align:middle line:84% setting up the concept, why we actually care about this. 00:39:30.390 --> 00:39:32.090 align:middle line:84% But now let's get into a few algorithms 00:39:32.090 --> 00:39:35.670 align:middle line:84% and see how representation learning actually can work. 00:39:35.670 --> 00:39:39.470 align:middle line:90% 00:39:39.470 --> 00:39:41.770 align:middle line:84% So just, first, what are properties 00:39:41.770 --> 00:39:42.990 align:middle line:90% of a good representation? 00:39:42.990 --> 00:39:47.610 align:middle line:84% The optional reading for today gives some ideas. 00:39:47.610 --> 00:39:49.370 align:middle line:84% So let me give two or three, and then I'll 00:39:49.370 --> 00:39:50.590 align:middle line:90% ask if you have any others. 00:39:50.590 --> 00:39:52.510 align:middle line:84% So what are properties of good representation? 00:39:52.510 --> 00:39:54.170 align:middle line:90% What are we looking for? 00:39:54.170 --> 00:39:56.310 align:middle line:84% So I said that I can pre-train somehow, 00:39:56.310 --> 00:39:58.870 align:middle line:84% but how should I pre-train to get the best representation? 00:39:58.870 --> 00:40:01.550 align:middle line:84% What are the properties I'm looking for? 00:40:01.550 --> 00:40:04.350 align:middle line:84% So one property is that you often want your representation 00:40:04.350 --> 00:40:06.910 align:middle line:90% to be somehow small. 00:40:06.910 --> 00:40:10.082 align:middle line:84% You want it to be something that is compact so that-- 00:40:10.082 --> 00:40:11.290 align:middle line:90% there's two reasons for this. 00:40:11.290 --> 00:40:13.650 align:middle line:84% One is that this will just use less memory. 00:40:13.650 --> 00:40:17.510 align:middle line:84% You can store low-dimensional, compact features on your disk 00:40:17.510 --> 00:40:20.130 align:middle line:84% with less memory than the raw data itself. 00:40:20.130 --> 00:40:21.650 align:middle line:90% That's the most obvious one. 00:40:21.650 --> 00:40:26.700 align:middle line:84% And the other one is an Occam's razor generalization theory type 00:40:26.700 --> 00:40:28.950 align:middle line:84% of analysis, where you would say that the more compact 00:40:28.950 --> 00:40:31.285 align:middle line:84% representations will be somehow simpler and generalize 00:40:31.285 --> 00:40:32.910 align:middle line:84% better, or learn more abstractions that 00:40:32.910 --> 00:40:34.970 align:middle line:90% generalize better. 00:40:34.970 --> 00:40:37.730 align:middle line:84% But you don't just want your representation to be compact. 00:40:37.730 --> 00:40:39.570 align:middle line:84% You also want it to be explanatory. 00:40:39.570 --> 00:40:42.650 align:middle line:84% And one of the technical terms for this is sufficient. 00:40:42.650 --> 00:40:44.990 align:middle line:84% You want minimal and sufficient representations. 00:40:44.990 --> 00:40:47.390 align:middle line:84% Minimal, they're very low-dimensional or 00:40:47.390 --> 00:40:49.510 align:middle line:84% low-information, but sufficient, that they 00:40:49.510 --> 00:40:51.770 align:middle line:84% are sufficient statistics for solving your tasks. 00:40:51.770 --> 00:40:54.430 align:middle line:84% They actually are informative toward the tasks of interest 00:40:54.430 --> 00:40:55.770 align:middle line:90% that you care about. 00:40:55.770 --> 00:40:57.470 align:middle line:84% So this cartoon on the right is meant 00:40:57.470 --> 00:40:59.990 align:middle line:84% to be like a minimal sufficient representation 00:40:59.990 --> 00:41:02.060 align:middle line:84% for semantic understanding of a scene. 00:41:02.060 --> 00:41:04.560 align:middle line:84% Just three things and where they are, but not all the weird, 00:41:04.560 --> 00:41:06.352 align:middle line:84% photometric details that might not actually 00:41:06.352 --> 00:41:09.720 align:middle line:84% matter for driving a car or for the semantic problems 00:41:09.720 --> 00:41:12.080 align:middle line:90% of interest. 00:41:12.080 --> 00:41:17.160 align:middle line:84% Another one might be that your representation has disentangled 00:41:17.160 --> 00:41:20.200 align:middle line:84% the interesting factors of variation in your data 00:41:20.200 --> 00:41:24.520 align:middle line:84% into independent dimensions, so axis-aligned representation. 00:41:24.520 --> 00:41:26.640 align:middle line:84% We'll talk a little bit about that. 00:41:26.640 --> 00:41:29.020 align:middle line:84% I have two more, but I want to query the class now. 00:41:29.020 --> 00:41:30.252 align:middle line:90% What are your ideas? 00:41:30.252 --> 00:41:32.460 align:middle line:84% What would you want in a good representation of data? 00:41:32.460 --> 00:41:37.720 align:middle line:90% 00:41:37.720 --> 00:41:41.840 align:middle line:84% What else is there besides this, minimal, compact, sufficient, 00:41:41.840 --> 00:41:43.260 align:middle line:90% disentangled? 00:41:43.260 --> 00:41:43.760 align:middle line:90% Yeah? 00:41:43.760 --> 00:41:44.920 align:middle line:90% Let's go here. 00:41:44.920 --> 00:41:46.680 align:middle line:84% AUDIENCE: Somehow human interpretable? 00:41:46.680 --> 00:41:48.840 align:middle line:84% PHILLIP ISOLA: Yes, interpretable. 00:41:48.840 --> 00:41:51.640 align:middle line:84% You got exactly the one I was thinking of next. 00:41:51.640 --> 00:41:53.960 align:middle line:84% So, yeah, there's other considerations. 00:41:53.960 --> 00:41:56.460 align:middle line:84% We're going to have to interface with these representations. 00:41:56.460 --> 00:41:57.600 align:middle line:90% OK, another one? 00:41:57.600 --> 00:42:00.700 align:middle line:84% AUDIENCE: Similar dimensional size, each dimension, 00:42:00.700 --> 00:42:02.680 align:middle line:90% [INAUDIBLE] something. 00:42:02.680 --> 00:42:05.900 align:middle line:84% PHILLIP ISOLA: Yeah, you maybe want your vector embeddings 00:42:05.900 --> 00:42:10.250 align:middle line:84% to be unit variance or have some nice numerical properties. 00:42:10.250 --> 00:42:11.000 align:middle line:90% That's a good one. 00:42:11.000 --> 00:42:11.958 align:middle line:90% I didn't think of that. 00:42:11.958 --> 00:42:15.900 align:middle line:84% AUDIENCE: [INAUDIBLE] between explanatory and interpretable. 00:42:15.900 --> 00:42:16.877 align:middle line:90% It seems similar. 00:42:16.877 --> 00:42:19.460 align:middle line:84% PHILLIP ISOLA: Explanatory and interpretable-- by explanatory, 00:42:19.460 --> 00:42:24.420 align:middle line:84% I meant that it is explanatory toward the automated 00:42:24.420 --> 00:42:26.603 align:middle line:84% tasks of interest, like the machine will 00:42:26.603 --> 00:42:28.020 align:middle line:84% be able to use that representation 00:42:28.020 --> 00:42:30.045 align:middle line:84% to solve problems and make accurate predictions. 00:42:30.045 --> 00:42:31.420 align:middle line:84% By interpretable, I meant a human 00:42:31.420 --> 00:42:34.380 align:middle line:84% can use that representation to solve problems and understand 00:42:34.380 --> 00:42:34.880 align:middle line:90% things. 00:42:34.880 --> 00:42:36.940 align:middle line:84% They're definitely related, but that was the distinction 00:42:36.940 --> 00:42:37.940 align:middle line:90% I was trying to make. 00:42:37.940 --> 00:42:38.980 align:middle line:90% Back here? 00:42:38.980 --> 00:42:40.220 align:middle line:90% AUDIENCE: Context aware. 00:42:40.220 --> 00:42:41.678 align:middle line:84% PHILLIP ISOLA: Context aware, yeah. 00:42:41.678 --> 00:42:45.060 align:middle line:84% You want your representation to know as much as possible 00:42:45.060 --> 00:42:48.340 align:middle line:84% about the universe when it's making a decision. 00:42:48.340 --> 00:42:49.435 align:middle line:90% Another one? 00:42:49.435 --> 00:42:52.060 align:middle line:84% AUDIENCE: Something similar, I was just going to say resilient. 00:42:52.060 --> 00:42:53.740 align:middle line:84% PHILLIP ISOLA: Resilient, robust, yeah. 00:42:53.740 --> 00:42:56.540 align:middle line:84% So you want representations that are not going 00:42:56.540 --> 00:42:59.097 align:middle line:90% to be fragile and vary a lot. 00:42:59.097 --> 00:43:01.430 align:middle line:84% When the input data changes, you want the representation 00:43:01.430 --> 00:43:03.390 align:middle line:90% to be stable. 00:43:03.390 --> 00:43:05.190 align:middle line:84% And you don't want it to be you can just 00:43:05.190 --> 00:43:07.065 align:middle line:84% add a little noise to the data and, suddenly, 00:43:07.065 --> 00:43:09.790 align:middle line:84% the representation will go crazy. 00:43:09.790 --> 00:43:11.170 align:middle line:90% And so those are all good. 00:43:11.170 --> 00:43:12.690 align:middle line:90% I just had this one final-- 00:43:12.690 --> 00:43:15.350 align:middle line:84% yeah, good representation, fundamentally, it's 00:43:15.350 --> 00:43:18.870 align:middle line:84% just something that makes subsequent problem-solving easy. 00:43:18.870 --> 00:43:21.773 align:middle line:90% Think of the 4DA transform. 00:43:21.773 --> 00:43:23.190 align:middle line:84% It's a representation of data that 00:43:23.190 --> 00:43:24.490 align:middle line:90% makes convolution really easy. 00:43:24.490 --> 00:43:26.910 align:middle line:84% It makes convolution just into a product. 00:43:26.910 --> 00:43:28.930 align:middle line:84% So that's a classic representation, 00:43:28.930 --> 00:43:31.190 align:middle line:84% but all representations have this property 00:43:31.190 --> 00:43:33.990 align:middle line:90% or should have this property. 00:43:33.990 --> 00:43:37.170 align:middle line:90% You can think of more. 00:43:37.170 --> 00:43:40.870 align:middle line:84% So how do we learn representations? 00:43:40.870 --> 00:43:44.470 align:middle line:84% So one way is you just train your neural network on whatever 00:43:44.470 --> 00:43:46.130 align:middle line:84% task you want, like music prediction, 00:43:46.130 --> 00:43:47.810 align:middle line:84% and you hope that it learned a good representation. 00:43:47.810 --> 00:43:48.570 align:middle line:90% And that's OK. 00:43:48.570 --> 00:43:50.228 align:middle line:90% That works sometimes. 00:43:50.228 --> 00:43:51.770 align:middle line:84% That's just like supervised learning. 00:43:51.770 --> 00:43:56.030 align:middle line:84% And it will find an f, and maybe that f is a good representation. 00:43:56.030 --> 00:43:58.970 align:middle line:84% But we're now going to talk about methods 00:43:58.970 --> 00:44:01.250 align:middle line:84% that don't target some supervised task, 00:44:01.250 --> 00:44:04.050 align:middle line:84% they just try to learn good representations generically. 00:44:04.050 --> 00:44:06.050 align:middle line:84% And we sometimes call this unsupervised learning 00:44:06.050 --> 00:44:08.010 align:middle line:84% or self-supervised learning, where 00:44:08.010 --> 00:44:11.777 align:middle line:84% we have input data, which is not xy pairs, but it's just x. 00:44:11.777 --> 00:44:13.610 align:middle line:84% There's no target, but I want to still learn 00:44:13.610 --> 00:44:16.650 align:middle line:90% a good representation. 00:44:16.650 --> 00:44:19.188 align:middle line:84% And so what would a good representation be? 00:44:19.188 --> 00:44:20.730 align:middle line:84% So maybe it's some embedding vectors. 00:44:20.730 --> 00:44:22.310 align:middle line:84% I said that's the main one we'll focus on. 00:44:22.310 --> 00:44:24.060 align:middle line:84% But there's other things it could be, too. 00:44:24.060 --> 00:44:25.610 align:middle line:90% It could be clusters. 00:44:25.610 --> 00:44:27.670 align:middle line:90% It could be distance metric. 00:44:27.670 --> 00:44:29.770 align:middle line:84% You could learn a proper distance metric 00:44:29.770 --> 00:44:31.890 align:middle line:90% for your data domain x. 00:44:31.890 --> 00:44:34.785 align:middle line:84% A metric is a function of x1 and x2, 00:44:34.785 --> 00:44:36.410 align:middle line:84% so it's a bivariate function as opposed 00:44:36.410 --> 00:44:38.770 align:middle line:84% to an embedding, which is a univariate function. 00:44:38.770 --> 00:44:40.690 align:middle line:90% It just is a function of x. 00:44:40.690 --> 00:44:42.790 align:middle line:84% We'll talk about metrics in the next lectures. 00:44:42.790 --> 00:44:45.725 align:middle line:90% 00:44:45.725 --> 00:44:47.850 align:middle line:84% So we're going to talk about the vector embeddings. 00:44:47.850 --> 00:44:52.850 align:middle line:84% And in my opinion, there's two general principles 00:44:52.850 --> 00:44:55.210 align:middle line:84% for how to learn a good vector embedding without having 00:44:55.210 --> 00:44:58.220 align:middle line:90% an explicit supervised task. 00:44:58.220 --> 00:45:00.160 align:middle line:84% So the one principle is compression, 00:45:00.160 --> 00:45:03.700 align:middle line:84% just find a way to find a good compression of your data. 00:45:03.700 --> 00:45:05.320 align:middle line:84% And the other principle is prediction, 00:45:05.320 --> 00:45:08.580 align:middle line:84% be able to predict missing data when you hold that data out. 00:45:08.580 --> 00:45:10.820 align:middle line:90% We'll talk about both of these. 00:45:10.820 --> 00:45:13.980 align:middle line:84% And in the reading, there's a catalog of some of the methods 00:45:13.980 --> 00:45:15.960 align:middle line:84% we'll see and how at least, in my view, 00:45:15.960 --> 00:45:18.792 align:middle line:84% they can all be understood as compression or prediction. 00:45:18.792 --> 00:45:20.500 align:middle line:84% And if you want to, you can actually even 00:45:20.500 --> 00:45:23.360 align:middle line:84% see the compression prediction as fundamentally the same. 00:45:23.360 --> 00:45:25.360 align:middle line:84% I'm not sure if there's really a difference, 00:45:25.360 --> 00:45:28.300 align:middle line:84% but at least it's useful, intuitively, 00:45:28.300 --> 00:45:30.240 align:middle line:84% to think about compression versus prediction. 00:45:30.240 --> 00:45:32.412 align:middle line:90% So let's see both of these. 00:45:32.412 --> 00:45:34.120 align:middle line:84% That's a question for you to think about. 00:45:34.120 --> 00:45:36.203 align:middle line:84% Is there actually a connection between compression 00:45:36.203 --> 00:45:38.260 align:middle line:90% and prediction? 00:45:38.260 --> 00:45:41.840 align:middle line:84% So learning via compression will work as follows. 00:45:41.840 --> 00:45:43.580 align:middle line:84% We'll take our data, and we'll try 00:45:43.580 --> 00:45:46.800 align:middle line:84% to find that compact mental representation. 00:45:46.800 --> 00:45:50.267 align:middle line:84% But what is that going to actually do mechanistically? 00:45:50.267 --> 00:45:51.600 align:middle line:90% So we're going to take our data. 00:45:51.600 --> 00:45:52.800 align:middle line:84% We're going to find a vector embedding, 00:45:52.800 --> 00:45:55.110 align:middle line:84% and then we're going to find a lower-dimensional vector 00:45:55.110 --> 00:45:56.510 align:middle line:84% embedding and a lower-dimensional vector 00:45:56.510 --> 00:45:56.930 align:middle line:90% embedding. 00:45:56.930 --> 00:45:58.513 align:middle line:84% And we'll just do that by constructing 00:45:58.513 --> 00:46:02.350 align:middle line:84% a neural network, where the width decreases 00:46:02.350 --> 00:46:03.370 align:middle line:90% as a function of depth. 00:46:03.370 --> 00:46:06.790 align:middle line:84% So each new layer will be lower width. 00:46:06.790 --> 00:46:08.850 align:middle line:90% So that is all fine. 00:46:08.850 --> 00:46:12.010 align:middle line:84% I'll learn a low dimensional compressed representation, 00:46:12.010 --> 00:46:14.270 align:middle line:84% but I also need it to be sufficient in explanatory 00:46:14.270 --> 00:46:14.810 align:middle line:90% of the data. 00:46:14.810 --> 00:46:18.508 align:middle line:84% I don't want to just map my input to 0, right? 00:46:18.508 --> 00:46:20.550 align:middle line:84% I want to map it to something that actually still 00:46:20.550 --> 00:46:22.430 align:middle line:90% can reconstruct the data. 00:46:22.430 --> 00:46:25.710 align:middle line:84% So we call that embedding vector-- canonically, 00:46:25.710 --> 00:46:27.590 align:middle line:90% we'll call it z. 00:46:27.590 --> 00:46:31.310 align:middle line:84% And we also want to be able to decode 00:46:31.310 --> 00:46:34.150 align:middle line:84% the image from the compressed representation. 00:46:34.150 --> 00:46:35.670 align:middle line:84% I'm using a lot of image examples, 00:46:35.670 --> 00:46:37.190 align:middle line:90% just because I like images. 00:46:37.190 --> 00:46:38.990 align:middle line:90% I often work on images. 00:46:38.990 --> 00:46:42.270 align:middle line:84% But think of image as just a placeholder for whatever data 00:46:42.270 --> 00:46:45.242 align:middle line:84% you want to put there, whatever domain you care about. 00:46:45.242 --> 00:46:46.450 align:middle line:90% So what is this thing called? 00:46:46.450 --> 00:46:48.430 align:middle line:90% It's called an autoencoder. 00:46:48.430 --> 00:46:53.390 align:middle line:84% And this is representation learning algorithm 101, 00:46:53.390 --> 00:46:56.530 align:middle line:84% just the most basic vanilla one, and also my favorite, 00:46:56.530 --> 00:46:57.670 align:middle line:90% and probably the best one. 00:46:57.670 --> 00:46:59.833 align:middle line:84% And I think it will just win out in the end. 00:46:59.833 --> 00:47:01.250 align:middle line:84% So many things in machine learning 00:47:01.250 --> 00:47:03.490 align:middle line:84% can be viewed as some kind of autoencoder 00:47:03.490 --> 00:47:05.350 align:middle line:84% or some variation on an autoencoder. 00:47:05.350 --> 00:47:07.230 align:middle line:84% So this is a very fundamental concept. 00:47:07.230 --> 00:47:09.850 align:middle line:84% Just map the data to a simpler form 00:47:09.850 --> 00:47:13.650 align:middle line:84% such that you can decode the original data from that simpler 00:47:13.650 --> 00:47:15.570 align:middle line:90% form. 00:47:15.570 --> 00:47:20.610 align:middle line:84% Autoencoders is just such a beautiful idea. 00:47:20.610 --> 00:47:23.350 align:middle line:84% This is not the original reference for autoencoders. 00:47:23.350 --> 00:47:27.570 align:middle line:84% But Hinton, who just won the physics Nobel Prize, of course, 00:47:27.570 --> 00:47:31.290 align:middle line:84% was one of the people that really helped develop 00:47:31.290 --> 00:47:32.898 align:middle line:90% autoencoders in his early work. 00:47:32.898 --> 00:47:35.190 align:middle line:84% I'm not sure there's a single inventor of autoencoders. 00:47:35.190 --> 00:47:36.410 align:middle line:90% That would probably be-- 00:47:36.410 --> 00:47:39.370 align:middle line:84% yeah, it's an idea that's existed for centuries 00:47:39.370 --> 00:47:42.010 align:middle line:90% in some form or another. 00:47:42.010 --> 00:47:44.550 align:middle line:90% So that's an autoencoder. 00:47:44.550 --> 00:47:46.170 align:middle line:84% So let's look at it mathematically. 00:47:46.170 --> 00:47:48.190 align:middle line:84% So we have our input data distribution, 00:47:48.190 --> 00:47:49.590 align:middle line:90% which is all complicated. 00:47:49.590 --> 00:47:51.940 align:middle line:84% We map it to a compressed, lower-dimensional 00:47:51.940 --> 00:47:53.660 align:middle line:90% representation. 00:47:53.660 --> 00:47:57.068 align:middle line:84% So I'm drawing it as a circle because it's a smaller object. 00:47:57.068 --> 00:47:58.360 align:middle line:90% It's a lower-dimensional thing. 00:47:58.360 --> 00:48:01.620 align:middle line:84% It might not actually be like a Gaussian object. 00:48:01.620 --> 00:48:03.742 align:middle line:84% In order for it to actually be a Gaussian circle, 00:48:03.742 --> 00:48:06.200 align:middle line:84% we'll have to do something called variational autoencoders. 00:48:06.200 --> 00:48:07.242 align:middle line:90% We'll come to that later. 00:48:07.242 --> 00:48:09.500 align:middle line:84% But just keep in mind that autoencoders 00:48:09.500 --> 00:48:12.820 align:middle line:84% do have a way of actually mapping to the circle. 00:48:12.820 --> 00:48:15.860 align:middle line:84% But we'll talk about just vanilla autoencoders for now. 00:48:15.860 --> 00:48:19.180 align:middle line:84% And then we'll train the decoder, g, 00:48:19.180 --> 00:48:21.682 align:middle line:84% which will predict the data in such a way 00:48:21.682 --> 00:48:23.140 align:middle line:84% that we minimize the reconstruction 00:48:23.140 --> 00:48:26.380 align:middle line:84% error between the reconstructed predicted data and the input 00:48:26.380 --> 00:48:27.380 align:middle line:90% data. 00:48:27.380 --> 00:48:28.960 align:middle line:84% So here's our objective function. 00:48:28.960 --> 00:48:30.700 align:middle line:84% We're just going to say that encode 00:48:30.700 --> 00:48:32.240 align:middle line:84% x into a lower-dimensional format, 00:48:32.240 --> 00:48:34.780 align:middle line:84% decode it back to the original dimensionality 00:48:34.780 --> 00:48:35.860 align:middle line:90% should be an identity. 00:48:35.860 --> 00:48:37.620 align:middle line:90% G of f should be an identity. 00:48:37.620 --> 00:48:40.700 align:middle line:84% And that should just be reconstructing the raw data x 00:48:40.700 --> 00:48:44.700 align:middle line:84% in expectation over the training set. 00:48:44.700 --> 00:48:49.560 align:middle line:90% So that's an autoencoder. 00:48:49.560 --> 00:48:55.520 align:middle line:84% So here's a learning diagram of what the autoencoder objective 00:48:55.520 --> 00:48:56.740 align:middle line:90% and hypothesis spaces. 00:48:56.740 --> 00:48:59.080 align:middle line:84% So the critical thing to note here 00:48:59.080 --> 00:49:01.580 align:middle line:84% is that the autoencoder objective just says, 00:49:01.580 --> 00:49:03.920 align:middle line:84% I'm going to define big F as being the composition 00:49:03.920 --> 00:49:06.640 align:middle line:90% of the decoder and the encoder. 00:49:06.640 --> 00:49:10.160 align:middle line:84% So it's encode and then decode becomes this one big function F. 00:49:10.160 --> 00:49:12.080 align:middle line:84% So the funny thing about the autoencoder 00:49:12.080 --> 00:49:15.440 align:middle line:84% is that this big function F should ideally just 00:49:15.440 --> 00:49:16.620 align:middle line:90% be an identity. 00:49:16.620 --> 00:49:17.740 align:middle line:90% It should do nothing. 00:49:17.740 --> 00:49:19.405 align:middle line:90% So the objective is trivial. 00:49:19.405 --> 00:49:21.780 align:middle line:84% The objective is you just don't do anything to your data, 00:49:21.780 --> 00:49:25.360 align:middle line:90% just learn identity. 00:49:25.360 --> 00:49:27.420 align:middle line:84% If I had no constraints on the architecture, 00:49:27.420 --> 00:49:29.120 align:middle line:84% on the hypothesis space, the autoencoder 00:49:29.120 --> 00:49:30.760 align:middle line:84% would be a trivial thing to learn, 00:49:30.760 --> 00:49:31.980 align:middle line:90% and it would have no utility. 00:49:31.980 --> 00:49:36.600 align:middle line:84% It would just spit out an exact copy of the data. 00:49:36.600 --> 00:49:38.840 align:middle line:84% But the key thing is the autoencoder 00:49:38.840 --> 00:49:40.240 align:middle line:84% works because you put constraints 00:49:40.240 --> 00:49:41.540 align:middle line:90% over the architecture. 00:49:41.540 --> 00:49:45.017 align:middle line:84% In our formal language, we'll call it the hypothesis space. 00:49:45.017 --> 00:49:46.600 align:middle line:84% So the key thing is that we're mapping 00:49:46.600 --> 00:49:50.210 align:middle line:84% from a high-dimensional space to a lower-dimensional embedding 00:49:50.210 --> 00:49:51.010 align:middle line:90% and back. 00:49:51.010 --> 00:49:53.290 align:middle line:84% So typically, m should be less than n. 00:49:53.290 --> 00:49:55.210 align:middle line:84% Now there are variations on autoencoders 00:49:55.210 --> 00:49:56.710 align:middle line:90% where you don't need that. 00:49:56.710 --> 00:49:59.930 align:middle line:84% You can impose simplicity on the embeddings in other ways 00:49:59.930 --> 00:50:01.690 align:middle line:90% than dimensionality reduction. 00:50:01.690 --> 00:50:04.643 align:middle line:84% But for vanilla autoencoders, m is less than n. 00:50:04.643 --> 00:50:06.310 align:middle line:84% And so for an image, it looks like this. 00:50:06.310 --> 00:50:09.970 align:middle line:84% We have a low-dimensional representation z. 00:50:09.970 --> 00:50:15.530 align:middle line:90% So question for the audience. 00:50:15.530 --> 00:50:18.753 align:middle line:90% What if f and g are both linear? 00:50:18.753 --> 00:50:20.170 align:middle line:84% So I learn a linear transformation 00:50:20.170 --> 00:50:21.753 align:middle line:84% to a low-dimensional space, and then I 00:50:21.753 --> 00:50:24.530 align:middle line:84% learn a linear transformation that tries to invert that. 00:50:24.530 --> 00:50:26.470 align:middle line:84% What do you think that is going to result in? 00:50:26.470 --> 00:50:27.880 align:middle line:90% Does anybody know? 00:50:27.880 --> 00:50:30.130 align:middle line:84% It's a model that I've encountered in previous machine 00:50:30.130 --> 00:50:31.150 align:middle line:90% learning or statistics. 00:50:31.150 --> 00:50:31.650 align:middle line:90% Yeah? 00:50:31.650 --> 00:50:34.170 align:middle line:84% AUDIENCE: So will it be another linear? 00:50:34.170 --> 00:50:36.730 align:middle line:84% PHILLIP ISOLA: That will be another linear function. 00:50:36.730 --> 00:50:39.050 align:middle line:90% Yeah, that's right. 00:50:39.050 --> 00:50:40.990 align:middle line:84% OK, yeah, that's one good response. 00:50:40.990 --> 00:50:43.130 align:middle line:90% Any others? 00:50:43.130 --> 00:50:44.530 align:middle line:90% Let's go over here. 00:50:44.530 --> 00:50:50.310 align:middle line:84% AUDIENCE: [INAUDIBLE] the equivalent of PCA, but-- 00:50:50.310 --> 00:50:52.030 align:middle line:84% PHILLIP ISOLA: Yeah, equivalent of PCA. 00:50:52.030 --> 00:50:52.530 align:middle line:90% OK, right. 00:50:52.530 --> 00:50:54.990 align:middle line:90% So you're on to this. 00:50:54.990 --> 00:50:56.050 align:middle line:90% What is PCA doing? 00:50:56.050 --> 00:51:00.270 align:middle line:84% PCA can be understood as trying to maximize the variance I'm 00:51:00.270 --> 00:51:05.750 align:middle line:84% capturing in the signal via some linear orthogonal transformation 00:51:05.750 --> 00:51:07.732 align:middle line:90% of the vector. 00:51:07.732 --> 00:51:09.190 align:middle line:84% And if I'm maximizing the variance, 00:51:09.190 --> 00:51:11.230 align:middle line:90% I'm able to best reconstruct. 00:51:11.230 --> 00:51:15.110 align:middle line:84% And that actually is exactly equivalent to the L2 00:51:15.110 --> 00:51:16.370 align:middle line:90% reconstruction objective. 00:51:16.370 --> 00:51:18.830 align:middle line:84% So PCA can be understood as doing 00:51:18.830 --> 00:51:20.230 align:middle line:90% essentially the same thing. 00:51:20.230 --> 00:51:21.430 align:middle line:84% So if they're both linear, we'll just 00:51:21.430 --> 00:51:23.030 align:middle line:84% replace it with the encoders of matrix 00:51:23.030 --> 00:51:27.710 align:middle line:84% W. The decoder is a matrix Wg, and the encoders matrix Wf. 00:51:27.710 --> 00:51:30.670 align:middle line:84% And we're trying to say, if I pass encode, decode, 00:51:30.670 --> 00:51:35.030 align:middle line:84% I will obtain my original data matrix again. 00:51:35.030 --> 00:51:37.550 align:middle line:84% That's the objective of an autoencoder 00:51:37.550 --> 00:51:40.630 align:middle line:90% written with linear f and g. 00:51:40.630 --> 00:51:47.000 align:middle line:84% And PCA is a variant of this, where we assume that the encoder 00:51:47.000 --> 00:51:49.000 align:middle line:84% and decoder are the same matrix W. 00:51:49.000 --> 00:51:52.720 align:middle line:84% And they have this property that it's an orthogonal matrix, 00:51:52.720 --> 00:51:54.063 align:middle line:90% so that's the constraint. 00:51:54.063 --> 00:51:55.480 align:middle line:84% It's a slightly different setting, 00:51:55.480 --> 00:51:57.080 align:middle line:90% but it's almost the same. 00:51:57.080 --> 00:52:00.400 align:middle line:84% And so this is the objective that PCA tries to minimize. 00:52:00.400 --> 00:52:04.400 align:middle line:84% It tries to find W such that this transformation will 00:52:04.400 --> 00:52:07.760 align:middle line:84% result in decoding back to the original data. 00:52:07.760 --> 00:52:09.660 align:middle line:84% So you can work through the math. 00:52:09.660 --> 00:52:11.520 align:middle line:84% If you're more familiar with PCA maximize 00:52:11.520 --> 00:52:14.560 align:middle line:84% the variance in the projected space, 00:52:14.560 --> 00:52:20.960 align:middle line:84% then you can work out that this thing under this condition 00:52:20.960 --> 00:52:25.360 align:middle line:84% is equivalent to minimize the variance in x 00:52:25.360 --> 00:52:27.960 align:middle line:84% minus the variance in the transformed version of x. 00:52:27.960 --> 00:52:33.080 align:middle line:84% And then we can maximize and change the sign 00:52:33.080 --> 00:52:35.200 align:middle line:90% to be plus instead of minus. 00:52:35.200 --> 00:52:38.080 align:middle line:84% And then we can remove this term because this term doesn't 00:52:38.080 --> 00:52:40.380 align:middle line:84% depend on W, and we're maximizing over w. 00:52:40.380 --> 00:52:42.560 align:middle line:84% And then this is just maximize the variance captured 00:52:42.560 --> 00:52:45.090 align:middle line:90% in the reconstructions. 00:52:45.090 --> 00:52:50.090 align:middle line:84% And that might be more familiar PCA that you're used to. 00:52:50.090 --> 00:52:54.570 align:middle line:84% And what can be proven is that the embeddings 00:52:54.570 --> 00:52:57.010 align:middle line:84% learned by a linear autoencoder span 00:52:57.010 --> 00:53:00.330 align:middle line:84% the same subspace as embeddings learned by PCA. 00:53:00.330 --> 00:53:03.210 align:middle line:84% So this is the rough the rough argument there. 00:53:03.210 --> 00:53:07.690 align:middle line:84% So vanilla, most intro statistical model 00:53:07.690 --> 00:53:09.650 align:middle line:84% that you can think of, the most standard one 00:53:09.650 --> 00:53:12.070 align:middle line:84% that's been around for hundreds of years, I believe, 00:53:12.070 --> 00:53:14.530 align:middle line:90% or quite a long time, PCA. 00:53:14.530 --> 00:53:16.692 align:middle line:84% Well, an autoencoder, the very best representation 00:53:16.692 --> 00:53:18.650 align:middle line:84% learning method in the family of reconstruction 00:53:18.650 --> 00:53:21.290 align:middle line:84% based representation learning is just nonlinear PCA. 00:53:21.290 --> 00:53:26.650 align:middle line:84% That's a generalization of PCA to nonlinear representations. 00:53:26.650 --> 00:53:31.650 align:middle line:84% By the way, I was trying to work out these equations. 00:53:31.650 --> 00:53:36.050 align:middle line:84% And I found that ChatGPT has gotten incredibly good at this. 00:53:36.050 --> 00:53:37.970 align:middle line:84% So if you're a little bit confused about-- 00:53:37.970 --> 00:53:40.110 align:middle line:84% I didn't put in every single step here. 00:53:40.110 --> 00:53:42.630 align:middle line:84% But if you're a little confused, asked 00:53:42.630 --> 00:53:46.550 align:middle line:84% ChatGPT, what's the connection between PCA 00:53:46.550 --> 00:53:49.790 align:middle line:90% and linear autoencoders? 00:53:49.790 --> 00:53:52.430 align:middle line:84% And it will just give this wonderful derivation 00:53:52.430 --> 00:53:53.090 align:middle line:90% of everything. 00:53:53.090 --> 00:53:54.230 align:middle line:90% I think it's correct. 00:53:54.230 --> 00:53:55.130 align:middle line:90% I did check. 00:53:55.130 --> 00:53:56.350 align:middle line:90% I thought about it. 00:53:56.350 --> 00:53:59.010 align:middle line:84% But my point is that these tools are getting a lot better. 00:53:59.010 --> 00:54:01.350 align:middle line:84% So if you're confused about a concept in this class, 00:54:01.350 --> 00:54:03.590 align:middle line:84% not a homework question, but a concept, just 00:54:03.590 --> 00:54:05.910 align:middle line:84% ask ChatGPT, or Claude, or whatever language 00:54:05.910 --> 00:54:07.510 align:middle line:90% model you prefer. 00:54:07.510 --> 00:54:10.670 align:middle line:84% They're really good at explaining concepts. 00:54:10.670 --> 00:54:13.650 align:middle line:84% Anyway, of course, they can be wrong. 00:54:13.650 --> 00:54:15.910 align:middle line:84% You have to be very aware of that. 00:54:15.910 --> 00:54:18.550 align:middle line:84% But I'm just finding it to be quite a nice resource 00:54:18.550 --> 00:54:19.290 align:middle line:90% for teaching. 00:54:19.290 --> 00:54:21.310 align:middle line:84% And I'd recommend it for students, too. 00:54:21.310 --> 00:54:24.093 align:middle line:84% So let's do a little experiment now. 00:54:24.093 --> 00:54:26.510 align:middle line:84% Are autoencoder is actually learning good representations? 00:54:26.510 --> 00:54:27.682 align:middle line:90% Or it sounds nice. 00:54:27.682 --> 00:54:29.890 align:middle line:84% I learn a low-dimensional, compressed representation. 00:54:29.890 --> 00:54:32.950 align:middle line:84% But is there any point to doing this besides just compression 00:54:32.950 --> 00:54:34.970 align:middle line:90% and saving memory? 00:54:34.970 --> 00:54:37.330 align:middle line:84% OK, obviously, I save memory by doing this. 00:54:37.330 --> 00:54:39.590 align:middle line:84% I have a smaller object to put on my disk, 00:54:39.590 --> 00:54:41.600 align:middle line:84% but do I get any other advantage out? 00:54:41.600 --> 00:54:43.210 align:middle line:84% We said that we think we might get 00:54:43.210 --> 00:54:44.960 align:middle line:84% representation learning, where we actually 00:54:44.960 --> 00:54:47.360 align:middle line:84% get a meaningful representation, so do we? 00:54:47.360 --> 00:54:48.380 align:middle line:90% So here's an experiment. 00:54:48.380 --> 00:54:49.838 align:middle line:84% You'll do some of these experiments 00:54:49.838 --> 00:54:52.360 align:middle line:84% on this same type of data on your p set 3. 00:54:52.360 --> 00:54:54.480 align:middle line:84% So it's a data set of colored shapes. 00:54:54.480 --> 00:54:57.820 align:middle line:84% We have three different shapes-- triangle, circle, and square. 00:54:57.820 --> 00:55:01.440 align:middle line:84% And we have nine or so different colors. 00:55:01.440 --> 00:55:05.700 align:middle line:84% And we're going to try to train an autoencoder on this data. 00:55:05.700 --> 00:55:12.525 align:middle line:84% So we'll encode some x, and we'll visualize. 00:55:12.525 --> 00:55:13.900 align:middle line:84% To understand the representation, 00:55:13.900 --> 00:55:16.200 align:middle line:84% we're going to do this particular probe called 00:55:16.200 --> 00:55:17.840 align:middle line:90% a nearest neighbor probe. 00:55:17.840 --> 00:55:20.500 align:middle line:84% We're going to look in the representation space, 00:55:20.500 --> 00:55:22.460 align:middle line:84% so we're going to embed x into z. 00:55:22.460 --> 00:55:25.720 align:middle line:84% And then we're going to look at the nearest neighbors to z. 00:55:25.720 --> 00:55:29.160 align:middle line:84% We're going to look at the images whose z vectors 00:55:29.160 --> 00:55:32.600 align:middle line:84% are the nearest neighbors to our query image. 00:55:32.600 --> 00:55:34.840 align:middle line:84% So in the representation space z, 00:55:34.840 --> 00:55:37.250 align:middle line:84% the nearest neighbors to this triangle 00:55:37.250 --> 00:55:39.000 align:middle line:84% are going to be these other triangles that 00:55:39.000 --> 00:55:40.620 align:middle line:90% are roughly the same color. 00:55:40.620 --> 00:55:44.900 align:middle line:84% And the nearest neighbor to this square will be blue squares. 00:55:44.900 --> 00:55:46.120 align:middle line:90% So what is this saying? 00:55:46.120 --> 00:55:48.900 align:middle line:84% This is saying that our representation space 00:55:48.900 --> 00:55:53.100 align:middle line:84% has organized the data in a way that seems meaningful 00:55:53.100 --> 00:55:54.920 align:middle line:84% and seems to match human perception. 00:55:54.920 --> 00:55:57.420 align:middle line:84% So nearby embeddings in representation space, 00:55:57.420 --> 00:55:59.180 align:middle line:84% actually, are similar images, according 00:55:59.180 --> 00:56:01.160 align:middle line:90% to their color and their shape. 00:56:01.160 --> 00:56:02.660 align:middle line:84% So it looks like the autoencoder did 00:56:02.660 --> 00:56:04.940 align:middle line:84% learn a meaningful representation that 00:56:04.940 --> 00:56:08.500 align:middle line:84% encodes properties of both color and shape. 00:56:08.500 --> 00:56:10.500 align:middle line:84% Another thing that we can do is we 00:56:10.500 --> 00:56:14.060 align:middle line:84% can look at where that representation becomes 00:56:14.060 --> 00:56:14.960 align:middle line:90% effective. 00:56:14.960 --> 00:56:16.817 align:middle line:90% So we can now ask-- 00:56:16.817 --> 00:56:18.900 align:middle line:84% we'll measure the quality of the nearest neighbors 00:56:18.900 --> 00:56:21.320 align:middle line:84% by using them to do classification. 00:56:21.320 --> 00:56:24.425 align:middle line:84% So we'll do a KNN or, actually, one nearest-neighbor classifier. 00:56:24.425 --> 00:56:26.300 align:middle line:84% So what is a one nearest-neighbor classifier? 00:56:26.300 --> 00:56:29.020 align:middle line:84% That just says, is my first nearest neighbor in a data 00:56:29.020 --> 00:56:32.440 align:middle line:90% set the same class as the query? 00:56:32.440 --> 00:56:35.860 align:middle line:84% So is it the same color class, and is it the same shape class? 00:56:35.860 --> 00:56:38.430 align:middle line:84% And I'm plotting accuracy of the one nearest neighbor 00:56:38.430 --> 00:56:41.510 align:middle line:84% classifier for the shape class and the color class, 00:56:41.510 --> 00:56:46.870 align:middle line:84% and I'm plotting that at every layer of a encoder. 00:56:46.870 --> 00:56:49.990 align:middle line:84% This is a convolutional encoder applied to this data. 00:56:49.990 --> 00:56:51.870 align:middle line:90% So what's going on here? 00:56:51.870 --> 00:56:55.467 align:middle line:84% The accuracy at shape is getting better, 00:56:55.467 --> 00:56:56.550 align:middle line:90% which kind of makes sense. 00:56:56.550 --> 00:56:58.508 align:middle line:84% I'm learning a better and better representation 00:56:58.508 --> 00:57:00.550 align:middle line:84% so that similar shapes are embedded 00:57:00.550 --> 00:57:02.490 align:middle line:84% near each other in representation space. 00:57:02.490 --> 00:57:04.330 align:middle line:84% Why is the accuracy at color getting worse? 00:57:04.330 --> 00:57:05.210 align:middle line:90% Anyone have an idea? 00:57:05.210 --> 00:57:05.772 align:middle line:90% Yeah? 00:57:05.772 --> 00:57:07.230 align:middle line:84% AUDIENCE: In a very crude way, it's 00:57:07.230 --> 00:57:09.990 align:middle line:84% perhaps learning the principal components of shape 00:57:09.990 --> 00:57:14.072 align:middle line:84% and separating it from the principal component of color. 00:57:14.072 --> 00:57:16.030 align:middle line:84% PHILLIP ISOLA: Yeah, so it's learning something 00:57:16.030 --> 00:57:19.122 align:middle line:84% like the one principal component dimension is shape 00:57:19.122 --> 00:57:20.330 align:middle line:90% and the other is maybe color. 00:57:20.330 --> 00:57:22.470 align:middle line:84% That would be if I had the linear autoencoder, that 00:57:22.470 --> 00:57:23.968 align:middle line:90% might be exactly what it does. 00:57:23.968 --> 00:57:25.510 align:middle line:84% This one's non-linear, but it's going 00:57:25.510 --> 00:57:28.150 align:middle line:84% to be something kind of like that. 00:57:28.150 --> 00:57:30.990 align:middle line:84% So anyone have an idea for why color did not 00:57:30.990 --> 00:57:32.730 align:middle line:84% improve as I go deeper into the network? 00:57:32.730 --> 00:57:34.113 align:middle line:90% I mean, it got worse. 00:57:34.113 --> 00:57:36.530 align:middle line:84% But let's just ask the question, why didn't it get better? 00:57:36.530 --> 00:57:37.050 align:middle line:90% Yeah? 00:57:37.050 --> 00:57:39.810 align:middle line:84% AUDIENCE: You don't need that much stuff to-- 00:57:39.810 --> 00:57:42.410 align:middle line:84% PHILLIP ISOLA: You don't need that much stuff to decode color. 00:57:42.410 --> 00:57:43.070 align:middle line:90% That's the answer? 00:57:43.070 --> 00:57:43.590 align:middle line:90% OK, right. 00:57:43.590 --> 00:57:47.470 align:middle line:84% So in pixel space, pixel space is a representation of the data. 00:57:47.470 --> 00:57:50.090 align:middle line:84% It's just the raw format that we take the data in. 00:57:50.090 --> 00:57:53.650 align:middle line:84% And color is very superficial and explicit in pixel space. 00:57:53.650 --> 00:57:57.037 align:middle line:84% And the nearest-neighbor image in pixel space to this query 00:57:57.037 --> 00:57:58.370 align:middle line:90% is going to have the same color. 00:57:58.370 --> 00:58:00.570 align:middle line:84% Color is explicitly represented in that space. 00:58:00.570 --> 00:58:04.010 align:middle line:84% Shape is not explicitly represented in pixel space. 00:58:04.010 --> 00:58:09.150 align:middle line:84% If I do L2 distance between two images represented as pixels, 00:58:09.150 --> 00:58:12.050 align:middle line:84% just a vector of pixels, it won't 00:58:12.050 --> 00:58:13.510 align:middle line:84% put similar shapes near each other, 00:58:13.510 --> 00:58:15.850 align:middle line:84% but it will put similar colors near each other. 00:58:15.850 --> 00:58:19.290 align:middle line:84% You can think about why that is, but this is the general-- 00:58:19.290 --> 00:58:21.970 align:middle line:84% the general point is that every representation 00:58:21.970 --> 00:58:23.990 align:middle line:84% is good at some things and bad at other things, 00:58:23.990 --> 00:58:25.170 align:middle line:90% and there's trade-offs. 00:58:25.170 --> 00:58:28.050 align:middle line:84% So in this case, the pixels are very good representation 00:58:28.050 --> 00:58:30.450 align:middle line:84% of color, but a bad representation of shape. 00:58:30.450 --> 00:58:33.110 align:middle line:84% And the embeddings are a good representation of shape, 00:58:33.110 --> 00:58:34.870 align:middle line:84% but a worse representation of color, 00:58:34.870 --> 00:58:36.780 align:middle line:84% in fact, because the pixels were already, 00:58:36.780 --> 00:58:39.380 align:middle line:84% I guess, the best you could do in this case. 00:58:39.380 --> 00:58:41.437 align:middle line:84% So autoencoding doesn't strictly result 00:58:41.437 --> 00:58:42.520 align:middle line:90% in better representations. 00:58:42.520 --> 00:58:44.520 align:middle line:84% It results in different representations. 00:58:44.520 --> 00:58:45.900 align:middle line:90% That's a fundamental principle. 00:58:45.900 --> 00:58:47.900 align:middle line:84% All representation learning is making trade-offs 00:58:47.900 --> 00:58:50.980 align:middle line:84% to reformat the data into a way that makes some tasks easier, 00:58:50.980 --> 00:58:53.260 align:middle line:84% shape classification and other tasks harder, color 00:58:53.260 --> 00:58:56.940 align:middle line:90% classification in this example. 00:58:56.940 --> 00:59:04.740 align:middle line:84% So I said autoencoders are the deep learning version of PCA. 00:59:04.740 --> 00:59:07.900 align:middle line:84% And now I'm going to say that another model is 00:59:07.900 --> 00:59:09.720 align:middle line:84% the deep learning version of k-means. 00:59:09.720 --> 00:59:13.332 align:middle line:84% So PCA and k-means are the two most important models, 00:59:13.332 --> 00:59:15.540 align:middle line:84% and everything else is just a generalization of them. 00:59:15.540 --> 00:59:20.053 align:middle line:84% That's just some opinionated statement. 00:59:20.053 --> 00:59:21.720 align:middle line:84% But that's one way of thinking about it. 00:59:21.720 --> 00:59:24.380 align:middle line:90% So let's look at clustering now. 00:59:24.380 --> 00:59:29.380 align:middle line:84% So clustering is the problem of taking data and assigning 00:59:29.380 --> 00:59:31.500 align:middle line:90% each data point to a cluster. 00:59:31.500 --> 00:59:35.470 align:middle line:84% So a representation-learning lens on clustering 00:59:35.470 --> 00:59:38.050 align:middle line:84% is you're learning an encoder that doesn't output a vector. 00:59:38.050 --> 00:59:39.505 align:middle line:90% It outputs an integer. 00:59:39.505 --> 00:59:41.630 align:middle line:84% So for every data point, I output an integer, which 00:59:41.630 --> 00:59:42.910 align:middle line:90% is the cluster assignment. 00:59:42.910 --> 00:59:44.050 align:middle line:90% It still is an encoder. 00:59:44.050 --> 00:59:46.390 align:middle line:84% It's still representation learning. 00:59:46.390 --> 00:59:52.670 align:middle line:84% So at inference time, if I just apply my learned encoder 00:59:52.670 --> 00:59:57.230 align:middle line:84% that outputs integers to some data points, it will tell me, 00:59:57.230 --> 00:59:58.230 align:middle line:90% this is the third class. 00:59:58.230 --> 00:59:59.470 align:middle line:90% And I'll color that red. 00:59:59.470 --> 01:00:03.710 align:middle line:84% And that has identified clusters So clustering is learning 01:00:03.710 --> 01:00:07.630 align:middle line:90% an encoder to integers. 01:00:07.630 --> 01:00:12.190 align:middle line:84% So why is clustering a good representation? 01:00:12.190 --> 01:00:14.455 align:middle line:84% So from the representation-learning angle, 01:00:14.455 --> 01:00:15.830 align:middle line:84% clustering is learning a function 01:00:15.830 --> 01:00:17.538 align:middle line:84% f that outputs an integer, will represent 01:00:17.538 --> 01:00:19.777 align:middle line:84% the integers of one-hot code, in this example here. 01:00:19.777 --> 01:00:21.610 align:middle line:84% So we can think of it as a vector embedding, 01:00:21.610 --> 01:00:23.470 align:middle line:84% but it's just a one-hot vector embedding 01:00:23.470 --> 01:00:26.530 align:middle line:84% that's isomorphic with the integers, 01:00:26.530 --> 01:00:29.790 align:middle line:90% for a finite set of integers. 01:00:29.790 --> 01:00:31.470 align:middle line:90% So. 01:00:31.470 --> 01:00:33.210 align:middle line:84% what's the best representation that-- 01:00:33.210 --> 01:00:34.890 align:middle line:84% this is another subjective statement. 01:00:34.890 --> 01:00:36.410 align:middle line:84% What's the best representation that humans 01:00:36.410 --> 01:00:37.290 align:middle line:90% have come up with so far? 01:00:37.290 --> 01:00:38.040 align:middle line:90% What do you think? 01:00:38.040 --> 01:00:40.370 align:middle line:84% What's the best representation that the human brain 01:00:40.370 --> 01:00:42.630 align:middle line:90% has discovered? 01:00:42.630 --> 01:00:43.130 align:middle line:90% Yeah? 01:00:43.130 --> 01:00:43.570 align:middle line:90% AUDIENCE: Words. 01:00:43.570 --> 01:00:44.612 align:middle line:90% PHILLIP ISOLA: Words, OK. 01:00:44.612 --> 01:00:46.410 align:middle line:90% That was my answer. 01:00:46.410 --> 01:00:47.150 align:middle line:90% You can disagree. 01:00:47.150 --> 01:00:48.530 align:middle line:90% I don't know if this is true. 01:00:48.530 --> 01:00:51.710 align:middle line:84% Traditionally, I come from a computer vision background, 01:00:51.710 --> 01:00:55.102 align:middle line:84% and so this is kind of blasphemy, I suppose. 01:00:55.102 --> 01:00:57.810 align:middle line:84% But I think language is just the best representation of the world 01:00:57.810 --> 01:00:59.190 align:middle line:90% that humans have discovered. 01:00:59.190 --> 01:01:00.607 align:middle line:84% There's other representations that 01:01:00.607 --> 01:01:02.110 align:middle line:84% are good and have their trade-offs, 01:01:02.110 --> 01:01:04.090 align:middle line:84% but language is like the pinnacle. 01:01:04.090 --> 01:01:06.290 align:middle line:84% And words are the atoms of language. 01:01:06.290 --> 01:01:08.590 align:middle line:84% There's grammar and there's structures between words, 01:01:08.590 --> 01:01:11.490 align:middle line:84% but words are the important component of language. 01:01:11.490 --> 01:01:14.560 align:middle line:84% And clustering is the problem of making up new words for things. 01:01:14.560 --> 01:01:16.310 align:middle line:84% That's one way of understanding it, right? 01:01:16.310 --> 01:01:21.170 align:middle line:84% A word is like a symbol that at least-- 01:01:21.170 --> 01:01:23.670 align:middle line:84% there might be some linguists that would quibble about this, 01:01:23.670 --> 01:01:26.410 align:middle line:84% but one rough definition of word is 01:01:26.410 --> 01:01:29.110 align:middle line:84% it's just like it is a symbol that denotes a set, 01:01:29.110 --> 01:01:31.070 align:middle line:84% and that's what clustering is doing. 01:01:31.070 --> 01:01:34.860 align:middle line:84% So this is like saying, oh, I will map every image 01:01:34.860 --> 01:01:38.780 align:middle line:84% like this to this cluster, and I can just label it arbitrarily 01:01:38.780 --> 01:01:40.940 align:middle line:90% with a word. 01:01:40.940 --> 01:01:42.840 align:middle line:84% Language has more structure beyond the words, 01:01:42.840 --> 01:01:45.120 align:middle line:84% but this is but words are an important part of it. 01:01:45.120 --> 01:01:46.860 align:middle line:84% So clustering is the representation 01:01:46.860 --> 01:01:51.900 align:middle line:84% learning problem of finding new words for concepts. 01:01:51.900 --> 01:01:53.500 align:middle line:90% Let's talk about k-means. 01:01:53.500 --> 01:01:54.960 align:middle line:90% So this should be review. 01:01:54.960 --> 01:01:56.760 align:middle line:84% I hope you've seen PCA and k-means before. 01:01:56.760 --> 01:01:58.593 align:middle line:84% So if you haven't, go back and review those. 01:01:58.593 --> 01:02:01.420 align:middle line:84% Because as I said, deep learning is just generalizing those, 01:02:01.420 --> 01:02:04.360 align:middle line:84% representation learning is generalizing these concepts. 01:02:04.360 --> 01:02:08.980 align:middle line:84% So what does k-means do? k-means finds k different clusters 01:02:08.980 --> 01:02:10.060 align:middle line:90% in your data. 01:02:10.060 --> 01:02:12.340 align:middle line:84% And each cluster is represented with what's 01:02:12.340 --> 01:02:15.100 align:middle line:84% called the mean, which is just going 01:02:15.100 --> 01:02:18.100 align:middle line:84% to be the average of all the data points in that cluster. 01:02:18.100 --> 01:02:20.420 align:middle line:84% So it maps data points to integers. 01:02:20.420 --> 01:02:22.780 align:middle line:84% And it does that in such a way that each data point is 01:02:22.780 --> 01:02:26.220 align:middle line:84% as close as possible to the mean of the data 01:02:26.220 --> 01:02:30.920 align:middle line:84% points in the cluster it is assigned to. 01:02:30.920 --> 01:02:36.040 align:middle line:84% So here's the representation learning view of k-means. 01:02:36.040 --> 01:02:38.900 align:middle line:84% I'm going to train an encoder that will output one-hot codes. 01:02:38.900 --> 01:02:42.647 align:middle line:84% And then my decoder will just be-- 01:02:42.647 --> 01:02:43.980 align:middle line:90% think of it like a lookup table. 01:02:43.980 --> 01:02:46.360 align:middle line:84% It takes in a one-hot code, and it outputs a vector. 01:02:46.360 --> 01:02:48.840 align:middle line:84% And it's trying to output the vector that will reconstruct 01:02:48.840 --> 01:02:51.960 align:middle line:90% the input as best as possible. 01:02:51.960 --> 01:02:54.640 align:middle line:84% If I only can output a single vector 01:02:54.640 --> 01:02:57.560 align:middle line:84% for every item in a cluster, every item in the cluster 01:02:57.560 --> 01:02:59.300 align:middle line:90% goes to the same one-hot code. 01:02:59.300 --> 01:03:02.320 align:middle line:84% So now I can decode that into just a single vector. 01:03:02.320 --> 01:03:05.100 align:middle line:84% Think of g as a matrix W applied to a one-hot code. 01:03:05.100 --> 01:03:08.240 align:middle line:90% It selects a row of that matrix. 01:03:08.240 --> 01:03:14.480 align:middle line:84% What is going to be the decoder that minimizes the Euclidean 01:03:14.480 --> 01:03:18.340 align:middle line:84% distance reconstruction error, the L2 reconstruction error? 01:03:18.340 --> 01:03:20.240 align:middle line:84% What is the name for the vector that 01:03:20.240 --> 01:03:23.920 align:middle line:84% minimizes the distance to all points 01:03:23.920 --> 01:03:27.060 align:middle line:90% in a set, the L2 distance? 01:03:27.060 --> 01:03:27.560 align:middle line:90% Yeah? 01:03:27.560 --> 01:03:28.050 align:middle line:90% AUDIENCE: Mean. 01:03:28.050 --> 01:03:28.970 align:middle line:90% PHILLIP ISOLA: The mean, OK. 01:03:28.970 --> 01:03:30.928 align:middle line:84% I think a lot of that, maybe some of you don't. 01:03:30.928 --> 01:03:34.792 align:middle line:84% But, yes, the vector that minimizes the L2 distance 01:03:34.792 --> 01:03:37.250 align:middle line:84% to a set of other points is the mean of those other points. 01:03:37.250 --> 01:03:38.890 align:middle line:90% You can show that. 01:03:38.890 --> 01:03:42.933 align:middle line:84% So that means that k-means is an L2 autoencoder. 01:03:42.933 --> 01:03:44.850 align:middle line:84% The only difference from the other autoencoder 01:03:44.850 --> 01:03:47.582 align:middle line:84% that I showed you is that the hypothesis space doesn't have 01:03:47.582 --> 01:03:48.790 align:middle line:90% a low-dimensional bottleneck. 01:03:48.790 --> 01:03:53.050 align:middle line:84% It has an integer bottleneck, OK? 01:03:53.050 --> 01:03:57.550 align:middle line:84% So that's just a mapping of k-means onto autoencoders. 01:03:57.550 --> 01:04:00.570 align:middle line:84% They're the same model with a different hypothesis space. 01:04:00.570 --> 01:04:03.190 align:middle line:84% k-means, this hypothesis space is not differentiable. 01:04:03.190 --> 01:04:05.448 align:middle line:84% It has some properties that can be exploited, 01:04:05.448 --> 01:04:06.990 align:middle line:84% some structure that can be exploited. 01:04:06.990 --> 01:04:10.710 align:middle line:84% So we optimize it with a different algorithm than SGD. 01:04:10.710 --> 01:04:13.690 align:middle line:84% And that's where you might have encountered the expectation 01:04:13.690 --> 01:04:17.030 align:middle line:84% maximization method for doing k-means and so forth. 01:04:17.030 --> 01:04:20.490 align:middle line:84% And, yeah, that's just because this problem 01:04:20.490 --> 01:04:21.910 align:middle line:90% has some special structure. 01:04:21.910 --> 01:04:26.230 align:middle line:90% 01:04:26.230 --> 01:04:28.590 align:middle line:84% And then the deep learning version of k-means 01:04:28.590 --> 01:04:31.670 align:middle line:84% is called a vector quantized autoencoder. 01:04:31.670 --> 01:04:33.870 align:middle line:84% So a vector quantized autoencoder is exactly 01:04:33.870 --> 01:04:36.070 align:middle line:84% the same as k-means, except that f is 01:04:36.070 --> 01:04:38.270 align:middle line:90% non-linear instead of linear. 01:04:38.270 --> 01:04:41.090 align:middle line:84% And there's a whole bunch of variations on VQ models. 01:04:41.090 --> 01:04:43.670 align:middle line:84% There's VQGAN, VQVAE, et cetera, et cetera. 01:04:43.670 --> 01:04:46.130 align:middle line:84% These have bells and whistles, but this is the gist of it. 01:04:46.130 --> 01:04:49.810 align:middle line:84% I'm trying to learn an encoding into a set of integers 01:04:49.810 --> 01:04:52.270 align:middle line:84% and a decoding that minimizes reconstruction error. 01:04:52.270 --> 01:04:56.950 align:middle line:84% And f will be a deep network, or little f in little g 01:04:56.950 --> 01:05:00.750 align:middle line:84% are both going to be deep neural networks. 01:05:00.750 --> 01:05:03.130 align:middle line:84% So you can read up on exactly how that method works, 01:05:03.130 --> 01:05:07.870 align:middle line:84% but it's just k-means with deep nets. 01:05:07.870 --> 01:05:08.590 align:middle line:90% Yeah? 01:05:08.590 --> 01:05:10.670 align:middle line:84% AUDIENCE: In this k-means analogy, 01:05:10.670 --> 01:05:13.090 align:middle line:84% does the autoencoder learn to find the best k? 01:05:13.090 --> 01:05:15.950 align:middle line:90% 01:05:15.950 --> 01:05:16.730 align:middle line:90% PHILLIP ISOLA: No. 01:05:16.730 --> 01:05:18.890 align:middle line:84% So does the autoencoder learn to find the best k's? 01:05:18.890 --> 01:05:21.830 align:middle line:84% So k is a hyperparameter and, typically, the user 01:05:21.830 --> 01:05:24.510 align:middle line:90% has to define it. 01:05:24.510 --> 01:05:27.800 align:middle line:84% Yeah, you can have methods that try to automatically determine 01:05:27.800 --> 01:05:31.040 align:middle line:84% k, but that's beyond the scope of these vanilla methods. 01:05:31.040 --> 01:05:34.840 align:middle line:90% 01:05:34.840 --> 01:05:37.000 align:middle line:84% So data compression-- that was the first way 01:05:37.000 --> 01:05:39.640 align:middle line:90% of learning representations. 01:05:39.640 --> 01:05:46.000 align:middle line:84% And it shows up in PCA, and k-means, and VQVAEs, and VQGANs, 01:05:46.000 --> 01:05:49.065 align:middle line:90% and autoencoders, and so forth. 01:05:49.065 --> 01:05:50.440 align:middle line:84% There's another thing you can do, 01:05:50.440 --> 01:05:53.060 align:middle line:84% and that will be the prediction method. 01:05:53.060 --> 01:05:55.520 align:middle line:84% So now rather than learning representations 01:05:55.520 --> 01:05:58.680 align:middle line:84% by compressing data, we're going to try to learn representations 01:05:58.680 --> 01:06:02.040 align:middle line:90% by predicting held-out data. 01:06:02.040 --> 01:06:04.885 align:middle line:84% And this is actually the kind I started this lecture with. 01:06:04.885 --> 01:06:06.760 align:middle line:84% I said that we could just pre-train a network 01:06:06.760 --> 01:06:08.420 align:middle line:90% on music classification. 01:06:08.420 --> 01:06:11.280 align:middle line:84% That's a prediction problem, and it induces an OK representation 01:06:11.280 --> 01:06:13.820 align:middle line:90% for music classification. 01:06:13.820 --> 01:06:16.320 align:middle line:84% But we're instead going to do something else, which is we'll 01:06:16.320 --> 01:06:17.420 align:middle line:90% say, we didn't want labels. 01:06:17.420 --> 01:06:19.712 align:middle line:84% We don't want to be we don't want a representation that 01:06:19.712 --> 01:06:21.395 align:middle line:90% is tied to a specific task. 01:06:21.395 --> 01:06:24.020 align:middle line:84% So we don't want to have to have humans label some narrow task. 01:06:24.020 --> 01:06:26.090 align:middle line:84% We do something much more generic 01:06:26.090 --> 01:06:29.930 align:middle line:84% that doesn't require annotations or task specificity. 01:06:29.930 --> 01:06:31.583 align:middle line:84% And one way to do that is we say, we're 01:06:31.583 --> 01:06:32.750 align:middle line:90% not going to predict labels. 01:06:32.750 --> 01:06:35.570 align:middle line:84% We're going to predict the raw data itself. 01:06:35.570 --> 01:06:38.030 align:middle line:84% And this is called self-supervised learning. 01:06:38.030 --> 01:06:39.677 align:middle line:84% So it's called self-supervised learning 01:06:39.677 --> 01:06:42.010 align:middle line:84% because it's using the machinery of supervised learning, 01:06:42.010 --> 01:06:45.690 align:middle line:84% meaning predict y from x, except that we define y 01:06:45.690 --> 01:06:50.510 align:middle line:84% as being some part of the raw data as opposed to some label. 01:06:50.510 --> 01:06:52.810 align:middle line:84% So this is like an autoencoder, except I'm 01:06:52.810 --> 01:06:56.810 align:middle line:84% predicting half of the data from the other half of the data. 01:06:56.810 --> 01:07:00.230 align:middle line:84% And interestingly, this works really well. 01:07:00.230 --> 01:07:03.170 align:middle line:84% This tends to work a lot better than autoencoders. 01:07:03.170 --> 01:07:05.822 align:middle line:84% So here's an example of this setup. 01:07:05.822 --> 01:07:08.030 align:middle line:84% This is the colorization problem I showed you before. 01:07:08.030 --> 01:07:10.890 align:middle line:84% Try to predict the colors from a black and white image? 01:07:10.890 --> 01:07:13.238 align:middle line:84% And this is a self-supervised method 01:07:13.238 --> 01:07:15.030 align:middle line:84% that tries to predict part of the raw data, 01:07:15.030 --> 01:07:17.280 align:middle line:84% the color channels, from another part of the raw data, 01:07:17.280 --> 01:07:20.070 align:middle line:90% the black and white channel. 01:07:20.070 --> 01:07:22.790 align:middle line:84% So we can now take that model that's 01:07:22.790 --> 01:07:25.590 align:middle line:84% just trained to do this self-supervised prediction 01:07:25.590 --> 01:07:26.950 align:middle line:90% of the missing colors. 01:07:26.950 --> 01:07:30.510 align:middle line:84% Now it's free labels because color images 01:07:30.510 --> 01:07:32.163 align:middle line:90% have the colors built in. 01:07:32.163 --> 01:07:33.330 align:middle line:90% I just took the color image. 01:07:33.330 --> 01:07:35.872 align:middle line:84% I split it into the luminance channel and the color channels. 01:07:35.872 --> 01:07:37.410 align:middle line:90% So it's cheap. 01:07:37.410 --> 01:07:40.790 align:middle line:84% It's easy to run this on just raw, unlabeled data. 01:07:40.790 --> 01:07:43.310 align:middle line:84% Now I can investigate, did this system actually learn 01:07:43.310 --> 01:07:44.658 align:middle line:90% a meaningful representation? 01:07:44.658 --> 01:07:46.450 align:middle line:84% So we'll do the deep net electrophysiology. 01:07:46.450 --> 01:07:48.158 align:middle line:84% I'll poke my probe in there and ask, what 01:07:48.158 --> 01:07:52.230 align:middle line:90% are the neurons sensitive to? 01:07:52.230 --> 01:07:55.430 align:middle line:84% So here, I guess I'll ask the class. 01:07:55.430 --> 01:07:56.830 align:middle line:90% What do you think? 01:07:56.830 --> 01:08:00.530 align:middle line:84% Remember when I talked about the Zeiler and Fergus model, 01:08:00.530 --> 01:08:03.030 align:middle line:84% I said that they were like face selective neurons, 01:08:03.030 --> 01:08:05.830 align:middle line:84% and dog face selective neurons, and other things. 01:08:05.830 --> 01:08:08.550 align:middle line:84% What do you think the neurons will be sensitive to 01:08:08.550 --> 01:08:09.980 align:middle line:90% for predicting colors? 01:08:09.980 --> 01:08:11.730 align:middle line:84% And let's look at layer five of a network, 01:08:11.730 --> 01:08:14.590 align:middle line:90% so deep inside the network. 01:08:14.590 --> 01:08:15.570 align:middle line:90% What do you all think? 01:08:15.570 --> 01:08:19.029 align:middle line:90% 01:08:19.029 --> 01:08:19.838 align:middle line:90% Yeah, over here? 01:08:19.838 --> 01:08:20.630 align:middle line:90% AUDIENCE: Textures. 01:08:20.630 --> 01:08:22.047 align:middle line:84% PHILLIP ISOLA: Textures, OK, good. 01:08:22.047 --> 01:08:23.175 align:middle line:90% Yeah, textures seem to be-- 01:08:23.175 --> 01:08:24.300 align:middle line:90% color is kind of low level. 01:08:24.300 --> 01:08:25.200 align:middle line:84% It seems like it should be something 01:08:25.200 --> 01:08:27.340 align:middle line:84% that's a photometric property, like texture. 01:08:27.340 --> 01:08:28.729 align:middle line:90% Any other ideas? 01:08:28.729 --> 01:08:29.479 align:middle line:90% Yeah, in the back. 01:08:29.479 --> 01:08:30.375 align:middle line:90% AUDIENCE: Objects. 01:08:30.375 --> 01:08:31.500 align:middle line:90% PHILLIP ISOLA: Objects, OK. 01:08:31.500 --> 01:08:34.960 align:middle line:90% Yeah, so why objects? 01:08:34.960 --> 01:08:36.880 align:middle line:84% AUDIENCE: Different classes of objects 01:08:36.880 --> 01:08:39.237 align:middle line:84% have different colors like [INAUDIBLE] 01:08:39.237 --> 01:08:41.279 align:middle line:84% PHILLIP ISOLA: Yeah, different classes of objects 01:08:41.279 --> 01:08:42.720 align:middle line:90% have different colors. 01:08:42.720 --> 01:08:44.120 align:middle line:84% If I know something's strawberry, 01:08:44.120 --> 01:08:45.500 align:middle line:90% I can say it's probably red. 01:08:45.500 --> 01:08:48.760 align:middle line:84% So knowing the object category tells me a lot about the color. 01:08:48.760 --> 01:08:49.660 align:middle line:90% Anything else? 01:08:49.660 --> 01:08:53.000 align:middle line:90% 01:08:53.000 --> 01:08:58.500 align:middle line:84% So, yes, it turns out objects is one of the answers. 01:08:58.500 --> 01:09:01.240 align:middle line:84% So here are our three different neurons. 01:09:01.240 --> 01:09:03.960 align:middle line:84% And this is looking at the feature maps. 01:09:03.960 --> 01:09:06.520 align:middle line:84% And we're just shading-- we're blacking out 01:09:06.520 --> 01:09:08.120 align:middle line:84% all the parts of the feature maps 01:09:08.120 --> 01:09:10.260 align:middle line:84% where the activations are below a threshold. 01:09:10.260 --> 01:09:12.279 align:middle line:84% So this is like, where are the neurons firing 01:09:12.279 --> 01:09:15.760 align:middle line:84% at some convolutional layer 5 of this network? 01:09:15.760 --> 01:09:17.920 align:middle line:84% And, yeah, on lower layers, it is textures 01:09:17.920 --> 01:09:19.060 align:middle line:90% and other things like this. 01:09:19.060 --> 01:09:21.620 align:middle line:84% But the interesting thing is that it doesn't really matter 01:09:21.620 --> 01:09:22.880 align:middle line:90% how you train these networks. 01:09:22.880 --> 01:09:25.140 align:middle line:84% If you train them to predict classes, 01:09:25.140 --> 01:09:26.640 align:middle line:84% if you train them to predict colors, 01:09:26.640 --> 01:09:28.478 align:middle line:84% if you train them to inpaint missing pixels, 01:09:28.478 --> 01:09:30.020 align:middle line:84% just pick the right half of the image 01:09:30.020 --> 01:09:33.740 align:middle line:84% from the left half, the units that 01:09:33.740 --> 01:09:35.859 align:middle line:84% carve the world at its joints, that are predictive 01:09:35.859 --> 01:09:39.268 align:middle line:84% of everything, turn out to be objects and semantics 01:09:39.268 --> 01:09:40.560 align:middle line:90% and the words that humans have. 01:09:40.560 --> 01:09:41.840 align:middle line:84% So it's like words are not arbitrary. 01:09:41.840 --> 01:09:43.548 align:middle line:84% We have the words we have because they're 01:09:43.548 --> 01:09:46.779 align:middle line:84% very predictive statistically of missing data. 01:09:46.779 --> 01:09:49.859 align:middle line:84% So this is discovery of semantic words 01:09:49.859 --> 01:09:53.160 align:middle line:90% without any semantic labels. 01:09:53.160 --> 01:09:54.620 align:middle line:84% And this has been the big finding 01:09:54.620 --> 01:09:57.900 align:middle line:84% over the last decade that led to this revolution in how 01:09:57.900 --> 01:09:59.620 align:middle line:84% we do deep learning, which was the move 01:09:59.620 --> 01:10:02.340 align:middle line:84% from supervised learning to self-supervised learning. 01:10:02.340 --> 01:10:04.603 align:middle line:84% So the self-supervised learning is-- 01:10:04.603 --> 01:10:06.020 align:middle line:84% we used to try to do what's called 01:10:06.020 --> 01:10:08.420 align:middle line:84% unsupervised learning, which is just 01:10:08.420 --> 01:10:10.160 align:middle line:90% learn from raw, unlabeled data. 01:10:10.160 --> 01:10:13.500 align:middle line:84% But then we switch it into the mathematical machinery 01:10:13.500 --> 01:10:17.740 align:middle line:84% of supervised learning by just predicting some fake labels, 01:10:17.740 --> 01:10:19.990 align:middle line:84% which are just raw data, which you call pretext labels 01:10:19.990 --> 01:10:23.630 align:middle line:84% or pretext tasks from the data itself. 01:10:23.630 --> 01:10:25.670 align:middle line:84% And there's so many different ways 01:10:25.670 --> 01:10:27.790 align:middle line:84% of doing this learning by prediction. 01:10:27.790 --> 01:10:29.430 align:middle line:84% So you could just predict class labels. 01:10:29.430 --> 01:10:32.310 align:middle line:84% That would be called supervised learning of representations. 01:10:32.310 --> 01:10:35.690 align:middle line:84% You could predict the next frame in a video, the future. 01:10:35.690 --> 01:10:38.910 align:middle line:84% You could predict the next pixel in a sequence and the same thing 01:10:38.910 --> 01:10:41.550 align:middle line:84% for any other data modality, too. 01:10:41.550 --> 01:10:44.990 align:middle line:84% So the two on the right are self-supervised because no human 01:10:44.990 --> 01:10:47.012 align:middle line:90% had to provide the label target. 01:10:47.012 --> 01:10:48.970 align:middle line:84% It's the next frame in the video, the raw data. 01:10:48.970 --> 01:10:51.710 align:middle line:84% It's the next pixel in a sequence. 01:10:51.710 --> 01:10:54.470 align:middle line:84% And as you probably know, the way that language models work 01:10:54.470 --> 01:10:57.330 align:middle line:84% is they predict the next word in a sequence, 01:10:57.330 --> 01:11:01.110 align:middle line:84% so they're mostly in the family of this type of learning. 01:11:01.110 --> 01:11:03.910 align:middle line:84% And most people still call language models 01:11:03.910 --> 01:11:07.570 align:middle line:84% self-supervised because they're just predicting the raw data. 01:11:07.570 --> 01:11:09.770 align:middle line:84% It happens the raw data is semantic and is words, 01:11:09.770 --> 01:11:11.888 align:middle line:84% but in that context, it's not like they're 01:11:11.888 --> 01:11:12.930 align:middle line:90% predicting the sentiment. 01:11:12.930 --> 01:11:15.310 align:middle line:84% They're just predicting the next word. 01:11:15.310 --> 01:11:19.610 align:middle line:84% So all of these self-supervised tasks 01:11:19.610 --> 01:11:22.610 align:middle line:84% can be understood as something we call imputation, which just 01:11:22.610 --> 01:11:26.450 align:middle line:84% means take your data-- it could be a matrix, a tensor, some data 01:11:26.450 --> 01:11:26.990 align:middle line:90% object. 01:11:26.990 --> 01:11:29.470 align:middle line:84% A video would be like time by x by y. 01:11:29.470 --> 01:11:32.590 align:middle line:84% A sentence would be time by the content at that time, 01:11:32.590 --> 01:11:34.090 align:middle line:90% the word at that time. 01:11:34.090 --> 01:11:39.210 align:middle line:84% And mask part of it, and put that into an encoder. 01:11:39.210 --> 01:11:42.530 align:middle line:84% Decode it to predict the masked part of that input. 01:11:42.530 --> 01:11:44.490 align:middle line:84% So masked prediction or imputation 01:11:44.490 --> 01:11:48.370 align:middle line:84% is the standard pretext tasks that people 01:11:48.370 --> 01:11:51.090 align:middle line:84% like to use these days to learn representations. 01:11:51.090 --> 01:11:55.690 align:middle line:84% So spatial imputation, just predict the next pixel 01:11:55.690 --> 01:11:56.750 align:middle line:90% from the previous pixel. 01:11:56.750 --> 01:11:58.583 align:middle line:84% Temporal imputation-- predict the next frame 01:11:58.583 --> 01:12:00.090 align:middle line:90% from the previous frame. 01:12:00.090 --> 01:12:03.930 align:middle line:84% Channel imputation-- predict the colors from the black and white 01:12:03.930 --> 01:12:06.150 align:middle line:84% and, again, on other modalities, you can do the same. 01:12:06.150 --> 01:12:11.050 align:middle line:90% 01:12:11.050 --> 01:12:14.123 align:middle line:84% Yeah, you can do this in a way that frames it 01:12:14.123 --> 01:12:16.540 align:middle line:84% just like an autoencoder, again, but it will be called now 01:12:16.540 --> 01:12:17.640 align:middle line:90% a masked autoencoder. 01:12:17.640 --> 01:12:22.220 align:middle line:84% So masked autoencoder is you take your data, 01:12:22.220 --> 01:12:25.660 align:middle line:90% you mask random chunks of it. 01:12:25.660 --> 01:12:29.460 align:middle line:84% And the little interesting trick is that in masked autoencoder 01:12:29.460 --> 01:12:31.340 align:middle line:90% is applied to images. 01:12:31.340 --> 01:12:33.240 align:middle line:84% You're using a vision transformer. 01:12:33.240 --> 01:12:34.900 align:middle line:84% And the vision transformer already 01:12:34.900 --> 01:12:37.705 align:middle line:84% tokenized the image into the first-- remember, 01:12:37.705 --> 01:12:39.080 align:middle line:84% we talked about vision transform. 01:12:39.080 --> 01:12:41.380 align:middle line:84% We said that we are going to create a set of tokens 01:12:41.380 --> 01:12:43.060 align:middle line:84% by chopping the image into patches 01:12:43.060 --> 01:12:47.140 align:middle line:84% and then mapping those patches to vector embeddings or tokens 01:12:47.140 --> 01:12:48.188 align:middle line:90% that then get processed. 01:12:48.188 --> 01:12:49.980 align:middle line:84% And so they just said, well, what if I just 01:12:49.980 --> 01:12:51.700 align:middle line:84% remove some of the tokens, and then I 01:12:51.700 --> 01:12:52.840 align:middle line:90% predict the missing tokens? 01:12:52.840 --> 01:12:58.140 align:middle line:84% And the really interesting thing about the attention architecture 01:12:58.140 --> 01:13:01.640 align:middle line:84% is that I can mask different ratios. 01:13:01.640 --> 01:13:04.200 align:middle line:84% I can only keep four of these tokens, 01:13:04.200 --> 01:13:07.300 align:middle line:84% but then the attention mechanism will 01:13:07.300 --> 01:13:09.500 align:middle line:84% scale in a way that is proportional to the number 01:13:09.500 --> 01:13:10.368 align:middle line:90% of tokens. 01:13:10.368 --> 01:13:12.160 align:middle line:84% Each token will attend to each other token. 01:13:12.160 --> 01:13:13.880 align:middle line:84% So if I have four, it will be four attending to four. 01:13:13.880 --> 01:13:15.700 align:middle line:84% And it will output predictions for four. 01:13:15.700 --> 01:13:17.908 align:middle line:84% If I have eight, it will be eight attending to eight. 01:13:17.908 --> 01:13:21.160 align:middle line:84% So it has this nice kind of architectural invariance 01:13:21.160 --> 01:13:23.560 align:middle line:84% to the number of tokens you put in. 01:13:23.560 --> 01:13:24.960 align:middle line:90% And then I decode. 01:13:24.960 --> 01:13:28.360 align:middle line:84% I just have some blank tokens I put into another transformer. 01:13:28.360 --> 01:13:31.760 align:middle line:84% And then I have trained it so that those blank tokens get 01:13:31.760 --> 01:13:36.920 align:middle line:84% filled in with the prediction of the pixels in the missing 01:13:36.920 --> 01:13:39.800 align:middle line:84% tokens, or the prediction of the data in the missing tokens. 01:13:39.800 --> 01:13:42.440 align:middle line:84% And masked autoencoders are just a new name 01:13:42.440 --> 01:13:47.440 align:middle line:84% for another model which was very popular called BERT. 01:13:47.440 --> 01:13:49.357 align:middle line:90% So there are differences. 01:13:49.357 --> 01:13:51.440 align:middle line:84% Engineering this to work on language versus vision 01:13:51.440 --> 01:13:53.815 align:middle line:84% is a very important difference but, conceptually, they're 01:13:53.815 --> 01:13:55.040 align:middle line:90% almost the same thing. 01:13:55.040 --> 01:13:57.640 align:middle line:84% So BERT was a language model that was very popular 01:13:57.640 --> 01:13:58.760 align:middle line:90% some years ago. 01:13:58.760 --> 01:14:01.320 align:middle line:84% It's gone a bit out of fashion, but BERT is 01:14:01.320 --> 01:14:04.220 align:middle line:90% roughly the same thing on text. 01:14:04.220 --> 01:14:05.420 align:middle line:90% So I just take my text. 01:14:05.420 --> 01:14:06.140 align:middle line:90% I tokenize it. 01:14:06.140 --> 01:14:07.620 align:middle line:90% I mask some of the tokens. 01:14:07.620 --> 01:14:10.500 align:middle line:84% I run it through a transformer, and I predict all the tokens. 01:14:10.500 --> 01:14:14.130 align:middle line:84% So I'm now doing masked prediction with language. 01:14:14.130 --> 01:14:18.250 align:middle line:84% And the autoregressive models that 01:14:18.250 --> 01:14:20.210 align:middle line:84% try to predict the next word in a sentence 01:14:20.210 --> 01:14:21.850 align:middle line:84% are just the same, except they're only 01:14:21.850 --> 01:14:24.730 align:middle line:84% masking the final word as opposed to interleaving words. 01:14:24.730 --> 01:14:26.450 align:middle line:84% And that has some advantages in that 01:14:26.450 --> 01:14:29.450 align:middle line:84% I can decode in sequential order as opposed 01:14:29.450 --> 01:14:33.530 align:middle line:90% to in out-of-time order. 01:14:33.530 --> 01:14:36.130 align:middle line:90% So, yeah, question? 01:14:36.130 --> 01:14:38.587 align:middle line:84% AUDIENCE: Why is BERT getting out of fashion? 01:14:38.587 --> 01:14:40.670 align:middle line:84% PHILLIP ISOLA: Why is BERT getting out of fashion? 01:14:40.670 --> 01:14:46.210 align:middle line:84% So I think it's because masking the final token, naturally, 01:14:46.210 --> 01:14:49.810 align:middle line:84% can be used for generating sentences autoregressively, 01:14:49.810 --> 01:14:51.690 align:middle line:90% which we talked about before. 01:14:51.690 --> 01:14:53.690 align:middle line:84% If I mask the interleaving, then it's 01:14:53.690 --> 01:14:57.310 align:middle line:84% like I'm generating words out of temporal order. 01:14:57.310 --> 01:14:59.730 align:middle line:84% So if I'm talking with a human, time 01:14:59.730 --> 01:15:03.630 align:middle line:84% is an axis that is important and not symmetric with other axes. 01:15:03.630 --> 01:15:06.450 align:middle line:84% I have to answer the question after the question has 01:15:06.450 --> 01:15:08.650 align:middle line:84% been asked by the person I'm talking to. 01:15:08.650 --> 01:15:11.580 align:middle line:84% And so having these causal attention 01:15:11.580 --> 01:15:13.720 align:middle line:84% mechanisms for the masking is in causal order. 01:15:13.720 --> 01:15:16.320 align:middle line:84% I'm only masking the future in conditioning on the past. 01:15:16.320 --> 01:15:18.360 align:middle line:84% It just fits into language models, 01:15:18.360 --> 01:15:20.420 align:middle line:90% which operate in causal order. 01:15:20.420 --> 01:15:22.980 align:middle line:84% And once that became popular, the biggest models 01:15:22.980 --> 01:15:26.500 align:middle line:84% were all masking only the tokens in the future, given 01:15:26.500 --> 01:15:27.900 align:middle line:90% the tokens in the past. 01:15:27.900 --> 01:15:30.077 align:middle line:84% And because those were the biggest models, 01:15:30.077 --> 01:15:31.160 align:middle line:90% they just worked the best. 01:15:31.160 --> 01:15:32.960 align:middle line:84% But if I want to learn a sentence embedding, 01:15:32.960 --> 01:15:34.140 align:middle line:84% I bet the BERT method is still going 01:15:34.140 --> 01:15:35.848 align:middle line:84% to work better if scaled the same amount. 01:15:35.848 --> 01:15:40.140 align:middle line:90% 01:15:40.140 --> 01:15:44.100 align:middle line:84% OK, so here's an interesting empirical finding. 01:15:44.100 --> 01:15:46.120 align:middle line:84% This is going back to that colorization paper, 01:15:46.120 --> 01:15:48.600 align:middle line:84% but it's been shown in a lot of work, 01:15:48.600 --> 01:15:53.060 align:middle line:84% which is that masked prediction just tends to always work better 01:15:53.060 --> 01:15:55.360 align:middle line:90% than autoencoding. 01:15:55.360 --> 01:15:57.720 align:middle line:84% So if I I'm going to look at the accuracy, 01:15:57.720 --> 01:16:00.460 align:middle line:84% I'm going to look at the representations 01:16:00.460 --> 01:16:04.020 align:middle line:84% at each layer of an autoencoder versus a masked prediction 01:16:04.020 --> 01:16:06.120 align:middle line:84% network, for colorization in this case. 01:16:06.120 --> 01:16:09.740 align:middle line:84% And I'm going to assess performance 01:16:09.740 --> 01:16:11.560 align:middle line:84% seeing how well I can linearly classify 01:16:11.560 --> 01:16:13.400 align:middle line:84% given the representation on that layer. 01:16:13.400 --> 01:16:15.700 align:middle line:84% Imagenet categories, in this case, is like a probe, 01:16:15.700 --> 01:16:17.520 align:middle line:84% just like we were doing with the shapes, where we had the nearest 01:16:17.520 --> 01:16:18.145 align:middle line:90% neighbor probe. 01:16:18.145 --> 01:16:20.200 align:middle line:84% Now we have a linear classifier probe. 01:16:20.200 --> 01:16:21.220 align:middle line:90% So here's what happens. 01:16:21.220 --> 01:16:22.760 align:middle line:84% As I go deeper in the network, you 01:16:22.760 --> 01:16:24.960 align:middle line:84% get this separation, where autoencoding 01:16:24.960 --> 01:16:27.960 align:middle line:84% learns an OK representation, but masked 01:16:27.960 --> 01:16:30.120 align:middle line:84% prediction learns a representation which 01:16:30.120 --> 01:16:31.180 align:middle line:90% is more semantic. 01:16:31.180 --> 01:16:35.280 align:middle line:84% It linearly decodes ImageNet categories better. 01:16:35.280 --> 01:16:38.880 align:middle line:84% And this is something that I thought I had a good explanation 01:16:38.880 --> 01:16:40.935 align:middle line:84% for, and then I realized I don't. 01:16:40.935 --> 01:16:42.560 align:middle line:84% So I'm going to call it ongoing science 01:16:42.560 --> 01:16:44.700 align:middle line:84% and leave it as a puzzle for the class. 01:16:44.700 --> 01:16:46.680 align:middle line:84% And maybe this would be a good final project. 01:16:46.680 --> 01:16:50.840 align:middle line:84% Why does masked prediction work better than autoencoding? 01:16:50.840 --> 01:16:52.740 align:middle line:84% So here are three hypotheses to think about. 01:16:52.740 --> 01:16:57.960 align:middle line:84% So one is that autoencoding controls compression 01:16:57.960 --> 01:17:00.280 align:middle line:90% via dimensional bottleneck. 01:17:00.280 --> 01:17:03.640 align:middle line:84% Masked prediction controls compression 01:17:03.640 --> 01:17:07.040 align:middle line:84% by the non-overlap between the outputs you're predicting 01:17:07.040 --> 01:17:10.033 align:middle line:84% and the inputs you're conditioning on. 01:17:10.033 --> 01:17:12.450 align:middle line:84% So the only thing that's useful for predicting the outputs 01:17:12.450 --> 01:17:14.283 align:middle line:84% is the mutual information between the inputs 01:17:14.283 --> 01:17:14.950 align:middle line:90% and the outputs. 01:17:14.950 --> 01:17:16.428 align:middle line:84% And if these are separate, then I 01:17:16.428 --> 01:17:17.970 align:middle line:84% will just forget all the stuff that's 01:17:17.970 --> 01:17:21.190 align:middle line:84% specific to the inputs that's not relevant for the output. 01:17:21.190 --> 01:17:23.950 align:middle line:84% So that's how it does compression. 01:17:23.950 --> 01:17:26.370 align:middle line:84% So it could be that just controlling compression 01:17:26.370 --> 01:17:29.330 align:middle line:84% via dimensional bottlenecks is really hard to make work. 01:17:29.330 --> 01:17:30.190 align:middle line:90% It's very finicky. 01:17:30.190 --> 01:17:31.715 align:middle line:84% It requires an architecture that has 01:17:31.715 --> 01:17:33.090 align:middle line:84% constraints on the dimensionality 01:17:33.090 --> 01:17:34.250 align:middle line:90% of the embeddings. 01:17:34.250 --> 01:17:37.470 align:middle line:84% And we know that low-dimensional things, 01:17:37.470 --> 01:17:41.130 align:middle line:84% anything with low dimension and deep learning just is hard. 01:17:41.130 --> 01:17:44.890 align:middle line:84% It interacts with optimization in weird ways and interacts 01:17:44.890 --> 01:17:47.070 align:middle line:84% with BatchNorm and LayerNorm in weird ways. 01:17:47.070 --> 01:17:49.602 align:middle line:84% So low-dimensional embeddings have some bad properties, 01:17:49.602 --> 01:17:51.810 align:middle line:84% and maybe it's just really hard to use dimensionality 01:17:51.810 --> 01:17:55.530 align:middle line:84% of the embeddings as the way to learn a good representation. 01:17:55.530 --> 01:18:00.730 align:middle line:84% Hypothesis two is autoencoders are learning shortcuts. 01:18:00.730 --> 01:18:03.888 align:middle line:84% So in order for me to reconstruct the signal-- 01:18:03.888 --> 01:18:04.430 align:middle line:90% imagine this. 01:18:04.430 --> 01:18:06.910 align:middle line:84% What if I had an autoencoder with skip connections, 01:18:06.910 --> 01:18:08.210 align:middle line:90% with residual connections? 01:18:08.210 --> 01:18:10.910 align:middle line:90% Would that be a good idea? 01:18:10.910 --> 01:18:13.392 align:middle line:84% What would be wrong with that, autoencoder f, 01:18:13.392 --> 01:18:15.350 align:middle line:84% g, but they just have some residual connections 01:18:15.350 --> 01:18:17.513 align:middle line:90% in between f and g? 01:18:17.513 --> 01:18:19.930 align:middle line:84% AUDIENCE: Not forcing this low-dimensional representation. 01:18:19.930 --> 01:18:20.030 align:middle line:90% PHILLIP ISOLA: Yeah. 01:18:20.030 --> 01:18:21.453 align:middle line:90% It skips the bottleneck. 01:18:21.453 --> 01:18:23.370 align:middle line:84% So there's all these little gotchas like that. 01:18:23.370 --> 01:18:24.953 align:middle line:84% And it could be that autoencoders just 01:18:24.953 --> 01:18:30.390 align:middle line:84% have this tendency to copy the local information that they're 01:18:30.390 --> 01:18:31.130 align:middle line:90% processing. 01:18:31.130 --> 01:18:33.170 align:middle line:84% And that will be a decent solution, 01:18:33.170 --> 01:18:35.543 align:middle line:84% even though it would be better for them 01:18:35.543 --> 01:18:37.710 align:middle line:84% to capture global properties in order to reconstruct 01:18:37.710 --> 01:18:39.990 align:middle line:90% the entire signal. 01:18:39.990 --> 01:18:41.435 align:middle line:84% And then maybe math prediction is 01:18:41.435 --> 01:18:43.810 align:middle line:84% closer to the prediction problems we actually care about. 01:18:43.810 --> 01:18:45.730 align:middle line:84% So somehow it's closer to the actual use case. 01:18:45.730 --> 01:18:47.980 align:middle line:84% We're going to try to predict the future from the past 01:18:47.980 --> 01:18:49.390 align:middle line:90% or something like this. 01:18:49.390 --> 01:18:50.830 align:middle line:90% So I don't know. 01:18:50.830 --> 01:18:56.462 align:middle line:84% I asked Kaiming, who's a professor here who was the first 01:18:56.462 --> 01:18:58.170 align:middle line:84% author of "The Masked Autoencoder" paper. 01:18:58.170 --> 01:19:00.890 align:middle line:84% And he said, well, at the end of the day, it's just empiricism. 01:19:00.890 --> 01:19:04.410 align:middle line:84% So mass prediction works better than autoencoding. 01:19:04.410 --> 01:19:07.160 align:middle line:84% And I think it's a really interesting question, 01:19:07.160 --> 01:19:09.080 align:middle line:84% theoretically why that should be the case. 01:19:09.080 --> 01:19:09.720 align:middle line:90% Yeah? 01:19:09.720 --> 01:19:11.678 align:middle line:84% AUDIENCE: [INAUDIBLE] beginning of the lecture, 01:19:11.678 --> 01:19:14.480 align:middle line:84% you said that in the scope of representation learning, 01:19:14.480 --> 01:19:19.320 align:middle line:84% you think this framework of [INAUDIBLE] is simplest 01:19:19.320 --> 01:19:20.225 align:middle line:90% and will win out. 01:19:20.225 --> 01:19:21.600 align:middle line:84% I was wondering if you could talk 01:19:21.600 --> 01:19:24.640 align:middle line:90% about why you think that is? 01:19:24.640 --> 01:19:26.480 align:middle line:84% PHILLIP ISOLA: Yeah, so ultimately, it just 01:19:26.480 --> 01:19:28.337 align:middle line:84% goes back to some first principle argument 01:19:28.337 --> 01:19:29.420 align:middle line:90% that I can't really prove. 01:19:29.420 --> 01:19:34.300 align:middle line:84% But there are arguments along the lines of Occam's razor 01:19:34.300 --> 01:19:39.160 align:middle line:84% and formalisms of that the compression is somehow 01:19:39.160 --> 01:19:40.977 align:middle line:90% equivalent to prediction. 01:19:40.977 --> 01:19:42.560 align:middle line:84% And the most compressed representation 01:19:42.560 --> 01:19:45.240 align:middle line:84% will make the most accurate predictions about the future. 01:19:45.240 --> 01:19:47.760 align:middle line:84% And autoencoders are just a simple embodiment 01:19:47.760 --> 01:19:51.360 align:middle line:84% of the idea of take your data, compress it as much as possible, 01:19:51.360 --> 01:19:54.500 align:middle line:90% and remove all redundancies. 01:19:54.500 --> 01:19:58.260 align:middle line:84% And it just seems, from first principles, 01:19:58.260 --> 01:19:59.760 align:middle line:84% if compression is all you need, then 01:19:59.760 --> 01:20:01.400 align:middle line:90% autoencoders are all you need. 01:20:01.400 --> 01:20:03.025 align:middle line:84% I think there's more to say about that. 01:20:03.025 --> 01:20:05.140 align:middle line:90% I don't have time right now. 01:20:05.140 --> 01:20:06.580 align:middle line:84% So, yeah, still an open question. 01:20:06.580 --> 01:20:10.060 align:middle line:84% I'll end with I think this is a nice idea that Yann LeCun put 01:20:10.060 --> 01:20:12.340 align:middle line:84% out some years ago, which is that intelligence 01:20:12.340 --> 01:20:13.780 align:middle line:90% is like this cake. 01:20:13.780 --> 01:20:18.260 align:middle line:84% A lot of you may have seen this, where the bulk of the cake 01:20:18.260 --> 01:20:20.392 align:middle line:90% is representation learning. 01:20:20.392 --> 01:20:22.100 align:middle line:84% You spend a lot of data and a lot of time 01:20:22.100 --> 01:20:23.892 align:middle line:84% coming up with a really good representation 01:20:23.892 --> 01:20:26.620 align:middle line:84% of the world in a self-supervised way 01:20:26.620 --> 01:20:29.160 align:middle line:84% or an unsupervised way, not tied to any specific task. 01:20:29.160 --> 01:20:31.867 align:middle line:84% So self-supervision is about not having labels, 01:20:31.867 --> 01:20:34.200 align:middle line:84% but it's also about not being narrow and tied to a task. 01:20:34.200 --> 01:20:36.380 align:middle line:84% It's just generally compress the universe 01:20:36.380 --> 01:20:40.525 align:middle line:84% into something that's predictive of the future and is compact. 01:20:40.525 --> 01:20:41.900 align:middle line:84% And then the icing and the cherry 01:20:41.900 --> 01:20:43.620 align:middle line:84% are all the rest of machine learning-- supervised 01:20:43.620 --> 01:20:45.860 align:middle line:84% learning, adaptation, post-training, reinforcement 01:20:45.860 --> 01:20:46.487 align:middle line:90% learning. 01:20:46.487 --> 01:20:48.820 align:middle line:84% So he's making the point that representation learning is 01:20:48.820 --> 01:20:51.080 align:middle line:84% the bulk of intelligence, and I agree with that point. 01:20:51.080 --> 01:20:53.060 align:middle line:90% So I will end there. 01:20:53.060 --> 01:20:55.410 align:middle line:90% Summary-- you can read online. 01:20:55.410 --> 01:21:04.000 align:middle line:90%