WEBVTT 00:00:00.000 --> 00:00:01.970 align:middle line:90% [SQUEAKING] 00:00:01.970 --> 00:00:03.930 align:middle line:90% [RUSTLING] 00:00:03.930 --> 00:00:07.360 align:middle line:90% [CLICKING] 00:00:07.360 --> 00:00:13.060 align:middle line:90% 00:00:13.060 --> 00:00:15.040 align:middle line:84% SARA BEERY: So, today, we're going 00:00:15.040 --> 00:00:18.520 align:middle line:84% to start off the first of a series of lectures 00:00:18.520 --> 00:00:21.400 align:middle line:84% on machine learning architectures. 00:00:21.400 --> 00:00:27.760 align:middle line:84% And so moving beyond the very vanilla architecture 00:00:27.760 --> 00:00:29.360 align:middle line:84% we've been talking about so far, which 00:00:29.360 --> 00:00:31.480 align:middle line:84% is the multilayer perceptron, and starting 00:00:31.480 --> 00:00:33.840 align:middle line:84% to discuss different ways that we 00:00:33.840 --> 00:00:37.160 align:middle line:84% can encode structure or essentially build hypotheses 00:00:37.160 --> 00:00:41.000 align:middle line:84% about the space of models that we want to optimize over 00:00:41.000 --> 00:00:43.600 align:middle line:84% directly into the architecture of the model 00:00:43.600 --> 00:00:46.320 align:middle line:90% that we're going to optimize. 00:00:46.320 --> 00:00:48.540 align:middle line:84% So what that's going to look like today-- first, 00:00:48.540 --> 00:00:51.760 align:middle line:84% we're going to have a broader discussion about why 00:00:51.760 --> 00:00:53.560 align:middle line:84% we want to build better architectures 00:00:53.560 --> 00:00:56.400 align:middle line:84% and what makes an architecture better. 00:00:56.400 --> 00:00:59.000 align:middle line:84% And then we're going to do a bit of a deep dive 00:00:59.000 --> 00:01:03.420 align:middle line:84% on convolutional layers as one specific mechanism you can build 00:01:03.420 --> 00:01:05.820 align:middle line:84% into an architecture, one that works particularly 00:01:05.820 --> 00:01:07.860 align:middle line:90% well for grids. 00:01:07.860 --> 00:01:09.680 align:middle line:84% And this might be a review for some of you, 00:01:09.680 --> 00:01:12.705 align:middle line:84% but we're still going to spend some time on it. 00:01:12.705 --> 00:01:14.580 align:middle line:84% Then we're going to talk a bit about pyramids 00:01:14.580 --> 00:01:16.900 align:middle line:90% and the idea of receptive field. 00:01:16.900 --> 00:01:19.700 align:middle line:84% Then I'm going to go through a bit of a high-level overview 00:01:19.700 --> 00:01:22.460 align:middle line:84% of a bunch of different architectures 00:01:22.460 --> 00:01:24.580 align:middle line:84% and architecture components that have 00:01:24.580 --> 00:01:28.780 align:middle line:84% made up a lot of the progress in the last 10 years or so. 00:01:28.780 --> 00:01:32.140 align:middle line:84% And then I'm going to finish with a first touch 00:01:32.140 --> 00:01:34.240 align:middle line:84% on neural fields and positional encoding. 00:01:34.240 --> 00:01:36.380 align:middle line:84% So we'll talk a lot more about this 00:01:36.380 --> 00:01:39.980 align:middle line:84% when we get into lectures on transformers later on. 00:01:39.980 --> 00:01:44.100 align:middle line:84% So this is what we've been talking about so far, 00:01:44.100 --> 00:01:46.500 align:middle line:90% the multilayer perceptron. 00:01:46.500 --> 00:01:50.340 align:middle line:84% So this is, at a high level, the building block 00:01:50.340 --> 00:01:52.740 align:middle line:84% is this linear combination of neurons 00:01:52.740 --> 00:01:55.220 align:middle line:84% or linear combination of inputs, a neuron 00:01:55.220 --> 00:01:57.540 align:middle line:84% wise or a point-wise non-linearity and then 00:01:57.540 --> 00:01:59.640 align:middle line:90% another linear combination. 00:01:59.640 --> 00:02:02.320 align:middle line:84% And one of the benefits of this, like got described 00:02:02.320 --> 00:02:04.640 align:middle line:84% in the last lecture, is the fact that it can-- 00:02:04.640 --> 00:02:06.180 align:middle line:90% it's a universal approximator. 00:02:06.180 --> 00:02:08.039 align:middle line:84% You can build any model and prove 00:02:08.039 --> 00:02:12.560 align:middle line:84% that you can build any model with this type of architecture. 00:02:12.560 --> 00:02:17.640 align:middle line:84% Another benefit of a multi-layer perceptron is it's quite simple. 00:02:17.640 --> 00:02:19.320 align:middle line:84% And one of the things this simplicity 00:02:19.320 --> 00:02:22.240 align:middle line:84% gives you is the ability to write really elegant theory 00:02:22.240 --> 00:02:22.980 align:middle line:90% about this model. 00:02:22.980 --> 00:02:24.160 align:middle line:84% And this is one of the only models 00:02:24.160 --> 00:02:26.080 align:middle line:84% that we actually have really elegant theory 00:02:26.080 --> 00:02:28.873 align:middle line:84% for because the complexity is such 00:02:28.873 --> 00:02:30.540 align:middle line:84% that we can make reasonable assumptions, 00:02:30.540 --> 00:02:35.240 align:middle line:84% it maps well into our mathematical tool blocks. 00:02:35.240 --> 00:02:38.360 align:middle line:84% And then another huge benefit of the multi-layer perceptron 00:02:38.360 --> 00:02:42.360 align:middle line:84% is it's what we call embarrassingly parallel. 00:02:42.360 --> 00:02:45.600 align:middle line:84% Essentially, this means that this type of architecture 00:02:45.600 --> 00:02:47.440 align:middle line:84% can be really easily parallelized 00:02:47.440 --> 00:02:49.320 align:middle line:84% in terms of computation, which means 00:02:49.320 --> 00:02:51.520 align:middle line:84% we can make use of things like GPUs 00:02:51.520 --> 00:02:54.660 align:middle line:84% that enable a lot of parallel multiplications. 00:02:54.660 --> 00:02:57.100 align:middle line:84% So one obvious example of that is 00:02:57.100 --> 00:03:00.460 align:middle line:84% that that neuron-wise or that point-wise non-linearity, that's 00:03:00.460 --> 00:03:03.340 align:middle line:84% calculated independently for every single output 00:03:03.340 --> 00:03:04.520 align:middle line:90% neuron of that first layer. 00:03:04.520 --> 00:03:06.437 align:middle line:84% And so that can happen completely in parallel. 00:03:06.437 --> 00:03:09.300 align:middle line:84% There's no dependence across the data points. 00:03:09.300 --> 00:03:13.300 align:middle line:84% Another nice example is that linear combination of neurons, 00:03:13.300 --> 00:03:16.740 align:middle line:84% that densely connected layer or that weight, 00:03:16.740 --> 00:03:19.900 align:middle line:84% that can happen in parallel across a very large, 00:03:19.900 --> 00:03:21.740 align:middle line:90% in some cases, batch of data. 00:03:21.740 --> 00:03:25.140 align:middle line:84% That's done independently per data point. 00:03:25.140 --> 00:03:26.580 align:middle line:84% So what are some of the bad things 00:03:26.580 --> 00:03:28.900 align:middle line:90% about the multilayer perceptron? 00:03:28.900 --> 00:03:34.160 align:middle line:84% So first one is that it has very weak inductive biases. 00:03:34.160 --> 00:03:38.640 align:middle line:84% And so what we mean by this is that the model itself, 00:03:38.640 --> 00:03:40.120 align:middle line:90% it's a universal approximator. 00:03:40.120 --> 00:03:43.420 align:middle line:84% But there's not a lot of structure or intuition baked 00:03:43.420 --> 00:03:45.140 align:middle line:90% into this model. 00:03:45.140 --> 00:03:48.820 align:middle line:84% It doesn't assume much about the structure of the model 00:03:48.820 --> 00:03:51.860 align:middle line:90% you want to learn coming out. 00:03:51.860 --> 00:03:53.820 align:middle line:84% And so one of the things that implies 00:03:53.820 --> 00:03:57.520 align:middle line:84% is that though you can use this model-- 00:03:57.520 --> 00:04:01.000 align:middle line:84% so you can use a multilayer perceptron to approximate any 00:04:01.000 --> 00:04:03.720 align:middle line:90% of mathematical function. 00:04:03.720 --> 00:04:09.120 align:middle line:84% It might be very difficult to do so given the data that you have. 00:04:09.120 --> 00:04:12.080 align:middle line:84% It's very sample inefficient or data hungry. 00:04:12.080 --> 00:04:14.360 align:middle line:84% This basically means that in order 00:04:14.360 --> 00:04:16.880 align:middle line:84% to get this multilayer perceptron to learn maybe 00:04:16.880 --> 00:04:21.279 align:middle line:84% a very complex function, you might require so much data 00:04:21.279 --> 00:04:22.960 align:middle line:84% that it's not actually tractable, 00:04:22.960 --> 00:04:25.000 align:middle line:84% like more data than exists on Earth to be 00:04:25.000 --> 00:04:27.160 align:middle line:84% able to really generalize and robustly 00:04:27.160 --> 00:04:31.480 align:middle line:84% learn a function with this particular architecture that 00:04:31.480 --> 00:04:33.440 align:middle line:90% generalizes well. 00:04:33.440 --> 00:04:36.520 align:middle line:84% And then another downside is that dense layers, those fully 00:04:36.520 --> 00:04:39.800 align:middle line:84% connected layers, particularly if you want a model that's 00:04:39.800 --> 00:04:42.480 align:middle line:84% very wide because maybe the function you need to approximate 00:04:42.480 --> 00:04:45.120 align:middle line:84% is quite complex, those can be really expensive. 00:04:45.120 --> 00:04:48.960 align:middle line:84% If your input data is an image, for example, a high resolution 00:04:48.960 --> 00:04:52.280 align:middle line:84% image that you might take on your iPhone today, 00:04:52.280 --> 00:04:55.510 align:middle line:84% then you need to flatten that out into one long vector. 00:04:55.510 --> 00:04:59.090 align:middle line:84% That's going to be thousands of input dimensions and 1,000 00:04:59.090 --> 00:05:02.450 align:middle line:84% by 1,000 multiplication in a dense way is a very expensive 00:05:02.450 --> 00:05:04.690 align:middle line:90% computational cost. 00:05:04.690 --> 00:05:08.672 align:middle line:90% So why use other architectures? 00:05:08.672 --> 00:05:11.130 align:middle line:84% Well, one, we might want to address some of those downsides 00:05:11.130 --> 00:05:12.370 align:middle line:90% that we talked about. 00:05:12.370 --> 00:05:18.330 align:middle line:84% But let's take a meta view of this and zoom out a little. 00:05:18.330 --> 00:05:21.730 align:middle line:84% So if this is the set of all possible mappings from X 00:05:21.730 --> 00:05:23.290 align:middle line:84% to Y-- so this is like all functions 00:05:23.290 --> 00:05:26.690 align:middle line:84% is contained in this little gray box. 00:05:26.690 --> 00:05:29.410 align:middle line:84% And maybe this is the true solution we're trying to learn. 00:05:29.410 --> 00:05:36.010 align:middle line:84% This is the correct model for our problem. 00:05:36.010 --> 00:05:40.290 align:middle line:84% Then possibly given any data set that we're 00:05:40.290 --> 00:05:42.490 align:middle line:84% going to learn from, that data set 00:05:42.490 --> 00:05:44.410 align:middle line:90% will constrain the space a bit. 00:05:44.410 --> 00:05:47.810 align:middle line:84% So not every model will fit the data 00:05:47.810 --> 00:05:50.170 align:middle line:84% that we're using to learn the data. 00:05:50.170 --> 00:05:53.190 align:middle line:84% And so now maybe you have this subset 00:05:53.190 --> 00:05:56.390 align:middle line:84% of all mappings that actually fits your training data. 00:05:56.390 --> 00:05:58.270 align:middle line:84% And then within that, based on something 00:05:58.270 --> 00:06:01.310 align:middle line:84% about your optimization scheme, you learn some solution, 00:06:01.310 --> 00:06:03.113 align:middle line:90% and it will be in that space. 00:06:03.113 --> 00:06:05.030 align:middle line:84% It will fit the data, but it won't necessarily 00:06:05.030 --> 00:06:08.510 align:middle line:84% be very close to the true solution. 00:06:08.510 --> 00:06:12.690 align:middle line:84% So what would happen to this high-level diagram, 00:06:12.690 --> 00:06:15.630 align:middle line:90% if we add more data? 00:06:15.630 --> 00:06:16.230 align:middle line:90% Anyone? 00:06:16.230 --> 00:06:18.580 align:middle line:90% What would happen? 00:06:18.580 --> 00:06:20.855 align:middle line:90% Mm-hmm? 00:06:20.855 --> 00:06:22.310 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:06:22.310 --> 00:06:23.447 align:middle line:90% 00:06:23.447 --> 00:06:24.530 align:middle line:90% SARA BEERY: Yeah, exactly. 00:06:24.530 --> 00:06:28.530 align:middle line:84% So that green space, that'll get smaller. 00:06:28.530 --> 00:06:30.790 align:middle line:84% There will be fewer models that fit all of the data 00:06:30.790 --> 00:06:31.930 align:middle line:90% if we have more data. 00:06:31.930 --> 00:06:33.770 align:middle line:90% So that's kind of nice. 00:06:33.770 --> 00:06:36.403 align:middle line:84% So if you can add more data, then potentially you 00:06:36.403 --> 00:06:38.070 align:middle line:84% shrink the space you're optimizing over, 00:06:38.070 --> 00:06:40.670 align:middle line:84% and then you'll learn a maybe more optimal solution 00:06:40.670 --> 00:06:42.790 align:middle line:84% or something that's closer to the true solution 00:06:42.790 --> 00:06:44.750 align:middle line:90% because the space is different. 00:06:44.750 --> 00:06:48.230 align:middle line:84% But what if you don't have more data? 00:06:48.230 --> 00:06:49.870 align:middle line:90% That's not an option for you. 00:06:49.870 --> 00:06:55.210 align:middle line:84% So then another alternative is to define a hypothesis space. 00:06:55.210 --> 00:06:58.170 align:middle line:84% And this is adding some structure into the model itself. 00:06:58.170 --> 00:07:00.130 align:middle line:84% And that hypothesis space is essentially 00:07:00.130 --> 00:07:03.810 align:middle line:84% going to shine a spotlight on the space of all possible models 00:07:03.810 --> 00:07:06.150 align:middle line:84% is the way we've kind of denoted it here. 00:07:06.150 --> 00:07:10.490 align:middle line:84% But essentially, it's a stronger inductive bias or a stronger 00:07:10.490 --> 00:07:14.370 align:middle line:84% prior over what we think the model is 00:07:14.370 --> 00:07:16.130 align:middle line:90% going to take the form of. 00:07:16.130 --> 00:07:19.010 align:middle line:84% And then if you have that intersection of the models that 00:07:19.010 --> 00:07:21.270 align:middle line:84% fit the data and your inductive hypothesis, 00:07:21.270 --> 00:07:26.070 align:middle line:84% then you might actually be able to use less data, 00:07:26.070 --> 00:07:28.690 align:middle line:84% but still find a better architecture. 00:07:28.690 --> 00:07:31.210 align:middle line:84% So the idea is if we can pin down 00:07:31.210 --> 00:07:33.610 align:middle line:84% or if we can get closer to the true model 00:07:33.610 --> 00:07:36.255 align:middle line:84% by either adding more data or by using 00:07:36.255 --> 00:07:37.630 align:middle line:84% a more constrained architecture-- 00:07:37.630 --> 00:07:39.755 align:middle line:84% and you can imagine that the combination of the two 00:07:39.755 --> 00:07:40.810 align:middle line:90% would be even better. 00:07:40.810 --> 00:07:42.950 align:middle line:90% So more data helps. 00:07:42.950 --> 00:07:45.850 align:middle line:84% Better architecture or stronger inductive bias helps. 00:07:45.850 --> 00:07:50.870 align:middle line:84% And those two things combined is ideal. 00:07:50.870 --> 00:07:56.070 align:middle line:84% So let's look at this from a very, very simplistic model. 00:07:56.070 --> 00:07:57.650 align:middle line:90% This is a one-dimensional model. 00:07:57.650 --> 00:08:00.670 align:middle line:84% It's a mapping of f of x, where x is a scalar. 00:08:00.670 --> 00:08:03.790 align:middle line:90% So here if we use, for example-- 00:08:03.790 --> 00:08:05.130 align:middle line:90% these are true outputs. 00:08:05.130 --> 00:08:08.870 align:middle line:84% If we use a 5-layer ReLU-net, so essentially densely connected 00:08:08.870 --> 00:08:13.270 align:middle line:84% layers with point-wise ReLU-based activations. 00:08:13.270 --> 00:08:16.280 align:middle line:84% And we fit that to a single data point. 00:08:16.280 --> 00:08:18.030 align:middle line:84% Well, now the space of all possible models 00:08:18.030 --> 00:08:20.530 align:middle line:84% that fit that data point is very, very large. 00:08:20.530 --> 00:08:23.590 align:middle line:84% And the one that actually gets learned is just this flat line. 00:08:23.590 --> 00:08:26.950 align:middle line:84% So that's not necessarily a very good learned solution. 00:08:26.950 --> 00:08:29.190 align:middle line:84% As we start adding more data, you 00:08:29.190 --> 00:08:31.590 align:middle line:84% can see that you maybe start to fit slightly 00:08:31.590 --> 00:08:35.049 align:middle line:84% better in the region where we add data in the middle, 00:08:35.049 --> 00:08:36.850 align:middle line:84% but it's still not a very good solution. 00:08:36.850 --> 00:08:38.887 align:middle line:84% And then as you start to add a lot of data, what 00:08:38.887 --> 00:08:40.929 align:middle line:84% you see on the right is you start getting really, 00:08:40.929 --> 00:08:43.510 align:middle line:84% really nice fits in the distribution where 00:08:43.510 --> 00:08:46.630 align:middle line:84% you have data, but you get really poor generalization 00:08:46.630 --> 00:08:48.210 align:middle line:90% out of distribution. 00:08:48.210 --> 00:08:54.825 align:middle line:84% And so the intuition here is really that model, hypothesis, 00:08:54.825 --> 00:08:56.450 align:middle line:84% that hypothesis space you build, that's 00:08:56.450 --> 00:08:58.650 align:middle line:84% really something that helps us generalize out 00:08:58.650 --> 00:09:01.010 align:middle line:84% of the distribution of the data we have. 00:09:01.010 --> 00:09:04.570 align:middle line:84% And so what if instead we had a really strong 00:09:04.570 --> 00:09:07.890 align:middle line:84% prior over what the model was supposed to be? 00:09:07.890 --> 00:09:11.710 align:middle line:84% So in this case, y equals ax plus sine bx squared. 00:09:11.710 --> 00:09:13.510 align:middle line:90% That's very specific. 00:09:13.510 --> 00:09:16.730 align:middle line:84% It turns out that it's exactly the correct model 00:09:16.730 --> 00:09:18.370 align:middle line:90% for this function. 00:09:18.370 --> 00:09:20.850 align:middle line:84% And you can see that if you only have a single data point, 00:09:20.850 --> 00:09:23.512 align:middle line:84% that really good model is insufficient. 00:09:23.512 --> 00:09:24.970 align:middle line:84% You still aren't able to figure out 00:09:24.970 --> 00:09:30.450 align:middle line:84% where in that set of hypotheses the correct model is. 00:09:30.450 --> 00:09:33.110 align:middle line:84% But even with very, very, very few data points, 00:09:33.110 --> 00:09:35.050 align:middle line:84% we start getting a really nice fit. 00:09:35.050 --> 00:09:37.752 align:middle line:84% And, actually, even when we add a lot more data points, 00:09:37.752 --> 00:09:39.210 align:middle line:84% we get really strong generalization 00:09:39.210 --> 00:09:40.030 align:middle line:90% out of distribution. 00:09:40.030 --> 00:09:41.410 align:middle line:84% But the difference between those two in the middle 00:09:41.410 --> 00:09:42.970 align:middle line:90% is actually not that big. 00:09:42.970 --> 00:09:45.890 align:middle line:84% So we can get a really nice fit even with just a little bit 00:09:45.890 --> 00:09:52.950 align:middle line:84% of data when we have the exact precise correct hypothesis. 00:09:52.950 --> 00:09:54.450 align:middle line:84% And so that's what we're showing. 00:09:54.450 --> 00:09:56.990 align:middle line:84% Architectures enable us to generalize outside the training 00:09:56.990 --> 00:09:58.230 align:middle line:90% distribution. 00:09:58.230 --> 00:09:59.950 align:middle line:84% And a good architecture is going to be 00:09:59.950 --> 00:10:03.650 align:middle line:84% one that can represent the true function of the data 00:10:03.650 --> 00:10:05.250 align:middle line:90% and then is otherwise minimal. 00:10:05.250 --> 00:10:07.350 align:middle line:84% We want it to be easy to search over 00:10:07.350 --> 00:10:09.030 align:middle line:90% via gradient-based learning. 00:10:09.030 --> 00:10:12.430 align:middle line:84% We want it to be easy to parallelize, fast on GPU, 00:10:12.430 --> 00:10:14.190 align:middle line:90% et cetera, et cetera. 00:10:14.190 --> 00:10:16.390 align:middle line:84% And so here's a bit of a preview. 00:10:16.390 --> 00:10:18.230 align:middle line:84% But it turns out that better architectures 00:10:18.230 --> 00:10:20.190 align:middle line:84% can approximate important function 00:10:20.190 --> 00:10:21.710 align:middle line:90% classes more efficiently. 00:10:21.710 --> 00:10:27.410 align:middle line:84% So this is a result from a paper by another MIT professor, 00:10:27.410 --> 00:10:31.150 align:middle line:84% Vincent Sitzmann, where they introduced this model over here 00:10:31.150 --> 00:10:33.990 align:middle line:90% on the far right called SIREN. 00:10:33.990 --> 00:10:39.030 align:middle line:84% And SIREN actually uses a sinusoidal activation space, 00:10:39.030 --> 00:10:41.410 align:middle line:84% and that fits images more efficiently. 00:10:41.410 --> 00:10:43.210 align:middle line:84% If anyone's taken signal processing, 00:10:43.210 --> 00:10:45.170 align:middle line:84% this idea might be a bit familiar. 00:10:45.170 --> 00:10:47.870 align:middle line:84% A Fourier basis is a good basis for representing images, 00:10:47.870 --> 00:10:50.650 align:middle line:84% and the fundamental building blocks of Fourier basis 00:10:50.650 --> 00:10:52.290 align:middle line:90% are sinusoids. 00:10:52.290 --> 00:10:54.930 align:middle line:84% So if we take a look at what this is, if you look at ReLU-net 00:10:54.930 --> 00:10:57.130 align:middle line:84% or a TanH-net or these different models 00:10:57.130 --> 00:10:58.470 align:middle line:90% that are being compared here. 00:10:58.470 --> 00:11:00.303 align:middle line:84% And we're going to look at how they actually 00:11:00.303 --> 00:11:02.650 align:middle line:84% try to fit this specific image over time 00:11:02.650 --> 00:11:04.450 align:middle line:84% by sampling pixels from that image 00:11:04.450 --> 00:11:07.890 align:middle line:84% and then fitting that as the training data. 00:11:07.890 --> 00:11:09.610 align:middle line:84% You can see that you start getting 00:11:09.610 --> 00:11:12.850 align:middle line:84% a pretty good approximation from SIREN 00:11:12.850 --> 00:11:16.010 align:middle line:84% much, much faster than you do from some of these other models. 00:11:16.010 --> 00:11:19.530 align:middle line:84% And, of course, after sampling enough points, 00:11:19.530 --> 00:11:22.090 align:middle line:84% eventually, all the models are getting 00:11:22.090 --> 00:11:24.070 align:middle line:90% reasonably good at estimating. 00:11:24.070 --> 00:11:26.850 align:middle line:84% But that SIREN model, it's a better hypothesis 00:11:26.850 --> 00:11:28.390 align:middle line:84% to start with for the image space. 00:11:28.390 --> 00:11:31.690 align:middle line:84% And so you get much more efficient convergence. 00:11:31.690 --> 00:11:34.270 align:middle line:84% And so in this particular instance, 00:11:34.270 --> 00:11:36.950 align:middle line:84% now instead of looking at a 5-layer ReLU-net, 00:11:36.950 --> 00:11:39.370 align:middle line:84% we're going to look at a 5-layer sin-net where now those 00:11:39.370 --> 00:11:42.690 align:middle line:84% activations, those nonlinear activations, are sinusoids. 00:11:42.690 --> 00:11:45.450 align:middle line:84% And you can see here that with a single data point, 00:11:45.450 --> 00:11:47.590 align:middle line:90% it still doesn't fit super well. 00:11:47.590 --> 00:11:49.870 align:middle line:84% But even just with a couple data points, 00:11:49.870 --> 00:11:52.510 align:middle line:84% we start seeing some periodicity that 00:11:52.510 --> 00:11:54.970 align:middle line:84% maybe starts to match the periodicity in the function. 00:11:54.970 --> 00:11:56.610 align:middle line:84% And even though we're not generalizing perfectly, 00:11:56.610 --> 00:11:58.230 align:middle line:84% we're still seeing some periodicity 00:11:58.230 --> 00:12:00.990 align:middle line:84% that seems to be moving more towards what 00:12:00.990 --> 00:12:02.130 align:middle line:90% the true function is. 00:12:02.130 --> 00:12:06.470 align:middle line:84% So why would you have that periodicity in the output space? 00:12:06.470 --> 00:12:08.010 align:middle line:84% It's kind of an obvious question. 00:12:08.010 --> 00:12:09.522 align:middle line:90% AUDIENCE: [INAUDIBLE] sine. 00:12:09.522 --> 00:12:10.730 align:middle line:90% SARA BEERY: Yes, it's a sine. 00:12:10.730 --> 00:12:12.190 align:middle line:90% Sines are periodic. 00:12:12.190 --> 00:12:14.310 align:middle line:90% Very nice. 00:12:14.310 --> 00:12:17.870 align:middle line:84% But so essentially, we've just added an inductive bias 00:12:17.870 --> 00:12:20.310 align:middle line:84% in our model architecture that says that the distribution 00:12:20.310 --> 00:12:21.732 align:middle line:90% should be periodic. 00:12:21.732 --> 00:12:23.190 align:middle line:84% And then it tries to find solutions 00:12:23.190 --> 00:12:26.390 align:middle line:84% that are periodic outside of the distribution where 00:12:26.390 --> 00:12:27.950 align:middle line:90% we have training data. 00:12:27.950 --> 00:12:28.450 align:middle line:90% Cool. 00:12:28.450 --> 00:12:30.950 align:middle line:90% Any questions about this? 00:12:30.950 --> 00:12:33.450 align:middle line:84% It's really just intended to build high-level intuition. 00:12:33.450 --> 00:12:33.950 align:middle line:90% Yeah? 00:12:33.950 --> 00:12:36.430 align:middle line:84% AUDIENCE: So I guess I'm wondering how constrained 00:12:36.430 --> 00:12:40.430 align:middle line:84% is this to the number of pixels because the sinusoid period 00:12:40.430 --> 00:12:42.468 align:middle line:84% seems like it would be affected by that one. 00:12:42.468 --> 00:12:44.760 align:middle line:84% SARA BEERY: So one of the things with the Fourier basis 00:12:44.760 --> 00:12:47.960 align:middle line:90% is that you are building out-- 00:12:47.960 --> 00:12:51.440 align:middle line:84% the basis assumes different periodicities of sine. 00:12:51.440 --> 00:12:53.400 align:middle line:84% And so you're reconstructing the image 00:12:53.400 --> 00:12:55.200 align:middle line:84% using different periodicities and weights 00:12:55.200 --> 00:12:56.700 align:middle line:90% of the sinusoid function. 00:12:56.700 --> 00:12:58.862 align:middle line:84% So that it handles some of the structure well. 00:12:58.862 --> 00:13:00.320 align:middle line:84% And it's one of the reasons that we 00:13:00.320 --> 00:13:02.112 align:middle line:84% can compress images really well with things 00:13:02.112 --> 00:13:04.480 align:middle line:90% like the fast Fourier transform. 00:13:04.480 --> 00:13:06.166 align:middle line:90% Yeah. 00:13:06.166 --> 00:13:07.060 align:middle line:90% Cool. 00:13:07.060 --> 00:13:11.920 align:middle line:84% So let's now talk about convolutional neural networks. 00:13:11.920 --> 00:13:15.320 align:middle line:90% So here's an image. 00:13:15.320 --> 00:13:17.376 align:middle line:90% What's in this image? 00:13:17.376 --> 00:13:18.744 align:middle line:90% AUDIENCE: Birds. 00:13:18.744 --> 00:13:20.940 align:middle line:90% SARA BEERY: Birds, OK. 00:13:20.940 --> 00:13:24.400 align:middle line:90% So what else is in this image? 00:13:24.400 --> 00:13:25.200 align:middle line:90% AUDIENCE: Sky. 00:13:25.200 --> 00:13:26.720 align:middle line:90% SARA BEERY: Sky. 00:13:26.720 --> 00:13:29.407 align:middle line:84% So if I had a whole image classifier, if I took this image 00:13:29.407 --> 00:13:31.240 align:middle line:84% and I ran it through one of the models we've 00:13:31.240 --> 00:13:33.820 align:middle line:84% talked about so far, what would we want it to say? 00:13:33.820 --> 00:13:36.480 align:middle line:90% 00:13:36.480 --> 00:13:39.666 align:middle line:84% What's the category of this image? 00:13:39.666 --> 00:13:41.237 align:middle line:90% AUDIENCE: Birds and sky. 00:13:41.237 --> 00:13:42.320 align:middle line:90% SARA BEERY: Birds and sky. 00:13:42.320 --> 00:13:43.445 align:middle line:90% I mean, that would be nice. 00:13:43.445 --> 00:13:45.820 align:middle line:84% We haven't talked about multi-output classification, 00:13:45.820 --> 00:13:46.320 align:middle line:90% though. 00:13:46.320 --> 00:13:46.820 align:middle line:90% Yeah? 00:13:46.820 --> 00:13:47.640 align:middle line:90% AUDIENCE: Nature. 00:13:47.640 --> 00:13:50.660 align:middle line:90% SARA BEERY: Nature, yeah. 00:13:50.660 --> 00:13:51.543 align:middle line:90% I love that answer. 00:13:51.543 --> 00:13:53.460 align:middle line:84% So one of the things that's really interesting 00:13:53.460 --> 00:13:55.940 align:middle line:84% is you can really see this trade-off between generality 00:13:55.940 --> 00:13:57.540 align:middle line:90% and specificity. 00:13:57.540 --> 00:13:59.880 align:middle line:84% And so, yeah, it's like, oh, well, that's too hard. 00:13:59.880 --> 00:14:03.460 align:middle line:84% I'm just going to predict something coarser grained. 00:14:03.460 --> 00:14:06.640 align:middle line:84% And that's a reasonable fallback. 00:14:06.640 --> 00:14:10.300 align:middle line:84% So what I was trying to get at here is actually 00:14:10.300 --> 00:14:14.180 align:middle line:84% for a lot of things we might want to understand in an image, 00:14:14.180 --> 00:14:17.640 align:middle line:84% maybe a single category isn't sufficient information. 00:14:17.640 --> 00:14:20.520 align:middle line:84% This could be flock of birds, for example. 00:14:20.520 --> 00:14:22.980 align:middle line:90% I mean, maybe that's a category. 00:14:22.980 --> 00:14:25.620 align:middle line:84% But really, it's just that a single label is usually 00:14:25.620 --> 00:14:29.340 align:middle line:84% insufficient to represent the semantic concepts that 00:14:29.340 --> 00:14:30.400 align:middle line:90% are captured in images. 00:14:30.400 --> 00:14:32.900 align:middle line:84% And this is actually, in the grand scheme of things, 00:14:32.900 --> 00:14:34.000 align:middle line:90% a pretty simple image. 00:14:34.000 --> 00:14:37.080 align:middle line:84% I mean, think about this versus an image like of Times Square, 00:14:37.080 --> 00:14:39.960 align:middle line:84% for example, how complex that image signals semantically 00:14:39.960 --> 00:14:41.400 align:middle line:90% could be. 00:14:41.400 --> 00:14:44.687 align:middle line:84% So if you were to take a picture, for example, from where 00:14:44.687 --> 00:14:47.020 align:middle line:84% you're sitting, I don't know, out there in the audience, 00:14:47.020 --> 00:14:49.280 align:middle line:84% think about what would you categorize that picture? 00:14:49.280 --> 00:14:50.740 align:middle line:90% Is it lecture hall? 00:14:50.740 --> 00:14:51.700 align:middle line:90% Is that students? 00:14:51.700 --> 00:14:53.400 align:middle line:90% Is it projector? 00:14:53.400 --> 00:14:54.300 align:middle line:90% Is it MIT? 00:14:54.300 --> 00:14:56.560 align:middle line:90% It's all those things. 00:14:56.560 --> 00:14:58.920 align:middle line:84% But so when we start to think about these more 00:14:58.920 --> 00:15:01.920 align:middle line:84% complex semantic signals, and we'll 00:15:01.920 --> 00:15:05.400 align:middle line:84% get into a lot more of this throughout the course, 00:15:05.400 --> 00:15:07.880 align:middle line:84% one thing that starts to be more and more true 00:15:07.880 --> 00:15:14.320 align:middle line:84% is that there's often maybe different categories 00:15:14.320 --> 00:15:17.440 align:middle line:84% or different bits of information that are somewhat locally 00:15:17.440 --> 00:15:18.233 align:middle line:90% constrained. 00:15:18.233 --> 00:15:20.400 align:middle line:84% And so you can think about breaking down the problem 00:15:20.400 --> 00:15:22.317 align:middle line:84% into something that actually looks at location 00:15:22.317 --> 00:15:24.680 align:middle line:90% as well as semantics. 00:15:24.680 --> 00:15:26.560 align:middle line:84% So one way that we could try to address 00:15:26.560 --> 00:15:29.880 align:middle line:84% the fact that this image contains multiple things is we 00:15:29.880 --> 00:15:32.327 align:middle line:84% could first subdivide the image into patches. 00:15:32.327 --> 00:15:34.660 align:middle line:84% And then we could try to classify each patch separately. 00:15:34.660 --> 00:15:38.800 align:middle line:84% So you could take this image, chop it up into little pieces, 00:15:38.800 --> 00:15:40.840 align:middle line:84% and then maybe you have a classifier. 00:15:40.840 --> 00:15:44.620 align:middle line:84% And now you send this patch to the classifier and it says bird. 00:15:44.620 --> 00:15:47.960 align:middle line:84% And then you could run that classifier over the entire grid. 00:15:47.960 --> 00:15:50.140 align:middle line:84% And you would take all those patches independently. 00:15:50.140 --> 00:15:51.580 align:middle line:90% And you could then-- 00:15:51.580 --> 00:15:52.360 align:middle line:90% hey, bird. 00:15:52.360 --> 00:15:54.460 align:middle line:90% This one is sky. 00:15:54.460 --> 00:15:59.220 align:middle line:84% And you could densely populate that entire space 00:15:59.220 --> 00:16:04.380 align:middle line:84% and get out the category per little block or little patch. 00:16:04.380 --> 00:16:06.780 align:middle line:84% And so if we think about this and we 00:16:06.780 --> 00:16:08.820 align:middle line:84% have some function that's going to crop this up 00:16:08.820 --> 00:16:13.960 align:middle line:84% and map every patch to a category, in some way, 00:16:13.960 --> 00:16:17.600 align:middle line:84% this is actually giving us an image output. 00:16:17.600 --> 00:16:19.520 align:middle line:84% It's a bit like a low-resolution image. 00:16:19.520 --> 00:16:23.540 align:middle line:84% It's got some spatial context, which is useful. 00:16:23.540 --> 00:16:26.460 align:middle line:84% But there are a couple patches here 00:16:26.460 --> 00:16:28.420 align:middle line:84% where the objects don't necessarily 00:16:28.420 --> 00:16:30.660 align:middle line:90% fit super neatly inside them. 00:16:30.660 --> 00:16:33.500 align:middle line:84% So I don't know, maybe this one over here 00:16:33.500 --> 00:16:36.240 align:middle line:84% that's got like the tip of the bird's wing 00:16:36.240 --> 00:16:38.400 align:middle line:84% just in that one edge of the patch. 00:16:38.400 --> 00:16:40.260 align:middle line:90% It gets a little ambiguous. 00:16:40.260 --> 00:16:42.428 align:middle line:84% Do you want that to be that patch, 00:16:42.428 --> 00:16:44.220 align:middle line:84% that whole patch to be categorized as bird, 00:16:44.220 --> 00:16:45.460 align:middle line:90% if there's any bird at all? 00:16:45.460 --> 00:16:47.252 align:middle line:84% Do you want it to be-- what is the majority 00:16:47.252 --> 00:16:49.480 align:middle line:90% of the pixels in the patch? 00:16:49.480 --> 00:16:52.660 align:middle line:84% And so maybe we want a higher resolution output map. 00:16:52.660 --> 00:16:55.400 align:middle line:84% We don't want to just know at this really coarse resolution. 00:16:55.400 --> 00:16:59.320 align:middle line:84% We actually want a finer resolution of the output. 00:16:59.320 --> 00:17:00.880 align:middle line:84% So one way you could do that is you 00:17:00.880 --> 00:17:03.160 align:middle line:84% could use smaller and smaller patches. 00:17:03.160 --> 00:17:05.079 align:middle line:84% But it turns out that as the patches get 00:17:05.079 --> 00:17:09.440 align:middle line:84% smaller and at the far extreme, the patches are a single pixel. 00:17:09.440 --> 00:17:11.680 align:middle line:84% From a single pixel, you don't have enough context 00:17:11.680 --> 00:17:14.880 align:middle line:84% to determine what the semantic category is. 00:17:14.880 --> 00:17:16.800 align:middle line:84% It's not easy to recognize content 00:17:16.800 --> 00:17:19.280 align:middle line:90% as those patches get small. 00:17:19.280 --> 00:17:24.520 align:middle line:84% So instead, the winning idea here 00:17:24.520 --> 00:17:25.960 align:middle line:84% with convolutional neural networks 00:17:25.960 --> 00:17:31.840 align:middle line:84% is we use large but overlapping patches. 00:17:31.840 --> 00:17:35.860 align:middle line:84% The very high-level intuition behind all of this is that you 00:17:35.860 --> 00:17:39.420 align:middle line:84% want context that's local to be-- 00:17:39.420 --> 00:17:41.100 align:middle line:84% you want to be able to look around 00:17:41.100 --> 00:17:44.300 align:middle line:84% the pixel you're predicting at when you make a prediction, 00:17:44.300 --> 00:17:48.460 align:middle line:84% but you also want to do this in a dense way. 00:17:48.460 --> 00:17:52.463 align:middle line:84% So the way that would look is, OK, you have your patch size, 00:17:52.463 --> 00:17:54.880 align:middle line:84% and that patch size is probably a parameter in your model. 00:17:54.880 --> 00:17:56.420 align:middle line:84% You want to make sure that context window is 00:17:56.420 --> 00:17:57.795 align:middle line:84% kind of big enough that you maybe 00:17:57.795 --> 00:18:00.380 align:middle line:90% see the whole object of bird. 00:18:00.380 --> 00:18:02.280 align:middle line:84% But then when you're actually predicting, 00:18:02.280 --> 00:18:05.940 align:middle line:84% you're sliding this over as you go. 00:18:05.940 --> 00:18:08.980 align:middle line:84% And then maybe you're only predicting, for example, 00:18:08.980 --> 00:18:12.140 align:middle line:84% the object class of the center pixel but given the context 00:18:12.140 --> 00:18:15.100 align:middle line:90% around that pixel. 00:18:15.100 --> 00:18:19.140 align:middle line:84% And so now your training data would be that patch and then 00:18:19.140 --> 00:18:23.540 align:middle line:84% the output category of that center pixel, for example. 00:18:23.540 --> 00:18:27.380 align:middle line:84% And now, if you actually run this categorizer 00:18:27.380 --> 00:18:29.620 align:middle line:84% over your entire image, you start 00:18:29.620 --> 00:18:33.320 align:middle line:84% to get something that actually looks pretty cool. 00:18:33.320 --> 00:18:37.600 align:middle line:84% That starts to look like, semantically, the boundaries 00:18:37.600 --> 00:18:39.980 align:middle line:90% of one category and another. 00:18:39.980 --> 00:18:42.960 align:middle line:84% And this problem of categorizing every pixel 00:18:42.960 --> 00:18:44.760 align:middle line:84% instead of categorizing a whole image 00:18:44.760 --> 00:18:46.960 align:middle line:90% is called semantic segmentation. 00:18:46.960 --> 00:18:50.040 align:middle line:84% And here we're categorizing every pixel as one a set 00:18:50.040 --> 00:18:52.200 align:middle line:90% of categories. 00:18:52.200 --> 00:18:53.420 align:middle line:90% This can be extended. 00:18:53.420 --> 00:18:57.255 align:middle line:84% And maybe we'll talk a bit later about instance segmentation, 00:18:57.255 --> 00:18:58.380 align:middle line:90% which is a similar problem. 00:18:58.380 --> 00:19:01.320 align:middle line:84% But now you want to identify unique instances of a given 00:19:01.320 --> 00:19:04.520 align:middle line:84% category and capture them separately. 00:19:04.520 --> 00:19:09.800 align:middle line:84% So now you have essentially the building blocks 00:19:09.800 --> 00:19:13.240 align:middle line:84% of a convolutional neural network. 00:19:13.240 --> 00:19:15.800 align:middle line:84% And one thing that's nice about this 00:19:15.800 --> 00:19:18.920 align:middle line:84% is because you're processing each of these patches 00:19:18.920 --> 00:19:21.520 align:middle line:84% independently and identically, that 00:19:21.520 --> 00:19:25.760 align:middle line:84% means that the actual model itself is 00:19:25.760 --> 00:19:28.880 align:middle line:84% invariant to translations of the input. 00:19:28.880 --> 00:19:32.440 align:middle line:84% So this is an equivariant mapping essentially. 00:19:32.440 --> 00:19:34.660 align:middle line:84% So since you're processing every single patch 00:19:34.660 --> 00:19:38.500 align:middle line:84% in the same way, that means that it doesn't matter if the bird is 00:19:38.500 --> 00:19:42.260 align:middle line:84% in the top corner, the center, the bottom left, those pixels 00:19:42.260 --> 00:19:44.940 align:middle line:84% are still going to be categorized as bird. 00:19:44.940 --> 00:19:46.460 align:middle line:84% And another way to think about this 00:19:46.460 --> 00:19:49.940 align:middle line:84% is if you took your function and you applied it 00:19:49.940 --> 00:19:51.900 align:middle line:84% to a translation of the input, it 00:19:51.900 --> 00:19:55.780 align:middle line:84% would be the same as if you translated a function 00:19:55.780 --> 00:19:57.660 align:middle line:90% of the original input. 00:19:57.660 --> 00:20:00.340 align:middle line:84% And why would this maybe be a nice thing 00:20:00.340 --> 00:20:03.740 align:middle line:84% to have for something like semantic segmentation? 00:20:03.740 --> 00:20:07.540 align:middle line:84% Why does this make sense as an inductive bias or a hypothesis 00:20:07.540 --> 00:20:12.290 align:middle line:84% for images and for categorizing like this? 00:20:12.290 --> 00:20:13.345 align:middle line:90% Yeah? 00:20:13.345 --> 00:20:15.220 align:middle line:84% AUDIENCE: Like the answer for each pixel kind 00:20:15.220 --> 00:20:18.120 align:middle line:84% of corresponds to the [INAUDIBLE] next slide 00:20:18.120 --> 00:20:20.052 align:middle line:90% [INAUDIBLE]. 00:20:20.052 --> 00:20:21.260 align:middle line:90% SARA BEERY: Yeah, yeah, yeah. 00:20:21.260 --> 00:20:27.140 align:middle line:84% So the answer corresponds to that local context window. 00:20:27.140 --> 00:20:29.660 align:middle line:84% Yeah, if you took an image of a bird 00:20:29.660 --> 00:20:32.760 align:middle line:84% and then you moved that portion of the image corresponding 00:20:32.760 --> 00:20:35.360 align:middle line:84% to a bird to a different part of the image or imagine 00:20:35.360 --> 00:20:37.480 align:middle line:84% you were actually taking a photo and then you 00:20:37.480 --> 00:20:39.140 align:middle line:84% shifted your camera a little bit, 00:20:39.140 --> 00:20:42.080 align:middle line:84% the object you're taking a photo of is the same. 00:20:42.080 --> 00:20:46.920 align:middle line:84% So there actually is this type of symmetry innate to categories 00:20:46.920 --> 00:20:48.020 align:middle line:90% in the real world. 00:20:48.020 --> 00:20:49.520 align:middle line:84% And this symmetry is something we've 00:20:49.520 --> 00:20:53.000 align:middle line:84% built into our hypothesis space via the model structure. 00:20:53.000 --> 00:20:57.040 align:middle line:84% And so, yeah, the world is somewhat translation invariant. 00:20:57.040 --> 00:21:00.345 align:middle line:84% A bird should look the same no matter where it is in an image. 00:21:00.345 --> 00:21:01.720 align:middle line:84% Of course, we're kind of glossing 00:21:01.720 --> 00:21:06.140 align:middle line:84% over pose, which is a more complicated factor here. 00:21:06.140 --> 00:21:07.920 align:middle line:84% But if you have something at the same pose 00:21:07.920 --> 00:21:09.840 align:middle line:84% and you move it somewhere else, it still 00:21:09.840 --> 00:21:12.880 align:middle line:90% maintains the same category. 00:21:12.880 --> 00:21:15.020 align:middle line:90% So cool. 00:21:15.020 --> 00:21:19.720 align:middle line:84% One possible version of this, of this model, 00:21:19.720 --> 00:21:24.240 align:middle line:84% would be to compute a weighted sum of the pixels in each patch. 00:21:24.240 --> 00:21:27.000 align:middle line:84% And then the weights of those that sum, that's 00:21:27.000 --> 00:21:28.260 align:middle line:90% what we're going to learn. 00:21:28.260 --> 00:21:32.250 align:middle line:84% And so if you do this, then essentially 00:21:32.250 --> 00:21:37.210 align:middle line:84% that set of linear weights, W, is a convolutional kernel 00:21:37.210 --> 00:21:41.633 align:middle line:84% that then we can apply to the entire image. 00:21:41.633 --> 00:21:44.050 align:middle line:84% And so what does this mean/ What does this kind of concept 00:21:44.050 --> 00:21:47.730 align:middle line:84% of convolution and a filter or a kernel? 00:21:47.730 --> 00:21:52.010 align:middle line:84% So a convolution is a linear, shift-invariant transformation 00:21:52.010 --> 00:21:56.130 align:middle line:90% of grid-structured input data. 00:21:56.130 --> 00:21:58.210 align:middle line:84% And essentially, in signal processing, 00:21:58.210 --> 00:22:00.017 align:middle line:84% convolution is a way that you do filtering. 00:22:00.017 --> 00:22:01.850 align:middle line:84% So how many of you have taken, I don't know, 00:22:01.850 --> 00:22:04.770 align:middle line:84% a signal processing class or learned signal processing 00:22:04.770 --> 00:22:06.710 align:middle line:84% maybe in like an electrical engineering class? 00:22:06.710 --> 00:22:08.210 align:middle line:90% Raise hands. 00:22:08.210 --> 00:22:12.730 align:middle line:84% Cool, so a good number of you have seen this before. 00:22:12.730 --> 00:22:15.410 align:middle line:84% And here, the filter or the weights 00:22:15.410 --> 00:22:19.027 align:middle line:84% we use to take that linear weighted sum. 00:22:19.027 --> 00:22:21.610 align:middle line:84% And then mathematically, if we want to structure that out here 00:22:21.610 --> 00:22:25.515 align:middle line:84% at the bottom, the output for a given location 00:22:25.515 --> 00:22:26.890 align:middle line:84% when applying the filter is going 00:22:26.890 --> 00:22:31.190 align:middle line:84% to be some bias, plus a linear combination of the filter weight 00:22:31.190 --> 00:22:33.830 align:middle line:84% for that coordinate times the pixel value for that 00:22:33.830 --> 00:22:35.390 align:middle line:84% coordinate in the image, and then 00:22:35.390 --> 00:22:37.598 align:middle line:84% summed over all the filter weights and all the pixels 00:22:37.598 --> 00:22:38.630 align:middle line:90% in the patch. 00:22:38.630 --> 00:22:41.750 align:middle line:84% So really what it's doing is for any given 00:22:41.750 --> 00:22:43.590 align:middle line:84% filter-- like in this case, this filter 00:22:43.590 --> 00:22:46.550 align:middle line:84% is kind of diffused all around, and then you 00:22:46.550 --> 00:22:49.870 align:middle line:84% have something dark and something light. 00:22:49.870 --> 00:22:53.750 align:middle line:84% So this filter is going to look for that pattern. 00:22:53.750 --> 00:22:56.610 align:middle line:84% By look for, that's very anthropomorphized. 00:22:56.610 --> 00:22:59.710 align:middle line:84% But you're going to produce maximum output 00:22:59.710 --> 00:23:01.950 align:middle line:84% values when you have that same kind 00:23:01.950 --> 00:23:03.868 align:middle line:90% of dark to light transition. 00:23:03.868 --> 00:23:05.410 align:middle line:84% And so what that actually looks like, 00:23:05.410 --> 00:23:08.630 align:middle line:84% if you run that filter over this image, 00:23:08.630 --> 00:23:11.510 align:middle line:84% is you start to get something that's highlighting areas 00:23:11.510 --> 00:23:16.550 align:middle line:84% in the image that match the filter the most, so 00:23:16.550 --> 00:23:17.970 align:middle line:90% match it most closely. 00:23:17.970 --> 00:23:21.070 align:middle line:84% So here we can see that we're getting really high values, so 00:23:21.070 --> 00:23:23.950 align:middle line:84% the white values along the boundaries between dark 00:23:23.950 --> 00:23:24.710 align:middle line:90% to light. 00:23:24.710 --> 00:23:26.810 align:middle line:84% And we're actually getting really low values 00:23:26.810 --> 00:23:28.830 align:middle line:84% along the boundaries from light to dark 00:23:28.830 --> 00:23:31.050 align:middle line:84% because that's the place that's going to be maximally 00:23:31.050 --> 00:23:34.330 align:middle line:90% opposite to that filter. 00:23:34.330 --> 00:23:36.810 align:middle line:84% So in this particular case, what we've got 00:23:36.810 --> 00:23:39.770 align:middle line:84% is a filter that's finding us horizontal edges 00:23:39.770 --> 00:23:42.810 align:middle line:90% between dark and light values. 00:23:42.810 --> 00:23:43.590 align:middle line:90% Any questions? 00:23:43.590 --> 00:23:44.690 align:middle line:90% Yeah? 00:23:44.690 --> 00:23:46.330 align:middle line:84% AUDIENCE: Can you explain [INAUDIBLE] 00:23:46.330 --> 00:23:48.662 align:middle line:84% variance transformation in this context? 00:23:48.662 --> 00:23:49.370 align:middle line:90% SARA BEERY: Yeah. 00:23:49.370 --> 00:23:55.570 align:middle line:84% So, again, here if I took this image of a clownfish 00:23:55.570 --> 00:24:00.770 align:middle line:84% and I shifted it over and I ran this convolutional filter 00:24:00.770 --> 00:24:04.810 align:middle line:84% over it, the output would be this shifted over. 00:24:04.810 --> 00:24:07.050 align:middle line:84% And so you could shift the input, 00:24:07.050 --> 00:24:08.353 align:middle line:90% then apply the convolution. 00:24:08.353 --> 00:24:09.770 align:middle line:84% Or you could apply the convolution 00:24:09.770 --> 00:24:12.312 align:middle line:84% and then shift the output, and the results would be the same. 00:24:12.312 --> 00:24:15.330 align:middle line:90% 00:24:15.330 --> 00:24:19.610 align:middle line:84% So now let's look at this from a more neural networky 00:24:19.610 --> 00:24:21.170 align:middle line:90% perspective. 00:24:21.170 --> 00:24:22.910 align:middle line:84% So this is a fully connected layer. 00:24:22.910 --> 00:24:24.750 align:middle line:90% This is what we've seen so far. 00:24:24.750 --> 00:24:29.430 align:middle line:84% And here all the outputs depend on all of the inputs. 00:24:29.430 --> 00:24:31.510 align:middle line:84% And this can be quite computationally expensive 00:24:31.510 --> 00:24:33.750 align:middle line:90% when things get very large. 00:24:33.750 --> 00:24:36.790 align:middle line:84% Instead of a fully connected network, 00:24:36.790 --> 00:24:40.470 align:middle line:84% a convolutional network is essentially a locally connected 00:24:40.470 --> 00:24:41.630 align:middle line:90% network. 00:24:41.630 --> 00:24:45.950 align:middle line:84% And so often we can assume that the output then of this model 00:24:45.950 --> 00:24:48.950 align:middle line:84% is a local function of the input. 00:24:48.950 --> 00:24:52.270 align:middle line:84% So here if we use the same weights 00:24:52.270 --> 00:24:56.870 align:middle line:84% for these linear combinations, this w matrix, 00:24:56.870 --> 00:24:59.230 align:middle line:84% if we use those weights the same regardless 00:24:59.230 --> 00:25:04.870 align:middle line:84% of where that initial input values are coming from, 00:25:04.870 --> 00:25:08.170 align:middle line:84% that gives us a convolutional neural network. 00:25:08.170 --> 00:25:10.310 align:middle line:84% And the way that we actually denote that 00:25:10.310 --> 00:25:13.470 align:middle line:84% is with this star operation for convolution. 00:25:13.470 --> 00:25:15.470 align:middle line:84% If you have take signal processing, 00:25:15.470 --> 00:25:17.830 align:middle line:84% actually, this is not quite the same definition 00:25:17.830 --> 00:25:20.490 align:middle line:84% as familiarly used in signal processing for convolution. 00:25:20.490 --> 00:25:23.970 align:middle line:84% Usually, it has like a flipped value. 00:25:23.970 --> 00:25:26.950 align:middle line:84% But here when we talk about neural networks, 00:25:26.950 --> 00:25:29.337 align:middle line:84% instead of calling this cross correlation, which 00:25:29.337 --> 00:25:31.670 align:middle line:84% might be what it is in the signal processing literature, 00:25:31.670 --> 00:25:33.227 align:middle line:90% we just call this convolution. 00:25:33.227 --> 00:25:35.310 align:middle line:84% One of those reasons is because in these networks, 00:25:35.310 --> 00:25:37.190 align:middle line:84% it's very easy to learn how to flip a sign, 00:25:37.190 --> 00:25:38.430 align:middle line:90% so it doesn't matter so much. 00:25:38.430 --> 00:25:38.930 align:middle line:90% Yeah? 00:25:38.930 --> 00:25:41.170 align:middle line:84% AUDIENCE: Is it always a five-point star? 00:25:41.170 --> 00:25:44.250 align:middle line:84% SARA BEERY: I mean, people do all sorts of stuff. 00:25:44.250 --> 00:25:46.730 align:middle line:84% We are going to use a five-point star in our lectures 00:25:46.730 --> 00:25:49.010 align:middle line:90% and in the notes. 00:25:49.010 --> 00:25:52.090 align:middle line:90% I think it's kind of pretty. 00:25:52.090 --> 00:25:53.830 align:middle line:84% So what does this actually look like? 00:25:53.830 --> 00:25:57.350 align:middle line:84% So here we have this same linear combination of weights. 00:25:57.350 --> 00:26:00.202 align:middle line:84% We're running it over the local structure. 00:26:00.202 --> 00:26:01.910 align:middle line:84% And then you're taking that weighted sum. 00:26:01.910 --> 00:26:03.810 align:middle line:84% And so the outputs for each of these values 00:26:03.810 --> 00:26:06.845 align:middle line:84% are going to be corresponding to just that local input. 00:26:06.845 --> 00:26:08.970 align:middle line:84% But it's going to be the same weights that are used 00:26:08.970 --> 00:26:10.870 align:middle line:90% across the entire input vector. 00:26:10.870 --> 00:26:13.450 align:middle line:90% 00:26:13.450 --> 00:26:16.930 align:middle line:84% And the nice thing with that thing 00:26:16.930 --> 00:26:20.370 align:middle line:84% is that you get this, again, embarrassing parallelism 00:26:20.370 --> 00:26:26.750 align:middle line:84% because now you can apply that same weight matrix across only 00:26:26.750 --> 00:26:29.050 align:middle line:90% smaller patches of the input. 00:26:29.050 --> 00:26:30.790 align:middle line:90% But you can do that in parallel. 00:26:30.790 --> 00:26:34.870 align:middle line:84% And if you want to think about what 00:26:34.870 --> 00:26:39.150 align:middle line:84% that looks like in this broader sense, if you're taking a fully 00:26:39.150 --> 00:26:43.090 align:middle line:84% connected layer, that's the same as taking your input vector, 00:26:43.090 --> 00:26:46.390 align:middle line:84% multiplying it by a dense weight matrix, where 00:26:46.390 --> 00:26:49.110 align:middle line:84% every single one of those values matters. 00:26:49.110 --> 00:26:51.150 align:middle line:84% And then that's how you get your output. 00:26:51.150 --> 00:26:53.670 align:middle line:84% And now you could think about actually 00:26:53.670 --> 00:26:56.670 align:middle line:84% a convolutional layer is the same. 00:26:56.670 --> 00:27:00.190 align:middle line:84% But now the weight matrix has a very specific structure. 00:27:00.190 --> 00:27:02.770 align:middle line:84% So instead of being dense, it's going to be very sparse. 00:27:02.770 --> 00:27:04.728 align:middle line:84% So in this particular instance where we're only 00:27:04.728 --> 00:27:06.870 align:middle line:84% looking at a sliding window of maybe three 00:27:06.870 --> 00:27:10.910 align:middle line:84% elements of that input vector, those three elements 00:27:10.910 --> 00:27:14.530 align:middle line:84% are going to be the only-- above and below that diagonal, 00:27:14.530 --> 00:27:17.167 align:middle line:84% those are going to be the only components that have values, 00:27:17.167 --> 00:27:18.750 align:middle line:84% and the values are going to be exactly 00:27:18.750 --> 00:27:22.330 align:middle line:84% the same along the entire diagonal of that matrix. 00:27:22.330 --> 00:27:24.950 align:middle line:84% And there's a special name for this type of matrix. 00:27:24.950 --> 00:27:28.890 align:middle line:84% It's a specific form of a Toeplitz matrix. 00:27:28.890 --> 00:27:30.490 align:middle line:84% And so one of the things that's cool 00:27:30.490 --> 00:27:33.650 align:middle line:84% here is that it's essentially a constraint 00:27:33.650 --> 00:27:35.810 align:middle line:90% on a standard linear layer. 00:27:35.810 --> 00:27:37.633 align:middle line:84% And there are fewer parameters that 00:27:37.633 --> 00:27:39.050 align:middle line:84% can be learned because you're just 00:27:39.050 --> 00:27:43.070 align:middle line:84% learning essentially parameters for just that local structure. 00:27:43.070 --> 00:27:45.670 align:middle line:84% So there's less of them, and they're going to be repeated. 00:27:45.670 --> 00:27:48.770 align:middle line:84% And so that means you can have a lot less overfitting. 00:27:48.770 --> 00:27:51.533 align:middle line:90% There's less open flexibility. 00:27:51.533 --> 00:27:53.950 align:middle line:84% And you're building some nice constraints into the system. 00:27:53.950 --> 00:27:55.190 align:middle line:90% You're making it equivariant. 00:27:55.190 --> 00:27:57.850 align:middle line:84% You're making it translation invariant. 00:27:57.850 --> 00:28:00.330 align:middle line:84% And then another nice benefit of this 00:28:00.330 --> 00:28:02.970 align:middle line:84% is because this thing has this specific structure where 00:28:02.970 --> 00:28:04.790 align:middle line:84% you have zeros almost everywhere, 00:28:04.790 --> 00:28:09.090 align:middle line:84% except for just above and below the diagonal, 00:28:09.090 --> 00:28:12.970 align:middle line:84% this can generalize to different input sizes 00:28:12.970 --> 00:28:16.190 align:middle line:84% because you can take that same weight matrix 00:28:16.190 --> 00:28:17.230 align:middle line:90% and you can scale it. 00:28:17.230 --> 00:28:18.630 align:middle line:84% It's not going to change anything 00:28:18.630 --> 00:28:20.230 align:middle line:90% about the model you've learned. 00:28:20.230 --> 00:28:22.390 align:middle line:84% And so that means that convolutional layers 00:28:22.390 --> 00:28:24.390 align:middle line:84% can be applied to arbitrarily sized 00:28:24.390 --> 00:28:26.670 align:middle line:84% inputs, which generalizes beyond the training 00:28:26.670 --> 00:28:28.650 align:middle line:90% data in a really nice way. 00:28:28.650 --> 00:28:31.350 align:middle line:84% Whereas if you took an MLP and you applied it 00:28:31.350 --> 00:28:34.150 align:middle line:84% to an image and then you tried to apply it to a larger image, 00:28:34.150 --> 00:28:35.610 align:middle line:90% you couldn't do it. 00:28:35.610 --> 00:28:38.330 align:middle line:84% This isn't something that you could apply to a larger image. 00:28:38.330 --> 00:28:40.950 align:middle line:84% You just wouldn't have weights for some of that size. 00:28:40.950 --> 00:28:42.397 align:middle line:84% And so one way to look at this is 00:28:42.397 --> 00:28:44.230 align:middle line:84% like you could take that same filter that we 00:28:44.230 --> 00:28:46.970 align:middle line:84% applied to the clownfish, and you could apply it to any image. 00:28:46.970 --> 00:28:49.830 align:middle line:84% And you would always get out these edges, 00:28:49.830 --> 00:28:51.990 align:middle line:90% these specific things. 00:28:51.990 --> 00:28:56.590 align:middle line:84% So five kind of high-level views-- yeah? 00:28:56.590 --> 00:28:58.370 align:middle line:84% AUDIENCE: If you go back just like before. 00:28:58.370 --> 00:28:59.230 align:middle line:90% Yeah. 00:28:59.230 --> 00:29:03.390 align:middle line:84% If you scale the matrix, does each pixel color 00:29:03.390 --> 00:29:05.430 align:middle line:90% here correspond to one value? 00:29:05.430 --> 00:29:09.630 align:middle line:84% SARA BEERY: So we talked about the weights, 00:29:09.630 --> 00:29:12.770 align:middle line:90% so w1, w0, w minus 1. 00:29:12.770 --> 00:29:16.330 align:middle line:84% So that's like if you're taking this from a vector perspective, 00:29:16.330 --> 00:29:18.890 align:middle line:84% you want to look at the wait just before and just after. 00:29:18.890 --> 00:29:22.610 align:middle line:84% That means that you're only going to have three values. 00:29:22.610 --> 00:29:24.610 align:middle line:84% And those three values, if you transfer this 00:29:24.610 --> 00:29:26.330 align:middle line:84% into a linear layer, those are going 00:29:26.330 --> 00:29:29.530 align:middle line:84% to be the values above, along, and below the diagonal 00:29:29.530 --> 00:29:30.430 align:middle line:90% for the whole matrix. 00:29:30.430 --> 00:29:32.930 align:middle line:84% So you just take that and slide like the ladder 00:29:32.930 --> 00:29:34.150 align:middle line:90% out along the diagonal. 00:29:34.150 --> 00:29:37.410 align:middle line:90% It'd be the same, like that. 00:29:37.410 --> 00:29:41.050 align:middle line:90% 00:29:41.050 --> 00:29:41.570 align:middle line:90% Cool. 00:29:41.570 --> 00:29:45.070 align:middle line:84% So some different ways to think about convolution. 00:29:45.070 --> 00:29:49.230 align:middle line:84% So, one, convolutional layers are equivariant to translation. 00:29:49.230 --> 00:29:51.430 align:middle line:84% So this is a built-in hypothesis. 00:29:51.430 --> 00:29:55.430 align:middle line:84% Two, convolutional layers enable patch processing. 00:29:55.430 --> 00:29:58.050 align:middle line:84% It builds in explicit parallelism 00:29:58.050 --> 00:29:59.390 align:middle line:90% into our computation. 00:29:59.390 --> 00:30:02.330 align:middle line:84% It makes it computationally efficient. 00:30:02.330 --> 00:30:04.090 align:middle line:84% Three, convolutional layers can be 00:30:04.090 --> 00:30:06.410 align:middle line:84% thought of as kind of image filters, things 00:30:06.410 --> 00:30:10.090 align:middle line:84% that are highlighting specific structures in the input data 00:30:10.090 --> 00:30:13.130 align:middle line:84% in some useful way that's learned. 00:30:13.130 --> 00:30:16.420 align:middle line:84% Or you can think of them as parameter sharing. 00:30:16.420 --> 00:30:19.340 align:middle line:84% You're using the same parameters across the entire input 00:30:19.340 --> 00:30:22.620 align:middle line:84% data in a computationally efficient way. 00:30:22.620 --> 00:30:25.140 align:middle line:84% And the last point is you can think of them as a way 00:30:25.140 --> 00:30:28.380 align:middle line:84% to process variable size tensors because you 00:30:28.380 --> 00:30:31.620 align:middle line:84% have this built-in generalization across the input 00:30:31.620 --> 00:30:33.040 align:middle line:90% size. 00:30:33.040 --> 00:30:35.660 align:middle line:90% Cool. 00:30:35.660 --> 00:30:39.140 align:middle line:84% So what happens when you stack convolutional layers? 00:30:39.140 --> 00:30:43.780 align:middle line:84% So here what we've got is two sets of convolutional layers. 00:30:43.780 --> 00:30:46.300 align:middle line:84% So, first, our input data comes in. 00:30:46.300 --> 00:30:48.300 align:middle line:84% We have a first convolutional layer. 00:30:48.300 --> 00:30:50.140 align:middle line:90% Then we have a non-linearity. 00:30:50.140 --> 00:30:52.540 align:middle line:84% And then we have a second convolutional layer. 00:30:52.540 --> 00:30:54.780 align:middle line:84% And it turns out that if you think of it this way, 00:30:54.780 --> 00:30:59.380 align:middle line:84% actually then the entire CNN is a convolutional filter, but one 00:30:59.380 --> 00:31:00.780 align:middle line:90% that's nonlinear. 00:31:00.780 --> 00:31:03.900 align:middle line:84% So you can really think of this way. 00:31:03.900 --> 00:31:06.980 align:middle line:84% And so here now this output depends only 00:31:06.980 --> 00:31:08.840 align:middle line:84% on the local structure of that input. 00:31:08.840 --> 00:31:11.580 align:middle line:84% And that function, even though it has that non-linearity 00:31:11.580 --> 00:31:15.380 align:middle line:84% in the middle, is the same for anywhere, 00:31:15.380 --> 00:31:17.440 align:middle line:84% any of these local patches in the input image. 00:31:17.440 --> 00:31:18.640 align:middle line:90% Does this make sense? 00:31:18.640 --> 00:31:19.220 align:middle line:90% Yeah? 00:31:19.220 --> 00:31:20.960 align:middle line:90% AUDIENCE: Why is it [INAUDIBLE]? 00:31:20.960 --> 00:31:24.620 align:middle line:90% 00:31:24.620 --> 00:31:25.360 align:middle line:90% SARA BEERY: Yeah. 00:31:25.360 --> 00:31:27.900 align:middle line:84% So I'm just saying here in this particular instance, 00:31:27.900 --> 00:31:30.920 align:middle line:84% assume now that you have a single convolutional layer, 00:31:30.920 --> 00:31:34.440 align:middle line:84% a point-wise non-linearity just like we had for MLP, and then 00:31:34.440 --> 00:31:37.120 align:middle line:90% another convolutional layer. 00:31:37.120 --> 00:31:39.080 align:middle line:84% So since we said that a convolutional layer is 00:31:39.080 --> 00:31:41.420 align:middle line:84% essentially a constrained version of a linear layer, 00:31:41.420 --> 00:31:43.520 align:middle line:84% you can essentially take it the same way. 00:31:43.520 --> 00:31:45.840 align:middle line:84% But the nice thing is that constrained version 00:31:45.840 --> 00:31:49.000 align:middle line:84% of the linear layer, it maintains that local constraint 00:31:49.000 --> 00:31:53.160 align:middle line:84% even as you start to stack layers. 00:31:53.160 --> 00:31:57.760 align:middle line:84% But one thing that's very clear is as you're adding more depth, 00:31:57.760 --> 00:31:59.800 align:middle line:84% you're increasing what we call the receptive 00:31:59.800 --> 00:32:01.480 align:middle line:90% field of the output. 00:32:01.480 --> 00:32:04.420 align:middle line:90% So you're adding more layers. 00:32:04.420 --> 00:32:06.960 align:middle line:84% That means that the local patch that 00:32:06.960 --> 00:32:09.480 align:middle line:84% corresponds to, for example, xL of j 00:32:09.480 --> 00:32:12.860 align:middle line:84% is larger than the local patch of a single one 00:32:12.860 --> 00:32:14.980 align:middle line:84% of those same layers because you're 00:32:14.980 --> 00:32:16.940 align:middle line:84% taking an input from a larger size 00:32:16.940 --> 00:32:18.400 align:middle line:84% and then you're doing that again. 00:32:18.400 --> 00:32:23.160 align:middle line:84% And this is what we call a spatial pyramid, for example. 00:32:23.160 --> 00:32:23.660 align:middle line:90% Yeah? 00:32:23.660 --> 00:32:26.090 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:32:26.090 --> 00:32:26.840 align:middle line:90% SARA BEERY: Sorry. 00:32:26.840 --> 00:32:27.965 align:middle line:90% Can you be a little louder? 00:32:27.965 --> 00:32:29.750 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:32:29.750 --> 00:32:30.860 align:middle line:90% SARA BEERY: Oh yeah, yeah. 00:32:30.860 --> 00:32:33.860 align:middle line:84% So the only reason this is nonlinear 00:32:33.860 --> 00:32:36.380 align:middle line:84% is because there's a non-linearity in the middle. 00:32:36.380 --> 00:32:39.100 align:middle line:84% So from between that first layer-- 00:32:39.100 --> 00:32:41.220 align:middle line:84% so you have this first convolutional layer, 00:32:41.220 --> 00:32:43.660 align:middle line:84% and then see where you have those nodes in the middle. 00:32:43.660 --> 00:32:45.700 align:middle line:84% What I'm saying is here we've applied 00:32:45.700 --> 00:32:48.060 align:middle line:84% an activation, a nonlinear activation. 00:32:48.060 --> 00:32:49.560 align:middle line:84% And then you have that second layer. 00:32:49.560 --> 00:32:51.352 align:middle line:84% Maybe that's something that in future years 00:32:51.352 --> 00:32:53.040 align:middle line:84% I'll just put that in the diagram 00:32:53.040 --> 00:32:54.593 align:middle line:84% so you can see that non-linearity, 00:32:54.593 --> 00:32:55.760 align:middle line:90% so it'd be a little clearer. 00:32:55.760 --> 00:32:59.500 align:middle line:90% But I'm just saying it exists. 00:32:59.500 --> 00:33:01.440 align:middle line:84% So now you have a convolutional filter, 00:33:01.440 --> 00:33:04.180 align:middle line:84% but it's nonlinear because you have that nonlinear activation 00:33:04.180 --> 00:33:06.120 align:middle line:90% that you've put in the middle. 00:33:06.120 --> 00:33:06.620 align:middle line:90% Yeah? 00:33:06.620 --> 00:33:10.993 align:middle line:84% AUDIENCE: What is it called when there's [INAUDIBLE] established 00:33:10.993 --> 00:33:13.160 align:middle line:84% when you're pulling weights from previous dependents 00:33:13.160 --> 00:33:14.880 align:middle line:90% and previous layers [INAUDIBLE]? 00:33:14.880 --> 00:33:19.000 align:middle line:84% SARA BEERY: A receptive field is one term that people will use. 00:33:19.000 --> 00:33:21.640 align:middle line:84% So if you're like increasing the receptive field of your model, 00:33:21.640 --> 00:33:23.140 align:middle line:84% you can do that in a couple of ways. 00:33:23.140 --> 00:33:25.400 align:middle line:84% You can make the original patch size, 00:33:25.400 --> 00:33:28.520 align:middle line:84% input size larger, so the kernel size of the filter size 00:33:28.520 --> 00:33:31.060 align:middle line:84% could get larger, or you can just add more depth. 00:33:31.060 --> 00:33:34.520 align:middle line:84% And that will also increase the receptive field. 00:33:34.520 --> 00:33:35.240 align:middle line:90% Yeah? 00:33:35.240 --> 00:33:37.040 align:middle line:84% AUDIENCE: A couple slides back, you 00:33:37.040 --> 00:33:40.840 align:middle line:90% represented acts as a vector. 00:33:40.840 --> 00:33:43.652 align:middle line:84% So that's an image that's been flattened into a vector. 00:33:43.652 --> 00:33:44.360 align:middle line:90% SARA BEERY: Yeah. 00:33:44.360 --> 00:33:46.068 align:middle line:84% So right now, we're talking about vectors 00:33:46.068 --> 00:33:48.232 align:middle line:84% because they're really clean and easy to look at. 00:33:48.232 --> 00:33:50.440 align:middle line:84% But, yeah, you can essentially say, you take an image 00:33:50.440 --> 00:33:52.533 align:middle line:84% and you flatten it out, though I think 00:33:52.533 --> 00:33:54.200 align:middle line:84% it's a little more complicated than that 00:33:54.200 --> 00:33:57.320 align:middle line:84% because you wouldn't take an image 00:33:57.320 --> 00:33:59.740 align:middle line:84% and flatten it out and then apply convolution like this 00:33:59.740 --> 00:34:02.120 align:middle line:84% because you would have this weird like stripe-wise 00:34:02.120 --> 00:34:02.700 align:middle line:90% structure. 00:34:02.700 --> 00:34:05.880 align:middle line:84% So, actually, we do take those convolutions in two dimensions. 00:34:05.880 --> 00:34:07.080 align:middle line:90% Yeah? 00:34:07.080 --> 00:34:10.675 align:middle line:84% AUDIENCE: In the second layer, would you use the same kernel? 00:34:10.675 --> 00:34:11.300 align:middle line:90% SARA BEERY: No. 00:34:11.300 --> 00:34:12.967 align:middle line:84% And that's why the colors are different. 00:34:12.967 --> 00:34:13.980 align:middle line:90% I mean, you could. 00:34:13.980 --> 00:34:15.475 align:middle line:90% But usually, we don't. 00:34:15.475 --> 00:34:17.100 align:middle line:84% Those weights are learned, but they are 00:34:17.100 --> 00:34:18.600 align:middle line:90% shared across the entire layer. 00:34:18.600 --> 00:34:19.968 align:middle line:90% Yeah? 00:34:19.968 --> 00:34:22.260 align:middle line:84% AUDIENCE: So just to make sure I understand [INAUDIBLE] 00:34:22.260 --> 00:34:27.260 align:middle line:84% each weight matrix in each layer [INAUDIBLE] 00:34:27.260 --> 00:34:38.380 align:middle line:84% except for the [INAUDIBLE] layers in our network in order 00:34:38.380 --> 00:34:41.778 align:middle line:90% to represent the [INAUDIBLE]. 00:34:41.778 --> 00:34:44.320 align:middle line:84% SARA BEERY: So the question is like for each of these layers, 00:34:44.320 --> 00:34:46.260 align:middle line:84% now that we have a convolutional layer, 00:34:46.260 --> 00:34:49.340 align:middle line:84% they can be represented as that sparse matrix where you only 00:34:49.340 --> 00:34:51.219 align:middle line:90% are looking along the diagonal. 00:34:51.219 --> 00:34:54.980 align:middle line:84% We don't always necessarily actually compute it that way. 00:34:54.980 --> 00:34:56.800 align:middle line:84% For efficiency or for parallelism, 00:34:56.800 --> 00:34:59.580 align:middle line:84% often, we just really do compute it patch wise. 00:34:59.580 --> 00:35:03.820 align:middle line:84% But your point was if you're now really restricting the search 00:35:03.820 --> 00:35:07.980 align:middle line:84% space by only allowing like a few parameters per layer, 00:35:07.980 --> 00:35:10.920 align:middle line:84% doesn't that mean that you would really struggle 00:35:10.920 --> 00:35:13.640 align:middle line:90% to represent images or learn? 00:35:13.640 --> 00:35:16.980 align:middle line:90% And the answer is yes. 00:35:16.980 --> 00:35:19.233 align:middle line:84% And so that's why often with convolutional layers, 00:35:19.233 --> 00:35:21.400 align:middle line:84% either we'll have quite a few different filters that 00:35:21.400 --> 00:35:23.525 align:middle line:84% are happening in parallel, we'll get into it later, 00:35:23.525 --> 00:35:26.400 align:middle line:84% or we add a lot more layers in a dense way. 00:35:26.400 --> 00:35:28.135 align:middle line:84% But what we have found experimentally, 00:35:28.135 --> 00:35:29.760 align:middle line:84% and there's some nice theory around it, 00:35:29.760 --> 00:35:33.400 align:middle line:84% is that for things where this is [? an ?] [? appropriate ?] 00:35:33.400 --> 00:35:36.140 align:middle line:84% inductive bias, you can learn more efficiently. 00:35:36.140 --> 00:35:39.060 align:middle line:84% You get a better fit with less data. 00:35:39.060 --> 00:35:40.800 align:middle line:90% Cool. 00:35:40.800 --> 00:35:43.240 align:middle line:90% So what if we have color? 00:35:43.240 --> 00:35:45.820 align:middle line:84% I mean, we're really taking a very image perspective here. 00:35:45.820 --> 00:35:47.987 align:middle line:84% But essentially, what we're talking about with color 00:35:47.987 --> 00:35:51.180 align:middle line:84% is multiple input channels, so not just spatial extent, 00:35:51.180 --> 00:35:55.080 align:middle line:84% but you have a tensor that also has something like depth. 00:35:55.080 --> 00:35:58.780 align:middle line:84% So if you have multi-channel inputs-- so say you do have RGB. 00:35:58.780 --> 00:36:00.920 align:middle line:90% That's what we're showing here. 00:36:00.920 --> 00:36:04.040 align:middle line:84% What you do to do convolution over multi-channel inputs 00:36:04.040 --> 00:36:08.020 align:middle line:84% is you learn a different set of weights for each of those input 00:36:08.020 --> 00:36:08.840 align:middle line:90% channels. 00:36:08.840 --> 00:36:13.340 align:middle line:84% And then you sum over all of that to get the output value. 00:36:13.340 --> 00:36:16.380 align:middle line:84% So you sum essentially over the convolutions 00:36:16.380 --> 00:36:19.020 align:middle line:84% for each of the input channels to get the output value 00:36:19.020 --> 00:36:21.180 align:middle line:90% you're summing per channel. 00:36:21.180 --> 00:36:25.060 align:middle line:84% So now what that looks like is you 00:36:25.060 --> 00:36:28.920 align:middle line:84% would still have a single output value for each filter. 00:36:28.920 --> 00:36:33.300 align:middle line:84% But now the filter is going to have per channel weights that 00:36:33.300 --> 00:36:34.180 align:middle line:90% are learned. 00:36:34.180 --> 00:36:37.880 align:middle line:84% And you could arguably force them to be shared. 00:36:37.880 --> 00:36:41.180 align:middle line:84% But often, we let them be separate. 00:36:41.180 --> 00:36:43.980 align:middle line:84% What about multi-channel outputs? 00:36:43.980 --> 00:36:47.998 align:middle line:84% So if instead of having just a single output 00:36:47.998 --> 00:36:49.540 align:middle line:84% of your convolutional layer, you want 00:36:49.540 --> 00:36:51.957 align:middle line:84% to have multiple channels of outputs, 00:36:51.957 --> 00:36:53.540 align:middle line:84% here what you actually do is you learn 00:36:53.540 --> 00:36:58.620 align:middle line:84% what we call a filter bank or a set of different output 00:36:58.620 --> 00:36:59.560 align:middle line:90% convolutions. 00:36:59.560 --> 00:37:05.180 align:middle line:84% So here instead of just learning a single linear layer, 00:37:05.180 --> 00:37:08.000 align:middle line:84% you're going to learn two in parallel, and each of them 00:37:08.000 --> 00:37:11.320 align:middle line:84% get to build their own output dimension. 00:37:11.320 --> 00:37:12.720 align:middle line:84% And so now you have a filter bank 00:37:12.720 --> 00:37:14.920 align:middle line:84% of let's say C different filters. 00:37:14.920 --> 00:37:17.240 align:middle line:84% And you get a different output activation 00:37:17.240 --> 00:37:20.800 align:middle line:84% for each of those filters from the same input data. 00:37:20.800 --> 00:37:22.380 align:middle line:90% So then you can generalize this. 00:37:22.380 --> 00:37:24.680 align:middle line:84% And generally, in most of our convolutional layers 00:37:24.680 --> 00:37:30.140 align:middle line:84% in deep learning, we have multi-input and multi-output 00:37:30.140 --> 00:37:31.280 align:middle line:90% functions. 00:37:31.280 --> 00:37:36.040 align:middle line:84% And so here you'll have essentially your output 00:37:36.040 --> 00:37:38.560 align:middle line:84% is going to have some number of dimensions 00:37:38.560 --> 00:37:41.560 align:middle line:84% or some number of feature channels. 00:37:41.560 --> 00:37:45.000 align:middle line:84% The input will have some number of input, dimensions or feature 00:37:45.000 --> 00:37:45.940 align:middle line:90% channels. 00:37:45.940 --> 00:37:49.560 align:middle line:84% And then you're going to have some number of filters 00:37:49.560 --> 00:37:52.760 align:middle line:84% that you convolve with that input to get this output. 00:37:52.760 --> 00:37:54.600 align:middle line:84% And the number of features that you have, 00:37:54.600 --> 00:37:58.440 align:middle line:84% you're going to have the number of output dimensions, number 00:37:58.440 --> 00:38:01.280 align:middle line:90% of features or filters. 00:38:01.280 --> 00:38:03.860 align:middle line:84% Each of those filters is going to have some kernel size. 00:38:03.860 --> 00:38:07.210 align:middle line:84% That's K here, so some of spatial dimensions. 00:38:07.210 --> 00:38:09.410 align:middle line:84% And then for each of them, you're 00:38:09.410 --> 00:38:13.130 align:middle line:84% going to have enough channels to match your input data. 00:38:13.130 --> 00:38:15.090 align:middle line:84% So let me show a few more examples 00:38:15.090 --> 00:38:16.410 align:middle line:90% of what this looks like. 00:38:16.410 --> 00:38:18.850 align:middle line:84% So here's, for example, a bank of two filters 00:38:18.850 --> 00:38:21.970 align:middle line:84% where you have multiple inputs and multiple outputs. 00:38:21.970 --> 00:38:24.510 align:middle line:84% So here you have two input dimensions. 00:38:24.510 --> 00:38:28.970 align:middle line:84% You've now learned this linear sum for your height 00:38:28.970 --> 00:38:32.850 align:middle line:84% by width or this kernel size, which in this case 00:38:32.850 --> 00:38:35.210 align:middle line:90% looks like 3 by 3. 00:38:35.210 --> 00:38:37.542 align:middle line:84% So each of these sets of weights, F1 and F2, 00:38:37.542 --> 00:38:39.250 align:middle line:84% those are learned separately, but they're 00:38:39.250 --> 00:38:42.010 align:middle line:90% shared across the entire image. 00:38:42.010 --> 00:38:44.370 align:middle line:84% And so what those feature maps might actually 00:38:44.370 --> 00:38:50.730 align:middle line:84% look like, they get increasingly complex, I would say. 00:38:50.730 --> 00:38:54.210 align:middle line:90% So here, this is your input. 00:38:54.210 --> 00:38:59.170 align:middle line:84% This is using an AlexNet model, so a convolutional model. 00:38:59.170 --> 00:39:01.170 align:middle line:90% Now you take that input. 00:39:01.170 --> 00:39:03.310 align:middle line:84% After the first convolutional layer, 00:39:03.310 --> 00:39:06.070 align:middle line:84% this is the number of feature banks that are learned 00:39:06.070 --> 00:39:08.990 align:middle line:84% or output channels that are learned. 00:39:08.990 --> 00:39:14.710 align:middle line:84% This initial input is convolved with this many different 00:39:14.710 --> 00:39:15.627 align:middle line:90% convolutional filters. 00:39:15.627 --> 00:39:18.168 align:middle line:84% And so now we're showing all of those different filters, what 00:39:18.168 --> 00:39:20.310 align:middle line:84% actually activates when you send in this image. 00:39:20.310 --> 00:39:22.477 align:middle line:84% And so you can see, I don't know, maybe some of them 00:39:22.477 --> 00:39:24.610 align:middle line:84% are capturing things like certain types of edges, 00:39:24.610 --> 00:39:26.870 align:middle line:90% certain types of structures. 00:39:26.870 --> 00:39:29.170 align:middle line:84% And then after that second convolutional layer, 00:39:29.170 --> 00:39:32.790 align:middle line:84% these things start to get maybe a little bit more complex. 00:39:32.790 --> 00:39:35.150 align:middle line:84% They can often be quite difficult to interpret. 00:39:35.150 --> 00:39:38.310 align:middle line:84% But every layer can be thought of as a set of C feature maps 00:39:38.310 --> 00:39:39.470 align:middle line:90% or channels. 00:39:39.470 --> 00:39:42.270 align:middle line:84% And every feature map is essentially 00:39:42.270 --> 00:39:43.787 align:middle line:90% like structural representation. 00:39:43.787 --> 00:39:44.370 align:middle line:90% It's an image. 00:39:44.370 --> 00:39:47.190 align:middle line:90% It has height and width. 00:39:47.190 --> 00:39:47.830 align:middle line:90% Yeah? 00:39:47.830 --> 00:39:50.750 align:middle line:84% AUDIENCE: How do you ensure that each feature map is 00:39:50.750 --> 00:39:52.990 align:middle line:84% uniquely different so that you have 00:39:52.990 --> 00:39:54.732 align:middle line:90% a set of ones that can combine? 00:39:54.732 --> 00:39:56.190 align:middle line:84% SARA BEERY: So how do you make sure 00:39:56.190 --> 00:39:58.830 align:middle line:84% that all of the different feature maps are different? 00:39:58.830 --> 00:40:02.250 align:middle line:84% So that's not actually something that's innately built in, 00:40:02.250 --> 00:40:05.730 align:middle line:84% but one of the ways that people will do this 00:40:05.730 --> 00:40:09.930 align:middle line:84% is through different types of regularization in the model, 00:40:09.930 --> 00:40:11.610 align:middle line:84% so sometimes things like dropout that 00:40:11.610 --> 00:40:17.290 align:middle line:84% tries to make sure that you don't make the model too brittle 00:40:17.290 --> 00:40:20.130 align:middle line:90% or reliant on different things. 00:40:20.130 --> 00:40:24.370 align:middle line:84% But usually, the model will learn each of them 00:40:24.370 --> 00:40:27.610 align:middle line:84% uniquely because that gives it more capacity. 00:40:27.610 --> 00:40:30.210 align:middle line:84% There's nothing in here forcing it to keep them all the same. 00:40:30.210 --> 00:40:35.410 align:middle line:84% And so why would it not learn to maximize the types of variants 00:40:35.410 --> 00:40:38.050 align:middle line:90% that it could have. 00:40:38.050 --> 00:40:40.450 align:middle line:84% Depending on the data, and the architecture, 00:40:40.450 --> 00:40:43.170 align:middle line:84% and your optimization, you can have feature collapse 00:40:43.170 --> 00:40:45.998 align:middle line:84% sometimes where some stuff just gets ignored that you might not 00:40:45.998 --> 00:40:46.790 align:middle line:90% want to be ignored. 00:40:46.790 --> 00:40:48.998 align:middle line:84% And then you can try to build in different mechanisms 00:40:48.998 --> 00:40:50.810 align:middle line:90% to make that not happen. 00:40:50.810 --> 00:40:54.810 align:middle line:84% All right, so we're going to do like a simple little quiz. 00:40:54.810 --> 00:41:00.050 align:middle line:90% So here's a layer. 00:41:00.050 --> 00:41:01.790 align:middle line:90% We have this input. 00:41:01.790 --> 00:41:05.650 align:middle line:84% It's three channels, so RGB by 128 by 128. 00:41:05.650 --> 00:41:08.390 align:middle line:84% Very standard image size for an input. 00:41:08.390 --> 00:41:11.330 align:middle line:84% And then we have a filter bank full of 3 by 3 filters. 00:41:11.330 --> 00:41:13.785 align:middle line:84% So the spatial extent of those filters is 3 by 3. 00:41:13.785 --> 00:41:15.410 align:middle line:84% This is often the way we talk about it. 00:41:15.410 --> 00:41:16.850 align:middle line:84% So we're not talking about the channel dimensions. 00:41:16.850 --> 00:41:18.870 align:middle line:84% We just talk about the spatial extent 00:41:18.870 --> 00:41:20.670 align:middle line:90% of the size of the filters. 00:41:20.670 --> 00:41:25.710 align:middle line:84% And then our output is 96 channels by 128 00:41:25.710 --> 00:41:28.150 align:middle line:90% by 128, so by height and width. 00:41:28.150 --> 00:41:36.470 align:middle line:84% So, first, how many parameters does each of those filters have/ 00:41:36.470 --> 00:41:37.327 align:middle line:90% AUDIENCE: 9. 00:41:37.327 --> 00:41:37.910 align:middle line:90% SARA BEERY: 9. 00:41:37.910 --> 00:41:39.290 align:middle line:90% So why 9? 00:41:39.290 --> 00:41:40.750 align:middle line:90% AUDIENCE: It's 3 by 3. 00:41:40.750 --> 00:41:42.917 align:middle line:84% SARA BEERY: But how many input channels did we have? 00:41:42.917 --> 00:41:45.430 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:41:45.430 --> 00:41:46.910 align:middle line:90% SARA BEERY: All good. 00:41:46.910 --> 00:41:48.910 align:middle line:90% Yeah. 00:41:48.910 --> 00:41:52.810 align:middle line:84% Every single one of these filters has 27 parameters. 00:41:52.810 --> 00:41:54.970 align:middle line:84% We put this in because it's a little confusing. 00:41:54.970 --> 00:41:57.150 align:middle line:84% We talk about the filters being 3 by 3. 00:41:57.150 --> 00:41:58.930 align:middle line:90% But the thing is when-- 00:41:58.930 --> 00:42:00.230 align:middle line:90% this is just convention. 00:42:00.230 --> 00:42:02.290 align:middle line:84% When we talk about filter size, we often only 00:42:02.290 --> 00:42:04.305 align:middle line:84% talk about the spatial extent of the filters. 00:42:04.305 --> 00:42:05.930 align:middle line:84% And the assumption is we don't actually 00:42:05.930 --> 00:42:10.077 align:middle line:84% convolve in channel space, though, of course, you could. 00:42:10.077 --> 00:42:11.910 align:middle line:84% That's something you could totally build in. 00:42:11.910 --> 00:42:14.202 align:middle line:84% So maybe you have something like a hyperspectral input. 00:42:14.202 --> 00:42:16.890 align:middle line:84% You actually want convolution across the spectral bands. 00:42:16.890 --> 00:42:18.650 align:middle line:84% You could build that into your model. 00:42:18.650 --> 00:42:26.170 align:middle line:84% But standard, we often use independent per input channel 00:42:26.170 --> 00:42:27.530 align:middle line:90% weights that are learned. 00:42:27.530 --> 00:42:30.970 align:middle line:84% And so then we only talk about the spatial extent. 00:42:30.970 --> 00:42:33.710 align:middle line:90% So the filter banks are 3 by 3. 00:42:33.710 --> 00:42:39.630 align:middle line:84% Now how many filters are in our filter bank? 00:42:39.630 --> 00:42:40.508 align:middle line:90% Yeah? 00:42:40.508 --> 00:42:41.050 align:middle line:90% AUDIENCE: 96. 00:42:41.050 --> 00:42:43.450 align:middle line:90% SARA BEERY: 96, exactly. 00:42:43.450 --> 00:42:48.170 align:middle line:84% There are 96 outputs here, which means we get 96 filters. 00:42:48.170 --> 00:42:51.490 align:middle line:84% And then each of those filters will have those 27 parameters. 00:42:51.490 --> 00:42:55.210 align:middle line:84% So you're doing this-- for each of those 27 parameter input 00:42:55.210 --> 00:42:57.790 align:middle line:84% filters, you're running those over the entire image, 00:42:57.790 --> 00:43:00.902 align:middle line:84% and you are now taking a linear sum over all three 00:43:00.902 --> 00:43:03.110 align:middle line:84% of those channels, and you're getting a single output 00:43:03.110 --> 00:43:03.770 align:middle line:90% dimension. 00:43:03.770 --> 00:43:06.050 align:middle line:84% And so if you have 96 output channels, 00:43:06.050 --> 00:43:08.950 align:middle line:90% you need 96 different filters. 00:43:08.950 --> 00:43:10.640 align:middle line:90% Nice. 00:43:10.640 --> 00:43:11.490 align:middle line:90% Cool. 00:43:11.490 --> 00:43:19.530 align:middle line:84% So high level, filter sizes, if you're mapping from some x of l, 00:43:19.530 --> 00:43:23.990 align:middle line:84% so this current layer where you have C sub l channels 00:43:23.990 --> 00:43:27.350 align:middle line:84% and then N by M output dimensions 00:43:27.350 --> 00:43:31.550 align:middle line:84% and you're going to map to the next layer where you have some C 00:43:31.550 --> 00:43:34.230 align:middle line:84% sub L plus 1 output dimensions and N 00:43:34.230 --> 00:43:37.190 align:middle line:84% by M spatial dimensions and your filter 00:43:37.190 --> 00:43:41.310 align:middle line:84% has spatial extent K1 by K2, then the number 00:43:41.310 --> 00:43:46.750 align:middle line:84% of parameters per filter will be K1 times K2 times C sub l. 00:43:46.750 --> 00:43:49.310 align:middle line:84% And then the number of filters will just be C sub l 00:43:49.310 --> 00:43:52.550 align:middle line:84% plus 1, the number of output channels that you have. 00:43:52.550 --> 00:43:53.190 align:middle line:90% Yeah? 00:43:53.190 --> 00:43:58.730 align:middle line:84% AUDIENCE: So I [INAUDIBLE] square 00:43:58.730 --> 00:44:00.990 align:middle line:84% filters when dealing with convolutions. 00:44:00.990 --> 00:44:03.930 align:middle line:84% I'm just curious where you have rectangular filters 00:44:03.930 --> 00:44:05.530 align:middle line:90% or [INAUDIBLE]? 00:44:05.530 --> 00:44:07.970 align:middle line:84% SARA BEERY: Absolutely, normally, the filters 00:44:07.970 --> 00:44:08.690 align:middle line:90% are square. 00:44:08.690 --> 00:44:10.390 align:middle line:90% That is totally standard. 00:44:10.390 --> 00:44:11.930 align:middle line:90% They don't have to be. 00:44:11.930 --> 00:44:15.530 align:middle line:84% A convolution can be defined in a way that is not square. 00:44:15.530 --> 00:44:18.910 align:middle line:84% But I would say normally we see square convolutions. 00:44:18.910 --> 00:44:20.470 align:middle line:90% Yeah. 00:44:20.470 --> 00:44:22.290 align:middle line:90% Cool. 00:44:22.290 --> 00:44:26.970 align:middle line:84% So you have say your input image. 00:44:26.970 --> 00:44:30.010 align:middle line:84% Now you're going to run it through a first layer. 00:44:30.010 --> 00:44:32.350 align:middle line:84% So essentially, for each of your filters, 00:44:32.350 --> 00:44:36.950 align:middle line:84% you have a red, a green, and a blue filter. 00:44:36.950 --> 00:44:38.150 align:middle line:90% Those get applied. 00:44:38.150 --> 00:44:40.390 align:middle line:84% And then you can think about, OK, 00:44:40.390 --> 00:44:43.645 align:middle line:84% so now if you take just one of those filters, that's 00:44:43.645 --> 00:44:45.770 align:middle line:84% going to give you just a single one of those output 00:44:45.770 --> 00:44:49.650 align:middle line:84% maps in your feature maps at layer 1. 00:44:49.650 --> 00:44:51.050 align:middle line:84% A single one of those output maps 00:44:51.050 --> 00:44:54.870 align:middle line:84% is going to correspond to one of those filters over RGB. 00:44:54.870 --> 00:44:57.690 align:middle line:84% And now if you run that a second layer, 00:44:57.690 --> 00:45:00.150 align:middle line:84% second convolutional layer, now you 00:45:00.150 --> 00:45:03.050 align:middle line:84% have layer 2, your whole sets of features. 00:45:03.050 --> 00:45:05.290 align:middle line:84% Now each of those are going to be-- instead of RGB, 00:45:05.290 --> 00:45:09.110 align:middle line:84% they're going to be across the entire set of feature maps 00:45:09.110 --> 00:45:10.590 align:middle line:90% in your first layer. 00:45:10.590 --> 00:45:13.910 align:middle line:84% But then one of the outputs after that will only 00:45:13.910 --> 00:45:17.190 align:middle line:84% correspond to a single one of those C2 filters. 00:45:17.190 --> 00:45:19.670 align:middle line:84% So you can see how the actual representation gets 00:45:19.670 --> 00:45:23.110 align:middle line:90% highly complex, highly quickly. 00:45:23.110 --> 00:45:25.130 align:middle line:84% Even just one or two layers of depth, 00:45:25.130 --> 00:45:27.270 align:middle line:84% when you have convolution, it can 00:45:27.270 --> 00:45:30.050 align:middle line:90% map really complex functions. 00:45:30.050 --> 00:45:32.710 align:middle line:90% 00:45:32.710 --> 00:45:34.990 align:middle line:84% So another thing that we often use 00:45:34.990 --> 00:45:37.310 align:middle line:84% when we're building convolutional neural networks 00:45:37.310 --> 00:45:39.910 align:middle line:90% is this idea of pooling. 00:45:39.910 --> 00:45:44.650 align:middle line:84% So here, what we have is we have this convolutional layer. 00:45:44.650 --> 00:45:47.830 align:middle line:84% So you have your convolutional weights, you have your bias, 00:45:47.830 --> 00:45:50.790 align:middle line:84% and then you have your point-wise non-linearity. 00:45:50.790 --> 00:45:53.290 align:middle line:84% But then one of the things that you can do 00:45:53.290 --> 00:45:55.650 align:middle line:84% is you can do what's called pooling over those output 00:45:55.650 --> 00:45:56.310 align:middle line:90% features. 00:45:56.310 --> 00:45:57.730 align:middle line:84% One of the reasons you might want to do that 00:45:57.730 --> 00:45:59.813 align:middle line:84% is because you can see that the dimensionality can 00:45:59.813 --> 00:46:02.210 align:middle line:90% get really large, really fast. 00:46:02.210 --> 00:46:04.730 align:middle line:84% And so one thing, very, very common thing, 00:46:04.730 --> 00:46:07.410 align:middle line:84% is something we call max pooling, where here you 00:46:07.410 --> 00:46:12.850 align:middle line:84% just take some local window and you just use the maximum value. 00:46:12.850 --> 00:46:15.610 align:middle line:84% And you use that to represent your output. 00:46:15.610 --> 00:46:18.730 align:middle line:84% Or you can also think about something like mean pooling. 00:46:18.730 --> 00:46:20.930 align:middle line:84% And here you're just taking the average 00:46:20.930 --> 00:46:23.450 align:middle line:90% over some smaller local window. 00:46:23.450 --> 00:46:24.830 align:middle line:84% So why do you want to do pooling? 00:46:24.830 --> 00:46:26.497 align:middle line:84% So one thing that's really nice about it 00:46:26.497 --> 00:46:28.690 align:middle line:84% is pooling across spatial locations 00:46:28.690 --> 00:46:31.130 align:middle line:90% gives you some stability. 00:46:31.130 --> 00:46:39.050 align:middle line:84% So for example, here, if you're taking the max over this window, 00:46:39.050 --> 00:46:42.150 align:middle line:84% you're still going to identify that there's an edge there. 00:46:42.150 --> 00:46:47.250 align:middle line:84% There's still this white value that will get passed through, 00:46:47.250 --> 00:46:51.880 align:middle line:84% and that will be stable as the inputs move. 00:46:51.880 --> 00:46:55.100 align:middle line:84% So you'll get a large response to something 00:46:55.100 --> 00:46:56.560 align:middle line:90% like an edge filter. 00:46:56.560 --> 00:46:59.220 align:middle line:84% Regardless of the exact position of that edge, 00:46:59.220 --> 00:47:01.520 align:middle line:84% you get some more stability in the outputs. 00:47:01.520 --> 00:47:07.220 align:middle line:90% 00:47:07.220 --> 00:47:10.940 align:middle line:84% And then sometimes we also will do pooling across channels. 00:47:10.940 --> 00:47:12.860 align:middle line:84% And one cool thing you can think about here 00:47:12.860 --> 00:47:14.540 align:middle line:84% is if you pool across feature channels 00:47:14.540 --> 00:47:17.720 align:middle line:84% where all the feature channels are looking for those edges 00:47:17.720 --> 00:47:20.260 align:middle line:84% but across different orientations of that edge 00:47:20.260 --> 00:47:24.800 align:middle line:84% filter, then you can get some new kinds of invariances. 00:47:24.800 --> 00:47:27.420 align:middle line:84% So for example, this would give you 00:47:27.420 --> 00:47:30.740 align:middle line:84% a filter that was finding edges and was 00:47:30.740 --> 00:47:33.620 align:middle line:84% invariant to the orientation of the edge. 00:47:33.620 --> 00:47:36.540 align:middle line:90% Seems nice. 00:47:36.540 --> 00:47:37.460 align:middle line:90% Cool. 00:47:37.460 --> 00:47:42.240 align:middle line:84% So now if we actually look at a lot of the time 00:47:42.240 --> 00:47:44.380 align:middle line:84% what computation and neural network looks like, 00:47:44.380 --> 00:47:48.180 align:middle line:84% a convolutional neural network, we will usually take an input, 00:47:48.180 --> 00:47:51.660 align:middle line:84% and then we will run it through stacks of this building block. 00:47:51.660 --> 00:47:55.080 align:middle line:84% And this building block is some computation 00:47:55.080 --> 00:47:59.920 align:middle line:84% over some set of filters, a nonlinear activation, so a ReLU, 00:47:59.920 --> 00:48:01.640 align:middle line:90% and then something like pooling. 00:48:01.640 --> 00:48:05.200 align:middle line:84% And then we'll repeat that over and over and over again. 00:48:05.200 --> 00:48:08.457 align:middle line:84% But this is a really large network 00:48:08.457 --> 00:48:09.540 align:middle line:90% that has a lot of density. 00:48:09.540 --> 00:48:12.620 align:middle line:84% And as we maybe add more feature channels as we keep going, 00:48:12.620 --> 00:48:13.998 align:middle line:90% this could get really big. 00:48:13.998 --> 00:48:16.040 align:middle line:84% So particularly, if you're trying to do something 00:48:16.040 --> 00:48:19.400 align:middle line:84% like classification, where the only thing you want to get out 00:48:19.400 --> 00:48:22.420 align:middle line:84% is a low rank representation of the input, 00:48:22.420 --> 00:48:27.360 align:middle line:84% just something like a word or a specific one hot encoding, here 00:48:27.360 --> 00:48:30.300 align:middle line:84% we actually often use something called downsampling, 00:48:30.300 --> 00:48:32.320 align:middle line:84% where we assume that we don't actually 00:48:32.320 --> 00:48:37.320 align:middle line:84% need that high-dimensional input structure at every dimension 00:48:37.320 --> 00:48:38.300 align:middle line:90% of the network. 00:48:38.300 --> 00:48:40.500 align:middle line:84% And so here, instead of just pooling, 00:48:40.500 --> 00:48:42.640 align:middle line:90% we might also downsample. 00:48:42.640 --> 00:48:44.240 align:middle line:90% And what that looks like-- 00:48:44.240 --> 00:48:47.440 align:middle line:84% so here if you have pooling, you've now done something 00:48:47.440 --> 00:48:51.720 align:middle line:84% like your convolutional layer, your activation function, 00:48:51.720 --> 00:48:53.860 align:middle line:84% and then something like max pooling, 00:48:53.860 --> 00:48:56.680 align:middle line:84% then downsampling would look something like this. 00:48:56.680 --> 00:48:59.940 align:middle line:84% So maybe you would only keep some subset 00:48:59.940 --> 00:49:03.140 align:middle line:90% of those actual values. 00:49:03.140 --> 00:49:05.700 align:middle line:84% And then instead of doing the pooling across all of them, 00:49:05.700 --> 00:49:10.180 align:middle line:84% you can also just do just straight downsampling. 00:49:10.180 --> 00:49:12.020 align:middle line:84% And one way that you can actually 00:49:12.020 --> 00:49:16.020 align:middle line:84% do both pooling and downsampling together 00:49:16.020 --> 00:49:19.100 align:middle line:84% is through something called a strided operation. 00:49:19.100 --> 00:49:21.940 align:middle line:84% So this actually combines something like convolution 00:49:21.940 --> 00:49:23.720 align:middle line:90% or pooling and downsampling. 00:49:23.720 --> 00:49:26.900 align:middle line:84% And the way that it does that is instead of going and sliding 00:49:26.900 --> 00:49:31.420 align:middle line:84% your filter by a single pixel every single time, instead, 00:49:31.420 --> 00:49:34.820 align:middle line:84% you increase what we call the stride of the filter. 00:49:34.820 --> 00:49:38.940 align:middle line:84% So now you operate your convolution. 00:49:38.940 --> 00:49:41.500 align:middle line:84% And then instead of just moving one, you might move two 00:49:41.500 --> 00:49:43.317 align:middle line:90% or you might move five. 00:49:43.317 --> 00:49:44.900 align:middle line:84% In this particular example, here we're 00:49:44.900 --> 00:49:46.880 align:middle line:84% showing a stride that's quite big. 00:49:46.880 --> 00:49:49.760 align:middle line:84% Usually, for these models, we often 00:49:49.760 --> 00:49:52.120 align:middle line:84% have that stride be somewhat overlapping, 00:49:52.120 --> 00:49:54.240 align:middle line:84% so you don't want to lose any of the context 00:49:54.240 --> 00:49:55.323 align:middle line:90% from the original input. 00:49:55.323 --> 00:49:57.740 align:middle line:84% So we'll have that stride be something that's overlapping. 00:49:57.740 --> 00:50:01.420 align:middle line:84% But maybe your input kernel is like 7 by 7. 00:50:01.420 --> 00:50:05.040 align:middle line:84% Your stride might be 5, something like that. 00:50:05.040 --> 00:50:11.960 align:middle line:84% And so here you're combining that pooling, that convolution, 00:50:11.960 --> 00:50:15.400 align:middle line:84% and the downsampling into a single element. 00:50:15.400 --> 00:50:17.880 align:middle line:84% And you can also think about doing something that we 00:50:17.880 --> 00:50:20.600 align:middle line:90% call using a dilated filter. 00:50:20.600 --> 00:50:22.520 align:middle line:84% And essentially, all this is it's just 00:50:22.520 --> 00:50:25.880 align:middle line:84% a filter where we've set some of the values to 0 and baked that 00:50:25.880 --> 00:50:27.240 align:middle line:90% into our model. 00:50:27.240 --> 00:50:30.515 align:middle line:84% And this gives you a larger receptive field, so what 00:50:30.515 --> 00:50:31.640 align:middle line:90% I was talking about before. 00:50:31.640 --> 00:50:34.120 align:middle line:84% You get more information from a larger spatial context 00:50:34.120 --> 00:50:36.620 align:middle line:84% or spatial extent with fewer parameters. 00:50:36.620 --> 00:50:37.860 align:middle line:90% There's less flexibility. 00:50:37.860 --> 00:50:39.580 align:middle line:84% There's less parameters that are learned, 00:50:39.580 --> 00:50:44.080 align:middle line:84% but you're kind of getting that larger context window. 00:50:44.080 --> 00:50:48.940 align:middle line:84% And that's kind of a trade-off that you can optimize for. 00:50:48.940 --> 00:50:50.220 align:middle line:90% Yeah? 00:50:50.220 --> 00:50:51.380 align:middle line:90% AUDIENCE: Oh, sorry. 00:50:51.380 --> 00:50:55.820 align:middle line:84% So when you downsample, you cannot go back to the previous 00:50:55.820 --> 00:50:59.160 align:middle line:84% dimension to input dimension unless you give information 00:50:59.160 --> 00:51:01.597 align:middle line:90% across the network, right? 00:51:01.597 --> 00:51:03.680 align:middle line:84% SARA BEERY: We'll talk about that in a little bit. 00:51:03.680 --> 00:51:05.940 align:middle line:84% So the question was if you downsample, 00:51:05.940 --> 00:51:08.220 align:middle line:84% you can't get back to the original input dimension 00:51:08.220 --> 00:51:10.880 align:middle line:84% unless you add some more information. 00:51:10.880 --> 00:51:12.940 align:middle line:90% That's not necessarily true. 00:51:12.940 --> 00:51:14.852 align:middle line:84% But you might not do it very well. 00:51:14.852 --> 00:51:16.560 align:middle line:84% So we'll talk about that in a little bit. 00:51:16.560 --> 00:51:18.140 align:middle line:84% AUDIENCE: What are the current norms 00:51:18.140 --> 00:51:25.685 align:middle line:84% for when it's acceptable to use striding [INAUDIBLE]? 00:51:25.685 --> 00:51:28.060 align:middle line:84% SARA BEERY: What are the current norms for when you would 00:51:28.060 --> 00:51:29.980 align:middle line:90% or would not use a stride? 00:51:29.980 --> 00:51:34.180 align:middle line:84% So I would say that this search space of possible architectures 00:51:34.180 --> 00:51:35.540 align:middle line:90% is vast. 00:51:35.540 --> 00:51:38.180 align:middle line:84% And even if we only consider the building blocks 00:51:38.180 --> 00:51:40.980 align:middle line:84% we've talked about so far, which is like linear layers 00:51:40.980 --> 00:51:42.980 align:middle line:90% and convolutional layers-- 00:51:42.980 --> 00:51:45.100 align:middle line:84% and then you can have varied input sizes, 00:51:45.100 --> 00:51:46.860 align:middle line:84% varied depth, various numbers of channels. 00:51:46.860 --> 00:51:49.152 align:middle line:84% All of a sudden, you already have a really large search 00:51:49.152 --> 00:51:50.240 align:middle line:90% space. 00:51:50.240 --> 00:51:51.800 align:middle line:84% There's entire fields of research 00:51:51.800 --> 00:51:53.800 align:middle line:84% called neural architecture search that actually 00:51:53.800 --> 00:51:58.440 align:middle line:84% specifically tried to optimize over the space of architectures 00:51:58.440 --> 00:52:00.220 align:middle line:84% to find the best possible solution. 00:52:00.220 --> 00:52:03.720 align:middle line:84% Pretty computationally expensive to actually train those. 00:52:03.720 --> 00:52:08.840 align:middle line:84% But I would say that one of the things that's kind of proven out 00:52:08.840 --> 00:52:13.220 align:middle line:84% in the literature over the last of maybe 10 years 00:52:13.220 --> 00:52:15.000 align:middle line:84% just experimentally is there seem 00:52:15.000 --> 00:52:20.440 align:middle line:84% to be some standards that seem to be trends that are long held. 00:52:20.440 --> 00:52:22.940 align:middle line:84% So for example, in the original AlexNet paper, 00:52:22.940 --> 00:52:24.400 align:middle line:90% the filters were quite big. 00:52:24.400 --> 00:52:26.080 align:middle line:84% They had pretty large spatial extent, 00:52:26.080 --> 00:52:28.320 align:middle line:84% and then there were less depth in the layers. 00:52:28.320 --> 00:52:29.800 align:middle line:84% One of the things that we've seen more and more 00:52:29.800 --> 00:52:32.160 align:middle line:84% is now we actually often see filters that are reasonably 00:52:32.160 --> 00:52:37.080 align:middle line:90% small, 7 or 5 spatial extent. 00:52:37.080 --> 00:52:40.620 align:middle line:84% But maybe then they're actually stacked quite deep. 00:52:40.620 --> 00:52:43.400 align:middle line:84% So then you end up with really large receptive fields, 00:52:43.400 --> 00:52:48.140 align:middle line:84% even though the initial filters were spatially quite small. 00:52:48.140 --> 00:52:49.740 align:middle line:84% And then with striding, again, it's 00:52:49.740 --> 00:52:52.420 align:middle line:84% just that trade off between the amount of computation 00:52:52.420 --> 00:52:53.840 align:middle line:90% and the receptive field. 00:52:53.840 --> 00:52:57.182 align:middle line:84% So there's different kind of optimality. 00:52:57.182 --> 00:52:59.140 align:middle line:84% One of the things you can do is you can go look 00:52:59.140 --> 00:53:01.915 align:middle line:84% at a bunch of the standard architectures, 00:53:01.915 --> 00:53:04.040 align:middle line:84% and often they'll have these architecture diagrams. 00:53:04.040 --> 00:53:07.980 align:middle line:84% You've seen them before probably where it's like the layer, 00:53:07.980 --> 00:53:10.620 align:middle line:84% and then it'll have just like a layer size for each thing. 00:53:10.620 --> 00:53:13.420 align:middle line:84% And you can look at, what are the similarities 00:53:13.420 --> 00:53:15.640 align:middle line:84% across the structures of these different models? 00:53:15.640 --> 00:53:17.560 align:middle line:84% AUDIENCE: Does the complexity of the picture matter at all, 00:53:17.560 --> 00:53:19.060 align:middle line:84% for example, the first example, you 00:53:19.060 --> 00:53:22.620 align:middle line:84% gave, which is Times Square versus the nature? 00:53:22.620 --> 00:53:24.420 align:middle line:84% SARA BEERY: Yeah, experimentally, yes. 00:53:24.420 --> 00:53:29.580 align:middle line:84% So if you have really non-complex pictures like CIFAR, 00:53:29.580 --> 00:53:33.860 align:middle line:84% for example, these 100 by 100 really simple photos, 00:53:33.860 --> 00:53:38.780 align:middle line:84% then the network itself often needs to be much less complex. 00:53:38.780 --> 00:53:41.080 align:middle line:84% If fine-grained feature details actually 00:53:41.080 --> 00:53:42.800 align:middle line:84% matter a lot, which happens for us a lot 00:53:42.800 --> 00:53:45.060 align:middle line:84% with the species images, for example, 00:53:45.060 --> 00:53:46.560 align:middle line:84% then one of the things that actually 00:53:46.560 --> 00:53:48.840 align:middle line:84% tends to matter more than almost anything else 00:53:48.840 --> 00:53:51.160 align:middle line:84% is the input image size that you use. 00:53:51.160 --> 00:53:54.020 align:middle line:84% So a lot of these models, the input image size, 00:53:54.020 --> 00:53:55.960 align:middle line:84% if you scale it up, then the computation 00:53:55.960 --> 00:53:59.720 align:middle line:90% scales up quite a lot with that. 00:53:59.720 --> 00:54:02.607 align:middle line:84% So we'll often downsample images to some smaller size 00:54:02.607 --> 00:54:04.440 align:middle line:84% before we run them through a network as part 00:54:04.440 --> 00:54:05.720 align:middle line:90% of the pre-processing. 00:54:05.720 --> 00:54:08.877 align:middle line:84% But we find that increasing the scale of that input image 00:54:08.877 --> 00:54:10.960 align:middle line:84% and then maybe downscaling some of the computation 00:54:10.960 --> 00:54:12.260 align:middle line:90% is sometimes a better trade-off. 00:54:12.260 --> 00:54:14.218 align:middle line:84% If you really care about fine-grained features, 00:54:14.218 --> 00:54:16.920 align:middle line:84% you don't want to lose that detail. 00:54:16.920 --> 00:54:18.040 align:middle line:90% Yeah? 00:54:18.040 --> 00:54:20.480 align:middle line:84% AUDIENCE: So normally, in signal processing, 00:54:20.480 --> 00:54:24.540 align:middle line:84% we're [INAUDIBLE] filters for a given purpose. 00:54:24.540 --> 00:54:25.510 align:middle line:90% [INAUDIBLE] 00:54:25.510 --> 00:54:28.800 align:middle line:90% 00:54:28.800 --> 00:54:35.040 align:middle line:84% And in this case, the weights, that would be [INAUDIBLE]. 00:54:35.040 --> 00:54:39.790 align:middle line:84% You mentioned earlier the model will learn [INAUDIBLE]. 00:54:39.790 --> 00:54:41.850 align:middle line:90% So how would you-- 00:54:41.850 --> 00:54:45.210 align:middle line:84% or do we include inductive biases 00:54:45.210 --> 00:54:47.210 align:middle line:84% in how we train our models so that we 00:54:47.210 --> 00:54:50.430 align:middle line:84% can design our models to then do a certain purpose, 00:54:50.430 --> 00:54:56.290 align:middle line:84% no different than designing an [INAUDIBLE] for a signal? 00:54:56.290 --> 00:54:57.723 align:middle line:90% How does the model [INAUDIBLE]? 00:54:57.723 --> 00:54:59.390 align:middle line:84% SARA BEERY: Yeah, it's a great question. 00:54:59.390 --> 00:55:02.030 align:middle line:84% So the question was in signal processing, 00:55:02.030 --> 00:55:06.030 align:middle line:84% we're often very carefully handcrafting our filters 00:55:06.030 --> 00:55:08.690 align:middle line:84% so that they are capturing a certain dimension of the signal 00:55:08.690 --> 00:55:11.370 align:middle line:84% that we want to capture, whereas here we're 00:55:11.370 --> 00:55:13.970 align:middle line:84% talking about filters that are actually being learned. 00:55:13.970 --> 00:55:15.510 align:middle line:90% And what's the trade-off? 00:55:15.510 --> 00:55:18.570 align:middle line:84% When does it make sense to try to actually craft 00:55:18.570 --> 00:55:22.490 align:middle line:84% the filters to match our intuition about the problem? 00:55:22.490 --> 00:55:25.490 align:middle line:84% So what you're talking about is actually this massive paradigm 00:55:25.490 --> 00:55:29.810 align:middle line:84% shift that happened maybe 10, 15 years ago in the field where 00:55:29.810 --> 00:55:34.610 align:middle line:84% we went from most of the work was 00:55:34.610 --> 00:55:38.248 align:middle line:84% on what we called feature engineering in computer vision 00:55:38.248 --> 00:55:40.790 align:middle line:84% or in deep learning, where you would be building these really 00:55:40.790 --> 00:55:44.750 align:middle line:84% handcrafted structures that then you would combine together 00:55:44.750 --> 00:55:48.390 align:middle line:84% into some larger system that would recognize stuff. 00:55:48.390 --> 00:55:51.490 align:middle line:84% And when we started to see that filters, 00:55:51.490 --> 00:55:54.370 align:middle line:84% just like these models being learned end to end 00:55:54.370 --> 00:55:58.630 align:middle line:84% were more predictive, it was a bit of like a catastrophe 00:55:58.630 --> 00:56:00.210 align:middle line:84% emotionally for many researchers. 00:56:00.210 --> 00:56:02.030 align:middle line:84% Because you were like, no, like I 00:56:02.030 --> 00:56:04.470 align:middle line:84% spent my life learning how to craft these filters 00:56:04.470 --> 00:56:06.570 align:middle line:90% in this really specific way. 00:56:06.570 --> 00:56:08.670 align:middle line:84% I don't want to just cram a bunch of data 00:56:08.670 --> 00:56:10.805 align:middle line:84% through a structure and have that be learned 00:56:10.805 --> 00:56:11.930 align:middle line:90% and then have it be better. 00:56:11.930 --> 00:56:16.950 align:middle line:84% And it turns out that actually the trade-off often comes 00:56:16.950 --> 00:56:18.630 align:middle line:90% down to how much data you have. 00:56:18.630 --> 00:56:21.450 align:middle line:84% So if you don't have enough data to learn the filters, 00:56:21.450 --> 00:56:26.430 align:middle line:84% well, then more handcrafting, more knowledge, more inductive 00:56:26.430 --> 00:56:30.270 align:middle line:84% bias tends to be better because you actually 00:56:30.270 --> 00:56:31.890 align:middle line:84% can put more constraints on the system 00:56:31.890 --> 00:56:33.950 align:middle line:84% so that you can optimize better with less data. 00:56:33.950 --> 00:56:37.450 align:middle line:84% But if you're in a space where you can get huge amounts of data 00:56:37.450 --> 00:56:40.690 align:middle line:84% to learn from, often, we don't know as much 00:56:40.690 --> 00:56:43.530 align:middle line:84% as we think we know about what optimality might be in terms 00:56:43.530 --> 00:56:45.530 align:middle line:84% of the filters that are learned, especially when 00:56:45.530 --> 00:56:47.738 align:middle line:84% you think about-- it can be really intuitive to think 00:56:47.738 --> 00:56:49.370 align:middle line:90% about crafting a single filter. 00:56:49.370 --> 00:56:54.530 align:middle line:84% But how would you handcraft like the seventh convolutional layer 00:56:54.530 --> 00:56:58.590 align:middle line:84% in a 128 layer network for a single channel? 00:56:58.590 --> 00:57:02.378 align:middle line:84% It very rapidly gets to a place where the system is so complex 00:57:02.378 --> 00:57:04.170 align:middle line:84% that it's very difficult to build intuition 00:57:04.170 --> 00:57:06.950 align:middle line:90% about what it should be for us. 00:57:06.950 --> 00:57:07.450 align:middle line:90% Cool. 00:57:07.450 --> 00:57:10.970 align:middle line:90% 00:57:10.970 --> 00:57:12.110 align:middle line:90% So receptive fields. 00:57:12.110 --> 00:57:14.170 align:middle line:84% So one of the things we talked about 00:57:14.170 --> 00:57:18.410 align:middle line:84% is if you let of input size of the patch get too small, 00:57:18.410 --> 00:57:20.290 align:middle line:84% you maybe won't be able to actually identify 00:57:20.290 --> 00:57:21.890 align:middle line:90% the things you care about. 00:57:21.890 --> 00:57:24.970 align:middle line:84% So in this particular instance, say maybe the input 00:57:24.970 --> 00:57:26.890 align:middle line:90% patch is this entire bird. 00:57:26.890 --> 00:57:30.930 align:middle line:84% And then we're running it through convolutional network, 00:57:30.930 --> 00:57:32.590 align:middle line:90% ReLU, convolutional layer. 00:57:32.590 --> 00:57:34.830 align:middle line:90% And then we get some output. 00:57:34.830 --> 00:57:37.090 align:middle line:84% And that output for that patch is bird. 00:57:37.090 --> 00:57:41.230 align:middle line:84% So here what we would say is the receptive field 00:57:41.230 --> 00:57:46.532 align:middle line:84% of that x2 output is the input patch there. 00:57:46.532 --> 00:57:47.990 align:middle line:84% Because receptive field essentially 00:57:47.990 --> 00:57:50.870 align:middle line:84% just says what parts of the initial input 00:57:50.870 --> 00:57:54.050 align:middle line:84% have any influence on this part of the output. 00:57:54.050 --> 00:57:55.470 align:middle line:84% Because we're doing this in a way 00:57:55.470 --> 00:57:57.170 align:middle line:90% that's translation invariant. 00:57:57.170 --> 00:58:00.510 align:middle line:84% We're doing the same computation across different patches. 00:58:00.510 --> 00:58:03.590 align:middle line:84% Outside of that receptive field, this model 00:58:03.590 --> 00:58:05.610 align:middle line:84% doesn't have any input information. 00:58:05.610 --> 00:58:08.110 align:middle line:84% So if there was something just outside of that field of view 00:58:08.110 --> 00:58:11.255 align:middle line:84% that was really informative and it wasn't actually in here, 00:58:11.255 --> 00:58:12.630 align:middle line:84% then there's no way for the model 00:58:12.630 --> 00:58:15.550 align:middle line:84% to get information outside of that receptive field. 00:58:15.550 --> 00:58:19.370 align:middle line:84% You can also talk about the receptive field of input layers. 00:58:19.370 --> 00:58:22.210 align:middle line:84% So you can say, OK, like in this first layer, 00:58:22.210 --> 00:58:24.510 align:middle line:84% the receptive field of that component 00:58:24.510 --> 00:58:28.550 align:middle line:84% of the output of that first layer is that part of the input. 00:58:28.550 --> 00:58:34.450 align:middle line:84% And what that looks like-- as you pass through the network, 00:58:34.450 --> 00:58:36.870 align:middle line:84% you can think about looking at the activations, 00:58:36.870 --> 00:58:40.370 align:middle line:84% and you can see that they go from being very fine grained 00:58:40.370 --> 00:58:42.110 align:middle line:84% or very detailed in the early layers 00:58:42.110 --> 00:58:46.130 align:middle line:84% because maybe you're running them through convolutions 00:58:46.130 --> 00:58:47.550 align:middle line:90% that are much more local. 00:58:47.550 --> 00:58:51.710 align:middle line:84% And then as you get further out or further along in AlexNet, 00:58:51.710 --> 00:58:54.410 align:middle line:84% VGG 16, ResNet 18, what you start to see 00:58:54.410 --> 00:58:59.010 align:middle line:84% are feature activations that are kind of much more diffuse. 00:58:59.010 --> 00:59:00.930 align:middle line:84% And this has to do with that receptive field 00:59:00.930 --> 00:59:03.690 align:middle line:84% because now each one of these output dimensions 00:59:03.690 --> 00:59:06.530 align:middle line:84% is actually collecting really complex information 00:59:06.530 --> 00:59:08.970 align:middle line:84% and mapping it through a bunch of different layers 00:59:08.970 --> 00:59:10.490 align:middle line:90% from this input. 00:59:10.490 --> 00:59:13.810 align:middle line:84% So maybe you're getting output feature activations 00:59:13.810 --> 00:59:18.130 align:middle line:84% that are more semantically meaningful but less 00:59:18.130 --> 00:59:24.370 align:middle line:84% affected by subtle texture or variation in the initial input 00:59:24.370 --> 00:59:26.130 align:middle line:90% image possibly. 00:59:26.130 --> 00:59:27.430 align:middle line:90% Yeah? 00:59:27.430 --> 00:59:31.967 align:middle line:84% AUDIENCE: Similar computation to how you do Gaussian blurring? 00:59:31.967 --> 00:59:33.550 align:middle line:84% SARA BEERY: So the question is if it's 00:59:33.550 --> 00:59:35.842 align:middle line:84% similar in computation to how you do Gaussian blurring. 00:59:35.842 --> 00:59:38.550 align:middle line:90% 00:59:38.550 --> 00:59:41.953 align:middle line:84% I think at a high level, like the intuition 00:59:41.953 --> 00:59:43.870 align:middle line:84% is a little similar, but computationally, it's 00:59:43.870 --> 00:59:45.070 align:middle line:90% quite different. 00:59:45.070 --> 00:59:45.790 align:middle line:90% Yeah? 00:59:45.790 --> 00:59:49.710 align:middle line:84% AUDIENCE: In that last image, are those, 1, 6, and 8, 00:59:49.710 --> 00:59:55.830 align:middle line:84% are-- the ones on the right, are those the same pixel size 00:59:55.830 --> 00:59:59.710 align:middle line:84% as the initial inputs, or are they downsampled and you just 00:59:59.710 --> 01:00:01.210 align:middle line:90% resize them to all be the same? 01:00:01.210 --> 01:00:02.630 align:middle line:84% SARA BEERY: We've resized them all to be the same size. 01:00:02.630 --> 01:00:04.880 align:middle line:84% But it actually is very dependent on the architecture. 01:00:04.880 --> 01:00:09.110 align:middle line:84% So ResNets, the actual spatial extent, once you get to block 8 01:00:09.110 --> 01:00:10.090 align:middle line:90% is much lower. 01:00:10.090 --> 01:00:13.990 align:middle line:84% And then there's a lot more channels versus the AlexNet one. 01:00:13.990 --> 01:00:14.850 align:middle line:90% Yeah. 01:00:14.850 --> 01:00:15.575 align:middle line:90% Cool. 01:00:15.575 --> 01:00:16.950 align:middle line:84% So in the last little bit, I want 01:00:16.950 --> 01:00:19.070 align:middle line:84% to talk through at a pretty high level 01:00:19.070 --> 01:00:21.830 align:middle line:84% a bunch of these different high-level ideas, maybe some 01:00:21.830 --> 01:00:25.110 align:middle line:84% of these what one of you mentioned, some of these things 01:00:25.110 --> 01:00:32.210 align:middle line:84% that have tended to continue to be used over time. 01:00:32.210 --> 01:00:35.570 align:middle line:84% So we talked about one very, very classic version 01:00:35.570 --> 01:00:37.170 align:middle line:84% of a convolutional neural network 01:00:37.170 --> 01:00:40.610 align:middle line:84% is something like a classification network. 01:00:40.610 --> 01:00:42.930 align:middle line:84% And most classification networks have 01:00:42.930 --> 01:00:44.510 align:middle line:90% something like this structure. 01:00:44.510 --> 01:00:48.410 align:middle line:84% We'll have filters, some sort of point-wise non-linearity, 01:00:48.410 --> 01:00:50.230 align:middle line:84% and then some form of downsampling. 01:00:50.230 --> 01:00:52.147 align:middle line:84% And we run through a bunch of different layers 01:00:52.147 --> 01:00:55.130 align:middle line:84% like that to get out some of output classification. 01:00:55.130 --> 01:00:59.210 align:middle line:84% But another really common structure in using convolutions 01:00:59.210 --> 01:01:03.610 align:middle line:84% is what we call an encoder-decoder architecture. 01:01:03.610 --> 01:01:06.410 align:middle line:84% So here what you're doing is you're taking that input 01:01:06.410 --> 01:01:09.970 align:middle line:84% and you're doing convolution, non-linearity, subsampling 01:01:09.970 --> 01:01:12.850 align:middle line:84% or downsampling to get down to some very 01:01:12.850 --> 01:01:16.690 align:middle line:84% low dimensional representation, for example this z space vector. 01:01:16.690 --> 01:01:19.350 align:middle line:84% And then in the decoder component, 01:01:19.350 --> 01:01:22.070 align:middle line:84% you're going to take that low dimensional representation, 01:01:22.070 --> 01:01:25.570 align:middle line:84% and you're going to now do convolution non-linearity 01:01:25.570 --> 01:01:27.170 align:middle line:90% upsampling. 01:01:27.170 --> 01:01:29.030 align:middle line:84% So in that upsampling component, you're 01:01:29.030 --> 01:01:31.790 align:middle line:84% actually just increasing the spatial extent of each 01:01:31.790 --> 01:01:33.190 align:middle line:90% of the layers going forward. 01:01:33.190 --> 01:01:37.830 align:middle line:84% And the training signal here is you want the decoded output 01:01:37.830 --> 01:01:42.710 align:middle line:84% to match the encoded input as close as possible. 01:01:42.710 --> 01:01:46.190 align:middle line:90% Now this is also used. 01:01:46.190 --> 01:01:48.270 align:middle line:84% So this is used as one way to get 01:01:48.270 --> 01:01:51.310 align:middle line:84% a low-dimensional representation of an image. 01:01:51.310 --> 01:01:54.110 align:middle line:84% This type of encoder-decoder architecture, this 01:01:54.110 --> 01:01:57.530 align:middle line:84% is built into things like variational autoencoders. 01:01:57.530 --> 01:02:00.590 align:middle line:84% Masked autoencoders are increasingly widely used, 01:02:00.590 --> 01:02:04.990 align:middle line:84% though those use attention as opposed 01:02:04.990 --> 01:02:08.270 align:middle line:84% to convolution as a way to share information over space. 01:02:08.270 --> 01:02:12.003 align:middle line:90% 01:02:12.003 --> 01:02:13.670 align:middle line:84% You can take these different components, 01:02:13.670 --> 01:02:14.795 align:middle line:90% and you can break them out. 01:02:14.795 --> 01:02:18.830 align:middle line:84% So we talked about differential programming. 01:02:18.830 --> 01:02:20.730 align:middle line:84% If you train an encoder-decoder architecture, 01:02:20.730 --> 01:02:23.470 align:middle line:84% you could use the encoder to build some representation 01:02:23.470 --> 01:02:24.330 align:middle line:90% of input images. 01:02:24.330 --> 01:02:27.810 align:middle line:84% But you can also use the decoder to generate images. 01:02:27.810 --> 01:02:29.890 align:middle line:84% So these components can be used separately 01:02:29.890 --> 01:02:32.028 align:middle line:90% and can be valuable separately. 01:02:32.028 --> 01:02:33.570 align:middle line:84% But another way you might want to use 01:02:33.570 --> 01:02:36.370 align:middle line:84% this type of encoder-decoder structure of architecture 01:02:36.370 --> 01:02:42.010 align:middle line:84% would be to do something like get out semantic segmentation 01:02:42.010 --> 01:02:46.157 align:middle line:84% because you want an output space that has spatial extent, 01:02:46.157 --> 01:02:48.490 align:middle line:84% doesn't necessarily need to be the exact same resolution 01:02:48.490 --> 01:02:50.530 align:middle line:90% as the input, though it can be. 01:02:50.530 --> 01:02:53.330 align:middle line:84% But you want something that has a spatial structure 01:02:53.330 --> 01:02:57.730 align:middle line:84% in the output space as opposed to just like a single category. 01:02:57.730 --> 01:03:01.410 align:middle line:90% Now this seems nice. 01:03:01.410 --> 01:03:06.030 align:middle line:84% But it turns out that this is both a blessing and a curse, 01:03:06.030 --> 01:03:07.370 align:middle line:90% this bottleneck in the middle. 01:03:07.370 --> 01:03:09.370 align:middle line:84% It's forcing the model to learn something that's 01:03:09.370 --> 01:03:12.210 align:middle line:84% kind of semantically useful, but it also 01:03:12.210 --> 01:03:15.890 align:middle line:84% means that often it's not possible with an architecture 01:03:15.890 --> 01:03:18.450 align:middle line:84% that just has downsampling, then upsampling 01:03:18.450 --> 01:03:21.850 align:middle line:84% to get a really precise output that's 01:03:21.850 --> 01:03:24.338 align:middle line:84% the same size as the input because you've lost 01:03:24.338 --> 01:03:25.630 align:middle line:90% a lot of the spatial structure. 01:03:25.630 --> 01:03:27.300 align:middle line:84% You've lost a lot of the information 01:03:27.300 --> 01:03:30.260 align:middle line:84% about the fine-grained details in that spatial structure. 01:03:30.260 --> 01:03:33.763 align:middle line:84% And so an alternative would be to take an approach 01:03:33.763 --> 01:03:36.180 align:middle line:84% to doing image-to-image, like semantic segmentation, where 01:03:36.180 --> 01:03:38.340 align:middle line:84% you don't lose any spatial structure. 01:03:38.340 --> 01:03:41.220 align:middle line:84% So here you do convolution, ReLU, convolution, Softmax, 01:03:41.220 --> 01:03:43.900 align:middle line:84% is a very, very simple version of that. 01:03:43.900 --> 01:03:46.400 align:middle line:84% But every single one of these inputs, 01:03:46.400 --> 01:03:48.700 align:middle line:84% because you're keeping the entire dimension, 01:03:48.700 --> 01:03:51.100 align:middle line:84% this can be really computationally expensive. 01:03:51.100 --> 01:03:54.420 align:middle line:84% So the size of this model would be really huge. 01:03:54.420 --> 01:03:57.060 align:middle line:84% So to bridge and get the best of both worlds, where 01:03:57.060 --> 01:04:00.680 align:middle line:84% you're keeping a lot of the high-resolution structure 01:04:00.680 --> 01:04:03.340 align:middle line:84% but you're also building models that are computationally 01:04:03.340 --> 01:04:05.060 align:middle line:84% efficient and that are forced to learn 01:04:05.060 --> 01:04:07.940 align:middle line:84% something that's kind of semantically meaningful 01:04:07.940 --> 01:04:09.485 align:middle line:90% is the U-net. 01:04:09.485 --> 01:04:10.860 align:middle line:84% So probably many of you have seen 01:04:10.860 --> 01:04:12.740 align:middle line:84% this or different variations of this. 01:04:12.740 --> 01:04:15.180 align:middle line:84% But the big intuition here is we have 01:04:15.180 --> 01:04:17.580 align:middle line:84% what are called these skip connections 01:04:17.580 --> 01:04:22.180 align:middle line:84% across the different encoder and decoder structures 01:04:22.180 --> 01:04:23.740 align:middle line:90% of this architecture. 01:04:23.740 --> 01:04:25.600 align:middle line:84% So you actually have a connection 01:04:25.600 --> 01:04:29.280 align:middle line:84% that takes whatever the size is, and it just 01:04:29.280 --> 01:04:34.280 align:middle line:84% takes that via the identity and lets it be skipped over 01:04:34.280 --> 01:04:35.560 align:middle line:90% towards the output. 01:04:35.560 --> 01:04:37.240 align:middle line:84% And what this means is the model is not 01:04:37.240 --> 01:04:40.680 align:middle line:84% forced to explicitly try to shove 01:04:40.680 --> 01:04:43.320 align:middle line:84% a bunch of fine-grained spatial detail 01:04:43.320 --> 01:04:46.360 align:middle line:84% into some very low rank, non-spatially dimensional 01:04:46.360 --> 01:04:48.600 align:middle line:84% embedding, which is somewhat impossible. 01:04:48.600 --> 01:04:52.120 align:middle line:84% It's able to learn the semantic details through this information 01:04:52.120 --> 01:04:55.280 align:middle line:84% bottleneck, but then it also gets a lot of the spatial detail 01:04:55.280 --> 01:04:57.560 align:middle line:84% through those skipped connections. 01:04:57.560 --> 01:04:58.900 align:middle line:90% And this is really powerful. 01:04:58.900 --> 01:05:00.460 align:middle line:84% U-net is a really old architecture, 01:05:00.460 --> 01:05:02.835 align:middle line:84% and it is still one of the most widely used architectures 01:05:02.835 --> 01:05:04.220 align:middle line:90% for semantic segmentation. 01:05:04.220 --> 01:05:09.200 align:middle line:84% It's used all the time in things like remote sensing, 01:05:09.200 --> 01:05:12.160 align:middle line:84% like land cover mapping, for example. 01:05:12.160 --> 01:05:14.320 align:middle line:84% And this idea of a skipped connection, this 01:05:14.320 --> 01:05:18.360 align:middle line:84% actually is a really useful computational tool. 01:05:18.360 --> 01:05:20.600 align:middle line:84% And it's also one of the fundamental underpinnings 01:05:20.600 --> 01:05:22.880 align:middle line:84% of what we call the ResNet architecture. 01:05:22.880 --> 01:05:25.820 align:middle line:84% So ResNet was invented by Kaiming He, who is also 01:05:25.820 --> 01:05:27.780 align:middle line:90% a professor now here at MIT. 01:05:27.780 --> 01:05:30.980 align:middle line:84% And this architecture is amazing. 01:05:30.980 --> 01:05:33.660 align:middle line:84% We still use this all the time, and it's like 10 years 01:05:33.660 --> 01:05:35.500 align:middle line:90% old at this point. 01:05:35.500 --> 01:05:38.740 align:middle line:84% But really, one of the important things that was recognized 01:05:38.740 --> 01:05:40.320 align:middle line:90% is essentially-- 01:05:40.320 --> 01:05:42.580 align:middle line:84% so one cool intuition about ResNet 01:05:42.580 --> 01:05:45.200 align:middle line:84% is every single layer has this skipped connection. 01:05:45.200 --> 01:05:47.220 align:middle line:84% So it has this identity pathway that 01:05:47.220 --> 01:05:52.500 align:middle line:84% lets you choose whether you want to run the input 01:05:52.500 --> 01:05:57.060 align:middle line:84% data through that layer or just skip it, just take this identity 01:05:57.060 --> 01:06:00.300 align:middle line:84% mapping and just pass it through to the next layer. 01:06:00.300 --> 01:06:02.220 align:middle line:84% So one kind of cool intuition about this 01:06:02.220 --> 01:06:04.580 align:middle line:84% is this actually means that this is 01:06:04.580 --> 01:06:08.660 align:middle line:84% a model that can learn something about its architecture. 01:06:08.660 --> 01:06:10.760 align:middle line:90% It can learn an optimal depth. 01:06:10.760 --> 01:06:13.340 align:middle line:84% Because you can essentially say, I 01:06:13.340 --> 01:06:15.923 align:middle line:84% want to just pass this through maybe three of these layers. 01:06:15.923 --> 01:06:17.340 align:middle line:84% And then the rest of the time, I'm 01:06:17.340 --> 01:06:20.860 align:middle line:84% just going to take these skip connections out to the output. 01:06:20.860 --> 01:06:23.160 align:middle line:84% And so then the model itself can implicitly 01:06:23.160 --> 01:06:26.640 align:middle line:84% learn the optimal depth for the task that you want to learn. 01:06:26.640 --> 01:06:29.960 align:middle line:84% So one kind of cool intuition, I think, about resonance 01:06:29.960 --> 01:06:32.720 align:middle line:84% is that they give the model the flexibility 01:06:32.720 --> 01:06:35.560 align:middle line:84% to learn something optimal about the architecture 01:06:35.560 --> 01:06:38.800 align:middle line:84% without needing to test out explicitly 01:06:38.800 --> 01:06:42.080 align:middle line:84% training a bunch of different architectures side by side. 01:06:42.080 --> 01:06:44.920 align:middle line:84% And these residual connections essentially 01:06:44.920 --> 01:06:48.260 align:middle line:84% are just x out equals function of x in, 01:06:48.260 --> 01:06:51.840 align:middle line:84% so mapping it through that layer plus the input dimension itself. 01:06:51.840 --> 01:06:54.482 align:middle line:84% And if you want to actually change the dimensionality, 01:06:54.482 --> 01:06:56.440 align:middle line:84% so you want to have something like downsampling 01:06:56.440 --> 01:06:58.520 align:middle line:84% within a ResNet, then you can also 01:06:58.520 --> 01:07:02.440 align:middle line:84% have another set of linear weights that downsamples in that 01:07:02.440 --> 01:07:04.260 align:middle line:90% skip connection. 01:07:04.260 --> 01:07:05.460 align:middle line:90% Cool. 01:07:05.460 --> 01:07:08.460 align:middle line:84% And this is a really, really powerful tool. 01:07:08.460 --> 01:07:09.240 align:middle line:90% Yeah? 01:07:09.240 --> 01:07:11.160 align:middle line:84% AUDIENCE: So with these skip connections, 01:07:11.160 --> 01:07:14.560 align:middle line:84% is the skip becoming another channel that's also like 01:07:14.560 --> 01:07:18.160 align:middle line:84% an output channel in the middle, or is it like actually choosing 01:07:18.160 --> 01:07:27.380 align:middle line:90% one or the other/ 01:07:27.380 --> 01:07:31.720 align:middle line:84% SARA BEERY: So it's kind of choosing one or the other. 01:07:31.720 --> 01:07:34.300 align:middle line:90% 01:07:34.300 --> 01:07:37.300 align:middle line:84% So the residual connection is essentially-- 01:07:37.300 --> 01:07:42.723 align:middle line:84% you assume that the output of your layer 01:07:42.723 --> 01:07:44.640 align:middle line:84% and the residual connection are the same size. 01:07:44.640 --> 01:07:47.860 align:middle line:84% And then you're just adding them together. 01:07:47.860 --> 01:07:51.180 align:middle line:84% But there's different implementations 01:07:51.180 --> 01:07:54.840 align:middle line:84% of this that will have some of learned weight. 01:07:54.840 --> 01:07:57.900 align:middle line:84% So, OK, maybe you learn to completely ignore that identity. 01:07:57.900 --> 01:08:00.740 align:middle line:84% That same type of skipped connection 01:08:00.740 --> 01:08:04.300 align:middle line:84% is also something that gets surfaced via self-attention 01:08:04.300 --> 01:08:06.100 align:middle line:84% in transformers, which we'll talk 01:08:06.100 --> 01:08:08.040 align:middle line:84% about more in the transformer architecture. 01:08:08.040 --> 01:08:10.060 align:middle line:84% But essentially, just this ability 01:08:10.060 --> 01:08:14.500 align:middle line:84% to learn how to combine information is valuable. 01:08:14.500 --> 01:08:19.253 align:middle line:84% If it is just an addition or a concatenation or however 01:08:19.253 --> 01:08:20.920 align:middle line:84% you want to do that skip connection, one 01:08:20.920 --> 01:08:24.340 align:middle line:84% of the cool things about linear layers or convolutional layers, 01:08:24.340 --> 01:08:25.960 align:middle line:84% which are constrained linear layers, 01:08:25.960 --> 01:08:30.220 align:middle line:84% is essentially even if you just add them together, 01:08:30.220 --> 01:08:32.960 align:middle line:84% there's some version of the next layer that could basically 01:08:32.960 --> 01:08:37.560 align:middle line:84% learn how to ignore certain dimensions, that input. 01:08:37.560 --> 01:08:39.288 align:middle line:90% But yeah. 01:08:39.288 --> 01:08:41.080 align:middle line:84% Like, for example, here, if you do actually 01:08:41.080 --> 01:08:43.477 align:middle line:84% want to change the dimensionality, that W layer, 01:08:43.477 --> 01:08:45.060 align:middle line:84% that could be learned to be all zeros. 01:08:45.060 --> 01:08:47.200 align:middle line:84% So you could just not keep any of that at all 01:08:47.200 --> 01:08:50.499 align:middle line:84% if that's what the model wants to learn. 01:08:50.499 --> 01:08:52.080 align:middle line:90% Cool. 01:08:52.080 --> 01:08:55.319 align:middle line:84% So far, we've talked about two-dimensional or 01:08:55.319 --> 01:08:57.160 align:middle line:90% three-dimensional grids. 01:08:57.160 --> 01:08:59.120 align:middle line:84% But one of the things that, of course, comes up 01:08:59.120 --> 01:09:04.319 align:middle line:84% all the time in the real world are additional dimensionalities. 01:09:04.319 --> 01:09:07.000 align:middle line:90% One really simple one is time. 01:09:07.000 --> 01:09:09.700 align:middle line:84% So here, if you think about convolutions in time, 01:09:09.700 --> 01:09:11.920 align:middle line:84% so maybe one version of this is you actually 01:09:11.920 --> 01:09:14.920 align:middle line:84% have just a single scalar valued input, maybe something 01:09:14.920 --> 01:09:17.340 align:middle line:84% like temperature at a temperature sensor. 01:09:17.340 --> 01:09:19.783 align:middle line:84% And then you can think about doing convolutions over time. 01:09:19.783 --> 01:09:22.200 align:middle line:84% So instead of doing convolutions over a spatial dimension, 01:09:22.200 --> 01:09:24.580 align:middle line:84% you could take convolutions over a temporal dimension 01:09:24.580 --> 01:09:27.140 align:middle line:84% and build some representation with time 01:09:27.140 --> 01:09:29.979 align:middle line:84% as one of the dimensions of that convolution. 01:09:29.979 --> 01:09:32.700 align:middle line:84% And now if you actually have something like video-- 01:09:32.700 --> 01:09:34.420 align:middle line:84% so, for example, now you actually 01:09:34.420 --> 01:09:36.993 align:middle line:84% have an image with red, green, and blue channels. 01:09:36.993 --> 01:09:38.660 align:middle line:84% And then you have these different frames 01:09:38.660 --> 01:09:40.899 align:middle line:90% of an image over time. 01:09:40.899 --> 01:09:44.580 align:middle line:84% You can think about building like what 01:09:44.580 --> 01:09:49.180 align:middle line:84% we'll call another tensor representation or this block 01:09:49.180 --> 01:09:53.359 align:middle line:84% representation where you have the spatial extent. 01:09:53.359 --> 01:09:57.260 align:middle line:84% And here we're now using that third channel to represent time. 01:09:57.260 --> 01:09:59.640 align:middle line:84% But then you also have RGB for each of these. 01:09:59.640 --> 01:10:02.300 align:middle line:84% So it's actually a four dimensional input. 01:10:02.300 --> 01:10:06.300 align:middle line:84% And then you can think of building these 3D convolutions 01:10:06.300 --> 01:10:10.780 align:middle line:84% that actually convolve over space and time. 01:10:10.780 --> 01:10:14.060 align:middle line:84% And then your output would be maybe a lower dimensional 01:10:14.060 --> 01:10:15.960 align:middle line:84% representation that would actually 01:10:15.960 --> 01:10:18.320 align:middle line:84% have those space and time dimensions for each 01:10:18.320 --> 01:10:20.120 align:middle line:90% of your tensors. 01:10:20.120 --> 01:10:21.658 align:middle line:84% And this is a type of architecture. 01:10:21.658 --> 01:10:24.200 align:middle line:84% This type of 3D convolutions is often used when you're trying 01:10:24.200 --> 01:10:27.880 align:middle line:84% to do something like action recognition in video. 01:10:27.880 --> 01:10:30.200 align:middle line:84% Now, one thing that this gives you is it 01:10:30.200 --> 01:10:33.280 align:middle line:84% makes you shift invariant in time. 01:10:33.280 --> 01:10:35.280 align:middle line:84% But you can imagine actually that for some types 01:10:35.280 --> 01:10:36.780 align:middle line:84% of recognition tasks, you don't want 01:10:36.780 --> 01:10:38.420 align:middle line:90% to be shift invariant in time. 01:10:38.420 --> 01:10:40.960 align:middle line:84% You don't want like picking a cup up 01:10:40.960 --> 01:10:43.600 align:middle line:84% to be the same action as putting a cup down. 01:10:43.600 --> 01:10:45.953 align:middle line:90% And I mean, they're-- 01:10:45.953 --> 01:10:48.620 align:middle line:84% ignoring the motion between, if you had a really low frame rate, 01:10:48.620 --> 01:10:50.880 align:middle line:84% if you had this frame and that frame, 01:10:50.880 --> 01:10:52.980 align:middle line:84% and then maybe you had that frame and this frame, 01:10:52.980 --> 01:10:55.320 align:middle line:84% you want the fact that one happened before the other 01:10:55.320 --> 01:10:56.180 align:middle line:90% to matter. 01:10:56.180 --> 01:10:59.840 align:middle line:84% That translation invariance is not something you always want. 01:10:59.840 --> 01:11:02.300 align:middle line:84% And so if you don't want to be shift invariant, 01:11:02.300 --> 01:11:05.180 align:middle line:84% there's a couple of different approaches that we can use. 01:11:05.180 --> 01:11:07.940 align:middle line:84% So one, use an architecture that's not shift invariant, 01:11:07.940 --> 01:11:09.400 align:middle line:90% so something like an MLP. 01:11:09.400 --> 01:11:12.400 align:middle line:84% There the position in time would actually matter. 01:11:12.400 --> 01:11:14.740 align:middle line:84% But like we said, these are it's quite difficult 01:11:14.740 --> 01:11:17.180 align:middle line:84% to get these MLPs to learn really complex functions 01:11:17.180 --> 01:11:18.940 align:middle line:90% of that almost infinite data. 01:11:18.940 --> 01:11:23.220 align:middle line:84% So another option, and one that is really shown to be a winner, 01:11:23.220 --> 01:11:26.980 align:middle line:84% is this idea of adding location information 01:11:26.980 --> 01:11:29.300 align:middle line:84% to the input to the convolutional filters 01:11:29.300 --> 01:11:32.420 align:middle line:84% through something that we call a positional encoding. 01:11:32.420 --> 01:11:36.940 align:middle line:84% So here what we do is you have your input signal. 01:11:36.940 --> 01:11:40.600 align:middle line:84% And then you also have this separate positional encoding, 01:11:40.600 --> 01:11:42.280 align:middle line:84% which is something you construct. 01:11:42.280 --> 01:11:43.822 align:middle line:84% This is not something that's learned. 01:11:43.822 --> 01:11:46.220 align:middle line:84% It's actually constructed and added to the input. 01:11:46.220 --> 01:11:48.180 align:middle line:84% And you learn an additional weight 01:11:48.180 --> 01:11:50.180 align:middle line:84% that incorporates that positional encoding 01:11:50.180 --> 01:11:52.260 align:middle line:90% as part of your convolution. 01:11:52.260 --> 01:11:54.500 align:middle line:84% And so there you're doing the same thing 01:11:54.500 --> 01:11:56.600 align:middle line:84% that you might with a normal convolutional layer, 01:11:56.600 --> 01:11:58.475 align:middle line:84% but you're adding this additional information 01:11:58.475 --> 01:11:59.517 align:middle line:90% from position. 01:11:59.517 --> 01:12:01.100 align:middle line:84% And so here we might want to introduce 01:12:01.100 --> 01:12:04.860 align:middle line:84% the concept of what gets sometimes called a neural field. 01:12:04.860 --> 01:12:09.340 align:middle line:84% So a field from physics, it can be a varying physical quantity 01:12:09.340 --> 01:12:12.540 align:middle line:84% of both spatial and temporal coordinates. 01:12:12.540 --> 01:12:15.110 align:middle line:84% And a neural field is a field that 01:12:15.110 --> 01:12:19.773 align:middle line:84% is parameterized maybe fully or in part by a neural network. 01:12:19.773 --> 01:12:21.190 align:middle line:84% So there's different ways that you 01:12:21.190 --> 01:12:23.270 align:middle line:84% can think about doing this and representing 01:12:23.270 --> 01:12:24.950 align:middle line:90% these neural fields. 01:12:24.950 --> 01:12:27.910 align:middle line:84% But one way to think about it-- so SIREN 01:12:27.910 --> 01:12:30.630 align:middle line:84% is an example of one of these neural fields, that model we 01:12:30.630 --> 01:12:31.990 align:middle line:90% talked about earlier. 01:12:31.990 --> 01:12:33.630 align:middle line:84% This is a convolutional neural network 01:12:33.630 --> 01:12:37.930 align:middle line:84% that's applied per pixel to map from a coordinate grid, 01:12:37.930 --> 01:12:41.270 align:middle line:84% so now the input is the positional encoding in an image, 01:12:41.270 --> 01:12:42.750 align:middle line:90% to a color. 01:12:42.750 --> 01:12:46.550 align:middle line:84% So you're actually learning to model a single image 01:12:46.550 --> 01:12:49.870 align:middle line:84% by learning a functional mapping from a position in the image 01:12:49.870 --> 01:12:51.092 align:middle line:90% to a color. 01:12:51.092 --> 01:12:52.550 align:middle line:84% And the way that they're doing that 01:12:52.550 --> 01:12:54.810 align:middle line:84% is through these sinusoidal things. 01:12:54.810 --> 01:12:56.590 align:middle line:84% And so here what you actually do is 01:12:56.590 --> 01:12:59.010 align:middle line:84% you take-- the input is actually just the position, 01:12:59.010 --> 01:13:03.210 align:middle line:84% so in some coordinate space, and these could be explicit. 01:13:03.210 --> 01:13:04.970 align:middle line:90% They could be batched up. 01:13:04.970 --> 01:13:07.830 align:middle line:84% Or they also could be continuous because the idea 01:13:07.830 --> 01:13:12.010 align:middle line:84% is that this is something that's mapping from some of position 01:13:12.010 --> 01:13:12.750 align:middle line:90% to an output. 01:13:12.750 --> 01:13:17.610 align:middle line:84% So now you run these inputs through a model 01:13:17.610 --> 01:13:19.370 align:middle line:90% that's just taking x and y. 01:13:19.370 --> 01:13:24.010 align:middle line:84% And then you're outputting the color of that value. 01:13:24.010 --> 01:13:26.050 align:middle line:84% And this can take continuous coordinates 01:13:26.050 --> 01:13:29.890 align:middle line:84% as input, which means that we can then do something 01:13:29.890 --> 01:13:33.730 align:middle line:84% like think about learning these functional 01:13:33.730 --> 01:13:38.090 align:middle line:84% representations of input space as opposed to only gridwise 01:13:38.090 --> 01:13:39.610 align:middle line:90% representations. 01:13:39.610 --> 01:13:43.250 align:middle line:84% And this idea is the fundamental underpinning 01:13:43.250 --> 01:13:46.650 align:middle line:84% of quite a popular architecture recently called 01:13:46.650 --> 01:13:49.970 align:middle line:84% NeRF, which is essentially a convolutional network applied 01:13:49.970 --> 01:13:53.010 align:middle line:84% to a map from a five-dimensional coordinate grid, which 01:13:53.010 --> 01:13:56.170 align:middle line:84% is a position and direction of the camera 01:13:56.170 --> 01:13:57.730 align:middle line:90% relative to the scene. 01:13:57.730 --> 01:14:02.510 align:middle line:84% And then it gives you an output of color and volumetric density. 01:14:02.510 --> 01:14:06.330 align:middle line:84% And this allows us to then actually generate 01:14:06.330 --> 01:14:10.790 align:middle line:84% what that same object would look like from different positions 01:14:10.790 --> 01:14:13.590 align:middle line:84% than what was in the original training data. 01:14:13.590 --> 01:14:15.510 align:middle line:84% So what that actually looks like is 01:14:15.510 --> 01:14:17.990 align:middle line:84% you can take some images of a scene 01:14:17.990 --> 01:14:21.750 align:middle line:84% and then generate these essentially walkthroughs 01:14:21.750 --> 01:14:22.492 align:middle line:90% of scenes. 01:14:22.492 --> 01:14:24.950 align:middle line:84% And these have actually gotten better and better over time. 01:14:24.950 --> 01:14:26.750 align:middle line:84% And one of the ways that this is really-- 01:14:26.750 --> 01:14:28.250 align:middle line:84% one of the fundamental underpinnings 01:14:28.250 --> 01:14:31.150 align:middle line:84% of this idea of building these appropriate positional 01:14:31.150 --> 01:14:32.670 align:middle line:90% encodings. 01:14:32.670 --> 01:14:34.350 align:middle line:90% Cool. 01:14:34.350 --> 01:14:36.430 align:middle line:84% NeRFs are really fun to play with. 01:14:36.430 --> 01:14:38.590 align:middle line:84% Recently, some people at Woods Hole 01:14:38.590 --> 01:14:41.430 align:middle line:84% made a NeRF for under the sea that handles the way 01:14:41.430 --> 01:14:42.770 align:middle line:90% that light bends under water. 01:14:42.770 --> 01:14:44.390 align:middle line:90% Very neat. 01:14:44.390 --> 01:14:47.990 align:middle line:90% So concluding remarks here. 01:14:47.990 --> 01:14:50.950 align:middle line:84% Convolution is really just a fundamental operation 01:14:50.950 --> 01:14:54.070 align:middle line:84% for image processing or actually for processing anything 01:14:54.070 --> 01:14:56.250 align:middle line:90% that has some structure. 01:14:56.250 --> 01:14:58.590 align:middle line:84% So this might be something like time 01:14:58.590 --> 01:15:04.010 align:middle line:84% or even time, space, and multiple input dimensions. 01:15:04.010 --> 01:15:05.950 align:middle line:84% So say you wanted to use convolution 01:15:05.950 --> 01:15:10.050 align:middle line:84% over hyperspectral satellite imagery taken once 01:15:10.050 --> 01:15:13.050 align:middle line:90% a week from the entire Earth. 01:15:13.050 --> 01:15:15.465 align:middle line:84% Essentially, what it means is you're just chopping up 01:15:15.465 --> 01:15:17.090 align:middle line:84% the image into patches, and then you're 01:15:17.090 --> 01:15:19.230 align:middle line:84% applying the same function to each patch, 01:15:19.230 --> 01:15:22.410 align:middle line:84% but you're doing it through this slide and sum operation. 01:15:22.410 --> 01:15:24.410 align:middle line:84% And this concept appears in basically 01:15:24.410 --> 01:15:26.270 align:middle line:90% almost all modern architectures. 01:15:26.270 --> 01:15:29.440 align:middle line:84% It appears in CNNs, but also things like transformers, 01:15:29.440 --> 01:15:31.690 align:middle line:84% though then the patch-wise operation isn't necessarily 01:15:31.690 --> 01:15:34.190 align:middle line:84% convolution, use attention instead, 01:15:34.190 --> 01:15:36.290 align:middle line:90% things like NeRFs and more. 01:15:36.290 --> 01:15:38.810 align:middle line:84% So, today, we talked about building better architectures 01:15:38.810 --> 01:15:39.930 align:middle line:90% and why. 01:15:39.930 --> 01:15:42.910 align:middle line:84% We talked about convolutional layers, pyramids, 01:15:42.910 --> 01:15:44.810 align:middle line:90% and receptive fields. 01:15:44.810 --> 01:15:46.690 align:middle line:84% We went through at a very high level 01:15:46.690 --> 01:15:48.450 align:middle line:84% some of these different ideas that 01:15:48.450 --> 01:15:50.663 align:middle line:84% built out different types of architectures. 01:15:50.663 --> 01:15:53.330 align:middle line:84% And then we talked about neural fields and positional encodings, 01:15:53.330 --> 01:15:55.622 align:middle line:84% though we'll talk a lot more about positional encodings 01:15:55.622 --> 01:15:57.890 align:middle line:90% in the transformers lecture. 01:15:57.890 --> 01:15:58.732 align:middle line:90% Any final questions? 01:15:58.732 --> 01:16:00.690 align:middle line:84% And if you are going to leave during questions, 01:16:00.690 --> 01:16:04.170 align:middle line:84% just be quiet so that we can address them. 01:16:04.170 --> 01:16:05.050 align:middle line:90% Yeah? 01:16:05.050 --> 01:16:07.693 align:middle line:84% AUDIENCE: Could you use NeRFs or upscaling? 01:16:07.693 --> 01:16:09.610 align:middle line:84% SARA BEERY: Could you use NeRFs for upscaling? 01:16:09.610 --> 01:16:10.110 align:middle line:90% Yes. 01:16:10.110 --> 01:16:11.212 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:16:11.212 --> 01:16:12.170 align:middle line:90% SARA BEERY: Yeah, yeah. 01:16:12.170 --> 01:16:15.230 align:middle line:84% So one of the ways that people have 01:16:15.230 --> 01:16:18.305 align:middle line:84% used NeRFs is to increase the resolution of, let's 01:16:18.305 --> 01:16:19.930 align:middle line:84% say, like a point cloud representation, 01:16:19.930 --> 01:16:22.078 align:middle line:84% not exactly like the original NeRFs architecture, 01:16:22.078 --> 01:16:23.870 align:middle line:84% but there's been adaptations of it that try 01:16:23.870 --> 01:16:26.950 align:middle line:90% to increase spatial resolution. 01:16:26.950 --> 01:16:29.430 align:middle line:90% Yeah. 01:16:29.430 --> 01:16:32.190 align:middle line:90% Any other questions? 01:16:32.190 --> 01:16:33.270 align:middle line:90% Yes? 01:16:33.270 --> 01:16:34.850 align:middle line:84% And then quiet, please, you guys. 01:16:34.850 --> 01:16:35.810 align:middle line:90% Thank you so much. 01:16:35.810 --> 01:16:38.730 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:16:38.730 --> 01:16:40.666 align:middle line:90% 01:16:40.666 --> 01:16:41.470 align:middle line:90% SARA BEERY: Yeah. 01:16:41.470 --> 01:16:45.470 align:middle line:84% So something like strided convolutions, 01:16:45.470 --> 01:16:48.470 align:middle line:84% that's actually explicitly doing some form of pooling 01:16:48.470 --> 01:16:51.550 align:middle line:90% and downsampling together. 01:16:51.550 --> 01:16:54.070 align:middle line:84% So one way you could think about it is like pooling 01:16:54.070 --> 01:16:57.410 align:middle line:84% is essentially the same as running a convolutional layer. 01:16:57.410 --> 01:16:59.050 align:middle line:84% But now you're not learning weights. 01:16:59.050 --> 01:17:03.950 align:middle line:84% You're just taking, for example, like the maximum function 01:17:03.950 --> 01:17:05.250 align:middle line:90% or that average function. 01:17:05.250 --> 01:17:06.833 align:middle line:84% That's like that's the function that's 01:17:06.833 --> 01:17:08.410 align:middle line:90% applied on that local patch. 01:17:08.410 --> 01:17:10.630 align:middle line:84% So essentially, it's a constrained version, 01:17:10.630 --> 01:17:13.010 align:middle line:84% a nonlearned version of a convolutional layer. 01:17:13.010 --> 01:17:14.050 align:middle line:90% Yeah. 01:17:14.050 --> 01:17:18.030 align:middle line:84% AUDIENCE: And then downsampling is like [INAUDIBLE]? 01:17:18.030 --> 01:17:20.350 align:middle line:90% SARA BEERY: Yeah. 01:17:20.350 --> 01:17:21.730 align:middle line:90% Yes? 01:17:21.730 --> 01:17:25.050 align:middle line:84% AUDIENCE: Could you explain a little bit on the neural fields? 01:17:25.050 --> 01:17:25.970 align:middle line:90% [INAUDIBLE]? 01:17:25.970 --> 01:17:29.330 align:middle line:90% 01:17:29.330 --> 01:17:30.770 align:middle line:90% SARA BEERY: Yeah. 01:17:30.770 --> 01:17:32.970 align:middle line:84% So the way that you train it is you 01:17:32.970 --> 01:17:35.490 align:middle line:84% take a bunch of different images of a scene 01:17:35.490 --> 01:17:36.670 align:middle line:90% from different positions. 01:17:36.670 --> 01:17:38.650 align:middle line:84% So often, the training data for NeRFs 01:17:38.650 --> 01:17:41.770 align:middle line:84% will be someone taking a video, like walking around 01:17:41.770 --> 01:17:43.530 align:middle line:90% an object, for example. 01:17:43.530 --> 01:17:46.250 align:middle line:84% And now you have a bunch of examples where now you're 01:17:46.250 --> 01:17:49.690 align:middle line:84% going to estimate the relative position to the scene. 01:17:49.690 --> 01:17:54.410 align:middle line:84% And then now you can estimate the position and the direction 01:17:54.410 --> 01:17:56.750 align:middle line:90% from the input image. 01:17:56.750 --> 01:17:59.330 align:middle line:84% Now for any given point, that's your training data. 01:17:59.330 --> 01:18:04.550 align:middle line:84% And the output is the color of the image at that pixel. 01:18:04.550 --> 01:18:10.350 align:middle line:84% AUDIENCE: Is that generalizable, or is just used for picture 01:18:10.350 --> 01:18:12.670 align:middle line:90% to video only used for that? 01:18:12.670 --> 01:18:15.150 align:middle line:84% SARA BEERY: It can be used for a lot of different things. 01:18:15.150 --> 01:18:18.970 align:middle line:84% So I will say that only just recently, like last week, 01:18:18.970 --> 01:18:21.510 align:middle line:84% I saw the first kind of large-scale paper that was 01:18:21.510 --> 01:18:26.510 align:middle line:84% taking this concept of NeRF, which gives you this creation 01:18:26.510 --> 01:18:28.283 align:middle line:84% of video that you can interpolate between, 01:18:28.283 --> 01:18:30.950 align:middle line:84% and then actually trying to move that directly to something that 01:18:30.950 --> 01:18:32.990 align:middle line:90% was 3D modeling the scene. 01:18:32.990 --> 01:18:36.690 align:middle line:84% Because the output of NeRF is not a 3D model of the scene. 01:18:36.690 --> 01:18:42.030 align:middle line:84% It's what the image would look like taken from a different 01:18:42.030 --> 01:18:45.250 align:middle line:84% direction, which is not the same as a 3D model of the scene. 01:18:45.250 --> 01:18:49.310 align:middle line:84% And, actually, a lot of real-world applications we would 01:18:49.310 --> 01:18:53.310 align:middle line:84% like to actually be able to use sparse monocular image 01:18:53.310 --> 01:18:56.120 align:middle line:84% representations to get out a robust 3D model of the scene. 01:18:56.120 --> 01:18:57.870 align:middle line:84% And so they've just started to bring those 01:18:57.870 --> 01:19:02.110 align:middle line:84% two things together in recent work from Ben Mildenhall, 01:19:02.110 --> 01:19:04.450 align:middle line:90% I think, was the person. 01:19:04.450 --> 01:19:07.910 align:middle line:84% I will say these are also really not super robust. 01:19:07.910 --> 01:19:10.170 align:middle line:90% You need a lot of input data. 01:19:10.170 --> 01:19:12.170 align:middle line:84% So classical NeRF, like if you only 01:19:12.170 --> 01:19:15.210 align:middle line:84% have a few sparse images of a scene, 01:19:15.210 --> 01:19:18.090 align:middle line:84% it can be really hard to fit a NeRF appropriately. 01:19:18.090 --> 01:19:21.250 align:middle line:84% And the other thing is it works really well 01:19:21.250 --> 01:19:23.710 align:middle line:90% with stuff that is not moving. 01:19:23.710 --> 01:19:28.050 align:middle line:84% But if you try and do it with something like a waving tree 01:19:28.050 --> 01:19:32.850 align:middle line:84% because the actual thing itself won't-- the math 01:19:32.850 --> 01:19:36.530 align:middle line:84% doesn't necessarily optimize because the thing that 01:19:36.530 --> 01:19:39.730 align:middle line:84% you're taking images of is changing position over time. 01:19:39.730 --> 01:19:41.870 align:middle line:84% That makes it really hard to fit these models. 01:19:41.870 --> 01:19:42.370 align:middle line:90% Yeah. 01:19:42.370 --> 01:19:42.850 align:middle line:90% AUDIENCE: Thank you so much. 01:19:42.850 --> 01:19:43.810 align:middle line:90% SARA BEERY: Yeah. 01:19:43.810 --> 01:19:44.312 align:middle line:90% Yes? 01:19:44.312 --> 01:19:45.770 align:middle line:84% AUDIENCE: What about problems where 01:19:45.770 --> 01:19:48.890 align:middle line:84% the input is not just a picture, but a picture and constants 01:19:48.890 --> 01:19:51.210 align:middle line:90% like scalar numbers? 01:19:51.210 --> 01:19:53.130 align:middle line:84% [INAUDIBLE] contribute to how the picture 01:19:53.130 --> 01:19:54.770 align:middle line:90% should be understood. 01:19:54.770 --> 01:19:57.850 align:middle line:84% SARA BEERY: So maybe like additional metadata. 01:19:57.850 --> 01:20:01.340 align:middle line:84% So for example, I might have a picture and a GPS location 01:20:01.340 --> 01:20:02.883 align:middle line:90% where the picture was taken. 01:20:02.883 --> 01:20:04.300 align:middle line:84% So there's a lot of different ways 01:20:04.300 --> 01:20:06.420 align:middle line:84% that we try to incorporate that side information 01:20:06.420 --> 01:20:08.848 align:middle line:90% into these architectures. 01:20:08.848 --> 01:20:10.640 align:middle line:84% And you can see lots of different examples. 01:20:10.640 --> 01:20:12.900 align:middle line:84% But some really obvious stuff is you can just 01:20:12.900 --> 01:20:15.520 align:middle line:84% run your image through that first convolutional layer, 01:20:15.520 --> 01:20:17.860 align:middle line:84% and then you can have a separate type 01:20:17.860 --> 01:20:21.820 align:middle line:84% of convolution or representation of that positional encoding. 01:20:21.820 --> 01:20:24.640 align:middle line:84% And you can either concatenate it to the entire input image. 01:20:24.640 --> 01:20:27.140 align:middle line:84% You can introduce it slightly later in the network. 01:20:27.140 --> 01:20:28.598 align:middle line:84% There's a lot of different ways you 01:20:28.598 --> 01:20:31.580 align:middle line:84% can append additional information into the network 01:20:31.580 --> 01:20:32.860 align:middle line:90% during training. 01:20:32.860 --> 01:20:33.740 align:middle line:90% Yeah. 01:20:33.740 --> 01:20:34.980 align:middle line:90% Yeah. 01:20:34.980 --> 01:20:36.380 align:middle line:90% Cool. 01:20:36.380 --> 01:20:37.120 align:middle line:90% Thanks, guys. 01:20:37.120 --> 01:20:38.260 align:middle line:90% Oh yeah? 01:20:38.260 --> 01:20:43.420 align:middle line:84% AUDIENCE: When you say there's [INAUDIBLE] video, 01:20:43.420 --> 01:20:47.420 align:middle line:84% can you just add additional channels for each frame versus 01:20:47.420 --> 01:20:49.860 align:middle line:84% just doing a 3D convolution [INAUDIBLE]? 01:20:49.860 --> 01:20:52.577 align:middle line:90% 01:20:52.577 --> 01:20:54.160 align:middle line:84% SARA BEERY: Is there a big difference. 01:20:54.160 --> 01:20:57.860 align:middle line:84% So if you add if you add channels for all 01:20:57.860 --> 01:21:01.040 align:middle line:84% of the different frames in the video, essentially, 01:21:01.040 --> 01:21:03.060 align:middle line:90% that is 3D convolution. 01:21:03.060 --> 01:21:05.560 align:middle line:84% Yeah, because you're taking those channels 01:21:05.560 --> 01:21:09.120 align:middle line:84% and you're maybe running your patch-wise operation almost 01:21:09.120 --> 01:21:10.475 align:middle line:90% like a sliding window over time. 01:21:10.475 --> 01:21:13.100 align:middle line:84% AUDIENCE: Yeah, I guess like you don't add another [INAUDIBLE]. 01:21:13.100 --> 01:21:17.760 align:middle line:90% 01:21:17.760 --> 01:21:19.700 align:middle line:84% SARA BEERY: So sorry, I'm a bit confused. 01:21:19.700 --> 01:21:22.600 align:middle line:84% So if you have an image model that's 01:21:22.600 --> 01:21:26.920 align:middle line:84% taking an RGB and height and width, 01:21:26.920 --> 01:21:29.000 align:middle line:84% now you take that image model and now you're 01:21:29.000 --> 01:21:35.080 align:middle line:84% just saying like stack it over the different frames. 01:21:35.080 --> 01:21:38.200 align:middle line:84% How would you not add another dimension 01:21:38.200 --> 01:21:39.520 align:middle line:90% if you have a time dimension? 01:21:39.520 --> 01:21:41.800 align:middle line:84% AUDIENCE: Like RGB channels, so you just 01:21:41.800 --> 01:21:44.300 align:middle line:84% add more RGB [INAUDIBLE] for each frame. 01:21:44.300 --> 01:21:47.040 align:middle line:90% SARA BEERY: Oh, I see. 01:21:47.040 --> 01:21:48.140 align:middle line:90% You could do that. 01:21:48.140 --> 01:21:51.280 align:middle line:90% It would get really expensive. 01:21:51.280 --> 01:21:54.600 align:middle line:84% And I think one of the things that we definitely 01:21:54.600 --> 01:21:58.180 align:middle line:90% see with things like video-- 01:21:58.180 --> 01:22:01.980 align:middle line:84% very rarely are those convolutions 01:22:01.980 --> 01:22:04.287 align:middle line:84% in the temporal space worth it computationally. 01:22:04.287 --> 01:22:06.620 align:middle line:84% So if you actually look at a lot of the state of the art 01:22:06.620 --> 01:22:10.020 align:middle line:84% models for video, often, they are downsampling 01:22:10.020 --> 01:22:13.060 align:middle line:84% in the temporal space because you can actually 01:22:13.060 --> 01:22:16.700 align:middle line:84% recognize, like most actions through what 01:22:16.700 --> 01:22:18.580 align:middle line:90% we call maybe a bag of images. 01:22:18.580 --> 01:22:20.160 align:middle line:84% So you don't need every single image. 01:22:20.160 --> 01:22:22.160 align:middle line:84% And the difference between the individual frames 01:22:22.160 --> 01:22:23.540 align:middle line:90% is really small. 01:22:23.540 --> 01:22:26.900 align:middle line:84% And so like Alyosha Efros has a nice paper 01:22:26.900 --> 01:22:29.780 align:middle line:84% where they talk about the complexity of actions 01:22:29.780 --> 01:22:33.060 align:middle line:84% by essentially taking any action recognition 01:22:33.060 --> 01:22:35.660 align:middle line:84% problem with a video clip and then 01:22:35.660 --> 01:22:37.940 align:middle line:84% using the number of frames from that video clip 01:22:37.940 --> 01:22:39.940 align:middle line:84% that you need to be able to accurately recognize 01:22:39.940 --> 01:22:42.140 align:middle line:84% that action as a representation of the complexity 01:22:42.140 --> 01:22:43.180 align:middle line:90% of the problem. 01:22:43.180 --> 01:22:45.980 align:middle line:84% And most actions are it's a single frame. 01:22:45.980 --> 01:22:47.900 align:middle line:84% You can recognize it from a single frame. 01:22:47.900 --> 01:22:50.580 align:middle line:84% So because of the computational complexity and the fact that 01:22:50.580 --> 01:22:53.150 align:middle line:84% often we want these models to maybe run in real time, 01:22:53.150 --> 01:22:54.900 align:middle line:84% a lot of times the state-of-the-art models 01:22:54.900 --> 01:22:58.780 align:middle line:84% for video are doing something that we call fast and slow, 01:22:58.780 --> 01:23:02.200 align:middle line:84% where they will have a really low-dimensional subsampling 01:23:02.200 --> 01:23:05.600 align:middle line:84% spatially and maybe even also temporally that's running more 01:23:05.600 --> 01:23:10.160 align:middle line:84% frequently and then some of secondary channel that's taking 01:23:10.160 --> 01:23:13.440 align:middle line:84% high-resolution inputs, so maybe the full image frame size 01:23:13.440 --> 01:23:14.980 align:middle line:90% or something quite close to it. 01:23:14.980 --> 01:23:16.898 align:middle line:84% But it's only doing that periodically. 01:23:16.898 --> 01:23:18.440 align:middle line:84% And either it's doing it periodically 01:23:18.440 --> 01:23:21.800 align:middle line:84% kind of in some pre-defined way, or it's actually 01:23:21.800 --> 01:23:25.900 align:middle line:84% deciding when to run an image through that high-resolution 01:23:25.900 --> 01:23:26.760 align:middle line:90% channel. 01:23:26.760 --> 01:23:29.520 align:middle line:84% So they'll take this two-pronged approach 01:23:29.520 --> 01:23:31.540 align:middle line:84% where one model is maybe modeling something 01:23:31.540 --> 01:23:33.040 align:middle line:84% like the motion and the other one is 01:23:33.040 --> 01:23:36.615 align:middle line:84% modeling some fine-grained semantic details. 01:23:36.615 --> 01:23:37.240 align:middle line:90% AUDIENCE: Cool. 01:23:37.240 --> 01:23:37.860 align:middle line:90% Thank you. 01:23:37.860 --> 01:23:40.200 align:middle line:84% SARA BEERY: Yeah, video is fascinating and quite 01:23:40.200 --> 01:23:45.540 align:middle line:84% complicated and expensive, expensive to store, 01:23:45.540 --> 01:23:48.960 align:middle line:84% expensive to train models on, expensive to run inference on. 01:23:48.960 --> 01:23:50.890 align:middle line:90% Yeah, it's a mess. 01:23:50.890 --> 01:23:56.000 align:middle line:90%