WEBVTT 00:00:00.000 --> 00:00:02.880 align:middle line:90% [SQUEAKING] 00:00:02.880 --> 00:00:04.308 align:middle line:90% [RUSTLING] 00:00:04.308 --> 00:00:09.567 align:middle line:90% [CLICKING] 00:00:09.567 --> 00:00:11.400 align:middle line:84% JOSH MCDERMOTT: So I want to start out today 00:00:11.400 --> 00:00:15.120 align:middle line:84% by talking about the generic viewpoint assumption, which 00:00:15.120 --> 00:00:17.133 align:middle line:84% is relevant to a few things that we've 00:00:17.133 --> 00:00:18.300 align:middle line:90% covered so far in the class. 00:00:18.300 --> 00:00:20.467 align:middle line:84% And then we're going to get into object recognition. 00:00:20.467 --> 00:00:26.620 align:middle line:84% So the generic viewpoint assumption, 00:00:26.620 --> 00:00:29.400 align:middle line:84% it relates to this idea of an accidental viewpoint. 00:00:29.400 --> 00:00:32.200 align:middle line:84% So there's oftentimes these viewpoints 00:00:32.200 --> 00:00:35.440 align:middle line:84% that will create a special image on the retina. 00:00:35.440 --> 00:00:38.160 align:middle line:84% And in particular, they're special in the sense 00:00:38.160 --> 00:00:40.460 align:middle line:84% that they're not robust to small changes in viewpoint. 00:00:40.460 --> 00:00:41.940 align:middle line:90% So here's an example. 00:00:41.940 --> 00:00:43.880 align:middle line:84% So this is like a scene in the world 00:00:43.880 --> 00:00:47.440 align:middle line:84% where you have these four different surfaces that 00:00:47.440 --> 00:00:50.520 align:middle line:84% have different orientations and they're at different depths. 00:00:50.520 --> 00:00:53.580 align:middle line:84% But if you view it from exactly the right point, 00:00:53.580 --> 00:00:55.800 align:middle line:90% it looks like that image. 00:00:55.800 --> 00:00:56.800 align:middle line:90% All right? 00:00:56.800 --> 00:00:59.400 align:middle line:84% So everything kind of lines up in exactly the right way. 00:00:59.400 --> 00:01:02.380 align:middle line:84% And there's lots of examples of things like that. 00:01:02.380 --> 00:01:06.120 align:middle line:84% So here we've got the good old Necker cube 00:01:06.120 --> 00:01:09.120 align:middle line:90% viewed from a generic position. 00:01:09.120 --> 00:01:11.040 align:middle line:84% And then here's the accidental view, 00:01:11.040 --> 00:01:15.400 align:middle line:84% where if you view it such that two of the corners kind of line 00:01:15.400 --> 00:01:18.200 align:middle line:84% up, you get an image that looks like this. 00:01:18.200 --> 00:01:22.240 align:middle line:84% So all of these images, they could be a cube. 00:01:22.240 --> 00:01:24.920 align:middle line:84% But interpreting c as a cube would require 00:01:24.920 --> 00:01:26.260 align:middle line:90% an accidental viewpoint. 00:01:26.260 --> 00:01:28.640 align:middle line:84% And we don't see a cube when we look at this. 00:01:28.640 --> 00:01:31.815 align:middle line:84% And for a long time, this was proposed 00:01:31.815 --> 00:01:33.440 align:middle line:84% to be consistent with this idea that we 00:01:33.440 --> 00:01:35.820 align:middle line:90% prefer a generic viewpoint. 00:01:35.820 --> 00:01:37.800 align:middle line:84% We assume what's called a generic viewpoint. 00:01:37.800 --> 00:01:41.000 align:middle line:90% So here's some other examples. 00:01:41.000 --> 00:01:44.080 align:middle line:84% This is a cube viewed from some different perspectives, 00:01:44.080 --> 00:01:45.960 align:middle line:90% generic and non-generic. 00:01:45.960 --> 00:01:48.862 align:middle line:90% 00:01:48.862 --> 00:01:49.820 align:middle line:90% Here's another example. 00:01:49.820 --> 00:01:55.320 align:middle line:84% Here's an image of something that in the world 00:01:55.320 --> 00:01:57.930 align:middle line:84% could be this, could be this, could be this. 00:01:57.930 --> 00:02:00.350 align:middle line:84% But in these latter two cases, you've 00:02:00.350 --> 00:02:02.870 align:middle line:84% got this accidental view that kind of lines 00:02:02.870 --> 00:02:06.990 align:middle line:84% up in a special way with the thing in the world. 00:02:06.990 --> 00:02:10.270 align:middle line:84% This is kind of a cool real-world example of this that 00:02:10.270 --> 00:02:11.930 align:middle line:90% was made here in this building. 00:02:11.930 --> 00:02:15.610 align:middle line:84% I don't know if you recognize the entrance here. 00:02:15.610 --> 00:02:27.710 align:middle line:90% 00:02:27.710 --> 00:02:28.710 align:middle line:90% Whoa. 00:02:28.710 --> 00:02:33.110 align:middle line:90% What's going on there? 00:02:33.110 --> 00:02:34.570 align:middle line:90% It's an accidental view. 00:02:34.570 --> 00:02:46.710 align:middle line:90% 00:02:46.710 --> 00:02:49.330 align:middle line:84% This was made on April Fool's Day about 10 years ago. 00:02:49.330 --> 00:03:00.590 align:middle line:90% 00:03:00.590 --> 00:03:03.350 align:middle line:90% And lo and behold. 00:03:03.350 --> 00:03:04.395 align:middle line:90% AUDIENCE: Oh, Wow. 00:03:04.395 --> 00:03:05.270 align:middle line:90% JOSH MCDERMOTT: Yeah. 00:03:05.270 --> 00:03:06.937 align:middle line:84% So there was some tape that was put down 00:03:06.937 --> 00:03:11.630 align:middle line:84% in a very tricky arrangement such that when 00:03:11.630 --> 00:03:17.230 align:middle line:84% you view it from a certain perspective, everything lines 00:03:17.230 --> 00:03:18.230 align:middle line:90% up. 00:03:18.230 --> 00:03:24.110 align:middle line:84% OK, so that's the idea of an accidental viewpoint, right? 00:03:24.110 --> 00:03:27.950 align:middle line:84% And it raises the question of what 00:03:27.950 --> 00:03:29.710 align:middle line:84% is it about accidental views that 00:03:29.710 --> 00:03:31.610 align:middle line:90% causes us not to assume them? 00:03:31.610 --> 00:03:34.390 align:middle line:84% So intuitively, you kind of look at that image, 00:03:34.390 --> 00:03:36.710 align:middle line:84% and it seems the inference that that's 00:03:36.710 --> 00:03:38.970 align:middle line:84% a cube doesn't seem like the best idea. 00:03:38.970 --> 00:03:42.170 align:middle line:90% But it's hard to make precise. 00:03:42.170 --> 00:03:45.030 align:middle line:84% And so there was this beautiful paper that came out 00:03:45.030 --> 00:03:48.930 align:middle line:90% in the mid-'90s by Bill Freeman. 00:03:48.930 --> 00:03:52.350 align:middle line:84% So Bill Freeman is a well-known computer vision researcher 00:03:52.350 --> 00:03:54.470 align:middle line:90% who works in CSAIL. 00:03:54.470 --> 00:03:57.750 align:middle line:84% At the time he was at MERL, Mitsubishi Electric Research 00:03:57.750 --> 00:03:58.250 align:middle line:90% Labs. 00:03:58.250 --> 00:04:01.150 align:middle line:84% That's like right around the corner. 00:04:01.150 --> 00:04:07.670 align:middle line:84% And he kind of came up with this really elegant way 00:04:07.670 --> 00:04:09.950 align:middle line:84% of thinking about the generic viewpoint 00:04:09.950 --> 00:04:14.990 align:middle line:84% assumption that kind of clarifies everything 00:04:14.990 --> 00:04:17.550 align:middle line:90% in a really beautiful way. 00:04:17.550 --> 00:04:20.950 align:middle line:84% And it also kind of links up to some of the Bayesian inference 00:04:20.950 --> 00:04:23.930 align:middle line:84% principles that we've talked about elsewhere in this class. 00:04:23.930 --> 00:04:25.430 align:middle line:84% And so that's one of the reasons why 00:04:25.430 --> 00:04:26.810 align:middle line:90% I kind of wanted to go over it. 00:04:26.810 --> 00:04:30.850 align:middle line:84% So in all of these examples that we've been talking about so far, 00:04:30.850 --> 00:04:32.590 align:middle line:84% we've been talking about the notion 00:04:32.590 --> 00:04:35.590 align:middle line:84% of generic and accidental viewpoints. 00:04:35.590 --> 00:04:41.350 align:middle line:84% But the notion that a scene can be generic or accidental 00:04:41.350 --> 00:04:43.710 align:middle line:90% extends to other things as well. 00:04:43.710 --> 00:04:47.230 align:middle line:84% So this is an example where this notion of generic and accidental 00:04:47.230 --> 00:04:50.230 align:middle line:84% extends to shape and illumination. 00:04:50.230 --> 00:04:54.150 align:middle line:84% So remember, shape from shading is 00:04:54.150 --> 00:04:56.110 align:middle line:84% one of the ways by which we infer 00:04:56.110 --> 00:04:58.450 align:middle line:84% the three-dimensional shape of things in the world. 00:04:58.450 --> 00:05:01.258 align:middle line:84% And so the idea is that if surfaces are Lambertian, 00:05:01.258 --> 00:05:03.550 align:middle line:84% then the amount of light that is reflected by a surface 00:05:03.550 --> 00:05:07.590 align:middle line:84% depends on the angle between the incident illumination 00:05:07.590 --> 00:05:10.670 align:middle line:90% and the surface orientation. 00:05:10.670 --> 00:05:12.750 align:middle line:90% So here's an example image. 00:05:12.750 --> 00:05:14.670 align:middle line:84% It's got one of these little gradients that 00:05:14.670 --> 00:05:17.510 align:middle line:84% sometimes looks like a bump, sometimes looks like a crater. 00:05:17.510 --> 00:05:21.950 align:middle line:84% And this is a set of possible scenes 00:05:21.950 --> 00:05:24.050 align:middle line:90% that could generate the image. 00:05:24.050 --> 00:05:26.670 align:middle line:84% And so each one of these is a three-dimensional shape 00:05:26.670 --> 00:05:29.550 align:middle line:84% indicated by this grid paired with the direction 00:05:29.550 --> 00:05:30.670 align:middle line:90% of illumination. 00:05:30.670 --> 00:05:33.210 align:middle line:84% So far when we talked about shape from shading, 00:05:33.210 --> 00:05:38.117 align:middle line:84% we talked about the idea that the shape and the illumination 00:05:38.117 --> 00:05:39.450 align:middle line:90% could combine in different ways. 00:05:39.450 --> 00:05:40.550 align:middle line:90% So the problem is ill-posed. 00:05:40.550 --> 00:05:41.967 align:middle line:84% You can only do shape from shading 00:05:41.967 --> 00:05:43.950 align:middle line:84% if you know the direction of the illumination. 00:05:43.950 --> 00:05:46.750 align:middle line:84% And so in the classic example of craters and bumps, 00:05:46.750 --> 00:05:48.750 align:middle line:84% they can be craters with the illumination coming 00:05:48.750 --> 00:05:51.300 align:middle line:84% from one direction or bumps with the illumination coming 00:05:51.300 --> 00:05:52.100 align:middle line:90% from the other. 00:05:52.100 --> 00:05:54.660 align:middle line:84% So that's what we have on the left and the right, number 00:05:54.660 --> 00:05:55.900 align:middle line:90% 1 and number 5. 00:05:55.900 --> 00:05:58.740 align:middle line:84% But they're also like all of these kind 00:05:58.740 --> 00:06:01.460 align:middle line:84% of funky shapes, which when paired 00:06:01.460 --> 00:06:04.580 align:middle line:84% with exactly the right type of illumination, 00:06:04.580 --> 00:06:05.800 align:middle line:90% everything kind of lines up. 00:06:05.800 --> 00:06:08.540 align:middle line:84% The illumination lines up with the shape variation 00:06:08.540 --> 00:06:12.980 align:middle line:84% in exactly the right way to give you that observed image. 00:06:12.980 --> 00:06:17.662 align:middle line:84% So the idea is that this is kind of like an accidental alignment 00:06:17.662 --> 00:06:19.120 align:middle line:84% of the illumination with the shape. 00:06:19.120 --> 00:06:22.120 align:middle line:84% It's conceptually similar to the idea of an accidental viewpoint, 00:06:22.120 --> 00:06:24.860 align:middle line:84% where things line up in exactly the right way to create 00:06:24.860 --> 00:06:26.220 align:middle line:90% a special image. 00:06:26.220 --> 00:06:34.380 align:middle line:84% And this turns out to be kind of an analytically tractable 00:06:34.380 --> 00:06:36.860 align:middle line:84% version of the problem with which 00:06:36.860 --> 00:06:40.100 align:middle line:84% to think about the kind of general issue. 00:06:40.100 --> 00:06:42.460 align:middle line:90% And so the essential idea here-- 00:06:42.460 --> 00:06:47.860 align:middle line:84% so the question is, it's intuitively obvious 00:06:47.860 --> 00:06:52.160 align:middle line:84% that the good or the correct interpretations of that image 00:06:52.160 --> 00:06:54.180 align:middle line:90% is this and this. 00:06:54.180 --> 00:06:56.720 align:middle line:84% These seem like they're not very good interpretations. 00:06:56.720 --> 00:06:58.160 align:middle line:90% The question is, why is that? 00:06:58.160 --> 00:07:00.260 align:middle line:84% And in what sense are they not good? 00:07:00.260 --> 00:07:04.780 align:middle line:84% And the essential intuition is that these situations where 00:07:04.780 --> 00:07:09.340 align:middle line:84% you get accidents in the scene interpretation 00:07:09.340 --> 00:07:11.660 align:middle line:84% are situations where the image that 00:07:11.660 --> 00:07:13.460 align:middle line:84% would be rendered by that accidental scene 00:07:13.460 --> 00:07:16.800 align:middle line:84% is very unstable with respect to the scene parameters. 00:07:16.800 --> 00:07:21.660 align:middle line:84% So here we have these two possible shapes 00:07:21.660 --> 00:07:23.540 align:middle line:84% that, with the right illumination, 00:07:23.540 --> 00:07:28.360 align:middle line:84% can produce the observed image, which was something like this. 00:07:28.360 --> 00:07:33.040 align:middle line:84% And what is shown here are five different images. 00:07:33.040 --> 00:07:35.680 align:middle line:84% So the one in the middle is the one that is the observed image. 00:07:35.680 --> 00:07:37.700 align:middle line:84% And the other ones are the images 00:07:37.700 --> 00:07:40.460 align:middle line:84% that would be rendered for slightly different directions 00:07:40.460 --> 00:07:42.220 align:middle line:90% of illumination. 00:07:42.220 --> 00:07:47.500 align:middle line:84% And so the point is that for the bottom shape, for the bump, 00:07:47.500 --> 00:07:52.380 align:middle line:84% the image that would be generated with these slightly 00:07:52.380 --> 00:07:53.980 align:middle line:84% different directions of illumination 00:07:53.980 --> 00:07:55.360 align:middle line:90% does not change very much. 00:07:55.360 --> 00:07:57.018 align:middle line:90% Those all look the same. 00:07:57.018 --> 00:07:59.060 align:middle line:84% So they're slightly different, but the difference 00:07:59.060 --> 00:08:01.380 align:middle line:90% is pretty small. 00:08:01.380 --> 00:08:03.780 align:middle line:84% Whereas for the accidental scenario-- 00:08:03.780 --> 00:08:07.580 align:middle line:84% so given this shape, if you change the illumination just 00:08:07.580 --> 00:08:11.542 align:middle line:84% a hair, the image that you would get 00:08:11.542 --> 00:08:13.500 align:middle line:84% from that combination of shape and illumination 00:08:13.500 --> 00:08:15.390 align:middle line:84% changes a lot, because you really 00:08:15.390 --> 00:08:17.140 align:middle line:84% have to get the shape and the illumination 00:08:17.140 --> 00:08:21.340 align:middle line:84% to line up exactly right in order to render that image. 00:08:21.340 --> 00:08:23.420 align:middle line:84% So that's the first idea, is that there's 00:08:23.420 --> 00:08:29.520 align:middle line:84% something very different about these scene interpretations, 00:08:29.520 --> 00:08:33.659 align:middle line:84% which is this one is stable and this one is unstable. 00:08:33.659 --> 00:08:34.159 align:middle line:90% All right. 00:08:34.159 --> 00:08:35.440 align:middle line:90% Well, so what, you might say. 00:08:35.440 --> 00:08:39.299 align:middle line:84% So let's think about the relevance of this 00:08:39.299 --> 00:08:45.660 align:middle line:84% for good old Bayesian theories of perception. 00:08:45.660 --> 00:08:49.900 align:middle line:84% OK, so remember how the probability of a scene 00:08:49.900 --> 00:08:52.180 align:middle line:84% interpretation given the image can be 00:08:52.180 --> 00:08:53.920 align:middle line:90% formalized with Bayes' theorem. 00:08:53.920 --> 00:08:58.340 align:middle line:84% So we generally think of the goal of perception 00:08:58.340 --> 00:09:05.220 align:middle line:84% as finding the scene that is most likely given some observed 00:09:05.220 --> 00:09:06.940 align:middle line:90% stimulus, like an image. 00:09:06.940 --> 00:09:09.980 align:middle line:84% So that's the posterior probability 00:09:09.980 --> 00:09:11.420 align:middle line:90% of the scene given the image. 00:09:11.420 --> 00:09:14.500 align:middle line:84% We want to maybe maximize that or find a few scenes 00:09:14.500 --> 00:09:16.620 align:middle line:90% for which that's high. 00:09:16.620 --> 00:09:18.600 align:middle line:90% So here's the posterior. 00:09:18.600 --> 00:09:20.500 align:middle line:84% All right, now, this looks a little different 00:09:20.500 --> 00:09:22.320 align:middle line:84% than what you have been used to seeing, 00:09:22.320 --> 00:09:25.840 align:middle line:84% because now there are two variables here. 00:09:25.840 --> 00:09:28.540 align:middle line:90% There's beta and there's x. 00:09:28.540 --> 00:09:30.860 align:middle line:84% So we're now dealing with a situation 00:09:30.860 --> 00:09:34.900 align:middle line:84% where the scene is defined by at least two variables. 00:09:34.900 --> 00:09:38.420 align:middle line:84% So in this particular setting, one of those variables 00:09:38.420 --> 00:09:41.700 align:middle line:84% is the shape of the surface and the other variable 00:09:41.700 --> 00:09:45.090 align:middle line:84% is the direction of the illumination. 00:09:45.090 --> 00:09:48.810 align:middle line:84% And so we're assuming that this is 00:09:48.810 --> 00:09:52.010 align:middle line:84% a setting in which one of those variables 00:09:52.010 --> 00:09:54.470 align:middle line:84% is something that you particularly care about. 00:09:54.470 --> 00:09:56.570 align:middle line:84% So in this particular case, you want 00:09:56.570 --> 00:09:58.290 align:middle line:90% to estimate what the shape is. 00:09:58.290 --> 00:10:03.070 align:middle line:84% But there's also this generic variable or parameter, 00:10:03.070 --> 00:10:06.290 align:middle line:84% the illumination that's going to affect whether or not 00:10:06.290 --> 00:10:10.290 align:middle line:90% the shape can explain the image. 00:10:10.290 --> 00:10:12.450 align:middle line:90% So this is just Bayes' theorem. 00:10:12.450 --> 00:10:15.410 align:middle line:90% So we've got the posterior here. 00:10:15.410 --> 00:10:17.310 align:middle line:84% Anybody remember what is this thing called? 00:10:17.310 --> 00:10:21.010 align:middle line:90% 00:10:21.010 --> 00:10:22.330 align:middle line:90% y is the image. 00:10:22.330 --> 00:10:22.830 align:middle line:90% What's that? 00:10:22.830 --> 00:10:24.170 align:middle line:90% What is this? 00:10:24.170 --> 00:10:26.928 align:middle line:84% Probability of the image given the scene interpretation, yeah? 00:10:26.928 --> 00:10:27.970 align:middle line:90% AUDIENCE: The likelihood. 00:10:27.970 --> 00:10:29.350 align:middle line:84% JOSH MCDERMOTT: Yep, that's the likelihood. 00:10:29.350 --> 00:10:30.808 align:middle line:84% So it just looks a little different 00:10:30.808 --> 00:10:33.330 align:middle line:84% because you've got two variables here. 00:10:33.330 --> 00:10:36.356 align:middle line:90% And remember, what is this? 00:10:36.356 --> 00:10:37.050 align:middle line:90% AUDIENCE: Prior. 00:10:37.050 --> 00:10:38.675 align:middle line:84% JOSH MCDERMOTT: That's the prior, yeah. 00:10:38.675 --> 00:10:42.170 align:middle line:84% So you've got two priors, one over the shape 00:10:42.170 --> 00:10:44.262 align:middle line:90% and one over the illumination. 00:10:44.262 --> 00:10:46.470 align:middle line:84% So this is just the thing that you're used to seeing. 00:10:46.470 --> 00:10:48.637 align:middle line:84% It just looks a little different because there's now 00:10:48.637 --> 00:10:49.450 align:middle line:90% two variables. 00:10:49.450 --> 00:10:53.190 align:middle line:90% OK, so here's the catch. 00:10:53.190 --> 00:10:56.170 align:middle line:84% So let's just suppose that what we really want to do here 00:10:56.170 --> 00:10:57.530 align:middle line:90% is estimate shape. 00:10:57.530 --> 00:11:02.510 align:middle line:84% So really what you want to do is maximize this. 00:11:02.510 --> 00:11:04.850 align:middle line:84% You want to find the shape that's 00:11:04.850 --> 00:11:07.490 align:middle line:90% most probable given the image. 00:11:07.490 --> 00:11:09.210 align:middle line:90% So how do you get that? 00:11:09.210 --> 00:11:12.370 align:middle line:84% Well, you take the joint distribution 00:11:12.370 --> 00:11:17.050 align:middle line:84% of shape and illumination, and you integrate with respect 00:11:17.050 --> 00:11:21.410 align:middle line:84% to the generic variable, the illumination, which is x. 00:11:21.410 --> 00:11:24.130 align:middle line:84% So if you've got a joint distribution and you want to get 00:11:24.130 --> 00:11:26.308 align:middle line:84% the marginal distribution-- so in this case, 00:11:26.308 --> 00:11:28.350 align:middle line:84% you start out with the distribution of beta and x 00:11:28.350 --> 00:11:30.770 align:middle line:84% and you want to just get the distribution over beta-- 00:11:30.770 --> 00:11:33.390 align:middle line:84% you integrate over the other variable. 00:11:33.390 --> 00:11:37.450 align:middle line:84% So this is just standard probability theory. 00:11:37.450 --> 00:11:41.530 align:middle line:84% And so this is just the integral of this thing 00:11:41.530 --> 00:11:44.970 align:middle line:84% with the quantities that don't depend on x kind of brought out 00:11:44.970 --> 00:11:48.350 align:middle line:90% in front of the integral. 00:11:48.350 --> 00:11:50.370 align:middle line:84% So then what's inside the integral? 00:11:50.370 --> 00:11:53.650 align:middle line:84% Well, we've got our likelihood here 00:11:53.650 --> 00:11:57.650 align:middle line:84% and then the prior probability of the illumination. 00:11:57.650 --> 00:12:00.650 align:middle line:84% But the key thing here that's going to be really important 00:12:00.650 --> 00:12:01.590 align:middle line:90% is the likelihood. 00:12:01.590 --> 00:12:03.370 align:middle line:90% So what is this integral doing? 00:12:03.370 --> 00:12:07.010 align:middle line:84% Well, you can think of it as adding up the likelihood here 00:12:07.010 --> 00:12:10.390 align:middle line:84% for all different possible values of the illumination. 00:12:10.390 --> 00:12:15.290 align:middle line:90% 00:12:15.290 --> 00:12:17.350 align:middle line:84% All right, so let's review the likelihood. 00:12:17.350 --> 00:12:24.650 align:middle line:84% And I'll say there was a question on the last exam asking 00:12:24.650 --> 00:12:28.070 align:middle line:84% about the likelihood, and not everybody got it right. 00:12:28.070 --> 00:12:30.150 align:middle line:84% Some people kind of got it partially right. 00:12:30.150 --> 00:12:32.290 align:middle line:84% So this is a good thing to review. 00:12:32.290 --> 00:12:35.290 align:middle line:84% So the key idea of the likelihood 00:12:35.290 --> 00:12:38.570 align:middle line:84% is it measures the extent to which particular scene 00:12:38.570 --> 00:12:43.770 align:middle line:84% parameters can account for an observed image y. 00:12:43.770 --> 00:12:46.010 align:middle line:84% So typically, likelihood functions 00:12:46.010 --> 00:12:49.250 align:middle line:84% will assume a Gaussian probability. 00:12:49.250 --> 00:12:51.610 align:middle line:90% And that's what's written here. 00:12:51.610 --> 00:12:53.070 align:middle line:90% So it's our likelihood. 00:12:53.070 --> 00:12:55.150 align:middle line:84% And it just says that it equals this thing. 00:12:55.150 --> 00:12:57.530 align:middle line:84% But the key thing to note here is 00:12:57.530 --> 00:13:04.222 align:middle line:84% that we've got y minus this function f of x and beta. 00:13:04.222 --> 00:13:05.930 align:middle line:84% So remember, x and beta are the variables 00:13:05.930 --> 00:13:07.970 align:middle line:90% that describe the scene. 00:13:07.970 --> 00:13:11.450 align:middle line:84% x is the illumination, beta is the shape, 00:13:11.450 --> 00:13:15.650 align:middle line:90% and y is the observed image. 00:13:15.650 --> 00:13:19.570 align:middle line:84% f here is a rendering function, which 00:13:19.570 --> 00:13:23.130 align:middle line:84% gives the image created by the generic-- that's 00:13:23.130 --> 00:13:25.450 align:middle line:84% the illumination-- and scene parameters-- that's 00:13:25.450 --> 00:13:30.110 align:middle line:84% the shape-- x and beta So what is this doing? 00:13:30.110 --> 00:13:34.050 align:middle line:84% Well, it's a comparison between the observed image and the image 00:13:34.050 --> 00:13:38.560 align:middle line:84% that would be generated by these hypothesized scene parameters. 00:13:38.560 --> 00:13:40.600 align:middle line:84% So that's what the likelihood does. 00:13:40.600 --> 00:13:44.120 align:middle line:84% It says, for a given hypothesis about the world, what 00:13:44.120 --> 00:13:47.140 align:middle line:84% is the stimulus that that would generate? 00:13:47.140 --> 00:13:50.960 align:middle line:84% And how similar is that to the observed stimulus? 00:13:50.960 --> 00:13:54.023 align:middle line:84% So this is just one example of a likelihood function. 00:13:54.023 --> 00:13:55.440 align:middle line:84% So the key thing is the difference 00:13:55.440 --> 00:13:58.573 align:middle line:84% between the observed image and the rendered image 00:13:58.573 --> 00:14:00.240 align:middle line:84% that you would get for particular values 00:14:00.240 --> 00:14:02.960 align:middle line:90% of illumination and shape. 00:14:02.960 --> 00:14:07.360 align:middle line:84% And the fact that this is inside this normal distribution 00:14:07.360 --> 00:14:09.880 align:middle line:84% is because we're assuming Gaussian noise in this case. 00:14:09.880 --> 00:14:12.380 align:middle line:84% But the key idea is that it's just taking a difference here. 00:14:12.380 --> 00:14:16.240 align:middle line:90% 00:14:16.240 --> 00:14:16.900 align:middle line:90% OK. 00:14:16.900 --> 00:14:22.160 align:middle line:84% So we're trying to compute the probability of a given 00:14:22.160 --> 00:14:23.920 align:middle line:90% shape given the image. 00:14:23.920 --> 00:14:27.520 align:middle line:84% We've got to integrate the likelihood here 00:14:27.520 --> 00:14:31.520 align:middle line:90% over illumination. 00:14:31.520 --> 00:14:33.040 align:middle line:84% And each value of the likelihood is 00:14:33.040 --> 00:14:34.457 align:middle line:84% going to be comparing the observed 00:14:34.457 --> 00:14:38.120 align:middle line:84% image to the image that's predicted by particular values 00:14:38.120 --> 00:14:41.920 align:middle line:90% of illumination and shape. 00:14:41.920 --> 00:14:45.640 align:middle line:84% So that predicted image is what is shown up here. 00:14:45.640 --> 00:14:50.240 align:middle line:84% So these images are the output of a computer graphics program 00:14:50.240 --> 00:14:52.560 align:middle line:84% that takes shape and illumination 00:14:52.560 --> 00:14:54.800 align:middle line:90% and generates an image. 00:14:54.800 --> 00:14:59.960 align:middle line:84% So we're taking an integral, remember, over the illumination 00:14:59.960 --> 00:15:01.320 align:middle line:90% direction. 00:15:01.320 --> 00:15:05.240 align:middle line:84% And that integral is going to add up the likelihood for each 00:15:05.240 --> 00:15:06.760 align:middle line:90% of these images. 00:15:06.760 --> 00:15:10.800 align:middle line:84% The likelihood will be high whenever the rendered image is 00:15:10.800 --> 00:15:13.440 align:middle line:90% similar to the observed image. 00:15:13.440 --> 00:15:16.080 align:middle line:84% So the key idea here is that in this scenario here 00:15:16.080 --> 00:15:21.320 align:middle line:84% for this shape, you're adding up a bunch of likelihoods that 00:15:21.320 --> 00:15:24.800 align:middle line:84% are going to be consistently high, because irrespective 00:15:24.800 --> 00:15:27.320 align:middle line:84% of the exact direction of the illumination, 00:15:27.320 --> 00:15:30.880 align:middle line:84% you get out an image that is pretty close to the observed 00:15:30.880 --> 00:15:32.920 align:middle line:90% image. 00:15:32.920 --> 00:15:35.600 align:middle line:84% Whereas for the funny shape, the likelihood 00:15:35.600 --> 00:15:38.320 align:middle line:84% is only high for this one special direction 00:15:38.320 --> 00:15:39.280 align:middle line:90% of illumination. 00:15:39.280 --> 00:15:41.680 align:middle line:84% And for all other slightly different directions 00:15:41.680 --> 00:15:44.030 align:middle line:84% of illumination, the image that is rendered 00:15:44.030 --> 00:15:45.780 align:middle line:84% is very different from the observed image, 00:15:45.780 --> 00:15:48.120 align:middle line:84% and so the likelihood will be low. 00:15:48.120 --> 00:15:52.680 align:middle line:84% And so the consequence is that when you do this integral, 00:15:52.680 --> 00:15:57.120 align:middle line:84% for the funny shapes, this sum or integral is going 00:15:57.120 --> 00:15:58.560 align:middle line:90% to end up being relatively low. 00:15:58.560 --> 00:16:01.760 align:middle line:90% 00:16:01.760 --> 00:16:03.660 align:middle line:90% And that's what happens. 00:16:03.660 --> 00:16:07.900 align:middle line:84% So here, we end up with the total score. 00:16:07.900 --> 00:16:11.600 align:middle line:84% It's like the posterior of all of these different shapes. 00:16:11.600 --> 00:16:17.000 align:middle line:84% And so the crater and the bump come out as pretty probable. 00:16:17.000 --> 00:16:20.320 align:middle line:84% And then the funny shapes come out as less probable. 00:16:20.320 --> 00:16:23.880 align:middle line:90% 00:16:23.880 --> 00:16:25.660 align:middle line:84% What questions do you have about that? 00:16:25.660 --> 00:16:40.080 align:middle line:90% 00:16:40.080 --> 00:16:40.955 align:middle line:90% Yeah. 00:16:40.955 --> 00:16:42.580 align:middle line:84% AUDIENCE: Like, what are these numbers? 00:16:42.580 --> 00:16:46.187 align:middle line:90% 00:16:46.187 --> 00:16:48.520 align:middle line:84% JOSH MCDERMOTT: There's some constant of proportionality 00:16:48.520 --> 00:16:51.520 align:middle line:84% that is going to turn them into probabilities. 00:16:51.520 --> 00:16:52.680 align:middle line:90% I don't remember. 00:16:52.680 --> 00:16:57.082 align:middle line:84% Yeah, it's proportional to the posterior. 00:16:57.082 --> 00:16:58.040 align:middle line:90% I don't know what the-- 00:16:58.040 --> 00:16:59.200 align:middle line:90% I forget what the units are. 00:16:59.200 --> 00:17:01.783 align:middle line:84% AUDIENCE: [INAUDIBLE] you expect the ones they have to do with 00:17:01.783 --> 00:17:02.363 align:middle line:90% 1/2? 00:17:02.363 --> 00:17:03.280 align:middle line:90% JOSH MCDERMOTT: To be? 00:17:03.280 --> 00:17:04.974 align:middle line:90% AUDIENCE: Sort of close to 1/2? 00:17:04.974 --> 00:17:07.099 align:middle line:84% JOSH MCDERMOTT: Ah, if this was actual probability? 00:17:07.099 --> 00:17:08.619 align:middle line:84% Yeah, it's not actual probability. 00:17:08.619 --> 00:17:10.940 align:middle line:84% It's something that's proportional to that. 00:17:10.940 --> 00:17:12.148 align:middle line:90% Yeah. 00:17:12.148 --> 00:17:14.440 align:middle line:84% So what matters is the relative height of these things. 00:17:14.440 --> 00:17:15.815 align:middle line:84% AUDIENCE: So it would be linearly 00:17:15.815 --> 00:17:18.020 align:middle line:84% proportional to the actual number? 00:17:18.020 --> 00:17:19.960 align:middle line:90% JOSH MCDERMOTT: I think so. 00:17:19.960 --> 00:17:23.319 align:middle line:84% It's been a long time since I read the paper in detail, 00:17:23.319 --> 00:17:24.640 align:middle line:90% so I don't remember. 00:17:24.640 --> 00:17:25.830 align:middle line:84% What really matters here is just the fact 00:17:25.830 --> 00:17:27.030 align:middle line:84% that these are a lot higher than that. 00:17:27.030 --> 00:17:28.990 align:middle line:84% So I'm just trying to give you the intuitive 00:17:28.990 --> 00:17:32.270 align:middle line:90% kind of explanation here. 00:17:32.270 --> 00:17:33.030 align:middle line:90% Yeah. 00:17:33.030 --> 00:17:34.010 align:middle line:90% Other questions? 00:17:34.010 --> 00:17:38.230 align:middle line:90% 00:17:38.230 --> 00:17:42.110 align:middle line:90% So here's another example. 00:17:42.110 --> 00:17:47.710 align:middle line:84% So this is an image of a piece of sculpture. 00:17:47.710 --> 00:17:51.350 align:middle line:84% And it's been corrupted by noise. 00:17:51.350 --> 00:17:55.670 align:middle line:84% And so here we have two different possible shapes that 00:17:55.670 --> 00:17:59.030 align:middle line:90% could account for the image. 00:17:59.030 --> 00:18:01.150 align:middle line:84% What you see, obviously, is this. 00:18:01.150 --> 00:18:04.930 align:middle line:84% But this kind of crazy-looking shape, 00:18:04.930 --> 00:18:07.170 align:middle line:84% if lit from exactly the right direction, 00:18:07.170 --> 00:18:12.510 align:middle line:84% would generate something very, very close to that image. 00:18:12.510 --> 00:18:19.910 align:middle line:84% But the key concept is that because the illumination has 00:18:19.910 --> 00:18:23.470 align:middle line:84% to be exactly right for that shape to generate the image, 00:18:23.470 --> 00:18:26.750 align:middle line:84% if you integrate out illumination, 00:18:26.750 --> 00:18:29.950 align:middle line:84% this thing ends up being a lot more likely. 00:18:29.950 --> 00:18:32.310 align:middle line:90% That's what you see. 00:18:32.310 --> 00:18:32.810 align:middle line:90% All right. 00:18:32.810 --> 00:18:36.670 align:middle line:84% So the take-home message here is that we don't actually 00:18:36.670 --> 00:18:39.130 align:middle line:84% really need a generic viewpoint assumption. 00:18:39.130 --> 00:18:42.230 align:middle line:90% No assumption is really needed. 00:18:42.230 --> 00:18:48.030 align:middle line:84% Interpretations under generic views or generic anything 00:18:48.030 --> 00:18:50.750 align:middle line:84% are just more likely given standard probabilistic 00:18:50.750 --> 00:18:54.350 align:middle line:84% inference, assuming that we integrate over variables that we 00:18:54.350 --> 00:18:56.910 align:middle line:84% don't need to estimate precisely. 00:18:56.910 --> 00:19:01.970 align:middle line:84% And so that's actually a slightly non-trivial statement. 00:19:01.970 --> 00:19:07.910 align:middle line:84% So all of this is positing that when you are seeing shape, 00:19:07.910 --> 00:19:09.690 align:middle line:84% you're integrating over illumination. 00:19:09.690 --> 00:19:11.330 align:middle line:84% It's like you don't really care about the illumination. 00:19:11.330 --> 00:19:12.450 align:middle line:84% You're not trying to estimate that. 00:19:12.450 --> 00:19:14.150 align:middle line:84% You're just trying to estimate shape. 00:19:14.150 --> 00:19:19.450 align:middle line:84% And that is maybe an assumption about the content of perception, 00:19:19.450 --> 00:19:20.530 align:middle line:90% which is non-trivial. 00:19:20.530 --> 00:19:21.030 align:middle line:90% Yeah. 00:19:21.030 --> 00:19:23.550 align:middle line:84% AUDIENCE: My question is in regards to the noise. 00:19:23.550 --> 00:19:26.407 align:middle line:84% Because we saw that in the other photo, it was noisy. 00:19:26.407 --> 00:19:28.490 align:middle line:84% And that's why there could be two interpretations. 00:19:28.490 --> 00:19:31.950 align:middle line:84% Is there a certain percentage, certain amount of noise 00:19:31.950 --> 00:19:34.030 align:middle line:84% in which those two kind of perceptions 00:19:34.030 --> 00:19:36.102 align:middle line:84% would be more equal in our brains? 00:19:36.102 --> 00:19:37.810 align:middle line:84% Like, the thing gets noisier and noisier, 00:19:37.810 --> 00:19:40.435 align:middle line:84% and eventually we can't really tell the difference? 00:19:40.435 --> 00:19:41.310 align:middle line:90% JOSH MCDERMOTT: Yeah. 00:19:41.310 --> 00:19:42.685 align:middle line:84% OK, so first of all, I should say 00:19:42.685 --> 00:19:45.870 align:middle line:84% that in this particular case, the noise here 00:19:45.870 --> 00:19:52.310 align:middle line:84% was added to actually cause this shape to better account 00:19:52.310 --> 00:19:54.030 align:middle line:90% for the image than this one. 00:19:54.030 --> 00:19:58.950 align:middle line:84% So given exactly the right illumination, 00:19:58.950 --> 00:20:01.942 align:middle line:84% the likelihood is actually higher for this than for this. 00:20:01.942 --> 00:20:03.650 align:middle line:84% And that's what this is supposed to show. 00:20:03.650 --> 00:20:05.790 align:middle line:84% So these are the images that would be 00:20:05.790 --> 00:20:07.067 align:middle line:90% rendered by these two shapes. 00:20:07.067 --> 00:20:09.150 align:middle line:84% And so the point is that this one doesn't actually 00:20:09.150 --> 00:20:12.030 align:middle line:84% model the noise, whereas this one does. 00:20:12.030 --> 00:20:14.190 align:middle line:84% So it's just sort of a contrived scenario that's 00:20:14.190 --> 00:20:16.190 align:middle line:84% supposed to make the point that even if you have 00:20:16.190 --> 00:20:19.470 align:middle line:84% a situation where you could, in principle, better 00:20:19.470 --> 00:20:23.927 align:middle line:84% account for the image with this crazy interpretation, 00:20:23.927 --> 00:20:26.510 align:middle line:84% once you take into account the fact that you need to integrate 00:20:26.510 --> 00:20:29.230 align:middle line:84% over illumination, this is going to end up being 00:20:29.230 --> 00:20:30.830 align:middle line:90% the preferred explanation. 00:20:30.830 --> 00:20:32.910 align:middle line:90% Yeah. 00:20:32.910 --> 00:20:33.410 align:middle line:90% OK. 00:20:33.410 --> 00:20:37.070 align:middle line:84% So key concept here is, again, the idea 00:20:37.070 --> 00:20:40.350 align:middle line:84% that we can think about perception 00:20:40.350 --> 00:20:44.990 align:middle line:84% as finding the most likely scene interpretation, given 00:20:44.990 --> 00:20:47.990 align:middle line:84% an image or sound or whatever kind of sensory input 00:20:47.990 --> 00:20:53.230 align:middle line:84% you're dealing with, that we're trying to maximize 00:20:53.230 --> 00:20:55.430 align:middle line:84% the posterior, that you can decompose 00:20:55.430 --> 00:20:58.270 align:middle line:84% the posterior into the likelihood and the prior. 00:20:58.270 --> 00:21:00.950 align:middle line:84% That the likelihood involves a comparison 00:21:00.950 --> 00:21:04.630 align:middle line:84% between the observed stimulus and the stimulus that would 00:21:04.630 --> 00:21:07.470 align:middle line:90% be generated by a hypothesis. 00:21:07.470 --> 00:21:10.890 align:middle line:84% And then in this particular application of this, 00:21:10.890 --> 00:21:14.150 align:middle line:84% there's this additional wrinkle that if you've 00:21:14.150 --> 00:21:17.070 align:middle line:84% got multiple variables in your scene interpretation 00:21:17.070 --> 00:21:19.500 align:middle line:84% and you're trying to estimate one of them, 00:21:19.500 --> 00:21:21.360 align:middle line:84% then you iterate over the other one. 00:21:21.360 --> 00:21:24.940 align:middle line:84% And then that kind of causes this generic viewpoint 00:21:24.940 --> 00:21:26.400 align:middle line:90% assumption. 00:21:26.400 --> 00:21:27.900 align:middle line:84% And so the other thing that I'll say 00:21:27.900 --> 00:21:31.300 align:middle line:84% is that there's lots of different applications 00:21:31.300 --> 00:21:37.020 align:middle line:84% of Bayesian approaches to perception. 00:21:37.020 --> 00:21:38.780 align:middle line:84% We've talked about some applications 00:21:38.780 --> 00:21:44.140 align:middle line:84% where the work is really being done by the prior. 00:21:44.140 --> 00:21:47.540 align:middle line:84% So for instance, when we talked about motion perception, 00:21:47.540 --> 00:21:51.500 align:middle line:84% the critical thing to explaining some of these motion illusions 00:21:51.500 --> 00:21:54.620 align:middle line:84% is that there's a prior that favors slow speeds. 00:21:54.620 --> 00:21:57.740 align:middle line:84% That kind of does a lot of the work for you. 00:21:57.740 --> 00:22:00.700 align:middle line:84% This is a case where actually the prior is not 00:22:00.700 --> 00:22:01.640 align:middle line:90% really super relevant. 00:22:01.640 --> 00:22:03.917 align:middle line:84% In fact, I think they assumed a uniform prior. 00:22:03.917 --> 00:22:06.500 align:middle line:84% You could imagine maybe making this more realistic by actually 00:22:06.500 --> 00:22:08.740 align:middle line:84% having a prior that favors illumination from above, which 00:22:08.740 --> 00:22:10.282 align:middle line:84% maybe is a little bit more realistic. 00:22:10.282 --> 00:22:13.300 align:middle line:84% But here, really, the work is done by the likelihood 00:22:13.300 --> 00:22:16.380 align:middle line:84% and by essentially probability theory, just basically saying 00:22:16.380 --> 00:22:19.860 align:middle line:84% that you have to integrate over the other variable 00:22:19.860 --> 00:22:21.660 align:middle line:84% that you're not trying to estimate. 00:22:21.660 --> 00:22:24.900 align:middle line:84% But it's the same basic kind of mathematical framework 00:22:24.900 --> 00:22:27.140 align:middle line:84% and the same kinds of ingredients. 00:22:27.140 --> 00:22:31.900 align:middle line:84% So this same kind of idea comes up 00:22:31.900 --> 00:22:36.100 align:middle line:84% a lot in a lot of different places in perception. 00:22:36.100 --> 00:22:38.740 align:middle line:84% There's some interesting ones with illusory contours 00:22:38.740 --> 00:22:41.280 align:middle line:84% that are kind of fun to look at and think about. 00:22:41.280 --> 00:22:44.940 align:middle line:84% So if you look at these two displays, the one on the left, 00:22:44.940 --> 00:22:47.900 align:middle line:84% people typically see much stronger illusory contour 00:22:47.900 --> 00:22:50.340 align:middle line:90% than the one on the right. 00:22:50.340 --> 00:22:52.500 align:middle line:90% Is that what you all are seeing? 00:22:52.500 --> 00:22:55.360 align:middle line:84% Very clear kind of illusory square on the left, 00:22:55.360 --> 00:22:58.820 align:middle line:90% not so clear on the right. 00:22:58.820 --> 00:23:02.420 align:middle line:84% And you can at least qualitatively 00:23:02.420 --> 00:23:09.660 align:middle line:84% think about this in terms of generic and accidental views. 00:23:09.660 --> 00:23:14.420 align:middle line:90% So this particular display-- 00:23:14.420 --> 00:23:17.620 align:middle line:90% so let me back up. 00:23:17.620 --> 00:23:21.620 align:middle line:84% One way to think about what we see when we look at this 00:23:21.620 --> 00:23:25.780 align:middle line:84% is the idea that there are two main possible explanations 00:23:25.780 --> 00:23:28.620 align:middle line:90% for the image. 00:23:28.620 --> 00:23:30.965 align:middle line:84% One is the kind of thing that most people see here, 00:23:30.965 --> 00:23:32.340 align:middle line:84% which is like, well, there's just 00:23:32.340 --> 00:23:36.500 align:middle line:84% a bunch of little patterns in the image that are 00:23:36.500 --> 00:23:38.740 align:middle line:90% all kind of just lying there. 00:23:38.740 --> 00:23:40.510 align:middle line:84% And the other is what people typically 00:23:40.510 --> 00:23:42.260 align:middle line:84% say they see when they look at this, which 00:23:42.260 --> 00:23:44.700 align:middle line:84% is that there's a white square which is kind 00:23:44.700 --> 00:23:47.060 align:middle line:90% of on top of some other stuff. 00:23:47.060 --> 00:23:50.360 align:middle line:84% Now, this one could, in principle, 00:23:50.360 --> 00:23:52.900 align:middle line:84% not have the white square and could just be explained 00:23:52.900 --> 00:23:55.460 align:middle line:90% by these features in the image. 00:23:55.460 --> 00:24:00.780 align:middle line:84% This one could, in principle, also have the white square. 00:24:00.780 --> 00:24:06.180 align:middle line:84% So the idea is that if the explanation of this stimulus 00:24:06.180 --> 00:24:10.860 align:middle line:84% is that there was a white square here, 00:24:10.860 --> 00:24:16.540 align:middle line:84% you would have to posit an accidental dental viewpoint. 00:24:16.540 --> 00:24:19.540 align:middle line:84% And specifically-- so imagine there's a white square here 00:24:19.540 --> 00:24:21.420 align:middle line:90% and you change your view. 00:24:21.420 --> 00:24:23.500 align:middle line:84% Then the image is going to change, 00:24:23.500 --> 00:24:27.260 align:middle line:84% because the square is now going to overlap with these features 00:24:27.260 --> 00:24:29.380 align:middle line:84% differently than it otherwise would. 00:24:29.380 --> 00:24:32.660 align:middle line:84% So you need to posit an accidental viewpoint here. 00:24:32.660 --> 00:24:35.745 align:middle line:84% Whereas in this case, that's not so true. 00:24:35.745 --> 00:24:37.120 align:middle line:84% Like, from a different viewpoint, 00:24:37.120 --> 00:24:38.495 align:middle line:84% well, you still get line endings. 00:24:38.495 --> 00:24:40.287 align:middle line:84% They just kind of move around a little bit. 00:24:40.287 --> 00:24:42.500 align:middle line:84% But the image doesn't really qualitatively change. 00:24:42.500 --> 00:24:50.980 align:middle line:84% So another case where this idea of generic and accidental views 00:24:50.980 --> 00:24:54.700 align:middle line:84% seems to have some explanatory power. 00:24:54.700 --> 00:24:56.780 align:middle line:84% And the framework that we just talked about 00:24:56.780 --> 00:25:02.020 align:middle line:84% gives us some mathematical basis for thinking about that, which 00:25:02.020 --> 00:25:04.780 align:middle line:84% is the idea is that you don't normally care about exactly 00:25:04.780 --> 00:25:07.322 align:middle line:84% what the viewpoint is, so you're going to integrate over that 00:25:07.322 --> 00:25:13.478 align:middle line:84% in order to estimate the things that you are trying to estimate. 00:25:13.478 --> 00:25:14.270 align:middle line:90% Here's another one. 00:25:14.270 --> 00:25:16.030 align:middle line:84% This one is maybe a little bit stronger. 00:25:16.030 --> 00:25:17.650 align:middle line:90% Same idea. 00:25:17.650 --> 00:25:20.570 align:middle line:84% So pretty strong illusory contours on the left 00:25:20.570 --> 00:25:23.410 align:middle line:90% and not so strong on the right. 00:25:23.410 --> 00:25:25.690 align:middle line:90% OK. 00:25:25.690 --> 00:25:26.190 align:middle line:90% OK. 00:25:26.190 --> 00:25:27.470 align:middle line:90% Any questions about that? 00:25:27.470 --> 00:25:31.730 align:middle line:90% 00:25:31.730 --> 00:25:32.530 align:middle line:90% Yeah. 00:25:32.530 --> 00:25:35.090 align:middle line:84% AUDIENCE: Could you explain how an accidental view 00:25:35.090 --> 00:25:39.690 align:middle line:84% is necessary for the stimulus on the right? 00:25:39.690 --> 00:25:42.672 align:middle line:84% JOSH MCDERMOTT: Well, yeah, so the notion is that-- 00:25:42.672 --> 00:25:45.130 align:middle line:84% so the question is, all right, why don't you actually infer 00:25:45.130 --> 00:25:47.870 align:middle line:90% that there's a square here? 00:25:47.870 --> 00:25:50.290 align:middle line:84% And in order to actually evaluate 00:25:50.290 --> 00:25:53.690 align:middle line:84% the probability of that interpretation, 00:25:53.690 --> 00:25:56.450 align:middle line:84% the proposal is that, well, you have to integrate 00:25:56.450 --> 00:25:58.470 align:middle line:90% over different viewpoints. 00:25:58.470 --> 00:26:00.490 align:middle line:84% And the point is that let's imagine 00:26:00.490 --> 00:26:03.110 align:middle line:90% that there was a square here. 00:26:03.110 --> 00:26:05.210 align:middle line:84% There would be one particular viewpoint 00:26:05.210 --> 00:26:07.890 align:middle line:84% where the combination of the square and all 00:26:07.890 --> 00:26:11.530 align:middle line:84% of these funny things would generate this image. 00:26:11.530 --> 00:26:14.530 align:middle line:84% And the idea is that in a world where there's 00:26:14.530 --> 00:26:17.070 align:middle line:84% a white square kind of hovering out in front of this thing, 00:26:17.070 --> 00:26:19.570 align:middle line:84% if the viewpoint changes a little bit, well then 00:26:19.570 --> 00:26:22.710 align:middle line:84% the square is going to start to occlude these corners. 00:26:22.710 --> 00:26:26.250 align:middle line:84% And the image that you would get from that potential scene 00:26:26.250 --> 00:26:31.010 align:middle line:84% would be very different from the one that is actually observed. 00:26:31.010 --> 00:26:34.650 align:middle line:84% So it's exactly analogous to that shape from shading example, 00:26:34.650 --> 00:26:38.290 align:middle line:84% where there's this one viewpoint where with the white square, 00:26:38.290 --> 00:26:40.150 align:middle line:84% you get exactly the observed image. 00:26:40.150 --> 00:26:42.450 align:middle line:84% And then for other viewpoints, you 00:26:42.450 --> 00:26:44.250 align:middle line:84% get something that deviates quite a lot 00:26:44.250 --> 00:26:46.090 align:middle line:90% from the observed image. 00:26:46.090 --> 00:26:48.650 align:middle line:84% Whereas with this example, that's not the case, 00:26:48.650 --> 00:26:51.930 align:middle line:84% because you don't have these funny corners here 00:26:51.930 --> 00:26:53.790 align:middle line:84% that would be able to get occluded or not. 00:26:53.790 --> 00:26:54.670 align:middle line:90% Does that make sense? 00:26:54.670 --> 00:26:57.370 align:middle line:90% 00:26:57.370 --> 00:26:57.870 align:middle line:90% All right. 00:26:57.870 --> 00:26:59.430 align:middle line:84% Let's talk about object recognition. 00:26:59.430 --> 00:27:03.910 align:middle line:90% 00:27:03.910 --> 00:27:11.130 align:middle line:84% So this is a simplified view of visual processing. 00:27:11.130 --> 00:27:13.647 align:middle line:84% So we spent the first part of the section 00:27:13.647 --> 00:27:15.730 align:middle line:84% of the class on vision, sort of talking about what 00:27:15.730 --> 00:27:16.790 align:middle line:90% we called early vision. 00:27:16.790 --> 00:27:18.710 align:middle line:84% You can think of that as feature extraction. 00:27:18.710 --> 00:27:21.490 align:middle line:84% You got all these filters in your visual cortex 00:27:21.490 --> 00:27:23.770 align:middle line:84% that make all these useful measurements of orientation 00:27:23.770 --> 00:27:27.810 align:middle line:84% and spatial frequency and binocular disparity 00:27:27.810 --> 00:27:29.750 align:middle line:84% and wavelength and stuff like that. 00:27:29.750 --> 00:27:33.202 align:middle line:90% 00:27:33.202 --> 00:27:35.890 align:middle line:84% We then talked about all of these processes 00:27:35.890 --> 00:27:39.810 align:middle line:84% that we kind of loosely think of as mid-level vision, 00:27:39.810 --> 00:27:44.490 align:middle line:84% many of which are different forms of shape estimation 00:27:44.490 --> 00:27:49.210 align:middle line:84% or things that might be relevant to recognizing what's there-- 00:27:49.210 --> 00:27:52.990 align:middle line:84% color estimation, shape from contour, from shading, 00:27:52.990 --> 00:27:55.170 align:middle line:90% from texture. 00:27:55.170 --> 00:27:57.790 align:middle line:84% So these all kind of, from a certain perspective, 00:27:57.790 --> 00:28:00.770 align:middle line:84% give you information about three-dimensional structure 00:28:00.770 --> 00:28:04.290 align:middle line:84% and other properties of the world. 00:28:04.290 --> 00:28:06.530 align:middle line:84% And we often think that all of that stuff 00:28:06.530 --> 00:28:09.390 align:middle line:84% is in the service of recognition, 00:28:09.390 --> 00:28:12.890 align:middle line:84% telling us what's out there in the world. 00:28:12.890 --> 00:28:16.150 align:middle line:84% And that often is thought to be the realm of high-level vision. 00:28:16.150 --> 00:28:18.850 align:middle line:84% And again, these are all very loose terms. 00:28:18.850 --> 00:28:20.570 align:middle line:84% And in reality, there's kind of lots 00:28:20.570 --> 00:28:22.710 align:middle line:84% of interactions and feedback and things like that. 00:28:22.710 --> 00:28:27.510 align:middle line:84% But it's still often kind of a useful division, 00:28:27.510 --> 00:28:29.810 align:middle line:84% at least in the kinds of things that people study. 00:28:29.810 --> 00:28:32.810 align:middle line:84% All right, so what happens when we recognize things? 00:28:32.810 --> 00:28:35.770 align:middle line:90% So here's an image. 00:28:35.770 --> 00:28:41.210 align:middle line:84% You can look at this image and name lots of things 00:28:41.210 --> 00:28:47.110 align:middle line:84% that you see, some of the things that are circled here. 00:28:47.110 --> 00:28:50.170 align:middle line:90% 00:28:50.170 --> 00:28:53.610 align:middle line:84% This process is usually pretty effortless for humans. 00:28:53.610 --> 00:28:56.030 align:middle line:84% And you really just look around the image 00:28:56.030 --> 00:28:58.310 align:middle line:90% and you can see all this stuff. 00:28:58.310 --> 00:29:01.290 align:middle line:84% So the fact that it's pretty effortless for us 00:29:01.290 --> 00:29:05.260 align:middle line:84% for a long time kind of belied the challenge of the problem. 00:29:05.260 --> 00:29:11.080 align:middle line:84% So this is a really famous memo that was put out here 00:29:11.080 --> 00:29:13.380 align:middle line:90% at MIT in 1966. 00:29:13.380 --> 00:29:16.400 align:middle line:84% How many of you have seen this before? 00:29:16.400 --> 00:29:17.380 align:middle line:90% OK, a couple people. 00:29:17.380 --> 00:29:17.920 align:middle line:90% Yeah. 00:29:17.920 --> 00:29:20.560 align:middle line:84% So this often gets trotted out as an example 00:29:20.560 --> 00:29:27.920 align:middle line:84% of how our perceptual competencies are 00:29:27.920 --> 00:29:31.760 align:middle line:84% deceptively-- they can seem a lot simpler than they actually 00:29:31.760 --> 00:29:32.260 align:middle line:90% are. 00:29:32.260 --> 00:29:37.420 align:middle line:84% So this is a memo that was issued 00:29:37.420 --> 00:29:38.920 align:middle line:84% by the Artificial Intelligence Group 00:29:38.920 --> 00:29:41.680 align:middle line:90% here at MIT a long time ago. 00:29:41.680 --> 00:29:46.040 align:middle line:84% And the proposal was to solve the problem of vision one 00:29:46.040 --> 00:29:47.740 align:middle line:90% summer in 1966. 00:29:47.740 --> 00:29:49.540 align:middle line:84% So it's called the Summer Vision Project. 00:29:49.540 --> 00:29:51.100 align:middle line:90% This was assigned to a student. 00:29:51.100 --> 00:29:55.240 align:middle line:84% So it says the final goal is object identification, which 00:29:55.240 --> 00:29:57.320 align:middle line:84% will actually name objects by matching them 00:29:57.320 --> 00:30:00.720 align:middle line:84% with a vocabulary of known objects. 00:30:00.720 --> 00:30:02.840 align:middle line:90% All right. 00:30:02.840 --> 00:30:04.700 align:middle line:84% And so, of course, this was a long time ago. 00:30:04.700 --> 00:30:06.800 align:middle line:84% And it's only been very, very recently that we've 00:30:06.800 --> 00:30:09.900 align:middle line:84% had machine vision systems that can do anything close to this. 00:30:09.900 --> 00:30:12.880 align:middle line:90% So it's very easy for humans. 00:30:12.880 --> 00:30:15.480 align:middle line:84% And people didn't initially understand 00:30:15.480 --> 00:30:17.960 align:middle line:90% the complexity of the problem. 00:30:17.960 --> 00:30:22.120 align:middle line:84% So humans and other primates, when 00:30:22.120 --> 00:30:24.480 align:middle line:84% we solve object recognition, we're 00:30:24.480 --> 00:30:26.652 align:middle line:84% mostly using the central 10 degrees of vision, 00:30:26.652 --> 00:30:28.360 align:middle line:84% because we just kind look around at stuff 00:30:28.360 --> 00:30:31.800 align:middle line:90% that we want to recognize. 00:30:31.800 --> 00:30:35.480 align:middle line:84% So we make eye movements around scenes. 00:30:35.480 --> 00:30:38.800 align:middle line:84% Can make three to four saccades a second. 00:30:38.800 --> 00:30:41.480 align:middle line:84% So you can think of object recognition 00:30:41.480 --> 00:30:43.800 align:middle line:84% as consisting of these brief snapshots, maybe 00:30:43.800 --> 00:30:44.940 align:middle line:90% 200 milliseconds. 00:30:44.940 --> 00:30:47.680 align:middle line:84% You look from one thing to the next. 00:30:47.680 --> 00:30:50.520 align:middle line:84% And you're mostly able to recognize what's there 00:30:50.520 --> 00:30:53.480 align:middle line:90% from these brief snapshots. 00:30:53.480 --> 00:30:53.980 align:middle line:90% All right. 00:30:53.980 --> 00:30:55.480 align:middle line:90% So why is the problem hard? 00:30:55.480 --> 00:30:59.673 align:middle line:84% So one reason why it's a difficult problem 00:30:59.673 --> 00:31:01.840 align:middle line:84% is that there are lots of different kinds of objects 00:31:01.840 --> 00:31:02.980 align:middle line:90% that you can recognize. 00:31:02.980 --> 00:31:07.628 align:middle line:84% So this is just a small subset of possible things, 00:31:07.628 --> 00:31:09.920 align:middle line:84% different types of animals, different types of objects. 00:31:09.920 --> 00:31:15.360 align:middle line:84% There's lots and lots of things that you can ascribe names to. 00:31:15.360 --> 00:31:21.000 align:middle line:84% Another reason why it's hard is that the same physical source 00:31:21.000 --> 00:31:23.160 align:middle line:84% in the world-- so the same kind of thing-- 00:31:23.160 --> 00:31:26.240 align:middle line:84% can produce many different types of images. 00:31:26.240 --> 00:31:29.440 align:middle line:84% So you can vary the position, the size, the pose, 00:31:29.440 --> 00:31:31.860 align:middle line:84% the illumination that's directed towards an object, 00:31:31.860 --> 00:31:36.120 align:middle line:84% like the background that it's on. 00:31:36.120 --> 00:31:38.520 align:middle line:84% And the ability to tolerate this kind of variation 00:31:38.520 --> 00:31:40.847 align:middle line:84% is normally referred to as invariance. 00:31:40.847 --> 00:31:43.180 align:middle line:84% And we talked about this at the very start of the class. 00:31:43.180 --> 00:31:45.160 align:middle line:84% One of the things that's hard about perception 00:31:45.160 --> 00:31:47.400 align:middle line:84% is that you have to be able to recognize 00:31:47.400 --> 00:31:51.240 align:middle line:84% things invariant to all these sources of variation. 00:31:51.240 --> 00:31:54.000 align:middle line:84% So the images that are produced by an object 00:31:54.000 --> 00:31:55.337 align:middle line:90% are highly variable. 00:31:55.337 --> 00:31:57.920 align:middle line:84% This can be due to occlusions, illumination changes, viewpoint 00:31:57.920 --> 00:32:00.880 align:middle line:84% differences, non-rigid deformations. 00:32:00.880 --> 00:32:06.600 align:middle line:84% So here you're able to tell that one of these faces on the right 00:32:06.600 --> 00:32:09.260 align:middle line:84% is the same as the one on the left, and that one is different. 00:32:09.260 --> 00:32:13.440 align:middle line:90% 00:32:13.440 --> 00:32:15.600 align:middle line:84% So as a consequence of this, the same object 00:32:15.600 --> 00:32:17.680 align:middle line:84% can produce wildly different images depending 00:32:17.680 --> 00:32:18.940 align:middle line:90% on the viewing conditions. 00:32:18.940 --> 00:32:21.580 align:middle line:84% So these are all the same pair of shoes. 00:32:21.580 --> 00:32:25.200 align:middle line:84% But the pixel array that you get in each of these cases 00:32:25.200 --> 00:32:26.380 align:middle line:90% is very different. 00:32:26.380 --> 00:32:29.960 align:middle line:90% 00:32:29.960 --> 00:32:32.800 align:middle line:84% And also different exemplars of a given object category 00:32:32.800 --> 00:32:34.840 align:middle line:84% can also generate very different images. 00:32:34.840 --> 00:32:39.280 align:middle line:84% But you can tell that all of these things are shoes. 00:32:39.280 --> 00:32:43.340 align:middle line:84% So what parts of the brain are involved in visual recognition? 00:32:43.340 --> 00:32:46.360 align:middle line:84% Everybody remembers how we think of the visual system very 00:32:46.360 --> 00:32:51.200 align:middle line:84% coarsely as being organized into dorsal and ventral pathways. 00:32:51.200 --> 00:32:53.120 align:middle line:84% And we think of the ventral pathway 00:32:53.120 --> 00:32:57.030 align:middle line:84% as mediating our recognition abilities. 00:32:57.030 --> 00:33:00.230 align:middle line:84% And there's now lots and lots of evidence in support of this. 00:33:00.230 --> 00:33:04.370 align:middle line:84% So the initial clues to this came from lesion studies. 00:33:04.370 --> 00:33:06.850 align:middle line:84% So these are individuals who have brain damage. 00:33:06.850 --> 00:33:11.390 align:middle line:84% So if you have damage to the temporal lobe, 00:33:11.390 --> 00:33:12.910 align:middle line:84% either in humans or monkeys, that 00:33:12.910 --> 00:33:16.150 align:middle line:84% leads to something called visual agnosia, which is the inability 00:33:16.150 --> 00:33:18.910 align:middle line:90% to recognize objects. 00:33:18.910 --> 00:33:22.590 align:middle line:84% And in monkeys in particular, there's 00:33:22.590 --> 00:33:24.390 align:middle line:84% some pretty powerful dissociations 00:33:24.390 --> 00:33:27.630 align:middle line:84% between the consequences of lesions to the temporal lobe 00:33:27.630 --> 00:33:29.670 align:middle line:84% and lesions to the parietal lobe. 00:33:29.670 --> 00:33:33.830 align:middle line:84% So these are two example kinds of tasks that were 00:33:33.830 --> 00:33:35.590 align:middle line:90% used in the service of this. 00:33:35.590 --> 00:33:39.190 align:middle line:84% So temporal lobe lesions leads to deficits 00:33:39.190 --> 00:33:41.690 align:middle line:90% on shape discrimination tasks. 00:33:41.690 --> 00:33:44.710 align:middle line:84% So if you show a monkey a particular shape before 00:33:44.710 --> 00:33:48.910 align:middle line:84% the trial and then ask it to pick which one is similar to it 00:33:48.910 --> 00:33:52.110 align:middle line:84% with temporal lobe lesions, that task gets really difficult. 00:33:52.110 --> 00:33:56.070 align:middle line:84% By comparison, if the task here is to choose the object 00:33:56.070 --> 00:33:59.910 align:middle line:84% that's close to the landmark-- in this case, the cylinder-- 00:33:59.910 --> 00:34:07.270 align:middle line:84% parietal lesions cause this task to become difficult. 00:34:07.270 --> 00:34:09.750 align:middle line:84% So this is an example of the responses 00:34:09.750 --> 00:34:12.909 align:middle line:84% of a patient with temporal lobe damage, studied by Martha 00:34:12.909 --> 00:34:14.170 align:middle line:90% Farah, a neuropsychologist. 00:34:14.170 --> 00:34:17.909 align:middle line:84% So the task here is to look at the object on the left 00:34:17.909 --> 00:34:21.469 align:middle line:84% and then choose the object in the row that's 00:34:21.469 --> 00:34:23.989 align:middle line:84% to the right of it that has the same shape. 00:34:23.989 --> 00:34:26.830 align:middle line:84% And you can see that the person is not 00:34:26.830 --> 00:34:29.389 align:middle line:84% able to choose the correct shape. 00:34:29.389 --> 00:34:30.190 align:middle line:90% Yeah. 00:34:30.190 --> 00:34:32.469 align:middle line:84% AUDIENCE: Does the person ever adapt 00:34:32.469 --> 00:34:37.429 align:middle line:84% or is it they permanently can never tell an object? 00:34:37.429 --> 00:34:39.310 align:middle line:84% JOSH MCDERMOTT: Yeah, I mean, there's 00:34:39.310 --> 00:34:43.710 align:middle line:84% usually some degree of recovery that I 00:34:43.710 --> 00:34:46.389 align:middle line:90% think can vary in the extent. 00:34:46.389 --> 00:34:50.090 align:middle line:84% So the brain is remarkably plastic, right? 00:34:50.090 --> 00:34:55.110 align:middle line:84% And so typically, most of the cases 00:34:55.110 --> 00:34:57.750 align:middle line:84% where a human has brain damage of this sort 00:34:57.750 --> 00:34:59.822 align:middle line:90% are the result of stroke. 00:34:59.822 --> 00:35:02.030 align:middle line:84% There's some kinds of other things that can cause it, 00:35:02.030 --> 00:35:02.988 align:middle line:90% but it's mostly stroke. 00:35:02.988 --> 00:35:05.750 align:middle line:84% So the blood flow to a particular part of the brain 00:35:05.750 --> 00:35:06.250 align:middle line:90% is cut off. 00:35:06.250 --> 00:35:08.438 align:middle line:84% It's deprived of oxygen. Neurons die 00:35:08.438 --> 00:35:10.230 align:middle line:84% if they're deprived of oxygen. And so there 00:35:10.230 --> 00:35:12.510 align:middle line:84% will be some part of the brain that can be damaged. 00:35:12.510 --> 00:35:14.410 align:middle line:84% Can be pretty small, can be pretty big. 00:35:14.410 --> 00:35:16.790 align:middle line:90% It varies. 00:35:16.790 --> 00:35:19.990 align:middle line:84% And so typically, my understanding 00:35:19.990 --> 00:35:23.030 align:middle line:84% is that you can have very profound deficits, 00:35:23.030 --> 00:35:25.910 align:middle line:84% like in the immediate aftermath of that. 00:35:25.910 --> 00:35:28.390 align:middle line:84% And then things often get better. 00:35:28.390 --> 00:35:30.370 align:middle line:84% So there's some degree of partial recovery. 00:35:30.370 --> 00:35:32.990 align:middle line:84% So in some cases, we think that's because other parts 00:35:32.990 --> 00:35:36.510 align:middle line:84% of the brain can end up mediating some of the functions 00:35:36.510 --> 00:35:40.150 align:middle line:84% that the damaged part would carry out. 00:35:40.150 --> 00:35:41.817 align:middle line:90% It's often not complete. 00:35:41.817 --> 00:35:44.150 align:middle line:84% I think it depends a little bit on which area is damaged 00:35:44.150 --> 00:35:46.130 align:middle line:84% and how big it is and things like that. 00:35:46.130 --> 00:35:49.570 align:middle line:84% So I think this patient actually had 00:35:49.570 --> 00:35:51.485 align:middle line:84% had the brain damage for quite a while, 00:35:51.485 --> 00:35:53.610 align:middle line:84% and so I think these deficits are fairly permanent. 00:35:53.610 --> 00:35:55.490 align:middle line:84% But I don't remember, to be sure. 00:35:55.490 --> 00:35:58.950 align:middle line:90% 00:35:58.950 --> 00:36:00.830 align:middle line:84% So this is a really interesting case 00:36:00.830 --> 00:36:05.470 align:middle line:84% of a patient who is unable to identify 00:36:05.470 --> 00:36:08.830 align:middle line:84% line drawings of common objects and also couldn't copy them, 00:36:08.830 --> 00:36:10.630 align:middle line:90% but could draw from memory. 00:36:10.630 --> 00:36:15.790 align:middle line:84% So this is a situation where they were shown these two line 00:36:15.790 --> 00:36:18.990 align:middle line:84% drawings, asked to name them, and they couldn't, asked to copy 00:36:18.990 --> 00:36:21.135 align:middle line:90% them, and basically couldn't. 00:36:21.135 --> 00:36:22.510 align:middle line:84% Although it's kind of interesting 00:36:22.510 --> 00:36:24.990 align:middle line:84% that you can see what looks like maybe 00:36:24.990 --> 00:36:27.710 align:middle line:84% they see some of the texture, like on the page, 00:36:27.710 --> 00:36:31.390 align:middle line:84% and are trying to replicate that. 00:36:31.390 --> 00:36:35.670 align:middle line:84% But when told or when asked to please draw an apple, 00:36:35.670 --> 00:36:39.230 align:middle line:84% like from memory, they're able to do something reasonable, 00:36:39.230 --> 00:36:42.110 align:middle line:84% or please draw a book, they're able to do something reasonable. 00:36:42.110 --> 00:36:46.790 align:middle line:84% So the inference from this is that what is damaged here 00:36:46.790 --> 00:36:50.340 align:middle line:90% is the recognition machinery. 00:36:50.340 --> 00:36:56.060 align:middle line:84% So they can't look at the image and generate a representation 00:36:56.060 --> 00:36:58.780 align:middle line:90% of shape from the visual input. 00:36:58.780 --> 00:37:01.460 align:middle line:84% But it appears that they have some form 00:37:01.460 --> 00:37:06.960 align:middle line:84% of intact memory for the shape and are able to draw from it. 00:37:06.960 --> 00:37:09.700 align:middle line:90% 00:37:09.700 --> 00:37:11.320 align:middle line:90% So pretty interesting. 00:37:11.320 --> 00:37:15.620 align:middle line:90% 00:37:15.620 --> 00:37:19.260 align:middle line:84% So we think, as a consequence of the evidence 00:37:19.260 --> 00:37:21.900 align:middle line:84% that I just showed you and lots of other kind of similar pieces 00:37:21.900 --> 00:37:23.780 align:middle line:84% of evidence, that our recognition 00:37:23.780 --> 00:37:26.580 align:middle line:84% abilities are mediated by the visual ventral stream shown 00:37:26.580 --> 00:37:27.080 align:middle line:90% here. 00:37:27.080 --> 00:37:27.960 align:middle line:90% Yeah. 00:37:27.960 --> 00:37:33.420 align:middle line:84% AUDIENCE: On the previous slide, if you give them color and ask 00:37:33.420 --> 00:37:39.060 align:middle line:84% them to copy a filled-in color drawing of an apple, 00:37:39.060 --> 00:37:42.340 align:middle line:84% how do they perform if it's [INAUDIBLE]? 00:37:42.340 --> 00:37:45.100 align:middle line:84% JOSH MCDERMOTT: So I'm pretty sure that somebody with agnosia 00:37:45.100 --> 00:37:47.080 align:middle line:84% would have pretty normal color vision, 00:37:47.080 --> 00:37:49.060 align:middle line:90% but I don't know for a fact. 00:37:49.060 --> 00:37:50.780 align:middle line:90% Yeah. 00:37:50.780 --> 00:37:52.720 align:middle line:84% You certainly see the opposite dissociation. 00:37:52.720 --> 00:37:55.620 align:middle line:84% Like, we saw examples of people with achromatopsia, 00:37:55.620 --> 00:37:59.420 align:middle line:84% where you can suffer from brain damage that 00:37:59.420 --> 00:38:03.540 align:middle line:84% will impair color vision, but leave object recognition 00:38:03.540 --> 00:38:06.087 align:middle line:84% and most other aspects of vision pretty normal. 00:38:06.087 --> 00:38:07.920 align:middle line:84% So I'm pretty sure the opposite can be true. 00:38:07.920 --> 00:38:08.300 align:middle line:90% Yeah. 00:38:08.300 --> 00:38:10.342 align:middle line:84% AUDIENCE: Does it impair color vision in the eyes 00:38:10.342 --> 00:38:12.260 align:middle line:84% or does it impair your understanding of color? 00:38:12.260 --> 00:38:15.500 align:middle line:84% JOSH MCDERMOTT: Oh, achromatopsia that we discussed 00:38:15.500 --> 00:38:16.900 align:middle line:90% results from cortical damage. 00:38:16.900 --> 00:38:22.020 align:middle line:84% So again, somebody has a stroke, so the eyes are fine. 00:38:22.020 --> 00:38:24.540 align:middle line:84% And the cones are fine and normal. 00:38:24.540 --> 00:38:28.940 align:middle line:84% It's just that the cortical machinery that 00:38:28.940 --> 00:38:31.940 align:middle line:84% is mediating the conscious perception of color 00:38:31.940 --> 00:38:33.700 align:middle line:84% and presumably color constancy and things 00:38:33.700 --> 00:38:35.320 align:middle line:90% like that, that's been damaged. 00:38:35.320 --> 00:38:36.460 align:middle line:90% Yeah. 00:38:36.460 --> 00:38:37.140 align:middle line:90% Yeah. 00:38:37.140 --> 00:38:38.932 align:middle line:84% AUDIENCE: I know not everyone is an artist, 00:38:38.932 --> 00:38:41.520 align:middle line:84% but the drawings from memory aren't exactly the best. 00:38:41.520 --> 00:38:44.012 align:middle line:84% Is that related to the [INAUDIBLE]? 00:38:44.012 --> 00:38:45.220 align:middle line:90% JOSH MCDERMOTT: I don't know. 00:38:45.220 --> 00:38:48.540 align:middle line:90% Yeah, I'm not sure. 00:38:48.540 --> 00:38:51.020 align:middle line:90% So Martha Farah has a nice book. 00:38:51.020 --> 00:38:53.280 align:middle line:84% I think it might be called Visual Object Recognition. 00:38:53.280 --> 00:38:56.520 align:middle line:84% It's this short little book that you can find in the library, 00:38:56.520 --> 00:39:00.100 align:middle line:84% if you want to read more about these patients. 00:39:00.100 --> 00:39:00.660 align:middle line:90% Yeah. 00:39:00.660 --> 00:39:04.980 align:middle line:84% AUDIENCE: [INAUDIBLE] copies being that they cannot recognize 00:39:04.980 --> 00:39:07.380 align:middle line:84% what the model is supposed to be. 00:39:07.380 --> 00:39:13.060 align:middle line:84% Is that the inference from this experiment? 00:39:13.060 --> 00:39:14.860 align:middle line:84% JOSH MCDERMOTT: I think the inference 00:39:14.860 --> 00:39:18.560 align:middle line:84% is that they're unable to take visual input 00:39:18.560 --> 00:39:22.180 align:middle line:84% and generate a representation of object shape. 00:39:22.180 --> 00:39:28.320 align:middle line:84% And so that renders them unable to then name the object, 00:39:28.320 --> 00:39:32.260 align:middle line:84% because the naming is dependent on having a representation 00:39:32.260 --> 00:39:33.580 align:middle line:90% of object shape. 00:39:33.580 --> 00:39:38.020 align:middle line:84% And it also means that they have a hard time copying the object, 00:39:38.020 --> 00:39:43.020 align:middle line:84% presumably because they can't relate what they are 00:39:43.020 --> 00:39:45.120 align:middle line:90% drawing to what's on the page. 00:39:45.120 --> 00:39:50.340 align:middle line:84% They're not able to extract a shape representation from that. 00:39:50.340 --> 00:39:52.640 align:middle line:84% AUDIENCE: If they were shown the model 00:39:52.640 --> 00:39:55.240 align:middle line:84% and they also told that this is an apple, for example, 00:39:55.240 --> 00:39:58.942 align:middle line:84% would they draw something closer to a copy? 00:39:58.942 --> 00:40:00.400 align:middle line:84% JOSH MCDERMOTT: Yeah, I don't know. 00:40:00.400 --> 00:40:01.760 align:middle line:84% I mean, they can draw from memory, right? 00:40:01.760 --> 00:40:03.760 align:middle line:84% So presumably if you told them to draw an apple, 00:40:03.760 --> 00:40:05.400 align:middle line:84% they would do something like that. 00:40:05.400 --> 00:40:08.140 align:middle line:90% Yeah. 00:40:08.140 --> 00:40:08.640 align:middle line:90% All right. 00:40:08.640 --> 00:40:10.420 align:middle line:90% So we got the ventral stream. 00:40:10.420 --> 00:40:12.500 align:middle line:84% So we think of the ventral stream 00:40:12.500 --> 00:40:16.740 align:middle line:84% as culminating in inferotemporal cortex or IT. 00:40:16.740 --> 00:40:19.040 align:middle line:84% And then there's a few other stages. 00:40:19.040 --> 00:40:21.360 align:middle line:84% So we've got a pixel image as input. 00:40:21.360 --> 00:40:22.720 align:middle line:90% There's Retinal Ganglion Cells. 00:40:22.720 --> 00:40:23.860 align:middle line:90% That's RGS. 00:40:23.860 --> 00:40:26.900 align:middle line:84% The LGN, that's the visual part of the thalamus. 00:40:26.900 --> 00:40:29.140 align:middle line:90% V1, primary visual cortex. 00:40:29.140 --> 00:40:32.460 align:middle line:84% V2, V4, and IT, kind of major components 00:40:32.460 --> 00:40:34.060 align:middle line:90% of the ventral stream. 00:40:34.060 --> 00:40:37.020 align:middle line:84% And one way to think about this is 00:40:37.020 --> 00:40:42.610 align:middle line:84% that each of these visual stages is performing 00:40:42.610 --> 00:40:47.050 align:middle line:90% a transformation on the input. 00:40:47.050 --> 00:40:50.730 align:middle line:84% So remember, we previously introduced the idea 00:40:50.730 --> 00:40:54.090 align:middle line:84% that you could use convolution as a way 00:40:54.090 --> 00:40:56.810 align:middle line:84% to think about what a particular type of receptive field 00:40:56.810 --> 00:41:01.050 align:middle line:84% would do if it was applied, say, at every location in an image. 00:41:01.050 --> 00:41:03.090 align:middle line:84% That's what a big set of retinal ganglion cells 00:41:03.090 --> 00:41:04.812 align:middle line:84% or a big set of simple cells would do. 00:41:04.812 --> 00:41:06.770 align:middle line:84% So you can kind of think of that transformation 00:41:06.770 --> 00:41:08.650 align:middle line:84% as kind of like a convolution operation 00:41:08.650 --> 00:41:13.090 align:middle line:84% with a particular set of filters. 00:41:13.090 --> 00:41:15.330 align:middle line:84% So each of these regions is doing something 00:41:15.330 --> 00:41:17.490 align:middle line:90% to the representation. 00:41:17.490 --> 00:41:19.170 align:middle line:84% And then there's also potentially 00:41:19.170 --> 00:41:21.690 align:middle line:84% feedback and recurrent connections. 00:41:21.690 --> 00:41:24.930 align:middle line:84% And these numbers here are the response latencies. 00:41:24.930 --> 00:41:28.970 align:middle line:84% So it's about 60 milliseconds from when 00:41:28.970 --> 00:41:31.290 align:middle line:84% the image hits the retina to when you 00:41:31.290 --> 00:41:32.870 align:middle line:90% get the initial response of V1. 00:41:32.870 --> 00:41:38.290 align:middle line:84% And about 100 milliseconds to when you get a response in IT. 00:41:38.290 --> 00:41:43.290 align:middle line:84% So one kind of early important result on human object 00:41:43.290 --> 00:41:48.530 align:middle line:84% recognition indicated that recognition is pretty fast. 00:41:48.530 --> 00:41:51.330 align:middle line:84% So this was a study by Simon Thorpe and colleagues. 00:41:51.330 --> 00:41:54.010 align:middle line:84% And the experiment involved looking 00:41:54.010 --> 00:41:57.608 align:middle line:84% at a very rapidly-presented sequence of images. 00:41:57.608 --> 00:42:00.150 align:middle line:84% So each one of these would are presented one at a time, boom, 00:42:00.150 --> 00:42:01.530 align:middle line:90% boom, boom, boom, boom. 00:42:01.530 --> 00:42:03.530 align:middle line:84% They'd be up for 20 milliseconds. 00:42:03.530 --> 00:42:08.610 align:middle line:84% And the task is just to press a key if you see an animal. 00:42:08.610 --> 00:42:11.610 align:middle line:84% So you would press the key there, because that's a fish. 00:42:11.610 --> 00:42:15.050 align:middle line:90% Press this key there. 00:42:15.050 --> 00:42:17.090 align:middle line:84% I'm not sure what that is, but maybe you 00:42:17.090 --> 00:42:18.810 align:middle line:90% would press it there too. 00:42:18.810 --> 00:42:19.690 align:middle line:90% All right. 00:42:19.690 --> 00:42:22.030 align:middle line:90% So you press keys for animals. 00:42:22.030 --> 00:42:24.890 align:middle line:90% 00:42:24.890 --> 00:42:28.910 align:middle line:84% So this is a histogram of the reaction time. 00:42:28.910 --> 00:42:32.290 align:middle line:84% So it's like how long it takes people 00:42:32.290 --> 00:42:34.850 align:middle line:84% to press the button from the time in which they 00:42:34.850 --> 00:42:35.850 align:middle line:90% see the image. 00:42:35.850 --> 00:42:41.650 align:middle line:84% So the average reaction time is something like 400 milliseconds. 00:42:41.650 --> 00:42:43.550 align:middle line:84% And there's a speed-accuracy trade off. 00:42:43.550 --> 00:42:46.250 align:middle line:84% So if the reaction time is shorter, 00:42:46.250 --> 00:42:48.780 align:middle line:84% people tend to do a little bit less accurate. 00:42:48.780 --> 00:42:51.030 align:middle line:84% If they take a little bit longer to make the decision, 00:42:51.030 --> 00:42:53.490 align:middle line:90% they're more accurate. 00:42:53.490 --> 00:42:54.030 align:middle line:90% All right. 00:42:54.030 --> 00:42:57.550 align:middle line:84% But you can't really draw much of an inference from that 400 00:42:57.550 --> 00:43:02.530 align:middle line:84% millisecond number about the time it takes the visual system 00:43:02.530 --> 00:43:06.570 align:middle line:84% to recognize the object, because that reaction time involves 00:43:06.570 --> 00:43:08.390 align:middle line:84% a lot of things other than recognition. 00:43:08.390 --> 00:43:10.350 align:middle line:84% In particular, you have to press the button. 00:43:10.350 --> 00:43:16.210 align:middle line:84% So once your visual system generates some recognition, 00:43:16.210 --> 00:43:19.213 align:middle line:84% then somehow it's got to communicate to motor cortex 00:43:19.213 --> 00:43:20.630 align:middle line:84% that you need to press the button. 00:43:20.630 --> 00:43:22.628 align:middle line:84% Motor cortex has got to send the motor command, 00:43:22.628 --> 00:43:25.170 align:middle line:84% travels down your nerves, and eventually the muscles contract 00:43:25.170 --> 00:43:26.350 align:middle line:90% and you press the button. 00:43:26.350 --> 00:43:27.308 align:middle line:90% OK? 00:43:27.308 --> 00:43:28.850 align:middle line:84% So there's a lot of that that's going 00:43:28.850 --> 00:43:30.710 align:middle line:90% into this 400 milliseconds. 00:43:30.710 --> 00:43:32.410 align:middle line:84% And so what they did in this study 00:43:32.410 --> 00:43:39.050 align:middle line:84% was instead to just let the brain reveal 00:43:39.050 --> 00:43:45.130 align:middle line:84% when there was information about recognition made available. 00:43:45.130 --> 00:43:49.130 align:middle line:84% So this is a brain response called an ERP 00:43:49.130 --> 00:43:50.590 align:middle line:90% or an Event-Related Potential. 00:43:50.590 --> 00:43:52.632 align:middle line:84% And the way that you make this measurement is you 00:43:52.632 --> 00:43:54.450 align:middle line:84% put a bunch of electrodes on the scalp. 00:43:54.450 --> 00:43:58.370 align:middle line:84% So you're recording scalp potentials. 00:43:58.370 --> 00:44:00.750 align:middle line:90% And this is showing-- 00:44:00.750 --> 00:44:03.530 align:middle line:90% 00:44:03.530 --> 00:44:04.870 align:middle line:90% so there's a lot of electrodes. 00:44:04.870 --> 00:44:08.850 align:middle line:84% And this is one way to depict the voltage that's 00:44:08.850 --> 00:44:11.250 align:middle line:84% being measured at different points on the skull. 00:44:11.250 --> 00:44:15.537 align:middle line:84% You can see that it evolves over time in some particular way. 00:44:15.537 --> 00:44:16.870 align:middle line:90% This is not what's so important. 00:44:16.870 --> 00:44:22.057 align:middle line:84% So here, this is a particular part of that brain response. 00:44:22.057 --> 00:44:24.390 align:middle line:84% And what they've done here-- so event-related potential, 00:44:24.390 --> 00:44:26.250 align:middle line:84% so what that means is that you're recording 00:44:26.250 --> 00:44:28.110 align:middle line:90% these continuous signals. 00:44:28.110 --> 00:44:31.170 align:middle line:84% And then you average them in some particular way, 00:44:31.170 --> 00:44:33.790 align:middle line:84% typically with respect to a stimulus. 00:44:33.790 --> 00:44:35.720 align:middle line:84% So what they're doing in this case 00:44:35.720 --> 00:44:38.320 align:middle line:84% is they're taking all of the times 00:44:38.320 --> 00:44:41.650 align:middle line:84% when an animal was presented on the screen 00:44:41.650 --> 00:44:43.400 align:middle line:84% and then averaging the brain response that 00:44:43.400 --> 00:44:44.020 align:middle line:90% occurred after that. 00:44:44.020 --> 00:44:45.978 align:middle line:84% So it's kind of like a spike-triggered average, 00:44:45.978 --> 00:44:50.760 align:middle line:84% except it's a stimulus-triggered average, where the stimulus-- 00:44:50.760 --> 00:44:53.160 align:middle line:84% you're averaging it separately, depending 00:44:53.160 --> 00:44:55.720 align:middle line:84% on whether it's an animal or whether it's a non-animal. 00:44:55.720 --> 00:44:57.960 align:middle line:84% So you get one average brain response 00:44:57.960 --> 00:45:00.880 align:middle line:84% for all the animal images and one average brain response 00:45:00.880 --> 00:45:02.760 align:middle line:90% for all the non-animal images. 00:45:02.760 --> 00:45:04.820 align:middle line:90% And so this is what you get. 00:45:04.820 --> 00:45:06.320 align:middle line:90% So this is time. 00:45:06.320 --> 00:45:08.065 align:middle line:90% This is the response. 00:45:08.065 --> 00:45:09.440 align:middle line:84% Whether it's positive or negative 00:45:09.440 --> 00:45:10.840 align:middle line:84% doesn't really mean anything because you just 00:45:10.840 --> 00:45:12.040 align:middle line:84% have these electrodes and you're measuring 00:45:12.040 --> 00:45:13.180 align:middle line:90% differences in voltage. 00:45:13.180 --> 00:45:16.800 align:middle line:84% And what you really care about is at what point in time 00:45:16.800 --> 00:45:19.720 align:middle line:84% there will be some difference in the brain 00:45:19.720 --> 00:45:23.200 align:middle line:84% depending on whether the image is an animal or a non-animal. 00:45:23.200 --> 00:45:24.960 align:middle line:90% And what does this show? 00:45:24.960 --> 00:45:27.600 align:middle line:84% Well, you get this response here. 00:45:27.600 --> 00:45:31.960 align:middle line:84% And they're very, very similar up until looks 00:45:31.960 --> 00:45:33.760 align:middle line:90% like about 150 milliseconds. 00:45:33.760 --> 00:45:37.960 align:middle line:84% At that point, the animal and the non-animal curves diverge. 00:45:37.960 --> 00:45:40.600 align:middle line:84% And so this curve up above them is 00:45:40.600 --> 00:45:42.320 align:middle line:90% the difference between the two. 00:45:42.320 --> 00:45:44.120 align:middle line:84% So you can see that that is close to zero 00:45:44.120 --> 00:45:46.035 align:middle line:90% until about 150 milliseconds. 00:45:46.035 --> 00:45:48.160 align:middle line:84% And then you see a difference between the responses 00:45:48.160 --> 00:45:49.740 align:middle line:90% to animals and non-animals. 00:45:49.740 --> 00:45:52.710 align:middle line:90% 00:45:52.710 --> 00:45:56.800 align:middle line:84% So at 150 milliseconds, the brain 00:45:56.800 --> 00:45:59.640 align:middle line:84% is able to discriminate whether you're looking 00:45:59.640 --> 00:46:00.985 align:middle line:90% at an animal or a non-animal. 00:46:00.985 --> 00:46:02.360 align:middle line:84% And the idea is that because this 00:46:02.360 --> 00:46:05.880 align:middle line:84% is being averaged across a gazillion different images, that 00:46:05.880 --> 00:46:08.760 align:middle line:84% probably is indicative of some fairly 00:46:08.760 --> 00:46:10.900 align:middle line:90% abstract property of the image. 00:46:10.900 --> 00:46:13.840 align:middle line:90% 00:46:13.840 --> 00:46:14.340 align:middle line:90% All right. 00:46:14.340 --> 00:46:18.420 align:middle line:84% So what's significant about that number of 150 milliseconds? 00:46:18.420 --> 00:46:21.000 align:middle line:84% Well, the latency of the V1 response 00:46:21.000 --> 00:46:23.800 align:middle line:84% is about 70 milliseconds in humans. 00:46:23.800 --> 00:46:27.600 align:middle line:84% And so this result implies that there's 00:46:27.600 --> 00:46:32.600 align:middle line:84% pretty fast recognition of objects, potentially mostly 00:46:32.600 --> 00:46:33.960 align:middle line:90% feedforward. 00:46:33.960 --> 00:46:37.100 align:middle line:84% So the idea is that you're able to recognize objects, 00:46:37.100 --> 00:46:39.360 align:middle line:84% at least at the level of animals versus non-animals, 00:46:39.360 --> 00:46:45.240 align:middle line:84% with one pass through this processing pipeline. 00:46:45.240 --> 00:46:47.900 align:middle line:84% And so these numbers here are for the macaque. 00:46:47.900 --> 00:46:50.000 align:middle line:84% They would be longer for the humans, 00:46:50.000 --> 00:46:51.760 align:middle line:84% because humans have a bigger brain. 00:46:51.760 --> 00:46:55.560 align:middle line:84% So 150 milliseconds is probably pretty 00:46:55.560 --> 00:46:58.120 align:middle line:84% close to the time at which things 00:46:58.120 --> 00:47:02.200 align:middle line:90% get to IT in the first pass. 00:47:02.200 --> 00:47:02.700 align:middle line:90% All right. 00:47:02.700 --> 00:47:08.320 align:middle line:84% And so this suggests that at least there's 00:47:08.320 --> 00:47:10.520 align:middle line:84% some component of recognition that we 00:47:10.520 --> 00:47:13.140 align:middle line:84% can think of as a pretty feedforward process. 00:47:13.140 --> 00:47:16.440 align:middle line:84% It's not to say that all of recognition is always like that, 00:47:16.440 --> 00:47:19.560 align:middle line:84% but there's some component of it that is potentially mostly 00:47:19.560 --> 00:47:22.840 align:middle line:90% feedforward and pretty fast. 00:47:22.840 --> 00:47:24.140 align:middle line:90% Any questions about that? 00:47:24.140 --> 00:47:36.880 align:middle line:90% 00:47:36.880 --> 00:47:41.160 align:middle line:84% OK, so here's the ventral pathway plotted 00:47:41.160 --> 00:47:42.480 align:middle line:90% in a slightly different way. 00:47:42.480 --> 00:47:44.300 align:middle line:84% So we've got these different areas. 00:47:44.300 --> 00:47:47.400 align:middle line:84% And these are the latencies in the macaque. 00:47:47.400 --> 00:47:51.020 align:middle line:84% We've already talked quite a lot about this stage here, V1. 00:47:51.020 --> 00:47:55.840 align:middle line:84% So remember, in V1, we've got simple cells and complex cells. 00:47:55.840 --> 00:47:58.660 align:middle line:84% So simple cells are orientation selective. 00:47:58.660 --> 00:48:00.360 align:middle line:84% Complex cells are orientation selective, 00:48:00.360 --> 00:48:02.300 align:middle line:84% but they're more tolerant to position. 00:48:02.300 --> 00:48:04.580 align:middle line:84% So they have some simple form of invariance. 00:48:04.580 --> 00:48:07.160 align:middle line:90% 00:48:07.160 --> 00:48:09.855 align:middle line:84% We've talked a little bit about V2. 00:48:09.855 --> 00:48:11.480 align:middle line:84% So one of the things that you see in V2 00:48:11.480 --> 00:48:16.640 align:middle line:84% is some more sophisticated forms of selectivity for edges, 00:48:16.640 --> 00:48:18.940 align:middle line:84% like that you get responses to illusory contours. 00:48:18.940 --> 00:48:21.920 align:middle line:84% There's some sensitivity to figure ground relations, border 00:48:21.920 --> 00:48:23.560 align:middle line:90% ownership cells you find there. 00:48:23.560 --> 00:48:25.380 align:middle line:90% We talked a little bit about V4. 00:48:25.380 --> 00:48:27.910 align:middle line:84% So we talked about how lesions of V4 00:48:27.910 --> 00:48:30.870 align:middle line:84% tend to often produce deficits in color perception. 00:48:30.870 --> 00:48:33.230 align:middle line:84% But V4 also kind of exhibits neurons 00:48:33.230 --> 00:48:37.070 align:middle line:84% that have some interesting selectivity for shape. 00:48:37.070 --> 00:48:41.990 align:middle line:84% So here's one fairly well-known study that presented neurons 00:48:41.990 --> 00:48:45.550 align:middle line:84% in V4 with these kind of funny-looking gratings, 00:48:45.550 --> 00:48:47.930 align:middle line:84% just different kinds of complicated patterns. 00:48:47.930 --> 00:48:50.990 align:middle line:84% And essentially, the finding is that a lot of the neurons 00:48:50.990 --> 00:48:54.330 align:middle line:84% are tuned to specific types of these patterns. 00:48:54.330 --> 00:49:00.470 align:middle line:84% So this one really liked these few patterns the most. 00:49:00.470 --> 00:49:02.778 align:middle line:84% This one really likes spiral shapes. 00:49:02.778 --> 00:49:05.070 align:middle line:84% So just kind of you get slightly more complicated shape 00:49:05.070 --> 00:49:05.650 align:middle line:90% selectivity. 00:49:05.650 --> 00:49:07.775 align:middle line:84% So if you did this in V1, you wouldn't see anything 00:49:07.775 --> 00:49:09.950 align:middle line:84% like this, because the responses there would just 00:49:09.950 --> 00:49:13.310 align:middle line:84% be determined by the approximate dot product between one 00:49:13.310 --> 00:49:15.730 align:middle line:84% of those simple cell receptive fields and these patterns. 00:49:15.730 --> 00:49:18.790 align:middle line:90% 00:49:18.790 --> 00:49:23.590 align:middle line:84% Here's another study from Ed Connor which 00:49:23.590 --> 00:49:25.410 align:middle line:90% created a basis for shapes. 00:49:25.410 --> 00:49:28.270 align:middle line:84% Essentially the shapes can be curved and then 00:49:28.270 --> 00:49:30.290 align:middle line:84% pointy in these different directions. 00:49:30.290 --> 00:49:33.230 align:middle line:84% So you get this big variation of these simple shapes. 00:49:33.230 --> 00:49:36.310 align:middle line:84% You can measure the response of V4 neurons 00:49:36.310 --> 00:49:39.990 align:middle line:84% and find that individual neurons can 00:49:39.990 --> 00:49:42.990 align:middle line:84% be-- the responses can be explained fairly well in terms 00:49:42.990 --> 00:49:45.310 align:middle line:84% of properties of these particular shapes. 00:49:45.310 --> 00:49:47.230 align:middle line:84% So this particular one is it really 00:49:47.230 --> 00:49:50.710 align:middle line:84% likes pointy things that are pointing up, looks like. 00:49:50.710 --> 00:49:55.190 align:middle line:90% 00:49:55.190 --> 00:49:56.950 align:middle line:84% So you move up the ventral stream, 00:49:56.950 --> 00:50:00.310 align:middle line:84% you see selectivity for more complicated kinds 00:50:00.310 --> 00:50:04.870 align:middle line:84% of things culminating in inferotemporal cortex. 00:50:04.870 --> 00:50:08.470 align:middle line:84% So a few things to know about inferotemporal cortex. 00:50:08.470 --> 00:50:12.590 align:middle line:84% One is that it's dominated by the central visual field. 00:50:12.590 --> 00:50:16.150 align:middle line:84% So these are the results of a tracer 00:50:16.150 --> 00:50:21.630 align:middle line:84% study from Leslie Ungerleider's lab, where they injected tracers 00:50:21.630 --> 00:50:26.670 align:middle line:84% in either the foveal or the peripheral part of V4, 00:50:26.670 --> 00:50:32.030 align:middle line:84% and then looked at where they landed. 00:50:32.030 --> 00:50:36.270 align:middle line:84% And so what you can see-- so this is different parts 00:50:36.270 --> 00:50:38.510 align:middle line:90% of inferotemporal cortex here. 00:50:38.510 --> 00:50:42.310 align:middle line:84% And the main take-home message is that there's a lot of red. 00:50:42.310 --> 00:50:45.950 align:middle line:84% So the injections to the foveal part of V4 00:50:45.950 --> 00:50:49.070 align:middle line:84% end up really labeling inferotemporal cortex pretty 00:50:49.070 --> 00:50:50.030 align:middle line:90% thoroughly. 00:50:50.030 --> 00:50:52.950 align:middle line:90% And there's not as much green. 00:50:52.950 --> 00:50:55.350 align:middle line:84% So that's consistent with this idea that 00:50:55.350 --> 00:50:57.310 align:middle line:84% object recognition is really predominantly 00:50:57.310 --> 00:50:59.910 align:middle line:84% a function of central vision, because you just 00:50:59.910 --> 00:51:02.150 align:middle line:84% look at the stuff that you want to recognize. 00:51:02.150 --> 00:51:05.070 align:middle line:84% And so IT is really dominated by inputs 00:51:05.070 --> 00:51:06.890 align:middle line:90% from the central part of vision. 00:51:06.890 --> 00:51:11.390 align:middle line:90% 00:51:11.390 --> 00:51:14.750 align:middle line:84% And so there are lots of these famous reports 00:51:14.750 --> 00:51:17.830 align:middle line:84% of pretty complicated forms of stimulus selectivity 00:51:17.830 --> 00:51:19.430 align:middle line:90% in this part of the brain. 00:51:19.430 --> 00:51:22.470 align:middle line:84% So Charlie Gross was a professor at Princeton 00:51:22.470 --> 00:51:24.910 align:middle line:84% who pioneered the study of inferotemporal cortex. 00:51:24.910 --> 00:51:27.870 align:middle line:84% And a couple of our faculty here in this department 00:51:27.870 --> 00:51:31.510 align:middle line:84% worked in his lab, Bob Desimone and Earl Miller. 00:51:31.510 --> 00:51:34.070 align:middle line:84% And these are different stimuli that 00:51:34.070 --> 00:51:36.310 align:middle line:84% were presented when they were recording 00:51:36.310 --> 00:51:39.310 align:middle line:90% from neurons in the IT. 00:51:39.310 --> 00:51:41.670 align:middle line:84% And it looks as though the neuron 00:51:41.670 --> 00:51:44.390 align:middle line:84% is really sensitive to the shape of a hand. 00:51:44.390 --> 00:51:48.510 align:middle line:84% And so this is an excerpt from one of their papers. 00:51:48.510 --> 00:51:50.550 align:middle line:84% They say the use of these stimuli 00:51:50.550 --> 00:51:52.910 align:middle line:84% was begun one day when, having failed 00:51:52.910 --> 00:51:55.490 align:middle line:84% to drive a unit with any light stimulus, 00:51:55.490 --> 00:51:57.550 align:middle line:84% we waved a hand at the stimulus screen 00:51:57.550 --> 00:52:00.110 align:middle line:84% and elicited a very vigorous response from the previously 00:52:00.110 --> 00:52:01.830 align:middle line:90% unresponsive neuron. 00:52:01.830 --> 00:52:05.550 align:middle line:84% We then spent the next 12 hours testing various paper cutouts 00:52:05.550 --> 00:52:08.550 align:middle line:84% in an attempt to find the trigger feature for this unit. 00:52:08.550 --> 00:52:10.182 align:middle line:84% When the entire set of stimuli used 00:52:10.182 --> 00:52:12.390 align:middle line:84% were ranked according to the strength of the response 00:52:12.390 --> 00:52:15.430 align:middle line:84% that they produced, we could not find a simple physical dimension 00:52:15.430 --> 00:52:17.430 align:middle line:84% that correlated with this rank order. 00:52:17.430 --> 00:52:19.860 align:middle line:84% However, the rank order of adequate stimuli 00:52:19.860 --> 00:52:22.540 align:middle line:84% did correlate with similarity for us 00:52:22.540 --> 00:52:25.540 align:middle line:90% to the shadow of a monkey hand. 00:52:25.540 --> 00:52:29.140 align:middle line:84% So selective for some complicated kind 00:52:29.140 --> 00:52:30.400 align:middle line:90% of meaningful structure. 00:52:30.400 --> 00:52:33.500 align:middle line:90% 00:52:33.500 --> 00:52:35.940 align:middle line:90% Here are some other examples. 00:52:35.940 --> 00:52:39.620 align:middle line:84% So this is another early study that kind of 00:52:39.620 --> 00:52:41.860 align:middle line:84% found another example of a neuron that's 00:52:41.860 --> 00:52:44.338 align:middle line:90% tuned to the shape of a hand. 00:52:44.338 --> 00:52:46.380 align:middle line:84% Here's an example where there's some neurons that 00:52:46.380 --> 00:52:48.740 align:middle line:90% seem to be selective for faces. 00:52:48.740 --> 00:52:50.140 align:middle line:84% So the idea is that they're tuned 00:52:50.140 --> 00:52:52.140 align:middle line:84% to these very specific combinations of features. 00:52:52.140 --> 00:52:54.875 align:middle line:90% They're highly selective. 00:52:54.875 --> 00:52:56.500 align:middle line:84% Now, one of the other things that we've 00:52:56.500 --> 00:52:59.420 align:middle line:84% talked a lot about in the context of the visual cortex 00:52:59.420 --> 00:53:02.665 align:middle line:84% is the idea that the cortex is organized into maps. 00:53:02.665 --> 00:53:04.540 align:middle line:84% So one of the main principles we talked about 00:53:04.540 --> 00:53:07.340 align:middle line:84% is retinotopy, where you have maps of the visual field 00:53:07.340 --> 00:53:09.900 align:middle line:84% kind of laid out in the visual cortex. 00:53:09.900 --> 00:53:12.500 align:middle line:84% And so you don't see very clear retinotopy 00:53:12.500 --> 00:53:14.027 align:middle line:84% in inferotemporal cortex, in part 00:53:14.027 --> 00:53:15.860 align:middle line:84% because the receptive fields are very large. 00:53:15.860 --> 00:53:18.940 align:middle line:84% But there's nonetheless some functional organization. 00:53:18.940 --> 00:53:20.900 align:middle line:84% So this is the result of a study that 00:53:20.900 --> 00:53:26.380 align:middle line:84% was done here in this department where they presented 00:53:26.380 --> 00:53:28.120 align:middle line:90% a very large set of images. 00:53:28.120 --> 00:53:29.760 align:middle line:90% So this is the object ID number. 00:53:29.760 --> 00:53:33.180 align:middle line:84% There's like 77 images of just kind of complicated shapes-- 00:53:33.180 --> 00:53:35.780 align:middle line:84% monkey faces, cars, elephants, just 00:53:35.780 --> 00:53:40.140 align:middle line:84% like a big heterogeneous set of object shapes. 00:53:40.140 --> 00:53:42.740 align:middle line:84% And there was an electrode that's kind of inserted 00:53:42.740 --> 00:53:43.680 align:middle line:90% into the brain. 00:53:43.680 --> 00:53:45.860 align:middle line:84% And they measure the responses to each one 00:53:45.860 --> 00:53:50.180 align:middle line:84% of these shapes at different points along that red line. 00:53:50.180 --> 00:53:54.420 align:middle line:84% And so each one of these rows is the response 00:53:54.420 --> 00:53:58.140 align:middle line:84% that's measured at a different point kind of in the brain. 00:53:58.140 --> 00:54:01.240 align:middle line:84% And presumably, these are multi-unit recordings. 00:54:01.240 --> 00:54:03.840 align:middle line:84% So they reflect the responses of a bunch of neurons, 00:54:03.840 --> 00:54:07.540 align:middle line:84% but in some spatial neighborhood. 00:54:07.540 --> 00:54:09.220 align:middle line:84% And so you just get some big vector 00:54:09.220 --> 00:54:12.900 align:middle line:84% for every one of these recording sites, a 77 dimensional vector. 00:54:12.900 --> 00:54:16.300 align:middle line:84% And then what is plotted on this graph here 00:54:16.300 --> 00:54:20.300 align:middle line:84% is the similarity of those vectors 00:54:20.300 --> 00:54:24.820 align:middle line:84% as a function of the separation between the recording sites. 00:54:24.820 --> 00:54:29.480 align:middle line:84% And the point is that this kind of drops. 00:54:29.480 --> 00:54:32.260 align:middle line:84% So when the spatial separation is small 00:54:32.260 --> 00:54:35.020 align:middle line:84% that you have fairly similar responses, even though they're 00:54:35.020 --> 00:54:37.580 align:middle line:84% complicated and hard to describe in words or in terms 00:54:37.580 --> 00:54:40.300 align:middle line:84% of physical stimulus dimensions, and then 00:54:40.300 --> 00:54:43.000 align:middle line:84% as you get to further points in the brain, 00:54:43.000 --> 00:54:45.660 align:middle line:84% the responses kind of become more distinct. 00:54:45.660 --> 00:54:50.020 align:middle line:84% So there's some degree of spatial organization. 00:54:50.020 --> 00:54:50.520 align:middle line:90% All right. 00:54:50.520 --> 00:54:52.940 align:middle line:84% So another general principle of organization 00:54:52.940 --> 00:54:56.380 align:middle line:84% in the ventral pathway is that receptive field sizes increase 00:54:56.380 --> 00:54:58.680 align:middle line:90% as you move up the pathway. 00:54:58.680 --> 00:55:01.180 align:middle line:84% So in particular by the time you get 00:55:01.180 --> 00:55:03.140 align:middle line:84% to anterior inferotemporal cortex, 00:55:03.140 --> 00:55:06.283 align:middle line:84% you have these very large receptive fields, 00:55:06.283 --> 00:55:08.700 align:middle line:84% even larger than what you get kind of a little bit earlier 00:55:08.700 --> 00:55:10.900 align:middle line:90% in inferotemporal cortex. 00:55:10.900 --> 00:55:13.060 align:middle line:90% So big receptive fields. 00:55:13.060 --> 00:55:17.340 align:middle line:84% Individual IT neurons typically show some degree of tolerance 00:55:17.340 --> 00:55:19.780 align:middle line:84% to variation in position and size. 00:55:19.780 --> 00:55:23.180 align:middle line:84% So this is an early study by Nikos Logothetis. 00:55:23.180 --> 00:55:27.380 align:middle line:84% This graph is plotting the extent of response 00:55:27.380 --> 00:55:30.840 align:middle line:84% to a preferred stimulus at a whole bunch of different sizes. 00:55:30.840 --> 00:55:33.940 align:middle line:84% So this is degrees of visual angle from small to big. 00:55:33.940 --> 00:55:36.220 align:middle line:84% And the response is pretty consistent. 00:55:36.220 --> 00:55:38.980 align:middle line:84% This is the stimulus presented at different locations 00:55:38.980 --> 00:55:40.420 align:middle line:90% on the screen. 00:55:40.420 --> 00:55:42.860 align:middle line:84% Maybe there's some spatial selectivity, 00:55:42.860 --> 00:55:46.740 align:middle line:90% but it's quite broad. 00:55:46.740 --> 00:55:47.360 align:middle line:90% OK. 00:55:47.360 --> 00:55:50.100 align:middle line:84% So these are all various kinds of clues 00:55:50.100 --> 00:55:54.860 align:middle line:84% about what is presumptively the locus of visual recognition. 00:55:54.860 --> 00:55:55.360 align:middle line:90% All right. 00:55:55.360 --> 00:55:57.100 align:middle line:84% What we're going to talk about now 00:55:57.100 --> 00:56:03.940 align:middle line:84% is how we can think about what these tuning properties kind 00:56:03.940 --> 00:56:09.000 align:middle line:84% of might be doing in a more rigorous sense. 00:56:09.000 --> 00:56:11.380 align:middle line:84% So I want to pose to you the question of what 00:56:11.380 --> 00:56:15.050 align:middle line:84% is the computational function of the ventral pathway, or really 00:56:15.050 --> 00:56:18.730 align:middle line:84% of any system that is attempting to do recognition? 00:56:18.730 --> 00:56:24.050 align:middle line:84% And in order to pose this question, 00:56:24.050 --> 00:56:28.830 align:middle line:84% we're going to think about the representation of images. 00:56:28.830 --> 00:56:31.180 align:middle line:84% And this generalized could be any kind of stimulus, 00:56:31.180 --> 00:56:33.430 align:middle line:84% but we're going to be talking about it here in vision. 00:56:33.430 --> 00:56:35.970 align:middle line:84% We're going to think about the representation of images 00:56:35.970 --> 00:56:39.490 align:middle line:90% in a population of neurons. 00:56:39.490 --> 00:56:42.250 align:middle line:84% So the idea is that you look at an image, 00:56:42.250 --> 00:56:45.250 align:middle line:84% and then some stage of your visual system 00:56:45.250 --> 00:56:48.810 align:middle line:84% generates a representation of that image via the responses 00:56:48.810 --> 00:56:50.050 align:middle line:90% of its neurons. 00:56:50.050 --> 00:56:51.770 align:middle line:84% So we can think of the representation 00:56:51.770 --> 00:56:56.130 align:middle line:84% of one image as a point in a very high-dimensional space 00:56:56.130 --> 00:56:59.650 align:middle line:84% where the axes of that space are the responses 00:56:59.650 --> 00:57:01.800 align:middle line:90% of different neurons. 00:57:01.800 --> 00:57:03.550 align:middle line:84% There might be thousands of these neurons, 00:57:03.550 --> 00:57:06.570 align:middle line:84% so it could be a thousand-dimensional space. 00:57:06.570 --> 00:57:08.610 align:middle line:84% So you see an image and there's a response 00:57:08.610 --> 00:57:10.230 align:middle line:90% generated in your brain. 00:57:10.230 --> 00:57:12.090 align:middle line:84% Now in actuality, it's more complicated 00:57:12.090 --> 00:57:14.430 align:middle line:84% than that, because maybe the response evolves over time. 00:57:14.430 --> 00:57:15.930 align:middle line:84% But let's for the moment just assume 00:57:15.930 --> 00:57:21.850 align:middle line:84% that you get one number, which is maybe the average response. 00:57:21.850 --> 00:57:22.590 align:middle line:90% All right. 00:57:22.590 --> 00:57:27.090 align:middle line:84% And the other thing that we're going to keep in mind here 00:57:27.090 --> 00:57:30.652 align:middle line:84% is that we're talking about the problem of object recognition. 00:57:30.652 --> 00:57:32.610 align:middle line:84% And one of the challenges of object recognition 00:57:32.610 --> 00:57:35.290 align:middle line:84% is the need to be invariant to all 00:57:35.290 --> 00:57:37.610 align:middle line:84% of the different factors that can 00:57:37.610 --> 00:57:40.930 align:middle line:84% cause the image that's produced by an object to change. 00:57:40.930 --> 00:57:43.350 align:middle line:84% So in this particular case, it's the viewpoint. 00:57:43.350 --> 00:57:46.850 align:middle line:84% So there's this person, evidently named Joe, 00:57:46.850 --> 00:57:50.090 align:middle line:84% who's being seen from different views. 00:57:50.090 --> 00:57:54.850 align:middle line:84% And those different views create different images. 00:57:54.850 --> 00:57:58.593 align:middle line:84% So each one of those images will generate a representation, 00:57:58.593 --> 00:58:00.010 align:middle line:84% which again, in this framework, is 00:58:00.010 --> 00:58:04.690 align:middle line:84% a point in this high-dimensional space of neural responses. 00:58:04.690 --> 00:58:06.570 align:middle line:84% And the collection of all of those 00:58:06.570 --> 00:58:10.290 align:middle line:84% points, we're going to call that a manifold that 00:58:10.290 --> 00:58:11.610 align:middle line:90% corresponds to Joe. 00:58:11.610 --> 00:58:15.250 align:middle line:90% 00:58:15.250 --> 00:58:18.210 align:middle line:84% And so the proposal here is that what 00:58:18.210 --> 00:58:21.350 align:middle line:84% we would like to achieve with our brain, 00:58:21.350 --> 00:58:23.930 align:middle line:84% in particular with our visual system, 00:58:23.930 --> 00:58:27.610 align:middle line:84% is a good set of visual features that 00:58:27.610 --> 00:58:30.170 align:middle line:84% has this property that we're going 00:58:30.170 --> 00:58:34.030 align:middle line:84% to call an explicit representation of object shape. 00:58:34.030 --> 00:58:37.930 align:middle line:84% And so this property is going to be that when you look at all 00:58:37.930 --> 00:58:41.850 align:middle line:84% of the images corresponding to one particular type of objects-- 00:58:41.850 --> 00:58:43.770 align:middle line:90% in this case, Joe-- 00:58:43.770 --> 00:58:47.130 align:middle line:84% they're going to reside in one part 00:58:47.130 --> 00:58:49.650 align:middle line:90% of this high-dimensional space. 00:58:49.650 --> 00:58:52.810 align:middle line:84% Whereas the images of some other kind of object, 00:58:52.810 --> 00:58:57.770 align:middle line:84% Sam, will be in another part of the space. 00:58:57.770 --> 00:59:02.410 align:middle line:84% And we will evaluate whether the manifold for Joe 00:59:02.410 --> 00:59:08.250 align:middle line:84% and the manifold for Sam are really separate by testing 00:59:08.250 --> 00:59:12.270 align:middle line:84% whether we can separate them with a plane, or a hyperplane 00:59:12.270 --> 00:59:14.690 align:middle line:84% in this case, because it's a high-dimensional space. 00:59:14.690 --> 00:59:19.950 align:middle line:84% And you can evaluate that with a linear classifier. 00:59:19.950 --> 00:59:22.490 align:middle line:84% So a linear classifier by definition 00:59:22.490 --> 00:59:23.990 align:middle line:90% is based on a hyperplane. 00:59:23.990 --> 00:59:27.190 align:middle line:90% And it has this hyperplane. 00:59:27.190 --> 00:59:29.370 align:middle line:84% And when points are on one side of the hyperplane, 00:59:29.370 --> 00:59:30.710 align:middle line:90% it says it's class 1. 00:59:30.710 --> 00:59:32.918 align:middle line:84% And when they're on the other side of the hyperplane, 00:59:32.918 --> 00:59:33.970 align:middle line:90% it says it's class 2. 00:59:33.970 --> 00:59:41.330 align:middle line:90% OK, so that's the proposal. 00:59:41.330 --> 00:59:44.370 align:middle line:84% This hypothesis is that the visual system should 00:59:44.370 --> 00:59:48.690 align:middle line:84% be performing transformations that produce representations 00:59:48.690 --> 00:59:53.250 align:middle line:84% like this, so that the images corresponding 00:59:53.250 --> 00:59:56.632 align:middle line:84% to one kind of thing end up in one part of this representation 00:59:56.632 --> 00:59:58.090 align:middle line:84% space, and the images corresponding 00:59:58.090 --> 01:00:02.690 align:middle line:84% to other parts of things lie in another part of that space, such 01:00:02.690 --> 01:00:04.480 align:middle line:84% that you can separate them with a plane 01:00:04.480 --> 01:00:09.600 align:middle line:84% and thus classify things with a linear classifier. 01:00:09.600 --> 01:00:11.340 align:middle line:84% OK, so what is a linear classifier? 01:00:11.340 --> 01:00:16.760 align:middle line:84% So a linear classifier uses projection onto a vector. 01:00:16.760 --> 01:00:22.600 align:middle line:84% So the classifier is defined by a direction, in this case b, 01:00:22.600 --> 01:00:25.160 align:middle line:90% and a threshold. 01:00:25.160 --> 01:00:30.320 align:middle line:84% So you take some observed response to an image. 01:00:30.320 --> 01:00:31.360 align:middle line:90% Could be a. 01:00:31.360 --> 01:00:35.280 align:middle line:84% You project it onto the classification vector. 01:00:35.280 --> 01:00:39.240 align:middle line:84% And that projection, depending on which side 01:00:39.240 --> 01:00:41.640 align:middle line:84% of the classification plane it is, 01:00:41.640 --> 01:00:45.720 align:middle line:84% would determine whether you're looking at class 1 or class 2. 01:00:45.720 --> 01:00:46.300 align:middle line:90% All right? 01:00:46.300 --> 01:00:48.960 align:middle line:90% 01:00:48.960 --> 01:00:52.800 align:middle line:84% So it just involves dot products and thresholds. 01:00:52.800 --> 01:00:56.120 align:middle line:84% And it's natural to think that you could potentially 01:00:56.120 --> 01:00:57.940 align:middle line:90% instantiate this with a neuron. 01:00:57.940 --> 01:01:01.880 align:middle line:84% So we often think of what a single neuron might be doing 01:01:01.880 --> 01:01:03.300 align:middle line:90% as computing a dot product. 01:01:03.300 --> 01:01:07.577 align:middle line:84% So think of it like synaptic weights and taking an input 01:01:07.577 --> 01:01:09.160 align:middle line:84% and multiplying it by all the weights, 01:01:09.160 --> 01:01:10.202 align:middle line:90% and then adding those up. 01:01:10.202 --> 01:01:12.520 align:middle line:84% And that gives you the membrane potential. 01:01:12.520 --> 01:01:15.180 align:middle line:84% And then there's a threshold inside the soma, 01:01:15.180 --> 01:01:17.300 align:middle line:84% which determines whether you respond or not. 01:01:17.300 --> 01:01:20.688 align:middle line:84% Again, that's a super oversimplified description 01:01:20.688 --> 01:01:21.980 align:middle line:90% of what a neuron actually does. 01:01:21.980 --> 01:01:25.640 align:middle line:84% But again, very loosely could be mapped on to this kind of math. 01:01:25.640 --> 01:01:28.280 align:middle line:84% So we often think that neurons could potentially 01:01:28.280 --> 01:01:30.300 align:middle line:84% be instantiating linear classifiers. 01:01:30.300 --> 01:01:33.420 align:middle line:90% 01:01:33.420 --> 01:01:33.920 align:middle line:90% All right. 01:01:33.920 --> 01:01:36.020 align:middle line:90% So here's the problem of vision. 01:01:36.020 --> 01:01:38.600 align:middle line:84% So we talked about how recognition is hard 01:01:38.600 --> 01:01:41.760 align:middle line:84% because there's all these ways in which images 01:01:41.760 --> 01:01:46.000 align:middle line:84% of the same kind of thing can vary. 01:01:46.000 --> 01:01:50.080 align:middle line:84% And another way of stating that is 01:01:50.080 --> 01:01:53.200 align:middle line:84% that the manifolds that correspond 01:01:53.200 --> 01:01:56.800 align:middle line:84% to different types of objects, when viewed in the pixel space, 01:01:56.800 --> 01:02:02.360 align:middle line:84% in the input space, are very tangled. 01:02:02.360 --> 01:02:04.920 align:middle line:84% So in the pixel space, what makes 01:02:04.920 --> 01:02:09.320 align:middle line:84% recognition hard is that all of the images of one person 01:02:09.320 --> 01:02:11.160 align:middle line:84% are not cleanly separated from all 01:02:11.160 --> 01:02:13.320 align:middle line:90% the images of another person. 01:02:13.320 --> 01:02:16.840 align:middle line:84% So those manifolds are all tangled up. 01:02:16.840 --> 01:02:19.640 align:middle line:84% This is why you need a visual system. 01:02:19.640 --> 01:02:22.480 align:middle line:84% So the hypothesis is that what the visual system is doing 01:02:22.480 --> 01:02:27.640 align:middle line:84% is performing transformations on the representations 01:02:27.640 --> 01:02:34.120 align:middle line:84% of the images to end up with something where 01:02:34.120 --> 01:02:36.537 align:middle line:84% the representations of all the images of one kind of thing 01:02:36.537 --> 01:02:38.537 align:middle line:84% are in one part of the space, and all the others 01:02:38.537 --> 01:02:40.040 align:middle line:90% are in the other part of space. 01:02:40.040 --> 01:02:42.940 align:middle line:90% And we call that explicit. 01:02:42.940 --> 01:02:47.400 align:middle line:84% So the notion of making something explicit 01:02:47.400 --> 01:02:51.360 align:middle line:84% is that you make it easy to read off 01:02:51.360 --> 01:02:54.320 align:middle line:84% from the neural population or some other kind 01:02:54.320 --> 01:02:55.720 align:middle line:90% of representation. 01:02:55.720 --> 01:03:04.120 align:middle line:84% And easy to read off is typically instantiated as being 01:03:04.120 --> 01:03:06.260 align:middle line:84% able to use a very simple classifier-- 01:03:06.260 --> 01:03:09.600 align:middle line:84% and a linear classifier is the simplest type of classifier-- 01:03:09.600 --> 01:03:12.680 align:middle line:84% and being able to extract that information. 01:03:12.680 --> 01:03:14.940 align:middle line:84% So that's the notion of something being explicit. 01:03:14.940 --> 01:03:18.800 align:middle line:84% So the idea is that the identity of the individual 01:03:18.800 --> 01:03:22.200 align:middle line:90% in the pixel input is implicit. 01:03:22.200 --> 01:03:25.720 align:middle line:84% So it's there because the visual system 01:03:25.720 --> 01:03:27.280 align:middle line:84% just gets that image as input and is 01:03:27.280 --> 01:03:29.000 align:middle line:90% able to recognize something. 01:03:29.000 --> 01:03:30.340 align:middle line:90% But it's not explicit. 01:03:30.340 --> 01:03:32.640 align:middle line:90% It's not easy to read off. 01:03:32.640 --> 01:03:35.680 align:middle line:84% But then you perform this series of transformations 01:03:35.680 --> 01:03:37.760 align:middle line:84% and you generate these representations 01:03:37.760 --> 01:03:41.380 align:middle line:84% where by hypothesis, things are explicit. 01:03:41.380 --> 01:03:42.860 align:middle line:90% So that's the idea. 01:03:42.860 --> 01:03:45.140 align:middle line:84% Now, we haven't said what these transformations are. 01:03:45.140 --> 01:03:47.320 align:middle line:84% And in fact, it's kind of non-obvious 01:03:47.320 --> 01:03:49.560 align:middle line:84% how to write down transformations 01:03:49.560 --> 01:03:51.020 align:middle line:90% that would do this. 01:03:51.020 --> 01:03:53.800 align:middle line:84% But we can ask the question of whether or not 01:03:53.800 --> 01:03:58.230 align:middle line:84% this actually happens in the human or the monkey 01:03:58.230 --> 01:03:59.250 align:middle line:90% visual system. 01:03:59.250 --> 01:04:02.190 align:middle line:90% 01:04:02.190 --> 01:04:04.630 align:middle line:84% OK, so what I'm going to tell you about here 01:04:04.630 --> 01:04:11.470 align:middle line:84% is a test of this idea in inferotemporal cortex. 01:04:11.470 --> 01:04:14.510 align:middle line:84% So this is an experiment where monkeys 01:04:14.510 --> 01:04:17.990 align:middle line:84% were presented with a big set of 78 objects 01:04:17.990 --> 01:04:20.870 align:middle line:90% from eight categories. 01:04:20.870 --> 01:04:25.750 align:middle line:84% So the objects could vary in their scale and position 01:04:25.750 --> 01:04:27.770 align:middle line:84% and also in what kind of thing they were. 01:04:27.770 --> 01:04:28.570 align:middle line:90% So here's a monkey. 01:04:28.570 --> 01:04:29.330 align:middle line:90% Here's a tank. 01:04:29.330 --> 01:04:31.790 align:middle line:90% Here's some kind of toy. 01:04:31.790 --> 01:04:33.750 align:middle line:84% OK, so lots of different objects. 01:04:33.750 --> 01:04:37.150 align:middle line:90% They're presented very briefly. 01:04:37.150 --> 01:04:39.550 align:middle line:90% The monkeys are fixating. 01:04:39.550 --> 01:04:42.830 align:middle line:84% And you're recording from a whole bunch of neurons 01:04:42.830 --> 01:04:45.310 align:middle line:84% in the inferotemporal cortex while the monkeys 01:04:45.310 --> 01:04:48.230 align:middle line:90% are looking at these objects. 01:04:48.230 --> 01:04:54.190 align:middle line:84% So this is the outcome of the neurophysiological experiment. 01:04:54.190 --> 01:04:57.750 align:middle line:90% So we've got 78 images here. 01:04:57.750 --> 01:05:00.350 align:middle line:84% So the columns represent images and the rows 01:05:00.350 --> 01:05:01.988 align:middle line:90% represent recording sites. 01:05:01.988 --> 01:05:04.030 align:middle line:84% So you kind of loosely think of those as neurons. 01:05:04.030 --> 01:05:07.790 align:middle line:84% In actuality, each one of these is more than one neuron. 01:05:07.790 --> 01:05:11.263 align:middle line:84% But whether or not it's one neuron or several neurons 01:05:11.263 --> 01:05:12.930 align:middle line:84% is not really critical for our purposes. 01:05:12.930 --> 01:05:14.930 align:middle line:84% It's a different little bit of the brain. 01:05:14.930 --> 01:05:18.710 align:middle line:84% And so you can see that particular recording 01:05:18.710 --> 01:05:21.130 align:middle line:84% sites respond more to particular images than others. 01:05:21.130 --> 01:05:24.430 align:middle line:84% Just this big complicated set of responses. 01:05:24.430 --> 01:05:27.670 align:middle line:84% So now what we're going to try to ask here 01:05:27.670 --> 01:05:34.230 align:middle line:84% is whether or not this neural representation makes information 01:05:34.230 --> 01:05:36.910 align:middle line:90% about objects explicit. 01:05:36.910 --> 01:05:39.350 align:middle line:84% So remember, the idea is that we're 01:05:39.350 --> 01:05:41.110 align:middle line:84% going to think of a particular image 01:05:41.110 --> 01:05:45.270 align:middle line:84% now as being represented in a point in-- 01:05:45.270 --> 01:05:48.710 align:middle line:84% well, here we're seeing 63 recording sites. 01:05:48.710 --> 01:05:50.770 align:middle line:90% They had about 350 in total. 01:05:50.770 --> 01:05:55.990 align:middle line:84% So it's a point in a 350-dimensional space. 01:05:55.990 --> 01:05:57.810 align:middle line:84% Different images are different points. 01:05:57.810 --> 01:06:01.830 align:middle line:84% And the question is whether we're 01:06:01.830 --> 01:06:04.630 align:middle line:84% dealing with an implicit representation-- 01:06:04.630 --> 01:06:09.070 align:middle line:84% so this would be an example where the green points 01:06:09.070 --> 01:06:11.550 align:middle line:84% correspond to one object and the red points correspond 01:06:11.550 --> 01:06:13.150 align:middle line:90% to another object. 01:06:13.150 --> 01:06:16.010 align:middle line:84% And so here they're a little bit intermingled. 01:06:16.010 --> 01:06:21.030 align:middle line:84% You can't like lay down a plane and classify these things 01:06:21.030 --> 01:06:22.510 align:middle line:90% linearly. 01:06:22.510 --> 01:06:23.830 align:middle line:90% Here, you can. 01:06:23.830 --> 01:06:25.210 align:middle line:90% These are linearly separable. 01:06:25.210 --> 01:06:29.230 align:middle line:84% So there's a plane that puts the green dots on one side 01:06:29.230 --> 01:06:31.870 align:middle line:90% and the red dots on the other. 01:06:31.870 --> 01:06:35.510 align:middle line:84% So we call this an explicit representation and this 01:06:35.510 --> 01:06:36.690 align:middle line:90% an implicit representation. 01:06:36.690 --> 01:06:38.690 align:middle line:84% And of course, it could be much worse than this. 01:06:38.690 --> 01:06:41.390 align:middle line:84% The red and the green could be completely intermingled 01:06:41.390 --> 01:06:42.150 align:middle line:90% in principle. 01:06:42.150 --> 01:06:46.270 align:middle line:90% 01:06:46.270 --> 01:06:49.830 align:middle line:90% So how do we test this? 01:06:49.830 --> 01:06:53.430 align:middle line:84% Well, you take the neural responses 01:06:53.430 --> 01:06:57.910 align:middle line:84% and you fit a classifier for a particular type of object. 01:06:57.910 --> 01:07:02.830 align:middle line:84% And I'm not going to tell you exactly how you fit that. 01:07:02.830 --> 01:07:06.410 align:middle line:84% But there's established ways to do that. 01:07:06.410 --> 01:07:08.510 align:middle line:84% And then you can test it on some separate images 01:07:08.510 --> 01:07:11.330 align:middle line:84% and see whether it classifies things correctly. 01:07:11.330 --> 01:07:16.590 align:middle line:84% So you're trying to find the best hyperplane that's 01:07:16.590 --> 01:07:18.950 align:middle line:84% going to divide the red and the green dots 01:07:18.950 --> 01:07:20.485 align:middle line:90% for some subset of the images. 01:07:20.485 --> 01:07:22.110 align:middle line:84% And then you're going to see whether it 01:07:22.110 --> 01:07:26.670 align:middle line:84% gets them correct on some held-out images. 01:07:26.670 --> 01:07:29.390 align:middle line:90% And this is the results of that. 01:07:29.390 --> 01:07:35.270 align:middle line:84% So the y-axis here shows classification performance. 01:07:35.270 --> 01:07:38.830 align:middle line:84% And the x-axis is the number of sites 01:07:38.830 --> 01:07:40.990 align:middle line:84% that the classifier can operate on. 01:07:40.990 --> 01:07:44.350 align:middle line:84% So you can ask, how does the ability to classify things 01:07:44.350 --> 01:07:46.010 align:middle line:90% vary with the number of sites? 01:07:46.010 --> 01:07:47.170 align:middle line:90% And it gets better. 01:07:47.170 --> 01:07:49.147 align:middle line:90% And the point is that-- 01:07:49.147 --> 01:07:50.980 align:middle line:84% and there's two curves here, because there's 01:07:50.980 --> 01:07:54.940 align:middle line:84% two kinds of classification that can be done. 01:07:54.940 --> 01:07:57.500 align:middle line:84% There's categorization, which I think 01:07:57.500 --> 01:08:03.860 align:middle line:84% is like whether you have the general class of object correct, 01:08:03.860 --> 01:08:06.560 align:middle line:84% so like if it's a monkey versus something else. 01:08:06.560 --> 01:08:08.140 align:middle line:84% And then identification, which is 01:08:08.140 --> 01:08:11.940 align:middle line:84% whether you have the specific exemplar within that class 01:08:11.940 --> 01:08:12.440 align:middle line:90% correct. 01:08:12.440 --> 01:08:15.080 align:middle line:84% And so the identification task is harder, 01:08:15.080 --> 01:08:17.520 align:middle line:84% which is why the blue curve is below the yellow curve. 01:08:17.520 --> 01:08:19.479 align:middle line:84% But the point is that in both cases, 01:08:19.479 --> 01:08:22.819 align:middle line:84% the classification performance from the brain responses just 01:08:22.819 --> 01:08:25.899 align:middle line:84% using a linear classifier increases with the number 01:08:25.899 --> 01:08:29.620 align:middle line:90% of recording sites. 01:08:29.620 --> 01:08:33.779 align:middle line:84% So this implies that information about objects 01:08:33.779 --> 01:08:36.620 align:middle line:84% is made explicit in inferotemporal cortex. 01:08:36.620 --> 01:08:41.939 align:middle line:84% And so critically, you can't do this, or at least not as well, 01:08:41.939 --> 01:08:43.979 align:middle line:90% in earlier visual areas. 01:08:43.979 --> 01:08:47.880 align:middle line:84% And so this is one graph that kind of shows this. 01:08:47.880 --> 01:08:50.700 align:middle line:84% So in this analysis, it's the same kind of experiment 01:08:50.700 --> 01:08:52.620 align:middle line:84% that I just showed you from a different paper. 01:08:52.620 --> 01:08:55.037 align:middle line:84% So you measure responses to a very large number of objects 01:08:55.037 --> 01:08:56.620 align:middle line:84% and a very large number of neurons. 01:08:56.620 --> 01:09:00.660 align:middle line:84% But here this was done in inferotemporal cortex as well as 01:09:00.660 --> 01:09:01.359 align:middle line:90% V4. 01:09:01.359 --> 01:09:02.899 align:middle line:90% So that's the green curve. 01:09:02.899 --> 01:09:06.359 align:middle line:84% And so in addition-- so they measured two things here. 01:09:06.359 --> 01:09:09.500 align:middle line:84% One is performance, so how well objectively you 01:09:09.500 --> 01:09:11.740 align:middle line:90% can classify the object images. 01:09:11.740 --> 01:09:14.880 align:middle line:84% But the other is consistency with human judgments. 01:09:14.880 --> 01:09:17.060 align:middle line:84% So they showed these images to humans 01:09:17.060 --> 01:09:19.979 align:middle line:84% and had them classify the images. 01:09:19.979 --> 01:09:22.979 align:middle line:84% And humans are good at this task, but they're not perfect. 01:09:22.979 --> 01:09:26.020 align:middle line:84% And so you can measure the consistency 01:09:26.020 --> 01:09:30.580 align:middle line:84% of these brain classifiers with human judgments. 01:09:30.580 --> 01:09:35.340 align:middle line:84% So on the x-axis is classification performance 01:09:35.340 --> 01:09:38.180 align:middle line:84% and on the y-axis is consistency. 01:09:38.180 --> 01:09:40.859 align:middle line:84% And so the blue curve here shows the results 01:09:40.859 --> 01:09:44.580 align:middle line:84% of this analysis for recordings from inferotemporal cortex. 01:09:44.580 --> 01:09:48.300 align:middle line:84% The different dots are different numbers of recording sites. 01:09:48.300 --> 01:09:51.100 align:middle line:84% And so as the number of recording sites increases, 01:09:51.100 --> 01:09:55.260 align:middle line:84% the classification performance increases and the consistency 01:09:55.260 --> 01:09:57.920 align:middle line:90% with human judgments increases. 01:09:57.920 --> 01:10:00.360 align:middle line:84% So the idea is that with a modest number of sites, 01:10:00.360 --> 01:10:03.500 align:middle line:84% like a couple hundred sites, you can classify things about 01:10:03.500 --> 01:10:07.660 align:middle line:84% as well as people can and with pretty good consistency 01:10:07.660 --> 01:10:09.340 align:middle line:90% with human judgments. 01:10:09.340 --> 01:10:11.500 align:middle line:84% By comparison, if you do the exact same thing 01:10:11.500 --> 01:10:16.180 align:middle line:84% in an earlier stage of the ventral stream, V4, even with-- 01:10:16.180 --> 01:10:19.260 align:middle line:84% I guess the most they analyzed here is 128, 01:10:19.260 --> 01:10:21.800 align:middle line:84% and the classification performance is above chance, 01:10:21.800 --> 01:10:24.100 align:middle line:90% but not very good. 01:10:24.100 --> 01:10:27.060 align:middle line:84% And things are not very consistent with human judgments. 01:10:27.060 --> 01:10:30.060 align:middle line:90% 01:10:30.060 --> 01:10:32.460 align:middle line:84% And the trajectory of this curve suggests 01:10:32.460 --> 01:10:34.720 align:middle line:84% that if you continued to analyze more neurons, 01:10:34.720 --> 01:10:39.620 align:middle line:84% it's not going to get dramatically better. 01:10:39.620 --> 01:10:47.340 align:middle line:84% So this again suggests that infratemporal cortex contains 01:10:47.340 --> 01:10:48.960 align:middle line:84% information about object identity, 01:10:48.960 --> 01:10:54.220 align:middle line:84% but in particular its making object identity explicit 01:10:54.220 --> 01:10:58.140 align:middle line:84% in the sense that different images that are of the same type 01:10:58.140 --> 01:11:01.580 align:middle line:84% are in a particular part of the representational space, 01:11:01.580 --> 01:11:06.060 align:middle line:84% and linearly separated from other kinds of images. 01:11:06.060 --> 01:11:07.900 align:middle line:84% And that enables recognition because 01:11:07.900 --> 01:11:10.740 align:middle line:84% with a linear classifier, as you could potentially 01:11:10.740 --> 01:11:13.340 align:middle line:84% implement with a downstream neuron, 01:11:13.340 --> 01:11:15.380 align:middle line:84% you can tell which kind of object 01:11:15.380 --> 01:11:18.940 align:middle line:84% you're dealing with just from the neural responses. 01:11:18.940 --> 01:11:25.060 align:middle line:84% So the conclusion is that IT neurons mediate object 01:11:25.060 --> 01:11:29.660 align:middle line:84% recognition by making object identity explicit, which again 01:11:29.660 --> 01:11:32.900 align:middle line:84% means easily read out, which again means that you 01:11:32.900 --> 01:11:35.380 align:middle line:90% can linearly classify it. 01:11:35.380 --> 01:11:40.980 align:middle line:84% So the key concepts here are this idea 01:11:40.980 --> 01:11:47.730 align:middle line:84% that sensory systems consist of this cascade of transforms that 01:11:47.730 --> 01:11:52.410 align:middle line:84% change the representation, and that those transformations may 01:11:52.410 --> 01:11:57.890 align:middle line:84% have either evolved or been learned to make information 01:11:57.890 --> 01:11:59.652 align:middle line:84% that matters to us, for instance, 01:11:59.652 --> 01:12:01.610 align:middle line:84% about objects in the case of visual recognition 01:12:01.610 --> 01:12:08.570 align:middle line:84% explicit, such that things of the same type of object category 01:12:08.570 --> 01:12:11.190 align:middle line:84% are kind of in the same part of the representational space. 01:12:11.190 --> 01:12:12.982 align:middle line:84% And so we're thinking of the representation 01:12:12.982 --> 01:12:15.490 align:middle line:84% here as this high-dimensional space 01:12:15.490 --> 01:12:19.650 align:middle line:84% where the different axes of the responses of different neurons. 01:12:19.650 --> 01:12:20.870 align:middle line:90% Questions about that? 01:12:20.870 --> 01:12:30.250 align:middle line:90% 01:12:30.250 --> 01:12:31.330 align:middle line:90% Yeah. 01:12:31.330 --> 01:12:35.090 align:middle line:84% AUDIENCE: So is learning to tell apart similar things 01:12:35.090 --> 01:12:40.850 align:middle line:84% just developing the procedures in order to make them explicit? 01:12:40.850 --> 01:12:43.570 align:middle line:84% JOSH MCDERMOTT: Yeah, so that is a great question. 01:12:43.570 --> 01:12:46.930 align:middle line:84% It's like, well, what happens if, say, 01:12:46.930 --> 01:12:52.530 align:middle line:84% we invent a new type of object that nobody's ever seen before? 01:12:52.530 --> 01:12:54.250 align:middle line:84% And you get a lot of experience with that 01:12:54.250 --> 01:12:57.050 align:middle line:84% object such that you're really good at recognizing it. 01:12:57.050 --> 01:12:59.130 align:middle line:84% Is the representation changing to make 01:12:59.130 --> 01:13:00.810 align:middle line:90% that a lot more explicit? 01:13:00.810 --> 01:13:04.850 align:middle line:84% So that's a very plausible hypothesis. 01:13:04.850 --> 01:13:07.210 align:middle line:84% To my knowledge, we do not have very good evidence 01:13:07.210 --> 01:13:10.410 align:middle line:90% for that at the moment. 01:13:10.410 --> 01:13:13.690 align:middle line:84% So I think a lot of this work that I talked to you about 01:13:13.690 --> 01:13:19.527 align:middle line:84% was conducted in the lab of Jim DiCarlo here in the building. 01:13:19.527 --> 01:13:21.110 align:middle line:84% I had dinner with him the other night, 01:13:21.110 --> 01:13:24.010 align:middle line:84% and we were actually talking about exactly that issue. 01:13:24.010 --> 01:13:26.730 align:middle line:84% And I think his stance on this is 01:13:26.730 --> 01:13:30.670 align:middle line:84% that there are not very many clear effects of learning. 01:13:30.670 --> 01:13:33.570 align:middle line:90% 01:13:33.570 --> 01:13:36.150 align:middle line:90% But I think it's an open issue. 01:13:36.150 --> 01:13:38.150 align:middle line:84% So people have tried to look at it a little bit, 01:13:38.150 --> 01:13:41.350 align:middle line:84% but hasn't really definitively been looked at. 01:13:41.350 --> 01:13:43.150 align:middle line:84% But it's a very plausible hypothesis. 01:13:43.150 --> 01:13:43.650 align:middle line:90% Yeah. 01:13:43.650 --> 01:13:49.890 align:middle line:90% 01:13:49.890 --> 01:13:50.390 align:middle line:90% Yeah. 01:13:50.390 --> 01:13:52.810 align:middle line:84% And so one kind of pretty interesting question 01:13:52.810 --> 01:13:59.890 align:middle line:84% that I think is still kind of open is the extent to which-- 01:13:59.890 --> 01:14:06.410 align:middle line:84% like, these responses are baked into the visual system. 01:14:06.410 --> 01:14:12.690 align:middle line:84% Does evolution give us this set of featural transforms and we're 01:14:12.690 --> 01:14:14.410 align:middle line:90% mostly born with it? 01:14:14.410 --> 01:14:17.210 align:middle line:84% Is it mostly learned after development? 01:14:17.210 --> 01:14:20.570 align:middle line:84% It's very hard to look at, just because it's 01:14:20.570 --> 01:14:22.690 align:middle line:84% kind of hard to do these experiments 01:14:22.690 --> 01:14:25.450 align:middle line:90% in very young organisms. 01:14:25.450 --> 01:14:26.950 align:middle line:84% So that's kind of an open question. 01:14:26.950 --> 01:14:30.490 align:middle line:84% But there's a lot of interest in that right now. 01:14:30.490 --> 01:14:34.570 align:middle line:84% Different people disagree about the extent to which these things 01:14:34.570 --> 01:14:36.750 align:middle line:90% are innate versus learned. 01:14:36.750 --> 01:14:41.370 align:middle line:90% 01:14:41.370 --> 01:14:43.790 align:middle line:90% OK. 01:14:43.790 --> 01:14:49.050 align:middle line:84% So when we come back, we'll talk about another wrinkle 01:14:49.050 --> 01:14:54.570 align:middle line:84% in all of this, which is pieces of evidence for specialization 01:14:54.570 --> 01:14:58.050 align:middle line:84% of the recognition machinery in the brain. 01:14:58.050 --> 01:15:00.490 align:middle line:84% In particular, evidence that neurons 01:15:00.490 --> 01:15:03.890 align:middle line:84% that are responsive to certain classes of objects 01:15:03.890 --> 01:15:05.630 align:middle line:84% seem to be segregated in the brain. 01:15:05.630 --> 01:15:07.422 align:middle line:84% So a lot of the evidence for this in humans 01:15:07.422 --> 01:15:09.370 align:middle line:84% comes from our own Nancy Kanwisher 01:15:09.370 --> 01:15:15.410 align:middle line:84% via fMRI, who has accumulated a lot of evidence 01:15:15.410 --> 01:15:18.930 align:middle line:84% for these different pieces of the brain that 01:15:18.930 --> 01:15:21.430 align:middle line:84% are functionally specialized for fairly specific things. 01:15:21.430 --> 01:15:23.810 align:middle line:84% And maybe the best example is faces. 01:15:23.810 --> 01:15:27.850 align:middle line:84% So this is a picture that is showing a few well-known brain 01:15:27.850 --> 01:15:33.210 align:middle line:84% areas-- the fusiform face area that's very responsive to faces, 01:15:33.210 --> 01:15:36.040 align:middle line:84% the parahippocampal place area that's 01:15:36.040 --> 01:15:40.960 align:middle line:84% very responsive to pictures of places, the extrastriate body 01:15:40.960 --> 01:15:45.680 align:middle line:84% area that responds to bodies, and then MT, 01:15:45.680 --> 01:15:47.380 align:middle line:90% which is responsive to motion. 01:15:47.380 --> 01:15:51.520 align:middle line:84% So these are examples of functional specialization. 01:15:51.520 --> 01:15:56.520 align:middle line:84% And so that's another kind of interesting property 01:15:56.520 --> 01:15:58.212 align:middle line:84% of our visual recognition mechanisms. 01:15:58.212 --> 01:15:59.920 align:middle line:84% And so we'll talk a little bit about that 01:15:59.920 --> 01:16:02.510 align:middle line:90% when we come back next time. 01:16:02.510 --> 01:16:33.000 align:middle line:90%