WEBVTT 00:00:00.000 --> 00:00:09.223 align:middle line:84% [SQUEAKING] [RUSTLING] [CLICKING] 00:00:09.223 --> 00:00:10.640 align:middle line:84% JOSH MCDERMOTT: Today, we're going 00:00:10.640 --> 00:00:13.840 align:middle line:84% to finish up this lecture on mid-level vision, 00:00:13.840 --> 00:00:16.960 align:middle line:84% and then start talking about lightness perception. 00:00:16.960 --> 00:00:17.960 align:middle line:90% And so I want to start-- 00:00:17.960 --> 00:00:20.300 align:middle line:84% I know it's been a little while since we last met. 00:00:20.300 --> 00:00:22.840 align:middle line:84% So I want to just start by briefly recapping 00:00:22.840 --> 00:00:26.510 align:middle line:90% where we left things last time. 00:00:26.510 --> 00:00:29.840 align:middle line:84% So we talked a little bit about the broader structure 00:00:29.840 --> 00:00:32.400 align:middle line:84% of the visual system, the idea that it's divided up 00:00:32.400 --> 00:00:35.440 align:middle line:84% into a whole bunch of areas, that these areas are delineated 00:00:35.440 --> 00:00:37.760 align:middle line:90% by having retinotopic maps. 00:00:37.760 --> 00:00:41.400 align:middle line:84% We talked a bit about how those are measured. 00:00:41.400 --> 00:00:43.880 align:middle line:84% We reviewed the relationship between the left 00:00:43.880 --> 00:00:48.200 align:middle line:84% and the right visual hemifields and the representation 00:00:48.200 --> 00:00:49.000 align:middle line:90% in the cortex. 00:00:49.000 --> 00:00:52.318 align:middle line:84% The central idea is that the left side of visual space 00:00:52.318 --> 00:00:54.860 align:middle line:84% is represented on the right side of the brain and vice versa. 00:00:54.860 --> 00:00:57.200 align:middle line:90% We saw some examples of this. 00:00:57.200 --> 00:01:00.790 align:middle line:84% And then talked about how the visual system continues 00:01:00.790 --> 00:01:02.290 align:middle line:90% to be organized in pathways. 00:01:02.290 --> 00:01:03.710 align:middle line:84% So you see these differences that 00:01:03.710 --> 00:01:07.710 align:middle line:84% start at the retina, also very salient in the LGN. 00:01:07.710 --> 00:01:11.070 align:middle line:84% And they continue into the visual cortex. 00:01:11.070 --> 00:01:13.030 align:middle line:84% And there's this broad proposed organization 00:01:13.030 --> 00:01:15.350 align:middle line:84% into dorsal and ventral pathways. 00:01:15.350 --> 00:01:17.390 align:middle line:84% But there's lots of visual areas that 00:01:17.390 --> 00:01:19.590 align:middle line:90% are connected in lots of ways. 00:01:19.590 --> 00:01:22.775 align:middle line:84% But one of the characteristics of the organization 00:01:22.775 --> 00:01:24.150 align:middle line:84% of the visual pathway is the fact 00:01:24.150 --> 00:01:26.430 align:middle line:84% that we think of it as being hierarchical. 00:01:26.430 --> 00:01:27.990 align:middle line:84% So there are some regions that are 00:01:27.990 --> 00:01:29.573 align:middle line:84% closer to the input, some regions that 00:01:29.573 --> 00:01:30.850 align:middle line:90% are closer to the output. 00:01:30.850 --> 00:01:32.890 align:middle line:84% And as you move up the hierarchy, in general, 00:01:32.890 --> 00:01:35.410 align:middle line:84% the responses get more complicated, 00:01:35.410 --> 00:01:38.510 align:middle line:84% selective for complex structures that are often 00:01:38.510 --> 00:01:40.110 align:middle line:90% behaviorally meaningful. 00:01:40.110 --> 00:01:42.030 align:middle line:84% So we saw a few examples of that, 00:01:42.030 --> 00:01:43.710 align:middle line:84% where you can see neurons that are 00:01:43.710 --> 00:01:46.230 align:middle line:84% selective for particular individuals 00:01:46.230 --> 00:01:49.950 align:middle line:84% that you might see on TV or in movies. 00:01:49.950 --> 00:01:53.470 align:middle line:84% And then we talked about this idea of moving from early vision 00:01:53.470 --> 00:01:55.630 align:middle line:84% to mid-level vision, where we think of early vision 00:01:55.630 --> 00:01:59.590 align:middle line:84% as involving a big set of measurements, 00:01:59.590 --> 00:02:03.770 align:middle line:84% often starting from the retina and going up to roughly V1. 00:02:03.770 --> 00:02:05.270 align:middle line:84% These are measurements that we often 00:02:05.270 --> 00:02:09.273 align:middle line:84% think were made with filters that are relatively linear. 00:02:09.273 --> 00:02:11.190 align:middle line:84% And then there's a lot of perceptual phenomena 00:02:11.190 --> 00:02:13.390 align:middle line:84% that we can link to these measurements-- things 00:02:13.390 --> 00:02:16.790 align:middle line:84% like the tilt aftereffect, the contrast sensitivity function, 00:02:16.790 --> 00:02:19.870 align:middle line:84% effects of adaptation, stuff like that. 00:02:19.870 --> 00:02:22.270 align:middle line:84% By contrast, mid-level vision refers 00:02:22.270 --> 00:02:26.350 align:middle line:84% to things that are a little bit more related to inferences 00:02:26.350 --> 00:02:28.190 align:middle line:84% that we make about the world, that 00:02:28.190 --> 00:02:32.630 align:middle line:84% are based on the measurements of the early visual system that 00:02:32.630 --> 00:02:35.110 align:middle line:84% lead up to object recognition and scene perception, which 00:02:35.110 --> 00:02:39.470 align:middle line:84% are often thought of as high-level vision. 00:02:39.470 --> 00:02:41.430 align:middle line:84% And unlike early vision, mid-level vision 00:02:41.430 --> 00:02:44.630 align:middle line:84% is less well-linked to specific anatomical stages 00:02:44.630 --> 00:02:46.870 align:middle line:90% and to individual neurons. 00:02:46.870 --> 00:02:48.810 align:middle line:84% But there are nonetheless some linkages. 00:02:48.810 --> 00:02:50.630 align:middle line:84% And we'll talk a little bit about that. 00:02:50.630 --> 00:02:52.830 align:middle line:84% So one key idea that we talked about last time 00:02:52.830 --> 00:02:55.290 align:middle line:84% is this idea that local measurements are ambiguous. 00:02:55.290 --> 00:02:58.140 align:middle line:84% This is important because we think of the early visual system 00:02:58.140 --> 00:03:00.140 align:middle line:90% as making local measurements. 00:03:00.140 --> 00:03:02.640 align:middle line:84% Neurons typically have associated receptive fields. 00:03:02.640 --> 00:03:05.460 align:middle line:84% These are small regions of visual space 00:03:05.460 --> 00:03:07.300 align:middle line:90% to which the neuron responds. 00:03:07.300 --> 00:03:09.480 align:middle line:84% These might be a fraction of a degree. 00:03:09.480 --> 00:03:12.220 align:middle line:84% So it's like every neuron is looking 00:03:12.220 --> 00:03:15.260 align:middle line:84% at this tiny little portion of the world. 00:03:15.260 --> 00:03:18.540 align:middle line:84% And when you look at tiny little portions of the visual world, 00:03:18.540 --> 00:03:21.980 align:middle line:84% what you see-- what is evident in that small region 00:03:21.980 --> 00:03:23.040 align:middle line:90% is often ambiguous. 00:03:23.040 --> 00:03:25.000 align:middle line:84% And we saw a whole bunch of examples of that, 00:03:25.000 --> 00:03:28.780 align:middle line:84% like not being able to tell the difference between edges that 00:03:28.780 --> 00:03:34.020 align:middle line:84% are due to paint versus shading, or sometimes object edges that 00:03:34.020 --> 00:03:36.840 align:middle line:84% may be very apparent when you just view the image as a whole. 00:03:36.840 --> 00:03:39.500 align:middle line:84% If you zoom in and look at what the evidence is 00:03:39.500 --> 00:03:43.300 align:middle line:84% in a local region, it might be pretty weak. 00:03:43.300 --> 00:03:46.740 align:middle line:84% And then we got into this idea of perceptual grouping, which 00:03:46.740 --> 00:03:52.060 align:middle line:84% is a particular perceptual phenomenon that we often view 00:03:52.060 --> 00:03:54.060 align:middle line:90% as part of mid-level vision. 00:03:54.060 --> 00:03:57.660 align:middle line:84% That refers to the fact that different elements 00:03:57.660 --> 00:04:00.020 align:middle line:84% of a visual display, in many cases 00:04:00.020 --> 00:04:01.400 align:middle line:90% will tend to group together. 00:04:01.400 --> 00:04:04.260 align:middle line:84% And typically, they'll be grouped in part because they're 00:04:04.260 --> 00:04:05.700 align:middle line:90% similar in some way. 00:04:05.700 --> 00:04:08.580 align:middle line:84% And so this was a major focus of gestalt psychology. 00:04:08.580 --> 00:04:12.140 align:middle line:84% And the phenomena continue to be important today. 00:04:12.140 --> 00:04:14.637 align:middle line:84% So we saw a bunch of examples of grouping by similarity. 00:04:14.637 --> 00:04:16.720 align:middle line:84% And when we talk about the phenomenon of grouping, 00:04:16.720 --> 00:04:18.387 align:middle line:84% we're referring to the fact that, if you 00:04:18.387 --> 00:04:19.640 align:middle line:90% look at a display like this-- 00:04:19.640 --> 00:04:23.380 align:middle line:84% and this display is constructed by putting down, laying down 00:04:23.380 --> 00:04:24.880 align:middle line:84% a whole bunch of different elements. 00:04:24.880 --> 00:04:28.300 align:middle line:84% But when you look at it, there's a sense in which you perceive it 00:04:28.300 --> 00:04:31.580 align:middle line:90% as being organized into rows. 00:04:31.580 --> 00:04:34.300 align:middle line:84% And it's just subjectively obvious. 00:04:34.300 --> 00:04:37.180 align:middle line:84% So there's some aspect of how you 00:04:37.180 --> 00:04:41.260 align:middle line:84% perceive this in which those elements are grouped. 00:04:41.260 --> 00:04:43.420 align:middle line:84% So you can also group things by common fate, 00:04:43.420 --> 00:04:49.060 align:middle line:84% as shown here, by texture, by proximity. 00:04:49.060 --> 00:04:53.300 align:middle line:84% We talked again, revisiting this idea of levels of analysis 00:04:53.300 --> 00:04:55.330 align:middle line:84% that we've talked about several times 00:04:55.330 --> 00:04:59.450 align:middle line:84% previously in this class, how you can take some problem that's 00:04:59.450 --> 00:05:03.038 align:middle line:84% being solved by the visual system or any sensory system 00:05:03.038 --> 00:05:05.330 align:middle line:84% and think about it in terms of these different levels-- 00:05:05.330 --> 00:05:07.890 align:middle line:84% computational, algorithmic, and implementation. 00:05:07.890 --> 00:05:09.690 align:middle line:84% And you can often explain phenomena 00:05:09.690 --> 00:05:10.990 align:middle line:90% at these different levels. 00:05:10.990 --> 00:05:14.130 align:middle line:84% And in some cases, it's easier to explain things 00:05:14.130 --> 00:05:15.710 align:middle line:90% at some levels than others. 00:05:15.710 --> 00:05:19.970 align:middle line:84% And grouping is a great example of this, where you can dream up 00:05:19.970 --> 00:05:22.610 align:middle line:84% these implementation-level explanations of grouping, 00:05:22.610 --> 00:05:26.410 align:middle line:84% in terms of how neurons in the brain might be wired up. 00:05:26.410 --> 00:05:29.130 align:middle line:84% But you can also talk about grouping 00:05:29.130 --> 00:05:31.570 align:middle line:84% as a process of probabilistic inference, whereby 00:05:31.570 --> 00:05:35.890 align:middle line:84% what you perceive to be grouped are actually the latent 00:05:35.890 --> 00:05:39.050 align:middle line:84% causes of these elements in the world, the idea 00:05:39.050 --> 00:05:42.050 align:middle line:84% that there are these two clumps of dots here because maybe 00:05:42.050 --> 00:05:43.650 align:middle line:84% there's one process that generated 00:05:43.650 --> 00:05:48.530 align:middle line:84% one clump and another process that generated another clump. 00:05:48.530 --> 00:05:50.170 align:middle line:84% We saw other examples of grouping, 00:05:50.170 --> 00:05:52.370 align:middle line:84% like good continuation, where you 00:05:52.370 --> 00:05:55.170 align:middle line:84% tend to see things as grouped if they 00:05:55.170 --> 00:05:56.853 align:middle line:90% result in continuous contours. 00:05:56.853 --> 00:05:58.270 align:middle line:84% So you look at an image like this. 00:05:58.270 --> 00:06:01.190 align:middle line:84% And you perceive this as being a circle on top of a square, 00:06:01.190 --> 00:06:03.650 align:middle line:84% even though there are alternative explanations that 00:06:03.650 --> 00:06:05.250 align:middle line:90% are possible. 00:06:05.250 --> 00:06:08.210 align:middle line:84% Grouping by closure-- so this is a whole bunch of elements 00:06:08.210 --> 00:06:14.210 align:middle line:84% that you could be perceiving as a repeating unit 00:06:14.210 --> 00:06:15.150 align:middle line:90% of different sorts. 00:06:15.150 --> 00:06:17.690 align:middle line:84% So each of those three building blocks 00:06:17.690 --> 00:06:22.650 align:middle line:84% could potentially be essentially an explanation of the elements 00:06:22.650 --> 00:06:23.790 align:middle line:90% that you see there. 00:06:23.790 --> 00:06:29.010 align:middle line:84% But almost everybody tends to see these things as circular. 00:06:29.010 --> 00:06:32.810 align:middle line:84% And then we wrapped up by talking about this phenomenon 00:06:32.810 --> 00:06:33.550 align:middle line:90% of popout. 00:06:33.550 --> 00:06:36.570 align:middle line:84% So popout you can think of as the flip side of grouping. 00:06:36.570 --> 00:06:39.050 align:middle line:84% So things that are similar tend to group together. 00:06:39.050 --> 00:06:42.278 align:middle line:84% Things that are different tend to pop out as separate. 00:06:42.278 --> 00:06:44.570 align:middle line:84% So when you look at this, it's just immediately obvious 00:06:44.570 --> 00:06:45.987 align:middle line:84% that there's this one thing that's 00:06:45.987 --> 00:06:47.690 align:middle line:84% kind of different from everything else. 00:06:47.690 --> 00:06:51.220 align:middle line:84% And popout tends to work for simple, 00:06:51.220 --> 00:06:56.400 align:middle line:84% easily computed properties like color, or polarity, 00:06:56.400 --> 00:07:07.200 align:middle line:84% or brightness, or orientation, or size, or motion, or flicker, 00:07:07.200 --> 00:07:11.320 align:middle line:84% or depth-- here, we're using a drop shadow to indicate depth-- 00:07:11.320 --> 00:07:12.640 align:middle line:90% or shape. 00:07:12.640 --> 00:07:17.440 align:middle line:84% So the point is that, as soon as I pop this display up, 00:07:17.440 --> 00:07:19.960 align:middle line:84% everybody immediately knows that there's one thing that's 00:07:19.960 --> 00:07:21.160 align:middle line:90% different from the rest. 00:07:21.160 --> 00:07:23.130 align:middle line:84% So that's the phenomenon of popout. 00:07:23.130 --> 00:07:25.380 align:middle line:84% You don't really have to search around looking for it. 00:07:25.380 --> 00:07:27.280 align:middle line:90% It just is immediately apparent. 00:07:27.280 --> 00:07:30.920 align:middle line:84% And that's not true for all dimensions 00:07:30.920 --> 00:07:34.060 align:middle line:84% by which you could distinguish one element for another. 00:07:34.060 --> 00:07:36.610 align:middle line:84% So I'm going to show you a display. 00:07:36.610 --> 00:07:38.360 align:middle line:84% And I want you to raise your hand when you 00:07:38.360 --> 00:07:42.320 align:middle line:90% see the thing that's different. 00:07:42.320 --> 00:07:42.920 align:middle line:90% OK, good. 00:07:42.920 --> 00:07:43.420 align:middle line:90% You got it. 00:07:43.420 --> 00:07:44.380 align:middle line:90% But it took you a little while. 00:07:44.380 --> 00:07:46.300 align:middle line:84% You have to look around a little bit for it. 00:07:46.300 --> 00:07:48.780 align:middle line:90% So one of the S's is backwards. 00:07:48.780 --> 00:07:51.680 align:middle line:84% And that doesn't really pop out to the same extent. 00:07:51.680 --> 00:07:54.000 align:middle line:84% We're going to do the same thing here. 00:07:54.000 --> 00:07:58.456 align:middle line:84% Raise your hand when you see the thing that's different. 00:07:58.456 --> 00:08:00.940 align:middle line:90% Again, it took a little while. 00:08:00.940 --> 00:08:03.720 align:middle line:84% So this is an example where the odd one out 00:08:03.720 --> 00:08:07.323 align:middle line:84% is distinguished by a conjunction of basic features. 00:08:07.323 --> 00:08:09.740 align:middle line:84% So there's a bunch of-- there's some things that are blue. 00:08:09.740 --> 00:08:10.660 align:middle line:84% There's some things that are green. 00:08:10.660 --> 00:08:11.740 align:middle line:84% There's some things that are X's. 00:08:11.740 --> 00:08:13.115 align:middle line:84% There's some things that are O's. 00:08:13.115 --> 00:08:15.440 align:middle line:84% But there's only one thing that's a blue X. 00:08:15.440 --> 00:08:17.200 align:middle line:84% And that takes a little while to find. 00:08:17.200 --> 00:08:20.080 align:middle line:90% 00:08:20.080 --> 00:08:23.320 align:middle line:84% So element grouping and popout, again, this 00:08:23.320 --> 00:08:25.780 align:middle line:84% is a sloppy, intuitive description. 00:08:25.780 --> 00:08:28.280 align:middle line:84% But they're usually based on simple properties 00:08:28.280 --> 00:08:30.240 align:middle line:84% where we think of simple properties 00:08:30.240 --> 00:08:34.000 align:middle line:84% as plausibly being computed by the machinery of early vision 00:08:34.000 --> 00:08:36.679 align:middle line:84% that could potentially involve comparisons between filter 00:08:36.679 --> 00:08:39.520 align:middle line:90% outputs. 00:08:39.520 --> 00:08:43.679 align:middle line:84% So we talked about how there's a computational level description 00:08:43.679 --> 00:08:46.320 align:middle line:84% of grouping, in terms of probability, 00:08:46.320 --> 00:08:50.110 align:middle line:84% where the Helmholtzian view of these things is that what we see 00:08:50.110 --> 00:08:53.870 align:middle line:84% is our best guess as to what's in the world based on the input 00:08:53.870 --> 00:08:56.630 align:middle line:90% data and our prior experience. 00:08:56.630 --> 00:08:58.990 align:middle line:84% And so grouping by proximity could potentially 00:08:58.990 --> 00:09:01.590 align:middle line:84% be explained in this way, by the notion 00:09:01.590 --> 00:09:03.890 align:middle line:84% that when things are close to each other in the image, 00:09:03.890 --> 00:09:05.390 align:middle line:84% there's a good chance that they were 00:09:05.390 --> 00:09:06.970 align:middle line:84% part of the same object in the world. 00:09:06.970 --> 00:09:11.070 align:middle line:84% And so we have a tendency to see them as part of the same object. 00:09:11.070 --> 00:09:15.290 align:middle line:84% So that's an intuitively appealing view of grouping. 00:09:15.290 --> 00:09:17.350 align:middle line:84% And the question is whether that is 00:09:17.350 --> 00:09:20.270 align:middle line:84% something that can be made a little bit more precise. 00:09:20.270 --> 00:09:24.950 align:middle line:84% And grouping is, in fact, a nice place 00:09:24.950 --> 00:09:28.390 align:middle line:84% where of computational level description has been worked out 00:09:28.390 --> 00:09:29.610 align:middle line:90% in a fair amount of detail. 00:09:29.610 --> 00:09:32.990 align:middle line:84% And the essential idea is that you can actually 00:09:32.990 --> 00:09:37.150 align:middle line:84% measure-- at least for certain simple visual elements 00:09:37.150 --> 00:09:40.572 align:middle line:84% or properties, you can measure the likelihood 00:09:40.572 --> 00:09:42.030 align:middle line:84% that two elements are actually part 00:09:42.030 --> 00:09:45.656 align:middle line:90% of the same thing in the world. 00:09:45.656 --> 00:09:49.190 align:middle line:84% And so here's the essential idea, 00:09:49.190 --> 00:09:53.530 align:middle line:84% that we've got a line segment here, 00:09:53.530 --> 00:09:55.950 align:middle line:84% and then a bunch of other line segments. 00:09:55.950 --> 00:09:59.070 align:middle line:84% And we can ask, just given the nature 00:09:59.070 --> 00:10:02.950 align:middle line:84% of images in the world, what is the likelihood that each 00:10:02.950 --> 00:10:05.830 align:middle line:84% of these other four line segments 00:10:05.830 --> 00:10:09.550 align:middle line:84% is actually part of the same thing as this line segment? 00:10:09.550 --> 00:10:15.590 align:middle line:84% And the intuition here is that the edges of things in the world 00:10:15.590 --> 00:10:17.790 align:middle line:90% tend to be pretty smooth. 00:10:17.790 --> 00:10:21.590 align:middle line:84% And so this line segment is actually pretty likely 00:10:21.590 --> 00:10:23.830 align:middle line:84% to be part of the same thing as this one, 00:10:23.830 --> 00:10:26.052 align:middle line:84% whereas this one is not as likely. 00:10:26.052 --> 00:10:27.510 align:middle line:84% And there's some relationship there 00:10:27.510 --> 00:10:31.370 align:middle line:84% based on the difference in the orientation, maybe the position. 00:10:31.370 --> 00:10:35.330 align:middle line:84% So that's just an intuitive explanation. 00:10:35.330 --> 00:10:38.150 align:middle line:84% You can actually measure this kind of thing and quantify it. 00:10:38.150 --> 00:10:41.690 align:middle line:84% And that's what is shown in this beautiful looking picture here. 00:10:41.690 --> 00:10:45.820 align:middle line:84% So what this picture is showing is the empirical distribution 00:10:45.820 --> 00:10:52.620 align:middle line:84% of edge orientations given a horizontal edge at the center. 00:10:52.620 --> 00:10:56.340 align:middle line:84% So supposing there is a horizontal edge at the center, 00:10:56.340 --> 00:11:00.340 align:middle line:84% we can then ask, what is the probability that there would 00:11:00.340 --> 00:11:05.260 align:middle line:84% be an oriented line segment at each of these possible positions 00:11:05.260 --> 00:11:08.240 align:middle line:84% and at each orientation at each possible position? 00:11:08.240 --> 00:11:10.220 align:middle line:84% And so the way that this is displayed 00:11:10.220 --> 00:11:13.300 align:middle line:84% is that the line segment is color-coded 00:11:13.300 --> 00:11:16.520 align:middle line:84% based on how likely it is to be present, 00:11:16.520 --> 00:11:18.820 align:middle line:84% given, again, that there is a horizontal edge right 00:11:18.820 --> 00:11:21.260 align:middle line:90% here at the center. 00:11:21.260 --> 00:11:22.520 align:middle line:90% So what does this reveal? 00:11:22.520 --> 00:11:24.800 align:middle line:84% So the idea is that this is measured empirically. 00:11:24.800 --> 00:11:27.100 align:middle line:84% So the way that you would compute something 00:11:27.100 --> 00:11:30.200 align:middle line:84% like this is you'd have a big set of images. 00:11:30.200 --> 00:11:34.060 align:middle line:84% You'd have people go through and label the edges of objects. 00:11:34.060 --> 00:11:37.060 align:middle line:84% And each edge gets divided up into these little segments that 00:11:37.060 --> 00:11:39.560 align:middle line:84% are approximately straight and have orientations. 00:11:39.560 --> 00:11:43.410 align:middle line:84% And so from that huge set of actual object 00:11:43.410 --> 00:11:50.190 align:middle line:84% contours in images of the world, you can compute this thing. 00:11:50.190 --> 00:11:52.970 align:middle line:84% And so what this tells you is that if there is an edge 00:11:52.970 --> 00:11:56.170 align:middle line:84% element that is here and that is horizontal, 00:11:56.170 --> 00:11:59.610 align:middle line:84% it's pretty likely that you'll have other edges 00:11:59.610 --> 00:12:03.290 align:middle line:84% out here that are pretty similar in orientation. 00:12:03.290 --> 00:12:05.570 align:middle line:84% So the idea is that edges of objects 00:12:05.570 --> 00:12:07.630 align:middle line:90% tend to be straight and smooth. 00:12:07.630 --> 00:12:14.450 align:middle line:84% And as you move away, there's also some likelihood 00:12:14.450 --> 00:12:17.070 align:middle line:84% that you have edges that are at nearby orientations 00:12:17.070 --> 00:12:19.370 align:middle line:84% because, again, there could be a smooth contour that 00:12:19.370 --> 00:12:21.250 align:middle line:90% passes through those points. 00:12:21.250 --> 00:12:24.450 align:middle line:90% 00:12:24.450 --> 00:12:26.790 align:middle line:84% But if the orientation is very different, 00:12:26.790 --> 00:12:29.830 align:middle line:84% like it would be at these other orientations, 00:12:29.830 --> 00:12:35.830 align:middle line:84% either here or here, the probability is much lower. 00:12:35.830 --> 00:12:39.330 align:middle line:84% So this is a property of the world. 00:12:39.330 --> 00:12:42.770 align:middle line:84% It just says, how likely is it that there 00:12:42.770 --> 00:12:46.630 align:middle line:84% are edges at these different orientations and positions, 00:12:46.630 --> 00:12:48.970 align:middle line:90% given that there's one here? 00:12:48.970 --> 00:12:51.650 align:middle line:90% So it's a property of the world. 00:12:51.650 --> 00:12:57.370 align:middle line:84% Now, the proposal, that computational level explanation 00:12:57.370 --> 00:13:01.250 align:middle line:84% of grouping, is that perceptual grouping, our tendency 00:13:01.250 --> 00:13:04.530 align:middle line:84% to see things as belonging together, 00:13:04.530 --> 00:13:10.330 align:middle line:84% has implicitly internalized these probabilities so that we 00:13:10.330 --> 00:13:16.410 align:middle line:84% would tend to see this line segment and this line segment 00:13:16.410 --> 00:13:20.570 align:middle line:84% as being part of the same thing because, empirically, they 00:13:20.570 --> 00:13:22.610 align:middle line:84% are likely to be part of the same object, 00:13:22.610 --> 00:13:24.530 align:middle line:84% just given the way the world works. 00:13:24.530 --> 00:13:28.530 align:middle line:84% And so the proposal is that, either over evolution 00:13:28.530 --> 00:13:33.570 align:middle line:84% or over development, we have internalized 00:13:33.570 --> 00:13:36.750 align:middle line:84% this probability distribution, this property of the world. 00:13:36.750 --> 00:13:42.600 align:middle line:84% And in general, for grouping and for really most other things 00:13:42.600 --> 00:13:45.160 align:middle line:84% in perception, we still don't really 00:13:45.160 --> 00:13:48.760 align:middle line:84% know the extent to which this is a product of evolution 00:13:48.760 --> 00:13:49.860 align:middle line:90% versus development. 00:13:49.860 --> 00:13:52.580 align:middle line:90% It's probably some mix of both. 00:13:52.580 --> 00:13:55.760 align:middle line:84% It's usually pretty hard to tease those things apart. 00:13:55.760 --> 00:13:59.320 align:middle line:84% But the idea is that, over the course of evolution, 00:13:59.320 --> 00:14:02.460 align:middle line:84% and then as you grow up in the world, 00:14:02.460 --> 00:14:05.440 align:middle line:84% you get lots of experience looking at objects, picking them 00:14:05.440 --> 00:14:06.480 align:middle line:90% up. 00:14:06.480 --> 00:14:08.800 align:middle line:90% You learn what objects are. 00:14:08.800 --> 00:14:13.760 align:middle line:84% And one way or another, you internalize these relationships. 00:14:13.760 --> 00:14:19.520 align:middle line:84% So the prediction then is that, if that is really 00:14:19.520 --> 00:14:22.080 align:middle line:84% how grouping works and why it is the way that it 00:14:22.080 --> 00:14:26.520 align:middle line:84% is, that you ought to see traces of these kinds of distributions 00:14:26.520 --> 00:14:28.880 align:middle line:90% in perceptual grouping. 00:14:28.880 --> 00:14:33.578 align:middle line:84% So we've already seen some of simple examples of this, 00:14:33.578 --> 00:14:35.120 align:middle line:84% grouping via good continuation, where 00:14:35.120 --> 00:14:37.960 align:middle line:84% it's just obvious from looking at these things 00:14:37.960 --> 00:14:39.940 align:middle line:84% that when you see something like this, 00:14:39.940 --> 00:14:42.840 align:middle line:84% you're going to see these two continuous curves rather 00:14:42.840 --> 00:14:46.200 align:middle line:90% than this kind of thing. 00:14:46.200 --> 00:14:50.560 align:middle line:84% This has actually been studied in more detail and more 00:14:50.560 --> 00:14:51.500 align:middle line:90% rigorously. 00:14:51.500 --> 00:14:55.600 align:middle line:84% And this is one pretty well-known method 00:14:55.600 --> 00:14:57.400 align:middle line:90% for looking at this. 00:14:57.400 --> 00:15:01.300 align:middle line:84% So this is the experimental stimulus here. 00:15:01.300 --> 00:15:02.800 align:middle line:90% It's a bunch of Gabors. 00:15:02.800 --> 00:15:05.720 align:middle line:84% Remember, Gabor functions are the product 00:15:05.720 --> 00:15:08.880 align:middle line:90% of a sinusoid and a Gaussian. 00:15:08.880 --> 00:15:11.735 align:middle line:84% So each one of these things is a Gabor function. 00:15:11.735 --> 00:15:13.360 align:middle line:84% They're all the same spatial frequency, 00:15:13.360 --> 00:15:16.800 align:middle line:84% but they're different orientations. 00:15:16.800 --> 00:15:19.520 align:middle line:84% And the idea is that this is a stimulus that 00:15:19.520 --> 00:15:23.040 align:middle line:84% contains a continuous contour amid a whole bunch 00:15:23.040 --> 00:15:24.360 align:middle line:90% of other Gabors. 00:15:24.360 --> 00:15:27.020 align:middle line:84% And the continuous contour is this one here. 00:15:27.020 --> 00:15:29.160 align:middle line:84% And what makes it a continuous contour 00:15:29.160 --> 00:15:33.960 align:middle line:84% is that there is this smooth relationship between position 00:15:33.960 --> 00:15:35.680 align:middle line:90% and orientation. 00:15:35.680 --> 00:15:41.470 align:middle line:84% And what you're supposed to see when you look at this 00:15:41.470 --> 00:15:45.550 align:middle line:84% is that that continuous contour pops out a little bit. 00:15:45.550 --> 00:15:47.550 align:middle line:84% You can look at that and see that there's 00:15:47.550 --> 00:15:54.750 align:middle line:84% this continuous thing amid this field of random-ish Gabors. 00:15:54.750 --> 00:15:59.870 align:middle line:84% And so the argument that is a consequence 00:15:59.870 --> 00:16:01.050 align:middle line:90% of perceptual grouping. 00:16:01.050 --> 00:16:03.070 align:middle line:84% And so you could measure perceptual grouping 00:16:03.070 --> 00:16:05.590 align:middle line:84% by showing people displays like this 00:16:05.590 --> 00:16:07.710 align:middle line:84% and asking them to detect whether there 00:16:07.710 --> 00:16:10.887 align:middle line:84% is a continuous contour in there or not. 00:16:10.887 --> 00:16:12.470 align:middle line:84% And then you could vary the properties 00:16:12.470 --> 00:16:15.990 align:middle line:84% of how that contour is defined and see 00:16:15.990 --> 00:16:18.710 align:middle line:84% whether the grouping occurs when it 00:16:18.710 --> 00:16:23.390 align:middle line:84% is defined by the sorts of properties 00:16:23.390 --> 00:16:24.990 align:middle line:90% that you see in the world. 00:16:24.990 --> 00:16:29.230 align:middle line:84% And so it turns out that this task 00:16:29.230 --> 00:16:33.990 align:middle line:84% of detecting the snake or the contour 00:16:33.990 --> 00:16:36.910 align:middle line:84% is easy if the orientation varies smoothly 00:16:36.910 --> 00:16:38.930 align:middle line:90% from one Gabor to the next. 00:16:38.930 --> 00:16:41.750 align:middle line:84% And it becomes more difficult if it varies less smoothly. 00:16:41.750 --> 00:16:44.990 align:middle line:84% So this is a diagram of how these experimental stimuli are 00:16:44.990 --> 00:16:46.390 align:middle line:90% constructed. 00:16:46.390 --> 00:16:47.530 align:middle line:90% So they're on a grid. 00:16:47.530 --> 00:16:51.990 align:middle line:84% And you can vary the orientation and the position. 00:16:51.990 --> 00:16:56.190 align:middle line:84% And so this is a graph that shows what would happen 00:16:56.190 --> 00:16:57.930 align:middle line:90% when you ran this experiment. 00:16:57.930 --> 00:17:00.710 align:middle line:84% So again, you show people these displays. 00:17:00.710 --> 00:17:03.450 align:middle line:84% Sometimes, they have one of these contours in them. 00:17:03.450 --> 00:17:04.470 align:middle line:90% Sometimes, they don't. 00:17:04.470 --> 00:17:06.430 align:middle line:84% People have to say which is which. 00:17:06.430 --> 00:17:08.990 align:middle line:84% So this is a graph that shows proportion correct 00:17:08.990 --> 00:17:14.310 align:middle line:84% as a function of the angle between adjacent Gabors 00:17:14.310 --> 00:17:16.589 align:middle line:90% on the contour. 00:17:16.589 --> 00:17:20.390 align:middle line:84% And so here's a case where the adjacent Gabor 00:17:20.390 --> 00:17:22.170 align:middle line:90% has changed by kind of a lot. 00:17:22.170 --> 00:17:26.950 align:middle line:84% So it's less smooth right, turns around like that. 00:17:26.950 --> 00:17:30.150 align:middle line:84% And what you're supposed to see when you look at that display 00:17:30.150 --> 00:17:34.560 align:middle line:84% is that it's harder to see the contour, 00:17:34.560 --> 00:17:36.980 align:middle line:84% because the relationships between the elements 00:17:36.980 --> 00:17:40.620 align:middle line:84% are less like the statistical relationships 00:17:40.620 --> 00:17:42.120 align:middle line:90% that you see in the world. 00:17:42.120 --> 00:17:46.340 align:middle line:84% And so what the graph is showing is that performance on this task 00:17:46.340 --> 00:17:49.840 align:middle line:84% gets worse as the angle between the elements goes down. 00:17:49.840 --> 00:17:51.680 align:middle line:84% These are two different participants. 00:17:51.680 --> 00:17:53.472 align:middle line:84% The two curves are two different durations, 00:17:53.472 --> 00:17:56.120 align:middle line:84% which, as the slide says, is not important for our purposes. 00:17:56.120 --> 00:17:58.900 align:middle line:90% 00:17:58.900 --> 00:18:02.580 align:middle line:84% So the task is also harder if the orientation and the position 00:18:02.580 --> 00:18:05.480 align:middle line:84% don't co-vary so as to form a smooth contour. 00:18:05.480 --> 00:18:09.220 align:middle line:84% So here, the Gabors are all arranged such 00:18:09.220 --> 00:18:11.480 align:middle line:84% that their positions form a smooth contour. 00:18:11.480 --> 00:18:15.500 align:middle line:84% But the orientations don't follow that orientation 00:18:15.500 --> 00:18:16.840 align:middle line:90% or the global orientation. 00:18:16.840 --> 00:18:18.420 align:middle line:84% And so then you look at that display. 00:18:18.420 --> 00:18:20.140 align:middle line:84% And it's a lot harder to see that thing. 00:18:20.140 --> 00:18:23.740 align:middle line:90% 00:18:23.740 --> 00:18:25.500 align:middle line:90% This one is also kind of cool. 00:18:25.500 --> 00:18:29.380 align:middle line:84% So here, the way that the contours are constructed 00:18:29.380 --> 00:18:34.220 align:middle line:84% is-- it's like the same as this initial one, where it's 00:18:34.220 --> 00:18:37.220 align:middle line:84% a well-behaved contour, but then you just rotate each Gabor 00:18:37.220 --> 00:18:40.740 align:middle line:90% by 90 degrees. 00:18:40.740 --> 00:18:44.580 align:middle line:84% So now, the orientation is orthogonal to the direction 00:18:44.580 --> 00:18:46.220 align:middle line:90% of the contour. 00:18:46.220 --> 00:18:48.400 align:middle line:90% And that is also hard to detect. 00:18:48.400 --> 00:18:53.060 align:middle line:90% 00:18:53.060 --> 00:18:53.660 align:middle line:90% Yeah? 00:18:53.660 --> 00:18:55.493 align:middle line:84% STUDENT: So in these experiments [INAUDIBLE] 00:18:55.493 --> 00:18:58.580 align:middle line:84% JOSH MCDERMOTT: No, we're cheating. 00:18:58.580 --> 00:19:01.260 align:middle line:84% Yeah, in these class demonstrations, yes. 00:19:01.260 --> 00:19:01.760 align:middle line:90% No. 00:19:01.760 --> 00:19:03.740 align:middle line:84% The experiment involves just seeing this 00:19:03.740 --> 00:19:06.780 align:middle line:84% and saying whether there's one of these contours in it or not. 00:19:06.780 --> 00:19:08.600 align:middle line:84% Yeah, and there's almost surely going 00:19:08.600 --> 00:19:10.100 align:middle line:84% to be some effect of looking at that 00:19:10.100 --> 00:19:14.443 align:middle line:84% and then looking at that, as you probably could detect. 00:19:14.443 --> 00:19:16.360 align:middle line:84% The problem is that if I just showed you this, 00:19:16.360 --> 00:19:18.777 align:middle line:84% you wouldn't believe me that there was something in there. 00:19:18.777 --> 00:19:20.188 align:middle line:84% So I've got to show you that one. 00:19:20.188 --> 00:19:21.480 align:middle line:90% I guess I could've animated it. 00:19:21.480 --> 00:19:24.300 align:middle line:90% 00:19:24.300 --> 00:19:26.700 align:middle line:84% So these are all things that make it harder. 00:19:26.700 --> 00:19:29.580 align:middle line:84% On the other hand, randomizing the phase of the Gabors 00:19:29.580 --> 00:19:31.540 align:middle line:90% has no effect. 00:19:31.540 --> 00:19:33.830 align:middle line:84% So you can still see this thing really easily. 00:19:33.830 --> 00:19:37.850 align:middle line:84% But if you look carefully, the phase is different from Gabor 00:19:37.850 --> 00:19:38.550 align:middle line:90% to Gabor. 00:19:38.550 --> 00:19:43.210 align:middle line:84% So the black and the white stripes don't line up exactly. 00:19:43.210 --> 00:19:45.450 align:middle line:84% Again, we haven't really given you 00:19:45.450 --> 00:19:46.950 align:middle line:84% a clear explanation for why that is, 00:19:46.950 --> 00:19:51.690 align:middle line:84% but it's just part of the phenomenon. 00:19:51.690 --> 00:19:53.730 align:middle line:84% But presumably, you might be able to relate 00:19:53.730 --> 00:19:55.610 align:middle line:84% this to actually measurements that you 00:19:55.610 --> 00:19:59.570 align:middle line:90% would make in actual images. 00:19:59.570 --> 00:20:04.930 align:middle line:84% So big picture here is that this phenomenon of grouping, which 00:20:04.930 --> 00:20:07.972 align:middle line:84% seems like this loose, subjective thing-- you 00:20:07.972 --> 00:20:08.930 align:middle line:90% look at these displays. 00:20:08.930 --> 00:20:11.850 align:middle line:84% And they kind of are organized in a certain way-- 00:20:11.850 --> 00:20:16.390 align:middle line:84% we think is actually rooted in the way the world works. 00:20:16.390 --> 00:20:19.970 align:middle line:84% So there are probabilistic relationships 00:20:19.970 --> 00:20:21.762 align:middle line:84% between the elements that you see in images 00:20:21.762 --> 00:20:23.553 align:middle line:84% and whether or not they're likely to belong 00:20:23.553 --> 00:20:24.570 align:middle line:90% to the same thing. 00:20:24.570 --> 00:20:26.770 align:middle line:84% Your brain has internalized those. 00:20:26.770 --> 00:20:30.450 align:middle line:84% And that causes you to see these structures that 00:20:30.450 --> 00:20:31.930 align:middle line:84% are likely to actually correspond 00:20:31.930 --> 00:20:34.370 align:middle line:90% to objects in the world. 00:20:34.370 --> 00:20:36.170 align:middle line:84% And if the world were set up differently, 00:20:36.170 --> 00:20:38.730 align:middle line:84% with different statistics, then your perceptual grouping 00:20:38.730 --> 00:20:40.910 align:middle line:84% would presumably work differently. 00:20:40.910 --> 00:20:43.410 align:middle line:84% Again, that's a very difficult experiment to do. 00:20:43.410 --> 00:20:45.430 align:middle line:84% But that would be the expectation. 00:20:45.430 --> 00:20:50.890 align:middle line:90% 00:20:50.890 --> 00:20:54.570 align:middle line:84% So as we said, we can talk about these things 00:20:54.570 --> 00:20:55.430 align:middle line:90% at different levels. 00:20:55.430 --> 00:20:59.970 align:middle line:84% So I just gave you a computational level explanation 00:20:59.970 --> 00:21:00.870 align:middle line:90% of grouping. 00:21:00.870 --> 00:21:03.770 align:middle line:84% The question is, what is the problem that's being solved? 00:21:03.770 --> 00:21:06.290 align:middle line:84% And what are the constraints that get imposed to solve it? 00:21:06.290 --> 00:21:08.410 align:middle line:84% The problem consists of an inference 00:21:08.410 --> 00:21:11.450 align:middle line:84% of which things are likely to be part of the same thing. 00:21:11.450 --> 00:21:13.762 align:middle line:84% You employ this constraint of knowing 00:21:13.762 --> 00:21:15.970 align:middle line:84% that the relationships between the elements of things 00:21:15.970 --> 00:21:18.130 align:middle line:84% in the actual world are not random. 00:21:18.130 --> 00:21:19.850 align:middle line:90% They have this structure. 00:21:19.850 --> 00:21:23.650 align:middle line:84% And you internalize that and apply that to make an inference. 00:21:23.650 --> 00:21:25.830 align:middle line:84% So that's the computational level description. 00:21:25.830 --> 00:21:28.490 align:middle line:84% But we can also ask, how would this be implemented 00:21:28.490 --> 00:21:31.840 align:middle line:90% in neural circuitry? 00:21:31.840 --> 00:21:34.920 align:middle line:84% And there's a fairly well-known proposal 00:21:34.920 --> 00:21:37.680 align:middle line:84% for how this particular aspect of grouping 00:21:37.680 --> 00:21:41.060 align:middle line:84% of elements into contours might arise in the brain. 00:21:41.060 --> 00:21:42.920 align:middle line:90% The main idea is really simple. 00:21:42.920 --> 00:21:46.320 align:middle line:84% And it's that you could set the brain up in such a way 00:21:46.320 --> 00:21:51.400 align:middle line:84% that neurons that would be stimulated by the same contour 00:21:51.400 --> 00:21:54.460 align:middle line:84% would be wired up so as to excite each other, 00:21:54.460 --> 00:21:57.840 align:middle line:84% so as to enhance the representation of the contour. 00:21:57.840 --> 00:22:00.740 align:middle line:90% So this is a simple diagram. 00:22:00.740 --> 00:22:02.800 align:middle line:84% So here, each one of these things 00:22:02.800 --> 00:22:06.577 align:middle line:84% now represents a receptive field of a single neuron. 00:22:06.577 --> 00:22:08.660 align:middle line:84% So it's located at a particular position in space. 00:22:08.660 --> 00:22:10.420 align:middle line:84% It's got a particular orientation. 00:22:10.420 --> 00:22:14.080 align:middle line:84% So these are different receptive fields of different neurons. 00:22:14.080 --> 00:22:17.640 align:middle line:84% The solid lines here represent excitatory interactions. 00:22:17.640 --> 00:22:20.440 align:middle line:84% The dashed lines represent inhibitory interactions. 00:22:20.440 --> 00:22:24.840 align:middle line:84% So the notion here is that this receptive field and this one, 00:22:24.840 --> 00:22:27.500 align:middle line:84% or this one, and this one, they would excite each other. 00:22:27.500 --> 00:22:30.150 align:middle line:84% What that means is that, if they're both stimulated-- 00:22:30.150 --> 00:22:31.990 align:middle line:84% so let's say there's some orientation energy 00:22:31.990 --> 00:22:34.310 align:middle line:84% here and some orientation energy here-- 00:22:34.310 --> 00:22:37.570 align:middle line:84% then they boost the responses of each other. 00:22:37.570 --> 00:22:40.830 align:middle line:84% And that ought to cause that set of neurons 00:22:40.830 --> 00:22:44.630 align:middle line:84% to then accentuated in the neural representation. 00:22:44.630 --> 00:22:47.830 align:middle line:84% By comparison, these inhibitory interactions 00:22:47.830 --> 00:22:51.470 align:middle line:90% wouldn't cause that to happen. 00:22:51.470 --> 00:22:55.070 align:middle line:84% So this is an implementation level explanation 00:22:55.070 --> 00:22:56.510 align:middle line:90% of this kind of phenomena. 00:22:56.510 --> 00:23:00.030 align:middle line:90% 00:23:00.030 --> 00:23:00.530 align:middle line:90% All right. 00:23:00.530 --> 00:23:03.930 align:middle line:84% Any questions about grouping or levels of explanation? 00:23:03.930 --> 00:23:09.070 align:middle line:90% 00:23:09.070 --> 00:23:16.270 align:middle line:84% So another place where grouping processes are evident 00:23:16.270 --> 00:23:20.310 align:middle line:84% are cases where we perceive contours 00:23:20.310 --> 00:23:23.630 align:middle line:84% in the absence of local image evidence. 00:23:23.630 --> 00:23:27.310 align:middle line:84% These are also often known as illusory contours. 00:23:27.310 --> 00:23:32.910 align:middle line:84% And these two displays illustrate this. 00:23:32.910 --> 00:23:36.150 align:middle line:84% So in this particular case, when you look at this, 00:23:36.150 --> 00:23:37.670 align:middle line:84% you all probably have the impression 00:23:37.670 --> 00:23:43.870 align:middle line:84% that there's a white strip that's vertically oriented. 00:23:43.870 --> 00:23:46.510 align:middle line:84% And we call this an illusory contour 00:23:46.510 --> 00:23:48.670 align:middle line:84% because there's a sense in which it looks 00:23:48.670 --> 00:23:50.390 align:middle line:90% like there's an edge here. 00:23:50.390 --> 00:23:52.830 align:middle line:84% But actually, there's no contrast 00:23:52.830 --> 00:23:54.530 align:middle line:90% in the image at that location. 00:23:54.530 --> 00:23:56.150 align:middle line:90% The whole thing is just white. 00:23:56.150 --> 00:24:00.130 align:middle line:84% And in fact, the way that you create this illusion, 00:24:00.130 --> 00:24:06.190 align:middle line:84% if you will, is just by taking a bunch of these semicircle things 00:24:06.190 --> 00:24:09.990 align:middle line:84% and laying them down so that the endpoints all align. 00:24:09.990 --> 00:24:13.510 align:middle line:84% So that's one way that you could create this display. 00:24:13.510 --> 00:24:17.750 align:middle line:84% Another way you could create the display is like the way 00:24:17.750 --> 00:24:21.030 align:middle line:84% that we see it, which is you take a whole bunch of ovals 00:24:21.030 --> 00:24:22.227 align:middle line:90% that are concentric. 00:24:22.227 --> 00:24:23.810 align:middle line:84% You put a bunch here and a bunch here. 00:24:23.810 --> 00:24:26.540 align:middle line:84% And then you put a white strip on top of them. 00:24:26.540 --> 00:24:29.100 align:middle line:84% And so that's really the explanation 00:24:29.100 --> 00:24:32.040 align:middle line:84% that you are implicitly inferring when you look at this. 00:24:32.040 --> 00:24:35.400 align:middle line:90% 00:24:35.400 --> 00:24:39.260 align:middle line:84% So you see this edge here, even though there's not 00:24:39.260 --> 00:24:42.380 align:middle line:90% actual contrast in the image. 00:24:42.380 --> 00:24:45.300 align:middle line:84% Now, there's this other situation over here, 00:24:45.300 --> 00:24:47.460 align:middle line:84% where you're also perceiving an edge 00:24:47.460 --> 00:24:51.500 align:middle line:84% in the absence of explicit local information in the image 00:24:51.500 --> 00:24:53.940 align:middle line:84% because, in this case, it actually 00:24:53.940 --> 00:24:56.540 align:middle line:84% looks like this light gray rectangle extends 00:24:56.540 --> 00:25:00.820 align:middle line:90% beyond or behind the dark ovals. 00:25:00.820 --> 00:25:05.120 align:middle line:84% So again, in this region here, there's no contrast here. 00:25:05.120 --> 00:25:07.380 align:middle line:84% But you have the sense that there's 00:25:07.380 --> 00:25:10.800 align:middle line:90% a thing extending behind them. 00:25:10.800 --> 00:25:14.140 align:middle line:84% So the difference between this and this-- 00:25:14.140 --> 00:25:17.580 align:middle line:84% well, there's a couple differences. 00:25:17.580 --> 00:25:22.300 align:middle line:84% One is that, in this case, the thing that you're perceiving 00:25:22.300 --> 00:25:25.300 align:middle line:90% is in front of the other stuff. 00:25:25.300 --> 00:25:27.108 align:middle line:84% In this case, the thing that you're 00:25:27.108 --> 00:25:28.900 align:middle line:84% perceiving in the absence of local evidence 00:25:28.900 --> 00:25:31.340 align:middle line:90% is behind the other stuff. 00:25:31.340 --> 00:25:33.900 align:middle line:84% But the other thing, the other difference, 00:25:33.900 --> 00:25:38.580 align:middle line:84% which I think is deep and cool, is that, in this case, 00:25:38.580 --> 00:25:41.400 align:middle line:90% you actually see the edge. 00:25:41.400 --> 00:25:45.740 align:middle line:84% It actually looks like there is an edge there that you can see. 00:25:45.740 --> 00:25:47.620 align:middle line:84% In this case, like no one's going 00:25:47.620 --> 00:25:49.700 align:middle line:84% to say that they actually see the edge there. 00:25:49.700 --> 00:25:51.860 align:middle line:84% You sense that the edge is there. 00:25:51.860 --> 00:25:54.000 align:middle line:84% You perceive that it's there at some level. 00:25:54.000 --> 00:25:55.280 align:middle line:90% But you don't actually see it. 00:25:55.280 --> 00:25:57.880 align:middle line:90% 00:25:57.880 --> 00:26:02.180 align:middle line:84% So these are often referred to as completion, 00:26:02.180 --> 00:26:05.623 align:middle line:84% in the sense that there's local evidence here and here. 00:26:05.623 --> 00:26:07.540 align:middle line:84% And then there's something that gets completed 00:26:07.540 --> 00:26:09.580 align:middle line:84% that causes you to see something in the absence 00:26:09.580 --> 00:26:10.740 align:middle line:90% of local evidence. 00:26:10.740 --> 00:26:14.620 align:middle line:84% This one is often referred to as modal completion. 00:26:14.620 --> 00:26:18.300 align:middle line:84% This is typically referred to as amodal completion. 00:26:18.300 --> 00:26:24.550 align:middle line:84% So modal-- I think it's Latin or Greek that refers to sensation. 00:26:24.550 --> 00:26:26.370 align:middle line:84% And so the idea is that you're actually 00:26:26.370 --> 00:26:29.290 align:middle line:90% seeing the contour here. 00:26:29.290 --> 00:26:31.110 align:middle line:84% Here, you're not actually seeing it. 00:26:31.110 --> 00:26:32.570 align:middle line:90% It's amodal. 00:26:32.570 --> 00:26:34.970 align:middle line:84% But there's still some sense in which you're representing 00:26:34.970 --> 00:26:36.970 align:middle line:90% a completed contour. 00:26:36.970 --> 00:26:43.210 align:middle line:84% And so both of these things happen all the time. 00:26:43.210 --> 00:26:46.850 align:middle line:84% Again, most of the time, you're just not aware of it. 00:26:46.850 --> 00:26:50.130 align:middle line:84% But sometimes, there are these funny coincidences 00:26:50.130 --> 00:26:55.890 align:middle line:84% in images that reveal the completion processes at work. 00:26:55.890 --> 00:27:00.330 align:middle line:84% So this is a funny one, where there's 00:27:00.330 --> 00:27:04.210 align:middle line:84% a cylinder or something that's placed in front of this person's 00:27:04.210 --> 00:27:04.710 align:middle line:90% hands. 00:27:04.710 --> 00:27:07.250 align:middle line:84% And it makes it look like they have a really long finger. 00:27:07.250 --> 00:27:12.410 align:middle line:84% Here, this is a case where there's 00:27:12.410 --> 00:27:14.370 align:middle line:90% a bunch of objects on the table. 00:27:14.370 --> 00:27:18.590 align:middle line:84% They're coincidentally covered up by this purple thing. 00:27:18.590 --> 00:27:20.850 align:middle line:84% And so it looks like there's a continuous table 00:27:20.850 --> 00:27:25.170 align:middle line:90% behind the purple thing. 00:27:25.170 --> 00:27:27.650 align:middle line:90% This is a really long cow. 00:27:27.650 --> 00:27:33.170 align:middle line:84% So it's these sorts of examples that really drive home 00:27:33.170 --> 00:27:35.450 align:middle line:90% that this is a real thing. 00:27:35.450 --> 00:27:37.790 align:middle line:84% On one level, it sounds very subjective, 00:27:37.790 --> 00:27:41.690 align:middle line:84% this idea that you're sensing this contour. 00:27:41.690 --> 00:27:45.250 align:middle line:84% But it really does affect the way that you perceive shapes. 00:27:45.250 --> 00:27:49.410 align:middle line:84% And because occlusion-- so occlusion refers to the fact 00:27:49.410 --> 00:27:51.530 align:middle line:84% that something can be in front of another 00:27:51.530 --> 00:27:57.130 align:middle line:84% and block your view of it and part of it in some region. 00:27:57.130 --> 00:28:00.330 align:middle line:84% And so objects in vision occlude each other. 00:28:00.330 --> 00:28:02.690 align:middle line:84% So you have one object that's in front of the other. 00:28:02.690 --> 00:28:04.230 align:middle line:90% It blocks your view temporarily. 00:28:04.230 --> 00:28:07.227 align:middle line:84% And so this business of amodal completion 00:28:07.227 --> 00:28:08.810 align:middle line:84% happens all the time because occlusion 00:28:08.810 --> 00:28:11.670 align:middle line:90% is just everywhere in vision. 00:28:11.670 --> 00:28:17.610 align:middle line:90% 00:28:17.610 --> 00:28:19.160 align:middle line:84% You might ask, well, what determines 00:28:19.160 --> 00:28:22.040 align:middle line:90% whether this completion happens? 00:28:22.040 --> 00:28:25.160 align:middle line:84% And one important property is what 00:28:25.160 --> 00:28:29.800 align:middle line:84% is depicted here, which is referred to as relatability. 00:28:29.800 --> 00:28:33.040 align:middle line:84% And this was popularized by two people, Kellman and Shipley. 00:28:33.040 --> 00:28:36.220 align:middle line:84% And so the idea is that when you look at this one, 00:28:36.220 --> 00:28:39.080 align:middle line:84% it looks like the two gray things are connected and are 00:28:39.080 --> 00:28:41.975 align:middle line:84% a single object, whereas, in this one, it really doesn't. 00:28:41.975 --> 00:28:44.600 align:middle line:84% It really looks like there's two different gray rectangles that 00:28:44.600 --> 00:28:47.440 align:middle line:84% are just sticking under the same black blob. 00:28:47.440 --> 00:28:50.920 align:middle line:84% And so the intuition is that the contours that 00:28:50.920 --> 00:28:53.020 align:middle line:84% are on either side of the occluder, 00:28:53.020 --> 00:28:55.440 align:middle line:84% they should be aiming towards the same place 00:28:55.440 --> 00:28:57.240 align:middle line:84% behind the occluder and shouldn't 00:28:57.240 --> 00:28:59.160 align:middle line:90% be at too steep of an angle. 00:28:59.160 --> 00:29:03.460 align:middle line:84% So that's what this term relatability really refers to. 00:29:03.460 --> 00:29:08.200 align:middle line:84% It's whether the contours can be connected in a smooth way. 00:29:08.200 --> 00:29:11.100 align:middle line:84% So here, we have an example of something that is relatable. 00:29:11.100 --> 00:29:14.160 align:middle line:84% So you can imagine this smooth curve 00:29:14.160 --> 00:29:16.200 align:middle line:84% joining the things behind an occluder. 00:29:16.200 --> 00:29:17.940 align:middle line:90% And these are unrelatable. 00:29:17.940 --> 00:29:20.300 align:middle line:84% There's an inflection point in the curve. 00:29:20.300 --> 00:29:23.580 align:middle line:90% 00:29:23.580 --> 00:29:27.000 align:middle line:84% And so again, this is something that's 00:29:27.000 --> 00:29:30.120 align:middle line:84% subjectively evident that is probably 00:29:30.120 --> 00:29:32.140 align:middle line:84% due to the statistics of the world. 00:29:32.140 --> 00:29:34.200 align:middle line:84% It's probably a function of which 00:29:34.200 --> 00:29:37.360 align:middle line:84% edges are empirically likely to actually correspond 00:29:37.360 --> 00:29:39.215 align:middle line:90% to objects in the world. 00:29:39.215 --> 00:29:40.840 align:middle line:84% And the idea is that this kind of thing 00:29:40.840 --> 00:29:42.557 align:middle line:84% is more probable than this kind of thing. 00:29:42.557 --> 00:29:44.140 align:middle line:84% It's not that this could never happen. 00:29:44.140 --> 00:29:46.080 align:middle line:90% It obviously could happen. 00:29:46.080 --> 00:29:47.420 align:middle line:90% But it's just less likely. 00:29:47.420 --> 00:29:49.640 align:middle line:84% And you're making a probabilistic inference 00:29:49.640 --> 00:29:51.340 align:middle line:84% when you look at one of these displays. 00:29:51.340 --> 00:29:55.340 align:middle line:90% 00:29:55.340 --> 00:30:00.800 align:middle line:84% So I want to conclude this lecture 00:30:00.800 --> 00:30:06.240 align:middle line:84% by talking about some evidence for how 00:30:06.240 --> 00:30:08.160 align:middle line:84% these illusory contours might be represented 00:30:08.160 --> 00:30:10.080 align:middle line:90% in the visual system. 00:30:10.080 --> 00:30:12.720 align:middle line:84% And one really interesting finding 00:30:12.720 --> 00:30:16.230 align:middle line:84% is that you can find individual neurons 00:30:16.230 --> 00:30:19.010 align:middle line:84% in an area of the visual system, V2, 00:30:19.010 --> 00:30:22.270 align:middle line:84% which is one stage beyond V1, that seem to respond 00:30:22.270 --> 00:30:23.990 align:middle line:90% to these illusory contours. 00:30:23.990 --> 00:30:25.870 align:middle line:84% So these were discovered by Rüdiger von der 00:30:25.870 --> 00:30:29.110 align:middle line:84% Heydt. he was a vision scientist who worked at Johns Hopkins 00:30:29.110 --> 00:30:30.430 align:middle line:90% for many years. 00:30:30.430 --> 00:30:34.910 align:middle line:84% And the very first paper was from back in 1984. 00:30:34.910 --> 00:30:38.150 align:middle line:84% And so the idea is to measure responses 00:30:38.150 --> 00:30:42.470 align:middle line:84% to stimuli that have an illusory contour, so cases 00:30:42.470 --> 00:30:45.310 align:middle line:84% where you see something there where there's no local image 00:30:45.310 --> 00:30:46.230 align:middle line:90% evidence. 00:30:46.230 --> 00:30:51.030 align:middle line:84% And so the oval here represents the receptive field of a neuron. 00:30:51.030 --> 00:30:54.110 align:middle line:84% And the rest of the stuff here is the stimulus. 00:30:54.110 --> 00:30:56.910 align:middle line:84% And so this is just a regular line 00:30:56.910 --> 00:30:59.830 align:middle line:84% that gets swept through the receptive field. 00:30:59.830 --> 00:31:02.630 align:middle line:84% And this is what's called a raster plot. 00:31:02.630 --> 00:31:05.990 align:middle line:84% So each row represents one trial where you sweep the thing 00:31:05.990 --> 00:31:07.450 align:middle line:90% through the receptive field. 00:31:07.450 --> 00:31:10.490 align:middle line:84% And the dots represent spikes that are fired by the neuron. 00:31:10.490 --> 00:31:12.790 align:middle line:84% And so the point is that there's just a lot of dots 00:31:12.790 --> 00:31:15.935 align:middle line:84% there, which is what you would expect because there's 00:31:15.935 --> 00:31:16.810 align:middle line:90% an edge in the image. 00:31:16.810 --> 00:31:18.518 align:middle line:84% You sweep it through the receptive field. 00:31:18.518 --> 00:31:19.650 align:middle line:90% And you get a response. 00:31:19.650 --> 00:31:22.750 align:middle line:90% 00:31:22.750 --> 00:31:26.638 align:middle line:84% And down here, D is just what happens if there's 00:31:26.638 --> 00:31:27.930 align:middle line:90% nothing in the receptive field. 00:31:27.930 --> 00:31:29.570 align:middle line:84% So it's just spontaneous activity. 00:31:29.570 --> 00:31:32.128 align:middle line:84% So you don't have a whole lot of spikes there. 00:31:32.128 --> 00:31:33.670 align:middle line:84% And so the two interesting conditions 00:31:33.670 --> 00:31:35.198 align:middle line:84% are the ones that are shown here. 00:31:35.198 --> 00:31:36.490 align:middle line:90% So this is a really old figure. 00:31:36.490 --> 00:31:37.830 align:middle line:90% And it's a little hard to see. 00:31:37.830 --> 00:31:40.190 align:middle line:90% But this is an illusory contour. 00:31:40.190 --> 00:31:44.690 align:middle line:84% So we've got a black line here and a black line here. 00:31:44.690 --> 00:31:46.730 align:middle line:84% And then this region here is also black. 00:31:46.730 --> 00:31:49.910 align:middle line:84% So there's no image contrast here. 00:31:49.910 --> 00:31:52.310 align:middle line:84% But if you looked at this, especially 00:31:52.310 --> 00:31:55.130 align:middle line:84% if this receptive field diagram wasn't there, 00:31:55.130 --> 00:31:58.110 align:middle line:84% you would perceive there to be a black line extending across 00:31:58.110 --> 00:32:00.510 align:middle line:90% here. 00:32:00.510 --> 00:32:03.490 align:middle line:84% This stimulus, maybe if you're sitting all the way in the back, 00:32:03.490 --> 00:32:05.010 align:middle line:84% this actually looks exactly the same as this. 00:32:05.010 --> 00:32:07.070 align:middle line:84% But it's actually different in an important way. 00:32:07.070 --> 00:32:09.790 align:middle line:84% And that is that there's a thin white line happening 00:32:09.790 --> 00:32:12.510 align:middle line:90% here and here. 00:32:12.510 --> 00:32:15.380 align:middle line:84% And so this is an analogous example 00:32:15.380 --> 00:32:17.740 align:middle line:84% shown at better resolution, so you can get the idea. 00:32:17.740 --> 00:32:20.360 align:middle line:84% So this is a stimulus where you would see an illusory contour. 00:32:20.360 --> 00:32:24.420 align:middle line:84% It looks like there's a white rectangle extending across 00:32:24.420 --> 00:32:25.900 align:middle line:90% between the two circles. 00:32:25.900 --> 00:32:30.060 align:middle line:84% Here, we've drawn these thin white lines, here and here. 00:32:30.060 --> 00:32:32.880 align:middle line:84% And that kills the percept of the illusory contour. 00:32:32.880 --> 00:32:34.540 align:middle line:84% If you look at that, you might actually 00:32:34.540 --> 00:32:36.340 align:middle line:84% see that there's a white rectangle that's 00:32:36.340 --> 00:32:39.420 align:middle line:84% behind some other white thing, like maybe you're 00:32:39.420 --> 00:32:42.780 align:middle line:84% viewing something through two round windows. 00:32:42.780 --> 00:32:45.900 align:middle line:84% But you don't see the illusory contour here that you see here. 00:32:45.900 --> 00:32:50.020 align:middle line:84% And so the point here is that this is a very small change 00:32:50.020 --> 00:32:52.620 align:middle line:84% to the image that makes a big change 00:32:52.620 --> 00:32:55.760 align:middle line:90% to the subjective perception. 00:32:55.760 --> 00:32:59.700 align:middle line:84% And so von der Heydt used this to actually probe for responses 00:32:59.700 --> 00:33:00.840 align:middle line:90% to the illusory contour. 00:33:00.840 --> 00:33:03.060 align:middle line:84% So the idea is that you have these two stimuli that 00:33:03.060 --> 00:33:06.860 align:middle line:84% are almost the same, except for this really small difference. 00:33:06.860 --> 00:33:08.720 align:middle line:84% And in this case, you see the contour. 00:33:08.720 --> 00:33:09.880 align:middle line:90% In this case, you don't. 00:33:09.880 --> 00:33:12.110 align:middle line:84% And what he found was that, in this case, 00:33:12.110 --> 00:33:15.830 align:middle line:84% you get a big response, so lots of dots on the raster here, 00:33:15.830 --> 00:33:17.090 align:middle line:90% action potentials. 00:33:17.090 --> 00:33:19.090 align:middle line:90% And in this case, you don't. 00:33:19.090 --> 00:33:23.730 align:middle line:84% So it's evidence that these neurons are responding 00:33:23.730 --> 00:33:26.130 align:middle line:90% to these illusory contours. 00:33:26.130 --> 00:33:29.993 align:middle line:84% And importantly, so this is interesting both just 00:33:29.993 --> 00:33:31.410 align:middle line:84% because it's interesting, but also 00:33:31.410 --> 00:33:36.450 align:middle line:84% because this is something that apparently differentiates 00:33:36.450 --> 00:33:37.790 align:middle line:90% V2 from V1. 00:33:37.790 --> 00:33:40.770 align:middle line:84% So you don't see this, or not very much of it, 00:33:40.770 --> 00:33:44.050 align:middle line:84% in primary visual cortex, something that emerges 00:33:44.050 --> 00:33:46.810 align:middle line:90% a little bit downstream. 00:33:46.810 --> 00:33:48.030 align:middle line:90% Here's just another example. 00:33:48.030 --> 00:33:51.410 align:middle line:84% This is a different type of illusory contour, 00:33:51.410 --> 00:33:55.010 align:middle line:84% where, if you have these gratings that are slightly 00:33:55.010 --> 00:33:57.850 align:middle line:84% offset, you see an edge here, even 00:33:57.850 --> 00:34:00.130 align:middle line:84% though all of the actual orientation 00:34:00.130 --> 00:34:05.650 align:middle line:84% energy in the stimulus is perpendicular to that. 00:34:05.650 --> 00:34:08.170 align:middle line:84% And so this is showing you that this 00:34:08.170 --> 00:34:10.130 align:middle line:84% is-- so this is a neuron that's being 00:34:10.130 --> 00:34:11.710 align:middle line:90% stimulated with these lines. 00:34:11.710 --> 00:34:15.030 align:middle line:84% And so when you have a line that's nearly horizontal, 00:34:15.030 --> 00:34:16.469 align:middle line:90% you get almost no response. 00:34:16.469 --> 00:34:18.469 align:middle line:84% When you have a line that's nearly vertical, 00:34:18.469 --> 00:34:20.170 align:middle line:90% you get big responses. 00:34:20.170 --> 00:34:24.650 align:middle line:84% And here, we have illusory contours 00:34:24.650 --> 00:34:26.443 align:middle line:90% with those same orientations. 00:34:26.443 --> 00:34:28.610 align:middle line:84% And the illusory contour, when it's almost vertical, 00:34:28.610 --> 00:34:30.929 align:middle line:84% gives you a big response, even though all 00:34:30.929 --> 00:34:34.330 align:middle line:84% of the lines that are in the stimulus are mostly horizontal. 00:34:34.330 --> 00:34:37.690 align:middle line:84% So the neuron's really responding to the percept 00:34:37.690 --> 00:34:40.370 align:middle line:84% of the-- or it's responding to some correlate of that illusory 00:34:40.370 --> 00:34:41.889 align:middle line:90% contour-- 00:34:41.889 --> 00:34:43.830 align:middle line:84% again, happening in an area of V2. 00:34:43.830 --> 00:34:46.650 align:middle line:90% 00:34:46.650 --> 00:34:47.150 align:middle line:90% All right. 00:34:47.150 --> 00:34:49.050 align:middle line:84% So it's natural to wonder like, well, 00:34:49.050 --> 00:34:53.290 align:middle line:84% what are the mechanisms that give rise 00:34:53.290 --> 00:34:55.889 align:middle line:90% to these illusory contours. 00:34:55.889 --> 00:34:58.850 align:middle line:84% And I would say we don't entirely know. 00:34:58.850 --> 00:35:04.290 align:middle line:84% There have been proposals for models of illusory contours 00:35:04.290 --> 00:35:07.770 align:middle line:84% based on local low-level features. 00:35:07.770 --> 00:35:09.960 align:middle line:84% Here's a diagram for one such paper. 00:35:09.960 --> 00:35:11.920 align:middle line:84% And so the idea is that you could 00:35:11.920 --> 00:35:15.480 align:middle line:84% have local receptive fields that would detect 00:35:15.480 --> 00:35:20.480 align:middle line:84% the elements of these displays-- the line endings, for instance. 00:35:20.480 --> 00:35:23.200 align:middle line:84% And you could imagine that you would 00:35:23.200 --> 00:35:25.080 align:middle line:84% try to create a neuron that would respond 00:35:25.080 --> 00:35:28.480 align:middle line:84% to these illusory contours in a way that's analogous to the way 00:35:28.480 --> 00:35:33.240 align:middle line:84% that we think orientation selectivity is produced 00:35:33.240 --> 00:35:36.335 align:middle line:84% by a bunch of center surround receptive field inputs. 00:35:36.335 --> 00:35:38.960 align:middle line:84% Remember, we talked about that model of orientation selectivity 00:35:38.960 --> 00:35:39.960 align:middle line:84% where you have these center surround 00:35:39.960 --> 00:35:41.440 align:middle line:90% receptive fields in the LGN. 00:35:41.440 --> 00:35:43.907 align:middle line:84% And they converge on some downstream neuron. 00:35:43.907 --> 00:35:46.240 align:middle line:84% And if you have the right combination of receptive field 00:35:46.240 --> 00:35:48.780 align:middle line:84% inputs, you get an orientation selective receptive field. 00:35:48.780 --> 00:35:51.880 align:middle line:84% So you can imagine the same principle applying here, 00:35:51.880 --> 00:35:54.000 align:middle line:84% that you would have a bunch of receptive fields 00:35:54.000 --> 00:35:59.420 align:middle line:84% that would be tuned to the individual elements of, say, 00:35:59.420 --> 00:36:00.620 align:middle line:90% a display like this. 00:36:00.620 --> 00:36:03.860 align:middle line:84% And then if those provided input to a downstream neuron, 00:36:03.860 --> 00:36:06.040 align:middle line:84% you would then get something that would respond 00:36:06.040 --> 00:36:08.660 align:middle line:90% to the illusory contour. 00:36:08.660 --> 00:36:12.240 align:middle line:84% And that might be part of the story. 00:36:12.240 --> 00:36:14.240 align:middle line:84% But one really interesting property 00:36:14.240 --> 00:36:17.160 align:middle line:84% of these illusory contours is that they're 00:36:17.160 --> 00:36:21.000 align:middle line:84% very sensitive to manipulations that you wouldn't really 00:36:21.000 --> 00:36:23.940 align:middle line:84% think would affect the low-level responses that much. 00:36:23.940 --> 00:36:25.940 align:middle line:84% And so in particular, in this case, 00:36:25.940 --> 00:36:29.280 align:middle line:84% you actually don't really see a very strong illusory contour, 00:36:29.280 --> 00:36:31.200 align:middle line:90% whereas in this case, you do. 00:36:31.200 --> 00:36:35.360 align:middle line:84% And so you have roughly the same number of line endings 00:36:35.360 --> 00:36:38.180 align:middle line:84% and the same amount of alignment in these two different cases. 00:36:38.180 --> 00:36:41.880 align:middle line:84% But in this case, the fact that the line segments 00:36:41.880 --> 00:36:47.960 align:middle line:84% are randomly oriented makes the illusory contour a lot stronger. 00:36:47.960 --> 00:36:53.400 align:middle line:84% And intuitively, one way to think about this 00:36:53.400 --> 00:36:57.240 align:middle line:84% is that there are other ways to actually explain 00:36:57.240 --> 00:37:00.120 align:middle line:84% the fact that these things are all lined up 00:37:00.120 --> 00:37:01.600 align:middle line:84% that don't necessarily require you 00:37:01.600 --> 00:37:03.640 align:middle line:84% to postulate that there is some object here 00:37:03.640 --> 00:37:04.720 align:middle line:90% that's occluding them. 00:37:04.720 --> 00:37:07.190 align:middle line:84% So in fact, these are all exactly the same length. 00:37:07.190 --> 00:37:08.690 align:middle line:90% And they're all lined up. 00:37:08.690 --> 00:37:10.390 align:middle line:84% And so you might imagine that there 00:37:10.390 --> 00:37:13.870 align:middle line:84% was some other process that caused all these things to be 00:37:13.870 --> 00:37:16.170 align:middle line:84% lined up in this way, whereas, in this case, 00:37:16.170 --> 00:37:18.310 align:middle line:84% really the most likely explanation is probably 00:37:18.310 --> 00:37:21.070 align:middle line:84% just that there's an occluding surface that 00:37:21.070 --> 00:37:22.350 align:middle line:90% happens to be white. 00:37:22.350 --> 00:37:25.070 align:middle line:84% So it's really not so obvious how 00:37:25.070 --> 00:37:28.310 align:middle line:84% to account for these kinds of differences 00:37:28.310 --> 00:37:33.610 align:middle line:84% in terms of these low-level circuit-style models. 00:37:33.610 --> 00:37:37.190 align:middle line:90% 00:37:37.190 --> 00:37:42.930 align:middle line:84% And as I've noted here, so this model here is probably-- 00:37:42.930 --> 00:37:45.813 align:middle line:84% I don't know-- 20, 25, 30 years old, something like that. 00:37:45.813 --> 00:37:47.230 align:middle line:84% This is the kind of problem that I 00:37:47.230 --> 00:37:51.070 align:middle line:84% think is ripe for revisiting in the current era. 00:37:51.070 --> 00:37:53.990 align:middle line:84% So we now have very different modeling tools than were 00:37:53.990 --> 00:37:57.990 align:middle line:84% available back when there was a lot of interest in this, 00:37:57.990 --> 00:38:00.070 align:middle line:90% in the '80s and early '90s. 00:38:00.070 --> 00:38:06.430 align:middle line:84% And so yeah, one of you could work on this, potentially. 00:38:06.430 --> 00:38:06.970 align:middle line:90% All right. 00:38:06.970 --> 00:38:10.830 align:middle line:90% So we think that these are-- 00:38:10.830 --> 00:38:12.930 align:middle line:84% at some level, these illusory contours, 00:38:12.930 --> 00:38:14.157 align:middle line:90% they're illusions, right? 00:38:14.157 --> 00:38:15.990 align:middle line:84% We call them illusions because you're seeing 00:38:15.990 --> 00:38:17.250 align:middle line:90% something that's not there. 00:38:17.250 --> 00:38:20.370 align:middle line:84% But as is the case with most illusions, 00:38:20.370 --> 00:38:23.470 align:middle line:84% we actually think that these represent engineering solutions 00:38:23.470 --> 00:38:27.030 align:middle line:90% that help us see in the world. 00:38:27.030 --> 00:38:29.190 align:middle line:84% And probably because of phenomena like 00:38:29.190 --> 00:38:31.910 align:middle line:84% this, that it just happens all the time, that there 00:38:31.910 --> 00:38:34.550 align:middle line:84% will be the edge of an object that just 00:38:34.550 --> 00:38:36.910 align:middle line:84% because the object happens to be pretty 00:38:36.910 --> 00:38:40.590 align:middle line:84% similar color to the background, at least in some place, 00:38:40.590 --> 00:38:42.810 align:middle line:84% there's not much local evidence for the edge. 00:38:42.810 --> 00:38:45.990 align:middle line:90% And so here, you have a log. 00:38:45.990 --> 00:38:50.450 align:middle line:84% And it's pretty similar in gray level to the background. 00:38:50.450 --> 00:38:52.550 align:middle line:84% And so locally, there's just not a whole lot 00:38:52.550 --> 00:38:55.470 align:middle line:90% of orientation energy. 00:38:55.470 --> 00:38:58.110 align:middle line:84% But you can still see the edge pretty clearly, 00:38:58.110 --> 00:39:01.110 align:middle line:84% probably because, well, there's some high contrast stuff here 00:39:01.110 --> 00:39:02.690 align:middle line:84% and some high contrast stuff here. 00:39:02.690 --> 00:39:07.900 align:middle line:84% And so you get these mechanisms that contour completion. 00:39:07.900 --> 00:39:12.060 align:middle line:84% So this sort of completion is probably happening all the time 00:39:12.060 --> 00:39:16.100 align:middle line:84% without us really thinking a whole lot about it. 00:39:16.100 --> 00:39:20.600 align:middle line:84% What questions do you have about modal or amodal completion? 00:39:20.600 --> 00:39:29.980 align:middle line:90% 00:39:29.980 --> 00:39:30.753 align:middle line:90% Yeah? 00:39:30.753 --> 00:39:32.420 align:middle line:84% STUDENT: This is just a clarifying point 00:39:32.420 --> 00:39:35.540 align:middle line:90% about the von der Heydt paper. 00:39:35.540 --> 00:39:36.420 align:middle line:90% JOSH MCDERMOTT: Yeah? 00:39:36.420 --> 00:39:39.020 align:middle line:84% STUDENT: When it's talking about some neurons in V2 00:39:39.020 --> 00:39:43.060 align:middle line:84% responding to illusory contours, is it responding similarly 00:39:43.060 --> 00:39:45.640 align:middle line:84% to illusory contours as it does to just a line? 00:39:45.640 --> 00:39:47.385 align:middle line:90% Or is it [INAUDIBLE] 00:39:47.385 --> 00:39:48.760 align:middle line:84% JOSH MCDERMOTT: Well, apparently. 00:39:48.760 --> 00:39:51.400 align:middle line:84% So that's kind of what this shows, right? 00:39:51.400 --> 00:39:53.840 align:middle line:84% So this is the response to just a line. 00:39:53.840 --> 00:39:56.100 align:middle line:84% And this is the illusory contour. 00:39:56.100 --> 00:39:58.760 align:middle line:84% I mean, it's not shown in these graphs. 00:39:58.760 --> 00:40:01.843 align:middle line:84% But this would be a neuron that is orientation-selective. 00:40:01.843 --> 00:40:04.260 align:middle line:84% So you change the orientation of the line and the response 00:40:04.260 --> 00:40:05.023 align:middle line:90% would go down. 00:40:05.023 --> 00:40:06.940 align:middle line:84% And you'd see the same orientation selectivity 00:40:06.940 --> 00:40:08.500 align:middle line:90% to the illusory contour. 00:40:08.500 --> 00:40:12.180 align:middle line:84% So yeah, it does seem like it's like the same code 00:40:12.180 --> 00:40:16.300 align:middle line:84% for a regular contour that has some image contrast as 00:40:16.300 --> 00:40:18.020 align:middle line:90% for an illusory contour. 00:40:18.020 --> 00:40:20.620 align:middle line:84% And the next example, I think, also shows that maybe more 00:40:20.620 --> 00:40:24.980 align:middle line:84% clearly, where here you can see that this is orientation 00:40:24.980 --> 00:40:26.960 align:middle line:90% selective to the visible line. 00:40:26.960 --> 00:40:29.740 align:middle line:84% So it responds to this orientation, not this one. 00:40:29.740 --> 00:40:32.020 align:middle line:84% And here, you see comparable selectivity 00:40:32.020 --> 00:40:35.220 align:middle line:90% to the illusory contour. 00:40:35.220 --> 00:40:35.980 align:middle line:90% Yeah. 00:40:35.980 --> 00:40:37.540 align:middle line:90% Yeah? 00:40:37.540 --> 00:40:40.420 align:middle line:90% STUDENT: [INAUDIBLE] 00:40:40.420 --> 00:40:45.213 align:middle line:90% 00:40:45.213 --> 00:40:47.380 align:middle line:84% JOSH MCDERMOTT: What do you mean how are we supposed 00:40:47.380 --> 00:40:48.580 align:middle line:90% to interpret the two halves? 00:40:48.580 --> 00:40:52.540 align:middle line:84% STUDENT: Like the two circles [INAUDIBLE] 00:40:52.540 --> 00:40:55.380 align:middle line:90% 00:40:55.380 --> 00:40:59.460 align:middle line:84% JOSH MCDERMOTT: So this is a stimulus, OK? 00:40:59.460 --> 00:41:02.870 align:middle line:84% And the stimulus is just-- it just looks like this. 00:41:02.870 --> 00:41:04.690 align:middle line:90% So it's just a line. 00:41:04.690 --> 00:41:06.890 align:middle line:84% And this is being presented to-- essentially, 00:41:06.890 --> 00:41:07.950 align:middle line:90% presented to a neuron. 00:41:07.950 --> 00:41:11.290 align:middle line:84% So the neuron's receptive field would be here. 00:41:11.290 --> 00:41:12.910 align:middle line:90% And then these are raster plots. 00:41:12.910 --> 00:41:17.118 align:middle line:84% So a whole bunch of these are presentations of this stimulus. 00:41:17.118 --> 00:41:19.410 align:middle line:84% And the point is that you don't see very many dots here 00:41:19.410 --> 00:41:21.830 align:middle line:90% because the dots are spikes. 00:41:21.830 --> 00:41:24.310 align:middle line:84% And so the neuron is just not responding to the stimulus. 00:41:24.310 --> 00:41:27.430 align:middle line:84% These correspond to presentation of this stimulus. 00:41:27.430 --> 00:41:30.330 align:middle line:84% And there's a whole bunch of dots here because it's 00:41:30.330 --> 00:41:31.310 align:middle line:90% orientation-selective. 00:41:31.310 --> 00:41:34.790 align:middle line:84% And it's responding to that line and not to that one. 00:41:34.790 --> 00:41:36.810 align:middle line:84% And this is the same thing, only presenting 00:41:36.810 --> 00:41:40.350 align:middle line:84% these illusory contours at these two orientations. 00:41:40.350 --> 00:41:41.830 align:middle line:84% So these are called raster plots. 00:41:41.830 --> 00:41:44.510 align:middle line:84% These are very common in old school neurophysiology papers. 00:41:44.510 --> 00:41:49.090 align:middle line:84% So again, each row represents a trial-- so a presentation 00:41:49.090 --> 00:41:50.410 align:middle line:90% of a stimulus. 00:41:50.410 --> 00:41:52.330 align:middle line:90% This is time. 00:41:52.330 --> 00:41:54.730 align:middle line:84% And each little dot here is a spike. 00:41:54.730 --> 00:41:58.520 align:middle line:90% 00:41:58.520 --> 00:42:05.760 align:middle line:84% OK, so one last related concept is this notion figure 00:42:05.760 --> 00:42:06.740 align:middle line:90% and ground. 00:42:06.740 --> 00:42:11.120 align:middle line:84% So empirically, we often interpret images 00:42:11.120 --> 00:42:13.540 align:middle line:84% as consisting of figure and ground. 00:42:13.540 --> 00:42:15.520 align:middle line:84% And this is a really famous illusion 00:42:15.520 --> 00:42:20.280 align:middle line:84% that illustrates this point where, when you look at this, 00:42:20.280 --> 00:42:25.298 align:middle line:84% sometimes you see a vase and sometimes you see two faces. 00:42:25.298 --> 00:42:26.840 align:middle line:84% And you tend to see one or the other. 00:42:26.840 --> 00:42:27.860 align:middle line:90% And it's bistable. 00:42:27.860 --> 00:42:31.227 align:middle line:84% So if you stare at this, sometimes you'll see two faces. 00:42:31.227 --> 00:42:33.560 align:middle line:84% And then it'll switch and it'll look mostly like a vase. 00:42:33.560 --> 00:42:34.580 align:middle line:90% It goes back and forth. 00:42:34.580 --> 00:42:36.580 align:middle line:84% And the point is that, at any moment in time, 00:42:36.580 --> 00:42:39.440 align:middle line:84% you tend to see one or the other. 00:42:39.440 --> 00:42:41.600 align:middle line:90% So that edge is always there. 00:42:41.600 --> 00:42:47.060 align:middle line:84% But the edge is either owned by one side or the other. 00:42:47.060 --> 00:42:50.140 align:middle line:84% So since the edge is caused by one side or the other, 00:42:50.140 --> 00:42:51.880 align:middle line:84% and that side has caused the figure, 00:42:51.880 --> 00:42:55.260 align:middle line:84% the other side is called the ground. 00:42:55.260 --> 00:42:58.760 align:middle line:84% So if this one doesn't work on you, look at this. 00:42:58.760 --> 00:43:05.640 align:middle line:84% So this is one of these displays that makes your brain hurt. 00:43:05.640 --> 00:43:10.560 align:middle line:84% And what is happening here is that these little crosses, 00:43:10.560 --> 00:43:12.400 align:middle line:90% for brief periods of time, move. 00:43:12.400 --> 00:43:16.400 align:middle line:84% And that causes them to look like objects and figure. 00:43:16.400 --> 00:43:18.200 align:middle line:90% And then they become static. 00:43:18.200 --> 00:43:19.920 align:middle line:90% And something else moves. 00:43:19.920 --> 00:43:21.920 align:middle line:84% And so the what's figure and what's ground 00:43:21.920 --> 00:43:24.800 align:middle line:84% is changing from moment to moment. 00:43:24.800 --> 00:43:27.520 align:middle line:84% But you'll initially see these white crosses, 00:43:27.520 --> 00:43:29.020 align:middle line:84% and then see these black crosses. 00:43:29.020 --> 00:43:32.560 align:middle line:84% And so it's just supposed to make the point 00:43:32.560 --> 00:43:37.800 align:middle line:84% that you tend to assign edges as belonging 00:43:37.800 --> 00:43:40.240 align:middle line:90% to one side or the other. 00:43:40.240 --> 00:43:42.680 align:middle line:84% And the side that belongs to them, 00:43:42.680 --> 00:43:44.460 align:middle line:90% that owns them is "figure." 00:43:44.460 --> 00:43:48.320 align:middle line:90% 00:43:48.320 --> 00:43:52.340 align:middle line:84% So what's figure and what's ground is inherently ambiguous. 00:43:52.340 --> 00:43:58.430 align:middle line:84% So it's another example of an ill-posed problem. 00:43:58.430 --> 00:44:00.355 align:middle line:84% But the visual system relies on many cues. 00:44:00.355 --> 00:44:02.230 align:middle line:84% And again, these are things that were studied 00:44:02.230 --> 00:44:04.090 align:middle line:90% by the gestalt psychologists. 00:44:04.090 --> 00:44:08.310 align:middle line:84% So for instance, things that are symmetric 00:44:08.310 --> 00:44:10.670 align:middle line:90% tend to be seen as figure. 00:44:10.670 --> 00:44:13.670 align:middle line:84% Things that are smaller tend to be seen as figure. 00:44:13.670 --> 00:44:17.190 align:middle line:84% So the dark orange things here look like figure. 00:44:17.190 --> 00:44:21.470 align:middle line:84% And the light orange things look like ground. 00:44:21.470 --> 00:44:23.710 align:middle line:84% Parallel lines tend to be interpreted 00:44:23.710 --> 00:44:26.270 align:middle line:90% as the borders of figure. 00:44:26.270 --> 00:44:31.950 align:middle line:84% So the things that are parallel here tend to be seen as figure. 00:44:31.950 --> 00:44:34.630 align:middle line:84% And so as with perceptual grouping, 00:44:34.630 --> 00:44:37.190 align:middle line:84% this was initially just studied with this very 00:44:37.190 --> 00:44:39.890 align:middle line:84% empirical, subjective, intuitive methodology. 00:44:39.890 --> 00:44:42.790 align:middle line:84% People would make these displays and look at them. 00:44:42.790 --> 00:44:45.410 align:middle line:84% And oftentimes, the effects were just obvious. 00:44:45.410 --> 00:44:48.880 align:middle line:90% 00:44:48.880 --> 00:44:52.070 align:middle line:90% And intuitively, we think-- 00:44:52.070 --> 00:44:56.550 align:middle line:84% people thought for a long time that these relationships 00:44:56.550 --> 00:45:00.190 align:middle line:84% are the way they are because of the way the world is. 00:45:00.190 --> 00:45:03.570 align:middle line:84% But it was hard to make that really argument precise. 00:45:03.570 --> 00:45:06.390 align:middle line:90% 00:45:06.390 --> 00:45:08.070 align:middle line:84% But nowadays, we have the ability 00:45:08.070 --> 00:45:10.630 align:middle line:84% to make these large scale measurements 00:45:10.630 --> 00:45:13.170 align:middle line:84% of images and measure probabilities of things. 00:45:13.170 --> 00:45:14.950 align:middle line:84% And so we can actually give teeth 00:45:14.950 --> 00:45:17.150 align:middle line:84% to these kinds of computational-level 00:45:17.150 --> 00:45:18.750 align:middle line:90% explanations. 00:45:18.750 --> 00:45:21.190 align:middle line:84% And so this is just an example of a paper that 00:45:21.190 --> 00:45:24.053 align:middle line:84% provides evidence for the idea that the cues to figure 00:45:24.053 --> 00:45:26.470 align:middle line:84% and ground are actually rooted in statistics of the world. 00:45:26.470 --> 00:45:29.790 align:middle line:84% And so the way that this works is-- 00:45:29.790 --> 00:45:31.990 align:middle line:84% this was a group at Berkeley that 00:45:31.990 --> 00:45:35.670 align:middle line:84% took lots of images of the world, asked 00:45:35.670 --> 00:45:39.830 align:middle line:84% human observers to label all of the edges of objects 00:45:39.830 --> 00:45:42.490 align:middle line:90% in the image. 00:45:42.490 --> 00:45:44.790 align:middle line:84% And they probably had a computer vision algorithm 00:45:44.790 --> 00:45:46.070 align:middle line:90% that would help them out. 00:45:46.070 --> 00:45:47.830 align:middle line:90% You find all of these edges. 00:45:47.830 --> 00:45:53.740 align:middle line:84% And then you get humans to label the sections of edges 00:45:53.740 --> 00:45:56.980 align:middle line:90% that correspond to a figure. 00:45:56.980 --> 00:46:01.040 align:middle line:84% So in this particular case, there's a person. 00:46:01.040 --> 00:46:03.400 align:middle line:84% And so that boundary is the boundary of the person. 00:46:03.400 --> 00:46:07.660 align:middle line:84% And so you have a way of labeling 00:46:07.660 --> 00:46:10.180 align:middle line:84% which side of the contour is the figure 00:46:10.180 --> 00:46:11.440 align:middle line:90% and which side is the ground. 00:46:11.440 --> 00:46:12.720 align:middle line:90% Here's another one. 00:46:12.720 --> 00:46:14.415 align:middle line:84% This is like the edge of the cow. 00:46:14.415 --> 00:46:16.040 align:middle line:84% And so you have a way of labeling that. 00:46:16.040 --> 00:46:18.140 align:middle line:84% So you do that for all of the different edges 00:46:18.140 --> 00:46:19.460 align:middle line:90% here in the image. 00:46:19.460 --> 00:46:25.100 align:middle line:84% So for every point on every edge in a big set of images, 00:46:25.100 --> 00:46:29.900 align:middle line:84% you have what, in theory, is ground truth of which side 00:46:29.900 --> 00:46:33.740 align:middle line:84% of the edge is actually the figure, where ground truth here 00:46:33.740 --> 00:46:40.060 align:middle line:84% is derived using the entirety of a bunch of human visual systems. 00:46:40.060 --> 00:46:43.140 align:middle line:90% 00:46:43.140 --> 00:46:48.500 align:middle line:84% So the question is whether the kinds 00:46:48.500 --> 00:46:53.860 align:middle line:84% of cues that people classically talked about out actually 00:46:53.860 --> 00:46:56.200 align:middle line:84% fall out of these kinds of probabilities. 00:46:56.200 --> 00:46:58.220 align:middle line:84% So remember, we talked about the idea 00:46:58.220 --> 00:47:00.540 align:middle line:84% that things that are smaller are more 00:47:00.540 --> 00:47:04.100 align:middle line:90% likely to be seen as figure. 00:47:04.100 --> 00:47:07.780 align:middle line:84% And so you can take a point on one of these contours. 00:47:07.780 --> 00:47:11.720 align:middle line:84% You have a little circle that you draw around the point. 00:47:11.720 --> 00:47:13.760 align:middle line:84% There's a contour that runs through that. 00:47:13.760 --> 00:47:18.900 align:middle line:84% And you can compute the area on either side of the contour. 00:47:18.900 --> 00:47:21.920 align:middle line:84% And so in this particular case, the F stands for figure, 00:47:21.920 --> 00:47:23.360 align:middle line:90% the G stands for ground. 00:47:23.360 --> 00:47:25.500 align:middle line:84% And the figure side of the contour 00:47:25.500 --> 00:47:28.040 align:middle line:84% has less area than the ground side of the contour. 00:47:28.040 --> 00:47:33.500 align:middle line:84% And you can quantify that as the ratio between the two areas. 00:47:33.500 --> 00:47:37.540 align:middle line:84% Similarly, we often think that things that are convex 00:47:37.540 --> 00:47:38.960 align:middle line:90% are likely to be figure. 00:47:38.960 --> 00:47:39.960 align:middle line:90% So again, here's a case. 00:47:39.960 --> 00:47:41.405 align:middle line:84% This is like the foot of the bear. 00:47:41.405 --> 00:47:42.280 align:middle line:90% So that's the figure. 00:47:42.280 --> 00:47:43.030 align:middle line:90% That's the ground. 00:47:43.030 --> 00:47:44.440 align:middle line:90% This is the heel of the bear. 00:47:44.440 --> 00:47:46.180 align:middle line:90% So this is a convex contour. 00:47:46.180 --> 00:47:49.900 align:middle line:84% And so you can measure the convexity. 00:47:49.900 --> 00:47:52.250 align:middle line:84% And so the question is whether those cues 00:47:52.250 --> 00:47:55.650 align:middle line:84% are predictive of whether something actually is 00:47:55.650 --> 00:47:59.090 align:middle line:90% figure or actually is ground. 00:47:59.090 --> 00:48:02.110 align:middle line:84% And so this is the result of the analysis. 00:48:02.110 --> 00:48:06.450 align:middle line:84% So these are histograms that are plotting 00:48:06.450 --> 00:48:11.430 align:middle line:84% the frequency of occurrence of these different measures. 00:48:11.430 --> 00:48:14.250 align:middle line:84% This is the size measure and the convexity measure 00:48:14.250 --> 00:48:16.530 align:middle line:84% for the figure side of the figure 00:48:16.530 --> 00:48:18.370 align:middle line:84% and the ground side of the figure. 00:48:18.370 --> 00:48:22.770 align:middle line:84% And so the point is that, for this size cue, 00:48:22.770 --> 00:48:24.450 align:middle line:84% the figure side of the figure tends 00:48:24.450 --> 00:48:27.210 align:middle line:84% to be smaller than the ground side of the figure. 00:48:27.210 --> 00:48:31.450 align:middle line:84% The red curve is to the left of the blue curve. 00:48:31.450 --> 00:48:34.170 align:middle line:84% And for the convexity cue, the figure 00:48:34.170 --> 00:48:36.710 align:middle line:84% tends to be more convex than the ground side of the curve. 00:48:36.710 --> 00:48:38.730 align:middle line:90% So it's to the right. 00:48:38.730 --> 00:48:42.690 align:middle line:90% 00:48:42.690 --> 00:48:43.310 align:middle line:90% OK. 00:48:43.310 --> 00:48:48.570 align:middle line:84% So this is essentially saying that these so-called cues that 00:48:48.570 --> 00:48:52.330 align:middle line:84% were often proposed by gestalt psychologists, 00:48:52.330 --> 00:48:56.950 align:middle line:84% that they actually are borne out by the statistics of the world. 00:48:56.950 --> 00:49:01.890 align:middle line:84% So the things that are figure do tend to be smaller and do 00:49:01.890 --> 00:49:03.510 align:middle line:90% tend to be more convex. 00:49:03.510 --> 00:49:06.970 align:middle line:90% 00:49:06.970 --> 00:49:07.470 align:middle line:90% All right. 00:49:07.470 --> 00:49:10.490 align:middle line:84% So the question is, these local cues, 00:49:10.490 --> 00:49:13.490 align:middle line:84% if we actually measure these cues, 00:49:13.490 --> 00:49:19.250 align:middle line:84% can we actually use them to classify which side of a contour 00:49:19.250 --> 00:49:20.290 align:middle line:90% is the figure? 00:49:20.290 --> 00:49:22.530 align:middle line:84% And so that's what this graph is showing. 00:49:22.530 --> 00:49:25.462 align:middle line:84% And so there were three cues here that were measured. 00:49:25.462 --> 00:49:26.670 align:middle line:90% I told you about two of them. 00:49:26.670 --> 00:49:30.810 align:middle line:84% There's this other one, L, which we can skip over 00:49:30.810 --> 00:49:32.030 align:middle line:90% in the interest of time. 00:49:32.030 --> 00:49:34.450 align:middle line:84% The point is there are three cues. 00:49:34.450 --> 00:49:36.970 align:middle line:84% And so you can build a classifier 00:49:36.970 --> 00:49:40.090 align:middle line:84% that has access to either all three, or pairs of them, 00:49:40.090 --> 00:49:41.490 align:middle line:90% or one of them. 00:49:41.490 --> 00:49:44.930 align:middle line:84% And then you can vary the radius of the window over which 00:49:44.930 --> 00:49:46.730 align:middle line:90% those cues are measured. 00:49:46.730 --> 00:49:51.700 align:middle line:84% And the graph here is plotting classification performance, 00:49:51.700 --> 00:49:55.000 align:middle line:84% so how accurately you can predict 00:49:55.000 --> 00:49:59.520 align:middle line:84% which side of the contour is the figure based on individual cues 00:49:59.520 --> 00:50:01.080 align:middle line:90% or combinations of cues. 00:50:01.080 --> 00:50:05.855 align:middle line:84% And what you can see is that, as you make the window bigger, 00:50:05.855 --> 00:50:06.980 align:middle line:90% you do a little bit better. 00:50:06.980 --> 00:50:12.480 align:middle line:84% And then you reach some radius and things level off. 00:50:12.480 --> 00:50:15.800 align:middle line:84% And unsurprisingly, when you have all three cues-- that's 00:50:15.800 --> 00:50:18.900 align:middle line:84% the red curve-- you do better than when you have fewer cues. 00:50:18.900 --> 00:50:25.400 align:middle line:84% But what's interesting is that performance seems 00:50:25.400 --> 00:50:29.440 align:middle line:90% to level out at about 75%. 00:50:29.440 --> 00:50:34.600 align:middle line:84% So that's well above chance, but it's not perfect. 00:50:34.600 --> 00:50:39.480 align:middle line:84% So let's just assume that humans viewing these images 00:50:39.480 --> 00:50:41.740 align:middle line:90% would be close to 100% correct. 00:50:41.740 --> 00:50:44.960 align:middle line:84% And in fact, I mean, humans were used to label the images. 00:50:44.960 --> 00:50:47.680 align:middle line:84% So in some sense, they have to be 100% correct. 00:50:47.680 --> 00:50:53.080 align:middle line:84% So this is showing that these local cues allow 00:50:53.080 --> 00:50:55.760 align:middle line:84% you to predict which side is figure, but certainly 00:50:55.760 --> 00:51:00.480 align:middle line:84% not at 100%, only at 75% So what do you think might account 00:51:00.480 --> 00:51:01.980 align:middle line:90% for the performance gap? 00:51:01.980 --> 00:51:04.965 align:middle line:90% 00:51:04.965 --> 00:51:07.080 align:middle line:90% Mm-hmm? 00:51:07.080 --> 00:51:08.880 align:middle line:84% STUDENT: Global context of the scenes. 00:51:08.880 --> 00:51:11.960 align:middle line:84% JOSH MCDERMOTT: Yeah, so the global context of a scene, 00:51:11.960 --> 00:51:15.960 align:middle line:84% that could be it-- so your knowledge of the world, that you 00:51:15.960 --> 00:51:18.760 align:middle line:84% recognize that something is a cow and cows 00:51:18.760 --> 00:51:20.660 align:middle line:84% are going to be figure, stuff like that. 00:51:20.660 --> 00:51:23.360 align:middle line:90% So yeah, that could be it. 00:51:23.360 --> 00:51:26.320 align:middle line:90% Anything else? 00:51:26.320 --> 00:51:27.080 align:middle line:90% Yeah? 00:51:27.080 --> 00:51:29.400 align:middle line:84% STUDENT: Yeah, so is this just looking at one circle at a time 00:51:29.400 --> 00:51:31.620 align:middle line:84% and then being like, I think this is the figure versus this? 00:51:31.620 --> 00:51:32.495 align:middle line:90% JOSH MCDERMOTT: Yeah. 00:51:32.495 --> 00:51:38.840 align:middle line:84% STUDENT: Well, it's probably the fact that [INAUDIBLE] Well, then 00:51:38.840 --> 00:51:42.440 align:middle line:84% leaves one figure, even if I'm classifying 25% of them 00:51:42.440 --> 00:51:43.590 align:middle line:90% correctly. 00:51:43.590 --> 00:51:45.830 align:middle line:84% JOSH MCDERMOTT: So you're saying that if you just 00:51:45.830 --> 00:51:48.250 align:middle line:84% combined information from all of those little windows, 00:51:48.250 --> 00:51:48.750 align:middle line:90% you could-- 00:51:48.750 --> 00:51:49.010 align:middle line:90% STUDENT: Yeah. 00:51:49.010 --> 00:51:50.450 align:middle line:84% JOSH MCDERMOTT: Yeah, so that could also be true. 00:51:50.450 --> 00:51:52.550 align:middle line:84% Yeah, and that's a slightly different version 00:51:52.550 --> 00:51:53.610 align:middle line:90% of the global context. 00:51:53.610 --> 00:51:56.550 align:middle line:84% It's not necessarily taking into account knowledge of the world, 00:51:56.550 --> 00:51:59.643 align:middle line:90% but just combining cues, yeah. 00:51:59.643 --> 00:52:00.810 align:middle line:90% So yeah, so that's possible. 00:52:00.810 --> 00:52:03.280 align:middle line:90% Yeah? 00:52:03.280 --> 00:52:05.030 align:middle line:84% STUDENT: I might be restating some things, 00:52:05.030 --> 00:52:08.272 align:middle line:84% but maybe it's related to grouping? 00:52:08.272 --> 00:52:10.230 align:middle line:84% JOSH MCDERMOTT: What do you mean by that, maybe 00:52:10.230 --> 00:52:11.870 align:middle line:90% it's related to grouping? 00:52:11.870 --> 00:52:18.130 align:middle line:84% STUDENT: So if a stimulus seems to appear to be an edge. 00:52:18.130 --> 00:52:22.050 align:middle line:84% And other things also seem to point to something bigger. 00:52:22.050 --> 00:52:25.575 align:middle line:84% They could be put together and considered one thing. 00:52:25.575 --> 00:52:26.950 align:middle line:84% JOSH MCDERMOTT: Yeah, yeah, yeah. 00:52:26.950 --> 00:52:27.530 align:middle line:90% That's true. 00:52:27.530 --> 00:52:29.750 align:middle line:84% So the strategy of combining cues 00:52:29.750 --> 00:52:33.050 align:middle line:84% could definitely be related to grouping via good continuation. 00:52:33.050 --> 00:52:34.930 align:middle line:84% Yeah, those are all good suggestions. 00:52:34.930 --> 00:52:37.190 align:middle line:84% Another possibility is that these three cues are not 00:52:37.190 --> 00:52:38.270 align:middle line:90% exhaustive. 00:52:38.270 --> 00:52:42.030 align:middle line:84% These are just three things that human scientists thought up. 00:52:42.030 --> 00:52:44.680 align:middle line:84% It could be other stuff that you could measure. 00:52:44.680 --> 00:52:48.340 align:middle line:84% But at any rate, it's one approach 00:52:48.340 --> 00:52:52.380 align:middle line:84% to showing how these perceptual phenomena can be related 00:52:52.380 --> 00:52:55.000 align:middle line:84% to the nature of the world-- in particular, 00:52:55.000 --> 00:52:58.220 align:middle line:84% like the probabilities of things being one 00:52:58.220 --> 00:53:02.050 align:middle line:90% way or the other in the world. 00:53:02.050 --> 00:53:04.160 align:middle line:84% Any questions about figure or ground? 00:53:04.160 --> 00:53:07.980 align:middle line:90% 00:53:07.980 --> 00:53:08.500 align:middle line:90% OK. 00:53:08.500 --> 00:53:10.580 align:middle line:84% Last thing I want to tell you about-- 00:53:10.580 --> 00:53:12.180 align:middle line:90% this is also pretty cool. 00:53:12.180 --> 00:53:16.260 align:middle line:84% So there's also a pretty intriguing neural correlate 00:53:16.260 --> 00:53:18.300 align:middle line:84% of these figure ground relationships 00:53:18.300 --> 00:53:20.660 align:middle line:90% that also is found in area V2. 00:53:20.660 --> 00:53:23.487 align:middle line:84% And this was also from the lab of Rüdiger von der 00:53:23.487 --> 00:53:25.820 align:middle line:84% Heydt, who was very interested in these mid-level visual 00:53:25.820 --> 00:53:27.580 align:middle line:90% phenomena. 00:53:27.580 --> 00:53:29.580 align:middle line:84% And these are the existence of neurons 00:53:29.580 --> 00:53:33.120 align:middle line:84% that seem to encode the direction of border ownership. 00:53:33.120 --> 00:53:36.460 align:middle line:84% And so what we mean by border ownership 00:53:36.460 --> 00:53:39.320 align:middle line:84% is it's really synonymous with figure and ground. 00:53:39.320 --> 00:53:41.500 align:middle line:84% So the idea is that you have some contour. 00:53:41.500 --> 00:53:43.160 align:middle line:90% And one side of that is figure. 00:53:43.160 --> 00:53:44.940 align:middle line:84% The other side of that is ground. 00:53:44.940 --> 00:53:48.180 align:middle line:84% The side that is figure, we say, owns the border, 00:53:48.180 --> 00:53:50.300 align:middle line:84% in the sense that that's the object that 00:53:50.300 --> 00:53:55.460 align:middle line:90% actually caused that contour. 00:53:55.460 --> 00:53:57.362 align:middle line:84% And so here's essentially the phenomenon. 00:53:57.362 --> 00:53:58.820 align:middle line:84% So this is the same kind of display 00:53:58.820 --> 00:53:59.945 align:middle line:90% that we've been looking at. 00:53:59.945 --> 00:54:00.920 align:middle line:90% These are raster plots. 00:54:00.920 --> 00:54:04.740 align:middle line:84% So each row is a trial, so a presentation of the stimulus. 00:54:04.740 --> 00:54:07.020 align:middle line:90% The x-axis is time. 00:54:07.020 --> 00:54:11.060 align:middle line:84% Each little line segment here is an action potential. 00:54:11.060 --> 00:54:12.440 align:middle line:90% This is the visual display. 00:54:12.440 --> 00:54:14.420 align:middle line:84% And you can see the receptive field of the cell 00:54:14.420 --> 00:54:18.388 align:middle line:84% as that little oval that is centered on part of the display. 00:54:18.388 --> 00:54:20.180 align:middle line:84% And so the idea is that the receptive field 00:54:20.180 --> 00:54:24.480 align:middle line:90% is positioned on an edge there. 00:54:24.480 --> 00:54:27.980 align:middle line:90% And we've got four stimuli here. 00:54:27.980 --> 00:54:32.720 align:middle line:84% And what is varied is both the sign of the contrast-- so here, 00:54:32.720 --> 00:54:35.180 align:middle line:84% we have a light square on a dark background. 00:54:35.180 --> 00:54:37.080 align:middle line:84% Here, we have a square in the same position. 00:54:37.080 --> 00:54:40.770 align:middle line:84% But now, the square is dark gray and the background is light. 00:54:40.770 --> 00:54:43.930 align:middle line:84% And so now, the stimulus that's inside the receptive field 00:54:43.930 --> 00:54:46.770 align:middle line:84% has actually flipped the sign of the contrast. 00:54:46.770 --> 00:54:50.290 align:middle line:84% Here, we've again got a square on a background. 00:54:50.290 --> 00:54:52.690 align:middle line:84% And the stimulus that's inside the receptive field 00:54:52.690 --> 00:54:55.650 align:middle line:90% is identical to the one up here. 00:54:55.650 --> 00:54:57.830 align:middle line:84% So on the left side, we have light. 00:54:57.830 --> 00:54:59.810 align:middle line:84% And on the right side, we have dark. 00:54:59.810 --> 00:55:04.650 align:middle line:84% But now, the square is on the opposite side. 00:55:04.650 --> 00:55:07.010 align:middle line:84% So the idea with these two displays 00:55:07.010 --> 00:55:09.370 align:middle line:84% is that, at this particular point on the contour, 00:55:09.370 --> 00:55:13.050 align:middle line:84% in this case, the left side owns the contour. 00:55:13.050 --> 00:55:16.290 align:middle line:84% And in this case, the right side owns the contour. 00:55:16.290 --> 00:55:18.430 align:middle line:90% The local stimulus is the same. 00:55:18.430 --> 00:55:22.090 align:middle line:84% So this is another example of this local ambiguity. 00:55:22.090 --> 00:55:24.850 align:middle line:84% But the rest of the image gives you a bunch of information 00:55:24.850 --> 00:55:27.410 align:middle line:84% indicating that either this side is figure 00:55:27.410 --> 00:55:30.970 align:middle line:90% or this side is figure. 00:55:30.970 --> 00:55:33.610 align:middle line:84% And then here's the alternative version 00:55:33.610 --> 00:55:35.710 align:middle line:84% where, again, the square is on the right side, 00:55:35.710 --> 00:55:37.350 align:middle line:90% but the contrast is flipped. 00:55:37.350 --> 00:55:41.810 align:middle line:84% So now, the local stimulus is the same as the one up here. 00:55:41.810 --> 00:55:42.310 align:middle line:90% All right. 00:55:42.310 --> 00:55:47.650 align:middle line:84% And so what you see from this is that this neuron is giving you 00:55:47.650 --> 00:55:50.690 align:middle line:84% a big response whenever the figure is 00:55:50.690 --> 00:55:53.130 align:middle line:90% on the left side of the contour. 00:55:53.130 --> 00:55:55.650 align:middle line:84% So it's when the left side owns the contour. 00:55:55.650 --> 00:55:57.530 align:middle line:84% And it gives you a small response 00:55:57.530 --> 00:56:00.010 align:middle line:84% whenever the figure is on the other side. 00:56:00.010 --> 00:56:05.910 align:middle line:84% So this seems to be coding the direction of border ownership. 00:56:05.910 --> 00:56:09.890 align:middle line:84% It's a representation of which side of the contour 00:56:09.890 --> 00:56:11.450 align:middle line:90% is the figure. 00:56:11.450 --> 00:56:17.490 align:middle line:84% So this doesn't tell us how the neuron figures out one side 00:56:17.490 --> 00:56:18.790 align:middle line:90% is figure and one is the other. 00:56:18.790 --> 00:56:21.570 align:middle line:84% It just says that this is evidence that quantity 00:56:21.570 --> 00:56:25.130 align:middle line:90% is being represented. 00:56:25.130 --> 00:56:25.710 align:middle line:90% Yeah? 00:56:25.710 --> 00:56:28.770 align:middle line:84% STUDENT: What is the y-axis on these graphs? 00:56:28.770 --> 00:56:31.370 align:middle line:84% JOSH MCDERMOTT: These are different trials, yeah. 00:56:31.370 --> 00:56:34.330 align:middle line:90% 00:56:34.330 --> 00:56:37.070 align:middle line:84% Yeah, so each one is just a different stimulus presentation. 00:56:37.070 --> 00:56:40.600 align:middle line:90% 00:56:40.600 --> 00:56:41.320 align:middle line:90% OK. 00:56:41.320 --> 00:56:47.160 align:middle line:84% So big picture here, talking about mid-level vision, 00:56:47.160 --> 00:56:50.600 align:middle line:84% these interesting aspects of perceptual organization, 00:56:50.600 --> 00:56:53.620 align:middle line:84% they're rooted in probabilities in the world. 00:56:53.620 --> 00:56:57.920 align:middle line:84% And we have various little bits of hints of how 00:56:57.920 --> 00:56:59.560 align:middle line:90% they're coded in the brain. 00:56:59.560 --> 00:57:02.280 align:middle line:84% We're very far from having complete explanations 00:57:02.280 --> 00:57:04.840 align:middle line:84% of these things, in terms of neural circuitry. 00:57:04.840 --> 00:57:08.720 align:middle line:84% But in terms of illusory contours and border ownership, 00:57:08.720 --> 00:57:12.400 align:middle line:84% there are some indications that V2 is representing some 00:57:12.400 --> 00:57:14.800 align:middle line:90% of the relevant quantities. 00:57:14.800 --> 00:57:18.440 align:middle line:84% So summary of what we've talked about-- 00:57:18.440 --> 00:57:20.020 align:middle line:84% at the very start of the lecture, 00:57:20.020 --> 00:57:21.937 align:middle line:84% we talked about how the visual system consists 00:57:21.937 --> 00:57:25.200 align:middle line:84% of many regions that are distinguished by retinotopy 00:57:25.200 --> 00:57:28.840 align:middle line:84% and that are organized into hierarchical pathways. 00:57:28.840 --> 00:57:31.960 align:middle line:84% Mid-level vision loosely refers to a set of perceptual phenomena 00:57:31.960 --> 00:57:34.225 align:middle line:84% that involve inferences about the world. 00:57:34.225 --> 00:57:35.600 align:middle line:84% They're not always clearly linked 00:57:35.600 --> 00:57:37.220 align:middle line:90% to neural mechanisms or stages. 00:57:37.220 --> 00:57:40.640 align:middle line:84% Although, sometimes we have little glimpses of that. 00:57:40.640 --> 00:57:43.000 align:middle line:84% We talked in particular about grouping, and popout, 00:57:43.000 --> 00:57:44.520 align:middle line:90% and figure and ground. 00:57:44.520 --> 00:57:46.880 align:middle line:84% In both cases, they seem to be linked 00:57:46.880 --> 00:57:51.360 align:middle line:84% to the statistics of the world, which nowadays can be measured 00:57:51.360 --> 00:57:53.600 align:middle line:84% and compared to human perception. 00:57:53.600 --> 00:57:56.520 align:middle line:84% And then we also talked about modal and amodal completion. 00:57:56.520 --> 00:57:59.720 align:middle line:84% These are processes that seem to represent perceptual inferences 00:57:59.720 --> 00:58:02.560 align:middle line:84% of object edges, even in cases where you don't have 00:58:02.560 --> 00:58:06.760 align:middle line:90% local evidence for those edges. 00:58:06.760 --> 00:58:08.940 align:middle line:84% What questions do you have about mid-level vision? 00:58:08.940 --> 00:58:19.240 align:middle line:90% 00:58:19.240 --> 00:58:20.960 align:middle line:90% OK. 00:58:20.960 --> 00:58:24.160 align:middle line:84% Let's talk about lightness perception. 00:58:24.160 --> 00:58:26.880 align:middle line:84% I reorganized the course just a little bit 00:58:26.880 --> 00:58:28.840 align:middle line:90% for logistical reasons. 00:58:28.840 --> 00:58:31.440 align:middle line:90% So this used to come later. 00:58:31.440 --> 00:58:32.640 align:middle line:90% But we're doing it earlier. 00:58:32.640 --> 00:58:35.140 align:middle line:84% And we're going to talk about lightness perception, and then 00:58:35.140 --> 00:58:39.750 align:middle line:84% color perception, and then get into depth and motion. 00:58:39.750 --> 00:58:41.130 align:middle line:84% All right, so here's the problem. 00:58:41.130 --> 00:58:46.630 align:middle line:84% So organisms need to estimate surface pigmentation. 00:58:46.630 --> 00:58:49.950 align:middle line:84% So pigmentation tells us a lot of important things 00:58:49.950 --> 00:58:52.310 align:middle line:90% about what things are made of. 00:58:52.310 --> 00:58:54.870 align:middle line:84% So for instance, it would tell you whether a piece of fruit 00:58:54.870 --> 00:58:56.750 align:middle line:90% is ripe or not. 00:58:56.750 --> 00:58:59.523 align:middle line:84% It might tell you whether your baby's got a fever or not. 00:58:59.523 --> 00:59:01.190 align:middle line:84% Lots of important things about the world 00:59:01.190 --> 00:59:05.100 align:middle line:90% are signaled by pigmentation. 00:59:05.100 --> 00:59:08.830 align:middle line:84% And so lightness perception and color perception 00:59:08.830 --> 00:59:13.390 align:middle line:84% are really just about estimating surface pigmentation. 00:59:13.390 --> 00:59:16.550 align:middle line:84% Lightness perception really is the one-dimensional problem. 00:59:16.550 --> 00:59:20.170 align:middle line:84% So we're going to first consider the problem ignoring wavelength 00:59:20.170 --> 00:59:23.110 align:middle line:84% and just talk about the problem of estimating what we call 00:59:23.110 --> 00:59:25.910 align:middle line:84% reflectance, where we're going to define reflectance 00:59:25.910 --> 00:59:28.253 align:middle line:84% as the proportion of light that a surface reflects. 00:59:28.253 --> 00:59:29.670 align:middle line:84% So this would be like if there was 00:59:29.670 --> 00:59:31.270 align:middle line:84% a world where everything was just 00:59:31.270 --> 00:59:33.550 align:middle line:84% on the continuum between black and white. 00:59:33.550 --> 00:59:37.430 align:middle line:84% And so the term lightness is used 00:59:37.430 --> 00:59:39.310 align:middle line:84% to refer to the perceptual correlate 00:59:39.310 --> 00:59:41.510 align:middle line:90% of this notion of reflectance. 00:59:41.510 --> 00:59:45.470 align:middle line:84% So it's like our estimate inside our heads 00:59:45.470 --> 00:59:48.510 align:middle line:84% of the extent to which something is white, black, gray, dark 00:59:48.510 --> 00:59:49.990 align:middle line:90% gray, et cetera. 00:59:49.990 --> 00:59:52.790 align:middle line:90% 00:59:52.790 --> 00:59:56.070 align:middle line:84% So the whole reason that we can talk about this 00:59:56.070 --> 01:00:00.550 align:middle line:84% is because many surfaces in the world exhibit 01:00:00.550 --> 01:00:03.790 align:middle line:84% something that's well-approximated as Lambertian 01:00:03.790 --> 01:00:04.950 align:middle line:90% reflectance. 01:00:04.950 --> 01:00:08.910 align:middle line:84% So a surface that is Lambertian scatters light 01:00:08.910 --> 01:00:13.890 align:middle line:90% uniformly in all directions. 01:00:13.890 --> 01:00:18.630 align:middle line:90% So here's the technical details. 01:00:18.630 --> 01:00:21.630 align:middle line:84% So I is the amount of incident light. 01:00:21.630 --> 01:00:26.390 align:middle line:84% Theta is the angle of the illumination 01:00:26.390 --> 01:00:28.610 align:middle line:90% relative to the surface normal. 01:00:28.610 --> 01:00:35.180 align:middle line:84% And so what this means is that, if the light is directed 01:00:35.180 --> 01:00:36.700 align:middle line:84% perpendicular to the surface, there 01:00:36.700 --> 01:00:38.740 align:middle line:84% will be a lot of light that is scattered. 01:00:38.740 --> 01:00:41.800 align:middle line:84% And if it is oblique, there will be less light that is scattered. 01:00:41.800 --> 01:00:44.620 align:middle line:84% But the key concept is that the light gets scattered 01:00:44.620 --> 01:00:46.860 align:middle line:90% uniformly in all directions. 01:00:46.860 --> 01:00:52.580 align:middle line:84% So that's the definition of a Lambertian surface or material. 01:00:52.580 --> 01:00:54.940 align:middle line:84% So that turns out to be important 01:00:54.940 --> 01:00:59.380 align:middle line:84% because a Lambertian surface's reflectance can be characterized 01:00:59.380 --> 01:01:03.500 align:middle line:84% by just one number, which is the proportion of light that 01:01:03.500 --> 01:01:05.200 align:middle line:90% gets reflected. 01:01:05.200 --> 01:01:08.540 align:middle line:84% So this allows us to refer to the lightness of a surface 01:01:08.540 --> 01:01:11.740 align:middle line:90% and have that have some meaning. 01:01:11.740 --> 01:01:13.480 align:middle line:84% So in the world of lightness perception, 01:01:13.480 --> 01:01:16.105 align:middle line:84% which is what we're going to be talking about in this lecture-- 01:01:16.105 --> 01:01:18.740 align:middle line:84% so this is where we don't care about wavelength-- 01:01:18.740 --> 01:01:22.060 align:middle line:84% this means that the problem of lightness perception 01:01:22.060 --> 01:01:25.180 align:middle line:84% or reflectance estimation can be summarized by this equation. 01:01:25.180 --> 01:01:28.980 align:middle line:84% So there's this quantity of an object in the world called 01:01:28.980 --> 01:01:30.220 align:middle line:90% reflectance. 01:01:30.220 --> 01:01:33.340 align:middle line:84% That's the proportion of light that the surface will reflect. 01:01:33.340 --> 01:01:38.060 align:middle line:84% There is illumination that's coming from a light source. 01:01:38.060 --> 01:01:40.540 align:middle line:84% And the luminance that is reflected by the surface 01:01:40.540 --> 01:01:42.460 align:middle line:84% is the product of the reflectance 01:01:42.460 --> 01:01:46.030 align:middle line:90% and the illumination. 01:01:46.030 --> 01:01:50.320 align:middle line:84% So critically, this is what you measure. 01:01:50.320 --> 01:01:51.880 align:middle line:84% Light is reflected off of a surface. 01:01:51.880 --> 01:01:52.900 align:middle line:90% And it enters your eye. 01:01:52.900 --> 01:01:56.060 align:middle line:90% And your eye records that. 01:01:56.060 --> 01:02:00.980 align:middle line:84% But that light is caused by two different variables-- 01:02:00.980 --> 01:02:03.220 align:middle line:84% the illumination, so the light source in the world, 01:02:03.220 --> 01:02:07.420 align:middle line:84% and the reflectance of a particular object. 01:02:07.420 --> 01:02:07.920 align:middle line:90% All right. 01:02:07.920 --> 01:02:10.140 align:middle line:84% And so all of this really, it makes sense 01:02:10.140 --> 01:02:13.780 align:middle line:84% if we think of surfaces as being Lambertian. 01:02:13.780 --> 01:02:17.990 align:middle line:84% So importantly, many surfaces are not fully Lambertian. 01:02:17.990 --> 01:02:19.740 align:middle line:84% So this is a case where you have something 01:02:19.740 --> 01:02:22.060 align:middle line:90% that's pretty Lambertian. 01:02:22.060 --> 01:02:26.220 align:middle line:84% Sometimes though, you get non-Lambertian reflectance, 01:02:26.220 --> 01:02:28.490 align:middle line:84% in particular specular reflections. 01:02:28.490 --> 01:02:30.370 align:middle line:84% So this is a case where the light 01:02:30.370 --> 01:02:34.490 align:middle line:84% comes in and gets reflected in some particular direction. 01:02:34.490 --> 01:02:37.950 align:middle line:84% So that's like what a mirror would do. 01:02:37.950 --> 01:02:40.910 align:middle line:84% And oftentimes, the surfaces that we encounter in the world, 01:02:40.910 --> 01:02:44.428 align:middle line:84% they have a combination of diffuse and specular reflection. 01:02:44.428 --> 01:02:45.970 align:middle line:84% So you get something like this, where 01:02:45.970 --> 01:02:48.387 align:middle line:84% there's a whole bunch of light that's scattered uniformly. 01:02:48.387 --> 01:02:52.690 align:middle line:84% But then there'll be a special direction that 01:02:52.690 --> 01:02:54.250 align:middle line:90% emits a fair amount of light. 01:02:54.250 --> 01:02:56.650 align:middle line:84% And then in addition, sometimes what you 01:02:56.650 --> 01:03:00.410 align:middle line:84% can have is this diffuse reflectance, 01:03:00.410 --> 01:03:02.810 align:middle line:84% and then some specular reflections 01:03:02.810 --> 01:03:06.130 align:middle line:84% that have some amount of blur, that get blurred out spatially. 01:03:06.130 --> 01:03:09.090 align:middle line:84% And this is a function of the material properties. 01:03:09.090 --> 01:03:11.370 align:middle line:84% And our visual systems are amazingly 01:03:11.370 --> 01:03:13.890 align:middle line:90% attuned to these properties. 01:03:13.890 --> 01:03:16.410 align:middle line:84% And you can get a sense of this just by looking 01:03:16.410 --> 01:03:18.410 align:middle line:90% at computer graphics. 01:03:18.410 --> 01:03:23.610 align:middle line:84% So this is a case where there is a specularity on the surface. 01:03:23.610 --> 01:03:25.730 align:middle line:90% So that's what this is. 01:03:25.730 --> 01:03:33.120 align:middle line:84% So this is rendered with a model of a specular reflection. 01:03:33.120 --> 01:03:38.080 align:middle line:84% And that causes the ball to appear to be somewhat glossy. 01:03:38.080 --> 01:03:44.520 align:middle line:84% And one way to see that is to just selectively remove 01:03:44.520 --> 01:03:46.480 align:middle line:90% the specular reflection. 01:03:46.480 --> 01:03:49.340 align:middle line:84% And now, the ball looks like it's matte. 01:03:49.340 --> 01:03:52.780 align:middle line:84% So the material appearance is quite different. 01:03:52.780 --> 01:03:56.720 align:middle line:84% So the local presence of those specularities 01:03:56.720 --> 01:04:01.640 align:middle line:84% causes you to infer this property of the entire surface 01:04:01.640 --> 01:04:04.815 align:middle line:90% as being shiny. 01:04:04.815 --> 01:04:06.440 align:middle line:84% At the start of the class, I showed you 01:04:06.440 --> 01:04:12.040 align:middle line:84% this photo that I like, where you look at this person's legs 01:04:12.040 --> 01:04:14.145 align:middle line:90% and they look shiny. 01:04:14.145 --> 01:04:15.520 align:middle line:84% So I think what actually happened 01:04:15.520 --> 01:04:19.480 align:middle line:84% is there's some suntan lotion that just got streaked along. 01:04:19.480 --> 01:04:22.280 align:middle line:84% But your visual system mistakes this for specular reflections. 01:04:22.280 --> 01:04:26.640 align:middle line:84% And it looks like the person's legs are shiny. 01:04:26.640 --> 01:04:30.800 align:middle line:84% Here's some graphical rendering of different surface materials 01:04:30.800 --> 01:04:32.740 align:middle line:90% that vary in their roughness. 01:04:32.740 --> 01:04:34.560 align:middle line:84% So the surface roughness will determine 01:04:34.560 --> 01:04:39.480 align:middle line:84% the extent to which the specular reflections are blurred 01:04:39.480 --> 01:04:42.260 align:middle line:84% and the amount of specular reflectance. 01:04:42.260 --> 01:04:44.720 align:middle line:90% So it's how shiny the thing is. 01:04:44.720 --> 01:04:48.280 align:middle line:84% And so all of these look like spheres 01:04:48.280 --> 01:04:50.480 align:middle line:84% that you might encounter in the world. 01:04:50.480 --> 01:04:54.540 align:middle line:84% And they may look like familiar objects. 01:04:54.540 --> 01:04:58.600 align:middle line:84% And they just differ in the way that they reflect light. 01:04:58.600 --> 01:05:02.120 align:middle line:84% And your visual system is pretty attuned to that. 01:05:02.120 --> 01:05:04.680 align:middle line:84% So there's a whole field now of material perception 01:05:04.680 --> 01:05:08.120 align:middle line:90% that relates to stuff like this. 01:05:08.120 --> 01:05:11.740 align:middle line:84% So we often, in the context of lightness perception, 01:05:11.740 --> 01:05:14.760 align:middle line:84% will just explore what happens if surfaces 01:05:14.760 --> 01:05:16.960 align:middle line:90% are purely Lambertian. 01:05:16.960 --> 01:05:20.800 align:middle line:84% But in the real world, things are more complicated. 01:05:20.800 --> 01:05:22.300 align:middle line:84% There's additional complications, 01:05:22.300 --> 01:05:24.310 align:middle line:90% which is translucency. 01:05:24.310 --> 01:05:29.670 align:middle line:84% So translucent objects are those where light enters an object, 01:05:29.670 --> 01:05:33.750 align:middle line:84% and then emerges a little bit later. 01:05:33.750 --> 01:05:36.330 align:middle line:90% So you can have a light source. 01:05:36.330 --> 01:05:38.130 align:middle line:84% There can be some diffuse reflectance, 01:05:38.130 --> 01:05:40.030 align:middle line:84% some specular reflectance that will 01:05:40.030 --> 01:05:41.870 align:middle line:90% cause the perception of gloss. 01:05:41.870 --> 01:05:44.830 align:middle line:84% And then photons can enter the material, 01:05:44.830 --> 01:05:48.030 align:middle line:84% and then bounce around, and then later emerge. 01:05:48.030 --> 01:05:52.950 align:middle line:84% And that will cause something to appear translucent. 01:05:52.950 --> 01:05:56.550 align:middle line:84% Here's another picture that gives the basic idea. 01:05:56.550 --> 01:06:00.150 align:middle line:84% So translucency is actually really important in perception, 01:06:00.150 --> 01:06:04.410 align:middle line:84% in part because skin has elements of translucency. 01:06:04.410 --> 01:06:06.510 align:middle line:84% So this is a diagram that shows you what happens 01:06:06.510 --> 01:06:08.590 align:middle line:90% when light interacts with skin. 01:06:08.590 --> 01:06:14.150 align:middle line:84% So your skin consists of these different levels of tissue. 01:06:14.150 --> 01:06:17.490 align:middle line:84% And there will be some amount of reflection off of the skin. 01:06:17.490 --> 01:06:20.490 align:middle line:84% But then photons enter the skin, and then eventually emerge. 01:06:20.490 --> 01:06:24.110 align:middle line:84% And so if you want to convincingly render 01:06:24.110 --> 01:06:26.630 align:middle line:84% the appearance of a person, you have 01:06:26.630 --> 01:06:30.030 align:middle line:84% to model what's called subsurface scattering, so 01:06:30.030 --> 01:06:34.910 align:middle line:90% this element of translucency. 01:06:34.910 --> 01:06:39.412 align:middle line:84% And in inanimate objects, the rendering 01:06:39.412 --> 01:06:40.870 align:middle line:84% of subsurface scattering can really 01:06:40.870 --> 01:06:42.870 align:middle line:84% change the appearance of material. 01:06:42.870 --> 01:06:45.270 align:middle line:90% So this is a little figurine. 01:06:45.270 --> 01:06:49.050 align:middle line:84% The one on the left looks like it's ceramic. 01:06:49.050 --> 01:06:52.190 align:middle line:84% And the one on the right looks like maybe wax. 01:06:52.190 --> 01:06:55.710 align:middle line:84% So this is done with computer graphic rendering. 01:06:55.710 --> 01:06:58.070 align:middle line:84% So that's just a brief little aside 01:06:58.070 --> 01:07:00.790 align:middle line:84% to say that surfaces in the world 01:07:00.790 --> 01:07:04.010 align:middle line:84% are rich and complicated in the way that they reflect light. 01:07:04.010 --> 01:07:06.550 align:middle line:84% And your visual system is highly attuned to the way 01:07:06.550 --> 01:07:07.810 align:middle line:90% that they reflect light. 01:07:07.810 --> 01:07:12.350 align:middle line:84% And we could have an entire class just about that. 01:07:12.350 --> 01:07:16.270 align:middle line:84% But it's nonetheless the case that many surfaces 01:07:16.270 --> 01:07:17.890 align:middle line:90% are approximately Lambertian. 01:07:17.890 --> 01:07:20.108 align:middle line:84% And you can learn a lot just from thinking 01:07:20.108 --> 01:07:21.900 align:middle line:84% about the Lambertian case, even though it's 01:07:21.900 --> 01:07:24.860 align:middle line:84% a little bit of a simplification. 01:07:24.860 --> 01:07:25.360 align:middle line:90% All right. 01:07:25.360 --> 01:07:27.100 align:middle line:84% So the focus of this lecture will 01:07:27.100 --> 01:07:31.500 align:middle line:84% be on the phenomenon of what is called lightness constancy. 01:07:31.500 --> 01:07:33.220 align:middle line:84% So this refers to the fact that people 01:07:33.220 --> 01:07:37.460 align:middle line:84% are pretty good at correctly estimating reflectance. 01:07:37.460 --> 01:07:42.100 align:middle line:90% So I can show you this eraser. 01:07:42.100 --> 01:07:44.480 align:middle line:84% And you'll probably say this is pretty black. 01:07:44.480 --> 01:07:49.340 align:middle line:84% I can show you this piece of paper. 01:07:49.340 --> 01:07:52.220 align:middle line:84% And everybody would agree this is white. 01:07:52.220 --> 01:07:54.780 align:middle line:84% But we're in a room that actually has 01:07:54.780 --> 01:07:56.160 align:middle line:90% relatively low light levels. 01:07:56.160 --> 01:07:58.120 align:middle line:84% So if we walk out there in the atrium, 01:07:58.120 --> 01:08:00.700 align:middle line:84% the amount of illumination will go up a fair bit. 01:08:00.700 --> 01:08:03.627 align:middle line:84% But if we take the eraser and the white piece of paper 01:08:03.627 --> 01:08:05.460 align:middle line:84% out there, this is still going to look black 01:08:05.460 --> 01:08:07.043 align:middle line:84% and this is still going to look white. 01:08:07.043 --> 01:08:10.500 align:middle line:84% And if we go out on the street, out on Vassar Street, 01:08:10.500 --> 01:08:13.420 align:middle line:84% there's a lot more illumination than there is in the atrium. 01:08:13.420 --> 01:08:16.819 align:middle line:84% This will still look black and this will still look white. 01:08:16.819 --> 01:08:17.319 align:middle line:90% All right. 01:08:17.319 --> 01:08:19.720 align:middle line:84% So the point is that the illumination changes. 01:08:19.720 --> 01:08:22.140 align:middle line:84% And that means that the amount of luminance that's 01:08:22.140 --> 01:08:24.060 align:middle line:84% being reflected off of the surfaces 01:08:24.060 --> 01:08:27.120 align:middle line:84% will change dramatically from environment to environment. 01:08:27.120 --> 01:08:32.140 align:middle line:84% But our perception of what the surface is made of, 01:08:32.140 --> 01:08:34.760 align:middle line:84% our subjective sense of lightness, 01:08:34.760 --> 01:08:37.580 align:middle line:84% which, again, is an implicit estimate of the reflectance, 01:08:37.580 --> 01:08:40.899 align:middle line:84% that's relatively unaffected by that. 01:08:40.899 --> 01:08:43.260 align:middle line:84% So a gray piece of paper looks about the same shade 01:08:43.260 --> 01:08:44.880 align:middle line:84% of gray indoors as it does outdoors, 01:08:44.880 --> 01:08:48.060 align:middle line:84% despite reflecting dramatically more light outdoors. 01:08:48.060 --> 01:08:51.859 align:middle line:84% So this suggests, anecdotally, that we're 01:08:51.859 --> 01:08:54.660 align:middle line:84% fairly accurate in estimating the reflectance 01:08:54.660 --> 01:08:59.760 align:middle line:90% of everyday objects. 01:08:59.760 --> 01:09:04.380 align:middle line:84% So I want to end by giving you an additional piece of evidence 01:09:04.380 --> 01:09:06.660 align:middle line:90% that we estimate reflectance. 01:09:06.660 --> 01:09:11.660 align:middle line:84% And that lies in the subjective observation 01:09:11.660 --> 01:09:14.300 align:middle line:84% that luminance changes that we interpret 01:09:14.300 --> 01:09:17.060 align:middle line:84% as being due to illumination look 01:09:17.060 --> 01:09:22.189 align:middle line:84% really different when we instead interpret them as reflectance. 01:09:22.189 --> 01:09:27.010 align:middle line:84% And so we're going to attempt to do this demo. 01:09:27.010 --> 01:09:31.029 align:middle line:84% And I want to explain to you how this is supposed to work. 01:09:31.029 --> 01:09:34.210 align:middle line:84% And then we'll try it out up here. 01:09:34.210 --> 01:09:35.410 align:middle line:90% So here's what happens. 01:09:35.410 --> 01:09:40.109 align:middle line:84% We're going to turn shadows into paint in your head. 01:09:40.109 --> 01:09:43.952 align:middle line:84% So to do this, you place an object and a light source 01:09:43.952 --> 01:09:45.910 align:middle line:84% such that a shadow is cast on a piece of paper. 01:09:45.910 --> 01:09:47.569 align:middle line:84% And so we're going to do it up here. 01:09:47.569 --> 01:09:48.850 align:middle line:84% We're not going to do it right now because we 01:09:48.850 --> 01:09:50.017 align:middle line:90% have to turn out the lights. 01:09:50.017 --> 01:09:54.050 align:middle line:84% So we're going to do it using this lamp, which is on, good. 01:09:54.050 --> 01:09:57.070 align:middle line:84% So the object is going to cast a shadow on the paper. 01:09:57.070 --> 01:09:59.290 align:middle line:84% What we're going to do then is we're 01:09:59.290 --> 01:10:01.150 align:middle line:84% going to take this black magic marker. 01:10:01.150 --> 01:10:05.570 align:middle line:84% And we're going to trace the outline of the shadow. 01:10:05.570 --> 01:10:09.370 align:middle line:84% And it will look like the region within the traced outline 01:10:09.370 --> 01:10:12.410 align:middle line:84% has a different color than the rest of the paper. 01:10:12.410 --> 01:10:16.290 align:middle line:84% Now, in fact, the number of photons 01:10:16.290 --> 01:10:18.690 align:middle line:84% that's being reflected by the paper and the pattern 01:10:18.690 --> 01:10:20.450 align:middle line:84% inside here will be exactly the same. 01:10:20.450 --> 01:10:22.890 align:middle line:84% We're just going to add a little outline around it. 01:10:22.890 --> 01:10:27.370 align:middle line:84% But your brain will interpret those photons 01:10:27.370 --> 01:10:28.770 align:middle line:90% completely differently. 01:10:28.770 --> 01:10:31.490 align:middle line:84% And the intuition is just that, well, shadows never 01:10:31.490 --> 01:10:32.890 align:middle line:90% have these black outlines. 01:10:32.890 --> 01:10:35.850 align:middle line:84% So when I put the black outline around it, 01:10:35.850 --> 01:10:39.970 align:middle line:84% your brain refuses to interpret that as an illumination 01:10:39.970 --> 01:10:40.750 align:middle line:90% boundary. 01:10:40.750 --> 01:10:43.450 align:middle line:84% And so the point here is that this 01:10:43.450 --> 01:10:48.330 align:middle line:84% is going to illustrate that your visual system codes reflectance 01:10:48.330 --> 01:10:50.890 align:middle line:84% and illumination as very different things, 01:10:50.890 --> 01:10:54.650 align:middle line:84% more evidence that you actually estimate reflectance. 01:10:54.650 --> 01:10:58.230 align:middle line:84% There is this observed image intensity-- this is luminance. 01:10:58.230 --> 01:11:01.450 align:middle line:84% This is registered by your photoreceptors-- 01:11:01.450 --> 01:11:05.410 align:middle line:84% that is the result of the incident 01:11:05.410 --> 01:11:08.530 align:middle line:84% light, the illumination, and the reflectance. 01:11:08.530 --> 01:11:13.030 align:middle line:84% That's a quality or property of the pigmentation of the surface. 01:11:13.030 --> 01:11:17.160 align:middle line:84% So you can think of the generative process as this. 01:11:17.160 --> 01:11:19.040 align:middle line:84% So at every point in space, there 01:11:19.040 --> 01:11:21.220 align:middle line:84% is an amount of incident illumination. 01:11:21.220 --> 01:11:24.320 align:middle line:84% There's also a certain amount of reflectance. 01:11:24.320 --> 01:11:25.320 align:middle line:90% Those get multiplied. 01:11:25.320 --> 01:11:27.760 align:middle line:84% And that generates a luminance image. 01:11:27.760 --> 01:11:30.720 align:middle line:84% So at every pixel, you have one number. 01:11:30.720 --> 01:11:33.440 align:middle line:90% And you want to estimate two-- 01:11:33.440 --> 01:11:35.160 align:middle line:90% or at least one of the two. 01:11:35.160 --> 01:11:36.660 align:middle line:90% You can't unmultiply. 01:11:36.660 --> 01:11:41.480 align:middle line:84% So it's a classic example of an ill-posed problem. 01:11:41.480 --> 01:11:43.090 align:middle line:90% But humans seem to do it. 01:11:43.090 --> 01:11:45.340 align:middle line:84% So we correctly perceive reflectance most of the time. 01:11:45.340 --> 01:11:47.800 align:middle line:84% And this is the phenomenon of lightness constancy. 01:11:47.800 --> 01:11:50.160 align:middle line:90% So how do we do it? 01:11:50.160 --> 01:11:53.440 align:middle line:84% And what we're going to talk about when we resume next time 01:11:53.440 --> 01:11:56.240 align:middle line:84% is a variety of illusions that provide 01:11:56.240 --> 01:11:58.040 align:middle line:90% some clues as to how we do it. 01:11:58.040 --> 01:12:00.620 align:middle line:84% So this is one of the very earliest lightness illusions. 01:12:00.620 --> 01:12:02.800 align:middle line:84% It's called simultaneous contrast. 01:12:02.800 --> 01:12:06.040 align:middle line:84% So it's an illusion because these two squares are 01:12:06.040 --> 01:12:08.000 align:middle line:90% the same physical gray level. 01:12:08.000 --> 01:12:09.600 align:middle line:84% But when you look at them, it probably 01:12:09.600 --> 01:12:12.320 align:middle line:84% looks like the one on the left is lighter 01:12:12.320 --> 01:12:14.080 align:middle line:90% than the one on the right. 01:12:14.080 --> 01:12:16.960 align:middle line:84% Now, you'll also probably notice that the one on the left 01:12:16.960 --> 01:12:19.405 align:middle line:84% is embedded in a black-- or a dark surround. 01:12:19.405 --> 01:12:21.780 align:middle line:84% And the one on the right is embedded in a light surround. 01:12:21.780 --> 01:12:23.640 align:middle line:84% And indeed, that's what causes the one 01:12:23.640 --> 01:12:26.320 align:middle line:84% on the left to look lighter than the one on the right. 01:12:26.320 --> 01:12:28.740 align:middle line:84% But both of the squares have the same luminance, 01:12:28.740 --> 01:12:31.080 align:middle line:84% the same physical intensity in the image. 01:12:31.080 --> 01:12:34.080 align:middle line:84% And when we come back, we will talk 01:12:34.080 --> 01:12:37.820 align:middle line:84% about some different classical explanations for this 01:12:37.820 --> 01:12:40.620 align:middle line:84% and what it tells us about lightness perception. 01:12:40.620 --> 01:12:43.720 align:middle line:84% And we will also get to much more striking lightness 01:12:43.720 --> 01:12:44.920 align:middle line:90% illusions. 01:12:44.920 --> 01:12:47.290 align:middle line:90% So to be continued. 01:12:47.290 --> 01:13:14.000 align:middle line:90%