WEBVTT 00:00:00.000 --> 00:00:01.964 align:middle line:90% [SQUEAKING] 00:00:01.964 --> 00:00:03.916 align:middle line:90% [RUSTLING] 00:00:03.916 --> 00:00:05.380 align:middle line:90% [CLICKING] 00:00:05.380 --> 00:00:11.112 align:middle line:90% 00:00:11.112 --> 00:00:12.820 align:middle line:84% JOSH MCDERMOTT: Happy Thursday, everyone. 00:00:12.820 --> 00:00:15.440 align:middle line:84% So today we're going to continue talking 00:00:15.440 --> 00:00:19.320 align:middle line:84% about lightness perception and then get into color. 00:00:19.320 --> 00:00:23.560 align:middle line:84% So remember, the problem that we're talking about 00:00:23.560 --> 00:00:27.720 align:middle line:84% is that organisms need to estimate surface pigmentation. 00:00:27.720 --> 00:00:30.040 align:middle line:84% And the problem of lightness perception 00:00:30.040 --> 00:00:33.480 align:middle line:84% is the one-dimensional version of estimating pigmentation. 00:00:33.480 --> 00:00:35.640 align:middle line:84% So we're going to ignore effects of wavelength. 00:00:35.640 --> 00:00:38.968 align:middle line:84% So we'll talk about this quantity called reflectance 00:00:38.968 --> 00:00:40.760 align:middle line:84% that corresponds to the proportion of light 00:00:40.760 --> 00:00:42.760 align:middle line:84% that a surface reflects, and lightness 00:00:42.760 --> 00:00:44.940 align:middle line:84% is the perceptual correlate of reflectance. 00:00:44.940 --> 00:00:46.960 align:middle line:84% And so last time we talked about how this really 00:00:46.960 --> 00:00:48.480 align:middle line:84% is-- this only really makes sense 00:00:48.480 --> 00:00:50.700 align:middle line:84% in the context of Lambertian reflectance, 00:00:50.700 --> 00:00:56.080 align:middle line:84% so these are surfaces that scatter light in all directions. 00:00:56.080 --> 00:00:57.720 align:middle line:84% And we talked about exceptions to that. 00:00:57.720 --> 00:01:01.770 align:middle line:84% There's specular reflections and then different degrees 00:01:01.770 --> 00:01:06.130 align:middle line:84% of diffuse reflections and so forth, translucency, 00:01:06.130 --> 00:01:09.710 align:middle line:84% but that many surfaces are approximately Lambertian, 00:01:09.710 --> 00:01:12.930 align:middle line:84% making it a pretty good starting point. 00:01:12.930 --> 00:01:15.450 align:middle line:84% And so the problem of-- or the phenomenon of lightness 00:01:15.450 --> 00:01:17.730 align:middle line:84% constancy refers to the fact that people 00:01:17.730 --> 00:01:19.450 align:middle line:84% are pretty good at correctly estimating 00:01:19.450 --> 00:01:20.830 align:middle line:90% Lambertian reflectance. 00:01:20.830 --> 00:01:23.170 align:middle line:84% So a great piece of paper looks about the same shade 00:01:23.170 --> 00:01:25.145 align:middle line:84% of gray indoors as it does outdoors 00:01:25.145 --> 00:01:26.770 align:middle line:84% even though if you go outdoors, they'll 00:01:26.770 --> 00:01:28.290 align:middle line:84% be dramatically more light that's 00:01:28.290 --> 00:01:30.970 align:middle line:90% reflecting off of the surface. 00:01:30.970 --> 00:01:33.690 align:middle line:84% And then we ended the lecture last time 00:01:33.690 --> 00:01:37.210 align:middle line:84% with this demonstration of the kind of contrast 00:01:37.210 --> 00:01:40.850 align:middle line:84% between perceiving light variation 00:01:40.850 --> 00:01:43.750 align:middle line:84% as being due to illumination versus being due to reflectance. 00:01:43.750 --> 00:01:46.083 align:middle line:84% So you can take a shadow, you draw an outline around it, 00:01:46.083 --> 00:01:48.810 align:middle line:84% and it really changes how it looks. 00:01:48.810 --> 00:01:53.250 align:middle line:84% So the formal problem that we're talking about here 00:01:53.250 --> 00:01:55.490 align:middle line:90% is shown here. 00:01:55.490 --> 00:01:59.930 align:middle line:84% So we've got a luminance image that gets measured by your eye. 00:01:59.930 --> 00:02:02.250 align:middle line:84% That luminance image is generated 00:02:02.250 --> 00:02:05.330 align:middle line:84% by a combination of illumination, 00:02:05.330 --> 00:02:08.250 align:middle line:84% comes from a light source, and reflectance. 00:02:08.250 --> 00:02:10.210 align:middle line:84% And so at every point in the image, 00:02:10.210 --> 00:02:13.262 align:middle line:84% there's some amount of incident illumination. 00:02:13.262 --> 00:02:14.970 align:middle line:84% And then there will be reflectance that's 00:02:14.970 --> 00:02:18.770 align:middle line:84% determined by the pigmentation of the surface in the world 00:02:18.770 --> 00:02:22.276 align:middle line:90% at that point in the image. 00:02:22.276 --> 00:02:25.530 align:middle line:84% So the problem here is that we're trying to-- 00:02:25.530 --> 00:02:29.050 align:middle line:84% we would like to estimate R. The idea is that reflectance 00:02:29.050 --> 00:02:30.690 align:middle line:84% is an important property of objects, 00:02:30.690 --> 00:02:32.190 align:middle line:84% tells us about what they're made of. 00:02:32.190 --> 00:02:33.970 align:middle line:84% So we would like to estimate that. 00:02:33.970 --> 00:02:37.770 align:middle line:84% What we observe is L. OK, L is the product 00:02:37.770 --> 00:02:38.790 align:middle line:90% of these two variables. 00:02:38.790 --> 00:02:41.890 align:middle line:84% And you don't know I either, so I and R unknown. 00:02:41.890 --> 00:02:42.950 align:middle line:90% So it's ill posed. 00:02:42.950 --> 00:02:45.770 align:middle line:84% You got one equation, two unknowns. 00:02:45.770 --> 00:02:48.690 align:middle line:84% So the question is, how can we do this? 00:02:48.690 --> 00:02:51.190 align:middle line:84% And so people have been interested in this a long time. 00:02:51.190 --> 00:02:53.770 align:middle line:84% And many clues to how we solve this problem 00:02:53.770 --> 00:02:57.860 align:middle line:84% have come from, as you might expect, from illusions. 00:02:57.860 --> 00:03:01.220 align:middle line:84% And so last time, we left off with this classic illusion 00:03:01.220 --> 00:03:02.720 align:middle line:90% of simultaneous contrast. 00:03:02.720 --> 00:03:06.340 align:middle line:84% So the two squares are the same physical shade of gray, 00:03:06.340 --> 00:03:07.780 align:middle line:90% but they look different. 00:03:07.780 --> 00:03:10.340 align:middle line:84% So the one on the left looks lighter 00:03:10.340 --> 00:03:13.500 align:middle line:90% than the one on the right. 00:03:13.500 --> 00:03:16.540 align:middle line:84% All right, so historically, there 00:03:16.540 --> 00:03:20.260 align:middle line:84% were a variety of different ways of explaining this phenomenon. 00:03:20.260 --> 00:03:24.100 align:middle line:84% And people debated this pretty fiercely, back in the 1800s. 00:03:24.100 --> 00:03:27.380 align:middle line:84% So there was one fairly prominent perceptual scientist 00:03:27.380 --> 00:03:30.820 align:middle line:84% of the time, Hering, who favored explanations 00:03:30.820 --> 00:03:34.420 align:middle line:84% of this type of phenomena in terms of neural processing. 00:03:34.420 --> 00:03:38.420 align:middle line:84% So remember, we talked about how there 00:03:38.420 --> 00:03:41.300 align:middle line:84% are these center-surround receptive fields 00:03:41.300 --> 00:03:45.820 align:middle line:84% throughout the early stages of the visual system, particularly 00:03:45.820 --> 00:03:47.460 align:middle line:90% the retina and the LGN. 00:03:47.460 --> 00:03:49.860 align:middle line:84% You can think of these center-surround receptive fields 00:03:49.860 --> 00:03:53.953 align:middle line:84% as making a comparison between the light that 00:03:53.953 --> 00:03:55.620 align:middle line:84% falls in their center and the light that 00:03:55.620 --> 00:03:57.620 align:middle line:90% falls into in their surround. 00:03:57.620 --> 00:04:00.460 align:middle line:84% And so imagine that you have a center-surround receptive field 00:04:00.460 --> 00:04:02.428 align:middle line:84% that's positioned on each of those two squares, 00:04:02.428 --> 00:04:04.220 align:middle line:84% it's going to give you a different response 00:04:04.220 --> 00:04:07.860 align:middle line:84% by virtue of the fact that the luminance in the surround 00:04:07.860 --> 00:04:09.480 align:middle line:90% would, on average, be different. 00:04:09.480 --> 00:04:14.620 align:middle line:84% And so this particular receptive field, if it's centered on this, 00:04:14.620 --> 00:04:18.980 align:middle line:84% will give you a bigger response than if it's centered on this. 00:04:18.980 --> 00:04:24.140 align:middle line:84% And just as we explored these explanations of phenomena 00:04:24.140 --> 00:04:27.927 align:middle line:84% like Mach bands and the Hermann grid 00:04:27.927 --> 00:04:29.760 align:middle line:84% in terms of these kinds of receptive fields, 00:04:29.760 --> 00:04:31.980 align:middle line:84% they were also offered as explanations 00:04:31.980 --> 00:04:33.820 align:middle line:90% of simultaneous contrast. 00:04:33.820 --> 00:04:35.940 align:middle line:84% And so Hering was the person who favored 00:04:35.940 --> 00:04:37.737 align:middle line:90% these sorts of explanations. 00:04:37.737 --> 00:04:39.820 align:middle line:84% All right, so Helmholtz was another very prominent 00:04:39.820 --> 00:04:42.580 align:middle line:90% perceptual scientist of the era. 00:04:42.580 --> 00:04:44.860 align:middle line:84% Helmholtz is normally credited with this idea 00:04:44.860 --> 00:04:46.940 align:middle line:84% of unconscious inference, the idea 00:04:46.940 --> 00:04:50.860 align:middle line:84% that, when you are observing an image or a sound, 00:04:50.860 --> 00:04:53.220 align:middle line:84% you're trying to infer the state of the world that 00:04:53.220 --> 00:04:56.630 align:middle line:84% is most likely given the observed sensory signal. 00:04:56.630 --> 00:04:59.390 align:middle line:84% And so the proposal here was that this illusion 00:04:59.390 --> 00:05:02.510 align:middle line:84% results from an attempt to discount illumination. 00:05:02.510 --> 00:05:06.150 align:middle line:84% So the concept is that, when presented 00:05:06.150 --> 00:05:08.923 align:middle line:84% with a stimulus like this, your visual system 00:05:08.923 --> 00:05:10.590 align:middle line:84% is trying to come up with an explanation 00:05:10.590 --> 00:05:13.270 align:middle line:84% of the stimulus in terms of the things 00:05:13.270 --> 00:05:14.630 align:middle line:90% that the world is made of. 00:05:14.630 --> 00:05:16.910 align:middle line:84% And so in the context of this problem, 00:05:16.910 --> 00:05:19.910 align:middle line:84% the world is defined by illumination and reflectance. 00:05:19.910 --> 00:05:22.630 align:middle line:84% And so the notion is that you might observe an image like this 00:05:22.630 --> 00:05:26.310 align:middle line:84% and then hypothesize that what could cause that image would 00:05:26.310 --> 00:05:29.612 align:middle line:84% be a stimulus like this in the world, 00:05:29.612 --> 00:05:31.070 align:middle line:84% so a surface like this in the world 00:05:31.070 --> 00:05:34.470 align:middle line:84% where there are two patches that actually are 00:05:34.470 --> 00:05:36.510 align:middle line:84% different reflectances but that they 00:05:36.510 --> 00:05:39.310 align:middle line:84% are in different levels of illumination. 00:05:39.310 --> 00:05:44.450 align:middle line:84% So this one is in shadow, this one is in higher illumination. 00:05:44.450 --> 00:05:47.310 align:middle line:84% And the notion is that the surrounds here 00:05:47.310 --> 00:05:50.710 align:middle line:84% could be interpreted as being due to the illumination. 00:05:50.710 --> 00:05:52.360 align:middle line:84% So the explanation of the stimulus 00:05:52.360 --> 00:05:55.920 align:middle line:84% would be that this is physically a light patch that's 00:05:55.920 --> 00:05:58.960 align:middle line:84% in dim illumination, and this is physically in a dark patch 00:05:58.960 --> 00:06:00.740 align:middle line:90% that's in bright illumination. 00:06:00.740 --> 00:06:03.440 align:middle line:84% And so the combination of the light patch in the dim 00:06:03.440 --> 00:06:05.920 align:middle line:84% illumination and the dark patch in the bright illumination 00:06:05.920 --> 00:06:08.660 align:middle line:84% produces the same physical gray level, 00:06:08.660 --> 00:06:11.880 align:middle line:84% so the same amount of light that comes off of the page, 00:06:11.880 --> 00:06:13.560 align:middle line:84% and that the explanation in the world 00:06:13.560 --> 00:06:16.760 align:middle line:84% is that you have the combination of causal factors 00:06:16.760 --> 00:06:21.000 align:middle line:84% and that your brain is trying to infer that and that what you 00:06:21.000 --> 00:06:24.920 align:middle line:84% see, when you look at this is your brain's estimate for what 00:06:24.920 --> 00:06:27.440 align:middle line:90% the reflectance plausibly is. 00:06:27.440 --> 00:06:31.680 align:middle line:84% So that's the Helmholtzian explanation. 00:06:31.680 --> 00:06:34.600 align:middle line:84% And as I said, these things were debated. 00:06:34.600 --> 00:06:37.760 align:middle line:84% So Hering supported a mechanistic so-called bottom-up 00:06:37.760 --> 00:06:39.060 align:middle line:90% view of visual processing. 00:06:39.060 --> 00:06:41.600 align:middle line:84% Helmholtz supported a more inferential view 00:06:41.600 --> 00:06:46.280 align:middle line:84% based on experience, what we now refer to as priors and lightness 00:06:46.280 --> 00:06:47.640 align:middle line:90% perception. 00:06:47.640 --> 00:06:49.320 align:middle line:84% So the problem of reflectance estimation 00:06:49.320 --> 00:06:50.780 align:middle line:90% was a major battleground. 00:06:50.780 --> 00:06:53.875 align:middle line:84% So this is a quote from a book by William James. 00:06:53.875 --> 00:06:56.000 align:middle line:84% William James is another pretty famous psychologist 00:06:56.000 --> 00:06:57.920 align:middle line:90% from that era. 00:06:57.920 --> 00:07:01.120 align:middle line:84% And he writes, Helmholtz maintains 00:07:01.120 --> 00:07:04.560 align:middle line:84% that the neural process and the corresponding sensation 00:07:04.560 --> 00:07:08.440 align:middle line:84% also remain unchanged but are differently interpreted. 00:07:08.440 --> 00:07:11.240 align:middle line:84% Hering, that the neural process and the sensation 00:07:11.240 --> 00:07:14.120 align:middle line:84% are themselves changed, and that the interpretation 00:07:14.120 --> 00:07:15.680 align:middle line:84% is the direct conscious correlate 00:07:15.680 --> 00:07:18.060 align:middle line:84% of the altered retinal conditions. 00:07:18.060 --> 00:07:20.340 align:middle line:84% According to the one, the contrast-- 00:07:20.340 --> 00:07:22.240 align:middle line:90% so simultaneous contrast here-- 00:07:22.240 --> 00:07:23.780 align:middle line:90% is psychological in its origin. 00:07:23.780 --> 00:07:27.840 align:middle line:84% According to the other, it's purely physiological. 00:07:27.840 --> 00:07:30.700 align:middle line:84% And it references this distinction between sensation, 00:07:30.700 --> 00:07:33.640 align:middle line:84% so the idea of the measurement process at the front end 00:07:33.640 --> 00:07:35.800 align:middle line:90% of the system, and perception. 00:07:35.800 --> 00:07:41.720 align:middle line:84% And so of course, we read this kind of stuff nowadays, 00:07:41.720 --> 00:07:43.520 align:middle line:84% and the debate seems a little bit silly 00:07:43.520 --> 00:07:46.360 align:middle line:84% because we could think of these two explanations 00:07:46.360 --> 00:07:48.060 align:middle line:90% as not being mutually exclusive. 00:07:48.060 --> 00:07:51.030 align:middle line:84% So the Helmholtzian explanation, of course, 00:07:51.030 --> 00:07:53.370 align:middle line:84% would have to be implemented neurally in some way. 00:07:53.370 --> 00:07:55.453 align:middle line:84% And potentially, the filtering that's 00:07:55.453 --> 00:07:57.870 align:middle line:84% happened in the early visual system could be part of that. 00:07:57.870 --> 00:08:01.330 align:middle line:84% But at the time, this notion of different levels of explanation 00:08:01.330 --> 00:08:07.450 align:middle line:84% wasn't really as clearly understood or acknowledged. 00:08:07.450 --> 00:08:10.770 align:middle line:84% So this is just a little taste of the history 00:08:10.770 --> 00:08:12.230 align:middle line:90% of this kind of problem. 00:08:12.230 --> 00:08:14.930 align:middle line:90% 00:08:14.930 --> 00:08:20.090 align:middle line:84% Another major development in how people thought about lightness 00:08:20.090 --> 00:08:25.410 align:middle line:84% perception came in the '70s with Edwin Land. 00:08:25.410 --> 00:08:28.090 align:middle line:84% How many people have heard of Edwin Land? 00:08:28.090 --> 00:08:29.690 align:middle line:84% Yeah, so he used to be pretty famous. 00:08:29.690 --> 00:08:32.715 align:middle line:84% So Edwin Land invented something called the Polaroid camera. 00:08:32.715 --> 00:08:34.590 align:middle line:84% How many people know what Polaroid camera is? 00:08:34.590 --> 00:08:35.230 align:middle line:90% OK, yeah. 00:08:35.230 --> 00:08:38.210 align:middle line:84% So Polaroids have made a comeback. 00:08:38.210 --> 00:08:42.370 align:middle line:84% When I was a kid, everybody had a Polaroid camera. 00:08:42.370 --> 00:08:45.290 align:middle line:84% So Polaroid camera is this type of camera 00:08:45.290 --> 00:08:47.500 align:middle line:84% that instantly generates a picture. 00:08:47.500 --> 00:08:50.448 align:middle line:84% So you press the button, and it's 00:08:50.448 --> 00:08:52.740 align:middle line:84% got the film inside the camera, and it kind of develops 00:08:52.740 --> 00:08:53.480 align:middle line:90% like on its own. 00:08:53.480 --> 00:08:57.340 align:middle line:84% And so Edwin Land founded Polaroid 00:08:57.340 --> 00:09:04.580 align:middle line:84% and was a major presence in the Cambridge area a while back. 00:09:04.580 --> 00:09:06.260 align:middle line:84% So he got interested in the problem 00:09:06.260 --> 00:09:09.220 align:middle line:84% of reflectance estimation because, in order 00:09:09.220 --> 00:09:11.920 align:middle line:84% to take pictures, cameras need to have the correct exposure. 00:09:11.920 --> 00:09:15.460 align:middle line:84% So they need to open the shutter for the right amount of time. 00:09:15.460 --> 00:09:17.840 align:middle line:84% So they need to divide out the effects of illumination. 00:09:17.840 --> 00:09:21.200 align:middle line:84% And so he got interested in how humans solve analogous problems. 00:09:21.200 --> 00:09:24.060 align:middle line:84% And so he came up with what's called the Retinex 00:09:24.060 --> 00:09:26.940 align:middle line:84% Theory of lightness constancy, where retinex is 00:09:26.940 --> 00:09:29.820 align:middle line:90% short for retina plus cortex. 00:09:29.820 --> 00:09:35.020 align:middle line:84% And it's a pretty cool and elegant theory. 00:09:35.020 --> 00:09:40.340 align:middle line:84% So the key idea of the retinex algorithm or theory 00:09:40.340 --> 00:09:44.540 align:middle line:84% is to take advantage of regularities in the world, i.e. 00:09:44.540 --> 00:09:46.020 align:middle line:90% scene statistics. 00:09:46.020 --> 00:09:50.240 align:middle line:84% And so the notion here is that some scene interpretations, 00:09:50.240 --> 00:09:53.940 align:middle line:84% the explanations of images, are more likely than others. 00:09:53.940 --> 00:09:59.780 align:middle line:84% And so the proposal that Land and his collaborator McCann made 00:09:59.780 --> 00:10:02.060 align:middle line:84% was that it's common for illumination 00:10:02.060 --> 00:10:05.580 align:middle line:84% to vary gradually over space, like with distance 00:10:05.580 --> 00:10:07.260 align:middle line:90% from a light source. 00:10:07.260 --> 00:10:11.800 align:middle line:84% But by contrast, it's common for reflectance to vary abruptly, 00:10:11.800 --> 00:10:13.480 align:middle line:90% so to be piecewise constant. 00:10:13.480 --> 00:10:17.180 align:middle line:84% So the notion here is that a lot of objects, my shirt, 00:10:17.180 --> 00:10:19.265 align:middle line:90% are fairly constant reflectance. 00:10:19.265 --> 00:10:21.140 align:middle line:84% And so then you get to the edge of an object, 00:10:21.140 --> 00:10:25.860 align:middle line:84% and the reflectance kind of makes some large change. 00:10:25.860 --> 00:10:30.940 align:middle line:84% All right, and so to explore the consequences of these generative 00:10:30.940 --> 00:10:36.100 align:middle line:84% assumptions, Land and McCann used this other very influential 00:10:36.100 --> 00:10:38.180 align:middle line:84% approach of employing a toy world. 00:10:38.180 --> 00:10:41.540 align:middle line:84% So they used images that they called Mondrians, so 00:10:41.540 --> 00:10:43.520 align:middle line:90% sets of random patches. 00:10:43.520 --> 00:10:48.590 align:middle line:84% So the proposal is that this is an approximation of the world. 00:10:48.590 --> 00:10:49.850 align:middle line:90% So this is reflectance. 00:10:49.850 --> 00:10:53.230 align:middle line:84% So it's an image that consists of all these patches 00:10:53.230 --> 00:10:56.590 align:middle line:84% of different reflectance values, and then the illumination 00:10:56.590 --> 00:10:58.310 align:middle line:90% would vary gradually. 00:10:58.310 --> 00:11:00.910 align:middle line:84% And then to generate the image that would be observed, 00:11:00.910 --> 00:11:03.130 align:middle line:90% you multiply these two things. 00:11:03.130 --> 00:11:06.070 align:middle line:84% And this gives you the luminance image. 00:11:06.070 --> 00:11:07.770 align:middle line:84% So this is the generative process. 00:11:07.770 --> 00:11:10.010 align:middle line:84% So we live in this simplified world. 00:11:10.010 --> 00:11:12.070 align:middle line:84% The illumination varies gradually. 00:11:12.070 --> 00:11:15.950 align:middle line:84% The reflectance changes discretely. 00:11:15.950 --> 00:11:19.030 align:middle line:84% And then they get multiplied, and you get the luminance image. 00:11:19.030 --> 00:11:21.990 align:middle line:90% 00:11:21.990 --> 00:11:25.550 align:middle line:84% All right, so in this generative model, 00:11:25.550 --> 00:11:27.530 align:middle line:90% these things get multiplied. 00:11:27.530 --> 00:11:32.510 align:middle line:84% To simplify it further, you can take the log of both quantities 00:11:32.510 --> 00:11:35.630 align:middle line:84% to turn that multiplication into an addition. 00:11:35.630 --> 00:11:38.870 align:middle line:84% So we're going to work with the log of the illumination 00:11:38.870 --> 00:11:39.830 align:middle line:90% and so forth. 00:11:39.830 --> 00:11:45.150 align:middle line:84% We'll just, for simplicity, still call it illumination. 00:11:45.150 --> 00:11:48.190 align:middle line:84% So this is what this looks like in 1D. 00:11:48.190 --> 00:11:51.290 align:middle line:84% So this is the image-formation process. 00:11:51.290 --> 00:11:52.910 align:middle line:84% That's the top half of the slide. 00:11:52.910 --> 00:11:54.467 align:middle line:90% So we've got illumination. 00:11:54.467 --> 00:11:56.050 align:middle line:84% So that varies as a function of space. 00:11:56.050 --> 00:12:00.110 align:middle line:84% This is just a 1D depiction of this for simplicity. 00:12:00.110 --> 00:12:02.190 align:middle line:84% So illumination is on the y-axis here. 00:12:02.190 --> 00:12:04.910 align:middle line:84% Space is on the x-axis, position in image. 00:12:04.910 --> 00:12:06.170 align:middle line:90% This is the reflectance. 00:12:06.170 --> 00:12:07.970 align:middle line:84% So the illumination varies gradually. 00:12:07.970 --> 00:12:09.250 align:middle line:90% This is just one example. 00:12:09.250 --> 00:12:12.870 align:middle line:84% Here's a reflectance image where it varies abruptly. 00:12:12.870 --> 00:12:15.670 align:middle line:84% Those get added because we're now in the log domain. 00:12:15.670 --> 00:12:18.350 align:middle line:84% And so you get this luminance image. 00:12:18.350 --> 00:12:21.970 align:middle line:84% So that's the image-formation process. 00:12:21.970 --> 00:12:25.270 align:middle line:84% That's this theory or an idealization of how 00:12:25.270 --> 00:12:28.430 align:middle line:90% the world generates images. 00:12:28.430 --> 00:12:32.390 align:middle line:84% So vision is the inverse process of starting 00:12:32.390 --> 00:12:36.630 align:middle line:84% with this luminance image and somehow estimating 00:12:36.630 --> 00:12:38.530 align:middle line:84% the illumination and the reflectance. 00:12:38.530 --> 00:12:41.452 align:middle line:84% Now, normally, you're mostly interested in the reflectance. 00:12:41.452 --> 00:12:43.160 align:middle line:84% But if you estimate the reflectance, then 00:12:43.160 --> 00:12:45.340 align:middle line:84% you probably could also estimate the illumination, 00:12:45.340 --> 00:12:48.800 align:middle line:84% and maybe you would need to in order to do that. 00:12:48.800 --> 00:12:51.800 align:middle line:84% All right, so the key proposal on how 00:12:51.800 --> 00:12:54.280 align:middle line:84% you would solve this inverse process 00:12:54.280 --> 00:12:58.600 align:middle line:84% is to take advantage of this assumption about the world 00:12:58.600 --> 00:13:02.240 align:middle line:84% that illumination varies gradually and reflectance varies 00:13:02.240 --> 00:13:03.360 align:middle line:90% abruptly. 00:13:03.360 --> 00:13:08.780 align:middle line:84% And so given that assumption, if you look at spatial derivatives, 00:13:08.780 --> 00:13:14.520 align:middle line:84% so how the luminance changes as a function of space, 00:13:14.520 --> 00:13:19.040 align:middle line:84% it follows that big changes, abrupt changes should 00:13:19.040 --> 00:13:22.140 align:middle line:84% be due to the reflectance and small changes, 00:13:22.140 --> 00:13:25.760 align:middle line:84% small derivative values, should be due to illumination. 00:13:25.760 --> 00:13:30.320 align:middle line:84% So the idea is that you take your input, 00:13:30.320 --> 00:13:32.800 align:middle line:90% you compute spatial derivatives. 00:13:32.800 --> 00:13:35.130 align:middle line:84% And what you can see here-- so there's 00:13:35.130 --> 00:13:37.380 align:middle line:84% this constant illumination gradient that's been added. 00:13:37.380 --> 00:13:39.700 align:middle line:84% So you have a small positive derivative here. 00:13:39.700 --> 00:13:43.050 align:middle line:84% And then you hit this big step, and so the derivative shoots up. 00:13:43.050 --> 00:13:45.870 align:middle line:84% Then you go back to the small positive value. 00:13:45.870 --> 00:13:47.070 align:middle line:90% Then it goes down. 00:13:47.070 --> 00:13:49.870 align:middle line:84% So you get a larger derivative down and so on and so forth. 00:13:49.870 --> 00:13:52.490 align:middle line:84% So these are the derivatives here. 00:13:52.490 --> 00:13:54.650 align:middle line:90% 0 would be somewhere here. 00:13:54.650 --> 00:13:58.450 align:middle line:84% And so the idea is that you go through, 00:13:58.450 --> 00:14:01.770 align:middle line:84% and in any place where the derivative is large, 00:14:01.770 --> 00:14:04.850 align:middle line:84% so either strongly positive or strongly negative, 00:14:04.850 --> 00:14:10.030 align:middle line:84% you classify that as having been due to reflectance. 00:14:10.030 --> 00:14:13.210 align:middle line:84% Any place where it's small, you suppose that that 00:14:13.210 --> 00:14:15.450 align:middle line:90% was due to illumination. 00:14:15.450 --> 00:14:18.290 align:middle line:84% So you then separate out the big derivatives 00:14:18.290 --> 00:14:19.750 align:middle line:90% and the small derivatives. 00:14:19.750 --> 00:14:21.650 align:middle line:84% And then you integrate the big ones 00:14:21.650 --> 00:14:24.090 align:middle line:90% to retrieve the reflectance. 00:14:24.090 --> 00:14:27.290 align:middle line:84% And then the remainder is the illuminance. 00:14:27.290 --> 00:14:30.450 align:middle line:84% So we take all of the derivatives 00:14:30.450 --> 00:14:32.850 align:middle line:90% and then integrate them. 00:14:32.850 --> 00:14:35.050 align:middle line:84% And you get this thing out that's an estimate 00:14:35.050 --> 00:14:37.530 align:middle line:84% of the reflectance, so super simple. 00:14:37.530 --> 00:14:46.010 align:middle line:90% 00:14:46.010 --> 00:14:49.730 align:middle line:84% OK, so here's a flow chart providing another example 00:14:49.730 --> 00:14:50.910 align:middle line:90% of how this would work. 00:14:50.910 --> 00:14:54.650 align:middle line:84% So again, here is an example reflectance image and an example 00:14:54.650 --> 00:14:56.570 align:middle line:90% illumination image. 00:14:56.570 --> 00:14:58.730 align:middle line:90% Those get combined. 00:14:58.730 --> 00:15:00.470 align:middle line:90% So here's the observed images. 00:15:00.470 --> 00:15:03.110 align:middle line:84% So again, the signals, in actuality, multiply. 00:15:03.110 --> 00:15:06.010 align:middle line:84% But we're going to work in the log domain, so they add. 00:15:06.010 --> 00:15:07.430 align:middle line:90% You get this observed image. 00:15:07.430 --> 00:15:10.850 align:middle line:84% You take the derivatives, split them up 00:15:10.850 --> 00:15:14.450 align:middle line:84% into the strong derivatives and the weak derivatives. 00:15:14.450 --> 00:15:17.330 align:middle line:84% Integrate each of these as a function of space. 00:15:17.330 --> 00:15:19.730 align:middle line:84% And you get the estimated reflectance and the estimated 00:15:19.730 --> 00:15:20.570 align:middle line:90% illumination. 00:15:20.570 --> 00:15:27.410 align:middle line:90% 00:15:27.410 --> 00:15:32.370 align:middle line:84% All right, any questions about how this works or why? 00:15:32.370 --> 00:15:33.570 align:middle line:90% Yeah. 00:15:33.570 --> 00:15:36.378 align:middle line:84% AUDIENCE: How was this pattern discovered? 00:15:36.378 --> 00:15:37.420 align:middle line:90% JOSH MCDERMOTT: How was-- 00:15:37.420 --> 00:15:39.660 align:middle line:90% AUDIENCE: How was it discovered? 00:15:39.660 --> 00:15:41.340 align:middle line:84% JOSH MCDERMOTT: They thought it up. 00:15:41.340 --> 00:15:47.580 align:middle line:84% I mean, yeah, it's just human intuition, really. 00:15:47.580 --> 00:15:48.120 align:middle line:90% Yeah. 00:15:48.120 --> 00:15:52.260 align:middle line:90% 00:15:52.260 --> 00:15:58.358 align:middle line:84% Yeah, I mean, it's an interesting example where-- 00:15:58.358 --> 00:16:00.400 align:middle line:84% I mean, we'll talk about this a little bit later. 00:16:00.400 --> 00:16:06.900 align:middle line:84% I mean, part of why this was such an important development 00:16:06.900 --> 00:16:10.140 align:middle line:84% was it was just a nice example of how to approach this problem. 00:16:10.140 --> 00:16:12.540 align:middle line:84% So they simplified it down to something 00:16:12.540 --> 00:16:15.660 align:middle line:84% that's pretty simple but not so simple 00:16:15.660 --> 00:16:18.000 align:middle line:84% as to potentially be totally irrelevant. 00:16:18.000 --> 00:16:20.780 align:middle line:84% And then because they simplified it down, 00:16:20.780 --> 00:16:24.133 align:middle line:84% they could make these assumptions 00:16:24.133 --> 00:16:26.300 align:middle line:84% and build that into an algorithm that could actually 00:16:26.300 --> 00:16:29.960 align:middle line:90% solve the inverse problem. 00:16:29.960 --> 00:16:31.760 align:middle line:84% But it was basically just a good idea. 00:16:31.760 --> 00:16:32.260 align:middle line:90% Yeah. 00:16:32.260 --> 00:16:33.020 align:middle line:90% Yeah. 00:16:33.020 --> 00:16:34.437 align:middle line:84% AUDIENCE: Once they integrate, how 00:16:34.437 --> 00:16:36.280 align:middle line:84% did they know what constant to add 00:16:36.280 --> 00:16:38.428 align:middle line:90% to each of those bottom graphs? 00:16:38.428 --> 00:16:40.220 align:middle line:84% JOSH MCDERMOTT: Yeah, it's a good question. 00:16:40.220 --> 00:16:45.140 align:middle line:84% And I think the major answer is the constant offset often 00:16:45.140 --> 00:16:46.280 align:middle line:90% doesn't matter that much. 00:16:46.280 --> 00:16:49.540 align:middle line:90% 00:16:49.540 --> 00:16:55.060 align:middle line:84% But, yeah, that's of an open question, I would say. 00:16:55.060 --> 00:16:55.660 align:middle line:90% Yeah. 00:16:55.660 --> 00:16:57.077 align:middle line:84% AUDIENCE: Is that empirically true 00:16:57.077 --> 00:16:59.457 align:middle line:84% that natural images have this property? 00:16:59.457 --> 00:17:01.040 align:middle line:84% JOSH MCDERMOTT: It's a great question. 00:17:01.040 --> 00:17:04.819 align:middle line:84% So hold that, and let's reevaluate that. 00:17:04.819 --> 00:17:06.980 align:middle line:90% That's a very good question. 00:17:06.980 --> 00:17:08.140 align:middle line:90% OK. 00:17:08.140 --> 00:17:11.060 align:middle line:84% All right, so how could you implement 00:17:11.060 --> 00:17:13.579 align:middle line:90% this in neural hardware? 00:17:13.579 --> 00:17:16.220 align:middle line:84% So one idea that kind of resonated at the time 00:17:16.220 --> 00:17:19.420 align:middle line:84% was that you could use the kinds of filters 00:17:19.420 --> 00:17:22.540 align:middle line:84% that we find in primary visual cortex, which 00:17:22.540 --> 00:17:25.900 align:middle line:84% you can think of as taking an approximate spatial derivative. 00:17:25.900 --> 00:17:27.980 align:middle line:90% So what is a derivative? 00:17:27.980 --> 00:17:29.480 align:middle line:84% Well, it's a differencing operation. 00:17:29.480 --> 00:17:31.420 align:middle line:84% It's telling you how much something changes from one 00:17:31.420 --> 00:17:32.360 align:middle line:90% point to the next. 00:17:32.360 --> 00:17:35.330 align:middle line:84% And if you think about a simple cell receptive field, 00:17:35.330 --> 00:17:38.110 align:middle line:84% you can think of that as taking a difference between two 00:17:38.110 --> 00:17:39.670 align:middle line:90% adjacent regions of an image. 00:17:39.670 --> 00:17:42.677 align:middle line:84% You got an excitatory lobe and an inhibitory lobe. 00:17:42.677 --> 00:17:45.010 align:middle line:84% So you can think of that as taking a spatial derivative. 00:17:45.010 --> 00:17:46.750 align:middle line:90% So you get an input like this. 00:17:46.750 --> 00:17:49.670 align:middle line:84% And the cell will give a positive response 00:17:49.670 --> 00:17:51.150 align:middle line:90% when there's an increase-- 00:17:51.150 --> 00:17:52.750 align:middle line:84% a particular cell might give a positive response 00:17:52.750 --> 00:17:55.167 align:middle line:84% when there's an increase and then a negative response when 00:17:55.167 --> 00:17:56.130 align:middle line:90% there's a decrease. 00:17:56.130 --> 00:17:59.658 align:middle line:90% 00:17:59.658 --> 00:18:02.970 align:middle line:84% So the kinds of ingredients in the algorithm, 00:18:02.970 --> 00:18:04.970 align:middle line:84% at least some of them in the way of derivatives, 00:18:04.970 --> 00:18:06.345 align:middle line:84% seem kind of compatible with what 00:18:06.345 --> 00:18:09.470 align:middle line:84% was known about early vision at that point in time. 00:18:09.470 --> 00:18:13.950 align:middle line:84% But another reason why people got pretty excited about retinex 00:18:13.950 --> 00:18:18.710 align:middle line:84% was that it seemed consistent with some illusions that 00:18:18.710 --> 00:18:21.810 align:middle line:84% were discovered-- or popularized at that time. 00:18:21.810 --> 00:18:24.590 align:middle line:84% So this is a very well-known illusion called 00:18:24.590 --> 00:18:27.350 align:middle line:84% the Craik-O'Brien-Cornsweet effect. 00:18:27.350 --> 00:18:30.270 align:middle line:84% Craik, O'Brien, and Cornsweet were humans that, I guess, 00:18:30.270 --> 00:18:32.310 align:middle line:84% discovered this, I think maybe independently, 00:18:32.310 --> 00:18:35.640 align:middle line:84% or I can't remember if they were working together. 00:18:35.640 --> 00:18:38.642 align:middle line:84% And the stimulus is what's shown here, 00:18:38.642 --> 00:18:41.100 align:middle line:84% so this is showing the luminance as a function of position. 00:18:41.100 --> 00:18:42.625 align:middle line:84% So it's just these two gradients, 00:18:42.625 --> 00:18:45.000 align:middle line:84% where the luminance kind of starts out high and gets low, 00:18:45.000 --> 00:18:48.240 align:middle line:90% starts out high and gets low. 00:18:48.240 --> 00:18:51.180 align:middle line:84% And the illusion is that, when you look at this, 00:18:51.180 --> 00:18:53.720 align:middle line:84% it looks like the thing on the left 00:18:53.720 --> 00:18:56.640 align:middle line:84% is darker than the thing on the right. 00:18:56.640 --> 00:19:00.398 align:middle line:84% But in fact, they're physically identical. 00:19:00.398 --> 00:19:01.940 align:middle line:84% All right, so let's think about this. 00:19:01.940 --> 00:19:04.023 align:middle line:84% How is this consistent with the retinex algorithm? 00:19:04.023 --> 00:19:05.720 align:middle line:84% Well, what retinex is going to do 00:19:05.720 --> 00:19:08.340 align:middle line:84% when it looks at this is take all these derivatives. 00:19:08.340 --> 00:19:11.500 align:middle line:84% And so you get small negative derivatives here 00:19:11.500 --> 00:19:13.640 align:middle line:84% and then this big positive derivative and then 00:19:13.640 --> 00:19:15.220 align:middle line:90% a small negative derivative. 00:19:15.220 --> 00:19:19.640 align:middle line:84% And so if you just assume that that one big positive derivative 00:19:19.640 --> 00:19:22.420 align:middle line:84% is due to the reflectance and then you integrate that, 00:19:22.420 --> 00:19:25.920 align:middle line:84% it's going to tell you that this is lighter than this. 00:19:25.920 --> 00:19:28.800 align:middle line:90% 00:19:28.800 --> 00:19:31.480 align:middle line:84% All right, so this kind of illusion and others 00:19:31.480 --> 00:19:34.920 align:middle line:84% like it indicates that the visual system gives 00:19:34.920 --> 00:19:40.280 align:middle line:84% very strong weight to edges in the computation of lightness. 00:19:40.280 --> 00:19:42.020 align:middle line:84% And not so much to slow gradients. 00:19:42.020 --> 00:19:44.480 align:middle line:84% And so this seemed like suggestive evidence 00:19:44.480 --> 00:19:47.800 align:middle line:84% that the visual system might be doing something similar 00:19:47.800 --> 00:19:49.220 align:middle line:90% to retinex. 00:19:49.220 --> 00:19:52.110 align:middle line:90% 00:19:52.110 --> 00:19:54.320 align:middle line:84% And there are a lot of other illusions 00:19:54.320 --> 00:19:55.740 align:middle line:84% that are kind of the same flavor. 00:19:55.740 --> 00:19:59.120 align:middle line:84% This is another one that's kind of nice. 00:19:59.120 --> 00:20:02.840 align:middle line:84% This is the Cornsweet square wave grating. 00:20:02.840 --> 00:20:05.580 align:middle line:84% So again, the physical stimulus is what's depicted here. 00:20:05.580 --> 00:20:09.120 align:middle line:84% So this is, again, luminance as a function of position. 00:20:09.120 --> 00:20:11.040 align:middle line:84% And so the point of this stimulus 00:20:11.040 --> 00:20:15.480 align:middle line:84% is that the middle regions of each of the squares 00:20:15.480 --> 00:20:17.620 align:middle line:90% are the same gray level. 00:20:17.620 --> 00:20:20.280 align:middle line:84% But then there's this scalloping here, 00:20:20.280 --> 00:20:22.260 align:middle line:90% creating these big step edges. 00:20:22.260 --> 00:20:24.500 align:middle line:84% And so the point is that, when you look at this, 00:20:24.500 --> 00:20:29.740 align:middle line:84% it looks like this is lighter than that when, in fact, 00:20:29.740 --> 00:20:33.250 align:middle line:84% the middle portions of them are the same. 00:20:33.250 --> 00:20:39.110 align:middle line:84% And so this is showing how the stimulus is created. 00:20:39.110 --> 00:20:40.550 align:middle line:90% So you start with a square wave. 00:20:40.550 --> 00:20:46.010 align:middle line:84% You high-pass filter it, you get this is the physical stimulus. 00:20:46.010 --> 00:20:52.610 align:middle line:84% And then you've got these big derivatives here 00:20:52.610 --> 00:20:54.510 align:middle line:84% that are happening kind of at the edges 00:20:54.510 --> 00:20:56.802 align:middle line:84% and then an integration process that would cause things 00:20:56.802 --> 00:21:02.570 align:middle line:84% to fill in so again, consistent with what you might expect 00:21:02.570 --> 00:21:04.950 align:middle line:90% from a retinex type algorithm. 00:21:04.950 --> 00:21:07.890 align:middle line:90% 00:21:07.890 --> 00:21:11.330 align:middle line:84% All right, so one of the major themes of this class 00:21:11.330 --> 00:21:12.970 align:middle line:84% is that illusions reveal the workings 00:21:12.970 --> 00:21:14.650 align:middle line:90% of our perceptual systems. 00:21:14.650 --> 00:21:18.530 align:middle line:84% So we call illusions illusions because they're, 00:21:18.530 --> 00:21:21.970 align:middle line:84% from a certain perspective, an example of the visual system 00:21:21.970 --> 00:21:23.195 align:middle line:90% getting things wrong. 00:21:23.195 --> 00:21:24.570 align:middle line:84% So we have these two stimuli that 00:21:24.570 --> 00:21:26.190 align:middle line:84% are actually physically identical, 00:21:26.190 --> 00:21:27.230 align:middle line:90% but they look different. 00:21:27.230 --> 00:21:28.980 align:middle line:84% But we actually think that these illusions 00:21:28.980 --> 00:21:33.020 align:middle line:84% result from the action of successful engineering designs 00:21:33.020 --> 00:21:35.180 align:middle line:84% that we place in unusual circumstances. 00:21:35.180 --> 00:21:38.340 align:middle line:84% So by studying illusions, we can study the hidden design 00:21:38.340 --> 00:21:41.620 align:middle line:84% rules underlying those engineering designs. 00:21:41.620 --> 00:21:42.843 align:middle line:90% All right. 00:21:42.843 --> 00:21:44.260 align:middle line:84% Another kind of theme of the class 00:21:44.260 --> 00:21:46.080 align:middle line:84% is this idea of levels of analysis. 00:21:46.080 --> 00:21:49.667 align:middle line:84% So as famously articulated by Marr, 00:21:49.667 --> 00:21:51.500 align:middle line:84% so we've got the computational level talking 00:21:51.500 --> 00:21:53.500 align:middle line:84% about what the problem is that's being solved, 00:21:53.500 --> 00:21:56.000 align:middle line:84% the algorithmic level of what the approach is to solving it, 00:21:56.000 --> 00:21:58.167 align:middle line:84% and the implementation level, how to actually put it 00:21:58.167 --> 00:22:00.500 align:middle line:84% into hardware, and reflectance estimation 00:22:00.500 --> 00:22:01.720 align:middle line:90% is a nice example of this. 00:22:01.720 --> 00:22:05.340 align:middle line:84% And so this is actually an excerpt of the book 00:22:05.340 --> 00:22:07.260 align:middle line:90% that Marr wrote. 00:22:07.260 --> 00:22:11.000 align:middle line:90% So this was his magnum opus. 00:22:11.000 --> 00:22:14.900 align:middle line:84% So Marr was a researcher here back in the '70s, and he got 00:22:14.900 --> 00:22:19.040 align:middle line:84% cancer at a very young age and wrote this book on his deathbed. 00:22:19.040 --> 00:22:20.900 align:middle line:84% And then the book came out, I think, right 00:22:20.900 --> 00:22:23.060 align:middle line:90% around when he died. 00:22:23.060 --> 00:22:26.340 align:middle line:84% And it just has his worldview for how 00:22:26.340 --> 00:22:29.300 align:middle line:84% to study vision and maybe perception more generally. 00:22:29.300 --> 00:22:32.300 align:middle line:84% And the retinex algorithm is featured fairly prominently 00:22:32.300 --> 00:22:35.000 align:middle line:84% in there because it's a nice example of these principles. 00:22:35.000 --> 00:22:38.820 align:middle line:84% And in particular, he gives the Retinex Theory 00:22:38.820 --> 00:22:42.140 align:middle line:84% as an example where the same computation can be implemented 00:22:42.140 --> 00:22:43.320 align:middle line:90% in two different algorithms. 00:22:43.320 --> 00:22:47.320 align:middle line:84% And so this is an excerpt from the book that illustrates this. 00:22:47.320 --> 00:22:49.680 align:middle line:84% And so I'll just read you what he says. 00:22:49.680 --> 00:22:52.298 align:middle line:84% He says, the retinex computation-- 00:22:52.298 --> 00:22:54.340 align:middle line:84% again, that's the computational-level description 00:22:54.340 --> 00:22:56.140 align:middle line:90% of this-- 00:22:56.140 --> 00:22:58.260 align:middle line:84% so the problem being solved here is the estimation 00:22:58.260 --> 00:23:00.300 align:middle line:84% of reflectance, leveraging assumptions 00:23:00.300 --> 00:23:02.887 align:middle line:84% about the nature of illumination and reflectance. 00:23:02.887 --> 00:23:04.720 align:middle line:84% That's what allows the problem to be solved. 00:23:04.720 --> 00:23:06.620 align:middle line:84% So that's a computational-level description. 00:23:06.620 --> 00:23:08.420 align:middle line:84% So that computation has been implemented 00:23:08.420 --> 00:23:09.860 align:middle line:90% in at least two ways. 00:23:09.860 --> 00:23:12.300 align:middle line:84% So Land and McCann themselves used 00:23:12.300 --> 00:23:14.820 align:middle line:84% the one-dimensional approach illustrated 00:23:14.820 --> 00:23:18.980 align:middle line:84% in figure 383, which I've shown here for your convenience. 00:23:18.980 --> 00:23:23.100 align:middle line:84% If we trace the image intensities along any path 00:23:23.100 --> 00:23:26.570 align:middle line:84% from A to B, they will have the form shown as blah, blah, blah. 00:23:26.570 --> 00:23:31.230 align:middle line:84% OK, so he talks about this approach, but then he says Horn, 00:23:31.230 --> 00:23:34.670 align:middle line:84% another vision researcher from back in the day, 00:23:34.670 --> 00:23:37.870 align:middle line:84% derived a two-dimensional analog of this algorithm illustrated 00:23:37.870 --> 00:23:41.530 align:middle line:84% in figure 3-83(b), which is shown here. 00:23:41.530 --> 00:23:44.390 align:middle line:90% 00:23:44.390 --> 00:23:47.030 align:middle line:84% Consisting of essentially the same three steps, 00:23:47.030 --> 00:23:49.750 align:middle line:90% but now in two-dimensional form. 00:23:49.750 --> 00:23:51.830 align:middle line:84% So these are two different algorithms 00:23:51.830 --> 00:23:55.630 align:middle line:84% for solving the same computation. 00:23:55.630 --> 00:23:59.090 align:middle line:84% So a nice example of levels of explanation. 00:23:59.090 --> 00:24:02.950 align:middle line:84% So the ideas that we illustrate here with retinex 00:24:02.950 --> 00:24:04.840 align:middle line:84% are, number one, levels of analysis, 00:24:04.840 --> 00:24:06.590 align:middle line:84% so the same computation can be implemented 00:24:06.590 --> 00:24:08.270 align:middle line:90% with different algorithms. 00:24:08.270 --> 00:24:14.230 align:middle line:84% Number two, scene statistics help with ill-posed problems. 00:24:14.230 --> 00:24:16.430 align:middle line:84% So this is what we think of as priors, the idea 00:24:16.430 --> 00:24:18.750 align:middle line:84% that certain things are a lot more likely in the world 00:24:18.750 --> 00:24:21.610 align:middle line:84% than others, in this case illumination varying gradually, 00:24:21.610 --> 00:24:23.910 align:middle line:90% reflectance varying abruptly. 00:24:23.910 --> 00:24:27.670 align:middle line:84% They're not explicitly defined as such, but same idea. 00:24:27.670 --> 00:24:30.110 align:middle line:84% Another thing that's illustrated with retinex 00:24:30.110 --> 00:24:32.390 align:middle line:84% that was very influential is this idea 00:24:32.390 --> 00:24:35.610 align:middle line:84% of using a toy world to help think through a problem. 00:24:35.610 --> 00:24:39.150 align:middle line:84% So the use of these Mondrians and illumination gradients. 00:24:39.150 --> 00:24:41.830 align:middle line:84% And it was also very influential because it 00:24:41.830 --> 00:24:45.150 align:middle line:84% was this really nice convergence of ideas from computation, 00:24:45.150 --> 00:24:47.070 align:middle line:84% from psychophysics with these new illusions, 00:24:47.070 --> 00:24:49.110 align:middle line:84% and physiology, where some of the operations 00:24:49.110 --> 00:24:52.350 align:middle line:84% seem consistent with the physiological things that 00:24:52.350 --> 00:24:55.950 align:middle line:84% were happening, that were being observed in the visual system. 00:24:55.950 --> 00:24:59.970 align:middle line:84% Any questions about retinex or any of this? 00:24:59.970 --> 00:25:09.030 align:middle line:90% 00:25:09.030 --> 00:25:10.870 align:middle line:84% All right, so another thing that's 00:25:10.870 --> 00:25:13.310 align:middle line:84% really important about retinex algorithm 00:25:13.310 --> 00:25:15.910 align:middle line:84% is what's called stimulus computability. 00:25:15.910 --> 00:25:20.750 align:middle line:84% So sometimes, the theories that you encounter 00:25:20.750 --> 00:25:24.120 align:middle line:84% when you're dealing with things related to the mind 00:25:24.120 --> 00:25:26.600 align:middle line:90% are verbally expressed. 00:25:26.600 --> 00:25:29.300 align:middle line:84% It's like there's an idea, gets described in words, 00:25:29.300 --> 00:25:33.600 align:middle line:84% and it's not always clear how to say, take an image, 00:25:33.600 --> 00:25:37.460 align:middle line:84% and actually figure out what the theory predicts. 00:25:37.460 --> 00:25:40.000 align:middle line:84% The retinex algorithm has this property 00:25:40.000 --> 00:25:41.260 align:middle line:90% of stimulus computability. 00:25:41.260 --> 00:25:43.240 align:middle line:84% And what that means is that it is specified 00:25:43.240 --> 00:25:46.520 align:middle line:84% with enough precision that you can apply it 00:25:46.520 --> 00:25:49.820 align:middle line:84% to any arbitrary signal, and it will do something. 00:25:49.820 --> 00:25:52.560 align:middle line:84% And that defines the prediction of the theory. 00:25:52.560 --> 00:25:55.360 align:middle line:84% And that's really important because it 00:25:55.360 --> 00:26:00.360 align:middle line:84% enables you to actually evaluate the theory fairly exhaustively 00:26:00.360 --> 00:26:02.900 align:middle line:84% because you can test it on anything essentially. 00:26:02.900 --> 00:26:06.610 align:middle line:84% And so that's what we aspire to always have. 00:26:06.610 --> 00:26:07.860 align:middle line:90% It's often kind of hard to do. 00:26:07.860 --> 00:26:10.000 align:middle line:84% But this was an early example of this. 00:26:10.000 --> 00:26:13.200 align:middle line:84% So stimulus computable means that a model or algorithm 00:26:13.200 --> 00:26:15.480 align:middle line:84% or theory can be applied to any stimulus. 00:26:15.480 --> 00:26:17.860 align:middle line:84% Not all models or algorithms are defined in this way. 00:26:17.860 --> 00:26:19.300 align:middle line:90% Some are defined with words. 00:26:19.300 --> 00:26:21.300 align:middle line:84% Others work on abstract variables. 00:26:21.300 --> 00:26:24.090 align:middle line:84% So we assume that there are these variables that are 00:26:24.090 --> 00:26:25.710 align:middle line:90% extracted by earlier processes. 00:26:25.710 --> 00:26:29.270 align:middle line:84% And so the theory is trying to model a second stage, 00:26:29.270 --> 00:26:32.022 align:middle line:84% but it means you can't really test it on actual stimuli. 00:26:32.022 --> 00:26:33.730 align:middle line:84% All right, and so the consequence of this 00:26:33.730 --> 00:26:36.250 align:middle line:84% is that we can actually apply the retinex 00:26:36.250 --> 00:26:40.010 align:middle line:84% algorithm to real images and see what it does. 00:26:40.010 --> 00:26:41.530 align:middle line:90% And it does funny things. 00:26:41.530 --> 00:26:43.010 align:middle line:84% And this gets at the question that 00:26:43.010 --> 00:26:45.635 align:middle line:84% was asked earlier of the extent to which these assumptions that 00:26:45.635 --> 00:26:49.090 align:middle line:84% are embodied in the algorithm are really accurate. 00:26:49.090 --> 00:26:52.250 align:middle line:84% So here, we have an image that's just 00:26:52.250 --> 00:26:55.970 align:middle line:90% taken of a toy in the world. 00:26:55.970 --> 00:26:58.910 align:middle line:84% And you can apply the retinex algorithm, 00:26:58.910 --> 00:27:01.450 align:middle line:84% and it outputs an illumination estimate 00:27:01.450 --> 00:27:03.050 align:middle line:90% and a reflectance estimate. 00:27:03.050 --> 00:27:05.690 align:middle line:90% 00:27:05.690 --> 00:27:10.370 align:middle line:84% So the reflectance estimate looks pretty reasonable. 00:27:10.370 --> 00:27:13.650 align:middle line:84% So the tires are dark, and the hair is dark. 00:27:13.650 --> 00:27:16.990 align:middle line:84% And the bucket is pretty dark, and the shirt is lighter. 00:27:16.990 --> 00:27:18.830 align:middle line:90% The background is lighter right. 00:27:18.830 --> 00:27:20.850 align:middle line:90% It looks OK. 00:27:20.850 --> 00:27:23.230 align:middle line:84% The illumination estimate, on the other hand, 00:27:23.230 --> 00:27:25.970 align:middle line:90% looks pretty funny. 00:27:25.970 --> 00:27:28.970 align:middle line:84% And so some of what's happening here 00:27:28.970 --> 00:27:31.790 align:middle line:84% is some of the luminance variation, 00:27:31.790 --> 00:27:36.370 align:middle line:84% like, for instance, on the tires is not due to illumination, 00:27:36.370 --> 00:27:40.090 align:middle line:84% but rather, it's due to shape, so the surface orientation 00:27:40.090 --> 00:27:43.690 align:middle line:84% changing and, as a consequence, there being less reflected 00:27:43.690 --> 00:27:45.210 align:middle line:90% light. 00:27:45.210 --> 00:27:47.090 align:middle line:84% Now that happens gradually because it's just 00:27:47.090 --> 00:27:49.830 align:middle line:84% a curved surface, and the shape changes gradually, 00:27:49.830 --> 00:27:52.870 align:middle line:84% and so that produces gradual changes in luminance. 00:27:52.870 --> 00:27:58.050 align:middle line:84% And because the world model that is implicit in retinex really 00:27:58.050 --> 00:28:00.370 align:middle line:84% only knows about illumination and reflectance. 00:28:00.370 --> 00:28:03.390 align:middle line:84% It doesn't know anything about three-dimensional shape. 00:28:03.390 --> 00:28:07.530 align:middle line:84% It assumes that every luminance change that happens in the image 00:28:07.530 --> 00:28:10.370 align:middle line:84% is either due to illumination or reflectance. 00:28:10.370 --> 00:28:12.630 align:middle line:84% And that's not true of the world. 00:28:12.630 --> 00:28:14.410 align:middle line:84% There's luminance variation due to lots 00:28:14.410 --> 00:28:17.450 align:middle line:84% of other causal factors, three-dimensional shape 00:28:17.450 --> 00:28:21.220 align:middle line:90% being a big one of them. 00:28:21.220 --> 00:28:26.720 align:middle line:84% So on the other hand, it is getting this right. 00:28:26.720 --> 00:28:28.700 align:middle line:84% This looks like maybe there was sunlight 00:28:28.700 --> 00:28:33.000 align:middle line:90% through a window or something. 00:28:33.000 --> 00:28:35.940 align:middle line:84% But part of the importance of these things being stimulus 00:28:35.940 --> 00:28:38.340 align:middle line:84% computable is that you can test them on real images, 00:28:38.340 --> 00:28:39.400 align:middle line:90% and they break. 00:28:39.400 --> 00:28:43.100 align:middle line:84% And then you get insight into what's missing from the theory. 00:28:43.100 --> 00:28:46.740 align:middle line:84% OK, so what's wrong with retinex-style theories 00:28:46.740 --> 00:28:48.420 align:middle line:90% of lightness constancy? 00:28:48.420 --> 00:28:52.260 align:middle line:84% So one of the big things that's wrong-- 00:28:52.260 --> 00:28:54.620 align:middle line:84% and so I just alluded to something else that's 00:28:54.620 --> 00:28:56.300 align:middle line:84% wrong about it, which is the world model 00:28:56.300 --> 00:28:58.955 align:middle line:84% is incomplete, that luminance changes 00:28:58.955 --> 00:29:00.080 align:middle line:90% can come from other things. 00:29:00.080 --> 00:29:03.260 align:middle line:84% But even if you accept the idea that you just 00:29:03.260 --> 00:29:05.980 align:middle line:84% have reflectance and illumination, 00:29:05.980 --> 00:29:09.060 align:middle line:84% this other big problem is that illumination changes are not 00:29:09.060 --> 00:29:10.500 align:middle line:90% always gradual. 00:29:10.500 --> 00:29:13.500 align:middle line:84% Sometimes they're abrupt, and so shadows in particular 00:29:13.500 --> 00:29:16.670 align:middle line:90% often have fairly crisp edges. 00:29:16.670 --> 00:29:22.470 align:middle line:84% And evidence that retinex is incomplete as an explanation 00:29:22.470 --> 00:29:24.950 align:middle line:84% of lightness perception comes from the fact 00:29:24.950 --> 00:29:28.630 align:middle line:84% that humans appear to discount illumination 00:29:28.630 --> 00:29:31.750 align:middle line:84% despite the absence of gradual change in some cases. 00:29:31.750 --> 00:29:36.830 align:middle line:84% And so there were these early illusions from the '70s that 00:29:36.830 --> 00:29:38.370 align:middle line:90% seemed consistent with retinex. 00:29:38.370 --> 00:29:40.870 align:middle line:84% And then there was another wave of illusions 00:29:40.870 --> 00:29:45.510 align:middle line:84% in the '90s and 2000s that kind of revealed the limitations. 00:29:45.510 --> 00:29:47.190 align:middle line:90% So this is one example. 00:29:47.190 --> 00:29:50.010 align:middle line:84% And a lot of this work was done by Ted Adelson, 00:29:50.010 --> 00:29:53.090 align:middle line:84% who is a professor in this department in CSAIL. 00:29:53.090 --> 00:29:55.070 align:middle line:90% He was my PhD advisor. 00:29:55.070 --> 00:29:57.970 align:middle line:84% And this was what he was working on back at that period in time. 00:29:57.970 --> 00:30:01.190 align:middle line:84% So this is one example called the tips and strips illusion. 00:30:01.190 --> 00:30:04.910 align:middle line:84% So we have tips and strips, and it's an illusion 00:30:04.910 --> 00:30:08.710 align:middle line:84% because all of these gray regions 00:30:08.710 --> 00:30:11.190 align:middle line:84% are physically the same shade of gray. 00:30:11.190 --> 00:30:14.830 align:middle line:84% But when you look at this, I mean, these look dark, 00:30:14.830 --> 00:30:18.590 align:middle line:90% and these look light. 00:30:18.590 --> 00:30:22.550 align:middle line:84% And one explanation for what's going on 00:30:22.550 --> 00:30:26.390 align:middle line:84% is that there is evidence that this region is 00:30:26.390 --> 00:30:32.630 align:middle line:84% in high illumination and this is in shadow. 00:30:32.630 --> 00:30:35.110 align:middle line:84% But you can see that the border between these two 00:30:35.110 --> 00:30:38.930 align:middle line:84% potentially different regions of illumination is quite abrupt. 00:30:38.930 --> 00:30:42.070 align:middle line:84% So if you gave this to retinex, what 00:30:42.070 --> 00:30:43.810 align:middle line:90% would the retinex algorithm do? 00:30:43.810 --> 00:30:51.190 align:middle line:90% 00:30:51.190 --> 00:30:51.910 align:middle line:90% Yeah. 00:30:51.910 --> 00:30:54.830 align:middle line:84% AUDIENCE: It would assume that it's all reflectance, not 00:30:54.830 --> 00:30:56.450 align:middle line:90% due to different illumination. 00:30:56.450 --> 00:30:57.700 align:middle line:90% JOSH MCDERMOTT: Exactly, yeah. 00:30:57.700 --> 00:30:59.930 align:middle line:84% So it would say, this is all reflectance. 00:30:59.930 --> 00:31:03.470 align:middle line:84% Now in actuality, this is all reflectance. 00:31:03.470 --> 00:31:07.510 align:middle line:84% This is just an image that was created in Photoshop 00:31:07.510 --> 00:31:09.930 align:middle line:90% or something, but, yeah. 00:31:09.930 --> 00:31:13.190 align:middle line:84% So it would not do what the human visual system is doing, 00:31:13.190 --> 00:31:16.760 align:middle line:84% which is essentially inferring that the reflectance is lower 00:31:16.760 --> 00:31:19.880 align:middle line:90% here than it is here. 00:31:19.880 --> 00:31:24.240 align:middle line:84% This is another one with the same idea, the snake illusion. 00:31:24.240 --> 00:31:27.920 align:middle line:84% So the diamonds there are always the same gray level. 00:31:27.920 --> 00:31:33.000 align:middle line:84% And so you can have this variant here where you essentially 00:31:33.000 --> 00:31:34.660 align:middle line:90% just get simultaneous contrast. 00:31:34.660 --> 00:31:37.240 align:middle line:84% So there's a little bit of difference between the lightness 00:31:37.240 --> 00:31:38.300 align:middle line:90% of this and this. 00:31:38.300 --> 00:31:40.720 align:middle line:84% But then this version here where things are just 00:31:40.720 --> 00:31:42.880 align:middle line:84% tweaked a little bit, and now there's 00:31:42.880 --> 00:31:46.600 align:middle line:84% a pretty big difference between the two sets of diamonds. 00:31:46.600 --> 00:31:48.662 align:middle line:84% And so again, the idea is that there's 00:31:48.662 --> 00:31:50.620 align:middle line:84% some evidence that this is in high illumination 00:31:50.620 --> 00:31:52.320 align:middle line:90% and that's in low illumination. 00:31:52.320 --> 00:31:55.040 align:middle line:84% So the same physical shade of gray 00:31:55.040 --> 00:31:58.360 align:middle line:84% would be explained as there being 00:31:58.360 --> 00:32:01.760 align:middle line:84% low reflectance and high illumination here and high 00:32:01.760 --> 00:32:05.120 align:middle line:84% reflectance and low illumination there, so the Helmholtzian 00:32:05.120 --> 00:32:07.880 align:middle line:84% explanation we talked about earlier. 00:32:07.880 --> 00:32:12.240 align:middle line:84% So this also made it onto a T-shirt. 00:32:12.240 --> 00:32:14.400 align:middle line:84% So this is a T-shirt that I happen 00:32:14.400 --> 00:32:20.120 align:middle line:84% to own that was printed up 25 years ago or something. 00:32:20.120 --> 00:32:22.320 align:middle line:84% And this is a particularly cool version 00:32:22.320 --> 00:32:26.900 align:middle line:84% of this because these phenomena that we're talking about, 00:32:26.900 --> 00:32:30.400 align:middle line:84% they cause this to look darker and not to look lighter. 00:32:30.400 --> 00:32:32.920 align:middle line:84% But in actuality, there's just some dark stuff 00:32:32.920 --> 00:32:35.420 align:middle line:84% and some light stuff that's printed on the same color cloth. 00:32:35.420 --> 00:32:37.760 align:middle line:84% So this and this are just physically the same stuff. 00:32:37.760 --> 00:32:41.100 align:middle line:84% So you can tell when you look at it it's the same cloth, 00:32:41.100 --> 00:32:44.920 align:middle line:84% but it ends up looking different. 00:32:44.920 --> 00:32:49.280 align:middle line:84% All right, so how can we explain this kind of stuff? 00:32:49.280 --> 00:32:52.040 align:middle line:84% So I'll start by just saying we don't actually 00:32:52.040 --> 00:32:56.240 align:middle line:84% have complete stimulus-computable explanations 00:32:56.240 --> 00:32:59.440 align:middle line:84% that can work on all of these classes of illusions. 00:32:59.440 --> 00:33:02.120 align:middle line:84% That is something that would be good to revisit. 00:33:02.120 --> 00:33:05.120 align:middle line:84% But all of the pieces of evidence that we have 00:33:05.120 --> 00:33:09.920 align:middle line:84% suggest a potential alternative approach. 00:33:09.920 --> 00:33:13.930 align:middle line:84% And one such approach is to not try to estimate illumination 00:33:13.930 --> 00:33:17.650 align:middle line:84% explicitly but rather try to find boundaries between regions 00:33:17.650 --> 00:33:20.570 align:middle line:84% that are likely to be in different illumination 00:33:20.570 --> 00:33:23.370 align:middle line:84% and then to look at the range of luminance values 00:33:23.370 --> 00:33:26.455 align:middle line:84% that you have within these regions that are plausibly 00:33:26.455 --> 00:33:27.830 align:middle line:84% different regions of illumination 00:33:27.830 --> 00:33:32.410 align:middle line:84% and then try to map that onto expected ranges of reflectance. 00:33:32.410 --> 00:33:35.530 align:middle line:84% So I'll just give you a little flavor of how this could work. 00:33:35.530 --> 00:33:39.770 align:middle line:84% So again, let's think about the world consisting 00:33:39.770 --> 00:33:42.170 align:middle line:84% of a set of random gray patches that are viewed 00:33:42.170 --> 00:33:43.750 align:middle line:90% in different illuminations. 00:33:43.750 --> 00:33:46.170 align:middle line:84% And so if we have some patches that 00:33:46.170 --> 00:33:49.770 align:middle line:84% can range from black to white in terms of reflectance, 00:33:49.770 --> 00:33:52.050 align:middle line:84% in high illumination, that would create 00:33:52.050 --> 00:33:55.250 align:middle line:84% a big range of luminance values from low to high. 00:33:55.250 --> 00:33:58.090 align:middle line:84% In lower illumination, that would create a smaller range 00:33:58.090 --> 00:34:00.250 align:middle line:84% of luminance values because you get 00:34:00.250 --> 00:34:01.810 align:middle line:84% smaller numbers that are multiplying 00:34:01.810 --> 00:34:03.970 align:middle line:90% the same reflectances. 00:34:03.970 --> 00:34:05.610 align:middle line:84% There are also some viewing conditions 00:34:05.610 --> 00:34:07.470 align:middle line:84% where there's an additive component, 00:34:07.470 --> 00:34:09.659 align:middle line:84% if you're viewing something through fog, 00:34:09.659 --> 00:34:12.080 align:middle line:84% and then that causes everything to get shifted up. 00:34:12.080 --> 00:34:14.980 align:middle line:84% That's another interesting wrinkle. 00:34:14.980 --> 00:34:17.020 align:middle line:84% And this is what these would actually look like. 00:34:17.020 --> 00:34:21.719 align:middle line:84% So we've got this set of random gray patches here. 00:34:21.719 --> 00:34:23.980 align:middle line:90% They're stripes. 00:34:23.980 --> 00:34:27.620 align:middle line:84% And in the high-illumination condition, 00:34:27.620 --> 00:34:30.780 align:middle line:84% the reflectance here gets mapped onto this particular range 00:34:30.780 --> 00:34:32.179 align:middle line:90% of luminance. 00:34:32.179 --> 00:34:34.000 align:middle line:84% In the lower-illumination condition, 00:34:34.000 --> 00:34:37.380 align:middle line:84% it gets mapped onto a smaller and lower range 00:34:37.380 --> 00:34:38.699 align:middle line:90% of luminance values. 00:34:38.699 --> 00:34:41.780 align:middle line:84% And then here's the hazy condition where there's 00:34:41.780 --> 00:34:44.040 align:middle line:90% this-- it gets shifted up. 00:34:44.040 --> 00:34:48.739 align:middle line:90% 00:34:48.739 --> 00:34:52.340 align:middle line:84% And so the notion here is that what could be happening 00:34:52.340 --> 00:34:55.940 align:middle line:84% is that if you have some way of finding boundaries 00:34:55.940 --> 00:34:58.180 align:middle line:84% between regions that are in different illumination, 00:34:58.180 --> 00:35:02.300 align:middle line:84% you could just sample the luminance values that you're 00:35:02.300 --> 00:35:04.340 align:middle line:90% getting in a particular region. 00:35:04.340 --> 00:35:06.820 align:middle line:84% And if you assume that there's some-- 00:35:06.820 --> 00:35:12.100 align:middle line:84% that those samples come from a particular range of reflectance 00:35:12.100 --> 00:35:16.572 align:middle line:84% from light to dark, you would then take the highest luminance 00:35:16.572 --> 00:35:18.780 align:middle line:84% that you would see and map that onto something that's 00:35:18.780 --> 00:35:22.460 align:middle line:84% pretty light and the lowest luminance that you see and map 00:35:22.460 --> 00:35:24.180 align:middle line:84% that onto something that's pretty dark. 00:35:24.180 --> 00:35:25.980 align:middle line:84% And so the idea is that you're going 00:35:25.980 --> 00:35:29.020 align:middle line:84% to have these different ranges for different illumination 00:35:29.020 --> 00:35:32.260 align:middle line:84% conditions, but you would take this particular range 00:35:32.260 --> 00:35:35.567 align:middle line:84% and call this something that's pretty light and take this range 00:35:35.567 --> 00:35:37.400 align:middle line:84% and call this something that's pretty light. 00:35:37.400 --> 00:35:40.380 align:middle line:90% 00:35:40.380 --> 00:35:42.460 align:middle line:84% And again, this is all a little bit hand-wavy 00:35:42.460 --> 00:35:46.540 align:middle line:84% because we don't really have very precise implementations 00:35:46.540 --> 00:35:47.780 align:middle line:90% of this. 00:35:47.780 --> 00:35:53.860 align:middle line:84% But that's something that could potentially be developed. 00:35:53.860 --> 00:35:57.300 align:middle line:84% So the idea is that you could potentially 00:35:57.300 --> 00:36:00.563 align:middle line:84% sample the ranges of luminances that you have in an image. 00:36:00.563 --> 00:36:02.980 align:middle line:84% Ideally, you'd like to get enough samples that would cover 00:36:02.980 --> 00:36:05.300 align:middle line:90% the full range of reflectance. 00:36:05.300 --> 00:36:09.270 align:middle line:84% If you make the window in which you're sampling things larger, 00:36:09.270 --> 00:36:11.030 align:middle line:84% you're more likely to accurately sample 00:36:11.030 --> 00:36:13.930 align:middle line:84% the full range of reflectance, but on the other hand, 00:36:13.930 --> 00:36:15.950 align:middle line:84% you could mix viewing conditions. 00:36:15.950 --> 00:36:18.710 align:middle line:84% So the idea is that this whole thing only makes sense 00:36:18.710 --> 00:36:20.350 align:middle line:84% if the things that you're sampling 00:36:20.350 --> 00:36:22.050 align:middle line:90% are under the same illumination. 00:36:22.050 --> 00:36:26.150 align:middle line:84% And so here's a situation where this is pointing up 00:36:26.150 --> 00:36:27.770 align:middle line:90% in higher illumination. 00:36:27.770 --> 00:36:30.135 align:middle line:84% This is pointing down in lower illumination. 00:36:30.135 --> 00:36:32.510 align:middle line:84% And you really don't want to be comparing things that are 00:36:32.510 --> 00:36:35.770 align:middle line:90% in these two different states. 00:36:35.770 --> 00:36:39.310 align:middle line:90% 00:36:39.310 --> 00:36:43.150 align:middle line:84% And so one idea is to try-- that the visual system might 00:36:43.150 --> 00:36:46.910 align:middle line:84% be trying to collect enough samples so that you could get 00:36:46.910 --> 00:36:49.670 align:middle line:84% some sense of the range of reflectance while avoiding 00:36:49.670 --> 00:36:51.950 align:middle line:90% mixing illumination conditions. 00:36:51.950 --> 00:36:55.270 align:middle line:84% And thus, it would be taking advantage 00:36:55.270 --> 00:36:59.350 align:middle line:84% of strong cues to illumination boundaries and such 00:36:59.350 --> 00:37:01.710 align:middle line:84% that your lightness percept would be really modulated 00:37:01.710 --> 00:37:03.570 align:middle line:90% by these strong cues. 00:37:03.570 --> 00:37:06.150 align:middle line:90% 00:37:06.150 --> 00:37:08.950 align:middle line:84% So we'll come back to how that might work in a second. 00:37:08.950 --> 00:37:12.230 align:middle line:84% And so I showed you some of these examples where 00:37:12.230 --> 00:37:17.150 align:middle line:84% cues to illumination boundaries, so these junctions here 00:37:17.150 --> 00:37:20.430 align:middle line:84% that suggest that you might have, darker illumination here 00:37:20.430 --> 00:37:22.950 align:middle line:84% and high illumination here, usually 00:37:22.950 --> 00:37:24.370 align:middle line:90% produce lightness differences. 00:37:24.370 --> 00:37:27.190 align:middle line:84% You get these big illusions in cases 00:37:27.190 --> 00:37:29.630 align:middle line:84% where there's very strong evidence of illumination 00:37:29.630 --> 00:37:30.130 align:middle line:90% boundaries. 00:37:30.130 --> 00:37:34.510 align:middle line:84% And then these other cases where you have similar gray levels 00:37:34.510 --> 00:37:36.990 align:middle line:84% scattered throughout the image, but things are not 00:37:36.990 --> 00:37:40.490 align:middle line:84% set up so as to cause this sense of illumination boundaries. 00:37:40.490 --> 00:37:43.870 align:middle line:84% You don't really get these differences. 00:37:43.870 --> 00:37:48.070 align:middle line:84% So if we take this back to what we started with, so 00:37:48.070 --> 00:37:50.450 align:middle line:84% this is the classic simultaneous contrast effect, 00:37:50.450 --> 00:37:52.050 align:middle line:90% which we started out with. 00:37:52.050 --> 00:37:53.790 align:middle line:90% So you got these two squares. 00:37:53.790 --> 00:37:55.170 align:middle line:84% They look a little bit different. 00:37:55.170 --> 00:37:58.710 align:middle line:84% The one on the left is darker than the one on the right. 00:37:58.710 --> 00:38:01.710 align:middle line:84% So there's an illusion there, but it's a small effect. 00:38:01.710 --> 00:38:05.320 align:middle line:84% So here we've taken those same grade levels here 00:38:05.320 --> 00:38:07.560 align:middle line:84% but then set them up in such a way 00:38:07.560 --> 00:38:10.000 align:middle line:84% that you have all these additional cues 00:38:10.000 --> 00:38:13.280 align:middle line:84% that this and this would be in different regions 00:38:13.280 --> 00:38:14.600 align:middle line:90% of illumination. 00:38:14.600 --> 00:38:18.240 align:middle line:90% So it looks like they're-- 00:38:18.240 --> 00:38:20.820 align:middle line:84% the surface orientation is different. 00:38:20.820 --> 00:38:22.977 align:middle line:84% There's these other regions here that 00:38:22.977 --> 00:38:24.560 align:middle line:84% are consistent with the idea that this 00:38:24.560 --> 00:38:27.000 align:middle line:90% is in high illumination. 00:38:27.000 --> 00:38:29.000 align:middle line:84% And so one way to think about the fact 00:38:29.000 --> 00:38:32.760 align:middle line:84% that the illusion here is a lot stronger than the illusion here 00:38:32.760 --> 00:38:35.880 align:middle line:84% is that there's stronger evidence for an illumination 00:38:35.880 --> 00:38:36.520 align:middle line:90% edge. 00:38:36.520 --> 00:38:40.400 align:middle line:84% And so the visual system is less likely to mix up the viewing 00:38:40.400 --> 00:38:44.195 align:middle line:84% conditions whereas, in classic simultaneous contrast, 00:38:44.195 --> 00:38:45.820 align:middle line:84% that could be an illumination boundary, 00:38:45.820 --> 00:38:48.200 align:middle line:84% but you don't have very strong evidence for that. 00:38:48.200 --> 00:38:52.200 align:middle line:84% So it could be partially influenced 00:38:52.200 --> 00:38:55.600 align:middle line:90% by the stuff on the other side. 00:38:55.600 --> 00:39:01.040 align:middle line:84% So that's a hand-wavy indication of really how some of this 00:39:01.040 --> 00:39:02.510 align:middle line:90% might work. 00:39:02.510 --> 00:39:04.750 align:middle line:84% Now, you look at something like this, 00:39:04.750 --> 00:39:07.410 align:middle line:84% and it's natural to maybe suppose 00:39:07.410 --> 00:39:11.970 align:middle line:84% that the lightness computation is actually maybe influenced 00:39:11.970 --> 00:39:15.275 align:middle line:84% by a three-dimensional interpretation of the scene. 00:39:15.275 --> 00:39:16.650 align:middle line:84% So you look at this thing, and it 00:39:16.650 --> 00:39:19.830 align:middle line:84% looks like it's folded like this, 00:39:19.830 --> 00:39:24.502 align:middle line:84% with this being oriented differently than this. 00:39:24.502 --> 00:39:27.130 align:middle line:84% So is that actually what's going on? 00:39:27.130 --> 00:39:29.890 align:middle line:84% And so there have been a few examples that kind of actually 00:39:29.890 --> 00:39:34.530 align:middle line:84% suggest that we're not really using a full 3D model 00:39:34.530 --> 00:39:37.370 align:middle line:84% of surfaces but instead using something that's a little bit 00:39:37.370 --> 00:39:39.450 align:middle line:84% simpler, maybe more appropriately described 00:39:39.450 --> 00:39:40.390 align:middle line:90% as mid-level. 00:39:40.390 --> 00:39:43.130 align:middle line:90% So here's one example. 00:39:43.130 --> 00:39:49.890 align:middle line:84% So we've got these different gray patches here, 00:39:49.890 --> 00:39:51.253 align:middle line:90% shown here and here. 00:39:51.253 --> 00:39:53.170 align:middle line:84% And the things that the arrows are pointing to 00:39:53.170 --> 00:39:55.530 align:middle line:84% are always physically the same shade of gray. 00:39:55.530 --> 00:39:57.490 align:middle line:84% So up here, they look like they're 00:39:57.490 --> 00:39:59.090 align:middle line:90% roughly the same shade of gray. 00:39:59.090 --> 00:40:01.170 align:middle line:90% Down here, they don't. 00:40:01.170 --> 00:40:06.050 align:middle line:84% That looks darker than that, so that's the illusion. 00:40:06.050 --> 00:40:08.010 align:middle line:84% So in this case, this makes sense. 00:40:08.010 --> 00:40:09.250 align:middle line:90% Why does it make sense? 00:40:09.250 --> 00:40:11.930 align:middle line:84% Well, the three-dimensional shape of this surface 00:40:11.930 --> 00:40:14.190 align:middle line:84% is such that it looks like this is poking out. 00:40:14.190 --> 00:40:18.090 align:middle line:84% So this surface would be angled up, potentially towards the sun. 00:40:18.090 --> 00:40:20.890 align:middle line:84% This one is angled down, thus, in shadow. 00:40:20.890 --> 00:40:22.510 align:middle line:90% You get a lot of cues here. 00:40:22.510 --> 00:40:23.750 align:middle line:90% This is high illumination. 00:40:23.750 --> 00:40:24.743 align:middle line:90% Got a lot of cues here. 00:40:24.743 --> 00:40:25.910 align:middle line:90% This is in low illumination. 00:40:25.910 --> 00:40:28.090 align:middle line:84% All the luminance values are low. 00:40:28.090 --> 00:40:33.490 align:middle line:84% So it makes sense that there would be a difference here. 00:40:33.490 --> 00:40:37.390 align:middle line:84% All right, so this was discovered and published. 00:40:37.390 --> 00:40:41.490 align:middle line:84% And then somebody came along and came up 00:40:41.490 --> 00:40:44.010 align:middle line:84% with this variant where you can see 00:40:44.010 --> 00:40:49.550 align:middle line:84% that this has got the same shades of gray as this, 00:40:49.550 --> 00:40:51.610 align:middle line:90% but this one is reconfigured. 00:40:51.610 --> 00:40:54.150 align:middle line:84% And now the three-dimensional interpretation is different. 00:40:54.150 --> 00:40:56.807 align:middle line:84% And it looks like the orientation 00:40:56.807 --> 00:40:59.140 align:middle line:84% of this surface and this surface are actually the same-- 00:40:59.140 --> 00:41:02.460 align:middle line:90% they would both be pointing up. 00:41:02.460 --> 00:41:04.860 align:middle line:84% So logically, that should suggest 00:41:04.860 --> 00:41:08.140 align:middle line:84% that they would be in the same illumination, pointing 00:41:08.140 --> 00:41:11.180 align:middle line:84% in the same direction, about the same place. 00:41:11.180 --> 00:41:13.440 align:middle line:84% But the lightness difference persists, 00:41:13.440 --> 00:41:17.700 align:middle line:84% so that looks darker than that by about the same amount that 00:41:17.700 --> 00:41:19.420 align:middle line:90% looks darker than that. 00:41:19.420 --> 00:41:25.420 align:middle line:84% So that's arguing against this idea that you're using this 00:41:25.420 --> 00:41:32.340 align:middle line:84% model of 3D shape to determine whether or not the illumination 00:41:32.340 --> 00:41:34.620 align:middle line:90% is likely to be the same or not. 00:41:34.620 --> 00:41:37.860 align:middle line:84% Instead, it suggests that there's something simpler 00:41:37.860 --> 00:41:43.640 align:middle line:84% going on where you're using some local cues and again, 00:41:43.640 --> 00:41:46.780 align:middle line:84% in an unconscious way, supposing that this stuff all 00:41:46.780 --> 00:41:49.320 align:middle line:84% has high illumination and this has low. 00:41:49.320 --> 00:41:50.980 align:middle line:90% Yeah. 00:41:50.980 --> 00:41:54.220 align:middle line:84% AUDIENCE: Doesn't that feel like a super convincing case 00:41:54.220 --> 00:41:58.190 align:middle line:84% where it's like everybody's going to view this as having 00:41:58.190 --> 00:41:59.190 align:middle line:90% the same illumination? 00:41:59.190 --> 00:42:01.970 align:middle line:84% You can also see it as something flat 00:42:01.970 --> 00:42:07.910 align:middle line:84% and then going up and then going down, to the right? 00:42:07.910 --> 00:42:11.790 align:middle line:84% So it feels like both are maybe possible. 00:42:11.790 --> 00:42:14.090 align:middle line:84% JOSH MCDERMOTT: Yeah, that's a good point. 00:42:14.090 --> 00:42:19.670 align:middle line:84% Yeah, that's, potentially, an alternative explanation. 00:42:19.670 --> 00:42:22.830 align:middle line:84% AUDIENCE: Is this super convincing in terms 00:42:22.830 --> 00:42:25.062 align:middle line:90% of discounting the 3D case? 00:42:25.062 --> 00:42:27.270 align:middle line:84% JOSH MCDERMOTT: All right, yeah, so we have a skeptic 00:42:27.270 --> 00:42:31.070 align:middle line:84% in the audience who's arguing that this maybe is not the most 00:42:31.070 --> 00:42:36.390 align:middle line:84% compelling evidence that 3D shape is not being used. 00:42:36.390 --> 00:42:40.590 align:middle line:84% I mean, I think one question that you could ask yourself is, 00:42:40.590 --> 00:42:43.710 align:middle line:84% if you look at this and the 3D shape is bistable and you can 00:42:43.710 --> 00:42:48.430 align:middle line:84% see it in a few different ways, does the perceived lightness 00:42:48.430 --> 00:42:51.110 align:middle line:84% actually change when your three-dimensional interpretation 00:42:51.110 --> 00:42:51.730 align:middle line:90% changes? 00:42:51.730 --> 00:42:56.880 align:middle line:84% So for me, I see both of those as is pointing up. 00:42:56.880 --> 00:42:59.130 align:middle line:84% I get a pretty good lightness difference, nonetheless. 00:42:59.130 --> 00:43:02.950 align:middle line:90% 00:43:02.950 --> 00:43:03.770 align:middle line:90% But we will see-- 00:43:03.770 --> 00:43:05.650 align:middle line:84% I mean, in the next lecture on color, 00:43:05.650 --> 00:43:09.350 align:middle line:84% I will show you some cases where three-dimensional structure 00:43:09.350 --> 00:43:13.390 align:middle line:84% actually does seem to actually influence color in some cases. 00:43:13.390 --> 00:43:17.570 align:middle line:84% So I would say none of this stuff is like perfectly settled. 00:43:17.570 --> 00:43:21.390 align:middle line:84% It's more just like, this is some evidence 00:43:21.390 --> 00:43:24.550 align:middle line:84% that this could be based on some local heuristics 00:43:24.550 --> 00:43:27.970 align:middle line:84% rather than something related to the global interpretation. 00:43:27.970 --> 00:43:30.440 align:middle line:90% 00:43:30.440 --> 00:43:32.190 align:middle line:84% So the thought is that a lot of the action 00:43:32.190 --> 00:43:34.510 align:middle line:84% may be happening at a mid-level representation using 00:43:34.510 --> 00:43:35.810 align:middle line:90% sensible heuristics. 00:43:35.810 --> 00:43:37.470 align:middle line:84% Here's another piece of evidence that 00:43:37.470 --> 00:43:39.670 align:middle line:90% maybe is consistent with that. 00:43:39.670 --> 00:43:44.470 align:middle line:84% So here we've got a pretty strong illusion 00:43:44.470 --> 00:43:47.510 align:middle line:84% where this looks darker than this same physical shade 00:43:47.510 --> 00:43:48.350 align:middle line:90% of gray. 00:43:48.350 --> 00:43:51.390 align:middle line:84% Again, same idea, a lot of evidence for an illumination 00:43:51.390 --> 00:43:53.110 align:middle line:90% boundary here. 00:43:53.110 --> 00:43:56.023 align:middle line:84% This is consistent with high illumination, this, 00:43:56.023 --> 00:43:56.940 align:middle line:90% with low illumination. 00:43:56.940 --> 00:43:58.720 align:middle line:84% And so under the story we were telling, 00:43:58.720 --> 00:44:01.780 align:middle line:84% you got evidence for illumination boundaries, 00:44:01.780 --> 00:44:06.000 align:middle line:84% and so within this region, this is the lowest luminance value. 00:44:06.000 --> 00:44:09.420 align:middle line:84% And thus, you would map that onto something dark 00:44:09.420 --> 00:44:11.860 align:middle line:84% whereas this is the highest luminance value, 00:44:11.860 --> 00:44:13.480 align:middle line:90% and that would look light. 00:44:13.480 --> 00:44:18.200 align:middle line:84% Now, in this region, this is now the highest luminance value. 00:44:18.200 --> 00:44:21.060 align:middle line:84% And so that would get mapped onto something that is light. 00:44:21.060 --> 00:44:23.220 align:middle line:84% And in this region, these are darker and so forth. 00:44:23.220 --> 00:44:25.600 align:middle line:84% So again, you don't need to explicitly estimate 00:44:25.600 --> 00:44:27.800 align:middle line:84% the high and the low illumination necessarily. 00:44:27.800 --> 00:44:29.560 align:middle line:84% All right, so that all makes sense. 00:44:29.560 --> 00:44:33.320 align:middle line:84% Now, in the one on the right, everything is the same. 00:44:33.320 --> 00:44:36.600 align:middle line:84% It's just that we've moved these ovals further away. 00:44:36.600 --> 00:44:40.800 align:middle line:84% So these kind of local cues here for the illumination boundary 00:44:40.800 --> 00:44:43.240 align:middle line:84% are pushed further away from the diamonds. 00:44:43.240 --> 00:44:47.680 align:middle line:84% And the argument is that the illusion gets weaker. 00:44:47.680 --> 00:44:51.360 align:middle line:84% So the argument is that the difference between these two 00:44:51.360 --> 00:44:57.920 align:middle line:84% diamonds is not as pronounced as it is here. 00:44:57.920 --> 00:45:00.520 align:middle line:84% And again, logically, the evidence 00:45:00.520 --> 00:45:03.760 align:middle line:84% that there is an illumination boundary here 00:45:03.760 --> 00:45:08.560 align:middle line:84% is just as high in this image as it is in that one. 00:45:08.560 --> 00:45:09.900 align:middle line:90% You got all the same stuff. 00:45:09.900 --> 00:45:11.500 align:middle line:90% It just got moved further away. 00:45:11.500 --> 00:45:14.935 align:middle line:90% 00:45:14.935 --> 00:45:19.200 align:middle line:84% So it suggests that there's some kind 00:45:19.200 --> 00:45:26.760 align:middle line:84% of limited spatial propagation that goes on and argues 00:45:26.760 --> 00:45:31.120 align:middle line:84% against a high-level cognitive interpretation of illumination 00:45:31.120 --> 00:45:32.520 align:middle line:90% boundaries. 00:45:32.520 --> 00:45:34.427 align:middle line:90% Does that make sense? 00:45:34.427 --> 00:45:36.760 align:middle line:84% AUDIENCE: Why does it argue against high-level cognitive 00:45:36.760 --> 00:45:37.920 align:middle line:90% interpretation? 00:45:37.920 --> 00:45:40.720 align:middle line:84% JOSH MCDERMOTT: Because if I were reasoning about this-- 00:45:40.720 --> 00:45:44.880 align:middle line:84% the evidence that there is an illumination boundary here is-- 00:45:44.880 --> 00:45:47.360 align:middle line:84% there's the same evidence here as here. 00:45:47.360 --> 00:45:49.485 align:middle line:84% You got all the same local features. 00:45:49.485 --> 00:45:51.610 align:middle line:84% They just got moved further away from the diamonds. 00:45:51.610 --> 00:45:55.650 align:middle line:84% So there's some effect of the distance 00:45:55.650 --> 00:45:58.490 align:middle line:84% that I think you wouldn't necessarily expect in that case. 00:45:58.490 --> 00:46:01.993 align:middle line:84% AUDIENCE: Because distance has to do with mid-level? 00:46:01.993 --> 00:46:03.910 align:middle line:84% JOSH MCDERMOTT: Well, no, it's just that the-- 00:46:03.910 --> 00:46:08.130 align:middle line:90% 00:46:08.130 --> 00:46:11.270 align:middle line:90% you got this straight line here. 00:46:11.270 --> 00:46:14.730 align:middle line:84% So the question is, what causes that straight line? 00:46:14.730 --> 00:46:16.930 align:middle line:84% And you've got these bits of local evidence 00:46:16.930 --> 00:46:19.310 align:middle line:84% that that's due to an illumination boundary 00:46:19.310 --> 00:46:22.570 align:middle line:84% because you've got these x junctions here. 00:46:22.570 --> 00:46:25.650 align:middle line:84% And all that same evidence is here and here. 00:46:25.650 --> 00:46:29.810 align:middle line:84% And so, yeah, I mean, certainly, if we were reasoning about it 00:46:29.810 --> 00:46:32.370 align:middle line:84% logically, there's as much evidence 00:46:32.370 --> 00:46:33.710 align:middle line:90% in one case as the other. 00:46:33.710 --> 00:46:35.418 align:middle line:84% AUDIENCE: So you're saying that mid-level 00:46:35.418 --> 00:46:39.382 align:middle line:84% has to do with local things, like physiological effects? 00:46:39.382 --> 00:46:40.590 align:middle line:90% JOSH MCDERMOTT: Very vaguely. 00:46:40.590 --> 00:46:46.110 align:middle line:90% Yeah, it's very underspecified. 00:46:46.110 --> 00:46:46.610 align:middle line:90% Yeah. 00:46:46.610 --> 00:46:51.380 align:middle line:90% 00:46:51.380 --> 00:46:56.140 align:middle line:84% OK, so if you take a lot of different cues 00:46:56.140 --> 00:47:01.280 align:middle line:84% to illumination, different regions of illumination, 00:47:01.280 --> 00:47:05.020 align:middle line:84% and you combine them, you can get very powerful illusions. 00:47:05.020 --> 00:47:08.377 align:middle line:84% So here, we've got these ovals on the top and these ovals 00:47:08.377 --> 00:47:08.960 align:middle line:90% on the bottom. 00:47:08.960 --> 00:47:10.900 align:middle line:84% Those are the same physical shade of gray, 00:47:10.900 --> 00:47:14.980 align:middle line:84% but they look like they're really, really different. 00:47:14.980 --> 00:47:16.940 align:middle line:84% And there's lots of cues that are all 00:47:16.940 --> 00:47:18.600 align:middle line:90% pointing in the same direction. 00:47:18.600 --> 00:47:20.120 align:middle line:90% So there's local contrast. 00:47:20.120 --> 00:47:22.360 align:middle line:84% So this is by far the darkest thing up here. 00:47:22.360 --> 00:47:25.100 align:middle line:84% This is by far the lightest thing down here. 00:47:25.100 --> 00:47:27.720 align:middle line:84% You've got this thing called a penumbra. 00:47:27.720 --> 00:47:30.440 align:middle line:84% This is the edge of a shadow, which tends to be blurry, 00:47:30.440 --> 00:47:32.957 align:middle line:84% which indicates that this is in shadow. 00:47:32.957 --> 00:47:34.540 align:middle line:84% There are these three-dimensional cues 00:47:34.540 --> 00:47:36.900 align:middle line:84% to depth, which may or may not matter, 00:47:36.900 --> 00:47:40.020 align:middle line:84% that would suggest that this is on the underside of a cylinder 00:47:40.020 --> 00:47:42.300 align:middle line:90% and this is on the top. 00:47:42.300 --> 00:47:44.738 align:middle line:84% There's a little bit of a specular highlight 00:47:44.738 --> 00:47:46.780 align:middle line:84% that indicates that there's a light source that's 00:47:46.780 --> 00:47:47.860 align:middle line:90% coming from above. 00:47:47.860 --> 00:47:49.860 align:middle line:84% So there's lots of cues here, and they're all 00:47:49.860 --> 00:47:51.780 align:middle line:90% pointing in the same direction. 00:47:51.780 --> 00:47:54.540 align:middle line:84% And that gives you a pretty big effect. 00:47:54.540 --> 00:47:58.280 align:middle line:84% And this is another one with the same idea. 00:47:58.280 --> 00:48:00.860 align:middle line:84% So the arrows here are all the same, 00:48:00.860 --> 00:48:02.440 align:middle line:84% but they look very, very different. 00:48:02.440 --> 00:48:06.460 align:middle line:90% 00:48:06.460 --> 00:48:08.020 align:middle line:90% This one is also pretty cool. 00:48:08.020 --> 00:48:11.740 align:middle line:84% So the two blocks are actually the same color. 00:48:11.740 --> 00:48:15.000 align:middle line:84% So if you put your fingers covering this up, 00:48:15.000 --> 00:48:16.360 align:middle line:90% you'll be able to see that. 00:48:16.360 --> 00:48:22.060 align:middle line:90% 00:48:22.060 --> 00:48:25.980 align:middle line:90% So just to summarize. 00:48:25.980 --> 00:48:30.140 align:middle line:84% Many surfaces have approximately Lambertian reflectance, though, 00:48:30.140 --> 00:48:34.260 align:middle line:90% as we talked about, not all. 00:48:34.260 --> 00:48:37.100 align:middle line:84% So that means that reflectance can be summarized as one number. 00:48:37.100 --> 00:48:40.900 align:middle line:84% Lightness is the perceptual correlate of reflectance. 00:48:40.900 --> 00:48:43.580 align:middle line:84% Lightness illusions are probably caused by processes 00:48:43.580 --> 00:48:45.180 align:middle line:90% at multiple levels-- 00:48:45.180 --> 00:48:47.550 align:middle line:84% low level, mid-level, and high level. 00:48:47.550 --> 00:48:50.030 align:middle line:84% Although, exactly what that means 00:48:50.030 --> 00:48:52.590 align:middle line:84% and how those things operate and getting 00:48:52.590 --> 00:48:56.190 align:middle line:84% complete theories of this is still an ongoing thing. 00:48:56.190 --> 00:48:59.830 align:middle line:84% We talked about retinex as one example of an influential model, 00:48:59.830 --> 00:49:01.510 align:middle line:84% illustrates some important principles 00:49:01.510 --> 00:49:04.330 align:middle line:90% that probably have some truth. 00:49:04.330 --> 00:49:08.910 align:middle line:84% The idea that abrupt edges have a big influence 00:49:08.910 --> 00:49:11.330 align:middle line:84% on perceived lightness and reflectance estimates, 00:49:11.330 --> 00:49:13.850 align:middle line:84% but it's pretty clearly not fully, fully correct. 00:49:13.850 --> 00:49:15.990 align:middle line:84% In particular, the main shortcoming 00:49:15.990 --> 00:49:19.870 align:middle line:84% is that it assumes that illumination varies smoothly, 00:49:19.870 --> 00:49:21.870 align:middle line:90% and that is not always the case. 00:49:21.870 --> 00:49:27.030 align:middle line:84% And you're able to actually make reflectance estimates that 00:49:27.030 --> 00:49:31.550 align:middle line:84% seem to incorporate the effects of abrupt illumination 00:49:31.550 --> 00:49:32.870 align:middle line:90% boundaries. 00:49:32.870 --> 00:49:34.950 align:middle line:84% So we currently lack a stimulus-computable model 00:49:34.950 --> 00:49:36.990 align:middle line:84% of lightness perception that can explain 00:49:36.990 --> 00:49:38.550 align:middle line:90% most lightness illusions. 00:49:38.550 --> 00:49:41.710 align:middle line:84% We also don't have clear neural correlates of lightness. 00:49:41.710 --> 00:49:44.590 align:middle line:90% So that's an open question. 00:49:44.590 --> 00:49:48.450 align:middle line:84% So we think that a lot of the action happens at a mid-level, 00:49:48.450 --> 00:49:52.770 align:middle line:84% where 2D heuristics about grouping contours, surfaces, 00:49:52.770 --> 00:49:54.090 align:middle line:90% and junctions are being used. 00:49:54.090 --> 00:49:57.730 align:middle line:84% But as I said, I haven't given you a complete theory of this. 00:49:57.730 --> 00:50:00.350 align:middle line:84% It's just some hints as to what some of the ingredients 00:50:00.350 --> 00:50:04.910 align:middle line:84% and what that kind of explanation might look like. 00:50:04.910 --> 00:50:07.070 align:middle line:84% What questions do you have about lightness? 00:50:07.070 --> 00:50:07.830 align:middle line:90% Yeah. 00:50:07.830 --> 00:50:10.330 align:middle line:84% AUDIENCE: So some of those later illusions, 00:50:10.330 --> 00:50:12.490 align:middle line:84% like the snake cylinder and the one after it, 00:50:12.490 --> 00:50:15.570 align:middle line:84% there's a lot of squiggly lines that are being used. 00:50:15.570 --> 00:50:17.030 align:middle line:90% So is there a reason for those? 00:50:17.030 --> 00:50:19.350 align:middle line:84% Do they add to the illusion in some way? 00:50:19.350 --> 00:50:22.090 align:middle line:90% Or would it still-- 00:50:22.090 --> 00:50:22.990 align:middle line:90% I'm curious. 00:50:22.990 --> 00:50:24.970 align:middle line:84% JOSH MCDERMOTT: Like these squiggly lines? 00:50:24.970 --> 00:50:27.030 align:middle line:84% AUDIENCE: Yeah, on the cylinder too, 00:50:27.030 --> 00:50:33.710 align:middle line:84% is it to add more surrounding stuff to it [INAUDIBLE]? 00:50:33.710 --> 00:50:38.590 align:middle line:84% JOSH MCDERMOTT: OK, so here, they definitely help give you 00:50:38.590 --> 00:50:43.630 align:middle line:84% this percept of 3D shape because you can see that the squiggles 00:50:43.630 --> 00:50:46.500 align:middle line:84% are closer together here than here, 00:50:46.500 --> 00:50:50.560 align:middle line:84% which is an indication that the surface is oblique. 00:50:50.560 --> 00:50:52.800 align:middle line:90% So they'll help with that. 00:50:52.800 --> 00:50:55.780 align:middle line:84% Again, how much that actually benefits the illusion 00:50:55.780 --> 00:51:00.640 align:middle line:84% I'm not sure in this particular case, but that's one reason. 00:51:00.640 --> 00:51:02.600 align:middle line:84% It makes the three-dimensionality 00:51:02.600 --> 00:51:05.320 align:middle line:84% of the thing a little bit easier to convey 00:51:05.320 --> 00:51:08.560 align:middle line:90% in a synthetic image like this. 00:51:08.560 --> 00:51:15.260 align:middle line:84% And then in these kinds of things, 00:51:15.260 --> 00:51:20.800 align:middle line:84% I think the curves, they probably enhance the sense-- 00:51:20.800 --> 00:51:22.360 align:middle line:84% they probably strengthen the evidence 00:51:22.360 --> 00:51:24.960 align:middle line:90% for the illumination boundary. 00:51:24.960 --> 00:51:27.240 align:middle line:84% If you made these things rectangular, 00:51:27.240 --> 00:51:32.320 align:middle line:84% I think it'd probably be easier to explain this 00:51:32.320 --> 00:51:35.300 align:middle line:84% as two different things that happen to be placed down. 00:51:35.300 --> 00:51:38.720 align:middle line:84% That's just my intuition, but that's just 00:51:38.720 --> 00:51:40.100 align:middle line:90% a hand-wavy intuition. 00:51:40.100 --> 00:51:42.460 align:middle line:84% So, yeah, my guess is that it helps a little bit, 00:51:42.460 --> 00:51:44.930 align:middle line:90% but, yeah, I don't know. 00:51:44.930 --> 00:51:48.010 align:middle line:90% Any other questions? 00:51:48.010 --> 00:51:49.990 align:middle line:84% We're now going to talk about color. 00:51:49.990 --> 00:51:52.490 align:middle line:84% So I'll talk about this for the rest of the lecture and then 00:51:52.490 --> 00:51:54.530 align:middle line:90% probably next time. 00:51:54.530 --> 00:51:57.490 align:middle line:90% So the world is full of color. 00:51:57.490 --> 00:52:00.190 align:middle line:84% So this is different types of fruit. 00:52:00.190 --> 00:52:02.010 align:middle line:90% They're different colors. 00:52:02.010 --> 00:52:04.103 align:middle line:84% Color is really useful for lots of things. 00:52:04.103 --> 00:52:05.770 align:middle line:84% You can get insight into this by looking 00:52:05.770 --> 00:52:08.450 align:middle line:84% at images that are not colored and then 00:52:08.450 --> 00:52:10.150 align:middle line:90% switching to colored versions. 00:52:10.150 --> 00:52:13.850 align:middle line:84% So it's a little unclear what is in that plant. 00:52:13.850 --> 00:52:18.970 align:middle line:84% And then you add color, and you can see the fruit in the bush. 00:52:18.970 --> 00:52:21.890 align:middle line:84% Here's two different versions of the same image, 00:52:21.890 --> 00:52:23.810 align:middle line:84% black and white version and a color version. 00:52:23.810 --> 00:52:25.728 align:middle line:84% And it's much, much easier to see 00:52:25.728 --> 00:52:27.270 align:middle line:84% the flowers in the color version than 00:52:27.270 --> 00:52:28.562 align:middle line:90% in the black-and-white version. 00:52:28.562 --> 00:52:32.970 align:middle line:84% So color serves to help us segregate objects. 00:52:32.970 --> 00:52:35.410 align:middle line:84% Also helps us with object recognition. 00:52:35.410 --> 00:52:40.290 align:middle line:84% So if you had to identify what type of fruit 00:52:40.290 --> 00:52:42.310 align:middle line:84% our former president is eating here, 00:52:42.310 --> 00:52:44.090 align:middle line:90% you would have a hard time. 00:52:44.090 --> 00:52:46.210 align:middle line:84% But when we add color to the image, 00:52:46.210 --> 00:52:50.410 align:middle line:84% it becomes a lot easier to make out. 00:52:50.410 --> 00:52:55.450 align:middle line:84% Color also gives us information about object state. 00:52:55.450 --> 00:53:00.430 align:middle line:84% So those different pieces of fruit all look pretty similar, 00:53:00.430 --> 00:53:03.810 align:middle line:84% but some are a lot riper than others. 00:53:03.810 --> 00:53:06.710 align:middle line:84% Similarly, depending on your taste, 00:53:06.710 --> 00:53:10.970 align:middle line:84% you'd probably want to have some of these pieces of steak and not 00:53:10.970 --> 00:53:13.370 align:middle line:90% others. 00:53:13.370 --> 00:53:15.650 align:middle line:84% So it tells us about object state. 00:53:15.650 --> 00:53:19.570 align:middle line:84% Color also tells us a lot about the state of the world. 00:53:19.570 --> 00:53:24.050 align:middle line:84% So here, we have two different natural scenes. 00:53:24.050 --> 00:53:26.850 align:middle line:84% They don't look all that different in black and white, 00:53:26.850 --> 00:53:30.170 align:middle line:84% but with color, you can tell that one is quite lush. 00:53:30.170 --> 00:53:32.850 align:middle line:90% The other one is pretty arid. 00:53:32.850 --> 00:53:36.130 align:middle line:84% All right, so color is really important. 00:53:36.130 --> 00:53:39.070 align:middle line:84% And it's also an important part of art. 00:53:39.070 --> 00:53:43.915 align:middle line:84% Some people have a favorite color. 00:53:43.915 --> 00:53:46.040 align:middle line:84% It's really important to how you pick your clothes. 00:53:46.040 --> 00:53:49.420 align:middle line:84% Some combinations of colors look good and bad. 00:53:49.420 --> 00:53:50.460 align:middle line:90% It's a big part of life. 00:53:50.460 --> 00:53:58.380 align:middle line:84% So color is a consequence of light. 00:53:58.380 --> 00:54:00.620 align:middle line:90% It's part of vision. 00:54:00.620 --> 00:54:02.740 align:middle line:84% And if you observe different wavelengths 00:54:02.740 --> 00:54:05.760 align:middle line:84% of light in isolation, they appear to be particular colors. 00:54:05.760 --> 00:54:07.980 align:middle line:84% So this is the spectrum of visible light, 00:54:07.980 --> 00:54:12.460 align:middle line:84% from roughly 400 to 700 nanometers. 00:54:12.460 --> 00:54:14.960 align:middle line:84% The short wavelengths, the shortest wavelengths 00:54:14.960 --> 00:54:16.960 align:middle line:84% that are visible when you view them in isolation 00:54:16.960 --> 00:54:18.242 align:middle line:90% look pretty blue. 00:54:18.242 --> 00:54:20.700 align:middle line:84% The longest wavelengths that are visible when you view them 00:54:20.700 --> 00:54:22.460 align:middle line:90% in isolation tend to look red. 00:54:22.460 --> 00:54:24.680 align:middle line:84% In between, you get this spectrum. 00:54:24.680 --> 00:54:26.460 align:middle line:84% So this is why, when you take sunlight 00:54:26.460 --> 00:54:31.340 align:middle line:84% and you pass it through a prism, you see a rainbow. 00:54:31.340 --> 00:54:33.020 align:middle line:84% All right, so different wavelengths 00:54:33.020 --> 00:54:35.280 align:middle line:84% of light in isolation appear to be particular colors. 00:54:35.280 --> 00:54:38.510 align:middle line:84% But we really believe that the function of color biologically 00:54:38.510 --> 00:54:41.710 align:middle line:84% is to provide information about pigmentation. 00:54:41.710 --> 00:54:44.350 align:middle line:84% And so these different wavelengths of light 00:54:44.350 --> 00:54:48.110 align:middle line:84% just really provide a signal for the different pigments 00:54:48.110 --> 00:54:50.550 align:middle line:90% that things can be made of. 00:54:50.550 --> 00:54:53.210 align:middle line:84% So this is the elementary school explanation of color. 00:54:53.210 --> 00:54:54.790 align:middle line:84% I call it the elementary school explanation color 00:54:54.790 --> 00:54:56.850 align:middle line:84% because this is what I learned when I was in elementary school. 00:54:56.850 --> 00:54:59.390 align:middle line:84% Maybe they have an even more sophisticated version 00:54:59.390 --> 00:55:01.670 align:middle line:90% that is popular now. 00:55:01.670 --> 00:55:04.430 align:middle line:84% But the elementary school explanation that I got 00:55:04.430 --> 00:55:08.630 align:middle line:84% is that objects appear colored if they reflect some wavelengths 00:55:08.630 --> 00:55:10.870 align:middle line:90% of light better than others. 00:55:10.870 --> 00:55:15.070 align:middle line:84% So the idea is that the light from the sun 00:55:15.070 --> 00:55:18.550 align:middle line:84% is approximately white, so it contains all wavelengths. 00:55:18.550 --> 00:55:21.630 align:middle line:84% So you get all of these different wavelengths of light 00:55:21.630 --> 00:55:24.190 align:middle line:84% that are kind of impinging on a surface. 00:55:24.190 --> 00:55:26.830 align:middle line:84% And if the object is red, it will 00:55:26.830 --> 00:55:31.630 align:middle line:84% tend to reflect the long wavelengths, the red light more 00:55:31.630 --> 00:55:33.390 align:middle line:90% than the shorter wavelengths. 00:55:33.390 --> 00:55:35.550 align:middle line:84% If the object is green, it will tend 00:55:35.550 --> 00:55:37.930 align:middle line:84% to reflect the medium wavelength, 00:55:37.930 --> 00:55:40.630 align:middle line:84% the green wavelengths more than the short and the long, 00:55:40.630 --> 00:55:43.643 align:middle line:90% and so on and so forth. 00:55:43.643 --> 00:55:45.310 align:middle line:84% How many people learned this explanation 00:55:45.310 --> 00:55:46.910 align:middle line:90% in elementary school? 00:55:46.910 --> 00:55:48.930 align:middle line:84% Yeah, so I see it's still fairly popular. 00:55:48.930 --> 00:55:52.890 align:middle line:84% And there's a lot of truth to this. 00:55:52.890 --> 00:55:56.950 align:middle line:84% So you can actually measure the spectral reflectance 00:55:56.950 --> 00:55:59.483 align:middle line:84% of different surfaces in the world. 00:55:59.483 --> 00:56:00.650 align:middle line:90% So that's what's shown here. 00:56:00.650 --> 00:56:02.567 align:middle line:84% So what this is a graph of is it's showing you 00:56:02.567 --> 00:56:07.190 align:middle line:84% for six different kinds of surfaces of the relative energy 00:56:07.190 --> 00:56:10.270 align:middle line:84% that they reflect as a function of wavelength. 00:56:10.270 --> 00:56:12.210 align:middle line:84% So for the things that don't look colored, 00:56:12.210 --> 00:56:15.517 align:middle line:84% so the white and the gray and the black surface, they differ. 00:56:15.517 --> 00:56:17.350 align:middle line:84% So that's what we'd be studying in the world 00:56:17.350 --> 00:56:20.410 align:middle line:84% of lightness perception, so no color there. 00:56:20.410 --> 00:56:22.810 align:middle line:84% But they differ in how much light they reflect. 00:56:22.810 --> 00:56:25.250 align:middle line:84% So the black surface doesn't reflect much light. 00:56:25.250 --> 00:56:28.070 align:middle line:84% The gray surface reflects intermediate levels 00:56:28.070 --> 00:56:30.250 align:middle line:84% of light, again, across all wavelengths. 00:56:30.250 --> 00:56:32.590 align:middle line:84% And the white surface reflects a lot of light 00:56:32.590 --> 00:56:34.270 align:middle line:90% across all wavelengths. 00:56:34.270 --> 00:56:36.480 align:middle line:84% But the colored objects, the leaf 00:56:36.480 --> 00:56:39.840 align:middle line:84% and the orange in the tomato, they reflect some wavelengths 00:56:39.840 --> 00:56:42.560 align:middle line:84% a lot more than others, so some wavelengths get absorbed 00:56:42.560 --> 00:56:44.160 align:middle line:90% and others get reflected. 00:56:44.160 --> 00:56:47.680 align:middle line:84% So the tomato, which is red, is predominantly 00:56:47.680 --> 00:56:49.820 align:middle line:84% reflecting long wavelengths of light, 00:56:49.820 --> 00:56:53.360 align:middle line:84% absorbing everything below about 600 nanometers. 00:56:53.360 --> 00:56:56.160 align:middle line:84% The orange is again predominantly reflecting 00:56:56.160 --> 00:56:58.680 align:middle line:84% long waves of lengths of light but a little bit bigger 00:56:58.680 --> 00:57:00.080 align:middle line:90% range than the red thing. 00:57:00.080 --> 00:57:03.200 align:middle line:84% And the leaf, which is green, is predominantly reflecting 00:57:03.200 --> 00:57:06.942 align:middle line:90% the middle wavelength of light. 00:57:06.942 --> 00:57:08.400 align:middle line:84% All right, so this all seems pretty 00:57:08.400 --> 00:57:12.000 align:middle line:84% consistent with the elementary school theory of color. 00:57:12.000 --> 00:57:16.240 align:middle line:84% The additional wrinkle, which we will address a little bit later, 00:57:16.240 --> 00:57:19.960 align:middle line:84% is that, as in the case of lightness perception, 00:57:19.960 --> 00:57:22.840 align:middle line:84% reflected light is not just a function 00:57:22.840 --> 00:57:25.660 align:middle line:84% of the reflectance properties of an object. 00:57:25.660 --> 00:57:29.120 align:middle line:84% It's also a function of the spectrum of the illumination. 00:57:29.120 --> 00:57:31.680 align:middle line:84% So it's the same kind of problem that we talked about 00:57:31.680 --> 00:57:33.020 align:middle line:90% in lightness perception. 00:57:33.020 --> 00:57:35.160 align:middle line:84% Only now, we've got lots of different wavelengths 00:57:35.160 --> 00:57:36.100 align:middle line:90% to worry about. 00:57:36.100 --> 00:57:39.530 align:middle line:90% 00:57:39.530 --> 00:57:41.280 align:middle line:84% All right, but it all starts with the fact 00:57:41.280 --> 00:57:43.447 align:middle line:84% that there are these different wavelengths of light. 00:57:43.447 --> 00:57:48.040 align:middle line:84% And our ability to detect differences in wavelength 00:57:48.040 --> 00:57:51.280 align:middle line:84% relies on the fact that we have three different types of cones. 00:57:51.280 --> 00:57:54.240 align:middle line:84% So remember the retina, the photoreceptors 00:57:54.240 --> 00:57:56.280 align:middle line:90% are either rods or cones. 00:57:56.280 --> 00:57:59.020 align:middle line:84% The cones come in these three variants, 00:57:59.020 --> 00:58:03.380 align:middle line:84% and those are differentiated by their spectral sensitivities. 00:58:03.380 --> 00:58:05.280 align:middle line:84% So this is the spectral sensitivity 00:58:05.280 --> 00:58:07.840 align:middle line:90% of the three types of cones. 00:58:07.840 --> 00:58:11.280 align:middle line:84% The scientific term for them is S cones, M cones, and L cones. 00:58:11.280 --> 00:58:13.840 align:middle line:84% That stands for Short wavelengths, Medium wavelengths, 00:58:13.840 --> 00:58:15.320 align:middle line:90% and Long wavelengths. 00:58:15.320 --> 00:58:17.320 align:middle line:84% They're often colloquially called 00:58:17.320 --> 00:58:20.660 align:middle line:90% the blue, green, and red cones. 00:58:20.660 --> 00:58:24.760 align:middle line:84% So this graph is plotting normalized sensitivity 00:58:24.760 --> 00:58:26.680 align:middle line:90% as a function of wavelength. 00:58:26.680 --> 00:58:32.330 align:middle line:84% So you can think of these curves as reflecting the probability 00:58:32.330 --> 00:58:34.890 align:middle line:84% that one of these types of receptor 00:58:34.890 --> 00:58:38.050 align:middle line:84% will absorb a photon at a particular wavelength. 00:58:38.050 --> 00:58:40.610 align:middle line:84% So it says that, for the S cone, it's 00:58:40.610 --> 00:58:45.170 align:middle line:84% really good at absorbing 450 nanometers and not so 00:58:45.170 --> 00:58:49.550 align:middle line:84% good and, really, not absorbing at all 600 and 700 nanometers. 00:58:49.550 --> 00:58:52.082 align:middle line:90% 00:58:52.082 --> 00:58:55.970 align:middle line:84% So if you imagine that you've got a photoreceptor that's 00:58:55.970 --> 00:58:58.170 align:middle line:84% in the business of absorbing photons and generating 00:58:58.170 --> 00:59:02.110 align:middle line:84% a response, if you have a whole bunch of those receptors, 00:59:02.110 --> 00:59:05.010 align:middle line:84% so if you sum over many receptors or over time, 00:59:05.010 --> 00:59:07.130 align:middle line:84% this is going to be the magnitude of response 00:59:07.130 --> 00:59:11.210 align:middle line:84% that you would get for a given intensity of a given wavelength. 00:59:11.210 --> 00:59:17.470 align:middle line:84% So this graph is, again, showing the cone spectral sensitivities, 00:59:17.470 --> 00:59:20.110 align:middle line:84% but we've also added in the rods just for comparison. 00:59:20.110 --> 00:59:21.980 align:middle line:90% That's the dotted line. 00:59:21.980 --> 00:59:23.230 align:middle line:90% And there's only one of those. 00:59:23.230 --> 00:59:25.390 align:middle line:84% You've got three cones and one rod. 00:59:25.390 --> 00:59:28.850 align:middle line:90% 00:59:28.850 --> 00:59:32.020 align:middle line:84% So one other way to think about these curves 00:59:32.020 --> 00:59:35.780 align:middle line:84% is that if you had only one receptor type, then 00:59:35.780 --> 00:59:39.400 align:middle line:84% the receptor would determine psychophysical sensitivity. 00:59:39.400 --> 00:59:44.000 align:middle line:84% And there's a scenario in which you have only one receptor type. 00:59:44.000 --> 00:59:46.980 align:middle line:84% And that is when the light level is very low. 00:59:46.980 --> 00:59:50.560 align:middle line:84% So this is scotopic conditions, remember? 00:59:50.560 --> 00:59:53.460 align:middle line:84% So under scotopic conditions, the cones 00:59:53.460 --> 00:59:55.060 align:middle line:90% are not doing anything. 00:59:55.060 --> 00:59:57.740 align:middle line:84% You've just got this one type of rod. 00:59:57.740 --> 01:00:02.800 align:middle line:84% And so this is a graph that is comparing the rhodopsin-- 01:00:02.800 --> 01:00:06.400 align:middle line:84% so that's the molecule in the rods that absorbs photons, 01:00:06.400 --> 01:00:09.500 align:middle line:84% the absorption probabilities of that-- 01:00:09.500 --> 01:00:11.840 align:middle line:90% to human scotopic sensitivity. 01:00:11.840 --> 01:00:15.180 align:middle line:84% So scotopic sensitivity, remember, sensitivity 01:00:15.180 --> 01:00:16.840 align:middle line:90% is the inverse of a threshold. 01:00:16.840 --> 01:00:18.820 align:middle line:84% So it's what the light level needs 01:00:18.820 --> 01:00:21.060 align:middle line:90% to be for you to detect it. 01:00:21.060 --> 01:00:24.220 align:middle line:84% So you could, in principle, measure detection thresholds. 01:00:24.220 --> 01:00:28.020 align:middle line:84% And the point is that these two curves are essentially the same. 01:00:28.020 --> 01:00:31.260 align:middle line:84% So the ability of the rods to absorb-- 01:00:31.260 --> 01:00:32.860 align:middle line:84% the photons of different wavelengths 01:00:32.860 --> 01:00:35.980 align:middle line:84% determines how sensitive you are to them. 01:00:35.980 --> 01:00:38.220 align:middle line:84% Another way that you could think about measuring 01:00:38.220 --> 01:00:40.820 align:middle line:84% scotopic sensitivity is to do an experiment 01:00:40.820 --> 01:00:44.380 align:middle line:84% where an observer adjusts lights of different wavelengths 01:00:44.380 --> 01:00:47.193 align:middle line:84% until they appear to be equal in brightness. 01:00:47.193 --> 01:00:48.860 align:middle line:84% And they will appear equal in brightness 01:00:48.860 --> 01:00:52.100 align:middle line:84% because they are producing the same response in the rods. 01:00:52.100 --> 01:00:58.080 align:middle line:84% So I can give you 450-nanometer light and a 600-nanometer light, 01:00:58.080 --> 01:01:00.105 align:middle line:84% and I could ask you to adjust their intensities, 01:01:00.105 --> 01:01:01.480 align:middle line:84% and you would be able to do that, 01:01:01.480 --> 01:01:03.878 align:middle line:84% and you will be able to make them look identical. 01:01:03.878 --> 01:01:05.420 align:middle line:84% Now there'll be different intensities 01:01:05.420 --> 01:01:09.060 align:middle line:84% and at different wavelengths, but they'll look the same. 01:01:09.060 --> 01:01:11.300 align:middle line:84% And this is possible because of this thing 01:01:11.300 --> 01:01:13.500 align:middle line:84% that we talked about a little while ago called 01:01:13.500 --> 01:01:15.660 align:middle line:90% the principle of univariance. 01:01:15.660 --> 01:01:17.660 align:middle line:84% So once photons are absorbed, they 01:01:17.660 --> 01:01:20.740 align:middle line:84% have exactly the same effect on the photoreceptor. 01:01:20.740 --> 01:01:23.005 align:middle line:84% So the only thing that really determines 01:01:23.005 --> 01:01:24.380 align:middle line:84% the response of the photoreceptor 01:01:24.380 --> 01:01:26.040 align:middle line:84% is whether the photons get absorbed or not. 01:01:26.040 --> 01:01:28.290 align:middle line:84% Some types of photons are more likely to get absorbed, 01:01:28.290 --> 01:01:30.270 align:middle line:84% and that means they give you a bigger response. 01:01:30.270 --> 01:01:33.450 align:middle line:84% So all the receptors can do is respond more or less. 01:01:33.450 --> 01:01:35.038 align:middle line:84% So they vary along one dimension. 01:01:35.038 --> 01:01:37.330 align:middle line:84% So that's why it's called the principle of univariance. 01:01:37.330 --> 01:01:39.230 align:middle line:90% So it just goes up or down. 01:01:39.230 --> 01:01:44.230 align:middle line:90% 01:01:44.230 --> 01:01:48.110 align:middle line:84% All right, so we can think of the initial stage 01:01:48.110 --> 01:01:51.110 align:middle line:84% of color vision that's happening in the retina with the cones 01:01:51.110 --> 01:01:54.790 align:middle line:84% as a mapping from the space of spectra 01:01:54.790 --> 01:01:57.430 align:middle line:90% to the space of cone responses. 01:01:57.430 --> 01:02:01.030 align:middle line:84% So this is an example to help us think about this. 01:02:01.030 --> 01:02:06.990 align:middle line:84% So what is depicted here is the spectrum of a particular light. 01:02:06.990 --> 01:02:10.430 align:middle line:84% So the y-axis here is power, and this 01:02:10.430 --> 01:02:13.210 align:middle line:84% is wavelength in nanometers from 400 to 700. 01:02:13.210 --> 01:02:15.310 align:middle line:84% That's the region that you can see. 01:02:15.310 --> 01:02:18.230 align:middle line:84% So this particular spectrum is the sum 01:02:18.230 --> 01:02:21.350 align:middle line:90% of 31 monochromatic lights. 01:02:21.350 --> 01:02:23.270 align:middle line:84% A monochromatic light is a light source that 01:02:23.270 --> 01:02:26.510 align:middle line:90% emits a particular wavelength. 01:02:26.510 --> 01:02:29.440 align:middle line:84% And they're spaced every 10 nanometers. 01:02:29.440 --> 01:02:31.520 align:middle line:84% All right, so to fully specify-- and you 01:02:31.520 --> 01:02:35.720 align:middle line:84% can see that the intensities of those 31 lights 01:02:35.720 --> 01:02:37.720 align:middle line:90% vary from one to the other. 01:02:37.720 --> 01:02:40.480 align:middle line:84% So to fully specify that mixture, 01:02:40.480 --> 01:02:42.600 align:middle line:90% you would need 31 numbers. 01:02:42.600 --> 01:02:44.915 align:middle line:84% So you'd need a vector of length 31. 01:02:44.915 --> 01:02:47.040 align:middle line:84% So we could also say there's 31 degrees of freedom, 01:02:47.040 --> 01:02:48.800 align:middle line:90% 31 dimensions. 01:02:48.800 --> 01:02:50.680 align:middle line:90% All means the same thing. 01:02:50.680 --> 01:02:55.800 align:middle line:84% Now, most colors in nature have continuous spectra. 01:02:55.800 --> 01:02:57.840 align:middle line:84% So to specify a continuous spectrum, 01:02:57.840 --> 01:03:00.120 align:middle line:84% you would need an infinite set of numbers. 01:03:00.120 --> 01:03:04.580 align:middle line:84% So this is an infinite dimensional quantity. 01:03:04.580 --> 01:03:07.380 align:middle line:84% Now, the output of a single cone, 01:03:07.380 --> 01:03:11.680 align:middle line:84% when it is stimulated by a given light with a given spectrum, 01:03:11.680 --> 01:03:14.480 align:middle line:90% is a single number. 01:03:14.480 --> 01:03:19.360 align:middle line:84% Thus, a cone maps an infinite dimensional vector 01:03:19.360 --> 01:03:22.680 align:middle line:90% into a single number. 01:03:22.680 --> 01:03:25.440 align:middle line:84% Now, you have three types of cones. 01:03:25.440 --> 01:03:27.880 align:middle line:84% So those three cones together map 01:03:27.880 --> 01:03:30.800 align:middle line:84% that infinite dimensional input, so that's 01:03:30.800 --> 01:03:33.000 align:middle line:84% the light at a given point in space 01:03:33.000 --> 01:03:36.302 align:middle line:90% to a three-dimensional vector. 01:03:36.302 --> 01:03:38.760 align:middle line:84% So another way to say it is that each spectral distribution 01:03:38.760 --> 01:03:41.140 align:middle line:84% is a point in an infinite dimensional space. 01:03:41.140 --> 01:03:43.240 align:middle line:84% The cones project this point to a point 01:03:43.240 --> 01:03:44.900 align:middle line:90% in a three-dimensional space. 01:03:44.900 --> 01:03:50.600 align:middle line:90% 01:03:50.600 --> 01:03:53.800 align:middle line:84% OK, so here's a picture that depicts what happens when 01:03:53.800 --> 01:03:55.580 align:middle line:90% you create a spectral image. 01:03:55.580 --> 01:03:57.240 align:middle line:90% So you got a person. 01:03:57.240 --> 01:04:00.020 align:middle line:84% They are looking at something, a surface. 01:04:00.020 --> 01:04:02.720 align:middle line:84% There's a light source, the illumination. 01:04:02.720 --> 01:04:07.360 align:middle line:84% So photons from the light source hit the surface. 01:04:07.360 --> 01:04:08.460 align:middle line:90% Some of them are absorbed. 01:04:08.460 --> 01:04:10.040 align:middle line:90% Some of them are reflected back. 01:04:10.040 --> 01:04:13.080 align:middle line:84% They enter the eye and are absorbed by the cones producing 01:04:13.080 --> 01:04:14.140 align:middle line:90% cone responses. 01:04:14.140 --> 01:04:17.120 align:middle line:84% And so the bottom row shows the different quantities that 01:04:17.120 --> 01:04:18.960 align:middle line:90% are involved in this process. 01:04:18.960 --> 01:04:21.540 align:middle line:84% So here's the spectrum of this illumination. 01:04:21.540 --> 01:04:23.320 align:middle line:90% So this is a light bulb. 01:04:23.320 --> 01:04:26.760 align:middle line:84% Light bulbs tend to have more power at long wavelengths. 01:04:26.760 --> 01:04:28.510 align:middle line:84% So you get something that looks like that. 01:04:28.510 --> 01:04:30.260 align:middle line:84% So that's the spectrum of the illumination 01:04:30.260 --> 01:04:32.650 align:middle line:90% in this particular example. 01:04:32.650 --> 01:04:34.330 align:middle line:90% Here's the reflectance. 01:04:34.330 --> 01:04:36.550 align:middle line:90% So this is a blue surface. 01:04:36.550 --> 01:04:40.770 align:middle line:84% And so it reflects more light at short wavelengths 01:04:40.770 --> 01:04:43.550 align:middle line:90% than at long wavelengths. 01:04:43.550 --> 01:04:46.090 align:middle line:84% Now, the scattered light that comes off the page 01:04:46.090 --> 01:04:48.490 align:middle line:84% is the product of the illumination 01:04:48.490 --> 01:04:49.470 align:middle line:90% and the reflectance. 01:04:49.470 --> 01:04:52.810 align:middle line:84% So you just pointwise multiply this and this. 01:04:52.810 --> 01:04:55.650 align:middle line:90% And so that gives you this. 01:04:55.650 --> 01:04:58.850 align:middle line:84% And so you can immediately see that there's a tricky problem 01:04:58.850 --> 01:05:00.870 align:middle line:90% that the visual system faces. 01:05:00.870 --> 01:05:04.010 align:middle line:84% So the blueness of the surface shows up in the fact 01:05:04.010 --> 01:05:05.890 align:middle line:90% that there's a bump down here. 01:05:05.890 --> 01:05:08.290 align:middle line:84% But the fact that the illumination has so much more 01:05:08.290 --> 01:05:10.290 align:middle line:84% power at the long wavelengths means 01:05:10.290 --> 01:05:14.410 align:middle line:84% that the scattered light is very biased towards long wavelengths, 01:05:14.410 --> 01:05:16.890 align:middle line:90% or red light. 01:05:16.890 --> 01:05:20.930 align:middle line:84% So this is the spectrum that enters the eye at a given point. 01:05:20.930 --> 01:05:23.890 align:middle line:84% These are the cone sensitivity functions, the things 01:05:23.890 --> 01:05:27.490 align:middle line:90% that we just looked at. 01:05:27.490 --> 01:05:32.090 align:middle line:84% This gets multiplied by each of these 01:05:32.090 --> 01:05:37.210 align:middle line:84% to yield the responses of the L, M, and S cones. 01:05:37.210 --> 01:05:39.630 align:middle line:84% And it would be a lot better if these were flipped. 01:05:39.630 --> 01:05:43.670 align:middle line:84% I don't know why they are not, but short, medium, 01:05:43.670 --> 01:05:44.970 align:middle line:90% and long wavelength cones. 01:05:44.970 --> 01:05:48.650 align:middle line:84% So even though this is a blue surface, 01:05:48.650 --> 01:05:50.450 align:middle line:84% the response in the S cones is actually 01:05:50.450 --> 01:05:52.970 align:middle line:84% lower than the response in the L cones 01:05:52.970 --> 01:05:54.890 align:middle line:84% because the illumination is so heavily biased 01:05:54.890 --> 01:05:57.530 align:middle line:90% towards long wavelengths. 01:05:57.530 --> 01:05:59.550 align:middle line:90% OK, so that's process here. 01:05:59.550 --> 01:06:01.810 align:middle line:84% We've got this infinite dimensional thing here 01:06:01.810 --> 01:06:04.530 align:middle line:84% that's coming into the eye, the spectrum of the scattered light, 01:06:04.530 --> 01:06:06.970 align:middle line:84% that's caused by these two things 01:06:06.970 --> 01:06:10.890 align:middle line:84% in the world, the illumination and the reflectance. 01:06:10.890 --> 01:06:15.610 align:middle line:84% That produces three responses in each of the three cones. 01:06:15.610 --> 01:06:18.410 align:middle line:84% And your percept of color has to be 01:06:18.410 --> 01:06:24.080 align:middle line:84% derived from these three numbers across the entire image. 01:06:24.080 --> 01:06:26.860 align:middle line:90% 01:06:26.860 --> 01:06:28.440 align:middle line:84% What questions you got about that? 01:06:28.440 --> 01:06:31.020 align:middle line:90% 01:06:31.020 --> 01:06:35.843 align:middle line:84% AUDIENCE: So why is it blue if the red is higher? 01:06:35.843 --> 01:06:36.760 align:middle line:90% Why does it look blue? 01:06:36.760 --> 01:06:38.343 align:middle line:84% JOSH MCDERMOTT: Why does it look blue? 01:06:38.343 --> 01:06:40.860 align:middle line:84% So this is the problem of color constancy. 01:06:40.860 --> 01:06:46.940 align:middle line:84% And so you would have to have some way of discounting 01:06:46.940 --> 01:06:49.580 align:middle line:84% the illumination, either estimating the illumination 01:06:49.580 --> 01:06:52.960 align:middle line:84% and dividing it out or doing something else. 01:06:52.960 --> 01:06:55.940 align:middle line:84% And so we're going to talk a lot about that in just a little bit. 01:06:55.940 --> 01:06:58.680 align:middle line:90% 01:06:58.680 --> 01:07:01.347 align:middle line:84% OK, so the key idea here is you're 01:07:01.347 --> 01:07:03.180 align:middle line:84% critically dependent on the fact that you've 01:07:03.180 --> 01:07:04.097 align:middle line:90% got these three cones. 01:07:04.097 --> 01:07:05.807 align:middle line:84% So if you just have a single receptor, 01:07:05.807 --> 01:07:08.140 align:middle line:84% you can't tell anything about the wavelength composition 01:07:08.140 --> 01:07:09.000 align:middle line:90% of light. 01:07:09.000 --> 01:07:11.540 align:middle line:84% So here's an example where we just 01:07:11.540 --> 01:07:14.380 align:middle line:84% look at the responses of the M cones 01:07:14.380 --> 01:07:18.920 align:middle line:84% to these two individual wavelengths. 01:07:18.920 --> 01:07:21.810 align:middle line:84% And so in this particular example, 01:07:21.810 --> 01:07:24.750 align:middle line:84% these two wavelengths have the same-- 01:07:24.750 --> 01:07:28.070 align:middle line:84% so the M cone has the same sensitivity to, 01:07:28.070 --> 01:07:34.990 align:middle line:84% it looks like about, 450 and about 625 nanometers. 01:07:34.990 --> 01:07:38.310 align:middle line:84% So you get the same response in the M cone. 01:07:38.310 --> 01:07:42.398 align:middle line:84% So even though these are two physically distinct wavelengths, 01:07:42.398 --> 01:07:44.190 align:middle line:84% you get the same response, and you wouldn't 01:07:44.190 --> 01:07:45.990 align:middle line:90% be able to tell them apart. 01:07:45.990 --> 01:07:49.310 align:middle line:84% But when you add in the other two cones, 01:07:49.310 --> 01:07:52.110 align:middle line:84% you can tell the difference because the other two cones 01:07:52.110 --> 01:07:53.430 align:middle line:90% give you different responses. 01:07:53.430 --> 01:07:56.590 align:middle line:84% So even though the M cones give you the same response, 01:07:56.590 --> 01:08:00.130 align:middle line:84% the L cones respond a lot to the longer wavelength 01:08:00.130 --> 01:08:02.150 align:middle line:84% and not at all to the shorter wavelength. 01:08:02.150 --> 01:08:05.430 align:middle line:84% And the S cones respond a lot to the shorter wavelength. 01:08:05.430 --> 01:08:08.270 align:middle line:84% And-- sorry, I said it backwards. 01:08:08.270 --> 01:08:12.210 align:middle line:84% So to the longer wavelength, the L cones respond a lot. 01:08:12.210 --> 01:08:13.830 align:middle line:84% The S cones don't respond at all. 01:08:13.830 --> 01:08:16.080 align:middle line:84% To the shorter wavelengths, the S cones respond a lot, 01:08:16.080 --> 01:08:18.069 align:middle line:90% and the L cones respond less. 01:08:18.069 --> 01:08:23.950 align:middle line:84% So the three-dimensional response is different. 01:08:23.950 --> 01:08:27.670 align:middle line:84% All right, so even though you are 01:08:27.670 --> 01:08:29.430 align:middle line:84% mapping this infinite dimensional space 01:08:29.430 --> 01:08:33.430 align:middle line:84% down onto three dimensions, those three dimensions 01:08:33.430 --> 01:08:39.310 align:middle line:84% allow you to distinguish different colors to some extent. 01:08:39.310 --> 01:08:42.350 align:middle line:84% So it's often very convenient and clarifying 01:08:42.350 --> 01:08:43.910 align:middle line:84% to think of the color transduction 01:08:43.910 --> 01:08:47.149 align:middle line:90% process as linear algebra. 01:08:47.149 --> 01:08:50.810 align:middle line:84% So we've got the light that's projected into the eye. 01:08:50.810 --> 01:08:53.790 align:middle line:84% Think of that as a vector, a very long vector 01:08:53.790 --> 01:08:56.270 align:middle line:84% that tells you the power of the light 01:08:56.270 --> 01:08:58.470 align:middle line:90% at each different wavelength. 01:08:58.470 --> 01:09:01.170 align:middle line:84% And then you have three spectral sensitivity curves. 01:09:01.170 --> 01:09:03.229 align:middle line:84% Those curves that were plotted on the graph. 01:09:03.229 --> 01:09:06.069 align:middle line:84% So that tells you, for every wavelength, 01:09:06.069 --> 01:09:09.149 align:middle line:84% what is the sensitivity of the L cone, the M cone, and the S 01:09:09.149 --> 01:09:11.729 align:middle line:90% cone? 01:09:11.729 --> 01:09:13.937 align:middle line:84% That vector gets multiplied by that matrix. 01:09:13.937 --> 01:09:15.729 align:middle line:84% And you get out a three-dimensional vector, 01:09:15.729 --> 01:09:17.771 align:middle line:84% which gives you the response of the L cone, the M 01:09:17.771 --> 01:09:18.779 align:middle line:90% cone, and the S cone. 01:09:18.779 --> 01:09:22.920 align:middle line:90% 01:09:22.920 --> 01:09:26.399 align:middle line:84% All right, so one really important consequence 01:09:26.399 --> 01:09:30.160 align:middle line:84% and one really intriguing consequence of this, namely 01:09:30.160 --> 01:09:34.600 align:middle line:84% the fact that you have this very high-dimensional thing coming 01:09:34.600 --> 01:09:36.840 align:middle line:84% into your eye that gets projected down 01:09:36.840 --> 01:09:39.680 align:middle line:84% onto this three-dimensional space, 01:09:39.680 --> 01:09:44.920 align:middle line:84% one consequence is that there are many physically distinct 01:09:44.920 --> 01:09:49.359 align:middle line:84% spectra that will produce the same cone responses 01:09:49.359 --> 01:09:52.952 align:middle line:84% and thus will be indistinguishable to a human 01:09:52.952 --> 01:09:55.160 align:middle line:84% even though physically you could measure them and see 01:09:55.160 --> 01:09:58.493 align:middle line:90% that they're different. 01:09:58.493 --> 01:10:01.320 align:middle line:90% And these are called metamers. 01:10:01.320 --> 01:10:05.440 align:middle line:84% So metamers are physically distinct stimuli 01:10:05.440 --> 01:10:09.880 align:middle line:84% that are indistinguishable to an observer. 01:10:09.880 --> 01:10:12.040 align:middle line:84% And so in the case of color, anytime 01:10:12.040 --> 01:10:16.760 align:middle line:84% you excite the cones in the same way you experience metamers. 01:10:16.760 --> 01:10:18.280 align:middle line:90% So these are two examples. 01:10:18.280 --> 01:10:20.480 align:middle line:90% These are two spectra. 01:10:20.480 --> 01:10:23.440 align:middle line:84% So power versus wavelength, they're physically distinct. 01:10:23.440 --> 01:10:26.040 align:middle line:84% This is the approximate spectrum from a tungsten light 01:10:26.040 --> 01:10:29.280 align:middle line:84% bulb, the kind of light bulb we used to all have in our houses. 01:10:29.280 --> 01:10:33.120 align:middle line:84% This is a metameric match from a color monitor. 01:10:33.120 --> 01:10:35.600 align:middle line:84% So this is a physically distinct spectrum 01:10:35.600 --> 01:10:37.680 align:middle line:84% that produces the same cone responses 01:10:37.680 --> 01:10:42.060 align:middle line:84% and would look exactly the same to a human viewing it. 01:10:42.060 --> 01:10:45.375 align:middle line:90% 01:10:45.375 --> 01:10:47.440 align:middle line:84% So metamers are different mixtures of light 01:10:47.440 --> 01:10:49.560 align:middle line:84% that excite the cones in the same way. 01:10:49.560 --> 01:10:52.180 align:middle line:84% And this is a consequence of, again, 01:10:52.180 --> 01:10:55.322 align:middle line:84% this property of univariance, the fact that the cones only 01:10:55.322 --> 01:10:57.280 align:middle line:84% respond on this one dimension and then the fact 01:10:57.280 --> 01:10:58.880 align:middle line:84% that you have only three of them. 01:10:58.880 --> 01:11:05.680 align:middle line:84% So these are three examples of stimuli that produce 01:11:05.680 --> 01:11:07.060 align:middle line:90% the same responses in the cone. 01:11:07.060 --> 01:11:08.960 align:middle line:84% And so for simplicity here, these 01:11:08.960 --> 01:11:12.440 align:middle line:84% are stimuli that don't stimulate the cone, 01:11:12.440 --> 01:11:17.650 align:middle line:84% but they are cleverly arranged to produce the same response 01:11:17.650 --> 01:11:21.650 align:middle line:90% in the M and the L cones. 01:11:21.650 --> 01:11:23.970 align:middle line:90% And there are lots of these. 01:11:23.970 --> 01:11:27.690 align:middle line:84% OK, so metamers produce exactly the same neural response 01:11:27.690 --> 01:11:28.950 align:middle line:90% starting at the cones. 01:11:28.950 --> 01:11:32.090 align:middle line:90% So here's an imaginary example. 01:11:32.090 --> 01:11:36.150 align:middle line:84% Lights A and B. So light A is projected on this half. 01:11:36.150 --> 01:11:39.050 align:middle line:84% Light B is projected on this half of the circle. 01:11:39.050 --> 01:11:41.810 align:middle line:90% They have different spectra. 01:11:41.810 --> 01:11:44.210 align:middle line:84% But if they stimulate the three cones in the same way, 01:11:44.210 --> 01:11:45.510 align:middle line:90% they will be indistinguishable. 01:11:45.510 --> 01:11:47.330 align:middle line:84% So they'll look like the same color. 01:11:47.330 --> 01:11:51.130 align:middle line:84% And no later processing in the brain can tell them apart. 01:11:51.130 --> 01:11:53.530 align:middle line:84% So the thing that distinguishes them 01:11:53.530 --> 01:11:56.938 align:middle line:84% was lost at the moment of photon absorption 01:11:56.938 --> 01:11:58.730 align:middle line:84% because the only thing that your brain gets 01:11:58.730 --> 01:12:00.410 align:middle line:90% is the cone responses. 01:12:00.410 --> 01:12:03.810 align:middle line:84% So to tell these things apart, you'd have to get a photometer 01:12:03.810 --> 01:12:08.470 align:middle line:84% and measure it with something else other than your eye. 01:12:08.470 --> 01:12:10.190 align:middle line:84% So here are some puzzles to think about. 01:12:10.190 --> 01:12:14.145 align:middle line:84% So suppose you adapt to a uniform red field. 01:12:14.145 --> 01:12:15.520 align:middle line:84% So I show you a big sheet of red, 01:12:15.520 --> 01:12:16.880 align:middle line:84% and I make you look at it for a minute, 01:12:16.880 --> 01:12:18.338 align:middle line:84% like we sometimes do in this class. 01:12:18.338 --> 01:12:23.102 align:middle line:84% That will reduce the gain, the sensitivity of the L cones, 01:12:23.102 --> 01:12:25.060 align:middle line:84% and the consequence of that is that if you then 01:12:25.060 --> 01:12:26.840 align:middle line:84% go around and look at some other stuff, 01:12:26.840 --> 01:12:29.780 align:middle line:84% everything will look a bit more greenish than it normally does. 01:12:29.780 --> 01:12:35.580 align:middle line:84% So if you do that and you then look at lights A and B, 01:12:35.580 --> 01:12:38.340 align:middle line:84% both of them will look more greenish than they did before, 01:12:38.340 --> 01:12:40.420 align:middle line:90% but they'll look the same. 01:12:40.420 --> 01:12:42.420 align:middle line:84% So you're reducing the sensitivity of one 01:12:42.420 --> 01:12:44.500 align:middle line:90% of the types of cones. 01:12:44.500 --> 01:12:46.440 align:middle line:84% So it's response gets a little bit lower. 01:12:46.440 --> 01:12:50.060 align:middle line:84% But because of the linearity of this process, 01:12:50.060 --> 01:12:53.480 align:middle line:84% light A and light B will still generate the same cone responses 01:12:53.480 --> 01:12:56.540 align:middle line:84% even though they'll be different than they were before. 01:12:56.540 --> 01:12:57.980 align:middle line:90% Here's another puzzle. 01:12:57.980 --> 01:13:00.180 align:middle line:84% So imagine we have lights A and B 01:13:00.180 --> 01:13:02.380 align:middle line:84% and then we introduce a wash light. 01:13:02.380 --> 01:13:06.580 align:middle line:84% So this is a separate light that gets 01:13:06.580 --> 01:13:10.860 align:middle line:84% added and is shown on the entire circle. 01:13:10.860 --> 01:13:14.260 align:middle line:84% So you can think of that as adding some additional light 01:13:14.260 --> 01:13:19.260 align:middle line:84% to both A and B. So A and B will now, 01:13:19.260 --> 01:13:21.940 align:middle line:84% again, both look different than before, 01:13:21.940 --> 01:13:24.500 align:middle line:84% but they will still look the same as each other. 01:13:24.500 --> 01:13:28.260 align:middle line:84% And so I want you all to think about why this is 01:13:28.260 --> 01:13:31.260 align:middle line:84% and again, to try to explain this 01:13:31.260 --> 01:13:33.480 align:middle line:90% in these linear algebraic terms. 01:13:33.480 --> 01:13:36.780 align:middle line:84% So this is a good case where thinking 01:13:36.780 --> 01:13:38.695 align:middle line:84% about this as matrix multiplication 01:13:38.695 --> 01:13:40.820 align:middle line:84% ends up greatly simplifying these things that would 01:13:40.820 --> 01:13:42.600 align:middle line:90% otherwise make your brain hurt. 01:13:42.600 --> 01:13:47.940 align:middle line:84% So something like this will be on an upcoming problem set. 01:13:47.940 --> 01:13:51.860 align:middle line:84% All right, so metamers are kind of cool. 01:13:51.860 --> 01:13:54.780 align:middle line:84% They're also practically really important 01:13:54.780 --> 01:14:00.800 align:middle line:84% because they are the reason why monitors, color TVs, cameras, 01:14:00.800 --> 01:14:03.980 align:middle line:84% film, et cetera work because what this means is 01:14:03.980 --> 01:14:06.900 align:middle line:84% that, with just three different color guns, so with three 01:14:06.900 --> 01:14:12.830 align:middle line:84% different light sources, you can create any color experience 01:14:12.830 --> 01:14:14.390 align:middle line:84% because the space of color is defined 01:14:14.390 --> 01:14:16.510 align:middle line:90% by this three-dimensional space. 01:14:16.510 --> 01:14:20.170 align:middle line:84% So if you have these three things that you can adjust, 01:14:20.170 --> 01:14:22.110 align:middle line:84% you can move the cone responses around 01:14:22.110 --> 01:14:25.230 align:middle line:84% to match any particular color experience. 01:14:25.230 --> 01:14:29.390 align:middle line:90% So super important practically. 01:14:29.390 --> 01:14:33.510 align:middle line:84% So now we know about the different types of cones 01:14:33.510 --> 01:14:34.510 align:middle line:90% nowadays-- 01:14:34.510 --> 01:14:38.230 align:middle line:84% because, in part, you can chop up the retina and measure stuff 01:14:38.230 --> 01:14:40.510 align:middle line:84% and find the pigments and measure their responses 01:14:40.510 --> 01:14:41.050 align:middle line:90% and things. 01:14:41.050 --> 01:14:44.750 align:middle line:84% But that's a relatively recent development in human history. 01:14:44.750 --> 01:14:50.150 align:middle line:84% But the idea that color vision is trichromatic predates all 01:14:50.150 --> 01:14:55.830 align:middle line:84% of that kind of modern biology and was hypothesized, really, 01:14:55.830 --> 01:15:01.550 align:middle line:84% back in the 1800s by Young, a vision scientist, and Helmholtz, 01:15:01.550 --> 01:15:03.310 align:middle line:84% who we hear about a lot in this class. 01:15:03.310 --> 01:15:06.590 align:middle line:84% So they proposed that different-color experiences 01:15:06.590 --> 01:15:09.620 align:middle line:84% are due to the activation of just three receptor types, 01:15:09.620 --> 01:15:12.840 align:middle line:84% again, long before anybody could actually measure the actual cone 01:15:12.840 --> 01:15:13.720 align:middle line:90% properties. 01:15:13.720 --> 01:15:15.920 align:middle line:84% And this was based on the observation 01:15:15.920 --> 01:15:18.680 align:middle line:84% that you could generate all possible colors just 01:15:18.680 --> 01:15:22.320 align:middle line:84% by mixing lights of three different wavelengths. 01:15:22.320 --> 01:15:26.898 align:middle line:84% So this just turns out to be a fact about color vision. 01:15:26.898 --> 01:15:29.440 align:middle line:84% So that led them to hypothesize that there are these receptor 01:15:29.440 --> 01:15:30.060 align:middle line:90% types. 01:15:30.060 --> 01:15:32.720 align:middle line:90% It turned out to be correct. 01:15:32.720 --> 01:15:37.360 align:middle line:84% All right, but this was controversial at the time. 01:15:37.360 --> 01:15:41.920 align:middle line:84% And the reason for this is that there are some, what seem like, 01:15:41.920 --> 01:15:46.800 align:middle line:84% problems with this idea that color is purely three 01:15:46.800 --> 01:15:47.840 align:middle line:90% dimensional. 01:15:47.840 --> 01:15:53.520 align:middle line:84% So one problem is that sometimes people are colorblind. 01:15:53.520 --> 01:15:55.520 align:middle line:84% Colorblindness is fairly common, so you probably 01:15:55.520 --> 01:15:59.280 align:middle line:90% know somebody who's colorblind. 01:15:59.280 --> 01:16:02.040 align:middle line:84% When people are colorblind, they are typically 01:16:02.040 --> 01:16:06.600 align:middle line:84% missing one or more receptors in the eye. 01:16:06.600 --> 01:16:09.920 align:middle line:84% But when this happens, colorblindness typically 01:16:09.920 --> 01:16:11.720 align:middle line:90% occurs in pairs. 01:16:11.720 --> 01:16:15.380 align:middle line:84% So you're not just blind to red or just blind to green. 01:16:15.380 --> 01:16:18.560 align:middle line:84% Typically, you're red-green colorblind 01:16:18.560 --> 01:16:20.600 align:middle line:90% or blue-yellow colorblind. 01:16:20.600 --> 01:16:22.500 align:middle line:90% And the pairs are not arbitrary. 01:16:22.500 --> 01:16:23.860 align:middle line:90% So it's not just any pairs. 01:16:23.860 --> 01:16:26.340 align:middle line:84% So you don't ever get blue-red colorblindness. 01:16:26.340 --> 01:16:31.080 align:middle line:84% So again, people were aware that colorblindness existed. 01:16:31.080 --> 01:16:38.040 align:middle line:84% And that led people to question the fact that-- or the proposal 01:16:38.040 --> 01:16:41.680 align:middle line:84% that vision was purely trichromatic. 01:16:41.680 --> 01:16:45.200 align:middle line:84% Trichromacy also really doesn't give a very clear explanation 01:16:45.200 --> 01:16:48.560 align:middle line:84% for the fact that it's impossible to create 01:16:48.560 --> 01:16:50.100 align:middle line:90% certain kinds of color mixtures. 01:16:50.100 --> 01:16:55.000 align:middle line:84% So there's really no such thing as a bluish yellow or a greenish 01:16:55.000 --> 01:16:56.440 align:middle line:90% red. 01:16:56.440 --> 01:16:59.680 align:middle line:84% And so this gave rise to a competing theory. 01:16:59.680 --> 01:17:02.320 align:middle line:84% Again, this is from Hering, who we 01:17:02.320 --> 01:17:06.530 align:middle line:84% talked about in the context of lightness perception. 01:17:06.530 --> 01:17:10.010 align:middle line:84% So Hering made this observation that a red 01:17:10.010 --> 01:17:12.610 align:middle line:84% can look yellowish or bluish, but not greenish. 01:17:12.610 --> 01:17:15.730 align:middle line:84% A yellow can look greenish or reddish, but not bluish. 01:17:15.730 --> 01:17:18.690 align:middle line:84% And inferred that, in the visual system, 01:17:18.690 --> 01:17:23.090 align:middle line:84% there were opponent color axes, so something that 01:17:23.090 --> 01:17:25.870 align:middle line:84% was computing the difference between red and green 01:17:25.870 --> 01:17:29.570 align:middle line:90% and between blue and yellow. 01:17:29.570 --> 01:17:32.650 align:middle line:84% So Hering thought that opponency was generated 01:17:32.650 --> 01:17:34.070 align:middle line:90% at the level of receptors. 01:17:34.070 --> 01:17:36.230 align:middle line:84% So he thought that Young and Helmholtz were wrong. 01:17:36.230 --> 01:17:40.227 align:middle line:84% It wasn't the case that there were these three receptor types. 01:17:40.227 --> 01:17:42.810 align:middle line:84% He thought that there were these opponent receptors, something 01:17:42.810 --> 01:17:44.670 align:middle line:84% that was measuring the difference between red and green 01:17:44.670 --> 01:17:46.420 align:middle line:84% and the difference between yellow and blue 01:17:46.420 --> 01:17:49.050 align:middle line:84% and the difference between white and black. 01:17:49.050 --> 01:17:52.490 align:middle line:84% And so there were all these vigorous arguments about that. 01:17:52.490 --> 01:17:57.450 align:middle line:84% Nowadays, as is sometimes the case with these debates, 01:17:57.450 --> 01:18:01.570 align:middle line:84% we believe that both parties were correct. 01:18:01.570 --> 01:18:04.850 align:middle line:84% In fact, we start out vision with three cone types 01:18:04.850 --> 01:18:08.690 align:middle line:84% that then get translated into opponent channels. 01:18:08.690 --> 01:18:12.510 align:middle line:84% And that happens actually in the retina. 01:18:12.510 --> 01:18:15.010 align:middle line:84% So we've got these three different types of cones 01:18:15.010 --> 01:18:17.670 align:middle line:84% and that those get combined in different ways, 01:18:17.670 --> 01:18:21.810 align:middle line:84% taking differences and sums to generate opponent channels 01:18:21.810 --> 01:18:23.410 align:middle line:84% that you can see in retinal ganglion 01:18:23.410 --> 01:18:26.170 align:middle line:84% cells with these on/off receptive fields 01:18:26.170 --> 01:18:28.690 align:middle line:84% where the center will have input from one cone type 01:18:28.690 --> 01:18:30.930 align:middle line:84% and the surround will have input from another. 01:18:30.930 --> 01:18:35.650 align:middle line:84% And again, the pairings here are not arbitrary but rather 01:18:35.650 --> 01:18:38.050 align:middle line:84% systematic into these red-green and blue-yellow opponent 01:18:38.050 --> 01:18:39.090 align:middle line:90% channels. 01:18:39.090 --> 01:18:41.250 align:middle line:84% So we're going to end there today. 01:18:41.250 --> 01:18:44.330 align:middle line:84% And when we come back, we'll continue talking about color 01:18:44.330 --> 01:18:46.770 align:middle line:84% and particularly get into color constancy 01:18:46.770 --> 01:18:50.160 align:middle line:84% and then how color is mediated in the brain. 01:18:50.160 --> 01:19:03.000 align:middle line:90%