WEBVTT 00:00:00.000 --> 00:00:01.970 align:middle line:90% [SQUEAKING] 00:00:01.970 --> 00:00:03.930 align:middle line:90% [RUSTLING] 00:00:03.930 --> 00:00:05.400 align:middle line:90% [CLICKING] 00:00:05.400 --> 00:00:08.545 align:middle line:90% 00:00:08.545 --> 00:00:09.920 align:middle line:84% JOSH MCDERMOTT: Today we're going 00:00:09.920 --> 00:00:11.880 align:middle line:84% to wrap up talking about object recognition, 00:00:11.880 --> 00:00:13.140 align:middle line:90% get into texture recognition. 00:00:13.140 --> 00:00:18.700 align:middle line:84% So remember, object recognition is a thing that we can do. 00:00:18.700 --> 00:00:23.290 align:middle line:84% You can look around scenes and name objects, OK? 00:00:23.290 --> 00:00:26.920 align:middle line:84% It's mostly effortless for people, but a pretty challenging 00:00:26.920 --> 00:00:28.880 align:middle line:90% computational problem. 00:00:28.880 --> 00:00:31.800 align:middle line:84% In humans and other primates, object recognition 00:00:31.800 --> 00:00:35.380 align:middle line:84% is something that we typically do when we foveate objects. 00:00:35.380 --> 00:00:38.455 align:middle line:84% So it's kind of dominated by central vision. 00:00:38.455 --> 00:00:40.580 align:middle line:84% And so, oftentimes, when you're looking at a scene, 00:00:40.580 --> 00:00:42.000 align:middle line:84% you'll be making a series of saccades 00:00:42.000 --> 00:00:44.120 align:middle line:84% around things in the scene, recognizing whatever 00:00:44.120 --> 00:00:49.080 align:middle line:90% it is that your eyes focus on. 00:00:49.080 --> 00:00:51.527 align:middle line:84% So the problem is hard, for a couple different reasons. 00:00:51.527 --> 00:00:53.360 align:middle line:84% One is that there's a large number of things 00:00:53.360 --> 00:00:55.340 align:middle line:90% that you can recognize, OK? 00:00:55.340 --> 00:00:57.400 align:middle line:90% This is just a small subset. 00:00:57.400 --> 00:01:01.250 align:middle line:84% And then, the second issue is that the same kind of thing 00:01:01.250 --> 00:01:02.849 align:middle line:84% can produce totally different images, 00:01:02.849 --> 00:01:04.849 align:middle line:84% like depending on the viewing conditions, right? 00:01:04.849 --> 00:01:06.990 align:middle line:84% The viewpoint from which you view an object, 00:01:06.990 --> 00:01:10.250 align:middle line:84% or the illumination conditions, or what the background is that 00:01:10.250 --> 00:01:14.770 align:middle line:84% the object is on, or whether it's occluded, or whether, 00:01:14.770 --> 00:01:17.350 align:middle line:84% like if you're talking about a biological organism, 00:01:17.350 --> 00:01:18.850 align:middle line:84% whether we're non-rigidly deformed-- 00:01:18.850 --> 00:01:21.740 align:middle line:84% I can do all kinds of weird things with my body, right? 00:01:21.740 --> 00:01:22.990 align:middle line:90% Totally changes the image, OK? 00:01:22.990 --> 00:01:26.930 align:middle line:84% So you get wildly different images, depending 00:01:26.930 --> 00:01:28.870 align:middle line:90% on the viewing conditions, OK? 00:01:28.870 --> 00:01:32.930 align:middle line:84% So we talked last time about how there's lots of evidence 00:01:32.930 --> 00:01:36.090 align:middle line:84% that object recognition is largely mediated 00:01:36.090 --> 00:01:38.630 align:middle line:90% by the ventral visual stream. 00:01:38.630 --> 00:01:41.015 align:middle line:84% Remember, dorsal ventral streams. 00:01:41.015 --> 00:01:42.890 align:middle line:84% We talked about evidence from lesion studies, 00:01:42.890 --> 00:01:44.730 align:middle line:84% where you lesion the temporal lobe, 00:01:44.730 --> 00:01:48.790 align:middle line:84% and monkeys and humans end up with recognition difficulties. 00:01:48.790 --> 00:01:50.330 align:middle line:90% We talked about agnosia. 00:01:50.330 --> 00:01:52.810 align:middle line:84% This is the inability to recognize 00:01:52.810 --> 00:01:55.610 align:middle line:84% objects that occasionally results from brain damage 00:01:55.610 --> 00:01:58.530 align:middle line:90% following strokes. 00:01:58.530 --> 00:02:02.290 align:middle line:84% And then, we talked about the ventral visual stream 00:02:02.290 --> 00:02:07.110 align:middle line:84% and evidence that object recognition is fast. 00:02:07.110 --> 00:02:09.810 align:middle line:84% And thus, the inference from that being that, in many cases, 00:02:09.810 --> 00:02:13.050 align:middle line:84% object recognition is mediated by largely feed-forward 00:02:13.050 --> 00:02:14.130 align:middle line:90% processing, OK? 00:02:14.130 --> 00:02:16.750 align:middle line:84% So this is in the macaque visual system. 00:02:16.750 --> 00:02:19.310 align:middle line:84% We've got latencies of about 60 milliseconds at V1. 00:02:19.310 --> 00:02:20.752 align:middle line:90% 100 milliseconds at IT. 00:02:20.752 --> 00:02:22.210 align:middle line:84% In humans, it's a little bit bigger 00:02:22.210 --> 00:02:23.523 align:middle line:90% because the head's bigger. 00:02:23.523 --> 00:02:25.690 align:middle line:84% And then, we talked about various pieces of evidence 00:02:25.690 --> 00:02:29.250 align:middle line:84% that recognition is a quick thing, in some cases, OK? 00:02:29.250 --> 00:02:31.610 align:middle line:84% And so this is the study where people looked 00:02:31.610 --> 00:02:33.305 align:middle line:90% at series-- sequence of images. 00:02:33.305 --> 00:02:34.930 align:middle line:84% And these are event-related potentials, 00:02:34.930 --> 00:02:35.870 align:middle line:90% in response to images. 00:02:35.870 --> 00:02:38.410 align:middle line:84% And you can see that they diverge for categories 00:02:38.410 --> 00:02:41.050 align:middle line:84% of images, in this case, animals and non-animals, 00:02:41.050 --> 00:02:43.630 align:middle line:84% at a relatively short latency of 150 milliseconds, 00:02:43.630 --> 00:02:46.930 align:middle line:84% which is on the order of the latency of the early response 00:02:46.930 --> 00:02:50.130 align:middle line:90% in IT. 00:02:50.130 --> 00:02:54.690 align:middle line:84% OK, and we talked a little bit about the differences 00:02:54.690 --> 00:02:57.820 align:middle line:84% that you see between different stages of the ventral processing 00:02:57.820 --> 00:03:00.980 align:middle line:84% stream, which much of which we've 00:03:00.980 --> 00:03:03.180 align:middle line:84% talked about at earlier stages of the class 00:03:03.180 --> 00:03:04.760 align:middle line:90% and to varying extents. 00:03:04.760 --> 00:03:06.980 align:middle line:84% So in V1, you have relatively simple tuning, 00:03:06.980 --> 00:03:08.780 align:middle line:84% orientation selectivity, and then, 00:03:08.780 --> 00:03:10.220 align:middle line:84% in simple cells and complex cells 00:03:10.220 --> 00:03:12.840 align:middle line:84% you get some tolerance to position. 00:03:12.840 --> 00:03:14.260 align:middle line:90% Question there. 00:03:14.260 --> 00:03:18.500 align:middle line:84% AUDIENCE: For the feed-forward processing in object 00:03:18.500 --> 00:03:24.080 align:middle line:84% recognition is that-- that works for like these broad categories? 00:03:24.080 --> 00:03:26.580 align:middle line:84% If you ask for more specificity, like rather 00:03:26.580 --> 00:03:29.660 align:middle line:84% than recognizing a face, getting the person, 00:03:29.660 --> 00:03:31.100 align:middle line:90% does that require feedback? 00:03:31.100 --> 00:03:35.400 align:middle line:84% Or does it just require a longer chain of feed-forward? 00:03:35.400 --> 00:03:40.080 align:middle line:84% JOSH MCDERMOTT: So I think that, on the whole, 00:03:40.080 --> 00:03:42.800 align:middle line:84% there's evidence that, to a large extent, 00:03:42.800 --> 00:03:44.900 align:middle line:84% even more subtle distinctions can often 00:03:44.900 --> 00:03:49.278 align:middle line:84% be made with largely feed-forward processing. 00:03:49.278 --> 00:03:50.820 align:middle line:84% I mean, there's also lots of evidence 00:03:50.820 --> 00:03:54.780 align:middle line:84% now, especially over the last five to 10 years, 00:03:54.780 --> 00:04:02.320 align:middle line:84% that images that are hard to recognize, for whatever reason-- 00:04:02.320 --> 00:04:05.500 align:middle line:84% like maybe an object is occluded pretty substantially 00:04:05.500 --> 00:04:09.810 align:middle line:84% or that are otherwise kind of unusual, in some way, where 00:04:09.810 --> 00:04:12.060 align:middle line:84% the information is degraded-- that those might require 00:04:12.060 --> 00:04:13.940 align:middle line:90% recurrent processing, OK? 00:04:13.940 --> 00:04:16.940 align:middle line:84% But I think, in some of the things 00:04:16.940 --> 00:04:19.160 align:middle line:84% that I'll get to in just a moment, 00:04:19.160 --> 00:04:22.900 align:middle line:84% that we talked a little bit about last time, where you look 00:04:22.900 --> 00:04:26.220 align:middle line:84% at whether you can decode object information from brain 00:04:26.220 --> 00:04:28.500 align:middle line:84% responses, the brain responses that are used 00:04:28.500 --> 00:04:32.507 align:middle line:84% to do the decoding are often kind of fairly early responses 00:04:32.507 --> 00:04:34.340 align:middle line:84% that you would think would be kind of mostly 00:04:34.340 --> 00:04:35.632 align:middle line:90% feed-forward, in origin, right? 00:04:35.632 --> 00:04:39.740 align:middle line:84% And so it's not just this distinction, in other words, 00:04:39.740 --> 00:04:40.240 align:middle line:90% right? 00:04:40.240 --> 00:04:42.740 align:middle line:84% I agree, this is a pretty coarse distinction between animals 00:04:42.740 --> 00:04:43.480 align:middle line:90% and non-animals. 00:04:43.480 --> 00:04:45.180 align:middle line:84% So there are certainly things that are more fine-grained 00:04:45.180 --> 00:04:46.555 align:middle line:84% that I think you can also account 00:04:46.555 --> 00:04:49.440 align:middle line:84% for with a fairly feed-forward mode of processing, 00:04:49.440 --> 00:04:51.440 align:middle line:84% but it's clearly not the whole story, all right? 00:04:51.440 --> 00:04:54.650 align:middle line:90% Yeah, good question. 00:04:54.650 --> 00:04:59.430 align:middle line:84% OK, so you've got selectivity for fairly simple features 00:04:59.430 --> 00:05:00.730 align:middle line:90% in primary visual cortex. 00:05:00.730 --> 00:05:02.150 align:middle line:90% Think of that as early vision. 00:05:02.150 --> 00:05:04.350 align:middle line:84% You go up to V4, things get a little bit more 00:05:04.350 --> 00:05:07.790 align:middle line:84% complicated, maybe harder to describe. 00:05:07.790 --> 00:05:09.790 align:middle line:84% And then in IT, which we spent most 00:05:09.790 --> 00:05:11.790 align:middle line:84% of our time talking about last time, 00:05:11.790 --> 00:05:14.930 align:middle line:84% you see that it's dominated by the central visual field. 00:05:14.930 --> 00:05:16.430 align:middle line:84% So most of the projections into IT 00:05:16.430 --> 00:05:19.350 align:middle line:84% are from the central part of the visual field. 00:05:19.350 --> 00:05:21.590 align:middle line:84% And how there's lots of these examples now 00:05:21.590 --> 00:05:25.310 align:middle line:84% of pretty complicated neural tuning, 00:05:25.310 --> 00:05:27.830 align:middle line:84% often for things that are kind of behaviorally meaningful, 00:05:27.830 --> 00:05:30.777 align:middle line:84% things like hands, or faces, and so forth. 00:05:30.777 --> 00:05:33.110 align:middle line:84% We talked about how there's some functional organization 00:05:33.110 --> 00:05:36.345 align:middle line:84% within IT and how there's this other major organizational 00:05:36.345 --> 00:05:38.470 align:middle line:84% principle that you see throughout the visual system 00:05:38.470 --> 00:05:41.053 align:middle line:84% that is also very prominent in IT, which is that the receptive 00:05:41.053 --> 00:05:44.310 align:middle line:84% field sizes increase as you go from the beginning 00:05:44.310 --> 00:05:46.910 align:middle line:84% of the visual system up to the deeper stages, OK? 00:05:46.910 --> 00:05:48.990 align:middle line:84% So even within IT, you see a gradient 00:05:48.990 --> 00:05:51.160 align:middle line:90% of receptive field sizes. 00:05:51.160 --> 00:05:54.400 align:middle line:84% Talked about how individual neurons in inferotemporal cortex 00:05:54.400 --> 00:05:58.400 align:middle line:84% show some degree of invariance to position and size. 00:05:58.400 --> 00:06:02.800 align:middle line:84% And then, we transitioned into trying 00:06:02.800 --> 00:06:05.310 align:middle line:84% to think about what these properties mean, 00:06:05.310 --> 00:06:06.560 align:middle line:90% in computational terms, right? 00:06:06.560 --> 00:06:10.080 align:middle line:84% So these are all like different sort of clues 00:06:10.080 --> 00:06:13.080 align:middle line:84% to the basis of recognition and inferotemporal cortex, OK? 00:06:13.080 --> 00:06:15.000 align:middle line:84% But how can we actually turn this 00:06:15.000 --> 00:06:18.080 align:middle line:84% into working models and a computational understanding? 00:06:18.080 --> 00:06:20.720 align:middle line:84% And the key thing that we talked about here 00:06:20.720 --> 00:06:24.400 align:middle line:84% was this idea of thinking about images or stimuli, 00:06:24.400 --> 00:06:27.740 align:middle line:84% in general, as being represented by populations of neurons. 00:06:27.740 --> 00:06:31.760 align:middle line:84% So you can think of the images being represented 00:06:31.760 --> 00:06:35.320 align:middle line:84% in this high-dimensional space, where every axis of the space 00:06:35.320 --> 00:06:36.920 align:middle line:90% is the response of a neuron, OK? 00:06:36.920 --> 00:06:40.888 align:middle line:84% So every image gets represented as a point in that space, OK? 00:06:40.888 --> 00:06:42.680 align:middle line:84% And if you're dealing with different images 00:06:42.680 --> 00:06:44.440 align:middle line:84% of the same thing, you have a set 00:06:44.440 --> 00:06:48.520 align:middle line:84% of points, right, that will form what we often call a manifold. 00:06:48.520 --> 00:06:54.280 align:middle line:84% And we proposed that what a good recognition system ought to do 00:06:54.280 --> 00:06:56.600 align:middle line:84% is generate representations of images 00:06:56.600 --> 00:06:58.960 align:middle line:84% that have this particular property, whereby 00:06:58.960 --> 00:07:01.605 align:middle line:84% all of the images that correspond to one type of thing 00:07:01.605 --> 00:07:02.980 align:middle line:84% would be in one part of the space 00:07:02.980 --> 00:07:05.605 align:middle line:84% and all the other images would be in another part of the space. 00:07:05.605 --> 00:07:07.960 align:middle line:84% And the way that you quantitatively assess that 00:07:07.960 --> 00:07:11.880 align:middle line:84% is by asking whether there is a plane or a hyperplane that 00:07:11.880 --> 00:07:14.880 align:middle line:84% can separate those different sets of points. 00:07:14.880 --> 00:07:17.080 align:middle line:84% And if there is such a plane, then you 00:07:17.080 --> 00:07:20.400 align:middle line:84% can use that to build a linear classifier that 00:07:20.400 --> 00:07:22.320 align:middle line:84% can take the brain response and tell you 00:07:22.320 --> 00:07:26.140 align:middle line:84% whether you're dealing with Joe or Sam, in this particular case, 00:07:26.140 --> 00:07:26.640 align:middle line:90% OK? 00:07:26.640 --> 00:07:30.560 align:middle line:84% So we talked about how linear classifier 00:07:30.560 --> 00:07:32.760 align:middle line:84% is based on a projection operation 00:07:32.760 --> 00:07:35.160 align:middle line:84% onto a vector, which in this case, 00:07:35.160 --> 00:07:38.147 align:middle line:84% would be perpendicular to that hyperplane. 00:07:38.147 --> 00:07:39.980 align:middle line:84% And so the crux of the recognition problem-- 00:07:39.980 --> 00:07:42.000 align:middle line:84% the reason that recognition is hard-- is 00:07:42.000 --> 00:07:44.960 align:middle line:84% that in the input representation, the classes 00:07:44.960 --> 00:07:46.840 align:middle line:84% that you care about, one kind of thing 00:07:46.840 --> 00:07:49.890 align:middle line:84% versus another kind of thing, they're all tangled up, right? 00:07:49.890 --> 00:07:52.370 align:middle line:84% So the points that correspond to one person and the points 00:07:52.370 --> 00:07:53.930 align:middle line:84% that correspond to another person, they're all sort of 00:07:53.930 --> 00:07:56.910 align:middle line:84% intermixed, in a complicated way, in the pixel space, right? 00:07:56.910 --> 00:08:00.890 align:middle line:84% That's kind of why you need a brain. 00:08:00.890 --> 00:08:02.290 align:middle line:84% And there's this proposal that we 00:08:02.290 --> 00:08:06.630 align:middle line:84% tried to evaluate that what a sensory system might do-- 00:08:06.630 --> 00:08:09.730 align:middle line:84% in this case, the ventral visual stream-- is perform 00:08:09.730 --> 00:08:13.570 align:middle line:84% a series of transformations on the input that would end up 00:08:13.570 --> 00:08:15.650 align:middle line:84% with a representational space that would have 00:08:15.650 --> 00:08:18.360 align:middle line:84% this property, OK, where all of the representations of one 00:08:18.360 --> 00:08:20.110 align:middle line:84% kind of thing are in one part of the space 00:08:20.110 --> 00:08:22.410 align:middle line:84% and the representations of another kind of thing 00:08:22.410 --> 00:08:23.410 align:middle line:90% are in a different part. 00:08:23.410 --> 00:08:25.868 align:middle line:84% And so then we talked about evidence that this is the case. 00:08:25.868 --> 00:08:27.450 align:middle line:84% And so this is the experiment that I 00:08:27.450 --> 00:08:30.530 align:middle line:84% was alluding to in my answer to the question, just now. 00:08:30.530 --> 00:08:34.370 align:middle line:84% So we looked at experiments where 00:08:34.370 --> 00:08:38.169 align:middle line:84% responses from a big set of neurons and inferotemporal 00:08:38.169 --> 00:08:40.490 align:middle line:84% cortex were measured to a big set of images, OK? 00:08:40.490 --> 00:08:42.330 align:middle line:84% So here, we have all these images 00:08:42.330 --> 00:08:44.418 align:middle line:84% and all these different recording sites, 00:08:44.418 --> 00:08:46.710 align:middle line:84% which you can think of as neurons or groups of neurons, 00:08:46.710 --> 00:08:47.410 align:middle line:90% OK? 00:08:47.410 --> 00:08:49.750 align:middle line:84% Some neurons respond more to some images than others. 00:08:49.750 --> 00:08:51.570 align:middle line:84% And the question is whether or not 00:08:51.570 --> 00:08:55.250 align:middle line:84% the representation that is implied by that data 00:08:55.250 --> 00:08:59.010 align:middle line:84% would be good, in the sense that we've discussed, 00:08:59.010 --> 00:09:01.890 align:middle line:84% in making object identity explicit-- which, again, is 00:09:01.890 --> 00:09:06.530 align:middle line:84% evaluated by asking whether the two different objects are 00:09:06.530 --> 00:09:08.690 align:middle line:84% linearly separable-- which, again, you evaluate 00:09:08.690 --> 00:09:11.330 align:middle line:84% by fitting a linear classifier, and then measuring 00:09:11.330 --> 00:09:12.570 align:middle line:90% classification performance. 00:09:12.570 --> 00:09:15.410 align:middle line:84% AUDIENCE: Also to test like different orientations and 3D 00:09:15.410 --> 00:09:17.330 align:middle line:90% rotations of the objects. 00:09:17.330 --> 00:09:21.630 align:middle line:84% JOSH MCDERMOTT: So in this particular experiment, 00:09:21.630 --> 00:09:24.730 align:middle line:84% I'm not sure if they looked at different views. 00:09:24.730 --> 00:09:28.210 align:middle line:84% This is showing you different sizes and positions. 00:09:28.210 --> 00:09:30.790 align:middle line:90% That's certainly been done. 00:09:30.790 --> 00:09:33.430 align:middle line:84% And I think would have a similar outcome. 00:09:33.430 --> 00:09:37.690 align:middle line:84% But I don't remember about this particular study, OK. 00:09:37.690 --> 00:09:40.450 align:middle line:84% OK, and so the outcome of this experiment 00:09:40.450 --> 00:09:43.850 align:middle line:84% was that you can classify objects pretty well 00:09:43.850 --> 00:09:49.420 align:middle line:84% using, with a linear classifier and using 00:09:49.420 --> 00:09:52.018 align:middle line:90% the responses of neurons in IT. 00:09:52.018 --> 00:09:54.060 align:middle line:84% So this graph plots the number of recording sites 00:09:54.060 --> 00:09:55.880 align:middle line:84% versus the classification performance. 00:09:55.880 --> 00:09:59.300 align:middle line:84% And once you have a couple hundred recording sites, 00:09:59.300 --> 00:10:01.260 align:middle line:90% you do fairly well, OK? 00:10:01.260 --> 00:10:04.040 align:middle line:84% And we saw that this was at least somewhat specific to IT. 00:10:04.040 --> 00:10:07.140 align:middle line:84% So if you do the same analysis in inferotemporal cortex 00:10:07.140 --> 00:10:10.940 align:middle line:84% and in V4, which we can think of as the area 00:10:10.940 --> 00:10:14.780 align:middle line:84% of the ventral stream that precedes inferotemporal cortex, 00:10:14.780 --> 00:10:16.920 align:middle line:84% classification performance is worse. 00:10:16.920 --> 00:10:19.300 align:middle line:84% Again, these are measured as a function of the number 00:10:19.300 --> 00:10:21.620 align:middle line:90% of recording sites. 00:10:21.620 --> 00:10:25.220 align:middle line:84% And also, the consistency of the classification judgments 00:10:25.220 --> 00:10:27.280 align:middle line:90% with humans is a lot lower. 00:10:27.280 --> 00:10:29.220 align:middle line:84% And so the conclusion from this is 00:10:29.220 --> 00:10:32.660 align:middle line:84% that IT neurons mediate object recognition 00:10:32.660 --> 00:10:36.310 align:middle line:84% by making object identity explicit, where 00:10:36.310 --> 00:10:38.060 align:middle line:84% what it means for something to be explicit 00:10:38.060 --> 00:10:39.980 align:middle line:90% is that it is easily read out. 00:10:39.980 --> 00:10:42.500 align:middle line:84% And what it means for something to be easily read out 00:10:42.500 --> 00:10:46.110 align:middle line:84% is that you can build a linear classifier that 00:10:46.110 --> 00:10:48.320 align:middle line:84% will enable you to classify the bit of information 00:10:48.320 --> 00:10:49.070 align:middle line:90% you're interested. 00:10:49.070 --> 00:10:53.030 align:middle line:84% In this case, object identity, OK? 00:10:53.030 --> 00:10:56.550 align:middle line:84% OK, so that's where we left off, last time. 00:10:56.550 --> 00:10:59.090 align:middle line:84% Are there any questions about that before we move on? 00:10:59.090 --> 00:11:08.830 align:middle line:90% 00:11:08.830 --> 00:11:11.910 align:middle line:84% And so I'll note that in this description 00:11:11.910 --> 00:11:13.570 align:middle line:84% of the basis of object recognition, 00:11:13.570 --> 00:11:20.110 align:middle line:84% we haven't really talked about the mechanistic details of what 00:11:20.110 --> 00:11:22.770 align:middle line:84% these transformations are, at each stage, right? 00:11:22.770 --> 00:11:25.810 align:middle line:84% We've talked about the outcome of those transformations, 00:11:25.810 --> 00:11:29.890 align:middle line:84% right, in terms of a property of the representation, OK. 00:11:29.890 --> 00:11:31.510 align:middle line:84% OK, and those transformations, they 00:11:31.510 --> 00:11:33.910 align:middle line:84% might be complicated and not that easy to describe, 00:11:33.910 --> 00:11:34.730 align:middle line:90% potentially. 00:11:34.730 --> 00:11:38.190 align:middle line:90% 00:11:38.190 --> 00:11:39.810 align:middle line:90% OK, all right. 00:11:39.810 --> 00:11:43.270 align:middle line:84% So another kind of important aspect of our recognition 00:11:43.270 --> 00:11:47.670 align:middle line:84% machinery is that there's evidence for specialization 00:11:47.670 --> 00:11:50.870 align:middle line:84% of this recognition machinery for particular classes 00:11:50.870 --> 00:11:54.590 align:middle line:84% of objects that are maybe especially distinctive 00:11:54.590 --> 00:11:56.170 align:middle line:90% or especially important, OK? 00:11:56.170 --> 00:11:58.990 align:middle line:84% So in particular, neurons that are 00:11:58.990 --> 00:12:01.630 align:middle line:84% responsive to certain special classes of objects 00:12:01.630 --> 00:12:04.270 align:middle line:84% seem to be segregated in the brain. 00:12:04.270 --> 00:12:07.090 align:middle line:84% And we talked about our colleague, Nancy Kanwisher, 00:12:07.090 --> 00:12:09.370 align:middle line:84% who's done a lot of work on this. 00:12:09.370 --> 00:12:11.710 align:middle line:84% And so one particular class, where 00:12:11.710 --> 00:12:16.030 align:middle line:84% there's especially good evidence for this, is faces. 00:12:16.030 --> 00:12:19.910 align:middle line:84% So there are lots of patches of cortex, 00:12:19.910 --> 00:12:25.510 align:middle line:84% in both monkeys and humans, within which the neurons respond 00:12:25.510 --> 00:12:28.350 align:middle line:84% a lot more to faces than to other kinds of images, OK? 00:12:28.350 --> 00:12:32.450 align:middle line:84% And so Nancy Kanwisher discovered one of these, 00:12:32.450 --> 00:12:33.490 align:middle line:90% the fusiform face area. 00:12:33.490 --> 00:12:35.390 align:middle line:90% There's a few others in humans. 00:12:35.390 --> 00:12:39.760 align:middle line:84% And this is the result of an fMRI experiment in macaques, 00:12:39.760 --> 00:12:43.720 align:middle line:84% where they are comparing brain responses to faces to brain 00:12:43.720 --> 00:12:47.320 align:middle line:84% responses to other types of objects. 00:12:47.320 --> 00:12:50.740 align:middle line:84% And there's a bunch of these patches that kind of emerge, OK? 00:12:50.740 --> 00:12:52.400 align:middle line:84% So and then, in this particular study 00:12:52.400 --> 00:12:57.440 align:middle line:84% by Doris Tsao and colleagues, they then use the fMRI results 00:12:57.440 --> 00:12:59.640 align:middle line:90% to direct electrodes, OK? 00:12:59.640 --> 00:13:03.640 align:middle line:84% So they put electrodes into these face patches 00:13:03.640 --> 00:13:07.160 align:middle line:84% and now measured the responses of individual neurons. 00:13:07.160 --> 00:13:09.240 align:middle line:84% So that each row here is a neuron 00:13:09.240 --> 00:13:11.480 align:middle line:84% that they recorded from in one of these patches 00:13:11.480 --> 00:13:13.160 align:middle line:90% to a big set of images, OK? 00:13:13.160 --> 00:13:14.660 align:middle line:90% So there's 96 different images. 00:13:14.660 --> 00:13:17.940 align:middle line:84% And the images are in different categories-- faces bodies, 00:13:17.940 --> 00:13:19.827 align:middle line:90% fruits, gadgets-- 00:13:19.827 --> 00:13:21.660 align:middle line:84% I'm not sure exactly what goes into gadgets, 00:13:21.660 --> 00:13:24.120 align:middle line:90% but I guess gadgets-- 00:13:24.120 --> 00:13:27.380 align:middle line:84% hands, and I think that's scrambled, images, OK? 00:13:27.380 --> 00:13:30.075 align:middle line:84% And so what you're supposed to take away from this-- and then, 00:13:30.075 --> 00:13:32.450 align:middle line:84% the color here represents the firing rate of the neurons, 00:13:32.450 --> 00:13:32.950 align:middle line:90% right? 00:13:32.950 --> 00:13:36.480 align:middle line:84% And so the vast majority of the neurons in these face patches 00:13:36.480 --> 00:13:40.170 align:middle line:84% are responding a lot to all of the images of faces and much 00:13:40.170 --> 00:13:45.434 align:middle line:84% less to all the other kinds of images, OK? 00:13:45.434 --> 00:13:49.690 align:middle line:84% So here's another bit of evidence for the same of thing. 00:13:49.690 --> 00:13:54.750 align:middle line:84% This is a plot of lots and lots of recording sites. 00:13:54.750 --> 00:14:00.770 align:middle line:84% So over 1,000 in macaque inferotemporal cortex. 00:14:00.770 --> 00:14:02.950 align:middle line:84% And each recording site is a dot here. 00:14:02.950 --> 00:14:05.930 align:middle line:84% So they have lots of electrode penetrations. 00:14:05.930 --> 00:14:09.010 align:middle line:84% And then, they're color-coding the electrode sites, 00:14:09.010 --> 00:14:12.290 align:middle line:84% as a function of the extent to which they prefer faces 00:14:12.290 --> 00:14:15.570 align:middle line:90% versus non-face objects, OK? 00:14:15.570 --> 00:14:18.690 align:middle line:84% And so the point is just that you 00:14:18.690 --> 00:14:22.350 align:middle line:84% see these regions where there's a lot of red dots, right? 00:14:22.350 --> 00:14:26.450 align:middle line:84% There's one here, and there's one here, maybe one up there. 00:14:26.450 --> 00:14:30.270 align:middle line:84% OK, and so these various lines of evidence, 00:14:30.270 --> 00:14:34.302 align:middle line:84% they address this kind of long-standing question 00:14:34.302 --> 00:14:36.010 align:middle line:84% in the realm of visual recognition, which 00:14:36.010 --> 00:14:38.790 align:middle line:84% is like whether faces are kind of special, in different ways. 00:14:38.790 --> 00:14:40.930 align:middle line:84% And there's four pieces of evidence 00:14:40.930 --> 00:14:44.690 align:middle line:84% that I think are all kind consistent with the notion 00:14:44.690 --> 00:14:47.370 align:middle line:84% that there is some degree of specialized machinery 00:14:47.370 --> 00:14:49.290 align:middle line:90% for recognizing faces. 00:14:49.290 --> 00:14:51.450 align:middle line:84% The first is what we've already discussed, 00:14:51.450 --> 00:14:53.530 align:middle line:84% which is evidence for a brain region 00:14:53.530 --> 00:14:57.370 align:middle line:84% that responds a lot more to faces than to non-faces. 00:14:57.370 --> 00:15:01.010 align:middle line:84% This is in humans, the fusiform face area, mostly characterized 00:15:01.010 --> 00:15:02.290 align:middle line:90% with fMRI. 00:15:02.290 --> 00:15:03.830 align:middle line:90% So it's right there. 00:15:03.830 --> 00:15:06.370 align:middle line:90% 00:15:06.370 --> 00:15:09.730 align:middle line:90% Here's another view. 00:15:09.730 --> 00:15:11.730 align:middle line:84% Another piece of evidence that there's 00:15:11.730 --> 00:15:13.810 align:middle line:84% something special about face recognition 00:15:13.810 --> 00:15:16.730 align:middle line:84% comes from inversion effects, OK? 00:15:16.730 --> 00:15:21.250 align:middle line:84% So it turns out that face recognition is much better 00:15:21.250 --> 00:15:25.130 align:middle line:84% when the faces are upright than when they are upside down, OK? 00:15:25.130 --> 00:15:28.830 align:middle line:84% And this is not necessarily true for other objects. 00:15:28.830 --> 00:15:31.850 align:middle line:84% And the classic piece of evidence for this 00:15:31.850 --> 00:15:34.410 align:middle line:84% is something called the Thatcher illusion, OK? 00:15:34.410 --> 00:15:36.320 align:middle line:84% And so it's called the Thatcher illusion, 00:15:36.320 --> 00:15:39.100 align:middle line:84% because the original example of this 00:15:39.100 --> 00:15:42.380 align:middle line:84% was performed on the face of Margaret Thatcher-- 00:15:42.380 --> 00:15:44.480 align:middle line:84% because it was done by somebody in Britain, 00:15:44.480 --> 00:15:47.420 align:middle line:84% and Margaret Thatcher was the prime minister of the UK, 00:15:47.420 --> 00:15:48.320 align:middle line:90% at that time, OK? 00:15:48.320 --> 00:15:50.760 align:middle line:84% And so the illusion here-- and so here, 00:15:50.760 --> 00:15:53.540 align:middle line:84% it's done on the face of somebody else-- this is just 00:15:53.540 --> 00:15:55.940 align:middle line:90% what I happen to have handy. 00:15:55.940 --> 00:15:57.820 align:middle line:84% The illusion here results from the fact 00:15:57.820 --> 00:15:59.820 align:middle line:90% that the face is distorted, OK? 00:15:59.820 --> 00:16:03.820 align:middle line:84% So there's a few sections of the image 00:16:03.820 --> 00:16:06.700 align:middle line:84% of the face that have actually just been locally flipped upside 00:16:06.700 --> 00:16:07.580 align:middle line:90% down, OK? 00:16:07.580 --> 00:16:10.720 align:middle line:84% But when you look at it upside down, it just looks fine. 00:16:10.720 --> 00:16:12.220 align:middle line:90% It looks like a face, right? 00:16:12.220 --> 00:16:14.060 align:middle line:90% But what it actually is-- 00:16:14.060 --> 00:16:16.980 align:middle line:84% and this becomes very obvious when it's right side up-- 00:16:16.980 --> 00:16:18.760 align:middle line:90% is this, right? 00:16:18.760 --> 00:16:22.100 align:middle line:84% So you can tell that this is not how the face is 00:16:22.100 --> 00:16:23.500 align:middle line:90% supposed to look, right? 00:16:23.500 --> 00:16:26.340 align:middle line:84% So that and that are exactly the same, 00:16:26.340 --> 00:16:34.648 align:middle line:84% just with the exception of being rotated 180 degrees, OK? 00:16:34.648 --> 00:16:36.440 align:middle line:84% And this is not the only piece of evidence. 00:16:36.440 --> 00:16:37.732 align:middle line:90% There's lots of other evidence. 00:16:37.732 --> 00:16:40.180 align:middle line:84% You can measure the ability to recognize or discriminate 00:16:40.180 --> 00:16:43.720 align:middle line:84% faces and compare that for upright versus inverted images. 00:16:43.720 --> 00:16:47.820 align:middle line:84% And there's a huge advantage for when the faces are upright, OK? 00:16:47.820 --> 00:16:48.682 align:middle line:90% Yeah. 00:16:48.682 --> 00:16:50.140 align:middle line:84% AUDIENCE: Do you know if this-- how 00:16:50.140 --> 00:16:55.053 align:middle line:84% this affects the facial recognition systems, possibly? 00:16:55.053 --> 00:16:57.220 align:middle line:84% JOSH MCDERMOTT: So the question is, does this affect 00:16:57.220 --> 00:16:58.480 align:middle line:90% facial recognition systems? 00:16:58.480 --> 00:17:00.420 align:middle line:84% So are you asking whether a computer vision, 00:17:00.420 --> 00:17:03.777 align:middle line:84% face recognition system would exhibit the same phenomenon? 00:17:03.777 --> 00:17:05.819 align:middle line:84% AUDIENCE: Yeah, so does it fall for the illusion? 00:17:05.819 --> 00:17:09.077 align:middle line:84% Or does it say, like, this is messed up? 00:17:09.077 --> 00:17:10.660 align:middle line:84% JOSH MCDERMOTT: Yeah, so my guess is-- 00:17:10.660 --> 00:17:12.560 align:middle line:84% I actually don't know whether that's true or not. 00:17:12.560 --> 00:17:14.280 align:middle line:84% And somebody has probably looked at this. 00:17:14.280 --> 00:17:16.099 align:middle line:84% My guess is that they probably would 00:17:16.099 --> 00:17:20.045 align:middle line:84% because we think that this comes from the training data, 00:17:20.045 --> 00:17:20.920 align:middle line:90% at some level, right? 00:17:20.920 --> 00:17:24.420 align:middle line:84% Like the naive interpretation of these inversion 00:17:24.420 --> 00:17:29.940 align:middle line:84% effects is just that most of the time faces are upright. 00:17:29.940 --> 00:17:33.710 align:middle line:84% And so your face recognition system 00:17:33.710 --> 00:17:35.012 align:middle line:90% is trained up on all this data. 00:17:35.012 --> 00:17:37.470 align:middle line:84% I mean, again, we don't know whether this is over evolution 00:17:37.470 --> 00:17:39.370 align:middle line:90% or learned via development. 00:17:39.370 --> 00:17:41.390 align:middle line:84% But either way, most of the training data, 00:17:41.390 --> 00:17:44.830 align:middle line:84% essentially, for the recognition system is in one orientation. 00:17:44.830 --> 00:17:48.010 align:middle line:84% And a lot of objects are not like that, right? 00:17:48.010 --> 00:17:52.750 align:middle line:84% So you see chairs from all kinds of different viewpoints. 00:17:52.750 --> 00:17:56.990 align:middle line:90% And are less sensitive to that. 00:17:56.990 --> 00:17:58.605 align:middle line:84% But yeah, my guess is that probably 00:17:58.605 --> 00:18:00.230 align:middle line:84% machine facial recognition systems will 00:18:00.230 --> 00:18:03.510 align:middle line:90% show something kind of similar. 00:18:03.510 --> 00:18:09.230 align:middle line:84% Yeah, and this is part of a long-standing debate 00:18:09.230 --> 00:18:16.430 align:middle line:84% about the extent of viewpoint invariance of human recognition. 00:18:16.430 --> 00:18:20.550 align:middle line:84% So one of the really interesting things about object recognition 00:18:20.550 --> 00:18:22.970 align:middle line:84% is that it's pretty invariant, as we've been discussing. 00:18:22.970 --> 00:18:25.190 align:middle line:84% And one of the most impressive forms of invariance 00:18:25.190 --> 00:18:27.182 align:middle line:84% is invariance to viewpoint, right? 00:18:27.182 --> 00:18:29.390 align:middle line:84% So I can look at this cup from this direction or this 00:18:29.390 --> 00:18:30.480 align:middle line:90% under this direction. 00:18:30.480 --> 00:18:32.355 align:middle line:84% I'm going to be able to tell that it's a cup, 00:18:32.355 --> 00:18:34.320 align:middle line:84% but the image that is produced by the cup 00:18:34.320 --> 00:18:36.903 align:middle line:84% is like totally different in all these different cases, right? 00:18:36.903 --> 00:18:40.880 align:middle line:84% And so there was like a long history 00:18:40.880 --> 00:18:44.120 align:middle line:84% of people thinking about the basis of that viewpoint 00:18:44.120 --> 00:18:44.820 align:middle line:90% and variance. 00:18:44.820 --> 00:18:48.800 align:middle line:84% And so some people have postulated that you actually 00:18:48.800 --> 00:18:50.420 align:middle line:90% infer 3D models of objects. 00:18:50.420 --> 00:18:52.080 align:middle line:84% And that's the basis of recognition. 00:18:52.080 --> 00:18:54.740 align:middle line:84% Others say that you have all these stored 2D views, 00:18:54.740 --> 00:18:56.800 align:middle line:84% but they're kind of linked together. 00:18:56.800 --> 00:18:58.560 align:middle line:84% And there's been lots of experiments 00:18:58.560 --> 00:19:00.880 align:middle line:84% on new kinds of objects, like where you expose people 00:19:00.880 --> 00:19:03.100 align:middle line:84% to particular views of the objects and not others. 00:19:03.100 --> 00:19:06.020 align:middle line:84% And then show that they don't generalize perfectly. 00:19:06.020 --> 00:19:09.405 align:middle line:84% And that suggests that viewpoint invariance is not automatic. 00:19:09.405 --> 00:19:11.280 align:middle line:84% So I think there's various pieces of evidence 00:19:11.280 --> 00:19:16.120 align:middle line:84% that you're only as invariant as your training 00:19:16.120 --> 00:19:18.320 align:middle line:84% data kind of forces you to be, in a lot of cases. 00:19:18.320 --> 00:19:21.180 align:middle line:84% And faces may be kind of an extreme example of that, OK? 00:19:21.180 --> 00:19:24.120 align:middle line:84% But at any rate, like this is not 00:19:24.120 --> 00:19:25.920 align:middle line:84% something that is as pronounced for most 00:19:25.920 --> 00:19:27.140 align:middle line:90% other classes of objects. 00:19:27.140 --> 00:19:31.240 align:middle line:84% So one piece of evidence that face recognition is special. 00:19:31.240 --> 00:19:35.080 align:middle line:84% Another piece of evidence is prosopagnosia. 00:19:35.080 --> 00:19:40.880 align:middle line:84% So there are these examples of people with brain damage 00:19:40.880 --> 00:19:42.840 align:middle line:84% that end up with selective impairments 00:19:42.840 --> 00:19:44.020 align:middle line:90% in recognizing faces. 00:19:44.020 --> 00:19:45.720 align:middle line:84% And that's known as prosopagnosia. 00:19:45.720 --> 00:19:47.640 align:middle line:84% And so the most striking cases of this 00:19:47.640 --> 00:19:50.560 align:middle line:84% come from people who've had a stroke, which 00:19:50.560 --> 00:19:53.200 align:middle line:84% is the typical cause of brain damage in people. 00:19:53.200 --> 00:19:56.640 align:middle line:84% And so here's a quote from a case study of this person, Dr. 00:19:56.640 --> 00:19:59.410 align:middle line:84% P. It says "By and large, he recognized nobody, 00:19:59.410 --> 00:20:01.660 align:middle line:84% neither his family nor his colleagues, nor his pupils, 00:20:01.660 --> 00:20:02.640 align:middle line:90% nor himself. 00:20:02.640 --> 00:20:04.160 align:middle line:84% He recognized a portrait of Einstein 00:20:04.160 --> 00:20:05.993 align:middle line:84% because he picked up the characteristic hair 00:20:05.993 --> 00:20:06.680 align:middle line:90% and mustache. 00:20:06.680 --> 00:20:11.760 align:middle line:84% In the absence of obvious markers, he was utterly lost." 00:20:11.760 --> 00:20:15.680 align:middle line:84% Here's another report-- this is of a person describing 00:20:15.680 --> 00:20:16.560 align:middle line:90% themselves. 00:20:16.560 --> 00:20:18.760 align:middle line:84% "At the club, I saw someone strange staring at me 00:20:18.760 --> 00:20:20.420 align:middle line:84% and asked the steward who it was. 00:20:20.420 --> 00:20:21.440 align:middle line:90% You'll laugh at me. 00:20:21.440 --> 00:20:23.160 align:middle line:84% I've been looking at myself in a mirror." 00:20:23.160 --> 00:20:28.290 align:middle line:84% All right, so people can't even recognize themselves, OK? 00:20:28.290 --> 00:20:31.170 align:middle line:84% And nowadays, so these classic examples 00:20:31.170 --> 00:20:34.530 align:middle line:84% come from people that have pretty serious brain damage. 00:20:34.530 --> 00:20:39.602 align:middle line:84% There's now, I think, growing evidence that face recognition-- 00:20:39.602 --> 00:20:41.310 align:middle line:84% probably a lot of recognition abilities-- 00:20:41.310 --> 00:20:42.312 align:middle line:90% they lie on a continuum. 00:20:42.312 --> 00:20:43.770 align:middle line:84% And some people are actually really 00:20:43.770 --> 00:20:45.270 align:middle line:90% good at recognizing faces. 00:20:45.270 --> 00:20:46.690 align:middle line:90% Others are less good. 00:20:46.690 --> 00:20:51.090 align:middle line:84% And at the extreme end of the less good end 00:20:51.090 --> 00:20:52.610 align:middle line:84% are people who might be considered 00:20:52.610 --> 00:20:55.610 align:middle line:84% to be prosopagnosia, but just congenitally prosopagnosia, OK? 00:20:55.610 --> 00:20:58.710 align:middle line:84% And so there's thought to be a genetic basis of this. 00:20:58.710 --> 00:21:00.730 align:middle line:84% It runs in families and so forth. 00:21:00.730 --> 00:21:04.730 align:middle line:84% So but prosopagnosia is classically 00:21:04.730 --> 00:21:06.970 align:middle line:90% a neuropsychological phenomenon. 00:21:06.970 --> 00:21:10.330 align:middle line:84% And the inference of it from this, right, is like-- 00:21:10.330 --> 00:21:13.890 align:middle line:84% and this is the power of all of these examples of brain 00:21:13.890 --> 00:21:16.850 align:middle line:84% lesions-- is that if you can selectively 00:21:16.850 --> 00:21:22.530 align:middle line:84% disrupt a particular ability, with other abilities remaining 00:21:22.530 --> 00:21:26.232 align:middle line:84% intact, that's evidence for modularity in the brain. 00:21:26.232 --> 00:21:27.690 align:middle line:84% That there is some bit of the brain 00:21:27.690 --> 00:21:30.830 align:middle line:84% that is specifically especially involved in that function. 00:21:30.830 --> 00:21:33.410 align:middle line:84% And if you're unlucky enough to have that particular bit 00:21:33.410 --> 00:21:35.495 align:middle line:84% of the brain damaged, you end up with a deficit 00:21:35.495 --> 00:21:37.870 align:middle line:84% in that particular thing, in this case, face recognition. 00:21:37.870 --> 00:21:40.770 align:middle line:84% So we saw this with achromatopsia, 00:21:40.770 --> 00:21:46.410 align:middle line:84% and here it is with face recognition. 00:21:46.410 --> 00:21:49.410 align:middle line:84% And then we've just seen these examples of face cells, OK? 00:21:49.410 --> 00:21:51.490 align:middle line:84% So a variety of pieces of evidence 00:21:51.490 --> 00:21:54.490 align:middle line:84% that there is some degree of specialization 00:21:54.490 --> 00:21:57.650 align:middle line:84% of the recognition machinery for faces. 00:21:57.650 --> 00:21:58.150 align:middle line:90% All right. 00:21:58.150 --> 00:22:00.650 align:middle line:84% Any questions about face recognition 00:22:00.650 --> 00:22:02.230 align:middle line:90% or this particular idea? 00:22:02.230 --> 00:22:06.933 align:middle line:90% 00:22:06.933 --> 00:22:08.810 align:middle line:90% Mm-hmm? 00:22:08.810 --> 00:22:12.650 align:middle line:84% AUDIENCE: So the time series here-- 00:22:12.650 --> 00:22:17.050 align:middle line:84% or in the responses, it like recognizes it, and then 00:22:17.050 --> 00:22:19.690 align:middle line:90% stops recognizing it? 00:22:19.690 --> 00:22:24.720 align:middle line:84% I'm a little confused as to why it's a lot more spread out, 00:22:24.720 --> 00:22:25.600 align:middle line:90% like the bottom one. 00:22:25.600 --> 00:22:28.520 align:middle line:84% Like it seems like it's continuing to fire later. 00:22:28.520 --> 00:22:29.660 align:middle line:90% JOSH MCDERMOTT: Oh, yeah. 00:22:29.660 --> 00:22:33.418 align:middle line:84% Yeah, I mean, so these are like responses for individual trials. 00:22:33.418 --> 00:22:33.960 align:middle line:90% AUDIENCE: Oh. 00:22:33.960 --> 00:22:36.360 align:middle line:84% JOSH MCDERMOTT: And I mean, neurons are stochastic devices, 00:22:36.360 --> 00:22:36.640 align:middle line:90% right? 00:22:36.640 --> 00:22:37.140 align:middle line:90% So-- 00:22:37.140 --> 00:22:38.220 align:middle line:90% AUDIENCE: OK. 00:22:38.220 --> 00:22:40.428 align:middle line:84% JOSH MCDERMOTT: Yeah, I wouldn't over-interpret this. 00:22:40.428 --> 00:22:42.920 align:middle line:84% I mean, I think if you repeated the experiment, 00:22:42.920 --> 00:22:44.420 align:middle line:84% like the exact pattern of the spikes 00:22:44.420 --> 00:22:46.353 align:middle line:84% is going to be different every time, yeah. 00:22:46.353 --> 00:22:48.020 align:middle line:84% So the main thing to take away from this 00:22:48.020 --> 00:22:52.500 align:middle line:84% is like there's a lot of spikes up here, kind of less here, 00:22:52.500 --> 00:22:54.518 align:middle line:90% some here, less there. 00:22:54.518 --> 00:22:55.560 align:middle line:90% That's sort of the point. 00:22:55.560 --> 00:22:59.420 align:middle line:90% 00:22:59.420 --> 00:23:10.540 align:middle line:84% OK, so this large-scale organization 00:23:10.540 --> 00:23:12.820 align:middle line:84% of selectivity for different kinds of categories that 00:23:12.820 --> 00:23:15.940 align:middle line:84% shows up in these face patches and also things 00:23:15.940 --> 00:23:21.140 align:middle line:84% like the region that's selected for bodies, other things, that 00:23:21.140 --> 00:23:26.550 align:middle line:84% is evident in what we call representational dissimilarity, 00:23:26.550 --> 00:23:27.190 align:middle line:90% OK? 00:23:27.190 --> 00:23:35.190 align:middle line:84% And so this is now a pretty well established and widely used 00:23:35.190 --> 00:23:38.090 align:middle line:84% method for looking at neural representations. 00:23:38.090 --> 00:23:40.493 align:middle line:84% And so it's worth walking through. 00:23:40.493 --> 00:23:41.910 align:middle line:84% Just curious, how many people have 00:23:41.910 --> 00:23:45.630 align:middle line:84% encountered representational dissimilarity before? 00:23:45.630 --> 00:23:46.950 align:middle line:90% OK, a few. 00:23:46.950 --> 00:23:48.270 align:middle line:90% What, in Nancy's class? 00:23:48.270 --> 00:23:49.870 align:middle line:90% Yeah. 00:23:49.870 --> 00:23:53.670 align:middle line:90% OK, so the essential idea here-- 00:23:53.670 --> 00:23:56.050 align:middle line:84% so let me tell you how you construct this, OK? 00:23:56.050 --> 00:23:59.630 align:middle line:84% So these are what are called representational dissimilarity 00:23:59.630 --> 00:24:00.950 align:middle line:90% matrices, OK? 00:24:00.950 --> 00:24:10.110 align:middle line:84% So the matrices here consist of sets of points, OK? 00:24:10.110 --> 00:24:14.990 align:middle line:84% And each point represents the dissimilarity of the brain 00:24:14.990 --> 00:24:18.510 align:middle line:90% responds to two images, OK? 00:24:18.510 --> 00:24:20.870 align:middle line:84% And so the dissimilarity similarity 00:24:20.870 --> 00:24:24.790 align:middle line:84% is, essentially, 1 minus the correlation between the brain 00:24:24.790 --> 00:24:26.200 align:middle line:90% responses to the two images. 00:24:26.200 --> 00:24:27.450 align:middle line:90% So what is the brain response? 00:24:27.450 --> 00:24:29.190 align:middle line:84% The brain response, in this case, 00:24:29.190 --> 00:24:32.810 align:middle line:84% is measured with fMRI from a whole bunch of different voxels, 00:24:32.810 --> 00:24:33.310 align:middle line:90% OK? 00:24:33.310 --> 00:24:37.070 align:middle line:84% So this is an experiment where you present 00:24:37.070 --> 00:24:38.810 align:middle line:90% this large set of images. 00:24:38.810 --> 00:24:40.630 align:middle line:84% Looks like there's a couple hundred here, 00:24:40.630 --> 00:24:42.430 align:middle line:90% OK, to every image. 00:24:42.430 --> 00:24:46.190 align:middle line:84% You measure the fMRI response in this big chunk of visual cortex. 00:24:46.190 --> 00:24:48.090 align:middle line:84% So that's a whole bunch of voxels. 00:24:48.090 --> 00:24:51.170 align:middle line:84% So for each Voxel you get a response to each image, OK? 00:24:51.170 --> 00:24:54.990 align:middle line:84% So you can think of the response to each of these images 00:24:54.990 --> 00:24:56.290 align:middle line:90% as like a big vector. 00:24:56.290 --> 00:24:59.190 align:middle line:84% So let's suppose there's 1,000 voxels in the region that you 00:24:59.190 --> 00:25:00.050 align:middle line:90% measure, OK? 00:25:00.050 --> 00:25:03.670 align:middle line:84% So you get 1,000 numbers for every image, OK, image? 00:25:03.670 --> 00:25:06.470 align:middle line:84% Which is the response to that image in every voxel, OK? 00:25:06.470 --> 00:25:09.230 align:middle line:84% And so then what you can ask is for a pair 00:25:09.230 --> 00:25:14.370 align:middle line:84% of images, how similar, overall, is the brain response, OK? 00:25:14.370 --> 00:25:17.030 align:middle line:84% And so you can measure that as the correlation. 00:25:17.030 --> 00:25:18.530 align:middle line:90% And here, this is dissimilarity. 00:25:18.530 --> 00:25:20.988 align:middle line:84% So dissimilarity, so it's just 1 minus the correlation, OK? 00:25:20.988 --> 00:25:26.360 align:middle line:84% And so blue, here, means that things are not 00:25:26.360 --> 00:25:27.500 align:middle line:90% at all dissimilar. 00:25:27.500 --> 00:25:28.740 align:middle line:90% So they're very similar. 00:25:28.740 --> 00:25:31.600 align:middle line:84% Red, here, means that they're very dissimilar, very different, 00:25:31.600 --> 00:25:32.640 align:middle line:90% OK? 00:25:32.640 --> 00:25:34.280 align:middle line:84% And so, in this experiment, there 00:25:34.280 --> 00:25:39.880 align:middle line:84% were like 200 different images of objects and other things, OK? 00:25:39.880 --> 00:25:43.240 align:middle line:84% And so there's lots and lots of these pairs, OK? 00:25:43.240 --> 00:25:45.120 align:middle line:90% All right. 00:25:45.120 --> 00:25:47.343 align:middle line:84% So what do you get from this, OK? 00:25:47.343 --> 00:25:48.760 align:middle line:84% And it has to be symmetric, right? 00:25:48.760 --> 00:25:52.760 align:middle line:84% Because the rows and the columns represent 00:25:52.760 --> 00:25:56.440 align:middle line:90% the same sets of things, OK? 00:25:56.440 --> 00:25:58.680 align:middle line:84% And also, the diagonal, thus, has 00:25:58.680 --> 00:26:01.040 align:middle line:84% to be 0 because you're just comparing the response 00:26:01.040 --> 00:26:04.830 align:middle line:90% to one thing to itself. 00:26:04.830 --> 00:26:07.420 align:middle line:90% OK, so this is what you get. 00:26:07.420 --> 00:26:09.360 align:middle line:84% And this is a particular study that 00:26:09.360 --> 00:26:12.560 align:middle line:84% compared the representational dissimilarity 00:26:12.560 --> 00:26:14.600 align:middle line:90% in monkeys and humans, OK? 00:26:14.600 --> 00:26:17.940 align:middle line:84% And we'll talk again about why you would want to do that, 00:26:17.940 --> 00:26:20.900 align:middle line:84% but let's just first get into what is actually shown here, OK? 00:26:20.900 --> 00:26:23.800 align:middle line:84% And the other thing I should tell you 00:26:23.800 --> 00:26:26.280 align:middle line:84% is that the images that are presented in the experiment 00:26:26.280 --> 00:26:29.000 align:middle line:84% are kind of organized here, according 00:26:29.000 --> 00:26:31.860 align:middle line:90% to categories of images. 00:26:31.860 --> 00:26:35.640 align:middle line:84% So the first little bunch that you run into here 00:26:35.640 --> 00:26:37.160 align:middle line:90% are images of humans. 00:26:37.160 --> 00:26:38.260 align:middle line:90% So these are faces. 00:26:38.260 --> 00:26:39.880 align:middle line:90% Those are bodies. 00:26:39.880 --> 00:26:45.480 align:middle line:84% These are images of animate things that are not human. 00:26:45.480 --> 00:26:46.293 align:middle line:90% Again, faces. 00:26:46.293 --> 00:26:47.960 align:middle line:84% So these would be animal faces and these 00:26:47.960 --> 00:26:49.600 align:middle line:90% would be animal bodies. 00:26:49.600 --> 00:26:52.180 align:middle line:84% And then, down here, you have inanimate things. 00:26:52.180 --> 00:26:55.200 align:middle line:84% And that's divided up into natural and artificial. 00:26:55.200 --> 00:26:59.200 align:middle line:84% So you got a banana here and like a gun. 00:26:59.200 --> 00:27:01.760 align:middle line:84% It's just all kinds of random images, OK? 00:27:01.760 --> 00:27:02.880 align:middle line:90% All right. 00:27:02.880 --> 00:27:06.120 align:middle line:84% And so when you look at the matrix here, 00:27:06.120 --> 00:27:09.480 align:middle line:84% there's a few things that kind of pop out. 00:27:09.480 --> 00:27:11.240 align:middle line:84% The first is that there's what we call 00:27:11.240 --> 00:27:13.280 align:middle line:90% block diagonal structure, OK? 00:27:13.280 --> 00:27:16.050 align:middle line:84% So if you step back and squint your eyes, 00:27:16.050 --> 00:27:19.770 align:middle line:84% like there's kind of a blue square here, right? 00:27:19.770 --> 00:27:24.310 align:middle line:84% And kind of a blue square here, and then red here and red here, 00:27:24.310 --> 00:27:24.810 align:middle line:90% OK? 00:27:24.810 --> 00:27:29.130 align:middle line:84% And so that's an indication that the images 00:27:29.130 --> 00:27:32.230 align:middle line:84% of the animate objects-- so that's the first half, 00:27:32.230 --> 00:27:35.090 align:middle line:84% all right, are kind of producing similar brain 00:27:35.090 --> 00:27:36.890 align:middle line:90% responses, on average, right? 00:27:36.890 --> 00:27:39.690 align:middle line:84% And the inanimate objects are kind of producing similar brain 00:27:39.690 --> 00:27:40.750 align:middle line:90% responses, on average. 00:27:40.750 --> 00:27:43.850 align:middle line:84% But that the animate and the inanimate images 00:27:43.850 --> 00:27:48.110 align:middle line:84% tend to produce pretty different brain responses, OK? 00:27:48.110 --> 00:27:50.650 align:middle line:84% So you can also see some really strong block diagonal 00:27:50.650 --> 00:27:52.330 align:middle line:90% structure, right here. 00:27:52.330 --> 00:27:53.190 align:middle line:90% And so what is that? 00:27:53.190 --> 00:27:56.410 align:middle line:90% Well, that's faces, right? 00:27:56.410 --> 00:28:00.710 align:middle line:90% So you've got human faces here. 00:28:00.710 --> 00:28:02.870 align:middle line:90% So that's really solidly blue. 00:28:02.870 --> 00:28:07.290 align:middle line:84% So all of the face images are generating very similar 00:28:07.290 --> 00:28:11.290 align:middle line:90% aggregate brain responses. 00:28:11.290 --> 00:28:13.050 align:middle line:84% And then, if you look down here, there's 00:28:13.050 --> 00:28:15.120 align:middle line:84% another kind of blue square, all right? 00:28:15.120 --> 00:28:17.180 align:middle line:90% So that's all the animal faces. 00:28:17.180 --> 00:28:20.120 align:middle line:84% OK, so those are also generating very similar brain responses. 00:28:20.120 --> 00:28:24.530 align:middle line:84% But then, you also get a blue square here, OK? 00:28:24.530 --> 00:28:26.440 align:middle line:90% So What does that tell us? 00:28:26.440 --> 00:28:28.820 align:middle line:90% Anybody want to tell me? 00:28:28.820 --> 00:28:30.500 align:middle line:84% AUDIENCE: Animal face, human face. 00:28:30.500 --> 00:28:31.820 align:middle line:84% JOSH MCDERMOTT: Yeah, so the animal faces 00:28:31.820 --> 00:28:34.300 align:middle line:84% and the human faces are also producing pretty similar brain 00:28:34.300 --> 00:28:37.980 align:middle line:90% responses, OK? 00:28:37.980 --> 00:28:42.660 align:middle line:84% OK, so this is one way to characterize the brain's 00:28:42.660 --> 00:28:46.100 align:middle line:90% representation of objects, OK? 00:28:46.100 --> 00:28:50.440 align:middle line:84% Now, one of the things that this was used for-- 00:28:50.440 --> 00:28:53.300 align:middle line:84% and this was really the point of this particular paper, which 00:28:53.300 --> 00:28:55.700 align:middle line:84% introduced this method-- but then what they were really 00:28:55.700 --> 00:28:57.900 align:middle line:84% using it for was to try to compare 00:28:57.900 --> 00:29:00.700 align:middle line:84% the monkey visual system and the human visual system, all right? 00:29:00.700 --> 00:29:02.575 align:middle line:84% And so the monkey and the human visual system 00:29:02.575 --> 00:29:04.300 align:middle line:84% have-- there's certain homologies, 00:29:04.300 --> 00:29:05.240 align:middle line:90% in terms of the areas. 00:29:05.240 --> 00:29:08.860 align:middle line:84% But really, they're kind of different, right? 00:29:08.860 --> 00:29:11.300 align:middle line:84% So like, in humans, like the visual system 00:29:11.300 --> 00:29:13.540 align:middle line:84% is kind of scrunched into the back of the brain 00:29:13.540 --> 00:29:16.527 align:middle line:84% because you use your temporal lobes for language 00:29:16.527 --> 00:29:17.360 align:middle line:90% and stuff like that. 00:29:17.360 --> 00:29:19.360 align:middle line:84% Whereas in the monkey, the visual system kind of 00:29:19.360 --> 00:29:21.740 align:middle line:84% extends into the temporal lobe, all right? 00:29:21.740 --> 00:29:23.620 align:middle line:84% So the actual sort of physical substrate 00:29:23.620 --> 00:29:24.900 align:middle line:90% is kind of different, OK? 00:29:24.900 --> 00:29:26.905 align:middle line:84% So it's not really totally obvious, 00:29:26.905 --> 00:29:28.780 align:middle line:84% like how you would actually make a comparison 00:29:28.780 --> 00:29:30.488 align:middle line:84% of the human and the monkey visual system 00:29:30.488 --> 00:29:33.040 align:middle line:84% at some of really fine-grained quantitative level, OK? 00:29:33.040 --> 00:29:36.140 align:middle line:84% But this particular analysis method 00:29:36.140 --> 00:29:38.343 align:middle line:84% gives you a way to actually make comparisons 00:29:38.343 --> 00:29:40.760 align:middle line:84% across brains, even if it's like different species, right? 00:29:40.760 --> 00:29:46.260 align:middle line:84% So you can ask is the structure of the representational space, 00:29:46.260 --> 00:29:49.340 align:middle line:84% as measured in this particular way, of like the similarity 00:29:49.340 --> 00:29:50.940 align:middle line:84% of the responses to different images, 00:29:50.940 --> 00:29:53.662 align:middle line:84% is that's shared between monkeys and humans? 00:29:53.662 --> 00:29:55.620 align:middle line:84% And what you're supposed to take away from this 00:29:55.620 --> 00:29:57.953 align:middle line:84% is that the thing on the left looks a lot like the thing 00:29:57.953 --> 00:29:58.860 align:middle line:90% on the right, OK? 00:29:58.860 --> 00:30:04.860 align:middle line:84% So in both species, you see this coarse scale structure 00:30:04.860 --> 00:30:08.140 align:middle line:84% of different responses to animate and inanimate objects. 00:30:08.140 --> 00:30:12.310 align:middle line:84% Faces seem to really stand out as being kind of special. 00:30:12.310 --> 00:30:15.270 align:middle line:84% Animal and human faces seem to be represented similarly, 00:30:15.270 --> 00:30:17.330 align:middle line:90% both in monkeys and humans. 00:30:17.330 --> 00:30:20.430 align:middle line:90% 00:30:20.430 --> 00:30:21.990 align:middle line:90% Yeah. 00:30:21.990 --> 00:30:25.190 align:middle line:84% So the take home message here is that this 00:30:25.190 --> 00:30:29.370 align:middle line:84% is one way to analyze brain representations. 00:30:29.370 --> 00:30:31.170 align:middle line:84% It's called representational dissimilarity. 00:30:31.170 --> 00:30:34.750 align:middle line:84% It's in pretty widespread use now. 00:30:34.750 --> 00:30:37.555 align:middle line:84% It's based on population level responses. 00:30:37.555 --> 00:30:39.430 align:middle line:84% So in this case, it's like the fMRI responses 00:30:39.430 --> 00:30:42.790 align:middle line:84% from a big chunk of brain, OK, to like a pretty large set 00:30:42.790 --> 00:30:45.990 align:middle line:90% of stimuli, OK? 00:30:45.990 --> 00:30:49.030 align:middle line:84% And we believe that the structure here is mostly 00:30:49.030 --> 00:30:51.790 align:middle line:84% driven by the large-scale organization of category 00:30:51.790 --> 00:30:52.570 align:middle line:90% selectivity. 00:30:52.570 --> 00:30:55.550 align:middle line:84% So the fact that there's this fusiform face 00:30:55.550 --> 00:30:58.490 align:middle line:84% area that responds a lot more to faces than to other stuff. 00:30:58.490 --> 00:31:01.830 align:middle line:84% And so what that means is that if you 00:31:01.830 --> 00:31:06.150 align:middle line:84% look at the response of the entire visual system to objects, 00:31:06.150 --> 00:31:07.950 align:middle line:84% faces are going to tend to all produce 00:31:07.950 --> 00:31:10.075 align:middle line:84% pretty similar responses-- because there's this one 00:31:10.075 --> 00:31:12.480 align:middle line:84% set of voxels that generates a huge response to faces 00:31:12.480 --> 00:31:14.772 align:middle line:84% and not much of a response to anything else, all right? 00:31:14.772 --> 00:31:17.320 align:middle line:84% And you see kind of similar stuff for bodies. 00:31:17.320 --> 00:31:21.360 align:middle line:84% So this large scale category selectivity that we certainly 00:31:21.360 --> 00:31:24.120 align:middle line:84% see, at the level of fMRI, contributes a lot 00:31:24.120 --> 00:31:26.800 align:middle line:90% to this kind of thing. 00:31:26.800 --> 00:31:28.880 align:middle line:90% Ask me questions. 00:31:28.880 --> 00:31:29.440 align:middle line:90% Yeah. 00:31:29.440 --> 00:31:31.640 align:middle line:84% AUDIENCE: What's the benefit of using dissimilarity 00:31:31.640 --> 00:31:33.568 align:middle line:90% for second level correlation? 00:31:33.568 --> 00:31:35.860 align:middle line:84% JOSH MCDERMOTT: I'm not sure that there is much of one. 00:31:35.860 --> 00:31:37.780 align:middle line:84% I think there's some reason for it, 00:31:37.780 --> 00:31:40.120 align:middle line:90% and I forget what it is, yeah. 00:31:40.120 --> 00:31:44.492 align:middle line:84% I mean, it's got the same information in it, yeah. 00:31:44.492 --> 00:31:46.700 align:middle line:84% AUDIENCE: Yeah, so it just seems like one extra step. 00:31:46.700 --> 00:31:49.240 align:middle line:90% 00:31:49.240 --> 00:31:53.770 align:middle line:84% JOSH MCDERMOTT: Well, yeah, I guess 00:31:53.770 --> 00:31:56.020 align:middle line:84% the idea is that it's more like a distance, all right? 00:31:56.020 --> 00:31:59.760 align:middle line:84% And so people often think of this as like capturing geometry. 00:31:59.760 --> 00:32:02.240 align:middle line:84% And so just dealing with a distance-based measure 00:32:02.240 --> 00:32:04.740 align:middle line:84% is appealing, from that standpoint, right? 00:32:04.740 --> 00:32:08.220 align:middle line:84% But I think you could draw a lot of the same inferences 00:32:08.220 --> 00:32:11.000 align:middle line:90% without doing that, yeah. 00:32:11.000 --> 00:32:11.680 align:middle line:90% Yeah? 00:32:11.680 --> 00:32:14.680 align:middle line:84% AUDIENCE: Can you use this to draw any information 00:32:14.680 --> 00:32:16.820 align:middle line:84% on differences between monkeys and humans? 00:32:16.820 --> 00:32:19.600 align:middle line:84% Like, for example, like the human one 00:32:19.600 --> 00:32:22.740 align:middle line:84% looks to be like a lot more well defined, in terms of borders. 00:32:22.740 --> 00:32:30.160 align:middle line:84% And also, like for example, in the monkey one, 00:32:30.160 --> 00:32:33.920 align:middle line:84% high dissimilarity between human face and not human body. 00:32:33.920 --> 00:32:36.200 align:middle line:84% But you don't really see that in humans. 00:32:36.200 --> 00:32:37.360 align:middle line:90% Like, I don't know. 00:32:37.360 --> 00:32:40.600 align:middle line:84% Is there some explanation for these kinds of things? 00:32:40.600 --> 00:32:42.240 align:middle line:84% JOSH MCDERMOTT: Yeah, I mean, I think 00:32:42.240 --> 00:32:43.860 align:middle line:90% your observation is correct. 00:32:43.860 --> 00:32:45.560 align:middle line:84% There are some subtle differences. 00:32:45.560 --> 00:32:48.560 align:middle line:84% I mean, exactly what that means and why 00:32:48.560 --> 00:32:52.080 align:middle line:90% there is not totally clear. 00:32:52.080 --> 00:32:54.640 align:middle line:84% But yes, you can conclude that the representations are not 00:32:54.640 --> 00:32:57.560 align:middle line:90% identical. 00:32:57.560 --> 00:32:59.600 align:middle line:84% And without something like this, it's 00:32:59.600 --> 00:33:04.520 align:middle line:84% not even clear how would you go about evaluating that, right? 00:33:04.520 --> 00:33:06.390 align:middle line:84% So yeah, there's some similarities, 00:33:06.390 --> 00:33:08.970 align:middle line:84% but they're also not exactly the same, and. 00:33:08.970 --> 00:33:11.490 align:middle line:90% 00:33:11.490 --> 00:33:12.770 align:middle line:90% Yeah. 00:33:12.770 --> 00:33:15.930 align:middle line:84% AUDIENCE: Has this been done for animals other than monkeys? 00:33:15.930 --> 00:33:19.550 align:middle line:84% Like, is there any comparison between the different types 00:33:19.550 --> 00:33:21.028 align:middle line:90% of animals and humans? 00:33:21.028 --> 00:33:22.570 align:middle line:84% JOSH MCDERMOTT: I'm not aware of this 00:33:22.570 --> 00:33:25.350 align:middle line:84% having been done with any animal other than monkeys. 00:33:25.350 --> 00:33:32.050 align:middle line:90% 00:33:32.050 --> 00:33:35.330 align:middle line:84% Yeah, I mean, it's interesting that the animal models 00:33:35.330 --> 00:33:39.650 align:middle line:84% that are popular in neuroscience has changed over the years. 00:33:39.650 --> 00:33:45.010 align:middle line:84% So the early days, like the '60s and '70s, like there were lots 00:33:45.010 --> 00:33:49.370 align:middle line:84% of experiments done on cats and some on monkeys. 00:33:49.370 --> 00:33:51.490 align:middle line:84% And then, but really, hardly anybody like, does 00:33:51.490 --> 00:33:53.570 align:middle line:90% anything with cats anymore. 00:33:53.570 --> 00:33:55.530 align:middle line:84% And then, nowadays, really because of 00:33:55.530 --> 00:33:59.150 align:middle line:84% the molecular genetic tools that are available in rodents, 00:33:59.150 --> 00:34:04.890 align:middle line:84% there's lots of people who work on mice and rats. 00:34:04.890 --> 00:34:08.010 align:middle line:84% But their visual system is very different from humans. 00:34:08.010 --> 00:34:11.477 align:middle line:84% So I don't think anybody has done an experiment 00:34:11.477 --> 00:34:12.310 align:middle line:90% like this with them. 00:34:12.310 --> 00:34:14.630 align:middle line:84% That would presumably would look totally different. 00:34:14.630 --> 00:34:16.630 align:middle line:84% Big area for cheese or something, right? 00:34:16.630 --> 00:34:19.210 align:middle line:90% So yeah. 00:34:19.210 --> 00:34:21.690 align:middle line:84% AUDIENCE: I'm curious if a similar type of representation 00:34:21.690 --> 00:34:23.250 align:middle line:84% of the similarity matrix has been 00:34:23.250 --> 00:34:25.909 align:middle line:84% done at different levels of development. 00:34:25.909 --> 00:34:29.730 align:middle line:84% So like if these associations are more ingrained 00:34:29.730 --> 00:34:32.250 align:middle line:90% or they're learned through time. 00:34:32.250 --> 00:34:35.210 align:middle line:84% JOSH MCDERMOTT: Yeah, so I don't believe it has. 00:34:35.210 --> 00:34:38.929 align:middle line:84% And the reason for this is that doing an experiment like this 00:34:38.929 --> 00:34:45.330 align:middle line:90% in children is really hard. 00:34:45.330 --> 00:34:48.008 align:middle line:84% Because really, the number one thing 00:34:48.008 --> 00:34:50.050 align:middle line:84% that determines the success of an fMRI experiment 00:34:50.050 --> 00:34:52.250 align:middle line:84% is whether the participants hold still. 00:34:52.250 --> 00:34:56.070 align:middle line:84% And kids are just not as good at staying still. 00:34:56.070 --> 00:34:58.945 align:middle line:84% And so it's really hard to get the kind of power 00:34:58.945 --> 00:35:00.570 align:middle line:84% that you need to do an experiment where 00:35:00.570 --> 00:35:02.220 align:middle line:90% you have like 200 stimuli. 00:35:02.220 --> 00:35:04.220 align:middle line:84% So I mean, there are people in this building, 00:35:04.220 --> 00:35:06.340 align:middle line:84% in like Rebecca Saxe's lab that are 00:35:06.340 --> 00:35:10.660 align:middle line:84% working on doing infant fMRI to look at questions like this. 00:35:10.660 --> 00:35:14.285 align:middle line:84% But it's because it's so much harder to get good data, 00:35:14.285 --> 00:35:16.660 align:middle line:84% like typically the experiments will have maybe four kinds 00:35:16.660 --> 00:35:18.700 align:middle line:90% of conditions, rather than 200. 00:35:18.700 --> 00:35:22.660 align:middle line:84% So I mean, you could look at a very, very coarse 00:35:22.660 --> 00:35:24.280 align:middle line:84% representational dissimilarity matrix. 00:35:24.280 --> 00:35:26.405 align:middle line:84% In fact, I think they may have done that, actually. 00:35:26.405 --> 00:35:29.260 align:middle line:84% But nothing like this, with like 200, yeah. 00:35:29.260 --> 00:35:30.560 align:middle line:90% So yeah. 00:35:30.560 --> 00:35:32.320 align:middle line:84% So it's a really interesting question. 00:35:32.320 --> 00:35:36.140 align:middle line:84% But they do find that you kind of see evidence 00:35:36.140 --> 00:35:40.340 align:middle line:84% for face selectivity in really young infants. 00:35:40.340 --> 00:35:42.520 align:middle line:84% And I think, also, maybe body selectivity, yeah. 00:35:42.520 --> 00:35:44.695 align:middle line:84% So Heather Kosakowski is a grad student 00:35:44.695 --> 00:35:45.820 align:middle line:90% who did a lot of that work. 00:35:45.820 --> 00:35:47.220 align:middle line:90% You should check out her papers. 00:35:47.220 --> 00:35:49.260 align:middle line:90% Yeah. 00:35:49.260 --> 00:35:57.620 align:middle line:84% OK, so we talked about this idea that we can kind of think 00:35:57.620 --> 00:36:02.350 align:middle line:84% of recognition systems as having a computational goal, right? 00:36:02.350 --> 00:36:04.030 align:middle line:84% Which is that you take this image 00:36:04.030 --> 00:36:06.790 align:middle line:84% and you want to transform the representation of that image 00:36:06.790 --> 00:36:11.710 align:middle line:84% into a representation that makes object categories explicit, 00:36:11.710 --> 00:36:13.870 align:middle line:84% right, by kind of putting them in different parts 00:36:13.870 --> 00:36:15.412 align:middle line:84% of the representational space, right? 00:36:15.412 --> 00:36:17.630 align:middle line:84% And so we talked about evidence that if you 00:36:17.630 --> 00:36:20.210 align:middle line:84% look at the representations in inferotemporal cortex, 00:36:20.210 --> 00:36:21.920 align:middle line:90% they have that property. 00:36:21.920 --> 00:36:23.670 align:middle line:84% But we didn't really talk at all about how 00:36:23.670 --> 00:36:27.910 align:middle line:84% you would go about achieving that property, OK? 00:36:27.910 --> 00:36:32.310 align:middle line:84% And so one kind of interesting development in this field 00:36:32.310 --> 00:36:35.870 align:middle line:84% is the advent of artificial neural networks 00:36:35.870 --> 00:36:39.310 align:middle line:84% that can recognize objects from images pretty well, OK? 00:36:39.310 --> 00:36:43.768 align:middle line:84% And this is something that it just wasn't the case 00:36:43.768 --> 00:36:44.810 align:middle line:90% even 10 years ago, right? 00:36:44.810 --> 00:36:48.510 align:middle line:84% So this is a pretty recent kind of development. 00:36:48.510 --> 00:36:52.590 align:middle line:84% And so this has led to a lot of interesting analogies 00:36:52.590 --> 00:36:55.830 align:middle line:84% between, in this case, the ventral visual stream 00:36:55.830 --> 00:36:58.590 align:middle line:84% and a neural network, an artificial neural network 00:36:58.590 --> 00:37:02.123 align:middle line:90% that recognizes objects, OK? 00:37:02.123 --> 00:37:06.710 align:middle line:84% So this is a diagram of one common type 00:37:06.710 --> 00:37:08.590 align:middle line:84% of artificial neural network called 00:37:08.590 --> 00:37:11.750 align:middle line:90% a convolutional neural network. 00:37:11.750 --> 00:37:14.030 align:middle line:84% And so these are machine systems that 00:37:14.030 --> 00:37:20.670 align:middle line:84% consist of cascades of simple operations. 00:37:20.670 --> 00:37:23.550 align:middle line:84% The operations that are in these kinds of systems 00:37:23.550 --> 00:37:26.310 align:middle line:84% are, arguably, loosely inspired from things 00:37:26.310 --> 00:37:30.510 align:middle line:84% that people initially observed or theorized about in the brain. 00:37:30.510 --> 00:37:33.110 align:middle line:84% So they perform filtering operations, 00:37:33.110 --> 00:37:37.270 align:middle line:84% thresholding operations, pooling, normalization. 00:37:37.270 --> 00:37:40.770 align:middle line:84% And these are each individually simple operations, 00:37:40.770 --> 00:37:44.230 align:middle line:84% but then when you stack them, OK, you 00:37:44.230 --> 00:37:46.170 align:middle line:84% can get pretty complicated behavior out, OK? 00:37:46.170 --> 00:37:48.430 align:middle line:84% And the critical thing that kind of 00:37:48.430 --> 00:37:52.270 align:middle line:84% makes these architectures work and makes them useful 00:37:52.270 --> 00:37:54.430 align:middle line:84% is that they're pretty easy to optimize, all right? 00:37:54.430 --> 00:37:57.070 align:middle line:84% So people, for a long time, we're 00:37:57.070 --> 00:37:59.720 align:middle line:84% not actually able to successfully optimize 00:37:59.720 --> 00:38:02.520 align:middle line:84% big systems that looked like this. 00:38:02.520 --> 00:38:04.360 align:middle line:84% But nowadays, we just know how to do it, OK? 00:38:04.360 --> 00:38:07.600 align:middle line:84% And the essential way by which this works 00:38:07.600 --> 00:38:11.615 align:middle line:90% is really pretty simple. 00:38:11.615 --> 00:38:12.740 align:middle line:90% And I'll tell you about it. 00:38:12.740 --> 00:38:17.280 align:middle line:84% But as I said, the most common architecture for sensory tasks, 00:38:17.280 --> 00:38:21.000 align:middle line:84% in the one that's really useful for making comparisons 00:38:21.000 --> 00:38:22.945 align:middle line:84% to biological sensory systems, is 00:38:22.945 --> 00:38:24.820 align:middle line:84% what's called a convolutional neural network. 00:38:24.820 --> 00:38:27.560 align:middle line:84% So you remember we talked about the operation of a convolution, 00:38:27.560 --> 00:38:28.060 align:middle line:90% right? 00:38:28.060 --> 00:38:31.320 align:middle line:84% So what a convolution is, is you've got a filter. 00:38:31.320 --> 00:38:34.240 align:middle line:84% And you apply that filter at all the locations in your signal, 00:38:34.240 --> 00:38:34.740 align:middle line:90% right? 00:38:34.740 --> 00:38:38.840 align:middle line:84% So in an image that would be all the x and the y locations, 00:38:38.840 --> 00:38:39.340 align:middle line:90% right? 00:38:39.340 --> 00:38:44.440 align:middle line:84% In a sound, this would be all the temporal positions, OK? 00:38:44.440 --> 00:38:48.640 align:middle line:84% And so we talked about this idea that, in the context of vision, 00:38:48.640 --> 00:38:51.200 align:middle line:84% you can think of the convolution operation 00:38:51.200 --> 00:38:55.920 align:middle line:84% as representing what a population of neurons that 00:38:55.920 --> 00:38:58.140 align:middle line:84% have the same kind of receptive field, 00:38:58.140 --> 00:38:59.980 align:middle line:84% just at different spatial positions-- 00:38:59.980 --> 00:39:02.280 align:middle line:84% so think of center surround receptive fields, 00:39:02.280 --> 00:39:06.480 align:middle line:84% in the retina or orientation selective receptive fields 00:39:06.480 --> 00:39:10.360 align:middle line:84% in V1, what they would essentially do to an image 00:39:10.360 --> 00:39:12.520 align:middle line:84% when they're all acting in concert, OK? 00:39:12.520 --> 00:39:14.600 align:middle line:84% And so one of these neural networks 00:39:14.600 --> 00:39:16.380 align:middle line:90% is performing that operation. 00:39:16.380 --> 00:39:18.600 align:middle line:84% So each one of these sheets kind of 00:39:18.600 --> 00:39:21.000 align:middle line:84% represents the response of a filter 00:39:21.000 --> 00:39:23.740 align:middle line:84% at all the different possible locations in an image, OK? 00:39:23.740 --> 00:39:26.880 align:middle line:84% And the different sheets kind of represent different filters. 00:39:26.880 --> 00:39:29.380 align:middle line:84% So this one could be horizontal, this one could be vertical, 00:39:29.380 --> 00:39:34.560 align:middle line:84% this one could be diagonal, and so forth, OK? 00:39:34.560 --> 00:39:36.160 align:middle line:84% All right, so you have multiple layers 00:39:36.160 --> 00:39:40.160 align:middle line:84% of these filters separated by non-linearities and pooling 00:39:40.160 --> 00:39:41.020 align:middle line:90% operations. 00:39:41.020 --> 00:39:46.080 align:middle line:84% And what makes them useful, what makes them really 00:39:46.080 --> 00:39:51.760 align:middle line:84% work is that you can learn the filter parameters using 00:39:51.760 --> 00:39:53.840 align:middle line:84% gradient descent, in order to maximize 00:39:53.840 --> 00:39:56.730 align:middle line:84% the performance of a task, right? 00:39:56.730 --> 00:40:00.050 align:middle line:84% And really a very popular task and one that's very widely used 00:40:00.050 --> 00:40:02.370 align:middle line:90% is object classification, OK? 00:40:02.370 --> 00:40:08.450 align:middle line:84% And so these systems are trained using these big data 00:40:08.450 --> 00:40:10.825 align:middle line:84% sets with lots and lots of images, where people have gone 00:40:10.825 --> 00:40:12.992 align:middle line:84% through, and with every image, they label the image, 00:40:12.992 --> 00:40:14.810 align:middle line:84% and say, that's a cat, that's a dog, that's 00:40:14.810 --> 00:40:17.870 align:middle line:84% a chair, that's a building, and so on, and so forth. 00:40:17.870 --> 00:40:20.170 align:middle line:84% So you have this huge set of images. 00:40:20.170 --> 00:40:23.355 align:middle line:84% You can present the images to the model. 00:40:23.355 --> 00:40:24.730 align:middle line:84% The model will give you an answer 00:40:24.730 --> 00:40:26.450 align:middle line:90% for what class it thinks it is. 00:40:26.450 --> 00:40:29.310 align:middle line:84% Initially, like the weight the filters will be randomly set. 00:40:29.310 --> 00:40:31.210 align:middle line:84% And so the answers will mostly be wrong. 00:40:31.210 --> 00:40:34.530 align:middle line:84% But the gradient tells you how should I 00:40:34.530 --> 00:40:36.370 align:middle line:84% change the parameters of the filters, 00:40:36.370 --> 00:40:40.810 align:middle line:84% in order to have the model make fewer errors, OK? 00:40:40.810 --> 00:40:44.490 align:middle line:84% So you do gradient descent over many, many iterations. 00:40:44.490 --> 00:40:48.410 align:middle line:84% And the model moves into a place where it can perform well 00:40:48.410 --> 00:40:49.570 align:middle line:90% at the task, OK? 00:40:49.570 --> 00:40:54.260 align:middle line:84% And so this is now kind of become 00:40:54.260 --> 00:40:56.580 align:middle line:84% one of the dominant methods in engineering. 00:40:56.580 --> 00:41:00.600 align:middle line:84% And object recognition was like a key in early success story, 00:41:00.600 --> 00:41:01.760 align:middle line:90% OK. 00:41:01.760 --> 00:41:05.600 align:middle line:84% So in some sense, this is the realm of computer vision. 00:41:05.600 --> 00:41:10.140 align:middle line:84% But these systems, because they're solving a problem 00:41:10.140 --> 00:41:12.740 align:middle line:84% that we think is also important for biological vision, 00:41:12.740 --> 00:41:17.700 align:middle line:84% they give you a useful computational model 00:41:17.700 --> 00:41:21.880 align:middle line:84% that you can propose as a model of biological vision. 00:41:21.880 --> 00:41:24.300 align:middle line:84% And you can ask, well, does this model actually 00:41:24.300 --> 00:41:27.740 align:middle line:84% do some of the same kinds of things that humans do, OK? 00:41:27.740 --> 00:41:29.900 align:middle line:84% So this is now sort of a cottage industry of people 00:41:29.900 --> 00:41:32.780 align:middle line:84% building these models, and analyzing them, and comparing 00:41:32.780 --> 00:41:37.060 align:middle line:84% them to human sensory systems, in different ways. 00:41:37.060 --> 00:41:39.420 align:middle line:84% And so one kind of thing that you can do 00:41:39.420 --> 00:41:42.660 align:middle line:84% is measure representational similarity, OK? 00:41:42.660 --> 00:41:47.478 align:middle line:84% So these are representational dissimilarity matrices. 00:41:47.478 --> 00:41:49.520 align:middle line:84% This is the kind of thing that we just looked at. 00:41:49.520 --> 00:41:52.620 align:middle line:84% This is from human IT using fMRI. 00:41:52.620 --> 00:41:55.220 align:middle line:84% So again, you get this block diagonal structure. 00:41:55.220 --> 00:41:58.980 align:middle line:84% And this is that same quantity measured in a-- 00:41:58.980 --> 00:42:02.340 align:middle line:84% HCNN is Hierarchical Convolutional Neural Network 00:42:02.340 --> 00:42:05.060 align:middle line:84% that's trained to recognize objects. 00:42:05.060 --> 00:42:08.760 align:middle line:84% And you can see that they're not exactly the same. 00:42:08.760 --> 00:42:10.520 align:middle line:84% But there are some common features. 00:42:10.520 --> 00:42:12.980 align:middle line:84% So again, you see this pretty clear distinction 00:42:12.980 --> 00:42:16.100 align:middle line:84% between animate and inanimate objects. 00:42:16.100 --> 00:42:19.620 align:middle line:84% You see that faces are kind of special. 00:42:19.620 --> 00:42:22.120 align:middle line:84% It's interesting that, in the neural network, 00:42:22.120 --> 00:42:25.500 align:middle line:84% maybe you don't see this kind of quite as 00:42:25.500 --> 00:42:27.427 align:middle line:90% strongly as you do in human IT. 00:42:27.427 --> 00:42:28.760 align:middle line:90% So there's some differences too. 00:42:28.760 --> 00:42:33.020 align:middle line:84% They're not exactly the same, but there's some similarities. 00:42:33.020 --> 00:42:38.020 align:middle line:84% So this is kind of one way that you can make these model brain 00:42:38.020 --> 00:42:41.140 align:middle line:90% comparisons. 00:42:41.140 --> 00:42:44.980 align:middle line:84% Another very popular way to compare brains to models 00:42:44.980 --> 00:42:48.220 align:middle line:84% is to use the features in the model, 00:42:48.220 --> 00:42:50.350 align:middle line:84% as what's called an encoding model, OK? 00:42:50.350 --> 00:42:53.710 align:middle line:84% And so the way that this works is 00:42:53.710 --> 00:42:57.270 align:middle line:84% that you try to model each neuron or each unit 00:42:57.270 --> 00:43:04.710 align:middle line:84% of measurement in a brain as a weighted sum of unit responses 00:43:04.710 --> 00:43:07.070 align:middle line:90% in some stage of your model, OK? 00:43:07.070 --> 00:43:11.723 align:middle line:84% So here's just a schematic to illustrate this. 00:43:11.723 --> 00:43:13.390 align:middle line:84% So imagine you do the kind of experiment 00:43:13.390 --> 00:43:14.890 align:middle line:84% that we've been talking about, where 00:43:14.890 --> 00:43:18.750 align:middle line:84% there's a big set of images that gets presented to a person. 00:43:18.750 --> 00:43:20.770 align:middle line:84% And you're measuring their brain responses, OK? 00:43:20.770 --> 00:43:22.970 align:middle line:90% So let's say with fMRI, OK? 00:43:22.970 --> 00:43:25.710 align:middle line:90% So here's a voxel. 00:43:25.710 --> 00:43:29.070 align:middle line:84% And the color here is supposed to represent the response 00:43:29.070 --> 00:43:31.510 align:middle line:90% of the voxel to each image. 00:43:31.510 --> 00:43:34.810 align:middle line:84% So every voxel gives you a vector, 00:43:34.810 --> 00:43:37.110 align:middle line:84% which is the response to each image, OK? 00:43:37.110 --> 00:43:39.910 align:middle line:90% So here's our neural network. 00:43:39.910 --> 00:43:42.550 align:middle line:90% Consists of a bunch of stages. 00:43:42.550 --> 00:43:45.990 align:middle line:84% You can think of each stage of the model 00:43:45.990 --> 00:43:48.510 align:middle line:84% as consisting of a bunch of units, all right? 00:43:48.510 --> 00:43:50.230 align:middle line:84% You could think of that as a filter 00:43:50.230 --> 00:43:53.110 align:middle line:84% at a particular point in the image, for instance. 00:43:53.110 --> 00:43:55.310 align:middle line:84% And so each one of those units will give you 00:43:55.310 --> 00:43:58.615 align:middle line:84% a numerical response to any image that you present to it. 00:43:58.615 --> 00:43:59.990 align:middle line:84% So you can take all of the images 00:43:59.990 --> 00:44:02.090 align:middle line:84% that you used in your fMRI experiment. 00:44:02.090 --> 00:44:04.710 align:middle line:84% And you can present those to the model, all right? 00:44:04.710 --> 00:44:09.150 align:middle line:84% And you'll get some big complicated matrix of responses. 00:44:09.150 --> 00:44:13.390 align:middle line:84% All right, and so one way to ask whether the model is 00:44:13.390 --> 00:44:17.110 align:middle line:84% representing the same information that is represented 00:44:17.110 --> 00:44:20.830 align:middle line:84% by the brain is to try to predict the brain responses, 00:44:20.830 --> 00:44:22.630 align:middle line:90% using the model responses, OK? 00:44:22.630 --> 00:44:27.710 align:middle line:84% And the standard way to do this is to take the model responses 00:44:27.710 --> 00:44:31.150 align:middle line:90% and to fit a linear mapping-- 00:44:31.150 --> 00:44:34.790 align:middle line:84% here, that's f-- between the model responses and the brain 00:44:34.790 --> 00:44:35.910 align:middle line:90% responses, OK? 00:44:35.910 --> 00:44:40.910 align:middle line:84% So you take the model responses to some subset of images 00:44:40.910 --> 00:44:43.050 align:middle line:84% and you optimize this linear mapping. 00:44:43.050 --> 00:44:46.920 align:middle line:84% So it's a weighted combination of the units in the model that 00:44:46.920 --> 00:44:50.260 align:middle line:84% allow you to predict the responses to some subset, 00:44:50.260 --> 00:44:53.160 align:middle line:90% usually half of the images. 00:44:53.160 --> 00:44:58.300 align:middle line:84% And then what you do is you take that matrix, 00:44:58.300 --> 00:45:02.360 align:middle line:84% that linear mapping, and you evaluate the predictions 00:45:02.360 --> 00:45:05.000 align:middle line:90% on another set of images, OK? 00:45:05.000 --> 00:45:09.560 align:middle line:84% And so you ask, does the predicted response 00:45:09.560 --> 00:45:11.880 align:middle line:84% that you're getting from the model features actually 00:45:11.880 --> 00:45:14.740 align:middle line:84% match the measured response from that little bit of the brain, 00:45:14.740 --> 00:45:15.240 align:middle line:90% OK? 00:45:15.240 --> 00:45:19.280 align:middle line:84% And so you can think of what the linear mapping, 00:45:19.280 --> 00:45:23.100 align:middle line:84% it's kind of helping you align the coordinate systems, right? 00:45:23.100 --> 00:45:29.320 align:middle line:84% So it's telling you how to rotate and scale 00:45:29.320 --> 00:45:32.257 align:middle line:84% the model responses to get them to match up 00:45:32.257 --> 00:45:33.340 align:middle line:90% with the brain, all right? 00:45:33.340 --> 00:45:35.848 align:middle line:84% But that doesn't guarantee you that you'll be 00:45:35.848 --> 00:45:37.140 align:middle line:90% able to make a good prediction. 00:45:37.140 --> 00:45:38.807 align:middle line:84% So if the features that are in the model 00:45:38.807 --> 00:45:41.160 align:middle line:84% are kind of totally unrelated to the kinds of things 00:45:41.160 --> 00:45:43.120 align:middle line:84% the brain is measuring, the predictions 00:45:43.120 --> 00:45:44.530 align:middle line:90% might be terrible, all right? 00:45:44.530 --> 00:45:46.550 align:middle line:90% Sometimes they are. 00:45:46.550 --> 00:45:48.050 align:middle line:84% All right, so here's just an example 00:45:48.050 --> 00:45:50.470 align:middle line:90% of what would come out of this. 00:45:50.470 --> 00:45:55.650 align:middle line:84% So these are example IT neural responses 00:45:55.650 --> 00:45:57.790 align:middle line:90% to a big set of images. 00:45:57.790 --> 00:46:01.850 align:middle line:84% So every point here is the response to one image. 00:46:01.850 --> 00:46:06.070 align:middle line:84% And these shaded regions are kind of categories of images. 00:46:06.070 --> 00:46:08.810 align:middle line:90% 00:46:08.810 --> 00:46:14.250 align:middle line:84% So the black line is the actual measured response 00:46:14.250 --> 00:46:17.090 align:middle line:90% of a particular neuron in IT. 00:46:17.090 --> 00:46:22.330 align:middle line:84% And then the red line is the predicted response using 00:46:22.330 --> 00:46:25.410 align:middle line:84% the artificial neural networks features, OK? 00:46:25.410 --> 00:46:28.310 align:middle line:84% And you can see that there is not like a perfect match. 00:46:28.310 --> 00:46:31.810 align:middle line:84% But the prediction here is good, in the sense 00:46:31.810 --> 00:46:33.970 align:middle line:84% that it's kind of high, for the images 00:46:33.970 --> 00:46:38.250 align:middle line:84% in this particular region, which happened to be images of faces 00:46:38.250 --> 00:46:42.810 align:middle line:90% and low for everything else, OK? 00:46:42.810 --> 00:46:48.130 align:middle line:84% OK, so the outcome of this exercise 00:46:48.130 --> 00:46:51.950 align:middle line:84% is typically expressed as variance explained, OK? 00:46:51.950 --> 00:46:56.290 align:middle line:84% So there are some variants in the brain responds. 00:46:56.290 --> 00:46:58.770 align:middle line:84% And then you ask whether that variance is 00:46:58.770 --> 00:47:01.070 align:middle line:90% matched by the predictions, OK? 00:47:01.070 --> 00:47:02.510 align:middle line:90% And that gives you a number. 00:47:02.510 --> 00:47:05.850 align:middle line:84% And that's a quantitative metric of the extent 00:47:05.850 --> 00:47:09.250 align:middle line:84% to which the model is replicating 00:47:09.250 --> 00:47:13.530 align:middle line:90% the brain's representation, OK? 00:47:13.530 --> 00:47:15.370 align:middle line:84% All right, so here's one finding that kind 00:47:15.370 --> 00:47:20.730 align:middle line:90% comes out of this exercise. 00:47:20.730 --> 00:47:25.290 align:middle line:84% And that is that particular stages 00:47:25.290 --> 00:47:27.570 align:middle line:84% of artificial neural networks tend 00:47:27.570 --> 00:47:31.730 align:middle line:84% to be best at predicting particular stages 00:47:31.730 --> 00:47:33.770 align:middle line:90% of the visual system, OK? 00:47:33.770 --> 00:47:36.390 align:middle line:84% And so this is one early example of this. 00:47:36.390 --> 00:47:41.020 align:middle line:84% So the graph on the left is showing the predictions 00:47:41.020 --> 00:47:44.940 align:middle line:84% of V4 responses by different sets of models 00:47:44.940 --> 00:47:48.792 align:middle line:84% and different stages of this artificial neural network. 00:47:48.792 --> 00:47:51.000 align:middle line:84% And the graph on the right is showing the same thing, 00:47:51.000 --> 00:47:52.400 align:middle line:90% but for inferotemporal cortex. 00:47:52.400 --> 00:47:54.340 align:middle line:84% So remember, V4, we kind of think 00:47:54.340 --> 00:47:57.580 align:middle line:84% of as an earlier stage of the visual system than IT, OK? 00:47:57.580 --> 00:48:00.660 align:middle line:84% So the y-axis here is variance explained. 00:48:00.660 --> 00:48:04.060 align:middle line:84% So bigger numbers mean better predictions, OK? 00:48:04.060 --> 00:48:08.740 align:middle line:84% 0 is like about the worst that you could do. 00:48:08.740 --> 00:48:14.160 align:middle line:84% And the gray bars here are a bunch of old school models. 00:48:14.160 --> 00:48:19.520 align:middle line:84% So some of this is a model of V1 that's based on Gabor filters. 00:48:19.520 --> 00:48:20.340 align:middle line:90% Remember Gabor? 00:48:20.340 --> 00:48:24.060 align:middle line:84% It's one model of V1 simple cells. 00:48:24.060 --> 00:48:27.720 align:middle line:84% SIFT is like old school computer vision features. 00:48:27.720 --> 00:48:32.500 align:middle line:84% HMAX is a early model from this building from 20 years ago, 00:48:32.500 --> 00:48:33.680 align:middle line:90% so on and so forth, OK? 00:48:33.680 --> 00:48:38.860 align:middle line:84% And then the red bars, here, are the different layers 00:48:38.860 --> 00:48:40.380 align:middle line:84% of this artificial neural network 00:48:40.380 --> 00:48:44.260 align:middle line:84% that's been trained to recognize objects. 00:48:44.260 --> 00:48:46.460 align:middle line:84% And so what you're supposed to take away from this 00:48:46.460 --> 00:48:47.540 align:middle line:90% is two things. 00:48:47.540 --> 00:48:51.180 align:middle line:84% One is that, in the best cases, the red bars are pretty far 00:48:51.180 --> 00:48:52.740 align:middle line:90% above the gray bars, OK? 00:48:52.740 --> 00:48:55.180 align:middle line:84% So the neural networks are giving us better predictions 00:48:55.180 --> 00:48:56.820 align:middle line:90% of the brain responses. 00:48:56.820 --> 00:49:00.060 align:middle line:84% And in V4, the best predictions kind 00:49:00.060 --> 00:49:04.020 align:middle line:84% come from the middle layers here, layer 2 and layer 3, OK? 00:49:04.020 --> 00:49:09.880 align:middle line:84% If you look at IT, the red bars are still doing the best. 00:49:09.880 --> 00:49:12.780 align:middle line:84% But now, the red bar that produces the best predictions 00:49:12.780 --> 00:49:19.470 align:middle line:84% is actually the deeper layer, layer 4, OK? 00:49:19.470 --> 00:49:22.700 align:middle line:84% So it's consistent with this idea 00:49:22.700 --> 00:49:25.900 align:middle line:84% that the hierarchy that we believe 00:49:25.900 --> 00:49:29.020 align:middle line:84% exists in the visual ventral stream, where it consists 00:49:29.020 --> 00:49:33.220 align:middle line:84% of the sequence of regions, and as you go from one region 00:49:33.220 --> 00:49:35.320 align:middle line:84% to the next things change, right? 00:49:35.320 --> 00:49:37.000 align:middle line:90% The receptive fields get bigger. 00:49:37.000 --> 00:49:39.190 align:middle line:84% The responses is get more complicated, 00:49:39.190 --> 00:49:42.630 align:middle line:84% in sort of an intuitive sense, OK? 00:49:42.630 --> 00:49:45.135 align:middle line:84% Object information becomes more explicit. 00:49:45.135 --> 00:49:46.510 align:middle line:84% All right, so remember, we talked 00:49:46.510 --> 00:49:49.230 align:middle line:84% about how you can read out object information pretty well 00:49:49.230 --> 00:49:51.310 align:middle line:84% from IT less well from V4, right? 00:49:51.310 --> 00:49:52.530 align:middle line:90% So things change. 00:49:52.530 --> 00:49:55.270 align:middle line:84% We think that there's this series of transformations going 00:49:55.270 --> 00:49:57.950 align:middle line:84% from region to region that culminates in representations 00:49:57.950 --> 00:49:59.910 align:middle line:84% that can mediate object recognition. 00:49:59.910 --> 00:50:01.890 align:middle line:84% And in this artificial machine system, 00:50:01.890 --> 00:50:03.710 align:middle line:84% the artificial neural network that 00:50:03.710 --> 00:50:05.850 align:middle line:84% is also optimized to recognize objects, 00:50:05.850 --> 00:50:09.710 align:middle line:84% there's some evidence of an analogous hierarchy, right? 00:50:09.710 --> 00:50:11.870 align:middle line:84% In particular, the middle layers are giving you 00:50:11.870 --> 00:50:15.210 align:middle line:84% the best predictions of a middle layer of the ventral stream. 00:50:15.210 --> 00:50:17.990 align:middle line:84% And the deep layer is giving you the best predictions 00:50:17.990 --> 00:50:22.070 align:middle line:84% of the deep stage of the ventral stream, OK? 00:50:22.070 --> 00:50:23.590 align:middle line:90% Questions about that? 00:50:23.590 --> 00:50:24.710 align:middle line:90% Yeah. 00:50:24.710 --> 00:50:26.910 align:middle line:84% AUDIENCE: Any like current research 00:50:26.910 --> 00:50:30.430 align:middle line:84% being done on CNN's or artificial networks 00:50:30.430 --> 00:50:34.070 align:middle line:84% to model or like to perform object recognition, 00:50:34.070 --> 00:50:40.040 align:middle line:84% is like the focus more so on editing the actual models 00:50:40.040 --> 00:50:44.340 align:middle line:84% to make them more replicated of the brain 00:50:44.340 --> 00:50:48.687 align:middle line:84% or changing the data that we're giving them, at least? 00:50:48.687 --> 00:50:50.520 align:middle line:84% JOSH MCDERMOTT: Yeah, it's a great question. 00:50:50.520 --> 00:50:52.760 align:middle line:84% So the question is like if we want 00:50:52.760 --> 00:50:56.520 align:middle line:84% to make these models better, do we change the training data 00:50:56.520 --> 00:50:59.640 align:middle line:84% or do we introduce architectural constraints that would 00:50:59.640 --> 00:51:01.920 align:middle line:90% make them more brain-like. 00:51:01.920 --> 00:51:05.920 align:middle line:84% And I think there's a lot of interest in both. 00:51:05.920 --> 00:51:09.480 align:middle line:84% And I think it's an open question, 00:51:09.480 --> 00:51:13.040 align:middle line:84% as to what the relative importance of those things 00:51:13.040 --> 00:51:14.640 align:middle line:90% will be. 00:51:14.640 --> 00:51:18.720 align:middle line:84% There's one perspective that really-- 00:51:18.720 --> 00:51:22.640 align:middle line:84% like the task and the diet kind of predominantly 00:51:22.640 --> 00:51:27.000 align:middle line:84% determine the characteristics of the representations. 00:51:27.000 --> 00:51:31.520 align:middle line:84% There's another perspective that anatomical constraints really 00:51:31.520 --> 00:51:35.040 align:middle line:90% kind of matter a lot. 00:51:35.040 --> 00:51:37.000 align:middle line:84% So I mean, and the other thing I should say 00:51:37.000 --> 00:51:39.800 align:middle line:84% is that when we talk about training a neural network, 00:51:39.800 --> 00:51:40.300 align:middle line:90% right? 00:51:40.300 --> 00:51:44.200 align:middle line:84% So training often kind of means adapting the weights 00:51:44.200 --> 00:51:45.800 align:middle line:90% via gradient descent. 00:51:45.800 --> 00:51:49.230 align:middle line:84% But it can also mean optimizing the architecture. 00:51:49.230 --> 00:51:50.980 align:middle line:84% So any time you build one of these models, 00:51:50.980 --> 00:51:52.397 align:middle line:84% there's all these choices that you 00:51:52.397 --> 00:51:55.080 align:middle line:84% make about the number of stages, and like the operations that 00:51:55.080 --> 00:51:57.680 align:middle line:84% are in each stage, and the sequence of the operations, 00:51:57.680 --> 00:52:00.300 align:middle line:84% and how big the filters are, and how many there are. 00:52:00.300 --> 00:52:02.640 align:middle line:84% And there's a million choices that you make, right? 00:52:02.640 --> 00:52:05.400 align:middle line:90% And those choices can matter. 00:52:05.400 --> 00:52:08.160 align:middle line:84% And so that sometimes that can actually 00:52:08.160 --> 00:52:10.100 align:middle line:84% be the subject of an optimization process. 00:52:10.100 --> 00:52:12.800 align:middle line:84% So you search over like a big set of architectures 00:52:12.800 --> 00:52:16.280 align:middle line:84% to find something that kind of works well. 00:52:16.280 --> 00:52:21.980 align:middle line:84% And that may, also, be one way to potentially find 00:52:21.980 --> 00:52:23.980 align:middle line:84% architectures that are computationally promising 00:52:23.980 --> 00:52:26.640 align:middle line:84% and maybe that also end up kind of resembling the visual system 00:52:26.640 --> 00:52:28.280 align:middle line:90% in some abstract way, yeah. 00:52:28.280 --> 00:52:30.720 align:middle line:84% So these are kind of, I think, these are open questions 00:52:30.720 --> 00:52:33.770 align:middle line:84% that lots of people are interested in, right now, yeah. 00:52:33.770 --> 00:52:37.650 align:middle line:90% 00:52:37.650 --> 00:52:40.850 align:middle line:84% OK, so this is one kind of early piece of evidence 00:52:40.850 --> 00:52:45.170 align:middle line:84% that there is some tendency for artificial networks 00:52:45.170 --> 00:52:47.170 align:middle line:84% to replicate what we think of as the hierarchy 00:52:47.170 --> 00:52:50.450 align:middle line:90% in the ventral visual stream. 00:52:50.450 --> 00:52:54.770 align:middle line:84% And this is from recordings in monkey V4 and IT. 00:52:54.770 --> 00:52:56.530 align:middle line:84% This is another more recent piece 00:52:56.530 --> 00:53:00.450 align:middle line:84% of evidence using fMRI responses in humans. 00:53:00.450 --> 00:53:03.650 align:middle line:84% So this is a comparison between a visual convolutional neural 00:53:03.650 --> 00:53:05.830 align:middle line:84% network and human fMRI responses. 00:53:05.830 --> 00:53:12.830 align:middle line:84% And so this is a big flat map of the visual system in humans. 00:53:12.830 --> 00:53:21.450 align:middle line:84% So we've got V1, V2, V3, V4, the extrastriate body area, LO, 00:53:21.450 --> 00:53:25.810 align:middle line:84% the fusiform face area, parahippocampal place area, OK. 00:53:25.810 --> 00:53:27.350 align:middle line:90% It's the human visual system. 00:53:27.350 --> 00:53:33.140 align:middle line:84% And then the color denotes the stage of this neural network 00:53:33.140 --> 00:53:36.540 align:middle line:84% model that produced the best predictions of the voxels 00:53:36.540 --> 00:53:40.800 align:middle line:84% in a particular bit of the brain, OK? 00:53:40.800 --> 00:53:42.300 align:middle line:84% And these are different hemispheres, 00:53:42.300 --> 00:53:43.405 align:middle line:90% like left and right, OK? 00:53:43.405 --> 00:53:44.780 align:middle line:84% And so the take home message here 00:53:44.780 --> 00:53:48.340 align:middle line:84% is that the early visual areas, like V1 and V2, 00:53:48.340 --> 00:53:53.300 align:middle line:84% are best predicted by lower levels of the model. 00:53:53.300 --> 00:53:56.615 align:middle line:84% And you then kind of have places like V4, 00:53:56.615 --> 00:53:58.740 align:middle line:84% which we think of as intermediate visual areas that 00:53:58.740 --> 00:53:59.720 align:middle line:90% are kind of green. 00:53:59.720 --> 00:54:01.740 align:middle line:90% Those are in the middle. 00:54:01.740 --> 00:54:04.760 align:middle line:84% And then some other regions here that are red. 00:54:04.760 --> 00:54:06.860 align:middle line:84% So they're best predicted by kind of deeper stages 00:54:06.860 --> 00:54:08.360 align:middle line:90% of this model, OK? 00:54:08.360 --> 00:54:11.620 align:middle line:84% So again, another piece of evidence 00:54:11.620 --> 00:54:14.540 align:middle line:84% that the hierarchy that we see in the visual system 00:54:14.540 --> 00:54:17.140 align:middle line:84% can, in some cases, kind be mirrored 00:54:17.140 --> 00:54:20.242 align:middle line:84% by the differences in representations 00:54:20.242 --> 00:54:22.700 align:middle line:84% across stages of this artificial neural network that's been 00:54:22.700 --> 00:54:27.020 align:middle line:90% trained to recognize objects. 00:54:27.020 --> 00:54:30.420 align:middle line:84% So these findings are not unique to the visual system. 00:54:30.420 --> 00:54:33.740 align:middle line:84% So this is some work from my lab doing 00:54:33.740 --> 00:54:36.580 align:middle line:84% something kind of analogous in the auditory system. 00:54:36.580 --> 00:54:39.500 align:middle line:84% So this is a similar kind of comparison 00:54:39.500 --> 00:54:45.620 align:middle line:84% between fMRI responses in humans to a big set of natural sounds 00:54:45.620 --> 00:54:50.220 align:middle line:84% and deep neural networks that are trained to recognize words. 00:54:50.220 --> 00:54:54.180 align:middle line:84% So same kind of analysis, where the voxels here 00:54:54.180 --> 00:54:56.860 align:middle line:84% are color-coded based on the stage of the model 00:54:56.860 --> 00:54:58.240 align:middle line:90% that produces the best response. 00:54:58.240 --> 00:55:01.520 align:middle line:84% So pink is kind of an intermediate stage of the model. 00:55:01.520 --> 00:55:04.160 align:middle line:90% So here's early to late. 00:55:04.160 --> 00:55:05.620 align:middle line:84% So pink is kind of in the middle. 00:55:05.620 --> 00:55:08.460 align:middle line:90% And the cyan is deeper stages. 00:55:08.460 --> 00:55:10.460 align:middle line:84% And so you can see that there's this region here 00:55:10.460 --> 00:55:13.120 align:middle line:84% where you get the voxels colored pink. 00:55:13.120 --> 00:55:14.620 align:middle line:84% And that kind of loosely corresponds 00:55:14.620 --> 00:55:16.980 align:middle line:84% to what we think of as primary auditory cortex. 00:55:16.980 --> 00:55:19.220 align:middle line:84% So these black outlines, here, are 00:55:19.220 --> 00:55:24.020 align:middle line:84% three anatomical subdivisions of Heschl's gyrus, 00:55:24.020 --> 00:55:25.913 align:middle line:84% the site of primary auditory cortex. 00:55:25.913 --> 00:55:27.580 align:middle line:84% So things around primary auditory cortex 00:55:27.580 --> 00:55:28.470 align:middle line:90% are tweaks are pink. 00:55:28.470 --> 00:55:33.150 align:middle line:84% And then, beyond that, you see a lot of blue, OK? 00:55:33.150 --> 00:55:36.570 align:middle line:90% OK, any questions about that? 00:55:36.570 --> 00:55:44.345 align:middle line:90% 00:55:44.345 --> 00:55:47.968 align:middle line:84% OK, so this is, as I said, a very active area of research. 00:55:47.968 --> 00:55:49.510 align:middle line:84% Lots and lots of interest, right now, 00:55:49.510 --> 00:55:53.578 align:middle line:84% in the extent to which artificial neural networks will 00:55:53.578 --> 00:55:55.870 align:middle line:84% mirror the structure of sensory systems, and under what 00:55:55.870 --> 00:56:01.190 align:middle line:84% conditions, and how you can use them to do interesting things. 00:56:01.190 --> 00:56:03.830 align:middle line:84% So overall summary of what we talked about, about 00:56:03.830 --> 00:56:05.750 align:middle line:90% object recognition. 00:56:05.750 --> 00:56:07.230 align:middle line:84% So the ventral stream is believed 00:56:07.230 --> 00:56:09.230 align:middle line:90% to mediate object recognition. 00:56:09.230 --> 00:56:12.350 align:middle line:84% Overall, the complexity of the neuronal stimulus tuning 00:56:12.350 --> 00:56:15.790 align:middle line:90% seems to increase from V1 to IT. 00:56:15.790 --> 00:56:18.070 align:middle line:84% The key computational challenge of object recognition 00:56:18.070 --> 00:56:23.110 align:middle line:84% can be thought of as untangling object representations. 00:56:23.110 --> 00:56:25.790 align:middle line:84% We talked about evidence that basic object recognition 00:56:25.790 --> 00:56:26.950 align:middle line:90% is pretty quick. 00:56:26.950 --> 00:56:31.590 align:middle line:84% And we believe, likely, mostly feedforward. 00:56:31.590 --> 00:56:34.430 align:middle line:84% We discussed evidence that IT representations 00:56:34.430 --> 00:56:36.870 align:middle line:90% make object identity explicit. 00:56:36.870 --> 00:56:39.750 align:middle line:84% Where again, explicit means that they enable linear readout 00:56:39.750 --> 00:56:42.102 align:middle line:90% of the object identity. 00:56:42.102 --> 00:56:44.310 align:middle line:84% We discussed evidence that artificial neural networks 00:56:44.310 --> 00:56:47.830 align:middle line:84% replicate aspects of visual ventral stream processing. 00:56:47.830 --> 00:56:50.830 align:middle line:84% And we talked about evidence for object class-related brain 00:56:50.830 --> 00:56:52.350 align:middle line:90% specialization. 00:56:52.350 --> 00:56:56.590 align:middle line:84% The poster child being face recognition, OK? 00:56:56.590 --> 00:56:59.930 align:middle line:84% We just talked about object recognition. 00:56:59.930 --> 00:57:03.990 align:middle line:84% It's tempting to conclude that object recognition is kind 00:57:03.990 --> 00:57:05.710 align:middle line:90% of the point of vision, right? 00:57:05.710 --> 00:57:08.030 align:middle line:84% I showed you of simple schematic, 00:57:08.030 --> 00:57:10.008 align:middle line:84% where we got early vision and mid-level vision. 00:57:10.008 --> 00:57:12.050 align:middle line:84% And then it culminates in object recognition, OK? 00:57:12.050 --> 00:57:17.630 align:middle line:84% But the point of this lecture is that perception is not 00:57:17.630 --> 00:57:20.150 align:middle line:90% just about objects, all right? 00:57:20.150 --> 00:57:22.350 align:middle line:84% So on the left, we have an object. 00:57:22.350 --> 00:57:24.350 align:middle line:90% What is that object? 00:57:24.350 --> 00:57:25.200 align:middle line:90% AUDIENCE: A leaf. 00:57:25.200 --> 00:57:26.158 align:middle line:90% JOSH MCDERMOTT: A leaf. 00:57:26.158 --> 00:57:28.240 align:middle line:90% Yes, it's a leaf, all right? 00:57:28.240 --> 00:57:32.000 align:middle line:84% On the right, we have lots of those objects, all right? 00:57:32.000 --> 00:57:35.440 align:middle line:84% And when you look at that thing, if you look at it closely, 00:57:35.440 --> 00:57:38.300 align:middle line:84% you can probably pick out individual leaves, right? 00:57:38.300 --> 00:57:42.820 align:middle line:84% But at a glance, you can tell that that's leaves, all right? 00:57:42.820 --> 00:57:45.520 align:middle line:84% So there's some collective properties, 00:57:45.520 --> 00:57:48.640 align:middle line:84% when you have a lot of leaves that create 00:57:48.640 --> 00:57:50.840 align:middle line:90% what we call a texture, OK? 00:57:50.840 --> 00:57:56.600 align:middle line:84% So textures are everywhere in the visual world. 00:57:56.600 --> 00:57:58.060 align:middle line:90% They tell us about materials. 00:57:58.060 --> 00:57:59.680 align:middle line:84% So you can glance at these things 00:57:59.680 --> 00:58:04.160 align:middle line:84% and quickly tell that you're dealing with grass or gravel. 00:58:04.160 --> 00:58:06.860 align:middle line:84% We've already talked about how they're an important depth cue. 00:58:06.860 --> 00:58:09.600 align:middle line:84% So they give us information about shape. 00:58:09.600 --> 00:58:11.720 align:middle line:84% So just via the texture gradient here, 00:58:11.720 --> 00:58:15.400 align:middle line:84% you get a sense of the three-dimensionality here. 00:58:15.400 --> 00:58:18.425 align:middle line:84% Texture also helps us to segment scenes. 00:58:18.425 --> 00:58:20.800 align:middle line:84% So you can see how there are different image regions here 00:58:20.800 --> 00:58:22.968 align:middle line:84% that have different textural properties. 00:58:22.968 --> 00:58:25.010 align:middle line:84% You can distinguish the lawn from different parts 00:58:25.010 --> 00:58:26.490 align:middle line:90% of the house. 00:58:26.490 --> 00:58:29.130 align:middle line:84% The roof that's kind of covered in ivy. 00:58:29.130 --> 00:58:30.350 align:middle line:90% OK, all right. 00:58:30.350 --> 00:58:38.450 align:middle line:84% So texture is an important part of the visual world and vision. 00:58:38.450 --> 00:58:41.370 align:middle line:84% And humans are pretty good at detecting some texture 00:58:41.370 --> 00:58:42.130 align:middle line:90% boundaries, OK? 00:58:42.130 --> 00:58:44.490 align:middle line:84% So if you look at the image on the left, 00:58:44.490 --> 00:58:47.170 align:middle line:84% you can very clearly see this distinction between the left 00:58:47.170 --> 00:58:48.530 align:middle line:90% and the right side, OK? 00:58:48.530 --> 00:58:50.650 align:middle line:84% But some kinds of texture boundaries 00:58:50.650 --> 00:58:52.270 align:middle line:84% are a lot easier to see than others. 00:58:52.270 --> 00:58:54.610 align:middle line:84% So in the image on the right, it's 00:58:54.610 --> 00:58:59.758 align:middle line:84% harder to tell that there's one kind of thing on the left 00:58:59.758 --> 00:59:01.550 align:middle line:84% and another kind of thing on the right, OK? 00:59:01.550 --> 00:59:05.140 align:middle line:84% You have to inspect it a little bit more. 00:59:05.140 --> 00:59:07.450 align:middle line:84% And there's lots of examples of this. 00:59:07.450 --> 00:59:12.270 align:middle line:84% So all of these images contain texture boundaries. 00:59:12.270 --> 00:59:18.690 align:middle line:84% Some of them are very salient, and others, like this one, 00:59:18.690 --> 00:59:21.490 align:middle line:84% are really kind of hard to see, OK? 00:59:21.490 --> 00:59:25.410 align:middle line:90% 00:59:25.410 --> 00:59:30.850 align:middle line:84% OK, so one question that people have historically been 00:59:30.850 --> 00:59:33.585 align:middle line:84% interested in is what determines the salience 00:59:33.585 --> 00:59:34.710 align:middle line:90% of these boundaries, right? 00:59:34.710 --> 00:59:38.110 align:middle line:84% So here's just a bunch more examples where, in some cases, 00:59:38.110 --> 00:59:39.658 align:middle line:84% it's super obvious that you've got 00:59:39.658 --> 00:59:40.950 align:middle line:90% two different types of texture. 00:59:40.950 --> 00:59:43.650 align:middle line:84% In other cases, it's not so obvious. 00:59:43.650 --> 00:59:45.890 align:middle line:84% And so just informal observation kind of 00:59:45.890 --> 00:59:49.770 align:middle line:84% suggests that the texture differences that are salient 00:59:49.770 --> 00:59:51.770 align:middle line:84% involve differences in what we would 00:59:51.770 --> 00:59:54.230 align:middle line:84% think of as basic features-- things like color, shape, 00:59:54.230 --> 00:59:55.810 align:middle line:84% orientation, size-- that we often 00:59:55.810 --> 01:00:01.170 align:middle line:84% think of as being kind of detected in early vision. 01:00:01.170 --> 01:00:03.770 align:middle line:84% And so one kind of obvious question 01:00:03.770 --> 01:00:06.770 align:middle line:84% is whether we can predict the salience of these texture 01:00:06.770 --> 01:00:09.570 align:middle line:84% boundaries using the models of early vision 01:00:09.570 --> 01:00:13.570 align:middle line:84% that we've learned about and grown to love, right? 01:00:13.570 --> 01:00:16.170 align:middle line:90% So here, we have an image. 01:00:16.170 --> 01:00:22.500 align:middle line:84% It's convolved with two Gabor filters or approximately Gabor 01:00:22.500 --> 01:00:23.780 align:middle line:90% filters. 01:00:23.780 --> 01:00:24.880 align:middle line:90% And this is the output. 01:00:24.880 --> 01:00:27.980 align:middle line:84% So what you can see is that when you convolve it 01:00:27.980 --> 01:00:31.140 align:middle line:84% with a filter that's tuned to horizontal orientations, 01:00:31.140 --> 01:00:34.380 align:middle line:84% the regions that have more horizontal energy in the middle 01:00:34.380 --> 01:00:37.580 align:middle line:90% give you larger responses. 01:00:37.580 --> 01:00:39.540 align:middle line:84% The reasons that are more vertical 01:00:39.540 --> 01:00:40.840 align:middle line:90% give you smaller responses. 01:00:40.840 --> 01:00:45.380 align:middle line:84% And the vertical filter, it's the opposite, OK? 01:00:45.380 --> 01:00:47.860 align:middle line:84% All right, so how can we actually take 01:00:47.860 --> 01:00:50.460 align:middle line:84% these responses, and then and turn this 01:00:50.460 --> 01:00:51.940 align:middle line:84% into something quantitative that we 01:00:51.940 --> 01:00:56.020 align:middle line:84% can compare with our perception of boundaries? 01:00:56.020 --> 01:00:58.340 align:middle line:84% So here, we have one of those convolutions 01:00:58.340 --> 01:01:00.160 align:middle line:90% that we just looked at, right? 01:01:00.160 --> 01:01:02.780 align:middle line:84% And we're going to take a 1D slice through the convolution, 01:01:02.780 --> 01:01:04.420 align:middle line:90% and we plot it, all right? 01:01:04.420 --> 01:01:06.940 align:middle line:84% So this is the filter response as a function of space, 01:01:06.940 --> 01:01:07.500 align:middle line:90% all right? 01:01:07.500 --> 01:01:09.260 align:middle line:90% The y-axis is the response. 01:01:09.260 --> 01:01:10.980 align:middle line:90% The x-axis is space. 01:01:10.980 --> 01:01:13.380 align:middle line:84% And so you can see that this response wiggles around, 01:01:13.380 --> 01:01:14.980 align:middle line:90% from positive to negative. 01:01:14.980 --> 01:01:16.680 align:middle line:84% But then you get here, in the middle. 01:01:16.680 --> 01:01:20.733 align:middle line:84% And the amplitude of the wiggles is kind of smaller, OK? 01:01:20.733 --> 01:01:22.400 align:middle line:84% All right, so we can make that explicit. 01:01:22.400 --> 01:01:24.960 align:middle line:84% So this is like a simple cell response, more or less, right? 01:01:24.960 --> 01:01:27.983 align:middle line:84% So we can make that explicit by squaring the filter response. 01:01:27.983 --> 01:01:29.400 align:middle line:84% So that makes everything positive. 01:01:29.400 --> 01:01:35.380 align:middle line:84% So now, you've got responses here that get big 01:01:35.380 --> 01:01:36.560 align:middle line:90% and go down to 0. 01:01:36.560 --> 01:01:39.180 align:middle line:84% And then, here, they just don't get that big, OK? 01:01:39.180 --> 01:01:44.100 align:middle line:84% And then to summarize that, we can average over local regions 01:01:44.100 --> 01:01:45.760 align:middle line:84% with some kind of smoothing operation. 01:01:45.760 --> 01:01:48.560 align:middle line:84% And then we get this thing that's shown here, all right? 01:01:48.560 --> 01:01:51.820 align:middle line:84% So this, now, has a big response over here and a small response 01:01:51.820 --> 01:01:52.320 align:middle line:90% here. 01:01:52.320 --> 01:01:53.760 align:middle line:90% And then another big response. 01:01:53.760 --> 01:01:56.340 align:middle line:84% So sort of distinguishing between these regions 01:01:56.340 --> 01:01:57.220 align:middle line:90% that we see. 01:01:57.220 --> 01:01:59.400 align:middle line:84% All right, so what we just did here, 01:01:59.400 --> 01:02:01.860 align:middle line:84% which is squaring and averaging, gives us 01:02:01.860 --> 01:02:06.020 align:middle line:84% a quantity that is a lot like an energy measure or an envelope 01:02:06.020 --> 01:02:08.240 align:middle line:84% that we've talked about in other parts of the class. 01:02:08.240 --> 01:02:11.940 align:middle line:84% Remember, when we were talking about simple cells 01:02:11.940 --> 01:02:15.020 align:middle line:84% and complex cells, we talked about this fact 01:02:15.020 --> 01:02:16.870 align:middle line:84% that complex cell responses can be 01:02:16.870 --> 01:02:20.350 align:middle line:84% modeled by combining even and odd simple 01:02:20.350 --> 01:02:21.590 align:middle line:90% cell responses, right? 01:02:21.590 --> 01:02:24.762 align:middle line:84% So remember, you've got an even in an odd simple cell response. 01:02:24.762 --> 01:02:26.970 align:middle line:84% These are Gabor functions with just different phases. 01:02:26.970 --> 01:02:28.762 align:middle line:84% So then, you square them, and you add them, 01:02:28.762 --> 01:02:30.710 align:middle line:84% and you get this energy quantity. 01:02:30.710 --> 01:02:33.270 align:middle line:84% And so the energy quantity is always 01:02:33.270 --> 01:02:36.790 align:middle line:84% positive and smooths out these effects of phase 01:02:36.790 --> 01:02:41.290 align:middle line:84% that you see with the simple cell receptive fields, OK? 01:02:41.290 --> 01:02:44.910 align:middle line:84% And so this is one way of computing energy 01:02:44.910 --> 01:02:47.890 align:middle line:84% that is especially simple to think about. 01:02:47.890 --> 01:02:50.750 align:middle line:84% But you can get a similar measure by squaring and just 01:02:50.750 --> 01:02:52.010 align:middle line:90% averaging over space, right? 01:02:52.010 --> 01:02:53.990 align:middle line:84% So if you think of the sine and the cosine 01:02:53.990 --> 01:02:56.230 align:middle line:84% as having similar receptive fields, 01:02:56.230 --> 01:02:59.110 align:middle line:84% just at slightly different spatial phases-- 01:02:59.110 --> 01:03:01.330 align:middle line:84% if you took one of them and you averaged over space, 01:03:01.330 --> 01:03:02.790 align:middle line:84% you'd get something pretty similar. 01:03:02.790 --> 01:03:05.110 align:middle line:84% And so that's what we're looking at here. 01:03:05.110 --> 01:03:07.270 align:middle line:84% So this is an energy measure that 01:03:07.270 --> 01:03:11.390 align:middle line:84% is derived from a simple cell-like receptive field that's 01:03:11.390 --> 01:03:14.560 align:middle line:90% been squared and averaged, OK? 01:03:14.560 --> 01:03:16.920 align:middle line:84% And in some cases, these energy measures 01:03:16.920 --> 01:03:19.160 align:middle line:84% can predict texture boundaries pretty well. 01:03:19.160 --> 01:03:22.060 align:middle line:84% So we've got different texture here, in the middle. 01:03:22.060 --> 01:03:25.240 align:middle line:84% And that kind of shows up as a difference 01:03:25.240 --> 01:03:28.960 align:middle line:90% in the amount of energy, OK? 01:03:28.960 --> 01:03:31.480 align:middle line:84% All right, so the key idea that I 01:03:31.480 --> 01:03:33.680 align:middle line:84% want you to take away from this is 01:03:33.680 --> 01:03:38.040 align:middle line:84% that texture is defined by the average properties of an image 01:03:38.040 --> 01:03:38.660 align:middle line:90% region. 01:03:38.660 --> 01:03:40.720 align:middle line:84% OK, so in this case, the texture kind of 01:03:40.720 --> 01:03:43.220 align:middle line:84% is generated by all these elements, OK? 01:03:43.220 --> 01:03:47.800 align:middle line:84% But what kind characterizes the look of the texture-- or so 01:03:47.800 --> 01:03:49.640 align:middle line:84% that our hypothesis for what characterizes 01:03:49.640 --> 01:03:52.620 align:middle line:84% the look of the texture is its average properties, OK? 01:03:52.620 --> 01:03:55.600 align:middle line:84% And so this is one way to turn that 01:03:55.600 --> 01:04:01.320 align:middle line:84% into a quantifiable, measurable thing by taking some response, 01:04:01.320 --> 01:04:04.600 align:middle line:84% and then kind of locally averaging it over space, OK? 01:04:04.600 --> 01:04:07.960 align:middle line:90% 01:04:07.960 --> 01:04:10.540 align:middle line:84% All right, so this is a really simple idea. 01:04:10.540 --> 01:04:13.940 align:middle line:84% So what about seemingly more complicated textures? 01:04:13.940 --> 01:04:15.440 align:middle line:84% Can we account for the boundaries 01:04:15.440 --> 01:04:19.540 align:middle line:84% that we see in something like this with such a simple idea, 01:04:19.540 --> 01:04:20.040 align:middle line:90% OK? 01:04:20.040 --> 01:04:23.920 align:middle line:84% And the answer is that these very simple 01:04:23.920 --> 01:04:26.900 align:middle line:84% energy-based measures do a pretty good job, OK. 01:04:26.900 --> 01:04:28.800 align:middle line:84% So here's a bunch of examples where 01:04:28.800 --> 01:04:31.520 align:middle line:84% we've got an image on the left and the output 01:04:31.520 --> 01:04:35.000 align:middle line:84% of an energy-tuned measure on the right. 01:04:35.000 --> 01:04:37.600 align:middle line:84% And you get differences in the responses that 01:04:37.600 --> 01:04:42.172 align:middle line:84% co-vary with your subjective sense of the salience 01:04:42.172 --> 01:04:43.380 align:middle line:90% of the texture region, right? 01:04:43.380 --> 01:04:45.720 align:middle line:84% So this one is pretty obvious visually, 01:04:45.720 --> 01:04:48.040 align:middle line:84% and you get a pretty big difference, OK? 01:04:48.040 --> 01:04:50.900 align:middle line:84% Here's another one that is even more obvious. 01:04:50.900 --> 01:04:52.760 align:middle line:84% And you get a really big difference. 01:04:52.760 --> 01:04:55.032 align:middle line:84% Here's one where it's really actually 01:04:55.032 --> 01:04:57.240 align:middle line:84% kind of hard to see the difference between the center 01:04:57.240 --> 01:04:58.700 align:middle line:90% region and the surround region. 01:04:58.700 --> 01:05:00.660 align:middle line:84% So the center region has got these little Xs. 01:05:00.660 --> 01:05:01.900 align:middle line:90% The surround has got the Ls. 01:05:01.900 --> 01:05:04.932 align:middle line:84% But the sizes of them have been adjusted. 01:05:04.932 --> 01:05:06.140 align:middle line:90% And it's a lot harder to see. 01:05:06.140 --> 01:05:10.400 align:middle line:84% And lo and behold, the energy measure 01:05:10.400 --> 01:05:14.850 align:middle line:84% also doesn't give a very big difference in the response, OK? 01:05:14.850 --> 01:05:17.930 align:middle line:84% All right, so this is kind of one early development 01:05:17.930 --> 01:05:19.230 align:middle line:90% in the world of texture. 01:05:19.230 --> 01:05:22.610 align:middle line:84% So this is an example paper by Bergen and Adelson 01:05:22.610 --> 01:05:25.530 align:middle line:90% that espoused this sort of idea. 01:05:25.530 --> 01:05:32.530 align:middle line:84% OK, so energy measures can predict some texture boundaries. 01:05:32.530 --> 01:05:34.570 align:middle line:84% But are they a complete description 01:05:34.570 --> 01:05:39.210 align:middle line:84% of what we actually see when we look at this texture, all right? 01:05:39.210 --> 01:05:43.450 align:middle line:84% And I'm going to make the claim that if we have correctly 01:05:43.450 --> 01:05:47.490 align:middle line:84% identified the brain's representation of texture, 01:05:47.490 --> 01:05:51.770 align:middle line:84% then if we measure that representation for an example 01:05:51.770 --> 01:05:56.370 align:middle line:90% texture, like this, OK? 01:05:56.370 --> 01:05:59.850 align:middle line:84% And then we generate synthetic textures 01:05:59.850 --> 01:06:03.690 align:middle line:84% that produce the same values of that candidate representation, 01:06:03.690 --> 01:06:05.150 align:middle line:84% well, then the synthetic textures, 01:06:05.150 --> 01:06:08.730 align:middle line:84% they ought to look like the original example, OK? 01:06:08.730 --> 01:06:11.530 align:middle line:84% So that would be like the purest test of our theory 01:06:11.530 --> 01:06:12.597 align:middle line:90% of representation, OK? 01:06:12.597 --> 01:06:14.430 align:middle line:84% And so that's what we're going to try to do. 01:06:14.430 --> 01:06:17.130 align:middle line:84% And this is a pretty influential method 01:06:17.130 --> 01:06:22.810 align:middle line:84% for evaluating texture models known as texture synthesis, OK? 01:06:22.810 --> 01:06:28.270 align:middle line:84% All right, so the idea is that, for any given type of texture, 01:06:28.270 --> 01:06:29.810 align:middle line:84% There's like lots of example images 01:06:29.810 --> 01:06:31.030 align:middle line:90% that are the same texture. 01:06:31.030 --> 01:06:32.730 align:middle line:90% So if you just look down-- 01:06:32.730 --> 01:06:35.630 align:middle line:84% everybody look down-- all right, there's a carpet, right? 01:06:35.630 --> 01:06:38.610 align:middle line:84% And that carpet has a texture, OK? 01:06:38.610 --> 01:06:40.630 align:middle line:84% There's a microtexture and a macro texture. 01:06:40.630 --> 01:06:43.813 align:middle line:84% But you can imagine taking lots of images of that carpet. 01:06:43.813 --> 01:06:45.730 align:middle line:84% They would all be the same texture, all right? 01:06:45.730 --> 01:06:47.390 align:middle line:84% Different images, but same texture. 01:06:47.390 --> 01:06:50.282 align:middle line:90% They have the same properties. 01:06:50.282 --> 01:06:51.990 align:middle line:84% So we would call those equivalent, right? 01:06:51.990 --> 01:06:55.153 align:middle line:84% So in this case, those different images of the same texture, 01:06:55.153 --> 01:06:56.570 align:middle line:84% they would be generated by exactly 01:06:56.570 --> 01:06:57.945 align:middle line:84% the same physical process, right, 01:06:57.945 --> 01:07:02.836 align:middle line:84% which is like somebody sewing together the carpet, OK? 01:07:02.836 --> 01:07:05.495 align:middle line:84% Now we think that when you look at the textures 01:07:05.495 --> 01:07:07.870 align:middle line:84% and you have the sense that they're the same thing, well, 01:07:07.870 --> 01:07:09.480 align:middle line:84% your brain is doing something, right? 01:07:09.480 --> 01:07:11.480 align:middle line:84% There's some representation, in your brain, that 01:07:11.480 --> 01:07:13.420 align:middle line:84% is being produced that is the same for all 01:07:13.420 --> 01:07:14.900 align:middle line:90% of those different images, OK? 01:07:14.900 --> 01:07:17.780 align:middle line:84% All right, and so those brain representations 01:07:17.780 --> 01:07:21.420 align:middle line:84% will cause there to be a set of perceptually equivalent textures 01:07:21.420 --> 01:07:24.520 align:middle line:84% for any example image that we might come up with, OK? 01:07:24.520 --> 01:07:28.603 align:middle line:84% And that might not be exactly the same as the set of textures 01:07:28.603 --> 01:07:31.020 align:middle line:84% that is generated by the same physical process, all right? 01:07:31.020 --> 01:07:35.460 align:middle line:84% But these are things that would look the same to you, OK? 01:07:35.460 --> 01:07:39.340 align:middle line:84% OK, so the synthesis concept is that we're 01:07:39.340 --> 01:07:43.900 align:middle line:84% going to use image synthesis to test 01:07:43.900 --> 01:07:45.860 align:middle line:84% whether a representation captures 01:07:45.860 --> 01:07:47.832 align:middle line:90% texture perception, all right? 01:07:47.832 --> 01:07:49.540 align:middle line:84% And so the way that this is going to work 01:07:49.540 --> 01:07:54.580 align:middle line:84% is we'll have some example texture image, like this, right? 01:07:54.580 --> 01:07:57.220 align:middle line:84% It's just got a certain look to it, OK? 01:07:57.220 --> 01:07:59.620 align:middle line:84% We're going to start out with a noise image. 01:07:59.620 --> 01:08:01.780 align:middle line:84% And we're going to try to change that noise 01:08:01.780 --> 01:08:05.340 align:middle line:84% image to cause its representation, 01:08:05.340 --> 01:08:09.510 align:middle line:84% in our texture representation, to enter this green set, OK? 01:08:09.510 --> 01:08:13.690 align:middle line:84% And so here, here we've moved it, to this point. 01:08:13.690 --> 01:08:15.450 align:middle line:84% It's not all the way in the green set. 01:08:15.450 --> 01:08:18.130 align:middle line:84% You can see that actually doesn't look exactly like this. 01:08:18.130 --> 01:08:21.510 align:middle line:84% Although, it's maybe part of the way there, OK? 01:08:21.510 --> 01:08:22.229 align:middle line:90% All right. 01:08:22.229 --> 01:08:26.029 align:middle line:84% And the idea is that if we have the right representation 01:08:26.029 --> 01:08:29.470 align:middle line:84% of texture, right, that when we do this, when we take that noise 01:08:29.470 --> 01:08:32.029 align:middle line:84% image, and we cause it to have the same representation 01:08:32.029 --> 01:08:34.149 align:middle line:84% as some candidate image, they should 01:08:34.149 --> 01:08:37.649 align:middle line:84% look like the same type of thing, OK, in all respects. 01:08:37.649 --> 01:08:41.390 align:middle line:90% 01:08:41.390 --> 01:08:48.470 align:middle line:84% OK, so the first example that I'm going to give you of this 01:08:48.470 --> 01:08:52.189 align:middle line:84% was really the first attempt that people 01:08:52.189 --> 01:08:55.149 align:middle line:84% made at instantiating this general kind 01:08:55.149 --> 01:08:56.450 align:middle line:90% of scientific philosophy, OK? 01:08:56.450 --> 01:08:59.569 align:middle line:84% And they did it in a particular way that was simple and elegant. 01:08:59.569 --> 01:09:02.890 align:middle line:84% And so I'm going to walk you through how that worked, 01:09:02.890 --> 01:09:03.590 align:middle line:90% all right? 01:09:03.590 --> 01:09:08.270 align:middle line:84% And so the essential idea is that the representation 01:09:08.270 --> 01:09:11.910 align:middle line:84% of a texture consists of the distribution of filter 01:09:11.910 --> 01:09:13.270 align:middle line:90% responses, OK? 01:09:13.270 --> 01:09:17.270 align:middle line:84% So that our energy measure is the average squared filter 01:09:17.270 --> 01:09:18.550 align:middle line:90% response. 01:09:18.550 --> 01:09:19.970 align:middle line:84% So we measure the filter response. 01:09:19.970 --> 01:09:23.149 align:middle line:84% And then we square it and we average it, all right? 01:09:23.149 --> 01:09:25.810 align:middle line:84% And you can think of that as kind of matching one statistic. 01:09:25.810 --> 01:09:28.790 align:middle line:84% So the average of the square is more or less like the variance, 01:09:28.790 --> 01:09:31.770 align:middle line:90% if the thing has 0 mean, OK? 01:09:31.770 --> 01:09:33.510 align:middle line:84% So what we're going to try to do here 01:09:33.510 --> 01:09:35.750 align:middle line:84% is to match the entire distribution of the filter's 01:09:35.750 --> 01:09:36.590 align:middle line:90% response, OK? 01:09:36.590 --> 01:09:38.710 align:middle line:84% So this has the consequence of matching 01:09:38.710 --> 01:09:41.029 align:middle line:90% all possible statistics. 01:09:41.029 --> 01:09:44.390 align:middle line:84% And whereas, statistic is capturing some average tendency 01:09:44.390 --> 01:09:46.550 align:middle line:90% of a signal, OK? 01:09:46.550 --> 01:09:47.060 align:middle line:90% All right. 01:09:47.060 --> 01:09:48.810 align:middle line:84% So this may not make a whole lot of sense. 01:09:48.810 --> 01:09:52.109 align:middle line:84% So let's walk through an example, OK? 01:09:52.109 --> 01:09:55.910 align:middle line:84% All right, and so we're going to do this in a filter bank 01:09:55.910 --> 01:09:57.210 align:middle line:90% representation of an image. 01:09:57.210 --> 01:09:59.250 align:middle line:90% So let's review filter bank. 01:09:59.250 --> 01:10:01.310 align:middle line:90% So here's an image, OK? 01:10:01.310 --> 01:10:05.280 align:middle line:84% This is a set of filters, this row. 01:10:05.280 --> 01:10:07.940 align:middle line:90% They vary in spatial scale. 01:10:07.940 --> 01:10:10.560 align:middle line:84% So we have things over here that are 01:10:10.560 --> 01:10:13.400 align:middle line:84% small at high spatial frequencies and things over here 01:10:13.400 --> 01:10:15.820 align:middle line:84% that are bigger and at lower spatial frequencies. 01:10:15.820 --> 01:10:17.500 align:middle line:84% And they vary in orientation, all right? 01:10:17.500 --> 01:10:19.940 align:middle line:84% So we've got vertical, horizontal, 01:10:19.940 --> 01:10:21.220 align:middle line:90% and the two diagonals. 01:10:21.220 --> 01:10:22.840 align:middle line:90% All right. 01:10:22.840 --> 01:10:27.520 align:middle line:84% The next row plots the convolution of that image 01:10:27.520 --> 01:10:30.640 align:middle line:84% at the top with each one of these filters, OK? 01:10:30.640 --> 01:10:32.320 align:middle line:84% So remember, the convolution gives you 01:10:32.320 --> 01:10:34.940 align:middle line:84% a filter response at every point in the original image. 01:10:34.940 --> 01:10:38.360 align:middle line:84% So you can display it like an image. 01:10:38.360 --> 01:10:39.220 align:middle line:90% And what do you see? 01:10:39.220 --> 01:10:43.400 align:middle line:84% Well, the low spatial frequency filters, their response 01:10:43.400 --> 01:10:45.020 align:middle line:84% varies kind of slowly over space. 01:10:45.020 --> 01:10:47.240 align:middle line:84% They're picking out low frequency structure 01:10:47.240 --> 01:10:48.760 align:middle line:90% at different orientations. 01:10:48.760 --> 01:10:50.960 align:middle line:84% So the horizontal filter responds a lot, whenever 01:10:50.960 --> 01:10:52.847 align:middle line:90% there's horizontal edges. 01:10:52.847 --> 01:10:54.680 align:middle line:84% The vertical filter responds a lot, whenever 01:10:54.680 --> 01:10:56.520 align:middle line:90% there's vertical edges, OK? 01:10:56.520 --> 01:10:58.560 align:middle line:84% And then the high spatial frequency filters 01:10:58.560 --> 01:11:02.030 align:middle line:84% do the same thing, but at a higher spatial frequency scale, 01:11:02.030 --> 01:11:04.730 align:middle line:90% OK? 01:11:04.730 --> 01:11:09.490 align:middle line:84% All right, so the technical term for each one of these things 01:11:09.490 --> 01:11:11.210 align:middle line:90% is a subband, OK? 01:11:11.210 --> 01:11:15.490 align:middle line:84% So that's what people refer to the output of a bandpass filter 01:11:15.490 --> 01:11:16.950 align:middle line:90% as in the engineering world. 01:11:16.950 --> 01:11:19.210 align:middle line:84% So but for us, it's just the convolution 01:11:19.210 --> 01:11:20.550 align:middle line:90% of the image with the filters. 01:11:20.550 --> 01:11:22.728 align:middle line:84% And from the prism of neuroscience, 01:11:22.728 --> 01:11:24.770 align:middle line:84% you can think of each one of these images as kind 01:11:24.770 --> 01:11:28.550 align:middle line:84% of a simulation of a population of neural responses, all right? 01:11:28.550 --> 01:11:31.410 align:middle line:84% So it's like all of the neurons in the visual cortex that 01:11:31.410 --> 01:11:33.950 align:middle line:84% have receptive fields that look like that, where 01:11:33.950 --> 01:11:35.450 align:middle line:84% every neuron has the receptive field 01:11:35.450 --> 01:11:37.610 align:middle line:84% at a different spatial position, OK? 01:11:37.610 --> 01:11:40.610 align:middle line:90% 01:11:40.610 --> 01:11:42.570 align:middle line:84% All right, so the model of texture perception 01:11:42.570 --> 01:11:45.730 align:middle line:84% that we're going to test here is one 01:11:45.730 --> 01:11:47.610 align:middle line:84% in which textures are represented 01:11:47.610 --> 01:11:51.010 align:middle line:84% with the distribution of activity in these subbands. 01:11:51.010 --> 01:11:54.130 align:middle line:84% Again, simulating the response of oriented spatial 01:11:54.130 --> 01:11:56.365 align:middle line:90% frequency-tuned neurons. 01:11:56.365 --> 01:11:58.490 align:middle line:84% All right, so this image that we're looking at here 01:11:58.490 --> 01:11:59.910 align:middle line:90% is not a texture. 01:11:59.910 --> 01:12:01.690 align:middle line:90% It's like just something else. 01:12:01.690 --> 01:12:03.290 align:middle line:90% Here, we have a texture. 01:12:03.290 --> 01:12:06.430 align:middle line:84% And this is the subband representation of the texture. 01:12:06.430 --> 01:12:09.417 align:middle line:84% So this is like the exactly the same thing that we saw here. 01:12:09.417 --> 01:12:11.250 align:middle line:84% So you're taking the image and convolving it 01:12:11.250 --> 01:12:13.130 align:middle line:90% with all of the filters-- 01:12:13.130 --> 01:12:22.730 align:middle line:84% with one wrinkle, which is that when the filters are coarser, 01:12:22.730 --> 01:12:24.710 align:middle line:84% you can downsample the image, OK? 01:12:24.710 --> 01:12:28.250 align:middle line:84% And so one kind of common type of representation 01:12:28.250 --> 01:12:31.650 align:middle line:84% in image processing is called a pyramid representation, where 01:12:31.650 --> 01:12:34.210 align:middle line:84% the low spatial frequencies get downsampled, 01:12:34.210 --> 01:12:36.930 align:middle line:84% so that they can be represented with fewer numbers, OK? 01:12:36.930 --> 01:12:40.360 align:middle line:84% So here, we have the high spatial frequencies. 01:12:40.360 --> 01:12:41.110 align:middle line:90% Here's the middle. 01:12:41.110 --> 01:12:42.070 align:middle line:90% Here's the lower. 01:12:42.070 --> 01:12:45.370 align:middle line:84% But they've been downsampled, and so the actual image 01:12:45.370 --> 01:12:47.190 align:middle line:84% representation of this is smaller. 01:12:47.190 --> 01:12:49.010 align:middle line:84% It's got the same information content. 01:12:49.010 --> 01:12:51.290 align:middle line:84% And so it's called a pyramid representation 01:12:51.290 --> 01:12:55.090 align:middle line:84% because you have these subbands at the base that are big, 01:12:55.090 --> 01:12:55.710 align:middle line:90% all right? 01:12:55.710 --> 01:12:58.010 align:middle line:84% And then it progresses to subbands 01:12:58.010 --> 01:13:00.080 align:middle line:84% that can be represented as smaller things. 01:13:00.080 --> 01:13:04.820 align:middle line:90% OK, so pyramid representation. 01:13:04.820 --> 01:13:08.740 align:middle line:84% Now what we're going to do is take each of these images 01:13:08.740 --> 01:13:10.940 align:middle line:84% and turn it into a histogram, OK? 01:13:10.940 --> 01:13:15.480 align:middle line:84% So each one of these things is an image. 01:13:15.480 --> 01:13:17.440 align:middle line:84% So at every point, you get a number, 01:13:17.440 --> 01:13:20.100 align:middle line:84% which is the response of the filter, all right? 01:13:20.100 --> 01:13:22.793 align:middle line:84% And this is just a histogram of all of those numbers. 01:13:22.793 --> 01:13:24.460 align:middle line:84% It's like you throw them all into a bag, 01:13:24.460 --> 01:13:26.210 align:middle line:84% and you just measure the distribution, OK? 01:13:26.210 --> 01:13:30.100 align:middle line:90% So that's the histogram. 01:13:30.100 --> 01:13:33.360 align:middle line:84% This here is a histogram of the pixel intensities, all right? 01:13:33.360 --> 01:13:35.065 align:middle line:90% So what does this show? 01:13:35.065 --> 01:13:36.940 align:middle line:84% Well, obviously, when you look at this image, 01:13:36.940 --> 01:13:38.077 align:middle line:90% it's mostly white. 01:13:38.077 --> 01:13:40.160 align:middle line:84% And then, there's some little bits that are black. 01:13:40.160 --> 01:13:43.180 align:middle line:84% So the histogram here has a small little peak 01:13:43.180 --> 01:13:44.080 align:middle line:90% down here at 0. 01:13:44.080 --> 01:13:45.600 align:middle line:84% That's all the bits that are black. 01:13:45.600 --> 01:13:49.860 align:middle line:84% And then a big peak up here at 255 or 1-- 01:13:49.860 --> 01:13:52.120 align:middle line:84% I forget what the scale is that they're using. 01:13:52.120 --> 01:13:54.080 align:middle line:84% So that's all the white pixels, OK? 01:13:54.080 --> 01:13:58.380 align:middle line:84% So this here is the histogram for one 01:13:58.380 --> 01:14:00.140 align:middle line:90% of these particular subbands. 01:14:00.140 --> 01:14:02.940 align:middle line:84% And so you can see that this has a big peak at 0. 01:14:02.940 --> 01:14:04.280 align:middle line:90% So that's what gray is, right? 01:14:04.280 --> 01:14:06.180 align:middle line:84% So there's all these places where the filter is not 01:14:06.180 --> 01:14:07.180 align:middle line:90% giving you any response. 01:14:07.180 --> 01:14:10.580 align:middle line:84% And that's because the image is uniform, OK? 01:14:10.580 --> 01:14:13.580 align:middle line:84% And then, these places, where the response is either 01:14:13.580 --> 01:14:16.380 align:middle line:90% lower or higher than 0. 01:14:16.380 --> 01:14:18.700 align:middle line:84% All right, so that's what you get. 01:14:18.700 --> 01:14:21.420 align:middle line:84% And so the proposal is that the representation of texture 01:14:21.420 --> 01:14:24.300 align:middle line:84% is going to be a whole bunch of these histograms, one 01:14:24.300 --> 01:14:27.100 align:middle line:90% for each one of these filters. 01:14:27.100 --> 01:14:29.673 align:middle line:84% OK, so what does this mean to actually have 01:14:29.673 --> 01:14:30.840 align:middle line:90% this as your representation. 01:14:30.840 --> 01:14:33.980 align:middle line:84% Well, OK, the histogram is throwing out all information 01:14:33.980 --> 01:14:35.600 align:middle line:90% about space, all right? 01:14:35.600 --> 01:14:37.840 align:middle line:84% So you're taking all of these values. 01:14:37.840 --> 01:14:40.260 align:middle line:84% You put them in a bag, and then you just 01:14:40.260 --> 01:14:43.000 align:middle line:84% measure how frequently you get one value or another, OK? 01:14:43.000 --> 01:14:46.540 align:middle line:84% So that's a pretty big change from this 01:14:46.540 --> 01:14:47.720 align:middle line:90% as a representation, right? 01:14:47.720 --> 01:14:50.580 align:middle line:90% This has spatial structure, OK? 01:14:50.580 --> 01:14:53.423 align:middle line:84% OK, so instead what we're just what this is capturing 01:14:53.423 --> 01:14:55.590 align:middle line:84% is kind of the average properties of this thing when 01:14:55.590 --> 01:14:59.374 align:middle line:90% you pull it over space. 01:14:59.374 --> 01:15:01.750 align:middle line:84% All right, so this will capture the energy. 01:15:01.750 --> 01:15:03.950 align:middle line:84% So the energy would be captured by the variance 01:15:03.950 --> 01:15:05.890 align:middle line:90% of that distribution. 01:15:05.890 --> 01:15:08.470 align:middle line:84% But it'll also capture all kinds of higher order statistics, 01:15:08.470 --> 01:15:10.930 align:middle line:90% which might be important. 01:15:10.930 --> 01:15:12.670 align:middle line:84% OK, so now, what we're going to try 01:15:12.670 --> 01:15:16.790 align:middle line:84% to do to test this theory of texture representation 01:15:16.790 --> 01:15:19.270 align:middle line:84% is to measure these histograms and some example images, 01:15:19.270 --> 01:15:20.510 align:middle line:90% like that one there. 01:15:20.510 --> 01:15:23.150 align:middle line:84% And now synthesize an image that we're 01:15:23.150 --> 01:15:26.230 align:middle line:84% going to force to match all of the histogram, all 01:15:26.230 --> 01:15:29.150 align:middle line:84% of the subband histograms, OK, the histograms of all 01:15:29.150 --> 01:15:34.085 align:middle line:90% of these filter responses, OK? 01:15:34.085 --> 01:15:36.230 align:middle line:84% So we're going to start with noise 01:15:36.230 --> 01:15:37.890 align:middle line:84% and equate the histograms, all right? 01:15:37.890 --> 01:15:40.050 align:middle line:84% So you'll start with some kind of distribution. 01:15:40.050 --> 01:15:42.190 align:middle line:84% And then, you're going to mess with the image 01:15:42.190 --> 01:15:44.310 align:middle line:90% to change that distribution. 01:15:44.310 --> 01:15:47.150 align:middle line:84% So it turns out that this process of histogram 01:15:47.150 --> 01:15:49.430 align:middle line:90% equalization is super simple. 01:15:49.430 --> 01:15:51.070 align:middle line:84% So it's a kind of classic trick that 01:15:51.070 --> 01:15:54.240 align:middle line:90% is used in image processing. 01:15:54.240 --> 01:15:57.800 align:middle line:84% So the way that this works is you 01:15:57.800 --> 01:16:02.440 align:middle line:84% have the image that you want to match to. 01:16:02.440 --> 01:16:03.565 align:middle line:90% You generate its histogram. 01:16:03.565 --> 01:16:05.690 align:middle line:84% So this is now just done with the pixel intensities 01:16:05.690 --> 01:16:07.120 align:middle line:90% to convey the idea, all right? 01:16:07.120 --> 01:16:09.703 align:middle line:84% So remember, there's a little bit of black and a lot of white. 01:16:09.703 --> 01:16:11.600 align:middle line:84% You generate the cumulative histogram, OK? 01:16:11.600 --> 01:16:13.960 align:middle line:84% So that's just kind of integrating this thing 01:16:13.960 --> 01:16:16.963 align:middle line:90% over the intensity axis. 01:16:16.963 --> 01:16:18.880 align:middle line:84% And this is the same thing for the noise image 01:16:18.880 --> 01:16:19.700 align:middle line:90% that we start with. 01:16:19.700 --> 01:16:20.742 align:middle line:90% So this is a noise image. 01:16:20.742 --> 01:16:23.460 align:middle line:84% And so it's got a pretty uniform distribution. 01:16:23.460 --> 01:16:27.760 align:middle line:84% And so its cumulative histogram looks like this, OK? 01:16:27.760 --> 01:16:30.640 align:middle line:84% And so, now, all that we're going to do 01:16:30.640 --> 01:16:33.440 align:middle line:84% is generate a lookup table, all right, 01:16:33.440 --> 01:16:38.480 align:middle line:84% that basically says that you take a pixel here. 01:16:38.480 --> 01:16:41.000 align:middle line:90% And you find what its value is. 01:16:41.000 --> 01:16:45.240 align:middle line:84% And then you just map that onto the cumulative histogram 01:16:45.240 --> 01:16:47.980 align:middle line:84% that you get for the other image, OK? 01:16:47.980 --> 01:16:49.840 align:middle line:84% And so, in this particular case, something 01:16:49.840 --> 01:16:55.040 align:middle line:84% that has a value of 0.8 gets turned into having a value of 1, 01:16:55.040 --> 01:16:56.440 align:middle line:90% all right? 01:16:56.440 --> 01:17:00.280 align:middle line:84% So this image, here, is that noise image 01:17:00.280 --> 01:17:02.760 align:middle line:84% that's now been histogram-equalized 01:17:02.760 --> 01:17:07.600 align:middle line:90% to that texture, all right? 01:17:07.600 --> 01:17:09.400 align:middle line:84% But just in the pixel domain, OK? 01:17:09.400 --> 01:17:13.145 align:middle line:84% So now we end up with a noise image that's mostly white 01:17:13.145 --> 01:17:15.520 align:middle line:84% and a little bit of black, all right, but otherwise quite 01:17:15.520 --> 01:17:16.220 align:middle line:90% random. 01:17:16.220 --> 01:17:19.047 align:middle line:90% 01:17:19.047 --> 01:17:21.380 align:middle line:84% All right, now, we're going to try to do the same thing, 01:17:21.380 --> 01:17:24.360 align:middle line:84% but now in the domain of the subbands, the filter responses, 01:17:24.360 --> 01:17:24.860 align:middle line:90% OK? 01:17:24.860 --> 01:17:31.040 align:middle line:84% So here, we have the filter responses of our texture 01:17:31.040 --> 01:17:32.300 align:middle line:90% that we're interested in. 01:17:32.300 --> 01:17:35.120 align:middle line:84% And now, we have the filter responses of our noise 01:17:35.120 --> 01:17:37.440 align:middle line:84% that's just been histogram-equalized 01:17:37.440 --> 01:17:39.920 align:middle line:84% in the pixel domain to the texture, all right? 01:17:39.920 --> 01:17:41.820 align:middle line:84% And so those look very different. 01:17:41.820 --> 01:17:43.400 align:middle line:84% And lo and behold, the histograms 01:17:43.400 --> 01:17:45.220 align:middle line:90% look totally different, OK? 01:17:45.220 --> 01:17:47.820 align:middle line:84% So this one is very peaked at 0 and has these long tails. 01:17:47.820 --> 01:17:49.760 align:middle line:84% This looks a lot more Gaussian, OK? 01:17:49.760 --> 01:17:52.730 align:middle line:84% So we're going to do the exact same thing, 01:17:52.730 --> 01:17:55.950 align:middle line:84% but on these images, all right, these subbands. 01:17:55.950 --> 01:17:58.570 align:middle line:84% So we'll equate their histograms. 01:17:58.570 --> 01:18:02.770 align:middle line:84% And so here we have the new subband 01:18:02.770 --> 01:18:05.970 align:middle line:84% that has now been histogram-equalized 01:18:05.970 --> 01:18:07.730 align:middle line:90% to the original texture. 01:18:07.730 --> 01:18:10.970 align:middle line:84% All right, they've got the same distribution. 01:18:10.970 --> 01:18:13.970 align:middle line:84% So we're just going to do this for every subband, OK? 01:18:13.970 --> 01:18:16.530 align:middle line:90% 01:18:16.530 --> 01:18:19.330 align:middle line:84% This actually kind of has to be done iteratively. 01:18:19.330 --> 01:18:20.263 align:middle line:90% But it's super simple. 01:18:20.263 --> 01:18:21.430 align:middle line:90% So you just take your image. 01:18:21.430 --> 01:18:22.630 align:middle line:90% You split it up into sub bands. 01:18:22.630 --> 01:18:24.380 align:middle line:84% Each one of those gets histogram-equalized 01:18:24.380 --> 01:18:25.233 align:middle line:90% to the original. 01:18:25.233 --> 01:18:27.650 align:middle line:84% And then you can add them back up to get an original image 01:18:27.650 --> 01:18:28.750 align:middle line:90% and do this iteratively. 01:18:28.750 --> 01:18:33.210 align:middle line:84% And so what you can see is the red and the blue plots here. 01:18:33.210 --> 01:18:34.710 align:middle line:84% One of them is the target histogram. 01:18:34.710 --> 01:18:36.930 align:middle line:84% The other is the histogram of the synthetic image. 01:18:36.930 --> 01:18:39.130 align:middle line:84% Each one of these is for a different subband, so 01:18:39.130 --> 01:18:40.440 align:middle line:90% a different filter. 01:18:40.440 --> 01:18:42.690 align:middle line:84% All right, so then we're going to add these things up, 01:18:42.690 --> 01:18:44.530 align:middle line:84% and we'll get an image, all right? 01:18:44.530 --> 01:18:47.610 align:middle line:84% And so the key question here is whether these things look 01:18:47.610 --> 01:18:48.730 align:middle line:90% like the original, right? 01:18:48.730 --> 01:18:52.130 align:middle line:84% So remember, the idea is that if this candidate texture 01:18:52.130 --> 01:18:55.890 align:middle line:84% representation captures what is in our brain's representation 01:18:55.890 --> 01:18:58.510 align:middle line:84% of texture, then the synthetic examples, 01:18:58.510 --> 01:19:00.570 align:middle line:84% they should look like the original, OK? 01:19:00.570 --> 01:19:04.770 align:middle line:84% And when we come back on Tuesday, 01:19:04.770 --> 01:19:06.810 align:middle line:90% I will tell you whether they do. 01:19:06.810 --> 01:19:07.410 align:middle line:90% [LAUGHTER] 01:19:07.410 --> 01:19:08.960 align:middle line:90% OK. 01:19:08.960 --> 01:19:48.000 align:middle line:90%