WEBVTT 00:00:00.000 --> 00:00:01.934 align:middle line:90% [SQUEAKING] 00:00:01.934 --> 00:00:03.368 align:middle line:90% [RUSTLING] 00:00:03.368 --> 00:00:08.863 align:middle line:90% [CLICKING] 00:00:08.863 --> 00:00:10.280 align:middle line:84% JOSH MCDERMOTT: Today, we're going 00:00:10.280 --> 00:00:13.760 align:middle line:84% to talk about texture perception, which we started 00:00:13.760 --> 00:00:15.000 align:middle line:90% talking about last time. 00:00:15.000 --> 00:00:20.160 align:middle line:84% So remember, this distinction between things, 00:00:20.160 --> 00:00:22.980 align:middle line:90% like objects and stuff. 00:00:22.980 --> 00:00:24.360 align:middle line:90% So objects are things. 00:00:24.360 --> 00:00:25.740 align:middle line:90% They're kind of individuated. 00:00:25.740 --> 00:00:27.240 align:middle line:84% We often think of the goal of vision 00:00:27.240 --> 00:00:28.500 align:middle line:90% as being object recognition. 00:00:28.500 --> 00:00:30.720 align:middle line:84% But the world also has a lot of stuff. 00:00:30.720 --> 00:00:33.060 align:middle line:84% When you have lots of objects all together, 00:00:33.060 --> 00:00:34.540 align:middle line:84% they create what we call a texture, 00:00:34.540 --> 00:00:35.915 align:middle line:84% and it's got a certain appearance 00:00:35.915 --> 00:00:38.360 align:middle line:90% and you can discriminate it. 00:00:38.360 --> 00:00:40.360 align:middle line:84% And we use it for all kinds of different things. 00:00:40.360 --> 00:00:44.680 align:middle line:84% You can use texture to tell about material, about shape, 00:00:44.680 --> 00:00:47.183 align:middle line:90% to segment scenes. 00:00:47.183 --> 00:00:48.600 align:middle line:84% And we're pretty good at detecting 00:00:48.600 --> 00:00:50.180 align:middle line:90% boundaries between textures. 00:00:50.180 --> 00:00:52.760 align:middle line:84% And what we initially talked about last time was the fact 00:00:52.760 --> 00:00:55.940 align:middle line:84% that when you look at these texture boundaries, 00:00:55.940 --> 00:00:58.880 align:middle line:84% in some cases, they're really easy to see, and in other cases, 00:00:58.880 --> 00:01:01.450 align:middle line:84% they're are very subtle and nonobvious 00:01:01.450 --> 00:01:02.830 align:middle line:90% and don't really pop out at you. 00:01:02.830 --> 00:01:04.989 align:middle line:84% So we saw lots of examples of this. 00:01:04.989 --> 00:01:06.722 align:middle line:84% So each of these things has got kind 00:01:06.722 --> 00:01:08.930 align:middle line:84% of a region in the middle that is a different texture 00:01:08.930 --> 00:01:09.830 align:middle line:90% than the surround. 00:01:09.830 --> 00:01:12.890 align:middle line:84% In some cases, it's obvious, in some cases, it's not. 00:01:12.890 --> 00:01:15.110 align:middle line:84% Each of these has a texture boundary. 00:01:15.110 --> 00:01:18.610 align:middle line:84% Some of them are obvious, some of them are not. 00:01:18.610 --> 00:01:21.170 align:middle line:84% And we spent some time talking about 00:01:21.170 --> 00:01:24.650 align:middle line:84% whether we can account for the salience of texture boundaries 00:01:24.650 --> 00:01:26.627 align:middle line:84% using pretty standard models of early vision, 00:01:26.627 --> 00:01:28.210 align:middle line:84% like the stuff that we kind of learned 00:01:28.210 --> 00:01:29.590 align:middle line:90% about earlier in the class. 00:01:29.590 --> 00:01:31.733 align:middle line:84% So this is an example of a texture image. 00:01:31.733 --> 00:01:33.150 align:middle line:84% So there's a region in the middle. 00:01:33.150 --> 00:01:35.330 align:middle line:84% It's a different texture than the rest. 00:01:35.330 --> 00:01:39.170 align:middle line:84% You convolve it with these two Gabor-like filters, stand-ins 00:01:39.170 --> 00:01:41.545 align:middle line:84% for filters in V1, this is the result of the convolution. 00:01:41.545 --> 00:01:43.837 align:middle line:84% Remember, the convolution means that the filter kind of 00:01:43.837 --> 00:01:46.570 align:middle line:84% gets placed down at every possible location in the image. 00:01:46.570 --> 00:01:50.770 align:middle line:84% And then we can view the convolution as an image. 00:01:50.770 --> 00:01:54.690 align:middle line:84% You can think of this as the response of a big population 00:01:54.690 --> 00:01:57.230 align:middle line:84% of neurons that all have the same type of receptive field, 00:01:57.230 --> 00:01:59.170 align:middle line:84% just positioned position at different places 00:01:59.170 --> 00:02:01.690 align:middle line:90% in the visual field. 00:02:01.690 --> 00:02:05.090 align:middle line:84% And so the filter responses are different in the center region 00:02:05.090 --> 00:02:06.770 align:middle line:90% than in the surround. 00:02:06.770 --> 00:02:08.370 align:middle line:84% And we talked about how you might 00:02:08.370 --> 00:02:10.770 align:middle line:84% be able to make that more explicit 00:02:10.770 --> 00:02:12.410 align:middle line:84% by performing some simple operations 00:02:12.410 --> 00:02:13.750 align:middle line:90% on the output of the filter. 00:02:13.750 --> 00:02:16.970 align:middle line:84% So this is just the raw filter output, the convolution 00:02:16.970 --> 00:02:18.310 align:middle line:90% of the filter with the image. 00:02:18.310 --> 00:02:21.673 align:middle line:84% And this is a horizontal slice there. 00:02:21.673 --> 00:02:23.090 align:middle line:84% If you square the filter response, 00:02:23.090 --> 00:02:24.730 align:middle line:84% you can make everything positive. 00:02:24.730 --> 00:02:27.250 align:middle line:84% And then if you average over local regions, 00:02:27.250 --> 00:02:31.490 align:middle line:84% you get this quantity that's a lot like the energy measure 00:02:31.490 --> 00:02:34.630 align:middle line:84% that we talked about as being a good model of complex cells. 00:02:34.630 --> 00:02:36.650 align:middle line:84% And in this case, the energy measure 00:02:36.650 --> 00:02:40.530 align:middle line:84% has a different value kind of in the surround than in the center. 00:02:40.530 --> 00:02:42.490 align:middle line:84% So it kind of makes explicit the idea 00:02:42.490 --> 00:02:44.650 align:middle line:84% that there's a different kind of stuff 00:02:44.650 --> 00:02:47.310 align:middle line:84% in the center region of the image than in the surround. 00:02:47.310 --> 00:02:50.310 align:middle line:90% 00:02:50.310 --> 00:02:54.250 align:middle line:84% So these energy measures can predict some texture boundaries. 00:02:54.250 --> 00:02:57.220 align:middle line:84% And this is that energy measure displayed 00:02:57.220 --> 00:03:00.780 align:middle line:90% as a convolution-like operation. 00:03:00.780 --> 00:03:03.700 align:middle line:84% And it captures this idea that texture is really 00:03:03.700 --> 00:03:05.820 align:middle line:84% defined by the average properties of a local image 00:03:05.820 --> 00:03:06.320 align:middle line:90% region. 00:03:06.320 --> 00:03:09.260 align:middle line:84% So what makes a texture a texture 00:03:09.260 --> 00:03:12.580 align:middle line:90% is what it's like on average. 00:03:12.580 --> 00:03:15.540 align:middle line:84% So in this case, there's some kind 00:03:15.540 --> 00:03:18.660 align:middle line:84% of orientation energy in one part of the image, 00:03:18.660 --> 00:03:22.320 align:middle line:90% and a different type in another. 00:03:22.320 --> 00:03:24.380 align:middle line:84% So then we asked whether we could potentially 00:03:24.380 --> 00:03:27.980 align:middle line:84% account for the perception of texture boundaries 00:03:27.980 --> 00:03:30.060 align:middle line:84% in some of these more complicated textures 00:03:30.060 --> 00:03:32.223 align:middle line:84% with these similarly simple models, 00:03:32.223 --> 00:03:34.140 align:middle line:84% and saw a bunch of examples that kind indicate 00:03:34.140 --> 00:03:36.100 align:middle line:90% that they do a pretty good job. 00:03:36.100 --> 00:03:38.140 align:middle line:84% So specifically, we can look at a whole bunch 00:03:38.140 --> 00:03:40.860 align:middle line:84% of different images of this nature, 00:03:40.860 --> 00:03:44.140 align:middle line:84% and the extent to which you really 00:03:44.140 --> 00:03:47.420 align:middle line:84% see a salient region in the middle 00:03:47.420 --> 00:03:49.060 align:middle line:84% is kind of mirrored by the extent 00:03:49.060 --> 00:03:51.460 align:middle line:84% to which these simple mechanisms show 00:03:51.460 --> 00:03:52.840 align:middle line:90% differences in their responses. 00:03:52.840 --> 00:03:56.150 align:middle line:84% So here's a case where the center region is really obvious 00:03:56.150 --> 00:03:57.670 align:middle line:84% and you see a really big difference. 00:03:57.670 --> 00:04:02.470 align:middle line:84% Here's a case where the little elements in the texture 00:04:02.470 --> 00:04:04.230 align:middle line:90% have been scaled. 00:04:04.230 --> 00:04:06.470 align:middle line:84% And at this particular relative scale, 00:04:06.470 --> 00:04:08.670 align:middle line:84% the difference doesn't really pop out at you, 00:04:08.670 --> 00:04:11.690 align:middle line:84% and it's also not really evident in this energy-based mechanism. 00:04:11.690 --> 00:04:14.450 align:middle line:90% 00:04:14.450 --> 00:04:19.550 align:middle line:84% So then we kind of pivoted to this question of whether 00:04:19.550 --> 00:04:22.430 align:middle line:84% or not these kind of very simple models that 00:04:22.430 --> 00:04:25.310 align:middle line:84% are based on the ingredients that we think 00:04:25.310 --> 00:04:28.390 align:middle line:84% of as being present in V1, whether they can actually 00:04:28.390 --> 00:04:31.110 align:middle line:84% account for the way that textures look. 00:04:31.110 --> 00:04:35.990 align:middle line:84% And the critical idea that we discussed is that 00:04:35.990 --> 00:04:38.550 align:middle line:84% if we've correctly identified the brain's representation 00:04:38.550 --> 00:04:41.810 align:middle line:84% of texture, then if we measure that representation, 00:04:41.810 --> 00:04:44.610 align:middle line:84% for some example natural texture-- 00:04:44.610 --> 00:04:46.390 align:middle line:90% some image that you give me-- 00:04:46.390 --> 00:04:50.310 align:middle line:84% and then we generate synthetic textures that 00:04:50.310 --> 00:04:52.930 align:middle line:84% have the same values of that representation, 00:04:52.930 --> 00:04:54.370 align:middle line:84% well, then the synthetic textures, 00:04:54.370 --> 00:04:55.650 align:middle line:84% they should look like the example. 00:04:55.650 --> 00:04:57.608 align:middle line:84% They should look like the same type of texture. 00:04:57.608 --> 00:05:01.910 align:middle line:84% So that's the principle that we're working with here. 00:05:01.910 --> 00:05:08.390 align:middle line:84% And so the key notion here is that a given type of texture 00:05:08.390 --> 00:05:10.568 align:middle line:84% can be present in a lot of different examples. 00:05:10.568 --> 00:05:12.610 align:middle line:84% So there's lots of examples of any given texture. 00:05:12.610 --> 00:05:14.990 align:middle line:84% And we looked at the carpet, and you 00:05:14.990 --> 00:05:17.010 align:middle line:84% could look at different regions of the carpet. 00:05:17.010 --> 00:05:18.030 align:middle line:84% They're all the same kind of texture 00:05:18.030 --> 00:05:20.010 align:middle line:84% even though they're slightly different images. 00:05:20.010 --> 00:05:23.430 align:middle line:84% So textures kind of consist of an equivalence class 00:05:23.430 --> 00:05:25.270 align:middle line:84% of all these different images that share 00:05:25.270 --> 00:05:26.450 align:middle line:90% some kind of property. 00:05:26.450 --> 00:05:29.927 align:middle line:84% And in this particular case, there's 00:05:29.927 --> 00:05:31.510 align:middle line:84% an equivalence class of all the things 00:05:31.510 --> 00:05:33.750 align:middle line:84% that are generated by the same physical process that 00:05:33.750 --> 00:05:38.990 align:middle line:84% kind of created carpet, for instance, or leaves. 00:05:38.990 --> 00:05:42.890 align:middle line:84% But if we're really interested in how textures look, 00:05:42.890 --> 00:05:44.910 align:middle line:84% then what really matters to us is 00:05:44.910 --> 00:05:46.890 align:middle line:84% whether textures are perceptually equivalent, 00:05:46.890 --> 00:05:51.480 align:middle line:84% so whether they'll look the same to a human observer. 00:05:51.480 --> 00:05:53.640 align:middle line:84% And so the game that we're going to play 00:05:53.640 --> 00:05:56.800 align:middle line:84% is to try to synthesize images that 00:05:56.800 --> 00:06:00.640 align:middle line:84% will have the same values of some candidate representation 00:06:00.640 --> 00:06:04.120 align:middle line:84% that embodies a theory of what is represented in your head 00:06:04.120 --> 00:06:05.520 align:middle line:90% when you look at a texture. 00:06:05.520 --> 00:06:06.840 align:middle line:84% And then we're going to look at the textures 00:06:06.840 --> 00:06:08.673 align:middle line:84% and actually see whether they look the same. 00:06:08.673 --> 00:06:12.160 align:middle line:84% So the idea is that we're going to take noise images, 00:06:12.160 --> 00:06:14.920 align:middle line:84% and then try to alter them in various ways 00:06:14.920 --> 00:06:19.220 align:middle line:84% to actually cause them to reside in this part of the space 00:06:19.220 --> 00:06:22.260 align:middle line:84% that's got the same representation as some example. 00:06:22.260 --> 00:06:25.180 align:middle line:84% So this was a case where maybe we got part of the way there, 00:06:25.180 --> 00:06:26.460 align:middle line:90% but not all the way there. 00:06:26.460 --> 00:06:30.420 align:middle line:90% 00:06:30.420 --> 00:06:33.100 align:middle line:84% And so the initial example that we talked about-- 00:06:33.100 --> 00:06:35.280 align:middle line:84% and this is where we ended last time-- 00:06:35.280 --> 00:06:40.480 align:middle line:84% was this initial attempt at implementing this methodology 00:06:40.480 --> 00:06:44.360 align:middle line:84% using a candidate representation of texture 00:06:44.360 --> 00:06:49.040 align:middle line:84% that consists of marginal histograms of filter responses, 00:06:49.040 --> 00:06:54.650 align:middle line:84% so the distribution of the filter responses. 00:06:54.650 --> 00:06:57.330 align:middle line:84% And so this is the representation 00:06:57.330 --> 00:06:58.830 align:middle line:90% that this is based on. 00:06:58.830 --> 00:07:00.590 align:middle line:90% So we're taking an image. 00:07:00.590 --> 00:07:02.330 align:middle line:84% We convolve it with a big set of filters 00:07:02.330 --> 00:07:05.090 align:middle line:84% that are tuned in orientation and spatial frequency, 00:07:05.090 --> 00:07:08.010 align:middle line:84% and you get a bunch of convolutions. 00:07:08.010 --> 00:07:11.050 align:middle line:84% These are often referred to as subbands. 00:07:11.050 --> 00:07:12.930 align:middle line:84% And so the model of texture perception 00:07:12.930 --> 00:07:16.170 align:middle line:84% will be to represent textures with the distribution 00:07:16.170 --> 00:07:18.810 align:middle line:84% of the activity in these different subbands, which 00:07:18.810 --> 00:07:21.810 align:middle line:84% we take as simulations of the responses of oriented, spatial 00:07:21.810 --> 00:07:23.265 align:middle line:90% frequency tuned neurons. 00:07:23.265 --> 00:07:25.890 align:middle line:84% So we talked about the idea that this is often called a pyramid 00:07:25.890 --> 00:07:27.930 align:middle line:90% representation of an image. 00:07:27.930 --> 00:07:31.750 align:middle line:84% So we've got different spatial scales of images, 00:07:31.750 --> 00:07:34.290 align:middle line:84% and when the spatial scale is coarse, 00:07:34.290 --> 00:07:35.717 align:middle line:90% the image can be downsampled. 00:07:35.717 --> 00:07:37.550 align:middle line:84% And so that's what these things are up here. 00:07:37.550 --> 00:07:39.710 align:middle line:84% These are the low spatial frequencies of the image. 00:07:39.710 --> 00:07:43.585 align:middle line:84% And they can be represented with a smaller pixel array. 00:07:43.585 --> 00:07:45.710 align:middle line:84% So that's why it's called a pyramid representation. 00:07:45.710 --> 00:07:49.200 align:middle line:84% So you have these big subband representations here, 00:07:49.200 --> 00:07:52.340 align:middle line:84% and then they get smaller as you move up the pyramid. 00:07:52.340 --> 00:07:57.500 align:middle line:84% So it's kind of a classic image processing-based representation. 00:07:57.500 --> 00:07:59.860 align:middle line:84% And so remember that the representation 00:07:59.860 --> 00:08:02.980 align:middle line:84% we're trying to evaluate here is one 00:08:02.980 --> 00:08:06.180 align:middle line:84% where you take each of these subband representations 00:08:06.180 --> 00:08:08.500 align:middle line:90% and you form a histogram. 00:08:08.500 --> 00:08:11.500 align:middle line:84% So the histogram is just the distribution 00:08:11.500 --> 00:08:15.460 align:middle line:90% of the values in this array. 00:08:15.460 --> 00:08:17.700 align:middle line:84% So it's like throwing all the values into a bag 00:08:17.700 --> 00:08:19.980 align:middle line:84% and then seeing what the distribution is. 00:08:19.980 --> 00:08:22.780 align:middle line:84% So you're throwing out all of the spatial organization 00:08:22.780 --> 00:08:23.740 align:middle line:90% of the image. 00:08:23.740 --> 00:08:25.780 align:middle line:84% And so this is really just capturing 00:08:25.780 --> 00:08:29.420 align:middle line:84% the average properties of that subband, 00:08:29.420 --> 00:08:32.860 align:middle line:84% in this case averaged across space. 00:08:32.860 --> 00:08:35.940 align:middle line:84% So it captures the variance as well as other as well as 00:08:35.940 --> 00:08:39.260 align:middle line:90% all other marginal statistics. 00:08:39.260 --> 00:08:41.520 align:middle line:90% So that's the proposal. 00:08:41.520 --> 00:08:45.060 align:middle line:84% So what we want to do is to take some candid image, 00:08:45.060 --> 00:08:47.180 align:middle line:84% generate the subband in representation, 00:08:47.180 --> 00:08:49.860 align:middle line:84% and then generate these histograms. 00:08:49.860 --> 00:08:53.220 align:middle line:84% And then we want to take a noise image, and then mess 00:08:53.220 --> 00:08:57.300 align:middle line:84% with the image so that it has the same histograms 00:08:57.300 --> 00:09:02.740 align:middle line:84% in these subbands in the filter responses. 00:09:02.740 --> 00:09:04.380 align:middle line:84% And so in this particular case, there 00:09:04.380 --> 00:09:07.380 align:middle line:84% was a pretty simple trick to do this called histogram 00:09:07.380 --> 00:09:09.020 align:middle line:90% equalization. 00:09:09.020 --> 00:09:11.820 align:middle line:84% So all that you do is you generate 00:09:11.820 --> 00:09:17.420 align:middle line:84% the cumulative histogram for an image or a subband. 00:09:17.420 --> 00:09:20.500 align:middle line:84% And then there's just a simple kind of mapping function 00:09:20.500 --> 00:09:25.620 align:middle line:84% where you take a given pixel in the noise image, 00:09:25.620 --> 00:09:29.380 align:middle line:84% and then you change its value depending 00:09:29.380 --> 00:09:33.440 align:middle line:84% on where it falls in the cumulative histogram. 00:09:33.440 --> 00:09:38.020 align:middle line:84% So in this particular case, this intensity value 00:09:38.020 --> 00:09:39.705 align:middle line:84% gives you this probability value. 00:09:39.705 --> 00:09:41.580 align:middle line:84% So you map that over here, and that tells you 00:09:41.580 --> 00:09:43.200 align:middle line:84% that this thing is going to be 1. 00:09:43.200 --> 00:09:45.710 align:middle line:84% And so in this case, we're just doing 00:09:45.710 --> 00:09:48.290 align:middle line:84% this on the pixel intensities, and so the noise image 00:09:48.290 --> 00:09:50.790 align:middle line:84% ends up getting turned into this thing that's kind of mostly 00:09:50.790 --> 00:09:54.110 align:middle line:84% white with a little bit of black because that's 00:09:54.110 --> 00:09:56.590 align:middle line:84% the pixel intensity histogram that you have 00:09:56.590 --> 00:09:58.170 align:middle line:90% in this original test image. 00:09:58.170 --> 00:10:00.750 align:middle line:90% 00:10:00.750 --> 00:10:03.910 align:middle line:84% So now we're going to do the same thing with the histograms 00:10:03.910 --> 00:10:05.470 align:middle line:90% of the subbands. 00:10:05.470 --> 00:10:09.670 align:middle line:84% So this original texture has subband histograms that 00:10:09.670 --> 00:10:11.630 align:middle line:90% have these very long tails. 00:10:11.630 --> 00:10:15.110 align:middle line:84% The noise image has something that's a lot more Gaussian. 00:10:15.110 --> 00:10:17.190 align:middle line:84% And so we do histogram equalization, 00:10:17.190 --> 00:10:20.810 align:middle line:84% and the noise subband gets kind of altered in this way, 00:10:20.810 --> 00:10:22.870 align:middle line:84% so that it's got a histogram that 00:10:22.870 --> 00:10:25.710 align:middle line:84% looks a lot more like the original one. 00:10:25.710 --> 00:10:29.030 align:middle line:84% So we do this for all of the different subbands. 00:10:29.030 --> 00:10:32.930 align:middle line:84% And then this subband representation can be inverted. 00:10:32.930 --> 00:10:36.190 align:middle line:84% You can take it and then essentially add these things 00:10:36.190 --> 00:10:39.390 align:middle line:84% back up in order to get an original image. 00:10:39.390 --> 00:10:42.310 align:middle line:84% So this is the result showing you that histograms are all 00:10:42.310 --> 00:10:43.440 align:middle line:90% matched. 00:10:43.440 --> 00:10:47.840 align:middle line:84% So we do this, and we've now taken this noise image, 00:10:47.840 --> 00:10:50.880 align:middle line:84% and we've altered it so that it now 00:10:50.880 --> 00:10:55.360 align:middle line:84% matches the representation for some natural texture. 00:10:55.360 --> 00:10:58.800 align:middle line:84% And remember, the whole point of doing this 00:10:58.800 --> 00:11:01.880 align:middle line:84% is to actually test whether that representation that we're 00:11:01.880 --> 00:11:05.640 align:middle line:84% matching is a good model for the texture 00:11:05.640 --> 00:11:07.480 align:middle line:90% representation in our brain. 00:11:07.480 --> 00:11:10.175 align:middle line:84% So if the texture representation that we're 00:11:10.175 --> 00:11:11.800 align:middle line:84% doing the matching with captures what's 00:11:11.800 --> 00:11:13.340 align:middle line:84% in our brain's representation of texture, 00:11:13.340 --> 00:11:14.715 align:middle line:84% then the synthetic examples ought 00:11:14.715 --> 00:11:16.140 align:middle line:84% to look like the original texture. 00:11:16.140 --> 00:11:18.080 align:middle line:84% Now, there's one other little detail here 00:11:18.080 --> 00:11:20.280 align:middle line:84% that I'll tell you about, which is 00:11:20.280 --> 00:11:24.560 align:middle line:84% that we're going to actually see this done on color images. 00:11:24.560 --> 00:11:26.880 align:middle line:84% So color images are slightly more 00:11:26.880 --> 00:11:28.680 align:middle line:84% complicated than the black and white image 00:11:28.680 --> 00:11:31.620 align:middle line:84% that I just showed you because there are three color channels. 00:11:31.620 --> 00:11:33.600 align:middle line:84% Remember, because there are three cones, 00:11:33.600 --> 00:11:36.240 align:middle line:84% three color channels are sufficient to recreate 00:11:36.240 --> 00:11:37.860 align:middle line:90% the perception of all colors. 00:11:37.860 --> 00:11:41.160 align:middle line:84% And so there's a little trick that's 00:11:41.160 --> 00:11:43.680 align:middle line:84% used to actually do this on color images, which 00:11:43.680 --> 00:11:46.880 align:middle line:84% is that you do this in the principal component 00:11:46.880 --> 00:11:49.220 align:middle line:90% space of the RGB values. 00:11:49.220 --> 00:11:53.240 align:middle line:84% So you essentially work with color axes that 00:11:53.240 --> 00:11:55.560 align:middle line:90% are orthogonal or uncorrelated. 00:11:55.560 --> 00:11:58.720 align:middle line:84% And that means that you can mess with them independently and then 00:11:58.720 --> 00:12:00.120 align:middle line:90% add them back up. 00:12:00.120 --> 00:12:02.700 align:middle line:90% So that's a small detail. 00:12:02.700 --> 00:12:08.440 align:middle line:84% But the big picture here is to inspect these pairs of images. 00:12:08.440 --> 00:12:11.500 align:middle line:84% One of these is the original, so that's the one on the left. 00:12:11.500 --> 00:12:13.960 align:middle line:84% And the right image is a synthetic image 00:12:13.960 --> 00:12:18.240 align:middle line:84% that was generated from noise just by matching 00:12:18.240 --> 00:12:20.440 align:middle line:90% the subband histograms. 00:12:20.440 --> 00:12:23.720 align:middle line:84% And so you can see that in some cases, 00:12:23.720 --> 00:12:26.820 align:middle line:84% this produces actually pretty good-looking textures right. 00:12:26.820 --> 00:12:28.540 align:middle line:90% So that one looks pretty good. 00:12:28.540 --> 00:12:31.280 align:middle line:90% 00:12:31.280 --> 00:12:33.020 align:middle line:84% That one maybe looks pretty good. 00:12:33.020 --> 00:12:34.815 align:middle line:84% In some cases, they don't look perfect. 00:12:34.815 --> 00:12:36.440 align:middle line:84% You can definitely see some things that 00:12:36.440 --> 00:12:39.400 align:middle line:90% are kind of funny about them. 00:12:39.400 --> 00:12:41.410 align:middle line:84% But there are a lot that look pretty good. 00:12:41.410 --> 00:12:44.370 align:middle line:84% Here's some other examples that look pretty reasonable. 00:12:44.370 --> 00:12:46.370 align:middle line:84% So maybe you can get something that 00:12:46.370 --> 00:12:49.750 align:middle line:84% looks like a leaf or some kind of carpet or something. 00:12:49.750 --> 00:12:53.585 align:middle line:90% 00:12:53.585 --> 00:12:55.890 align:middle line:84% But there's a lot of other cases where the synthesis 00:12:55.890 --> 00:12:57.270 align:middle line:90% pretty clearly fails. 00:12:57.270 --> 00:13:00.350 align:middle line:90% So for instance, what is this? 00:13:00.350 --> 00:13:03.170 align:middle line:90% 00:13:03.170 --> 00:13:05.222 align:middle line:90% What material is this? 00:13:05.222 --> 00:13:05.930 align:middle line:90% AUDIENCE: Marble. 00:13:05.930 --> 00:13:06.990 align:middle line:90% JOSH MCDERMOTT: Marble, exactly. 00:13:06.990 --> 00:13:08.110 align:middle line:90% Everyone recognizes this. 00:13:08.110 --> 00:13:08.670 align:middle line:90% It's marble. 00:13:08.670 --> 00:13:10.250 align:middle line:90% It looks like marble. 00:13:10.250 --> 00:13:11.670 align:middle line:90% This does not look like marble. 00:13:11.670 --> 00:13:16.730 align:middle line:84% So it just has a very different appearance. 00:13:16.730 --> 00:13:18.370 align:middle line:90% What is this? 00:13:18.370 --> 00:13:19.250 align:middle line:90% AUDIENCE: Hay. 00:13:19.250 --> 00:13:20.170 align:middle line:90% JOSH MCDERMOTT: Yep. 00:13:20.170 --> 00:13:23.092 align:middle line:90% That is not hay. 00:13:23.092 --> 00:13:24.550 align:middle line:84% I don't know what these things are. 00:13:24.550 --> 00:13:30.290 align:middle line:84% Something more exotic, but it also doesn't look the same. 00:13:30.290 --> 00:13:31.770 align:middle line:90% So what does this tell us? 00:13:31.770 --> 00:13:36.090 align:middle line:84% Well, this tells us that when you are looking at these images, 00:13:36.090 --> 00:13:39.300 align:middle line:84% your brain is generating a representation that 00:13:39.300 --> 00:13:42.220 align:middle line:84% differs from the one that we used 00:13:42.220 --> 00:13:44.120 align:middle line:90% to do this matching process. 00:13:44.120 --> 00:13:47.660 align:middle line:84% So you're representing something above and beyond 00:13:47.660 --> 00:13:50.120 align:middle line:90% these subband histograms. 00:13:50.120 --> 00:13:53.560 align:middle line:90% 00:13:53.560 --> 00:13:54.260 align:middle line:90% So what's wrong? 00:13:54.260 --> 00:13:56.080 align:middle line:90% Like what's missing here? 00:13:56.080 --> 00:13:57.680 align:middle line:84% So one thing that you might notice-- 00:13:57.680 --> 00:13:59.600 align:middle line:84% so when you look at the marble, for instance, 00:13:59.600 --> 00:14:02.100 align:middle line:84% the marble kind of has these long-- 00:14:02.100 --> 00:14:04.040 align:middle line:90% these elongated things in it. 00:14:04.040 --> 00:14:08.603 align:middle line:84% Some function of the composition of marble causes that to happen. 00:14:08.603 --> 00:14:10.020 align:middle line:84% I don't know what, but that's just 00:14:10.020 --> 00:14:11.720 align:middle line:84% part of what makes marble look like marble. 00:14:11.720 --> 00:14:13.700 align:middle line:84% And then when you look at the synthetic image, 00:14:13.700 --> 00:14:14.700 align:middle line:90% that's kind of missing. 00:14:14.700 --> 00:14:16.720 align:middle line:84% And you get the same issue with the hay. 00:14:16.720 --> 00:14:19.620 align:middle line:84% The hay has got all these strands of hay, 00:14:19.620 --> 00:14:21.240 align:middle line:84% and those are missing in the image. 00:14:21.240 --> 00:14:23.900 align:middle line:90% 00:14:23.900 --> 00:14:27.420 align:middle line:84% And so one of the things that characterizes 00:14:27.420 --> 00:14:29.180 align:middle line:84% these kind of long, elongated structures 00:14:29.180 --> 00:14:31.980 align:middle line:84% is that there are these statistical dependencies 00:14:31.980 --> 00:14:35.923 align:middle line:84% between orientation energy at different locations. 00:14:35.923 --> 00:14:37.340 align:middle line:84% And this is a little bit like what 00:14:37.340 --> 00:14:39.340 align:middle line:84% we talked about when we were talking 00:14:39.340 --> 00:14:42.460 align:middle line:84% about the statistical basis for grouping by good continuation, 00:14:42.460 --> 00:14:47.668 align:middle line:84% where you can measure the dependencies 00:14:47.668 --> 00:14:49.960 align:middle line:84% between orientations at different positions and images. 00:14:49.960 --> 00:14:52.020 align:middle line:90% And so edges tend to-- 00:14:52.020 --> 00:14:53.960 align:middle line:84% natural images have a lot of elongated edges. 00:14:53.960 --> 00:14:56.060 align:middle line:84% So you have an orientation here, you tend 00:14:56.060 --> 00:14:57.353 align:middle line:90% to have an orientation here. 00:14:57.353 --> 00:14:59.520 align:middle line:84% And so some of these textures have those properties, 00:14:59.520 --> 00:15:03.580 align:middle line:84% and that's not really being captured 00:15:03.580 --> 00:15:06.540 align:middle line:84% And so one way that you might imagine 00:15:06.540 --> 00:15:08.900 align:middle line:84% being able to capture that is by measuring 00:15:08.900 --> 00:15:12.500 align:middle line:84% the correlation between, say, a filter response here, 00:15:12.500 --> 00:15:14.200 align:middle line:90% and a filter response here. 00:15:14.200 --> 00:15:16.080 align:middle line:84% And this representation doesn't do that. 00:15:16.080 --> 00:15:17.940 align:middle line:84% So what does this representation do? 00:15:17.940 --> 00:15:20.340 align:middle line:84% Well, it measures-- the filter response 00:15:20.340 --> 00:15:22.220 align:middle line:90% is across this whole image. 00:15:22.220 --> 00:15:23.220 align:middle line:90% So you get some numbers. 00:15:23.220 --> 00:15:24.940 align:middle line:84% And then it just throws them in a bag. 00:15:24.940 --> 00:15:28.060 align:middle line:84% And you get the distribution of the bag. 00:15:28.060 --> 00:15:31.260 align:middle line:84% So whether or not like this is active at the same time 00:15:31.260 --> 00:15:33.720 align:middle line:84% as this is active is lost in that process. 00:15:33.720 --> 00:15:35.910 align:middle line:90% So it's a marginal distribution. 00:15:35.910 --> 00:15:38.190 align:middle line:84% You don't really have any information 00:15:38.190 --> 00:15:40.030 align:middle line:84% about the relationship between responses 00:15:40.030 --> 00:15:43.270 align:middle line:84% at, say, different points in space or different orientations. 00:15:43.270 --> 00:15:46.130 align:middle line:90% 00:15:46.130 --> 00:15:49.070 align:middle line:90% So big picture here. 00:15:49.070 --> 00:15:51.970 align:middle line:84% This was a very influential initial attempt at this. 00:15:51.970 --> 00:15:55.550 align:middle line:84% It was really like the first instance 00:15:55.550 --> 00:15:57.710 align:middle line:90% of the use of texture synthesis. 00:15:57.710 --> 00:16:01.630 align:middle line:84% It's very simple and elegant, but it's not 00:16:01.630 --> 00:16:06.490 align:middle line:84% great as a theory of texture perception. 00:16:06.490 --> 00:16:11.590 align:middle line:84% So as I said, the intuitive explanation of the failure 00:16:11.590 --> 00:16:14.750 align:middle line:84% is that there are these higher-order statistical 00:16:14.750 --> 00:16:18.830 align:middle line:84% properties that are present in these natural textures 00:16:18.830 --> 00:16:20.990 align:middle line:84% that we might plausibly be sensitive to that 00:16:20.990 --> 00:16:23.870 align:middle line:90% are missing from this model. 00:16:23.870 --> 00:16:25.770 align:middle line:90% So what if we include them? 00:16:25.770 --> 00:16:31.510 align:middle line:84% So what is shown here is a picture that you've seen before. 00:16:31.510 --> 00:16:35.120 align:middle line:84% This is a canonical model of early vision. 00:16:35.120 --> 00:16:36.940 align:middle line:84% So you start with an image up there, 00:16:36.940 --> 00:16:40.892 align:middle line:84% and then you have several stages of simple operations. 00:16:40.892 --> 00:16:42.600 align:middle line:84% So there's some center surround filtering 00:16:42.600 --> 00:16:44.160 align:middle line:84% kind of like what we think happens 00:16:44.160 --> 00:16:45.680 align:middle line:90% in the retina and the LGN. 00:16:45.680 --> 00:16:48.960 align:middle line:84% There's some spatial frequency tuned and orientation tuned 00:16:48.960 --> 00:16:51.880 align:middle line:84% filtering here, like what we think happens in V1. 00:16:51.880 --> 00:16:55.360 align:middle line:84% And then there's what's called envelope extraction, which is 00:16:55.360 --> 00:16:56.780 align:middle line:90% sort of like an energy measure. 00:16:56.780 --> 00:16:59.760 align:middle line:84% So that's like a complex cell kind of stage. 00:16:59.760 --> 00:17:03.860 align:middle line:84% So this is the energy kind of measure. 00:17:03.860 --> 00:17:05.859 align:middle line:84% So this is our standard model of early vision. 00:17:05.859 --> 00:17:09.560 align:middle line:84% And the little, red circles that are stuck on top of it 00:17:09.560 --> 00:17:11.240 align:middle line:84% are different statistics that are 00:17:11.240 --> 00:17:12.859 align:middle line:90% computed from different stages. 00:17:12.859 --> 00:17:16.900 align:middle line:84% So the word M stands for a Marginal statistic, 00:17:16.900 --> 00:17:20.160 align:middle line:84% so the average or the variance, things like that. 00:17:20.160 --> 00:17:21.920 align:middle line:84% And then the things that have a C in them 00:17:21.920 --> 00:17:24.260 align:middle line:84% are different types of Correlations. 00:17:24.260 --> 00:17:26.880 align:middle line:84% So AC is an Autocorrelation, so that's 00:17:26.880 --> 00:17:29.840 align:middle line:84% a correlation between different points in space. 00:17:29.840 --> 00:17:32.490 align:middle line:84% And then these other Cs are correlations 00:17:32.490 --> 00:17:37.690 align:middle line:84% between different orientations or different spatial scales. 00:17:37.690 --> 00:17:39.810 align:middle line:84% So it's measuring the joint dependence 00:17:39.810 --> 00:17:44.670 align:middle line:84% of the responses along these different dimensions. 00:17:44.670 --> 00:17:47.590 align:middle line:84% But essentially, it's just a big set of statistics. 00:17:47.590 --> 00:17:51.290 align:middle line:84% And all of them are kind of capturing the average properties 00:17:51.290 --> 00:17:55.930 align:middle line:84% over space of the image viewed through a certain type 00:17:55.930 --> 00:17:57.150 align:middle line:90% of representation. 00:17:57.150 --> 00:18:00.270 align:middle line:90% 00:18:00.270 --> 00:18:04.430 align:middle line:84% So what we'd like to do now is the same kind of thing. 00:18:04.430 --> 00:18:07.950 align:middle line:84% So we'd like to test this idea using synthesis. 00:18:07.950 --> 00:18:11.770 align:middle line:84% So what's shown here is a model whereby you 00:18:11.770 --> 00:18:13.710 align:middle line:90% can measure these statistics. 00:18:13.710 --> 00:18:16.110 align:middle line:84% So you take your model of early vision, 00:18:16.110 --> 00:18:17.870 align:middle line:84% and then we can measure these statistics. 00:18:17.870 --> 00:18:21.050 align:middle line:84% They're just mathematical functions of this. 00:18:21.050 --> 00:18:25.090 align:middle line:84% And so what we want to do is now generate a new image 00:18:25.090 --> 00:18:27.250 align:middle line:84% starting from some random image that's going 00:18:27.250 --> 00:18:30.290 align:middle line:90% to match these statistics. 00:18:30.290 --> 00:18:33.510 align:middle line:84% And so because of the fact that we're using correlations, 00:18:33.510 --> 00:18:38.930 align:middle line:84% we can't use the sort of simple trick of histogram equalization. 00:18:38.930 --> 00:18:42.210 align:middle line:84% So instead, people use gradient-based optimization, 00:18:42.210 --> 00:18:43.870 align:middle line:84% which nowadays is like super standard, 00:18:43.870 --> 00:18:45.630 align:middle line:90% and like pretty easy to do. 00:18:45.630 --> 00:18:48.670 align:middle line:84% So you could even think of this as a neural network. 00:18:48.670 --> 00:18:51.370 align:middle line:84% This is the output of the neural network. 00:18:51.370 --> 00:18:55.970 align:middle line:84% And what you're going to do is solve an optimization problem 00:18:55.970 --> 00:18:58.530 align:middle line:84% where you want to minimize the difference 00:18:58.530 --> 00:19:02.147 align:middle line:84% between the statistics of a synthetic texture-- which is not 00:19:02.147 --> 00:19:03.730 align:middle line:84% shown here, so that would be something 00:19:03.730 --> 00:19:07.370 align:middle line:84% that's initialized as noise-- and those of a natural texture. 00:19:07.370 --> 00:19:10.610 align:middle line:84% So the natural texture, you get this set of numbers. 00:19:10.610 --> 00:19:13.190 align:middle line:84% And you can then write down a loss function, 00:19:13.190 --> 00:19:16.130 align:middle line:84% which tells you about the difference between the numbers 00:19:16.130 --> 00:19:17.850 align:middle line:84% from this image and the numbers that you 00:19:17.850 --> 00:19:21.290 align:middle line:84% would get from a noise image passed through that same model. 00:19:21.290 --> 00:19:23.970 align:middle line:84% And then you can compute the gradient of that loss function 00:19:23.970 --> 00:19:25.978 align:middle line:90% with respect to the noise image. 00:19:25.978 --> 00:19:27.770 align:middle line:84% And that's going to tell you-- the gradient 00:19:27.770 --> 00:19:30.740 align:middle line:84% is going to tell you how to change the noise image so as 00:19:30.740 --> 00:19:32.440 align:middle line:90% to minimize that loss function. 00:19:32.440 --> 00:19:35.660 align:middle line:84% That is, to make the statistics of the noise image become 00:19:35.660 --> 00:19:40.198 align:middle line:84% similar to those of the natural image. 00:19:40.198 --> 00:19:42.240 align:middle line:84% So it's a different method of doing the matching, 00:19:42.240 --> 00:19:44.920 align:middle line:84% but the scientific purpose is really the same. 00:19:44.920 --> 00:19:48.180 align:middle line:84% You're trying to synthesize an image that 00:19:48.180 --> 00:19:52.300 align:middle line:84% has the same values of this candidate representation, which, 00:19:52.300 --> 00:19:57.000 align:middle line:84% in this case is a bunch of statistics measured on the image 00:19:57.000 --> 00:20:01.372 align:middle line:84% You can think of these that-- when we talk about correlations 00:20:01.372 --> 00:20:03.080 align:middle line:84% and stuff, it may sound kind of abstract, 00:20:03.080 --> 00:20:06.100 align:middle line:84% but you can think about building this into a pretty neurally 00:20:06.100 --> 00:20:07.560 align:middle line:90% plausible model. 00:20:07.560 --> 00:20:11.940 align:middle line:84% So this is showing several-- a few different stages 00:20:11.940 --> 00:20:14.360 align:middle line:84% by which you could compute these correlations. 00:20:14.360 --> 00:20:16.180 align:middle line:84% So you've got these original images. 00:20:16.180 --> 00:20:20.540 align:middle line:84% You convolve it with a set of V1-like filters. 00:20:20.540 --> 00:20:23.860 align:middle line:84% And then you could measure products between these filters, 00:20:23.860 --> 00:20:26.100 align:middle line:84% and then average those over space, 00:20:26.100 --> 00:20:28.410 align:middle line:84% and that will give you a correlation. 00:20:28.410 --> 00:20:31.270 align:middle line:84% So correlation is like the average between 00:20:31.270 --> 00:20:34.550 align:middle line:84% is the average of a product between two variables, 00:20:34.550 --> 00:20:38.580 align:middle line:84% maybe with the mean subtracted out. 00:20:38.580 --> 00:20:40.070 align:middle line:84% And so we're going to, again, try 00:20:40.070 --> 00:20:42.112 align:middle line:84% to minimize the difference between the statistics 00:20:42.112 --> 00:20:44.470 align:middle line:84% of a synthetic texture and those of a natural texture. 00:20:44.470 --> 00:20:46.370 align:middle line:84% So again, it's an optimization problem. 00:20:46.370 --> 00:20:51.962 align:middle line:84% And nowadays, we have lots of ways to solve this. 00:20:51.962 --> 00:20:53.670 align:middle line:84% Back when this was originally introduced, 00:20:53.670 --> 00:20:56.710 align:middle line:84% this was a little bit harder to do. 00:20:56.710 --> 00:20:58.510 align:middle line:84% And so what I'm going to show you here 00:20:58.510 --> 00:21:01.710 align:middle line:84% are a bunch of examples of synthetic textures 00:21:01.710 --> 00:21:03.590 align:middle line:90% that are generated in this way. 00:21:03.590 --> 00:21:06.550 align:middle line:90% So here is an original texture. 00:21:06.550 --> 00:21:11.230 align:middle line:84% This is synthesized just using these the marginal histograms 00:21:11.230 --> 00:21:14.110 align:middle line:84% of Heeger and Bergen, so it doesn't look that great. 00:21:14.110 --> 00:21:17.230 align:middle line:84% And this is synthesized with this newer version from Portilla 00:21:17.230 --> 00:21:20.590 align:middle line:84% and Simoncelli that incorporates correlations 00:21:20.590 --> 00:21:22.790 align:middle line:84% and other statistics, and it looks like a little bit 00:21:22.790 --> 00:21:23.990 align:middle line:90% of a better match. 00:21:23.990 --> 00:21:27.790 align:middle line:90% And here's a bunch of examples. 00:21:27.790 --> 00:21:31.210 align:middle line:84% So the top row here shows you three original images. 00:21:31.210 --> 00:21:34.310 align:middle line:84% And the bottom row shows you synthetic textures 00:21:34.310 --> 00:21:36.950 align:middle line:84% that result from matching the statistics that are measured 00:21:36.950 --> 00:21:38.570 align:middle line:90% from the original image. 00:21:38.570 --> 00:21:43.430 align:middle line:84% And they look really pretty good. 00:21:43.430 --> 00:21:47.590 align:middle line:84% So you get this thing out that's got all these plus signs in it. 00:21:47.590 --> 00:21:50.350 align:middle line:84% You get a pretty respectable-looking brick wall 00:21:50.350 --> 00:21:51.255 align:middle line:90% texture. 00:21:51.255 --> 00:21:52.630 align:middle line:84% And if you look very closely, you 00:21:52.630 --> 00:21:57.850 align:middle line:84% can see that it's not completely faithful. 00:21:57.850 --> 00:22:00.850 align:middle line:84% But at a glance, they look really, really similar. 00:22:00.850 --> 00:22:04.710 align:middle line:90% 00:22:04.710 --> 00:22:07.070 align:middle line:90% Here's some others. 00:22:07.070 --> 00:22:11.710 align:middle line:84% And so in particular, you can see that the synthetic textures 00:22:11.710 --> 00:22:14.190 align:middle line:84% that you get out here, it has a lot more 00:22:14.190 --> 00:22:17.910 align:middle line:84% of these elongated contours that were missing 00:22:17.910 --> 00:22:19.430 align:middle line:90% from the original model. 00:22:19.430 --> 00:22:23.340 align:middle line:84% So that's one of the things that those correlations buy you. 00:22:23.340 --> 00:22:24.920 align:middle line:90% Again, it's not perfect. 00:22:24.920 --> 00:22:30.400 align:middle line:84% So the beans don't quite look like beans. 00:22:30.400 --> 00:22:32.620 align:middle line:84% But again, at a glance, it might look pretty good. 00:22:32.620 --> 00:22:36.440 align:middle line:90% 00:22:36.440 --> 00:22:41.000 align:middle line:84% And so these models, like one of the strengths of this approach 00:22:41.000 --> 00:22:43.940 align:middle line:84% is that you can do this on any sort of image 00:22:43.940 --> 00:22:47.600 align:middle line:90% and you'll get something out. 00:22:47.600 --> 00:22:51.280 align:middle line:84% And so you can ask, well, what would the texture representation 00:22:51.280 --> 00:22:52.700 align:middle line:90% be for a crowd of people? 00:22:52.700 --> 00:22:54.880 align:middle line:84% And it's like this, and that seems like it's 00:22:54.880 --> 00:22:57.560 align:middle line:90% kind of missing something. 00:22:57.560 --> 00:22:59.440 align:middle line:84% There's definitely cases where maybe it 00:22:59.440 --> 00:23:01.120 align:middle line:90% doesn't work quite as well. 00:23:01.120 --> 00:23:04.622 align:middle line:84% So you can see some of the remaining inadequacies. 00:23:04.622 --> 00:23:06.080 align:middle line:84% But it nonetheless kind of accounts 00:23:06.080 --> 00:23:10.380 align:middle line:84% for a lot of what we see when we look at these images. 00:23:10.380 --> 00:23:13.840 align:middle line:90% 00:23:13.840 --> 00:23:17.520 align:middle line:84% Here are some other examples where the center thing is 00:23:17.520 --> 00:23:19.680 align:middle line:84% like an actual photo of a texture, 00:23:19.680 --> 00:23:22.000 align:middle line:84% and then the surround is synthesized 00:23:22.000 --> 00:23:25.090 align:middle line:84% from the statistics of the original constrained 00:23:25.090 --> 00:23:27.610 align:middle line:90% to agree at the boundary. 00:23:27.610 --> 00:23:30.410 align:middle line:84% So you can get this endless New York apartment 00:23:30.410 --> 00:23:34.410 align:middle line:84% building on the right, and lots and lots 00:23:34.410 --> 00:23:39.170 align:middle line:84% and lots of bell peppers on the left. 00:23:39.170 --> 00:23:40.590 align:middle line:90% Any questions about this? 00:23:40.590 --> 00:23:46.970 align:middle line:90% 00:23:46.970 --> 00:23:47.770 align:middle line:90% Yeah. 00:23:47.770 --> 00:23:53.010 align:middle line:84% AUDIENCE: Is this publicly available software? 00:23:53.010 --> 00:23:53.910 align:middle line:90% JOSH MCDERMOTT: Yes. 00:23:53.910 --> 00:23:57.990 align:middle line:84% Although it's probably only available in MATLAB. 00:23:57.990 --> 00:23:58.490 align:middle line:90% Yeah. 00:23:58.490 --> 00:24:03.330 align:middle line:90% So this particular method was-- 00:24:03.330 --> 00:24:05.850 align:middle line:84% the paper came out, I think, in 1999. 00:24:05.850 --> 00:24:07.290 align:middle line:84% So yeah, there's a MATLAB toolbox 00:24:07.290 --> 00:24:09.870 align:middle line:84% that you can download and try this kind of stuff out. 00:24:09.870 --> 00:24:14.090 align:middle line:84% And I mean, there's more recent variants 00:24:14.090 --> 00:24:17.410 align:middle line:90% of this that do similar things. 00:24:17.410 --> 00:24:19.610 align:middle line:90% But yeah, you can try it out. 00:24:19.610 --> 00:24:23.490 align:middle line:84% And I should emphasize that all of this-- these kinds of things 00:24:23.490 --> 00:24:26.030 align:middle line:84% have gotten much easier to implement nowadays. 00:24:26.030 --> 00:24:29.450 align:middle line:84% So one of the things that deep learning has resulted in 00:24:29.450 --> 00:24:32.570 align:middle line:84% is the widespread availability of really pretty 00:24:32.570 --> 00:24:34.330 align:middle line:90% good optimization tools. 00:24:34.330 --> 00:24:36.430 align:middle line:84% So things like computing gradients, 00:24:36.430 --> 00:24:38.410 align:middle line:84% all that nowadays is kind of automated. 00:24:38.410 --> 00:24:40.410 align:middle line:84% It used to be you would actually have to do math 00:24:40.410 --> 00:24:41.452 align:middle line:90% if you wanted to do this. 00:24:41.452 --> 00:24:45.728 align:middle line:84% Now, you just let the computer do it for you. 00:24:45.728 --> 00:24:47.770 align:middle line:84% So yeah, you could probably code this up yourself 00:24:47.770 --> 00:24:52.690 align:middle line:84% without an enormous amount of effort. 00:24:52.690 --> 00:24:56.730 align:middle line:84% So the key ideas that I want you to take away from this 00:24:56.730 --> 00:24:59.410 align:middle line:84% is the idea that textures are believed 00:24:59.410 --> 00:25:01.050 align:middle line:84% to be represented with statistics 00:25:01.050 --> 00:25:03.410 align:middle line:84% that capture their average properties 00:25:03.410 --> 00:25:08.010 align:middle line:90% over a region of an image. 00:25:08.010 --> 00:25:10.450 align:middle line:84% These statistics could plausibly be 00:25:10.450 --> 00:25:14.050 align:middle line:84% computed by neurons that are pooling the responses of neurons 00:25:14.050 --> 00:25:16.530 align:middle line:84% that have smaller receptive fields, like, 00:25:16.530 --> 00:25:19.500 align:middle line:84% again, pooling over some region of space. 00:25:19.500 --> 00:25:24.580 align:middle line:84% And we can test theories of texture using synthesis. 00:25:24.580 --> 00:25:26.460 align:middle line:84% So that's kind of an important idea. 00:25:26.460 --> 00:25:30.060 align:middle line:84% So we started out the lecture kind of just asking, well, 00:25:30.060 --> 00:25:32.140 align:middle line:84% can we account for the perception 00:25:32.140 --> 00:25:33.860 align:middle line:90% of texture boundaries? 00:25:33.860 --> 00:25:36.123 align:middle line:84% And that turns out to not be that difficult to do, 00:25:36.123 --> 00:25:37.540 align:middle line:84% like pretty simple things actually 00:25:37.540 --> 00:25:39.060 align:middle line:90% work pretty well for that. 00:25:39.060 --> 00:25:40.540 align:middle line:84% But if you actually want to account 00:25:40.540 --> 00:25:42.580 align:middle line:84% for the appearance of things, the way 00:25:42.580 --> 00:25:47.220 align:middle line:84% that it looks, then having a more complicated representation 00:25:47.220 --> 00:25:48.600 align:middle line:90% seems to pay dividends. 00:25:48.600 --> 00:25:52.480 align:middle line:90% 00:25:52.480 --> 00:25:53.920 align:middle line:84% What questions you got about that? 00:25:53.920 --> 00:26:06.980 align:middle line:90% 00:26:06.980 --> 00:26:08.020 align:middle line:90% Yep? 00:26:08.020 --> 00:26:10.020 align:middle line:84% AUDIENCE: What exactly do algorithms 00:26:10.020 --> 00:26:13.540 align:middle line:84% like texture synthesis actually recreate 00:26:13.540 --> 00:26:14.980 align:middle line:90% what you would call texture? 00:26:14.980 --> 00:26:19.370 align:middle line:84% Like say, for all the artifacts, which is just the image itself. 00:26:19.370 --> 00:26:21.030 align:middle line:84% But if you take a picture of a person 00:26:21.030 --> 00:26:23.830 align:middle line:84% and you run a texture synthesis, how much of that 00:26:23.830 --> 00:26:27.150 align:middle line:84% is actually texture, and how much is it 00:26:27.150 --> 00:26:30.355 align:middle line:84% just like a worse version of the original picture? 00:26:30.355 --> 00:26:31.230 align:middle line:90% JOSH MCDERMOTT: Yeah. 00:26:31.230 --> 00:26:35.110 align:middle line:84% So that's a really good question, is-- 00:26:35.110 --> 00:26:36.990 align:middle line:84% I think what you're essentially asking 00:26:36.990 --> 00:26:41.630 align:middle line:84% is like when we generate the synthetic image, 00:26:41.630 --> 00:26:44.630 align:middle line:84% is it possible that we're essentially 00:26:44.630 --> 00:26:48.350 align:middle line:84% like recreating the original image? 00:26:48.350 --> 00:26:48.990 align:middle line:90% Yeah. 00:26:48.990 --> 00:26:50.190 align:middle line:84% So because essentially, like what's 00:26:50.190 --> 00:26:51.690 align:middle line:84% happening here is that you can think 00:26:51.690 --> 00:26:54.050 align:middle line:84% of this at an abstract level, is you have this image, 00:26:54.050 --> 00:26:57.012 align:middle line:84% you make a whole bunch of measurements of that. 00:26:57.012 --> 00:26:58.470 align:middle line:84% And now you're generating something 00:26:58.470 --> 00:27:00.450 align:middle line:84% that gives you the same values of that measurement. 00:27:00.450 --> 00:27:02.030 align:middle line:84% And you might imagine that if there 00:27:02.030 --> 00:27:04.110 align:middle line:84% was enough different measurements, 00:27:04.110 --> 00:27:09.430 align:middle line:84% that would essentially uniquely specify the original image. 00:27:09.430 --> 00:27:13.270 align:middle line:84% And I think it's true, that if there were enough measurements, 00:27:13.270 --> 00:27:15.430 align:middle line:90% that would happen. 00:27:15.430 --> 00:27:17.900 align:middle line:84% This is not really close to that regime. 00:27:17.900 --> 00:27:22.340 align:middle line:84% I mean, one way to see that is like these models-- 00:27:22.340 --> 00:27:27.160 align:middle line:84% I think this one probably has maybe 700 parameters, 700 00:27:27.160 --> 00:27:29.680 align:middle line:84% measurements, so there's 700 statistics. 00:27:29.680 --> 00:27:34.160 align:middle line:84% The pixel arrays here, they're probably 256 by 256. 00:27:34.160 --> 00:27:35.498 align:middle line:90% What's 256 times 256? 00:27:35.498 --> 00:27:36.040 align:middle line:90% I don't know. 00:27:36.040 --> 00:27:39.320 align:middle line:90% It's a lot more than 700. 00:27:39.320 --> 00:27:43.440 align:middle line:84% But you can also just see from inspection 00:27:43.440 --> 00:27:46.178 align:middle line:84% that you're not getting something that's 00:27:46.178 --> 00:27:47.220 align:middle line:90% the same as the original. 00:27:47.220 --> 00:27:50.040 align:middle line:84% The other thing that you could do to test this 00:27:50.040 --> 00:27:53.240 align:middle line:84% would be to do this procedure with different samples of noise. 00:27:53.240 --> 00:27:56.360 align:middle line:84% And different samples of noise will give you different images. 00:27:56.360 --> 00:27:58.920 align:middle line:84% So you can essentially generate an infinite number 00:27:58.920 --> 00:28:01.320 align:middle line:90% of these samples more or less. 00:28:01.320 --> 00:28:03.520 align:middle line:90% Yeah. 00:28:03.520 --> 00:28:04.700 align:middle line:90% That's a good question. 00:28:04.700 --> 00:28:05.740 align:middle line:90% Any other questions? 00:28:05.740 --> 00:28:13.756 align:middle line:90% 00:28:13.756 --> 00:28:19.240 align:middle line:84% So this is where we're going to improvise some audio demos using 00:28:19.240 --> 00:28:20.260 align:middle line:90% my microphone. 00:28:20.260 --> 00:28:25.080 align:middle line:84% So so far, we've been talking about visual textures, 00:28:25.080 --> 00:28:29.720 align:middle line:84% but textures also occur in sound. 00:28:29.720 --> 00:28:32.280 align:middle line:84% In sound, textures result from large numbers 00:28:32.280 --> 00:28:35.240 align:middle line:84% of acoustic events, and they include lots of things 00:28:35.240 --> 00:28:40.760 align:middle line:84% that you hear all the time, like rain and wind or birds 00:28:40.760 --> 00:28:41.573 align:middle line:90% in a forest-- 00:28:41.573 --> 00:28:42.240 align:middle line:90% [AUDIO PLAYBACK] 00:28:42.240 --> 00:28:45.544 align:middle line:90% [MULTIPLE BIRDS CHIRPING] 00:28:45.544 --> 00:28:48.376 align:middle line:90% 00:28:48.376 --> 00:28:49.040 align:middle line:90% [END PLAYBACK] 00:28:49.040 --> 00:28:50.540 align:middle line:84% JOSH MCDERMOTT: --or running water-- 00:28:50.540 --> 00:28:51.207 align:middle line:90% [AUDIO PLAYBACK] 00:28:51.207 --> 00:28:54.498 align:middle line:90% [WATER GURGLING] 00:28:54.498 --> 00:28:58.857 align:middle line:90% 00:28:58.857 --> 00:28:59.440 align:middle line:90% [END PLAYBACK] 00:28:59.440 --> 00:29:01.093 align:middle line:90% JOSH MCDERMOTT: --or insects. 00:29:01.093 --> 00:29:01.760 align:middle line:90% [AUDIO PLAYBACK] 00:29:01.760 --> 00:29:04.694 align:middle line:90% [MULTIPLE INSECTS CHITTERING] 00:29:04.694 --> 00:29:07.139 align:middle line:90% 00:29:07.139 --> 00:29:08.120 align:middle line:90% [END PLAYBACK] 00:29:08.120 --> 00:29:10.564 align:middle line:84% JOSH MCDERMOTT: Crowd noise or applause-- 00:29:10.564 --> 00:29:11.231 align:middle line:90% [AUDIO PLAYBACK] 00:29:11.231 --> 00:29:13.586 align:middle line:90% [APPLAUSE] 00:29:13.586 --> 00:29:14.530 align:middle line:90% [END PLAYBACK] 00:29:14.530 --> 00:29:17.130 align:middle line:90% JOSH MCDERMOTT: --or fire. 00:29:17.130 --> 00:29:19.750 align:middle line:84% So these things are all over the place in the real world, 00:29:19.750 --> 00:29:23.210 align:middle line:90% just as they are for vision. 00:29:23.210 --> 00:29:27.250 align:middle line:84% And like what we were talking about when we discussed 00:29:27.250 --> 00:29:31.010 align:middle line:84% the sense of hearing in the first part of the class often 00:29:31.010 --> 00:29:33.590 align:middle line:84% really had to do with individual events. 00:29:33.590 --> 00:29:39.450 align:middle line:84% So these are sound waveforms of like two individual events. 00:29:39.450 --> 00:29:40.790 align:middle line:90% I can't remember what this is. 00:29:40.790 --> 00:29:41.290 align:middle line:90% [AUDIO PLAYBACK] 00:29:41.290 --> 00:29:41.650 align:middle line:90% [SQUEAKING] 00:29:41.650 --> 00:29:42.010 align:middle line:90% [END PLAYBACK] 00:29:42.010 --> 00:29:43.710 align:middle line:84% JOSH MCDERMOTT: So that's a baby toy. 00:29:43.710 --> 00:29:46.650 align:middle line:84% So these sounds, like they have a beginning 00:29:46.650 --> 00:29:49.070 align:middle line:84% and they have an end, and they have a temporal trajectory, 00:29:49.070 --> 00:29:51.670 align:middle line:84% and that's what makes them what they are in some sense. 00:29:51.670 --> 00:29:57.207 align:middle line:84% By comparison, if we just consider the sound of rain-- 00:29:57.207 --> 00:29:57.874 align:middle line:90% [AUDIO PLAYBACK] 00:29:57.874 --> 00:30:01.318 align:middle line:90% [RAIN FALLING] 00:30:01.318 --> 00:30:02.302 align:middle line:90% 00:30:02.302 --> 00:30:03.290 align:middle line:90% [END PLAYBACK] 00:30:03.290 --> 00:30:05.690 align:middle line:84% JOSH MCDERMOTT: I mean, at some point the rain started, 00:30:05.690 --> 00:30:07.090 align:middle line:84% and hopefully, at some point, it will end. 00:30:07.090 --> 00:30:08.290 align:middle line:84% But the beginning and the end, they're 00:30:08.290 --> 00:30:10.050 align:middle line:84% not really what make it sound like rain. 00:30:10.050 --> 00:30:12.092 align:middle line:84% The qualities that make it sound like rain there, 00:30:12.092 --> 00:30:14.580 align:middle line:90% they're just there. 00:30:14.580 --> 00:30:18.860 align:middle line:84% And so in this respect, I mean, textures 00:30:18.860 --> 00:30:20.480 align:middle line:90% are what we call stationary. 00:30:20.480 --> 00:30:23.920 align:middle line:84% So the essential properties don't change over time. 00:30:23.920 --> 00:30:26.652 align:middle line:90% 00:30:26.652 --> 00:30:29.100 align:middle line:84% And so you can think of sound textures 00:30:29.100 --> 00:30:32.907 align:middle line:84% as the time analog of image textures. 00:30:32.907 --> 00:30:34.740 align:middle line:84% In the sense that we think of image textures 00:30:34.740 --> 00:30:39.197 align:middle line:84% as being defined by statistical properties over space, 00:30:39.197 --> 00:30:40.780 align:middle line:84% sound textures we think of are defined 00:30:40.780 --> 00:30:43.520 align:middle line:84% by statistical properties over time. 00:30:43.520 --> 00:30:46.980 align:middle line:90% 00:30:46.980 --> 00:30:51.300 align:middle line:84% So just to be concrete about what we want to understand here, 00:30:51.300 --> 00:30:53.520 align:middle line:90% here's another texture. 00:30:53.520 --> 00:30:54.478 align:middle line:90% [AUDIO PLAYBACK] 00:30:54.478 --> 00:30:56.768 align:middle line:90% [CHATTER] 00:30:56.768 --> 00:30:58.775 align:middle line:90% [END PLAYBACK] 00:30:58.775 --> 00:31:00.400 align:middle line:84% JOSH MCDERMOTT: And here's another one. 00:31:00.400 --> 00:31:01.067 align:middle line:90% [AUDIO PLAYBACK] 00:31:01.067 --> 00:31:03.357 align:middle line:90% [CHATTER] 00:31:03.357 --> 00:31:03.940 align:middle line:90% [END PLAYBACK] 00:31:03.940 --> 00:31:04.400 align:middle line:90% JOSH MCDERMOTT: All right. 00:31:04.400 --> 00:31:05.897 align:middle line:84% And it's probably trivially obvious 00:31:05.897 --> 00:31:07.480 align:middle line:84% that those are the same kind of thing. 00:31:07.480 --> 00:31:09.022 align:middle line:84% Those are just two different excerpts 00:31:09.022 --> 00:31:12.620 align:middle line:84% of the same recording of a room full of people talking. 00:31:12.620 --> 00:31:15.260 align:middle line:84% But the sound waveforms that correspond to those excerpts 00:31:15.260 --> 00:31:16.540 align:middle line:90% are totally different. 00:31:16.540 --> 00:31:20.860 align:middle line:84% So there's some property that those sounds share that you 00:31:20.860 --> 00:31:26.470 align:middle line:84% immediately extract and that causes you to recognize 00:31:26.470 --> 00:31:27.720 align:middle line:90% that those are the same thing. 00:31:27.720 --> 00:31:30.260 align:middle line:84% So we'd like to know what that is 00:31:30.260 --> 00:31:32.553 align:middle line:84% and what would be different about, say-- 00:31:32.553 --> 00:31:33.220 align:middle line:90% [AUDIO PLAYBACK] 00:31:33.220 --> 00:31:35.140 align:middle line:90% [BUZZING] 00:31:35.140 --> 00:31:36.100 align:middle line:90% [END PLAYBACK] 00:31:36.100 --> 00:31:38.600 align:middle line:84% JOSH MCDERMOTT: --which is a very different kind of texture. 00:31:38.600 --> 00:31:40.660 align:middle line:84% So what is it that you store and-- 00:31:40.660 --> 00:31:43.780 align:middle line:84% what is it that you extract and store about these waveforms 00:31:43.780 --> 00:31:47.440 align:middle line:84% to recognize that they're the same kind of thing? 00:31:47.440 --> 00:31:50.780 align:middle line:84% So the key theoretical proposal, and this is echoing what we just 00:31:50.780 --> 00:31:55.660 align:middle line:84% talked about with images, is that because they're stationary, 00:31:55.660 --> 00:31:58.580 align:middle line:84% textures can be captured by statistics, that, in this case, 00:31:58.580 --> 00:32:00.680 align:middle line:84% are time averages of acoustic measurements. 00:32:00.680 --> 00:32:04.140 align:middle line:84% And so the proposal is that when you recognize the sound of fire 00:32:04.140 --> 00:32:08.810 align:middle line:84% or the sound of rain, you're recognizing these statistics. 00:32:08.810 --> 00:32:11.670 align:middle line:84% So what kinds of statistics might we be measuring? 00:32:11.670 --> 00:32:14.630 align:middle line:84% Well, it seems pretty plausible that whatever statistics 00:32:14.630 --> 00:32:18.110 align:middle line:84% the auditory system measures are derived from auditory system 00:32:18.110 --> 00:32:18.970 align:middle line:90% representations. 00:32:18.970 --> 00:32:20.805 align:middle line:84% And this is one pretty standard model 00:32:20.805 --> 00:32:22.430 align:middle line:84% of the front end of the auditory system 00:32:22.430 --> 00:32:25.810 align:middle line:84% that you probably recall from a month and a half ago or so. 00:32:25.810 --> 00:32:27.510 align:middle line:90% So we've got a sound signal. 00:32:27.510 --> 00:32:29.190 align:middle line:90% That's the input to the system. 00:32:29.190 --> 00:32:30.830 align:middle line:84% There's a bank of bandpass filters 00:32:30.830 --> 00:32:34.230 align:middle line:84% that mimics the frequency selectivity of the cochlea. 00:32:34.230 --> 00:32:37.190 align:middle line:84% We then often think that there's a process of extracting 00:32:37.190 --> 00:32:38.370 align:middle line:90% the amplitude envelope. 00:32:38.370 --> 00:32:41.030 align:middle line:84% There's compression that happens in the cochlea. 00:32:41.030 --> 00:32:44.390 align:middle line:84% And then there's often thought to be a second set of filters 00:32:44.390 --> 00:32:47.110 align:middle line:84% called modulation filters that operates 00:32:47.110 --> 00:32:49.950 align:middle line:84% on the envelope of the cochlear filters. 00:32:49.950 --> 00:32:55.150 align:middle line:84% And the output of that are what you might call modulation bands. 00:32:55.150 --> 00:32:56.710 align:middle line:84% So one idea is like, well, maybe we 00:32:56.710 --> 00:33:00.270 align:middle line:84% can take this pretty standard model 00:33:00.270 --> 00:33:02.470 align:middle line:84% of the front end of the auditory system 00:33:02.470 --> 00:33:05.950 align:middle line:84% and measure some statistics from those representations. 00:33:05.950 --> 00:33:12.200 align:middle line:84% And so here, again, the M stands for Marginal statistics, 00:33:12.200 --> 00:33:13.960 align:middle line:84% and particular moments, so things 00:33:13.960 --> 00:33:16.040 align:middle line:90% like the mean and the variance. 00:33:16.040 --> 00:33:17.900 align:middle line:90% The C stands for Correlations. 00:33:17.900 --> 00:33:21.600 align:middle line:90% 00:33:21.600 --> 00:33:25.000 align:middle line:84% So if these kinds of statistics are 00:33:25.000 --> 00:33:29.152 align:middle line:84% going to be useful for recognizing sound textures, 00:33:29.152 --> 00:33:31.360 align:middle line:84% minimally, they kind of need to give different values 00:33:31.360 --> 00:33:32.600 align:middle line:90% for different sounds. 00:33:32.600 --> 00:33:34.920 align:middle line:84% And so we can just take a look at a few of these 00:33:34.920 --> 00:33:38.160 align:middle line:84% and get a sense of what they might capture. 00:33:38.160 --> 00:33:40.520 align:middle line:84% So let's take a look at what you might 00:33:40.520 --> 00:33:45.680 align:middle line:84% get from looking at the marginal statistics 00:33:45.680 --> 00:33:49.880 align:middle line:84% in this representation, so the envelopes of cochlear frequency 00:33:49.880 --> 00:33:51.480 align:middle line:90% channels. 00:33:51.480 --> 00:33:53.700 align:middle line:84% So this is what we're dealing with here. 00:33:53.700 --> 00:33:56.720 align:middle line:84% So this is what we often call a cochleagram. 00:33:56.720 --> 00:33:58.480 align:middle line:84% So the y-axis here, you can think 00:33:58.480 --> 00:34:02.000 align:middle line:84% of as the cochlear channel, or the place along the cochlea. 00:34:02.000 --> 00:34:06.040 align:middle line:84% So remember, cochlea is tuned for frequency. 00:34:06.040 --> 00:34:10.540 align:middle line:84% What frequencies are transduced near the base of the cochlea? 00:34:10.540 --> 00:34:15.670 align:middle line:90% 00:34:15.670 --> 00:34:16.364 align:middle line:90% AUDIENCE: High. 00:34:16.364 --> 00:34:17.239 align:middle line:90% JOSH MCDERMOTT: High. 00:34:17.239 --> 00:34:17.780 align:middle line:90% Yes. 00:34:17.780 --> 00:34:18.679 align:middle line:90% Good. 00:34:18.679 --> 00:34:21.760 align:middle line:84% What frequencies are transduced near the apex? 00:34:21.760 --> 00:34:22.679 align:middle line:90% Low. 00:34:22.679 --> 00:34:23.280 align:middle line:90% OK. 00:34:23.280 --> 00:34:23.780 align:middle line:90% All right. 00:34:23.780 --> 00:34:27.360 align:middle line:84% So we've got cochlea laid out there. 00:34:27.360 --> 00:34:29.199 align:middle line:90% Time is on the x-axis. 00:34:29.199 --> 00:34:33.239 align:middle line:84% The gray level represents the firing rate or the amplitude 00:34:33.239 --> 00:34:35.880 align:middle line:90% in a frequency channel. 00:34:35.880 --> 00:34:39.120 align:middle line:84% If we just take a horizontal slice through the cochleagram, 00:34:39.120 --> 00:34:42.900 align:middle line:84% we got the blue curve here, which is like a subband. 00:34:42.900 --> 00:34:44.940 align:middle line:84% So that's the response of the filter. 00:34:44.940 --> 00:34:49.659 align:middle line:84% And then the red curve here is the envelope of the filter. 00:34:49.659 --> 00:34:54.000 align:middle line:84% And so the marginal statistics are 00:34:54.000 --> 00:34:57.680 align:middle line:84% going to capture properties of the marginal distribution 00:34:57.680 --> 00:34:58.460 align:middle line:90% of the envelope. 00:34:58.460 --> 00:34:59.960 align:middle line:84% The marginal distribution, remember, 00:34:59.960 --> 00:35:02.960 align:middle line:84% that's what you get if you take all of these red values, 00:35:02.960 --> 00:35:05.010 align:middle line:84% you throw them in a bag, and then 00:35:05.010 --> 00:35:06.595 align:middle line:90% you plot their distribution. 00:35:06.595 --> 00:35:07.970 align:middle line:84% So this is plotting the frequency 00:35:07.970 --> 00:35:10.610 align:middle line:84% of occurrence of different amplitudes here 00:35:10.610 --> 00:35:13.270 align:middle line:84% of sound energy in this particular frequency band. 00:35:13.270 --> 00:35:14.930 align:middle line:90% And so what does this tell us? 00:35:14.930 --> 00:35:19.570 align:middle line:84% Well, it says that on average, the amplitude 00:35:19.570 --> 00:35:22.690 align:middle line:84% is in this particular vicinity, and sometimes it's lower, 00:35:22.690 --> 00:35:24.370 align:middle line:90% and sometimes it's higher. 00:35:24.370 --> 00:35:26.330 align:middle line:84% So that may not seem super exciting, 00:35:26.330 --> 00:35:28.530 align:middle line:84% but if you look at these distributions 00:35:28.530 --> 00:35:30.950 align:middle line:84% for different sounds, you see that they look different. 00:35:30.950 --> 00:35:34.970 align:middle line:84% So here, we have the histogram of an envelope 00:35:34.970 --> 00:35:39.850 align:middle line:84% for one particular frequency channel at 220 hertz for three 00:35:39.850 --> 00:35:41.770 align:middle line:90% different sounds. 00:35:41.770 --> 00:35:44.970 align:middle line:90% So the red curve is for noise. 00:35:44.970 --> 00:35:46.970 align:middle line:90% The blue curve is for a stream. 00:35:46.970 --> 00:35:50.393 align:middle line:84% And the green curve is for geese like honking 00:35:50.393 --> 00:35:51.310 align:middle line:90% or-- what do geese do? 00:35:51.310 --> 00:35:52.370 align:middle line:90% They quack? 00:35:52.370 --> 00:35:53.210 align:middle line:90% Honk? 00:35:53.210 --> 00:35:53.370 align:middle line:90% AUDIENCE: Honk. 00:35:53.370 --> 00:35:54.703 align:middle line:90% JOSH MCDERMOTT: Yeah, they honk. 00:35:54.703 --> 00:36:01.210 align:middle line:84% So that's geese honking And so these particular examples, 00:36:01.210 --> 00:36:03.900 align:middle line:84% they were chosen so that they would have 00:36:03.900 --> 00:36:05.520 align:middle line:90% about the same average value. 00:36:05.520 --> 00:36:07.720 align:middle line:84% That's the little vertical line segments at the top. 00:36:07.720 --> 00:36:09.660 align:middle line:84% But you can see that the distributions 00:36:09.660 --> 00:36:10.820 align:middle line:90% have different shapes. 00:36:10.820 --> 00:36:14.220 align:middle line:84% So in particular, the natural sounds 00:36:14.220 --> 00:36:17.540 align:middle line:84% like the stream and the geese, the distributions are wider, 00:36:17.540 --> 00:36:19.500 align:middle line:90% so they have higher variance. 00:36:19.500 --> 00:36:22.440 align:middle line:84% And they also-- they vary in how skewed they are. 00:36:22.440 --> 00:36:25.900 align:middle line:84% So the green one is very positively skewed. 00:36:25.900 --> 00:36:29.260 align:middle line:84% And so if you look at the cochleagrams of these sounds, 00:36:29.260 --> 00:36:32.140 align:middle line:84% you can see where these distributions are coming from. 00:36:32.140 --> 00:36:35.220 align:middle line:84% So if you look at the distribution for pink noise-- 00:36:35.220 --> 00:36:37.740 align:middle line:84% sorry, not the distribution, the cochleagram. 00:36:37.740 --> 00:36:40.747 align:middle line:90% The cochleagram is pretty gray. 00:36:40.747 --> 00:36:42.580 align:middle line:84% So that means it doesn't deviate a whole lot 00:36:42.580 --> 00:36:45.220 align:middle line:90% from the average values. 00:36:45.220 --> 00:36:48.380 align:middle line:84% Whereas the stream, you can see more light and dark gray, 00:36:48.380 --> 00:36:50.040 align:middle line:90% fluctuates more in amplitude. 00:36:50.040 --> 00:36:52.700 align:middle line:84% And with the geese, there's a fair bit of white and black. 00:36:52.700 --> 00:36:57.110 align:middle line:84% So it's varying quite a lot in amplitude. 00:36:57.110 --> 00:36:58.940 align:middle line:84% And so in this particular case, this 00:36:58.940 --> 00:37:02.590 align:middle line:84% is an indication of the fact that a lot of natural signals 00:37:02.590 --> 00:37:05.230 align:middle line:90% are sparser than noise. 00:37:05.230 --> 00:37:08.310 align:middle line:84% And the intuition here is that natural sounds, 00:37:08.310 --> 00:37:12.550 align:middle line:84% they contain events, like raindrops or geese calls, 00:37:12.550 --> 00:37:15.210 align:middle line:84% and these events are infrequent, or relatively infrequent, 00:37:15.210 --> 00:37:17.750 align:middle line:84% but when they occur, they produce large amplitudes. 00:37:17.750 --> 00:37:21.190 align:middle line:84% So you get these pretty big swings 00:37:21.190 --> 00:37:23.030 align:middle line:90% in the amplitude of the signal. 00:37:23.030 --> 00:37:26.670 align:middle line:84% And critically-- so that shows up here 00:37:26.670 --> 00:37:30.710 align:middle line:84% in these relatively simple statistical properties 00:37:30.710 --> 00:37:33.590 align:middle line:84% of these distributions like the variance and the skew, 00:37:33.590 --> 00:37:37.010 align:middle line:90% which are examples of moments. 00:37:37.010 --> 00:37:39.048 align:middle line:84% So the point is that we could measure 00:37:39.048 --> 00:37:41.590 align:middle line:84% some fairly simple things right, just variance, for instance, 00:37:41.590 --> 00:37:47.008 align:middle line:84% or skew, and differentiate between these different sounds. 00:37:47.008 --> 00:37:48.550 align:middle line:84% Let's take a look at what information 00:37:48.550 --> 00:37:51.110 align:middle line:90% you might get from correlations. 00:37:51.110 --> 00:37:54.030 align:middle line:84% So we talked a little bit about how correlations in images 00:37:54.030 --> 00:37:57.470 align:middle line:84% might kind of capture like elongated edges. 00:37:57.470 --> 00:37:58.570 align:middle line:90% What about sound? 00:37:58.570 --> 00:38:03.070 align:middle line:84% So this is a cochleagram for fire. 00:38:03.070 --> 00:38:06.923 align:middle line:84% So fire has lots of crackles and pops. 00:38:06.923 --> 00:38:08.090 align:middle line:90% This is what it sounds like. 00:38:08.090 --> 00:38:08.757 align:middle line:90% [AUDIO PLAYBACK] 00:38:08.757 --> 00:38:11.810 align:middle line:90% [FIRE CRACKLING] 00:38:11.810 --> 00:38:12.607 align:middle line:90% 00:38:12.607 --> 00:38:13.190 align:middle line:90% [END PLAYBACK] 00:38:13.190 --> 00:38:14.273 align:middle line:90% JOSH MCDERMOTT: All right. 00:38:14.273 --> 00:38:16.590 align:middle line:90% Very crackly. 00:38:16.590 --> 00:38:20.790 align:middle line:84% So those crackles and pops are broadband events. 00:38:20.790 --> 00:38:22.290 align:middle line:90% So they're like impulsive. 00:38:22.290 --> 00:38:24.770 align:middle line:84% So they contain energy at all frequencies. 00:38:24.770 --> 00:38:26.470 align:middle line:84% And so they show up in the cochleagram 00:38:26.470 --> 00:38:28.510 align:middle line:90% as these vertical streaks. 00:38:28.510 --> 00:38:32.670 align:middle line:84% And those vertical streaks induce 00:38:32.670 --> 00:38:35.410 align:middle line:84% dependencies between different frequency channels. 00:38:35.410 --> 00:38:39.070 align:middle line:84% So they mean that the amplitude at one frequency 00:38:39.070 --> 00:38:41.150 align:middle line:84% is going to be correlated with the amplitude 00:38:41.150 --> 00:38:42.830 align:middle line:90% at another frequency. 00:38:42.830 --> 00:38:47.970 align:middle line:84% And so this is a matrix of all those correlation coefficients. 00:38:47.970 --> 00:38:51.630 align:middle line:84% So the axes here are the cochlear channels. 00:38:51.630 --> 00:38:55.483 align:middle line:84% And so the diagonal here has got to be 1, 00:38:55.483 --> 00:38:57.400 align:middle line:84% but the off-diagonals can be kind of whatever. 00:38:57.400 --> 00:39:00.240 align:middle line:84% But you can see that for fire, especially kind of either 00:39:00.240 --> 00:39:03.640 align:middle line:84% at the very low frequencies or the relatively high frequencies, 00:39:03.640 --> 00:39:04.720 align:middle line:90% there's a lot of red. 00:39:04.720 --> 00:39:06.700 align:middle line:84% So things are pretty strongly correlated. 00:39:06.700 --> 00:39:09.646 align:middle line:90% 00:39:09.646 --> 00:39:13.760 align:middle line:84% And this is not true of all sounds. 00:39:13.760 --> 00:39:16.840 align:middle line:84% So in particular, a lot of water sounds 00:39:16.840 --> 00:39:18.580 align:middle line:84% are actually pretty not correlated. 00:39:18.580 --> 00:39:20.320 align:middle line:90% So here is a stream. 00:39:20.320 --> 00:39:21.980 align:middle line:84% I don't have the audio demo for that, 00:39:21.980 --> 00:39:23.420 align:middle line:90% but I played you that earlier. 00:39:23.420 --> 00:39:25.000 align:middle line:84% So this is a stream, and you can see 00:39:25.000 --> 00:39:26.660 align:middle line:90% that it's not quite as streaky. 00:39:26.660 --> 00:39:29.600 align:middle line:84% And then if you look at the correlation matrix for a stream, 00:39:29.600 --> 00:39:31.840 align:middle line:84% you can see that things are more green, 00:39:31.840 --> 00:39:35.940 align:middle line:84% which corresponds to something pretty close to zero. 00:39:35.940 --> 00:39:37.840 align:middle line:84% So the point is that this correlation 00:39:37.840 --> 00:39:39.920 align:middle line:84% that you could measure between different frequency channels 00:39:39.920 --> 00:39:42.462 align:middle line:84% has got very different values depending on what kind of sound 00:39:42.462 --> 00:39:43.822 align:middle line:90% that you listen to. 00:39:43.822 --> 00:39:45.280 align:middle line:84% So things that are more crackly are 00:39:45.280 --> 00:39:48.480 align:middle line:90% going to have more correlation. 00:39:48.480 --> 00:39:51.840 align:middle line:84% Finally, just one more example here. 00:39:51.840 --> 00:39:54.120 align:middle line:84% Let's look at what kind of information 00:39:54.120 --> 00:39:58.050 align:middle line:84% you might get from measuring a correlation between the outputs 00:39:58.050 --> 00:39:59.550 align:middle line:90% of these modulation filters. 00:39:59.550 --> 00:40:02.410 align:middle line:84% So this is kind of one stage deeper in the auditory system. 00:40:02.410 --> 00:40:05.430 align:middle line:90% 00:40:05.430 --> 00:40:08.930 align:middle line:84% So these correlations are going to be measured 00:40:08.930 --> 00:40:14.730 align:middle line:84% between a particular modulation filter applied 00:40:14.730 --> 00:40:17.270 align:middle line:90% to different frequency channels. 00:40:17.270 --> 00:40:20.290 align:middle line:90% So this guy, and this guy. 00:40:20.290 --> 00:40:23.005 align:middle line:84% And so you get these correlation matrices, 00:40:23.005 --> 00:40:24.630 align:middle line:84% but now you have a whole bunch of them. 00:40:24.630 --> 00:40:28.970 align:middle line:84% Each one kind of corresponds to a particular modulation filter. 00:40:28.970 --> 00:40:31.370 align:middle line:84% And the main thing to take away from this is that here, 00:40:31.370 --> 00:40:35.130 align:middle line:84% these statistics are plotted for waves-- that's on the left-- 00:40:35.130 --> 00:40:37.462 align:middle line:90% and fire on the right. 00:40:37.462 --> 00:40:39.170 align:middle line:84% And the main thing to take away from this 00:40:39.170 --> 00:40:40.337 align:middle line:90% is that they look different. 00:40:40.337 --> 00:40:45.210 align:middle line:84% In particular, fire has got a lot more of these correlations 00:40:45.210 --> 00:40:48.590 align:middle line:84% at fast modulation rates, like 50 hertz and 100 hertz, 00:40:48.590 --> 00:40:50.030 align:middle line:90% whereas waves basically doesn't. 00:40:50.030 --> 00:40:52.770 align:middle line:84% It's only got them at the slow modulations. 00:40:52.770 --> 00:40:57.490 align:middle line:84% So this is just a statistical acoustic fact about these types 00:40:57.490 --> 00:40:58.710 align:middle line:90% of natural sounds. 00:40:58.710 --> 00:41:03.430 align:middle line:90% 00:41:03.430 --> 00:41:07.250 align:middle line:84% So the big picture here is that this little tour 00:41:07.250 --> 00:41:09.650 align:middle line:84% of these statistics that we can measure 00:41:09.650 --> 00:41:12.970 align:middle line:84% from our model of the auditory system 00:41:12.970 --> 00:41:14.850 align:middle line:90% capture variation across sound. 00:41:14.850 --> 00:41:18.390 align:middle line:84% They got different values for different types of sounds. 00:41:18.390 --> 00:41:20.450 align:middle line:84% So now what we're going to ask here 00:41:20.450 --> 00:41:23.010 align:middle line:84% is whether they can account for the perception 00:41:23.010 --> 00:41:24.730 align:middle line:90% of real-world textures. 00:41:24.730 --> 00:41:26.490 align:middle line:84% And so our key methodological proposal, 00:41:26.490 --> 00:41:27.930 align:middle line:84% again echoing what we kind of just 00:41:27.930 --> 00:41:29.850 align:middle line:84% talked about for visual texture, is 00:41:29.850 --> 00:41:34.810 align:middle line:84% that synthesis is a powerful way to test a perceptual theory. 00:41:34.810 --> 00:41:38.290 align:middle line:84% And so the logic here is that if your brain represents sounds 00:41:38.290 --> 00:41:40.930 align:middle line:84% with a set of measurements, then signals 00:41:40.930 --> 00:41:43.330 align:middle line:84% that have the same values of those measurements 00:41:43.330 --> 00:41:46.090 align:middle line:90% ought to sound the same to us. 00:41:46.090 --> 00:41:50.250 align:middle line:84% And so in particular, sounds that we synthesize 00:41:50.250 --> 00:41:53.100 align:middle line:84% to have the same measurements as some particular real-world 00:41:53.100 --> 00:41:56.320 align:middle line:84% recording ought to sound like that real-world recording 00:41:56.320 --> 00:41:59.580 align:middle line:84% if the measurements that we use to do the synthesis 00:41:59.580 --> 00:42:05.040 align:middle line:84% are like the ones that the brain is using to represent sound. 00:42:05.040 --> 00:42:08.380 align:middle line:84% So I'm going to walk you through a really simple example, just 00:42:08.380 --> 00:42:11.280 align:middle line:84% to make sure everybody's on board with the logic here. 00:42:11.280 --> 00:42:14.940 align:middle line:84% So let's suppose that we have the world's simplest 00:42:14.940 --> 00:42:16.900 align:middle line:84% theory of texture perception, which 00:42:16.900 --> 00:42:21.300 align:middle line:84% is that our representation of texture consists 00:42:21.300 --> 00:42:24.460 align:middle line:84% of something that approximates the power spectrum. 00:42:24.460 --> 00:42:26.943 align:middle line:84% So the power spectrum is a super standard thing 00:42:26.943 --> 00:42:27.860 align:middle line:90% to measure with sound. 00:42:27.860 --> 00:42:29.402 align:middle line:84% On some of your earlier problem sets, 00:42:29.402 --> 00:42:31.740 align:middle line:84% you actually plotted the power spectrum. 00:42:31.740 --> 00:42:33.980 align:middle line:84% In this model, the power spectrum 00:42:33.980 --> 00:42:39.780 align:middle line:84% is approximated by the mean of each cochlear envelope. 00:42:39.780 --> 00:42:42.580 align:middle line:84% So remember, we've got our bank of cochlear filters. 00:42:42.580 --> 00:42:44.860 align:middle line:84% The envelope is the instantaneous amplitude. 00:42:44.860 --> 00:42:46.780 align:middle line:84% And so if you just average that over time, 00:42:46.780 --> 00:42:48.900 align:middle line:84% it's going to tell you about how much power 00:42:48.900 --> 00:42:51.030 align:middle line:84% is in that particular frequency channel. 00:42:51.030 --> 00:42:53.930 align:middle line:84% So let's suppose we have 30 cochlear filters. 00:42:53.930 --> 00:42:58.610 align:middle line:84% We're going to get 30 numbers, one for each frequency channel. 00:42:58.610 --> 00:43:02.270 align:middle line:84% So the procedure here, in order to test this theory, is first, 00:43:02.270 --> 00:43:04.495 align:middle line:84% we're going to measure this quantity, 00:43:04.495 --> 00:43:05.870 align:middle line:84% this proposed representation-- so 00:43:05.870 --> 00:43:07.990 align:middle line:84% in this case, the average value of each envelope 00:43:07.990 --> 00:43:11.870 align:middle line:84% in a real-world texture like a recording of water. 00:43:11.870 --> 00:43:14.270 align:middle line:84% We're then going to synthesize a random signal 00:43:14.270 --> 00:43:19.850 align:middle line:84% that's got the same values for those envelope means. 00:43:19.850 --> 00:43:21.350 align:middle line:84% And so in this particular case, this 00:43:21.350 --> 00:43:22.950 align:middle line:90% is actually pretty easy to do. 00:43:22.950 --> 00:43:24.870 align:middle line:84% So we're going to start with noise. 00:43:24.870 --> 00:43:28.310 align:middle line:84% We will pass the noise through a bank of bandpass filters 00:43:28.310 --> 00:43:31.510 align:middle line:84% that mimics the cochlea, so we get noise subbands. 00:43:31.510 --> 00:43:34.510 align:middle line:90% 00:43:34.510 --> 00:43:37.170 align:middle line:84% We're then going to just rescale each one of those 00:43:37.170 --> 00:43:40.350 align:middle line:84% so that it's got the correct average power. 00:43:40.350 --> 00:43:44.690 align:middle line:84% And then we can add those back up, and we can get a new signal 00:43:44.690 --> 00:43:48.158 align:middle line:84% And there's a few little details of the procedure 00:43:48.158 --> 00:43:50.450 align:middle line:84% for doing this to do it correctly that I'm leaving out. 00:43:50.450 --> 00:43:52.430 align:middle line:84% You actually need to do this iteratively, 00:43:52.430 --> 00:43:55.390 align:middle line:84% but it's still very, very simple. 00:43:55.390 --> 00:43:57.870 align:middle line:84% So now what we want to do is listen to the sounds, 00:43:57.870 --> 00:44:01.050 align:middle line:84% and just ask ourselves whether they sound like the real thing. 00:44:01.050 --> 00:44:04.030 align:middle line:84% So I'm going to play you some synthetic sounds that 00:44:04.030 --> 00:44:07.310 align:middle line:84% are generated from noise just by matching this proposed 00:44:07.310 --> 00:44:11.490 align:middle line:84% representation to that of each of the originals. 00:44:11.490 --> 00:44:13.218 align:middle line:90% So here's what they sound like. 00:44:13.218 --> 00:44:13.885 align:middle line:90% [AUDIO PLAYBACK] 00:44:13.885 --> 00:44:15.282 align:middle line:90% [MID-FREQUENCY STATIC] 00:44:15.282 --> 00:44:15.865 align:middle line:90% [END PLAYBACK] 00:44:15.865 --> 00:44:16.532 align:middle line:90% [AUDIO PLAYBACK] 00:44:16.532 --> 00:44:19.242 align:middle line:90% [LOWER-FREQUENCY STATIC] 00:44:19.242 --> 00:44:19.825 align:middle line:90% [END PLAYBACK] 00:44:19.825 --> 00:44:20.492 align:middle line:90% [AUDIO PLAYBACK] 00:44:20.492 --> 00:44:21.805 align:middle line:90% [LOW-FREQUENCY STATIC] 00:44:21.805 --> 00:44:22.623 align:middle line:90% [END PLAYBACK] 00:44:22.623 --> 00:44:23.290 align:middle line:90% [AUDIO PLAYBACK] 00:44:23.290 --> 00:44:24.687 align:middle line:90% [VERY LOW-FREQUENCY STATIC] 00:44:24.687 --> 00:44:25.270 align:middle line:90% [END PLAYBACK] 00:44:25.270 --> 00:44:25.937 align:middle line:90% [AUDIO PLAYBACK] 00:44:25.937 --> 00:44:27.745 align:middle line:90% [MID-FREQUENCY STATIC] 00:44:27.745 --> 00:44:29.730 align:middle line:90% [END PLAYBACK] 00:44:29.730 --> 00:44:32.030 align:middle line:84% JOSH MCDERMOTT: Now each of those sounded different, 00:44:32.030 --> 00:44:34.590 align:middle line:84% and you may have even actually been 00:44:34.590 --> 00:44:37.110 align:middle line:84% able to convince yourself that, yeah, that maybe sounds 00:44:37.110 --> 00:44:40.822 align:middle line:90% a little bit like applause. 00:44:40.822 --> 00:44:44.163 align:middle line:84% But they don't sound like the real thing. 00:44:44.163 --> 00:44:44.830 align:middle line:90% [AUDIO PLAYBACK] 00:44:44.830 --> 00:44:47.593 align:middle line:90% [APPLAUSE] 00:44:47.593 --> 00:44:48.176 align:middle line:90% [END PLAYBACK] 00:44:48.176 --> 00:44:48.654 align:middle line:90% JOSH MCDERMOTT: Or-- 00:44:48.654 --> 00:44:49.321 align:middle line:90% [AUDIO PLAYBACK] 00:44:49.321 --> 00:44:51.044 align:middle line:90% [WATER GURGLING] 00:44:51.044 --> 00:44:53.320 align:middle line:90% [END PLAYBACK] 00:44:53.320 --> 00:44:56.040 align:middle line:84% JOSH MCDERMOTT: So we have ruled out the power spectrum theory 00:44:56.040 --> 00:44:58.920 align:middle line:90% of texture with this exercise. 00:44:58.920 --> 00:45:01.830 align:middle line:90% And now the question is-- 00:45:01.830 --> 00:45:04.080 align:middle line:84% so since this is not realistic, everything pretty much 00:45:04.080 --> 00:45:05.160 align:middle line:90% sounds like noise. 00:45:05.160 --> 00:45:09.080 align:middle line:84% And the question is, well, we can do better 00:45:09.080 --> 00:45:12.520 align:middle line:84% with some additional simple statistics? 00:45:12.520 --> 00:45:17.160 align:middle line:90% So more of these red circles. 00:45:17.160 --> 00:45:19.440 align:middle line:84% So just as kind of was the case when 00:45:19.440 --> 00:45:21.300 align:middle line:84% we were talking about image textures, 00:45:21.300 --> 00:45:23.960 align:middle line:84% so the fact that we're using this big set of statistics 00:45:23.960 --> 00:45:25.760 align:middle line:84% means that the synthesis process has 00:45:25.760 --> 00:45:28.320 align:middle line:84% to work a little bit differently. 00:45:28.320 --> 00:45:31.340 align:middle line:84% And it's, again, going to be gradient-based optimization. 00:45:31.340 --> 00:45:33.360 align:middle line:84% So there's this initial step where 00:45:33.360 --> 00:45:34.960 align:middle line:84% you take your original sound, you 00:45:34.960 --> 00:45:36.680 align:middle line:84% pass it through your auditory model 00:45:36.680 --> 00:45:39.080 align:middle line:90% and you measure its statistics. 00:45:39.080 --> 00:45:43.200 align:middle line:84% And then there's a synthesis procedure where you take a noise 00:45:43.200 --> 00:45:45.320 align:middle line:84% signal, you initially measure its statistics, 00:45:45.320 --> 00:45:47.110 align:middle line:84% and you get an error signal here. 00:45:47.110 --> 00:45:50.032 align:middle line:84% So this is the loss function you're trying to minimize, 00:45:50.032 --> 00:45:52.490 align:middle line:84% which is the difference between the statistics of the noise 00:45:52.490 --> 00:45:55.850 align:middle line:84% and the statistics of rain or fire or whatever it is. 00:45:55.850 --> 00:45:58.850 align:middle line:84% And then you're going to try to impose that 00:45:58.850 --> 00:46:02.690 align:middle line:84% with the gradient on the original thing 00:46:02.690 --> 00:46:06.980 align:middle line:84% And again, back when we originally did this stuff, 00:46:06.980 --> 00:46:09.230 align:middle line:84% this was a little bit more complicated than it is now, 00:46:09.230 --> 00:46:11.890 align:middle line:84% and it's pretty straightforward to do nowadays. 00:46:11.890 --> 00:46:14.690 align:middle line:84% And there is MATLAB code for this too if you're interested. 00:46:14.690 --> 00:46:18.990 align:middle line:84% Lots of MATLAB code from back in the day. 00:46:18.990 --> 00:46:21.850 align:middle line:84% So the result of this procedure is 00:46:21.850 --> 00:46:24.290 align:middle line:84% that you get a signal that shares the statistics 00:46:24.290 --> 00:46:25.470 align:middle line:90% of some real-world sound. 00:46:25.470 --> 00:46:29.668 align:middle line:84% And again, what we're interested in is what they sound like. 00:46:29.668 --> 00:46:32.210 align:middle line:84% And again, the reason that we care about what they sound like 00:46:32.210 --> 00:46:35.350 align:middle line:84% is that if the statistics account for texture perception, 00:46:35.350 --> 00:46:38.257 align:middle line:84% then the synthetic signals that we synthesize in this way, 00:46:38.257 --> 00:46:40.590 align:middle line:84% they ought to sound like new examples of the real thing. 00:46:40.590 --> 00:46:42.890 align:middle line:84% And so what I'm going to play you here 00:46:42.890 --> 00:46:46.250 align:middle line:84% are a whole bunch of examples of synthetic sounds 00:46:46.250 --> 00:46:51.090 align:middle line:84% that are generated from noise just by causing the noise 00:46:51.090 --> 00:46:53.990 align:middle line:84% to match the statistics of each of these examples. 00:46:53.990 --> 00:46:59.543 align:middle line:84% And in many cases, they actually sound pretty good. 00:46:59.543 --> 00:47:00.210 align:middle line:90% [AUDIO PLAYBACK] 00:47:00.210 --> 00:47:01.710 align:middle line:90% [RAIN FALLING] 00:47:01.710 --> 00:47:02.710 align:middle line:90% [END PLAYBACK] 00:47:02.710 --> 00:47:04.670 align:middle line:84% JOSH MCDERMOTT: So that's pretty good rain. 00:47:04.670 --> 00:47:05.337 align:middle line:90% [AUDIO PLAYBACK] 00:47:05.337 --> 00:47:06.940 align:middle line:90% [WATER GURGLING] 00:47:06.940 --> 00:47:07.850 align:middle line:90% [END PLAYBACK] 00:47:07.850 --> 00:47:09.473 align:middle line:90% JOSH MCDERMOTT: The stream. 00:47:09.473 --> 00:47:10.140 align:middle line:90% [AUDIO PLAYBACK] 00:47:10.140 --> 00:47:11.595 align:middle line:90% [BUBBLES POPPING] 00:47:11.595 --> 00:47:12.383 align:middle line:90% [END PLAYBACK] 00:47:12.383 --> 00:47:13.050 align:middle line:90% [AUDIO PLAYBACK] 00:47:13.050 --> 00:47:15.475 align:middle line:90% [FIRE CRACKLING] 00:47:15.475 --> 00:47:16.450 align:middle line:90% [END PLAYBACK] 00:47:16.450 --> 00:47:17.360 align:middle line:90% JOSH MCDERMOTT: Fire. 00:47:17.360 --> 00:47:18.027 align:middle line:90% [AUDIO PLAYBACK] 00:47:18.027 --> 00:47:20.889 align:middle line:90% [APPLAUSE] 00:47:20.889 --> 00:47:21.850 align:middle line:90% [END PLAYBACK] 00:47:21.850 --> 00:47:23.439 align:middle line:90% JOSH MCDERMOTT: Applause. 00:47:23.439 --> 00:47:24.106 align:middle line:90% [AUDIO PLAYBACK] 00:47:24.106 --> 00:47:27.034 align:middle line:90% [WIND BLOWING] 00:47:27.034 --> 00:47:27.935 align:middle line:90% [END PLAYBACK] 00:47:27.935 --> 00:47:28.810 align:middle line:90% JOSH MCDERMOTT: Wind. 00:47:28.810 --> 00:47:30.124 align:middle line:90% Here's insects. 00:47:30.124 --> 00:47:30.791 align:middle line:90% [AUDIO PLAYBACK] 00:47:30.791 --> 00:47:33.518 align:middle line:90% [MULTIPLE INSECTS CHITTERING] 00:47:33.518 --> 00:47:34.018 align:middle line:90% 00:47:34.018 --> 00:47:35.410 align:middle line:90% [END PLAYBACK] 00:47:35.410 --> 00:47:36.789 align:middle line:90% JOSH MCDERMOTT: Birds. 00:47:36.789 --> 00:47:37.456 align:middle line:90% [AUDIO PLAYBACK] 00:47:37.456 --> 00:47:39.348 align:middle line:90% [MULTIPLE BIRDS CHIRPING] 00:47:39.348 --> 00:47:40.517 align:middle line:90% [END PLAYBACK] 00:47:40.517 --> 00:47:41.850 align:middle line:90% JOSH MCDERMOTT: And crowd noise. 00:47:41.850 --> 00:47:42.517 align:middle line:90% [AUDIO PLAYBACK] 00:47:42.517 --> 00:47:45.104 align:middle line:90% [AMBIENT CROWD NOISE] 00:47:45.104 --> 00:47:46.940 align:middle line:90% [END PLAYBACK] 00:47:46.940 --> 00:47:49.900 align:middle line:90% JOSH MCDERMOTT: OK. 00:47:49.900 --> 00:47:54.680 align:middle line:84% So with marginal moments and these pairwise correlations, 00:47:54.680 --> 00:47:57.820 align:middle line:84% the synthesis is often pretty compelling. 00:47:57.820 --> 00:47:59.322 align:middle line:84% It also works for a lot of things 00:47:59.322 --> 00:48:01.280 align:middle line:84% that you might consider to be unnatural sounds. 00:48:01.280 --> 00:48:02.488 align:middle line:90% I'll play you a few examples. 00:48:02.488 --> 00:48:03.728 align:middle line:90% Here's rustling paper. 00:48:03.728 --> 00:48:04.395 align:middle line:90% [AUDIO PLAYBACK] 00:48:04.395 --> 00:48:06.542 align:middle line:90% [PAPER RUSTLING] 00:48:06.542 --> 00:48:07.125 align:middle line:90% [END PLAYBACK] 00:48:07.125 --> 00:48:09.083 align:middle line:84% JOSH MCDERMOTT: That's holding a piece of paper 00:48:09.083 --> 00:48:10.380 align:middle line:90% and going like that. 00:48:10.380 --> 00:48:12.673 align:middle line:90% And the jackhammer. 00:48:12.673 --> 00:48:13.340 align:middle line:90% [AUDIO PLAYBACK] 00:48:13.340 --> 00:48:15.690 align:middle line:90% [JACKHAMMER HAMMERING] 00:48:15.690 --> 00:48:19.180 align:middle line:90% [END PLAYBACK] 00:48:19.180 --> 00:48:19.680 align:middle line:90% OK. 00:48:19.680 --> 00:48:24.460 align:middle line:84% So the success of the synthesis suggests that these statistics 00:48:24.460 --> 00:48:26.060 align:middle line:84% could underlie the representation 00:48:26.060 --> 00:48:29.593 align:middle line:90% and the recognition of textures. 00:48:29.593 --> 00:48:34.260 align:middle line:84% So there's also this kind of interesting possibility, which 00:48:34.260 --> 00:48:38.040 align:middle line:84% is that if we take this idea seriously, 00:48:38.040 --> 00:48:41.870 align:middle line:84% that texture is represented with statistics that kind of capture 00:48:41.870 --> 00:48:44.990 align:middle line:84% the time-averaged properties, then if really, your brain 00:48:44.990 --> 00:48:48.790 align:middle line:84% is just representing those time-averaged statistics, 00:48:48.790 --> 00:48:51.630 align:middle line:84% then I think the conclusion that follows 00:48:51.630 --> 00:48:54.950 align:middle line:84% from that is that different exemplars of the same texture 00:48:54.950 --> 00:48:58.270 align:middle line:84% should be difficult to tell apart. 00:48:58.270 --> 00:49:02.670 align:middle line:84% So these are two different excerpts of fire 00:49:02.670 --> 00:49:05.750 align:middle line:84% that were synthesized from two different samples of noise. 00:49:05.750 --> 00:49:08.750 align:middle line:84% If you look closely, you can see that the patterns 00:49:08.750 --> 00:49:11.670 align:middle line:84% of the vertical streaks, so the clicks and the pops, 00:49:11.670 --> 00:49:13.610 align:middle line:84% are different in the two excerpts, 00:49:13.610 --> 00:49:15.870 align:middle line:84% but they also kind of look sort of the same. 00:49:15.870 --> 00:49:19.043 align:middle line:90% They've got the same texture. 00:49:19.043 --> 00:49:27.230 align:middle line:84% And so this is a graph that is plotting the standard deviation 00:49:27.230 --> 00:49:29.972 align:middle line:90% of one particular statistic-- 00:49:29.972 --> 00:49:32.430 align:middle line:84% you would see a similar graph regardless of which statistic 00:49:32.430 --> 00:49:32.972 align:middle line:90% you measured. 00:49:32.972 --> 00:49:35.230 align:middle line:84% But this is one example statistic 00:49:35.230 --> 00:49:39.310 align:middle line:84% measured in different excerpts of different lengths. 00:49:39.310 --> 00:49:43.920 align:middle line:84% And so what this shows is that if you take very short excerpts 00:49:43.920 --> 00:49:46.640 align:middle line:84% of like 40 or 80 milliseconds, and you measure 00:49:46.640 --> 00:49:48.580 align:middle line:84% the statistic in different excerpts, 00:49:48.580 --> 00:49:50.400 align:middle line:90% the standard deviation is high. 00:49:50.400 --> 00:49:52.200 align:middle line:84% So the value of the statistic that you 00:49:52.200 --> 00:49:55.440 align:middle line:84% will measure from different excerpts will be different. 00:49:55.440 --> 00:49:58.080 align:middle line:84% Because you're getting these small samples, and sometimes 00:49:58.080 --> 00:50:00.080 align:middle line:84% maybe you have a couple of clicks, and other times maybe 00:50:00.080 --> 00:50:01.080 align:middle line:90% you don't, for instance. 00:50:01.080 --> 00:50:05.040 align:middle line:84% But what this shows is that as the excerpts become longer 00:50:05.040 --> 00:50:08.400 align:middle line:84% and get out to a second or two, the standard deviation 00:50:08.400 --> 00:50:10.160 align:middle line:90% becomes quite small. 00:50:10.160 --> 00:50:13.800 align:middle line:84% So what that means is that the statistics that you measure 00:50:13.800 --> 00:50:16.240 align:middle line:84% from these different excerpts of a texture, they 00:50:16.240 --> 00:50:18.600 align:middle line:90% converge to the same values. 00:50:18.600 --> 00:50:23.760 align:middle line:84% So again, the idea is that if you do an experiment where 00:50:23.760 --> 00:50:26.460 align:middle line:84% you ask people to try to tell these things apart, well, 00:50:26.460 --> 00:50:28.120 align:middle line:84% the prediction here is that if they're just representing 00:50:28.120 --> 00:50:29.720 align:middle line:84% the statistics, they should be able to tell them apart 00:50:29.720 --> 00:50:32.420 align:middle line:84% when the excerpts are short, but not when they're long. 00:50:32.420 --> 00:50:36.720 align:middle line:90% 00:50:36.720 --> 00:50:40.760 align:middle line:84% So first, we're going to do kind of a vanilla experiment, 00:50:40.760 --> 00:50:44.280 align:middle line:84% just measuring the ability to discriminate different textures. 00:50:44.280 --> 00:50:48.060 align:middle line:84% So these are signals that have different long-term statistics. 00:50:48.060 --> 00:50:49.460 align:middle line:90% So here's the task. 00:50:49.460 --> 00:50:53.240 align:middle line:84% So people are hear three excerpts of sound-- 00:50:53.240 --> 00:50:55.880 align:middle line:90% examples are shown up top. 00:50:55.880 --> 00:50:57.980 align:middle line:84% Two of the excerpts-- in this case, 00:50:57.980 --> 00:50:59.680 align:middle line:90% it's the second and the third-- 00:50:59.680 --> 00:51:02.100 align:middle line:84% are different excerpts of the same texture. 00:51:02.100 --> 00:51:03.740 align:middle line:90% So it could be rain. 00:51:03.740 --> 00:51:06.480 align:middle line:84% The third excerpt, which in this case 00:51:06.480 --> 00:51:09.200 align:middle line:84% is first, but it can either go first or last, 00:51:09.200 --> 00:51:11.460 align:middle line:84% is an excerpt of a different texture. 00:51:11.460 --> 00:51:14.360 align:middle line:90% So it could be fire. 00:51:14.360 --> 00:51:16.640 align:middle line:84% And the task that the person has to perform 00:51:16.640 --> 00:51:20.220 align:middle line:84% is to say which sound was produced by a different source. 00:51:20.220 --> 00:51:22.360 align:middle line:84% In this case, the answer would be first, 00:51:22.360 --> 00:51:24.720 align:middle line:84% because it's a different texture. 00:51:24.720 --> 00:51:27.760 align:middle line:84% And this graph is going to show you how accurately people can 00:51:27.760 --> 00:51:29.120 align:middle line:90% perform this task. 00:51:29.120 --> 00:51:31.160 align:middle line:84% So the y-axis plots proportion correct 00:51:31.160 --> 00:51:35.760 align:middle line:84% as a function of the duration of the stimulus excerpts. 00:51:35.760 --> 00:51:38.290 align:middle line:84% And what you see is not super surprising, 00:51:38.290 --> 00:51:43.090 align:middle line:84% is that as the excerpts get longer, people get better. 00:51:43.090 --> 00:51:46.050 align:middle line:84% It's not surprising in the sense that as the stimulus excerpts 00:51:46.050 --> 00:51:49.090 align:middle line:84% get longer, you're giving people more information with which 00:51:49.090 --> 00:51:52.370 align:middle line:84% to do the task in which to tell that these two things have 00:51:52.370 --> 00:51:56.593 align:middle line:84% something in common, and that first one is different. 00:51:56.593 --> 00:51:58.010 align:middle line:84% So this is kind of standardly what 00:51:58.010 --> 00:52:00.650 align:middle line:84% you would expect to get on most psychophysical tasks, 00:52:00.650 --> 00:52:02.310 align:middle line:84% is that when the stimuli are longer, 00:52:02.310 --> 00:52:03.570 align:middle line:90% people are going to be better. 00:52:03.570 --> 00:52:06.450 align:middle line:84% And in this particular case, it's 00:52:06.450 --> 00:52:07.970 align:middle line:84% certainly consistent with the idea 00:52:07.970 --> 00:52:11.730 align:middle line:84% that people are using statistics because, as the excerpt gets 00:52:11.730 --> 00:52:15.130 align:middle line:84% longer, the statistics of these two 00:52:15.130 --> 00:52:18.010 align:middle line:84% will kind of converge to the same values 00:52:18.010 --> 00:52:23.090 align:middle line:84% and be reliably different from the statistics of the first one. 00:52:23.090 --> 00:52:29.170 align:middle line:84% So now, the really interesting experiment is this one. 00:52:29.170 --> 00:52:32.450 align:middle line:84% So in this particular case, people 00:52:32.450 --> 00:52:36.820 align:middle line:84% hear three excerpts of the same texture. 00:52:36.820 --> 00:52:40.900 align:middle line:84% Two of them, in this case, it is the second and third, 00:52:40.900 --> 00:52:42.360 align:middle line:90% are physically identical. 00:52:42.360 --> 00:52:45.940 align:middle line:84% So it's the exact same excerpt of rain. 00:52:45.940 --> 00:52:50.460 align:middle line:84% The first one is a different excerpt of rain, for instance. 00:52:50.460 --> 00:52:54.540 align:middle line:84% And the task is just to say which sound was 00:52:54.540 --> 00:52:56.220 align:middle line:90% different from the other two. 00:52:56.220 --> 00:52:59.140 align:middle line:84% So this is what's called exemplar discrimination. 00:52:59.140 --> 00:53:02.460 align:middle line:90% 00:53:02.460 --> 00:53:07.180 align:middle line:84% And so the idea here is that if, when you listen to these texture 00:53:07.180 --> 00:53:11.740 align:middle line:84% sounds, you're representing them with statistics, 00:53:11.740 --> 00:53:15.260 align:middle line:84% well, then, as the excerpts get longer, 00:53:15.260 --> 00:53:17.500 align:middle line:84% the statistics of these three sounds 00:53:17.500 --> 00:53:20.320 align:middle line:84% should all converge to very similar values. 00:53:20.320 --> 00:53:22.780 align:middle line:84% And it should be difficult to tell which one is 00:53:22.780 --> 00:53:24.460 align:middle line:90% different from the other two. 00:53:24.460 --> 00:53:29.420 align:middle line:84% Whereas when the excerpts are short, as we just saw, 00:53:29.420 --> 00:53:31.700 align:middle line:84% the statistics will tend to be different 00:53:31.700 --> 00:53:33.860 align:middle line:84% because you have small samples, and so you ought 00:53:33.860 --> 00:53:35.200 align:middle line:90% to be able to tell them apart. 00:53:35.200 --> 00:53:40.300 align:middle line:84% So there's this kind of wacky prediction 00:53:40.300 --> 00:53:44.300 align:middle line:84% that for this particular task, if the hypothesis is correct 00:53:44.300 --> 00:53:47.760 align:middle line:84% that you're representing these sounds just with statistics, 00:53:47.760 --> 00:53:50.740 align:middle line:84% then performance ought to get worse as the sounds 00:53:50.740 --> 00:53:52.460 align:middle line:90% increase in duration. 00:53:52.460 --> 00:53:55.680 align:middle line:84% And that's, in fact, what happens. 00:53:55.680 --> 00:53:59.260 align:middle line:84% So you can see that people are very good at this task at 90% 00:53:59.260 --> 00:54:02.038 align:middle line:84% correct when the excerpts of the textures are very short. 00:54:02.038 --> 00:54:04.080 align:middle line:84% But then as they get longer, up to a few seconds, 00:54:04.080 --> 00:54:05.600 align:middle line:90% people get worse and worse. 00:54:05.600 --> 00:54:10.120 align:middle line:90% 00:54:10.120 --> 00:54:11.960 align:middle line:84% So let's just think about what this means. 00:54:11.960 --> 00:54:17.500 align:middle line:84% So you might imagine that you can't really 00:54:17.500 --> 00:54:19.940 align:middle line:84% tell one example of rain from another example, 00:54:19.940 --> 00:54:22.620 align:middle line:84% just because they all sound the same. 00:54:22.620 --> 00:54:24.780 align:middle line:84% Maybe they don't actually have discriminable detail 00:54:24.780 --> 00:54:26.500 align:middle line:84% that you could use to tell them apart. 00:54:26.500 --> 00:54:30.700 align:middle line:84% But we can rule that out, because when you just 00:54:30.700 --> 00:54:34.310 align:middle line:84% give people these very, very short excerpts of rain, or fire, 00:54:34.310 --> 00:54:38.190 align:middle line:84% or whatever it is, people can very accurately discriminate 00:54:38.190 --> 00:54:38.690 align:middle line:90% them. 00:54:38.690 --> 00:54:40.857 align:middle line:84% They can tell whether they're the same or different. 00:54:40.857 --> 00:54:45.910 align:middle line:84% So that says that these sounds do contain discriminable detail. 00:54:45.910 --> 00:54:49.830 align:middle line:84% It's just somehow, when the duration gets long, 00:54:49.830 --> 00:54:52.510 align:middle line:90% you're not able to access it. 00:54:52.510 --> 00:54:56.030 align:middle line:84% You might also imagine that-- so this is a discrimination task 00:54:56.030 --> 00:54:59.470 align:middle line:90% with 22.5 second-long stimuli. 00:54:59.470 --> 00:55:03.030 align:middle line:84% You might think it's really hard to do tasks with long stimuli. 00:55:03.030 --> 00:55:06.712 align:middle line:84% But on the other hand, when it's a texture discrimination task, 00:55:06.712 --> 00:55:08.670 align:middle line:84% and one of the stimuli has different statistics 00:55:08.670 --> 00:55:10.628 align:middle line:84% from the other, people are really good at that. 00:55:10.628 --> 00:55:13.750 align:middle line:84% So it's not like there's some issue with doing tasks 00:55:13.750 --> 00:55:16.430 align:middle line:90% with things that are long. 00:55:16.430 --> 00:55:20.030 align:middle line:84% Moreover, the fact that performance on this texture 00:55:20.030 --> 00:55:23.550 align:middle line:84% discrimination task gets better with duration 00:55:23.550 --> 00:55:28.290 align:middle line:84% indicates that the details that are present in these sounds, 00:55:28.290 --> 00:55:30.200 align:middle line:90% they do get accumulated. 00:55:30.200 --> 00:55:34.000 align:middle line:84% So the longer the sound is, the better you are at the task. 00:55:34.000 --> 00:55:38.840 align:middle line:84% So all that details that's kind of streaming into your ear, 00:55:38.840 --> 00:55:41.920 align:middle line:84% you're able to use that for performance 00:55:41.920 --> 00:55:43.700 align:middle line:84% on a texture discrimination task. 00:55:43.700 --> 00:55:46.120 align:middle line:84% So it seems like they contribute to your estimate 00:55:46.120 --> 00:55:47.180 align:middle line:90% of the statistics. 00:55:47.180 --> 00:55:50.100 align:middle line:84% And when the task is based on differences in statistics, 00:55:50.100 --> 00:55:53.630 align:middle line:90% you get better and better. 00:55:53.630 --> 00:55:58.240 align:middle line:84% But it seems that those details are not otherwise retained. 00:55:58.240 --> 00:56:02.120 align:middle line:84% So that when you have different excerpts of the same texture, 00:56:02.120 --> 00:56:05.820 align:middle line:84% you have a hard time telling them apart. 00:56:05.820 --> 00:56:09.040 align:middle line:84% So all of this is consistent with the idea 00:56:09.040 --> 00:56:11.760 align:middle line:84% that when you hear texture-like things, 00:56:11.760 --> 00:56:15.280 align:middle line:84% your brain turns them into a statistical representation 00:56:15.280 --> 00:56:17.760 align:middle line:84% that kind of averages information over time, 00:56:17.760 --> 00:56:19.820 align:middle line:84% and you lose access to the detail, 00:56:19.820 --> 00:56:23.300 align:middle line:84% even though that detail is used to compute the statistics. 00:56:23.300 --> 00:56:26.520 align:middle line:90% 00:56:26.520 --> 00:56:30.640 align:middle line:84% What questions you got about this? 00:56:30.640 --> 00:56:31.670 align:middle line:90% Yeah. 00:56:31.670 --> 00:56:33.420 align:middle line:84% AUDIENCE: Well, it's about discrimination, 00:56:33.420 --> 00:56:36.720 align:middle line:84% is it strictly decreasing with the excerpt duration, 00:56:36.720 --> 00:56:41.160 align:middle line:84% or is that supposed to be a turning point where it initially 00:56:41.160 --> 00:56:44.200 align:middle line:84% gets better and then it gets worse? 00:56:44.200 --> 00:56:46.700 align:middle line:84% JOSH MCDERMOTT: Yeah, I mean, if you made this short enough, 00:56:46.700 --> 00:56:51.227 align:middle line:84% you're going to get worse, because at some point they'll-- 00:56:51.227 --> 00:56:53.560 align:middle line:84% like if they get short enough, they'll probably all just 00:56:53.560 --> 00:56:55.360 align:middle line:84% sound like clicks, is my guess, and they'll 00:56:55.360 --> 00:56:57.200 align:middle line:90% get harder to discriminate. 00:56:57.200 --> 00:56:57.820 align:middle line:90% Yeah. 00:56:57.820 --> 00:57:00.160 align:middle line:84% But the experiment didn't go down that short. 00:57:00.160 --> 00:57:02.120 align:middle line:90% Yeah. 00:57:02.120 --> 00:57:02.620 align:middle line:90% Yeah. 00:57:02.620 --> 00:57:05.080 align:middle line:84% I mean 40 milliseconds is still-- 00:57:05.080 --> 00:57:07.080 align:middle line:90% that's like a blink of an eye. 00:57:07.080 --> 00:57:08.460 align:middle line:90% So that's pretty brief. 00:57:08.460 --> 00:57:12.280 align:middle line:84% But it's possible that if you went down to 10, at some level, 00:57:12.280 --> 00:57:13.740 align:middle line:84% if you just end up with one sample, 00:57:13.740 --> 00:57:18.300 align:middle line:90% they're all going to be clicks. 00:57:18.300 --> 00:57:22.320 align:middle line:84% So you might also be worried about that maybe what's going on 00:57:22.320 --> 00:57:25.980 align:middle line:90% here is related to memory. 00:57:25.980 --> 00:57:28.890 align:middle line:84% So this is what the stimuli actually 00:57:28.890 --> 00:57:31.970 align:middle line:84% look like on a long-duration trial 00:57:31.970 --> 00:57:33.830 align:middle line:84% versus a short-duration trial right. 00:57:33.830 --> 00:57:37.265 align:middle line:84% So if the inner stimulus interval is the same, 00:57:37.265 --> 00:57:39.890 align:middle line:84% you might worry about the fact that, well, when the stimuli are 00:57:39.890 --> 00:57:43.250 align:middle line:84% really long, the bits that you have to compare 00:57:43.250 --> 00:57:45.390 align:middle line:84% are kind of separated like further in time, 00:57:45.390 --> 00:57:47.410 align:middle line:84% and so maybe it's hard to remember, 00:57:47.410 --> 00:57:50.170 align:middle line:90% and that kind of messes you up. 00:57:50.170 --> 00:57:52.290 align:middle line:84% So you can do a control experiment where you just 00:57:52.290 --> 00:57:55.130 align:middle line:84% kind of space the short ones out in time, 00:57:55.130 --> 00:57:57.075 align:middle line:90% and that has very little effect. 00:57:57.075 --> 00:57:58.450 align:middle line:84% So you're still much, much better 00:57:58.450 --> 00:58:02.320 align:middle line:84% when they're short than when they're long. 00:58:02.320 --> 00:58:06.730 align:middle line:84% So the take-home message here is that the results suggest 00:58:06.730 --> 00:58:08.850 align:middle line:84% that people are using a representation 00:58:08.850 --> 00:58:12.610 align:middle line:90% of time-averaged statistics. 00:58:12.610 --> 00:58:15.130 align:middle line:84% And again, based on this phenomenon 00:58:15.130 --> 00:58:18.850 align:middle line:84% that the statistics will converge 00:58:18.850 --> 00:58:20.497 align:middle line:90% as the sample size increases. 00:58:20.497 --> 00:58:22.830 align:middle line:84% In this case, the sample size is determined by duration. 00:58:22.830 --> 00:58:26.620 align:middle line:90% 00:58:26.620 --> 00:58:29.620 align:middle line:84% So textures in the real world are normally 00:58:29.620 --> 00:58:31.740 align:middle line:84% generated from a superposition of sources. 00:58:31.740 --> 00:58:34.680 align:middle line:84% So you have rain, it consists of a lot of raindrops, or insects, 00:58:34.680 --> 00:58:41.620 align:middle line:84% it's a lot of insects, things like that And so, for instance, 00:58:41.620 --> 00:58:43.353 align:middle line:90% like here's one person talking. 00:58:43.353 --> 00:58:44.020 align:middle line:90% [AUDIO PLAYBACK] 00:58:44.020 --> 00:58:46.485 align:middle line:90% - [GERMAN] 00:58:46.485 --> 00:58:48.445 align:middle line:90% [END PLAYBACK] 00:58:48.445 --> 00:58:50.320 align:middle line:84% JOSH MCDERMOTT: And here's 29 people talking. 00:58:50.320 --> 00:58:51.800 align:middle line:90% This is a German cocktail party. 00:58:51.800 --> 00:58:52.467 align:middle line:90% [AUDIO PLAYBACK] 00:58:52.467 --> 00:58:54.192 align:middle line:84% [MULTIPLE CONVERSATIONS IN GERMAN] 00:58:54.192 --> 00:58:55.140 align:middle line:90% [END PLAYBACK] 00:58:55.140 --> 00:58:57.200 align:middle line:84% JOSH MCDERMOTT: And that's a texture. 00:58:57.200 --> 00:59:00.060 align:middle line:84% So we can generate, in principle, 00:59:00.060 --> 00:59:03.580 align:middle line:84% kind of a continuum of sounds, where on one end 00:59:03.580 --> 00:59:05.840 align:middle line:84% you have a single source, and on the other hand, 00:59:05.840 --> 00:59:07.880 align:middle line:90% you have a lot of sources. 00:59:07.880 --> 00:59:12.340 align:middle line:84% And we can ask, well what happens 00:59:12.340 --> 00:59:15.740 align:middle line:84% to this phenomenon of being dependent on statistical 00:59:15.740 --> 00:59:20.020 align:middle line:84% representations as a function of the extent to which the sounds 00:59:20.020 --> 00:59:22.180 align:middle line:90% are really texture-like? 00:59:22.180 --> 00:59:23.440 align:middle line:90% And so here's what happens. 00:59:23.440 --> 00:59:26.130 align:middle line:84% So this is a graph that-- it's the same task. 00:59:26.130 --> 00:59:30.820 align:middle line:84% This is exemplar discrimination, but with four types of signals. 00:59:30.820 --> 00:59:33.070 align:middle line:84% So here, you have something that's very texture-like-- 00:59:33.070 --> 00:59:37.830 align:middle line:84% it's 115 speakers at this party-- 00:59:37.830 --> 00:59:39.170 align:middle line:90% all the way up to one speaker. 00:59:39.170 --> 00:59:41.650 align:middle line:84% So the graph plots proportion correct versus excerpt duration. 00:59:41.650 --> 00:59:43.858 align:middle line:84% So here, there's just two durations, one that's short 00:59:43.858 --> 00:59:45.430 align:middle line:90% and one there's one that's long. 00:59:45.430 --> 00:59:47.310 align:middle line:84% And so the black curve is just kind 00:59:47.310 --> 00:59:49.670 align:middle line:84% of a replication of the thing that we saw previously 00:59:49.670 --> 00:59:53.230 align:middle line:84% for textures, which is that you have a hard time discriminating 00:59:53.230 --> 00:59:56.030 align:middle line:84% excerpts when they're long, consistent with the idea 00:59:56.030 --> 00:59:58.270 align:middle line:84% that you become reliant on statistics. 00:59:58.270 --> 01:00:02.110 align:middle line:84% But what's pretty interesting is that when 01:00:02.110 --> 01:00:05.070 align:middle line:84% the density of this mixture is varied, 01:00:05.070 --> 01:00:06.330 align:middle line:90% the behavior really changes. 01:00:06.330 --> 01:00:08.950 align:middle line:84% And so in the extreme case where you have one speaker, 01:00:08.950 --> 01:00:13.150 align:middle line:84% you actually get better at the task as the duration increases. 01:00:13.150 --> 01:00:18.470 align:middle line:84% So with a single speaker, you're not completely 01:00:18.470 --> 01:00:20.323 align:middle line:84% reducing the representation to these time 01:00:20.323 --> 01:00:21.990 align:middle line:84% average statistics, which at some level, 01:00:21.990 --> 01:00:24.090 align:middle line:84% is not very surprising, because speech 01:00:24.090 --> 01:00:26.190 align:middle line:84% has got a lot of temporal structure, 01:00:26.190 --> 01:00:28.870 align:middle line:84% and allows you to understand what someone says, 01:00:28.870 --> 01:00:31.800 align:middle line:84% is the temporal sequence of the phonemes 01:00:31.800 --> 01:00:33.050 align:middle line:90% and syllables that they utter. 01:00:33.050 --> 01:00:36.850 align:middle line:84% But you can also see that there's this gradual transition. 01:00:36.850 --> 01:00:38.990 align:middle line:84% So as the thing becomes more and more texture-like, 01:00:38.990 --> 01:00:42.830 align:middle line:84% you seem to become more and more dependent 01:00:42.830 --> 01:00:45.910 align:middle line:84% on this statistical representation. 01:00:45.910 --> 01:00:48.690 align:middle line:84% And in all of the cases at the short durations, 01:00:48.690 --> 01:00:54.200 align:middle line:84% you're about equally good at discriminating the sounds. 01:00:54.200 --> 01:00:56.370 align:middle line:84% So we can ask whether this is specific to speech. 01:00:56.370 --> 01:00:59.770 align:middle line:84% So here's the same idea, but with drum hits. 01:00:59.770 --> 01:01:03.490 align:middle line:84% So this is a very sparse sequence of drum hits. 01:01:03.490 --> 01:01:04.485 align:middle line:90% [AUDIO PLAYBACK] 01:01:04.485 --> 01:01:06.960 align:middle line:90% [SPARSE DRUM HITS] 01:01:06.960 --> 01:01:07.950 align:middle line:90% [END PLAYBACK] 01:01:07.950 --> 01:01:09.950 align:middle line:84% JOSH MCDERMOTT: And here's, it's more dense. 01:01:09.950 --> 01:01:12.250 align:middle line:90% [RAPID DRUM HITS] 01:01:12.250 --> 01:01:13.630 align:middle line:90% [END PLAYBACK] 01:01:13.630 --> 01:01:16.330 align:middle line:84% JOSH MCDERMOTT: And you get the same phenomenon. 01:01:16.330 --> 01:01:19.950 align:middle line:84% So the very sparse signal, five hits per second, 01:01:19.950 --> 01:01:23.280 align:middle line:84% shows an improvement in this discrimination task 01:01:23.280 --> 01:01:28.280 align:middle line:84% with duration, whereas the dense ones show this decrease 01:01:28.280 --> 01:01:29.700 align:middle line:90% in discrimination. 01:01:29.700 --> 01:01:32.740 align:middle line:90% 01:01:32.740 --> 01:01:34.958 align:middle line:84% So in all these, cases, again the same points apply. 01:01:34.958 --> 01:01:37.000 align:middle line:84% So the high performance with these short excerpts 01:01:37.000 --> 01:01:39.440 align:middle line:84% indicates that all of the stimuli 01:01:39.440 --> 01:01:41.840 align:middle line:90% contain discriminable variation. 01:01:41.840 --> 01:01:45.800 align:middle line:84% It's not the case that you can't discriminate these things, 01:01:45.800 --> 01:01:48.600 align:middle line:84% just because there isn't discriminable detail that's 01:01:48.600 --> 01:01:49.960 align:middle line:90% contained in the sound. 01:01:49.960 --> 01:01:52.200 align:middle line:84% So they all contain discriminable variation, 01:01:52.200 --> 01:01:55.480 align:middle line:84% but the temporal detail is not retained when 01:01:55.480 --> 01:01:57.680 align:middle line:90% the signals are texture-like. 01:01:57.680 --> 01:02:03.040 align:middle line:84% So that seems to be a fundamental thing 01:02:03.040 --> 01:02:06.020 align:middle line:84% about the representation of textures. 01:02:06.020 --> 01:02:09.560 align:middle line:84% And we think there's probably similar principles that 01:02:09.560 --> 01:02:13.160 align:middle line:84% would apply for visual textures, but only across space. 01:02:13.160 --> 01:02:16.260 align:middle line:84% So as the size of a texture would increase, 01:02:16.260 --> 01:02:20.010 align:middle line:84% you'd probably see similar decreases in the ability 01:02:20.010 --> 01:02:22.410 align:middle line:90% to discriminate examples. 01:02:22.410 --> 01:02:23.510 align:middle line:90% Questions about that? 01:02:23.510 --> 01:02:30.390 align:middle line:90% 01:02:30.390 --> 01:02:32.970 align:middle line:84% So one other question you might wonder about 01:02:32.970 --> 01:02:35.390 align:middle line:84% is, will any set of statistics do? 01:02:35.390 --> 01:02:39.330 align:middle line:84% So we just kind of made up this set of statistics, 01:02:39.330 --> 01:02:44.730 align:middle line:84% but it was based on our best bet for a model of the early stages 01:02:44.730 --> 01:02:45.750 align:middle line:90% of the auditory system. 01:02:45.750 --> 01:02:48.970 align:middle line:84% So there are these three stages of this model. 01:02:48.970 --> 01:02:52.250 align:middle line:84% There are these filters that are based on the cochlea. 01:02:52.250 --> 01:02:54.690 align:middle line:84% There's this compressive nonlinearity 01:02:54.690 --> 01:02:56.210 align:middle line:84% that is motivated by the compression 01:02:56.210 --> 01:02:57.650 align:middle line:84% that we think happens in the ear. 01:02:57.650 --> 01:02:59.830 align:middle line:84% And then we've got these modulation filters, 01:02:59.830 --> 01:03:01.290 align:middle line:90% the second stage of filters. 01:03:01.290 --> 01:03:05.130 align:middle line:84% And we could ask, well, what happens if we actually 01:03:05.130 --> 01:03:08.290 align:middle line:84% change this in a way to make it less consistent 01:03:08.290 --> 01:03:10.330 align:middle line:90% with what we know about biology? 01:03:10.330 --> 01:03:13.530 align:middle line:84% So this is called a log-spaced filter bank. 01:03:13.530 --> 01:03:16.130 align:middle line:84% You all know that the filter bank in the ear 01:03:16.130 --> 01:03:18.410 align:middle line:84% is not exactly log spaced, but so it's kind 01:03:18.410 --> 01:03:19.910 align:middle line:90% of approximately log spaced. 01:03:19.910 --> 01:03:21.660 align:middle line:84% But remember, the filters in the ear, they 01:03:21.660 --> 01:03:24.090 align:middle line:84% get broader at high frequencies than at low frequencies. 01:03:24.090 --> 01:03:25.507 align:middle line:84% So you can ask, well, what happens 01:03:25.507 --> 01:03:28.730 align:middle line:84% if we use an incorrect auditory model where the filters are 01:03:28.730 --> 01:03:30.957 align:middle line:84% linearly spaced, like they're all the same bandwidth? 01:03:30.957 --> 01:03:32.790 align:middle line:84% So this violates what we know about the ear. 01:03:32.790 --> 01:03:35.370 align:middle line:90% 01:03:35.370 --> 01:03:39.330 align:middle line:84% Similarly, we could take that compressive nonlinearity that 01:03:39.330 --> 01:03:42.770 align:middle line:84% happens in the ear and get rid of that 01:03:42.770 --> 01:03:45.310 align:middle line:84% and end up with different auditory models, 01:03:45.310 --> 01:03:48.690 align:middle line:84% measure their statistics, and then ask 01:03:48.690 --> 01:03:52.090 align:middle line:84% if it makes a difference in how the textures sound. 01:03:52.090 --> 01:03:55.010 align:middle line:84% And so this was measured with an experiment where 01:03:55.010 --> 01:03:57.770 align:middle line:84% people heard the original texture, and then 01:03:57.770 --> 01:03:59.850 align:middle line:90% two synthetic versions. 01:03:59.850 --> 01:04:01.610 align:middle line:84% One of the synthetic versions came 01:04:01.610 --> 01:04:05.178 align:middle line:84% from the biologically correct model, or at least our best bet 01:04:05.178 --> 01:04:06.470 align:middle line:90% at what's biologically correct. 01:04:06.470 --> 01:04:09.630 align:middle line:84% And the other one came from an alternative model. 01:04:09.630 --> 01:04:11.690 align:middle line:84% And so there were four alternative models, 01:04:11.690 --> 01:04:13.610 align:middle line:84% one where we got rid of the compression, one 01:04:13.610 --> 01:04:15.627 align:middle line:84% where we made the cochlea linearly spaced, 01:04:15.627 --> 01:04:17.960 align:middle line:84% one when we made the modulation filters linearly spaced, 01:04:17.960 --> 01:04:19.580 align:middle line:84% and the one where we did all three. 01:04:19.580 --> 01:04:23.340 align:middle line:84% And so the prediction here is that it doesn't really 01:04:23.340 --> 01:04:24.980 align:middle line:84% matter what model you use and what 01:04:24.980 --> 01:04:26.560 align:middle line:90% statistics you're measuring. 01:04:26.560 --> 01:04:29.420 align:middle line:84% If you just need to measure a big set of statistics, 01:04:29.420 --> 01:04:31.460 align:middle line:84% then people shouldn't have, on average, 01:04:31.460 --> 01:04:34.120 align:middle line:84% a preference for the textures from one model over another. 01:04:34.120 --> 01:04:36.740 align:middle line:84% And so they should be right at 50%, that dashed line. 01:04:36.740 --> 01:04:41.660 align:middle line:84% But instead what happens is that in every case, 01:04:41.660 --> 01:04:47.260 align:middle line:84% people systematically prefer the biologically plausible model. 01:04:47.260 --> 01:04:50.040 align:middle line:90% And here's just a few examples. 01:04:50.040 --> 01:04:54.020 align:middle line:84% So this is crowd noise generated from the biologically plausible 01:04:54.020 --> 01:04:54.680 align:middle line:90% model. 01:04:54.680 --> 01:04:55.347 align:middle line:90% [AUDIO PLAYBACK] 01:04:55.347 --> 01:04:57.635 align:middle line:90% [CROWD NOISE] 01:04:57.635 --> 01:04:58.560 align:middle line:90% [END PLAYBACK] 01:04:58.560 --> 01:05:00.560 align:middle line:84% JOSH MCDERMOTT: And from the non-biological one. 01:05:00.560 --> 01:05:01.227 align:middle line:90% [AUDIO PLAYBACK] 01:05:01.227 --> 01:05:03.790 align:middle line:90% [DISTORTED CROWD NOISE] 01:05:03.790 --> 01:05:04.700 align:middle line:90% [END PLAYBACK] 01:05:04.700 --> 01:05:06.325 align:middle line:84% JOSH MCDERMOTT: Probably here it sounds 01:05:06.325 --> 01:05:10.060 align:middle line:84% kind of weird and a little bit garbled or something. 01:05:10.060 --> 01:05:12.994 align:middle line:84% And here's one more example, a helicopter. 01:05:12.994 --> 01:05:13.661 align:middle line:90% [AUDIO PLAYBACK] 01:05:13.661 --> 01:05:15.693 align:middle line:90% [HELICOPTER WHIRRING] 01:05:15.693 --> 01:05:17.110 align:middle line:84% JOSH MCDERMOTT: Can you hear that? 01:05:17.110 --> 01:05:17.830 align:middle line:90% AUDIENCE: Yeah. 01:05:17.830 --> 01:05:17.930 align:middle line:90% [END PLAYBACK] 01:05:17.930 --> 01:05:19.970 align:middle line:84% JOSH MCDERMOTT: And then here's from the nonbiological model. 01:05:19.970 --> 01:05:20.637 align:middle line:90% [AUDIO PLAYBACK] 01:05:20.637 --> 01:05:24.287 align:middle line:90% [WARBLY HELICOPTER WHIRRING] 01:05:24.287 --> 01:05:24.870 align:middle line:90% [END PLAYBACK] 01:05:24.870 --> 01:05:26.550 align:middle line:84% JOSH MCDERMOTT: So you can hear the modulations don't quite 01:05:26.550 --> 01:05:27.050 align:middle line:90% sound right. 01:05:27.050 --> 01:05:29.092 align:middle line:84% And that's probably because the modulation filter 01:05:29.092 --> 01:05:31.610 align:middle line:84% bank is kind of not spaced in the way that ours is. 01:05:31.610 --> 01:05:33.710 align:middle line:84% And so our sensitivity to modulation 01:05:33.710 --> 01:05:38.320 align:middle line:84% is mismatched with that of the model. 01:05:38.320 --> 01:05:42.430 align:middle line:84% And so the idea here is that the procedure 01:05:42.430 --> 01:05:43.825 align:middle line:90% is initialized with noise. 01:05:43.825 --> 01:05:45.450 align:middle line:84% So we get a different sound every time, 01:05:45.450 --> 01:05:47.470 align:middle line:84% sharing only the statistical properties. 01:05:47.470 --> 01:05:50.090 align:middle line:84% And the statistics define a class of sounds. 01:05:50.090 --> 01:05:52.450 align:middle line:84% This is same idea we talked about with visual textures. 01:05:52.450 --> 01:05:54.550 align:middle line:84% There's an equivalence class that's 01:05:54.550 --> 01:05:56.308 align:middle line:90% defined by these statistics. 01:05:56.308 --> 01:05:58.350 align:middle line:84% And so the idea is that if the statistics measure 01:05:58.350 --> 01:06:00.910 align:middle line:84% what the brain is measuring, then the samples 01:06:00.910 --> 01:06:04.430 align:middle line:84% should sound like another example of the original sound. 01:06:04.430 --> 01:06:08.290 align:middle line:84% But if the statistics don't measure 01:06:08.290 --> 01:06:09.790 align:middle line:84% what the brain is measuring, then we 01:06:09.790 --> 01:06:11.443 align:middle line:84% get a different equivalence class. 01:06:11.443 --> 01:06:13.110 align:middle line:84% So in this case, we think the statistics 01:06:13.110 --> 01:06:14.710 align:middle line:84% of that nonbiological model, they 01:06:14.710 --> 01:06:16.630 align:middle line:84% define a different class of sound. 01:06:16.630 --> 01:06:18.850 align:middle line:84% And when we run the synthesis procedure-- 01:06:18.850 --> 01:06:23.130 align:middle line:84% so both sets of sounds, like the set for what our brain is using, 01:06:23.130 --> 01:06:26.070 align:middle line:84% which is symbolized in blue, and the set for this incorrect model 01:06:26.070 --> 01:06:28.310 align:middle line:84% in red, both of them contain the original, 01:06:28.310 --> 01:06:31.390 align:middle line:84% because the original defines the set of statistics 01:06:31.390 --> 01:06:34.370 align:middle line:84% along with the model that defines that set. 01:06:34.370 --> 01:06:36.270 align:middle line:84% And when we run the synthesis, we're 01:06:36.270 --> 01:06:38.770 align:middle line:84% pulling an example from that set. 01:06:38.770 --> 01:06:40.952 align:middle line:84% And so if we've got the wrong set of statistics, 01:06:40.952 --> 01:06:42.410 align:middle line:84% we get the wrong equivalence class, 01:06:42.410 --> 01:06:46.430 align:middle line:84% and you get something that just doesn't sound right to us. 01:06:46.430 --> 01:06:48.230 align:middle line:84% So the point is that it does actually 01:06:48.230 --> 01:06:53.970 align:middle line:84% kind of matter to use a model that we think is correct. 01:06:53.970 --> 01:06:56.430 align:middle line:84% Last thing I want to show you, this 01:06:56.430 --> 01:07:00.950 align:middle line:84% is an experiment where people were played an original sound, 01:07:00.950 --> 01:07:04.870 align:middle line:84% and then a synthetic sound, like generated from the biologically 01:07:04.870 --> 01:07:05.637 align:middle line:90% plausible model. 01:07:05.637 --> 01:07:07.470 align:middle line:84% And they were just asked to rate the realism 01:07:07.470 --> 01:07:11.440 align:middle line:84% of the synthetic sound like on a scale of 1 to 7. 01:07:11.440 --> 01:07:13.360 align:middle line:84% And they did this for 170 sounds. 01:07:13.360 --> 01:07:16.440 align:middle line:84% So this is a histogram that just plots the distribution 01:07:16.440 --> 01:07:19.460 align:middle line:84% of the average realism ratings for each one of those 170 01:07:19.460 --> 01:07:22.400 align:middle line:84% sounds, like averaged across a bunch of participants. 01:07:22.400 --> 01:07:24.800 align:middle line:84% There's just two things to take away from this. 01:07:24.800 --> 01:07:28.220 align:middle line:84% The first is that there's a big bump up here kind of at the top. 01:07:28.220 --> 01:07:29.680 align:middle line:84% So that is encouraging, because it 01:07:29.680 --> 01:07:31.680 align:middle line:84% means that for a lot of sounds, people 01:07:31.680 --> 01:07:34.560 align:middle line:84% are saying that the synthesis is fairly realistic. 01:07:34.560 --> 01:07:38.080 align:middle line:84% But the second thing is that there's this tail down here. 01:07:38.080 --> 01:07:40.200 align:middle line:84% And this is quite interesting because these 01:07:40.200 --> 01:07:44.360 align:middle line:84% are cases where, despite the fact that we have matched 01:07:44.360 --> 01:07:47.280 align:middle line:84% these sounds along this big set of statistics, 01:07:47.280 --> 01:07:49.280 align:middle line:84% people are saying that they don't sound anything 01:07:49.280 --> 01:07:50.220 align:middle line:90% like the original. 01:07:50.220 --> 01:07:53.080 align:middle line:84% So it means that our brain is measuring something 01:07:53.080 --> 01:07:54.460 align:middle line:90% that the model is not. 01:07:54.460 --> 01:07:55.960 align:middle line:84% And so these failures, again, always 01:07:55.960 --> 01:07:57.800 align:middle line:84% give us kind of interesting clues 01:07:57.800 --> 01:08:01.960 align:middle line:84% for how we can make our theories better. 01:08:01.960 --> 01:08:06.960 align:middle line:84% And so this is a list of the 15 or so sounds 01:08:06.960 --> 01:08:09.120 align:middle line:90% that got the lowest rating. 01:08:09.120 --> 01:08:11.332 align:middle line:84% And just to make it easy on you, I'll 01:08:11.332 --> 01:08:13.290 align:middle line:84% put some labels next to them, because they tend 01:08:13.290 --> 01:08:17.352 align:middle line:90% to have one of three things. 01:08:17.352 --> 01:08:19.810 align:middle line:84% So they tend to either be sounds that involve pitch, sounds 01:08:19.810 --> 01:08:22.050 align:middle line:84% that have some kind of rhythm, or sounds 01:08:22.050 --> 01:08:23.569 align:middle line:90% that have reverberation. 01:08:23.569 --> 01:08:25.649 align:middle line:84% And so I'll play you both the original 01:08:25.649 --> 01:08:29.932 align:middle line:84% and the synthetic version, just to give you a flavor. 01:08:29.932 --> 01:08:30.599 align:middle line:90% [AUDIO PLAYBACK] 01:08:30.599 --> 01:08:33.074 align:middle line:90% [RAILROAD BELLS] 01:08:33.074 --> 01:08:33.892 align:middle line:90% [END PLAYBACK] 01:08:33.892 --> 01:08:34.559 align:middle line:90% [AUDIO PLAYBACK] 01:08:34.559 --> 01:08:38.024 align:middle line:90% [DISTORTED RAILROAD BELLS] 01:08:38.024 --> 01:08:39.337 align:middle line:90% [END PLAYBACK] 01:08:39.337 --> 01:08:40.004 align:middle line:90% [AUDIO PLAYBACK] 01:08:40.004 --> 01:08:43.464 align:middle line:90% [BANGING] 01:08:43.464 --> 01:08:43.964 align:middle line:90% 01:08:43.964 --> 01:08:45.277 align:middle line:90% [END PLAYBACK] 01:08:45.277 --> 01:08:45.944 align:middle line:90% [AUDIO PLAYBACK] 01:08:45.944 --> 01:08:49.409 align:middle line:90% [DISTORTED BANGING] 01:08:49.409 --> 01:08:53.197 align:middle line:90% 01:08:53.197 --> 01:08:53.864 align:middle line:90% [AUDIO PLAYBACK] 01:08:53.864 --> 01:08:56.834 align:middle line:90% [TAPPING] 01:08:56.834 --> 01:08:58.642 align:middle line:90% [END PLAYBACK] 01:08:58.642 --> 01:08:59.309 align:middle line:90% [AUDIO PLAYBACK] 01:08:59.309 --> 01:09:02.774 align:middle line:90% [DISTORTED TAPPING] 01:09:02.774 --> 01:09:03.764 align:middle line:90% 01:09:03.764 --> 01:09:05.078 align:middle line:90% [END PLAYBACK] 01:09:05.078 --> 01:09:06.870 align:middle line:84% JOSH MCDERMOTT: So there's some taps there, 01:09:06.870 --> 01:09:09.420 align:middle line:90% but the rhythm is kind of lost. 01:09:09.420 --> 01:09:12.300 align:middle line:84% And I should emphasize that it wasn't 01:09:12.300 --> 01:09:16.060 align:middle line:84% really obvious before doing the whole exercise of synthesizing 01:09:16.060 --> 01:09:18.859 align:middle line:84% the sounds that these statistics would not capture rhythm. 01:09:18.859 --> 01:09:21.359 align:middle line:84% We actually thought there was a good chance that they might. 01:09:21.359 --> 01:09:25.420 align:middle line:84% But then you go and do it, and you're forced to face the truth. 01:09:25.420 --> 01:09:27.353 align:middle line:84% It really doesn't do a great job with music. 01:09:27.353 --> 01:09:28.020 align:middle line:90% [AUDIO PLAYBACK] 01:09:28.020 --> 01:09:31.317 align:middle line:90% [MAMBO MUSIC] 01:09:31.317 --> 01:09:32.260 align:middle line:90% [END PLAYBACK] 01:09:32.260 --> 01:09:33.700 align:middle line:90% JOSH MCDERMOTT: And here you go. 01:09:33.700 --> 01:09:37.011 align:middle line:90% [DISTORTED MAMBO MUSIC] 01:09:37.011 --> 01:09:38.430 align:middle line:90% 01:09:38.430 --> 01:09:41.899 align:middle line:90% [END PLAYBACK] 01:09:41.899 --> 01:09:43.420 align:middle line:84% JOSH MCDERMOTT: So these failures 01:09:43.420 --> 01:09:44.899 align:middle line:84% are kind of exciting in the sense 01:09:44.899 --> 01:09:47.300 align:middle line:84% that they point the direction to things 01:09:47.300 --> 01:09:48.880 align:middle line:90% that we need to understand. 01:09:48.880 --> 01:09:52.819 align:middle line:84% And so this is the value of having models that 01:09:52.819 --> 01:09:54.660 align:middle line:90% can be applied to any sound. 01:09:54.660 --> 01:09:57.500 align:middle line:84% That's what we aspire to in perceptual science, 01:09:57.500 --> 01:10:00.113 align:middle line:84% because they force you to face the truth. 01:10:00.113 --> 01:10:00.780 align:middle line:90% [AUDIO PLAYBACK] 01:10:00.780 --> 01:10:01.530 align:middle line:90% [DISTORTED SPEECH] 01:10:01.530 --> 01:10:02.540 align:middle line:90% [END PLAYBACK] 01:10:02.540 --> 01:10:05.460 align:middle line:84% JOSH MCDERMOTT: So take-home messages. 01:10:05.460 --> 01:10:08.220 align:middle line:84% We talked about the idea that synthesis 01:10:08.220 --> 01:10:11.040 align:middle line:84% can help us test and explore theories of perception, 01:10:11.040 --> 01:10:13.700 align:middle line:84% and we saw that applied to texture in both vision 01:10:13.700 --> 01:10:15.420 align:middle line:90% and audition. 01:10:15.420 --> 01:10:17.300 align:middle line:84% Variables that produce compelling synthesis 01:10:17.300 --> 01:10:20.930 align:middle line:84% are things that could plausibly underlie perception. 01:10:20.930 --> 01:10:23.180 align:middle line:84% Synthesis failures are really interesting because they 01:10:23.180 --> 01:10:25.220 align:middle line:84% point the way to new variables that 01:10:25.220 --> 01:10:27.860 align:middle line:84% might be important for the perceptual system. 01:10:27.860 --> 01:10:29.740 align:middle line:84% We saw evidence that many natural sounds 01:10:29.740 --> 01:10:32.900 align:middle line:84% may be recognized with relatively simple statistics 01:10:32.900 --> 01:10:35.260 align:middle line:84% of early auditory representations. 01:10:35.260 --> 01:10:37.340 align:middle line:84% So the very simplest statistics capturing 01:10:37.340 --> 01:10:39.773 align:middle line:84% things like the spectrum and are not that informative, 01:10:39.773 --> 01:10:41.440 align:middle line:84% but if you go a little bit more complex, 01:10:41.440 --> 01:10:43.740 align:middle line:84% you get representations that are pretty powerful. 01:10:43.740 --> 01:10:46.780 align:middle line:84% We saw evidence that for sufficiently texture-like 01:10:46.780 --> 01:10:50.205 align:middle line:84% sounds, statistics may be all you have, 01:10:50.205 --> 01:10:51.580 align:middle line:84% which actually makes it difficult 01:10:51.580 --> 01:10:55.500 align:middle line:84% to discriminate different excerpts of different textures. 01:10:55.500 --> 01:10:58.700 align:middle line:84% And I'll just highlight that textures are one domain where 01:10:58.700 --> 01:11:01.660 align:middle line:84% I think we have models that do a pretty good job of accounting 01:11:01.660 --> 01:11:03.140 align:middle line:90% for perception. 01:11:03.140 --> 01:11:06.990 align:middle line:84% And we're not quite this far in really any other aspect 01:11:06.990 --> 01:11:10.230 align:middle line:90% of perception at this point. 01:11:10.230 --> 01:11:14.125 align:middle line:84% So I'm going to end there, but I'm happy to take questions. 01:11:14.125 --> 01:11:15.250 align:middle line:90% What questions do you have? 01:11:15.250 --> 01:11:21.710 align:middle line:90% 01:11:21.710 --> 01:11:22.290 align:middle line:90% Yeah. 01:11:22.290 --> 01:11:25.210 align:middle line:84% AUDIENCE: This might be kind of like a weird question, 01:11:25.210 --> 01:11:27.590 align:middle line:84% but I feel like texture makes a lot of sense 01:11:27.590 --> 01:11:30.610 align:middle line:84% in the modalities of like hearing and seeing, 01:11:30.610 --> 01:11:33.630 align:middle line:84% but are there other sensory textures? 01:11:33.630 --> 01:11:35.050 align:middle line:90% Are there like touch textures? 01:11:35.050 --> 01:11:35.530 align:middle line:90% JOSH MCDERMOTT: Oh yeah. 01:11:35.530 --> 01:11:36.890 align:middle line:84% I mean, it's super important for touch. 01:11:36.890 --> 01:11:38.770 align:middle line:84% I mean, it's one of the most important thing. 01:11:38.770 --> 01:11:44.043 align:middle line:84% I mean, just think about feeling different materials. 01:11:44.043 --> 01:11:45.710 align:middle line:84% I mean, a lot of what happens with touch 01:11:45.710 --> 01:11:49.990 align:middle line:84% is you're sweeping usually your fingertips 01:11:49.990 --> 01:11:56.070 align:middle line:84% like over some surface, and getting some statistical sense 01:11:56.070 --> 01:11:58.390 align:middle line:90% of the textural properties. 01:11:58.390 --> 01:12:01.770 align:middle line:84% Yeah, so texture is really important for touch. 01:12:01.770 --> 01:12:04.480 align:middle line:90% 01:12:04.480 --> 01:12:07.620 align:middle line:84% I mean, it's really important for eating and drinking as well, 01:12:07.620 --> 01:12:09.620 align:middle line:84% but that's also touched just inside your mouth, 01:12:09.620 --> 01:12:14.280 align:middle line:84% so maybe interacts with flavor and taste in some kind 01:12:14.280 --> 01:12:16.120 align:middle line:90% of complicated way. 01:12:16.120 --> 01:12:18.800 align:middle line:84% But yeah, vision, hearing, and touch, it's 01:12:18.800 --> 01:12:20.120 align:middle line:90% important in all of them. 01:12:20.120 --> 01:12:22.200 align:middle line:90% Yeah. 01:12:22.200 --> 01:12:23.840 align:middle line:84% And I think all of these same ideas, 01:12:23.840 --> 01:12:25.442 align:middle line:90% they could be applied to touch. 01:12:25.442 --> 01:12:27.400 align:middle line:84% It's just a lot harder because it's much harder 01:12:27.400 --> 01:12:28.660 align:middle line:90% to generate the stimuli. 01:12:28.660 --> 01:12:31.280 align:middle line:90% 01:12:31.280 --> 01:12:33.200 align:middle line:84% And this is one of many reasons why 01:12:33.200 --> 01:12:36.660 align:middle line:84% there's lots of work on vision and hearing and less on touch. 01:12:36.660 --> 01:12:39.320 align:middle line:84% It's just like just manipulating stimuli with touch 01:12:39.320 --> 01:12:41.720 align:middle line:90% is challenging-- 01:12:41.720 --> 01:12:45.040 align:middle line:90% not impossible, but hard. 01:12:45.040 --> 01:12:46.460 align:middle line:90% What other questions you got? 01:12:46.460 --> 01:12:51.560 align:middle line:90% 01:12:51.560 --> 01:12:52.280 align:middle line:90% Yeah? 01:12:52.280 --> 01:12:55.020 align:middle line:84% AUDIENCE: Can you play the English one, the last one? 01:12:55.020 --> 01:13:03.376 align:middle line:90% 01:13:03.376 --> 01:13:04.360 align:middle line:90% [AUDIO PLAYBACK] 01:13:04.360 --> 01:13:06.180 align:middle line:90% - A boy fell from the window. 01:13:06.180 --> 01:13:07.700 align:middle line:90% The wife helped her husband. 01:13:07.700 --> 01:13:09.620 align:middle line:90% Big dogs can be dangerous. 01:13:09.620 --> 01:13:11.060 align:middle line:90% Her shoes were very dirty. 01:13:11.060 --> 01:13:11.560 align:middle line:90% The-- 01:13:11.560 --> 01:13:12.347 align:middle line:90% [END PLAYBACK] 01:13:12.347 --> 01:13:13.014 align:middle line:90% [AUDIO PLAYBACK] 01:13:13.014 --> 01:13:16.360 align:middle line:90% [DISTORTED SPEECH] 01:13:16.360 --> 01:13:18.010 align:middle line:90% 01:13:18.010 --> 01:13:18.727 align:middle line:90% [END PLAYBACK] 01:13:18.727 --> 01:13:20.560 align:middle line:84% JOSH MCDERMOTT: And that's also interesting, 01:13:20.560 --> 01:13:23.080 align:middle line:84% because on the one hand, it's probably totally obvious 01:13:23.080 --> 01:13:25.960 align:middle line:90% that is supposed to be speech. 01:13:25.960 --> 01:13:29.080 align:middle line:84% So speech definitely has distinctive properties 01:13:29.080 --> 01:13:30.900 align:middle line:84% in this kind of texture representation. 01:13:30.900 --> 01:13:33.040 align:middle line:84% But it's also missing a whole bunch of stuff that 01:13:33.040 --> 01:13:34.500 align:middle line:90% is present in normal speech. 01:13:34.500 --> 01:13:38.200 align:middle line:84% So speech and music in particular 01:13:38.200 --> 01:13:40.140 align:middle line:84% involve more complicated measurements. 01:13:40.140 --> 01:13:42.328 align:middle line:84% And I think if we had the right model, 01:13:42.328 --> 01:13:44.120 align:middle line:84% we could probably generate speech and music 01:13:44.120 --> 01:13:46.750 align:middle line:90% textures in the same way. 01:13:46.750 --> 01:14:01.000 align:middle line:90%