WEBVTT 00:00:00.000 --> 00:00:02.432 align:middle line:90% [SQUEAKING] 00:00:02.432 --> 00:00:03.881 align:middle line:90% [RUSTLING] 00:00:03.881 --> 00:00:06.296 align:middle line:90% [CLICKING] 00:00:06.296 --> 00:00:08.303 align:middle line:90% 00:00:08.303 --> 00:00:09.720 align:middle line:84% JOSH MCDERMOTT: Today, we're going 00:00:09.720 --> 00:00:12.440 align:middle line:84% to talk about pitch perception, and then get into auditory scene 00:00:12.440 --> 00:00:13.840 align:middle line:90% analysis, all right? 00:00:13.840 --> 00:00:17.760 align:middle line:84% So one of the really remarkable things 00:00:17.760 --> 00:00:22.600 align:middle line:84% about the sense of hearing is that most natural sounds contain 00:00:22.600 --> 00:00:24.680 align:middle line:90% many different frequencies. 00:00:24.680 --> 00:00:26.960 align:middle line:84% And yet, they sound like one thing. 00:00:26.960 --> 00:00:31.920 align:middle line:84% So in particular, voices and instruments, among other sounds, 00:00:31.920 --> 00:00:35.240 align:middle line:84% tend to produce frequencies that are harmonics 00:00:35.240 --> 00:00:37.480 align:middle line:90% of a fundamental frequency. 00:00:37.480 --> 00:00:41.320 align:middle line:84% And so what that means is that the frequencies in the sound 00:00:41.320 --> 00:00:45.720 align:middle line:84% are integer multiples of a fundamental frequency. 00:00:45.720 --> 00:00:48.160 align:middle line:90% So here's just an example. 00:00:48.160 --> 00:00:49.900 align:middle line:84% This is a schematic power spectrum. 00:00:49.900 --> 00:00:52.160 align:middle line:90% So we've got amplitude here. 00:00:52.160 --> 00:00:54.440 align:middle line:90% And this is frequency in hertz. 00:00:54.440 --> 00:00:56.520 align:middle line:84% But the harmonics here are labeled 00:00:56.520 --> 00:00:58.520 align:middle line:90% by their harmonic number. 00:00:58.520 --> 00:01:00.080 align:middle line:84% And that's because the frequencies 00:01:00.080 --> 00:01:04.260 align:middle line:84% that are in the sound have this very regular pattern to them. 00:01:04.260 --> 00:01:06.680 align:middle line:84% So we've got the fundamental frequency 00:01:06.680 --> 00:01:09.960 align:middle line:84% down here and 2 times that, 3 times 00:01:09.960 --> 00:01:12.320 align:middle line:84% that, 4 times that, all the way up. 00:01:12.320 --> 00:01:15.280 align:middle line:84% So on a linear frequency scale, these things 00:01:15.280 --> 00:01:16.860 align:middle line:90% are regularly spaced. 00:01:16.860 --> 00:01:19.760 align:middle line:90% So these are harmonics. 00:01:19.760 --> 00:01:24.860 align:middle line:84% And so typically, when we talk about the sounds having a pitch, 00:01:24.860 --> 00:01:27.700 align:middle line:84% it usually refers to the fact that they're harmonic. 00:01:27.700 --> 00:01:30.640 align:middle line:84% And the pitch is thought to be the perceptual correlate 00:01:30.640 --> 00:01:32.420 align:middle line:90% of that fundamental frequency. 00:01:32.420 --> 00:01:34.220 align:middle line:84% So if you're playing a musical instrument 00:01:34.220 --> 00:01:36.720 align:middle line:84% and you're varying the pitch of the notes, what you're doing 00:01:36.720 --> 00:01:38.937 align:middle line:84% is varying the fundamental frequency of the notes. 00:01:38.937 --> 00:01:40.520 align:middle line:84% And that can be because you're varying 00:01:40.520 --> 00:01:44.280 align:middle line:84% the string length, for instance, or something 00:01:44.280 --> 00:01:47.120 align:middle line:90% else inside the instrument. 00:01:47.120 --> 00:01:47.620 align:middle line:90% All right. 00:01:47.620 --> 00:01:52.720 align:middle line:84% So I was just playing you an excerpt of a tune called "Summer 00:01:52.720 --> 00:01:54.880 align:middle line:90% Madness" by Kool and the Gang. 00:01:54.880 --> 00:01:56.680 align:middle line:84% And I always like showing this when 00:01:56.680 --> 00:02:00.000 align:middle line:84% we talk about pitch because this has these very 00:02:00.000 --> 00:02:01.460 align:middle line:90% prominent octave jumps. 00:02:01.460 --> 00:02:04.240 align:middle line:90% So an octave is a ratio of 2. 00:02:04.240 --> 00:02:08.372 align:middle line:84% And so when you have two notes that are separated by an octave, 00:02:08.372 --> 00:02:10.080 align:middle line:84% that means that the fundamental frequency 00:02:10.080 --> 00:02:12.360 align:middle line:90% differs by a factor of 2. 00:02:12.360 --> 00:02:14.940 align:middle line:84% And then you can see all of the harmonics kind of stretch. 00:02:14.940 --> 00:02:17.520 align:middle line:84% And so we can just look at that one more time. 00:02:17.520 --> 00:02:21.360 align:middle line:90% So here's our live spectrogram. 00:02:21.360 --> 00:02:23.440 align:middle line:90% And this is Kool and the Gang. 00:02:23.440 --> 00:02:26.912 align:middle line:84% [KOOL AND THE GANG, "SUMMER MADNESS"] 00:02:26.912 --> 00:02:53.200 align:middle line:90% 00:02:53.200 --> 00:02:57.730 align:middle line:84% And so what you hopefully saw is that, after each one 00:02:57.730 --> 00:03:00.870 align:middle line:84% of those intervals, the harmonics all doubled. 00:03:00.870 --> 00:03:02.630 align:middle line:84% So the fundamental frequency doubled. 00:03:02.630 --> 00:03:04.970 align:middle line:84% And as a consequence of that, the spacing 00:03:04.970 --> 00:03:06.810 align:middle line:90% between the harmonics doubled. 00:03:06.810 --> 00:03:14.730 align:middle line:90% 00:03:14.730 --> 00:03:17.670 align:middle line:84% So that's the nature of harmonic sounds. 00:03:17.670 --> 00:03:23.430 align:middle line:84% And so sounds can have the same fundamental frequency. 00:03:23.430 --> 00:03:25.770 align:middle line:84% And you might describe them as having the same pitch, 00:03:25.770 --> 00:03:28.350 align:middle line:84% despite them differing in other ways. 00:03:28.350 --> 00:03:30.930 align:middle line:84% So these are, again, schematic power spectra 00:03:30.930 --> 00:03:33.450 align:middle line:84% of six different sounds that all have 00:03:33.450 --> 00:03:36.290 align:middle line:84% the same fundamental frequency, despite having very 00:03:36.290 --> 00:03:37.790 align:middle line:90% different harmonic amplitudes. 00:03:37.790 --> 00:03:40.810 align:middle line:84% So we've got a piano, a trombone, tenor sax and then 00:03:40.810 --> 00:03:43.850 align:middle line:90% three spoken vowels. 00:03:43.850 --> 00:03:49.250 align:middle line:84% And so you can see that the lowest frequency is the same 00:03:49.250 --> 00:03:50.390 align:middle line:90% across all of these sounds. 00:03:50.390 --> 00:03:52.850 align:middle line:84% And then the spacing between the frequencies they contain 00:03:52.850 --> 00:03:53.490 align:middle line:90% is the same. 00:03:53.490 --> 00:03:55.730 align:middle line:84% So they've got the same harmonic series, 00:03:55.730 --> 00:03:58.250 align:middle line:84% but they've been passed through different filters. 00:03:58.250 --> 00:04:00.130 align:middle line:84% And at the end of the lecture, we're 00:04:00.130 --> 00:04:01.788 align:middle line:90% going to talk about timbre. 00:04:01.788 --> 00:04:03.330 align:middle line:84% So these sounds we would all describe 00:04:03.330 --> 00:04:05.170 align:middle line:90% as having different timbres. 00:04:05.170 --> 00:04:10.090 align:middle line:84% But they've got the same fundamental frequency, or f0. 00:04:10.090 --> 00:04:12.330 align:middle line:84% So another kind of important concept 00:04:12.330 --> 00:04:14.490 align:middle line:84% is that the fundamental frequency, it determines 00:04:14.490 --> 00:04:17.410 align:middle line:84% the period of the sound, even when 00:04:17.410 --> 00:04:19.010 align:middle line:90% it's not physically present. 00:04:19.010 --> 00:04:23.690 align:middle line:84% So this is a sound that has a missing fundamental. 00:04:23.690 --> 00:04:27.410 align:middle line:84% And it consists of the second, the third, 00:04:27.410 --> 00:04:31.010 align:middle line:84% and the fourth harmonic of 250 hertz. 00:04:31.010 --> 00:04:33.970 align:middle line:84% So the second harmonic of 250 hertz 00:04:33.970 --> 00:04:36.330 align:middle line:90% is 500 hertz, so twice that. 00:04:36.330 --> 00:04:37.990 align:middle line:90% Third harmonic is 750. 00:04:37.990 --> 00:04:40.090 align:middle line:90% And the fourth is 1,000. 00:04:40.090 --> 00:04:48.690 align:middle line:84% And so you can see that, within this period of the sound, 00:04:48.690 --> 00:04:52.370 align:middle line:84% the 500 hertz component has two cycles. 00:04:52.370 --> 00:04:55.370 align:middle line:84% The 750 component has three cycles. 00:04:55.370 --> 00:04:57.990 align:middle line:84% And the 1,000 hertz component has four cycles. 00:04:57.990 --> 00:05:00.290 align:middle line:84% Now, if you add them all together 00:05:00.290 --> 00:05:02.890 align:middle line:84% to get the sound that this would correspond to, 00:05:02.890 --> 00:05:04.510 align:middle line:90% this is the waveform. 00:05:04.510 --> 00:05:07.690 align:middle line:84% So this is the sum of these three sinusoids. 00:05:07.690 --> 00:05:09.610 align:middle line:90% And it's got some funny shape. 00:05:09.610 --> 00:05:13.610 align:middle line:84% But you can see that the period of the sound-- so the time 00:05:13.610 --> 00:05:17.770 align:middle line:84% interval over which the sound exerts one cycle after which it 00:05:17.770 --> 00:05:23.250 align:middle line:84% repeats-- is 4 milliseconds, which 00:05:23.250 --> 00:05:26.930 align:middle line:84% is 1 over the fundamental frequency of 250 hertz. 00:05:26.930 --> 00:05:30.650 align:middle line:84% So the fundamental frequency of this sound 00:05:30.650 --> 00:05:33.690 align:middle line:84% is 250 hertz, even though the sound does not actually 00:05:33.690 --> 00:05:37.890 align:middle line:90% contain any power at 250 hertz. 00:05:37.890 --> 00:05:40.450 align:middle line:84% So that 250 hertz component is missing 00:05:40.450 --> 00:05:42.530 align:middle line:90% physically from the sound. 00:05:42.530 --> 00:05:46.650 align:middle line:84% So the fundamental frequency is a more abstract notion. 00:05:46.650 --> 00:05:50.390 align:middle line:84% It describes the repetition rate of the sound. 00:05:50.390 --> 00:05:54.210 align:middle line:90% 00:05:54.210 --> 00:05:57.290 align:middle line:84% And so the classical notion of pitch 00:05:57.290 --> 00:05:59.490 align:middle line:84% is that it is the perceptual analog 00:05:59.490 --> 00:06:03.650 align:middle line:84% of the fundamental frequency, or f0. 00:06:03.650 --> 00:06:05.210 align:middle line:84% And we typically think that sounds 00:06:05.210 --> 00:06:08.450 align:middle line:84% that have the same periodicity, which equivalently 00:06:08.450 --> 00:06:13.650 align:middle line:84% means they have the same f0, usually share the pitch. 00:06:13.650 --> 00:06:14.790 align:middle line:90% So here are some examples. 00:06:14.790 --> 00:06:16.450 align:middle line:90% So here's a pure tone. 00:06:16.450 --> 00:06:17.850 align:middle line:90% [PURE TONE] 00:06:17.850 --> 00:06:20.330 align:middle line:90% Here's a harmonic complex tone. 00:06:20.330 --> 00:06:22.290 align:middle line:90% [HARMONIC COMPLEX TONE] 00:06:22.290 --> 00:06:25.552 align:middle line:84% So this harmonic complex tone has the fundamental frequency 00:06:25.552 --> 00:06:27.510 align:middle line:84% that's equal to the frequency of the pure tone. 00:06:27.510 --> 00:06:30.530 align:middle line:84% But then it's got all these other harmonics on top of it. 00:06:30.530 --> 00:06:32.610 align:middle line:84% And so the waveforms, which are shown here, 00:06:32.610 --> 00:06:33.987 align:middle line:90% look really different. 00:06:33.987 --> 00:06:35.570 align:middle line:84% But when you listen to them, you might 00:06:35.570 --> 00:06:37.403 align:middle line:84% think there's something they have in common. 00:06:37.403 --> 00:06:40.250 align:middle line:84% Now, in the bottom it's that same harmonic complex, 00:06:40.250 --> 00:06:42.970 align:middle line:84% but without the fundamental component. 00:06:42.970 --> 00:06:48.130 align:middle line:84% So the fundamental frequency of the sound is still that same f0, 00:06:48.130 --> 00:06:51.370 align:middle line:84% but the F0 is now physically missing from the sound. 00:06:51.370 --> 00:06:54.890 align:middle line:90% [TWO NOTES] 00:06:54.890 --> 00:06:55.390 align:middle line:90% All right. 00:06:55.390 --> 00:06:57.890 align:middle line:84% So you can tell that those sounds are not exactly the same, 00:06:57.890 --> 00:07:00.110 align:middle line:84% but there's some quality that they have in common. 00:07:00.110 --> 00:07:03.450 align:middle line:84% And we would often call that quality the pitch. 00:07:03.450 --> 00:07:05.630 align:middle line:90% So why does pitch matter? 00:07:05.630 --> 00:07:08.710 align:middle line:84% Well, it's a diagnostic property of some sounds. 00:07:08.710 --> 00:07:12.850 align:middle line:84% And for some familiar sounds, you know what their pitch is. 00:07:12.850 --> 00:07:17.370 align:middle line:84% And voices are a good example of this. 00:07:17.370 --> 00:07:22.610 align:middle line:84% So this is a demo, where I have a recording 00:07:22.610 --> 00:07:25.810 align:middle line:84% of a person whose voice probably all of you 00:07:25.810 --> 00:07:27.610 align:middle line:90% are fairly familiar with. 00:07:27.610 --> 00:07:30.610 align:middle line:84% And then I have pitch shifted it artificially. 00:07:30.610 --> 00:07:34.690 align:middle line:84% So I'm going to play you the pitch-shifted versions. 00:07:34.690 --> 00:07:37.370 align:middle line:84% And then we'll play you the correct version. 00:07:37.370 --> 00:07:38.982 align:middle line:84% Raise your hand-- don't say who you 00:07:38.982 --> 00:07:40.690 align:middle line:84% think this is, but raise your hand if you 00:07:40.690 --> 00:07:42.743 align:middle line:90% think you can tell who this is. 00:07:42.743 --> 00:07:43.410 align:middle line:90% [AUDIO PLAYBACK] 00:07:43.410 --> 00:07:45.535 align:middle line:84% - There are a lot of points I want to make tonight, 00:07:45.535 --> 00:07:47.217 align:middle line:90% but the most important one-- 00:07:47.217 --> 00:07:47.800 align:middle line:90% [END PLAYBACK] 00:07:47.800 --> 00:07:49.633 align:middle line:84% JOSH MCDERMOTT: OK, so a couple of you know. 00:07:49.633 --> 00:07:51.673 align:middle line:84% All right, let's play the really low one. 00:07:51.673 --> 00:07:52.340 align:middle line:90% [AUDIO PLAYBACK] 00:07:52.340 --> 00:07:54.465 align:middle line:84% - There are a lot of points I want to make tonight. 00:07:54.465 --> 00:07:55.980 align:middle line:90% But the most important one-- 00:07:55.980 --> 00:07:56.740 align:middle line:90% [END PLAYBACK] 00:07:56.740 --> 00:07:59.393 align:middle line:84% JOSH MCDERMOTT: OK, let's go a little closer to the true pitch. 00:07:59.393 --> 00:08:00.060 align:middle line:90% [AUDIO PLAYBACK] 00:08:00.060 --> 00:08:02.185 align:middle line:84% - There are a lot of points I want to make tonight, 00:08:02.185 --> 00:08:04.828 align:middle line:90% but the most important one-- 00:08:04.828 --> 00:08:05.804 align:middle line:90% [END PLAYBACK] 00:08:05.804 --> 00:08:06.515 align:middle line:90% [AUDIO PLAYBACK] 00:08:06.515 --> 00:08:08.640 align:middle line:84% - There are a lot of points I want to make tonight, 00:08:08.640 --> 00:08:10.197 align:middle line:90% but the most important one-- 00:08:10.197 --> 00:08:10.780 align:middle line:90% [END PLAYBACK] 00:08:10.780 --> 00:08:12.040 align:middle line:84% JOSH MCDERMOTT: All right, how many people know who that is? 00:08:12.040 --> 00:08:13.200 align:middle line:90% Raise your hand. 00:08:13.200 --> 00:08:14.200 align:middle line:90% All right, a few of you. 00:08:14.200 --> 00:08:15.753 align:middle line:90% But then here's the correct one. 00:08:15.753 --> 00:08:16.420 align:middle line:90% [AUDIO PLAYBACK] 00:08:16.420 --> 00:08:18.545 align:middle line:84% - There are a lot of points I want to make tonight, 00:08:18.545 --> 00:08:22.660 align:middle line:84% but the most important one is that, 20 years ago, I 00:08:22.660 --> 00:08:25.540 align:middle line:84% became the luckiest man on Earth because Michelle Obama agreed 00:08:25.540 --> 00:08:26.360 align:middle line:90% to marry me. 00:08:26.360 --> 00:08:26.620 align:middle line:90% [END PLAYBACK] 00:08:26.620 --> 00:08:29.037 align:middle line:84% JOSH MCDERMOTT: All right, so that's our former president. 00:08:29.037 --> 00:08:30.850 align:middle line:90% 00:08:30.850 --> 00:08:32.350 align:middle line:84% And I think probably you would agree 00:08:32.350 --> 00:08:34.267 align:middle line:84% that it's a little more recognizable when it's 00:08:34.267 --> 00:08:35.520 align:middle line:90% played at the correct pitch. 00:08:35.520 --> 00:08:37.419 align:middle line:84% And even those of you who could recognize 00:08:37.419 --> 00:08:39.900 align:middle line:84% that that was Barack Obama could probably tell 00:08:39.900 --> 00:08:41.520 align:middle line:90% that the pitch was not correct. 00:08:41.520 --> 00:08:47.380 align:middle line:84% So you know what the pitch is of familiar voices. 00:08:47.380 --> 00:08:48.820 align:middle line:84% This is kind of a cool experiment 00:08:48.820 --> 00:08:50.740 align:middle line:84% that actually shows that you rely 00:08:50.740 --> 00:08:53.280 align:middle line:90% on pitch to recognize voices. 00:08:53.280 --> 00:08:56.720 align:middle line:84% So this was an experiment conducted by Malinda McPherson, 00:08:56.720 --> 00:08:59.980 align:middle line:84% a former graduate student here, who measured the ability 00:08:59.980 --> 00:09:01.880 align:middle line:90% to recognize celebrity voices. 00:09:01.880 --> 00:09:05.340 align:middle line:84% So she took all these recordings of famous people, 00:09:05.340 --> 00:09:08.660 align:middle line:84% did an experiment where participants heard excerpts 00:09:08.660 --> 00:09:09.980 align:middle line:90% of these people talking. 00:09:09.980 --> 00:09:13.580 align:middle line:84% And those excerpts could either be played at the original pitch 00:09:13.580 --> 00:09:16.460 align:middle line:84% or could be pitch-shifted up or down by up to an octave 00:09:16.460 --> 00:09:18.027 align:middle line:90% in either direction. 00:09:18.027 --> 00:09:19.860 align:middle line:84% People would hear the excerpt, and then they 00:09:19.860 --> 00:09:21.800 align:middle line:84% would type in who they thought it was. 00:09:21.800 --> 00:09:22.900 align:middle line:90% And that was coded. 00:09:22.900 --> 00:09:26.300 align:middle line:84% And so this graph plots percent correct recognition 00:09:26.300 --> 00:09:28.220 align:middle line:84% as a function of the pitch shift. 00:09:28.220 --> 00:09:31.340 align:middle line:84% And so you can see that, when it's not shifted at all, 00:09:31.340 --> 00:09:34.320 align:middle line:84% people are recognizing a little over 50% of the celebrities. 00:09:34.320 --> 00:09:36.335 align:middle line:84% And that's just because lots of celebrities 00:09:36.335 --> 00:09:38.460 align:middle line:84% only some people know because not everybody watches 00:09:38.460 --> 00:09:41.060 align:middle line:84% the same TV shows and stuff like that. 00:09:41.060 --> 00:09:43.540 align:middle line:84% But as you shift the pitch, the recognition 00:09:43.540 --> 00:09:45.200 align:middle line:90% drops fairly precipitously. 00:09:45.200 --> 00:09:48.520 align:middle line:84% And even a pitch shift of three semitones-- 00:09:48.520 --> 00:09:52.160 align:middle line:84% that's a quarter of an octave-- is enough to give you a deficit. 00:09:52.160 --> 00:09:55.460 align:middle line:90% 00:09:55.460 --> 00:09:58.140 align:middle line:84% So it's also often the case that information 00:09:58.140 --> 00:10:01.240 align:middle line:84% is conveyed by the way that pitch changes over time. 00:10:01.240 --> 00:10:03.660 align:middle line:84% And so that's certainly also very important in pitch 00:10:03.660 --> 00:10:04.720 align:middle line:90% and also in music. 00:10:04.720 --> 00:10:07.340 align:middle line:84% So prosody is the word that we use 00:10:07.340 --> 00:10:09.960 align:middle line:84% to refer to the pitch changes that occur in music. 00:10:09.960 --> 00:10:13.380 align:middle line:84% They often convey emotion and emphasis. 00:10:13.380 --> 00:10:15.833 align:middle line:90% So here's a particular example. 00:10:15.833 --> 00:10:16.500 align:middle line:90% [AUDIO PLAYBACK] 00:10:16.500 --> 00:10:17.677 align:middle line:90% - 903. 00:10:17.677 --> 00:10:18.260 align:middle line:90% [END PLAYBACK] 00:10:18.260 --> 00:10:19.843 align:middle line:84% JOSH MCDERMOTT: OK, so that's a person 00:10:19.843 --> 00:10:23.537 align:middle line:84% who is saying a number that doesn't 00:10:23.537 --> 00:10:25.120 align:middle line:84% have any particular emotional content. 00:10:25.120 --> 00:10:28.220 align:middle line:84% But you can tell they're really excited. 00:10:28.220 --> 00:10:28.873 align:middle line:90% Conversely-- 00:10:28.873 --> 00:10:29.540 align:middle line:90% [AUDIO PLAYBACK] 00:10:29.540 --> 00:10:30.957 align:middle line:90% - December 4. 00:10:30.957 --> 00:10:31.540 align:middle line:90% [END PLAYBACK] 00:10:31.540 --> 00:10:33.460 align:middle line:84% JOSH MCDERMOTT: That person is really sad. 00:10:33.460 --> 00:10:35.020 align:middle line:90% They're just saying a date. 00:10:35.020 --> 00:10:38.600 align:middle line:84% But importantly, if we just shift the pitch up, 00:10:38.600 --> 00:10:41.140 align:middle line:84% so the pitch gets much higher, you 00:10:41.140 --> 00:10:43.240 align:middle line:84% can still kind of hear the emotion in their voice, 00:10:43.240 --> 00:10:45.960 align:middle line:84% because it's conveyed by the way the pitch is changing over time. 00:10:45.960 --> 00:10:46.627 align:middle line:90% [AUDIO PLAYBACK] 00:10:46.627 --> 00:10:48.000 align:middle line:90% - 903. 00:10:48.000 --> 00:10:48.786 align:middle line:90% [END PLAYBACK] 00:10:48.786 --> 00:10:49.453 align:middle line:90% [AUDIO PLAYBACK] 00:10:49.453 --> 00:10:51.077 align:middle line:90% - December 4. 00:10:51.077 --> 00:10:51.660 align:middle line:90% [END PLAYBACK] 00:10:51.660 --> 00:10:54.820 align:middle line:84% JOSH MCDERMOTT: OK, so that's relative pitch. 00:10:54.820 --> 00:10:55.320 align:middle line:90% All right. 00:10:55.320 --> 00:10:57.900 align:middle line:84% So classically, the problem of pitch perception 00:10:57.900 --> 00:11:00.740 align:middle line:84% has been construed as trying to understand 00:11:00.740 --> 00:11:07.340 align:middle line:84% how the brain estimates the f0 of a sound from its harmonics. 00:11:07.340 --> 00:11:11.580 align:middle line:84% And part of the idea of that is that a sound that has an f0 00:11:11.580 --> 00:11:12.480 align:middle line:90% is harmonic. 00:11:12.480 --> 00:11:14.980 align:middle line:84% It consists of these different frequency components. 00:11:14.980 --> 00:11:20.500 align:middle line:84% And the ear-- we often think of the ear as kind 00:11:20.500 --> 00:11:22.858 align:middle line:84% of doing a frequency analysis of a sound. 00:11:22.858 --> 00:11:25.400 align:middle line:84% So the sound comes into the ear, it's just a waveform, right? 00:11:25.400 --> 00:11:27.600 align:middle line:84% But it contains these discrete frequency components. 00:11:27.600 --> 00:11:30.560 align:middle line:84% And the ear is doing an analysis of the frequency content. 00:11:30.560 --> 00:11:33.780 align:middle line:84% And so, naively, you might think that the ear takes 00:11:33.780 --> 00:11:36.860 align:middle line:84% that sound, takes it apart into these constituent frequencies. 00:11:36.860 --> 00:11:38.740 align:middle line:84% And then you have to determine the properties 00:11:38.740 --> 00:11:43.300 align:middle line:84% of the sound, one of which is the fundamental frequency. 00:11:43.300 --> 00:11:43.800 align:middle line:90% All right. 00:11:43.800 --> 00:11:47.780 align:middle line:84% And so one key idea in all of this 00:11:47.780 --> 00:11:50.220 align:middle line:84% is that the information that we have 00:11:50.220 --> 00:11:52.260 align:middle line:84% about the harmonics and the sound 00:11:52.260 --> 00:11:55.380 align:middle line:84% is limited by the frequency resolution of the cochlea. 00:11:55.380 --> 00:11:59.060 align:middle line:84% So what's shown here is a schematic power spectrum 00:11:59.060 --> 00:12:00.033 align:middle line:90% of a harmonic sound. 00:12:00.033 --> 00:12:01.700 align:middle line:84% This is a harmonic sound that's actually 00:12:01.700 --> 00:12:03.100 align:middle line:90% missing the fundamental. 00:12:03.100 --> 00:12:05.300 align:middle line:84% But you have all of the other harmonics. 00:12:05.300 --> 00:12:07.900 align:middle line:84% And this is a schematic of the filters 00:12:07.900 --> 00:12:11.060 align:middle line:84% that we think of as being in your cochlea. 00:12:11.060 --> 00:12:13.580 align:middle line:84% So remember how, when we learned about the filters 00:12:13.580 --> 00:12:17.100 align:middle line:84% in your cochlea, we talked about how, in absolute terms, 00:12:17.100 --> 00:12:20.340 align:middle line:84% the filters at the low end of the cochlea-- so actually, 00:12:20.340 --> 00:12:22.640 align:middle line:84% at the apex, but tuned to low frequencies-- 00:12:22.640 --> 00:12:24.840 align:middle line:84% those tend to be narrower in hertz. 00:12:24.840 --> 00:12:28.860 align:middle line:84% And then as you move up the spectrum, the filters get wider. 00:12:28.860 --> 00:12:30.213 align:middle line:90% And that's what's shown here. 00:12:30.213 --> 00:12:32.380 align:middle line:84% So down here at the low end, the filters are narrow. 00:12:32.380 --> 00:12:35.260 align:middle line:84% And they get wider as you move up the spectrum. 00:12:35.260 --> 00:12:38.230 align:middle line:84% So let's think about the representation 00:12:38.230 --> 00:12:41.790 align:middle line:84% of this harmonic sound when passed through these filters. 00:12:41.790 --> 00:12:43.550 align:middle line:84% So what's shown at the top is what's 00:12:43.550 --> 00:12:45.550 align:middle line:90% called an excitation pattern. 00:12:45.550 --> 00:12:48.750 align:middle line:84% Think of this as the magnitude of the response 00:12:48.750 --> 00:12:51.030 align:middle line:90% at each place along the cochlea. 00:12:51.030 --> 00:12:53.670 align:middle line:84% And you can see that you get these peaks 00:12:53.670 --> 00:12:57.350 align:middle line:84% and valleys here down at the apical end 00:12:57.350 --> 00:13:00.310 align:middle line:84% of the cochlea in response to some of the individual frequency 00:13:00.310 --> 00:13:00.930 align:middle line:90% components. 00:13:00.930 --> 00:13:04.490 align:middle line:84% But you can also see that those peaks and valleys peter out. 00:13:04.490 --> 00:13:06.770 align:middle line:84% So the higher numbered harmonics, 00:13:06.770 --> 00:13:11.230 align:middle line:84% you don't produce these discernible peaks and valleys. 00:13:11.230 --> 00:13:12.630 align:middle line:90% So why is this? 00:13:12.630 --> 00:13:16.710 align:middle line:84% Well, let's think about the interactions of these harmonics 00:13:16.710 --> 00:13:18.390 align:middle line:90% with the filters. 00:13:18.390 --> 00:13:21.950 align:middle line:90% So this is harmonics 2 and 3. 00:13:21.950 --> 00:13:24.430 align:middle line:84% Harmonics 2 and 3 are actually quite widely spaced 00:13:24.430 --> 00:13:26.790 align:middle line:84% compared to the cochlear filter bank. 00:13:26.790 --> 00:13:31.630 align:middle line:84% And so the consequence is that they hit different filters. 00:13:31.630 --> 00:13:37.482 align:middle line:84% And so in particular, in between them, there's a filter that-- 00:13:37.482 --> 00:13:38.690 align:middle line:90% there's another filter there. 00:13:38.690 --> 00:13:40.030 align:middle line:90% That's the purple one. 00:13:40.030 --> 00:13:42.490 align:middle line:84% And so you get a peak and a trough there. 00:13:42.490 --> 00:13:46.510 align:middle line:84% And so that's true for a lot of these low-numbered harmonics. 00:13:46.510 --> 00:13:52.070 align:middle line:84% So we often call these harmonics resolved because they produce 00:13:52.070 --> 00:13:55.950 align:middle line:84% these discernible peaks in the excitation 00:13:55.950 --> 00:13:58.710 align:middle line:90% pattern of the cochlea. 00:13:58.710 --> 00:14:02.950 align:middle line:84% By contrast, if you look up here at the high-numbered harmonics, 00:14:02.950 --> 00:14:06.510 align:middle line:84% you've got two filters here that are separated 00:14:06.510 --> 00:14:08.930 align:middle line:90% by the same number of hertz. 00:14:08.930 --> 00:14:10.550 align:middle line:84% And so they have the same separation 00:14:10.550 --> 00:14:11.770 align:middle line:90% relative to the harmonics. 00:14:11.770 --> 00:14:12.930 align:middle line:90% But they're much broader. 00:14:12.930 --> 00:14:15.470 align:middle line:84% And so now, the filters up there at the high 00:14:15.470 --> 00:14:19.870 align:middle line:84% are looking at a whole bunch of harmonics at once 00:14:19.870 --> 00:14:22.630 align:middle line:84% And so the consequence is that the responses 00:14:22.630 --> 00:14:27.110 align:middle line:84% of different filters actually varies quite minimally. 00:14:27.110 --> 00:14:30.110 align:middle line:84% So even though you have a harmonic here 00:14:30.110 --> 00:14:32.910 align:middle line:84% and a harmonic here, there's very little variation 00:14:32.910 --> 00:14:35.840 align:middle line:90% in the response. 00:14:35.840 --> 00:14:38.230 align:middle line:84% And so this regime is classically referred to 00:14:38.230 --> 00:14:39.870 align:middle line:90% as being unresolved. 00:14:39.870 --> 00:14:43.070 align:middle line:84% So you're not able to resolve the individual components 00:14:43.070 --> 00:14:46.470 align:middle line:84% because of the coarseness of the tuning of the filter. 00:14:46.470 --> 00:14:49.150 align:middle line:90% 00:14:49.150 --> 00:14:49.650 align:middle line:90% All right. 00:14:49.650 --> 00:14:51.950 align:middle line:84% So this is that same kind of concept depicted 00:14:51.950 --> 00:14:52.922 align:middle line:90% in a different picture. 00:14:52.922 --> 00:14:54.630 align:middle line:84% So now, we've got like the power spectrum 00:14:54.630 --> 00:14:56.330 align:middle line:84% up top, pass through the filter bank, 00:14:56.330 --> 00:14:57.890 align:middle line:90% you get the excitation pattern. 00:14:57.890 --> 00:15:01.710 align:middle line:84% The addition here is now, for different places 00:15:01.710 --> 00:15:05.150 align:middle line:84% along the cochlea, we're looking at the basilar membrane 00:15:05.150 --> 00:15:08.590 align:middle line:90% vibration over time 00:15:08.590 --> 00:15:09.090 align:middle line:90% All right. 00:15:09.090 --> 00:15:12.350 align:middle line:84% And there's a few important things about this picture. 00:15:12.350 --> 00:15:15.150 align:middle line:84% So the first thing to note is that, down here 00:15:15.150 --> 00:15:17.070 align:middle line:84% at the apical end of the cochlea where 00:15:17.070 --> 00:15:22.830 align:middle line:84% it's tuned to low frequencies, if you look at a place that's 00:15:22.830 --> 00:15:25.150 align:middle line:84% being stimulated by a single harmonic, 00:15:25.150 --> 00:15:28.310 align:middle line:84% you can see that the basilar membrane vibration is dominated 00:15:28.310 --> 00:15:29.170 align:middle line:90% by that harmonic. 00:15:29.170 --> 00:15:32.430 align:middle line:90% It looks like a sinusoid. 00:15:32.430 --> 00:15:35.230 align:middle line:84% So in this sense, the cochlea is really picking out 00:15:35.230 --> 00:15:38.788 align:middle line:90% that one low-numbered harmonic. 00:15:38.788 --> 00:15:40.330 align:middle line:84% Here's another place a little way up. 00:15:40.330 --> 00:15:41.870 align:middle line:90% It's the same kind of thing. 00:15:41.870 --> 00:15:46.150 align:middle line:84% But if you move a little ways up further, what you're seeing 00:15:46.150 --> 00:15:49.430 align:middle line:84% is a response that's no longer purely sinusoidal. 00:15:49.430 --> 00:15:51.370 align:middle line:84% In fact, there's amplitude modulation. 00:15:51.370 --> 00:15:55.370 align:middle line:84% Anybody remember what we call that? 00:15:55.370 --> 00:15:56.960 align:middle line:90% AUDIENCE: Beating. 00:15:56.960 --> 00:15:58.210 align:middle line:90% JOSH MCDERMOTT: Beating, yeah. 00:15:58.210 --> 00:15:59.570 align:middle line:90% So we're getting some beating. 00:15:59.570 --> 00:16:02.910 align:middle line:84% Remember, beating happens whenever there are two frequency 00:16:02.910 --> 00:16:06.210 align:middle line:84% components that are getting passed through the same-- 00:16:06.210 --> 00:16:07.192 align:middle line:90% so we hear beating. 00:16:07.192 --> 00:16:09.150 align:middle line:84% And it gets registered by the ear whenever they 00:16:09.150 --> 00:16:10.650 align:middle line:90% pass through the same filter. 00:16:10.650 --> 00:16:13.002 align:middle line:84% And that causes amplitude modulation. 00:16:13.002 --> 00:16:14.210 align:middle line:90% So that's what you're seeing. 00:16:14.210 --> 00:16:17.270 align:middle line:84% So this is beating in the response of the ear 00:16:17.270 --> 00:16:18.550 align:middle line:90% to the sound. 00:16:18.550 --> 00:16:20.710 align:middle line:84% And then as we move up the spectrum, 00:16:20.710 --> 00:16:22.550 align:middle line:84% the beating becomes more and more pronounced 00:16:22.550 --> 00:16:24.790 align:middle line:84% because there are more and more harmonics 00:16:24.790 --> 00:16:29.710 align:middle line:84% that are essentially making it through the same filter. 00:16:29.710 --> 00:16:33.530 align:middle line:84% So up here at the basal end of the cochlea, 00:16:33.530 --> 00:16:36.190 align:middle line:84% when you're responding to high frequencies, 00:16:36.190 --> 00:16:38.790 align:middle line:84% the harmonics are unresolved, in the sense 00:16:38.790 --> 00:16:42.230 align:middle line:84% that the pattern of excitation along the cochlea 00:16:42.230 --> 00:16:43.990 align:middle line:90% is undifferentiated. 00:16:43.990 --> 00:16:46.910 align:middle line:84% But you do have this big temporal signal here. 00:16:46.910 --> 00:16:49.510 align:middle line:84% And in fact, the beating here happens 00:16:49.510 --> 00:16:52.470 align:middle line:90% at the fundamental frequency. 00:16:52.470 --> 00:16:56.710 align:middle line:84% So this has got the same period as the original sound. 00:16:56.710 --> 00:17:01.990 align:middle line:84% So big picture is that there's these two cues that pop out 00:17:01.990 --> 00:17:04.510 align:middle line:84% of these kinds of diagrams as being 00:17:04.510 --> 00:17:07.710 align:middle line:84% plausible sources of information about the fundamental frequency 00:17:07.710 --> 00:17:08.990 align:middle line:90% of a sound. 00:17:08.990 --> 00:17:11.810 align:middle line:84% One is what are called place cues. 00:17:11.810 --> 00:17:13.230 align:middle line:84% So these are the peaks and valleys 00:17:13.230 --> 00:17:15.109 align:middle line:90% in the excitation pattern. 00:17:15.109 --> 00:17:18.530 align:middle line:84% And the other are timing cues here, 00:17:18.530 --> 00:17:21.430 align:middle line:84% which exist even for these high-numbered unresolved 00:17:21.430 --> 00:17:24.109 align:middle line:90% harmonics. 00:17:24.109 --> 00:17:27.960 align:middle line:84% So now, I'm going to tell you a fact about pitch perception. 00:17:27.960 --> 00:17:33.100 align:middle line:84% And one very salient fact is that pitch discrimination 00:17:33.100 --> 00:17:37.360 align:middle line:84% turns out to be better in humans when resolved harmonics are 00:17:37.360 --> 00:17:39.280 align:middle line:90% present in the sound. 00:17:39.280 --> 00:17:41.560 align:middle line:84% And so you can do an experiment where 00:17:41.560 --> 00:17:45.280 align:middle line:84% you vary the harmonics that are present in a sound. 00:17:45.280 --> 00:17:47.713 align:middle line:84% And you can measure discrimination thresholds. 00:17:47.713 --> 00:17:49.880 align:middle line:84% So that would be like the just noticeable difference 00:17:49.880 --> 00:17:51.020 align:middle line:90% in fundamental frequency. 00:17:51.020 --> 00:17:53.760 align:middle line:84% It's how much you have to increase the pitch for somebody 00:17:53.760 --> 00:17:56.000 align:middle line:90% to tell that it went up or down. 00:17:56.000 --> 00:17:58.860 align:middle line:84% So that's what's plotted here on the y-axis. 00:17:58.860 --> 00:18:01.100 align:middle line:84% This is the fundamental frequency difference 00:18:01.100 --> 00:18:04.680 align:middle line:84% that is the threshold for a bunch of humans. 00:18:04.680 --> 00:18:06.420 align:middle line:84% There are two different frequency ranges, 00:18:06.420 --> 00:18:07.940 align:middle line:84% which for our purposes doesn't really matter. 00:18:07.940 --> 00:18:09.398 align:middle line:84% And then the critical variable here 00:18:09.398 --> 00:18:11.080 align:middle line:84% is what's plotted on the x-axis, which 00:18:11.080 --> 00:18:15.520 align:middle line:84% is the lowest harmonic number in the stimulus. 00:18:15.520 --> 00:18:18.040 align:middle line:84% And so if the lowest harmonic number is 3, 00:18:18.040 --> 00:18:21.540 align:middle line:84% that means that stimulus contains harmonics 3, 4, 5, 6, 00:18:21.540 --> 00:18:23.680 align:middle line:90% 7, up to some large number. 00:18:23.680 --> 00:18:25.838 align:middle line:84% If the lowest harmonic number is 10, 00:18:25.838 --> 00:18:27.880 align:middle line:84% that means that you start with the 10th harmonic. 00:18:27.880 --> 00:18:30.220 align:middle line:84% And you have 10, 11, 12, 13, 14, 15. 00:18:30.220 --> 00:18:34.040 align:middle line:84% And so what is evident from this results graph 00:18:34.040 --> 00:18:37.520 align:middle line:84% is that thresholds tend to be pretty good when 00:18:37.520 --> 00:18:40.920 align:middle line:84% the stimulus includes low-numbered harmonics. 00:18:40.920 --> 00:18:44.042 align:middle line:84% And then as you start to lose those low-numbered harmonics-- 00:18:44.042 --> 00:18:46.000 align:middle line:84% so when the lowest harmonic number is 8, or 10, 00:18:46.000 --> 00:18:48.700 align:middle line:84% or 12, or 15-- the thresholds increase. 00:18:48.700 --> 00:18:53.440 align:middle line:90% 00:18:53.440 --> 00:18:53.940 align:middle line:90% All right. 00:18:53.940 --> 00:18:55.760 align:middle line:90% So here's a demo of that. 00:18:55.760 --> 00:18:59.500 align:middle line:84% So this is an example trial from an experiment. 00:18:59.500 --> 00:19:02.080 align:middle line:84% And so in this particular experiment, 00:19:02.080 --> 00:19:04.000 align:middle line:90% there are three stimuli. 00:19:04.000 --> 00:19:05.760 align:middle line:84% And you have to say which of them 00:19:05.760 --> 00:19:09.020 align:middle line:84% has a different pitch from the others. 00:19:09.020 --> 00:19:12.370 align:middle line:84% So let's start with something easy. 00:19:12.370 --> 00:19:16.560 align:middle line:90% [THREE PITCHES] 00:19:16.560 --> 00:19:17.740 align:middle line:90% 1, 2, or 3? 00:19:17.740 --> 00:19:18.320 align:middle line:90% AUDIENCE: 3. 00:19:18.320 --> 00:19:19.500 align:middle line:90% JOSH MCDERMOTT: 3, good. 00:19:19.500 --> 00:19:23.260 align:middle line:84% So this was a stimulus that contained resolved harmonics. 00:19:23.260 --> 00:19:27.200 align:middle line:84% So I'm not going to go through the details of exactly 00:19:27.200 --> 00:19:28.942 align:middle line:90% how these stimuli were made. 00:19:28.942 --> 00:19:30.400 align:middle line:84% But they're passed through a filter 00:19:30.400 --> 00:19:33.000 align:middle line:84% and manipulated so that you've either 00:19:33.000 --> 00:19:36.157 align:middle line:84% got low-numbered harmonics that are resolved, 00:19:36.157 --> 00:19:38.240 align:middle line:84% thought to be resolved, or high-numbered harmonics 00:19:38.240 --> 00:19:39.340 align:middle line:90% that we call unresolved. 00:19:39.340 --> 00:19:42.600 align:middle line:84% So that's a 10% difference in fundamental frequency. 00:19:42.600 --> 00:19:45.540 align:middle line:84% So just for comparison, a semitone, 00:19:45.540 --> 00:19:48.360 align:middle line:84% which is like the difference between adjacent white and black 00:19:48.360 --> 00:19:51.140 align:middle line:84% keys on a piano, that's about 6%. 00:19:51.140 --> 00:19:55.520 align:middle line:84% So 10%, in musical terms, is pretty substantial. 00:19:55.520 --> 00:19:57.925 align:middle line:84% So we expect you to be able to hear that difference 00:19:57.925 --> 00:19:59.800 align:middle line:84% because, if you couldn't, you wouldn't really 00:19:59.800 --> 00:20:01.518 align:middle line:84% be able to recognize a lot of melodies 00:20:01.518 --> 00:20:03.060 align:middle line:84% So that's for the resolved harmonics. 00:20:03.060 --> 00:20:05.602 align:middle line:84% Here's the same thing, but when the harmonics are unresolved. 00:20:05.602 --> 00:20:07.400 align:middle line:84% And it'll probably be harder for you 00:20:07.400 --> 00:20:09.160 align:middle line:90% to tell which one is different. 00:20:09.160 --> 00:20:12.680 align:middle line:90% [THREE PITCHES] 00:20:12.680 --> 00:20:14.015 align:middle line:90% Which one? 00:20:14.015 --> 00:20:15.140 align:middle line:90% Yeah, I think that's right. 00:20:15.140 --> 00:20:16.182 align:middle line:90% But it's a little harder. 00:20:16.182 --> 00:20:17.960 align:middle line:90% Now, let's go down to 2%. 00:20:17.960 --> 00:20:22.240 align:middle line:90% [THREE PITCHES] 00:20:22.240 --> 00:20:24.440 align:middle line:90% Which one is different? 00:20:24.440 --> 00:20:27.140 align:middle line:84% But that was obviously a little bit harder. 00:20:27.140 --> 00:20:29.200 align:middle line:84% But now, let's try it with the unresolved. 00:20:29.200 --> 00:20:33.040 align:middle line:90% [THREE PITCHES] 00:20:33.040 --> 00:20:34.440 align:middle line:90% Yeah, so that's pretty hard. 00:20:34.440 --> 00:20:39.200 align:middle line:84% So you're all normal, which is to say 00:20:39.200 --> 00:20:41.160 align:middle line:84% your pitch discrimination is better when you 00:20:41.160 --> 00:20:42.380 align:middle line:90% have low-numbered harmonics. 00:20:42.380 --> 00:20:44.780 align:middle line:84% So that's just this fact about human hearing. 00:20:44.780 --> 00:20:49.520 align:middle line:84% So the standard inference from that result 00:20:49.520 --> 00:20:54.560 align:middle line:84% is that it is suggestive that pitch perception involves 00:20:54.560 --> 00:20:56.980 align:middle line:84% some form of spectral pattern recognition. 00:20:56.980 --> 00:21:00.920 align:middle line:84% So the fact that you're better when the stimulus contains 00:21:00.920 --> 00:21:03.280 align:middle line:84% these low-numbered harmonics suggests 00:21:03.280 --> 00:21:06.520 align:middle line:84% that these peaks and valleys are part 00:21:06.520 --> 00:21:10.080 align:middle line:84% of what is being recognized when you're detecting the pitch. 00:21:10.080 --> 00:21:13.640 align:middle line:84% And in this regime up here, where the harmonics are 00:21:13.640 --> 00:21:17.120 align:middle line:84% unresolved and all you have is that temporal response, 00:21:17.120 --> 00:21:19.260 align:middle line:84% that's where pitch perception is worse. 00:21:19.260 --> 00:21:23.650 align:middle line:84% You don't have that spectral pattern information. 00:21:23.650 --> 00:21:25.310 align:middle line:84% And so that seems to be important. 00:21:25.310 --> 00:21:28.050 align:middle line:90% 00:21:28.050 --> 00:21:28.590 align:middle line:90% All right. 00:21:28.590 --> 00:21:34.030 align:middle line:84% So we often see these phenomena in perception. 00:21:34.030 --> 00:21:36.290 align:middle line:84% And it's natural to wonder why things 00:21:36.290 --> 00:21:39.450 align:middle line:90% are the way that they are. 00:21:39.450 --> 00:21:43.123 align:middle line:84% And so some of the new tools that we have at our disposal, 00:21:43.123 --> 00:21:44.790 align:middle line:84% due to the advances of machine learning, 00:21:44.790 --> 00:21:46.730 align:middle line:84% give us some new ways to investigate 00:21:46.730 --> 00:21:49.990 align:middle line:84% some of these questions about why things are the way they are. 00:21:49.990 --> 00:21:52.570 align:middle line:84% And so one of the things that we can 00:21:52.570 --> 00:21:55.970 align:middle line:84% do to try to understand the nature of human perception 00:21:55.970 --> 00:21:58.650 align:middle line:84% is to optimize machine systems to solve 00:21:58.650 --> 00:22:00.690 align:middle line:84% what we think are some of the same problems 00:22:00.690 --> 00:22:03.350 align:middle line:84% that human perceptual systems are designed to solve. 00:22:03.350 --> 00:22:05.610 align:middle line:84% And then we can look at whether they do what people do 00:22:05.610 --> 00:22:06.830 align:middle line:90% and under what conditions. 00:22:06.830 --> 00:22:10.850 align:middle line:84% So this is a model that was optimized 00:22:10.850 --> 00:22:14.290 align:middle line:84% to estimate the fundamental frequency of excerpts of speech 00:22:14.290 --> 00:22:16.210 align:middle line:90% or music superimposed on noise. 00:22:16.210 --> 00:22:18.690 align:middle line:84% So there's a model of the cochlea, and then 00:22:18.690 --> 00:22:20.270 align:middle line:90% a convolutional neural network. 00:22:20.270 --> 00:22:22.490 align:middle line:84% And so the model is trained to do this. 00:22:22.490 --> 00:22:24.970 align:middle line:84% And then you can simulate experiments on the model 00:22:24.970 --> 00:22:27.890 align:middle line:84% and ask whether the model exhibits 00:22:27.890 --> 00:22:31.650 align:middle line:84% the same kind of characteristics that are seen in humans. 00:22:31.650 --> 00:22:39.530 align:middle line:84% So this is a results graph from human pitch perception that just 00:22:39.530 --> 00:22:41.830 align:middle line:90% summarizes what we saw earlier. 00:22:41.830 --> 00:22:45.832 align:middle line:84% So what's being plotted here is the f0 discrimination threshold. 00:22:45.832 --> 00:22:47.790 align:middle line:84% So that's your pitch discrimination threshold-- 00:22:47.790 --> 00:22:50.330 align:middle line:90% so down is better, up is worse-- 00:22:50.330 --> 00:22:53.130 align:middle line:84% as a function of the lowest harmonic number. 00:22:53.130 --> 00:22:55.890 align:middle line:84% And so what this shows is that when the lowest harmonic 00:22:55.890 --> 00:22:58.570 align:middle line:84% number is low, the discrimination thresholds 00:22:58.570 --> 00:23:00.648 align:middle line:90% are pretty good, i.e. small. 00:23:00.648 --> 00:23:02.690 align:middle line:84% And then as the lowest harmonic number increases, 00:23:02.690 --> 00:23:03.810 align:middle line:90% they get worse. 00:23:03.810 --> 00:23:05.730 align:middle line:84% There's also this phase manipulation here 00:23:05.730 --> 00:23:09.490 align:middle line:84% that is not terribly relevant for our purposes. 00:23:09.490 --> 00:23:12.550 align:middle line:84% And this is the model when you run it on a similar experiment. 00:23:12.550 --> 00:23:14.050 align:middle line:84% You can see that, qualitatively, you 00:23:14.050 --> 00:23:16.690 align:middle line:84% get a similar kind of result, that the model has 00:23:16.690 --> 00:23:19.102 align:middle line:84% low discrimination thresholds, i.e. good discrimination 00:23:19.102 --> 00:23:21.810 align:middle line:84% thresholds, in this regime where you have low-numbered harmonics, 00:23:21.810 --> 00:23:23.450 align:middle line:90% and then they get worse. 00:23:23.450 --> 00:23:26.890 align:middle line:84% So this suggests that this particular performance 00:23:26.890 --> 00:23:29.210 align:middle line:84% characteristic emerges as a consequence 00:23:29.210 --> 00:23:32.350 align:middle line:84% of optimizing a system to estimate fundamental frequency. 00:23:32.350 --> 00:23:34.790 align:middle line:84% Now, of course, it doesn't tell you why that happens. 00:23:34.790 --> 00:23:37.490 align:middle line:84% But with the model, you can do experiments that would 00:23:37.490 --> 00:23:40.970 align:middle line:90% be difficult to do in biology. 00:23:40.970 --> 00:23:43.610 align:middle line:84% And in particular, you can ask whether 00:23:43.610 --> 00:23:46.650 align:middle line:84% the perceptual characteristics that are observed 00:23:46.650 --> 00:23:51.890 align:middle line:84% depend on the auditory diet that the system is optimized for. 00:23:51.890 --> 00:23:54.770 align:middle line:84% So the results that I just showed 00:23:54.770 --> 00:23:58.090 align:middle line:84% you came from a model that was trained 00:23:58.090 --> 00:24:01.250 align:middle line:84% on excerpts of speech or music, so natural or naturalistic 00:24:01.250 --> 00:24:02.090 align:middle line:90% sounds. 00:24:02.090 --> 00:24:05.890 align:middle line:84% One of the most salient features of natural sounds 00:24:05.890 --> 00:24:07.390 align:middle line:84% is that they tend to be low pass. 00:24:07.390 --> 00:24:09.770 align:middle line:84% They tend to have more power at low frequencies 00:24:09.770 --> 00:24:10.750 align:middle line:90% than high frequencies. 00:24:10.750 --> 00:24:12.310 align:middle line:90% And that's evident here. 00:24:12.310 --> 00:24:16.190 align:middle line:84% So this is a graph that shows a power spectrum. 00:24:16.190 --> 00:24:17.950 align:middle line:90% So the y-axis is power. 00:24:17.950 --> 00:24:19.590 align:middle line:90% The x-axis is frequency. 00:24:19.590 --> 00:24:22.330 align:middle line:84% The black line is the average spectrum 00:24:22.330 --> 00:24:23.725 align:middle line:90% of speech or music stimuli. 00:24:23.725 --> 00:24:25.850 align:middle line:84% And so you can see they have a fair amount of power 00:24:25.850 --> 00:24:26.710 align:middle line:90% at low frequencies. 00:24:26.710 --> 00:24:27.970 align:middle line:90% And then it drops off. 00:24:27.970 --> 00:24:33.190 align:middle line:84% And so what was done in this study is to create new sounds, 00:24:33.190 --> 00:24:35.030 align:middle line:84% either by filtering speech or music, 00:24:35.030 --> 00:24:36.850 align:middle line:84% or by generating synthetic tones that 00:24:36.850 --> 00:24:40.570 align:middle line:84% have particular spectral properties so that they were 00:24:40.570 --> 00:24:42.810 align:middle line:90% either low pass or high pass. 00:24:42.810 --> 00:24:45.730 align:middle line:84% So high pass sounds instead have more power 00:24:45.730 --> 00:24:48.210 align:middle line:90% at higher frequencies. 00:24:48.210 --> 00:24:51.250 align:middle line:84% So you can take these new auditory diets, 00:24:51.250 --> 00:24:54.810 align:middle line:84% and then optimize systems to estimate fundamental frequency 00:24:54.810 --> 00:24:59.090 align:middle line:84% under these different conditions. 00:24:59.090 --> 00:25:00.890 align:middle line:84% And what was found is that you get 00:25:00.890 --> 00:25:04.930 align:middle line:84% very different psychophysical results 00:25:04.930 --> 00:25:06.570 align:middle line:84% on these models that are optimized 00:25:06.570 --> 00:25:08.250 align:middle line:90% on different kinds of sounds. 00:25:08.250 --> 00:25:09.910 align:middle line:84% And so this is that same experiment. 00:25:09.910 --> 00:25:12.410 align:middle line:84% So again, we have f0 discrimination thresholds 00:25:12.410 --> 00:25:14.770 align:middle line:84% as a function of the lowest harmonic number. 00:25:14.770 --> 00:25:17.970 align:middle line:84% So the black curve is the one that you 00:25:17.970 --> 00:25:20.875 align:middle line:84% get if you train the model on natural sounds. 00:25:20.875 --> 00:25:22.250 align:middle line:84% And then if you filter the sounds 00:25:22.250 --> 00:25:25.250 align:middle line:84% to either be even more low pass or high pass, 00:25:25.250 --> 00:25:27.690 align:middle line:84% you get very different results curves. 00:25:27.690 --> 00:25:31.090 align:middle line:84% And here, if you take synthetic sounds, 00:25:31.090 --> 00:25:33.490 align:middle line:84% if you match the spectrum of the synthetic sounds 00:25:33.490 --> 00:25:35.990 align:middle line:84% to these natural sounds, you get the black curve out, 00:25:35.990 --> 00:25:38.350 align:middle line:84% which, again, looks qualitatively human-like. 00:25:38.350 --> 00:25:40.810 align:middle line:84% And then if you do these anti-match sounds, where they're 00:25:40.810 --> 00:25:45.970 align:middle line:84% instead high pass, you get kind of the opposite sort of pattern. 00:25:45.970 --> 00:25:50.410 align:middle line:84% So this is suggesting that the dependence of human pitch 00:25:50.410 --> 00:25:53.330 align:middle line:84% perception on these low-numbered harmonics 00:25:53.330 --> 00:25:56.410 align:middle line:84% is because that's where the information is 00:25:56.410 --> 00:25:59.170 align:middle line:84% in natural sounds about fundamental frequency, 00:25:59.170 --> 00:26:01.470 align:middle line:84% in the sense that if you take a machine system, 00:26:01.470 --> 00:26:04.650 align:middle line:84% you train it to estimate fundamental frequency, that's 00:26:04.650 --> 00:26:08.010 align:middle line:84% the information that it ends up learning to use. 00:26:08.010 --> 00:26:10.560 align:middle line:84% And if you train it on different kinds of sounds, 00:26:10.560 --> 00:26:12.560 align:middle line:84% it develops a different strategy. 00:26:12.560 --> 00:26:15.420 align:middle line:90% 00:26:15.420 --> 00:26:17.040 align:middle line:84% What questions do you have about that? 00:26:17.040 --> 00:26:38.060 align:middle line:90% 00:26:38.060 --> 00:26:38.560 align:middle line:90% OK. 00:26:38.560 --> 00:26:43.260 align:middle line:84% So big picture is we can take machine learning models 00:26:43.260 --> 00:26:47.200 align:middle line:84% and optimize them for biological problems, 00:26:47.200 --> 00:26:49.820 align:middle line:90% and then look at what they do. 00:26:49.820 --> 00:26:52.380 align:middle line:84% And what you can do with these machine learning models 00:26:52.380 --> 00:26:54.460 align:middle line:84% that you can't do with actual biological systems 00:26:54.460 --> 00:26:58.060 align:middle line:84% is you can optimize them for different kinds of worlds. 00:26:58.060 --> 00:27:01.460 align:middle line:84% Think of this as like simulating evolution and development 00:27:01.460 --> 00:27:04.460 align:middle line:84% in alternative worlds where the nature of sounds 00:27:04.460 --> 00:27:05.480 align:middle line:90% is very different. 00:27:05.480 --> 00:27:08.580 align:middle line:84% So it can give you a glimpse of what might we 00:27:08.580 --> 00:27:13.460 align:middle line:84% be like if we had evolved and grown up in a very 00:27:13.460 --> 00:27:14.780 align:middle line:90% different kind of world. 00:27:14.780 --> 00:27:17.100 align:middle line:84% And in this particular case, it suggests that our pitch 00:27:17.100 --> 00:27:18.400 align:middle line:90% perception would be different. 00:27:18.400 --> 00:27:21.180 align:middle line:84% So it gives us one way to understand 00:27:21.180 --> 00:27:24.120 align:middle line:84% why we have these particular traits that we do. 00:27:24.120 --> 00:27:28.020 align:middle line:90% 00:27:28.020 --> 00:27:28.520 align:middle line:90% OK. 00:27:28.520 --> 00:27:31.877 align:middle line:84% So let's talk very briefly about where all this might 00:27:31.877 --> 00:27:32.960 align:middle line:90% be happening in the brain. 00:27:32.960 --> 00:27:37.500 align:middle line:84% So this is a cross-section of the head showing you 00:27:37.500 --> 00:27:40.020 align:middle line:84% the key components of the auditory system. 00:27:40.020 --> 00:27:43.260 align:middle line:84% So we've got these different subcortical weigh stations, 00:27:43.260 --> 00:27:48.100 align:middle line:84% and then auditory cortex here in the temporal lobe. 00:27:48.100 --> 00:27:52.540 align:middle line:84% So this is a view of the macaque brain from the side. 00:27:52.540 --> 00:27:57.400 align:middle line:84% And auditory cortex is mostly buried inside a sulcus. 00:27:57.400 --> 00:28:01.660 align:middle line:84% And so to view it, you typically cut out a piece of the brain. 00:28:01.660 --> 00:28:04.340 align:middle line:84% And it's very common to distinguish in the cortex 00:28:04.340 --> 00:28:06.060 align:middle line:90% between three regions-- 00:28:06.060 --> 00:28:09.180 align:middle line:84% the core, the belt that goes around it, and then 00:28:09.180 --> 00:28:13.820 align:middle line:90% the parabelt. 00:28:13.820 --> 00:28:16.185 align:middle line:84% And these different regions are distinguished in part 00:28:16.185 --> 00:28:17.560 align:middle line:84% by different response properties, 00:28:17.560 --> 00:28:19.000 align:middle line:90% but also by their connections. 00:28:19.000 --> 00:28:23.220 align:middle line:84% So this is a crude diagram that denotes different areas 00:28:23.220 --> 00:28:24.520 align:middle line:90% and their connections. 00:28:24.520 --> 00:28:26.445 align:middle line:84% So this thing in the middle here is the core. 00:28:26.445 --> 00:28:27.820 align:middle line:84% And that's conventionally thought 00:28:27.820 --> 00:28:30.300 align:middle line:84% to consist of three regions, and then 00:28:30.300 --> 00:28:34.980 align:middle line:84% surrounded by the belt, and then the parabelt. 00:28:34.980 --> 00:28:36.700 align:middle line:84% Now, one of the key things that you 00:28:36.700 --> 00:28:39.020 align:middle line:84% see when you look at the auditory cortex 00:28:39.020 --> 00:28:40.900 align:middle line:90% is what's known as tonotopy. 00:28:40.900 --> 00:28:43.340 align:middle line:84% And that refers to the fact that there is 00:28:43.340 --> 00:28:47.360 align:middle line:90% a map of frequency selectivity. 00:28:47.360 --> 00:28:51.340 align:middle line:84% So this is a picture that is the result of an fMRI experiment, 00:28:51.340 --> 00:28:53.580 align:middle line:84% where people are in an fMRI scanner 00:28:53.580 --> 00:28:55.440 align:middle line:84% and they're listening to pure tones. 00:28:55.440 --> 00:28:59.140 align:middle line:84% And the pure tones vary in their frequency. 00:28:59.140 --> 00:29:02.620 align:middle line:84% So what an fMRI scanner does is it measures a signal 00:29:02.620 --> 00:29:05.180 align:middle line:84% from the brain that is believed to be 00:29:05.180 --> 00:29:07.420 align:middle line:90% related to neural activity. 00:29:07.420 --> 00:29:10.040 align:middle line:84% So you can then analyze the results. 00:29:10.040 --> 00:29:13.220 align:middle line:84% And for every little piece of the brain, 00:29:13.220 --> 00:29:18.180 align:middle line:84% you can ask which frequency caused it to respond the most. 00:29:18.180 --> 00:29:20.540 align:middle line:84% And so that's turned into a map here, 00:29:20.540 --> 00:29:23.940 align:middle line:84% where the voxels, or the little pieces of brain, 00:29:23.940 --> 00:29:26.780 align:middle line:84% are color-coded based on the frequency that 00:29:26.780 --> 00:29:28.580 align:middle line:90% caused the biggest response. 00:29:28.580 --> 00:29:31.820 align:middle line:84% So very low, best frequencies would be dark blue 00:29:31.820 --> 00:29:34.140 align:middle line:90% and very high would be red. 00:29:34.140 --> 00:29:36.940 align:middle line:84% And so what you see when you do an experiment like this 00:29:36.940 --> 00:29:38.980 align:middle line:84% is typically this kind of gradient, 00:29:38.980 --> 00:29:42.100 align:middle line:84% where you see a stripe here where 00:29:42.100 --> 00:29:43.900 align:middle line:90% the best frequency is high. 00:29:43.900 --> 00:29:47.572 align:middle line:84% And then it transitions to the best frequency being low. 00:29:47.572 --> 00:29:49.780 align:middle line:84% And then it switches back again to the best frequency 00:29:49.780 --> 00:29:51.500 align:middle line:90% being high. 00:29:51.500 --> 00:29:55.700 align:middle line:84% So it's a very prominent, functional, anatomical thing 00:29:55.700 --> 00:29:57.500 align:middle line:84% that you see in the auditory cortex. 00:29:57.500 --> 00:30:00.980 align:middle line:84% Now, it's not totally clear why this is there 00:30:00.980 --> 00:30:02.200 align:middle line:90% and why it's important. 00:30:02.200 --> 00:30:05.420 align:middle line:84% I mean, it's essentially recapitulating the tonotopy 00:30:05.420 --> 00:30:07.660 align:middle line:84% that you see as early as the cochlea. 00:30:07.660 --> 00:30:08.640 align:middle line:90% But it's there. 00:30:08.640 --> 00:30:09.140 align:middle line:90% Yeah? 00:30:09.140 --> 00:30:11.420 align:middle line:84% AUDIENCE: Is this the fMRI scan of the macaque brain? 00:30:11.420 --> 00:30:11.640 align:middle line:90% Or-- 00:30:11.640 --> 00:30:13.180 align:middle line:84% JOSH MCDERMOTT: No, this is a human. 00:30:13.180 --> 00:30:14.700 align:middle line:90% Yeah. 00:30:14.700 --> 00:30:18.797 align:middle line:84% Yeah, and so I would say, although it's not 00:30:18.797 --> 00:30:20.380 align:middle line:84% totally clear why this is particularly 00:30:20.380 --> 00:30:22.353 align:middle line:84% interesting or important, it's there. 00:30:22.353 --> 00:30:23.520 align:middle line:90% And it's there in everybody. 00:30:23.520 --> 00:30:25.800 align:middle line:84% And it pretty much always looks pretty much the same. 00:30:25.800 --> 00:30:27.288 align:middle line:84% So everybody's got a map like this. 00:30:27.288 --> 00:30:29.080 align:middle line:84% It's more or less always in the same place. 00:30:29.080 --> 00:30:29.700 align:middle line:90% Yeah? 00:30:29.700 --> 00:30:33.940 align:middle line:84% AUDIENCE: Is there a pattern to it that I'm not seeing? 00:30:33.940 --> 00:30:36.100 align:middle line:90% JOSH MCDERMOTT: High, low, high. 00:30:36.100 --> 00:30:37.980 align:middle line:90% So high is red. 00:30:37.980 --> 00:30:39.040 align:middle line:90% Low is blue. 00:30:39.040 --> 00:30:40.220 align:middle line:90% High is red. 00:30:40.220 --> 00:30:42.100 align:middle line:84% So that's the stereotyped pattern 00:30:42.100 --> 00:30:44.220 align:middle line:84% that you see when you do the experiment. 00:30:44.220 --> 00:30:47.660 align:middle line:84% And then the high regions come close to connecting at the top. 00:30:47.660 --> 00:30:49.160 align:middle line:90% Yeah. 00:30:49.160 --> 00:30:51.740 align:middle line:90% 00:30:51.740 --> 00:30:55.620 align:middle line:84% So one of the reasons why this is worth knowing about 00:30:55.620 --> 00:30:58.040 align:middle line:84% is because it's there and it's because it's prominent. 00:30:58.040 --> 00:31:00.415 align:middle line:84% It's also worth knowing about because, scientifically, it 00:31:00.415 --> 00:31:02.210 align:middle line:84% turns out to be a fairly useful landmark. 00:31:02.210 --> 00:31:07.490 align:middle line:84% So oftentimes, we will measure tonotopy as a reference point. 00:31:07.490 --> 00:31:09.435 align:middle line:84% And it's useful to compare other responses 00:31:09.435 --> 00:31:11.310 align:middle line:84% that you might observe in the auditory cortex 00:31:11.310 --> 00:31:12.850 align:middle line:90% to that tonotopic map. 00:31:12.850 --> 00:31:17.790 align:middle line:84% And so one of the places where this turns out to be useful 00:31:17.790 --> 00:31:22.150 align:middle line:84% is when we think about the neural basis of pitch. 00:31:22.150 --> 00:31:25.030 align:middle line:84% So one approach that has been used for a long time 00:31:25.030 --> 00:31:26.890 align:middle line:84% to try to isolate parts of the brain 00:31:26.890 --> 00:31:30.590 align:middle line:84% that process pitch is to look for regions that respond 00:31:30.590 --> 00:31:32.790 align:middle line:90% more to tones than to noise. 00:31:32.790 --> 00:31:34.770 align:middle line:84% And that's what's shown in the top plots. 00:31:34.770 --> 00:31:38.350 align:middle line:84% So we have the right hemisphere and the left hemisphere. 00:31:38.350 --> 00:31:41.390 align:middle line:84% And these are probabilistic maps of the likelihood 00:31:41.390 --> 00:31:45.350 align:middle line:84% that a voxel will significantly respond more to tones, so 00:31:45.350 --> 00:31:47.710 align:middle line:90% harmonic tones, than to noise. 00:31:47.710 --> 00:31:52.750 align:middle line:84% And so you see these regions that have that property. 00:31:52.750 --> 00:31:56.310 align:middle line:84% And so the bottom part, the borders of that region 00:31:56.310 --> 00:31:59.730 align:middle line:84% are compared to that tonotopic map that we just looked at. 00:31:59.730 --> 00:32:02.950 align:middle line:84% So again, we've got high, low, high. 00:32:02.950 --> 00:32:06.710 align:middle line:84% And you can see that this region that is responding more to tones 00:32:06.710 --> 00:32:09.270 align:middle line:84% than to noise, that a lot of people have hypothesized 00:32:09.270 --> 00:32:12.630 align:middle line:84% is involved in pitch perception, overlaps with the low frequency 00:32:12.630 --> 00:32:16.170 align:middle line:84% part of the tonotopic map, and then extends beyond it. 00:32:16.170 --> 00:32:18.550 align:middle line:84% So again, the reasons why this is where it is 00:32:18.550 --> 00:32:20.750 align:middle line:90% are not well understood. 00:32:20.750 --> 00:32:24.310 align:middle line:84% But it's a regularity that is there and is 00:32:24.310 --> 00:32:25.750 align:middle line:90% kind of interesting. 00:32:25.750 --> 00:32:26.250 align:middle line:90% All right. 00:32:26.250 --> 00:32:27.990 align:middle line:84% So what is the evidence that this thing is actually 00:32:27.990 --> 00:32:29.510 align:middle line:90% involved in pitch perception? 00:32:29.510 --> 00:32:32.830 align:middle line:84% Well, I just told you about how pitch perception seems 00:32:32.830 --> 00:32:35.110 align:middle line:84% to be dominated by low-numbered harmonics, 00:32:35.110 --> 00:32:38.390 align:middle line:84% in the sense that your discrimination thresholds are 00:32:38.390 --> 00:32:41.150 align:middle line:84% best, they're lowest, when the stimulus contains 00:32:41.150 --> 00:32:42.630 align:middle line:90% low-numbered harmonics. 00:32:42.630 --> 00:32:45.270 align:middle line:84% And so it turns out that, if you measure 00:32:45.270 --> 00:32:49.030 align:middle line:84% the response of this brain region as a function 00:32:49.030 --> 00:32:51.630 align:middle line:84% of the harmonic composition of tones 00:32:51.630 --> 00:32:54.350 align:middle line:84% that people are listening to, the response 00:32:54.350 --> 00:32:58.390 align:middle line:84% is largest when the tones contain low-numbered harmonics. 00:32:58.390 --> 00:33:01.750 align:middle line:84% So what these graphs are plotting is the fMRI response 00:33:01.750 --> 00:33:04.790 align:middle line:84% measured as percent signal change-- that's on the y-axis-- 00:33:04.790 --> 00:33:06.870 align:middle line:84% for different stimulus conditions 00:33:06.870 --> 00:33:10.327 align:middle line:84% that vary in the lowest harmonic number. 00:33:10.327 --> 00:33:12.410 align:middle line:84% And again, there's two different frequency ranges. 00:33:12.410 --> 00:33:15.270 align:middle line:84% And it's not critical for our purposes. 00:33:15.270 --> 00:33:19.230 align:middle line:84% And what you can see is that, when the stimulus contains 00:33:19.230 --> 00:33:21.630 align:middle line:84% low numbered harmonics up to maybe the eighth, 00:33:21.630 --> 00:33:23.250 align:middle line:90% you get a higher response. 00:33:23.250 --> 00:33:25.090 align:middle line:84% And then after that, it kind of drops off. 00:33:25.090 --> 00:33:29.310 align:middle line:84% And so these two graphs, they look like the flip 00:33:29.310 --> 00:33:30.910 align:middle line:90% of these two graphs. 00:33:30.910 --> 00:33:33.910 align:middle line:84% So the response of the brain is high 00:33:33.910 --> 00:33:35.830 align:middle line:84% whenever the discrimination thresholds are 00:33:35.830 --> 00:33:37.390 align:middle line:90% low-- that is, they're good. 00:33:37.390 --> 00:33:39.270 align:middle line:84% And the response gets lower whenever 00:33:39.270 --> 00:33:42.550 align:middle line:90% discrimination is worse. 00:33:42.550 --> 00:33:45.230 align:middle line:84% And so the fact that there's this relationship 00:33:45.230 --> 00:33:51.030 align:middle line:84% between this stimulus parameter that affects perception 00:33:51.030 --> 00:33:53.470 align:middle line:84% is consistent with the idea that this brain region 00:33:53.470 --> 00:33:56.070 align:middle line:84% is involved in some way in mediating 00:33:56.070 --> 00:33:57.210 align:middle line:90% the perception of pitch. 00:33:57.210 --> 00:33:59.910 align:middle line:90% 00:33:59.910 --> 00:34:01.110 align:middle line:90% Yeah? 00:34:01.110 --> 00:34:06.357 align:middle line:84% AUDIENCE: Why is it higher for the [INAUDIBLE]? 00:34:06.357 --> 00:34:07.690 align:middle line:90% JOSH MCDERMOTT: Say that louder? 00:34:07.690 --> 00:34:10.310 align:middle line:84% AUDIENCE: Why is it higher for the high frequency ranges? 00:34:10.310 --> 00:34:11.070 align:middle line:84% JOSH MCDERMOTT: You mean, why does this 00:34:11.070 --> 00:34:12.270 align:middle line:90% go up a little bit higher? 00:34:12.270 --> 00:34:14.230 align:middle line:90% I don't know. 00:34:14.230 --> 00:34:16.989 align:middle line:84% And if you squint, maybe it looks 00:34:16.989 --> 00:34:20.750 align:middle line:84% like there's a similar effect here perceptually. 00:34:20.750 --> 00:34:21.929 align:middle line:90% This might just be a fluke. 00:34:21.929 --> 00:34:22.889 align:middle line:90% I mean, real data. 00:34:22.889 --> 00:34:25.290 align:middle line:84% So we don't expect them to be exactly the same. 00:34:25.290 --> 00:34:29.469 align:middle line:84% I'm not sure if that's worth writing home about or not. 00:34:29.469 --> 00:34:31.989 align:middle line:90% Yeah, it's a good question. 00:34:31.989 --> 00:34:33.290 align:middle line:90% What other questions you got? 00:34:33.290 --> 00:34:40.110 align:middle line:90% 00:34:40.110 --> 00:34:40.610 align:middle line:90% All right. 00:34:40.610 --> 00:34:42.230 align:middle line:84% We're going to talk about timbre now. 00:34:42.230 --> 00:34:45.429 align:middle line:84% So the last thing we're going to talk about 00:34:45.429 --> 00:34:48.130 align:middle line:84% before we get into auditory scene analysis is timbre. 00:34:48.130 --> 00:34:51.100 align:middle line:90% 00:34:51.100 --> 00:34:58.550 align:middle line:84% So timbre is-- it's frustrating because, on the one hand, 00:34:58.550 --> 00:35:01.430 align:middle line:84% it's maybe the most interesting thing about sound. 00:35:01.430 --> 00:35:07.830 align:middle line:84% On the other hand, it's defined as what a whole bunch of things 00:35:07.830 --> 00:35:08.830 align:middle line:90% leave out. 00:35:08.830 --> 00:35:11.730 align:middle line:84% So it's classically defined as the other dimension, 00:35:11.730 --> 00:35:13.950 align:middle line:84% in addition to loudness and pitch, 00:35:13.950 --> 00:35:17.190 align:middle line:84% along which our perception of sounds can vary. 00:35:17.190 --> 00:35:20.543 align:middle line:84% And it's pretty clearly not just one dimension. 00:35:20.543 --> 00:35:22.210 align:middle line:84% But the number of dimensions is unclear. 00:35:22.210 --> 00:35:26.150 align:middle line:84% It's not even really even clear how to think about it. 00:35:26.150 --> 00:35:31.550 align:middle line:84% But it's nonetheless a thing that people talk about a lot. 00:35:31.550 --> 00:35:35.390 align:middle line:84% And it references lots of things that are important. 00:35:35.390 --> 00:35:37.460 align:middle line:84% So classically, timbre is the thing 00:35:37.460 --> 00:35:39.210 align:middle line:84% that differentiates different instruments. 00:35:39.210 --> 00:35:40.170 align:middle line:90% So this is an example. 00:35:40.170 --> 00:35:42.270 align:middle line:84% We've got a piano note, a trombone note, and a tenor sax 00:35:42.270 --> 00:35:42.770 align:middle line:90% note. 00:35:42.770 --> 00:35:45.040 align:middle line:84% And the fundamental frequencies are all the same. 00:35:45.040 --> 00:35:46.790 align:middle line:84% So classically and musically, we would say 00:35:46.790 --> 00:35:48.370 align:middle line:90% that they have the same pitch. 00:35:48.370 --> 00:35:50.590 align:middle line:90% They're the same note. 00:35:50.590 --> 00:35:53.160 align:middle line:90% But they sound really different. 00:35:53.160 --> 00:35:54.858 align:middle line:84% So what is it that determines timbre? 00:35:54.858 --> 00:35:56.400 align:middle line:84% And one factor that determines timbre 00:35:56.400 --> 00:35:58.100 align:middle line:84% is the relative amplitude of overtones. 00:35:58.100 --> 00:36:00.360 align:middle line:84% And so this is a kind of a cool demo 00:36:00.360 --> 00:36:05.140 align:middle line:84% that shows how different overtones contribute to timbre. 00:36:05.140 --> 00:36:06.000 align:middle line:90% So listen to this. 00:36:06.000 --> 00:36:06.667 align:middle line:90% [AUDIO PLAYBACK] 00:36:06.667 --> 00:36:10.640 align:middle line:84% - The effect of spectrum on timbre. 00:36:10.640 --> 00:36:13.400 align:middle line:84% You will hear the sounds of two instruments built up 00:36:13.400 --> 00:36:17.891 align:middle line:84% by adding partials one at a time. 00:36:17.891 --> 00:36:20.873 align:middle line:90% [SINGLE NOTE] 00:36:20.873 --> 00:36:23.855 align:middle line:90% [TWO NOTES] 00:36:23.855 --> 00:36:25.346 align:middle line:90% 00:36:25.346 --> 00:36:28.328 align:middle line:90% [THREE NOTES] 00:36:28.328 --> 00:36:29.322 align:middle line:90% 00:36:29.322 --> 00:36:31.807 align:middle line:90% [MULTIPLE NOTES, INSTRUMENTS] 00:36:31.807 --> 00:36:33.795 align:middle line:90% 00:36:33.795 --> 00:36:36.280 align:middle line:90% [MULTIPLE NOTES, INSTRUMENTS] 00:36:36.280 --> 00:36:37.274 align:middle line:90% 00:36:37.274 --> 00:36:40.256 align:middle line:90% [MULTIPLE NOTES, INSTRUMENTS] 00:36:40.256 --> 00:36:41.250 align:middle line:90% 00:36:41.250 --> 00:36:44.232 align:middle line:90% [MULTIPLE NOTES, INSTRUMENTS] 00:36:44.232 --> 00:36:45.226 align:middle line:90% 00:36:45.226 --> 00:36:48.208 align:middle line:90% [MULTIPLE NOTES, INSTRUMENTS] 00:36:48.208 --> 00:36:49.699 align:middle line:90% 00:36:49.699 --> 00:36:50.340 align:middle line:90% [END PLAYBACK] 00:36:50.340 --> 00:36:51.840 align:middle line:84% JOSH MCDERMOTT: So the point of that 00:36:51.840 --> 00:36:55.960 align:middle line:84% is that thing initially didn't sound like what 00:36:55.960 --> 00:36:57.220 align:middle line:90% it sounded like at the end. 00:36:57.220 --> 00:36:59.320 align:middle line:84% And then we add in those frequency components. 00:36:59.320 --> 00:37:02.060 align:middle line:84% And you build up this sense of what it is. 00:37:02.060 --> 00:37:04.658 align:middle line:90% And here's the same thing here. 00:37:04.658 --> 00:37:08.144 align:middle line:90% [SINGLE RESONATING NOTE] 00:37:08.144 --> 00:37:11.630 align:middle line:90% [SINGLE RESONATING NOTE] 00:37:11.630 --> 00:37:14.618 align:middle line:90% [TWO RESONATING NOTES] 00:37:14.618 --> 00:37:16.112 align:middle line:90% 00:37:16.112 --> 00:37:19.100 align:middle line:90% [MULTIPLE RESONATING NOTES] 00:37:19.100 --> 00:37:20.096 align:middle line:90% 00:37:20.096 --> 00:37:23.084 align:middle line:90% [MULTIPLE RESONATING NOTES] 00:37:23.084 --> 00:37:24.080 align:middle line:90% 00:37:24.080 --> 00:37:27.068 align:middle line:90% [MULTIPLE RESONATING NOTES] 00:37:27.068 --> 00:37:28.064 align:middle line:90% 00:37:28.064 --> 00:37:31.052 align:middle line:90% [MULTIPLE RESONATING NOTES] 00:37:31.052 --> 00:37:32.048 align:middle line:90% 00:37:32.048 --> 00:37:35.036 align:middle line:90% [MULTIPLE RESONATING NOTES] 00:37:35.036 --> 00:37:38.080 align:middle line:90% 00:37:38.080 --> 00:37:39.800 align:middle line:84% So hopefully, you had the experience 00:37:39.800 --> 00:37:42.240 align:middle line:84% that that gradually became more and more 00:37:42.240 --> 00:37:45.140 align:middle line:84% of a realistic rendition of a guitar being plucked. 00:37:45.140 --> 00:37:48.660 align:middle line:84% So the timbre, the sense of what the instrument is, 00:37:48.660 --> 00:37:54.460 align:middle line:84% it's some operation that's being performed on the entire sound. 00:37:54.460 --> 00:37:57.040 align:middle line:84% And the relative amplitudes of the different overtones 00:37:57.040 --> 00:37:59.300 align:middle line:90% is part of the story. 00:37:59.300 --> 00:38:02.240 align:middle line:90% 00:38:02.240 --> 00:38:04.280 align:middle line:84% But another interesting fact, just 00:38:04.280 --> 00:38:06.000 align:middle line:84% at least about musical instruments, 00:38:06.000 --> 00:38:08.880 align:middle line:84% is that the high and the low tones from a musical instrument, 00:38:08.880 --> 00:38:12.320 align:middle line:84% they generally have different amplitude spectra. 00:38:12.320 --> 00:38:15.840 align:middle line:84% And what I mean by that is that the shape of the spectrum 00:38:15.840 --> 00:38:16.900 align:middle line:90% is distinct. 00:38:16.900 --> 00:38:19.760 align:middle line:84% It's not the case that you can take the amplitude spectrum 00:38:19.760 --> 00:38:25.117 align:middle line:84% of a very low note and just translate that up and end up 00:38:25.117 --> 00:38:27.200 align:middle line:84% with a high note that sounds like that instrument. 00:38:27.200 --> 00:38:30.160 align:middle line:84% So for instance, on the piano, the low notes 00:38:30.160 --> 00:38:32.700 align:middle line:84% will typically have a lot more energy in the high overtones, 00:38:32.700 --> 00:38:34.575 align:middle line:84% whereas the high notes have a lot more energy 00:38:34.575 --> 00:38:35.707 align:middle line:90% at the fundamental. 00:38:35.707 --> 00:38:37.540 align:middle line:84% And so the consequence of this, like I said, 00:38:37.540 --> 00:38:40.560 align:middle line:84% is if you take one note played on an instrument 00:38:40.560 --> 00:38:42.880 align:middle line:84% and simply scale all the frequencies up or down, 00:38:42.880 --> 00:38:45.140 align:middle line:84% it doesn't sound like the same instrument. 00:38:45.140 --> 00:38:48.720 align:middle line:84% So this is a classic demonstration of this effect. 00:38:48.720 --> 00:38:49.660 align:middle line:90% So check this out. 00:38:49.660 --> 00:38:51.000 align:middle line:90% [AUDIO PLAYBACK] 00:38:51.000 --> 00:38:54.720 align:middle line:84% - Change in timbre with transposition-- 00:38:54.720 --> 00:38:56.760 align:middle line:84% a three-octave scale on a bassoon 00:38:56.760 --> 00:39:00.400 align:middle line:84% is presented, followed by a three octave scale of notes 00:39:00.400 --> 00:39:03.680 align:middle line:84% that are simple transpositions of the instrument's highest 00:39:03.680 --> 00:39:05.080 align:middle line:90% tone. 00:39:05.080 --> 00:39:08.960 align:middle line:84% This is how the bassoon would sound if all its tones had 00:39:08.960 --> 00:39:12.182 align:middle line:90% the same relative spectrum. 00:39:12.182 --> 00:39:15.128 align:middle line:90% [BASSOON TUNE] 00:39:15.128 --> 00:39:24.948 align:middle line:90% 00:39:24.948 --> 00:39:27.894 align:middle line:90% [ELECTRONIC BASSOON TUNE] 00:39:27.894 --> 00:39:35.750 align:middle line:90% 00:39:35.750 --> 00:39:37.005 align:middle line:90% [END PLAYBACK] 00:39:37.005 --> 00:39:39.380 align:middle line:84% JOSH MCDERMOTT: So you can tell, in that version, really, 00:39:39.380 --> 00:39:41.963 align:middle line:84% it's only the high notes that actually sound like the bassoon. 00:39:41.963 --> 00:39:44.740 align:middle line:84% So the low notes have that same spectrum as the high note, 00:39:44.740 --> 00:39:46.460 align:middle line:90% but just transposed down. 00:39:46.460 --> 00:39:48.360 align:middle line:84% And your brain has learned that that's not 00:39:48.360 --> 00:39:50.240 align:middle line:84% what a bass sounds soon like-- that's not 00:39:50.240 --> 00:39:54.520 align:middle line:90% what a bassoon sounds like. 00:39:54.520 --> 00:39:59.640 align:middle line:84% So the basis for that kind of effect is still not known. 00:39:59.640 --> 00:40:01.780 align:middle line:84% It's probably learned from exposure to instruments, 00:40:01.780 --> 00:40:03.200 align:middle line:90% but we don't really know. 00:40:03.200 --> 00:40:09.000 align:middle line:84% And really, I should say that the reason why the bassoon has 00:40:09.000 --> 00:40:12.360 align:middle line:84% that acoustic property, I think, is fairly straightforward, which 00:40:12.360 --> 00:40:16.760 align:middle line:84% is that a lot of instruments have a relatively fixed filter. 00:40:16.760 --> 00:40:20.920 align:middle line:84% So the body of the instrument is this thing that's constant. 00:40:20.920 --> 00:40:24.600 align:middle line:84% And then it's excited with a sound that can 00:40:24.600 --> 00:40:26.060 align:middle line:90% vary in fundamental frequency. 00:40:26.060 --> 00:40:30.400 align:middle line:84% But because the filter is fixed, the shape of the sound 00:40:30.400 --> 00:40:32.537 align:middle line:84% ends up being competent, in some sense. 00:40:32.537 --> 00:40:34.620 align:middle line:84% But then you're varying the fundamental frequency. 00:40:34.620 --> 00:40:36.320 align:middle line:84% And that causes the relative amplitudes 00:40:36.320 --> 00:40:38.320 align:middle line:84% of the different harmonics to change, 00:40:38.320 --> 00:40:41.400 align:middle line:84% depending on the pitch of the note. 00:40:41.400 --> 00:40:44.010 align:middle line:84% But how you actually learn that that's how instruments 00:40:44.010 --> 00:40:47.050 align:middle line:90% do their thing is unclear. 00:40:47.050 --> 00:40:49.890 align:middle line:84% So another stimulus property that influences timbre 00:40:49.890 --> 00:40:51.310 align:middle line:90% is the envelope of a sound. 00:40:51.310 --> 00:40:54.070 align:middle line:84% So that refers to the way that the intensity changes over time. 00:40:54.070 --> 00:40:56.237 align:middle line:84% So this can be really different for something that's 00:40:56.237 --> 00:40:57.970 align:middle line:90% plucked versus struck or blown. 00:40:57.970 --> 00:41:01.418 align:middle line:84% So these are waveforms of two notes played on a violin. 00:41:01.418 --> 00:41:02.710 align:middle line:90% In the first one, it's plucked. 00:41:02.710 --> 00:41:04.530 align:middle line:84% So when you pluck the string, you 00:41:04.530 --> 00:41:08.730 align:middle line:84% get this very rapid increase in energy that slowly decays, 00:41:08.730 --> 00:41:12.190 align:middle line:84% whereas when you bow a string, there's a slow buildup. 00:41:12.190 --> 00:41:14.450 align:middle line:84% And then it's more sustained as the bow 00:41:14.450 --> 00:41:17.090 align:middle line:90% travels across the string. 00:41:17.090 --> 00:41:19.050 align:middle line:84% And so the importance of the envelope 00:41:19.050 --> 00:41:21.810 align:middle line:84% can be seen by playing things backwards, 00:41:21.810 --> 00:41:23.890 align:middle line:84% because the amplitude spectrum is unchanged 00:41:23.890 --> 00:41:25.230 align:middle line:90% when you play things backwards. 00:41:25.230 --> 00:41:26.330 align:middle line:84% And so if you've ever played around 00:41:26.330 --> 00:41:28.130 align:middle line:84% with playing sounds backwards, you'll 00:41:28.130 --> 00:41:31.090 align:middle line:84% know that they sound kind of funny. 00:41:31.090 --> 00:41:34.447 align:middle line:90% And here's an example. 00:41:34.447 --> 00:41:35.114 align:middle line:90% [AUDIO PLAYBACK] 00:41:35.114 --> 00:41:39.770 align:middle line:84% - The effect of tone envelope on timbre. 00:41:39.770 --> 00:41:45.400 align:middle line:84% You will hear a recording of a Bach chorale played on a piano. 00:41:45.400 --> 00:41:48.370 align:middle line:90% [BACH CHORALE] 00:41:48.370 --> 00:42:18.070 align:middle line:90% 00:42:18.070 --> 00:42:19.758 align:middle line:90% [END PLAYBACK] 00:42:19.758 --> 00:42:20.550 align:middle line:90% JOSH MCDERMOTT: OK. 00:42:20.550 --> 00:42:22.610 align:middle line:90% Now, what they did-- 00:42:22.610 --> 00:42:25.070 align:middle line:90% this is a really old demo. 00:42:25.070 --> 00:42:28.730 align:middle line:84% And so they got a very talented musician 00:42:28.730 --> 00:42:30.970 align:middle line:84% to play the Bach chorale backwards. 00:42:30.970 --> 00:42:32.450 align:middle line:90% They recorded that. 00:42:32.450 --> 00:42:35.090 align:middle line:84% And then they time-reversed the tape. 00:42:35.090 --> 00:42:39.450 align:middle line:84% So now, you have the Bach chorale where all the notes are 00:42:39.450 --> 00:42:41.133 align:middle line:84% in the correct order, but each note 00:42:41.133 --> 00:42:43.550 align:middle line:84% is actually the time reversal of what it's supposed to be. 00:42:43.550 --> 00:42:45.063 align:middle line:90% And so it sounds pretty funny. 00:42:45.063 --> 00:42:45.730 align:middle line:90% [AUDIO PLAYBACK] 00:42:45.730 --> 00:42:47.690 align:middle line:84% - Now, the tape of the last recording 00:42:47.690 --> 00:42:51.770 align:middle line:84% is played backwards so that the chorale is heard forward again, 00:42:51.770 --> 00:42:55.000 align:middle line:84% but with an interesting difference. 00:42:55.000 --> 00:42:57.970 align:middle line:90% [BACH CHORALE, STRETCHED] 00:42:57.970 --> 00:43:08.860 align:middle line:90% 00:43:08.860 --> 00:43:09.557 align:middle line:90% [END PLAYBACK] 00:43:09.557 --> 00:43:11.390 align:middle line:84% JOSH MCDERMOTT: All right, you get the idea. 00:43:11.390 --> 00:43:12.932 align:middle line:84% So that's the piano played backwards. 00:43:12.932 --> 00:43:14.970 align:middle line:84% It doesn't sound at all like a piano anymore. 00:43:14.970 --> 00:43:16.970 align:middle line:84% And so obviously, music producers 00:43:16.970 --> 00:43:19.330 align:middle line:84% use effects like this now in modern times 00:43:19.330 --> 00:43:21.450 align:middle line:84% all the time to create psychedelic effects 00:43:21.450 --> 00:43:23.810 align:middle line:90% and things like that. 00:43:23.810 --> 00:43:27.810 align:middle line:84% So the amplitude envelope matters a lot. 00:43:27.810 --> 00:43:30.370 align:middle line:84% There's all kinds of really interesting timbral effects 00:43:30.370 --> 00:43:32.130 align:middle line:90% with voices. 00:43:32.130 --> 00:43:35.810 align:middle line:84% And in general, a lot of people will often 00:43:35.810 --> 00:43:39.250 align:middle line:84% say that musical instruments are intended to mimic voices. 00:43:39.250 --> 00:43:42.030 align:middle line:84% And in some contexts, maybe that's true. 00:43:42.030 --> 00:43:44.530 align:middle line:84% But voices do a lot of things that musical instruments often 00:43:44.530 --> 00:43:45.030 align:middle line:90% don't. 00:43:45.030 --> 00:43:48.250 align:middle line:84% And in particular, typically, the fundamental frequency 00:43:48.250 --> 00:43:51.310 align:middle line:84% of the voice will change fairly rapidly over time. 00:43:51.310 --> 00:43:53.490 align:middle line:84% Certainly, that happens in speech 00:43:53.490 --> 00:43:57.030 align:middle line:84% and often, when people sing, if there's any kind of vibrato. 00:43:57.030 --> 00:44:00.330 align:middle line:84% And so that turns out to be this very distinctive thing that 00:44:00.330 --> 00:44:03.490 align:middle line:84% makes something sound like a voice. 00:44:03.490 --> 00:44:06.063 align:middle line:90% Let's look at the spectrogram. 00:44:06.063 --> 00:44:06.730 align:middle line:90% [AUDIO PLAYBACK] 00:44:06.730 --> 00:44:11.050 align:middle line:84% - Demonstration 24, role of frequency modulation 00:44:11.050 --> 00:44:13.370 align:middle line:90% on voice perception. 00:44:13.370 --> 00:44:15.470 align:middle line:84% This demonstration is quite complex. 00:44:15.470 --> 00:44:19.050 align:middle line:84% Please refer to the booklet for a complete explanation. 00:44:19.050 --> 00:44:20.930 align:middle line:90% First, we hear a pure tone. 00:44:20.930 --> 00:44:22.730 align:middle line:84% Then it is joined by other partials 00:44:22.730 --> 00:44:25.050 align:middle line:90% that could make up a vowel. 00:44:25.050 --> 00:44:28.050 align:middle line:84% Our perception is that the pure tone continues on, 00:44:28.050 --> 00:44:30.290 align:middle line:90% accompanied by a rich one. 00:44:30.290 --> 00:44:32.930 align:middle line:84% Finally, when an identical vibrato 00:44:32.930 --> 00:44:37.490 align:middle line:84% is added to all the harmonics, the sounds fuse and a sung vowel 00:44:37.490 --> 00:44:38.610 align:middle line:90% emerges. 00:44:38.610 --> 00:44:41.430 align:middle line:84% This sequence is heard for three different vowels. 00:44:41.430 --> 00:44:44.410 align:middle line:90% 00:44:44.410 --> 00:44:47.386 align:middle line:90% [VIBRATING TONES] 00:44:47.386 --> 00:45:22.106 align:middle line:90% 00:45:22.106 --> 00:45:24.120 align:middle line:90% [END PLAYBACK] 00:45:24.120 --> 00:45:25.870 align:middle line:84% JOSH MCDERMOTT: OK, I think that's enough. 00:45:25.870 --> 00:45:27.010 align:middle line:90% You get the idea. 00:45:27.010 --> 00:45:28.130 align:middle line:90% But it's quite striking. 00:45:28.130 --> 00:45:29.850 align:middle line:84% So you get this thing that goes from sounding 00:45:29.850 --> 00:45:32.058 align:middle line:84% like a synthetic tone to something that really sounds 00:45:32.058 --> 00:45:34.100 align:middle line:84% voice-like, because you've learned 00:45:34.100 --> 00:45:38.452 align:middle line:84% to associate vibrato with that, presumably. 00:45:38.452 --> 00:45:40.160 align:middle line:84% What questions have you got about timbre? 00:45:40.160 --> 00:45:42.980 align:middle line:90% 00:45:42.980 --> 00:45:43.498 align:middle line:90% Yeah? 00:45:43.498 --> 00:45:45.540 align:middle line:84% AUDIENCE: I'm kind of curious how the effect of-- 00:45:45.540 --> 00:45:48.580 align:middle line:84% I'm not sure what to call it, reverb? 00:45:48.580 --> 00:45:51.700 align:middle line:84% But the effect of the space that the sound exists in 00:45:51.700 --> 00:45:52.720 align:middle line:90% has on timbre. 00:45:52.720 --> 00:45:54.800 align:middle line:84% I'm thinking like with the saxophone. 00:45:54.800 --> 00:45:57.320 align:middle line:84% Sometimes, if you play with a different mouthpiece, 00:45:57.320 --> 00:45:59.260 align:middle line:84% it will be described as having a brassier 00:45:59.260 --> 00:46:02.425 align:middle line:90% timbre or a non-brassy timbre. 00:46:02.425 --> 00:46:03.300 align:middle line:90% JOSH MCDERMOTT: Yeah. 00:46:03.300 --> 00:46:04.667 align:middle line:90% Well, that I mean-- 00:46:04.667 --> 00:46:07.000 align:middle line:84% would you describe that as-- that's not about the space. 00:46:07.000 --> 00:46:09.540 align:middle line:84% That's just you're changing the instrument, right? 00:46:09.540 --> 00:46:10.620 align:middle line:90% AUDIENCE: Well, I guess. 00:46:10.620 --> 00:46:14.500 align:middle line:84% Yeah, the place where the sound first comes in. 00:46:14.500 --> 00:46:16.060 align:middle line:90% JOSH MCDERMOTT: Yeah. 00:46:16.060 --> 00:46:17.860 align:middle line:84% I mean, again, you can think about this 00:46:17.860 --> 00:46:18.720 align:middle line:90% in a lot of different ways. 00:46:18.720 --> 00:46:20.380 align:middle line:84% But I mean, you are changing the sound. 00:46:20.380 --> 00:46:23.180 align:middle line:84% It's going to probably filter the sound in different ways. 00:46:23.180 --> 00:46:29.520 align:middle line:84% And the question is, why does that make it feel different? 00:46:29.520 --> 00:46:32.060 align:middle line:84% Why do we associate those sounds with-- 00:46:32.060 --> 00:46:34.840 align:middle line:84% I don't know-- different emotional emphasis or something? 00:46:34.840 --> 00:46:38.980 align:middle line:90% And it's not totally clear. 00:46:38.980 --> 00:46:42.700 align:middle line:84% Some people would say that the emotional associations that you 00:46:42.700 --> 00:46:46.380 align:middle line:84% have with timbre maybe derive from experience with voices, 00:46:46.380 --> 00:46:50.460 align:middle line:84% that you hear people vocalizing under different emotional 00:46:50.460 --> 00:46:51.080 align:middle line:90% conditions. 00:46:51.080 --> 00:46:54.560 align:middle line:84% And then some instrument effects are trying to mimic that. 00:46:54.560 --> 00:46:57.340 align:middle line:84% That's possible and probably true to some extent. 00:46:57.340 --> 00:47:00.870 align:middle line:90% 00:47:00.870 --> 00:47:03.160 align:middle line:84% But yeah, it's always a little hard to say. 00:47:03.160 --> 00:47:05.820 align:middle line:90% 00:47:05.820 --> 00:47:07.668 align:middle line:84% Yeah, so timbre, it's very-- yeah, 00:47:07.668 --> 00:47:08.960 align:middle line:90% it's a lot of fun to listen to. 00:47:08.960 --> 00:47:10.420 align:middle line:90% It's hard to talk about. 00:47:10.420 --> 00:47:13.660 align:middle line:84% And yeah, often hard to think about. 00:47:13.660 --> 00:47:14.380 align:middle line:90% Yeah? 00:47:14.380 --> 00:47:17.000 align:middle line:84% AUDIENCE: I guess you preempted my question a bit. 00:47:17.000 --> 00:47:21.900 align:middle line:84% But has anyone tried to subdivide timbre into things 00:47:21.900 --> 00:47:23.980 align:middle line:84% aside from what we just discussed about, 00:47:23.980 --> 00:47:25.755 align:middle line:90% like envelope and stuff? 00:47:25.755 --> 00:47:26.880 align:middle line:90% JOSH MCDERMOTT: Not really. 00:47:26.880 --> 00:47:29.020 align:middle line:84% So the main thing that's been done 00:47:29.020 --> 00:47:33.140 align:middle line:84% is there's been a fair bit of work on instrument sounds, 00:47:33.140 --> 00:47:37.980 align:middle line:84% where you can do things like ask people to rate 00:47:37.980 --> 00:47:40.363 align:middle line:84% the similarity of different instrument sounds. 00:47:40.363 --> 00:47:41.780 align:middle line:84% And then there's analyses that you 00:47:41.780 --> 00:47:44.460 align:middle line:84% can do to try to figure out what are the main dimensions that 00:47:44.460 --> 00:47:46.980 align:middle line:84% determine the variation and similarity. 00:47:46.980 --> 00:47:48.740 align:middle line:84% And the main thing that comes out of that 00:47:48.740 --> 00:47:51.380 align:middle line:90% are really two main dimensions. 00:47:51.380 --> 00:47:53.740 align:middle line:90% One is the speed of the attack. 00:47:53.740 --> 00:47:57.080 align:middle line:84% So it's how abrupt the amplitude envelope is. 00:47:57.080 --> 00:47:59.300 align:middle line:84% So that's one of the things that we talked about. 00:47:59.300 --> 00:48:03.420 align:middle line:84% And then the other is the spectral centroid-- so how 00:48:03.420 --> 00:48:05.660 align:middle line:90% bright or dull the sound is. 00:48:05.660 --> 00:48:09.442 align:middle line:84% So some sounds have a lot more energy at high frequencies. 00:48:09.442 --> 00:48:11.400 align:middle line:84% Some have a lot more energy at low frequencies. 00:48:11.400 --> 00:48:13.380 align:middle line:84% So that's what comes out of that. 00:48:13.380 --> 00:48:15.780 align:middle line:84% And it's always a little underwhelming, 00:48:15.780 --> 00:48:17.420 align:middle line:84% because you listen to instruments. 00:48:17.420 --> 00:48:19.160 align:middle line:84% And they're so much fun to listen to. 00:48:19.160 --> 00:48:20.480 align:middle line:90% And it feels very rich. 00:48:20.480 --> 00:48:23.923 align:middle line:84% And obviously, when you listen to a musician, 00:48:23.923 --> 00:48:26.340 align:middle line:84% you can hear all kinds of stuff about how they're playing. 00:48:26.340 --> 00:48:28.598 align:middle line:90% So there's a lot more to it. 00:48:28.598 --> 00:48:30.640 align:middle line:84% But yeah, that's the main thing that's been done. 00:48:30.640 --> 00:48:32.500 align:middle line:90% Yeah? 00:48:32.500 --> 00:48:35.860 align:middle line:84% AUDIENCE: Besides bright and dull, 00:48:35.860 --> 00:48:41.740 align:middle line:84% how do people describe timbre, especially in the science 00:48:41.740 --> 00:48:42.980 align:middle line:90% fields? 00:48:42.980 --> 00:48:44.980 align:middle line:84% JOSH MCDERMOTT: Yeah, they don't. 00:48:44.980 --> 00:48:50.365 align:middle line:84% So we talk about brightness and sharpness. 00:48:50.365 --> 00:48:52.240 align:middle line:84% I mean, there's a very small number of words. 00:48:52.240 --> 00:48:53.900 align:middle line:90% So yeah, there's sharpness. 00:48:53.900 --> 00:48:55.140 align:middle line:90% There's roughness. 00:48:55.140 --> 00:48:57.880 align:middle line:84% That's how amplitude-modulated something is. 00:48:57.880 --> 00:49:00.060 align:middle line:84% So if a sound has got a lot of amplitude modulation 00:49:00.060 --> 00:49:01.685 align:middle line:84% in a certain range, we'll call it rough 00:49:01.685 --> 00:49:04.580 align:middle line:90% because it kind of sounds rough. 00:49:04.580 --> 00:49:08.023 align:middle line:84% And yeah, I mean, people just make stuff up. 00:49:08.023 --> 00:49:09.940 align:middle line:84% And different people will use different words. 00:49:09.940 --> 00:49:12.273 align:middle line:84% And they're inconsistent in how they use them and stuff. 00:49:12.273 --> 00:49:15.140 align:middle line:84% It's hard to talk about these things. 00:49:15.140 --> 00:49:18.220 align:middle line:84% Yeah, so it's interesting that it's not very 00:49:18.220 --> 00:49:22.340 align:middle line:90% well-connected to language. 00:49:22.340 --> 00:49:25.980 align:middle line:90% Yeah, OK. 00:49:25.980 --> 00:49:31.500 align:middle line:84% Last thing we'll talk about here is intensity and loudness. 00:49:31.500 --> 00:49:35.003 align:middle line:84% So loudness is like this other very obvious property of sound. 00:49:35.003 --> 00:49:35.920 align:middle line:90% Some sounds are quiet. 00:49:35.920 --> 00:49:37.820 align:middle line:90% Some sounds are loud. 00:49:37.820 --> 00:49:40.260 align:middle line:84% It's a very important thing to actually understand 00:49:40.260 --> 00:49:41.860 align:middle line:84% because it's one of the main things 00:49:41.860 --> 00:49:45.272 align:middle line:90% that people complain about. 00:49:45.272 --> 00:49:47.222 align:middle line:90% So you're living in a city. 00:49:47.222 --> 00:49:48.180 align:middle line:90% There's too much noise. 00:49:48.180 --> 00:49:49.520 align:middle line:84% And so you complain to the authorities. 00:49:49.520 --> 00:49:50.720 align:middle line:84% And then they're supposed to do something. 00:49:50.720 --> 00:49:52.262 align:middle line:84% And so in order to do that, they need 00:49:52.262 --> 00:49:55.020 align:middle line:84% to be able to measure or estimate the loudness. 00:49:55.020 --> 00:49:56.860 align:middle line:84% Or if you're a car manufacturer, people 00:49:56.860 --> 00:49:58.600 align:middle line:84% will complain if the car is too loud. 00:49:58.600 --> 00:50:01.660 align:middle line:84% And so you want to be able to model and predict 00:50:01.660 --> 00:50:05.300 align:middle line:90% how loud the car noise is. 00:50:05.300 --> 00:50:07.300 align:middle line:84% And so this is something that's been the subject 00:50:07.300 --> 00:50:10.820 align:middle line:90% of quite a bit of research. 00:50:10.820 --> 00:50:13.300 align:middle line:84% In the very early days of psychophysics, 00:50:13.300 --> 00:50:15.620 align:middle line:84% there was a lot of work done on what's 00:50:15.620 --> 00:50:18.460 align:middle line:84% called magnitude estimation, where people would 00:50:18.460 --> 00:50:20.700 align:middle line:84% do these experiments where they'd play people, 00:50:20.700 --> 00:50:22.180 align:middle line:90% in this case a sound. 00:50:22.180 --> 00:50:25.670 align:middle line:84% And they'd ask them to rate how loud it is, just using a number. 00:50:25.670 --> 00:50:31.510 align:middle line:84% And so the person who most is associated with this approach is 00:50:31.510 --> 00:50:34.590 align:middle line:84% S. S. Stevens, who is a psychophysicist who was working 00:50:34.590 --> 00:50:37.110 align:middle line:90% at Harvard like in the '40s. 00:50:37.110 --> 00:50:39.750 align:middle line:84% And the conclusion of these experiments 00:50:39.750 --> 00:50:42.630 align:middle line:84% is that loudness can be described as a power law. 00:50:42.630 --> 00:50:45.150 align:middle line:84% So the loudness that someone will ascribe to a sound 00:50:45.150 --> 00:50:47.830 align:middle line:84% is proportional to the physical intensity 00:50:47.830 --> 00:50:52.350 align:middle line:84% of the sound raised to a power of approximately 0.3. 00:50:52.350 --> 00:50:54.170 align:middle line:90% It's an empirical fact. 00:50:54.170 --> 00:50:56.910 align:middle line:84% So if you have sound intensity on this axis 00:50:56.910 --> 00:50:59.330 align:middle line:84% and the magnitude estimation on this axis, 00:50:59.330 --> 00:51:03.976 align:middle line:90% you get a lawful relationship. 00:51:03.976 --> 00:51:05.890 align:middle line:84% So what's significant about this? 00:51:05.890 --> 00:51:08.770 align:middle line:84% Well, again, this tends to be fairly consistent across people. 00:51:08.770 --> 00:51:12.430 align:middle line:84% And you can use this to predict the extent to which 00:51:12.430 --> 00:51:15.790 align:middle line:84% a change in sound level, that you can measure in decibels, 00:51:15.790 --> 00:51:17.570 align:middle line:84% would translate to a change in loudness. 00:51:17.570 --> 00:51:22.390 align:middle line:84% And so what that formula tells us is that a 10-decibel increase 00:51:22.390 --> 00:51:26.070 align:middle line:84% in the level of a sound will give approximately a doubling 00:51:26.070 --> 00:51:27.550 align:middle line:90% in loudness. 00:51:27.550 --> 00:51:29.390 align:middle line:84% So if you go from something that's 00:51:29.390 --> 00:51:31.270 align:middle line:84% 60 dB to something that's 70 dB, it 00:51:31.270 --> 00:51:33.230 align:middle line:84% should sound about twice as loud. 00:51:33.230 --> 00:51:37.610 align:middle line:84% And that's because, when you have a 10 dB increase in level, 00:51:37.610 --> 00:51:39.990 align:middle line:84% that's a 10-fold increase in intensity. 00:51:39.990 --> 00:51:43.590 align:middle line:84% And so 10 to the power of 0.3 is about 2. 00:51:43.590 --> 00:51:44.090 align:middle line:90% All right. 00:51:44.090 --> 00:51:47.950 align:middle line:84% So that's just a useful rule of thumb to know about. 00:51:47.950 --> 00:51:50.790 align:middle line:84% So that's one fact that's worth knowing, I think, 00:51:50.790 --> 00:51:54.643 align:middle line:90% about loudness and intensity. 00:51:54.643 --> 00:51:56.310 align:middle line:84% Another fact that is worth knowing about 00:51:56.310 --> 00:51:57.830 align:middle line:84% is something that we talked a little 00:51:57.830 --> 00:51:59.670 align:middle line:90% bit about earlier in the class. 00:51:59.670 --> 00:52:01.630 align:middle line:84% And that is that it's kind of interesting 00:52:01.630 --> 00:52:03.750 align:middle line:84% that people can discriminate intensities 00:52:03.750 --> 00:52:07.630 align:middle line:84% quite well over this enormous dynamic range. 00:52:07.630 --> 00:52:09.440 align:middle line:84% And so this is something that, again, 00:52:09.440 --> 00:52:10.690 align:middle line:90% has been measured quite a lot. 00:52:10.690 --> 00:52:12.450 align:middle line:90% This is one example. 00:52:12.450 --> 00:52:15.630 align:middle line:84% So this is a graph that is plotting the intensity 00:52:15.630 --> 00:52:18.790 align:middle line:84% discrimination threshold, in decibels, 00:52:18.790 --> 00:52:22.430 align:middle line:84% as a function of the intensity of the sounds. 00:52:22.430 --> 00:52:26.970 align:middle line:84% So 0 here, this is in SPL, dB SPL. 00:52:26.970 --> 00:52:30.050 align:middle line:84% So remember, that's relative to this reference that's 00:52:30.050 --> 00:52:32.050 align:middle line:84% thought to be close to the threshold of hearing. 00:52:32.050 --> 00:52:36.170 align:middle line:84% So 0 dB SPL basically is just what you can just barely hear. 00:52:36.170 --> 00:52:40.110 align:middle line:84% So when you get down to these really low stimulus intensities, 00:52:40.110 --> 00:52:41.850 align:middle line:90% the thresholds are not so good. 00:52:41.850 --> 00:52:46.670 align:middle line:84% But then once you're above about dB SPL, all the way up to 85, 00:52:46.670 --> 00:52:52.390 align:middle line:84% the thresholds are less than 1 decibel. 00:52:52.390 --> 00:52:53.490 align:middle line:90% All right. 00:52:53.490 --> 00:52:57.290 align:middle line:84% Again, it's just a fact about what people can discriminate. 00:52:57.290 --> 00:53:00.848 align:middle line:84% Intensity resolution is pretty good. 00:53:00.848 --> 00:53:02.390 align:middle line:84% And so one of the reasons that people 00:53:02.390 --> 00:53:04.218 align:middle line:84% have thought this was interesting 00:53:04.218 --> 00:53:06.010 align:middle line:84% is that, if you look in the nervous system, 00:53:06.010 --> 00:53:08.190 align:middle line:84% in particular in the auditory nerve, 00:53:08.190 --> 00:53:12.870 align:middle line:84% most auditory nerve fibers have narrow dynamic ranges. 00:53:12.870 --> 00:53:15.750 align:middle line:84% So what that means is that if you measure the response 00:53:15.750 --> 00:53:18.710 align:middle line:84% of a nerve fiber-- in this case, this is spikes per second-- 00:53:18.710 --> 00:53:23.310 align:middle line:84% as a function of the intensity of a stimulus-- in this case, 00:53:23.310 --> 00:53:26.870 align:middle line:84% this is the tone at the characteristic frequency-- 00:53:26.870 --> 00:53:29.750 align:middle line:84% what happens is that, well, when the intensity is too low, 00:53:29.750 --> 00:53:32.497 align:middle line:84% you just get the spontaneous rate of the fiber. 00:53:32.497 --> 00:53:34.330 align:middle line:84% And then at some point, you get a threshold. 00:53:34.330 --> 00:53:37.830 align:middle line:84% And that means that the response starts to increase. 00:53:37.830 --> 00:53:40.010 align:middle line:84% But then at some point, you saturate. 00:53:40.010 --> 00:53:43.030 align:middle line:84% It's like that's the highest intensity over which 00:53:43.030 --> 00:53:45.670 align:middle line:90% the response will change. 00:53:45.670 --> 00:53:47.710 align:middle line:84% And so what you can see here is that, here, 00:53:47.710 --> 00:53:49.890 align:middle line:84% the threshold of the tone is maybe about decibels. 00:53:49.890 --> 00:53:52.130 align:middle line:84% And then it's saturating here at maybe-- 00:53:52.130 --> 00:53:53.970 align:middle line:90% I don't know-- 50 or 55. 00:53:53.970 --> 00:53:58.870 align:middle line:84% So that's a very fairly narrow dynamic range of maybe 25-30 dB. 00:53:58.870 --> 00:54:01.935 align:middle line:90% 00:54:01.935 --> 00:54:03.310 align:middle line:84% It gets a little more complicated 00:54:03.310 --> 00:54:06.030 align:middle line:84% because, if you look at the auditory nerve, 00:54:06.030 --> 00:54:09.150 align:middle line:84% there tend to be these different types of fibers. 00:54:09.150 --> 00:54:11.910 align:middle line:84% So some of the fibers are like this one, 00:54:11.910 --> 00:54:14.150 align:middle line:84% where the dynamic range is fairly narrow 00:54:14.150 --> 00:54:17.150 align:middle line:84% and the spontaneous rate is high. 00:54:17.150 --> 00:54:19.890 align:middle line:84% So on its own, without any sound stimulation, 00:54:19.890 --> 00:54:22.570 align:middle line:84% the neuron is spiking around 30 spikes per second. 00:54:22.570 --> 00:54:27.470 align:middle line:84% You also have nerve fibers that have lower spontaneous rates 00:54:27.470 --> 00:54:31.110 align:middle line:84% and that have a wider dynamic range, 00:54:31.110 --> 00:54:33.390 align:middle line:84% and a fact about the auditory nerve. 00:54:33.390 --> 00:54:36.830 align:middle line:84% So the puzzle here is that we seem 00:54:36.830 --> 00:54:39.670 align:middle line:84% to be able to register and discriminate intensities 00:54:39.670 --> 00:54:42.190 align:middle line:84% over this enormous range, from all the way 00:54:42.190 --> 00:54:45.950 align:middle line:84% down here to all the way up like above here. 00:54:45.950 --> 00:54:48.830 align:middle line:84% But the nerve fibers only change their response 00:54:48.830 --> 00:54:50.770 align:middle line:90% over a fairly narrow range. 00:54:50.770 --> 00:54:52.710 align:middle line:84% So the question is, well, once the nerve fiber 00:54:52.710 --> 00:54:55.350 align:middle line:84% is saturated, the response isn't changing, 00:54:55.350 --> 00:54:57.650 align:middle line:84% if you increase the stimulus beyond that. 00:54:57.650 --> 00:55:00.430 align:middle line:84% And so how could you tell that the intensity 00:55:00.430 --> 00:55:03.830 align:middle line:90% was continuing to go up? 00:55:03.830 --> 00:55:04.330 align:middle line:90% All right. 00:55:04.330 --> 00:55:06.413 align:middle line:84% So that's what's called the dynamic range problem. 00:55:06.413 --> 00:55:10.397 align:middle line:84% It's a well-known thing people have puzzled about. 00:55:10.397 --> 00:55:12.230 align:middle line:84% And there's a handful of solutions that have 00:55:12.230 --> 00:55:14.200 align:middle line:90% been proposed over the years. 00:55:14.200 --> 00:55:16.960 align:middle line:84% One is what is called off-frequency listening. 00:55:16.960 --> 00:55:20.240 align:middle line:84% And so the idea there is that-- so let's suppose 00:55:20.240 --> 00:55:22.860 align:middle line:84% you're discriminating the intensity of a tone at 1 00:55:22.860 --> 00:55:23.460 align:middle line:90% kilohertz. 00:55:23.460 --> 00:55:25.480 align:middle line:84% When the intensity gets really high, 00:55:25.480 --> 00:55:30.280 align:middle line:84% the nerve fibers that have a characteristic frequency 00:55:30.280 --> 00:55:32.940 align:middle line:84% near a kilohertz may be saturated. 00:55:32.940 --> 00:55:35.680 align:middle line:84% But the ones that are further away won't be. 00:55:35.680 --> 00:55:38.000 align:middle line:84% And so you could somehow listen to those. 00:55:38.000 --> 00:55:40.080 align:middle line:84% So that's off-frequency listening. 00:55:40.080 --> 00:55:42.760 align:middle line:84% There's also the notion that you might use temporal information, 00:55:42.760 --> 00:55:44.240 align:middle line:90% like phase locking. 00:55:44.240 --> 00:55:47.698 align:middle line:84% So you see increases in synchrony with levels. 00:55:47.698 --> 00:55:49.240 align:middle line:84% Of course, you can only use this when 00:55:49.240 --> 00:55:51.240 align:middle line:84% you have phase locking, which is for frequencies 00:55:51.240 --> 00:55:53.160 align:middle line:90% that are below 4 kilohertz. 00:55:53.160 --> 00:55:56.560 align:middle line:84% And then the other proposal is that these low spontaneous rate 00:55:56.560 --> 00:55:58.600 align:middle line:84% fibers might be really important. 00:55:58.600 --> 00:56:02.098 align:middle line:84% So those have higher thresholds, but greater dynamic range. 00:56:02.098 --> 00:56:03.640 align:middle line:84% And people have wondered about, well, 00:56:03.640 --> 00:56:05.720 align:middle line:84% if they're so important, why are they 00:56:05.720 --> 00:56:09.800 align:middle line:84% a relatively small proportion of the auditory nerve? 00:56:09.800 --> 00:56:10.300 align:middle line:90% All right. 00:56:10.300 --> 00:56:12.440 align:middle line:84% So that's the dynamic range problem. 00:56:12.440 --> 00:56:14.880 align:middle line:84% And I would say the solution to this 00:56:14.880 --> 00:56:17.040 align:middle line:84% is still something that is not-- there's not 00:56:17.040 --> 00:56:21.320 align:middle line:84% total consensus about how this works. 00:56:21.320 --> 00:56:27.760 align:middle line:84% So we've talked about, over the past couple lectures, 00:56:27.760 --> 00:56:30.100 align:middle line:84% the properties of individual sounds. 00:56:30.100 --> 00:56:31.880 align:middle line:84% We describe individual sounds in terms 00:56:31.880 --> 00:56:34.060 align:middle line:84% of their loudness, their location, their pitch, 00:56:34.060 --> 00:56:35.560 align:middle line:90% their timbre. 00:56:35.560 --> 00:56:38.208 align:middle line:84% So all of these are perceptual variables that tell us things 00:56:38.208 --> 00:56:40.000 align:middle line:84% about the physical events in the world that 00:56:40.000 --> 00:56:42.840 align:middle line:84% produce the sounds, which is presumably the point of hearing 00:56:42.840 --> 00:56:44.080 align:middle line:90% them in the first place. 00:56:44.080 --> 00:56:46.400 align:middle line:84% So timbre is the richest, but it's 00:56:46.400 --> 00:56:49.400 align:middle line:84% the least defined and the least understood of these dimensions. 00:56:49.400 --> 00:56:51.240 align:middle line:84% And what we're going to talk about now 00:56:51.240 --> 00:56:55.560 align:middle line:84% is what happens when more than one sound occurs at once, when 00:56:55.560 --> 00:56:57.600 align:middle line:90% we have an auditory scene. 00:56:57.600 --> 00:57:01.740 align:middle line:84% Before I do that, any questions about loudness or anything else? 00:57:01.740 --> 00:57:05.600 align:middle line:90% 00:57:05.600 --> 00:57:06.100 align:middle line:90% Yeah? 00:57:06.100 --> 00:57:08.420 align:middle line:84% AUDIENCE: I have a question about the tonotopy 00:57:08.420 --> 00:57:10.010 align:middle line:90% that we saw in the fMRI scan. 00:57:10.010 --> 00:57:10.885 align:middle line:90% JOSH MCDERMOTT: Yeah? 00:57:10.885 --> 00:57:13.320 align:middle line:84% AUDIENCE: [INAUDIBLE] similar attributions 00:57:13.320 --> 00:57:15.700 align:middle line:84% in the auditory cortex in deaf individuals? 00:57:15.700 --> 00:57:19.920 align:middle line:90% 00:57:19.920 --> 00:57:23.200 align:middle line:84% JOSH MCDERMOTT: So you're asking, is there a way 00:57:23.200 --> 00:57:27.600 align:middle line:84% to activate this bit of cortex in someone who's deaf? 00:57:27.600 --> 00:57:28.585 align:middle line:90% AUDIENCE: Yeah. 00:57:28.585 --> 00:57:29.460 align:middle line:90% JOSH MCDERMOTT: Yeah. 00:57:29.460 --> 00:57:37.880 align:middle line:84% So I think that the standard thinking on someone who lacks 00:57:37.880 --> 00:57:41.080 align:middle line:90% a sense-- so deaf or blind-- 00:57:41.080 --> 00:57:45.040 align:middle line:84% is that the cortex that's normally allocated to that sense 00:57:45.040 --> 00:57:47.120 align:middle line:90% tends to get repurposed. 00:57:47.120 --> 00:57:51.760 align:middle line:84% AUDIENCE: So what kind of stimuli would activate-- 00:57:51.760 --> 00:57:54.160 align:middle line:84% produce similar activations, I guess? 00:57:54.160 --> 00:57:56.160 align:middle line:84% JOSH MCDERMOTT: Well, I don't know if what would 00:57:56.160 --> 00:57:57.380 align:middle line:90% produce similar activations. 00:57:57.380 --> 00:57:59.120 align:middle line:90% But I think it seems-- 00:57:59.120 --> 00:58:01.460 align:middle line:84% I'm not sure if this is actually known. 00:58:01.460 --> 00:58:03.080 align:middle line:84% But it seems very plausible that, 00:58:03.080 --> 00:58:06.200 align:middle line:84% in someone who is deaf, that you might 00:58:06.200 --> 00:58:10.882 align:middle line:84% get visual stimuli actually activating the auditory cortex. 00:58:10.882 --> 00:58:12.340 align:middle line:84% So you often see the reverse thing. 00:58:12.340 --> 00:58:15.800 align:middle line:84% So people who are blind, when they're listening to sounds, 00:58:15.800 --> 00:58:17.840 align:middle line:84% you often get a lot of visual cortex activation 00:58:17.840 --> 00:58:20.760 align:middle line:84% that you wouldn't necessarily see in someone who's sighted. 00:58:20.760 --> 00:58:24.900 align:middle line:84% So the idea is that the cortex gets repurposed. 00:58:24.900 --> 00:58:29.080 align:middle line:84% And there's lots of debate over like, well, 00:58:29.080 --> 00:58:32.260 align:middle line:84% exactly what can the cortex be repurposed for. 00:58:32.260 --> 00:58:36.240 align:middle line:90% And I think that's unresolved. 00:58:36.240 --> 00:58:39.040 align:middle line:90% Yeah. 00:58:39.040 --> 00:58:42.520 align:middle line:90% What other questions you got? 00:58:42.520 --> 00:58:43.080 align:middle line:90% Yeah? 00:58:43.080 --> 00:58:45.640 align:middle line:84% AUDIENCE: Does the loudness estimate 00:58:45.640 --> 00:58:48.720 align:middle line:84% depend on the frequency of the sound? 00:58:48.720 --> 00:58:52.780 align:middle line:84% Are there some frequencies that may appear louder or quieter? 00:58:52.780 --> 00:58:55.247 align:middle line:90% 00:58:55.247 --> 00:58:56.080 align:middle line:90% JOSH MCDERMOTT: Yes. 00:58:56.080 --> 00:58:59.360 align:middle line:84% There are probably small effects of frequency. 00:58:59.360 --> 00:59:02.160 align:middle line:84% I mean, obviously, your detection threshold varies quite 00:59:02.160 --> 00:59:05.160 align:middle line:90% a lot depending on frequency. 00:59:05.160 --> 00:59:08.598 align:middle line:84% And so you'd want to compensate for that. 00:59:08.598 --> 00:59:10.140 align:middle line:84% And once you've compensated for that, 00:59:10.140 --> 00:59:11.300 align:middle line:90% there's probably some effects. 00:59:11.300 --> 00:59:12.880 align:middle line:84% I don't actually know what they are. 00:59:12.880 --> 00:59:17.280 align:middle line:84% The thing that has a bigger effect on loudness 00:59:17.280 --> 00:59:21.860 align:middle line:84% is bandwidth, which is on your problem set. 00:59:21.860 --> 00:59:25.880 align:middle line:90% 00:59:25.880 --> 00:59:26.680 align:middle line:90% Cool. 00:59:26.680 --> 00:59:31.100 align:middle line:84% OK, let's start talking about auditory scene analysis. 00:59:31.100 --> 00:59:34.480 align:middle line:90% 00:59:34.480 --> 00:59:38.960 align:middle line:84% So the problem that we're going to be talking about 00:59:38.960 --> 00:59:43.840 align:middle line:84% is that many acoustic events happen in the world at once. 00:59:43.840 --> 00:59:48.000 align:middle line:84% But your ear only receives one sound wave. 00:59:48.000 --> 00:59:50.840 align:middle line:84% And so the sound wave that is received by your ear 00:59:50.840 --> 00:59:53.160 align:middle line:84% is the sum of those that would have 00:59:53.160 --> 00:59:55.320 align:middle line:84% been produced by the individual events 00:59:55.320 --> 00:59:59.040 align:middle line:90% if they happened on their own. 00:59:59.040 --> 01:00:02.960 align:middle line:84% Somehow, the brain has to distinguish the different events 01:00:02.960 --> 01:00:05.250 align:middle line:84% that happened in the world from the single signal 01:00:05.250 --> 01:00:08.210 align:middle line:90% that it receives. 01:00:08.210 --> 01:00:11.770 align:middle line:84% So for instance, if I'm talking and somebody walks in 01:00:11.770 --> 01:00:13.530 align:middle line:84% and you hear the door close, you need 01:00:13.530 --> 01:00:17.090 align:middle line:84% to be able to distinguish that door closing from my voice, 01:00:17.090 --> 01:00:20.210 align:middle line:84% because you might want to know whether somebody came in. 01:00:20.210 --> 01:00:21.787 align:middle line:84% Similarly, if you're at a restaurant, 01:00:21.787 --> 01:00:23.370 align:middle line:84% and you're trying to talk to somebody, 01:00:23.370 --> 01:00:24.370 align:middle line:84% and there's all this noise, you need 01:00:24.370 --> 01:00:26.250 align:middle line:84% to be able to distinguish the different sound sources. 01:00:26.250 --> 01:00:27.190 align:middle line:90% So this happens all the time. 01:00:27.190 --> 01:00:29.070 align:middle line:84% And people are really pretty good at it. 01:00:29.070 --> 01:00:33.100 align:middle line:84% So here's the classic cocktail party version of the problem. 01:00:33.100 --> 01:00:34.350 align:middle line:90% So you're talking to somebody. 01:00:34.350 --> 01:00:35.590 align:middle line:90% You want to understand this. 01:00:35.590 --> 01:00:35.770 align:middle line:90% [AUDIO PLAYBACK] 01:00:35.770 --> 01:00:37.150 align:middle line:90% - She argues with her sister. 01:00:37.150 --> 01:00:39.608 align:middle line:84% JOSH MCDERMOTT: But maybe what comes into your ear is this. 01:00:39.608 --> 01:00:40.910 align:middle line:90% - She argues with her sister. 01:00:40.910 --> 01:00:41.450 align:middle line:90% JOSH MCDERMOTT: Or this. 01:00:41.450 --> 01:00:42.610 align:middle line:90% [INTERPOSING VOICES] 01:00:42.610 --> 01:00:43.970 align:middle line:90% JOSH MCDERMOTT: Or this. 01:00:43.970 --> 01:00:46.330 align:middle line:90% [INTERPOSING VOICES] 01:00:46.330 --> 01:00:47.770 align:middle line:90% [END PLAYBACK] 01:00:47.770 --> 01:00:50.610 align:middle line:84% JOSH MCDERMOTT: So the presence of the other talkers 01:00:50.610 --> 01:00:52.030 align:middle line:90% obscures a lot of the structure. 01:00:52.030 --> 01:00:55.530 align:middle line:84% And the target utterance creates this very complicated signal. 01:00:55.530 --> 01:00:58.110 align:middle line:84% But the speech remains largely intelligible. 01:00:58.110 --> 01:01:00.410 align:middle line:84% And people are still substantially better 01:01:00.410 --> 01:01:03.550 align:middle line:84% at this than present day speech recognition algorithms. 01:01:03.550 --> 01:01:08.650 align:middle line:84% Although, that may not be true for very much longer. 01:01:08.650 --> 01:01:09.150 align:middle line:90% All right. 01:01:09.150 --> 01:01:12.430 align:middle line:84% So one interesting perspective on this problem, 01:01:12.430 --> 01:01:14.810 align:middle line:84% which is what we talked about in the very first lecture 01:01:14.810 --> 01:01:17.530 align:middle line:84% to the class, is that you can think of the problem 01:01:17.530 --> 01:01:19.330 align:middle line:84% that the auditory system is solving 01:01:19.330 --> 01:01:22.530 align:middle line:84% as being akin to me giving you this equation 01:01:22.530 --> 01:01:24.930 align:middle line:90% and asking you to solve for x. 01:01:24.930 --> 01:01:28.610 align:middle line:84% So that's a ridiculous thing for me to ask you to do. 01:01:28.610 --> 01:01:31.030 align:middle line:84% But that's like exactly what's happening at some level, 01:01:31.030 --> 01:01:34.730 align:middle line:84% because you're measuring this one signal that 01:01:34.730 --> 01:01:38.250 align:middle line:84% is the sum of multiple signals that 01:01:38.250 --> 01:01:40.010 align:middle line:90% were generated in the world. 01:01:40.010 --> 01:01:42.300 align:middle line:84% And then you're trying to estimate one or more 01:01:42.300 --> 01:01:44.050 align:middle line:84% of those one signal, the thing that you're 01:01:44.050 --> 01:01:46.850 align:middle line:90% trying to understand. 01:01:46.850 --> 01:01:47.350 align:middle line:90% All right. 01:01:47.350 --> 01:01:50.190 align:middle line:84% So this is a classic example of an ill-posed problem. 01:01:50.190 --> 01:01:55.370 align:middle line:84% It's ill-posed because there are many combinations of source 01:01:55.370 --> 01:01:59.490 align:middle line:84% signals that could add up to be equal to the observed mixture 01:01:59.490 --> 01:02:01.833 align:middle line:90% signal. 01:02:01.833 --> 01:02:03.750 align:middle line:84% So for every sound wave that the ear receives, 01:02:03.750 --> 01:02:06.210 align:middle line:84% there are really an infinite set of sound sources 01:02:06.210 --> 01:02:07.850 align:middle line:90% that could have generated it. 01:02:07.850 --> 01:02:09.690 align:middle line:90% So it's ill-posed. 01:02:09.690 --> 01:02:11.230 align:middle line:84% So as with any ill-posed problem, 01:02:11.230 --> 01:02:14.010 align:middle line:84% the brain has to make its best guess as to how many sources 01:02:14.010 --> 01:02:16.850 align:middle line:84% there are and what they are based on what 01:02:16.850 --> 01:02:18.730 align:middle line:90% it knows about the world. 01:02:18.730 --> 01:02:23.330 align:middle line:84% So another kind of fun way to think about this problem 01:02:23.330 --> 01:02:25.090 align:middle line:84% is this pictorial description that's 01:02:25.090 --> 01:02:27.090 align:middle line:90% normally attributed to Bregman. 01:02:27.090 --> 01:02:30.190 align:middle line:84% So imagine the situation here, where we've got a lake. 01:02:30.190 --> 01:02:32.570 align:middle line:84% There's a bunch of boats on the lake. 01:02:32.570 --> 01:02:34.610 align:middle line:84% There are these two little inlets 01:02:34.610 --> 01:02:37.850 align:middle line:84% that we've carved out of the edge of the lake. 01:02:37.850 --> 01:02:40.130 align:middle line:84% And we lay down two handkerchiefs 01:02:40.130 --> 01:02:43.370 align:middle line:84% so that the handkerchiefs float on the surface of the water. 01:02:43.370 --> 01:02:48.810 align:middle line:84% And we are allowed to measure the motion of these two 01:02:48.810 --> 01:02:50.170 align:middle line:90% handkerchiefs. 01:02:50.170 --> 01:02:53.110 align:middle line:84% And from the motion of the pieces of cloth, 01:02:53.110 --> 01:02:55.930 align:middle line:84% the handkerchiefs, we have to determine how many boats 01:02:55.930 --> 01:02:59.870 align:middle line:84% are in the lake, where they are in the lake, how big they are, 01:02:59.870 --> 01:03:02.690 align:middle line:84% or maybe where they're going, stuff like that. 01:03:02.690 --> 01:03:03.190 align:middle line:90% All right. 01:03:03.190 --> 01:03:04.310 align:middle line:90% So the analogy here-- 01:03:04.310 --> 01:03:05.610 align:middle line:90% we've got two ears. 01:03:05.610 --> 01:03:07.750 align:middle line:84% The ears are just measuring vibrations in the air. 01:03:07.750 --> 01:03:11.490 align:middle line:84% Those vibrations are originating with sources in the world. 01:03:11.490 --> 01:03:16.150 align:middle line:90% So it's kind of amazing. 01:03:16.150 --> 01:03:17.470 align:middle line:90% So how can we do this? 01:03:17.470 --> 01:03:20.010 align:middle line:84% Well, I think the crux of the issue 01:03:20.010 --> 01:03:22.030 align:middle line:84% is that the sounds that we have to hear, 01:03:22.030 --> 01:03:25.850 align:middle line:84% the sounds that occur in the real world, are not random. 01:03:25.850 --> 01:03:28.543 align:middle line:84% So this is a brief demonstration of the fact that sounds 01:03:28.543 --> 01:03:29.710 align:middle line:90% in the world are not random. 01:03:29.710 --> 01:03:32.382 align:middle line:84% So here are some real-world sounds. 01:03:32.382 --> 01:03:34.350 align:middle line:90% [ENGINE ROARING] 01:03:34.350 --> 01:03:37.794 align:middle line:90% [DOORBELL RINGING] 01:03:37.794 --> 01:03:39.762 align:middle line:90% [HAWK CALLING] 01:03:39.762 --> 01:03:42.658 align:middle line:90% [SWOOSHING] 01:03:42.658 --> 01:03:44.450 align:middle line:84% OK, so those are just sounds that you might 01:03:44.450 --> 01:03:46.890 align:middle line:90% hear in your everyday life. 01:03:46.890 --> 01:03:49.910 align:middle line:84% Let's compare those to sounds that are random. 01:03:49.910 --> 01:03:52.792 align:middle line:84% So I made these sounds to be fully random. 01:03:52.792 --> 01:03:53.750 align:middle line:90% So what does that mean? 01:03:53.750 --> 01:03:56.170 align:middle line:84% Well, a sound is just a big vector 01:03:56.170 --> 01:03:59.180 align:middle line:84% of numbers, where the number at every point in time 01:03:59.180 --> 01:04:01.120 align:middle line:84% tells you the pressure at that point in time. 01:04:01.120 --> 01:04:03.980 align:middle line:84% So you can make a random sound by randomly 01:04:03.980 --> 01:04:06.260 align:middle line:84% sampling each one of those numbers 01:04:06.260 --> 01:04:08.320 align:middle line:84% from an independent distribution. 01:04:08.320 --> 01:04:10.620 align:middle line:90% I used a Gaussian, I think. 01:04:10.620 --> 01:04:11.120 align:middle line:90% All right. 01:04:11.120 --> 01:04:13.080 align:middle line:84% So you could do this yourself in Python. 01:04:13.080 --> 01:04:14.880 align:middle line:84% So these are fully random sounds. 01:04:14.880 --> 01:04:16.780 align:middle line:84% Each sample is an independent draw. 01:04:16.780 --> 01:04:18.460 align:middle line:90% Here's what they sound like. 01:04:18.460 --> 01:04:19.260 align:middle line:90% [STATIC SOUND] 01:04:19.260 --> 01:04:19.760 align:middle line:90% All right. 01:04:19.760 --> 01:04:20.927 align:middle line:90% Let's listen to another one. 01:04:20.927 --> 01:04:22.060 align:middle line:90% [STATIC SOUND] 01:04:22.060 --> 01:04:22.560 align:middle line:90% All right. 01:04:22.560 --> 01:04:23.940 align:middle line:90% Let's do it again. 01:04:23.940 --> 01:04:25.380 align:middle line:90% [STATIC SOUND] 01:04:25.380 --> 01:04:27.900 align:middle line:84% So it's pretty clear that we would 01:04:27.900 --> 01:04:31.880 align:middle line:84% have to sit here generating these random sounds for a very, 01:04:31.880 --> 01:04:36.180 align:middle line:84% very, very long time before you got a doorbell or a hawk 01:04:36.180 --> 01:04:37.560 align:middle line:90% or the sound of toothbrushing. 01:04:37.560 --> 01:04:42.700 align:middle line:84% And so what that tells us is that the sounds that we hear 01:04:42.700 --> 01:04:48.180 align:middle line:84% in life are a tiny, tiny, tiny fraction 01:04:48.180 --> 01:04:50.980 align:middle line:84% of the space of all possible sounds. 01:04:50.980 --> 01:04:53.820 align:middle line:84% And that's really the key to us being 01:04:53.820 --> 01:04:57.260 align:middle line:84% able to solve auditory scene analysis, 01:04:57.260 --> 01:05:02.560 align:middle line:84% is that, when we describe the problem like this-- well, 01:05:02.560 --> 01:05:06.220 align:middle line:84% we're giving you this equation with two unknowns. 01:05:06.220 --> 01:05:12.980 align:middle line:84% The fact is that x and y can't take on every possible value. 01:05:12.980 --> 01:05:16.780 align:middle line:84% They live in this very, very specific subset of the space 01:05:16.780 --> 01:05:18.400 align:middle line:90% of all possible numbers. 01:05:18.400 --> 01:05:21.070 align:middle line:84% And so that's what makes the problem solvable. 01:05:21.070 --> 01:05:22.820 align:middle line:84% Real world sounds are a very small portion 01:05:22.820 --> 01:05:26.100 align:middle line:90% of all possible sounds. 01:05:26.100 --> 01:05:29.140 align:middle line:84% So we talked, at the very start of the class, 01:05:29.140 --> 01:05:31.740 align:middle line:84% of how we often think of perception 01:05:31.740 --> 01:05:34.680 align:middle line:84% as unconscious probabilistic inference. 01:05:34.680 --> 01:05:37.180 align:middle line:84% It's an inference process because the problems 01:05:37.180 --> 01:05:38.293 align:middle line:90% are ill-posed. 01:05:38.293 --> 01:05:39.960 align:middle line:84% The solution is not uniquely determined. 01:05:39.960 --> 01:05:42.460 align:middle line:84% And you have to make your best guess as to what 01:05:42.460 --> 01:05:45.000 align:middle line:84% happened in the world to cause a stimulus, 01:05:45.000 --> 01:05:46.883 align:middle line:84% given what you know about the world. 01:05:46.883 --> 01:05:48.300 align:middle line:84% So that's probabilistic inference, 01:05:48.300 --> 01:05:49.740 align:middle line:90% making your best guess. 01:05:49.740 --> 01:05:51.940 align:middle line:84% It's unconscious, because you're not actually 01:05:51.940 --> 01:05:53.460 align:middle line:90% aware of that happening. 01:05:53.460 --> 01:05:56.140 align:middle line:84% You just end up hearing something or seeing something. 01:05:56.140 --> 01:05:59.460 align:middle line:84% This is just happening under the Hood. 01:05:59.460 --> 01:06:01.140 align:middle line:84% And the general framework that we 01:06:01.140 --> 01:06:06.380 align:middle line:84% use to think about probabilistic inference is Bayes' rule. 01:06:06.380 --> 01:06:08.280 align:middle line:90% So here's the setting. 01:06:08.280 --> 01:06:10.860 align:middle line:84% When we're perceiving something, there 01:06:10.860 --> 01:06:14.600 align:middle line:84% is an observation, which is the sensory data that we receive. 01:06:14.600 --> 01:06:16.900 align:middle line:90% In this case, this is an image. 01:06:16.900 --> 01:06:18.460 align:middle line:90% It could be a sound. 01:06:18.460 --> 01:06:21.140 align:middle line:90% So that's the image. 01:06:21.140 --> 01:06:23.700 align:middle line:84% There is a set of possible hypotheses 01:06:23.700 --> 01:06:27.700 align:middle line:84% for what could have happened in the world to cause the image. 01:06:27.700 --> 01:06:30.900 align:middle line:84% So hypothesis A is that there are 01:06:30.900 --> 01:06:33.260 align:middle line:90% these two objects in the world. 01:06:33.260 --> 01:06:34.620 align:middle line:90% One is light green. 01:06:34.620 --> 01:06:36.957 align:middle line:84% And it's got that funny pattern of stripes. 01:06:36.957 --> 01:06:38.040 align:middle line:90% The other is darker green. 01:06:38.040 --> 01:06:42.100 align:middle line:84% And it's got that curved thing on it. 01:06:42.100 --> 01:06:45.900 align:middle line:84% Hypothesis B is that there are these two objects in the world. 01:06:45.900 --> 01:06:49.920 align:middle line:84% There's one surface that is like a two-tone surface. 01:06:49.920 --> 01:06:52.580 align:middle line:84% And then there's the letter B. Hypothesis 01:06:52.580 --> 01:06:54.680 align:middle line:84% C is that you have that same surface, 01:06:54.680 --> 01:06:57.100 align:middle line:84% but now there's a different letter there. 01:06:57.100 --> 01:07:02.300 align:middle line:90% So that's the observation. 01:07:02.300 --> 01:07:05.780 align:middle line:84% Now, when we are doing probabilistic inference 01:07:05.780 --> 01:07:09.260 align:middle line:84% in this setting, we are trying to find 01:07:09.260 --> 01:07:14.120 align:middle line:84% the hypothesis that is most likely given the observation. 01:07:14.120 --> 01:07:15.980 align:middle line:90% You get sensory data. 01:07:15.980 --> 01:07:21.660 align:middle line:84% And you want to figure out what possible state of the world 01:07:21.660 --> 01:07:24.740 align:middle line:84% is most likely to have caused that sensory data. 01:07:24.740 --> 01:07:28.780 align:middle line:84% And the hypotheses here are different possible states 01:07:28.780 --> 01:07:32.220 align:middle line:84% of the world that might have caused that sensory data. 01:07:32.220 --> 01:07:35.180 align:middle line:84% So how do we evaluate that probability? 01:07:35.180 --> 01:07:37.742 align:middle line:84% Well, this is an expression for that probability. 01:07:37.742 --> 01:07:39.700 align:middle line:84% That's what's called the posterior probability. 01:07:39.700 --> 01:07:41.980 align:middle line:84% It's the probability of the hypothesis, 01:07:41.980 --> 01:07:43.540 align:middle line:90% given the observation. 01:07:43.540 --> 01:07:46.740 align:middle line:90% That vertical line means given. 01:07:46.740 --> 01:07:50.560 align:middle line:84% And Bayes' rule states that the posterior probability, 01:07:50.560 --> 01:07:54.140 align:middle line:84% the probability of A, in this case, given O, 01:07:54.140 --> 01:07:58.680 align:middle line:84% is equal to the product of the probability of A on its own-- 01:07:58.680 --> 01:08:00.340 align:middle line:90% that's called the prior-- 01:08:00.340 --> 01:08:04.260 align:middle line:84% and the likelihood, the probability of the observation 01:08:04.260 --> 01:08:06.020 align:middle line:90% given the hypothesis. 01:08:06.020 --> 01:08:09.160 align:middle line:84% And then it's normalized by the probability of the observation. 01:08:09.160 --> 01:08:11.260 align:middle line:84% But typically, we're working in a setting 01:08:11.260 --> 01:08:13.200 align:middle line:90% where the observation is known. 01:08:13.200 --> 01:08:15.460 align:middle line:84% And so that's going to be the same across all possible 01:08:15.460 --> 01:08:17.540 align:middle line:90% hypotheses that we consider. 01:08:17.540 --> 01:08:20.580 align:middle line:84% So really, the thing that will cause the posterior 01:08:20.580 --> 01:08:25.859 align:middle line:84% to vary across hypotheses are the prior and the likelihood. 01:08:25.859 --> 01:08:29.260 align:middle line:84% So let's think through, in this simple setting, what 01:08:29.260 --> 01:08:30.800 align:middle line:90% that would correspond to. 01:08:30.800 --> 01:08:34.939 align:middle line:84% So the prior is a reflection of the fact 01:08:34.939 --> 01:08:39.420 align:middle line:84% that some hypotheses are a priori much more likely 01:08:39.420 --> 01:08:42.660 align:middle line:84% than others, just based on the way the world works. 01:08:42.660 --> 01:08:46.220 align:middle line:84% And so the idea in this simple example 01:08:46.220 --> 01:08:50.830 align:middle line:84% is that this hypothesis is a priori not very likely 01:08:50.830 --> 01:08:54.550 align:middle line:84% because objects like this tend not to occur in the world, 01:08:54.550 --> 01:08:57.590 align:middle line:84% whereas these two hypotheses have 01:08:57.590 --> 01:09:00.029 align:middle line:90% fairly high prior probability. 01:09:00.029 --> 01:09:02.050 align:middle line:84% This is just a reasonable looking surface. 01:09:02.050 --> 01:09:04.229 align:middle line:84% And then the world has lots of letters in it. 01:09:04.229 --> 01:09:07.830 align:middle line:84% So the idea in this contrived example 01:09:07.830 --> 01:09:13.590 align:middle line:84% is that the prior is high for these two and low for that one. 01:09:13.590 --> 01:09:18.430 align:middle line:84% Now, the likelihood is the probability of the observation 01:09:18.430 --> 01:09:20.229 align:middle line:90% given the hypothesis. 01:09:20.229 --> 01:09:24.029 align:middle line:84% That's going to tell us to what extent 01:09:24.029 --> 01:09:28.550 align:middle line:84% could a hypothesis actually have caused the observation. 01:09:28.550 --> 01:09:31.229 align:middle line:84% And so in this contrived example, 01:09:31.229 --> 01:09:37.310 align:middle line:84% the idea is that hypothesis A and B both do a very good job 01:09:37.310 --> 01:09:41.310 align:middle line:84% of explaining the observation, because if you put these two 01:09:41.310 --> 01:09:44.729 align:middle line:84% objects next to each other, or you put the B on top of this, 01:09:44.729 --> 01:09:49.390 align:middle line:84% you're going to reproduce pretty much exactly the observed image. 01:09:49.390 --> 01:09:54.350 align:middle line:84% So the likelihood, the probability of this observation, 01:09:54.350 --> 01:09:58.310 align:middle line:84% given this hypothesis or this hypothesis, is high. 01:09:58.310 --> 01:10:03.070 align:middle line:84% By comparison, this hypothesis produces fairly low likelihood 01:10:03.070 --> 01:10:04.910 align:middle line:84% because it's got the wrong letter. 01:10:04.910 --> 01:10:06.550 align:middle line:84% So the image that would be created 01:10:06.550 --> 01:10:09.070 align:middle line:84% by putting the C on top of this surface 01:10:09.070 --> 01:10:14.510 align:middle line:84% is not going to be very similar to that observed image. 01:10:14.510 --> 01:10:15.010 align:middle line:90% All right. 01:10:15.010 --> 01:10:16.950 align:middle line:84% So the idea is that we're trying to find 01:10:16.950 --> 01:10:21.530 align:middle line:84% the hypothesis that gives us a high posterior probability. 01:10:21.530 --> 01:10:23.870 align:middle line:84% And that's proportional to the product of the prior 01:10:23.870 --> 01:10:24.835 align:middle line:90% and the likelihood. 01:10:24.835 --> 01:10:26.210 align:middle line:84% So you want the prior to be high. 01:10:26.210 --> 01:10:28.070 align:middle line:84% And you also want the likelihood to be high. 01:10:28.070 --> 01:10:30.830 align:middle line:84% The prior being high tells you that the hypothesis 01:10:30.830 --> 01:10:32.052 align:middle line:90% is reasonable. 01:10:32.052 --> 01:10:33.510 align:middle line:84% The likelihood being high tells you 01:10:33.510 --> 01:10:36.590 align:middle line:84% that it can explain the sensory data. 01:10:36.590 --> 01:10:38.490 align:middle line:84% And so in this contrived example, 01:10:38.490 --> 01:10:41.910 align:middle line:84% the idea is that really only hypothesis B 01:10:41.910 --> 01:10:45.990 align:middle line:84% has both high prior probability and a high likelihood. 01:10:45.990 --> 01:10:48.430 align:middle line:84% Hypothesis A has a high likelihood, 01:10:48.430 --> 01:10:50.230 align:middle line:90% but low prior probability. 01:10:50.230 --> 01:10:56.410 align:middle line:84% Hypothesis C has high prior probability, but low likelihood. 01:10:56.410 --> 01:10:58.710 align:middle line:84% So this is a mathematical framework 01:10:58.710 --> 01:11:01.750 align:middle line:84% for thinking about probabilistic inference, key idea 01:11:01.750 --> 01:11:04.730 align:middle line:84% being that there are these two ingredients here. 01:11:04.730 --> 01:11:09.110 align:middle line:84% There's the prior-- how likely different states of the world 01:11:09.110 --> 01:11:11.230 align:middle line:84% are, given what we know about the world-- 01:11:11.230 --> 01:11:14.150 align:middle line:84% and the likelihood, which is, can the hypothesis explain 01:11:14.150 --> 01:11:15.690 align:middle line:90% the data that we observe? 01:11:15.690 --> 01:11:20.070 align:middle line:90% 01:11:20.070 --> 01:11:20.570 align:middle line:90% All right. 01:11:20.570 --> 01:11:24.350 align:middle line:84% So if we want to explain perception in this kind of way, 01:11:24.350 --> 01:11:27.110 align:middle line:84% as perceptual inference, there's a lot 01:11:27.110 --> 01:11:29.250 align:middle line:84% of things that actually have to be specified. 01:11:29.250 --> 01:11:31.550 align:middle line:84% So one of the things that you have to specify 01:11:31.550 --> 01:11:35.810 align:middle line:90% is the hypothesis space. 01:11:35.810 --> 01:11:39.390 align:middle line:84% So how are we actually going to define the states of the world 01:11:39.390 --> 01:11:43.910 align:middle line:84% that the perceptual system is actually trying to estimate? 01:11:43.910 --> 01:11:45.970 align:middle line:84% If we define the hypothesis space, 01:11:45.970 --> 01:11:49.270 align:middle line:84% then we also need to know the prior probability 01:11:49.270 --> 01:11:54.430 align:middle line:84% of all the different hypotheses that live in that space. 01:11:54.430 --> 01:11:56.270 align:middle line:84% We need a likelihood function, which 01:11:56.270 --> 01:12:01.150 align:middle line:84% tells us how to compare a hypothesis to the sensory input. 01:12:01.150 --> 01:12:04.390 align:middle line:84% If we've got the prior and the likelihood, 01:12:04.390 --> 01:12:06.310 align:middle line:90% that gives us the posterior. 01:12:06.310 --> 01:12:08.910 align:middle line:84% But the whole idea behind the perceptual inference 01:12:08.910 --> 01:12:11.710 align:middle line:84% is you want to find the hypothesis that gives you 01:12:11.710 --> 01:12:13.630 align:middle line:90% a high value for the posterior. 01:12:13.630 --> 01:12:16.390 align:middle line:84% And so that means you need some way to find 01:12:16.390 --> 01:12:20.110 align:middle line:84% the hypothesis that has, say, the highest posterior 01:12:20.110 --> 01:12:21.150 align:middle line:90% probability. 01:12:21.150 --> 01:12:24.230 align:middle line:84% And in practice, that actually is often highly non-trivial 01:12:24.230 --> 01:12:27.870 align:middle line:84% because the space of all the hypotheses may be enormous. 01:12:27.870 --> 01:12:32.550 align:middle line:84% And the posterior distribution may be multimodal. 01:12:32.550 --> 01:12:35.830 align:middle line:84% So finding the value that has the-- finding the hypothesis 01:12:35.830 --> 01:12:38.670 align:middle line:84% that has the highest posterior may be a really difficult search 01:12:38.670 --> 01:12:39.530 align:middle line:90% problem. 01:12:39.530 --> 01:12:43.310 align:middle line:84% And then ultimately, if you want to understand 01:12:43.310 --> 01:12:45.150 align:middle line:84% how all this happens in the brain, 01:12:45.150 --> 01:12:48.270 align:middle line:84% we need a description of the underlying representations 01:12:48.270 --> 01:12:52.470 align:middle line:84% and of the inference in terms of neurons. 01:12:52.470 --> 01:12:52.970 align:middle line:90% All right. 01:12:52.970 --> 01:12:54.190 align:middle line:84% So there's lots of different things 01:12:54.190 --> 01:12:56.482 align:middle line:84% that we actually really need to nail down to really say 01:12:56.482 --> 01:12:58.070 align:middle line:90% that we understand this. 01:12:58.070 --> 01:13:02.030 align:middle line:84% And these different ingredients that are here on this slide 01:13:02.030 --> 01:13:04.590 align:middle line:84% are related to this idea that there 01:13:04.590 --> 01:13:07.270 align:middle line:84% are different levels of analysis at which you 01:13:07.270 --> 01:13:11.710 align:middle line:84% can talk about an information processing problem. 01:13:11.710 --> 01:13:14.350 align:middle line:84% And the idea of there being levels of analysis was most 01:13:14.350 --> 01:13:17.130 align:middle line:84% famously articulated by somebody named David Marr, 01:13:17.130 --> 01:13:20.430 align:middle line:84% who was a vision researcher who worked here at MIT back 01:13:20.430 --> 01:13:22.150 align:middle line:90% in the '70s. 01:13:22.150 --> 01:13:24.790 align:middle line:84% He wrote a very influential book called 01:13:24.790 --> 01:13:28.390 align:middle line:84% Vision, which was his view for how to study vision. 01:13:28.390 --> 01:13:31.390 align:middle line:84% And the lasting contribution of that 01:13:31.390 --> 01:13:34.510 align:middle line:84% is the set of distinctions into levels of analysis 01:13:34.510 --> 01:13:37.250 align:middle line:84% that you still hear a lot about today. 01:13:37.250 --> 01:13:39.920 align:middle line:84% And you've probably heard about them in other classes. 01:13:39.920 --> 01:13:43.740 align:middle line:84% So Marr distinguished between three levels of analysis. 01:13:43.740 --> 01:13:48.800 align:middle line:84% The first and the highest level is the computational level. 01:13:48.800 --> 01:13:51.640 align:middle line:84% And that involves asking what the problem is 01:13:51.640 --> 01:13:54.020 align:middle line:84% that is being solved by the system 01:13:54.020 --> 01:13:56.080 align:middle line:84% and what the constraints are that allow 01:13:56.080 --> 01:13:58.120 align:middle line:90% that problem to be solved. 01:13:58.120 --> 01:14:01.480 align:middle line:84% So defining the problem to be solved 01:14:01.480 --> 01:14:04.740 align:middle line:84% is akin to identifying the hypothesis space. 01:14:04.740 --> 01:14:06.960 align:middle line:84% So what is the thing that you're trying to estimate 01:14:06.960 --> 01:14:09.040 align:middle line:90% from the sensory data? 01:14:09.040 --> 01:14:11.180 align:middle line:84% So it could be the fundamental frequency. 01:14:11.180 --> 01:14:14.000 align:middle line:84% It could be like where something is coming from in the world. 01:14:14.000 --> 01:14:17.840 align:middle line:84% It could be the shape of an object. 01:14:17.840 --> 01:14:21.520 align:middle line:84% And it also involves determining how you would assign 01:14:21.520 --> 01:14:24.360 align:middle line:90% probabilities to hypotheses-- 01:14:24.360 --> 01:14:27.320 align:middle line:84% so asserting that there's prior probability that 01:14:27.320 --> 01:14:30.360 align:middle line:84% varies over hypotheses, likelihood, and so forth. 01:14:30.360 --> 01:14:32.640 align:middle line:84% So you can also think of this computational level 01:14:32.640 --> 01:14:34.920 align:middle line:84% as nailing down the mapping between the input 01:14:34.920 --> 01:14:36.840 align:middle line:90% and the output. 01:14:36.840 --> 01:14:39.520 align:middle line:84% So a computational level description of a problem 01:14:39.520 --> 01:14:40.692 align:middle line:90% defines that mapping. 01:14:40.692 --> 01:14:42.400 align:middle line:84% So it says, what are the assumptions that 01:14:42.400 --> 01:14:45.300 align:middle line:84% are being made by the perceptual system to solve this problem? 01:14:45.300 --> 01:14:48.080 align:middle line:84% And those assumptions determine the mapping 01:14:48.080 --> 01:14:50.800 align:middle line:84% between the input, say the sound or the image, 01:14:50.800 --> 01:14:52.520 align:middle line:84% and the output, which is your estimate 01:14:52.520 --> 01:14:56.000 align:middle line:84% of the property of the world that you're interested in. 01:14:56.000 --> 01:14:58.200 align:middle line:84% Now, then there's also the algorithmic level, 01:14:58.200 --> 01:15:00.800 align:middle line:84% which involves identifying the approach 01:15:00.800 --> 01:15:02.360 align:middle line:90% to finding the solution. 01:15:02.360 --> 01:15:03.900 align:middle line:84% And so one way to think about this, 01:15:03.900 --> 01:15:05.980 align:middle line:84% if we're thinking about probabilistic inference, 01:15:05.980 --> 01:15:08.840 align:middle line:84% is in describing an optimization procedure 01:15:08.840 --> 01:15:12.040 align:middle line:84% to find the best hypothesis, or the most probable hypothesis, 01:15:12.040 --> 01:15:13.460 align:middle line:90% given the sensory input. 01:15:13.460 --> 01:15:15.960 align:middle line:84% And as I said, that often involves very hard search 01:15:15.960 --> 01:15:18.660 align:middle line:84% problems because there's a huge hypothesis space. 01:15:18.660 --> 01:15:21.108 align:middle line:84% It's not obvious which one is the best. 01:15:21.108 --> 01:15:23.400 align:middle line:84% And then there's the level of the implementation, which 01:15:23.400 --> 01:15:25.640 align:middle line:84% is how you would instantiate a solution 01:15:25.640 --> 01:15:27.460 align:middle line:90% to a problem in hardware. 01:15:27.460 --> 01:15:29.800 align:middle line:84% And so this, in the context of neuroscience, 01:15:29.800 --> 01:15:34.160 align:middle line:84% would involve a description of neural circuitry. 01:15:34.160 --> 01:15:40.840 align:middle line:84% So this is an excerpt of a figure from Marr's book, where-- 01:15:40.840 --> 01:15:44.160 align:middle line:84% these are, in his words, those three levels. 01:15:44.160 --> 01:15:46.920 align:middle line:84% So there's the computational theory, the representation 01:15:46.920 --> 01:15:52.080 align:middle line:84% and algorithm, and then hardware implementation. 01:15:52.080 --> 01:15:54.380 align:middle line:84% Any questions about levels of analysis? 01:15:54.380 --> 01:15:59.080 align:middle line:90% 01:15:59.080 --> 01:16:03.320 align:middle line:84% And I should say, we will return to these ideas 01:16:03.320 --> 01:16:05.420 align:middle line:84% repeatedly over the course of the class. 01:16:05.420 --> 01:16:07.440 align:middle line:84% And you'll think about them as being 01:16:07.440 --> 01:16:10.100 align:middle line:84% applied to lots of different aspects of perception. 01:16:10.100 --> 01:16:19.400 align:middle line:90% 01:16:19.400 --> 01:16:19.940 align:middle line:90% All right. 01:16:19.940 --> 01:16:23.640 align:middle line:84% So we're going to talk about auditory scene analysis, which 01:16:23.640 --> 01:16:26.640 align:middle line:84% you can think of as this problem of inferring 01:16:26.640 --> 01:16:29.560 align:middle line:84% sources in the world from a sound signal that 01:16:29.560 --> 01:16:32.120 align:middle line:90% reaches your ears. 01:16:32.120 --> 01:16:34.840 align:middle line:84% So as we've discussed, if we want 01:16:34.840 --> 01:16:37.300 align:middle line:84% to think about this as perceptual inference, 01:16:37.300 --> 01:16:39.220 align:middle line:84% we need to specify the hypothesis space, 01:16:39.220 --> 01:16:40.760 align:middle line:84% the prior probability of hypotheses, 01:16:40.760 --> 01:16:43.920 align:middle line:90% a whole bunch of stuff. 01:16:43.920 --> 01:16:45.960 align:middle line:84% And as a starting point, let's think 01:16:45.960 --> 01:16:47.540 align:middle line:90% about this about these priors. 01:16:47.540 --> 01:16:51.080 align:middle line:84% So priors reflect the regularities 01:16:51.080 --> 01:16:52.260 align:middle line:90% of real world sounds. 01:16:52.260 --> 01:16:53.840 align:middle line:84% So we've just seen this demonstration 01:16:53.840 --> 01:16:56.620 align:middle line:84% that the sounds that we hear in the real world are not random. 01:16:56.620 --> 01:17:01.400 align:middle line:84% So some sounds are a lot more likely than others. 01:17:01.400 --> 01:17:03.940 align:middle line:84% So the doorbell is the kind of thing that can occur in life. 01:17:03.940 --> 01:17:06.280 align:middle line:84% There's other things that wouldn't. 01:17:06.280 --> 01:17:08.563 align:middle line:84% And these regularities that exist in real world 01:17:08.563 --> 01:17:10.980 align:middle line:84% sounds, they derive from the processes that generate them. 01:17:10.980 --> 01:17:14.040 align:middle line:84% So sounds, they're generated by lawful physical processes, 01:17:14.040 --> 01:17:14.980 align:middle line:90% for the most part. 01:17:14.980 --> 01:17:17.580 align:middle line:84% And those are governed by physical laws. 01:17:17.580 --> 01:17:20.620 align:middle line:84% And so you can't get any old thing. 01:17:20.620 --> 01:17:24.400 align:middle line:84% There's certain things that can result from that. 01:17:24.400 --> 01:17:27.720 align:middle line:84% And so as with other areas of perception, 01:17:27.720 --> 01:17:29.740 align:middle line:90% illusions can tell us a lot. 01:17:29.740 --> 01:17:32.240 align:middle line:84% So illusions can reveal the constraints 01:17:32.240 --> 01:17:35.080 align:middle line:84% on sound generation that have been internalized 01:17:35.080 --> 01:17:37.840 align:middle line:84% by our brains, that make auditory scene 01:17:37.840 --> 01:17:40.560 align:middle line:84% analysis possible via priors that we 01:17:40.560 --> 01:17:44.740 align:middle line:84% have on what kinds of sounds are likely and what are not. 01:17:44.740 --> 01:17:47.520 align:middle line:90% 01:17:47.520 --> 01:17:50.080 align:middle line:84% And so this is often talked about in the language 01:17:50.080 --> 01:17:50.788 align:middle line:90% of grouping cues. 01:17:50.788 --> 01:17:52.622 align:middle line:84% So we talked about cues when we were talking 01:17:52.622 --> 01:17:53.780 align:middle line:90% about sound localization. 01:17:53.780 --> 01:17:57.320 align:middle line:84% So the notion of a cue is a stimulus property 01:17:57.320 --> 01:17:59.673 align:middle line:84% that's informative about something in the world. 01:17:59.673 --> 01:18:01.840 align:middle line:84% So we talked about how there are binaural cues, like 01:18:01.840 --> 01:18:04.145 align:middle line:84% interaural time differences and level differences, 01:18:04.145 --> 01:18:05.520 align:middle line:84% and spectral cues from your pinna 01:18:05.520 --> 01:18:08.640 align:middle line:90% that tell you about elevation. 01:18:08.640 --> 01:18:12.160 align:middle line:84% Similarly, or analogously, grouping cues 01:18:12.160 --> 01:18:14.040 align:middle line:84% are thought of as stimulus properties 01:18:14.040 --> 01:18:16.800 align:middle line:84% that are informative about the organization of sound 01:18:16.800 --> 01:18:18.180 align:middle line:90% into sources. 01:18:18.180 --> 01:18:21.555 align:middle line:90% 01:18:21.555 --> 01:18:24.160 align:middle line:84% We believe that these cues derive 01:18:24.160 --> 01:18:27.320 align:middle line:84% from statistical properties of natural sounds. 01:18:27.320 --> 01:18:29.250 align:middle line:84% And we'll talk about a number of examples. 01:18:29.250 --> 01:18:34.930 align:middle line:84% And really, two obvious examples are shown here-- 01:18:34.930 --> 01:18:36.590 align:middle line:84% onset and offset and harmonicity. 01:18:36.590 --> 01:18:39.570 align:middle line:84% And this is just a cochleagram, a cochlear representation 01:18:39.570 --> 01:18:41.050 align:middle line:90% of a speech signal. 01:18:41.050 --> 01:18:45.490 align:middle line:84% And you can see that you get these patterns here, 01:18:45.490 --> 01:18:48.130 align:middle line:84% where the energy begins and ends, 01:18:48.130 --> 01:18:49.870 align:middle line:84% across a wide range of frequencies, 01:18:49.870 --> 01:18:52.010 align:middle line:90% more or less at the same time. 01:18:52.010 --> 01:18:54.582 align:middle line:84% So there's a common onset or a common offset. 01:18:54.582 --> 01:18:56.790 align:middle line:84% And that's because people open and close their mouth. 01:18:56.790 --> 01:19:00.610 align:middle line:84% And so you get a termination of energy or a burst of energy. 01:19:00.610 --> 01:19:02.330 align:middle line:84% And that common onset and offset is 01:19:02.330 --> 01:19:05.430 align:middle line:84% something that happens a lot in natural sounds. 01:19:05.430 --> 01:19:07.610 align:middle line:84% And the idea is that, when that is observed 01:19:07.610 --> 01:19:09.130 align:middle line:84% in the sound signal, that gives you 01:19:09.130 --> 01:19:11.890 align:middle line:84% a clue that all of those frequencies that 01:19:11.890 --> 01:19:16.370 align:middle line:84% have the same onset are caused by the same thing in the world. 01:19:16.370 --> 01:19:19.330 align:middle line:84% Similarly, we just talked about how lots of sounds in the world 01:19:19.330 --> 01:19:20.810 align:middle line:84% are harmonic, in particular sounds 01:19:20.810 --> 01:19:23.010 align:middle line:90% that are produced by our voice. 01:19:23.010 --> 01:19:26.450 align:middle line:84% And that shows up as these horizontal stripes 01:19:26.450 --> 01:19:28.590 align:middle line:90% here for individual harmonics. 01:19:28.590 --> 01:19:32.530 align:middle line:84% And so the idea is that if you observe harmonic frequencies 01:19:32.530 --> 01:19:35.690 align:middle line:84% in the input to the ear, that's a pretty strong cue 01:19:35.690 --> 01:19:38.450 align:middle line:84% that those frequencies actually came from the same source 01:19:38.450 --> 01:19:42.650 align:middle line:84% in the world, because sources in the world tend to be harmonic. 01:19:42.650 --> 01:19:46.130 align:middle line:84% And so what we're going to get into next time 01:19:46.130 --> 01:19:49.850 align:middle line:84% is a bunch of really remarkable demonstrations 01:19:49.850 --> 01:19:52.690 align:middle line:84% that give us evidence that your brain has internalized 01:19:52.690 --> 01:19:54.890 align:middle line:84% a whole bunch of these properties of sound. 01:19:54.890 --> 01:19:58.650 align:middle line:84% And it causes you to organize sound into distinct sources. 01:19:58.650 --> 01:20:03.290 align:middle line:84% These are illusions of perceptual organization. 01:20:03.290 --> 01:20:04.855 align:middle line:90% So we'll end there. 01:20:04.855 --> 01:20:05.730 align:middle line:90% Have a great weekend. 01:20:05.730 --> 01:20:06.870 align:middle line:90% Recitations tomorrow. 01:20:06.870 --> 01:20:09.730 align:middle line:90% Problem set is due tomorrow. 01:20:09.730 --> 01:20:11.800 align:middle line:90% I'll see you on Tuesday. 01:20:11.800 --> 01:20:26.000 align:middle line:90%