WEBVTT 00:00:00.000 --> 00:00:02.440 align:middle line:90% 00:00:02.440 --> 00:00:09.025 align:middle line:90% [SQUEAKING] 00:00:09.025 --> 00:00:11.400 align:middle line:84% JOSH MCDERMOTT: The problem that we've been talking about 00:00:11.400 --> 00:00:16.160 align:middle line:84% is the problem that there's this world out there 00:00:16.160 --> 00:00:19.340 align:middle line:84% with multiple sources, typically, that will make sound. 00:00:19.340 --> 00:00:24.040 align:middle line:84% And you receive a single sound waveform that enters your ears. 00:00:24.040 --> 00:00:26.400 align:middle line:84% You're trying to infer what happened in the world 00:00:26.400 --> 00:00:28.140 align:middle line:90% to cause the sound. 00:00:28.140 --> 00:00:32.159 align:middle line:84% So that's a specific example of the general problem 00:00:32.159 --> 00:00:33.820 align:middle line:90% of perceptual inference. 00:00:33.820 --> 00:00:38.360 align:middle line:84% And so we've talked about how the typical formalization 00:00:38.360 --> 00:00:42.920 align:middle line:84% that we use to think about perceptual inference problems 00:00:42.920 --> 00:00:46.880 align:middle line:84% is given by Bayes' rule, where the idea is that we want 00:00:46.880 --> 00:00:51.920 align:middle line:84% to find the hypothesis, which is the potential state of the world 00:00:51.920 --> 00:00:54.060 align:middle line:84% that is most probable given the observation. 00:00:54.060 --> 00:00:56.040 align:middle line:84% So we're trying to find something that 00:00:56.040 --> 00:00:57.920 align:middle line:90% has high posterior probability. 00:00:57.920 --> 00:01:02.170 align:middle line:84% You can break that down in terms of the prior and the likelihood. 00:01:02.170 --> 00:01:04.177 align:middle line:90% And so you look at Bayes' rule. 00:01:04.177 --> 00:01:06.010 align:middle line:84% And it kind of makes it seem like everything 00:01:06.010 --> 00:01:08.450 align:middle line:90% is kind of nice and simple. 00:01:08.450 --> 00:01:11.890 align:middle line:84% But in actuality, if you think deeply about what's 00:01:11.890 --> 00:01:15.112 align:middle line:84% required in order to really have a thorough understanding of one 00:01:15.112 --> 00:01:16.570 align:middle line:84% of these problems, there's a number 00:01:16.570 --> 00:01:18.262 align:middle line:84% of things that require specifying, 00:01:18.262 --> 00:01:19.470 align:middle line:90% which we've discussed before. 00:01:19.470 --> 00:01:22.590 align:middle line:84% So we've got to say what the hypothesis space is. 00:01:22.590 --> 00:01:26.810 align:middle line:84% So what's the candidate set of states of the world 00:01:26.810 --> 00:01:29.250 align:middle line:84% that we're going to try to differentiate between? 00:01:29.250 --> 00:01:31.010 align:middle line:84% And that's often kind of non-obvious, 00:01:31.010 --> 00:01:32.850 align:middle line:84% or at least there's lots of different ways 00:01:32.850 --> 00:01:35.610 align:middle line:90% to specify the hypothesis space. 00:01:35.610 --> 00:01:39.010 align:middle line:84% So for instance, you could think of the hypothesis space 00:01:39.010 --> 00:01:42.130 align:middle line:84% as the actual waveforms that are produced by the sources. 00:01:42.130 --> 00:01:45.690 align:middle line:84% You could think of it as like parameters of those sources, 00:01:45.690 --> 00:01:48.622 align:middle line:84% for instance, like the pitch of the notes in some way 00:01:48.622 --> 00:01:49.830 align:middle line:90% of characterizing the timbre. 00:01:49.830 --> 00:01:52.410 align:middle line:84% There's lots of different ways to think about what exactly 00:01:52.410 --> 00:01:53.370 align:middle line:90% that is. 00:01:53.370 --> 00:01:55.370 align:middle line:84% Then you have to establish the prior probability 00:01:55.370 --> 00:01:57.900 align:middle line:90% of the different hypotheses. 00:01:57.900 --> 00:01:59.500 align:middle line:84% You need a likelihood function, which 00:01:59.500 --> 00:02:04.100 align:middle line:84% involves a comparison of a given hypothesis to the sensory input. 00:02:04.100 --> 00:02:06.780 align:middle line:84% You need a means to find the hypothesis with the highest 00:02:06.780 --> 00:02:08.580 align:middle line:90% posterior probability. 00:02:08.580 --> 00:02:11.980 align:middle line:84% So just because you know how to calculate the posterior for one 00:02:11.980 --> 00:02:14.660 align:middle line:84% example hypothesis doesn't give you necessarily a way 00:02:14.660 --> 00:02:17.260 align:middle line:90% to find the one that is best. 00:02:17.260 --> 00:02:19.980 align:middle line:84% And then we ultimately want to describe all this stuff 00:02:19.980 --> 00:02:22.820 align:middle line:90% in terms of neurons. 00:02:22.820 --> 00:02:23.320 align:middle line:90% OK. 00:02:23.320 --> 00:02:25.020 align:middle line:84% So where does all of the things we've been talking about 00:02:25.020 --> 00:02:25.520 align:middle line:90% leave us? 00:02:25.520 --> 00:02:29.460 align:middle line:84% So one of the basic questions that you 00:02:29.460 --> 00:02:31.660 align:middle line:84% could ask about auditory scene analysis is, 00:02:31.660 --> 00:02:33.740 align:middle line:84% what is the hypothesis space, and what's 00:02:33.740 --> 00:02:36.160 align:middle line:84% the nature of the prior over those hypotheses? 00:02:36.160 --> 00:02:42.380 align:middle line:84% And this involves implicitly making assumptions about sources 00:02:42.380 --> 00:02:44.180 align:middle line:90% and how they produce sounds. 00:02:44.180 --> 00:02:48.420 align:middle line:84% And so we often think of the problem of perceptual inference 00:02:48.420 --> 00:02:51.180 align:middle line:84% as there being like a model of the world that 00:02:51.180 --> 00:02:53.240 align:middle line:90% defines a space of hypotheses. 00:02:53.240 --> 00:02:55.630 align:middle line:84% So in the case of auditory scene analysis, 00:02:55.630 --> 00:02:58.270 align:middle line:84% the world model would consist of sources in the world that 00:02:58.270 --> 00:03:00.110 align:middle line:90% can make sounds. 00:03:00.110 --> 00:03:02.650 align:middle line:84% They generate sounds, and they produce an observation, 00:03:02.650 --> 00:03:04.030 align:middle line:84% which is like a mixture of all the sounds that 00:03:04.030 --> 00:03:05.410 align:middle line:90% would be produced on their own. 00:03:05.410 --> 00:03:08.470 align:middle line:84% OK, so Bayes' rule gives us a formalization 00:03:08.470 --> 00:03:10.850 align:middle line:84% for using a world model to do inference. 00:03:10.850 --> 00:03:13.350 align:middle line:90% And it looks really simple. 00:03:13.350 --> 00:03:15.230 align:middle line:84% You've got probability of the world state 00:03:15.230 --> 00:03:18.350 align:middle line:84% given the data being proportional to the likelihood 00:03:18.350 --> 00:03:19.790 align:middle line:90% times the prior. 00:03:19.790 --> 00:03:26.870 align:middle line:84% But there's a lot of other stuff that has to go into this. 00:03:26.870 --> 00:03:29.510 align:middle line:84% Now so what's the contribution of all these illusions 00:03:29.510 --> 00:03:31.670 align:middle line:90% that we've been discussing? 00:03:31.670 --> 00:03:33.550 align:middle line:84% So I think one way to look at this 00:03:33.550 --> 00:03:36.670 align:middle line:84% is that these illusions suggest constraints 00:03:36.670 --> 00:03:39.490 align:middle line:84% on the brain's hypothesis space for sound sources. 00:03:39.490 --> 00:03:42.030 align:middle line:84% So we've been looking at all of these examples that 00:03:42.030 --> 00:03:46.390 align:middle line:84% seem to suggest that our auditory system is assuming 00:03:46.390 --> 00:03:48.950 align:middle line:84% the tendency of sources to make sounds that 00:03:48.950 --> 00:03:54.120 align:middle line:84% have abrupt onsets, that contain harmonic frequencies, that 00:03:54.120 --> 00:03:56.680 align:middle line:84% contain repeating elements, and that 00:03:56.680 --> 00:03:59.295 align:middle line:84% exhibits similar acoustic properties over time. 00:03:59.295 --> 00:04:00.920 align:middle line:84% Because all of those illusions, they're 00:04:00.920 --> 00:04:04.400 align:middle line:84% essentially sounds that when you listen to them 00:04:04.400 --> 00:04:07.600 align:middle line:84% get organized in your head in a certain way. 00:04:07.600 --> 00:04:10.293 align:middle line:84% So for instance, we saw the harmonica's tuning effect. 00:04:10.293 --> 00:04:11.960 align:middle line:84% So you take a frequency, and you tune it 00:04:11.960 --> 00:04:14.700 align:middle line:84% and that as part of a harmonic complex. 00:04:14.700 --> 00:04:16.440 align:middle line:84% And then you hear it as a separate sound. 00:04:16.440 --> 00:04:19.000 align:middle line:84% So that's indicating that there's this assumption that's 00:04:19.000 --> 00:04:22.160 align:middle line:84% implicit in the way your auditory system works 00:04:22.160 --> 00:04:25.940 align:middle line:84% that sources in the world make harmonic frequencies. 00:04:25.940 --> 00:04:28.280 align:middle line:84% So that's one of the contributions 00:04:28.280 --> 00:04:30.920 align:middle line:84% of a lot of these illusions is revealing 00:04:30.920 --> 00:04:34.620 align:middle line:84% these constraints on sound sources 00:04:34.620 --> 00:04:37.740 align:middle line:84% and thus on the brain's hypothesis space. 00:04:37.740 --> 00:04:43.960 align:middle line:84% And now, of course, these constraints, at some level, 00:04:43.960 --> 00:04:46.200 align:middle line:90% they come from the world. 00:04:46.200 --> 00:04:48.840 align:middle line:84% So we believe that these perceptual effects 00:04:48.840 --> 00:04:51.440 align:middle line:84% that relate to harmonic frequencies are abrupt onsets. 00:04:51.440 --> 00:04:54.800 align:middle line:84% They're there due to the fact that the sources in the world 00:04:54.800 --> 00:04:57.320 align:middle line:84% behave in a certain way due to physical laws. 00:04:57.320 --> 00:04:59.160 align:middle line:84% So there's sound sources in the world that 00:04:59.160 --> 00:05:03.480 align:middle line:84% are periodic in time that are harmonic in frequency 00:05:03.480 --> 00:05:05.840 align:middle line:84% or that tend to produce sounds in a way that 00:05:05.840 --> 00:05:07.980 align:middle line:84% creates this common onset across frequencies. 00:05:07.980 --> 00:05:11.160 align:middle line:84% So at some level, this stuff originates in the world. 00:05:11.160 --> 00:05:13.160 align:middle line:84% But then the illusions kind of both 00:05:13.160 --> 00:05:15.818 align:middle line:84% reveal those as being important things in the world 00:05:15.818 --> 00:05:17.360 align:middle line:84% but also things that are internalized 00:05:17.360 --> 00:05:18.800 align:middle line:90% by your auditory system. 00:05:18.800 --> 00:05:21.440 align:middle line:90% 00:05:21.440 --> 00:05:23.280 align:middle line:84% We also looked at examples that speak 00:05:23.280 --> 00:05:25.360 align:middle line:84% to the tendency for environmental surfaces 00:05:25.360 --> 00:05:26.380 align:middle line:90% to produce reflections. 00:05:26.380 --> 00:05:29.000 align:middle line:84% Again, that's a statement about acoustics. 00:05:29.000 --> 00:05:30.960 align:middle line:84% But the illusion that is associated 00:05:30.960 --> 00:05:32.560 align:middle line:84% with that, the precedence effect, 00:05:32.560 --> 00:05:35.400 align:middle line:84% is demonstrating that that's been internalized 00:05:35.400 --> 00:05:38.720 align:middle line:90% by the auditory system. 00:05:38.720 --> 00:05:41.280 align:middle line:84% The illusions also kind of reveal 00:05:41.280 --> 00:05:44.040 align:middle line:84% that the auditory system is implicitly 00:05:44.040 --> 00:05:45.960 align:middle line:84% cognizant of the possibility that sounds 00:05:45.960 --> 00:05:49.440 align:middle line:84% can occur concurrently and that they can continue 00:05:49.440 --> 00:05:51.610 align:middle line:84% in the absence of overt evidence, all 00:05:51.610 --> 00:05:55.910 align:middle line:84% of these filling in illusions that we saw. 00:05:55.910 --> 00:05:59.250 align:middle line:84% So, as we said, Bayes' rule gives us 00:05:59.250 --> 00:06:02.970 align:middle line:84% a formalization for using a model of the world 00:06:02.970 --> 00:06:06.432 align:middle line:90% to do inference. 00:06:06.432 --> 00:06:08.390 align:middle line:84% But it doesn't tell us how to do the inference. 00:06:08.390 --> 00:06:10.090 align:middle line:84% And this actually in practice turns out 00:06:10.090 --> 00:06:16.290 align:middle line:84% to be like often one of the hardest parts of making 00:06:16.290 --> 00:06:18.330 align:middle line:90% models of perception work. 00:06:18.330 --> 00:06:21.130 align:middle line:84% And so what I mean by telling us how to do the inference 00:06:21.130 --> 00:06:25.130 align:middle line:84% is determining how to find the most probable hypothesis 00:06:25.130 --> 00:06:27.930 align:middle line:90% for a given observation. 00:06:27.930 --> 00:06:35.370 align:middle line:84% And the reason that this is in practice often pretty hard 00:06:35.370 --> 00:06:38.690 align:middle line:90% is indicated by this picture. 00:06:38.690 --> 00:06:42.570 align:middle line:84% And this is just like-- it's a very simplistic attempt 00:06:42.570 --> 00:06:44.510 align:middle line:90% to convey some of the issues. 00:06:44.510 --> 00:06:49.140 align:middle line:84% So this picture is plotting a hypothetical posterior 00:06:49.140 --> 00:06:52.260 align:middle line:84% distribution over a hypothesis space 00:06:52.260 --> 00:06:54.100 align:middle line:90% that has only two parameters. 00:06:54.100 --> 00:06:56.700 align:middle line:84% So let's suppose that the hypothesis space that you're 00:06:56.700 --> 00:07:00.245 align:middle line:84% considering would be determined by these two parameters, 00:07:00.245 --> 00:07:02.120 align:middle line:84% say the frequency and the duration of a tone. 00:07:02.120 --> 00:07:04.100 align:middle line:84% So that would be like an incredibly simple hypothesis 00:07:04.100 --> 00:07:04.760 align:middle line:90% space. 00:07:04.760 --> 00:07:08.920 align:middle line:84% But as you can see in this simple example, 00:07:08.920 --> 00:07:12.820 align:middle line:84% the posterior probability has these multiple peaks 00:07:12.820 --> 00:07:13.940 align:middle line:90% throughout this space. 00:07:13.940 --> 00:07:16.860 align:middle line:90% 00:07:16.860 --> 00:07:19.220 align:middle line:84% Now in actual problems, there are 00:07:19.220 --> 00:07:20.590 align:middle line:90% way more than two parameters. 00:07:20.590 --> 00:07:22.340 align:middle line:84% If you think of all of the parameters that 00:07:22.340 --> 00:07:25.140 align:middle line:84% would be necessary to specify even 00:07:25.140 --> 00:07:26.880 align:middle line:84% a moderately complicated auditory scene, 00:07:26.880 --> 00:07:28.020 align:middle line:90% it's quite a lot. 00:07:28.020 --> 00:07:30.660 align:middle line:84% So we've got this probability distribution 00:07:30.660 --> 00:07:33.140 align:middle line:84% defined over a pretty high-dimensional space. 00:07:33.140 --> 00:07:36.060 align:middle line:90% It's typically multimodal. 00:07:36.060 --> 00:07:38.980 align:middle line:84% And so like in this particular example, 00:07:38.980 --> 00:07:41.360 align:middle line:84% we've plotted the full distribution. 00:07:41.360 --> 00:07:43.920 align:middle line:84% And so you can look at it with your visual system and say, 00:07:43.920 --> 00:07:46.830 align:middle line:90% oh yeah, this is the best point. 00:07:46.830 --> 00:07:50.270 align:middle line:84% But normally, you can't plot it, and you can't look at it 00:07:50.270 --> 00:07:52.070 align:middle line:90% with your visual system. 00:07:52.070 --> 00:07:54.150 align:middle line:84% And so you have to have some other means 00:07:54.150 --> 00:07:57.770 align:middle line:84% of finding the hypothesis that is the best. 00:07:57.770 --> 00:08:01.430 align:middle line:84% Now the other added complexity is that in a lot of cases-- 00:08:01.430 --> 00:08:03.430 align:middle line:84% this picture is actually a little bit misleading 00:08:03.430 --> 00:08:06.510 align:middle line:84% because the number of parameters that defines your hypothesis 00:08:06.510 --> 00:08:07.650 align:middle line:90% space is not even fixed. 00:08:07.650 --> 00:08:10.590 align:middle line:84% So let's suppose, for instance, in the domain of auditory scene 00:08:10.590 --> 00:08:14.085 align:middle line:84% analysis, you often don't know how many sources 00:08:14.085 --> 00:08:15.210 align:middle line:90% are out there in the world. 00:08:15.210 --> 00:08:17.410 align:middle line:84% And so one of the questions is, well, is there two sources? 00:08:17.410 --> 00:08:18.077 align:middle line:90% Are there three? 00:08:18.077 --> 00:08:18.950 align:middle line:90% Are there four? 00:08:18.950 --> 00:08:21.350 align:middle line:84% And for those different numbers of sources, 00:08:21.350 --> 00:08:24.350 align:middle line:84% that would create different numbers of parameters. 00:08:24.350 --> 00:08:26.470 align:middle line:84% So you also have to be able to consider 00:08:26.470 --> 00:08:30.310 align:middle line:90% those different possibilities. 00:08:30.310 --> 00:08:33.950 align:middle line:84% But the crux of the issue is that you 00:08:33.950 --> 00:08:37.750 align:middle line:84% can think of perceptual inference as a search problem. 00:08:37.750 --> 00:08:40.890 align:middle line:84% So we've got this huge space of possible hypotheses. 00:08:40.890 --> 00:08:42.630 align:middle line:90% You have an observation. 00:08:42.630 --> 00:08:45.880 align:middle line:84% And you somehow have to search through that space of hypotheses 00:08:45.880 --> 00:08:50.080 align:middle line:84% to find one that has good posterior probability given 00:08:50.080 --> 00:08:52.760 align:middle line:90% the observation. 00:08:52.760 --> 00:08:54.560 align:middle line:84% And these are hard search problems 00:08:54.560 --> 00:08:57.320 align:middle line:84% because the optimization landscape is complicated, 00:08:57.320 --> 00:09:00.280 align:middle line:90% and it's very high dimensional. 00:09:00.280 --> 00:09:02.560 align:middle line:84% So one kind of algorithmic approach, 00:09:02.560 --> 00:09:05.280 align:middle line:84% which you can think of as mapping onto some of the things 00:09:05.280 --> 00:09:08.440 align:middle line:84% that we know about sensory systems, 00:09:08.440 --> 00:09:14.400 align:middle line:84% is to use a feed-forward algorithm 00:09:14.400 --> 00:09:17.800 align:middle line:84% to take clues that are in the sensory input 00:09:17.800 --> 00:09:20.780 align:middle line:84% and get you to the right part of the hypothesis space. 00:09:20.780 --> 00:09:23.800 align:middle line:84% And this is actually like a pretty common 00:09:23.800 --> 00:09:29.020 align:middle line:84% computational technique that is used in this neck of the woods. 00:09:29.020 --> 00:09:30.800 align:middle line:84% So the idea is that you could take 00:09:30.800 --> 00:09:34.320 align:middle line:84% a feed-forward neural network that takes the sensory input 00:09:34.320 --> 00:09:38.520 align:middle line:84% and make some initial guesses as to what the parameters are. 00:09:38.520 --> 00:09:43.370 align:middle line:84% Now in practice, those guesses are not perfect. 00:09:43.370 --> 00:09:45.690 align:middle line:84% But they are often useful to get you 00:09:45.690 --> 00:09:47.690 align:middle line:90% to the right part of the space. 00:09:47.690 --> 00:09:50.490 align:middle line:84% And then you could do some kind of optimization procedure 00:09:50.490 --> 00:09:55.930 align:middle line:84% to try to find your way to the peak of the local distribution. 00:09:55.930 --> 00:09:59.050 align:middle line:84% But that's the key thing to take away 00:09:59.050 --> 00:10:02.170 align:middle line:84% from this is that when we think about perception as Bayesian 00:10:02.170 --> 00:10:07.970 align:middle line:84% inference, instantiating that for the real problems 00:10:07.970 --> 00:10:11.250 align:middle line:84% that we solve is pretty difficult. 00:10:11.250 --> 00:10:14.530 align:middle line:84% And doing that right now is like on the bleeding edge of what is 00:10:14.530 --> 00:10:17.170 align:middle line:90% kind of possible in our field. 00:10:17.170 --> 00:10:19.810 align:middle line:84% And the crux of the issue is that you 00:10:19.810 --> 00:10:22.470 align:middle line:84% have to solve these very challenging search problems. 00:10:22.470 --> 00:10:27.210 align:middle line:84% So initially, you can try to define a hypothesis space, 00:10:27.210 --> 00:10:28.850 align:middle line:84% but then you still have this problem 00:10:28.850 --> 00:10:34.770 align:middle line:84% of finding which one is best given an observed sensory input. 00:10:34.770 --> 00:10:39.330 align:middle line:84% Any questions about that way of thinking about things? 00:10:39.330 --> 00:10:40.690 align:middle line:90% Yeah, Christa. 00:10:40.690 --> 00:10:43.460 align:middle line:84% STUDENT: Did you say it's accurate to describe illusions 00:10:43.460 --> 00:10:46.320 align:middle line:84% as like finding a local but not local optimum? 00:10:46.320 --> 00:10:50.240 align:middle line:84% Or should you say that the analogy kind of breaks down? 00:10:50.240 --> 00:10:51.800 align:middle line:90% JOSH MCDERMOTT: Well, OK. 00:10:51.800 --> 00:10:56.460 align:middle line:90% So illusions are interesting. 00:10:56.460 --> 00:10:59.420 align:middle line:84% They relate to this kind of picture in an interesting way 00:10:59.420 --> 00:11:03.780 align:middle line:84% sometimes because some illusions are bistable in the sense 00:11:03.780 --> 00:11:06.500 align:middle line:84% that you can actually hear them in one of two 00:11:06.500 --> 00:11:09.140 align:middle line:90% or sometimes three ways. 00:11:09.140 --> 00:11:11.380 align:middle line:84% And that kind of suggests that that's 00:11:11.380 --> 00:11:13.540 align:middle line:84% a situation where maybe there are 00:11:13.540 --> 00:11:18.100 align:middle line:84% two modes of the posterior that are both pretty good, 00:11:18.100 --> 00:11:21.080 align:middle line:84% and your perceptual system kind of switches between them. 00:11:21.080 --> 00:11:23.500 align:middle line:84% And that typically doesn't happen very much 00:11:23.500 --> 00:11:24.460 align:middle line:90% in the real world. 00:11:24.460 --> 00:11:27.100 align:middle line:84% One of the remarkable things about perception 00:11:27.100 --> 00:11:29.820 align:middle line:84% in the real world is you pretty much just always see or hear 00:11:29.820 --> 00:11:31.580 align:middle line:84% this one thing, and it usually seems 00:11:31.580 --> 00:11:34.377 align:middle line:84% to be pretty close to ground truth. 00:11:34.377 --> 00:11:36.460 align:middle line:84% And illusions are interesting just because they're 00:11:36.460 --> 00:11:37.480 align:middle line:90% often not like that. 00:11:37.480 --> 00:11:38.445 align:middle line:90% So I would say that's-- 00:11:38.445 --> 00:11:40.070 align:middle line:84% I mean, that's one respect in which you 00:11:40.070 --> 00:11:43.090 align:middle line:84% can think of illusions, as being related to this, 00:11:43.090 --> 00:11:46.350 align:middle line:84% is that sometimes you experience bistability. 00:11:46.350 --> 00:11:48.750 align:middle line:84% And you could think of that as there 00:11:48.750 --> 00:11:53.190 align:middle line:84% being two good solutions or two approximately equally 00:11:53.190 --> 00:11:54.930 align:middle line:90% good explanations of a stimulus. 00:11:54.930 --> 00:11:57.510 align:middle line:90% Yeah. 00:11:57.510 --> 00:12:00.950 align:middle line:84% STUDENT: How good or bad are current networks 00:12:00.950 --> 00:12:03.450 align:middle line:84% at doing auditory scene analysis? 00:12:03.450 --> 00:12:06.710 align:middle line:90% 00:12:06.710 --> 00:12:09.770 align:middle line:84% JOSH MCDERMOTT: The problem is certainly unsolved. 00:12:09.770 --> 00:12:12.550 align:middle line:90% So I mean, I would say the-- 00:12:12.550 --> 00:12:19.730 align:middle line:84% I mean, as with a lot of problems that involve signals, 00:12:19.730 --> 00:12:23.310 align:middle line:84% there's been enormous progress in the last 10 years. 00:12:23.310 --> 00:12:27.150 align:middle line:84% But in terms of having models that 00:12:27.150 --> 00:12:29.710 align:middle line:84% solve the problems in the way that people do, 00:12:29.710 --> 00:12:31.210 align:middle line:84% we're still fairly far from that. 00:12:31.210 --> 00:12:33.430 align:middle line:84% So one domain in which there's been a lot of progress 00:12:33.430 --> 00:12:35.250 align:middle line:84% is what's called source separation. 00:12:35.250 --> 00:12:38.200 align:middle line:84% So that's really the problem of taking 00:12:38.200 --> 00:12:40.920 align:middle line:84% a mixture of source signals and then estimating 00:12:40.920 --> 00:12:42.100 align:middle line:90% the constituent signals. 00:12:42.100 --> 00:12:45.400 align:middle line:84% So imagine we have a recording of two people talking. 00:12:45.400 --> 00:12:47.000 align:middle line:90% You add that together. 00:12:47.000 --> 00:12:49.120 align:middle line:84% And then the task is to take that mixture 00:12:49.120 --> 00:12:51.140 align:middle line:84% and estimate the two source signals. 00:12:51.140 --> 00:12:54.720 align:middle line:84% So that's a problem that neural networks actually do 00:12:54.720 --> 00:12:56.300 align:middle line:90% a pretty good job of solving. 00:12:56.300 --> 00:12:59.400 align:middle line:90% 00:12:59.400 --> 00:13:01.960 align:middle line:84% You might ask, OK, well, can we take one of those systems 00:13:01.960 --> 00:13:05.040 align:middle line:84% and ask whether it explains all of the phenomena 00:13:05.040 --> 00:13:07.700 align:middle line:84% that we've been looking at in human auditory perception? 00:13:07.700 --> 00:13:09.700 align:middle line:84% And we've actually done some of this in our lab. 00:13:09.700 --> 00:13:12.000 align:middle line:84% And typically, those systems, they really 00:13:12.000 --> 00:13:14.560 align:middle line:84% don't reproduce almost any of the phenomena 00:13:14.560 --> 00:13:16.040 align:middle line:90% that we've seen here. 00:13:16.040 --> 00:13:17.720 align:middle line:84% They seem to be doing something that's 00:13:17.720 --> 00:13:19.880 align:middle line:84% a bit different and a little bit more specific. 00:13:19.880 --> 00:13:22.560 align:middle line:84% So I would say yeah, we're a little ways off 00:13:22.560 --> 00:13:26.600 align:middle line:84% from solving this in the way that people do. 00:13:26.600 --> 00:13:29.680 align:middle line:84% And part of it is related to some 00:13:29.680 --> 00:13:31.600 align:middle line:84% of these sorts of issues, which just turn out 00:13:31.600 --> 00:13:32.900 align:middle line:90% to be very challenging. 00:13:32.900 --> 00:13:36.400 align:middle line:90% Yeah. 00:13:36.400 --> 00:13:37.540 align:middle line:90% Any other questions? 00:13:37.540 --> 00:13:43.060 align:middle line:90% 00:13:43.060 --> 00:13:45.680 align:middle line:84% So I want to turn briefly to thinking 00:13:45.680 --> 00:13:49.920 align:middle line:84% about what happens in the brain when we segregate sounds. 00:13:49.920 --> 00:13:53.200 align:middle line:84% And we're going to make a brief detour 00:13:53.200 --> 00:13:56.040 align:middle line:84% to talk in a little bit more detail 00:13:56.040 --> 00:13:58.680 align:middle line:84% about some issues in auditory neuroscience 00:13:58.680 --> 00:14:00.480 align:middle line:84% and in particular about receptive fields 00:14:00.480 --> 00:14:04.600 align:middle line:84% in the auditory system and how neurons represent sound. 00:14:04.600 --> 00:14:06.560 align:middle line:84% So this slide is depicting something called 00:14:06.560 --> 00:14:08.080 align:middle line:90% the spike-triggered average. 00:14:08.080 --> 00:14:11.880 align:middle line:84% How many people have encountered a spike-triggered average? 00:14:11.880 --> 00:14:13.058 align:middle line:90% Yeah, maybe just a few. 00:14:13.058 --> 00:14:15.600 align:middle line:84% So they teach this in 940, which I think probably most of you 00:14:15.600 --> 00:14:16.960 align:middle line:90% haven't taken yet. 00:14:16.960 --> 00:14:22.720 align:middle line:84% But so it's a method for characterizing the sensitivity 00:14:22.720 --> 00:14:24.620 align:middle line:90% of neurons in sensory systems. 00:14:24.620 --> 00:14:27.680 align:middle line:84% And the basic idea is actually pretty simple and pretty cool. 00:14:27.680 --> 00:14:31.320 align:middle line:84% So the notion is let's suppose you do an experiment where 00:14:31.320 --> 00:14:35.170 align:middle line:84% you're recording from a neuron in the brain of some organism. 00:14:35.170 --> 00:14:40.470 align:middle line:84% And you expose the organism to a continuous stimulus. 00:14:40.470 --> 00:14:42.830 align:middle line:84% So in this particular case, it's a visual-- 00:14:42.830 --> 00:14:46.570 align:middle line:84% it's a visual stimulus, which is a sequence of noise patterns-- 00:14:46.570 --> 00:14:48.310 align:middle line:84% pattern one, pattern two, pattern three, 00:14:48.310 --> 00:14:50.450 align:middle line:90% rapidly one after another. 00:14:50.450 --> 00:14:52.610 align:middle line:84% So you're recording from the neuron, 00:14:52.610 --> 00:14:56.530 align:middle line:84% and you keep track of when it fires an action potential. 00:14:56.530 --> 00:14:58.890 align:middle line:84% And so the notion is that every time there 00:14:58.890 --> 00:15:02.330 align:middle line:84% is an action potential, you look back in the stimulus history, 00:15:02.330 --> 00:15:08.250 align:middle line:84% and you grab the stimulus that preceded the action potential 00:15:08.250 --> 00:15:12.230 align:middle line:84% by some amount of time, say 30 milliseconds, 40 milliseconds. 00:15:12.230 --> 00:15:16.243 align:middle line:84% And you collect that set of stimuli. 00:15:16.243 --> 00:15:17.910 align:middle line:84% So there's this long sequence of frames, 00:15:17.910 --> 00:15:20.910 align:middle line:84% and these are just the five ones that preceded these spikes. 00:15:20.910 --> 00:15:23.330 align:middle line:84% So you collect all these stimuli that preceded the spikes, 00:15:23.330 --> 00:15:24.970 align:middle line:90% and then you average them. 00:15:24.970 --> 00:15:27.810 align:middle line:84% It's called the spike-triggered average. 00:15:27.810 --> 00:15:31.250 align:middle line:84% So every time there's a spike, you 00:15:31.250 --> 00:15:33.920 align:middle line:84% average the stimulus that preceded the spike. 00:15:33.920 --> 00:15:37.500 align:middle line:84% And so what that gives you is the average stimulus that 00:15:37.500 --> 00:15:38.640 align:middle line:90% precedes a spike. 00:15:38.640 --> 00:15:42.740 align:middle line:84% So it's kind of like you could think of that as representing 00:15:42.740 --> 00:15:46.980 align:middle line:84% the feature in the input that's the neurons responding to it. 00:15:46.980 --> 00:15:52.200 align:middle line:84% And it turns out that if under certain assumptions, 00:15:52.200 --> 00:15:55.180 align:middle line:84% you can think of the spike-triggered average 00:15:55.180 --> 00:15:59.460 align:middle line:84% as an estimate of the filter that the neuron is 00:15:59.460 --> 00:16:00.160 align:middle line:90% instantiating. 00:16:00.160 --> 00:16:02.980 align:middle line:84% So if the neuron is a filter followed 00:16:02.980 --> 00:16:06.140 align:middle line:84% by a simple non-linearity, the spike-triggered average 00:16:06.140 --> 00:16:09.520 align:middle line:90% will give you the filter. 00:16:09.520 --> 00:16:13.700 align:middle line:84% So it's a fairly common method for characterizing 00:16:13.700 --> 00:16:17.325 align:middle line:84% the tuning properties of neurons in sensory systems. 00:16:17.325 --> 00:16:19.200 align:middle line:84% So this is an example from the visual domain. 00:16:19.200 --> 00:16:21.033 align:middle line:84% And we'll talk more about this kind of stuff 00:16:21.033 --> 00:16:21.980 align:middle line:90% when we get to vision. 00:16:21.980 --> 00:16:24.860 align:middle line:84% But you can also do this in the auditory domain. 00:16:24.860 --> 00:16:29.100 align:middle line:84% So imagine instead of showing you these sequence of frames, 00:16:29.100 --> 00:16:31.324 align:middle line:90% we play you this. 00:16:31.324 --> 00:16:36.430 align:middle line:90% [RANDOM NOISES] 00:16:36.430 --> 00:16:37.570 align:middle line:90% OK, you get the idea. 00:16:37.570 --> 00:16:40.750 align:middle line:84% So it's a very extended kind of random noise stimulus. 00:16:40.750 --> 00:16:44.630 align:middle line:84% And we record from a neuron in your brain. 00:16:44.630 --> 00:16:46.430 align:middle line:84% And then every time there's a spike, 00:16:46.430 --> 00:16:48.750 align:middle line:84% we average the little section of the stimulus 00:16:48.750 --> 00:16:50.570 align:middle line:90% that preceded the spike. 00:16:50.570 --> 00:16:53.310 align:middle line:84% And so if we did this in a stage of your brain 00:16:53.310 --> 00:16:56.910 align:middle line:84% called the inferior colliculus-- so this is a subcortical region 00:16:56.910 --> 00:16:58.990 align:middle line:84% of the auditory pathway like a few synapses up 00:16:58.990 --> 00:17:01.150 align:middle line:90% from the cochlea. 00:17:01.150 --> 00:17:05.589 align:middle line:84% This is the spectrogram of the stimulus 00:17:05.589 --> 00:17:08.089 align:middle line:90% that would precede a spike. 00:17:08.089 --> 00:17:11.270 align:middle line:84% And so what you can see is that there 00:17:11.270 --> 00:17:13.630 align:middle line:90% is a little burst of energy-- 00:17:13.630 --> 00:17:17.010 align:middle line:84% that's the red-- and then a little dip in the energy. 00:17:17.010 --> 00:17:18.589 align:middle line:84% And at a particular frequency, it 00:17:18.589 --> 00:17:21.730 align:middle line:84% looks like it's around 8 kilohertz. 00:17:21.730 --> 00:17:23.790 align:middle line:84% So this particular neuron is being 00:17:23.790 --> 00:17:29.060 align:middle line:84% driven by amplitude modulation at a particular rate. 00:17:29.060 --> 00:17:31.880 align:middle line:84% So you can see that the cycle time here 00:17:31.880 --> 00:17:34.900 align:middle line:84% is 4 or 5 milliseconds, something like that. 00:17:34.900 --> 00:17:35.840 align:middle line:90% So it's pretty rapid. 00:17:35.840 --> 00:17:40.280 align:middle line:84% It's a very rapid modulation at a particular audio frequency. 00:17:40.280 --> 00:17:44.720 align:middle line:84% So this is known as a spectro-temporal receptive 00:17:44.720 --> 00:17:45.880 align:middle line:90% field. 00:17:45.880 --> 00:17:48.760 align:middle line:90% It is usually abbreviated STRF. 00:17:48.760 --> 00:17:51.720 align:middle line:84% And people who work in the field will just call this a STRF. 00:17:51.720 --> 00:17:53.753 align:middle line:84% So if you go to a neuroscience conference 00:17:53.753 --> 00:17:55.420 align:middle line:84% and you hear people talking about STRFs, 00:17:55.420 --> 00:17:57.160 align:middle line:90% that's what they're talking to. 00:17:57.160 --> 00:17:59.300 align:middle line:84% That's what they're talking about. 00:17:59.300 --> 00:18:01.400 align:middle line:84% So this is an example where this neuron 00:18:01.400 --> 00:18:03.080 align:middle line:84% is responding to changes in amplitude 00:18:03.080 --> 00:18:04.960 align:middle line:90% in a particular frequency range. 00:18:04.960 --> 00:18:08.180 align:middle line:90% 00:18:08.180 --> 00:18:10.760 align:middle line:84% So remember how when we were talking 00:18:10.760 --> 00:18:13.060 align:middle line:84% about the cochlea and auditory nerve fibers, 00:18:13.060 --> 00:18:16.840 align:middle line:84% we talked about how they are tuned in frequency. 00:18:16.840 --> 00:18:19.000 align:middle line:84% And when we were discussing that, 00:18:19.000 --> 00:18:21.600 align:middle line:84% we were talking about audio frequency. 00:18:21.600 --> 00:18:23.128 align:middle line:90% So you can take a waveform. 00:18:23.128 --> 00:18:24.420 align:middle line:90% You can do a Fourier transform. 00:18:24.420 --> 00:18:26.020 align:middle line:84% And you decompose that into sinusoids. 00:18:26.020 --> 00:18:27.510 align:middle line:90% Those are audio frequencies. 00:18:27.510 --> 00:18:31.290 align:middle line:84% But there's a lot of information in sound 00:18:31.290 --> 00:18:34.110 align:middle line:84% that is conveyed by amplitude modulation. 00:18:34.110 --> 00:18:35.690 align:middle line:84% And remember how we talked about how 00:18:35.690 --> 00:18:38.130 align:middle line:90% a sound can have an envelope? 00:18:38.130 --> 00:18:40.170 align:middle line:90% So here is an example sound. 00:18:40.170 --> 00:18:41.652 align:middle line:90% The waveform is in blue. 00:18:41.652 --> 00:18:43.110 align:middle line:84% It's actually wiggling up and down. 00:18:43.110 --> 00:18:44.818 align:middle line:84% It's just so dense that you don't see it. 00:18:44.818 --> 00:18:47.550 align:middle line:84% But there's the instantaneous amplitude, 00:18:47.550 --> 00:18:51.530 align:middle line:84% which is plotted in red, that kind of goes up and down. 00:18:51.530 --> 00:18:58.270 align:middle line:84% And if we zoom in on a particular frequency channel, 00:18:58.270 --> 00:19:00.370 align:middle line:84% you can see that the blue waveform here 00:19:00.370 --> 00:19:02.130 align:middle line:84% is oscillating up and down, and then 00:19:02.130 --> 00:19:07.770 align:middle line:84% the red envelope is kind of changing at a much slower rate 00:19:07.770 --> 00:19:10.750 align:middle line:84% So this is what we call a cochleagram. 00:19:10.750 --> 00:19:12.690 align:middle line:84% So what the way that you generate 00:19:12.690 --> 00:19:15.370 align:middle line:84% this picture is you take a filter 00:19:15.370 --> 00:19:18.690 align:middle line:84% bank that's kind of replicating the frequency tuning of the ear. 00:19:18.690 --> 00:19:22.570 align:middle line:84% And you apply that filter bank to a sound. 00:19:22.570 --> 00:19:27.440 align:middle line:84% You then measure the envelope of every frequency channel's 00:19:27.440 --> 00:19:32.300 align:middle line:84% output, and you plot that in grayscale as a row. 00:19:32.300 --> 00:19:37.060 align:middle line:84% So every row in this picture is the instantaneous amplitude 00:19:37.060 --> 00:19:39.640 align:middle line:84% that varies over time of a particular filter. 00:19:39.640 --> 00:19:44.220 align:middle line:84% You can think of that as a particular place on the cochlea. 00:19:44.220 --> 00:19:46.160 align:middle line:84% And so this looks a lot like a spectrogram. 00:19:46.160 --> 00:19:48.460 align:middle line:84% The only difference being that we made it 00:19:48.460 --> 00:19:50.062 align:middle line:90% with a model of the ear. 00:19:50.062 --> 00:19:52.520 align:middle line:84% And so it's got the tuning properties that you see in here. 00:19:52.520 --> 00:19:55.187 align:middle line:84% So remember how in the ear, the filters get broader when they're 00:19:55.187 --> 00:19:56.320 align:middle line:90% tuned to high frequencies. 00:19:56.320 --> 00:19:58.020 align:middle line:90% So that happens here-- 00:19:58.020 --> 00:20:00.320 align:middle line:90% stuff like that. 00:20:00.320 --> 00:20:04.620 align:middle line:84% Now what's shown here is the power spectrum 00:20:04.620 --> 00:20:07.140 align:middle line:84% of the response of one particular filter. 00:20:07.140 --> 00:20:08.820 align:middle line:84% In this case, it's one that's centered 00:20:08.820 --> 00:20:12.620 align:middle line:90% at looks like 350 hertz-- 00:20:12.620 --> 00:20:16.440 align:middle line:84% so kind of down here And there's two. 00:20:16.440 --> 00:20:18.380 align:middle line:84% So remember, a power spectrum plots power 00:20:18.380 --> 00:20:21.380 align:middle line:90% as a function of frequency. 00:20:21.380 --> 00:20:25.870 align:middle line:84% And the blue graph here shows you the power spectrum 00:20:25.870 --> 00:20:28.950 align:middle line:90% of the output of the filter. 00:20:28.950 --> 00:20:32.510 align:middle line:84% So this tells you the audio frequencies that 00:20:32.510 --> 00:20:34.570 align:middle line:90% are present in that output. 00:20:34.570 --> 00:20:36.050 align:middle line:90% And this is a bandpass filter. 00:20:36.050 --> 00:20:41.030 align:middle line:84% And so you can see that there's power between around 280 00:20:41.030 --> 00:20:42.870 align:middle line:90% and 400 hertz. 00:20:42.870 --> 00:20:46.070 align:middle line:84% So there's positive stuff here, and then it goes to 0. 00:20:46.070 --> 00:20:48.510 align:middle line:90% So that's a bandpass filter. 00:20:48.510 --> 00:20:53.190 align:middle line:84% Now the red plot here is the power spectrum 00:20:53.190 --> 00:20:56.390 align:middle line:84% of the envelope of the filter of the red curve 00:20:56.390 --> 00:21:00.330 align:middle line:84% up here And what does this show us? 00:21:00.330 --> 00:21:02.590 align:middle line:84% Well, it shows us that the frequencies 00:21:02.590 --> 00:21:06.610 align:middle line:84% that are in the envelope are low frequencies. 00:21:06.610 --> 00:21:08.470 align:middle line:90% The envelope is low pass. 00:21:08.470 --> 00:21:11.510 align:middle line:84% So that says that the instantaneous amplitude 00:21:11.510 --> 00:21:16.130 align:middle line:84% of the output of that filter is varying relatively slowly. 00:21:16.130 --> 00:21:20.670 align:middle line:90% It's defined by low frequencies. 00:21:20.670 --> 00:21:24.620 align:middle line:84% And these are modulation frequencies. 00:21:24.620 --> 00:21:26.960 align:middle line:84% So the key takeaway message from this slide 00:21:26.960 --> 00:21:29.400 align:middle line:84% is that we have audio frequencies. 00:21:29.400 --> 00:21:31.160 align:middle line:84% That's kind of what you get if you do 00:21:31.160 --> 00:21:33.960 align:middle line:90% a Fourier transform of sound. 00:21:33.960 --> 00:21:36.260 align:middle line:84% We also have modulation frequencies. 00:21:36.260 --> 00:21:40.680 align:middle line:84% That's what you get if you take the envelope of part of a sound, 00:21:40.680 --> 00:21:44.720 align:middle line:84% and you look at the frequency decomposition of that envelope. 00:21:44.720 --> 00:21:48.500 align:middle line:84% And the envelope tends to vary more slowly. 00:21:48.500 --> 00:21:53.760 align:middle line:84% And so it contains low frequencies. 00:21:53.760 --> 00:21:56.960 align:middle line:84% And so whereas the auditory nerve 00:21:56.960 --> 00:21:59.100 align:middle line:90% is tuned to audio frequency. 00:21:59.100 --> 00:22:01.080 align:middle line:84% If you look later in the auditory system, 00:22:01.080 --> 00:22:05.320 align:middle line:84% you'll find tuning to modulation frequency. 00:22:05.320 --> 00:22:07.680 align:middle line:84% And so we just saw an example of that 00:22:07.680 --> 00:22:10.280 align:middle line:84% with the spectro-temporal receptive field. 00:22:10.280 --> 00:22:13.600 align:middle line:84% This is an example of a bunch of different neurons 00:22:13.600 --> 00:22:14.660 align:middle line:90% in the midbrain. 00:22:14.660 --> 00:22:19.360 align:middle line:84% And each one of these curves is a tuning curve 00:22:19.360 --> 00:22:21.760 align:middle line:84% that plots the response of the neuron 00:22:21.760 --> 00:22:26.000 align:middle line:84% as a function of the modulation frequency. 00:22:26.000 --> 00:22:29.840 align:middle line:84% So we have a stimulus, and we are varying how rapidly 00:22:29.840 --> 00:22:33.260 align:middle line:90% the amplitude is modulated. 00:22:33.260 --> 00:22:36.220 align:middle line:90% [MODULATION SOUNDS] 00:22:36.220 --> 00:22:37.940 align:middle line:90% So that's modulation frequency. 00:22:37.940 --> 00:22:41.240 align:middle line:84% And what you can see-- so this particular neuron 00:22:41.240 --> 00:22:46.880 align:middle line:84% here responds best at looks like around 70 hertz. 00:22:46.880 --> 00:22:48.600 align:middle line:84% This particular neuron here responds 00:22:48.600 --> 00:22:50.200 align:middle line:90% best at around 40 hertz. 00:22:50.200 --> 00:22:53.160 align:middle line:84% This particular neuron here is down around 25 hertz. 00:22:53.160 --> 00:22:56.460 align:middle line:84% And this looks like a filter bank. 00:22:56.460 --> 00:22:59.320 align:middle line:84% So it's a set of bandpass filters 00:22:59.320 --> 00:23:01.280 align:middle line:84% that are tuned to different frequencies, 00:23:01.280 --> 00:23:03.560 align:middle line:84% but they're modulation frequencies. 00:23:03.560 --> 00:23:06.960 align:middle line:84% So it's kind of conceptually similar to the filter bank 00:23:06.960 --> 00:23:09.360 align:middle line:84% that we use to model the cochlea. 00:23:09.360 --> 00:23:11.480 align:middle line:84% But instead of being tuned to audio frequency, 00:23:11.480 --> 00:23:13.460 align:middle line:84% it's tuned to modulation frequency. 00:23:13.460 --> 00:23:19.200 align:middle line:90% 00:23:19.200 --> 00:23:22.950 align:middle line:84% So this is a picture that kind of captures that idea. 00:23:22.950 --> 00:23:26.130 align:middle line:84% So this is a model of auditory signal processing 00:23:26.130 --> 00:23:30.270 align:middle line:84% that extends from the cochlea to the midbrain and thalamus. 00:23:30.270 --> 00:23:31.950 align:middle line:90% And so what happens here? 00:23:31.950 --> 00:23:34.450 align:middle line:84% Well, we start out with a sound waveform. 00:23:34.450 --> 00:23:37.670 align:middle line:84% That gets processed by a bank of filters. 00:23:37.670 --> 00:23:40.290 align:middle line:84% So each one of these pictures here 00:23:40.290 --> 00:23:44.530 align:middle line:84% represents the frequency tuning of one filter in your cochlea. 00:23:44.530 --> 00:23:46.990 align:middle line:84% So this is frequency, and that's the response. 00:23:46.990 --> 00:23:48.830 align:middle line:84% So this is a high-frequency filter. 00:23:48.830 --> 00:23:50.650 align:middle line:90% That's a low-frequency filter. 00:23:50.650 --> 00:23:52.730 align:middle line:84% This is the output of that filter when it's 00:23:52.730 --> 00:23:54.610 align:middle line:90% applied to that sound signal. 00:23:54.610 --> 00:23:56.207 align:middle line:84% So this is a low-frequency filter, 00:23:56.207 --> 00:23:58.290 align:middle line:84% and you can see the blue curve is kind of wiggling 00:23:58.290 --> 00:24:00.250 align:middle line:90% at a pretty slow rate. 00:24:00.250 --> 00:24:01.670 align:middle line:84% This is a higher-frequency filter, 00:24:01.670 --> 00:24:06.330 align:middle line:84% and you can see the blue curve is wiggling at a faster rate. 00:24:06.330 --> 00:24:08.610 align:middle line:84% But the output of each of those filters 00:24:08.610 --> 00:24:10.310 align:middle line:90% is characterized by an envelope. 00:24:10.310 --> 00:24:13.190 align:middle line:84% And you can extract that with a nonlinear operation. 00:24:13.190 --> 00:24:15.370 align:middle line:84% And that's just what these blue curves are. 00:24:15.370 --> 00:24:16.940 align:middle line:84% And then these envelopes can then 00:24:16.940 --> 00:24:18.760 align:middle line:84% be passed through a second set of filters. 00:24:18.760 --> 00:24:20.780 align:middle line:90% These are modulation filters. 00:24:20.780 --> 00:24:22.480 align:middle line:84% So they are also tuned in frequency. 00:24:22.480 --> 00:24:24.000 align:middle line:90% But this modulation frequency. 00:24:24.000 --> 00:24:26.680 align:middle line:84% So again, this one is tuned to low frequencies. 00:24:26.680 --> 00:24:29.718 align:middle line:84% And so you can see that the output varies rather slowly. 00:24:29.718 --> 00:24:31.260 align:middle line:84% This one's tuned to high frequencies, 00:24:31.260 --> 00:24:34.840 align:middle line:84% and the output varies more quickly. 00:24:34.840 --> 00:24:37.500 align:middle line:84% So we often think of sensory systems 00:24:37.500 --> 00:24:42.220 align:middle line:84% as kind of having these cascades of filtering operations 00:24:42.220 --> 00:24:44.060 align:middle line:84% followed by non-linear operations followed 00:24:44.060 --> 00:24:45.440 align:middle line:90% by more filtering operations. 00:24:45.440 --> 00:24:47.960 align:middle line:84% And this is just one example in the auditory system. 00:24:47.960 --> 00:24:55.580 align:middle line:90% 00:24:55.580 --> 00:24:59.020 align:middle line:84% So I showed you an example of a STRF, an STRF, 00:24:59.020 --> 00:25:02.420 align:middle line:84% in the inferior colliculus where things look nice and simple. 00:25:02.420 --> 00:25:05.220 align:middle line:84% If you make the same kind of measurement 00:25:05.220 --> 00:25:08.800 align:middle line:84% in the auditory cortex, things are a little more complicated. 00:25:08.800 --> 00:25:12.340 align:middle line:84% So these are four examples of spectro-temporal receptive 00:25:12.340 --> 00:25:15.190 align:middle line:84% fields measured in the auditory cortex. 00:25:15.190 --> 00:25:17.550 align:middle line:84% So again, you can think of this as the stimulus that's 00:25:17.550 --> 00:25:21.550 align:middle line:84% most likely to elicit a spike in a particular neuron. 00:25:21.550 --> 00:25:26.270 align:middle line:84% And the general takeaway is they're more complicated. 00:25:26.270 --> 00:25:27.830 align:middle line:84% You can see that some of the neurons 00:25:27.830 --> 00:25:32.190 align:middle line:84% are tuned for it looks like kind of ripples in the spectrum, 00:25:32.190 --> 00:25:36.710 align:middle line:84% maybe with some kind of difference in frequency content. 00:25:36.710 --> 00:25:39.250 align:middle line:84% There's still some tuning to amplitude modulation, 00:25:39.250 --> 00:25:42.550 align:middle line:84% but it just gets more complicated. 00:25:42.550 --> 00:25:47.310 align:middle line:84% And so there is a standard model of the auditory cortex that's 00:25:47.310 --> 00:25:50.750 align:middle line:84% inspired by these kinds of operations. 00:25:50.750 --> 00:25:53.350 align:middle line:84% And it looks like this, where you've 00:25:53.350 --> 00:25:56.750 align:middle line:84% got the output of the early auditory 00:25:56.750 --> 00:26:01.670 align:middle line:84% system, which we think of as a time-frequency representation. 00:26:01.670 --> 00:26:03.390 align:middle line:84% That then gets passed through a set 00:26:03.390 --> 00:26:09.030 align:middle line:84% of filters that are now tuned in both time and frequency. 00:26:09.030 --> 00:26:16.200 align:middle line:84% And so they're kind of sensitive to these features in the earlier 00:26:16.200 --> 00:26:17.500 align:middle line:90% representation. 00:26:17.500 --> 00:26:20.875 align:middle line:84% So these are again modulation filters, 00:26:20.875 --> 00:26:23.000 align:middle line:84% but they're now spectro-temporal modulation filters 00:26:23.000 --> 00:26:24.750 align:middle line:84% that are tuned in both time and frequency. 00:26:24.750 --> 00:26:28.280 align:middle line:84% But the important thing to take away from this 00:26:28.280 --> 00:26:32.160 align:middle line:84% is that we get of second set of filters that's 00:26:32.160 --> 00:26:35.320 align:middle line:84% kind of tuned to higher-order structure that's 00:26:35.320 --> 00:26:39.320 align:middle line:84% operating on the output of an earlier set of filters. 00:26:39.320 --> 00:26:43.160 align:middle line:84% Any questions about audio frequency or modulation 00:26:43.160 --> 00:26:48.120 align:middle line:84% frequency or these cascades of filters? 00:26:48.120 --> 00:26:49.040 align:middle line:90% Yeah. 00:26:49.040 --> 00:26:51.320 align:middle line:90% STUDENT: I'm curious. 00:26:51.320 --> 00:26:53.015 align:middle line:84% These all have very nice waveforms. 00:26:53.015 --> 00:26:55.140 align:middle line:84% But like when you're actually looking at the brain, 00:26:55.140 --> 00:26:58.600 align:middle line:84% can you directly measure this and see this kind signal 00:26:58.600 --> 00:27:01.700 align:middle line:84% transformation, or is it more abstract? 00:27:01.700 --> 00:27:07.000 align:middle line:90% 00:27:07.000 --> 00:27:10.600 align:middle line:84% JOSH MCDERMOTT: I mean, OK, the difference is that normally, 00:27:10.600 --> 00:27:12.470 align:middle line:84% if you're making a measurement from neurons, 00:27:12.470 --> 00:27:14.650 align:middle line:84% you're measuring action potentials. 00:27:14.650 --> 00:27:20.530 align:middle line:84% And so like the notion is that typically-- 00:27:20.530 --> 00:27:25.130 align:middle line:84% when we talk about these filters kind of existing in the brain, 00:27:25.130 --> 00:27:28.090 align:middle line:84% normally, what we mean is that the neuron is implementing 00:27:28.090 --> 00:27:32.010 align:middle line:84% that filter plus a stage of generating action potentials. 00:27:32.010 --> 00:27:34.050 align:middle line:84% So there's this spike generation phase 00:27:34.050 --> 00:27:37.430 align:middle line:84% that's kind of missing from the picture. 00:27:37.430 --> 00:27:41.850 align:middle line:84% And so that adds a layer of complexity. 00:27:41.850 --> 00:27:44.210 align:middle line:84% But modulo that, yeah, I mean, you do 00:27:44.210 --> 00:27:48.330 align:middle line:90% see traces of stuff like this. 00:27:48.330 --> 00:27:51.350 align:middle line:84% I mean, these pictures-- like this picture. 00:27:51.350 --> 00:27:53.490 align:middle line:84% I mean, this is derived from measurements 00:27:53.490 --> 00:27:56.130 align:middle line:84% that are made from one neuron in the inferior colliculus. 00:27:56.130 --> 00:27:58.053 align:middle line:84% I mean, these pictures are derived 00:27:58.053 --> 00:27:59.970 align:middle line:84% from measurements made from individual neurons 00:27:59.970 --> 00:28:00.750 align:middle line:90% in the cortex. 00:28:00.750 --> 00:28:05.150 align:middle line:84% So yeah, I mean, this is what you get. 00:28:05.150 --> 00:28:05.650 align:middle line:90% Yeah. 00:28:05.650 --> 00:28:10.300 align:middle line:90% 00:28:10.300 --> 00:28:14.540 align:middle line:84% So what is all this stuff good for? 00:28:14.540 --> 00:28:18.020 align:middle line:84% So one kind of interesting application of this 00:28:18.020 --> 00:28:21.340 align:middle line:84% is to understand the representation 00:28:21.340 --> 00:28:24.340 align:middle line:90% of sounds in the human brain. 00:28:24.340 --> 00:28:26.900 align:middle line:84% And one kind of opportunity to do this kind 00:28:26.900 --> 00:28:29.320 align:middle line:90% comes from epilepsy patients. 00:28:29.320 --> 00:28:31.980 align:middle line:84% So in general, with human neuroscience, 00:28:31.980 --> 00:28:33.920 align:middle line:84% because typically the methods are noninvasive, 00:28:33.920 --> 00:28:36.640 align:middle line:84% they're very coarse both spatially and temporally. 00:28:36.640 --> 00:28:38.220 align:middle line:90% So we have fMRI. 00:28:38.220 --> 00:28:39.040 align:middle line:90% We have MEG. 00:28:39.040 --> 00:28:41.020 align:middle line:90% We have EEG. 00:28:41.020 --> 00:28:44.700 align:middle line:84% And they're making measurements of signals from the brain. 00:28:44.700 --> 00:28:46.500 align:middle line:84% But typically, those measurements 00:28:46.500 --> 00:28:48.380 align:middle line:84% are kind of pooling over time and space 00:28:48.380 --> 00:28:50.500 align:middle line:90% to a pretty good degree. 00:28:50.500 --> 00:28:51.820 align:middle line:90% And they're pretty indirect. 00:28:51.820 --> 00:28:55.780 align:middle line:90% 00:28:55.780 --> 00:28:58.125 align:middle line:84% If you have somebody who has bad epilepsy, 00:28:58.125 --> 00:28:59.500 align:middle line:84% sometimes the epilepsy interferes 00:28:59.500 --> 00:29:02.093 align:middle line:84% with quality of life to the extent 00:29:02.093 --> 00:29:04.260 align:middle line:84% that they decide that they want to have an operation 00:29:04.260 --> 00:29:06.940 align:middle line:84% to try to remove the part of the brain that's 00:29:06.940 --> 00:29:08.410 align:middle line:90% causing their seizures. 00:29:08.410 --> 00:29:11.910 align:middle line:84% So the person will go to the hospital, 00:29:11.910 --> 00:29:15.790 align:middle line:84% and oftentimes there will be a grid of electrodes 00:29:15.790 --> 00:29:18.150 align:middle line:90% that is placed on their cortex. 00:29:18.150 --> 00:29:21.453 align:middle line:84% And they'll be monitored for some period of time. 00:29:21.453 --> 00:29:22.870 align:middle line:84% Often, this is because the surgeon 00:29:22.870 --> 00:29:25.430 align:middle line:84% is trying to determine exactly where the seizure is 00:29:25.430 --> 00:29:26.348 align:middle line:90% coming from. 00:29:26.348 --> 00:29:28.390 align:middle line:84% So the person will be hanging out in the hospital 00:29:28.390 --> 00:29:31.015 align:middle line:84% with a grid of electrodes on their brain, 00:29:31.015 --> 00:29:32.390 align:middle line:84% essentially waiting for a seizure 00:29:32.390 --> 00:29:35.070 align:middle line:84% to happen so that they can make measurements and figure out 00:29:35.070 --> 00:29:37.750 align:middle line:84% where they need to perform the operation. 00:29:37.750 --> 00:29:39.910 align:middle line:84% And during this period of time, they're 00:29:39.910 --> 00:29:42.230 align:middle line:84% often willing to participate in experiments. 00:29:42.230 --> 00:29:45.430 align:middle line:84% And so this is of a branch of human neuroscience 00:29:45.430 --> 00:29:48.350 align:middle line:84% where people make intracranial recordings 00:29:48.350 --> 00:29:49.463 align:middle line:90% in this kind of setting. 00:29:49.463 --> 00:29:51.630 align:middle line:84% And so you can, for instance, show the person movies 00:29:51.630 --> 00:29:55.110 align:middle line:84% or play them sounds, or ask them to do math problems and measure 00:29:55.110 --> 00:29:56.830 align:middle line:90% responses in their brain. 00:29:56.830 --> 00:29:58.870 align:middle line:84% And the key difference over a lot 00:29:58.870 --> 00:30:01.510 align:middle line:84% of the methods that we normally use in human neuroscience 00:30:01.510 --> 00:30:04.130 align:middle line:84% is that the resolution is substantially better. 00:30:04.130 --> 00:30:06.760 align:middle line:84% So you place a grid of electrodes on the brain, 00:30:06.760 --> 00:30:09.160 align:middle line:84% and the temporal resolution is typically much better 00:30:09.160 --> 00:30:10.880 align:middle line:90% than you would get with fMRI. 00:30:10.880 --> 00:30:13.080 align:middle line:84% And the spatial resolution is way better 00:30:13.080 --> 00:30:14.820 align:middle line:84% than you would get with MEG or EEG 00:30:14.820 --> 00:30:20.243 align:middle line:84% and usually better than what you would get with fMRI as well. 00:30:20.243 --> 00:30:22.160 align:middle line:84% So I'm going to tell you about one application 00:30:22.160 --> 00:30:25.560 align:middle line:84% of this type of method to understanding auditory scene 00:30:25.560 --> 00:30:26.400 align:middle line:90% analysis. 00:30:26.400 --> 00:30:29.647 align:middle line:84% And there's two things that I have to tell you about. 00:30:29.647 --> 00:30:31.980 align:middle line:84% And you're going to have to mostly accept them on faith. 00:30:31.980 --> 00:30:34.543 align:middle line:90% 00:30:34.543 --> 00:30:35.460 align:middle line:90% It's really one thing. 00:30:35.460 --> 00:30:39.040 align:middle line:84% So we've talked about how you can make measurements 00:30:39.040 --> 00:30:42.200 align:middle line:84% from the brain and infer spectro-temporal receptive 00:30:42.200 --> 00:30:42.880 align:middle line:90% fields. 00:30:42.880 --> 00:30:46.400 align:middle line:84% And what you're going to have to accept 00:30:46.400 --> 00:30:49.373 align:middle line:84% is that if you have characterized the receptive 00:30:49.373 --> 00:30:51.040 align:middle line:84% fields of the neurons and if you measure 00:30:51.040 --> 00:30:53.480 align:middle line:84% their responses to a sound, you can 00:30:53.480 --> 00:30:56.440 align:middle line:84% reconstruct an estimate of the stimulus 00:30:56.440 --> 00:30:58.480 align:middle line:90% that the person is hearing. 00:30:58.480 --> 00:31:02.940 align:middle line:84% So given you can measure the responses to a sound. 00:31:02.940 --> 00:31:05.357 align:middle line:84% And if you have some prior measurements of their responses 00:31:05.357 --> 00:31:06.732 align:middle line:84% to sound and you've characterized 00:31:06.732 --> 00:31:08.180 align:middle line:84% like what the neurons respond to, 00:31:08.180 --> 00:31:11.360 align:middle line:84% you can then estimate what the person's listening to. 00:31:11.360 --> 00:31:15.320 align:middle line:84% And so the study that I'm going to tell you about 00:31:15.320 --> 00:31:18.400 align:middle line:84% is an attempt to answer the question of, 00:31:18.400 --> 00:31:22.600 align:middle line:84% what stimulus is represented by the brain when you are listening 00:31:22.600 --> 00:31:25.320 align:middle line:84% to a mixture of sounds and trying to pay attention 00:31:25.320 --> 00:31:26.480 align:middle line:90% to one of them? 00:31:26.480 --> 00:31:28.300 align:middle line:84% So it's the classic cocktail party problem. 00:31:28.300 --> 00:31:30.160 align:middle line:84% You have two people who are talking. 00:31:30.160 --> 00:31:33.260 align:middle line:84% You're trying to listen to person A. The question is, 00:31:33.260 --> 00:31:35.720 align:middle line:84% what is being represented in your brain 00:31:35.720 --> 00:31:37.020 align:middle line:90% when that is happening? 00:31:37.020 --> 00:31:40.198 align:middle line:84% And so the way that works is you do 00:31:40.198 --> 00:31:42.740 align:middle line:84% an experiment where you play the person a mixture of talkers. 00:31:42.740 --> 00:31:45.582 align:middle line:84% You cue them to attend to talker one. 00:31:45.582 --> 00:31:47.040 align:middle line:84% And then you do the reconstruction, 00:31:47.040 --> 00:31:52.020 align:middle line:84% and you compare that to the spectrogram of the mixture. 00:31:52.020 --> 00:31:53.860 align:middle line:84% And as well as of the individual talkers. 00:31:53.860 --> 00:31:56.080 align:middle line:84% And the question is, will the representation 00:31:56.080 --> 00:31:59.120 align:middle line:84% in your brain actually be a lot more similar to the voice 00:31:59.120 --> 00:32:01.040 align:middle line:84% that you're attending than to the voice 00:32:01.040 --> 00:32:02.710 align:middle line:90% that you're trying to ignore? 00:32:02.710 --> 00:32:06.010 align:middle line:90% 00:32:06.010 --> 00:32:09.730 align:middle line:84% So here's an example of where these-- 00:32:09.730 --> 00:32:10.990 align:middle line:90% of these recordings are made. 00:32:10.990 --> 00:32:12.930 align:middle line:84% So this is in the auditory cortex, 00:32:12.930 --> 00:32:16.890 align:middle line:84% typically over the superior-temporal gyrus. 00:32:16.890 --> 00:32:21.648 align:middle line:84% This is an example of the spike-triggered average 00:32:21.648 --> 00:32:24.190 align:middle line:84% or the receptive field that you would get from one electrode. 00:32:24.190 --> 00:32:25.690 align:middle line:84% So this is an electrode that happens 00:32:25.690 --> 00:32:29.530 align:middle line:84% to be responsive to very high frequencies. 00:32:29.530 --> 00:32:38.090 align:middle line:84% And this is a depiction of one example stimulus of a mixture 00:32:38.090 --> 00:32:39.610 align:middle line:90% of two people talking. 00:32:39.610 --> 00:32:41.710 align:middle line:90% So this is frequency. 00:32:41.710 --> 00:32:42.850 align:middle line:90% This is time. 00:32:42.850 --> 00:32:45.770 align:middle line:84% And now every point in the spectrogram 00:32:45.770 --> 00:32:50.610 align:middle line:84% has been color-coded depending on whether speaker 1 or speaker 00:32:50.610 --> 00:32:53.450 align:middle line:84% 2 is kind of dominating the mixture 00:32:53.450 --> 00:32:55.230 align:middle line:90% at that particular point. 00:32:55.230 --> 00:32:58.890 align:middle line:84% So usually, the energy in a mixture 00:32:58.890 --> 00:33:02.500 align:middle line:84% is predominantly coming from one speaker or the other. 00:33:02.500 --> 00:33:04.760 align:middle line:84% And so at all the places where it's blue, 00:33:04.760 --> 00:33:06.598 align:middle line:84% that's where speaker 1 has the most energy. 00:33:06.598 --> 00:33:08.140 align:middle line:84% All the places where it's red, that's 00:33:08.140 --> 00:33:10.160 align:middle line:84% where speaker 2 has the most energy. 00:33:10.160 --> 00:33:12.700 align:middle line:90% 00:33:12.700 --> 00:33:15.060 align:middle line:84% And so this is an example here where 00:33:15.060 --> 00:33:18.340 align:middle line:84% they're plotting the response of this one particular electrode, 00:33:18.340 --> 00:33:21.160 align:middle line:84% which is tuned to these high frequencies. 00:33:21.160 --> 00:33:23.873 align:middle line:84% And you can see in that high-frequency band, 00:33:23.873 --> 00:33:26.040 align:middle line:84% there are these two places that are kind of circled. 00:33:26.040 --> 00:33:28.940 align:middle line:84% So one of the circles is a place where 00:33:28.940 --> 00:33:31.880 align:middle line:84% speaker 1 has this high-frequency energy, 00:33:31.880 --> 00:33:33.700 align:middle line:84% and another place is where speaker 2 00:33:33.700 --> 00:33:35.380 align:middle line:90% has this high-frequency energy. 00:33:35.380 --> 00:33:41.520 align:middle line:84% So first, look at the dashed lines here. 00:33:41.520 --> 00:33:43.340 align:middle line:84% Those show the responses that we've 00:33:43.340 --> 00:33:47.180 align:middle line:84% measured in the brain when either speaker 1 or speaker 2 00:33:47.180 --> 00:33:49.180 align:middle line:90% is being presented alone. 00:33:49.180 --> 00:33:53.100 align:middle line:84% And so you can see that the dashed line here 00:33:53.100 --> 00:33:56.580 align:middle line:84% has a peak which corresponds to that burst 00:33:56.580 --> 00:33:58.870 align:middle line:84% of high-frequency energy that's in speaker 2. 00:33:58.870 --> 00:34:01.430 align:middle line:84% The dashed line, when it's read, has a peak 00:34:01.430 --> 00:34:04.430 align:middle line:84% that's later in the trial that corresponds to that later burst 00:34:04.430 --> 00:34:05.650 align:middle line:90% of high-frequency energy. 00:34:05.650 --> 00:34:07.510 align:middle line:84% So the point is that the electrode is just 00:34:07.510 --> 00:34:10.530 align:middle line:84% responding to the content of the stimulus here. 00:34:10.530 --> 00:34:13.710 align:middle line:84% Now the really critical case is where the person 00:34:13.710 --> 00:34:16.190 align:middle line:84% is being presented with a mixture of these talkers 00:34:16.190 --> 00:34:18.710 align:middle line:84% and either is attending to speaker 1 00:34:18.710 --> 00:34:20.790 align:middle line:90% or attending to speaker 2. 00:34:20.790 --> 00:34:25.429 align:middle line:84% And that's the solid lines, the solid red and the solid blue. 00:34:25.429 --> 00:34:28.670 align:middle line:84% So this is a case where the physical stimulus is the same, 00:34:28.670 --> 00:34:30.670 align:middle line:84% but the listener is directing their attention 00:34:30.670 --> 00:34:32.090 align:middle line:90% to one talker or the other. 00:34:32.090 --> 00:34:34.429 align:middle line:84% And what you're supposed to take away from this 00:34:34.429 --> 00:34:38.389 align:middle line:84% is that the brain responds during that condition. 00:34:38.389 --> 00:34:41.389 align:middle line:84% When you're attending to speaker 1 is more like the response 00:34:41.389 --> 00:34:45.449 align:middle line:84% to speaker 1 alone, and when you're attending to speaker 2, 00:34:45.449 --> 00:34:47.889 align:middle line:84% it's more like the response to speaker 2 alone. 00:34:47.889 --> 00:34:50.730 align:middle line:90% 00:34:50.730 --> 00:34:56.040 align:middle line:84% So the inference here is that the brain is preferentially 00:34:56.040 --> 00:35:00.900 align:middle line:84% representing the talker that you're paying attention to. 00:35:00.900 --> 00:35:04.960 align:middle line:84% So this is just one kind of example from one electrode. 00:35:04.960 --> 00:35:08.800 align:middle line:84% This is what happens when you reconstruct the full stimulus. 00:35:08.800 --> 00:35:13.140 align:middle line:84% So here, we have speaker 1 and speaker 2. 00:35:13.140 --> 00:35:15.700 align:middle line:84% Speaker one is saying the sentence that's up at the top-- 00:35:15.700 --> 00:35:17.220 align:middle line:84% ready tiger go to green five now. 00:35:17.220 --> 00:35:19.560 align:middle line:84% Speaker 2 saying ready ringo go to red two now. 00:35:19.560 --> 00:35:21.740 align:middle line:84% You may wonder, why are the sentences so weird? 00:35:21.740 --> 00:35:24.922 align:middle line:84% And the reason is that this is a corpus of sentence 00:35:24.922 --> 00:35:27.380 align:middle line:84% that was specifically designed for this type of experiment. 00:35:27.380 --> 00:35:31.860 align:middle line:84% And the idea is that there is a call sign on, 00:35:31.860 --> 00:35:34.320 align:middle line:84% which in this case, would be tiger or Ringo. 00:35:34.320 --> 00:35:37.340 align:middle line:84% And ahead of the trial, you tell the person, 00:35:37.340 --> 00:35:39.600 align:middle line:84% you got to pay attention to the person who says tiger. 00:35:39.600 --> 00:35:41.860 align:middle line:90% So then you hear the mixture. 00:35:41.860 --> 00:35:43.720 align:middle line:84% You hear somebody say tiger, and then you're 00:35:43.720 --> 00:35:45.803 align:middle line:84% supposed to be tracking the rest of what they say. 00:35:45.803 --> 00:35:47.920 align:middle line:84% And you have to report the color and the number. 00:35:47.920 --> 00:35:50.560 align:middle line:84% Like green 5 would be the correct answer here. 00:35:50.560 --> 00:35:52.640 align:middle line:84% And so the idea is that this gives you 00:35:52.640 --> 00:35:56.542 align:middle line:84% a way to ask people what they're hearing, 00:35:56.542 --> 00:35:58.250 align:middle line:84% and they have this discrete set of things 00:35:58.250 --> 00:35:59.333 align:middle line:90% that they can choose from. 00:35:59.333 --> 00:36:02.950 align:middle line:84% And so the experiment is kind of simple to conduct and code. 00:36:02.950 --> 00:36:05.970 align:middle line:84% So now we get a mixture of the two talkers-- again, 00:36:05.970 --> 00:36:09.210 align:middle line:84% color-coded in the way that we discussed. 00:36:09.210 --> 00:36:12.010 align:middle line:84% And then these four panels are showing you 00:36:12.010 --> 00:36:15.450 align:middle line:84% the stimulus reconstructions that's made from the recordings 00:36:15.450 --> 00:36:16.490 align:middle line:90% from the brain. 00:36:16.490 --> 00:36:19.390 align:middle line:90% 00:36:19.390 --> 00:36:22.825 align:middle line:84% And so these first two that's called single sp1 and sp2, 00:36:22.825 --> 00:36:24.450 align:middle line:84% those are the cases where the person is 00:36:24.450 --> 00:36:26.850 align:middle line:90% hearing only a single talker. 00:36:26.850 --> 00:36:31.090 align:middle line:84% And the argument is that the reconstruction of the stimulus 00:36:31.090 --> 00:36:33.930 align:middle line:84% is supposed to look like the original stimulus but kind 00:36:33.930 --> 00:36:34.750 align:middle line:90% of blurrier. 00:36:34.750 --> 00:36:37.050 align:middle line:84% So there's definitely some information 00:36:37.050 --> 00:36:40.170 align:middle line:84% that gets lost because the measurements from the brain 00:36:40.170 --> 00:36:41.610 align:middle line:90% are still very coarse. 00:36:41.610 --> 00:36:43.970 align:middle line:84% But you capture the coarse spectro-temporal structure 00:36:43.970 --> 00:36:45.290 align:middle line:90% of the original talker. 00:36:45.290 --> 00:36:46.950 align:middle line:84% And these and these look different. 00:36:46.950 --> 00:36:47.910 align:middle line:84% And that one looks like that one. 00:36:47.910 --> 00:36:48.950 align:middle line:84% And that one looks like that one. 00:36:48.950 --> 00:36:51.075 align:middle line:84% That's what you're supposed to take away from this. 00:36:51.075 --> 00:36:53.220 align:middle line:84% So now you get the reconstruction 00:36:53.220 --> 00:36:56.260 align:middle line:84% of the mixture where the person is attending to speaker 1 00:36:56.260 --> 00:36:57.680 align:middle line:90% or attending to speaker 2. 00:36:57.680 --> 00:37:00.740 align:middle line:84% And so what you're supposed to take away from this 00:37:00.740 --> 00:37:02.980 align:middle line:84% is that this reconstruction kind of looks 00:37:02.980 --> 00:37:06.595 align:middle line:84% like the reconstruction of the individual talker on its own. 00:37:06.595 --> 00:37:09.220 align:middle line:84% And this reconstruction, where the persons attending to speaker 00:37:09.220 --> 00:37:12.960 align:middle line:90% 2 looks like this one. 00:37:12.960 --> 00:37:15.460 align:middle line:84% And what's shown down here is kind 00:37:15.460 --> 00:37:18.760 align:middle line:84% of a superposition of all of these things 00:37:18.760 --> 00:37:20.520 align:middle line:90% so you can make that comparison. 00:37:20.520 --> 00:37:22.620 align:middle line:84% But I think just this visual comparison 00:37:22.620 --> 00:37:23.840 align:middle line:90% is supposed to be enough. 00:37:23.840 --> 00:37:26.020 align:middle line:90% So this is just one example. 00:37:26.020 --> 00:37:29.860 align:middle line:84% And to quantify this across like a large set of trials, 00:37:29.860 --> 00:37:34.300 align:middle line:84% they did this analysis where they computed the correlation 00:37:34.300 --> 00:37:36.780 align:middle line:84% between the stimulus reconstructions 00:37:36.780 --> 00:37:42.260 align:middle line:84% from the mixtures and the individual talkers. 00:37:42.260 --> 00:37:47.340 align:middle line:84% And specifically, you've got the correlation between the mixture 00:37:47.340 --> 00:37:49.340 align:middle line:84% and the reconstruction of speaker 1 00:37:49.340 --> 00:37:51.270 align:middle line:84% and the reconstruction of speaker 2 00:37:51.270 --> 00:37:53.668 align:middle line:84% when you're either attending to speaker 2-- that's 00:37:53.668 --> 00:37:55.710 align:middle line:84% the red dots-- or attending to speaker 1-- that's 00:37:55.710 --> 00:37:57.250 align:middle line:90% the blue dots. 00:37:57.250 --> 00:37:59.910 align:middle line:84% And so what you're supposed to take away from this graph 00:37:59.910 --> 00:38:04.170 align:middle line:84% is that the red dots are mostly above the diagonal, 00:38:04.170 --> 00:38:06.450 align:middle line:84% and the blue dots are mostly below the diagonal. 00:38:06.450 --> 00:38:10.150 align:middle line:84% And so that means that when you're attending to speaker 2, 00:38:10.150 --> 00:38:13.950 align:middle line:84% the reconstruction is more like the reconstruction for speaker 00:38:13.950 --> 00:38:17.590 align:middle line:84% 2 than the reconstruction of speaker 1 and vice versa 00:38:17.590 --> 00:38:20.750 align:middle line:90% when the attention is switched. 00:38:20.750 --> 00:38:22.750 align:middle line:84% So what this is showing us again is 00:38:22.750 --> 00:38:26.990 align:middle line:84% that the representation in the auditory cortex 00:38:26.990 --> 00:38:29.790 align:middle line:84% seems to be predominantly that of the talker 00:38:29.790 --> 00:38:31.990 align:middle line:84% that you are paying attention to. 00:38:31.990 --> 00:38:36.750 align:middle line:84% The other thing that they did that's kind of cool 00:38:36.750 --> 00:38:39.430 align:middle line:84% is they separately did this analysis for trials where 00:38:39.430 --> 00:38:41.890 align:middle line:90% people answered correctly. 00:38:41.890 --> 00:38:44.670 align:middle line:84% So again, they're cued to attend to one of the talkers, 00:38:44.670 --> 00:38:48.350 align:middle line:84% and they correctly report the words that talker said. 00:38:48.350 --> 00:38:49.920 align:middle line:84% When they do the analysis on trials 00:38:49.920 --> 00:38:52.040 align:middle line:84% where people made mistakes, you no longer 00:38:52.040 --> 00:38:54.960 align:middle line:84% see the difference between the red and the blue dots. 00:38:54.960 --> 00:38:56.560 align:middle line:84% And that's consistent with the idea 00:38:56.560 --> 00:38:58.760 align:middle line:84% that the mistakes that people make 00:38:58.760 --> 00:39:01.100 align:middle line:84% are kind of attentional selection errors. 00:39:01.100 --> 00:39:03.120 align:middle line:84% They don't manage to correctly attend 00:39:03.120 --> 00:39:05.600 align:middle line:84% to the person they're trying to pay attention to. 00:39:05.600 --> 00:39:07.800 align:middle line:84% And so the brain representation doesn't 00:39:07.800 --> 00:39:10.540 align:middle line:84% isolate the representation of that talker. 00:39:10.540 --> 00:39:14.237 align:middle line:90% 00:39:14.237 --> 00:39:15.820 align:middle line:84% What questions do you have about this? 00:39:15.820 --> 00:39:20.480 align:middle line:90% 00:39:20.480 --> 00:39:22.360 align:middle line:84% STUDENT: Yeah, so when the dots are 00:39:22.360 --> 00:39:24.860 align:middle line:84% on the wrong side for the correct trials, 00:39:24.860 --> 00:39:27.460 align:middle line:84% when the dots are on the wrong side of the graph, 00:39:27.460 --> 00:39:29.700 align:middle line:84% is that more like the participant-- 00:39:29.700 --> 00:39:32.060 align:middle line:84% like it was a fluke or something? 00:39:32.060 --> 00:39:34.022 align:middle line:90% They just guessed correctly? 00:39:34.022 --> 00:39:35.480 align:middle line:84% JOSH MCDERMOTT: I mean, my guess is 00:39:35.480 --> 00:39:37.340 align:middle line:84% a lot of this is just measurement noise. 00:39:37.340 --> 00:39:39.800 align:middle line:84% I mean, like the brain measurements are not-- 00:39:39.800 --> 00:39:40.880 align:middle line:90% none of this is perfect. 00:39:40.880 --> 00:39:42.700 align:middle line:84% The brain measurements are not perfect. 00:39:42.700 --> 00:39:44.620 align:middle line:84% The reconstruction algorithm is not perfect. 00:39:44.620 --> 00:39:47.800 align:middle line:90% 00:39:47.800 --> 00:39:51.880 align:middle line:84% Yeah, you're always going to get some of that. 00:39:51.880 --> 00:39:53.600 align:middle line:84% Yeah, because you just have this one set 00:39:53.600 --> 00:39:55.683 align:middle line:84% of electrodes that's on the surface of the cortex. 00:39:55.683 --> 00:39:57.500 align:middle line:90% And yeah. 00:39:57.500 --> 00:40:00.140 align:middle line:84% So I think that's some of that is bound to happen. 00:40:00.140 --> 00:40:02.380 align:middle line:84% So I'm not sure that is necessarily meaningful. 00:40:02.380 --> 00:40:07.330 align:middle line:90% 00:40:07.330 --> 00:40:09.080 align:middle line:84% In principle, I mean, it could be the case 00:40:09.080 --> 00:40:13.600 align:middle line:84% that maybe some of those dots that are on the wrong side 00:40:13.600 --> 00:40:15.920 align:middle line:84% are cases where the person wasn't 00:40:15.920 --> 00:40:18.407 align:middle line:84% as attending as successfully, and maybe 00:40:18.407 --> 00:40:20.740 align:middle line:84% they were not as confident in their answer or something. 00:40:20.740 --> 00:40:23.820 align:middle line:90% It's possible, but hard to know. 00:40:23.820 --> 00:40:24.320 align:middle line:90% Yeah. 00:40:24.320 --> 00:40:29.180 align:middle line:90% 00:40:29.180 --> 00:40:32.640 align:middle line:84% So what have we learned from these kinds of experiments? 00:40:32.640 --> 00:40:37.000 align:middle line:84% So I would say that these type of experiments, they 00:40:37.000 --> 00:40:39.880 align:middle line:84% give us some indication of where in the brain 00:40:39.880 --> 00:40:43.400 align:middle line:84% individual sources can be selected with attention. 00:40:43.400 --> 00:40:47.388 align:middle line:84% In this case, it's the superior temporal gyrus. 00:40:47.388 --> 00:40:49.930 align:middle line:84% So what does this tell us about the problem of auditory scene 00:40:49.930 --> 00:40:53.650 align:middle line:84% analysis of the inference of sources in the world? 00:40:53.650 --> 00:40:58.610 align:middle line:84% Well, in order for you to select a source with your attention-- 00:40:58.610 --> 00:41:00.930 align:middle line:84% the sources kind of have to be segregated. 00:41:00.930 --> 00:41:04.370 align:middle line:84% So it gives us some constraints on where 00:41:04.370 --> 00:41:06.810 align:middle line:90% that segregation happens. 00:41:06.810 --> 00:41:08.650 align:middle line:84% But I would say we still really know 00:41:08.650 --> 00:41:11.410 align:middle line:84% very little about the neural implementation 00:41:11.410 --> 00:41:14.430 align:middle line:84% of the inference of individual sources from mixtures. 00:41:14.430 --> 00:41:19.130 align:middle line:84% And part of the challenge in that 00:41:19.130 --> 00:41:22.510 align:middle line:84% is that it's not so obvious actually 00:41:22.510 --> 00:41:28.330 align:middle line:84% how to go looking for neural correlates of auditory scene 00:41:28.330 --> 00:41:29.095 align:middle line:90% analysis. 00:41:29.095 --> 00:41:30.970 align:middle line:84% So looking for neural correlates of attention 00:41:30.970 --> 00:41:32.887 align:middle line:84% is pretty straightforward because attention is 00:41:32.887 --> 00:41:34.190 align:middle line:90% directed to a particular thing. 00:41:34.190 --> 00:41:36.570 align:middle line:84% And you can ask whether the representation 00:41:36.570 --> 00:41:39.210 align:middle line:84% of that particular thing is different compared to the thing 00:41:39.210 --> 00:41:40.970 align:middle line:84% that you're not paying attention to. 00:41:40.970 --> 00:41:44.460 align:middle line:84% But the question of how is it that when the two people are 00:41:44.460 --> 00:41:46.792 align:middle line:84% talking, you actually represent in your brain 00:41:46.792 --> 00:41:48.500 align:middle line:84% the fact that there are these two things? 00:41:48.500 --> 00:41:50.125 align:middle line:84% That's actually something that we don't 00:41:50.125 --> 00:41:52.620 align:middle line:90% know very much about still. 00:41:52.620 --> 00:41:54.260 align:middle line:84% And my guess is there probably is 00:41:54.260 --> 00:41:58.380 align:middle line:84% some of fundamental representational difference 00:41:58.380 --> 00:42:01.000 align:middle line:84% when you're listening to one thing versus two in part. 00:42:01.000 --> 00:42:02.580 align:middle line:84% So we saw those examples last time 00:42:02.580 --> 00:42:06.520 align:middle line:84% of when you perceive streaming between these two sets of tones, 00:42:06.520 --> 00:42:08.540 align:middle line:84% you become unable to discriminate 00:42:08.540 --> 00:42:10.360 align:middle line:84% the temporal relations between those tones. 00:42:10.360 --> 00:42:11.860 align:middle line:84% So that's some evidence that there's 00:42:11.860 --> 00:42:14.620 align:middle line:84% something pretty different about the representation 00:42:14.620 --> 00:42:16.580 align:middle line:90% of one thing versus two things. 00:42:16.580 --> 00:42:20.220 align:middle line:84% But what that is at this point is still essentially unknown. 00:42:20.220 --> 00:42:22.080 align:middle line:84% So that's like a major open problem, 00:42:22.080 --> 00:42:27.780 align:middle line:90% I would say, in neuroscience. 00:42:27.780 --> 00:42:32.300 align:middle line:84% Any other questions about neural correlates 00:42:32.300 --> 00:42:33.760 align:middle line:90% of auditory scene analysis? 00:42:33.760 --> 00:42:37.700 align:middle line:90% 00:42:37.700 --> 00:42:40.500 align:middle line:84% The next topic we're going to discuss-- 00:42:40.500 --> 00:42:44.110 align:middle line:84% and this will really be the last part of the section 00:42:44.110 --> 00:42:46.190 align:middle line:90% of the course on hearing-- 00:42:46.190 --> 00:42:47.450 align:middle line:90% is speech perception. 00:42:47.450 --> 00:42:51.070 align:middle line:84% And you may wonder why are we talking about speech 00:42:51.070 --> 00:42:52.350 align:middle line:90% in a perception class. 00:42:52.350 --> 00:42:55.510 align:middle line:84% You often think of language as being a high-level cognitive 00:42:55.510 --> 00:42:57.790 align:middle line:90% phenomenon. 00:42:57.790 --> 00:43:00.470 align:middle line:84% And the answer is that speech is typically 00:43:00.470 --> 00:43:03.030 align:middle line:84% received by the brain as a sound signal, 00:43:03.030 --> 00:43:05.790 align:middle line:84% and perceptual processes are needed 00:43:05.790 --> 00:43:08.990 align:middle line:84% to transform the sound into a form 00:43:08.990 --> 00:43:11.430 align:middle line:84% that semantic and syntactic processes that we associate 00:43:11.430 --> 00:43:13.430 align:middle line:90% with language can handle. 00:43:13.430 --> 00:43:16.750 align:middle line:84% And this requires solving some challenging perceptual problems. 00:43:16.750 --> 00:43:19.390 align:middle line:90% 00:43:19.390 --> 00:43:21.750 align:middle line:84% Speech is also like one of the most important things 00:43:21.750 --> 00:43:25.150 align:middle line:84% that we do with our auditory system. 00:43:25.150 --> 00:43:29.390 align:middle line:84% It's like an important thing to understand in the context 00:43:29.390 --> 00:43:31.690 align:middle line:90% of the auditory system. 00:43:31.690 --> 00:43:36.430 align:middle line:84% So one of the key concepts to take away from today's lecture 00:43:36.430 --> 00:43:37.810 align:middle line:90% is the source-filter model. 00:43:37.810 --> 00:43:41.280 align:middle line:84% And that refers to how we think of the way in which speech 00:43:41.280 --> 00:43:42.900 align:middle line:90% is generated. 00:43:42.900 --> 00:43:46.200 align:middle line:84% So the idea is that speech is produced 00:43:46.200 --> 00:43:49.320 align:middle line:84% by a source that generates a sound that 00:43:49.320 --> 00:43:51.880 align:middle line:90% gets passed through a filter. 00:43:51.880 --> 00:43:55.200 align:middle line:90% The sound source is the larynx. 00:43:55.200 --> 00:43:57.392 align:middle line:84% This is a video of somebody's larynx. 00:43:57.392 --> 00:43:59.600 align:middle line:84% So what happened was there's a little camera that got 00:43:59.600 --> 00:44:00.820 align:middle line:90% stuck down somebody's throat. 00:44:00.820 --> 00:44:04.160 align:middle line:84% And then they had to say something like a vowel. 00:44:04.160 --> 00:44:06.440 align:middle line:84% And then they slowed down the movie a lot. 00:44:06.440 --> 00:44:10.960 align:middle line:84% So these are the vocal folds or vocal cords. 00:44:10.960 --> 00:44:13.480 align:middle line:84% So there are these bits of tissue 00:44:13.480 --> 00:44:15.360 align:middle line:90% that kind of open and close. 00:44:15.360 --> 00:44:17.640 align:middle line:84% They open and close when air is passed through them. 00:44:17.640 --> 00:44:20.160 align:middle line:84% So air gets blown through the vocal folds, 00:44:20.160 --> 00:44:22.880 align:middle line:84% and they open and close at a particular rate. 00:44:22.880 --> 00:44:24.440 align:middle line:84% The rate at which they open and close 00:44:24.440 --> 00:44:28.700 align:middle line:84% depends on the tension in the muscles of the larynx. 00:44:28.700 --> 00:44:30.560 align:middle line:84% So that's something that you can modulate. 00:44:30.560 --> 00:44:33.880 align:middle line:84% So I can talk like this, or I can talk like this. 00:44:33.880 --> 00:44:38.140 align:middle line:84% So that's just changes in the larynx. 00:44:38.140 --> 00:44:41.650 align:middle line:90% So that produces a sound. 00:44:41.650 --> 00:44:44.650 align:middle line:84% And then there are resonators that 00:44:44.650 --> 00:44:48.570 align:middle line:84% are part of-- that are essentially the vocal tract. 00:44:48.570 --> 00:44:50.870 align:middle line:84% And those are things that you can change. 00:44:50.870 --> 00:44:52.490 align:middle line:84% And when you talk, you are constantly 00:44:52.490 --> 00:44:53.790 align:middle line:90% changing those resonators. 00:44:53.790 --> 00:44:56.390 align:middle line:84% The sound passes through the pharynx or the throat, 00:44:56.390 --> 00:44:58.090 align:middle line:84% the mouth, the lips, and the nose, 00:44:58.090 --> 00:45:01.070 align:middle line:90% and that filters the sound. 00:45:01.070 --> 00:45:04.470 align:middle line:84% So here's a sideways view of the vocal production apparatus. 00:45:04.470 --> 00:45:07.855 align:middle line:84% We've got the vocal folds down here that generate sound. 00:45:07.855 --> 00:45:09.730 align:middle line:84% And then you can see the sound kind of passes 00:45:09.730 --> 00:45:14.690 align:middle line:84% through these cavities before emerging and traveling 00:45:14.690 --> 00:45:16.190 align:middle line:84% to whoever you're trying to talk to. 00:45:16.190 --> 00:45:19.650 align:middle line:90% 00:45:19.650 --> 00:45:23.410 align:middle line:84% So this is the spectrum of the sound source 00:45:23.410 --> 00:45:26.730 align:middle line:84% from the larynx when the speech is voiced-- so 00:45:26.730 --> 00:45:28.930 align:middle line:90% when the larynx is vibrating. 00:45:28.930 --> 00:45:33.290 align:middle line:84% And typically, the opening and closing of the larynx 00:45:33.290 --> 00:45:36.610 align:middle line:84% will be periodic in time, characterized 00:45:36.610 --> 00:45:39.958 align:middle line:84% by a fundamental frequency that produces harmonics. 00:45:39.958 --> 00:45:42.500 align:middle line:84% So you see how there are these frequencies that are regularly 00:45:42.500 --> 00:45:43.160 align:middle line:90% spaced? 00:45:43.160 --> 00:45:45.380 align:middle line:90% That's a harmonic signal. 00:45:45.380 --> 00:45:47.260 align:middle line:84% So that's what kind comes out of the larynx. 00:45:47.260 --> 00:45:50.280 align:middle line:84% That then gets passed through your vocal tract. 00:45:50.280 --> 00:45:53.700 align:middle line:84% And the vocal tract can be configured in different shapes, 00:45:53.700 --> 00:45:57.500 align:middle line:84% and that causes it to filter the sound differently. 00:45:57.500 --> 00:46:00.020 align:middle line:84% So these are three shapes corresponding 00:46:00.020 --> 00:46:02.900 align:middle line:90% to three different vowels. 00:46:02.900 --> 00:46:06.740 align:middle line:84% This is a schematic of the transfer function 00:46:06.740 --> 00:46:08.920 align:middle line:84% that is associated with those three shapes. 00:46:08.920 --> 00:46:10.460 align:middle line:84% So the transfer function essentially 00:46:10.460 --> 00:46:13.380 align:middle line:84% defines the frequency response of the filter. 00:46:13.380 --> 00:46:16.620 align:middle line:84% So this is like the gain that would 00:46:16.620 --> 00:46:17.800 align:middle line:90% be applied to the frequency. 00:46:17.800 --> 00:46:19.220 align:middle line:90% And this is frequency. 00:46:19.220 --> 00:46:21.140 align:middle line:84% And you can see that these transfer functions, 00:46:21.140 --> 00:46:23.660 align:middle line:84% they have peaks associated with them. 00:46:23.660 --> 00:46:26.060 align:middle line:84% Those peaks are known as formants. 00:46:26.060 --> 00:46:28.632 align:middle line:84% And critically, the peaks are in different places. 00:46:28.632 --> 00:46:30.340 align:middle line:84% And so that's what makes different vowels 00:46:30.340 --> 00:46:32.830 align:middle line:90% sound different. 00:46:32.830 --> 00:46:34.580 align:middle line:84% So the signal that comes out of your mouth 00:46:34.580 --> 00:46:37.870 align:middle line:84% you can think of as having the spectrum of the source 00:46:37.870 --> 00:46:40.770 align:middle line:84% multiplied by the transfer function of the filter. 00:46:40.770 --> 00:46:43.830 align:middle line:84% And that gives you this signal or this one or this one. 00:46:43.830 --> 00:46:47.590 align:middle line:84% So source and filter-- that's the source-filter model 00:46:47.590 --> 00:46:49.090 align:middle line:90% of speech production. 00:46:49.090 --> 00:46:52.230 align:middle line:90% 00:46:52.230 --> 00:46:55.190 align:middle line:84% So speech is typically characterized 00:46:55.190 --> 00:46:57.910 align:middle line:84% by having a fundamental frequency. 00:46:57.910 --> 00:47:00.750 align:middle line:84% So everybody's vocal folds can produce some range 00:47:00.750 --> 00:47:02.550 align:middle line:90% of fundamental frequencies. 00:47:02.550 --> 00:47:07.310 align:middle line:84% In men, that would typically be from around 80 to 240 hertz. 00:47:07.310 --> 00:47:09.737 align:middle line:84% In women, it's higher-- from 140 to 500 hertz. 00:47:09.737 --> 00:47:11.070 align:middle line:90% In children, it's higher still-- 00:47:11.070 --> 00:47:12.467 align:middle line:90% 170 to 600. 00:47:12.467 --> 00:47:14.550 align:middle line:84% This is determined by the length and the thickness 00:47:14.550 --> 00:47:16.430 align:middle line:90% of the vocal cords. 00:47:16.430 --> 00:47:22.590 align:middle line:84% And it's also different obviously across individuals. 00:47:22.590 --> 00:47:24.750 align:middle line:84% The sound signal is typically harmonic. 00:47:24.750 --> 00:47:26.010 align:middle line:90% And then there are resonators. 00:47:26.010 --> 00:47:28.230 align:middle line:84% These are cavities that amplify certain frequencies 00:47:28.230 --> 00:47:29.530 align:middle line:90% and dampen others. 00:47:29.530 --> 00:47:33.030 align:middle line:84% In general, bigger cavities have lower resonant frequencies 00:47:33.030 --> 00:47:34.860 align:middle line:84% and produce lower-frequency sounds. 00:47:34.860 --> 00:47:37.480 align:middle line:84% Smaller cavities have higher resonant frequencies 00:47:37.480 --> 00:47:40.600 align:middle line:90% and produce higher sounds, 00:47:40.600 --> 00:47:43.720 align:middle line:84% So the filter resonance is those peaks in the transfer function 00:47:43.720 --> 00:47:46.520 align:middle line:84% are known in speech known as formants, 00:47:46.520 --> 00:47:51.360 align:middle line:84% and they will be abbreviated as F1, F2, F3, 00:47:51.360 --> 00:47:53.460 align:middle line:84% where F1 would be the lowest peak. 00:47:53.460 --> 00:47:54.860 align:middle line:90% F2 would be the next one up. 00:47:54.860 --> 00:47:57.737 align:middle line:84% F3 would be the next one up, and so on and so forth. 00:47:57.737 --> 00:48:00.320 align:middle line:84% And these result from the size and the shape of the resonating 00:48:00.320 --> 00:48:01.640 align:middle line:90% cavities. 00:48:01.640 --> 00:48:07.020 align:middle line:84% And so sound is modulated by manipulating the articulators. 00:48:07.020 --> 00:48:09.480 align:middle line:84% So these are the movable parts of the vocal apparatus-- 00:48:09.480 --> 00:48:11.440 align:middle line:90% tongue, lips, palate. 00:48:11.440 --> 00:48:15.280 align:middle line:84% And these movements change the resonant properties, 00:48:15.280 --> 00:48:16.980 align:middle line:84% and they also change the airflow. 00:48:16.980 --> 00:48:22.420 align:middle line:90% 00:48:22.420 --> 00:48:27.860 align:middle line:84% So phonemes are kind of an important idea in speech. 00:48:27.860 --> 00:48:31.960 align:middle line:84% We often think of them as the goal of speech perception. 00:48:31.960 --> 00:48:33.520 align:middle line:84% So phonemes are typically defined 00:48:33.520 --> 00:48:35.120 align:middle line:84% as the smallest unit of sound that 00:48:35.120 --> 00:48:38.360 align:middle line:84% can make a difference in the meaning of speech-- so the thing 00:48:38.360 --> 00:48:41.840 align:middle line:90% that differentiates pot and dot. 00:48:41.840 --> 00:48:44.520 align:middle line:84% It's that initial sound-- the "puh" and the "duh." 00:48:44.520 --> 00:48:50.280 align:middle line:84% So phonemes often have a rough correspondence with letters 00:48:50.280 --> 00:48:53.420 align:middle line:90% when you write words down. 00:48:53.420 --> 00:48:54.420 align:middle line:90% But they're not letters. 00:48:54.420 --> 00:48:56.440 align:middle line:84% There's not a one-to-one correspondence. 00:48:56.440 --> 00:49:00.440 align:middle line:90% They're a part of sound. 00:49:00.440 --> 00:49:03.080 align:middle line:84% So we can think of the problem of speech perception 00:49:03.080 --> 00:49:06.320 align:middle line:84% as the problem of extracting a string of phonemes 00:49:06.320 --> 00:49:07.920 align:middle line:90% from the speech signal. 00:49:07.920 --> 00:49:12.200 align:middle line:84% And the issue is that this is hard to do. 00:49:12.200 --> 00:49:13.840 align:middle line:84% And that's because the sound waveform 00:49:13.840 --> 00:49:15.320 align:middle line:84% that results from a given phoneme 00:49:15.320 --> 00:49:18.540 align:middle line:84% can be highly variable across different conditions. 00:49:18.540 --> 00:49:22.560 align:middle line:90% 00:49:22.560 --> 00:49:26.400 align:middle line:84% So English is typically thought to have around 40 phonemes. 00:49:26.400 --> 00:49:29.080 align:middle line:84% Different languages have different numbers of phonemes. 00:49:29.080 --> 00:49:34.050 align:middle line:84% They vary from as low as 11 to as high as 140. 00:49:34.050 --> 00:49:37.050 align:middle line:84% The total inventory pooled across all the languages 00:49:37.050 --> 00:49:39.510 align:middle line:84% of the world is thought to be in the thousands, 00:49:39.510 --> 00:49:41.810 align:middle line:84% but there are some phonemes that are very 00:49:41.810 --> 00:49:44.390 align:middle line:90% common across most languages. 00:49:44.390 --> 00:49:49.290 align:middle line:90% 00:49:49.290 --> 00:49:51.850 align:middle line:84% So one type of phoneme are vowels. 00:49:51.850 --> 00:49:54.570 align:middle line:84% Vowels are phonemes that are produced 00:49:54.570 --> 00:50:00.170 align:middle line:84% by an unrestricted vocal tract, and different vowels 00:50:00.170 --> 00:50:02.290 align:middle line:84% are distinguished by how the sound produced 00:50:02.290 --> 00:50:04.150 align:middle line:90% by the vocal cords is filtered. 00:50:04.150 --> 00:50:08.530 align:middle line:84% And typically, the resonances can be altered in two main ways. 00:50:08.530 --> 00:50:13.830 align:middle line:84% One is by which part of the tongue gets moved, 00:50:13.830 --> 00:50:16.490 align:middle line:84% either the front or the back, and the other 00:50:16.490 --> 00:50:19.290 align:middle line:84% is the position of the tongue in the mouth-- 00:50:19.290 --> 00:50:21.970 align:middle line:90% can be high or low. 00:50:21.970 --> 00:50:24.290 align:middle line:90% So let's practice. 00:50:24.290 --> 00:50:29.220 align:middle line:84% Everybody say bet, bet, bet, bet, and pay attention to what's 00:50:29.220 --> 00:50:33.760 align:middle line:84% happening to your tongue-- bet, bet, bet. 00:50:33.760 --> 00:50:40.760 align:middle line:84% Now say buh, buh, buh, bet, but, bet, but, bet, but. 00:50:40.760 --> 00:50:42.720 align:middle line:84% So you feel your tongue kind of moving around? 00:50:42.720 --> 00:50:43.380 align:middle line:90% Yeah? 00:50:43.380 --> 00:50:43.880 align:middle line:90% Yeah. 00:50:43.880 --> 00:50:47.140 align:middle line:84% So you're normally not aware of this stuff. 00:50:47.140 --> 00:50:48.320 align:middle line:90% But that's what's happening. 00:50:48.320 --> 00:50:55.700 align:middle line:84% Let's do this one-- beat, beat, beat, bat, bat, bat, beet, bat, 00:50:55.700 --> 00:51:02.000 align:middle line:84% bat, beet, bat, bat, beet, bat, bat. 00:51:02.000 --> 00:51:02.500 align:middle line:90% Yeah. 00:51:02.500 --> 00:51:07.380 align:middle line:90% 00:51:07.380 --> 00:51:10.700 align:middle line:90% So here are three examples. 00:51:10.700 --> 00:51:14.340 align:middle line:84% So we talked about how the key feature of these transfer 00:51:14.340 --> 00:51:20.300 align:middle line:84% functions are these peaks, the formants-- so F1, F2, F3. 00:51:20.300 --> 00:51:23.180 align:middle line:90% Here you can see F4. 00:51:23.180 --> 00:51:26.730 align:middle line:84% And the first two formants do a pretty good job, actually, 00:51:26.730 --> 00:51:29.430 align:middle line:84% of distinguishing different vowels. 00:51:29.430 --> 00:51:34.350 align:middle line:84% So this is a graph that is plotting the frequencies 00:51:34.350 --> 00:51:36.990 align:middle line:84% of the first formant and the second formant for a whole bunch 00:51:36.990 --> 00:51:42.470 align:middle line:84% of different vowels-- heed, hid, head, had, hawed, hood, who'd. 00:51:42.470 --> 00:51:44.710 align:middle line:84% So when I just did that, my vocal tract 00:51:44.710 --> 00:51:47.030 align:middle line:84% was rapidly reconfiguring itself, 00:51:47.030 --> 00:51:49.390 align:middle line:84% causing these resonant frequencies to move around 00:51:49.390 --> 00:51:50.690 align:middle line:90% in this particular way. 00:51:50.690 --> 00:51:56.150 align:middle line:84% So heed happens to have a very low first formant 00:51:56.150 --> 00:51:59.310 align:middle line:90% and a very high second formant. 00:51:59.310 --> 00:52:02.970 align:middle line:84% Who'd has a pretty low first formant 00:52:02.970 --> 00:52:06.590 align:middle line:84% but a relatively low second formant, and so on and so forth. 00:52:06.590 --> 00:52:13.070 align:middle line:84% So it's pretty common to plot vowels 00:52:13.070 --> 00:52:16.030 align:middle line:84% in this space of the first formant 00:52:16.030 --> 00:52:17.950 align:middle line:90% versus the second formant. 00:52:17.950 --> 00:52:20.870 align:middle line:84% And to first order, the first formant 00:52:20.870 --> 00:52:24.680 align:middle line:84% is kind of mostly determined by the position of the tongue 00:52:24.680 --> 00:52:28.040 align:middle line:84% on the high-low axis, and the second formant 00:52:28.040 --> 00:52:30.600 align:middle line:84% is predominantly determined by the position of the tongue 00:52:30.600 --> 00:52:32.460 align:middle line:90% on the front-back axis. 00:52:32.460 --> 00:52:36.520 align:middle line:90% So let's do boot versus beat-- 00:52:36.520 --> 00:52:40.630 align:middle line:90% boot, beet, boot, beet. 00:52:40.630 --> 00:52:42.380 align:middle line:84% So you feel like when you're saying "ooh," 00:52:42.380 --> 00:52:45.380 align:middle line:84% the tongue is low in the mouth, and "EE" gets higher-- 00:52:45.380 --> 00:52:51.260 align:middle line:90% 00:52:51.260 --> 00:52:59.420 align:middle line:84% ooh ee-- or sorry, we messed this up. 00:52:59.420 --> 00:53:04.380 align:middle line:84% Ooh, ooh-- the tongue is more towards the back. 00:53:04.380 --> 00:53:08.148 align:middle line:84% Ee is more towards the front-- ooh, ee, ooh, ee. 00:53:08.148 --> 00:53:09.940 align:middle line:84% Feel your tongue moving from front to back? 00:53:09.940 --> 00:53:11.160 align:middle line:90% Yeah. 00:53:11.160 --> 00:53:15.200 align:middle line:84% So now if we do a comparison between boot and bot, 00:53:15.200 --> 00:53:17.560 align:middle line:84% so we should be able to detect the tongue moving 00:53:17.560 --> 00:53:18.800 align:middle line:90% from low to high-- 00:53:18.800 --> 00:53:24.370 align:middle line:90% ooh, aah, ooh, aah, ooh, aah. 00:53:24.370 --> 00:53:26.252 align:middle line:90% Yeah, that works. 00:53:26.252 --> 00:53:27.710 align:middle line:84% You can have lots of fun with this. 00:53:27.710 --> 00:53:31.570 align:middle line:90% 00:53:31.570 --> 00:53:37.530 align:middle line:84% These are spectrograms of these vowels. 00:53:37.530 --> 00:53:39.370 align:middle line:84% These look differently from spectrograms 00:53:39.370 --> 00:53:42.290 align:middle line:90% that you have seen before. 00:53:42.290 --> 00:53:44.970 align:middle line:84% And this is because of something that we haven't really 00:53:44.970 --> 00:53:47.790 align:middle line:84% talked about, which is that when you are generating 00:53:47.790 --> 00:53:51.170 align:middle line:84% a spectrogram, what happens is you take the sound signal, 00:53:51.170 --> 00:53:54.170 align:middle line:84% and you take a little window of that signal 00:53:54.170 --> 00:53:55.570 align:middle line:90% at a given point in time. 00:53:55.570 --> 00:53:58.250 align:middle line:84% And you take the Fourier transform of that, 00:53:58.250 --> 00:53:59.970 align:middle line:90% and then you plot that. 00:53:59.970 --> 00:54:01.490 align:middle line:84% And the Fried parameter, when you're 00:54:01.490 --> 00:54:05.530 align:middle line:84% generating the spectrogram, is the size of the window. 00:54:05.530 --> 00:54:09.410 align:middle line:84% And because of the uncertainty principle between the time 00:54:09.410 --> 00:54:15.050 align:middle line:84% and frequency domains, when the time window is very short, 00:54:15.050 --> 00:54:16.910 align:middle line:84% the frequency resolution will be poor. 00:54:16.910 --> 00:54:18.550 align:middle line:84% When the time window gets longer, 00:54:18.550 --> 00:54:21.075 align:middle line:84% the frequency resolution will get better. 00:54:21.075 --> 00:54:22.700 align:middle line:84% But your time resolution will get worse 00:54:22.700 --> 00:54:25.180 align:middle line:84% because you're kind of averaging the signal over a longer 00:54:25.180 --> 00:54:26.040 align:middle line:90% period of time. 00:54:26.040 --> 00:54:28.560 align:middle line:84% And so that's a choice that gets made. 00:54:28.560 --> 00:54:32.060 align:middle line:84% And so often in the spectrograms that we have seen previously 00:54:32.060 --> 00:54:34.870 align:middle line:84% in the class, we will be choosing the settings 00:54:34.870 --> 00:54:36.620 align:middle line:84% of the spectrogram such that the frequency 00:54:36.620 --> 00:54:38.180 align:middle line:84% resolution is pretty good so you can 00:54:38.180 --> 00:54:40.220 align:middle line:90% see the individual harmonics. 00:54:40.220 --> 00:54:42.300 align:middle line:84% However, it's pretty common with speech 00:54:42.300 --> 00:54:45.580 align:middle line:84% to actually set things up so that the frequency resolution is 00:54:45.580 --> 00:54:46.860 align:middle line:90% not so good. 00:54:46.860 --> 00:54:49.820 align:middle line:84% And so when you do that, you get these stripes here 00:54:49.820 --> 00:54:51.940 align:middle line:84% that really correspond to the formants. 00:54:51.940 --> 00:54:54.100 align:middle line:84% And so the reason that choice is made 00:54:54.100 --> 00:54:55.678 align:middle line:84% is because in the analysis of speech, 00:54:55.678 --> 00:54:57.720 align:middle line:84% you often don't care so much about the harmonics, 00:54:57.720 --> 00:54:58.980 align:middle line:84% but you really care about the formants 00:54:58.980 --> 00:55:00.840 align:middle line:84% because that differentiates phonemes. 00:55:00.840 --> 00:55:04.977 align:middle line:84% So that's why these look the way that they do. 00:55:04.977 --> 00:55:07.060 align:middle line:84% Here's a spectrogram that's made a little bit more 00:55:07.060 --> 00:55:09.020 align:middle line:90% like what you're used to. 00:55:09.020 --> 00:55:14.531 align:middle line:84% So this is just a recording that I made of two different vowels. 00:55:14.531 --> 00:55:15.198 align:middle line:90% [AUDIO PLAYBACK] 00:55:15.198 --> 00:55:19.680 align:middle line:90% - La, looh. 00:55:19.680 --> 00:55:24.087 align:middle line:90% 00:55:24.087 --> 00:55:24.670 align:middle line:90% [END PLAYBACK] 00:55:24.670 --> 00:55:25.470 align:middle line:84% JOSH MCDERMOTT: And so you can see 00:55:25.470 --> 00:55:27.670 align:middle line:84% that there is a fundamental frequency here 00:55:27.670 --> 00:55:30.430 align:middle line:90% that's around 250 hertz. 00:55:30.430 --> 00:55:34.710 align:middle line:84% You've got harmonics of that fundamental frequency. 00:55:34.710 --> 00:55:37.350 align:middle line:84% But the relative amplitude of those harmonics 00:55:37.350 --> 00:55:38.970 align:middle line:90% is pretty different. 00:55:38.970 --> 00:55:41.950 align:middle line:84% So the "ooh" sound kind of has all the energy down here. 00:55:41.950 --> 00:55:44.870 align:middle line:84% The "ahh" sound kind of has peaks there and there. 00:55:44.870 --> 00:55:47.510 align:middle line:84% But this is supposed to make the point 00:55:47.510 --> 00:55:51.350 align:middle line:84% that you can maintain the fundamental frequency 00:55:51.350 --> 00:55:53.830 align:middle line:84% and keep the pitch the same and still 00:55:53.830 --> 00:55:56.750 align:middle line:84% have the vowel sounds be totally different because the formants 00:55:56.750 --> 00:55:58.030 align:middle line:90% will be moving around. 00:55:58.030 --> 00:56:01.690 align:middle line:84% So in this case, the filter is changing, and the source is not. 00:56:01.690 --> 00:56:04.230 align:middle line:90% 00:56:04.230 --> 00:56:06.213 align:middle line:84% So that's kind of the story with vowels. 00:56:06.213 --> 00:56:08.130 align:middle line:84% They're made with an unrestricted vocal tract. 00:56:08.130 --> 00:56:10.150 align:middle line:84% They're characterized by the formants. 00:56:10.150 --> 00:56:13.670 align:middle line:84% That's the classical textbook story. 00:56:13.670 --> 00:56:15.950 align:middle line:84% Consonants are the other main type of phonemes, 00:56:15.950 --> 00:56:18.400 align:middle line:84% and consonants are generated when the vocal tract is 00:56:18.400 --> 00:56:21.280 align:middle line:90% restricted in some way. 00:56:21.280 --> 00:56:23.680 align:middle line:84% And the different types of consonants that you can make 00:56:23.680 --> 00:56:26.280 align:middle line:84% are distinguished by differences in what's called 00:56:26.280 --> 00:56:27.740 align:middle line:90% the place of articulation. 00:56:27.740 --> 00:56:30.840 align:middle line:84% So that's where the restriction is occurring. 00:56:30.840 --> 00:56:32.650 align:middle line:90% And so it could be dental. 00:56:32.650 --> 00:56:34.900 align:middle line:84% That refers to the restriction happening at the teeth. 00:56:34.900 --> 00:56:36.400 align:middle line:84% We'll see some examples in a second. 00:56:36.400 --> 00:56:39.080 align:middle line:84% Velar-- that means the roof of the mouth. 00:56:39.080 --> 00:56:42.480 align:middle line:84% They can also be distinguished by the manner of articulation. 00:56:42.480 --> 00:56:45.400 align:middle line:84% So the restriction can be complete. 00:56:45.400 --> 00:56:47.680 align:middle line:84% So in other words, the vocal tract-- 00:56:47.680 --> 00:56:50.620 align:middle line:84% the airflow will be completely cut off and then released. 00:56:50.620 --> 00:56:52.520 align:middle line:90% So that's a stop. 00:56:52.520 --> 00:56:56.160 align:middle line:84% You can get nasal articulation where the airflow is forced up 00:56:56.160 --> 00:56:58.320 align:middle line:90% through the nose. 00:56:58.320 --> 00:57:00.600 align:middle line:84% Fricatives are where the airflow kind of gets 00:57:00.600 --> 00:57:02.540 align:middle line:90% blown through a narrow gap. 00:57:02.540 --> 00:57:07.800 align:middle line:84% So that's like where you force the air through. 00:57:07.800 --> 00:57:10.100 align:middle line:84% And then they also differ in the voicing. 00:57:10.100 --> 00:57:12.140 align:middle line:84% So consonants can be voiced or unvoiced. 00:57:12.140 --> 00:57:15.960 align:middle line:84% And that refers to whether the vocal cords are vibrating 00:57:15.960 --> 00:57:17.920 align:middle line:90% at the start of the consonant. 00:57:17.920 --> 00:57:21.300 align:middle line:90% We'll see some examples of that. 00:57:21.300 --> 00:57:26.600 align:middle line:84% So here are examples of stops that 00:57:26.600 --> 00:57:28.800 align:middle line:84% differ in the place of articulation 00:57:28.800 --> 00:57:30.640 align:middle line:84% as well as whether they are voiced. 00:57:30.640 --> 00:57:34.420 align:middle line:90% So let's look at the first two. 00:57:34.420 --> 00:57:39.080 align:middle line:84% So "buh" and "puh"-- everybody go buh, puh, buh, puh. 00:57:39.080 --> 00:57:41.960 align:middle line:84% So you can tell when you were doing that, your lips were 00:57:41.960 --> 00:57:43.140 align:middle line:90% opening and closing. 00:57:43.140 --> 00:57:45.560 align:middle line:90% That's a labial stop-- 00:57:45.560 --> 00:57:47.740 align:middle line:90% buh puh, buh puh. 00:57:47.740 --> 00:57:50.460 align:middle line:84% So what causes "buh" and "puh" to sound different? 00:57:50.460 --> 00:57:52.040 align:middle line:84% Well, the difference is that when 00:57:52.040 --> 00:57:55.640 align:middle line:84% you say buh, buh, like at the moment 00:57:55.640 --> 00:57:59.360 align:middle line:84% when the stop is released, the vocal cords vibrate. 00:57:59.360 --> 00:58:03.000 align:middle line:84% Whereas with puh, puh, there's this little moment 00:58:03.000 --> 00:58:04.820 align:middle line:84% where the vocal cords are not vibrating, 00:58:04.820 --> 00:58:08.640 align:middle line:84% and so you just get turbulent airflow being forced out. 00:58:08.640 --> 00:58:10.520 align:middle line:84% And then the vocal cords are vibrating, 00:58:10.520 --> 00:58:14.040 align:middle line:84% and you get the periodic voiced signal coming through-- 00:58:14.040 --> 00:58:15.090 align:middle line:90% buh, ph. 00:58:15.090 --> 00:58:22.070 align:middle line:84% So let's do the second one-- deh, teh, deh, teh, deh, teh. 00:58:22.070 --> 00:58:23.470 align:middle line:90% So that's a dental stop. 00:58:23.470 --> 00:58:25.810 align:middle line:84% So your tongue is coming up against your teeth 00:58:25.810 --> 00:58:26.950 align:middle line:90% and then releasing. 00:58:26.950 --> 00:58:29.930 align:middle line:84% And again, though the difference between deh and teh 00:58:29.930 --> 00:58:32.190 align:middle line:84% is whether or not they are voiced-- 00:58:32.190 --> 00:58:34.370 align:middle line:84% so whether the vocal cords vibrate right 00:58:34.370 --> 00:58:39.050 align:middle line:84% at the start of the release or not. 00:58:39.050 --> 00:58:42.810 align:middle line:84% And so when you are learning how to talk, 00:58:42.810 --> 00:58:44.690 align:middle line:84% you're picking up on these distinctions that 00:58:44.690 --> 00:58:46.355 align:middle line:84% exist in your language and figuring out 00:58:46.355 --> 00:58:47.230 align:middle line:90% how to do this stuff. 00:58:47.230 --> 00:58:49.355 align:middle line:84% And then it all happens just kind of unconsciously. 00:58:49.355 --> 00:58:52.370 align:middle line:84% So it's like incredibly precise motor control of this stuff-- 00:58:52.370 --> 00:58:56.130 align:middle line:84% coordination between what your lips and mouth are doing 00:58:56.130 --> 00:58:58.370 align:middle line:84% and what the vocal folds are doing. 00:58:58.370 --> 00:59:03.730 align:middle line:84% Finally, let's do the third one-- guh, kuh, guh, kuh, 00:59:03.730 --> 00:59:05.870 align:middle line:90% guh, kuh. 00:59:05.870 --> 00:59:07.930 align:middle line:84% So there the stop is happening at the back 00:59:07.930 --> 00:59:09.290 align:middle line:90% of the roof of the mouth. 00:59:09.290 --> 00:59:11.250 align:middle line:84% And again, we get the distinction 00:59:11.250 --> 00:59:14.020 align:middle line:84% between voiced and unvoiced consonants. 00:59:14.020 --> 00:59:19.240 align:middle line:90% 00:59:19.240 --> 00:59:23.780 align:middle line:84% So this is making the same point that I was just saying earlier. 00:59:23.780 --> 00:59:25.340 align:middle line:84% So these kinds of spectrograms are 00:59:25.340 --> 00:59:28.740 align:middle line:84% conventional for speech analysis where 00:59:28.740 --> 00:59:31.860 align:middle line:84% they use small time windows that gives you coarse frequency 00:59:31.860 --> 00:59:32.800 align:middle line:90% resolution. 00:59:32.800 --> 00:59:34.833 align:middle line:84% So the horizontal stripes here are formants. 00:59:34.833 --> 00:59:36.500 align:middle line:84% And then the thing that you can see here 00:59:36.500 --> 00:59:39.060 align:middle line:84% that wasn't so evident on the ones we saw earlier 00:59:39.060 --> 00:59:42.300 align:middle line:84% is that the source, which are these vocal folds that 00:59:42.300 --> 00:59:49.267 align:middle line:84% are opening and closing, is evident as thin vertical lines. 00:59:49.267 --> 00:59:51.100 align:middle line:84% Each of those vertical lines, it corresponds 00:59:51.100 --> 00:59:54.700 align:middle line:84% to one pitch pulse-- so one opening and closing 00:59:54.700 --> 00:59:57.740 align:middle line:90% of the vocal cords. 00:59:57.740 --> 01:00:01.060 align:middle line:84% So it's kind of cool that you can look at it 01:00:01.060 --> 01:00:04.380 align:middle line:90% with these small time windows. 01:00:04.380 --> 01:00:06.140 align:middle line:84% And the periodicity of the source 01:00:06.140 --> 01:00:08.940 align:middle line:90% shows up in the time domain. 01:00:08.940 --> 01:00:11.372 align:middle line:84% Or you can look at it with longer time windows, 01:00:11.372 --> 01:00:13.830 align:middle line:84% and it would actually show up as harmonics in the frequency 01:00:13.830 --> 01:00:14.330 align:middle line:90% domain. 01:00:14.330 --> 01:00:17.170 align:middle line:90% 01:00:17.170 --> 01:00:19.630 align:middle line:84% So that's the thing to take away from this 01:00:19.630 --> 01:00:22.830 align:middle line:84% is that those small kind of thin vertical stripes 01:00:22.830 --> 01:00:24.070 align:middle line:90% are the voicing. 01:00:24.070 --> 01:00:28.150 align:middle line:84% That's the vocal cords oscillating periodically. 01:00:28.150 --> 01:00:29.710 align:middle line:84% Now this slide is supposed to show 01:00:29.710 --> 01:00:34.190 align:middle line:84% you the difference between voiced consonants and voiceless 01:00:34.190 --> 01:00:35.870 align:middle line:90% or unvoiced consonants. 01:00:35.870 --> 01:00:38.810 align:middle line:84% And that shows up at the very start of the consonant. 01:00:38.810 --> 01:00:41.230 align:middle line:84% So you can see that in all of the voiced cases, 01:00:41.230 --> 01:00:45.350 align:middle line:84% you see those pitch pulses kind of right from the get-go. 01:00:45.350 --> 01:00:49.190 align:middle line:84% Whereas in the unvoiced cases, you don't see the pitch pulses. 01:00:49.190 --> 01:00:51.050 align:middle line:84% Well, the time axis is not labeled here, 01:00:51.050 --> 01:00:53.250 align:middle line:84% but it would be like 40 milliseconds or something. 01:00:53.250 --> 01:00:57.090 align:middle line:84% It's a pretty short amount of time. 01:00:57.090 --> 01:00:59.650 align:middle line:84% So that's like voiced versus unvoiced. 01:00:59.650 --> 01:01:04.663 align:middle line:90% 01:01:04.663 --> 01:01:06.830 align:middle line:84% Another there are a couple other types of consonants 01:01:06.830 --> 01:01:08.720 align:middle line:90% that we'll talk about. 01:01:08.720 --> 01:01:13.380 align:middle line:84% Fricatives are cases where there's a partial restriction, 01:01:13.380 --> 01:01:14.560 align:middle line:90% a narrowing. 01:01:14.560 --> 01:01:17.400 align:middle line:84% And typically, there'll be some turbulent energy 01:01:17.400 --> 01:01:22.600 align:middle line:84% that gets forced through this thin aperture. 01:01:22.600 --> 01:01:24.680 align:middle line:90% So let's do "zuh" versus "suh"-- 01:01:24.680 --> 01:01:28.600 align:middle line:90% zuh, suh, zuh, suh. 01:01:28.600 --> 01:01:31.600 align:middle line:84% So there's a thin gap between your teeth. 01:01:31.600 --> 01:01:34.380 align:middle line:90% But they differ in voicing. 01:01:34.380 --> 01:01:36.380 align:middle line:90% So zuh is voiced. 01:01:36.380 --> 01:01:38.440 align:middle line:84% Your vocal cords vibrate from the get-go. 01:01:38.440 --> 01:01:40.458 align:middle line:90% Suh, suh is unvoiced. 01:01:40.458 --> 01:01:43.000 align:middle line:84% There's this little moment at the start where the vocal cords 01:01:43.000 --> 01:01:44.400 align:middle line:90% aren't doing their thing. 01:01:44.400 --> 01:01:47.580 align:middle line:84% So that's an example of fricatives, 01:01:47.580 --> 01:01:49.920 align:middle line:84% where air is forced through a narrow gap between two 01:01:49.920 --> 01:01:51.020 align:middle line:90% articulators. 01:01:51.020 --> 01:01:53.360 align:middle line:84% And then there are nasal consonants 01:01:53.360 --> 01:01:56.680 align:middle line:84% where air is forced up through the nose, and that kind of just 01:01:56.680 --> 01:01:59.760 align:middle line:90% makes it sound a certain way. 01:01:59.760 --> 01:02:03.000 align:middle line:90% So let's do "muh," "nuh"-- 01:02:03.000 --> 01:02:08.552 align:middle line:90% muh, nuh, muh, nuh, muh, nuh. 01:02:08.552 --> 01:02:10.010 align:middle line:84% And so the difference between those 01:02:10.010 --> 01:02:14.268 align:middle line:84% is again where the stoppage in the mouth is occurring. 01:02:14.268 --> 01:02:16.310 align:middle line:84% And then the air gets forced up through the nose. 01:02:16.310 --> 01:02:17.910 align:middle line:90% And that changes the resonances. 01:02:17.910 --> 01:02:21.110 align:middle line:84% It just creates different sounds. 01:02:21.110 --> 01:02:23.610 align:middle line:84% So those are examples of phonemes 01:02:23.610 --> 01:02:25.450 align:middle line:90% and how they are produced. 01:02:25.450 --> 01:02:29.630 align:middle line:84% So you can see a lot of structure 01:02:29.630 --> 01:02:32.530 align:middle line:84% if you look at spectrograms of speech. 01:02:32.530 --> 01:02:36.530 align:middle line:84% So this is an old-school spectrogram 01:02:36.530 --> 01:02:39.570 align:middle line:84% of somebody saying I can see you. 01:02:39.570 --> 01:02:41.710 align:middle line:84% So this is the fricative of the S. 01:02:41.710 --> 01:02:45.010 align:middle line:84% You can see this broadband high-frequency noise. 01:02:45.010 --> 01:02:47.370 align:middle line:90% Here are the vowels-- eeh, euu. 01:02:47.370 --> 01:02:48.190 align:middle line:90% These are formants. 01:02:48.190 --> 01:02:53.450 align:middle line:84% You can see that they change as you move from the eeh, euu. 01:02:53.450 --> 01:02:56.130 align:middle line:84% The other thing that's kind of important to note 01:02:56.130 --> 01:03:00.730 align:middle line:84% about spectrograms of speech is that the things that we perceive 01:03:00.730 --> 01:03:03.570 align:middle line:84% to be words are typically not well-separated 01:03:03.570 --> 01:03:06.580 align:middle line:90% in the actual signal. 01:03:06.580 --> 01:03:10.520 align:middle line:84% So "I can see you" has four words in it. 01:03:10.520 --> 01:03:14.400 align:middle line:90% 01:03:14.400 --> 01:03:17.440 align:middle line:84% The gap between the C and the U is just not there in the signal. 01:03:17.440 --> 01:03:19.500 align:middle line:90% There's this continuous thing. 01:03:19.500 --> 01:03:22.820 align:middle line:84% And so that's kind of an important kind of hard problem 01:03:22.820 --> 01:03:24.980 align:middle line:84% that also has to get solved and figuring out 01:03:24.980 --> 01:03:28.940 align:middle line:84% how this continuous signal actually consists 01:03:28.940 --> 01:03:32.700 align:middle line:90% of these discrete words. 01:03:32.700 --> 01:03:34.480 align:middle line:90% So I just said all this stuff. 01:03:34.480 --> 01:03:38.300 align:middle line:90% 01:03:38.300 --> 01:03:41.620 align:middle line:84% So whispering is another interesting thing 01:03:41.620 --> 01:03:43.340 align:middle line:84% that happens with speech. (WHISPERING) 01:03:43.340 --> 01:03:45.220 align:middle line:90% So can talk like this. 01:03:45.220 --> 01:03:47.800 align:middle line:84% Everyone can probably understand me. 01:03:47.800 --> 01:03:49.820 align:middle line:90% But it's very different. 01:03:49.820 --> 01:03:51.900 align:middle line:84% So whispering-- what happens with whispering 01:03:51.900 --> 01:03:54.140 align:middle line:84% is that the vocal cords don't vibrate. 01:03:54.140 --> 01:03:58.380 align:middle line:90% So how does that work? 01:03:58.380 --> 01:04:01.540 align:middle line:84% How would you be able to understand me talk when 01:04:01.540 --> 01:04:05.430 align:middle line:90% my vocal cords don't vibrate? 01:04:05.430 --> 01:04:08.270 align:middle line:90% Anybody have any idea? 01:04:08.270 --> 01:04:08.790 align:middle line:90% Yeah. 01:04:08.790 --> 01:04:12.390 align:middle line:84% STUDENT: You can hear the noise in the air. 01:04:12.390 --> 01:04:13.735 align:middle line:90% It's not going to be vocal. 01:04:13.735 --> 01:04:15.610 align:middle line:84% JOSH MCDERMOTT: Yeah, you can hear the noise, 01:04:15.610 --> 01:04:20.590 align:middle line:84% but how would you be able to detect the structural features 01:04:20.590 --> 01:04:23.390 align:middle line:90% of these phonemes? 01:04:23.390 --> 01:04:23.950 align:middle line:90% Yeah. 01:04:23.950 --> 01:04:26.543 align:middle line:84% STUDENT: By seeing you speak, like seeing-- 01:04:26.543 --> 01:04:28.710 align:middle line:84% JOSH MCDERMOTT: (WHISPERING) But I can go like this, 01:04:28.710 --> 01:04:32.670 align:middle line:84% and you can still understand me just fine. 01:04:32.670 --> 01:04:36.790 align:middle line:84% STUDENT: The noise is still shaped by the resonances 01:04:36.790 --> 01:04:39.450 align:middle line:90% of your mouth area. 01:04:39.450 --> 01:04:39.950 align:middle line:90% Yeah. 01:04:39.950 --> 01:04:42.283 align:middle line:84% JOSH MCDERMOTT: So that's probably like a big part of it 01:04:42.283 --> 01:04:46.490 align:middle line:84% is that the filters are doing more or less the same thing. 01:04:46.490 --> 01:04:50.430 align:middle line:84% And so you can probably detect the formants. 01:04:50.430 --> 01:04:52.470 align:middle line:84% They're just excited by a noise signal 01:04:52.470 --> 01:04:55.470 align:middle line:84% rather than by a periodic harmonic. 01:04:55.470 --> 01:04:57.830 align:middle line:84% But William is correct that it definitely 01:04:57.830 --> 01:05:00.037 align:middle line:84% helps to be able to see what people are doing 01:05:00.037 --> 01:05:01.120 align:middle line:90% with their mouths as well. 01:05:01.120 --> 01:05:02.780 align:middle line:84% And we'll see some examples of that. 01:05:02.780 --> 01:05:05.780 align:middle line:90% 01:05:05.780 --> 01:05:08.720 align:middle line:84% So naively, you might imagine that you 01:05:08.720 --> 01:05:11.600 align:middle line:84% could build detectors for the phonemes, 01:05:11.600 --> 01:05:14.040 align:middle line:84% and speech reception would be solved. 01:05:14.040 --> 01:05:17.280 align:middle line:84% And the challenge that's classically 01:05:17.280 --> 01:05:19.120 align:middle line:84% associated with speech perception 01:05:19.120 --> 01:05:20.920 align:middle line:84% is that phonemes are produced differently 01:05:20.920 --> 01:05:22.300 align:middle line:90% depending on lots of factors. 01:05:22.300 --> 01:05:25.200 align:middle line:84% And so this results in a constancy problem, 01:05:25.200 --> 01:05:27.820 align:middle line:84% a lot like the others that we often encounter in perception, 01:05:27.820 --> 01:05:31.480 align:middle line:84% like the problem of recognizing objects across viewpoints. 01:05:31.480 --> 01:05:34.080 align:middle line:84% And so one factor that's kind of important in this 01:05:34.080 --> 01:05:36.160 align:middle line:90% is what's called coarticulation. 01:05:36.160 --> 01:05:38.240 align:middle line:84% And this refers to the fact that phonemes 01:05:38.240 --> 01:05:40.680 align:middle line:84% are produced differently depending on what 01:05:40.680 --> 01:05:42.480 align:middle line:90% comes before and after them. 01:05:42.480 --> 01:05:45.880 align:middle line:84% And the crazy thing is that even though 01:05:45.880 --> 01:05:48.800 align:middle line:84% the underlying acoustic signature that corresponds 01:05:48.800 --> 01:05:51.600 align:middle line:84% to the phoneme kind of differs, they sound like the same thing 01:05:51.600 --> 01:05:53.090 align:middle line:90% to us. 01:05:53.090 --> 01:05:54.840 align:middle line:84% So somehow, they're recognized as the same 01:05:54.840 --> 01:05:56.900 align:middle line:84% despite producing different patterns of energy. 01:05:56.900 --> 01:06:04.080 align:middle line:84% And so this is an example where We have three syllables that 01:06:04.080 --> 01:06:08.880 align:middle line:84% start with the same consonant-- so "baah," "boo," "bee" 01:06:08.880 --> 01:06:12.840 align:middle line:84% or "dah," "doo," "dee," or "gah," "goo," "gee." 01:06:12.840 --> 01:06:16.120 align:middle line:84% And these are spectrograms of each of those syllables. 01:06:16.120 --> 01:06:18.680 align:middle line:84% And the point is that-- so the consonant kind 01:06:18.680 --> 01:06:20.500 align:middle line:84% of lives at the start of the syllable. 01:06:20.500 --> 01:06:22.880 align:middle line:84% And if you look at what is actually 01:06:22.880 --> 01:06:26.680 align:middle line:84% going on in that initial little bit of syllable, 01:06:26.680 --> 01:06:29.860 align:middle line:84% it's quite different in the three cases 01:06:29.860 --> 01:06:32.240 align:middle line:84% because it varies depending on what vowel is going 01:06:32.240 --> 01:06:33.660 align:middle line:90% to come after the consonant. 01:06:33.660 --> 01:06:36.660 align:middle line:84% And that's true in all of these cases. 01:06:36.660 --> 01:06:40.200 align:middle line:90% 01:06:40.200 --> 01:06:44.440 align:middle line:84% Here's another case that's pretty interesting. 01:06:44.440 --> 01:06:48.280 align:middle line:84% So these are spectrograms of somebody saying "eebah," 01:06:48.280 --> 01:06:51.760 align:middle line:90% "oobah," "eedah," "oodah." 01:06:51.760 --> 01:06:56.960 align:middle line:84% And the point of this is that the acoustic signature 01:06:56.960 --> 01:07:01.570 align:middle line:84% of the "buh" sound when you're seeing eebah looks a lot like 01:07:01.570 --> 01:07:04.490 align:middle line:84% the acoustic signature of the duh sound when you're seeing 01:07:04.490 --> 01:07:07.530 align:middle line:90% oodah. 01:07:07.530 --> 01:07:12.010 align:middle line:84% So the physical signal in this little bit of sound 01:07:12.010 --> 01:07:14.010 align:middle line:84% is pretty similar for the buh and the dub, 01:07:14.010 --> 01:07:18.450 align:middle line:84% but eebah sounds like a buh, and oodah sounds like a duh. 01:07:18.450 --> 01:07:19.290 align:middle line:90% Yeah. 01:07:19.290 --> 01:07:21.970 align:middle line:84% STUDENT: Is this only for consonants? 01:07:21.970 --> 01:07:26.750 align:middle line:84% Are vowels also changing based on preceding? 01:07:26.750 --> 01:07:28.250 align:middle line:84% JOSH MCDERMOTT: Most of the examples 01:07:28.250 --> 01:07:30.083 align:middle line:84% that I know about are with consonants, yeah. 01:07:30.083 --> 01:07:33.610 align:middle line:84% I'm not aware of this happening a whole lot with vowels, 01:07:33.610 --> 01:07:36.370 align:middle line:84% although there may be some effects of that too. 01:07:36.370 --> 01:07:38.130 align:middle line:90% Yeah. 01:07:38.130 --> 01:07:38.630 align:middle line:90% Yeah. 01:07:38.630 --> 01:07:41.430 align:middle line:84% So co-articulation-- phonemes are produced differently. 01:07:41.430 --> 01:07:44.450 align:middle line:84% So they have distinct acoustic signatures 01:07:44.450 --> 01:07:47.250 align:middle line:84% depending on what comes before and after them. 01:07:47.250 --> 01:07:51.490 align:middle line:84% And the reason for this, we think 01:07:51.490 --> 01:07:54.810 align:middle line:84% it really has to do with mechanical constraints 01:07:54.810 --> 01:07:59.440 align:middle line:84% on how quickly you can change your vocal tract shape. 01:07:59.440 --> 01:08:02.322 align:middle line:84% So you got to get from point A to point B 01:08:02.322 --> 01:08:04.280 align:middle line:84% if you want to make one sound and then another. 01:08:04.280 --> 01:08:05.655 align:middle line:84% And that kind of forces you to go 01:08:05.655 --> 01:08:08.300 align:middle line:84% through certain trajectories in the kind of space 01:08:08.300 --> 01:08:10.500 align:middle line:84% of the state of your vocal tract. 01:08:10.500 --> 01:08:12.913 align:middle line:84% And you can't just make arbitrary 01:08:12.913 --> 01:08:14.580 align:middle line:84% sounds with your vocal tract, and that's 01:08:14.580 --> 01:08:16.620 align:middle line:90% probably why this happens. 01:08:16.620 --> 01:08:18.500 align:middle line:84% But there's lots of other factors 01:08:18.500 --> 01:08:22.420 align:middle line:84% that kind of affect the speech signal. 01:08:22.420 --> 01:08:23.859 align:middle line:90% So there's prosody. 01:08:23.859 --> 01:08:26.960 align:middle line:84% So you can vary stress to convey emphasis. 01:08:26.960 --> 01:08:32.340 align:middle line:90% 01:08:32.340 --> 01:08:36.100 align:middle line:90% We can say permit or permit. 01:08:36.100 --> 01:08:38.380 align:middle line:84% So you need a permit to skip class, 01:08:38.380 --> 01:08:40.225 align:middle line:90% or I permit you to skip class. 01:08:40.225 --> 01:08:41.600 align:middle line:84% Those mean very different things. 01:08:41.600 --> 01:08:44.819 align:middle line:84% And it's conveyed by the stress on the different parts 01:08:44.819 --> 01:08:46.460 align:middle line:90% of the word. 01:08:46.460 --> 01:08:48.380 align:middle line:90% There's intonation. 01:08:48.380 --> 01:08:50.460 align:middle line:90% Dad wants me to mow the lawn. 01:08:50.460 --> 01:08:52.500 align:middle line:90% Dad wants me to mow the lawn. 01:08:52.500 --> 01:08:53.990 align:middle line:90% Those mean different things. 01:08:53.990 --> 01:08:57.149 align:middle line:84% And it results in the speech having 01:08:57.149 --> 01:08:58.810 align:middle line:84% different acoustic characteristics. 01:08:58.810 --> 01:09:00.602 align:middle line:84% There's also differences in emotional state 01:09:00.602 --> 01:09:03.590 align:middle line:84% that can change the way people talk. 01:09:03.590 --> 01:09:05.090 align:middle line:84% Different speakers sound different, 01:09:05.090 --> 01:09:07.790 align:middle line:90% so people vary in accents. 01:09:07.790 --> 01:09:09.742 align:middle line:90% It's a function of gender, age. 01:09:09.742 --> 01:09:11.950 align:middle line:84% So the point is that the same phoneme can be realized 01:09:11.950 --> 01:09:14.710 align:middle line:90% in many different ways. 01:09:14.710 --> 01:09:18.670 align:middle line:84% And sometimes the variation in phoneme acoustics 01:09:18.670 --> 01:09:20.370 align:middle line:84% produces ambiguity in what is said. 01:09:20.370 --> 01:09:22.287 align:middle line:84% So this was an example that everybody probably 01:09:22.287 --> 01:09:24.990 align:middle line:84% knows because it kind of went viral like a while ago. 01:09:24.990 --> 01:09:29.750 align:middle line:84% So this speech signal that people disagree about. 01:09:29.750 --> 01:09:31.810 align:middle line:90% I'll play it for you. 01:09:31.810 --> 01:09:32.950 align:middle line:90% [AUDIO PLAYBACK] 01:09:32.950 --> 01:09:45.847 align:middle line:84% - Laurel, Laurel, Laurel, Laurel. 01:09:45.847 --> 01:09:46.430 align:middle line:90% [END PLAYBACK] 01:09:46.430 --> 01:09:49.149 align:middle line:84% JOSH MCDERMOTT: How many people think that says Yanny? 01:09:49.149 --> 01:09:51.402 align:middle line:84% How many people think it says Laurel? 01:09:51.402 --> 01:09:52.819 align:middle line:84% Yeah, so it's about half and half. 01:09:52.819 --> 01:09:56.600 align:middle line:90% 01:09:56.600 --> 01:09:59.640 align:middle line:84% And the presumptive explanation here 01:09:59.640 --> 01:10:04.120 align:middle line:84% is that there is some ambiguity in what was said. 01:10:04.120 --> 01:10:09.200 align:middle line:84% And different people interpreted it a little bit differently. 01:10:09.200 --> 01:10:10.760 align:middle line:84% So there was lots of experiments that 01:10:10.760 --> 01:10:13.343 align:middle line:84% were done on this like when this was first kind of discovered. 01:10:13.343 --> 01:10:15.240 align:middle line:84% This was like now five years ago. 01:10:15.240 --> 01:10:19.820 align:middle line:84% And there are definite effects of filtering the stimulus. 01:10:19.820 --> 01:10:23.160 align:middle line:84% So for instance, if you low-pass filter the stimulus, 01:10:23.160 --> 01:10:27.078 align:middle line:84% people are a lot more likely to say that they hear Laurel. 01:10:27.078 --> 01:10:28.620 align:middle line:84% If you high-pass filter the stimulus, 01:10:28.620 --> 01:10:31.160 align:middle line:84% people are more likely to say that they hear Yanny. 01:10:31.160 --> 01:10:33.520 align:middle line:84% But some people consistently hear one 01:10:33.520 --> 01:10:36.640 align:middle line:90% or the other, no matter what. 01:10:36.640 --> 01:10:39.160 align:middle line:84% And I think the explanation for why for the individual 01:10:39.160 --> 01:10:42.040 align:middle line:84% differences remains a little bit unclear. 01:10:42.040 --> 01:10:47.000 align:middle line:84% So here's just a continuum where the-- so I think 01:10:47.000 --> 01:10:50.550 align:middle line:90% this is the high-low ratio. 01:10:50.550 --> 01:10:52.250 align:middle line:84% So this one is probably pretty low. 01:10:52.250 --> 01:10:53.370 align:middle line:90% - Laurel. 01:10:53.370 --> 01:10:54.770 align:middle line:84% JOSH MCDERMOTT: So that should mostly sound like Laurel 01:10:54.770 --> 01:10:55.437 align:middle line:90% to a lot of you. 01:10:55.437 --> 01:10:56.370 align:middle line:90% Is that right? 01:10:56.370 --> 01:10:58.810 align:middle line:84% How many people say that's Laurel? 01:10:58.810 --> 01:11:01.150 align:middle line:84% Now, let's just do the controlled experiment. 01:11:01.150 --> 01:11:02.010 align:middle line:90% - Yanny. 01:11:02.010 --> 01:11:04.170 align:middle line:84% JOSH MCDERMOTT: How many people say that's Laurel? 01:11:04.170 --> 01:11:07.950 align:middle line:84% Nobody-- so everybody thinks this one's Yanny, 01:11:07.950 --> 01:11:13.090 align:middle line:84% and a lot of people think that one is Laurel. 01:11:13.090 --> 01:11:15.490 align:middle line:84% This is a sequence where the thing 01:11:15.490 --> 01:11:17.965 align:middle line:90% is the cutoff of the filter. 01:11:17.965 --> 01:11:19.590 align:middle line:84% I don't know exactly how they did this, 01:11:19.590 --> 01:11:21.630 align:middle line:84% but they're changing the frequency content. 01:11:21.630 --> 01:11:24.010 align:middle line:90% And it goes from low to high. 01:11:24.010 --> 01:11:26.450 align:middle line:84% So that means that it should tend 01:11:26.450 --> 01:11:27.970 align:middle line:84% to sound more like Laurel initially 01:11:27.970 --> 01:11:30.130 align:middle line:84% and then eventually maybe transition. 01:11:30.130 --> 01:11:34.290 align:middle line:84% - Laurel, Laurel, Laurel, Laurel, Laurel, Laurel, Laurel, 01:11:34.290 --> 01:11:36.812 align:middle line:90% Laurel, Laurel, Laurel, Laurel. 01:11:36.812 --> 01:11:38.770 align:middle line:84% JOSH MCDERMOTT: Never transitioned for me-- how 01:11:38.770 --> 01:11:39.570 align:middle line:90% about you? 01:11:39.570 --> 01:11:40.810 align:middle line:90% Laurel all the time? 01:11:40.810 --> 01:11:41.350 align:middle line:90% Some of you. 01:11:41.350 --> 01:11:42.830 align:middle line:90% Now, here's the reverse. 01:11:42.830 --> 01:11:46.174 align:middle line:84% - Laurel, Laurel, Laurel, Laurel, Laurel, Laurel, Laurel, 01:11:46.174 --> 01:11:50.940 align:middle line:84% Laurel, Laurel, Laurel, Laurel, Laurel, Laurel. 01:11:50.940 --> 01:11:55.860 align:middle line:84% JOSH MCDERMOTT: So what you probably experience here 01:11:55.860 --> 01:11:58.800 align:middle line:84% is that the perception of these two continua is different. 01:11:58.800 --> 01:12:02.440 align:middle line:84% And this illustrates something called hysteresis. 01:12:02.440 --> 01:12:06.460 align:middle line:84% So hysteresis refers to the fact that there 01:12:06.460 --> 01:12:09.080 align:middle line:84% tends to be some persistence of the perceptual interpretation. 01:12:09.080 --> 01:12:10.660 align:middle line:84% So in other words, if you started out 01:12:10.660 --> 01:12:11.860 align:middle line:84% hearing the thing as Laurel, you'll 01:12:11.860 --> 01:12:13.440 align:middle line:84% tend to hear it as Laurel for a while. 01:12:13.440 --> 01:12:14.800 align:middle line:84% If you start out hearing the thing as Yanny, 01:12:14.800 --> 01:12:16.633 align:middle line:84% you'll tend to hear it as Yanny for a while. 01:12:16.633 --> 01:12:20.940 align:middle line:84% And so this is a graph that shows the proportion of times 01:12:20.940 --> 01:12:25.820 align:middle line:84% that people respond with the thing being Laurel. 01:12:25.820 --> 01:12:31.580 align:middle line:84% And this is as a function of the high-to-low ratio. 01:12:31.580 --> 01:12:35.580 align:middle line:84% And what this is showing is that-- so there's some people 01:12:35.580 --> 01:12:39.500 align:middle line:84% here-- this is the blue line-- who basically always hear 01:12:39.500 --> 01:12:43.287 align:middle line:84% Laurel, kind of independent of where the cutoff is set. 01:12:43.287 --> 01:12:45.620 align:middle line:84% There's some people, as the yellow line, who pretty much 01:12:45.620 --> 01:12:47.750 align:middle line:90% always say they hear Yanny. 01:12:47.750 --> 01:12:49.870 align:middle line:84% But then there's a group, a very substantial group 01:12:49.870 --> 01:12:56.470 align:middle line:84% of people, that transition from hearing Laurel to Yanny. 01:12:56.470 --> 01:13:00.110 align:middle line:84% And this is a plot that shows how confident people are 01:13:00.110 --> 01:13:00.810 align:middle line:90% in the judgment. 01:13:00.810 --> 01:13:03.050 align:middle line:84% And the point is that people are very confident. 01:13:03.050 --> 01:13:04.630 align:middle line:84% So you're like, usually pretty sure 01:13:04.630 --> 01:13:06.130 align:middle line:84% that it's either Laurel or Yanny, 01:13:06.130 --> 01:13:08.130 align:middle line:84% and then it just kind of switches at some point. 01:13:08.130 --> 01:13:09.932 align:middle line:84% So there are these occasional kind 01:13:09.932 --> 01:13:11.390 align:middle line:84% of ambiguities in the speech signal 01:13:11.390 --> 01:13:13.810 align:middle line:84% that kind come out in these particular ways. 01:13:13.810 --> 01:13:16.670 align:middle line:84% And now that there's social media and the internet, 01:13:16.670 --> 01:13:19.003 align:middle line:84% and it's a lot easier to discover these things 01:13:19.003 --> 01:13:20.670 align:middle line:84% because people start talking about them. 01:13:20.670 --> 01:13:22.325 align:middle line:90% And then they become known. 01:13:22.325 --> 01:13:23.950 align:middle line:84% And we don't so far actually have a way 01:13:23.950 --> 01:13:26.450 align:middle line:84% to discover these things automatically. 01:13:26.450 --> 01:13:30.350 align:middle line:84% They just kind of happen by accident. 01:13:30.350 --> 01:13:33.910 align:middle line:84% So the point is that speech perception 01:13:33.910 --> 01:13:36.590 align:middle line:84% is thought to be a challenging perceptual problem because 01:13:36.590 --> 01:13:41.270 align:middle line:84% of all of the variation in how particular phonemes are produced 01:13:41.270 --> 01:13:44.930 align:middle line:90% across conditions. 01:13:44.930 --> 01:13:48.960 align:middle line:84% So what are some potential solutions to this? 01:13:48.960 --> 01:13:51.080 align:middle line:84% Well, it's often thought that there are probably 01:13:51.080 --> 01:13:54.560 align:middle line:84% some phonemes that are pretty invariant-- that is, 01:13:54.560 --> 01:13:56.000 align:middle line:84% whose acoustic signature is fairly 01:13:56.000 --> 01:13:58.000 align:middle line:90% consistent across conditions. 01:13:58.000 --> 01:14:00.820 align:middle line:84% So stops pretty much always produce bursts of energy. 01:14:00.820 --> 01:14:02.880 align:middle line:84% Turbulence are typically associated-- sorry, 01:14:02.880 --> 01:14:05.172 align:middle line:84% fricatives are typically associated with turbulence, so 01:14:05.172 --> 01:14:06.480 align:middle line:90% broad-spectrum energy. 01:14:06.480 --> 01:14:08.777 align:middle line:84% Vowels usually have steady-state formants, 01:14:08.777 --> 01:14:10.360 align:middle line:84% and the relations between the formants 01:14:10.360 --> 01:14:12.640 align:middle line:90% is often pretty consistent. 01:14:12.640 --> 01:14:17.200 align:middle line:84% Nasals are also fairly consistent. 01:14:17.200 --> 01:14:20.620 align:middle line:84% There's also other things that have been proposed, though, 01:14:20.620 --> 01:14:24.345 align:middle line:84% and one of the most famous ideas that 01:14:24.345 --> 01:14:25.720 align:middle line:84% has come out of speech perception 01:14:25.720 --> 01:14:27.740 align:middle line:84% is this idea of categorical perception. 01:14:27.740 --> 01:14:32.080 align:middle line:84% And this is the idea that we learn and impose categories 01:14:32.080 --> 01:14:36.360 align:middle line:84% on physically continuous stimuli. 01:14:36.360 --> 01:14:38.360 align:middle line:84% So we'll do a quick demonstration of this 01:14:38.360 --> 01:14:41.480 align:middle line:90% before we break. 01:14:41.480 --> 01:14:44.440 align:middle line:84% And the way that this works is we'll take two phonemes, 01:14:44.440 --> 01:14:47.040 align:middle line:90% in this case "cah" and "gah." 01:14:47.040 --> 01:14:52.260 align:middle line:84% So those differ in voicing-- cah, gah, cah, gah. 01:14:52.260 --> 01:14:54.320 align:middle line:84% And you can define that difference 01:14:54.320 --> 01:14:56.480 align:middle line:84% in terms of the voice onset time. 01:14:56.480 --> 01:15:00.880 align:middle line:84% That is how long after the onset of the consonant 01:15:00.880 --> 01:15:03.700 align:middle line:90% does the voicing kick in? 01:15:03.700 --> 01:15:06.220 align:middle line:84% And that will be in like tens of milliseconds, typically. 01:15:06.220 --> 01:15:09.320 align:middle line:90% 01:15:09.320 --> 01:15:11.800 align:middle line:84% And we can artificially create a continuum 01:15:11.800 --> 01:15:15.600 align:middle line:84% by manipulating the voice onset time using a speech synthesizer. 01:15:15.600 --> 01:15:18.860 align:middle line:84% So when the voice onset time is small, the thing will be voiced. 01:15:18.860 --> 01:15:20.780 align:middle line:84% When it's large, it will be unvoiced. 01:15:20.780 --> 01:15:23.440 align:middle line:84% So we can create a physical continuum of stimuli 01:15:23.440 --> 01:15:25.560 align:middle line:84% where the voice onset time is going to vary 01:15:25.560 --> 01:15:28.000 align:middle line:90% from 0 to 80 milliseconds. 01:15:28.000 --> 01:15:29.780 align:middle line:84% And then we can ask you what you hear. 01:15:29.780 --> 01:15:32.720 align:middle line:84% And the phenomenon of categorical perception 01:15:32.720 --> 01:15:36.780 align:middle line:84% is that people tend to hear one phoneme or the other, 01:15:36.780 --> 01:15:39.580 align:middle line:84% even though you're drawing these things from a continuum. 01:15:39.580 --> 01:15:41.490 align:middle line:84% So these are stimuli that are drawn 01:15:41.490 --> 01:15:45.650 align:middle line:84% from the continuum that will be presented in random order. 01:15:45.650 --> 01:15:47.750 align:middle line:90% So just listen to these. 01:15:47.750 --> 01:15:52.650 align:middle line:84% And so I'm pretty sure this is the labeling of the stimuli, 01:15:52.650 --> 01:15:55.712 align:middle line:84% and then this is telling you what people normally hear. 01:15:55.712 --> 01:15:57.170 align:middle line:84% But if you don't want to be biased, 01:15:57.170 --> 01:15:59.010 align:middle line:90% just kind of close your eyes. 01:15:59.010 --> 01:15:59.976 align:middle line:90% [AUDIO PLAYBACK] 01:15:59.976 --> 01:16:11.090 align:middle line:84% - Gah, kah, kah, gah, gah, gah, kah, kah. 01:16:11.090 --> 01:16:12.690 align:middle line:90% [END PLAYBACK] 01:16:12.690 --> 01:16:15.650 align:middle line:84% JOSH MCDERMOTT: So this is the answer key here. 01:16:15.650 --> 01:16:18.115 align:middle line:84% And that was fairly consistent with what I heard. 01:16:18.115 --> 01:16:19.990 align:middle line:84% So there's maybe one or two that's uncertain, 01:16:19.990 --> 01:16:21.910 align:middle line:84% but people are usually fairly confident. 01:16:21.910 --> 01:16:25.770 align:middle line:84% And so if you ask people to categorize the phoneme as being 01:16:25.770 --> 01:16:29.010 align:middle line:84% either gah or kah and you measure the categorization 01:16:29.010 --> 01:16:32.950 align:middle line:84% judgments as a function of the voice onset timing, 01:16:32.950 --> 01:16:35.850 align:middle line:84% you find that there's this very rapid transition between them 01:16:35.850 --> 01:16:37.930 align:middle line:84% saying that they hear gah and then switching 01:16:37.930 --> 01:16:41.260 align:middle line:90% to saying that they hear kah. 01:16:41.260 --> 01:16:44.660 align:middle line:84% So that's one feature of categorical perception. 01:16:44.660 --> 01:16:50.540 align:middle line:84% The second is this idea that discrimination will be 01:16:50.540 --> 01:16:53.180 align:middle line:90% best at the category boundary. 01:16:53.180 --> 01:16:55.620 align:middle line:84% And so the kind of naive idea around that 01:16:55.620 --> 01:16:59.300 align:middle line:84% is that if you've taken this speech signal 01:16:59.300 --> 01:17:02.460 align:middle line:84% and mapped it onto a category, then 01:17:02.460 --> 01:17:05.020 align:middle line:84% if I give you two examples that are 01:17:05.020 --> 01:17:07.500 align:middle line:84% kind of within that same category, 01:17:07.500 --> 01:17:10.418 align:middle line:84% if you've thrown out like the physical differences 01:17:10.418 --> 01:17:12.960 align:middle line:84% between them and you're just representing them as a category, 01:17:12.960 --> 01:17:14.100 align:middle line:84% then it should be difficult to tell 01:17:14.100 --> 01:17:15.850 align:middle line:84% the difference between those two examples, 01:17:15.850 --> 01:17:17.620 align:middle line:84% even though they're physically distinct. 01:17:17.620 --> 01:17:21.140 align:middle line:84% And so the idea is that we can do an experiment where 01:17:21.140 --> 01:17:24.100 align:middle line:84% we measure discrimination between pairs of stimuli 01:17:24.100 --> 01:17:25.700 align:middle line:90% along this continuum. 01:17:25.700 --> 01:17:28.515 align:middle line:84% The category boundary's here kind of in the middle. 01:17:28.515 --> 01:17:30.640 align:middle line:84% And the idea is that if you're within the category, 01:17:30.640 --> 01:17:34.020 align:middle line:84% so over here or over here, your discrimination will be poor. 01:17:34.020 --> 01:17:37.640 align:middle line:84% But if the pair of points straddles the category boundary, 01:17:37.640 --> 01:17:39.010 align:middle line:90% your discrimination is good. 01:17:39.010 --> 01:17:40.910 align:middle line:84% And so this is the discrimination function 01:17:40.910 --> 01:17:42.350 align:middle line:90% that would be predicted. 01:17:42.350 --> 01:17:46.110 align:middle line:84% So those are the two characteristics 01:17:46.110 --> 01:17:48.330 align:middle line:84% of classical categorical perception. 01:17:48.330 --> 01:17:51.910 align:middle line:84% So the first is steep categorization functions. 01:17:51.910 --> 01:17:54.110 align:middle line:84% So if you ask people to categorize the stimulus, 01:17:54.110 --> 01:17:55.590 align:middle line:84% there's this very rapid transition 01:17:55.590 --> 01:17:58.650 align:middle line:84% from people hearing it as one thing to hearing it as another. 01:17:58.650 --> 01:18:02.050 align:middle line:84% And the second is that if you do a discrimination experiment, 01:18:02.050 --> 01:18:05.230 align:middle line:84% discrimination is best at a category boundary. 01:18:05.230 --> 01:18:10.830 align:middle line:84% And when we resume next week, I'll 01:18:10.830 --> 01:18:12.470 align:middle line:84% show you some empirical data that's 01:18:12.470 --> 01:18:14.590 align:middle line:90% kind of in support of this. 01:18:14.590 --> 01:18:16.950 align:middle line:84% And then we will finish talking about speech perception. 01:18:16.950 --> 01:18:20.350 align:middle line:90% 01:18:20.350 --> 01:18:22.560 align:middle line:90% Have a good weekend. 01:18:22.560 --> 01:18:37.000 align:middle line:90%