The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
HYNEK HERMANSKY: So we have this wanted information
and unwanted information.
Not-- I call-- unwanted information noise
and wanted information signal.
Not all noises are created equal.
There are some noises which are partially understood,
and I'm claim this is what we should strip off very quickly,
like linear distortions of speaker dependencies.
Those are two things I will be talking about.
You can easily do it in a future instruction.
There are noises which are expected.
But efforts may not be well understood.
These should go into machine land,
but this is what I'm going--
this is something which we--
whatever you don't know, you better
have the machine to learn.
It's always better to use the dumb machine
with a lot of training data than putting in something
which you don't know for sure.
But what you know for sure, it should go here.
And then there is an interesting set of the whole noises, which
you don't know that they are--
they even exist.
These are the ones which I'm especially interested,
because they cause us the biggest problems.
Basically, the word you don't know that it exists,
noises which somebody introduces--
they've never talked about, and so on, and so on.
So I think this is an interesting problem.
Hopefully, I will get to it towards the end of the talk--
at least a little bit.
So some noise is with known effects.
One is like this.
When you have a--
you have a speech--
oh.
VOICE RECORDING: You are yo-yo.
HYNEK HERMANSKY: And you have another speech
which looks very different.
VOICE RECORDING: You are yo-yo!
HYNEK HERMANSKY: But it says the same thing, right?
I mean this is a child.
This is a adult. And you can tell this was me.
This was my daughter when she was 4-- not 30.
The problem is that different human beings
have different vocal tracts.
Especially when it comes to children,
vocal tract is much, much shorter.
And I was showing you the effects
of what happens is that you get a very different set
of formants like these dark lines, which a number of people
believe we should look at if we want to understand what's
being said.
We have four formants here, but we have only two formants here.
They are in approximately similar positions,
but where you had a fourth formant,
you have only second formant here.
So what we want--
we want some techniques which would work more
like a human perception, not look at the spectral envelopes.
But mainly you look at the whole clusters.
So this is a technique which has been developed a long time ago,
but I still mention it because it's an interesting way
of going about the things.
So it uses several things.
One is that it suppresses the signals at low frequencies.
You basically use this equal loudness curve.
So you emphasize the parts of the signal
each child heard well.
Second one that uses this is critical bands.
Because you say, first step which you want to do
is to integrate overcritical band.
This is the simplest way of processing
within the band-- is you integrate
what's happening inside.
So what you do, you take your full ear spectrum.
This is the spectrum which has equal frequency resolution
at all times and a lot of details with this case,
because it's a fundamental frequency.
And here you integrate over these different frequency
bands.
They are narrower at low frequencies
and getting broader, and broader, and broader,
very much how we learned from the experiment
with the simultaneous masking.
So this is a textbook knowledge.
So you get a different spectrum, which is unequally sampled.
So, of course, you go back into the equal sampling,
but you know that there is fewer samples
at the high frequencies, because you are integrating
more spectral energy at high frequencies
than in low frequencies.
And you multiply these outputs through these equal loudness
curves.
So from the spectrum you get something
which has a resolution, which is more like auditory-like.
Then you put it through the equal loudness curves,
because you know the loudness depends
on a cubic root of intensity.
So you get a modified spectrum.
And then you find some approximation
to this spectrum--
auditory spectrum-- saying, that I
don't think that all these details still
have to be important.
I would like to have some control
of how much spectral detail I want to keep in.
So the whole thing looks like-- let's start with the spectrum.
You go through-- a number of steps.
And you end up with the spectrum, which
is, of course, related to the origin of the spectrum,
but it's much simpler.
So we eliminated information about fundamental frequency.
We merged number of foramens and so on, and so on.
So we follow our philosophy.
Leave out the stuff which you think may not be important.
You don't know how much stuff you should leave out.
So if you don't know something, and you are engineer,
you run an experiment.
You know, research is what I'm doing when
I don't know what I'm doing--
supposingly.
I don't know where now-- from Brown or somebody
was saying that.
So we didn't know what's-- how much smoothing we should do
if we want to do our-- speaker independent representation.
So we ran an experiment for a number of smoothing,
a number of complex poles telling you
how much smoothing you get through auto-regressive model.
And there was a very distinct peak in the situations
where we had a training coming for--
templates coming from one speaker and the test
coming from another speaker.
Then we used this kind of representation
in a speech recognition--
I mean in a--
to derive the features from the speech.
Suddenly, these two pictures start looking much more
similar, because what this technique is doing
is basically interpreting the spectrum in a way
the hearing might be doing.
It has much lower resolution that normally people would use.
It has only two peaks, right?
But they say that it was good enough for speech recognition.
What was more interesting about--
a little bit of interesting science
is they also-- found that difference between production
of the adults and the children might be just
in the length of the pharynx.
This is a back part of the vocal tract; that the children may
be producing speech in such a way
that they are putting already the--
[AUDIO OUT] constriction into right position
against the palate.
And because they know--
or they-- well, whatever-- mother nature
taught them that pharynx will grow in the lifetime.
But the front part of the vocal tract is going to be similar.
So it is the front cavity which is speaker independent,
and it is the back cavity, the rest of the vocal tract,
which may be introducing speaker dependencies.
It's quite possible if you will--
if you ask people how they've been treated-- like actors--
on how they are being trained to generate the different voices,
they are being trained to modify back part of the-- vocal tract.
Normally, we don't know how to do.
But there is some circumstantial evidence
for this might be at least partially true.
So what is nice is that when we synthesize the speech
and we made sure that front cavity is always
in the same place, even when the foramens were
in different positions, we were getting very similar results.
So we have this theory.
The message is encoded in the shape of the front cavity.
Through speaker-dependent vocal tracts,
you generate the speech spectrum with all the formants.
But then there comes the speech perception fine
point, which extracts what is called
perceptual second formant.
Don't worry about that.
Basic-- [AUDIO OUT] on the--
at most, two peaks from the spectrum.
And this is being used for decoding of the signal, speaker
independently.
However, I told you one thing, which is
don't use the textbook data and be the exact--
[AUDIO OUT]
And so I was challenged by my friend, late
Professor Fred Jalinek.
He is claimed to say, airplanes don't flap wings,
so why should we be putting the knowledge of the hearing
in-- actually, he said something quite different.
This is what The New York Times quoted after he passed away,
because that was one of-- supposedly one
of his famous quotes.
No, he said something else.
Well, if airplanes do not flap wings,
but they have a wings nevertheless.
They use some knowledge from the nature
in order to get the job done.
The flapping of the wings is not important.
Having the wings is important if you
want to create the machine which is heavier than the air
and flies.
So we should try to include everything
what we know about human perception, and production,
and so on.
However, we need to estimate the parameters from the data,
because don't trust the textbooks and that sort
of thing.
You have to derive in such a way that is relevant to your task.
What I wanted to say--
you can use the data to derive the similar knowledge.
And I want to show it to you.
What you can do is to use a technique again,
known from the '30s, called Linear Discriminant Analysis.
This is the statistician's friend.
For this you need a within-class covariance and between class
covariance matrix.
You need the labeled data.
And you need to make some assumptions, which turns out
are not very critical.
But I mean, they are approximately
satisfied when you are working with the spectra.
So what we did was we would take this spectrogram,
and we would generate the spectral vectors from that.
So we would always cut part of the spectrum,
or short term the spectrum, and we assign it
to the label from which part of speech it came from.
So this one would have a label, "yo," right?
And so you get the big box full of vectors.
All of them are labeled.
So you can do LDA.
And you can look what they are--
discriminants are telling you.
From LDA, you get the discriminant matrix
and each of the row of the--
column of the-- column or row--
whatever-- creates the basis on which you should project
the whole spectrum, right?
These are the four obvious ones here.
You also have the amount of variability
present in this discriminant matrix which you started with.
What you observe, which is very interesting,
is that these bases tend to project the spectrum
at the beginning, with more detail,
than the spectrum at the end.
So essentially, they tend-- they appear--
in the first-- group they appear to be emulating properties
of human hearing, with some of the well known properties
of human hearing, namely, non-equal spectral resolution
being verified in any ways-- many, many ways.
Among them was one--
I was showing you this masking experiment of Harvey Fletcher.
There is a number of reasons to believe
that this is a good thing.
This is what you see.
So essentially, if you look at the zero crossings
of these bases-- this is the first base--
they are getting broader and broader.
So you are integrating more and more spectrum, right?
This is all right-- so that I leave it.
Oh, this is from another experiment
with a-- very large database, very much and very similar
thing.
Eigenvalues quickly decay.
And what is interesting-- you can actually formally
ask, "What is your resolution?" by doing what is
called perturbation analysis.
So you take some-- say, some signal here.
This is a Gaussian.
And you project this on this LDA basis.
Then you perturb it.
You move it.
And you ask, how much effect this movement of this, say,
emulated spectral element of the speech causes the output--
seen as the output of this projection of this many--
on these many bases?
And what do you see is, as I was suggesting,
spectral sensitivity to the movements of the formant
is much higher at the beginning of the spectrum
and much less at the end of the spectrum.
You can actually compare it to what
we had-- initially in the PLP analysis when
we integrated the spectrum based on the knowledge coming
from the textbook.
And it's very much the same if there
were just a plain cosine basis computing
mel cepstrum sensitivity is the same at all frequencies.
But these bases-- are from the LDA,
they're very much doing the thing which physical bin
analysis would be doing.
And so this is a--
you can look at it.
It was a PhD thesis from Oregon Graduate Institute by Naren
Malayath, who is now a big--
you better be friends with him.
He's at Qualcomm.
I think he's a head of Image Processing department.
[COUGHS] We better-- better be good friends with him.
[INAUDIBLE]
[LAUGHTER]
OK.
Another problem with linear distortions-- linear
distortions void was a problem.
Now it's not problem anymore, but in the old days,
this was a problem.
A problem shows up in a rather dramatic way, in following way.
Here we have one sound.
VOICE RECORDING: Beat.
HYNEK HERMANSKY: Beat.
So "buh ee- tuh."
Here is the very distinct E. Every phonetician would agree.
This is E-- high foramen, cluster of high of foramens,
and so on.
Some vicious person, namely one of my graduate students,
took this spectral envelope, designed a filter,
which is exactly in reverse of that,
and put this speech through this inverse filter
so it looked like this.
This was-- there is a spectrum where there were nine formants.
It's entirely flat.
And if you listened to it, you've
probably already guessed what you will hear.
VOICE RECORDING: Beat.
HYNEK HERMANSKY: You'll hear the first speech, right?
But you--
VOICE RECORDING: Beat.
HYNEK HERMANSKY: It's OK when this-- oops!
That's what you would--
sorry.
VOICE RECORDING: Beat.
Beat.
HYNEK HERMANSKY: But whoever doesn't hear E--
don't spoil my talk.
I mean, I think everybody has to hear E,
even though any phonetician would get very upset.
Because they say this is not E. Because, of course,
what is happening, human perception is taking relative--
percept relative to these neighboring sounds, right?
And since we filtered everything with the same filter, I mean,
relative percepts is still the same.
So this is something which we needed to put into our machine.
And we did-- actually, this is a very straight-- signal
processing wise, the things are very straightforward,
because what you have, the signal is actually
signal of the speech convolved with the spectrum
of the impulse response of the environment.
So in logarithmic domain this is the signal processing stuff.
Basically, you have a logarithmic spectrum
of the signal plus logarithmic spectrum of the environment,
which is fixed.
So-- what we're finding here is that if you remove somehow this
environment, or if you make it invariant to this environment
thing, then you maybe win--
you may be winning.
The problem here is that each frequency
you have a different amount of additive constant,
because this is a spectrum, right?
If it was just a constant at all frequencies,
you just subtract it.
But in this case, you can use the trick.
You remember what Josh told us this morning.
Hearing is doing spectral analysis.
And what I was trying to tell you, that each--
at each frequency, each critical band is trajectory
of the spectral energy is independent of-- in the first
approximation, of the other--
others.
You can do independent processing
at each frequency band--
and maybe don't screw up that many things.
So this was a step which we took.
We said, OK.
We will treat each sample trajectory differently, right?
But we will filter out stuff which is not changing.
So related to different frequency channel,
do the independent processing channel.
And processing was that we would take first logarithm,
and then we would do-- then we put each trajectory
through a bandpass filter, which would--
main thing is which would be suppressing DC,
and slowly changing components.
Mainly it was suppressing anything which
was slower than one hertz.
And also, it turned out it was useful to separate things which
are higher than about 15 hertz.
So what you get out-- this was the origin spectrogram.
This was the modified spectrogram.
It seems it got a little bit-- this trajectory got
a little bit smoother.
Transitions got smoothed out because there
was a bandpass filter.
There was a high pass elements to that.
Very much what we thought-- well, this is interesting.
Maybe this is what a human hearing might be doing.
To tell you the truth, we didn't know.
It was-- for the people who are from MIT
and who are working in Image, it was
inspired by some work on a perception of lightness, what
David Marr called lightness.
And here was the type--
thing which I told you about--
6 by-- 6 by 749.
David Marr was talking about processing in space.
We applied it in processing in time.
But it was still good enough.
I mean, so that we definitely got rid of the problem.
So here it is.
The spectrograms, which look very different,
look-- suddenly start looking very similar.
I was good seeing what's here.
Remember, I'm an engineer.
I was working for a telephone company at the time.
It was still working better in some problems
which we had before.
And we had a severely mismatched environment,
getting the training data from the labs
and testing it in the US west, in Colorado.
So it didn't work at all; recognized
after this processing everything was cool and dandy.
OK.
So now we have a RASTA LDA.
And we can do the same trick.
How about that?
So we take-- you take the spectral temporal vectors,
and you label each of these vector
by the label which is of the phoneme, which
is in the center of this trajectory.
And just for the sake of--
just to have some fun, we took a rather long vector.
It was about one second.
And we said, well, what kind of projections would these
temporal trajectories would go on if we wanted to get rid
of a speaker-dependent--
I mean, of an environment-dependent
information?
Well, these were the impulse responses.
These were the frequency responses.
Because in this case, you get a FIR filters.
These discriminants are FIR filters,
which are to be applied to temporal trajectory-- so
spectral energies.
Because it's just basically projection
of the spectrum on the basis.
And basis is one second long.
This is impulse response.
It cannot be all that long, because eventually,
they become zero, right, if you should do nothing to it.
You do do nothing.
But you can see active part is about a couple
of hundred millisecond, maybe a little bit more.
And these are the bandpass filters.
So essentially-- passing frequency between the 1 hertz
and 10, 15 hertz--
very similar at all frequencies.
There was another thing which we were very interested in.
We should really do different things
at different frequencies.
Answer is pretty much no.
And so that was very exciting.
What-- what I-- well, anyway, so let me tell you-- yet
another experiment which hasn't happened and is going
to be presented next week.
Well, we wanted to move in the 21st century,
so we did convolutive neural network.
And our convolutive network is maybe not
what you are used to when you have a 2D convolutions there.
But we just said, we will have a 1D filter
as a first processing step in this deep neural network.
So we postulated the filter, the input, to the neural network.
But in this case, we trained the whole thing together.
So it wasn't just LDA and that sort of thing.
So we forced all filters at all frequencies
be the same, because we expected that's what we want to get.
And we were asking how these forces look like,
which come from the convolutive neural network.
Well, again, I wouldn't be showing it
if it wasn't really somehow supportive of what I
want to say.
They don't look all that different for what
we were guessing from LDA.
They definitely are enhancing the important modulation
frequencies around four hertz, right?
They are passing a number of them.
I'm showing three here, which are somehow arbitrary.
They are passing-- and most of them look like that.
And we will use the 16 of them.
Passing between in 1 and 10 hertz in modulation
spectral domain, so changes which are 1 to 10 times
a second.
It's coming out in a paper, so you can look it up if you want.
Last thing which I still wanted to do--
I said, well, maybe it has something
to do with the hearing after all.
We were are deriving everything from speech.
There was no knowledge about hearing,
except that we said we think that we should be looking
at long segments of the signal, and we
expect that this filtering will be very much the same
at all frequencies.
Actually, not even-- earlier it come out automatically.
There wasn't much of the knowledge from human hearing--
[AUDIO OUT] on.
In the first one, when I was showing you critical
band spectral resolution, we started this full ear spectrum.
We didn't tell anything about human hearing.
And what comes out is a property of human hearing.
I mean, tell me if there is yet another strong evidence
that speech is processed in such a way that fits human hearing,
because the only thing which was used here
was the speech, labor, into certain--
into classes which we are using for recognizing it--
speech sounds.
So what we did was-- that was with Nema, with Garani,
and my students.
We took a number of these cortical receptive fields--
which we talk about it before a little bit--
about 2,000, 3,000, whatever-- we spread--
we basically spread to the floor at the University of Maryland
and computed principal components
from these fields in both spectral and temporal domain.
But here I'm showing the temporal domain.
How they look-- they are very much like the rest of filter.
This is what is happening.
It's a bandpass.
Peak is somewhere around four hertz.
Essentially, I'm showing you here what
I understood might be a transfer function of the auditory cortex
derived with all the usual disclaimers,
like, this is a linear approximation to the receptive
fields.
And there might have been problems with collecting it,
and so on, and so on.
But this is what we are getting as a possible function
of auditory cortex.
I'm doing fine with the time, right?
So you can do experiment in this case.
You can actually generate the speech
which has certain rates of change eliminated.
By doing all this, computing the cepstrum,
do the filtering of each trajectory,
and reconstruct the speech.
And ask people what do they hear?
How do they recognize speech?
You can also ask, "Do you recognize it?"
For this you don't have to regenerate the speech.
But you just use therapy C cepstrum.
This is the full experiment with the-- this
is for-- this is called a residual-excited LPC vocoder.
But it's modified in such a way that you artificially
slow down or modify the temporal trajectories, which are being--
if there is no filter, you cannot make a replica
of the origin, the signal here.
So just the bottom line of the experiment here is what--
if you start removing components which
are somewhere between 1 and 16 hertz,
you are getting hurt significantly.
Most you are getting hurt in performance
when they are removing component between 2 and 4 hertz.
This is a-- that's how you are getting biggest hit.
Here we are showing how much is--
how much these bands contribute to recognition and performance
by humans.
This is a white bars; and to speech recognize it.
Those are the black bars.
So you can see that in speech recognition, machine
recognition, you can safely remove
stuff between 0 and 1 hertz.
It's not going to hurt you.
It's only helps you in this task.
Speech perception, there is a little bit of hit,
but certainly not as much hit as you
are getting when you are moving to the part
where you hear the--
[AUDIO OUT]
And certainly the component that is higher than 16 or 20 hertz
are not important.
Then, already, Homer Dudley, he knew, in 1930,
when he was designing his vocoder.
But it was a nice experiment.
It came out in just--
so you can look it up and-- if you want to have a go.
Just to summarize what I told you so far,
Homer Dudley was telling us information from the--
information about the message in the slow modulation,
slow movements of the vocal tract, which modulates
the carrier; information about the message
in slow modulations of the signal--
slow changes of speech signal in individual frequency bands.
Slow modulations imply long impulse responses, right?
So 5 hertz, I sense something around 200 millisecond.
My magic number of what physically needs to allow,
which we have observed in this summation of sub-threshold--
the signals and temporal masking.
And so I have to hear a number of things which I listed.
Frequency discrimination improved.
If you are longer than 200 millisecond,
below 200 milliseconds of signal,
you don't get such a good frequency discrimination.
Loudness increases up to 200 millisecond,
then it stays constant.
It depends on amplitude.
Effect of forward masking--
I was showing you--
asked about 200 millisecond independent
of the amplitude of the masker.
And sub-threshold integration is also showing you.
So I'm suggesting there seem to be some temporal buffer
in human hearing on some level.
I suspect it's cortical level, which is processing.
Whatever happens within this buffer,
it's a fair thing to treat as a one element.
So you can do filtering on it.
You can integrate it; any-- basically, all kinds of thing.
If things are happening outside this buffer,
these parts should be treated--
[AUDIO OUT] in parts.
So how does it help us?
You remember the story about the phonemes.
You remember that phonemes don't look like this,
but they look like this.
Length of the coarticulation pattern is about 200
millisecond, perhaps more.
So what is a good thing about it is
that if you look at sufficiently long segment of this signal,
you will get whole coarticulation pattern in.
And then you have a chance that your classifier is getting
all the information about the speech
sound for finding the sound.
And then you may have a chance to get a good estimate
of the speech sounds.
But you need to use these long temporal segments.
And here I can say it even to YouTube.
I think we should claim the full victory here,
because most of the speech recognition systems
do it nowadays.
They use the long segments of the signal as a first step
of the processing.
So I can happily retire telling my grandchildren,
well, we knew it.
We were the only ones.
But I mean, we were certainly using it for a long time
and probably for a long time, in such a way
that we even designed several techniques that you do there.
So this is a classifying speech recognition
from the temporal patterns directly.
So we would take these long segments of the speech
through some processing, put neural nets on every--
every temporal structure, trying to estimate the sound
at each frequency--
each carrier frequency.
And then we would fuse all these decisions
from different frequency bands.
And then we would use the final vector
of posterior probabilities.
Unlike what people do very often--
most often-- that they just take the short term spectra,
and then they maybe now take the longer segment of this block
of these short term spectra.
We say, short term spectrum is good for nothing.
We just cut it into pieces.
And we classify each temporal trajectory individually
in the first step.
Tell now that it was used--
it may be useful later when I will
be telling you about dealing with some kind of noises.
But you understand what we did here, right?
Instead of using the spectral temporal blocks,
we would be using temporal trajectories
at each critical event, very much
along the lines of what we think that hearing is
doing with the speech signal.
First thing is, hearing is doing.
It takes the signal, sub divides it into individual frequency
bands, and then it treats each temporal trajectory
coming from each of these cochlear filters
to extract the information.
And then it tries to figure out what
to do with this information later, right?
Well, we have another technique called MRASTA, just
for people who are interested in cochlear--
I mean, in-- cortical modeling.
You take this data.
We project a number of projections
with variable resolution.
So we get a huge vector of the data
coming from different parts of the spectrum.
And then we feed it into speech recognizers.
The first test looked like this.
I mean, they have a different temporal resolution,
spectral resolution.
We are pretty much integrating or differentiating over
three critical bands following some of the filters,
coming from these three--
I mean, old PLP low order model and three bark three bark
critical event integration.
So these ones look a bit like what people
would call Gabor filters.
But they are just put together, basically,
from these two places in time and in frequency--
different temporal resolution enhancing different components
of moderating the spectrum.
Again, you may be claiming that this is something
which resembles the Thorston--
[AUDIO OUT] Josh was mentioning in the morning.
It's cochlear filter banks--
[CLEARS THROAT] auditory, of course.
I mixed up cochlear and cortical-- cortical filter
bank, modulation filter banks.
So there are some novel aspects in this type of processing
I want to impress.
It was novel in 1998.
That is, as I said, this is fortunately becoming less novel
15 years later.
Use is rather long temporal context
of the signal as a input.
It uses already hierarchical neural nets.
So deep neural network processing,
which wasn't around in 1998.
The only thing was that there was independent processing
of frequency of neural net estimator at frequencies.
The only thing which we didn't do at the time,
and I don't know how important it is.
I don't think it doesn't hurt--
it hurts anybody.
They were training these parts of the system,
this deep neural net individually.
And it's just concatenated output.
So we never did training all together
as we do now in convoluted nets and that sort of thing.
Because simply, we didn't even dream about doing that,
because we didn't have the hardware.
That was one thing which I tried to point out during this panel.
A lot of progress in neural nets research
and success of neural nets comes from the fact
that we have very, very powerful hardware, which we didn't have.
So we didn't dream about many things doing, even when
they might have made sense.
So, OK.
Where are we?
Oh, I see-- one more thing.
Coarticulation.
This is a problem which is known since people started
looking at the spectrograms.
There's some consonants like a "kuh", or "huh."
They are very dependent on what's following.
So "kuh" in front of "ee, kee, koo, kah, koo-kah,"
we [AUDIO OUT] has a burst here.
In front of "ooh" has a burst here.
And in front of "ah" there's a burst here.
So the phonemes are very different
depending on the environment.
When you start using these long temporal segments,
you know all the tricks, or some of the tricks, I showed you
about, what comes out are the posteriogram
in which the "kuh" almost looks the same as a "kuh."
It basically recognizes this--
recognizes that since it looks at the whole coarticulation
pattern or group of the phonemes,
in order to recognize this sound, it does the right thing.
So I suspect that success of these long temporal context
which people are using now with speech recognition,
comes from the fact that this partially
compensates for the problems with-- by problems
for the coarticulation.
And what I also want is to say-- coarticulation
is not really a problem.
It just spreads the information for a long period of the time.
If you know how to suck it out, it can be useful.
But it's a terrible thing if you start just
looking at individual frequency events,
even with your frequency's slices of the short term
spectrum.
So it's another deep net--
deep net from, I don't know the name.
Sorry.
It was already almost legal deep net.
You do the-- you estimate the posteriorgram
from the short window in the first step for about 40
mm length window.
And then you take the long--
I mean, big window of the posteriors' tree.
Another neural net you get much better, which also work better.
Again, the mainstream technique nowadays
is being used in most of the DARPA systems.
Oh, yes-- one more thing.
I want to stress this one.
[LAUGHS] So, I'm sorry I didn't want
to show it all at the same.
But anyways, I don't think that there's
anything which is terribly special about short term
spectrum of speech.
I think what really matters is how
you process the temporal trajectories of the spectra
energies.
This is what the human hearing is
doing that seems to be doing a good job on our speech
recognizers.
So essentially, this is one message which I want to say.
Don't be afraid to treat different parts of the spectrum
different.
Individually you may get some advantages from them.
It started with your stub, but it
shows up over and over again.
So away from the short term spectrum, go away,
they start using what hearing is doing--
start using a temporal trajectories
of the spectra energies coming from your analysis.
To the point that we did this work on real [INAUDIBLE],,
on the-- going directly, don't do this.
And don't get your time frequency patterns
from the short term spectra.
I think about always how to get directly what you want.
It turns out that there is a nice way of doing--
for estimating directed hilbert envelopes
of the signal in the frequency bands called frequency domain
linear prediction.
[STATIC]
Mario says-- there's his PhD thesis.
And we were working together for a couple of years.
So what you do, instead of using the time trajectory,
so use this case, autoregressive modeling--
LPC modeling-- and put the windows on a time to get
the frequency--
frequency vectors.
You do it on a cosine transform of the signal.
So you move the signal into a frequency domain.
And then you put the windows on this cosine transform
of the signal.
And you derive directly the--
all polar approximations to hilbert envelopes
of the signal in the sub bands.
You don't ever do the hilbert transform.
You just use the usual techniques
from autoaggressive modeling.
The only difference is--
[AUDIO OUT] on the cosine transform of the signal.
And your windowing determines which frequency
range you are looking.
So, of course, what you typically do,
you can use the longer windows at higher frequencies, shorter
window of lower frequencies.
You do all these things.
But this is a convenient way.
[COUGHS] It's convenient, and this is more and more like fun.
But maybe somebody might be interested in that.
So essentially, what you do-- oops.
Sorry.
What you do is that you take the signal,
and you eliminate modulation component
out of that AM component which the signal is being modulated.
So this carries the information about the message.
And this is the carrier itself.
And you can do what is called channel vocoder, which we did.
And you can listen to the signal.
So this is-- in some ways it's interesting-- original signal.
VOICE RECORDING: They are both trend-following methods.
HYNEK HERMANKSY: Oops.
I tried to make it somehow--
[AUDIO OUT]
VOICE RECORDING: They are both trend-following methods.
HYNEK HERMANKSY: Somebody may recognize Jim Glass from MIT
in that.
VOICE RECORDING: In an ideological argument,
the participants tend to dump the table.
HYNEK HERMANKSY: So this is silly, right?
Now you can look at what you get if you just
keep the modulations and excite you know, with the white noise.
Oops.
Sorry.
Oops!
What am I doing?
Oh, here.
VOICE RECORDING: (WHISPERING) They are
both trend-following methods.
HYNEK HERMANSKY: Do you recognize Jim Glass?
I can.
VOICE RECORDING: (WHISPERING) In an ideological argument
the participants tend to dump the table.
HYNEK HERMANSKY: And then you can also
listen to what is left after you eliminate the message.
VOICE RECORDING: Mm-hmm.
Ha, ha.
[LAUGHTER]
HYNEK HERMANSKY: Maybe it's a male, right?
VOICE RECORDING: Mm-mm [VOCALIZING]
HYNEK HERMANSKY: Oh, this is fun.
This is [CHUCKLES] fun.
It may have some implication for speech recognition.
But certainly, if I have seen one verification of what old
Homer Dudley was telling us-- where the message is--
I mean, this is it.
All right?
Anyways, what is good in-- for that,
is that once you get an open--
[AUDIO OUT] it's relatively very easy
to compensate for your ear distortions.
Because main effect of linear distortions
is basically shifting the energy a different frequency by--
bends by different amounts.
But all this information is in the gain of the model.
It's one parameter which you essentially
ignore after you do this frequency domain
linear prediction.
And you get a very similar trajectory for both.
This is a telephone speech and clean speech
which differed quite a bit.
And I hope that I have-- oh, this is for reverberant speech.
There seem to be also some advantage,
because reverberation is in the first information,
it is a convolution with the impulse response of the room.
So you make the--
if you use a truly long segments-- in this case,
we used about 10 seconds of the signal approximating
by this open model, and eliminated the DC from that.
You know, it seems to be getting some advanced--
[AUDIO OUT]
So known noise with unknown effects.
I say train the machine on that.
Here is the one example, right?
You have a phoneme error rates, noise estimate.
If everything is good, clean, trading clean test,
you have about 20% phoneme accuracy.
This is a stage of the result--
reasonable result. But once you start adding a noise,
things quickly go south.
Typical way of dealing with it is if you train multi-style.
So if you know which choices you are going to deal with,
you train on them.
And things get better, but you pay some price.
I mean, certainly, you pay the price on clean,
because you recognize your model became must mushier, basically.
It's not a very sharp model.
So here we had a wonderful 21%.
We paid 10% relative price for getting this better performance
on the noises.
What we observe is that you get much better results, most
noticeably better results, if you
would have different recognizers for each type of noise.
But of course, the problem is that you
have different types of noise.
So you have this number of recognizers.
But now you need to pick up the best stream.
And how do you do that?
This is something, again, which I was mentioning also earlier.
This is something which we are struggling with
and we don't know how to do that.
If you are a human being, maybe you
can just look at the output.
And you can see, just keep switching after your message
starts looking reasonably well.
But if you want to do it fully automatically,
I don't know why we want to only build
a fully automatic recognizers, but that's what we are doing.
So you want to pick up the--
you want a system to pick up the best stream.
So how do we do that?
First thing is, of course--
one way is to recognize type of noise.
This is a typical system nowadays.
You recognize type of the noise, and you use
the appropriate recognizers.
BBN is doing it.
My feeling is that it's somehow cleaner and more
elegant to be able to figure out what
is the right output, because what--
neither.
It's not what is the signal, but what is the signal interacting
with the classifier?
So for this we have to figure out what the best means.
So here we have two posteriograms.
If you look at it, if you know that these are trajectories
of the posteriors--
of the speech sounds, you know this one is good.
This one is not so good.
Because the word is nine--
"ne-ine," "ne," right?
Here is a lot of garbage.
So I will know that--
I will do it automatically.
So ideally, I would pick up the stream which
gives me the lowest error.
But I don't know what the lowest error is, because I don't know
what the correct answer is.
That's the problem, right?
So one is to maybe try to see what I-- what my I did?
Try to figure out which posteriogram is the cleanest.
Another one is following thinking.
When I trained the--
I trained the neural net on something.
It's going to work well on the data on which it was trained.
So I have some gold standard output.
And then I will try to see how much my output differs
if the test data, which are not the same as the data on which
the recognizer was trained.
So both of these tricks we were using.
So first one uses a technique which is like this.
You look at the differences between posteriors
or KL divergence areas a certain distance from each other,
and you will slice this window.
You cumulatively cover as much data as you possibly can.
And what you observe is that if you have a good, clean data,
this cumulative divergence keeps increasing.
And after you cross the point where there is a coarticulation
pattern or coarticulation ceases,
suddenly you start getting pretty much the fixed high tail
divergence-- cumulative KL divergence.
If we have a noisy data, the noise
start dominating this KL divergences and differences RS.
Because the signal in the first place
carries the information, and the information is in the changes.
But noise is creating the information
which it doesn't have these segments, or something.
So this is one technique which we use.
Another technique which is even more, now, popular,
in at least in my lab, training of another unit.
We trained this autoencoder on the output of a classifier.
And we say-- so we--
on the output of the classifier as
it's being used on the training data, and we say autoencoder.
Then we learn how in average the output
from the classifier used on its training data looks like.
And if-- and then we use it on the output from the classifier
used to unknown data.
And if the autoencoders then knew,
then we tried to predict input on its output.
This is how it is being trained.
So if the output--
if the prediction is not very good,
then we say we are probably dealing
with the data for which the classifier is not good.
It's how it works.
I mean, you know, it's honest.
If you are looking at the output of the neural net
which has been applied to a--
towards training data, the test is-- or the test is maybe--
the training data is matched.
Pretty much the error is very similar
as it goes on the training data.
When you apply to a data for which the classifier wasn't
trained, your error is, of course, much larger.
Prediction is prediction of the output of the--
so there's a double-- there is a double deep net.
One is classifying, and then another one
is predicting its output.
One-- and the one which predicts output
is trained to predict its best output it can possibly have,
which is the output on the training data
of the previous classifier.
I don't know if you follow me or if it
is becoming too complicated.
[CHUCKLES] But essentially, we are
trying to figure out if anything like
it is looking on a training-- and applied on the training
data.
So it seems to be working to some extent.
Here we have a multi-style results.
Here we have a matched result. This
is what we would like to achieve.
Of course this is oracle.
This is what we-- it will be ideal
if we knew which stream is best.
This is what we would be getting.
But what we are getting is not that terribly bad.
I mean, certainly, it is typically better
than a multi-style training.
All right.
And we have some ways to go to oracle like--
not too far from the matched trace.
Sometimes it's even there, because it's making
a decision on every utterance.
So sometimes it can-- going to do quite well.
So we were capable of picking up the good streams
and leaving out the best bad streams, even
at the output of the classifier.
How does it work on previously unseen noise?
Fortunately, for this example, we still
seem to be getting some advantage.
We are using noise which has never
been seen by the classifiers.
But still, it was capable of picking up
the good classifier, which is actually better
than any of these classifiers.
Sort of a, so this seems to be good.
Another technique of dealing with unseen noises
is actually one which I like maybe even a bit more,
which is you do the pre-processing,
and in some processing in frequency bands,
hoping the main effect of the different noises
is in the shape, spectral shape, of different noise.
So if you are doing recommission in the sub bands,
then the noise start looking-- in each sub band
starts looking more like a white noise except
of different levels.
So meaning, it's here, that maybe the signal to noise ratio
is higher.
Here it is more miserable.
But if I have the classifier, which
is strained on multiple levels of the white noise,
in each frequency band perhaps I can get some advantage.
So I do what in the cochlear might be doing,
which is I divide each--
the signal into a number of frequency bands, and then I
have one fusion DNN which will try
to put these things together.
But each of these nets are going to be
trained on multiple levels, but this time of the white noise.
And it's going to get back to noises which are not white,
or the noise which is not white.
So how it works, you can figure it out
to see if it was done in the case of Aurora.
So here we have the examples here--
how it works on a-- in matched situations.
Here is multi-style training.
But here it is what you are getting
if you apply this technique.
This is what you get now, multi-style training.
But in the sub band recommendation,
you are getting 1/2 of the error rate.
Just a simple trick which I think
is reasonable, which is you do the sub band recognition.
There's a number of power recognized,
each of them paying attention to part of the spectrum.
And then you-- each of them is being
trained to head the white noise, a simple white noise.
But you turn, in some ways, arbitrary additive noise,
this car noise, into white-like noise in each sub band.
And that what's you get.
So in general, dealing with unexpected noise is--
you want to do the adaptation.
You want to modify your classifier on the fly.
You want to have parts of the classifier or some streams
which are doing well.
And some of the parts--
so parts of the classifier are still reliable.
And you want to pick up these streams which are
reliable on unseen situation.
So this is what we call this multi-stream recognition--
adapt to multi-stream adaptation to unknown noise.
So you assume that not all the streams are going to give you
good result, but you assume that at least some of the streams
are going to have good results.
And all these streams are being trained on, say, clean speech
or something.
So this is this multi-band processing, all right?
So this is what we do.
We do different frequency ranges.
And then we use our performance monitor
to pick up the best stream.
So here is the experiment which we did.
So you would have the 31 processing
streams created from all combinations
of fine frequencies.
And one stream was looking at full spectrum.
And the other things were only looking
at the parts of the spectrum.
So more black ones, there is a more spectrum; more white,
some of them are looking only at a single frequency band.
So we have a decent number of processing channels,
and we would hope that if the noise comes here,
maybe this one is going to be good, because this one is not
going to be--
not worth picking up this noise, or recognize
that basically which only uses the bands which are not noisy.
It's going to be good.
So this is the whole system.
It was published in 230--
[STATIC] like a entire speech.
We had this sub band recognition, fuse,
and performance monitor, and the selecting of a stream.
This is how it works.
This is again, showing for car noise.
Car noise is very nice, because it mainly, it
corrupts the low frequencies.
So all these sub band techniques work quite well.
But you can see that it's pretty impressive, and it's--
if you didn't do anything, you get 50% error.
With this one you get 38% error.
If you knew which bands to pickup
you would be getting oracle experiment-- cheating
experiment.
You would be getting about 35%.
So that was, I thought, quite nice.
Just to conclude-- so, auditory system doesn't only look like
this, that it starts with a signal as the analysis,
and then it reduces the--
reduces the bit rate; but it also
increases number of views of the signal.
And this is based on the fact that there
is a massive increase in number of the cortical neurons
on the level of the cortex.
So there is many ways of describing information
at high level of perception.
Essentially, the signal doesn't go through one pass,
but it goes through many, many passes.
And then we need to have some means, or we have some means
to pick up the good ones and ignore the other ones--
maybe switch them entirely off.
This is like vision.
So this is the path of processing, in general, signal.
You get the different probability estimates
for different streams.
And then you need to do some fusion and decide on--
based on the level of the fusion.
How you can create the streams-- but we were showing you
differently trained probability estimates on different noises--
different aspects of signal, that
is, different parts of the spectrum of the signal.
But you can go wild.
You can start thinking about different modalities,
because there-- as we talked about it also in the panel,
you know, very often audiovisual model,
if it carries the same information
about the same things, [STATIC] infusion
of audio visual streams.
You can also imagine the fusion from streams
with different levels of priors, different levels
of hallucinations.
So basically, this is what I see human beings are
doing very often.
If the signal is very noisy, you are at a cocktail party,
you are guessing, because that's the best
way to get through if the communication is not
very important.
It's not about your salary increase,
but about the weather--
so basically, guessing what the other people are saying,
especially if they speak the way I
do, right, with a strong accent or something.
So the priors are very important.
And streams with priors are very important.
We use this to some extent, I was mentioning,
by comparing the streams of different-- with different
prior to discover if the signal is biased in the wrong way
by priors.
So stream formation-- there is a number of PhD theses
right here, right?
I think.
Fusion-- oh, why?
It's select the best probability estimates.
I tell you.
This is the problem which I was actually asking for,
please help me to solve it.
Because we still don't know how to do that.
I have a-- I suspect that especially
in human communications, people are doing it
like this, which starts making a sense
if they use the certain processing strategy.
So people can tell if the output of our--
this perceptual system makes sense or not.
Our machines don't know how to do it yet.
Conclusion.
So some problems with the noise are simple.
You know, you can deal with it on a signal processing level
by filtering the spectrum, filtering the trajectories,
because these effects are very predictable.
And if you understand them, you should do it.
Because there's no need to train on that.
They said there is--
you just do it.
And things may be working well.
Unpredictable effects of noise can be--
typically are being handled by now by multi-style training.
And these amounts of training are enormous nowadays.
You know, if you talk to Google people,
they say we are not deeply interested in this-- what you
are doing, because we can always collect more
data from new environments.
But I think it's not--
I shouldn't say dishonest.
I'm sorry.
Scratch it.
[LAUGHTER]
It's not the best engineering way
of dealing with these things, because I
think the good engineering way of dealing with those things
is to get away with less training and that sort
of thing; and maybe follow what I believe that human beings are
doing.
So we have a lot of parallel experts working
with the different aspects of the signal,
giving us different pictures.
And then we need to pick up the good ways of being.