SPEAKER: The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation, or to view additional material from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: OK I want to review a little bit what we said
about detection at the end of last hour, last hour and a
half, because we were going through it relatively quickly.
Detection is a very funny subject.
Particularly the way that we do it here because we go
through a bunch of very, very simple steps.
Everything looks trivial--
I hope it looks trivial-- as we go, at least after you
think about it for awhile, and you work with it for awhile,
you will come back and look at it and you will say, "yes, in
fact that is trivial." Nothing when you look at it for the
first time is trivial.
And the kind of detection problems that we're
interested in is--
to start out with-- we want to look just at binary detection.
We're sending a binary signal, it's going to go through some
signal encoder, which is the kind of channel encoder we've
been thinking about.
It's going to go through a baseband modulator, a baseband
to passband modulator.
It's going to have white Gaussian noise, or some kind
of noise, added to it.
It's going to come out the other end.
We come back from passband to baseband.
We then go through a baseband demodulator.
We then sample at that point.
And, the point is, when you're all done with all of that,
what you've done is you've started out sending either
plus a or minus a as a one- dimensional numerical signal.
And when you're all through, there's some one- dimensional
number that comes out, v. And on the basis of that one-
dimensional number, v, you're supposed to guess whether the
output is zero or one.
Now, one of the things that we're doing right now is we're
simplifying the problem in the sense that we're not looking
at a sequence of inputs coming in, and we're not looking at a
sequence of outputs coming out.
We're only looking at a single input coming in.
In other words, you build this piece of communication
equipment, you get it all tuned up, you get it into
steady state.
You send one bit.
You receive something.
You try to guess at the receiver, what was sent.
And at that point you tear the whole thing down and you wait
a year until you've set it up perfectly again.
You send another bit.
And we're not going to worry at all about what happens with
the sequence, we're only going to worry about
this one shot problem.
You sort of have some kind of clue that if you send the
whole sequence of bits, in a system like this, and you
don't have intersymbol interference, and the noise is
white, so it's sort of independent from time to time.
You sort of have a clue that you're going to get the same
answer whether you send the sequence of data or whether
you just send a single bit.
And we're going to show later that that, in fact, is true.
But for the time being we want to understand what's going on,
and to understand what's going on we take this simplest
possible case where there's only one bit that's being
transmitted.
It's the question, "Are we going to destroy ourselves in
the next five years or not?" And this question is important
to most of us, and at the output we find out, in fact,
whether we're going to destroy ourselves or not.
So it's one bit, but it's one important bit.
OK, why are we doing things this way?
Want to tell you a little story about the first time I
really talked to Claude Shannon.
I was a young member of the faculty at that time, and I
was working on a problem which I thought was
really a neat problem.
It was interesting theoretically.
It was important practically.
And I thought, "gee, I finally have something I can go to
this great man and talk to him about." So I screwed up my
courage for about two days.
Finally I saw his door open and him sitting there, so I
went in and started to tell him about this problem.
He's a very kind person and he listened very patiently.
And after about 15 minutes he said, "My god!
I'm just sort of lost with all of this.
There's so much stuff going on in this problem.
Can't we simplify it a little bit by throwing out this kind
of practical constraint you've put on it?" I said, "yeah, I
guess so." So we threw that out and the we went on for
awhile later, and then he said, "My god, I'm still
terribly confused about this whole thing.
Why don't we simplify it in some other way?"
And this went on for about an hour.
As I say, he was a very patient guy.
And at the end of an hour I was getting really depressed.
Because here was this beautiful problem that I
thought was going to make me famous, give me tenure, do all
these neat things.
And here he'd reduced the thing to a
totally trivial toy problem.
And we looked at it.
And we said, yes this is a trivial toy problem.
This is the answer.
The problem is solved.
But so what?
And then he suggested putting some of those
constraints back in again.
And as we started putting the constraints back in, one- by-
one, we saw that each time we put a new constraint in--
since we understood the problem and its simplest
form-- putting the constraint in, it was still simple.
And by the time we built the whole thing back up again, it
was clear what the answer was.
OK, in other words, what theory means is really solving
these toy problems.
And solving the toy problems first.
And in terms of practice, some people think the most
practical thing is to be practical.
And the whole point of this course, and this particular
subject of detection is a wonderful example of this, the
most practical thing is to be theoretical.
I mean, you need to add practice to the theory, but
the way you do things is you start with a theory-- which
means you start with the toy problems, you build up from
those toy problems, and after you build up for awhile,
understanding what the practical problem is also--
you then understand how to deal with
the practical problem.
And the practical engineer who doesn't have any of that
fundamental knowledge about how to deal with these
problems is always submerged in a sea of complexity.
Always doing simulations of something that he doesn't, he
or she doesn't understand.
Always trying to interpret something from it, but with
just too many things going on to have any idea of what it
really means.
Ok, so that's why we're making this trivial assumption here.
We're only putting one bit in.
We're ignoring what happens all the
way through the system.
We only get one number out.
We're going to assume that this one number here is either
plus or minus a, plus the Gaussian
random noise variable.
And we're not quite sure why it's going to be plus or minus
a, plus a Gaussian noise random variable, but we're
going to assume that for the time being.
OK?
So the detector observes the sample values of the random
variable for which this is the sample value, and then guesses
what the value of this random variable, h, which is what we
call the input now.
Because we view it from the standpoint of the detector--
the detector has two possible hypotheses--
one is that a zero was sent, and the other
that a one was sent.
And on the basis of this observation, you take first
the hypothesis zero and you say, "Is this a reasonable
hypothesis?" Then you look at the hypothesis one, say, "Is
this a reasonable hypothesis?" And then you guess whether you
think zero was more likely or one is more likely, given this
observation that you've had.
So what they detector has, at this point, is a full
statistical characterization of the entire problem.
Mainly you have a model of the problem.
You understand every probability in the universe
that might have any effect on this.
And what might have any effect on this--
as far as the way we've set up the problem--
is only the question of what are the probabilities that
you're going to send one or the other of
these signals here?
And conditional on each of these, what are the
probabilities of this random variable
appearing at the output?
Because you have to base your decision only on this.
So all of the probabilities only give you
this one simple thing.
Hypothesis testing, decision making, decoding, all mean the
same thing.
They mean exactly the same thing.
And they're just done by different people.
OK, so what that says is we're assuming the detector uses a
known probability model.
And in designing the detector, you know what that
probability model is.
It might not be the right probability model, and one of
the things that many people interested in detection study
is the question of when you think the probability model is
one thing and it's actually something else, how well does
the detection work?
It's a little like the Lempel-Ziv algorithm that we
studied earlier for doing source coding.
Which is, how do you do source coding when you don't know
what the probabilities are?
And we found the best way to study that, of course, was to
first find out how to do source encoding when you did
know what the probabilities were.
So we're doing the same thing here.
We assume the detector is designed to maximize the
probability of guessing correctly.
In other words, it's trying to minimize the
probability of error.
We call that a MAP detector-- maximum a posteriori
probability decoding.
You can try to do other things.
You can say that there's a cost of one kind of error, and
there's another cost of another kind of error.
I mean, if you're doing medical testing or something.
If you guess wrong in one way, you tell the patient there's
nothing wrong with them, the patient goes out, drops dead
the next day.
And you don't care about that, of course, but you care about
the fact that the patient is going to sue the hospital for
100 million dollars and you're going to lose your
job because of it.
So there's a big cost to guessing wrong in that way.
But for now, we're not going to bother about the costs.
One of the things that you'll see when we get all done is
that putting in cost doesn't make the
problem any harder, really.
You really wind up with the same kind of problem.
OK, so h is the random variable that will be
detected, and v is the random variable
that's going to be observed.
The experiment is performed.
Some sample value of v is observed, and some sample
value of the hypothesis has actually happened.
In other words, what has happened is you prepared the
whole system.
Then at the input end to the whole system, the input to the
channel, somebody has chosen a one or a zero without the
knowledge of the receiver.
That one or zero has been sent through this whole system, the
receiver has observed some output, v, so in fact we're
now dealing with the sample values of
two different things.
The sample value of the input, which is h, the sample value
of the output, which is v, and in terms of the sample value
of the output, we're trying to guess what the sample value of
the input is.
OK, an error then occurs if, after the output chooses a
particular hypothesis as its guess, and that hypothesis,
then, is a function of what it receives.
In other words, after you receive something, what the
detector has to do is somehow map what gets received, which
is some number, into either zero or one.
It's like what a Quantizer does.
Mainly, it maps the whole region into two
different sub- regions.
Some things are mapped into zero, Some things
are mapped into one.
This H bar then becomes a random variable, but is a
random variable that is a function of what's received.
So we have one random variable, H, which is what
actually happened.
There's another random variable, H hat, which is what
the detector guesses will happen.
This is an unusual random variable, because it's not
determined ahead of time.
It's determined only in terms of what you decide your
detection rule is going to be.
This is a random variable that you have some control over.
These other random variables you have no
control over at all.
So that's the random variable we're going to choose.
And, in fact, what we're going to do is we're going to say
what we want to do is this MAP decoding, maximum a posteriori
probability decoding, where we're trying to minimize the
probability of screwing up.
And we don't care whether we make an error of one kind or
make an error of the other kind.
OK, is that formulation of the problem crystal clear?
Anybody have any questions about it?
I mean, the easiest way to get screwed up with detection is
at a certain point to be going through, studying a detection
problem, and then you suddenly realize you
don't understand what--
you don't understand what the whole problem is about.
OK, let's assume we do know what the problem is, then.
In principle it's simple.
Given a particular observed value, what we're going to do
is we're going to calculate the-- what we call the a
posteriori probability, the probability given that
particular sample value of the observation--
we're going to calculate the probability that what went
into the system is a zero and what went into
the system is a one.
OK?
This is the probability that j is the sample value of H,
conditional on what we observed.
OK, if you can calculate this quantity, it tells you if I
decide that--
if I guess that H is equal to j-- this in fact tells me that
this is the probability that guess is correct.
And if this is the probability that guess is correct, and I
want to maximize my probability of guessing
correctly, what do I do?
Well, what I do is my MAP rule is arg max of this
probability.
And "arg max" means instead of trying to maximize this
instead of trying to maximize this quantity over something,
what we're doing is trying to find the value of j, which
maximizes this.
In other words, we calculate this for each value of j, and
then we picked the j for which this quantity is largest.
In other words, we maximize this but we're not interested
in the maximum value of it at this point.
We're interested in it later because that's the probability
that we're choosing correctly.
What we're interested in, for now, is what is the hypothesis
that we're going to guess.
OK?
So this probability of being correct is going to the this
probability for this maximal j.
And when we average over v we get the overall probability of
being correct.
There's a theorem which is stated in the notes, which is
one of the more trivial theorems you can think of,
which says that if you do the best thing for every sample
point you, in fact, have done the best thing on average.
I think that's pretty clear.
If you, well I mean you can read it if you want to form a
proof, but if you do the best thing all the time, then it's
the overall best thing.
OK, so that's the general idea of detection.
And in doing this we have to be able to calculate these
probabilities, so that's the only constraint.
These are probabilities, which means that this set of
hypothesis is discrete.
If you have an uncountable infinite number of hypotheses,
at that point you're dealing with an estimation problem.
Because you don't have any chance in hell of getting the
right answer, exactly the right answer.
And therefore you have to have some criterion for
how close you are.
And that's what's important in estimation.
And here what's important is really, do we guess right or
don't we guess right.
And we don't, we don't care.
There aren't any near misses here.
You either get it on the nose or you don't.
OK, so we want to study binary detection now
to start off with.
We want to trivialize the problem because even that
problem we just stated is too hard.
So we're going to trivialize it in two ways.
We're going to assume that there are only two hypotheses.
That it's a binary detection problem.
And we're also going to assume that it's Gaussian noise.
And that will make it sort of transparent what's happening.
So H takes the values zero or one.
And we'll call the probabilities with which it
takes those values P zero and P one.
These are called a priori probabilities.
In other words, these are the probabilities that the
hypothesis takes to value zero or one before seeing any
observation.
And the probabilities after you see the observation are
called a posteriori probabilities.
In other words, probabilities after the observation and the
probabilities before the observation.
Up until about 1950 statisticians used to argue
terribly about whether it was valid to assume a priori
probabilities.
And as you can see by thinking about it a little bit, the
problem they were facing was they couldn't separate the
problem of choosing a mathematical model and
analyzing it from the problem of figuring out whether the
model was valid or not.
And at that point people studying in that area had not
gotten to the point where they could say, "Well, maybe I
ought to analyze the problem for different models, and then
after I understand what happens for different models I
then ought to go back because I'll know what's important to
find out in the real problem." But up until that time, there
was just fighting among everyone.
Bayes was the person who decided you really ought to
assume that there's a model to start with.
And he developed most of detection
theory at an early time.
And people used to think that Bayes was a terrible fraud.
Because in fact he was using models of the problem rather
than nothing.
But anyway, that's where we were.
We're also going to assume that, after we get all through
with modulation and demodulation, and we really
want to look at a general problem here.
There's one discrete random variable, H, one analog random
variable, which has a probability density, v, what
we want to assume is that there's a probability density
that we know, which is the probability density of the
observation conditional on the hypothesis.
We're assuming that we know this, and we know this--
we call these things likelihoods--
and in most communication problems anyway, it's far
easier to get your hand on these likelihoods than it is
to get your hand the a posteriori probabilities,
which you're really interested in.
So we find these likelihoods, we can find the marginal
density of the observation.
Which is just the weighted sum.
The probability that the hypothesis is zero times a
conditional probability of the observation and so forth.
So we're going to assume that those densities exist.
We're going to assume that we know them.
And then with a great feat of probability theory, we say the
a posteriori probability is equal to the a priori
probability times the likelihood divided by the
marginal probability of v. OK?
What was the first thing you learn in probability when you
started studying random variables?
It was probably this formula, which says when you have a
joint density of two random variables you can
write it in two ways.
You can either write, "density of one times density of two
conditional and one is equal to the density of two times
density of one conditional in two." And then you think about
it a little bit and you say, "A ha!" It doesn't matter
whether the first one is the density or whether it's the
probability.
You can deal with it in the same way.
And you get this formula, which I hope is
not unusual to you.
OK, so our MAP decision rule then, our MAP decision rule,
remember, is to pick the more, is to find the a posteriori
probability which is most probable.
Because that is the probability of being correct.
So, in fact, if this probability is bigger than
this probability, this is the a posteriori probability that
H is equal to zero.
This is the a posteriori probability that
H is equal to one.
So we're just going to compare those two and pick the larger.
And if this one is larger than this one, we pick our choice
equal zero.
And if it's smaller, we pick our choice equal to one.
So this is what MAP detection is.
Why did I make this greater than or equal
and this less than?
Well, if we have densities, it doesn't often make any
difference.
Strangely enough, it does sometimes make a difference.
Because sometimes you can have a density, and the densities
are the same for both of these likelihoods.
And you can find situations where it's important.
But when the two probabilities are the same, the probability
of being correct is the same in both cases, so it doesn't
make any difference what you do when you
have a quality here.
And therefore we've just made a decision.
We've said, OK, what we're going to do is whenever this
is equal to this, we're going to choose zero.
If you prefer choosing one, be my guest.
All of your MAP error probabilities will
be exactly the same.
Nothing will change.
It just is easier to do the same thing all the time.
OK, well then we look at this formula, and we say, "Well, I
can simplify this a little bit." If I take this
likelihood and move it over to this side, and if I take this
marginal density and move it over to this side, and if I
take p zero and move it over to this side, then the
marginal densities cancel out.
They had nothing to do with the problem.
And I wind up with a ratio of the likelihoods.
And what do you think the ratio of the
likelihoods is called?
Somebody got the smart idea of calling that
a likelihood ratio.
Somehow the people in statistics were much better at
generating notation than the people in communication theory
who have done just an abominable job of choosing
notation for things.
But anyway, they call this a likelihood ratio.
And the rule then becomes: if the likelihood ratio is
greater than or equal to the ratio of p1 to
p0, we choose zero.
And if it's less, we choose one.
And we call this ratio the threshold.
So in fact what this says is binary MAP tests are always
threshold tests.
And by a threshold test I mean finds the likelihood ratio,
compare the likelihood ratio with the threshold-- the
threshold in fact is this ratio of a priori
probabilities--
and at that point you have actually
achieved the MAP test.
In other words, you have done something which actually, for
real, minimizes the probability of error.
Maximizes the probability of being correct.
Well because of that, this thing here, this likelihood
ratio, is called a sufficient statistic.
And it's called a sufficient statistic because you can do
math decoding just by knowing this number.
OK?
In other words, it says you can you can calculate these
likelihoods.
You can find the ratio of them-- which is this
likelihood ratio-- and after you know the likelihood ratio,
you don't have to worry about these likelihoods anymore.
This is the only thing relevant to the problem.
Now this doesn't seem to be a huge saving, because here
we're dealing with two real numbers-- well here we've
reduced it to one real number-- which is something.
When we start dealing with vectors, when we start dealing
with wave forms, this is really a big thing.
Because what you're doing is reducing
the vectors to numbers.
And when you reduce accountibly infinite
dimensional vector to a number,
that's a big advantage.
It also, in terms of the communication problems we're
facing, breaks up a detector into two pieces in an
interesting way.
Mainly it says there are things you do with the wave
form in order to calculate what this likelihood ratio is,
and then after you find the likelihood ratio you just
forget about what the wave form was and you
deal only with that.
What we're going to find out is in this problem we were
looking at here--
we're going to find out later when we look
at the vector problem--
in fact this thing here is in fact the likelihood ratio if
you make an observation out at this point here.
In other words, right at the front end of the receiver,
that's where you have all the information you
can possibly have.
If you calculate likelihood ratios at that point what
you're going to do is to find the likelihood ratio you're
going to go through all this stuff right here and wind up
with that which is work which is proportional to the
likelihood ratio right here.
OK, so one of the things we're doing right now is we're not
looking at that problem.
We're only looking at the simpler problem, assuming a
one dimensional problem.
But the reason we're looking at it is that later we're
going to show that this is, in fact, the solution to the more
general problem.
Which was Shannon's idea in the first place.
Of, how do you solve the trivial problem first and then
see what the complicated problem is.
OK, so that's what we're trying to do, summarized here
for any binary detection problem where the observation
has a sample value of a random something.
Namely, a random vector, a random process, a random
variable, a complex variable, a complex anything.
Anything whatsoever, so long as you can assign a
probability density to it.
You calculate the likelihood ratio, which is this ratio
here, so long as you have then cities even talk about.
The MAP rule is to compare this likelihood ratio with the
threshold data-- which is just the ratio of the a priori
probabilities--
if this is greater than or equal to
that, you choose zero.
Otherwise you choose one.
The MAP rule, as I said before, partitions this
observation space into two pieces.
Into two segments.
And one of those pieces gets mapped into zero.
One of the pieces gets mapped into one.
It's exactly like a binary quanitizer.
Except the rule you use to choose the the quantization
regions is different.
But a quanitizer maps a space into a finite set of regions.
And this detection rule does exactly the same thing.
And since the beginning of information theory people have
been puzzling over how to make use of the correspondence
between quanitization on one hand and detection on the
other hand.
And there are some correspondences but they
aren't all that good most of the time.
OK, so you get an error when the actual hypothesis that
occurred, namely the bit that got sent was i and if the
observation landed in the other subset.
We know that the MAP rule minimizes the error
probability.
So you have a rule which you can use for all binary
detection problems so long as you have the density.
And if you don't have a density you can generalize it
without too much trouble.
OK, so we want to look at the problem in Gaussian noise.
In particular we want to look at it for 2PAM.
In other words for a standard PAM system, where zero gets
mapped into plus a and one gets mapped into minus a.
This is often called antipodal signaling because you're
sending a plus something and a minus something.
They are at opposite ends of the spectrum.
You push them as far away as you can, because as you push
them further and further away it requires
more and more energy.
So you use the energy you have.
You get them as far apart as you can, and you hope that's
going to help you.
And we'll see that it does help you.
OK, so what you receive then, we'll assume, is either plus
or minus a--
depending on which hypothesis occured--
plus a Gaussian random variable.
And here's where the notation of communication theorists
rears its ugly head.
We call the variance of this random variable n 0 over 2.
I would prefer to call it sigma squared but
unfortunately you can't fight city hall on
something like this.
And everybody talks about n 0 and n 0 over 2.
And you got to get used to it.
So here's where we're starting to get used to it.
So that's the variance of this noise random variable.
OK, we're only going to send one binary digit, H, so this
is the only, this is the sole problem we have to deal with.
We've made a binary choice.
Added one Gaussian random variable to it.
You observe the sum, and you guess.
So what are these likelihoods in this case?
Well, the likelihood if H is equal zero, in other words if
you're sending a plus a, the likelihood is just a Gaussian
density shifted over by a.
And if you're sending, on the other hand, a one--
which means you're sending minus a-- you have a Gaussian
density shifted over the other way.
Let me show you a picture of that.
We'll come back to analyze more things about the picture
in a little bit, so don't worry about most of the
picture at this point.
OK, this is the likelihood probability density of the
output given that you sent a zero.
Mainly that you sent plus a.
So we have a Gaussian density-- this bell shaped
curve-- centered around plus a.
If you sent a one, you're sending minus a-- one gets
mapped into minus a-- and you have the same bell shaped
curve centered around minus a.
If you receive any particular value of v, mainly suppose you
receive the value of v here, you calculate these two
likelihoods.
One of them is this.
One of them is that.
You compare them, the ratio with the threshold, and you
make your choice.
OK, so let's go back to do it.
To do the arithmetic.
Here are the two likelihoods.
You take the ratio of these two things.
When you take the ratio of them, what happens?
And this sort of always happens in
these Gaussian problems.
These terms cancel out.
Well it always happens in these
additive Gaussian problems.
These terms cancel out.
You take a ratio of two exponents.
You just get the difference.
So the likelihood ratio-- this divided by this--
is then e to the minus v minus a squared over n0.
and v plus a squared over n0.
OK?
Because normally the Gaussian density is something divided
by two sigma squared, and sigma squared here is n0 over
2, so the 2's cancel out.
One nice thing about the notation anyway, you get rid
of one factor of two in it.
Well so you have this minus this.
When you take the difference of these two things the v
squareds cancel out.
Because one of these things is in the numerator, the other
one was in the denominator.
So you have this term comes through as is.
This is--
you're dividing by this--
so when you multiply this turns into a plus sign.
So the v squared here cancels out with the v squared here.
The a squared here cancels out with the a squared here.
And it's only the inner product term that survives
this whole thing.
And here you have plus 2va.
Here you have plus 2va.
So you wind up with e to the 4av divided by n0.
Which is very nice, because what it says is this
likelihood, which is what determines everything in the
world, is just a scalar in multiple of the observation.
And that that's going to simplify things a fair amount.
It's why that picture comes out as simply is it does.
OK, so to do a little more of the arithmetic.
This is the likelihood here, e to the 4av over n0.
So our rule is you compare this likelihood to the
threshold-- which is p1 over p0, which we call eta--
and you look at that for awhile and you say, "Gee, this
is going to be much easier to deal with.
Instead of looking at the likelihood ratio, I look at
the log likelihood ratio."
And people who deal with Gaussian problems a lot, you
never hear them talk about likelihood ratios, you always
hear them talk about log likelihood ratios.
And you can find one from the other, so either one is
equally good.
In other words, the log likelihood ratio is a
sufficient statistic, because you can calculate the
likelihood ratio from it.
So this is a sufficient statistic.
It's equal to 4av over n0.
And when this is greater than or equal the log of the
threshold, you go this way.
When it's less than, you go this way.
So when you then multiply by n0 over 4a, your decision rule
is you just look at the observation.
You compare it with n0 times log of eta divided by 4a.
And at this point we can go back to this picture and sort
of sort out what all of it means.
Because this point here is now the threshold.
It's n0 times log of eta divided 4eta.
By 4a.
That's what we said the threshold had to be.
So we have these two Gaussian curves now.
Why do we have to go back and look at these Gaussian curves?
I told you that once we calculated the likelihood
ratio we could forget about the curves.
So why do I want to put the curves back in?
Well because I want to calculate the probability of
error at this point.
OK?
And it's easier to calculate the probability of error if,
in fact, I draw curve for myself and I look at
what's going on.
So here's the threshold.
Here's the density when H equals one is the correct
hypothesis.
The probability of error is the probability that when I
send one, the observation is going to be a random variable
with this probability density and if it's a wild case, and I
got an enormous value of noise, positive noise, the
noise is going to push me over that threshold there and I'm
going to make a mistake.
So, in fact, the probability of error-- conditional on
sending one--
is just this probability of that little space in there.
OK?
Which is the probability that I'm going to say that zero
occurred when, in fact, one occurred.
So that's my probability of error when one occurs.
What's the probability of error when zero occurs?
Well it's the same analysis.
When zero occurs--
mainly when the correct hypothesis is zero--
the output, v, follows this probability density here.
And I'm going to screw up if the noise carries me beyond
this point here.
So you can see what the threshold is doing now.
I mean, when you choose a threshold which is positive it
makes it much harder to screw up when you send a minus a.
It makes it much easier to screw up when
you send a plus a.
But you see that's what we wanted to do, because the
threshold was positive in this case, because p1 was so much
larger than p0.
And because p1 is so much larger than p0--
p1 happens almost all the time--
and therefore you would normally almost choose p1
without looking at v. Which says you want to push it over
that way a little bit.
OK, when you calculate this probability of error, it's the
probability of the tail of a Gaussian random variable.
So you define this tail variable, q of x, is the
complimentary distribution function of a
normal random variable.
It's the integral from x to infinity of one over the
square root of 2pi, e to the minus c squared over 2.
I guess this would make better sense if this were a z--
one and--
oh, no.
No, I did it right the first time.
That's an x, because x is the limit in there, you see.
So I'm calculating all of the probability density that's off
to the right of x.
And the probability of error when H is equal to one is this
probability-- which looks like it's the tail on the negative
side but if you think about it a little bit, since the
Gaussian curve is symmetric, you can also look at it as a q
function where now when you have this is equal to zero,
this corresponds to changing this plus to a minus here and
that's the only change.
OK, so this looks a little ugly and it
looks a little strange.
I mean you can sort of interpret
this part of it here--
I can interpret this part if I'm using
maximum likelihood decoding--
maximum likelihood is mapped decoding where the threshold
is equal to one.
In other words, it's where you're assuming that the
hypothesis is equally likely to be zero or one-- a priori--
which is a good assumption almost always in communication
because we work very hard in doing source coding to make
those binary digits equally likely zero or one.
And there are other reasons for choosing maximum
likelihood.
If you don't know anything about the probabilities it's a
good assumption in a sort of a max/min sense.
It sort of limits how much you can screw up by having the
wrong probability, so it's a very robust choice also.
OK, but now this, we're taking the ratio of a with the square
root of n0.
Well the square root of n0 over 2 is really the standard
deviation of the noise.
So what we're doing is comparing the amount of input
we've put in with the standard deviation of the noise.
Now does that make any sense?
The probability of error depends on that ratio?
Well yeah, it makes a whole lot of sense.
Because if, for example, I wanted to look at this problem
in a different scaling system--
if this is volts and I want to look at it in milli-volts--
I'm going to divide a by, I'm going to multiply a by 1000.
I'm going to multiply the standard deviation of the
noise by 1000.
Because one of the things we always do here-- the way we
choose n0 over 2--
n0 over 2 is sort of a meaningless quantity.
It's the noise energy in one degree of freedom in the
scaling reference that we're using for the data.
OK?
And that's the only definition you can come up with that
makes any sense.
I mean you scale the data in whatever way you please, and
when we've gone from baseband to passband, we in fact have
multiplied the energy in the input by a factor of two.
And therefore, because of that, we're going to--
n0 at passband is going to be a square root of 2 bigger than
it is at baseband.
If you don't like that, live with it.
That's the way it is.
Nobody will change n0, no matter who wants
them to change it.
OK, so this term make sense.
It's the ratio of the signal amplitude to the standard
deviation of the noise.
And that should be the only way that the signal amplitude
or the standard deviation of the noise enters in, because
it's really the ratio that has to be important.
Why this crazy term?
Well if you look at the curve you can sort of see why it is.
The threshold test is comparing the likelihood ratio
of this curve with the likelihood
ratio of this curve.
What's going to happen as a gets very, very large?
You move a out, and the thing that's happening then is this
curve-- which is now coming down in a modest way here-- if
you move a way out here, you're going to have almost
nothing there.
And it's going to be going down very fast.
It's going to be going down very fast
relative to its magnitude.
In other words the bigger a gets, the bigger this
difference is going to be for any given threshold.
And that's why you get a over square root of n0 here.
And here you get exactly the opposite thing.
That's because for a given threshold, as this signal to
noise ratio gets bigger, this threshold term becomes almost
totally unimportant.
I mean you get so much information out of the reading
you're making, because it's so reliable, that having a
threshold is almost completely irrelevant.
And therefore you can sort of forget about it.
If a is very large, this term is zilch.
OK?
So if you want to have reliable communication and you
use a large amount of single noise ratio to get it, that's
another reason for forgetting about whether the threshold is
one or something else.
And we would certainly like to deal with problems where the
threshold is equal to one because most people who
remember q of a signal to noise ratio.
I don't know anybody who can remember this formula.
I'm sure there's some people, but I don't think anybody who
works in the communication field ever thinks
about this at all.
Except the first time they derive it and the say, "Oh,
that's very nice." And then they promptly forget about it.
The only reason I think about it more is I
teach the course sometimes.
Otherwise I would forget about it, too.
OK, which is what this says.
For communication we assume p0 is equal to p1.
So we assume that eta is equal to one.
So the probability of error, which is also the probability
of error when H is equal to one-- in other words when a
one actually enters the communication system-- is
equal to the probability of error when H is equal to 0.
In other words these two tails here, when the threshold is
equal to one you set the threshold right there.
The probability of this tail is clearly equal to the
probability of this tail.
Just by symmetry.
So these two error probabilities are the same.
And in fact they are just q of a over the square
root of n0 over 2.
It's nice to put this in terms of energy.
We said before that energy is sort of important in the
communication field.
So we call e sub b the energy per bit that we're
spending to send data.
I mean don't worry about the fact that we're only sending
one bit and then we're tearing the communication system down.
Because pretty soon we're going to send multiple bits.
But the amount of energy we're spending sending this one bit
is a squared.
At least back in this frame of reference that we're looking
at now, where we're just looking at this discrete
signal and single variable noise.
And n0 over 2 is the noise variance of this particular
random variable, z, so when we write this out in terms of Eb,
which is a squared, it looks like this-- it's 2eb over n0.
So the probability of error for this binary communication
problem is just the square root of 2Eb over n0.
Which is a formula that you want to remember.
It's the error probability for binary detection when n0 over
2 is the noise energy on this one degree of freedom and Eb
is the amount of energy you're spending on this
one degree of freedom.
You will see about 50 variations of
this as we go on.
If you try to remember this fundamental definition, it'll
save you a lot of agony.
Even so, everybody I know who deals with this kind of thing
always screws up the factors of twos.
And finally when they get all done, they try to figure out
from common sense or from something else, what the
factors of two ought to be.
And they reduce their probability of error to about
a quarter after they're all done with doing all of that.
OK, so we spent a lot of time analyzing
binary antipodal signals.
What about the binary non antipodal signals?
This is beautiful example of Shannon's idea of studying the
simplest cases first.
If you have two signals, one of which is b and one of which
is b prime, and they can be put anywhere on the real line.
And what I've done, because I didn't want to plot this whole
picture again, is I just took the zero out, and I replaced
the zero by the point halfway between these two points which
is b plus b prime over 2.
And then we look at it and we say what happens if you have
an arbitrary set of two points anywhere on the real line?
Well when I send this point the likelihood, conditional on
this point being sent, is this Gaussian curve
centered on b prime.
When I send b the likelihood is a Gaussian
curve centered on b.
And it is in fact the same curve that we drew before.
If in fact I replaced zero with a center point between
these two, which is b plus b prime over 2.
And if I then define a as this distance in here, the
probability of error that I calculated before is the same
as it was before.
Now I would suggest to all of you that you try to find the
probability of error for this system here, just not using
what we've already done, and just writing out the
likelihood ratios as a function of an arbitrary b
prime and an arbitrary b and finding a likelihood ratio,
and calculating through all of that.
And most of you are capable of calculating through all of it.
But when you do so, you will get a god awful looking
formula, which just looks totally messy.
And by looking at the formula you are not going to be able
to realize that what's going on is what we see here from
the picture.
And the only reason we figured out what was going like the
picture is we already solved the problem
for the Simpson case.
OK so it never makes any sense in this problem to look at
this general case.
You always want to say the general case is just a special
case of a special case.
Where you just have to define things slightly differently.
OK, so we're going to do the center point, then, I mean it
might be a pilot tone.
It might be any other non information bearing signal.
In other words were sending the one bit.
Sometimes for some reason or other, you need to get
synchronization.
You need to get other things in a communication system.
And for that reason, you send other things.
We'll talk about a lot of those later.
But they don't change the error probability at all.
The error probability is determined solely by the
distance between these two points which we call 2a.
So probability of error remains the same in terms of
this distance.
The energy per bit now changes.
The energy per bit is the energy here
plus the energy here.
Which in fact is the energy in the center
point plus a squared.
I mean we've done that in a number of contexts, the way to
find the energy in a binary random variable is to take the
energy in the center point plus the energy in the
difference.
It's the same as finding the fluctuation plus the
square of the mean.
It's that same underlying idea.
So any time you use non antipodal and you shift things
off the mean, you can see what's going
on very, very easily.
You waste energy.
I mean it might not be wasted, you might have to waste it for
some reason.
But as far as a communication is concerned you're simply
wasting it.
So your energy per bit changes, but your probability
of error remains the same.
Because of that, you get a very clear cut idea of what
it's costing you to send that pilot tone.
Because in fact what you've done is to just increase this
energy, which we talk about in terms of db.
If c is equal to a in this case which, as will see, is a
common thing that happens in a lot of systems, what you've
lost is a factor three db.
Because you're using twice as much energy which is three db
more energy, than you have to use for the pure
communication.
So it's costing you three db to do whatever silly thing you
want to do for synchronization or something else.
Which is why people work very hard to try to send signals
which carry their own synchronization
information in them.
And we will talk about that more when we get to wireless
and things like that.
OK.
Let's go on to real antipodal vectors in
white Gaussian noise.
And again, let me point out to you again that one of the
remarkable things about detection theory is once you
understand detection for antipodal binary signals and
Gaussian noise, everything else just follows along.
OK, so here what we're going to do is to assume that under
the hypothesis H equals zero-- in other words conditional on
a zero entering the communication system--
what we're going to send is not a single degree, is not
something in a single degree of freedom.
But we're actually going to send a vector.
And you can think of that if you want to as sending a way
form and breaking up the waveform into an orthonormal
expansion and a1 to ak as being the
coefficients in that expansion.
So we're going to use several degrees of
freedom to send one signal.
Yes?
AUDIENCE: [INAUDIBLE]
PROFESSOR: I'm only sending one bit.
And on Wednesday I'm going to talk about what happens when
we want to send multiple bits or when we want to send a
large number of signals in one degree of freedom, or all
those cases, multiple hypotheses.
You know what's going to happen?
It's going to turn out to be a trivial problem again.
Multiple hypotheses are no harder than just binary
hypotheses.
So again, once you understand the simplest case, all
Gaussian problems turn out to be solved.
Just with minor variations and minor tweaks.
OK, so I have antipodal vectors.
One vector is a1 to a sub k.
Under the other hypothesis we're going to send minus a,
which is the opposite vector.
So if we're dealing with two dimensional space with
coordinates here and here, if I send this I'm
going to send this.
If I send that I'm going to send that, and so forth.
As the opposite alternative.
The likelihood ratio is then, the probability of vector v
conditional on sending this.
I'm assuming here that the noise is IID and each noise
variable has mean zero and variance n0 over 2.
Namely, we're pretending we're communication people here,
using and n0 over 2 here.
So the conditional density--
the likelihood of this output vector given zero-- is just
this density of z shifted over by a.
So it's what we've talked about as the Gaussian density.
Just this, which is just the energy in v minus a is what
that turns out to be.
So the log likelihood ratio is the ratio of this quantity to
the density where H is equal to one.
And when H is equal to one, what happens is the same thing
as happened before.
One makes this sign turn into a plus sign.
So when I look at the log likelihood ratio, I want to
take the ratio of this quantity to the same quantity
with a plus put into it.
And when I take the log of that, what happens is I get
this term minus the opposite term of the opposite side.
So I have minus the norm squared of v minus a plus the
norm squared of v plus a over n0.
And again, if you multiply this out, the v squareds
cancel out.
The a squareds cancel out.
And you just get the inner product terms.
And strangely enough you get the same formula that you got
before, almost, except here you have the inner product of
v with a instead of just the product of v times a.
So in fact we just have a slight generalization of the
thing that we did before.
In other words, the scalar product is
a sufficient statistic.
Now what does that tell you?
It tells you how to build a detector.
OK?
It tells you when you have a vector detection problem, the
thing that you want to do is to take this vector, v, that
you have and form the inner product of v times a.
If in fact v is a waveform and a is a waveform,
what do you do then?
Well the first thing you do is to think of
v as being a vector--
where it's the vector of coefficients-- in the
expansion for that waveform and a in the same way.
You look at what the inner product is then, and then you
say, "well what does that correspond to when I deal with
L2 waveforms?" What's the inner
product for L2 waveforms?
It's the integral of the product of the waveforms.
And how do you form the integral of the
product of the waveforms?
You take this waveform here.
You turn it around and you call it a matched filter to a.
And you take the received waveform.
You pass it through the max filter for a, and you look at
the output for it.
Now, let's go back and look at what all of this was doing.
And for now let's forget about the baseband to
the passband business.
Let's just look at this part here because it's a little
easier to see this first.
So this comes in here.
Now remember what we were saying
when we studied Nyquist.
We said a neat thing to do was to use a square root of the
Nyquist pulse at the transmitter.
When you use a square root of the Nyquist pulse at the
transmitter, what you have is orthogonality between the
pulse and all of its shifts.
Well now we don't much care about the orthogonality
between the pulse and all of the shifts because we're only
sending this one bit anyway.
But it sort of looks like we're going to be able to put
that back in in a nice convenient way.
So we're sending this one pulse, p of t, city and what
did we do in this baseband demodulator?
We passed this through another filter, q of t, which was the
matched filter to p of t.
What's our optimal detector for maximum likelihood?
It's to take whatever this waveform was, pass it through
the matched filter.
In other words, to calculate that inner product we just
talked about.
OK?
So in fact when we were looking at the Nyquist problem
and worrying about inner symbol interference, in fact
what we were doing was also doing the first part of an
optimal MAP detector.
And at this point what comes out of here is a single
number, v, which in fact now is the inner product of this
waveform at this point.
With the waveform, a, that we sent.
OK?
In other words, we started out by saying, "let's suppose that
what we have here is a number.
What's the optimal detector to build?" And then we go on and
say, "OK, let's suppose we look at the problem here.
What's the optimal detector to build now?" And the optimal
detector to build now at this point is this matched filter
to this input waveform.
Followed by the inner product here-- which is what the match
filter does for us-- followed by our binary antipodal
detector again.
OK?
So by studying the problem at this point, we now understand
what happens at this point.
And do I have time to show you what happens at this point?
I don't know.
Let me--
let's not do that at least right now--
let's look at the picture of this that we get when we just
look at the problem when we have two dimensions.
So we're either going to transmit a vector, a, or we're
going to transmit a vector, minus a.
And think of this in two dimensions.
When we transmit the vector, a, we have
two dimensional noise.
We've already pointed out that two dimensional Gaussian noise
has circular symmetry.
Spherical symmetry in an arbitrary number of dimension.
So what happens is you get these equal probability
regions which are spreading out like when you drop a rock
into a pool of water.
You see all of these things spreading out in circles.
And you then say, "OK, what's this inner product going to
correspond to?" Finding the inner product and comparing it
with a threshold.
Well you can see geometrically what's going to happen here.
You're trying to do maximum likelihood.
And we already know we're supposed to calculate the
inner product, so what the inner product is going to do
is take whatever v that we receive-- it's going to
project it on to this line between 0 and a.
So if I got a v here I'm going to project it down to here.
And then what I'm going to do is I'm going to compare the
distance from here to there with the distance
from here to there.
Which says first project, then do the old decision in a one
dimensional way.
Now geometrically, this distance squared is equal to
this distance squared plus this distance squared.
And this distance squared is equal to the same distance
squared plus this distance squared.
So whatever you decide to do in terms of these distances,
you will also decide to do in terms of these distances.
Which also means that the maximum likelihood regions
that you're going to develop, or in fact the maximum a
posteriori probability regions are simply planes.
Which are perpendicular to the line between
minus a and plus a.
OK?
So if you're doing maximum likelihood you just form a
plane halfway between these two points.
Yeah?
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: We got the error probability just by first
doing the projection and then turning it into this scale or
problem again.
So in fact the error probability--
What?
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: The probability of error is just the probability
of error in the projection.
Did I write it down someplace?
Oh yeah, I did write it down.
But I wrote it down, well I sort of cheated.
It's in the notes.
I mean the likelihood ratio is just a center product here
which is a number.
And when you find the error probability, you just use the
same q formula that we used before.
And in place of a you substitute--
in place of a you substitute the inner product of v with a.
Which is the corresponding quantity.
So it's q of 4va divided by n0.
OK?
So that's the maximum likelihood error probability.
OK?
In other words, nothing new has happened here.
You just go through the match filter and then you do this
same one dimensional problem that we've already
figured out how to do.
I think I'm going to stop there and we'll do the complex
case which really corresponds to what happens after baseband
to passband then passband to baseband.