PROFESSOR: We are still in chapter 12 on
the sum-product algorithm.
I hope to get through that today, fairly expeditiously,
and to start with chapter 13.
The handouts for today will be chapter 13.
And, as usual, problem set 9 with problem 8.3 has been
moved up into problem set 9.
Problem set 8 solutions for 8.1 and 8.2.
So I'm going to start off on a different tack of presenting
the sum-product algorithm.
Where is everybody?
1, 2, 3, 4, 5, 6, 7, 8, 9.
Is there any competitive event that I should know about?
I know the traffic was bad and it's a rainy, late morning.
Problem getting here myself.
Well, this is improving.
The write up of the sum-product in the notes is in
terms of equations.
And I've never been terribly satisfied with this write up.
It's concise, but we have to invent some new notation,
which I don't consider to be completely transparent.
I've never found it particularly easy to present
in class, and it's been a little frustrating to me.
So I'd like to try a different approach.
I'd like in class just to go over through a very explicit
example, our favorite 844 code.
What we're trying to achieve with the sum-product algorithm
and how the sum-product algorithm helps us to achieve
it in an efficient way.
So it's proof by example.
It won't be very satisfying to those of you who like to see
real proofs, but that you can get by going back and
re-reading the notes.
So I'm going to try doing it this way and you will let me
know by your looks of gladness or of frustration whether you
like it or not.
So let's start at the beginning again.
What are we trying to do?
We're trying to compute the a posteriori probability of
every possible value of every symbol, internal and external,
that appears in a graphical realization for a code.
And we will consider these to be APP vectors.
In other words, if a symbol is eight valued, there will be
eight APP values in the vector.
And we're going to assume that we have a cycle-free graph
realization, so that we can proceed by the cut-set idea to
solve independent parts of the graph.
And at the very end, we'll relax that and see whether we
can still do OK.
All right, one concept that seems very simple in theory
but that some people don't get till they really try to do it
in practice is that an APP vector is really a likelihood
weight vector normalized so that the sums of all the
probabilities are equal to 1.
For instance, if we send plus 1 or minus 1 through an
additive white Gaussian Noise channel with sigma squared
equal to 1 or whatever, then the probability of--
we call this y and this r, the probability of r given y
equals whatever.
This is probability of r given y, is 1 over square root of 2
pi sigma squared e to the minus y minus r-squared over 2
sigma-squared.
And so we can compute that for y equals 1 and y
equals minus 1.
And we'll get a vector of two things.
The APP vector would then consist of this evaluated for
y equals 1.
Let me just take this part of it, a to the minus r minus 1
squared over 2 sigma squared and e to the minus r plus 1
squared over 2 sigma-squared.
Something proportional to these two vectors normalized
and that's why I continue to use this proportional to
symbol, which you may not have seen before.
Or maybe you have.
So the APP vector in this case would be proportional to--
well, actually 1 over square root of 2 pi sigma-squared
times each of these two things.
But this would basically tell you how much weight to give to
plus 1 and minus 1 as possible transmitted symbols.
And all this matters --
there's one extra degree of freedom in here.
You can scale this, the whole vector, by any scale factor
you like and it gives you the same information.
Because you can always normalize the two terms to be
equal to 1.
So if this, for instance, was 0.12 and this was 0.03, that's
equivalent to 0.8 and 0.2.
Two probabilities.
Actual a-posteriori probabilities that sum to 1,
all you've got to do is keep the ratio right.
This is four times as large as that, so what that really
means is 0.8 and 0.2.
And in implementation, so the sum-product algorithm, we
never really worry about constant scaling factors.
As long as they appear in every term we can just forget
about scaling.
I mean we don't want things to get too large or too small, so
we can apply an overall scaling factor.
But as long as we've got something, the relative
weights, then we have what we need to know.
Let's go to the 844 code since we know that very well and ask
ourselves what it is we're trying to compute.
We're given APP vectors for each of the eight received
bits, let's say.
We transmit y, we receive some r, perhaps over an additive
white Gaussian noise channel.
So we get a little 2 term APP vector for each of these y's,
telling us the relative probability that a 0 was sent
or a 1 was sent.
That's called the intrinsic information.
Let me just call that.
So for each one we get p0 and--
let me make it p0 and q0, the two terms for y0; p1 and q1
for y1 and so forth.
So we have that information for each of
the received symbols.
What's the likelihood of each of these code words then given
these individual bitwise likelihoods, the probability
of the all zero code word.
It's likelihood, let's say, is then proportional to p0, p1,
p2, p3, p4, p5, p6, p7.
Whereas the probability at 1, 1, 1, 1, 0, 0, 0, 0 is q0, q1,
q2, q3, p4, p5, p6, p7 and so forth.
Each one is a product of the eight corresponding terms.
Yes?
AUDIENCE: You're assuming that the bits are independent.
PROFESSOR: Yes, it's a memoryless channel.
So the noise in every symbol transmission is independent.
And that's important too.
AUDIENCE: So basically once again, the second bit is
independent of the first bit in transmission?
PROFESSOR: I'm sorry, I was writing and I
missed what you said.
AUDIENCE: Well, I mean, if the first bit is 0, maybe that is
a code where if the first bit is 0, the second
bit has to be 0.
PROFESSOR: Yeah, there's certainly dependencies among
the code words.
That's what we're trying to take into account.
This is how to take it into account.
So I'm assuming a memoryless channel.
But I'm of course, assuming dependencies within the code.
How do I do that?
One way is maximum likelihood decoding.
What do we do when we do maximum
likelihood decoding in principle?
We exhaustively computed each of these 16 products, each of
these 16 likelihoods for each of the 16 code words.
We'd simply pick the one that's greatest and that's the
code word that is most likely to have been sent.
So that's what exhaustive maximum likelihood decoding
consists of, finding the maximum likelihood word.
And that clearly takes into account the dependencies.
We're only considering valid code
sequences when we do that.
In APP decoding we're doing something else, but it can be
characterized quite simply.
What's the a-posteriori probability that say, the
first bit, is equal to a 0 or is equal to a 1?
We can get the APP by summing up the likelihoods of all the
code words that are associated with the value of the bit that
we want to see.
So we take the eight code words that have a 0 in this
place and we'd sum up these likelihoods for those eight
code words, and that would be the weight, the likelihood
weight of y0 being 0.
And the likelihood weight of y0 being 1 would be the sum of
the other eight.
And again you see that scale factors aren't
going to matter here.
All we want is the relative weights of these things.
So that's what we're trying to do in APP decoding, but we're
trying to do it now for every single variable.
At this point, I've only shown you the symbol variables, the
external variables.
So a brute force way of doing that would be to fill out this
table, all 16 likelihoods, do these sums and then we'd have
the APP vector for each of these.
Now we're looking for a more efficient way of doing this.
The efficient way of doing this is going to be based on
the fact we have a cycle-free graph realization.
Which in this case, is a trellis realization.
I'm going to use this two section trellis where each
section is a four-tuple.
We've drawn it two different ways.
This is a very explicit way showing the two parallel
transitions that go from the initial node to the central
state, and then two more down here.
The set of code words is the set of all
possible paths through--
corresponds to the set of all paths through this trellis.
So for instance, includes the all zero word, (0,0,1,1,1),
(1,1,1,0,0,0), (1,1,1,1,1,1,1), and so forth.
And these I happened to have listed as the first four here.
Or we now have this more abstract way of writing this
same thing.
This says there are four external variables and two
state variables or a vector space of dimension 2 here and
a vector space with dimension 4 here.
That the dimension of this constraint is 3.
That means there are eight possible values for these
combinations of 4 and 2.
They're shown explicitly up here.
And they form a linear vector space.
Either way, this is our trellis realization.
Now, we know that to do maximum likelihood decoding,
it's useful to have this trellis realization, because,
for instance, we can do Viterbi decoding.
And that's a more efficient way of finding what the
maximum likelihood sequence is.
Basically, because we go through and at this point we
can make a decision between 0, 0, 0, and 1, 1, 1, 1.
We only have to do that based on this four-tuple and then we
can live with that decision regardless of what else we see
in the other half.
And likewise in the other direction.
So now we'd like to use this same structure to simplify the
APP decoding calculation, which looks like a more
complicated calculation.
So how do we do that?
We introduce two state bits here.
All of these have state bit 0, 0, 0, 0.
Sorry, this point, we've called this state 0, 1.
Just to keep it visually straight, 0, 1, 0, 1, 0, 1,
and then there are eight more corresponding to the other two
possible values of the state bits over here.
This is a quaternary variable.
We want it to assume that it's a four-valued variable and to
make the realization cycle-free, I don't want to
consider it to be two independent bits.
So there are four possible values
for this state variable.
So maybe I just ought to call this my state space --
which is a four-valued state space.
Now, what's the a-posteriori probability of the state
being say, 0, 0?
I compute that in the same way.
I compute that simply by summing up the four
likelihoods of the code for the external variables, the
code words that are associated with state variable 0, 0 --
the first possible values of this quaternary variable.
So it'd be the sum of these four things.
Yes?
AUDIENCE: [UNINTELLIGIBLE PHRASE]
PROFESSOR: I've only listed half the code words.
This is only 8 of the 16 code words.
You want me to write the other 8?
And by the way, there's at least an implicit assumption
here that the code sequence determines the state sequence.
As we found when we did minimal trellis realizations,
the code sequence always does determine the state sequence.
So if I know the entire code word that also tells me what
the values of all the state variables are.
There's a 1:1 correspondence between code words and
trajectories in a minimal trellis.
And I believe I'm sometimes implicitly using that
assumption.
But anyway, it holds whenever we have a cycle-free graph.
In a cycle-free graph, we have a well-defined minimal
realization.
And the minimal realization does have this 1:1 property.
So the state variables are determined
by the symbol variables.
Suppose I want to compute the values of these state
variables here.
Let me write this out.
This will be p0, p1, p2, p3, q1--
sorry ---
q4, q5, q6, q7.
And then there's one that's all q's.
q0, q1, q2, q3, q4, q5, q6, q7.
So basically, what I want to do is take the sum of these
four things, and that's just a sum of products and I could
just do it.
But the Cartesian product lemma gives us a simpler way
of doing it.
We notice that I can have either of these two
four-tuples as the first half and either of these two
four-tuples as a second half.
In fact, I see over here, explicitly, these four code
words are the Cartesian product of two four-tuple
codes, the repetition code on 4.
So I can use the Cartesian product lemma to
write this as --
again, let me write it very explicitly.
p0, p1, p2, p3, which you recognize as the probability
of this four-tuple plus q0, q1, q2, q3.
The probability of this four-tuple times p4, p5, p6,
p7 plus q4, q5, q6, q7.
Now do you agree that the product of those two terms is
equal to this, the sum of these four terms?
Yes, there are four terms in this sum and
they're what we want.
And it's because we have this Cartesian product structure,
which we always have in the state space.
This is the basic Markov property of a state variable.
So that's a simpler calculation.
This requires four multiplications, each one is
seven-fold multiplication.
This only requires one multip -- well, it requires one, two,
three, four-fold multiplications, plus one
overall multiplication.
And it's certainly a simpler way to compute that.
And I can similarly do that for each of these down here.
What this leads me to is what I called last time the past
future decomposition.
We have one APP vector over here that corresponds to the--
let's call it the APP vector of the state given the past
received symbols.
We had something like that.
In other words, given what was received for y0, y1, y2, y3,
what's the probability, the partial a-posteriori
probability for each of the four possible state variables
given this past?
And similarly, we have another vector which I will consider
to be a message coming in this direction, a message
consisting of a vector of four values.
Which is what's the probability the state vector
given these four received simple, which is computed in
the same way?
That's this here.
And the past future rule is that just the overall
probability for any of these internal state variables, the
overall APP vector, is just obtained by a component-wise
multiplication.
This would be the APP of the state vector being equal to 0,
0 given the past.
This would be APP of the state vector given
0, 0 given the future.
We'd have the same thing for 0, 1, 1, 0, 1, 1.
And to get the overall APP vector given all of r, the
rule is just multiply them component-wise.
We take the likelihood weight for 0, 0 given the past and
multiply it by likelihood weight of
0, 0 given the future.
And that gives us the total likelihood weight up to a
scale factor.
The notes derive this in equation terms.
You see it basically comes from this Cartesian product
decomposition, which we always get for any
internal state variable.
The possible code words consistent with that state
consist of a certain set of possible past code words
consistent with that state.
Cartesian product with a set of possible future code words
consistent with that state.
And as a result of that Cartesian product
decomposition, we always get this APP vector decomposition.
That's what the proof says in the notes.
Do you like this way of arguing?
A couple people are nodding at least.
It's an experiment.
All right, so that tells us what we're going to want to
do, at least, for the internal variables.
And actually, the same is true up here.
The variables going in are easy.
That's just p0, p1, p2, p3, or whatever.
The ones coming out are--
this is called the intrinsic variable, the one that we get
from direct observation, the intrinsic APP
vector, excuse me.
The extrinsic APP vector is the one that we get from all
the rest of the inputs and symbols and the other
parts of the graph.
And we combine these component-wise to get the
overall APP vector.
All right, but we now have to do this for
every part of the graph.
Let me go through another calculation, which will--
how do we, for instance, find the APP vector for y7, the
extrinsic APP vector for y7, given that we have a
trellis like this?
In general, for y7, we're going to be adding up all the
zeros, these guys.
And four more down here, and we'll make that the extrinsic
APP vector of 0.
So we want to add up this and this and this and this and so
forth, eight of those.
We want to do that in an efficient way.
This defines a code word with eight possible components.
Let me move this up now.
So let's talk about this 6, 3 code.
What does it look like?
With state vectors 0, 0, I can have y4, y5, y6,
y7 be 0, 0, 0, 0.
Or I can have it be 1, 1, 1, 1.
The state vector is 0, 1.
I can have 0, 1, 0, 1 or I could have 1, 0, 1, 0.
With state vectors variable 1, 0, I can have 0, 0, 1,
1 or 1, 1, 0, 0.
And with 1, 1, I can have 0, 1, 1, 0 or 1, 1, 1, 0, 0, 1.
All right, so that's precisely the 6, 3
code I'm talking about.
Now suppose I have the APP vector for each of these.
The APP vector for each of these, 0, 0 we've already
calculated.
It'd be p0, p1, p2, p3 plus q0, q1, q2, q3.
That's the likelihood vector for 0, 0 coming from the past.
So that's one part of this message here.
For either of these I get p0, q1, p2, q2, q3
plus q0, p1, q2, p3.
Reflecting the two possible ways I can get to this value.
I'm going to get four terms that I've computed for the
past message.
I'm going to have four likelihood weights
corresponding to each of the four values
of this state variable.
So I've got that vector.
I already computed it as part of the computation to do the
APP for here.
It's the past part of it.
So now, I want to compute the probability that y7 is a 0.
The rule here is to take every combination in this code and
again, I'm going to get a--
well, let me write this part of it.
This is going to be p4, p5, p6, p7.
That's the probability of this four-tuple.
This is q4, q5, q6, q7.
This is p4, q5, p6, q7, q4, p5, q6, p7 and so forth.
That's the weights I want to assign to each of these.
So now I get a weight for each of these six-tuples.
I multiply this by this to get the weight, the likelihood
weight, for this code word.
This by this to get the likelihood weight for this
code word and so forth.
And I add these to buckets for--
I simply go through this and I'll add this one to the
bucket for y7 equals 0.
And I'll add this to the bucket for y7 equals 1.
And this to the bucket for y7 equals 1.
And this to the bucket for y7 equals 0 and so forth.
It's clear that in this bucket I'm always going to get the
same value for p--
I'm always going to get a p7 for 0 and a q7 for 1.
So just looking at y7, I don't have to compute the seventh
value here.
So if I did this separately as a little y7 variable, then the
incoming message, the intrinsic information, would
just be this p7, q7 likelihood weight vector.
And what I really want to compute is the outgoing part,
which has to do with y4 through y6.
So let me draw it that way.
This I don't think is very good exposition.
Bear with me.
So now I'm going to put this times this in my bucket for 0.
This times this in my bucket for 1, and so forth.
The question here is, does this method give me the same
answer as my exhaustive method up here?
And again, because of the Cartesian product character of
the states, you can convince yourself that it does.
In other words, I can use this as a summary of everything in
the past that got to the state value equal to 0, 0, 0.
Any time I got to 0, 0, 0, I got through one
of these two things.
And so for any possible continuation over here, I
could have got to it in one of these two ways.
And I'm going to find terms, e.g., these two terms.
Let's see.
What do I want?
I want two ways of getting to 0, 0, 0.
I'm sorry, these two.
I claim that this times these is equal to the
sum of these two.
And it is.
I can get to 0, 0, 0, 0, 0 either through this or this.
And I get to 1, 1, 1, 1, which is this one, either through
this or this.
And again, it's a fundamental property of the state that
whenever I get one of these, I'm going to get both of them
in equal combination.
So I can summarize things in this way, which leads to an
efficiency of computation.
So what I'm basically explaining to you now is the
sum-product update rule, which is how to propagate messages.
Propagating messages, which are APP vectors.
In general, my situation is going to be like this.
I'm going to have some constraint code, (n,k), I'm
going to have some incoming messages over here which tell
me about everything that's happened in this past, p
prime, and in this past, p prime.
Give me a summary of all the weights corresponding to that.
I'm going to have some output symbol over here and I want to
compute now what the APP vector is for the possible
values of this output symbol, which is going to be some
aggregate of this down here.
What I do is I go through all 2 to the k code words, call
this constraint code.
Sorry, I'm using k again.
That's 2 to the k code words.
And compute the product of the inputs.
Of input APP vectors according to the possible combinations
of the input variables and the output variables that are
allowed by this constraint.
That's explicitly what I'm doing here.
Here are all the possible combinations of state
variables and symbol variables that are allowed by this 6, 3
constraint code.
There are eight of them.
For each one, I'm going to compute the relevant product
of APP vectors and I'm going to add it to a bin.
So add to a bin.
Bins which correspond to--
draw it like that--
output.
I'm calling this the output variable values.
OK, and that'll give me the summary APP vector of the
outputs for now the past, which consists of everything
that could have happened in this side of the graph.
Again, I'm using the cycle-free assumption in
several ways.
I'm saying that everything that happens up here is
independent of everything that happens here.
Everything that happens completely in this past is
just the sum of what happens here.
What happens here are subject to this constraint and has
nothing to do with what happens
over here in the future.
Eventually I'm going to do the same thing in the future.
I'm going to compute a future component that's based on all
this stuff.
I'm going to get messages going in each direction and
I'm going to sum them up.
OK, now I'm seeing a fair amount of puzzlement on
people's faces.
Do you get abstractly what I'm trying to do here?
Does anyone want to ask a clarifying question?
So now--
AUDIENCE: [UNINTELLIGIBLE PHRASE]
the bottom right, output variable?
PROFESSOR: Output variable values, I'm sorry.
I think there's a little bit of confusion here because
sometimes I've considered this output to
be a 16-valued output.
Here, in fact, well, it has eight actually valid values.
And if I was considering it that way, then I would just
have one bin for each of these likelihoods.
But now if I consider these as four binary variables, then I
would aggregate these things according to the values of
each of the binary values.
And I'd get four vectors of length 2 for
the individual APP.
Which is ultimately what I want here, so I'm sorry I've
gone back and forth between subtly different realizations.
I do find this very hard to explain.
Perhaps there's someone else who can do a better job.
You have a homework problem which asks you to do this, and
I think having done that you'll then get the idea.
But it's a matter of going back and forth between the
gross structure and the actual equations and
just working it out.
All right, so the key things in the sum-product algorithm.
I've already explained some level to them.
We have this sum-product update.
That basically explains given messages coming into a node,
how do we compute the message going out of the node?
We take contributions, products corresponding to what
the code allows.
We dump them into the appropriate bins in the output
vector and that's the sum-product update rule.
We have the past future decomposition
or combination rule.
And that says at the end of the day, you're finally going
to get messages going in both directions.
And you just combine them component-wise as
we started out with.
And finally, we need to talk about the overall schedule of
the algorithm.
So let me draw an arbitrary cycle-free graph.
OK, suppose I have an arbitrary cycle-free graph.
It consists of external variables out here, internal
variables and constraint nodes.
How am I going to schedule these computations in order to
do APP decoding of this graph?
Well, what do I have at the beginning?
At the beginning, I measure the received variables
corresponding to each symbol, and that gives me the
intrinsic information, which is incoming message you can
consider from every symbol variable.
Basically, what did I see on the channel?
That's what that likelihood weight vector corresponds to.
All right.
What do I need in order to do the sum-product update rule?
I need the inputs.
Think of this now as a directed graph.
I need the incoming messages on all of the incident edges
on this node except for one.
If I have all but one, I can compute the last one as an
outgoing message.
So that's the structure of that update.
So at this point, can I compute anything here?
No, I would need the inputs.
But I can compute this outgoing message.
I can compute this outgoing message.
I haven't drawn these of very high degree.
Normally we have degree higher than two.
If I had another input here, however, I could still get
that output.
All right, so that's time one.
Think of there being a clock and at time one I can
propagate messages into the graph to
depth one if you like.
There's going to be a systematic way that we can
assign depths.
All right, now at time two what can I compute?
I can compute this guy because I have this input.
Now I have both inputs coming in here, I can
compute this guy.
Anything else?
So maybe 0, 1, 2.
I should put times on each of these,
so I don't get confused.
0, 0, 1, 0.
So this is 2 at the output.
Now I'm at time three, what can I compute?
At time three, I now have two inputs coming in here.
I can compute this output.
Actually, from these two inputs, I can compute this
output at time three.
And from these two, I can compute this outgoing message
at time three.
Are there other things I can compute?
Can I compute this?
No, I don't have this yet.
That's probably it.
Does anyone else see anything else I can compute at time
three based on the time zero, one, and two messages?
No.
All right, but I'm making progress.
And should be clear that I'm always going to be able to
make some progress.
At time four I can compute this
message in that direction.
Let's see, I now have everything I need to compute
this message in that direction.
And I'm almost done.
And finally, I can compute this message in this direction
at time four, the extrinsic
information for those variables.
I guess I can do this at time four.
I had that at time three.
And now at time five, I can get
everything else that I need.
Time five I get that out.
I get this out based on this input and this input.
And I get this out based on this input and this input.
So in a finite amount of time, in this case, five
computational cycles, I can compute all of the messages in
both directions for every variable in the graph.
Now as a final cleanup step, I simply component-wise multiply
the vectors in the two directions, and that gives me
the APPs for all the variables.
This is a kind of parallel computation of all the APPs
for all variables in the graph.
And I claim two things.
It's finite and it's exact.
And the exactness is proved from the equations and the
cycle-free assumptions.
So we can always decompose things as Cartesian products.
Independent components.
The finiteness, well, I assumed
this is a finite graph.
What is this number 5?
It's really the maximum length of time it takes me to get
from any of these external symbol variables
to any other external--
from leaf to leaf in graph theory terms.
And that's called the diameter of the graph.
And again, some people count the last little symbol node
and some people don't count the last little symbol node.
But it's clear that a finite graph is going to have finite
diameter and that the propagation time is basically
going to be equal to the diameter.
That's how long it takes to get across the graph.
So each of these messages in each direction
has a certain depth.
That's the distance in the graph to the most
distant leaf node.
And the maximum sum of these is always going to
be 5 in this case.
Notice that it is.
A tiny little graph theory theorem.
AUDIENCE: [UNINTELLIGIBLE PHRASE]
you see why if you sum these messages flying through those
past and future of the single edge, you would get the
overall APP because you're summing the contribution from
both the set of outputs and those [UNINTELLIGIBLE PHRASE].
How would you prove that that should be overall the APP of
the symbols?
PROFESSOR: Recall the state space theorem.
Let's take any edge in here.
Suppose we think about cutting it.
A cut-set through that edge.
Suppose we consider agglomerating everything back
here and calling that the past.
Agglomerating everything out here and
calling that the future.
This is past and future.
The idea of the state space theorem was that this is
exactly the same situation as a two section trellis.
Because everything is linear you get the state space
theorem basically, which says that you get a Cartesian
product of a certain past, a certain future.
You get a minimal state space, which is kind
of the total code.
The modulo, the part of the code that lives purely on the
past, and the part that lives purely on the future.
That quotient space is really what it is, corresponds to the
state space at this time.
Because every edge in a cycle-free graph is by itself
a cut-set, you can do this for every edge in
any cycle-free graph.
You can make this same argument.
So you get a well-defined minimal state space.
You get a Cartesian product decomposition, which has a
number of terms equal to the size of the state space, due
to the dimension of the state space over binary field.
And so everything goes through just as it did
in the trellis case.
That was the argument.
So mushing all this together, you get
the sum-product algorithm.
This is the fundamental reason why I took the time to go
through minimal trellis realizations of block codes.
Not because that's actually the way we decode block codes
all by themselves, or even use block codes.
We use convolutional codes.
It's so that we could prove these very fundamental
theorems about graph decoding.
And we aren't even going to decode cycle-free graphs.
But that's the intellectual line here.
Thank you for that question, it was very well timed.
AUDIENCE: [UNINTELLIGIBLE PHRASE]
PROFESSOR: Let's take a look at the structure of this.
Where do the computations occur?
Every node there's a computation.
What's the complexity of that computation?
It's somehow approximately equal to the size of the
constraint code.
We do one thing for every possible combination of
legitimate combination of variables, which is what we
mean by the size of the constraint code.
So if this is a 6, 3 code, we need to do eight things here.
Basically, eight multiplications.
So this, in more general cycle-free graphs, that's what
corresponds to branch complexity and trellis graphs.
We found that the branch space corresponded to a constraint
code in trellises.
So the computation is basically determined by the
constraint code complexity and really, overall what we'd like
to do is minimize the constraint code, the maximum
size of any constraint code because that's going to
dominate computation.
So we try to keep the k in these
things as small as possible.
Yes?
AUDIENCE: [UNINTELLIGIBLE PHRASE]
and you get larger dimension constraint for
[UNINTELLIGIBLE PHRASE].
PROFESSOR: Yeah, we're actually not too concerned
about diameter.
Because things go up exponentially with constraint
code dimension, just that single parameter tends to be
the thing we want to focus on.
But we remember that's hard to minimize in a trellis.
We showed that basically our only trick there after
ordering the coordinates was sectionalization and that we
could not reduce the constraint code complexity,
the branch complexity, by sectionalization.
And then again, we build on that and we went through this
argument that said, gee, this corresponds to some trellis.
We could draw a trellis where this, literally was the past
and this was the future.
And the minimal size state space in this cycle-free graph
would be the same as in that trellis.
What I guess I didn't show is you can't reduce the
constraint code complexity by significantly --
actually, I'm not sure there's a crisp theorem.
But the fact that you now are constrained to size of state
space, this is certainly going to be at least as large as
state space sizes.
Tends to indicate that you're not going to be able to do a
lot about constraint code complexity by going to more
elaborate cycle-free graphs than just the chain trellis
graph either.
That's, I think, a missing theorem that
would be nice to have.
I had a paper about a year or two ago, I'm not sure it ever
quite gets to that theorem.
But that's the theorem you'd like to see.
You really can't improve this.
So ultimately, we're bound by what you can do in a trellis.
And in a trellis you can't do too much about this branch
complexity parameter once you've ordered the
coordinates.
A bit of hand-waving here, but we're going to need to go away
from cycle-free graphs.
One reason I like normal graphs is that you get this
very nice separation of function.
You get the idea that nodes
correspond to little computers.
You could actually realize this with little computers.
Put a computer in here for each node.
Its job is to do the sum-product update equation.
And then, well, what are the edges?
They're for communication.
These are the wires that run around on your chip.
And so when we talk about state space complexity, we're
really talking about something like bandwidth.
How wide do you have to make these wires?
Do you have 6 bits, or 9 bits, or whatever?
So state space really has to do with communication
complexity.
Constraint or branch has to do with the computational
complexity, which sometimes one's more
important than the other.
Computational tends to be the thing we focus on.
But we know more and more as we get to more complicated
chips, communications complexity
tends to become dominant.
And this, of course, is the I/O. These are your paths
which go to the outside world, your I/O paths.
So it's really a good way to think about this.
And in fact, some people like Andy [UNINTELLIGIBLE]
and his students have experimented with building
analog sum-product decoders.
The general idea here is that you can compute an output as
soon as you have the two corresponding inputs.
Well, imagine another schedule where this thing just computes
all the time based on whatever inputs it has.
In a cycle-free graph, say you're in the interior.
Initially, what you have on your inputs is garbage.
But eventually, you're going to get the right things on all
your inputs.
And therefore, you're going to put the right things on all
your outputs.
So again, up to some propagation times, just
putting in non-clocked little computers here, whose function
is to generate the right outputs given the inputs in
each direction, is eventually going to be right in a
cycle-free graph.
And in fact, there are very nice little analog
implementations of this where it turns out the sum-product
update rule is extremely amenable to transistor
implementation nonlinear rule.
So you just put these in, little computing nodes, and
you put whatever you received on here as the received
symbols, and you let it go.
And it computes incredibly quickly for
a cycle-free graph.
Where we're going to go is we're going to say, well, gee,
we got this nice little local rule that we can implement
with local computers for the sum-product algorithm.
Maybe it'll work if we do it on a graph with cycles.
And on a graph with cycles this kind of parallel or
flooding schedule, where every little computer is computing
something at every instant of time, is probably the most
popular schedule.
And again, you build an analog implementation of that and it
goes extremely fast and it converges to something.
It just relaxes to something, there's a local equilibrium in
all these constraint nodes.
And therefore, some global APP vector that satisfies the
whole thing.
I think this is a nice attribute of this kind of
graphical realization.
All right, a particular example of how
this schedule works.
The cycle-free realization that we care most
about is the trellis.
The BCJR algorithm is simply the implementation of the
sum-product algorithm on a trellis.
SP on a trellis.
So generically, what does a trellis look like?
I guess I should allow for some nontrivial code here.
So all trellises have this same boring graph.
What is this schedule?
It's a cycle-free graph, so we could operate according to
this nice controlled schedule, compute everything just once.
And how does that operate?
We get intrinsic information that's measured at each of
these symbols here.
What did we receive on the channel?
We put that through a constraint code and we get
here, let's give this a name.
This is called intrinsic 0.
Here is alpha 1, which is just a function of the intrinsic
information at time zero.
And we combine that with the intrinsic information at time
one and we get alpha 2, and so forth.
So we get a forward-going path down the trellis.
If any of you knows what a Kalman filter or a smoother
does, this is really exactly the same thing.
This is finding the conditional APP probabilities
of this state vector given everything in the past.
In Kalman filtering or smoothing, it's done for
Gaussian vectors.
Here we're talking about discrete vectors.
But in principle, it's doing exactly the same thing.
It's computing conditional probabilities given everything
observed in the past.
And so the diameter here is just the length
of the trellis basically.
So this is i n minus 1, i n.
And here we have alpha n.
When we have alpha n finally in here, we can compute the
extrinsic information based on alpha n as just extrinsic n.
We combine these two and we're done for that guy.
Meanwhile, we have to compute the backward-going information
on each of these branches.
So we compute first b n based on i n.
Then we get b n minus 1 based on i n minus 1
and b n, and so forth.
We get a backward-going propagation of conditional
information.
Each of these betas represents the a-posteriori probability
vector given the future from it.
And finally, when we get down to here we can compute
extrinsic 0 from data 1.
And by this time, when we have both alpha 1 and beta 2 coming
into here, we can, of course, get extrinsic 1.
When we get both the alpha and beta coming in here, we can
get extrinsic 2, and so forth.
Of course, you can write this down as equations.
If you want to read the equations, you look in the
original Bahl, Cocke, Jelinek, and Raviv article in 1973.
This was again, a kind of lost paper because just for
decoding a trellis, the Viterbi algorithm
is much less complex.
And gives you approximately the same thing.
I mean, I talked last time how probably what you really want
to do in decoding a block code is maximum likelihood.
That's minimum probability of error in a block-wise basis.
What APP decoding gives you minimum probability of error.
If you then make a decision based on each of these
extrinsic information, or on the combination of these two
to get the overall APP vector, that's called a map algorithm,
maximum a-posteriori probability.
If you make a decision based on that, that will give you
the minimal bit error probability, but it may not
necessarily give you a code word, and it will not minimize
the probability of making any error, which is generally what
you want to minimize.
So for trellis decoding of block codes
this was not favored.
However, a block code as a component of a larger code,
you want the APP vector to feed off to something else.
And so with the advent of much larger codes, of which block
codes are components or convolutional codes, BCJR is
the way you want to go.
And some people say it's about three times as complicated as
the Viterbi algorithm.
All right, so you'll have an exercise with doing that in
the homework.
There is a closely related algorithm called the min-sum
algorithm, or the max-product algorithm.
And the idea of that is that you can really carry through
the same logical computations except when you get to one of
these combining operations, rather than doing a sum of
products, you can do a max of products.
And that the Cartesian product law still holds, the same kind
of decompositions that we have still hold.
For instance, in this example, to get the state 0, 0, 0
before what we said we're going to get an APP vector
where this is the past APP for the 0, 0 value of the middle
state vector.
We simply combine the likelihood weights of the two
possible ways of getting to that state and we sum them.
Instead of doing sum, let's just take max.
The maximum probability of getting--
in other words, what's the best way of getting here?
Same question as we ask in the Viterbi algorithm.
We'll let that be the likelihood weight for the
state vector.
Otherwise, the logic is exactly the same.
Similarly, coming this way, what's the max of these two?
OK, what's the best way of getting to the state vector in
terms of likelihood from the future?
And what do you know, if we combine these things at this
point, we now have a past vector and a future vector.
Let's take the max of this times the max of that.
That gives us the maximum likelihood way of going
through the 0, 0 value of the state.
If we do exactly the same logic for decoding this
trellis, we get a two-way algorithm.
It's like this.
It's like the BCJR algorithm, where at every point we're
computing the maximum likelihood.
It's not hard to show that what this gives you is that
each of these symbols out here, it gives you the symbol
that belongs to the maximum likelihood code word.
How does it differ from the Viterbi algorithm?
It gives you the same answer.
It gives you the symbols, one by one, of the maximum
likelihood code word.
It doesn't have to remember anything.
It doesn't store survivors.
That's its advantage.
It's disadvantage is it's a two-way algorithm.
You have to start from both ends and propagate your
information in the same way.
And for block codes, maybe that's OK.
For convolutional codes that's certainly not something you
want to do because the code word may go on indefinitely.
So the Viterbi algorithm gets rid of the backward step by
always remembering the survivor.
It remembers the history of how it got to where it is.
So that once you get the max, you don't have to--
you simply read it out rather than having--
this backward part amounts to a trace back operation.
That's probably much too quick to be absorbed.
But I just wanted to mention there is a max product
version, which if you take log likelihoods becomes a max-sum.
Or if you take negative log
likelihoods it becomes min-sum.
So this is called the min-sum algorithm.
And it does maximum likelihood decoding
rather than APP decoding.
And sometimes it's used as an approximation or very
intermediate approximations between these two.
AUDIENCE: [UNINTELLIGIBLE PHRASE]
PROFESSOR: No, if you use the min-sum, basically everybody
will converge on the same maximum likelihood code word.
So all these constraints will be consistent with--
you'd be taking a single max at each point.
But at each point you will solve for the maximum
likelihood code word.
And what you'll see up here is the symbol that belongs to the
maximum likelihood code word.
This is maybe a bit of a miracle when you first see it,
but you'll independently get each of the bits in the
maximum likelihood code word.
Check it out at home.
AUDIENCE: Why not always do this?
PROFESSOR: Why not always do that?
It turns out that we want softer decisions when this is
part of a big graph.
We actually want the APPs, the relative probabilities of each
of the values.
The max, in effect, is making a hard decision.
AUDIENCE: [UNINTELLIGIBLE PHRASE]
PROFESSOR: Yeah, all right.
It does give reliability information.
I'm not sure I can give you a conclusive answer to that
question except APP works better.
That's a more empirical answer.
Yes?
AUDIENCE: [UNINTELLIGIBLE PHRASE]
Where does that flexibility of the input symbols being
[UNINTELLIGIBLE PHRASE].
We should be able to take that into account
[UNINTELLIGIBLE PHRASE].
PROFESSOR: Yeah.
OK, well, if the input symbols are independently not
equiprobable, you can feed that in as--
you know, once they came out of the channel, once we see
the channel information, the two symbols are not
equiprobable anymore.
So if somehow they a priori independently were not equally
probable, you could just feed that in as part of this
intrinsic information vector.
That's almost never the case in coding.
The intrinsic information is sometimes regarded as a-priori
and this has a-posteriori, or the combination of the two of
them is then a-posteriori when this goes off somewhere else.
But it doesn't really mean that the symbol itself was--
it means that the evidence is biased one way or the other.
And when this appears somewhere else in the graph,
this will now appear somewhere down in some other graph.
It's called the a-priori information, but what it is is
the a-posteriori information given all of the received
symbols in this part of the graph.
AUDIENCE: [UNINTELLIGIBLE PHRASE]
Does the i0 of that other graph contain this table
sharing information added to it?
PROFESSOR: Yeah, the i0 for another graph is just the e0
of this graph.
I mean, if you consider it all as one big graph,
that's what you want.
You want the message that goes in this direction, which
basically summarizes all of the inputs from this graph.
Generally, you'd have a little equals node here.
You'd have the actual intrinsic information coming
from the outside world there.
You would compute the e0 from this graph.
And at this point, you would combine e0 and i0, and that
would go around here.
So the very last page of chapter 12 just says, OK,
we've got a nice, clean, finite exact algorithm that
will compute all these APPs on a cycle-free graph.
Now, suppose we're faced with a graph that has cycles.
And here's the situation.
Suppose the graph has cycles in it.
Then first thing that might occur to you, at least after
this development is, well, we now have a local rule for
updating at each of these computational nodes, each of
these constraint nodes, in this representation.
Why don't we just let it rip?
See what happens.
We're going to put in some intrinsic information at the
outputs as before.
We can compute some output message here based on-- well,
eventually we're going to get some input message from here.
And we can compute some output message here based on all of
these and we'll pass that over.
Let's assume what's called a parallel schedule or a
flooding schedule, where in each cycle every one of these
little guys does its thing.
It takes all the inputs that it has available at that time
and it generates outputs.
Well, hope for the best.
It will compute something.
You can sort of intuitively see that it'll settle.
It'll converge, perhaps, into some kind of equilibrium.
Or hopefully it will converge into some kind of equilibrium.
There is a basic fallacy here, which is that the information
that comes in here is used to compute part of this message.
Which is used to compute part of this message.
Which ultimately comes back and is used to compute part of
this message.
And then part of this message again, so that information
goes around in cycles and it tends to be used again and
again and again.
Of course, this tends to reinforce
previously computed messages.
It tends to make you overconfident.
You heard some rumor and then the rumor goes all around the
class and it comes back and you hear it again.
And that tends to confirm the rumor.
So it's that kind of telephone situation, and it's not really
independent information.
Intuitively, makes sense that if all of the
cycles are very large--
in other words, the rumor has to go to China and back before
you get it.
That it probably is highly attenuated by
the time you get.
It's mixed up with all kinds of other information, so maybe
it doesn't hurt you too much if the cycles are very large.
And in fact, that's the way these capacity approaching
codes are designed.
Whereas, as I mentioned last time, if you're in a physical
situation where basically your graph looks something like
this, you don't have the freedom to make your cycles
large, then this is probably a bad idea.
And this isn't why in fields, like vision, for instance,
people try to use this--
the sum-product algorithm is the belief propagation
algorithm, if you've ever heard of that.
Widely used in inference on Bayesian networks.
And the religion in belief propagation used to be that
you simply can't use belief propagation on graphs that
have cycles.
Why?
Because most of the graphs they dealt
with look like this.
And in fact, it works terribly on that.
It was a great shock on that field when the coding people
came along and said, well, we use belief propagation on
graphs with cycles and it works great.
And this, as I understand it, really had an enormous impact
on the belief propagation community.
But it was realized that gee, coding people have it nice.
You make your own graphs.
And you make them so this is going to be a very low order,
low impact event.
And that's right.
But coding we do have this freedom to
design our own graphs.
We realize from the basic arguments that I put up before
that we're really going to need to have cycles.
As long as we have cycle-free graphs, we're going to get
this kind of exponentially increasing complexity, which
is characteristic of trellis graphs.
So we're going to need cycles, but as long as the cycles are
very long and attenuated, as long as the girth of the graph
is great, maybe we can get away with it.
And that, in fact, is the way the story has developed.
And Gallager basically saw this back in 1961, but it took
a long time for it to catch.
So I didn't even get into chapter 13.
Chapter 13 we're going to first just introduce several
classes, the most popular classes of capacity
approaching codes.
And then I'm going to give you some real performance
analysis, how you would design,
simulate, in these codes.
And it's what people actually use.
OK, see you next time.