PROFESSOR: Chapter six, problem set three, problem set
two solutions.
OK.
Good morning.
Nice to see you all again on this mild day.
I want to start off by thanking Ashish greatly for
giving the last three lectures.
I have very high confidence in Ashish, and I'm sure he give
you excellent lectures and was able to answer all your
questions, right?
But if you feel dissatisfied at any point, you'll have an
opportunity to ask the question again today.
We will continue now in chapter six.
I understand Ashish got up to section six point one already,
so that makes my life a little easier.
And we'll just continue to plow ahead through the notes.
You have as handouts today chapter six, the outgoing
problem set, the solutions to the incoming problem set.
So does anyone have any comments or questions or
suggestions as we go forward at this point?
Let's say I'll do a very quick review of everything so far,
at least what we need to proceed, so you'll have a
chance to ask questions along there if you like.
Anyone have anything to say?
Anyone want to say thanks to Ashish?
All right.
Great.
OK.
So a very quick review.
We first said we were going to work with a continuous time
Additive White Gaussian noise channel, but we quickly
reduced it to a discrete time additive white
Gaussian noise model.
Vector model, Y equals X plus N, where the vectors may go on
as long as you like.
And implicit in here is we have some kind of power
limitation expressed by various --
I know I've probably introduced too many parameters
in the early part of the course.
But you'll see them all in the literature, and I thought you
might as well see the relation between them all at once.
The noise.
The main thing about this is it's an independent,
identically distributed noise, also independent of X, and
it's characterized simply by its variance per symbol, or
average power.
Again, there are various notations that we
can use for its power.
OK.
However you express the power of the signal of the noise,
you're going to get some signal-to-noise ratio out of
that, as long as you use consistent the same units for
signal and noise.
And so the key parameters of this channel turn
out to be just two.
One is the signal-to-noise ratio.
And the other is something which in the discrete time
domain we might think of as a data rate, rho, the number of
bits per two dimensions.
But we were at pains to show --
and the reason we measure it in bits per two dimensions is
that it converts directly to the spectral efficiency, or we
sometimes say, the nominal spectral efficiency, or the
Nyquist spectral efficiency.
Basically, this is the spectral efficiency if you
assume that you only use the nominal Nyquist bandwidth.
And both in 450 and here it was shown that you can really
get as close to the nominal bandwidth as you like, its
sharp or roll off as you like, by using filters.
Very sharp filters or very sharp digital filters that
today we can really realistically get effectively
to the Nyquist bandwidth.
So this is a good measure of nominal spectral efficiency,
which is an extremely important parameter, the
spectral efficiency in continuous time.
And then the Shannon limit is expressed in this very simple
form that we can never do better in data rate or nominal
spectral efficiency than log of 1 plus SNR.
And the units are bits per two dimensions.
So there's an awful lot about parameters and units in the
first part of the course, and sometimes it's a little
overwhelming and confusing.
But in my experience in engineering is, it's extremely
important to get the units right.
It helps you in thinking clearly.
It helps you to focus on the right things.
And so in the first couple of chapters, there's a lot of
emphasis on getting the units right.
For instance, bits per two dimensions.
I could simply assert to you that things are always to come
out better if you do things per two dimensions.
This basically has to do with Gaussian distributions being
simpler in two dimensions than they are in one dimension.
So you know you can get closed form integrals in two and four
and so forth dimensions that you can't get in one and three
and so forth dimensions.
So you know, the geometrical formulas for the volume of the
sphere and so forth are much simpler in even dimensions
than they are in odd dimensions.
This all, I think, has to do with the fact that really a
one-dimensional complex Gaussian variable, or in
general, complex Gaussian variables, are in some sense
mathematically simpler than real Gaussian variables.
For instance, the Q function is a very ugly thing.
In two dimensions, you can closed-form integrals with
probability of error.
So there's a lot of clues that we really want to think in
terms of pairs of real dimensions.
All right.
And then we introduced a couple more parameters.
And I understand Ashish got the questions
that I get every year.
Why do we introduce both of these things?
SNR norm, which is SNR normalized by 2 to the rho
minus 1, that's motivated directly by the
Shannon limit formula.
Or Eb over N_0, which is SNR divided by rho.
You know, these are both more or less
equivalent to each other.
From this Eb over N_0 is just 2 to the rho minus 1 over rho
times SNR norm, so why do we introduce both of them?
And there's not an intellectually very strong
argument to introduce both of them.
Because one is easily translated into the other.
For fixed nominal spectral efficiency rho, there clearly
is just a one-to-one translation.
We could use either one.
If you graph something versus Eb over N_0 or versus SNR
norm, it's just a matter of shifting it by this factor.
They're going to be exactly the same graph with the 0 dB
point in a different place, according to where 0 dB is.
Philosophically, of course, SNR norm, which is sometimes
called the gap to capacity, is exactly a measure of how many
dBs away are we from the Shannon limit.
Measuring things on a log scale in dB.
And so 0 dB is always the Shannon limit point with SNR
norm, because this statement translates into SNR norm is
greater than 1, which is 0 dB.
So here the focus is always, how far are you from capacity?
How far are you from the Shannon limit?
And that really is the modern view.
In the early days of coding, the view was, well, how much
coding gain can we get over no coding?
And as we'll see, Eb over N_0 is often a very convenient
parameter to use when we're focusing on coding gain.
In fact, for binary linear block codes, which is what
we're talking about in chapter six, we get an extremely
simple expression that Eb over N_0 it's just kd over N, where
N, k, d are the basic parameters of a block code.
If you don't know those yet, you will by
the end of this lecture.
And Eb over N_0 is what was put forward in the early days,
when the principal coding application was coding for
deep space communications.
It makes sense in the power-limited regime.
In the power-limited regime, we showed that this
essentially goes -- this is rho log2 over rho, so up to a
factor of natural logarithm of two, these are the same,
almost independent of the spectral efficiency, as well.
And so this is a very natural thing to use, especially in
the power-limited regime.
In the bandwidth-limited regime, as rho gets large,
then these things become very different.
SNR norm always keeps us in the neighborhood of 0 dB.
This thing goes up to 10 dB, 20 dB, 30 dB, and so forth.
Nonetheless, you'll see in some literature, people
continue to measure against Eb over N_0 maybe I would say,
just because they don't know any better.
Anyway, since I started writing these notes about
eight or nine years ago, I've been advocating SNR norm.
It's more and more widely used in our business,
in the coding business.
Or equivalently, one always shows nowadays how far are you
from capacity, and that's what SNR norm is about.
And you can always translate this into this and this into
this by this simple formula.
So that's why we introduce them both.
Eb over N_0 is traditional.
SNR norm is more of the modern gap to capacity viewpoint.
Any questions about that?
Because I understood that Ashish got a number of
questions about that.
Yes?
AUDIENCE: And the probability of error you mentioned
[UNINTELLIGIBLE] is natural with SNR model.
PROFESSOR: Well, this is a slightly
different point, actually.
So we go on from this to talk about the power-limited
regime, which we defined more or less arbitrarily as the
regime where rho is less than or equal to two bits per two
dimensions, and the bandwidth-limited regime,
which is where rho is larger.
And at this point, I simply assert that it's better to do
everything per two dimensions in the bandwidth-limited
regime and to do everything per bit in the
power-limited regime.
And the reason for this is basically long practice and
intuition and experience that things do work out better, and
this is the proper normalization.
But I think at this point in the course, with your
background, this is only an assertion, all right?
So I simply say, this is the way we're going to do things
in bandwidth-limited and power-limited.
For most of the rest of the course, we're going to be in
the power-limited regime.
Then we'll come back to the bandwidth-limited very late in
the course.
So you can now forget this assertion for a while.
Yes?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Well, I'm trying to focus on that here.
That if two systems have different rhos, then you
should take that into account in their comparison.
And when you see charts of a rate 7/8 system, or a rho 7/4
system versus one that's rate 1/8, those are very different
regimes, and it probably isn't fair to compare two systems,
just in terms of their probability of error versus Eb
over N_0 at two different spectral efficiencies.
What is more fair is to compare them in terms of their
gap to capacity.
You can get more powerful codes with more
error-correcting capability the lower in rate that you go.
And so you should start from the point of view that two
systems with different rho are incomparable, and then using
this you can say, but, you know, if we can get within 1
dB of the Shannon limit with both of them, then they're
both approximately equally powerful.
That would be the modern point of view.
But it's really apples and oranges if they have different
rates, different rhos.
Yes?
AUDIENCE: [INAUDIBLE]
If [UNINTELLIGIBLE]
systems modulation [INAUDIBLE].
Then is it fair --
because if one of them has coding --
PROFESSOR: If none of them have coding, well --
AUDIENCE: No, just modulation.
PROFESSOR: So what shall I say to that.
In the power-limited regime, an uncoded system is simply
binary modulation or 4-QAM, and there aren't different
systems to compare, really.
There's only one baseline system.
As you go up into the bandwidth-limited regime, then
it is fair to compare systems of the same
class, like M by M QAM.
That's a simple uncoded system.
Now there, rho keeps changing.
rho equals 2, rho equals 4, rho equals 6, rho equals 8.
Or you can easily find intermediate
rates that are uncoded.
And there you find that this normalization makes all these
systems comparable.
In fact, we saw that we got a universal baseline curve for
the bandwidth-limited regime, which was independent of rho,
for, say, the class of 4 by 4, by the class of M by M
bandwidth-limited QAM systems.
I'm sorry.
I got off a plane at nine o'clock last night, so I may
not be totally coherent today.
But all of these systems have exactly the same baseline
curve up to the approximations we've made that their exactly
four nearest neighbors and so forth.
But on a curve of probability of error per two dimensions,
per QAM symbol, versus SNR norm, we
get a universal curve.
So that indicates they really are comparable.
They form a class where the performance of the class is
independent of rho.
And that's sort of typical of the bandwidth-limited regime.
Two systems that differ, that basically use the same code
with smaller constellation or a larger-based uncoded
constellation are going to be directly comparable, and are
probably going to have the same gap to capacity.
Going to be just as far away from the Shannon limit.
But we don't have that phenomenon in the
power-limited case.
Other questions, or shall I proceed?
OK.
Again continuing what I hope will be a quick review, but I
don't want to go any faster than you're comfortable with.
We get this baseline curve for 2-PAM or 2
by 2 QAM, same thing.
So this is the baseline of Pb of E on a log scale, where we
typically go down to ten to the minus six, ten to the
minus five.
Here's 0 dB.
This goes down through about 9.6 dB here or
about 10.5 dB here.
And the ultimate Shannon limit is over here.
The ultimate Shannon limit for very low rho, as rho goes to
0, is about minus 1.6 dB.
And we get expressions.
0 dB is the Shannon limit at rho equals 1.
At rho equals 2, it's up about 1.8 dB.
Shannon limit for rho equals 2, and so forth.
And anyway, so we see as a function of rho, we can
measure the gap to capacity at different probabilities of
error and see how much coding gain is,
in principle, possible.
And then for a coded system, we can put that on here.
The effective coding gain is the distance
between here and here.
At some target probability of error.
It's going to differ according to probably of error.
You found a way of getting a good, rough-cut estimate, at
least for not very complicated codes called the union bound
estimate for any signal constellation in any number of
dimensions.
So if we have a signal constellation A, which
consists of M points in N dimensions, so forth,
basically found that we could, in the power-limited regime,
get an approximate expression from considering pairwise
error probabilities that's very simple.
And I used this notation.
Q of the square root of some coding gain of the
constellation times 2 Eb over N_0.
If it's just 2-PAM then the coding gain becomes 1.
This is the average number of nearest neighbors per
transmitted bit.
And so the whole thing reduces to this expression in terms of
couple of parameters.
the principal parameter is the nominal coding gain, which is
the minimum squared distance of A over 4 times the energy
per bit of this constellation A. So we really only need to
know this kind of normalized measure of goodness of the
constellation.
And K_b of A Is the average number of nearest neighbors.
I forget what we call it.
What is the numerator here?
K_min of A, which itself is an average, over the number of
bits that we're actually sending, which is log2 of the
size of A.
So basically we only need to know a couple of parameters of
this signal constellation.
Its minimum square distance is very important.
Energy parameter, which we choose to make the energy per
bit, or so that we get this expression.
And the average number of nearest neighbors per bit that
we transmit.
OK.
And our best example so far is the tetrahedral constellation,
where we basically pick every other
point from the 4 simplex.
Maybe I should do that in a different color.
I don't think that's a different color.
Nope.
Anyway.
You know this quite well by now.
And if we do that, we find that normalizing everything,
this is 4/3 or 1.25 dB.
And every point has three nearest neighbors, and we're
sending 2 bits, so this is 3/2.
So then we get this handy dandy engineering rule to make
a quick plot of the union bound estimate.
Given that we decided to put this on a log-log scale to
start with, all we have to do to plot this expression --
if we're given, as we always are, the baseline curve, which
is simply Q to the square root of 2 Eb over N_0.
How do you convert this curve to this curve
on a log-log plot?
Well, you simply move it to the left by the coding gain.
And you move it up by the log of whatever K_b is.
So if the dominant effect is moving it left, in this case,
by 1.25 dB --
I'm going to wind up getting about this curve, so I won't
draw it again --
then we move it up by a factor of 3/2, we developed a rule of
thumb that said a factor of two is basically going to cost
you about 0.2 dB.
Around ten to the minus five.
This is just based on the slope of this baseline curve,
as long as we're somewhat in that region.
So this will cost us about 1/10 of a dB, so we'll get an
effective coding gain of about 1.15 dB.
So graphically, we take this curve bodily.
We move it over 1.25 dB, and we move it up
by a factor of 3/2.
And what we'll find is the effective coding gain is
thereby reduced to, I estimate, about 1.15 dB.
And this is all just engineering rules of thumb.
Nice template.
I sometimes say you should fill out a copy of this
baseline curve, cut it out, put it in your wallet for the
duration of this course, because you'll have the
opportunity to make this kind of
calculation again and again.
All right.
So that's union bound SNR.
For simple constellations, the union bound
estimate is very accurate.
So this is a very good way to proceed, from an engineering
point of view.
We want to write a paper?
You know, it's a little late to write a paper on the
performance of the four simplex signal constellations,
but if you wanted to, and you wanted to have one graph in
that paper, you would have the graph of --
well, you would actually probably compute the exact
error of probability, either by analysis or by Monte Carlo
simulation, you would put that in there.
But in your first draft of the paper, you would put in union
bound estimate and you would find that wasn't far off.
OK.
Any questions on that?
This has basically got us up to chapter six.
So in chapter six now, which Ashish got into last time.
This is basically about binary linear block codes.
And well, we first just start talking about
binary block codes.
That's what Ashish did last time.
We basically take 0 and 1 as our binary alphabet.
We take a blocks of length n.
Sorry.
Are we using little n or big N?
Little n, where n is called the block plank.
So we take the set of all binary symbols of length n,
and we're going to convert this to real n space.
How do we convert it?
Component-wise, by the standard 2-PAM map.
We map 0 into plus alpha, 1 into minus alpha.
So this maps into plus or minus alpha to the n.
So the entire universe of possible code words that we
have is this set of 2 to the n real n-tuples, which are
simply the vertices of an n cube of size 2 alpha,
obviously, right?
Just like this.
We have eight possible vertices in 3-space.
They obviously all have the same power.
They all lie on the surface of a sphere of squared radius n
alpha squared.
So they're not only the vertices of a cube, they're
vertices that are spread on an equal energy sphere.
And the whole idea is that our code is going to be some
subset of this, and the code will map under the same map,
which we call S, into some subset of the
vertices of the n-cube.
OK?
And again, our favorite example is the tetrahedron.
For instance, if we take the code as being 0, 0, 0, 0, 1,
1, 1, 0, 1 1, 1, 0, those for binary three-tuples, this maps
into T, the tetrahedron.
OK?
Well, we get this signal structure, which we've already
found the coding gain.
It has a little bit of coding gain.
We actually accomplished something.
All right.
So that's the basic idea that we go through in 6.1.
That's our favorite example.
And Ashish also shows you that, you know, you might
think this is an awfully restricted way of designing
constellations.
But when we're talking about low spectral efficiencies, by
going through the capacity calculation, we can assure
ourselves that in principle, this is not going to be very
sub-optimal for it.
Capacity is about the regime where n gets very large for
very long block codes.
We don't lose much in potential coding gain, or in
the Shannon limit, equivalently, by restricting
the input output input alphabet to be just these two
numbers, plus or minus alpha, I should say, rather than
using the whole real line as an input, as long as the
nominal spectral efficiency is less than or equal to one bit
per two dimensions.
And the exact numbers --
at rho equals 1, you lose about 0.2 dB, in principle,
using the Shannon limit as your guide.
And for lower rho, it just goes to 0.
And you did this last time.
OK.
So it's a very reasonable, and obviously attractive from an
implementation point of view, way of designing signal
constellations.
And again, this is basically all we're going to be talking
about for the majority of the course, is signal
constellations like this.
But of course as we get to codes that are thousands of
bits long, or perhaps even infinitely long, as in the
case of convolutional codes and trellis codes, sometimes
you forget that we're really talking about just putting
points in Euclidian signal space.
Because we're going to be very abstracted back into the
binary domain.
But in this course, we're always thinking of codes as
means of designing signal constellations.
Of course, codes are used far more widely than just for
signaling over the additive white Gaussian noise channel.
I've sort of packaged this course as a search to get to
capacity, we have added the additive white
Gaussian noise channel.
First of all because this corresponds
very closely to history.
Within this package, we can talk about all of the
principal classes of codes that have been developed to
date, and we can compare them in some performance terms.
How close do they get to capacity on the additive white
Gaussian noise channel.
So you get most of what you would get if you took a
communications-free view of coding theory.
You know, people are interested in codes for
computer memories, for God knows what,
lots of other things.
And for many of these other applications, you are going to
be interested in the same class of codes.
Basically, you're going to want to maximize the distance
between code words for a given rate, which has to do with the
size of the code.
And so you're going to be interested in the same codes.
Putting it in the framework of getting to capacity on the
additive white Gaussian noise channel gives a motivation,
gives a very nice story, because over 50 years, we were
able to get to capacity, and gives it a real
communications flavor.
Some of you, I'm sure, are here without any interest in
communications whatsoever.
You simply want to know about coding.
You will still get that story, but you'll get it
in this nice package.
That's why I do it this way.
OK.
I'm about to get into new stuff.
End of review.
Any questions?
So you must have done a great job.
Everyone understands perfectly.
All right.
So now let's talk about, as I said, binary
linear block codes.
When you see the word "linear," that's a signal that
there's some algebra ahead.
And so this is the very first point at which we get into
what's called algebraic coding theory.
The algebra will be extremely simple at this
point, not to worry.
And what are we doing here?
First thing we do is to identify 0 and 1 with the
binary field, which I'm always going to write as F2.
The older way of writing this is GF(2), for Galois field
with two elements.
They mean exactly the same thing.
Nowadays most people write just F2.
Now we have [UNINTELLIGIBLE], we can write this blackboard
F.
And OK.
Step one.
We've identified our alphabet with a field.
So algebraically, this is a field.
Some of you know exactly what I mean by that.
Others don't.
We'll come back to this again.
Informally, I would say a field is simply something
where you can add, subtract, multiply, or divide.
Our best examples of that before now have been the real
and complex fields.
There is a more formal definition of that.
In this case, we have only two elements.
So let me just write down the tables, which you all know,
regardless of your background.
What's the addition table for this field?
Well, 0 is the additive identity.
So we know that 0 added to anything gives itself, so that
gives us three of the entries of this table.
What do we put down here?
There's really only one choice to satisfy one of the axioms
of the field, which is that under addition, the field must
form a group.
In particular, that means that all elements are invertible.
That means that each row or column has to be a permutation
of the group elements.
And so the only possibility here is to put in a 0.
And well, we are forced, if we're going to make 0 and 1
into a field to find an addition table which is the
table of mod2 addition.
You've had this stated as an axiom before.
You can derive it just from the fact that 0 needs to be
the additive identity, and then we need to put a 0 in
here in order to get invertibility.
Under multiplication.
What is the multiplication table?
Well, the additive identity is also a nullifier under
multiplication in any field.
So 0 times anything is equal to 0.
1 times anything is equal to itself.
So that completely determines the multiplication table.
Again, just the table of mod2 multiplication.
So if I'd done this axiomatically, I would have
given you the axioms of field.
Then I would show that under this binary addition
operation, this multiplication operation,
we satisfy the axioms.
I'll do that as we get into chapter seven.
OK.
So now we have 0 1 to the n, n-tuples, binary n-tuples, we
will now regard as n-tuples of field elements, F2.
And these will be regarded as vectors in a vector space.
Well, I'll just say that F2 to the n is a vector space, which
clearly has 2 to the n elements.
Now again, informally.
What's a vector space?
A vector space is always over a given field.
In this case, it's going to be over the
binary field, of course.
F2.
The given field is called a scalar.
Just like vector space over the reals, the scalars are
real numbers.
Here the scalars are elements of F2.
And what do we have to have to make something algebraically a
vector space?
We have to have that the addition of two vectors is
well-defined and gives another vector, and that the
multiplication of a vector by a scalar is well-defined and
gives another vector in our vector space.
All right?
OK.
So how are we going to define that to addition?
If we want to add two n-tuples --
again, you aren't going to see anything here that you haven't
already seen in some other context.
What do we do?
We add them component-wise, using the component-wise rules
of field addition.
OK.
So we just add two vector n-tuples,
component by component.
We obviously get some result, which is itself a binary
vector or an F2 vector.
No problem there.
Except that --
well, all right.
As long as we're talking about all possible n-tuples, the
result is certainly in the vector space, right?
We add two binary n-tuples, we get a binary n-tuple.
So the result is in F2 to the n.
This is going to be the key test when we get to subspaces
of F2 to the n.
OK.
And multiplication is even easier.
How do we define multiplication by scalars?
Well, we only have two scalars --
0 and 1.
So 0 times any vector is going to equal the all 0 vector.
Again, you can view that as just component-wise
multiplication of everything by 0, and since 0 times
anything equals 0, we're always going
to get the 0 vector.
All right?
So is the 0 vector in the vector space?
Well, yes.
If we're talking about several n-tuples, it of
course always is.
And one times anything, again, we can just do this
component-wise, and it just gives itself.
Trivially.
So since this was a vector in the vector space, this is
certainly a vector in the vector space.
OK.
This seems pretty trivial so far.
But what's a binary linear block code?
Again, focusing on the linear, is a --
now I'll give a formal definition.
it's a subspace of F2 to the n for some n
called the block length.
All right.
What do I mean when I say a subspace?
I mean subsets of the elements of a vector space that itself
forms a vector space.
So in this case when I say subset, I mean a
set of binary n-tuples.
OK?
That itself forms a vector space.
OK.
What are the components of that?
To check that it forms a vector space, let's see.
Multiplication by scalars is, again, easy to check.
If it's going to be a subspace, then the all 0 --
so it has to contain the all 0 vector in order that when I
multiply by the scalar 0, I get another
element of this subspace.
Multiplication by 1 is always trivially satisfied.
If I start off with a set, I multiply by 1, I'm going to
get the same set.
So let's check whether the elements of a subspace are
closed under vector addition.
What do I mean by that?
I mean if you add two elements of the of the subspace
together, you get another element of the subspace.
That's the key property to check.
Key property -- we can write that as closure
under vector addition.
Which is also called the group property.
It means that just under addition, under vector
addition, the set of elements that you have forms a group.
You add any two elements, you get another
element of the subset.
The example is our favorite example so far.
Let's take this little code, and I'm going to ask.
Is that a subspace of F2 to the three?
So does this equal subspace of the set of all binary
three-tuples?
Anyone care to hazard a guess whether it is or isn't?
AUDIENCE: It is.
PROFESSOR: It is?
Why?
AUDIENCE: It has the all 0 vector.
PROFESSOR: It has the all 0 vector, first of all.
Good.
AUDIENCE: [INAUDIBLE]
PROFESSOR: And it's closed under addition.
Now how might we see that?
You could, of course, just take all pair-wise --
you could form the addition table of these four elements,
and you would find that you always get another one of
these elements.
For instance, 0, 1, 1 plus 1, 0, 1, is equal to 1, 1, 0.
In fact, you easily see that you take any two of these, add
them together, you get the third one.
If you call this a, b, and c, a plus b plus c equals 0,
addition is the same as subtraction, because we're in
a binary field.
So that means that a plus b equals c, a equals b plus c, c
equals b plus a, whatever you like.
And of course, if you add 0 to anything, it's trivially
closed under that.
All right.
So it is.
It satisfies --
it's all you've got to check.
A more abstract proof of this would be, this is the set of
all even-weight three-tuples.
If I add even to even, I'm going to get even.
So of course my result is going to be another
even-weight three-tuple, therefore in the set.
That's the more algebraic proof.
All right.
Suppose I just add 0, 0, 1 to all of these things.
I'll get the set of all odd-weight n-tuples.
Add any odd-weight n-tuple to this.
Let me take the C prime, which is the set of
all odd-weight n-tuples.
Is that a vector space?
It doesn't have 0, and in fact, it's not even closed
under vector addition.
All right.
If I take in C double prime is equal to 0, 0, 0 0, 1, 1, 1,
0, 1 and I stop there, is that a subspace?
No.
Because?
Not closed.
For instance, if I add these two together, I would get
that, and that's missing.
OK.
So actually, everything is much simpler when we're
talking about finite fields.
All the finite dimensional vector spaces consist of a
finite number of elements.
It's easier than real and complex vector spaces.
There's no analysis involved.
So forth.
All right.
So a binary linear block code is in a subspace
of F2 to the n.
What are some of the key algebraic facts we know about
vector spaces from our study of linear algebra, which I
assume all of you have had in some form or another?
What's a key algebraic property of a vector space?
What's the very first property?
All vector spaces have a --
AUDIENCE: [INAUDIBLE]
PROFESSOR: Norm?
No.
Dimension.
Dimension, that's where I'm going.
What's the significance of dimension?
AUDIENCE: [INAUDIBLE]
PROFESSOR: The definition of dimension is the number of
generators in any basis.
So we're talking about generators of the vector
space, or a set of generators which form a basis.
If you think we have the same properties here in the case of
vector spaces over finite fields, well, we probably do.
But let's see how it works out.
So we're talking about things like
generators, basis, dimension.
These are all closely interlinked properties.
Again, the fact that everything is finite here
gives us very elementary ways of addressing
all of these concepts.
We don't have any geometry yet.
We don't have norms.
We don't have distances.
We don't have angles.
We'll talk about that in a minute.
Here we just have a set that basically has these two
properties.
It contains the all-0 vector, and it's closed
as the group property.
It's closed under vector addition.
All right.
So the first property is that the all-0 vector is always in
the subspace, which I'll represent by C, meaning code.
So when I talk about a code now, I'm talking about a
binary linear block code, which by definition is a
vector space, a subspace of F2 to the n, where
n is the code length.
All right.
Let me try to find a set of generators for the code.
All right?
If somebody gives me a code, they say, this is a binary
linear block code.
Let me see if I can find a set of generators for it.
If I find three generators, then I'll know the
dimension is three.
That's basically where I'm going.
I'll state this a little bit more formally as we go ahead.
So suppose I'm given a code.
I know it's a binary linear block code, so I know it has
the all-0 element.
So how might I go about finding a set of generators?
Let's just take a greedy algorithm.
All right?
Suppose the code contains only the all-0 vector.
Is that a subspace?
What's its dimension?
0.
All right?
So that's a very trivial vector space, but it's a
vector space.
Satisfies the axioms.
Suppose it is not the trivial code.
That means it has more than the all-0 vector.
So I take as my first generator any non-zero vector.
OK?
I can always do that.
Don't have the axiom of choice involved here, because
everything is finite.
So I'm going to take g1 to be any non-zero vector.
Now I've got a generator.
And how many code words does it generate?
I want to take the set of all binary linear combinations of
all the generators that I have so far.
At this point, the binary linear combinations are 0
times g1 and 1 times g1 And that just gives me this word
and this word.
So now I have counted for two words with one generator.
Could it be that that's the whole code?
Sure.
The all-0 vector and any non-zero vector together form
a one-dimensional subspace.
That's all you can get from one dimension.
So I could be finished here now.
But if I'm not finished, there's still more code words
that I haven't accounted for.
Then I greedily pick a second generator.
So this is now, let's say, any vector not generated by g1.
So I have a branch here.
Either I've finished or I can pick another generator g2.
Now with g1 and g-two, how many vectors can I generate?
Let me take all binary linear combinations.
So a binary linear combination is any vector of the form
alpha1 g1 plus alpha2 g2 where these are both scalars.
And therefore this can be 0 or 1, this could be 0 or 1.
So what have I got now?
I've got 0, g1 g2 and g1 plus g2 I've got four binary linear
combinations of two generators.
Could that be the whole code?
Certainly.
At this point, again, consider our standing example.
I start out what I take as my first generator.
Let me take g1 equal 1 0, 1, g2 equals whatever, 0, 1, 1.
Then if I take all binary linear combinations of these
two generators, I'm going to get the whole code, right?
These four code words.
They can all be expressed in this form.
Or I'm not done, and then I have to pick g3.
And how many binary linear combinations are there of g1,
g2, and g3?
Eight.
Are they all necessarily in my subspace?
Yes, by the fact that the subspace is closed under
scalar multiplication.
Alpha g1.
Alpha1 g1, alpha2 g2 alpha3 g3 are all in the subspace.
Any vector addition of any of these scalar multiples is in
the subspace.
So I get now eight possible elements of the subspace, and
I either may be done or not.
Continuing in this way, I get some number gk of generators,
just by picking greedily the next one until I'm done.
All right?
When I'm done --
so I have to stop.
Why do I have to stop?
All right.
Let's look at the size.
At each point here, this accounts for two code words,
this for four, this for eight, this for 2 to the k.
How many binary n-tuples are there?
2 to the n.
All right?
So I clearly can't find more than n generators.
More more than n independent generators, in the sense that
the set of all the binary linear
combinations are distinct.
All right.
So k is, at most, going to be n.
So I will stop in a finite number of steps.
I'll stop at some number k.
When I've stopped, that's because the code consists of
all binary linear combinations of these k generators, and
therefore has size 2 to the k.
So the only possible size of a subspace is 2 to the k for k
less than n.
A power of 2 where the power is, at most, n.
So any subspace besides the subspace --
I'm repeating myself.
OK.
So this means I found a basis.
Does this mean that all possible bases of any subspace
have the same size?
Well yeah, it must.
I mean, I've proved now that any subspace has to have this
size 2 to the k for some k.
So obviously, if I go through this process, no matter how I
choose my generators, I could choose any pair of these
non-zero n-tuples as my generators.
So that would be a legitimate basis.
Take any two out of these three.
But it's always going to take exactly two of them, right?
Why?
Because the size of this subspace is four.
Two to the two.
So if I go through this process, I'm always going to
come up with k generators, where k is the log to the base
2 of the size of this code that I was given, which I was
told was a linear code, meaning it's a subspace.
So I'm somewhat free to choose the generators, but I'm always
going to come up with k of them if the code has
size 2 to the k.
So a basis is a set of k linearly independent k-tuples,
where linear independence has the same meaning here as
you're accustomed to, meaning that all linear combinations
of the k generators are distinct.
So I get 2 to the k distinct elements of the code.
And I say the dimension of the code is k.
In this case, basically it's the size of the code is 2 to
the k, then its dimension is k.
Has to be.
And all basis have k generators in them.
And there are, in general, many ways to pick them.
All right?
So just by considering this greedy basis construction
algorithm, I basically find the size of any subspace is a
power of two, and the power is equal to the dimension.
And any basis is going to have that cardinality.
Are you with me?
This is a pretty simple proof.
And I call this an n, k binary linear block code.
n being the length.
That just means that every code word is an n-tuple over
the binary field.
And k is the dimension.
So an n, k binary linear block code has size 2 to the k.
That's 2 to the k distinct code words.
Main example is this guy.
Easy?
Any questions?
I think this is clear.
AUDIENCE: Can n use the number of code word, I take
like 2 to the k?
PROFESSOR: n is something I specify a priori as the length
of every vector in the code.
In other words, it has size 2 to the k, and it's a subset of
the set of all binary n-tuples, which
I write like that.
In other words, the elements of the
code are binary n-tuples.
If I write them out, each one has length n.
OK?
We're good?
What are some other examples?
The simplest codes you can think of is, first of
all, an n, 0 code.
That means a code with dimension zero, has size what?
1.
And what does it consist of?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Yeah.
So this is the so-called trivial code, just containing
the all-0 word.
Has to mention 0.
It is a binary linear block code, but it's not much use
for communication.
So but nonetheless, we include that in this family.
Another trivial one is n, n.
What's that?
AUDIENCE: [INAUDIBLE]
PROFESSOR: F2 to the n, right.
What's its size?
2 to the n.
That means it has to contain all distinct binary n-tuples.
So this is called the trivial code.
This is called the universe code.
Contains the entire universe of binary n-tuples.
Let's get some slightly less trivial ones.
n, 1.
What would that be?
An n, 1 code.
What's its size?
2.
What does it consist of?
The 0 word, and?
Any other non-zero generator.
And that's correct.
This can be 0 and any generator, two words.
In communications, where we want to maximize the distance,
in some sense, between the two code words, what is g
always taken as?
0,1's, right.
So if it's in particular the all-0 and the all-1 word,
which I might write as a vector of 0's and a vector of
1's, this is called the repetition code.
The binary repetition code of length n.
It either gives me a 0 and I repeat it n times, or gives me
a 1 and I repeat it n times.
So whenever you see n,1, you can pretty well assume it's
the repetition code, though it might be any pair 0, g.
And n, minus 1.
This is an interesting one.
Again, while this could be a lot of things, in
communications, whenever you see this, this will always be
the set of all even-weight n-tuples.
In other words, the set of all n-tuples with even parity such
that if you sum up all of the components of any
vector, mod2 equals 0.
OK?
So this I will call the single parity check, or more briefly,
the SPC code, or the even-weight code.
That's equally good.
And here we maybe should do a little bit more work.
Say, is this in fact a subspace?
Does it include the all-zero code word?
Yes, all-zero has even weight.
The sum of any two even-weight code words, an even-weight
code word, an even-weight n-tuple.
Yes.
As here.
This is an example.
This is the three, two SPC code.
OK.
Why is this dimension n minus one?
AUDIENCE: [INAUDIBLE]
Is every code word orthogonal to the one-vector?
PROFESSOR: OK.
That's an excellent answer.
It's a little advanced for us right now.
I'm looking for an elementary argument.
AUDIENCE: [INAUDIBLE]
PROFESSOR: OK.
So you're saying we have a set of generators
that looks like this.
Is that what you're saying?
You are correct.
And how many such generators are there?
There are n minus 1 of them.
I always like to find the most elementary argument possible.
I think the most elementary argument here is that the
number of even-weight n-tuples is equal to the number of
odd-weight n-tuples, and together they form the set of
all n-tuples.
So exactly half of the n-tuples are even weight.
That means there are 2 to the n minus 1 of them, 2 to the n
over 2, and therefore, the dimension must be n minus 1.
But perhaps this is just as elementary a proof.
Well, however you do it, you'll find that there are 2
to the n minus 1 of them, or that here is clearly a set of
generators.
It might take a few more lines to show that every even-weight
code word is a linear combination of this particular
set of generators, but it's certainly true.
All right.
So these four classes of codes, these two entirely
trivial ones, these two which are actually somewhat more
interesting for coding purposes --
we've already seen, we can get a coding game with this length
three, dimension two code --
are basically the simplest codes we can think of.
The simplest binary linear block codes.
Now we'll see them again and again.
They turn up.
All right?
So the whole course is going to be about
finding 1's in between.
More complicated ones.
There's clearly more room to play.
For instance, if k is equal to half of n, which means that
rho is equal to one bit per two dimensions, there's a lot
of possibilities.
And so we're going to explore those possibilities.
AUDIENCE: [INAUDIBLE]
[UNINTELLIGIBLE]
PROFESSOR: Sure.
Let's take the 6,5 code generated by these five
generators.
Well, it contains some
odd-weight code words, obviously.
But it's not as interesting from a coding point of view.
It's not the only one, but it's the only one
you'll ever see here.
All right.
Let's see.
One thing I didn't point out in the notes but probably
should have here is, what is rho for an n, k binary linear
block code?
How many bits can we send?
Suppose we take the Euclidean image of this.
It is going to have 2 to the k points, 2 to the k vertices of
the n-cube, and so what is rho?
What is the rate in bits per two dimensions?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Right.
The rate is basically k over n over 2, if you like, or 2 k
over n bits per two dimensions.
You can send k bits in n dimensions, or 2 k over n bits
per two dimensions.
And since k can't be larger than n, this is going to be
less than or equal to two bits per two dimensions.
So again, we see we're definitely in the
power-limited regime, and that we can really get any nominal
spectral efficiency between 0 and 2 by choosing k and n
appropriately.
All right.
So n and k determine the rate, determine the nominal spectral
efficiency.
All right.
Let's now talk about things that I think are also very old
hat to you.
I mean weight and distance.
But we begin to get into areas that show us that these vector
spaces are very different from the real and complex vector
spaces that we're accustomed to.
So what are we doing now?
We're starting to get into the geometry of this vector space.
The geometry is not Euclidean geometry, but
it's Hamming geometry.
We define the Hamming weight of a vector as simply the
number of 1's in v. So the Hamming weight of the all-0
vector is 0, the Hamming weight of the all-1 vector is
n, and in general, the Hamming weight is somewhere
between 0 and n.
Just the number of 1's.
Pretty simple.
And given two vectors x and y, what is their distance?
The Hamming distance between x and y is equal to the Hamming
weight of x minus y.
This is the standard way of converting a weight metric
into a distance metric.
Or because addition is the same as subtraction in a
binary vector space, we might equally well write this is as
the Hamming weight of x plus y.
And more informally, this is simply the number of places in
which they differ.
OK.
So if x and y are identical, then x plus y is equal to 0
and the distance is 0.
If they are complementary, y is the complement of x, then
they differ in every place.
The sum will then be the all-1 vector, and the weight, the
Hamming distance, will be n.
And so again, the Hamming distance is somewhere between
0 and n, measures how different they are.
Clearly going to be important for coding.
It's going to translate directly into Euclidean
distance under this standard 2-PAM map.
OK.
Again, let's check that it satisfies the distance axioms.
I don't know how many of you have seen this, but let's see.
What are the distance axioms?
Strict non-negativity.
In other words, the Hamming distance between x and y --
that's a single comma --
is greater than or equal to 0, and equality if and
only if x equals y.
That's what strict means.
So if we find the Hamming distance is 0, we can assert
that x equals y.
We have, of course, symmetry.
The Hamming distance between x and y is the same as the
Hamming distance between y and x.
Order doesn't matter.
And finally we have the triangle inequality, that the
Hamming distance between x and z certainly can't be more than
the Hamming distance between x and y plus the Hamming
distance between y and z.
If x differs from y in only n1 places, and y differs from z
in only n2 places, then clearly z can't differ from x
in more than n1 plus n2 places.
So check, check, check.
This is a legitimate metric for defining a geometry on the
space, and this is the one that we use on the space of
all n-tuples.
But notice it's not all like the Euclidean Distance.
Now when we have a linear code --
let's combine these things.
When we have a linear code, we have a group property which
is, let me write it this way.
If we take any code word and add it to any other code word,
that's in the code.
And furthermore, c plus c prime is not equal to c prime
plus c single prime, c plus c single prime,
because we can --
well, I'll finish it.
Unless c prime equals c double prime.
Why is that?
We can do subtraction, cancellation.
Cancel c out from each side.
So if we add c to c double prime, we're going to get a
different result from adding c to c prime, if c prime and c
double prime are different.
So this implies that c plus C --
I write that as an abbreviation for the set of
all c plus c prime, as a c prime runs through the code C.
So this is the 2 to the k sums of the code plus any code word
in the code.
Sorry if I don't write down all steps.
What is that going to be?
C. How did we conclude that?
AUDIENCE: [INAUDIBLE]
[UNINTELLIGIBLE]
PROFESSOR: Perfect.
Did you all hear that?
By the group property, each one of these
things is in the code.
By this argument, no two of them are the same.
That means I get 2 to the k distinct elements all in the
code, that's got to be the code, because the code only
has 2 to the k elements.
All right.
So if I add a code word --
in other words, if I write down the code --
0, 0, 0, 0, 1, 1, 1 0 1, 1 1 0- and I add any code word to
it -- say I add 0 1 1 to the code.
So let me just do one column of the addition table.
I get 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1.
I'll get the code itself again.
This has a very important property.
The set of Hamming distances C and c prime, as c prime runs
through c, from any given code word C, it is independent of
C. So I can start from any code word, measure the Hamming
distances, the 2 to the k Hamming distances to all other
code words, including C itself --
let c prime run through the entire code, I'm going to get
a set of 2 to the k distances called the distance
profile of the code.
And I claim that it doesn't matter which code
word I start from.
I'm going to get the same distance profile regardless of
where I start.
In other words, the set of all distances from the all-0 word
here, which is 0, 2, 2, 2, is the same as the set of all
distances from the 0, 1, 1, code word,
which is 2, 0, 2, 2.
And the proof is basically that this is simply the
Hamming weight of C plus c prime as c prime runs through
C. What is this equal to?
So the proof is that this is equal to the Hamming weight of
c prime as c prime runs through C.
So the distance profile from any code word is simply equal
to the weight profile of the code itself.
The weight profile of this code is 0, 2, 2, 2, Start from
any code word, measure the distances to other code words,
I'm always going to get 0, 2, 2, 2, 0, to
itself, and the others.
Sounds sort of like the tetrahedron, doesn't it?
It's zero distance from a code word to itself and equal
distance to all the other code words, in
that particular case.
OK.
So again, everything is very elementary here.
The distance profile is independent of C and equal to
the weight profile.
So this has an extremely important corollary.
What's the minimum Hamming distance of the code?
You might expect this is going to be an
important code parameter.
AUDIENCE: [INAUDIBLE]
PROFESSOR: The minimum Hamming distance between any two code
words is going to be equal to --
I think you said it.
AUDIENCE: The 0 [UNINTELLIGIBLE].
PROFESSOR: Non-zero is important here.
If the distance profile is equal to the weight profile,
one of the weights is always going to be zero.
And that corresponds to the distance between a code word
and itself.
All right.
If I go through all the other 2 to the k minus 1 distances,
they're going to be weights.
They're going to be the distances from a code to all
other code words.
And the minimum distance is simply going to be the minimum
non-zero weight of the code.
For example, in this code, the minimum Hamming distance
between any two distinct code words is going to be equal to
the minimum distance from the 0 code word --
that's another way of doing it.
Since it's independent of C, we may as well take the base
code word C to be zero.
And then what's the minimum distance to 0?
To the 0-code word?
It's the minimum weight.
From the 0 code word, distance equals weight.
So the minimum distance is the minimum weight of any non-zero
code word, which for this code is two.
Now here the weight profile is 0, 2, 2, 2, 0.
The distance profile from any code word to all the others is
0, 2, 2,2 2.
This is always the distance to itself.
So minimum distance to other code words is
always going to be two.
Furthermore, the number of nearest neighbors --
to go back and use chapter five terminology --
the number of nearest neighbors is going to be the
number of code words that have that minimum weight --
in this case, three.
Still sounding a lot like a tetrahedron, isn't it?
This easy map between Hamming distance and Euclidean
distance for this case and in general for all of our cases.
So corollary.
The minimum Hamming distance, which implicitly means between
two distinct code words of C, is equal to the minimum
non-zero weight of C, and the number of minimum weight code
words is independent --
I'm doing this backwards.
From any code word, the number of code words that distance,
let's call this d, is equal to the number of
weight d code words.
Sorry, you probably can't see that.
All right.
So we get this symmetry property for codes that
follows from the group property of the code that if
we stand on any code word and look out, we're always going
to see the same thing.
We have this constant.
It's actually easiest to see this when we make the map from
the code to the Euclidean image of the code.
So the Euclidean image S of C of the code word is going to
be some set of 2 to the k vertices of an
n-cube of side alpha.
Let's talk about the Euclidean image of these properties.
If the minimum Hamming distance of the code is d,
what's the minimum squared Euclidean distance between
elements of S of C going to be?
Well, let's do it coordinate by coordinates.
Let's take two code words, C and c prime, let's say.
And let's suppose we have some Hamming distance
between C and c prime.
That means that c and c prime differ in the Hamming
distance, number of places.
So if we map this into the corresponding pair of vertices
of the n-cube in Euclidean space, S of C and S of c
prime, how many coordinates are these going to differ in?
It's going to differ in same number of coordinates, D_h.
If they don't differ, what's Euclidean squared distance in
those coordinates?
0.
If they do differ, the Euclidean squared distance is
4 alpha squared.
So the Euclidean distance D_e between S of C and S of c
prime is simply going to be 4 alpha squared times the
Hamming distance between C and c prime, yes?
So I should say this is the squared Euclidean distance.
Why do we always talk about the
squared Euclidean distance?
Because it's additive, coordinate-wise.
And the Hamming distance is additive, coordinate-wise.
So there's a nice easy map here.
So what does this mean d_min squared is going to be?
d_min squared of, let's say, S of C. This constellation that
we've formed by taking the Euclidean image of c.
The minimum square distance between points in S of C is
just going to be 4 alpha squared times d, where I don't
think I ever --
d equals min Hamming distance.
And we're always going to talk about n, k, d as the three key
parameters of a binary linear block code.
n is the code length, F2 to the n, k is the dimension, d
is the minimum Hamming distance.
So by going into this Hamming geometry, we've got a third
key property of the code.
And we see it's key, because we can get the minimum squared
distance between this Euclidean image constellation,
just 4 alpha squared d.
AUDIENCE: nd makes a probability of
[UNINTELLIGIBLE] that is dependent on [UNINTELLIGIBLE]
then.
PROFESSOR: Correct.
This is all we need to know to get the union bound estimate.
Well, a few more things.
We need to know what K_min average of S of C. And what is
that going to be?
This is simply going to be the number of words in the code.
To get this minimum squared distance, we need a Hamming
distance of d.
So the number of words in the code of distance d, which is
given by the parameter n sub d, is simply going to be the
number of nearest neighbors.
Not just the average distance, but I want to emphasize this
symmetry property.
If we stand on any point, on any vertex of this cube in
n-space, which is the code vertex, and we look at all the
other points in the constellation, no matter which
point we stand on, we will always see the
same profile of distances.
We'll see precisely nd code words at Euclidean distance 4
alpha squared d.
We'll see nd plus 1 at Euclidean squared distance 4
alpha squared d plus 1, and so forth, right up the profile.
So there's complete symmetry in the constellation.
In that universe, you don't know which code point you're
standing on just by looking out, because the world looks
the same to you.
Is that clear?
OK.
So from a communications point of view, this is important,
because it means it doesn't matter what code word we send.
The probability of error from any code word is going to be
the same as the probability of error from any other code
word, because the geometry is exactly the same.
The Voronoi regions are all the same shape.
So given the exact probability of error, not just the
union-bound estimate, is going to be independent of which
code word was sent.
This all follows from the fact that it's a linear code and
therefore has the group property, which translates
into this very strong geometrical uniformity
property in Euclidean space.
Or actually in Hamming space too, but it's more striking in
Euclidean space.
OK?
So we have everything we need to write down
the union bound estimate.
Union bound estimate was just the probably of error per bit
is well approximated by K_b of constellation, in this case, S
of C, times Q of the square root of the coding gain of the
constellation times 2 Eb over N_0.