The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or to view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: I'm going to spend a couple of minutes reviewing
the major things that we talked about last time and
then get into discrete source coding, which is the major
topic for today.
The first major thing that we talked about last time, along
with all of the philosophy and all those other things, was
the sense of what digital communication really is.
I said that what digital communication is, is it's
communication where there's a binary interface between
source and destination.
The source is very often analog.
The most interesting sources are analog.
The channel is often analog, most interesting.
Channels are analog, and we'll say more about what I mean by
analog later.
What's important is that you have this binary interface
between source and channel coding.
We said a little bit about why we wanted a binary interface,
aside from the fact that it's there now, there's nothing you
can do about it even if you don't like it.
One reason is standardization, which means it simplifies
implementation, which means you can do
everything in the same way.
If you have ten different kinds of channel coding and
you have ten different kinds of source coding and you have
a binary interface, it means you need to develop 20
different things -- ten at the source and ten at the decoder.
If you don't have that standardization with a binary
interface between them, you need 100 different things.
You need to match every kind of source with every kind of
destination.
That raises the price of all chips enormously.
One of the other things we said is the price of chips is
very much the cost of development divided by the
number of them that you stamp out.
That's not quite true, but it's a good first
approximation.
In other words, standardization is important.
Layering.
Layering is in many ways very similar to standardization
because this binary interface is also a layer between source
and destination.
But the idea there is not that it standardizes to make things
cheaper, but it simplifies the conceptualization of
what's going on.
You can look at a source and only focus on one thing.
How do I take that source and turn it into the smallest
number of binary digits possible?
We'll talk a good deal about what that means later because
there's something stochastic involved in there and will
take us awhile to really understand that.
Finally, using a binary interface loses nothing in
performance.
That's what Shannon said, it's what he proved.
There's some questions there when you get to networks, but
the important thing is the places where you want to study
non-binary interfaces, you will never get a clue of what
it is that you're looking at or why if you don't first very
well understand why you want a binary
interface to start with.
In other words, if you look at these other cases, there's
exceptions to the rule, and if you don't know what the rule
is, you certainly can't understand what
the exceptions are.
So for today we're going to start out by studying this
part of the problem in here.
Namely, how do you turn a source and put a general
source input into binary digits that you're going to
put into the channel.
How do I study this without studying that?
Well, one thing is these are binary digits here.
But the other thing is we're going to assume that what
binary digits go in here come out here.
In other words, there aren't any errors.
It's an error-free system.
Part of the purpose of studying channel encoding and
channel decoding is to say how is it that you get that
error-free performance.
You can't quite get error-free performance, you get almost
error-free performance, but the idea is when errors come
out here, it's not this guy's fault, it's this guy's fault.
Therefore, what we're going to study here is how we do our
job over here.
Namely, how we deal with decoding, the same string of
bits that went into there and decode them coming out.
So that's where we'll be for the next three weeks or so.
We talked a little bit last time about how do you layer
source coding itself.
I want to come back, because we were talking about so many
things last time, and emphasize what this means a
little bit.
We're going to break source coding up into three different
layers again.
You start out with some kind of input wave form or image or
video or whatever the heck it is.
You're going to do something like sampling it or expanding
it in some kind of expansion, and we'll talk a great deal
about that later.
That's not an obvious thing, how to do that.
When you finish doing that, you wind up
with an analog sequence.
In other words, you wind up with a sequence of real
numbers or sequence of complex numbers.
Those go into a quantizer.
What the quantizer does is to turn an uncountably infinite
set of things into a finite set of things.
When you turn an uncountably infinite set of possibilities
into a finite set of possibilities, you get
distortion.
There's no way you can avoid it.
So that's a part of what happens there.
Then at this point you have a finite alphabet of symbols.
That goes into the discrete coder, goes through what we're
now calling a reliable binary channel and comes out here.
What we're going to be studying for the next two
weeks or so is this piece of the system right in here.
Again, what we're going to be doing is assuming a reliable
binary channel to the right of this, which is
what we already assumed.
We're going to assume that these things do
whatever they have to.
But this problem here, this isolated problem is important
because this is dealing with the entire problem of text,
and you know what text is, it's computer files, it's
English language text, it's Chinese text, it's whatever
kind of text.
If we understand how to do that, we can then go on to
talk about quantization because we'll have some idea
of what we're trying to accomplish with quantization.
Without that we won't know what the purpose of
quantization is.
Without the quantization we won't know what we're trying
to accomplish over here.
There's another reason for studying this problem, which
is that virtually all the ideas that come into this
whole bunch of things are all tucked into this one subject
in the simplest possible way.
One of the nice things about information theory, which
we're going to touch on I said in this course, is that one of
the reasons for studying these simple things first is that
information theory is really like a symphony.
You see themes coming out, those themes get repeated,
they get repeated again with more and more complexity each
time, and when you understand the simple idea of the theme,
then you understand what's going on.
So, that's the other reason for dealing with that.
To summarize those things -- most of this I already said.
Examples of analog sources are voice, music, video, images.
We're going to restrict this to just wave form sources,
which is voice and music.
In other words, an image is something where you're mapping
from two dimensions this way and this way into a sequence
of binary digits.
So it's a mapping after you get done sampling from r
square, which is this axis and this axis, into your outut.
Namely, for each point in this plane, there's some real
number that represents the amplitude at that point.
Video is a three-dimensional to one-dimensional thing,
namely, you have time.
You also have this way, you have this way, so you're
mapping from r cubed into r.
We're not going to deal with those because really all the
ideas are just contained in dealing
with wave form sources.
In other words, the conventional functions that
you're used to seeing.
Namely, things that you can draw on a piece of paper and
you can understand what's going on with them.
These are usually samples or expanded into series
expansions almost invariably, and we'll
understand why later.
That, in fact, is a major portion of the course.
That's where all of the stuff from signals and
systems come in.
We'll have to expand that a whole lot because you didn't
learn enough there.
We need a lot of other things, and that's what we need to
deal with wave forms.
We'll take the sequence of numbers that
comes out of the sampler.
We're then going to quantize that sequence of numbers.
That's the next thing we're going to study.
Then we're going to get into analog and discrete sources,
which is the topic we will study right now.
So we're going to study this.
After we get done this, we're going to study this also.
When we study this, we'll have what we know about this as a
way of knowing how to deal with the whole problem from
here out to here.
Finally, we'll deal with wave forms and deal with the whole
problem from here out to here.
So that's our plan.
In fact, this whole course is devoted to studying this
problem, then this problem, then this problem -- that's
the source part of the course.
Then dealing with -- if I can find it again -- with the
various parts of this problem.
So first we study sources, then we study channels.
Because of the binary interface, when we're all done
with that we understand digital communication.
When we get towards the end of the term we'll be looking at
more sophisticated kinds of channels than we look at
earlier, which are really models for wireless channels.
So that's where we're going to end up.
So discrete source coding, which is what we want
to deal with now.
What's the objective?
We're going to map a sequence of symbols into a binary
sequence and we're going to do it with unique decodability.
I'm not going to define unique decodability at this point.
I'm going to define it a little bit later.
But roughly what it means is this.
We have a sequence of symbols which come into
the encoding encoder.
They go through this binary channel.
They come out as a sequence of binary digits.
Unique decodability says if this guy does his job, can
this guy do his job?
If this guy can always do his job when these digits are
correct, then you have something called unique
decodability.
Namely, you can guarantee that whatever comes out here,
whatever comes in here, will turn into a
sequence of binary digits.
That sequence of binary digits goes through here.
These symbols are the same as these symbols.
In other words, you are reproducing things error-free
if, in fact, this reproduces things error-free.
So that's our objective.
There's a very trivial approach to this, and I hope
all of you will agree that this is
really, in fact, trivial.
You map each source symbol into an
l-tuple of binary digits.
If you have an alphabet of size m, how many different
binary strings are there of length l?
Well, there are 2 to the l of them.
If l is equal to 2, you have 0, 0, 0, 1, 1, 0, and 1, 1.
If l is equal to 3, you have strings of 3, which is 0, 0,
0, 0, 0, 1, 0, 1, 0, blah, blah, blah, blah, blah.
What comes out to be 2 to the 3, which is equal to 8.
So what we need if we're going to use this approach, which is
the simplest possible approach, which is called the
fixed length approach, is you need the alphabet size to be
less than or equal to the number of binary digits that
you use in these strings.
Now, is that trivial or isn't it trivial?
I hope it's trivial.
We don't want to waste bits when we're doing this,
particularly, so we don't want to make l any bigger than we
have to, because for every symbol that comes in, we get l
symbols coming out.
So we'd like to minimize l subject to this constraint
that 2 to the l has to be bigger, greater
than or equal to m.
So, what we want to do is we want to choose l as the
smallest integer which satisfies this.
In other words, when you take the logarithm to the base 2 of
this, you get log to the base 2 of m has to be less than or
equal to l, and l is then going to be less than log to
the base 2 of m plus 1.
This is the constraint which says you don't make l any
bigger than you have to make it.
So in other words, we're going to choose l equal to the
ceiling function of log to the base m.
In other words, this is the integer which is greater than
or equal to log to the base 2 of m.
So let me give you a couple of examples of that.
Excuse me for boring you with something which really is
trivial, but there's notation here you have to get used to.
You get confused with this because there's the alphabet
size which we call m, there's the string length which we
call l, and you keep getting mixed up between these two.
Everybody gets mixed up between them.
I had a doctoral student the other day who got mixed up in
it, and I read what she had written four times and I
didn't catch it either.
So this does get confusing at times.
If you have an alphabet, which is five different kinds of the
letter a --
that's one reason why these source codes get messy, you
have too many different kinds of each letter, which
technical people who like a lot of jargon use all of them.
In fact, when people start writing papers and books you
find many more than five there.
In terms of Latex, you get math cow, you get math gold,
you get math blah, blah, blah.
Everything in little and big.
You get the Greek version.
You get the Roman version and the Arabic version, if you're
smart enough to know that language, those languages.
What we mean by code is alpha gets mapped into 0, 0, 0. a
gets mapped into 0, 0, 1.
Capital A into this and so forth.
Does it make any difference what mapping you use here?
Can you find any possible reason why it wouldn't make a
difference whether I map alpha into 0, 0, 0, and a into 0, 0,
1 or vice versa?
I can't find any reason for that.
Would it make any difference of instead of having this
alphabet I had beta b, capital B, script b, and capital B
with a line over it?
I can't see any reason why that would make
a difference either.
In other words, when we're talking about fixed length
codes, there are only two things of importance.
One of them is how big is the alphabet -- that's why we talk
about alphabets all the time.
After you know how big the alphabet is and after you know
you want to do a fixed length binary encoding, then you just
assign a binary string to each of these letters.
In other words, there's nothing
important in these symbols.
This is a very important principle
of information theory.
It sort of underlines the whole subject.
I'm not really talking about information theory here, as I
said, we're talking about communication.
But communication these days is built on these information
theoretic ideas.
Symbols don't have any inherent meaning.
As far as communication is concerned, all you're
interested in is what is the set of things --
I could call this a1, a2, a3, a4, a5, and we're going to
start doing this after awhile because we will recognize that
the name of the symbols don't make any difference.
If you listen to a political speech if it's by a Republican
there are n different things they might say, and you might
as well number them a1 to a sub n.
If you listen to one of the Democratic candidates there
are m different things they might say.
You can number them 1 to m, and you can talk to other
people about it and say oh, he said a1 today, which is how do
we get out of the war in Iraq.
Or he said number 2 today, which is we need more taxes or
less taxes and so forth.
So it's not what they say as far as communication is
concerned, it's just distinguishing the different
possible symbols.
So, you can easily decode this --
you see three bits and you decode them.
Can I?
Is this right or is there something missing here?
Of course, there's something missing.
You need synchronization if you're going to do this.
If I see a very long string of binary digits and I'm going to
decode them into these letters here, I need to know where the
beginning is.
In other words, if it's a semi-infinite string of binary
digits, I don't know how to look at it.
So, inherently, we believe that somebody else gives us
synchronization.
This is one of these things we always assume.
When you start building a system after you decide how to
do this kind of coding, somebody at some point has to
go through and decide where do you get the
synchronization from.
But you shouldn't think of the synchronization first.
If I'm encoding 10 million symbols and it takes me 1,000
bits to achieve the synchronization, that 1,000
bits gets amortized over 10 million different symbols, and
therefore, it doesn't make any difference, and therefore,
we're going to ignore it.
It's an important problem but we ignore it.
The ASCII code is a more important example in this.
It was invented many, many years ago.
It was a mapping from 256 different symbols which are
all the letters, all the numbers, all the things that
people used on typewriters.
Anybody remember what a typewriter is?
Well, it's something people used to use before they had
computers, and these typewriters had a lot of
different keys on them and they had a lot of special
things you could do with them.
And somebody dreamed up 256 different things that they
might want to do.
Why do they use l equals 8?
Nothing to do with communication or with
information theory or with any of these things.
It was that 8 is a nice number.
It's 2 to the 3.
In other words, this was a standard length of both
computer words and of lots of other things.
Everybody likes to deal with 8 bits, which you call a byte,
rather than 7 bits which is sort of awkward or 6 bits
which was an earlier standard, which would have been
perfectly adequate for most things that people wanted.
But no, they had to go to 8 bits because it
just sounded nicer.
These codes are called fixed length codes.
I'd like to say more about them but there really isn't
much more to say about them.
There is a more general version of them, which we'll
call generalized fixed length codes.
The idea there is to segment the source sequence.
In other words, we're always visualizing now having a
sequence of symbols which starts at
time zero, runs forever.
We want to segment that into blocks of length n.
Namely, you pick off the first n symbols, you find the code
word for those n symbols, then you find the code word for the
next n symbols, then you find the code word for the next n
symbols and so forth.
So it's really the same problem that
we looked at before.
It's just that the alphabet before had the number of
symbols as the alphabet size.
Now, instead of having an alphabet size which is m,
we're looking at blocks of m symbols and how many possible
combinations are there of blocks where every symbol is
one of m different things.
Well, if you have two symbols, the first one can be any one
of m things, the second one can be any one of m things.
So there are m squared possible combinations for the
first two symbols, there are m cubed possible combinations
for the first three symbols and so forth.
So we're going to have an alphabet on blocks of m to the
n different n tuples of source letters.
Well, once you see that we're done because what we're going
to do is find a binary sequence for every one of
these blocks of m to the n symbols.
As I said before, the only thing important is
how many are there.
It doesn't matter that they're blocks or that they're stacked
this way or that they're stacked around in a circle or
anything else.
All you're interested in is how many of them are there.
So there are m to the n of them.
So, what we want to do is make the binary length that we're
dealing with equal to log to the base 2 of m to the n, the
ceiling function of that.
Which says log to the base 2 of m is less than or equal to
l bar where l bar is going to be the bits per source symbol.
I'm going to abbreviate that bits per source symbol.
I would like to abbreviate it bps, but I and everyone else
will keep thinking that bps means bits per second.
We don't have to worry about seconds here, seconds had
nothing to do with this problem.
We're just dealing with sequences of things and we
don't care how often they occur.
They might just be sitting in a computer file and we're
doing them offline, so seconds has nothing to
do with this problem.
So, log to the base 2 of m is less than or equal to l over
n, which is less than log to the base 2 of m plus 1 over n.
In other words, we're just taking this dividing it by n,
we're taking this dividing by n, the ceiling function is
between log to the base 2 of m to the n, and log to the base
2 of m to the n plus 1.
When we divide by n, that 1 becomes 1 over n.
What happens when you make n large? l approaches log to the
base 2 of m from above.
Therefore, fixed length coding requires log to the base 2 of
n bits per source symbol if, in fact, you
make n large enough.
In other words, for the example of five different
kinds of a's, we had m equal to 5.
So if you have m equal to 5, that leads to m squared equals
25, that leads to l equals -- what's the ceiling function of
log of this?
It's 5.
l bar is equal to -- what's half of 5?
2 and 1/2, yes.
As you get older you can't do arithmetic anymore.
So look what we've accomplished.
We've gone from three bits per symbol down to two and and
half bits per symbol, isn't that exciting?
Well, you look at it and you say no,
that's not very exciting.
I mean yes, you can do it, but most people don't do that.
So why do we bother with this?
Well, it's the same reason we bother with a lot of things in
this course, and the whole first two weeks of this course
will be dealing with things where when you look at them
and you ask is this important, you have to answer no, it's
not important, it doesn't really have much to do with
anything, it's a mathematical idea.
What it does have to do with is the principle involved here
is important.
It says that the lower limit of what you can do with fixed
coding is log to the base 2 of m.
You have an alphabet of size m, you can get as close to
this as you want to.
We will find out later that if you have equally likely
symbols when we get to talking about probability, we will
find out that nothing in the world can do
any better than this.
That's the more important thing, because what we're
eventually interested in is what's the best you can do if
you do things very complicated.
Why do you want to know what the best is if you do
something very complicated?
Because if you can do that simply then you know you don't
have to look any further.
So that's the important thing.
Namely, it lets you do something simple and know
that, in fact, what you're doing makes sense.
That's why we do all of that.
But then after we say well there's no place else to go on
fixed length codes, we say well, let's look at variable
length codes.
The motivation for variable length codes is that probable
symbols should probably have shorter code words than very
unlikely symbols.
And Morse thought of this a long, long time ago when Morse
code came along.
Probably other people thought of it earlier, but he actually
developed the system and it worked.
Everyone since then has understood that if you have a
symbol that only occurs very, very, very rarely, you would
like to do something, make a code word which is very long
for it so it doesn't interfere with other code words.
Namely, one of the things that you often do when you're
developing a code is think of a whole bunch of things which
are sort of exceptions.
They hardly ever happen.
You use the fixed length code for all the things that happen
all the time, and you make one extra code word for all these
exceptions.
Then you have this exception and paste it on at the end of
the exception is a number which represents which
exception you're looking at.
Presto, you have a variable length code.
Namely, you have two different possible code lengths -- one
of them for all of the likely things and the indication that
there is an exception, and two, all the unlikely things.
There's an important feature there.
You can't drop out having the code word
saying this is an exception.
If you just have a bunch of short code words and a bunch
of long code words, then you see a short code word and you
don't know -- well, if you see a long code word starting or
you have a short code word, you don't know which it is and
you're stuck.
So one example of a variable length code -- we'll use some
jargon here.
We'll call the code a script c.
We'll think of script c as a mapping which goes from the
symbols onto binary strings.
In other words, c of x is the code word corresponding
to the symbol x.
So for each x in the alphabet, capital X, and we have to
think of what the capital X is.
But as we say, the only thing we're really interested in is
how big is this alphabet --
that's the only thing of importance.
So if we have an alphabet which consists of the three
letters a, b and c, we might make a code where the code
word for a is equal to zero, the code word for b is equal
to 1, zero, and the code word for c is equal to 1,1.
Now it turns out that's a perfectly fine
code and that works.
Let me show you another example of a code.
Let me just show you an example of a code here so we
can see that not everything works.
Suppose c of a is zero, c of b is 1, and c of c is -- this is
a script c, that's a little c -- is 1, zero.
Does that work?
Well, all of the symbols have different code words, but this
is an incredibly stupid thing to do.
It's an incredibly stupid thing to do because if I send
a b followed by an a, what the poor decoder sees is 1
followed by zero.
In other words, one of the things that I didn't tell you
about is when we're using variable length codes we're
just concatenating all of these code words together.
We don't put any spaces between them.
We don't put any commas between them.
If, in fact, I put a space between them, I would really
have not a binary alphabet but a ternary alphabet.
I would have zeros and I would have 1's and I would have
spaces, and you don't like to do that
because it's much harder.
When we start to study channels we'll see that
ternary alphabets are much more difficult to work with
than binary alphabets.
So this doesn't work, this does work.
Part of what we're going to be interested in is what are the
conditions under why this works and why
this doesn't work.
Again, when you understand this problem you will say it's
very simple, and then you come back to look at it again and
you'll say it's complicated and then it looks simple.
It's one of these problems that looks simple when you
look at it in the right way, and it looks complicated when
you get turned around and you look at it backwards.
So the success of code words of a variable length code are
all transmitted just as a continuing sequence of bits.
You don't have any of these commas or spaces in them.
If I have a sequence of symbols which come into the
encoder, those get mapped into a sequence of bits, variable
length sequences of bits which come out.
They all get pushed together and just come out
one after the other.
Buffering can be a problem here, because when you have a
variable length code --
I mean look at what happens here.
If I've got a very long string of a's coming in, I got a very
short string of bits coming out.
If I have a long string of b's and c's coming in, I have a
very long string of bits coming out.
Now usually the way the channels work is that you put
in bits at a fixed rate in time.
Usually the way that sources work is that symbols arrive at
a fixed rate in time.
Therefore, here, if symbols are coming in at a fixed rate
in time, they're going out at a non-fixed rate in time.
We have to bring them into a channel at a fixed rate in
time, so we need a buffer to take care of the difference
between the rate at which they come out and the rate at which
they go in.
We will talk about that problem later, but for now we
just say OK, we have a buffer, we'll put
them all in a buffer.
If the buffer ever empties out -- well, that's sort of like
the problem of initial synchronization.
It's something that doesn't happen very often, and we'll
put some junior engineer on that because it's a hard
problem, and seeing your engineers never deal with the
hard problems, they always give those to the junior
engineers so that they can assert their superiority over
the junior engineers.
It's a standard thing you find in the industry.
We also require unique decodability.
Namely, the encoded bit stream has to be uniquely deparsed at
the decoder.
I have to have some way of taking that long string of
bits and figuring out where the commas would have gone if
I put commas in it.
Then from that I have to decode things.
In other words, it means that every symbol in the alphabet
has to have a distinct code word connected with it.
We have that here.
Every symbol has a distinct code word.
But it has to be more than that.
I'm not even going to talk about precisely what that more
means for a little bit.
We also assume to make life easy for the decoder that it
has initial synchronization.
There's another obvious property that we have.
Namely, both the encoder and the decoder know what the code
is to start with.
In other words, the code is built into these devices.
When you design a coder and a decoder, what you're doing is
you figure out what an appropriate code should be,
you give it to both the encoder and the decoder, both
of them know what the code is and therefore, both of them
can start decoding.
A piece of confusion.
We have an alphabet here which has a list of symbols in it.
So there's a symbol a1, a2, a3, up to a sub m.
We're sending a sequence of symbols, and we usually call
the sequence of symbols we're sending x1, x2, x3,
x4, x5 and so forth.
The difference is the symbols in the alphabet are all
distinct, we're listing them one after the other.
Usually there's a finite number of them.
Incidentally, we could have a countable number of symbols.
You could try to do everything we're doing here say with the
integers, and there's a countable number of integers.
All of this theory pretty much carries through with various
little complications.
We're leaving that out here because after you understand
what we're doing, making it apply to integers is
straightforward.
Putting in the integers to start with, you'll always be
fussing about various silly little special cases, and I
don't know a single situation where anybody deals with a
countable alphabet, except by truncating it.
When you truncate an infinite alphabet you
get a finite alphabet.
So, we'll assume initial synchronization, we'll also
assume that there's a finite alphabet.
You should always make sure that you know whether you're
talking about a listing of the symbols in the alphabet or a
listing of the symbols in a sequence.
The symbols in a sequence can all be the same, they can all
be different.
They can be anything at all.
The listing of symbols in the alphabet, there's just one for
each symbol.
We're going to talk about a very simple case of uniquely
decodable codes which are called prefix-free codes.
A code is prefix-free if no code word is a prefix of any
other code word.
In other words, a code word is a string of binary digits.
A prefix of a string of binary digits.
For example, if we have the binary string 1, 0, 1, 1, 1.
What are the prefixes of that?
Well, one prefix is 1, 0, 1, 1.
Another one is 1, 0, 1.
Another one is 1, 0.
Another is 1.
In other words, it's what you get by starting out at the
beginning and not quite getting to the end.
All of these things are called prefixes.
If you want to be general you could call 1, 0, 1, 1, 1, a
prefix of itself.
We won't bother to do that because it just is -- that's
the kind of things that mathematicians do to save a
few words in the proofs that they give and we
won't bother with that.
We will rely a little more on common sense.
Incidentally, I prove a lot of things in these notes here.
I will ask you to prove a lot of things.
One of the questions that people always have is what
does a proof really mean?
I mean what is a proof and what isn't a proof?
When you take mathematics courses you get one idea of
what a proof is, which is appropriate
for mathematics courses.
Namely, you prove things using the correct terminology for
proving them.
Namely, everything that you deal with you define it ahead
of time so that all of the terminology you're using all
has correct definitions.
Then everything should follow from those definitions and you
should be able to follow a proof through without any
insight at all about what is going on.
You should be able to follow a mathematical proof
step-by-step without knowing anything about what this is
going to be used for, why anybody is interested in it or
anything else, and that's an important thing to learn.
That's not what we're interested in here.
What we're interested in here for a proof --
I mean yes, you know all of the things around this
particular proof that we're dealing with, and what you're
trying to do is to construct a proof that covers
all possible cases.
You're going to use insight for that, you're going to use
common sense, you're going to use whatever you have to use.
And eventually you start to get some sort of second sense
about when you're leaving something out that really
should be there.
That's what we're going to be focusing on when we worry
about trying to be precise here.
When I start proving things about prefix codes, I think
you'll see this because you will look at it and say that's
not a proof, and, in fact, it really is a proof.
Any good mathematician would look at it and say yes, that
is a proof.
Bad mathematicians sometimes look at it and say well, it
doesn't look like proof so it can't be a proof.
But they are.
So here we have prefix-free codes.
The definition is no code word is a prefix of
any other code word.
If you have a prefix-free code, you can express it in
terms of a binary tree.
Now a binary tree starts at a root, this is the beginning,
moves off to the right -- you might have it start at the
bottom and move up or whatever direction you want to go in,
it doesn't make any difference.
If you take the zero path you come to some leaf.
If you take the one path you come to some
intermediate node here.
From the intermediate node, you either go
up or you go down.
Namely, you have a 1 or a zero.
From this intermediate node you go up and you go down.
In other words, a binary tree, every node in it is either an
intermediate node, which means there are two branches going
out from it, or it's a leaf which means there aren't any
branches going out from it.
You can't, in a binary tree, have just one branch coming
out of a node.
There are either no branches or two branches, just by
definition of what we mean by a binary tree --
binary says two.
So, here this tree corresponds where we label
various ones of the leafs.
It corresponds to the code where a corresponds to the
string zero, b corresponds to the string 1, 1, and c
corresponds to the string 1, 0, 1.
Now you look at this and when you look at the tree, when you
look at this as a code, it's not.
Obvious that it's something really stupid about it.
When you look at the tree, it's pretty obvious that
there's something stupid about it, because here we have this
c here, which is sitting off on this leaf, and here we have
this leaf here which isn't doing anything for us at all.
We say gee, we could still keep this prefix condition if
we moved this into here and we drop this off.
So any time that there's something hanging here without
corresponding to a symbol, you would really
like to shorten it.
When you shorten these things and you can't shorten anything
else, namely, when every leaf has a symbol on it you call it
a full tree.
So a full tree is more than a tree, a full tree is a code
tree where the leaves correspond to symbols.
So a full tree has no empty leaves.
Empty leaves can be shortened just like I showed you here,
so we'll talk about full trees, and full trees are sort
of the good trees.
But prefix-free codes don't necessarily have to worry
about that.
Well, now I'm going to prove something to you, and at this
point you really should object, but I don't care.
We will come back and you'll get
straightened out on it later.
I'm going to prove that prefix-free codes are uniquely
decodable, and you should cry foul because I really haven't
defined what uniquely decodable means yet.
You think you know what uniquely decodable means,
which is good.
It means physically that you can look at a string of code
words and you can pick out what all of them are.
We will define it later and you'll find out
it's not that simple.
As we move on, when we start talking about Lempel Ziv codes
and things like that.
You will start to really wonder what
uniquely decodable means.
So it's not quite as simple as it looks.
But anyway, let's prove that prefix-free codes are uniquely
decodable anyway, because prefix-free codes are a
particularly simple example of uniquely decodable codes, and
it's sort of clear that you can, in fact, decode them
because of one of the properties that they have.
The way we're going to prove this is we want to look at a
sequence of symbols or a string of symbols that come
out of the source.
As that string of symbols come out of the source, each symbol
in the string gets mapped into a binary string, and then we
concatenate all those binary strings together.
That's a big mouthful.
So let's look at this code we were just talking about where
the code words are b, c and a.
So if a 1 comes out of the source and then another 1, it
corresponds to the first letter b.
If a 1, zero comes out, it corresponds to the
first letter c.
If a zero comes out, that corresponds to the letter a.
Well now the second symbol comes in and what happens on
that second symbol is if the first symbol was an a, the
second symbol could be a b or a c or an a, which gives rise
to this little sub-tree here.
If the first letter is a b, the second letter could be
either an a, b or a c, which gives rise to this little
sub-tree here.
If we have a c followed by anything, that gives rise to
this little sub-tree here.
You can imagine growing this tree as far as you want to,
although it gets hard to write down.
How do you decode this?
Well, as many things, you want to start at the beginning, and
we know where the beginning is.
That's a basic assumption on all of this source coding.
So knowing where the beginning is, you sit there and you look
at it, and you see a zero as the first letter as a first
binary digit, and zero says I move this way in the tree, and
presto, I say gee, an a must have occurred as the first
source letter.
So what do I do?
I remove the a, I print out a, and then I start to look at
this point.
At this point I'm back where I started at, so if I can decode
the first letter, I can certainly
decode everything else.
If the first letter is a b, what I see is a 1
followed by a 1.
Namely, when I see the first binary 1 come out of the
channel, I don't know what was said.
I know either a b or c was sent.
I have to look at the second letter, the second binary
digit resolves my confusion.
I know that the first source letter was in a b, if it's 1
1, or a c, if it's 1 zero.
I decode that first source letter and then where am I?
I'm either on this tree or on this tree, each of which goes
extending off into the wild blue yonder.
So this says if I know where the beginning is, I can decode
the first letter.
But if I can decode the first letter, I know where the
beginning is for everything else.
Therefore, I can decode that also.
Well, aside from any small amount of confusion about what
uniquely decodable means, that's a perfectly fine
mathematical proof.
So, prefix-free codes are, in fact, uniquely decodable and
that's nice.
So then there's a question.
What is the condition on the lengths of a prefix-free code
which allow you to have unique decodability?
The Kraft inequality is a test on whether there are
prefix-free codes or there are not prefix-free codes
connected with any given set a code word lengths.
This is a very interesting enough inequality.
This is one of the relatively few things in information
theory that was not invented by Claude Shannon.
You sit there and you wonder why didn't Claude Shannon
realize this?
Well, it's because I think he sort of
realized that it was trivial.
He sort of understood it and he was really eager to get on
to the meat of things, which is unusual for him because he
was somebody, more than anyone else I know, who really
understood why you should understand the simple things
before you go on to the more complicated thing.
But anyway, he missed this.
Bob Fano, who some of you might know, who was a
professor emeritus over in LCS, was interested in
information theory.
Then he was teaching a graduate course back in the
'50s here at MIT, and as he often did, he threw out these
problems and said nobody knows how to figure this out.
How kinds of lengths can you have on prefix-free codes, and
what kinds of lengths can't you have?
Kraft was a graduate student at the time.
The next day he came in with this beautiful, elegant proof
and everybody's always known who Kraft is ever since then.
Nobody's ever known what he did after that.
But at least he made his mark on the world
as a graduate student.
So, in a sense, those were good days to be around,
because all the obvious things hadn't been done yet.
But the other thing is you never know what the obvious
things are until you do them.
This didn't look like an obvious problem ahead of time.
Don't talk about a number of other obvious things that cuts
off, because somebody was looking at it in a slightly
different way than other people were looking at it.
You see, back then people said we want to look at these
variable length codes because we want to have some
capability of mapping improbable symbols into long
code words and probable symbols into short code words.
You'll notice that I've done something strange here.
That was our motivation for looking at variable length
codes, but I haven't said a thing about probability.
All I'm dealing with now is the question of what is
possible and what is not possible.
We'll bring in probability later, but now all we're
trying to figure out is what are the sets of code word
lengths you can use, and what are the sets of code word
lengths you can't use.
So what Kraft said is every prefix-free code for an
alphabet x with code word lengths l of x for each letter
in the alphabet x satisfies the sum 2 to the minus length
less than or equal to 1.
In other words, you take all of the code words in the
alphabet, you take the length of each of those code words,
you take 2 to the minus l of that length.
And if this inequality is not satisfied, your code does not
satisfy the prefix condition, there's no way you can create
a prefix-free code which has these lengths, so
you're out of luck.
So you better create a new set of lengths which satisfies
this inequality.
There's also a simple procedure you can go through
which lets you construct a code which has these lengths.
So, in other words, this, in a sense, is a necessary and
sufficient condition on the possibility of constructing
codes with a particular set of lengths.
It has nothing to do with probability.
So it's, in a sense, cleaner than these other results.
So, conversely, if this inequality is satisfied, you
can construct a prefix-free code, and even more strangely,
you can construct it very, very easily, as we'll see.
Finally, a prefix-free code is full -- you remember what a
full prefix-free code is?
It's a code where the tree has nothing that's unused if and
only if this inequality is satisfied with a quality.
So it's a neat result.
It's useful in a lot of places other than source coding.
If you ever get involved with designing protocols for
computer networks or protocols for any kind of computer
communication, you'll find that you use this all the
time, because this says you can do some things, you can't
do other things.
So let's see why it's true.
I'll give you another funny proof that doesn't look like a
proof but it really is.
What I'm going to do is to associate code words with base
2 expansions.
There's a little Genie that early in the morning leaves
things out of these slides when I make them.
It wasn't me, I put it in.
So we're going to prove this by associating code words with
base 2 expansions, which are like decimals, but decimals to
the base 2.
In other words, we're going to take a code word, y1, y2 up to
y sub m where y1 is a binary digit, y2 is a binary digit.
This is a string of binary digits, and we're going to
represent this as a real number.
The real number is the decimal, but it's not a
decimal, it's a becimal, if you will, which is dot y1, y2
up to y sub m.
Which means y1 over 2 plus y2 over 4 plus dot dot dot plus y
sub m over 2 to the minus m.
If you think of it, an ordinary becimal, y1, y2 up to
y sub m, means y1 over 10 plus y2 over 100 plus y3 over 1,000
and so forth.
So this is what people would have developed for decimals
if, in fact, we lived in a base 2 world instead
of a base 10 world.
If you were born without fingers and you only had two
fingers, this is the number system you would use.
When we think about decimals there's
something more involved.
We use decimals all the time to approximate things.
Namely, if I say that a number is 0.12, I don't mean usually
that it's exactly 12 one hundredths.
Usually I mean it's about 12 one hundredths.
The easiest way to do this is to round things down to two
decimal points.
In other words, when I say 0.12, what I really mean is I
am talking about a real number which lies between 12 one
hundredths and 13 one hundredths.
It's greater than or equal to 12 one hundredths and it's
less than 13 one hundredths.
I'll do the same thing in base 2.
As soon as I do this you'll see where the Kraft inequality
comes from.
So I'm going to have this interval here, which the
interval associated with a binary expansion to m digits,
there's a number associated with it which
is this number here.
There's also an interval associated with it, which is 2
to the minus m.
So if I have a code consisting of 0, 0, 0, 1 and 1, what I'm
going to do is represent zero zero as a binary expansion, so
0, 0, is a binary expansion is 0.00, which is zero.
But also as an approximation it's between zero and 1/4.
So I have this interval associated with 0, 0, which is
the interval from zero up to 1/4.
For the code word zero 1, if I'm trying to see whether that
is part of a prefix code, I map it into a number, 0.01 as
a binary expansion.
This number corresponds to the number 1/4, and it also
corresponds into sub length 2 to an interval of size 1/4.
So we go from 1/4 up to 1/2.
Finally, I have 1, which corresponds to the number 1/2,
and since it's only one binary digit long, it corresponds to
the interval 1/2 to 1.
Namely, if I truncate thing to one binary digit, I'm talking
about the entire interval from 1/2 to 1.
So where does the Kraft inequality come from and what
does it have to do with this?
Incidentally, this isn't the way that Kraft proved it.
Kraft was very smart.
He did this as his Master's thesis, too, I believe, and
since he wanted it to be his Master's thesis he didn't want
to make it look quite that trivial or Bob Fano would have
said oh, you ought to do something else for a Master's
thesis also.
So he was cagey and made his proof look a little more
complicated.
So, if a code word x is a prefix of code word y, in
other words, y has some binary expansion, x has some binary
expansion which is the first few letters of y.
Then the number corresponding to x and the interval
corresponding to x, namely, x covers that entire range of
decimal expansions which start with x and goes up to
something which differs from x only in that mth binary digit.
In other words, let me show you what that
means in terms of here.
If I tried to create a code word 0, 0, 0, 1, 0, 0, 0, 1
would correspond to the number 1/16.
1/16 lies in that interval there.
In other words, any time I create a code word which lies
in the interval corresponding to another code word, it means
that this code word has a prefix of that code word.
Sure enough it does --
0, 0, 0, 1, this has this as a prefix.
In other words, there is a perfect mapping between
intervals associated with code words and
prefixes of code words.
So in other words, if we have a prefix-free code, the
intervals for each of these code words has to be distinct.
Well, now we're in nice shape because we know what the size
of each of these intervals is.
The size of the interval associated with a code word of
length 2 is 2 to the minus 2.
To be a prefix-free code, all these
intervals have to be disjoint.
But everything is contained here between zero and 1, and
therefore, when we add up all of these intervals we get a
number which is at most 1.
That's the Kraft inequality.
That's all there is to it.
There was one more thing in it.
It's a full code if and only if the Kraft inequality is
satisfied with a quality.
Where was that?
The code is full if and only if the expansion intervals
fill up zero and 1.
In other words, suppose this was 1 zero, which would lead
into 0.1 with an interval 1/2 to 3/4, and this was all you
had, then this interval up here would be empty, and, in
fact, since this interval is empty you could
shorten the code down.
In other words, you'd have intervals which weren't full
which means that you would have code words that could be
put in there which are not there.
So, that completes the proof.
So now finally, it's time to define unique decodability.
The definition in the notes is a mouthful, so I broke it
apart into a bunch of different pieces here.
A code c for a discrete source is uniquely decodable if for
each string of source letters, x1 up to x sub m, these are
not distinct letters of the alphabet, these are just the
things that might come out of the source. x1 could be the
same as x2, it could be different from x2.
If all of these letters coming out of the source, that
corresponds to some concatenation of these code
words, namely, c of x1, c of x2 up to c of x sub m.
So I have this coming out of the source, this is a string
of binary digits that come out corresponding to this, and I
require that this differs from the concatenation of the code
words c of x1 prime up to c of xm prime.
For any other string, x1 prime x2 prime, x of m prime of
source letters.
Example of this, the thing that we were trying to
construct before c of a equals 1c of b equals zero, c of c
equals 1 zero, doesn't work because the concatenation of a
and b yields 1 zero, c of x1 -- take x1 to be a,
take x2 to be b.
This concatenation, c of x1, c of x2 is c of a, c
of b equals 1 zero.
C of c equals 1 zero, and therefore, you don't have
something that works.
Note that n here can be different from m here.
You'll deal with that in the homework a little bit, not
this week's set.
But that's what unique decodability says.
Let me give you an example.
Here's an example.
Turns out that all uniquely decodable codes have to
satisfy the Kraft inequality also.
Kraft didn't prove this.
In fact, it's a bit of a bear to prove it, and
we'll prove it later.
I suspect that about 2/3 of you will see the proof and say
ugh, and 1/3 of you will say oh, this is really, really
interesting.
I sort of say gee, this is interesting sometimes, and
more often I say ugh, why do we have to do this?
But one example of a code which is uniquely decodable is
first code word is 1, second code word is 1, 0, third is 1,
0, 0, and the fourth is 1, 0, 0, 0.
It doesn't satisfy the Kraft inequality with the quality,
it satisfies it with inequality.
It is uniquely decodable.
How do I know it's uniquely decodable by
just looking at it?
Because any time I see a 1 I know it's the
beginning of a code word.
So I look at some along binary string, it starts out with the
1, I just read digits till it comes to the next one, I say
ah-ha, that next 1 is the first binary digit in the
second code word, the third 1 that I see is the first digit
in the third code word and so forth.
You might say why don't I make the 1 the end of the code word
instead of the beginning of the code word and then we'll
have the prefix condition again.
All I can say is because I want to be perverse and I want
to give you an example of something that is uniquely
decodable but doesn't satisfy the Kraft inequality.
So it's a question.
Why don't we just stick the prefix-free codes and forget
about unique decodability?
You won't understand the answer to that really until we
start looking at things like Lempel Ziv codes, which are,
in fact, a bunch of different things all put together which
are, in fact, very, very practical codes.
But they're not prefix-free codes, and you'll see why
they're not prefix-free codes when we study them.
Then you will see why we want to have a definition of
something which is more involved than that.
So don't worry about that for the time being.
For the time being, the correct idea to take away from
this is that why not just use prefix-free codes, and the
answer is for quite a while we will.
We know that anything we can do with prefix-free codes we
can also do with uniquely decodable codes, anything we
can do with uniquely decodable codes, we can do with
prefix-free codes.
Namely, any old code that you invent has like certain set of
lengths associated with the code words, and if it
satisfies the Kraft inequality, you can easily
develop a prefix-free code which has those lengths and
you might as well do it because then it makes the
coding a lot easier.
Namely, if we have a prefix-free code -- let's go
back and look at that because I never mentioned it and it
really is one of the important advantages
of prefix-free codes.
When I look at this picture and I look at the proof of how
I saw that this was uniquely decodable, what we said was
you start at the beginning and as soon as the decoder sees
the last binary digit of a code word, the decoder can say
ah-ah, it's that code word.
So it's instantaneously decodable.
In other words, all you need to see is the end of the code
word and at that point you know it's the end.
Incidentally, that makes figuring out when you have a
long sequence of code words and you want to stop the whole
thing, it makes things a little bit easier.
This example we started out with of --
I can't find it anymore --
but the example of a uniquely decodable, but non-prefix-free
code, you always had to look at the first digit of the next
code word to know that the old code word was finished.
So, prefix-free codes have that advantage also.
The next topic that we're going to take up is discrete
memoryless sources.
Namely, at this point we have gone as far as we can in
studying prefix-free codes and uniquely decodable codes
strictly in terms of their non-probabalistic properties.
Namely, the question of what set of lengths can you use in
a prefix-free code or uniquely decodable code, and what sets
of lengths can't you use.
So the next thing we want to do is to start looking at the
probabilities of these different symbols and looking
at the probabilities of the different symbols.
We want to find out what sort of lengths we want to choose.
There will be a simple answer to that.
In fact, there'll be two ways of looking at it, one of which
will lead to the idea of entropy, and the other which
will lead to the idea of generating an optimal code.
Both of those approaches are extremely interesting.
But to do that we have to think about a very
simple kind of source.
The simple kind of source is called a
discrete memoryless source.
We know what a discrete source is -- it's a source which
spews out a sequence of symbols from this finite
alphabet that we know and the decoder knows.
The next thing we have to do is to put a probability
measure on the output of the source.
There's a little review of probability at
the end of this lecture.
You should read it carefully.
When you study probability, you have undoubtedly studied
it like most students do, as a way of learning
how to do the problems.
You don't necessarily think of the generalizations of this,
you don't necessarily think of why is it that when you define
a probability space you start out with a sample space and
you talk about elements in the sample space, what's their
sample points.
What do those sample points have to do with random
variables and all of that stuff?
That's the first thing you forget when you haven't been
looking at probability for a while.
Unfortunately, it's something you have to understand when
we're dealing with this because we have a bunch of
things which are not random variables here.
These letters here are things which we
will call chance variables.
A chance variable is just like a random variable but the set
of possible values that it has are not necessarily numbers,
they're just events, as it turns out.
So the sample space is just some set of letters, as we
call them, which are really events in
this probability space.
The probability space assigns probabilities to
sequences of letters.
What we're assuming here is that the sequence of letters
are all statistically independent of each other.
So for example, if you go to Las Vegas and you're reporting
the outcome of some gambling game and you're sending it
back your home computer and your home computer is figuring
out what your odds are in black jack or something, then
every time the dice are rolled you get an independent -- we
hope it's independent if the game is fair --
outcome of the dice.
So that what we're sending then, what we're going to
encode is a sequence of independent, random -- not
random variables because it's not necessarily numbers that
you're interested in, it's this sequence of symbols.
But if we deal with the English text, for example, the
idea that the letters in English text are independent
of each other is absolutely ludicrous.
If it's early enough in the term that you're not
overloaded already, I would suggest that those of you with
a little time go back and read at least the first part of
Shannon's original article about information theory where
he talks about the problem of modeling English.
It's a beautiful treatment, because he starts out same way
we are, dealing with sources which are independent,
identically distributed chance variables.
Then he goes from there, as we will, to looking at Markov
chains of source variables.
Some of you will cringe at this because you might have
seen Markov chains and forgotten about them or you
might have never seen them.
Don't worry about it, there's not that much that's peculiar
about them.
Then he goes on from there to talk about
actual English language.
But the point that he makes is that when you want to study
something as complicated as the English language, the way
that you do it is not to start out by taking a lot of
statistics about English.
If you want to encode English, you start out by making highly
simplifying assumptions, like the assumption that we're
making here that we're dealing with a
discrete memoryless source.
You then learn how to encode discrete memoryless sources.
You then look at blocks of letters out of these sources,
and if they're not independent you look at the probabilities
of these blocks.
If you know how to generate an optimal code for IID letters,
then all you have to do is take these blocks of length m
where you'd have a probability on each possible block, and
your generate a code for the block.
You don't worry about the statistical relationships
between different blocks.
You just say well, if I make my block long enough I don't
care about what happens at the edges, and I'm going to get
everything of interest.
So the idea is by starting out here you have all the clues
you need to start looking at the more interesting cases.
As it turns out with source coding there's another
advantage involved -- looking at independent letters is in
some sense a worst case.
When you look at this worst case, in fact, presto, you
will say if the letters are statistically related, fine.
I'd do even better.
I could do better if I took that into account, but if I'm
not taking it into account, I know exactly
how well I can do.
So what's the definition of that?
Source output is an unending sequence -- x1, x2, x3 --
of randomly selected letters, and these randomly selected
letters are called chance variables.
Each source output is selected from the alphabet using a
common probability measure.
In other words, they're identically distributed.
Each source output is statistically independent of
the other source outputs, x1 up to x k plus 1.
We will call that independent identically distributed, and
we'll abbreviate it IID.
It doesn't mean that the probability measure is 1 over
m for each letter, it's not what we were assuming before.
It means you can have an arbitrary probability
assignment on the different letters, but every letter has
the same probability assignment on it and they're
all independent of each other.
So that's the kind of source we're going to be
dealing with first.
We will find out everything we want to know about how we deal
with that source.
You will understand that source completely and the
other sources you will half understand a little later.