WEBVTT 00:00:00.000 --> 00:00:02.475 align:middle line:90% [SQUEAKING] 00:00:02.475 --> 00:00:03.960 align:middle line:90% [RUSTLING] 00:00:03.960 --> 00:00:06.435 align:middle line:90% [CLICKING] 00:00:06.435 --> 00:00:11.223 align:middle line:90% 00:00:11.223 --> 00:00:12.640 align:middle line:84% ANKUR MOITRA: So today we're going 00:00:12.640 --> 00:00:14.307 align:middle line:84% to be talking about Huffman codes, which 00:00:14.307 --> 00:00:15.800 align:middle line:90% is a pretty elegant topic. 00:00:15.800 --> 00:00:18.260 align:middle line:84% So I just want to give you a reminder from last time. 00:00:18.260 --> 00:00:20.280 align:middle line:84% So what did we talk about last time 00:00:20.280 --> 00:00:22.720 align:middle line:84% was we talked about this notion of a first order 00:00:22.720 --> 00:00:26.240 align:middle line:84% source that defines a distribution on an alphabet 00:00:26.240 --> 00:00:27.140 align:middle line:90% of size k. 00:00:27.140 --> 00:00:29.360 align:middle line:84% So you think about the entries in the set A 00:00:29.360 --> 00:00:32.119 align:middle line:84% as being all the symbols that could arise. 00:00:32.119 --> 00:00:34.240 align:middle line:84% And then what we wanted was we wanted a way 00:00:34.240 --> 00:00:38.080 align:middle line:84% to compress the outputs of our first order channel. 00:00:38.080 --> 00:00:40.800 align:middle line:84% So we did last time was we had this crucial definition, which 00:00:40.800 --> 00:00:42.800 align:middle line:90% was called a coding function. 00:00:42.800 --> 00:00:44.440 align:middle line:84% So we talked about a function which 00:00:44.440 --> 00:00:49.120 align:middle line:84% maps from length n sequences of the alphabet 00:00:49.120 --> 00:00:52.580 align:middle line:84% into some binary code of some varying lengths. 00:00:52.580 --> 00:00:55.040 align:middle line:84% So just the sequence of zeros and ones. 00:00:55.040 --> 00:00:58.800 align:middle line:84% And we want the crucial property that it can be decoded. 00:00:58.800 --> 00:01:02.050 align:middle line:84% So we want the property that for any two sequences 00:01:02.050 --> 00:01:05.470 align:middle line:84% that our channel could generate, as long as they're different, 00:01:05.470 --> 00:01:07.930 align:middle line:90% they map to different outputs. 00:01:07.930 --> 00:01:09.490 align:middle line:84% And last time what we talked about 00:01:09.490 --> 00:01:11.610 align:middle line:84% was we talked about Shannon's theorem 00:01:11.610 --> 00:01:13.410 align:middle line:84% that gave us both upper and lower 00:01:13.410 --> 00:01:15.530 align:middle line:90% bounds on how well we could do. 00:01:15.530 --> 00:01:17.570 align:middle line:84% So in particular, the main thing we proved 00:01:17.570 --> 00:01:20.050 align:middle line:84% was this upper bound that we defined 00:01:20.050 --> 00:01:23.770 align:middle line:84% this quantity called the binary entropy of the channel. 00:01:23.770 --> 00:01:26.610 align:middle line:84% And we showed that the expected length of our code 00:01:26.610 --> 00:01:31.050 align:middle line:84% is at most n times the per symbol entropy plus little 00:01:31.050 --> 00:01:32.890 align:middle line:90% o of n. 00:01:32.890 --> 00:01:35.030 align:middle line:84% So that was the context of what we did last time, 00:01:35.030 --> 00:01:38.130 align:middle line:84% and today we're going to drastically improve upon this. 00:01:38.130 --> 00:01:40.490 align:middle line:84% So first of all, you can wonder how small 00:01:40.490 --> 00:01:42.810 align:middle line:90% we can make this little o of n. 00:01:42.810 --> 00:01:46.130 align:middle line:84% But there's another point, which is a little bit more subtle. 00:01:46.130 --> 00:01:49.970 align:middle line:84% See, this scheme is not really so computationally efficient, 00:01:49.970 --> 00:01:52.450 align:middle line:84% because if I think about what I have to do, 00:01:52.450 --> 00:01:55.810 align:middle line:84% I might, in principle, have to write down 00:01:55.810 --> 00:02:01.860 align:middle line:84% one code word for each possible output of my channel. 00:02:01.860 --> 00:02:05.420 align:middle line:84% So as the length of my channel becomes larger and larger, 00:02:05.420 --> 00:02:08.419 align:middle line:84% now I have k to the n possible strings. 00:02:08.419 --> 00:02:10.900 align:middle line:84% So I'm writing down this giant table 00:02:10.900 --> 00:02:13.340 align:middle line:84% of what are all the possible outputs, 00:02:13.340 --> 00:02:15.340 align:middle line:84% and that might be a very inefficient way 00:02:15.340 --> 00:02:18.340 align:middle line:84% to do these things, because maybe I don't want to come up 00:02:18.340 --> 00:02:20.520 align:middle line:84% with my coding scheme by brute force, 00:02:20.520 --> 00:02:23.220 align:middle line:84% by writing down everything that could possibly output 00:02:23.220 --> 00:02:26.420 align:middle line:84% from the channel and mapping it to what the coding 00:02:26.420 --> 00:02:28.180 align:middle line:90% function would send it to. 00:02:28.180 --> 00:02:30.820 align:middle line:84% So we're going to do a lot better today with something 00:02:30.820 --> 00:02:32.500 align:middle line:90% called Huffman coding. 00:02:32.500 --> 00:02:34.860 align:middle line:84% In fact, let me tell you a bit of the history of Huffman 00:02:34.860 --> 00:02:35.900 align:middle line:90% coding. 00:02:35.900 --> 00:02:45.620 align:middle line:84% So Huffman coding was invented back in 1951 by Huffman. 00:02:45.620 --> 00:02:50.960 align:middle line:84% And it was invented right here at MIT, in fact, 00:02:50.960 --> 00:02:54.460 align:middle line:84% not by a professor, but by a grad student. 00:02:54.460 --> 00:02:56.430 align:middle line:84% So what happened here was that the grad student 00:02:56.430 --> 00:02:58.870 align:middle line:90% was taking a graduate seminar. 00:02:58.870 --> 00:03:01.710 align:middle line:84% And as is often the case with these types of things, 00:03:01.710 --> 00:03:03.810 align:middle line:84% students don't want to take final exams. 00:03:03.810 --> 00:03:06.070 align:middle line:90% I'm sure you guys don't. 00:03:06.070 --> 00:03:10.870 align:middle line:84% And so his professor, Bob Fano, who's still here, 00:03:10.870 --> 00:03:14.070 align:middle line:84% he offered him the choice of whether he could write a term 00:03:14.070 --> 00:03:16.230 align:middle line:90% paper or do the final. 00:03:16.230 --> 00:03:19.350 align:middle line:84% So he chose, of course, writing a term paper. 00:03:19.350 --> 00:03:22.310 align:middle line:84% And the term paper had to be about improving something 00:03:22.310 --> 00:03:24.550 align:middle line:84% in the state of the art in coding theory. 00:03:24.550 --> 00:03:26.430 align:middle line:84% Coding theory was a topic that was just 00:03:26.430 --> 00:03:28.670 align:middle line:90% developing around that time. 00:03:28.670 --> 00:03:29.890 align:middle line:90% He was pretty stuck. 00:03:29.890 --> 00:03:32.950 align:middle line:84% And then he thought, OK, I better cut my losses 00:03:32.950 --> 00:03:34.910 align:middle line:84% and start studying for the final. 00:03:34.910 --> 00:03:36.370 align:middle line:84% And then at the very last minute, 00:03:36.370 --> 00:03:38.610 align:middle line:84% he came up with this amazing new algorithm, 00:03:38.610 --> 00:03:41.690 align:middle line:84% which we teach in a lot of our undergraduate classes, 00:03:41.690 --> 00:03:45.727 align:middle line:84% as a neat example and a powerful example. 00:03:45.727 --> 00:03:47.310 align:middle line:84% So that's what we'll be covering today 00:03:47.310 --> 00:03:51.910 align:middle line:84% is some of the work that went into Huffman's PhD thesis. 00:03:51.910 --> 00:03:54.230 align:middle line:84% So as I mentioned, we'll come back 00:03:54.230 --> 00:03:58.400 align:middle line:84% to the comparison to Shannon later on. 00:03:58.400 --> 00:04:01.120 align:middle line:84% But let me give you a crucial definition. 00:04:01.120 --> 00:04:03.560 align:middle line:84% So we're going to be interested in a stronger 00:04:03.560 --> 00:04:05.840 align:middle line:90% notion of an encoding. 00:04:05.840 --> 00:04:09.280 align:middle line:84% We'll define what's called a prefix code. 00:04:09.280 --> 00:04:11.640 align:middle line:84% Now in comparison to the coding function, 00:04:11.640 --> 00:04:15.520 align:middle line:84% that took an entire sequence of symbols of length n 00:04:15.520 --> 00:04:17.959 align:middle line:84% and it outputs some binary string. 00:04:17.959 --> 00:04:22.520 align:middle line:84% A prefix code is just going to take one single symbol 00:04:22.520 --> 00:04:24.520 align:middle line:84% from our alphabet, and it's going 00:04:24.520 --> 00:04:28.880 align:middle line:84% to output some binary string of variable length. 00:04:28.880 --> 00:04:30.640 align:middle line:84% But now we have a stronger condition 00:04:30.640 --> 00:04:32.270 align:middle line:84% on what we want out of this code. 00:04:32.270 --> 00:04:37.520 align:middle line:90% 00:04:37.520 --> 00:04:41.920 align:middle line:84% So we want it to satisfy the condition that for any two input 00:04:41.920 --> 00:04:44.880 align:middle line:84% symbols ai and aj that are different, 00:04:44.880 --> 00:04:58.250 align:middle line:84% we want that phi of ai is not a prefix of phi of aj. 00:04:58.250 --> 00:05:01.810 align:middle line:84% So this is a stronger condition than them merely being different 00:05:01.810 --> 00:05:03.890 align:middle line:84% in terms of their binary strings. 00:05:03.890 --> 00:05:07.090 align:middle line:84% We want that whatever ai is mapped to, 00:05:07.090 --> 00:05:09.730 align:middle line:84% we want that that binary string doesn't 00:05:09.730 --> 00:05:14.050 align:middle line:84% occur in the front and the beginning of what 00:05:14.050 --> 00:05:16.290 align:middle line:90% aj is mapped to and vice versa. 00:05:16.290 --> 00:05:18.290 align:middle line:84% Of course, we want this to be true for all pairs 00:05:18.290 --> 00:05:21.050 align:middle line:90% of distinct symbols. 00:05:21.050 --> 00:05:22.850 align:middle line:84% So in fact, what I claim is that there's 00:05:22.850 --> 00:05:28.250 align:middle line:84% a nice way to think about a prefix code as a binary tree. 00:05:28.250 --> 00:05:31.890 align:middle line:84% So let's develop a way to think about binary 00:05:31.890 --> 00:05:35.876 align:middle line:90% codes, prefix free codes, as-- 00:05:35.876 --> 00:05:40.690 align:middle line:90% sorry, prefix free codes. 00:05:40.690 --> 00:05:45.040 align:middle line:84% Let's develop a way to visualize them as binary trees. 00:05:45.040 --> 00:05:49.130 align:middle line:90% 00:05:49.130 --> 00:05:54.790 align:middle line:84% So first let me give you an example of a prefix free code. 00:05:54.790 --> 00:05:57.710 align:middle line:84% So let's imagine-- and this will be our running example today 00:05:57.710 --> 00:06:01.100 align:middle line:84% as we see the algorithm under the hood of Huffman coding. 00:06:01.100 --> 00:06:06.350 align:middle line:90% 00:06:06.350 --> 00:06:12.950 align:middle line:84% So let's imagine that we have symbols a, b, C, d, e, and f. 00:06:12.950 --> 00:06:19.070 align:middle line:84% And I'm going to map these to the code words 1, 0, 0, 0, 0, 1, 00:06:19.070 --> 00:06:27.550 align:middle line:84% 1, 0, 0, 1, 0, 0, 1, and 1, 1, 1. 00:06:27.550 --> 00:06:31.750 align:middle line:84% So you can check that this is a prefix free code, because you 00:06:31.750 --> 00:06:34.270 align:middle line:84% can check for each of the possible outputs 00:06:34.270 --> 00:06:35.990 align:middle line:90% for your encoding function. 00:06:35.990 --> 00:06:40.430 align:middle line:84% 1, 0 doesn't appear at the start of any of these strings, 00:06:40.430 --> 00:06:44.310 align:middle line:84% and that's true for all of the other encodings for each 00:06:44.310 --> 00:06:45.830 align:middle line:90% of these symbols. 00:06:45.830 --> 00:06:47.830 align:middle line:84% And one of the ways that we can think about this 00:06:47.830 --> 00:06:52.320 align:middle line:84% is we can think about it as a binary tree in which each branch 00:06:52.320 --> 00:06:55.640 align:middle line:84% in the binary tree tells us whether the next symbol is 00:06:55.640 --> 00:06:57.680 align:middle line:90% either a 0 or 1. 00:06:57.680 --> 00:06:59.560 align:middle line:84% So in this case, let's arrange these 00:06:59.560 --> 00:07:04.160 align:middle line:84% into a binary tree where the leaves are going to represent 00:07:04.160 --> 00:07:06.640 align:middle line:90% what the actual symbols are. 00:07:06.640 --> 00:07:10.000 align:middle line:84% So when I have a symbol like d that my prefix free code is 00:07:10.000 --> 00:07:13.000 align:middle line:84% mapping to the string 0, 1, then I 00:07:13.000 --> 00:07:16.920 align:middle line:84% get that just by tracing from my root to that leaf, 00:07:16.920 --> 00:07:20.960 align:middle line:84% I concatenate these symbols in order and I get 0, 1. 00:07:20.960 --> 00:07:25.400 align:middle line:84% And let me do this for all of the different symbols 00:07:25.400 --> 00:07:28.160 align:middle line:90% in my prefix free code. 00:07:28.160 --> 00:07:30.120 align:middle line:84% And every time going left down the tree 00:07:30.120 --> 00:07:33.200 align:middle line:84% is going to correspond to concatenating with a 0, 00:07:33.200 --> 00:07:35.560 align:middle line:84% and every time going right is going to correspond 00:07:35.560 --> 00:07:38.080 align:middle line:90% to concatenating with a 1. 00:07:38.080 --> 00:07:39.760 align:middle line:84% So when I put this all together I'll 00:07:39.760 --> 00:07:42.910 align:middle line:84% get a binary tree that looks like this. 00:07:42.910 --> 00:07:47.840 align:middle line:90% 00:07:47.840 --> 00:07:49.930 align:middle line:84% And you can check me on the details 00:07:49.930 --> 00:07:53.210 align:middle line:84% that indeed each symbol is mapped 00:07:53.210 --> 00:07:56.130 align:middle line:84% to the position in the binary tree, where 00:07:56.130 --> 00:07:59.650 align:middle line:84% its associated word comes from just following the root to leaf 00:07:59.650 --> 00:08:01.010 align:middle line:90% path. 00:08:01.010 --> 00:08:07.210 align:middle line:84% And now what is the condition that this code is prefix free? 00:08:07.210 --> 00:08:11.810 align:middle line:84% Well, if I had one leaf, if I had one symbol appear right 00:08:11.810 --> 00:08:14.130 align:middle line:84% here, right in the middle of this tree, 00:08:14.130 --> 00:08:17.370 align:middle line:84% I would have another symbol that's a descendant of it. 00:08:17.370 --> 00:08:20.370 align:middle line:84% And that would correspond to a code that's 00:08:20.370 --> 00:08:23.427 align:middle line:84% not prefix free, because in that case, 00:08:23.427 --> 00:08:25.010 align:middle line:84% the way that I would reach this symbol 00:08:25.010 --> 00:08:28.450 align:middle line:84% would be a prefix of how I reach this later symbol. 00:08:28.450 --> 00:08:31.710 align:middle line:84% So in fact, what this constraint on prefix freeness is, 00:08:31.710 --> 00:08:33.929 align:middle line:84% another way to think about it is that when 00:08:33.929 --> 00:08:37.650 align:middle line:84% I arrange all of the symbols into this binary tree describing 00:08:37.650 --> 00:08:40.169 align:middle line:84% their encoding, I want the property 00:08:40.169 --> 00:08:43.049 align:middle line:84% that each position in the tree that 00:08:43.049 --> 00:08:46.010 align:middle line:84% describes the encoding of a symbol 00:08:46.010 --> 00:08:49.460 align:middle line:84% has no other descendants that are symbols beneath it. 00:08:49.460 --> 00:08:52.660 align:middle line:84% So that's an equivalent way to think about prefix free. 00:08:52.660 --> 00:08:55.000 align:middle line:84% So any questions about this key definition? 00:08:55.000 --> 00:08:56.780 align:middle line:90% Does this make sense? 00:08:56.780 --> 00:08:57.580 align:middle line:90% Yeah? 00:08:57.580 --> 00:08:59.740 align:middle line:90% All right. 00:08:59.740 --> 00:09:02.620 align:middle line:84% So now we can talk about what's the important thing 00:09:02.620 --> 00:09:06.300 align:middle line:90% about prefix free codes. 00:09:06.300 --> 00:09:08.500 align:middle line:84% So what I claim is that it actually makes 00:09:08.500 --> 00:09:11.220 align:middle line:90% decoding much, much easier. 00:09:11.220 --> 00:09:12.720 align:middle line:90% So let's see why that is. 00:09:12.720 --> 00:09:22.260 align:middle line:90% 00:09:22.260 --> 00:09:24.340 align:middle line:84% So when I talked about Shannon, I 00:09:24.340 --> 00:09:28.460 align:middle line:84% had to specify how the entire string of length n 00:09:28.460 --> 00:09:30.980 align:middle line:90% was mapped to a binary output. 00:09:30.980 --> 00:09:34.060 align:middle line:84% But now what I can do is I have a much simpler code that 00:09:34.060 --> 00:09:36.962 align:middle line:84% operates on a symbol by symbol basis. 00:09:36.962 --> 00:09:38.420 align:middle line:84% So all I'm going to do is I'm going 00:09:38.420 --> 00:09:41.140 align:middle line:84% to take my sequence of symbols, and I'm just 00:09:41.140 --> 00:09:43.460 align:middle line:84% going to map them to 0, 1 strings using 00:09:43.460 --> 00:09:46.070 align:middle line:90% my prefix free encoding. 00:09:46.070 --> 00:09:49.590 align:middle line:84% And then I'm going to concatenate them all together. 00:09:49.590 --> 00:09:51.210 align:middle line:90% So let's do an example. 00:09:51.210 --> 00:09:54.150 align:middle line:84% What if I tell you I started off with a word 00:09:54.150 --> 00:09:57.720 align:middle line:84% and I ended up with this binary string. 00:09:57.720 --> 00:10:00.550 align:middle line:90% 00:10:00.550 --> 00:10:03.430 align:middle line:84% How would I figure out what was the sequence of symbols 00:10:03.430 --> 00:10:05.550 align:middle line:90% that I started off from? 00:10:05.550 --> 00:10:07.550 align:middle line:84% So we're going to have to use the fact that it's 00:10:07.550 --> 00:10:10.030 align:middle line:90% prefix free crucially. 00:10:10.030 --> 00:10:13.050 align:middle line:84% So how can I go about figuring out what the first letter is? 00:10:13.050 --> 00:10:16.070 align:middle line:84% Any guesses for the first letter? 00:10:16.070 --> 00:10:17.510 align:middle line:90% f. 00:10:17.510 --> 00:10:19.430 align:middle line:84% So in fact, what I can do is I can 00:10:19.430 --> 00:10:23.170 align:middle line:84% see that if I chop off these first three symbols, 00:10:23.170 --> 00:10:25.150 align:middle line:90% that's the encoding of f. 00:10:25.150 --> 00:10:28.910 align:middle line:90% What's my next symbol? 00:10:28.910 --> 00:10:30.110 align:middle line:90% a. 00:10:30.110 --> 00:10:32.630 align:middle line:84% So you can see what's tricky here 00:10:32.630 --> 00:10:36.030 align:middle line:84% is that because my output has variable length, 00:10:36.030 --> 00:10:38.110 align:middle line:84% I don't know where the breaks are. 00:10:38.110 --> 00:10:42.230 align:middle line:84% I smushed everything together when I did this, 00:10:42.230 --> 00:10:48.160 align:middle line:84% but how did I know that the next break should be distance two? 00:10:48.160 --> 00:10:51.400 align:middle line:84% How did you guys figure that out? 00:10:51.400 --> 00:10:52.040 align:middle line:90% Yeah? 00:10:52.040 --> 00:10:54.055 align:middle line:84% AUDIENCE: You traverse down the tree. 00:10:54.055 --> 00:10:55.180 align:middle line:90% ANKUR MOITRA: That's right. 00:10:55.180 --> 00:10:56.040 align:middle line:90% That's right. 00:10:56.040 --> 00:10:58.820 align:middle line:84% And so if I traverse down the tree and hit a leaf, 00:10:58.820 --> 00:10:59.720 align:middle line:90% that's right. 00:10:59.720 --> 00:11:03.040 align:middle line:84% So we can continue this, we'll get the next letter is d, 00:11:03.040 --> 00:11:05.060 align:middle line:84% and last but not least, we'll get e. 00:11:05.060 --> 00:11:07.600 align:middle line:84% So I started off with the word fate. 00:11:07.600 --> 00:11:10.520 align:middle line:84% But now here's the crucial question. 00:11:10.520 --> 00:11:14.280 align:middle line:84% Why couldn't there be some other way, hypothetically, 00:11:14.280 --> 00:11:20.280 align:middle line:84% to break up the string into pieces in a different way that 00:11:20.280 --> 00:11:24.360 align:middle line:84% results in a totally different decoding of what 00:11:24.360 --> 00:11:26.800 align:middle line:90% my original string was. 00:11:26.800 --> 00:11:30.240 align:middle line:84% You see, when I started off with Shannon's encoding theorem, 00:11:30.240 --> 00:11:33.120 align:middle line:84% I was doing it on the entire string, 00:11:33.120 --> 00:11:35.320 align:middle line:84% and the property was that two different strings 00:11:35.320 --> 00:11:37.560 align:middle line:90% mapped to different outputs. 00:11:37.560 --> 00:11:40.920 align:middle line:84% Now I'm doing this on a symbol by symbol basis. 00:11:40.920 --> 00:11:45.690 align:middle line:84% Why am I guaranteed, with my prefix free encoding, 00:11:45.690 --> 00:11:49.210 align:middle line:84% that there's no other way to break this up that makes sense? 00:11:49.210 --> 00:11:52.290 align:middle line:84% It's really because of the prefix freeness. 00:11:52.290 --> 00:11:54.650 align:middle line:84% So what if the break point had really happened 00:11:54.650 --> 00:11:57.330 align:middle line:84% at this yellow line, so that I pulled out the first two 00:11:57.330 --> 00:11:58.530 align:middle line:90% symbols? 00:11:58.530 --> 00:12:01.590 align:middle line:84% Well, if that really mapped to another symbol, 00:12:01.590 --> 00:12:04.400 align:middle line:84% if there was a symbol whose associated code word was 1, 00:12:04.400 --> 00:12:07.790 align:middle line:84% 1, then the trouble is that my code word, my encoding scheme, 00:12:07.790 --> 00:12:11.130 align:middle line:84% would not be prefix free, because that symbol would 00:12:11.130 --> 00:12:15.690 align:middle line:84% have an encoding that's a prefix of that of f. 00:12:15.690 --> 00:12:18.730 align:middle line:84% So prefix freeness is just the constraint 00:12:18.730 --> 00:12:22.770 align:middle line:84% that you can sequentially do decoding, symbol by symbol, 00:12:22.770 --> 00:12:24.770 align:middle line:90% in an unambiguous way. 00:12:24.770 --> 00:12:28.010 align:middle line:84% I always know where the breaks are, even when I erase them. 00:12:28.010 --> 00:12:30.170 align:middle line:90% So does that make sense? 00:12:30.170 --> 00:12:31.730 align:middle line:90% All right. 00:12:31.730 --> 00:12:35.650 align:middle line:84% So prefix freeness is an awesome property. 00:12:35.650 --> 00:12:38.690 align:middle line:84% It's not obvious how it compares to what 00:12:38.690 --> 00:12:40.330 align:middle line:84% the fundamental limits were that we 00:12:40.330 --> 00:12:44.260 align:middle line:90% proved last time using Shannon. 00:12:44.260 --> 00:12:47.580 align:middle line:84% We also don't know what's the best way to-- what's 00:12:47.580 --> 00:12:49.940 align:middle line:84% the way to choose the prefix free code that 00:12:49.940 --> 00:12:52.500 align:middle line:90% minimizes the expected length. 00:12:52.500 --> 00:12:54.500 align:middle line:84% So that'll be our goal today is we're 00:12:54.500 --> 00:12:56.340 align:middle line:84% going to be interested in the following very 00:12:56.340 --> 00:12:58.700 align:middle line:90% basic algorithmic problem. 00:12:58.700 --> 00:13:07.660 align:middle line:84% We want to design a prefix free code 00:13:07.660 --> 00:13:12.330 align:middle line:90% that minimizes expected length. 00:13:12.330 --> 00:13:17.780 align:middle line:90% 00:13:17.780 --> 00:13:21.000 align:middle line:84% And just to be clear, what is our expected length here? 00:13:21.000 --> 00:13:24.100 align:middle line:84% It's just on a by symbol basis, we're 00:13:24.100 --> 00:13:27.420 align:middle line:84% thinking about outputs of our first order source 00:13:27.420 --> 00:13:30.340 align:middle line:84% that are just length 1 for simplicity. 00:13:30.340 --> 00:13:33.860 align:middle line:84% And what I want is I want that the sum over all of the symbols 00:13:33.860 --> 00:13:37.060 align:middle line:84% in my alphabet, the probability that the symbol 00:13:37.060 --> 00:13:43.750 align:middle line:84% is chosen times la, where la is the length 00:13:43.750 --> 00:13:46.630 align:middle line:90% of the encoding of phi of a. 00:13:46.630 --> 00:13:50.090 align:middle line:84% I want to minimize this quantity, because intuitively, 00:13:50.090 --> 00:13:52.750 align:middle line:84% now, when I really care about the case 00:13:52.750 --> 00:13:56.010 align:middle line:84% where n is large and is not 1, well, 00:13:56.010 --> 00:13:59.230 align:middle line:84% my expected length is just going to be n times whatever I can 00:13:59.230 --> 00:14:01.990 align:middle line:90% achieve for a single symbol. 00:14:01.990 --> 00:14:05.110 align:middle line:84% And we know how encoding and decoding work, 00:14:05.110 --> 00:14:08.230 align:middle line:84% so our goal is just to solve this algorithmic problem. 00:14:08.230 --> 00:14:11.470 align:middle line:84% And then we're going to compare it back with Shannon. 00:14:11.470 --> 00:14:13.670 align:middle line:90% So are there any questions? 00:14:13.670 --> 00:14:16.030 align:middle line:90% Makes sense? 00:14:16.030 --> 00:14:17.550 align:middle line:90% All right. 00:14:17.550 --> 00:14:20.830 align:middle line:84% So as I told you, Huffman, in lieu 00:14:20.830 --> 00:14:22.750 align:middle line:84% of taking a final exam at MIT, he 00:14:22.750 --> 00:14:25.190 align:middle line:84% designed a simple greedy algorithm 00:14:25.190 --> 00:14:27.590 align:middle line:84% for optimally solving this problem. 00:14:27.590 --> 00:14:29.470 align:middle line:84% And that's what we're going to do. 00:14:29.470 --> 00:14:34.030 align:middle line:84% We'll explain how the approach works through a running example. 00:14:34.030 --> 00:14:36.910 align:middle line:84% And our running example will be the same one 00:14:36.910 --> 00:14:39.120 align:middle line:90% that I wrote up there. 00:14:39.120 --> 00:14:41.200 align:middle line:84% Our running example-- let me just write it 00:14:41.200 --> 00:14:42.540 align:middle line:90% here so that we have it. 00:14:42.540 --> 00:14:51.000 align:middle line:90% 00:14:51.000 --> 00:14:56.120 align:middle line:84% We'll consider the problem with symbols a, b, c, d, e, 00:14:56.120 --> 00:14:59.860 align:middle line:84% and f and their associated probabilities, 00:14:59.860 --> 00:15:02.880 align:middle line:84% because, of course, the optimal encoding depends crucially 00:15:02.880 --> 00:15:16.080 align:middle line:84% on these probabilities, will be 0.4, 0.05, 0.18, 0.07, 0.20, 00:15:16.080 --> 00:15:18.080 align:middle line:90% 0.10. 00:15:18.080 --> 00:15:21.120 align:middle line:84% And if I did this right, they all sum up to 1. 00:15:21.120 --> 00:15:23.240 align:middle line:84% So what we want to do is we want to understand 00:15:23.240 --> 00:15:25.840 align:middle line:84% what is the optimal prefix free encoding 00:15:25.840 --> 00:15:29.240 align:middle line:84% for this particular first order source. 00:15:29.240 --> 00:15:32.100 align:middle line:84% And we'll see that in solving this problem, 00:15:32.100 --> 00:15:34.440 align:middle line:84% we're going to come up with a general algorithm that 00:15:34.440 --> 00:15:38.180 align:middle line:90% works really for any channel. 00:15:38.180 --> 00:15:39.350 align:middle line:90% OK. 00:15:39.350 --> 00:15:43.900 align:middle line:90% So now let's get started. 00:15:43.900 --> 00:15:47.780 align:middle line:84% So let's imagine, let me tell you what 00:15:47.780 --> 00:15:50.100 align:middle line:90% our strategy is going to be. 00:15:50.100 --> 00:15:53.300 align:middle line:84% We're going to start off with some encoding, which 00:15:53.300 --> 00:15:56.420 align:middle line:84% we remember we think about as a binary tree. 00:15:56.420 --> 00:15:59.940 align:middle line:84% And we're going to pretend that this encoding is optimal 00:15:59.940 --> 00:16:02.620 align:middle line:84% and we're going to derive structural properties 00:16:02.620 --> 00:16:06.180 align:middle line:84% about this optimal encoding that will be enough to figure out 00:16:06.180 --> 00:16:07.940 align:middle line:90% what the encoding is. 00:16:07.940 --> 00:16:09.580 align:middle line:90% So that's our game plan. 00:16:09.580 --> 00:16:13.380 align:middle line:84% So I'm going to pretend that someone has given me already 00:16:13.380 --> 00:16:16.590 align:middle line:90% an optimal prefix free code. 00:16:16.590 --> 00:16:23.798 align:middle line:90% 00:16:23.798 --> 00:16:25.340 align:middle line:84% And what I'm going to do is I'm going 00:16:25.340 --> 00:16:31.380 align:middle line:84% to reverse engineer properties of this binary tree. 00:16:31.380 --> 00:16:34.020 align:middle line:84% Now, just for simplicity, I'm going 00:16:34.020 --> 00:16:39.310 align:middle line:84% to assume that there are no ties. 00:16:39.310 --> 00:16:41.270 align:middle line:84% What I mean by that is when I start off 00:16:41.270 --> 00:16:43.990 align:middle line:84% with all of these symbols, none of the symbols 00:16:43.990 --> 00:16:46.390 align:middle line:84% have exactly the same probability. 00:16:46.390 --> 00:16:47.950 align:middle line:84% It'll just be a bit more of a pain 00:16:47.950 --> 00:16:50.630 align:middle line:84% to talk about what the structural properties are when 00:16:50.630 --> 00:16:53.790 align:middle line:84% we allow ties, even though the same algorithm is essentially 00:16:53.790 --> 00:16:55.590 align:middle line:90% going to work as is. 00:16:55.590 --> 00:16:58.150 align:middle line:84% So I'll assume that none of these symbols 00:16:58.150 --> 00:17:00.670 align:middle line:84% have probabilities that are tied. 00:17:00.670 --> 00:17:03.290 align:middle line:84% And we're going to create new synthetic symbols along the way. 00:17:03.290 --> 00:17:05.415 align:middle line:84% And I'm going to assume that there are no ties even 00:17:05.415 --> 00:17:08.790 align:middle line:90% after we make those operations. 00:17:08.790 --> 00:17:11.390 align:middle line:84% So now let's state the first key Lemma 00:17:11.390 --> 00:17:14.030 align:middle line:84% that's going to get us started in terms 00:17:14.030 --> 00:17:17.190 align:middle line:84% of structural properties of this tree t. 00:17:17.190 --> 00:17:24.170 align:middle line:84% So what I claim is that the smallest probability symbol, 00:17:24.170 --> 00:17:27.109 align:middle line:84% so whichever symbol has the smallest 00:17:27.109 --> 00:17:30.030 align:middle line:84% probability in our first order source, 00:17:30.030 --> 00:17:36.500 align:middle line:84% has the property that it's at least tied for the longest 00:17:36.500 --> 00:17:37.000 align:middle line:90% encoding. 00:17:37.000 --> 00:17:43.240 align:middle line:90% 00:17:43.240 --> 00:17:45.640 align:middle line:84% So what I'm claiming is that whatever is the symbol 00:17:45.640 --> 00:17:47.760 align:middle line:90% with the smallest probability-- 00:17:47.760 --> 00:17:52.400 align:middle line:84% in this case, it's the symbol b here. 00:17:52.400 --> 00:17:55.560 align:middle line:84% I claim that symbol had better be 00:17:55.560 --> 00:18:00.380 align:middle line:84% in one of the deepest nodes in my binary tree 00:18:00.380 --> 00:18:03.760 align:middle line:84% that corresponds to my prefix free code. 00:18:03.760 --> 00:18:05.040 align:middle line:90% OK? 00:18:05.040 --> 00:18:07.320 align:middle line:90% So let's prove this fact. 00:18:07.320 --> 00:18:09.240 align:middle line:84% And then we're going to prove a related fact 00:18:09.240 --> 00:18:14.520 align:middle line:84% about the symbol with the second smallest probability. 00:18:14.520 --> 00:18:17.880 align:middle line:84% So this thing is very easy to prove once you know 00:18:17.880 --> 00:18:19.110 align:middle line:90% what you're trying to prove. 00:18:19.110 --> 00:18:22.600 align:middle line:90% 00:18:22.600 --> 00:18:25.160 align:middle line:84% Does anyone have any intuition for how we could try and prove 00:18:25.160 --> 00:18:27.423 align:middle line:90% this lemma? 00:18:27.423 --> 00:18:28.340 align:middle line:90% Why should it be true? 00:18:28.340 --> 00:18:29.680 align:middle line:90% Yeah. 00:18:29.680 --> 00:18:31.570 align:middle line:84% AUDIENCE: If it is not tied for the longest, 00:18:31.570 --> 00:18:33.450 align:middle line:84% then you could swap it with something 00:18:33.450 --> 00:18:34.470 align:middle line:90% with higher probability. 00:18:34.470 --> 00:18:35.150 align:middle line:90% ANKUR MOITRA: That's right. 00:18:35.150 --> 00:18:35.852 align:middle line:90% That's right. 00:18:35.852 --> 00:18:37.810 align:middle line:84% So that's exactly how our proof is going to go, 00:18:37.810 --> 00:18:40.990 align:middle line:84% because you can see the way that this proof is structured, 00:18:40.990 --> 00:18:44.610 align:middle line:84% it makes it a lot easier to have the algorithm fall out 00:18:44.610 --> 00:18:48.050 align:middle line:84% at the end because I'm not, in one shot, trying to tell you 00:18:48.050 --> 00:18:49.970 align:middle line:90% what the optimal tree is. 00:18:49.970 --> 00:18:53.030 align:middle line:84% I'm just trying to say, if you gave me the optimal tree, 00:18:53.030 --> 00:18:56.650 align:middle line:84% I could find some interesting structural properties 00:18:56.650 --> 00:19:00.850 align:middle line:84% that constrain something about the tree that I'm looking for. 00:19:00.850 --> 00:19:02.590 align:middle line:90% So that's exactly right. 00:19:02.590 --> 00:19:04.750 align:middle line:84% Our proof is going to be, if not, 00:19:04.750 --> 00:19:08.090 align:middle line:84% we're going to swap and contradict the optimality of t 00:19:08.090 --> 00:19:09.450 align:middle line:90% to begin with. 00:19:09.450 --> 00:19:11.890 align:middle line:84% So let's do a proof by contradiction 00:19:11.890 --> 00:19:14.090 align:middle line:90% following exactly that logic. 00:19:14.090 --> 00:19:17.810 align:middle line:84% So suppose that this lemma is not true. 00:19:17.810 --> 00:19:29.290 align:middle line:84% Well, let's let b be the symbol with the lowest probability, 00:19:29.290 --> 00:19:31.220 align:middle line:90% like in our example. 00:19:31.220 --> 00:19:36.540 align:middle line:84% And let's let c be the symbol which 00:19:36.540 --> 00:19:41.060 align:middle line:90% is tied for longest encoding. 00:19:41.060 --> 00:19:46.860 align:middle line:90% 00:19:46.860 --> 00:19:49.380 align:middle line:84% Now the crucial thing is that if we 00:19:49.380 --> 00:19:59.860 align:middle line:90% swap the encodings of b and c. 00:19:59.860 --> 00:20:04.660 align:middle line:84% Well, first of all that's still a valid prefix free code, 00:20:04.660 --> 00:20:08.140 align:middle line:84% because the property of being a prefix free code 00:20:08.140 --> 00:20:10.620 align:middle line:84% doesn't actually depend on how the symbols are 00:20:10.620 --> 00:20:12.540 align:middle line:90% mapped to the binary strings. 00:20:12.540 --> 00:20:16.340 align:middle line:84% It's purely a property of the binary strings I've selected. 00:20:16.340 --> 00:20:18.500 align:middle line:84% When you think about graphically how 00:20:18.500 --> 00:20:21.300 align:middle line:84% we visualize the prefix free code, 00:20:21.300 --> 00:20:24.580 align:middle line:84% it just had this property that all of these nodes, which 00:20:24.580 --> 00:20:28.180 align:middle line:84% were boxes that correspond to symbols being mapped to them, 00:20:28.180 --> 00:20:30.350 align:middle line:84% we need the property that no box had 00:20:30.350 --> 00:20:33.030 align:middle line:90% another box as its descendant. 00:20:33.030 --> 00:20:36.630 align:middle line:84% And now if I interchange the contents of these boxes, 00:20:36.630 --> 00:20:40.150 align:middle line:84% if I swap b and c, then it doesn't actually 00:20:40.150 --> 00:20:43.830 align:middle line:84% change this combinatorial condition of being prefix free. 00:20:43.830 --> 00:20:45.290 align:middle line:90% So this is a valid move. 00:20:45.290 --> 00:20:47.510 align:middle line:84% We're allowed to swap the encodings. 00:20:47.510 --> 00:20:50.630 align:middle line:84% And what we can do is if we swap these encodings, 00:20:50.630 --> 00:20:55.150 align:middle line:84% we can track how the expected length changes. 00:20:55.150 --> 00:20:59.420 align:middle line:84% So let's in particular compute how the expected length changes. 00:20:59.420 --> 00:21:04.870 align:middle line:90% 00:21:04.870 --> 00:21:10.390 align:middle line:84% So we have originally-- well, our new encoding 00:21:10.390 --> 00:21:12.630 align:middle line:84% is going to have the probability of the symbol 00:21:12.630 --> 00:21:16.630 align:middle line:84% b times the length of the encoding for symbol 00:21:16.630 --> 00:21:21.630 align:middle line:84% c plus the probability of the symbol c 00:21:21.630 --> 00:21:24.790 align:middle line:84% times the length of the encoding for b. 00:21:24.790 --> 00:21:26.430 align:middle line:84% That will be some of contribution 00:21:26.430 --> 00:21:30.480 align:middle line:84% to the new expected length, because every time I sample 00:21:30.480 --> 00:21:34.440 align:middle line:84% the symbol b, its encoding is what the encoding of c 00:21:34.440 --> 00:21:36.040 align:middle line:90% previously was. 00:21:36.040 --> 00:21:38.460 align:middle line:84% And every time I sample the symbol c, 00:21:38.460 --> 00:21:40.760 align:middle line:84% its length will be the length of the encoding 00:21:40.760 --> 00:21:43.440 align:middle line:90% of what b previously was. 00:21:43.440 --> 00:21:47.800 align:middle line:84% But now I have to subtract off what it was before. 00:21:47.800 --> 00:21:53.720 align:middle line:84% So I'm going to get Pb times the length of the encoding for b 00:21:53.720 --> 00:21:59.640 align:middle line:84% minus Pc times the length of the encoding for c. 00:21:59.640 --> 00:22:03.060 align:middle line:84% So in particular, when I have my formula for the expected length, 00:22:03.060 --> 00:22:07.460 align:middle line:84% which is right here, notice that for none of the other symbols 00:22:07.460 --> 00:22:12.240 align:middle line:84% am I changing their probability or the length of their encoding. 00:22:12.240 --> 00:22:15.680 align:middle line:84% The only changes are happening between b and c. 00:22:15.680 --> 00:22:17.920 align:middle line:84% So when I look at the difference in the expected 00:22:17.920 --> 00:22:22.240 align:middle line:84% length of my old prefix free code and my new one, 00:22:22.240 --> 00:22:25.120 align:middle line:84% the change in the length is this quantity 00:22:25.120 --> 00:22:29.930 align:middle line:84% right here because all of the other things cancel out. 00:22:29.930 --> 00:22:32.420 align:middle line:84% So let's figure out what's going on here. 00:22:32.420 --> 00:22:36.610 align:middle line:90% 00:22:36.610 --> 00:22:39.490 align:middle line:84% We can write this out in a more convenient way. 00:22:39.490 --> 00:22:45.230 align:middle line:84% This is just Pb times delta minus Pc times delta, 00:22:45.230 --> 00:22:48.290 align:middle line:84% where delta is equal to the length 00:22:48.290 --> 00:22:56.070 align:middle line:84% of the encoding of c minus the length of the encoding of b. 00:22:56.070 --> 00:22:58.930 align:middle line:90% 00:22:58.930 --> 00:23:01.770 align:middle line:84% And we know by assumption, because we're 00:23:01.770 --> 00:23:04.330 align:middle line:84% trying to prove something by contradiction, 00:23:04.330 --> 00:23:09.570 align:middle line:84% that by assumption, that c is the symbol with the longest 00:23:09.570 --> 00:23:10.810 align:middle line:90% encoding. 00:23:10.810 --> 00:23:15.730 align:middle line:84% So b is not tied for the same length of the encoding. 00:23:15.730 --> 00:23:18.770 align:middle line:84% So we know that the difference between these encoding lengths 00:23:18.770 --> 00:23:20.970 align:middle line:90% is at least 1. 00:23:20.970 --> 00:23:23.010 align:middle line:90% So it's strictly positive. 00:23:23.010 --> 00:23:30.540 align:middle line:84% And we also know that Pb is less than Pc. 00:23:30.540 --> 00:23:31.900 align:middle line:90% Why is that true? 00:23:31.900 --> 00:23:34.940 align:middle line:84% Because of the way that I defined Pb. 00:23:34.940 --> 00:23:38.900 align:middle line:84% Pb was the symbol with the smallest probability, 00:23:38.900 --> 00:23:41.380 align:middle line:84% and I assume that there were no ties. 00:23:41.380 --> 00:23:45.700 align:middle line:84% So whatever Pb is, Pc is strictly larger than that. 00:23:45.700 --> 00:23:47.900 align:middle line:84% So if you look at these two quantities 00:23:47.900 --> 00:23:50.300 align:middle line:84% together, then what this means is 00:23:50.300 --> 00:23:56.900 align:middle line:84% that you actually have a change that's strictly negative. 00:23:56.900 --> 00:24:00.820 align:middle line:84% So I've actually decreased the expected length of my encoding. 00:24:00.820 --> 00:24:03.900 align:middle line:84% So that's the proof of the first key lemma. 00:24:03.900 --> 00:24:06.900 align:middle line:84% It tells us this really nice structural property. 00:24:06.900 --> 00:24:09.340 align:middle line:84% And you can see that the proof is a very simple proof 00:24:09.340 --> 00:24:11.940 align:middle line:84% by contradiction, where I just have to figure out 00:24:11.940 --> 00:24:15.660 align:middle line:84% that I want to use the swap and make the calculation of how 00:24:15.660 --> 00:24:18.620 align:middle line:84% much I've improved the length, and argue that it really 00:24:18.620 --> 00:24:20.180 align:middle line:90% is an improvement. 00:24:20.180 --> 00:24:22.880 align:middle line:84% So are there any questions so far? 00:24:22.880 --> 00:24:24.560 align:middle line:90% Good? 00:24:24.560 --> 00:24:26.840 align:middle line:90% All right. 00:24:26.840 --> 00:24:29.960 align:middle line:84% So I claim that there's a companion lemma 00:24:29.960 --> 00:24:33.480 align:middle line:84% that the same lemma holds also for the symbol 00:24:33.480 --> 00:24:36.240 align:middle line:84% with the second smallest probability. 00:24:36.240 --> 00:24:38.600 align:middle line:84% So let me state what I mean exactly 00:24:38.600 --> 00:24:41.400 align:middle line:84% by a version of this lemma that holds 00:24:41.400 --> 00:24:43.840 align:middle line:84% for the second smallest probability. 00:24:43.840 --> 00:24:44.860 align:middle line:90% And then we'll prove it. 00:24:44.860 --> 00:24:48.280 align:middle line:84% The proof will build on our proof of lemma one, 00:24:48.280 --> 00:24:50.720 align:middle line:84% and then we'll be off to the races. 00:24:50.720 --> 00:24:54.880 align:middle line:84% So what I claim is that this lemma 00:24:54.880 --> 00:25:10.840 align:middle line:84% holds for second smallest probability too. 00:25:10.840 --> 00:25:13.260 align:middle line:84% So let's formally state what this lemma is. 00:25:13.260 --> 00:25:20.810 align:middle line:90% 00:25:20.810 --> 00:25:31.290 align:middle line:84% The second smallest symbol, I claim, 00:25:31.290 --> 00:25:35.770 align:middle line:84% is second smallest probability symbol, because that's 00:25:35.770 --> 00:25:38.290 align:middle line:84% how we're measuring their contribution, 00:25:38.290 --> 00:25:41.700 align:middle line:84% is tied for the longest encoding. 00:25:41.700 --> 00:25:49.650 align:middle line:90% 00:25:49.650 --> 00:25:53.370 align:middle line:84% And so the proof for this lemma is going to very closely mirror 00:25:53.370 --> 00:25:56.890 align:middle line:84% the proof for lemma 1, but it's just going to build on it. 00:25:56.890 --> 00:25:58.420 align:middle line:90% So let's prove this fact. 00:25:58.420 --> 00:26:01.170 align:middle line:90% 00:26:01.170 --> 00:26:05.350 align:middle line:84% Actually, in my example, what is my second smallest probability? 00:26:05.350 --> 00:26:07.410 align:middle line:90% Let's double check. 00:26:07.410 --> 00:26:09.780 align:middle line:90% So it's d. 00:26:09.780 --> 00:26:13.770 align:middle line:90% 00:26:13.770 --> 00:26:16.690 align:middle line:84% So now what I'm claiming between lemma one and lemma two 00:26:16.690 --> 00:26:19.220 align:middle line:84% is that these two symbols both had better be 00:26:19.220 --> 00:26:21.780 align:middle line:90% tied for the longest encoding. 00:26:21.780 --> 00:26:24.380 align:middle line:90% So let's follow our lemma. 00:26:24.380 --> 00:26:34.500 align:middle line:84% So let's let b be the symbol with the lowest probability. 00:26:34.500 --> 00:26:44.780 align:middle line:84% Let's let d be the symbol with the second lowest probability. 00:26:44.780 --> 00:26:52.785 align:middle line:84% And now what we know from lemma one, what we already proved, 00:26:52.785 --> 00:26:55.140 align:middle line:84% is that if we look in our optimal tree 00:26:55.140 --> 00:26:59.620 align:middle line:84% we start off at the root, we follow some path, 00:26:59.620 --> 00:27:03.220 align:middle line:84% and then we reach our symbol b, which 00:27:03.220 --> 00:27:06.740 align:middle line:84% is the symbol with the smallest probability. 00:27:06.740 --> 00:27:11.540 align:middle line:84% And let's imagine that we call P the parent. 00:27:11.540 --> 00:27:14.660 align:middle line:90% So whoever comes before. 00:27:14.660 --> 00:27:17.630 align:middle line:84% So this is not the image of a symbol, 00:27:17.630 --> 00:27:20.370 align:middle line:84% because we have this prefix free constraint, 00:27:20.370 --> 00:27:22.590 align:middle line:84% but it's still a node in our binary tree. 00:27:22.590 --> 00:27:26.830 align:middle line:84% There is some parent of wherever b gets mapped to. 00:27:26.830 --> 00:27:33.870 align:middle line:84% And now what I claim is that but P, the way that I've drawn it, 00:27:33.870 --> 00:27:37.040 align:middle line:90% I claim must have another child. 00:27:37.040 --> 00:27:43.750 align:middle line:90% 00:27:43.750 --> 00:27:49.150 align:middle line:84% I claim that there must be some node in my binary tree 00:27:49.150 --> 00:27:52.030 align:middle line:90% that's the sibling of b. 00:27:52.030 --> 00:27:54.310 align:middle line:84% So let's make sure we're on the same page. 00:27:54.310 --> 00:27:56.950 align:middle line:90% So why is this claim true. 00:27:56.950 --> 00:27:59.660 align:middle line:84% Why can I assert that P has another child? 00:27:59.660 --> 00:28:03.310 align:middle line:90% 00:28:03.310 --> 00:28:05.290 align:middle line:84% If it didn't, what would it contradict? 00:28:05.290 --> 00:28:19.500 align:middle line:90% 00:28:19.500 --> 00:28:20.000 align:middle line:90% Yeah? 00:28:20.000 --> 00:28:21.400 align:middle line:84% AUDIENCE: Would it not be a binary tree, 00:28:21.400 --> 00:28:22.813 align:middle line:90% then or cannot be represented? 00:28:22.813 --> 00:28:24.980 align:middle line:84% ANKUR MOITRA: Well, it would still be a binary tree. 00:28:24.980 --> 00:28:28.080 align:middle line:84% So we'll allow 0, 1, or 2 children. 00:28:28.080 --> 00:28:30.300 align:middle line:84% But there's something else that's really crucial. 00:28:30.300 --> 00:28:32.840 align:middle line:84% So think back to my proof strategy. 00:28:32.840 --> 00:28:34.060 align:middle line:90% What am I trying to do? 00:28:34.060 --> 00:28:38.520 align:middle line:84% I'm trying to say that t is an optimal prefix free code. 00:28:38.520 --> 00:28:41.960 align:middle line:84% And then my first lemma was a structural statement 00:28:41.960 --> 00:28:45.720 align:middle line:84% about some property that must be true of that optimal binary 00:28:45.720 --> 00:28:47.440 align:middle line:90% tree. 00:28:47.440 --> 00:28:50.200 align:middle line:84% Lemma two is an even stronger structural property 00:28:50.200 --> 00:28:53.040 align:middle line:90% about that optimal binary tree. 00:28:53.040 --> 00:28:55.640 align:middle line:84% So what would happen if P really didn't 00:28:55.640 --> 00:28:58.660 align:middle line:90% have another child right here? 00:28:58.660 --> 00:28:59.160 align:middle line:90% Yeah? 00:28:59.160 --> 00:29:00.740 align:middle line:84% AUDIENCE: No longer be optimal code. 00:29:00.740 --> 00:29:01.040 align:middle line:90% ANKUR MOITRA: Yes. 00:29:01.040 --> 00:29:02.460 align:middle line:90% It would no longer be optimal. 00:29:02.460 --> 00:29:03.292 align:middle line:90% Why? 00:29:03.292 --> 00:29:05.000 align:middle line:84% AUDIENCE: Because you have an empty space 00:29:05.000 --> 00:29:05.760 align:middle line:90% where you could have just-- 00:29:05.760 --> 00:29:06.160 align:middle line:90% ANKUR MOITRA: Right. 00:29:06.160 --> 00:29:07.535 align:middle line:84% But another way to think about it 00:29:07.535 --> 00:29:11.080 align:middle line:84% is that I know that there's no symbol that's mapped 00:29:11.080 --> 00:29:13.320 align:middle line:90% to P because it's prefix free. 00:29:13.320 --> 00:29:16.250 align:middle line:84% If P didn't have another child, I 00:29:16.250 --> 00:29:20.170 align:middle line:84% could just promote this node and delete everything underneath. 00:29:20.170 --> 00:29:22.890 align:middle line:84% So in this case, I would be strictly 00:29:22.890 --> 00:29:27.370 align:middle line:84% improving the expected length of my prefix free code 00:29:27.370 --> 00:29:29.410 align:middle line:84% because I'm not changing anything 00:29:29.410 --> 00:29:32.170 align:middle line:84% except for the length of the encoding of b, 00:29:32.170 --> 00:29:34.210 align:middle line:90% which is decreasing by 1. 00:29:34.210 --> 00:29:37.370 align:middle line:84% So what I claim is that P must have another child, 00:29:37.370 --> 00:29:43.020 align:middle line:84% otherwise t is definitely not optimal. 00:29:43.020 --> 00:29:45.730 align:middle line:90% 00:29:45.730 --> 00:29:48.440 align:middle line:90% Does that make sense? 00:29:48.440 --> 00:29:52.890 align:middle line:84% So now I know that this child really must be there. 00:29:52.890 --> 00:29:58.490 align:middle line:84% And let's call this child c, which is just the sibling. 00:29:58.490 --> 00:30:03.650 align:middle line:84% And so what I claim now is that if I look in this tree 00:30:03.650 --> 00:30:07.770 align:middle line:84% and I have some path that goes to my other symbol, which 00:30:07.770 --> 00:30:12.610 align:middle line:84% is d, my second smallest probability, well, what could 00:30:12.610 --> 00:30:17.300 align:middle line:84% I do if d was not tied for the second-- 00:30:17.300 --> 00:30:20.140 align:middle line:84% not tied for the longest encoding? 00:30:20.140 --> 00:30:22.740 align:middle line:84% We know that b has the longest encoding, 00:30:22.740 --> 00:30:25.660 align:middle line:84% so d better have an encoding length that's 00:30:25.660 --> 00:30:27.260 align:middle line:90% less than or equal to it. 00:30:27.260 --> 00:30:30.260 align:middle line:84% But what if it was strictly less than the length 00:30:30.260 --> 00:30:32.100 align:middle line:90% of the encoding of b? 00:30:32.100 --> 00:30:34.340 align:middle line:84% We know that there's another child c that 00:30:34.340 --> 00:30:36.860 align:middle line:84% has the same length encoding as b, 00:30:36.860 --> 00:30:41.060 align:middle line:84% so what should I do if d wasn't tied for the second longest 00:30:41.060 --> 00:30:42.020 align:middle line:90% encoding? 00:30:42.020 --> 00:30:44.880 align:middle line:84% I can do the same trick as before, right? 00:30:44.880 --> 00:30:46.740 align:middle line:90% I can swap. 00:30:46.740 --> 00:30:53.900 align:middle line:84% So what I claim is that now, if d 00:30:53.900 --> 00:31:03.620 align:middle line:84% has a strictly shorter encoding, what I claim 00:31:03.620 --> 00:31:14.740 align:middle line:84% is that I can swap c and d and improve the expected length. 00:31:14.740 --> 00:31:19.990 align:middle line:90% 00:31:19.990 --> 00:31:22.615 align:middle line:84% So this is the same proof as before. 00:31:22.615 --> 00:31:23.990 align:middle line:84% So this is a bit subtle, but it's 00:31:23.990 --> 00:31:26.070 align:middle line:90% a purely combinatorial proof. 00:31:26.070 --> 00:31:27.090 align:middle line:90% Are there any questions? 00:31:27.090 --> 00:31:30.470 align:middle line:84% So the way that this worked logically, from lemma one, 00:31:30.470 --> 00:31:33.630 align:middle line:84% we know that b is tied for the longest encoding. 00:31:33.630 --> 00:31:37.950 align:middle line:84% So nothing can have an encoding length that's strictly longer. 00:31:37.950 --> 00:31:42.750 align:middle line:84% We argued that b has a parent P and that P had better 00:31:42.750 --> 00:31:44.350 align:middle line:90% have another child. 00:31:44.350 --> 00:31:46.510 align:middle line:84% And if P has another child, this would better 00:31:46.510 --> 00:31:49.990 align:middle line:84% be some other symbol in terms of the encoding. 00:31:49.990 --> 00:31:52.110 align:middle line:84% And if d were strictly off this line 00:31:52.110 --> 00:31:54.110 align:middle line:84% and the length of its coding were shorter 00:31:54.110 --> 00:31:57.150 align:middle line:84% than the length of the encoding of b and c, what we could do 00:31:57.150 --> 00:31:59.310 align:middle line:90% is we could just swap d and c. 00:31:59.310 --> 00:32:01.030 align:middle line:84% And then the same type of computation 00:32:01.030 --> 00:32:02.990 align:middle line:84% that we did in the proof of lemma one 00:32:02.990 --> 00:32:05.490 align:middle line:84% will show us that we've strictly improved. 00:32:05.490 --> 00:32:05.990 align:middle line:90% Why? 00:32:05.990 --> 00:32:07.870 align:middle line:84% Because the probability of d is strictly 00:32:07.870 --> 00:32:11.520 align:middle line:90% less than the probability of c. 00:32:11.520 --> 00:32:14.440 align:middle line:84% So these are our two key structural lemmas. 00:32:14.440 --> 00:32:16.940 align:middle line:84% And now we're actually almost in good shape. 00:32:16.940 --> 00:32:22.800 align:middle line:84% So we're almost ready to talk about Huffman's algorithm. 00:32:22.800 --> 00:32:26.420 align:middle line:90% So let me do one more claim. 00:32:26.420 --> 00:32:28.200 align:middle line:84% This last structural property is the thing 00:32:28.200 --> 00:32:33.000 align:middle line:84% that's going to make Huffman's greedy algorithm work. 00:32:33.000 --> 00:32:35.820 align:middle line:84% So I claim building off of lemma one and lemma two, 00:32:35.820 --> 00:32:39.040 align:middle line:84% we can say something even stronger. 00:32:39.040 --> 00:32:43.400 align:middle line:84% What I claim is that without loss of generality, these two 00:32:43.400 --> 00:32:48.920 align:middle line:84% symbols, b and d, which are the smallest and second smallest 00:32:48.920 --> 00:32:51.400 align:middle line:84% probabilities, respectively, I claim 00:32:51.400 --> 00:32:57.140 align:middle line:84% without loss of generality their siblings in the tree. 00:32:57.140 --> 00:33:02.200 align:middle line:90% 00:33:02.200 --> 00:33:03.960 align:middle line:90% So why is this true? 00:33:03.960 --> 00:33:14.510 align:middle line:84% Well, from lemma one and lemma two, what do we know? 00:33:14.510 --> 00:33:16.770 align:middle line:90% We can draw it out pictorially. 00:33:16.770 --> 00:33:19.410 align:middle line:84% We know that we start off from the root. 00:33:19.410 --> 00:33:22.570 align:middle line:84% We take some path to the parent P 00:33:22.570 --> 00:33:26.450 align:middle line:90% and then we reach our symbol d. 00:33:26.450 --> 00:33:30.610 align:middle line:84% And we know that P has some other child. 00:33:30.610 --> 00:33:32.290 align:middle line:90% Maybe it's c. 00:33:32.290 --> 00:33:35.050 align:middle line:84% And then we know that there's a path that 00:33:35.050 --> 00:33:40.570 align:middle line:84% reaches the other symbol b which has the second smallest 00:33:40.570 --> 00:33:43.210 align:middle line:84% probability according to our channel. 00:33:43.210 --> 00:33:44.770 align:middle line:84% And we know that all of these things 00:33:44.770 --> 00:33:47.130 align:middle line:90% are tied for the same length. 00:33:47.130 --> 00:33:49.210 align:middle line:84% So what could I do in this case is I could just 00:33:49.210 --> 00:33:51.890 align:middle line:90% swap the positions of b and c. 00:33:51.890 --> 00:33:53.970 align:middle line:84% That doesn't change anything about the length 00:33:53.970 --> 00:33:56.730 align:middle line:84% of the encoding, but it has the property now 00:33:56.730 --> 00:34:01.970 align:middle line:84% that b and d are siblings of each other. 00:34:01.970 --> 00:34:04.730 align:middle line:84% So the proof for all of these structural statements, lemma 00:34:04.730 --> 00:34:09.909 align:middle line:84% one, lemma two, and this claim is really just, if not, swap. 00:34:09.909 --> 00:34:12.870 align:middle line:84% Lemma one was if b isn't the longest, then swap 00:34:12.870 --> 00:34:14.670 align:middle line:90% and you'll improve the length. 00:34:14.670 --> 00:34:17.550 align:middle line:84% Lemma two was, well, we know where b is, 00:34:17.550 --> 00:34:19.690 align:middle line:84% but if d is also not the same length, 00:34:19.690 --> 00:34:21.830 align:middle line:84% then swap and improve the length. 00:34:21.830 --> 00:34:24.429 align:middle line:84% And finally, even knowing that they're 00:34:24.429 --> 00:34:26.870 align:middle line:84% tied for the same length of the encoding, 00:34:26.870 --> 00:34:29.590 align:middle line:90% swap so that they're siblings. 00:34:29.590 --> 00:34:31.429 align:middle line:84% So this is the key structural property 00:34:31.429 --> 00:34:32.750 align:middle line:90% that we're going to use. 00:34:32.750 --> 00:34:36.190 align:middle line:84% Without loss of generality, b and d are siblings. 00:34:36.190 --> 00:34:38.730 align:middle line:84% And now we can come back to our running example 00:34:38.730 --> 00:34:41.510 align:middle line:84% and we can build up what the optimal tree is. 00:34:41.510 --> 00:34:43.070 align:middle line:90% And we'll be able to-- 00:34:43.070 --> 00:34:44.510 align:middle line:84% Huffman's algorithm is just going 00:34:44.510 --> 00:34:47.645 align:middle line:90% to fall out as a consequence. 00:34:47.645 --> 00:34:53.590 align:middle line:84% So let's do the same intuition on this. 00:34:53.590 --> 00:34:57.110 align:middle line:84% So we don't know what our optimal tree looks like. 00:34:57.110 --> 00:34:59.790 align:middle line:90% I'll just draw it as a big blob. 00:34:59.790 --> 00:35:05.640 align:middle line:84% But we know from our claim that without loss of generality 00:35:05.640 --> 00:35:10.280 align:middle line:84% b and d are siblings of each other. 00:35:10.280 --> 00:35:12.140 align:middle line:90% And so now here's the idea. 00:35:12.140 --> 00:35:15.160 align:middle line:84% What I'm going to do is I'm going to take these two symbols, 00:35:15.160 --> 00:35:19.360 align:middle line:84% and I'm going to replace them with a new symbol. 00:35:19.360 --> 00:35:22.360 align:middle line:84% So I'm going to write down a new symbol alpha. 00:35:22.360 --> 00:35:25.760 align:middle line:84% And so here's the step in my Huffman coding algorithm. 00:35:25.760 --> 00:35:33.560 align:middle line:84% I'm going to replace these symbols b and d 00:35:33.560 --> 00:35:39.450 align:middle line:84% with a new symbol, which I'm going to call alpha. 00:35:39.450 --> 00:35:42.280 align:middle line:90% 00:35:42.280 --> 00:35:43.800 align:middle line:84% And I'm going to have the property 00:35:43.800 --> 00:35:46.720 align:middle line:84% that my probability of this new symbol 00:35:46.720 --> 00:35:50.520 align:middle line:84% is going to be the sum of the probabilities of the symbols 00:35:50.520 --> 00:35:52.040 align:middle line:90% that I've merged. 00:35:52.040 --> 00:35:53.640 align:middle line:90% So that's the key idea. 00:35:53.640 --> 00:35:55.200 align:middle line:84% So I'm going to set its probability 00:35:55.200 --> 00:35:58.340 align:middle line:84% to be equal to the sum of the two old probabilities. 00:35:58.340 --> 00:36:02.360 align:middle line:84% This way when I get rid of these two symbols from my list 00:36:02.360 --> 00:36:05.507 align:middle line:84% and instead add my symbol alpha at the bottom, 00:36:05.507 --> 00:36:08.090 align:middle line:84% I'm still going to have a valid distribution because I've just 00:36:08.090 --> 00:36:10.090 align:middle line:90% merged those two probabilities. 00:36:10.090 --> 00:36:14.290 align:middle line:84% But now what we're going to do, the main idea behind Huffman, 00:36:14.290 --> 00:36:18.290 align:middle line:84% is we're going to solve the optimal problem on the smaller 00:36:18.290 --> 00:36:20.296 align:middle line:90% set of symbols. 00:36:20.296 --> 00:36:24.710 align:middle line:84% And then from the optimal binary tree for this new problem, 00:36:24.710 --> 00:36:27.410 align:middle line:84% we're going to derive what the optimal binary tree was 00:36:27.410 --> 00:36:29.690 align:middle line:90% for our original problem. 00:36:29.690 --> 00:36:31.730 align:middle line:84% So are there any questions about the strategy? 00:36:31.730 --> 00:36:34.490 align:middle line:90% Does this make sense? 00:36:34.490 --> 00:36:36.050 align:middle line:90% All right. 00:36:36.050 --> 00:36:40.810 align:middle line:84% So let's do that and let's see what Huffman would do. 00:36:40.810 --> 00:36:42.930 align:middle line:90% All right. 00:36:42.930 --> 00:36:45.480 align:middle line:84% So the corresponding transformation on the tree. 00:36:45.480 --> 00:36:53.810 align:middle line:90% 00:36:53.810 --> 00:36:56.970 align:middle line:84% We started off with this tree t and now we're 00:36:56.970 --> 00:36:59.530 align:middle line:90% going to have this tree t prime. 00:36:59.530 --> 00:37:01.290 align:middle line:84% And what's going to happen is I've 00:37:01.290 --> 00:37:08.340 align:middle line:84% replaced the parent of b and d instead with this new symbol 00:37:08.340 --> 00:37:11.980 align:middle line:84% alpha that occurs in the same exact place in the tree. 00:37:11.980 --> 00:37:15.740 align:middle line:84% But then I've changed nothing else about the tree. 00:37:15.740 --> 00:37:18.790 align:middle line:90% So now we get to the key lemma. 00:37:18.790 --> 00:37:25.380 align:middle line:90% 00:37:25.380 --> 00:37:29.700 align:middle line:84% What I claim is that this new tree 00:37:29.700 --> 00:37:44.380 align:middle line:84% t prime is also optimal for our new problem, where 00:37:44.380 --> 00:37:47.020 align:middle line:90% our new alphabet is A prime. 00:37:47.020 --> 00:37:50.580 align:middle line:84% So we take our old alphabet A, we 00:37:50.580 --> 00:37:54.100 align:middle line:90% add in this new element alpha. 00:37:54.100 --> 00:37:57.700 align:middle line:84% We subtract off the two symbols b and d. 00:37:57.700 --> 00:38:01.750 align:middle line:84% And we've already talked about how we update the probabilities. 00:38:01.750 --> 00:38:05.110 align:middle line:84% So this is the key property that I 00:38:05.110 --> 00:38:08.230 align:middle line:84% was assuming that someone gave me the optimal tree 00:38:08.230 --> 00:38:10.750 align:middle line:90% t for my original problem. 00:38:10.750 --> 00:38:13.110 align:middle line:84% Now I said something structurally about without loss 00:38:13.110 --> 00:38:14.670 align:middle line:84% of generality, where b and d have 00:38:14.670 --> 00:38:16.870 align:middle line:84% to occur in relation to each other. 00:38:16.870 --> 00:38:18.910 align:middle line:84% And now I've created a subproblem 00:38:18.910 --> 00:38:21.270 align:middle line:90% on a smaller number of symbols. 00:38:21.270 --> 00:38:26.790 align:middle line:84% But I claim that just this new tree, t prime that I get, 00:38:26.790 --> 00:38:31.070 align:middle line:84% had actually be an optimal answer to the new problem 00:38:31.070 --> 00:38:33.270 align:middle line:90% that I've constructed. 00:38:33.270 --> 00:38:36.270 align:middle line:84% So does this make sense intuitively? 00:38:36.270 --> 00:38:37.310 align:middle line:90% OK. 00:38:37.310 --> 00:38:38.640 align:middle line:90% So let's prove this lemma. 00:38:38.640 --> 00:38:43.630 align:middle line:90% 00:38:43.630 --> 00:38:53.710 align:middle line:84% So just for notation, let's let L be the expected length of T. 00:38:53.710 --> 00:38:58.070 align:middle line:84% I'm going to let L prime be the expected length of T 00:38:58.070 --> 00:39:01.480 align:middle line:84% prime, this new tree that I've constructed. 00:39:01.480 --> 00:39:07.200 align:middle line:84% And let's prove this lemma that T prime is optimal. 00:39:07.200 --> 00:39:09.100 align:middle line:84% So to make sure you guys are following, 00:39:09.100 --> 00:39:10.640 align:middle line:84% so does anyone have any intuition 00:39:10.640 --> 00:39:14.670 align:middle line:84% for how I can prove this lemma that T prime is again optimal? 00:39:14.670 --> 00:39:19.200 align:middle line:90% 00:39:19.200 --> 00:39:22.090 align:middle line:84% There's basically only one proof we have today. 00:39:22.090 --> 00:39:26.400 align:middle line:90% 00:39:26.400 --> 00:39:27.708 align:middle line:90% Yeah? 00:39:27.708 --> 00:39:29.500 align:middle line:84% AUDIENCE: I was thinking you could swap it, 00:39:29.500 --> 00:39:32.960 align:middle line:84% but I was a little bit concerned because if you 00:39:32.960 --> 00:39:37.560 align:middle line:84% swap the probabilities together, maybe it gets big. 00:39:37.560 --> 00:39:38.180 align:middle line:90% Yeah. 00:39:38.180 --> 00:39:39.960 align:middle line:84% ANKUR MOITRA: So let's think about what 00:39:39.960 --> 00:39:40.900 align:middle line:90% we're trying to prove. 00:39:40.900 --> 00:39:42.920 align:middle line:84% We're trying to prove that T prime is 00:39:42.920 --> 00:39:47.200 align:middle line:84% an optimal prefix free code for this new subproblem I've 00:39:47.200 --> 00:39:51.320 align:middle line:84% constructed on a smaller alphabet. 00:39:51.320 --> 00:39:54.520 align:middle line:84% So what proof strategy do you think 00:39:54.520 --> 00:39:57.400 align:middle line:84% makes sense to prove that T prime is optimal. 00:39:57.400 --> 00:40:00.690 align:middle line:90% 00:40:00.690 --> 00:40:03.900 align:middle line:84% Greedy, induction, some other proof strategy? 00:40:03.900 --> 00:40:15.210 align:middle line:90% 00:40:15.210 --> 00:40:16.830 align:middle line:90% Proof by contradiction? 00:40:16.830 --> 00:40:23.010 align:middle line:90% 00:40:23.010 --> 00:40:24.055 align:middle line:90% Yeah? 00:40:24.055 --> 00:40:25.430 align:middle line:84% AUDIENCE: Proof by contradiction. 00:40:25.430 --> 00:40:26.138 align:middle line:90% ANKUR MOITRA: OK. 00:40:26.138 --> 00:40:27.850 align:middle line:90% That's right. 00:40:27.850 --> 00:40:31.650 align:middle line:84% Because in fact, the easiest way to prove this 00:40:31.650 --> 00:40:34.850 align:middle line:84% is what if T prime is not optimal? 00:40:34.850 --> 00:40:39.010 align:middle line:84% Then what I can do is I could hope to construct-- 00:40:39.010 --> 00:40:42.570 align:middle line:84% I could take whatever tree is optimal for this subproblem. 00:40:42.570 --> 00:40:46.250 align:middle line:84% And then what would I want to do with that tree? 00:40:46.250 --> 00:40:48.210 align:middle line:84% Let's say that proof by contradiction 00:40:48.210 --> 00:40:50.530 align:middle line:90% is the right way to do this. 00:40:50.530 --> 00:40:51.860 align:middle line:90% So suppose not. 00:40:51.860 --> 00:40:54.490 align:middle line:90% 00:40:54.490 --> 00:41:02.740 align:middle line:90% Suppose there's a better tree. 00:41:02.740 --> 00:41:06.780 align:middle line:84% Let's call this tree, I don't know, S. 00:41:06.780 --> 00:41:09.460 align:middle line:84% What should I do with my better tree? 00:41:09.460 --> 00:41:11.760 align:middle line:84% How am I trying to prove a proof by contradiction? 00:41:11.760 --> 00:41:20.700 align:middle line:90% 00:41:20.700 --> 00:41:22.820 align:middle line:84% So you can either answer the question 00:41:22.820 --> 00:41:26.140 align:middle line:84% or ask me a question about what we covered so far. 00:41:26.140 --> 00:41:29.700 align:middle line:84% So it's your choice, but someone's got to answer. 00:41:29.700 --> 00:41:30.280 align:middle line:90% Yeah? 00:41:30.280 --> 00:41:35.132 align:middle line:84% AUDIENCE: I presume you can show that S is actually just deeper. 00:41:35.132 --> 00:41:37.980 align:middle line:84% ANKUR MOITRA: S is actually T prime? 00:41:37.980 --> 00:41:38.880 align:middle line:90% That's true. 00:41:38.880 --> 00:41:41.130 align:middle line:90% But it's harder than I want. 00:41:41.130 --> 00:41:44.645 align:middle line:90% 00:41:44.645 --> 00:41:46.020 align:middle line:84% So some of these things, you have 00:41:46.020 --> 00:41:48.700 align:middle line:84% to have a little bit of intuition about which proof 00:41:48.700 --> 00:41:50.900 align:middle line:90% technique would be the easiest. 00:41:50.900 --> 00:41:53.700 align:middle line:84% So we already agreed it's proof by contradiction. 00:41:53.700 --> 00:41:56.960 align:middle line:84% But the question is what am I trying to contradict and how? 00:41:56.960 --> 00:41:58.960 align:middle line:84% Because that's how proof by contradiction works. 00:41:58.960 --> 00:42:02.150 align:middle line:90% 00:42:02.150 --> 00:42:04.330 align:middle line:90% So what can I do with S? 00:42:04.330 --> 00:42:04.830 align:middle line:90% Yeah? 00:42:04.830 --> 00:42:10.630 align:middle line:90% 00:42:10.630 --> 00:42:13.710 align:middle line:84% AUDIENCE: Can you backtrack to make another version of T? 00:42:13.710 --> 00:42:14.510 align:middle line:90% ANKUR MOITRA: Yeah. 00:42:14.510 --> 00:42:15.510 align:middle line:90% OK. 00:42:15.510 --> 00:42:18.430 align:middle line:84% So let's think about S. I like that. 00:42:18.430 --> 00:42:21.950 align:middle line:90% So S is some other blob. 00:42:21.950 --> 00:42:22.640 align:middle line:90% And it's better. 00:42:22.640 --> 00:42:25.750 align:middle line:90% 00:42:25.750 --> 00:42:29.430 align:middle line:84% What are the symbols that occur in S? 00:42:29.430 --> 00:42:37.430 align:middle line:84% Well, a, c, e, f, and alpha, my new symbol. 00:42:37.430 --> 00:42:39.440 align:middle line:90% So alpha appears somewhere. 00:42:39.440 --> 00:42:42.870 align:middle line:90% 00:42:42.870 --> 00:42:45.830 align:middle line:90% What should I do with alpha now? 00:42:45.830 --> 00:42:49.070 align:middle line:84% Well, maybe I can try and construct a better encoding 00:42:49.070 --> 00:42:52.646 align:middle line:90% for my original problem. 00:42:52.646 --> 00:42:55.000 align:middle line:84% So that's the idea is that there's 00:42:55.000 --> 00:42:56.760 align:middle line:90% a proof by contradiction. 00:42:56.760 --> 00:43:00.840 align:middle line:84% So we suppose that there's a better tree S. 00:43:00.840 --> 00:43:05.280 align:middle line:84% And this better tree is a better tree for this particular problem 00:43:05.280 --> 00:43:08.120 align:middle line:90% on this new alphabet A prime. 00:43:08.120 --> 00:43:12.600 align:middle line:84% And what I want to do is I want to use this S to construct 00:43:12.600 --> 00:43:16.400 align:middle line:84% a better answer to my original problem that contradicts 00:43:16.400 --> 00:43:19.600 align:middle line:90% the optimality of T. 00:43:19.600 --> 00:43:21.860 align:middle line:84% So think about how the logic of this proof worked. 00:43:21.860 --> 00:43:24.280 align:middle line:84% Someone gave me an optimal tree T. 00:43:24.280 --> 00:43:29.640 align:middle line:84% I proved a structural property about where b and d occur. 00:43:29.640 --> 00:43:33.640 align:middle line:84% And now what I'm doing is I'm saying that I want-- 00:43:33.640 --> 00:43:37.120 align:middle line:84% that when I contract b and d, the result I get 00:43:37.120 --> 00:43:40.120 align:middle line:84% is optimal for the corresponding subproblem. 00:43:40.120 --> 00:43:42.640 align:middle line:84% Because if it weren't optimal for the corresponding 00:43:42.640 --> 00:43:45.160 align:middle line:84% subproblem, then my original tree 00:43:45.160 --> 00:43:48.480 align:middle line:84% T could not have been optimal to begin with. 00:43:48.480 --> 00:43:50.780 align:middle line:84% So this is the key point for the logic of the proof. 00:43:50.780 --> 00:43:52.820 align:middle line:84% Even though the proof is very simple, 00:43:52.820 --> 00:43:56.660 align:middle line:84% it takes a little bit to wrap your head around the logic. 00:43:56.660 --> 00:43:59.860 align:middle line:84% So let's imagine that there's a better prefix code 00:43:59.860 --> 00:44:02.260 align:middle line:90% S than the associated tree. 00:44:02.260 --> 00:44:08.840 align:middle line:84% And let's say that it has expected length, 00:44:08.840 --> 00:44:11.100 align:middle line:90% which is L double prime. 00:44:11.100 --> 00:44:12.140 align:middle line:90% OK? 00:44:12.140 --> 00:44:15.620 align:middle line:84% Remember, L prime was the expected length of T prime. 00:44:15.620 --> 00:44:18.980 align:middle line:84% So we're assuming for the purpose of contradiction 00:44:18.980 --> 00:44:23.380 align:middle line:84% that L double prime is strictly less than L prime, that it 00:44:23.380 --> 00:44:25.220 align:middle line:90% does strictly better. 00:44:25.220 --> 00:44:29.260 align:middle line:84% So in this case, what we can do is we can convert. 00:44:29.260 --> 00:44:31.570 align:middle line:84% We can invert the operation that we just did. 00:44:31.570 --> 00:44:39.770 align:middle line:90% 00:44:39.770 --> 00:44:47.300 align:middle line:84% So we can invert the operation, which 00:44:47.300 --> 00:44:50.990 align:middle line:84% took us from the problem on the full alphabet A 00:44:50.990 --> 00:44:54.270 align:middle line:84% to the subproblem on alphabet A prime. 00:44:54.270 --> 00:44:56.390 align:middle line:84% We can replace the symbol alpha that 00:44:56.390 --> 00:45:05.096 align:middle line:84% was our new synthetic symbol by this gadget right here. 00:45:05.096 --> 00:45:08.230 align:middle line:84% And so when we do that, let's track what 00:45:08.230 --> 00:45:11.830 align:middle line:90% the new expected length is. 00:45:11.830 --> 00:45:21.230 align:middle line:84% So the new expected length is L double prime, 00:45:21.230 --> 00:45:26.310 align:middle line:84% because I still pay the expected length, until I-- 00:45:26.310 --> 00:45:29.110 align:middle line:90% for this answer S. 00:45:29.110 --> 00:45:34.430 align:middle line:84% But now by replacing alpha with b and d, every time I incur-- 00:45:34.430 --> 00:45:38.430 align:middle line:84% encounter either b or d, I'm going 00:45:38.430 --> 00:45:43.310 align:middle line:84% to pay one extra unit of length, because previously I 00:45:43.310 --> 00:45:45.770 align:middle line:90% stopped at the symbol alpha. 00:45:45.770 --> 00:45:47.470 align:middle line:90% And now I have to continue. 00:45:47.470 --> 00:45:49.860 align:middle line:84% Wherever alpha was, that's not the end. 00:45:49.860 --> 00:45:52.640 align:middle line:84% I have to go further down to either b or d. 00:45:52.640 --> 00:45:56.800 align:middle line:84% That happens with probability Pb plus Pd, respectively. 00:45:56.800 --> 00:45:59.480 align:middle line:84% So this is my new expected length. 00:45:59.480 --> 00:46:04.360 align:middle line:84% When I take my new better answer for this subproblem 00:46:04.360 --> 00:46:09.500 align:middle line:84% and replace it with inverting this operation, 00:46:09.500 --> 00:46:11.880 align:middle line:90% that's my new expected length. 00:46:11.880 --> 00:46:13.840 align:middle line:84% But when I think about how this compares 00:46:13.840 --> 00:46:19.000 align:middle line:84% to my old expected length, see, if I had the original tree T 00:46:19.000 --> 00:46:23.840 align:middle line:84% prime here, then I pay expected length L prime. 00:46:23.840 --> 00:46:26.480 align:middle line:84% And the way that I get from T prime to T 00:46:26.480 --> 00:46:29.920 align:middle line:84% was I do that same operation, where I take alpha 00:46:29.920 --> 00:46:32.560 align:middle line:90% and replace it with b and d. 00:46:32.560 --> 00:46:37.920 align:middle line:84% So my old expected length was actually L prime plus Pd. 00:46:37.920 --> 00:46:41.800 align:middle line:90% Sorry, plus Pb plus Pd. 00:46:41.800 --> 00:46:44.360 align:middle line:84% And this is just equal to L. So does 00:46:44.360 --> 00:46:45.580 align:middle line:90% everyone see how I got this? 00:46:45.580 --> 00:46:47.635 align:middle line:84% This is the crucial point in the proof. 00:46:47.635 --> 00:46:49.010 align:middle line:84% If you don't understand this, you 00:46:49.010 --> 00:46:52.422 align:middle line:90% don't understand Huffman coding. 00:46:52.422 --> 00:46:54.850 align:middle line:84% And so the crucial thing right here 00:46:54.850 --> 00:46:58.530 align:middle line:84% is that now I've come up with a better solution 00:46:58.530 --> 00:47:00.250 align:middle line:90% for my original problem. 00:47:00.250 --> 00:47:13.090 align:middle line:84% And this contradicts the optimality of T. OK. 00:47:13.090 --> 00:47:15.410 align:middle line:90% So this is the proof. 00:47:15.410 --> 00:47:20.650 align:middle line:84% And I'm not going to go on further until you convince me 00:47:20.650 --> 00:47:24.170 align:middle line:90% you understand the proof. 00:47:24.170 --> 00:47:27.490 align:middle line:84% So who can explain to me how this proof 00:47:27.490 --> 00:47:29.370 align:middle line:90% works in your own words? 00:47:29.370 --> 00:47:33.010 align:middle line:90% The one that's on the board. 00:47:33.010 --> 00:47:34.710 align:middle line:90% Or ask me a question. 00:47:34.710 --> 00:47:37.130 align:middle line:90% Those are your two options. 00:47:37.130 --> 00:47:40.110 align:middle line:84% I guess the third option is try and avoid eye contact, 00:47:40.110 --> 00:47:41.590 align:middle line:90% but that won't be as effective. 00:47:41.590 --> 00:47:47.700 align:middle line:90% 00:47:47.700 --> 00:47:49.080 align:middle line:90% Yeah, go for it. 00:47:49.080 --> 00:47:50.580 align:middle line:84% AUDIENCE: So basically, we're trying 00:47:50.580 --> 00:47:52.900 align:middle line:84% to do it by contradiction, by saying that there's 00:47:52.900 --> 00:47:55.540 align:middle line:84% a more optimal tree S, and we're going 00:47:55.540 --> 00:47:58.180 align:middle line:90% to show that the expected-- 00:47:58.180 --> 00:48:01.740 align:middle line:84% that when we make S back into the same alphabet with T 00:48:01.740 --> 00:48:06.668 align:middle line:84% that S finds that another shape is more optimal. 00:48:06.668 --> 00:48:07.460 align:middle line:90% ANKUR MOITRA: Yeah. 00:48:07.460 --> 00:48:07.920 align:middle line:90% Yeah. 00:48:07.920 --> 00:48:09.340 align:middle line:84% AUDIENCE: Contradiction [INAUDIBLE]. 00:48:09.340 --> 00:48:10.200 align:middle line:90% ANKUR MOITRA: That's right. 00:48:10.200 --> 00:48:12.380 align:middle line:84% AUDIENCE: So to do that, we calculated the expected 00:48:12.380 --> 00:48:16.220 align:middle line:84% length for S, and then given that S is more optimal than t 00:48:16.220 --> 00:48:19.740 align:middle line:84% Prime, you can know that optimal length is 00:48:19.740 --> 00:48:23.500 align:middle line:84% less than L prime plus the reverting 00:48:23.500 --> 00:48:25.572 align:middle line:90% L prime, which is equal to T. 00:48:25.572 --> 00:48:26.280 align:middle line:90% ANKUR MOITRA: OK. 00:48:26.280 --> 00:48:26.900 align:middle line:90% Yeah. 00:48:26.900 --> 00:48:28.060 align:middle line:90% Very good. 00:48:28.060 --> 00:48:30.580 align:middle line:90% That was perfect. 00:48:30.580 --> 00:48:32.842 align:middle line:84% Let's stress test our understanding. 00:48:32.842 --> 00:48:34.300 align:middle line:84% I think the right way to understand 00:48:34.300 --> 00:48:37.440 align:middle line:84% proofs is to think about them like an engineer, 00:48:37.440 --> 00:48:38.920 align:middle line:84% to look at each individual piece. 00:48:38.920 --> 00:48:42.100 align:middle line:84% So we proved a bunch of things today, 00:48:42.100 --> 00:48:46.702 align:middle line:90% all getting to this key lemma. 00:48:46.702 --> 00:48:50.390 align:middle line:84% And one of the key things we had was this claim, 00:48:50.390 --> 00:48:54.830 align:middle line:84% that without loss of generality, in the optimal tree b and d, 00:48:54.830 --> 00:48:58.790 align:middle line:84% the symbols with the smallest and second smallest probability 00:48:58.790 --> 00:49:01.070 align:middle line:90% are siblings. 00:49:01.070 --> 00:49:05.750 align:middle line:84% Where in the proof of this, or in the setup of this lemma three 00:49:05.750 --> 00:49:06.980 align:middle line:90% did I use this claim? 00:49:06.980 --> 00:49:12.390 align:middle line:90% 00:49:12.390 --> 00:49:13.225 align:middle line:90% Yeah? 00:49:13.225 --> 00:49:15.350 align:middle line:84% AUDIENCE: They have to be siblings in order for you 00:49:15.350 --> 00:49:18.470 align:middle line:90% to remove b and d. 00:49:18.470 --> 00:49:20.030 align:middle line:84% ANKUR MOITRA: That's exactly right. 00:49:20.030 --> 00:49:23.230 align:middle line:84% Because I used it even in the setup the definition. 00:49:23.230 --> 00:49:25.150 align:middle line:90% So how did I define T prime? 00:49:25.150 --> 00:49:29.250 align:middle line:84% T prime was what I got from taking an optimal tree T 00:49:29.250 --> 00:49:32.150 align:middle line:84% and then taking the symbols with the smallest 00:49:32.150 --> 00:49:36.750 align:middle line:84% and second smallest probability and replacing them with alpha. 00:49:36.750 --> 00:49:40.230 align:middle line:84% And in the tree, the only way that that can make sense 00:49:40.230 --> 00:49:43.240 align:middle line:84% is if they're actually siblings of each other. 00:49:43.240 --> 00:49:46.160 align:middle line:84% So I invoked this claim implicitly 00:49:46.160 --> 00:49:50.200 align:middle line:84% when I said that let T be an optimal tree without loss 00:49:50.200 --> 00:49:53.080 align:middle line:84% of generality, the two symbols b and d 00:49:53.080 --> 00:49:54.840 align:middle line:90% are siblings of each other. 00:49:54.840 --> 00:49:57.360 align:middle line:84% And then this operation makes sense. 00:49:57.360 --> 00:50:00.400 align:middle line:84% This contraction operation where I replaced b and d 00:50:00.400 --> 00:50:02.440 align:middle line:90% with the new symbol alpha. 00:50:02.440 --> 00:50:05.846 align:middle line:84% And that was what allowed me to proceed with this proof, 00:50:05.846 --> 00:50:09.040 align:middle line:84% because once I had that that operation made sense. 00:50:09.040 --> 00:50:11.300 align:middle line:84% The key point was exactly as you said, 00:50:11.300 --> 00:50:15.600 align:middle line:84% that if this new tree T prime weren't optimal, 00:50:15.600 --> 00:50:19.160 align:middle line:84% then I could do better by inverting my operation, 00:50:19.160 --> 00:50:21.800 align:middle line:84% calculating the expected length, and that would contradict 00:50:21.800 --> 00:50:24.760 align:middle line:90% the optimality of T. OK? 00:50:24.760 --> 00:50:26.980 align:middle line:84% So now are we all on the same page? 00:50:26.980 --> 00:50:27.960 align:middle line:90% We're good? 00:50:27.960 --> 00:50:29.440 align:middle line:90% Happy? 00:50:29.440 --> 00:50:30.160 align:middle line:90% OK. 00:50:30.160 --> 00:50:32.480 align:middle line:90% All right. 00:50:32.480 --> 00:50:34.500 align:middle line:84% When you're trying to learn things yourself, 00:50:34.500 --> 00:50:37.040 align:middle line:84% I think this is a good way to try and do it because you 00:50:37.040 --> 00:50:39.440 align:middle line:90% should stress test yourself. 00:50:39.440 --> 00:50:40.770 align:middle line:90% All right. 00:50:40.770 --> 00:50:43.550 align:middle line:84% But I claim that now we're basically done. 00:50:43.550 --> 00:50:47.290 align:middle line:84% So we're going to have Huffman's main result, because it'll just 00:50:47.290 --> 00:50:50.530 align:middle line:90% be a lot of pretty pictures. 00:50:50.530 --> 00:50:51.940 align:middle line:90% So let's do the pretty pictures. 00:50:51.940 --> 00:50:58.930 align:middle line:90% 00:50:58.930 --> 00:51:01.690 align:middle line:84% And don't fear, I still have more questions for the audience 00:51:01.690 --> 00:51:02.460 align:middle line:90% as we do this. 00:51:02.460 --> 00:51:07.290 align:middle line:90% 00:51:07.290 --> 00:51:10.530 align:middle line:84% So let's just keep track of applying this lemma again 00:51:10.530 --> 00:51:11.976 align:middle line:90% and again. 00:51:11.976 --> 00:51:15.450 align:middle line:84% What I know is that when I replace the symbols 00:51:15.450 --> 00:51:17.330 align:middle line:84% with the smallest and second smallest 00:51:17.330 --> 00:51:20.250 align:middle line:84% probabilities with a new symbol, I 00:51:20.250 --> 00:51:24.450 align:middle line:84% know that my finding the optimal solution to that problem 00:51:24.450 --> 00:51:27.290 align:middle line:84% will help me solve my original problem. 00:51:27.290 --> 00:51:30.050 align:middle line:84% So let's just track it through and figure out 00:51:30.050 --> 00:51:32.250 align:middle line:84% what happens to our original problem. 00:51:32.250 --> 00:51:37.010 align:middle line:84% So our original problem that we cared about was just we have 00:51:37.010 --> 00:51:44.420 align:middle line:84% a, b, c, d, e, and f, and they had associated probabilities 00:51:44.420 --> 00:51:55.780 align:middle line:84% 0.4, 0.05, 0.18, 0.07, 0.20, 0.10. 00:51:55.780 --> 00:51:59.380 align:middle line:84% And then what I'm going to do in my next problem 00:51:59.380 --> 00:52:06.180 align:middle line:84% is I'm going to have symbols a, c, b went away, e, f, and alpha. 00:52:06.180 --> 00:52:08.700 align:middle line:84% And their associated probabilities. 00:52:08.700 --> 00:52:10.240 align:middle line:90% Well, a stays the same. 00:52:10.240 --> 00:52:12.180 align:middle line:90% We haven't touched it. 00:52:12.180 --> 00:52:18.060 align:middle line:84% c stays the same, e stays the same, f stays the same. 00:52:18.060 --> 00:52:21.460 align:middle line:84% And alpha is now 0.12, because I added up 00:52:21.460 --> 00:52:24.300 align:middle line:90% the probabilities of b and c. 00:52:24.300 --> 00:52:28.300 align:middle line:84% What's my next step in this procedure? 00:52:28.300 --> 00:52:31.060 align:middle line:90% Can anyone help me out? 00:52:31.060 --> 00:52:33.380 align:middle line:90% What should I do? 00:52:33.380 --> 00:52:36.140 align:middle line:84% I'm trying to reduce my original problem down 00:52:36.140 --> 00:52:38.870 align:middle line:84% to smaller problems in such a way 00:52:38.870 --> 00:52:44.030 align:middle line:84% that I can back out what the optimal solution is. 00:52:44.030 --> 00:52:45.270 align:middle line:90% What should I do now? 00:52:45.270 --> 00:52:48.040 align:middle line:84% I've got this new problem here on this alphabet A prime. 00:52:48.040 --> 00:52:51.770 align:middle line:90% 00:52:51.770 --> 00:52:52.270 align:middle line:90% Yeah? 00:52:52.270 --> 00:52:54.270 align:middle line:84% AUDIENCE: You do a similar operation with f and alpha. 00:52:54.270 --> 00:52:55.353 align:middle line:90% ANKUR MOITRA: f and alpha. 00:52:55.353 --> 00:52:55.990 align:middle line:90% That's right. 00:52:55.990 --> 00:52:59.150 align:middle line:84% So these become the two symbols with the smallest 00:52:59.150 --> 00:53:01.430 align:middle line:90% and second smallest probability. 00:53:01.430 --> 00:53:03.990 align:middle line:84% And I can replace them with the new symbol beta. 00:53:03.990 --> 00:53:08.750 align:middle line:90% So I get a, c, e, and beta. 00:53:08.750 --> 00:53:11.070 align:middle line:84% And now their associated probabilities 00:53:11.070 --> 00:53:16.610 align:middle line:90% are still 0.4 for a. 00:53:16.610 --> 00:53:17.830 align:middle line:90% I haven't touched it. 00:53:17.830 --> 00:53:23.190 align:middle line:90% 0.18, 0.2, and 0.22. 00:53:23.190 --> 00:53:26.710 align:middle line:84% And after this, my symbols with the smallest probabilities 00:53:26.710 --> 00:53:28.550 align:middle line:90% are c and e. 00:53:28.550 --> 00:53:31.450 align:middle line:84% So I'm going to replace them with a new symbol gamma. 00:53:31.450 --> 00:53:39.140 align:middle line:84% So I have a gamma beta, and they have associated probability 0.4, 00:53:39.140 --> 00:53:43.560 align:middle line:90% 0.38, 0.22. 00:53:43.560 --> 00:53:46.880 align:middle line:84% And last but not least, I take the two ones with the smallest 00:53:46.880 --> 00:53:49.400 align:middle line:84% probabilities right here, and I replace 00:53:49.400 --> 00:53:50.850 align:middle line:90% some of the new symbol delta. 00:53:50.850 --> 00:53:58.120 align:middle line:90% 00:53:58.120 --> 00:54:01.070 align:middle line:84% So what's my optimal prefix code for this? 00:54:01.070 --> 00:54:04.440 align:middle line:90% 00:54:04.440 --> 00:54:08.760 align:middle line:84% If I gave you a problem with just an alphabet of size two, 00:54:08.760 --> 00:54:14.170 align:middle line:84% and the two symbols had probability 0.4 and 0.6, 00:54:14.170 --> 00:54:16.760 align:middle line:84% my goal is to try and take my original problem, which 00:54:16.760 --> 00:54:20.360 align:middle line:84% I didn't know the optimal solution for, and hopefully 00:54:20.360 --> 00:54:25.280 align:middle line:84% reduce it to a problem that's so simple that I can solve it. 00:54:25.280 --> 00:54:26.540 align:middle line:90% So I'm done reducing it. 00:54:26.540 --> 00:54:29.240 align:middle line:90% I ran out of board space. 00:54:29.240 --> 00:54:31.480 align:middle line:84% What's the optimal solution to this last problem 00:54:31.480 --> 00:54:32.700 align:middle line:90% at the very end of the chain? 00:54:32.700 --> 00:54:34.970 align:middle line:84% What's my optimal prefix free code? 00:54:34.970 --> 00:54:38.060 align:middle line:90% 00:54:38.060 --> 00:54:39.100 align:middle line:90% Yeah? 00:54:39.100 --> 00:54:42.580 align:middle line:84% AUDIENCE: Could you say just, like, gamma is like 1 00:54:42.580 --> 00:54:43.595 align:middle line:90% and then like A01? 00:54:43.595 --> 00:54:44.720 align:middle line:90% ANKUR MOITRA: That's right. 00:54:44.720 --> 00:54:46.220 align:middle line:90% Perfect. 00:54:46.220 --> 00:54:48.720 align:middle line:84% Because remember, what's going on? 00:54:48.720 --> 00:54:50.980 align:middle line:84% These prefix free codes, I can think about them 00:54:50.980 --> 00:54:52.820 align:middle line:90% as binary trees. 00:54:52.820 --> 00:54:56.260 align:middle line:84% There's only one binary tree I can do on these two things. 00:54:56.260 --> 00:55:00.060 align:middle line:84% So my optimal code is just this guy right here. 00:55:00.060 --> 00:55:06.060 align:middle line:84% I have the root and then my two symbols get encoded with 00:55:06.060 --> 00:55:07.200 align:middle line:90% either a 0 or 1. 00:55:07.200 --> 00:55:09.660 align:middle line:84% Doesn't matter which one is which. 00:55:09.660 --> 00:55:12.973 align:middle line:84% And now for the question for the audience. 00:55:12.973 --> 00:55:16.100 align:middle line:84% OK, my goal wasn't to get down to a trivial problem 00:55:16.100 --> 00:55:17.740 align:middle line:90% that I can solve. 00:55:17.740 --> 00:55:21.020 align:middle line:84% If we give you on your exam three a question like this, 00:55:21.020 --> 00:55:23.420 align:middle line:84% you can't just replace it with your own favorite trivial 00:55:23.420 --> 00:55:25.460 align:middle line:90% question and solve it. 00:55:25.460 --> 00:55:27.890 align:middle line:84% I care about the solution to my original problem. 00:55:27.890 --> 00:55:30.460 align:middle line:90% 00:55:30.460 --> 00:55:33.410 align:middle line:84% So how can I try and get the solution to my original problem? 00:55:33.410 --> 00:55:36.390 align:middle line:90% Maybe let's start simple. 00:55:36.390 --> 00:55:40.190 align:middle line:84% What's the optimal solution to this problem right here? 00:55:40.190 --> 00:55:41.950 align:middle line:90% Slightly harder. 00:55:41.950 --> 00:55:45.150 align:middle line:84% There is an alphabet of size three. 00:55:45.150 --> 00:55:47.910 align:middle line:84% We know what the optimal solution is on the associated 00:55:47.910 --> 00:55:49.230 align:middle line:90% subproblem. 00:55:49.230 --> 00:55:52.390 align:middle line:84% So how can I get my next optimal solution trying 00:55:52.390 --> 00:55:55.470 align:middle line:90% to go the other direction? 00:55:55.470 --> 00:55:57.410 align:middle line:84% Or what would the optimal solution be? 00:55:57.410 --> 00:56:02.770 align:middle line:90% 00:56:02.770 --> 00:56:03.270 align:middle line:90% Yeah? 00:56:03.270 --> 00:56:05.445 align:middle line:84% AUDIENCE: Adding a branch off of delta. 00:56:05.445 --> 00:56:06.570 align:middle line:90% ANKUR MOITRA: That's right. 00:56:06.570 --> 00:56:07.450 align:middle line:90% That's right. 00:56:07.450 --> 00:56:08.850 align:middle line:90% Because how do we get delta? 00:56:08.850 --> 00:56:11.070 align:middle line:90% It was merging gamma and beta. 00:56:11.070 --> 00:56:13.350 align:middle line:84% What we did was we invoked the claim 00:56:13.350 --> 00:56:15.790 align:middle line:84% that says that the optimal tree for this 00:56:15.790 --> 00:56:19.070 align:middle line:84% had better have the symbol with the smallest and second smallest 00:56:19.070 --> 00:56:21.990 align:middle line:90% probabilities be siblings. 00:56:21.990 --> 00:56:25.070 align:middle line:84% And now I can undo my operation where I merge them together 00:56:25.070 --> 00:56:27.750 align:middle line:84% into a new symbol delta, and what I'll get 00:56:27.750 --> 00:56:30.190 align:middle line:84% is I'll get this new binary tree. 00:56:30.190 --> 00:56:31.580 align:middle line:90% A stays where it is. 00:56:31.580 --> 00:56:33.440 align:middle line:90% That's still encoded as 0. 00:56:33.440 --> 00:56:35.600 align:middle line:84% But now we get a deeper binary tree 00:56:35.600 --> 00:56:42.800 align:middle line:84% where I replace delta with beta and gamma. 00:56:42.800 --> 00:56:50.124 align:middle line:84% So beta gets mapped to 1, 0 and gamma gets mapped to 1, 1. 00:56:50.124 --> 00:56:50.932 align:middle line:90% OK? 00:56:50.932 --> 00:56:53.140 align:middle line:84% Just to make sure that we're really on the same page, 00:56:53.140 --> 00:56:55.000 align:middle line:90% what should I do next? 00:56:55.000 --> 00:56:57.010 align:middle line:90% What's my next optimal solution? 00:56:57.010 --> 00:57:00.200 align:middle line:90% 00:57:00.200 --> 00:57:02.940 align:middle line:90% What should I do? 00:57:02.940 --> 00:57:03.440 align:middle line:90% Yeah? 00:57:03.440 --> 00:57:04.320 align:middle line:90% AUDIENCE: Split up gamma. 00:57:04.320 --> 00:57:05.528 align:middle line:90% ANKUR MOITRA: Split up gamma. 00:57:05.528 --> 00:57:06.120 align:middle line:90% Perfect. 00:57:06.120 --> 00:57:09.000 align:middle line:84% So now I think you guys see how this works. 00:57:09.000 --> 00:57:14.080 align:middle line:84% So my new solution right here would be a is over here. 00:57:14.080 --> 00:57:19.000 align:middle line:84% I still have beta where it was before I split up gamma 00:57:19.000 --> 00:57:25.040 align:middle line:90% into c and e. 00:57:25.040 --> 00:57:27.880 align:middle line:84% And then I go backwards once more. 00:57:27.880 --> 00:57:29.420 align:middle line:84% I'm going to run out of space soon. 00:57:29.420 --> 00:57:33.730 align:middle line:84% So this will be the last one that I'll do. 00:57:33.730 --> 00:57:35.110 align:middle line:84% Actually, maybe I'll do one more. 00:57:35.110 --> 00:57:35.750 align:middle line:90% That's fine. 00:57:35.750 --> 00:57:45.890 align:middle line:90% 00:57:45.890 --> 00:57:48.970 align:middle line:84% And my last one, which I'll need a bit of space to draw. 00:57:48.970 --> 00:57:51.770 align:middle line:90% 00:57:51.770 --> 00:57:54.290 align:middle line:90% OK. 00:57:54.290 --> 00:57:56.020 align:middle line:90% I guess I'll draw it over here. 00:57:56.020 --> 00:58:08.650 align:middle line:90% 00:58:08.650 --> 00:58:11.600 align:middle line:84% So my last one is this binary tree. 00:58:11.600 --> 00:58:30.220 align:middle line:90% 00:58:30.220 --> 00:58:34.660 align:middle line:84% So this is a valid prefix free encoding. 00:58:34.660 --> 00:58:36.680 align:middle line:84% What can I say about this tree T? 00:58:36.680 --> 00:58:39.500 align:middle line:90% 00:58:39.500 --> 00:58:40.740 align:middle line:90% It's optimal. 00:58:40.740 --> 00:58:42.860 align:middle line:90% Why is it optimal? 00:58:42.860 --> 00:58:44.700 align:middle line:84% I claim that this is an optimal solution 00:58:44.700 --> 00:58:46.580 align:middle line:84% to the original running example I gave you 00:58:46.580 --> 00:58:48.420 align:middle line:90% at the beginning of class. 00:58:48.420 --> 00:58:49.630 align:middle line:90% Why is this tree optimal? 00:58:49.630 --> 00:58:55.540 align:middle line:90% 00:58:55.540 --> 00:59:00.980 align:middle line:84% Because in each step, all we did was we reasoned about any tree 00:59:00.980 --> 00:59:03.720 align:middle line:84% that's optimal without loss of generality 00:59:03.720 --> 00:59:06.940 align:middle line:84% had the property that b and d were siblings. 00:59:06.940 --> 00:59:09.480 align:middle line:84% And then for the new problem on my alphabet 00:59:09.480 --> 00:59:11.920 align:middle line:84% A prime, without loss of generality, 00:59:11.920 --> 00:59:16.100 align:middle line:84% it had the property that f and alpha were siblings. 00:59:16.100 --> 00:59:17.940 align:middle line:84% And for my new subproblem, it had 00:59:17.940 --> 00:59:21.100 align:middle line:84% the property with loss of generality and so on. 00:59:21.100 --> 00:59:23.420 align:middle line:84% So in fact, we argued in each step 00:59:23.420 --> 00:59:26.790 align:middle line:84% that without loss of generality, our optimal tree 00:59:26.790 --> 00:59:28.750 align:middle line:90% could take this form. 00:59:28.750 --> 00:59:31.030 align:middle line:84% Now, there could be other ways of doing it because I 00:59:31.030 --> 00:59:33.350 align:middle line:90% could swap f and c together. 00:59:33.350 --> 00:59:36.190 align:middle line:84% But we argued that whatever we get out of this procedure 00:59:36.190 --> 00:59:38.230 align:middle line:90% had actually better be optimal. 00:59:38.230 --> 00:59:41.350 align:middle line:90% So this was Huffman coding. 00:59:41.350 --> 00:59:42.730 align:middle line:90% In fact, we're not done. 00:59:42.730 --> 00:59:44.470 align:middle line:84% There's one more crucial question 00:59:44.470 --> 00:59:47.870 align:middle line:84% that we want to ask and answer about it. 00:59:47.870 --> 00:59:49.310 align:middle line:84% But right now, what we did was we 00:59:49.310 --> 00:59:51.510 align:middle line:84% gave you a simple greedy algorithm 00:59:51.510 --> 00:59:54.870 align:middle line:84% that finds the optimal prefix free code. 00:59:54.870 --> 00:59:57.870 align:middle line:84% It's not like Shannon where I say it's approximately optimal 00:59:57.870 --> 00:59:59.170 align:middle line:90% up to some additive terms. 00:59:59.170 --> 01:00:02.070 align:middle line:84% It is the optimal prefix free code. 01:00:02.070 --> 01:00:04.630 align:middle line:84% But there's one more crucial question, which 01:00:04.630 --> 01:00:07.290 align:middle line:90% is, how good is this code? 01:00:07.290 --> 01:00:12.030 align:middle line:84% So when we talked about Shannon, I have it conveniently up here. 01:00:12.030 --> 01:00:15.670 align:middle line:84% It's not just that we argue that there's a coding function and we 01:00:15.670 --> 01:00:17.110 align:middle line:90% constructed it. 01:00:17.110 --> 01:00:20.170 align:middle line:84% It's that we actually argued about how good it is. 01:00:20.170 --> 01:00:22.550 align:middle line:84% We bounded its performance, and we 01:00:22.550 --> 01:00:27.120 align:middle line:84% saw that the binary entropy fell out from the end of that. 01:00:27.120 --> 01:00:31.640 align:middle line:84% So our last step is to bound the performance of the Huffman code 01:00:31.640 --> 01:00:33.920 align:middle line:90% and show that it's very good. 01:00:33.920 --> 01:00:38.360 align:middle line:84% In fact, it'll really beat what we proved in Shannon last time. 01:00:38.360 --> 01:00:42.920 align:middle line:84% So that's the last part of today's lecture. 01:00:42.920 --> 01:00:45.300 align:middle line:84% So let me state one helper lemma, 01:00:45.300 --> 01:00:50.240 align:middle line:84% which you're going to give an alternative proof 01:00:50.240 --> 01:00:53.600 align:middle line:84% for in recitation and talk about what are figures 01:00:53.600 --> 01:00:56.040 align:middle line:90% to illustrate this proof. 01:00:56.040 --> 01:00:58.020 align:middle line:84% This proof is called the Kraft inequality. 01:00:58.020 --> 01:01:00.200 align:middle line:84% And once we have this Kraft inequality, 01:01:00.200 --> 01:01:03.440 align:middle line:84% we'll be able to get a very nice bound on the performance 01:01:03.440 --> 01:01:04.370 align:middle line:90% of Huffman codes. 01:01:04.370 --> 01:01:07.320 align:middle line:90% 01:01:07.320 --> 01:01:09.670 align:middle line:84% So this is a purely combinatorial lemma. 01:01:09.670 --> 01:01:18.400 align:middle line:90% 01:01:18.400 --> 01:01:22.640 align:middle line:84% And it's called the Kraft inequality. 01:01:22.640 --> 01:01:29.690 align:middle line:84% So what I claim is that for any natural numbers l1, 01:01:29.690 --> 01:01:33.010 align:middle line:90% l2 all the way up to lk. 01:01:33.010 --> 01:01:34.730 align:middle line:84% So of course, natural number just 01:01:34.730 --> 01:01:37.590 align:middle line:90% means non-negative integers. 01:01:37.590 --> 01:01:39.930 align:middle line:90% So any non-negative integers. 01:01:39.930 --> 01:01:55.290 align:middle line:84% I claim that there is a binary tree with leaves at depths 01:01:55.290 --> 01:01:57.170 align:middle line:90% given by these values. 01:01:57.170 --> 01:02:00.250 align:middle line:84% So at depths l1, l2, all the way up 01:02:00.250 --> 01:02:05.410 align:middle line:84% to lq, if and only if the following condition holds. 01:02:05.410 --> 01:02:13.730 align:middle line:84% The sum of i equals 1 to k of 2 to the minus li is at most 1. 01:02:13.730 --> 01:02:16.770 align:middle line:84% So this is a little bit of a mouthful. 01:02:16.770 --> 01:02:20.150 align:middle line:84% So what I'm asking you is some of packing constraint. 01:02:20.150 --> 01:02:24.420 align:middle line:84% Obviously you can't have three leaves at depth two 01:02:24.420 --> 01:02:27.500 align:middle line:90% because it's a binary tree. 01:02:27.500 --> 01:02:29.780 align:middle line:84% And what this Kraft inequality tells you is it 01:02:29.780 --> 01:02:32.540 align:middle line:84% tells you a necessary and sufficient condition. 01:02:32.540 --> 01:02:35.700 align:middle line:84% If you give me the sequence of depths, whether or not there 01:02:35.700 --> 01:02:40.900 align:middle line:84% is a tree with leaves at those depths. 01:02:40.900 --> 01:02:44.420 align:middle line:84% So let's do a simple example just to make sure 01:02:44.420 --> 01:02:46.150 align:middle line:90% that we're on the same page. 01:02:46.150 --> 01:02:55.380 align:middle line:90% 01:02:55.380 --> 01:03:00.340 align:middle line:84% So let's start off with some depths of the leaves 01:03:00.340 --> 01:03:03.260 align:middle line:84% that satisfy the Kraft inequality. 01:03:03.260 --> 01:03:10.540 align:middle line:84% Let's say l1 equals 2, l2 equals 3, l3 equals 3. 01:03:10.540 --> 01:03:14.060 align:middle line:84% You can check when you add up the Kraft inequality 01:03:14.060 --> 01:03:15.860 align:middle line:90% that you're good. 01:03:15.860 --> 01:03:18.820 align:middle line:90% 01:03:18.820 --> 01:03:25.960 align:middle line:84% Actually, sorry this should be depth 1, 2, 2. 01:03:25.960 --> 01:03:29.740 align:middle line:90% Depth 1, 2, 2. 01:03:29.740 --> 01:03:33.480 align:middle line:84% And so what would be the associated tree here? 01:03:33.480 --> 01:03:37.400 align:middle line:84% Well, here's one example of a binary tree 01:03:37.400 --> 01:03:42.680 align:middle line:84% that satisfies these conditions as promised by Kraft inequality. 01:03:42.680 --> 01:03:45.400 align:middle line:84% So I take this associated binary tree. 01:03:45.400 --> 01:03:48.720 align:middle line:84% It has one leaf of depth 1 because we're going down 01:03:48.720 --> 01:03:51.040 align:middle line:90% distance 1 from the root. 01:03:51.040 --> 01:03:54.600 align:middle line:90% It has one leaf of depth 2. 01:03:54.600 --> 01:03:57.920 align:middle line:84% And its third leaf is also of depth 2. 01:03:57.920 --> 01:04:01.600 align:middle line:84% And you can check that 1/2 plus 1/4 plus 1/4 is less than 01:04:01.600 --> 01:04:02.800 align:middle line:90% or equal to 1. 01:04:02.800 --> 01:04:06.720 align:middle line:84% So this is an example which both satisfies this Kraft condition 01:04:06.720 --> 01:04:10.200 align:middle line:84% and has an associated binary tree. 01:04:10.200 --> 01:04:12.520 align:middle line:84% So this is just one illustration. 01:04:12.520 --> 01:04:18.480 align:middle line:84% But what if instead I take l3 and replace it by l3 01:04:18.480 --> 01:04:19.520 align:middle line:90% is equal to-- 01:04:19.520 --> 01:04:23.090 align:middle line:90% 01:04:23.090 --> 01:04:23.850 align:middle line:90% let's see. 01:04:23.850 --> 01:04:25.930 align:middle line:90% 3. 01:04:25.930 --> 01:04:29.250 align:middle line:90% And l4 is equal to 3. 01:04:29.250 --> 01:04:32.010 align:middle line:84% This still satisfies the Kraft inequality. 01:04:32.010 --> 01:04:36.890 align:middle line:84% What would I do is I would just replace this part with a deeper 01:04:36.890 --> 01:04:42.130 align:middle line:84% binary tree with a3 here and a4 here. 01:04:42.130 --> 01:04:43.810 align:middle line:84% So the proof for the Kraft inequality 01:04:43.810 --> 01:04:47.490 align:middle line:84% essentially goes the way you'd expect it to is you 01:04:47.490 --> 01:04:49.280 align:middle line:90% just build up a binary tree. 01:04:49.280 --> 01:05:04.810 align:middle line:90% 01:05:04.810 --> 01:05:08.310 align:middle line:84% The key point is that if I had all of the same lengths, 01:05:08.310 --> 01:05:12.170 align:middle line:84% like I had l1 equals 1 and l2 equals 1, 01:05:12.170 --> 01:05:15.490 align:middle line:84% then it's just a regular binary tree. 01:05:15.490 --> 01:05:17.300 align:middle line:84% And every time what's going on is 01:05:17.300 --> 01:05:19.380 align:middle line:84% that when I have larger depths, it really 01:05:19.380 --> 01:05:23.900 align:middle line:84% corresponds to replacing the leaf with some deeper binary 01:05:23.900 --> 01:05:26.580 align:middle line:84% tree, exactly as I did in this example. 01:05:26.580 --> 01:05:29.200 align:middle line:84% So you're going to prove this in recitation. 01:05:29.200 --> 01:05:31.800 align:middle line:84% So I'll just leave the proof as to be continued. 01:05:31.800 --> 01:05:34.780 align:middle line:84% You're going to work on both understanding the proof of this 01:05:34.780 --> 01:05:38.020 align:middle line:84% and drawing diagrams to illustrate the proof. 01:05:38.020 --> 01:05:40.580 align:middle line:84% But for us, we're going to take this for granted, 01:05:40.580 --> 01:05:42.740 align:middle line:84% and we're going to prove our main theorem 01:05:42.740 --> 01:05:46.420 align:middle line:84% about Huffman codes, which is not just that there's 01:05:46.420 --> 01:05:49.192 align:middle line:84% an algorithm for building Huffman codes, 01:05:49.192 --> 01:05:51.400 align:middle line:84% but we're going to give a bound on their performance, 01:05:51.400 --> 01:05:54.620 align:middle line:84% and we're going to relate it back to what we proved 01:05:54.620 --> 01:05:57.390 align:middle line:90% on Thursday using Shannon. 01:05:57.390 --> 01:06:00.380 align:middle line:90% 01:06:00.380 --> 01:06:03.440 align:middle line:84% So here's our last statement for today that we're going to prove, 01:06:03.440 --> 01:06:07.140 align:middle line:84% and we're going to build on this Kraft inequality. 01:06:07.140 --> 01:06:27.110 align:middle line:84% What I claim is that the Huffman code has expected length l, 01:06:27.110 --> 01:06:33.470 align:middle line:90% which is between h and h plus 1. 01:06:33.470 --> 01:06:36.350 align:middle line:84% So this is an incredibly strong statement 01:06:36.350 --> 01:06:40.150 align:middle line:84% because I'm saying that the binary entropy really tightly 01:06:40.150 --> 01:06:43.110 align:middle line:84% controls what the expected length of our optimal prefix 01:06:43.110 --> 01:06:44.910 align:middle line:90% free code is. 01:06:44.910 --> 01:06:47.350 align:middle line:84% I'm not going to prove this part. 01:06:47.350 --> 01:06:50.730 align:middle line:84% This part, in effect, we already proved through Shannon, 01:06:50.730 --> 01:06:53.350 align:middle line:84% but I'll argue about why we already proved that. 01:06:53.350 --> 01:06:57.750 align:middle line:84% Let's prove the other side, that the Huffman code is good. 01:06:57.750 --> 01:06:58.250 align:middle line:90% OK. 01:06:58.250 --> 01:07:02.250 align:middle line:90% 01:07:02.250 --> 01:07:06.270 align:middle line:90% So here's our idea. 01:07:06.270 --> 01:07:14.480 align:middle line:84% Let's let la be the length of the encoding for symbol a, 01:07:14.480 --> 01:07:18.480 align:middle line:84% and we're going to very cleverly choose it 01:07:18.480 --> 01:07:22.320 align:middle line:90% to be the ceiling of log of Pa. 01:07:22.320 --> 01:07:26.080 align:middle line:84% So what's going on here is that the depths of my leaves 01:07:26.080 --> 01:07:28.460 align:middle line:90% had better be natural numbers. 01:07:28.460 --> 01:07:31.000 align:middle line:90% They're non-negative integers. 01:07:31.000 --> 01:07:36.160 align:middle line:84% And Pa is the associated probability of the symbol a. 01:07:36.160 --> 01:07:38.980 align:middle line:84% So I take log of Pa, I'm going to get a negative number. 01:07:38.980 --> 01:07:41.920 align:middle line:84% I multiply it by minus 1, I'm going to get a positive number. 01:07:41.920 --> 01:07:45.280 align:middle line:84% And I'm just going to round up that number. 01:07:45.280 --> 01:07:48.720 align:middle line:84% And those are going to be my depths la. 01:07:48.720 --> 01:07:51.840 align:middle line:84% And so the first thing I have to argue about 01:07:51.840 --> 01:07:55.420 align:middle line:84% is that these are valid depths for the leaves, 01:07:55.420 --> 01:07:59.120 align:middle line:84% because that's where I want to put each of these symbols. 01:07:59.120 --> 01:08:02.040 align:middle line:84% So how do I argue that these are valid depths is I 01:08:02.040 --> 01:08:04.760 align:middle line:90% appeal to the Kraft inequality. 01:08:04.760 --> 01:08:06.640 align:middle line:84% So the Kraft inequality tells me I just 01:08:06.640 --> 01:08:09.640 align:middle line:84% have to check a particular inequality. 01:08:09.640 --> 01:08:17.490 align:middle line:84% So let's verify Kraft, that the conditions of Kraft inequality 01:08:17.490 --> 01:08:18.410 align:middle line:90% hold. 01:08:18.410 --> 01:08:23.130 align:middle line:84% So I want that the sum over all symbols in my alphabet of 2 01:08:23.130 --> 01:08:28.649 align:middle line:84% to the minus la, I want to show that this is at most 1. 01:08:28.649 --> 01:08:31.130 align:middle line:84% So let's write out what this thing is. 01:08:31.130 --> 01:08:34.250 align:middle line:84% Well, by definition, the way we've constructed la, 01:08:34.250 --> 01:08:38.130 align:middle line:84% this is still the sum over all symbols in our alphabet of 2 01:08:38.130 --> 01:08:43.930 align:middle line:84% to the minus ceiling of minus log Pa. 01:08:43.930 --> 01:08:46.649 align:middle line:84% And I claim that this is upper bounded 01:08:46.649 --> 01:08:50.890 align:middle line:84% by the sum over all the symbols in our alphabet of 2 01:08:50.890 --> 01:08:54.770 align:middle line:90% to the log Pa. 01:08:54.770 --> 01:08:58.569 align:middle line:84% So what's going on here is that remember inside I 01:08:58.569 --> 01:09:01.950 align:middle line:84% have a positive number, because it's minus log Pa. 01:09:01.950 --> 01:09:03.970 align:middle line:90% Log Pa is negative. 01:09:03.970 --> 01:09:07.729 align:middle line:84% So when I'm rounding it up and then I'm multiplying by minus 1 01:09:07.729 --> 01:09:11.020 align:middle line:84% I'm making this associated number smaller. 01:09:11.020 --> 01:09:12.859 align:middle line:84% So I can make the number only bigger 01:09:12.859 --> 01:09:15.340 align:middle line:84% by removing the ceiling function. 01:09:15.340 --> 01:09:16.920 align:middle line:84% When I remove the ceiling function, 01:09:16.920 --> 01:09:21.479 align:middle line:84% the two minuses cancel and I get log Pa, and now 2 to the log 01:09:21.479 --> 01:09:22.720 align:middle line:90% Pa is very simple. 01:09:22.720 --> 01:09:23.939 align:middle line:90% It's just Pa. 01:09:23.939 --> 01:09:26.100 align:middle line:84% So I get sum over all of the symbols 01:09:26.100 --> 01:09:29.779 align:middle line:84% in my alphabet of Pa, which is 1. 01:09:29.779 --> 01:09:33.140 align:middle line:84% So this verifies the Kraft inequality, 01:09:33.140 --> 01:09:38.020 align:middle line:84% because all of these lengths la that I've conveniently guessed, 01:09:38.020 --> 01:09:40.660 align:middle line:84% they really do correspond to a binary tree 01:09:40.660 --> 01:09:44.020 align:middle line:84% with those associated depths of the leaves. 01:09:44.020 --> 01:09:45.700 align:middle line:84% And now the last part I have to do 01:09:45.700 --> 01:09:48.540 align:middle line:84% is I just have to analyze how good 01:09:48.540 --> 01:09:53.060 align:middle line:90% this particular construction is. 01:09:53.060 --> 01:09:56.220 align:middle line:84% And then because the Huffman code is optimal, 01:09:56.220 --> 01:09:58.500 align:middle line:84% it'll only ever be better than whatever 01:09:58.500 --> 01:10:01.700 align:middle line:84% I get for this particular code that 01:10:01.700 --> 01:10:03.870 align:middle line:90% follows by appealing to Kraft. 01:10:03.870 --> 01:10:09.710 align:middle line:90% 01:10:09.710 --> 01:10:12.660 align:middle line:84% So now let's calculate the performance of the code. 01:10:12.660 --> 01:10:15.470 align:middle line:90% 01:10:15.470 --> 01:10:21.350 align:middle line:84% So we know, by lemma four, we know 01:10:21.350 --> 01:10:28.470 align:middle line:84% that there is a binary tree with these depths, 01:10:28.470 --> 01:10:29.720 align:middle line:90% the depths that I've guessed. 01:10:29.720 --> 01:10:32.310 align:middle line:90% 01:10:32.310 --> 01:10:34.890 align:middle line:84% And let's calculate its expected length. 01:10:34.890 --> 01:10:39.670 align:middle line:84% So I know that l, the expected length of my Huffman code, 01:10:39.670 --> 01:10:42.390 align:middle line:84% is only ever better than whatever expected lengths 01:10:42.390 --> 01:10:46.310 align:middle line:84% I'm going to get for this tree that I'm guessing from Kraft. 01:10:46.310 --> 01:10:49.190 align:middle line:84% And this right here is the sum over all symbols 01:10:49.190 --> 01:10:54.350 align:middle line:84% in my alphabet of Pa times la, the length of the associated 01:10:54.350 --> 01:10:55.430 align:middle line:90% symbol. 01:10:55.430 --> 01:10:57.570 align:middle line:90% Well, I can replace this again. 01:10:57.570 --> 01:11:01.670 align:middle line:84% This is the sum over all symbols in my alphabet of Pa 01:11:01.670 --> 01:11:06.710 align:middle line:84% times the ceiling of minus log Pa. 01:11:06.710 --> 01:11:08.880 align:middle line:84% And now I claim that this is upper bounded 01:11:08.880 --> 01:11:11.880 align:middle line:84% just by adding one instead of taking the ceiling, 01:11:11.880 --> 01:11:14.400 align:middle line:84% because at most I bump it up by 1. 01:11:14.400 --> 01:11:18.680 align:middle line:84% So this is the sum over all symbols in my alphabet of Pa 01:11:18.680 --> 01:11:23.320 align:middle line:90% times minus log Pa plus 1. 01:11:23.320 --> 01:11:26.160 align:middle line:84% But now I'm in good shape, because this term 01:11:26.160 --> 01:11:27.180 align:middle line:90% should look familiar. 01:11:27.180 --> 01:11:29.160 align:middle line:90% This is the binary entropy. 01:11:29.160 --> 01:11:32.080 align:middle line:84% And this term right here, when I'm summing up all the Pa's, I 01:11:32.080 --> 01:11:33.600 align:middle line:90% get 1. 01:11:33.600 --> 01:11:36.040 align:middle line:84% So in total the thing that I get is my upper bound 01:11:36.040 --> 01:11:38.880 align:middle line:90% is the binary entropy plus 1. 01:11:38.880 --> 01:11:41.880 align:middle line:90% So that's the proof. 01:11:41.880 --> 01:11:46.120 align:middle line:84% The Huffman code is very, very good because it's optimal 01:11:46.120 --> 01:11:50.920 align:middle line:84% and because there exists a good code by the Kraft inequality 01:11:50.920 --> 01:11:55.720 align:middle line:90% whose performance is H plus 1. 01:11:55.720 --> 01:11:57.600 align:middle line:84% Now, in the last couple minutes, I 01:11:57.600 --> 01:12:00.960 align:middle line:84% want to take this way further and connect it back to Shannon. 01:12:00.960 --> 01:12:02.980 align:middle line:90% So we proved a bunch of things. 01:12:02.980 --> 01:12:05.730 align:middle line:84% We proved that we found an algorithm, 01:12:05.730 --> 01:12:09.170 align:middle line:84% Huffman's coding algorithm for finding the optimal prefix 01:12:09.170 --> 01:12:10.250 align:middle line:90% free code. 01:12:10.250 --> 01:12:14.410 align:middle line:84% We analyzed its performance in terms of its expected length. 01:12:14.410 --> 01:12:16.570 align:middle line:84% But now what I want to argue about 01:12:16.570 --> 01:12:20.825 align:middle line:84% is how we can do much better than Shannon. 01:12:20.825 --> 01:12:24.450 align:middle line:90% So we talked about how Shannon-- 01:12:24.450 --> 01:12:27.610 align:middle line:90% I think I still have it here. 01:12:27.610 --> 01:12:29.810 align:middle line:84% It gets a bound when you apply it 01:12:29.810 --> 01:12:34.850 align:middle line:84% to length n sequences instead of a single length single symbol. 01:12:34.850 --> 01:12:38.610 align:middle line:84% It gets n times entropy plus some other term. 01:12:38.610 --> 01:12:41.890 align:middle line:84% So if I naively use my Huffman code, 01:12:41.890 --> 01:12:46.330 align:middle line:84% it'll sound like I'll get n times entropy plus n 01:12:46.330 --> 01:12:50.730 align:middle line:84% because I'm paying that plus 1 every time. 01:12:50.730 --> 01:12:53.450 align:middle line:84% But there's a trick where we can replace that. 01:12:53.450 --> 01:12:55.550 align:middle line:84% So that's the last thing I want to show you. 01:12:55.550 --> 01:13:03.090 align:middle line:90% 01:13:03.090 --> 01:13:05.230 align:middle line:84% And this is one of the really cool punch lines. 01:13:05.230 --> 01:13:07.780 align:middle line:90% 01:13:07.780 --> 01:13:10.380 align:middle line:84% So let's imagine, and then the length n case 01:13:10.380 --> 01:13:12.580 align:middle line:84% will follow immediately from this. 01:13:12.580 --> 01:13:16.420 align:middle line:84% What if we want to encode outputs of length 2? 01:13:16.420 --> 01:13:16.920 align:middle line:90% OK. 01:13:16.920 --> 01:13:26.988 align:middle line:90% 01:13:26.988 --> 01:13:31.630 align:middle line:84% So what if our source outputs two symbols instead? 01:13:31.630 --> 01:13:34.460 align:middle line:90% 01:13:34.460 --> 01:13:39.820 align:middle line:84% Well, we can still think about them as a single symbol, 01:13:39.820 --> 01:13:45.820 align:middle line:84% just on a larger alphabet, because originally we 01:13:45.820 --> 01:13:51.700 align:middle line:84% had six symbols, but now we'll have 36 in our alphabet. 01:13:51.700 --> 01:13:53.500 align:middle line:84% Each of these pairs of characters 01:13:53.500 --> 01:13:58.300 align:middle line:84% is a new symbol in my alphabet, aa, ab, ac. 01:13:58.300 --> 01:14:00.140 align:middle line:84% We still have a first order source which 01:14:00.140 --> 01:14:03.400 align:middle line:90% just has a larger value for k. 01:14:03.400 --> 01:14:06.746 align:middle line:84% What I can do is I can compute the entropy. 01:14:06.746 --> 01:14:14.400 align:middle line:84% Sp what's the entropy of this source? 01:14:14.400 --> 01:14:20.700 align:middle line:84% Well, now it's minus the sum over all i, the sum over all j, 01:14:20.700 --> 01:14:23.560 align:middle line:84% because I'm taking two symbols, i and j, 01:14:23.560 --> 01:14:25.400 align:middle line:84% the probability that our first symbol is i, 01:14:25.400 --> 01:14:28.400 align:middle line:84% the probability that our second symbol is j, 01:14:28.400 --> 01:14:33.240 align:middle line:90% times log of Pi, Pj. 01:14:33.240 --> 01:14:35.720 align:middle line:84% But now something amazing happens, 01:14:35.720 --> 01:14:38.680 align:middle line:84% which is that I can just algebraically manipulate 01:14:38.680 --> 01:14:40.240 align:middle line:90% this statement. 01:14:40.240 --> 01:14:48.280 align:middle line:84% I'll get minus the sum over i, the sum over j of Pi Pj 01:14:48.280 --> 01:14:56.080 align:middle line:90% log Pi plus Pi Pj log Pj. 01:14:56.080 --> 01:14:59.640 align:middle line:84% This is just using properties of the log. 01:14:59.640 --> 01:15:02.690 align:middle line:84% And now look at this term right here. 01:15:02.690 --> 01:15:06.810 align:middle line:84% I can sum up over j, and then I can replace the Pj with 1 01:15:06.810 --> 01:15:08.810 align:middle line:90% because the Pj's sum to 1. 01:15:08.810 --> 01:15:12.210 align:middle line:84% So I'll get the sum over i of Pi log Pi. 01:15:12.210 --> 01:15:14.770 align:middle line:84% And the same way here I can sum up over i 01:15:14.770 --> 01:15:17.290 align:middle line:84% because i doesn't appear inside the log. 01:15:17.290 --> 01:15:21.530 align:middle line:84% So I can replace this Pi with one and get rid of the sum. 01:15:21.530 --> 01:15:24.970 align:middle line:84% So this entire thing is then equal to minus 01:15:24.970 --> 01:15:35.010 align:middle line:84% the sum over i of Pi log Pi minus the sum over j Pj log Pj. 01:15:35.010 --> 01:15:37.682 align:middle line:84% All this is is 2 times the entropy. 01:15:37.682 --> 01:15:40.610 align:middle line:90% 01:15:40.610 --> 01:15:42.850 align:middle line:84% It's 2 times the entropy because each of these terms 01:15:42.850 --> 01:15:44.850 align:middle line:90% is computing the entropy. 01:15:44.850 --> 01:15:48.210 align:middle line:84% So this is one of the amazing facts about entropy 01:15:48.210 --> 01:15:52.490 align:middle line:84% is that when you look at the entropy of the output at length 01:15:52.490 --> 01:15:55.850 align:middle line:84% 2, it's just twice the entropy of the per symbol 01:15:55.850 --> 01:16:00.020 align:middle line:84% entropy that you started with H. And now something 01:16:00.020 --> 01:16:03.860 align:middle line:84% amazing happens, which is if I think about this theorem, 01:16:03.860 --> 01:16:07.580 align:middle line:84% and instead of encoding my output for my channel symbol 01:16:07.580 --> 01:16:12.580 align:middle line:84% by symbol, what if I encode it as a pair? 01:16:12.580 --> 01:16:17.100 align:middle line:84% So instead, when I look at the output as being two symbols, 01:16:17.100 --> 01:16:20.220 align:middle line:90% my new entropy is 2H. 01:16:20.220 --> 01:16:23.380 align:middle line:84% So I'll have that my expected length is at most 2H, 01:16:23.380 --> 01:16:26.540 align:middle line:90% but still plus 1. 01:16:26.540 --> 01:16:29.260 align:middle line:84% When I do it for an output that's size n, 01:16:29.260 --> 01:16:32.140 align:middle line:84% it'll be n times H, but still plus 1. 01:16:32.140 --> 01:16:33.860 align:middle line:84% That plus 1 is just the fixed thing 01:16:33.860 --> 01:16:37.740 align:middle line:84% that we add independently of how large our alphabet is. 01:16:37.740 --> 01:16:40.460 align:middle line:84% So this result that we had from Shannon, 01:16:40.460 --> 01:16:44.620 align:middle line:84% where there was a bound and the suboptimality that's 01:16:44.620 --> 01:16:48.260 align:middle line:84% some little o of n that's like n over log n or root n, 01:16:48.260 --> 01:16:52.300 align:middle line:84% it turns out that we can replace this with plus 1. 01:16:52.300 --> 01:16:53.980 align:middle line:84% So this is an amazingly sharp thing 01:16:53.980 --> 01:16:57.260 align:middle line:84% that follows immediately from our proof of Huffman codes. 01:16:57.260 --> 01:16:59.830 align:middle line:84% So we improved upon two things here. 01:16:59.830 --> 01:17:02.430 align:middle line:84% First of all, we have a more explicit coding function 01:17:02.430 --> 01:17:05.150 align:middle line:84% because it follows just from this greedy algorithm. 01:17:05.150 --> 01:17:06.790 align:middle line:84% And second of all, we drastically 01:17:06.790 --> 01:17:10.230 align:middle line:84% improve this additive loss in our original bound. 01:17:10.230 --> 01:17:14.870 align:middle line:84% So Huffman is really one of the crown jewels in coding theory, 01:17:14.870 --> 01:17:18.030 align:middle line:84% and it was proven, really, by a grad student, which 01:17:18.030 --> 01:17:21.510 align:middle line:84% is pretty amazing, who wanted to get out of taking a final exam. 01:17:21.510 --> 01:17:24.630 align:middle line:84% So with that, I'll leave you as an invitation. 01:17:24.630 --> 01:17:28.530 align:middle line:84% If you also want to get out of your exam three, 01:17:28.530 --> 01:17:30.030 align:middle line:84% just prove something else that we're 01:17:30.030 --> 01:17:32.010 align:middle line:84% going to teach in lecture next semester 01:17:32.010 --> 01:17:34.750 align:middle line:84% and I will happily give you an A for the class. 01:17:34.750 --> 01:17:38.190 align:middle line:84% If not, come to office hours and we'll help you study. 01:17:38.190 --> 01:17:39.390 align:middle line:90% All right. 01:17:39.390 --> 01:17:43.170 align:middle line:84% So we'll continue next time with Shannon's noisy coding theorem. 01:17:43.170 --> 01:17:45.670 align:middle line:84% That'll probably be the hardest lecture in the coding theory 01:17:45.670 --> 01:17:48.690 align:middle line:90% unit, but come prepared. 01:17:48.690 --> 01:17:50.570 align:middle line:84% Be awake and ready to answer questions. 01:17:50.570 --> 01:17:52.340 align:middle line:90% See you then. 01:17:52.340 --> 01:17:58.000 align:middle line:90%