WEBVTT 00:00:00.000 --> 00:00:01.984 align:middle line:90% [SQUEAKING] 00:00:01.984 --> 00:00:03.472 align:middle line:90% [RUSTLING] 00:00:03.472 --> 00:00:05.952 align:middle line:90% [CLICKING] 00:00:05.952 --> 00:00:11.920 align:middle line:90% 00:00:11.920 --> 00:00:14.120 align:middle line:84% ANKUR MOITRA: These are fairly challenging topics 00:00:14.120 --> 00:00:18.000 align:middle line:84% because we're really just going to be proving one theorem today. 00:00:18.000 --> 00:00:19.480 align:middle line:84% But this theorem, which is called 00:00:19.480 --> 00:00:21.540 align:middle line:84% Shannon's noiseless coding theorem, 00:00:21.540 --> 00:00:23.480 align:middle line:84% will involve a lot of the probabilistic tools 00:00:23.480 --> 00:00:26.200 align:middle line:84% that we've developed earlier in the class. 00:00:26.200 --> 00:00:28.720 align:middle line:84% So I'll give you an introduction to what data compression is 00:00:28.720 --> 00:00:31.360 align:middle line:84% and really how Shannon's work put this 00:00:31.360 --> 00:00:34.000 align:middle line:84% on a rigorous mathematical foundation. 00:00:34.000 --> 00:00:36.120 align:middle line:84% One of the cool historical facts about what 00:00:36.120 --> 00:00:37.920 align:middle line:84% I'm going to be teaching you about today 00:00:37.920 --> 00:00:40.560 align:middle line:84% is that it literally happened in this building 00:00:40.560 --> 00:00:41.877 align:middle line:90% pre the renovation. 00:00:41.877 --> 00:00:43.460 align:middle line:84% But the theorems I'm going to tell you 00:00:43.460 --> 00:00:46.440 align:middle line:84% were all laid out right here, not 00:00:46.440 --> 00:00:48.480 align:middle line:84% down the hall in a different building, 00:00:48.480 --> 00:00:50.180 align:middle line:84% literally right here in Building 2. 00:00:50.180 --> 00:00:51.800 align:middle line:90% So this will be cool. 00:00:51.800 --> 00:00:54.840 align:middle line:84% So let me start off by giving you some historical context 00:00:54.840 --> 00:00:56.060 align:middle line:90% for the lead-in for this. 00:00:56.060 --> 00:01:01.980 align:middle line:84% So how many people here use text messages? 00:01:01.980 --> 00:01:03.113 align:middle line:90% OK, should be everyone. 00:01:03.113 --> 00:01:04.280 align:middle line:90% You're all paying attention. 00:01:04.280 --> 00:01:05.740 align:middle line:90% That's good. 00:01:05.740 --> 00:01:07.840 align:middle line:84% Of course, when we're texting each other, 00:01:07.840 --> 00:01:11.332 align:middle line:90% we use short forms, like BRB. 00:01:11.332 --> 00:01:13.060 align:middle line:84% In fact, every once in a while, I'll 00:01:13.060 --> 00:01:18.300 align:middle line:84% see a crazy acronym that I don't, but maybe you guys know. 00:01:18.300 --> 00:01:21.460 align:middle line:84% And so you can think that the acronyms, especially as you 00:01:21.460 --> 00:01:24.620 align:middle line:84% get older and you stop being as plugged into it, at some point, 00:01:24.620 --> 00:01:27.900 align:middle line:84% you have no idea what people are talking about anymore. 00:01:27.900 --> 00:01:30.620 align:middle line:84% But this isn't even a new phenomenon. 00:01:30.620 --> 00:01:32.533 align:middle line:84% So let me give you some historical context. 00:01:32.533 --> 00:01:36.660 align:middle line:90% 00:01:36.660 --> 00:01:39.100 align:middle line:84% Back in the day, a long, long time 00:01:39.100 --> 00:01:42.780 align:middle line:84% ago, the way that people used to send messages 00:01:42.780 --> 00:01:45.300 align:middle line:90% was through telegrams. 00:01:45.300 --> 00:01:47.180 align:middle line:84% And the problem with telegrams was 00:01:47.180 --> 00:01:50.020 align:middle line:90% that they cost a lot of money. 00:01:50.020 --> 00:01:54.460 align:middle line:84% So we're talking about the cost $1 per word. 00:01:54.460 --> 00:01:56.760 align:middle line:84% So people wanted to send each other telegrams. 00:01:56.760 --> 00:01:59.920 align:middle line:84% But the same way that when we have text messages, 00:01:59.920 --> 00:02:02.320 align:middle line:84% we have common phrases and strings 00:02:02.320 --> 00:02:05.880 align:middle line:84% that we say over and over again, so we figure out the short form, 00:02:05.880 --> 00:02:09.639 align:middle line:84% well, here, they would create some of code book 00:02:09.639 --> 00:02:11.920 align:middle line:90% to compress communication. 00:02:11.920 --> 00:02:14.380 align:middle line:84% And some of these things were pretty obscure things. 00:02:14.380 --> 00:02:17.080 align:middle line:84% You can go back through and look at the telegram books from way 00:02:17.080 --> 00:02:18.600 align:middle line:90% back in the day. 00:02:18.600 --> 00:02:21.740 align:middle line:84% And you can find lots of gems in these code books, 00:02:21.740 --> 00:02:25.080 align:middle line:84% like if you send someone a telegram that 00:02:25.080 --> 00:02:28.600 align:middle line:84% had this word, which I can't pronounce, 00:02:28.600 --> 00:02:32.280 align:middle line:84% what it meant was it translated to the phrase 00:02:32.280 --> 00:02:38.200 align:middle line:90% has not been reinsured. 00:02:38.200 --> 00:02:41.480 align:middle line:84% That's a pretty technical concept. 00:02:41.480 --> 00:02:47.740 align:middle line:84% They also had another one, A-Z-K-H-E. This word, 00:02:47.740 --> 00:02:50.600 align:middle line:84% if you sent it through a telegram back in the day, 00:02:50.600 --> 00:02:56.740 align:middle line:84% this would get translated to clean bill of health. 00:02:56.740 --> 00:02:58.900 align:middle line:84% So all of these phrases are saving you 00:02:58.900 --> 00:03:02.620 align:middle line:84% money because, instead of sending this many words 00:03:02.620 --> 00:03:05.620 align:middle line:84% in this phrase, I'm getting away with sending just a single word 00:03:05.620 --> 00:03:07.660 align:middle line:90% for a single dollar. 00:03:07.660 --> 00:03:10.160 align:middle line:84% Some of these really go off the rails. 00:03:10.160 --> 00:03:12.780 align:middle line:84% So this is my favorite one when I look through these telegram 00:03:12.780 --> 00:03:14.820 align:middle line:90% books-- 00:03:14.820 --> 00:03:16.180 align:middle line:90% NBET. 00:03:16.180 --> 00:03:18.180 align:middle line:90% So what did this mean? 00:03:18.180 --> 00:03:22.660 align:middle line:90% This meant captain is insane. 00:03:22.660 --> 00:03:24.920 align:middle line:84% Hopefully, that one doesn't come up that often. 00:03:24.920 --> 00:03:28.540 align:middle line:84% But that really was one of the phrases in these code books 00:03:28.540 --> 00:03:30.340 align:middle line:90% from way back when. 00:03:30.340 --> 00:03:32.020 align:middle line:84% So what we're going to talk about today 00:03:32.020 --> 00:03:34.780 align:middle line:84% is really the mathematical foundation 00:03:34.780 --> 00:03:37.080 align:middle line:84% of what it means to compress data. 00:03:37.080 --> 00:03:38.580 align:middle line:84% And even better, we're going to talk 00:03:38.580 --> 00:03:42.695 align:middle line:84% about what it means to find the best compression for data. 00:03:42.695 --> 00:03:44.320 align:middle line:84% So I'm going to give you some examples. 00:03:44.320 --> 00:03:45.612 align:middle line:90% I'll give you some definitions. 00:03:45.612 --> 00:03:48.180 align:middle line:84% And we'll think intuitively about what a good compression 00:03:48.180 --> 00:03:49.432 align:middle line:90% ought to look like. 00:03:49.432 --> 00:03:52.140 align:middle line:84% And we're going to be working up to this beautiful theorem, which 00:03:52.140 --> 00:03:55.000 align:middle line:84% is called Shannon's noiseless coding theorem, that's 00:03:55.000 --> 00:03:57.160 align:middle line:84% going to tell us the best possible way 00:03:57.160 --> 00:03:59.840 align:middle line:90% to do data compression. 00:03:59.840 --> 00:04:04.920 align:middle line:84% So let's start off with the key definition 00:04:04.920 --> 00:04:06.600 align:middle line:90% at least that kicks it off. 00:04:06.600 --> 00:04:10.800 align:middle line:84% And as I mentioned, this happened all in this building. 00:04:10.800 --> 00:04:17.399 align:middle line:84% So in 1948, in this building, Claude Shannon 00:04:17.399 --> 00:04:20.270 align:middle line:84% founded the field of information theory. 00:04:20.270 --> 00:04:24.120 align:middle line:90% 00:04:24.120 --> 00:04:27.940 align:middle line:84% And this field, you can take whole graduate seminars on it. 00:04:27.940 --> 00:04:31.920 align:middle line:84% It's a beautiful, rich, thriving field. 00:04:31.920 --> 00:04:34.360 align:middle line:84% He's really one of the mathematical heroes 00:04:34.360 --> 00:04:37.905 align:middle line:84% of this entire area of modern research. 00:04:37.905 --> 00:04:39.280 align:middle line:84% And we're going to tell you about 00:04:39.280 --> 00:04:42.400 align:middle line:90% his first beautiful theorem. 00:04:42.400 --> 00:04:45.853 align:middle line:84% In the lecture on Tuesday, I'm going 00:04:45.853 --> 00:04:47.520 align:middle line:84% to tell you about an amazing result that 00:04:47.520 --> 00:04:51.420 align:middle line:84% was proved by a graduate student here on an exam. 00:04:51.420 --> 00:04:54.380 align:middle line:84% So you guys all took an exam on Thursday. 00:04:54.380 --> 00:04:56.540 align:middle line:84% And it turns out that the theorem 00:04:56.540 --> 00:04:58.820 align:middle line:84% I'm going to tell you about on Tuesday 00:04:58.820 --> 00:05:01.100 align:middle line:84% was proved by a graduate student who was given 00:05:01.100 --> 00:05:03.580 align:middle line:90% a question for his project. 00:05:03.580 --> 00:05:06.880 align:middle line:84% And he invented something that we still teach to this day. 00:05:06.880 --> 00:05:08.360 align:middle line:90% It's amazing and beautiful. 00:05:08.360 --> 00:05:09.860 align:middle line:84% And then in the third lecture, we're 00:05:09.860 --> 00:05:12.860 align:middle line:84% going to talk about Shannon's second main theorem. 00:05:12.860 --> 00:05:19.340 align:middle line:84% So first, let's define what we mean by our data source. 00:05:19.340 --> 00:05:23.700 align:middle line:84% This is the source of what we want to compress. 00:05:23.700 --> 00:05:27.500 align:middle line:84% And you can consider much more complicated sources. 00:05:27.500 --> 00:05:30.020 align:middle line:84% But let's keep it simple for today. 00:05:30.020 --> 00:05:32.980 align:middle line:84% So a first order source is described 00:05:32.980 --> 00:05:34.900 align:middle line:90% by the following things. 00:05:34.900 --> 00:05:38.540 align:middle line:84% So first of all, there's an alphabet, 00:05:38.540 --> 00:05:40.920 align:middle line:84% which is just the set of all possible symbols. 00:05:40.920 --> 00:05:42.660 align:middle line:84% And we'll denote that by capital A. 00:05:42.660 --> 00:05:48.680 align:middle line:84% And the individual symbols are a1, a2, all the way up to ak. 00:05:48.680 --> 00:05:51.280 align:middle line:90% 00:05:51.280 --> 00:05:53.200 align:middle line:84% And then the other component we have 00:05:53.200 --> 00:05:57.240 align:middle line:84% is that we have a distribution, which 00:05:57.240 --> 00:06:01.960 align:middle line:84% is going to be a distribution p on the alphabet. 00:06:01.960 --> 00:06:04.700 align:middle line:84% So it'll tell you the probability of the first symbol. 00:06:04.700 --> 00:06:07.240 align:middle line:84% It'll tell you the probability of the second symbol, all 00:06:07.240 --> 00:06:09.560 align:middle line:90% the way up to the k-th symbol. 00:06:09.560 --> 00:06:16.120 align:middle line:84% So all this first order source is it's a k-sided unfair die. 00:06:16.120 --> 00:06:17.480 align:middle line:90% You roll the die. 00:06:17.480 --> 00:06:21.160 align:middle line:84% And whatever symbol from your alphabet comes up on top, 00:06:21.160 --> 00:06:24.000 align:middle line:84% that's your next symbol in your message. 00:06:24.000 --> 00:06:26.160 align:middle line:84% So what happens is that the first order source 00:06:26.160 --> 00:06:28.560 align:middle line:84% is going to define a long string that we just 00:06:28.560 --> 00:06:31.160 align:middle line:90% take by taking IID samples. 00:06:31.160 --> 00:06:32.760 align:middle line:84% And so the way that this works is it's 00:06:32.760 --> 00:06:41.040 align:middle line:84% going to generate length n strings 00:06:41.040 --> 00:06:50.740 align:middle line:84% where each symbol is IID from this associated distribution. 00:06:50.740 --> 00:06:53.460 align:middle line:84% So the way that I'm constructing a long phrase 00:06:53.460 --> 00:06:56.460 align:middle line:84% is I take my k-sided die that's unfair. 00:06:56.460 --> 00:06:58.380 align:middle line:90% I roll it n times. 00:06:58.380 --> 00:07:02.540 align:middle line:84% And then I just concatenate all of those letters in my alphabet 00:07:02.540 --> 00:07:06.158 align:middle line:84% together, keeping track the order in which I rolled them. 00:07:06.158 --> 00:07:07.700 align:middle line:84% Now, you can think about the alphabet 00:07:07.700 --> 00:07:09.720 align:middle line:90% as being composed of letters. 00:07:09.720 --> 00:07:11.220 align:middle line:84% But really, you could think about it 00:07:11.220 --> 00:07:13.580 align:middle line:90% as being composed of words, too. 00:07:13.580 --> 00:07:18.380 align:middle line:84% Maybe the way that I create a sentence is by rolling my die 00:07:18.380 --> 00:07:20.400 align:middle line:84% and getting my first word, rolling my die, 00:07:20.400 --> 00:07:22.180 align:middle line:90% and getting my second word. 00:07:22.180 --> 00:07:25.660 align:middle line:84% This would be a terrible model for English language. 00:07:25.660 --> 00:07:27.320 align:middle line:84% And there are a lot richer models. 00:07:27.320 --> 00:07:29.280 align:middle line:84% You can do things like Markov chains, 00:07:29.280 --> 00:07:31.965 align:middle line:84% where a lot of this theory is going to carry over. 00:07:31.965 --> 00:07:33.340 align:middle line:84% But we're going to keep it simple 00:07:33.340 --> 00:07:36.340 align:middle line:84% and just talk about this as our data source. 00:07:36.340 --> 00:07:39.500 align:middle line:84% So I'm trying to send a message through a telegram to someone 00:07:39.500 --> 00:07:40.140 align:middle line:90% else. 00:07:40.140 --> 00:07:43.020 align:middle line:84% And the way I decide on what message I want to send 00:07:43.020 --> 00:07:45.940 align:middle line:84% is I use my first order source to generate it. 00:07:45.940 --> 00:07:48.000 align:middle line:84% And then I care about how short I 00:07:48.000 --> 00:07:50.920 align:middle line:84% can make my communication be to transmit 00:07:50.920 --> 00:07:53.160 align:middle line:90% that message from my source. 00:07:53.160 --> 00:07:54.980 align:middle line:84% So are there any questions about the setup? 00:07:54.980 --> 00:07:57.760 align:middle line:84% Do people understand what a first order source is 00:07:57.760 --> 00:08:00.040 align:middle line:84% and what the communication problem is? 00:08:00.040 --> 00:08:05.080 align:middle line:84% OK, so as I mentioned, there are more complex sources, 00:08:05.080 --> 00:08:07.280 align:middle line:90% like Markov chains. 00:08:07.280 --> 00:08:09.960 align:middle line:84% But now, let me define precisely what 00:08:09.960 --> 00:08:14.400 align:middle line:84% I mean by a codebook and a coding scheme. 00:08:14.400 --> 00:08:17.600 align:middle line:84% So what is a strategy for compressing communication? 00:08:17.600 --> 00:08:20.400 align:middle line:84% This will be our second key definition. 00:08:20.400 --> 00:08:26.680 align:middle line:84% So we'll say that a coding function, phi, 00:08:26.680 --> 00:08:31.960 align:middle line:84% it's a function that maps symbols of length 00:08:31.960 --> 00:08:34.880 align:middle line:84% n from my alphabet, which are just 00:08:34.880 --> 00:08:41.880 align:middle line:84% things generated from my first order source, to 0, 1 strings. 00:08:41.880 --> 00:08:43.980 align:middle line:84% Now, if you haven't seen this notation before, 00:08:43.980 --> 00:08:46.500 align:middle line:84% this notation star here means that I 00:08:46.500 --> 00:08:48.780 align:middle line:90% can take any length I want. 00:08:48.780 --> 00:08:51.660 align:middle line:84% So I'm mapping anything that my first order 00:08:51.660 --> 00:08:56.120 align:middle line:84% source generates to a string of a variable length. 00:08:56.120 --> 00:08:58.500 align:middle line:84% But it's composed of 0's and 1's. 00:08:58.500 --> 00:09:02.100 align:middle line:84% So maybe one of my messages gets mapped to the 00 string. 00:09:02.100 --> 00:09:06.420 align:middle line:84% Maybe one of my messages gets mapped to the 111 string. 00:09:06.420 --> 00:09:08.700 align:middle line:84% So in fact, the length of the output 00:09:08.700 --> 00:09:12.060 align:middle line:84% doesn't need to be fixed across the different messages. 00:09:12.060 --> 00:09:13.660 align:middle line:84% And that's exactly where we're going 00:09:13.660 --> 00:09:16.220 align:middle line:84% to win because somehow the intuition is 00:09:16.220 --> 00:09:19.100 align:middle line:84% that more frequent messages should 00:09:19.100 --> 00:09:21.900 align:middle line:90% be mapped to shorter strings. 00:09:21.900 --> 00:09:26.180 align:middle line:84% Now, first of all, I need not just that this coding function 00:09:26.180 --> 00:09:29.900 align:middle line:84% maps every message to an output string 00:09:29.900 --> 00:09:31.280 align:middle line:90% because that wouldn't help me. 00:09:31.280 --> 00:09:35.180 align:middle line:84% What if I mapped everything to the string 0 or everything 00:09:35.180 --> 00:09:37.100 align:middle line:90% to the string the empty set? 00:09:37.100 --> 00:09:39.260 align:middle line:84% That wouldn't help me decode and figure out 00:09:39.260 --> 00:09:41.260 align:middle line:90% what the original message is. 00:09:41.260 --> 00:09:48.360 align:middle line:84% So we need the property that we can successfully decode. 00:09:48.360 --> 00:09:50.320 align:middle line:84% In particular, what we want is that 00:09:50.320 --> 00:09:55.240 align:middle line:84% for any pair of different messages 00:09:55.240 --> 00:09:58.160 align:middle line:84% that are generated by our first order source, 00:09:58.160 --> 00:10:04.240 align:middle line:84% we want that their image under this mapping is not the same. 00:10:04.240 --> 00:10:07.200 align:middle line:84% So that's the key property because what this really means 00:10:07.200 --> 00:10:08.620 align:middle line:90% is just that we can decode. 00:10:08.620 --> 00:10:18.040 align:middle line:90% 00:10:18.040 --> 00:10:21.160 align:middle line:84% So this part is a little bit subtle. 00:10:21.160 --> 00:10:25.440 align:middle line:84% Later on, when we talk about Huffman codes on Tuesday next 00:10:25.440 --> 00:10:28.400 align:middle line:84% week, we're going to have some constraints which are called 00:10:28.400 --> 00:10:32.440 align:middle line:84% prefix-free constraints that we're really going to talk about 00:10:32.440 --> 00:10:37.520 align:middle line:84% a coding function that maps symbol to symbol-- 00:10:37.520 --> 00:10:39.160 align:middle line:90% symbol to sequence. 00:10:39.160 --> 00:10:42.820 align:middle line:84% But here, the way we're thinking about it is n is gigantic 00:10:42.820 --> 00:10:44.420 align:middle line:90% and it's fixed. 00:10:44.420 --> 00:10:47.500 align:middle line:84% And then what happens is I decide on my coding function 00:10:47.500 --> 00:10:48.580 align:middle line:90% phi. 00:10:48.580 --> 00:10:52.180 align:middle line:84% And I'm just going to one-shot transmit some variable length 00:10:52.180 --> 00:10:53.080 align:middle line:90% message. 00:10:53.080 --> 00:10:57.100 align:middle line:84% And at the end of the message, we say END, termination over. 00:10:57.100 --> 00:11:01.260 align:middle line:84% And then the key point is just that things which are different 00:11:01.260 --> 00:11:03.120 align:middle line:90% get mapped to different outputs. 00:11:03.120 --> 00:11:05.740 align:middle line:84% And we see where the actual thing ends. 00:11:05.740 --> 00:11:09.380 align:middle line:84% And then we want that we can recover x and y from this. 00:11:09.380 --> 00:11:12.260 align:middle line:84% So are there any questions about what a first order source is 00:11:12.260 --> 00:11:14.940 align:middle line:90% or what a coding function is? 00:11:14.940 --> 00:11:17.540 align:middle line:90% Does this make sense? 00:11:17.540 --> 00:11:18.100 align:middle line:90% Yeah? 00:11:18.100 --> 00:11:20.460 align:middle line:90% OK, all right. 00:11:20.460 --> 00:11:22.520 align:middle line:90% So let's do an example. 00:11:22.520 --> 00:11:25.060 align:middle line:84% And really, our main question is going 00:11:25.060 --> 00:11:27.340 align:middle line:84% to be this question we're going to address 00:11:27.340 --> 00:11:29.620 align:middle line:84% at the end with Shannon's theorem. 00:11:29.620 --> 00:11:34.010 align:middle line:84% So what is the best coding function? 00:11:34.010 --> 00:11:38.570 align:middle line:90% 00:11:38.570 --> 00:11:41.890 align:middle line:84% And I even have to define what I mean by best here. 00:11:41.890 --> 00:11:46.330 align:middle line:84% But let's do an example to make all of this explicit. 00:11:46.330 --> 00:11:50.410 align:middle line:84% Let's say that our first order source has two letters. 00:11:50.410 --> 00:11:53.170 align:middle line:90% So our alphabet is of size 2. 00:11:53.170 --> 00:11:56.130 align:middle line:84% And these two letters are a1 and a2. 00:11:56.130 --> 00:12:03.010 align:middle line:84% And we're going to have p1 is equal to 7/8 and p2 is equal 00:12:03.010 --> 00:12:04.850 align:middle line:90% to 1/8. 00:12:04.850 --> 00:12:07.850 align:middle line:84% So already here, there's a huge asymmetry 00:12:07.850 --> 00:12:10.090 align:middle line:84% because the first letter in my alphabet 00:12:10.090 --> 00:12:14.450 align:middle line:84% is way more likely than my second letter. 00:12:14.450 --> 00:12:17.390 align:middle line:90% And I'm going to set n equals 2. 00:12:17.390 --> 00:12:20.010 align:middle line:84% So I'm looking at length 2 messages 00:12:20.010 --> 00:12:22.090 align:middle line:90% from my first order source. 00:12:22.090 --> 00:12:25.170 align:middle line:84% And what are all of the messages that I could 00:12:25.170 --> 00:12:27.650 align:middle line:90% want to transmit to someone? 00:12:27.650 --> 00:12:38.190 align:middle line:84% I could want to transmit a1 a1, a1 a2, a2 a1, and a2 a2. 00:12:38.190 --> 00:12:41.710 align:middle line:84% Each of these messages has a different probability, 00:12:41.710 --> 00:12:45.130 align:middle line:84% like a1 a1 is much more likely than the others. 00:12:45.130 --> 00:12:48.310 align:middle line:90% It has probability 49 over 64. 00:12:48.310 --> 00:12:55.430 align:middle line:84% This is 7 over 64, 7 over 64, and finally 1 over 64. 00:12:55.430 --> 00:13:00.390 align:middle line:84% So the chance that I get a2 a2 is very unlikely. 00:13:00.390 --> 00:13:04.110 align:middle line:84% And now, what we can do is we can think about candidate 00:13:04.110 --> 00:13:07.070 align:middle line:90% codebooks and coding functions. 00:13:07.070 --> 00:13:09.190 align:middle line:84% So let's do the naive thing first 00:13:09.190 --> 00:13:11.550 align:middle line:84% and see how good the naive thing is. 00:13:11.550 --> 00:13:16.870 align:middle line:84% So the naive thing is just to map the symbol a1 to 0 and a2 00:13:16.870 --> 00:13:18.030 align:middle line:90% to 1. 00:13:18.030 --> 00:13:20.470 align:middle line:84% So in this case, when I get my code, 00:13:20.470 --> 00:13:24.750 align:middle line:84% it's just going to output 00 because I literally do it just 00:13:24.750 --> 00:13:26.350 align:middle line:90% symbol by symbol. 00:13:26.350 --> 00:13:30.270 align:middle line:84% a1 gets replaced with 0. a2 gets replaced with 1. 00:13:30.270 --> 00:13:35.370 align:middle line:84% My code would be 01 here, 10 here, and 11. 00:13:35.370 --> 00:13:39.930 align:middle line:84% And now, I should define what I mean by the best coding scheme. 00:13:39.930 --> 00:13:42.250 align:middle line:84% What we care about is how much money 00:13:42.250 --> 00:13:45.650 align:middle line:84% I pay on average with this coding scheme. 00:13:45.650 --> 00:13:50.170 align:middle line:84% And the cost of my transmission, the same way 00:13:50.170 --> 00:13:52.730 align:middle line:84% it is for things like telegrams, it 00:13:52.730 --> 00:13:57.210 align:middle line:84% will be some amount per symbol in this case, not per word. 00:13:57.210 --> 00:14:03.558 align:middle line:84% So we can compute the expected length 00:14:03.558 --> 00:14:05.830 align:middle line:90% of this particular encoding. 00:14:05.830 --> 00:14:09.130 align:middle line:84% Notice that the expected length is a property of the first order 00:14:09.130 --> 00:14:12.570 align:middle line:84% source and the choice of the coding function. 00:14:12.570 --> 00:14:15.070 align:middle line:90% So what is the expected length? 00:14:15.070 --> 00:14:17.230 align:middle line:90% Well, we can do this. 00:14:17.230 --> 00:14:19.410 align:middle line:90% This is a very easy computation. 00:14:19.410 --> 00:14:22.210 align:middle line:84% What's the probability I output the first symbol, and then 00:14:22.210 --> 00:14:24.850 align:middle line:90% I output a length 2 message? 00:14:24.850 --> 00:14:27.850 align:middle line:84% What's the probability I have the second message? 00:14:27.850 --> 00:14:30.610 align:middle line:84% And again, I output a length 2 message. 00:14:30.610 --> 00:14:33.510 align:middle line:84% What's the probability I output the third message 00:14:33.510 --> 00:14:37.350 align:middle line:84% from my first order source and so on. 00:14:37.350 --> 00:14:39.430 align:middle line:84% So this is a very easy computation 00:14:39.430 --> 00:14:41.710 align:middle line:90% to do because it's just 2. 00:14:41.710 --> 00:14:44.730 align:middle line:84% No matter what my first order source outputs, 00:14:44.730 --> 00:14:46.340 align:middle line:90% I always pay two symbols. 00:14:46.340 --> 00:14:50.250 align:middle line:90% 00:14:50.250 --> 00:14:54.190 align:middle line:90% So I claim this is not optimal. 00:14:54.190 --> 00:14:55.580 align:middle line:90% So can we do better? 00:14:55.580 --> 00:15:03.270 align:middle line:90% 00:15:03.270 --> 00:15:04.870 align:middle line:84% Anyone have any intuition for how 00:15:04.870 --> 00:15:09.190 align:middle line:84% I could design a better codebook for this particular sequence? 00:15:09.190 --> 00:15:11.270 align:middle line:84% This one's not too bad because there's really 00:15:11.270 --> 00:15:16.630 align:middle line:84% one dominant event, which is the event that my first order 00:15:16.630 --> 00:15:19.710 align:middle line:90% source outputs a1 a1. 00:15:19.710 --> 00:15:22.870 align:middle line:84% So originally, when that happened, 00:15:22.870 --> 00:15:26.230 align:middle line:90% I outputted a length 2 code. 00:15:26.230 --> 00:15:28.310 align:middle line:90% I want to do better. 00:15:28.310 --> 00:15:31.370 align:middle line:90% So any ideas? 00:15:31.370 --> 00:15:32.370 align:middle line:90% Yeah? 00:15:32.370 --> 00:15:33.390 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:15:33.390 --> 00:15:35.990 align:middle line:84% ANKUR MOITRA: Yeah, like 0, for example. 00:15:35.990 --> 00:15:39.430 align:middle line:84% So in this case, for example, my new codebook-- 00:15:39.430 --> 00:15:42.250 align:middle line:90% 00:15:42.250 --> 00:15:52.130 align:middle line:84% well, I'll have my message a1 a1, a1 a2, a2 a1, a2 a2. 00:15:52.130 --> 00:15:56.570 align:middle line:90% In this case, I could output 0. 00:15:56.570 --> 00:16:00.410 align:middle line:84% But now, what I need is I need to-- 00:16:00.410 --> 00:16:02.950 align:middle line:84% I could do-- let's just do something else for this. 00:16:02.950 --> 00:16:08.230 align:middle line:84% I could do 10, 110, and then 111. 00:16:08.230 --> 00:16:10.410 align:middle line:84% I'm not trying to get optimality yet. 00:16:10.410 --> 00:16:12.930 align:middle line:84% I'm just trying to get an improvement. 00:16:12.930 --> 00:16:15.900 align:middle line:84% And we can calculate the new expected length. 00:16:15.900 --> 00:16:23.890 align:middle line:90% 00:16:23.890 --> 00:16:26.690 align:middle line:84% And now, with this very large probability 00:16:26.690 --> 00:16:30.670 align:middle line:84% that I output the first message, well, I'm 00:16:30.670 --> 00:16:33.350 align:middle line:90% only paying 1 with this. 00:16:33.350 --> 00:16:36.130 align:middle line:90% I can add up 7 over 64. 00:16:36.130 --> 00:16:37.990 align:middle line:90% I'm still paying 2. 00:16:37.990 --> 00:16:39.890 align:middle line:84% And then on these other messages, 00:16:39.890 --> 00:16:46.070 align:middle line:84% I'm losing because now this third message, a2a1, instead 00:16:46.070 --> 00:16:49.190 align:middle line:84% of its length being 2, its length is 3. 00:16:49.190 --> 00:16:51.510 align:middle line:84% And on this last message, I'm losing again 00:16:51.510 --> 00:16:54.590 align:middle line:84% because it used to be 2 and now it's 3. 00:16:54.590 --> 00:16:57.070 align:middle line:84% But if you work it out, what this works out to be 00:16:57.070 --> 00:17:03.150 align:middle line:84% is 87 over 64, which indeed is less than 2. 00:17:03.150 --> 00:17:05.829 align:middle line:84% So this was an example where the naive encoding we 00:17:05.829 --> 00:17:08.030 align:middle line:90% could improve upon it. 00:17:08.030 --> 00:17:09.950 align:middle line:84% And we showed that we could do better. 00:17:09.950 --> 00:17:15.349 align:middle line:84% But really, the question now is, what's the best that we can do? 00:17:15.349 --> 00:17:18.750 align:middle line:84% So it turns out that there's a very clean and elegant answer 00:17:18.750 --> 00:17:21.550 align:middle line:90% to what's the best we can do. 00:17:21.550 --> 00:17:25.190 align:middle line:84% And it's related to really the first key notion 00:17:25.190 --> 00:17:29.010 align:middle line:84% from information theory, which is called the entropy. 00:17:29.010 --> 00:17:30.870 align:middle line:84% So let me just define what this is. 00:17:30.870 --> 00:17:33.970 align:middle line:84% And then we'll talk about Shannon's theorem 00:17:33.970 --> 00:17:35.530 align:middle line:90% that we're going to prove. 00:17:35.530 --> 00:17:42.730 align:middle line:84% So the binary entropy of a distribution-- 00:17:42.730 --> 00:17:47.690 align:middle line:84% in our case p, which is given by probabilities p1, p2, up to pk-- 00:17:47.690 --> 00:17:52.970 align:middle line:84% well, we write it as h of p1 up to pk. 00:17:52.970 --> 00:17:54.830 align:middle line:84% And it's the following expression. 00:17:54.830 --> 00:18:03.560 align:middle line:84% It's the negative sum from i equals 1 to k of pi log pi. 00:18:03.560 --> 00:18:06.290 align:middle line:90% 00:18:06.290 --> 00:18:09.290 align:middle line:84% So in particular, you have to be careful with the negative signs 00:18:09.290 --> 00:18:10.010 align:middle line:90% here. 00:18:10.010 --> 00:18:12.170 align:middle line:90% So this log is base 2. 00:18:12.170 --> 00:18:13.870 align:middle line:90% This pi is less than 1. 00:18:13.870 --> 00:18:16.410 align:middle line:84% So this term inside here is negative. 00:18:16.410 --> 00:18:19.050 align:middle line:84% But we're fixing it with the negative outside. 00:18:19.050 --> 00:18:23.770 align:middle line:84% So the entropy is something that's non-negative. 00:18:23.770 --> 00:18:26.990 align:middle line:84% And what I claim is that this simple definition right here 00:18:26.990 --> 00:18:30.830 align:middle line:84% is the key to understanding what is possible in terms of data 00:18:30.830 --> 00:18:33.630 align:middle line:84% compression and what the optimal scheme is. 00:18:33.630 --> 00:18:38.070 align:middle line:84% And we're going to unravel that as we prove Shannon's theorem. 00:18:38.070 --> 00:18:40.400 align:middle line:84% So let me now tell you Shannon's theorem. 00:18:40.400 --> 00:18:48.110 align:middle line:90% 00:18:48.110 --> 00:18:50.590 align:middle line:90% It comes in two parts. 00:18:50.590 --> 00:18:59.510 align:middle line:84% So we fix a first order source exactly as we've been doing. 00:18:59.510 --> 00:19:03.310 align:middle line:84% And let's say that this first order source has entropy 00:19:03.310 --> 00:19:09.030 align:middle line:84% H. We just plug in whatever our probabilities are 00:19:09.030 --> 00:19:12.710 align:middle line:84% that define the first order source into this binary entropy 00:19:12.710 --> 00:19:14.830 align:middle line:84% function that I defined right here. 00:19:14.830 --> 00:19:16.390 align:middle line:84% And let's say that the result we get 00:19:16.390 --> 00:19:21.070 align:middle line:84% is this numerical value H that's some constant. 00:19:21.070 --> 00:19:26.330 align:middle line:84% And what I claim is that for any coding function, 00:19:26.330 --> 00:19:32.290 align:middle line:84% no matter how clever you try to be, any valid coding function 00:19:32.290 --> 00:19:38.170 align:middle line:84% phi, we have the property that the expected length-- 00:19:38.170 --> 00:19:46.850 align:middle line:84% and what I mean by this is the length of phi of x1 up to xn, 00:19:46.850 --> 00:19:51.170 align:middle line:84% where this x1 up to xn comes from our first order source. 00:19:51.170 --> 00:19:55.730 align:middle line:84% I claim that this is at least the entropy H times 00:19:55.730 --> 00:19:58.890 align:middle line:90% n minus little o of n. 00:19:58.890 --> 00:20:01.050 align:middle line:84% So just to make sure-- you guys may not 00:20:01.050 --> 00:20:03.850 align:middle line:84% have all seen asymptotic notation before. 00:20:03.850 --> 00:20:06.370 align:middle line:84% What this expression means right here, little o of n 00:20:06.370 --> 00:20:11.290 align:middle line:84% just means that it's a function that grows smaller than n. 00:20:11.290 --> 00:20:15.130 align:middle line:84% So you should think about n over log n or root n. 00:20:15.130 --> 00:20:18.610 align:middle line:84% So the main point is that as n becomes large, 00:20:18.610 --> 00:20:20.930 align:middle line:90% this is the term that dominates. 00:20:20.930 --> 00:20:26.150 align:middle line:84% And H is the answer to what the per symbol length is that we 00:20:26.150 --> 00:20:28.630 align:middle line:90% pay in our encoding function. 00:20:28.630 --> 00:20:31.210 align:middle line:84% So this bound only kicks in when n is large, 00:20:31.210 --> 00:20:33.830 align:middle line:84% because when n is small, this term might actually 00:20:33.830 --> 00:20:37.150 align:middle line:84% be more significant than this right here. 00:20:37.150 --> 00:20:39.570 align:middle line:84% But what we're saying is that, at least in the limit, 00:20:39.570 --> 00:20:42.560 align:middle line:84% we can understand what is the best possible coding scheme. 00:20:42.560 --> 00:20:45.740 align:middle line:90% 00:20:45.740 --> 00:20:50.750 align:middle line:84% So this is one part of Shannon's noiseless coding theorem. 00:20:50.750 --> 00:20:52.230 align:middle line:84% Let me state the other part, which 00:20:52.230 --> 00:20:55.270 align:middle line:84% is just the natural converse, that you can actually 00:20:55.270 --> 00:20:56.390 align:middle line:90% achieve it. 00:20:56.390 --> 00:21:00.670 align:middle line:84% So moreover, this really is the answer. 00:21:00.670 --> 00:21:08.430 align:middle line:84% So there is a coding function phi with the property 00:21:08.430 --> 00:21:12.910 align:middle line:84% that its expected length l is defined the same way 00:21:12.910 --> 00:21:18.190 align:middle line:84% as at most the entropy times n plus little o of n. 00:21:18.190 --> 00:21:21.690 align:middle line:84% So up to this plus minus little o of n, 00:21:21.690 --> 00:21:25.570 align:middle line:84% we've resolved what the right answer is for the best possible 00:21:25.570 --> 00:21:27.010 align:middle line:90% coding scheme. 00:21:27.010 --> 00:21:29.770 align:middle line:84% And amazingly, this kind of mysterious function 00:21:29.770 --> 00:21:32.010 align:middle line:90% called the entropy pops out. 00:21:32.010 --> 00:21:35.170 align:middle line:84% And that's really the thing that governs what the best you can do 00:21:35.170 --> 00:21:36.250 align:middle line:90% is. 00:21:36.250 --> 00:21:40.690 align:middle line:84% So are there any questions about the definition so far? 00:21:40.690 --> 00:21:42.230 align:middle line:84% We've talked about coding functions. 00:21:42.230 --> 00:21:44.610 align:middle line:84% We've talked about the binary entropy function. 00:21:44.610 --> 00:21:46.050 align:middle line:84% And now, we have this key theorem 00:21:46.050 --> 00:21:49.370 align:middle line:90% that we're going to prove today. 00:21:49.370 --> 00:21:51.450 align:middle line:90% Does this make sense? 00:21:51.450 --> 00:21:54.410 align:middle line:90% Yeah, OK, good. 00:21:54.410 --> 00:21:56.290 align:middle line:90% All right. 00:21:56.290 --> 00:21:58.790 align:middle line:84% And just to be clear, I said this in words. 00:21:58.790 --> 00:22:01.610 align:middle line:84% But what Shannon's theorem is telling 00:22:01.610 --> 00:22:12.810 align:middle line:84% us is that H is the best per symbol from our alphabet length 00:22:12.810 --> 00:22:16.290 align:middle line:84% that you can do for a coding scheme. 00:22:16.290 --> 00:22:19.910 align:middle line:84% So before we prove this theorem, let's work up to it 00:22:19.910 --> 00:22:21.440 align:middle line:90% with some intuition first. 00:22:21.440 --> 00:22:27.990 align:middle line:90% 00:22:27.990 --> 00:22:38.390 align:middle line:84% So let me ask you to test your intuition now 00:22:38.390 --> 00:22:40.530 align:middle line:84% that we know what target we're shooting for. 00:22:40.530 --> 00:22:47.590 align:middle line:90% 00:22:47.590 --> 00:22:57.330 align:middle line:84% So what distribution p on k symbols-- 00:22:57.330 --> 00:23:01.270 align:middle line:84% we're still fixing our alphabet size-- 00:23:01.270 --> 00:23:03.040 align:middle line:90% requires the longest encoding? 00:23:03.040 --> 00:23:14.322 align:middle line:90% 00:23:14.322 --> 00:23:16.370 align:middle line:84% This is just for intuition's sake 00:23:16.370 --> 00:23:20.330 align:middle line:84% because we already know what the answer is for the best 00:23:20.330 --> 00:23:21.950 align:middle line:90% possible per symbol length. 00:23:21.950 --> 00:23:23.970 align:middle line:84% We know it's the entropy function. 00:23:23.970 --> 00:23:26.770 align:middle line:84% So you can think about this question mathematically. 00:23:26.770 --> 00:23:31.090 align:middle line:84% If I have any distribution p on k things, 00:23:31.090 --> 00:23:33.050 align:middle line:84% what distribution on k things is going 00:23:33.050 --> 00:23:35.890 align:middle line:84% to maximize that binary entropy function? 00:23:35.890 --> 00:23:37.890 align:middle line:84% That's one way to think about mathematically 00:23:37.890 --> 00:23:39.570 align:middle line:90% what I'm asking here. 00:23:39.570 --> 00:23:41.490 align:middle line:84% But maybe the better way to think about it is 00:23:41.490 --> 00:23:43.970 align:middle line:90% to go back to your intuition. 00:23:43.970 --> 00:23:46.010 align:middle line:84% So if I just started off with this, 00:23:46.010 --> 00:23:48.250 align:middle line:84% and I told you that there were first order sources, 00:23:48.250 --> 00:23:51.890 align:middle line:84% and that we were looking at coding functions, intuitively, 00:23:51.890 --> 00:23:53.610 align:middle line:84% what kind of first order source would 00:23:53.610 --> 00:23:57.850 align:middle line:90% be the hardest to compress? 00:23:57.850 --> 00:23:59.590 align:middle line:84% If you think back to this example 00:23:59.590 --> 00:24:03.450 align:middle line:84% we had right here, how were we able to win over 00:24:03.450 --> 00:24:06.490 align:middle line:84% the naive coding scheme was we exploited 00:24:06.490 --> 00:24:09.450 align:middle line:84% the fact that the probabilities were very 00:24:09.450 --> 00:24:15.110 align:middle line:84% different because there was this message a1a1 which was so much 00:24:15.110 --> 00:24:16.550 align:middle line:90% more likely. 00:24:16.550 --> 00:24:19.270 align:middle line:84% So we could encode that very frequent outcome 00:24:19.270 --> 00:24:21.070 align:middle line:90% with a very short string. 00:24:21.070 --> 00:24:23.308 align:middle line:84% But maybe that's not always possible. 00:24:23.308 --> 00:24:25.350 align:middle line:84% So you can think about this thing mathematically, 00:24:25.350 --> 00:24:27.430 align:middle line:84% in terms of the binary entropy function. 00:24:27.430 --> 00:24:29.910 align:middle line:84% Or you can think about it from first principles, 00:24:29.910 --> 00:24:33.790 align:middle line:84% just from your guess instead of doing the calculus. 00:24:33.790 --> 00:24:35.030 align:middle line:90% Yeah? 00:24:35.030 --> 00:24:36.670 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:24:36.670 --> 00:24:37.850 align:middle line:90% 00:24:37.850 --> 00:24:39.350 align:middle line:84% ANKUR MOITRA: That's exactly right. 00:24:39.350 --> 00:24:41.470 align:middle line:90% That is completely correct. 00:24:41.470 --> 00:24:43.990 align:middle line:84% So in fact, the answer to this question 00:24:43.990 --> 00:24:46.310 align:middle line:84% is what you'd expect, what you could 00:24:46.310 --> 00:24:48.590 align:middle line:84% have guessed even before class started, 00:24:48.590 --> 00:24:50.170 align:middle line:90% the uniform distribution. 00:24:50.170 --> 00:24:53.830 align:middle line:90% 00:24:53.830 --> 00:24:59.750 align:middle line:84% So the uniform distribution, p1 equals p2 all the way up to pk 00:24:59.750 --> 00:25:01.990 align:middle line:90% is all equal to 1 over k. 00:25:01.990 --> 00:25:05.350 align:middle line:84% And in this case, what does the entropy look like? 00:25:05.350 --> 00:25:09.510 align:middle line:84% Well, it's minus the sum of i equals 1 to k 00:25:09.510 --> 00:25:19.450 align:middle line:84% of pi log pi, which is the same thing as log k. 00:25:19.450 --> 00:25:23.730 align:middle line:84% And what this corresponds to is this corresponds to the naive 00:25:23.730 --> 00:25:28.490 align:middle line:84% encoding, at least if k is a power of 2, 00:25:28.490 --> 00:25:31.930 align:middle line:84% because one of the things I can do is I 00:25:31.930 --> 00:25:37.490 align:middle line:90% can just take my first symbol. 00:25:37.490 --> 00:25:39.250 align:middle line:90% Let's say it's ai. 00:25:39.250 --> 00:25:45.650 align:middle line:84% And I can encode this using log k bits because you can tell me 00:25:45.650 --> 00:25:48.250 align:middle line:90% which of the k symbols it is. 00:25:48.250 --> 00:25:53.050 align:middle line:84% And I can take the second symbol and encode it using log k bits. 00:25:53.050 --> 00:25:56.250 align:middle line:84% So in fact, this is an example, the uniform distribution, 00:25:56.250 --> 00:25:59.450 align:middle line:84% where the naive encoding cannot be beaten. 00:25:59.450 --> 00:26:01.450 align:middle line:84% One of the ways you can prove that this really 00:26:01.450 --> 00:26:04.530 align:middle line:84% is the answer, at least once we have Shannon's theorem, 00:26:04.530 --> 00:26:06.610 align:middle line:84% is you can solve the calculus problem-- 00:26:06.610 --> 00:26:09.450 align:middle line:84% which pi, which is a distribution 00:26:09.450 --> 00:26:12.670 align:middle line:84% and has non-negative entries that sum to 1, 00:26:12.670 --> 00:26:15.170 align:middle line:84% has the property that it maximizes the entropy function? 00:26:15.170 --> 00:26:17.330 align:middle line:84% You'll find out that this is the answer. 00:26:17.330 --> 00:26:20.430 align:middle line:84% But it really corresponds to this intuitive fact 00:26:20.430 --> 00:26:22.230 align:middle line:84% that, when there's no discrepancy 00:26:22.230 --> 00:26:23.830 align:middle line:84% in the probabilities, you can't have 00:26:23.830 --> 00:26:28.190 align:middle line:84% any win in terms of making more common things shorter code 00:26:28.190 --> 00:26:30.070 align:middle line:90% words. 00:26:30.070 --> 00:26:32.270 align:middle line:90% OK, good. 00:26:32.270 --> 00:26:35.750 align:middle line:90% So we have our intuition. 00:26:35.750 --> 00:26:39.280 align:middle line:84% Let's work towards proving the first part of Shannon's theorem. 00:26:39.280 --> 00:26:41.830 align:middle line:90% 00:26:41.830 --> 00:26:45.230 align:middle line:84% So let me state one of the key lemmas, which 00:26:45.230 --> 00:26:49.690 align:middle line:84% really corresponds to this thing that you have intuition for now. 00:26:49.690 --> 00:27:08.630 align:middle line:90% 00:27:08.630 --> 00:27:18.170 align:middle line:84% So what I claim is that any coding function 00:27:18.170 --> 00:27:30.490 align:middle line:84% for the uniform distribution on k symbols, 00:27:30.490 --> 00:27:36.450 align:middle line:84% it has the expected length being at least log 00:27:36.450 --> 00:27:39.230 align:middle line:90% k minus some constant. 00:27:39.230 --> 00:27:41.110 align:middle line:90% So o of 1 is some constant. 00:27:41.110 --> 00:27:44.290 align:middle line:84% You'll see in the proof why this constant shows up. 00:27:44.290 --> 00:27:48.090 align:middle line:84% It really comes from the fact that, for my coding function, 00:27:48.090 --> 00:27:51.010 align:middle line:84% I still can encode some of my symbols 00:27:51.010 --> 00:27:52.970 align:middle line:90% using very short strings. 00:27:52.970 --> 00:27:56.210 align:middle line:84% And then I can fill out things with longer strings, too. 00:27:56.210 --> 00:27:59.230 align:middle line:84% So there is some slight improvement you can get. 00:27:59.230 --> 00:28:00.730 align:middle line:84% But for reasons that we'll get into, 00:28:00.730 --> 00:28:03.010 align:middle line:90% that'll come out in the wash. 00:28:03.010 --> 00:28:08.150 align:middle line:84% So the sketch for this is the following. 00:28:08.150 --> 00:28:11.310 align:middle line:84% All of these symbols are interchangeable. 00:28:11.310 --> 00:28:14.510 align:middle line:90% 00:28:14.510 --> 00:28:24.510 align:middle line:84% We can think of phi as mapping symbols, each 00:28:24.510 --> 00:28:34.380 align:middle line:84% of these k symbols, to nodes in a binary tree. 00:28:34.380 --> 00:28:38.910 align:middle line:90% 00:28:38.910 --> 00:28:41.790 align:middle line:84% So in particular, I can start with this binary tree, 00:28:41.790 --> 00:28:45.110 align:middle line:84% where the root is the empty set, because I output nothing. 00:28:45.110 --> 00:28:46.370 align:middle line:90% There are no 0's and 1's. 00:28:46.370 --> 00:28:48.670 align:middle line:90% My message just says END. 00:28:48.670 --> 00:28:51.450 align:middle line:84% And then maybe down here, I output the symbol 0. 00:28:51.450 --> 00:28:53.590 align:middle line:84% Down here, I output the symbol 1. 00:28:53.590 --> 00:28:59.650 align:middle line:84% Down here, I output the code 00, 01, and so on. 00:28:59.650 --> 00:29:03.161 align:middle line:90% 00:29:03.161 --> 00:29:08.130 align:middle line:84% So my constraint that phi is a code, is a valid code, 00:29:08.130 --> 00:29:12.210 align:middle line:84% is just the constraint that I map each of my symbols 00:29:12.210 --> 00:29:15.770 align:middle line:84% to distinct nodes in this binary tree. 00:29:15.770 --> 00:29:18.143 align:middle line:90% I can't have any collisions. 00:29:18.143 --> 00:29:19.810 align:middle line:84% As long as there are no collisions, then 00:29:19.810 --> 00:29:24.370 align:middle line:84% I meet the definition of being a valid coding function. 00:29:24.370 --> 00:29:26.330 align:middle line:84% And now, the intuition is you want 00:29:26.330 --> 00:29:29.650 align:middle line:84% to minimize the expected length for what the depth is 00:29:29.650 --> 00:29:32.650 align:middle line:84% of the symbol you're searching for wherever you map it 00:29:32.650 --> 00:29:34.010 align:middle line:90% in this tree. 00:29:34.010 --> 00:29:36.330 align:middle line:84% So what should you do for your first symbol? 00:29:36.330 --> 00:29:39.450 align:middle line:84% Maybe you would map it to the empty set 00:29:39.450 --> 00:29:41.970 align:middle line:90% because that's pretty shallow. 00:29:41.970 --> 00:29:44.470 align:middle line:84% Maybe your next symbol you would map somewhere here, 00:29:44.470 --> 00:29:47.090 align:middle line:84% and then you'd map here, and you'd map here. 00:29:47.090 --> 00:29:49.250 align:middle line:84% And basically, you'd fill out the levels 00:29:49.250 --> 00:29:51.430 align:middle line:90% of this tree in order. 00:29:51.430 --> 00:29:54.330 align:middle line:84% You'd fill out all previous things 00:29:54.330 --> 00:29:56.590 align:middle line:84% because all these symbols are all equivalent. 00:29:56.590 --> 00:29:59.650 align:middle line:84% And this will minimize the expected length. 00:29:59.650 --> 00:30:02.250 align:middle line:84% And you can check that most of the symbols 00:30:02.250 --> 00:30:04.630 align:middle line:84% are going to get mapped down here, 00:30:04.630 --> 00:30:08.310 align:middle line:84% where their length of the message will be log k. 00:30:08.310 --> 00:30:11.210 align:middle line:84% And maybe you're off by some constant factor 00:30:11.210 --> 00:30:14.390 align:middle line:84% that depends on the fact that there are symbols before it. 00:30:14.390 --> 00:30:16.230 align:middle line:84% But if you work out the details, this 00:30:16.230 --> 00:30:18.950 align:middle line:90% is what you can show explicitly. 00:30:18.950 --> 00:30:21.350 align:middle line:84% It's just this lemma that, for the uniform distribution 00:30:21.350 --> 00:30:23.990 align:middle line:84% for our coding function, you can't do better 00:30:23.990 --> 00:30:27.776 align:middle line:90% than log k minus some constant. 00:30:27.776 --> 00:30:30.050 align:middle line:84% But now, we're going to use this simple lemma. 00:30:30.050 --> 00:30:33.350 align:middle line:84% And we're going to appeal to it in powerful ways. 00:30:33.350 --> 00:30:35.630 align:middle line:84% So let's do a thought experiment before we 00:30:35.630 --> 00:30:39.710 align:middle line:84% get to the first part of Shannon's theorem. 00:30:39.710 --> 00:30:42.030 align:middle line:84% And let me ask a simpler sounding 00:30:42.030 --> 00:30:45.540 align:middle line:84% question, which is going to be the heart of the lower bound. 00:30:45.540 --> 00:30:48.550 align:middle line:90% 00:30:48.550 --> 00:30:51.950 align:middle line:84% So what if I made your life easier? 00:30:51.950 --> 00:30:54.830 align:middle line:84% You're trying to encode a general message 00:30:54.830 --> 00:30:57.030 align:middle line:90% from this first order source. 00:30:57.030 --> 00:30:58.710 align:middle line:84% But what if I made your life easier 00:30:58.710 --> 00:31:02.610 align:middle line:84% by promising you something structurally about the message 00:31:02.610 --> 00:31:06.250 align:middle line:84% you're encoding that gave you more information? 00:31:06.250 --> 00:31:26.050 align:middle line:84% So what if you know there are n1 occurrences of the symbol a1, 00:31:26.050 --> 00:31:34.150 align:middle line:90% n2 occurrences of a2, and so on? 00:31:34.150 --> 00:31:36.970 align:middle line:84% So I'm making your life easier because instead 00:31:36.970 --> 00:31:40.210 align:middle line:84% of having to worry about a general message 00:31:40.210 --> 00:31:42.610 align:middle line:84% from your first order source, what 00:31:42.610 --> 00:31:46.290 align:middle line:84% if I told you something about that message and promised you 00:31:46.290 --> 00:31:49.950 align:middle line:84% I'm not going to tell you where these symbols of a1 appear, 00:31:49.950 --> 00:31:52.610 align:middle line:84% but I'm telling you, there are exactly n1 of them. 00:31:52.610 --> 00:31:55.230 align:middle line:84% I'm not going to tell you where these symbols a2 occur, 00:31:55.230 --> 00:31:57.170 align:middle line:84% but there are exactly n2 of them. 00:31:57.170 --> 00:32:00.710 align:middle line:84% What if I told you this for each different symbol type, 00:32:00.710 --> 00:32:03.870 align:middle line:84% I told you exactly what their count was? 00:32:03.870 --> 00:32:08.350 align:middle line:84% So now, what we can do is we can ask, first of all, how 00:32:08.350 --> 00:32:12.650 align:middle line:90% many such sequences are there. 00:32:12.650 --> 00:32:16.150 align:middle line:90% 00:32:16.150 --> 00:32:19.150 align:middle line:84% If I promise you that this condition holds, 00:32:19.150 --> 00:32:21.010 align:middle line:84% then I can ask a counting question, 00:32:21.010 --> 00:32:23.150 align:middle line:84% which is how many possible messages could there 00:32:23.150 --> 00:32:26.430 align:middle line:84% be under this promise that I've given you? 00:32:26.430 --> 00:32:28.070 align:middle line:84% You can see now this will connect back 00:32:28.070 --> 00:32:31.350 align:middle line:84% to a lot of the topics we did earlier in the class. 00:32:31.350 --> 00:32:35.550 align:middle line:84% So the key is really the multinomial. 00:32:35.550 --> 00:32:38.550 align:middle line:84% If we had only two symbols and I told you 00:32:38.550 --> 00:32:39.770 align:middle line:90% how many 1's there are-- 00:32:39.770 --> 00:32:42.670 align:middle line:84% I told you there were k 1's out of n things-- 00:32:42.670 --> 00:32:47.190 align:middle line:84% the answer for how many strings there would be n choose k. 00:32:47.190 --> 00:32:50.450 align:middle line:84% But in general, if I have k larger than 2, 00:32:50.450 --> 00:32:52.950 align:middle line:84% when I tell you all of those types, what you're going to get 00:32:52.950 --> 00:32:55.910 align:middle line:84% is you're going to get this multinomial, which 00:32:55.910 --> 00:33:02.210 align:middle line:84% is called n choose n1 n2 all the way up to nk. 00:33:02.210 --> 00:33:04.350 align:middle line:84% And it's defined-- we've seen it before. 00:33:04.350 --> 00:33:11.690 align:middle line:84% It's just n factorial over n1 factorial n2 factorial 00:33:11.690 --> 00:33:14.250 align:middle line:90% all the way up to nk factorial. 00:33:14.250 --> 00:33:16.010 align:middle line:84% That just counts the number of ways 00:33:16.010 --> 00:33:19.290 align:middle line:84% of coloring a set of integers from 1 00:33:19.290 --> 00:33:22.850 align:middle line:84% to n with k different colors once I promise you 00:33:22.850 --> 00:33:27.610 align:middle line:84% that there are n1 reds, n2 blues, and so on. 00:33:27.610 --> 00:33:30.450 align:middle line:84% So let's just denote this quantity as m 00:33:30.450 --> 00:33:32.650 align:middle line:84% because this will be important for us. 00:33:32.650 --> 00:33:36.490 align:middle line:84% And now, we can get to a key observation. 00:33:36.490 --> 00:33:39.130 align:middle line:84% So it sounds like, so far in proving Shannon's theorem, 00:33:39.130 --> 00:33:41.355 align:middle line:84% we've only thought about special cases. 00:33:41.355 --> 00:33:42.730 align:middle line:84% We've talked about the case where 00:33:42.730 --> 00:33:46.210 align:middle line:84% we had the uniform distribution on k symbols. 00:33:46.210 --> 00:33:48.130 align:middle line:84% But what I claim here is that now we're 00:33:48.130 --> 00:33:50.770 align:middle line:84% back to the case of the uniform distribution. 00:33:50.770 --> 00:33:53.290 align:middle line:84% So we know how many sequences there are. 00:33:53.290 --> 00:33:59.630 align:middle line:84% And moreover, each of these sequences 00:33:59.630 --> 00:34:02.990 align:middle line:90% has the same probability. 00:34:02.990 --> 00:34:08.389 align:middle line:84% So the probability that they have is p1 to the n1 p2 00:34:08.389 --> 00:34:13.630 align:middle line:90% to the n2 times pk to the nk. 00:34:13.630 --> 00:34:16.469 align:middle line:84% So there are a ton of possible sequences. 00:34:16.469 --> 00:34:20.250 align:middle line:84% But each particular sequence you give me, well, 00:34:20.250 --> 00:34:24.469 align:middle line:84% every time I see a symbol of a1, the probability that happened 00:34:24.469 --> 00:34:25.290 align:middle line:90% was p1. 00:34:25.290 --> 00:34:27.030 align:middle line:90% So I multiply p1. 00:34:27.030 --> 00:34:31.790 align:middle line:84% Every time I see a symbol of type a2, I multiply by p2. 00:34:31.790 --> 00:34:34.830 align:middle line:84% So what this means is we're back in the uniform distribution 00:34:34.830 --> 00:34:39.510 align:middle line:84% setting because when I made you this promise about the number 00:34:39.510 --> 00:34:42.070 align:middle line:84% of counts of each type of symbol, 00:34:42.070 --> 00:34:45.510 align:middle line:84% well, I know how many messages there are. 00:34:45.510 --> 00:34:48.070 align:middle line:84% And I know that it's the uniform distribution 00:34:48.070 --> 00:34:50.630 align:middle line:90% on those possible messages. 00:34:50.630 --> 00:34:54.860 align:middle line:84% So the intuition behind the first part of Shannon's theorem 00:34:54.860 --> 00:34:58.100 align:middle line:90% really comes from two parts. 00:34:58.100 --> 00:35:01.320 align:middle line:84% We're going to expand what's happening with this expression. 00:35:01.320 --> 00:35:03.380 align:middle line:84% We're going to use Stirling's approximation. 00:35:03.380 --> 00:35:06.940 align:middle line:84% And the binary entropy function will pop out. 00:35:06.940 --> 00:35:10.780 align:middle line:84% But then it'll turn out that we can remove this assumption 00:35:10.780 --> 00:35:14.380 align:middle line:84% because when we think back to tail bounds, 00:35:14.380 --> 00:35:17.700 align:middle line:84% if I had a first order source where the probability was 00:35:17.700 --> 00:35:21.300 align:middle line:84% p1 for the first symbol and p2 for the second symbol-- 00:35:21.300 --> 00:35:24.780 align:middle line:84% and really, that's a coin that has bias p1-- 00:35:24.780 --> 00:35:27.900 align:middle line:84% and I flip it n times, what we know 00:35:27.900 --> 00:35:31.980 align:middle line:84% is that the empirical number of times we see heads 00:35:31.980 --> 00:35:36.100 align:middle line:84% is going to converge very quickly to p1 as a fraction. 00:35:36.100 --> 00:35:38.100 align:middle line:84% So in fact, this assumption that you 00:35:38.100 --> 00:35:42.220 align:middle line:84% know what n1, n2, and all the way up to nk are 00:35:42.220 --> 00:35:44.500 align:middle line:84% is not really such a big assumption. 00:35:44.500 --> 00:35:47.000 align:middle line:84% Just because of large deviation principles, 00:35:47.000 --> 00:35:50.220 align:middle line:84% I should approximately know the number of symbols of each type 00:35:50.220 --> 00:35:51.222 align:middle line:90% anyways. 00:35:51.222 --> 00:35:52.680 align:middle line:84% And that's the way that we're going 00:35:52.680 --> 00:35:53.837 align:middle line:90% to piece this all together. 00:35:53.837 --> 00:35:55.420 align:middle line:84% So that's the preview of the argument. 00:35:55.420 --> 00:35:58.080 align:middle line:90% But let's do it. 00:35:58.080 --> 00:36:07.840 align:middle line:84% So now, what we can do is, from lemma 1, 00:36:07.840 --> 00:36:11.600 align:middle line:84% we know that the expected length of my encoding, 00:36:11.600 --> 00:36:15.840 align:middle line:84% even if I condition on telling you the information that there 00:36:15.840 --> 00:36:22.040 align:middle line:84% are n1 occurrences of a1, n2 occurrences of a2 00:36:22.040 --> 00:36:25.920 align:middle line:84% and so on, we know, by the first lemma, 00:36:25.920 --> 00:36:33.040 align:middle line:84% that the expected length must be log m minus maybe some constant. 00:36:33.040 --> 00:36:35.240 align:middle line:84% So here, this constant isn't so bad 00:36:35.240 --> 00:36:39.060 align:middle line:84% because log m is a really gigantic thing. 00:36:39.060 --> 00:36:41.800 align:middle line:84% You should think of it as being like exponential in n 00:36:41.800 --> 00:36:45.800 align:middle line:84% And this other o of 1 is just a constant. 00:36:45.800 --> 00:36:47.600 align:middle line:84% So now, the entire name of the game 00:36:47.600 --> 00:36:51.300 align:middle line:84% is just to figure out what log m is. 00:36:51.300 --> 00:36:52.940 align:middle line:84% And this is where the binary entropy 00:36:52.940 --> 00:36:54.400 align:middle line:90% function is going to pop out. 00:36:54.400 --> 00:36:59.300 align:middle line:90% 00:36:59.300 --> 00:37:00.440 align:middle line:90% So let's do that. 00:37:00.440 --> 00:37:03.580 align:middle line:90% 00:37:03.580 --> 00:37:06.780 align:middle line:84% So we can just plug in for our expression for m 00:37:06.780 --> 00:37:11.260 align:middle line:84% this multinomial expression we wrote down. 00:37:11.260 --> 00:37:13.780 align:middle line:84% And then we're going to use Stirling's formula to group 00:37:13.780 --> 00:37:16.700 align:middle line:90% together terms and simplify. 00:37:16.700 --> 00:37:19.700 align:middle line:84% So plugging in the multinomial, we 00:37:19.700 --> 00:37:26.580 align:middle line:84% get that log m is equal to log of n factorial minus the sum 00:37:26.580 --> 00:37:33.180 align:middle line:84% from i equals 1 to k of log ni factorial. 00:37:33.180 --> 00:37:36.660 align:middle line:84% And then I have my constant sitting right here, 00:37:36.660 --> 00:37:39.940 align:middle line:84% which is the slack in this expression. 00:37:39.940 --> 00:37:42.140 align:middle line:84% So this is just using the multinomial expression 00:37:42.140 --> 00:37:44.860 align:middle line:90% and using properties of the log. 00:37:44.860 --> 00:37:48.290 align:middle line:84% And now, what I can do is I can use Stirling. 00:37:48.290 --> 00:37:51.120 align:middle line:90% 00:37:51.120 --> 00:37:56.840 align:middle line:84% So Stirling's formula gives us an expression 00:37:56.840 --> 00:38:03.760 align:middle line:84% that n factorial behaves like square root of 2 pi n 00:38:03.760 --> 00:38:07.080 align:middle line:90% times n over e to the n. 00:38:07.080 --> 00:38:10.000 align:middle line:84% And really, this part isn't going 00:38:10.000 --> 00:38:13.580 align:middle line:84% to matter because I'm taking the log of all of these expressions. 00:38:13.580 --> 00:38:15.760 align:middle line:84% That's just going to get thrown into the constant. 00:38:15.760 --> 00:38:19.340 align:middle line:84% The important thing is this n over e to the n. 00:38:19.340 --> 00:38:20.840 align:middle line:84% So when we plug that in, we're going 00:38:20.840 --> 00:38:22.660 align:middle line:90% to get a very nice expression. 00:38:22.660 --> 00:38:24.440 align:middle line:90% So let's use Stirling. 00:38:24.440 --> 00:38:27.080 align:middle line:84% Well, we'll get that the expected length 00:38:27.080 --> 00:38:40.920 align:middle line:84% is at least n log n minus the sum i equals 1 to k ni log ni. 00:38:40.920 --> 00:38:43.520 align:middle line:84% And I'll have some more expressions right here. 00:38:43.520 --> 00:38:47.300 align:middle line:84% All I'm doing is I'm pulling out this n to the n term. 00:38:47.300 --> 00:38:50.580 align:middle line:84% So when I have n factorial behaves like n to the n, 00:38:50.580 --> 00:38:52.180 align:middle line:90% I get n log n. 00:38:52.180 --> 00:38:54.600 align:middle line:90% This behaves like ni to the ni. 00:38:54.600 --> 00:38:57.020 align:middle line:90% So I get ni log ni. 00:38:57.020 --> 00:39:00.060 align:middle line:84% But now, I have to deal with all of these e to the n correction 00:39:00.060 --> 00:39:01.220 align:middle line:90% terms. 00:39:01.220 --> 00:39:06.060 align:middle line:84% So I'm going to have a minus n log of e, 00:39:06.060 --> 00:39:14.660 align:middle line:84% and then a plus sum from i equals 1 to k of ni log e. 00:39:14.660 --> 00:39:20.120 align:middle line:84% And then I'll have my o of 1 sitting in here. 00:39:20.120 --> 00:39:22.220 align:middle line:90% So this is my expression. 00:39:22.220 --> 00:39:26.820 align:middle line:84% But now, life is good because I claim that this expression just 00:39:26.820 --> 00:39:28.980 align:middle line:90% goes away. 00:39:28.980 --> 00:39:31.980 align:middle line:90% So this is n log e. 00:39:31.980 --> 00:39:36.980 align:middle line:84% And this is the sum over i from 1 to k of ni log e. 00:39:36.980 --> 00:39:41.380 align:middle line:84% So what is the sum of all of the ni's? 00:39:41.380 --> 00:39:44.680 align:middle line:90% n Why? 00:39:44.680 --> 00:39:48.080 align:middle line:84% AUDIENCE: Because like, that's the tall number of each, 00:39:48.080 --> 00:39:49.160 align:middle line:90% I guess-- 00:39:49.160 --> 00:39:50.160 align:middle line:90% ANKUR MOITRA: Perfect. 00:39:50.160 --> 00:39:53.100 align:middle line:84% So the important point is just to not forget what the ni's are. 00:39:53.100 --> 00:39:56.000 align:middle line:84% They count the symbols of type i. 00:39:56.000 --> 00:39:59.560 align:middle line:84% And the total number of symbols of type i when we sum over i 00:39:59.560 --> 00:40:02.160 align:middle line:84% is just the total number of symbols, which is n. 00:40:02.160 --> 00:40:04.415 align:middle line:84% So this part completely goes away. 00:40:04.415 --> 00:40:06.040 align:middle line:84% And now, we're in good shape because we 00:40:06.040 --> 00:40:10.260 align:middle line:84% can start to see where the entropy function comes in. 00:40:10.260 --> 00:40:12.720 align:middle line:84% So how can we think about this in terms 00:40:12.720 --> 00:40:14.640 align:middle line:90% of the entropy function? 00:40:14.640 --> 00:40:16.440 align:middle line:84% Well, we can rewrite this expression 00:40:16.440 --> 00:40:23.600 align:middle line:84% right here as the sum from i equals 1 to k of ni log n 00:40:23.600 --> 00:40:26.060 align:middle line:84% because then it's just the knee that's varying. 00:40:26.060 --> 00:40:29.240 align:middle line:84% I'm just splitting n up into the sum of the ni's. 00:40:29.240 --> 00:40:33.640 align:middle line:84% And then when I arrange these expressions, 00:40:33.640 --> 00:40:34.980 align:middle line:90% what I'm going to get-- 00:40:34.980 --> 00:40:43.180 align:middle line:84% so this part right here is the sum from i equals 1 to k of ni. 00:40:43.180 --> 00:40:53.512 align:middle line:90% And then I have log n over ni. 00:40:53.512 --> 00:41:00.940 align:middle line:84% And when I put a minus in front of this, I can switch these two. 00:41:00.940 --> 00:41:04.920 align:middle line:84% And then I'll have my minus o of 1 sitting here. 00:41:04.920 --> 00:41:10.420 align:middle line:84% So in fact, now, what I can do is I can divide through by n. 00:41:10.420 --> 00:41:16.000 align:middle line:84% And what I'll get is ni over n that looks like a probability. 00:41:16.000 --> 00:41:18.740 align:middle line:84% So it'll be equal, this expression right here, 00:41:18.740 --> 00:41:25.140 align:middle line:84% to the binary entropy function of n1 over n n2 over n 00:41:25.140 --> 00:41:28.180 align:middle line:90% all the way up to nk over n. 00:41:28.180 --> 00:41:36.380 align:middle line:84% And still now I have my little o of actually n term here. 00:41:36.380 --> 00:41:39.500 align:middle line:90% So I'm in good shape. 00:41:39.500 --> 00:41:43.640 align:middle line:84% So you can see that when I expect each of these ni's-- 00:41:43.640 --> 00:41:46.800 align:middle line:84% the ni over n-- to behave like pi, 00:41:46.800 --> 00:41:49.920 align:middle line:84% then I'm pulling out this binary entropy function. 00:41:49.920 --> 00:41:52.820 align:middle line:84% And this is the first part of Shannon's theorem. 00:41:52.820 --> 00:41:56.000 align:middle line:84% This gives us our valid lower bound. 00:41:56.000 --> 00:41:57.500 align:middle line:90% It's actually almost there. 00:41:57.500 --> 00:42:00.680 align:middle line:90% Let me just clean up the proof. 00:42:00.680 --> 00:42:03.880 align:middle line:84% So what this tells us really is that, 00:42:03.880 --> 00:42:06.160 align:middle line:84% even if you were given the promise 00:42:06.160 --> 00:42:09.080 align:middle line:84% that there are ni symbols of type i, 00:42:09.080 --> 00:42:12.360 align:middle line:84% then you can't do better than this expression. 00:42:12.360 --> 00:42:15.800 align:middle line:84% But in principle, the thing you might be worried about 00:42:15.800 --> 00:42:22.360 align:middle line:84% is maybe every configuration of the ni symbols is very unlikely. 00:42:22.360 --> 00:42:25.960 align:middle line:84% So now, let's turn this lower bound into an actual lower 00:42:25.960 --> 00:42:29.280 align:middle line:90% bound for Shannon's theorem. 00:42:29.280 --> 00:42:29.840 align:middle line:90% Let's see. 00:42:29.840 --> 00:42:30.890 align:middle line:90% I can do it over here. 00:42:30.890 --> 00:42:44.300 align:middle line:90% 00:42:44.300 --> 00:42:49.300 align:middle line:84% So let me just reiterate why I'm not quite done yet. 00:42:49.300 --> 00:42:51.980 align:middle line:84% So I claim that what we've done so far 00:42:51.980 --> 00:43:02.860 align:middle line:84% doesn't quite prove what we're after because the thing we could 00:43:02.860 --> 00:43:22.420 align:middle line:84% be worried about is that there might not be any choice of ni's 00:43:22.420 --> 00:43:23.350 align:middle line:90% that's very common. 00:43:23.350 --> 00:43:28.780 align:middle line:90% 00:43:28.780 --> 00:43:30.160 align:middle line:90% So let's pop up a level. 00:43:30.160 --> 00:43:32.660 align:middle line:84% So what I'm saying here in this proof-- 00:43:32.660 --> 00:43:35.320 align:middle line:84% the way the proof is structured is I'm saying, 00:43:35.320 --> 00:43:40.440 align:middle line:84% aha, even if I told you that ni is the number of occurrences 00:43:40.440 --> 00:43:44.020 align:middle line:84% of symbol type i, well, in that case, 00:43:44.020 --> 00:43:47.080 align:middle line:84% you can't beat the binary entropy function. 00:43:47.080 --> 00:43:50.880 align:middle line:84% The trouble is that it might be very unlikely that configuration 00:43:50.880 --> 00:43:53.080 align:middle line:90% of ni's is really what happens. 00:43:53.080 --> 00:43:55.678 align:middle line:84% So you might tell me, all right, when that happens, I'm dead. 00:43:55.678 --> 00:43:56.220 align:middle line:90% You're right. 00:43:56.220 --> 00:43:58.880 align:middle line:84% I have to pay the binary entropy function. 00:43:58.880 --> 00:44:02.980 align:middle line:84% But maybe I just never have to-- that situation never arises. 00:44:02.980 --> 00:44:04.480 align:middle line:84% So that's the last part of the proof 00:44:04.480 --> 00:44:05.897 align:middle line:84% that we have to worry about, is we 00:44:05.897 --> 00:44:09.160 align:middle line:84% have to argue why these ni's, at least some suitable choice 00:44:09.160 --> 00:44:11.760 align:middle line:84% of them, is actually quite likely. 00:44:11.760 --> 00:44:16.520 align:middle line:84% And so the key for this is the following definition. 00:44:16.520 --> 00:44:19.280 align:middle line:84% And then we'll connect it to our earlier discussions 00:44:19.280 --> 00:44:20.880 align:middle line:90% of tail bounds. 00:44:20.880 --> 00:44:29.360 align:middle line:84% So we'll say that a string is epsilon typical if we 00:44:29.360 --> 00:44:31.840 align:middle line:90% have the following property-- 00:44:31.840 --> 00:44:37.820 align:middle line:84% for all i, for all of the symbols in our alphabet-- 00:44:37.820 --> 00:44:39.460 align:middle line:90% well, we take that string. 00:44:39.460 --> 00:44:41.560 align:middle line:90% And we look at what its ni is. 00:44:41.560 --> 00:44:43.420 align:middle line:84% So what is its count of symbol type i? 00:44:43.420 --> 00:44:44.580 align:middle line:90% We divide it by n. 00:44:44.580 --> 00:44:48.500 align:middle line:84% And I want this to be very close to pi. 00:44:48.500 --> 00:44:51.580 align:middle line:84% In fact, I want it to be epsilon close. 00:44:51.580 --> 00:44:54.020 align:middle line:84% So this is what I was alluding to before, 00:44:54.020 --> 00:44:57.620 align:middle line:84% was that this is the type of thing we proved for tail bounds. 00:44:57.620 --> 00:45:00.740 align:middle line:84% When k equals 2, when we just have heads or tails, 00:45:00.740 --> 00:45:03.260 align:middle line:84% we want the empirical number of heads 00:45:03.260 --> 00:45:07.620 align:middle line:84% in our string to be very close to the true bias of my coin. 00:45:07.620 --> 00:45:10.900 align:middle line:84% I want it to be close up to an additive epsilon. 00:45:10.900 --> 00:45:12.240 align:middle line:90% And this is the same thing. 00:45:12.240 --> 00:45:14.300 align:middle line:84% It's just saying that the empirical average 00:45:14.300 --> 00:45:15.820 align:middle line:84% of the number of times we actually 00:45:15.820 --> 00:45:18.420 align:middle line:84% roll a particular side for our die 00:45:18.420 --> 00:45:21.700 align:middle line:84% is epsilon close to the true probability. 00:45:21.700 --> 00:45:25.020 align:middle line:84% So we'll say that a string from our first order source 00:45:25.020 --> 00:45:29.900 align:middle line:84% is epsilon typical if this condition holds for all i. 00:45:29.900 --> 00:45:32.700 align:middle line:84% And now, the main part is really that we 00:45:32.700 --> 00:45:34.500 align:middle line:90% can come back to Chebyshev. 00:45:34.500 --> 00:45:37.800 align:middle line:90% 00:45:37.800 --> 00:45:42.720 align:middle line:84% Remember that we used tools like Chebyshev, Chernoff inequality, 00:45:42.720 --> 00:45:49.960 align:middle line:84% and so on to prove that the empirical number of heads 00:45:49.960 --> 00:45:53.440 align:middle line:84% converges very quickly to the expected number of heads. 00:45:53.440 --> 00:45:57.680 align:middle line:84% And this works even when we have a k-sided die instead. 00:45:57.680 --> 00:46:00.920 align:middle line:84% So what I claim is really the consequence of what we proved 00:46:00.920 --> 00:46:03.360 align:middle line:84% by Chebyshev's inequality quite a few lectures ago 00:46:03.360 --> 00:46:06.600 align:middle line:84% at this point, is that the probability 00:46:06.600 --> 00:46:10.120 align:middle line:84% that a random string from our first order source 00:46:10.120 --> 00:46:13.760 align:middle line:84% is epsilon typical is actually quite large. 00:46:13.760 --> 00:46:20.120 align:middle line:84% It's like 1 minus maybe k over epsilon squared n. 00:46:20.120 --> 00:46:22.600 align:middle line:84% So the point was that when we had Chebyshev, 00:46:22.600 --> 00:46:24.800 align:middle line:84% we could argue that, as n was going 00:46:24.800 --> 00:46:28.600 align:middle line:84% to infinity, because the variances behaved nicely 00:46:28.600 --> 00:46:31.120 align:middle line:84% for the sum of independent random variables, 00:46:31.120 --> 00:46:34.060 align:middle line:84% that the probability of being far away from our expectation 00:46:34.060 --> 00:46:38.558 align:middle line:84% was going down rapidly with n, at a 1 over n kind of rate. 00:46:38.558 --> 00:46:40.100 align:middle line:84% And then maybe I have to do some kind 00:46:40.100 --> 00:46:42.620 align:middle line:84% of union bound over each of these different k 00:46:42.620 --> 00:46:43.980 align:middle line:90% sides for the die. 00:46:43.980 --> 00:46:47.520 align:middle line:84% Any one of those goes wrong, I fail typicality. 00:46:47.520 --> 00:46:49.020 align:middle line:84% But at the end of the day, the point 00:46:49.020 --> 00:46:51.140 align:middle line:84% is, when you look at this expression, 00:46:51.140 --> 00:46:54.220 align:middle line:90% k and epsilon are fixed. 00:46:54.220 --> 00:46:57.100 align:middle line:84% So k is fixed by our first order source. 00:46:57.100 --> 00:46:59.420 align:middle line:84% And you should think of epsilon as being your target 00:46:59.420 --> 00:47:01.500 align:middle line:90% accuracy in Shannon's theorem. 00:47:01.500 --> 00:47:03.780 align:middle line:84% Maybe I want to get up to optimality up 00:47:03.780 --> 00:47:07.220 align:middle line:90% to a 0.01 or a 0.001. 00:47:07.220 --> 00:47:10.420 align:middle line:84% The point is just that if you take n large enough, 00:47:10.420 --> 00:47:12.860 align:middle line:84% then this term is still going to 0. 00:47:12.860 --> 00:47:15.740 align:middle line:90% So you're almost always typical. 00:47:15.740 --> 00:47:17.900 align:middle line:84% So one of the hard parts about the results 00:47:17.900 --> 00:47:20.380 align:middle line:84% we're doing in coding theory is keeping all the parameters 00:47:20.380 --> 00:47:22.660 align:middle line:84% straight in your head, in terms of what's 00:47:22.660 --> 00:47:24.980 align:middle line:84% the order in which things go to infinity. 00:47:24.980 --> 00:47:26.740 align:middle line:90% So k is some fixed thing. 00:47:26.740 --> 00:47:28.600 align:middle line:90% Epsilon is merely tiny. 00:47:28.600 --> 00:47:32.440 align:middle line:84% And then n can be gigantic in relation to things like 1 00:47:32.440 --> 00:47:34.383 align:middle line:90% over epsilon. 00:47:34.383 --> 00:47:35.800 align:middle line:84% Any questions about this statement 00:47:35.800 --> 00:47:37.240 align:middle line:90% that I'm asserting right here? 00:47:37.240 --> 00:47:39.760 align:middle line:84% This makes sense and it seems to jive with the things we 00:47:39.760 --> 00:47:40.940 align:middle line:90% covered for tail bounds? 00:47:40.940 --> 00:47:41.760 align:middle line:90% Yeah? 00:47:41.760 --> 00:47:42.920 align:middle line:90% OK. 00:47:42.920 --> 00:47:46.760 align:middle line:84% So now, let's put this all together. 00:47:46.760 --> 00:47:54.110 align:middle line:84% So putting this all together, let's see what we have. 00:47:54.110 --> 00:48:03.840 align:middle line:90% 00:48:03.840 --> 00:48:07.480 align:middle line:84% So you give me some coding function. 00:48:07.480 --> 00:48:10.760 align:middle line:84% And I want to argue that that coding function cannot do super 00:48:10.760 --> 00:48:11.960 align:middle line:90% duper well. 00:48:11.960 --> 00:48:15.480 align:middle line:84% I care about the expected length of your coding function. 00:48:15.480 --> 00:48:19.160 align:middle line:84% And we can break this up into two parts using 00:48:19.160 --> 00:48:21.600 align:middle line:90% the law of total expectation. 00:48:21.600 --> 00:48:27.240 align:middle line:84% We can look at the expected length given that the string is 00:48:27.240 --> 00:48:31.300 align:middle line:84% epsilon typical times the probability that it's 00:48:31.300 --> 00:48:34.900 align:middle line:90% epsilon typical. 00:48:34.900 --> 00:48:39.340 align:middle line:84% And we can add in the expected length given 00:48:39.340 --> 00:48:47.900 align:middle line:84% that it's not epsilon typical times the probability that it's 00:48:47.900 --> 00:48:51.380 align:middle line:90% not epsilon typical. 00:48:51.380 --> 00:48:54.380 align:middle line:84% So all I'm doing is I'm using the law of total expectation 00:48:54.380 --> 00:48:57.540 align:middle line:84% to break up my computation because what 00:48:57.540 --> 00:49:01.700 align:middle line:84% we know how to reason about is really, this part right here. 00:49:01.700 --> 00:49:03.580 align:middle line:84% When we have a typical sequence, we 00:49:03.580 --> 00:49:07.100 align:middle line:84% can reason through this log of the multinomial what 00:49:07.100 --> 00:49:09.740 align:middle line:90% the expected length must be. 00:49:09.740 --> 00:49:11.340 align:middle line:90% So what I can do is I can just-- 00:49:11.340 --> 00:49:12.760 align:middle line:90% I'm interested in a lower bound. 00:49:12.760 --> 00:49:14.860 align:middle line:90% So I can just drop this term. 00:49:14.860 --> 00:49:17.860 align:middle line:84% So this entire thing is at least 1 00:49:17.860 --> 00:49:21.220 align:middle line:84% minus k over epsilon squared n because that's 00:49:21.220 --> 00:49:23.460 align:middle line:90% my probability of being typical. 00:49:23.460 --> 00:49:26.520 align:middle line:84% And then I'm going to use this fact that I had 00:49:26.520 --> 00:49:29.720 align:middle line:84% about how it relates to the binary entropy function. 00:49:29.720 --> 00:49:34.480 align:middle line:84% So I'll put in minus n sum from I equals 1 00:49:34.480 --> 00:49:45.760 align:middle line:84% to k of pi plus or minus epsilon log of pi plus or minus epsilon. 00:49:45.760 --> 00:49:51.680 align:middle line:84% And then the entire thing will be minus some little o of n. 00:49:51.680 --> 00:49:54.840 align:middle line:84% So here, I'm being slightly sloppy because the way that I 00:49:54.840 --> 00:49:57.440 align:middle line:84% reasoned about here was I gave you 00:49:57.440 --> 00:50:00.680 align:middle line:84% what n1, n2, all the way up to nk were. 00:50:00.680 --> 00:50:05.360 align:middle line:84% And then I talked about log of capital M being the lower bound. 00:50:05.360 --> 00:50:09.080 align:middle line:84% So here, really, epsilon typicality is not any one 00:50:09.080 --> 00:50:11.080 align:middle line:90% configuration of the ni's. 00:50:11.080 --> 00:50:14.000 align:middle line:84% It's a range of configurations for those ni's. 00:50:14.000 --> 00:50:19.040 align:middle line:84% But I know that each of those ni's are equal to ni over n 00:50:19.040 --> 00:50:22.680 align:middle line:84% is equal to pi plus or minus epsilon because that's 00:50:22.680 --> 00:50:25.460 align:middle line:84% the thing we plug into the binary entropy function. 00:50:25.460 --> 00:50:28.460 align:middle line:84% But the point is that this expression right here, it 00:50:28.460 --> 00:50:30.620 align:middle line:84% doesn't actually vary that much as you 00:50:30.620 --> 00:50:32.780 align:middle line:90% range epsilon over some tiny-- 00:50:32.780 --> 00:50:35.020 align:middle line:90% some constant. 00:50:35.020 --> 00:50:38.260 align:middle line:84% So in particular, if I take this expression 00:50:38.260 --> 00:50:42.645 align:middle line:84% and I look at epsilon, which is little o of 1-- 00:50:42.645 --> 00:50:44.020 align:middle line:84% so it's something that's actually 00:50:44.020 --> 00:50:47.620 align:middle line:84% going to 0 as the length of my code increases, 00:50:47.620 --> 00:50:49.540 align:middle line:90% the length of my message-- 00:50:49.540 --> 00:50:52.980 align:middle line:84% then what I'm going to get here is that this behaves basically 00:50:52.980 --> 00:50:57.940 align:middle line:84% like the entropy function at pi, up to some small slack. 00:50:57.940 --> 00:51:00.540 align:middle line:84% So what this will give me is what I was after, 00:51:00.540 --> 00:51:06.580 align:middle line:84% that this is equal to n times entropy minus little o of n. 00:51:06.580 --> 00:51:09.080 align:middle line:84% So really, this proof, putting it all together, 00:51:09.080 --> 00:51:11.140 align:middle line:84% it mimics what I did in the special case 00:51:11.140 --> 00:51:13.358 align:middle line:84% where I promised you what the ni's are. 00:51:13.358 --> 00:51:15.900 align:middle line:84% But it just uses the fact that maybe I don't want to tell you 00:51:15.900 --> 00:51:16.640 align:middle line:90% the ni's. 00:51:16.640 --> 00:51:19.020 align:middle line:84% I just want to tell you it's epsilon typical. 00:51:19.020 --> 00:51:21.540 align:middle line:84% And then I want to use the fact that the binary entropy 00:51:21.540 --> 00:51:25.080 align:middle line:84% function doesn't vary much, as long as the pi's 00:51:25.080 --> 00:51:28.360 align:middle line:84% are what they're supposed to be plus or minus some little o 00:51:28.360 --> 00:51:29.520 align:middle line:90% of 1. 00:51:29.520 --> 00:51:31.380 align:middle line:84% So these are all computations you can do. 00:51:31.380 --> 00:51:34.080 align:middle line:84% But this is just the intuition for where it is. 00:51:34.080 --> 00:51:36.820 align:middle line:84% And this gives us the first part of Shannon's theorem. 00:51:36.820 --> 00:51:40.120 align:middle line:84% We have our bona fide lower bound 00:51:40.120 --> 00:51:42.480 align:middle line:84% on the best possible coding function. 00:51:42.480 --> 00:51:45.040 align:middle line:84% So are there any questions about the proof of the first part 00:51:45.040 --> 00:51:47.640 align:middle line:90% of Shannon's theorem? 00:51:47.640 --> 00:51:51.760 align:middle line:90% This was a lot of ingredients. 00:51:51.760 --> 00:51:52.982 align:middle line:90% Yeah? 00:51:52.982 --> 00:51:54.440 align:middle line:84% AUDIENCE: Can you explain again why 00:51:54.440 --> 00:51:57.160 align:middle line:84% we don't need to consider the square root of n? 00:51:57.160 --> 00:51:59.960 align:middle line:90% The square root of pi n? 00:51:59.960 --> 00:52:01.640 align:middle line:84% ANKUR MOITRA: The square root 2 pi n? 00:52:01.640 --> 00:52:02.440 align:middle line:90% Yeah. 00:52:02.440 --> 00:52:05.480 align:middle line:84% Well, OK, so here, what's going to happen 00:52:05.480 --> 00:52:09.080 align:middle line:90% is that all of these things-- 00:52:09.080 --> 00:52:13.120 align:middle line:84% I'm really plugging Stirling in into this expression. 00:52:13.120 --> 00:52:16.080 align:middle line:84% So when I take log of this expression, 00:52:16.080 --> 00:52:18.920 align:middle line:84% I'm going to have these giant terms in here. 00:52:18.920 --> 00:52:22.500 align:middle line:84% Log of n to the n looks like n log n. 00:52:22.500 --> 00:52:24.098 align:middle line:84% I'm going to add in some other terms. 00:52:24.098 --> 00:52:24.640 align:middle line:90% You're right. 00:52:24.640 --> 00:52:26.780 align:middle line:84% I could worry about what happens here. 00:52:26.780 --> 00:52:28.900 align:middle line:90% And that would be a log n term. 00:52:28.900 --> 00:52:31.360 align:middle line:84% But the point is that, at the end of the day, 00:52:31.360 --> 00:52:34.620 align:middle line:84% the thing that I want is this little o of n. 00:52:34.620 --> 00:52:38.220 align:middle line:84% So all of those log n's are just tiny by comparison. 00:52:38.220 --> 00:52:39.940 align:middle line:84% So even though, by this vantage point, 00:52:39.940 --> 00:52:41.760 align:middle line:90% that looks like a serious term-- 00:52:41.760 --> 00:52:43.660 align:middle line:90% it's a root n sitting in there-- 00:52:43.660 --> 00:52:46.300 align:middle line:84% But after I take the logs, really, the action 00:52:46.300 --> 00:52:48.400 align:middle line:84% is all happening in this part right here. 00:52:48.400 --> 00:52:49.620 align:middle line:90% But good question. 00:52:49.620 --> 00:52:51.460 align:middle line:84% So that's what makes some of these coding 00:52:51.460 --> 00:52:54.200 align:middle line:84% theorems a bit tricky, is that I promised you, 00:52:54.200 --> 00:52:56.580 align:middle line:84% when we did things like probability and counting, 00:52:56.580 --> 00:52:58.820 align:middle line:84% that we would see ways that these tools would come up 00:52:58.820 --> 00:53:00.700 align:middle line:90% in very complicated things. 00:53:00.700 --> 00:53:04.500 align:middle line:84% This uses a lot of material from earlier in the class. 00:53:04.500 --> 00:53:06.900 align:middle line:84% If you just think about what all has happened, 00:53:06.900 --> 00:53:10.660 align:middle line:84% we used tail bounds to argue about typicality. 00:53:10.660 --> 00:53:13.260 align:middle line:84% We used combinatorics to understand 00:53:13.260 --> 00:53:16.420 align:middle line:84% what exactly this multinomial works out to be. 00:53:16.420 --> 00:53:18.500 align:middle line:84% And then we use Stirling's formula in order 00:53:18.500 --> 00:53:20.400 align:middle line:84% to massage it into something which has 00:53:20.400 --> 00:53:22.280 align:middle line:90% the binary entropy function. 00:53:22.280 --> 00:53:24.440 align:middle line:84% And all of these things required a lot of estimates 00:53:24.440 --> 00:53:27.400 align:middle line:84% about what the error terms are and how they add up. 00:53:27.400 --> 00:53:31.080 align:middle line:84% But the intuition, at its core, really comes down to this idea 00:53:31.080 --> 00:53:33.360 align:middle line:90% that you're encoding. 00:53:33.360 --> 00:53:37.920 align:middle line:84% If it works, I mean, typicality is a common thing that happens. 00:53:37.920 --> 00:53:39.400 align:middle line:84% And in that case, the probabilities 00:53:39.400 --> 00:53:42.400 align:middle line:84% of the different outcomes are very close to each other. 00:53:42.400 --> 00:53:46.240 align:middle line:84% So then you can't really beat log of the number of strings. 00:53:46.240 --> 00:53:49.520 align:middle line:84% So at the core, this lower bound in first part 00:53:49.520 --> 00:53:52.600 align:middle line:84% of Shannon's theorem is exactly this kind of statement 00:53:52.600 --> 00:53:55.280 align:middle line:84% that we all had intuition for, that you 00:53:55.280 --> 00:53:58.760 align:middle line:84% can't beat the naive encoding for the uniform distribution. 00:53:58.760 --> 00:54:00.440 align:middle line:84% And we're just reasoning about why 00:54:00.440 --> 00:54:02.560 align:middle line:84% this holds even when it's really close 00:54:02.560 --> 00:54:04.520 align:middle line:90% to the uniform distribution. 00:54:04.520 --> 00:54:05.060 align:middle line:90% All right. 00:54:05.060 --> 00:54:08.120 align:middle line:90% So this was a challenging proof. 00:54:08.120 --> 00:54:11.480 align:middle line:84% Tuesday's lecture will be an easier proof. 00:54:11.480 --> 00:54:15.440 align:middle line:84% Thursday's lecture will make this look like a cakewalk. 00:54:15.440 --> 00:54:17.900 align:middle line:90% I'm sorry, but it's true. 00:54:17.900 --> 00:54:19.160 align:middle line:90% So let me pause. 00:54:19.160 --> 00:54:20.485 align:middle line:90% Any other questions about this? 00:54:20.485 --> 00:54:21.360 align:middle line:90% Does this make sense? 00:54:21.360 --> 00:54:23.220 align:middle line:84% You guys all comfortable with it? 00:54:23.220 --> 00:54:24.260 align:middle line:90% Yeah? 00:54:24.260 --> 00:54:25.140 align:middle line:90% All right. 00:54:25.140 --> 00:54:28.540 align:middle line:84% So the good news is that now we're 00:54:28.540 --> 00:54:31.220 align:middle line:84% in a position where we can prove the other side of Shannon's 00:54:31.220 --> 00:54:32.940 align:middle line:90% theorem very easily. 00:54:32.940 --> 00:54:34.620 align:middle line:84% So the second part actually follows 00:54:34.620 --> 00:54:37.068 align:middle line:84% from roughly the same kind of argument. 00:54:37.068 --> 00:54:38.860 align:middle line:84% In fact, the way that I'm going to write it 00:54:38.860 --> 00:54:42.220 align:middle line:84% is really as a block diagram for what this encoding 00:54:42.220 --> 00:54:45.300 align:middle line:90% function looks like. 00:54:45.300 --> 00:54:51.190 align:middle line:84% So let's prove the second part of Shannon's theorem. 00:54:51.190 --> 00:55:12.180 align:middle line:90% 00:55:12.180 --> 00:55:14.730 align:middle line:84% So let me ask a question first, which will get us started. 00:55:14.730 --> 00:55:17.520 align:middle line:90% 00:55:17.520 --> 00:55:24.460 align:middle line:84% How many epsilon typical sequences are there? 00:55:24.460 --> 00:55:32.440 align:middle line:90% 00:55:32.440 --> 00:55:33.860 align:middle line:90% Well, let's just work it out. 00:55:33.860 --> 00:55:37.160 align:middle line:90% Let's do the combinatorics. 00:55:37.160 --> 00:55:38.960 align:middle line:84% So what I claim is that the answer 00:55:38.960 --> 00:55:42.280 align:middle line:90% is a sum over signatures-- 00:55:42.280 --> 00:55:49.290 align:middle line:84% all of these n1 n2 up to nk that are epsilon typical. 00:55:49.290 --> 00:55:53.960 align:middle line:90% 00:55:53.960 --> 00:55:57.600 align:middle line:84% And we have this multinomial here, 00:55:57.600 --> 00:56:03.200 align:middle line:84% n choose n1 n2 all the way up to nk. 00:56:03.200 --> 00:56:05.640 align:middle line:84% So this is just, by definition, the answer. 00:56:05.640 --> 00:56:07.800 align:middle line:84% I sum over all of the signatures that 00:56:07.800 --> 00:56:10.520 align:middle line:84% meet my definition of epsilon typicality. 00:56:10.520 --> 00:56:15.940 align:middle line:84% And I count the number of things that meet that signature. 00:56:15.940 --> 00:56:19.660 align:middle line:84% But let's get an upper bound on what this is. 00:56:19.660 --> 00:56:23.200 align:middle line:84% So first of all, I can think about this expression. 00:56:23.200 --> 00:56:25.340 align:middle line:84% I'll still be a bit sloppy here, the same way 00:56:25.340 --> 00:56:27.620 align:middle line:84% I was last time, where I'm going to think 00:56:27.620 --> 00:56:32.460 align:middle line:84% about this multinomial as really being 00:56:32.460 --> 00:56:41.340 align:middle line:84% p1 plus or minus epsilon times n p2 plus or minus epsilon times 00:56:41.340 --> 00:56:43.100 align:middle line:90% n and so on. 00:56:43.100 --> 00:56:46.460 align:middle line:84% So I don't know what these n1 n2's are. 00:56:46.460 --> 00:56:49.660 align:middle line:84% But certainly, I could choose whatever choices 00:56:49.660 --> 00:56:53.360 align:middle line:84% that are close to pi that maximize this expression. 00:56:53.360 --> 00:56:56.540 align:middle line:84% And I could use it as an upper bound for every term that 00:56:56.540 --> 00:56:58.300 align:middle line:90% shows up in the sum. 00:56:58.300 --> 00:57:01.860 align:middle line:84% So the same way I used this notation earlier just 00:57:01.860 --> 00:57:04.460 align:middle line:84% to think about all possible values that 00:57:04.460 --> 00:57:06.680 align:middle line:84% meet this condition of being close to the pi's, 00:57:06.680 --> 00:57:08.300 align:middle line:90% I'm doing the same thing here. 00:57:08.300 --> 00:57:10.500 align:middle line:84% But now, I have to worry about how many terms there 00:57:10.500 --> 00:57:12.000 align:middle line:90% are in the sum. 00:57:12.000 --> 00:57:13.710 align:middle line:84% And this is where we get a huge win. 00:57:13.710 --> 00:57:16.500 align:middle line:90% 00:57:16.500 --> 00:57:19.840 align:middle line:84% How many times does the sum execute? 00:57:19.840 --> 00:57:22.920 align:middle line:84% How many things are we summing over for this multinomial? 00:57:22.920 --> 00:57:25.220 align:middle line:84% It's actually not exponential in n. 00:57:25.220 --> 00:57:28.360 align:middle line:90% That's the key point. 00:57:28.360 --> 00:57:31.400 align:middle line:84% So in fact, how many different signatures 00:57:31.400 --> 00:57:34.600 align:middle line:84% meet the condition of being epsilon typical? 00:57:34.600 --> 00:57:36.640 align:middle line:84% Well, I could just ignore the constraint 00:57:36.640 --> 00:57:38.920 align:middle line:84% that it's epsilon typical and just 00:57:38.920 --> 00:57:41.920 align:middle line:84% look at all of the possible signatures. 00:57:41.920 --> 00:57:43.860 align:middle line:84% How many choices are there for n1? 00:57:43.860 --> 00:57:45.720 align:middle line:90% There's n possibilities. 00:57:45.720 --> 00:57:47.540 align:middle line:84% How many choices are there for n2? 00:57:47.540 --> 00:57:50.020 align:middle line:84% There's n possibilities and so on. 00:57:50.020 --> 00:57:51.840 align:middle line:90% So I get n to the k. 00:57:51.840 --> 00:57:53.327 align:middle line:84% So this term right here, remember 00:57:53.327 --> 00:57:55.160 align:middle line:84% that we're going to be taking the log of all 00:57:55.160 --> 00:57:56.840 align:middle line:90% of these expressions. 00:57:56.840 --> 00:58:00.120 align:middle line:84% So this term right here has the same property that it's not 00:58:00.120 --> 00:58:01.820 align:middle line:90% exponentially large in n. 00:58:01.820 --> 00:58:05.800 align:middle line:84% And that's a huge win for reasons we'll see. 00:58:05.800 --> 00:58:09.300 align:middle line:84% So I'm going to call this term right here m prime, 00:58:09.300 --> 00:58:12.390 align:middle line:84% even though it's really a function of whatever I jitter 00:58:12.390 --> 00:58:14.110 align:middle line:90% these pi's to be. 00:58:14.110 --> 00:58:16.430 align:middle line:84% But now, I promised you that, really, our proof 00:58:16.430 --> 00:58:18.840 align:middle line:84% is, in essence, going to be a block diagram. 00:58:18.840 --> 00:58:22.030 align:middle line:90% 00:58:22.030 --> 00:58:25.550 align:middle line:90% So let's prove it. 00:58:25.550 --> 00:58:30.630 align:middle line:84% Let's explicitly come up with a good coding scheme 00:58:30.630 --> 00:58:32.020 align:middle line:90% for this first order source. 00:58:32.020 --> 00:58:46.910 align:middle line:90% 00:58:46.910 --> 00:58:51.470 align:middle line:84% So we start off with our message. 00:58:51.470 --> 00:58:54.710 align:middle line:84% And then in my block diagram, I'm first going to ask, 00:58:54.710 --> 00:58:58.200 align:middle line:90% is the message epsilon typical? 00:58:58.200 --> 00:59:01.470 align:middle line:90% 00:59:01.470 --> 00:59:03.670 align:middle line:90% You give me the message. 00:59:03.670 --> 00:59:06.708 align:middle line:84% I can certainly compute what all the ni's are. 00:59:06.708 --> 00:59:08.250 align:middle line:84% And I can plug it into the expression 00:59:08.250 --> 00:59:10.090 align:middle line:84% and decide whether it meets my definition 00:59:10.090 --> 00:59:13.170 align:middle line:90% for some value of epsilon. 00:59:13.170 --> 00:59:17.650 align:middle line:84% Now, in the case where the answer is yes, 00:59:17.650 --> 00:59:21.090 align:middle line:84% how am I going to encode this message? 00:59:21.090 --> 00:59:26.890 align:middle line:84% The first thing that I'm going to do is I'm going to ask you, 00:59:26.890 --> 00:59:30.090 align:middle line:90% what is n1 up to nk? 00:59:30.090 --> 00:59:34.870 align:middle line:84% And then I can ask you, what string is it? 00:59:34.870 --> 00:59:41.130 align:middle line:90% 00:59:41.130 --> 00:59:43.650 align:middle line:84% Let's worry about the no branch later. 00:59:43.650 --> 00:59:45.930 align:middle line:84% But let's compute the number of bits 00:59:45.930 --> 00:59:48.330 align:middle line:90% that this coding scheme uses. 00:59:48.330 --> 00:59:51.810 align:middle line:90% So how long is my transmission? 00:59:51.810 --> 00:59:54.590 align:middle line:84% Well, I have to tell you whether the answer is yes or no. 00:59:54.590 --> 00:59:55.730 align:middle line:90% That's 1 bit. 00:59:55.730 --> 00:59:58.130 align:middle line:90% I tell you yes or no. 00:59:58.130 --> 01:00:03.610 align:middle line:84% What about telling you what the signature n1 up to nk is? 01:00:03.610 --> 01:00:07.160 align:middle line:90% How many bits is that at most? 01:00:07.160 --> 01:00:14.030 align:middle line:90% 01:00:14.030 --> 01:00:14.890 align:middle line:90% Yeah? 01:00:14.890 --> 01:00:15.640 align:middle line:90% AUDIENCE: k log n. 01:00:15.640 --> 01:00:18.030 align:middle line:90% ANKUR MOITRA: Perfect, k log n. 01:00:18.030 --> 01:00:22.275 align:middle line:84% Because I only pay the log of the number of possibilities, 01:00:22.275 --> 01:00:24.150 align:middle line:84% I just have to write out a binary string that 01:00:24.150 --> 01:00:26.790 align:middle line:84% tells me which of the possibilities it is. 01:00:26.790 --> 01:00:28.670 align:middle line:84% So this is a huge win because all 01:00:28.670 --> 01:00:31.230 align:middle line:84% of these questions, the answer to them 01:00:31.230 --> 01:00:33.650 align:middle line:84% is really hidden in this little o of n term 01:00:33.650 --> 01:00:35.710 align:middle line:90% because it's not linear in n. 01:00:35.710 --> 01:00:38.310 align:middle line:84% The only place where I'm going to pay linear in n 01:00:38.310 --> 01:00:42.390 align:middle line:84% is this last part right here, which is what is the string? 01:00:42.390 --> 01:00:47.150 align:middle line:84% This right here will get me log of M prime. 01:00:47.150 --> 01:00:50.250 align:middle line:84% And what we just argued was that log of M prime, 01:00:50.250 --> 01:00:53.910 align:middle line:84% even when I maximize over what p prime i to put 01:00:53.910 --> 01:00:58.150 align:middle line:84% in here that's epsilon close to pi, I'm going to get something-- 01:00:58.150 --> 01:01:04.130 align:middle line:84% the log of Mi is going to behave like the binary entropy function 01:01:04.130 --> 01:01:07.570 align:middle line:84% times n up to some lower order terms. 01:01:07.570 --> 01:01:10.010 align:middle line:84% So now, down this entire branch, I'm 01:01:10.010 --> 01:01:13.650 align:middle line:84% only paying exactly what Shannon wants me to pay. 01:01:13.650 --> 01:01:18.650 align:middle line:84% I'm paying up to leading order the binary entropy times n. 01:01:18.650 --> 01:01:21.290 align:middle line:84% Now, you guys can help me out with what 01:01:21.290 --> 01:01:23.690 align:middle line:90% happens in the rest of the way. 01:01:23.690 --> 01:01:27.630 align:middle line:84% So here, I could pay plus 1 bit because I have to tell you, no, 01:01:27.630 --> 01:01:30.810 align:middle line:90% it's not epsilon typical. 01:01:30.810 --> 01:01:34.310 align:middle line:84% And here, what I can do, as it turns out-- 01:01:34.310 --> 01:01:38.930 align:middle line:84% any guesses what I can do down this no branch? 01:01:38.930 --> 01:01:40.350 align:middle line:90% Anyone have any intuition? 01:01:40.350 --> 01:01:47.210 align:middle line:90% 01:01:47.210 --> 01:01:50.690 align:middle line:84% So let me ask a leading question. 01:01:50.690 --> 01:01:54.610 align:middle line:84% See, Shannon's theorem, I have this slack 01:01:54.610 --> 01:01:56.510 align:middle line:90% built in, little o of n. 01:01:56.510 --> 01:01:59.370 align:middle line:90% 01:01:59.370 --> 01:02:03.230 align:middle line:84% What's the probability that I go down this branch, at least if I 01:02:03.230 --> 01:02:05.090 align:middle line:90% set parameters correctly? 01:02:05.090 --> 01:02:06.210 align:middle line:90% What do I want it to be? 01:02:06.210 --> 01:02:10.110 align:middle line:84% Do I want it to be a half probability, something 01:02:10.110 --> 01:02:13.310 align:middle line:90% closer to 1, something tiny? 01:02:13.310 --> 01:02:16.030 align:middle line:90% Maybe a little o of 1? 01:02:16.030 --> 01:02:18.670 align:middle line:84% So if I set these parameters the right way, 01:02:18.670 --> 01:02:22.670 align:middle line:84% the probability that I fail and the thing is not epsilon typical 01:02:22.670 --> 01:02:24.630 align:middle line:84% is little o of 1 because, remember, 01:02:24.630 --> 01:02:26.750 align:middle line:84% it was this k over epsilon squared 01:02:26.750 --> 01:02:29.230 align:middle line:90% n for my Chebyshev bound. 01:02:29.230 --> 01:02:32.390 align:middle line:84% So if I set the parameters right, that's tiny. 01:02:32.390 --> 01:02:36.310 align:middle line:84% And if this is little o of 1, the probability I go down 01:02:36.310 --> 01:02:38.670 align:middle line:84% this branch, I can actually afford 01:02:38.670 --> 01:02:41.550 align:middle line:84% to do something really dumb here. 01:02:41.550 --> 01:02:43.830 align:middle line:84% So what could I do is I could just 01:02:43.830 --> 01:02:47.600 align:middle line:90% output the trivial encoding. 01:02:47.600 --> 01:02:52.150 align:middle line:90% 01:02:52.150 --> 01:02:55.110 align:middle line:84% So how much does this trivial encoding pay? 01:02:55.110 --> 01:02:59.610 align:middle line:84% It pays a constant times n, maybe 100 times n. 01:02:59.610 --> 01:03:00.990 align:middle line:90% But so what? 01:03:00.990 --> 01:03:03.490 align:middle line:84% It only happens with probability 1 over log n 01:03:03.490 --> 01:03:05.330 align:middle line:90% that we go down this branch. 01:03:05.330 --> 01:03:06.890 align:middle line:84% So it doesn't actually prevent me 01:03:06.890 --> 01:03:08.910 align:middle line:84% from proving Shannon's second theorem. 01:03:08.910 --> 01:03:10.630 align:middle line:90% So this is very unlikely. 01:03:10.630 --> 01:03:12.330 align:middle line:84% So I can just give up whenever that 01:03:12.330 --> 01:03:14.650 align:middle line:90% happens and do the naive thing. 01:03:14.650 --> 01:03:17.570 align:middle line:84% So let's analyze this and just write down 01:03:17.570 --> 01:03:20.730 align:middle line:84% what the expression is, even though this block diagram really 01:03:20.730 --> 01:03:21.490 align:middle line:90% does the trick. 01:03:21.490 --> 01:03:24.290 align:middle line:90% 01:03:24.290 --> 01:03:27.250 align:middle line:84% So let's compute the expected length. 01:03:27.250 --> 01:03:29.490 align:middle line:90% So the expected length-- 01:03:29.490 --> 01:03:32.570 align:middle line:84% I always have to tell you the answer to the question that's 01:03:32.570 --> 01:03:35.170 align:middle line:90% yes or no is it epsilon typical. 01:03:35.170 --> 01:03:37.450 align:middle line:84% If it's epsilon typical, which happens 01:03:37.450 --> 01:03:42.050 align:middle line:84% with 1 minus little o of 1 probability, 01:03:42.050 --> 01:03:45.810 align:middle line:84% because this n is gigantic compared to k and 1 over epsilon 01:03:45.810 --> 01:03:51.970 align:middle line:84% squared, in that case, I pay this k log n 01:03:51.970 --> 01:03:56.170 align:middle line:90% plus this log M prime. 01:03:56.170 --> 01:04:00.070 align:middle line:84% And otherwise, I pay this k over epsilon 01:04:00.070 --> 01:04:07.950 align:middle line:84% squared n all times maybe log k times n because this 01:04:07.950 --> 01:04:11.030 align:middle line:84% is my trivial encoding, is that for each symbol I just tell you 01:04:11.030 --> 01:04:13.390 align:middle line:90% which particular symbol it is. 01:04:13.390 --> 01:04:16.670 align:middle line:84% But the point is that, once I tell you what k is-- maybe 01:04:16.670 --> 01:04:17.790 align:middle line:90% it's 10. 01:04:17.790 --> 01:04:22.190 align:middle line:84% Once I fix my target accuracy-- maybe it's 0.00001-- 01:04:22.190 --> 01:04:24.910 align:middle line:84% n can be large enough so that this term 01:04:24.910 --> 01:04:27.830 align:middle line:90% becomes arbitrarily small. 01:04:27.830 --> 01:04:30.290 align:middle line:84% And then it's multiplied by something linear in n. 01:04:30.290 --> 01:04:33.010 align:middle line:84% But then it becomes much, much smaller. 01:04:33.010 --> 01:04:37.030 align:middle line:84% Any constant times n, I can make it as small as I want. 01:04:37.030 --> 01:04:43.430 align:middle line:84% So this is the proof of Shannon's second main theorem. 01:04:43.430 --> 01:04:44.490 align:middle line:90% Any questions? 01:04:44.490 --> 01:04:50.110 align:middle line:90% 01:04:50.110 --> 01:04:51.730 align:middle line:84% You guys should have some questions. 01:04:51.730 --> 01:04:59.290 align:middle line:90% 01:04:59.290 --> 01:05:02.810 align:middle line:84% So just to summarize, really, the way that all of this fit 01:05:02.810 --> 01:05:06.210 align:middle line:84% together was a bit magical, but it's really based on this block 01:05:06.210 --> 01:05:10.330 align:middle line:84% diagram, is that even though a priori we were worried about how 01:05:10.330 --> 01:05:12.250 align:middle line:84% do I encode an arbitrary thing that 01:05:12.250 --> 01:05:14.890 align:middle line:84% could come out of my first order source, 01:05:14.890 --> 01:05:17.770 align:middle line:84% the point is that there's this event that happens that makes it 01:05:17.770 --> 01:05:19.590 align:middle line:90% a lot easier to reason about. 01:05:19.590 --> 01:05:22.090 align:middle line:84% This event almost always happens. 01:05:22.090 --> 01:05:25.410 align:middle line:84% And once it's epsilon typical, then the problem 01:05:25.410 --> 01:05:29.930 align:middle line:84% behaves like the uniform distribution on some universe. 01:05:29.930 --> 01:05:33.350 align:middle line:84% And we just care about how big is that universe, M prime. 01:05:33.350 --> 01:05:34.970 align:middle line:90% That's a multinomial. 01:05:34.970 --> 01:05:37.170 align:middle line:84% And when we use Stirling's approximation, 01:05:37.170 --> 01:05:39.250 align:middle line:90% the binary entropy pops out. 01:05:39.250 --> 01:05:41.290 align:middle line:84% But this block diagram is really how 01:05:41.290 --> 01:05:45.490 align:middle line:84% the proof goes, both for the upper and the lower bounds. 01:05:45.490 --> 01:05:47.910 align:middle line:84% So we're probably going to finish a couple minutes early. 01:05:47.910 --> 01:05:51.250 align:middle line:84% But let me tell you where we're headed for this. 01:05:51.250 --> 01:05:55.070 align:middle line:90% So this theorem looks amazing. 01:05:55.070 --> 01:05:57.350 align:middle line:84% We started off with this question about what 01:05:57.350 --> 01:05:59.470 align:middle line:90% is the best encoding. 01:05:59.470 --> 01:06:01.550 align:middle line:84% We saw that there were naive encodings 01:06:01.550 --> 01:06:03.390 align:middle line:90% that we could improve upon. 01:06:03.390 --> 01:06:06.550 align:middle line:84% And then we figured out what is the natural limit, in terms 01:06:06.550 --> 01:06:08.950 align:middle line:84% of what's the best that we could ever hope for, 01:06:08.950 --> 01:06:10.950 align:middle line:84% at least in the asymptotic regime where 01:06:10.950 --> 01:06:12.710 align:middle line:90% n is going to infinity. 01:06:12.710 --> 01:06:15.270 align:middle line:84% But this coding scheme would actually be pretty terrible 01:06:15.270 --> 01:06:21.390 align:middle line:84% from a practical standpoint, if this were all there would be. 01:06:21.390 --> 01:06:24.670 align:middle line:84% So if you think about it, when I defined what a coding 01:06:24.670 --> 01:06:27.310 align:middle line:84% function was, a coding function is just 01:06:27.310 --> 01:06:31.950 align:middle line:84% a mapping from a possible message to what I'm going 01:06:31.950 --> 01:06:34.630 align:middle line:90% to encode it as, the codebook. 01:06:34.630 --> 01:06:38.350 align:middle line:84% But in principle, as n becomes gigantic, 01:06:38.350 --> 01:06:39.810 align:middle line:90% how large is that codebook? 01:06:39.810 --> 01:06:42.070 align:middle line:84% It's becoming exponentially large in M 01:06:42.070 --> 01:06:44.710 align:middle line:84% because there are all kinds of messages. 01:06:44.710 --> 01:06:46.150 align:middle line:84% So this would be terrible if I had 01:06:46.150 --> 01:06:50.990 align:middle line:84% to do this because I would have to pre-guess whatever message 01:06:50.990 --> 01:06:55.290 align:middle line:84% anyone would ever want to send and fix what it is I'm going 01:06:55.290 --> 01:06:57.450 align:middle line:90% to transmit along the channel. 01:06:57.450 --> 01:06:59.430 align:middle line:84% That would be a really bad way to do things, 01:06:59.430 --> 01:07:03.370 align:middle line:84% because it's not an efficient way to do the encoding, 01:07:03.370 --> 01:07:05.410 align:middle line:90% let alone the decoding. 01:07:05.410 --> 01:07:09.077 align:middle line:84% If I actually receive some message, then how in the world 01:07:09.077 --> 01:07:11.410 align:middle line:84% am I going to figure out what was the original thing you 01:07:11.410 --> 01:07:14.050 align:middle line:84% started from other than looking at everything 01:07:14.050 --> 01:07:16.696 align:middle line:84% and figuring out what its image was? 01:07:16.696 --> 01:07:18.570 align:middle line:84% So what we're going to cover on Tuesday 01:07:18.570 --> 01:07:20.250 align:middle line:84% is a really amazing scheme, which 01:07:20.250 --> 01:07:23.450 align:middle line:84% is going to give us a different way to think about compression. 01:07:23.450 --> 01:07:25.950 align:middle line:84% And it's a scheme that has a lot of beautiful combinatorics. 01:07:25.950 --> 01:07:27.570 align:middle line:90% It's called the Huffman code. 01:07:27.570 --> 01:07:29.490 align:middle line:84% And what will be great about the Huffman code 01:07:29.490 --> 01:07:32.570 align:middle line:84% is that it'll naturally give us efficient encodings 01:07:32.570 --> 01:07:34.130 align:middle line:90% and decodings. 01:07:34.130 --> 01:07:36.267 align:middle line:84% So that'll be the Tuesday lecture. 01:07:36.267 --> 01:07:37.850 align:middle line:84% And then on Thursday, what we're going 01:07:37.850 --> 01:07:42.290 align:middle line:84% to talk about is the other big result of Shannon 01:07:42.290 --> 01:07:45.090 align:middle line:84% was it's not just about data compression. 01:07:45.090 --> 01:07:48.810 align:middle line:84% But when you're transmitting messages at long distances, 01:07:48.810 --> 01:07:51.910 align:middle line:84% inevitably there are errors in your transmission. 01:07:51.910 --> 01:07:55.190 align:middle line:84% So what happens when some of the symbols that you're 01:07:55.190 --> 01:07:59.230 align:middle line:84% sending along the telephone wire get corrupted? 01:07:59.230 --> 01:08:01.870 align:middle line:84% Is there a way to recover and decode 01:08:01.870 --> 01:08:05.070 align:middle line:84% what was the original thing you'd meant to send? 01:08:05.070 --> 01:08:07.830 align:middle line:84% It'll turn out that you can do amazing things. 01:08:07.830 --> 01:08:09.590 align:middle line:84% These are really the building blocks 01:08:09.590 --> 01:08:11.350 align:middle line:84% of the starting point of what's called 01:08:11.350 --> 01:08:14.110 align:middle line:84% coding theory that has a lot of applications 01:08:14.110 --> 01:08:16.390 align:middle line:84% and other areas of computer science. 01:08:16.390 --> 01:08:18.710 align:middle line:84% And in fact, one of the biggest challenges 01:08:18.710 --> 01:08:20.350 align:middle line:84% for things like quantum computation 01:08:20.350 --> 01:08:23.430 align:middle line:84% is really how do we do these schemes in quantum computers, 01:08:23.430 --> 01:08:24.310 align:middle line:90% too? 01:08:24.310 --> 01:08:25.770 align:middle line:90% So we'll stop early. 01:08:25.770 --> 01:08:30.170 align:middle line:84% And as I told you in the beginning, come to office hours, 01:08:30.170 --> 01:08:30.970 align:middle line:90% ask questions. 01:08:30.970 --> 01:08:31.870 align:middle line:90% We're here. 01:08:31.870 --> 01:08:33.590 align:middle line:84% Don't take your foot off the gas pedal 01:08:33.590 --> 01:08:36.510 align:middle line:84% quite yet because there's still a lot of coursework to go. 01:08:36.510 --> 01:08:38.609 align:middle line:84% But otherwise, I'll see you next week, 01:08:38.609 --> 01:08:41.109 align:middle line:84% where we'll continue our coding theory in information theory 01:08:41.109 --> 01:08:42.149 align:middle line:90% units. 01:08:42.149 --> 01:08:44.220 align:middle line:90% So see you then. 01:08:44.220 --> 01:08:50.000 align:middle line:90%