WEBVTT 00:00:00.000 --> 00:00:03.381 align:middle line:84% [SQUEAKING] [RUSTLING] [CLICKING] 00:00:03.381 --> 00:00:10.927 align:middle line:90% 00:00:10.927 --> 00:00:12.260 align:middle line:90% ANKUR MOITRA: Let's get started. 00:00:12.260 --> 00:00:13.740 align:middle line:90% So we're in the home stretch. 00:00:13.740 --> 00:00:17.960 align:middle line:84% We've got one more of these topics in coding theory. 00:00:17.960 --> 00:00:20.740 align:middle line:84% Peter will be covering a related topic next week, 00:00:20.740 --> 00:00:23.360 align:middle line:84% but that'll be a bit of a different flavor. 00:00:23.360 --> 00:00:27.040 align:middle line:84% So today, we're going to cover one of the other great results 00:00:27.040 --> 00:00:31.040 align:middle line:84% in coding theory, which has to do with error correction. 00:00:31.040 --> 00:00:34.280 align:middle line:84% So far, we've been exclusively talking about compression, 00:00:34.280 --> 00:00:37.440 align:middle line:84% how you can take a message and reduce the number of bits 00:00:37.440 --> 00:00:39.800 align:middle line:84% you need to transmit in order to reliably 00:00:39.800 --> 00:00:43.360 align:middle line:84% be able to recover what was the original message. 00:00:43.360 --> 00:00:46.180 align:middle line:84% Now, today, we're going to talk about a very realistic scenario, 00:00:46.180 --> 00:00:48.760 align:middle line:84% which is when the way that you're transmitting 00:00:48.760 --> 00:00:50.920 align:middle line:90% introduces errors. 00:00:50.920 --> 00:00:54.320 align:middle line:84% So there's a very simple model, which is a good way 00:00:54.320 --> 00:01:02.640 align:middle line:84% to get started, which is called the binary symmetric channel, 00:01:02.640 --> 00:01:04.860 align:middle line:90% or BSC for short. 00:01:04.860 --> 00:01:06.380 align:middle line:84% The way that this channel works is 00:01:06.380 --> 00:01:10.500 align:middle line:84% that every time you input a bit, which is either 0 or 1, 00:01:10.500 --> 00:01:15.140 align:middle line:84% then it has some probability of flipping the outcome, the output 00:01:15.140 --> 00:01:16.340 align:middle line:90% of the bit. 00:01:16.340 --> 00:01:21.842 align:middle line:84% So this BSC channel is going to be governed by some parameter p. 00:01:21.842 --> 00:01:23.300 align:middle line:84% And it's going to have the property 00:01:23.300 --> 00:01:26.820 align:middle line:84% that, with p probability, it ends up 00:01:26.820 --> 00:01:30.460 align:middle line:90% flipping the input and output. 00:01:30.460 --> 00:01:32.700 align:middle line:84% And with the remaining probability, 1 minus p, 00:01:32.700 --> 00:01:36.780 align:middle line:84% it just sends through whatever bit you sent along the channel. 00:01:36.780 --> 00:01:39.660 align:middle line:84% The trouble is that you don't know when and where 00:01:39.660 --> 00:01:41.940 align:middle line:90% these errors have occurred. 00:01:41.940 --> 00:01:44.140 align:middle line:84% So the question is, you can think about p just 00:01:44.140 --> 00:01:47.780 align:middle line:84% for concreteness as being something like 1/3. 00:01:47.780 --> 00:01:50.400 align:middle line:84% If I'm trying to send a message along the channel to you, 00:01:50.400 --> 00:01:52.900 align:middle line:84% and I've already done my compression to get it down 00:01:52.900 --> 00:01:56.180 align:middle line:84% to the shortest possible string of 0's and 1's, now, 00:01:56.180 --> 00:02:00.080 align:middle line:84% when I send it along to you, and each bit gets flipped 00:02:00.080 --> 00:02:02.720 align:middle line:84% independently with probability of 1/3, 00:02:02.720 --> 00:02:06.240 align:middle line:84% what hope do you have of recovering the original message? 00:02:06.240 --> 00:02:08.160 align:middle line:84% And what are schemes you can design 00:02:08.160 --> 00:02:12.760 align:middle line:84% that combat the fact that you're anticipating there'll be errors? 00:02:12.760 --> 00:02:14.680 align:middle line:84% So just to be clear, we're talking 00:02:14.680 --> 00:02:24.200 align:middle line:84% about each bit is flipped independently 00:02:24.200 --> 00:02:28.720 align:middle line:90% with this probability p. 00:02:28.720 --> 00:02:32.800 align:middle line:84% And now we can ask all of the natural questions 00:02:32.800 --> 00:02:36.240 align:middle line:84% about communication and the presence of errors. 00:02:36.240 --> 00:02:38.040 align:middle line:84% So the main thing we care about, which 00:02:38.040 --> 00:02:40.900 align:middle line:84% will be what we'll address today, 00:02:40.900 --> 00:02:49.540 align:middle line:84% is it still possible to transmit information reliably? 00:02:49.540 --> 00:02:55.520 align:middle line:90% 00:02:55.520 --> 00:03:01.950 align:middle line:84% So let's think about what this question means 00:03:01.950 --> 00:03:05.650 align:middle line:84% with some simple ideas for how to combat the errors. 00:03:05.650 --> 00:03:08.310 align:middle line:84% So one simple example you can come up 00:03:08.310 --> 00:03:11.710 align:middle line:84% with for how to transmit along a noisy channel 00:03:11.710 --> 00:03:13.980 align:middle line:84% is to use what are called repetition codes. 00:03:13.980 --> 00:03:19.230 align:middle line:90% 00:03:19.230 --> 00:03:25.270 align:middle line:84% So here, the idea is that I take my original message. 00:03:25.270 --> 00:03:29.750 align:middle line:84% Let's say it's a string of 0's and 1's. 00:03:29.750 --> 00:03:32.750 align:middle line:84% And what I'm going to do for what I'll transmit along 00:03:32.750 --> 00:03:38.950 align:middle line:84% the channel is I'm just going to keep repeating each input 00:03:38.950 --> 00:03:43.430 align:middle line:84% bit a certain number of times so that, even if some of them 00:03:43.430 --> 00:03:45.870 align:middle line:90% get corrupted, maybe it's OK. 00:03:45.870 --> 00:03:48.470 align:middle line:84% So I'll take my first bit, which is a 1, 00:03:48.470 --> 00:03:53.190 align:middle line:84% and I'm going to repeat it T times. 00:03:53.190 --> 00:03:55.670 align:middle line:84% And I'll do the same thing for my next bit. 00:03:55.670 --> 00:03:56.470 align:middle line:90% That's a 0. 00:03:56.470 --> 00:04:00.370 align:middle line:90% I'm going to repeat it T times. 00:04:00.370 --> 00:04:01.930 align:middle line:84% And I'll do the same thing for all 00:04:01.930 --> 00:04:05.638 align:middle line:84% of the bits in my original message. 00:04:05.638 --> 00:04:07.550 align:middle line:84% So now you can see that, already, 00:04:07.550 --> 00:04:09.890 align:middle line:84% the way that I'm combating the fact that the channel is 00:04:09.890 --> 00:04:12.930 align:middle line:84% introducing errors is that I'm increasing 00:04:12.930 --> 00:04:14.970 align:middle line:90% the length of my transmission. 00:04:14.970 --> 00:04:18.570 align:middle line:84% I'm padding it with extra information in such a way 00:04:18.570 --> 00:04:20.970 align:middle line:84% that, hopefully, I can detect and correct 00:04:20.970 --> 00:04:24.450 align:middle line:84% whatever errors the channel has introduced. 00:04:24.450 --> 00:04:27.290 align:middle line:84% So the name of the game for today will be, 00:04:27.290 --> 00:04:29.410 align:middle line:84% what are the best possible schemes 00:04:29.410 --> 00:04:31.850 align:middle line:90% for how to pad the extra-- 00:04:31.850 --> 00:04:34.010 align:middle line:84% with extra information, in such a way 00:04:34.010 --> 00:04:38.690 align:middle line:84% that I can reliably figure out what my original message was? 00:04:38.690 --> 00:04:42.130 align:middle line:84% So just to think about this, so let's say 00:04:42.130 --> 00:04:46.090 align:middle line:84% that the original message had length k. 00:04:46.090 --> 00:04:51.850 align:middle line:84% Then what I can do is I claim a good choice for how many times 00:04:51.850 --> 00:04:53.450 align:middle line:90% to repeat each message. 00:04:53.450 --> 00:04:58.750 align:middle line:84% There's maybe something like 100 log k. 00:04:58.750 --> 00:05:00.830 align:middle line:84% So my original message was length k. 00:05:00.830 --> 00:05:06.670 align:middle line:84% And then each bit, I repeat 100 log k times. 00:05:06.670 --> 00:05:10.170 align:middle line:84% So intuitively, if I took this transmission 00:05:10.170 --> 00:05:13.550 align:middle line:84% and I sent it through the channel, some of these 1's 00:05:13.550 --> 00:05:16.270 align:middle line:84% get flipped to 0's independently. 00:05:16.270 --> 00:05:19.950 align:middle line:84% Some of these 0's get flipped to 1's independently. 00:05:19.950 --> 00:05:22.870 align:middle line:84% How would you try and decode what you receive to figure out 00:05:22.870 --> 00:05:25.830 align:middle line:90% what the original message is? 00:05:25.830 --> 00:05:26.460 align:middle line:90% Any ideas? 00:05:26.460 --> 00:05:32.870 align:middle line:90% 00:05:32.870 --> 00:05:34.843 align:middle line:84% And keep in mind, just to be concrete, 00:05:34.843 --> 00:05:37.510 align:middle line:84% you can think about the failure of the flip probability as being 00:05:37.510 --> 00:05:39.310 align:middle line:90% something like 1/3. 00:05:39.310 --> 00:05:42.390 align:middle line:84% So maybe a bit is more likely to come through 00:05:42.390 --> 00:05:44.670 align:middle line:84% in the clear without being flipped 00:05:44.670 --> 00:05:47.190 align:middle line:90% than it is of being flipped. 00:05:47.190 --> 00:05:50.590 align:middle line:84% So if I sent you this long, padded transmission, 00:05:50.590 --> 00:05:53.810 align:middle line:84% how do you think I'd figure out what the first symbol is 00:05:53.810 --> 00:05:56.610 align:middle line:90% based on what I receive? 00:05:56.610 --> 00:05:57.220 align:middle line:90% Any ideas? 00:05:57.220 --> 00:06:00.390 align:middle line:90% 00:06:00.390 --> 00:06:00.890 align:middle line:90% Yeah? 00:06:00.890 --> 00:06:02.170 align:middle line:84% AUDIENCE: The majority of the block. 00:06:02.170 --> 00:06:03.837 align:middle line:84% ANKUR MOITRA: Perfect, the majority vote 00:06:03.837 --> 00:06:08.410 align:middle line:84% because, intuitively, if I'm repeating this T times, 00:06:08.410 --> 00:06:11.330 align:middle line:84% then I would expect that the number of bit 00:06:11.330 --> 00:06:14.850 align:middle line:84% flips that I have is around T over 3. 00:06:14.850 --> 00:06:16.690 align:middle line:84% And so that means the majority of them 00:06:16.690 --> 00:06:21.170 align:middle line:84% should actually correspond to what my original bit was. 00:06:21.170 --> 00:06:22.850 align:middle line:84% In fact, what I care about is not 00:06:22.850 --> 00:06:27.010 align:middle line:84% just what the expected number of bit flips are, 00:06:27.010 --> 00:06:29.810 align:middle line:84% but I care about getting bounds on how likely it 00:06:29.810 --> 00:06:32.810 align:middle line:84% is that my decoding rule would fail. 00:06:32.810 --> 00:06:35.770 align:middle line:84% So what kind of tool that we've talked about earlier in class 00:06:35.770 --> 00:06:37.530 align:middle line:90% might be relevant here? 00:06:37.530 --> 00:06:42.770 align:middle line:84% So I started off with bit flip probability, 1/3. 00:06:42.770 --> 00:06:45.090 align:middle line:84% And then I sent this bit through the channel 00:06:45.090 --> 00:06:47.210 align:middle line:90% with a repetition code. 00:06:47.210 --> 00:06:51.320 align:middle line:84% Now, my chance of getting the wrong bit for my majority vote 00:06:51.320 --> 00:06:56.680 align:middle line:84% decoding is the chance that the expected number of flips is 1/3. 00:06:56.680 --> 00:06:59.240 align:middle line:84% But really, the true number of flips is much larger 00:06:59.240 --> 00:07:01.480 align:middle line:90% because it's larger than 1/2. 00:07:01.480 --> 00:07:05.040 align:middle line:84% That's the only way I would get the wrong decoding for this bit 00:07:05.040 --> 00:07:09.480 align:middle line:84% is if I ended up with half errors instead of 1/3. 00:07:09.480 --> 00:07:12.320 align:middle line:84% So what kind of tool that we've covered earlier in class 00:07:12.320 --> 00:07:15.120 align:middle line:84% might be useful to get a bound on the failure 00:07:15.120 --> 00:07:18.270 align:middle line:84% probability, the chance that my decoder fails here? 00:07:18.270 --> 00:07:21.760 align:middle line:90% 00:07:21.760 --> 00:07:22.280 align:middle line:90% Yeah? 00:07:22.280 --> 00:07:23.360 align:middle line:90% AUDIENCE: Markov. 00:07:23.360 --> 00:07:26.040 align:middle line:84% ANKUR MOITRA: Markov would do it. 00:07:26.040 --> 00:07:27.860 align:middle line:90% So I could certainly use Markov. 00:07:27.860 --> 00:07:30.400 align:middle line:84% That's an example of a tail bound. 00:07:30.400 --> 00:07:32.800 align:middle line:84% Now, one problem, though, is that Markov 00:07:32.800 --> 00:07:36.560 align:middle line:84% uses very little information about the random variables. 00:07:36.560 --> 00:07:39.560 align:middle line:84% So if I did this, what I care about is, 00:07:39.560 --> 00:07:44.320 align:middle line:84% what's the probability that I fail for this bit? 00:07:44.320 --> 00:07:48.040 align:middle line:84% What's the probability that I fail for this bit and so on? 00:07:48.040 --> 00:07:50.580 align:middle line:84% And it turns out that if you used Markov, 00:07:50.580 --> 00:07:53.980 align:middle line:84% if you wanted to argue that, with good chance, 00:07:53.980 --> 00:07:57.380 align:middle line:84% I fail on none of these bits and I use the union bound, 00:07:57.380 --> 00:07:59.860 align:middle line:84% I would end up having to pad the length of the message 00:07:59.860 --> 00:08:03.237 align:middle line:84% by a factor of k instead of log k. 00:08:03.237 --> 00:08:04.820 align:middle line:84% So are there any ideas for how I could 00:08:04.820 --> 00:08:07.220 align:middle line:84% hope to get sharper bounds on how much I need 00:08:07.220 --> 00:08:09.000 align:middle line:90% to increase the padding by? 00:08:09.000 --> 00:08:09.500 align:middle line:90% Yeah. 00:08:09.500 --> 00:08:10.060 align:middle line:90% AUDIENCE: Chernoff. 00:08:10.060 --> 00:08:11.018 align:middle line:90% ANKUR MOITRA: Chernoff. 00:08:11.018 --> 00:08:12.020 align:middle line:90% That's right. 00:08:12.020 --> 00:08:16.380 align:middle line:84% So the key point is that, if I have this repetition code 00:08:16.380 --> 00:08:22.093 align:middle line:84% and I do it 100 log k times, I can use the Chernoff bound. 00:08:22.093 --> 00:08:24.260 align:middle line:84% So you remember, the way to think about the Chernoff 00:08:24.260 --> 00:08:28.660 align:middle line:84% bound is that the error exponent goes down exponentially 00:08:28.660 --> 00:08:31.380 align:middle line:84% in the length and the number of independent samples 00:08:31.380 --> 00:08:32.740 align:middle line:90% you're taking. 00:08:32.740 --> 00:08:37.020 align:middle line:84% So the probability that I have too many flips, 00:08:37.020 --> 00:08:40.380 align:middle line:84% that I was expecting 1/3 flips and I get 1/2, 00:08:40.380 --> 00:08:43.860 align:middle line:84% is something that goes down exponentially with log k. 00:08:43.860 --> 00:08:46.120 align:middle line:84% So if I choose this constant in front of it, 00:08:46.120 --> 00:08:49.800 align:middle line:84% the point is that my failure probability is something like 1 00:08:49.800 --> 00:08:51.320 align:middle line:90% over k squared. 00:08:51.320 --> 00:08:54.080 align:middle line:84% And then I can say that I don't fail on this bit, 00:08:54.080 --> 00:08:56.580 align:middle line:84% I don't fail on this bit, I don't fail on this bit, 00:08:56.580 --> 00:08:59.000 align:middle line:90% I don't fail on any of the bits. 00:08:59.000 --> 00:09:00.640 align:middle line:90% So that's exactly right. 00:09:00.640 --> 00:09:03.360 align:middle line:84% If we use the Chernoff bound, we could then 00:09:03.360 --> 00:09:07.920 align:middle line:84% take the majority vote in each one of these blocks, 00:09:07.920 --> 00:09:09.560 align:middle line:84% and we could show that the majority 00:09:09.560 --> 00:09:15.480 align:middle line:84% vote succeeds with probability, let's say, 00:09:15.480 --> 00:09:17.260 align:middle line:90% 1 minus little O of 1. 00:09:17.260 --> 00:09:19.600 align:middle line:84% So maybe I fail with some probability 1 00:09:19.600 --> 00:09:22.200 align:middle line:90% over k, something like that. 00:09:22.200 --> 00:09:24.920 align:middle line:84% So we're not going to fully proof these things right here 00:09:24.920 --> 00:09:29.000 align:middle line:84% because we're going to design a much, much better coding scheme. 00:09:29.000 --> 00:09:30.760 align:middle line:84% So repetition codes, the first time 00:09:30.760 --> 00:09:34.240 align:middle line:84% you see this problem of communicating on a noisy channel 00:09:34.240 --> 00:09:36.600 align:middle line:84% is the first scheme you come up with. 00:09:36.600 --> 00:09:39.200 align:middle line:84% But the big surprise is that you can do way better 00:09:39.200 --> 00:09:41.305 align:middle line:84% than just repeating your message. 00:09:41.305 --> 00:09:42.680 align:middle line:84% So what I'm going to do right now 00:09:42.680 --> 00:09:45.580 align:middle line:84% is I'm going to tell you about, what are the desiderata? 00:09:45.580 --> 00:09:48.620 align:middle line:84% So what are the things we want out of our code? 00:09:48.620 --> 00:09:51.880 align:middle line:84% And then we'll talk about Shannon's noisy coding theorem, 00:09:51.880 --> 00:09:54.660 align:middle line:90% and we'll prove it. 00:09:54.660 --> 00:09:57.020 align:middle line:84% So I claim this code is pretty good, but not 00:09:57.020 --> 00:10:00.980 align:middle line:84% really the kind of thing that we're aiming for. 00:10:00.980 --> 00:10:03.020 align:middle line:84% So let's make precise some of the things 00:10:03.020 --> 00:10:05.160 align:middle line:84% that we talked about intuitively. 00:10:05.160 --> 00:10:09.460 align:middle line:84% So we can talk about the rate of a code. 00:10:09.460 --> 00:10:13.500 align:middle line:84% The rate just describes, however long my transmission 00:10:13.500 --> 00:10:17.340 align:middle line:84% is, how long is that compared to how much information I'm 00:10:17.340 --> 00:10:19.100 align:middle line:90% actually transmitting? 00:10:19.100 --> 00:10:23.540 align:middle line:84% So in particular, what it is, is the length 00:10:23.540 --> 00:10:31.840 align:middle line:84% of the message divided by the length of the transmission. 00:10:31.840 --> 00:10:38.420 align:middle line:90% 00:10:38.420 --> 00:10:41.470 align:middle line:84% So in particular, what do we get for repetition codes? 00:10:41.470 --> 00:10:45.060 align:middle line:90% 00:10:45.060 --> 00:10:47.690 align:middle line:84% How good are they, according to this desiderata 00:10:47.690 --> 00:10:49.610 align:middle line:90% of having a good rate? 00:10:49.610 --> 00:10:52.650 align:middle line:84% Well, we can check that the rate that we get 00:10:52.650 --> 00:11:01.570 align:middle line:84% is 1 over 100 log k because my original length of my message 00:11:01.570 --> 00:11:02.970 align:middle line:90% was k. 00:11:02.970 --> 00:11:07.370 align:middle line:84% My new length of my message is k times T. And I set T to be 00:11:07.370 --> 00:11:09.010 align:middle line:90% 100 log k. 00:11:09.010 --> 00:11:12.210 align:middle line:84% So I get this 1 over T type of rate. 00:11:12.210 --> 00:11:16.470 align:middle line:84% So the trouble with this is that this rate, 00:11:16.470 --> 00:11:20.090 align:middle line:84% if you look at it, as the length of my message k 00:11:20.090 --> 00:11:24.210 align:middle line:84% goes to infinity, my rate is going to 0. 00:11:24.210 --> 00:11:25.350 align:middle line:90% So that's the question. 00:11:25.350 --> 00:11:28.970 align:middle line:84% That's the main question, really, is that necessary? 00:11:28.970 --> 00:11:32.290 align:middle line:84% So when we send longer and longer transmissions, 00:11:32.290 --> 00:11:35.450 align:middle line:84% does it mean that I'm communicating asymptotically 00:11:35.450 --> 00:11:38.410 align:middle line:84% very little negligible information when 00:11:38.410 --> 00:11:40.090 align:middle line:90% I'm sending things to you? 00:11:40.090 --> 00:11:42.850 align:middle line:84% Or is it possible to achieve a constant rate that's 00:11:42.850 --> 00:11:46.310 align:middle line:84% independent of the length of the message? 00:11:46.310 --> 00:11:49.170 align:middle line:84% So this came as a pretty big surprise. 00:11:49.170 --> 00:11:51.440 align:middle line:84% And this is our main guiding question for today. 00:11:51.440 --> 00:11:54.110 align:middle line:90% 00:11:54.110 --> 00:12:02.270 align:middle line:84% Let's ask, can we make the probability of getting an error? 00:12:02.270 --> 00:12:05.390 align:middle line:84% And what I mean by that is the probability that we 00:12:05.390 --> 00:12:09.230 align:middle line:84% don't successfully decode the message we started from. 00:12:09.230 --> 00:12:11.960 align:middle line:84% I want the probability of my error going to 0. 00:12:11.960 --> 00:12:15.950 align:middle line:90% 00:12:15.950 --> 00:12:21.050 align:middle line:84% But I want to keep the rate constant. 00:12:21.050 --> 00:12:24.750 align:middle line:90% 00:12:24.750 --> 00:12:26.710 align:middle line:84% And here, what I mean by going to 0 00:12:26.710 --> 00:12:30.510 align:middle line:84% is we're taking the limit as the length of my input message, k, 00:12:30.510 --> 00:12:32.430 align:middle line:90% as going to infinity. 00:12:32.430 --> 00:12:34.470 align:middle line:84% So is it possible to do much better 00:12:34.470 --> 00:12:36.750 align:middle line:90% than these repetition codes? 00:12:36.750 --> 00:12:40.570 align:middle line:84% We got error, probability of error, was going to 0. 00:12:40.570 --> 00:12:41.670 align:middle line:90% That's fine. 00:12:41.670 --> 00:12:44.410 align:middle line:84% But we were transmitting very little information 00:12:44.410 --> 00:12:45.390 align:middle line:90% at the end of the day. 00:12:45.390 --> 00:12:48.570 align:middle line:90% 00:12:48.570 --> 00:12:52.890 align:middle line:84% So, in fact, I need to make some other notions precise too 00:12:52.890 --> 00:12:58.810 align:middle line:84% because we have to talk about, what is our coding scheme? 00:12:58.810 --> 00:13:01.670 align:middle line:84% See, for the repetition codes, I did this all informally. 00:13:01.670 --> 00:13:06.290 align:middle line:84% But really, there are two pieces to the puzzle. 00:13:06.290 --> 00:13:09.010 align:middle line:84% One is the encoding, which is how 00:13:09.010 --> 00:13:12.610 align:middle line:84% you go from the message to what you actually transmit along 00:13:12.610 --> 00:13:13.610 align:middle line:90% the channel. 00:13:13.610 --> 00:13:16.290 align:middle line:84% It's the way that you're encoding the information. 00:13:16.290 --> 00:13:18.610 align:middle line:84% Repetition is a very simple way to do that, just 00:13:18.610 --> 00:13:20.850 align:middle line:90% by repeating myself. 00:13:20.850 --> 00:13:22.450 align:middle line:84% And the other piece of the puzzle 00:13:22.450 --> 00:13:25.770 align:middle line:84% is the decoding, which takes whatever 00:13:25.770 --> 00:13:28.610 align:middle line:84% is the output of the channel, where some of the bits 00:13:28.610 --> 00:13:29.790 align:middle line:90% have been flipped. 00:13:29.790 --> 00:13:33.050 align:middle line:84% And we try to go back to the message we started from. 00:13:33.050 --> 00:13:35.570 align:middle line:84% So in our case, the encoding was repetition 00:13:35.570 --> 00:13:38.010 align:middle line:84% and the decoding was majority vote. 00:13:38.010 --> 00:13:41.090 align:middle line:84% But in general, we could think about very different kinds 00:13:41.090 --> 00:13:42.796 align:middle line:90% of schemes. 00:13:42.796 --> 00:13:57.590 align:middle line:84% So we'll say that an Rn, n encoding is a function, 00:13:57.590 --> 00:14:02.670 align:middle line:84% and which we'll write as Enc, that goes from 0, 00:14:02.670 --> 00:14:09.190 align:middle line:90% 1 to the Rn to 0, 1 to the n. 00:14:09.190 --> 00:14:11.530 align:middle line:90% So R here represents the rate. 00:14:11.530 --> 00:14:15.790 align:middle line:84% And it's something that, for us, will be between 0 and 1. 00:14:15.790 --> 00:14:18.770 align:middle line:84% And you can see that I'm going from a shorter message. 00:14:18.770 --> 00:14:21.310 align:middle line:84% This is my k right here is the Rn. 00:14:21.310 --> 00:14:24.830 align:middle line:84% This is the input that I want to send to another party. 00:14:24.830 --> 00:14:27.310 align:middle line:84% And then I decide how I'm going to transmit it 00:14:27.310 --> 00:14:30.690 align:middle line:84% along the channel using a longer message. 00:14:30.690 --> 00:14:33.990 align:middle line:90% So it goes from Rn to n. 00:14:33.990 --> 00:14:36.990 align:middle line:84% So that's the first part, is that this is our encoding. 00:14:36.990 --> 00:14:40.980 align:middle line:84% And you can see that R right here is literally 00:14:40.980 --> 00:14:43.420 align:middle line:90% the rate of my code. 00:14:43.420 --> 00:14:45.740 align:middle line:84% So that's a parameter I'll care about. 00:14:45.740 --> 00:14:55.590 align:middle line:84% And moreover, an Rn, n decoding is also a function. 00:14:55.590 --> 00:14:59.740 align:middle line:90% 00:14:59.740 --> 00:15:06.380 align:middle line:84% All it does is decode, goes from a received message, Rn, 00:15:06.380 --> 00:15:10.380 align:middle line:84% back to-- oh, sorry, n, back to what 00:15:10.380 --> 00:15:14.000 align:middle line:84% we think the person was transmitting to begin with. 00:15:14.000 --> 00:15:17.205 align:middle line:90% 00:15:17.205 --> 00:15:24.900 align:middle line:84% Now, the key point is how these two functions encode and decode, 00:15:24.900 --> 00:15:27.380 align:middle line:84% work in the presence of errors and bit 00:15:27.380 --> 00:15:29.420 align:middle line:84% flips, like when we send it along the binary 00:15:29.420 --> 00:15:31.140 align:middle line:90% symmetric channel. 00:15:31.140 --> 00:15:39.960 align:middle line:84% So in particular, we'll be interested 00:15:39.960 --> 00:15:41.350 align:middle line:90% in the following quantity. 00:15:41.350 --> 00:15:44.000 align:middle line:90% 00:15:44.000 --> 00:15:46.900 align:middle line:84% This is a bit of a mouthful, so let me write it down, 00:15:46.900 --> 00:15:49.320 align:middle line:84% and then we'll parse it together. 00:15:49.320 --> 00:15:52.280 align:middle line:84% So if you give me two parameters, R, your target rate, 00:15:52.280 --> 00:15:57.640 align:middle line:84% and n, which is how long you want the transmission to be, 00:15:57.640 --> 00:16:02.540 align:middle line:84% well, I'm going to look at the min over all Rn, 00:16:02.540 --> 00:16:08.240 align:middle line:84% n encoding and decoding functions 00:16:08.240 --> 00:16:14.800 align:middle line:84% of the max over messages m, which are n, 00:16:14.800 --> 00:16:22.280 align:middle line:84% 0, 1 to the Rn of the probability that m tilde is not 00:16:22.280 --> 00:16:26.040 align:middle line:84% equal to m, where we think about it according 00:16:26.040 --> 00:16:27.840 align:middle line:90% to the following picture. 00:16:27.840 --> 00:16:30.280 align:middle line:84% So we start off with our message. 00:16:30.280 --> 00:16:33.600 align:middle line:84% That gets turned into some transmission, which 00:16:33.600 --> 00:16:35.840 align:middle line:84% is just the encoding of my message 00:16:35.840 --> 00:16:38.460 align:middle line:90% given my encoding function. 00:16:38.460 --> 00:16:43.820 align:middle line:84% We send this through the binary symmetric channel. 00:16:43.820 --> 00:16:46.460 align:middle line:90% And we receive C tilde. 00:16:46.460 --> 00:16:50.540 align:middle line:84% We pass C tilde into the decoding function 00:16:50.540 --> 00:16:55.460 align:middle line:84% to get back what we think was the original message. 00:16:55.460 --> 00:16:58.820 align:middle line:84% And the question is, how often do we have the property 00:16:58.820 --> 00:17:01.620 align:middle line:90% that m equals m tilde? 00:17:01.620 --> 00:17:05.140 align:middle line:84% So this is a lot of notation, but it's really very natural. 00:17:05.140 --> 00:17:08.940 align:middle line:84% So inside right here, I just care about the probability 00:17:08.940 --> 00:17:11.460 align:middle line:84% that I successfully decode the message 00:17:11.460 --> 00:17:15.339 align:middle line:84% you were trying to send to me, despite the fact that the binary 00:17:15.339 --> 00:17:19.060 align:middle line:84% symmetric channel has introduced errors. 00:17:19.060 --> 00:17:23.579 align:middle line:84% Now, I want this probability, the probability that I fail 00:17:23.579 --> 00:17:27.680 align:middle line:84% to get the correct message, to be small, 00:17:27.680 --> 00:17:31.460 align:middle line:84% irrespective of what message you're sending me. 00:17:31.460 --> 00:17:34.340 align:middle line:84% So I look at how bad the failure probability can 00:17:34.340 --> 00:17:37.160 align:middle line:84% be in decoding and the worst case 00:17:37.160 --> 00:17:39.680 align:middle line:90% over all of my input messages. 00:17:39.680 --> 00:17:42.680 align:middle line:84% And then this latter thing is I want 00:17:42.680 --> 00:17:45.880 align:middle line:84% to create an encoding and decoding scheme that is 00:17:45.880 --> 00:17:48.800 align:middle line:90% good for all possible messages. 00:17:48.800 --> 00:17:51.840 align:middle line:84% So I don't want to choose an arbitrary encoding, decoding 00:17:51.840 --> 00:17:54.320 align:middle line:84% function, like these repetition codes. 00:17:54.320 --> 00:17:57.000 align:middle line:84% But I want to figure out, what is actually 00:17:57.000 --> 00:18:01.040 align:middle line:84% the best encoding-decoding pair in terms of, 00:18:01.040 --> 00:18:03.160 align:middle line:90% you fixed what the rate is. 00:18:03.160 --> 00:18:06.720 align:middle line:84% How small can I get this failure probability to be? 00:18:06.720 --> 00:18:11.000 align:middle line:84% If I fix the rate to be a constant, like 1/2 or 1/3, 00:18:11.000 --> 00:18:14.040 align:middle line:84% can I get the failure probability for all messages 00:18:14.040 --> 00:18:18.160 align:middle line:84% to be going to 0 as n goes to infinity? 00:18:18.160 --> 00:18:20.440 align:middle line:90% So this is a bit of a mouthful. 00:18:20.440 --> 00:18:22.080 align:middle line:84% Let me give you a moment to parse 00:18:22.080 --> 00:18:25.120 align:middle line:84% this thing, but are there any questions about the terminology 00:18:25.120 --> 00:18:29.240 align:middle line:90% or what I'm asking for here? 00:18:29.240 --> 00:18:32.120 align:middle line:84% Give me a thumbs up if this makes sense. 00:18:32.120 --> 00:18:33.640 align:middle line:90% Awesome. 00:18:33.640 --> 00:18:43.550 align:middle line:84% All right, so let me tell you Shannon's second main theorem, 00:18:43.550 --> 00:18:47.630 align:middle line:84% Shannon's noisy coding theorem, also proven 00:18:47.630 --> 00:18:49.510 align:middle line:90% around the same time. 00:18:49.510 --> 00:18:52.110 align:middle line:84% And it asserts two things, which are upper 00:18:52.110 --> 00:18:55.270 align:middle line:90% and lower bounds that meet. 00:18:55.270 --> 00:18:59.710 align:middle line:84% So what I claim is that if the rate that you're trying to hit 00:18:59.710 --> 00:19:03.710 align:middle line:84% is larger than 1 minus entropy of p, 00:19:03.710 --> 00:19:09.990 align:middle line:84% then what I claim is that lambda star, Rn, the best 00:19:09.990 --> 00:19:13.070 align:middle line:84% you can do for encoding and decoding functions, 00:19:13.070 --> 00:19:19.950 align:middle line:84% that error rate goes to 1 as n goes to infinity. 00:19:19.950 --> 00:19:22.190 align:middle line:84% That's the first part for the theorem. 00:19:22.190 --> 00:19:23.490 align:middle line:90% Let me state the other part. 00:19:23.490 --> 00:19:25.630 align:middle line:84% We'll talk about what this tells us 00:19:25.630 --> 00:19:28.510 align:middle line:84% and how to connect it back to our intuition. 00:19:28.510 --> 00:19:35.250 align:middle line:84% So on the other side, if R is less than 1 minus entropy of p, 00:19:35.250 --> 00:19:41.370 align:middle line:84% then what I claim is that lambda star, R, n, 00:19:41.370 --> 00:19:47.050 align:middle line:84% is going to go to 0 as n goes to infinity. 00:19:47.050 --> 00:19:48.990 align:middle line:84% So let's parse these things together 00:19:48.990 --> 00:19:52.310 align:middle line:84% because, to really unwrap what this theorem is stating, 00:19:52.310 --> 00:19:56.170 align:middle line:84% we have to go back to this min-max formulation. 00:19:56.170 --> 00:19:59.010 align:middle line:84% So what this is saying is think about p 00:19:59.010 --> 00:20:01.850 align:middle line:84% as being a third in our binary symmetric channel. 00:20:01.850 --> 00:20:04.110 align:middle line:84% H is the binary entropy function. 00:20:04.110 --> 00:20:07.850 align:middle line:84% We talked about what that expression is, minus p log 00:20:07.850 --> 00:20:12.330 align:middle line:90% p plus 1 minus p log 1 minus p. 00:20:12.330 --> 00:20:16.210 align:middle line:84% So you can just plug-in for p is 1/3 and compute numerically what 00:20:16.210 --> 00:20:17.610 align:middle line:90% entropy of p is. 00:20:17.610 --> 00:20:19.570 align:middle line:90% It's some constant. 00:20:19.570 --> 00:20:21.210 align:middle line:84% And now the point is that if you're 00:20:21.210 --> 00:20:28.370 align:middle line:84% trying to transmit at a denser rate than 1 minus entropy of p, 00:20:28.370 --> 00:20:30.210 align:middle line:90% you're going to fail. 00:20:30.210 --> 00:20:31.990 align:middle line:90% You're going to fail why? 00:20:31.990 --> 00:20:35.190 align:middle line:84% Because no matter what encoding-decoding 00:20:35.190 --> 00:20:39.030 align:middle line:84% pair you choose, there will be messages 00:20:39.030 --> 00:20:42.630 align:middle line:84% where the probability that you successfully decode the message 00:20:42.630 --> 00:20:44.190 align:middle line:90% is actually going to 0. 00:20:44.190 --> 00:20:48.790 align:middle line:84% The probability that m tilde is not equal to m is going to 1. 00:20:48.790 --> 00:20:52.150 align:middle line:84% So when you're trying to transmit at too high a rate, 00:20:52.150 --> 00:20:56.030 align:middle line:84% you never decode the correct message. 00:20:56.030 --> 00:20:58.870 align:middle line:84% And on the other side, if the rate 00:20:58.870 --> 00:21:00.350 align:middle line:84% at which you're trying to transmit 00:21:00.350 --> 00:21:04.910 align:middle line:84% is less than the same quantity, 1 minus entropy of p, then 00:21:04.910 --> 00:21:08.370 align:middle line:84% what this means is that there is a good encoding-decoding pair. 00:21:08.370 --> 00:21:10.190 align:middle line:84% We'll have to talk about what this pair is 00:21:10.190 --> 00:21:14.270 align:middle line:84% and how does it work, that the error probability goes 00:21:14.270 --> 00:21:18.150 align:middle line:90% to 0 as n goes to infinity. 00:21:18.150 --> 00:21:20.630 align:middle line:84% So this is a really striking theorem 00:21:20.630 --> 00:21:23.830 align:middle line:84% because, usually, what you-- what the terminology 00:21:23.830 --> 00:21:31.210 align:middle line:84% is that you call 1 minus entropy of p, the capacity, 00:21:31.210 --> 00:21:34.570 align:middle line:84% it's just some kind of rate that the channel admits 00:21:34.570 --> 00:21:39.130 align:middle line:84% for how densely you can communicate with someone else. 00:21:39.130 --> 00:21:42.290 align:middle line:84% In fact, what's amazing is that these types of theorems 00:21:42.290 --> 00:21:45.850 align:middle line:84% are true not just for the binary symmetric channel, 00:21:45.850 --> 00:21:50.650 align:middle line:84% but if you invent your favorite stochastic channel, something 00:21:50.650 --> 00:21:53.690 align:middle line:84% which maybe erases symbols or deletes them, or has 00:21:53.690 --> 00:21:56.450 align:middle line:84% some hybrid model between these things, 00:21:56.450 --> 00:21:59.330 align:middle line:84% they always have a well-defined capacity, 00:21:59.330 --> 00:22:01.690 align:middle line:84% that if you're above the capacity, 00:22:01.690 --> 00:22:04.530 align:middle line:90% your error rate is going to 1. 00:22:04.530 --> 00:22:06.010 align:middle line:84% If you're below the capacity, you 00:22:06.010 --> 00:22:08.330 align:middle line:84% can get the error rate going to 0. 00:22:08.330 --> 00:22:10.830 align:middle line:84% But the interesting quirk is that, in a lot of cases, 00:22:10.830 --> 00:22:13.530 align:middle line:84% the channel you write down, even though we know there is some 00:22:13.530 --> 00:22:16.570 align:middle line:84% line in the sand that you cannot cross, 00:22:16.570 --> 00:22:19.170 align:middle line:84% we don't actually know numerically what it is 00:22:19.170 --> 00:22:23.090 align:middle line:84% because it turns into an extremely hard math problem, 00:22:23.090 --> 00:22:26.130 align:middle line:84% to give an explicit characterization the way that we 00:22:26.130 --> 00:22:28.922 align:middle line:90% have one here. 00:22:28.922 --> 00:22:33.000 align:middle line:90% So this is a lot to digest. 00:22:33.000 --> 00:22:36.600 align:middle line:84% Let's try and get some intuition for this. 00:22:36.600 --> 00:22:40.260 align:middle line:84% So I told you H of p is the binary entropy. 00:22:40.260 --> 00:22:48.840 align:middle line:84% So recall that the entropy of p is just minus p log base 00:22:48.840 --> 00:22:55.960 align:middle line:84% 2 of p minus 1 minus p log base 2 of 1 minus p. 00:22:55.960 --> 00:22:58.560 align:middle line:90% 00:22:58.560 --> 00:23:03.520 align:middle line:84% So in particular, what this function looks like is, 00:23:03.520 --> 00:23:08.840 align:middle line:84% if you choose p is 1/2, what's going to happen is that 00:23:08.840 --> 00:23:15.120 align:middle line:90% your entropy of p is 1. 00:23:15.120 --> 00:23:18.680 align:middle line:84% So what you can show is that this binary entropy function, 00:23:18.680 --> 00:23:22.840 align:middle line:90% it's this nice concave function. 00:23:22.840 --> 00:23:26.960 align:middle line:84% And its maximum value corresponds to p is 1/2, 00:23:26.960 --> 00:23:32.060 align:middle line:84% at which point, it equals the value 1. 00:23:32.060 --> 00:23:34.180 align:middle line:84% So let's figure out what Shannon is saying here 00:23:34.180 --> 00:23:35.440 align:middle line:90% and see if it makes sense. 00:23:35.440 --> 00:23:39.420 align:middle line:84% So I have my binary symmetric channel. 00:23:39.420 --> 00:23:45.800 align:middle line:84% And if p is 1, then I can compute the channel capacity. 00:23:45.800 --> 00:23:47.160 align:middle line:90% That's 1 minus 1. 00:23:47.160 --> 00:23:48.860 align:middle line:90% So that's 0. 00:23:48.860 --> 00:23:52.740 align:middle line:84% So Shannon is telling me that you can't communicate at any 00:23:52.740 --> 00:23:56.860 align:middle line:84% positive rate when your flip probability for your binary 00:23:56.860 --> 00:23:58.140 align:middle line:90% symmetric channel is 1/2. 00:23:58.140 --> 00:24:01.580 align:middle line:90% 00:24:01.580 --> 00:24:02.530 align:middle line:90% Does that make sense? 00:24:02.530 --> 00:24:05.060 align:middle line:90% 00:24:05.060 --> 00:24:07.580 align:middle line:90% Let's think about that one. 00:24:07.580 --> 00:24:11.100 align:middle line:84% So why is that reasonable, that when p is 1/2, 00:24:11.100 --> 00:24:15.980 align:middle line:84% that I can't actually transmit information to another party? 00:24:15.980 --> 00:24:16.480 align:middle line:90% Yeah. 00:24:16.480 --> 00:24:18.272 align:middle line:84% AUDIENCE: There's no reasonable way for you 00:24:18.272 --> 00:24:21.595 align:middle line:84% to tell every bit that comes through [INAUDIBLE]. 00:24:21.595 --> 00:24:22.720 align:middle line:90% ANKUR MOITRA: That's right. 00:24:22.720 --> 00:24:25.100 align:middle line:84% So if you just look back to my definition of the binary 00:24:25.100 --> 00:24:29.680 align:middle line:84% symmetric channel, well, 1 minus p is 1/2, p is 1/2. 00:24:29.680 --> 00:24:31.720 align:middle line:84% So what that means is you sit there 00:24:31.720 --> 00:24:35.120 align:middle line:84% and you receive my first symbol, which is a 0 or 1. 00:24:35.120 --> 00:24:37.860 align:middle line:84% It has the same probability of being 0 and 1, 00:24:37.860 --> 00:24:42.000 align:middle line:84% irrespective of whether you are sending a 0 or 1 to begin with. 00:24:42.000 --> 00:24:45.440 align:middle line:84% So literally, my channel just doesn't look at the message, 00:24:45.440 --> 00:24:47.720 align:middle line:84% and it just spits out random bits. 00:24:47.720 --> 00:24:51.240 align:middle line:84% Of course, there's no way to communicate. 00:24:51.240 --> 00:24:55.480 align:middle line:84% When p is 1/3, Shannon's theorem is telling us that we can do way 00:24:55.480 --> 00:24:59.040 align:middle line:84% better than the repetition code because this will be a constant 00:24:59.040 --> 00:25:01.880 align:middle line:90% because we'll be somewhere here. 00:25:01.880 --> 00:25:03.800 align:middle line:84% And whatever that value is numerically 00:25:03.800 --> 00:25:07.840 align:middle line:84% is the fixed line in the sand, we can communicate at that rate 00:25:07.840 --> 00:25:09.080 align:middle line:90% and know better. 00:25:09.080 --> 00:25:12.080 align:middle line:84% And we don't have something like the repetition code, where 00:25:12.080 --> 00:25:13.500 align:middle line:90% the rate is going to 0. 00:25:13.500 --> 00:25:15.600 align:middle line:84% It's actually a universal constant, 00:25:15.600 --> 00:25:18.800 align:middle line:84% even though the error probability is going to 0. 00:25:18.800 --> 00:25:21.140 align:middle line:84% Let me test your intuition again. 00:25:21.140 --> 00:25:26.500 align:middle line:84% See, what's interesting is that this entropy function, H of p, 00:25:26.500 --> 00:25:32.420 align:middle line:84% is symmetric around p equals 1/2 because, when I substitute p 00:25:32.420 --> 00:25:36.180 align:middle line:84% with 1 minus p, it doesn't change the expression. 00:25:36.180 --> 00:25:39.980 align:middle line:84% So the same thing is true for 1 minus entropy of p. 00:25:39.980 --> 00:25:43.500 align:middle line:84% So what Shannon's theorem is telling me is that, 00:25:43.500 --> 00:25:47.420 align:middle line:84% when my bit flip probability is 2/3, 00:25:47.420 --> 00:25:50.560 align:middle line:84% I can also communicate at some positive rate. 00:25:50.560 --> 00:25:53.220 align:middle line:90% Why is that intuitive? 00:25:53.220 --> 00:25:57.580 align:middle line:84% If I use something like the repetition code majority vote 00:25:57.580 --> 00:26:00.440 align:middle line:84% doesn't work anymore, but something else does. 00:26:00.440 --> 00:26:03.000 align:middle line:84% What would work if my bit flip probability were two thirds? 00:26:03.000 --> 00:26:03.500 align:middle line:90% Yeah? 00:26:03.500 --> 00:26:04.700 align:middle line:90% AUDIENCE: [INAUDIBLE] minority? 00:26:04.700 --> 00:26:05.658 align:middle line:90% ANKUR MOITRA: Minority. 00:26:05.658 --> 00:26:06.580 align:middle line:90% Exactly. 00:26:06.580 --> 00:26:10.180 align:middle line:84% All I do is I'm thinking that there'll be 2/3 flips here, 00:26:10.180 --> 00:26:14.672 align:middle line:84% and so I take the opposite of the majority vote. 00:26:14.672 --> 00:26:17.380 align:middle line:90% So this theorem is pretty deep. 00:26:17.380 --> 00:26:19.880 align:middle line:84% It takes a little bit to wrap your head around it. 00:26:19.880 --> 00:26:21.340 align:middle line:90% But so far, it makes sense. 00:26:21.340 --> 00:26:23.930 align:middle line:90% p is 1/2 makes sense. 00:26:23.930 --> 00:26:26.090 align:middle line:84% The fact that it's symmetric and p-- 00:26:26.090 --> 00:26:30.362 align:middle line:84% interchanging p and 1 minus p makes sense. 00:26:30.362 --> 00:26:34.450 align:middle line:84% And we're going to prove this theorem today. 00:26:34.450 --> 00:26:39.260 align:middle line:84% And let me give you the proof right here. 00:26:39.260 --> 00:26:44.810 align:middle line:90% 00:26:44.810 --> 00:26:47.850 align:middle line:90% I'll put proof in quotes. 00:26:47.850 --> 00:26:51.250 align:middle line:84% So, really, the question is like, repetition codes didn't 00:26:51.250 --> 00:26:53.850 align:middle line:90% work for my encoding function. 00:26:53.850 --> 00:26:57.530 align:middle line:84% What should I use for my encoding function? 00:26:57.530 --> 00:27:02.980 align:middle line:84% So what you do is you let encoding be a random function. 00:27:02.980 --> 00:27:07.650 align:middle line:90% 00:27:07.650 --> 00:27:08.630 align:middle line:90% And that's it. 00:27:08.630 --> 00:27:10.050 align:middle line:90% That's the proof. 00:27:10.050 --> 00:27:12.370 align:middle line:84% The entire rest of today's lecture 00:27:12.370 --> 00:27:15.650 align:middle line:84% will be removing the quotes around this proof 00:27:15.650 --> 00:27:17.490 align:middle line:84% because, in order to actually analyze 00:27:17.490 --> 00:27:19.130 align:middle line:84% what happens when we choose encoding 00:27:19.130 --> 00:27:20.843 align:middle line:84% to be a random function, well, we're 00:27:20.843 --> 00:27:22.510 align:middle line:84% going to have to get into a lot of tools 00:27:22.510 --> 00:27:25.950 align:middle line:90% from the probabilistic method. 00:27:25.950 --> 00:27:28.950 align:middle line:90% So this is our strategy. 00:27:28.950 --> 00:27:34.093 align:middle line:84% Let me introduce a bit more about what the strategy is. 00:27:34.093 --> 00:27:35.510 align:middle line:84% So what we're going to do is we're 00:27:35.510 --> 00:27:41.230 align:middle line:84% going to prove this second statement first, 00:27:41.230 --> 00:27:45.310 align:middle line:84% which is, of course, the positive statement 00:27:45.310 --> 00:27:49.510 align:middle line:84% that we really can communicate at some positive rate. 00:27:49.510 --> 00:27:52.510 align:middle line:90% So let's prove 2 first. 00:27:52.510 --> 00:27:57.150 align:middle line:84% And let me be more precise about what I mean by encode. 00:27:57.150 --> 00:28:01.630 align:middle line:84% So I'm going to choose M, capital M, 00:28:01.630 --> 00:28:09.910 align:middle line:84% equals 2 times 2 to the Rn strings 00:28:09.910 --> 00:28:18.980 align:middle line:84% from my domain, 0, 1 to the n, uniformly at random. 00:28:18.980 --> 00:28:22.170 align:middle line:90% 00:28:22.170 --> 00:28:23.990 align:middle line:84% So if you think back to what's going on, 00:28:23.990 --> 00:28:26.170 align:middle line:90% what is an encoding function? 00:28:26.170 --> 00:28:29.650 align:middle line:84% Well it's mapping each of the possible messages-- 00:28:29.650 --> 00:28:32.250 align:middle line:84% and there are 2 to the Rn of these-- 00:28:32.250 --> 00:28:35.330 align:middle line:90% to strings, and 0, 1 to the n. 00:28:35.330 --> 00:28:38.410 align:middle line:84% And what I'm doing here, it sort of looks like I'm looking-- 00:28:38.410 --> 00:28:42.770 align:middle line:84% I'm choosing the images of what those messages would be. 00:28:42.770 --> 00:28:47.330 align:middle line:84% There's 2 to the Rn messages that I could want to encode, 00:28:47.330 --> 00:28:48.810 align:middle line:84% and I'm just going to encode them 00:28:48.810 --> 00:28:53.490 align:middle line:84% with a uniformly random string from 0, 1 to the n. 00:28:53.490 --> 00:28:55.250 align:middle line:84% For each one of the messages, I just 00:28:55.250 --> 00:28:59.130 align:middle line:84% choose the bits in its output completely at random. 00:28:59.130 --> 00:29:01.970 align:middle line:84% I have this extra factor of 2 for reasons 00:29:01.970 --> 00:29:05.890 align:middle line:84% that will become clear much later because this scheme 00:29:05.890 --> 00:29:08.210 align:middle line:90% by itself won't quite work. 00:29:08.210 --> 00:29:11.390 align:middle line:84% We'll have to remove some of these strings to really get by, 00:29:11.390 --> 00:29:13.090 align:middle line:90% and we'll talk about that later. 00:29:13.090 --> 00:29:16.050 align:middle line:84% But that's at least the intuition. 00:29:16.050 --> 00:29:22.270 align:middle line:84% And here I'm going to define the codebook, script C. That's just 00:29:22.270 --> 00:29:25.990 align:middle line:84% all of these different strings that I've chosen, 00:29:25.990 --> 00:29:31.950 align:middle line:90% C1 up to Cm, all right? 00:29:31.950 --> 00:29:38.410 align:middle line:84% So now, let me tell you one more key definition. 00:29:38.410 --> 00:29:42.310 align:middle line:84% And then we can get to at least the statement 00:29:42.310 --> 00:29:43.540 align:middle line:90% of some of the key lemmas. 00:29:43.540 --> 00:29:51.350 align:middle line:90% 00:29:51.350 --> 00:29:55.230 align:middle line:84% So the last key definition I need 00:29:55.230 --> 00:30:04.750 align:middle line:84% is the notion of a ring of width gamma around some point C. 00:30:04.750 --> 00:30:08.470 align:middle line:84% So C here is a vector in 0, 1 to the n. 00:30:08.470 --> 00:30:13.590 align:middle line:84% And I'm going to let this ring be the set of all C tildes, 00:30:13.590 --> 00:30:16.883 align:middle line:90% such that the Hamming distance-- 00:30:16.883 --> 00:30:18.800 align:middle line:84% I'll tell you what I mean by that in a second, 00:30:18.800 --> 00:30:20.740 align:middle line:84% in case you haven't seen this term. 00:30:20.740 --> 00:30:26.940 align:middle line:84% The Hamming distance between C and C tilde minus np 00:30:26.940 --> 00:30:29.860 align:middle line:90% is, at most, gamma n. 00:30:29.860 --> 00:30:34.420 align:middle line:84% All right, so let me be precise, so the Hamming distance is just 00:30:34.420 --> 00:30:45.020 align:middle line:84% the number of coordinates, i, where Ci is not equal to C 00:30:45.020 --> 00:30:46.980 align:middle line:90% tilde i. 00:30:46.980 --> 00:30:51.500 align:middle line:84% So my ring is going to be centered at some vector C. And C 00:30:51.500 --> 00:30:55.540 align:middle line:84% right here, remember, is a vector in 0, 1 to the n. 00:30:55.540 --> 00:30:58.880 align:middle line:84% And my ring is going to be a subset of 0, 1 to the n. 00:30:58.880 --> 00:31:01.500 align:middle line:84% It's the set of all C tildes that 00:31:01.500 --> 00:31:04.380 align:middle line:90% are the right distance from C. 00:31:04.380 --> 00:31:07.900 align:middle line:84% In particular, I look at how many coordinates between C and C 00:31:07.900 --> 00:31:09.780 align:middle line:90% tilde are different. 00:31:09.780 --> 00:31:12.540 align:middle line:84% np is maybe the expected number of coordinates 00:31:12.540 --> 00:31:14.620 align:middle line:84% where they would be different because I'm 00:31:14.620 --> 00:31:18.240 align:middle line:84% sending C along the binary symmetric channel, 00:31:18.240 --> 00:31:20.400 align:middle line:84% and it flips each bit independently 00:31:20.400 --> 00:31:22.120 align:middle line:90% with probability p. 00:31:22.120 --> 00:31:26.240 align:middle line:84% So in expectation, if C tilde is the output of my channel, 00:31:26.240 --> 00:31:30.160 align:middle line:84% I would expect the Hamming distance to be n times p. 00:31:30.160 --> 00:31:32.662 align:middle line:84% And here I just added a bit of slack. 00:31:32.662 --> 00:31:35.120 align:middle line:84% You should think of gamma as being something really, really 00:31:35.120 --> 00:31:39.420 align:middle line:84% tiny that's going to 0, and it just adds a little bit of slack 00:31:39.420 --> 00:31:43.560 align:middle line:84% because my ring now just looks like not the set of things 00:31:43.560 --> 00:31:47.040 align:middle line:84% at some fixed distance, but within some tolerance 00:31:47.040 --> 00:31:48.800 align:middle line:90% of that distance. 00:31:48.800 --> 00:31:52.060 align:middle line:84% So the way to think about it as a ring, just pictorially, 00:31:52.060 --> 00:31:54.520 align:middle line:90% is that I have my vector C here. 00:31:54.520 --> 00:32:00.800 align:middle line:84% And then I'm looking at some disk around it of radius pn 00:32:00.800 --> 00:32:04.000 align:middle line:84% in terms of how many coordinates are difference between C and C 00:32:04.000 --> 00:32:05.040 align:middle line:90% tilde. 00:32:05.040 --> 00:32:10.360 align:middle line:84% And then I flatten this ring a little bit by some width, gamma. 00:32:10.360 --> 00:32:13.560 align:middle line:84% And this shaded region in between them 00:32:13.560 --> 00:32:18.800 align:middle line:84% is exactly what the set ring gamma around C is. 00:32:18.800 --> 00:32:22.420 align:middle line:84% So it has a center and it has a thickness. 00:32:22.420 --> 00:32:25.740 align:middle line:90% So any questions about that? 00:32:25.740 --> 00:32:27.820 align:middle line:90% This is just the definition. 00:32:27.820 --> 00:32:32.180 align:middle line:84% All right, and so now let me tell you 00:32:32.180 --> 00:32:40.140 align:middle line:84% the key lemma, which is going to tell us-- 00:32:40.140 --> 00:32:43.300 align:middle line:84% it'll be the key for proving the second part of Shannon's 00:32:43.300 --> 00:32:44.460 align:middle line:90% theorem. 00:32:44.460 --> 00:32:48.460 align:middle line:84% And it'll even tell us what the decoding function should be. 00:32:48.460 --> 00:32:50.760 align:middle line:84% We'll have to work up towards proving this lemma, 00:32:50.760 --> 00:32:53.780 align:middle line:84% but I can already state the lemma right now. 00:32:53.780 --> 00:33:07.220 align:middle line:84% So here's the key lemma, the probability that if we take C, 00:33:07.220 --> 00:33:11.340 align:middle line:84% and we send it through our binary symmetric channel, 00:33:11.340 --> 00:33:20.120 align:middle line:84% and we get out C tilde, we want it to satisfy two conditions. 00:33:20.120 --> 00:33:22.520 align:middle line:84% We want it to satisfy the condition 1, 00:33:22.520 --> 00:33:30.650 align:middle line:84% that C tilde belongs to the ring around Ci of some width, gamma. 00:33:30.650 --> 00:33:33.440 align:middle line:90% 00:33:33.440 --> 00:33:35.800 align:middle line:84% And we want that C tilde does not 00:33:35.800 --> 00:33:41.440 align:middle line:84% belong to any other ring of width gamma 00:33:41.440 --> 00:33:47.160 align:middle line:84% around any other Cj for all j not equal to i. 00:33:47.160 --> 00:33:54.920 align:middle line:84% Well, I claim that this is at least 1 minus epsilon. 00:33:54.920 --> 00:34:01.120 align:middle line:84% And this will be true for appropriate choice of gamma 00:34:01.120 --> 00:34:03.320 align:middle line:90% and as n goes to infinity. 00:34:03.320 --> 00:34:04.700 align:middle line:90% So let's parse this thing. 00:34:04.700 --> 00:34:07.200 align:middle line:84% This still requires a lot of digestion 00:34:07.200 --> 00:34:09.280 align:middle line:90% to understand what's going on. 00:34:09.280 --> 00:34:12.260 align:middle line:84% So I'm punting on the question, what is gamma? 00:34:12.260 --> 00:34:14.690 align:middle line:90% We'll deal with that later. 00:34:14.690 --> 00:34:16.630 align:middle line:84% And all of these things I care about 00:34:16.630 --> 00:34:22.670 align:middle line:84% is really asymptotically as a function of n, all right? 00:34:22.670 --> 00:34:26.270 align:middle line:84% And, ultimately, I'm going to want this epsilon, actually, 00:34:26.270 --> 00:34:27.197 align:middle line:90% to go to 0 too. 00:34:27.197 --> 00:34:28.989 align:middle line:84% So you should think of epsilon as something 00:34:28.989 --> 00:34:31.949 align:middle line:84% that's going to 0 as n goes to infinity. 00:34:31.949 --> 00:34:33.449 align:middle line:84% So I haven't told you what gamma is. 00:34:33.449 --> 00:34:34.949 align:middle line:90% We'll deal with that later. 00:34:34.949 --> 00:34:39.630 align:middle line:90% But each of these C's are-- 00:34:39.630 --> 00:34:42.469 align:middle line:84% actually, sorry, I should say Ci here. 00:34:42.469 --> 00:34:45.150 align:middle line:84% Each of these Ci's are the messages 00:34:45.150 --> 00:34:48.790 align:middle line:84% from my codebook, script C. Remember, 00:34:48.790 --> 00:34:51.270 align:middle line:90% there's capital M of them. 00:34:51.270 --> 00:34:54.070 align:middle line:84% And what I'm saying is, when I choose my encoding 00:34:54.070 --> 00:34:58.750 align:middle line:84% scheme that way, where each of these Ci's are a random length n 00:34:58.750 --> 00:35:00.790 align:middle line:84% string, what I'm saying is, when I 00:35:00.790 --> 00:35:04.310 align:middle line:84% pass that message through the binary symmetric channel, 00:35:04.310 --> 00:35:08.110 align:middle line:84% and introduce flips, and get C tilde, well, 00:35:08.110 --> 00:35:12.810 align:middle line:84% with high probability, Ci belong-- 00:35:12.810 --> 00:35:15.890 align:middle line:90% C tilde belongs to Ci's ring. 00:35:15.890 --> 00:35:19.010 align:middle line:90% And he belongs to no other ring. 00:35:19.010 --> 00:35:21.250 align:middle line:84% So if this event really did happen 00:35:21.250 --> 00:35:23.970 align:middle line:84% with all but vanishing probability, 00:35:23.970 --> 00:35:27.010 align:middle line:84% I would be in great shape because it would give me 00:35:27.010 --> 00:35:29.090 align:middle line:90% my decoding algorithm. 00:35:29.090 --> 00:35:32.090 align:middle line:84% So once I've chosen this encoding function, 00:35:32.090 --> 00:35:36.450 align:middle line:84% where each of the messages, the 2 to the Rn of them, 00:35:36.450 --> 00:35:40.170 align:middle line:84% goes to a random image, one of these Ci's, then 00:35:40.170 --> 00:35:42.870 align:middle line:84% all I have to do is I pass my Ci through the channel, 00:35:42.870 --> 00:35:45.490 align:middle line:84% I receive C tilde, and my decoder 00:35:45.490 --> 00:35:47.570 align:middle line:84% is just going to check one-by-one 00:35:47.570 --> 00:35:51.130 align:middle line:84% each of the messages in my codebook, which 00:35:51.130 --> 00:35:53.930 align:middle line:90% ring C tilde belongs to. 00:35:53.930 --> 00:35:56.650 align:middle line:84% And the point is that the only ring it will belong to 00:35:56.650 --> 00:36:00.370 align:middle line:84% is the actual message I started from if both of these conditions 00:36:00.370 --> 00:36:01.810 align:middle line:90% hold. 00:36:01.810 --> 00:36:05.350 align:middle line:84% So that's the main point, is that, if this event happens, 00:36:05.350 --> 00:36:06.890 align:middle line:90% I'm golden. 00:36:06.890 --> 00:36:08.650 align:middle line:84% So does everyone see why decoding 00:36:08.650 --> 00:36:11.630 align:middle line:90% would succeed if this happened? 00:36:11.630 --> 00:36:13.530 align:middle line:90% Does this make sense? 00:36:13.530 --> 00:36:14.030 align:middle line:90% Yeah? 00:36:14.030 --> 00:36:18.270 align:middle line:90% 00:36:18.270 --> 00:36:21.190 align:middle line:84% I saw less vigorous head-nodding. 00:36:21.190 --> 00:36:22.350 align:middle line:90% Yes? 00:36:22.350 --> 00:36:24.390 align:middle line:90% No? 00:36:24.390 --> 00:36:28.170 align:middle line:84% All right, so we're going to have to prove this lemma. 00:36:28.170 --> 00:36:32.212 align:middle line:84% That's one of the key things we're going to prove today. 00:36:32.212 --> 00:36:33.670 align:middle line:84% A bunch of these things, I'm trying 00:36:33.670 --> 00:36:36.350 align:middle line:84% to do the calculations in a way that 00:36:36.350 --> 00:36:43.260 align:middle line:84% makes them as unpainful as possible, but there are limits. 00:36:43.260 --> 00:36:45.693 align:middle line:90% 00:36:45.693 --> 00:36:47.110 align:middle line:84% So the first thing I want to do is 00:36:47.110 --> 00:36:49.990 align:middle line:84% I want to tell you a back of the envelope calculation 00:36:49.990 --> 00:36:53.870 align:middle line:84% because all these things, like this binary entropy, H of p, 00:36:53.870 --> 00:36:55.870 align:middle line:90% is going to have to show up. 00:36:55.870 --> 00:36:58.910 align:middle line:84% And somehow it shows up from the combinatorics when we look 00:36:58.910 --> 00:37:01.590 align:middle line:90% at the size of these rings. 00:37:01.590 --> 00:37:04.590 align:middle line:84% So let me tell you a back of the envelope calculation, which 00:37:04.590 --> 00:37:07.750 align:middle line:90% will be very useful for us. 00:37:07.750 --> 00:37:11.290 align:middle line:84% And you can make all of this precise, 00:37:11.290 --> 00:37:16.210 align:middle line:84% but let's do this a little bit more heuristically. 00:37:16.210 --> 00:37:19.330 align:middle line:84% So we know from Stirling's approximation 00:37:19.330 --> 00:37:25.170 align:middle line:84% that n factorial behaves like n over e to the n times 00:37:25.170 --> 00:37:28.153 align:middle line:90% square root of 2 pi n. 00:37:28.153 --> 00:37:29.570 align:middle line:84% Now, I'll tell you right now, what 00:37:29.570 --> 00:37:33.490 align:middle line:84% I want to do is I want to figure out what the size of the rings 00:37:33.490 --> 00:37:34.290 align:middle line:90% are. 00:37:34.290 --> 00:37:36.570 align:middle line:84% So I want to use Stirling's approximation 00:37:36.570 --> 00:37:41.650 align:middle line:84% to figure out some of expression for a binomial coefficient. 00:37:41.650 --> 00:37:44.610 align:middle line:84% And that's where the entropy is going to pop out. 00:37:44.610 --> 00:37:47.410 align:middle line:84% Now, I can tell you right now that, for Stirling's 00:37:47.410 --> 00:37:50.870 align:middle line:84% approximation, when we apply it to the thing I'm about to do, 00:37:50.870 --> 00:37:52.290 align:middle line:90% this term doesn't matter. 00:37:52.290 --> 00:37:54.070 align:middle line:84% So if you want, you can just ignore it. 00:37:54.070 --> 00:37:56.130 align:middle line:90% I'll still keep track of it. 00:37:56.130 --> 00:37:59.450 align:middle line:90% But let's progress. 00:37:59.450 --> 00:38:01.090 align:middle line:90% So let's move on. 00:38:01.090 --> 00:38:05.330 align:middle line:84% So what I really care about is understanding the behavior of n 00:38:05.330 --> 00:38:07.990 align:middle line:90% choose pn. 00:38:07.990 --> 00:38:13.170 align:middle line:84% This makes sense because I start off with a message which is, 00:38:13.170 --> 00:38:14.510 align:middle line:90% say, all 0's. 00:38:14.510 --> 00:38:16.030 align:middle line:84% Let's say that's the transmission 00:38:16.030 --> 00:38:18.070 align:middle line:90% that I send along the channel. 00:38:18.070 --> 00:38:20.830 align:middle line:84% And what I could care about is, how many 00:38:20.830 --> 00:38:24.270 align:middle line:90% outputs are reasonably likely? 00:38:24.270 --> 00:38:27.310 align:middle line:90% Certainly, if my trend-- 00:38:27.310 --> 00:38:30.310 align:middle line:84% my flip probability on the channel is 1/3, 00:38:30.310 --> 00:38:34.010 align:middle line:84% then I should expect that my output started off with no 1's. 00:38:34.010 --> 00:38:36.230 align:middle line:84% And after I send it through the channel, 00:38:36.230 --> 00:38:39.070 align:middle line:84% maybe I end up with n over three 1's. 00:38:39.070 --> 00:38:41.070 align:middle line:90% That's just p times n. 00:38:41.070 --> 00:38:43.630 align:middle line:84% And what I care about is how many 00:38:43.630 --> 00:38:47.030 align:middle line:84% outputs I'm reasonably likely to get from my channel. 00:38:47.030 --> 00:38:50.050 align:middle line:84% So I care about, what are the asymptotics of this quantity? 00:38:50.050 --> 00:38:54.150 align:middle line:84% And I can use the fact that I can express the binomial 00:38:54.150 --> 00:38:57.710 align:middle line:90% as ratios of factorials. 00:38:57.710 --> 00:38:59.430 align:middle line:84% And I'll get out-- when I plug-in 00:38:59.430 --> 00:39:03.190 align:middle line:84% Stirling's approximation, I'll get this mess of an expression 00:39:03.190 --> 00:39:04.950 align:middle line:90% that I can simplify. 00:39:04.950 --> 00:39:07.160 align:middle line:90% So I get n over e to the n. 00:39:07.160 --> 00:39:11.660 align:middle line:90% I get np over e to the np. 00:39:11.660 --> 00:39:19.140 align:middle line:84% I get n times 1 minus p over e to the n times 1 minus p. 00:39:19.140 --> 00:39:21.340 align:middle line:84% And then I get some other stuff that I already 00:39:21.340 --> 00:39:24.150 align:middle line:84% told you isn't going to matter, but let me keep track of it 00:39:24.150 --> 00:39:24.650 align:middle line:90% anyways. 00:39:24.650 --> 00:39:36.030 align:middle line:90% 00:39:36.030 --> 00:39:40.220 align:middle line:84% And this is not mysterious because this binomial is just 00:39:40.220 --> 00:39:47.220 align:middle line:84% n factorial over pn factorial times 1 minus pn factorial. 00:39:47.220 --> 00:39:50.360 align:middle line:84% And these are the leading order terms because these are much, 00:39:50.360 --> 00:39:52.260 align:middle line:90% much smaller. 00:39:52.260 --> 00:39:55.860 align:middle line:84% Now, what I can do is I can cancel things out because this n 00:39:55.860 --> 00:40:00.820 align:middle line:84% to the n cancels with the n to the np times n to the n of 1 00:40:00.820 --> 00:40:01.700 align:middle line:90% minus 1p-- 00:40:01.700 --> 00:40:03.540 align:middle line:90% 1 minus p. 00:40:03.540 --> 00:40:07.280 align:middle line:84% This e to the n cancels with the e's down here. 00:40:07.280 --> 00:40:12.200 align:middle line:84% The only thing that'll be left over is I'll get 1 over p 00:40:12.200 --> 00:40:19.360 align:middle line:84% to the np times 1 minus p to the 1 minus pn. 00:40:19.360 --> 00:40:23.680 align:middle line:84% And then I can cancel a bunch of these expressions if I want to. 00:40:23.680 --> 00:40:29.080 align:middle line:84% This is square root 2 pi np times 1 minus p. 00:40:29.080 --> 00:40:30.580 align:middle line:90% So just some algebra. 00:40:30.580 --> 00:40:31.946 align:middle line:90% Nothing fancy so far. 00:40:31.946 --> 00:40:36.182 align:middle line:90% 00:40:36.182 --> 00:40:37.640 align:middle line:84% And now I'm in good shape because I 00:40:37.640 --> 00:40:39.580 align:middle line:90% can take this expression. 00:40:39.580 --> 00:40:42.480 align:middle line:84% And I can take the log and divide by n. 00:40:42.480 --> 00:40:49.200 align:middle line:84% So what this tells me is that 1 over n times log of n choose pn, 00:40:49.200 --> 00:40:53.520 align:middle line:84% well, this is exactly where the binary entropy pops out. 00:40:53.520 --> 00:41:02.560 align:middle line:84% So I get minus p log p minus 1 minus p log 1 minus p. 00:41:02.560 --> 00:41:06.540 align:middle line:84% And then all the other terms are much smaller order 00:41:06.540 --> 00:41:09.700 align:middle line:84% because they behave like log n over n. 00:41:09.700 --> 00:41:11.860 align:middle line:90% They're tiny compared to it. 00:41:11.860 --> 00:41:13.580 align:middle line:84% So really, what's going on is all 00:41:13.580 --> 00:41:16.340 align:middle line:84% of the terms that don't cancel when I take their log 00:41:16.340 --> 00:41:20.540 align:middle line:84% and divide by n, that's exactly where the binary entropy pops 00:41:20.540 --> 00:41:22.220 align:middle line:90% out. 00:41:22.220 --> 00:41:25.940 align:middle line:84% And that's why entropy shows up in Shannon's theorem. 00:41:25.940 --> 00:41:30.340 align:middle line:84% So this is really just saying that the limit as n 00:41:30.340 --> 00:41:33.980 align:middle line:84% goes to infinity of 1 over n times log 00:41:33.980 --> 00:41:38.580 align:middle line:84% of this binomial coefficient, n choose pn, 00:41:38.580 --> 00:41:42.620 align:middle line:84% behaves like the binary entropy of p. 00:41:42.620 --> 00:41:47.180 align:middle line:84% So this is a key algebraic fact that we're 00:41:47.180 --> 00:41:50.080 align:middle line:84% going to appeal to in our proof of correctness. 00:41:50.080 --> 00:41:54.300 align:middle line:84% And our proof that lemma 1 holds is just this fact. 00:41:54.300 --> 00:41:56.822 align:middle line:84% We're going to see a bunch of binomials in there, 00:41:56.822 --> 00:41:58.780 align:middle line:84% and I'm just going to think about them in terms 00:41:58.780 --> 00:42:00.411 align:middle line:90% of the binary entropy instead. 00:42:00.411 --> 00:42:05.440 align:middle line:90% 00:42:05.440 --> 00:42:08.420 align:middle line:84% So now let's go back to this lemma 1. 00:42:08.420 --> 00:42:10.320 align:middle line:84% And let me tell you something basic facts 00:42:10.320 --> 00:42:12.840 align:middle line:84% that are going to be key ingredients 00:42:12.840 --> 00:42:14.300 align:middle line:90% on the proof of lemma 1. 00:42:14.300 --> 00:42:28.800 align:middle line:90% 00:42:28.800 --> 00:42:30.880 align:middle line:84% This part is really just calculational 00:42:30.880 --> 00:42:33.080 align:middle line:90% to get to the proof of lemma 1. 00:42:33.080 --> 00:42:36.180 align:middle line:84% But let's state one of our key facts. 00:42:36.180 --> 00:42:44.020 align:middle line:84% Fact one, well, I claim that ring gamma of C, the size of it, 00:42:44.020 --> 00:42:47.720 align:middle line:84% the number of strings that are contained in this set, 00:42:47.720 --> 00:42:56.440 align:middle line:84% is bounded by 2 to the entropy of p plus or minus gamma times 00:42:56.440 --> 00:42:58.680 align:middle line:90% n. 00:42:58.680 --> 00:43:02.750 align:middle line:84% So all this is, is, really, it's related to this fact. 00:43:02.750 --> 00:43:06.310 align:middle line:84% Think about the case where gamma is 0 00:43:06.310 --> 00:43:08.270 align:middle line:90% for our definition of the ring. 00:43:08.270 --> 00:43:09.790 align:middle line:84% All that's happening is the number 00:43:09.790 --> 00:43:15.030 align:middle line:84% of things that are in our ring is n choose pn. 00:43:15.030 --> 00:43:17.910 align:middle line:84% And so our fact tells us that, in that case, when 00:43:17.910 --> 00:43:24.270 align:middle line:84% gamma equals 0, it behaves like 2 to the entropy of p times n. 00:43:24.270 --> 00:43:25.950 align:middle line:84% That's literally what this expression 00:43:25.950 --> 00:43:27.910 align:middle line:84% means when I multiply through by n 00:43:27.910 --> 00:43:33.230 align:middle line:84% and take everything in the exponent of 2. 00:43:33.230 --> 00:43:36.650 align:middle line:84% And as I take this thickness, gamma, to be something, 00:43:36.650 --> 00:43:39.510 align:middle line:84% but, ultimately, I'm going to care about this thickness going 00:43:39.510 --> 00:43:42.310 align:middle line:90% to 0 as n goes to infinity. 00:43:42.310 --> 00:43:46.190 align:middle line:84% It's only going to be some fudge factor in the exponent. 00:43:46.190 --> 00:43:49.230 align:middle line:84% So you can write down this fact more precisely. 00:43:49.230 --> 00:43:51.310 align:middle line:84% It's a bit tedious because it involves 00:43:51.310 --> 00:43:54.270 align:middle line:84% a lot of algebraic expressions, but this is really 00:43:54.270 --> 00:43:56.910 align:middle line:84% how the size of the ring behaves up 00:43:56.910 --> 00:43:58.590 align:middle line:84% to some fudge factor that depends 00:43:58.590 --> 00:44:00.330 align:middle line:90% on the width of the ring. 00:44:00.330 --> 00:44:03.250 align:middle line:84% It's 2 to the entropy of p times n. 00:44:03.250 --> 00:44:06.490 align:middle line:90% That's our first fact. 00:44:06.490 --> 00:44:16.090 align:middle line:84% And so now what I claim is that we can get into these questions 00:44:16.090 --> 00:44:18.610 align:middle line:90% like, how should I choose gamma? 00:44:18.610 --> 00:44:25.930 align:middle line:84% What I claim is that there is a gamma, such that 00:44:25.930 --> 00:44:34.330 align:middle line:84% with probability at least 1 minus epsilon over 2, 00:44:34.330 --> 00:44:41.330 align:middle line:84% p minus gamma n is a lower bound for the Hamming 00:44:41.330 --> 00:44:45.210 align:middle line:84% distance between C tilde and C. And p 00:44:45.210 --> 00:44:48.970 align:middle line:90% plus gamma n is an upper bound. 00:44:48.970 --> 00:44:51.890 align:middle line:84% So we can forget about how we're connecting this to coding theory 00:44:51.890 --> 00:44:52.430 align:middle line:90% right now. 00:44:52.430 --> 00:44:55.130 align:middle line:84% This is a purely probabilistic question. 00:44:55.130 --> 00:44:57.490 align:middle line:84% I take any message, C of length n, 00:44:57.490 --> 00:44:59.670 align:middle line:90% and I send it along the channel. 00:44:59.670 --> 00:45:02.590 align:middle line:90% And then I observe C tilde. 00:45:02.590 --> 00:45:04.510 align:middle line:84% And remember that C tilde just comes 00:45:04.510 --> 00:45:06.870 align:middle line:84% from flipping each bit independently 00:45:06.870 --> 00:45:07.990 align:middle line:90% with probability p. 00:45:07.990 --> 00:45:12.190 align:middle line:84% So I can care about how many flips were introduced. 00:45:12.190 --> 00:45:15.270 align:middle line:84% This is really just the standard coin-flipping question 00:45:15.270 --> 00:45:18.390 align:middle line:84% is I want bounds on the probability 00:45:18.390 --> 00:45:21.350 align:middle line:84% that the number of times I flipped the bit 00:45:21.350 --> 00:45:24.750 align:middle line:84% is larger than its expectation or is smaller 00:45:24.750 --> 00:45:26.670 align:middle line:90% than its expectation. 00:45:26.670 --> 00:45:30.830 align:middle line:84% And if my gamma is some appropriately-chosen slack, 00:45:30.830 --> 00:45:32.630 align:middle line:84% then I can guarantee that you're not 00:45:32.630 --> 00:45:36.510 align:middle line:84% outside of these bounds with all but some tiny failure 00:45:36.510 --> 00:45:38.710 align:middle line:90% probability. 00:45:38.710 --> 00:45:44.270 align:middle line:84% So all this is this follows because we've proven things 00:45:44.270 --> 00:45:45.490 align:middle line:90% very much like this. 00:45:45.490 --> 00:45:53.310 align:middle line:84% It follows from Chebyshev or the weak law of large numbers 00:45:53.310 --> 00:45:55.270 align:middle line:84% because, as n goes to infinity, you 00:45:55.270 --> 00:45:57.850 align:middle line:84% expect the empirical number of bit 00:45:57.850 --> 00:46:01.850 align:middle line:84% flips you get to be very, very close to the expected number, 00:46:01.850 --> 00:46:04.170 align:middle line:90% p times n. 00:46:04.170 --> 00:46:06.490 align:middle line:84% So with these two facts, we're now 00:46:06.490 --> 00:46:09.850 align:middle line:84% in good shape for trying to prove lemma 1. 00:46:09.850 --> 00:46:12.520 align:middle line:84% So let's put these things together and prove lemma 1. 00:46:12.520 --> 00:46:19.930 align:middle line:90% 00:46:19.930 --> 00:46:24.410 align:middle line:84% So how does the proof of lemma 1 go, our key lemma 00:46:24.410 --> 00:46:26.290 align:middle line:90% in Shannon's theorem? 00:46:26.290 --> 00:46:43.090 align:middle line:84% So first of all, from fact 2, we have the probability 00:46:43.090 --> 00:46:44.888 align:middle line:90% that C tilde-- 00:46:44.888 --> 00:46:45.930 align:middle line:90% how did I write it there? 00:46:45.930 --> 00:46:52.130 align:middle line:84% C tilde belongs to the ring for some appropriately-chosen gamma 00:46:52.130 --> 00:46:57.740 align:middle line:84% around Ci is at least 1 minus 1 over 2. 00:46:57.740 --> 00:47:02.160 align:middle line:84% So my failure probability is at most epsilon over 2. 00:47:02.160 --> 00:47:05.980 align:middle line:84% And now what we can do is let's bound the other probability. 00:47:05.980 --> 00:47:11.680 align:middle line:90% 00:47:11.680 --> 00:47:17.960 align:middle line:84% So let's bound the failure probability of the other event, 00:47:17.960 --> 00:47:23.200 align:middle line:84% namely that C tilde belongs to some other guy's ring. 00:47:23.200 --> 00:47:28.300 align:middle line:84% And so what we can do is we can fix j not equal to i, 00:47:28.300 --> 00:47:32.680 align:middle line:84% so one of the other code words in our codebook. 00:47:32.680 --> 00:47:38.840 align:middle line:84% And what we can do is, in that case, the probability 00:47:38.840 --> 00:47:47.120 align:middle line:84% that C tilde belongs to the ring of some width, gamma, around Cj, 00:47:47.120 --> 00:47:49.660 align:middle line:84% well, I claim that this is the same thing. 00:47:49.660 --> 00:47:52.160 align:middle line:84% This is one of the key tricks in here, 00:47:52.160 --> 00:47:53.900 align:middle line:84% is the same thing as the probability 00:47:53.900 --> 00:48:01.820 align:middle line:84% that Cj belongs to the ring of C tilde. 00:48:01.820 --> 00:48:04.220 align:middle line:84% So this looks like a pretty obvious statement 00:48:04.220 --> 00:48:09.100 align:middle line:84% because this notion of ring is just symmetric. 00:48:09.100 --> 00:48:14.020 align:middle line:84% It doesn't-- whether C tilde belongs to Ci's ring, 00:48:14.020 --> 00:48:17.240 align:middle line:84% that's the same thing as vice versa, 00:48:17.240 --> 00:48:20.580 align:middle line:90% if Ci belongs to C tilde's ring. 00:48:20.580 --> 00:48:22.140 align:middle line:84% But what's going on here is we have 00:48:22.140 --> 00:48:24.900 align:middle line:84% to go back to how exactly we chose the codebook 00:48:24.900 --> 00:48:29.360 align:middle line:84% to see why this is important, is that a priori, 00:48:29.360 --> 00:48:32.300 align:middle line:84% this might seem like a fairly complicated event 00:48:32.300 --> 00:48:37.820 align:middle line:84% to analyze because I have some message that's 00:48:37.820 --> 00:48:39.440 align:middle line:90% received by my decoder. 00:48:39.440 --> 00:48:41.580 align:middle line:90% That's C tilde. 00:48:41.580 --> 00:48:46.900 align:middle line:84% What's the chance he belongs to some other ring around Cj? 00:48:46.900 --> 00:48:48.980 align:middle line:84% The key point is that when I express 00:48:48.980 --> 00:48:51.340 align:middle line:84% the probability this way, remember, 00:48:51.340 --> 00:48:55.400 align:middle line:84% I chose each of those code words randomly. 00:48:55.400 --> 00:48:58.120 align:middle line:84% So what I could do is I could run the whole experiment 00:48:58.120 --> 00:49:02.640 align:middle line:84% in my head, but be lazy about when I choose Cj. 00:49:02.640 --> 00:49:06.100 align:middle line:84% So you tell me you're sending Ci along the channel. 00:49:06.100 --> 00:49:06.600 align:middle line:90% Fine. 00:49:06.600 --> 00:49:08.725 align:middle line:84% You tell me you're sending that particular message, 00:49:08.725 --> 00:49:12.360 align:middle line:84% I fix what Ci is, we send Ci through the channel, 00:49:12.360 --> 00:49:15.040 align:middle line:90% and we receive C tilde. 00:49:15.040 --> 00:49:21.760 align:middle line:84% But only now after I receive C tilde do I pick what Cj is. 00:49:21.760 --> 00:49:27.120 align:middle line:84% And the chance that Cj, if it's a uniformly random string, lands 00:49:27.120 --> 00:49:29.640 align:middle line:84% in this particular ring, I can bound that 00:49:29.640 --> 00:49:31.920 align:middle line:90% based on the size of the ring. 00:49:31.920 --> 00:49:35.720 align:middle line:90% So now I can appeal to fact 1. 00:49:35.720 --> 00:49:43.280 align:middle line:84% So we can upper bound this probability by this 2 00:49:43.280 --> 00:49:49.400 align:middle line:84% to the entropy of p plus or minus the slack gamma, n, over 2 00:49:49.400 --> 00:49:55.580 align:middle line:84% to the n, just because that's the size of the ring C tilde, 00:49:55.580 --> 00:49:58.940 align:middle line:84% is exactly this quantity from fact 1. 00:49:58.940 --> 00:50:01.740 align:middle line:84% And we're choosing Cj uniformly at random 00:50:01.740 --> 00:50:04.920 align:middle line:84% from all of the strings of length n that are binary. 00:50:04.920 --> 00:50:07.700 align:middle line:84% So we get exactly this expression. 00:50:07.700 --> 00:50:10.100 align:middle line:84% And so now we're in good shape because we 00:50:10.100 --> 00:50:18.100 align:middle line:84% can appeal to a union bound because we 00:50:18.100 --> 00:50:21.460 align:middle line:84% don't care about this just for one particular j. 00:50:21.460 --> 00:50:27.020 align:middle line:84% We care about it for all j not equal to i. 00:50:27.020 --> 00:50:37.940 align:middle line:84% So now, by a union bound, the probability that C tilde 00:50:37.940 --> 00:50:45.260 align:middle line:84% belongs to any ring around a Cj, where j is not 00:50:45.260 --> 00:50:47.060 align:middle line:84% equal to i, the probability that's true 00:50:47.060 --> 00:50:52.130 align:middle line:84% for any j is, at most, the size of our codebook times 00:50:52.130 --> 00:50:56.170 align:middle line:84% this probability, 2 to the entropy of p plus or minus 00:50:56.170 --> 00:51:00.330 align:middle line:90% gamma n over 2 to the n. 00:51:00.330 --> 00:51:02.150 align:middle line:84% So this is just a simple union bound. 00:51:02.150 --> 00:51:05.130 align:middle line:90% 00:51:05.130 --> 00:51:09.570 align:middle line:84% And now what we can do, if you ignore this gamma, what's 00:51:09.570 --> 00:51:14.850 align:middle line:84% going on is I have my codebook, remember, 00:51:14.850 --> 00:51:19.450 align:middle line:84% looks like 2 to the 1 minus entropy of-- it 00:51:19.450 --> 00:51:26.330 align:middle line:84% looks like this codebook right here, is 2 times 2 to the Rn. 00:51:26.330 --> 00:51:29.810 align:middle line:84% And we're assuming that R is strictly less than 1 00:51:29.810 --> 00:51:32.130 align:middle line:90% minus entropy of p. 00:51:32.130 --> 00:51:37.130 align:middle line:84% What's going on right here is I have 2 to the minus 1 00:51:37.130 --> 00:51:38.970 align:middle line:90% minus entropy of p. 00:51:38.970 --> 00:51:41.770 align:middle line:84% So the point is that, once my rate is strictly 00:51:41.770 --> 00:51:44.970 align:middle line:84% smaller than what's happening in the exponent over here, 00:51:44.970 --> 00:51:48.710 align:middle line:84% then I'm in good shape because this probability goes 00:51:48.710 --> 00:51:53.350 align:middle line:90% to 0 as n goes to infinity. 00:51:53.350 --> 00:51:56.150 align:middle line:84% So that's the proof of our key lemma. 00:51:56.150 --> 00:51:59.750 align:middle line:84% It's really just the counting argument at the end of the day, 00:51:59.750 --> 00:52:03.390 align:middle line:84% is that my codebook isn't so large, like I 00:52:03.390 --> 00:52:06.790 align:middle line:84% union bound over all of the things in my codebook. 00:52:06.790 --> 00:52:09.070 align:middle line:84% I look at the size of each of these rings 00:52:09.070 --> 00:52:12.990 align:middle line:84% compared to the size of the underlying domain. 00:52:12.990 --> 00:52:16.270 align:middle line:84% And as long as the size of my codebook times 00:52:16.270 --> 00:52:19.750 align:middle line:84% the size of the ring is strictly smaller than my domain, 00:52:19.750 --> 00:52:24.190 align:middle line:84% even asymptotically, then I'm in good shape. 00:52:24.190 --> 00:52:26.370 align:middle line:84% So this proves these two key facts. 00:52:26.370 --> 00:52:29.110 align:middle line:84% We know that the probability you don't belong to the ring 00:52:29.110 --> 00:52:31.390 align:middle line:84% you start with is very small, is at most 00:52:31.390 --> 00:52:34.990 align:middle line:84% epsilon over 2 by the weak law of large numbers. 00:52:34.990 --> 00:52:38.670 align:middle line:84% And we know by a union bound and a counting argument 00:52:38.670 --> 00:52:41.550 align:middle line:84% that the probability that you belong to any other ring 00:52:41.550 --> 00:52:43.070 align:middle line:90% is very small. 00:52:43.070 --> 00:52:47.130 align:middle line:84% And the key step that we used was really this transformation 00:52:47.130 --> 00:52:50.850 align:middle line:84% right here, that the probability that C tilde belongs 00:52:50.850 --> 00:52:54.710 align:middle line:84% to another ring because the Cj's are chosen randomly, 00:52:54.710 --> 00:52:56.330 align:middle line:84% that's the same as the probability 00:52:56.330 --> 00:53:01.287 align:middle line:84% that my newly sampled Cj belongs to the ring around C tilde. 00:53:01.287 --> 00:53:02.870 align:middle line:84% So are there any questions about this? 00:53:02.870 --> 00:53:05.410 align:middle line:84% This is one of the key proofs for today, 00:53:05.410 --> 00:53:07.850 align:middle line:90% is the proof of this key lemma. 00:53:07.850 --> 00:53:08.950 align:middle line:90% Any questions? 00:53:08.950 --> 00:53:11.530 align:middle line:90% 00:53:11.530 --> 00:53:12.970 align:middle line:90% Give me a thumbs up? 00:53:12.970 --> 00:53:13.690 align:middle line:90% Good? 00:53:13.690 --> 00:53:15.730 align:middle line:90% All right? 00:53:15.730 --> 00:53:18.850 align:middle line:90% So now let's use this lemma. 00:53:18.850 --> 00:53:20.750 align:middle line:90% And there'll be another trick. 00:53:20.750 --> 00:53:23.890 align:middle line:84% It's a subtle trick, but it's a cool trick. 00:53:23.890 --> 00:53:26.322 align:middle line:84% So I need a little bit more terminology. 00:53:26.322 --> 00:53:28.780 align:middle line:84% And then we're going to be able to prove Shannon's theorem. 00:53:28.780 --> 00:53:37.770 align:middle line:90% 00:53:37.770 --> 00:53:41.190 align:middle line:84% Because one of the things to think about is that, so far, 00:53:41.190 --> 00:53:43.690 align:middle line:84% our attempt at proving Shannon's theorem, 00:53:43.690 --> 00:53:45.680 align:middle line:84% we're choosing a random codebook. 00:53:45.680 --> 00:53:49.430 align:middle line:90% 00:53:49.430 --> 00:53:52.670 align:middle line:84% And that codebook then defines what the encoding 00:53:52.670 --> 00:53:54.910 align:middle line:90% and the decoding functions are. 00:53:54.910 --> 00:53:58.670 align:middle line:84% Our encoding sends each message to one of those randomly chosen 00:53:58.670 --> 00:54:00.710 align:middle line:90% codewords in our codebook. 00:54:00.710 --> 00:54:04.920 align:middle line:84% Our decoder just looks at which message's ring you belong to. 00:54:04.920 --> 00:54:07.170 align:middle line:84% But at the end of the day, to prove Shannon's theorem, 00:54:07.170 --> 00:54:10.950 align:middle line:84% I don't want to argue that a random code is usually good. 00:54:10.950 --> 00:54:14.790 align:middle line:84% I want to show that there's a code that's always good. 00:54:14.790 --> 00:54:17.750 align:middle line:84% So that subtlety is really where the probabilistic method is 00:54:17.750 --> 00:54:20.310 align:middle line:84% going to show up, is because we have 00:54:20.310 --> 00:54:22.390 align:middle line:84% to argue that there actually is one fixed 00:54:22.390 --> 00:54:26.590 align:middle line:84% choice of this random codebook that makes everything small. 00:54:26.590 --> 00:54:29.530 align:middle line:84% And this is a little bit subtle to really prove this statement, 00:54:29.530 --> 00:54:31.630 align:middle line:90% but I need some more intuition. 00:54:31.630 --> 00:54:33.470 align:middle line:90% I need some more notation. 00:54:33.470 --> 00:54:39.150 align:middle line:84% So I'm going to let Ei of my codebook, script C, 00:54:39.150 --> 00:54:47.420 align:middle line:84% be the event that everything goes well, 00:54:47.420 --> 00:54:52.500 align:middle line:84% that I send in my ith code word from my codebook 00:54:52.500 --> 00:54:55.860 align:middle line:84% through the binary symmetric channel. 00:54:55.860 --> 00:55:00.500 align:middle line:84% I get out some received message, C tilde, i. 00:55:00.500 --> 00:55:03.020 align:middle line:84% And I have exactly the properties 00:55:03.020 --> 00:55:06.220 align:middle line:84% that are promised to me in this key lemma, lemma 1. 00:55:06.220 --> 00:55:12.020 align:middle line:84% I want that my received thing belongs to the ring around Ci, 00:55:12.020 --> 00:55:14.360 align:middle line:90% and it belongs to no other ring. 00:55:14.360 --> 00:55:20.460 align:middle line:90% 00:55:20.460 --> 00:55:24.100 align:middle line:84% And this is true for all j not equal to i. 00:55:24.100 --> 00:55:26.740 align:middle line:84% So notice that this event now is a property 00:55:26.740 --> 00:55:29.640 align:middle line:90% of the actual codebook too. 00:55:29.640 --> 00:55:33.540 align:middle line:84% It's a property of what errors the channel introduces 00:55:33.540 --> 00:55:35.590 align:middle line:84% and a property of the choice of the codebook. 00:55:35.590 --> 00:55:38.708 align:middle line:90% 00:55:38.708 --> 00:55:40.500 align:middle line:84% And what we're going to do is we're finally 00:55:40.500 --> 00:55:48.440 align:middle line:84% going to let lambda i of C denote 1 minus the probability 00:55:48.440 --> 00:55:50.250 align:middle line:90% that this event happens. 00:55:50.250 --> 00:55:52.760 align:middle line:90% 00:55:52.760 --> 00:55:56.800 align:middle line:84% So it's really the probability that there's a decoding failure. 00:55:56.800 --> 00:55:58.920 align:middle line:84% But notice that the decoding failure then 00:55:58.920 --> 00:56:02.120 align:middle line:84% depends on what the original-- that probability depends 00:56:02.120 --> 00:56:04.000 align:middle line:90% on the choice of the codebook. 00:56:04.000 --> 00:56:10.000 align:middle line:84% And it depends on which message we're sending, all right? 00:56:10.000 --> 00:56:14.160 align:middle line:84% And so what I claim, we just proved through lemma 1, 00:56:14.160 --> 00:56:15.860 align:middle line:90% is the following statement. 00:56:15.860 --> 00:56:25.160 align:middle line:90% 00:56:25.160 --> 00:56:33.560 align:middle line:84% We proved that the average over all of the possible codebooks, 00:56:33.560 --> 00:56:43.860 align:middle line:84% because we're choosing these codebooks uniformly at random, 00:56:43.860 --> 00:56:46.900 align:middle line:84% the average over all of the messages 00:56:46.900 --> 00:56:55.380 align:middle line:84% that we want to send of having a decoding failure is small. 00:56:55.380 --> 00:56:57.260 align:middle line:90% So what does this thing say? 00:56:57.260 --> 00:56:59.700 align:middle line:90% Let's parse this in English. 00:56:59.700 --> 00:57:02.120 align:middle line:84% When I choose a random codebook-- 00:57:02.120 --> 00:57:04.900 align:middle line:84% this is the average over a random codebook. 00:57:04.900 --> 00:57:08.620 align:middle line:84% When I choose a random message that I want to encode, 00:57:08.620 --> 00:57:12.080 align:middle line:84% the probability that I have a decoding error for that codebook 00:57:12.080 --> 00:57:14.820 align:middle line:84% and that message on the binary symmetric channel 00:57:14.820 --> 00:57:17.500 align:middle line:90% is at most epsilon. 00:57:17.500 --> 00:57:19.300 align:middle line:84% This is not the same thing as saying 00:57:19.300 --> 00:57:21.820 align:middle line:84% that there exists a codebook that's 00:57:21.820 --> 00:57:24.900 align:middle line:90% always good for every message. 00:57:24.900 --> 00:57:26.660 align:middle line:84% And that's exactly what we have to try 00:57:26.660 --> 00:57:28.980 align:middle line:90% and twist the statement into. 00:57:28.980 --> 00:57:30.925 align:middle line:84% So we proved through lemma 1 was this. 00:57:30.925 --> 00:57:32.300 align:middle line:84% And we really need something else 00:57:32.300 --> 00:57:36.865 align:middle line:84% that gets rid of this average over codewords. 00:57:36.865 --> 00:57:49.000 align:middle line:84% And so what I claim now is, by the probabilistic method, 00:57:49.000 --> 00:57:52.760 align:middle line:84% we claim that there exists a codebook with the property 00:57:52.760 --> 00:58:01.400 align:middle line:84% that the average over all the messages of the decoding failure 00:58:01.400 --> 00:58:03.840 align:middle line:90% probability is at most epsilon. 00:58:03.840 --> 00:58:05.360 align:middle line:84% So the probabilistic method is one 00:58:05.360 --> 00:58:08.160 align:middle line:84% of these things where it looks very natural, sometimes, 00:58:08.160 --> 00:58:09.760 align:middle line:90% when you state it. 00:58:09.760 --> 00:58:11.840 align:middle line:84% But other times, when it's happening 00:58:11.840 --> 00:58:13.700 align:middle line:84% with other layers of abstraction, 00:58:13.700 --> 00:58:16.960 align:middle line:84% it's a bit harder to wrap your head around. 00:58:16.960 --> 00:58:21.970 align:middle line:84% So we use the probabilistic method to show things like, 00:58:21.970 --> 00:58:24.040 align:middle line:84% if we choose that there's a graph that 00:58:24.040 --> 00:58:26.780 align:middle line:84% has no large clique or no large independent set-- 00:58:26.780 --> 00:58:29.200 align:middle line:84% we'll talk about that, I guess, later. 00:58:29.200 --> 00:58:33.120 align:middle line:84% And here, we're really using the probabilistic method 00:58:33.120 --> 00:58:36.480 align:middle line:84% the same way that I know that this expectation 00:58:36.480 --> 00:58:40.670 align:middle line:84% of this quantity inside is small over a typical codebook. 00:58:40.670 --> 00:58:44.882 align:middle line:84% So there must exist one codebook where it's small as well. 00:58:44.882 --> 00:58:47.470 align:middle line:84% So does everyone see how I got from here to here using 00:58:47.470 --> 00:58:48.770 align:middle line:90% the probabilistic method? 00:58:48.770 --> 00:58:49.970 align:middle line:90% Is that all right? 00:58:49.970 --> 00:58:50.470 align:middle line:90% Yeah? 00:58:50.470 --> 00:58:53.350 align:middle line:90% 00:58:53.350 --> 00:58:55.950 align:middle line:84% So now we're in good shape, and I 00:58:55.950 --> 00:59:00.990 align:middle line:84% can tell you how to finish the rest of the argument. 00:59:00.990 --> 00:59:02.690 align:middle line:90% Let's see how much space I need. 00:59:02.690 --> 00:59:03.360 align:middle line:90% Not too much. 00:59:03.360 --> 00:59:09.670 align:middle line:90% 00:59:09.670 --> 00:59:11.430 align:middle line:84% So now we're starting to make progress 00:59:11.430 --> 00:59:15.070 align:middle line:84% because, instead of saying over a random codebook, 00:59:15.070 --> 00:59:18.670 align:middle line:84% over a random message, our decoding failure is small, 00:59:18.670 --> 00:59:22.350 align:middle line:84% we know that we have a codebook where 00:59:22.350 --> 00:59:26.350 align:middle line:84% the average over the messages, the decoding error rate 00:59:26.350 --> 00:59:29.350 align:middle line:90% is small. 00:59:29.350 --> 00:59:32.350 align:middle line:84% But we really care about, back to our original statement 00:59:32.350 --> 00:59:35.270 align:middle line:84% of what we wanted to prove, was we want that there's 00:59:35.270 --> 00:59:39.830 align:middle line:84% a codebook so that every message can be decoded well, 00:59:39.830 --> 00:59:41.810 align:middle line:90% not a random message. 00:59:41.810 --> 00:59:45.170 align:middle line:84% So we have one remaining expectation 00:59:45.170 --> 00:59:49.570 align:middle line:84% that we want to prune, which is this average over M, all right? 00:59:49.570 --> 00:59:52.930 align:middle line:84% And now we're going to do it in a kind of cool way, which 00:59:52.930 --> 00:59:56.220 align:middle line:90% is I claim by Markov's bound. 00:59:56.220 --> 01:00:01.570 align:middle line:90% 01:00:01.570 --> 01:00:12.050 align:middle line:84% We know that at least half of the i's 01:00:12.050 --> 01:00:21.850 align:middle line:84% satisfy the bound that lambda i of C is at most 2 epsilon. 01:00:21.850 --> 01:00:24.210 align:middle line:84% So this is our old friend, the Markov 01:00:24.210 --> 01:00:28.210 align:middle line:84% bound, because we're looking at random variables that 01:00:28.210 --> 01:00:32.530 align:middle line:84% are 0, 1 valued, like did you have a decoding failure? 01:00:32.530 --> 01:00:36.790 align:middle line:84% And if the average over these messages, 1 through M, 01:00:36.790 --> 01:00:40.870 align:middle line:84% that we want to send has the decoding failure probability 01:00:40.870 --> 01:00:45.310 align:middle line:84% bounded by epsilon on average, we know that for at least 1/2 01:00:45.310 --> 01:00:48.710 align:middle line:84% of the realizations, the probability that they're larger 01:00:48.710 --> 01:00:54.310 align:middle line:84% than 2 epsilon, the decoding failure probability, is small. 01:00:54.310 --> 01:00:56.700 align:middle line:84% So this is the usual thing, that if this weren't true, 01:00:56.700 --> 01:00:58.950 align:middle line:84% then the expectation would be higher because these are 01:00:58.950 --> 01:01:01.230 align:middle line:90% non-negative random variables. 01:01:01.230 --> 01:01:03.270 align:middle line:84% And so now what we can do is we can 01:01:03.270 --> 01:01:07.070 align:middle line:84% let C prime, our eventual codebook, 01:01:07.070 --> 01:01:18.630 align:middle line:84% be the codebook restricted to just these messages, just 01:01:18.630 --> 01:01:26.070 align:middle line:84% these codewords, the ones that satisfy this condition right 01:01:26.070 --> 01:01:29.790 align:middle line:84% here, that their lambda i for that particular choice 01:01:29.790 --> 01:01:32.990 align:middle line:84% of the codebook is at most 2 epsilon. 01:01:32.990 --> 01:01:36.270 align:middle line:84% So what does this mean, just putting this back into English? 01:01:36.270 --> 01:01:41.450 align:middle line:84% It means that this new codebook, C prime, 01:01:41.450 --> 01:01:46.640 align:middle line:90% is itself an Rn, n encoding. 01:01:46.640 --> 01:01:49.770 align:middle line:90% 01:01:49.770 --> 01:01:54.130 align:middle line:84% That's exactly why I started off with my original codebook was 01:01:54.130 --> 01:02:01.850 align:middle line:84% on capital M things, where M was equal to 2 times 2 to the Rn, 01:02:01.850 --> 01:02:04.272 align:middle line:84% was exactly this extra factor of 2 01:02:04.272 --> 01:02:05.730 align:middle line:84% that I was going to have to give up 01:02:05.730 --> 01:02:08.170 align:middle line:90% when I went from C to C prime. 01:02:08.170 --> 01:02:10.770 align:middle line:84% So whatever's left is still an Rn, 01:02:10.770 --> 01:02:14.890 align:middle line:84% n encoding because there are 2 to the RNA things in it. 01:02:14.890 --> 01:02:18.250 align:middle line:84% And we know, just by unwrapping the definition of what 01:02:18.250 --> 01:02:22.890 align:middle line:84% lambda iC is, what this means is the probability 01:02:22.890 --> 01:02:28.170 align:middle line:90% that C tilde i is not equal to-- 01:02:28.170 --> 01:02:34.400 align:middle line:84% sorry, that the probability that M tilde is not equal to M 01:02:34.400 --> 01:02:37.680 align:middle line:84% is at most 2 epsilon because, literally, 01:02:37.680 --> 01:02:42.560 align:middle line:84% the way we define this was how often we expect for the given 01:02:42.560 --> 01:02:45.880 align:middle line:84% code word to have a decoding failure probability 01:02:45.880 --> 01:02:48.720 align:middle line:90% on that particular message? 01:02:48.720 --> 01:02:51.280 align:middle line:84% So the only code words that survived 01:02:51.280 --> 01:02:54.440 align:middle line:84% were exactly 1's, who, when I fix the codebook 01:02:54.440 --> 01:02:57.840 align:middle line:84% and I fix the message, have a very small failure probability 01:02:57.840 --> 01:03:00.520 align:middle line:90% of getting the wrong decoding. 01:03:00.520 --> 01:03:03.360 align:middle line:84% So this proves the most interesting part 01:03:03.360 --> 01:03:07.400 align:middle line:84% of Shannon's theorem, that we can get positive rate and error 01:03:07.400 --> 01:03:09.200 align:middle line:90% probability going to 0. 01:03:09.200 --> 01:03:14.800 align:middle line:84% So we can drastically improve upon the repetition code. 01:03:14.800 --> 01:03:17.360 align:middle line:84% If you actually work out the details, what ends up happening 01:03:17.360 --> 01:03:20.560 align:middle line:84% is that it's not just that the probability of failure 01:03:20.560 --> 01:03:22.142 align:middle line:90% goes to 0. 01:03:22.142 --> 01:03:23.600 align:middle line:84% The probability of failure actually 01:03:23.600 --> 01:03:27.080 align:middle line:84% goes to 0 exponentially fast, which is really cool. 01:03:27.080 --> 01:03:29.400 align:middle line:84% So it's way better than the repetition code 01:03:29.400 --> 01:03:31.240 align:middle line:90% on a few different fronts. 01:03:31.240 --> 01:03:32.740 align:middle line:84% If I wanted the failure probability 01:03:32.740 --> 01:03:35.640 align:middle line:84% to be exponentially small for my repetition code, 01:03:35.640 --> 01:03:39.755 align:middle line:84% I would have had to increase it way beyond this log k blow up. 01:03:39.755 --> 01:03:41.380 align:middle line:84% So a bunch of people at the time really 01:03:41.380 --> 01:03:43.960 align:middle line:84% thought that the repetition code was optimal. 01:03:43.960 --> 01:03:49.620 align:middle line:84% So this came as quite a surprise that this was not true. 01:03:49.620 --> 01:03:52.260 align:middle line:84% I will briefly give you the intuition 01:03:52.260 --> 01:03:56.540 align:middle line:84% behind the first part of Shannon's theorem, 01:03:56.540 --> 01:03:57.960 align:middle line:90% and then we'll call it a day. 01:03:57.960 --> 01:04:00.980 align:middle line:90% 01:04:00.980 --> 01:04:07.140 align:middle line:84% So remember that the first part of Shannon's theorem 01:04:07.140 --> 01:04:10.740 align:middle line:84% was that you can't communicate at a rate 01:04:10.740 --> 01:04:14.580 align:middle line:84% larger than the capacity, 1 minus entropy of p. 01:04:14.580 --> 01:04:18.860 align:middle line:84% If you do, the probability of getting the correct message 01:04:18.860 --> 01:04:21.340 align:middle line:90% is going to go to 0. 01:04:21.340 --> 01:04:24.980 align:middle line:84% And so let me give you, very briefly, the intuition. 01:04:24.980 --> 01:04:30.320 align:middle line:84% So let's suppose that we had a coding scheme, where 01:04:30.320 --> 01:04:36.600 align:middle line:84% the rate was larger than the capacity, 1 minus entropy of p. 01:04:36.600 --> 01:04:39.720 align:middle line:84% And let's suppose, for the sake of contradiction, 01:04:39.720 --> 01:04:43.800 align:middle line:84% that it really wasn't such a bad coding scheme. 01:04:43.800 --> 01:04:45.880 align:middle line:84% Maybe the probability that it successfully 01:04:45.880 --> 01:04:49.360 align:middle line:84% decoded the message is at least some epsilon, 01:04:49.360 --> 01:04:53.680 align:middle line:90% like 0.01, let's say. 01:04:53.680 --> 01:04:57.760 align:middle line:84% Well, what we're going to argue is that we'll prove 01:04:57.760 --> 01:05:02.480 align:middle line:84% a contradiction, that the rate really cannot be this large. 01:05:02.480 --> 01:05:17.880 align:middle line:84% So the main point is that an epsilon over 2 fraction of ring 01:05:17.880 --> 01:05:27.410 align:middle line:90% gamma Ci must be decoded to Ci. 01:05:27.410 --> 01:05:30.900 align:middle line:90% 01:05:30.900 --> 01:05:33.980 align:middle line:84% So here the point is, originally, our decoder 01:05:33.980 --> 01:05:36.740 align:middle line:90% was just based on the ring. 01:05:36.740 --> 01:05:39.260 align:middle line:84% But now we have no idea what the decoder is. 01:05:39.260 --> 01:05:41.460 align:middle line:84% But the same way, whatever our decoder 01:05:41.460 --> 01:05:45.980 align:middle line:84% is, we know that when we pass through this transmission 01:05:45.980 --> 01:05:49.660 align:middle line:84% along the channel, we're going to end up in this ring 01:05:49.660 --> 01:05:54.060 align:middle line:84% because that was the first part of lemma 1, 01:05:54.060 --> 01:05:57.340 align:middle line:84% was just that we land in the ring with all but very small 01:05:57.340 --> 01:05:59.100 align:middle line:90% failure probability. 01:05:59.100 --> 01:06:01.620 align:middle line:84% And once we're in this ring, because that's always 01:06:01.620 --> 01:06:05.140 align:middle line:84% where we land, the fact that we're successfully 01:06:05.140 --> 01:06:08.300 align:middle line:84% decoding the original message is just saying that, 01:06:08.300 --> 01:06:13.900 align:middle line:84% among these outputs that we often reach, well, many of them 01:06:13.900 --> 01:06:16.680 align:middle line:84% must be decoded to the correct message. 01:06:16.680 --> 01:06:18.860 align:middle line:84% That's just the definition of-- that's 01:06:18.860 --> 01:06:22.460 align:middle line:84% just the translation of what this statement means. 01:06:22.460 --> 01:06:30.170 align:middle line:84% But now the point is that, but the C tildes 01:06:30.170 --> 01:06:45.741 align:middle line:84% that are decoded to different Ci's must be disjoint 01:06:45.741 --> 01:06:48.930 align:middle line:84% because your decoding function has to map you 01:06:48.930 --> 01:06:52.170 align:middle line:90% to a unique original message. 01:06:52.170 --> 01:06:55.290 align:middle line:84% So if I look at, over all these different messages, which 01:06:55.290 --> 01:07:00.130 align:middle line:84% things are decoded to it is each of these sets are disjoint. 01:07:00.130 --> 01:07:02.130 align:middle line:90% And so what does this mean? 01:07:02.130 --> 01:07:06.490 align:middle line:84% Well, if we add up just the total volume of these sets, 01:07:06.490 --> 01:07:08.330 align:middle line:84% we know that they're at least an epsilon 01:07:08.330 --> 01:07:10.530 align:middle line:90% over 2 fraction of the ring. 01:07:10.530 --> 01:07:13.290 align:middle line:84% We know that there's one of these rings for each 01:07:13.290 --> 01:07:14.730 align:middle line:90% of the codebook. 01:07:14.730 --> 01:07:20.250 align:middle line:84% We know that the ring has size 2 to the entropy of p times n. 01:07:20.250 --> 01:07:23.770 align:middle line:84% But this cannot be larger than 2 to the n 01:07:23.770 --> 01:07:27.010 align:middle line:84% because that's the number of possible C tildes there are, 01:07:27.010 --> 01:07:28.750 align:middle line:90% period. 01:07:28.750 --> 01:07:31.250 align:middle line:84% And so when we rearrange this bound, 01:07:31.250 --> 01:07:33.630 align:middle line:84% we'll get exactly the thing we wanted, 01:07:33.630 --> 01:07:38.910 align:middle line:84% that the size of my codebook is at most something like 2 01:07:38.910 --> 01:07:42.550 align:middle line:84% to the 1 minus entropy of p times n. 01:07:42.550 --> 01:07:44.370 align:middle line:84% So this is just meant as a sketch 01:07:44.370 --> 01:07:46.750 align:middle line:84% because, really, to do this precisely, 01:07:46.750 --> 01:07:49.270 align:middle line:84% I would have to prove a lot of the helper lemmas 01:07:49.270 --> 01:07:52.110 align:middle line:84% here and give similar kinds of bounds. 01:07:52.110 --> 01:07:54.210 align:middle line:84% But the intuition is much simpler. 01:07:54.210 --> 01:07:56.870 align:middle line:84% And we're only going to ask you about, really, the second part 01:07:56.870 --> 01:07:58.510 align:middle line:90% for Shannon's theorem. 01:07:58.510 --> 01:08:01.580 align:middle line:84% So are there any questions about this? 01:08:01.580 --> 01:08:04.310 align:middle line:90% 01:08:04.310 --> 01:08:04.830 align:middle line:90% Yeah? 01:08:04.830 --> 01:08:06.950 align:middle line:84% AUDIENCE: Sorry, can you explain the [? x1 ?] over 2 part? 01:08:06.950 --> 01:08:08.325 align:middle line:84% ANKUR MOITRA: Yeah, that's right. 01:08:08.325 --> 01:08:09.090 align:middle line:90% Sure. 01:08:09.090 --> 01:08:11.590 align:middle line:84% So what is the statement saying right here, 01:08:11.590 --> 01:08:16.109 align:middle line:84% is that at least an epsilon fraction-- so if I look at-- 01:08:16.109 --> 01:08:20.510 align:middle line:84% when I start off with Ci, and I look at the output 01:08:20.510 --> 01:08:23.310 align:middle line:84% after the binary symmetric channel, then 01:08:23.310 --> 01:08:26.850 align:middle line:84% at least an epsilon fraction of those outputs 01:08:26.850 --> 01:08:31.649 align:middle line:84% must be successfully decoded to the correct original message. 01:08:31.649 --> 01:08:35.250 align:middle line:84% Now, almost all of the outputs live in the ring. 01:08:35.250 --> 01:08:38.649 align:middle line:84% So if we just throw out all the things that aren't in the ring, 01:08:38.649 --> 01:08:41.689 align:middle line:84% we're not really changing what the output distribution 01:08:41.689 --> 01:08:43.729 align:middle line:90% looks like from our BSC. 01:08:43.729 --> 01:08:45.250 align:middle line:84% And that means that we still must 01:08:45.250 --> 01:08:48.370 align:middle line:84% have-- at least a large fraction of this ring 01:08:48.370 --> 01:08:51.130 align:middle line:84% must also be successfully decoded. 01:08:51.130 --> 01:08:53.170 align:middle line:84% To fully prove this would be a lot more work 01:08:53.170 --> 01:08:57.050 align:middle line:84% and would be a bit tedious, but that's the intuition. 01:08:57.050 --> 01:08:58.290 align:middle line:90% Any other questions? 01:08:58.290 --> 01:08:59.430 align:middle line:90% Great questions, though. 01:08:59.430 --> 01:09:04.090 align:middle line:90% 01:09:04.090 --> 01:09:08.270 align:middle line:84% So let me end with a little bit of a historical perspective. 01:09:08.270 --> 01:09:11.890 align:middle line:84% So this Shannon theorem is quite amazing. 01:09:11.890 --> 01:09:16.090 align:middle line:84% It's actually not such a perfectly ideal scheme. 01:09:16.090 --> 01:09:17.770 align:middle line:84% So we have the same kinds of issues 01:09:17.770 --> 01:09:22.616 align:middle line:84% that we had in Shannon's noiseless coding theorem. 01:09:22.616 --> 01:09:25.149 align:middle line:84% So we proved Shannon's noiseless coding theorem 01:09:25.149 --> 01:09:29.149 align:middle line:84% about how well we can compress a first-order source. 01:09:29.149 --> 01:09:30.750 align:middle line:84% But then the thing I complained about 01:09:30.750 --> 01:09:34.470 align:middle line:84% was the fact that, really, if we were to execute the scheme, 01:09:34.470 --> 01:09:39.870 align:middle line:84% I would have to write down the full set of encodings, 01:09:39.870 --> 01:09:41.090 align:middle line:90% and that would be gigantic. 01:09:41.090 --> 01:09:43.470 align:middle line:84% I'd have to anticipate everything you could ever 01:09:43.470 --> 01:09:46.590 align:middle line:84% want to compress and write down this codebook of what 01:09:46.590 --> 01:09:48.270 align:middle line:90% to compress it to. 01:09:48.270 --> 01:09:52.630 align:middle line:84% Same problem happens here, that choosing this codebook, script 01:09:52.630 --> 01:09:55.970 align:middle line:84% C, randomly is very convenient for the proof 01:09:55.970 --> 01:09:57.870 align:middle line:84% because it makes it easier to reason 01:09:57.870 --> 01:09:59.950 align:middle line:90% via the probabilistic method. 01:09:59.950 --> 01:10:03.190 align:middle line:84% It makes it easier to reason about failure probabilities 01:10:03.190 --> 01:10:05.750 align:middle line:84% of events because it boils down to some combinatorics 01:10:05.750 --> 01:10:07.950 align:middle line:90% of the sizes of rings. 01:10:07.950 --> 01:10:10.310 align:middle line:84% But the trouble is that this would be a horrible coding 01:10:10.310 --> 01:10:14.270 align:middle line:84% scheme if I really had to write down exponentially many things 01:10:14.270 --> 01:10:16.590 align:middle line:90% and define my codebook. 01:10:16.590 --> 01:10:20.070 align:middle line:84% So what Peter is going to be talking about on Tuesday 01:10:20.070 --> 01:10:24.340 align:middle line:84% is a very powerful paradigm, which is called a linear code. 01:10:24.340 --> 01:10:27.180 align:middle line:84% So instead of explicitly writing down all 01:10:27.180 --> 01:10:30.220 align:middle line:84% of the codes and codewords in my codebook, 01:10:30.220 --> 01:10:32.860 align:middle line:84% I can instead implicitly describe them 01:10:32.860 --> 01:10:35.060 align:middle line:90% as matrix vector products. 01:10:35.060 --> 01:10:38.300 align:middle line:84% All of my messages are vectors, and my codebook 01:10:38.300 --> 01:10:40.300 align:middle line:90% describes a matrix. 01:10:40.300 --> 01:10:44.060 align:middle line:84% And when I want to encode some message as what 01:10:44.060 --> 01:10:45.940 align:middle line:84% it maps to is the code word, I just 01:10:45.940 --> 01:10:48.220 align:middle line:84% take the matrix times the vector. 01:10:48.220 --> 01:10:50.900 align:middle line:84% So this now becomes a much more concise way 01:10:50.900 --> 01:10:53.180 align:middle line:90% to define the codebook. 01:10:53.180 --> 01:10:57.700 align:middle line:84% So he'll tell you about linear codes. 01:10:57.700 --> 01:11:01.780 align:middle line:84% And then the other thing, which that next lecture will touch on, 01:11:01.780 --> 01:11:03.580 align:middle line:84% is that, right now, the crucial assumption 01:11:03.580 --> 01:11:06.900 align:middle line:84% that we used in the proof of the theorems today 01:11:06.900 --> 01:11:09.980 align:middle line:84% was the fact that the binary symmetric channel introduced 01:11:09.980 --> 01:11:12.460 align:middle line:90% flips randomly. 01:11:12.460 --> 01:11:15.180 align:middle line:84% So bit-by-bit, it had a third probability 01:11:15.180 --> 01:11:17.180 align:middle line:90% of introducing a flip. 01:11:17.180 --> 01:11:18.900 align:middle line:84% Now, it turns out there are other models 01:11:18.900 --> 01:11:22.800 align:middle line:84% for codes, where you could instead think about. worst case. 01:11:22.800 --> 01:11:26.280 align:middle line:84% What if sitting on your channel is an adversary who gets 01:11:26.280 --> 01:11:29.960 align:middle line:84% to corrupt 1/3 of your bits, but they get to choose when and how 01:11:29.960 --> 01:11:31.640 align:middle line:90% to corrupt those bits? 01:11:31.640 --> 01:11:33.340 align:middle line:84% That's called adversarial coding, 01:11:33.340 --> 01:11:35.800 align:middle line:84% and we can get similar semantics there, 01:11:35.800 --> 01:11:39.600 align:middle line:84% where we want an encoding and decoding function that now works 01:11:39.600 --> 01:11:42.320 align:middle line:84% even if the channel is not stochastic, but is bounded 01:11:42.320 --> 01:11:44.800 align:middle line:84% in terms of its budget of corruptions. 01:11:44.800 --> 01:11:46.580 align:middle line:84% So for those types of problems in general, 01:11:46.580 --> 01:11:50.938 align:middle line:84% we don't understand such tight trade-offs as we do for Shannon. 01:11:50.938 --> 01:11:52.480 align:middle line:84% But he'll tell you about linear codes 01:11:52.480 --> 01:11:55.077 align:middle line:84% and he'll tell you about adversarial decoding. 01:11:55.077 --> 01:11:56.660 align:middle line:84% And then after Peter's done with that, 01:11:56.660 --> 01:11:58.320 align:middle line:84% I'll tell you about some other topics 01:11:58.320 --> 01:12:01.720 align:middle line:84% that's connected to, namely, cryptography and secret sharing. 01:12:01.720 --> 01:12:03.260 align:middle line:84% And then the semester will be over. 01:12:03.260 --> 01:12:05.135 align:middle line:84% So just to give you a little bit of a preview 01:12:05.135 --> 01:12:08.040 align:middle line:84% of what we're going to cover in the remaining lectures. 01:12:08.040 --> 01:12:10.040 align:middle line:84% But let's stop here, and I'll take any questions 01:12:10.040 --> 01:12:11.960 align:middle line:90% you guys have offline. 01:12:11.960 --> 01:12:13.510 align:middle line:90% Thanks. 01:12:13.510 --> 01:12:20.000 align:middle line:90%