WEBVTT 00:00:00.000 --> 00:00:02.455 align:middle line:90% [SQUEAKING] 00:00:02.455 --> 00:00:04.419 align:middle line:90% [RUSTLING] 00:00:04.419 --> 00:00:06.874 align:middle line:90% [CLICKING] 00:00:06.874 --> 00:00:11.753 align:middle line:90% 00:00:11.753 --> 00:00:13.420 align:middle line:84% ANKUR MOITRA: So remember, last lecture, 00:00:13.420 --> 00:00:16.079 align:middle line:84% I introduced you to the idea of tail bounds, 00:00:16.079 --> 00:00:19.000 align:middle line:84% where we want to get a rough handle on how likely it 00:00:19.000 --> 00:00:23.080 align:middle line:84% is for a random variable to be far away from its expectation. 00:00:23.080 --> 00:00:28.000 align:middle line:84% So we talked about Markov's bound, Chebyshev's bound. 00:00:28.000 --> 00:00:31.200 align:middle line:84% And each of these used more and more information 00:00:31.200 --> 00:00:33.720 align:middle line:84% about the random variables as a way 00:00:33.720 --> 00:00:37.120 align:middle line:84% to get a better bound on the tail probabilities. 00:00:37.120 --> 00:00:39.280 align:middle line:84% So Markov just used that the random variable 00:00:39.280 --> 00:00:40.800 align:middle line:90% was non-negative. 00:00:40.800 --> 00:00:43.240 align:middle line:84% And then we already could bound the probability 00:00:43.240 --> 00:00:46.320 align:middle line:84% that it was double its expectation. 00:00:46.320 --> 00:00:48.320 align:middle line:84% Chebyshev, we used its variance, which 00:00:48.320 --> 00:00:51.760 align:middle line:84% is a measure of how spread out the random variable is. 00:00:51.760 --> 00:00:58.080 align:middle line:84% And today, we're going to do the most important tool, the highest 00:00:58.080 --> 00:01:01.540 align:middle line:84% power tool, which is called the Chernoff bound. 00:01:01.540 --> 00:01:05.620 align:middle line:84% So this is going to use that not only do we know properties 00:01:05.620 --> 00:01:08.780 align:middle line:84% about x's expectation and its variance, 00:01:08.780 --> 00:01:12.140 align:middle line:84% but that x itself is composed of the sum of mutually 00:01:12.140 --> 00:01:14.620 align:middle line:90% independent random variables. 00:01:14.620 --> 00:01:17.420 align:middle line:84% So we'll see that implicitly in the proof of this. 00:01:17.420 --> 00:01:19.780 align:middle line:84% We're going to be using not just the first moment, which 00:01:19.780 --> 00:01:21.880 align:middle line:84% is the mean, and the second moment, 00:01:21.880 --> 00:01:25.860 align:middle line:84% but we're going to be using all of the higher moments at once. 00:01:25.860 --> 00:01:28.460 align:middle line:84% But the basic setup for the Chernoff bound, at least 00:01:28.460 --> 00:01:31.980 align:middle line:84% the plain vanilla version, is-- remember from last time, 00:01:31.980 --> 00:01:34.260 align:middle line:84% we have these random variables, xi, 00:01:34.260 --> 00:01:35.840 align:middle line:90% that are mutually independent. 00:01:35.840 --> 00:01:37.260 align:middle line:90% So this is critical. 00:01:37.260 --> 00:01:40.320 align:middle line:84% Chernoff bound only works when you have mutual independence, 00:01:40.320 --> 00:01:41.440 align:middle line:90% not just pairwise. 00:01:41.440 --> 00:01:43.940 align:middle line:90% It's completely false otherwise. 00:01:43.940 --> 00:01:45.620 align:middle line:84% And these xi's, we're going to assume 00:01:45.620 --> 00:01:49.140 align:middle line:84% for simplicity that they're Bernoulli random variables. 00:01:49.140 --> 00:01:52.940 align:middle line:84% So each of the xi's has probability pi of being 1 00:01:52.940 --> 00:01:54.780 align:middle line:90% and is otherwise 0. 00:01:54.780 --> 00:01:57.910 align:middle line:84% And just to have some notation, we'll let mu denote 00:01:57.910 --> 00:02:02.060 align:middle line:84% the expectation of x, which is the sum of all the xi's. 00:02:02.060 --> 00:02:04.310 align:middle line:84% But of course, that's the same thing as the sum of all 00:02:04.310 --> 00:02:06.110 align:middle line:90% of the pi's. 00:02:06.110 --> 00:02:09.130 align:middle line:84% So the Chernoff bound comes in a lot of different flavors. 00:02:09.130 --> 00:02:13.110 align:middle line:84% Let me state a few different versions you might care about. 00:02:13.110 --> 00:02:16.590 align:middle line:84% First, there's what's called the upper tail, which 00:02:16.590 --> 00:02:20.510 align:middle line:84% this is the probability that the random variable is way larger 00:02:20.510 --> 00:02:22.110 align:middle line:90% than its expectation. 00:02:22.110 --> 00:02:27.390 align:middle line:84% So for all delta greater than 0, we can look at the probability 00:02:27.390 --> 00:02:32.810 align:middle line:84% that x is larger than its mean by a 1 plus delta factor. 00:02:32.810 --> 00:02:34.310 align:middle line:84% So you should think about delta here 00:02:34.310 --> 00:02:38.030 align:middle line:90% as being something like 0.1. 00:02:38.030 --> 00:02:43.190 align:middle line:84% The chance that this happens is exponentially small. 00:02:43.190 --> 00:02:46.790 align:middle line:84% And it depends on this delta parameter, of course. 00:02:46.790 --> 00:02:50.430 align:middle line:90% But then it also depends on mu. 00:02:50.430 --> 00:02:52.770 align:middle line:84% So this is a complicated algebraic expression. 00:02:52.770 --> 00:02:55.030 align:middle line:84% But it's important to have some intuition for what's 00:02:55.030 --> 00:02:56.410 align:middle line:90% going on here. 00:02:56.410 --> 00:02:59.010 align:middle line:84% So if I take a fair coin and I flip it, 00:02:59.010 --> 00:03:02.090 align:middle line:84% I know the probability that it's heads is 50. 00:03:02.090 --> 00:03:03.930 align:middle line:90% It's 50%. 00:03:03.930 --> 00:03:08.090 align:middle line:84% And intuitively, if I flip my coin a whole bunch of times 00:03:08.090 --> 00:03:10.810 align:middle line:84% and keep track of how many heads I get, 00:03:10.810 --> 00:03:13.650 align:middle line:84% I should expect that the empirical average of the number 00:03:13.650 --> 00:03:17.290 align:middle line:84% of heads is very quickly going to 50/50. 00:03:17.290 --> 00:03:21.570 align:middle line:84% Maybe if I flip it three times, I'll get two heads out of three. 00:03:21.570 --> 00:03:24.050 align:middle line:84% And that'll be far away from 50/50. 00:03:24.050 --> 00:03:26.530 align:middle line:84% But the larger the number of flips, 00:03:26.530 --> 00:03:28.370 align:middle line:84% well, the more it should be driven 00:03:28.370 --> 00:03:30.390 align:middle line:90% to going back to the average. 00:03:30.390 --> 00:03:33.010 align:middle line:84% That's what things like the weak law of large numbers 00:03:33.010 --> 00:03:34.570 align:middle line:90% are telling us. 00:03:34.570 --> 00:03:38.850 align:middle line:84% And the way that this shows up is really in this mean mu 00:03:38.850 --> 00:03:42.310 align:middle line:84% because, as the number of flips n gets larger and larger, 00:03:42.310 --> 00:03:44.250 align:middle line:90% my mean is larger and larger. 00:03:44.250 --> 00:03:47.330 align:middle line:84% So the chance that I have 10% too many heads 00:03:47.330 --> 00:03:52.370 align:middle line:84% is going down exponentially with the number of flips I make. 00:03:52.370 --> 00:03:55.540 align:middle line:84% So this is the critical distinction between the Chernoff 00:03:55.540 --> 00:03:58.780 align:middle line:84% bound and things like the weak law of large numbers, 00:03:58.780 --> 00:04:01.460 align:middle line:84% is that, here, our failure probability, the chance 00:04:01.460 --> 00:04:05.900 align:middle line:84% that we're outside this range where we think it lands, 00:04:05.900 --> 00:04:09.220 align:middle line:84% is exponentially small in n, whereas for things 00:04:09.220 --> 00:04:11.980 align:middle line:84% like the weak law of large numbers, it was only inverse 00:04:11.980 --> 00:04:14.420 align:middle line:90% polynomially small in n. 00:04:14.420 --> 00:04:17.380 align:middle line:84% So that's called the upper tail because all I care about 00:04:17.380 --> 00:04:21.060 align:middle line:84% is the probability that x is too big compared to mu. 00:04:21.060 --> 00:04:23.540 align:middle line:84% There's a corresponding lower tail. 00:04:23.540 --> 00:04:26.300 align:middle line:84% The only reason we separate these two things out 00:04:26.300 --> 00:04:30.100 align:middle line:84% is because they have a slightly different dependence on delta. 00:04:30.100 --> 00:04:32.940 align:middle line:84% So here, it really only makes sense 00:04:32.940 --> 00:04:37.340 align:middle line:84% to have a delta that's between 0 and 1. 00:04:37.340 --> 00:04:43.100 align:middle line:84% And now, the probability that x is at most 1 minus delta times 00:04:43.100 --> 00:04:46.260 align:middle line:84% mu-- that I get to few heads instead-- 00:04:46.260 --> 00:04:49.300 align:middle line:84% is, again, something that's exponentially small. 00:04:49.300 --> 00:04:53.640 align:middle line:84% It just doesn't have this 2 plus delta in the base. 00:04:53.640 --> 00:04:56.280 align:middle line:90% So it looks like this instead. 00:04:56.280 --> 00:04:58.160 align:middle line:90% So this is the lower tail. 00:04:58.160 --> 00:05:01.600 align:middle line:84% And there's also a way that you can combine them, 00:05:01.600 --> 00:05:04.320 align:middle line:84% if you don't have to worry that much about what 00:05:04.320 --> 00:05:06.560 align:middle line:90% the dependence on delta is. 00:05:06.560 --> 00:05:09.300 align:middle line:84% So what I claim is that when you just put these bounds together-- 00:05:09.300 --> 00:05:12.640 align:middle line:84% so for all delta between 0 and 1, 00:05:12.640 --> 00:05:16.680 align:middle line:84% you can just write it instead, the probability that x minus mu, 00:05:16.680 --> 00:05:19.980 align:middle line:84% in absolute value, is larger than delta mu-- 00:05:19.980 --> 00:05:23.600 align:middle line:84% so it's more than delta mu away from its expectation-- 00:05:23.600 --> 00:05:31.000 align:middle line:84% goes down like 2 times e to the minus mu delta squared over 3. 00:05:31.000 --> 00:05:34.540 align:middle line:84% So Chernoff bounds aren't really just one bound. 00:05:34.540 --> 00:05:36.720 align:middle line:84% There's a whole family of bounds. 00:05:36.720 --> 00:05:38.800 align:middle line:84% The ones I'm stating here, these are 00:05:38.800 --> 00:05:41.280 align:middle line:90% for Bernoulli random variables. 00:05:41.280 --> 00:05:47.160 align:middle line:84% And we have an upper tail, a lower tail, a combined bound. 00:05:47.160 --> 00:05:49.320 align:middle line:84% In fact, there are versions of Chernoff bounds 00:05:49.320 --> 00:05:53.050 align:middle line:84% where the xi's aren't Bernoulli random variables. 00:05:53.050 --> 00:05:55.170 align:middle line:84% There are other kinds of random variables 00:05:55.170 --> 00:05:57.370 align:middle line:90% with bounded second moments. 00:05:57.370 --> 00:06:00.290 align:middle line:84% So the way you should think about Chernoff bounds 00:06:00.290 --> 00:06:03.070 align:middle line:84% is I'm going to give you one proof of the Chernoff bound. 00:06:03.070 --> 00:06:06.890 align:middle line:84% It'll be in this restricted case of Bernoulli random variables. 00:06:06.890 --> 00:06:10.890 align:middle line:84% The same basic proof works for a lot of other exotic variants 00:06:10.890 --> 00:06:12.290 align:middle line:90% of the Chernoff bound. 00:06:12.290 --> 00:06:14.210 align:middle line:84% And when you need a different kind of Chernoff 00:06:14.210 --> 00:06:16.690 align:middle line:84% bound, like your xi's are not Bernoulli, 00:06:16.690 --> 00:06:19.010 align:middle line:84% you just look it up and quote the bound. 00:06:19.010 --> 00:06:20.970 align:middle line:84% So the key thing is really just to understand 00:06:20.970 --> 00:06:23.450 align:middle line:84% one of these bounds, how the proof works, 00:06:23.450 --> 00:06:26.890 align:middle line:84% and intuitively why the proof should extend beyond literally 00:06:26.890 --> 00:06:29.290 align:middle line:90% the assumptions I'm making. 00:06:29.290 --> 00:06:33.170 align:middle line:84% So are there any questions about it? 00:06:33.170 --> 00:06:39.010 align:middle line:84% So this is probably the most complicated proof 00:06:39.010 --> 00:06:41.312 align:middle line:90% we've done up until this point. 00:06:41.312 --> 00:06:43.770 align:middle line:84% And it's something which we're going to spend a bit of time 00:06:43.770 --> 00:06:45.390 align:middle line:90% in recitation thinking about. 00:06:45.390 --> 00:06:46.110 align:middle line:90% Yeah? 00:06:46.110 --> 00:06:49.450 align:middle line:84% STUDENT: Does mu have to be positive? 00:06:49.450 --> 00:06:53.110 align:middle line:84% ANKUR MOITRA: So mu is positive here just by assumption 00:06:53.110 --> 00:06:55.830 align:middle line:84% because my xi's are non-negative. 00:06:55.830 --> 00:06:58.562 align:middle line:84% I mean, if mu were 0, it would be kind of trivial. 00:06:58.562 --> 00:07:00.270 align:middle line:84% So you have to be a bit more careful when 00:07:00.270 --> 00:07:05.030 align:middle line:84% you have a Chernoff bound with random variables which are not 00:07:05.030 --> 00:07:07.550 align:middle line:90% non-negative. 00:07:07.550 --> 00:07:09.330 align:middle line:84% Yeah, you have to shift and scale it. 00:07:09.330 --> 00:07:11.788 align:middle line:84% And you'll see through the proof that a lot of these things 00:07:11.788 --> 00:07:12.550 align:middle line:90% can be done. 00:07:12.550 --> 00:07:15.430 align:middle line:84% It's just that it changes subtleties like what exactly 00:07:15.430 --> 00:07:17.630 align:middle line:90% happens in the exponent here. 00:07:17.630 --> 00:07:20.250 align:middle line:84% So there might be other quantities that show up. 00:07:20.250 --> 00:07:22.550 align:middle line:84% But the important thing is really the mu 00:07:22.550 --> 00:07:24.670 align:middle line:90% and the delta dependence. 00:07:24.670 --> 00:07:27.250 align:middle line:84% And then the other scalars might change in different ways. 00:07:27.250 --> 00:07:28.430 align:middle line:90% But good question. 00:07:28.430 --> 00:07:29.230 align:middle line:90% All right. 00:07:29.230 --> 00:07:32.770 align:middle line:84% So we're going to cover this in more detail in the recitation. 00:07:32.770 --> 00:07:33.730 align:middle line:90% But I'm warning you. 00:07:33.730 --> 00:07:35.870 align:middle line:84% So this will be one of the most complicated proofs 00:07:35.870 --> 00:07:38.212 align:middle line:90% we've done up until this point. 00:07:38.212 --> 00:07:39.670 align:middle line:84% But what I'm going to try and do is 00:07:39.670 --> 00:07:43.430 align:middle line:84% I'm going to explain the high level strategy for the proof. 00:07:43.430 --> 00:07:45.470 align:middle line:84% And that way, we can then try and fill 00:07:45.470 --> 00:07:48.600 align:middle line:84% in some of the technical details, the actual estimates 00:07:48.600 --> 00:07:50.960 align:middle line:84% to get the overall strategy to work. 00:07:50.960 --> 00:07:52.680 align:middle line:84% So what I really want to focus on 00:07:52.680 --> 00:07:56.840 align:middle line:84% is how we're going to prove this bound at a high level 00:07:56.840 --> 00:08:01.160 align:middle line:84% because, in all proofs, there are the hard parts of how you're 00:08:01.160 --> 00:08:03.120 align:middle line:84% going to fit the pieces together, 00:08:03.120 --> 00:08:05.080 align:middle line:84% and then there are the tedious parts of how 00:08:05.080 --> 00:08:08.200 align:middle line:84% do you do the literal computations to execute 00:08:08.200 --> 00:08:10.720 align:middle line:84% the different steps of your strategy. 00:08:10.720 --> 00:08:13.200 align:middle line:84% So let's start with the main idea. 00:08:13.200 --> 00:08:15.240 align:middle line:84% And this will be a very similar trick 00:08:15.240 --> 00:08:18.520 align:middle line:84% to what happened when we proved the Chebyshev bound. 00:08:18.520 --> 00:08:21.560 align:middle line:84% So let's say I have my random variable x. 00:08:21.560 --> 00:08:24.240 align:middle line:84% And for a that I'm going to set later, 00:08:24.240 --> 00:08:27.720 align:middle line:84% because I'm going to set it to be equal to this 1 plus delta mu 00:08:27.720 --> 00:08:30.680 align:middle line:84% or 1 minus delta, that kind of thing, 00:08:30.680 --> 00:08:32.720 align:middle line:84% let's say I'm interested in the probability 00:08:32.720 --> 00:08:36.720 align:middle line:84% that x is very large, that it's at least some value a. 00:08:36.720 --> 00:08:41.320 align:middle line:84% I claim that this is the same thing as the probability 00:08:41.320 --> 00:08:46.860 align:middle line:84% that e to the bx is at least e to the ba. 00:08:46.860 --> 00:08:52.020 align:middle line:84% And this is true for all b that are non-negative. 00:08:52.020 --> 00:08:55.980 align:middle line:90% So this is the first key trick. 00:08:55.980 --> 00:08:59.420 align:middle line:84% So remember, when we had the proof of the Chebyshev bound, 00:08:59.420 --> 00:09:02.580 align:middle line:84% Chebyshev doesn't assume any kind of non-negativity 00:09:02.580 --> 00:09:04.260 align:middle line:90% about the random variable. 00:09:04.260 --> 00:09:05.900 align:middle line:84% And I had this trick where when I 00:09:05.900 --> 00:09:09.420 align:middle line:84% wanted to look at the probability that x is too large, 00:09:09.420 --> 00:09:11.040 align:middle line:90% I squared both sides. 00:09:11.040 --> 00:09:14.100 align:middle line:84% And then I defined a new random variable y. 00:09:14.100 --> 00:09:17.555 align:middle line:84% And I said that that random variable y is non-negative. 00:09:17.555 --> 00:09:19.180 align:middle line:84% And I just said those two probabilities 00:09:19.180 --> 00:09:22.500 align:middle line:84% are the same because they're literally the same event. 00:09:22.500 --> 00:09:25.420 align:middle line:84% So that's exactly what's happening here. 00:09:25.420 --> 00:09:28.500 align:middle line:84% You can take b to be 1, just for simplicity. 00:09:28.500 --> 00:09:32.540 align:middle line:84% I'm just saying that these two events are the same. 00:09:32.540 --> 00:09:35.900 align:middle line:84% x being at least a is the same thing as e 00:09:35.900 --> 00:09:38.380 align:middle line:84% to the x being at least e to the a. 00:09:38.380 --> 00:09:44.430 align:middle line:84% And that's just because e is a nice, monotone function. 00:09:44.430 --> 00:09:47.390 align:middle line:84% And this is true now even if I put in a scalar b 00:09:47.390 --> 00:09:50.070 align:middle line:84% in there, as long as b is non-negative. 00:09:50.070 --> 00:09:51.810 align:middle line:84% So is everyone on board with this? 00:09:51.810 --> 00:09:53.590 align:middle line:90% This is fine? 00:09:53.590 --> 00:09:54.230 align:middle line:90% Yeah? 00:09:54.230 --> 00:09:57.020 align:middle line:84% STUDENT: Why does b need to be strictly greater than 0? 00:09:57.020 --> 00:10:02.985 align:middle line:90% 00:10:02.985 --> 00:10:04.110 align:middle line:90% ANKUR MOITRA: I don't know. 00:10:04.110 --> 00:10:05.490 align:middle line:90% Let's just make it strict. 00:10:05.490 --> 00:10:07.510 align:middle line:84% I'm not going to use the equality anyways. 00:10:07.510 --> 00:10:09.050 align:middle line:84% And we'll get into a technicality. 00:10:09.050 --> 00:10:10.750 align:middle line:90% But thanks for the catch. 00:10:10.750 --> 00:10:14.190 align:middle line:90% Any other questions? 00:10:14.190 --> 00:10:15.390 align:middle line:90% All right. 00:10:15.390 --> 00:10:17.110 align:middle line:90% So that's the first idea. 00:10:17.110 --> 00:10:22.120 align:middle line:84% So just to belabor the point right, so why is this true? 00:10:22.120 --> 00:10:24.630 align:middle line:90% 00:10:24.630 --> 00:10:29.680 align:middle line:84% It's true because e to the y is an increasing function. 00:10:29.680 --> 00:10:37.230 align:middle line:90% 00:10:37.230 --> 00:10:39.510 align:middle line:84% And so that really means that these two events 00:10:39.510 --> 00:10:44.270 align:middle line:84% are the same-- the set of all x's where x is at least a 00:10:44.270 --> 00:10:49.130 align:middle line:84% are the same as the set of all x's where e to the bx 00:10:49.130 --> 00:10:52.410 align:middle line:90% is at least e to the ba. 00:10:52.410 --> 00:10:54.310 align:middle line:84% So they are literally the same event. 00:10:54.310 --> 00:10:57.330 align:middle line:84% So this is the first thing in our trick. 00:10:57.330 --> 00:11:00.875 align:middle line:84% And so now, exactly like in the proof of the Chebyshev bound, 00:11:00.875 --> 00:11:02.250 align:middle line:84% what I'm going to do is I'm going 00:11:02.250 --> 00:11:09.130 align:middle line:84% to define a new random variable y which is e to the bx. 00:11:09.130 --> 00:11:11.070 align:middle line:84% So this is a really interesting trick. 00:11:11.070 --> 00:11:14.170 align:middle line:84% Just like for Chebyshev, I defined a new random variable 00:11:14.170 --> 00:11:18.530 align:middle line:84% y, which was the square compared to the expectation. 00:11:18.530 --> 00:11:20.170 align:middle line:84% Now, I'm doing this funny thing where 00:11:20.170 --> 00:11:23.610 align:middle line:84% I take the exponential of my random variable. 00:11:23.610 --> 00:11:28.122 align:middle line:84% And I'm using that to define this new random variable y. 00:11:28.122 --> 00:11:34.150 align:middle line:84% So now, just by construction, y is non-negative. 00:11:34.150 --> 00:11:38.866 align:middle line:90% 00:11:38.866 --> 00:11:42.100 align:middle line:84% It's a non-negative random variable, which 00:11:42.100 --> 00:11:43.590 align:middle line:90% means we can apply Markov. 00:11:43.590 --> 00:11:49.900 align:middle line:90% 00:11:49.900 --> 00:11:53.260 align:middle line:84% So let's apply Markov and see what we get. 00:11:53.260 --> 00:12:00.700 align:middle line:84% So the probability that x is at least a 00:12:00.700 --> 00:12:09.780 align:middle line:84% is at most e expectation of e to the bx over e to the ba. 00:12:09.780 --> 00:12:13.360 align:middle line:90% 00:12:13.360 --> 00:12:17.980 align:middle line:84% So what this is doing right here is I'm just using two facts. 00:12:17.980 --> 00:12:21.740 align:middle line:84% So first of all, this one right here is just the probability 00:12:21.740 --> 00:12:25.860 align:middle line:90% that y is at least e to the ba. 00:12:25.860 --> 00:12:30.380 align:middle line:84% And I'm using the fact that, if I had probability of y at least 00:12:30.380 --> 00:12:34.220 align:middle line:84% e to the ba, this would be my expectation of y. 00:12:34.220 --> 00:12:35.680 align:middle line:90% This would be my threshold. 00:12:35.680 --> 00:12:37.440 align:middle line:90% So this is just Markov. 00:12:37.440 --> 00:12:41.640 align:middle line:84% And now, I'm using this fact that these two probabilities are 00:12:41.640 --> 00:12:45.040 align:middle line:84% the same to replace my left-hand side with the probability x 00:12:45.040 --> 00:12:47.240 align:middle line:90% is at least a. 00:12:47.240 --> 00:12:49.000 align:middle line:90% So this is subtle. 00:12:49.000 --> 00:12:51.560 align:middle line:90% Does everyone follow that? 00:12:51.560 --> 00:12:54.720 align:middle line:90% Any questions? 00:12:54.720 --> 00:12:56.480 align:middle line:84% Give me a thumbs up if it makes sense. 00:12:56.480 --> 00:12:58.400 align:middle line:90% OK, good. 00:12:58.400 --> 00:12:59.920 align:middle line:90% All right. 00:12:59.920 --> 00:13:04.820 align:middle line:84% So now, it turns out that the quantity on the right-hand side, 00:13:04.820 --> 00:13:10.400 align:middle line:84% this thing right here, this is important for its own right. 00:13:10.400 --> 00:13:13.840 align:middle line:84% It actually has a very special name and a special role. 00:13:13.840 --> 00:13:16.500 align:middle line:84% So before we go on with the proof, 00:13:16.500 --> 00:13:19.420 align:middle line:84% let me explain what this quantity is 00:13:19.420 --> 00:13:21.560 align:middle line:84% and why it's so important, because otherwise it 00:13:21.560 --> 00:13:23.720 align:middle line:90% will seem a bit too magical. 00:13:23.720 --> 00:13:26.290 align:middle line:84% So this is called the moment-generating function. 00:13:26.290 --> 00:13:28.800 align:middle line:90% 00:13:28.800 --> 00:13:32.960 align:middle line:84% So the moment-generating function-- 00:13:32.960 --> 00:13:39.686 align:middle line:84% we've seen generating functions in the context of combinatorics. 00:13:39.686 --> 00:13:43.290 align:middle line:84% And here, it's a generating function in a different disguise 00:13:43.290 --> 00:13:45.857 align:middle line:84% because it'll be keeping track of other things. 00:13:45.857 --> 00:13:47.690 align:middle line:84% So when we talked about generating functions 00:13:47.690 --> 00:13:51.570 align:middle line:84% in the context of combinatorics, the coefficients that showed up 00:13:51.570 --> 00:13:53.890 align:middle line:90% in different powers of a-- 00:13:53.890 --> 00:13:56.970 align:middle line:84% different powers of x, that kept track of things we 00:13:56.970 --> 00:13:58.850 align:middle line:90% wanted to count. 00:13:58.850 --> 00:14:02.450 align:middle line:84% So it turns out that this quantity right here 00:14:02.450 --> 00:14:07.410 align:middle line:84% keeps track of important quantities of x. 00:14:07.410 --> 00:14:11.170 align:middle line:84% But the way it keeps track of it is that there's a free parameter 00:14:11.170 --> 00:14:12.130 align:middle line:90% b. 00:14:12.130 --> 00:14:17.090 align:middle line:84% So what I really care about is how this term depends on b. 00:14:17.090 --> 00:14:19.850 align:middle line:84% And so it turns out that the series expansion 00:14:19.850 --> 00:14:24.370 align:middle line:84% I get in terms of b keeps track a lot of interesting properties 00:14:24.370 --> 00:14:25.970 align:middle line:90% of the random variable. 00:14:25.970 --> 00:14:29.050 align:middle line:84% That's why we call this the moment-generating function. 00:14:29.050 --> 00:14:31.770 align:middle line:84% So the moment-generating function of x 00:14:31.770 --> 00:14:36.070 align:middle line:84% is literally just the thing on the right-hand side. 00:14:36.070 --> 00:14:38.590 align:middle line:90% We denote it by mx of b. 00:14:38.590 --> 00:14:43.870 align:middle line:84% And it's equal to the expectation of e to the bx. 00:14:43.870 --> 00:14:46.210 align:middle line:84% So even though x is our random variable, 00:14:46.210 --> 00:14:47.790 align:middle line:84% that's not the thing that shows up 00:14:47.790 --> 00:14:49.870 align:middle line:84% in the moment-generating function 00:14:49.870 --> 00:14:52.790 align:middle line:84% because I'm taking the expectation over x. 00:14:52.790 --> 00:14:54.230 align:middle line:84% The important thing is really just 00:14:54.230 --> 00:14:57.950 align:middle line:84% how that expectation depends on b. 00:14:57.950 --> 00:15:00.270 align:middle line:84% So let's tease out these properties 00:15:00.270 --> 00:15:02.210 align:middle line:84% of the moment-generating function, 00:15:02.210 --> 00:15:05.830 align:middle line:84% just to get some intuition for what properties 00:15:05.830 --> 00:15:09.550 align:middle line:84% of the random variable this proof of the Chernoff bound 00:15:09.550 --> 00:15:10.830 align:middle line:90% is using. 00:15:10.830 --> 00:15:12.690 align:middle line:90% So let's do some intuition. 00:15:12.690 --> 00:15:17.430 align:middle line:90% 00:15:17.430 --> 00:15:22.790 align:middle line:84% So first of all, mx of b is a function of b. 00:15:22.790 --> 00:15:26.630 align:middle line:84% In fact, one way that we could think about it is we 00:15:26.630 --> 00:15:31.430 align:middle line:84% could take the Taylor expansion of e to the bx. 00:15:31.430 --> 00:15:34.080 align:middle line:84% So if we take the Taylor expansion of e to the bx 00:15:34.080 --> 00:15:36.080 align:middle line:84% because, inside the expectation, x 00:15:36.080 --> 00:15:39.480 align:middle line:84% is just some particular realization. 00:15:39.480 --> 00:15:46.680 align:middle line:84% And it could look like 1 plus bx plus 1 over 2 factorial b 00:15:46.680 --> 00:15:51.640 align:middle line:84% squared x squared plus 1 over 3 factorial 00:15:51.640 --> 00:15:56.640 align:middle line:90% b cubed x cubed and so on. 00:15:56.640 --> 00:15:58.640 align:middle line:84% So if we just take the Taylor expansion 00:15:58.640 --> 00:16:01.920 align:middle line:84% of our exponential function, this is what we get. 00:16:01.920 --> 00:16:05.840 align:middle line:84% And now, we can use linearity of expectation. 00:16:05.840 --> 00:16:08.080 align:middle line:84% And this is the same thing as the sum from i 00:16:08.080 --> 00:16:14.200 align:middle line:84% equals 0 to infinity of 1 over i factorial b to the i 00:16:14.200 --> 00:16:19.520 align:middle line:90% the expectation of x to the i. 00:16:19.520 --> 00:16:22.240 align:middle line:90% So this is what I promised you. 00:16:22.240 --> 00:16:24.000 align:middle line:84% So the moment-generating function 00:16:24.000 --> 00:16:28.360 align:middle line:84% is this generating function that depends on b. 00:16:28.360 --> 00:16:31.900 align:middle line:84% And what is the first term of this? 00:16:31.900 --> 00:16:34.740 align:middle line:84% Well, it's trivial when I equals 0 00:16:34.740 --> 00:16:36.940 align:middle line:90% because I just get 1 out of it. 00:16:36.940 --> 00:16:41.540 align:middle line:84% But now, when I do the next term corresponding to i equals 1, 00:16:41.540 --> 00:16:42.600 align:middle line:90% what am I getting? 00:16:42.600 --> 00:16:47.500 align:middle line:84% Well, the coefficient of b is the expectation of x. 00:16:47.500 --> 00:16:50.860 align:middle line:84% And when I look at i equals 2, it's not the variance of x 00:16:50.860 --> 00:16:54.420 align:middle line:84% But it's the second moment of x, and so on, 00:16:54.420 --> 00:16:55.900 align:middle line:90% and the third moments. 00:16:55.900 --> 00:16:57.340 align:middle line:84% So the moment-generating function 00:16:57.340 --> 00:17:00.780 align:middle line:90% keeps track of the moments of x. 00:17:00.780 --> 00:17:02.860 align:middle line:84% And in fact, what I claim is that you 00:17:02.860 --> 00:17:09.380 align:middle line:84% can extract the moments from the moment-generating function. 00:17:09.380 --> 00:17:11.610 align:middle line:84% So we'll see how this works just in a simple case. 00:17:11.610 --> 00:17:16.220 align:middle line:90% 00:17:16.220 --> 00:17:18.260 align:middle line:84% And this will parallel what Peter taught you 00:17:18.260 --> 00:17:20.940 align:middle line:90% about generating functions. 00:17:20.940 --> 00:17:23.260 align:middle line:84% So how could I go about extracting 00:17:23.260 --> 00:17:32.510 align:middle line:84% a particular moment of x from the moment-generating function? 00:17:32.510 --> 00:17:34.710 align:middle line:84% Well, in particular, the way that I 00:17:34.710 --> 00:17:38.550 align:middle line:84% can recover the first moment of x 00:17:38.550 --> 00:17:46.070 align:middle line:84% is just by taking my moment-generating function, 00:17:46.070 --> 00:17:49.230 align:middle line:84% taking the partial derivative with respect to b, 00:17:49.230 --> 00:17:53.470 align:middle line:84% and then evaluating it at b equals 0. 00:17:53.470 --> 00:17:56.150 align:middle line:84% All that would do is it would kill the first term that's 00:17:56.150 --> 00:17:57.430 align:middle line:90% just a constant. 00:17:57.430 --> 00:18:00.510 align:middle line:84% And I just pick up the expectation. 00:18:00.510 --> 00:18:03.750 align:middle line:84% And then when I evaluate it at b equals 0, 00:18:03.750 --> 00:18:06.950 align:middle line:84% all of the higher order terms would disappear. 00:18:06.950 --> 00:18:10.950 align:middle line:84% So you can check that this really does give you 00:18:10.950 --> 00:18:13.350 align:middle line:90% what you want right. 00:18:13.350 --> 00:18:19.170 align:middle line:84% In particular, the partial with respect to b of mx of b, 00:18:19.170 --> 00:18:21.070 align:middle line:84% we can just apply it to this expression 00:18:21.070 --> 00:18:23.190 align:middle line:90% that I wrote out right here. 00:18:23.190 --> 00:18:26.570 align:middle line:84% And that's the same thing as the partial with respect 00:18:26.570 --> 00:18:33.130 align:middle line:90% to b of 1 plus b bx and so on. 00:18:33.130 --> 00:18:35.330 align:middle line:84% And then we'll get this expression 00:18:35.330 --> 00:18:42.490 align:middle line:84% that looks like ex plus bex squared and so on. 00:18:42.490 --> 00:18:44.350 align:middle line:90% And we evaluate at b equals 0. 00:18:44.350 --> 00:18:47.410 align:middle line:90% We get exactly what we wanted. 00:18:47.410 --> 00:18:50.610 align:middle line:84% So for example, how could I extract the second moment 00:18:50.610 --> 00:18:53.912 align:middle line:84% of x from the moment-generating function, 00:18:53.912 --> 00:18:55.620 align:middle line:84% just to make sure we're on the same page? 00:18:55.620 --> 00:19:00.510 align:middle line:90% 00:19:00.510 --> 00:19:01.010 align:middle line:90% Yeah? 00:19:01.010 --> 00:19:02.610 align:middle line:84% STUDENT: You take the second derivative. 00:19:02.610 --> 00:19:03.670 align:middle line:90% ANKUR MOITRA: Second derivative. 00:19:03.670 --> 00:19:05.628 align:middle line:84% You just have to be careful with the constants. 00:19:05.628 --> 00:19:08.370 align:middle line:84% But they all work out nicely here. 00:19:08.370 --> 00:19:11.810 align:middle line:84% So this gives us some intuition about-- 00:19:11.810 --> 00:19:13.650 align:middle line:84% I told you that the Chernoff bound 00:19:13.650 --> 00:19:17.330 align:middle line:84% would be using higher order moments of my random variable. 00:19:17.330 --> 00:19:19.890 align:middle line:84% I'm not just using the first and second moment, like I 00:19:19.890 --> 00:19:22.130 align:middle line:90% am with Markov and Chebyshev. 00:19:22.130 --> 00:19:24.470 align:middle line:84% And the way that I'm using it is really 00:19:24.470 --> 00:19:26.830 align:middle line:84% what's happening on the right-hand side. 00:19:26.830 --> 00:19:28.990 align:middle line:84% That moment-generating function contains 00:19:28.990 --> 00:19:31.790 align:middle line:84% all of the information about x that we'll want. 00:19:31.790 --> 00:19:37.030 align:middle line:84% So now, let's get back to the proof 00:19:37.030 --> 00:19:40.330 align:middle line:84% and figure out why this is a productive thing to do. 00:19:40.330 --> 00:19:44.990 align:middle line:90% 00:19:44.990 --> 00:19:47.310 align:middle line:90% So here's our first key lemma. 00:19:47.310 --> 00:19:49.310 align:middle line:84% So in the setting of the Chernoff bound, 00:19:49.310 --> 00:19:55.390 align:middle line:84% where we have x equals the sum of these xi's and all 00:19:55.390 --> 00:20:08.110 align:middle line:84% these Xi's are mutually independent, then 00:20:08.110 --> 00:20:12.350 align:middle line:84% what I claim is that the moment-generating function of x 00:20:12.350 --> 00:20:16.910 align:middle line:84% evaluated at b is the same thing as the product of all 00:20:16.910 --> 00:20:20.670 align:middle line:84% of the constituent moment-generating functions 00:20:20.670 --> 00:20:25.120 align:middle line:84% for each one of the xi's all evaluated at the same thing, b. 00:20:25.120 --> 00:20:28.680 align:middle line:90% 00:20:28.680 --> 00:20:30.380 align:middle line:84% So that's the first key statement. 00:20:30.380 --> 00:20:32.520 align:middle line:84% This is why the moment-generating function plays 00:20:32.520 --> 00:20:34.920 align:middle line:84% nicely with mutual independence, is 00:20:34.920 --> 00:20:37.500 align:middle line:84% because it decomposes as a product of a lot of things. 00:20:37.500 --> 00:20:38.460 align:middle line:90% Was there a question? 00:20:38.460 --> 00:20:38.960 align:middle line:90% No? 00:20:38.960 --> 00:20:39.800 align:middle line:90% OK. 00:20:39.800 --> 00:20:40.840 align:middle line:90% All right. 00:20:40.840 --> 00:20:42.820 align:middle line:84% So the proof for this is very simple. 00:20:42.820 --> 00:20:47.020 align:middle line:84% It's just about unpacking what's happening in this definition. 00:20:47.020 --> 00:20:50.760 align:middle line:90% 00:20:50.760 --> 00:20:57.400 align:middle line:84% So by definition and our assumption, 00:20:57.400 --> 00:20:59.640 align:middle line:84% we just have to remember what the moment generating 00:20:59.640 --> 00:21:00.500 align:middle line:90% function is. 00:21:00.500 --> 00:21:06.140 align:middle line:84% So mx of b is defined to be the expectation of e to the bx. 00:21:06.140 --> 00:21:07.880 align:middle line:90% That's where it came from. 00:21:07.880 --> 00:21:14.400 align:middle line:84% And then we can remember what exactly x is because x is 00:21:14.400 --> 00:21:19.920 align:middle line:84% the sum from i equals 1 to n of the xi's. 00:21:19.920 --> 00:21:23.160 align:middle line:84% And now, what should I do in my last step right here? 00:21:23.160 --> 00:21:25.520 align:middle line:90% So how am I going to proceed? 00:21:25.520 --> 00:21:26.500 align:middle line:90% Yeah? 00:21:26.500 --> 00:21:28.820 align:middle line:84% STUDENT: You can break the [INAUDIBLE]. 00:21:28.820 --> 00:21:32.460 align:middle line:90% 00:21:32.460 --> 00:21:33.740 align:middle line:90% ANKUR MOITRA: That's right. 00:21:33.740 --> 00:21:40.620 align:middle line:84% So just to belabor this, let me write it out in two steps first. 00:21:40.620 --> 00:21:45.260 align:middle line:84% I can write this as the product from i equals 1 to n of each 00:21:45.260 --> 00:21:48.540 align:middle line:90% of these eb xi's. 00:21:48.540 --> 00:21:51.180 align:middle line:84% But now, the crucial thing is that, 00:21:51.180 --> 00:21:57.380 align:middle line:84% by mutual independence of the xi's, I actually know that not 00:21:57.380 --> 00:22:01.540 align:middle line:84% only are the xi's independent, but so are the e to the b xi's. 00:22:01.540 --> 00:22:02.220 align:middle line:90% As well. 00:22:02.220 --> 00:22:13.400 align:middle line:84% And so by mutual independence, we can pull the product outside. 00:22:13.400 --> 00:22:17.460 align:middle line:84% This is where I crucially need not just pairwise independence. 00:22:17.460 --> 00:22:21.030 align:middle line:84% And I'll get the product of all of these expectations of e 00:22:21.030 --> 00:22:22.870 align:middle line:90% to the b xi. 00:22:22.870 --> 00:22:24.950 align:middle line:84% And that's literally the product of 00:22:24.950 --> 00:22:30.950 align:middle line:84% their moment-generating functions exactly as I promised. 00:22:30.950 --> 00:22:33.270 align:middle line:84% So that's the first lemma, the first key thing 00:22:33.270 --> 00:22:35.530 align:middle line:84% we need for the ingredient for the Chernoff bound. 00:22:35.530 --> 00:22:38.430 align:middle line:90% 00:22:38.430 --> 00:22:39.730 align:middle line:90% Any questions? 00:22:39.730 --> 00:22:43.990 align:middle line:90% 00:22:43.990 --> 00:22:46.110 align:middle line:84% So this is the basic architecture 00:22:46.110 --> 00:22:48.390 align:middle line:84% of the proof for the Chernoff bound, 00:22:48.390 --> 00:22:51.230 align:middle line:84% is that we define this new random variable 00:22:51.230 --> 00:22:53.910 align:middle line:90% y that's the exponential of x. 00:22:53.910 --> 00:22:56.917 align:middle line:84% And then when we applied Markov, we 00:22:56.917 --> 00:22:58.750 align:middle line:84% ended up with the moment-generating function 00:22:58.750 --> 00:23:00.030 align:middle line:90% the right-hand side. 00:23:00.030 --> 00:23:01.470 align:middle line:84% And the moment generating function 00:23:01.470 --> 00:23:03.950 align:middle line:84% turns into the product of all of the constituent 00:23:03.950 --> 00:23:06.310 align:middle line:90% moment-generating functions. 00:23:06.310 --> 00:23:09.110 align:middle line:84% So now, what we need to do is we need to actually estimate 00:23:09.110 --> 00:23:12.470 align:middle line:84% the individual terms that go into the right-hand side. 00:23:12.470 --> 00:23:16.790 align:middle line:84% So now, the whole name of the game is executing the strategy 00:23:16.790 --> 00:23:19.450 align:middle line:84% and figuring out what the right-hand side looks 00:23:19.450 --> 00:23:22.890 align:middle line:84% like in this one bound that I wrote down right there. 00:23:22.890 --> 00:23:25.570 align:middle line:84% So now, we're actually going to use the properties of like 00:23:25.570 --> 00:23:27.050 align:middle line:90% the pi's. 00:23:27.050 --> 00:23:35.130 align:middle line:84% So let me tell you the second key lemma, 00:23:35.130 --> 00:23:38.810 align:middle line:84% which will be our second main ingredient in the proof. 00:23:38.810 --> 00:23:42.290 align:middle line:84% So let's figure out what these moment-generating functions are 00:23:42.290 --> 00:23:45.130 align:middle line:90% for each of these x sub i's. 00:23:45.130 --> 00:23:46.310 align:middle line:90% So here's what I claim. 00:23:46.310 --> 00:23:47.550 align:middle line:90% This is our lemma 2. 00:23:47.550 --> 00:23:51.170 align:middle line:90% 00:23:51.170 --> 00:23:56.250 align:middle line:90% So if x sub i is 1 or 0-- 00:23:56.250 --> 00:24:00.670 align:middle line:84% again, with probability pi and 0 otherwise, 00:24:00.670 --> 00:24:04.130 align:middle line:84% so same as the assumptions of the Chernoff bound-- 00:24:04.130 --> 00:24:09.170 align:middle line:84% then what I claim is that the moment-generating function of xi 00:24:09.170 --> 00:24:16.780 align:middle line:84% evaluated at b is at most e to the pi e to the b minus 1. 00:24:16.780 --> 00:24:21.460 align:middle line:84% So this is now just the numerical value for it. 00:24:21.460 --> 00:24:24.460 align:middle line:84% Obviously, it has to depend on what the probability 00:24:24.460 --> 00:24:28.700 align:middle line:84% pi is of xi being 1 for that Bernoulli random variable. 00:24:28.700 --> 00:24:30.100 align:middle line:84% And then it also depends on where 00:24:30.100 --> 00:24:32.620 align:middle line:84% we're evaluating the moment-generating function 00:24:32.620 --> 00:24:36.310 align:middle line:84% at because that shows up as a double exponential bound. 00:24:36.310 --> 00:24:38.980 align:middle line:90% 00:24:38.980 --> 00:24:40.620 align:middle line:84% So what we're going to do to prove 00:24:40.620 --> 00:24:54.980 align:middle line:84% this is we'll use the fact that 1 plus y is at most e to the y. 00:24:54.980 --> 00:24:59.500 align:middle line:90% And this is true for all y. 00:24:59.500 --> 00:25:01.060 align:middle line:84% And so this is a very simple thing 00:25:01.060 --> 00:25:02.680 align:middle line:90% to prove just with calculus. 00:25:02.680 --> 00:25:05.300 align:middle line:84% Or you can just plot it to check yourself. 00:25:05.300 --> 00:25:09.140 align:middle line:84% You plot 1 plus y and look at e to the y. 00:25:09.140 --> 00:25:16.200 align:middle line:84% And you'll see that the line is below the exponential curve. 00:25:16.200 --> 00:25:18.760 align:middle line:84% So this inequality is going to play a crucial role 00:25:18.760 --> 00:25:21.920 align:middle line:84% in getting a convenient form for the right-hand side. 00:25:21.920 --> 00:25:23.790 align:middle line:90% So now, let's prove this lemma. 00:25:23.790 --> 00:25:29.840 align:middle line:90% 00:25:29.840 --> 00:25:33.480 align:middle line:84% Well, we're going to prove this just by direct computation. 00:25:33.480 --> 00:25:36.000 align:middle line:84% And eventually, we're going to appeal to this fact 00:25:36.000 --> 00:25:40.440 align:middle line:84% in order to write it in a more convenient form. 00:25:40.440 --> 00:25:46.200 align:middle line:84% So what is the moment-generating function of xi? 00:25:46.200 --> 00:25:48.160 align:middle line:84% Well, we can write it out because there 00:25:48.160 --> 00:25:50.240 align:middle line:90% are only two possibilities. 00:25:50.240 --> 00:25:57.390 align:middle line:84% So either xi equals 1 or xi equals 0. 00:25:57.390 --> 00:26:02.600 align:middle line:90% 00:26:02.600 --> 00:26:05.480 align:middle line:84% In the case where it's 0, our quantity 00:26:05.480 --> 00:26:08.720 align:middle line:84% contributes a 1 because we get e to the 0. 00:26:08.720 --> 00:26:12.890 align:middle line:84% In the case where it's a 1, we get an e to the b. 00:26:12.890 --> 00:26:15.210 align:middle line:84% So this is just literally using the definition 00:26:15.210 --> 00:26:17.250 align:middle line:90% of the expectation. 00:26:17.250 --> 00:26:21.050 align:middle line:84% And now, we can plug- in for what the probability of xi 00:26:21.050 --> 00:26:22.570 align:middle line:90% being 1 is. 00:26:22.570 --> 00:26:24.530 align:middle line:84% So this gives us, on the right-hand side, 00:26:24.530 --> 00:26:31.250 align:middle line:90% pi e to the b plus 1 minus pi. 00:26:31.250 --> 00:26:35.130 align:middle line:84% And we can write this in a slightly more convenient form. 00:26:35.130 --> 00:26:40.610 align:middle line:84% We can write this out instead as 1 plus pi times e 00:26:40.610 --> 00:26:42.190 align:middle line:90% to the b minus 1. 00:26:42.190 --> 00:26:45.890 align:middle line:84% So I'm just collecting the two terms. 00:26:45.890 --> 00:26:48.450 align:middle line:84% And now, I'm almost done because I 00:26:48.450 --> 00:26:54.850 align:middle line:84% should use this fact I told you I needed about the exponential. 00:26:54.850 --> 00:26:58.010 align:middle line:84% So I'm going to think about my y variable 00:26:58.010 --> 00:27:01.450 align:middle line:84% as just being this pi quantity on the right-hand side. 00:27:01.450 --> 00:27:04.030 align:middle line:84% I have something that looks like 1 plus y. 00:27:04.030 --> 00:27:07.010 align:middle line:84% And I'm going to upper bound it by e to the y. 00:27:07.010 --> 00:27:11.990 align:middle line:84% So using our fact about exponentials, 00:27:11.990 --> 00:27:14.190 align:middle line:84% we get that the moment-generating function 00:27:14.190 --> 00:27:20.990 align:middle line:84% is now, at most, e to the pi times e to the B minus 1. 00:27:20.990 --> 00:27:24.430 align:middle line:84% So literally, all I'm doing is I'm appealing to the fact 00:27:24.430 --> 00:27:26.950 align:middle line:90% where this is my y here. 00:27:26.950 --> 00:27:31.670 align:middle line:84% And that's the end of the proof for lemma 2. 00:27:31.670 --> 00:27:33.990 align:middle line:84% So right now, it's not totally obvious 00:27:33.990 --> 00:27:36.790 align:middle line:84% why we want an expression like this. 00:27:36.790 --> 00:27:40.807 align:middle line:84% Why did we not just stop at the expression I had above? 00:27:40.807 --> 00:27:42.390 align:middle line:84% Let me give you a little foreshadowing 00:27:42.390 --> 00:27:43.612 align:middle line:90% of where we're going. 00:27:43.612 --> 00:27:46.070 align:middle line:84% See, at the end of the day, the way that the Chernoff bound 00:27:46.070 --> 00:27:48.510 align:middle line:84% works is that the right-hand side has this quantity 00:27:48.510 --> 00:27:53.070 align:middle line:84% mu in it, which is the average of your random variable. 00:27:53.070 --> 00:27:56.430 align:middle line:84% And that quantity mu is the sum of the pi's. 00:27:56.430 --> 00:27:59.390 align:middle line:84% So in our lemma 1, we're taking the product 00:27:59.390 --> 00:28:01.892 align:middle line:84% of all these moment-generating functions. 00:28:01.892 --> 00:28:03.350 align:middle line:84% And so what's going to happen is we 00:28:03.350 --> 00:28:05.790 align:middle line:84% take the product of terms that look like this, where 00:28:05.790 --> 00:28:08.360 align:middle line:90% the pi is happening upstairs. 00:28:08.360 --> 00:28:10.840 align:middle line:84% And then we're going to get some of the pi's. 00:28:10.840 --> 00:28:13.440 align:middle line:84% And that's where our mu is going to fall out. 00:28:13.440 --> 00:28:16.140 align:middle line:84% So if I kept that other bound over there, 00:28:16.140 --> 00:28:20.360 align:middle line:84% that would have been still true and just an even sharper bound. 00:28:20.360 --> 00:28:22.640 align:middle line:84% But it doesn't give me a convenient expression 00:28:22.640 --> 00:28:27.560 align:middle line:84% for actually getting the Chernoff bound that I want. 00:28:27.560 --> 00:28:29.790 align:middle line:84% So now, we can prove the Chernoff bound. 00:28:29.790 --> 00:28:34.480 align:middle line:90% 00:28:34.480 --> 00:28:37.800 align:middle line:84% And I'm just going to prove the upper tail. 00:28:37.800 --> 00:28:41.020 align:middle line:84% But you can check the notes for the proof of the lower tail. 00:28:41.020 --> 00:28:44.600 align:middle line:90% And it works very similarly. 00:28:44.600 --> 00:28:47.160 align:middle line:90% What I'm going to do is-- 00:28:47.160 --> 00:28:49.720 align:middle line:84% at the end of the day, I have to figure out 00:28:49.720 --> 00:28:53.560 align:middle line:84% how to execute this proof strategy because I 00:28:53.560 --> 00:28:56.800 align:middle line:84% have a whole bunch of parameters that I now need to set. 00:28:56.800 --> 00:29:00.520 align:middle line:84% So I have to set my threshold a in that 00:29:00.520 --> 00:29:02.520 align:middle line:90% bound that I wrote up there. 00:29:02.520 --> 00:29:06.620 align:middle line:84% I have to choose my parameter b, which is my helper parameter, so 00:29:06.620 --> 00:29:09.180 align:middle line:84% that I get a tight enough right-hand side that I 00:29:09.180 --> 00:29:12.500 align:middle line:84% get the upper tail for the Chernoff bound popping out. 00:29:12.500 --> 00:29:16.220 align:middle line:84% So let me just tell you what I'm going to choose for these a 00:29:16.220 --> 00:29:17.620 align:middle line:90% and b. 00:29:17.620 --> 00:29:20.220 align:middle line:84% So a is the obvious thing because it corresponds 00:29:20.220 --> 00:29:21.760 align:middle line:90% to the event we care about. 00:29:21.760 --> 00:29:24.700 align:middle line:84% So it's 1 plus delta times mu because that's our upper tail 00:29:24.700 --> 00:29:26.260 align:middle line:90% condition. 00:29:26.260 --> 00:29:30.660 align:middle line:84% And b we're just going to set to be equal to the natural log of 1 00:29:30.660 --> 00:29:32.420 align:middle line:90% plus delta. 00:29:32.420 --> 00:29:34.000 align:middle line:84% So this might look a little strange. 00:29:34.000 --> 00:29:35.940 align:middle line:84% But the way that you would prove this yourself 00:29:35.940 --> 00:29:38.620 align:middle line:84% or come up with this b is you would just 00:29:38.620 --> 00:29:41.420 align:middle line:84% work through the calculations I'm about to do, 00:29:41.420 --> 00:29:43.860 align:middle line:84% but keeping B as a free parameter. 00:29:43.860 --> 00:29:46.700 align:middle line:84% You would get a right-hand side that depends on b. 00:29:46.700 --> 00:29:48.420 align:middle line:84% And then you would optimize over what 00:29:48.420 --> 00:29:50.500 align:middle line:84% is the best choice of b that gives you 00:29:50.500 --> 00:29:52.220 align:middle line:90% the tightest upper bound. 00:29:52.220 --> 00:29:54.640 align:middle line:84% So all I'm doing here is I'm doing the work for you. 00:29:54.640 --> 00:29:57.260 align:middle line:84% And I'm telling you that this will be the answer for b. 00:29:57.260 --> 00:29:58.900 align:middle line:90% So let's just work with this b. 00:29:58.900 --> 00:30:03.340 align:middle line:84% And then we'll check what happens for the upper tail. 00:30:03.340 --> 00:30:05.140 align:middle line:84% So now, we can put this all together. 00:30:05.140 --> 00:30:12.750 align:middle line:90% 00:30:12.750 --> 00:30:16.750 align:middle line:84% I told you this would be a long proof. 00:30:16.750 --> 00:30:20.800 align:middle line:84% All right, so let's put this all together. 00:30:20.800 --> 00:30:32.510 align:middle line:90% 00:30:32.510 --> 00:30:35.710 align:middle line:84% Well, we care about the probability 00:30:35.710 --> 00:30:37.670 align:middle line:90% that x is at least a. 00:30:37.670 --> 00:30:41.670 align:middle line:84% And remember, a is 1 plus delta times mu. 00:30:41.670 --> 00:30:47.830 align:middle line:84% We know that this is the same as the probability that e to the bx 00:30:47.830 --> 00:30:54.790 align:middle line:84% is at least e to the ab, where a is, again, my 1 plus delta mu. 00:30:54.790 --> 00:30:58.390 align:middle line:84% This uses that first fact that these two events are the same. 00:30:58.390 --> 00:31:05.210 align:middle line:84% We know by Markov that this is at most the expectation of e 00:31:05.210 --> 00:31:05.990 align:middle line:90% to the bx. 00:31:05.990 --> 00:31:09.490 align:middle line:84% That's where we first saw the moment-generating function. 00:31:09.490 --> 00:31:13.690 align:middle line:84% Now, we can appeal to lemma 1, which told us 00:31:13.690 --> 00:31:15.570 align:middle line:84% how to write the moment-generating function 00:31:15.570 --> 00:31:18.370 align:middle line:84% as the product of the moment-generating functions 00:31:18.370 --> 00:31:20.170 align:middle line:90% of its constituents. 00:31:20.170 --> 00:31:23.290 align:middle line:84% So we get the product from i equals 1 to n 00:31:23.290 --> 00:31:27.210 align:middle line:84% of all of these individual moment-generating functions 00:31:27.210 --> 00:31:30.690 align:middle line:90% still over this denominator eab. 00:31:30.690 --> 00:31:35.730 align:middle line:84% And finally, we can use lemma 2, which 00:31:35.730 --> 00:31:38.810 align:middle line:84% gave us a good quantitative upper bound for the setting 00:31:38.810 --> 00:31:40.450 align:middle line:90% of the Chernoff bound. 00:31:40.450 --> 00:31:43.810 align:middle line:84% We get the product from i equals 1 to n of e 00:31:43.810 --> 00:31:52.330 align:middle line:84% to the pi e to the b minus 1 all over e to the ab. 00:31:52.330 --> 00:31:56.105 align:middle line:84% And I'm going to call this right-hand side star. 00:31:56.105 --> 00:31:57.730 align:middle line:84% And the name of the game is really just 00:31:57.730 --> 00:32:01.660 align:middle line:84% to figure out what star is so that the Chernoff bound pops 00:32:01.660 --> 00:32:03.100 align:middle line:90% out. 00:32:03.100 --> 00:32:05.980 align:middle line:84% So our bound star right here, we can 00:32:05.980 --> 00:32:09.180 align:middle line:84% plug in for all of our values of a and b. 00:32:09.180 --> 00:32:15.340 align:middle line:84% We'll get the product of i equals 1 to n of e 00:32:15.340 --> 00:32:22.365 align:middle line:84% to the pi times delta because b is natural log of 1 plus delta. 00:32:22.365 --> 00:32:23.740 align:middle line:84% So when I plug this in, I'm going 00:32:23.740 --> 00:32:26.900 align:middle line:84% to get delta sitting out right here. 00:32:26.900 --> 00:32:31.620 align:middle line:84% And then my denominator, well, that's e to the ab. 00:32:31.620 --> 00:32:37.380 align:middle line:84% So I claim it's equal to 1 plus delta to the 1 plus delta times 00:32:37.380 --> 00:32:38.500 align:middle line:90% mu. 00:32:38.500 --> 00:32:41.540 align:middle line:84% Because again, when I have e to the ab, 00:32:41.540 --> 00:32:43.260 align:middle line:90% I can take the b downstairs. 00:32:43.260 --> 00:32:46.580 align:middle line:84% And I'll get 1 plus delta instead of my e. 00:32:46.580 --> 00:32:50.060 align:middle line:84% And that'll be being raised to the a-th power, which is exactly 00:32:50.060 --> 00:32:53.340 align:middle line:90% this 1 plus delta times mu. 00:32:53.340 --> 00:32:55.000 align:middle line:84% So you see, for the Chernoff bound, 00:32:55.000 --> 00:32:56.980 align:middle line:84% it's very easy to potentially get lost 00:32:56.980 --> 00:32:58.480 align:middle line:90% in some of the computations. 00:32:58.480 --> 00:33:00.920 align:middle line:84% But the highest level point is really 00:33:00.920 --> 00:33:04.320 align:middle line:84% what's happening over here, the architecture of how the pi's 00:33:04.320 --> 00:33:06.000 align:middle line:90% of the proof fit together. 00:33:06.000 --> 00:33:09.520 align:middle line:84% The rest of it now, from now on, is just calculus 00:33:09.520 --> 00:33:11.640 align:middle line:84% because all we have to do is figure out 00:33:11.640 --> 00:33:14.160 align:middle line:84% how to actually bound the right-hand side so 00:33:14.160 --> 00:33:16.860 align:middle line:84% that the upper tail for the Chernoff pops out. 00:33:16.860 --> 00:33:20.520 align:middle line:90% 00:33:20.520 --> 00:33:23.600 align:middle line:84% So now, we have this expression right here. 00:33:23.600 --> 00:33:25.820 align:middle line:90% And we can write this out. 00:33:25.820 --> 00:33:28.560 align:middle line:84% Of course, the sum of the pi's is mu. 00:33:28.560 --> 00:33:34.400 align:middle line:84% So we're going to get e to the delta mu all over 1 plus delta 00:33:34.400 --> 00:33:37.360 align:middle line:90% to the 1 plus delta times mu. 00:33:37.360 --> 00:33:41.400 align:middle line:84% And what I claim-- this will be my lemma 3, The Last piece 00:33:41.400 --> 00:33:43.400 align:middle line:84% that we need for the Chernoff bound. 00:33:43.400 --> 00:33:45.600 align:middle line:84% I claim that this right-hand side 00:33:45.600 --> 00:33:50.280 align:middle line:84% is at most whatever shows up in the upper tail, 00:33:50.280 --> 00:33:54.320 align:middle line:84% this e to the minus delta squared mu type of term. 00:33:54.320 --> 00:33:57.710 align:middle line:84% So once I prove that last lemma 3, we'll be done. 00:33:57.710 --> 00:34:00.810 align:middle line:90% 00:34:00.810 --> 00:34:02.300 align:middle line:90% So let's prove that last part. 00:34:02.300 --> 00:34:05.050 align:middle line:90% 00:34:05.050 --> 00:34:06.200 align:middle line:90% And it's not too bad. 00:34:06.200 --> 00:34:15.090 align:middle line:90% 00:34:15.090 --> 00:34:16.810 align:middle line:84% So the way that we're going to do this 00:34:16.810 --> 00:34:19.530 align:middle line:84% is like this is what our lemma 3 is. 00:34:19.530 --> 00:34:23.017 align:middle line:84% We want to show that this is at most this quantity right here. 00:34:23.017 --> 00:34:24.850 align:middle line:84% So what I'm going to do is I'm going to take 00:34:24.850 --> 00:34:26.539 align:middle line:90% the natural log of both sides. 00:34:26.539 --> 00:34:40.409 align:middle line:90% 00:34:40.409 --> 00:34:45.290 align:middle line:84% So the natural log of the left-hand side of my expression 00:34:45.290 --> 00:34:52.570 align:middle line:84% is equal to delta mu minus 1 plus delta 00:34:52.570 --> 00:34:58.750 align:middle line:90% mu natural log of 1 plus delta. 00:34:58.750 --> 00:35:03.910 align:middle line:84% And I can rewrite this expression and pull out the mu. 00:35:03.910 --> 00:35:07.470 align:middle line:84% So I'll get mu times delta minus 1 00:35:07.470 --> 00:35:13.430 align:middle line:84% plus delta natural log of 1 plus delta. 00:35:13.430 --> 00:35:16.790 align:middle line:90% Nothing fancy is happening here. 00:35:16.790 --> 00:35:19.207 align:middle line:84% And now, we need our last basic fact 00:35:19.207 --> 00:35:20.790 align:middle line:84% that we're going to take advantage of, 00:35:20.790 --> 00:35:27.790 align:middle line:84% which is that I claim, for all x that are strictly positive, 00:35:27.790 --> 00:35:29.200 align:middle line:90% we have the following bound. 00:35:29.200 --> 00:35:31.950 align:middle line:90% 00:35:31.950 --> 00:35:35.510 align:middle line:84% So the log of 1 plus x is lower bounded 00:35:35.510 --> 00:35:40.430 align:middle line:90% by x over 1 plus x over 2. 00:35:40.430 --> 00:35:41.890 align:middle line:90% So this is all very natural. 00:35:41.890 --> 00:35:45.190 align:middle line:84% I mean, I told you these things follow from the Chernoff bound, 00:35:45.190 --> 00:35:48.390 align:middle line:90% from calculus types of proofs. 00:35:48.390 --> 00:35:50.470 align:middle line:84% And the idea is that if I took the Taylor 00:35:50.470 --> 00:35:53.520 align:middle line:84% expansion of natural log of 1 plus x, 00:35:53.520 --> 00:35:56.420 align:middle line:90% my linear term would be x. 00:35:56.420 --> 00:35:57.920 align:middle line:84% And now, what I want to do is I just 00:35:57.920 --> 00:36:01.400 align:middle line:84% want to make sure that the way that I get rid 00:36:01.400 --> 00:36:05.640 align:middle line:84% of the other terms, I always have a valid lower bound. 00:36:05.640 --> 00:36:07.800 align:middle line:84% So here, what I'm going to do is you 00:36:07.800 --> 00:36:09.360 align:middle line:84% have to choose the right expression 00:36:09.360 --> 00:36:10.920 align:middle line:90% to put in the denominator. 00:36:10.920 --> 00:36:13.640 align:middle line:84% But this is, again, something you can check either 00:36:13.640 --> 00:36:16.400 align:middle line:84% with elementary calculus or just by plotting these things 00:36:16.400 --> 00:36:18.720 align:middle line:90% to convince yourself it's true. 00:36:18.720 --> 00:36:20.920 align:middle line:84% But now, we're in good shape because what 00:36:20.920 --> 00:36:24.180 align:middle line:90% this tells us is that-- 00:36:24.180 --> 00:36:25.140 align:middle line:90% oh, boy. 00:36:25.140 --> 00:36:28.390 align:middle line:90% 00:36:28.390 --> 00:36:29.890 align:middle line:84% Sorry, I'll have to do it over here. 00:36:29.890 --> 00:36:32.560 align:middle line:90% 00:36:32.560 --> 00:36:35.560 align:middle line:84% So using this fact and our expression 00:36:35.560 --> 00:36:45.880 align:middle line:84% for the left-hand side, this is at most mu delta minus mu 1 00:36:45.880 --> 00:36:52.460 align:middle line:84% plus delta delta over 1 plus delta over 2. 00:36:52.460 --> 00:36:54.180 align:middle line:84% So you can see what's going on here 00:36:54.180 --> 00:36:59.940 align:middle line:84% is that I'm just plugging in this natural log of 1 plus x 00:36:59.940 --> 00:37:03.500 align:middle line:84% in order to get rid of this term right here, because I'm 00:37:03.500 --> 00:37:07.820 align:middle line:84% replacing this with delta over 1 plus delta over 2. 00:37:07.820 --> 00:37:10.100 align:middle line:84% And then I'll get exactly this expression 00:37:10.100 --> 00:37:12.420 align:middle line:84% that I've written down right here. 00:37:12.420 --> 00:37:15.340 align:middle line:84% And now, when we collect terms, we'll be home free. 00:37:15.340 --> 00:37:19.040 align:middle line:84% So this is equal to mu times a whole bunch of things. 00:37:19.040 --> 00:37:23.780 align:middle line:84% So I'm going to multiply this part by 1 plus delta over 2, 00:37:23.780 --> 00:37:25.380 align:middle line:84% just so that I can put everything 00:37:25.380 --> 00:37:28.820 align:middle line:84% to have this denominator of 1 plus delta over 2. 00:37:28.820 --> 00:37:33.500 align:middle line:84% And then I'll get delta plus delta squared minus delta 00:37:33.500 --> 00:37:36.220 align:middle line:90% minus delta squared. 00:37:36.220 --> 00:37:41.060 align:middle line:84% And then life is good because I get minus mu delta squared 00:37:41.060 --> 00:37:43.355 align:middle line:90% over 2 plus delta. 00:37:43.355 --> 00:37:44.230 align:middle line:90% And that's the proof. 00:37:44.230 --> 00:37:48.100 align:middle line:90% 00:37:48.100 --> 00:37:50.200 align:middle line:84% So one of the points that I want to make-- so 00:37:50.200 --> 00:37:53.000 align:middle line:84% we'll talk more about this in recitation. 00:37:53.000 --> 00:37:55.440 align:middle line:84% So the Chernoff bound is definitely 00:37:55.440 --> 00:37:57.000 align:middle line:84% one of the most complicated proofs 00:37:57.000 --> 00:37:59.200 align:middle line:90% we've done up until this point. 00:37:59.200 --> 00:38:01.080 align:middle line:84% But when you're presenting things 00:38:01.080 --> 00:38:03.760 align:middle line:84% that are this complicated or things like this in your term 00:38:03.760 --> 00:38:06.200 align:middle line:84% paper, one of the things to keep in mind 00:38:06.200 --> 00:38:11.160 align:middle line:84% is that it's not really just begin proof end proof. 00:38:11.160 --> 00:38:13.160 align:middle line:84% So the way that you break up the proof 00:38:13.160 --> 00:38:17.640 align:middle line:84% makes it a lot easier to understand the overall strategy. 00:38:17.640 --> 00:38:20.080 align:middle line:84% So for example, the way that I broke it up 00:38:20.080 --> 00:38:24.720 align:middle line:84% into these lemma 1, lemma 2, and then lemma 3, 00:38:24.720 --> 00:38:27.920 align:middle line:84% it allows me to give an overview of the Chernoff bound, even 00:38:27.920 --> 00:38:31.160 align:middle line:84% though it was a full lecture-long proof, that 00:38:31.160 --> 00:38:33.600 align:middle line:90% fits very compactly. 00:38:33.600 --> 00:38:36.980 align:middle line:84% So a lot of times, when you're first doing proof-based math, 00:38:36.980 --> 00:38:40.218 align:middle line:84% you have this temptation to prove everything linearly, 00:38:40.218 --> 00:38:41.760 align:middle line:84% where you just start at the beginning 00:38:41.760 --> 00:38:45.520 align:middle line:84% and you keep proving things until you're at the end. 00:38:45.520 --> 00:38:48.290 align:middle line:84% When I was first starting out, that 00:38:48.290 --> 00:38:50.530 align:middle line:90% was also my temptation, too. 00:38:50.530 --> 00:38:52.430 align:middle line:84% I think my first paper, when I wrote it, 00:38:52.430 --> 00:38:55.810 align:middle line:84% it was a 15-page begin proof end proof. 00:38:55.810 --> 00:38:57.820 align:middle line:84% And then my advisor told me afterwards 00:38:57.820 --> 00:38:59.570 align:middle line:84% that he didn't understand the proof, which 00:38:59.570 --> 00:39:01.650 align:middle line:90% was a little bit worrisome. 00:39:01.650 --> 00:39:03.370 align:middle line:84% In fact, it wasn't even so obvious 00:39:03.370 --> 00:39:05.010 align:middle line:84% from the way I'd written it that, 00:39:05.010 --> 00:39:07.290 align:middle line:84% when you look at the dependencies 00:39:07.290 --> 00:39:11.210 align:middle line:84% between the different pieces, that they were actually acyclic. 00:39:11.210 --> 00:39:13.290 align:middle line:84% So to a certain extent, sometimes, you 00:39:13.290 --> 00:39:17.010 align:middle line:84% can refactor a proof so that it's not just linear so that you 00:39:17.010 --> 00:39:19.530 align:middle line:84% can give people a preview of how exactly 00:39:19.530 --> 00:39:24.570 align:middle line:84% it fits together because there's a difference between what 00:39:24.570 --> 00:39:27.090 align:middle line:84% are the parts where the action is happening, where you're 00:39:27.090 --> 00:39:30.090 align:middle line:84% telling me about what's the strategy you're going to use, 00:39:30.090 --> 00:39:32.610 align:middle line:84% where you're going to use mutual independence, 00:39:32.610 --> 00:39:35.090 align:middle line:84% and the parts which are mechanical 00:39:35.090 --> 00:39:37.770 align:middle line:84% and are more technical to actually execute, 00:39:37.770 --> 00:39:41.510 align:middle line:84% because a lot of these things are algebraic computations. 00:39:41.510 --> 00:39:44.370 align:middle line:84% I can follow these things and check them. 00:39:44.370 --> 00:39:46.770 align:middle line:90% And I can work them out myself. 00:39:46.770 --> 00:39:49.990 align:middle line:84% But the truth is that I don't learn a whole heck of a lot 00:39:49.990 --> 00:39:53.590 align:middle line:84% from the proof of lemma 3, except for the fact that, 00:39:53.590 --> 00:39:55.530 align:middle line:84% if I massage things algebraically, 00:39:55.530 --> 00:39:58.572 align:middle line:84% I'll get the right expression that I wanted. 00:39:58.572 --> 00:40:01.030 align:middle line:84% So you should think about a lot of these pedagogical things 00:40:01.030 --> 00:40:05.670 align:middle line:84% for your own writing, because are there 00:40:05.670 --> 00:40:10.190 align:middle line:84% ways to assert some pieces of the lemma 00:40:10.190 --> 00:40:13.190 align:middle line:84% so that you can give people a blueprint for how the pieces fit 00:40:13.190 --> 00:40:17.550 align:middle line:84% together, and then maybe fill in the parts which are the most 00:40:17.550 --> 00:40:21.550 align:middle line:84% interesting first and table the parts which are more mechanical 00:40:21.550 --> 00:40:23.330 align:middle line:90% and algebraic for later? 00:40:23.330 --> 00:40:27.270 align:middle line:84% So another thing I did to try and present this proof 00:40:27.270 --> 00:40:30.390 align:middle line:84% was that there were things that I explained that I didn't 00:40:30.390 --> 00:40:32.750 align:middle line:90% literally need to explain. 00:40:32.750 --> 00:40:35.450 align:middle line:84% So can you give me any examples of things in the proof, 00:40:35.450 --> 00:40:37.670 align:middle line:84% the way that I presented it, that aren't strictly 00:40:37.670 --> 00:40:41.670 align:middle line:84% necessary for getting the upper tail for the Chernoff bound? 00:40:41.670 --> 00:40:42.930 align:middle line:90% Just out of curiosity. 00:40:42.930 --> 00:40:52.680 align:middle line:90% 00:40:52.680 --> 00:40:53.830 align:middle line:90% Most of it's on the board. 00:40:53.830 --> 00:40:59.705 align:middle line:90% 00:40:59.705 --> 00:41:01.080 align:middle line:84% And also, are there any questions 00:41:01.080 --> 00:41:02.500 align:middle line:90% about the Chernoff bound proof? 00:41:02.500 --> 00:41:09.840 align:middle line:90% 00:41:09.840 --> 00:41:10.360 align:middle line:90% Yeah? 00:41:10.360 --> 00:41:11.735 align:middle line:84% STUDENT: So at the beginning, you 00:41:11.735 --> 00:41:14.540 align:middle line:84% said if you want to adapt it to a different variable, 00:41:14.540 --> 00:41:18.360 align:middle line:84% we could use bounds on the moments? 00:41:18.360 --> 00:41:20.920 align:middle line:90% Where exactly in the proof-- 00:41:20.920 --> 00:41:23.840 align:middle line:84% let's say we want to use Bernoulli inequality. 00:41:23.840 --> 00:41:26.780 align:middle line:84% Where would we incorporate that information? 00:41:26.780 --> 00:41:28.000 align:middle line:90% ANKUR MOITRA: Great question. 00:41:28.000 --> 00:41:31.040 align:middle line:84% So where in this proof did I use the fact 00:41:31.040 --> 00:41:34.600 align:middle line:90% that the xi's were Bernoulli? 00:41:34.600 --> 00:41:38.840 align:middle line:84% So now, when I have this architecture of the proof just 00:41:38.840 --> 00:41:41.760 align:middle line:84% sitting on the board, it becomes a lot easier 00:41:41.760 --> 00:41:44.380 align:middle line:84% to think about each individual piece. 00:41:44.380 --> 00:41:47.060 align:middle line:84% That's the advantage of presenting it this way. 00:41:47.060 --> 00:41:50.260 align:middle line:90% Did I use Bernoulli here? 00:41:50.260 --> 00:41:51.900 align:middle line:90% No. 00:41:51.900 --> 00:41:54.660 align:middle line:90% Did I use Bernoulli here? 00:41:54.660 --> 00:41:55.940 align:middle line:90% No. 00:41:55.940 --> 00:41:57.540 align:middle line:90% What about here? 00:41:57.540 --> 00:41:59.523 align:middle line:90% This was the place I used it. 00:41:59.523 --> 00:42:00.940 align:middle line:84% So that's what I was saying when I 00:42:00.940 --> 00:42:03.260 align:middle line:84% meant that the Chernoff bound can be adapted 00:42:03.260 --> 00:42:04.900 align:middle line:90% to many different settings. 00:42:04.900 --> 00:42:06.700 align:middle line:84% It's really just a blanket statement 00:42:06.700 --> 00:42:11.100 align:middle line:84% that sums up independent random variables have 00:42:11.100 --> 00:42:13.240 align:middle line:90% nice tail-bound properties. 00:42:13.240 --> 00:42:14.740 align:middle line:84% And the only thing that would change 00:42:14.740 --> 00:42:17.820 align:middle line:84% was, if you gave me non Bernoulli random variables, 00:42:17.820 --> 00:42:20.780 align:middle line:84% I would have other explicit estimates. 00:42:20.780 --> 00:42:24.860 align:middle line:84% Now here, I just used Bernoulli to figure out 00:42:24.860 --> 00:42:27.720 align:middle line:84% what exactly the moment-generating function was 00:42:27.720 --> 00:42:28.400 align:middle line:90% literally. 00:42:28.400 --> 00:42:29.900 align:middle line:90% This is an equality. 00:42:29.900 --> 00:42:32.340 align:middle line:84% But then I had to do a bit of massaging to get 00:42:32.340 --> 00:42:35.100 align:middle line:90% the pi's to be in the exponent. 00:42:35.100 --> 00:42:37.080 align:middle line:84% So that's the part that would change, 00:42:37.080 --> 00:42:39.780 align:middle line:84% is that if you gave me some other random variables, 00:42:39.780 --> 00:42:41.910 align:middle line:84% I could plug in and compute literally 00:42:41.910 --> 00:42:44.607 align:middle line:84% what the moment-generating functions are for them. 00:42:44.607 --> 00:42:46.190 align:middle line:84% But then I would have the same problem 00:42:46.190 --> 00:42:49.110 align:middle line:84% about how to massage that into an expression 00:42:49.110 --> 00:42:51.650 align:middle line:84% where, when I take the product of all of them, 00:42:51.650 --> 00:42:54.030 align:middle line:90% I get the mu sitting out there. 00:42:54.030 --> 00:42:56.910 align:middle line:84% So those are things which can be done. 00:42:56.910 --> 00:42:59.410 align:middle line:84% But for the most part, they're not very instructive. 00:42:59.410 --> 00:43:01.990 align:middle line:84% So you can just cite whichever Chernoff bounds you 00:43:01.990 --> 00:43:03.460 align:middle line:90% want when you have to use it. 00:43:03.460 --> 00:43:05.710 align:middle line:84% And you should know that the architecture of the proof 00:43:05.710 --> 00:43:09.150 align:middle line:90% is not very surprising. 00:43:09.150 --> 00:43:11.707 align:middle line:84% Are there any other questions about it? 00:43:11.707 --> 00:43:13.790 align:middle line:84% So the other thing I'll ask, which I asked before, 00:43:13.790 --> 00:43:17.990 align:middle line:84% is what part of my exposition was not literally needed 00:43:17.990 --> 00:43:19.530 align:middle line:90% to prove the Chernoff bound? 00:43:19.530 --> 00:43:20.810 align:middle line:90% Yeah? 00:43:20.810 --> 00:43:21.710 align:middle line:90% STUDENT: [INAUDIBLE] 00:43:21.710 --> 00:43:22.410 align:middle line:90% ANKUR MOITRA: Yeah. 00:43:22.410 --> 00:43:24.702 align:middle line:84% Actually, I didn't need to talk about moment-generating 00:43:24.702 --> 00:43:27.710 align:middle line:90% functions at all. 00:43:27.710 --> 00:43:30.710 align:middle line:84% So I could have just said that the right-hand side is 00:43:30.710 --> 00:43:32.470 align:middle line:90% the right-hand side. 00:43:32.470 --> 00:43:36.790 align:middle line:84% It is whatever it is, expectation of e to the bx. 00:43:36.790 --> 00:43:40.250 align:middle line:84% And then I could have just said, because x 00:43:40.250 --> 00:43:43.490 align:middle line:84% is the sum of xi's, it's equal to the product 00:43:43.490 --> 00:43:44.630 align:middle line:90% of their expectations. 00:43:44.630 --> 00:43:47.515 align:middle line:84% And we could have skipped straight to Lemma 1. 00:43:47.515 --> 00:43:49.390 align:middle line:84% That's one way I could have proved the thing. 00:43:49.390 --> 00:43:52.730 align:middle line:84% And there would be nothing mathematically wrong with that. 00:43:52.730 --> 00:43:54.750 align:middle line:90% But this class is a CIM. 00:43:54.750 --> 00:43:56.770 align:middle line:84% So it's not just about writing mathematically 00:43:56.770 --> 00:43:58.410 align:middle line:90% correct statements. 00:43:58.410 --> 00:44:02.850 align:middle line:84% But it's also about thinking about how to do the exposition 00:44:02.850 --> 00:44:05.925 align:middle line:84% to get the audience to better understand and appreciate it. 00:44:05.925 --> 00:44:08.550 align:middle line:84% Why do you think I talked about the moment-generating function, 00:44:08.550 --> 00:44:12.610 align:middle line:84% even though I didn't literally need to? 00:44:12.610 --> 00:44:14.190 align:middle line:84% Sorry to put you guys on the spot, 00:44:14.190 --> 00:44:15.607 align:middle line:84% but this is the way I designed it, 00:44:15.607 --> 00:44:17.970 align:middle line:84% was I want to interrogate you guys about why 00:44:17.970 --> 00:44:21.450 align:middle line:90% I presented it the way I did. 00:44:21.450 --> 00:44:22.590 align:middle line:90% Just for fun? 00:44:22.590 --> 00:44:23.105 align:middle line:90% Stalling? 00:44:23.105 --> 00:44:24.730 align:middle line:84% STUDENT: It's more modular [INAUDIBLE]. 00:44:24.730 --> 00:44:26.188 align:middle line:84% ANKUR MOITRA: So it's more modular? 00:44:26.188 --> 00:44:27.430 align:middle line:90% Yeah, that's one good thing. 00:44:27.430 --> 00:44:30.690 align:middle line:84% Any other ideas why I did it that way? 00:44:30.690 --> 00:44:32.770 align:middle line:84% Modularity was definitely very important 00:44:32.770 --> 00:44:35.810 align:middle line:84% for being able to fit together a blueprint of the proof 00:44:35.810 --> 00:44:38.060 align:middle line:84% on just one section of the board. 00:44:38.060 --> 00:44:39.780 align:middle line:84% But there are other reasons I used 00:44:39.780 --> 00:44:41.540 align:middle line:90% the moment-generating function. 00:44:41.540 --> 00:44:42.460 align:middle line:90% Yeah? 00:44:42.460 --> 00:44:44.940 align:middle line:90% STUDENT: So [INAUDIBLE]. 00:44:44.940 --> 00:44:46.920 align:middle line:84% ANKUR MOITRA: Yeah, that's a big reason. 00:44:46.920 --> 00:44:49.820 align:middle line:84% So I used it to demystify because otherwise this 00:44:49.820 --> 00:44:51.940 align:middle line:90% would look like black magic. 00:44:51.940 --> 00:44:54.100 align:middle line:84% We proved the Chebyshev bound, where 00:44:54.100 --> 00:44:58.860 align:middle line:84% we did that same trick of creating a new random variable. 00:44:58.860 --> 00:45:01.860 align:middle line:84% And that was very natural because we had a random variable 00:45:01.860 --> 00:45:03.320 align:middle line:84% before, which wasn't non-negative. 00:45:03.320 --> 00:45:06.540 align:middle line:84% So we squared it to create a non-negative random variable. 00:45:06.540 --> 00:45:08.260 align:middle line:84% But here, where would you come up 00:45:08.260 --> 00:45:10.140 align:middle line:84% with the idea of taking the random variable 00:45:10.140 --> 00:45:11.900 align:middle line:90% and exponentiating it? 00:45:11.900 --> 00:45:13.360 align:middle line:90% That's a very clever idea. 00:45:13.360 --> 00:45:15.280 align:middle line:84% And it's used throughout probability. 00:45:15.280 --> 00:45:18.260 align:middle line:84% We're not going to see much more exotic examples. 00:45:18.260 --> 00:45:21.080 align:middle line:84% But otherwise, this would seem totally exotic. 00:45:21.080 --> 00:45:23.660 align:middle line:84% Why wouldn't I try other functions? 00:45:23.660 --> 00:45:25.620 align:middle line:84% It's really because the exponential 00:45:25.620 --> 00:45:29.500 align:middle line:84% is the thing that keeps track of all of the relevant information 00:45:29.500 --> 00:45:30.680 align:middle line:90% about the random variable. 00:45:30.680 --> 00:45:32.140 align:middle line:90% I could care about. 00:45:32.140 --> 00:45:34.060 align:middle line:84% So I not only told you that this is 00:45:34.060 --> 00:45:36.480 align:middle line:84% called the moment-generating function, 00:45:36.480 --> 00:45:39.560 align:middle line:84% but I explained how you could extract properties 00:45:39.560 --> 00:45:43.440 align:middle line:84% of the random variable from that moment-generating function. 00:45:43.440 --> 00:45:45.920 align:middle line:84% That way, we can talk about what properties 00:45:45.920 --> 00:45:49.040 align:middle line:84% are we literally using about the random variable. 00:45:49.040 --> 00:45:51.680 align:middle line:84% There's another reason I introduced the moment generating 00:45:51.680 --> 00:45:53.480 align:middle line:90% function. 00:45:53.480 --> 00:45:55.980 align:middle line:84% I introduced it for several reasons. 00:45:55.980 --> 00:45:56.480 align:middle line:90% Yeah? 00:45:56.480 --> 00:45:59.600 align:middle line:90% STUDENT: Maybe [INAUDIBLE]. 00:45:59.600 --> 00:46:01.600 align:middle line:84% ANKUR MOITRA: Yeah, that's a good thought. 00:46:01.600 --> 00:46:03.040 align:middle line:90% That's a good idea. 00:46:03.040 --> 00:46:07.180 align:middle line:84% But it also has to do with you, the audience. 00:46:07.180 --> 00:46:09.580 align:middle line:84% So if I were teaching a different class, 00:46:09.580 --> 00:46:11.360 align:middle line:84% I might not have said the things I 00:46:11.360 --> 00:46:13.920 align:middle line:84% said about the moment-generating function. 00:46:13.920 --> 00:46:16.840 align:middle line:84% What did I use about the moment-generating function 00:46:16.840 --> 00:46:19.280 align:middle line:84% that was specific to you and what else you've 00:46:19.280 --> 00:46:21.272 align:middle line:90% learned in this class? 00:46:21.272 --> 00:46:23.175 align:middle line:84% STUDENT: We just covered generating-- 00:46:23.175 --> 00:46:24.300 align:middle line:90% ANKUR MOITRA: That's right. 00:46:24.300 --> 00:46:26.680 align:middle line:84% We just covered generating functions. 00:46:26.680 --> 00:46:29.160 align:middle line:84% So the way that I chose my exposition was also 00:46:29.160 --> 00:46:32.040 align:middle line:84% based on what I konw you know and how 00:46:32.040 --> 00:46:36.130 align:middle line:84% to make connections between it because, when we started 00:46:36.130 --> 00:46:38.130 align:middle line:84% the probability unit, we then switched 00:46:38.130 --> 00:46:39.890 align:middle line:90% gears and talked about counting. 00:46:39.890 --> 00:46:42.140 align:middle line:84% And there are a lot of connections between probability 00:46:42.140 --> 00:46:45.290 align:middle line:84% and counting, things like inclusion exclusion formulas. 00:46:45.290 --> 00:46:49.370 align:middle line:84% We already saw binomial coefficients coming up. 00:46:49.370 --> 00:46:52.450 align:middle line:84% And the same way, I wanted to further strengthen 00:46:52.450 --> 00:46:54.570 align:middle line:84% the connections between probability and counting 00:46:54.570 --> 00:46:58.890 align:middle line:84% because, usually, they're just different languages for talking 00:46:58.890 --> 00:47:02.290 align:middle line:90% about the same thing. 00:47:02.290 --> 00:47:02.790 align:middle line:90% All right. 00:47:02.790 --> 00:47:06.010 align:middle line:84% So these are all things you should think about when you're 00:47:06.010 --> 00:47:08.330 align:middle line:90% doing your own exposition. 00:47:08.330 --> 00:47:10.470 align:middle line:90% I'll tell you one last thing. 00:47:10.470 --> 00:47:13.090 align:middle line:90% And then we'll call it a day. 00:47:13.090 --> 00:47:17.090 align:middle line:84% So the Chernoff bound is this amazingly powerful tool. 00:47:17.090 --> 00:47:19.130 align:middle line:84% It was invented by Herman Chernoff, 00:47:19.130 --> 00:47:22.050 align:middle line:90% who was here for a long time. 00:47:22.050 --> 00:47:24.230 align:middle line:84% It's one of his most famous discoveries. 00:47:24.230 --> 00:47:27.230 align:middle line:90% But what's kind of funny is he-- 00:47:27.230 --> 00:47:29.510 align:middle line:84% I mean, he actually just turned, I think, 100. 00:47:29.510 --> 00:47:35.030 align:middle line:84% So he had his 100-year birthday recently. 00:47:35.030 --> 00:47:36.910 align:middle line:84% In fact, another fun fact about him 00:47:36.910 --> 00:47:39.130 align:middle line:90% is that his wife is alive, too. 00:47:39.130 --> 00:47:41.870 align:middle line:84% And they're believed to be the oldest living couple 00:47:41.870 --> 00:47:43.990 align:middle line:90% in the state of Massachusetts. 00:47:43.990 --> 00:47:46.350 align:middle line:90% So that's quite something. 00:47:46.350 --> 00:47:52.030 align:middle line:84% But the Chernoff bound also gets misused quite a lot because it 00:47:52.030 --> 00:47:56.150 align:middle line:84% sounds like this amazing fact that, when I first introduced 00:47:56.150 --> 00:47:59.910 align:middle line:84% it, I talked about how we wanted to drive the failure 00:47:59.910 --> 00:48:03.790 align:middle line:84% probability of some randomized algorithm down. 00:48:03.790 --> 00:48:05.730 align:middle line:84% And if we repeat it a bunch of times, 00:48:05.730 --> 00:48:07.550 align:middle line:84% things like the weak law of large numbers 00:48:07.550 --> 00:48:10.750 align:middle line:84% would tell us, to get our failure probability down to 10 00:48:10.750 --> 00:48:12.950 align:middle line:84% to the minus 30, we'd have to repeat it 00:48:12.950 --> 00:48:15.790 align:middle line:90% like 10 to the 30 times. 00:48:15.790 --> 00:48:17.710 align:middle line:84% The Chernoff bound would tell us that, 00:48:17.710 --> 00:48:20.470 align:middle line:84% because the right-hand side is this exponentially 00:48:20.470 --> 00:48:22.950 align:middle line:84% small decaying probability, we only 00:48:22.950 --> 00:48:26.470 align:middle line:84% have to repeat it natural log of that many times. 00:48:26.470 --> 00:48:29.110 align:middle line:84% So it tells us that random variables concentrate 00:48:29.110 --> 00:48:31.600 align:middle line:90% at extremely fast rates. 00:48:31.600 --> 00:48:35.280 align:middle line:84% But what's really critical about it is the assumption 00:48:35.280 --> 00:48:37.580 align:middle line:90% of mutual independence. 00:48:37.580 --> 00:48:39.300 align:middle line:84% All of this was predicated on it. 00:48:39.300 --> 00:48:40.800 align:middle line:84% The moment-generating function keeps 00:48:40.800 --> 00:48:43.400 align:middle line:84% track of not just pairwise information, 00:48:43.400 --> 00:48:45.840 align:middle line:90% but all higher order moments. 00:48:45.840 --> 00:48:48.720 align:middle line:84% And there are a lot of famous examples 00:48:48.720 --> 00:48:52.200 align:middle line:84% that affect all of us of people thinking 00:48:52.200 --> 00:48:54.760 align:middle line:84% about using the Chernoff bound, even though it doesn't really 00:48:54.760 --> 00:48:56.160 align:middle line:90% apply. 00:48:56.160 --> 00:48:58.460 align:middle line:84% So this is probably pre your time. 00:48:58.460 --> 00:49:03.240 align:middle line:84% But I'll tell you a story about the financial markets, which 00:49:03.240 --> 00:49:07.520 align:middle line:84% was certainly relevant when I was finishing up college. 00:49:07.520 --> 00:49:13.720 align:middle line:84% So right around that time, there was this big financial crisis 00:49:13.720 --> 00:49:16.960 align:middle line:90% in 2008. 00:49:16.960 --> 00:49:20.680 align:middle line:84% And so here, you really have to be careful 00:49:20.680 --> 00:49:23.120 align:middle line:84% when you try and use this reasoning 00:49:23.120 --> 00:49:27.600 align:middle line:84% and when you do and don't have independence. 00:49:27.600 --> 00:49:30.100 align:middle line:90% So it's not just important-- 00:49:30.100 --> 00:49:33.220 align:middle line:84% I want to say that it's not just important to know 00:49:33.220 --> 00:49:34.720 align:middle line:90% how to use the Chernoff bound. 00:49:34.720 --> 00:49:40.220 align:middle line:84% But I want to claim that it's just as important to know 00:49:40.220 --> 00:49:41.930 align:middle line:90% when tail bounds don't apply. 00:49:41.930 --> 00:49:53.420 align:middle line:90% 00:49:53.420 --> 00:49:56.720 align:middle line:84% So just a little bit of personal background-- 00:49:56.720 --> 00:49:58.960 align:middle line:84% so both my parents are computer scientists. 00:49:58.960 --> 00:50:01.220 align:middle line:84% My whole family is computer scientists. 00:50:01.220 --> 00:50:03.780 align:middle line:84% So I really didn't want to be a computer scientist. 00:50:03.780 --> 00:50:06.100 align:middle line:90% I really showed them. 00:50:06.100 --> 00:50:11.060 align:middle line:84% But I tried out all kinds of things when I was at your age. 00:50:11.060 --> 00:50:14.520 align:middle line:84% So around that time, I worked in Wall Street. 00:50:14.520 --> 00:50:17.660 align:middle line:84% And I worked on some quantitative finance problems. 00:50:17.660 --> 00:50:19.880 align:middle line:84% And this was pre the financial crisis. 00:50:19.880 --> 00:50:21.740 align:middle line:84% But I remember one of the things that they 00:50:21.740 --> 00:50:25.500 align:middle line:84% were big into were very fancy assets. 00:50:25.500 --> 00:50:28.310 align:middle line:84% So there are things called collateralized mortgage 00:50:28.310 --> 00:50:30.190 align:middle line:90% obligations. 00:50:30.190 --> 00:50:32.110 align:middle line:84% Has anyone heard of collateralized mortgage 00:50:32.110 --> 00:50:33.710 align:middle line:90% operations? 00:50:33.710 --> 00:50:34.270 align:middle line:90% CMOs? 00:50:34.270 --> 00:50:35.280 align:middle line:90% No? 00:50:35.280 --> 00:50:37.990 align:middle line:84% So the way that this works is that you 00:50:37.990 --> 00:50:42.430 align:middle line:84% have a whole bunch of mortgages that you sell to people for them 00:50:42.430 --> 00:50:44.150 align:middle line:90% to buy their house. 00:50:44.150 --> 00:50:46.970 align:middle line:84% And then implicitly, when you're giving someone a mortgage, 00:50:46.970 --> 00:50:49.910 align:middle line:84% you have to assess their credit worthiness 00:50:49.910 --> 00:50:51.750 align:middle line:84% to figure out the likelihood that they're 00:50:51.750 --> 00:50:54.190 align:middle line:90% going to pay back the loan. 00:50:54.190 --> 00:50:56.290 align:middle line:84% Now, for people who have great credit ratings, 00:50:56.290 --> 00:50:58.550 align:middle line:84% maybe that's not such a big risk. 00:50:58.550 --> 00:51:02.370 align:middle line:84% But what happens if you're making riskier mortgages? 00:51:02.370 --> 00:51:05.570 align:middle line:84% So those are things that you can charge more for. 00:51:05.570 --> 00:51:08.350 align:middle line:84% You can have a higher interest rate. 00:51:08.350 --> 00:51:10.070 align:middle line:84% But you have to deal with the fact 00:51:10.070 --> 00:51:12.630 align:middle line:84% that the probability that they pay it back 00:51:12.630 --> 00:51:16.250 align:middle line:84% might be less than you would hope. 00:51:16.250 --> 00:51:18.870 align:middle line:84% So you had all of these issues in 2008 00:51:18.870 --> 00:51:22.310 align:middle line:84% that there was a huge amount of construction going on. 00:51:22.310 --> 00:51:24.050 align:middle line:84% People were buying a lot of these houses. 00:51:24.050 --> 00:51:26.050 align:middle line:90% Some people had multiple houses. 00:51:26.050 --> 00:51:28.530 align:middle line:84% And they had very poor credit ratings. 00:51:28.530 --> 00:51:31.690 align:middle line:84% But the banks didn't want to hold on to these mortgages 00:51:31.690 --> 00:51:33.730 align:middle line:90% because they're risky products. 00:51:33.730 --> 00:51:37.370 align:middle line:84% So how exactly-- what do you do with these financial products? 00:51:37.370 --> 00:51:39.170 align:middle line:84% So there was this great idea, which 00:51:39.170 --> 00:51:42.730 align:middle line:84% was to create synthetic products instead. 00:51:42.730 --> 00:51:45.170 align:middle line:84% So what you do is you take a whole bunch of what 00:51:45.170 --> 00:51:47.310 align:middle line:90% are called subprime mortgages. 00:51:47.310 --> 00:51:49.930 align:middle line:90% 00:51:49.930 --> 00:51:53.730 align:middle line:84% And you bundle them together because the hope 00:51:53.730 --> 00:51:56.770 align:middle line:84% is that all of these small probability 00:51:56.770 --> 00:52:00.610 align:middle line:84% events of the person failing and defaulting on their loan, 00:52:00.610 --> 00:52:03.090 align:middle line:84% maybe if you combine them together, you 00:52:03.090 --> 00:52:05.610 align:middle line:84% can appeal to things like the Chernoff bound 00:52:05.610 --> 00:52:10.650 align:middle line:84% and say that things concentrate more around their expectation. 00:52:10.650 --> 00:52:13.050 align:middle line:84% So you have the issue that these subprime mortgages still 00:52:13.050 --> 00:52:15.690 align:middle line:84% have a reasonably large failure probability. 00:52:15.690 --> 00:52:16.950 align:middle line:90% So no one would buy them. 00:52:16.950 --> 00:52:20.970 align:middle line:84% So what you do is you create what's called a waterfall payoff 00:52:20.970 --> 00:52:22.330 align:middle line:90% structure. 00:52:22.330 --> 00:52:25.140 align:middle line:84% So you do something called creating tranches. 00:52:25.140 --> 00:52:29.300 align:middle line:84% So you have this baseline of all of these subprime mortgages. 00:52:29.300 --> 00:52:31.900 align:middle line:84% And then the first people who default 00:52:31.900 --> 00:52:35.060 align:middle line:84% on their loans, the people who are going to eat the losses 00:52:35.060 --> 00:52:37.280 align:middle line:84% are the people at the bottom of the waterfall. 00:52:37.280 --> 00:52:39.660 align:middle line:84% That's where the water dries up the first. 00:52:39.660 --> 00:52:43.100 align:middle line:84% And the people at the very top of the waterfall, 00:52:43.100 --> 00:52:47.340 align:middle line:84% they only get losses if basically everything collapses 00:52:47.340 --> 00:52:51.020 align:middle line:84% or a large fraction of the mortgages collapse. 00:52:51.020 --> 00:52:55.460 align:middle line:84% So maybe each mortgage has a 10% chance of defaulting. 00:52:55.460 --> 00:52:59.080 align:middle line:84% And in order for this person at the very top to eat any losses, 00:52:59.080 --> 00:53:03.700 align:middle line:84% maybe 40% of the mortgages have to default. 00:53:03.700 --> 00:53:06.020 align:middle line:84% So if you thought about these probabilities 00:53:06.020 --> 00:53:10.540 align:middle line:84% as being independent, this would be an extremely safe asset. 00:53:10.540 --> 00:53:14.460 align:middle line:84% What's the chance that I take a coin that is heads with 10% 00:53:14.460 --> 00:53:18.620 align:middle line:84% probability and I flip it 1,000 times and I end up with 40% 00:53:18.620 --> 00:53:19.460 align:middle line:90% heads? 00:53:19.460 --> 00:53:21.480 align:middle line:84% It's ridiculously small because we just 00:53:21.480 --> 00:53:24.200 align:middle line:90% proofed it with the upper tail. 00:53:24.200 --> 00:53:27.360 align:middle line:84% So that allowed people to get credit rating 00:53:27.360 --> 00:53:30.760 align:middle line:84% agencies to say that these are very safe assets. 00:53:30.760 --> 00:53:33.040 align:middle line:84% So when you have a retirement plan 00:53:33.040 --> 00:53:35.340 align:middle line:84% and you put money into your retirement account, 00:53:35.340 --> 00:53:37.040 align:middle line:84% there are certain rules and regulations 00:53:37.040 --> 00:53:41.200 align:middle line:84% about what the financial risk and the gradation for the asset 00:53:41.200 --> 00:53:44.840 align:middle line:84% needs to be in order for these retirement companies 00:53:44.840 --> 00:53:47.857 align:middle line:84% to even purchase the asset because these things that 00:53:47.857 --> 00:53:49.440 align:middle line:84% are managing your retirement shouldn't 00:53:49.440 --> 00:53:51.800 align:middle line:90% be purchasing risky accounts. 00:53:51.800 --> 00:53:53.600 align:middle line:84% So all the credit rating agencies 00:53:53.600 --> 00:53:58.120 align:middle line:84% rated these things as if the failure of the mortgages 00:53:58.120 --> 00:54:00.280 align:middle line:90% were independent events. 00:54:00.280 --> 00:54:02.200 align:middle line:84% And that made them say that, actually, these 00:54:02.200 --> 00:54:03.980 align:middle line:90% are extremely safe assets. 00:54:03.980 --> 00:54:08.760 align:middle line:84% Maybe they're even safer than things like the US currency. 00:54:08.760 --> 00:54:12.360 align:middle line:84% The trouble is that Chernoff bounds don't really apply. 00:54:12.360 --> 00:54:13.780 align:middle line:90% So what's the problem with this? 00:54:13.780 --> 00:54:15.500 align:middle line:90% Why don't tail bounds apply? 00:54:15.500 --> 00:54:16.360 align:middle line:90% Yeah? 00:54:16.360 --> 00:54:17.340 align:middle line:84% STUDENT: They're not independent. 00:54:17.340 --> 00:54:18.923 align:middle line:84% ANKUR MOITRA: They're not independent. 00:54:18.923 --> 00:54:19.690 align:middle line:90% That's right. 00:54:19.690 --> 00:54:23.810 align:middle line:84% What causes someone to default is a lot of economic strife. 00:54:23.810 --> 00:54:26.530 align:middle line:84% And when that happens on a massive scale, all of a sudden, 00:54:26.530 --> 00:54:28.410 align:middle line:90% you have this domino effect. 00:54:28.410 --> 00:54:30.690 align:middle line:84% And these things that were previously 00:54:30.690 --> 00:54:35.440 align:middle line:84% viewed to be as very safe things are not really safe, even 00:54:35.440 --> 00:54:37.690 align:middle line:84% though at the time I remember people were using things 00:54:37.690 --> 00:54:40.370 align:middle line:84% like Gaussian approximations to figure out 00:54:40.370 --> 00:54:42.090 align:middle line:90% their probability of default. 00:54:42.090 --> 00:54:44.310 align:middle line:84% So I'll leave you with that last nugget, 00:54:44.310 --> 00:54:47.570 align:middle line:84% which is probability is very important for your lives. 00:54:47.570 --> 00:54:49.050 align:middle line:84% You should use the Chernoff bounds 00:54:49.050 --> 00:54:51.450 align:middle line:84% in the classes we give you in problem sets. 00:54:51.450 --> 00:54:54.930 align:middle line:84% But in real life, you should be critical about whether or not 00:54:54.930 --> 00:54:57.050 align:middle line:84% it applies because the dependence 00:54:57.050 --> 00:54:59.712 align:middle line:84% between these random variables is a nasty thing. 00:54:59.712 --> 00:55:01.170 align:middle line:84% And there are whole courses you can 00:55:01.170 --> 00:55:03.390 align:middle line:84% teach on dealing with dependent random variables. 00:55:03.390 --> 00:55:06.050 align:middle line:90% But to be continued. 00:55:06.050 --> 00:55:07.270 align:middle line:90% So see you next time. 00:55:07.270 --> 00:55:10.050 align:middle line:84% And we'll be starting our modular arithmetic unit 00:55:10.050 --> 00:55:11.840 align:middle line:90% on Tuesday. 00:55:11.840 --> 00:55:18.000 align:middle line:90%