WEBVTT 00:00:00.000 --> 00:00:02.415 align:middle line:90% [SQUEAKING] 00:00:02.415 --> 00:00:03.864 align:middle line:90% [RUSTLING] 00:00:03.864 --> 00:00:07.728 align:middle line:90% [CLICKING] 00:00:07.728 --> 00:00:11.382 align:middle line:90% 00:00:11.382 --> 00:00:12.840 align:middle line:84% ANKUR MOITRA: So today, we're going 00:00:12.840 --> 00:00:15.020 align:middle line:84% to be continuing our probability unit. 00:00:15.020 --> 00:00:17.100 align:middle line:84% So I just want to remind you where we were. 00:00:17.100 --> 00:00:19.560 align:middle line:84% So when I was last lecturing, I introduced 00:00:19.560 --> 00:00:21.520 align:middle line:90% the basics of probability. 00:00:21.520 --> 00:00:24.120 align:middle line:84% I told you what a sample space was. 00:00:24.120 --> 00:00:26.563 align:middle line:84% And then it led to things like events, 00:00:26.563 --> 00:00:28.480 align:middle line:84% where we talked about what was the probability 00:00:28.480 --> 00:00:29.313 align:middle line:90% of different events. 00:00:29.313 --> 00:00:31.480 align:middle line:90% We talked about independence. 00:00:31.480 --> 00:00:33.320 align:middle line:84% We talked about random variables that 00:00:33.320 --> 00:00:37.120 align:middle line:84% are functions from our sample space, the real values. 00:00:37.120 --> 00:00:38.960 align:middle line:84% And once we had random variables, 00:00:38.960 --> 00:00:41.380 align:middle line:84% that allowed us to do a lot of exciting things, 00:00:41.380 --> 00:00:44.800 align:middle line:84% like we could talk about the expectation and the variance 00:00:44.800 --> 00:00:46.720 align:middle line:90% of the random variables. 00:00:46.720 --> 00:00:49.680 align:middle line:84% It allowed us to do things talk about conditioning, like when 00:00:49.680 --> 00:00:53.440 align:middle line:84% we condition on some event in the sample space occurring-- 00:00:53.440 --> 00:00:56.400 align:middle line:84% how does that change the probability of other events, 00:00:56.400 --> 00:01:00.160 align:middle line:84% how does that change the expectation of random variables. 00:01:00.160 --> 00:01:02.120 align:middle line:84% And then we talked about this powerful tool 00:01:02.120 --> 00:01:04.599 align:middle line:90% of inclusion-exclusion. 00:01:04.599 --> 00:01:06.160 align:middle line:84% And then after that, Peter gave you 00:01:06.160 --> 00:01:09.000 align:middle line:84% a bunch of lectures on counting methods, on generating 00:01:09.000 --> 00:01:10.040 align:middle line:90% functions. 00:01:10.040 --> 00:01:12.355 align:middle line:84% And the truth is, for the introductory lectures 00:01:12.355 --> 00:01:14.480 align:middle line:84% in this class, there are a lot of tight connections 00:01:14.480 --> 00:01:16.560 align:middle line:84% between probability and counting. 00:01:16.560 --> 00:01:18.580 align:middle line:84% So we really need both perspectives. 00:01:18.580 --> 00:01:21.160 align:middle line:84% And we're going to go back and forth between them. 00:01:21.160 --> 00:01:23.080 align:middle line:84% But today, we're going to cover one 00:01:23.080 --> 00:01:26.120 align:middle line:84% of the other important topics in probability, something 00:01:26.120 --> 00:01:28.520 align:middle line:84% that we're going to spend a bunch of time on, 00:01:28.520 --> 00:01:30.960 align:middle line:90% which are called tail bounds. 00:01:30.960 --> 00:01:34.160 align:middle line:84% So the idea for a tail bound is to get a handle 00:01:34.160 --> 00:01:37.800 align:middle line:84% on how likely it is for random variables 00:01:37.800 --> 00:01:40.720 align:middle line:84% to be far away from their expectation. 00:01:40.720 --> 00:01:42.880 align:middle line:84% So remember that we take a random variable 00:01:42.880 --> 00:01:45.040 align:middle line:90% and we look at its expectation. 00:01:45.040 --> 00:01:47.960 align:middle line:84% And that tells us, on average, what we expect 00:01:47.960 --> 00:01:49.880 align:middle line:90% the random variable to be. 00:01:49.880 --> 00:01:51.640 align:middle line:84% But of course, some random variables 00:01:51.640 --> 00:01:53.840 align:middle line:84% might have large averages just because they 00:01:53.840 --> 00:01:58.920 align:middle line:84% have some very tiny probability of having a gigantic value. 00:01:58.920 --> 00:02:02.440 align:middle line:84% So then we talked about notions variation and variance 00:02:02.440 --> 00:02:05.680 align:middle line:84% that told us how well a random variable concentrates 00:02:05.680 --> 00:02:07.588 align:middle line:90% around its expectation. 00:02:07.588 --> 00:02:09.380 align:middle line:84% So today, we're going to make that precise. 00:02:09.380 --> 00:02:11.280 align:middle line:84% We're actually going to get bounds 00:02:11.280 --> 00:02:14.160 align:middle line:84% on what's the probability that a random variable is very far 00:02:14.160 --> 00:02:16.640 align:middle line:90% away from its expectation. 00:02:16.640 --> 00:02:21.040 align:middle line:84% So let me tell you one simple tale bound in action. 00:02:21.040 --> 00:02:25.680 align:middle line:84% And really, the theme of today and Thursday's lecture 00:02:25.680 --> 00:02:28.800 align:middle line:84% will be we're going to use more and more information 00:02:28.800 --> 00:02:33.040 align:middle line:84% about random variables to get tighter and tighter control. 00:02:33.040 --> 00:02:37.880 align:middle line:84% So let's start off with a very basic example of what theorem 00:02:37.880 --> 00:02:41.160 align:middle line:84% needs very little information about my random variable, 00:02:41.160 --> 00:02:44.360 align:middle line:84% but gives me a rudimentary baseline of what's 00:02:44.360 --> 00:02:48.440 align:middle line:84% the probability it's far away from its expectation. 00:02:48.440 --> 00:02:52.060 align:middle line:84% So this is a very simple to state and prove bound. 00:02:52.060 --> 00:02:55.160 align:middle line:90% It's called Markov's inequality. 00:02:55.160 --> 00:02:57.960 align:middle line:84% And here, we're going to assume that x is not just 00:02:57.960 --> 00:03:02.130 align:middle line:84% a random variable, but it's a non-negative random variable. 00:03:02.130 --> 00:03:05.360 align:middle line:90% 00:03:05.360 --> 00:03:12.120 align:middle line:84% Then what I claim is that the probability that x is larger 00:03:12.120 --> 00:03:18.840 align:middle line:84% than some constant C is bounded by the expectation of x all 00:03:18.840 --> 00:03:21.080 align:middle line:90% divided by C. 00:03:21.080 --> 00:03:24.280 align:middle line:84% So this makes intuitive sense because, if I 00:03:24.280 --> 00:03:26.760 align:middle line:84% have this non-negative random variable, 00:03:26.760 --> 00:03:30.680 align:middle line:84% as C gets larger and larger, I'm going way out into the tails 00:03:30.680 --> 00:03:32.520 align:middle line:90% of my random variable. 00:03:32.520 --> 00:03:35.600 align:middle line:84% The probability that the random variable is actually that large 00:03:35.600 --> 00:03:38.880 align:middle line:84% should be getting smaller and smaller. 00:03:38.880 --> 00:03:42.680 align:middle line:84% Now, the important thing is that the expectation shows up 00:03:42.680 --> 00:03:45.280 align:middle line:84% on the right-hand side, because that gives us 00:03:45.280 --> 00:03:50.200 align:middle line:84% some baseline of how large we expect x to be, on average. 00:03:50.200 --> 00:03:52.140 align:middle line:84% So the larger its expectation is, 00:03:52.140 --> 00:03:55.710 align:middle line:84% the weaker my bound on this tail probability is. 00:03:55.710 --> 00:03:59.470 align:middle line:84% But as C increases, I also get some width. 00:03:59.470 --> 00:04:01.710 align:middle line:84% So we're going to prove this bound. 00:04:01.710 --> 00:04:04.870 align:middle line:84% And then we'll state another corollary, an equivalent way 00:04:04.870 --> 00:04:06.530 align:middle line:90% to rephrase Markov's bound. 00:04:06.530 --> 00:04:08.670 align:middle line:84% And you can go back and forth between them, 00:04:08.670 --> 00:04:10.870 align:middle line:84% depending on what's more convenient. 00:04:10.870 --> 00:04:13.270 align:middle line:84% But one thing I want to emphasize 00:04:13.270 --> 00:04:16.110 align:middle line:84% is that, here, I'm crucially going 00:04:16.110 --> 00:04:19.550 align:middle line:84% to need the fact that x is non-negative. 00:04:19.550 --> 00:04:23.030 align:middle line:84% So just as a spoiler, I'm going to ask you where in this proof 00:04:23.030 --> 00:04:25.372 align:middle line:84% I'm really appealing to non-negativity. 00:04:25.372 --> 00:04:26.830 align:middle line:84% And after I give you the proof, I'm 00:04:26.830 --> 00:04:31.950 align:middle line:84% going to expect you guys to tell me where this fact was used. 00:04:31.950 --> 00:04:34.710 align:middle line:90% So let's prove this. 00:04:34.710 --> 00:04:38.470 align:middle line:84% And the proof for this is really just breaking up the expectation 00:04:38.470 --> 00:04:40.510 align:middle line:90% into different pieces. 00:04:40.510 --> 00:04:45.090 align:middle line:84% So let's start off with our random variable x. 00:04:45.090 --> 00:04:48.470 align:middle line:90% We know its expectation, E of x. 00:04:48.470 --> 00:04:52.430 align:middle line:84% And I'm going to break this up using conditional expectation. 00:04:52.430 --> 00:04:58.910 align:middle line:90% 00:04:58.910 --> 00:05:01.390 align:middle line:84% So I'm going to break it up into two terms. 00:05:01.390 --> 00:05:06.110 align:middle line:84% I look at the conditional expectation 00:05:06.110 --> 00:05:09.990 align:middle line:84% of x given that x is at least some value 00:05:09.990 --> 00:05:15.430 align:middle line:84% C times the probability that x is at least C. 00:05:15.430 --> 00:05:18.870 align:middle line:84% And then I'm going to add the other contribution, which 00:05:18.870 --> 00:05:22.950 align:middle line:84% is the conditional expectation of x if that event does not 00:05:22.950 --> 00:05:26.230 align:middle line:84% happen, if x is strictly less than C, 00:05:26.230 --> 00:05:29.990 align:middle line:84% times the probability that x is less than C. 00:05:29.990 --> 00:05:31.910 align:middle line:90% So this should look familiar. 00:05:31.910 --> 00:05:34.950 align:middle line:84% This is basically a random variable version of something 00:05:34.950 --> 00:05:37.550 align:middle line:84% we covered way back when, which is 00:05:37.550 --> 00:05:40.310 align:middle line:84% we talked about the law of total probability. 00:05:40.310 --> 00:05:44.310 align:middle line:84% We had things, the probability of some event B. 00:05:44.310 --> 00:05:46.190 align:middle line:84% And we talked about it as the probability 00:05:46.190 --> 00:05:49.590 align:middle line:84% that B happens conditioned on event A happening 00:05:49.590 --> 00:05:53.790 align:middle line:84% times the probability that event A happens plus the probability 00:05:53.790 --> 00:05:56.270 align:middle line:84% that B happens conditioned on A not 00:05:56.270 --> 00:05:59.870 align:middle line:84% happening times the probability A does not happen. 00:05:59.870 --> 00:06:02.110 align:middle line:84% So when we talked about conditional probability, 00:06:02.110 --> 00:06:04.690 align:middle line:84% this was one of our first applications, 00:06:04.690 --> 00:06:07.030 align:middle line:84% once we had Bayes' rule, was we used 00:06:07.030 --> 00:06:10.270 align:middle line:84% it to reason using the law of total probability. 00:06:10.270 --> 00:06:13.230 align:middle line:84% And this is just the same thing, but expectations 00:06:13.230 --> 00:06:14.690 align:middle line:90% of random variables. 00:06:14.690 --> 00:06:16.190 align:middle line:84% Sometimes, you'll hear this referred 00:06:16.190 --> 00:06:18.830 align:middle line:84% to as the law of total expectation. 00:06:18.830 --> 00:06:20.670 align:middle line:84% But it's just breaking up my space 00:06:20.670 --> 00:06:22.350 align:middle line:90% into two different things-- 00:06:22.350 --> 00:06:25.310 align:middle line:84% either this event A happens, that x is at least C, 00:06:25.310 --> 00:06:26.870 align:middle line:90% or it does not happen. 00:06:26.870 --> 00:06:29.270 align:middle line:84% And then I average the conditional expectation 00:06:29.270 --> 00:06:32.590 align:middle line:84% in those two different possibilities. 00:06:32.590 --> 00:06:37.030 align:middle line:84% And so now, what I claim is that I can 00:06:37.030 --> 00:06:39.710 align:middle line:90% look at each of these pieces. 00:06:39.710 --> 00:06:42.950 align:middle line:84% I claim that the first piece right here 00:06:42.950 --> 00:06:46.150 align:middle line:84% is at least C. And then I'm going to pick up 00:06:46.150 --> 00:06:49.110 align:middle line:84% this probability of x at least C. 00:06:49.110 --> 00:06:51.830 align:middle line:84% And then I claim that the second piece right here 00:06:51.830 --> 00:06:53.910 align:middle line:90% is non-negative. 00:06:53.910 --> 00:06:56.710 align:middle line:90% So that I'm just going to drop. 00:06:56.710 --> 00:06:58.330 align:middle line:90% And I'll get a lower bound. 00:06:58.330 --> 00:07:01.630 align:middle line:84% So I've really just used the fact 00:07:01.630 --> 00:07:06.510 align:middle line:84% that the expectation of x, given that x is at least C, 00:07:06.510 --> 00:07:09.590 align:middle line:84% had better be a value at least C because x always 00:07:09.590 --> 00:07:11.505 align:middle line:84% takes on a value that's at least C. 00:07:11.505 --> 00:07:13.630 align:middle line:84% And then I'm still going to pick up the probability 00:07:13.630 --> 00:07:15.615 align:middle line:90% that that event happens. 00:07:15.615 --> 00:07:16.990 align:middle line:84% And for this term, I'm just going 00:07:16.990 --> 00:07:21.190 align:middle line:84% to drop it, because this term is non-negative. 00:07:21.190 --> 00:07:23.070 align:middle line:84% And so now, what I can do is I can just 00:07:23.070 --> 00:07:25.600 align:middle line:90% rearrange to finish the proof. 00:07:25.600 --> 00:07:32.950 align:middle line:90% 00:07:32.950 --> 00:07:33.990 align:middle line:90% That's it. 00:07:33.990 --> 00:07:36.510 align:middle line:90% So that's our first tail bound. 00:07:36.510 --> 00:07:39.330 align:middle line:84% Our tail bounds will get a bit more sophisticated from this. 00:07:39.330 --> 00:07:42.790 align:middle line:84% But this already gives us some rudimentary understanding 00:07:42.790 --> 00:07:45.972 align:middle line:84% of the probability that x is very, very large. 00:07:45.972 --> 00:07:47.430 align:middle line:84% So now, let me ask you the question 00:07:47.430 --> 00:07:49.590 align:middle line:84% that I promised I was going to ask you. 00:07:49.590 --> 00:07:55.790 align:middle line:84% Where in this proof did I use the fact that x is non-negative? 00:07:55.790 --> 00:07:57.150 align:middle line:90% Yeah? 00:07:57.150 --> 00:07:59.110 align:middle line:84% STUDENT: When you were counting x [INAUDIBLE] 00:07:59.110 --> 00:08:02.805 align:middle line:84% x given, x less than c, [INAUDIBLE]. 00:08:02.805 --> 00:08:03.930 align:middle line:90% ANKUR MOITRA: That's right. 00:08:03.930 --> 00:08:07.030 align:middle line:84% That's exactly right, because if x were not 00:08:07.030 --> 00:08:09.830 align:middle line:84% a non-negative random variable, I 00:08:09.830 --> 00:08:13.308 align:middle line:84% couldn't say that this term is non-negative. 00:08:13.308 --> 00:08:14.850 align:middle line:84% This term is definitely non-negative. 00:08:14.850 --> 00:08:17.050 align:middle line:84% But otherwise, if x took on negative values, 00:08:17.050 --> 00:08:18.390 align:middle line:90% this could be negative. 00:08:18.390 --> 00:08:21.970 align:middle line:84% And dropping it, I wouldn't get an inequality like this. 00:08:21.970 --> 00:08:23.630 align:middle line:90% Perfect. 00:08:23.630 --> 00:08:25.710 align:middle line:84% So we definitely have to use this. 00:08:25.710 --> 00:08:28.590 align:middle line:84% And that's crucial for Markov's bound, at least 00:08:28.590 --> 00:08:30.310 align:middle line:90% the intuition behind it. 00:08:30.310 --> 00:08:32.010 align:middle line:84% I mean, this is the algebraic proof. 00:08:32.010 --> 00:08:35.030 align:middle line:84% But the intuition is just that if x 00:08:35.030 --> 00:08:38.390 align:middle line:84% had a very large probability of being at least C, then 00:08:38.390 --> 00:08:40.190 align:middle line:84% its expectation could not be what 00:08:40.190 --> 00:08:43.230 align:middle line:84% I promised it was, because the expectation would actually 00:08:43.230 --> 00:08:45.190 align:middle line:90% be too large. 00:08:45.190 --> 00:08:47.390 align:middle line:90% Any questions? 00:08:47.390 --> 00:08:49.110 align:middle line:90% Make sense? 00:08:49.110 --> 00:08:51.230 align:middle line:90% All right. 00:08:51.230 --> 00:08:55.150 align:middle line:84% So let me restate this in another form. 00:08:55.150 --> 00:08:57.050 align:middle line:90% I'll state it as a corollary. 00:08:57.050 --> 00:09:01.050 align:middle line:84% This is just a restatement of Markov's bound. 00:09:01.050 --> 00:09:04.350 align:middle line:84% But sometimes, you'll find it convenient to invoke it 00:09:04.350 --> 00:09:06.670 align:middle line:90% in one form versus the other. 00:09:06.670 --> 00:09:09.130 align:middle line:84% So I claim, under the same conditions, 00:09:09.130 --> 00:09:18.870 align:middle line:84% if x is non-negative, then the probability 00:09:18.870 --> 00:09:27.390 align:middle line:84% that x is at least C times its expectation of x is at most 1 00:09:27.390 --> 00:09:29.950 align:middle line:90% over C. 00:09:29.950 --> 00:09:32.990 align:middle line:84% So let me make sure we're all on the same page. 00:09:32.990 --> 00:09:35.090 align:middle line:90% So why is this a corollary? 00:09:35.090 --> 00:09:38.390 align:middle line:90% 00:09:38.390 --> 00:09:43.070 align:middle line:84% How do I get from here, the thing I proved, to here? 00:09:43.070 --> 00:09:44.890 align:middle line:84% In fact, let me make it a bit easier. 00:09:44.890 --> 00:09:47.270 align:middle line:84% Let me just call this a different constant, C-prime. 00:09:47.270 --> 00:09:51.350 align:middle line:90% 00:09:51.350 --> 00:09:52.610 align:middle line:90% Why is this a corollary? 00:09:52.610 --> 00:09:53.370 align:middle line:90% They seem related. 00:09:53.370 --> 00:09:53.870 align:middle line:90% Yeah? 00:09:53.870 --> 00:09:58.910 align:middle line:84% STUDENT: You just substitute c prime [INAUDIBLE] c times e. 00:09:58.910 --> 00:09:59.567 align:middle line:90% [INAUDIBLE] 00:09:59.567 --> 00:10:01.650 align:middle line:84% ANKUR MOITRA: Yeah, so let me do it the other way. 00:10:01.650 --> 00:10:07.150 align:middle line:84% Let me substitute for C is equal to C-prime times Ex. 00:10:07.150 --> 00:10:10.110 align:middle line:84% Then I get exactly this bound right here. 00:10:10.110 --> 00:10:13.190 align:middle line:84% And then on the right-hand side, I cancel out the E of x. 00:10:13.190 --> 00:10:15.190 align:middle line:90% And I get 1 over C-prime. 00:10:15.190 --> 00:10:17.150 align:middle line:84% So this is very intuitive because, if I 00:10:17.150 --> 00:10:19.390 align:middle line:84% have a non-negative random variable, 00:10:19.390 --> 00:10:22.830 align:middle line:84% the probability that it's twice as large as its expectation 00:10:22.830 --> 00:10:24.587 align:middle line:90% is at most 0.5. 00:10:24.587 --> 00:10:26.670 align:middle line:84% Because otherwise, if that probability were larger 00:10:26.670 --> 00:10:29.080 align:middle line:84% than 0.5, the expectation would be wrong. 00:10:29.080 --> 00:10:31.960 align:middle line:90% 00:10:31.960 --> 00:10:34.990 align:middle line:90% So any questions about this? 00:10:34.990 --> 00:10:36.270 align:middle line:90% All right. 00:10:36.270 --> 00:10:41.530 align:middle line:84% So just to illustrate my point that, in tail bounds, 00:10:41.530 --> 00:10:44.870 align:middle line:84% we're going to be asking for tighter and tighter bounds that 00:10:44.870 --> 00:10:48.470 align:middle line:84% use more and more information about our random variable, 00:10:48.470 --> 00:10:50.070 align:middle line:84% let me do a running example, which 00:10:50.070 --> 00:10:55.910 align:middle line:84% we'll use as a way to check how much progress we're making. 00:10:55.910 --> 00:10:58.530 align:middle line:84% So let's do the following running example. 00:10:58.530 --> 00:11:02.270 align:middle line:84% And we'll come back to it with our next tail bound. 00:11:02.270 --> 00:11:17.690 align:middle line:84% So in the US, the average male height is about 69 inches. 00:11:17.690 --> 00:11:24.150 align:middle line:84% So we can use this fact to reason about the probability 00:11:24.150 --> 00:11:26.190 align:middle line:84% that someone is super duper tall, 00:11:26.190 --> 00:11:31.070 align:middle line:84% because x being the height of a random male in the US 00:11:31.070 --> 00:11:32.870 align:middle line:90% is a random variable. 00:11:32.870 --> 00:11:35.350 align:middle line:90% So we know its expectation. 00:11:35.350 --> 00:11:39.190 align:middle line:84% And we can use Markov's bound and just see 00:11:39.190 --> 00:11:41.550 align:middle line:90% what Markov's bound tells us. 00:11:41.550 --> 00:11:44.460 align:middle line:84% It tells us the probability that x, which, remember, 00:11:44.460 --> 00:11:46.020 align:middle line:84% is our random variable-- this would 00:11:46.020 --> 00:11:54.820 align:middle line:84% be the height of a randomly chosen US male. 00:11:54.820 --> 00:12:00.140 align:middle line:84% The probability that it's at least 2 times its expectation, 00:12:00.140 --> 00:12:06.820 align:middle line:84% which is equal to 138 inches, or 11 foot 6 00:12:06.820 --> 00:12:10.580 align:middle line:84% inches, that probability is at most 0.5. 00:12:10.580 --> 00:12:13.660 align:middle line:84% So we've proven this amazing fact. 00:12:13.660 --> 00:12:16.300 align:middle line:84% You're walking down the street in Boston. 00:12:16.300 --> 00:12:19.500 align:middle line:84% The probability you see someone who's above 11 foot 6 00:12:19.500 --> 00:12:21.900 align:middle line:90% is at most 0.5. 00:12:21.900 --> 00:12:24.620 align:middle line:90% So pretty good or maybe not. 00:12:24.620 --> 00:12:26.660 align:middle line:84% So see, the problem is that Markov's 00:12:26.660 --> 00:12:30.060 align:middle line:84% bound-- you can actually check-- really is tight. 00:12:30.060 --> 00:12:33.000 align:middle line:84% It's just that real random variables, a lot of times, 00:12:33.000 --> 00:12:34.920 align:middle line:84% there's more information we know about it-- 00:12:34.920 --> 00:12:38.060 align:middle line:84% things like the variation of the random variable. 00:12:38.060 --> 00:12:41.380 align:middle line:84% The more information you tell me about the random variable, 00:12:41.380 --> 00:12:43.980 align:middle line:84% the tighter my tail bound will be. 00:12:43.980 --> 00:12:45.940 align:middle line:84% And this Markov is just the get me 00:12:45.940 --> 00:12:50.220 align:middle line:84% over first warm-up for a tail bound. 00:12:50.220 --> 00:12:53.060 align:middle line:84% So we'll come back to this example. 00:12:53.060 --> 00:12:56.700 align:middle line:84% I want to make sure we still have the right intuition 00:12:56.700 --> 00:12:58.220 align:middle line:90% for this proof. 00:12:58.220 --> 00:13:03.020 align:middle line:84% So let me ask you a diagnostic question, 00:13:03.020 --> 00:13:05.340 align:middle line:84% just to make sure we're on the same page. 00:13:05.340 --> 00:13:07.600 align:middle line:84% I think probably you can guess the answer to this. 00:13:07.600 --> 00:13:10.820 align:middle line:84% But what I really care about is to find an example. 00:13:10.820 --> 00:13:18.560 align:middle line:84% So does Markov's bound hold without non-negativity? 00:13:18.560 --> 00:13:24.540 align:middle line:90% 00:13:24.540 --> 00:13:26.920 align:middle line:84% So in the statement of the theorem, 00:13:26.920 --> 00:13:29.180 align:middle line:84% I definitely assumed that the random variable 00:13:29.180 --> 00:13:30.540 align:middle line:90% was non-negative. 00:13:30.540 --> 00:13:33.660 align:middle line:84% In the proof I gave you, I definitely used the fact 00:13:33.660 --> 00:13:36.180 align:middle line:90% that x was non-negative. 00:13:36.180 --> 00:13:38.780 align:middle line:84% We talked about where we used that. 00:13:38.780 --> 00:13:41.660 align:middle line:84% But could I hope for some bound that looks like this, 00:13:41.660 --> 00:13:43.800 align:middle line:90% even when x is not non-negative? 00:13:43.800 --> 00:13:46.020 align:middle line:84% Maybe there's some other proof out there 00:13:46.020 --> 00:13:49.460 align:middle line:84% that doesn't use non-negativity that still bounds 00:13:49.460 --> 00:13:54.480 align:middle line:84% the probability that x is twice its expectation by 0.5 00:13:54.480 --> 00:13:56.260 align:middle line:90% or something like this. 00:13:56.260 --> 00:13:59.040 align:middle line:84% So I claim that this is spectacularly false, 00:13:59.040 --> 00:14:00.900 align:middle line:90% but this is good intuition. 00:14:00.900 --> 00:14:04.220 align:middle line:84% So how would I come up with some random variable that 00:14:04.220 --> 00:14:07.780 align:middle line:84% has negative values and completely breaks 00:14:07.780 --> 00:14:10.700 align:middle line:84% any kind of statement like this, that the expectation gives you 00:14:10.700 --> 00:14:13.980 align:middle line:90% a handle on the tails? 00:14:13.980 --> 00:14:16.660 align:middle line:90% Anyone have any intuition? 00:14:16.660 --> 00:14:17.460 align:middle line:90% Yeah? 00:14:17.460 --> 00:14:19.500 align:middle line:90% STUDENT: [INAUDIBLE] 00:14:19.500 --> 00:14:22.223 align:middle line:90% 00:14:22.223 --> 00:14:23.140 align:middle line:90% ANKUR MOITRA: Perfect. 00:14:23.140 --> 00:14:24.820 align:middle line:90% I love it. 00:14:24.820 --> 00:14:26.760 align:middle line:90% So this is definitely not true. 00:14:26.760 --> 00:14:29.260 align:middle line:90% It's spectacularly false. 00:14:29.260 --> 00:14:32.740 align:middle line:84% So let's just consider some x where 00:14:32.740 --> 00:14:34.840 align:middle line:84% the right-hand side doesn't make any sense 00:14:34.840 --> 00:14:36.980 align:middle line:90% because the expectation is 0. 00:14:36.980 --> 00:14:44.380 align:middle line:84% So what if x is 1 with probability 0.5 and it's minus 1 00:14:44.380 --> 00:14:45.900 align:middle line:90% otherwise? 00:14:45.900 --> 00:14:51.100 align:middle line:84% The trouble is that the expectation of x is equal to 0. 00:14:51.100 --> 00:14:55.700 align:middle line:84% But the probability that, say, x is at least 3 times its 00:14:55.700 --> 00:15:03.580 align:middle line:84% expectation, which is still 0, is definitely not less than 1/3. 00:15:03.580 --> 00:15:05.760 align:middle line:84% So non-negativity really is essential. 00:15:05.760 --> 00:15:07.420 align:middle line:84% And sometimes, these examples are 00:15:07.420 --> 00:15:10.540 align:middle line:84% very useful for both building intuition 00:15:10.540 --> 00:15:12.763 align:middle line:84% and conveying a proof to your reader. 00:15:12.763 --> 00:15:14.180 align:middle line:84% So these kinds of things, when you 00:15:14.180 --> 00:15:16.380 align:middle line:84% start thinking about your term paper, 00:15:16.380 --> 00:15:19.000 align:middle line:84% even if I didn't literally need to show this to you, 00:15:19.000 --> 00:15:21.180 align:middle line:84% this is the good kind of thing to keep in mind 00:15:21.180 --> 00:15:23.700 align:middle line:90% as a pedagogical device. 00:15:23.700 --> 00:15:30.620 align:middle line:84% All right, so let's prove our next tail bound. 00:15:30.620 --> 00:15:33.620 align:middle line:84% This one will be fairly simple as well, 00:15:33.620 --> 00:15:35.820 align:middle line:90% but it's much more powerful. 00:15:35.820 --> 00:15:40.020 align:middle line:84% And we're going to get some awesome applications of it. 00:15:40.020 --> 00:15:43.620 align:middle line:84% And this is called Chebyshev's bound. 00:15:43.620 --> 00:15:45.500 align:middle line:84% Actually, I'm not even sure about the history 00:15:45.500 --> 00:15:49.420 align:middle line:84% because I think Chebyshev's bound was proofed 00:15:49.420 --> 00:15:52.220 align:middle line:84% before Markov's bound, even though it's going to use 00:15:52.220 --> 00:15:53.833 align:middle line:90% the proof of Markov's bound. 00:15:53.833 --> 00:15:54.500 align:middle line:90% So I don't know. 00:15:54.500 --> 00:15:56.580 align:middle line:84% Maybe Markov just looked at his proof 00:15:56.580 --> 00:16:01.420 align:middle line:84% and called the thing inside it his own bound. 00:16:01.420 --> 00:16:03.980 align:middle line:84% So in any case, now, we're going to start off 00:16:03.980 --> 00:16:06.000 align:middle line:90% with any random variable x. 00:16:06.000 --> 00:16:07.380 align:middle line:84% So this is already very different 00:16:07.380 --> 00:16:09.700 align:middle line:84% than Markov's bound because I'm not 00:16:09.700 --> 00:16:11.540 align:middle line:90% going to assume non-negativity. 00:16:11.540 --> 00:16:14.260 align:middle line:84% I don't even need it for this proof. 00:16:14.260 --> 00:16:18.300 align:middle line:84% So for any random variable x, what I claim 00:16:18.300 --> 00:16:21.660 align:middle line:84% is that the probability that the absolute value 00:16:21.660 --> 00:16:28.300 align:middle line:84% of x minus the expectation of x is larger than C-- 00:16:28.300 --> 00:16:31.440 align:middle line:84% so it deviates not by a multiplicative factor of C, 00:16:31.440 --> 00:16:35.300 align:middle line:84% but by some additive factor of C from its expectation-- 00:16:35.300 --> 00:16:43.180 align:middle line:84% is bounded by the variance of x all over C squared. 00:16:43.180 --> 00:16:47.500 align:middle line:84% So this will be my standard form of Chebyshev's bound. 00:16:47.500 --> 00:16:49.600 align:middle line:84% And the same way for Markov's bound-- 00:16:49.600 --> 00:16:53.220 align:middle line:84% I stated it in one form, and then I stated a corollary-- 00:16:53.220 --> 00:16:56.260 align:middle line:84% I'll do the same after we prove this bound. 00:16:56.260 --> 00:16:58.740 align:middle line:84% But this is what I promised because now we're 00:16:58.740 --> 00:17:01.340 align:middle line:90% going to use extra information. 00:17:01.340 --> 00:17:05.593 align:middle line:84% See, Markov's bound did not have the variance showing up. 00:17:05.593 --> 00:17:08.260 align:middle line:84% So you have to actually be able to compute the variance in order 00:17:08.260 --> 00:17:10.435 align:middle line:84% to appeal to the Chebyshev bound. 00:17:10.435 --> 00:17:12.060 align:middle line:84% But as we'll see, it's going to give us 00:17:12.060 --> 00:17:13.859 align:middle line:84% much tighter bounds and some really 00:17:13.859 --> 00:17:16.940 align:middle line:90% cool qualitative corollaries. 00:17:16.940 --> 00:17:19.680 align:middle line:84% So let's prove Chebyshev's bound. 00:17:19.680 --> 00:17:22.020 align:middle line:84% And then I'll state the other corollary. 00:17:22.020 --> 00:17:25.260 align:middle line:84% The proof will also be short and sweet. 00:17:25.260 --> 00:17:28.540 align:middle line:84% But it's going to have a really cool trick to it, which 00:17:28.540 --> 00:17:31.700 align:middle line:84% is a hint about the tricks to come later when 00:17:31.700 --> 00:17:35.460 align:middle line:90% we do much fancier tail bounds. 00:17:35.460 --> 00:17:37.560 align:middle line:84% So the first thing I'm going to do-- 00:17:37.560 --> 00:17:41.020 align:middle line:90% 00:17:41.020 --> 00:17:47.620 align:middle line:84% I can rewrite the quantity I care 00:17:47.620 --> 00:17:52.740 align:middle line:84% about, the probability that the absolute value of x minus E 00:17:52.740 --> 00:18:01.260 align:middle line:84% of x is at least C I claim is equal to the probability 00:18:01.260 --> 00:18:11.152 align:middle line:84% that x minus E of X quantity squared is at least C squared. 00:18:11.152 --> 00:18:15.100 align:middle line:84% So this is trick number 1 for Chebyshev's bound. 00:18:15.100 --> 00:18:16.920 align:middle line:84% It's not a hard trick to explain. 00:18:16.920 --> 00:18:18.500 align:middle line:90% But this is a trick. 00:18:18.500 --> 00:18:21.500 align:middle line:84% Why are these two probabilities equal? 00:18:21.500 --> 00:18:23.860 align:middle line:84% Why am I allowed to say that these two probabilities are 00:18:23.860 --> 00:18:24.360 align:middle line:90% equal? 00:18:24.360 --> 00:18:28.260 align:middle line:90% 00:18:28.260 --> 00:18:31.370 align:middle line:84% It's because they're the same event. 00:18:31.370 --> 00:18:33.850 align:middle line:84% So at the end of the day, what's happening in here is 00:18:33.850 --> 00:18:35.410 align:middle line:90% I have an event. 00:18:35.410 --> 00:18:39.210 align:middle line:84% Once you fix the random variable, I know its expectation 00:18:39.210 --> 00:18:42.450 align:middle line:84% and you fix C. I have this event, whether or not 00:18:42.450 --> 00:18:45.930 align:middle line:90% x is far from its expectation. 00:18:45.930 --> 00:18:48.070 align:middle line:84% And I claim that these two probabilities-- 00:18:48.070 --> 00:18:50.810 align:middle line:84% what's going on in here are two different events. 00:18:50.810 --> 00:18:52.930 align:middle line:90% But these events coincide. 00:18:52.930 --> 00:18:54.770 align:middle line:84% The only way that one event triggers 00:18:54.770 --> 00:18:58.170 align:middle line:84% is if the other one does too, just because we're 00:18:58.170 --> 00:19:00.250 align:middle line:90% taking the absolute value. 00:19:00.250 --> 00:19:02.570 align:middle line:90% Make sense? 00:19:02.570 --> 00:19:06.250 align:middle line:84% So that's trick number one for Chebyshev. 00:19:06.250 --> 00:19:08.890 align:middle line:84% And now, here's the really cool trick. 00:19:08.890 --> 00:19:11.930 align:middle line:84% What we're going to do is we're going to define 00:19:11.930 --> 00:19:13.360 align:middle line:90% a new random variable. 00:19:13.360 --> 00:19:16.050 align:middle line:90% 00:19:16.050 --> 00:19:18.850 align:middle line:84% So I'm going to call this random variable y. 00:19:18.850 --> 00:19:21.650 align:middle line:84% And it's going to be equal to just the thing that shows up 00:19:21.650 --> 00:19:24.050 align:middle line:90% in here on the right-hand side-- 00:19:24.050 --> 00:19:29.410 align:middle line:84% x minus the expectation of x quantity squared. 00:19:29.410 --> 00:19:31.170 align:middle line:84% See, I told you, in Chebyshev, we're 00:19:31.170 --> 00:19:35.570 align:middle line:84% going to appeal to Markov's bound in order to prove it. 00:19:35.570 --> 00:19:37.510 align:middle line:84% But if you remember, for Markov's bound, 00:19:37.510 --> 00:19:40.090 align:middle line:84% we needed that the random variable we are looking at 00:19:40.090 --> 00:19:41.810 align:middle line:90% is non-negative. 00:19:41.810 --> 00:19:44.010 align:middle line:84% For Chebyshev, I'm not assuming that x 00:19:44.010 --> 00:19:45.850 align:middle line:90% is non-negative to begin with. 00:19:45.850 --> 00:19:48.090 align:middle line:84% So what I have to do is I have to manufacture 00:19:48.090 --> 00:19:50.850 align:middle line:84% my own non-negative random variable. 00:19:50.850 --> 00:19:53.770 align:middle line:84% And that's this new random variable y 00:19:53.770 --> 00:19:56.050 align:middle line:84% that I just constructed right here. 00:19:56.050 --> 00:19:58.130 align:middle line:84% This is definitely non-negative because I'm 00:19:58.130 --> 00:20:00.450 align:middle line:90% taking the square of a value. 00:20:00.450 --> 00:20:04.370 align:middle line:84% And now, I can appeal to Markov's bound. 00:20:04.370 --> 00:20:13.370 align:middle line:84% So using Markov on y, we know that this quantity 00:20:13.370 --> 00:20:16.490 align:middle line:84% right here, which I'll call star-- 00:20:16.490 --> 00:20:24.130 align:middle line:84% star is equal to the probability that y is at least C squared. 00:20:24.130 --> 00:20:26.850 align:middle line:84% That's literally just using the definition of what 00:20:26.850 --> 00:20:29.490 align:middle line:90% my random variable y was. 00:20:29.490 --> 00:20:31.730 align:middle line:84% And now, appealing to Markov's bound, which 00:20:31.730 --> 00:20:34.650 align:middle line:84% is allowed because y is non-negative, 00:20:34.650 --> 00:20:42.290 align:middle line:84% this is at most the expectation of y all over C squared. 00:20:42.290 --> 00:20:45.690 align:middle line:84% Notice that my threshold here is C squared, not C, 00:20:45.690 --> 00:20:48.810 align:middle line:84% as in the original statement of Markov's bound. 00:20:48.810 --> 00:20:53.970 align:middle line:84% And now, what I can do is I can just remember what y is. 00:20:53.970 --> 00:20:57.730 align:middle line:90% So y is just the expectation-- 00:20:57.730 --> 00:21:02.130 align:middle line:84% y itself is x minus the expectation 00:21:02.130 --> 00:21:05.050 align:middle line:90% of x quantity squared. 00:21:05.050 --> 00:21:07.210 align:middle line:90% I still have my C squared. 00:21:07.210 --> 00:21:10.050 align:middle line:84% And when I take the expectation of y, 00:21:10.050 --> 00:21:12.330 align:middle line:84% this is literally the definition I gave you 00:21:12.330 --> 00:21:15.810 align:middle line:90% earlier for the variance of x. 00:21:15.810 --> 00:21:16.930 align:middle line:90% So that's my proof. 00:21:16.930 --> 00:21:20.330 align:middle line:90% 00:21:20.330 --> 00:21:22.890 align:middle line:90% Any questions? 00:21:22.890 --> 00:21:24.930 align:middle line:90% Good? 00:21:24.930 --> 00:21:26.130 align:middle line:90% All right. 00:21:26.130 --> 00:21:30.490 align:middle line:84% So as before, I'm going to state another equivalent way 00:21:30.490 --> 00:21:32.810 align:middle line:84% to think about Chebyshev, which is sometimes 00:21:32.810 --> 00:21:36.050 align:middle line:90% a bit more convenient. 00:21:36.050 --> 00:21:40.650 align:middle line:84% So feel free to use whichever is easier in a proof. 00:21:40.650 --> 00:21:44.810 align:middle line:84% So again, for any random variable x, 00:21:44.810 --> 00:21:49.410 align:middle line:84% the probability that the absolute value 00:21:49.410 --> 00:21:55.950 align:middle line:84% of x minus its expectation is larger than C times sigma of x-- 00:21:55.950 --> 00:21:58.422 align:middle line:90% I'll define what sigma of x is-- 00:21:58.422 --> 00:22:01.890 align:middle line:90% is at most 1 over C squared. 00:22:01.890 --> 00:22:06.670 align:middle line:84% And sigma of x is called the standard deviation. 00:22:06.670 --> 00:22:10.010 align:middle line:90% 00:22:10.010 --> 00:22:12.770 align:middle line:84% And it's just defined to be the square root 00:22:12.770 --> 00:22:16.370 align:middle line:90% of the variance of x. 00:22:16.370 --> 00:22:19.490 align:middle line:84% So the same way I had this simple version of Markov that 00:22:19.490 --> 00:22:21.000 align:middle line:90% allowed me to remember it-- 00:22:21.000 --> 00:22:23.250 align:middle line:84% the probability that a non-negative random variable is 00:22:23.250 --> 00:22:26.770 align:middle line:84% twice its expectation patient is at most 0.5-- 00:22:26.770 --> 00:22:31.930 align:middle line:84% the probability that a random variable is 2 times away 00:22:31.930 --> 00:22:35.490 align:middle line:84% from its expectation in terms of its standard deviation 00:22:35.490 --> 00:22:41.290 align:middle line:90% is at most a 0.25. 00:22:41.290 --> 00:22:44.630 align:middle line:84% All right, so let's go back to our running example. 00:22:44.630 --> 00:22:49.310 align:middle line:84% And let's see why Chebyshev is a lot more useful for us. 00:22:49.310 --> 00:23:01.890 align:middle line:90% 00:23:01.890 --> 00:23:05.130 align:middle line:84% So in particular, I need to tell you more information 00:23:05.130 --> 00:23:08.370 align:middle line:84% about this random variable, about the height 00:23:08.370 --> 00:23:11.650 align:middle line:90% of a random male, US male. 00:23:11.650 --> 00:23:18.570 align:middle line:84% And what I claim is that the standard deviation 00:23:18.570 --> 00:23:27.810 align:middle line:90% of male height is 2.7 inches. 00:23:27.810 --> 00:23:31.230 align:middle line:84% So now, we're in good shape because instead of using Markov, 00:23:31.230 --> 00:23:33.670 align:middle line:90% we can instead use Chebyshev. 00:23:33.670 --> 00:23:38.050 align:middle line:90% 00:23:38.050 --> 00:23:40.970 align:middle line:84% We have the probability that x, where 00:23:40.970 --> 00:23:43.930 align:middle line:84% x is our same random variable as before-- 00:23:43.930 --> 00:23:48.130 align:middle line:84% the probability that you're 2 standard deviations 00:23:48.130 --> 00:23:56.170 align:middle line:84% above average, which is the same thing as 6 foot 2.4, 00:23:56.170 --> 00:23:59.250 align:middle line:90% is at most 0.25. 00:23:59.250 --> 00:24:01.350 align:middle line:84% So this is still not a great bound. 00:24:01.350 --> 00:24:03.250 align:middle line:84% But it's at least not a nonsensical 00:24:03.250 --> 00:24:07.610 align:middle line:84% bound, like our 11 foot 6 inches version before. 00:24:07.610 --> 00:24:11.490 align:middle line:84% And so you can do the same types of exercise. 00:24:11.490 --> 00:24:13.530 align:middle line:84% When you tell me not just the variance 00:24:13.530 --> 00:24:18.142 align:middle line:84% of x but more information about the moments of x, 00:24:18.142 --> 00:24:20.010 align:middle line:84% you would get tighter and tighter bounds 00:24:20.010 --> 00:24:22.390 align:middle line:84% until you would actually get something pretty reasonable. 00:24:22.390 --> 00:24:26.650 align:middle line:90% 00:24:26.650 --> 00:24:27.430 align:middle line:90% All right. 00:24:27.430 --> 00:24:30.010 align:middle line:90% 00:24:30.010 --> 00:24:33.650 align:middle line:84% So now, I want to tell you-- now that we have Chebyshev, 00:24:33.650 --> 00:24:35.690 align:middle line:84% it may not be obvious, but it turns out 00:24:35.690 --> 00:24:37.990 align:middle line:84% to be very qualitatively powerful, 00:24:37.990 --> 00:24:41.330 align:middle line:84% not just numerically in these examples like height. 00:24:41.330 --> 00:24:45.810 align:middle line:84% So let me tell you about one really cool corollary. 00:24:45.810 --> 00:24:47.030 align:middle line:90% I'll state it as a theorem. 00:24:47.030 --> 00:24:52.050 align:middle line:84% And then we'll talk about some applications to gambling. 00:24:52.050 --> 00:24:56.290 align:middle line:84% So Chebyshev allows you to prove one 00:24:56.290 --> 00:25:02.570 align:middle line:84% of the most important results in tail bounds, which is called 00:25:02.570 --> 00:25:04.992 align:middle line:90% the weak law of large numbers. 00:25:04.992 --> 00:25:06.450 align:middle line:84% Now, the weak law of large numbers, 00:25:06.450 --> 00:25:08.730 align:middle line:84% we're going to strengthen it at various times 00:25:08.730 --> 00:25:11.170 align:middle line:84% and get better quantitative bounds. 00:25:11.170 --> 00:25:16.290 align:middle line:84% But already, we can see some of the power of tail bounds. 00:25:16.290 --> 00:25:21.290 align:middle line:84% So let me state what this weak law of large numbers is. 00:25:21.290 --> 00:25:23.610 align:middle line:84% And then we're going to prove it just using 00:25:23.610 --> 00:25:26.310 align:middle line:84% the tail bound technology we've developed so far. 00:25:26.310 --> 00:25:33.690 align:middle line:90% 00:25:33.690 --> 00:25:37.130 align:middle line:84% Now, for the weak law of large numbers, 00:25:37.130 --> 00:25:39.470 align:middle line:84% you have to be very careful with the quantifiers. 00:25:39.470 --> 00:25:40.710 align:middle line:90% So let me write it down. 00:25:40.710 --> 00:25:42.730 align:middle line:90% And then we'll talk about it. 00:25:42.730 --> 00:25:46.730 align:middle line:84% So first, I'm going to fix some positive epsilon. 00:25:46.730 --> 00:25:50.410 align:middle line:84% That'll be some measure of how close I want a random variable 00:25:50.410 --> 00:25:52.422 align:middle line:90% to be to its expectation. 00:25:52.422 --> 00:25:56.970 align:middle line:90% 00:25:56.970 --> 00:26:01.410 align:middle line:84% And I'm going to let x1 all the way up 00:26:01.410 --> 00:26:17.290 align:middle line:84% to xn be independent copies of some fixed random variable x. 00:26:17.290 --> 00:26:23.720 align:middle line:84% And moreover, let's assume that the expectation of x is equal 00:26:23.720 --> 00:26:27.540 align:middle line:84% to some value C. And I'm also going to assume-- 00:26:27.540 --> 00:26:30.920 align:middle line:84% and this is critical for the weak law of large numbers-- 00:26:30.920 --> 00:26:35.880 align:middle line:84% that its variance is not infinity-- 00:26:35.880 --> 00:26:38.600 align:middle line:84% so just some basic bounds that my random variable 00:26:38.600 --> 00:26:41.520 align:middle line:90% is somewhat well-behaved. 00:26:41.520 --> 00:26:44.160 align:middle line:84% Well, what I'm going to do is I'm 00:26:44.160 --> 00:26:47.800 align:middle line:84% going to look at averages of these random variables 00:26:47.800 --> 00:26:52.720 align:middle line:84% as a way to try and approximate this expectation C. 00:26:52.720 --> 00:26:56.480 align:middle line:84% So I'm going to define a new random variable, Sn, that's 00:26:56.480 --> 00:27:00.080 align:middle line:90% just going to be the average. 00:27:00.080 --> 00:27:08.920 align:middle line:84% It'll be 1 over n sum from i equals 1 to n of xi. 00:27:08.920 --> 00:27:11.220 align:middle line:84% So just to make sure that we're on the same page, 00:27:11.220 --> 00:27:14.130 align:middle line:84% let's keep some mental model of this in mind. 00:27:14.130 --> 00:27:16.480 align:middle line:84% So one of the most important applications 00:27:16.480 --> 00:27:20.840 align:middle line:84% of the weak law of large numbers is to something like polling. 00:27:20.840 --> 00:27:23.880 align:middle line:84% So maybe you have some random variable, 00:27:23.880 --> 00:27:28.840 align:middle line:84% which is the probability that a random person selected 00:27:28.840 --> 00:27:32.240 align:middle line:84% from the electorate votes for one candidate, candidate 00:27:32.240 --> 00:27:35.000 align:middle line:90% A over candidate B. 00:27:35.000 --> 00:27:39.000 align:middle line:84% Now, you want to know what's the expectation of x 00:27:39.000 --> 00:27:42.080 align:middle line:84% because you want to know who's going to win the election. 00:27:42.080 --> 00:27:45.600 align:middle line:84% So you want to do this forecasting before election day. 00:27:45.600 --> 00:27:48.080 align:middle line:84% The natural thing you could try is 00:27:48.080 --> 00:27:52.000 align:middle line:84% you can try and get independent samples from x just 00:27:52.000 --> 00:27:54.400 align:middle line:84% by calling random people and asking them 00:27:54.400 --> 00:27:55.772 align:middle line:90% who they're going to vote for. 00:27:55.772 --> 00:27:57.480 align:middle line:84% Now, there are a lot of issues with that. 00:27:57.480 --> 00:28:00.560 align:middle line:84% That won't really give you independent random variables. 00:28:00.560 --> 00:28:03.080 align:middle line:84% But if we pretend for the moment that it did, 00:28:03.080 --> 00:28:06.480 align:middle line:84% each of these xi's is a realization 00:28:06.480 --> 00:28:08.320 align:middle line:90% of these random variables. 00:28:08.320 --> 00:28:10.880 align:middle line:90% What would my polling scheme do? 00:28:10.880 --> 00:28:13.360 align:middle line:84% I would just choose some large denominator n 00:28:13.360 --> 00:28:15.680 align:middle line:84% for the number of people I'm going to poll. 00:28:15.680 --> 00:28:19.600 align:middle line:84% I'm going to average up all of their votes, which way they're 00:28:19.600 --> 00:28:20.100 align:middle line:90% voting. 00:28:20.100 --> 00:28:23.000 align:middle line:84% Are they voting for candidate A or candidate B? 00:28:23.000 --> 00:28:25.920 align:middle line:84% And what I'm hoping is that the empirical average 00:28:25.920 --> 00:28:28.880 align:middle line:84% of their votes of these random variables 00:28:28.880 --> 00:28:33.480 align:middle line:84% converges very well to my true expectation 00:28:33.480 --> 00:28:36.400 align:middle line:84% so that I can forecast who's going to win the election. 00:28:36.400 --> 00:28:39.720 align:middle line:84% And so that's exactly what the weak law of large numbers 00:28:39.720 --> 00:28:40.500 align:middle line:90% tells us. 00:28:40.500 --> 00:28:43.720 align:middle line:84% You just have to be careful with all the quantifiers. 00:28:43.720 --> 00:28:54.040 align:middle line:84% Then what I claim is that the probability that Sn is epsilon 00:28:54.040 --> 00:28:58.960 align:middle line:84% away from the true value C that this quantity, as you take 00:28:58.960 --> 00:29:01.080 align:middle line:84% the limit, as the number of people 00:29:01.080 --> 00:29:05.040 align:middle line:84% you're sampling in your poll goes to infinity, 00:29:05.040 --> 00:29:07.600 align:middle line:90% is going to go to 0. 00:29:07.600 --> 00:29:10.520 align:middle line:84% So there are a ton of quantifiers here. 00:29:10.520 --> 00:29:14.220 align:middle line:84% So let's say that I fix epsilon at the very outset. 00:29:14.220 --> 00:29:16.240 align:middle line:84% I want to figure out the probability people 00:29:16.240 --> 00:29:19.960 align:middle line:84% are voting for candidate A within some additive tolerance, 00:29:19.960 --> 00:29:21.520 align:middle line:90% 0.01. 00:29:21.520 --> 00:29:24.600 align:middle line:84% And so once I've fixed my epsilon, 00:29:24.600 --> 00:29:29.000 align:middle line:84% this is my success criteria, is that my empirical average 00:29:29.000 --> 00:29:30.760 align:middle line:90% of the people I poll-- 00:29:30.760 --> 00:29:35.960 align:middle line:84% that empirical average should be within an additive 0.01 00:29:35.960 --> 00:29:37.320 align:middle line:90% of the truth. 00:29:37.320 --> 00:29:40.520 align:middle line:84% And the claim is whatever you fix for epsilon, 00:29:40.520 --> 00:29:43.400 align:middle line:84% there's an n that's large enough so 00:29:43.400 --> 00:29:47.320 align:middle line:84% that the probability that you're far away by more than this 0.01 00:29:47.320 --> 00:29:49.680 align:middle line:90% is going to go to 0. 00:29:49.680 --> 00:29:51.360 align:middle line:84% So the weak law of large numbers is 00:29:51.360 --> 00:29:53.300 align:middle line:90% deliberately non-quantitative. 00:29:53.300 --> 00:29:55.700 align:middle line:84% And we will get quantitative versions out of it, 00:29:55.700 --> 00:29:58.280 align:middle line:84% even just from the proof and from strengthening 00:29:58.280 --> 00:29:59.840 align:middle line:90% of our tail bounds. 00:29:59.840 --> 00:30:02.520 align:middle line:84% But it just tells you some basic conditions 00:30:02.520 --> 00:30:05.020 align:middle line:84% that, if you're given enough samples, 00:30:05.020 --> 00:30:09.040 align:middle line:84% you really can approximate the expectation to any desired 00:30:09.040 --> 00:30:10.040 align:middle line:90% accuracy. 00:30:10.040 --> 00:30:14.160 align:middle line:84% And in the proof, we'll see how large little n needs to be. 00:30:14.160 --> 00:30:17.040 align:middle line:84% So this is quite a mouthful of a statement. 00:30:17.040 --> 00:30:19.550 align:middle line:84% Are there any questions about the weak law of large numbers? 00:30:19.550 --> 00:30:23.960 align:middle line:90% 00:30:23.960 --> 00:30:25.360 align:middle line:84% Give me a thumbs up if this makes 00:30:25.360 --> 00:30:26.940 align:middle line:90% sense, at least as a statement. 00:30:26.940 --> 00:30:28.360 align:middle line:90% OK, good. 00:30:28.360 --> 00:30:29.560 align:middle line:90% All right. 00:30:29.560 --> 00:30:32.380 align:middle line:84% So the order of quantifiers is super important for this. 00:30:32.380 --> 00:30:36.200 align:middle line:84% Now, let's prove the weak law of large numbers. 00:30:36.200 --> 00:30:43.360 align:middle line:84% The good thing is that the proof will be pretty straightforward 00:30:43.360 --> 00:30:48.680 align:middle line:84% now that we have Chebyshev's bound. 00:30:48.680 --> 00:30:53.760 align:middle line:84% So first, I claim that the expectation of Sn-- 00:30:53.760 --> 00:30:55.880 align:middle line:84% remember, that's a random variable itself 00:30:55.880 --> 00:30:59.440 align:middle line:84% because each of the xi's is a random variable. 00:30:59.440 --> 00:31:03.812 align:middle line:84% And I could first care about-- is its expectation correct? 00:31:03.812 --> 00:31:07.960 align:middle line:84% So now, I can use linearity of expectation 00:31:07.960 --> 00:31:12.480 align:middle line:84% to write it out as the sum of the expectations of each 00:31:12.480 --> 00:31:16.280 align:middle line:84% of the individual xi's that form it. 00:31:16.280 --> 00:31:18.600 align:middle line:84% Now, because each of these xis are 00:31:18.600 --> 00:31:24.320 align:middle line:84% assumed to be independent copies of this fixed random variable x, 00:31:24.320 --> 00:31:28.920 align:middle line:84% whose expectation is C, I know that each of their expectations 00:31:28.920 --> 00:31:32.420 align:middle line:84% is C. So that means, when I average their expectations, 00:31:32.420 --> 00:31:34.260 align:middle line:90% I get the same thing back again. 00:31:34.260 --> 00:31:39.040 align:middle line:84% I get C. So this is just a basic sanity check. 00:31:39.040 --> 00:31:41.640 align:middle line:84% If I'm trying to approximate the expectation 00:31:41.640 --> 00:31:46.560 align:middle line:84% of my random variable x with another random variable Sn, 00:31:46.560 --> 00:31:50.140 align:middle line:84% I better have that the expectation of Sn is equal to x. 00:31:50.140 --> 00:31:51.820 align:middle line:84% Otherwise, I'm going to have trouble. 00:31:51.820 --> 00:31:53.280 align:middle line:90% So this makes sense. 00:31:53.280 --> 00:31:56.600 align:middle line:84% This just means that I have an unbiased estimator of the thing 00:31:56.600 --> 00:31:58.960 align:middle line:90% that I really care about. 00:31:58.960 --> 00:32:01.780 align:middle line:84% Now, the crucial thing is computing the variance. 00:32:01.780 --> 00:32:04.700 align:middle line:84% We're not going to be able to get by by using Markov's bound. 00:32:04.700 --> 00:32:07.240 align:middle line:84% We're really going to have to use Chebyshev. 00:32:07.240 --> 00:32:10.400 align:middle line:84% And that's where independence is going to save us. 00:32:10.400 --> 00:32:12.340 align:middle line:84% So remember, for Chebyshev's bound, 00:32:12.340 --> 00:32:14.840 align:middle line:84% I need to know what the variance of my random variable 00:32:14.840 --> 00:32:19.480 align:middle line:84% is in order to quantify how close a random variable will be 00:32:19.480 --> 00:32:22.320 align:middle line:84% to its expectation, because that's the thing that shows up 00:32:22.320 --> 00:32:24.680 align:middle line:90% in my statement of Chebyshev. 00:32:24.680 --> 00:32:26.840 align:middle line:84% The main action for the weak law of large numbers 00:32:26.840 --> 00:32:30.760 align:middle line:84% is just computing what this variance of Sn is. 00:32:30.760 --> 00:32:32.700 align:middle line:84% So I can do the same thing as before. 00:32:32.700 --> 00:32:39.080 align:middle line:84% I can replace Sn with what it is defined as, in terms 00:32:39.080 --> 00:32:39.890 align:middle line:90% of the xi's. 00:32:39.890 --> 00:32:42.520 align:middle line:90% 00:32:42.520 --> 00:32:46.000 align:middle line:84% And now, what happens when I scale a random variable 00:32:46.000 --> 00:32:48.360 align:middle line:90% by some scalar C-- 00:32:48.360 --> 00:32:50.200 align:middle line:90% in this case 1 over n? 00:32:50.200 --> 00:32:54.160 align:middle line:84% Well, the variance is going to scale by a C squared, 00:32:54.160 --> 00:32:56.800 align:middle line:84% because I'm taking the expectation of some quantity 00:32:56.800 --> 00:32:58.640 align:middle line:90% squared in there. 00:32:58.640 --> 00:33:01.840 align:middle line:84% So this is the same thing as 1 over n 00:33:01.840 --> 00:33:08.430 align:middle line:84% squared times the variance of the sum of the xi's from i 00:33:08.430 --> 00:33:10.950 align:middle line:90% equals 1 to n. 00:33:10.950 --> 00:33:13.910 align:middle line:84% And now, the main trick is that I claim 00:33:13.910 --> 00:33:15.450 align:middle line:90% that this is now equal to-- 00:33:15.450 --> 00:33:18.190 align:middle line:84% I'll carry through my 1 over n squared. 00:33:18.190 --> 00:33:24.430 align:middle line:84% I claim that this is equal to n times the variance of each 00:33:24.430 --> 00:33:26.470 align:middle line:90% of these random variables x. 00:33:26.470 --> 00:33:30.110 align:middle line:84% So this is the key step in the weak law of large numbers. 00:33:30.110 --> 00:33:32.950 align:middle line:84% This is the part that I want to dwell on. 00:33:32.950 --> 00:33:35.430 align:middle line:90% So why is this step OK? 00:33:35.430 --> 00:33:38.350 align:middle line:84% So how did I go from here to here? 00:33:38.350 --> 00:33:40.270 align:middle line:84% We have to remember way back when with what 00:33:40.270 --> 00:33:42.130 align:middle line:84% I told you when we did the probability unit. 00:33:42.130 --> 00:33:42.910 align:middle line:90% Yes? 00:33:42.910 --> 00:33:44.050 align:middle line:84% STUDENT: Because they're independent. 00:33:44.050 --> 00:33:45.370 align:middle line:84% ANKUR MOITRA: Because they're independent. 00:33:45.370 --> 00:33:46.190 align:middle line:90% That's right. 00:33:46.190 --> 00:33:49.930 align:middle line:84% So remember, we had things like the linearity of expectation. 00:33:49.930 --> 00:33:51.370 align:middle line:90% We add two random variables. 00:33:51.370 --> 00:33:53.550 align:middle line:84% Their expectation of their sum is 00:33:53.550 --> 00:33:55.390 align:middle line:90% the sum of their expectations. 00:33:55.390 --> 00:33:57.650 align:middle line:84% That assumed nothing about the random variables. 00:33:57.650 --> 00:34:00.110 align:middle line:90% It was always true. 00:34:00.110 --> 00:34:02.230 align:middle line:84% But when I talked about the variance of two 00:34:02.230 --> 00:34:04.350 align:middle line:84% random variables, the variance of the sum 00:34:04.350 --> 00:34:08.510 align:middle line:84% of two random variables is equal to the sum of their variances, 00:34:08.510 --> 00:34:11.630 align:middle line:90% but only if I have independence. 00:34:11.630 --> 00:34:14.350 align:middle line:84% So the crucial assumption that these are independent 00:34:14.350 --> 00:34:17.373 align:middle line:84% draws I need in order to get this step. 00:34:17.373 --> 00:34:18.790 align:middle line:84% And that's where we're going to be 00:34:18.790 --> 00:34:21.469 align:middle line:84% winning, because you can see that I pulled out a 1 00:34:21.469 --> 00:34:22.989 align:middle line:90% over n squared. 00:34:22.989 --> 00:34:25.750 align:middle line:84% And it's only canceling out one of these. 00:34:25.750 --> 00:34:26.989 align:middle line:90% It'll cancel out this n. 00:34:26.989 --> 00:34:29.870 align:middle line:84% But I'll still have a 1 over n left over. 00:34:29.870 --> 00:34:32.310 align:middle line:84% So that's why, as I take more and more samples, 00:34:32.310 --> 00:34:35.110 align:middle line:84% I'm going to get better and better concentration. 00:34:35.110 --> 00:34:37.989 align:middle line:84% Just to completely drive home the point, 00:34:37.989 --> 00:34:42.230 align:middle line:84% we needed independence in order to get this last step. 00:34:42.230 --> 00:34:44.989 align:middle line:84% If I didn't have independence here, 00:34:44.989 --> 00:34:47.830 align:middle line:84% what would my polling methodology be? 00:34:47.830 --> 00:34:50.790 align:middle line:84% I'm trying to get a hint about who's going to win the election. 00:34:50.790 --> 00:34:54.429 align:middle line:84% And I call the same guy over and over again and ask him 00:34:54.429 --> 00:34:55.790 align:middle line:90% his opinion. 00:34:55.790 --> 00:34:57.390 align:middle line:84% That would be a pretty ridiculous way 00:34:57.390 --> 00:35:00.270 align:middle line:84% to get an idea about who's going to win the election. 00:35:00.270 --> 00:35:02.670 align:middle line:84% So you crucially need independence. 00:35:02.670 --> 00:35:04.610 align:middle line:90% But now, we can write this out. 00:35:04.610 --> 00:35:10.270 align:middle line:84% This is the variance of x over n. 00:35:10.270 --> 00:35:13.990 align:middle line:84% And now, we can just appeal to Chebyshev. 00:35:13.990 --> 00:35:17.710 align:middle line:84% And Chebyshev is going to get us our weak law of large numbers. 00:35:17.710 --> 00:35:24.470 align:middle line:84% So by Chebyshev, we know that the probability 00:35:24.470 --> 00:35:28.590 align:middle line:84% that Sn is off from its expectation 00:35:28.590 --> 00:35:32.670 align:middle line:84% by an additive epsilon is at most 00:35:32.670 --> 00:35:37.830 align:middle line:84% the variance of Sn over epsilon squared, which 00:35:37.830 --> 00:35:42.710 align:middle line:84% our right-hand side is the same thing as the variance of x 00:35:42.710 --> 00:35:47.070 align:middle line:90% over n times epsilon squared. 00:35:47.070 --> 00:35:49.230 align:middle line:84% So this tells me immediately that, 00:35:49.230 --> 00:35:53.350 align:middle line:84% as I want some better and better target accuracy 00:35:53.350 --> 00:35:55.630 align:middle line:90% for how accurate my poll is-- 00:35:55.630 --> 00:35:58.410 align:middle line:84% maybe I don't want like 0.05 error. 00:35:58.410 --> 00:36:01.670 align:middle line:84% I want 0.01-- what's going to happen 00:36:01.670 --> 00:36:05.750 align:middle line:84% is that my epsilon is getting 5 times as small. 00:36:05.750 --> 00:36:06.690 align:middle line:90% So what do I need? 00:36:06.690 --> 00:36:10.630 align:middle line:84% I need my n to be 25 times as big in order 00:36:10.630 --> 00:36:14.008 align:middle line:84% to get the same bound on the tail probability. 00:36:14.008 --> 00:36:15.550 align:middle line:84% So actually, in the course of proving 00:36:15.550 --> 00:36:17.350 align:middle line:84% the weak law of large numbers, we even 00:36:17.350 --> 00:36:20.430 align:middle line:84% got quantitative bounds that tell us 00:36:20.430 --> 00:36:24.190 align:middle line:84% exactly how large our n needs to be instead of hiding it 00:36:24.190 --> 00:36:27.910 align:middle line:90% all within this limit. 00:36:27.910 --> 00:36:30.370 align:middle line:84% So this is one of our cool applications. 00:36:30.370 --> 00:36:31.410 align:middle line:90% Any questions? 00:36:31.410 --> 00:36:34.990 align:middle line:90% 00:36:34.990 --> 00:36:36.590 align:middle line:90% Good? 00:36:36.590 --> 00:36:37.210 align:middle line:90% All right. 00:36:37.210 --> 00:36:41.510 align:middle line:90% 00:36:41.510 --> 00:36:44.830 align:middle line:84% So let's talk about an application to gambling. 00:36:44.830 --> 00:36:49.110 align:middle line:84% I hope none of you all are big gamblers. 00:36:49.110 --> 00:36:50.790 align:middle line:84% Actually, it turned out-- so when 00:36:50.790 --> 00:36:54.270 align:middle line:84% we bought our house in Belmont, we 00:36:54.270 --> 00:36:57.370 align:middle line:84% kept getting all these weird packages and fruits in the mail, 00:36:57.370 --> 00:36:59.055 align:middle line:90% like very fancy things. 00:36:59.055 --> 00:37:00.430 align:middle line:84% And around Christmas time, people 00:37:00.430 --> 00:37:03.850 align:middle line:84% would send us these very expensive wreaths. 00:37:03.850 --> 00:37:05.350 align:middle line:84% And it took us a while to figure out 00:37:05.350 --> 00:37:07.910 align:middle line:84% that that was because the person who owned our place before 00:37:07.910 --> 00:37:11.550 align:middle line:90% was the CTO of DraftKings. 00:37:11.550 --> 00:37:16.070 align:middle line:84% So now, we get a lot of free fruits in the mail because 00:37:16.070 --> 00:37:17.130 align:middle line:90% of gambling. 00:37:17.130 --> 00:37:18.390 align:middle line:90% So that's good. 00:37:18.390 --> 00:37:20.870 align:middle line:84% But let's talk about an application 00:37:20.870 --> 00:37:25.470 align:middle line:84% of the weak law of large numbers to gambling, or maybe a caution 00:37:25.470 --> 00:37:28.450 align:middle line:84% about why it's not a great idea to gamble too much, 00:37:28.450 --> 00:37:31.390 align:middle line:90% especially in Vegas. 00:37:31.390 --> 00:37:35.100 align:middle line:84% So there's an important example, which is called gambler's ruin. 00:37:35.100 --> 00:37:37.750 align:middle line:90% 00:37:37.750 --> 00:37:41.550 align:middle line:84% So the way that it's colloquially phrased 00:37:41.550 --> 00:37:59.030 align:middle line:84% is that playing a game with negative expected value, 00:37:59.030 --> 00:38:00.520 align:middle line:90% you'll eventually go broke. 00:38:00.520 --> 00:38:09.870 align:middle line:90% 00:38:09.870 --> 00:38:12.670 align:middle line:90% So this makes sense. 00:38:12.670 --> 00:38:15.350 align:middle line:84% When you gamble in Vegas, definitely all the games, 00:38:15.350 --> 00:38:19.270 align:middle line:84% aside from maybe blackjack, if you're counting cards, 00:38:19.270 --> 00:38:23.415 align:middle line:84% have this property that they have negative expected value. 00:38:23.415 --> 00:38:25.790 align:middle line:84% And intuitively, it makes sense that if you have negative 00:38:25.790 --> 00:38:28.670 align:middle line:84% expected value, on average, you're giving up some money 00:38:28.670 --> 00:38:30.670 align:middle line:90% every time you play. 00:38:30.670 --> 00:38:32.330 align:middle line:90% But maybe you get really lucky. 00:38:32.330 --> 00:38:36.323 align:middle line:84% Maybe there's some probability that you just keep on winning. 00:38:36.323 --> 00:38:37.990 align:middle line:84% Well, gambler's ruin is going to tell us 00:38:37.990 --> 00:38:42.270 align:middle line:84% that that's not the case, that the probability that you 00:38:42.270 --> 00:38:46.590 align:middle line:84% go broke is going to converge to 1 as soon as the expected value 00:38:46.590 --> 00:38:48.310 align:middle line:90% is negative. 00:38:48.310 --> 00:38:51.570 align:middle line:84% So let's do this, for example, with a concrete thing. 00:38:51.570 --> 00:38:56.510 align:middle line:84% So if you ignore multiplayer betting games, 00:38:56.510 --> 00:39:01.830 align:middle line:84% like blackjack, if you just stick to the standard ones, 00:39:01.830 --> 00:39:05.470 align:middle line:84% all of the games in Vegas have negative expected value. 00:39:05.470 --> 00:39:08.710 align:middle line:84% But the one that has the least negative expected value 00:39:08.710 --> 00:39:09.360 align:middle line:90% is craps. 00:39:09.360 --> 00:39:12.990 align:middle line:90% 00:39:12.990 --> 00:39:16.350 align:middle line:90% Has anyone played craps before? 00:39:16.350 --> 00:39:17.270 align:middle line:90% No? 00:39:17.270 --> 00:39:19.250 align:middle line:84% OK, I'm talking to the wrong audience. 00:39:19.250 --> 00:39:20.950 align:middle line:90% That's good. 00:39:20.950 --> 00:39:24.550 align:middle line:84% So craps is one where you roll dice. 00:39:24.550 --> 00:39:29.190 align:middle line:84% And then depending on whether it lands on black or red, 00:39:29.190 --> 00:39:31.670 align:middle line:84% you make different kinds of wagers. 00:39:31.670 --> 00:39:35.190 align:middle line:84% And there's basically a bet you can make that's very close 00:39:35.190 --> 00:39:36.950 align:middle line:90% to 50/50. 00:39:36.950 --> 00:39:39.910 align:middle line:84% So you have to get to that point in the game first. 00:39:39.910 --> 00:39:44.230 align:middle line:84% But the house edge, if you play that strategy in craps, 00:39:44.230 --> 00:39:48.990 align:middle line:90% is just 0.0141. 00:39:48.990 --> 00:39:52.070 align:middle line:84% So we can think about a simplified version 00:39:52.070 --> 00:39:57.910 align:middle line:84% of making that bet in craps, that with some probability 00:39:57.910 --> 00:39:59.510 align:middle line:90% you're going to win $1. 00:39:59.510 --> 00:40:02.310 align:middle line:84% And what is that probability going to be? 00:40:02.310 --> 00:40:11.910 align:middle line:84% Well, it'll be 0.4929 because you want that the probability 00:40:11.910 --> 00:40:13.950 align:middle line:90% that you win-- 00:40:13.950 --> 00:40:16.590 align:middle line:84% the probability you lose minus the probability you win 00:40:16.590 --> 00:40:20.330 align:middle line:84% is equal to this house edge here. 00:40:20.330 --> 00:40:25.550 align:middle line:84% And then otherwise, you're going to lose $1. 00:40:25.550 --> 00:40:32.870 align:middle line:84% And so what I claim is now that, by weak law of large numbers, 00:40:32.870 --> 00:40:35.430 align:middle line:84% well, you can look at Sn, which is just 00:40:35.430 --> 00:40:40.030 align:middle line:84% your cumulative winnings-- so you can add up in each game 00:40:40.030 --> 00:40:42.870 align:middle line:84% whether you win $1 or you lose $1. 00:40:42.870 --> 00:40:46.830 align:middle line:84% Now, the point is that that average is negative. 00:40:46.830 --> 00:40:49.990 align:middle line:84% So the probability that you're epsilon away from your average 00:40:49.990 --> 00:40:53.310 align:middle line:84% is actually going to 0 as you play more and more games. 00:40:53.310 --> 00:40:55.940 align:middle line:84% So on average, you really will be losing 00:40:55.940 --> 00:40:57.800 align:middle line:90% some amount of money per step. 00:40:57.800 --> 00:41:00.100 align:middle line:84% And you will not get that lucky with probability 00:41:00.100 --> 00:41:01.860 align:middle line:90% converging to 0. 00:41:01.860 --> 00:41:04.460 align:middle line:84% So by the weak law of large numbers, 00:41:04.460 --> 00:41:09.900 align:middle line:84% except with vanishing probability-- 00:41:09.900 --> 00:41:14.140 align:middle line:84% something that gets swallowed up by the limit-- 00:41:14.140 --> 00:41:24.420 align:middle line:84% I claim the winning per round will go negative, 00:41:24.420 --> 00:41:27.800 align:middle line:84% which, of course, since it's an average over your rounds, 00:41:27.800 --> 00:41:31.700 align:middle line:84% if you play infinitely long, you will go broke. 00:41:31.700 --> 00:41:35.420 align:middle line:84% So this is called gambler's ruin. 00:41:35.420 --> 00:41:38.598 align:middle line:84% And I want to give one more motivating example before we 00:41:38.598 --> 00:41:40.140 align:middle line:84% take a step back and I tell you about 00:41:40.140 --> 00:41:42.340 align:middle line:90% another probabilistic tool. 00:41:42.340 --> 00:41:44.860 align:middle line:84% So for us, so far, we've been talking 00:41:44.860 --> 00:41:48.380 align:middle line:84% about tail bounds as a way to get a handle 00:41:48.380 --> 00:41:53.060 align:middle line:84% on how likely or unlikely it is for a random variable 00:41:53.060 --> 00:41:56.300 align:middle line:84% to be far away from their expectation. 00:41:56.300 --> 00:41:58.820 align:middle line:84% Now, it turns out that, really, the reason 00:41:58.820 --> 00:42:04.740 align:middle line:84% we care about these tail bounds isn't just for their own sake. 00:42:04.740 --> 00:42:08.020 align:middle line:84% But it really comes up in all kinds of applications 00:42:08.020 --> 00:42:09.860 align:middle line:90% in discrete math. 00:42:09.860 --> 00:42:12.700 align:middle line:84% So we'll see probability throughout some 00:42:12.700 --> 00:42:15.340 align:middle line:90% of the later units in class. 00:42:15.340 --> 00:42:17.900 align:middle line:84% But I want to give you a little bit of a hint about why 00:42:17.900 --> 00:42:20.660 align:middle line:84% we care about tail bounds, especially when it 00:42:20.660 --> 00:42:22.230 align:middle line:90% comes to algorithm design. 00:42:22.230 --> 00:42:24.892 align:middle line:90% 00:42:24.892 --> 00:42:28.400 align:middle line:84% So let's talk about an algorithmic application. 00:42:28.400 --> 00:42:31.380 align:middle line:90% 00:42:31.380 --> 00:42:35.060 align:middle line:84% And this is a real application, something 00:42:35.060 --> 00:42:36.550 align:middle line:90% that gets used all the time. 00:42:36.550 --> 00:42:39.700 align:middle line:90% 00:42:39.700 --> 00:42:46.880 align:middle line:84% So let's say that we're given a randomized algorithm. 00:42:46.880 --> 00:42:54.460 align:middle line:90% 00:42:54.460 --> 00:42:58.100 align:middle line:90% And let's call it Aq. 00:42:58.100 --> 00:43:01.820 align:middle line:84% So A is the algorithm. q is the input. 00:43:01.820 --> 00:43:06.220 align:middle line:84% And let's say this algorithm, this randomized algorithm, 00:43:06.220 --> 00:43:09.780 align:middle line:84% has the property that when we run on A on q, well, 00:43:09.780 --> 00:43:11.720 align:middle line:90% its answer isn't deterministic. 00:43:11.720 --> 00:43:13.980 align:middle line:90% It's a randomized algorithm. 00:43:13.980 --> 00:43:17.380 align:middle line:84% And let's assume that it gets the answer right more 00:43:17.380 --> 00:43:19.420 align:middle line:90% often than it gets it wrong. 00:43:19.420 --> 00:43:24.060 align:middle line:84% So let's say it outputs the correct answer 00:43:24.060 --> 00:43:26.260 align:middle line:90% with some probability. 00:43:26.260 --> 00:43:28.190 align:middle line:90% Let's just say 2/3. 00:43:28.190 --> 00:43:31.700 align:middle line:90% 00:43:31.700 --> 00:43:36.610 align:middle line:84% And otherwise, it's going to output the incorrect answer. 00:43:36.610 --> 00:43:43.940 align:middle line:90% 00:43:43.940 --> 00:43:45.460 align:middle line:84% And the basic question we can ask 00:43:45.460 --> 00:43:49.180 align:middle line:84% is how many times we have to repeat this algorithm in order 00:43:49.180 --> 00:43:52.940 align:middle line:84% to get a good to boost its success probability. 00:43:52.940 --> 00:43:54.580 align:middle line:90% So one way to think about it-- 00:43:54.580 --> 00:43:57.220 align:middle line:84% our next unit after we do tail bounds-- 00:43:57.220 --> 00:44:01.780 align:middle line:84% is I'll be teaching you about modular arithmetic. 00:44:01.780 --> 00:44:04.900 align:middle line:84% And the basic thing we want to do there is-- imagine I'm 00:44:04.900 --> 00:44:06.940 align:middle line:90% given some giant number. 00:44:06.940 --> 00:44:09.780 align:middle line:84% A lot of properties of the factorization of this number 00:44:09.780 --> 00:44:15.580 align:middle line:84% are going to affect how it works when we do modular arithmetic. 00:44:15.580 --> 00:44:18.280 align:middle line:84% And later on, much later in the class, 00:44:18.280 --> 00:44:19.900 align:middle line:84% we're going to talk about applications 00:44:19.900 --> 00:44:23.260 align:middle line:84% of modular arithmetic to things like cryptography-- 00:44:23.260 --> 00:44:27.060 align:middle line:84% so communicating securely with a public encryption scheme. 00:44:27.060 --> 00:44:30.820 align:middle line:84% These are things that get used all the time in e-commerce. 00:44:30.820 --> 00:44:34.820 align:middle line:84% But when I'm creating a scheme for communicating securely, 00:44:34.820 --> 00:44:38.540 align:middle line:84% I have to start off with some number that's 00:44:38.540 --> 00:44:41.140 align:middle line:90% the product of just two primes. 00:44:41.140 --> 00:44:44.940 align:middle line:84% So what I can do is I can try and generate a random integer 00:44:44.940 --> 00:44:46.820 align:middle line:90% and hope that it's prime. 00:44:46.820 --> 00:44:48.840 align:middle line:84% And that turns out to work pretty well. 00:44:48.840 --> 00:44:51.540 align:middle line:84% You're reasonably likely to get a prime. 00:44:51.540 --> 00:44:55.620 align:middle line:84% But the trouble is that our best algorithms for determining 00:44:55.620 --> 00:45:00.460 align:middle line:84% whether or not a given integer is prime, they're randomized. 00:45:00.460 --> 00:45:04.020 align:middle line:84% So maybe I'm running my cryptographic scheme. 00:45:04.020 --> 00:45:06.460 align:middle line:90% And I have some integer q. 00:45:06.460 --> 00:45:08.260 align:middle line:84% And I want to figure out if it's prime 00:45:08.260 --> 00:45:10.940 align:middle line:84% so that I can safely use it in the context 00:45:10.940 --> 00:45:13.260 align:middle line:90% of my secure communication. 00:45:13.260 --> 00:45:16.680 align:middle line:84% So you have an algorithm that, with 2/3 probability, 00:45:16.680 --> 00:45:19.180 align:middle line:84% tells you the right answer and otherwise tells you the wrong 00:45:19.180 --> 00:45:20.220 align:middle line:90% answer. 00:45:20.220 --> 00:45:24.100 align:middle line:84% But if you are wrong, there's a huge penalty to pay. 00:45:24.100 --> 00:45:26.940 align:middle line:84% If you're wrong and the thing wasn't prime 00:45:26.940 --> 00:45:29.380 align:middle line:84% and you thought it was, then maybe you've 00:45:29.380 --> 00:45:31.940 align:middle line:84% just leaked all of the sensitive information, 00:45:31.940 --> 00:45:35.340 align:middle line:84% like your bank account, your social security number. 00:45:35.340 --> 00:45:38.620 align:middle line:84% So maybe this failure probability of 1/3 is just not 00:45:38.620 --> 00:45:40.030 align:middle line:90% good enough. 00:45:40.030 --> 00:45:41.780 align:middle line:84% What kind of failure probability would you 00:45:41.780 --> 00:45:44.390 align:middle line:84% be OK with your social security number leaking? 00:45:44.390 --> 00:45:48.180 align:middle line:90% 00:45:48.180 --> 00:45:50.660 align:middle line:90% Just give me a number. 00:45:50.660 --> 00:45:51.720 align:middle line:90% 10 to the minus 30. 00:45:51.720 --> 00:45:53.140 align:middle line:90% OK. 00:45:53.140 --> 00:45:54.800 align:middle line:84% I was going to say 10 to the minus 10, 00:45:54.800 --> 00:45:56.780 align:middle line:90% but that's even better. 00:45:56.780 --> 00:45:59.900 align:middle line:84% You're even more paranoid than I am, which is a good thing. 00:45:59.900 --> 00:46:05.160 align:middle line:84% So let's say I want to get a really tiny failure probability, 00:46:05.160 --> 00:46:07.980 align:middle line:90% like 10 to the minus 10. 00:46:07.980 --> 00:46:19.140 align:middle line:84% So how many times should I repeat this algorithm 00:46:19.140 --> 00:46:31.400 align:middle line:84% to make its failure probability less than 10 to the minus 10 00:46:31.400 --> 00:46:35.780 align:middle line:84% or maybe even 10 to the negative 30? 00:46:35.780 --> 00:46:38.380 align:middle line:84% So if you think about it, this is really 00:46:38.380 --> 00:46:41.020 align:middle line:84% the setting of the weak law of large numbers. 00:46:41.020 --> 00:46:42.740 align:middle line:90% It's the same kind of thing. 00:46:42.740 --> 00:46:45.580 align:middle line:84% Each of my runs of this random variable 00:46:45.580 --> 00:46:49.700 align:middle line:84% is a random variable xi that tells me whether or not 00:46:49.700 --> 00:46:51.420 align:middle line:90% q is prime. 00:46:51.420 --> 00:46:55.520 align:middle line:84% And what I'm guaranteed is that, as I take the limit, 00:46:55.520 --> 00:46:58.460 align:middle line:84% as the number of repetitions goes to infinity, 00:46:58.460 --> 00:47:04.940 align:middle line:84% the average of the number of outputs that say that it's prime 00:47:04.940 --> 00:47:08.560 align:middle line:84% is going to converge to 2/3, if it really is prime. 00:47:08.560 --> 00:47:11.060 align:middle line:84% And the number that are going to say it's not prime is going 00:47:11.060 --> 00:47:14.420 align:middle line:90% to converge to 2/3 if it isn't. 00:47:14.420 --> 00:47:16.840 align:middle line:84% So this is the setting of the weak law of large numbers. 00:47:16.840 --> 00:47:20.980 align:middle line:84% I'm just voting with runs of my algorithm. 00:47:20.980 --> 00:47:23.260 align:middle line:84% And we've already seen some quantitative bounds 00:47:23.260 --> 00:47:24.780 align:middle line:84% from the weak law of large numbers 00:47:24.780 --> 00:47:29.730 align:middle line:84% about how large n needs to be in order to get some valid epsilon. 00:47:29.730 --> 00:47:32.580 align:middle line:90% 00:47:32.580 --> 00:47:34.860 align:middle line:84% And if we think about it, if we just 00:47:34.860 --> 00:47:37.160 align:middle line:84% think about applying the weak law of large numbers, 00:47:37.160 --> 00:47:41.220 align:middle line:84% we're actually going to get very pessimistic bounds even here. 00:47:41.220 --> 00:47:44.800 align:middle line:84% So let's think about what the weak law of large numbers 00:47:44.800 --> 00:47:46.320 align:middle line:90% and Chebyshev would tell us. 00:47:46.320 --> 00:47:58.060 align:middle line:90% 00:47:58.060 --> 00:48:01.900 align:middle line:84% So the weak law of large numbers slash our Chebyshev proof that 00:48:01.900 --> 00:48:04.940 align:middle line:90% gave us a quantitative bound-- 00:48:04.940 --> 00:48:07.840 align:middle line:84% well, it's going to tell us that the right-hand side, 00:48:07.840 --> 00:48:14.220 align:middle line:84% the probability of failure is bounded by the variance of x 00:48:14.220 --> 00:48:17.780 align:middle line:90% n over epsilon squared. 00:48:17.780 --> 00:48:26.940 align:middle line:84% And in our case, we can take x to be just whether or not 00:48:26.940 --> 00:48:29.700 align:middle line:90% the output of our random-- 00:48:29.700 --> 00:48:34.620 align:middle line:84% of our randomized algorithm is correct or not. 00:48:34.620 --> 00:48:40.300 align:middle line:84% And you can compute that the variance of x is going to be 2/3 00:48:40.300 --> 00:48:42.810 align:middle line:90% minus 4/9. 00:48:42.810 --> 00:48:44.750 align:middle line:90% So it's some constant. 00:48:44.750 --> 00:48:46.850 align:middle line:90% That's a reasonable constant. 00:48:46.850 --> 00:48:52.290 align:middle line:84% And what we can do is we can set epsilon to be something like 00:48:52.290 --> 00:48:53.770 align:middle line:90% 1/9. 00:48:53.770 --> 00:48:57.530 align:middle line:84% So x is supposed to tell us whether or not 00:48:57.530 --> 00:48:59.170 align:middle line:90% the thing is prime or not. 00:48:59.170 --> 00:49:02.290 align:middle line:84% We only have to set our tolerance really to split 00:49:02.290 --> 00:49:04.790 align:middle line:84% the difference between 2/3 and 1/3. 00:49:04.790 --> 00:49:07.350 align:middle line:84% That's not where any of the action is happening. 00:49:07.350 --> 00:49:09.330 align:middle line:90% These are all just constants. 00:49:09.330 --> 00:49:12.930 align:middle line:84% But what's going on is that the probability 00:49:12.930 --> 00:49:18.770 align:middle line:84% that our average is off from the truth, 00:49:18.770 --> 00:49:23.570 align:middle line:84% we can bound it by something like 18 over n. 00:49:23.570 --> 00:49:25.370 align:middle line:84% Now, none of the other computations 00:49:25.370 --> 00:49:26.470 align:middle line:90% really matter too much. 00:49:26.470 --> 00:49:27.710 align:middle line:90% You can compute the variance. 00:49:27.710 --> 00:49:30.330 align:middle line:84% You can compute what epsilon makes sense. 00:49:30.330 --> 00:49:32.210 align:middle line:84% But the crucial thing is really the fact 00:49:32.210 --> 00:49:35.450 align:middle line:84% that the n shows up in the denominator. 00:49:35.450 --> 00:49:38.850 align:middle line:84% So if I want my failure probability 00:49:38.850 --> 00:49:43.530 align:middle line:84% to be absolutely tiny, what kind of n would I have to take? 00:49:43.530 --> 00:49:47.770 align:middle line:84% I'd have to take it to be like 10 to the 10 or 10 to the 30. 00:49:47.770 --> 00:49:49.970 align:middle line:90% That would be ridiculous. 00:49:49.970 --> 00:49:52.450 align:middle line:84% So I want to make sure that I have vanishing probability 00:49:52.450 --> 00:49:56.690 align:middle line:84% of using something that violates security in my crypto scheme. 00:49:56.690 --> 00:49:59.770 align:middle line:84% And I just have to repeat this so many times 00:49:59.770 --> 00:50:02.930 align:middle line:84% that I can't really do a transaction online 00:50:02.930 --> 00:50:06.234 align:middle line:84% because I would just clog up the computer. 00:50:06.234 --> 00:50:09.630 align:middle line:84% So this takes me to the last main topic. 00:50:09.630 --> 00:50:11.770 align:middle line:84% This is something which I want to introduce now. 00:50:11.770 --> 00:50:15.210 align:middle line:84% You're going to start to cover it in recitation. 00:50:15.210 --> 00:50:20.330 align:middle line:84% But this tail bound is not at all obvious how to prove it. 00:50:20.330 --> 00:50:23.370 align:middle line:84% And it'll be basically the full lecture on Thursday. 00:50:23.370 --> 00:50:26.770 align:middle line:84% But this was really just the motivation for it. 00:50:26.770 --> 00:50:30.230 align:middle line:84% So you see that, like I told you, 00:50:30.230 --> 00:50:35.370 align:middle line:84% the theme for today's lecture was the more information you 00:50:35.370 --> 00:50:39.530 align:middle line:84% tell me about the random variables, 00:50:39.530 --> 00:50:42.450 align:middle line:84% the better I get for my tail bounds. 00:50:42.450 --> 00:50:46.770 align:middle line:84% So let's take a minute to think about the proof of this weak law 00:50:46.770 --> 00:50:49.010 align:middle line:90% of large numbers. 00:50:49.010 --> 00:50:52.130 align:middle line:84% So the setting for this was that I assumed that the xi's 00:50:52.130 --> 00:50:56.610 align:middle line:90% are independent draws from x. 00:50:56.610 --> 00:51:02.310 align:middle line:84% But I claim my proof didn't really use full independence. 00:51:02.310 --> 00:51:04.850 align:middle line:90% 00:51:04.850 --> 00:51:11.730 align:middle line:84% I claim all it used was that they're pairwise independent, 00:51:11.730 --> 00:51:14.770 align:middle line:84% that if you look at any pair of them, they're independent. 00:51:14.770 --> 00:51:18.010 align:middle line:84% But I didn't use the fact that any triple or any quadruple 00:51:18.010 --> 00:51:20.410 align:middle line:90% of them are independent. 00:51:20.410 --> 00:51:23.330 align:middle line:84% So this requires a little bit of thinking. 00:51:23.330 --> 00:51:28.330 align:middle line:84% So where in my proof of the weak, law of large numbers 00:51:28.330 --> 00:51:30.930 align:middle line:90% did I actually use independence? 00:51:30.930 --> 00:51:35.330 align:middle line:84% It was only in this step right here. 00:51:35.330 --> 00:51:39.210 align:middle line:84% And I didn't actually need that all of these random variables 00:51:39.210 --> 00:51:40.510 align:middle line:90% are mutually independent. 00:51:40.510 --> 00:51:43.690 align:middle line:84% I just needed that they're pairwise independent. 00:51:43.690 --> 00:51:46.650 align:middle line:84% That's exactly what got me the quantitative bound. 00:51:46.650 --> 00:51:49.050 align:middle line:84% But that's the weakness in applying Chebyshev, 00:51:49.050 --> 00:51:53.210 align:middle line:84% is that I'm not using as much information as I could hope to. 00:51:53.210 --> 00:51:55.250 align:middle line:90% Does that make sense? 00:51:55.250 --> 00:51:57.930 align:middle line:84% So that's what leads me to the next bound, which 00:51:57.930 --> 00:52:00.690 align:middle line:84% is called the Chernoff bound, which is not just 00:52:00.690 --> 00:52:02.550 align:middle line:84% going to use pairwise independence, 00:52:02.550 --> 00:52:05.530 align:middle line:84% but it's going to use mutual independence. 00:52:05.530 --> 00:52:08.310 align:middle line:84% So let's state the Chernoff bound. 00:52:08.310 --> 00:52:12.650 align:middle line:84% This was actually proven at MIT, presumably 00:52:12.650 --> 00:52:15.290 align:middle line:84% in this building even, which is cool. 00:52:15.290 --> 00:52:19.250 align:middle line:84% So this was proven by our former colleague, Herman Chernoff. 00:52:19.250 --> 00:52:23.570 align:middle line:84% And let's assume that x1 up to xn, 00:52:23.570 --> 00:52:26.470 align:middle line:84% let them be mutually independent. 00:52:26.470 --> 00:52:30.810 align:middle line:84% So this is a much stronger notion of independence, 00:52:30.810 --> 00:52:32.570 align:middle line:84% because I need all subsets of them 00:52:32.570 --> 00:52:35.490 align:middle line:84% to be independent from all other subsets. 00:52:35.490 --> 00:52:38.870 align:middle line:84% So let's say they're mutually independent random variables. 00:52:38.870 --> 00:52:43.410 align:middle line:90% 00:52:43.410 --> 00:52:48.850 align:middle line:84% And let's just say that xi is equal to 1 or 0 00:52:48.850 --> 00:52:51.330 align:middle line:90% with probability pi. 00:52:51.330 --> 00:52:54.040 align:middle line:90% And otherwise, it's equal to 0. 00:52:54.040 --> 00:52:56.650 align:middle line:90% 00:52:56.650 --> 00:52:59.450 align:middle line:84% Now, I'm going to do the same type of thing I did before. 00:52:59.450 --> 00:53:02.530 align:middle line:84% I'm going to look at some kind of average or sum 00:53:02.530 --> 00:53:03.710 align:middle line:90% of these random variables. 00:53:03.710 --> 00:53:06.010 align:middle line:84% So let me look at capital x, which 00:53:06.010 --> 00:53:10.290 align:middle line:84% is the sum from i equals 1 to little n of xi. 00:53:10.290 --> 00:53:14.450 align:middle line:84% And let me look at the expectation of x. 00:53:14.450 --> 00:53:16.930 align:middle line:90% Let me call that quantity mu. 00:53:16.930 --> 00:53:20.290 align:middle line:84% But remember, that's the same thing as just the sum from i 00:53:20.290 --> 00:53:23.130 align:middle line:90% equals 1 to n of the pi's. 00:53:23.130 --> 00:53:25.010 align:middle line:90% So this is all just setup. 00:53:25.010 --> 00:53:28.770 align:middle line:84% I have mutually independent random variables, the x sub i's. 00:53:28.770 --> 00:53:31.610 align:middle line:84% In fact, they're Bernoulli random variables because they 00:53:31.610 --> 00:53:34.090 align:middle line:90% just take on the value 1 or 0. 00:53:34.090 --> 00:53:36.690 align:middle line:84% And I'm telling you that the probability that they're 1 00:53:36.690 --> 00:53:37.930 align:middle line:90% is pi. 00:53:37.930 --> 00:53:39.810 align:middle line:84% So this is already a bit of a departure 00:53:39.810 --> 00:53:42.130 align:middle line:84% from the weak law of large numbers 00:53:42.130 --> 00:53:44.110 align:middle line:84% because, in the weak law of large numbers, 00:53:44.110 --> 00:53:47.610 align:middle line:84% I assumed that all of these xi's were identical. 00:53:47.610 --> 00:53:52.210 align:middle line:84% They were all draws of this random variable x. 00:53:52.210 --> 00:53:54.210 align:middle line:84% Here, I'm actually allowing these to be 00:53:54.210 --> 00:53:55.670 align:middle line:90% different random variables. 00:53:55.670 --> 00:53:58.650 align:middle line:84% So that's even one extra layer of power. 00:53:58.650 --> 00:54:03.450 align:middle line:84% But the real power just comes from the right-hand side. 00:54:03.450 --> 00:54:07.370 align:middle line:84% So let me write down one form of the Chernoff bound. 00:54:07.370 --> 00:54:10.090 align:middle line:84% So the probability that x is very far 00:54:10.090 --> 00:54:14.970 align:middle line:84% away from its average mu by some delta 00:54:14.970 --> 00:54:23.280 align:middle line:84% is at most 2 times e to the minus mu delta squared over 3. 00:54:23.280 --> 00:54:27.330 align:middle line:90% 00:54:27.330 --> 00:54:32.150 align:middle line:90% I think I need the average here. 00:54:32.150 --> 00:54:33.490 align:middle line:90% OK. 00:54:33.490 --> 00:54:35.290 align:middle line:90% No. 00:54:35.290 --> 00:54:36.450 align:middle line:90% OK. 00:54:36.450 --> 00:54:38.890 align:middle line:84% All right, so this is the conclusion 00:54:38.890 --> 00:54:40.370 align:middle line:90% for the Chernoff bound. 00:54:40.370 --> 00:54:44.170 align:middle line:84% So the important thing is really that what 00:54:44.170 --> 00:54:47.530 align:middle line:84% I had on the right-hand side when I had Markov's bound 00:54:47.530 --> 00:54:48.870 align:middle line:90% was a kind of weak bound. 00:54:48.870 --> 00:54:52.370 align:middle line:84% It looked like 1 over C. So as you increase C, you just 00:54:52.370 --> 00:54:56.530 align:middle line:84% improve the right-hand side by the same factor. 00:54:56.530 --> 00:54:58.890 align:middle line:84% When I had Chebyshev, I had 1 over C 00:54:58.890 --> 00:55:00.890 align:middle line:90% squared on the other side. 00:55:00.890 --> 00:55:03.770 align:middle line:84% So the probability that I'm C away 00:55:03.770 --> 00:55:07.130 align:middle line:84% was degrading still at an inverse poly rate. 00:55:07.130 --> 00:55:11.290 align:middle line:84% The crucial thing is that, here, I have an exponential rate. 00:55:11.290 --> 00:55:12.930 align:middle line:84% And this is what mutual independence 00:55:12.930 --> 00:55:15.450 align:middle line:84% buys me, is that the probability that you're 00:55:15.450 --> 00:55:17.090 align:middle line:84% far away from the average is just 00:55:17.090 --> 00:55:20.570 align:middle line:84% plummeting to 0 once you have this extra information 00:55:20.570 --> 00:55:22.530 align:middle line:90% about independence. 00:55:22.530 --> 00:55:27.370 align:middle line:84% So the Chernoff bound, there are many different forms for it. 00:55:27.370 --> 00:55:30.170 align:middle line:84% We're going to prove this next time. 00:55:30.170 --> 00:55:32.440 align:middle line:84% But it turns out that, if we were 00:55:32.440 --> 00:55:36.560 align:middle line:84% to use Chernoff in our numerical example 00:55:36.560 --> 00:55:40.440 align:middle line:84% about testing out primes, well, we would only 00:55:40.440 --> 00:55:43.840 align:middle line:84% have to take n, our number of repetitions, 00:55:43.840 --> 00:55:46.480 align:middle line:84% to behave like some constant times 00:55:46.480 --> 00:55:50.200 align:middle line:84% the natural log of whatever we want 00:55:50.200 --> 00:55:52.480 align:middle line:84% to be in our failure probability over here 00:55:52.480 --> 00:55:54.040 align:middle line:90% on the right-hand side. 00:55:54.040 --> 00:55:55.600 align:middle line:84% So if we had 10 to the 10, really, 00:55:55.600 --> 00:55:58.280 align:middle line:84% the number of repetitions would be much tamer. 00:55:58.280 --> 00:56:01.280 align:middle line:84% It would just be the natural log of 10 to the 10. 00:56:01.280 --> 00:56:04.060 align:middle line:84% So even if I put a 10 or a 30 in the exponent, 00:56:04.060 --> 00:56:08.000 align:middle line:84% it's not such a giant overhead to actually drive the failure 00:56:08.000 --> 00:56:10.800 align:middle line:90% probability basically to 0. 00:56:10.800 --> 00:56:13.100 align:middle line:84% So that's what we'll do next time. 00:56:13.100 --> 00:56:15.120 align:middle line:90% We'll do the Chernoff bound. 00:56:15.120 --> 00:56:18.680 align:middle line:84% But I want to cover one last topic in probability 00:56:18.680 --> 00:56:21.600 align:middle line:84% before we get to the Chernoff bound on Thursday. 00:56:21.600 --> 00:56:23.080 align:middle line:84% This will be something that's going 00:56:23.080 --> 00:56:27.240 align:middle line:84% to help you a lot for this week's assignment, which 00:56:27.240 --> 00:56:30.080 align:middle line:84% is a writing problem that involves 00:56:30.080 --> 00:56:32.040 align:middle line:90% using the probabilistic method. 00:56:32.040 --> 00:56:34.840 align:middle line:84% So it's something we've talked a little bit about on some 00:56:34.840 --> 00:56:35.780 align:middle line:90% of the P sets. 00:56:35.780 --> 00:56:39.040 align:middle line:84% But I just want to be explicit and show you some neat examples. 00:56:39.040 --> 00:56:41.160 align:middle line:84% So that will be our last topic for today, 00:56:41.160 --> 00:56:42.470 align:middle line:90% is the probabilistic method. 00:56:42.470 --> 00:56:48.840 align:middle line:90% 00:56:48.840 --> 00:56:51.480 align:middle line:84% So you should think of this as like one more 00:56:51.480 --> 00:56:56.320 align:middle line:84% important application of probability tools. 00:56:56.320 --> 00:57:00.320 align:middle line:84% So far, we've talked about probability for its own sake, 00:57:00.320 --> 00:57:03.160 align:middle line:84% like looking at the deviation of random variables. 00:57:03.160 --> 00:57:04.920 align:middle line:84% We talked about probability for the sake 00:57:04.920 --> 00:57:07.040 align:middle line:84% of analyzing the failure probability 00:57:07.040 --> 00:57:09.480 align:middle line:90% of randomized algorithms. 00:57:09.480 --> 00:57:12.680 align:middle line:84% And we'll also be interested in using probability 00:57:12.680 --> 00:57:15.280 align:middle line:90% in the context of combinatorics. 00:57:15.280 --> 00:57:18.240 align:middle line:84% It turns out that, in many applications, 00:57:18.240 --> 00:57:21.760 align:middle line:84% there are exotic types of combinatorial objects 00:57:21.760 --> 00:57:25.480 align:middle line:84% that the only way we know how to reason about why they exist 00:57:25.480 --> 00:57:27.760 align:middle line:90% is by using randomness. 00:57:27.760 --> 00:57:30.760 align:middle line:84% So this leads to a very important paradigm 00:57:30.760 --> 00:57:36.040 align:middle line:84% in discrete math, which is called the probabilistic method. 00:57:36.040 --> 00:57:41.720 align:middle line:84% You can take entire classes on using sophisticated tools 00:57:41.720 --> 00:57:43.560 align:middle line:90% to do things like this. 00:57:43.560 --> 00:57:47.240 align:middle line:84% But let me give you the high level bumper sticker 00:57:47.240 --> 00:57:49.560 align:middle line:84% version first before we talk about 00:57:49.560 --> 00:57:52.160 align:middle line:90% some concrete applications. 00:57:52.160 --> 00:57:54.520 align:middle line:84% So what we'll be interested in is 00:57:54.520 --> 00:57:56.360 align:middle line:84% we'll be interested in constructing 00:57:56.360 --> 00:58:04.360 align:middle line:84% certain kinds of random objects or structures, which are exotic 00:58:04.360 --> 00:58:07.040 align:middle line:84% and have some pretty wild properties. 00:58:07.040 --> 00:58:09.800 align:middle line:84% And we're going to show that these things exist 00:58:09.800 --> 00:58:12.500 align:middle line:84% not by building them with our own hands, 00:58:12.500 --> 00:58:20.130 align:middle line:84% but by arguing that a random object succeeds. 00:58:20.130 --> 00:58:22.640 align:middle line:90% 00:58:22.640 --> 00:58:24.680 align:middle line:90% So that's the high level idea. 00:58:24.680 --> 00:58:26.560 align:middle line:84% And there are many instantiations 00:58:26.560 --> 00:58:27.900 align:middle line:90% of this probabilistic method. 00:58:27.900 --> 00:58:30.520 align:middle line:84% It depends on what kinds of structures I want to prove 00:58:30.520 --> 00:58:31.340 align:middle line:90% exist. 00:58:31.340 --> 00:58:33.360 align:middle line:84% We'll talk about some concrete applications. 00:58:33.360 --> 00:58:36.360 align:middle line:84% And we'll see how the probabilistic method helps. 00:58:36.360 --> 00:58:40.160 align:middle line:84% But we're going to apply this to another important branch 00:58:40.160 --> 00:58:45.140 align:middle line:84% of applied mathematics, which is called Ramsey theory. 00:58:45.140 --> 00:58:49.640 align:middle line:84% So I'll introduce both of these topics right now. 00:58:49.640 --> 00:58:52.400 align:middle line:90% So what's Ramsey theory? 00:58:52.400 --> 00:58:55.340 align:middle line:84% Well, let me give you a famous example of it. 00:58:55.340 --> 00:59:00.040 align:middle line:90% 00:59:00.040 --> 00:59:13.120 align:middle line:84% So in every party with at least six people, 00:59:13.120 --> 00:59:19.760 align:middle line:84% I claim that either we have the situation that there 00:59:19.760 --> 00:59:24.800 align:middle line:84% are three people all of whom are friends with each other, 00:59:24.800 --> 00:59:28.560 align:middle line:84% all pairs of these three people are friends with each other, 00:59:28.560 --> 00:59:34.880 align:middle line:84% or there are three people all of which 00:59:34.880 --> 00:59:40.720 align:middle line:84% are strangers, none of which knows the other person. 00:59:40.720 --> 00:59:45.880 align:middle line:84% So Ramsey theory was actually first invented or realized 00:59:45.880 --> 00:59:49.320 align:middle line:84% in Hungary, where there was a Hungarian sociologist who 00:59:49.320 --> 00:59:54.680 align:middle line:84% realized that he was studying social groups of kids. 00:59:54.680 --> 01:00:00.000 align:middle line:84% And he realized that in every group of, I think, 20 kids, 01:00:00.000 --> 01:00:02.800 align:middle line:84% there were always four, a group of four people 01:00:02.800 --> 01:00:04.880 align:middle line:84% he could find who all knew each other 01:00:04.880 --> 01:00:07.080 align:middle line:84% or who all didn't know each other. 01:00:07.080 --> 01:00:09.860 align:middle line:84% And you might think that this is a sociological phenomenon. 01:00:09.860 --> 01:00:12.920 align:middle line:84% Maybe it has to do with the way kids play with each other, 01:00:12.920 --> 01:00:15.640 align:middle line:84% that they form some of interconnected groups 01:00:15.640 --> 01:00:17.760 align:middle line:84% and they don't know other groups. 01:00:17.760 --> 01:00:20.500 align:middle line:84% But he consulted with his mathematician friends, 01:00:20.500 --> 01:00:22.520 align:middle line:90% including Erdos and others. 01:00:22.520 --> 01:00:25.140 align:middle line:84% And he realized that it wasn't a sociological phenomenon. 01:00:25.140 --> 01:00:26.880 align:middle line:84% It was just the property of graphs. 01:00:26.880 --> 01:00:30.760 align:middle line:84% So what this is really saying, like the same way that we can 01:00:30.760 --> 01:00:35.440 align:middle line:84% think about representing friendship relations with 01:00:35.440 --> 01:00:37.880 align:middle line:84% a graph, what we're really saying is that, 01:00:37.880 --> 01:00:43.520 align:middle line:84% on any graph with six nodes, no matter how I draw on the edges-- 01:00:43.520 --> 01:00:46.560 align:middle line:84% and the edges represent who's friends with who-- 01:00:46.560 --> 01:00:50.440 align:middle line:84% there must either be a triangle where all the edges are present. 01:00:50.440 --> 01:00:53.040 align:middle line:84% Or there's an anti triangle, where there's a set of three 01:00:53.040 --> 01:00:55.600 align:middle line:84% and none of the edges are present. 01:00:55.600 --> 01:00:58.780 align:middle line:84% So this is the basics of Ramsey theory, 01:00:58.780 --> 01:01:04.280 align:middle line:84% is that we want to show that, in large enough objects, 01:01:04.280 --> 01:01:06.400 align:middle line:84% certain kinds of ordered substructures 01:01:06.400 --> 01:01:09.780 align:middle line:84% must appear because they always have to happen. 01:01:09.780 --> 01:01:12.080 align:middle line:90% There's no way of avoiding it. 01:01:12.080 --> 01:01:15.000 align:middle line:84% So in fact, there's a pretty famous and interesting example 01:01:15.000 --> 01:01:17.520 align:middle line:90% from pop culture. 01:01:17.520 --> 01:01:19.380 align:middle line:84% This happened like when I was younger. 01:01:19.380 --> 01:01:24.520 align:middle line:84% But in the mid-'90s, there was a lot of interest in something 01:01:24.520 --> 01:01:25.740 align:middle line:90% called the Bible codes. 01:01:25.740 --> 01:01:28.160 align:middle line:84% Has anyone heard of the Bible codes? 01:01:28.160 --> 01:01:28.680 align:middle line:90% No? 01:01:28.680 --> 01:01:29.260 align:middle line:90% OK. 01:01:29.260 --> 01:01:29.960 align:middle line:90% Yeah? 01:01:29.960 --> 01:01:32.235 align:middle line:90% You want to explain it? 01:01:32.235 --> 01:01:34.860 align:middle line:84% STUDENT: I think it's like maybe you skip letters in the Bible. 01:01:34.860 --> 01:01:36.440 align:middle line:90% You make sentences or something. 01:01:36.440 --> 01:01:37.060 align:middle line:90% ANKUR MOITRA: That's right. 01:01:37.060 --> 01:01:39.200 align:middle line:84% So basically, what happens, if you take the Hebrew 01:01:39.200 --> 01:01:41.040 align:middle line:84% translation-- if you take the Hebrew 01:01:41.040 --> 01:01:43.560 align:middle line:84% version of the Torah, what you can do 01:01:43.560 --> 01:01:48.600 align:middle line:84% is you look at evenly spaced characters in them. 01:01:48.600 --> 01:01:51.020 align:middle line:84% And if you choose the step size in the right way, 01:01:51.020 --> 01:01:56.624 align:middle line:84% you can find all these instances in the Torah of examples 01:01:56.624 --> 01:01:59.120 align:middle line:84% of words that cross with other words, which 01:01:59.120 --> 01:02:01.040 align:middle line:90% have historical significance. 01:02:01.040 --> 01:02:03.520 align:middle line:84% There would be examples of famous rabbis 01:02:03.520 --> 01:02:07.140 align:middle line:84% names with the dates in which they were later assassinated, 01:02:07.140 --> 01:02:09.400 align:middle line:84% like after the Torah was written. 01:02:09.400 --> 01:02:12.080 align:middle line:84% So this was quite amazing because people let 01:02:12.080 --> 01:02:15.040 align:middle line:90% their imaginations run wild. 01:02:15.040 --> 01:02:16.800 align:middle line:84% They thought that maybe the Torah 01:02:16.800 --> 01:02:20.120 align:middle line:84% was predicting the future in arithmetic 01:02:20.120 --> 01:02:23.206 align:middle line:90% progressions in Hebrew. 01:02:23.206 --> 01:02:27.240 align:middle line:84% You could certainly write some bad movies about it. 01:02:27.240 --> 01:02:29.280 align:middle line:84% But it turned out that it was really 01:02:29.280 --> 01:02:32.300 align:middle line:90% just Ramsey theory in disguise. 01:02:32.300 --> 01:02:34.560 align:middle line:84% And the things that people found weren't statistically 01:02:34.560 --> 01:02:37.400 align:middle line:84% significant, that, in fact, you could 01:02:37.400 --> 01:02:39.360 align:middle line:84% find the same types of things in the Hebrew 01:02:39.360 --> 01:02:41.140 align:middle line:90% translation of War and Peace. 01:02:41.140 --> 01:02:42.958 align:middle line:84% So that was an interesting rejoinder to it. 01:02:42.958 --> 01:02:45.500 align:middle line:84% But that's the kind of thing we're going to do with it today. 01:02:45.500 --> 01:02:48.760 align:middle line:84% Let me tell you about one example of structure 01:02:48.760 --> 01:02:51.600 align:middle line:84% that has to do with arithmetic progressions. 01:02:51.600 --> 01:02:53.600 align:middle line:84% And we're going to use the probabilistic method 01:02:53.600 --> 01:02:58.320 align:middle line:84% to prove types of lower bounds on these quantities, 01:02:58.320 --> 01:03:00.920 align:middle line:84% or at least very strong lower bounds. 01:03:00.920 --> 01:03:03.120 align:middle line:84% So let me tell you the last topic for today 01:03:03.120 --> 01:03:06.110 align:middle line:84% is something called van der Waerden's theorem. 01:03:06.110 --> 01:03:10.320 align:middle line:90% 01:03:10.320 --> 01:03:14.240 align:middle line:84% So let me first give you a definition. 01:03:14.240 --> 01:03:23.778 align:middle line:84% So a k term, arithmetic progression, 01:03:23.778 --> 01:03:29.550 align:middle line:84% is just a very particular type of structured subset. 01:03:29.550 --> 01:03:36.230 align:middle line:84% So it's a set of integers of the following form. 01:03:36.230 --> 01:03:38.790 align:middle line:84% And I'll call it Q sub ab because I'm 01:03:38.790 --> 01:03:40.470 align:middle line:90% going to use it later. 01:03:40.470 --> 01:03:42.990 align:middle line:84% But it's just the set of integers that looks like 01:03:42.990 --> 01:03:53.350 align:middle line:84% a, a plus b, a plus 2b, B all the way up to a plus k minus 1 01:03:53.350 --> 01:03:55.270 align:middle line:90% b. 01:03:55.270 --> 01:03:57.590 align:middle line:84% So this is an arithmetic progression. 01:03:57.590 --> 01:04:00.150 align:middle line:84% You start off with some base number a. 01:04:00.150 --> 01:04:03.710 align:middle line:84% And then your steps are of size b. 01:04:03.710 --> 01:04:06.350 align:middle line:90% And then you go for k steps. 01:04:06.350 --> 01:04:07.890 align:middle line:84% So that's an arithmetic progression. 01:04:07.890 --> 01:04:10.390 align:middle line:84% This was the kind of thing that they were using in the Bible 01:04:10.390 --> 01:04:13.750 align:middle line:84% codes, because you would look at arithmetic progressions 01:04:13.750 --> 01:04:19.470 align:middle line:84% and concatenate the symbols you get at those positions together. 01:04:19.470 --> 01:04:22.610 align:middle line:84% And so now, I can tell you this van der Waerden's theorem. 01:04:22.610 --> 01:04:23.930 align:middle line:90% We're not going to prove it. 01:04:23.930 --> 01:04:26.070 align:middle line:90% The proof is in the notes. 01:04:26.070 --> 01:04:28.470 align:middle line:84% But we're going to prove lower bounds for it. 01:04:28.470 --> 01:04:31.030 align:middle line:84% So here's a really cool statement. 01:04:31.030 --> 01:04:34.910 align:middle line:84% So for every k you give me, what I 01:04:34.910 --> 01:04:45.390 align:middle line:84% claim is that there is an n such that any two coloring 01:04:45.390 --> 01:04:54.830 align:middle line:90% of the integers 1, 2 up to n-- 01:04:54.830 --> 01:04:58.190 align:middle line:84% so before I get to the theorem statement, what's going on 01:04:58.190 --> 01:05:03.190 align:middle line:84% is I want to find a k term arithmetic progression. 01:05:03.190 --> 01:05:08.310 align:middle line:84% And what I'm going to do is you're going to fix some n. 01:05:08.310 --> 01:05:11.190 align:middle line:84% And then once you look at the integers from 1, 2, all the way 01:05:11.190 --> 01:05:13.830 align:middle line:84% up to n, you're going to color these integers 01:05:13.830 --> 01:05:16.350 align:middle line:90% with either red or blue. 01:05:16.350 --> 01:05:18.830 align:middle line:84% And what I'm looking for is what's 01:05:18.830 --> 01:05:22.390 align:middle line:84% called a monochromatic arithmetic progression. 01:05:22.390 --> 01:05:25.070 align:middle line:84% So I want some arithmetic progression of this form, 01:05:25.070 --> 01:05:27.630 align:middle line:84% like Qab, but with the property that I 01:05:27.630 --> 01:05:31.110 align:middle line:84% look at the colors of the integers along the sequence, 01:05:31.110 --> 01:05:34.150 align:middle line:84% that they all should be the same color. 01:05:34.150 --> 01:05:44.270 align:middle line:84% So this necessarily contains a monochromatic k term 01:05:44.270 --> 01:05:47.830 align:middle line:90% AP, arithmetic progression. 01:05:47.830 --> 01:05:49.173 align:middle line:90% So this is the statement. 01:05:49.173 --> 01:05:51.090 align:middle line:84% Again, there are a lot of quantifiers in this. 01:05:51.090 --> 01:05:53.030 align:middle line:84% So let's make sure we understand it. 01:05:53.030 --> 01:05:56.390 align:middle line:84% I want to find some k term arithmetic progression, one 01:05:56.390 --> 01:05:57.990 align:middle line:90% that's monochromatic. 01:05:57.990 --> 01:06:03.270 align:middle line:84% For whatever k I choose, there are some large enough n 01:06:03.270 --> 01:06:06.230 align:middle line:84% so that no matter how you two-color 01:06:06.230 --> 01:06:10.910 align:middle line:84% the integers from 1, 2 up to n, there's no way to avoid it. 01:06:10.910 --> 01:06:13.750 align:middle line:84% Somewhere in that set is sitting some kind 01:06:13.750 --> 01:06:18.110 align:middle line:84% of entirely red or entirely blue arithmetic progression. 01:06:18.110 --> 01:06:19.610 align:middle line:90% So this is a bit of a mouthful. 01:06:19.610 --> 01:06:23.150 align:middle line:84% But this is one example of a much more interesting version 01:06:23.150 --> 01:06:27.030 align:middle line:84% of Ramsey theory, is that any large enough object, which, 01:06:27.030 --> 01:06:29.830 align:middle line:84% in this case, is a two-coloring of the integers from 1, 01:06:29.830 --> 01:06:32.790 align:middle line:84% 2 up to n, must necessarily contain 01:06:32.790 --> 01:06:35.510 align:middle line:84% this ordered substructure, which, in our case, 01:06:35.510 --> 01:06:39.550 align:middle line:84% is this monochromatic k term arithmetic progression. 01:06:39.550 --> 01:06:41.042 align:middle line:84% So this has a lot of quantifiers. 01:06:41.042 --> 01:06:42.500 align:middle line:84% Are there any questions about this? 01:06:42.500 --> 01:06:45.070 align:middle line:90% 01:06:45.070 --> 01:06:45.600 align:middle line:90% Make sense? 01:06:45.600 --> 01:06:48.350 align:middle line:90% 01:06:48.350 --> 01:06:51.210 align:middle line:84% So that's van der Waerden's theorem. 01:06:51.210 --> 01:06:52.830 align:middle line:90% The proof of it's not too hard. 01:06:52.830 --> 01:06:56.110 align:middle line:84% But it just doesn't involve any probability. 01:06:56.110 --> 01:06:57.870 align:middle line:90% So we might cover it later. 01:06:57.870 --> 01:07:02.150 align:middle line:84% But it turns out that probability, especially using 01:07:02.150 --> 01:07:04.510 align:middle line:84% the probabilistic method, is a great way 01:07:04.510 --> 01:07:08.350 align:middle line:84% to prove lower bounds for this kind of theorem. 01:07:08.350 --> 01:07:10.610 align:middle line:84% So let's ask for quantitative things. 01:07:10.610 --> 01:07:22.280 align:middle line:84% Let's let W of k be the smallest n such that the property holds. 01:07:22.280 --> 01:07:27.110 align:middle line:90% 01:07:27.110 --> 01:07:29.190 align:middle line:84% Now, you have to be careful and unpack this. 01:07:29.190 --> 01:07:33.190 align:middle line:84% W of k is the smallest n so that which property holds? 01:07:33.190 --> 01:07:35.550 align:middle line:84% The property that every two coloring 01:07:35.550 --> 01:07:38.750 align:middle line:84% of the integers from 1 up to n necessarily 01:07:38.750 --> 01:07:41.110 align:middle line:84% has a k term arithmetic progression. 01:07:41.110 --> 01:07:43.270 align:middle line:84% So this property has an extra quantifier 01:07:43.270 --> 01:07:47.470 align:middle line:84% because I'm not telling you what the two-coloring is. 01:07:47.470 --> 01:07:55.630 align:middle line:84% Now, one of the things you can show that-- 01:07:55.630 --> 01:07:58.550 align:middle line:84% you can reason about small values 01:07:58.550 --> 01:08:01.890 align:middle line:90% of these types of quantities. 01:08:01.890 --> 01:08:03.990 align:middle line:84% So one of the concrete things you can do 01:08:03.990 --> 01:08:05.990 align:middle line:84% is you can ask what's the smallest n 01:08:05.990 --> 01:08:09.190 align:middle line:84% so that every two coloring has a three-term arithmetic 01:08:09.190 --> 01:08:11.110 align:middle line:90% progression. 01:08:11.110 --> 01:08:14.330 align:middle line:84% And it turns out to not be super easy to show this. 01:08:14.330 --> 01:08:16.950 align:middle line:90% But one side is easy. 01:08:16.950 --> 01:08:20.590 align:middle line:84% So the easy part is just to show a lower bound. 01:08:20.590 --> 01:08:25.310 align:middle line:84% So we're going to prove that W of 3 is at least 9. 01:08:25.310 --> 01:08:27.029 align:middle line:84% And the way that you do this is you just 01:08:27.029 --> 01:08:30.590 align:middle line:84% construct a valid two-coloring that 01:08:30.590 --> 01:08:35.229 align:middle line:84% avoids having a three-term arithmetic progression. 01:08:35.229 --> 01:08:39.550 align:middle line:84% So for example, maybe all of these things I circle are red. 01:08:39.550 --> 01:08:42.069 align:middle line:84% And everything I don't circle is blue. 01:08:42.069 --> 01:08:45.670 align:middle line:84% So you can check that this is a coloring of the integers from 1, 01:08:45.670 --> 01:08:47.510 align:middle line:90% 2 all the way up to 8. 01:08:47.510 --> 01:08:49.229 align:middle line:84% And you can just check, by trying out 01:08:49.229 --> 01:08:51.550 align:middle line:84% different choices of a and b, that there 01:08:51.550 --> 01:08:55.710 align:middle line:84% is no three-term arithmetic progression. 01:08:55.710 --> 01:08:58.990 align:middle line:84% The fact that everything that's larger 01:08:58.990 --> 01:09:01.430 align:middle line:84% than this is a lot harder to prove, 01:09:01.430 --> 01:09:03.390 align:middle line:84% because I would have to argue that when 01:09:03.390 --> 01:09:06.510 align:middle line:84% I add another number to this list, not just 01:09:06.510 --> 01:09:09.590 align:middle line:84% this coloring has a three-term arithmetic progression, 01:09:09.590 --> 01:09:12.510 align:middle line:84% but any other two coloring has a three-term arithmetic 01:09:12.510 --> 01:09:13.390 align:middle line:90% progression. 01:09:13.390 --> 01:09:16.710 align:middle line:84% So it involves a huge amount of brute force computer search 01:09:16.710 --> 01:09:19.390 align:middle line:90% to do these kinds of things. 01:09:19.390 --> 01:09:21.270 align:middle line:84% But what we're going to do instead 01:09:21.270 --> 01:09:24.029 align:middle line:84% is we're going to prove the following theorem, 01:09:24.029 --> 01:09:27.149 align:middle line:84% which is a beautiful theorem in its own right, which 01:09:27.149 --> 01:09:30.910 align:middle line:84% is we're going to show nice lower bounds on Wk that are 01:09:30.910 --> 01:09:33.029 align:middle line:90% going to kick in for large k. 01:09:33.029 --> 01:09:35.170 align:middle line:84% So they won't be tight for small values. 01:09:35.170 --> 01:09:38.109 align:middle line:84% But they'll give us a good handle on how large 01:09:38.109 --> 01:09:41.750 align:middle line:84% the n needs to be in this van der Waerden's theorem. 01:09:41.750 --> 01:09:46.870 align:middle line:84% So here's the theorem we're going to prove in what remains. 01:09:46.870 --> 01:09:52.950 align:middle line:84% What I claim is that, for any k, W 01:09:52.950 --> 01:09:56.710 align:middle line:84% of k, the smallest n such that this property holds 01:09:56.710 --> 01:10:01.590 align:middle line:84% is at least square root of k minus 1 times 2 to the k 01:10:01.590 --> 01:10:05.750 align:middle line:90% minus 1 over 2. 01:10:05.750 --> 01:10:08.500 align:middle line:84% So that's the thing we're going to prove. 01:10:08.500 --> 01:10:10.320 align:middle line:84% And the crucial thing about this, 01:10:10.320 --> 01:10:14.730 align:middle line:84% which I really want to explain, is the strategy. 01:10:14.730 --> 01:10:17.780 align:middle line:90% 01:10:17.780 --> 01:10:26.220 align:middle line:84% What we're going to show is that a random coloring 01:10:26.220 --> 01:10:28.940 align:middle line:84% for the choice of n equal to this 01:10:28.940 --> 01:10:34.180 align:middle line:84% has positive probability, probability greater than 0, 01:10:34.180 --> 01:10:43.380 align:middle line:90% of having no k term AP. 01:10:43.380 --> 01:10:46.520 align:middle line:84% So now, this is making good on some of the promises I made you. 01:10:46.520 --> 01:10:49.900 align:middle line:84% See, the whole point of the probabilistic method 01:10:49.900 --> 01:10:53.340 align:middle line:84% is that we want to show that certain things exist, 01:10:53.340 --> 01:10:57.740 align:middle line:84% even if it's hard to construct them by hand. 01:10:57.740 --> 01:11:00.100 align:middle line:84% So in order to prove this theorem, 01:11:00.100 --> 01:11:02.460 align:middle line:84% really, I have to choose some n that's 01:11:02.460 --> 01:11:04.060 align:middle line:84% equal to the right-hand side, that's 01:11:04.060 --> 01:11:07.500 align:middle line:84% equal to the square root of k minus 1 times 2 01:11:07.500 --> 01:11:09.460 align:middle line:90% to the k minus 1 over 2. 01:11:09.460 --> 01:11:11.900 align:middle line:84% If I choose this as my n, then the thing 01:11:11.900 --> 01:11:16.060 align:middle line:84% I have to construct and build is what is the valid two-color. 01:11:16.060 --> 01:11:19.120 align:middle line:84% That's very difficult. I could try some things. 01:11:19.120 --> 01:11:21.220 align:middle line:90% They might or might not work. 01:11:21.220 --> 01:11:23.002 align:middle line:90% And then I might get stuck. 01:11:23.002 --> 01:11:24.460 align:middle line:84% But it turns out that the way we're 01:11:24.460 --> 01:11:27.780 align:middle line:84% going to show that there is a coloring that avoids k term 01:11:27.780 --> 01:11:30.260 align:middle line:84% arithmetic progressions is just by showing 01:11:30.260 --> 01:11:33.860 align:middle line:84% that a random one works with positive probability. 01:11:33.860 --> 01:11:36.420 align:middle line:84% So that's what the probabilistic method is. 01:11:36.420 --> 01:11:40.020 align:middle line:90% And let's execute this strategy. 01:11:40.020 --> 01:11:42.860 align:middle line:84% So just to make sure we're on the same page, what 01:11:42.860 --> 01:11:48.460 align:middle line:84% I'm going to do is I'm going to consider a random variable 01:11:48.460 --> 01:11:56.459 align:middle line:84% x that's going to be equal to the number of monochromatic k 01:11:56.459 --> 01:12:00.580 align:middle line:90% term APs. 01:12:00.580 --> 01:12:02.020 align:middle line:90% I'm just going to count. 01:12:02.020 --> 01:12:04.000 align:middle line:84% So you give me a random coloring. 01:12:04.000 --> 01:12:06.100 align:middle line:84% The dumbest thing we could do is we randomly 01:12:06.100 --> 01:12:08.380 align:middle line:84% decide independently for each integer 01:12:08.380 --> 01:12:09.820 align:middle line:90% whether it's red or blue. 01:12:09.820 --> 01:12:11.860 align:middle line:84% And we're going to show that that works. 01:12:11.860 --> 01:12:14.260 align:middle line:84% And we're going to be interested in this random variable 01:12:14.260 --> 01:12:17.100 align:middle line:84% x that's the number of monochromatic k term 01:12:17.100 --> 01:12:20.700 align:middle line:90% APs in my random coloring. 01:12:20.700 --> 01:12:25.660 align:middle line:84% Now, the basis for making this strategy make sense 01:12:25.660 --> 01:12:28.940 align:middle line:84% is really this very simple lemma that's 01:12:28.940 --> 01:12:32.380 align:middle line:84% going to remind you of things we did earlier in lecture. 01:12:32.380 --> 01:12:34.800 align:middle line:84% What claim-- and this is just a general statement, 01:12:34.800 --> 01:12:37.500 align:middle line:84% but it'll explain why this strategy makes 01:12:37.500 --> 01:12:39.100 align:middle line:90% sense in the first place. 01:12:39.100 --> 01:12:43.030 align:middle line:84% Suppose x is a non-negative random variable. 01:12:43.030 --> 01:12:46.700 align:middle line:90% 01:12:46.700 --> 01:12:50.930 align:middle line:84% And let's say it's an integer valued random variable. 01:12:50.930 --> 01:12:54.500 align:middle line:90% 01:12:54.500 --> 01:13:00.900 align:middle line:84% And let's say that the expectation of x is less than 1. 01:13:00.900 --> 01:13:07.260 align:middle line:84% Then what I claim is that the probability that x equals 0 01:13:07.260 --> 01:13:09.260 align:middle line:90% is strictly positive. 01:13:09.260 --> 01:13:11.682 align:middle line:84% So this is the basis for our strategy, 01:13:11.682 --> 01:13:13.140 align:middle line:84% because the way we're going to show 01:13:13.140 --> 01:13:15.468 align:middle line:84% that a good two-coloring exists is 01:13:15.468 --> 01:13:17.260 align:middle line:84% we're going to look at this random variable 01:13:17.260 --> 01:13:18.520 align:middle line:90% x that's non-negative. 01:13:18.520 --> 01:13:21.580 align:middle line:84% It counts the number of monochromatic k term APs. 01:13:21.580 --> 01:13:23.340 align:middle line:90% It's integer valued. 01:13:23.340 --> 01:13:27.180 align:middle line:84% And as soon as we can show its expectation is less than 1, 01:13:27.180 --> 01:13:30.060 align:middle line:84% that means that the probability that it's actually equal to 0 01:13:30.060 --> 01:13:34.340 align:middle line:84% and there's no k term apes is strictly positive. 01:13:34.340 --> 01:13:37.100 align:middle line:84% So this is the basis for our strategy. 01:13:37.100 --> 01:13:38.400 align:middle line:90% So let's prove this lemma. 01:13:38.400 --> 01:13:40.300 align:middle line:90% This lemma is super simple. 01:13:40.300 --> 01:13:44.740 align:middle line:84% But then let's execute using the strategy in the case of van der 01:13:44.740 --> 01:13:46.980 align:middle line:90% Waerden's theorem. 01:13:46.980 --> 01:13:49.900 align:middle line:84% So we can just write out the expectation. 01:13:49.900 --> 01:13:53.580 align:middle line:84% So the expectation of x, because it's integer valued, 01:13:53.580 --> 01:13:57.940 align:middle line:84% is the sum from i equals 0 to infinity of the probability 01:13:57.940 --> 01:14:03.200 align:middle line:90% that x equals i times i. 01:14:03.200 --> 01:14:08.280 align:middle line:84% This is lower bounded by the sum from i equals 1 to infinity. 01:14:08.280 --> 01:14:10.180 align:middle line:90% So I've dropped the first term. 01:14:10.180 --> 01:14:13.140 align:middle line:84% And then I'm going to drop the i because each 01:14:13.140 --> 01:14:16.340 align:middle line:90% of these i's is at least 1. 01:14:16.340 --> 01:14:19.180 align:middle line:84% And this quantity right here on the right-hand side is just 1 01:14:19.180 --> 01:14:25.740 align:middle line:84% minus the probability that x equals 0 by definition. 01:14:25.740 --> 01:14:28.220 align:middle line:84% So now, if I rearrange this, I'm going 01:14:28.220 --> 01:14:33.700 align:middle line:84% to get that the probability of x being equal to 0 01:14:33.700 --> 01:14:37.540 align:middle line:84% is lower bounded by 1 minus the expectation of x. 01:14:37.540 --> 01:14:39.880 align:middle line:84% And the expectation of x is less than 1. 01:14:39.880 --> 01:14:42.780 align:middle line:90% 01:14:42.780 --> 01:14:46.540 align:middle line:84% So that simple proof is the basis for our strategy, 01:14:46.540 --> 01:14:49.020 align:middle line:84% because all we have to do now is we just 01:14:49.020 --> 01:14:52.820 align:middle line:84% have to compute what the expectation of x is. 01:14:52.820 --> 01:14:58.420 align:middle line:84% I told you what x represents, what random variable it is. 01:14:58.420 --> 01:15:01.580 align:middle line:84% I told you the fact that we're doing a random coloring. 01:15:01.580 --> 01:15:03.660 align:middle line:84% And I told you what the strategy is 01:15:03.660 --> 01:15:07.020 align:middle line:84% of how to reason about the probability that x is 0. 01:15:07.020 --> 01:15:09.500 align:middle line:84% So all that's left is really just the compute this 01:15:09.500 --> 01:15:14.460 align:middle line:84% expectation of x and hope that it's less than 1. 01:15:14.460 --> 01:15:20.740 align:middle line:84% So let's just compute expectation of x. 01:15:20.740 --> 01:15:26.580 align:middle line:84% So I'm going to let Iab denote the relevant indicator 01:15:26.580 --> 01:15:27.660 align:middle line:90% variables. 01:15:27.660 --> 01:15:32.980 align:middle line:84% It'll be 1 if Qab is monochromatic. 01:15:32.980 --> 01:15:35.980 align:middle line:90% 01:15:35.980 --> 01:15:39.500 align:middle line:84% And otherwise, it'll be 0 because these are just 01:15:39.500 --> 01:15:42.020 align:middle line:84% the events that are going to help me get a handle on what 01:15:42.020 --> 01:15:43.960 align:middle line:90% my random variable x is. 01:15:43.960 --> 01:15:47.780 align:middle line:84% x is just summing up all of these indicator variables. 01:15:47.780 --> 01:15:52.500 align:middle line:84% So in particular, then x, the thing I want to keep track of, 01:15:52.500 --> 01:16:00.180 align:middle line:84% is the sum over all ab pairs such that Qab is a subset of 1, 01:16:00.180 --> 01:16:01.620 align:middle line:90% 2 up to n-- 01:16:01.620 --> 01:16:05.540 align:middle line:84% those are the only arithmetic progressions I'll consider-- 01:16:05.540 --> 01:16:09.220 align:middle line:90% of this quantity Iab. 01:16:09.220 --> 01:16:12.620 align:middle line:84% And now, we're good because what I 01:16:12.620 --> 01:16:19.100 align:middle line:84% claim is that the expectation of Iab, the probability 01:16:19.100 --> 01:16:24.260 align:middle line:84% that this event happens, that this arithmetic progression 01:16:24.260 --> 01:16:26.900 align:middle line:90% Qab is monochromatic-- 01:16:26.900 --> 01:16:29.940 align:middle line:84% I claim that this is very simple to compute. 01:16:29.940 --> 01:16:34.300 align:middle line:84% It's equal to 1 over 2 to the k minus 1. 01:16:34.300 --> 01:16:36.800 align:middle line:84% So just to make sure we're on the same page, 01:16:36.800 --> 01:16:43.380 align:middle line:84% so Iab is the indicator event for Qab being monochromatic. 01:16:43.380 --> 01:16:47.060 align:middle line:84% Remember, Q is a k term arithmetic progression. 01:16:47.060 --> 01:16:48.840 align:middle line:90% So why is this probability true? 01:16:48.840 --> 01:16:50.970 align:middle line:84% What's the one line proof of this? 01:16:50.970 --> 01:16:53.660 align:middle line:90% 01:16:53.660 --> 01:16:57.380 align:middle line:84% I'm fixing a particular arithmetic progression. 01:16:57.380 --> 01:16:59.420 align:middle line:84% Why is the chance that it's monochromatic 01:16:59.420 --> 01:17:02.300 align:middle line:84% going down exponentially with the length of the arithmetic 01:17:02.300 --> 01:17:03.080 align:middle line:90% progression? 01:17:03.080 --> 01:17:05.900 align:middle line:90% 01:17:05.900 --> 01:17:06.960 align:middle line:90% Who can help me out? 01:17:06.960 --> 01:17:15.260 align:middle line:90% 01:17:15.260 --> 01:17:15.900 align:middle line:90% Yeah? 01:17:15.900 --> 01:17:18.275 align:middle line:84% STUDENT: Choose the color of each term independently with 01:17:18.275 --> 01:17:21.020 align:middle line:84% probability 1/2, and then multiply by 2 because you have 2 01:17:21.020 --> 01:17:21.860 align:middle line:90% [INAUDIBLE]. 01:17:21.860 --> 01:17:22.777 align:middle line:90% ANKUR MOITRA: Perfect. 01:17:22.777 --> 01:17:23.860 align:middle line:90% That's exactly right. 01:17:23.860 --> 01:17:26.300 align:middle line:84% So the proof of this is exactly what 01:17:26.300 --> 01:17:29.100 align:middle line:84% he said, which is that once we pick 01:17:29.100 --> 01:17:33.140 align:middle line:90% the color, the first color-- 01:17:33.140 --> 01:17:35.380 align:middle line:84% I'm just rephrasing your proof slightly. 01:17:35.380 --> 01:17:37.620 align:middle line:84% So we just pick the first color, namely 01:17:37.620 --> 01:17:39.620 align:middle line:84% the color of a, the first term that shows up 01:17:39.620 --> 01:17:41.500 align:middle line:90% in our arithmetic progression. 01:17:41.500 --> 01:17:44.180 align:middle line:84% The only choice-- let's say a was red. 01:17:44.180 --> 01:17:47.620 align:middle line:84% Then a plus b had better be red. a plus 2b had better be red, 01:17:47.620 --> 01:17:50.340 align:middle line:90% a plus 3b, and so on. 01:17:50.340 --> 01:17:53.340 align:middle line:84% Then all the other colors must match. 01:17:53.340 --> 01:18:01.940 align:middle line:84% All the other colors in Qab must match. 01:18:01.940 --> 01:18:04.890 align:middle line:90% 01:18:04.890 --> 01:18:06.830 align:middle line:84% And that's the proof because there 01:18:06.830 --> 01:18:09.290 align:middle line:90% are k minus 1 other things. 01:18:09.290 --> 01:18:10.850 align:middle line:90% So now, we're in good shape. 01:18:10.850 --> 01:18:13.390 align:middle line:84% We just have one very last ingredient. 01:18:13.390 --> 01:18:15.850 align:middle line:84% And then we'll get our bound to follow, 01:18:15.850 --> 01:18:21.250 align:middle line:84% which is I have to tell you how many Qabs there are. 01:18:21.250 --> 01:18:22.920 align:middle line:90% Lastly, I claim. 01:18:22.920 --> 01:18:25.450 align:middle line:90% 01:18:25.450 --> 01:18:32.170 align:middle line:84% the number of valid ab pairs which induce an arithmetic 01:18:32.170 --> 01:18:36.290 align:middle line:84% progression, which stays in the integers from 1 to n, 01:18:36.290 --> 01:18:41.598 align:middle line:84% is at most n times n over k minus 1. 01:18:41.598 --> 01:18:43.890 align:middle line:84% So the way to think about it-- remember, our arithmetic 01:18:43.890 --> 01:18:45.170 align:middle line:90% progression starts at a. 01:18:45.170 --> 01:18:49.570 align:middle line:84% And then it goes a plus b and a plus 2b all the way up to a plus 01:18:49.570 --> 01:18:51.610 align:middle line:90% k minus 1 times B. 01:18:51.610 --> 01:18:53.650 align:middle line:84% So in order for this arithmetic progression 01:18:53.650 --> 01:18:56.210 align:middle line:84% to stay in the integers from 1 to n, 01:18:56.210 --> 01:18:59.190 align:middle line:84% I better choose my a to be at most n. 01:18:59.190 --> 01:19:01.170 align:middle line:84% Otherwise, it's not going to be in the set. 01:19:01.170 --> 01:19:04.750 align:middle line:84% And then my b would better be at most n over k minus 1 01:19:04.750 --> 01:19:08.210 align:middle line:84% because I'm going to add b k minus 1 times. 01:19:08.210 --> 01:19:11.610 align:middle line:84% So if it were larger than this, I would fall outside the set. 01:19:11.610 --> 01:19:18.090 align:middle line:84% So putting this all together, we know that the expectation of x 01:19:18.090 --> 01:19:22.110 align:middle line:84% is at most n squared over k minus 1-- 01:19:22.110 --> 01:19:24.250 align:middle line:84% that's the number of valid things-- 01:19:24.250 --> 01:19:28.290 align:middle line:84% times the expectation of each of these Iab's, which is 1 over 2 01:19:28.290 --> 01:19:30.690 align:middle line:90% to the k minus 1. 01:19:30.690 --> 01:19:33.570 align:middle line:84% And now, you can check that that implies 01:19:33.570 --> 01:19:37.170 align:middle line:84% that if n is less than this quantity 01:19:37.170 --> 01:19:43.730 align:middle line:84% I told you, k minus 1 times 2 to the k minus 1 over 2, 01:19:43.730 --> 01:19:49.170 align:middle line:84% then it implies that the expectation of x is less than 1. 01:19:49.170 --> 01:19:50.810 align:middle line:90% So that's our proof. 01:19:50.810 --> 01:19:54.330 align:middle line:84% So this is a very beautiful but subtle proof because of the way 01:19:54.330 --> 01:19:56.170 align:middle line:90% we're using probability. 01:19:56.170 --> 01:19:58.550 align:middle line:84% I want to show that W of k is large 01:19:58.550 --> 01:20:01.850 align:middle line:84% because I want to construct a valid two-coloring that avoids 01:20:01.850 --> 01:20:03.835 align:middle line:90% all arithmetic progressions. 01:20:03.835 --> 01:20:05.210 align:middle line:84% What I'm going to do is I'm going 01:20:05.210 --> 01:20:07.690 align:middle line:84% to reason about the random variable that 01:20:07.690 --> 01:20:10.770 align:middle line:84% counts the number of k term APs as a way 01:20:10.770 --> 01:20:13.850 align:middle line:84% to argue that there is a good two-coloring to begin with. 01:20:13.850 --> 01:20:15.730 align:middle line:84% And the rest of it is a computation 01:20:15.730 --> 01:20:17.930 align:middle line:90% using linearity of expectation. 01:20:17.930 --> 01:20:21.090 align:middle line:84% But that allows us to prove that these exotic objects, which 01:20:21.090 --> 01:20:23.450 align:middle line:84% are hard to actually build and construct, 01:20:23.450 --> 01:20:26.643 align:middle line:84% must exist even if we can't find them. 01:20:26.643 --> 01:20:29.310 align:middle line:84% So that's one of the cool things about the probabilistic method. 01:20:29.310 --> 01:20:32.450 align:middle line:84% Sometimes, you can show very exotic things 01:20:32.450 --> 01:20:35.730 align:middle line:84% exist, even though we don't know how to build them from scratch. 01:20:35.730 --> 01:20:39.510 align:middle line:84% And we'll do some more applications of this later, too, 01:20:39.510 --> 01:20:40.570 align:middle line:90% and next time. 01:20:40.570 --> 01:20:42.710 align:middle line:84% And we'll also cover the Chernoff bound. 01:20:42.710 --> 01:20:43.670 align:middle line:90% So we'll stop there. 01:20:43.670 --> 01:20:46.400 align:middle line:84% And I'll see you guys on Thursday. 01:20:46.400 --> 01:20:52.000 align:middle line:90%