WEBVTT

00:00:00.000 --> 00:00:02.455 align:middle line:90%
[SQUEAKING]

00:00:02.455 --> 00:00:03.928 align:middle line:90%
[RUSTLING]

00:00:03.928 --> 00:00:05.401 align:middle line:90%
[CLICKING]

00:00:05.401 --> 00:00:11.790 align:middle line:90%


00:00:11.790 --> 00:00:13.680 align:middle line:84%
YUFEI ZHAO: In this
video, we'll look

00:00:13.680 --> 00:00:18.270 align:middle line:84%
at three basic yet important
inequalities in probability--

00:00:18.270 --> 00:00:22.830 align:middle line:84%
Markov's inequality, Chebyshev's
inequality, and the Chernoff

00:00:22.830 --> 00:00:24.480 align:middle line:90%
bound.

00:00:24.480 --> 00:00:28.350 align:middle line:84%
Markov's inequality says
that if we are given X,

00:00:28.350 --> 00:00:32.189 align:middle line:84%
a real valued non-negative
random variable,

00:00:32.189 --> 00:00:37.200 align:middle line:84%
then for every positive number
lambda we have the following

00:00:37.200 --> 00:00:38.610 align:middle line:90%
inequality--

00:00:38.610 --> 00:00:42.030 align:middle line:84%
the probability
that X is bigger--

00:00:42.030 --> 00:00:44.970 align:middle line:84%
is at least lambda,
this probability

00:00:44.970 --> 00:00:50.220 align:middle line:84%
is no more than the expectation
of X divided by lambda.

00:00:50.220 --> 00:00:52.230 align:middle line:84%
One way to interpret
this inequality

00:00:52.230 --> 00:00:55.890 align:middle line:84%
is that if X is a
non-negative random variable

00:00:55.890 --> 00:01:00.810 align:middle line:84%
with small expectation,
then it is unlikely for X

00:01:00.810 --> 00:01:02.820 align:middle line:90%
to be very high.

00:01:02.820 --> 00:01:06.030 align:middle line:84%
This is a very important
and useful inequality.

00:01:06.030 --> 00:01:07.240 align:middle line:90%
And let us prove it.

00:01:07.240 --> 00:01:08.495 align:middle line:90%
The proof is quite short.

00:01:08.495 --> 00:01:12.450 align:middle line:90%


00:01:12.450 --> 00:01:17.490 align:middle line:84%
We can start with
the expectation of X.

00:01:17.490 --> 00:01:22.170 align:middle line:84%
And rewrite this in
the following way.

00:01:22.170 --> 00:01:26.970 align:middle line:84%
It is at least the
expectation of X

00:01:26.970 --> 00:01:30.870 align:middle line:84%
times the indicator
function corresponding

00:01:30.870 --> 00:01:35.110 align:middle line:90%
to when X is at least lambda.

00:01:35.110 --> 00:01:45.090 align:middle line:84%
So this means it is 1 if X is
at least lambda and 0 otherwise.

00:01:45.090 --> 00:01:52.160 align:middle line:90%


00:01:52.160 --> 00:01:58.250 align:middle line:84%
Now, claim next
that the expression

00:01:58.250 --> 00:02:04.220 align:middle line:84%
inside the expectation is
at least lambda times 1

00:02:04.220 --> 00:02:07.910 align:middle line:84%
sub X being at least
lambda, because when

00:02:07.910 --> 00:02:11.190 align:middle line:84%
X is at least lambda, well,
X is at least a lambda.

00:02:11.190 --> 00:02:14.490 align:middle line:90%
Otherwise, both sides are 0.

00:02:14.490 --> 00:02:16.940 align:middle line:84%
So this inequality
holds as well.

00:02:16.940 --> 00:02:21.950 align:middle line:84%
And finally, we can
pull out the lambda

00:02:21.950 --> 00:02:25.010 align:middle line:84%
and get lambda times
the probability

00:02:25.010 --> 00:02:29.600 align:middle line:90%
that X is at least lambda.

00:02:29.600 --> 00:02:33.360 align:middle line:84%
So this finishes the proof
of Markov's inequality.

00:02:33.360 --> 00:02:36.190 align:middle line:90%


00:02:36.190 --> 00:02:40.030 align:middle line:84%
Next, let's move to
Chebyshev's inequality.

00:02:40.030 --> 00:02:42.700 align:middle line:84%
The statement of
Chebyshev's inequality

00:02:42.700 --> 00:02:46.090 align:middle line:84%
is that if we're given
a real random variable

00:02:46.090 --> 00:02:51.670 align:middle line:84%
X, then for every positive
real number lambda,

00:02:51.670 --> 00:02:57.070 align:middle line:84%
the probability that X
deviates from its mean

00:02:57.070 --> 00:03:00.940 align:middle line:84%
by at least lambda
times the square root

00:03:00.940 --> 00:03:02.460 align:middle line:90%
of the variance of X--

00:03:02.460 --> 00:03:07.920 align:middle line:84%
so this probability-- is at
most 1 over lambda squared.

00:03:07.920 --> 00:03:13.750 align:middle line:84%
Let me remind you
that the variance

00:03:13.750 --> 00:03:17.200 align:middle line:84%
of the random variable
X is the quantity which

00:03:17.200 --> 00:03:22.120 align:middle line:84%
is defined to be
the expectation of X

00:03:22.120 --> 00:03:25.390 align:middle line:84%
minus the expectation
of X squared.

00:03:25.390 --> 00:03:29.770 align:middle line:84%
And it is also equal
to the expectation

00:03:29.770 --> 00:03:36.040 align:middle line:84%
of X squared minus the square
of the expectation of X.

00:03:36.040 --> 00:03:42.790 align:middle line:84%
So this is the variance
of the random variable X.

00:03:42.790 --> 00:03:43.290 align:middle line:90%
Yeah.

00:03:43.290 --> 00:03:46.020 align:middle line:84%
So intuitively, what
Chebyshev's inequality

00:03:46.020 --> 00:03:49.870 align:middle line:84%
tells us is that if a random
variable has small variance,

00:03:49.870 --> 00:03:55.260 align:middle line:84%
then it is unlikely to
deviate too far from its mean.

00:03:55.260 --> 00:03:57.280 align:middle line:84%
Let us now prove
this inequality.

00:03:57.280 --> 00:04:01.710 align:middle line:90%


00:04:01.710 --> 00:04:06.420 align:middle line:84%
The left-hand side,
namely this probability,

00:04:06.420 --> 00:04:11.610 align:middle line:84%
we can rewrite it by
squaring the inequality

00:04:11.610 --> 00:04:16.329 align:middle line:84%
in this expression,
which we-- doing

00:04:16.329 --> 00:04:18.310 align:middle line:90%
this gets us the following.

00:04:18.310 --> 00:04:21.160 align:middle line:90%


00:04:21.160 --> 00:04:24.850 align:middle line:84%
On the left-hand side, we
get X minus the expectation

00:04:24.850 --> 00:04:26.650 align:middle line:90%
of X squared.

00:04:26.650 --> 00:04:30.040 align:middle line:84%
On the right-hand side,
we get lambda times lambda

00:04:30.040 --> 00:04:34.480 align:middle line:90%
squared times the variance of X.

00:04:34.480 --> 00:04:39.720 align:middle line:84%
And now let us apply
Markov's inequality.

00:04:39.720 --> 00:04:42.840 align:middle line:84%
So apply what we saw
earlier, apply Markov,

00:04:42.840 --> 00:04:48.960 align:middle line:84%
to get the expectation of X
minus the expectation of X

00:04:48.960 --> 00:04:55.860 align:middle line:84%
squared divided by lambda
squared, variance of X.

00:04:55.860 --> 00:05:00.090 align:middle line:84%
But you see, from the
definition of variance,

00:05:00.090 --> 00:05:03.780 align:middle line:84%
the expression circled in
red here is equal to 1.

00:05:03.780 --> 00:05:07.500 align:middle line:84%
And therefore, the
right-hand side is equal to 1

00:05:07.500 --> 00:05:10.380 align:middle line:90%
over lambda squared.

00:05:10.380 --> 00:05:13.870 align:middle line:84%
So that finishes the proof
of Chebyshev's inequality.

00:05:13.870 --> 00:05:16.900 align:middle line:90%


00:05:16.900 --> 00:05:18.670 align:middle line:84%
The third inequality
that we'll look at

00:05:18.670 --> 00:05:20.860 align:middle line:90%
is known as the Chernoff bound.

00:05:20.860 --> 00:05:23.540 align:middle line:84%
And unlike the
earlier inequalities,

00:05:23.540 --> 00:05:26.750 align:middle line:84%
which is for fairly
general random variables,

00:05:26.750 --> 00:05:30.370 align:middle line:84%
the Chernoff bound is for a
more specific setting, namely

00:05:30.370 --> 00:05:33.220 align:middle line:84%
a sum of independent
random variables.

00:05:33.220 --> 00:05:36.880 align:middle line:84%
And more specifically, let
us look at the case where

00:05:36.880 --> 00:05:43.300 align:middle line:84%
the random variable S sub n is
a sum of n different plus minus

00:05:43.300 --> 00:05:43.820 align:middle line:90%
1's.

00:05:43.820 --> 00:05:48.790 align:middle line:84%
So each Xi is plus 1 with
probability 1/2 and minus 1 with

00:05:48.790 --> 00:05:53.580 align:middle line:84%
probability 1/2, all chosen
independently at random.

00:05:53.580 --> 00:05:59.310 align:middle line:84%
So one way to interpret Sn is
that we walk on the number line,

00:05:59.310 --> 00:06:00.640 align:middle line:90%
take n steps.

00:06:00.640 --> 00:06:04.110 align:middle line:84%
Each step, we flip a coin
and whether walk either 1

00:06:04.110 --> 00:06:08.890 align:middle line:84%
step to the right or 1 step
to the left and walk n steps.

00:06:08.890 --> 00:06:12.730 align:middle line:84%
The conclusion of
Chernoff bound is

00:06:12.730 --> 00:06:18.910 align:middle line:84%
that for every positive lambda,
the probability that S sub n is

00:06:18.910 --> 00:06:25.310 align:middle line:84%
at least lambda times root n
is at most this quantity here,

00:06:25.310 --> 00:06:29.990 align:middle line:84%
which is e to the minus
lambda squared over 2.

00:06:29.990 --> 00:06:33.470 align:middle line:90%
So in other words, let S sub n--

00:06:33.470 --> 00:06:35.240 align:middle line:84%
so where we end
up in this walk--

00:06:35.240 --> 00:06:39.600 align:middle line:84%
cannot drift too far
away from the origin.

00:06:39.600 --> 00:06:44.130 align:middle line:84%
And too far here means some
large multiple of root n.

00:06:44.130 --> 00:06:55.170 align:middle line:84%
It is worth noting that root
n here is the variance of--

00:06:55.170 --> 00:07:01.260 align:middle line:84%
so the square root of
the variance of S sub n.

00:07:01.260 --> 00:07:04.950 align:middle line:84%
So if we simply apply
Chebyshev's inequality

00:07:04.950 --> 00:07:11.730 align:middle line:84%
to S sub n, we would arrive
at the following conclusion.

00:07:11.730 --> 00:07:15.450 align:middle line:84%
So Chebyshev tells us
that the probability

00:07:15.450 --> 00:07:21.930 align:middle line:84%
that Sn is at least
lambda is at most 1

00:07:21.930 --> 00:07:26.600 align:middle line:84%
over lambda squared, which is
already an interesting bound.

00:07:26.600 --> 00:07:29.820 align:middle line:84%
But the Chernoff bound gives
a much stronger conclusion.

00:07:29.820 --> 00:07:33.960 align:middle line:84%
So the right-hand side
decreases very rapidly

00:07:33.960 --> 00:07:37.230 align:middle line:84%
as a function of lambda
compared to the Chebyshev

00:07:37.230 --> 00:07:40.230 align:middle line:84%
bound, which decreases
only at the rate of 1

00:07:40.230 --> 00:07:41.450 align:middle line:90%
over lambda squared.

00:07:41.450 --> 00:07:47.190 align:middle line:90%


00:07:47.190 --> 00:07:50.570 align:middle line:84%
Let us now prove Chebyshev--
the Chebyshev bound.

00:07:50.570 --> 00:07:55.990 align:middle line:90%


00:07:55.990 --> 00:07:59.260 align:middle line:84%
For the proof, we'll need
to introduce a new idea.

00:07:59.260 --> 00:08:01.970 align:middle line:84%
And this is the idea of a
moment-generating function.

00:08:01.970 --> 00:08:07.450 align:middle line:84%
So let t be a
non-negative real number.

00:08:07.450 --> 00:08:23.810 align:middle line:84%
And let us consider the
moment-generating function,

00:08:23.810 --> 00:08:27.320 align:middle line:90%
in this case given as follows.

00:08:27.320 --> 00:08:31.160 align:middle line:84%
Consider the expectation
of the quantity

00:08:31.160 --> 00:08:36.210 align:middle line:90%
e to the t times S sub n.

00:08:36.210 --> 00:08:38.110 align:middle line:84%
So instead of
considering, for example,

00:08:38.110 --> 00:08:39.990 align:middle line:84%
the expectation
of S sub n, let's

00:08:39.990 --> 00:08:45.690 align:middle line:84%
consider the exponential
applied to the random variable.

00:08:45.690 --> 00:08:51.120 align:middle line:84%
So this is the expectation is
over the randomness in S sub n.

00:08:51.120 --> 00:08:55.500 align:middle line:84%
Well, let us now rewrite
this expression by expanding

00:08:55.500 --> 00:08:58.110 align:middle line:90%
the definition of S sub n.

00:08:58.110 --> 00:09:01.780 align:middle line:90%
And we get the following.

00:09:01.780 --> 00:09:06.470 align:middle line:90%


00:09:06.470 --> 00:09:08.600 align:middle line:84%
And the next step is
where we crucially

00:09:08.600 --> 00:09:13.500 align:middle line:84%
use the fact that the Xi's are
independent random variables,

00:09:13.500 --> 00:09:17.630 align:middle line:84%
which then allows us to split
this expectation as a product

00:09:17.630 --> 00:09:19.666 align:middle line:90%
of individual expectations.

00:09:19.666 --> 00:09:30.850 align:middle line:90%


00:09:30.850 --> 00:09:34.045 align:middle line:84%
Let's look at one of these
terms, one of these expectation

00:09:34.045 --> 00:09:34.545 align:middle line:90%
factors.

00:09:34.545 --> 00:09:37.800 align:middle line:90%


00:09:37.800 --> 00:09:43.440 align:middle line:84%
Well, X sub 1 is plus 1 with
probability 1/2 and minus 1 with

00:09:43.440 --> 00:09:44.730 align:middle line:90%
probability 1/2.

00:09:44.730 --> 00:09:48.870 align:middle line:84%
So this factor here
equals to e to the t

00:09:48.870 --> 00:09:53.160 align:middle line:90%
plus e to the minus t over 2.

00:09:53.160 --> 00:09:55.750 align:middle line:84%
And likewise with
all the other terms.

00:09:55.750 --> 00:09:58.590 align:middle line:84%
So the right-hand side
equals to this quantity

00:09:58.590 --> 00:09:59.800 align:middle line:90%
raised to the power n.

00:09:59.800 --> 00:10:02.470 align:middle line:90%


00:10:02.470 --> 00:10:06.610 align:middle line:84%
So this exactly computes the
moment-generating function

00:10:06.610 --> 00:10:11.230 align:middle line:90%
of this random variable S sub n.

00:10:11.230 --> 00:10:15.700 align:middle line:84%
Let us now try to manipulate
this right-hand side expression

00:10:15.700 --> 00:10:17.325 align:middle line:90%
to make it easier to work with.

00:10:17.325 --> 00:10:21.070 align:middle line:90%


00:10:21.070 --> 00:10:25.600 align:middle line:84%
So starting with
this expression,

00:10:25.600 --> 00:10:28.390 align:middle line:84%
for now temporarily
dropping the power

00:10:28.390 --> 00:10:31.870 align:middle line:84%
of n, so just look at what's
inside the parentheses,

00:10:31.870 --> 00:10:35.250 align:middle line:84%
we can apply a Taylor
series expansion.

00:10:35.250 --> 00:10:43.630 align:middle line:84%
So by Taylor series, I'm
going to do the steps slightly

00:10:43.630 --> 00:10:45.100 align:middle line:84%
on the quick side,
but I encourage

00:10:45.100 --> 00:10:48.200 align:middle line:84%
you to work it out yourself
if you can follow this step.

00:10:48.200 --> 00:10:50.620 align:middle line:84%
Basically, we apply the
Taylor series expansion

00:10:50.620 --> 00:10:55.600 align:middle line:84%
and note that the odd index
terms all cancel each other out.

00:10:55.600 --> 00:10:59.740 align:middle line:84%
So again, I encourage you to try
out this calculation yourself.

00:10:59.740 --> 00:11:01.790 align:middle line:90%
We get the following.

00:11:01.790 --> 00:11:09.750 align:middle line:90%


00:11:09.750 --> 00:11:16.470 align:middle line:84%
On the other hand, let's
look at the expression e

00:11:16.470 --> 00:11:20.400 align:middle line:90%
to the t squared over 2.

00:11:20.400 --> 00:11:23.130 align:middle line:84%
So we'll see where this
comes from in a second.

00:11:23.130 --> 00:11:25.360 align:middle line:90%
So this X should be a t.

00:11:25.360 --> 00:11:33.330 align:middle line:90%


00:11:33.330 --> 00:11:37.470 align:middle line:84%
So let's look at this e
to the t squared over 2.

00:11:37.470 --> 00:11:45.840 align:middle line:84%
By using Taylor expansion over
here, we get the following.

00:11:45.840 --> 00:11:57.940 align:middle line:84%
X of each term is t to the 2k
over k factorial, 2 to the k.

00:11:57.940 --> 00:12:00.730 align:middle line:90%


00:12:00.730 --> 00:12:03.580 align:middle line:84%
So this is by writing out
the Taylor series expansion

00:12:03.580 --> 00:12:05.650 align:middle line:90%
for both of these expressions.

00:12:05.650 --> 00:12:10.820 align:middle line:84%
And finally, let's note that
even on a term-by-term basis,

00:12:10.820 --> 00:12:14.410 align:middle line:84%
one has this inequality
by comparing what

00:12:14.410 --> 00:12:16.840 align:middle line:90%
happens in the denominators.

00:12:16.840 --> 00:12:18.380 align:middle line:84%
So I'll leave this
as an exercise.

00:12:18.380 --> 00:12:20.210 align:middle line:90%
But this is not too hard to see.

00:12:20.210 --> 00:12:23.410 align:middle line:90%


00:12:23.410 --> 00:12:28.890 align:middle line:84%
This allows us then to
rewrite the previous line

00:12:28.890 --> 00:12:33.630 align:middle line:84%
by upper bounding this
moment-generating function

00:12:33.630 --> 00:12:40.830 align:middle line:84%
by the expression e to
the nt squared over 2.

00:12:40.830 --> 00:12:43.070 align:middle line:90%
So that's what we just deduced.

00:12:43.070 --> 00:12:46.830 align:middle line:90%


00:12:46.830 --> 00:12:55.670 align:middle line:84%
Finally, let us use
Markov's inequality

00:12:55.670 --> 00:13:05.110 align:middle line:84%
to upper bound the probability
that S sub n is at least lambda

00:13:05.110 --> 00:13:06.160 align:middle line:90%
times root n.

00:13:06.160 --> 00:13:08.670 align:middle line:84%
So that's what we're
trying to bound.

00:13:08.670 --> 00:13:13.360 align:middle line:90%


00:13:13.360 --> 00:13:17.290 align:middle line:84%
As with earlier,
well, previously we

00:13:17.290 --> 00:13:18.350 align:middle line:90%
squared both sides.

00:13:18.350 --> 00:13:21.375 align:middle line:84%
But now let's take an
exponential of both sides.

00:13:21.375 --> 00:13:27.170 align:middle line:90%


00:13:27.170 --> 00:13:29.180 align:middle line:84%
So for this step,
actually, I think

00:13:29.180 --> 00:13:33.360 align:middle line:84%
I will need t to be strictly
positive for this step to hold.

00:13:33.360 --> 00:13:37.180 align:middle line:84%
So let me change
the hypothesis on t.

00:13:37.180 --> 00:13:39.580 align:middle line:90%
So this is true.

00:13:39.580 --> 00:13:45.940 align:middle line:84%
And then we can apply Markov
to upper bound this probability

00:13:45.940 --> 00:13:51.680 align:middle line:84%
by the expectation of this
expression e to the t sub n,

00:13:51.680 --> 00:13:55.970 align:middle line:84%
which is random, and then
divided by the right-hand side,

00:13:55.970 --> 00:13:56.925 align:middle line:90%
which is a constant.

00:13:56.925 --> 00:14:00.880 align:middle line:90%


00:14:00.880 --> 00:14:04.090 align:middle line:84%
And earlier, we had
determined an upper bound

00:14:04.090 --> 00:14:06.220 align:middle line:84%
for the moment-generating
function.

00:14:06.220 --> 00:14:11.020 align:middle line:84%
And plugging everything in, we
get this following expression--

00:14:11.020 --> 00:14:18.410 align:middle line:84%
e to the nt squared over 2
minus t lambda square root n.

00:14:18.410 --> 00:14:21.520 align:middle line:90%


00:14:21.520 --> 00:14:24.100 align:middle line:84%
So this expression,
this inequality,

00:14:24.100 --> 00:14:28.900 align:middle line:84%
is true for every
positive value of t.

00:14:28.900 --> 00:14:32.470 align:middle line:84%
And the next step then
is to use a value of t

00:14:32.470 --> 00:14:34.910 align:middle line:90%
that works most in our favor.

00:14:34.910 --> 00:14:39.730 align:middle line:84%
And one can do this by trying
to minimize the expression

00:14:39.730 --> 00:14:41.980 align:middle line:90%
on the right-hand side.

00:14:41.980 --> 00:14:43.880 align:middle line:90%
And you can do this.

00:14:43.880 --> 00:14:54.130 align:middle line:84%
And turns out that the optimal
t to set is lambda over root n.

00:14:54.130 --> 00:14:56.110 align:middle line:84%
So you can determine
the optimal value of t

00:14:56.110 --> 00:14:57.400 align:middle line:90%
by taking the derivative.

00:14:57.400 --> 00:14:59.200 align:middle line:90%
And this is the optimal value.

00:14:59.200 --> 00:15:12.100 align:middle line:84%
And set t into this bound
gives the desired conclusion.

00:15:12.100 --> 00:15:17.190 align:middle line:90%


00:15:17.190 --> 00:15:21.620 align:middle line:84%
So if you set this value of t
into this right-hand side bound,

00:15:21.620 --> 00:15:26.250 align:middle line:84%
you will see the right-hand side
of Chernoff bound popping out.

00:15:26.250 --> 00:15:30.500 align:middle line:84%
So that's the proof
of the Chernoff bound.

00:15:30.500 --> 00:15:32.510 align:middle line:84%
So the key idea
introduced here is

00:15:32.510 --> 00:15:34.760 align:middle line:90%
the moment-generating function.

00:15:34.760 --> 00:15:38.880 align:middle line:84%
And because of the independence
of the components of S sub n,

00:15:38.880 --> 00:15:41.990 align:middle line:84%
we can factorize the
moment-generating function

00:15:41.990 --> 00:15:44.610 align:middle line:84%
and, through some additional
algebraic manipulations,

00:15:44.610 --> 00:15:46.580 align:middle line:90%
derive the desired bound.

00:15:46.580 --> 00:15:48.830 align:middle line:84%
So all three of
these inequalities

00:15:48.830 --> 00:15:54.590 align:middle line:84%
are basic and powerful
techniques in probability.

00:15:54.590 --> 00:15:58.370 align:middle line:84%
The Chernoff bound in particular
gives an extraordinarily good

00:15:58.370 --> 00:16:01.220 align:middle line:84%
bound on the
probability that a sum

00:16:01.220 --> 00:16:03.320 align:middle line:84%
of independent
random variables can

00:16:03.320 --> 00:16:05.670 align:middle line:90%
deviate from its expectation.

00:16:05.670 --> 00:16:09.320 align:middle line:84%
And over here, although we only
looked at the case when Xi is

00:16:09.320 --> 00:16:12.200 align:middle line:84%
plus 1 or minus 1, each
with probability 1/2,

00:16:12.200 --> 00:16:18.320 align:middle line:84%
the same proof can be applied
in a more versatile way to other

00:16:18.320 --> 00:16:21.770 align:middle line:84%
settings, as long as you have
a sum of independent random

00:16:21.770 --> 00:16:23.620 align:middle line:90%
variables.

00:16:23.620 --> 00:16:30.000 align:middle line:90%