WEBVTT

00:00:01.270 --> 00:00:05.080
Let us now illustrate the linear
least mean squares estimation

00:00:05.080 --> 00:00:07.730
methodology in the
context of an example.

00:00:07.730 --> 00:00:11.230
And we're going to revisit
our familiar example

00:00:11.230 --> 00:00:15.780
that we considered earlier in
the context of general least

00:00:15.780 --> 00:00:17.790
mean squares estimation.

00:00:17.790 --> 00:00:19.740
Let us remind
ourselves what were

00:00:19.740 --> 00:00:22.540
the assumptions
behind this example.

00:00:22.540 --> 00:00:24.360
There is an unknown
random variable

00:00:24.360 --> 00:00:28.790
that we wish to estimate, and
that random variable happens

00:00:28.790 --> 00:00:34.610
to be uniform on the
range from 4 to 10.

00:00:34.610 --> 00:00:39.220
What we get to observe
is a random variable X,

00:00:39.220 --> 00:00:43.110
which is equal to Theta
plus or minus something.

00:00:43.110 --> 00:00:47.230
So X is Theta plus a noise term.

00:00:47.230 --> 00:00:51.370
And that noise term
can be anything

00:00:51.370 --> 00:00:57.800
in the range from
minus 1 to plus 1.

00:00:57.800 --> 00:01:00.540
Furthermore, the
distribution of U

00:01:00.540 --> 00:01:02.750
is this particular
uniform no matter

00:01:02.750 --> 00:01:07.710
what theta is, so Theta
and U are independent.

00:01:14.760 --> 00:01:18.090
These modeling assumptions
correspond to this picture.

00:01:18.090 --> 00:01:20.440
This is the range
of X and Theta.

00:01:20.440 --> 00:01:22.850
And the joint distribution
of X and Theta

00:01:22.850 --> 00:01:25.789
happens to be a
uniform distribution

00:01:25.789 --> 00:01:28.930
on this particular shape.

00:01:28.930 --> 00:01:32.280
However, we're not really going
to use this picture other than

00:01:32.280 --> 00:01:34.320
for illustration purposes.

00:01:34.320 --> 00:01:38.039
You could take just this as
the formulation of the problem

00:01:38.039 --> 00:01:40.470
that we're interested in.

00:01:40.470 --> 00:01:45.789
So now, to develop the form of
the optimal linear estimator,

00:01:45.789 --> 00:01:47.750
all we need to do
is to determine

00:01:47.750 --> 00:01:50.789
the various constants
that show up.

00:01:50.789 --> 00:01:53.550
So let's start
with expectations.

00:01:53.550 --> 00:01:56.960
Theta is uniform from 4 to 10.

00:01:56.960 --> 00:01:59.440
Therefore, the expected
value is the midpoint,

00:01:59.440 --> 00:02:01.710
which is equal to 7.

00:02:01.710 --> 00:02:05.980
U has a symmetric
distribution around 0,

00:02:05.980 --> 00:02:09.720
so its expected value is
going to be equal to 0.

00:02:09.720 --> 00:02:13.270
X is the sum of Theta
and U. Therefore,

00:02:13.270 --> 00:02:16.550
its expected value is the sum
of these two expected values

00:02:16.550 --> 00:02:19.800
and is equal to 7.

00:02:19.800 --> 00:02:22.600
Let us now look at variances.

00:02:22.600 --> 00:02:28.110
The variance of a uniform
is equal to the square

00:02:28.110 --> 00:02:31.930
of the length of the interval
on which the uniform is

00:02:31.930 --> 00:02:37.180
distributed-- in this case,
it's 6 squared-- divided always

00:02:37.180 --> 00:02:39.270
by a coefficient of 12.

00:02:39.270 --> 00:02:44.600
6 squared is 36,
so we obtain three.

00:02:44.600 --> 00:02:49.210
The variance of U is determined
by a similar formula,

00:02:49.210 --> 00:02:52.180
except that now we have
an interval of length 2,

00:02:52.180 --> 00:02:56.560
so we obtain 2 squared over 12.

00:02:56.560 --> 00:02:57.720
And this is 1/3.

00:03:00.750 --> 00:03:04.300
Now let us look at
the variance of X.

00:03:04.300 --> 00:03:08.510
Since X is the sum of Theta and
U, and since the two of them

00:03:08.510 --> 00:03:11.130
are independent,
the variance of X

00:03:11.130 --> 00:03:15.420
is going to be the sum of
these two variances, which

00:03:15.420 --> 00:03:17.755
is 10 over 3.

00:03:21.220 --> 00:03:25.070
Now let us try to calculate
the covariance term.

00:03:28.890 --> 00:03:36.560
The covariance of Theta
with X is this expression,

00:03:36.560 --> 00:03:40.480
because X is Theta
plus U. And then,

00:03:40.480 --> 00:03:44.210
using linearity
properties of covariances,

00:03:44.210 --> 00:03:48.400
this is the covariance
of Theta with itself,

00:03:48.400 --> 00:03:54.120
plus the covariance
of Theta with U. Now,

00:03:54.120 --> 00:03:59.280
Theta and U are independent, so
this covariance is equal to 0.

00:03:59.280 --> 00:04:01.320
The covariance of
Theta with itself

00:04:01.320 --> 00:04:03.440
is just the same
as the variance,

00:04:03.440 --> 00:04:09.030
so here we obtain
an answer of 3.

00:04:09.030 --> 00:04:12.140
And so now we have all
the pieces of information

00:04:12.140 --> 00:04:15.770
that we need, and we can
proceed and write down

00:04:15.770 --> 00:04:18.589
the form of the
linear estimator.

00:04:18.589 --> 00:04:23.080
The expected value
of Theta is 7.

00:04:23.080 --> 00:04:26.540
Then, the covariance
of Theta with X

00:04:26.540 --> 00:04:32.310
is 3, divided by the
variance, which is 10 over 3.

00:04:32.310 --> 00:04:34.380
So this ratio becomes 9/10.

00:04:37.780 --> 00:04:44.440
And then X minus the expected
value of X gives us this term.

00:04:44.440 --> 00:04:48.760
So this is the form of the
optimal linear estimator,

00:04:48.760 --> 00:04:54.110
and if you wish to plot it,
it is a curve of this kind.

00:04:54.110 --> 00:04:58.450
It is actually interesting
to contrast this solution

00:04:58.450 --> 00:05:02.710
to the shape of the optimal
estimator, the least mean

00:05:02.710 --> 00:05:05.530
squares estimator, or the
conditional expectation

00:05:05.530 --> 00:05:08.265
estimator, which we
had found earlier,

00:05:08.265 --> 00:05:12.960
and which corresponds to
this particular blue curve.

00:05:12.960 --> 00:05:15.910
So in some sense,
the linear estimator

00:05:15.910 --> 00:05:21.830
is a pretty good approximation
of the optimal non-linear one.

00:05:21.830 --> 00:05:24.240
It does the best
job that it can do,

00:05:24.240 --> 00:05:27.570
subject to the constraint that
it has to be a linear function.

00:05:30.200 --> 00:05:33.300
Notice also that this
coefficient here is, of course,

00:05:33.300 --> 00:05:34.290
positive.

00:05:34.290 --> 00:05:38.220
This reflects the fact that
the two random variables,

00:05:38.220 --> 00:05:41.780
X and Theta, are
positively correlated.

00:05:41.780 --> 00:05:43.800
This should be clear
from this diagram.

00:05:43.800 --> 00:05:48.200
When X increases, Theta tends to
also increase, and vice versa.

00:05:48.200 --> 00:05:51.430
It's also reflected in the
fact that the covariance

00:05:51.430 --> 00:05:54.600
is a positive number.

00:05:54.600 --> 00:05:56.740
On the other hand,
because this coefficient

00:05:56.740 --> 00:06:01.410
is 9/10 and not equal
to 1, this red line

00:06:01.410 --> 00:06:03.880
is somewhat slanted
in comparison

00:06:03.880 --> 00:06:07.405
with the orientation of
the diagram that we have.

00:06:12.060 --> 00:06:14.320
The calculations
that we went through

00:06:14.320 --> 00:06:18.710
in this particular example
are pretty generic.

00:06:18.710 --> 00:06:21.620
This is what you need
to do in general.

00:06:21.620 --> 00:06:24.100
You just look at the
random variables involved,

00:06:24.100 --> 00:06:27.840
you calculate their means,
you calculate their variances.

00:06:27.840 --> 00:06:30.420
Then you may have to
do some extra work

00:06:30.420 --> 00:06:33.800
to calculate the
covariance of interest.

00:06:33.800 --> 00:06:36.500
And once you're done, you
plug in the numerical values

00:06:36.500 --> 00:06:39.040
that you have found,
and you obtain

00:06:39.040 --> 00:06:41.900
the form of the
linear estimator.