WEBVTT
00:00:07.460 --> 00:00:09.910
OK.
00:00:09.910 --> 00:00:15.260
Here's lecture sixteen
and if you remember
00:00:15.260 --> 00:00:21.150
I ended up the last lecture
with this formula for what
00:00:21.150 --> 00:00:24.610
I called a projection matrix.
00:00:24.610 --> 00:00:31.610
And maybe I could just
recap for a minute what
00:00:31.610 --> 00:00:35.010
is that magic formula doing?
00:00:35.010 --> 00:00:38.390
For example, it's
supposed to be --
00:00:38.390 --> 00:00:40.230
it's supposed to
produce a projection,
00:00:40.230 --> 00:00:44.770
if I multiply by a b,
so I take P times b,
00:00:44.770 --> 00:00:51.240
I'm supposed to project that
vector b to the nearest point
00:00:51.240 --> 00:00:54.150
in the column space.
00:00:54.150 --> 00:00:54.800
OK.
00:00:54.800 --> 00:00:56.350
Can I just --
00:00:56.350 --> 00:01:01.420
one way to recap is to
take the two extreme cases.
00:01:01.420 --> 00:01:05.040
Suppose a vector b is
in the column space?
00:01:05.040 --> 00:01:10.030
Then what do I get when
I apply the projection P?
00:01:10.030 --> 00:01:13.610
So I'm projecting
into the column space
00:01:13.610 --> 00:01:18.780
but I'm starting with a vector
in this case that's already
00:01:18.780 --> 00:01:20.950
in the column
space, so of course
00:01:20.950 --> 00:01:25.860
when I project it I
get B again, right.
00:01:25.860 --> 00:01:29.860
And I want to show you how
that comes out of this formula.
00:01:29.860 --> 00:01:32.550
Let me do the other extreme.
00:01:32.550 --> 00:01:35.350
Suppose that vector is
perpendicular to the column
00:01:35.350 --> 00:01:36.210
space.
00:01:36.210 --> 00:01:38.860
So imagine this column
space as a plane
00:01:38.860 --> 00:01:42.550
and imagine b as sticking
straight up perpendicular
00:01:42.550 --> 00:01:43.490
to it.
00:01:43.490 --> 00:01:50.490
What's the nearest point in the
column space to b in that case?
00:01:50.490 --> 00:01:54.380
So what's the projection
onto the plane,
00:01:54.380 --> 00:01:57.980
the nearest point in the
plane, if the vector b that
00:01:57.980 --> 00:02:02.000
I'm looking at is -- got no
component in the column space,
00:02:02.000 --> 00:02:05.050
it's sticking completely
-- ninety degrees with it,
00:02:05.050 --> 00:02:10.220
then Pb should be zero, right.
00:02:10.220 --> 00:02:13.040
So those are the
two extreme cases.
00:02:13.040 --> 00:02:18.000
The average vector has a
component P in the column space
00:02:18.000 --> 00:02:20.930
and a component
perpendicular to it,
00:02:20.930 --> 00:02:25.930
and what the projection
does is it kills this part
00:02:25.930 --> 00:02:29.510
and it preserves this part.
00:02:29.510 --> 00:02:30.010
OK.
00:02:30.010 --> 00:02:32.230
Can we just see why that's true?
00:02:32.230 --> 00:02:37.140
Just -- that formula
ought to work.
00:02:37.140 --> 00:02:41.010
So let me start with this one.
00:02:41.010 --> 00:02:44.260
What vectors are in the -- are
perpendicular to the column
00:02:44.260 --> 00:02:45.240
space?
00:02:45.240 --> 00:02:48.220
How do I see that
I really get zero?
00:02:48.220 --> 00:02:50.850
I have to think, what does
it mean for a vector b
00:02:50.850 --> 00:02:54.410
to be perpendicular
to the column space?
00:02:54.410 --> 00:02:59.430
So if it's perpendicular
to all the columns,
00:02:59.430 --> 00:03:02.100
then it's in some other space.
00:03:02.100 --> 00:03:05.740
We've got our four spaces so
the reason I do this is it's
00:03:05.740 --> 00:03:10.030
perfectly using what we
know about our four spaces.
00:03:10.030 --> 00:03:13.860
What vectors are perpendicular
to the column space?
00:03:13.860 --> 00:03:19.190
Those are the guys in the
null space of A transpose,
00:03:19.190 --> 00:03:20.290
right?
00:03:20.290 --> 00:03:22.740
That's the first
section of this chapter,
00:03:22.740 --> 00:03:26.160
that's the key geometry
of these spaces.
00:03:26.160 --> 00:03:28.300
If I'm perpendicular
to the column space,
00:03:28.300 --> 00:03:30.961
I'm in the null
space of A transpose.
00:03:30.961 --> 00:03:31.460
OK.
00:03:31.460 --> 00:03:33.790
So if I'm in the null
space of A transpose,
00:03:33.790 --> 00:03:41.760
and I multiply this big formula
times b, so now I'm getting Pb,
00:03:41.760 --> 00:03:48.530
this is now the projection,
Pb, do you see that I get zero?
00:03:48.530 --> 00:03:50.470
Of course I get zero.
00:03:50.470 --> 00:03:52.950
Right at the end
there, A transpose b
00:03:52.950 --> 00:03:54.840
will give me zero right away.
00:03:54.840 --> 00:03:57.780
So that's why that zero's here.
00:03:57.780 --> 00:04:00.890
Because if I'm perpendicular
to the column space, then
00:04:00.890 --> 00:04:03.840
I'm in the null space of A
transpose and A transpose
00:04:03.840 --> 00:04:08.640
b is OK, what about
the other possibility.
00:04:08.640 --> 00:04:09.970
zilch.
00:04:09.970 --> 00:04:13.370
How do I see that this formula
gives me the right answer
00:04:13.370 --> 00:04:15.250
if b is in the column space?
00:04:18.230 --> 00:04:21.890
So what's a typical vector
in the column space?
00:04:21.890 --> 00:04:24.480
It's a combination
of the columns.
00:04:24.480 --> 00:04:27.240
How do I write a
combination of the columns?
00:04:27.240 --> 00:04:31.090
So tell me, how would
I write, you know,
00:04:31.090 --> 00:04:34.440
your everyday vector
that's in the column space?
00:04:34.440 --> 00:04:38.860
It would have the form
A times some x, right?
00:04:38.860 --> 00:04:42.280
That's what's in the column
space, A times something.
00:04:42.280 --> 00:04:44.730
That makes it a
combination of the columns.
00:04:44.730 --> 00:04:49.570
So these b's were in the
null space of A transpose.
00:04:49.570 --> 00:04:54.640
These guys in the column
space, those b's are Ax-s.
00:04:54.640 --> 00:04:55.210
Right?
00:04:55.210 --> 00:04:58.825
If b is in the column space
then it has the form Ax.
00:05:01.380 --> 00:05:04.450
I'm going to stick that on the
quiz or the final for sure.
00:05:04.450 --> 00:05:08.290
That you have to realize --
because we've said it like
00:05:08.290 --> 00:05:13.020
a thousand times that the things
in the column space are vectors
00:05:13.020 --> 00:05:14.241
A times x.
00:05:14.241 --> 00:05:14.740
OK.
00:05:14.740 --> 00:05:18.050
And do you see what happens
now if we use our formula?
00:05:18.050 --> 00:05:19.990
There's an A transpose A.
00:05:19.990 --> 00:05:21.860
Gets canceled by its inverse.
00:05:21.860 --> 00:05:25.800
We're left with an A times x.
00:05:25.800 --> 00:05:27.530
So the result was Ax.
00:05:27.530 --> 00:05:28.570
Which was b.
00:05:28.570 --> 00:05:30.100
Do you see that it works?
00:05:30.100 --> 00:05:32.750
This is that whole business.
00:05:32.750 --> 00:05:35.870
Cancel, cancel, leaving Ax.
00:05:35.870 --> 00:05:37.840
And Ax was b.
00:05:37.840 --> 00:05:43.730
So that turned out to
be b, in this case.
00:05:43.730 --> 00:05:51.300
OK, so geometrically what we're
seeing is we're taking a vector
00:05:51.300 --> 00:05:53.010
--
00:05:53.010 --> 00:06:00.770
we've got the column space
and perpendicular to that
00:06:00.770 --> 00:06:06.350
is the null space
of A transpose.
00:06:06.350 --> 00:06:10.230
And our typical
vector b is out here.
00:06:10.230 --> 00:06:12.970
There's zero, so there's
our typical vector b,
00:06:12.970 --> 00:06:19.460
and what we're doing is we're
projecting it to P. And the --
00:06:19.460 --> 00:06:22.300
and of course at the same time
we're finding the other part
00:06:22.300 --> 00:06:24.810
of it which is e.
00:06:24.810 --> 00:06:30.520
So the two pieces, the
projection piece and the error
00:06:30.520 --> 00:06:35.280
piece, add up to the original b.
00:06:35.280 --> 00:06:36.230
OK.
00:06:36.230 --> 00:06:39.520
That's like what
our matrix does.
00:06:39.520 --> 00:06:41.440
So this is P --
00:06:41.440 --> 00:06:48.260
P is -- this P is Ab, is sorry
-- is Pb, it's the projection,
00:06:48.260 --> 00:06:52.590
applied to b, and this one is --
00:06:52.590 --> 00:06:55.080
OK, that's a projection too.
00:06:55.080 --> 00:06:58.070
That's a projection
down onto that space.
00:06:58.070 --> 00:06:59.860
What's a good formula for it?
00:06:59.860 --> 00:07:05.340
Suppose I ask you for the
projection of the projection
00:07:05.340 --> 00:07:08.830
matrix onto the --
00:07:08.830 --> 00:07:13.240
this space, this
perpendicular space?
00:07:13.240 --> 00:07:16.960
So if this projection
was P, what's
00:07:16.960 --> 00:07:21.070
the projection that gives me e?
00:07:21.070 --> 00:07:24.170
It's the -- what I want is to
get the rest of the vector,
00:07:24.170 --> 00:07:30.790
so it'll be just I minus P times
b, that's a projection too.
00:07:30.790 --> 00:07:35.880
That's the projection onto
the perpendicular space.
00:07:38.950 --> 00:07:40.040
OK.
00:07:40.040 --> 00:07:44.150
So if P's a projection, I
minus P is a projection.
00:07:44.150 --> 00:07:47.790
If P is symmetric, I
minus P is symmetric.
00:07:47.790 --> 00:07:52.290
If P squared equals P, then I
minus P squared equals I minus
00:07:52.290 --> 00:07:55.690
P. It's just --
00:07:55.690 --> 00:08:00.460
the algebra -- is only
doing what your --
00:08:00.460 --> 00:08:05.270
picture is completely
telling you.
00:08:05.270 --> 00:08:08.122
But the algebra leads
to this expression.
00:08:11.820 --> 00:08:16.280
That expression for P given --
00:08:16.280 --> 00:08:19.810
given a basis for
the subspace, given
00:08:19.810 --> 00:08:25.690
the matrix A whose columns are
a basis for our column space.
00:08:25.690 --> 00:08:28.820
OK, that's recap because you
-- you need to see that formula
00:08:28.820 --> 00:08:30.460
more than once.
00:08:30.460 --> 00:08:34.669
And now can I pick
up on using it?
00:08:34.669 --> 00:08:37.789
So now -- and the --
00:08:37.789 --> 00:08:46.590
it's like, let me do that again,
I'll go right through a problem
00:08:46.590 --> 00:08:52.470
that I started at the end, which
is find a best straight line.
00:08:52.470 --> 00:08:53.820
You remember that problem, I --
00:08:53.820 --> 00:08:57.530
I picked a particular
set of points,
00:08:57.530 --> 00:09:00.820
they weren't specially
brilliant, t equal one,
00:09:00.820 --> 00:09:07.190
two, three, the heights were
one, two, and then two again.
00:09:07.190 --> 00:09:10.570
So they were -- heights
were that point, that point,
00:09:10.570 --> 00:09:13.320
which makes it look like I've
got a nice forty-five-degree
00:09:13.320 --> 00:09:18.800
line -- but then the third
point didn't lie on the line.
00:09:18.800 --> 00:09:22.500
And I wanted to find
the best straight line.
00:09:22.500 --> 00:09:26.404
So I'm looking for the
-- this line, y=C+Dt.
00:09:30.850 --> 00:09:35.110
And it's not going to go
through all three points,
00:09:35.110 --> 00:09:37.880
because no line goes
through all three points.
00:09:37.880 --> 00:09:42.190
So I'm going to pick
the best line, the --
00:09:42.190 --> 00:09:45.990
the best being the one that
makes the overall error
00:09:45.990 --> 00:09:48.430
as small as I can make it.
00:09:48.430 --> 00:09:52.390
Now I have to tell you,
what is that overall error?
00:09:52.390 --> 00:10:01.600
And -- because that determines
what's the winning line.
00:10:01.600 --> 00:10:02.740
If we don't know --
00:10:02.740 --> 00:10:06.810
I mean we have to decide
what we mean by the error --
00:10:06.810 --> 00:10:12.940
and then we minimize and we find
the right -- the best C and D.
00:10:12.940 --> 00:10:18.100
So if I went through this --
if I went through that point,
00:10:18.100 --> 00:10:18.600
OK.
00:10:18.600 --> 00:10:20.688
I would solve the
equation C+D=1.
00:10:23.680 --> 00:10:26.310
Because at t equal to one --
00:10:26.310 --> 00:10:30.050
I'd have C plus D, and
it would come out right.
00:10:30.050 --> 00:10:34.310
If it went through this point,
I'd have C plus two D equal to
00:10:34.310 --> 00:10:35.150
two.
00:10:35.150 --> 00:10:38.990
Because at t equal to two, I
would like to get the answer
00:10:38.990 --> 00:10:39.550
two.
00:10:39.550 --> 00:10:43.950
At the third point, I have
C plus three D because t is
00:10:43.950 --> 00:10:47.160
three, but the -- the
answer I'm shooting for is
00:10:47.160 --> 00:10:49.850
two again.
00:10:49.850 --> 00:10:52.680
So those are my three equations.
00:10:52.680 --> 00:10:55.720
And they don't have a solution.
00:10:55.720 --> 00:10:58.110
But they've got a best solution.
00:10:58.110 --> 00:10:59.890
What do I mean by best solution?
00:10:59.890 --> 00:11:04.220
So let me take time out to
remember what I'm talking
00:11:04.220 --> 00:11:06.550
about for best solution.
00:11:06.550 --> 00:11:11.960
So this is my equation Ax=b.
00:11:11.960 --> 00:11:18.440
A is this matrix, one,
one, one, one, two, three.
00:11:18.440 --> 00:11:22.580
x is my -- only have
two unknowns, C and D,
00:11:22.580 --> 00:11:27.440
and b is my right-hand
side, one, two, three.
00:11:27.440 --> 00:11:27.940
OK.
00:11:31.930 --> 00:11:34.670
No solution.
00:11:34.670 --> 00:11:37.300
Three eq- I have a
three by two matrix,
00:11:37.300 --> 00:11:40.200
I do have two
independent columns --
00:11:40.200 --> 00:11:42.540
so I do have a basis
for the column space,
00:11:42.540 --> 00:11:44.630
those two columns
are independent,
00:11:44.630 --> 00:11:46.420
they're a basis for
the column space,
00:11:46.420 --> 00:11:52.370
but the column space
doesn't include that vector.
00:11:52.370 --> 00:11:57.600
So best possible in this --
00:11:57.600 --> 00:12:01.540
what would best possible mean?
00:12:01.540 --> 00:12:05.750
The way that comes out to
linear equations is I --
00:12:05.750 --> 00:12:13.970
I want to minimize
the sum of these --
00:12:13.970 --> 00:12:15.457
I'm going to make an error here.
00:12:15.457 --> 00:12:16.790
I'm going to make an error here.
00:12:16.790 --> 00:12:18.950
I'm going to make
an error there.
00:12:18.950 --> 00:12:24.970
And I'm going to sum and
square and add up those errors.
00:12:24.970 --> 00:12:26.720
So it's a sum of squares.
00:12:26.720 --> 00:12:30.750
It's a least squares
solution I'm looking for.
00:12:30.750 --> 00:12:37.980
So if I -- those errors are the
difference between Ax and b.
00:12:37.980 --> 00:12:40.320
That's what I want
to make small.
00:12:40.320 --> 00:12:42.951
And the way I'm measuring
this -- this is a vector,
00:12:42.951 --> 00:12:43.450
right?
00:12:43.450 --> 00:12:45.480
This is e1,e2 ,e3.
00:12:45.480 --> 00:12:49.090
The Ax-b, this is the e.
00:12:49.090 --> 00:12:50.890
The error vector.
00:12:50.890 --> 00:12:55.890
And small means its length.
00:12:55.890 --> 00:12:57.662
The length of that vector.
00:12:57.662 --> 00:12:59.370
That's what I'm going
to try to minimize.
00:12:59.370 --> 00:13:04.280
And it's convenient to square.
00:13:04.280 --> 00:13:06.920
If I make something
small, I make --
00:13:09.620 --> 00:13:12.320
this is a never negative
quantity, right?
00:13:12.320 --> 00:13:13.690
The length of that vector.
00:13:16.760 --> 00:13:20.040
The length will be zero
exactly when the --
00:13:20.040 --> 00:13:21.990
when I have the
zero vector here.
00:13:21.990 --> 00:13:26.620
That's exactly the case
when I can solve exactly,
00:13:26.620 --> 00:13:29.730
b is in the column
space, all great.
00:13:29.730 --> 00:13:31.820
But I'm not in that case now.
00:13:31.820 --> 00:13:34.070
I'm going to have
an error vector, e.
00:13:34.070 --> 00:13:35.815
What's this error
vector in my picture?
00:13:38.340 --> 00:13:42.030
I guess what I'm trying
to say is there's --
00:13:42.030 --> 00:13:45.270
there's two pictures
of what's going on.
00:13:45.270 --> 00:13:47.540
There's two pictures
of what's going on.
00:13:47.540 --> 00:13:50.900
One picture is --
00:13:50.900 --> 00:13:55.020
in this is the three
points and the line.
00:13:55.020 --> 00:14:00.220
And in that picture, what
are the three errors?
00:14:00.220 --> 00:14:03.480
The three errors are what
I miss by in this equation.
00:14:03.480 --> 00:14:05.150
So it's this --
00:14:05.150 --> 00:14:06.740
this little bit here.
00:14:06.740 --> 00:14:08.950
That vertical distance
up to the line.
00:14:08.950 --> 00:14:12.780
There's one -- sorry there's
one, and there's C plus D.
00:14:12.780 --> 00:14:14.700
And it's that difference.
00:14:14.700 --> 00:14:17.720
Here's two and here's C+2D.
00:14:17.720 --> 00:14:20.600
So vertically it's
that distance --
00:14:20.600 --> 00:14:23.620
that little error there is e1.
00:14:23.620 --> 00:14:26.220
This little error here is e2.
00:14:26.220 --> 00:14:30.540
This little error
coming up is e3.
00:14:30.540 --> 00:14:32.350
e3.
00:14:32.350 --> 00:14:35.240
And what's my overall error?
00:14:35.240 --> 00:14:43.240
Is e1 square plus e2
squared plus e3 squared.
00:14:43.240 --> 00:14:44.920
That's what I'm
trying to make small.
00:14:44.920 --> 00:14:54.090
I -- some statisticians -- this
is a big part of statistics,
00:14:54.090 --> 00:14:56.360
fitting straight lines is
a big part of science --
00:14:56.360 --> 00:15:00.310
and specifically statistics,
where the right word to use
00:15:00.310 --> 00:15:02.210
would be regression.
00:15:02.210 --> 00:15:05.270
I'm doing regression here.
00:15:05.270 --> 00:15:06.145
Linear regression.
00:15:09.840 --> 00:15:12.820
And I'm using this
sum of squares
00:15:12.820 --> 00:15:15.270
as the measure of error.
00:15:15.270 --> 00:15:21.190
Again, some statisticians
would be -- they would say, OK,
00:15:21.190 --> 00:15:24.000
I'll solve that problem
because it's the clean problem.
00:15:24.000 --> 00:15:27.080
It leads to a beautiful
linear system.
00:15:27.080 --> 00:15:30.340
But they would be a little
careful about these squares,
00:15:30.340 --> 00:15:32.670
for -- in this case.
00:15:32.670 --> 00:15:35.990
If one of these
points was way off.
00:15:35.990 --> 00:15:39.040
Suppose I had a measurement at
t equal zero that was way off.
00:15:41.560 --> 00:15:44.880
Well, would the straight line,
would the best line be the same
00:15:44.880 --> 00:15:46.890
if I had this fourth point?
00:15:46.890 --> 00:15:50.180
Suppose I have this
fourth data point.
00:15:50.180 --> 00:15:54.880
No, certainly the line would --
00:15:54.880 --> 00:15:57.620
it wouldn't be the -- that
wouldn't be the best line.
00:15:57.620 --> 00:16:01.100
Because that line would
have a giant error --
00:16:01.100 --> 00:16:04.820
and when I squared it it
would be like way out of sight
00:16:04.820 --> 00:16:06.860
compared to the others.
00:16:06.860 --> 00:16:14.280
So this would be called by
statisticians an outlier,
00:16:14.280 --> 00:16:17.830
and they would not be happy to
see the whole problem turned
00:16:17.830 --> 00:16:21.150
topsy-turvy by this one outlier,
which could be a mistake,
00:16:21.150 --> 00:16:22.760
after all.
00:16:22.760 --> 00:16:26.500
So they wouldn't -- so they
wouldn't like maybe squaring,
00:16:26.500 --> 00:16:29.940
if there were outliers, they
would want to identify them.
00:16:29.940 --> 00:16:30.440
OK.
00:16:30.440 --> 00:16:35.800
I'm not going to --
00:16:35.800 --> 00:16:40.870
I don't want to suggest that
least squares isn't used,
00:16:40.870 --> 00:16:44.790
it's the most used, but
it's not exclusively used
00:16:44.790 --> 00:16:47.040
because it's a little --
00:16:47.040 --> 00:16:50.000
overcompensates for outliers.
00:16:50.000 --> 00:16:51.500
Because of that squaring.
00:16:51.500 --> 00:16:52.000
OK.
00:16:52.000 --> 00:16:54.300
So suppose we don't
have this guy,
00:16:54.300 --> 00:16:57.300
we just have these
three equations.
00:16:57.300 --> 00:17:01.680
And I want to make --
minimize this error.
00:17:01.680 --> 00:17:02.650
OK.
00:17:02.650 --> 00:17:08.069
Now, what I said is there's
two pictures to look at.
00:17:08.069 --> 00:17:10.940
One picture is this one.
00:17:10.940 --> 00:17:14.700
The three points, the best line.
00:17:14.700 --> 00:17:16.280
And the errors.
00:17:16.280 --> 00:17:20.760
Now, on this picture,
what are these points
00:17:20.760 --> 00:17:24.890
on the line, the points
that are really on the line?
00:17:24.890 --> 00:17:30.490
So they're -- points, let
me call them P1, P2, and P3,
00:17:30.490 --> 00:17:35.610
those are three numbers, so
this -- this height is P1,
00:17:35.610 --> 00:17:45.700
this height is P2, this height
is P3, and what are those guys?
00:17:45.700 --> 00:17:49.930
Suppose those were the
three values instead of --
00:17:49.930 --> 00:17:53.840
there's b1, ev- everybody's
seen all these -- sorry,
00:17:53.840 --> 00:17:57.090
my art is as usual
not the greatest,
00:17:57.090 --> 00:18:04.590
but there's the given b1, the
given b2, and the given b3.
00:18:04.590 --> 00:18:09.330
I promise not to put a single
letter more on that picture.
00:18:09.330 --> 00:18:10.050
OK.
00:18:10.050 --> 00:18:15.600
There's b1, P1 is the one on
the line, and e1 is the distance
00:18:15.600 --> 00:18:16.600
between.
00:18:16.600 --> 00:18:21.410
And same at points two
and same at points three.
00:18:21.410 --> 00:18:23.370
OK, so what's up?
00:18:23.370 --> 00:18:26.310
What's up with those Ps?
00:18:26.310 --> 00:18:29.930
P1, P2, P3, what are they?
00:18:29.930 --> 00:18:32.520
They're the components,
they lie on the line,
00:18:32.520 --> 00:18:33.720
right?
00:18:33.720 --> 00:18:38.420
They're the points
which if instead
00:18:38.420 --> 00:18:44.530
of one, two, two, which
were the b's, suppose I put
00:18:44.530 --> 00:18:47.230
P1, P2, P3 in here.
00:18:47.230 --> 00:18:50.150
I'll figure out in a minute
what those numbers are.
00:18:50.150 --> 00:18:53.040
But I just want to get the
picture of what I'm doing.
00:18:53.040 --> 00:18:56.390
If I put P1, P2, P3 in
those three equations,
00:18:56.390 --> 00:18:58.795
what would be good about
the three equations?
00:19:01.820 --> 00:19:03.820
I could solve them.
00:19:03.820 --> 00:19:06.420
A line goes through the Ps.
00:19:06.420 --> 00:19:10.400
So the P1, P2, P3 vector,
that's in the column
00:19:10.400 --> 00:19:11.320
space.
00:19:11.320 --> 00:19:14.480
That is a combination
of these columns.
00:19:14.480 --> 00:19:16.400
It's the closest combination.
00:19:16.400 --> 00:19:18.180
It's this picture.
00:19:18.180 --> 00:19:20.920
See, I've got the two
pictures like here's
00:19:20.920 --> 00:19:24.710
the picture that
shows the points, this
00:19:24.710 --> 00:19:28.240
is a picture in a
blackboard plane,
00:19:28.240 --> 00:19:34.310
here's a picture that's
showing the vectors.
00:19:34.310 --> 00:19:38.540
The vector b, which is in
this case, in this example
00:19:38.540 --> 00:19:42.090
is the vector one, two, two.
00:19:42.090 --> 00:19:47.940
The column space is in
this case spanned by the --
00:19:47.940 --> 00:19:49.720
well, you see A there.
00:19:49.720 --> 00:19:55.600
The column space of the matrix
one, one, one, one, two, three.
00:19:55.600 --> 00:20:01.540
And this picture shows
the nearest point.
00:20:01.540 --> 00:20:04.510
There's the -- that
point P1, P2, P3,
00:20:04.510 --> 00:20:08.050
which I'm going to compute
before the end of this hour,
00:20:08.050 --> 00:20:13.090
is the closest point
in the column space.
00:20:13.090 --> 00:20:13.780
OK.
00:20:13.780 --> 00:20:19.560
Let me -- t I don't dare
leave it any longer --
00:20:19.560 --> 00:20:21.650
can I just compute it now.
00:20:21.650 --> 00:20:24.850
So I want to compute --
00:20:24.850 --> 00:20:28.800
find P. All right.
00:20:28.800 --> 00:20:39.250
Find P. Find x, which
is CD, find P and P. OK.
00:20:39.250 --> 00:20:42.430
And I really should put
these little hats on
00:20:42.430 --> 00:20:49.830
to remind myself that they're
the estimated the best line,
00:20:49.830 --> 00:20:51.970
not the perfect line.
00:20:51.970 --> 00:20:53.050
OK.
00:20:53.050 --> 00:20:54.330
OK.
00:20:54.330 --> 00:20:55.540
How do I proceed?
00:20:55.540 --> 00:20:58.340
Let's just run
through the mechanics.
00:20:58.340 --> 00:21:02.530
What's the equation for x?
00:21:02.530 --> 00:21:04.620
The -- or x hat.
00:21:04.620 --> 00:21:10.390
The equation for that is A
transpose A x hat equals A
00:21:10.390 --> 00:21:12.500
transpose x --
00:21:12.500 --> 00:21:14.105
A transpose b.
00:21:18.020 --> 00:21:19.530
The most --
00:21:19.530 --> 00:21:23.620
I'm -- will venture to call
that the most important equation
00:21:23.620 --> 00:21:26.350
in statistics.
00:21:26.350 --> 00:21:28.560
And in estimation.
00:21:28.560 --> 00:21:33.140
And whatever you're -- wherever
you've got error and noise this
00:21:33.140 --> 00:21:36.980
is the estimate
that you use first.
00:21:36.980 --> 00:21:37.500
OK.
00:21:37.500 --> 00:21:42.740
Whenever you're fitting
things by a few parameters,
00:21:42.740 --> 00:21:44.700
that's the equation to use.
00:21:44.700 --> 00:21:46.500
OK, let's solve it.
00:21:46.500 --> 00:21:47.970
What is A transpose A?
00:21:47.970 --> 00:21:50.580
So I have to figure out
what these matrices are.
00:21:50.580 --> 00:21:56.860
One, one, one, one, two, three
and one, one, one, one, two,
00:21:56.860 --> 00:22:04.490
three, that gives me some
matrix, that gives me
00:22:04.490 --> 00:22:12.510
a matrix, what do I get out of
that, three, six, six, and one
00:22:12.510 --> 00:22:15.720
and four and nine, fourteen.
00:22:15.720 --> 00:22:17.040
OK.
00:22:17.040 --> 00:22:21.830
And what do I expect to see in
that matrix and I do see it,
00:22:21.830 --> 00:22:25.210
just before I keep going
with the calculation?
00:22:25.210 --> 00:22:28.450
I expect that matrix
to be symmetric.
00:22:28.450 --> 00:22:30.565
I expect it to be invertible.
00:22:34.100 --> 00:22:36.300
And near the end
of the course I'm
00:22:36.300 --> 00:22:39.060
going to say I expect it
to be positive definite,
00:22:39.060 --> 00:22:45.590
but that's a future fact
about this crucial matrix,
00:22:45.590 --> 00:22:47.050
A transpose A.
00:22:47.050 --> 00:22:47.670
OK.
00:22:47.670 --> 00:22:50.880
And now let me
figure A transpose b.
00:22:50.880 --> 00:22:57.280
So let me -- can I tack on b as
an extra column here, one, two,
00:22:57.280 --> 00:22:59.950
two?
00:22:59.950 --> 00:23:04.770
And tack on the extra
A transpose b is --
00:23:04.770 --> 00:23:09.580
looks like five and one
and four and six, eleven.
00:23:13.770 --> 00:23:20.760
I think my equations are three
C plus six D equals five,
00:23:20.760 --> 00:23:29.700
and six D plus fourt-six C
plus fourteen D is eleven.
00:23:29.700 --> 00:23:33.090
Can I just for safety
see if I did that right?
00:23:33.090 --> 00:23:37.350
One, one, one times
one, two, two is five.
00:23:37.350 --> 00:23:40.630
One, two, three, that's
one, four and six, eleven.
00:23:40.630 --> 00:23:42.667
Looks good.
00:23:42.667 --> 00:23:43.625
These are my equations.
00:23:48.860 --> 00:23:52.000
That's my -- they're called
the normal equations.
00:23:54.610 --> 00:23:56.984
I'll just write that
word down because it --
00:24:02.800 --> 00:24:04.470
so I solve them.
00:24:04.470 --> 00:24:10.270
I solve that for C and
D. I would like to --
00:24:10.270 --> 00:24:13.130
before I solve them could I
do one thing that's on the --
00:24:13.130 --> 00:24:16.570
that's just above here?
00:24:16.570 --> 00:24:18.110
I would like to --
00:24:18.110 --> 00:24:21.470
I'd like to find these
equations from calculus.
00:24:21.470 --> 00:24:26.320
I'd like to find them from
this minimizing thing.
00:24:26.320 --> 00:24:28.010
So what's the first error?
00:24:28.010 --> 00:24:32.690
The first error is what I
missed by in the first equation.
00:24:32.690 --> 00:24:36.250
C plus D minus one squared.
00:24:36.250 --> 00:24:40.010
And the second error is what
I miss in the second equation.
00:24:40.010 --> 00:24:44.110
C plus two D minus two squared.
00:24:44.110 --> 00:24:52.350
And the third error squared is C
plus three D minus two squared.
00:24:52.350 --> 00:24:56.410
That's my -- overall squared
error that I'm trying
00:24:56.410 --> 00:24:58.040
to minimize.
00:24:58.040 --> 00:24:58.610
OK.
00:24:58.610 --> 00:25:08.910
So how would you minimize that?
00:25:08.910 --> 00:25:16.270
OK, linear algebra has given us
the equations for the minimum.
00:25:16.270 --> 00:25:20.750
But we could use calculus too.
00:25:20.750 --> 00:25:24.440
That's a function of
two variables, C and D,
00:25:24.440 --> 00:25:28.010
and we're looking
for the minimum.
00:25:28.010 --> 00:25:31.140
So how do we find it?
00:25:31.140 --> 00:25:35.160
Directly from calculus, we
take partial derivatives,
00:25:35.160 --> 00:25:37.510
right, we've got two
variables, C and D,
00:25:37.510 --> 00:25:40.900
so take the partial
derivative with respect to C
00:25:40.900 --> 00:25:44.560
and set it to zero, and
you'll get that equation.
00:25:44.560 --> 00:25:47.140
Take the partial
derivative with respect --
00:25:47.140 --> 00:25:51.570
I'm not going to write it
all out, just -- you will.
00:25:51.570 --> 00:25:56.370
The partial derivative with
respect to D, it -- you know,
00:25:56.370 --> 00:25:59.340
it's going to be linear,
that's the beauty of these
00:25:59.340 --> 00:26:03.220
squares,that if I have the
square of something and I take
00:26:03.220 --> 00:26:07.520
its derivative I get something
And this is what I get. linear.
00:26:07.520 --> 00:26:11.440
So this is the derivative of
the error with respect to C
00:26:11.440 --> 00:26:13.770
being zero, and this
is the derivative
00:26:13.770 --> 00:26:17.850
of the error with
respect to D being zero.
00:26:17.850 --> 00:26:20.660
Wherever you look, these
equations keep coming.
00:26:20.660 --> 00:26:22.370
So now I guess I'm
going to solve it,
00:26:22.370 --> 00:26:25.830
what will I do, I'll subtract,
I'll do elimination of course,
00:26:25.830 --> 00:26:27.820
because that's the only
thing I know how to do.
00:26:27.820 --> 00:26:32.540
Two of these away from
this would give me --
00:26:32.540 --> 00:26:37.198
let's see, six, so would
that be two Ds equals one?
00:26:37.198 --> 00:26:37.697
Ha.
00:26:41.440 --> 00:26:43.050
So it wasn't --
00:26:43.050 --> 00:26:45.760
I was afraid these numbers
were going to come out awful.
00:26:45.760 --> 00:26:48.770
But if I take two of
those away from that,
00:26:48.770 --> 00:26:51.480
the equation I get left
is two D equals one,
00:26:51.480 --> 00:26:57.700
so I think D is a
half and C is whatever
00:26:57.700 --> 00:27:03.650
back substitution gives, six D
is three, so three C plus three
00:27:03.650 --> 00:27:07.060
is five, I'm doing back
substitution now, right, three,
00:27:07.060 --> 00:27:10.910
can I do it in light
letters, three C plus
00:27:10.910 --> 00:27:15.970
that six D is three equals
five, so three C is two,
00:27:15.970 --> 00:27:17.440
so I think C is two-thirds.
00:27:23.276 --> 00:27:24.275
One-half and two-thirds.
00:27:29.230 --> 00:27:38.640
So the best line, the best
line is the constant two-thirds
00:27:38.640 --> 00:27:42.760
plus one-half t.
00:27:42.760 --> 00:27:46.820
And I -- is my picture
more or less right?
00:27:46.820 --> 00:27:49.890
Let me write, let me copy
that best line down again,
00:27:49.890 --> 00:27:52.600
two-thirds and a half.
00:27:52.600 --> 00:27:55.510
Let me -- I'll put in the
two-thirds and the half.
00:27:59.890 --> 00:28:00.710
OK.
00:28:00.710 --> 00:28:05.360
So what's this P1, that's
the value at t equal to one.
00:28:05.360 --> 00:28:08.380
At t equal to one, I have
two-thirds plus a half,
00:28:08.380 --> 00:28:10.280
which is --
00:28:10.280 --> 00:28:13.400
what's that, four-sixths
and three-sixths, so P1, oh,
00:28:13.400 --> 00:28:18.400
I promised not to write
another thing on this --
00:28:18.400 --> 00:28:21.860
I'll erase P1 and
I'll put seven-sixths.
00:28:21.860 --> 00:28:22.580
OK.
00:28:22.580 --> 00:28:27.990
And yeah, it's above one,
and e1 is one-sixth, right.
00:28:27.990 --> 00:28:28.720
You see it all.
00:28:28.720 --> 00:28:29.220
Right?
00:28:29.220 --> 00:28:29.830
What's P2?
00:28:29.830 --> 00:28:31.660
OK.
00:28:31.660 --> 00:28:35.230
At point t equal to two,
where's my line here?
00:28:35.230 --> 00:28:38.920
At t equal to two, it's
two-thirds plus one, right?
00:28:38.920 --> 00:28:41.580
That's five-thirds.
00:28:41.580 --> 00:28:44.130
Two-thirds and t is two,
so that's two-thirds
00:28:44.130 --> 00:28:46.070
and one make five-thirds.
00:28:46.070 --> 00:28:49.320
And that's -- sure enough,
that's smaller than the exact
00:28:49.320 --> 00:28:50.280
two.
00:28:50.280 --> 00:28:55.180
And then final P3, when
t is three, oh, what's
00:28:55.180 --> 00:28:56.820
two-thirds plus three-halves?
00:29:01.390 --> 00:29:03.950
It's the same as
three-halves plus two-thirds.
00:29:03.950 --> 00:29:09.280
It's -- so maybe
four-sixths and nine-sixths,
00:29:09.280 --> 00:29:11.120
maybe thirteen-sixths.
00:29:11.120 --> 00:29:15.110
OK, and again, look,
oh, look at this, OK.
00:29:15.110 --> 00:29:19.840
You have to admire the
beauty of this answer.
00:29:19.840 --> 00:29:21.260
What's this first error?
00:29:21.260 --> 00:29:25.760
So here are the
errors. e1, e2 and e3.
00:29:25.760 --> 00:29:28.340
OK, what was that
first error, e1?
00:29:28.340 --> 00:29:32.640
Well, if we decide the
errors counting up,
00:29:32.640 --> 00:29:35.260
then it's one-sixth.
00:29:35.260 --> 00:29:38.670
And the last error,
thirteen-sixths
00:29:38.670 --> 00:29:43.420
minus the correct two
is one-sixth again.
00:29:43.420 --> 00:29:47.890
And what's this
error in the middle?
00:29:47.890 --> 00:29:52.530
Let's see, the correct
answer was two, two.
00:29:52.530 --> 00:29:55.900
And we got five-thirds and
it's the other direction,
00:29:55.900 --> 00:29:58.445
minus one-third,
minus two-sixths.
00:30:02.070 --> 00:30:04.560
That's our error vector.
00:30:04.560 --> 00:30:09.220
In our picture, in our
other picture, here it is.
00:30:09.220 --> 00:30:13.880
We just found P and e.
00:30:13.880 --> 00:30:19.120
e is this vector, one-sixth,
minus two-sixths, one-sixth,
00:30:19.120 --> 00:30:21.540
and P is this guy.
00:30:21.540 --> 00:30:23.540
Well, maybe I have
the signs of e wrong,
00:30:23.540 --> 00:30:26.880
I think I have, let me fix it.
00:30:26.880 --> 00:30:32.840
Because I would like
this one-sixth --
00:30:32.840 --> 00:30:37.690
I would like this plus the
P to give the original b.
00:30:37.690 --> 00:30:42.730
I want P plus e to match b.
00:30:42.730 --> 00:30:47.080
So I want minus a
sixth, plus seven-sixths
00:30:47.080 --> 00:30:50.650
to give the correct b equal one.
00:30:50.650 --> 00:30:52.090
OK.
00:30:52.090 --> 00:30:58.060
Now -- I'm going to
take a deep breath here,
00:30:58.060 --> 00:31:06.720
and ask what do we know
about this error vector e?
00:31:06.720 --> 00:31:09.790
You've seen now this whole
problem worked completely
00:31:09.790 --> 00:31:13.780
through, and I even think
the numbers are right.
00:31:13.780 --> 00:31:17.500
So there's P, so let me --
00:31:17.500 --> 00:31:24.840
I'll write -- if I can put
it down here, B is P plus e.
00:31:24.840 --> 00:31:29.110
b I believe was one, two, two.
00:31:29.110 --> 00:31:34.860
The nearest point
had seven-sixths,
00:31:34.860 --> 00:31:36.120
what were the others?
00:31:36.120 --> 00:31:40.590
Five-thirds and thirteen-sixths.
00:31:40.590 --> 00:31:46.950
And the e vector was
minus a sixth, two-sixths,
00:31:46.950 --> 00:31:49.360
one-third in other
words, and minus a sixth.
00:31:58.511 --> 00:31:59.010
OK.
00:31:59.010 --> 00:32:01.930
Tell me some stuff
about these two vectors.
00:32:01.930 --> 00:32:03.820
Tell me something about
those two vectors,
00:32:03.820 --> 00:32:06.480
well, they add to
b, right, great.
00:32:06.480 --> 00:32:07.070
OK.
00:32:07.070 --> 00:32:09.420
What else?
00:32:09.420 --> 00:32:12.520
What else about those
two vectors, the P,
00:32:12.520 --> 00:32:18.700
the projection vector P,
and the error vector e.
00:32:18.700 --> 00:32:21.470
What else do you
know about them?
00:32:21.470 --> 00:32:24.430
They're perpendicular, right.
00:32:24.430 --> 00:32:25.860
Do we dare verify that?
00:32:29.180 --> 00:32:32.230
Can you take the dot
product of those vectors?
00:32:32.230 --> 00:32:35.440
I'm like getting like minus
seven over thirty-six,
00:32:35.440 --> 00:32:36.850
can I change that to ten-sixths?
00:32:42.180 --> 00:32:45.250
Oh, God, come out right here.
00:32:45.250 --> 00:32:50.880
Minus seven over thirty-six,
plus twenty over thirty-six,
00:32:50.880 --> 00:32:53.080
minus thirteen over thirty-six.
00:32:56.730 --> 00:32:57.630
Thank you, God.
00:32:57.630 --> 00:32:59.120
OK.
00:32:59.120 --> 00:33:04.030
And what else should we
know about that vector?
00:33:04.030 --> 00:33:05.740
Actually we know --
00:33:05.740 --> 00:33:08.740
I've got to say we know
even a little more.
00:33:08.740 --> 00:33:13.510
This vector, e, is
perpendicular to P,
00:33:13.510 --> 00:33:18.480
but it's perpendicular
to other stuff too.
00:33:18.480 --> 00:33:22.220
It's perpendicular not just to
this guy in the column space,
00:33:22.220 --> 00:33:25.170
this is in the column
space for sure.
00:33:25.170 --> 00:33:27.680
This is perpendicular
to the column space.
00:33:27.680 --> 00:33:32.710
So like give me another
vector it's perpendicular to.
00:33:32.710 --> 00:33:35.000
Another because it's
perpendicular to the whole
00:33:35.000 --> 00:33:37.490
column space, not
just to this --
00:33:37.490 --> 00:33:40.780
this particular
projection that's --
00:33:40.780 --> 00:33:44.880
that is in the column space,
but it's perpendicular to other
00:33:44.880 --> 00:33:46.520
stuff, whatever's
in the column space,
00:33:46.520 --> 00:33:49.800
so tell me another vector
in the -- oh, well,
00:33:49.800 --> 00:33:53.080
I've written down the matrix,
so tell me another vector
00:33:53.080 --> 00:33:55.000
in the column space.
00:33:55.000 --> 00:33:58.000
Pick a nice one.
00:33:58.000 --> 00:33:59.450
One, one, one.
00:33:59.450 --> 00:34:01.490
That's what
everybody's thinking.
00:34:01.490 --> 00:34:04.230
OK, one, one, one is
in the column space.
00:34:04.230 --> 00:34:07.350
And this guy is supposed
to be perpendicular to one,
00:34:07.350 --> 00:34:08.090
one, one.
00:34:08.090 --> 00:34:10.000
Is it?
00:34:10.000 --> 00:34:10.659
Sure.
00:34:10.659 --> 00:34:12.550
If I take the dot
product with one,
00:34:12.550 --> 00:34:16.690
one, one I get minus a sixth,
plus two-sixths, minus a sixth,
00:34:16.690 --> 00:34:18.080
zero.
00:34:18.080 --> 00:34:20.659
And it's perpendicular
to one, two, three.
00:34:20.659 --> 00:34:23.020
Because if I take the
dot product with one,
00:34:23.020 --> 00:34:30.310
two, three I get minus one, plus
four, minus three, zero again.
00:34:30.310 --> 00:34:32.449
OK, do you see the --
00:34:32.449 --> 00:34:35.739
I hope you see the two pictures.
00:34:35.739 --> 00:34:41.110
The picture here for vectors
and, the picture here
00:34:41.110 --> 00:34:48.120
for the best line, and it's
the same picture, just --
00:34:48.120 --> 00:34:51.440
this one's in the plane
and it's showing the line,
00:34:51.440 --> 00:34:56.060
this one never did show the
line, this -- in this picture,
00:34:56.060 --> 00:34:59.160
C and D never showed up.
00:34:59.160 --> 00:35:02.040
In this picture, C and
D were -- you know,
00:35:02.040 --> 00:35:04.730
they determined that line.
00:35:04.730 --> 00:35:07.020
But the two are
exactly the same.
00:35:07.020 --> 00:35:10.540
C and D is the combination
of the two columns
00:35:10.540 --> 00:35:14.770
that gives P. OK.
00:35:14.770 --> 00:35:19.890
So that's these squares.
00:35:19.890 --> 00:35:23.680
And the special
but most important
00:35:23.680 --> 00:35:26.820
example of fitting by
straight line, so the homework
00:35:26.820 --> 00:35:29.670
that's coming then
Wednesday asks
00:35:29.670 --> 00:35:32.750
you to fit by straight lines.
00:35:32.750 --> 00:35:40.440
So you're just going to end
up solving the key equation.
00:35:40.440 --> 00:35:42.850
You're going to end up
solving that key equation
00:35:42.850 --> 00:35:47.100
and then P will be Ax hat.
00:35:47.100 --> 00:35:47.870
That's it.
00:35:51.640 --> 00:35:54.470
OK.
00:35:54.470 --> 00:35:59.650
Now, can I put in a little
piece of linear algebra
00:35:59.650 --> 00:36:03.350
that I mentioned
earlier, mentioned again,
00:36:03.350 --> 00:36:06.510
but I never did write?
00:36:06.510 --> 00:36:09.840
And I've -- I
should do it right.
00:36:09.840 --> 00:36:16.070
It's about this matrix
A transpose A. There.
00:36:21.840 --> 00:36:26.450
I was sure that that
matrix would be invertible.
00:36:26.450 --> 00:36:29.220
And of course I wanted to
be sure it was invertible,
00:36:29.220 --> 00:36:36.210
because I planned to solve this
system with with that matrix.
00:36:36.210 --> 00:36:40.620
So and I announced
like before --
00:36:40.620 --> 00:36:42.660
as the chapter
was just starting,
00:36:42.660 --> 00:36:45.390
I announced that it
would be invertible.
00:36:45.390 --> 00:36:48.615
But now I -- can I
come back to that?
00:36:48.615 --> 00:36:49.115
OK.
00:36:52.940 --> 00:36:56.050
So what I said was --
00:36:56.050 --> 00:37:07.440
that if A has
independent columns,
00:37:07.440 --> 00:37:14.616
then A transpose
A is invertible.
00:37:20.100 --> 00:37:24.080
And I would like to --
00:37:24.080 --> 00:37:27.250
first to repeat
that important fact,
00:37:27.250 --> 00:37:32.320
that that's the requirement
that makes everything go here.
00:37:32.320 --> 00:37:34.610
It's this independent
columns of A
00:37:34.610 --> 00:37:39.050
that guarantees
everything goes through.
00:37:39.050 --> 00:37:42.140
And think why.
00:37:42.140 --> 00:37:44.970
Why does this matrix
A transpose A,
00:37:44.970 --> 00:37:50.410
why is it invertible if the
columns of A are independent?
00:37:50.410 --> 00:38:01.840
OK, there's -- so if it
wasn't invertible, I'm --
00:38:01.840 --> 00:38:04.750
so I want to prove that.
00:38:04.750 --> 00:38:08.060
If it isn't
invertible, then what?
00:38:08.060 --> 00:38:10.610
I want to reach --
00:38:10.610 --> 00:38:13.010
I want to follow that
-- follow that line --
00:38:13.010 --> 00:38:15.400
of thinking and
see what I come to.
00:38:15.400 --> 00:38:17.480
Suppose, so proof.
00:38:17.480 --> 00:38:26.810
Suppose A transpose Ax is zero.
00:38:26.810 --> 00:38:28.400
I'm trying to prove this.
00:38:28.400 --> 00:38:30.440
This is now to prove.
00:38:30.440 --> 00:38:39.910
I don't like hammer away at
too many proofs in this course.
00:38:39.910 --> 00:38:41.690
But this is like
the central fact
00:38:41.690 --> 00:38:44.320
and it brings in all
the stuff we know.
00:38:44.320 --> 00:38:44.820
OK.
00:38:44.820 --> 00:38:46.700
So I'll start the proof.
00:38:46.700 --> 00:38:51.160
Suppose A transpose Ax is zero.
00:38:51.160 --> 00:38:56.110
What -- and I'm aiming to prove
A transpose A is invertible.
00:38:56.110 --> 00:38:58.150
So what do I want to prove now?
00:39:00.680 --> 00:39:03.560
So I'm aiming to
prove this fact.
00:39:03.560 --> 00:39:06.680
I'll use this, and I'm aiming
to prove that this matrix is
00:39:06.680 --> 00:39:11.740
invertible, OK, so if I
suppose A transpose Ax is zero,
00:39:11.740 --> 00:39:13.875
then what conclusion
do I want to reach?
00:39:16.450 --> 00:39:21.200
I'd like to know
that x must be zero.
00:39:21.200 --> 00:39:23.510
I want to show x must be zero.
00:39:23.510 --> 00:39:33.100
To show now -- to prove x
must be the zero vector.
00:39:33.100 --> 00:39:38.640
Is that right, that's what we
worked in the previous chapter
00:39:38.640 --> 00:39:43.850
to understand, that a
matrix was invertible
00:39:43.850 --> 00:39:51.960
when its null space is
only the zero vector.
00:39:51.960 --> 00:39:53.340
So that's what I want to show.
00:39:53.340 --> 00:40:00.520
How come if A transpose Ax is
zero, how come x must be zero?
00:40:00.520 --> 00:40:01.810
What's going to be the reason?
00:40:05.270 --> 00:40:06.880
Actually I have
two ways to do it.
00:40:10.270 --> 00:40:12.210
Let me show you one way.
00:40:12.210 --> 00:40:14.640
This is -- here, trick.
00:40:18.210 --> 00:40:22.880
Take the dot product
of both sides with x.
00:40:22.880 --> 00:40:25.980
So I'll multiply both
sides by x transpose.
00:40:25.980 --> 00:40:30.100
x transpose A transpose
Ax equals zero.
00:40:33.190 --> 00:40:35.100
I shouldn't have written trick.
00:40:35.100 --> 00:40:37.640
That makes it sound
like just a dumb idea.
00:40:37.640 --> 00:40:39.581
Brilliant idea, I
should have put.
00:40:39.581 --> 00:40:40.080
OK.
00:40:43.040 --> 00:40:44.215
I'll just put idea.
00:40:47.920 --> 00:40:49.230
OK.
00:40:49.230 --> 00:40:57.670
Now, I got to that equation,
x transpose A transpose Ax=0,
00:40:57.670 --> 00:41:06.229
and I'm hoping you can
see the right way to --
00:41:06.229 --> 00:41:07.270
to look at that equation.
00:41:12.760 --> 00:41:15.030
What can I conclude
from that equation,
00:41:15.030 --> 00:41:17.840
that if I have x
transpose A -- well,
00:41:17.840 --> 00:41:21.070
what is x transpose
A transpose Ax?
00:41:21.070 --> 00:41:25.360
Does that -- what
it's giving you?
00:41:29.620 --> 00:41:32.740
It's again going to be putting
in parentheses, I'm looking
00:41:32.740 --> 00:41:37.170
at Ax and what I seeing here?
00:41:37.170 --> 00:41:39.560
Its transpose.
00:41:39.560 --> 00:41:47.580
So I'm seeing here this
is Ax transpose Ax.
00:41:47.580 --> 00:41:48.325
Equaling zero.
00:41:51.640 --> 00:41:57.040
Now if Ax transpose Ax, so like
let's call it y or something,
00:41:57.040 --> 00:42:01.450
if y transpose y is zero,
what does that tell me?
00:42:06.780 --> 00:42:08.950
That the vector has
to be zero, right?
00:42:08.950 --> 00:42:10.650
This is the length
squared, that's
00:42:10.650 --> 00:42:15.730
the length of the vector Ax
squared, that's Ax times Ax.
00:42:15.730 --> 00:42:18.210
So I conclude that
Ax has to be zero.
00:42:23.474 --> 00:42:24.640
Well, I'm getting somewhere.
00:42:29.900 --> 00:42:34.610
Now that I know Ax
is zero, now I'm
00:42:34.610 --> 00:42:37.370
going to use my
little hypothesis.
00:42:37.370 --> 00:42:43.290
Somewhere every mathematician
has to use the hypothesis.
00:42:43.290 --> 00:42:45.050
Right?
00:42:45.050 --> 00:42:49.740
Now, if A has independent
columns and we've --
00:42:49.740 --> 00:42:55.580
we're at the point where Ax is
zero, what does that tell us?
00:42:55.580 --> 00:42:59.610
I could -- I mean that could
be like a fill-in question
00:42:59.610 --> 00:43:01.090
on the final exam.
00:43:01.090 --> 00:43:06.820
If A has independent columns
and if Ax equals zero then what?
00:43:10.390 --> 00:43:15.850
Please say it. x is zero, right.
00:43:15.850 --> 00:43:18.370
Which was just what
we wanted to prove.
00:43:18.370 --> 00:43:20.790
That -- do you see why that is?
00:43:20.790 --> 00:43:24.150
If Ax eq- equals zero,
now we're using --
00:43:24.150 --> 00:43:27.190
here we used this was
the square of something,
00:43:27.190 --> 00:43:30.810
so I'll put in
little parentheses
00:43:30.810 --> 00:43:35.720
the observation we made, that
was a square which is zero,
00:43:35.720 --> 00:43:37.610
so the thing has to be zero.
00:43:37.610 --> 00:43:43.130
Now we're using the hypothesis
of independent columns
00:43:43.130 --> 00:43:48.600
at the A has
independent columns.
00:43:48.600 --> 00:43:52.060
If A has independent
columns, this is telling me
00:43:52.060 --> 00:43:56.040
x is in its null space,
and the only thing
00:43:56.040 --> 00:44:00.510
in the null space of such a
matrix is the zero vector.
00:44:00.510 --> 00:44:01.320
OK.
00:44:01.320 --> 00:44:06.620
So that's the argument and
you see how it really used
00:44:06.620 --> 00:44:13.420
our understanding of the
-- of the null space.
00:44:13.420 --> 00:44:13.990
OK.
00:44:13.990 --> 00:44:15.650
That's great.
00:44:15.650 --> 00:44:16.420
All right.
00:44:16.420 --> 00:44:20.750
So where are we then?
00:44:20.750 --> 00:44:24.430
That board is like
the backup theory
00:44:24.430 --> 00:44:28.670
that tells me that
this matrix had
00:44:28.670 --> 00:44:32.610
to be invertible because these
columns were independent.
00:44:35.530 --> 00:44:38.360
OK.
00:44:38.360 --> 00:44:44.940
there's one case
of independent --
00:44:44.940 --> 00:44:50.540
there's one case where the
geometry gets even better.
00:44:50.540 --> 00:44:55.030
When the -- there's one case
when columns are sure to be
00:44:55.030 --> 00:44:56.610
independent.
00:44:56.610 --> 00:45:00.060
And let me put that -- let me
write that down and that'll be
00:45:00.060 --> 00:45:01.780
the subject for next time.
00:45:01.780 --> 00:45:07.040
Columns are sure -- are
certainly independent,
00:45:07.040 --> 00:45:23.290
definitely independent,
if they're perpendicular.
00:45:23.290 --> 00:45:25.190
Oh, I've got to rule
out the zero column,
00:45:25.190 --> 00:45:33.280
let me give them all length one,
so they can't be zero if they
00:45:33.280 --> 00:45:37.855
are perpendicular unit vectors.
00:45:42.870 --> 00:45:53.370
Like the vectors one, zero,
zero, zero, one, zero and zero,
00:45:53.370 --> 00:45:55.480
zero, one.
00:45:55.480 --> 00:46:00.660
Those vectors are unit
vectors, they're perpendicular,
00:46:00.660 --> 00:46:05.820
and they certainly
are independent.
00:46:05.820 --> 00:46:10.610
And what's more, suppose
they're -- oh, that's so nice,
00:46:10.610 --> 00:46:14.080
I mean what is A transpose
A for that matrix?
00:46:14.080 --> 00:46:16.470
For the matrix with
these three columns?
00:46:16.470 --> 00:46:18.280
It's the identity.
00:46:18.280 --> 00:46:23.090
So here's the key to the
lecture that's coming.
00:46:23.090 --> 00:46:27.210
If we're dealing with
perpendicular unit vectors
00:46:27.210 --> 00:46:32.000
and the word for that will
be -- see I could have said
00:46:32.000 --> 00:46:35.650
orthogonal, but I
said perpendicular --
00:46:35.650 --> 00:46:41.370
and this unit vectors gets
put in as the word normal.
00:46:41.370 --> 00:46:42.545
Orthonormal vectors.
00:46:46.070 --> 00:46:49.820
Those are the best
columns you could ask for.
00:46:49.820 --> 00:46:54.730
Matrices with -- whose
columns are orthonormal,
00:46:54.730 --> 00:46:56.950
they're perpendicular
to each other,
00:46:56.950 --> 00:47:01.010
and they're unit vectors, well,
they don't have to be those
00:47:01.010 --> 00:47:06.280
three, let me do a
final example over here,
00:47:06.280 --> 00:47:11.110
how about one at an angle like
that and one at ninety degrees,
00:47:11.110 --> 00:47:18.050
that vector would be cos theta,
sine theta, a unit vector,
00:47:18.050 --> 00:47:24.150
and this vector would be
minus sine theta cos theta.
00:47:24.150 --> 00:47:30.850
That is our absolute favorite
pair of orthonormal vectors.
00:47:30.850 --> 00:47:33.630
They're both unit vectors
and they're perpendicular.
00:47:33.630 --> 00:47:36.520
That angle is ninety degrees.
00:47:36.520 --> 00:47:41.500
So like our job next
time is first to see
00:47:41.500 --> 00:47:43.640
why orthonormal
vectors are great,
00:47:43.640 --> 00:47:47.240
and then to make vectors
orthonormal by picking
00:47:47.240 --> 00:47:49.530
the right basis.
00:47:49.530 --> 00:47:50.625
OK, see you.
00:47:57.070 --> 00:47:58.620
Thanks.