WEBVTT
00:00:14.160 --> 00:00:14.860
Okay.
00:00:14.860 --> 00:00:21.760
This is the lecture on the
singular value decomposition.
00:00:21.760 --> 00:00:27.030
But everybody calls it the SVD.
00:00:27.030 --> 00:00:34.350
So this is the final and best
factorization of a matrix.
00:00:34.350 --> 00:00:37.270
Let me tell you what's coming.
00:00:37.270 --> 00:00:43.330
The factors will be, orthogonal
matrix, diagonal matrix,
00:00:43.330 --> 00:00:45.610
orthogonal matrix.
00:00:45.610 --> 00:00:48.610
So it's things that
we've seen before,
00:00:48.610 --> 00:00:53.180
these special good matrices,
orthogonal diagonal.
00:00:53.180 --> 00:00:58.810
The new point is that we
need two orthogonal matrices.
00:00:58.810 --> 00:01:02.720
A can be any matrix whatsoever.
00:01:02.720 --> 00:01:07.140
Any matrix whatsoever has this
singular value decomposition,
00:01:07.140 --> 00:01:11.870
so a diagonal one in the middle,
but I need two different --
00:01:11.870 --> 00:01:17.030
probably different orthogonal
matrices to be able to do this.
00:01:17.030 --> 00:01:17.560
Okay.
00:01:17.560 --> 00:01:22.030
And this factorization
has jumped into importance
00:01:22.030 --> 00:01:27.240
and is properly, I think,
maybe the bringing together
00:01:27.240 --> 00:01:29.620
of everything in this course.
00:01:29.620 --> 00:01:36.970
One thing we'll bring together
is the very good family
00:01:36.970 --> 00:01:39.070
of matrices that
we just studied,
00:01:39.070 --> 00:01:41.120
symmetric, positive, definite.
00:01:41.120 --> 00:01:43.730
Do you remember the
stories with those guys?
00:01:43.730 --> 00:01:49.160
Because they were symmetric,
their eigenvectors were
00:01:49.160 --> 00:01:53.880
orthogonal, so I could produce
an orthogonal matrix --
00:01:53.880 --> 00:01:56.070
this is my usual one.
00:01:56.070 --> 00:01:59.630
My usual one is the
eigenvectors and eigenvalues In
00:01:59.630 --> 00:02:05.540
the symmetric case, the
eigenvectors are orthogonal,
00:02:05.540 --> 00:02:09.430
so I've got the good --
my ordinary s has become
00:02:09.430 --> 00:02:12.180
an especially good Q.
00:02:12.180 --> 00:02:15.610
And positive definite,
my ordinary lambda
00:02:15.610 --> 00:02:18.260
has become a positive lambda.
00:02:18.260 --> 00:02:24.420
So that's the singular
value decomposition in case
00:02:24.420 --> 00:02:28.240
our matrix is symmetric
positive definite --
00:02:28.240 --> 00:02:30.650
in that case, I
don't need two --
00:02:30.650 --> 00:02:35.795
U and a V -- one orthogonal
matrix will do for both sides.
00:02:38.970 --> 00:02:41.260
So this would be
no good in general,
00:02:41.260 --> 00:02:45.190
because usually the eigenvector
matrix isn't orthogonal.
00:02:45.190 --> 00:02:49.900
So that's not what I'm after.
00:02:49.900 --> 00:02:56.790
I'm looking for orthogonal
times diagonal times orthogonal.
00:02:56.790 --> 00:03:00.250
And let me show you what that
means and where it comes from.
00:03:00.250 --> 00:03:01.880
Okay.
00:03:01.880 --> 00:03:02.680
What does it mean?
00:03:05.440 --> 00:03:10.310
You remember the picture of
any linear transformation.
00:03:10.310 --> 00:03:16.670
This was, like, the
most important figure in
00:03:16.670 --> 00:03:19.810
And what I looking for now?
00:03:19.810 --> 00:03:26.020
A typical vector
in the row space --
00:03:26.020 --> 00:03:28.940
typical vector,
let me call it v1,
00:03:28.940 --> 00:03:35.470
gets taken over to some vector
in the column space, say u1.
00:03:35.470 --> 00:03:37.850
So u1 is Av1.
00:03:41.350 --> 00:03:43.320
Okay.
00:03:43.320 --> 00:03:48.800
Now, another vector gets
taken over here somewhere.
00:03:48.800 --> 00:03:50.290
What I looking for?
00:03:50.290 --> 00:03:54.590
In this SVD, this singular
value decomposition,
00:03:54.590 --> 00:03:59.950
what I'm looking for is
an orthogonal basis here
00:03:59.950 --> 00:04:06.130
that gets knocked over into an
orthogonal basis over there.
00:04:06.130 --> 00:04:11.810
See that's pretty special,
to have an orthogonal basis
00:04:11.810 --> 00:04:18.180
in the row space that goes over
into an orthogonal basis --
00:04:18.180 --> 00:04:21.490
so this is like a right angle
and this is a right angle --
00:04:21.490 --> 00:04:25.700
into an orthogonal basis
in the column space.
00:04:25.700 --> 00:04:30.940
So that's our
goal, is to find --
00:04:30.940 --> 00:04:34.230
do you see how things
are coming together?
00:04:34.230 --> 00:04:37.590
First of all, can I
find an orthogonal basis
00:04:37.590 --> 00:04:39.960
for this row space?
00:04:39.960 --> 00:04:41.100
Of course.
00:04:41.100 --> 00:04:44.210
No big deal to find
an orthogonal basis.
00:04:44.210 --> 00:04:46.870
Graham Schmidt tells
me how to do it.
00:04:46.870 --> 00:04:50.710
Start with any old basis and
grind through Graham Schmidt,
00:04:50.710 --> 00:04:54.100
out comes an orthogonal basis.
00:04:54.100 --> 00:04:58.280
But then, if I just take any
old orthogonal basis, then
00:04:58.280 --> 00:05:00.990
when I multiply by
A, there's no reason
00:05:00.990 --> 00:05:04.770
why it should be
orthogonal over here.
00:05:04.770 --> 00:05:06.800
So I'm looking for
this special set
00:05:06.800 --> 00:05:13.850
up where A takes these basis
vectors into orthogonal vectors
00:05:13.850 --> 00:05:14.780
over there.
00:05:14.780 --> 00:05:18.570
Now, you might have
noticed that the null space
00:05:18.570 --> 00:05:19.400
I didn't include.
00:05:19.400 --> 00:05:22.400
Why don't I stick that in?
00:05:22.400 --> 00:05:25.770
You remember our usual figure
had a little null space
00:05:25.770 --> 00:05:28.730
and a little null space.
00:05:28.730 --> 00:05:31.650
And those are no problems.
00:05:31.650 --> 00:05:34.000
Those null spaces
are going to show up
00:05:34.000 --> 00:05:37.510
as zeroes on the
diagonal of sigma,
00:05:37.510 --> 00:05:43.730
so that doesn't
present any difficulty.
00:05:43.730 --> 00:05:46.250
Our difficulty is to find these.
00:05:46.250 --> 00:05:48.220
So do you see what
this will mean?
00:05:48.220 --> 00:05:57.950
This will mean that A times
these v-s, v1, v2, up to --
00:05:57.950 --> 00:06:01.460
what's the dimension
of this row space?
00:06:01.460 --> 00:06:03.180
Vr.
00:06:03.180 --> 00:06:07.390
Sorry, make that V a
little smaller -- up to vr.
00:06:10.650 --> 00:06:12.990
So that's --
00:06:12.990 --> 00:06:16.680
Av1 is going to be
the first column,
00:06:16.680 --> 00:06:20.250
so here's what I'm achieving.
00:06:20.250 --> 00:06:24.390
Oh, I'm not only going
to make these orthogonal,
00:06:24.390 --> 00:06:27.030
but why not make
them orthonormal?
00:06:27.030 --> 00:06:28.880
Make them unit vectors.
00:06:28.880 --> 00:06:34.380
So maybe the unit vector
is here, is the u1,
00:06:34.380 --> 00:06:38.510
and this might be
a multiple of it.
00:06:38.510 --> 00:06:45.560
So really, what's happening
is Av1 is some multiple of u1,
00:06:45.560 --> 00:06:46.660
right?
00:06:46.660 --> 00:06:50.040
These guys will be unit
vectors and they'll
00:06:50.040 --> 00:06:52.940
go over into multiples
of unit vectors
00:06:52.940 --> 00:06:56.820
and the multiple I'm not
going to call lambda anymore.
00:06:56.820 --> 00:06:58.090
I'm calling it sigma.
00:06:58.090 --> 00:07:01.480
So that's the number --
the stretching number.
00:07:01.480 --> 00:07:06.000
And similarly, Av2
is sigma two u2.
00:07:06.000 --> 00:07:09.000
This is my goal.
00:07:09.000 --> 00:07:13.730
And now I want to express
that goal in matrix language.
00:07:13.730 --> 00:07:15.340
That's the usual step.
00:07:15.340 --> 00:07:18.400
Think of what you want
and then express it
00:07:18.400 --> 00:07:20.420
as a matrix multiplication.
00:07:20.420 --> 00:07:25.800
So Av1 is sigma one u1 --
00:07:25.800 --> 00:07:27.960
actually, here we go.
00:07:27.960 --> 00:07:30.710
Let me pull out these --
00:07:30.710 --> 00:07:36.765
u1, u2 to ur and then a
matrix with the sigmas.
00:07:40.460 --> 00:07:45.970
Everything now is going
to be in that little part
00:07:45.970 --> 00:07:46.720
of the blackboard.
00:07:46.720 --> 00:07:52.880
Do you see that this
equation says what I'm
00:07:52.880 --> 00:07:55.740
trying to do with my figure.
00:07:55.740 --> 00:08:00.770
A times the first basis vector
should be sigma one times
00:08:00.770 --> 00:08:04.580
the other basis -- the
other first basis vector.
00:08:04.580 --> 00:08:08.150
These are the basis
vectors in the row space,
00:08:08.150 --> 00:08:11.470
these are the basis
vectors in the column space
00:08:11.470 --> 00:08:14.680
and these are the
multiplying factors.
00:08:14.680 --> 00:08:23.660
So Av2 is sigma two times
u2, Avr is sigma r times ur.
00:08:23.660 --> 00:08:27.170
And then we've got a whole lot
of zeroes and maybe some zeroes
00:08:27.170 --> 00:08:31.480
at the end, but that's
the heart of it.
00:08:31.480 --> 00:08:34.064
And now if I express that in --
00:08:40.000 --> 00:08:43.350
as matrices, because you
knew that was coming --
00:08:43.350 --> 00:08:45.140
that's what I have.
00:08:45.140 --> 00:08:51.520
So, this is my goal, to
find an orthogonal basis
00:08:51.520 --> 00:08:55.500
in the orthonormal, even
-- basis in the row space
00:08:55.500 --> 00:09:00.420
and an orthonormal basis in the
column space so that I've sort
00:09:00.420 --> 00:09:02.590
of diagonalized the matrix.
00:09:02.590 --> 00:09:05.530
The matrix A is, like,
getting converted
00:09:05.530 --> 00:09:08.660
to this diagonal matrix sigma.
00:09:08.660 --> 00:09:13.140
And you notice that usually
I have to allow myself
00:09:13.140 --> 00:09:15.980
two different bases.
00:09:15.980 --> 00:09:19.230
My little comment about
symmetric positive definite
00:09:19.230 --> 00:09:24.860
was the one case where
it's A Q equal Q sigma,
00:09:24.860 --> 00:09:27.430
where V and U are the same Q.
00:09:27.430 --> 00:09:32.310
But mostly, you know, I'm going
to take a matrix like -- oh,
00:09:32.310 --> 00:09:38.240
let me take a matrix like
four four minus three three.
00:09:38.240 --> 00:09:39.540
Okay.
00:09:39.540 --> 00:09:42.420
There's a two by two matrix.
00:09:42.420 --> 00:09:48.430
It's invertible,
so it has rank two.
00:09:48.430 --> 00:09:51.210
So I'm going to look
for two vectors,
00:09:51.210 --> 00:09:55.230
v1 and v2 in the
row space, and U --
00:09:55.230 --> 00:10:00.770
so I'm going to look for
v1, v2 in the row space,
00:10:00.770 --> 00:10:05.350
which of course is R^2.
00:10:05.350 --> 00:10:10.280
And I'm going to look for
u1, u2 in the column space,
00:10:10.280 --> 00:10:15.620
which of course is also R^2, and
I'm going to look for numbers
00:10:15.620 --> 00:10:21.160
sigma one and sigma two so
that it all comes out right.
00:10:21.160 --> 00:10:26.060
So these guys are orthonormal,
these guys are orthonormal
00:10:26.060 --> 00:10:28.820
and these are the
scaling factors.
00:10:28.820 --> 00:10:33.510
So I'll do that example as
soon as I get the matrix
00:10:33.510 --> 00:10:35.030
picture straight.
00:10:35.030 --> 00:10:36.280
Okay.
00:10:36.280 --> 00:10:39.380
Do you see that this
expresses what I want?
00:10:39.380 --> 00:10:45.360
Can I just say two
words about null spaces?
00:10:45.360 --> 00:10:52.010
If there's some
null space, then we
00:10:52.010 --> 00:10:56.080
want to stick in a basis
for those, for that.
00:10:56.080 --> 00:11:02.700
So here comes a basis for the
null space, v(r+1) down to vm.
00:11:02.700 --> 00:11:06.590
So if we only had an r
dimensional row space
00:11:06.590 --> 00:11:11.160
and the other n-r dimensions
were in the null space -- okay,
00:11:11.160 --> 00:11:14.360
we'll take an orthogonal
-- orthonormal basis there.
00:11:14.360 --> 00:11:15.430
No problem.
00:11:15.430 --> 00:11:18.500
And then we'll just get zeroes.
00:11:18.500 --> 00:11:21.700
So, actually, w- those
zeroes will come out
00:11:21.700 --> 00:11:25.040
on the diagonal matrix.
00:11:25.040 --> 00:11:33.250
So I'll complete that to an
orthonormal basis for the whole
00:11:33.250 --> 00:11:35.420
space, R^m.
00:11:35.420 --> 00:11:38.080
I complete this to an
orthonormal basis for the whole
00:11:38.080 --> 00:11:42.420
space R^n and I complete
that with zeroes.
00:11:42.420 --> 00:11:46.870
Null spaces are no problem here.
00:11:46.870 --> 00:11:52.570
So really the true problem
is in a matrix like that,
00:11:52.570 --> 00:11:56.170
which isn't symmetric, so I
can't use its eigenvectors,
00:11:56.170 --> 00:11:58.110
they're not orthogonal --
00:11:58.110 --> 00:12:02.740
but somehow I have to get
these orthogonal -- in fact,
00:12:02.740 --> 00:12:09.470
orthonormal guys
that make it work.
00:12:09.470 --> 00:12:13.990
I have to find these orthonormal
guys, these orthonormal guys
00:12:13.990 --> 00:12:22.960
and I want Av1 to be sigma one
u1 and Av2 to be sigma two u2.
00:12:22.960 --> 00:12:23.460
Okay.
00:12:26.790 --> 00:12:28.260
That's my goal.
00:12:28.260 --> 00:12:34.310
Here's the matrices that
are going to get me there.
00:12:34.310 --> 00:12:36.500
Now these are
orthogonal matrices.
00:12:36.500 --> 00:12:41.610
I can put that -- if I multiply
on both sides by V inverse,
00:12:41.610 --> 00:12:47.740
I have A equals U
sigma V inverse,
00:12:47.740 --> 00:12:53.230
and of course you know the
other way I can write V inverse.
00:12:53.230 --> 00:12:57.200
This is one of those
square orthogonal matrices,
00:12:57.200 --> 00:13:02.731
so it's the same as
U sigma V transpose.
00:13:02.731 --> 00:13:03.230
Okay.
00:13:06.020 --> 00:13:08.900
Here's my problem.
00:13:08.900 --> 00:13:14.580
I've got two orthogonal
matrices here.
00:13:14.580 --> 00:13:17.330
And I don't want to
find them both at once.
00:13:17.330 --> 00:13:21.120
So I want to cook up
some expression that
00:13:21.120 --> 00:13:26.630
will make the Us disappear.
00:13:26.630 --> 00:13:29.010
I would like to make the
Us disappear and leave me
00:13:29.010 --> 00:13:31.070
only with the Vs.
00:13:31.070 --> 00:13:34.200
And here's how to do it.
00:13:34.200 --> 00:13:37.520
It's the same combination
that keeps showing up
00:13:37.520 --> 00:13:41.040
whenever we have a general
rectangular matrix,
00:13:41.040 --> 00:13:46.640
then it's A transpose A,
that's the great matrix.
00:13:46.640 --> 00:13:48.760
That's the great matrix.
00:13:48.760 --> 00:13:50.530
That's the matrix
that's symmetric,
00:13:50.530 --> 00:13:53.910
and in fact positive
definite or at least
00:13:53.910 --> 00:13:55.240
positive semi-definite.
00:13:55.240 --> 00:13:58.300
This is the matrix with nice
properties, so let's see what
00:13:58.300 --> 00:13:59.230
will it be?
00:13:59.230 --> 00:14:03.100
So if I took the transpose,
then, I would have --
00:14:03.100 --> 00:14:06.140
A transpose A will be what?
00:14:06.140 --> 00:14:06.920
What do I have?
00:14:06.920 --> 00:14:12.580
If I transpose that I have V
sigma transpose U transpose,
00:14:12.580 --> 00:14:14.900
that's the A transpose.
00:14:14.900 --> 00:14:18.350
Now the A --
00:14:18.350 --> 00:14:19.290
and what have I got?
00:14:21.930 --> 00:14:25.960
Looks like worse, because it's
got six things now together,
00:14:25.960 --> 00:14:30.660
but it's going to collapse
into something good.
00:14:30.660 --> 00:14:34.300
What does U transpose
U collapse into?
00:14:34.300 --> 00:14:35.770
I, the identity.
00:14:35.770 --> 00:14:37.330
So that's the key point.
00:14:37.330 --> 00:14:42.050
This is the identity and
we don't have U anymore.
00:14:42.050 --> 00:14:44.380
And sigma transpose
times sigma, those
00:14:44.380 --> 00:14:48.300
are diagonal matrixes,
so their product is just
00:14:48.300 --> 00:14:50.590
going to have sigma
squareds on the diagonal.
00:14:50.590 --> 00:14:52.650
So do you see what
we've got here?
00:14:52.650 --> 00:14:57.440
This is V times this
easy matrix sigma
00:14:57.440 --> 00:15:03.080
one squared sigma two
squared times V transpose.
00:15:03.080 --> 00:15:05.840
This is the A transpose A.
00:15:05.840 --> 00:15:07.590
This is -- let me copy down --
00:15:07.590 --> 00:15:09.185
A transpose A is that.
00:15:12.760 --> 00:15:14.170
Us are out of the picture, now.
00:15:14.170 --> 00:15:17.900
I'm only having to choose the
Vs, and what are these Vs?
00:15:17.900 --> 00:15:19.370
And what are these sigmas?
00:15:19.370 --> 00:15:26.750
Do you know what the Vs are?
00:15:26.750 --> 00:15:28.650
They're the eigenvectors
that -- see,
00:15:28.650 --> 00:15:34.120
this is a perfect
eigenvector, eigenvalue,
00:15:34.120 --> 00:15:41.920
Q lambda Q transpose for
the matrix A transpose A.
00:15:41.920 --> 00:15:45.020
A itself is nothing special.
00:15:45.020 --> 00:15:48.620
But A transpose A
will be special.
00:15:48.620 --> 00:15:50.700
It'll be symmetric
positive definite,
00:15:50.700 --> 00:15:55.340
so this will be its eigenvectors
and this'll be its eigenvalues.
00:15:55.340 --> 00:15:59.100
And the eigenvalues'll be
positive because this thing's
00:15:59.100 --> 00:16:02.400
positive definite.
00:16:02.400 --> 00:16:04.650
So this is my method.
00:16:04.650 --> 00:16:07.310
This tells me what the Vs are.
00:16:07.310 --> 00:16:09.210
And how I going to find the Us?
00:16:11.730 --> 00:16:18.570
Well, one way would be
to look at A A transpose.
00:16:18.570 --> 00:16:21.650
Multiply A by A transpose
in the opposite order.
00:16:21.650 --> 00:16:24.340
That will stick the
Vs in the middle,
00:16:24.340 --> 00:16:27.130
knock them out, and
leave me with the Us.
00:16:27.130 --> 00:16:29.620
So here's the overall
picture, then.
00:16:29.620 --> 00:16:36.270
The Vs are the eigenvectors
of A transpose A.
00:16:36.270 --> 00:16:38.180
The Us are the
eigenvectors of A A
00:16:38.180 --> 00:16:39.860
transpose, which are different.
00:16:39.860 --> 00:16:43.940
And the sigmas are
the square roots
00:16:43.940 --> 00:16:48.730
of these and the
positive square roots,
00:16:48.730 --> 00:16:50.580
so we have positive sigmas.
00:16:50.580 --> 00:16:52.780
Let me do it for that example.
00:16:52.780 --> 00:16:56.570
This is really what
you should know
00:16:56.570 --> 00:17:01.460
and be able to do for the SVD.
00:17:01.460 --> 00:17:02.520
Okay.
00:17:02.520 --> 00:17:03.740
Let me take that matrix.
00:17:03.740 --> 00:17:06.550
So what's my first step?
00:17:06.550 --> 00:17:12.191
Compute A transpose A, because
I want its eigenvectors.
00:17:12.191 --> 00:17:12.690
Okay.
00:17:12.690 --> 00:17:16.690
So I have to compute
A transpose A.
00:17:16.690 --> 00:17:22.240
So A transpose is four
four minus three three,
00:17:22.240 --> 00:17:26.970
and A is four four
minus three three,
00:17:26.970 --> 00:17:30.810
and I do that multiplication
and I get sixteen --
00:17:30.810 --> 00:17:32.930
I get twenty five --
00:17:32.930 --> 00:17:36.300
I get sixteen minus nine --
00:17:36.300 --> 00:17:37.870
is that seven?
00:17:37.870 --> 00:17:40.260
And it better come
out symmetric.
00:17:40.260 --> 00:17:43.190
And -- oh, okay, and
then it comes out 25.
00:17:43.190 --> 00:17:43.690
Okay.
00:17:47.360 --> 00:17:52.030
So, I want its eigenvectors
and its eigenvalues.
00:17:52.030 --> 00:17:55.680
Its eigenvectors will be
the Vs, its eigenvalues
00:17:55.680 --> 00:17:58.990
will be the squares
of the sigmas.
00:17:58.990 --> 00:17:59.570
Okay.
00:17:59.570 --> 00:18:04.900
What are the eigenvalues and
eigenvectors of this guy?
00:18:04.900 --> 00:18:10.690
Have you seen that two by two
example enough to recognize
00:18:10.690 --> 00:18:16.970
that the eigenvectors are --
that one one is an eigenvector?
00:18:16.970 --> 00:18:19.210
So this here is A transpose A.
00:18:19.210 --> 00:18:22.300
I'm looking for
its eigenvectors.
00:18:22.300 --> 00:18:27.620
So its eigenvectors, I think,
are one one and one minus one,
00:18:27.620 --> 00:18:29.800
because if I
multiply that matrix
00:18:29.800 --> 00:18:32.390
by one one, what do I get?
00:18:32.390 --> 00:18:37.720
If I multiply that matrix
by one one, I get 32 32,
00:18:37.720 --> 00:18:41.530
which is 32 of one one.
00:18:41.530 --> 00:18:45.240
So there's the
first eigenvector,
00:18:45.240 --> 00:18:49.360
and there's the eigenvalue
for A transpose A.
00:18:49.360 --> 00:18:59.120
So I'm going to take its
square root for sigma.
00:18:59.120 --> 00:18:59.620
Okay.
00:18:59.620 --> 00:19:01.310
What's the eigenvector
that goes --
00:19:01.310 --> 00:19:03.410
eigenvalue that
goes with this one?
00:19:03.410 --> 00:19:05.730
If I do that multiplication,
what do I get?
00:19:05.730 --> 00:19:12.420
I get some multiple of one minus
one, and what is that multiple?
00:19:12.420 --> 00:19:13.250
Looks like 18.
00:19:16.470 --> 00:19:17.340
Okay.
00:19:17.340 --> 00:19:20.880
So those are the two
eigenvectors, but -- oh,
00:19:20.880 --> 00:19:24.300
just a moment, I
didn't normalize them.
00:19:24.300 --> 00:19:27.310
To make everything
absolutely right,
00:19:27.310 --> 00:19:30.180
I ought to normalize
these eigenvectors,
00:19:30.180 --> 00:19:33.370
divide by their length,
square root of two.
00:19:33.370 --> 00:19:42.890
So all these guys should be true
unit vectors and, of course,
00:19:42.890 --> 00:19:46.900
that normalization didn't
change the 32 and the 18.
00:19:46.900 --> 00:19:48.500
Okay.
00:19:48.500 --> 00:19:51.980
So I'm happy with the Vs.
00:19:51.980 --> 00:19:53.380
Here are the Vs.
00:19:53.380 --> 00:19:57.490
So now let me put
together the pieces here.
00:19:57.490 --> 00:19:59.340
Here's my A.
00:19:59.340 --> 00:20:01.060
Here's my A.
00:20:01.060 --> 00:20:03.550
Let me write down A again.
00:20:07.860 --> 00:20:16.150
If life is right, we should get
U, which I don't yet know --
00:20:16.150 --> 00:20:19.870
U I don't yet know,
sigma I do now know.
00:20:19.870 --> 00:20:21.040
What's sigma?
00:20:21.040 --> 00:20:24.080
So I'm looking for a
U sigma V transpose.
00:20:24.080 --> 00:20:28.115
U, the diagonal guy
and V transpose.
00:20:31.500 --> 00:20:32.030
Okay.
00:20:32.030 --> 00:20:33.740
Let's just see that
come out right.
00:20:33.740 --> 00:20:36.780
So what are the sigmas?
00:20:36.780 --> 00:20:39.130
They're the square
roots of these things.
00:20:39.130 --> 00:20:43.820
So square root of 32
and square root of 18.
00:20:47.900 --> 00:20:49.120
Zero zero.
00:20:49.120 --> 00:20:50.210
Okay.
00:20:50.210 --> 00:20:52.040
What are the Vs?
00:20:52.040 --> 00:20:53.810
They're these two.
00:20:53.810 --> 00:20:56.900
And I have to transpose --
00:20:56.900 --> 00:20:59.450
maybe that just
leaves me with ones --
00:20:59.450 --> 00:21:03.790
with one over square root of two
in that row and the other one
00:21:03.790 --> 00:21:06.870
is one over square
root of two minus one
00:21:06.870 --> 00:21:08.010
over square root of two.
00:21:11.590 --> 00:21:15.030
Now finally, I've
got to know the Us.
00:21:15.030 --> 00:21:18.570
Well, actually, one way to do --
since I now know all the other
00:21:18.570 --> 00:21:21.100
pieces, I could put those
together and figure out what
00:21:21.100 --> 00:21:22.110
the Us are.
00:21:22.110 --> 00:21:25.780
But let me do it the
A A transpose way.
00:21:25.780 --> 00:21:26.280
Okay.
00:21:26.280 --> 00:21:27.380
Find the Us now.
00:21:31.390 --> 00:21:33.070
u1 and u2.
00:21:33.070 --> 00:21:34.650
And what are they?
00:21:37.220 --> 00:21:41.240
I look at A A transpose --
00:21:41.240 --> 00:21:47.230
so A is supposed to be U
sigma V transpose, and then
00:21:47.230 --> 00:21:52.652
when I transpose that I get V
sigma transpose U transpose.
00:21:57.210 --> 00:21:59.160
So I'm just doing it
in the opposite order,
00:21:59.160 --> 00:22:03.470
A times A transpose, and
what's the good part here?
00:22:03.470 --> 00:22:10.480
That in the middle, V transpose
V is going to be the identity.
00:22:10.480 --> 00:22:14.990
So this is just U
sigma sigma transpose,
00:22:14.990 --> 00:22:22.340
that's some diagonal matrix with
sigma squareds and U transpose.
00:22:22.340 --> 00:22:24.480
So what I seeing here?
00:22:24.480 --> 00:22:29.280
I'm seeing here, again, a
symmetric positive definite
00:22:29.280 --> 00:22:32.000
or at least semi-definite
matrix and I'm
00:22:32.000 --> 00:22:36.600
seeing its eigenvectors
and its eigenvalues.
00:22:36.600 --> 00:22:41.590
So if I compute A A
transpose, its eigenvectors
00:22:41.590 --> 00:22:44.060
will be the things
that go into U.
00:22:44.060 --> 00:22:47.470
Okay, so I need to
compute A A transpose.
00:22:47.470 --> 00:22:50.970
I guess I'm going
to have to go --
00:22:50.970 --> 00:22:53.450
can I move that
up just a little?
00:22:53.450 --> 00:22:56.580
Maybe a little more
and do A A transpose.
00:22:59.880 --> 00:23:01.820
So what's A?
00:23:01.820 --> 00:23:05.930
Four four minus three and three.
00:23:05.930 --> 00:23:07.550
And what's A transpose?
00:23:07.550 --> 00:23:10.310
Four four minus three and three.
00:23:10.310 --> 00:23:15.530
And when I do that
multiplication, what do I get?
00:23:15.530 --> 00:23:18.750
Sixteen and sixteen, thirty two.
00:23:18.750 --> 00:23:21.630
Uh, that one comes out zero.
00:23:21.630 --> 00:23:26.940
Oh, so this is a lucky case
and that one comes out 18.
00:23:26.940 --> 00:23:31.560
So this is an accident
that A A transpose
00:23:31.560 --> 00:23:38.030
happens to come out diagonal, so
we know easily its eigenvectors
00:23:38.030 --> 00:23:38.990
and eigenvalues.
00:23:38.990 --> 00:23:43.130
So its eigenvectors -- what's
the first eigenvector for this
00:23:43.130 --> 00:23:45.150
A A transpose matrix?
00:23:45.150 --> 00:23:49.940
It's just one zero, and when
I do that multiplication,
00:23:49.940 --> 00:23:54.020
I get 32 times one zero.
00:23:54.020 --> 00:23:57.380
And the other eigenvector
is just zero one
00:23:57.380 --> 00:24:00.350
and when I multiply
by that I get 18.
00:24:00.350 --> 00:24:04.720
So this is A A transpose.
00:24:04.720 --> 00:24:08.910
Multiplying that gives
me the 32 A A transpose.
00:24:08.910 --> 00:24:14.860
Multiplying this guy
gives me First of all,
00:24:14.860 --> 00:24:18.590
I got 32 and 18 again.
00:24:18.590 --> 00:24:19.740
Am I surprised?
00:24:19.740 --> 00:24:24.420
You know, it's clearly
not an accident.
00:24:24.420 --> 00:24:29.190
The eigenvalues of A A
transpose were exactly the same
00:24:29.190 --> 00:24:37.820
as the eigenvalues of --
this one was A transpose A.
00:24:37.820 --> 00:24:40.140
That's no surprise at all.
00:24:40.140 --> 00:24:47.140
The eigenvalues of A B are the
same as the eigenvalues of B A.
00:24:47.140 --> 00:24:50.530
That's a very nice
fact, that eigenvalues
00:24:50.530 --> 00:24:55.310
stay the same if I switch
the order of multiplication.
00:24:55.310 --> 00:25:01.650
So no surprise to see
32 and What I learned --
00:25:01.650 --> 00:25:05.550
first the check that things
were numerically correct,
00:25:05.550 --> 00:25:07.950
but now I've learned
these eigenvectors,
00:25:07.950 --> 00:25:11.980
and actually they're
about as nice as can be.
00:25:11.980 --> 00:25:16.045
They're the best orthogonal
matrix, just the identity.
00:25:18.511 --> 00:25:19.010
Okay.
00:25:21.940 --> 00:25:26.400
So my claim is that it
ought to all fit together,
00:25:26.400 --> 00:25:31.560
that these numbers
should come out right.
00:25:31.560 --> 00:25:33.440
The numbers should
come out right
00:25:33.440 --> 00:25:41.070
because the matrix
multiplications use
00:25:41.070 --> 00:25:42.450
the properties that we want.
00:25:42.450 --> 00:25:42.970
Okay.
00:25:42.970 --> 00:25:44.370
Shall we just check that?
00:25:44.370 --> 00:25:47.020
Here's the identity, so
not doing anything --
00:25:47.020 --> 00:25:50.530
square root of 32 is
multiplying that row,
00:25:50.530 --> 00:25:53.670
so that square root of 32
divided by square root of two
00:25:53.670 --> 00:25:58.150
means square root of
16, four, correct?
00:25:58.150 --> 00:26:01.680
And square root of 18 is
divided by square root of two,
00:26:01.680 --> 00:26:07.570
so that leaves me square root
of 9, which is three, but --
00:26:07.570 --> 00:26:11.100
well, Professor Strang,
you see the problem?
00:26:11.100 --> 00:26:12.740
Why is that --
00:26:12.740 --> 00:26:13.240
okay.
00:26:13.240 --> 00:26:16.790
Why I getting minus
three three here
00:26:16.790 --> 00:26:19.965
and here I'm getting
three minus three?
00:26:24.640 --> 00:26:26.240
Phooey.
00:26:26.240 --> 00:26:27.105
I don't know why.
00:26:30.980 --> 00:26:34.650
It shouldn't have
happened, but it did.
00:26:34.650 --> 00:26:38.402
Now, okay, you could
say, well, just --
00:26:41.300 --> 00:26:43.560
the eigenvector
there could have --
00:26:43.560 --> 00:26:47.410
I could have had the minus
sign here for that eigenvector,
00:26:47.410 --> 00:26:49.710
but I'm not happy about that.
00:26:49.710 --> 00:26:50.210
Hmm.
00:26:50.210 --> 00:26:50.710
Okay.
00:26:55.740 --> 00:26:58.480
So I realize there's a
little catch here somewhere
00:26:58.480 --> 00:27:02.630
and I may not see
it until Wednesday.
00:27:02.630 --> 00:27:04.830
Which then gives you a
very important reason
00:27:04.830 --> 00:27:09.930
to come back on Wednesday, to
catch that sine difference.
00:27:09.930 --> 00:27:14.310
So what did I do illegally?
00:27:14.310 --> 00:27:22.550
I think I put the eigenvectors
in that matrix V transpose --
00:27:22.550 --> 00:27:24.160
okay, I'm going
to have to think.
00:27:24.160 --> 00:27:29.460
Why did that come out with
with the opposite sines?
00:27:29.460 --> 00:27:30.510
So you see --
00:27:30.510 --> 00:27:35.000
I mean, if I had a minus
there, I would be all right,
00:27:35.000 --> 00:27:36.490
but I don't want that.
00:27:36.490 --> 00:27:45.331
I want positive entries down
the diagonal of sigma squared.
00:27:45.331 --> 00:27:45.830
Okay.
00:27:45.830 --> 00:27:51.590
It'll come to me, but, I'm
going to leave this example
00:27:51.590 --> 00:27:57.600
to finish.
00:27:57.600 --> 00:27:58.920
Okay.
00:27:58.920 --> 00:28:02.560
And the beauty of,
these sliding boards
00:28:02.560 --> 00:28:05.780
is I can make that go away.
00:28:05.780 --> 00:28:10.910
Can I,-- let me not
do it, though, yet.
00:28:10.910 --> 00:28:15.090
Let me take a second example.
00:28:15.090 --> 00:28:18.720
Let me take a second example
where the matrix is singular.
00:28:21.940 --> 00:28:24.090
So rank one.
00:28:24.090 --> 00:28:32.390
Okay, so let me take
as an example two,
00:28:32.390 --> 00:28:38.770
where my matrix A is going
to be rectangular again --
00:28:38.770 --> 00:28:43.220
let me just make it
four three eight six.
00:28:47.590 --> 00:28:48.090
Okay.
00:28:48.090 --> 00:28:50.830
That's a rank one matrix.
00:28:50.830 --> 00:28:57.430
So that has a null space and
only a one dimensional row
00:28:57.430 --> 00:28:59.370
space and column space.
00:28:59.370 --> 00:29:05.510
So actually, my picture
becomes easy for this matrix,
00:29:05.510 --> 00:29:09.510
because what's my row
space for this one?
00:29:09.510 --> 00:29:12.450
So this is two by two.
00:29:12.450 --> 00:29:17.000
So my pictures are
both two dimensional.
00:29:17.000 --> 00:29:22.390
My row space is all multiples
of that vector four three.
00:29:22.390 --> 00:29:24.760
So the whole -- the row
space is just a line, right?
00:29:27.770 --> 00:29:29.380
That's the row space.
00:29:29.380 --> 00:29:33.050
And the null space, of course,
is the perpendicular line.
00:29:33.050 --> 00:29:46.480
So the row space for this matrix
is multiples of four three.
00:29:46.480 --> 00:29:47.680
Typical row.
00:29:47.680 --> 00:29:48.520
Okay.
00:29:48.520 --> 00:29:50.060
What's the column space?
00:29:50.060 --> 00:29:55.980
The columns are all multiples of
four eight, three six, one two.
00:29:55.980 --> 00:30:00.315
The column space, then, goes
in, like, this direction.
00:30:03.040 --> 00:30:07.490
So the column space --
00:30:07.490 --> 00:30:09.650
when I look at those
columns, the column space --
00:30:09.650 --> 00:30:12.660
so it's only one dimensional,
because the rank is one.
00:30:12.660 --> 00:30:21.480
It's multiples of four eight.
00:30:21.480 --> 00:30:22.370
Okay.
00:30:22.370 --> 00:30:26.360
And what's the null
space of A transpose?
00:30:26.360 --> 00:30:30.270
It's the perpendicular guy.
00:30:30.270 --> 00:30:35.150
So this was the null
space of A and this is
00:30:35.150 --> 00:30:38.921
the null space of A transpose.
00:30:38.921 --> 00:30:39.420
Okay.
00:30:42.260 --> 00:30:48.650
What I want to say here is that
choosing these orthogonal bases
00:30:48.650 --> 00:30:53.750
for the row space and the column
space is, like, no problem.
00:30:53.750 --> 00:30:55.800
They're only one dimensional.
00:30:55.800 --> 00:30:58.020
So what should V be?
00:30:58.020 --> 00:31:02.620
V should be -- v1, but
-- yes, v1, rather --
00:31:02.620 --> 00:31:05.540
v1 is supposed to
be a unit vector.
00:31:05.540 --> 00:31:08.640
There's only one
v1 to choose here,
00:31:08.640 --> 00:31:11.210
only one dimension
in the row space.
00:31:11.210 --> 00:31:13.800
I just want to make
it a unit vector.
00:31:13.800 --> 00:31:17.180
So v1 will be --
00:31:17.180 --> 00:31:24.570
it'll be this vector, but made
into a unit vector, so four --
00:31:24.570 --> 00:31:25.970
point eight point six.
00:31:28.660 --> 00:31:30.670
Four fifths, three fifths.
00:31:30.670 --> 00:31:33.040
And what will be u1?
00:31:33.040 --> 00:31:35.740
u1 will be the
unit vector there.
00:31:35.740 --> 00:31:40.820
So I want to turn four eight
or one two into a unit vector,
00:31:40.820 --> 00:31:44.150
so u1 will be --
00:31:44.150 --> 00:31:47.760
let's see, if it's one two,
then what multiple of one two
00:31:47.760 --> 00:31:49.230
do I want?
00:31:49.230 --> 00:31:51.240
That has length
square root of five,
00:31:51.240 --> 00:31:53.460
so I have to divide by
square root of five.
00:31:56.140 --> 00:31:58.470
Let me complete
the singular value
00:31:58.470 --> 00:32:01.660
decomposition for this matrix.
00:32:01.660 --> 00:32:09.540
So this matrix, four
three eight six, is --
00:32:09.540 --> 00:32:11.520
so I know what u1 --
00:32:11.520 --> 00:32:19.570
here's A and I want to get U
the basis in the column space.
00:32:19.570 --> 00:32:24.020
And it has to start
with this guy, one
00:32:24.020 --> 00:32:27.290
over square root of five two
over square root of five.
00:32:30.520 --> 00:32:36.350
Then I want the sigma.
00:32:36.350 --> 00:32:37.540
Okay.
00:32:37.540 --> 00:32:40.370
What are we expecting
now for sigma?
00:32:44.400 --> 00:32:47.010
This is only a rank one matrix.
00:32:47.010 --> 00:32:51.720
We're only expecting a sigma
one, which I have to find,
00:32:51.720 --> 00:32:55.040
but zeroes here.
00:32:55.040 --> 00:32:55.540
Okay.
00:32:55.540 --> 00:32:57.420
So what's sigma one?
00:32:57.420 --> 00:33:02.950
It should be the --
00:33:02.950 --> 00:33:05.480
where did these
sigmas come from?
00:33:05.480 --> 00:33:08.370
They came from A
transpose A, so I --
00:33:08.370 --> 00:33:10.750
can I do that little
calculation over here?
00:33:10.750 --> 00:33:19.840
A transpose A is four three --
four three eight six times four
00:33:19.840 --> 00:33:23.340
three eight six.
00:33:23.340 --> 00:33:26.240
This had better -- this
is a rank one matrix,
00:33:26.240 --> 00:33:29.170
this is going to be -- the
whole thing will have rank one,
00:33:29.170 --> 00:33:40.240
that's 16 and 64 is 80, 12
and 48 is 60, 12 and 48 is 60,
00:33:40.240 --> 00:33:43.880
9 and 36 is 45.
00:33:43.880 --> 00:33:45.520
Okay.
00:33:45.520 --> 00:33:47.340
It's a rank one matrix.
00:33:47.340 --> 00:33:48.230
Of course.
00:33:48.230 --> 00:33:52.450
Every row is a
multiple of four three.
00:33:52.450 --> 00:33:56.820
And what's the eigen -- what are
the eigenvalues of that matrix?
00:33:56.820 --> 00:33:59.730
So this is the calculation
-- this is like practicing,
00:33:59.730 --> 00:34:00.230
now.
00:34:00.230 --> 00:34:04.510
What are the eigenvalues
of this rank one matrix?
00:34:04.510 --> 00:34:08.389
Well, tell me one eigenvalue,
since the rank is only one,
00:34:08.389 --> 00:34:11.920
one eigenvalue is
going to be zero.
00:34:11.920 --> 00:34:14.320
And then you know that
the other eigenvalue
00:34:14.320 --> 00:34:19.330
is going to be a
hundred and twenty five.
00:34:19.330 --> 00:34:22.760
So that's sigma squared,
right, in A transpose A.
00:34:22.760 --> 00:34:28.780
So this will be the square root
of a hundred and twenty five.
00:34:28.780 --> 00:34:34.900
And then finally,
the V transpose --
00:34:34.900 --> 00:34:37.750
the Vs will be --
00:34:37.750 --> 00:34:41.830
there's v1, and what's v2?
00:34:41.830 --> 00:34:45.170
What's v2 in the --
00:34:45.170 --> 00:34:50.739
how do I make this into
an orthonormal basis?
00:34:50.739 --> 00:34:55.480
Well, v2 is, in the
null space direction.
00:34:55.480 --> 00:34:59.810
It's perpendicular to that,
so point six and minus point
00:34:59.810 --> 00:35:00.790
eight.
00:35:00.790 --> 00:35:04.400
So those are the
Vs that go in here.
00:35:04.400 --> 00:35:11.920
Point eight, point six and
point six minus point eight.
00:35:11.920 --> 00:35:12.420
Okay.
00:35:14.960 --> 00:35:17.410
And I guess I better
finish this guy.
00:35:17.410 --> 00:35:21.580
So this guy, all I want is to
complete the orthonormal basis
00:35:21.580 --> 00:35:23.670
-- it'll be coming from there.
00:35:23.670 --> 00:35:28.770
It'll be a two over square
root of five and a minus one
00:35:28.770 --> 00:35:31.330
over square root of five.
00:35:31.330 --> 00:35:35.100
Let me take square root
of five out of that matrix
00:35:35.100 --> 00:35:38.080
to make it look better.
00:35:38.080 --> 00:35:44.850
So one over square root of five
times one two two minus one.
00:35:47.560 --> 00:35:49.920
Okay.
00:35:49.920 --> 00:35:53.160
So there I have -- including
the square root of five --
00:35:53.160 --> 00:35:56.870
that's an orthogonal matrix,
that's an orthogonal matrix,
00:35:56.870 --> 00:36:01.190
that's a diagonal matrix
and its rank is only one.
00:36:01.190 --> 00:36:03.800
And now if I do
that multiplication,
00:36:03.800 --> 00:36:07.550
I pray that it comes out right.
00:36:10.200 --> 00:36:12.022
The square root of
five will cancel
00:36:12.022 --> 00:36:13.480
into that square
root of one twenty
00:36:13.480 --> 00:36:16.390
five and leave me with the
square root of 25, which
00:36:16.390 --> 00:36:20.190
is five, and five will
multiply these numbers
00:36:20.190 --> 00:36:24.410
and I'll get whole numbers
and out will come A.
00:36:24.410 --> 00:36:25.230
Okay.
00:36:25.230 --> 00:36:31.110
That's like a second example
showing how the null space guy
00:36:31.110 --> 00:36:39.680
-- so this -- this vector
and this one were multiplied
00:36:39.680 --> 00:36:40.620
by this zero.
00:36:40.620 --> 00:36:46.370
So they were easy to deal with.
00:36:46.370 --> 00:36:50.960
Tthe key ones are the ones in
the column space and the row
00:36:50.960 --> 00:36:51.480
space.
00:36:51.480 --> 00:36:57.920
Do you see how I'm getting
columns here, diagonal here,
00:36:57.920 --> 00:37:02.330
rows here, coming
together to produce A.
00:37:02.330 --> 00:37:06.790
Okay, that's the singular
value decomposition.
00:37:06.790 --> 00:37:13.370
So, let me think what I want
to add to complete this topic.
00:37:18.130 --> 00:37:22.970
So that's two examples.
00:37:22.970 --> 00:37:25.740
And now let's think
what we're really doing.
00:37:25.740 --> 00:37:33.770
We're choosing the right
basis for the four subspaces
00:37:33.770 --> 00:37:35.010
of linear algebra.
00:37:35.010 --> 00:37:39.640
Let me write this down.
00:37:39.640 --> 00:37:52.785
So v1 up to vr is an orthonormal
basis for the row space.
00:37:58.220 --> 00:38:05.570
u1 up to ur is an orthonormal
basis for the column space.
00:38:09.420 --> 00:38:14.930
And then I just finish
those out by v(r+1),
00:38:14.930 --> 00:38:20.350
the rest up to vn is an
orthonormal basis for the null
00:38:20.350 --> 00:38:20.850
space.
00:38:24.510 --> 00:38:35.220
And finally, u(r+1) up to is an
orthonormal basis for the null
00:38:35.220 --> 00:38:36.320
space of A transpose.
00:38:39.670 --> 00:38:45.360
Do you see that we finally
got the bases right?
00:38:45.360 --> 00:38:51.170
They're right because they're
orthonormal, and also --
00:38:51.170 --> 00:38:55.100
again, Graham Schmidt would
have done this in chapter four.
00:38:55.100 --> 00:39:00.920
Here we needed eigenvalues,
because these bases
00:39:00.920 --> 00:39:03.070
make the matrix diagonal.
00:39:03.070 --> 00:39:09.400
A times V I is a
multiple of U I.
00:39:09.400 --> 00:39:11.250
So I'll put "and" --
00:39:14.540 --> 00:39:16.640
the matrix has
been made diagonal.
00:39:16.640 --> 00:39:24.730
When we choose these bases,
there's no coupling between Vs
00:39:24.730 --> 00:39:26.740
and no coupling between Us.
00:39:26.740 --> 00:39:30.550
Each A -- A times each
V is in the direction
00:39:30.550 --> 00:39:31.980
of the corresponding U.
00:39:31.980 --> 00:39:36.700
So it's exactly the
right basis for the four
00:39:36.700 --> 00:39:38.130
fundamental subspaces.
00:39:38.130 --> 00:39:41.570
And of course, their
dimensions are what we know.
00:39:41.570 --> 00:39:44.850
The dimension of
the row space is
00:39:44.850 --> 00:39:49.450
the rank r, and so is the
dimension of the column space.
00:39:49.450 --> 00:39:51.190
The dimension of
the null space is
00:39:51.190 --> 00:39:54.620
n-r, that's how many
vectors we need,
00:39:54.620 --> 00:39:59.790
and m-r basis vectors for the
left null space, the null space
00:39:59.790 --> 00:40:01.960
of A transpose.
00:40:01.960 --> 00:40:04.620
Okay.
00:40:04.620 --> 00:40:05.850
I'm going to stop there.
00:40:05.850 --> 00:40:10.720
I could develop
further from the SVD,
00:40:10.720 --> 00:40:14.140
but we'll see it again
in the very last lectures
00:40:14.140 --> 00:40:14.949
of the course.
00:40:14.949 --> 00:40:15.740
So there's the SVD.
00:40:15.740 --> 00:40:17.290
Thanks.