1
00:00:11 --> 00:00:12
Okay.
2
00:00:12 --> 00:00:19
This is the lecture on the
singular value decomposition.
3
00:00:19 --> 00:00:23
But everybody calls it the SVD.
4
00:00:23 --> 00:00:31
So this is the final and best
factorization of a matrix.
5
00:00:31 --> 00:00:35
Let me tell you what's coming.
6
00:00:35 --> 00:00:40
The factors will be,
orthogonal matrix,
7
00:00:40 --> 00:00:46
diagonal matrix,
orthogonal matrix.
8
00:00:46 --> 00:00:50
So it's things that we've seen
before, these special good
9
00:00:50 --> 00:00:52
matrices, orthogonal diagonal.
10
00:00:52 --> 00:00:55
The new point is that we need
two orthogonal matrices.
11
00:00:55 --> 00:00:57
A can be any matrix whatsoever.
12
00:00:57 --> 00:01:01
Any matrix whatsoever has this
singular value decomposition,
13
00:01:01 --> 00:01:05.48
so a diagonal one in the
middle, but I need two different
14
00:01:05.48 --> 00:01:10
-- probably different orthogonal
matrices to be able to do this.
15
00:01:10 --> 00:01:10
Okay.
16
00:01:10 --> 00:01:16
And this factorization has
jumped into importance and is
17
00:01:16 --> 00:01:21.1
properly, I think,
maybe the bringing together of
18
00:01:21.1 --> 00:01:23.65
everything in this course.
19
00:01:23.65 --> 00:01:29
One thing we'll bring together
is the very good family of
20
00:01:29 --> 00:01:34
matrices that we just studied,
symmetric, positive,
21
00:01:34 --> 00:01:36
definite.
22
00:01:36 --> 00:01:39
Do you remember the stories
with those guys?
23
00:01:39 --> 00:01:44
Because they were symmetric,
their eigenvectors were
24
00:01:44 --> 00:01:50.37
orthogonal, so I could produce
an orthogonal matrix -- this is
25
00:01:50.37 --> 00:01:51
my usual one.
26
00:01:51 --> 00:01:56
My usual one is the
eigenvectors and eigenvalues
27
00:01:56 --> 00:02:00
In the symmetric case,
the eigenvectors are
28
00:02:00 --> 00:02:07
orthogonal, so I've got the good
-- my ordinary s has become an
29
00:02:07 --> 00:02:08
especially good Q.
30
00:02:08 --> 00:02:14
And positive definite,
my ordinary lambda has become a
31
00:02:14 --> 00:02:16
positive lambda.
32
00:02:16 --> 00:02:21.96
So that's the singular value
decomposition in case our matrix
33
00:02:21.96 --> 00:02:26
is symmetric positive definite
-- in that case,
34
00:02:26 --> 00:02:31
I don't need two -- U and a V
-- one orthogonal matrix will do
35
00:02:31 --> 00:02:33
for both sides.
36
00:02:33 --> 00:02:40
So this would be no good in
general, because usually the
37
00:02:40 --> 00:02:44
eigenvector matrix isn't
orthogonal.
38
00:02:44 --> 00:02:47.66
So that's not what I'm after.
39
00:02:47.66 --> 00:02:55
I'm looking for orthogonal
times diagonal times orthogonal.
40
00:02:55 --> 00:03:01
And let me show you what that
means and where it comes from.
41
00:03:01 --> 00:03:02
Okay.
42
00:03:02 --> 00:03:03
What does it mean?
43
00:03:03 --> 00:03:09
You remember the picture of any
linear transformation.
44
00:03:09 --> 00:03:13
This was, like,
the most important figure in
45
00:03:13 --> 00:03:14
46
00:03:14 --> 00:03:14
47
00:03:14 --> 00:03:17
And what I looking for now?
48
00:03:17 --> 00:03:21.89
A typical vector in the row
space
49
00:03:21 --> 00:03:27
-- typical vector,
let me call it v1,
50
00:03:27.35 --> 00:03:34
gets taken over to some vector
in the column space,
51
00:03:34 --> 00:03:35
say u1.
52
00:03:35 --> 00:03:37
So u1 is Av1.
53
00:03:37 --> 00:03:38
Okay.
54
00:03:38 --> 00:03:46
Now, another vector gets taken
over here somewhere.
55
00:03:46 --> 00:03:50
What I looking for?
56
00:03:50 --> 00:03:53
In this SVD,
this singular value
57
00:03:53 --> 00:03:57.66
decomposition,
what I'm looking for is an
58
00:03:57.66 --> 00:04:03
orthogonal basis here that gets
knocked over into an orthogonal
59
00:04:03 --> 00:04:05
basis over there.
60
00:04:05 --> 00:04:10
See that's pretty special,
to have an orthogonal basis in
61
00:04:10 --> 00:04:17.35
the row space that goes over
into an orthogonal basis --
62
00:04:17.35 --> 00:04:21
so this is like a right angle
and this is a right angle --
63
00:04:21 --> 00:04:25
into an orthogonal basis in the
column space.
64
00:04:25 --> 00:04:28
So that's our goal,
is to find -- do you see how
65
00:04:28 --> 00:04:31
things are coming together?
66
00:04:31 --> 00:04:34
First of all,
can I find an orthogonal basis
67
00:04:34 --> 00:04:36.75
for this row space?
68
00:04:36.75 --> 00:04:37
Of course.
69
00:04:37 --> 00:04:39
No big deal to find an
orthogonal basis.
70
00:04:39 --> 00:04:42
Graham Schmidt tells me how to
do it.
71
00:04:42 --> 00:04:45
Start with any old basis and
grind through Graham Schmidt,
72
00:04:45 --> 00:04:47
out comes an orthogonal basis.
73
00:04:47 --> 00:04:50.61
But then, if I just take any
old orthogonal basis,
74
00:04:50.61 --> 00:04:54
then when I multiply by A,
there's no reason why it should
75
00:04:54 --> 00:04:56
be orthogonal over here.
76
00:04:56 --> 00:05:04
So I'm looking for this special
set up where A takes these basis
77
00:05:04 --> 00:05:10
vectors into orthogonal vectors
over there.
78
00:05:10 --> 00:05:17
Now, you might have noticed
that the null space I didn't
79
00:05:17 --> 00:05:18
include.
80
00:05:18 --> 00:05:22
Why don't I stick that in?
81
00:05:22 --> 00:05:27
You remember our usual figure
had a little null space and a
82
00:05:27 --> 00:05:29
little null space.
83
00:05:29 --> 00:05:31
And those are no problems.
84
00:05:31 --> 00:05:36
Those null spaces are going to
show up as zeroes on the
85
00:05:36 --> 00:05:40
diagonal of sigma,
so that doesn't present any
86
00:05:40 --> 00:05:41
difficulty.
87
00:05:41 --> 00:05:44
Our difficulty is to find
these.
88
00:05:44 --> 00:05:48.64
So do you see what this will
mean?
89
00:05:48.64 --> 00:05:53
This will mean that A times
these v-s, v1,
90
00:05:53 --> 00:05:59
v2, up to -- what's the
dimension of this row space?
91
00:05:59 --> 00:05:59
Vr.
92
00:05:59 --> 00:06:04
Sorry, make that V a little
smaller -- up to vr.
93
00:06:04 --> 00:06:10
So that's -- Av1 is going to be
the first column,
94
00:06:10 --> 00:06:15
so here's what I'm achieving.
95
00:06:15 --> 00:06:20
Oh, I'm not only going to make
these orthogonal,
96
00:06:20 --> 00:06:24.41
but why not make them
orthonormal?
97
00:06:24.41 --> 00:06:27
Make them unit vectors.
98
00:06:27 --> 00:06:32
So maybe the unit vector is
here, is the u1,
99
00:06:32 --> 00:06:37
and this might be a multiple of
it.
100
00:06:37 --> 00:06:42
So really, what's happening is
Av1 is some multiple of u1,
101
00:06:42 --> 00:06:42
right?
102
00:06:42 --> 00:06:47
These guys will be unit vectors
and they'll go over into
103
00:06:47 --> 00:06:53
multiples of unit vectors and
the multiple I'm not going to
104
00:06:53 --> 00:06:55
call lambda anymore.
105
00:06:55 --> 00:06:57
I'm calling it sigma.
106
00:06:57 --> 00:07:01
So that's the number -- the
stretching number.
107
00:07:01 --> 00:07:05
And similarly,
Av2 is sigma two u2.
108
00:07:05 --> 00:07:06
This is my goal.
109
00:07:06 --> 00:07:11
And now I want to express that
goal in matrix language.
110
00:07:11 --> 00:07:14
That's the usual step.
111
00:07:14 --> 00:07:20
Think of what you want and then
express it as a matrix
112
00:07:20 --> 00:07:21
multiplication.
113
00:07:21 --> 00:07:26
So Av1 is sigma one u1 --
actually, here we go.
114
00:07:26 --> 00:07:32
Let me pull out these -- u1,
u2 to ur and then a matrix with
115
00:07:32 --> 00:07:33.97
the sigmas.
116
00:07:33.97 --> 00:07:39
Everything now is going to be
in that little part of the
117
00:07:39 --> 00:07:42
blackboard.
118
00:07:42 --> 00:07:50.02
Do you see that this equation
says what I'm trying to do with
119
00:07:50.02 --> 00:07:51
my figure.
120
00:07:51 --> 00:07:59
A times the first basis vector
should be sigma one times the
121
00:07:59 --> 00:08:05
other basis -- the other first
basis vector.
122
00:08:05 --> 00:08:09
These are the basis vectors in
the row space,
123
00:08:09 --> 00:08:14
these are the basis vectors in
the column space and these are
124
00:08:14 --> 00:08:16
the multiplying factors.
125
00:08:16 --> 00:08:21
So Av2 is sigma two times u2,
Avr is sigma r times ur.
126
00:08:21 --> 00:08:26
And then we've got a whole lot
of zeroes and maybe some zeroes
127
00:08:26 --> 00:08:30
at the end, but that's the heart
of it.
128
00:08:30 --> 00:08:36
And now if I express that in --
as matrices, because you knew
129
00:08:36 --> 00:08:40
that was coming -- that's what I
have.
130
00:08:40 --> 00:08:45
So, this is my goal,
to find an orthogonal basis in
131
00:08:45 --> 00:08:50
the orthonormal,
even -- basis in the row space
132
00:08:50 --> 00:08:56
and an orthonormal basis in the
column space so that I've sort
133
00:08:56 --> 00:09:00
of diagonalized the matrix.
134
00:09:00 --> 00:09:03
The matrix A is,
like, getting converted to this
135
00:09:03 --> 00:09:04
diagonal matrix sigma.
136
00:09:04 --> 00:09:08
And you notice that usually I
have to allow myself two
137
00:09:08 --> 00:09:09
different bases.
138
00:09:09 --> 00:09:13
My little comment about
symmetric positive definite was
139
00:09:13 --> 00:09:15
the one case where it's A Q
equal Q sigma,
140
00:09:15 --> 00:09:18
where V and U are the same Q.
141
00:09:18 --> 00:09:25
But mostly, you know,
I'm going to take a matrix like
142
00:09:25 --> 00:09:33
-- oh, let me take a matrix like
four four minus three three.
143
00:09:33 --> 00:09:34
Okay.
144
00:09:34 --> 00:09:38.36
There's a two by two matrix.
145
00:09:38.36 --> 00:09:44
It's invertible,
so it has rank two.
146
00:09:44 --> 00:09:49
So I'm going to look for two
vectors, v1 and v2 in the row
147
00:09:49 --> 00:09:54
space, and U -- so I'm going to
look for v1, v2 in the row
148
00:09:54 --> 00:09:56
space, which of course is R^2.
149
00:09:56 --> 00:10:01
And I'm going to look for u1,
u2 in the column space,
150
00:10:01 --> 00:10:06
which of course is also R^2,
and I'm going to look for
151
00:10:06 --> 00:10:12
numbers sigma one and sigma two
so that it all comes out right.
152
00:10:12 --> 00:10:20
So these guys are orthonormal,
these guys are orthonormal and
153
00:10:20 --> 00:10:23
these are the scaling factors.
154
00:10:23 --> 00:10:30
So I'll do that example as soon
as I get the matrix picture
155
00:10:30 --> 00:10:31
straight.
156
00:10:31 --> 00:10:32
Okay.
157
00:10:32 --> 00:10:38
Do you see that this expresses
what I want?
158
00:10:38 --> 00:10:43
Can I just say two words about
null spaces?
159
00:10:43 --> 00:10:48
If there's some null space,
then we want to stick in a
160
00:10:48 --> 00:10:50
basis for those,
for that.
161
00:10:50 --> 00:10:56
So here comes a basis for the
null space, v(r+1) down to vm.
162
00:10:56 --> 00:11:01
So if we only had an r
dimensional row space and the
163
00:11:01 --> 00:11:07
other n-r dimensions were in the
null space --
164
00:11:07 --> 00:11:12
okay, we'll take an orthogonal
-- orthonormal basis there.
165
00:11:12 --> 00:11:12
No problem.
166
00:11:12 --> 00:11:15
And then we'll just get zeroes.
167
00:11:15 --> 00:11:19
So, actually,
w- those zeroes will come out
168
00:11:19 --> 00:11:21
on the diagonal matrix.
169
00:11:21 --> 00:11:26
So I'll complete that to an
orthonormal basis for the whole
170
00:11:26 --> 00:11:28
space, R^m.
171
00:11:28 --> 00:11:32
I complete this to an
orthonormal basis for the whole
172
00:11:32 --> 00:11:35
space R^n and I complete that
with zeroes.
173
00:11:35 --> 00:11:38
Null spaces are no problem
here.
174
00:11:38 --> 00:11:42.66
So really the true problem is
in a matrix like that,
175
00:11:42.66 --> 00:11:45.98
which isn't symmetric,
so I can't use its
176
00:11:45.98 --> 00:11:49
eigenvectors,
they're not orthogonal -- but
177
00:11:49 --> 00:11:53
somehow I have to get these
orthogonal --
178
00:11:53 --> 00:12:00
in fact, orthonormal guys that
make it work.
179
00:12:00 --> 00:12:06
I have to find these
orthonormal guys,
180
00:12:06 --> 00:12:14
these orthonormal guys and I
want Av1 to be sigma one u1 and
181
00:12:14 --> 00:12:19
Av2 to be sigma two u2.
182
00:12:19 --> 00:12:19
Okay.
183
00:12:19 --> 00:12:21.31
That's my goal.
184
00:12:21.31 --> 00:12:26
Here's the matrices that are
going to get me there.
185
00:12:26 --> 00:12:29.17
Now these are orthogonal
matrices.
186
00:12:29.17 --> 00:12:34
I can put that -- if I multiply
on both sides by V inverse,
187
00:12:34 --> 00:12:39
I have A equals U sigma V
inverse, and of course you know
188
00:12:39 --> 00:12:44
the other way I can write V
inverse.
189
00:12:44 --> 00:12:52
This is one of those square
orthogonal matrices,
190
00:12:52 --> 00:12:58
so it's the same as U sigma V
transpose.
191
00:12:58 --> 00:12:59
Okay.
192
00:12:59 --> 00:13:02
Here's my problem.
193
00:13:02 --> 00:13:08
I've got two orthogonal
matrices here.
194
00:13:08 --> 00:13:16
And I don't want to find them
both at once.
195
00:13:16 --> 00:13:21
So I want to cook up some
expression that will make the Us
196
00:13:21 --> 00:13:22
disappear.
197
00:13:22 --> 00:13:27
I would like to make the Us
disappear and leave me only with
198
00:13:27 --> 00:13:28
the Vs.
199
00:13:28 --> 00:13:30
And here's how to do it.
200
00:13:30 --> 00:13:35
It's the same combination that
keeps showing up whenever we
201
00:13:35 --> 00:13:40
have a general rectangular
matrix, then it's A transpose A,
202
00:13:40 --> 00:13:43
that's the great matrix.
203
00:13:43 --> 00:13:45
That's the great matrix.
204
00:13:45 --> 00:13:49
That's the matrix that's
symmetric, and in fact positive
205
00:13:49 --> 00:13:52.92
definite or at least positive
semi-definite.
206
00:13:52.92 --> 00:13:57
This is the matrix with nice
properties, so let's see what
207
00:13:57 --> 00:13:57
will it be?
208
00:13:57 --> 00:14:01
So if I took the transpose,
then, I would have -- A
209
00:14:01 --> 00:14:04.28
transpose A will be what?
210
00:14:04.28 --> 00:14:05
What do I have?
211
00:14:05 --> 00:14:11
If I transpose that I have V
sigma transpose U transpose,
212
00:14:11 --> 00:14:13
that's the A transpose.
213
00:14:13 --> 00:14:17.05
Now the A -- and what have I
got?
214
00:14:17.05 --> 00:14:21
Looks like worse,
because it's got six things now
215
00:14:21 --> 00:14:27.59
together, but it's going to
collapse into something good.
216
00:14:27.59 --> 00:14:32
What does U transpose U
collapse into?
217
00:14:32 --> 00:14:33
I, the identity.
218
00:14:33 --> 00:14:35
So that's the key point.
219
00:14:35 --> 00:14:38
This is the identity and we
don't have U anymore.
220
00:14:38 --> 00:14:42
And sigma transpose times
sigma, those are diagonal
221
00:14:42 --> 00:14:45
matrixes, so their product is
just going to have sigma
222
00:14:45 --> 00:14:47
squareds on the diagonal.
223
00:14:47 --> 00:14:49
So do you see what we've got
here?
224
00:14:49 --> 00:14:53
This is V times this easy
matrix sigma one squared sigma
225
00:14:53 --> 00:14:56
two squared times V transpose.
226
00:14:56 --> 00:15:00
This is the A transpose A.
227
00:15:00 --> 00:15:07
This is -- let me copy down --
A transpose A is that.
228
00:15:07 --> 00:15:11
Us are out of the picture,
now.
229
00:15:11 --> 00:15:18
I'm only having to choose the
Vs, and what are these Vs?
230
00:15:18 --> 00:15:23
And what are these sigmas?
231
00:15:23 --> 00:15:25.6
Do you know what the Vs are?
232
00:15:25.6 --> 00:15:30
They're the eigenvectors that
-- see, this is a perfect
233
00:15:30 --> 00:15:35.32
eigenvector, eigenvalue,
Q lambda Q transpose for the
234
00:15:35.32 --> 00:15:37
matrix A transpose A.
235
00:15:37 --> 00:15:39
A itself is nothing special.
236
00:15:39 --> 00:15:42
But A transpose A will be
special.
237
00:15:42 --> 00:15:47
It'll be symmetric positive
definite, so this will be its
238
00:15:47 --> 00:15:53
eigenvectors and this'll be its
eigenvalues.
239
00:15:53 --> 00:15:58
And the eigenvalues'll be
positive because this thing's
240
00:15:58 --> 00:16:00
positive definite.
241
00:16:00 --> 00:16:02
So this is my method.
242
00:16:02 --> 00:16:05
This tells me what the Vs are.
243
00:16:05 --> 00:16:09
And how I going to find the Us?
244
00:16:09 --> 00:16:13
Well, one way would be to look
at A A transpose.
245
00:16:13 --> 00:16:19
Multiply A by A transpose in
the opposite order.
246
00:16:19 --> 00:16:24.48
That will stick the Vs in the
middle, knock them out,
247
00:16:24.48 --> 00:16:26
and leave me with the Us.
248
00:16:26 --> 00:16:29
So here's the overall picture,
then.
249
00:16:29 --> 00:16:33
The Vs are the eigenvectors of
A transpose A.
250
00:16:33 --> 00:16:37
The Us are the eigenvectors of
A A transpose,
251
00:16:37 --> 00:16:40
which are different.
252
00:16:40 --> 00:16:47.09
And the sigmas are the square
roots of these and the positive
253
00:16:47.09 --> 00:16:51
square roots,
so we have positive sigmas.
254
00:16:51 --> 00:16:54
Let me do it for that example.
255
00:16:54 --> 00:17:01
This is really what you should
know and be able to do for the
256
00:17:01 --> 00:17:01.57
SVD.
257
00:17:01.57 --> 00:17:02.11
Okay.
258
00:17:02.11 --> 00:17:04
Let me take that matrix.
259
00:17:04 --> 00:17:08
So what's my first step?
260
00:17:08 --> 00:17:12
Compute A transpose A,
because I want its
261
00:17:12 --> 00:17:13
eigenvectors.
262
00:17:13 --> 00:17:13
Okay.
263
00:17:13 --> 00:17:16
So I have to compute A
transpose A.
264
00:17:16 --> 00:17:20
So A transpose is four four
minus three three,
265
00:17:20 --> 00:17:25
and A is four four minus three
three, and I do that
266
00:17:25 --> 00:17:30
multiplication and I get sixteen
-- I get twenty five -- I get
267
00:17:30 --> 00:17:35
sixteen minus nine -- is that
seven?
268
00:17:35 --> 00:17:38
And it better come out
symmetric.
269
00:17:38 --> 00:17:42
And -- oh, okay,
and then it comes out 25.
270
00:17:42 --> 00:17:42
Okay.
271
00:17:42 --> 00:17:47
So, I want its eigenvectors and
its eigenvalues.
272
00:17:47 --> 00:17:53
Its eigenvectors will be the
Vs, its eigenvalues will be the
273
00:17:53 --> 00:17:55
squares of the sigmas.
274
00:17:55 --> 00:17:55
Okay.
275
00:17:55 --> 00:18:02
What are the eigenvalues and
eigenvectors of this guy?
276
00:18:02 --> 00:18:09
Have you seen that two by two
example enough to recognize that
277
00:18:09 --> 00:18:16
the eigenvectors are -- that one
one is an eigenvector?
278
00:18:16 --> 00:18:19
So this here is A transpose A.
279
00:18:19 --> 00:18:24
I'm looking for its
eigenvectors.
280
00:18:24 --> 00:18:29
So its eigenvectors,
I think, are one one and one
281
00:18:29 --> 00:18:34
minus one, because if I multiply
that matrix by one one,
282
00:18:34 --> 00:18:36
what do I get?
283
00:18:36 --> 00:18:41
If I multiply that matrix by
one one, I get 32 32,
284
00:18:41 --> 00:18:43
which is 32 of one one.
285
00:18:43 --> 00:18:48
So there's the first
eigenvector, and there's the
286
00:18:48 --> 00:18:52
eigenvalue for A transpose A.
287
00:18:52 --> 00:18:56
So I'm going to take its square
root for sigma.
288
00:18:56 --> 00:18:57
Okay.
289
00:18:57 --> 00:19:02
What's the eigenvector that
goes -- eigenvalue that goes
290
00:19:02 --> 00:19:04
with this one?
291
00:19:04 --> 00:19:08
If I do that multiplication,
what do I get?
292
00:19:08 --> 00:19:13
I get some multiple of one
minus one, and what is that
293
00:19:13 --> 00:19:14
multiple?
294
00:19:14 --> 00:19:16
Looks like 18.
295
00:19:16 --> 00:19:17
Okay.
296
00:19:17 --> 00:19:21
So those are the two
eigenvectors,
297
00:19:21 --> 00:19:26
but -- oh, just a moment,
I didn't normalize them.
298
00:19:26 --> 00:19:33
To make everything absolutely
right, I ought to normalize
299
00:19:33 --> 00:19:37
these eigenvectors,
divide by their length,
300
00:19:37 --> 00:19:40
square root of two.
301
00:19:40 --> 00:19:46
So all these guys should be
true unit vectors and,
302
00:19:46 --> 00:19:53
of course, that normalization
didn't change the 32 and the 18.
303
00:19:53 --> 00:19:54
Okay.
304
00:19:54 --> 00:19:57
So I'm happy with the Vs.
305
00:19:57 --> 00:19:59
Here are the Vs.
306
00:19:59 --> 00:20:04
So now let me put together the
pieces here.
307
00:20:04 --> 00:20:05
Here's my A.
308
00:20:05 --> 00:20:06
Here's my A.
309
00:20:06 --> 00:20:11
Let me write down A again.
310
00:20:11 --> 00:20:14
If life is right,
we should get U,
311
00:20:14 --> 00:20:18
which I don't yet know -- U I
don't yet know,
312
00:20:18 --> 00:20:20
sigma I do now know.
313
00:20:20 --> 00:20:21
What's sigma?
314
00:20:21 --> 00:20:25.35
So I'm looking for a U sigma V
transpose.
315
00:20:25.35 --> 00:20:28
U, the diagonal guy and V
transpose.
316
00:20:28 --> 00:20:29
Okay.
317
00:20:29 --> 00:20:32
Let's just see that come out
right.
318
00:20:32 --> 00:20:35
So what are the sigmas?
319
00:20:35 --> 00:20:40
They're the square roots of
these things.
320
00:20:40 --> 00:20:45
So square root of 32 and square
root of 18.
321
00:20:45 --> 00:20:46
Zero zero.
322
00:20:46 --> 00:20:47
Okay.
323
00:20:47 --> 00:20:49
What are the Vs?
324
00:20:49 --> 00:20:51
They're these two.
325
00:20:51 --> 00:20:58.12
And I have to transpose --
maybe that just leaves me with
326
00:20:58.12 --> 00:21:02
ones --
with one over square root of
327
00:21:02 --> 00:21:06
two in that row and the other
one is one over square root of
328
00:21:06 --> 00:21:09
two minus one over square root
of two.
329
00:21:09 --> 00:21:11
Now finally,
I've got to know the Us.
330
00:21:11 --> 00:21:14
Well, actually,
one way to do -- since I now
331
00:21:14 --> 00:21:18
know all the other pieces,
I could put those together and
332
00:21:18 --> 00:21:21
figure out what the Us are.
333
00:21:21 --> 00:21:26
But let me do it the A A
transpose way.
334
00:21:26 --> 00:21:26
Okay.
335
00:21:26 --> 00:21:30.11
Find the Us now.
u1 and u2.
336
00:21:30.11 --> 00:21:32
And what are they?
337
00:21:32 --> 00:21:39
I look at A A transpose -- so A
is supposed to be U sigma V
338
00:21:39 --> 00:21:46
transpose, and then when I
transpose that I get V sigma
339
00:21:46 --> 00:21:50
transpose U transpose.
340
00:21:50 --> 00:21:55
So I'm just doing it in the
opposite order,
341
00:21:55 --> 00:22:00
A times A transpose,
and what's the good part here?
342
00:22:00 --> 00:22:05
That in the middle,
V transpose V is going to be
343
00:22:05 --> 00:22:07
the identity.
344
00:22:07 --> 00:22:13
So this is just U sigma sigma
transpose, that's some diagonal
345
00:22:13 --> 00:22:19
matrix with sigma squareds and U
transpose.
346
00:22:19 --> 00:22:21
So what I seeing here?
347
00:22:21 --> 00:22:25.01
I'm seeing here,
again, a symmetric positive
348
00:22:25.01 --> 00:22:29
definite or at least
semi-definite matrix and I'm
349
00:22:29 --> 00:22:33
seeing its eigenvectors and its
eigenvalues.
350
00:22:33 --> 00:22:38
So if I compute A A transpose,
its eigenvectors will be the
351
00:22:38 --> 00:22:41
things that go into U.
352
00:22:41 --> 00:22:46
Okay, so I need to compute A A
transpose.
353
00:22:46 --> 00:22:53
I guess I'm going to have to go
-- can I move that up just a
354
00:22:53 --> 00:22:54
little?
355
00:22:54 --> 00:22:59
Maybe a little more and do A A
transpose.
356
00:22:59 --> 00:23:00
So what's A?
357
00:23:00 --> 00:23:05
Four four minus three and
three.
358
00:23:05 --> 00:23:08
And what's A transpose?
359
00:23:08 --> 00:23:12
Four four minus three and
three.
360
00:23:12 --> 00:23:16
And when I do that
multiplication,
361
00:23:16 --> 00:23:17
what do I get?
362
00:23:17 --> 00:23:21.61
Sixteen and sixteen,
thirty two.
363
00:23:21.61 --> 00:23:25
Uh, that one comes out zero.
364
00:23:25 --> 00:23:32
Oh, so this is a lucky case and
that one comes out 18.
365
00:23:32 --> 00:23:38
So this is an accident that A A
transpose happens to come out
366
00:23:38 --> 00:23:43
diagonal, so we know easily its
eigenvectors and eigenvalues.
367
00:23:43 --> 00:23:48
So its eigenvectors -- what's
the first eigenvector for this A
368
00:23:48 --> 00:23:50
A transpose matrix?
369
00:23:50 --> 00:23:53
It's just one zero,
and when I do that
370
00:23:53 --> 00:23:57
multiplication,
I get 32 times one zero.
371
00:23:57 --> 00:24:03
And the other eigenvector is
just zero one and when I
372
00:24:03 --> 00:24:06
multiply by that I get 18.
373
00:24:06 --> 00:24:09
So this is A A transpose.
374
00:24:09 --> 00:24:13
Multiplying that gives me the
32 A A transpose.
375
00:24:13 --> 00:24:15
Multiplying this guy gives me
376
00:24:16 --> 00:24:17
377
00:24:17 --> 00:24:21.2
First of all,
I got 32 and 18 again.
378
00:24:21.2 --> 00:24:22
Am I surprised?
379
00:24:22 --> 00:24:28.3
You know, it's clearly not an
accident.
380
00:24:28.3 --> 00:24:32
The eigenvalues of A A
transpose were exactly the same
381
00:24:32 --> 00:24:36
as the eigenvalues of -- this
one was A transpose A.
382
00:24:36 --> 00:24:39
That's no surprise at all.
383
00:24:39 --> 00:24:44
The eigenvalues of A B are the
same as the eigenvalues of B A.
384
00:24:44.1 --> 00:24:48
That's a very nice fact,
that eigenvalues stay the same
385
00:24:48 --> 00:24:52
if I switch the order of
multiplication.
386
00:24:52 --> 00:24:53
So no surprise to see 32 and
387
00:24:54 --> 00:24:55
388
00:24:55 --> 00:25:00
What I learned -- first the
check that things were
389
00:25:00 --> 00:25:04
numerically correct,
but now I've learned these
390
00:25:04 --> 00:25:08
eigenvectors,
and actually they're about as
391
00:25:08 --> 00:25:09.77
nice as can be.
392
00:25:09.77 --> 00:25:15
They're the best orthogonal
matrix, just the identity.
393
00:25:15 --> 00:25:16
Okay.
394
00:25:16 --> 00:25:22
So my claim is that it ought to
all fit together,
395
00:25:22 --> 00:25:27
that these numbers should come
out right.
396
00:25:27 --> 00:25:34
The numbers should come out
right because the matrix
397
00:25:34 --> 00:25:40
multiplications use the
properties that we want.
398
00:25:40 --> 00:25:40
Okay.
399
00:25:40 --> 00:25:45.44
Shall we just check that?
400
00:25:45.44 --> 00:25:48
Here's the identity,
so not doing anything -- square
401
00:25:48 --> 00:25:52.85
root of 32 is multiplying that
row, so that square root of 32
402
00:25:52.85 --> 00:25:56
divided by square root of two
means square root of 16,
403
00:25:56 --> 00:25:57
four, correct?
404
00:25:57 --> 00:26:00
And square root of 18 is
divided by square root of two,
405
00:26:00 --> 00:26:04
so that leaves me square root
of 9, which is three,
406
00:26:04 --> 00:26:08
but --
well, Professor Strang,
407
00:26:08 --> 00:26:11
you see the problem?
408
00:26:11 --> 00:26:14.66
Why is that -- okay.
409
00:26:14.66 --> 00:26:23
Why I getting minus three three
here and here I'm getting three
410
00:26:23 --> 00:26:25
minus three?
411
00:26:25 --> 00:26:26
Phooey.
412
00:26:26 --> 00:26:28
I don't know why.
413
00:26:28 --> 00:26:35
It shouldn't have happened,
but it did.
414
00:26:35 --> 00:26:40
Now, okay, you could say,
well, just -- the eigenvector
415
00:26:40 --> 00:26:46
there could have -- I could have
had the minus sign here for that
416
00:26:46 --> 00:26:50
eigenvector, but I'm not happy
about that.
417
00:26:50 --> 00:26:51
Hmm.
418
00:26:51 --> 00:26:51
Okay.
419
00:26:51 --> 00:26:56
So I realize there's a little
catch here somewhere and I may
420
00:26:56 --> 00:27:00
not see it until Wednesday.
421
00:27:00 --> 00:27:06
Which then gives you a very
important reason to come back on
422
00:27:06 --> 00:27:10
Wednesday, to catch that sine
difference.
423
00:27:10 --> 00:27:13
So what did I do illegally?
424
00:27:13 --> 00:27:18
I think I put the eigenvectors
in that matrix V transpose --
425
00:27:18 --> 00:27:22
okay, I'm going to have to
think.
426
00:27:22 --> 00:27:28
Why did that come out with with
the opposite sines?
427
00:27:28 --> 00:27:33
So you see -- I mean,
if I had a minus there,
428
00:27:33 --> 00:27:39
I would be all right,
but I don't want that.
429
00:27:39 --> 00:27:46
I want positive entries down
the diagonal of sigma squared.
430
00:27:46 --> 00:27:47
Okay.
431
00:27:47 --> 00:27:53
It'll come to me,
but, I'm going to leave this
432
00:27:53 --> 00:27:56
example to finish.
433
00:27:56 --> 00:27:57
Okay.
434
00:27:57 --> 00:28:03
And the beauty of,
these sliding boards is I can
435
00:28:03 --> 00:28:05
make that go away.
436
00:28:05 --> 00:28:10
Can I,-- let me not do it,
though, yet.
437
00:28:10 --> 00:28:14
Let me take a second example.
438
00:28:14 --> 00:28:22
Let me take a second example
where the matrix is singular.
439
00:28:22 --> 00:28:24
So rank one.
440
00:28:24 --> 00:28:33
Okay, so let me take as an
example two, where my matrix A
441
00:28:33 --> 00:28:41
is going to be rectangular again
-- let me just make it four
442
00:28:41 --> 00:28:43
three eight six.
443
00:28:43 --> 00:28:44
Okay.
444
00:28:44 --> 00:28:48
That's a rank one matrix.
445
00:28:48 --> 00:28:56
So that has a null space and
only a one dimensional row space
446
00:28:56 --> 00:29:00
and column space.
447
00:29:00 --> 00:29:04
So actually,
my picture becomes easy for
448
00:29:04 --> 00:29:10
this matrix, because what's my
row space for this one?
449
00:29:10 --> 00:29:12
So this is two by two.
450
00:29:12 --> 00:29:16
So my pictures are both two
dimensional.
451
00:29:16 --> 00:29:22
My row space is all multiples
of that vector four three.
452
00:29:22 --> 00:29:27
So the whole -- the row space
is just a line,
453
00:29:27 --> 00:29:28
right?
454
00:29:28 --> 00:29:30
That's the row space.
455
00:29:30 --> 00:29:35
And the null space,
of course, is the perpendicular
456
00:29:35 --> 00:29:36
line.
457
00:29:36 --> 00:29:41
So the row space for this
matrix is multiples of four
458
00:29:41 --> 00:29:42
three.
459
00:29:42 --> 00:29:43
Typical row.
460
00:29:43 --> 00:29:43.97
Okay.
461
00:29:43.97 --> 00:29:46
What's the column space?
462
00:29:46 --> 00:29:50.98
The columns are all multiples
of four eight,
463
00:29:50.98 --> 00:29:54
three six, one two.
464
00:29:54 --> 00:29:58
The column space,
then, goes in,
465
00:29:58 --> 00:30:00
like, this direction.
466
00:30:00 --> 00:30:05
So the column space -- when I
look at those columns,
467
00:30:05 --> 00:30:11.06
the column space -- so it's
only one dimensional,
468
00:30:11.06 --> 00:30:13.67
because the rank is one.
469
00:30:13.67 --> 00:30:17
It's multiples of four eight.
470
00:30:17 --> 00:30:18
Okay.
471
00:30:18 --> 00:30:23
And what's the null space of A
transpose?
472
00:30:23 --> 00:30:26
It's the perpendicular guy.
473
00:30:26 --> 00:30:33
So this was the null space of A
and this is the null space of A
474
00:30:33 --> 00:30:34
transpose.
475
00:30:34 --> 00:30:35
Okay.
476
00:30:35 --> 00:30:42
What I want to say here is that
choosing these orthogonal bases
477
00:30:42 --> 00:30:47
for the row space and the column
space is, like,
478
00:30:47 --> 00:30:50
no problem.
479
00:30:50 --> 00:30:53
They're only one dimensional.
480
00:30:53 --> 00:30:55
So what should V be?
481
00:30:55 --> 00:30:58.84
V should be -- v1,
but -- yes, v1,
482
00:30:58.84 --> 00:31:03
rather -- v1 is supposed to be
a unit vector.
483
00:31:03 --> 00:31:09
There's only one v1 to choose
here, only one dimension in the
484
00:31:09 --> 00:31:11
row space.
485
00:31:11 --> 00:31:15
I just want to make it a unit
vector.
486
00:31:15 --> 00:31:22
So v1 will be -- it'll be this
vector, but made into a unit
487
00:31:22 --> 00:31:27
vector, so four -- point eight
point six.
488
00:31:27 --> 00:31:30
Four fifths,
three fifths.
489
00:31:30 --> 00:31:35.46
And what will be u1?
u1 will be the unit vector
490
00:31:35.46 --> 00:31:37
there.
491
00:31:37 --> 00:31:41.95
So I want to turn four eight or
one two into a unit vector,
492
00:31:41.95 --> 00:31:45
so u1 will be -- let's see,
if it's one two,
493
00:31:45 --> 00:31:48
then what multiple of one two
do I want?
494
00:31:48 --> 00:31:52
That has length square root of
five, so I have to divide by
495
00:31:52 --> 00:31:54
square root of five.
496
00:31:54 --> 00:31:58
Let me complete the singular
value decomposition for this
497
00:31:58 --> 00:32:00
matrix.
498
00:32:00 --> 00:32:05
So this matrix,
four three eight six,
499
00:32:05 --> 00:32:15
is -- so I know what u1 --
here's A and I want to get U the
500
00:32:15 --> 00:32:19
basis in the column space.
501
00:32:19 --> 00:32:28
And it has to start with this
guy, one over square root of
502
00:32:28 --> 00:32:35
five two over square root of
five.
503
00:32:35 --> 00:32:38
Then I want the sigma.
504
00:32:38 --> 00:32:38
Okay.
505
00:32:38 --> 00:32:43
What are we expecting now for
sigma?
506
00:32:43 --> 00:32:47
This is only a rank one matrix.
507
00:32:47 --> 00:32:54
We're only expecting a sigma
one, which I have to find,
508
00:32:54 --> 00:32:56
but zeroes here.
509
00:32:56 --> 00:32:57
Okay.
510
00:32:57 --> 00:32:59
So what's sigma one?
511
00:32:59 --> 00:33:07
It should be the -- where did
these sigmas come from?
512
00:33:07 --> 00:33:14.22
They came from A transpose A,
so I -- can I do that little
513
00:33:14.22 --> 00:33:16
calculation over here?
514
00:33:16 --> 00:33:23
A transpose A is four three --
four three eight six times four
515
00:33:23 --> 00:33:25
three eight six.
516
00:33:25 --> 00:33:30
This had better -- this is a
rank one matrix,
517
00:33:30 --> 00:33:36
this is going to be --
the whole thing will have rank
518
00:33:36 --> 00:33:41
one, that's 16 and 64 is 80,
12 and 48 is 60,
519
00:33:41 --> 00:33:44
12 and 48 is 60,
9 and 36 is 45.
520
00:33:44 --> 00:33:45
Okay.
521
00:33:45 --> 00:33:47
It's a rank one matrix.
522
00:33:47 --> 00:33:48
Of course.
523
00:33:48 --> 00:33:52
Every row is a multiple of four
three.
524
00:33:52 --> 00:33:58
And what's the eigen -- what
are the eigenvalues of that
525
00:33:58 --> 00:34:00
matrix?
526
00:34:00 --> 00:34:04
So this is the calculation --
this is like practicing,
527
00:34:04 --> 00:34:04
now.
528
00:34:04 --> 00:34:07
What are the eigenvalues of
this rank one matrix?
529
00:34:07 --> 00:34:11
Well, tell me one eigenvalue,
since the rank is only one,
530
00:34:11 --> 00:34:14
one eigenvalue is going to be
zero.
531
00:34:14 --> 00:34:18.18
And then you know that the
other eigenvalue is going to be
532
00:34:18.18 --> 00:34:20
a hundred and twenty five.
533
00:34:20 --> 00:34:26
So that's sigma squared,
right, in A transpose A.
534
00:34:26 --> 00:34:34
So this will be the square root
of a hundred and twenty five.
535
00:34:34 --> 00:34:40
And then finally,
the V transpose -- the Vs will
536
00:34:40 --> 00:34:44
be -- there's v1,
and what's v2?
537
00:34:44 --> 00:34:51.09
What's v2 in the -- how do I
make this into an orthonormal
538
00:34:51.09 --> 00:34:53
basis?
539
00:34:53 --> 00:34:56
Well, v2 is,
in the null space direction.
540
00:34:56 --> 00:35:02
It's perpendicular to that,
so point six and minus point
541
00:35:02 --> 00:35:02
eight.
542
00:35:02 --> 00:35:06
So those are the Vs that go in
here.
543
00:35:06 --> 00:35:09.99
Point eight,
point six and point six minus
544
00:35:09.99 --> 00:35:11
point eight.
545
00:35:11 --> 00:35:11
Okay.
546
00:35:11 --> 00:35:15
And I guess I better finish
this guy.
547
00:35:15 --> 00:35:18
So this guy,
all I want is to complete the
548
00:35:18 --> 00:35:24
orthonormal basis --
it'll be coming from there.
549
00:35:24 --> 00:35:30
It'll be a two over square root
of five and a minus one over
550
00:35:30 --> 00:35:32
square root of five.
551
00:35:32 --> 00:35:38
Let me take square root of five
out of that matrix to make it
552
00:35:38 --> 00:35:39
look better.
553
00:35:39 --> 00:35:45
So one over square root of five
times one two two minus one.
554
00:35:45 --> 00:35:46
Okay.
555
00:35:46 --> 00:35:52
So there I have -- including
the square root of five --
556
00:35:52 --> 00:35:58
that's an orthogonal matrix,
that's an orthogonal matrix,
557
00:35:58 --> 00:36:03
that's a diagonal matrix and
its rank is only one.
558
00:36:03 --> 00:36:07
And now if I do that
multiplication,
559
00:36:07 --> 00:36:11
I pray that it comes out right.
560
00:36:11 --> 00:36:17
The square root of five will
cancel into that square root of
561
00:36:17 --> 00:36:22
one twenty five and leave me
with the square root of 25,
562
00:36:22 --> 00:36:26
which is five,
and five will multiply these
563
00:36:26 --> 00:36:31
numbers and I'll get whole
numbers and out will come A.
564
00:36:31 --> 00:36:32
Okay.
565
00:36:32 --> 00:36:39
That's like a second example
showing how the null space guy
566
00:36:39 --> 00:36:46
-- so this -- this vector and
this one were multiplied by this
567
00:36:46 --> 00:36:47
zero.
568
00:36:47 --> 00:36:50
So they were easy to deal with.
569
00:36:50 --> 00:36:57
Tthe key ones are the ones in
the column space and the row
570
00:36:57 --> 00:36:59
space.
571
00:36:59 --> 00:37:03
Do you see how I'm getting
columns here,
572
00:37:03 --> 00:37:08
diagonal here,
rows here, coming together to
573
00:37:08 --> 00:37:09
produce A.
574
00:37:09 --> 00:37:13
Okay, that's the singular value
decomposition.
575
00:37:13 --> 00:37:19
So, let me think what I want to
add to complete this topic.
576
00:37:19 --> 00:37:22
So that's two examples.
577
00:37:22 --> 00:37:28.13
And now let's think what we're
really doing.
578
00:37:28.13 --> 00:37:41
We're choosing the right basis
for the four subspaces of linear
579
00:37:41 --> 00:37:43
algebra.
580
00:37:43 --> 00:37:48
Let me write this down.
581
00:37:48 --> 00:37:59
So v1 up to vr is an
orthonormal basis for the row
582
00:37:59 --> 00:38:06
space.
u1 up to ur is an orthonormal
583
00:38:06 --> 00:38:14
basis for the column space.
584
00:38:14 --> 00:38:20
And then I just finish those
out by v(r+1),
585
00:38:20 --> 00:38:28
the rest up to vn is an
orthonormal basis for the null
586
00:38:28 --> 00:38:29
space.
587
00:38:29 --> 00:38:35
And finally,
u(r+1) up to is an orthonormal
588
00:38:35 --> 00:38:41
basis for the null space of A
transpose.
589
00:38:41 --> 00:38:49
Do you see that we finally got
the bases right?
590
00:38:49 --> 00:38:55
They're right because they're
orthonormal, and also -- again,
591
00:38:55 --> 00:39:01
Graham Schmidt would have done
this in chapter four.
592
00:39:01 --> 00:39:07
Here we needed eigenvalues,
because these bases make the
593
00:39:07 --> 00:39:08
matrix diagonal.
594
00:39:08 --> 00:39:12
A times V I is a multiple of U
I.
595
00:39:12 --> 00:39:19
So I'll put "and" -- the matrix
has been made diagonal.
596
00:39:19 --> 00:39:24
When we choose these bases,
there's no coupling between Vs
597
00:39:24 --> 00:39:26.8
and no coupling between Us.
598
00:39:26.8 --> 00:39:31
Each A -- A times each V is in
the direction of the
599
00:39:31 --> 00:39:32
corresponding U.
600
00:39:32 --> 00:39:37.76
So it's exactly the right basis
for the four fundamental
601
00:39:37.76 --> 00:39:38
subspaces.
602
00:39:38 --> 00:39:42.47
And of course,
their dimensions are what we
603
00:39:42.47 --> 00:39:44.01
know.
604
00:39:44.01 --> 00:39:47.65
The dimension of the row space
is the rank r,
605
00:39:47.65 --> 00:39:51
and so is the dimension of the
column space.
606
00:39:51 --> 00:39:56
The dimension of the null space
is n-r, that's how many vectors
607
00:39:56 --> 00:40:00
we need, and m-r basis vectors
for the left null space,
608
00:40:00 --> 00:40:04
the null space of A transpose.
609
00:40:04 --> 00:40:04
Okay.
610
00:40:04 --> 00:40:06
I'm going to stop there.
611
00:40:06 --> 00:40:11.39
I could develop further from
the SVD, but we'll see it again
612
00:40:11.39 --> 00:40:14
in the very last lectures of the
course.
613
00:40:14 --> 00:40:16
So there's the SVD.
614
00:40:16 --> 00:40:19
Thanks.