1
00:00:11 --> 00:00:12
Okay.

2
00:00:12 --> 00:00:19
This is the lecture on the
singular value decomposition.

3
00:00:19 --> 00:00:23
But everybody calls it the SVD.

4
00:00:23 --> 00:00:31
So this is the final and best
factorization of a matrix.

5
00:00:31 --> 00:00:35
Let me tell you what's coming.

6
00:00:35 --> 00:00:40
The factors will be,
orthogonal matrix,

7
00:00:40 --> 00:00:46
diagonal matrix,
orthogonal matrix.

8
00:00:46 --> 00:00:50
So it's things that we've seen
before, these special good

9
00:00:50 --> 00:00:52
matrices, orthogonal diagonal.

10
00:00:52 --> 00:00:55
The new point is that we need
two orthogonal matrices.

11
00:00:55 --> 00:00:57
A can be any matrix whatsoever.

12
00:00:57 --> 00:01:01
Any matrix whatsoever has this
singular value decomposition,

13
00:01:01 --> 00:01:05.48
so a diagonal one in the
middle, but I need two different

14
00:01:05.48 --> 00:01:10
-- probably different orthogonal
matrices to be able to do this.

15
00:01:10 --> 00:01:10
Okay.

16
00:01:10 --> 00:01:16
And this factorization has
jumped into importance and is

17
00:01:16 --> 00:01:21.1
properly, I think,
maybe the bringing together of

18
00:01:21.1 --> 00:01:23.65
everything in this course.

19
00:01:23.65 --> 00:01:29
One thing we'll bring together
is the very good family of

20
00:01:29 --> 00:01:34
matrices that we just studied,
symmetric, positive,

21
00:01:34 --> 00:01:36
definite.

22
00:01:36 --> 00:01:39
Do you remember the stories
with those guys?

23
00:01:39 --> 00:01:44
Because they were symmetric,
their eigenvectors were

24
00:01:44 --> 00:01:50.37
orthogonal, so I could produce
an orthogonal matrix -- this is

25
00:01:50.37 --> 00:01:51
my usual one.

26
00:01:51 --> 00:01:56
My usual one is the
eigenvectors and eigenvalues

27
00:01:56 --> 00:02:00
In the symmetric case,
the eigenvectors are

28
00:02:00 --> 00:02:07
orthogonal, so I've got the good
-- my ordinary s has become an

29
00:02:07 --> 00:02:08
especially good Q.

30
00:02:08 --> 00:02:14
And positive definite,
my ordinary lambda has become a

31
00:02:14 --> 00:02:16
positive lambda.

32
00:02:16 --> 00:02:21.96
So that's the singular value
decomposition in case our matrix

33
00:02:21.96 --> 00:02:26
is symmetric positive definite
-- in that case,

34
00:02:26 --> 00:02:31
I don't need two -- U and a V
-- one orthogonal matrix will do

35
00:02:31 --> 00:02:33
for both sides.

36
00:02:33 --> 00:02:40
So this would be no good in
general, because usually the

37
00:02:40 --> 00:02:44
eigenvector matrix isn't
orthogonal.

38
00:02:44 --> 00:02:47.66
So that's not what I'm after.

39
00:02:47.66 --> 00:02:55
I'm looking for orthogonal
times diagonal times orthogonal.

40
00:02:55 --> 00:03:01
And let me show you what that
means and where it comes from.

41
00:03:01 --> 00:03:02
Okay.

42
00:03:02 --> 00:03:03
What does it mean?

43
00:03:03 --> 00:03:09
You remember the picture of any
linear transformation.

44
00:03:09 --> 00:03:13
This was, like,
the most important figure in

45
00:03:13 --> 00:03:14


46
00:03:14 --> 00:03:14


47
00:03:14 --> 00:03:17
And what I looking for now?

48
00:03:17 --> 00:03:21.89
A typical vector in the row
space

49
00:03:21 --> 00:03:27
-- typical vector,
let me call it v1,

50
00:03:27.35 --> 00:03:34
gets taken over to some vector
in the column space,

51
00:03:34 --> 00:03:35
say u1.

52
00:03:35 --> 00:03:37
So u1 is Av1.

53
00:03:37 --> 00:03:38
Okay.

54
00:03:38 --> 00:03:46
Now, another vector gets taken
over here somewhere.

55
00:03:46 --> 00:03:50
What I looking for?

56
00:03:50 --> 00:03:53
In this SVD,
this singular value

57
00:03:53 --> 00:03:57.66
decomposition,
what I'm looking for is an

58
00:03:57.66 --> 00:04:03
orthogonal basis here that gets
knocked over into an orthogonal

59
00:04:03 --> 00:04:05
basis over there.

60
00:04:05 --> 00:04:10
See that's pretty special,
to have an orthogonal basis in

61
00:04:10 --> 00:04:17.35
the row space that goes over
into an orthogonal basis --

62
00:04:17.35 --> 00:04:21
so this is like a right angle
and this is a right angle --

63
00:04:21 --> 00:04:25
into an orthogonal basis in the
column space.

64
00:04:25 --> 00:04:28
So that's our goal,
is to find -- do you see how

65
00:04:28 --> 00:04:31
things are coming together?

66
00:04:31 --> 00:04:34
First of all,
can I find an orthogonal basis

67
00:04:34 --> 00:04:36.75
for this row space?

68
00:04:36.75 --> 00:04:37
Of course.

69
00:04:37 --> 00:04:39
No big deal to find an
orthogonal basis.

70
00:04:39 --> 00:04:42
Graham Schmidt tells me how to
do it.

71
00:04:42 --> 00:04:45
Start with any old basis and
grind through Graham Schmidt,

72
00:04:45 --> 00:04:47
out comes an orthogonal basis.

73
00:04:47 --> 00:04:50.61
But then, if I just take any
old orthogonal basis,

74
00:04:50.61 --> 00:04:54
then when I multiply by A,
there's no reason why it should

75
00:04:54 --> 00:04:56
be orthogonal over here.

76
00:04:56 --> 00:05:04
So I'm looking for this special
set up where A takes these basis

77
00:05:04 --> 00:05:10
vectors into orthogonal vectors
over there.

78
00:05:10 --> 00:05:17
Now, you might have noticed
that the null space I didn't

79
00:05:17 --> 00:05:18
include.

80
00:05:18 --> 00:05:22
Why don't I stick that in?

81
00:05:22 --> 00:05:27
You remember our usual figure
had a little null space and a

82
00:05:27 --> 00:05:29
little null space.

83
00:05:29 --> 00:05:31
And those are no problems.

84
00:05:31 --> 00:05:36
Those null spaces are going to
show up as zeroes on the

85
00:05:36 --> 00:05:40
diagonal of sigma,
so that doesn't present any

86
00:05:40 --> 00:05:41
difficulty.

87
00:05:41 --> 00:05:44
Our difficulty is to find
these.

88
00:05:44 --> 00:05:48.64
So do you see what this will
mean?

89
00:05:48.64 --> 00:05:53
This will mean that A times
these v-s, v1,

90
00:05:53 --> 00:05:59
v2, up to -- what's the
dimension of this row space?

91
00:05:59 --> 00:05:59
Vr.

92
00:05:59 --> 00:06:04
Sorry, make that V a little
smaller -- up to vr.

93
00:06:04 --> 00:06:10
So that's -- Av1 is going to be
the first column,

94
00:06:10 --> 00:06:15
so here's what I'm achieving.

95
00:06:15 --> 00:06:20
Oh, I'm not only going to make
these orthogonal,

96
00:06:20 --> 00:06:24.41
but why not make them
orthonormal?

97
00:06:24.41 --> 00:06:27
Make them unit vectors.

98
00:06:27 --> 00:06:32
So maybe the unit vector is
here, is the u1,

99
00:06:32 --> 00:06:37
and this might be a multiple of
it.

100
00:06:37 --> 00:06:42
So really, what's happening is
Av1 is some multiple of u1,

101
00:06:42 --> 00:06:42
right?

102
00:06:42 --> 00:06:47
These guys will be unit vectors
and they'll go over into

103
00:06:47 --> 00:06:53
multiples of unit vectors and
the multiple I'm not going to

104
00:06:53 --> 00:06:55
call lambda anymore.

105
00:06:55 --> 00:06:57
I'm calling it sigma.

106
00:06:57 --> 00:07:01
So that's the number -- the
stretching number.

107
00:07:01 --> 00:07:05
And similarly,
Av2 is sigma two u2.

108
00:07:05 --> 00:07:06
This is my goal.

109
00:07:06 --> 00:07:11
And now I want to express that
goal in matrix language.

110
00:07:11 --> 00:07:14
That's the usual step.

111
00:07:14 --> 00:07:20
Think of what you want and then
express it as a matrix

112
00:07:20 --> 00:07:21
multiplication.

113
00:07:21 --> 00:07:26
So Av1 is sigma one u1 --
actually, here we go.

114
00:07:26 --> 00:07:32
Let me pull out these -- u1,
u2 to ur and then a matrix with

115
00:07:32 --> 00:07:33.97
the sigmas.

116
00:07:33.97 --> 00:07:39
Everything now is going to be
in that little part of the

117
00:07:39 --> 00:07:42
blackboard.

118
00:07:42 --> 00:07:50.02
Do you see that this equation
says what I'm trying to do with

119
00:07:50.02 --> 00:07:51
my figure.

120
00:07:51 --> 00:07:59
A times the first basis vector
should be sigma one times the

121
00:07:59 --> 00:08:05
other basis -- the other first
basis vector.

122
00:08:05 --> 00:08:09
These are the basis vectors in
the row space,

123
00:08:09 --> 00:08:14
these are the basis vectors in
the column space and these are

124
00:08:14 --> 00:08:16
the multiplying factors.

125
00:08:16 --> 00:08:21
So Av2 is sigma two times u2,
Avr is sigma r times ur.

126
00:08:21 --> 00:08:26
And then we've got a whole lot
of zeroes and maybe some zeroes

127
00:08:26 --> 00:08:30
at the end, but that's the heart
of it.

128
00:08:30 --> 00:08:36
And now if I express that in --
as matrices, because you knew

129
00:08:36 --> 00:08:40
that was coming -- that's what I
have.

130
00:08:40 --> 00:08:45
So, this is my goal,
to find an orthogonal basis in

131
00:08:45 --> 00:08:50
the orthonormal,
even -- basis in the row space

132
00:08:50 --> 00:08:56
and an orthonormal basis in the
column space so that I've sort

133
00:08:56 --> 00:09:00
of diagonalized the matrix.

134
00:09:00 --> 00:09:03
The matrix A is,
like, getting converted to this

135
00:09:03 --> 00:09:04
diagonal matrix sigma.

136
00:09:04 --> 00:09:08
And you notice that usually I
have to allow myself two

137
00:09:08 --> 00:09:09
different bases.

138
00:09:09 --> 00:09:13
My little comment about
symmetric positive definite was

139
00:09:13 --> 00:09:15
the one case where it's A Q
equal Q sigma,

140
00:09:15 --> 00:09:18
where V and U are the same Q.

141
00:09:18 --> 00:09:25
But mostly, you know,
I'm going to take a matrix like

142
00:09:25 --> 00:09:33
-- oh, let me take a matrix like
four four minus three three.

143
00:09:33 --> 00:09:34
Okay.

144
00:09:34 --> 00:09:38.36
There's a two by two matrix.

145
00:09:38.36 --> 00:09:44
It's invertible,
so it has rank two.

146
00:09:44 --> 00:09:49
So I'm going to look for two
vectors, v1 and v2 in the row

147
00:09:49 --> 00:09:54
space, and U -- so I'm going to
look for v1, v2 in the row

148
00:09:54 --> 00:09:56
space, which of course is R^2.

149
00:09:56 --> 00:10:01
And I'm going to look for u1,
u2 in the column space,

150
00:10:01 --> 00:10:06
which of course is also R^2,
and I'm going to look for

151
00:10:06 --> 00:10:12
numbers sigma one and sigma two
so that it all comes out right.

152
00:10:12 --> 00:10:20
So these guys are orthonormal,
these guys are orthonormal and

153
00:10:20 --> 00:10:23
these are the scaling factors.

154
00:10:23 --> 00:10:30
So I'll do that example as soon
as I get the matrix picture

155
00:10:30 --> 00:10:31
straight.

156
00:10:31 --> 00:10:32
Okay.

157
00:10:32 --> 00:10:38
Do you see that this expresses
what I want?

158
00:10:38 --> 00:10:43
Can I just say two words about
null spaces?

159
00:10:43 --> 00:10:48
If there's some null space,
then we want to stick in a

160
00:10:48 --> 00:10:50
basis for those,
for that.

161
00:10:50 --> 00:10:56
So here comes a basis for the
null space, v(r+1) down to vm.

162
00:10:56 --> 00:11:01
So if we only had an r
dimensional row space and the

163
00:11:01 --> 00:11:07
other n-r dimensions were in the
null space --

164
00:11:07 --> 00:11:12
okay, we'll take an orthogonal
-- orthonormal basis there.

165
00:11:12 --> 00:11:12
No problem.

166
00:11:12 --> 00:11:15
And then we'll just get zeroes.

167
00:11:15 --> 00:11:19
So, actually,
w- those zeroes will come out

168
00:11:19 --> 00:11:21
on the diagonal matrix.

169
00:11:21 --> 00:11:26
So I'll complete that to an
orthonormal basis for the whole

170
00:11:26 --> 00:11:28
space, R^m.

171
00:11:28 --> 00:11:32
I complete this to an
orthonormal basis for the whole

172
00:11:32 --> 00:11:35
space R^n and I complete that
with zeroes.

173
00:11:35 --> 00:11:38
Null spaces are no problem
here.

174
00:11:38 --> 00:11:42.66
So really the true problem is
in a matrix like that,

175
00:11:42.66 --> 00:11:45.98
which isn't symmetric,
so I can't use its

176
00:11:45.98 --> 00:11:49
eigenvectors,
they're not orthogonal -- but

177
00:11:49 --> 00:11:53
somehow I have to get these
orthogonal --

178
00:11:53 --> 00:12:00
in fact, orthonormal guys that
make it work.

179
00:12:00 --> 00:12:06
I have to find these
orthonormal guys,

180
00:12:06 --> 00:12:14
these orthonormal guys and I
want Av1 to be sigma one u1 and

181
00:12:14 --> 00:12:19
Av2 to be sigma two u2.

182
00:12:19 --> 00:12:19
Okay.

183
00:12:19 --> 00:12:21.31
That's my goal.

184
00:12:21.31 --> 00:12:26
Here's the matrices that are
going to get me there.

185
00:12:26 --> 00:12:29.17
Now these are orthogonal
matrices.

186
00:12:29.17 --> 00:12:34
I can put that -- if I multiply
on both sides by V inverse,

187
00:12:34 --> 00:12:39
I have A equals U sigma V
inverse, and of course you know

188
00:12:39 --> 00:12:44
the other way I can write V
inverse.

189
00:12:44 --> 00:12:52
This is one of those square
orthogonal matrices,

190
00:12:52 --> 00:12:58
so it's the same as U sigma V
transpose.

191
00:12:58 --> 00:12:59
Okay.

192
00:12:59 --> 00:13:02
Here's my problem.

193
00:13:02 --> 00:13:08
I've got two orthogonal
matrices here.

194
00:13:08 --> 00:13:16
And I don't want to find them
both at once.

195
00:13:16 --> 00:13:21
So I want to cook up some
expression that will make the Us

196
00:13:21 --> 00:13:22
disappear.

197
00:13:22 --> 00:13:27
I would like to make the Us
disappear and leave me only with

198
00:13:27 --> 00:13:28
the Vs.

199
00:13:28 --> 00:13:30
And here's how to do it.

200
00:13:30 --> 00:13:35
It's the same combination that
keeps showing up whenever we

201
00:13:35 --> 00:13:40
have a general rectangular
matrix, then it's A transpose A,

202
00:13:40 --> 00:13:43
that's the great matrix.

203
00:13:43 --> 00:13:45
That's the great matrix.

204
00:13:45 --> 00:13:49
That's the matrix that's
symmetric, and in fact positive

205
00:13:49 --> 00:13:52.92
definite or at least positive
semi-definite.

206
00:13:52.92 --> 00:13:57
This is the matrix with nice
properties, so let's see what

207
00:13:57 --> 00:13:57
will it be?

208
00:13:57 --> 00:14:01
So if I took the transpose,
then, I would have -- A

209
00:14:01 --> 00:14:04.28
transpose A will be what?

210
00:14:04.28 --> 00:14:05
What do I have?

211
00:14:05 --> 00:14:11
If I transpose that I have V
sigma transpose U transpose,

212
00:14:11 --> 00:14:13
that's the A transpose.

213
00:14:13 --> 00:14:17.05
Now the A -- and what have I
got?

214
00:14:17.05 --> 00:14:21
Looks like worse,
because it's got six things now

215
00:14:21 --> 00:14:27.59
together, but it's going to
collapse into something good.

216
00:14:27.59 --> 00:14:32
What does U transpose U
collapse into?

217
00:14:32 --> 00:14:33
I, the identity.

218
00:14:33 --> 00:14:35
So that's the key point.

219
00:14:35 --> 00:14:38
This is the identity and we
don't have U anymore.

220
00:14:38 --> 00:14:42
And sigma transpose times
sigma, those are diagonal

221
00:14:42 --> 00:14:45
matrixes, so their product is
just going to have sigma

222
00:14:45 --> 00:14:47
squareds on the diagonal.

223
00:14:47 --> 00:14:49
So do you see what we've got
here?

224
00:14:49 --> 00:14:53
This is V times this easy
matrix sigma one squared sigma

225
00:14:53 --> 00:14:56
two squared times V transpose.

226
00:14:56 --> 00:15:00
This is the A transpose A.

227
00:15:00 --> 00:15:07
This is -- let me copy down --
A transpose A is that.

228
00:15:07 --> 00:15:11
Us are out of the picture,
now.

229
00:15:11 --> 00:15:18
I'm only having to choose the
Vs, and what are these Vs?

230
00:15:18 --> 00:15:23
And what are these sigmas?

231
00:15:23 --> 00:15:25.6
Do you know what the Vs are?

232
00:15:25.6 --> 00:15:30
They're the eigenvectors that
-- see, this is a perfect

233
00:15:30 --> 00:15:35.32
eigenvector, eigenvalue,
Q lambda Q transpose for the

234
00:15:35.32 --> 00:15:37
matrix A transpose A.

235
00:15:37 --> 00:15:39
A itself is nothing special.

236
00:15:39 --> 00:15:42
But A transpose A will be
special.

237
00:15:42 --> 00:15:47
It'll be symmetric positive
definite, so this will be its

238
00:15:47 --> 00:15:53
eigenvectors and this'll be its
eigenvalues.

239
00:15:53 --> 00:15:58
And the eigenvalues'll be
positive because this thing's

240
00:15:58 --> 00:16:00
positive definite.

241
00:16:00 --> 00:16:02
So this is my method.

242
00:16:02 --> 00:16:05
This tells me what the Vs are.

243
00:16:05 --> 00:16:09
And how I going to find the Us?

244
00:16:09 --> 00:16:13
Well, one way would be to look
at A A transpose.

245
00:16:13 --> 00:16:19
Multiply A by A transpose in
the opposite order.

246
00:16:19 --> 00:16:24.48
That will stick the Vs in the
middle, knock them out,

247
00:16:24.48 --> 00:16:26
and leave me with the Us.

248
00:16:26 --> 00:16:29
So here's the overall picture,
then.

249
00:16:29 --> 00:16:33
The Vs are the eigenvectors of
A transpose A.

250
00:16:33 --> 00:16:37
The Us are the eigenvectors of
A A transpose,

251
00:16:37 --> 00:16:40
which are different.

252
00:16:40 --> 00:16:47.09
And the sigmas are the square
roots of these and the positive

253
00:16:47.09 --> 00:16:51
square roots,
so we have positive sigmas.

254
00:16:51 --> 00:16:54
Let me do it for that example.

255
00:16:54 --> 00:17:01
This is really what you should
know and be able to do for the

256
00:17:01 --> 00:17:01.57
SVD.

257
00:17:01.57 --> 00:17:02.11
Okay.

258
00:17:02.11 --> 00:17:04
Let me take that matrix.

259
00:17:04 --> 00:17:08
So what's my first step?

260
00:17:08 --> 00:17:12
Compute A transpose A,
because I want its

261
00:17:12 --> 00:17:13
eigenvectors.

262
00:17:13 --> 00:17:13
Okay.

263
00:17:13 --> 00:17:16
So I have to compute A
transpose A.

264
00:17:16 --> 00:17:20
So A transpose is four four
minus three three,

265
00:17:20 --> 00:17:25
and A is four four minus three
three, and I do that

266
00:17:25 --> 00:17:30
multiplication and I get sixteen
-- I get twenty five -- I get

267
00:17:30 --> 00:17:35
sixteen minus nine -- is that
seven?

268
00:17:35 --> 00:17:38
And it better come out
symmetric.

269
00:17:38 --> 00:17:42
And -- oh, okay,
and then it comes out 25.

270
00:17:42 --> 00:17:42
Okay.

271
00:17:42 --> 00:17:47
So, I want its eigenvectors and
its eigenvalues.

272
00:17:47 --> 00:17:53
Its eigenvectors will be the
Vs, its eigenvalues will be the

273
00:17:53 --> 00:17:55
squares of the sigmas.

274
00:17:55 --> 00:17:55
Okay.

275
00:17:55 --> 00:18:02
What are the eigenvalues and
eigenvectors of this guy?

276
00:18:02 --> 00:18:09
Have you seen that two by two
example enough to recognize that

277
00:18:09 --> 00:18:16
the eigenvectors are -- that one
one is an eigenvector?

278
00:18:16 --> 00:18:19
So this here is A transpose A.

279
00:18:19 --> 00:18:24
I'm looking for its
eigenvectors.

280
00:18:24 --> 00:18:29
So its eigenvectors,
I think, are one one and one

281
00:18:29 --> 00:18:34
minus one, because if I multiply
that matrix by one one,

282
00:18:34 --> 00:18:36
what do I get?

283
00:18:36 --> 00:18:41
If I multiply that matrix by
one one, I get 32 32,

284
00:18:41 --> 00:18:43
which is 32 of one one.

285
00:18:43 --> 00:18:48
So there's the first
eigenvector, and there's the

286
00:18:48 --> 00:18:52
eigenvalue for A transpose A.

287
00:18:52 --> 00:18:56
So I'm going to take its square
root for sigma.

288
00:18:56 --> 00:18:57
Okay.

289
00:18:57 --> 00:19:02
What's the eigenvector that
goes -- eigenvalue that goes

290
00:19:02 --> 00:19:04
with this one?

291
00:19:04 --> 00:19:08
If I do that multiplication,
what do I get?

292
00:19:08 --> 00:19:13
I get some multiple of one
minus one, and what is that

293
00:19:13 --> 00:19:14
multiple?

294
00:19:14 --> 00:19:16
Looks like 18.

295
00:19:16 --> 00:19:17
Okay.

296
00:19:17 --> 00:19:21
So those are the two
eigenvectors,

297
00:19:21 --> 00:19:26
but -- oh, just a moment,
I didn't normalize them.

298
00:19:26 --> 00:19:33
To make everything absolutely
right, I ought to normalize

299
00:19:33 --> 00:19:37
these eigenvectors,
divide by their length,

300
00:19:37 --> 00:19:40
square root of two.

301
00:19:40 --> 00:19:46
So all these guys should be
true unit vectors and,

302
00:19:46 --> 00:19:53
of course, that normalization
didn't change the 32 and the 18.

303
00:19:53 --> 00:19:54
Okay.

304
00:19:54 --> 00:19:57
So I'm happy with the Vs.

305
00:19:57 --> 00:19:59
Here are the Vs.

306
00:19:59 --> 00:20:04
So now let me put together the
pieces here.

307
00:20:04 --> 00:20:05
Here's my A.

308
00:20:05 --> 00:20:06
Here's my A.

309
00:20:06 --> 00:20:11
Let me write down A again.

310
00:20:11 --> 00:20:14
If life is right,
we should get U,

311
00:20:14 --> 00:20:18
which I don't yet know -- U I
don't yet know,

312
00:20:18 --> 00:20:20
sigma I do now know.

313
00:20:20 --> 00:20:21
What's sigma?

314
00:20:21 --> 00:20:25.35
So I'm looking for a U sigma V
transpose.

315
00:20:25.35 --> 00:20:28
U, the diagonal guy and V
transpose.

316
00:20:28 --> 00:20:29
Okay.

317
00:20:29 --> 00:20:32
Let's just see that come out
right.

318
00:20:32 --> 00:20:35
So what are the sigmas?

319
00:20:35 --> 00:20:40
They're the square roots of
these things.

320
00:20:40 --> 00:20:45
So square root of 32 and square
root of 18.

321
00:20:45 --> 00:20:46
Zero zero.

322
00:20:46 --> 00:20:47
Okay.

323
00:20:47 --> 00:20:49
What are the Vs?

324
00:20:49 --> 00:20:51
They're these two.

325
00:20:51 --> 00:20:58.12
And I have to transpose --
maybe that just leaves me with

326
00:20:58.12 --> 00:21:02
ones --
with one over square root of

327
00:21:02 --> 00:21:06
two in that row and the other
one is one over square root of

328
00:21:06 --> 00:21:09
two minus one over square root
of two.

329
00:21:09 --> 00:21:11
Now finally,
I've got to know the Us.

330
00:21:11 --> 00:21:14
Well, actually,
one way to do -- since I now

331
00:21:14 --> 00:21:18
know all the other pieces,
I could put those together and

332
00:21:18 --> 00:21:21
figure out what the Us are.

333
00:21:21 --> 00:21:26
But let me do it the A A
transpose way.

334
00:21:26 --> 00:21:26
Okay.

335
00:21:26 --> 00:21:30.11
Find the Us now.
u1 and u2.

336
00:21:30.11 --> 00:21:32
And what are they?

337
00:21:32 --> 00:21:39
I look at A A transpose -- so A
is supposed to be U sigma V

338
00:21:39 --> 00:21:46
transpose, and then when I
transpose that I get V sigma

339
00:21:46 --> 00:21:50
transpose U transpose.

340
00:21:50 --> 00:21:55
So I'm just doing it in the
opposite order,

341
00:21:55 --> 00:22:00
A times A transpose,
and what's the good part here?

342
00:22:00 --> 00:22:05
That in the middle,
V transpose V is going to be

343
00:22:05 --> 00:22:07
the identity.

344
00:22:07 --> 00:22:13
So this is just U sigma sigma
transpose, that's some diagonal

345
00:22:13 --> 00:22:19
matrix with sigma squareds and U
transpose.

346
00:22:19 --> 00:22:21
So what I seeing here?

347
00:22:21 --> 00:22:25.01
I'm seeing here,
again, a symmetric positive

348
00:22:25.01 --> 00:22:29
definite or at least
semi-definite matrix and I'm

349
00:22:29 --> 00:22:33
seeing its eigenvectors and its
eigenvalues.

350
00:22:33 --> 00:22:38
So if I compute A A transpose,
its eigenvectors will be the

351
00:22:38 --> 00:22:41
things that go into U.

352
00:22:41 --> 00:22:46
Okay, so I need to compute A A
transpose.

353
00:22:46 --> 00:22:53
I guess I'm going to have to go
-- can I move that up just a

354
00:22:53 --> 00:22:54
little?

355
00:22:54 --> 00:22:59
Maybe a little more and do A A
transpose.

356
00:22:59 --> 00:23:00
So what's A?

357
00:23:00 --> 00:23:05
Four four minus three and
three.

358
00:23:05 --> 00:23:08
And what's A transpose?

359
00:23:08 --> 00:23:12
Four four minus three and
three.

360
00:23:12 --> 00:23:16
And when I do that
multiplication,

361
00:23:16 --> 00:23:17
what do I get?

362
00:23:17 --> 00:23:21.61
Sixteen and sixteen,
thirty two.

363
00:23:21.61 --> 00:23:25
Uh, that one comes out zero.

364
00:23:25 --> 00:23:32
Oh, so this is a lucky case and
that one comes out 18.

365
00:23:32 --> 00:23:38
So this is an accident that A A
transpose happens to come out

366
00:23:38 --> 00:23:43
diagonal, so we know easily its
eigenvectors and eigenvalues.

367
00:23:43 --> 00:23:48
So its eigenvectors -- what's
the first eigenvector for this A

368
00:23:48 --> 00:23:50
A transpose matrix?

369
00:23:50 --> 00:23:53
It's just one zero,
and when I do that

370
00:23:53 --> 00:23:57
multiplication,
I get 32 times one zero.

371
00:23:57 --> 00:24:03
And the other eigenvector is
just zero one and when I

372
00:24:03 --> 00:24:06
multiply by that I get 18.

373
00:24:06 --> 00:24:09
So this is A A transpose.

374
00:24:09 --> 00:24:13
Multiplying that gives me the
32 A A transpose.

375
00:24:13 --> 00:24:15
Multiplying this guy gives me

376
00:24:16 --> 00:24:17


377
00:24:17 --> 00:24:21.2
First of all,
I got 32 and 18 again.

378
00:24:21.2 --> 00:24:22
Am I surprised?

379
00:24:22 --> 00:24:28.3
You know, it's clearly not an
accident.

380
00:24:28.3 --> 00:24:32
The eigenvalues of A A
transpose were exactly the same

381
00:24:32 --> 00:24:36
as the eigenvalues of -- this
one was A transpose A.

382
00:24:36 --> 00:24:39
That's no surprise at all.

383
00:24:39 --> 00:24:44
The eigenvalues of A B are the
same as the eigenvalues of B A.

384
00:24:44.1 --> 00:24:48
That's a very nice fact,
that eigenvalues stay the same

385
00:24:48 --> 00:24:52
if I switch the order of
multiplication.

386
00:24:52 --> 00:24:53
So no surprise to see 32 and

387
00:24:54 --> 00:24:55


388
00:24:55 --> 00:25:00
What I learned -- first the
check that things were

389
00:25:00 --> 00:25:04
numerically correct,
but now I've learned these

390
00:25:04 --> 00:25:08
eigenvectors,
and actually they're about as

391
00:25:08 --> 00:25:09.77
nice as can be.

392
00:25:09.77 --> 00:25:15
They're the best orthogonal
matrix, just the identity.

393
00:25:15 --> 00:25:16
Okay.

394
00:25:16 --> 00:25:22
So my claim is that it ought to
all fit together,

395
00:25:22 --> 00:25:27
that these numbers should come
out right.

396
00:25:27 --> 00:25:34
The numbers should come out
right because the matrix

397
00:25:34 --> 00:25:40
multiplications use the
properties that we want.

398
00:25:40 --> 00:25:40
Okay.

399
00:25:40 --> 00:25:45.44
Shall we just check that?

400
00:25:45.44 --> 00:25:48
Here's the identity,
so not doing anything -- square

401
00:25:48 --> 00:25:52.85
root of 32 is multiplying that
row, so that square root of 32

402
00:25:52.85 --> 00:25:56
divided by square root of two
means square root of 16,

403
00:25:56 --> 00:25:57
four, correct?

404
00:25:57 --> 00:26:00
And square root of 18 is
divided by square root of two,

405
00:26:00 --> 00:26:04
so that leaves me square root
of 9, which is three,

406
00:26:04 --> 00:26:08
but --
well, Professor Strang,

407
00:26:08 --> 00:26:11
you see the problem?

408
00:26:11 --> 00:26:14.66
Why is that -- okay.

409
00:26:14.66 --> 00:26:23
Why I getting minus three three
here and here I'm getting three

410
00:26:23 --> 00:26:25
minus three?

411
00:26:25 --> 00:26:26
Phooey.

412
00:26:26 --> 00:26:28
I don't know why.

413
00:26:28 --> 00:26:35
It shouldn't have happened,
but it did.

414
00:26:35 --> 00:26:40
Now, okay, you could say,
well, just -- the eigenvector

415
00:26:40 --> 00:26:46
there could have -- I could have
had the minus sign here for that

416
00:26:46 --> 00:26:50
eigenvector, but I'm not happy
about that.

417
00:26:50 --> 00:26:51
Hmm.

418
00:26:51 --> 00:26:51
Okay.

419
00:26:51 --> 00:26:56
So I realize there's a little
catch here somewhere and I may

420
00:26:56 --> 00:27:00
not see it until Wednesday.

421
00:27:00 --> 00:27:06
Which then gives you a very
important reason to come back on

422
00:27:06 --> 00:27:10
Wednesday, to catch that sine
difference.

423
00:27:10 --> 00:27:13
So what did I do illegally?

424
00:27:13 --> 00:27:18
I think I put the eigenvectors
in that matrix V transpose --

425
00:27:18 --> 00:27:22
okay, I'm going to have to
think.

426
00:27:22 --> 00:27:28
Why did that come out with with
the opposite sines?

427
00:27:28 --> 00:27:33
So you see -- I mean,
if I had a minus there,

428
00:27:33 --> 00:27:39
I would be all right,
but I don't want that.

429
00:27:39 --> 00:27:46
I want positive entries down
the diagonal of sigma squared.

430
00:27:46 --> 00:27:47
Okay.

431
00:27:47 --> 00:27:53
It'll come to me,
but, I'm going to leave this

432
00:27:53 --> 00:27:56
example to finish.

433
00:27:56 --> 00:27:57
Okay.

434
00:27:57 --> 00:28:03
And the beauty of,
these sliding boards is I can

435
00:28:03 --> 00:28:05
make that go away.

436
00:28:05 --> 00:28:10
Can I,-- let me not do it,
though, yet.

437
00:28:10 --> 00:28:14
Let me take a second example.

438
00:28:14 --> 00:28:22
Let me take a second example
where the matrix is singular.

439
00:28:22 --> 00:28:24
So rank one.

440
00:28:24 --> 00:28:33
Okay, so let me take as an
example two, where my matrix A

441
00:28:33 --> 00:28:41
is going to be rectangular again
-- let me just make it four

442
00:28:41 --> 00:28:43
three eight six.

443
00:28:43 --> 00:28:44
Okay.

444
00:28:44 --> 00:28:48
That's a rank one matrix.

445
00:28:48 --> 00:28:56
So that has a null space and
only a one dimensional row space

446
00:28:56 --> 00:29:00
and column space.

447
00:29:00 --> 00:29:04
So actually,
my picture becomes easy for

448
00:29:04 --> 00:29:10
this matrix, because what's my
row space for this one?

449
00:29:10 --> 00:29:12
So this is two by two.

450
00:29:12 --> 00:29:16
So my pictures are both two
dimensional.

451
00:29:16 --> 00:29:22
My row space is all multiples
of that vector four three.

452
00:29:22 --> 00:29:27
So the whole -- the row space
is just a line,

453
00:29:27 --> 00:29:28
right?

454
00:29:28 --> 00:29:30
That's the row space.

455
00:29:30 --> 00:29:35
And the null space,
of course, is the perpendicular

456
00:29:35 --> 00:29:36
line.

457
00:29:36 --> 00:29:41
So the row space for this
matrix is multiples of four

458
00:29:41 --> 00:29:42
three.

459
00:29:42 --> 00:29:43
Typical row.

460
00:29:43 --> 00:29:43.97
Okay.

461
00:29:43.97 --> 00:29:46
What's the column space?

462
00:29:46 --> 00:29:50.98
The columns are all multiples
of four eight,

463
00:29:50.98 --> 00:29:54
three six, one two.

464
00:29:54 --> 00:29:58
The column space,
then, goes in,

465
00:29:58 --> 00:30:00
like, this direction.

466
00:30:00 --> 00:30:05
So the column space -- when I
look at those columns,

467
00:30:05 --> 00:30:11.06
the column space -- so it's
only one dimensional,

468
00:30:11.06 --> 00:30:13.67
because the rank is one.

469
00:30:13.67 --> 00:30:17
It's multiples of four eight.

470
00:30:17 --> 00:30:18
Okay.

471
00:30:18 --> 00:30:23
And what's the null space of A
transpose?

472
00:30:23 --> 00:30:26
It's the perpendicular guy.

473
00:30:26 --> 00:30:33
So this was the null space of A
and this is the null space of A

474
00:30:33 --> 00:30:34
transpose.

475
00:30:34 --> 00:30:35
Okay.

476
00:30:35 --> 00:30:42
What I want to say here is that
choosing these orthogonal bases

477
00:30:42 --> 00:30:47
for the row space and the column
space is, like,

478
00:30:47 --> 00:30:50
no problem.

479
00:30:50 --> 00:30:53
They're only one dimensional.

480
00:30:53 --> 00:30:55
So what should V be?

481
00:30:55 --> 00:30:58.84
V should be -- v1,
but -- yes, v1,

482
00:30:58.84 --> 00:31:03
rather -- v1 is supposed to be
a unit vector.

483
00:31:03 --> 00:31:09
There's only one v1 to choose
here, only one dimension in the

484
00:31:09 --> 00:31:11
row space.

485
00:31:11 --> 00:31:15
I just want to make it a unit
vector.

486
00:31:15 --> 00:31:22
So v1 will be -- it'll be this
vector, but made into a unit

487
00:31:22 --> 00:31:27
vector, so four -- point eight
point six.

488
00:31:27 --> 00:31:30
Four fifths,
three fifths.

489
00:31:30 --> 00:31:35.46
And what will be u1?
u1 will be the unit vector

490
00:31:35.46 --> 00:31:37
there.

491
00:31:37 --> 00:31:41.95
So I want to turn four eight or
one two into a unit vector,

492
00:31:41.95 --> 00:31:45
so u1 will be -- let's see,
if it's one two,

493
00:31:45 --> 00:31:48
then what multiple of one two
do I want?

494
00:31:48 --> 00:31:52
That has length square root of
five, so I have to divide by

495
00:31:52 --> 00:31:54
square root of five.

496
00:31:54 --> 00:31:58
Let me complete the singular
value decomposition for this

497
00:31:58 --> 00:32:00
matrix.

498
00:32:00 --> 00:32:05
So this matrix,
four three eight six,

499
00:32:05 --> 00:32:15
is -- so I know what u1 --
here's A and I want to get U the

500
00:32:15 --> 00:32:19
basis in the column space.

501
00:32:19 --> 00:32:28
And it has to start with this
guy, one over square root of

502
00:32:28 --> 00:32:35
five two over square root of
five.

503
00:32:35 --> 00:32:38
Then I want the sigma.

504
00:32:38 --> 00:32:38
Okay.

505
00:32:38 --> 00:32:43
What are we expecting now for
sigma?

506
00:32:43 --> 00:32:47
This is only a rank one matrix.

507
00:32:47 --> 00:32:54
We're only expecting a sigma
one, which I have to find,

508
00:32:54 --> 00:32:56
but zeroes here.

509
00:32:56 --> 00:32:57
Okay.

510
00:32:57 --> 00:32:59
So what's sigma one?

511
00:32:59 --> 00:33:07
It should be the -- where did
these sigmas come from?

512
00:33:07 --> 00:33:14.22
They came from A transpose A,
so I -- can I do that little

513
00:33:14.22 --> 00:33:16
calculation over here?

514
00:33:16 --> 00:33:23
A transpose A is four three --
four three eight six times four

515
00:33:23 --> 00:33:25
three eight six.

516
00:33:25 --> 00:33:30
This had better -- this is a
rank one matrix,

517
00:33:30 --> 00:33:36
this is going to be --
the whole thing will have rank

518
00:33:36 --> 00:33:41
one, that's 16 and 64 is 80,
12 and 48 is 60,

519
00:33:41 --> 00:33:44
12 and 48 is 60,
9 and 36 is 45.

520
00:33:44 --> 00:33:45
Okay.

521
00:33:45 --> 00:33:47
It's a rank one matrix.

522
00:33:47 --> 00:33:48
Of course.

523
00:33:48 --> 00:33:52
Every row is a multiple of four
three.

524
00:33:52 --> 00:33:58
And what's the eigen -- what
are the eigenvalues of that

525
00:33:58 --> 00:34:00
matrix?

526
00:34:00 --> 00:34:04
So this is the calculation --
this is like practicing,

527
00:34:04 --> 00:34:04
now.

528
00:34:04 --> 00:34:07
What are the eigenvalues of
this rank one matrix?

529
00:34:07 --> 00:34:11
Well, tell me one eigenvalue,
since the rank is only one,

530
00:34:11 --> 00:34:14
one eigenvalue is going to be
zero.

531
00:34:14 --> 00:34:18.18
And then you know that the
other eigenvalue is going to be

532
00:34:18.18 --> 00:34:20
a hundred and twenty five.

533
00:34:20 --> 00:34:26
So that's sigma squared,
right, in A transpose A.

534
00:34:26 --> 00:34:34
So this will be the square root
of a hundred and twenty five.

535
00:34:34 --> 00:34:40
And then finally,
the V transpose -- the Vs will

536
00:34:40 --> 00:34:44
be -- there's v1,
and what's v2?

537
00:34:44 --> 00:34:51.09
What's v2 in the -- how do I
make this into an orthonormal

538
00:34:51.09 --> 00:34:53
basis?

539
00:34:53 --> 00:34:56
Well, v2 is,
in the null space direction.

540
00:34:56 --> 00:35:02
It's perpendicular to that,
so point six and minus point

541
00:35:02 --> 00:35:02
eight.

542
00:35:02 --> 00:35:06
So those are the Vs that go in
here.

543
00:35:06 --> 00:35:09.99
Point eight,
point six and point six minus

544
00:35:09.99 --> 00:35:11
point eight.

545
00:35:11 --> 00:35:11
Okay.

546
00:35:11 --> 00:35:15
And I guess I better finish
this guy.

547
00:35:15 --> 00:35:18
So this guy,
all I want is to complete the

548
00:35:18 --> 00:35:24
orthonormal basis --
it'll be coming from there.

549
00:35:24 --> 00:35:30
It'll be a two over square root
of five and a minus one over

550
00:35:30 --> 00:35:32
square root of five.

551
00:35:32 --> 00:35:38
Let me take square root of five
out of that matrix to make it

552
00:35:38 --> 00:35:39
look better.

553
00:35:39 --> 00:35:45
So one over square root of five
times one two two minus one.

554
00:35:45 --> 00:35:46
Okay.

555
00:35:46 --> 00:35:52
So there I have -- including
the square root of five --

556
00:35:52 --> 00:35:58
that's an orthogonal matrix,
that's an orthogonal matrix,

557
00:35:58 --> 00:36:03
that's a diagonal matrix and
its rank is only one.

558
00:36:03 --> 00:36:07
And now if I do that
multiplication,

559
00:36:07 --> 00:36:11
I pray that it comes out right.

560
00:36:11 --> 00:36:17
The square root of five will
cancel into that square root of

561
00:36:17 --> 00:36:22
one twenty five and leave me
with the square root of 25,

562
00:36:22 --> 00:36:26
which is five,
and five will multiply these

563
00:36:26 --> 00:36:31
numbers and I'll get whole
numbers and out will come A.

564
00:36:31 --> 00:36:32
Okay.

565
00:36:32 --> 00:36:39
That's like a second example
showing how the null space guy

566
00:36:39 --> 00:36:46
-- so this -- this vector and
this one were multiplied by this

567
00:36:46 --> 00:36:47
zero.

568
00:36:47 --> 00:36:50
So they were easy to deal with.

569
00:36:50 --> 00:36:57
Tthe key ones are the ones in
the column space and the row

570
00:36:57 --> 00:36:59
space.

571
00:36:59 --> 00:37:03
Do you see how I'm getting
columns here,

572
00:37:03 --> 00:37:08
diagonal here,
rows here, coming together to

573
00:37:08 --> 00:37:09
produce A.

574
00:37:09 --> 00:37:13
Okay, that's the singular value
decomposition.

575
00:37:13 --> 00:37:19
So, let me think what I want to
add to complete this topic.

576
00:37:19 --> 00:37:22
So that's two examples.

577
00:37:22 --> 00:37:28.13
And now let's think what we're
really doing.

578
00:37:28.13 --> 00:37:41
We're choosing the right basis
for the four subspaces of linear

579
00:37:41 --> 00:37:43
algebra.

580
00:37:43 --> 00:37:48
Let me write this down.

581
00:37:48 --> 00:37:59
So v1 up to vr is an
orthonormal basis for the row

582
00:37:59 --> 00:38:06
space.
u1 up to ur is an orthonormal

583
00:38:06 --> 00:38:14
basis for the column space.

584
00:38:14 --> 00:38:20
And then I just finish those
out by v(r+1),

585
00:38:20 --> 00:38:28
the rest up to vn is an
orthonormal basis for the null

586
00:38:28 --> 00:38:29
space.

587
00:38:29 --> 00:38:35
And finally,
u(r+1) up to is an orthonormal

588
00:38:35 --> 00:38:41
basis for the null space of A
transpose.

589
00:38:41 --> 00:38:49
Do you see that we finally got
the bases right?

590
00:38:49 --> 00:38:55
They're right because they're
orthonormal, and also -- again,

591
00:38:55 --> 00:39:01
Graham Schmidt would have done
this in chapter four.

592
00:39:01 --> 00:39:07
Here we needed eigenvalues,
because these bases make the

593
00:39:07 --> 00:39:08
matrix diagonal.

594
00:39:08 --> 00:39:12
A times V I is a multiple of U
I.

595
00:39:12 --> 00:39:19
So I'll put "and" -- the matrix
has been made diagonal.

596
00:39:19 --> 00:39:24
When we choose these bases,
there's no coupling between Vs

597
00:39:24 --> 00:39:26.8
and no coupling between Us.

598
00:39:26.8 --> 00:39:31
Each A -- A times each V is in
the direction of the

599
00:39:31 --> 00:39:32
corresponding U.

600
00:39:32 --> 00:39:37.76
So it's exactly the right basis
for the four fundamental

601
00:39:37.76 --> 00:39:38
subspaces.

602
00:39:38 --> 00:39:42.47
And of course,
their dimensions are what we

603
00:39:42.47 --> 00:39:44.01
know.

604
00:39:44.01 --> 00:39:47.65
The dimension of the row space
is the rank r,

605
00:39:47.65 --> 00:39:51
and so is the dimension of the
column space.

606
00:39:51 --> 00:39:56
The dimension of the null space
is n-r, that's how many vectors

607
00:39:56 --> 00:40:00
we need, and m-r basis vectors
for the left null space,

608
00:40:00 --> 00:40:04
the null space of A transpose.

609
00:40:04 --> 00:40:04
Okay.

610
00:40:04 --> 00:40:06
I'm going to stop there.

611
00:40:06 --> 00:40:11.39
I could develop further from
the SVD, but we'll see it again

612
00:40:11.39 --> 00:40:14
in the very last lectures of the
course.

613
00:40:14 --> 00:40:16
So there's the SVD.

614
00:40:16 --> 00:40:19
Thanks.