1
00:00:13,665 --> 00:00:15,540
GILBERT STRANG: Moving
now to the second half
2
00:00:15,540 --> 00:00:17,220
of linear algebra.
3
00:00:17,220 --> 00:00:20,160
It's about eigenvalues
and eigenvectors.
4
00:00:20,160 --> 00:00:22,320
The first half, I
just had a matrix.
5
00:00:22,320 --> 00:00:24,180
I solved equations.
6
00:00:24,180 --> 00:00:27,150
The second half,
you'll see the point
7
00:00:27,150 --> 00:00:29,820
of eigenvalues and
eigenvectors as a new way
8
00:00:29,820 --> 00:00:34,500
to look deeper into the matrix
to see what's important there.
9
00:00:34,500 --> 00:00:36,660
OK, so what are they?
10
00:00:36,660 --> 00:00:40,510
This is a big
equation, S time x.
11
00:00:40,510 --> 00:00:42,590
So S is our matrix.
12
00:00:42,590 --> 00:00:45,060
And I've called it
S because I'm taking
13
00:00:45,060 --> 00:00:47,400
it to be a symmetric matrix.
14
00:00:47,400 --> 00:00:49,920
What's on one side
of the diagonal
15
00:00:49,920 --> 00:00:52,530
is also on the other
side of the diagonal.
16
00:00:52,530 --> 00:00:54,800
So those have the
beautiful properties.
17
00:00:54,800 --> 00:00:57,270
Those are the kings
of linear algebra.
18
00:00:57,270 --> 00:01:02,280
Now, about eigenvectors
x and eigenvalues lambda.
19
00:01:02,280 --> 00:01:06,810
So what does that equation,
Sx equal lambda x, tell me?
20
00:01:06,810 --> 00:01:10,550
That says that I have
a special vector x.
21
00:01:10,550 --> 00:01:14,400
When I multiply it
by S, my matrix,
22
00:01:14,400 --> 00:01:17,730
I stay in the same
direction as the original x.
23
00:01:17,730 --> 00:01:19,810
It might get multiplied by 2.
24
00:01:19,810 --> 00:01:21,630
Lambda could be 2.
25
00:01:21,630 --> 00:01:23,670
It might get multiplied by 0.
26
00:01:23,670 --> 00:01:25,590
Lambda there could even be 0.
27
00:01:25,590 --> 00:01:28,740
It might get multiplied
by minus 2, whatever.
28
00:01:28,740 --> 00:01:30,680
But it's along the same line.
29
00:01:30,680 --> 00:01:35,760
So that's like taking a matrix
and discovering inside it
30
00:01:35,760 --> 00:01:39,740
something that stays on a line.
31
00:01:39,740 --> 00:01:43,640
That means that it's really a
sort of one dimensional problem
32
00:01:43,640 --> 00:01:47,060
if we're looking along
that eigenvector.
33
00:01:47,060 --> 00:01:51,380
And that makes computations
infinitely easier.
34
00:01:51,380 --> 00:01:55,070
The hard part of a matrix
is all the connections
35
00:01:55,070 --> 00:01:57,830
between different
rows and columns.
36
00:01:57,830 --> 00:02:00,050
So eigenvectors
are the guys that
37
00:02:00,050 --> 00:02:02,222
stay in that same direction.
38
00:02:04,830 --> 00:02:08,330
And y is another eigenvector.
39
00:02:08,330 --> 00:02:10,169
It has its own eigenvalue.
40
00:02:10,169 --> 00:02:14,200
It got multiplied by alpha
where Sx multiplied the x
41
00:02:14,200 --> 00:02:16,540
by some other number lambda.
42
00:02:16,540 --> 00:02:18,400
So there's our couple
of eigenvectors.
43
00:02:18,400 --> 00:02:23,600
And the beautiful fact is
that because S is symmetric,
44
00:02:23,600 --> 00:02:26,300
those two eigenvectors
are perpendicular.
45
00:02:26,300 --> 00:02:29,450
They are orthogonal,
as it says up there.
46
00:02:29,450 --> 00:02:32,450
So symmetric matrices
are really the best
47
00:02:32,450 --> 00:02:35,330
because their eigenvectors
are perpendicular.
48
00:02:35,330 --> 00:02:38,600
And we have a bunch of
one dimensional problems.
49
00:02:38,600 --> 00:02:41,780
And here, I've included a proof.
50
00:02:41,780 --> 00:02:47,460
You want a proof that the
eigenvectors are perpendicular?
51
00:02:47,460 --> 00:02:48,980
So what does perpendicular mean?
52
00:02:48,980 --> 00:02:55,980
It means that x transpose
times y, the dot product is 0.
53
00:02:55,980 --> 00:02:58,940
The angle is 90 degrees.
54
00:02:58,940 --> 00:03:01,640
The cosine is 1.
55
00:03:01,640 --> 00:03:03,050
OK.
56
00:03:03,050 --> 00:03:07,670
How to show the
cosine might be there.
57
00:03:07,670 --> 00:03:09,560
How to show that?
58
00:03:09,560 --> 00:03:10,500
Yeah, proof.
59
00:03:10,500 --> 00:03:13,730
This is just you can
tune out for two minutes
60
00:03:13,730 --> 00:03:15,410
if you hate proofs.
61
00:03:15,410 --> 00:03:17,990
OK, I start with what I know.
62
00:03:17,990 --> 00:03:20,200
What I know is in that box.
63
00:03:20,200 --> 00:03:21,640
Sx is lambda x.
64
00:03:21,640 --> 00:03:23,260
That's one eigenvector.
65
00:03:23,260 --> 00:03:25,570
That tells me the eigenvector y.
66
00:03:25,570 --> 00:03:27,940
This tells me the
eigenvalues are different.
67
00:03:27,940 --> 00:03:30,430
And that tells me the
matrix is symmetric.
68
00:03:30,430 --> 00:03:33,640
I'm just going to
juggle those four facts.
69
00:03:33,640 --> 00:03:39,040
And I'll end up with x
transpose y equals 0.
70
00:03:39,040 --> 00:03:41,350
That's orthogonality.
71
00:03:41,350 --> 00:03:42,400
OK.
72
00:03:42,400 --> 00:03:46,000
So I'll just do it
quickly, too quickly.
73
00:03:46,000 --> 00:03:49,900
So I take this first
thing, and I transpose
74
00:03:49,900 --> 00:03:52,740
it, turn it into row vectors.
75
00:03:52,740 --> 00:03:57,450
And then when I transpose
it, that transpose
76
00:03:57,450 --> 00:03:59,220
means I flip rows and columns.
77
00:03:59,220 --> 00:04:02,670
But for as symmetric
matrix, no different.
78
00:04:02,670 --> 00:04:05,390
So S transpose is the same as S.
79
00:04:05,390 --> 00:04:08,840
And then I look at this
one, and I multiply that
80
00:04:08,840 --> 00:04:13,130
by x transpose, both
sides by x transpose.
81
00:04:13,130 --> 00:04:16,700
And what I end up
with is recognizing
82
00:04:16,700 --> 00:04:19,250
that lambda times
that dot product
83
00:04:19,250 --> 00:04:22,140
equals alpha times
that dot product.
84
00:04:22,140 --> 00:04:24,690
But lambda is
different from alpha.
85
00:04:24,690 --> 00:04:26,760
So the only way lambda
times that number
86
00:04:26,760 --> 00:04:28,470
could equal alpha
times that number
87
00:04:28,470 --> 00:04:31,050
is that number has to be 0.
88
00:04:31,050 --> 00:04:32,460
And that's the answer.
89
00:04:32,460 --> 00:04:34,740
OK, so that's the
proof that used
90
00:04:34,740 --> 00:04:37,980
exactly every fact we knew.
91
00:04:37,980 --> 00:04:39,930
End of proof.
92
00:04:39,930 --> 00:04:42,720
Main point to
remember, eigenvectors
93
00:04:42,720 --> 00:04:47,130
are perpendicular when
the matrix is symmetric.
94
00:04:47,130 --> 00:04:49,710
OK.
95
00:04:49,710 --> 00:04:54,180
In that case, now, you always
want to express these facts
96
00:04:54,180 --> 00:04:59,030
as from multiplying matrices.
97
00:04:59,030 --> 00:05:02,180
That says everything
in a few symbols
98
00:05:02,180 --> 00:05:06,000
where I had to use all those
words on the previous slide.
99
00:05:06,000 --> 00:05:10,950
So that's the result
that I'm shooting for,
100
00:05:10,950 --> 00:05:14,450
that a symmetric matrix--
101
00:05:14,450 --> 00:05:19,150
just focus on that box.
102
00:05:19,150 --> 00:05:24,740
A symmetric matrix can be
broken up into its eigenvectors.
103
00:05:24,740 --> 00:05:27,740
Those are in Q. Its eigenvalues.
104
00:05:27,740 --> 00:05:28,910
Those are the lambdas.
105
00:05:28,910 --> 00:05:32,120
Those are the numbers
lambda 1 to lambda n
106
00:05:32,120 --> 00:05:34,070
on the diagonal of lambda.
107
00:05:34,070 --> 00:05:36,890
And then the transpose, so
the eigenvectors are now
108
00:05:36,890 --> 00:05:39,440
rows in Q transpose.
109
00:05:39,440 --> 00:05:42,160
That's just perfect.
110
00:05:42,160 --> 00:05:43,480
Perfect.
111
00:05:43,480 --> 00:05:47,020
Every symmetric matrix
is an orthogonal matrix
112
00:05:47,020 --> 00:05:51,490
times a diagonal matrix
times the transpose
113
00:05:51,490 --> 00:05:53,680
of the orthogonal matrix.
114
00:05:53,680 --> 00:05:55,810
Yeah, that's called
the spectral theorem.
115
00:05:55,810 --> 00:06:00,670
And you could say it's up there
with the most important facts
116
00:06:00,670 --> 00:06:04,410
in linear algebra and
in wider mathematics.
117
00:06:04,410 --> 00:06:12,580
Yeah, so that's the fact that
controls what we do here.
118
00:06:12,580 --> 00:06:18,160
Oh, now I have to say what's the
situation if the matrix is not
119
00:06:18,160 --> 00:06:19,860
symmetric.
120
00:06:19,860 --> 00:06:24,170
Now I am not going to get
perpendicular eigenvectors.
121
00:06:24,170 --> 00:06:27,470
That was a symmetric
thing mostly.
122
00:06:27,470 --> 00:06:30,260
But I'll get eigenvectors.
123
00:06:30,260 --> 00:06:35,140
So I'll get Ax equal lambda x.
124
00:06:35,140 --> 00:06:37,060
The first one won't
be perpendicular
125
00:06:37,060 --> 00:06:37,870
to the second one.
126
00:06:37,870 --> 00:06:40,520
The matrix A, it
has to be square,
127
00:06:40,520 --> 00:06:41,710
or this doesn't make sense.
128
00:06:41,710 --> 00:06:45,070
So eigenvalues and
eigenvectors are the way
129
00:06:45,070 --> 00:06:50,590
to break up a square matrix
and find this diagonal matrix
130
00:06:50,590 --> 00:06:54,640
lambda with the eigenvalues,
lambda 1, lambda 2, to lambda
131
00:06:54,640 --> 00:06:55,270
n.
132
00:06:55,270 --> 00:06:58,900
That's the purpose.
133
00:06:58,900 --> 00:07:01,510
And eigenvectors are
perpendicular when
134
00:07:01,510 --> 00:07:03,250
it's a symmetric matrix.
135
00:07:03,250 --> 00:07:08,710
Otherwise, I just have x and its
inverse matrix but no symmetry.
136
00:07:08,710 --> 00:07:09,490
OK.
137
00:07:09,490 --> 00:07:14,470
So that's the quick expression,
another factorization
138
00:07:14,470 --> 00:07:17,530
of eigenvalues in lambda.
139
00:07:17,530 --> 00:07:19,670
Diagonal, just numbers.
140
00:07:19,670 --> 00:07:23,370
And eigenvectors in
the columns of x.
141
00:07:23,370 --> 00:07:27,610
And now I'm not
going to-- oh, I was
142
00:07:27,610 --> 00:07:29,530
going to say I'm not
going to solve all
143
00:07:29,530 --> 00:07:31,030
the problems of applied math.
144
00:07:31,030 --> 00:07:33,850
But that's what these are for.
145
00:07:33,850 --> 00:07:38,400
Let's just see what's special
here about these eigenvectors.
146
00:07:38,400 --> 00:07:46,800
Suppose I multiply again by A.
I Start with Ax equal lambda x.
147
00:07:46,800 --> 00:07:49,650
Now I'm going to
multiply both sides by A.
148
00:07:49,650 --> 00:07:53,550
That'll tell me something
about eigenvalues of A squared.
149
00:07:53,550 --> 00:07:55,970
Because when I multiply by A--
150
00:07:55,970 --> 00:07:58,230
so let me start
with A squared now
151
00:07:58,230 --> 00:08:02,810
times x, which means A times Ax.
152
00:08:02,810 --> 00:08:04,400
A times Ax.
153
00:08:04,400 --> 00:08:06,440
But Ax is lambda x.
154
00:08:06,440 --> 00:08:09,320
So I have A times lambda x.
155
00:08:09,320 --> 00:08:12,180
And I pull out
that number lambda.
156
00:08:12,180 --> 00:08:15,040
And I still have a 1Ax.
157
00:08:15,040 --> 00:08:17,450
And that's also still lambda x.
158
00:08:17,450 --> 00:08:20,180
You see I'm just talking
around in a little circle
159
00:08:20,180 --> 00:08:24,080
here, just using Ax equal
lambda x a couple of times.
160
00:08:24,080 --> 00:08:25,860
And the result is--
161
00:08:25,860 --> 00:08:28,400
do you see what that
means, that result?
162
00:08:28,400 --> 00:08:33,080
That means that the eigenvalue
for A squared, same eigenvector
163
00:08:33,080 --> 00:08:33,830
x.
164
00:08:33,830 --> 00:08:37,230
The eigenvalue is
lambda squared.
165
00:08:37,230 --> 00:08:39,500
And if I add A
cubed, the eigenvalue
166
00:08:39,500 --> 00:08:41,299
would come out lambda cubed.
167
00:08:41,299 --> 00:08:44,990
And if I have a to
the-- yeah, yeah.
168
00:08:44,990 --> 00:08:49,640
So if I had A to the n
times, n multiplies-- so when
169
00:08:49,640 --> 00:08:53,090
would you have A
to a high power?
170
00:08:53,090 --> 00:08:56,480
That's a interesting matrix.
171
00:08:56,480 --> 00:08:58,910
Take a matrix and
square it, cube it,
172
00:08:58,910 --> 00:09:02,450
take high powers of it.
173
00:09:02,450 --> 00:09:04,520
The eigenvectors don't change.
174
00:09:04,520 --> 00:09:05,690
That's the great thing.
175
00:09:05,690 --> 00:09:07,850
That's the whole
point of eigenvectors.
176
00:09:07,850 --> 00:09:09,120
They don't change.
177
00:09:09,120 --> 00:09:12,540
And the eigenvalues just
get taken to the high power.
178
00:09:12,540 --> 00:09:16,010
So for example, we could
ask the question, when,
179
00:09:16,010 --> 00:09:19,160
if I multiply a matrix by itself
over and over and over again,
180
00:09:19,160 --> 00:09:20,840
when do I approach 0?
181
00:09:20,840 --> 00:09:24,920
Well, if these
numbers are below 1.
182
00:09:24,920 --> 00:09:27,260
So eigenvectors, eigenvalues
gives you something
183
00:09:27,260 --> 00:09:33,330
that you just could not see by
those column operations or L
184
00:09:33,330 --> 00:09:36,980
times U. This is looking deeper.
185
00:09:36,980 --> 00:09:38,120
OK.
186
00:09:38,120 --> 00:09:45,040
And OK, and then you'll see
we have almost already seen
187
00:09:45,040 --> 00:09:49,600
with least squares, this
combination A transpose A. So
188
00:09:49,600 --> 00:09:53,750
remember A is a
rectangular matrix, m by n.
189
00:09:53,750 --> 00:09:56,220
I multiply it by its transpose.
190
00:09:56,220 --> 00:10:02,750
When I transpose
it, I have n by m.
191
00:10:02,750 --> 00:10:05,450
And when I multiply them
together, I get n by n.
192
00:10:05,450 --> 00:10:10,880
So A transpose A is, for
theory, is a great matrix,
193
00:10:10,880 --> 00:10:14,030
A transpose times
A. It's symmetric.
194
00:10:14,030 --> 00:10:16,430
Yeah, let's just see
what we have about A.
195
00:10:16,430 --> 00:10:19,460
It's square for sure.
196
00:10:19,460 --> 00:10:20,000
Oh, yeah.
197
00:10:20,000 --> 00:10:22,125
This tells me that
it's symmetric.
198
00:10:22,125 --> 00:10:23,000
And you remember why.
199
00:10:23,000 --> 00:10:26,020
I'm always looking
for symmetric matrices
200
00:10:26,020 --> 00:10:29,530
because they have those
orthogonal eigenvectors.
201
00:10:29,530 --> 00:10:32,930
They're the beautiful
ones for eigenvectors.
202
00:10:32,930 --> 00:10:36,070
And A transpose A,
automatically symmetric.
203
00:10:36,070 --> 00:10:40,660
You just you're
multiplying something
204
00:10:40,660 --> 00:10:45,940
by its adjoint, its
transpose, and the result
205
00:10:45,940 --> 00:10:51,010
is that this matrix
is symmetric.
206
00:10:51,010 --> 00:10:55,920
And maybe there's even more
about A transpose A. Yes.
207
00:10:55,920 --> 00:10:56,580
What is that?
208
00:11:00,610 --> 00:11:03,260
Here is a final--
209
00:11:03,260 --> 00:11:07,020
I always say certain
matrices are important,
210
00:11:07,020 --> 00:11:10,330
but these are the winners.
211
00:11:10,330 --> 00:11:12,510
They are symmetric matrices.
212
00:11:12,510 --> 00:11:16,260
If I want beautiful
matrices, make them symmetric
213
00:11:16,260 --> 00:11:18,950
and make the
eigenvalues positive.
214
00:11:21,540 --> 00:11:26,160
Or non-negative allows 0.
215
00:11:26,160 --> 00:11:28,890
So I can either say
positive definite
216
00:11:28,890 --> 00:11:31,320
when the eigenvalues
are positive,
217
00:11:31,320 --> 00:11:34,880
or I can say non-negative,
which allows 0.
218
00:11:34,880 --> 00:11:38,700
And so I have greater
than or equal to 0.
219
00:11:38,700 --> 00:11:41,820
I just want to say
that bringing all
220
00:11:41,820 --> 00:11:44,520
the pieces of linear
algebra come together
221
00:11:44,520 --> 00:11:46,320
in these matrices.
222
00:11:46,320 --> 00:11:49,770
And we're seeing the
eigenvalue part of it.
223
00:11:49,770 --> 00:11:53,020
And here, I've mentioned
something called the energy.
224
00:11:53,020 --> 00:11:55,380
So that's a physical
quantity that
225
00:11:55,380 --> 00:11:58,230
also is greater or equal to 0.
226
00:11:58,230 --> 00:12:02,970
So that's A transpose
A is the matrix
227
00:12:02,970 --> 00:12:10,290
that I'm going to use in
the final part of this video
228
00:12:10,290 --> 00:12:15,060
to achieve the
greatest factorization.
229
00:12:15,060 --> 00:12:18,780
Q lambda, Q transpose
was fantastic.
230
00:12:18,780 --> 00:12:22,830
But for a non-square
matrix, it's not.
231
00:12:22,830 --> 00:12:25,260
For a non-square
matrix, they don't even
232
00:12:25,260 --> 00:12:27,840
have eigenvalues
and eigenvectors.
233
00:12:27,840 --> 00:12:31,650
But data comes in
non-square matrices.
234
00:12:31,650 --> 00:12:34,230
Data is about like we
have a bunch of diseases
235
00:12:34,230 --> 00:12:37,940
and a bunch of patients
or a bunch of medicines.
236
00:12:37,940 --> 00:12:39,470
And the number of
medicines is not
237
00:12:39,470 --> 00:12:42,090
equal the number of
patients or diseases.
238
00:12:42,090 --> 00:12:43,740
Those are different numbers.
239
00:12:43,740 --> 00:12:48,560
So the matrices that we see
in data are rectangular.
240
00:12:48,560 --> 00:12:52,800
And eigenvalues don't
make sense for those.
241
00:12:52,800 --> 00:12:56,650
And singular values take
the place of eigenvalues.
242
00:12:56,650 --> 00:12:59,280
So singular values,
and my hope is
243
00:12:59,280 --> 00:13:04,550
that linear algebra
courses, 18.06 for sure,
244
00:13:04,550 --> 00:13:08,690
will always reach,
after you explain
245
00:13:08,690 --> 00:13:12,170
eigenvalues that everybody
agrees is important,
246
00:13:12,170 --> 00:13:14,600
get singular values
into the course
247
00:13:14,600 --> 00:13:18,740
because they really have
come on as the big things
248
00:13:18,740 --> 00:13:20,600
to do in data.
249
00:13:20,600 --> 00:13:27,370
So that would be the last
part of this summary video
250
00:13:27,370 --> 00:13:30,780
for 2020 vision
of linear algebra
251
00:13:30,780 --> 00:13:33,520
is to get singular
values in there.
252
00:13:33,520 --> 00:13:36,120
OK, that's coming next.