WEBVTT
00:00:13.665 --> 00:00:15.540
GILBERT STRANG: Moving
now to the second half
00:00:15.540 --> 00:00:17.220
of linear algebra.
00:00:17.220 --> 00:00:20.160
It's about eigenvalues
and eigenvectors.
00:00:20.160 --> 00:00:22.320
The first half, I
just had a matrix.
00:00:22.320 --> 00:00:24.180
I solved equations.
00:00:24.180 --> 00:00:27.150
The second half,
you'll see the point
00:00:27.150 --> 00:00:29.820
of eigenvalues and
eigenvectors as a new way
00:00:29.820 --> 00:00:34.500
to look deeper into the matrix
to see what's important there.
00:00:34.500 --> 00:00:36.660
OK, so what are they?
00:00:36.660 --> 00:00:40.510
This is a big
equation, S time x.
00:00:40.510 --> 00:00:42.590
So S is our matrix.
00:00:42.590 --> 00:00:45.060
And I've called it
S because I'm taking
00:00:45.060 --> 00:00:47.400
it to be a symmetric matrix.
00:00:47.400 --> 00:00:49.920
What's on one side
of the diagonal
00:00:49.920 --> 00:00:52.530
is also on the other
side of the diagonal.
00:00:52.530 --> 00:00:54.800
So those have the
beautiful properties.
00:00:54.800 --> 00:00:57.270
Those are the kings
of linear algebra.
00:00:57.270 --> 00:01:02.280
Now, about eigenvectors
x and eigenvalues lambda.
00:01:02.280 --> 00:01:06.810
So what does that equation,
Sx equal lambda x, tell me?
00:01:06.810 --> 00:01:10.550
That says that I have
a special vector x.
00:01:10.550 --> 00:01:14.400
When I multiply it
by S, my matrix,
00:01:14.400 --> 00:01:17.730
I stay in the same
direction as the original x.
00:01:17.730 --> 00:01:19.810
It might get multiplied by 2.
00:01:19.810 --> 00:01:21.630
Lambda could be 2.
00:01:21.630 --> 00:01:23.670
It might get multiplied by 0.
00:01:23.670 --> 00:01:25.590
Lambda there could even be 0.
00:01:25.590 --> 00:01:28.740
It might get multiplied
by minus 2, whatever.
00:01:28.740 --> 00:01:30.680
But it's along the same line.
00:01:30.680 --> 00:01:35.760
So that's like taking a matrix
and discovering inside it
00:01:35.760 --> 00:01:39.740
something that stays on a line.
00:01:39.740 --> 00:01:43.640
That means that it's really a
sort of one dimensional problem
00:01:43.640 --> 00:01:47.060
if we're looking along
that eigenvector.
00:01:47.060 --> 00:01:51.380
And that makes computations
infinitely easier.
00:01:51.380 --> 00:01:55.070
The hard part of a matrix
is all the connections
00:01:55.070 --> 00:01:57.830
between different
rows and columns.
00:01:57.830 --> 00:02:00.050
So eigenvectors
are the guys that
00:02:00.050 --> 00:02:02.222
stay in that same direction.
00:02:04.830 --> 00:02:08.330
And y is another eigenvector.
00:02:08.330 --> 00:02:10.169
It has its own eigenvalue.
00:02:10.169 --> 00:02:14.200
It got multiplied by alpha
where Sx multiplied the x
00:02:14.200 --> 00:02:16.540
by some other number lambda.
00:02:16.540 --> 00:02:18.400
So there's our couple
of eigenvectors.
00:02:18.400 --> 00:02:23.600
And the beautiful fact is
that because S is symmetric,
00:02:23.600 --> 00:02:26.300
those two eigenvectors
are perpendicular.
00:02:26.300 --> 00:02:29.450
They are orthogonal,
as it says up there.
00:02:29.450 --> 00:02:32.450
So symmetric matrices
are really the best
00:02:32.450 --> 00:02:35.330
because their eigenvectors
are perpendicular.
00:02:35.330 --> 00:02:38.600
And we have a bunch of
one dimensional problems.
00:02:38.600 --> 00:02:41.780
And here, I've included a proof.
00:02:41.780 --> 00:02:47.460
You want a proof that the
eigenvectors are perpendicular?
00:02:47.460 --> 00:02:48.980
So what does perpendicular mean?
00:02:48.980 --> 00:02:55.980
It means that x transpose
times y, the dot product is 0.
00:02:55.980 --> 00:02:58.940
The angle is 90 degrees.
00:02:58.940 --> 00:03:01.640
The cosine is 1.
00:03:01.640 --> 00:03:03.050
OK.
00:03:03.050 --> 00:03:07.670
How to show the
cosine might be there.
00:03:07.670 --> 00:03:09.560
How to show that?
00:03:09.560 --> 00:03:10.500
Yeah, proof.
00:03:10.500 --> 00:03:13.730
This is just you can
tune out for two minutes
00:03:13.730 --> 00:03:15.410
if you hate proofs.
00:03:15.410 --> 00:03:17.990
OK, I start with what I know.
00:03:17.990 --> 00:03:20.200
What I know is in that box.
00:03:20.200 --> 00:03:21.640
Sx is lambda x.
00:03:21.640 --> 00:03:23.260
That's one eigenvector.
00:03:23.260 --> 00:03:25.570
That tells me the eigenvector y.
00:03:25.570 --> 00:03:27.940
This tells me the
eigenvalues are different.
00:03:27.940 --> 00:03:30.430
And that tells me the
matrix is symmetric.
00:03:30.430 --> 00:03:33.640
I'm just going to
juggle those four facts.
00:03:33.640 --> 00:03:39.040
And I'll end up with x
transpose y equals 0.
00:03:39.040 --> 00:03:41.350
That's orthogonality.
00:03:41.350 --> 00:03:42.400
OK.
00:03:42.400 --> 00:03:46.000
So I'll just do it
quickly, too quickly.
00:03:46.000 --> 00:03:49.900
So I take this first
thing, and I transpose
00:03:49.900 --> 00:03:52.740
it, turn it into row vectors.
00:03:52.740 --> 00:03:57.450
And then when I transpose
it, that transpose
00:03:57.450 --> 00:03:59.220
means I flip rows and columns.
00:03:59.220 --> 00:04:02.670
But for as symmetric
matrix, no different.
00:04:02.670 --> 00:04:05.390
So S transpose is the same as S.
00:04:05.390 --> 00:04:08.840
And then I look at this
one, and I multiply that
00:04:08.840 --> 00:04:13.130
by x transpose, both
sides by x transpose.
00:04:13.130 --> 00:04:16.700
And what I end up
with is recognizing
00:04:16.700 --> 00:04:19.250
that lambda times
that dot product
00:04:19.250 --> 00:04:22.140
equals alpha times
that dot product.
00:04:22.140 --> 00:04:24.690
But lambda is
different from alpha.
00:04:24.690 --> 00:04:26.760
So the only way lambda
times that number
00:04:26.760 --> 00:04:28.470
could equal alpha
times that number
00:04:28.470 --> 00:04:31.050
is that number has to be 0.
00:04:31.050 --> 00:04:32.460
And that's the answer.
00:04:32.460 --> 00:04:34.740
OK, so that's the
proof that used
00:04:34.740 --> 00:04:37.980
exactly every fact we knew.
00:04:37.980 --> 00:04:39.930
End of proof.
00:04:39.930 --> 00:04:42.720
Main point to
remember, eigenvectors
00:04:42.720 --> 00:04:47.130
are perpendicular when
the matrix is symmetric.
00:04:47.130 --> 00:04:49.710
OK.
00:04:49.710 --> 00:04:54.180
In that case, now, you always
want to express these facts
00:04:54.180 --> 00:04:59.030
as from multiplying matrices.
00:04:59.030 --> 00:05:02.180
That says everything
in a few symbols
00:05:02.180 --> 00:05:06.000
where I had to use all those
words on the previous slide.
00:05:06.000 --> 00:05:10.950
So that's the result
that I'm shooting for,
00:05:10.950 --> 00:05:14.450
that a symmetric matrix--
00:05:14.450 --> 00:05:19.150
just focus on that box.
00:05:19.150 --> 00:05:24.740
A symmetric matrix can be
broken up into its eigenvectors.
00:05:24.740 --> 00:05:27.740
Those are in Q. Its eigenvalues.
00:05:27.740 --> 00:05:28.910
Those are the lambdas.
00:05:28.910 --> 00:05:32.120
Those are the numbers
lambda 1 to lambda n
00:05:32.120 --> 00:05:34.070
on the diagonal of lambda.
00:05:34.070 --> 00:05:36.890
And then the transpose, so
the eigenvectors are now
00:05:36.890 --> 00:05:39.440
rows in Q transpose.
00:05:39.440 --> 00:05:42.160
That's just perfect.
00:05:42.160 --> 00:05:43.480
Perfect.
00:05:43.480 --> 00:05:47.020
Every symmetric matrix
is an orthogonal matrix
00:05:47.020 --> 00:05:51.490
times a diagonal matrix
times the transpose
00:05:51.490 --> 00:05:53.680
of the orthogonal matrix.
00:05:53.680 --> 00:05:55.810
Yeah, that's called
the spectral theorem.
00:05:55.810 --> 00:06:00.670
And you could say it's up there
with the most important facts
00:06:00.670 --> 00:06:04.410
in linear algebra and
in wider mathematics.
00:06:04.410 --> 00:06:12.580
Yeah, so that's the fact that
controls what we do here.
00:06:12.580 --> 00:06:18.160
Oh, now I have to say what's the
situation if the matrix is not
00:06:18.160 --> 00:06:19.860
symmetric.
00:06:19.860 --> 00:06:24.170
Now I am not going to get
perpendicular eigenvectors.
00:06:24.170 --> 00:06:27.470
That was a symmetric
thing mostly.
00:06:27.470 --> 00:06:30.260
But I'll get eigenvectors.
00:06:30.260 --> 00:06:35.140
So I'll get Ax equal lambda x.
00:06:35.140 --> 00:06:37.060
The first one won't
be perpendicular
00:06:37.060 --> 00:06:37.870
to the second one.
00:06:37.870 --> 00:06:40.520
The matrix A, it
has to be square,
00:06:40.520 --> 00:06:41.710
or this doesn't make sense.
00:06:41.710 --> 00:06:45.070
So eigenvalues and
eigenvectors are the way
00:06:45.070 --> 00:06:50.590
to break up a square matrix
and find this diagonal matrix
00:06:50.590 --> 00:06:54.640
lambda with the eigenvalues,
lambda 1, lambda 2, to lambda
00:06:54.640 --> 00:06:55.270
n.
00:06:55.270 --> 00:06:58.900
That's the purpose.
00:06:58.900 --> 00:07:01.510
And eigenvectors are
perpendicular when
00:07:01.510 --> 00:07:03.250
it's a symmetric matrix.
00:07:03.250 --> 00:07:08.710
Otherwise, I just have x and its
inverse matrix but no symmetry.
00:07:08.710 --> 00:07:09.490
OK.
00:07:09.490 --> 00:07:14.470
So that's the quick expression,
another factorization
00:07:14.470 --> 00:07:17.530
of eigenvalues in lambda.
00:07:17.530 --> 00:07:19.670
Diagonal, just numbers.
00:07:19.670 --> 00:07:23.370
And eigenvectors in
the columns of x.
00:07:23.370 --> 00:07:27.610
And now I'm not
going to-- oh, I was
00:07:27.610 --> 00:07:29.530
going to say I'm not
going to solve all
00:07:29.530 --> 00:07:31.030
the problems of applied math.
00:07:31.030 --> 00:07:33.850
But that's what these are for.
00:07:33.850 --> 00:07:38.400
Let's just see what's special
here about these eigenvectors.
00:07:38.400 --> 00:07:46.800
Suppose I multiply again by A.
I Start with Ax equal lambda x.
00:07:46.800 --> 00:07:49.650
Now I'm going to
multiply both sides by A.
00:07:49.650 --> 00:07:53.550
That'll tell me something
about eigenvalues of A squared.
00:07:53.550 --> 00:07:55.970
Because when I multiply by A--
00:07:55.970 --> 00:07:58.230
so let me start
with A squared now
00:07:58.230 --> 00:08:02.810
times x, which means A times Ax.
00:08:02.810 --> 00:08:04.400
A times Ax.
00:08:04.400 --> 00:08:06.440
But Ax is lambda x.
00:08:06.440 --> 00:08:09.320
So I have A times lambda x.
00:08:09.320 --> 00:08:12.180
And I pull out
that number lambda.
00:08:12.180 --> 00:08:15.040
And I still have a 1Ax.
00:08:15.040 --> 00:08:17.450
And that's also still lambda x.
00:08:17.450 --> 00:08:20.180
You see I'm just talking
around in a little circle
00:08:20.180 --> 00:08:24.080
here, just using Ax equal
lambda x a couple of times.
00:08:24.080 --> 00:08:25.860
And the result is--
00:08:25.860 --> 00:08:28.400
do you see what that
means, that result?
00:08:28.400 --> 00:08:33.080
That means that the eigenvalue
for A squared, same eigenvector
00:08:33.080 --> 00:08:33.830
x.
00:08:33.830 --> 00:08:37.230
The eigenvalue is
lambda squared.
00:08:37.230 --> 00:08:39.500
And if I add A
cubed, the eigenvalue
00:08:39.500 --> 00:08:41.299
would come out lambda cubed.
00:08:41.299 --> 00:08:44.990
And if I have a to
the-- yeah, yeah.
00:08:44.990 --> 00:08:49.640
So if I had A to the n
times, n multiplies-- so when
00:08:49.640 --> 00:08:53.090
would you have A
to a high power?
00:08:53.090 --> 00:08:56.480
That's a interesting matrix.
00:08:56.480 --> 00:08:58.910
Take a matrix and
square it, cube it,
00:08:58.910 --> 00:09:02.450
take high powers of it.
00:09:02.450 --> 00:09:04.520
The eigenvectors don't change.
00:09:04.520 --> 00:09:05.690
That's the great thing.
00:09:05.690 --> 00:09:07.850
That's the whole
point of eigenvectors.
00:09:07.850 --> 00:09:09.120
They don't change.
00:09:09.120 --> 00:09:12.540
And the eigenvalues just
get taken to the high power.
00:09:12.540 --> 00:09:16.010
So for example, we could
ask the question, when,
00:09:16.010 --> 00:09:19.160
if I multiply a matrix by itself
over and over and over again,
00:09:19.160 --> 00:09:20.840
when do I approach 0?
00:09:20.840 --> 00:09:24.920
Well, if these
numbers are below 1.
00:09:24.920 --> 00:09:27.260
So eigenvectors, eigenvalues
gives you something
00:09:27.260 --> 00:09:33.330
that you just could not see by
those column operations or L
00:09:33.330 --> 00:09:36.980
times U. This is looking deeper.
00:09:36.980 --> 00:09:38.120
OK.
00:09:38.120 --> 00:09:45.040
And OK, and then you'll see
we have almost already seen
00:09:45.040 --> 00:09:49.600
with least squares, this
combination A transpose A. So
00:09:49.600 --> 00:09:53.750
remember A is a
rectangular matrix, m by n.
00:09:53.750 --> 00:09:56.220
I multiply it by its transpose.
00:09:56.220 --> 00:10:02.750
When I transpose
it, I have n by m.
00:10:02.750 --> 00:10:05.450
And when I multiply them
together, I get n by n.
00:10:05.450 --> 00:10:10.880
So A transpose A is, for
theory, is a great matrix,
00:10:10.880 --> 00:10:14.030
A transpose times
A. It's symmetric.
00:10:14.030 --> 00:10:16.430
Yeah, let's just see
what we have about A.
00:10:16.430 --> 00:10:19.460
It's square for sure.
00:10:19.460 --> 00:10:20.000
Oh, yeah.
00:10:20.000 --> 00:10:22.125
This tells me that
it's symmetric.
00:10:22.125 --> 00:10:23.000
And you remember why.
00:10:23.000 --> 00:10:26.020
I'm always looking
for symmetric matrices
00:10:26.020 --> 00:10:29.530
because they have those
orthogonal eigenvectors.
00:10:29.530 --> 00:10:32.930
They're the beautiful
ones for eigenvectors.
00:10:32.930 --> 00:10:36.070
And A transpose A,
automatically symmetric.
00:10:36.070 --> 00:10:40.660
You just you're
multiplying something
00:10:40.660 --> 00:10:45.940
by its adjoint, its
transpose, and the result
00:10:45.940 --> 00:10:51.010
is that this matrix
is symmetric.
00:10:51.010 --> 00:10:55.920
And maybe there's even more
about A transpose A. Yes.
00:10:55.920 --> 00:10:56.580
What is that?
00:11:00.610 --> 00:11:03.260
Here is a final--
00:11:03.260 --> 00:11:07.020
I always say certain
matrices are important,
00:11:07.020 --> 00:11:10.330
but these are the winners.
00:11:10.330 --> 00:11:12.510
They are symmetric matrices.
00:11:12.510 --> 00:11:16.260
If I want beautiful
matrices, make them symmetric
00:11:16.260 --> 00:11:18.950
and make the
eigenvalues positive.
00:11:21.540 --> 00:11:26.160
Or non-negative allows 0.
00:11:26.160 --> 00:11:28.890
So I can either say
positive definite
00:11:28.890 --> 00:11:31.320
when the eigenvalues
are positive,
00:11:31.320 --> 00:11:34.880
or I can say non-negative,
which allows 0.
00:11:34.880 --> 00:11:38.700
And so I have greater
than or equal to 0.
00:11:38.700 --> 00:11:41.820
I just want to say
that bringing all
00:11:41.820 --> 00:11:44.520
the pieces of linear
algebra come together
00:11:44.520 --> 00:11:46.320
in these matrices.
00:11:46.320 --> 00:11:49.770
And we're seeing the
eigenvalue part of it.
00:11:49.770 --> 00:11:53.020
And here, I've mentioned
something called the energy.
00:11:53.020 --> 00:11:55.380
So that's a physical
quantity that
00:11:55.380 --> 00:11:58.230
also is greater or equal to 0.
00:11:58.230 --> 00:12:02.970
So that's A transpose
A is the matrix
00:12:02.970 --> 00:12:10.290
that I'm going to use in
the final part of this video
00:12:10.290 --> 00:12:15.060
to achieve the
greatest factorization.
00:12:15.060 --> 00:12:18.780
Q lambda, Q transpose
was fantastic.
00:12:18.780 --> 00:12:22.830
But for a non-square
matrix, it's not.
00:12:22.830 --> 00:12:25.260
For a non-square
matrix, they don't even
00:12:25.260 --> 00:12:27.840
have eigenvalues
and eigenvectors.
00:12:27.840 --> 00:12:31.650
But data comes in
non-square matrices.
00:12:31.650 --> 00:12:34.230
Data is about like we
have a bunch of diseases
00:12:34.230 --> 00:12:37.940
and a bunch of patients
or a bunch of medicines.
00:12:37.940 --> 00:12:39.470
And the number of
medicines is not
00:12:39.470 --> 00:12:42.090
equal the number of
patients or diseases.
00:12:42.090 --> 00:12:43.740
Those are different numbers.
00:12:43.740 --> 00:12:48.560
So the matrices that we see
in data are rectangular.
00:12:48.560 --> 00:12:52.800
And eigenvalues don't
make sense for those.
00:12:52.800 --> 00:12:56.650
And singular values take
the place of eigenvalues.
00:12:56.650 --> 00:12:59.280
So singular values,
and my hope is
00:12:59.280 --> 00:13:04.550
that linear algebra
courses, 18.06 for sure,
00:13:04.550 --> 00:13:08.690
will always reach,
after you explain
00:13:08.690 --> 00:13:12.170
eigenvalues that everybody
agrees is important,
00:13:12.170 --> 00:13:14.600
get singular values
into the course
00:13:14.600 --> 00:13:18.740
because they really have
come on as the big things
00:13:18.740 --> 00:13:20.600
to do in data.
00:13:20.600 --> 00:13:27.370
So that would be the last
part of this summary video
00:13:27.370 --> 00:13:30.780
for 2020 vision
of linear algebra
00:13:30.780 --> 00:13:33.520
is to get singular
values in there.
00:13:33.520 --> 00:13:36.120
OK, that's coming next.