WEBVTT
00:00:08.491 --> 00:00:09.490
PROFESSOR: Hi, everyone.
00:00:09.490 --> 00:00:10.780
Welcome back.
00:00:10.780 --> 00:00:14.050
So today, I'd like to talk about
positive definite matrices.
00:00:14.050 --> 00:00:17.650
And specifically, we're going
to analyze several properties
00:00:17.650 --> 00:00:19.440
of positive definite matrices.
00:00:19.440 --> 00:00:21.860
And specifically,
we're going to look
00:00:21.860 --> 00:00:24.710
at why each one of these
following statements is true.
00:00:24.710 --> 00:00:29.970
So first off, why every positive
definite matrix is invertible.
00:00:29.970 --> 00:00:32.270
Why the only positive
definite projection matrix
00:00:32.270 --> 00:00:34.860
is the identity matrix.
00:00:34.860 --> 00:00:38.320
If D is a diagonal matrix
with positive entries,
00:00:38.320 --> 00:00:41.010
show that it must also
be positive definite.
00:00:41.010 --> 00:00:44.260
And then lastly, if S is
a symmetric matrix where
00:00:44.260 --> 00:00:46.980
the determinant S
is bigger than 0,
00:00:46.980 --> 00:00:48.970
show why this might
not necessarily
00:00:48.970 --> 00:00:50.870
imply that it's
positive definite.
00:00:50.870 --> 00:00:53.239
So I'll let you think
about these for a moment.
00:00:53.239 --> 00:00:54.530
And I'll come back in a second.
00:01:08.730 --> 00:01:09.410
Hi, everyone.
00:01:09.410 --> 00:01:10.640
Welcome back.
00:01:10.640 --> 00:01:11.170
OK.
00:01:11.170 --> 00:01:15.820
So let's take a look
at part A. So part A
00:01:15.820 --> 00:01:21.140
is asking why every positive
definite matrix is invertible.
00:01:21.140 --> 00:01:24.680
Well, let's just recall
that if A is a matrix
00:01:24.680 --> 00:01:33.010
and if A is invertible,
then this necessarily
00:01:33.010 --> 00:01:37.795
implies that the determinant
of A is non-zero.
00:01:40.890 --> 00:01:42.500
And I'm going to
just write out det
00:01:42.500 --> 00:01:53.710
A as the product of the
eigenvalues of A. So lambda_1
00:01:53.710 --> 00:02:06.180
to lambda_n are the
eigenvalues of A.
00:02:06.180 --> 00:02:07.070
OK.
00:02:07.070 --> 00:02:18.480
In addition, if A is
positive definite,
00:02:18.480 --> 00:02:21.000
what does this say about
the eigenvalues of A?
00:02:21.000 --> 00:02:24.900
Well, it says that each
eigenvalue of A, lambda_1,
00:02:24.900 --> 00:02:30.700
lambda_2, dot, dot, dot, to
lambda_n, each one of them
00:02:30.700 --> 00:02:33.440
must be bigger than 0.
00:02:33.440 --> 00:02:36.700
So this statement that
each eigenvalue of A
00:02:36.700 --> 00:02:38.790
is bigger than 0 is
completely equivalent
00:02:38.790 --> 00:02:46.860
to A being positive definite
for symmetric matrices A.
00:02:46.860 --> 00:02:50.260
So if I have a whole bunch of
eigenvalues and each of them
00:02:50.260 --> 00:02:52.880
are bigger than 0, what
does this say about det A?
00:02:56.530 --> 00:02:58.780
Well, I can take the product
of all these eigenvalues.
00:03:03.460 --> 00:03:06.740
And of course, the product of a
whole bunch of positive numbers
00:03:06.740 --> 00:03:07.660
must also be positive.
00:03:10.540 --> 00:03:14.640
So if the quantity is positive,
then it certainly cannot equal
00:03:14.640 --> 00:03:15.800
0.
00:03:15.800 --> 00:03:18.770
So this proves that det
A is not equal to 0.
00:03:18.770 --> 00:03:22.590
Hence, A must be invertible.
00:03:22.590 --> 00:03:23.090
OK.
00:03:23.090 --> 00:03:29.850
So for part B,
we're asked to show
00:03:29.850 --> 00:03:31.810
that the only positive
definite projection
00:03:31.810 --> 00:03:35.070
matrix is the identity matrix.
00:03:35.070 --> 00:03:37.220
So again, how do we
tackle this problem?
00:03:37.220 --> 00:03:39.930
We're going to look
at the eigenvalues.
00:03:39.930 --> 00:03:49.690
So remember, if P
is a projection,
00:03:49.690 --> 00:03:52.080
what does it say about
the eigenvalues of P?
00:03:52.080 --> 00:04:03.520
Well, it says that
the eigenvalues of P
00:04:03.520 --> 00:04:05.370
are either 0 or 1.
00:04:09.390 --> 00:04:11.520
So this is point one.
00:04:11.520 --> 00:04:25.150
Point two, if P is
a positive definite,
00:04:25.150 --> 00:04:27.706
what does that say about
the eigenvalues of P?
00:04:27.706 --> 00:04:43.670
Well, as I've noted before,
it means that the eigenvalues
00:04:43.670 --> 00:04:44.580
are bigger than 0.
00:04:47.210 --> 00:04:51.700
So if P is a projection
and it's positive definite,
00:04:51.700 --> 00:04:54.540
the only possible
eigenvalues that
00:04:54.540 --> 00:04:59.310
are both 0 and 1 and
bigger than 0 are 1.
00:04:59.310 --> 00:05:09.540
So the conclusion is
that the eigenvalues of P
00:05:09.540 --> 00:05:12.950
must all equal 1.
00:05:12.950 --> 00:05:17.870
So which matrix has eigenvalues
1 and is also symmetric?
00:05:17.870 --> 00:05:26.110
Well, the only matrix that
satisfies this property
00:05:26.110 --> 00:05:28.660
is the identity matrix.
00:05:28.660 --> 00:05:31.270
Now you might ask, how do
you actually show that?
00:05:31.270 --> 00:05:35.170
Well, you could
argue as follows.
00:05:35.170 --> 00:05:47.350
If P is diagonalizable--
and every symmetric matrix
00:05:47.350 --> 00:05:50.980
is diagonalizable, so
I'm not making this up.
00:05:50.980 --> 00:05:52.770
So if P is
diagonalizable, then you
00:05:52.770 --> 00:05:56.870
can always write P
as some matrix, U,
00:05:56.870 --> 00:06:00.530
times a diagonal matrix--
and we know in this case,
00:06:00.530 --> 00:06:03.900
the diagonal matrix
has eigenvalues 1,
00:06:03.900 --> 00:06:08.907
so it's actually the identity
matrix-- times the inverse
00:06:08.907 --> 00:06:09.990
of the eigenvector matrix.
00:06:12.530 --> 00:06:17.400
But of course, this is
just U times U inverse,
00:06:17.400 --> 00:06:21.660
which then gives me the
identity at the end.
00:06:21.660 --> 00:06:23.970
So U times the identity
times U inverse.
00:06:23.970 --> 00:06:25.859
This is just U times U inverse.
00:06:25.859 --> 00:06:27.900
And of course, U and U
inverse collapse back down
00:06:27.900 --> 00:06:29.050
to the identity.
00:06:29.050 --> 00:06:32.850
So this shows you that the only
matrix that has eigenvalues
00:06:32.850 --> 00:06:35.420
of 1 is the identity matrix.
00:06:35.420 --> 00:06:39.550
So that's just to cross all
the T's and dot all the I's.
00:06:39.550 --> 00:06:40.050
OK.
00:06:40.050 --> 00:06:45.120
For part C, we're given
D as a diagonal matrix
00:06:45.120 --> 00:06:47.140
with positive entries
on the diagonal.
00:06:47.140 --> 00:06:50.140
Now we have to show that
it's positive definite.
00:06:50.140 --> 00:06:50.640
OK.
00:06:50.640 --> 00:06:53.102
So let me write D as follows.
00:06:53.102 --> 00:06:54.310
I'll just write it like this.
00:06:54.310 --> 00:06:57.580
I'm going to use a compact
notation, which is sometimes
00:06:57.580 --> 00:07:05.600
seen: diagonal d_1, d_2, d_n.
00:07:05.600 --> 00:07:09.010
So D is a diagonal matrix whose
elements along the diagonal
00:07:09.010 --> 00:07:13.130
are d_1, d_2, dot,
dot, dot, to d_n.
00:07:13.130 --> 00:07:16.220
Now what does it mean for a
matrix to be positive definite?
00:07:16.220 --> 00:07:28.150
Well, it means that for
any x, for any vector x,
00:07:28.150 --> 00:07:34.950
I have to look at the
product x transpose D*x.
00:07:34.950 --> 00:07:38.070
And I have to show that
it's bigger than 0.
00:07:38.070 --> 00:07:41.100
And I should
qualify this and say
00:07:41.100 --> 00:07:43.330
that the vector we're looking
at is x not equal to 0.
00:07:43.330 --> 00:07:46.050
So we're now looking
at the zero vector.
00:07:46.050 --> 00:07:47.710
But for D to be
positive definite,
00:07:47.710 --> 00:07:51.320
we have to show that x
transpose D*x is bigger than 0.
00:07:51.320 --> 00:07:53.894
This is just one way to show
that it's positive definite.
00:07:53.894 --> 00:07:54.810
It's not the only way.
00:08:00.160 --> 00:08:06.872
So if I write x out using
components, x_1, x_2, dot, dot,
00:08:06.872 --> 00:08:15.300
dot, to x_n-- I'll
write it like this--
00:08:15.300 --> 00:08:19.050
then you can work out the
quantity x transpose D*x.
00:08:19.050 --> 00:08:21.160
And we see that we
get a sum of squares.
00:08:21.160 --> 00:08:28.690
We get d_1 times x_1 squared
plus d_2 x_2 squared plus dot,
00:08:28.690 --> 00:08:33.694
dot, dot plus d_n x_n squared.
00:08:36.960 --> 00:08:41.750
Now by definition, each
coefficient is positive.
00:08:41.750 --> 00:08:44.290
A sum of a square is positive.
00:08:44.290 --> 00:08:45.980
so-- sorry.
00:08:45.980 --> 00:08:49.180
A product of a positive number
with a square is positive.
00:08:49.180 --> 00:08:51.220
And then of course, a
sum of positive numbers
00:08:51.220 --> 00:08:54.060
is going to be positive.
00:08:54.060 --> 00:08:57.340
So this means the whole
thing is positive.
00:08:57.340 --> 00:08:59.940
Now there's other
more efficient ways
00:08:59.940 --> 00:09:01.950
of getting at this using
other tricks we know.
00:09:01.950 --> 00:09:04.230
For example, if we're
given a diagonal matrix,
00:09:04.230 --> 00:09:06.700
we know its eigenvalues
are already d_1,
00:09:06.700 --> 00:09:09.130
d_2, dot, dot, dot, to d_n.
00:09:09.130 --> 00:09:13.240
And we know that a matrix
with positive eigenvalues
00:09:13.240 --> 00:09:16.010
is already positive definite.
00:09:16.010 --> 00:09:20.350
But this is kind of
starting from the base
00:09:20.350 --> 00:09:23.930
to show that it's
positive definite.
00:09:23.930 --> 00:09:27.910
And now lastly,
let's look at part D.
00:09:27.910 --> 00:09:32.730
So S is a symmetric matrix
with det S bigger than 0.
00:09:32.730 --> 00:09:34.510
Show that S might
not necessarily
00:09:34.510 --> 00:09:36.110
be positive definite.
00:09:36.110 --> 00:09:38.020
So there's lots of
counterexamples.
00:09:38.020 --> 00:09:39.480
I only need to construct one.
00:09:42.390 --> 00:09:44.837
So I'm looking at S, which
is a symmetric matrix.
00:09:44.837 --> 00:09:46.920
So I'm just going to throw
in some numbers on some
00:09:46.920 --> 00:09:47.930
off diagonals.
00:09:47.930 --> 00:09:50.400
So I'll just pick one.
00:09:50.400 --> 00:09:55.040
And now I need to pick some
numbers along the diagonal.
00:09:55.040 --> 00:09:56.710
And the easiest way
to do it is just
00:09:56.710 --> 00:09:59.900
to pick two negative
numbers on the diagonal.
00:09:59.900 --> 00:10:04.170
Because if there's a negative
number on a diagonal,
00:10:04.170 --> 00:10:06.972
then we know S can't
be positive definite.
00:10:06.972 --> 00:10:08.680
And I'll say more
about that in a second.
00:10:12.390 --> 00:10:14.380
So I can just pick
negative 2 and negative 3.
00:10:14.380 --> 00:10:15.955
So let's quickly
check what det S is.
00:10:21.520 --> 00:10:25.460
Well, it's negative 2
times negative 3 minus 1.
00:10:25.460 --> 00:10:29.820
So this gives me 6
minus 1, which is 5.
00:10:29.820 --> 00:10:33.610
So by construction, det S
is positive, which is good.
00:10:33.610 --> 00:10:37.410
And then, as I mentioned
before, if there's
00:10:37.410 --> 00:10:39.700
negative elements along
the diagonal of the matrix,
00:10:39.700 --> 00:10:41.500
that matrix can't be
positive definite.
00:10:41.500 --> 00:10:43.250
Well, why is that?
00:10:43.250 --> 00:10:45.850
Well, suppose I
wanted to take a look
00:10:45.850 --> 00:10:52.730
at this upper left
component, negative 3,
00:10:52.730 --> 00:10:55.370
and it's negative, how do
I show that that implies
00:10:55.370 --> 00:10:58.710
S is not positive definite?
00:10:58.710 --> 00:11:03.133
Well, what I can do is I can
look at the product x transpose
00:11:03.133 --> 00:11:03.633
S*x.
00:11:06.790 --> 00:11:08.750
And what I do is I look at it.
00:11:08.750 --> 00:11:12.170
I just take one value of x.
00:11:12.170 --> 00:11:15.050
So we know that this has to be
positive for every value of x.
00:11:15.050 --> 00:11:17.280
So I can pick any value I want.
00:11:17.280 --> 00:11:22.660
So I can take x, say,
[1, 0] transpose.
00:11:25.600 --> 00:11:33.930
And when I do this, we end up
getting that x transpose S*x is
00:11:33.930 --> 00:11:37.270
equal to negative 3.
00:11:37.270 --> 00:11:40.370
So notice how by taking 1
in the first entry and 0
00:11:40.370 --> 00:11:43.770
on the second entry, that picks
out the upper left corner,
00:11:43.770 --> 00:11:44.990
negative 3.
00:11:44.990 --> 00:11:48.010
If I were to take
0 here and 1 here,
00:11:48.010 --> 00:11:52.000
it would pick out the
negative 2 entry in S.
00:11:52.000 --> 00:11:58.200
So by picking 1 in
entry i of a vector x,
00:11:58.200 --> 00:12:01.530
and then computing this
product x transpose S*x,
00:12:01.530 --> 00:12:05.640
I pick out the i-th
element along the diagonal.
00:12:05.640 --> 00:12:09.740
And since that
element is negative,
00:12:09.740 --> 00:12:12.490
this shows me that along
the direction [1, 0],
00:12:12.490 --> 00:12:15.440
the product x transpose
S*x is also negative.
00:12:15.440 --> 00:12:19.090
And hence, S can't possibly
be positive definite.
00:12:19.090 --> 00:12:19.990
OK.
00:12:19.990 --> 00:12:21.730
So just to summarize,
we've taken a look
00:12:21.730 --> 00:12:25.120
at a couple matrices and a
couple different properties
00:12:25.120 --> 00:12:27.090
of positive definite matrices.
00:12:27.090 --> 00:12:30.510
And notably, we've
used the eigenvalues
00:12:30.510 --> 00:12:33.330
to get a handle of the
positive definite matrices.
00:12:33.330 --> 00:12:36.120
And I hope these provide
some useful tricks.
00:12:36.120 --> 00:12:38.570
And I'll see you next time.