1
00:00:08,491 --> 00:00:09,490
PROFESSOR: Hi, everyone.
2
00:00:09,490 --> 00:00:10,780
Welcome back.
3
00:00:10,780 --> 00:00:14,050
So today, I'd like to talk about
positive definite matrices.
4
00:00:14,050 --> 00:00:17,650
And specifically, we're going
to analyze several properties
5
00:00:17,650 --> 00:00:19,440
of positive definite matrices.
6
00:00:19,440 --> 00:00:21,860
And specifically,
we're going to look
7
00:00:21,860 --> 00:00:24,710
at why each one of these
following statements is true.
8
00:00:24,710 --> 00:00:29,970
So first off, why every positive
definite matrix is invertible.
9
00:00:29,970 --> 00:00:32,270
Why the only positive
definite projection matrix
10
00:00:32,270 --> 00:00:34,860
is the identity matrix.
11
00:00:34,860 --> 00:00:38,320
If D is a diagonal matrix
with positive entries,
12
00:00:38,320 --> 00:00:41,010
show that it must also
be positive definite.
13
00:00:41,010 --> 00:00:44,260
And then lastly, if S is
a symmetric matrix where
14
00:00:44,260 --> 00:00:46,980
the determinant S
is bigger than 0,
15
00:00:46,980 --> 00:00:48,970
show why this might
not necessarily
16
00:00:48,970 --> 00:00:50,870
imply that it's
positive definite.
17
00:00:50,870 --> 00:00:53,239
So I'll let you think
about these for a moment.
18
00:00:53,239 --> 00:00:54,530
And I'll come back in a second.
19
00:01:08,730 --> 00:01:09,410
Hi, everyone.
20
00:01:09,410 --> 00:01:10,640
Welcome back.
21
00:01:10,640 --> 00:01:11,170
OK.
22
00:01:11,170 --> 00:01:15,820
So let's take a look
at part A. So part A
23
00:01:15,820 --> 00:01:21,140
is asking why every positive
definite matrix is invertible.
24
00:01:21,140 --> 00:01:24,680
Well, let's just recall
that if A is a matrix
25
00:01:24,680 --> 00:01:33,010
and if A is invertible,
then this necessarily
26
00:01:33,010 --> 00:01:37,795
implies that the determinant
of A is non-zero.
27
00:01:40,890 --> 00:01:42,500
And I'm going to
just write out det
28
00:01:42,500 --> 00:01:53,710
A as the product of the
eigenvalues of A. So lambda_1
29
00:01:53,710 --> 00:02:06,180
to lambda_n are the
eigenvalues of A.
30
00:02:06,180 --> 00:02:07,070
OK.
31
00:02:07,070 --> 00:02:18,480
In addition, if A is
positive definite,
32
00:02:18,480 --> 00:02:21,000
what does this say about
the eigenvalues of A?
33
00:02:21,000 --> 00:02:24,900
Well, it says that each
eigenvalue of A, lambda_1,
34
00:02:24,900 --> 00:02:30,700
lambda_2, dot, dot, dot, to
lambda_n, each one of them
35
00:02:30,700 --> 00:02:33,440
must be bigger than 0.
36
00:02:33,440 --> 00:02:36,700
So this statement that
each eigenvalue of A
37
00:02:36,700 --> 00:02:38,790
is bigger than 0 is
completely equivalent
38
00:02:38,790 --> 00:02:46,860
to A being positive definite
for symmetric matrices A.
39
00:02:46,860 --> 00:02:50,260
So if I have a whole bunch of
eigenvalues and each of them
40
00:02:50,260 --> 00:02:52,880
are bigger than 0, what
does this say about det A?
41
00:02:56,530 --> 00:02:58,780
Well, I can take the product
of all these eigenvalues.
42
00:03:03,460 --> 00:03:06,740
And of course, the product of a
whole bunch of positive numbers
43
00:03:06,740 --> 00:03:07,660
must also be positive.
44
00:03:10,540 --> 00:03:14,640
So if the quantity is positive,
then it certainly cannot equal
45
00:03:14,640 --> 00:03:15,800
0.
46
00:03:15,800 --> 00:03:18,770
So this proves that det
A is not equal to 0.
47
00:03:18,770 --> 00:03:22,590
Hence, A must be invertible.
48
00:03:22,590 --> 00:03:23,090
OK.
49
00:03:23,090 --> 00:03:29,850
So for part B,
we're asked to show
50
00:03:29,850 --> 00:03:31,810
that the only positive
definite projection
51
00:03:31,810 --> 00:03:35,070
matrix is the identity matrix.
52
00:03:35,070 --> 00:03:37,220
So again, how do we
tackle this problem?
53
00:03:37,220 --> 00:03:39,930
We're going to look
at the eigenvalues.
54
00:03:39,930 --> 00:03:49,690
So remember, if P
is a projection,
55
00:03:49,690 --> 00:03:52,080
what does it say about
the eigenvalues of P?
56
00:03:52,080 --> 00:04:03,520
Well, it says that
the eigenvalues of P
57
00:04:03,520 --> 00:04:05,370
are either 0 or 1.
58
00:04:09,390 --> 00:04:11,520
So this is point one.
59
00:04:11,520 --> 00:04:25,150
Point two, if P is
a positive definite,
60
00:04:25,150 --> 00:04:27,706
what does that say about
the eigenvalues of P?
61
00:04:27,706 --> 00:04:43,670
Well, as I've noted before,
it means that the eigenvalues
62
00:04:43,670 --> 00:04:44,580
are bigger than 0.
63
00:04:47,210 --> 00:04:51,700
So if P is a projection
and it's positive definite,
64
00:04:51,700 --> 00:04:54,540
the only possible
eigenvalues that
65
00:04:54,540 --> 00:04:59,310
are both 0 and 1 and
bigger than 0 are 1.
66
00:04:59,310 --> 00:05:09,540
So the conclusion is
that the eigenvalues of P
67
00:05:09,540 --> 00:05:12,950
must all equal 1.
68
00:05:12,950 --> 00:05:17,870
So which matrix has eigenvalues
1 and is also symmetric?
69
00:05:17,870 --> 00:05:26,110
Well, the only matrix that
satisfies this property
70
00:05:26,110 --> 00:05:28,660
is the identity matrix.
71
00:05:28,660 --> 00:05:31,270
Now you might ask, how do
you actually show that?
72
00:05:31,270 --> 00:05:35,170
Well, you could
argue as follows.
73
00:05:35,170 --> 00:05:47,350
If P is diagonalizable--
and every symmetric matrix
74
00:05:47,350 --> 00:05:50,980
is diagonalizable, so
I'm not making this up.
75
00:05:50,980 --> 00:05:52,770
So if P is
diagonalizable, then you
76
00:05:52,770 --> 00:05:56,870
can always write P
as some matrix, U,
77
00:05:56,870 --> 00:06:00,530
times a diagonal matrix--
and we know in this case,
78
00:06:00,530 --> 00:06:03,900
the diagonal matrix
has eigenvalues 1,
79
00:06:03,900 --> 00:06:08,907
so it's actually the identity
matrix-- times the inverse
80
00:06:08,907 --> 00:06:09,990
of the eigenvector matrix.
81
00:06:12,530 --> 00:06:17,400
But of course, this is
just U times U inverse,
82
00:06:17,400 --> 00:06:21,660
which then gives me the
identity at the end.
83
00:06:21,660 --> 00:06:23,970
So U times the identity
times U inverse.
84
00:06:23,970 --> 00:06:25,859
This is just U times U inverse.
85
00:06:25,859 --> 00:06:27,900
And of course, U and U
inverse collapse back down
86
00:06:27,900 --> 00:06:29,050
to the identity.
87
00:06:29,050 --> 00:06:32,850
So this shows you that the only
matrix that has eigenvalues
88
00:06:32,850 --> 00:06:35,420
of 1 is the identity matrix.
89
00:06:35,420 --> 00:06:39,550
So that's just to cross all
the T's and dot all the I's.
90
00:06:39,550 --> 00:06:40,050
OK.
91
00:06:40,050 --> 00:06:45,120
For part C, we're given
D as a diagonal matrix
92
00:06:45,120 --> 00:06:47,140
with positive entries
on the diagonal.
93
00:06:47,140 --> 00:06:50,140
Now we have to show that
it's positive definite.
94
00:06:50,140 --> 00:06:50,640
OK.
95
00:06:50,640 --> 00:06:53,102
So let me write D as follows.
96
00:06:53,102 --> 00:06:54,310
I'll just write it like this.
97
00:06:54,310 --> 00:06:57,580
I'm going to use a compact
notation, which is sometimes
98
00:06:57,580 --> 00:07:05,600
seen: diagonal d_1, d_2, d_n.
99
00:07:05,600 --> 00:07:09,010
So D is a diagonal matrix whose
elements along the diagonal
100
00:07:09,010 --> 00:07:13,130
are d_1, d_2, dot,
dot, dot, to d_n.
101
00:07:13,130 --> 00:07:16,220
Now what does it mean for a
matrix to be positive definite?
102
00:07:16,220 --> 00:07:28,150
Well, it means that for
any x, for any vector x,
103
00:07:28,150 --> 00:07:34,950
I have to look at the
product x transpose D*x.
104
00:07:34,950 --> 00:07:38,070
And I have to show that
it's bigger than 0.
105
00:07:38,070 --> 00:07:41,100
And I should
qualify this and say
106
00:07:41,100 --> 00:07:43,330
that the vector we're looking
at is x not equal to 0.
107
00:07:43,330 --> 00:07:46,050
So we're now looking
at the zero vector.
108
00:07:46,050 --> 00:07:47,710
But for D to be
positive definite,
109
00:07:47,710 --> 00:07:51,320
we have to show that x
transpose D*x is bigger than 0.
110
00:07:51,320 --> 00:07:53,894
This is just one way to show
that it's positive definite.
111
00:07:53,894 --> 00:07:54,810
It's not the only way.
112
00:08:00,160 --> 00:08:06,872
So if I write x out using
components, x_1, x_2, dot, dot,
113
00:08:06,872 --> 00:08:15,300
dot, to x_n-- I'll
write it like this--
114
00:08:15,300 --> 00:08:19,050
then you can work out the
quantity x transpose D*x.
115
00:08:19,050 --> 00:08:21,160
And we see that we
get a sum of squares.
116
00:08:21,160 --> 00:08:28,690
We get d_1 times x_1 squared
plus d_2 x_2 squared plus dot,
117
00:08:28,690 --> 00:08:33,694
dot, dot plus d_n x_n squared.
118
00:08:36,960 --> 00:08:41,750
Now by definition, each
coefficient is positive.
119
00:08:41,750 --> 00:08:44,290
A sum of a square is positive.
120
00:08:44,290 --> 00:08:45,980
so-- sorry.
121
00:08:45,980 --> 00:08:49,180
A product of a positive number
with a square is positive.
122
00:08:49,180 --> 00:08:51,220
And then of course, a
sum of positive numbers
123
00:08:51,220 --> 00:08:54,060
is going to be positive.
124
00:08:54,060 --> 00:08:57,340
So this means the whole
thing is positive.
125
00:08:57,340 --> 00:08:59,940
Now there's other
more efficient ways
126
00:08:59,940 --> 00:09:01,950
of getting at this using
other tricks we know.
127
00:09:01,950 --> 00:09:04,230
For example, if we're
given a diagonal matrix,
128
00:09:04,230 --> 00:09:06,700
we know its eigenvalues
are already d_1,
129
00:09:06,700 --> 00:09:09,130
d_2, dot, dot, dot, to d_n.
130
00:09:09,130 --> 00:09:13,240
And we know that a matrix
with positive eigenvalues
131
00:09:13,240 --> 00:09:16,010
is already positive definite.
132
00:09:16,010 --> 00:09:20,350
But this is kind of
starting from the base
133
00:09:20,350 --> 00:09:23,930
to show that it's
positive definite.
134
00:09:23,930 --> 00:09:27,910
And now lastly,
let's look at part D.
135
00:09:27,910 --> 00:09:32,730
So S is a symmetric matrix
with det S bigger than 0.
136
00:09:32,730 --> 00:09:34,510
Show that S might
not necessarily
137
00:09:34,510 --> 00:09:36,110
be positive definite.
138
00:09:36,110 --> 00:09:38,020
So there's lots of
counterexamples.
139
00:09:38,020 --> 00:09:39,480
I only need to construct one.
140
00:09:42,390 --> 00:09:44,837
So I'm looking at S, which
is a symmetric matrix.
141
00:09:44,837 --> 00:09:46,920
So I'm just going to throw
in some numbers on some
142
00:09:46,920 --> 00:09:47,930
off diagonals.
143
00:09:47,930 --> 00:09:50,400
So I'll just pick one.
144
00:09:50,400 --> 00:09:55,040
And now I need to pick some
numbers along the diagonal.
145
00:09:55,040 --> 00:09:56,710
And the easiest way
to do it is just
146
00:09:56,710 --> 00:09:59,900
to pick two negative
numbers on the diagonal.
147
00:09:59,900 --> 00:10:04,170
Because if there's a negative
number on a diagonal,
148
00:10:04,170 --> 00:10:06,972
then we know S can't
be positive definite.
149
00:10:06,972 --> 00:10:08,680
And I'll say more
about that in a second.
150
00:10:12,390 --> 00:10:14,380
So I can just pick
negative 2 and negative 3.
151
00:10:14,380 --> 00:10:15,955
So let's quickly
check what det S is.
152
00:10:21,520 --> 00:10:25,460
Well, it's negative 2
times negative 3 minus 1.
153
00:10:25,460 --> 00:10:29,820
So this gives me 6
minus 1, which is 5.
154
00:10:29,820 --> 00:10:33,610
So by construction, det S
is positive, which is good.
155
00:10:33,610 --> 00:10:37,410
And then, as I mentioned
before, if there's
156
00:10:37,410 --> 00:10:39,700
negative elements along
the diagonal of the matrix,
157
00:10:39,700 --> 00:10:41,500
that matrix can't be
positive definite.
158
00:10:41,500 --> 00:10:43,250
Well, why is that?
159
00:10:43,250 --> 00:10:45,850
Well, suppose I
wanted to take a look
160
00:10:45,850 --> 00:10:52,730
at this upper left
component, negative 3,
161
00:10:52,730 --> 00:10:55,370
and it's negative, how do
I show that that implies
162
00:10:55,370 --> 00:10:58,710
S is not positive definite?
163
00:10:58,710 --> 00:11:03,133
Well, what I can do is I can
look at the product x transpose
164
00:11:03,133 --> 00:11:03,633
S*x.
165
00:11:06,790 --> 00:11:08,750
And what I do is I look at it.
166
00:11:08,750 --> 00:11:12,170
I just take one value of x.
167
00:11:12,170 --> 00:11:15,050
So we know that this has to be
positive for every value of x.
168
00:11:15,050 --> 00:11:17,280
So I can pick any value I want.
169
00:11:17,280 --> 00:11:22,660
So I can take x, say,
[1, 0] transpose.
170
00:11:25,600 --> 00:11:33,930
And when I do this, we end up
getting that x transpose S*x is
171
00:11:33,930 --> 00:11:37,270
equal to negative 3.
172
00:11:37,270 --> 00:11:40,370
So notice how by taking 1
in the first entry and 0
173
00:11:40,370 --> 00:11:43,770
on the second entry, that picks
out the upper left corner,
174
00:11:43,770 --> 00:11:44,990
negative 3.
175
00:11:44,990 --> 00:11:48,010
If I were to take
0 here and 1 here,
176
00:11:48,010 --> 00:11:52,000
it would pick out the
negative 2 entry in S.
177
00:11:52,000 --> 00:11:58,200
So by picking 1 in
entry i of a vector x,
178
00:11:58,200 --> 00:12:01,530
and then computing this
product x transpose S*x,
179
00:12:01,530 --> 00:12:05,640
I pick out the i-th
element along the diagonal.
180
00:12:05,640 --> 00:12:09,740
And since that
element is negative,
181
00:12:09,740 --> 00:12:12,490
this shows me that along
the direction [1, 0],
182
00:12:12,490 --> 00:12:15,440
the product x transpose
S*x is also negative.
183
00:12:15,440 --> 00:12:19,090
And hence, S can't possibly
be positive definite.
184
00:12:19,090 --> 00:12:19,990
OK.
185
00:12:19,990 --> 00:12:21,730
So just to summarize,
we've taken a look
186
00:12:21,730 --> 00:12:25,120
at a couple matrices and a
couple different properties
187
00:12:25,120 --> 00:12:27,090
of positive definite matrices.
188
00:12:27,090 --> 00:12:30,510
And notably, we've
used the eigenvalues
189
00:12:30,510 --> 00:12:33,330
to get a handle of the
positive definite matrices.
190
00:12:33,330 --> 00:12:36,120
And I hope these provide
some useful tricks.
191
00:12:36,120 --> 00:12:38,570
And I'll see you next time.