1
00:00:07,270 --> 00:00:12,540
OK, here's the last lecture in
the chapter on orthogonality.
2
00:00:12,540 --> 00:00:16,900
So we met orthogonal
vectors, two vectors,
3
00:00:16,900 --> 00:00:22,530
we met orthogonal subspaces,
like the row space and null
4
00:00:22,530 --> 00:00:23,590
space.
5
00:00:23,590 --> 00:00:28,720
Now today we meet
an orthogonal basis,
6
00:00:28,720 --> 00:00:31,370
and an orthogonal matrix.
7
00:00:31,370 --> 00:00:32,920
So we really --
8
00:00:32,920 --> 00:00:36,170
this chapter cleans
up orthogonality.
9
00:00:36,170 --> 00:00:38,930
And really I want --
10
00:00:38,930 --> 00:00:43,880
I should use the
word orthonormal.
11
00:00:43,880 --> 00:00:51,690
Orthogonal is -- so my
vectors are q1,q2 up to qn --
12
00:00:51,690 --> 00:00:56,180
I use the letter "q",
here, to remind me,
13
00:00:56,180 --> 00:01:01,270
I'm talking about orthogonal
things, not just any vectors,
14
00:01:01,270 --> 00:01:02,690
but orthogonal ones.
15
00:01:02,690 --> 00:01:04,230
So what does that mean?
16
00:01:04,230 --> 00:01:08,600
That means that every q is
orthogonal to every other q.
17
00:01:08,600 --> 00:01:13,880
It's a natural idea,
to have a basis that's
18
00:01:13,880 --> 00:01:17,060
headed off at
ninety-degree angles,
19
00:01:17,060 --> 00:01:19,120
the inner products are all zero.
20
00:01:19,120 --> 00:01:26,260
Of course if q is -- certainly
qi is not orthogonal to itself.
21
00:01:26,260 --> 00:01:29,200
But there we'll make
the best choice again,
22
00:01:29,200 --> 00:01:31,050
make it a unit vector.
23
00:01:31,050 --> 00:01:35,610
Then qi transpose qi is
one, for a unit vector.
24
00:01:35,610 --> 00:01:37,590
The length squared is one.
25
00:01:37,590 --> 00:01:41,430
And that's what I would
use the word normal.
26
00:01:41,430 --> 00:01:48,479
So for this part, normalized,
unit length for this part.
27
00:01:48,479 --> 00:01:48,979
OK.
28
00:01:48,979 --> 00:01:55,120
So first part of
the lecture is how
29
00:01:55,120 --> 00:01:59,320
does having an orthonormal
basis make things nice?
30
00:01:59,320 --> 00:02:00,330
It certainly does.
31
00:02:00,330 --> 00:02:02,490
It makes all the
calculations better,
32
00:02:02,490 --> 00:02:05,150
a whole lot of
numerical linear algebra
33
00:02:05,150 --> 00:02:10,520
is built around working
with orthonormal vectors,
34
00:02:10,520 --> 00:02:12,230
because they never
get out of hand,
35
00:02:12,230 --> 00:02:16,220
they never overflow
or underflow.
36
00:02:16,220 --> 00:02:20,390
And I'll put them
into a matrix Q,
37
00:02:20,390 --> 00:02:22,430
and then the second
part of the lecture
38
00:02:22,430 --> 00:02:26,630
will be suppose my
basis, my columns of A
39
00:02:26,630 --> 00:02:28,710
are not orthonormal.
40
00:02:28,710 --> 00:02:30,660
How do I make them so?
41
00:02:30,660 --> 00:02:35,110
And the two names associated
with that simple idea
42
00:02:35,110 --> 00:02:36,840
are Graham and Schmidt.
43
00:02:36,840 --> 00:02:43,410
So the first part is we've
got a basis like this.
44
00:02:43,410 --> 00:02:46,040
Let's put those into
the columns of a matrix.
45
00:02:48,620 --> 00:02:53,990
So a matrix Q that has --
46
00:02:53,990 --> 00:02:56,680
I'll put these
orthonormal vectors,
47
00:02:56,680 --> 00:03:01,650
q1 will be the first column,
qn will be the n-th column.
48
00:03:04,350 --> 00:03:09,610
And I want to say, I want
to write this property,
49
00:03:09,610 --> 00:03:13,890
qi transpose qj
being zero, I want
50
00:03:13,890 --> 00:03:17,640
to put that in a matrix form.
51
00:03:17,640 --> 00:03:23,390
And just the right thing is
to look at Q transpose Q.
52
00:03:23,390 --> 00:03:27,520
So this chapter has been
looking at A transpose A.
53
00:03:27,520 --> 00:03:30,470
So it's natural to
look at Q transpose Q.
54
00:03:30,470 --> 00:03:33,490
And the beauty is it
comes out perfectly.
55
00:03:33,490 --> 00:03:38,160
Because Q transpose has
these vectors in its rows,
56
00:03:38,160 --> 00:03:45,510
the first row is q1 transpose,
the nth row is qn transpose.
57
00:03:45,510 --> 00:03:49,070
So that's Q transpose.
58
00:03:49,070 --> 00:03:51,610
And now I want to multiply by Q.
59
00:03:51,610 --> 00:03:56,810
That has q1 along to
qn in the columns.
60
00:03:56,810 --> 00:03:58,430
That's Q.
61
00:03:58,430 --> 00:03:59,410
And what do I get?
62
00:04:01,940 --> 00:04:06,240
You really -- this is the
first simplest most basic fact,
63
00:04:06,240 --> 00:04:10,930
that how do orthonormal
vectors, orthonormal columns
64
00:04:10,930 --> 00:04:17,660
in a matrix, what happens
if I compute Q transpose Q?
65
00:04:17,660 --> 00:04:18,850
Do you see it?
66
00:04:18,850 --> 00:04:23,060
If I take the first row
times the first column,
67
00:04:23,060 --> 00:04:25,170
what do I get?
68
00:04:25,170 --> 00:04:26,680
A one.
69
00:04:26,680 --> 00:04:29,570
If I take the first row
times the second column,
70
00:04:29,570 --> 00:04:31,330
what do I get?
71
00:04:31,330 --> 00:04:32,280
Zero.
72
00:04:32,280 --> 00:04:34,170
That's the orthogonality.
73
00:04:34,170 --> 00:04:37,460
The first row times the
last column is zero.
74
00:04:37,460 --> 00:04:40,630
And so I'm getting
ones on the diagonal
75
00:04:40,630 --> 00:04:43,620
and I'm getting zeroes
everywhere else.
76
00:04:43,620 --> 00:04:45,080
I'm getting the identity matrix.
77
00:04:47,880 --> 00:04:50,920
You see how that's -- it's
just like the right calculation
78
00:04:50,920 --> 00:04:51,740
to do.
79
00:04:51,740 --> 00:04:55,540
If you have orthonormal
columns, and the matrix
80
00:04:55,540 --> 00:04:59,420
doesn't have to be square here.
81
00:04:59,420 --> 00:05:02,290
We might have just two columns.
82
00:05:02,290 --> 00:05:05,670
And they might have
four, lots of components.
83
00:05:05,670 --> 00:05:13,370
So but they're orthonormal, and
when we do Q transpose times Q,
84
00:05:13,370 --> 00:05:17,110
that Q transpose times
Q or A transpose A
85
00:05:17,110 --> 00:05:22,680
just asks for all
those dot products.
86
00:05:22,680 --> 00:05:24,530
Rows times columns.
87
00:05:24,530 --> 00:05:29,390
And in this orthonormal case,
we get the best possible answer,
88
00:05:29,390 --> 00:05:30,850
the identity.
89
00:05:30,850 --> 00:05:33,330
OK, so this is --
90
00:05:33,330 --> 00:05:39,690
so I mean now we have a new
bunch of important matrices.
91
00:05:39,690 --> 00:05:41,070
What have we seen previously?
92
00:05:41,070 --> 00:05:43,440
We've seen in the
distant past we
93
00:05:43,440 --> 00:05:48,180
had triangular matrices,
diagonal matrices, permutation
94
00:05:48,180 --> 00:05:51,760
matrices, that was
early chapters,
95
00:05:51,760 --> 00:05:59,190
then we had row echelon
forms, then in this chapter
96
00:05:59,190 --> 00:06:02,900
we've already seen
projection matrices,
97
00:06:02,900 --> 00:06:08,830
and now we're seeing this
new class of matrices
98
00:06:08,830 --> 00:06:11,490
with orthonormal columns.
99
00:06:11,490 --> 00:06:13,510
That's a very long expression.
100
00:06:13,510 --> 00:06:19,020
I sorry that I can't just
call them orthogonal matrices.
101
00:06:19,020 --> 00:06:22,090
But that word
orthogonal matrices --
102
00:06:22,090 --> 00:06:25,440
or maybe I should be able to
call it orthonormal matrices,
103
00:06:25,440 --> 00:06:28,090
why don't we call
it orthonormal --
104
00:06:28,090 --> 00:06:31,290
I mean that would be an
absolutely perfect name.
105
00:06:31,290 --> 00:06:33,860
For Q, call it an
orthonormal matrix
106
00:06:33,860 --> 00:06:36,370
because its columns
are orthonormal.
107
00:06:36,370 --> 00:06:42,270
OK, but the convention is
that we only use that name
108
00:06:42,270 --> 00:06:46,930
orthogonal matrix,
we only use this --
109
00:06:46,930 --> 00:06:49,670
this word orthogonal,
we don't even
110
00:06:49,670 --> 00:06:51,950
say orthonormal for
some unknown reason,
111
00:06:51,950 --> 00:06:54,530
matrix when it's square.
112
00:06:59,130 --> 00:07:04,740
So in the case when this is a
square matrix, that's the case
113
00:07:04,740 --> 00:07:07,830
we call it an orthogonal matrix.
114
00:07:07,830 --> 00:07:12,290
And what's special about
the case when it's square?
115
00:07:12,290 --> 00:07:19,390
When it's a square matrix,
we've got its inverse, so --
116
00:07:19,390 --> 00:07:33,850
so in the case if Q is square,
then Q transpose Q equals I
117
00:07:33,850 --> 00:07:35,730
tells us --
118
00:07:35,730 --> 00:07:37,820
let me write that underneath --
119
00:07:37,820 --> 00:07:47,670
tells us that Q
transpose is Q inverse.
120
00:07:47,670 --> 00:07:51,000
There we have the
easy to remember
121
00:07:51,000 --> 00:07:57,440
property for a square matrix
with orthonormal columns.
122
00:07:57,440 --> 00:08:01,960
That -- I need to write
some examples down.
123
00:08:01,960 --> 00:08:04,180
Let's see.
124
00:08:04,180 --> 00:08:07,750
Some examples like if I
take any -- so examples,
125
00:08:07,750 --> 00:08:08,720
let's do some examples.
126
00:08:12,100 --> 00:08:14,830
Any permutation matrix,
let me take just
127
00:08:14,830 --> 00:08:17,360
some random permutation matrix.
128
00:08:17,360 --> 00:08:23,910
Permutation Q equals let's say
oh, make it three by three,
129
00:08:23,910 --> 00:08:30,050
say zero, zero, one, one,
zero, zero, zero, one, zero.
130
00:08:30,050 --> 00:08:30,550
OK.
131
00:08:36,470 --> 00:08:41,860
That certainly has unit
vectors in its columns.
132
00:08:41,860 --> 00:08:45,940
Those vectors are certainly
perpendicular to each other.
133
00:08:45,940 --> 00:08:48,700
And if I -- and so that's it.
134
00:08:48,700 --> 00:08:50,440
That makes it a Q.
135
00:08:50,440 --> 00:08:56,360
And -- if I took its transpose,
if I multiplied by Q transpose,
136
00:08:56,360 --> 00:08:59,640
shall I do that -- and let
me stick in Q transpose
137
00:08:59,640 --> 00:09:00,400
here.
138
00:09:00,400 --> 00:09:02,990
Just to do that
multiplication once more,
139
00:09:02,990 --> 00:09:05,200
transpose it'll put the --
140
00:09:05,200 --> 00:09:08,720
make that into a column,
make that into a column,
141
00:09:08,720 --> 00:09:10,920
make that into a column.
142
00:09:10,920 --> 00:09:14,200
And the transpose is also --
143
00:09:14,200 --> 00:09:15,690
another Q.
144
00:09:15,690 --> 00:09:17,070
Another orthonormal matrix.
145
00:09:17,070 --> 00:09:22,110
And when I multiply that
product I get I. OK,
146
00:09:22,110 --> 00:09:23,890
so there's an example.
147
00:09:23,890 --> 00:09:26,860
And actually there's
a second example.
148
00:09:26,860 --> 00:09:29,420
But those are real
easy examples, right,
149
00:09:29,420 --> 00:09:34,780
I mean to get orthogonal
columns by just
150
00:09:34,780 --> 00:09:40,630
putting ones in different
places is like too easy.
151
00:09:40,630 --> 00:09:43,030
So let me keep
going with examples.
152
00:09:43,030 --> 00:09:46,170
So here's another
simple example.
153
00:09:46,170 --> 00:09:51,170
Cos theta sine theta,
there's a unit vector,
154
00:09:51,170 --> 00:09:53,920
oh, let me even
take it, well, yeah.
155
00:09:53,920 --> 00:09:58,500
Cos theta sine theta
and now the other way
156
00:09:58,500 --> 00:10:01,790
I want sine theta cos theta.
157
00:10:01,790 --> 00:10:05,700
But I want the inner
product to be zero.
158
00:10:05,700 --> 00:10:09,150
And if I put a minus
there, it'll do it.
159
00:10:09,150 --> 00:10:12,760
So that's -- unit vector,
that's a unit vector.
160
00:10:12,760 --> 00:10:16,920
And if I take the dot product,
I get minus plus zero.
161
00:10:19,470 --> 00:10:19,970
OK.
162
00:10:19,970 --> 00:10:27,230
For example Q equals say
one, one, one, minus one,
163
00:10:27,230 --> 00:10:30,710
is that an orthogonal matrix?
164
00:10:33,280 --> 00:10:35,600
I've got orthogonal
columns there,
165
00:10:35,600 --> 00:10:37,920
but it's not quite
an orthogonal matrix.
166
00:10:37,920 --> 00:10:43,000
How shall I fix it to
be an orthogonal matrix?
167
00:10:43,000 --> 00:10:45,780
Well, what's the length
of those column vectors,
168
00:10:45,780 --> 00:10:51,540
the dot product with themselves
is -- right now it's two,
169
00:10:51,540 --> 00:10:52,760
right, the --
170
00:10:52,760 --> 00:10:55,040
the length squared.
171
00:10:55,040 --> 00:10:57,710
The length squared would be
one plus one would be two,
172
00:10:57,710 --> 00:10:59,340
the length would be
square root of two,
173
00:10:59,340 --> 00:11:02,790
so I better divide by
square root of two.
174
00:11:02,790 --> 00:11:03,750
OK.
175
00:11:03,750 --> 00:11:08,080
So there's a -- there now I
have got an orthogonal matrix,
176
00:11:08,080 --> 00:11:13,350
in fact, it's this one --
when theta is pi over four.
177
00:11:13,350 --> 00:11:16,210
The cosines and
well almost, I guess
178
00:11:16,210 --> 00:11:19,610
the minus sine is down
there, so maybe, I
179
00:11:19,610 --> 00:11:23,650
don't know, maybe minus
pi over four or something.
180
00:11:23,650 --> 00:11:24,150
OK.
181
00:11:26,760 --> 00:11:28,760
Let me do one
final example, just
182
00:11:28,760 --> 00:11:32,000
to show that you
can get bigger ones.
183
00:11:32,000 --> 00:11:38,270
Q equals let me take that
matrix up in the corner
184
00:11:38,270 --> 00:11:41,830
and I'll sort of
repeat that pattern,
185
00:11:41,830 --> 00:11:47,010
repeat it again, and
then minus it down here.
186
00:11:51,110 --> 00:11:57,800
That's one of the world's
favorite orthogonal matrices.
187
00:11:57,800 --> 00:12:00,090
I hope I got it right, is --
188
00:12:00,090 --> 00:12:02,500
can you see whether --
189
00:12:02,500 --> 00:12:05,890
if I take the inner product of
one column with another one,
190
00:12:05,890 --> 00:12:07,990
let's see, if I take
the inner product
191
00:12:07,990 --> 00:12:11,070
of that column with that I have
two minuses and two pluses,
192
00:12:11,070 --> 00:12:11,960
that's good.
193
00:12:11,960 --> 00:12:14,280
When I take the inner
product of that with that
194
00:12:14,280 --> 00:12:17,390
I have a plus and a
minus, a minus and a plus.
195
00:12:17,390 --> 00:12:17,890
Good.
196
00:12:17,890 --> 00:12:19,530
I think it all works out.
197
00:12:19,530 --> 00:12:22,210
And what do I have
to divide by now?
198
00:12:22,210 --> 00:12:24,500
To make those into unit vectors.
199
00:12:27,320 --> 00:12:34,520
Right now the vector one,
one, one, one has length two.
200
00:12:34,520 --> 00:12:35,660
Square root of four.
201
00:12:35,660 --> 00:12:39,680
So I have to divide by two
to make it unit vector,
202
00:12:39,680 --> 00:12:41,190
so there's another.
203
00:12:41,190 --> 00:12:45,550
That's my entire array
of simple examples.
204
00:12:49,190 --> 00:12:56,610
This construction is named after
a guy called Adhemar and we
205
00:12:56,610 --> 00:13:02,960
know how to do it for
two, four, sixteen,
206
00:13:02,960 --> 00:13:10,670
sixty-four and so on, but we --
nobody knows exactly which size
207
00:13:10,670 --> 00:13:12,970
matrices have --
208
00:13:12,970 --> 00:13:18,250
which size -- which sizes allow
orthogonal matrices of ones
209
00:13:18,250 --> 00:13:19,300
and minus ones.
210
00:13:19,300 --> 00:13:24,450
So Adhemar matrix is an
orthogonal matrix that's got
211
00:13:24,450 --> 00:13:30,070
ones and minus ones, and a
lot of ones -- some we know,
212
00:13:30,070 --> 00:13:34,130
some other sizes, there couldn't
be a five by five I think.
213
00:13:34,130 --> 00:13:35,970
But there are some
sizes that nobody
214
00:13:35,970 --> 00:13:42,220
yet knows whether there could be
or can't be a matrix like that.
215
00:13:42,220 --> 00:13:43,160
OK.
216
00:13:43,160 --> 00:13:47,860
You see those
orthogonal matrices.
217
00:13:47,860 --> 00:13:54,910
Now let me ask what -- why
is it good to have orthogonal
218
00:13:54,910 --> 00:13:56,050
matrices?
219
00:13:56,050 --> 00:13:59,830
What calculation is made easy?
220
00:13:59,830 --> 00:14:02,040
If I have an orthogonal matrix.
221
00:14:02,040 --> 00:14:06,660
And -- let me remember that the
matrix could be rectangular.
222
00:14:06,660 --> 00:14:07,880
Shall I put down --
223
00:14:07,880 --> 00:14:10,680
I better put a
rectangular example down.
224
00:14:10,680 --> 00:14:13,270
So the -- these were
all square examples.
225
00:14:13,270 --> 00:14:14,910
Can I put down just --
226
00:14:14,910 --> 00:14:18,190
a rectangular one
just to be sure
227
00:14:18,190 --> 00:14:22,080
that we realize that
this is possible.
228
00:14:22,080 --> 00:14:23,930
let's help me out.
229
00:14:23,930 --> 00:14:33,870
Let's see, if I put like a
one, two, two and a minus two,
230
00:14:33,870 --> 00:14:35,630
minus one, two.
231
00:14:40,530 --> 00:14:45,340
That's -- a matrix -- oh its
columns aren't normalized yet.
232
00:14:45,340 --> 00:14:47,950
I always have to
remember to do that.
233
00:14:47,950 --> 00:14:50,590
I always do that last
because it's easy to do.
234
00:14:50,590 --> 00:14:53,460
What's the length
of those columns?
235
00:14:53,460 --> 00:14:56,420
So if I wanted them -- if I
wanted them to be length one,
236
00:14:56,420 --> 00:14:59,840
I should divide by their
length, which is --
237
00:14:59,840 --> 00:15:03,440
so I'd look at one squared plus
two squared plus two squared,
238
00:15:03,440 --> 00:15:06,420
that's one and four
and four is nine,
239
00:15:06,420 --> 00:15:11,420
so I take the square root and
I need to divide by three.
240
00:15:11,420 --> 00:15:11,920
OK.
241
00:15:11,920 --> 00:15:14,780
So there is --
242
00:15:14,780 --> 00:15:21,320
well, without that, I've
got one orthonormal vector.
243
00:15:21,320 --> 00:15:24,060
I mean just one unit vector.
244
00:15:24,060 --> 00:15:26,130
Now put that guy in.
245
00:15:26,130 --> 00:15:29,230
Now I have a basis
for the column
246
00:15:29,230 --> 00:15:34,980
space for a two-dimensional
space, an orthonormal basis,
247
00:15:34,980 --> 00:15:35,500
right?
248
00:15:35,500 --> 00:15:38,140
These two columns
are orthonormal,
249
00:15:38,140 --> 00:15:40,660
they would be an
orthonormal basis
250
00:15:40,660 --> 00:15:45,190
for this two-dimensional
space that they span.
251
00:15:45,190 --> 00:15:48,700
Orthonormal vectors by the way
have got to be independent.
252
00:15:48,700 --> 00:15:52,910
It's easy to show that
orthonormal vectors
253
00:15:52,910 --> 00:15:55,640
since they're headed off
all at ninety degrees
254
00:15:55,640 --> 00:15:58,630
there's no combination
that gives zero.
255
00:15:58,630 --> 00:16:06,970
Now if I wanted to
create now a third one,
256
00:16:06,970 --> 00:16:13,220
I could either just put in
some third vector that was
257
00:16:13,220 --> 00:16:18,800
independent and go to this
Graham-Schmidt calculation that
258
00:16:18,800 --> 00:16:22,860
I'm going to explain, or I
could be inspired and say look,
259
00:16:22,860 --> 00:16:26,990
that -- with that pattern,
why not put a one in there,
260
00:16:26,990 --> 00:16:29,650
and a two in there,
and a two in there,
261
00:16:29,650 --> 00:16:33,260
and try to fix up the
signs so that they worked.
262
00:16:36,530 --> 00:16:37,070
Hmm.
263
00:16:37,070 --> 00:16:41,250
I don't know if I've done
this too brilliantly.
264
00:16:41,250 --> 00:16:43,230
Let's see, what
signs, that's minus,
265
00:16:43,230 --> 00:16:49,270
maybe I'd make a minus sign
there, how would that be?
266
00:16:49,270 --> 00:16:52,990
Yeah, maybe that works.
267
00:16:52,990 --> 00:17:00,170
I think that those three columns
are orthonormal and they --
268
00:17:00,170 --> 00:17:03,720
the beauty of this -- this is
the last example I'll probably
269
00:17:03,720 --> 00:17:08,089
find where there's no
square root, the --
270
00:17:08,089 --> 00:17:11,250
the punishing thing
in Graham-Schmidt,
271
00:17:11,250 --> 00:17:14,720
maybe we better know
that in advance,
272
00:17:14,720 --> 00:17:19,950
is that because I want these
vectors to be unit vectors,
273
00:17:19,950 --> 00:17:21,900
I'm always running
into square roots.
274
00:17:21,900 --> 00:17:24,270
I'm always dividing by lengths.
275
00:17:24,270 --> 00:17:26,109
And those lengths
are square roots.
276
00:17:26,109 --> 00:17:29,900
So you'll see as soon as I
do a Graham-Schmidt example,
277
00:17:29,900 --> 00:17:32,190
square roots are
going to show up.
278
00:17:32,190 --> 00:17:34,960
But here are some
examples where we did it
279
00:17:34,960 --> 00:17:36,850
without any square root.
280
00:17:36,850 --> 00:17:38,900
OK.
281
00:17:38,900 --> 00:17:42,110
So -- so great.
282
00:17:42,110 --> 00:17:50,760
Now next question is what's
the good of having a Q?
283
00:17:50,760 --> 00:17:52,860
What formulas become easier?
284
00:17:52,860 --> 00:17:57,540
Suppose I want to
project, so suppose Q --
285
00:17:57,540 --> 00:18:03,010
suppose Q has
orthonormal columns.
286
00:18:03,010 --> 00:18:05,150
I'm using the letter
Q to mean this,
287
00:18:05,150 --> 00:18:07,130
I'll write it this
one more time,
288
00:18:07,130 --> 00:18:12,120
but I always mean
when I write a Q,
289
00:18:12,120 --> 00:18:14,910
I always mean that it
has orthonormal columns.
290
00:18:14,910 --> 00:18:25,910
So suppose I want to project
onto its column space.
291
00:18:31,590 --> 00:18:33,170
So what's the projection matrix?
292
00:18:36,480 --> 00:18:40,760
What's the projection matrix is
I project onto a column space?
293
00:18:40,760 --> 00:18:46,310
OK, that gives me a chance to
review the projection section,
294
00:18:46,310 --> 00:18:50,770
including that big formula,
which used to be --
295
00:18:50,770 --> 00:18:53,580
those four As in a
row, but now it's
296
00:18:53,580 --> 00:18:57,350
got Qs, because I'm projecting
onto the column space of Q,
297
00:18:57,350 --> 00:18:58,780
so do you remember what it was?
298
00:18:58,780 --> 00:19:05,700
It's Q Q transpose Q
inverse Q transpose.
299
00:19:08,500 --> 00:19:12,390
That's my four Qs in a row.
300
00:19:12,390 --> 00:19:13,690
But what's good here?
301
00:19:16,490 --> 00:19:21,040
What -- what makes this formula
nice if I'm projecting onto
302
00:19:21,040 --> 00:19:24,660
a column space when I have
orthonormal basis for that
303
00:19:24,660 --> 00:19:25,440
space?
304
00:19:25,440 --> 00:19:29,110
What makes it nice is
this is the identity.
305
00:19:29,110 --> 00:19:31,360
I don't have to
do any inversion.
306
00:19:31,360 --> 00:19:33,190
I just get Q Q transpose.
307
00:19:40,170 --> 00:19:44,250
So Q Q transpose is
a projection matrix.
308
00:19:44,250 --> 00:19:45,590
Oh, I can't help --
309
00:19:45,590 --> 00:19:47,970
I can't resist just
checking the properties,
310
00:19:47,970 --> 00:19:52,640
what are the properties
of a projection matrix?
311
00:19:52,640 --> 00:19:57,280
There are two properties to
know for any projection matrix.
312
00:19:57,280 --> 00:20:00,560
And I'm saying that this
is the right projection
313
00:20:00,560 --> 00:20:04,500
matrix when we've got
this orthonormal basis
314
00:20:04,500 --> 00:20:06,980
in the columns.
315
00:20:06,980 --> 00:20:07,540
OK.
316
00:20:07,540 --> 00:20:10,650
So there's the
projection matrix.
317
00:20:10,650 --> 00:20:13,670
Suppose the matrix is square.
318
00:20:13,670 --> 00:20:17,090
First just tell me
first this extreme case.
319
00:20:17,090 --> 00:20:22,190
If my matrix is square and it's
got these orthonormal columns,
320
00:20:22,190 --> 00:20:26,170
then what's the column space?
321
00:20:26,170 --> 00:20:31,270
If I have a square matrix and
I have independent columns,
322
00:20:31,270 --> 00:20:34,950
and even orthonormal columns,
then the column space
323
00:20:34,950 --> 00:20:37,170
is the whole space, right?
324
00:20:37,170 --> 00:20:41,250
And what's the projection
matrix onto the whole space?
325
00:20:41,250 --> 00:20:43,790
The identity matrix.
326
00:20:43,790 --> 00:20:45,360
If I'm projecting
in the whole space,
327
00:20:45,360 --> 00:20:49,690
every vector B is right
where it's supposed to be
328
00:20:49,690 --> 00:20:52,810
and I don't have to
move it by projection.
329
00:20:52,810 --> 00:20:57,800
So this would be --
330
00:20:57,800 --> 00:21:02,390
I'll put in parentheses
this is I if Q is square.
331
00:21:07,890 --> 00:21:10,360
Well that we said that already.
332
00:21:10,360 --> 00:21:14,990
If Q is square, that's the case
where Q transpose is Q inverse,
333
00:21:14,990 --> 00:21:18,140
we can put it on the right,
we can put it on the left,
334
00:21:18,140 --> 00:21:23,000
we always get the identity
matrix, if it's square.
335
00:21:23,000 --> 00:21:29,070
But if it's not a square
matrix then it's not --
336
00:21:29,070 --> 00:21:32,510
we don't get the
identity matrix.
337
00:21:32,510 --> 00:21:37,710
We have Q Q transpose,
and just again
338
00:21:37,710 --> 00:21:41,050
what are those two properties
of a projection matrix?
339
00:21:41,050 --> 00:21:44,400
First of all, it's symmetric.
340
00:21:44,400 --> 00:21:48,870
OK, no problem, that's
certainly a symmetric So what's
341
00:21:48,870 --> 00:21:50,860
that second property
of a projection?
342
00:21:50,860 --> 00:21:51,360
matrix.
343
00:21:51,360 --> 00:21:55,010
That if you project and project
again you don't move the second
344
00:21:55,010 --> 00:21:55,760
time.
345
00:21:55,760 --> 00:21:58,220
So the other property
of a projection matrix
346
00:21:58,220 --> 00:22:04,070
should be that Q
Q transpose twice
347
00:22:04,070 --> 00:22:09,320
should be the same as
Q Q transpose once.
348
00:22:09,320 --> 00:22:11,550
That's projection matrices.
349
00:22:11,550 --> 00:22:14,170
And that property
better fall out
350
00:22:14,170 --> 00:22:18,540
right away because
from the fact we
351
00:22:18,540 --> 00:22:24,060
know about orthonormal matrices,
Q transpose Q is I. OK,
352
00:22:24,060 --> 00:22:25,010
you see it.
353
00:22:25,010 --> 00:22:30,320
In the middle here is sitting
Q Q t- Q transpose Q, sorry,
354
00:22:30,320 --> 00:22:34,030
that's what I meant to
say, Q transpose Q is I.
355
00:22:34,030 --> 00:22:37,000
So that's sitting right in
the middle, that cancels out,
356
00:22:37,000 --> 00:22:40,760
to give the identity, we're
left with one Q Q transpose,
357
00:22:40,760 --> 00:22:43,480
and we're all set.
358
00:22:43,480 --> 00:22:44,180
OK.
359
00:22:44,180 --> 00:22:48,840
So this is the
projection matrix --
360
00:22:48,840 --> 00:22:54,670
all the equation -- all the
messy equations of this chapter
361
00:22:54,670 --> 00:22:59,370
become trivial
when our matrix --
362
00:22:59,370 --> 00:23:02,540
when we have this
orthonormal basis.
363
00:23:02,540 --> 00:23:04,630
I mean what do I mean
by all the equations?
364
00:23:04,630 --> 00:23:06,280
Well, the most
important equation
365
00:23:06,280 --> 00:23:10,680
was the normal equation, do
you remember old A transpose
366
00:23:10,680 --> 00:23:15,620
A x hat equals A transpose b?
367
00:23:15,620 --> 00:23:22,200
But now -- now A is Q.
368
00:23:22,200 --> 00:23:28,140
Now I'm thinking I have
Q transpose Q X hat
369
00:23:28,140 --> 00:23:32,000
equals Q transpose b.
370
00:23:32,000 --> 00:23:33,280
And what's good about that?
371
00:23:37,220 --> 00:23:42,950
What's good is that matrix on
the left side is the identity.
372
00:23:42,950 --> 00:23:46,110
The matrix on the left is
the identity, Q transpose Q,
373
00:23:46,110 --> 00:23:49,510
normally it isn't, normally it's
that matrix of inner products
374
00:23:49,510 --> 00:23:53,870
and you've to compute all those
dopey inner products and --
375
00:23:53,870 --> 00:23:56,030
and -- and solve the system.
376
00:23:56,030 --> 00:23:59,950
Here the inner products
are all one or zero.
377
00:23:59,950 --> 00:24:01,540
This is the identity matrix.
378
00:24:01,540 --> 00:24:03,280
It's gone.
379
00:24:03,280 --> 00:24:05,910
And there's the answer.
380
00:24:05,910 --> 00:24:09,140
There's no inversion involved.
381
00:24:09,140 --> 00:24:15,730
Each component of
x is a Q times b.
382
00:24:15,730 --> 00:24:21,240
What that equation is saying
is that the i-th component is
383
00:24:21,240 --> 00:24:26,420
the i-th basis vector times b.
384
00:24:26,420 --> 00:24:34,760
That's -- probably the most
important formula in some major
385
00:24:34,760 --> 00:24:40,720
parts of mathematics, that
if we have orthonormal basis,
386
00:24:40,720 --> 00:24:47,660
then the component in the --
in the i-th, along the i-th --
387
00:24:47,660 --> 00:24:54,630
the projection on the i-th basis
vector is just qi transpose b.
388
00:24:54,630 --> 00:25:00,580
That number x that we look
for is just a dot product.
389
00:25:00,580 --> 00:25:01,080
OK.
390
00:25:04,050 --> 00:25:10,040
OK, so I'm ready now for
the sort of like second half
391
00:25:10,040 --> 00:25:11,530
of the lecture.
392
00:25:11,530 --> 00:25:16,320
Where we don't start with
an orthogonal matrix,
393
00:25:16,320 --> 00:25:18,910
orthonormal vectors.
394
00:25:18,910 --> 00:25:21,610
We just start with
independent vectors
395
00:25:21,610 --> 00:25:25,370
and we want to make
them orthonormal.
396
00:25:25,370 --> 00:25:27,860
So I'm going to --
can I do that now?
397
00:25:27,860 --> 00:25:29,940
Now here comes Graham-Schmidt.
398
00:25:29,940 --> 00:25:31,490
So -- Graham-Schmidt.
399
00:25:39,460 --> 00:25:43,990
So this is a calculation,
I won't say --
400
00:25:43,990 --> 00:25:53,190
I can't quite say it's like
elimination, because it's
401
00:25:53,190 --> 00:25:56,630
different, our goal
isn't triangular anymore.
402
00:25:56,630 --> 00:26:00,780
With elimination our goal was
make the matrix triangular.
403
00:26:00,780 --> 00:26:04,200
Now our goal is make
the matrix orthogonal.
404
00:26:04,200 --> 00:26:07,580
Make those columns orthonormal.
405
00:26:07,580 --> 00:26:10,240
So let me start
with two columns.
406
00:26:10,240 --> 00:26:13,000
So I start with vectors a and b.
407
00:26:16,960 --> 00:26:20,430
And they're just like --
here, let me draw them.
408
00:26:20,430 --> 00:26:22,780
Here's a.
409
00:26:22,780 --> 00:26:23,400
Here's b.
410
00:26:26,370 --> 00:26:27,580
For example.
411
00:26:27,580 --> 00:26:29,700
A isn't specially
horizontal, wasn't
412
00:26:29,700 --> 00:26:34,150
meant to be, just a is
one vector, b is another.
413
00:26:34,150 --> 00:26:38,040
I want to produce
those two vectors,
414
00:26:38,040 --> 00:26:40,760
they might be in
twelve-dimensional space,
415
00:26:40,760 --> 00:26:43,940
or they might be in
two-dimensional space.
416
00:26:43,940 --> 00:26:46,300
They're independent, anyway.
417
00:26:46,300 --> 00:26:49,500
So I better be sure I say that.
418
00:26:49,500 --> 00:26:51,330
I start with
independent vectors.
419
00:26:54,860 --> 00:26:58,500
And I want to produce
out of that q 1 and q2,
420
00:26:58,500 --> 00:27:00,970
I want to produce
orthonormal vectors.
421
00:27:03,880 --> 00:27:09,730
And Graham and
Schmidt tell me how.
422
00:27:09,730 --> 00:27:10,260
OK.
423
00:27:10,260 --> 00:27:14,330
Well, actually you could tell me
how, we don't need -- frankly,
424
00:27:14,330 --> 00:27:17,800
I don't know -- there's
only one idea here,
425
00:27:17,800 --> 00:27:24,850
if Graham had the idea, I
don't know what Schmidt did.
426
00:27:24,850 --> 00:27:28,140
But OK.
427
00:27:28,140 --> 00:27:29,470
So you'll see it.
428
00:27:29,470 --> 00:27:31,900
We don't need either
of them, actually.
429
00:27:31,900 --> 00:27:33,630
OK, so what I going to do.
430
00:27:33,630 --> 00:27:36,630
I'll take that --
this first guy.
431
00:27:36,630 --> 00:27:37,970
OK.
432
00:27:37,970 --> 00:27:39,790
Well, he's fine.
433
00:27:43,480 --> 00:27:46,620
That direction is fine except --
434
00:27:46,620 --> 00:27:50,300
yeah, I'll say OK, I'll
settle for that direction.
435
00:27:50,300 --> 00:27:51,410
So I'm going to --
436
00:27:51,410 --> 00:27:53,630
I'm going to get, so
what I going to --
437
00:27:53,630 --> 00:27:59,370
my goal is I'm going to
get orthogonal vectors
438
00:27:59,370 --> 00:28:02,780
and I'll call those
capital A and B.
439
00:28:02,780 --> 00:28:07,590
So that's the key step is
to get from any two vectors
440
00:28:07,590 --> 00:28:09,430
to two orthogonal vectors.
441
00:28:09,430 --> 00:28:14,530
And then at the end, no problem,
I'll get orthonormal vectors,
442
00:28:14,530 --> 00:28:20,630
how will -- what will those
will be my qs, q1 and q2,
443
00:28:20,630 --> 00:28:21,450
and what will they
444
00:28:21,450 --> 00:28:21,950
be?
445
00:28:25,980 --> 00:28:30,270
Once I've got A and B
orthogonal, well, look,
446
00:28:30,270 --> 00:28:35,390
it's no big deal -- maybe
that's what Schmidt did, he,
447
00:28:35,390 --> 00:28:38,840
brilliant Schmidt, thought
OK, divide by the length,
448
00:28:38,840 --> 00:28:40,310
all right.
449
00:28:40,310 --> 00:28:42,820
That's Schmidt's contribution.
450
00:28:45,640 --> 00:28:46,140
OK.
451
00:28:51,250 --> 00:28:55,620
But Graham had a little
more thinking to do, right?
452
00:28:55,620 --> 00:28:58,780
We haven't done Graham's part.
453
00:28:58,780 --> 00:29:03,050
This part except OK,
I'm happy with A,
454
00:29:03,050 --> 00:29:07,170
A can be A. That first
direction is fine.
455
00:29:07,170 --> 00:29:09,620
Why should -- no
complaint about that.
456
00:29:09,620 --> 00:29:13,620
The trouble is the second
direction is not fine.
457
00:29:13,620 --> 00:29:18,170
Because it's not
orthogonal to the first.
458
00:29:18,170 --> 00:29:23,710
I'm looking for a vector
that's -- starts with B,
459
00:29:23,710 --> 00:29:29,550
but makes it orthogonal to A.
460
00:29:29,550 --> 00:29:30,980
What's the vector?
461
00:29:30,980 --> 00:29:32,870
How do I do that?
462
00:29:32,870 --> 00:29:35,960
How do I produce
from this vector
463
00:29:35,960 --> 00:29:41,820
a piece that's
orthogonal to this one?
464
00:29:41,820 --> 00:29:44,900
And the -- remember these
vectors might be in two
465
00:29:44,900 --> 00:29:48,570
dimensions or they might
be in twelve dimensions.
466
00:29:48,570 --> 00:29:51,210
I'm just looking for the idea.
467
00:29:51,210 --> 00:29:53,840
So what's the idea?
468
00:29:53,840 --> 00:29:57,110
Where did we have orthogonal --
469
00:29:57,110 --> 00:30:01,440
a vector showing up that
was orthogonal to this guy?
470
00:30:01,440 --> 00:30:03,990
Well, that was the
first basic calculation
471
00:30:03,990 --> 00:30:05,780
of the whole chapter.
472
00:30:05,780 --> 00:30:10,850
We -- we did a projection and
the projection gave us this
473
00:30:10,850 --> 00:30:15,890
part, which was the
part in the A direction.
474
00:30:15,890 --> 00:30:19,500
Now, the part we want is
the other part, the e part.
475
00:30:19,500 --> 00:30:21,050
This part.
476
00:30:21,050 --> 00:30:23,790
This is going to be our --
477
00:30:23,790 --> 00:30:25,450
that guy is that guy.
478
00:30:25,450 --> 00:30:31,110
This is our vector B. That gives
us that ninety-degree angle.
479
00:30:31,110 --> 00:30:33,240
So B is you could say --
480
00:30:33,240 --> 00:30:35,730
B is really what we
previously called
481
00:30:35,730 --> 00:30:37,140
e.
482
00:30:37,140 --> 00:30:41,060
The error vector.
483
00:30:41,060 --> 00:30:42,950
And what is it?
484
00:30:42,950 --> 00:30:45,950
I mean what do I
-- what is B here?
485
00:30:45,950 --> 00:30:47,690
A is A, no problem.
486
00:30:47,690 --> 00:30:52,410
B is --
487
00:30:52,410 --> 00:30:54,120
OK, what's this error piece?
488
00:30:54,120 --> 00:30:56,530
Do you remember?
489
00:30:56,530 --> 00:31:04,150
It's I start with the original
B and I take away what?
490
00:31:04,150 --> 00:31:08,670
Its projection, this P.
This -- the vector B,
491
00:31:08,670 --> 00:31:12,160
this error vector, is the
original vector removing
492
00:31:12,160 --> 00:31:12,890
the projection.
493
00:31:12,890 --> 00:31:15,590
So instead of wanting
the projection,
494
00:31:15,590 --> 00:31:20,260
now that's what I
want to throw away.
495
00:31:20,260 --> 00:31:22,780
I want to get the part
that's perpendicular.
496
00:31:22,780 --> 00:31:24,720
And there will be a
perpendicular part,
497
00:31:24,720 --> 00:31:25,450
it won't be zero.
498
00:31:28,890 --> 00:31:32,880
Because these vectors
were independent, so B --
499
00:31:32,880 --> 00:31:34,880
if B was along the
direction of A,
500
00:31:34,880 --> 00:31:37,830
then if the original B and A
were in the same direction,
501
00:31:37,830 --> 00:31:38,760
then I'm --
502
00:31:38,760 --> 00:31:40,500
I've only got one direction.
503
00:31:40,500 --> 00:31:43,440
But here they're in two
independent directions
504
00:31:43,440 --> 00:31:46,190
and all I'm doing
is getting that guy.
505
00:31:46,190 --> 00:31:49,550
So what's its formula?
506
00:31:49,550 --> 00:31:53,510
What's the formula
for that if --
507
00:31:53,510 --> 00:31:55,540
I want to subtract
the projection,
508
00:31:55,540 --> 00:31:57,450
so do you remember
the projection?
509
00:31:57,450 --> 00:32:03,490
It's some multiple of A
and what's that multiple?
510
00:32:03,490 --> 00:32:07,030
It's -- it's that thing we
called x in the very very first
511
00:32:07,030 --> 00:32:10,150
lecture on this chapter.
512
00:32:10,150 --> 00:32:16,540
There's an A transpose
A in the bottom
513
00:32:16,540 --> 00:32:23,990
and there's an A transpose
B, isn't that it?
514
00:32:29,780 --> 00:32:32,240
I think that's Graham's formula.
515
00:32:32,240 --> 00:32:33,300
Or Graham-Schmidt.
516
00:32:33,300 --> 00:32:34,600
No, that's Graham.
517
00:32:34,600 --> 00:32:38,120
Schmidt has got to divide the
whole thing by the length,
518
00:32:38,120 --> 00:32:39,660
so he --
519
00:32:39,660 --> 00:32:43,280
his formula makes a mess which
I'm not willing to write down.
520
00:32:43,280 --> 00:32:48,070
So let's just see that
what I saying here?
521
00:32:48,070 --> 00:32:51,670
I'm saying that this vector
is perpendicular to A.
522
00:32:51,670 --> 00:32:53,260
That these are orthogonal.
523
00:32:53,260 --> 00:32:56,710
A is perpendicular to B.
524
00:32:56,710 --> 00:32:58,490
Can you check that?
525
00:32:58,490 --> 00:33:01,430
How do you see that
yes, of course, we --
526
00:33:01,430 --> 00:33:04,160
our picture is telling
us, yes, we did it right.
527
00:33:04,160 --> 00:33:08,720
How would I check that this
matrix is perpendicular to A?
528
00:33:11,390 --> 00:33:16,740
I would multiply by A transpose
and I better get zero, right?
529
00:33:16,740 --> 00:33:18,410
I should check that.
530
00:33:18,410 --> 00:33:22,930
A transpose B should
come out zero.
531
00:33:22,930 --> 00:33:26,920
So this is A transpose times
-- now what did we say B was?
532
00:33:26,920 --> 00:33:30,370
We start with the original
B, and we take away
533
00:33:30,370 --> 00:33:38,090
this projection, and that
should come out zero.
534
00:33:38,090 --> 00:33:42,950
Well, here we get an
A transpose B minus --
535
00:33:42,950 --> 00:33:46,050
and here is another A
transpose B, and the --
536
00:33:46,050 --> 00:33:50,120
and it's an A transpose A
over A transpose A, a one,
537
00:33:50,120 --> 00:33:53,840
those cancel, and
we do get zero.
538
00:33:53,840 --> 00:33:54,340
Right.
539
00:33:58,360 --> 00:34:05,940
Now I guess I can
do numbers in there.
540
00:34:05,940 --> 00:34:09,420
But I have to take
a third vector
541
00:34:09,420 --> 00:34:13,330
to be sure we've got
this system down.
542
00:34:13,330 --> 00:34:20,250
So now I have to say if I have
independent vectors A, B and C,
543
00:34:20,250 --> 00:34:25,820
I'm looking for orthogonal
vectors A, B and capital C,
544
00:34:25,820 --> 00:34:29,170
and then of course the
third guy will just
545
00:34:29,170 --> 00:34:32,969
be C over its length,
the unit vector.
546
00:34:35,860 --> 00:34:39,980
So this is now the problem.
547
00:34:39,980 --> 00:34:42,929
I got B here.
548
00:34:42,929 --> 00:34:45,889
I got A very easily.
549
00:34:45,889 --> 00:34:52,610
And now -- if you see the idea,
we could figure out a formula
550
00:34:52,610 --> 00:35:01,390
for C. So now that -- so this
is like a typical homework quiz
551
00:35:01,390 --> 00:35:02,150
problem.
552
00:35:02,150 --> 00:35:07,260
I give you two vectors, you do
this, I give you three vectors,
553
00:35:07,260 --> 00:35:10,460
and you have to make
them orthonormal.
554
00:35:10,460 --> 00:35:13,520
So you do this again,
the first vector's fine,
555
00:35:13,520 --> 00:35:17,160
the second vector is
perpendicular to the first,
556
00:35:17,160 --> 00:35:19,720
and now I need a
third vector that's
557
00:35:19,720 --> 00:35:23,240
perpendicular to the first
one and the second one.
558
00:35:23,240 --> 00:35:25,560
Right?
559
00:35:25,560 --> 00:35:29,660
Tthis is the end of a -- the
lecture is to find this guy.
560
00:35:29,660 --> 00:35:33,980
Find this vector -- this vector
C, that's perpendicular we n-
561
00:35:33,980 --> 00:35:39,410
at this point we know A and B.
562
00:35:39,410 --> 00:35:44,540
But C, the little c that
we're given, is off in some --
563
00:35:44,540 --> 00:35:48,030
it's got to come out of the
blackboard to be independent,
564
00:35:48,030 --> 00:35:52,440
so -- so can I sort of draw
off -- off comes a c somewhere.
565
00:35:52,440 --> 00:35:54,240
I don't know, where I
going to put the darn
566
00:35:54,240 --> 00:35:55,050
thing?
567
00:35:55,050 --> 00:35:59,450
Maybe I'll put it
off, oh, I don't know,
568
00:35:59,450 --> 00:36:02,160
like that somehow, C, little c.
569
00:36:05,610 --> 00:36:08,930
And I already know that
perpendicular direction,
570
00:36:08,930 --> 00:36:10,750
that one and that one.
571
00:36:10,750 --> 00:36:13,800
So now what's the idea?
572
00:36:13,800 --> 00:36:17,080
Give me the Graham-Schmidt
formula for C.
573
00:36:17,080 --> 00:36:20,700
What is this C here?
574
00:36:20,700 --> 00:36:21,520
Equals what?
575
00:36:27,340 --> 00:36:28,160
What I going to do?
576
00:36:28,160 --> 00:36:31,300
I'll start with the given one.
577
00:36:31,300 --> 00:36:32,730
As before.
578
00:36:32,730 --> 00:36:33,280
Right?
579
00:36:33,280 --> 00:36:36,410
I start with the
vector I'm given.
580
00:36:36,410 --> 00:36:38,400
And what do I do with it?
581
00:36:38,400 --> 00:36:42,060
I want to remove out of
it, I want to subtract off,
582
00:36:42,060 --> 00:36:46,510
so I'll put a minus sign
in, I want to subtract off
583
00:36:46,510 --> 00:36:53,010
its components in the A, capital
A and capital B directions.
584
00:36:53,010 --> 00:36:55,850
I just want to get
those out of there.
585
00:36:55,850 --> 00:36:57,290
Well, I know how to do that.
586
00:36:57,290 --> 00:36:58,710
I did it with B.
587
00:36:58,710 --> 00:37:02,410
So I'll just -- so
let me take away --
588
00:37:02,410 --> 00:37:03,440
what if I do this?
589
00:37:07,960 --> 00:37:08,850
What have I done?
590
00:37:12,060 --> 00:37:16,520
I've got little c and what
have I subtracted from it?
591
00:37:16,520 --> 00:37:22,280
Its component, its projection
if you like, in the A direction.
592
00:37:25,010 --> 00:37:30,820
And now I've got to subtract
off its component B transpose
593
00:37:30,820 --> 00:37:35,200
C over B transpose B,
that multiple of B,
594
00:37:35,200 --> 00:37:37,570
is its component
in the B direction.
595
00:37:40,100 --> 00:37:47,480
And that gives me the vector
capital C that if anything is
596
00:37:47,480 --> 00:37:48,160
--
597
00:37:48,160 --> 00:37:54,700
if there's any justice, this
C should be perpendicular to A
598
00:37:54,700 --> 00:37:58,750
and it should be
perpendicular to B.
599
00:37:58,750 --> 00:38:02,660
And the only thing it's --
hasn't got is unit vector,
600
00:38:02,660 --> 00:38:05,830
so we divide by its
length to get that too.
601
00:38:08,660 --> 00:38:10,390
OK.
602
00:38:10,390 --> 00:38:14,940
Let me do an example.
603
00:38:14,940 --> 00:38:16,410
Can I --
604
00:38:16,410 --> 00:38:20,990
I'll make my life easy,
I'll just take two vectors.
605
00:38:20,990 --> 00:38:23,880
So let me do a
numerical example.
606
00:38:23,880 --> 00:38:26,480
If I'll give you
two vectors, you
607
00:38:26,480 --> 00:38:31,220
give me back the Graham-Schmidt
orthonormal basis,
608
00:38:31,220 --> 00:38:35,990
and we'll see how to
express that in matrix form.
609
00:38:35,990 --> 00:38:36,530
OK.
610
00:38:36,530 --> 00:38:41,080
So let me give you
the two vectors.
611
00:38:41,080 --> 00:38:46,420
So I'll take the vector A
equals let's say one, one, one,
612
00:38:46,420 --> 00:38:47,910
why not?
613
00:38:47,910 --> 00:38:55,810
And B equals let's say
one, zero, two, OK?
614
00:39:02,400 --> 00:39:05,300
I didn't want to cheat
and make them orthogonal
615
00:39:05,300 --> 00:39:07,330
in the first place because
then Graham-Schmidt
616
00:39:07,330 --> 00:39:08,430
wouldn't be needed.
617
00:39:08,430 --> 00:39:08,930
OK.
618
00:39:08,930 --> 00:39:10,760
So those are not orthogonal.
619
00:39:10,760 --> 00:39:12,330
So what is capital A?
620
00:39:12,330 --> 00:39:14,280
Well that's the same as big A.
621
00:39:14,280 --> 00:39:15,350
That was fine.
622
00:39:15,350 --> 00:39:18,590
What's B?
623
00:39:18,590 --> 00:39:21,600
So B is this b --
is the original B,
624
00:39:21,600 --> 00:39:29,680
and then I subtract off
some multiple of the A.
625
00:39:29,680 --> 00:39:30,900
And what's the multiple?
626
00:39:33,810 --> 00:39:36,520
What goes in here?
627
00:39:36,520 --> 00:39:41,240
B -- here's the A -- this is
the -- this is the little b,
628
00:39:41,240 --> 00:39:45,430
this is the big A, also
the little a, and I want
629
00:39:45,430 --> 00:39:48,680
to multiply it by that
right -- that right ratio,
630
00:39:48,680 --> 00:39:53,480
which has A transpose
A, here's my ratio.
631
00:39:53,480 --> 00:39:57,100
I'm just doing this.
632
00:39:57,100 --> 00:40:00,610
So it's A transpose B,
what is A transpose B,
633
00:40:00,610 --> 00:40:02,790
it looks like three.
634
00:40:02,790 --> 00:40:06,780
And what is A -- oh, my --
635
00:40:06,780 --> 00:40:08,470
what's A transpose A?
636
00:40:08,470 --> 00:40:08,970
Three.
637
00:40:11,540 --> 00:40:12,690
I'm sorry.
638
00:40:12,690 --> 00:40:14,990
I didn't know that
was going to happen.
639
00:40:14,990 --> 00:40:15,490
OK.
640
00:40:15,490 --> 00:40:16,440
But it happened.
641
00:40:16,440 --> 00:40:18,750
Why should we knock it?
642
00:40:18,750 --> 00:40:19,610
OK.
643
00:40:19,610 --> 00:40:21,430
So do you see it all right?
644
00:40:21,430 --> 00:40:25,040
That's A transpose B,
there's A transpose A, that's
645
00:40:25,040 --> 00:40:28,640
the fraction, so
I take this away,
646
00:40:28,640 --> 00:40:33,640
and I get one take away one
is a zero, zero minus this one
647
00:40:33,640 --> 00:40:39,180
is a minus one, and two
minus the one is a one.
648
00:40:39,180 --> 00:40:39,930
OK.
649
00:40:39,930 --> 00:40:42,330
And what's this vector
that we finally found?
650
00:40:42,330 --> 00:40:47,020
This is B.
651
00:40:47,020 --> 00:40:48,580
And how do I know it's right?
652
00:40:51,930 --> 00:40:54,690
How do I know I've
got a vector I want?
653
00:40:54,690 --> 00:40:57,700
I check that B is
perpendicular to --
654
00:40:57,700 --> 00:40:59,980
that A and B are perpendicular.
655
00:40:59,980 --> 00:41:02,280
That A is perpendicular to B.
656
00:41:02,280 --> 00:41:03,190
Just look at that.
657
00:41:03,190 --> 00:41:06,550
That one -- the dot product
of that with that is zero.
658
00:41:06,550 --> 00:41:07,180
OK.
659
00:41:07,180 --> 00:41:10,850
So now what is my q1 and q2?
660
00:41:14,720 --> 00:41:17,700
Why don't I put
them in a matrix?
661
00:41:17,700 --> 00:41:18,480
Of course.
662
00:41:18,480 --> 00:41:20,970
Since I'm always putting
these -- so the Q,
663
00:41:20,970 --> 00:41:24,260
I'll put the q1 and
the q2 in a matrix.
664
00:41:24,260 --> 00:41:25,720
And what are they?
665
00:41:29,270 --> 00:41:32,700
Now when I'm writing
q-s I'm supposed
666
00:41:32,700 --> 00:41:34,200
to make things normalized.
667
00:41:34,200 --> 00:41:36,450
I'm supposed to make
things unit vectors.
668
00:41:36,450 --> 00:41:39,740
So I'm going to take that A
but I'm going to divide it
669
00:41:39,740 --> 00:41:41,650
by square root of three.
670
00:41:46,920 --> 00:41:48,780
And I'm going to
take this B but I'm
671
00:41:48,780 --> 00:41:53,370
going to divide it
by square root of two
672
00:41:53,370 --> 00:41:57,720
to make it a unit vector,
and there is my matrix.
673
00:42:01,000 --> 00:42:05,430
That's my matrix with
orthonormal columns coming from
674
00:42:05,430 --> 00:42:08,790
Graham-Schmidt and
it sort of it --
675
00:42:08,790 --> 00:42:14,740
it came from the original
one, one, one, one, zero, two,
676
00:42:14,740 --> 00:42:15,240
right?
677
00:42:15,240 --> 00:42:16,620
That was my original guys.
678
00:42:20,880 --> 00:42:23,200
These were the two
I started with.
679
00:42:23,200 --> 00:42:25,950
These are the two that
I'm happy to end with.
680
00:42:25,950 --> 00:42:30,110
Because those are orthonormal.
681
00:42:30,110 --> 00:42:33,490
So that's what
Graham-Schmidt did.
682
00:42:33,490 --> 00:42:38,050
It -- well, tell me about
the column spaces of these
683
00:42:38,050 --> 00:42:40,260
matrices.
684
00:42:40,260 --> 00:42:44,180
How is the column space of Q
related to the column space of
685
00:42:44,180 --> 00:42:44,900
A?
686
00:42:44,900 --> 00:42:47,150
So I'm always asking
you things like this,
687
00:42:47,150 --> 00:42:49,970
and that makes you think,
OK, the column space
688
00:42:49,970 --> 00:42:54,550
is all combinations of the
columns, it's that plane,
689
00:42:54,550 --> 00:42:55,540
right?
690
00:42:55,540 --> 00:42:58,770
I've got two vectors in
three-dimensional space,
691
00:42:58,770 --> 00:43:03,630
their column space is a plane,
the column space of this matrix
692
00:43:03,630 --> 00:43:08,200
is a plane, what's the
relation between the planes?
693
00:43:08,200 --> 00:43:09,740
Between the two column spaces?
694
00:43:12,750 --> 00:43:15,190
They're one and the same, right?
695
00:43:15,190 --> 00:43:17,760
It's the same column space.
696
00:43:17,760 --> 00:43:23,580
All I'm taking is here this
B thing that I computed,
697
00:43:23,580 --> 00:43:30,120
this B thing that I computed
is a combination of B and A,
698
00:43:30,120 --> 00:43:34,630
and A was little A, so
I'm always working here
699
00:43:34,630 --> 00:43:36,920
with this in the same space.
700
00:43:36,920 --> 00:43:42,450
I'm just like getting
ninety-degree angles in there.
701
00:43:42,450 --> 00:43:47,450
Where my original column space
had a perfectly good basis,
702
00:43:47,450 --> 00:43:50,470
but it wasn't as
good as this basis,
703
00:43:50,470 --> 00:43:53,610
because it wasn't orthonormal.
704
00:43:53,610 --> 00:43:59,560
Now this one is orthonormal,
and I have a basis then that --
705
00:43:59,560 --> 00:44:03,050
so now projections, all the
calculations I would ever want
706
00:44:03,050 --> 00:44:09,570
to do are -- are a cinch
with this orthonormal basis.
707
00:44:09,570 --> 00:44:12,710
One final point.
708
00:44:12,710 --> 00:44:14,480
One final point in this chapter.
709
00:44:17,200 --> 00:44:21,240
And it's -- just
like elimination.
710
00:44:21,240 --> 00:44:23,530
We learned how to
do elimination,
711
00:44:23,530 --> 00:44:26,160
we know all the
steps, we can do it.
712
00:44:26,160 --> 00:44:34,720
But then I came back to it and
said look at it as a matrix
713
00:44:34,720 --> 00:44:40,140
in matrix language and
elimination gave me --
714
00:44:40,140 --> 00:44:42,480
what was elimination
in matrix language?
715
00:44:42,480 --> 00:44:44,000
I'll just put it up there.
716
00:44:44,000 --> 00:44:46,240
A was LU.
717
00:44:46,240 --> 00:44:49,720
That was matrix,
that was elimination.
718
00:44:49,720 --> 00:44:53,180
Now, I want to do the
same for Graham-Schmidt.
719
00:44:53,180 --> 00:44:56,200
Everybody who works
in linear algebra
720
00:44:56,200 --> 00:44:58,530
isn't going to write
out the columns
721
00:44:58,530 --> 00:45:01,220
are orthogonal, or orthonormal.
722
00:45:01,220 --> 00:45:04,530
And isn't going to write
out these formulas.
723
00:45:04,530 --> 00:45:08,820
They're going to write out the
connection between the matrix A
724
00:45:08,820 --> 00:45:11,320
and the matrix Q.
725
00:45:11,320 --> 00:45:14,230
And the two matrices have
the same column space,
726
00:45:14,230 --> 00:45:17,720
but there's some -- some
matrix is taking the --
727
00:45:17,720 --> 00:45:25,110
and I'm going to call it R, so
A equals QR is the magic formula
728
00:45:25,110 --> 00:45:25,610
here.
729
00:45:28,200 --> 00:45:30,260
It's the expression
of Graham-Schmidt.
730
00:45:32,860 --> 00:45:38,430
And I'll -- let me
just capture that.
731
00:45:38,430 --> 00:45:42,240
So that's the -- my final
step then is A equal QR.
732
00:45:42,240 --> 00:45:44,310
Maybe I can squeeze it in here.
733
00:45:47,300 --> 00:45:50,960
So A has columns,
let's say a1 and a2.
734
00:45:55,420 --> 00:45:59,150
Let me suppose n is
two, just two vectors.
735
00:45:59,150 --> 00:46:00,290
OK.
736
00:46:00,290 --> 00:46:06,300
So that's some
combination of q1 and q2.
737
00:46:06,300 --> 00:46:13,170
And times some matrix R.
738
00:46:13,170 --> 00:46:16,330
They have the same column space.
739
00:46:16,330 --> 00:46:20,420
This is just -- this matrix just
includes in it whatever these
740
00:46:20,420 --> 00:46:23,430
numbers like three over three
and one over square root
741
00:46:23,430 --> 00:46:25,360
of three and one over
square root of two,
742
00:46:25,360 --> 00:46:28,400
probably that's what it's got.
743
00:46:28,400 --> 00:46:31,140
One over square root of three,
one over square root of two,
744
00:46:31,140 --> 00:46:34,390
something there, but actually
it's got a zero there.
745
00:46:37,690 --> 00:46:45,260
So the main point about
this A equal QR is this R
746
00:46:45,260 --> 00:46:48,270
turns out to be
upper triangular.
747
00:46:48,270 --> 00:46:50,750
It turns out that this
zero is upper triangular.
748
00:46:53,510 --> 00:46:56,440
We could see why.
749
00:46:56,440 --> 00:47:00,640
Let me see, I can put in
general formulas for what these
750
00:47:00,640 --> 00:47:05,190
This I think in here should
be the inner product of a1
751
00:47:05,190 --> 00:47:05,740
with q1. are.
752
00:47:08,600 --> 00:47:12,160
And this one should be the --
753
00:47:12,160 --> 00:47:16,230
the inner product of a1 with q2.
754
00:47:16,230 --> 00:47:18,970
And that's what I
believe is zero.
755
00:47:21,710 --> 00:47:25,110
This will be something here,
and this will be something here
756
00:47:25,110 --> 00:47:33,960
with inner -- a1 transpose q2,
sorry a2 transpose q1 and a2
757
00:47:33,960 --> 00:47:35,060
transpose q2.
758
00:47:35,060 --> 00:47:37,130
But why is that guy zero?
759
00:47:40,100 --> 00:47:43,900
Why is a1 q2 zero?
760
00:47:43,900 --> 00:47:47,580
That's the key to this being
-- this R here being upper
761
00:47:47,580 --> 00:47:49,110
triangular.
762
00:47:49,110 --> 00:47:55,100
You know why a1q2 is
zero, because a1 --
763
00:47:55,100 --> 00:47:57,310
that was my --
764
00:47:57,310 --> 00:48:00,180
this was really a and b here.
765
00:48:00,180 --> 00:48:02,700
This was really a and b.
766
00:48:02,700 --> 00:48:05,780
So this is a transpose q2.
767
00:48:05,780 --> 00:48:09,380
And the whole point of
Graham-Schmidt was that we
768
00:48:09,380 --> 00:48:14,820
constructed these later q-s to
be perpendicular to the earlier
769
00:48:14,820 --> 00:48:19,030
vectors, to the earlier --
all the earlier vectors.
770
00:48:19,030 --> 00:48:21,090
So that's why we get
a triangular matrix.
771
00:48:23,800 --> 00:48:29,750
The -- result is
extremely satisfactory.
772
00:48:32,530 --> 00:48:37,450
That if I have a matrix
with independent columns,
773
00:48:37,450 --> 00:48:40,540
the Graham-Schmidt
produces a matrix
774
00:48:40,540 --> 00:48:44,780
with orthonormal columns, and
the connection between those
775
00:48:44,780 --> 00:48:48,230
is a triangular matrix.
776
00:48:48,230 --> 00:48:51,200
That last point, that the
connection is a triangular
777
00:48:51,200 --> 00:48:53,510
matrix, please look
in the book, you
778
00:48:53,510 --> 00:48:56,390
have to see that one more time.
779
00:48:56,390 --> 00:48:56,930
OK.
780
00:48:56,930 --> 00:48:59,170
Thanks, that's great.