1
00:00:06 --> 00:00:06
OK.
2
00:00:06 --> 00:00:14
Here's lecture sixteen and if
you remember I ended up the last
3
00:00:14 --> 00:00:21.5
lecture with this formula for
what I called a projection
4
00:00:21.5 --> 00:00:22
matrix.
5
00:00:22 --> 00:00:29
And maybe I could just recap
for a minute what is that magic
6
00:00:29 --> 00:00:31
formula doing?
7
00:00:31 --> 00:00:37
For example,
it's supposed to be --
8
00:00:37 --> 00:00:42
it's supposed to produce a
projection, if I multiply by a
9
00:00:42 --> 00:00:47
b, so I take P times b,
I'm supposed to project that
10
00:00:47 --> 00:00:52
vector b to the nearest point in
the column space.
11
00:00:52 --> 00:00:53
OK.
12
00:00:53 --> 00:00:58.7
Can I just -- one way to recap
is to take the two extreme
13
00:00:58.7 --> 00:00:59
cases.
14
00:00:59 --> 00:01:03
Suppose a vector b is in the
column space?
15
00:01:03 --> 00:01:09
Then what do I get when I apply
the projection P?
16
00:01:09 --> 00:01:13.76
So I'm projecting into the
column space but I'm starting
17
00:01:13.76 --> 00:01:17
with a vector in this case
that's already in the column
18
00:01:17 --> 00:01:21
space, so of course when I
project it I get B again,
19
00:01:21 --> 00:01:22
right.
20
00:01:22 --> 00:01:26
And I want to show you how that
comes out of this formula.
21
00:01:26 --> 00:01:28
Let me do the other extreme.
22
00:01:28 --> 00:01:32
Suppose that vector is
perpendicular to the column
23
00:01:32 --> 00:01:34
space.
24
00:01:34 --> 00:01:39
So imagine this column space as
a plane and imagine b as
25
00:01:39 --> 00:01:43
sticking straight up
perpendicular to it.
26
00:01:43 --> 00:01:48
What's the nearest point in the
column space to b in that case?
27
00:01:48 --> 00:01:54
So what's the projection onto
the plane, the nearest point in
28
00:01:54 --> 00:02:00.28
the plane, if the vector b that
I'm looking at is --
29
00:02:00.28 --> 00:02:04
got no component in the column
space, it's sticking completely
30
00:02:04 --> 00:02:08
-- ninety degrees with it,
then Pb should be zero,
31
00:02:08 --> 00:02:09
right.
32
00:02:09 --> 00:02:11
So those are the two extreme
cases.
33
00:02:11 --> 00:02:15.96
The average vector has a
component P in the column space
34
00:02:15.96 --> 00:02:20
and a component perpendicular to
it, and what the projection does
35
00:02:20 --> 00:02:25
is it kills this part and it
preserves this part.
36
00:02:25 --> 00:02:25
OK.
37
00:02:25 --> 00:02:29
Can we just see why that's
true?
38
00:02:29 --> 00:02:33
Just -- that formula ought to
work.
39
00:02:33 --> 00:02:36
So let me start with this one.
40
00:02:36 --> 00:02:43
What vectors are in the -- are
perpendicular to the column
41
00:02:43 --> 00:02:43
space?
42
00:02:43 --> 00:02:49.06
How do I see that I really get
zero?
43
00:02:49.06 --> 00:02:52
I have to think,
what does it mean for a vector
44
00:02:52 --> 00:02:56
b to be perpendicular to the
column space?
45
00:02:56 --> 00:03:00.98
So if it's perpendicular to all
the columns, then it's in some
46
00:03:00.98 --> 00:03:01
other space.
47
00:03:01 --> 00:03:06
We've got our four spaces so
the reason I do this is it's
48
00:03:06 --> 00:03:11
perfectly using what we know
about our four spaces.
49
00:03:11 --> 00:03:15
What vectors are perpendicular
to the column space?
50
00:03:15 --> 00:03:19
Those are the guys in the null
space of A transpose,
51
00:03:19 --> 00:03:19
right?
52
00:03:19 --> 00:03:22
That's the first section of
this chapter,
53
00:03:22 --> 00:03:25
that's the key geometry of
these spaces.
54
00:03:25 --> 00:03:28
If I'm perpendicular to the
column space,
55
00:03:28 --> 00:03:32
I'm in the null space of A
transpose.
56
00:03:32 --> 00:03:32
OK.
57
00:03:32 --> 00:03:37
So if I'm in the null space of
A transpose, and I multiply this
58
00:03:37 --> 00:03:41
big formula times b,
so now I'm getting Pb,
59
00:03:41 --> 00:03:46
this is now the projection,
Pb, do you see that I get zero?
60
00:03:46 --> 00:03:47
Of course I get zero.
61
00:03:47 --> 00:03:52
Right at the end there,
A transpose b will give me zero
62
00:03:52 --> 00:03:53
right away.
63
00:03:53 --> 00:03:56
So that's why that zero's here.
64
00:03:56 --> 00:04:00
Because if I'm perpendicular to
the column space,
65
00:04:00 --> 00:04:05.46
then I'm in the null space of A
transpose and A transpose b is
66
00:04:05.46 --> 00:04:05
zilch.
67
00:04:05 --> 00:04:08
OK, what about the other
possibility.
68
00:04:08 --> 00:04:13
How do I see that this formula
gives me the right answer if b
69
00:04:13 --> 00:04:16
is in the column space?
70
00:04:16 --> 00:04:20
So what's a typical vector in
the column space?
71
00:04:20 --> 00:04:23
It's a combination of the
columns.
72
00:04:23 --> 00:04:26
How do I write a combination of
the columns?
73
00:04:26 --> 00:04:31
So tell me, how would I write,
you know, your everyday vector
74
00:04:31 --> 00:04:33
that's in the column space?
75
00:04:33 --> 00:04:38
It would have the form A times
some x, right?
76
00:04:38 --> 00:04:42
That's what's in the column
space, A times something.
77
00:04:42 --> 00:04:45.34
That makes it a combination of
the columns.
78
00:04:45.34 --> 00:04:49
So these b's were in the null
space of A transpose.
79
00:04:49 --> 00:04:52
These guys in the column space,
those b's are Ax-s.
80
00:04:52 --> 00:04:53
Right?
81
00:04:53 --> 00:04:57.09
If b is in the column space
then it has the form Ax.
82
00:04:57.09 --> 00:05:02
I'm going to stick that on the
quiz or the final for sure.
83
00:05:02 --> 00:05:06
That you have to realize --
because we've said it like a
84
00:05:06 --> 00:05:12
thousand times that the things
in the column space are vectors
85
00:05:12 --> 00:05:13
A times x.
86
00:05:13 --> 00:05:13
OK.
87
00:05:13 --> 00:05:17
And do you see what happens now
if we use our formula?
88
00:05:17 --> 00:05:19
There's an A transpose A.
89
00:05:19 --> 00:05:23
Gets canceled by its inverse.
90
00:05:23 --> 00:05:26
We're left with an A times x.
91
00:05:26 --> 00:05:28
So the result was Ax.
92
00:05:28 --> 00:05:29
Which was b.
93
00:05:29 --> 00:05:32
Do you see that it works?
94
00:05:32 --> 00:05:35
This is that whole business.
95
00:05:35 --> 00:05:37
Cancel, cancel,
leaving Ax.
96
00:05:37 --> 00:05:39
And Ax was b.
97
00:05:39 --> 00:05:43
So that turned out to be b,
in this case.
98
00:05:43 --> 00:05:49
OK, so geometrically what we're
seeing is we're taking a vector
99
00:05:49 --> 00:05:54.71
--
we've got the column space and
100
00:05:54.71 --> 00:06:01
perpendicular to that is the
null space of A transpose.
101
00:06:01 --> 00:06:05.22
And our typical vector b is out
here.
102
00:06:05.22 --> 00:06:10
There's zero,
so there's our typical vector
103
00:06:10 --> 00:06:16
b, and what we're doing is we're
projecting it to P.
104
00:06:16 --> 00:06:21
And the --
and of course at the same time
105
00:06:21 --> 00:06:26
we're finding the other part of
it which is e.
106
00:06:26 --> 00:06:31
So the two pieces,
the projection piece and the
107
00:06:31 --> 00:06:35
error piece, add up to the
original b.
108
00:06:35 --> 00:06:36
OK.
109
00:06:36 --> 00:06:39
That's like what our matrix
does.
110
00:06:39 --> 00:06:44
So this is P -- P is --
this P is Ab,
111
00:06:44 --> 00:06:48
is sorry -- is Pb,
it's the projection,
112
00:06:48 --> 00:06:51
applied to b,
and this one is -- OK,
113
00:06:51 --> 00:06:53
that's a projection too.
114
00:06:53 --> 00:06:57
That's a projection down onto
that space.
115
00:06:57 --> 00:06:59
What's a good formula for it?
116
00:06:59 --> 00:07:04
Suppose I ask you for the
projection of the projection
117
00:07:04 --> 00:07:10
matrix onto the --
this space, this perpendicular
118
00:07:10 --> 00:07:11
space?
119
00:07:11 --> 00:07:17
So if this projection was P,
what's the projection that
120
00:07:17 --> 00:07:18
gives me e?
121
00:07:18 --> 00:07:24
It's the -- what I want is to
get the rest of the vector,
122
00:07:24 --> 00:07:30
so it'll be just I minus P
times b, that's a projection
123
00:07:30 --> 00:07:31
too.
124
00:07:31 --> 00:07:37
That's the projection onto the
perpendicular space.
125
00:07:37 --> 00:07:38
OK.
126
00:07:38 --> 00:07:43
So if P's a projection,
I minus P is a projection.
127
00:07:43 --> 00:07:47
If P is symmetric,
I minus P is symmetric.
128
00:07:47 --> 00:07:52.96
If P squared equals P,
then I minus P squared equals I
129
00:07:52.96 --> 00:07:53
minus P.
130
00:07:53 --> 00:08:00
It's just -- the algebra -- is
only doing what your --
131
00:08:00 --> 00:08:04
picture is completely telling
you.
132
00:08:04 --> 00:08:08
But the algebra leads to this
expression.
133
00:08:08 --> 00:08:16
That expression for P given --
given a basis for the subspace,
134
00:08:16 --> 00:08:22
given the matrix A whose
columns are a basis for our
135
00:08:22 --> 00:08:23
column space.
136
00:08:23 --> 00:08:31
OK, that's recap because you --
you need to see that formula
137
00:08:31 --> 00:08:32
more than once.
138
00:08:32 --> 00:08:35
And now can I pick up on using
it?
139
00:08:35 --> 00:08:40
So now -- and the -- it's like,
let me do that again,
140
00:08:40 --> 00:08:46
I'll go right through a problem
that I started at the end,
141
00:08:46 --> 00:08:49
which is find a best straight
line.
142
00:08:49 --> 00:08:53
You remember that problem,
I --
143
00:08:53 --> 00:08:58
I picked a particular set of
points, they weren't specially
144
00:08:58 --> 00:09:02
brilliant, t equal one,
two, three, the heights were
145
00:09:02 --> 00:09:05
one, two, and then two again.
146
00:09:05 --> 00:09:09
So they were -- heights were
that point, that point,
147
00:09:09 --> 00:09:14
which makes it look like I've
got a nice forty-five-degree
148
00:09:14 --> 00:09:18
line --
but then the third point didn't
149
00:09:18 --> 00:09:19
lie on the line.
150
00:09:19 --> 00:09:23.82
And I wanted to find the best
straight line.
151
00:09:23.82 --> 00:09:27
So I'm looking for the -- this
line, y=C+Dt.
152
00:09:27 --> 00:09:32
And it's not going to go
through all three points,
153
00:09:32 --> 00:09:36
because no line goes through
all three points.
154
00:09:36 --> 00:09:40
So I'm going to pick the best
line, the --
155
00:09:40 --> 00:09:46
the best being the one that
makes the overall error as small
156
00:09:46 --> 00:09:47
as I can make it.
157
00:09:47 --> 00:09:52.47
Now I have to tell you,
what is that overall error?
158
00:09:52.47 --> 00:09:57
And -- because that determines
what's the winning line.
159
00:09:57 --> 00:10:02
If we don't know -- I mean we
have to decide what we mean by
160
00:10:02 --> 00:10:07
the error --
and then we minimize and we
161
00:10:07 --> 00:10:11.2
find the right -- the best C and
D.
162
00:10:11.2 --> 00:10:11
OK.
163
00:10:11 --> 00:10:17
So if I went through this -- if
I went through that point,
164
00:10:17 --> 00:10:20
I would solve the equation
C+D=1.
165
00:10:20 --> 00:10:25
Because at t equal to one --
I'd have C plus D,
166
00:10:25 --> 00:10:28
and it would come out right.
167
00:10:28 --> 00:10:33
If it went through this point,
I'd have C plus two D equal to
168
00:10:33 --> 00:10:33
two.
169
00:10:33 --> 00:10:38
Because at t equal to two,
I would like to get the answer
170
00:10:38 --> 00:10:38
two.
171
00:10:38 --> 00:10:42.78
At the third point,
I have C plus three D because t
172
00:10:42.78 --> 00:10:48
is three, but the --
the answer I'm shooting for is
173
00:10:48 --> 00:10:48
two again.
174
00:10:48 --> 00:10:51
So those are my three
equations.
175
00:10:51 --> 00:10:54
And they don't have a solution.
176
00:10:54 --> 00:10:57
But they've got a best
solution.
177
00:10:57 --> 00:11:00
What do I mean by best
solution?
178
00:11:00 --> 00:11:05
So let me take time out to
remember what I'm talking about
179
00:11:05 --> 00:11:06.79
for best solution.
180
00:11:06.79 --> 00:11:10
So this is my equation Ax=b.
181
00:11:10 --> 00:11:14.36
A is this matrix,
one, one, one,
182
00:11:14.36 --> 00:11:19.39
one, two, three.
x is my -- only have two
183
00:11:19.39 --> 00:11:25
unknowns, C and D,
and b is my right-hand side,
184
00:11:25 --> 00:11:27
one, two, three.
185
00:11:27 --> 00:11:27
OK.
186
00:11:27 --> 00:11:29
No solution.
187
00:11:29 --> 00:11:35.61
Three eq- I have a three by two
matrix, I do have two
188
00:11:35.61 --> 00:11:42
independent columns --
so I do have a basis for the
189
00:11:42 --> 00:11:45
column space,
those two columns are
190
00:11:45 --> 00:11:50
independent, they're a basis for
the column space,
191
00:11:50 --> 00:11:54
but the column space doesn't
include that vector.
192
00:11:54 --> 00:12:00
So best possible in this --
what would best possible mean?
193
00:12:00 --> 00:12:06
The way that comes out to
linear equations is I --
194
00:12:06 --> 00:12:11
I want to minimize the sum of
these -- I'm going to make an
195
00:12:11 --> 00:12:13
error here.
196
00:12:13 --> 00:12:16
I'm going to make an error
here.
197
00:12:16 --> 00:12:19
I'm going to make an error
there.
198
00:12:19 --> 00:12:25.02
And I'm going to sum and square
and add up those errors.
199
00:12:25.02 --> 00:12:27
So it's a sum of squares.
200
00:12:27 --> 00:12:32.13
It's a least squares solution
I'm looking for.
201
00:12:32.13 --> 00:12:37
So if I --
those errors are the difference
202
00:12:37 --> 00:12:38
between Ax and b.
203
00:12:38 --> 00:12:41
That's what I want to make
small.
204
00:12:41 --> 00:12:45
And the way I'm measuring this
-- this is a vector,
205
00:12:45 --> 00:12:45
right?
206
00:12:45 --> 00:12:47
This is e1,e2 ,e3.
207
00:12:47 --> 00:12:49
The Ax-b, this is the e.
208
00:12:49 --> 00:12:50
The error vector.
209
00:12:50 --> 00:12:52.86
And small means its length.
210
00:12:52.86 --> 00:12:56
The length of that vector.
211
00:12:56 --> 00:13:00
That's what I'm going to try to
minimize.
212
00:13:00 --> 00:13:03
And it's convenient to square.
213
00:13:03 --> 00:13:09
If I make something small,
I make -- this is a never
214
00:13:09 --> 00:13:11
negative quantity,
right?
215
00:13:11 --> 00:13:14
The length of that vector.
216
00:13:14 --> 00:13:20
The length will be zero exactly
when the --
217
00:13:20 --> 00:13:22
when I have the zero vector
here.
218
00:13:22 --> 00:13:26
That's exactly the case when I
can solve exactly,
219
00:13:26 --> 00:13:29
b is in the column space,
all great.
220
00:13:29 --> 00:13:31
But I'm not in that case now.
221
00:13:31 --> 00:13:33
I'm going to have an error
vector, e.
222
00:13:33 --> 00:13:36
What's this error vector in my
picture?
223
00:13:36 --> 00:13:40
I guess what I'm trying to say
is there's --
224
00:13:40 --> 00:13:44
there's two pictures of what's
going on.
225
00:13:44 --> 00:13:47
There's two pictures of what's
going on.
226
00:13:47 --> 00:13:52
One picture is -- in this is
the three points and the line.
227
00:13:52 --> 00:13:56
And in that picture,
what are the three errors?
228
00:13:56 --> 00:14:01
The three errors are what I
miss by in this equation.
229
00:14:01 --> 00:14:05
So it's this -- this little bit
here.
230
00:14:05 --> 00:14:08
That vertical distance up to
the line.
231
00:14:08 --> 00:14:13
There's one -- sorry there's
one, and there's C plus D.
232
00:14:13 --> 00:14:15
And it's that difference.
233
00:14:15 --> 00:14:17
Here's two and here's C+2D.
234
00:14:17 --> 00:14:21
So vertically it's that
distance -- that little error
235
00:14:21 --> 00:14:22
there is e1.
236
00:14:22 --> 00:14:25
This little error here is e2.
237
00:14:25 --> 00:14:28
This little error coming up is
e3.
238
00:14:28 --> 00:14:29.24
e3.
239
00:14:29.24 --> 00:14:32
And what's my overall error?
240
00:14:32 --> 00:14:36
Is e1 square plus e2 squared
plus e3 squared.
241
00:14:36 --> 00:14:40
That's what I'm trying to make
small.
242
00:14:40 --> 00:14:46
I -- some statisticians -- this
is a big part of statistics,
243
00:14:46 --> 00:14:52
fitting straight lines is a big
part of science --
244
00:14:52 --> 00:14:58
and specifically statistics,
where the right word to use
245
00:14:58 --> 00:15:01
would be regression.
246
00:15:01 --> 00:15:04.22
I'm doing regression here.
247
00:15:04.22 --> 00:15:06
Linear regression.
248
00:15:06 --> 00:15:12
And I'm using this sum of
squares as the measure of error.
249
00:15:12 --> 00:15:18
Again, some statisticians would
be --
250
00:15:18 --> 00:15:21.42
they would say,
OK, I'll solve that problem
251
00:15:21.42 --> 00:15:23
because it's the clean problem.
252
00:15:23 --> 00:15:26
It leads to a beautiful linear
system.
253
00:15:26 --> 00:15:30
But they would be a little
careful about these squares,
254
00:15:30 --> 00:15:32
for -- in this case.
255
00:15:32 --> 00:15:34
If one of these points was way
off.
256
00:15:34 --> 00:15:40
Suppose I had a measurement at
t equal zero that was way off.
257
00:15:40 --> 00:15:45
Well, would the straight line,
would the best line be the same
258
00:15:45 --> 00:15:47
if I had this fourth point?
259
00:15:47 --> 00:15:50
Suppose I have this fourth data
point.
260
00:15:50 --> 00:15:54
No, certainly the line would --
it wouldn't be the -- that
261
00:15:54 --> 00:15:56
wouldn't be the best line.
262
00:15:56 --> 00:16:00.96
Because that line would have a
giant error --
263
00:16:00.96 --> 00:16:05
and when I squared it it would
be like way out of sight
264
00:16:05 --> 00:16:06
compared to the others.
265
00:16:06 --> 00:16:10.74
So this would be called by
statisticians an outlier,
266
00:16:10.74 --> 00:16:15
and they would not be happy to
see the whole problem turned
267
00:16:15 --> 00:16:19
topsy-turvy by this one outlier,
which could be a mistake,
268
00:16:19 --> 00:16:21
after all.
269
00:16:21 --> 00:16:26
So they wouldn't -- so they
wouldn't like maybe squaring,
270
00:16:26 --> 00:16:31
if there were outliers,
they would want to identify
271
00:16:31 --> 00:16:32
them.
272
00:16:32 --> 00:16:32
OK.
273
00:16:32 --> 00:16:37
I'm not going to -- I don't
want to suggest that least
274
00:16:37 --> 00:16:42
squares isn't used,
it's the most used,
275
00:16:42 --> 00:16:46
but it's not exclusively used
because it's a little --
276
00:16:46 --> 00:16:48
overcompensates for outliers.
277
00:16:48 --> 00:16:50
Because of that squaring.
278
00:16:50 --> 00:16:50
OK.
279
00:16:50 --> 00:16:54
So suppose we don't have this
guy, we just have these three
280
00:16:54 --> 00:16:55
equations.
281
00:16:55 --> 00:16:58
And I want to make -- minimize
this error.
282
00:16:58 --> 00:16:58
OK.
283
00:16:58 --> 00:17:03
Now, what I said is there's two
pictures to look at.
284
00:17:03 --> 00:17:06
One picture is this one.
285
00:17:06 --> 00:17:09
The three points,
the best line.
286
00:17:09 --> 00:17:11
And the errors.
287
00:17:11 --> 00:17:16
Now, on this picture,
what are these points on the
288
00:17:16 --> 00:17:20
line, the points that are really
on the line?
289
00:17:20 --> 00:17:24
So they're -- points,
let me call them P1,
290
00:17:24 --> 00:17:30
P2, and P3, those are three
numbers, so this --
291
00:17:30 --> 00:17:34
this height is P1,
this height is P2,
292
00:17:34 --> 00:17:39
this height is P3,
and what are those guys?
293
00:17:39 --> 00:17:46
Suppose those were the three
values instead of -- there's b1,
294
00:17:46 --> 00:17:52
ev- everybody's seen all these
-- sorry, my art is as usual not
295
00:17:52 --> 00:17:57
the greatest,
but there's the given b1,
296
00:17:57 --> 00:18:00
the given b2,
and the given b3.
297
00:18:00 --> 00:18:06
I promise not to put a single
letter more on that picture.
298
00:18:06 --> 00:18:06
OK.
299
00:18:06 --> 00:18:12
There's b1, P1 is the one on
the line, and e1 is the distance
300
00:18:12 --> 00:18:12
between.
301
00:18:12 --> 00:18:18
And same at points two and same
at points three.
302
00:18:18 --> 00:18:20
OK, so what's up?
303
00:18:20 --> 00:18:22
What's up with those Ps?
304
00:18:22 --> 00:18:24
P1, P2, P3, what are they?
305
00:18:24 --> 00:18:29
They're the components,
they lie on the line,
306
00:18:29 --> 00:18:29
right?
307
00:18:29 --> 00:18:33.76
They're the points which if
instead of one,
308
00:18:33.76 --> 00:18:38
two, two, which were the b's,
suppose I put P1,
309
00:18:38 --> 00:18:40
P2, P3 in here.
310
00:18:40 --> 00:18:45
I'll figure out in a minute
what those numbers are.
311
00:18:45 --> 00:18:51
But I just want to get the
picture of what I'm doing.
312
00:18:51 --> 00:18:54
If I put P1,
P2, P3 in those three
313
00:18:54 --> 00:19:00
equations, what would be good
about the three equations?
314
00:19:00 --> 00:19:02
I could solve them.
315
00:19:02 --> 00:19:04
A line goes through the Ps.
316
00:19:04 --> 00:19:08
So the P1, P2,
P3 vector, that's in the column
317
00:19:08 --> 00:19:08
space.
318
00:19:08 --> 00:19:11.72
That is a combination of these
columns.
319
00:19:11.72 --> 00:19:14
It's the closest combination.
320
00:19:14 --> 00:19:15
It's this picture.
321
00:19:15 --> 00:19:20
See, I've got the two pictures
like here's the picture that
322
00:19:20 --> 00:19:25.15
shows the points,
this is a picture in a
323
00:19:25.15 --> 00:19:30
blackboard plane,
here's a picture that's showing
324
00:19:30 --> 00:19:32
the vectors.
325
00:19:32 --> 00:19:36.22
The vector b,
which is in this case,
326
00:19:36.22 --> 00:19:41.23
in this example is the vector
one, two, two.
327
00:19:41.23 --> 00:19:47
The column space is in this
case spanned by the -- well,
328
00:19:47 --> 00:19:50
you see A there.
329
00:19:50 --> 00:19:55
The column space of the matrix
one, one, one,
330
00:19:55 --> 00:19:56
one, two, three.
331
00:19:56 --> 00:20:00
And this picture shows the
nearest point.
332
00:20:00 --> 00:20:06
There's the -- that point P1,
P2, P3, which I'm going to
333
00:20:06 --> 00:20:12
compute before the end of this
hour, is the closest point in
334
00:20:12 --> 00:20:14
the column space.
335
00:20:14 --> 00:20:14
OK.
336
00:20:14 --> 00:20:19
Let me -- t
I don't dare leave it any
337
00:20:19 --> 00:20:23
longer -- can I just compute it
now.
338
00:20:23 --> 00:20:27
So I want to compute -- find P.
339
00:20:27 --> 00:20:28
All right.
340
00:20:28 --> 00:20:29
Find P.
341
00:20:29 --> 00:20:33
Find x, which is CD,
find P and P.
342
00:20:33 --> 00:20:34
OK.
343
00:20:34 --> 00:20:41
And I really should put these
little hats on to remind myself
344
00:20:41 --> 00:20:50
that they're the estimated the
best line, not the perfect line.
345
00:20:50 --> 00:20:50
OK.
346
00:20:50 --> 00:20:51
OK.
347
00:20:51 --> 00:20:53
How do I proceed?
348
00:20:53 --> 00:20:57.41
Let's just run through the
mechanics.
349
00:20:57.41 --> 00:21:00
What's the equation for x?
350
00:21:00 --> 00:21:02
The -- or x hat.
351
00:21:02 --> 00:21:08
The equation for that is A
transpose A x hat equals A
352
00:21:08 --> 00:21:11
transpose x -- A transpose b.
353
00:21:11 --> 00:21:18
The most -- I'm -- will venture
to call that the most important
354
00:21:18 --> 00:21:22
equation in statistics.
355
00:21:22 --> 00:21:24
And in estimation.
356
00:21:24 --> 00:21:30
And whatever you're -- wherever
you've got error and noise this
357
00:21:30 --> 00:21:34
is the estimate that you use
first.
358
00:21:34 --> 00:21:34
OK.
359
00:21:34 --> 00:21:39
Whenever you're fitting things
by a few parameters,
360
00:21:39 --> 00:21:42
that's the equation to use.
361
00:21:42 --> 00:21:43.93
OK, let's solve it.
362
00:21:43.93 --> 00:21:47
What is A transpose A?
363
00:21:47 --> 00:21:51
So I have to figure out what
these matrices are.
364
00:21:51 --> 00:21:55
One, one, one,
one, two, three and one,
365
00:21:55 --> 00:21:59
one, one, one,
two, three, that gives me some
366
00:21:59 --> 00:22:04
matrix, that gives me a matrix,
what do I get out of that,
367
00:22:04 --> 00:22:08
three, six, six,
and one and four and nine,
368
00:22:08 --> 00:22:10
fourteen.
369
00:22:10 --> 00:22:11
OK.
370
00:22:11 --> 00:22:18
And what do I expect to see in
that matrix and I do see it,
371
00:22:18 --> 00:22:24
just before I keep going with
the calculation?
372
00:22:24 --> 00:22:29
I expect that matrix to be
symmetric.
373
00:22:29 --> 00:22:34.59
I expect it to be invertible.
374
00:22:34.59 --> 00:22:40
And near the end of the course
I'm going to say I expect it to
375
00:22:40 --> 00:22:44
be positive definite,
but that's a future fact about
376
00:22:44 --> 00:22:48
this crucial matrix,
A transpose A.
377
00:22:48 --> 00:22:48.43
OK.
378
00:22:48.43 --> 00:22:51
And now let me figure A
transpose b.
379
00:22:51 --> 00:22:56
So let me -- can I tack on b as
an extra column here,
380
00:22:56 --> 00:22:58
one, two, two?
381
00:22:58 --> 00:23:04
And tack on the extra A
transpose b is -- looks like
382
00:23:04 --> 00:23:09.18
five and one and four and six,
eleven.
383
00:23:09.18 --> 00:23:15
I think my equations are three
C plus six D equals five,
384
00:23:15 --> 00:23:21
and six D plus fourt-six C plus
fourteen D is eleven.
385
00:23:21 --> 00:23:26
Can I just for safety see if I
did that right?
386
00:23:26 --> 00:23:33
One, one, one times one,
two, two is five.
387
00:23:33 --> 00:23:39
One, two, three,
that's one, four and six,
388
00:23:39 --> 00:23:40
eleven.
389
00:23:40 --> 00:23:42
Looks good.
390
00:23:42 --> 00:23:45
These are my equations.
391
00:23:45 --> 00:23:53
That's my -- they're called the
normal equations.
392
00:23:53 --> 00:24:04
I'll just write that word down
because it -- so I solve them.
393
00:24:04 --> 00:24:06
I solve that for C and D.
394
00:24:06 --> 00:24:11.48
I would like to -- before I
solve them could I do one thing
395
00:24:11.48 --> 00:24:14
that's on the -- that's just
above here?
396
00:24:14 --> 00:24:19
I would like to -- I'd like to
find these equations from
397
00:24:19 --> 00:24:20
calculus.
398
00:24:20 --> 00:24:24
I'd like to find them from this
minimizing thing.
399
00:24:24 --> 00:24:28
So what's the first error?
400
00:24:28 --> 00:24:34
The first error is what I
missed by in the first equation.
401
00:24:34 --> 00:24:37
C plus D minus one squared.
402
00:24:37 --> 00:24:43.27
And the second error is what I
miss in the second equation.
403
00:24:43.27 --> 00:24:46
C plus two D minus two squared.
404
00:24:46 --> 00:24:52
And the third error squared is
C plus three D minus two
405
00:24:52 --> 00:24:53
squared.
406
00:24:53 --> 00:24:58
That's my -- overall squared
error that I'm trying to
407
00:24:58 --> 00:25:01
minimize.
408
00:25:01 --> 00:25:01
OK.
409
00:25:01 --> 00:25:05
So how would you minimize that?
410
00:25:05 --> 00:25:12
OK, linear algebra has given us
the equations for the minimum.
411
00:25:12 --> 00:25:15
But we could use calculus too.
412
00:25:15 --> 00:25:21
That's a function of two
variables, C and D,
413
00:25:21 --> 00:25:24
and we're looking for the
minimum.
414
00:25:24 --> 00:25:27.48
So how do we find it?
415
00:25:27.48 --> 00:25:33
Directly from calculus,
we take partial derivatives,
416
00:25:33 --> 00:25:38
right, we've got two variables,
C and D, so take the partial
417
00:25:38 --> 00:25:41
derivative with respect to C and
set it to zero,
418
00:25:41 --> 00:25:43
and you'll get that equation.
419
00:25:43 --> 00:25:48
Take the partial derivative
with respect -- I'm not going to
420
00:25:48 --> 00:25:51
write it all out,
just -- you will.
421
00:25:51 --> 00:25:54.33
The partial derivative with
respect to D,
422
00:25:54.33 --> 00:25:56
it -- you know,
it's going to be linear,
423
00:25:56 --> 00:26:00
that's the beauty of these
squares,that if I have the
424
00:26:00 --> 00:26:04.42
square of something and I take
its derivative I get something
425
00:26:04.42 --> 00:26:04
linear.
426
00:26:04 --> 00:26:06
And this is what I get.
427
00:26:06 --> 00:26:10
So this is the derivative of
the error with respect to C
428
00:26:10 --> 00:26:13
being zero,
and this is the derivative of
429
00:26:13 --> 00:26:16
the error with respect to D
being zero.
430
00:26:16 --> 00:26:20
Wherever you look,
these equations keep coming.
431
00:26:20 --> 00:26:24
So now I guess I'm going to
solve it, what will I do,
432
00:26:24 --> 00:26:27
I'll subtract,
I'll do elimination of course,
433
00:26:27 --> 00:26:31
because that's the only thing I
know how to do.
434
00:26:31 --> 00:26:36
Two of these away from this
would give me -- let's see,
435
00:26:36 --> 00:26:39
six, so would that be two Ds
equals one?
436
00:26:39 --> 00:26:39
Ha.
437
00:26:39 --> 00:26:44
So it wasn't -- I was afraid
these numbers were going to come
438
00:26:44 --> 00:26:45
out awful.
439
00:26:45 --> 00:26:49.8
But if I take two of those away
from that, the equation I get
440
00:26:49.8 --> 00:26:55.56
left is two D equals one,
so I think D is a half and C is
441
00:26:55.56 --> 00:27:00
whatever back substitution
gives, six D is three,
442
00:27:00 --> 00:27:05
so three C plus three is five,
I'm doing back substitution
443
00:27:05 --> 00:27:09
now, right, three,
can I do it in light letters,
444
00:27:09 --> 00:27:13
three C plus that six D is
three equals five,
445
00:27:13 --> 00:27:19
so three C is two,
so I think C is two-thirds.
446
00:27:19 --> 00:27:23
One-half and two-thirds.
447
00:27:23 --> 00:27:30
So the best line,
the best line is the constant
448
00:27:30 --> 00:27:35
two-thirds plus one-half t.
449
00:27:35 --> 00:27:41
And I -- is my picture more or
less right?
450
00:27:41 --> 00:27:48.98
Let me write,
let me copy that best line down
451
00:27:48.98 --> 00:27:55
again, two-thirds and a half.
452
00:27:55 --> 00:27:58
Let me -- I'll put in the
two-thirds and the half.
453
00:27:58 --> 00:27:59
OK.
454
00:27:59 --> 00:28:02
So what's this P1,
that's the value at t equal to
455
00:28:02 --> 00:28:03.11
one.
456
00:28:03.11 --> 00:28:06
At t equal to one,
I have two-thirds plus a half,
457
00:28:06 --> 00:28:10
which is -- what's that,
four-sixths and three-sixths,
458
00:28:10 --> 00:28:16
so P1, oh, I promised not to
write another thing on this --
459
00:28:16 --> 00:28:19
I'll erase P1 and I'll put
seven-sixths.
460
00:28:19 --> 00:28:20
OK.
461
00:28:20 --> 00:28:24
And yeah, it's above one,
and e1 is one-sixth,
462
00:28:24 --> 00:28:24
right.
463
00:28:24 --> 00:28:26
You see it all.
464
00:28:26 --> 00:28:26
Right?
465
00:28:26 --> 00:28:27
What's P2?
466
00:28:27 --> 00:28:28
OK.
467
00:28:28 --> 00:28:32
At point t equal to two,
where's my line here?
468
00:28:32 --> 00:28:36
At t equal to two,
it's two-thirds plus one,
469
00:28:36 --> 00:28:38
right?
470
00:28:38 --> 00:28:40
That's five-thirds.
471
00:28:40 --> 00:28:45
Two-thirds and t is two,
so that's two-thirds and one
472
00:28:45 --> 00:28:46
make five-thirds.
473
00:28:46 --> 00:28:51
And that's -- sure enough,
that's smaller than the exact
474
00:28:51 --> 00:28:52
two.
475
00:28:52 --> 00:28:55
And then final P3,
when t is three,
476
00:28:55 --> 00:29:00
oh, what's two-thirds plus
three-halves?
477
00:29:00 --> 00:29:04
It's the same as three-halves
plus two-thirds.
478
00:29:04 --> 00:29:08
It's -- so maybe four-sixths
and nine-sixths,
479
00:29:08 --> 00:29:10
maybe thirteen-sixths.
480
00:29:10 --> 00:29:13
OK, and again,
look, oh, look at this,
481
00:29:13 --> 00:29:14
OK.
482
00:29:14 --> 00:29:18
You have to admire the beauty
of this answer.
483
00:29:18 --> 00:29:20
What's this first error?
484
00:29:20 --> 00:29:24
So here are the errors.
e1, e2 and e3.
485
00:29:24 --> 00:29:27
OK, what was that first error,
e1?
486
00:29:27 --> 00:29:32
Well, if we decide the errors
counting up, then it's
487
00:29:32 --> 00:29:33
one-sixth.
488
00:29:33 --> 00:29:38
And the last error,
thirteen-sixths minus the
489
00:29:38 --> 00:29:41
correct two is one-sixth again.
490
00:29:41 --> 00:29:44
And what's this error in the
middle?
491
00:29:44 --> 00:29:49
Let's see, the correct answer
was two, two.
492
00:29:49 --> 00:29:56
And we got five-thirds and it's
the other direction,
493
00:29:56 --> 00:30:00
minus one-third,
minus two-sixths.
494
00:30:00 --> 00:30:03
That's our error vector.
495
00:30:03 --> 00:30:08
In our picture,
in our other picture,
496
00:30:08 --> 00:30:09
here it is.
497
00:30:09 --> 00:30:14
We just found P and e.
e is this vector,
498
00:30:14 --> 00:30:23
one-sixth, minus two-sixths,
one-sixth, and P is this guy.
499
00:30:23 --> 00:30:29
Well, maybe I have the signs of
e wrong, I think I have,
500
00:30:29 --> 00:30:30
let me fix it.
501
00:30:30 --> 00:30:36
Because I would like this
one-sixth -- I would like this
502
00:30:36 --> 00:30:39
plus the P to give the original
b.
503
00:30:39 --> 00:30:42
I want P plus e to match b.
504
00:30:42 --> 00:30:48
So I want minus a sixth,
plus seven-sixths to give the
505
00:30:48 --> 00:30:51
correct b equal one.
506
00:30:51 --> 00:30:52
OK.
507
00:30:52 --> 00:30:58.84
Now -- I'm going to take a deep
breath here, and ask what do we
508
00:30:58.84 --> 00:31:02.24
know about this error vector e?
509
00:31:02.24 --> 00:31:07
You've seen now this whole
problem worked completely
510
00:31:07 --> 00:31:12
through, and I even think the
numbers are right.
511
00:31:12 --> 00:31:18
So there's P,
so let me -- I'll write --
512
00:31:18 --> 00:31:23
if I can put it down here,
B is P plus e.
513
00:31:23 --> 00:31:27
b I believe was one,
two, two.
514
00:31:27 --> 00:31:32.06
The nearest point had
seven-sixths,
515
00:31:32.06 --> 00:31:34
what were the others?
516
00:31:34 --> 00:31:38
Five-thirds and
thirteen-sixths.
517
00:31:38 --> 00:31:45
And the e vector was minus a
sixth, two-sixths,
518
00:31:45 --> 00:31:52
one-third in other words,
and minus a sixth.
519
00:31:52 --> 00:31:52
OK.
520
00:31:52 --> 00:31:57
Tell me some stuff about these
two vectors.
521
00:31:57 --> 00:32:02
Tell me something about those
two vectors, well,
522
00:32:02 --> 00:32:05
they add to b,
right, great.
523
00:32:05 --> 00:32:06
OK.
524
00:32:06 --> 00:32:07
What else?
525
00:32:07 --> 00:32:11
What else about those two
vectors, the P,
526
00:32:11 --> 00:32:18.4
the projection vector P,
and the error vector e.
527
00:32:18.4 --> 00:32:21.78
What else do you know about
them?
528
00:32:21.78 --> 00:32:24
They're perpendicular,
right.
529
00:32:24 --> 00:32:27
Do we dare verify that?
530
00:32:27 --> 00:32:31
Can you take the dot product of
those vectors?
531
00:32:31 --> 00:32:37
I'm like getting like minus
seven over thirty-six,
532
00:32:37 --> 00:32:40
can I change that to
ten-sixths?
533
00:32:40 --> 00:32:43
Oh, God, come out right here.
534
00:32:43 --> 00:32:50
Minus seven over thirty-six,
plus twenty over thirty-six,
535
00:32:50 --> 00:32:54
minus thirteen over thirty-six.
536
00:32:54 --> 00:32:55
Thank you, God.
537
00:32:55 --> 00:32:56
OK.
538
00:32:56 --> 00:33:01
And what else should we know
about that vector?
539
00:33:01 --> 00:33:07
Actually we know -- I've got to
say we know even a little more.
540
00:33:07 --> 00:33:12
This vector,
e, is perpendicular to P,
541
00:33:12 --> 00:33:17
but it's perpendicular to other
stuff too.
542
00:33:17 --> 00:33:22
It's perpendicular not just to
this guy in the column space,
543
00:33:22 --> 00:33:24
this is in the column space for
sure.
544
00:33:24 --> 00:33:27
This is perpendicular to the
column space.
545
00:33:27 --> 00:33:31
So like give me another vector
it's perpendicular to.
546
00:33:31 --> 00:33:34
Another because it's
perpendicular to the whole
547
00:33:34 --> 00:33:37
column space,
not just to this -- this
548
00:33:37 --> 00:33:42
particular projection that's --
that is in the column space,
549
00:33:42 --> 00:33:46
but it's perpendicular to other
stuff, whatever's in the column
550
00:33:46 --> 00:33:49
space, so tell me another vector
in the -- oh,
551
00:33:49 --> 00:33:53
well, I've written down the
matrix, so tell me another
552
00:33:53 --> 00:33:55
vector in the column space.
553
00:33:55 --> 00:33:57
Pick a nice one.
554
00:33:57 --> 00:33:58
One, one, one.
555
00:33:58 --> 00:34:00
That's what everybody's
thinking.
556
00:34:00 --> 00:34:03
OK, one, one,
one is in the column space.
557
00:34:03 --> 00:34:06
And this guy is supposed to be
perpendicular to one,
558
00:34:06 --> 00:34:06
one, one.
559
00:34:06 --> 00:34:07
Is it?
560
00:34:07 --> 00:34:07
Sure.
561
00:34:07 --> 00:34:11.37
If I take the dot product with
one, one, one I get minus a
562
00:34:11.37 --> 00:34:13
sixth, plus two-sixths,
minus a sixth,
563
00:34:13 --> 00:34:15
zero.
564
00:34:15 --> 00:34:19
And it's perpendicular to one,
two, three.
565
00:34:19 --> 00:34:23
Because if I take the dot
product with one,
566
00:34:23 --> 00:34:28
two, three I get minus one,
plus four, minus three,
567
00:34:28 --> 00:34:29
zero again.
568
00:34:29 --> 00:34:34
OK, do you see the -- I hope
you see the two pictures.
569
00:34:34 --> 00:34:38
The picture here for vectors
and,
570
00:34:38 --> 00:34:44.74
the picture here for the best
line, and it's the same picture,
571
00:34:44.74 --> 00:34:50
just -- this one's in the plane
and it's showing the line,
572
00:34:50 --> 00:34:56
this one never did show the
line, this -- in this picture,
573
00:34:56 --> 00:34:59
C and D never showed up.
574
00:34:59 --> 00:35:03
In this picture,
C and D were --
575
00:35:03 --> 00:35:06.61
you know, they determined that
line.
576
00:35:06.61 --> 00:35:09
But the two are exactly the
same.
577
00:35:09 --> 00:35:15
C and D is the combination of
the two columns that gives P.
578
00:35:15 --> 00:35:15
OK.
579
00:35:15 --> 00:35:18
So that's these squares.
580
00:35:18 --> 00:35:23
And the special but most
important example of fitting by
581
00:35:23 --> 00:35:27
straight line,
so the homework that's coming
582
00:35:27 --> 00:35:33
then Wednesday asks you to fit
by straight lines.
583
00:35:33 --> 00:35:41.2
So you're just going to end up
solving the key equation.
584
00:35:41.2 --> 00:35:48
You're going to end up solving
that key equation and then P
585
00:35:48 --> 00:35:50
will be Ax hat.
586
00:35:50 --> 00:35:52
That's it.
587
00:35:52 --> 00:35:52
OK.
588
00:35:52 --> 00:35:59
Now, can I put in a little
piece of linear algebra that I
589
00:35:59 --> 00:36:04
mentioned earlier,
mentioned again,
590
00:36:04 --> 00:36:07
but I never did write?
591
00:36:07 --> 00:36:12
And I've --
I should do it right.
592
00:36:12 --> 00:36:17
It's about this matrix A
transpose A.
593
00:36:17 --> 00:36:18
There.
594
00:36:18 --> 00:36:23
I was sure that that matrix
would be invertible.
595
00:36:23 --> 00:36:29
And of course I wanted to be
sure it was invertible,
596
00:36:29 --> 00:36:38
because I planned to solve this
system with with that matrix.
597
00:36:38 --> 00:36:46
So and I announced like before
-- as the chapter was just
598
00:36:46 --> 00:36:54
starting, I announced that it
would be invertible.
599
00:36:54 --> 00:37:00
But now I -- can I come back to
that?
600
00:37:00 --> 00:37:00
OK.
601
00:37:00 --> 00:37:09
So what I said was -- that if A
has independent columns,
602
00:37:09 --> 00:37:15
then A transpose A is
invertible.
603
00:37:15 --> 00:37:23
And I would like to -- first to
repeat that important fact,
604
00:37:23 --> 00:37:30
that that's the requirement
that makes everything go here.
605
00:37:30 --> 00:37:37
It's this independent columns
of A that guarantees everything
606
00:37:37 --> 00:37:39
goes through.
607
00:37:39 --> 00:37:41
And think why.
608
00:37:41 --> 00:37:46.48
Why does this matrix A
transpose A,
609
00:37:46.48 --> 00:37:53
why is it invertible if the
columns of A are independent?
610
00:37:53 --> 00:38:00
OK, there's -- so if it wasn't
invertible, I'm -- so I want to
611
00:38:00 --> 00:38:02
prove that.
612
00:38:02 --> 00:38:06
If it isn't invertible,
then what?
613
00:38:06 --> 00:38:13
I want to reach -- I want to
follow that -- follow that line
614
00:38:13 --> 00:38:18.72
--
of thinking and see what I come
615
00:38:18.72 --> 00:38:19
to.
616
00:38:19 --> 00:38:21
Suppose, so proof.
617
00:38:21 --> 00:38:24
Suppose A transpose Ax is zero.
618
00:38:24 --> 00:38:27
I'm trying to prove this.
619
00:38:27 --> 00:38:29
This is now to prove.
620
00:38:29 --> 00:38:36
I don't like hammer away at too
many proofs in this course.
621
00:38:36 --> 00:38:42
But this is like the central
fact and it brings in all the
622
00:38:42 --> 00:38:45
stuff we know.
623
00:38:45 --> 00:38:46
OK.
624
00:38:46 --> 00:38:48
So I'll start the proof.
625
00:38:48 --> 00:38:51
Suppose A transpose Ax is zero.
626
00:38:51 --> 00:38:56
What -- and I'm aiming to prove
A transpose A is invertible.
627
00:38:56 --> 00:38:59
So what do I want to prove now?
628
00:38:59 --> 00:39:02
So I'm aiming to prove this
fact.
629
00:39:02 --> 00:39:06
I'll use this,
and I'm aiming to prove that
630
00:39:06 --> 00:39:12
this matrix is invertible,
OK, so if I suppose A transpose
631
00:39:12 --> 00:39:17
Ax is zero, then what conclusion
do I want to reach?
632
00:39:17 --> 00:39:21
I'd like to know that x must be
zero.
633
00:39:21 --> 00:39:23
I want to show x must be zero.
634
00:39:23 --> 00:39:28
To show now -- to prove x must
be the zero vector.
635
00:39:28 --> 00:39:32.75
Is that right,
that's what we worked in the
636
00:39:32.75 --> 00:39:40
previous chapter to understand,
that a matrix was invertible
637
00:39:40 --> 00:39:46
when its null space is only the
zero vector.
638
00:39:46 --> 00:39:51
So that's what I want to show.
639
00:39:51 --> 00:39:59
How come if A transpose Ax is
zero, how come x must be zero?
640
00:39:59 --> 00:40:03
What's going to be the reason?
641
00:40:03 --> 00:40:08
Actually I have two ways to do
it.
642
00:40:08 --> 00:40:11
Let me show you one way.
643
00:40:11 --> 00:40:16.67
This is -- here,
trick.
644
00:40:16.67 --> 00:40:21
Take the dot product of both
sides with x.
645
00:40:21 --> 00:40:25.92
So I'll multiply both sides by
x transpose.
646
00:40:25.92 --> 00:40:30
x transpose A transpose Ax
equals zero.
647
00:40:30 --> 00:40:33.61
I shouldn't have written trick.
648
00:40:33.61 --> 00:40:38
That makes it sound like just a
dumb idea.
649
00:40:38 --> 00:40:41
Brilliant idea,
I should have put.
650
00:40:41 --> 00:40:42
OK.
651
00:40:42 --> 00:40:45
I'll just put idea.
652
00:40:45 --> 00:40:46
OK.
653
00:40:46 --> 00:40:53
Now, I got to that equation,
x transpose A transpose Ax=0,
654
00:40:53 --> 00:41:01
and I'm hoping you can see the
right way to -- to look at that
655
00:41:01 --> 00:41:02
equation.
656
00:41:02 --> 00:41:09
What can I conclude from that
equation, that if I have x
657
00:41:09 --> 00:41:16
transpose A -- well,
what is x transpose A transpose
658
00:41:16 --> 00:41:18
Ax?
659
00:41:18 --> 00:41:22
Does that -- what it's giving
you?
660
00:41:22 --> 00:41:27
It's again going to be putting
in parentheses,
661
00:41:27 --> 00:41:32
I'm looking at Ax and what I
seeing here?
662
00:41:32 --> 00:41:34
Its transpose.
663
00:41:34 --> 00:41:40
So I'm seeing here this is Ax
transpose Ax.
664
00:41:40 --> 00:41:42
Equaling zero.
665
00:41:42 --> 00:41:49
Now if Ax transpose Ax,
so like let's call it y or
666
00:41:49 --> 00:41:57
something, if y transpose y is
zero, what does that tell me?
667
00:41:57 --> 00:42:02
That the vector has to be zero,
right?
668
00:42:02 --> 00:42:10.38
This is the length squared,
that's the length of the vector
669
00:42:10.38 --> 00:42:15
Ax squared,
that's Ax times Ax.
670
00:42:15 --> 00:42:21
So I conclude that Ax has to be
zero.
671
00:42:21 --> 00:42:25
Well, I'm getting somewhere.
672
00:42:25 --> 00:42:33
Now that I know Ax is zero,
now I'm going to use my little
673
00:42:33 --> 00:42:34
hypothesis.
674
00:42:34 --> 00:42:43.9
Somewhere every mathematician
has to use the hypothesis.
675
00:42:43.9 --> 00:42:44
Right?
676
00:42:44 --> 00:42:49.09
Now, if A has independent
columns and we've -- we're at
677
00:42:49.09 --> 00:42:53
the point where Ax is zero,
what does that tell us?
678
00:42:53 --> 00:42:58
I could -- I mean that could be
like a fill-in question on the
679
00:42:58 --> 00:42:59
final exam.
680
00:42:59 --> 00:43:04
If A has independent columns
and if Ax equals zero then what?
681
00:43:04 --> 00:43:07
Please say it.
x is zero, right.
682
00:43:07 --> 00:43:12
Which was just what we wanted
to prove.
683
00:43:12 --> 00:43:14
That -- do you see why that is?
684
00:43:14 --> 00:43:20
If Ax eq- equals zero,
now we're using -- here we used
685
00:43:20 --> 00:43:25.29
this was the square of
something, so I'll put in little
686
00:43:25.29 --> 00:43:31
parentheses the observation we
made, that was a square which is
687
00:43:31 --> 00:43:35
zero, so the thing has to be
zero.
688
00:43:35 --> 00:43:41.09
Now we're using the hypothesis
of independent columns at the A
689
00:43:41.09 --> 00:43:43.37
has independent columns.
690
00:43:43.37 --> 00:43:48
If A has independent columns,
this is telling me x is in its
691
00:43:48 --> 00:43:54.39
null space, and the only thing
in the null space of such a
692
00:43:54.39 --> 00:43:56.86
matrix is the zero vector.
693
00:43:56.86 --> 00:43:57
OK.
694
00:43:57 --> 00:44:02
So that's the argument and you
see how it really used our
695
00:44:02 --> 00:44:08
understanding of the --
of the null space.
696
00:44:08 --> 00:44:09
OK.
697
00:44:09 --> 00:44:11.05
That's great.
698
00:44:11.05 --> 00:44:12
All right.
699
00:44:12 --> 00:44:15
So where are we then?
700
00:44:15 --> 00:44:24
That board is like the backup
theory that tells me that this
701
00:44:24 --> 00:44:33
matrix had to be invertible
because these columns were
702
00:44:33 --> 00:44:35.03
independent.
703
00:44:35.03 --> 00:44:40.29
OK.
there's one case of independent
704
00:44:40.29 --> 00:44:45
--
there's one case where the
705
00:44:45 --> 00:44:48
geometry gets even better.
706
00:44:48 --> 00:44:54
When the -- there's one case
when columns are sure to be
707
00:44:54 --> 00:44:55
independent.
708
00:44:55 --> 00:45:01
And let me put that -- let me
write that down and that'll be
709
00:45:01 --> 00:45:04.79
the subject for next time.
710
00:45:04.79 --> 00:45:12
Columns are sure --
are certainly independent,
711
00:45:12 --> 00:45:20
definitely independent,
if they're perpendicular.
712
00:45:20 --> 00:45:30
Oh, I've got to rule out the
zero column, let me give them
713
00:45:30 --> 00:45:38
all length one,
so they can't be zero if they
714
00:45:38 --> 00:45:45
are perpendicular unit vectors.
715
00:45:45 --> 00:45:49
Like the vectors one,
zero, zero, zero,
716
00:45:49 --> 00:45:52
one, zero and zero,
zero, one.
717
00:45:52 --> 00:45:57.28
Those vectors are unit vectors,
they're perpendicular,
718
00:45:57.28 --> 00:46:00
and they certainly are
independent.
719
00:46:00 --> 00:46:04
And what's more,
suppose they're -- oh,
720
00:46:04 --> 00:46:08
that's so nice,
I mean what is A transpose A
721
00:46:08 --> 00:46:10
for that matrix?
722
00:46:10 --> 00:46:15
For the matrix with these three
columns?
723
00:46:15 --> 00:46:17
It's the identity.
724
00:46:17 --> 00:46:21.52
So here's the key to the
lecture that's coming.
725
00:46:21.52 --> 00:46:26
If we're dealing with
perpendicular unit vectors and
726
00:46:26 --> 00:46:32
the word for that will be -- see
I could have said orthogonal,
727
00:46:32 --> 00:46:38
but I said perpendicular -- and
this unit vectors gets put in as
728
00:46:38 --> 00:46:40
the word normal.
729
00:46:40 --> 00:46:42
Orthonormal vectors.
730
00:46:42 --> 00:46:46
Those are the best columns you
could ask for.
731
00:46:46 --> 00:46:50
Matrices with -- whose columns
are orthonormal,
732
00:46:50 --> 00:46:55
they're perpendicular to each
other, and they're unit vectors,
733
00:46:55 --> 00:47:00
well, they don't have to be
those three, let me do a final
734
00:47:00 --> 00:47:04
example over here,
how about one at an angle like
735
00:47:04 --> 00:47:11
that and one at ninety degrees,
that vector would be cos theta,
736
00:47:11 --> 00:47:17
sine theta, a unit vector,
and this vector would be minus
737
00:47:17 --> 00:47:20
sine theta cos theta.
738
00:47:20 --> 00:47:26
That is our absolute favorite
pair of orthonormal vectors.
739
00:47:26 --> 00:47:32.61
They're both unit vectors and
they're perpendicular.
740
00:47:32.61 --> 00:47:36
That angle is ninety degrees.
741
00:47:36 --> 00:47:43
So like our job next time is
first to see why orthonormal
742
00:47:43 --> 00:47:48
vectors are great,
and then to make vectors
743
00:47:48 --> 00:47:52
orthonormal by picking the right
basis.
744
00:47:52 --> 00:47:53
OK, see you.
745
00:47:53 --> 00:47:56
Thanks.