1
00:00:10,240 --> 00:00:17,470
OK, guys the -- we're almost
ready to make this lecture
2
00:00:17,470 --> 00:00:19,310
immortal.
3
00:00:19,310 --> 00:00:21,340
OK.
4
00:00:21,340 --> 00:00:22,720
Are we on?
5
00:00:22,720 --> 00:00:23,490
All right.
6
00:00:23,490 --> 00:00:26,820
This is an important lecture.
7
00:00:26,820 --> 00:00:30,170
It's about projections.
8
00:00:30,170 --> 00:00:36,430
Let me start by just projecting
a vector b down on a vector a.
9
00:00:36,430 --> 00:00:41,400
So just to so you see what the
geometry looks like in when
10
00:00:41,400 --> 00:00:44,060
I'm in -- in just
two dimensions,
11
00:00:44,060 --> 00:00:51,670
I'd like to find the point along
this line so that line through
12
00:00:51,670 --> 00:00:55,080
a is a one-dimensional subspace,
so I'm starting with one
13
00:00:55,080 --> 00:00:56,270
dimension.
14
00:00:56,270 --> 00:01:00,940
I'd like to find the point
on that line closest to a.
15
00:01:00,940 --> 00:01:03,390
Can I just take that
problem first and then I'll
16
00:01:03,390 --> 00:01:06,600
explain why I want to
do it and why I want
17
00:01:06,600 --> 00:01:09,620
to project on other subspaces.
18
00:01:09,620 --> 00:01:14,780
So where's the point closest
to b that's on that line?
19
00:01:14,780 --> 00:01:17,480
It's somewhere there.
20
00:01:17,480 --> 00:01:24,780
And let me connect that and
-- and what's the whole point
21
00:01:24,780 --> 00:01:25,800
of my picture now?
22
00:01:25,800 --> 00:01:29,440
What's the -- where does
orthogonality come into this
23
00:01:29,440 --> 00:01:30,350
picture?
24
00:01:30,350 --> 00:01:33,860
The whole point is that
this best point, that's
25
00:01:33,860 --> 00:01:44,200
the projection, P, of b onto
the line, where's orthogonality?
26
00:01:44,200 --> 00:01:46,820
It's the fact that
that's a right angle.
27
00:01:46,820 --> 00:01:50,990
That this -- the error -- this
is like how much I'm wrong
28
00:01:50,990 --> 00:01:51,940
by --
29
00:01:51,940 --> 00:01:57,730
this is the difference between
b and P, the whole point is --
30
00:01:57,730 --> 00:02:01,090
that's perpendicular to a.
31
00:02:01,090 --> 00:02:03,660
That's got to give
us the equation.
32
00:02:03,660 --> 00:02:07,370
That's got to tell us --
that's the one fact we know,
33
00:02:07,370 --> 00:02:11,090
that's got to tell us
where that projection is.
34
00:02:11,090 --> 00:02:14,200
Let me also say, look --
35
00:02:14,200 --> 00:02:16,970
I've drawn a triangle there.
36
00:02:16,970 --> 00:02:19,220
So if we were doing
trigonometry we
37
00:02:19,220 --> 00:02:25,680
would do like we would have
angles theta and distances that
38
00:02:25,680 --> 00:02:29,330
would involve sine
theta and cos theta that
39
00:02:29,330 --> 00:02:33,120
leads to lousy formulas
compared to linear algebra.
40
00:02:33,120 --> 00:02:40,750
The formula that we want comes
out nicely and what's the --
41
00:02:40,750 --> 00:02:42,260
what do we know?
42
00:02:42,260 --> 00:02:49,040
We know that P, this projection,
is some multiple of a, right?
43
00:02:49,040 --> 00:02:50,690
It's on that line.
44
00:02:50,690 --> 00:02:54,550
So we know it's in that
one-dimensional subspace, it's
45
00:02:54,550 --> 00:02:57,870
some multiple, let me
call that multiple x,
46
00:02:57,870 --> 00:02:59,060
of a.
47
00:02:59,060 --> 00:03:03,530
So really it's that
number x I'd like to find.
48
00:03:03,530 --> 00:03:07,500
So this is going to
be simple in 1-D,
49
00:03:07,500 --> 00:03:11,130
so let's just carry it
through, and then see
50
00:03:11,130 --> 00:03:13,890
how it goes in high dimensions.
51
00:03:13,890 --> 00:03:14,510
OK.
52
00:03:14,510 --> 00:03:17,120
The key fact is --
53
00:03:17,120 --> 00:03:21,000
the key to everything
is that perpendicular.
54
00:03:21,000 --> 00:03:31,150
The fact that a is perpendicular
to a is perpendicular to e.
55
00:03:31,150 --> 00:03:34,470
Which is (b-ax), xa.
56
00:03:34,470 --> 00:03:36,510
I don't care what --
57
00:03:36,510 --> 00:03:37,010
xa.
58
00:03:37,010 --> 00:03:41,950
That that equals zero.
59
00:03:41,950 --> 00:03:44,010
Do you see that as
the central equation,
60
00:03:44,010 --> 00:03:50,100
that's saying that this a
is perpendicular to this --
61
00:03:50,100 --> 00:03:53,700
correction, that's going
to tell us what x is.
62
00:03:53,700 --> 00:03:57,890
Let me just raise the board
and simplify that and out will
63
00:03:57,890 --> 00:03:59,790
come x.
64
00:03:59,790 --> 00:04:00,290
OK.
65
00:04:00,290 --> 00:04:06,540
So if I simplify that, let's
see, I'll move one to --
66
00:04:06,540 --> 00:04:09,580
one term to one side, the other
term will be on the other side,
67
00:04:09,580 --> 00:04:14,690
it looks to me like
x times a transpose a
68
00:04:14,690 --> 00:04:18,560
is equal to a transpose b.
69
00:04:21,370 --> 00:04:22,210
Right?
70
00:04:22,210 --> 00:04:25,240
I have a transpose b
as one f- one term,
71
00:04:25,240 --> 00:04:27,510
a transpose a as
the other, so right
72
00:04:27,510 --> 00:04:30,590
away here's my a transpose a.
73
00:04:30,590 --> 00:04:32,900
But it's just a number now.
74
00:04:32,900 --> 00:04:35,400
And I divide by it.
75
00:04:35,400 --> 00:04:42,200
And I get the answer. x is a
transpose b over a transpose a.
76
00:04:45,220 --> 00:04:48,320
And P, the projection
I wanted, is --
77
00:04:48,320 --> 00:04:50,530
that's the right multiple.
78
00:04:50,530 --> 00:04:52,720
That's got a cosine
theta built in.
79
00:04:52,720 --> 00:04:55,560
But we don't need
to look at angles.
80
00:04:55,560 --> 00:04:58,040
It's -- we've just
got vectors here.
81
00:04:58,040 --> 00:05:02,140
And the projection is
P is a times that x.
82
00:05:07,450 --> 00:05:08,349
Or x times that a.
83
00:05:08,349 --> 00:05:10,640
But I'm really going to --
eventually I'm going to want
84
00:05:10,640 --> 00:05:13,380
that x coming on
the right-hand side.
85
00:05:13,380 --> 00:05:20,630
So do you see that I've got two
of the three formulas already,
86
00:05:20,630 --> 00:05:24,750
right here, I've got the --
87
00:05:24,750 --> 00:05:29,140
that's the equation -- that
leads me to the answer,
88
00:05:29,140 --> 00:05:34,230
here's the answer for x,
and here's the projection.
89
00:05:34,230 --> 00:05:34,870
OK.
90
00:05:34,870 --> 00:05:37,520
can I do add just
one more thing to
91
00:05:37,520 --> 00:05:39,880
this one-dimensional problem?
92
00:05:39,880 --> 00:05:45,150
One more like lift it up into
linear algebra, into matrices.
93
00:05:48,460 --> 00:05:50,260
Here's the last
thing I want to do --
94
00:05:50,260 --> 00:05:52,120
but don't forget those formulas.
95
00:05:52,120 --> 00:05:54,760
a transpose b over a transpose
96
00:05:54,760 --> 00:05:55,680
a.
97
00:05:55,680 --> 00:06:06,020
Actually let's look at
that for a moment first.
98
00:06:06,020 --> 00:06:09,390
Suppose -- Let me
take this next step.
99
00:06:09,390 --> 00:06:11,490
So P is a times x.
100
00:06:11,490 --> 00:06:13,170
So can I write that then?
101
00:06:13,170 --> 00:06:20,800
P is a times this neat number, a
transpose b over a transpose a.
102
00:06:24,130 --> 00:06:26,690
That's our projection.
103
00:06:26,690 --> 00:06:30,060
Can I ask a couple of questions
about it, just while we look,
104
00:06:30,060 --> 00:06:32,510
get that digest that formula.
105
00:06:32,510 --> 00:06:37,020
Suppose b is doubled.
106
00:06:37,020 --> 00:06:41,600
Suppose I change b to two b.
107
00:06:41,600 --> 00:06:42,935
What happens to the projection?
108
00:06:46,200 --> 00:06:48,950
So suppose I instead of
that vector b that I drew
109
00:06:48,950 --> 00:06:51,920
on the board make it
two b, twice as long --
110
00:06:51,920 --> 00:06:55,540
what's the projection now?
111
00:06:55,540 --> 00:06:57,550
It's doubled too, right?
112
00:06:57,550 --> 00:07:01,060
It's going to be twice as
far, if b goes twice as far,
113
00:07:01,060 --> 00:07:03,030
the projection will
go twice as far.
114
00:07:03,030 --> 00:07:04,160
And you see it there.
115
00:07:04,160 --> 00:07:09,180
If I put in an extra factor
two, then P's got that factor
116
00:07:09,180 --> 00:07:09,840
too.
117
00:07:09,840 --> 00:07:14,040
Now what about if I double a?
118
00:07:14,040 --> 00:07:19,850
What if I double the vector
a that I'm projecting onto?
119
00:07:19,850 --> 00:07:23,220
What changes?
120
00:07:23,220 --> 00:07:25,320
The projection
doesn't change at all.
121
00:07:25,320 --> 00:07:25,820
Right?
122
00:07:25,820 --> 00:07:28,890
Because I'm just --
the line didn't change.
123
00:07:28,890 --> 00:07:31,620
If I double a or I take minus a.
124
00:07:31,620 --> 00:07:33,430
It's still that same line.
125
00:07:33,430 --> 00:07:35,310
The projection's still
in the same place.
126
00:07:35,310 --> 00:07:40,540
And of course if I double
a I get a four up above,
127
00:07:40,540 --> 00:07:43,600
and I get a four -- an extra
four below, they cancel out,
128
00:07:43,600 --> 00:07:45,580
and the projection is the same.
129
00:07:45,580 --> 00:07:46,350
OK.
130
00:07:46,350 --> 00:07:48,970
So really, this --
131
00:07:48,970 --> 00:07:54,750
I want to look at this
as the projection --
132
00:07:54,750 --> 00:07:57,630
there's a matrix here.
133
00:07:57,630 --> 00:08:01,960
The projection is carried
out by some matrix
134
00:08:01,960 --> 00:08:05,090
that I'm going to call
the projection matrix.
135
00:08:05,090 --> 00:08:07,970
And in other words
the projection
136
00:08:07,970 --> 00:08:12,240
is some matrix that
acts on this guy b
137
00:08:12,240 --> 00:08:14,100
and produces the projection.
138
00:08:14,100 --> 00:08:21,270
The projection P is the
projection matrix acting
139
00:08:21,270 --> 00:08:24,230
on whatever the input is.
140
00:08:24,230 --> 00:08:28,130
The input is b, the
projection matrix is P. OK.
141
00:08:28,130 --> 00:08:30,850
Actually you can tell
me right away what
142
00:08:30,850 --> 00:08:31,970
this projection matrix
143
00:08:31,970 --> 00:08:32,470
is.
144
00:08:32,470 --> 00:08:37,530
So this is a pretty
interesting matrix.
145
00:08:37,530 --> 00:08:40,650
What matrix is multiplying b?
146
00:08:40,650 --> 00:08:42,690
I'm just -- just
from my formula --
147
00:08:42,690 --> 00:08:44,120
I see what P is.
148
00:08:44,120 --> 00:08:51,650
P, this projection matrix, is --
149
00:08:51,650 --> 00:08:52,360
what is it?
150
00:08:52,360 --> 00:08:59,515
I see a a transpose above,
and I see a transpose a below.
151
00:09:02,470 --> 00:09:04,560
And those don't cancel.
152
00:09:04,560 --> 00:09:06,620
That's not one.
153
00:09:06,620 --> 00:09:07,260
Right?
154
00:09:07,260 --> 00:09:08,230
That's a matrix.
155
00:09:08,230 --> 00:09:10,580
Because down here,
the a transpose a,
156
00:09:10,580 --> 00:09:13,690
that's just a number,
a transpose a, that's
157
00:09:13,690 --> 00:09:18,060
the length of a squared, and
up above is a column times
158
00:09:18,060 --> 00:09:19,040
a row.
159
00:09:19,040 --> 00:09:22,050
Column times a row is a matrix.
160
00:09:22,050 --> 00:09:26,450
So this is a full-scale
n by n matrix, if I --
161
00:09:26,450 --> 00:09:28,880
if I'm in n dimensions.
162
00:09:28,880 --> 00:09:32,270
And it's kind of
an interesting one.
163
00:09:32,270 --> 00:09:36,510
And it's the one which if I
multiply by b then I get this,
164
00:09:36,510 --> 00:09:39,750
you see once again I'm
putting parentheses
165
00:09:39,750 --> 00:09:41,140
in different places.
166
00:09:41,140 --> 00:09:46,480
I'm putting the
parentheses right there.
167
00:09:46,480 --> 00:09:49,280
I'm saying OK, that's
really the matrix
168
00:09:49,280 --> 00:09:55,990
that produces this projection.
169
00:09:55,990 --> 00:09:56,980
OK.
170
00:09:56,980 --> 00:10:00,280
Now, tell me --
171
00:10:00,280 --> 00:10:02,910
all right, what are the
properties of that matrix?
172
00:10:05,420 --> 00:10:08,900
I'm just using letters here, a
and b, I could put in numbers,
173
00:10:08,900 --> 00:10:12,100
but I think it's -- for once,
it's clearer with letters,
174
00:10:12,100 --> 00:10:16,790
because all formulas are simple,
a transpose b over a transpose
175
00:10:16,790 --> 00:10:21,770
a -- that's the number that
multiplies the a, and then I
176
00:10:21,770 --> 00:10:26,160
see wait a minute, there's a
matrix here and what's the rank
177
00:10:26,160 --> 00:10:29,260
of that matrix, by the way?
178
00:10:29,260 --> 00:10:31,360
What's the rank of
that matrix, yeah --
179
00:10:31,360 --> 00:10:33,260
let me just ask you
about that matrix.
180
00:10:33,260 --> 00:10:38,250
Which looks a little strange,
a a transpose over this number.
181
00:10:38,250 --> 00:10:46,180
But well, I could ask
you its column space.
182
00:10:46,180 --> 00:10:48,920
Yeah, let me ask you
its column space.
183
00:10:48,920 --> 00:10:52,180
So what's the column
space of a matrix?
184
00:10:52,180 --> 00:10:56,460
If you multiply that
matrix by anything
185
00:10:56,460 --> 00:10:59,810
you always get in the
column space, right?
186
00:10:59,810 --> 00:11:04,900
The column space of a matrix
is when you multiply any vector
187
00:11:04,900 --> 00:11:09,900
by that matrix -- any
vector b, by the matrix,
188
00:11:09,900 --> 00:11:13,360
you always land in
the column space.
189
00:11:13,360 --> 00:11:16,320
That's what column spaces work.
190
00:11:16,320 --> 00:11:20,060
Now what space do
we always land in?
191
00:11:20,060 --> 00:11:22,110
What's the column space of --
192
00:11:22,110 --> 00:11:26,160
what's the result when I
multiply this any vector
193
00:11:26,160 --> 00:11:29,550
b by my matrix?
194
00:11:29,550 --> 00:11:34,710
So I have P times b, where I?
195
00:11:34,710 --> 00:11:36,170
I'm on that line, right?
196
00:11:36,170 --> 00:11:40,610
The column space, so here
are facts about this matrix.
197
00:11:40,610 --> 00:11:46,220
The column space of P, of
this projection matrix,
198
00:11:46,220 --> 00:11:49,190
is the line through a.
199
00:11:52,620 --> 00:12:03,720
And the rank of this matrix is
you can all say it at once one.
200
00:12:03,720 --> 00:12:04,520
Right.
201
00:12:04,520 --> 00:12:06,590
The rank is one.
202
00:12:06,590 --> 00:12:08,300
This is a rank one matrix.
203
00:12:08,300 --> 00:12:10,960
Actually it's exactly
the form that we're
204
00:12:10,960 --> 00:12:15,130
familiar with a rank one matrix.
205
00:12:15,130 --> 00:12:20,840
A column times a row, that's
a rank one matrix, that column
206
00:12:20,840 --> 00:12:27,400
is the basis for
the column space.
207
00:12:27,400 --> 00:12:28,770
Just one dimension.
208
00:12:28,770 --> 00:12:29,550
OK.
209
00:12:29,550 --> 00:12:32,560
So I know that much
about the matrix.
210
00:12:32,560 --> 00:12:35,490
But now there are two more
facts about the matrix
211
00:12:35,490 --> 00:12:40,230
that I want to notice.
212
00:12:40,230 --> 00:12:42,980
First of all is the
matrix symmetric?
213
00:12:42,980 --> 00:12:45,580
That's a natural
question for matrices.
214
00:12:45,580 --> 00:12:48,380
And the answer is yes.
215
00:12:48,380 --> 00:12:52,620
If I take the transpose of this
-- there's a number down there,
216
00:12:52,620 --> 00:12:57,300
the transpose of a a
transpose is a a transpose.
217
00:12:57,300 --> 00:12:59,200
So P is symmetric.
218
00:12:59,200 --> 00:13:04,500
P transpose equals P. So
this is a key property.
219
00:13:04,500 --> 00:13:07,730
That the projection
matrix is symmetric.
220
00:13:07,730 --> 00:13:12,430
One more property now
and this is the real one.
221
00:13:12,430 --> 00:13:18,570
What happens if I do
the projection twice?
222
00:13:18,570 --> 00:13:21,740
So I'm looking for
something, some information
223
00:13:21,740 --> 00:13:24,220
about P squared.
224
00:13:24,220 --> 00:13:27,530
But just give me in terms
of that picture, in terms
225
00:13:27,530 --> 00:13:33,940
my picture, I take any
vector b, I multiply it
226
00:13:33,940 --> 00:13:36,280
by my projection
matrix, and that
227
00:13:36,280 --> 00:13:41,010
puts me there, so this is Pb.
228
00:13:41,010 --> 00:13:45,170
And now I project again.
229
00:13:45,170 --> 00:13:46,790
What happens now?
230
00:13:46,790 --> 00:13:49,620
What happens when I apply
the projection matrix
231
00:13:49,620 --> 00:13:51,880
a second time?
232
00:13:51,880 --> 00:13:56,160
To this, so I'm applying
it once brings me here
233
00:13:56,160 --> 00:14:01,420
and the second time
brings me I stay put.
234
00:14:01,420 --> 00:14:02,170
Right?
235
00:14:02,170 --> 00:14:05,490
The projection for a point
on this line the projection
236
00:14:05,490 --> 00:14:08,530
is right where it is.
237
00:14:08,530 --> 00:14:10,610
The projection is
the same point.
238
00:14:10,610 --> 00:14:16,340
So that means that
if I project twice,
239
00:14:16,340 --> 00:14:21,430
I get the same answer as I
did in the first projection.
240
00:14:21,430 --> 00:14:26,510
So those are the
two properties that
241
00:14:26,510 --> 00:14:30,630
tell me I'm looking at
a projection matrix.
242
00:14:30,630 --> 00:14:35,070
It's symmetric and
it's square is itself.
243
00:14:35,070 --> 00:14:37,500
Because if I project
a second time,
244
00:14:37,500 --> 00:14:40,740
it's the same result
as the first result.
245
00:14:40,740 --> 00:14:41,940
OK.
246
00:14:41,940 --> 00:14:49,120
So that's -- and then here's the
exact formula when I know what
247
00:14:49,120 --> 00:14:53,450
I'm projecting onto, that line
through a, then I know what P
248
00:14:53,450 --> 00:14:54,460
is.
249
00:14:54,460 --> 00:14:57,070
So do you see that I
have all the pieces here
250
00:14:57,070 --> 00:14:59,990
for projection on a line?
251
00:14:59,990 --> 00:15:04,880
Now, and those --
please remember those.
252
00:15:04,880 --> 00:15:07,330
So there are three
formulas to remember.
253
00:15:07,330 --> 00:15:11,650
The formula for x, the formula
for P, which is just ax,
254
00:15:11,650 --> 00:15:17,680
and then the formula for
capital P, which is the matrix.
255
00:15:17,680 --> 00:15:18,430
Good.
256
00:15:18,430 --> 00:15:18,970
Good.
257
00:15:18,970 --> 00:15:20,400
OK.
258
00:15:20,400 --> 00:15:26,780
Now I want to move
to more dimensions.
259
00:15:26,780 --> 00:15:30,610
So we're going to have three
formulas again but you'll have
260
00:15:30,610 --> 00:15:32,170
to --
261
00:15:32,170 --> 00:15:35,830
they'll be a little different
because we won't have a single
262
00:15:35,830 --> 00:15:40,990
line but -- a plane
or three-dimensional
263
00:15:40,990 --> 00:15:44,530
or a n-dimensional subspace.
264
00:15:44,530 --> 00:15:45,230
OK.
265
00:15:45,230 --> 00:15:49,960
So now I'll move to
the next question.
266
00:15:49,960 --> 00:16:04,020
Maybe -- let me say first
why I want this projection,
267
00:16:04,020 --> 00:16:06,200
and then we'll figure
out what it is,
268
00:16:06,200 --> 00:16:11,040
we'll go completely in parallel
there, and then we'll use it.
269
00:16:11,040 --> 00:16:14,420
OK, why do I want
this projection?
270
00:16:14,420 --> 00:16:17,083
Well, so why project?
271
00:16:21,600 --> 00:16:32,010
It's because I'm as I mentioned
last time this new chapter
272
00:16:32,010 --> 00:16:47,890
deals with equations Ax=b
may have no solution.
273
00:16:54,540 --> 00:16:57,380
So that's really my
problem, that I'm
274
00:16:57,380 --> 00:17:00,240
given a bunch of
equations probably too
275
00:17:00,240 --> 00:17:04,450
many equations, more
equations than unknowns,
276
00:17:04,450 --> 00:17:08,119
and I can't solve them.
277
00:17:08,119 --> 00:17:09,079
OK.
278
00:17:09,079 --> 00:17:11,050
So what do I do?
279
00:17:11,050 --> 00:17:15,359
I solve the closest
problem that I can solve.
280
00:17:15,359 --> 00:17:17,730
And what's the closest one?
281
00:17:17,730 --> 00:17:21,460
Well, ax will always be
in the column space of a.
282
00:17:21,460 --> 00:17:22,970
That's my problem.
283
00:17:22,970 --> 00:17:27,190
My problem is ax has to
be in the column space
284
00:17:27,190 --> 00:17:31,130
and b is probably not
in the column space.
285
00:17:31,130 --> 00:17:36,620
So I change b to what?
286
00:17:36,620 --> 00:17:40,820
I choose the closest
vector in the column space,
287
00:17:40,820 --> 00:17:50,350
so I'll solve Ax
equal P instead.
288
00:17:50,350 --> 00:17:52,660
That one I can do.
289
00:17:52,660 --> 00:17:57,220
Where P is this
is the projection
290
00:17:57,220 --> 00:18:04,460
of b onto the column space.
291
00:18:04,460 --> 00:18:07,010
That's why I want to
be able to do this.
292
00:18:07,010 --> 00:18:10,460
Because I have to
find a solution here,
293
00:18:10,460 --> 00:18:13,370
and I'm going to put
a little hat there
294
00:18:13,370 --> 00:18:17,400
to indicate that
it's not the x, it's
295
00:18:17,400 --> 00:18:21,660
not the x that
doesn't exist, it's
296
00:18:21,660 --> 00:18:27,000
the x hat that's best possible.
297
00:18:27,000 --> 00:18:32,170
So I must be able
to figure out what's
298
00:18:32,170 --> 00:18:35,310
the good projection there.
299
00:18:35,310 --> 00:18:38,600
What's the good right-hand side
that is in the column space
300
00:18:38,600 --> 00:18:42,330
that's as close as possible
to b and then I'm --
301
00:18:42,330 --> 00:18:43,820
then I know what to do.
302
00:18:43,820 --> 00:18:44,620
OK.
303
00:18:44,620 --> 00:18:47,500
So now I've got that problem.
304
00:18:47,500 --> 00:18:50,190
So that's why I have
the problem again
305
00:18:50,190 --> 00:18:54,050
but now let me say I'm
in three dimensions,
306
00:18:54,050 --> 00:18:58,630
so I have a plane
maybe for example,
307
00:18:58,630 --> 00:19:06,390
and I have a vector b
that's not in the plane.
308
00:19:06,390 --> 00:19:10,840
And I want to project
b down into the plane.
309
00:19:13,420 --> 00:19:15,510
OK.
310
00:19:15,510 --> 00:19:17,540
So there's my question.
311
00:19:17,540 --> 00:19:19,890
How do I project a
vector and I'm --
312
00:19:19,890 --> 00:19:22,670
what I'm looking for
is a nice formula,
313
00:19:22,670 --> 00:19:28,120
and I'm counting on linear
algebra to just come out right,
314
00:19:28,120 --> 00:19:34,940
a nice formula for the
projection of b into the plane.
315
00:19:34,940 --> 00:19:36,310
The nearest point.
316
00:19:36,310 --> 00:19:41,990
So this again a right angle
is going to be crucial.
317
00:19:41,990 --> 00:19:42,570
OK.
318
00:19:42,570 --> 00:19:46,630
Now so what's -- first of all
I've got to say what is that
319
00:19:46,630 --> 00:19:48,100
plane.
320
00:19:48,100 --> 00:19:51,460
To get a formula I have to
tell you what the plane is.
321
00:19:51,460 --> 00:19:53,990
How I going to tell you a plane?
322
00:19:53,990 --> 00:19:56,580
I'll tell you a
basis for the plane,
323
00:19:56,580 --> 00:20:03,476
I'll tell you two vectors a one
and a two that give you a basis
324
00:20:03,476 --> 00:20:05,100
for the plane, so
that -- let us say --
325
00:20:05,100 --> 00:20:12,510
say there's an a one and
here's an a -- a vector a two.
326
00:20:12,510 --> 00:20:15,690
They don't have to
be perpendicular.
327
00:20:15,690 --> 00:20:18,921
But they better be independent,
because then that tells me the
328
00:20:18,921 --> 00:20:19,420
plane.
329
00:20:19,420 --> 00:20:24,880
The plane is the -- is the
plane of a one and a two.
330
00:20:31,760 --> 00:20:34,320
And actually going back to my --
331
00:20:34,320 --> 00:20:39,620
to this connection, this
plane is a column space,
332
00:20:39,620 --> 00:20:47,990
it's the column
space of what matrix?
333
00:20:47,990 --> 00:20:56,610
What matrix, so how do I
connect the two questions?
334
00:20:56,610 --> 00:21:01,220
I'm thinking how do I
project onto a plane
335
00:21:01,220 --> 00:21:06,030
and I want to get
a matrix in here.
336
00:21:06,030 --> 00:21:09,760
Everything's cleaner if I
write it in terms of a matrix.
337
00:21:09,760 --> 00:21:14,680
So what matrix has these
-- has that column space?
338
00:21:14,680 --> 00:21:17,270
Well of course it's
just the matrix
339
00:21:17,270 --> 00:21:20,240
that has a one in
the first column
340
00:21:20,240 --> 00:21:23,790
and a two in the second column.
341
00:21:23,790 --> 00:21:27,820
Right, just just let's be
sure we've got the question
342
00:21:27,820 --> 00:21:30,060
before we get to the answer.
343
00:21:30,060 --> 00:21:34,290
So I'm looking for
again I'm given
344
00:21:34,290 --> 00:21:38,140
a matrix a with two columns.
345
00:21:38,140 --> 00:21:44,110
And really I'm ready once I
get to two I'm ready for n.
346
00:21:44,110 --> 00:21:47,610
So it could be two columns,
it could be n columns.
347
00:21:47,610 --> 00:21:50,930
I'll write the answer in
terms of the matrix a.
348
00:21:50,930 --> 00:21:56,590
And the point will be those
two columns describe the plane,
349
00:21:56,590 --> 00:22:01,331
they describe the column
space, and I want to project.
350
00:22:01,331 --> 00:22:01,830
OK.
351
00:22:01,830 --> 00:22:06,190
And I'm given a vector b that's
probably not in the column
352
00:22:06,190 --> 00:22:06,770
space.
353
00:22:06,770 --> 00:22:11,280
Of course, if b is in the column
space, my projection is simple,
354
00:22:11,280 --> 00:22:13,420
it's just b.
355
00:22:13,420 --> 00:22:19,680
But most likely I have an
error e, this b minus P
356
00:22:19,680 --> 00:22:24,500
part, which is
probably not zero.
357
00:22:24,500 --> 00:22:25,560
OK.
358
00:22:25,560 --> 00:22:30,330
But the beauty is that I know --
359
00:22:30,330 --> 00:22:35,880
from geometry or I could get it
from calculus or I could get it
360
00:22:35,880 --> 00:22:38,724
from linear algebra that
that this this vector --
361
00:22:42,600 --> 00:22:45,830
this is the part
of b that's that's
362
00:22:45,830 --> 00:22:48,600
perpendicular to the plane.
363
00:22:48,600 --> 00:22:55,390
That e is perpendicular is
perpendicular to the plane.
364
00:22:58,030 --> 00:23:03,530
If your intuition is saying
that that's the crucial fact.
365
00:23:03,530 --> 00:23:06,900
That's going to
give us the answer.
366
00:23:06,900 --> 00:23:07,540
OK.
367
00:23:07,540 --> 00:23:10,320
So let me, that's the problem.
368
00:23:10,320 --> 00:23:14,550
Now for the answer.
369
00:23:14,550 --> 00:23:18,910
So this is a lecture that's
really like moving along.
370
00:23:18,910 --> 00:23:24,940
Because I'm just plotting that
problem up there and asking you
371
00:23:24,940 --> 00:23:27,130
what combination --
372
00:23:27,130 --> 00:23:29,920
now, yeah, so what is it?
373
00:23:29,920 --> 00:23:31,110
What is this projection P?
374
00:23:31,110 --> 00:23:42,300
P. This is projection P, is
some multiple of these basis
375
00:23:42,300 --> 00:23:46,740
guys, right, some
multiple of the columns.
376
00:23:46,740 --> 00:23:51,700
But I don't like writing out
x one a one plus x two a two,
377
00:23:51,700 --> 00:23:54,265
I would rather right that as ax.
378
00:23:56,790 --> 00:24:00,512
Well, actually I should put
if I'm really doing everything
379
00:24:00,512 --> 00:24:02,220
right, I should put
a little hat on it --
380
00:24:02,220 --> 00:24:05,500
to remember that this x --
381
00:24:05,500 --> 00:24:08,190
that those are the numbers
and I could have a put
382
00:24:08,190 --> 00:24:14,470
a hat way back there
is right, so this
383
00:24:14,470 --> 00:24:20,360
is this is the projection,
P. P is ax bar.
384
00:24:20,360 --> 00:24:22,830
And I'm looking for x bar.
385
00:24:22,830 --> 00:24:25,195
So that's what I
want an equation for.
386
00:24:28,410 --> 00:24:31,630
So now I've got
hold of the problem.
387
00:24:31,630 --> 00:24:36,330
The problem is find the right
combination of the columns
388
00:24:36,330 --> 00:24:44,070
so that the error vector is
perpendicular to the plane.
389
00:24:44,070 --> 00:24:48,840
Now let me turn that
into an equation.
390
00:24:48,840 --> 00:24:53,250
So I'll raise the board
and just turn that --
391
00:24:53,250 --> 00:24:54,910
what we've just done
into an equation.
392
00:24:58,490 --> 00:25:02,020
So let me I'll write
again the main point.
393
00:25:02,020 --> 00:25:06,600
The projection is ax b- x hat.
394
00:25:06,600 --> 00:25:08,770
And our problem is find x hat.
395
00:25:12,670 --> 00:25:23,060
And the key is that b minus
ax hat, that's the error.
396
00:25:23,060 --> 00:25:24,800
This is the e.
397
00:25:24,800 --> 00:25:30,390
Is perpendicular to the plane.
398
00:25:35,050 --> 00:25:38,490
That's got to give me
well what I looking
399
00:25:38,490 --> 00:25:40,720
for, I'm looking for
two equations now
400
00:25:40,720 --> 00:25:42,845
because I've got an
x one and an x two.
401
00:25:45,640 --> 00:25:49,910
And I'll get two equations
because so this thing e
402
00:25:49,910 --> 00:25:51,680
is perpendicular to the plane.
403
00:25:55,470 --> 00:25:56,650
So what does that mean?
404
00:25:56,650 --> 00:26:00,060
I guess it means it's
perpendicular to a one
405
00:26:00,060 --> 00:26:02,540
and also to a two.
406
00:26:02,540 --> 00:26:06,320
Right, those are two vectors
in the plane and the things
407
00:26:06,320 --> 00:26:08,240
that are perpendicular
to the plane
408
00:26:08,240 --> 00:26:10,810
are perpendicular
to a one and a two.
409
00:26:10,810 --> 00:26:12,350
Let me just repeat.
410
00:26:12,350 --> 00:26:15,480
This this guy then is
perpendicular to the plane
411
00:26:15,480 --> 00:26:18,460
so it's perpendicular to
that vector and that vector.
412
00:26:18,460 --> 00:26:22,000
Not -- it's perpendicular
to that of course.
413
00:26:22,000 --> 00:26:25,940
But it's perpendicular to
everything I the plane.
414
00:26:25,940 --> 00:26:31,500
And the plane is really
told me by a one and a two.
415
00:26:31,500 --> 00:26:39,150
So really I have the equations
a one transpose b minus ax
416
00:26:39,150 --> 00:26:41,050
is zero.
417
00:26:41,050 --> 00:26:48,130
And also a two transpose
b minus ax is zero.
418
00:26:52,321 --> 00:26:53,445
Those are my two equations.
419
00:26:56,950 --> 00:27:02,410
But I want those in matrix form.
420
00:27:02,410 --> 00:27:04,430
I want to put those
two equations together
421
00:27:04,430 --> 00:27:08,600
as a matrix equation and
it's just comes out right.
422
00:27:08,600 --> 00:27:12,100
Look at the matrix a transpose.
423
00:27:12,100 --> 00:27:16,050
Put a one a one transpose
is its first row,
424
00:27:16,050 --> 00:27:25,770
a two transpose is its second
row, that multiplies this b-ax,
425
00:27:25,770 --> 00:27:28,440
and gives me the
zero and the zero.
426
00:27:28,440 --> 00:27:29,410
I'm you see the --
427
00:27:39,220 --> 00:27:43,050
this is one way -- to come
up with this equation.
428
00:27:43,050 --> 00:27:44,600
So the equation
I'm coming up with
429
00:27:44,600 --> 00:27:52,550
is a transpose b-ax hat is zero.
430
00:27:55,250 --> 00:27:57,520
OK.
431
00:27:57,520 --> 00:27:59,791
That's my equation.
432
00:27:59,791 --> 00:28:00,290
All right.
433
00:28:00,290 --> 00:28:03,790
Now I want to stop for a
moment before I solve it
434
00:28:03,790 --> 00:28:06,280
and just think about it.
435
00:28:06,280 --> 00:28:13,230
First of all do you see that
that equation back in the very
436
00:28:13,230 --> 00:28:17,440
first problem I solved
on a line, what was --
437
00:28:17,440 --> 00:28:23,610
what was on a line the
matrix a only had one column,
438
00:28:23,610 --> 00:28:27,210
it was just little a.
439
00:28:27,210 --> 00:28:29,980
So in the first
problem I solved,
440
00:28:29,980 --> 00:28:33,680
projecting on a line,
this for capital
441
00:28:33,680 --> 00:28:35,510
a you just change
that to little a
442
00:28:35,510 --> 00:28:39,160
and you have the same equation
that we solved before.
443
00:28:39,160 --> 00:28:42,670
a transpose e equals zero.
444
00:28:42,670 --> 00:28:43,300
OK.
445
00:28:43,300 --> 00:28:48,500
Now a second thing,
second comment.
446
00:28:48,500 --> 00:28:54,220
I would like to since I know
about these four subspaces,
447
00:28:54,220 --> 00:28:57,720
I would like to get
them into this picture.
448
00:29:00,890 --> 00:29:07,270
So let me ask the question,
what subspace is this thing in?
449
00:29:07,270 --> 00:29:11,330
Which of the four subspaces
is that error vector e,
450
00:29:11,330 --> 00:29:13,920
this is this is nothing but e --
451
00:29:13,920 --> 00:29:21,204
this is this guy, coming in
down perpendicular to the plane.
452
00:29:21,204 --> 00:29:22,120
What subspace is e in?
453
00:29:22,120 --> 00:29:22,911
From this equation.
454
00:29:22,911 --> 00:29:34,470
Well the equation is saying
a transpose e is zero.
455
00:29:34,470 --> 00:29:39,630
So I'm learning here that
e is in the null space
456
00:29:39,630 --> 00:29:40,330
of a transpose.
457
00:29:44,660 --> 00:29:45,360
Right?
458
00:29:45,360 --> 00:29:47,000
That's my equation.
459
00:29:47,000 --> 00:29:51,300
And now I just want to see hey
of course that that was right.
460
00:29:51,300 --> 00:29:57,320
Because things that are in
the null space of a transpose,
461
00:29:57,320 --> 00:30:00,310
what do we know about the
null space of a transpose?
462
00:30:03,170 --> 00:30:06,320
So that last lecture
gave us the sort
463
00:30:06,320 --> 00:30:09,210
of the geometry of
these subspaces.
464
00:30:09,210 --> 00:30:11,200
And the orthogonality of them.
465
00:30:11,200 --> 00:30:12,950
And do you remember what it was?
466
00:30:12,950 --> 00:30:16,910
What on the right
side of our big figure
467
00:30:16,910 --> 00:30:20,200
we always have the null
space of a transpose
468
00:30:20,200 --> 00:30:27,270
and the column space of
a, and they're orthogonal.
469
00:30:27,270 --> 00:30:31,750
So e in the null
space of a transpose
470
00:30:31,750 --> 00:30:38,730
is saying e is perpendicular
to the column space of a.
471
00:30:38,730 --> 00:30:39,230
Yes.
472
00:30:44,850 --> 00:30:49,100
I just feel OK, the damn
thing came out right.
473
00:30:49,100 --> 00:30:58,140
The equation for the equation
that I struggled to find for e
474
00:30:58,140 --> 00:31:03,980
really said what I
wanted, that the error
475
00:31:03,980 --> 00:31:08,290
e is perpendicular to the
column space of a, just right.
476
00:31:08,290 --> 00:31:10,710
And from our four
fundamental subspaces
477
00:31:10,710 --> 00:31:14,780
we knew that that
is the same as that.
478
00:31:14,780 --> 00:31:17,530
To say e is in the null
space of a transpose says
479
00:31:17,530 --> 00:31:19,451
e's perpendicular
to the column space.
480
00:31:19,451 --> 00:31:19,950
OK.
481
00:31:19,950 --> 00:31:21,900
So we've got this equation.
482
00:31:21,900 --> 00:31:23,360
Now let's just solve it.
483
00:31:23,360 --> 00:31:23,860
All right.
484
00:31:23,860 --> 00:31:28,130
Let me just rewrite
it as a transpose
485
00:31:28,130 --> 00:31:34,100
a x hat equals a transpose b.
486
00:31:34,100 --> 00:31:37,700
That's our equation.
487
00:31:37,700 --> 00:31:42,500
That gives us x.
488
00:31:42,500 --> 00:31:48,140
And -- allow me to keep
remembering the one-dimensional
489
00:31:48,140 --> 00:31:48,970
case.
490
00:31:48,970 --> 00:31:54,260
The one-dimensional
case, this was little a.
491
00:31:54,260 --> 00:31:56,620
So this was just a number,
little a transpose,
492
00:31:56,620 --> 00:32:02,910
a transpose a was just a vector
row times a column, a number.
493
00:32:02,910 --> 00:32:04,560
And this was a number.
494
00:32:04,560 --> 00:32:07,480
And x was the ratio
of those numbers.
495
00:32:07,480 --> 00:32:12,350
But now we've got matrices,
this one is n by n.
496
00:32:12,350 --> 00:32:15,211
a transpose a is
an n by n matrix.
497
00:32:15,211 --> 00:32:15,710
OK.
498
00:32:15,710 --> 00:32:20,260
So can I move to the next
board for the solution?
499
00:32:26,260 --> 00:32:27,470
OK.
500
00:32:27,470 --> 00:32:30,900
This is the -- the key equation.
501
00:32:30,900 --> 00:32:35,160
Now I'm ready for the formulas
that we have to remember.
502
00:32:35,160 --> 00:32:38,010
What's x hat?
503
00:32:38,010 --> 00:32:41,570
What's the projection,
what's the projection matrix,
504
00:32:41,570 --> 00:32:43,620
those are my three questions.
505
00:32:43,620 --> 00:32:46,050
That we answered in
the 1-D case and now
506
00:32:46,050 --> 00:32:49,720
we're ready for in the
n-dimensional case.
507
00:32:49,720 --> 00:32:50,920
So what is x hat?
508
00:32:50,920 --> 00:32:55,930
Well, what can I
say but a transpose
509
00:32:55,930 --> 00:33:01,075
a inverse, a transpose b.
510
00:33:03,750 --> 00:33:07,840
That's the solution
to -- to our equation.
511
00:33:07,840 --> 00:33:08,890
OK.
512
00:33:08,890 --> 00:33:09,940
What's the projection?
513
00:33:09,940 --> 00:33:11,430
That's more interesting.
514
00:33:11,430 --> 00:33:13,500
What's the projection?
515
00:33:13,500 --> 00:33:18,520
The projection is a x hat.
516
00:33:18,520 --> 00:33:21,700
That's how x hat got into the
picture in the first place.
517
00:33:21,700 --> 00:33:28,050
x hat was the was the
combination of columns
518
00:33:28,050 --> 00:33:31,440
in the I had to look for those
numbers and now I found them.
519
00:33:31,440 --> 00:33:34,020
Was the combination
of the columns of a
520
00:33:34,020 --> 00:33:35,510
that gave me the projection.
521
00:33:35,510 --> 00:33:37,060
OK.
522
00:33:37,060 --> 00:33:40,000
So now I know what this guy is.
523
00:33:40,000 --> 00:33:42,260
So it's just I multiply by a.
524
00:33:42,260 --> 00:33:50,110
a a transpose a
inverse a transpose b.
525
00:33:56,010 --> 00:34:00,145
That's looking a little
messy but it's not bad.
526
00:34:03,270 --> 00:34:07,600
That that combination is is
our like magic combination.
527
00:34:07,600 --> 00:34:14,860
This is the thing which is which
use which is like what's it
528
00:34:14,860 --> 00:34:16,504
like, what was it
in one dimension?
529
00:34:19,130 --> 00:34:20,820
What was that we
had this we must
530
00:34:20,820 --> 00:34:25,139
have had this thing way back at
the beginning of the lecture.
531
00:34:25,139 --> 00:34:29,929
What did we -- oh that a was
just a column so it was little
532
00:34:29,929 --> 00:34:36,210
a, little a transpose
over a transpose a, right,
533
00:34:36,210 --> 00:34:46,110
that's what it was in 1-D. You
see what's happened in more
534
00:34:46,110 --> 00:34:47,520
dimensions, I --
535
00:34:47,520 --> 00:34:49,810
I'm not allowed
to to just divide
536
00:34:49,810 --> 00:34:52,440
because because I don't have a
number, I have to put inverse,
537
00:34:52,440 --> 00:34:55,320
because I have an n by n matrix.
538
00:34:55,320 --> 00:34:56,440
But same formula.
539
00:34:59,120 --> 00:35:03,320
And now tell me what's
the projection matrix?
540
00:35:03,320 --> 00:35:08,955
What matrix is multiplying
b to give the projection?
541
00:35:12,300 --> 00:35:13,660
Right there.
542
00:35:13,660 --> 00:35:15,010
Because there it --
543
00:35:15,010 --> 00:35:17,390
I even already underlined
it by accident.
544
00:35:17,390 --> 00:35:24,100
The projection matrix which
I use capital P is this,
545
00:35:24,100 --> 00:35:28,360
it's it's that thing, shall
I write it again, a times
546
00:35:28,360 --> 00:35:32,770
a transpose a inverse
times a transpose.
547
00:35:41,610 --> 00:35:49,080
Now if you'll bear with me I'll
think about what have I done
548
00:35:49,080 --> 00:35:49,640
here.
549
00:35:49,640 --> 00:35:53,030
I've got this formula.
550
00:35:53,030 --> 00:36:00,100
Now the first thing that
occurs to me is something bad.
551
00:36:00,100 --> 00:36:04,360
Look why don't I
just you know here's
552
00:36:04,360 --> 00:36:08,100
a product of two matrices
and I want its inverse,
553
00:36:08,100 --> 00:36:10,460
why don't I just
use the formula I
554
00:36:10,460 --> 00:36:14,030
know for the inverse of
a product and say OK,
555
00:36:14,030 --> 00:36:22,110
that's a inverse times a
transpose inverse, what
556
00:36:22,110 --> 00:36:25,480
will happen if I do that?
557
00:36:25,480 --> 00:36:29,370
What will happen
if I say hey this
558
00:36:29,370 --> 00:36:36,060
is a inverse times a transpose
inverse, then shall I do it?
559
00:36:36,060 --> 00:36:41,100
It's going to go on videotape
if I do it, and I don't --
560
00:36:41,100 --> 00:36:43,780
all right, I'll put it
there, but just like
561
00:36:43,780 --> 00:36:48,010
don't take the videotape
quite so carefully.
562
00:36:48,010 --> 00:36:48,770
OK.
563
00:36:48,770 --> 00:36:52,440
So if I put that thing it
-- it would be a a inverse
564
00:36:52,440 --> 00:36:59,387
a transpose inverse a
transpose and what's that?
565
00:36:59,387 --> 00:37:00,220
That's the identity.
566
00:37:03,610 --> 00:37:06,660
But what's going on?
567
00:37:06,660 --> 00:37:14,059
So why -- you see my question is
somehow I did something wrong.
568
00:37:14,059 --> 00:37:15,100
That that wasn't allowed.
569
00:37:15,100 --> 00:37:19,100
And and and why is that?
570
00:37:19,100 --> 00:37:23,530
Because a is not
a square matrix.
571
00:37:23,530 --> 00:37:25,060
a is not a square matrix.
572
00:37:25,060 --> 00:37:28,580
It doesn't have an inverse.
573
00:37:28,580 --> 00:37:31,850
So I have to leave it that way.
574
00:37:31,850 --> 00:37:33,310
This is not OK.
575
00:37:33,310 --> 00:37:36,950
If if a was a square
invertible matrix, then then
576
00:37:36,950 --> 00:37:37,800
I couldn't complain.
577
00:37:40,980 --> 00:37:44,280
Yeah I think -- let me
think about that case.
578
00:37:44,280 --> 00:37:46,740
But you but my main
case, the whole reason
579
00:37:46,740 --> 00:37:51,240
I'm doing all this, is
that a is this matrix that
580
00:37:51,240 --> 00:37:56,360
has x too many rows, it's
just got a couple of columns,
581
00:37:56,360 --> 00:38:01,280
like a one and a two,
but lots of rows.
582
00:38:01,280 --> 00:38:02,390
Not square.
583
00:38:02,390 --> 00:38:07,810
And if it's not square, this
a transpose a is square but I
584
00:38:07,810 --> 00:38:10,530
can't pull it apart like this --
585
00:38:10,530 --> 00:38:16,650
I'm not allowed to do this pull
apart, except if a was square.
586
00:38:16,650 --> 00:38:19,180
Now if a is square
what's what's going on
587
00:38:19,180 --> 00:38:20,780
if a is a square matrix?
588
00:38:20,780 --> 00:38:25,070
a nice square inv-
invertible matrix.
589
00:38:25,070 --> 00:38:26,290
Think.
590
00:38:26,290 --> 00:38:29,790
What's up with that
what's with that case.
591
00:38:29,790 --> 00:38:34,300
So this is that the formula
ought to work then too.
592
00:38:34,300 --> 00:38:38,910
If a is a nice square invertible
matrix what's its column space,
593
00:38:38,910 --> 00:38:44,070
so it's a nice n by n invertible
everything great matrix,
594
00:38:44,070 --> 00:38:50,670
what's its column
space, the whole of R^n.
595
00:38:50,670 --> 00:38:54,190
So what's the projection
matrix if I'm projecting
596
00:38:54,190 --> 00:38:56,900
onto the whole space?
597
00:38:56,900 --> 00:39:00,510
It's the identity matrix right?
598
00:39:00,510 --> 00:39:04,800
If I'm projecting b
onto the whole space,
599
00:39:04,800 --> 00:39:08,090
not just onto a plane,
but onto all of 3-D,
600
00:39:08,090 --> 00:39:11,550
then b is already
in the column space,
601
00:39:11,550 --> 00:39:15,930
the projection is the
identity, and this is gives me
602
00:39:15,930 --> 00:39:18,720
the correct formula, P is I.
603
00:39:18,720 --> 00:39:23,000
But if I'm projecting
onto a subspace then
604
00:39:23,000 --> 00:39:27,310
I can't split those apart and I
have to stay with that formula.
605
00:39:27,310 --> 00:39:30,040
OK.
606
00:39:30,040 --> 00:39:37,240
And what can I say if -- so I
remember this formula for 1-D
607
00:39:37,240 --> 00:39:40,760
and that's what it looks
like in n dimensions.
608
00:39:40,760 --> 00:39:44,350
And what are the properties that
I expected for any projection
609
00:39:44,350 --> 00:39:44,970
matrix?
610
00:39:44,970 --> 00:39:47,370
And I still expect for this one?
611
00:39:47,370 --> 00:39:50,170
That matrix should be
symmetric and it is.
612
00:39:50,170 --> 00:39:55,160
P transpose is P. Because
if I transpose this,
613
00:39:55,160 --> 00:39:59,280
this guy's symmetric, and
its inverse is symmetric,
614
00:39:59,280 --> 00:40:03,340
and if I transpose this
one when I transpose
615
00:40:03,340 --> 00:40:08,050
it will pop up there, become
a, that a transpose will pop up
616
00:40:08,050 --> 00:40:11,440
here, and I'm back to P again.
617
00:40:11,440 --> 00:40:14,720
And do we dare try
the other property
618
00:40:14,720 --> 00:40:18,182
which is P squared equal P?
619
00:40:24,790 --> 00:40:25,750
It's got to be right.
620
00:40:28,670 --> 00:40:33,540
Because we know geometrically
that the first projection pops
621
00:40:33,540 --> 00:40:37,090
us onto the column space and the
second one leaves us where we
622
00:40:37,090 --> 00:40:37,680
are.
623
00:40:37,680 --> 00:40:43,180
So I expect that if I
multiply by let me do it --
624
00:40:43,180 --> 00:40:48,030
if I multiply by another P,
so there's another a, another
625
00:40:48,030 --> 00:40:59,040
a transpose a inverse
a transpose, can you --
626
00:40:59,040 --> 00:41:03,960
eight (a)-s in a row
is quite obscene but --
627
00:41:03,960 --> 00:41:05,415
do you see that it works?
628
00:41:08,160 --> 00:41:11,240
So I'm squaring that so
what do I do-- how do I
629
00:41:11,240 --> 00:41:12,610
see that multiplication?
630
00:41:12,610 --> 00:41:17,480
Well, yeah, I just want to put
parentheses in good places,
631
00:41:17,480 --> 00:41:21,910
so I see what's happening, yeah,
here's an a transpose a sitting
632
00:41:21,910 --> 00:41:25,380
together -- so when that a
transpose a multiplies its
633
00:41:25,380 --> 00:41:29,840
inverse, all that
stuff goes, right.
634
00:41:29,840 --> 00:41:32,300
And leaves just the a
transpose at the end,
635
00:41:32,300 --> 00:41:35,480
which is just what we want.
636
00:41:35,480 --> 00:41:39,090
So P squared equals P.
So sure enough those two
637
00:41:39,090 --> 00:41:40,200
properties hold.
638
00:41:40,200 --> 00:41:40,730
OK.
639
00:41:40,730 --> 00:41:45,510
OK we really have got
now all the formulas.
640
00:41:45,510 --> 00:41:50,370
x hat, the projection P, and
the projection matrix capital
641
00:41:50,370 --> 00:41:58,710
P. And now my job
is to use them.
642
00:41:58,710 --> 00:41:59,210
OK.
643
00:41:59,210 --> 00:42:05,940
So when would I have
a bunch of equations,
644
00:42:05,940 --> 00:42:12,250
too many equations and yet I
want the best answer and the --
645
00:42:12,250 --> 00:42:19,210
the most important example,
the most common example is if I
646
00:42:19,210 --> 00:42:25,130
have points so here's the
-- here's the application.
647
00:42:25,130 --> 00:42:25,850
v squared.
648
00:42:25,850 --> 00:42:27,800
Fitting by a line.
649
00:42:27,800 --> 00:42:28,300
OK.
650
00:42:42,170 --> 00:42:46,700
So I'll start this application
today and there's more in it
651
00:42:46,700 --> 00:42:50,240
than I can do in
this same lecture.
652
00:42:50,240 --> 00:42:53,800
So that'll give me a chance
to recap the formulas
653
00:42:53,800 --> 00:42:59,910
and there they are,
and recap the ideas.
654
00:42:59,910 --> 00:43:03,050
So let me start
the problem today.
655
00:43:03,050 --> 00:43:10,500
I'm given a bunch
of data points.
656
00:43:10,500 --> 00:43:14,430
And they lie close to a
line but not on a line.
657
00:43:14,430 --> 00:43:15,660
Let me take that.
658
00:43:15,660 --> 00:43:19,960
Say a t equal to
one, two and three,
659
00:43:19,960 --> 00:43:25,740
I have one, and
two and two again.
660
00:43:25,740 --> 00:43:31,890
So my data points are this is
the like the time direction
661
00:43:31,890 --> 00:43:37,200
and this is like well let me
call that b or y or something.
662
00:43:37,200 --> 00:43:43,790
I'm given these three points and
I want to fit them by a line.
663
00:43:43,790 --> 00:43:46,130
By the best straight line.
664
00:43:46,130 --> 00:43:55,700
So the problem is fit the points
one, one is the first point --
665
00:43:55,700 --> 00:44:02,720
the second point is t
equals two, b equal one,
666
00:44:02,720 --> 00:44:06,735
and the third point is t
equal three, b equal to two.
667
00:44:09,570 --> 00:44:14,920
So those are my three points,
t equal sorry,that's two.
668
00:44:14,920 --> 00:44:17,400
Yeah, OK.
669
00:44:17,400 --> 00:44:19,380
So this is the point one, one.
670
00:44:19,380 --> 00:44:22,900
This is the point two, two, and
that's the point three, two.
671
00:44:22,900 --> 00:44:27,430
And of course there isn't a --
a line that goes through them.
672
00:44:27,430 --> 00:44:29,120
So I'm looking
for the best line.
673
00:44:29,120 --> 00:44:33,670
I'm looking for a line that
probably goes somewhere,
674
00:44:33,670 --> 00:44:38,080
do you think it goes
somewhere like that?
675
00:44:38,080 --> 00:44:41,300
I didn't mean to make it go
through that point, it won't.
676
00:44:41,300 --> 00:44:42,720
It'll kind of --
677
00:44:42,720 --> 00:44:47,180
it'll go between so the error
there and the error there
678
00:44:47,180 --> 00:44:54,780
and the error there are as
small as I can get them.
679
00:44:54,780 --> 00:45:00,200
OK, what I'd like to do
is find the matrix a.
680
00:45:00,200 --> 00:45:01,710
Because once I've
found the matrix
681
00:45:01,710 --> 00:45:05,030
a the formulas take over.
682
00:45:05,030 --> 00:45:11,110
So what I'm looking for
this line, b is C+Dt.
683
00:45:11,110 --> 00:45:16,970
So this is in the homework
that I sent out for today.
684
00:45:16,970 --> 00:45:18,310
Find the best line.
685
00:45:18,310 --> 00:45:21,170
So I'm looking
for these numbers.
686
00:45:21,170 --> 00:45:23,820
C and D.
687
00:45:23,820 --> 00:45:27,130
That tell me the
line and I want them
688
00:45:27,130 --> 00:45:30,560
to be as close to going
through those three points
689
00:45:30,560 --> 00:45:32,270
as I can get.
690
00:45:32,270 --> 00:45:35,320
I can't get exactly so
there are three equations
691
00:45:35,320 --> 00:45:37,870
to go through the three points.
692
00:45:37,870 --> 00:45:41,110
It would it will go
exactly through that point
693
00:45:41,110 --> 00:45:45,050
if let's see that first
point has t equal to one,
694
00:45:45,050 --> 00:45:48,710
so that would say C+D equaled 1.
695
00:45:48,710 --> 00:45:51,890
This is the one, one.
696
00:45:51,890 --> 00:45:55,250
The second point t is two.
697
00:45:55,250 --> 00:46:00,760
So C+2D should come
out to equal 2.
698
00:46:00,760 --> 00:46:05,190
But I also want to get the third
equation in and at that third
699
00:46:05,190 --> 00:46:11,400
equation t is three
so C+3D equals only 2.
700
00:46:17,400 --> 00:46:19,160
That's the key.
701
00:46:19,160 --> 00:46:23,450
Is to write down what equations
we would like to solve
702
00:46:23,450 --> 00:46:25,010
but can't.
703
00:46:25,010 --> 00:46:28,080
Reason we if we could solve them
that would mean that we could
704
00:46:28,080 --> 00:46:33,470
put a line through all
three points and of course
705
00:46:33,470 --> 00:46:39,100
if these numbers one, two, two
were different, we could do it.
706
00:46:39,100 --> 00:46:42,130
But with those numbers,
one, two, two, we can't.
707
00:46:42,130 --> 00:46:49,300
So what is our equation Ax equal
Ax equal b that we can't solve?
708
00:46:49,300 --> 00:46:52,840
I just want to say
what's the matrix here,
709
00:46:52,840 --> 00:46:56,090
what's the unknown x, and
what's the right-hand side.
710
00:46:56,090 --> 00:47:00,975
So this is the matrix is one,
one, one, one, two, three.
711
00:47:03,980 --> 00:47:07,650
The unknown is C and D.
712
00:47:07,650 --> 00:47:10,350
And the right-hand
side if one, two, two.
713
00:47:10,350 --> 00:47:18,080
Right, I've just
taken my equations
714
00:47:18,080 --> 00:47:28,810
and I've said what
is Ax and what is b.
715
00:47:28,810 --> 00:47:33,640
Then there's no solution, this
is the typical case where I
716
00:47:33,640 --> 00:47:36,860
have three equations --
two unknowns, no solution,
717
00:47:36,860 --> 00:47:40,640
but I'm still looking
for the best solution.
718
00:47:40,640 --> 00:47:45,780
And the best
solution is taken is
719
00:47:45,780 --> 00:47:49,390
is to solve not this
equation Ax equal
720
00:47:49,390 --> 00:47:55,960
b which has which has no
solution but the equation that
721
00:47:55,960 --> 00:48:01,350
does have a solution,
which was this one.
722
00:48:01,350 --> 00:48:03,720
So that's the equation to solve.
723
00:48:03,720 --> 00:48:06,110
That's the central
equation of the subject.
724
00:48:06,110 --> 00:48:12,540
I can't solve Ax=b but magically
when I multiply both sides
725
00:48:12,540 --> 00:48:22,280
by a transpose I get an equation
that I can solve and its
726
00:48:22,280 --> 00:48:29,770
solution gives me x, the
best x, the best projection,
727
00:48:29,770 --> 00:48:36,510
and I discover what's the
matrix that's behind it.
728
00:48:36,510 --> 00:48:37,260
OK.
729
00:48:37,260 --> 00:48:44,010
So next time I'll complete an
example, numerical example.
730
00:48:44,010 --> 00:48:49,250
today was all letters,
numbers next time.
731
00:48:49,250 --> 00:48:50,800
Thanks.