1
00:00:03,890 --> 00:00:09,130
Yes, OK, four, three,
two, one, OK, I
2
00:00:09,130 --> 00:00:10,880
see you guys are
in a happy mood.
3
00:00:10,880 --> 00:00:13,980
I don't know if that
means 18.06 is ending,
4
00:00:13,980 --> 00:00:16,680
or, the quiz was good.
5
00:00:16,680 --> 00:00:21,030
Uh, my birthday
conference was going
6
00:00:21,030 --> 00:00:24,880
on at the time of the quiz, and
in the conference, of course,
7
00:00:24,880 --> 00:00:27,190
everybody had to
say nice things,
8
00:00:27,190 --> 00:00:30,390
but I was wondering,
what would my 18.06
9
00:00:30,390 --> 00:00:36,200
class be saying, because it was
at the exactly the same time.
10
00:00:36,200 --> 00:00:39,300
But, what I know from
the grades so far,
11
00:00:39,300 --> 00:00:45,440
they're basically close to, and
maybe slightly above the grades
12
00:00:45,440 --> 00:00:48,590
that you got on quiz two.
13
00:00:48,590 --> 00:00:52,790
So, very satisfactory.
14
00:00:52,790 --> 00:00:56,550
And, then we have a
final exam coming up,
15
00:00:56,550 --> 00:01:00,780
and today's lecture,
as I told you by email,
16
00:01:00,780 --> 00:01:05,019
will be a first
step in the review,
17
00:01:05,019 --> 00:01:07,890
and then on Wednesday
I'll do all I can
18
00:01:07,890 --> 00:01:13,560
in reviewing the whole course.
19
00:01:13,560 --> 00:01:16,300
So my topic today
is -- actually,
20
00:01:16,300 --> 00:01:24,740
this is a lecture I have never
given before in this way,
21
00:01:24,740 --> 00:01:28,310
and it will -- well,
four subspaces,
22
00:01:28,310 --> 00:01:32,190
that's certainly fundamental,
and you know that,
23
00:01:32,190 --> 00:01:34,810
so I want to speak
about left-inverses
24
00:01:34,810 --> 00:01:37,150
and right-inverses and
then something called
25
00:01:37,150 --> 00:01:39,190
pseudo-inverses.
26
00:01:39,190 --> 00:01:43,110
And pseudo-inverses,
let me say right away,
27
00:01:43,110 --> 00:01:47,240
that comes in near the
end of chapter seven,
28
00:01:47,240 --> 00:01:52,170
and that would not be
expected on the final.
29
00:01:52,170 --> 00:01:54,840
But you'll see that
what I'm talking about
30
00:01:54,840 --> 00:02:01,600
is really the basic stuff that,
for an m-by-n matrix of rank r,
31
00:02:01,600 --> 00:02:06,080
we're going back to the most
fundamental picture in linear
32
00:02:06,080 --> 00:02:07,130
algebra.
33
00:02:07,130 --> 00:02:12,020
Nobody could forget
that picture, right?
34
00:02:12,020 --> 00:02:15,890
When you're my age, even,
you'll remember the row space,
35
00:02:15,890 --> 00:02:17,560
and the null space.
36
00:02:17,560 --> 00:02:21,130
Orthogonal complements over
there, the column space
37
00:02:21,130 --> 00:02:23,530
and the null space of
A transpose column,
38
00:02:23,530 --> 00:02:26,190
orthogonal
complements over here.
39
00:02:26,190 --> 00:02:29,220
And I want to speak
about inverses.
40
00:02:29,220 --> 00:02:30,320
OK.
41
00:02:30,320 --> 00:02:33,960
And I want to identify the
different possibilities.
42
00:02:33,960 --> 00:02:37,980
So first of all,
when does a matrix
43
00:02:37,980 --> 00:02:42,440
have a just a perfect
inverse, two-sided, you know,
44
00:02:42,440 --> 00:02:51,600
so the two-sided inverse is what
we just call inverse, right?
45
00:02:51,600 --> 00:02:57,720
And, so that means that
there's a matrix that
46
00:02:57,720 --> 00:03:02,010
produces the identity, whether
we write it on the left
47
00:03:02,010 --> 00:03:03,670
or on the right.
48
00:03:03,670 --> 00:03:10,170
And just tell me, how
are the numbers r,
49
00:03:10,170 --> 00:03:16,580
the rank, n the number of
columns, m the number of rows,
50
00:03:16,580 --> 00:03:19,070
how are those
numbers related when
51
00:03:19,070 --> 00:03:21,320
we have an invertible matrix?
52
00:03:21,320 --> 00:03:23,690
So this is the
matrix which was --
53
00:03:23,690 --> 00:03:26,540
chapter two was all
about matrices like this,
54
00:03:26,540 --> 00:03:30,420
the beginning of the course,
what was the relation of th-
55
00:03:30,420 --> 00:03:36,110
of r, m, and n,
for the nice case?
56
00:03:36,110 --> 00:03:39,360
They're all the same, all equal.
57
00:03:39,360 --> 00:03:43,260
So this is the case when r=m=n.
58
00:03:43,260 --> 00:03:46,910
Square matrix, full
rank, period, just --
59
00:03:46,910 --> 00:03:50,480
so I'll use the words full rank.
60
00:03:50,480 --> 00:03:51,840
OK, good.
61
00:03:51,840 --> 00:03:53,800
Everybody knows that.
62
00:03:53,800 --> 00:03:55,860
OK.
63
00:03:55,860 --> 00:03:56,875
Then chapter three.
64
00:03:59,660 --> 00:04:03,240
We began to deal with matrices
that were not of full rank,
65
00:04:03,240 --> 00:04:07,050
and they could have any rank,
and we learned what the rank
66
00:04:07,050 --> 00:04:08,480
was.
67
00:04:08,480 --> 00:04:12,140
And then we focused,
if you remember
68
00:04:12,140 --> 00:04:17,110
on some cases like
full column rank.
69
00:04:17,110 --> 00:04:20,810
Now, can you remember what was
the deal with full column rank?
70
00:04:20,810 --> 00:04:25,020
So, now, I think this
is the case in which we
71
00:04:25,020 --> 00:04:28,650
have a left-inverse,
and I'll try to find it.
72
00:04:32,930 --> 00:04:35,130
So we have a --
73
00:04:35,130 --> 00:04:37,800
what was the situation there?
74
00:04:37,800 --> 00:04:43,780
It's the case of full column
rank, and that means --
75
00:04:43,780 --> 00:04:47,160
what does that mean about r?
76
00:04:47,160 --> 00:04:51,770
It equals, what's
the deal with r, now,
77
00:04:51,770 --> 00:04:54,460
if we have full
column rank, I mean
78
00:04:54,460 --> 00:04:59,730
the columns are independent,
but maybe not the rows.
79
00:04:59,730 --> 00:05:03,780
So what is r equal
to in this case?
80
00:05:03,780 --> 00:05:04,450
n.
81
00:05:04,450 --> 00:05:05,780
Thanks.
82
00:05:05,780 --> 00:05:06,480
n.
83
00:05:06,480 --> 00:05:06,979
r=n.
84
00:05:06,979 --> 00:05:10,270
The n columns are
independent, but probably, we
85
00:05:10,270 --> 00:05:12,920
have more rows.
86
00:05:12,920 --> 00:05:17,060
What's the picture, and then
what's the null space for this?
87
00:05:17,060 --> 00:05:19,990
So the n columns
are independent,
88
00:05:19,990 --> 00:05:22,155
what's the null
space in this case?
89
00:05:25,420 --> 00:05:27,450
So of course, you
know what I'm asking.
90
00:05:27,450 --> 00:05:30,690
You're saying, why is this guy
asking something, I know that--
91
00:05:30,690 --> 00:05:32,680
I think about it in my sleep,
92
00:05:32,680 --> 00:05:33,260
right?
93
00:05:33,260 --> 00:05:36,900
So the null space of this
matrix if the rank is
94
00:05:36,900 --> 00:05:47,280
n, the null space is what
vectors are in the null space?
95
00:05:47,280 --> 00:05:48,410
Just the zero vector.
96
00:05:51,320 --> 00:05:51,820
Right?
97
00:05:51,820 --> 00:05:53,360
The columns are independent.
98
00:05:53,360 --> 00:05:54,300
Independent columns.
99
00:05:59,020 --> 00:06:03,320
No combination of the columns
gives zero except that one.
100
00:06:03,320 --> 00:06:06,270
And what's my picture over, --
101
00:06:06,270 --> 00:06:10,050
let me redraw my picture --
102
00:06:10,050 --> 00:06:14,330
the row space is everything.
103
00:06:18,450 --> 00:06:20,860
No.
104
00:06:20,860 --> 00:06:23,030
Is that right?
105
00:06:23,030 --> 00:06:26,580
Let's see, I often get
these turned around, right?
106
00:06:26,580 --> 00:06:31,640
So what's the deal?
107
00:06:31,640 --> 00:06:34,810
The columns are
independent, right?
108
00:06:34,810 --> 00:06:39,330
So the rank should be the full
number of columns, so what
109
00:06:39,330 --> 00:06:41,250
does that tell us?
110
00:06:41,250 --> 00:06:43,051
There's no null space, right.
111
00:06:43,051 --> 00:06:43,550
OK.
112
00:06:43,550 --> 00:06:45,400
The row space is
the whole thing.
113
00:06:45,400 --> 00:06:48,510
Yes, I won't even
draw the picture.
114
00:06:48,510 --> 00:06:51,640
And what was the deal with --
115
00:06:51,640 --> 00:06:56,960
and these were very important
in least squares problems
116
00:06:56,960 --> 00:06:58,750
because --
117
00:06:58,750 --> 00:07:06,230
So, what more is true here?
118
00:07:06,230 --> 00:07:09,460
If we have full column rank,
the null space is zero,
119
00:07:09,460 --> 00:07:14,640
we have independent
columns, the unique --
120
00:07:14,640 --> 00:07:22,990
so we have zero or
one solutions to Ax=b.
121
00:07:25,790 --> 00:07:28,660
There may not be any solutions,
but if there's a solution,
122
00:07:28,660 --> 00:07:32,910
there's only one solution
because other solutions are
123
00:07:32,910 --> 00:07:35,470
found by adding on stuff
from the null space,
124
00:07:35,470 --> 00:07:39,490
and there's nobody
there to add on.
125
00:07:39,490 --> 00:07:43,250
So the particular
solution is the solution,
126
00:07:43,250 --> 00:07:45,860
if there is a
particular solution.
127
00:07:45,860 --> 00:07:49,030
But of course, the
rows might not be -
128
00:07:49,030 --> 00:07:51,990
are probably not independent
-- and therefore,
129
00:07:51,990 --> 00:07:56,880
so right-hand sides won't end
up with a zero equal zero after
130
00:07:56,880 --> 00:08:00,990
elimination, so sometimes
we may have no solution,
131
00:08:00,990 --> 00:08:02,710
or one solution.
132
00:08:02,710 --> 00:08:03,380
OK.
133
00:08:03,380 --> 00:08:10,350
And what I want to say is
that for this matrix A --
134
00:08:10,350 --> 00:08:13,910
oh, yes, tell me something about
A transpose A in this case.
135
00:08:13,910 --> 00:08:20,240
So this whole part of the board,
now, is devoted to this case.
136
00:08:20,240 --> 00:08:23,690
What's the deal
with A transpose A?
137
00:08:23,690 --> 00:08:27,280
I've emphasized over and over
how important that combination
138
00:08:27,280 --> 00:08:32,120
is, for a rectangular
matrix, A transpose A
139
00:08:32,120 --> 00:08:37,120
is the good thing to look
at, and if the rank is n,
140
00:08:37,120 --> 00:08:39,750
if the null space
has only zero in it,
141
00:08:39,750 --> 00:08:43,340
then the same is true
of A transpose A.
142
00:08:43,340 --> 00:08:49,250
That's the beautiful fact, that
if the rank of A is n, well,
143
00:08:49,250 --> 00:08:52,300
we know this will be an
n by n symmetric matrix,
144
00:08:52,300 --> 00:08:53,820
and it will be full rank.
145
00:08:53,820 --> 00:08:55,230
So this is invertible.
146
00:08:55,230 --> 00:08:58,450
This matrix is invertible.
147
00:08:58,450 --> 00:08:59,960
That matrix is invertible.
148
00:08:59,960 --> 00:09:04,070
And now I want to show
you that A itself has
149
00:09:04,070 --> 00:09:06,880
a one-sided inverse.
150
00:09:06,880 --> 00:09:08,450
Here it is.
151
00:09:08,450 --> 00:09:17,190
The inverse of that, which
exists, times A transpose,
152
00:09:17,190 --> 00:09:20,390
there is a one-sided --
shall I call it A inverse?
153
00:09:20,390 --> 00:09:24,160
-- left of the matrix A.
154
00:09:24,160 --> 00:09:28,680
Why do I say that?
155
00:09:28,680 --> 00:09:35,740
Because if I multiply this
guy by A, what do I get?
156
00:09:35,740 --> 00:09:37,630
What does that
multiplication give?
157
00:09:37,630 --> 00:09:40,360
Of course, you
know it instantly,
158
00:09:40,360 --> 00:09:44,110
because I just put
the parentheses there,
159
00:09:44,110 --> 00:09:47,000
I have A transpose A
inverse times A transpose A
160
00:09:47,000 --> 00:09:49,090
so, of course,
it's the identity.
161
00:09:49,090 --> 00:09:52,120
So it's a left inverse.
162
00:09:52,120 --> 00:09:58,830
And this was the totally
crucial case for least squares,
163
00:09:58,830 --> 00:10:02,920
because you remember that least
squares, the central equation
164
00:10:02,920 --> 00:10:06,230
of least squares had this
matrix, A transpose A,
165
00:10:06,230 --> 00:10:08,920
as its coefficient matrix.
166
00:10:08,920 --> 00:10:11,770
And in the case of
full column rank,
167
00:10:11,770 --> 00:10:15,910
that matrix is
invertible, and we're go.
168
00:10:15,910 --> 00:10:19,830
So that's the case where
there is a left-inverse.
169
00:10:19,830 --> 00:10:26,140
So A does whatever it does,
we can find a matrix that
170
00:10:26,140 --> 00:10:30,530
brings it back to the identity.
171
00:10:30,530 --> 00:10:33,710
Now, is it true that,
in the other order --
172
00:10:33,710 --> 00:10:36,830
so A inverse left times
A is the identity.
173
00:10:42,131 --> 00:10:42,630
Right?
174
00:10:42,630 --> 00:10:46,100
This matrix is m by n.
175
00:10:46,100 --> 00:10:49,580
This matrix is n by m.
176
00:10:49,580 --> 00:10:52,060
The identity matrix is n by n.
177
00:10:52,060 --> 00:10:53,560
All good.
178
00:10:53,560 --> 00:10:56,300
All good if you're n.
179
00:10:56,300 --> 00:11:02,850
But if you try to put that
matrix on the other side,
180
00:11:02,850 --> 00:11:05,280
it would fail.
181
00:11:05,280 --> 00:11:12,240
If the full column rank --
if this is smaller than m,
182
00:11:12,240 --> 00:11:14,480
the case where they're
equals is the beautiful case,
183
00:11:14,480 --> 00:11:16,270
but that's all set.
184
00:11:16,270 --> 00:11:18,100
Now, we're looking
at the case where
185
00:11:18,100 --> 00:11:21,440
the columns are independent
but the rows are not.
186
00:11:21,440 --> 00:11:25,370
So this is invertible,
but what matrix is not
187
00:11:25,370 --> 00:11:26,670
invertible?
188
00:11:26,670 --> 00:11:30,890
A A transpose is
bad for this case.
189
00:11:30,890 --> 00:11:32,570
A transpose A is good.
190
00:11:32,570 --> 00:11:35,710
So we can multiply on the
left, everything good,
191
00:11:35,710 --> 00:11:39,130
we get the left inverse.
192
00:11:39,130 --> 00:11:42,140
But it would not be
a two-sided inverse.
193
00:11:42,140 --> 00:11:46,680
A rectangular matrix can't
have a two-sided inverse,
194
00:11:46,680 --> 00:11:50,900
because there's got to be
some null space, right?
195
00:11:50,900 --> 00:11:53,840
If I have a matrix
that's rectangular,
196
00:11:53,840 --> 00:11:58,920
then either that
matrix or its transpose
197
00:11:58,920 --> 00:12:02,340
has some null space, because
if n and m are different,
198
00:12:02,340 --> 00:12:06,440
then there's going to be
some free variables around,
199
00:12:06,440 --> 00:12:09,380
and we'll have some null
space in that direction.
200
00:12:09,380 --> 00:12:17,070
OK, tell me the corresponding
picture for the opposite case.
201
00:12:17,070 --> 00:12:20,840
So now I'm going to ask
you about right-inverses.
202
00:12:20,840 --> 00:12:22,325
A right-inverse.
203
00:12:26,010 --> 00:12:28,970
And you can fill
this all out, this
204
00:12:28,970 --> 00:12:31,410
is going to be the
case of full row rank.
205
00:12:34,420 --> 00:12:41,870
And then r is equal to m, now,
the m rows are independent,
206
00:12:41,870 --> 00:12:45,270
but the columns are not.
207
00:12:45,270 --> 00:12:47,000
So what's the deal on that?
208
00:12:47,000 --> 00:12:50,640
Well, just exactly
the flip of this one.
209
00:12:50,640 --> 00:12:57,400
The null space of A
transpose contains only zero,
210
00:12:57,400 --> 00:13:00,480
because there are no
combinations of the rows that
211
00:13:00,480 --> 00:13:02,390
give the zero row.
212
00:13:02,390 --> 00:13:03,750
We have independent rows.
213
00:13:08,860 --> 00:13:13,660
And in a minute, I'll give
an example of all these.
214
00:13:13,660 --> 00:13:17,545
So, how many solutions
to Ax=b in this case?
215
00:13:23,159 --> 00:13:24,200
The rows are independent.
216
00:13:27,050 --> 00:13:30,980
So we can always solve Ax=b.
217
00:13:30,980 --> 00:13:34,540
Whenever elimination
never produces a zero row,
218
00:13:34,540 --> 00:13:38,290
so we never get into that
zero equal one problem,
219
00:13:38,290 --> 00:13:43,470
so Ax=b always has a
solution, but too many.
220
00:13:43,470 --> 00:13:50,140
So there will be some null
space, the null space of A --
221
00:13:50,140 --> 00:13:54,460
what will be the dimension
of A's null space?
222
00:13:54,460 --> 00:13:58,910
How many free
variables have we got?
223
00:13:58,910 --> 00:14:03,410
How many special solutions in
that null space have we got?
224
00:14:03,410 --> 00:14:06,090
So how many free
variables in this setup?
225
00:14:06,090 --> 00:14:12,680
We've got n columns,
so n variables,
226
00:14:12,680 --> 00:14:16,530
and this tells us
how many are pivot
227
00:14:16,530 --> 00:14:19,170
variables, that tells us
how many pivots there are,
228
00:14:19,170 --> 00:14:21,930
so there are n-m free variables.
229
00:14:21,930 --> 00:14:27,940
So there are infinitely
many solutions to Ax=b.
230
00:14:27,940 --> 00:14:37,220
We have n-m free
variables in this case.
231
00:14:37,220 --> 00:14:38,240
OK.
232
00:14:38,240 --> 00:14:45,600
Now I wanted to ask about
this idea of a right-inverse.
233
00:14:45,600 --> 00:14:46,720
OK.
234
00:14:46,720 --> 00:14:52,710
So I'm going to have a matrix
A, my matrix A, and now
235
00:14:52,710 --> 00:14:54,960
there's going to be some
inverse on the right that
236
00:14:54,960 --> 00:14:57,710
will give the identity matrix.
237
00:14:57,710 --> 00:15:05,500
So it will be A times A inverse
on the right, will be I.
238
00:15:05,500 --> 00:15:12,290
And can you tell me
what, just by comparing
239
00:15:12,290 --> 00:15:18,560
with what we had up there,
what will be the right-inverse,
240
00:15:18,560 --> 00:15:21,150
we even have a formula for it.
241
00:15:21,150 --> 00:15:22,760
There will be other --
242
00:15:22,760 --> 00:15:24,820
actually, there are
other left-inverses,
243
00:15:24,820 --> 00:15:26,610
that's our favorite.
244
00:15:26,610 --> 00:15:28,350
There will be other
right-inverses,
245
00:15:28,350 --> 00:15:31,620
but tell me our favorite here,
what's the nice right-inverse?
246
00:15:35,240 --> 00:15:39,470
The nice right-inverse
will be, well, there we
247
00:15:39,470 --> 00:15:43,820
had A transpose A
was good, now it
248
00:15:43,820 --> 00:15:46,860
will be A A transpose
that's good.
249
00:15:46,860 --> 00:15:49,680
The good matrix,
the good right --
250
00:15:49,680 --> 00:15:53,200
the thing we can invert
is A A transpose,
251
00:15:53,200 --> 00:16:00,540
so now if I just do it that way,
there sits the right-inverse.
252
00:16:00,540 --> 00:16:03,930
You see how completely parallel
it is to the one above?
253
00:16:11,220 --> 00:16:11,964
Right.
254
00:16:11,964 --> 00:16:13,130
So that's the right-inverse.
255
00:16:13,130 --> 00:16:20,960
So that's the case
when there is --
256
00:16:20,960 --> 00:16:25,390
In terms of this
picture, tell me
257
00:16:25,390 --> 00:16:29,770
what the null spaces are like
so far for these three cases.
258
00:16:29,770 --> 00:16:32,530
What about case
one, where we had
259
00:16:32,530 --> 00:16:36,930
a two-sided inverse, full
rank, everything great.
260
00:16:36,930 --> 00:16:41,290
The null spaces were,
like, gone, right?
261
00:16:41,290 --> 00:16:44,480
The null spaces were
just the zero vectors.
262
00:16:44,480 --> 00:16:49,590
Then I took case two,
this null space was gone.
263
00:16:52,990 --> 00:16:58,890
Case three, this null space was
gone, and then case four is,
264
00:16:58,890 --> 00:17:04,250
like, the most general case when
this picture is all there --
265
00:17:04,250 --> 00:17:10,680
when all the null spaces --
this has dimension r, of course,
266
00:17:10,680 --> 00:17:14,700
this has dimension n-r,
this has dimension r,
267
00:17:14,700 --> 00:17:26,800
this has dimension m-r, and the
final case will be when r is
268
00:17:26,800 --> 00:17:29,210
smaller than m and n.
269
00:17:29,210 --> 00:17:38,940
But can I just,
before I leave here
270
00:17:38,940 --> 00:17:43,750
look a little more at this one?
271
00:17:43,750 --> 00:17:46,510
At this case of
full column rank?
272
00:17:46,510 --> 00:17:52,070
So A inverse on the left,
it has this left-inverse
273
00:17:52,070 --> 00:17:53,930
to give the identity.
274
00:17:53,930 --> 00:17:56,540
I said if we multiply
it in the other order,
275
00:17:56,540 --> 00:17:57,970
we wouldn't get the identity.
276
00:17:57,970 --> 00:18:02,650
But then I just realized that I
should ask you, what do we get?
277
00:18:02,650 --> 00:18:05,260
So if I put them in
the other order --
278
00:18:05,260 --> 00:18:19,010
if I continue this down below,
but I write A times A inverse
279
00:18:19,010 --> 00:18:21,040
left -- so there's A
times the left-inverse,
280
00:18:21,040 --> 00:18:23,370
but it's not on
the left any more.
281
00:18:23,370 --> 00:18:26,970
So it's not going to
come out perfectly.
282
00:18:26,970 --> 00:18:35,990
But everybody in this room
ought to recognize that matrix,
283
00:18:35,990 --> 00:18:38,150
right?
284
00:18:38,150 --> 00:18:42,080
Let's see, is that
the guy we know?
285
00:18:42,080 --> 00:18:43,195
Am I OK, here?
286
00:18:51,280 --> 00:18:53,000
What is that matrix?
287
00:18:53,000 --> 00:18:56,130
P. Thanks.
288
00:18:56,130 --> 00:18:59,230
P. That matrix --
289
00:18:59,230 --> 00:19:02,070
it's a projection.
290
00:19:02,070 --> 00:19:07,750
It's the projection
onto the column space.
291
00:19:07,750 --> 00:19:12,340
It's trying to be the
identity matrix, right?
292
00:19:12,340 --> 00:19:17,620
A projection matrix tries
to be the identity matrix,
293
00:19:17,620 --> 00:19:22,140
but you've given it,
an impossible job.
294
00:19:22,140 --> 00:19:25,240
So it's the identity
matrix where it can be,
295
00:19:25,240 --> 00:19:27,750
and elsewhere, it's
the zero matrix.
296
00:19:27,750 --> 00:19:29,820
So this is P, right.
297
00:19:29,820 --> 00:19:34,190
A projection onto
the column space.
298
00:19:34,190 --> 00:19:38,190
And if I asked you this one,
and put these in the opposite
299
00:19:38,190 --> 00:19:41,340
OK. order -- so this
came from up here.
300
00:19:41,340 --> 00:19:45,610
And similarly, if I try to put
the right inverse on the left
301
00:19:45,610 --> 00:19:48,380
--
302
00:19:48,380 --> 00:19:51,840
so that, like, came from above.
303
00:19:51,840 --> 00:19:53,810
This, coming from
this side, what
304
00:19:53,810 --> 00:19:56,620
happens if I try to put the
right inverse on the left?
305
00:19:56,620 --> 00:20:05,090
Then I would have A transpose
A, A transpose inverse A,
306
00:20:05,090 --> 00:20:08,350
if this matrix is
now on the left, what
307
00:20:08,350 --> 00:20:09,880
do you figure that matrix is?
308
00:20:09,880 --> 00:20:17,390
It's going to be a
projection, too, right?
309
00:20:17,390 --> 00:20:19,090
It looks very much
like this guy,
310
00:20:19,090 --> 00:20:22,400
except the only difference
is, A and A transpose
311
00:20:22,400 --> 00:20:24,100
have been reversed.
312
00:20:24,100 --> 00:20:27,960
So this is a projection,
this is another projection,
313
00:20:27,960 --> 00:20:29,710
onto the row space.
314
00:20:33,520 --> 00:20:35,850
Again, it's trying
to be the identity,
315
00:20:35,850 --> 00:20:39,980
but there's only so
much the matrix can do.
316
00:20:39,980 --> 00:20:44,580
And this is the projection
onto the column space.
317
00:20:44,580 --> 00:20:50,600
So let me now go back
to the main picture
318
00:20:50,600 --> 00:20:55,600
and tell you about the general
case, the pseudo-inverse.
319
00:20:55,600 --> 00:20:58,060
These are cases we know.
320
00:20:58,060 --> 00:21:01,500
So this was important review.
321
00:21:01,500 --> 00:21:08,480
You've got to know the
business about these ranks,
322
00:21:08,480 --> 00:21:11,350
and the free variables --
323
00:21:11,350 --> 00:21:14,960
really, this is linear
algebra coming together.
324
00:21:14,960 --> 00:21:19,130
And, you know, one nice
thing about teaching 18.06,
325
00:21:19,130 --> 00:21:23,260
It's not trivial.
326
00:21:23,260 --> 00:21:25,610
But it's --
327
00:21:25,610 --> 00:21:28,590
I don't know, somehow, it's
nice when it comes out right.
328
00:21:28,590 --> 00:21:31,360
I mean -- well, I shouldn't say
anything bad about calculus,
329
00:21:31,360 --> 00:21:33,050
but I will.
330
00:21:33,050 --> 00:21:35,150
I mean, like, you
know, you have formulas
331
00:21:35,150 --> 00:21:40,190
for surface area, and other
awful things and, you know,
332
00:21:40,190 --> 00:21:46,580
they do their best in
calculus, but it's not elegant.
333
00:21:46,580 --> 00:21:52,300
And, linear algebra just
is -- well, you know,
334
00:21:52,300 --> 00:21:54,770
linear algebra is about
the nice part of calculus,
335
00:21:54,770 --> 00:22:00,880
where everything's, like, flat,
and, the formulas come out
336
00:22:00,880 --> 00:22:01,830
right.
337
00:22:01,830 --> 00:22:04,040
And you can go into
high dimensions
338
00:22:04,040 --> 00:22:06,820
where, in calculus,
you're trying
339
00:22:06,820 --> 00:22:09,780
to visualize these things,
well, two or three dimensions
340
00:22:09,780 --> 00:22:10,810
is kind of the limit.
341
00:22:10,810 --> 00:22:12,430
But here, we don't --
342
00:22:12,430 --> 00:22:16,280
you know, I've stopped
doing two-by-twos,
343
00:22:16,280 --> 00:22:18,160
I'm just talking about
the general case.
344
00:22:18,160 --> 00:22:22,430
OK, now I really will speak
about the general case here.
345
00:22:22,430 --> 00:22:27,810
What could be the inverse --
346
00:22:27,810 --> 00:22:29,910
what's a kind of
reasonable inverse
347
00:22:29,910 --> 00:22:34,070
for a matrix for the
completely general matrix where
348
00:22:34,070 --> 00:22:38,410
there's a rank r, but
it's smaller than n,
349
00:22:38,410 --> 00:22:41,680
so there's some null space
left, and it's smaller
350
00:22:41,680 --> 00:22:44,930
than m, so a transpose
has some null space,
351
00:22:44,930 --> 00:22:48,430
and it's those null spaces
that are screwing up inverses,
352
00:22:48,430 --> 00:22:49,650
right?
353
00:22:49,650 --> 00:22:53,090
Because if a matrix
takes a vector to zero,
354
00:22:53,090 --> 00:23:01,390
well, there's no way an inverse
can, like, bring it back
355
00:23:01,390 --> 00:23:03,240
to life.
356
00:23:03,240 --> 00:23:05,810
My topic is now
the pseudo-inverse,
357
00:23:05,810 --> 00:23:09,190
and let's just by a
picture, see what's
358
00:23:09,190 --> 00:23:11,170
the best inverse we could have?
359
00:23:11,170 --> 00:23:15,820
So, here's a vector
x in the row space.
360
00:23:15,820 --> 00:23:18,170
I multiply by A.
361
00:23:18,170 --> 00:23:22,050
Now, the one thing everybody
knows is you take a vector,
362
00:23:22,050 --> 00:23:25,360
you multiply by A,
and you get an output,
363
00:23:25,360 --> 00:23:28,220
and where is that output?
364
00:23:28,220 --> 00:23:31,090
Where is Ax?
365
00:23:31,090 --> 00:23:35,050
Always in the
column space, right?
366
00:23:35,050 --> 00:23:37,750
Ax is a combination
of the columns.
367
00:23:37,750 --> 00:23:39,310
So Ax is somewhere here.
368
00:23:42,800 --> 00:23:46,750
So I could take all the
vectors in the row space.
369
00:23:46,750 --> 00:23:49,400
I could multiply them all by A.
370
00:23:49,400 --> 00:23:53,870
I would get a bunch of
vectors in the column space
371
00:23:53,870 --> 00:23:59,280
and what I think is, I'd get all
the vectors in the column space
372
00:23:59,280 --> 00:24:00,670
just right.
373
00:24:00,670 --> 00:24:03,440
I think that this
connection between an x
374
00:24:03,440 --> 00:24:07,257
in the row space and an Ax
in the column space, this
375
00:24:07,257 --> 00:24:07,840
is one-to-one.
376
00:24:12,160 --> 00:24:14,170
We got a chance, because
they have the same
377
00:24:14,170 --> 00:24:15,150
dimension.
378
00:24:15,150 --> 00:24:17,480
That's an r-dimensional
space, and that's
379
00:24:17,480 --> 00:24:20,060
an r-dimensional space.
380
00:24:20,060 --> 00:24:22,970
And somehow, the matrix A --
381
00:24:22,970 --> 00:24:27,260
it's got these null
spaces hanging around,
382
00:24:27,260 --> 00:24:30,820
where it's knocking vectors to
383
00:24:30,820 --> 00:24:33,510
And then it's got all the
vectors in between, zero.
384
00:24:33,510 --> 00:24:35,190
which is almost all vectors.
385
00:24:35,190 --> 00:24:38,460
Almost all vectors have
a row space component
386
00:24:38,460 --> 00:24:39,250
and a null space
387
00:24:39,250 --> 00:24:39,950
component.
388
00:24:39,950 --> 00:24:42,880
And it's killing the
null space component.
389
00:24:42,880 --> 00:24:45,690
But if I look at the vectors
that are in the row space,
390
00:24:45,690 --> 00:24:48,740
with no null space component,
just in the row space,
391
00:24:48,740 --> 00:24:51,110
then they all go into
the column space,
392
00:24:51,110 --> 00:24:55,340
so if I put another vector,
let's say, y, in the row space,
393
00:24:55,340 --> 00:25:02,730
I positive that wherever
Ay is, it won't hit Ax.
394
00:25:02,730 --> 00:25:04,680
Do you see what I'm saying?
395
00:25:04,680 --> 00:25:05,830
Let's see why.
396
00:25:09,340 --> 00:25:09,840
All right.
397
00:25:09,840 --> 00:25:12,600
So here's what I said.
398
00:25:12,600 --> 00:25:23,430
If x and y are in the
row space, then A x
399
00:25:23,430 --> 00:25:27,650
is not the same as A y.
400
00:25:27,650 --> 00:25:30,110
They're both in the
column space, of course,
401
00:25:30,110 --> 00:25:31,210
but they're different.
402
00:25:36,690 --> 00:25:39,780
That would be a perfect
question on a final exam,
403
00:25:39,780 --> 00:25:45,360
because that's what
I'm teaching you
404
00:25:45,360 --> 00:25:48,690
in that material
of chapter three
405
00:25:48,690 --> 00:25:53,920
and chapter four,
especially chapter three.
406
00:25:53,920 --> 00:25:58,430
If x and y are in the row space,
then Ax is different from Ay.
407
00:25:58,430 --> 00:26:01,280
So what this means --
408
00:26:01,280 --> 00:26:03,630
and we'll see why --
409
00:26:03,630 --> 00:26:09,000
is that, in words, from the
row space to the column space,
410
00:26:09,000 --> 00:26:12,980
A is perfect, it's
an invertible matrix.
411
00:26:12,980 --> 00:26:16,700
If we, like, limited
it to those spaces.
412
00:26:16,700 --> 00:26:19,930
And then, its
inverse will be what
413
00:26:19,930 --> 00:26:21,290
I'll call the pseudo-inverse.
414
00:26:21,290 --> 00:26:23,770
So that's that the
pseudo-inverse is.
415
00:26:23,770 --> 00:26:28,940
It's the inverse -- so A goes
this way, from x to y -- sorry,
416
00:26:28,940 --> 00:26:35,340
x to A x, from y to A y,
that's A, going that way.
417
00:26:35,340 --> 00:26:38,910
Then in the other direction,
anything in the column space
418
00:26:38,910 --> 00:26:41,400
comes from somebody
in the row space,
419
00:26:41,400 --> 00:26:45,010
and the reverse there is what
I'll call the pseudo-inverse,
420
00:26:45,010 --> 00:26:51,010
and the accepted
notation is A plus.
421
00:26:51,010 --> 00:26:55,390
So y will be A plus x.
422
00:26:55,390 --> 00:26:55,890
I'm sorry.
423
00:26:55,890 --> 00:27:05,900
No, y will be A plus times
whatever it started with, A y.
424
00:27:05,900 --> 00:27:09,030
Do you see my picture there?
425
00:27:09,030 --> 00:27:11,340
Same, of course, for x and A x.
426
00:27:11,340 --> 00:27:15,280
This way, A does it, the other
way is the pseudo-inverse,
427
00:27:15,280 --> 00:27:18,430
and the pseudo-inverse
just kills this stuff,
428
00:27:18,430 --> 00:27:20,360
and the matrix just kills this
429
00:27:20,360 --> 00:27:20,860
stuff.
430
00:27:20,860 --> 00:27:25,260
So everything that's really
serious here is going
431
00:27:25,260 --> 00:27:27,840
on in the row space and
the column space, and now,
432
00:27:27,840 --> 00:27:31,550
tell me --
433
00:27:31,550 --> 00:27:34,910
this is the fundamental
fact, that between those two
434
00:27:34,910 --> 00:27:37,560
r-dimensional spaces,
our matrix is perfect.
435
00:27:40,900 --> 00:27:41,400
Why?
436
00:27:44,960 --> 00:27:47,180
Suppose they weren't.
437
00:27:47,180 --> 00:27:49,230
Why do I get into trouble?
438
00:27:49,230 --> 00:27:51,780
Suppose -- so, proof.
439
00:27:51,780 --> 00:27:54,090
I haven't written
down proof very much,
440
00:27:54,090 --> 00:27:57,640
but I'm going to
use that word once.
441
00:27:57,640 --> 00:28:00,710
Suppose they were the same.
442
00:28:00,710 --> 00:28:07,990
Suppose these are supposed
to be two different vectors.
443
00:28:07,990 --> 00:28:10,540
Maybe I'd better make
the statement correctly.
444
00:28:10,540 --> 00:28:13,930
If x and y are different
vectors in the row space --
445
00:28:13,930 --> 00:28:20,790
maybe I'll better put if
x is different from y,
446
00:28:20,790 --> 00:28:23,370
both in the row space --
447
00:28:23,370 --> 00:28:25,620
so I'm starting with two
different vectors in the row
448
00:28:25,620 --> 00:28:29,900
space, I'm multiplying by A --
so these guys are in the column
449
00:28:29,900 --> 00:28:33,520
space, everybody knows
that, and the point is,
450
00:28:33,520 --> 00:28:36,980
they're different over there.
451
00:28:36,980 --> 00:28:38,950
So, suppose they weren't.
452
00:28:38,950 --> 00:28:40,570
Suppose A x=A y.
453
00:28:44,940 --> 00:28:48,800
Suppose, well, that's the
same as saying A(x-y) is zero.
454
00:28:54,960 --> 00:28:57,140
So what?
455
00:28:57,140 --> 00:29:00,090
So, what do I know
now about (x-y),
456
00:29:00,090 --> 00:29:03,500
what do I know
about this vector?
457
00:29:03,500 --> 00:29:07,810
Well, I can see right
away, what space is it in?
458
00:29:07,810 --> 00:29:10,390
It's sitting in the
null space, right?
459
00:29:10,390 --> 00:29:11,580
So it's in the null space.
460
00:29:14,520 --> 00:29:17,030
But what else do
I know about it?
461
00:29:17,030 --> 00:29:20,950
Here it was x in the row
space, y in the row space,
462
00:29:20,950 --> 00:29:23,170
what about x-y?
463
00:29:23,170 --> 00:29:29,870
It's also in the
row space, right?
464
00:29:29,870 --> 00:29:32,030
Heck, that thing
is a vector space,
465
00:29:32,030 --> 00:29:35,550
and if the vector space
is anything at all,
466
00:29:35,550 --> 00:29:38,610
if x is in the row space,
and y is in the row space,
467
00:29:38,610 --> 00:29:43,040
then the difference is also,
so it's also in the row space.
468
00:29:47,990 --> 00:29:48,850
So what?
469
00:29:48,850 --> 00:29:52,420
Now I've got a vector x-y
that's in the null space,
470
00:29:52,420 --> 00:29:56,270
and that's also in the row
space, so what vector is it?
471
00:29:56,270 --> 00:29:58,380
It's the zero vector.
472
00:29:58,380 --> 00:30:00,840
So I would conclude
from that that x-y
473
00:30:00,840 --> 00:30:07,200
had to be the zero vector,
x-y, so, in other words,
474
00:30:07,200 --> 00:30:09,120
if I start from two
different vectors,
475
00:30:09,120 --> 00:30:11,480
I get two different vectors.
476
00:30:11,480 --> 00:30:14,020
If these vectors are the
same, then those vectors
477
00:30:14,020 --> 00:30:16,050
had to be the same.
478
00:30:16,050 --> 00:30:21,600
That's like the algebra proof,
which we understand completely
479
00:30:21,600 --> 00:30:28,230
because we really understand
these subspaces of what
480
00:30:28,230 --> 00:30:32,480
I said in words, that
a matrix A is really
481
00:30:32,480 --> 00:30:37,560
a nice, invertible mapping
from row space to columns pace.
482
00:30:37,560 --> 00:30:40,160
If the null spaces
keep out of the way,
483
00:30:40,160 --> 00:30:43,200
then we have an inverse.
484
00:30:43,200 --> 00:30:46,840
And that inverse is
called the pseudo inverse,
485
00:30:46,840 --> 00:30:51,410
and it's a very, very,
useful in application.
486
00:30:51,410 --> 00:30:54,360
Statisticians
discovered, oh boy, this
487
00:30:54,360 --> 00:30:56,580
is the thing that we
needed all our lives,
488
00:30:56,580 --> 00:30:59,100
and here it finally showed
up, the pseudo-inverse
489
00:30:59,100 --> 00:31:02,070
is the right thing.
490
00:31:02,070 --> 00:31:04,300
Why do statisticians need it?
491
00:31:04,300 --> 00:31:11,360
And because statisticians
are like least-squares-happy.
492
00:31:11,360 --> 00:31:14,440
I mean they're always
doing least squares.
493
00:31:14,440 --> 00:31:20,700
And so this is their
central linear regression.
494
00:31:20,700 --> 00:31:22,920
Statisticians who may
watch this on video,
495
00:31:22,920 --> 00:31:28,940
please forgive that
description of your interests.
496
00:31:28,940 --> 00:31:35,170
One of your interests is linear
regression and this problem.
497
00:31:35,170 --> 00:31:41,150
But this problem is only OK
provided we have full column
498
00:31:41,150 --> 00:31:42,080
rank.
499
00:31:42,080 --> 00:31:46,810
And statisticians have to worry
all the time about, oh, God,
500
00:31:46,810 --> 00:31:49,900
maybe we just repeated
an experiment.
501
00:31:49,900 --> 00:31:52,220
You know, you're taking
all these measurements,
502
00:31:52,220 --> 00:31:54,750
maybe you just repeat
them a few times.
503
00:31:54,750 --> 00:31:56,730
You know, maybe they're
not independent.
504
00:31:56,730 --> 00:32:00,910
Well, in that case, that
A transpose A matrix
505
00:32:00,910 --> 00:32:04,230
that they depend on
becomes singular.
506
00:32:04,230 --> 00:32:06,890
So then that's when they
needed the pseudo-inverse,
507
00:32:06,890 --> 00:32:09,180
it just arrived at
the right moment,
508
00:32:09,180 --> 00:32:13,840
and it's the right quantity.
509
00:32:13,840 --> 00:32:14,440
OK.
510
00:32:14,440 --> 00:32:21,590
So now that you know what the
pseudo-inverse should do, let
511
00:32:21,590 --> 00:32:25,080
me see what it is.
512
00:32:25,080 --> 00:32:27,010
Can we find it?
513
00:32:27,010 --> 00:32:30,170
So this is my -- to
complete the lecture is --
514
00:32:30,170 --> 00:32:42,740
how do I find this
pseudo-inverse A plus?
515
00:32:42,740 --> 00:32:45,310
OK.
516
00:32:45,310 --> 00:32:46,340
OK.
517
00:32:46,340 --> 00:32:48,170
Well, here's one way.
518
00:32:50,710 --> 00:32:53,860
Everything I do today is
to try to review stuff.
519
00:32:53,860 --> 00:33:00,770
One way would be to
start from the SVD.
520
00:33:00,770 --> 00:33:02,630
The Singular Value
Decomposition.
521
00:33:02,630 --> 00:33:05,380
And you remember
that that factored A
522
00:33:05,380 --> 00:33:10,480
into an orthogonal matrix
times this diagonal matrix
523
00:33:10,480 --> 00:33:12,500
times this orthogonal matrix.
524
00:33:12,500 --> 00:33:16,330
But what did that
diagonal guy look like?
525
00:33:16,330 --> 00:33:25,510
This diagonal guy, sigma,
has some non-zeroes,
526
00:33:25,510 --> 00:33:28,280
and you remember, they
came from A transpose A,
527
00:33:28,280 --> 00:33:31,400
and A A transpose, these
are the good guys, and then
528
00:33:31,400 --> 00:33:34,680
some more zeroes, and all zeroes
there, and all zeroes there.
529
00:33:38,110 --> 00:33:41,700
So you can guess what
the pseudo-inverse is,
530
00:33:41,700 --> 00:33:45,030
I just invert stuff that's
nice to invert -- well,
531
00:33:45,030 --> 00:33:47,280
what's the
pseudo-inverse of this?
532
00:33:47,280 --> 00:33:50,610
That's what the
problem comes down to.
533
00:33:50,610 --> 00:33:55,310
What's the pseudo-inverse of
this beautiful diagonal matrix?
534
00:33:55,310 --> 00:33:58,380
But it's got a
null space, right?
535
00:33:58,380 --> 00:34:01,480
What's the rank of this matrix?
536
00:34:01,480 --> 00:34:06,420
What's the rank of
this diagonal matrix?
537
00:34:06,420 --> 00:34:07,430
r, of course.
538
00:34:07,430 --> 00:34:09,790
It's got r non-zeroes,
and then it's otherwise,
539
00:34:09,790 --> 00:34:10,780
zip.
540
00:34:10,780 --> 00:34:21,000
So it's got n columns, it's got
m rows, and it's got rank r.
541
00:34:21,000 --> 00:34:23,989
It's the best example, the
simplest example we could ever
542
00:34:23,989 --> 00:34:28,089
have of our general setup.
543
00:34:31,179 --> 00:34:31,699
OK?
544
00:34:31,699 --> 00:34:36,800
So what's the pseudo-inverse?
545
00:34:36,800 --> 00:34:39,020
What's the matrix --
546
00:34:39,020 --> 00:34:41,296
so I'll erase our columns,
because right below it,
547
00:34:41,296 --> 00:34:42,754
I want to write
the pseudo-inverse.
548
00:34:46,199 --> 00:34:49,179
OK, you can make a
pretty darn good guess.
549
00:34:49,179 --> 00:34:53,170
If it was a proper diagonal
matrix, invertible,
550
00:34:53,170 --> 00:34:57,000
if there weren't any zeroes
down here, if it was sigma one
551
00:34:57,000 --> 00:35:01,010
to sigma n, then everybody
knows what the inverse would be,
552
00:35:01,010 --> 00:35:08,760
the inverse would be one over
sigma one, down to one over s-
553
00:35:08,760 --> 00:35:13,340
but of course, I'll
have to stop at sigma r.
554
00:35:13,340 --> 00:35:17,350
And, it will be the rest,
zeroes again, of course.
555
00:35:17,350 --> 00:35:22,620
And now this one was
m by n, and this one
556
00:35:22,620 --> 00:35:25,340
is meant to have a slightly
different, you know,
557
00:35:25,340 --> 00:35:29,230
transpose shape, n by m.
558
00:35:29,230 --> 00:35:31,740
They both have that rank r.
559
00:35:39,820 --> 00:35:45,350
My idea is that the
pseudo-inverse is the best --
560
00:35:45,350 --> 00:35:48,200
is the closest I can
come to an inverse.
561
00:35:48,200 --> 00:35:53,220
So what is sigma times
its pseudo-inverse?
562
00:35:53,220 --> 00:35:56,020
Can you multiply sigma
by its pseudo-inverse?
563
00:35:56,020 --> 00:35:57,920
Multiply that by that?
564
00:35:57,920 --> 00:35:59,130
What matrix do you get?
565
00:36:05,040 --> 00:36:07,280
They're diagonal.
566
00:36:07,280 --> 00:36:10,200
Rectangular, of course.
567
00:36:10,200 --> 00:36:18,780
But of course, we're
going to get ones, R ones,
568
00:36:18,780 --> 00:36:20,590
and all the rest, zeroes.
569
00:36:20,590 --> 00:36:24,420
And the shape of that, this
whole matrix will be m by
570
00:36:24,420 --> 00:36:25,780
m.
571
00:36:25,780 --> 00:36:30,200
And suppose I did it
in the other order.
572
00:36:30,200 --> 00:36:32,067
Suppose I did sigma plus sigma.
573
00:36:32,067 --> 00:36:33,525
Why don't I do it
right underneath?
574
00:36:38,190 --> 00:36:40,500
in the opposite order?
575
00:36:40,500 --> 00:36:42,640
See, this matrix hasn't
got a left-inverse,
576
00:36:42,640 --> 00:36:45,170
it hasn't got a right-inverse,
but every matrix
577
00:36:45,170 --> 00:36:46,590
has got a pseudo-inverse.
578
00:36:46,590 --> 00:36:51,300
If I do it in the order sigma
plus sigma, what do I get?
579
00:36:51,300 --> 00:36:55,920
Square matrix, this is
m by n, this is m by m,
580
00:36:55,920 --> 00:37:01,590
my result is going to m by
m -- is going to be n by n,
581
00:37:01,590 --> 00:37:03,030
and what is it?
582
00:37:05,930 --> 00:37:08,260
Those are diagonal
matrices, it's
583
00:37:08,260 --> 00:37:10,600
going to be ones,
and then zeroes.
584
00:37:14,670 --> 00:37:19,020
It's not the same as that,
it's a different size --
585
00:37:19,020 --> 00:37:22,090
it's a projection.
586
00:37:22,090 --> 00:37:25,560
One is a projection matrix
onto the column space,
587
00:37:25,560 --> 00:37:30,590
and this one is the projection
matrix onto the row space.
588
00:37:30,590 --> 00:37:35,470
That's the best that
pseudo-inverse can do.
589
00:37:35,470 --> 00:37:38,030
So what the
pseudo-inverse does is,
590
00:37:38,030 --> 00:37:41,050
if you multiply on the left,
you don't get the identity,
591
00:37:41,050 --> 00:37:42,510
if you multiply
on the right, you
592
00:37:42,510 --> 00:37:46,080
don't get the identity, what
you get is the projection.
593
00:37:46,080 --> 00:37:51,300
It brings you into the two
good spaces, the row space
594
00:37:51,300 --> 00:37:52,760
and column space.
595
00:37:52,760 --> 00:37:55,240
And it just wipes
out the null space.
596
00:37:55,240 --> 00:37:57,800
So that's what the
pseudo-inverse of this diagonal
597
00:37:57,800 --> 00:38:04,710
one is, and then the
pseudo-inverse of A itself --
598
00:38:04,710 --> 00:38:06,290
this is perfectly invertible.
599
00:38:06,290 --> 00:38:09,180
What's the inverse
of V transpose?
600
00:38:09,180 --> 00:38:11,800
Just another tiny bit of review.
601
00:38:11,800 --> 00:38:18,590
That's an orthogonal matrix,
and its inverse is V, good.
602
00:38:18,590 --> 00:38:21,360
This guy has got all
the trouble in it,
603
00:38:21,360 --> 00:38:25,940
all the null space
is responsible for,
604
00:38:25,940 --> 00:38:28,610
so it doesn't have
a true inverse,
605
00:38:28,610 --> 00:38:32,010
it has a pseudo-inverse,
and then the inverse of U
606
00:38:32,010 --> 00:38:37,320
is U transpose, thanks.
607
00:38:37,320 --> 00:38:40,200
Or, of course, I
could write U inverse.
608
00:38:40,200 --> 00:38:43,640
So, that's the question of, how
do you find the pseudo-inverse
609
00:38:43,640 --> 00:38:45,110
--
610
00:38:45,110 --> 00:38:49,350
so what statisticians do
when they're in this --
611
00:38:49,350 --> 00:38:54,980
so this is like the case of
where least squares breaks down
612
00:38:54,980 --> 00:38:58,420
because the rank is --
you don't have full rank,
613
00:38:58,420 --> 00:39:05,020
and the beauty of the singular
value decomposition is,
614
00:39:05,020 --> 00:39:09,620
it puts all the problems into
this diagonal matrix where
615
00:39:09,620 --> 00:39:10,980
it's clear what to do.
616
00:39:10,980 --> 00:39:13,960
It's the best inverse you
could think of is clear.
617
00:39:13,960 --> 00:39:16,430
You see there could be other --
618
00:39:16,430 --> 00:39:18,720
I mean, we could put
some stuff down here,
619
00:39:18,720 --> 00:39:20,980
it would multiply these zeroes.
620
00:39:20,980 --> 00:39:27,270
It wouldn't have any effect,
but then the good pseudo-inverse
621
00:39:27,270 --> 00:39:33,290
is the one with no extra
stuff, it's sort of, like,
622
00:39:33,290 --> 00:39:36,430
as small as possible.
623
00:39:36,430 --> 00:39:41,040
It has to have those
to produce the ones.
624
00:39:41,040 --> 00:39:46,260
If it had other stuff, it
would just be a larger matrix,
625
00:39:46,260 --> 00:39:51,050
so this pseudo-inverse is kind
of the minimal matrix that
626
00:39:51,050 --> 00:39:53,960
gives the best result.
627
00:39:53,960 --> 00:39:57,250
Sigma sigma plus being r ones.
628
00:39:57,250 --> 00:39:59,710
SK.
629
00:39:59,710 --> 00:40:03,122
so I guess I'm hoping --
630
00:40:03,122 --> 00:40:04,830
pseudo-inverse, again,
let me repeat what
631
00:40:04,830 --> 00:40:06,038
I said at the very beginning.
632
00:40:09,080 --> 00:40:11,990
This pseudo-inverse,
which appears
633
00:40:11,990 --> 00:40:17,700
at the end, which is in
section seven point four,
634
00:40:17,700 --> 00:40:24,090
and probably I did more with
it here than I did in the book.
635
00:40:24,090 --> 00:40:28,460
The word pseudo-inverse will
not appear on an exam in this
636
00:40:28,460 --> 00:40:34,900
course, but I think if you
see this all will appear,
637
00:40:34,900 --> 00:40:40,010
because this is all what the
course was about, chapters one,
638
00:40:40,010 --> 00:40:41,527
two, three, four --
639
00:40:44,150 --> 00:40:47,900
but if you see all that,
then you probably see,
640
00:40:47,900 --> 00:40:51,810
well, OK, the general case
had both null spaces around,
641
00:40:51,810 --> 00:40:56,470
and this is the
natural thing to do.
642
00:40:56,470 --> 00:41:01,190
So, this is one way to
find the pseudo-inverse.
643
00:41:01,190 --> 00:41:03,500
Yes.
644
00:41:03,500 --> 00:41:07,260
The point of a pseudo-inverse,
of computing a pseudo-inverse
645
00:41:07,260 --> 00:41:10,220
is to get some factors
where you can find
646
00:41:10,220 --> 00:41:12,000
the pseudo-inverse quickly.
647
00:41:12,000 --> 00:41:15,720
And this is, like, the
champion, because this
648
00:41:15,720 --> 00:41:20,850
is where we can invert
those, and those two, easily,
649
00:41:20,850 --> 00:41:26,560
just by transposing, and we
know what to do with a diagonal.
650
00:41:26,560 --> 00:41:33,210
OK, that's as much
review, maybe --
651
00:41:33,210 --> 00:41:37,920
let's have a five-minute holiday
in 18.06 and, I'll see you
652
00:41:37,920 --> 00:41:40,060
Wednesday, then,
for the rest of this
653
00:41:40,060 --> 00:41:40,560
course.
654
00:41:40,560 --> 00:41:42,110
Thanks.