1
00:00:00 --> 00:00:01
2
00:00:01 --> 00:00:02
The following content is
provided under a Creative
3
00:00:02 --> 00:00:03
Commons license.
4
00:00:03 --> 00:00:06
Your support will help MIT
OpenCourseWare continue to
5
00:00:06 --> 00:00:09
offer high-quality educational
resources for free.
6
00:00:09 --> 00:00:13
To make a donation or to view
additional materials from
7
00:00:13 --> 00:00:15
hundreds of MIT courses, visit
MIT OpenCourseWare
8
00:00:15 --> 00:00:21
at ocw.mit.edu.
9
00:00:21 --> 00:00:21
PROFESSOR STRANG: OK.
10
00:00:21 --> 00:00:26
So this is Lecture 14, it's
also the lecture before the
11
00:00:26 --> 00:00:29
exam on Tuesday evening.
12
00:00:29 --> 00:00:35
I thought I would just go ahead
and tell you what questions
13
00:00:35 --> 00:00:37
there are, so you could see.
14
00:00:37 --> 00:00:42
I haven't filled in all the
numbers, but this will tell
15
00:00:42 --> 00:00:47
you, it's a way of reviewing
of course, to sort of see
16
00:00:47 --> 00:00:50
the things that we've done.
17
00:00:50 --> 00:00:52
Which is quite a bit, really.
18
00:00:52 --> 00:00:58
And also of course topics that
will not be in the exam.
19
00:00:58 --> 00:01:01
You can see that they're not
in the exam, for example.
20
00:01:01 --> 00:01:07
Topics from 1.7 on condition
number, or 2.3 on Gram-Schmidt
21
00:01:07 --> 00:01:09
that I can speak
about a little bit.
22
00:01:09 --> 00:01:13
So those are the, only
thing I didn't fill in
23
00:01:13 --> 00:01:16
is the n time there.
24
00:01:16 --> 00:01:19
So I don't usually ask
four questions, three
25
00:01:19 --> 00:01:20
is more typical.
26
00:01:20 --> 00:01:25
So it's a little bit longer but
it's not a difficult exam.
27
00:01:25 --> 00:01:33
And I never want time to be
the essential ingredient.
28
00:01:33 --> 00:01:38
So 7:30 to 9 is the nominal
time, but it would not
29
00:01:38 --> 00:01:43
be a surprise if you're
still going on after 9
30
00:01:43 --> 00:01:45
o'clock a little bit.
31
00:01:45 --> 00:01:50
And I'll try not to, like,
tear a paper away from you.
32
00:01:50 --> 00:01:55
So, just figure that if you
move along at a reasonable
33
00:01:55 --> 00:02:01
speed, and so you can
bring papers, the book,
34
00:02:01 --> 00:02:04
anything with you.
35
00:02:04 --> 00:02:09
And I'm open for questions
now about the exam.
36
00:02:09 --> 00:02:13
You know where 54-100 is, it's
a classroom unfortunately.
37
00:02:13 --> 00:02:17
Not a, sometimes we'll have the
top of Walker where you have a
38
00:02:17 --> 00:02:22
whole table to work on, that's
this 54-100 in the tallest
39
00:02:22 --> 00:02:27
building of MIT out in the
middle of the middle space out
40
00:02:27 --> 00:02:31
there, is a large classroom.
41
00:02:31 --> 00:02:38
So if you kind of spread out
your papers we'll be OK.
42
00:02:38 --> 00:02:40
It's a pretty big room.
43
00:02:40 --> 00:02:42
Questions.
44
00:02:42 --> 00:02:47
And, of course, don't forget
this afternoon in 1-190
45
00:02:47 --> 00:02:48
rather than here.
46
00:02:48 --> 00:02:58
So, review is in
1-190, 4 to 5 today.
47
00:02:58 --> 00:03:05
But if you're not free at that
time don't feel that you've
48
00:03:05 --> 00:03:07
missed anything essential.
49
00:03:07 --> 00:03:13
I thought this would be the
right way to tell you what
50
00:03:13 --> 00:03:18
the exam is and guide
your preparation for it.
51
00:03:18 --> 00:03:19
Questions.
52
00:03:19 --> 00:03:19
Yes, thanks.
53
00:03:19 --> 00:03:26
AUDIENCE: [INAUDIBLE].
54
00:03:26 --> 00:03:30
PROFESSOR STRANG: So these are
four different questions.
55
00:03:30 --> 00:03:36
So this would be a question
about getting to this matrix
56
00:03:36 --> 00:03:40
and what it's about,
A transpose C A.
57
00:03:40 --> 00:03:49
AUDIENCE: [INAUDIBLE]
58
00:03:49 --> 00:03:52
PROFESSOR STRANG: OK, I don't
use beam bending for that.
59
00:03:52 --> 00:03:53
I'm thinking of elastic bar.
60
00:03:53 --> 00:03:57
Yeah, stretching
equation, yeah.
61
00:03:57 --> 00:04:02
So the stretching equation, so
the first one we ever saw, the
62
00:04:02 --> 00:04:07
stretching equation, was
just u'', or -u''=1.
63
00:04:08 --> 00:04:14
And now that allows a C to
sneak in and that allows a
64
00:04:14 --> 00:04:20
matrix C to sneak in there, but
I think you'll see what they
65
00:04:20 --> 00:04:21
should look like, but yeah.
66
00:04:21 --> 00:04:24
AUDIENCE: [INAUDIBLE]
67
00:04:24 --> 00:04:26
PROFESSOR STRANG: Yeah, yeah.
68
00:04:26 --> 00:04:30
So this is Section 2.2,
directly out of the book.
69
00:04:30 --> 00:04:32
M will be the mass
matrix, K will be the
70
00:04:32 --> 00:04:35
stiffness matrix, yep.
71
00:04:35 --> 00:04:39
AUDIENCE: [INAUDIBLE]
72
00:04:39 --> 00:04:39
PROFESSOR STRANG: OK.
73
00:04:39 --> 00:04:43
So, good point to say.
74
00:04:43 --> 00:04:48
In the very first section, 1.1,
we gave the name K to a very
75
00:04:48 --> 00:04:51
special matrix, a specific one.
76
00:04:51 --> 00:04:57
But then later, now, I'm
using the same letter K
77
00:04:57 --> 00:04:59
for matrices of that type.
78
00:04:59 --> 00:05:03
That was the most special,
simplest completely understood
79
00:05:03 --> 00:05:09
case, but now I'll use K for
stiffness matrices and when
80
00:05:09 --> 00:05:13
we're doing finite elements in
a few weeks again it'll be K.
81
00:05:13 --> 00:05:15
Yeah, same name, right.
82
00:05:15 --> 00:05:21
So here you'll want to create
K and M and know how to deal
83
00:05:21 --> 00:05:26
with, this was our only
time dependent thing.
84
00:05:26 --> 00:05:29
So I guess what you're seeing
here is not only what time
85
00:05:29 --> 00:05:33
dependent equation will be
there but also that I'm not
86
00:05:33 --> 00:05:42
going in detail into those
trapezoidal difference methods.
87
00:05:42 --> 00:05:45
Important as they are, we can't
do everything on the quiz so
88
00:05:45 --> 00:05:50
I'm really focusing on things
that were central
89
00:05:50 --> 00:05:55
to our course.
90
00:05:55 --> 00:05:56
Good.
91
00:05:56 --> 00:05:57
Other questions.
92
00:05:57 --> 00:06:01
I'm very open for more
questions this afternoon.
93
00:06:01 --> 00:06:01
Yep.
94
00:06:01 --> 00:06:04
AUDIENCE: [INAUDIBLE]
95
00:06:04 --> 00:06:05
PROFESSOR STRANG: Others.
96
00:06:05 --> 00:06:05
OK.
97
00:06:05 --> 00:06:12
So, let me, I don't want to go
on and do new material, because
98
00:06:12 --> 00:06:15
we're focused on these things.
99
00:06:15 --> 00:06:19
And this course, the name of
this course is computational
100
00:06:19 --> 00:06:20
science and engineering.
101
00:06:20 --> 00:06:24
And by the way I just had an
email last week from the dean
102
00:06:24 --> 00:06:29
of engineering, or a bunch of
us did, to say that the school
103
00:06:29 --> 00:06:33
of engineering is establishing
a center for computational
104
00:06:33 --> 00:06:36
engineering, CCE.
105
00:06:36 --> 00:06:40
Several faculty members there
and, like myself in the School
106
00:06:40 --> 00:06:45
of Science and the Sloan School
are involved with computation,
107
00:06:45 --> 00:06:50
and this new center is
going to organize that.
108
00:06:50 --> 00:06:53
So, it's a good development.
109
00:06:53 --> 00:06:58
And it's headed by people
in Course 2 and Course 16.
110
00:06:58 --> 00:06:59
So.
111
00:06:59 --> 00:07:03
If we're talking about
computations, and I do have to
112
00:07:03 --> 00:07:06
say something about how you
would actually do
113
00:07:06 --> 00:07:08
the computations.
114
00:07:08 --> 00:07:12
And what are the issues
about accuracy.
115
00:07:12 --> 00:07:17
Speed and accuracy is what
you're aiming for in
116
00:07:17 --> 00:07:19
the computations.
117
00:07:19 --> 00:07:22
Of course, the first step
is to know what problem is
118
00:07:22 --> 00:07:23
that you want to compute.
119
00:07:23 --> 00:07:26
What do you want to solve,
what's the equation?
120
00:07:26 --> 00:07:28
That's what we've been
doing all along.
121
00:07:28 --> 00:07:33
Now, I just take a little
time out to say, suppose
122
00:07:33 --> 00:07:35
I have the equation.
123
00:07:35 --> 00:07:39
When I write K, I'm thinking of
a symmetric positive, definite
124
00:07:39 --> 00:07:42
or at least semi
definite matrix.
125
00:07:42 --> 00:07:46
When I write A I'm thinking
of any general, usually
126
00:07:46 --> 00:07:48
tall, thin, matrix.
127
00:07:48 --> 00:07:49
Rectangular.
128
00:07:49 --> 00:07:54
So that I would need the
squares for this guy, where
129
00:07:54 --> 00:07:58
straightforward elimination
would work for that one.
130
00:07:58 --> 00:08:02
And so my first question is
let's take this, so these
131
00:08:02 --> 00:08:05
are two topics for today.
132
00:08:05 --> 00:08:09
This one would come out
of 1.7, that discussion
133
00:08:09 --> 00:08:11
with condition number.
134
00:08:11 --> 00:08:16
This one would come out of 2.3,
the least squares section.
135
00:08:16 --> 00:08:23
OK, so if I give you, and I'm
thinking that the computational
136
00:08:23 --> 00:08:26
questions emerge when
the systems are large.
137
00:08:26 --> 00:08:28
So I'm thinking thousands
of unknowns here.
138
00:08:28 --> 00:08:31
Thousands of equations,
at the least.
139
00:08:31 --> 00:08:31
OK.
140
00:08:31 --> 00:08:40
So, and the question is, I do
Gaussian elimination here,
141
00:08:40 --> 00:08:42
ordinary elimination.
142
00:08:42 --> 00:08:43
Backslash.
143
00:08:43 --> 00:08:48
And how accurate is the answer?
144
00:08:48 --> 00:08:49
And how do you understand?
145
00:08:49 --> 00:08:52
I mean, the accuracy of the
answer is going to kind
146
00:08:52 --> 00:08:53
of depend on two things.
147
00:08:53 --> 00:08:55
And it's good to separate them.
148
00:08:55 --> 00:09:01
One is the method you
use, like elimination.
149
00:09:01 --> 00:09:03
Whatever adjustments
you might make.
150
00:09:03 --> 00:09:04
Pivoting.
151
00:09:04 --> 00:09:07
Exchanging rows to
get larger pivots.
152
00:09:07 --> 00:09:10
All that is in the
algorithm, in the code.
153
00:09:10 --> 00:09:13
And then the second, very
important aspect, is
154
00:09:13 --> 00:09:16
the matrix K itself.
155
00:09:16 --> 00:09:19
Is this a tough problem
to solve whatever method
156
00:09:19 --> 00:09:21
you're using, or is
it a simple problem?
157
00:09:21 --> 00:09:26
Is the problem ill condition,
meeting K would be like nearly
158
00:09:26 --> 00:09:30
singular, and then we would
know we had a tougher problem
159
00:09:30 --> 00:09:35
to solve whatever method, or is
K quite well conditioned, I
160
00:09:35 --> 00:09:39
mean the best condition would
be when all the columns
161
00:09:39 --> 00:09:43
are unit vectors and all
orthogonal to each other.
162
00:09:43 --> 00:09:46
Yeah, I mean that would be the
best conditioning of all.
163
00:09:46 --> 00:09:51
That condition number would
be one if this K, which,
164
00:09:51 --> 00:09:56
not too likely is a matrix
that I would call Q.
165
00:09:56 --> 00:10:01
Q, which is going to show up
over here, in the second
166
00:10:01 --> 00:10:06
problem is, Q stands
for a matrix which has
167
00:10:06 --> 00:10:08
orthonormal columns.
168
00:10:08 --> 00:10:17
So, you remember what
orthonormal means.
169
00:10:17 --> 00:10:20
Ortho is telling us
perpendicular, that's
170
00:10:20 --> 00:10:21
the key point.
171
00:10:21 --> 00:10:24
Normal is telling us that
they're unit vectors,
172
00:10:24 --> 00:10:25
lengths one.
173
00:10:25 --> 00:10:28
So that's the Q, and then
you might ask what's the R.
174
00:10:28 --> 00:10:35
And the R is upper triangular.
175
00:10:35 --> 00:10:38
OK.
176
00:10:38 --> 00:10:39
OK.
177
00:10:39 --> 00:10:42
So what I said about this
problem, that there's the
178
00:10:42 --> 00:10:47
method you use, and also the
sensitivity, the difficulty of
179
00:10:47 --> 00:10:51
the problem in the first place,
applies just the same here.
180
00:10:51 --> 00:10:58
There's the method you use, do
you use A transpose A to find u
181
00:10:58 --> 00:11:01
hat, this is, now we're looking
for u hat, of course,
182
00:11:01 --> 00:11:03
the best solution.
183
00:11:03 --> 00:11:05
Do I use A transpose A?
184
00:11:05 --> 00:11:09
Well, you would say of
course, what else.
185
00:11:09 --> 00:11:12
That equation, that least
squares equation has A
186
00:11:12 --> 00:11:16
transpose Au hat equal A
transpose b, what's the choice?
187
00:11:16 --> 00:11:25
But, if you're interested in
high accuracy, and stability,
188
00:11:25 --> 00:11:29
numerical stability, maybe you
don't go to A transpose A.
189
00:11:29 --> 00:11:34
Going to A transpose A kind of
squares the condition number.
190
00:11:34 --> 00:11:37
You get to an A transpose A,
that'll be our ok, but its
191
00:11:37 --> 00:11:43
condition number will somehow
be squared and if the problem
192
00:11:43 --> 00:11:46
is nice, you're OK with that.
193
00:11:46 --> 00:11:48
But if the problem is delicate?
194
00:11:48 --> 00:11:50
Now, what does delicate
mean for Au=b?
195
00:11:51 --> 00:11:55
I'm kind of giving you a
overview of the two problems
196
00:11:55 --> 00:11:57
before I start on this one.
197
00:11:57 --> 00:11:59
And then that one.
198
00:11:59 --> 00:12:01
So with this one the
problem was, is the
199
00:12:01 --> 00:12:03
matrix nearly singular.
200
00:12:03 --> 00:12:05
What does that mean?
201
00:12:05 --> 00:12:07
Does MATLAB tell you, what
does MATLAB tell you
202
00:12:07 --> 00:12:09
about the matrix?
203
00:12:09 --> 00:12:12
And that is measured by
the condition number.
204
00:12:12 --> 00:12:20
What's the issue here is, when
would this be a numerically
205
00:12:20 --> 00:12:22
difficult, sensitive problem?
206
00:12:22 --> 00:12:27
Well, the columns of A
are not orthonormal.
207
00:12:27 --> 00:12:30
If they are, then
you're golden.
208
00:12:30 --> 00:12:33
If the columns of A
are orthonormal,
209
00:12:33 --> 00:12:36
then you're all set.
210
00:12:36 --> 00:12:39
So what's the opposite?
211
00:12:39 --> 00:12:42
Well, the extreme opposite
would be when the columns
212
00:12:42 --> 00:12:44
of A are dependent.
213
00:12:44 --> 00:12:48
If the columns of A are
linearly dependent and some
214
00:12:48 --> 00:12:50
column is a combination
of other columns, you're
215
00:12:50 --> 00:12:52
in trouble right away.
216
00:12:52 --> 00:12:56
So that's like big trouble,
that's like K being singular.
217
00:12:56 --> 00:12:57
Those are the cases.
218
00:12:57 --> 00:13:03
K singular here,
dependent columns here.
219
00:13:03 --> 00:13:05
Not full rank.
220
00:13:05 --> 00:13:11
So, again we're supposing
we're not facing disaster.
221
00:13:11 --> 00:13:12
Just near disaster.
222
00:13:12 --> 00:13:16
So we want to know, is K near
the singular and how to measure
223
00:13:16 --> 00:13:22
that, and we want to know what
to do when the columns of A are
224
00:13:22 --> 00:13:28
independent but maybe not very.
225
00:13:28 --> 00:13:31
And that would show up in
a large condition number
226
00:13:31 --> 00:13:33
for A transpose A.
227
00:13:33 --> 00:13:36
And this happens all the time;
if you don't set up your
228
00:13:36 --> 00:13:44
problem well, your experimental
problem, you can easily get
229
00:13:44 --> 00:13:49
matrices A, whose columns
are not very independent.
230
00:13:49 --> 00:13:54
Measured by A transpose A
being close to singular.
231
00:13:54 --> 00:13:56
Right, everybody
here's got that idea.
232
00:13:56 --> 00:13:59
If the columns of A are
independent, A transpose
233
00:13:59 --> 00:14:01
A is non-singular.
234
00:14:01 --> 00:14:03
Fact, positive, definite.
235
00:14:03 --> 00:14:13
Then now we're talking about
when we have that property, but
236
00:14:13 --> 00:14:17
the columns of A are not very
independent and the matrix A
237
00:14:17 --> 00:14:20
transpose A is not
very invertible.
238
00:14:20 --> 00:14:23
OK, so that's what
the things are.
239
00:14:23 --> 00:14:28
And then just to, because, say
on this one, what's the good
240
00:14:28 --> 00:14:36
thing to do? the good thing to
do is to call the qr code which
241
00:14:36 --> 00:14:43
gets its name because it takes
the matrix A and it factors it.
242
00:14:43 --> 00:14:49
Of course, we all know
that lu is the code here.
243
00:14:49 --> 00:14:53
It factors K.
244
00:14:53 --> 00:15:01
And qr is the code that factors
A into a very good guy.
245
00:15:01 --> 00:15:03
An optimal queue.
246
00:15:03 --> 00:15:07
It couldn't be b. and an R,
that's upper triangular, and
247
00:15:07 --> 00:15:11
therefore in the simplest
form you see exactly what
248
00:15:11 --> 00:15:15
you're dealing with.
249
00:15:15 --> 00:15:24
Let me continue this
least squares idea.
250
00:15:24 --> 00:15:27
Because Q and R are
probably not so familiar.
251
00:15:27 --> 00:15:30
Maybe the name
Gram-Schmidt is familiar?
252
00:15:30 --> 00:15:34
How many have seen
Gram-Schmidt?
253
00:15:34 --> 00:15:38
The Gram-Schmidt idea I'll
describe quickly, but just do
254
00:15:38 --> 00:15:41
those words, do those guys'
names mean anything to?
255
00:15:41 --> 00:15:42
Yes, if they do.
256
00:15:42 --> 00:15:43
Quite a few.
257
00:15:43 --> 00:15:44
But not all.
258
00:15:44 --> 00:15:45
OK.
259
00:15:45 --> 00:15:47
OK.
260
00:15:47 --> 00:15:53
And I can I just say, also
Gram-Schmidt is kind of
261
00:15:53 --> 00:15:57
our name for getting
these two factors.
262
00:15:57 --> 00:16:01
And you'll see why, it's
very cool to have, why
263
00:16:01 --> 00:16:06
this is a good first step.
264
00:16:06 --> 00:16:09
It costs a little to take that
step, but if you're interested
265
00:16:09 --> 00:16:12
in safety, take it.
266
00:16:12 --> 00:16:16
It might cost twice as much as
solving the A transpose A
267
00:16:16 --> 00:16:21
equation, so you double the
cost by going this safer route.
268
00:16:21 --> 00:16:24
And double is not a
big deal, usually.
269
00:16:24 --> 00:16:26
OK.
270
00:16:26 --> 00:16:29
So, I was going to say that
Gram-Schmidt, that's the
271
00:16:29 --> 00:16:31
name everybody uses.
272
00:16:31 --> 00:16:37
But actually their method
is no longer the winner.
273
00:16:37 --> 00:16:42
And in Section 2.3 and I'll try
to describe a slightly better
274
00:16:42 --> 00:16:48
method than the Gram-Schmidt
idea to arrive at Q and R.
275
00:16:48 --> 00:16:50
But let's suppose
you got to Q and R.
276
00:16:50 --> 00:16:53
Then, what would be the
least squares equation?
277
00:16:53 --> 00:16:59
A transpose Au hat is
A transpose b, right?
278
00:16:59 --> 00:17:02
That's the equation
everybody knows.
279
00:17:02 --> 00:17:08
But now if we have A factored
into Q times R, let me
280
00:17:08 --> 00:17:09
see how that simplifies.
281
00:17:09 --> 00:17:15
So now A is QR, and A
transpose, of course, is R
282
00:17:15 --> 00:17:20
transpose Q transpose u hat,
and this is R transpose
283
00:17:20 --> 00:17:23
Q transpose b.
284
00:17:23 --> 00:17:27
Same equation, I'm just
supposing that I've got a into
285
00:17:27 --> 00:17:32
this nice form where Q has,
where I've taken these columns
286
00:17:32 --> 00:17:37
that possibly lined up too
close to each other like, you
287
00:17:37 --> 00:17:40
know, angles of one degree.
288
00:17:40 --> 00:17:42
And I've got better angles.
289
00:17:42 --> 00:17:47
I've got these columns
of A are too close.
290
00:17:47 --> 00:17:50
So I spread them out,
two columns of qQ that
291
00:17:50 --> 00:17:52
are at 90 degrees.
292
00:17:52 --> 00:17:54
Orthogonal columns.
293
00:17:54 --> 00:17:57
Now, what's the deal with
orthogonal columns?
294
00:17:57 --> 00:18:03
Let me just remember the
main point about Q.
295
00:18:03 --> 00:18:06
It has orthogonal columns,
right, and I'll call
296
00:18:06 --> 00:18:11
those q's. q_1 to q_n.
297
00:18:11 --> 00:18:12
OK.
298
00:18:12 --> 00:18:14
And the good deal is
what happens when I
299
00:18:14 --> 00:18:16
do Q transpose Q.
300
00:18:16 --> 00:18:23
So I do q_1 transpose, these
now rows to qn transpose.
301
00:18:23 --> 00:18:27
q_1 columns to q_n.
302
00:18:27 --> 00:18:31
And what do I get when I
multiply those matrices?
303
00:18:31 --> 00:18:34
Q transpose times Q.
304
00:18:34 --> 00:18:39
I get I. q_1 transpose q_1,
that's the length of q_1
305
00:18:39 --> 00:18:44
squared is one, and q_1 is
orthogonal to all the others.
306
00:18:44 --> 00:18:50
And then q_2, you see I get
the I. q_3, get an n by n.
307
00:18:50 --> 00:18:52
I get the identity matrix.
308
00:18:52 --> 00:18:55
Q transpose Q is I.
309
00:18:55 --> 00:19:00
That's the beautiful,
just remember that fact.
310
00:19:00 --> 00:19:02
And use it right away.
311
00:19:02 --> 00:19:05
You see where it's used,
Q transpose Q is I in
312
00:19:05 --> 00:19:08
the middle of that.
313
00:19:08 --> 00:19:12
So I can just delete that, I
just have R transpose R and I
314
00:19:12 --> 00:19:14
can even simplify this further.
315
00:19:14 --> 00:19:15
What can I do now?
316
00:19:15 --> 00:19:20
So that's the identity, so I
have R transpose R, but now I
317
00:19:20 --> 00:19:25
have an R transpose over
here so am I left with?
318
00:19:25 --> 00:19:29
I'll multiply both sides by R
transpose inverse and that will
319
00:19:29 --> 00:19:36
lead me to, Ru hat equals,
knocking out our transpose
320
00:19:36 --> 00:19:37
inverse on both sides.
321
00:19:37 --> 00:19:40
Q transpose b.
322
00:19:40 --> 00:19:45
Well, that's, our least
squares equation has become
323
00:19:45 --> 00:19:47
completely easy to solve.
324
00:19:47 --> 00:19:50
We've got a triangular matrix
here, I mean it's just
325
00:19:50 --> 00:19:52
back substitution.
326
00:19:52 --> 00:19:56
It's just back substitution
now, and a Q transpose
327
00:19:56 --> 00:19:57
b over there.
328
00:19:57 --> 00:20:05
So a very simple solution
for our equation after
329
00:20:05 --> 00:20:07
the initial work of A=QR.
330
00:20:09 --> 00:20:10
OK.
331
00:20:10 --> 00:20:18
But very safe, Q is a great
matrix to work with.
332
00:20:18 --> 00:20:23
In fact people, codes are
written so as to use
333
00:20:23 --> 00:20:27
orthogonal matrices Q
as often as they can.
334
00:20:27 --> 00:20:34
Alright, so you had a look
ahead of the computational side
335
00:20:34 --> 00:20:39
of 2.3, let me come back to the
most basic equations just
336
00:20:39 --> 00:20:42
symmetric, positive,
definite equations.
337
00:20:42 --> 00:20:52
Ku=f, and consider OK, how
do we measure whether
338
00:20:52 --> 00:20:55
K is nearly singular.
339
00:20:55 --> 00:20:57
OK, let me just ask
that question.
340
00:20:57 --> 00:21:00
That's the central question.
341
00:21:00 --> 00:21:08
How to measure, when K,
which is, we're assuming
342
00:21:08 --> 00:21:13
be symmetric positive
definite, nearly singular?
343
00:21:13 --> 00:21:20
How to measure that?
344
00:21:20 --> 00:21:24
How to know whether we're
in any danger or not?
345
00:21:24 --> 00:21:25
OK.
346
00:21:25 --> 00:21:30
Well, first you might think
OK, if it is singular
347
00:21:30 --> 00:21:33
its determinate is zero.
348
00:21:33 --> 00:21:36
So why not take
its determinant?
349
00:21:36 --> 00:21:41
Well determinants, as we've
said, are not a good
350
00:21:41 --> 00:21:45
idea numerically.
351
00:21:45 --> 00:21:47
First, they're not
fun to compute.
352
00:21:47 --> 00:21:53
Second, they depend on the
number of unknowns, right?
353
00:21:53 --> 00:21:58
If I just have twice the
identity, suppose k is
354
00:21:58 --> 00:22:01
twice the identity matrix.
355
00:22:01 --> 00:22:04
You could not get a better
problem than that, right?
356
00:22:04 --> 00:22:06
If k was twice the
identity matrix the
357
00:22:06 --> 00:22:08
whole thing's simple.
358
00:22:08 --> 00:22:12
Or if K is, suppose K is
one millionth of the
359
00:22:12 --> 00:22:15
identity matrix.
360
00:22:15 --> 00:22:18
OK, again, that's a
perfect problem, right?
361
00:22:18 --> 00:22:24
If K is one millionth of the
identity matrix, well to solve
362
00:22:24 --> 00:22:26
the problem you just multiply
by a million, you've
363
00:22:26 --> 00:22:27
got the answer.
364
00:22:27 --> 00:22:29
So those are good.
365
00:22:29 --> 00:22:34
And we have to have some
measure of bad or good that
366
00:22:34 --> 00:22:37
tells us those are good.
367
00:22:37 --> 00:22:40
OK.
368
00:22:40 --> 00:22:42
So the determinant won't do.
369
00:22:42 --> 00:22:47
Because the determinant of two
I would be two to the nth,
370
00:22:47 --> 00:22:48
the size of the matrix.
371
00:22:48 --> 00:22:52
Or the determinant of one
millionth identity would be
372
00:22:52 --> 00:22:53
one millionth to the n.
373
00:22:53 --> 00:22:56
Those are not numbers we want.
374
00:22:56 --> 00:22:57
What's a better number?
375
00:22:57 --> 00:23:01
Maybe you could suggest a
better number to measure
376
00:23:01 --> 00:23:05
how close is the matrix
to being singular.
377
00:23:05 --> 00:23:08
What would you say?
378
00:23:08 --> 00:23:10
I think if you think about
it a little, so what
379
00:23:10 --> 00:23:13
numbers do we know?
380
00:23:13 --> 00:23:17
Well eigenvalues jumps to mind.
381
00:23:17 --> 00:23:18
Eigenvalues jumps to mind.
382
00:23:18 --> 00:23:22
Because this matrix K, being
symmetric positive definite,
383
00:23:22 --> 00:23:31
has eigenvalues say lambda_1
less than lambda_2, so on.
384
00:23:31 --> 00:23:32
So on.
385
00:23:32 --> 00:23:40
Up to, so this is lambda_max
and that's lambda_min,
386
00:23:40 --> 00:23:42
and they're all positive.
387
00:23:42 --> 00:23:50
And so what's your idea
of whether the thing's
388
00:23:50 --> 00:23:52
nearly singular now?
389
00:23:52 --> 00:23:54
Look at lambda_1, right?
390
00:23:54 --> 00:23:57
If lambda_1 is near zero,
that somehow indicates
391
00:23:57 --> 00:23:59
near singular.
392
00:23:59 --> 00:24:02
So lambda_1 is sort
of a natural test.
393
00:24:02 --> 00:24:05
Not that I intend to compute
lambda_1, that would take
394
00:24:05 --> 00:24:08
longer than solving the system.
395
00:24:08 --> 00:24:10
But an estimate of
lambda_1 would be enough.
396
00:24:10 --> 00:24:13
OK.
397
00:24:13 --> 00:24:17
But my answer is
not just lambda_1.
398
00:24:17 --> 00:24:20
399
00:24:20 --> 00:24:22
And why is that?
400
00:24:22 --> 00:24:27
Because the examples I gave
you, when I had twice the
401
00:24:27 --> 00:24:31
identity, what would lambda_1
be there in that case?
402
00:24:31 --> 00:24:35
If my matrix K was beautiful,
twice the identity matrix,
403
00:24:35 --> 00:24:40
lambda_1 would be two.
404
00:24:40 --> 00:24:44
All the eigenvalues are two
for the identity matrix.
405
00:24:44 --> 00:24:47
Now if my matrix was
one millionth of the
406
00:24:47 --> 00:24:50
identity, again I have
a beautiful problem.
407
00:24:50 --> 00:24:52
Just as good, just as
beautiful problem.
408
00:24:52 --> 00:24:55
What's lambda_1 for that one?
409
00:24:55 --> 00:24:56
One millionth.
410
00:24:56 --> 00:24:59
It looks not as good, it looks
much more singular, but
411
00:24:59 --> 00:25:05
that's not really there.
412
00:25:05 --> 00:25:11
So you could say, we'll
scale your matrix.
413
00:25:11 --> 00:25:15
And scaling the matrices, in
fact scaling individual rows
414
00:25:15 --> 00:25:19
and columns to get it, you
might have used, your
415
00:25:19 --> 00:25:25
unknowns might be somehow
in the wrong units.
416
00:25:25 --> 00:25:29
So one of the answers is
way big and the second
417
00:25:29 --> 00:25:30
component is way small.
418
00:25:30 --> 00:25:33
That's not good.
419
00:25:33 --> 00:25:36
So scaling is important.
420
00:25:36 --> 00:25:43
But even then you still end
up with a matrix K, some
421
00:25:43 --> 00:25:47
eigenvalues and I'll tell
you the condition number.
422
00:25:47 --> 00:25:53
The condition number
of K is the ratio of
423
00:25:53 --> 00:25:55
this guy to this one.
424
00:25:55 --> 00:25:59
In other words, two K, or a
million K, or one millionth
425
00:25:59 --> 00:26:02
K, all have the same
condition number.
426
00:26:02 --> 00:26:06
Because those problems
are identical problems.
427
00:26:06 --> 00:26:09
Multiplying by two, multiplying
by a million, dividing by a
428
00:26:09 --> 00:26:12
million didn't change
reality there.
429
00:26:12 --> 00:26:17
So if we're in floating points,
it just didn't change.
430
00:26:17 --> 00:26:19
So the condition number
is going to be
431
00:26:19 --> 00:26:19
lambda_max/lambda_min.
432
00:26:19 --> 00:26:24
433
00:26:24 --> 00:26:31
And this is for symmetric
positive definite matrices.
434
00:26:31 --> 00:26:35
And and MATLAB will
print out that number.
435
00:26:35 --> 00:26:38
Or print an estimate for that
number; as I said we don't
436
00:26:38 --> 00:26:39
want to compute it exactly.
437
00:26:39 --> 00:26:40
Lambda_max/lambda_min.
438
00:26:41 --> 00:26:48
That measures how sensitive,
how tough your problem is.
439
00:26:48 --> 00:26:49
OK.
440
00:26:49 --> 00:26:55
And then I have to think how
does that come in, why is
441
00:26:55 --> 00:26:57
that an appropriate number?
442
00:26:57 --> 00:27:00
I guess I've tried to give
an instinct for why it's
443
00:27:00 --> 00:27:05
appropriate, but we can be
pretty specific about it.
444
00:27:05 --> 00:27:08
In fact, let's do that now.
445
00:27:08 --> 00:27:13
So what would be the condition
number of twice the identity?
446
00:27:13 --> 00:27:15
It would be one.
447
00:27:15 --> 00:27:17
Perfectly conditioned problem.
448
00:27:17 --> 00:27:20
What would be the
condition, yeah, OK.
449
00:27:20 --> 00:27:25
What would be the condition of
a diagonal matrix suppose K was
450
00:27:25 --> 00:27:32
a diagonal matrix
two, three, four?
451
00:27:32 --> 00:27:36
The condition number of
that matrix is two, right?
452
00:27:36 --> 00:27:39
Lambda_max is sitting there,
lambda_min is sitting
453
00:27:39 --> 00:27:41
there, the ratio is two.
454
00:27:41 --> 00:27:44
Of course, any condition
number under 100 or
455
00:27:44 --> 00:27:47
1000 is no problem.
456
00:27:47 --> 00:27:52
Roughly the rule of thumb
is that the, what's
457
00:27:52 --> 00:27:53
the rule of thumb?
458
00:27:53 --> 00:27:56
I think that maybe the number
of digits in the condition
459
00:27:56 --> 00:28:04
number, the number of digits,
maybe if the condition number
460
00:28:04 --> 00:28:08
was 1000 you would be taking a
chance that your last three
461
00:28:08 --> 00:28:14
digits, single precision that
would be three out of six,
462
00:28:14 --> 00:28:20
three bits, somehow the log of
the condition number, the
463
00:28:20 --> 00:28:23
number of digits in it, is some
measure of the number
464
00:28:23 --> 00:28:26
of digits you'd lose.
465
00:28:26 --> 00:28:30
Because you're doing floating
point, of course, here.
466
00:28:30 --> 00:28:33
That's it, totally well
conditioned matrix.
467
00:28:33 --> 00:28:35
I wouldn't touch that one.
468
00:28:35 --> 00:28:37
I mean that's just fine.
469
00:28:37 --> 00:28:45
But we can figure out, here's
the point that I should make.
470
00:28:45 --> 00:28:48
Because here's the
computational science point.
471
00:28:48 --> 00:28:57
When this is our special
K, our -1, 2, -1 matrix.
472
00:28:57 --> 00:29:03
-1, 2, -1 matrix of size
n, the condition number
473
00:29:03 --> 00:29:09
goes, like, n squared.
474
00:29:09 --> 00:29:11
Because we know the
eigenvalues of that
475
00:29:11 --> 00:29:13
matrix, we could see it.
476
00:29:13 --> 00:29:18
The largest eigenvalue, when
n is big, say n is 1000.
477
00:29:18 --> 00:29:21
And we're dealing with our
standard second difference
478
00:29:21 --> 00:29:24
matrix, the most important
example I could
479
00:29:24 --> 00:29:25
possibly present.
480
00:29:25 --> 00:29:30
Then the largest eigenvalues we
actually are there in Section
481
00:29:30 --> 00:29:37
1.5, we didn't do them in
detail, we'll probably come
482
00:29:37 --> 00:29:39
back to them when we need them.
483
00:29:39 --> 00:29:42
But the largest one
is about four.
484
00:29:42 --> 00:29:46
And the smallest one
is pretty small.
485
00:29:46 --> 00:29:51
The smallest one is sort
of like a sine squared
486
00:29:51 --> 00:29:54
of a small number.
487
00:29:54 --> 00:29:59
And so the smallest eigenvalue
is of order 1/n squared.
488
00:29:59 --> 00:30:05
And then when I do that ratio
of four, lambda_max is just
489
00:30:05 --> 00:30:10
like four, this lambda_min is
like 1/n squared, quite small.
490
00:30:10 --> 00:30:13
That ratio gives
me the n squared.
491
00:30:13 --> 00:30:16
So there's an indication.
492
00:30:16 --> 00:30:19
Basically, that's not bad.
493
00:30:19 --> 00:30:25
That's not bad if n is 1000 in
most engineering problems,
494
00:30:25 --> 00:30:29
that gives you extremely,
extremely good accuracy.
495
00:30:29 --> 00:30:33
Condition number of a million,
you could live with.
496
00:30:33 --> 00:30:37
If n is 100 more typical,
condition number of 10,000
497
00:30:37 --> 00:30:44
is basically, I think OK.
498
00:30:44 --> 00:30:46
And I would go with it.
499
00:30:46 --> 00:30:53
But if the condition number is
way up, then I'd think again,
500
00:30:53 --> 00:30:55
did I model the problem well.
501
00:30:55 --> 00:30:56
OK.
502
00:30:56 --> 00:30:57
Alright.
503
00:30:57 --> 00:31:01
So that's, now I have to tell
you why is this inappropriate?
504
00:31:01 --> 00:31:04
How do you look at the error?
505
00:31:04 --> 00:31:15
So can I write down a way of
approaching this does Ku=f?
506
00:31:15 --> 00:31:21
507
00:31:21 --> 00:31:24
So this is the first time I've
used the word round off error.
508
00:31:24 --> 00:31:29
So in all the calculations, in
all the calculations that you
509
00:31:29 --> 00:31:37
have to do, to get to u, those
row operations and you're doing
510
00:31:37 --> 00:31:41
them to the right side too, so
all those are floating point
511
00:31:41 --> 00:31:44
operations in which small
errors are sneaking in.
512
00:31:44 --> 00:31:51
And it was very unclear, in
the early years, whether the
513
00:31:51 --> 00:31:55
millions and millions of
operations that you do
514
00:31:55 --> 00:31:59
addition, subtractions,
multiplications, and
515
00:31:59 --> 00:32:07
elimination, do those, could
those add up, if they don't
516
00:32:07 --> 00:32:10
cancel, you've got
problems, right?
517
00:32:10 --> 00:32:16
But in general you would expect
somehow that these are just
518
00:32:16 --> 00:32:18
round off errors, you're making
them millions and millions of
519
00:32:18 --> 00:32:23
times, it would be pretty bad
luck, I mean like Red Sox
520
00:32:23 --> 00:32:29
twelfth inning bad luck to
have them pile up on you.
521
00:32:29 --> 00:32:32
So you don't expect that.
522
00:32:32 --> 00:32:37
Now, what you do solve, so
what you actually compute,
523
00:32:37 --> 00:32:41
so this is the exact.
524
00:32:41 --> 00:32:44
This it would be the computed.
525
00:32:44 --> 00:32:48
Let me suppose that the
computed one is sort of
526
00:32:48 --> 00:32:49
a, there's an error.
527
00:32:49 --> 00:32:51
I'll call it delta u.
528
00:32:51 --> 00:32:54
That's our error.
529
00:32:54 --> 00:32:58
And it's equal to
an f plus delta f.
530
00:32:58 --> 00:33:02
And this is our round off
error, this is error
531
00:33:02 --> 00:33:13
we make, and this is
error in the answer.
532
00:33:13 --> 00:33:18
In the final answer.
533
00:33:18 --> 00:33:21
OK, now I would like to
have an error equation.
534
00:33:21 --> 00:33:24
An equation for that error,
delta u, because that's what
535
00:33:24 --> 00:33:26
I'm trying to get an idea of.
536
00:33:26 --> 00:33:28
No problem.
537
00:33:28 --> 00:33:32
If I subtract the exact
equation from this equation, I
538
00:33:32 --> 00:33:35
have a simple error equation.
539
00:33:35 --> 00:33:43
So this is my error equation.
540
00:33:43 --> 00:33:45
OK.
541
00:33:45 --> 00:33:50
So I want to estimate
the size of that error,
542
00:33:50 --> 00:33:54
compared to the exact.
543
00:33:54 --> 00:33:58
You might say, and you would be
right, in saying, well wait a
544
00:33:58 --> 00:34:03
minute as you do all these
operations you're also
545
00:34:03 --> 00:34:05
creating errors in K.
546
00:34:05 --> 00:34:09
So I could have a K plus
delta K here, too.
547
00:34:09 --> 00:34:13
And actually it wouldn't be
difficult to deal with.
548
00:34:13 --> 00:34:18
And would certainly be there
in a proper error analysis.
549
00:34:18 --> 00:34:21
And it wouldn't make a big
difference, the condition
550
00:34:21 --> 00:34:23
number would still be
the right measure.
551
00:34:23 --> 00:34:30
So let me concentrate here on
the error in f, when
552
00:34:30 --> 00:34:34
subtracting one from the other
gives me this simple
553
00:34:34 --> 00:34:37
error equation.
554
00:34:37 --> 00:34:46
So my question is, when is
that error, delta u, big?
555
00:34:46 --> 00:34:49
When do I get a large error?
556
00:34:49 --> 00:34:51
And delta f, I'm
not controlling.
557
00:34:51 --> 00:34:55
I might control the size, but
the details of it I can't know.
558
00:34:55 --> 00:35:03
So what delta f, now I'll
take worst possible here.
559
00:35:03 --> 00:35:08
Suppose this is of some small
size, ten to the minus
560
00:35:08 --> 00:35:14
something, times some vector of
errors, but I don't know
561
00:35:14 --> 00:35:16
anything about that vector and
therefore I'd better take
562
00:35:16 --> 00:35:17
the worst possibility.
563
00:35:17 --> 00:35:19
What would be the
worst possibility?
564
00:35:19 --> 00:35:24
What right hand side would
give me the biggest delta u?
565
00:35:24 --> 00:35:24
Yeah.
566
00:35:24 --> 00:35:29
Maybe that's the right
question to ask.
567
00:35:29 --> 00:35:31
So now we're being a
little pessimistic.
568
00:35:31 --> 00:35:36
We're saying what right hand
side, what set of errors in
569
00:35:36 --> 00:35:40
the measurements or from the
calculations would give
570
00:35:40 --> 00:35:42
me the largest delta u?
571
00:35:42 --> 00:35:49
Well, so let's see.
572
00:35:49 --> 00:35:53
I'm thinking the worst case
would be if delta f was
573
00:35:53 --> 00:36:00
an eigenvector with the
smallest eigenvalue, right?
574
00:36:00 --> 00:36:05
If delta f is an eigenvector,
is x_1, the eigenvector that
575
00:36:05 --> 00:36:15
goes with lambda_1, the worst
case would be for that to
576
00:36:15 --> 00:36:17
be the first eigenvector.
577
00:36:17 --> 00:36:19
That would be the
worst direction.
578
00:36:19 --> 00:36:20
Of course, it would be
multiplied by some
579
00:36:20 --> 00:36:22
little number.
580
00:36:22 --> 00:36:24
Epsilon is every
mathematician's idea
581
00:36:24 --> 00:36:26
of a little number.
582
00:36:26 --> 00:36:26
OK.
583
00:36:26 --> 00:36:32
So epsilon x_1, then
what is delta u?
584
00:36:32 --> 00:36:39
Then the worst delta u is what?
585
00:36:39 --> 00:36:44
What would be the solution
to that equation?
586
00:36:44 --> 00:36:49
If the right hand side was
epsilon and an eigenvector?
587
00:36:49 --> 00:36:51
This is the whole point
of eigenvectors.
588
00:36:51 --> 00:36:54
You can tell me what
the solution is.
589
00:36:54 --> 00:36:57
Is it a multiple of
that eigenvector?
590
00:36:57 --> 00:36:59
You bet.
591
00:36:59 --> 00:37:02
If this is an eigenvector,
then I can put in the same
592
00:37:02 --> 00:37:05
eigenvector there, I just
have to scale it properly.
593
00:37:05 --> 00:37:14
So it'll be just this
side and what do I need?
594
00:37:14 --> 00:37:15
I think I just need lambda_1.
595
00:37:15 --> 00:37:19
596
00:37:19 --> 00:37:23
The worst K inverse can
be is like 1/lambda_1.
597
00:37:24 --> 00:37:30
Right, if I claim that that's
the answer, if the right hand
598
00:37:30 --> 00:37:35
side is sort of in the worst
direction, then the answer is
599
00:37:35 --> 00:37:37
that same right hand side
divided by lambda_1.
600
00:37:39 --> 00:37:40
Let me just check.
601
00:37:40 --> 00:37:44
If I multiply both sides
by K, I have K delta u
602
00:37:44 --> 00:37:46
equals K, what's K*x_1?
603
00:37:48 --> 00:37:49
Everybody with me?
604
00:37:49 --> 00:37:50
What's K*x_1?
605
00:37:51 --> 00:37:54
Lambda_1*x_1, so the lambda_1's
cancel, then I got the
606
00:37:54 --> 00:37:56
epsilon x_1 I want.
607
00:37:56 --> 00:37:59
So, no surprise.
608
00:37:59 --> 00:38:03
That's just telling us that the
worst error is an error in
609
00:38:03 --> 00:38:07
the direction of the low
eigenvector, and that error
610
00:38:07 --> 00:38:08
gets amplified by 1/lambda_1.
611
00:38:10 --> 00:38:11
OK.
612
00:38:11 --> 00:38:15
So that's brought lambda_1,
lambda_min into it,
613
00:38:15 --> 00:38:16
in the denominator.
614
00:38:16 --> 00:38:19
Now, here's another point.
615
00:38:19 --> 00:38:22
Second point now.
616
00:38:22 --> 00:38:25
So that would be the
absolute error.
617
00:38:25 --> 00:38:30
But we saw for those factors of
two and one millionth and so
618
00:38:30 --> 00:38:33
on, that really it's
the relative error.
619
00:38:33 --> 00:38:40
So I want to estimate not the
absolute error, delta u,
620
00:38:40 --> 00:38:45
but the error delta u
relative to u itself.
621
00:38:45 --> 00:38:48
So that if I scale the
whole problem, my relative
622
00:38:48 --> 00:38:50
error wouldn't change.
623
00:38:50 --> 00:38:57
So, in other words, what I want
to do is ask in this case,
624
00:38:57 --> 00:39:00
what's the right hand side?
625
00:39:00 --> 00:39:06
How big, yeah, I know, I
want to know how small
626
00:39:06 --> 00:39:08
u could be, right?
627
00:39:08 --> 00:39:10
I'm shooting for the worst.
628
00:39:10 --> 00:39:20
The relative error is
the size of u, maybe I
629
00:39:20 --> 00:39:21
should put it this way.
630
00:39:21 --> 00:39:27
The relative error is the
size of the error relative
631
00:39:27 --> 00:39:28
to the size of u.
632
00:39:28 --> 00:39:32
And I want to know how
big that could be.
633
00:39:32 --> 00:39:33
OK.
634
00:39:33 --> 00:39:37
So now I know how big delta u
could be, it could be that big.
635
00:39:37 --> 00:39:40
But u itself, how
big could u be?
636
00:39:40 --> 00:39:44
How small could u be, right?
u's in the denominator.
637
00:39:44 --> 00:39:46
So if I'm trying to make
this big, I'll try
638
00:39:46 --> 00:39:47
to make that small.
639
00:39:47 --> 00:39:50
So when is u the smallest?
640
00:39:50 --> 00:39:54
Over there I said when is delta
u the biggest, now I'm going to
641
00:39:54 --> 00:39:56
say when is u the smallest?
642
00:39:56 --> 00:40:03
What f would point me in
the direction in which
643
00:40:03 --> 00:40:07
u was the smallest?
644
00:40:07 --> 00:40:10
Got to be the other
eigenvector.
645
00:40:10 --> 00:40:11
This end.
646
00:40:11 --> 00:40:18
The worst case would be
when this is in the
647
00:40:18 --> 00:40:19
direction of x_n.
648
00:40:19 --> 00:40:21
The top eigenvector.
649
00:40:21 --> 00:40:27
In that case, what is u?
650
00:40:27 --> 00:40:34
So I'm saying the worst f is
the one that makes u smallest,
651
00:40:34 --> 00:40:39
and the worst delta f is the
one that makes delta u biggest.
652
00:40:39 --> 00:40:44
I'm going for the worst case
here, so if the right side is
653
00:40:44 --> 00:40:48
x_n, what is u? x_n/lambda_n.
654
00:40:48 --> 00:40:52
655
00:40:52 --> 00:40:56
Because when I multiply by K, K
times x_n brings me a lambda_n,
656
00:40:56 --> 00:40:58
cancel that lambda_n
I get it right.
657
00:40:58 --> 00:41:04
So there is the smallest u, and
here is the largest delta u.
658
00:41:04 --> 00:41:09
And the epsilon is coming from
the method we use, so that's
659
00:41:09 --> 00:41:12
not involved with the matrix K.
660
00:41:12 --> 00:41:17
So do you see on there, if
I'm trying to estimate
661
00:41:17 --> 00:41:23
delta u over u, that's b.
662
00:41:23 --> 00:41:27
The size of delta to u over
this, the size of delta u
663
00:41:27 --> 00:41:31
over the size of u is what?
664
00:41:31 --> 00:41:36
Delta u has some epsilon that
measures the machine number of
665
00:41:36 --> 00:41:41
digits we're keeping, the
machine length, word
666
00:41:41 --> 00:41:43
length and so on.
667
00:41:43 --> 00:41:46
This is a unit vector
over lambda_1.
668
00:41:46 --> 00:41:48
669
00:41:48 --> 00:41:52
That's delta u over lambda_1,
this u is in the denominator.
670
00:41:52 --> 00:41:57
lambda_n u is down here,
so the lambda_n flips up.
671
00:41:57 --> 00:42:00
Do you see it?
672
00:42:00 --> 00:42:07
By taking the worst case, I've
got the worst relative error.
673
00:42:07 --> 00:42:15
So that for other methods,
other f's and delta f's, they
674
00:42:15 --> 00:42:17
won't be the very worst ones.
675
00:42:17 --> 00:42:22
But here I've written
down what's the worst.
676
00:42:22 --> 00:42:27
And that's the reason that
this is the condition number.
677
00:42:27 --> 00:42:35
So I'm speaking about topics
that are there in 1.7, trying
678
00:42:35 --> 00:42:37
to give you the main point.
679
00:42:37 --> 00:42:41
The main point is, look at
relative error because that's
680
00:42:41 --> 00:42:43
the right thing to look at.
681
00:42:43 --> 00:42:47
Look at the worst cases, which
are in the directions of the
682
00:42:47 --> 00:42:49
top and bottom eigenvectors.
683
00:42:49 --> 00:42:52
In that case, that relative
error has this condition
684
00:42:52 --> 00:42:58
number, lambda_n/lambda_1, and
that's the good measure for
685
00:42:58 --> 00:43:01
how singular the matrix is.
686
00:43:01 --> 00:43:05
So one millionth of the
identity is not a nearly
687
00:43:05 --> 00:43:08
singular matrix, because
lambda_max and lambda_min are
688
00:43:08 --> 00:43:11
equal, that's a perfectly
conditioned matrix.
689
00:43:11 --> 00:43:14
This matrix has condition
number two, 4/2.
690
00:43:14 --> 00:43:16
It's quite good.
691
00:43:16 --> 00:43:21
This matrix is getting worse
with an n squared in there; if
692
00:43:21 --> 00:43:29
n is big and other matrices
could be worse than that.
693
00:43:29 --> 00:43:37
OK. so that's my discussion
of condition numbers.
694
00:43:37 --> 00:43:40
I'll add one more thing.
695
00:43:40 --> 00:43:45
These eigenvalues were a good
measure when my matrix was
696
00:43:45 --> 00:43:47
symmetric positive definite.
697
00:43:47 --> 00:43:53
If I have a matrix that I would
never call K, a matrix like
698
00:43:53 --> 00:44:00
one, one, zero and a million,
OK, that I would never
699
00:44:00 --> 00:44:02
write K for that.
700
00:44:02 --> 00:44:05
I would shoot myself
first before writing K.
701
00:44:05 --> 00:44:08
So what are the eigenvalues
of lambda_min lambda_max
702
00:44:08 --> 00:44:11
for that matrix?
703
00:44:11 --> 00:44:13
Are you up now on eigenvalues?
704
00:44:13 --> 00:44:17
We haven't done a lot of
eigenvalues but triangular
705
00:44:17 --> 00:44:19
matrices are really easy.
706
00:44:19 --> 00:44:21
The eigenvalues of
that matrix are?
707
00:44:21 --> 00:44:23
One and one.
708
00:44:23 --> 00:44:28
The condition number should not
be 1/1, that would be bad.
709
00:44:28 --> 00:44:33
So instead, if this was my
matrix and I wanted to know
710
00:44:33 --> 00:44:36
its condition number,
what would I do?
711
00:44:36 --> 00:44:40
How would I define the
condition number of A?
712
00:44:40 --> 00:44:43
You know what I do whenever
I have a matrix that
713
00:44:43 --> 00:44:46
is not symmetric.
714
00:44:46 --> 00:44:51
I get to a symmetric matrix by
forming A transpose A, I get a
715
00:44:51 --> 00:44:59
K, I take its condition number
by my formula, which I like in
716
00:44:59 --> 00:45:05
a symmetric case, and then I
would take the square root.
717
00:45:05 --> 00:45:08
So that would be a
pretty big number.
718
00:45:08 --> 00:45:13
For this, for that matrix A,
the condition number of that
719
00:45:13 --> 00:45:14
matrix A is up around 10^(6).
720
00:45:16 --> 00:45:19
The condition number of that
matrix is up around 10^(6) even
721
00:45:19 --> 00:45:24
though it's eigenvalues are one
and one because when I form A
722
00:45:24 --> 00:45:29
transpose A, those eigenvalues
will jump all over.
723
00:45:29 --> 00:45:36
And then probably this thing
will have eigenvalues way
724
00:45:36 --> 00:45:41
up and the condition
number will be high.
725
00:45:41 --> 00:45:42
OK.
726
00:45:42 --> 00:45:45
So that's a little, you'll
meet condition number.
727
00:45:45 --> 00:45:49
MATLAB shows it, and you
naturally wonder what is this?
728
00:45:49 --> 00:45:52
Well, if it's a positive
definite matrix, it's just the
729
00:45:52 --> 00:45:56
ratio lambda_max to lambda_min
and it tells you as it gets
730
00:45:56 --> 00:46:00
bigger that means the matrix
is tougher to work with.
731
00:46:00 --> 00:46:01
OK.
732
00:46:01 --> 00:46:05
We have just five minutes left
to say a few words about QR.
733
00:46:07 --> 00:46:11
OK, can I do that in
just a few minutes?
734
00:46:11 --> 00:46:16
And much more is in the codes.
735
00:46:16 --> 00:46:17
OK, what's the deal with QR?
736
00:46:19 --> 00:46:28
I'm starting with a matrix A.
737
00:46:28 --> 00:46:32
Let's make it two by
two, a two by two.
738
00:46:32 --> 00:46:34
OK, it's got a
couple of columns.
739
00:46:34 --> 00:46:36
Can I draw them?
740
00:46:36 --> 00:46:39
So it's got a column there,
that's its first column and
741
00:46:39 --> 00:46:42
it's got another column there,
maybe that's not a very
742
00:46:42 --> 00:46:44
well conditioned matrix.
743
00:46:44 --> 00:46:46
Those are the columns of a.
744
00:46:46 --> 00:46:48
Plotted in the
plane, two space.
745
00:46:48 --> 00:46:55
OK, so now the Gram-Schmidt
idea is out of those columns,
746
00:46:55 --> 00:46:57
get forced to normal columns.
747
00:46:57 --> 00:47:00
Get from A to Q.
748
00:47:00 --> 00:47:04
So the Gram-Schmidt idea is,
out of these two vectors, two
749
00:47:04 --> 00:47:10
axes that are not at 90
degrees produce vectors
750
00:47:10 --> 00:47:12
that are at 90 degrees.
751
00:47:12 --> 00:47:16
Actually, you can guess how
you're going to do it.
752
00:47:16 --> 00:47:19
Let me say, OK I'll settle for
that direction, that can be
753
00:47:19 --> 00:47:22
my first direction, q_1.
754
00:47:22 --> 00:47:26
What should be q_2?
755
00:47:26 --> 00:47:30
If that direction is the
right one for q_1 I say
756
00:47:30 --> 00:47:36
OK I'll settle for that,
what's the q_2 guy?
757
00:47:36 --> 00:47:38
Well, what am I going to do?
758
00:47:38 --> 00:47:43
I mean, Gram thought of it
and Schmidt thought of it.
759
00:47:43 --> 00:47:48
Schmidt was a little later,
but it wasn't, like that
760
00:47:48 --> 00:47:49
hard to think of it.
761
00:47:49 --> 00:47:52
What do you do here?
762
00:47:52 --> 00:47:54
Well, we know how to
do projections from
763
00:47:54 --> 00:47:59
this least squares.
764
00:47:59 --> 00:48:01
What am I looking for?
765
00:48:01 --> 00:48:03
Subtract off the
projection, right.
766
00:48:03 --> 00:48:08
Take the projection and
subtract it off and be
767
00:48:08 --> 00:48:12
left with the component
that's perpendicular.
768
00:48:12 --> 00:48:17
So this will be the q_1
direction and this will be,
769
00:48:17 --> 00:48:20
this guy e, that what we
call e would tell me
770
00:48:20 --> 00:48:22
the q_2 direction.
771
00:48:22 --> 00:48:26
And then we could make
those into unit vectors
772
00:48:26 --> 00:48:29
and we'd be golden.
773
00:48:29 --> 00:48:34
And if we did that we would
discover that the original a_1,
774
00:48:34 --> 00:48:40
a_2 and a_1, so this is the
original first column, and the
775
00:48:40 --> 00:48:46
original second column, would
be the good q_1 and the good
776
00:48:46 --> 00:48:50
q_2 times some matrix R.
777
00:48:50 --> 00:48:51
So here's our A=QR.
778
00:48:53 --> 00:48:57
It's your chance to see this
second major factorization
779
00:48:57 --> 00:48:59
of linear algebra.
780
00:48:59 --> 00:49:03
LU being the first,
QR being the second.
781
00:49:03 --> 00:49:05
So what's up?
782
00:49:05 --> 00:49:09
Well, compare first columns.
783
00:49:09 --> 00:49:12
First columns, I didn't
change direction.
784
00:49:12 --> 00:49:17
So all I have is here is
some scaling r_1,1, zero.
785
00:49:17 --> 00:49:20
Some number times q_1 is a_1.
786
00:49:20 --> 00:49:23
That direction was fine.
787
00:49:23 --> 00:49:31
The second direction, q_2 and
a_2, those involve also a_1.
788
00:49:31 --> 00:49:37
So there's an r_1,2
and an r_2,2 there.
789
00:49:37 --> 00:49:40
The point was, this
came out triangular.
790
00:49:40 --> 00:49:42
And that's what
makes things good.
791
00:49:42 --> 00:49:46
It came out triangular because
of the order that Gram
792
00:49:46 --> 00:49:48
and Schmidt worked.
793
00:49:48 --> 00:49:51
Gram and Schmidt settled
the first one in the
794
00:49:51 --> 00:49:53
first direction.
795
00:49:53 --> 00:49:58
Then they settled the first two
in the first two directions.
796
00:49:58 --> 00:50:01
If we were in three dimensions,
there'd be an a_3 somewhere
797
00:50:01 --> 00:50:04
here, coming out of the board.
798
00:50:04 --> 00:50:09
And then the q_3 would come
straight out of the board.
799
00:50:09 --> 00:50:11
Right?
800
00:50:11 --> 00:50:13
If you just see that
you've got Gram-Schmidt
801
00:50:13 --> 00:50:17
completely. a_1 is there.
802
00:50:17 --> 00:50:19
So is q_1. a_2 is there.
803
00:50:19 --> 00:50:26
I'm in the board still, in the
plane of a_1 and a_2, is the
804
00:50:26 --> 00:50:32
plane of q_1 and q_2, I'm just
getting right angles in. a_3,
805
00:50:32 --> 00:50:35
the third column in a three by
three case, is coming
806
00:50:35 --> 00:50:37
out at some angle.
807
00:50:37 --> 00:50:42
I want q_3 to come out
at a 90 degree angle.
808
00:50:42 --> 00:50:50
So that q_3 will involve some
combination of all the a's.
809
00:50:50 --> 00:50:54
So if it was three by three,
this would grow to q_1, q_2,
810
00:50:54 --> 00:50:59
q_3, and this would then have
three guys in its third column.
811
00:50:59 --> 00:51:02
But maybe you see that picture.
812
00:51:02 --> 00:51:06
So that's what
Gram-Schmidt achieves.
813
00:51:06 --> 00:51:12
And I just can't let time
right now without saying that
814
00:51:12 --> 00:51:14
this is a pretty good way.
815
00:51:14 --> 00:51:16
Actually, nobody's thought
there was a better
816
00:51:16 --> 00:51:19
one for centuries.
817
00:51:19 --> 00:51:24
But then a guy named
Householder came up with a
818
00:51:24 --> 00:51:29
different way, and a
numerically little better way.
819
00:51:29 --> 00:51:31
Numerically a
little better way.
820
00:51:31 --> 00:51:33
So this is the
Gram-Schmidt way.
821
00:51:33 --> 00:51:35
Can I just put those
words up here?
822
00:51:35 --> 00:51:38
So there's the Gram-Schmidt,
the classical Gram-Schmidt
823
00:51:38 --> 00:51:44
idea, which was what I
described, was the easy
824
00:51:44 --> 00:51:47
one, easy to describe.
825
00:51:47 --> 00:51:53
And then there's a method
called Householder, just named
826
00:51:53 --> 00:51:56
after him, MATLAB would follow.
827
00:51:56 --> 00:52:01
That every good qr code now
uses Householder matrices.
828
00:52:01 --> 00:52:04
It achieves the same results.
829
00:52:04 --> 00:52:06
And if I had a little bit
more time I could draw a
830
00:52:06 --> 00:52:08
picture of what it does.
831
00:52:08 --> 00:52:09
But there you go.
832
00:52:09 --> 00:52:15
So that's my lecture on,
my quick lecture on
833
00:52:15 --> 00:52:17
numerical linear algebra.
834
00:52:17 --> 00:52:23
These two essential points and
I'll see you this afternoon.
835
00:52:23 --> 00:52:27
Let me bring those quiz
questions down again, for any
836
00:52:27 --> 00:52:29
discussion about the quiz.