1
00:00:00 --> 00:00:01
2
00:00:01 --> 00:00:03
The following content is
provided under a Creative
3
00:00:03 --> 00:00:03
Commons license.
4
00:00:03 --> 00:00:06
Your support will help MIT
OpenCourseWare continue to
5
00:00:06 --> 00:00:10
offer high-quality educational
resources for free.
6
00:00:10 --> 00:00:13
To make a donation or to view
additional materials from
7
00:00:13 --> 00:00:15
hundreds of MIT courses visit
MIT OpenCourseWare
8
00:00:15 --> 00:00:19
at ocw.mit.edu.
9
00:00:19 --> 00:00:25
PROFESSOR STRANG: You can
pick up the homeworks after
10
00:00:25 --> 00:00:27
if you didn't get one.
11
00:00:27 --> 00:00:29
And the good grade is five.
12
00:00:29 --> 00:00:32
And the MATLABs are coming
in, that's all good.
13
00:00:32 --> 00:00:40
And I'll have some thoughts
about the MATLAB Monday.
14
00:00:40 --> 00:00:44
I wanted to say a little more
about finite differences
15
00:00:44 --> 00:00:47
in time just because
they're so important.
16
00:00:47 --> 00:00:54
And I got the idea of a forward
Euler and a backward Euler but
17
00:00:54 --> 00:00:57
I want to give you a couple of
more possibilities just so you
18
00:00:57 --> 00:01:02
see how finite difference
methods are created.
19
00:01:02 --> 00:01:07
And then I want to get started
on least squares, the next
20
00:01:07 --> 00:01:13
major topic, the next case
where A transpose A shows up
21
00:01:13 --> 00:01:15
at the core of everything.
22
00:01:15 --> 00:01:19
So what we did with finite
differences was this was our
23
00:01:19 --> 00:01:23
model problem for one spring
and this is our model
24
00:01:23 --> 00:01:27
problem for n springs
with n different masses.
25
00:01:27 --> 00:01:34
So this is like, scalar. u
is just a single unknown.
26
00:01:34 --> 00:01:37
Here u is a vector unknown.
27
00:01:37 --> 00:01:42
If I reduce things to like,
even this model problem, if I
28
00:01:42 --> 00:01:45
introduce the velocity city,
because everybody's also
29
00:01:45 --> 00:01:49
interested in computing
velocity, then the velocity is
30
00:01:49 --> 00:01:53
u' and the equation tells
me that v' is minus u.
31
00:01:53 --> 00:01:55
So I have a system.
32
00:01:55 --> 00:01:58
Better to think in terms of
first order systems because
33
00:01:58 --> 00:02:00
they include everything.
34
00:02:00 --> 00:02:03
So the first order system
there, do you see what the
35
00:02:03 --> 00:02:05
matrix is on that
right-hand side?
36
00:02:05 --> 00:02:08
I'm always seeing a matrix
that's multiplying uv.
37
00:02:10 --> 00:02:17
So this would be our example
of this general set-up.
38
00:02:17 --> 00:02:19
This general set-up.
39
00:02:19 --> 00:02:24
First order systems, taking
them to be linear here.
40
00:02:24 --> 00:02:29
And then I better say a little
bit about what happens when
41
00:02:29 --> 00:02:34
they're not linear
today and later.
42
00:02:34 --> 00:02:36
So this is my
problem. du/dt=Au.
43
00:02:36 --> 00:02:39
44
00:02:39 --> 00:02:42
And the exact solution would
come from the eigenvalues
45
00:02:42 --> 00:02:45
and eigenvectors of A.
46
00:02:45 --> 00:02:50
We would have the
e^(lambda*t)'s giving us the
47
00:02:50 --> 00:02:55
time growth or decay or
oscillation times
48
00:02:55 --> 00:02:56
their eigenvectors.
49
00:02:56 --> 00:03:02
And a combination of those
would be the exact answer.
50
00:03:02 --> 00:03:06
So that's the general method
for using eigenvalues and
51
00:03:06 --> 00:03:13
eigenvectors to get
exact solutions.
52
00:03:13 --> 00:03:15
I'm not speaking now
about exact solutions.
53
00:03:15 --> 00:03:22
I'm going to talk about finite
differences in time because for
54
00:03:22 --> 00:03:25
more general problems I
must expect to go to
55
00:03:25 --> 00:03:26
finite differences.
56
00:03:26 --> 00:03:31
I can't expect exact solutions
like pure cos(t) or sin(t).
57
00:03:31 --> 00:03:36
So here's my problem,
model problem.
58
00:03:36 --> 00:03:39
Here is Euler's first idea.
59
00:03:39 --> 00:03:45
So idea one from Euler was
replace the derivative by a
60
00:03:45 --> 00:03:52
finite difference taking time
steps forward in time and use
61
00:03:52 --> 00:03:56
the equation to tell you the
slope at the start of the step.
62
00:03:56 --> 00:04:01
Do you see Euler's
equation there?
63
00:04:01 --> 00:04:08
And that is definitely a
reasonable thing to start with.
64
00:04:08 --> 00:04:11
It's not very accurate.
65
00:04:11 --> 00:04:13
It's not perfectly stable.
66
00:04:13 --> 00:04:16
It doesn't preserve energy.
67
00:04:16 --> 00:04:23
We saw the answer spiraling out
but if you only want to compute
68
00:04:23 --> 00:04:28
up to a limited time you could
use it with a pretty
69
00:04:28 --> 00:04:30
small time step.
70
00:04:30 --> 00:04:34
But you can do better.
71
00:04:34 --> 00:04:36
Now backward Euler
was the other way.
72
00:04:36 --> 00:04:40
So it's just the contrast of
the two that you want to see.
73
00:04:40 --> 00:04:43
Neither one is an
outstanding method.
74
00:04:43 --> 00:04:48
They're both only
first order accurate.
75
00:04:48 --> 00:04:56
So backward Euler, this time
step in time is still the
76
00:04:56 --> 00:05:00
difference that replaces
the derivative.
77
00:05:00 --> 00:05:04
But now everybody notices
the big difference.
78
00:05:04 --> 00:05:08
The slope is being computed
at the end of the time step.
79
00:05:08 --> 00:05:09
And that's more stable.
80
00:05:09 --> 00:05:14
That's the one
that spiraled in.
81
00:05:14 --> 00:05:21
And I would call this
method explicit.
82
00:05:21 --> 00:05:23
What does explicit mean?
83
00:05:23 --> 00:05:27
It means that I can
find u_(n+1) directly.
84
00:05:27 --> 00:05:31
This method I would
call implicit.
85
00:05:31 --> 00:05:33
And what does implicit mean?
86
00:05:33 --> 00:05:37
Implicit means that u_(n+1) is
appearing on the right-hand
87
00:05:37 --> 00:05:40
side as well as the left so
I've got to move it over and I
88
00:05:40 --> 00:05:44
have to solve a system of
equations to find
89
00:05:44 --> 00:05:44
the next u_(n+1).
90
00:05:45 --> 00:05:51
So you see that basic
separation of ideas here.
91
00:05:51 --> 00:05:57
Forward, faster,
but less stable.
92
00:05:57 --> 00:06:01
Backward, slower, generally
more stable and in fact, in
93
00:06:01 --> 00:06:03
this case somehow, too stable.
94
00:06:03 --> 00:06:05
It dissipates more energy.
95
00:06:05 --> 00:06:10
You don't want to lose
all your energy.
96
00:06:10 --> 00:06:12
So you look for
something better.
97
00:06:12 --> 00:06:16
And also you look for
something more accurate.
98
00:06:16 --> 00:06:21
I want to suggest a more
accurate method, the
99
00:06:21 --> 00:06:23
trapezoidal method.
100
00:06:23 --> 00:06:27
Which just follows that basic
principle that if I center
101
00:06:27 --> 00:06:31
things at halfway I'm probably
going to pick up an
102
00:06:31 --> 00:06:33
order of accuracy.
103
00:06:33 --> 00:06:40
So centering it halfway I took
half of backward Euler and
104
00:06:40 --> 00:06:41
half of forward Euler.
105
00:06:41 --> 00:06:47
So is this one
explicit or implicit?
106
00:06:47 --> 00:06:47
Implicit.
107
00:06:47 --> 00:06:51
Implicit. u_(n+1) is still
appearing here, has
108
00:06:51 --> 00:06:52
to move over there.
109
00:06:52 --> 00:06:56
So if I move it over I could
say here I have, let me
110
00:06:56 --> 00:07:01
multiply up by delta t and
bring the u_(n+1) over here.
111
00:07:01 --> 00:07:02
So I'd have u_(n+1).
112
00:07:04 --> 00:07:08
I'll have minus
delta t over 2A.
113
00:07:10 --> 00:07:12
All that's what's
multiplying u_(n+1).
114
00:07:14 --> 00:07:15
Right?
115
00:07:15 --> 00:07:17
When I multiply up,
let me do that.
116
00:07:17 --> 00:07:19
Let me make it easy
for your eyes.
117
00:07:19 --> 00:07:23
I'll multiply up
by the delta t.
118
00:07:23 --> 00:07:26
And then I'm bringing this
part, the implicit part.
119
00:07:26 --> 00:07:29
So that's my left-hand side.
120
00:07:29 --> 00:07:33
And what's my right-hand side?
u_n is going over as the
121
00:07:33 --> 00:07:37
identity plus delta t over 2A.
122
00:07:38 --> 00:07:40
All that's what's
multiplying u_n.
123
00:07:41 --> 00:07:45
Good.
124
00:07:45 --> 00:07:49
So this is the matrix that has
to get inverted at every step.
125
00:07:49 --> 00:07:52
Of course, in this model
problem that's not expensive to
126
00:07:52 --> 00:07:59
do. if we have the same matrix
at every step, if we have a
127
00:07:59 --> 00:08:06
linear time-invariant system,
well we can just invert it once
128
00:08:06 --> 00:08:09
or factor it in into
L times U once.
129
00:08:09 --> 00:08:11
We don't have to do
it at every step.
130
00:08:11 --> 00:08:13
So that's, of
course, very fast.
131
00:08:13 --> 00:08:18
But when the problem becomes
non-linear we're going to start
132
00:08:18 --> 00:08:23
paying a price for a more
complicated implicit part.
133
00:08:23 --> 00:08:28
What I'm interested in right
now just to sort of see
134
00:08:28 --> 00:08:35
these differences, there's
a particular matrix A
135
00:08:35 --> 00:08:39
that is my model here.
136
00:08:39 --> 00:08:41
It's antisymmetric.
137
00:08:41 --> 00:08:45
Notice it's antisymmetric
and that sort of goes
138
00:08:45 --> 00:08:47
with conserving energy.
139
00:08:47 --> 00:08:51
I'm not going to in a first
order system, that
140
00:08:51 --> 00:08:56
antisymmetric tells me that the
eigenvalues of that matrix
141
00:08:56 --> 00:08:58
are pure imaginary.
142
00:08:58 --> 00:09:00
And so I'm going to get e^(it).
143
00:09:02 --> 00:09:05
I'm going to get things
that don't blow up,
144
00:09:05 --> 00:09:07
that don't decay.
145
00:09:07 --> 00:09:14
So I want to see what's
the growth factor?
146
00:09:14 --> 00:09:21
Suppose you want to understand
is this method good,
147
00:09:21 --> 00:09:22
what's its growth factor?
148
00:09:22 --> 00:09:24
So that will tell me
about stability.
149
00:09:24 --> 00:09:28
It'll tell me about
accuracy, too.
150
00:09:28 --> 00:09:39
So its growth factor, all I
want to do is put everything on
151
00:09:39 --> 00:09:42
the right side of the equation.
152
00:09:42 --> 00:09:44
I want to write
it in this form.
153
00:09:44 --> 00:09:46
So that G will be
the growth matrix.
154
00:09:46 --> 00:09:50
And what is G?
155
00:09:50 --> 00:09:55
Well, I just have
to invert that.
156
00:09:55 --> 00:09:57
That's I minus delta t over 2A.
157
00:09:58 --> 00:10:02
The inverse of that, so that's
the implicit part, times
158
00:10:02 --> 00:10:08
the explicit part.
159
00:10:08 --> 00:10:10
I'm not doing anything
difficult here.
160
00:10:10 --> 00:10:19
Just seeing how would you
approach to understand whether
161
00:10:19 --> 00:10:25
the solution to this is
growing, decaying, or staying
162
00:10:25 --> 00:10:31
as we hope with maybe,
constant energy.
163
00:10:31 --> 00:10:34
Because that's what the
true solution is doing.
164
00:10:34 --> 00:10:36
The true solution, you
remember, is just going around
165
00:10:36 --> 00:10:40
in a circle in our model
problem and just oscillating
166
00:10:40 --> 00:10:45
forever in our model system.
167
00:10:45 --> 00:10:51
What about the growth or
decay or whatever of
168
00:10:51 --> 00:10:53
that growth factor?
169
00:10:53 --> 00:10:55
Well again, this is the
point where I would
170
00:10:55 --> 00:10:57
look for eigenvalues.
171
00:10:57 --> 00:11:02
If I have a matrix G, and
everybody recognizes that after
172
00:11:02 --> 00:11:07
n time steps, what matrix
am I going to have?
173
00:11:07 --> 00:11:12
What matrix connects u_n
to the original u_0?
174
00:11:13 --> 00:11:15
That matrix is?
175
00:11:15 --> 00:11:15
G^n.
176
00:11:17 --> 00:11:21
At every step I'm
multiplying by a G.
177
00:11:21 --> 00:11:25
So with finite differences,
you have powers.
178
00:11:25 --> 00:11:26
That's the rule.
179
00:11:26 --> 00:11:30
With differential equations,
you have exponentials.
180
00:11:30 --> 00:11:36
With finite differences, finite
steps, you have powers of G.
181
00:11:36 --> 00:11:42
So I'm interested in G^n and
what tells me about that
182
00:11:42 --> 00:11:44
is the eigenvalues here.
183
00:11:44 --> 00:11:47
If I had to ask for the
eigenvalues of this
184
00:11:47 --> 00:11:52
matrix, they would be
the eigenvalues of G.
185
00:11:52 --> 00:11:59
So eig(G), can I
say for this case.
186
00:11:59 --> 00:12:04
Actually, can I just
say what it would be?
187
00:12:04 --> 00:12:07
What are the eigenvalues
of this guy?
188
00:12:07 --> 00:12:13
The eigenvalues of that factor
are one plus delta t over two
189
00:12:13 --> 00:12:16
times the eigenvalues of A.
190
00:12:16 --> 00:12:19
That's what we're
getting from here.
191
00:12:19 --> 00:12:20
And here, this is the inverse.
192
00:12:20 --> 00:12:26
So its eigenvalues will
come in the denominator.
193
00:12:26 --> 00:12:31
This'll be one minus delta
t over two times the
194
00:12:31 --> 00:12:34
eigenvalues of A.
195
00:12:34 --> 00:12:35
That's pretty good.
196
00:12:35 --> 00:12:37
That's pretty good.
197
00:12:37 --> 00:12:41
Actually this sort
of shows me a lot.
198
00:12:41 --> 00:12:46
Well, in the case of this model
example, the eigenvalues
199
00:12:46 --> 00:12:51
were i and minus i.
200
00:12:51 --> 00:12:57
This is for the special case
A equals [0, 1; -1, 0].
201
00:12:57 --> 00:13:04
This will be one plus i delta
t over two divided by one
202
00:13:04 --> 00:13:09
minus i delta t over two.
203
00:13:09 --> 00:13:12
And the complex conjugate
for the other eigenvalue.
204
00:13:12 --> 00:13:15
It'd be two eigenvalues.
205
00:13:15 --> 00:13:18
Let me work with the one, let
me take the one that's i
206
00:13:18 --> 00:13:22
and then there would be a
similar deal with minus i.
207
00:13:22 --> 00:13:30
That's our eigenvalue of G.
208
00:13:30 --> 00:13:34
Where in the complex
plane is that number?
209
00:13:34 --> 00:13:37
If I give you that
complex number.
210
00:13:37 --> 00:13:40
We're meeting complex
numbers here.
211
00:13:40 --> 00:13:43
We're not meeting complex
functions, just we have to
212
00:13:43 --> 00:13:48
be able to deal with
complex numbers.
213
00:13:48 --> 00:13:52
Actually, when I look at those
two numbers, what do I see
214
00:13:52 --> 00:13:56
right away about them?
215
00:13:56 --> 00:13:59
How are they related?
216
00:13:59 --> 00:14:02
They're conjugates, right?
217
00:14:02 --> 00:14:03
Conjugates of each other.
218
00:14:03 --> 00:14:08
This one is in the complex
plane, I would go along
219
00:14:08 --> 00:14:12
one, the real axis and
up delta t over two.
220
00:14:12 --> 00:14:15
And this one I would go
down delta t over two.
221
00:14:15 --> 00:14:21
And that symmetry is what
I, the word you used,
222
00:14:21 --> 00:14:25
the complex conjugate.
223
00:14:25 --> 00:14:30
Compare that length for the
top with that length for the
224
00:14:30 --> 00:14:32
bottom and what do you get?
225
00:14:32 --> 00:14:33
Same length.
226
00:14:33 --> 00:14:39
So I'm concluding that in this
case the magnitude of these
227
00:14:39 --> 00:14:43
eigenvalues of G is what?
228
00:14:43 --> 00:14:48
So magnitude, absolute value,
is just the absolute value
229
00:14:48 --> 00:14:51
of that divided by the
absolute value of that.
230
00:14:51 --> 00:14:54
It's this distance divided
by this distance.
231
00:14:54 --> 00:14:59
And you told me already
the answer is one.
232
00:14:59 --> 00:15:06
The eigenvalues are right
on the unit circle.
233
00:15:06 --> 00:15:09
In some ways that's wonderful.
234
00:15:09 --> 00:15:14
The solution if you
compute exactly will
235
00:15:14 --> 00:15:18
stay on the unit circle.
236
00:15:18 --> 00:15:25
Of course, it will not be
exactly the same as the
237
00:15:25 --> 00:15:27
continuous solution.
238
00:15:27 --> 00:15:31
But the energy won't change.
239
00:15:31 --> 00:15:32
We're not going to spiral out.
240
00:15:32 --> 00:15:34
We're not going to spiral in.
241
00:15:34 --> 00:15:37
It's this average and
it's more accurate.
242
00:15:37 --> 00:15:41
So trapezoidal method is
like, the workhorse for
243
00:15:41 --> 00:15:44
finite element codes.
244
00:15:44 --> 00:15:52
Trapezoidal method is a
pretty successful method.
245
00:15:52 --> 00:15:53
What's the price?
246
00:15:53 --> 00:15:58
The price is this implicit
stuff that you have to solve.
247
00:15:58 --> 00:16:01
So I should say it's
the workhorse among
248
00:16:01 --> 00:16:03
implicit methods.
249
00:16:03 --> 00:16:06
And it's simple.
250
00:16:06 --> 00:16:11
So just to have a picture,
finite element codes
251
00:16:11 --> 00:16:16
usually are not shooting
for great accuracy.
252
00:16:16 --> 00:16:19
Many finite element
calculations, you're happy with
253
00:16:19 --> 00:16:21
two or three decimal places.
254
00:16:21 --> 00:16:23
So we're not shooting
for great accuracy.
255
00:16:23 --> 00:16:27
Second order is very adequate.
256
00:16:27 --> 00:16:33
What we don't want, of
course, is to be unstable.
257
00:16:33 --> 00:16:39
We don't want to lose
all the energy.
258
00:16:39 --> 00:16:49
So this is really
a good method.
259
00:16:49 --> 00:17:02
I have to say it's a good
method, but not perfect.
260
00:17:02 --> 00:17:08
For linear problems, for my
model problem this is fine.
261
00:17:08 --> 00:17:19
For a real problem, a difficult
problem in mechanics you might
262
00:17:19 --> 00:17:27
find that the energy, you might
find it goes a little unstable
263
00:17:27 --> 00:17:32
and non-linearity tends to grab
onto a little instability
264
00:17:32 --> 00:17:33
and make it worse.
265
00:17:33 --> 00:17:39
So like Professor Bathe if you
take his course on finite
266
00:17:39 --> 00:17:46
elements, this trapezoidal
rule-- and many people
267
00:17:46 --> 00:17:50
have reinvented it.
268
00:17:50 --> 00:17:54
There's a Newmark family of
methods and with special
269
00:17:54 --> 00:17:57
parameter, you get back this
one and everybody makes
270
00:17:57 --> 00:17:59
that choice practically.
271
00:17:59 --> 00:18:04
There's just a host of
finite difference methods.
272
00:18:04 --> 00:18:06
I'm wondering whether you want
me to tell you one more.
273
00:18:06 --> 00:18:09
Do you want one more
finite difference method?
274
00:18:09 --> 00:18:13
Just to like, see what.
275
00:18:13 --> 00:18:15
You said yes, right?
276
00:18:15 --> 00:18:16
One more method.
277
00:18:16 --> 00:18:21
One more and then, this
is a subject on its own.
278
00:18:21 --> 00:18:23
But just to see what
else could you do.
279
00:18:23 --> 00:18:25
What else might you do?
280
00:18:25 --> 00:18:27
And let me say why you would.
281
00:18:27 --> 00:18:30
And Professor Bathe
had to do it.
282
00:18:30 --> 00:18:37
In some problems he found that
with non-linear problems, he
283
00:18:37 --> 00:18:42
found that this method, which
for a perfect linear problem
284
00:18:42 --> 00:18:48
stays exactly one that's great,
but you're playing with fire.
285
00:18:48 --> 00:18:54
To be right on the unit circle,
if non-linearity pushes you off
286
00:18:54 --> 00:19:00
then you wish you had not tried
tried for that perfection.
287
00:19:00 --> 00:19:10
So let me describe a
backward difference.
288
00:19:10 --> 00:19:18
Let me write down,
I'll call this BDF2.
289
00:19:18 --> 00:19:22
All these formulas have
names and numbers.
290
00:19:22 --> 00:19:25
So this would be Backward
Difference Formula
291
00:19:25 --> 00:19:27
second order accurate.
292
00:19:27 --> 00:19:30
You'll see, it goes
two steps back.
293
00:19:30 --> 00:19:38
So here it is. u_(n+1)-u_(n-1)
over delta t.
294
00:19:38 --> 00:19:42
And then there's another
term which gets the
295
00:19:42 --> 00:19:43
second order accuracy.
296
00:19:43 --> 00:19:51
It happens to be
u_(n+1)-2u_n+u_(n-1)
297
00:19:51 --> 00:19:54
over delta t.
298
00:19:54 --> 00:19:56
And then that
really is delta t.
299
00:19:56 --> 00:19:58
Equals Au(n+1).
300
00:19:58 --> 00:20:02
301
00:20:02 --> 00:20:04
It's good practice.
302
00:20:04 --> 00:20:11
That formula came
from somewhere.
303
00:20:11 --> 00:20:12
What if we look at it.
304
00:20:12 --> 00:20:15
What do we see?
305
00:20:15 --> 00:20:19
What's the picture for
a formula like that?
306
00:20:19 --> 00:20:25
Is it implicit or explicit
first, our first question.
307
00:20:25 --> 00:20:25
Implicit.
308
00:20:25 --> 00:20:29
Because it's using, this
right-hand side involves this.
309
00:20:29 --> 00:20:33
And if it was a non-linear
equation, whatever that
310
00:20:33 --> 00:20:37
right-hand side is, not linear,
would be evaluated at the new
311
00:20:37 --> 00:20:43
step and therefore would
have to go back over here.
312
00:20:43 --> 00:20:46
It is second order accurate
and I won't go through
313
00:20:46 --> 00:20:51
the checking on that.
314
00:20:51 --> 00:20:53
And is it stable?
315
00:20:53 --> 00:20:56
That's another question.
316
00:20:56 --> 00:20:59
We have to find
eigenvalues here.
317
00:20:59 --> 00:21:07
Let me not go through
all details there.
318
00:21:07 --> 00:21:09
It is stable.
319
00:21:09 --> 00:21:12
And it's slightly dissipative.
320
00:21:12 --> 00:21:16
It's not as dissipative
as backward Euler.
321
00:21:16 --> 00:21:19
There you're losing
energy fast.
322
00:21:19 --> 00:21:28
Here the eigenvalues, the thing
would stay much closer to the
323
00:21:28 --> 00:21:33
unit circle than
backward Euler.
324
00:21:33 --> 00:21:41
I'll just put up there
so that you see.
325
00:21:41 --> 00:21:45
What are other features
that you see right
326
00:21:45 --> 00:21:47
away from this method?
327
00:21:47 --> 00:21:51
The fact that it involves
not only u_(n+1) and
328
00:21:51 --> 00:21:52
u_n, but also u_(n-1).
329
00:21:54 --> 00:21:55
What does that mean?
330
00:21:55 --> 00:21:57
How do I get started
with that method?
331
00:21:57 --> 00:21:59
Right?
332
00:21:59 --> 00:22:02
This calls to find the new u.
333
00:22:02 --> 00:22:07
I need the previous one
and the one before that.
334
00:22:07 --> 00:22:09
No problem, I have them.
335
00:22:09 --> 00:22:11
Except at the start I don't.
336
00:22:11 --> 00:22:15
So it'll need a sort of special
start to be able to figure
337
00:22:15 --> 00:22:17
out what should u_n be.
338
00:22:17 --> 00:22:20
It'll need a separate
formula to decide.
339
00:22:20 --> 00:22:25
And it could use one step of
backward Euler to find u_1.
340
00:22:26 --> 00:22:33
And then take off, now finding
u_2 from u_1 and u_0.
341
00:22:33 --> 00:22:34
And then onward.
342
00:22:34 --> 00:22:36
So that's quite fast.
343
00:22:36 --> 00:22:40
More stable, good method.
344
00:22:40 --> 00:22:47
And you maybe you can see that
I could get more formulas by
345
00:22:47 --> 00:22:49
going back further in time.
346
00:22:49 --> 00:22:54
And by doing that I can
get the accuracy higher.
347
00:22:54 --> 00:22:58
So that's good to see
that particular BDF2.
348
00:22:58 --> 00:23:02
So that's a backward
difference formula.
349
00:23:02 --> 00:23:09
Oh, just to mention
what's developed.
350
00:23:09 --> 00:23:14
So this is stable.
351
00:23:14 --> 00:23:17
It actually loses
a little energy.
352
00:23:17 --> 00:23:22
So in fact now both Professor
Bathe and I have studied the
353
00:23:22 --> 00:23:28
possibility of taking a
trapezoidal step, which was a
354
00:23:28 --> 00:23:31
little dangerous in the
non-linear case because you
355
00:23:31 --> 00:23:34
were playing with fire, you
were right on the circle.
356
00:23:34 --> 00:23:40
And then put in a backward
difference step to recover
357
00:23:40 --> 00:23:43
a little stability.
358
00:23:43 --> 00:23:47
And then trapezoidal
backward difference.
359
00:23:47 --> 00:23:48
So split step.
360
00:23:48 --> 00:23:52
Split the step into a
trapezoidal part and a
361
00:23:52 --> 00:23:53
backward difference part.
362
00:23:53 --> 00:23:58
That's actually discussed
later in Chapter 2.
363
00:23:58 --> 00:24:01
Section 2.6 and I might
come back to that.
364
00:24:01 --> 00:24:04
I just wanted to say
that much this morning.
365
00:24:04 --> 00:24:10
Having got as far as forward
and backward Euler.
366
00:24:10 --> 00:24:14
I didn't want to leave you
without a better method which
367
00:24:14 --> 00:24:18
is the trapezoidal method.
368
00:24:18 --> 00:24:23
I could take any question.
369
00:24:23 --> 00:24:27
I would like to devote the
second half, if you're ok
370
00:24:27 --> 00:24:34
to just change gear, to
beginning on least squares.
371
00:24:34 --> 00:24:41
So this was our kind of
excitement to have time
372
00:24:41 --> 00:24:43
dependence for a little while.
373
00:24:43 --> 00:24:48
But now I'm going back to
steady state problems.
374
00:24:48 --> 00:24:49
So they're not moving.
375
00:24:49 --> 00:24:51
I'm looking at equilibrium.
376
00:24:51 --> 00:25:01
And what I'm going to do now in
then next lectures is more to
377
00:25:01 --> 00:25:07
see this framework that we
identified once for the masses
378
00:25:07 --> 00:25:09
and springs, to see
it again and again.
379
00:25:09 --> 00:25:12
Because it's the basic
framework of applied math.
380
00:25:12 --> 00:25:15
So now I'm ready
for least squares.
381
00:25:15 --> 00:25:17
Least squares.
382
00:25:17 --> 00:25:22
What's the problem?
383
00:25:22 --> 00:25:27
Well the problem is I'm given
a system of equations Au=f.
384
00:25:27 --> 00:25:34
385
00:25:34 --> 00:25:42
These f's are observations.
386
00:25:42 --> 00:25:45
The u is the unknown vector.
387
00:25:45 --> 00:25:47
You say what's different here?
388
00:25:47 --> 00:25:49
It's just a linear system.
389
00:25:49 --> 00:25:53
What's different is
too many equations.
390
00:25:53 --> 00:26:01
This is an m by n matrix
with m bigger than n.
391
00:26:01 --> 00:26:05
Maybe much bigger than n.
392
00:26:05 --> 00:26:07
So what do we do?
393
00:26:07 --> 00:26:11
We've got too many
equations and no solution,
394
00:26:11 --> 00:26:18
no exact solution.
395
00:26:18 --> 00:26:22
I would say probably that what
we're coming to, what we're
396
00:26:22 --> 00:26:26
starting on today is the most
important, the application of
397
00:26:26 --> 00:26:29
linear algebra that
I see the most.
398
00:26:29 --> 00:26:30
So it interests everybody.
399
00:26:30 --> 00:26:36
It interests engineers,
scientists, statisticians,
400
00:26:36 --> 00:26:43
everybody has to deal with this
problem of too many equations.
401
00:26:43 --> 00:26:46
And those equations come from
measurements so you don't
402
00:26:46 --> 00:26:48
want to throw them away.
403
00:26:48 --> 00:26:50
I don't want to
just throw away.
404
00:26:50 --> 00:26:54
I want to somehow get
the best solution.
405
00:26:54 --> 00:26:58
I'm looking for the u that
comes closest when I
406
00:26:58 --> 00:27:00
can't find an exact u.
407
00:27:00 --> 00:27:02
So that's the idea.
408
00:27:02 --> 00:27:05
There's no exact solution
and the problem is
409
00:27:05 --> 00:27:12
what is the best u.
410
00:27:12 --> 00:27:14
And I'm going to
call that u hat.
411
00:27:14 --> 00:27:18
My favorite choice
of u will be u hat.
412
00:27:18 --> 00:27:22
So I'm going to get an
equation for u hat.
413
00:27:22 --> 00:27:25
That is my goal.
414
00:27:25 --> 00:27:32
And what I'm starting here just
goes all the way to, there will
415
00:27:32 --> 00:27:36
be weighted least squares,
there will be Kalman filters.
416
00:27:36 --> 00:27:42
It's a giant world here of
estimating the best solution
417
00:27:42 --> 00:27:44
when there's noise in
the right-hand side.
418
00:27:44 --> 00:27:46
And what's the model problem?
419
00:27:46 --> 00:27:48
Always good to have
a model problem.
420
00:27:48 --> 00:27:51
Let me draw a model problem.
421
00:27:51 --> 00:28:00
Model problem is fit
by a straight line.
422
00:28:00 --> 00:28:10
So say C+Dt shall I
use that as the C+Dx.
423
00:28:11 --> 00:28:13
It's got two unknowns.
424
00:28:13 --> 00:28:15
So n is two.
425
00:28:15 --> 00:28:21
But m is big.
426
00:28:21 --> 00:28:22
What do I have?
427
00:28:22 --> 00:28:23
Let me draw the picture.
428
00:28:23 --> 00:28:26
You've seen this often.
429
00:28:26 --> 00:28:29
This is the t direction,
this is the f.
430
00:28:29 --> 00:28:31
These are the measurements.
431
00:28:31 --> 00:28:39
So I measure at time
t=0 let's say.
432
00:28:39 --> 00:28:45
I measure my position.
433
00:28:45 --> 00:28:46
That would be f_1.
434
00:28:47 --> 00:28:51
At time t=1 I've moved
somewhere, I've
435
00:28:51 --> 00:28:52
measured where I am.
436
00:28:52 --> 00:28:54
I'm tracking a
satellite, let's say.
437
00:28:54 --> 00:28:56
So I'm tracking this satellite.
438
00:28:56 --> 00:28:59
Well the times don't have
to be equally spaced.
439
00:28:59 --> 00:29:03
I'll take the next
time to be three.
440
00:29:03 --> 00:29:10
Let's say the engine is
off this satellite.
441
00:29:10 --> 00:29:14
If the measurements were
perfect-- Does that look
442
00:29:14 --> 00:29:16
too perfect to you?
443
00:29:16 --> 00:29:18
It's almost on a straight
line, isn't it?
444
00:29:18 --> 00:29:23
Of course my point is that
measurements, well I mean, of
445
00:29:23 --> 00:29:26
course in reality measurements
would be close to
446
00:29:26 --> 00:29:27
a straight line.
447
00:29:27 --> 00:29:31
I'm going to have to draw, in
order for you to see anything,
448
00:29:31 --> 00:29:35
I'm going to have
to draw a really.
449
00:29:35 --> 00:29:39
Suppose f_1 is one.
450
00:29:39 --> 00:29:45
Suppose it starts at position
one and suppose f_2 is
451
00:29:45 --> 00:29:50
two and this guy will be.
452
00:29:50 --> 00:29:53
Where do you want
me to take it?
453
00:29:53 --> 00:29:58
Let's see, if it was
linear, what would be the?
454
00:29:58 --> 00:30:01
It would be four, right?
455
00:30:01 --> 00:30:03
So can I take a
different number?
456
00:30:03 --> 00:30:04
Three.
457
00:30:04 --> 00:30:05
Is three okay?
458
00:30:05 --> 00:30:09
Because five I haven't
got space for.
459
00:30:09 --> 00:30:12
And you don't want to see pi
or some dumb thing or e.
460
00:30:12 --> 00:30:20
So let me take three.
461
00:30:20 --> 00:30:24
I want to fit that data which
is we're saying close to a
462
00:30:24 --> 00:30:28
straight line, I want to fit
it by the best straight line.
463
00:30:28 --> 00:30:32
So the best straight line would
go probably, I don't know what
464
00:30:32 --> 00:30:35
your eye suggests for the best
straight line through
465
00:30:35 --> 00:30:36
three points.
466
00:30:36 --> 00:30:40
Do you see I've got three
equations, two unknowns?
467
00:30:40 --> 00:30:41
That's the first point to see.
468
00:30:41 --> 00:30:47
Somehow I'm trying to fit three
things with only two degrees of
469
00:30:47 --> 00:30:50
freedom and I'm not going
to succeed usually.
470
00:30:50 --> 00:30:54
But I'm going to do my best and
probably the best line goes
471
00:30:54 --> 00:30:59
sort of, it won't exactly
go through any of them.
472
00:30:59 --> 00:31:03
So I'm doing the best least
squares approximation.
473
00:31:03 --> 00:31:05
And what does that mean?
474
00:31:05 --> 00:31:08
Well, what would the
three equations be?
475
00:31:08 --> 00:31:12
What does my linear equations,
my unsolvable ones,
476
00:31:12 --> 00:31:15
say that at time zero?
477
00:31:15 --> 00:31:20
So at time zero, at time one
and at time three, at each of
478
00:31:20 --> 00:31:24
those times I have an equation
C+Dt should agree with.
479
00:31:24 --> 00:31:28
So that C+D time zero
should match f_1.
480
00:31:31 --> 00:31:34
At t=1 my line will be C+D*1.
481
00:31:34 --> 00:31:37
482
00:31:37 --> 00:31:38
It should equal f_2.
483
00:31:40 --> 00:31:48
And at t=3, the height of the
line will be C+3d and I would
484
00:31:48 --> 00:31:50
like it to go through
that height f_3.
485
00:31:50 --> 00:31:53
486
00:31:53 --> 00:31:58
But I'm not going to be able.
487
00:31:58 --> 00:32:01
If there was noise in the
measurements that system,
488
00:32:01 --> 00:32:06
that's my unsolvable system.
489
00:32:06 --> 00:32:08
What's the matrix?
490
00:32:08 --> 00:32:12
I want to write three equations
and you're getting good at
491
00:32:12 --> 00:32:19
seeing three equations like so.
492
00:32:19 --> 00:32:24
So I've a 3 by 2 matrix.
493
00:32:24 --> 00:32:27
And my unknown u is C, D.
494
00:32:28 --> 00:32:30
Those are my unknowns.
495
00:32:30 --> 00:32:33
And my right-hand sides are
these heights, well, I decided
496
00:32:33 --> 00:32:40
on particular numbers,
one, two and three.
497
00:32:40 --> 00:32:42
One, two, three.
498
00:32:42 --> 00:32:44
And what's the matrix?
499
00:32:44 --> 00:32:47
What's the matrix A that
you read off when you see
500
00:32:47 --> 00:32:50
that system of equations?
501
00:32:50 --> 00:32:52
The first column
of the matrix is?
502
00:32:52 --> 00:32:55
All ones because that's
multiplying the C's.
503
00:32:55 --> 00:32:59
And the second column of
the matrix is the times.
504
00:32:59 --> 00:33:02
Zero, one, three,
is that right?
505
00:33:02 --> 00:33:04
That multiply the D.
506
00:33:04 --> 00:33:08
So this is the same as that.
507
00:33:08 --> 00:33:12
So here I'm in my set-up.
508
00:33:12 --> 00:33:24
I'll erase m equal big because
m was only three, not that big.
509
00:33:24 --> 00:33:30
What's the best answer?
510
00:33:30 --> 00:33:32
What's the best u hat?
511
00:33:32 --> 00:33:38
The best u hat will now
be C hat and D hat.
512
00:33:38 --> 00:33:45
The best I can do.
513
00:33:45 --> 00:33:50
I need some idea of
what does best mean.
514
00:33:50 --> 00:33:55
And there is not a single
possible meaning.
515
00:33:55 --> 00:33:59
There are many possible ways
I could say the best line.
516
00:33:59 --> 00:34:06
One way would be to make
the, well, what could
517
00:34:06 --> 00:34:09
a best line be?
518
00:34:09 --> 00:34:11
I'm going to have three
errors here, right?
519
00:34:11 --> 00:34:14
That did not go right
through the point.
520
00:34:14 --> 00:34:15
This did not go right
through the point.
521
00:34:15 --> 00:34:17
They came pretty close.
522
00:34:17 --> 00:34:22
I've got three small
errors. e_1, e_2, e_3.
523
00:34:23 --> 00:34:26
Those are the errors
in my equations.
524
00:34:26 --> 00:34:31
So I will get equality when
I add in the e_1, e_2, e_3,
525
00:34:31 --> 00:34:39
the little bits that will
bring it onto the line.
526
00:34:39 --> 00:34:41
One idea.
527
00:34:41 --> 00:34:46
Make the largest error
as small as I can.
528
00:34:46 --> 00:34:54
Minimize the maximum of the e's
Try to balance them so no e,
529
00:34:54 --> 00:34:58
no error is bigger
than the others.
530
00:34:58 --> 00:35:00
Look for that balance.
531
00:35:00 --> 00:35:04
That's a reasonable idea.
532
00:35:04 --> 00:35:07
But it's not the
least squares idea.
533
00:35:07 --> 00:35:11
So what's the least
squares idea?
534
00:35:11 --> 00:35:15
The least squares idea makes
the sum of the squares of the
535
00:35:15 --> 00:35:17
errors as small as possible.
536
00:35:17 --> 00:35:29
So the least squares idea will
be to minimize the sum of the
537
00:35:29 --> 00:35:35
squares of the errors. e_1
squared plus e_m squared.
538
00:35:35 --> 00:35:38
It would be just e_1
squared plus e_2 squared
539
00:35:38 --> 00:35:42
plus e_3 squared.
540
00:35:42 --> 00:35:44
What is this?
541
00:35:44 --> 00:35:48
Let me began to write
this in matrix.
542
00:35:48 --> 00:35:51
I want to bring in
the matrix here.
543
00:35:51 --> 00:35:56
This is the error.
544
00:35:56 --> 00:35:59
The error is the
difference between the
545
00:35:59 --> 00:36:01
measurements and Au.
546
00:36:02 --> 00:36:06
So that's what I'm
trying to make small.
547
00:36:06 --> 00:36:10
I'd love to make it zero
but I can't. i've got more
548
00:36:10 --> 00:36:12
equations than unknowns.
549
00:36:12 --> 00:36:14
There's no two unknowns
that will make all
550
00:36:14 --> 00:36:16
three errors zero.
551
00:36:16 --> 00:36:18
So I want to make
the errors small.
552
00:36:18 --> 00:36:23
And this is the
length of e squared.
553
00:36:23 --> 00:36:30
The length in this sum
of squares method.
554
00:36:30 --> 00:36:37
It's a pretty good
measure of the error.
555
00:36:37 --> 00:36:41
Gauss was the first to
apply least squares.
556
00:36:41 --> 00:36:45
What I'm going to
do today is Gauss.
557
00:36:45 --> 00:36:48
Who was, by the way, the
greatest mathematician
558
00:36:48 --> 00:36:52
of all time.
559
00:36:52 --> 00:36:54
And here, he was doing
astronomy actually.
560
00:36:54 --> 00:37:03
And writing in latin.
561
00:37:03 --> 00:37:08
The message got out somehow.
562
00:37:08 --> 00:37:12
So his idea was sum of squares.
563
00:37:12 --> 00:37:18
So this e is the distance
between f and Au.
564
00:37:18 --> 00:37:22
I have to begin to write.
565
00:37:22 --> 00:37:23
I have to write some things.
566
00:37:23 --> 00:37:28
I can write some things out in
detail, but then I also, at the
567
00:37:28 --> 00:37:33
same time, have to carry along
the way I would look at it for
568
00:37:33 --> 00:37:36
any matrix A and any
right-hand side f.
569
00:37:36 --> 00:37:39
So do you see that
this is the error?
570
00:37:39 --> 00:37:44
The meaning of this double
bars squared is exactly that.
571
00:37:44 --> 00:37:46
That it's the sum of the
squares of the components.
572
00:37:46 --> 00:37:54
So that's where the word
least squares come in.
573
00:37:54 --> 00:38:03
Can I just say what's better
about least squares and
574
00:38:03 --> 00:38:06
what's maybe a drawback.
575
00:38:06 --> 00:38:09
So actually this next
sentence is pretty
576
00:38:09 --> 00:38:12
important in practice.
577
00:38:12 --> 00:38:15
What's better about least
squares, what's really nice
578
00:38:15 --> 00:38:21
about least squares is well,
for one thing, the equations
579
00:38:21 --> 00:38:25
we get for the best C, D
will be linear, will
580
00:38:25 --> 00:38:29
be linear equations.
581
00:38:29 --> 00:38:33
You may say, not surprising,
I started out with a linear
582
00:38:33 --> 00:38:39
system and I'm going to end
up with a linear system.
583
00:38:39 --> 00:38:45
Actually I prefer to use
well, might be too late.
584
00:38:45 --> 00:38:47
Next lecture I'm going
to put b in there for
585
00:38:47 --> 00:38:49
the right-hand side.
586
00:38:49 --> 00:38:54
But I'll leave it
with that for now.
587
00:38:54 --> 00:38:57
So good point is we'll
get linear equations.
588
00:38:57 --> 00:39:06
The not so good point in some
applications is when I look at
589
00:39:06 --> 00:39:16
the squares of errors, well big
errors, outliers, really bad
590
00:39:16 --> 00:39:20
measurements have a big
influence on the answer
591
00:39:20 --> 00:39:22
because of getting squared.
592
00:39:22 --> 00:39:30
So if I have ten readings that
are very accurate but then in
593
00:39:30 --> 00:39:36
an eleventh reading that is way
off and I don't know it and if
594
00:39:36 --> 00:39:40
I don't realize that that's way
off, then that eleventh error
595
00:39:40 --> 00:39:45
will-- It's like having a whole
lot of points close to a line
596
00:39:45 --> 00:39:48
and then another point way off.
597
00:39:48 --> 00:39:55
That will have a significant
effect on the best line.
598
00:39:55 --> 00:40:02
So you might say too
great an effect.
599
00:40:02 --> 00:40:05
Depends on the application.
600
00:40:05 --> 00:40:11
I just had to say before
starting on least squares, as
601
00:40:11 --> 00:40:17
always, there are advantages
and disadvantages but
602
00:40:17 --> 00:40:18
the advantages are
very, very great.
603
00:40:18 --> 00:40:25
So it's an important idea
here, least squares.
604
00:40:25 --> 00:40:31
I'm ready now to ask for
the equation for u hat.
605
00:40:31 --> 00:40:38
So the equation for u hat is
the u that minimizes here.
606
00:40:38 --> 00:40:45
So we have touched on
minimizing quadratics.
607
00:40:45 --> 00:40:54
This is squares.
608
00:40:54 --> 00:40:59
I could expand that out as
f minus Au transpose times
609
00:40:59 --> 00:41:06
f minus Au just to see
another way to write it.
610
00:41:06 --> 00:41:11
The length squared of a vector
is always the transpose.
611
00:41:11 --> 00:41:14
It's inner product with itself.
612
00:41:14 --> 00:41:19
And I could split this out into
all these different terms.
613
00:41:19 --> 00:41:24
I would have then, some
quadratic equation to minimize.
614
00:41:24 --> 00:41:28
In other words, let me
jump to the answer.
615
00:41:28 --> 00:41:33
Let me jump to the
equation for the best u.
616
00:41:33 --> 00:41:38
And then come back to see why.
617
00:41:38 --> 00:41:41
Because you must see
what that equation is.
618
00:41:41 --> 00:41:46
It's the fundamental equation
of, this might be called linear
619
00:41:46 --> 00:41:49
regression, fitting data.
620
00:41:49 --> 00:41:51
You're just
constantly doing it.
621
00:41:51 --> 00:41:56
So what is the equation
for the best u hat?
622
00:41:56 --> 00:41:58
Can I put it here?
623
00:41:58 --> 00:42:01
This'll be equation
that we get to.
624
00:42:01 --> 00:42:04
It'll be A transpose A.
625
00:42:04 --> 00:42:07
You're not surprised to see
A transpose A up here.
626
00:42:07 --> 00:42:12
First of all because this is
18.085 and also because this A
627
00:42:12 --> 00:42:16
is rectangular and when you
have rectangular matrices,
628
00:42:16 --> 00:42:19
sooner or later A
transpose A comes up.
629
00:42:19 --> 00:42:21
So that's the matrix.
630
00:42:21 --> 00:42:26
And then the right-hand
side is A transpose f.
631
00:42:26 --> 00:42:30
So that's the key equation
for least squares.
632
00:42:30 --> 00:42:33
That's the central equation
of least squares.
633
00:42:33 --> 00:42:37
And let's just see
what it looks like.
634
00:42:37 --> 00:42:41
You could say the way I arrived
at it, Imean the short way
635
00:42:41 --> 00:42:45
is this is an equation
that I can't satisfy.
636
00:42:45 --> 00:42:56
I multiply both sides by A
transpose and now this is
637
00:42:56 --> 00:43:00
the equation for u hat.
638
00:43:00 --> 00:43:05
And what that did was kind of
average out the m equations.
639
00:43:05 --> 00:43:07
Because how many
equations do I now have?
640
00:43:07 --> 00:43:09
A as m by n.
641
00:43:09 --> 00:43:13
What's the shape
of A transpose A?
642
00:43:13 --> 00:43:15
Everybody's on top
of that, right?
643
00:43:15 --> 00:43:21
The shape of A transpose
A is square, n by n.
644
00:43:21 --> 00:43:28
Because A transpose is n
by m. n by m times m by
645
00:43:28 --> 00:43:30
n leaves us an n by n.
646
00:43:30 --> 00:43:34
So we've averaged the m
equations that were too
647
00:43:34 --> 00:43:38
many to get n equations.
648
00:43:38 --> 00:43:47
And of course this is what it
should be, n by m m by one.
649
00:43:47 --> 00:43:48
So it's m by one.
650
00:43:48 --> 00:43:51
It's a good right-hand side.
651
00:43:51 --> 00:44:02
That's the equation
of least squares.
652
00:44:02 --> 00:44:07
That's the equation I want to
explain, understand and solve.
653
00:44:07 --> 00:44:10
Actually why don't we solve it
for this particular problem
654
00:44:10 --> 00:44:17
just to see the whole
thing for this example.
655
00:44:17 --> 00:44:22
Just to do it.
656
00:44:22 --> 00:44:24
So there is A.
657
00:44:24 --> 00:44:25
Here is u.
658
00:44:25 --> 00:44:32
And here is f. what
shall I call these?
659
00:44:32 --> 00:44:37
They're mostly called the
normal equations. that's one
660
00:44:37 --> 00:44:42
possible word for the key
equation of least squares,
661
00:44:42 --> 00:44:44
the normal equations.
662
00:44:44 --> 00:44:47
Can you tell me these
matrices A transpose A.
663
00:44:47 --> 00:44:50
And u hat I know.
664
00:44:50 --> 00:44:53
That'll be the best
C and the best D.
665
00:44:53 --> 00:44:59
And over here can you
compute A transpose f?
666
00:44:59 --> 00:45:04
If I write A transpose above
it, will that help you do
667
00:45:04 --> 00:45:05
these multiplications?
668
00:45:05 --> 00:45:06
Let me just do.
669
00:45:06 --> 00:45:08
So there was the matrix A.
670
00:45:08 --> 00:45:12
Let me write A
transpose above it.
671
00:45:12 --> 00:45:22
So it has a row of ones
and then a row of times.
672
00:45:22 --> 00:45:25
So what shape is the matrix?
673
00:45:25 --> 00:45:30
The A transpose A matrix.
674
00:45:30 --> 00:45:33
It's going to be that A
transpose times that A.
675
00:45:33 --> 00:45:37
The size will be
two by two, right?
676
00:45:37 --> 00:45:41
Two by three times three by
two, it's averaging out to
677
00:45:41 --> 00:45:43
get me a two by two matrix.
678
00:45:43 --> 00:45:46
What's the first entry of this?
679
00:45:46 --> 00:45:50
Can you do A transpose times A
just so we see this matrix.
680
00:45:50 --> 00:45:52
Three.
681
00:45:52 --> 00:45:54
And off the diagonal?
682
00:45:54 --> 00:45:55
Four.
683
00:45:55 --> 00:45:56
And here?
684
00:45:56 --> 00:45:58
And you knew it
would be symmetric.
685
00:45:58 --> 00:45:59
And here?
686
00:45:59 --> 00:46:03
Ten.
687
00:46:03 --> 00:46:06
And tell me A transpose
f while we're at it.
688
00:46:06 --> 00:46:08
So that's just a vector.
689
00:46:08 --> 00:46:11
If you multiply that by
the right-hand sides,
690
00:46:11 --> 00:46:12
looks like a six.
691
00:46:12 --> 00:46:18
Is that right?
692
00:46:18 --> 00:46:21
11, maybe.
693
00:46:21 --> 00:46:24
Is that right?
694
00:46:24 --> 00:46:29
Two, nine making 11, yeah.
695
00:46:29 --> 00:46:31
So those are the numbers.
696
00:46:31 --> 00:46:36
I can't write those numbers,
write A transpose A without
697
00:46:36 --> 00:46:39
asking you to tell me one
more time what kind of a
698
00:46:39 --> 00:46:42
matrix have I got here?
699
00:46:42 --> 00:46:44
It's symmetric positive
definite, right?
700
00:46:44 --> 00:46:47
We know that's going to be.
701
00:46:47 --> 00:46:50
And we see it.
702
00:46:50 --> 00:46:54
Our test for positive definite
might be the determinant, that
703
00:46:54 --> 00:46:57
one by one determinant is
three, that two by two
704
00:46:57 --> 00:47:04
determinant is 30 minus
16; 14 positive.
705
00:47:04 --> 00:47:06
We got a good problem here.
706
00:47:06 --> 00:47:09
And I could solve for
C hat and D hat.
707
00:47:09 --> 00:47:14
I see the numbers are not
coming out fantastically but
708
00:47:14 --> 00:47:18
they would produce a line that
would, I'm pretty sure, it
709
00:47:18 --> 00:47:25
would be, this looks
optimal to me.
710
00:47:25 --> 00:47:29
If I rotated any, I'm going
to make things worse.
711
00:47:29 --> 00:47:31
I think it would
look like that.
712
00:47:31 --> 00:47:35
So that's the system
that you end up with.
713
00:47:35 --> 00:47:37
But why?
714
00:47:37 --> 00:47:41
You really have to
understand where did
715
00:47:41 --> 00:47:44
this equation come from.
716
00:47:44 --> 00:47:50
Where did it come from?
717
00:47:50 --> 00:47:56
It's worth understanding,
this least squares stuff.
718
00:47:56 --> 00:48:01
So I'm going to try to draw a
picture that makes it clear
719
00:48:01 --> 00:48:08
where that equation comes from.
720
00:48:08 --> 00:48:11
So what am I doing here?
721
00:48:11 --> 00:48:12
Au=f.
722
00:48:12 --> 00:48:15
723
00:48:15 --> 00:48:18
Start there.
724
00:48:18 --> 00:48:22
And the particular A was,
I'll even copy the A.
725
00:48:22 --> 00:48:27
It was 1, 1, 1; 0, 1, 3
multiplied u to give
726
00:48:27 --> 00:48:30
me f as .
727
00:48:30 --> 00:48:32
But of course, I couldn't
solve it because I don't
728
00:48:32 --> 00:48:35
have enough unknowns.
729
00:48:35 --> 00:48:39
What's the picture?
730
00:48:39 --> 00:48:46
Everybody likes to see what's
happening by a picture
731
00:48:46 --> 00:48:49
as well as by algebra.
732
00:48:49 --> 00:48:56
So the picture here is I'm
in three dimensions and I
733
00:48:56 --> 00:48:57
have a vector .
734
00:48:57 --> 00:49:01
So 0 in that direction,
one in that, three up.
735
00:49:01 --> 00:49:07
So somewhere there is my f.
736
00:49:07 --> 00:49:15
Now I'll put in C, D here.
737
00:49:15 --> 00:49:19
What is the equation
asking me to do?
738
00:49:19 --> 00:49:22
Which actually, I won't be able
to do because I can't solve it.
739
00:49:22 --> 00:49:35
But the equation, how do we see
a system of linear equations?
740
00:49:35 --> 00:49:39
If I have a system of linear
equations I'm looking for
741
00:49:39 --> 00:49:49
numbers C and D so that C
times column one plus D times
742
00:49:49 --> 00:49:52
column two gives me that.
743
00:49:52 --> 00:49:55
That's how I think of a
system of equations.
744
00:49:55 --> 00:50:00
A combination of the columns.
745
00:50:00 --> 00:50:04
Tell me what vectors do I
get if I take combinations
746
00:50:04 --> 00:50:05
of the columns.
747
00:50:05 --> 00:50:08
Well, if I took the combination
C=1, D=0 I would just
748
00:50:08 --> 00:50:09
get the first column.
749
00:50:09 --> 00:50:14
So that's a candidate. <1, 1,
1>, I don't know where that is.
750
00:50:14 --> 00:50:20
Wherever might be.
751
00:50:20 --> 00:50:23
I'm not too sure where
to draw .
752
00:50:23 --> 00:50:27
I want to go one there,
one there and one there.
753
00:50:27 --> 00:50:31
Damn.
754
00:50:31 --> 00:50:36
Let's define that to be the
vector right there.
755
00:50:36 --> 00:50:36
Wait.
756
00:50:36 --> 00:50:43
You let me put that up there
and I didn't mean to, right? f
757
00:50:43 --> 00:50:48
should be zero, what, sorry?
f was , yeah.
758
00:50:48 --> 00:50:49
Damn!
759
00:50:49 --> 00:50:52
Don't let me make mistakes.
760
00:50:52 --> 00:50:57
These mistakes are permanent
if you let them slide by.
761
00:50:57 --> 00:51:04
That's it, same point.
762
00:51:04 --> 00:51:06
I didn't have the point
right in the first place
763
00:51:06 --> 00:51:08
so now it's just perfect.
764
00:51:08 --> 00:51:15
There it is.
765
00:51:15 --> 00:51:18
Before of course, if I
had I could've
766
00:51:18 --> 00:51:21
solved the equation.
767
00:51:21 --> 00:51:26
But with I can't.
768
00:51:26 --> 00:51:27
Here's the situation.
769
00:51:27 --> 00:51:31
This vector is not a
combination of those two.
770
00:51:31 --> 00:51:36
Because the combinations of two
vectors, what's the picture?
771
00:51:36 --> 00:51:40
If I try to draw, if I look at
all combinations of two
772
00:51:40 --> 00:51:45
vectors, which is
that vector, which is
773
00:51:45 --> 00:51:49
maybe this vector
let's just say.
774
00:51:49 --> 00:51:52
If I take the combinations
of these two column
775
00:51:52 --> 00:51:55
vectors, what do I get?
776
00:51:55 --> 00:51:57
Now this is for
everybody to know.
777
00:51:57 --> 00:52:00
If I take the combinations
of two vectors here
778
00:52:00 --> 00:52:04
in three-dimensional
space I get a plane.
779
00:52:04 --> 00:52:08
I get the plane that
contains those vectors.
780
00:52:08 --> 00:52:14
So this I could call the column
plane or the column space.
781
00:52:14 --> 00:52:21
This is all combinations
of the columns.
782
00:52:21 --> 00:52:29
That's the same thing as saying
this is all the f's that
783
00:52:29 --> 00:52:38
have exact solutions.
784
00:52:38 --> 00:52:41
So let's just see this picture.
785
00:52:41 --> 00:52:45
This particular right-hand side
is not in the plane, right?
786
00:52:45 --> 00:52:46
That's my problem.
787
00:52:46 --> 00:52:51
This particular vector f
points out of the plane.
788
00:52:51 --> 00:52:53
But if I change it a
little, like if I
789
00:52:53 --> 00:52:56
change it to .
790
00:52:56 --> 00:52:57
Do you see that that would?
791
00:52:57 --> 00:53:00
What's different now that
I've changed it to <1,
792
00:53:00 --> 00:53:05
2, 3> for a moment?
793
00:53:05 --> 00:53:07
What's different
about ?
794
00:53:07 --> 00:53:11
Where is
in my picture?
795
00:53:11 --> 00:53:14
Do you see what's great
about ?
796
00:53:14 --> 00:53:16
It is a combination.
797
00:53:16 --> 00:53:18
Right?
is a combination.
798
00:53:18 --> 00:53:19
With C=1, D=1.
799
00:53:20 --> 00:53:24
It would satisfy the equation.
800
00:53:24 --> 00:53:28
So where is <1, 2,
4> in my picture?
801
00:53:28 --> 00:53:31
It's in the plane.
802
00:53:31 --> 00:53:38
The plane are the heights that
do lie on a straight line.
803
00:53:38 --> 00:53:42
So the plane are all the ones
that I can get exactly.
804
00:53:42 --> 00:53:49
But this vector, these
observations,
805
00:53:49 --> 00:53:50
I couldn't get exactly.
806
00:53:50 --> 00:53:55
So let me, in 30 seconds
or less, let me tell you
807
00:53:55 --> 00:53:57
the best thing to do.
808
00:53:57 --> 00:53:59
Or let you tell me the
best thing to do.
809
00:53:59 --> 00:54:02
I have a right-hand side
that's not in the plane.
810
00:54:02 --> 00:54:06
I can get my straight lines
correspond to vectors,
811
00:54:06 --> 00:54:07
right-hand sides that
are in the plane.
812
00:54:07 --> 00:54:10
So what do I do?
813
00:54:10 --> 00:54:12
I project.
814
00:54:12 --> 00:54:16
I take the nearest point
that is the plane as
815
00:54:16 --> 00:54:18
my right-hand side.
816
00:54:18 --> 00:54:21
I project down.
817
00:54:21 --> 00:54:24
And it's that projection that's
going to lead us to the
818
00:54:24 --> 00:54:30
equation that I'm shooting
for, A transpose Au hat
819
00:54:30 --> 00:54:32
equals A transpose f.
820
00:54:32 --> 00:54:40
This comes from projecting f
down into the plane where
821
00:54:40 --> 00:54:43
straight lines do work exactly.
822
00:54:43 --> 00:54:47
So there's an error here
that I can't deal with.
823
00:54:47 --> 00:54:50
And there's a part here,
the projection part,
824
00:54:50 --> 00:54:52
that I can deal with.
825
00:54:52 --> 00:54:56
This is important and
it's fun and we'll come
826
00:54:56 --> 00:54:57
back to it Monday.
827
00:54:57 --> 00:54:59
Thanks for patience.