1
00:00:00 --> 00:00:01
 
 

2
00:00:01 --> 00:00:03
The following content is
provided under a Creative

3
00:00:03 --> 00:00:03
Commons license.
 

4
00:00:03 --> 00:00:06
Your support will help MIT
OpenCourseWare continue to

5
00:00:06 --> 00:00:10
offer high-quality educational
resources for free.

6
00:00:10 --> 00:00:13
To make a donation or to view
additional materials from

7
00:00:13 --> 00:00:15
hundreds of MIT courses visit
MIT OpenCourseWare

8
00:00:15 --> 00:00:19
at ocw.mit.edu.
 

9
00:00:19 --> 00:00:25
PROFESSOR STRANG: You can
pick up the homeworks after

10
00:00:25 --> 00:00:27
if you didn't get one.
 

11
00:00:27 --> 00:00:29
And the good grade is five.
 

12
00:00:29 --> 00:00:32
And the MATLABs are coming
in, that's all good.

13
00:00:32 --> 00:00:40
And I'll have some thoughts
about the MATLAB Monday.

14
00:00:40 --> 00:00:44
I wanted to say a little more
about finite differences

15
00:00:44 --> 00:00:47
in time just because
they're so important.

16
00:00:47 --> 00:00:54
And I got the idea of a forward
Euler and a backward Euler but

17
00:00:54 --> 00:00:57
I want to give you a couple of
more possibilities just so you

18
00:00:57 --> 00:01:02
see how finite difference
methods are created.

19
00:01:02 --> 00:01:07
And then I want to get started
on least squares, the next

20
00:01:07 --> 00:01:13
major topic, the next case
where A transpose A shows up

21
00:01:13 --> 00:01:15
at the core of everything.
 

22
00:01:15 --> 00:01:19
So what we did with finite
differences was this was our

23
00:01:19 --> 00:01:23
model problem for one spring
and this is our model

24
00:01:23 --> 00:01:27
problem for n springs
with n different masses.

25
00:01:27 --> 00:01:34
So this is like, scalar. u
is just a single unknown.

26
00:01:34 --> 00:01:37
Here u is a vector unknown.
 

27
00:01:37 --> 00:01:42
If I reduce things to like,
even this model problem, if I

28
00:01:42 --> 00:01:45
introduce the velocity city,
because everybody's also

29
00:01:45 --> 00:01:49
interested in computing
velocity, then the velocity is

30
00:01:49 --> 00:01:53
u' and the equation tells
me that v' is minus u.

31
00:01:53 --> 00:01:55
So I have a system.
 

32
00:01:55 --> 00:01:58
Better to think in terms of
first order systems because

33
00:01:58 --> 00:02:00
they include everything.
 

34
00:02:00 --> 00:02:03
So the first order system
there, do you see what the

35
00:02:03 --> 00:02:05
matrix is on that
right-hand side?

36
00:02:05 --> 00:02:08
I'm always seeing a matrix
that's multiplying uv.

37
00:02:10 --> 00:02:17
So this would be our example
of this general set-up.

38
00:02:17 --> 00:02:19
This general set-up.
 

39
00:02:19 --> 00:02:24
First order systems, taking
them to be linear here.

40
00:02:24 --> 00:02:29
And then I better say a little
bit about what happens when

41
00:02:29 --> 00:02:34
they're not linear
today and later.

42
00:02:34 --> 00:02:36
So this is my
problem. du/dt=Au.

43
00:02:36 --> 00:02:39
 
 

44
00:02:39 --> 00:02:42
And the exact solution would
come from the eigenvalues

45
00:02:42 --> 00:02:45
and eigenvectors of A.
 

46
00:02:45 --> 00:02:50
We would have the
e^(lambda*t)'s giving us the

47
00:02:50 --> 00:02:55
time growth or decay or
oscillation times

48
00:02:55 --> 00:02:56
their eigenvectors.
 

49
00:02:56 --> 00:03:02
And a combination of those
would be the exact answer.

50
00:03:02 --> 00:03:06
So that's the general method
for using eigenvalues and

51
00:03:06 --> 00:03:13
eigenvectors to get
exact solutions.

52
00:03:13 --> 00:03:15
I'm not speaking now
about exact solutions.

53
00:03:15 --> 00:03:22
I'm going to talk about finite
differences in time because for

54
00:03:22 --> 00:03:25
more general problems I
must expect to go to

55
00:03:25 --> 00:03:26
finite differences.
 

56
00:03:26 --> 00:03:31
I can't expect exact solutions
like pure cos(t) or sin(t).

57
00:03:31 --> 00:03:36
So here's my problem,
model problem.

58
00:03:36 --> 00:03:39
Here is Euler's first idea.
 

59
00:03:39 --> 00:03:45
So idea one from Euler was
replace the derivative by a

60
00:03:45 --> 00:03:52
finite difference taking time
steps forward in time and use

61
00:03:52 --> 00:03:56
the equation to tell you the
slope at the start of the step.

62
00:03:56 --> 00:04:01
Do you see Euler's
equation there?

63
00:04:01 --> 00:04:08
And that is definitely a
reasonable thing to start with.

64
00:04:08 --> 00:04:11
It's not very accurate.
 

65
00:04:11 --> 00:04:13
It's not perfectly stable.
 

66
00:04:13 --> 00:04:16
It doesn't preserve energy.
 

67
00:04:16 --> 00:04:23
We saw the answer spiraling out
but if you only want to compute

68
00:04:23 --> 00:04:28
up to a limited time you could
use it with a pretty

69
00:04:28 --> 00:04:30
small time step.
 

70
00:04:30 --> 00:04:34
But you can do better.
 

71
00:04:34 --> 00:04:36
Now backward Euler
was the other way.

72
00:04:36 --> 00:04:40
So it's just the contrast of
the two that you want to see.

73
00:04:40 --> 00:04:43
Neither one is an
outstanding method.

74
00:04:43 --> 00:04:48
They're both only
first order accurate.

75
00:04:48 --> 00:04:56
So backward Euler, this time
step in time is still the

76
00:04:56 --> 00:05:00
difference that replaces
the derivative.

77
00:05:00 --> 00:05:04
But now everybody notices
the big difference.

78
00:05:04 --> 00:05:08
The slope is being computed
at the end of the time step.

79
00:05:08 --> 00:05:09
And that's more stable.
 

80
00:05:09 --> 00:05:14
That's the one
that spiraled in.

81
00:05:14 --> 00:05:21
And I would call this
method explicit.

82
00:05:21 --> 00:05:23
What does explicit mean?
 

83
00:05:23 --> 00:05:27
It means that I can
find u_(n+1) directly.

84
00:05:27 --> 00:05:31
This method I would
call implicit.

85
00:05:31 --> 00:05:33
And what does implicit mean?
 

86
00:05:33 --> 00:05:37
Implicit means that u_(n+1) is
appearing on the right-hand

87
00:05:37 --> 00:05:40
side as well as the left so
I've got to move it over and I

88
00:05:40 --> 00:05:44
have to solve a system of
equations to find

89
00:05:44 --> 00:05:44
the next u_(n+1).
 

90
00:05:45 --> 00:05:51
So you see that basic
separation of ideas here.

91
00:05:51 --> 00:05:57
Forward, faster,
but less stable.

92
00:05:57 --> 00:06:01
Backward, slower, generally
more stable and in fact, in

93
00:06:01 --> 00:06:03
this case somehow, too stable.
 

94
00:06:03 --> 00:06:05
It dissipates more energy.
 

95
00:06:05 --> 00:06:10
You don't want to lose
all your energy.

96
00:06:10 --> 00:06:12
So you look for
something better.

97
00:06:12 --> 00:06:16
And also you look for
something more accurate.

98
00:06:16 --> 00:06:21
I want to suggest a more
accurate method, the

99
00:06:21 --> 00:06:23
trapezoidal method.
 

100
00:06:23 --> 00:06:27
Which just follows that basic
principle that if I center

101
00:06:27 --> 00:06:31
things at halfway I'm probably
going to pick up an

102
00:06:31 --> 00:06:33
order of accuracy.
 

103
00:06:33 --> 00:06:40
So centering it halfway I took
half of backward Euler and

104
00:06:40 --> 00:06:41
half of forward Euler.
 

105
00:06:41 --> 00:06:47
So is this one
explicit or implicit?

106
00:06:47 --> 00:06:47
Implicit.
 

107
00:06:47 --> 00:06:51
Implicit. u_(n+1) is still
appearing here, has

108
00:06:51 --> 00:06:52
to move over there.
 

109
00:06:52 --> 00:06:56
So if I move it over I could
say here I have, let me

110
00:06:56 --> 00:07:01
multiply up by delta t and
bring the u_(n+1) over here.

111
00:07:01 --> 00:07:02
So I'd have u_(n+1).
 

112
00:07:04 --> 00:07:08
I'll have minus
delta t over 2A.

113
00:07:10 --> 00:07:12
All that's what's
multiplying u_(n+1).

114
00:07:14 --> 00:07:15
Right?
 

115
00:07:15 --> 00:07:17
When I multiply up,
let me do that.

116
00:07:17 --> 00:07:19
Let me make it easy
for your eyes.

117
00:07:19 --> 00:07:23
I'll multiply up
by the delta t.

118
00:07:23 --> 00:07:26
And then I'm bringing this
part, the implicit part.

119
00:07:26 --> 00:07:29
So that's my left-hand side.
 

120
00:07:29 --> 00:07:33
And what's my right-hand side?
u_n is going over as the

121
00:07:33 --> 00:07:37
identity plus delta t over 2A.
 

122
00:07:38 --> 00:07:40
All that's what's
multiplying u_n.

123
00:07:41 --> 00:07:45
Good.
 

124
00:07:45 --> 00:07:49
So this is the matrix that has
to get inverted at every step.

125
00:07:49 --> 00:07:52
Of course, in this model
problem that's not expensive to

126
00:07:52 --> 00:07:59
do. if we have the same matrix
at every step, if we have a

127
00:07:59 --> 00:08:06
linear time-invariant system,
well we can just invert it once

128
00:08:06 --> 00:08:09
or factor it in into
L times U once.

129
00:08:09 --> 00:08:11
We don't have to do
it at every step.

130
00:08:11 --> 00:08:13
So that's, of
course, very fast.

131
00:08:13 --> 00:08:18
But when the problem becomes
non-linear we're going to start

132
00:08:18 --> 00:08:23
paying a price for a more
complicated implicit part.

133
00:08:23 --> 00:08:28
What I'm interested in right
now just to sort of see

134
00:08:28 --> 00:08:35
these differences, there's
a particular matrix A

135
00:08:35 --> 00:08:39
that is my model here.
 

136
00:08:39 --> 00:08:41
It's antisymmetric.
 

137
00:08:41 --> 00:08:45
Notice it's antisymmetric
and that sort of goes

138
00:08:45 --> 00:08:47
with conserving energy.
 

139
00:08:47 --> 00:08:51
I'm not going to in a first
order system, that

140
00:08:51 --> 00:08:56
antisymmetric tells me that the
eigenvalues of that matrix

141
00:08:56 --> 00:08:58
are pure imaginary.
 

142
00:08:58 --> 00:09:00
And so I'm going to get e^(it).
 

143
00:09:02 --> 00:09:05
I'm going to get things
that don't blow up,

144
00:09:05 --> 00:09:07
that don't decay.
 

145
00:09:07 --> 00:09:14
So I want to see what's
the growth factor?

146
00:09:14 --> 00:09:21
Suppose you want to understand
is this method good,

147
00:09:21 --> 00:09:22
what's its growth factor?
 

148
00:09:22 --> 00:09:24
So that will tell me
about stability.

149
00:09:24 --> 00:09:28
It'll tell me about
accuracy, too.

150
00:09:28 --> 00:09:39
So its growth factor, all I
want to do is put everything on

151
00:09:39 --> 00:09:42
the right side of the equation.
 

152
00:09:42 --> 00:09:44
I want to write
it in this form.

153
00:09:44 --> 00:09:46
So that G will be
the growth matrix.

154
00:09:46 --> 00:09:50
And what is G?
 

155
00:09:50 --> 00:09:55
Well, I just have
to invert that.

156
00:09:55 --> 00:09:57
That's I minus delta t over 2A.
 

157
00:09:58 --> 00:10:02
The inverse of that, so that's
the implicit part, times

158
00:10:02 --> 00:10:08
the explicit part.
 

159
00:10:08 --> 00:10:10
I'm not doing anything
difficult here.

160
00:10:10 --> 00:10:19
Just seeing how would you
approach to understand whether

161
00:10:19 --> 00:10:25
the solution to this is
growing, decaying, or staying

162
00:10:25 --> 00:10:31
as we hope with maybe,
constant energy.

163
00:10:31 --> 00:10:34
Because that's what the
true solution is doing.

164
00:10:34 --> 00:10:36
The true solution, you
remember, is just going around

165
00:10:36 --> 00:10:40
in a circle in our model
problem and just oscillating

166
00:10:40 --> 00:10:45
forever in our model system.
 

167
00:10:45 --> 00:10:51
What about the growth or
decay or whatever of

168
00:10:51 --> 00:10:53
that growth factor?
 

169
00:10:53 --> 00:10:55
Well again, this is the
point where I would

170
00:10:55 --> 00:10:57
look for eigenvalues.
 

171
00:10:57 --> 00:11:02
If I have a matrix G, and
everybody recognizes that after

172
00:11:02 --> 00:11:07
n time steps, what matrix
am I going to have?

173
00:11:07 --> 00:11:12
What matrix connects u_n
to the original u_0?

174
00:11:13 --> 00:11:15
That matrix is?
 

175
00:11:15 --> 00:11:15
G^n.
 

176
00:11:17 --> 00:11:21
At every step I'm
multiplying by a G.

177
00:11:21 --> 00:11:25
So with finite differences,
you have powers.

178
00:11:25 --> 00:11:26
That's the rule.
 

179
00:11:26 --> 00:11:30
With differential equations,
you have exponentials.

180
00:11:30 --> 00:11:36
With finite differences, finite
steps, you have powers of G.

181
00:11:36 --> 00:11:42
So I'm interested in G^n and
what tells me about that

182
00:11:42 --> 00:11:44
is the eigenvalues here.
 

183
00:11:44 --> 00:11:47
If I had to ask for the
eigenvalues of this

184
00:11:47 --> 00:11:52
matrix, they would be
the eigenvalues of G.

185
00:11:52 --> 00:11:59
So eig(G), can I
say for this case.

186
00:11:59 --> 00:12:04
Actually, can I just
say what it would be?

187
00:12:04 --> 00:12:07
What are the eigenvalues
of this guy?

188
00:12:07 --> 00:12:13
The eigenvalues of that factor
are one plus delta t over two

189
00:12:13 --> 00:12:16
times the eigenvalues of A.
 

190
00:12:16 --> 00:12:19
That's what we're
getting from here.

191
00:12:19 --> 00:12:20
And here, this is the inverse.
 

192
00:12:20 --> 00:12:26
So its eigenvalues will
come in the denominator.

193
00:12:26 --> 00:12:31
This'll be one minus delta
t over two times the

194
00:12:31 --> 00:12:34
eigenvalues of A.
 

195
00:12:34 --> 00:12:35
That's pretty good.
 

196
00:12:35 --> 00:12:37
That's pretty good.
 

197
00:12:37 --> 00:12:41
Actually this sort
of shows me a lot.

198
00:12:41 --> 00:12:46
Well, in the case of this model
example, the eigenvalues

199
00:12:46 --> 00:12:51
were i and minus i.
 

200
00:12:51 --> 00:12:57
This is for the special case
A equals [0, 1; -1, 0].

201
00:12:57 --> 00:13:04
This will be one plus i delta
t over two divided by one

202
00:13:04 --> 00:13:09
minus i delta t over two.
 

203
00:13:09 --> 00:13:12
And the complex conjugate
for the other eigenvalue.

204
00:13:12 --> 00:13:15
It'd be two eigenvalues.
 

205
00:13:15 --> 00:13:18
Let me work with the one, let
me take the one that's i

206
00:13:18 --> 00:13:22
and then there would be a
similar deal with minus i.

207
00:13:22 --> 00:13:30
That's our eigenvalue of G.
 

208
00:13:30 --> 00:13:34
Where in the complex
plane is that number?

209
00:13:34 --> 00:13:37
If I give you that
complex number.

210
00:13:37 --> 00:13:40
We're meeting complex
numbers here.

211
00:13:40 --> 00:13:43
We're not meeting complex
functions, just we have to

212
00:13:43 --> 00:13:48
be able to deal with
complex numbers.

213
00:13:48 --> 00:13:52
Actually, when I look at those
two numbers, what do I see

214
00:13:52 --> 00:13:56
right away about them?
 

215
00:13:56 --> 00:13:59
How are they related?
 

216
00:13:59 --> 00:14:02
They're conjugates, right?
 

217
00:14:02 --> 00:14:03
Conjugates of each other.
 

218
00:14:03 --> 00:14:08
This one is in the complex
plane, I would go along

219
00:14:08 --> 00:14:12
one, the real axis and
up delta t over two.

220
00:14:12 --> 00:14:15
And this one I would go
down delta t over two.

221
00:14:15 --> 00:14:21
And that symmetry is what
I, the word you used,

222
00:14:21 --> 00:14:25
the complex conjugate.
 

223
00:14:25 --> 00:14:30
Compare that length for the
top with that length for the

224
00:14:30 --> 00:14:32
bottom and what do you get?
 

225
00:14:32 --> 00:14:33
Same length.
 

226
00:14:33 --> 00:14:39
So I'm concluding that in this
case the magnitude of these

227
00:14:39 --> 00:14:43
eigenvalues of G is what?
 

228
00:14:43 --> 00:14:48
So magnitude, absolute value,
is just the absolute value

229
00:14:48 --> 00:14:51
of that divided by the
absolute value of that.

230
00:14:51 --> 00:14:54
It's this distance divided
by this distance.

231
00:14:54 --> 00:14:59
And you told me already
the answer is one.

232
00:14:59 --> 00:15:06
The eigenvalues are right
on the unit circle.

233
00:15:06 --> 00:15:09
In some ways that's wonderful.
 

234
00:15:09 --> 00:15:14
The solution if you
compute exactly will

235
00:15:14 --> 00:15:18
stay on the unit circle.
 

236
00:15:18 --> 00:15:25
Of course, it will not be
exactly the same as the

237
00:15:25 --> 00:15:27
continuous solution.
 

238
00:15:27 --> 00:15:31
But the energy won't change.
 

239
00:15:31 --> 00:15:32
We're not going to spiral out.
 

240
00:15:32 --> 00:15:34
We're not going to spiral in.
 

241
00:15:34 --> 00:15:37
It's this average and
it's more accurate.

242
00:15:37 --> 00:15:41
So trapezoidal method is
like, the workhorse for

243
00:15:41 --> 00:15:44
finite element codes.
 

244
00:15:44 --> 00:15:52
Trapezoidal method is a
pretty successful method.

245
00:15:52 --> 00:15:53
What's the price?
 

246
00:15:53 --> 00:15:58
The price is this implicit
stuff that you have to solve.

247
00:15:58 --> 00:16:01
So I should say it's
the workhorse among

248
00:16:01 --> 00:16:03
implicit methods.
 

249
00:16:03 --> 00:16:06
And it's simple.
 

250
00:16:06 --> 00:16:11
So just to have a picture,
finite element codes

251
00:16:11 --> 00:16:16
usually are not shooting
for great accuracy.

252
00:16:16 --> 00:16:19
Many finite element
calculations, you're happy with

253
00:16:19 --> 00:16:21
two or three decimal places.
 

254
00:16:21 --> 00:16:23
So we're not shooting
for great accuracy.

255
00:16:23 --> 00:16:27
Second order is very adequate.
 

256
00:16:27 --> 00:16:33
What we don't want, of
course, is to be unstable.

257
00:16:33 --> 00:16:39
We don't want to lose
all the energy.

258
00:16:39 --> 00:16:49
So this is really
a good method.

259
00:16:49 --> 00:17:02
I have to say it's a good
method, but not perfect.

260
00:17:02 --> 00:17:08
For linear problems, for my
model problem this is fine.

261
00:17:08 --> 00:17:19
For a real problem, a difficult
problem in mechanics you might

262
00:17:19 --> 00:17:27
find that the energy, you might
find it goes a little unstable

263
00:17:27 --> 00:17:32
and non-linearity tends to grab
onto a little instability

264
00:17:32 --> 00:17:33
and make it worse.
 

265
00:17:33 --> 00:17:39
So like Professor Bathe if you
take his course on finite

266
00:17:39 --> 00:17:46
elements, this trapezoidal
rule-- and many people

267
00:17:46 --> 00:17:50
have reinvented it.
 

268
00:17:50 --> 00:17:54
There's a Newmark family of
methods and with special

269
00:17:54 --> 00:17:57
parameter, you get back this
one and everybody makes

270
00:17:57 --> 00:17:59
that choice practically.
 

271
00:17:59 --> 00:18:04
There's just a host of
finite difference methods.

272
00:18:04 --> 00:18:06
I'm wondering whether you want
me to tell you one more.

273
00:18:06 --> 00:18:09
Do you want one more
finite difference method?

274
00:18:09 --> 00:18:13
Just to like, see what.
 

275
00:18:13 --> 00:18:15
You said yes, right?
 

276
00:18:15 --> 00:18:16
One more method.
 

277
00:18:16 --> 00:18:21
One more and then, this
is a subject on its own.

278
00:18:21 --> 00:18:23
But just to see what
else could you do.

279
00:18:23 --> 00:18:25
What else might you do?
 

280
00:18:25 --> 00:18:27
And let me say why you would.
 

281
00:18:27 --> 00:18:30
And Professor Bathe
had to do it.

282
00:18:30 --> 00:18:37
In some problems he found that
with non-linear problems, he

283
00:18:37 --> 00:18:42
found that this method, which
for a perfect linear problem

284
00:18:42 --> 00:18:48
stays exactly one that's great,
but you're playing with fire.

285
00:18:48 --> 00:18:54
To be right on the unit circle,
if non-linearity pushes you off

286
00:18:54 --> 00:19:00
then you wish you had not tried
tried for that perfection.

287
00:19:00 --> 00:19:10
So let me describe a
backward difference.

288
00:19:10 --> 00:19:18
Let me write down,
I'll call this BDF2.

289
00:19:18 --> 00:19:22
All these formulas have
names and numbers.

290
00:19:22 --> 00:19:25
So this would be Backward
Difference Formula

291
00:19:25 --> 00:19:27
second order accurate.
 

292
00:19:27 --> 00:19:30
You'll see, it goes
two steps back.

293
00:19:30 --> 00:19:38
So here it is. u_(n+1)-u_(n-1)
over delta t.

294
00:19:38 --> 00:19:42
And then there's another
term which gets the

295
00:19:42 --> 00:19:43
second order accuracy.
 

296
00:19:43 --> 00:19:51
It happens to be
u_(n+1)-2u_n+u_(n-1)

297
00:19:51 --> 00:19:54
over delta t.
 

298
00:19:54 --> 00:19:56
And then that
really is delta t.

299
00:19:56 --> 00:19:58
Equals Au(n+1).
 

300
00:19:58 --> 00:20:02
 
 

301
00:20:02 --> 00:20:04
It's good practice.
 

302
00:20:04 --> 00:20:11
That formula came
from somewhere.

303
00:20:11 --> 00:20:12
What if we look at it.
 

304
00:20:12 --> 00:20:15
What do we see?
 

305
00:20:15 --> 00:20:19
What's the picture for
a formula like that?

306
00:20:19 --> 00:20:25
Is it implicit or explicit
first, our first question.

307
00:20:25 --> 00:20:25
Implicit.
 

308
00:20:25 --> 00:20:29
Because it's using, this
right-hand side involves this.

309
00:20:29 --> 00:20:33
And if it was a non-linear
equation, whatever that

310
00:20:33 --> 00:20:37
right-hand side is, not linear,
would be evaluated at the new

311
00:20:37 --> 00:20:43
step and therefore would
have to go back over here.

312
00:20:43 --> 00:20:46
It is second order accurate
and I won't go through

313
00:20:46 --> 00:20:51
the checking on that.
 

314
00:20:51 --> 00:20:53
And is it stable?
 

315
00:20:53 --> 00:20:56
That's another question.
 

316
00:20:56 --> 00:20:59
We have to find
eigenvalues here.

317
00:20:59 --> 00:21:07
Let me not go through
all details there.

318
00:21:07 --> 00:21:09
It is stable.
 

319
00:21:09 --> 00:21:12
And it's slightly dissipative.
 

320
00:21:12 --> 00:21:16
It's not as dissipative
as backward Euler.

321
00:21:16 --> 00:21:19
There you're losing
energy fast.

322
00:21:19 --> 00:21:28
Here the eigenvalues, the thing
would stay much closer to the

323
00:21:28 --> 00:21:33
unit circle than
backward Euler.

324
00:21:33 --> 00:21:41
I'll just put up there
so that you see.

325
00:21:41 --> 00:21:45
What are other features
that you see right

326
00:21:45 --> 00:21:47
away from this method?
 

327
00:21:47 --> 00:21:51
The fact that it involves
not only u_(n+1) and

328
00:21:51 --> 00:21:52
u_n, but also u_(n-1).
 

329
00:21:54 --> 00:21:55
What does that mean?
 

330
00:21:55 --> 00:21:57
How do I get started
with that method?

331
00:21:57 --> 00:21:59
Right?
 

332
00:21:59 --> 00:22:02
This calls to find the new u.
 

333
00:22:02 --> 00:22:07
I need the previous one
and the one before that.

334
00:22:07 --> 00:22:09
No problem, I have them.
 

335
00:22:09 --> 00:22:11
Except at the start I don't.
 

336
00:22:11 --> 00:22:15
So it'll need a sort of special
start to be able to figure

337
00:22:15 --> 00:22:17
out what should u_n be.
 

338
00:22:17 --> 00:22:20
It'll need a separate
formula to decide.

339
00:22:20 --> 00:22:25
And it could use one step of
backward Euler to find u_1.

340
00:22:26 --> 00:22:33
And then take off, now finding
u_2 from u_1 and u_0.

341
00:22:33 --> 00:22:34
And then onward.
 

342
00:22:34 --> 00:22:36
So that's quite fast.
 

343
00:22:36 --> 00:22:40
More stable, good method.
 

344
00:22:40 --> 00:22:47
And you maybe you can see that
I could get more formulas by

345
00:22:47 --> 00:22:49
going back further in time.
 

346
00:22:49 --> 00:22:54
And by doing that I can
get the accuracy higher.

347
00:22:54 --> 00:22:58
So that's good to see
that particular BDF2.

348
00:22:58 --> 00:23:02
So that's a backward
difference formula.

349
00:23:02 --> 00:23:09
Oh, just to mention
what's developed.

350
00:23:09 --> 00:23:14
So this is stable.
 

351
00:23:14 --> 00:23:17
It actually loses
a little energy.

352
00:23:17 --> 00:23:22
So in fact now both Professor
Bathe and I have studied the

353
00:23:22 --> 00:23:28
possibility of taking a
trapezoidal step, which was a

354
00:23:28 --> 00:23:31
little dangerous in the
non-linear case because you

355
00:23:31 --> 00:23:34
were playing with fire, you
were right on the circle.

356
00:23:34 --> 00:23:40
And then put in a backward
difference step to recover

357
00:23:40 --> 00:23:43
a little stability.
 

358
00:23:43 --> 00:23:47
And then trapezoidal
backward difference.

359
00:23:47 --> 00:23:48
So split step.
 

360
00:23:48 --> 00:23:52
Split the step into a
trapezoidal part and a

361
00:23:52 --> 00:23:53
backward difference part.
 

362
00:23:53 --> 00:23:58
That's actually discussed
later in Chapter 2.

363
00:23:58 --> 00:24:01
Section 2.6 and I might
come back to that.

364
00:24:01 --> 00:24:04
I just wanted to say
that much this morning.

365
00:24:04 --> 00:24:10
Having got as far as forward
and backward Euler.

366
00:24:10 --> 00:24:14
I didn't want to leave you
without a better method which

367
00:24:14 --> 00:24:18
is the trapezoidal method.
 

368
00:24:18 --> 00:24:23
I could take any question.
 

369
00:24:23 --> 00:24:27
I would like to devote the
second half, if you're ok

370
00:24:27 --> 00:24:34
to just change gear, to
beginning on least squares.

371
00:24:34 --> 00:24:41
So this was our kind of
excitement to have time

372
00:24:41 --> 00:24:43
dependence for a little while.
 

373
00:24:43 --> 00:24:48
But now I'm going back to
steady state problems.

374
00:24:48 --> 00:24:49
So they're not moving.
 

375
00:24:49 --> 00:24:51
I'm looking at equilibrium.
 

376
00:24:51 --> 00:25:01
And what I'm going to do now in
then next lectures is more to

377
00:25:01 --> 00:25:07
see this framework that we
identified once for the masses

378
00:25:07 --> 00:25:09
and springs, to see
it again and again.

379
00:25:09 --> 00:25:12
Because it's the basic
framework of applied math.

380
00:25:12 --> 00:25:15
So now I'm ready
for least squares.

381
00:25:15 --> 00:25:17
Least squares.
 

382
00:25:17 --> 00:25:22
What's the problem?
 

383
00:25:22 --> 00:25:27
Well the problem is I'm given
a system of equations Au=f.

384
00:25:27 --> 00:25:34
 
 

385
00:25:34 --> 00:25:42
These f's are observations.
 

386
00:25:42 --> 00:25:45
The u is the unknown vector.
 

387
00:25:45 --> 00:25:47
You say what's different here?
 

388
00:25:47 --> 00:25:49
It's just a linear system.
 

389
00:25:49 --> 00:25:53
What's different is
too many equations.

390
00:25:53 --> 00:26:01
This is an m by n matrix
with m bigger than n.

391
00:26:01 --> 00:26:05
Maybe much bigger than n.
 

392
00:26:05 --> 00:26:07
So what do we do?
 

393
00:26:07 --> 00:26:11
We've got too many
equations and no solution,

394
00:26:11 --> 00:26:18
no exact solution.
 

395
00:26:18 --> 00:26:22
I would say probably that what
we're coming to, what we're

396
00:26:22 --> 00:26:26
starting on today is the most
important, the application of

397
00:26:26 --> 00:26:29
linear algebra that
I see the most.

398
00:26:29 --> 00:26:30
So it interests everybody.
 

399
00:26:30 --> 00:26:36
It interests engineers,
scientists, statisticians,

400
00:26:36 --> 00:26:43
everybody has to deal with this
problem of too many equations.

401
00:26:43 --> 00:26:46
And those equations come from
measurements so you don't

402
00:26:46 --> 00:26:48
want to throw them away.
 

403
00:26:48 --> 00:26:50
I don't want to
just throw away.

404
00:26:50 --> 00:26:54
I want to somehow get
the best solution.

405
00:26:54 --> 00:26:58
I'm looking for the u that
comes closest when I

406
00:26:58 --> 00:27:00
can't find an exact u.
 

407
00:27:00 --> 00:27:02
So that's the idea.
 

408
00:27:02 --> 00:27:05
There's no exact solution
and the problem is

409
00:27:05 --> 00:27:12
what is the best u.
 

410
00:27:12 --> 00:27:14
And I'm going to
call that u hat.

411
00:27:14 --> 00:27:18
My favorite choice
of u will be u hat.

412
00:27:18 --> 00:27:22
So I'm going to get an
equation for u hat.

413
00:27:22 --> 00:27:25
That is my goal.
 

414
00:27:25 --> 00:27:32
And what I'm starting here just
goes all the way to, there will

415
00:27:32 --> 00:27:36
be weighted least squares,
there will be Kalman filters.

416
00:27:36 --> 00:27:42
It's a giant world here of
estimating the best solution

417
00:27:42 --> 00:27:44
when there's noise in
the right-hand side.

418
00:27:44 --> 00:27:46
And what's the model problem?
 

419
00:27:46 --> 00:27:48
Always good to have
a model problem.

420
00:27:48 --> 00:27:51
Let me draw a model problem.
 

421
00:27:51 --> 00:28:00
Model problem is fit
by a straight line.

422
00:28:00 --> 00:28:10
So say C+Dt shall I
use that as the C+Dx.

423
00:28:11 --> 00:28:13
It's got two unknowns.
 

424
00:28:13 --> 00:28:15
So n is two.
 

425
00:28:15 --> 00:28:21
But m is big.
 

426
00:28:21 --> 00:28:22
What do I have?
 

427
00:28:22 --> 00:28:23
Let me draw the picture.
 

428
00:28:23 --> 00:28:26
You've seen this often.
 

429
00:28:26 --> 00:28:29
This is the t direction,
this is the f.

430
00:28:29 --> 00:28:31
These are the measurements.
 

431
00:28:31 --> 00:28:39
So I measure at time
t=0 let's say.

432
00:28:39 --> 00:28:45
I measure my position.
 

433
00:28:45 --> 00:28:46
That would be f_1.
 

434
00:28:47 --> 00:28:51
At time t=1 I've moved
somewhere, I've

435
00:28:51 --> 00:28:52
measured where I am.
 

436
00:28:52 --> 00:28:54
I'm tracking a
satellite, let's say.

437
00:28:54 --> 00:28:56
So I'm tracking this satellite.
 

438
00:28:56 --> 00:28:59
Well the times don't have
to be equally spaced.

439
00:28:59 --> 00:29:03
I'll take the next
time to be three.

440
00:29:03 --> 00:29:10
Let's say the engine is
off this satellite.

441
00:29:10 --> 00:29:14
If the measurements were
perfect-- Does that look

442
00:29:14 --> 00:29:16
too perfect to you?
 

443
00:29:16 --> 00:29:18
It's almost on a straight
line, isn't it?

444
00:29:18 --> 00:29:23
Of course my point is that
measurements, well I mean, of

445
00:29:23 --> 00:29:26
course in reality measurements
would be close to

446
00:29:26 --> 00:29:27
a straight line.
 

447
00:29:27 --> 00:29:31
I'm going to have to draw, in
order for you to see anything,

448
00:29:31 --> 00:29:35
I'm going to have
to draw a really.

449
00:29:35 --> 00:29:39
Suppose f_1 is one.
 

450
00:29:39 --> 00:29:45
Suppose it starts at position
one and suppose f_2 is

451
00:29:45 --> 00:29:50
two and this guy will be.
 

452
00:29:50 --> 00:29:53
Where do you want
me to take it?

453
00:29:53 --> 00:29:58
Let's see, if it was
linear, what would be the?

454
00:29:58 --> 00:30:01
It would be four, right?
 

455
00:30:01 --> 00:30:03
So can I take a
different number?

456
00:30:03 --> 00:30:04
Three.
 

457
00:30:04 --> 00:30:05
Is three okay?
 

458
00:30:05 --> 00:30:09
Because five I haven't
got space for.

459
00:30:09 --> 00:30:12
And you don't want to see pi
or some dumb thing or e.

460
00:30:12 --> 00:30:20
So let me take three.
 

461
00:30:20 --> 00:30:24
I want to fit that data which
is we're saying close to a

462
00:30:24 --> 00:30:28
straight line, I want to fit
it by the best straight line.

463
00:30:28 --> 00:30:32
So the best straight line would
go probably, I don't know what

464
00:30:32 --> 00:30:35
your eye suggests for the best
straight line through

465
00:30:35 --> 00:30:36
three points.
 

466
00:30:36 --> 00:30:40
Do you see I've got three
equations, two unknowns?

467
00:30:40 --> 00:30:41
That's the first point to see.
 

468
00:30:41 --> 00:30:47
Somehow I'm trying to fit three
things with only two degrees of

469
00:30:47 --> 00:30:50
freedom and I'm not going
to succeed usually.

470
00:30:50 --> 00:30:54
But I'm going to do my best and
probably the best line goes

471
00:30:54 --> 00:30:59
sort of, it won't exactly
go through any of them.

472
00:30:59 --> 00:31:03
So I'm doing the best least
squares approximation.

473
00:31:03 --> 00:31:05
And what does that mean?
 

474
00:31:05 --> 00:31:08
Well, what would the
three equations be?

475
00:31:08 --> 00:31:12
What does my linear equations,
my unsolvable ones,

476
00:31:12 --> 00:31:15
say that at time zero?
 

477
00:31:15 --> 00:31:20
So at time zero, at time one
and at time three, at each of

478
00:31:20 --> 00:31:24
those times I have an equation
C+Dt should agree with.

479
00:31:24 --> 00:31:28
So that C+D time zero
should match f_1.

480
00:31:31 --> 00:31:34
At t=1 my line will be C+D*1.
 

481
00:31:34 --> 00:31:37
 
 

482
00:31:37 --> 00:31:38
It should equal f_2.
 

483
00:31:40 --> 00:31:48
And at t=3, the height of the
line will be C+3d and I would

484
00:31:48 --> 00:31:50
like it to go through
that height f_3.

485
00:31:50 --> 00:31:53
 
 

486
00:31:53 --> 00:31:58
But I'm not going to be able.
 

487
00:31:58 --> 00:32:01
If there was noise in the
measurements that system,

488
00:32:01 --> 00:32:06
that's my unsolvable system.
 

489
00:32:06 --> 00:32:08
What's the matrix?
 

490
00:32:08 --> 00:32:12
I want to write three equations
and you're getting good at

491
00:32:12 --> 00:32:19
seeing three equations like so.
 

492
00:32:19 --> 00:32:24
So I've a 3 by 2 matrix.
 

493
00:32:24 --> 00:32:27
And my unknown u is C, D.
 

494
00:32:28 --> 00:32:30
Those are my unknowns.
 

495
00:32:30 --> 00:32:33
And my right-hand sides are
these heights, well, I decided

496
00:32:33 --> 00:32:40
on particular numbers,
one, two and three.

497
00:32:40 --> 00:32:42
One, two, three.
 

498
00:32:42 --> 00:32:44
And what's the matrix?
 

499
00:32:44 --> 00:32:47
What's the matrix A that
you read off when you see

500
00:32:47 --> 00:32:50
that system of equations?
 

501
00:32:50 --> 00:32:52
The first column
of the matrix is?

502
00:32:52 --> 00:32:55
All ones because that's
multiplying the C's.

503
00:32:55 --> 00:32:59
And the second column of
the matrix is the times.

504
00:32:59 --> 00:33:02
Zero, one, three,
is that right?

505
00:33:02 --> 00:33:04
That multiply the D.
 

506
00:33:04 --> 00:33:08
So this is the same as that.
 

507
00:33:08 --> 00:33:12
So here I'm in my set-up.
 

508
00:33:12 --> 00:33:24
I'll erase m equal big because
m was only three, not that big.

509
00:33:24 --> 00:33:30
What's the best answer?
 

510
00:33:30 --> 00:33:32
What's the best u hat?
 

511
00:33:32 --> 00:33:38
The best u hat will now
be C hat and D hat.

512
00:33:38 --> 00:33:45
The best I can do.
 

513
00:33:45 --> 00:33:50
I need some idea of
what does best mean.

514
00:33:50 --> 00:33:55
And there is not a single
possible meaning.

515
00:33:55 --> 00:33:59
There are many possible ways
I could say the best line.

516
00:33:59 --> 00:34:06
One way would be to make
the, well, what could

517
00:34:06 --> 00:34:09
a best line be?
 

518
00:34:09 --> 00:34:11
I'm going to have three
errors here, right?

519
00:34:11 --> 00:34:14
That did not go right
through the point.

520
00:34:14 --> 00:34:15
This did not go right
through the point.

521
00:34:15 --> 00:34:17
They came pretty close.
 

522
00:34:17 --> 00:34:22
I've got three small
errors. e_1, e_2, e_3.

523
00:34:23 --> 00:34:26
Those are the errors
in my equations.

524
00:34:26 --> 00:34:31
So I will get equality when
I add in the e_1, e_2, e_3,

525
00:34:31 --> 00:34:39
the little bits that will
bring it onto the line.

526
00:34:39 --> 00:34:41
One idea.
 

527
00:34:41 --> 00:34:46
Make the largest error
as small as I can.

528
00:34:46 --> 00:34:54
Minimize the maximum of the e's
Try to balance them so no e,

529
00:34:54 --> 00:34:58
no error is bigger
than the others.

530
00:34:58 --> 00:35:00
Look for that balance.
 

531
00:35:00 --> 00:35:04
That's a reasonable idea.
 

532
00:35:04 --> 00:35:07
But it's not the
least squares idea.

533
00:35:07 --> 00:35:11
So what's the least
squares idea?

534
00:35:11 --> 00:35:15
The least squares idea makes
the sum of the squares of the

535
00:35:15 --> 00:35:17
errors as small as possible.
 

536
00:35:17 --> 00:35:29
So the least squares idea will
be to minimize the sum of the

537
00:35:29 --> 00:35:35
squares of the errors. e_1
squared plus e_m squared.

538
00:35:35 --> 00:35:38
It would be just e_1
squared plus e_2 squared

539
00:35:38 --> 00:35:42
plus e_3 squared.
 

540
00:35:42 --> 00:35:44
What is this?
 

541
00:35:44 --> 00:35:48
Let me began to write
this in matrix.

542
00:35:48 --> 00:35:51
I want to bring in
the matrix here.

543
00:35:51 --> 00:35:56
This is the error.
 

544
00:35:56 --> 00:35:59
The error is the
difference between the

545
00:35:59 --> 00:36:01
measurements and Au.
 

546
00:36:02 --> 00:36:06
So that's what I'm
trying to make small.

547
00:36:06 --> 00:36:10
I'd love to make it zero
but I can't. i've got more

548
00:36:10 --> 00:36:12
equations than unknowns.
 

549
00:36:12 --> 00:36:14
There's no two unknowns
that will make all

550
00:36:14 --> 00:36:16
three errors zero.
 

551
00:36:16 --> 00:36:18
So I want to make
the errors small.

552
00:36:18 --> 00:36:23
And this is the
length of e squared.

553
00:36:23 --> 00:36:30
The length in this sum
of squares method.

554
00:36:30 --> 00:36:37
It's a pretty good
measure of the error.

555
00:36:37 --> 00:36:41
Gauss was the first to
apply least squares.

556
00:36:41 --> 00:36:45
What I'm going to
do today is Gauss.

557
00:36:45 --> 00:36:48
Who was, by the way, the
greatest mathematician

558
00:36:48 --> 00:36:52
of all time.
 

559
00:36:52 --> 00:36:54
And here, he was doing
astronomy actually.

560
00:36:54 --> 00:37:03
And writing in latin.
 

561
00:37:03 --> 00:37:08
The message got out somehow.
 

562
00:37:08 --> 00:37:12
So his idea was sum of squares.
 

563
00:37:12 --> 00:37:18
So this e is the distance
between f and Au.

564
00:37:18 --> 00:37:22
I have to begin to write.
 

565
00:37:22 --> 00:37:23
I have to write some things.
 

566
00:37:23 --> 00:37:28
I can write some things out in
detail, but then I also, at the

567
00:37:28 --> 00:37:33
same time, have to carry along
the way I would look at it for

568
00:37:33 --> 00:37:36
any matrix A and any
right-hand side f.

569
00:37:36 --> 00:37:39
So do you see that
this is the error?

570
00:37:39 --> 00:37:44
The meaning of this double
bars squared is exactly that.

571
00:37:44 --> 00:37:46
That it's the sum of the
squares of the components.

572
00:37:46 --> 00:37:54
So that's where the word
least squares come in.

573
00:37:54 --> 00:38:03
Can I just say what's better
about least squares and

574
00:38:03 --> 00:38:06
what's maybe a drawback.
 

575
00:38:06 --> 00:38:09
So actually this next
sentence is pretty

576
00:38:09 --> 00:38:12
important in practice.
 

577
00:38:12 --> 00:38:15
What's better about least
squares, what's really nice

578
00:38:15 --> 00:38:21
about least squares is well,
for one thing, the equations

579
00:38:21 --> 00:38:25
we get for the best C, D
will be linear, will

580
00:38:25 --> 00:38:29
be linear equations.
 

581
00:38:29 --> 00:38:33
You may say, not surprising,
I started out with a linear

582
00:38:33 --> 00:38:39
system and I'm going to end
up with a linear system.

583
00:38:39 --> 00:38:45
Actually I prefer to use
well, might be too late.

584
00:38:45 --> 00:38:47
Next lecture I'm going
to put b in there for

585
00:38:47 --> 00:38:49
the right-hand side.
 

586
00:38:49 --> 00:38:54
But I'll leave it
with that for now.

587
00:38:54 --> 00:38:57
So good point is we'll
get linear equations.

588
00:38:57 --> 00:39:06
The not so good point in some
applications is when I look at

589
00:39:06 --> 00:39:16
the squares of errors, well big
errors, outliers, really bad

590
00:39:16 --> 00:39:20
measurements have a big
influence on the answer

591
00:39:20 --> 00:39:22
because of getting squared.
 

592
00:39:22 --> 00:39:30
So if I have ten readings that
are very accurate but then in

593
00:39:30 --> 00:39:36
an eleventh reading that is way
off and I don't know it and if

594
00:39:36 --> 00:39:40
I don't realize that that's way
off, then that eleventh error

595
00:39:40 --> 00:39:45
will-- It's like having a whole
lot of points close to a line

596
00:39:45 --> 00:39:48
and then another point way off.
 

597
00:39:48 --> 00:39:55
That will have a significant
effect on the best line.

598
00:39:55 --> 00:40:02
So you might say too
great an effect.

599
00:40:02 --> 00:40:05
Depends on the application.
 

600
00:40:05 --> 00:40:11
I just had to say before
starting on least squares, as

601
00:40:11 --> 00:40:17
always, there are advantages
and disadvantages but

602
00:40:17 --> 00:40:18
the advantages are
very, very great.

603
00:40:18 --> 00:40:25
So it's an important idea
here, least squares.

604
00:40:25 --> 00:40:31
I'm ready now to ask for
the equation for u hat.

605
00:40:31 --> 00:40:38
So the equation for u hat is
the u that minimizes here.

606
00:40:38 --> 00:40:45
So we have touched on
minimizing quadratics.

607
00:40:45 --> 00:40:54
This is squares.
 

608
00:40:54 --> 00:40:59
I could expand that out as
f minus Au transpose times

609
00:40:59 --> 00:41:06
f minus Au just to see
another way to write it.

610
00:41:06 --> 00:41:11
The length squared of a vector
is always the transpose.

611
00:41:11 --> 00:41:14
It's inner product with itself.
 

612
00:41:14 --> 00:41:19
And I could split this out into
all these different terms.

613
00:41:19 --> 00:41:24
I would have then, some
quadratic equation to minimize.

614
00:41:24 --> 00:41:28
In other words, let me
jump to the answer.

615
00:41:28 --> 00:41:33
Let me jump to the
equation for the best u.

616
00:41:33 --> 00:41:38
And then come back to see why.
 

617
00:41:38 --> 00:41:41
Because you must see
what that equation is.

618
00:41:41 --> 00:41:46
It's the fundamental equation
of, this might be called linear

619
00:41:46 --> 00:41:49
regression, fitting data.
 

620
00:41:49 --> 00:41:51
You're just
constantly doing it.

621
00:41:51 --> 00:41:56
So what is the equation
for the best u hat?

622
00:41:56 --> 00:41:58
Can I put it here?
 

623
00:41:58 --> 00:42:01
This'll be equation
that we get to.

624
00:42:01 --> 00:42:04
It'll be A transpose A.
 

625
00:42:04 --> 00:42:07
You're not surprised to see
A transpose A up here.

626
00:42:07 --> 00:42:12
First of all because this is
18.085 and also because this A

627
00:42:12 --> 00:42:16
is rectangular and when you
have rectangular matrices,

628
00:42:16 --> 00:42:19
sooner or later A
transpose A comes up.

629
00:42:19 --> 00:42:21
So that's the matrix.
 

630
00:42:21 --> 00:42:26
And then the right-hand
side is A transpose f.

631
00:42:26 --> 00:42:30
So that's the key equation
for least squares.

632
00:42:30 --> 00:42:33
That's the central equation
of least squares.

633
00:42:33 --> 00:42:37
And let's just see
what it looks like.

634
00:42:37 --> 00:42:41
You could say the way I arrived
at it, Imean the short way

635
00:42:41 --> 00:42:45
is this is an equation
that I can't satisfy.

636
00:42:45 --> 00:42:56
I multiply both sides by A
transpose and now this is

637
00:42:56 --> 00:43:00
the equation for u hat.
 

638
00:43:00 --> 00:43:05
And what that did was kind of
average out the m equations.

639
00:43:05 --> 00:43:07
Because how many
equations do I now have?

640
00:43:07 --> 00:43:09
A as m by n.
 

641
00:43:09 --> 00:43:13
What's the shape
of A transpose A?

642
00:43:13 --> 00:43:15
Everybody's on top
of that, right?

643
00:43:15 --> 00:43:21
The shape of A transpose
A is square, n by n.

644
00:43:21 --> 00:43:28
Because A transpose is n
by m. n by m times m by

645
00:43:28 --> 00:43:30
n leaves us an n by n.
 

646
00:43:30 --> 00:43:34
So we've averaged the m
equations that were too

647
00:43:34 --> 00:43:38
many to get n equations.
 

648
00:43:38 --> 00:43:47
And of course this is what it
should be, n by m m by one.

649
00:43:47 --> 00:43:48
So it's m by one.
 

650
00:43:48 --> 00:43:51
It's a good right-hand side.
 

651
00:43:51 --> 00:44:02
That's the equation
of least squares.

652
00:44:02 --> 00:44:07
That's the equation I want to
explain, understand and solve.

653
00:44:07 --> 00:44:10
Actually why don't we solve it
for this particular problem

654
00:44:10 --> 00:44:17
just to see the whole
thing for this example.

655
00:44:17 --> 00:44:22
Just to do it.
 

656
00:44:22 --> 00:44:24
So there is A.
 

657
00:44:24 --> 00:44:25
Here is u.
 

658
00:44:25 --> 00:44:32
And here is f. what
shall I call these?

659
00:44:32 --> 00:44:37
They're mostly called the
normal equations. that's one

660
00:44:37 --> 00:44:42
possible word for the key
equation of least squares,

661
00:44:42 --> 00:44:44
the normal equations.
 

662
00:44:44 --> 00:44:47
Can you tell me these
matrices A transpose A.

663
00:44:47 --> 00:44:50
And u hat I know.
 

664
00:44:50 --> 00:44:53
That'll be the best
C and the best D.

665
00:44:53 --> 00:44:59
And over here can you
compute A transpose f?

666
00:44:59 --> 00:45:04
If I write A transpose above
it, will that help you do

667
00:45:04 --> 00:45:05
these multiplications?
 

668
00:45:05 --> 00:45:06
Let me just do.
 

669
00:45:06 --> 00:45:08
So there was the matrix A.
 

670
00:45:08 --> 00:45:12
Let me write A
transpose above it.

671
00:45:12 --> 00:45:22
So it has a row of ones
and then a row of times.

672
00:45:22 --> 00:45:25
So what shape is the matrix?
 

673
00:45:25 --> 00:45:30
The A transpose A matrix.
 

674
00:45:30 --> 00:45:33
It's going to be that A
transpose times that A.

675
00:45:33 --> 00:45:37
The size will be
two by two, right?

676
00:45:37 --> 00:45:41
Two by three times three by
two, it's averaging out to

677
00:45:41 --> 00:45:43
get me a two by two matrix.
 

678
00:45:43 --> 00:45:46
What's the first entry of this?
 

679
00:45:46 --> 00:45:50
Can you do A transpose times A
just so we see this matrix.

680
00:45:50 --> 00:45:52
Three.
 

681
00:45:52 --> 00:45:54
And off the diagonal?
 

682
00:45:54 --> 00:45:55
Four.
 

683
00:45:55 --> 00:45:56
And here?
 

684
00:45:56 --> 00:45:58
And you knew it
would be symmetric.

685
00:45:58 --> 00:45:59
And here?
 

686
00:45:59 --> 00:46:03
Ten.
 

687
00:46:03 --> 00:46:06
And tell me A transpose
f while we're at it.

688
00:46:06 --> 00:46:08
So that's just a vector.
 

689
00:46:08 --> 00:46:11
If you multiply that by
the right-hand sides,

690
00:46:11 --> 00:46:12
looks like a six.
 

691
00:46:12 --> 00:46:18
Is that right?
 

692
00:46:18 --> 00:46:21
11, maybe.
 

693
00:46:21 --> 00:46:24
Is that right?
 

694
00:46:24 --> 00:46:29
Two, nine making 11, yeah.
 

695
00:46:29 --> 00:46:31
So those are the numbers.
 

696
00:46:31 --> 00:46:36
I can't write those numbers,
write A transpose A without

697
00:46:36 --> 00:46:39
asking you to tell me one
more time what kind of a

698
00:46:39 --> 00:46:42
matrix have I got here?
 

699
00:46:42 --> 00:46:44
It's symmetric positive
definite, right?

700
00:46:44 --> 00:46:47
We know that's going to be.
 

701
00:46:47 --> 00:46:50
And we see it.
 

702
00:46:50 --> 00:46:54
Our test for positive definite
might be the determinant, that

703
00:46:54 --> 00:46:57
one by one determinant is
three, that two by two

704
00:46:57 --> 00:47:04
determinant is 30 minus
16; 14 positive.

705
00:47:04 --> 00:47:06
We got a good problem here.
 

706
00:47:06 --> 00:47:09
And I could solve for
C hat and D hat.

707
00:47:09 --> 00:47:14
I see the numbers are not
coming out fantastically but

708
00:47:14 --> 00:47:18
they would produce a line that
would, I'm pretty sure, it

709
00:47:18 --> 00:47:25
would be, this looks
optimal to me.

710
00:47:25 --> 00:47:29
If I rotated any, I'm going
to make things worse.

711
00:47:29 --> 00:47:31
I think it would
look like that.

712
00:47:31 --> 00:47:35
So that's the system
that you end up with.

713
00:47:35 --> 00:47:37
But why?
 

714
00:47:37 --> 00:47:41
You really have to
understand where did

715
00:47:41 --> 00:47:44
this equation come from.
 

716
00:47:44 --> 00:47:50
Where did it come from?
 

717
00:47:50 --> 00:47:56
It's worth understanding,
this least squares stuff.

718
00:47:56 --> 00:48:01
So I'm going to try to draw a
picture that makes it clear

719
00:48:01 --> 00:48:08
where that equation comes from.
 

720
00:48:08 --> 00:48:11
So what am I doing here?
 

721
00:48:11 --> 00:48:12
Au=f.
 

722
00:48:12 --> 00:48:15
 
 

723
00:48:15 --> 00:48:18
Start there.
 

724
00:48:18 --> 00:48:22
And the particular A was,
I'll even copy the A.

725
00:48:22 --> 00:48:27
It was 1, 1, 1; 0, 1, 3
multiplied u to give

726
00:48:27 --> 00:48:30
me f as .
 

727
00:48:30 --> 00:48:32
But of course, I couldn't
solve it because I don't

728
00:48:32 --> 00:48:35
have enough unknowns.
 

729
00:48:35 --> 00:48:39
What's the picture?
 

730
00:48:39 --> 00:48:46
Everybody likes to see what's
happening by a picture

731
00:48:46 --> 00:48:49
as well as by algebra.
 

732
00:48:49 --> 00:48:56
So the picture here is I'm
in three dimensions and I

733
00:48:56 --> 00:48:57
have a vector .
 

734
00:48:57 --> 00:49:01
So 0 in that direction,
one in that, three up.

735
00:49:01 --> 00:49:07
So somewhere there is my f.
 

736
00:49:07 --> 00:49:15
Now I'll put in C, D here.
 

737
00:49:15 --> 00:49:19
What is the equation
asking me to do?

738
00:49:19 --> 00:49:22
Which actually, I won't be able
to do because I can't solve it.

739
00:49:22 --> 00:49:35
But the equation, how do we see
a system of linear equations?

740
00:49:35 --> 00:49:39
If I have a system of linear
equations I'm looking for

741
00:49:39 --> 00:49:49
numbers C and D so that C
times column one plus D times

742
00:49:49 --> 00:49:52
column two gives me that.
 

743
00:49:52 --> 00:49:55
That's how I think of a
system of equations.

744
00:49:55 --> 00:50:00
A combination of the columns.
 

745
00:50:00 --> 00:50:04
Tell me what vectors do I
get if I take combinations

746
00:50:04 --> 00:50:05
of the columns.
 

747
00:50:05 --> 00:50:08
Well, if I took the combination
C=1, D=0 I would just

748
00:50:08 --> 00:50:09
get the first column.
 

749
00:50:09 --> 00:50:14
So that's a candidate. <1, 1,
1>, I don't know where that is.

750
00:50:14 --> 00:50:20
Wherever  might be.
 

751
00:50:20 --> 00:50:23
I'm not too sure where
to draw .

752
00:50:23 --> 00:50:27
I want to go one there,
one there and one there.

753
00:50:27 --> 00:50:31
Damn.
 

754
00:50:31 --> 00:50:36
Let's define that to be the
vector  right there.

755
00:50:36 --> 00:50:36
Wait.
 

756
00:50:36 --> 00:50:43
You let me put that up there
and I didn't mean to, right? f

757
00:50:43 --> 00:50:48
should be zero, what, sorry?
f was , yeah.

758
00:50:48 --> 00:50:49
Damn!
 

759
00:50:49 --> 00:50:52
Don't let me make mistakes.
 

760
00:50:52 --> 00:50:57
These mistakes are permanent
if you let them slide by.

761
00:50:57 --> 00:51:04
That's it, same point.
 

762
00:51:04 --> 00:51:06
I didn't have the point
right in the first place

763
00:51:06 --> 00:51:08
so now it's just perfect.
 

764
00:51:08 --> 00:51:15
There it is.
 

765
00:51:15 --> 00:51:18
Before of course, if I
had  I could've

766
00:51:18 --> 00:51:21
solved the equation.
 

767
00:51:21 --> 00:51:26
But with  I can't.
 

768
00:51:26 --> 00:51:27
Here's the situation.
 

769
00:51:27 --> 00:51:31
This vector is not a
combination of those two.

770
00:51:31 --> 00:51:36
Because the combinations of two
vectors, what's the picture?

771
00:51:36 --> 00:51:40
If I try to draw, if I look at
all combinations of two

772
00:51:40 --> 00:51:45
vectors,  which is
that vector,  which is

773
00:51:45 --> 00:51:49
maybe this vector
let's just say.

774
00:51:49 --> 00:51:52
If I take the combinations
of these two column

775
00:51:52 --> 00:51:55
vectors, what do I get?
 

776
00:51:55 --> 00:51:57
Now this is for
everybody to know.

777
00:51:57 --> 00:52:00
If I take the combinations
of two vectors here

778
00:52:00 --> 00:52:04
in three-dimensional
space I get a plane.

779
00:52:04 --> 00:52:08
I get the plane that
contains those vectors.

780
00:52:08 --> 00:52:14
So this I could call the column
plane or the column space.

781
00:52:14 --> 00:52:21
This is all combinations
of the columns.

782
00:52:21 --> 00:52:29
That's the same thing as saying
this is all the f's that

783
00:52:29 --> 00:52:38
have exact solutions.
 

784
00:52:38 --> 00:52:41
So let's just see this picture.
 

785
00:52:41 --> 00:52:45
This particular right-hand side
is not in the plane, right?

786
00:52:45 --> 00:52:46
That's my problem.
 

787
00:52:46 --> 00:52:51
This particular vector f
points out of the plane.

788
00:52:51 --> 00:52:53
But if I change it a
little, like if I

789
00:52:53 --> 00:52:56
change it to .
 

790
00:52:56 --> 00:52:57
Do you see that that would?
 

791
00:52:57 --> 00:53:00
What's different now that
I've changed it to <1,

792
00:53:00 --> 00:53:05
2, 3> for a moment?
 

793
00:53:05 --> 00:53:07
What's different
about ?

794
00:53:07 --> 00:53:11
Where is 
in my picture?

795
00:53:11 --> 00:53:14
Do you see what's great
about ?

796
00:53:14 --> 00:53:16
It is a combination.
 

797
00:53:16 --> 00:53:18
Right? 
is a combination.

798
00:53:18 --> 00:53:19
With C=1, D=1.
 

799
00:53:20 --> 00:53:24
It would satisfy the equation.
 

800
00:53:24 --> 00:53:28
So where is <1, 2,
4> in my picture?

801
00:53:28 --> 00:53:31
It's in the plane.
 

802
00:53:31 --> 00:53:38
The plane are the heights that
do lie on a straight line.

803
00:53:38 --> 00:53:42
So the plane are all the ones
that I can get exactly.

804
00:53:42 --> 00:53:49
But this vector, these
observations, 

805
00:53:49 --> 00:53:50
I couldn't get exactly.
 

806
00:53:50 --> 00:53:55
So let me, in 30 seconds
or less, let me tell you

807
00:53:55 --> 00:53:57
the best thing to do.
 

808
00:53:57 --> 00:53:59
Or let you tell me the
best thing to do.

809
00:53:59 --> 00:54:02
I have a right-hand side
that's not in the plane.

810
00:54:02 --> 00:54:06
I can get my straight lines
correspond to vectors,

811
00:54:06 --> 00:54:07
right-hand sides that
are in the plane.

812
00:54:07 --> 00:54:10
So what do I do?
 

813
00:54:10 --> 00:54:12
I project.
 

814
00:54:12 --> 00:54:16
I take the nearest point
that is the plane as

815
00:54:16 --> 00:54:18
my right-hand side.
 

816
00:54:18 --> 00:54:21
I project down.
 

817
00:54:21 --> 00:54:24
And it's that projection that's
going to lead us to the

818
00:54:24 --> 00:54:30
equation that I'm shooting
for, A transpose Au hat

819
00:54:30 --> 00:54:32
equals A transpose f.
 

820
00:54:32 --> 00:54:40
This comes from projecting f
down into the plane where

821
00:54:40 --> 00:54:43
straight lines do work exactly.
 

822
00:54:43 --> 00:54:47
So there's an error here
that I can't deal with.

823
00:54:47 --> 00:54:50
And there's a part here,
the projection part,

824
00:54:50 --> 00:54:52
that I can deal with.
 

825
00:54:52 --> 00:54:56
This is important and
it's fun and we'll come

826
00:54:56 --> 00:54:57
back to it Monday.
 

827
00:54:57 --> 00:54:59
Thanks for patience.