WEBVTT

00:00:01.550 --> 00:00:03.920
The following content is
provided under a Creative

00:00:03.920 --> 00:00:05.310
Commons license.

00:00:05.310 --> 00:00:07.520
Your support will help
MIT OpenCourseWare

00:00:07.520 --> 00:00:11.610
continue to offer high quality
educational resources for free.

00:00:11.610 --> 00:00:14.180
To make a donation or to
view additional materials

00:00:14.180 --> 00:00:18.140
from hundreds of MIT courses,
visit MIT OpenCourseWare

00:00:18.140 --> 00:00:19.026
at ocw.mit.edu.

00:00:22.340 --> 00:00:26.860
GILBERT STRANG:
OK, so actually, I

00:00:26.860 --> 00:00:29.560
know where people are
working on projects,

00:00:29.560 --> 00:00:34.030
and you're not responsible
for any new material

00:00:34.030 --> 00:00:35.810
in the lectures.

00:00:35.810 --> 00:00:37.240
Thank you for coming.

00:00:37.240 --> 00:00:42.290
But I do have something,
an important topic,

00:00:42.290 --> 00:00:46.570
which is a revised version
about the construction

00:00:46.570 --> 00:00:48.910
of neural nets,
the basic structure

00:00:48.910 --> 00:00:51.130
that we're working with.

00:00:51.130 --> 00:01:00.310
So that's on the open
web at section 7.1,

00:01:00.310 --> 00:01:08.980
so Construction of Neural Nets.

00:01:13.890 --> 00:01:23.870
Really, it's a construction
of the learning function, F.

00:01:23.870 --> 00:01:27.870
So that's the function
that you optimize

00:01:27.870 --> 00:01:32.150
by gradient descent or
stochastic gradient descent,

00:01:32.150 --> 00:01:40.560
and you apply to the training
data to minimize the loss.

00:01:40.560 --> 00:01:45.470
So I'm just thinking about
it in a more organized way,

00:01:45.470 --> 00:01:49.130
because I wrote that section
before I knew anything

00:01:49.130 --> 00:01:52.400
more than how to
spell neural nets,

00:01:52.400 --> 00:01:56.170
but now I've thought
about it more.

00:01:56.170 --> 00:02:05.820
So the key point maybe, compared
to what I had in the past,

00:02:05.820 --> 00:02:09.930
is that I now think of
this as a function of two

00:02:09.930 --> 00:02:21.240
sets of variables, x and
v. So x are the weights,

00:02:21.240 --> 00:02:29.775
and v are the feature vectors,
the sample feature vectors.

00:02:36.490 --> 00:02:42.030
So those come from the training
data, either one at a time,

00:02:42.030 --> 00:02:44.220
if we're doing
stochastic gradient

00:02:44.220 --> 00:02:47.160
descent with mini-batch size 1.

00:02:47.160 --> 00:02:51.570
Or B at a time, if we're
doing mini-batch of size B,

00:02:51.570 --> 00:02:54.190
or the whole thing,
a whole epoch

00:02:54.190 --> 00:02:59.490
at once, if we're doing
full-scale gradient descent.

00:02:59.490 --> 00:03:01.890
So those are the
feature vectors,

00:03:01.890 --> 00:03:12.670
and these are the numbers in
the linear steps, the weights.

00:03:12.670 --> 00:03:24.360
So they're the matrices AK that
you multiply by, multiply v by.

00:03:24.360 --> 00:03:34.270
And also the bias vectors
bK that you add on

00:03:34.270 --> 00:03:36.970
to shift the origin.

00:03:36.970 --> 00:03:39.640
OK.

00:03:39.640 --> 00:03:44.260
It's these that you optimize,
those are to optimize.

00:03:49.010 --> 00:03:57.620
And what's the structure of the
whole of the learning function,

00:03:57.620 --> 00:03:59.610
and how do you use it?

00:03:59.610 --> 00:04:02.200
What does a neural
net look like?

00:04:02.200 --> 00:04:09.700
So you take F of a
first set of weights,

00:04:09.700 --> 00:04:17.500
so F of the first set of
weights would be A1 and B1,

00:04:17.500 --> 00:04:20.140
so that's x part.

00:04:20.140 --> 00:04:27.190
And the actual sample
vector, the sample vectors

00:04:27.190 --> 00:04:32.320
are v0 in the iteration.

00:04:32.320 --> 00:04:38.140
And then you do the nonlinear
step to each component,

00:04:38.140 --> 00:04:41.050
and that produces v1.

00:04:41.050 --> 00:04:44.620
So there is a typical--

00:04:44.620 --> 00:04:51.620
I could write out what this
is here, A1 v0 plus b1.

00:04:54.940 --> 00:04:57.950
The two steps are
the linear step.

00:04:57.950 --> 00:05:00.600
The endpoint is v0.

00:05:00.600 --> 00:05:06.240
You take the linear step using
the first weights, A1 and b1.

00:05:06.240 --> 00:05:11.190
Then, you takes a nonlinear
step, and that gives you v1.

00:05:11.190 --> 00:05:14.790
So that really better
than my line above,

00:05:14.790 --> 00:05:17.520
so I'll erase that line above.

00:05:17.520 --> 00:05:18.020
Yeah.

00:05:25.340 --> 00:05:32.070
So that produces v1 from
v0 and the first weights.

00:05:32.070 --> 00:05:36.980
And then the next
level inputs v1,

00:05:36.980 --> 00:05:43.550
so I'll just call
this vK or vK minus 1,

00:05:43.550 --> 00:05:45.970
and I'll call this one vK.

00:05:45.970 --> 00:05:51.320
OK, so K equal to 1 up
to however many layers,

00:05:51.320 --> 00:05:52.740
you are l layers.

00:05:56.790 --> 00:05:59.190
So the input was v0.

00:05:59.190 --> 00:06:03.420
So this v is really
v0, you could say.

00:06:03.420 --> 00:06:12.120
And this is the neural net, and
this is the input and output

00:06:12.120 --> 00:06:13.980
from each layer.

00:06:13.980 --> 00:06:20.050
And then vl is the final
output from the final layer.

00:06:20.050 --> 00:06:23.370
So let's just do a picture here.

00:06:23.370 --> 00:06:29.820
Here is v0, a sample
vector, or if we're

00:06:29.820 --> 00:06:37.320
doing image processing, it's
all the pixels in the data,

00:06:37.320 --> 00:06:39.570
in the training.

00:06:39.570 --> 00:06:43.440
From one sample, this
is one training sample.

00:06:50.340 --> 00:06:55.470
And then you multiply
by A1, and you add b1.

00:06:55.470 --> 00:07:04.190
And you take ReLU of that
vector, and that gives you v1.

00:07:04.190 --> 00:07:09.480
That gives you v1,
and then you iterate

00:07:09.480 --> 00:07:14.010
to finally vl, the last layer.

00:07:14.010 --> 00:07:16.680
You don't do ReLU
at the last layer,

00:07:16.680 --> 00:07:23.430
so it's just Al vl
minus 1 plus bl.

00:07:23.430 --> 00:07:27.480
And you may not do a bias
vector also at that layer,

00:07:27.480 --> 00:07:32.310
but you might, and this
is the finally the output.

00:07:35.910 --> 00:07:37.800
So that picture
is clearer for me

00:07:37.800 --> 00:07:41.220
than it was previously
to distinguish

00:07:41.220 --> 00:07:44.370
between the weights.

00:07:44.370 --> 00:07:49.410
So in the gradient
descent algorithm,

00:07:49.410 --> 00:07:51.990
it's these x's that
you're choosing.

00:07:51.990 --> 00:07:55.980
The v's are given by
the training data.

00:07:55.980 --> 00:07:59.850
That's not part of
the optimization part.

00:07:59.850 --> 00:08:02.850
It's x in chapter
6, where you're

00:08:02.850 --> 00:08:05.170
finding the optimal weights.

00:08:05.170 --> 00:08:12.980
So this x really stands
for all the weights

00:08:12.980 --> 00:08:26.370
that you compute up
to Al, bl, so that's

00:08:26.370 --> 00:08:27.840
a collection of all the weights.

00:08:27.840 --> 00:08:32.100
And the important part for
applications for practice is

00:08:32.100 --> 00:08:36.150
to realize that there are
often more weights and more

00:08:36.150 --> 00:08:39.299
components in the weights
than there are components

00:08:39.299 --> 00:08:43.679
in the feature vectors, in
the samples, in the v's.

00:08:43.679 --> 00:08:48.870
So often, the size of x is
greater than the size of v's

00:08:48.870 --> 00:08:54.720
which is an interesting and
sort of unexpected situation.

00:08:54.720 --> 00:08:58.160
So often, I'll just write that.

00:08:58.160 --> 00:09:03.290
Often, the x's are the weights.

00:09:06.800 --> 00:09:23.930
x's are underdetermined, because
the number of x's exceeds,

00:09:23.930 --> 00:09:27.300
and often far exceeds,
the number of v's,

00:09:27.300 --> 00:09:32.490
the number of the cardinality,
the number of weights.

00:09:32.490 --> 00:09:39.380
This is in the A's
and b's, and these

00:09:39.380 --> 00:09:43.180
are in the samples
in the training

00:09:43.180 --> 00:09:57.010
set, the number of features of
all the samples in the training

00:09:57.010 --> 00:09:58.060
set.

00:09:58.060 --> 00:10:05.710
So I'll get that new
section 7.1 up hopefully

00:10:05.710 --> 00:10:09.370
this week on the open--

00:10:09.370 --> 00:10:15.250
that's the open set-- and
I'll email to you on Stellar.

00:10:15.250 --> 00:10:17.890
Is there more I
should say about this?

00:10:17.890 --> 00:10:21.250
You see here, I can
draw the picture,

00:10:21.250 --> 00:10:23.140
but of course, a
hand-drawn picture

00:10:23.140 --> 00:10:30.850
is far inferior to a
machine-drawn picture

00:10:30.850 --> 00:10:33.820
an online picture,
but let me just do it.

00:10:33.820 --> 00:10:40.090
So there is v, the training
sample has some components,

00:10:40.090 --> 00:10:42.250
and then they're multiplied.

00:10:42.250 --> 00:10:51.970
Now, here is going to be v1,
the first hidden layer, and that

00:10:51.970 --> 00:11:02.620
can have a different number of
components in the first layer,

00:11:02.620 --> 00:11:04.600
a different number of neurons.

00:11:04.600 --> 00:11:10.960
And then each one
comes from the v's--

00:11:10.960 --> 00:11:16.780
so I will keep going here,
but you see the picture.

00:11:16.780 --> 00:11:21.400
So that describes a
matrix A1 that tells you

00:11:21.400 --> 00:11:24.100
what the weights are on
those, and then there's

00:11:24.100 --> 00:11:28.590
a b1 that's added.

00:11:28.590 --> 00:11:34.290
The bias vector is added
to all those to get the v1.

00:11:34.290 --> 00:11:42.220
so v1 is A1 v0 plus
b1, and then onwards.

00:11:42.220 --> 00:11:46.750
So this is the spot
where drawing it by hand

00:11:46.750 --> 00:11:53.560
is clearly inferior to any
other possible way to do it.

00:11:53.560 --> 00:11:55.690
OK.

00:11:55.690 --> 00:12:03.430
So now, I haven't yet put into
the picture the loss function.

00:12:03.430 --> 00:12:08.410
So that's the function
that you want to minimize.

00:12:08.410 --> 00:12:09.675
So what is the loss function?

00:12:13.420 --> 00:12:18.740
So we're choosing x to--

00:12:18.740 --> 00:12:20.830
that's all the A's and b's--

00:12:20.830 --> 00:12:27.250
to minimize the
loss, function L.

00:12:27.250 --> 00:12:28.600
OK.

00:12:28.600 --> 00:12:34.540
So it's this part that Professor
Sra's lecture was about.

00:12:34.540 --> 00:12:45.490
So he said, L is often a
finite sum over all of F.

00:12:45.490 --> 00:12:46.630
So what would that be?

00:12:46.630 --> 00:13:00.210
F of x, vi, so this is the
output from with weights

00:13:00.210 --> 00:13:03.600
in x from sample number i.

00:13:03.600 --> 00:13:06.600
And if we're doing batch
processing-- that is,

00:13:06.600 --> 00:13:08.700
we're doing the
whole batch at once--

00:13:08.700 --> 00:13:10.530
then we compute that for all i.

00:13:10.530 --> 00:13:14.970
And that's the computation
that's ridiculously expensive,

00:13:14.970 --> 00:13:19.020
and you go instead to
stochastic gradient.

00:13:19.020 --> 00:13:22.260
And you just choose one
of those, or b of those,

00:13:22.260 --> 00:13:27.430
a small number b, like
32 or 128 of these F's.

00:13:27.430 --> 00:13:34.290
But full-scale gradient
descent chooses the weights

00:13:34.290 --> 00:13:36.840
x to minimize the loss.

00:13:36.840 --> 00:13:40.500
Now, so I haven't got
the loss here yet.

00:13:40.500 --> 00:13:49.860
This function, the loss would
be minus the true result

00:13:49.860 --> 00:13:52.110
from sample i.

00:13:52.110 --> 00:13:53.940
I haven't got a good
notation for that.

00:13:53.940 --> 00:13:56.400
I'm open to suggestions.

00:13:56.400 --> 00:13:58.390
So how do I want
to write the error?

00:14:01.200 --> 00:14:03.330
So that would be--

00:14:03.330 --> 00:14:06.630
if it was least squares, I
would maybe be squaring that.

00:14:09.480 --> 00:14:12.300
So it would be a sum
of squares of errors

00:14:12.300 --> 00:14:15.930
squared over all the samples.

00:14:15.930 --> 00:14:18.810
Or if I'm doing stochastic
gradient descent,

00:14:18.810 --> 00:14:19.700
I would minimize.

00:14:19.700 --> 00:14:21.870
I guess I'm minimizing this.

00:14:21.870 --> 00:14:25.740
But the question is, do
I use the whole function

00:14:25.740 --> 00:14:32.580
L at each iteration, or do
I just pick one, or only b,

00:14:32.580 --> 00:14:37.830
of the samples to look
at iteration number K?

00:14:37.830 --> 00:14:43.890
So this is the L of x then.

00:14:43.890 --> 00:14:47.250
I've added up over all the v's.

00:14:47.250 --> 00:14:51.510
So just to keep the
notation straight,

00:14:51.510 --> 00:14:54.180
I have this function
of x and v's.

00:14:54.180 --> 00:14:55.530
I find it's output.

00:14:58.380 --> 00:15:03.150
This is what the
neural net produces.

00:15:03.150 --> 00:15:06.300
It's supposed to be
close to the true.

00:15:06.300 --> 00:15:08.010
We don't want it to be exactly--

00:15:08.010 --> 00:15:10.620
we don't expect this
to be exactly 0,

00:15:10.620 --> 00:15:15.540
but it could be, because we have
lots of weight to achieve that.

00:15:18.270 --> 00:15:22.020
So anyway, that would
be the loss we minimize,

00:15:22.020 --> 00:15:24.780
and it'd be squared
for square loss.

00:15:24.780 --> 00:15:30.720
I guess I haven't really
spoken about loss functions.

00:15:30.720 --> 00:15:38.520
Let me just put those
here, and actually these

00:15:38.520 --> 00:15:42.070
are popular loss functions.

00:15:42.070 --> 00:15:49.890
One would be the one we know
best, square loss, and number

00:15:49.890 --> 00:15:53.730
two, I've never seen it
used quite this directly,

00:15:53.730 --> 00:16:03.330
would be the l1 loss,
maybe the sum of L1 norms.

00:16:03.330 --> 00:16:10.250
This is sum of these errors
squared in the L2 norm.

00:16:10.250 --> 00:16:16.080
The L1 loss could be the
sum over i of the L1 losses.

00:16:22.510 --> 00:16:28.150
Well, this comes into specific
other problems like Lasso

00:16:28.150 --> 00:16:32.350
and other important problems
you're minimizing an L1 norm

00:16:32.350 --> 00:16:36.550
but not in deep learning.

00:16:36.550 --> 00:16:39.415
Now, and three
would be Hinge loss.

00:16:43.960 --> 00:16:48.070
Probably some of you know
better than I the formula

00:16:48.070 --> 00:16:52.390
and the background
behind hinge losses.

00:16:52.390 --> 00:16:57.565
This is for the minus 1,
1 classification problems.

00:17:05.349 --> 00:17:08.960
That would be appropriate
for regression,

00:17:08.960 --> 00:17:11.079
so this would be
for a regression.

00:17:14.670 --> 00:17:18.359
And then finally, the most
important for neural nets,

00:17:18.359 --> 00:17:21.539
is cross-entropy loss.

00:17:28.099 --> 00:17:29.570
This is for neural nets.

00:17:34.800 --> 00:17:42.690
So this is really the most
used loss function in the setup

00:17:42.690 --> 00:17:46.590
that we are mostly
thinking of, and I'll

00:17:46.590 --> 00:17:51.510
try to say more about that
before the course ends.

00:17:51.510 --> 00:17:52.470
So is that--

00:17:52.470 --> 00:17:54.090
I don't know.

00:17:54.090 --> 00:18:00.450
For me, I hadn't got this
straight until rewriting

00:18:00.450 --> 00:18:05.400
that section, and it's
now in better form,

00:18:05.400 --> 00:18:07.735
but comments are welcome.

00:18:07.735 --> 00:18:08.235
OK.

00:18:11.360 --> 00:18:14.570
So that just completes
what I wanted to say,

00:18:14.570 --> 00:18:16.325
and you'll see the new section.

00:18:19.750 --> 00:18:25.610
Any comment on that before I go
to a different topic entirely?

00:18:25.610 --> 00:18:26.600
OK.

00:18:26.600 --> 00:18:31.190
Oh, any questions before
I go to this topic?

00:18:31.190 --> 00:18:32.600
Which I'll tell you what it is.

00:18:36.880 --> 00:18:47.230
It's a short section in the
book about distance matrices,

00:18:47.230 --> 00:18:49.000
and the question is.

00:18:58.540 --> 00:19:07.160
We have a bunch of points
in space, and what we know

00:19:07.160 --> 00:19:19.700
is we know the distances
between the points,

00:19:19.700 --> 00:19:23.050
and it's convenient to talk
about distances squared here.

00:19:28.050 --> 00:19:30.610
OK.

00:19:30.610 --> 00:19:32.740
And how would we know
of these distances?

00:19:32.740 --> 00:19:39.820
Maybe by radar or
any measurement.

00:19:39.820 --> 00:19:47.110
They might be sensors,
which we've placed around,

00:19:47.110 --> 00:19:51.100
and we can measure the
distances between them.

00:19:51.100 --> 00:19:54.190
And the question is,
what's their position?

00:19:54.190 --> 00:19:59.450
So that's the question.

00:19:59.450 --> 00:20:01.660
So let me talk a little
bit about this question

00:20:01.660 --> 00:20:03.310
and then pause.

00:20:03.310 --> 00:20:14.200
Find positions in, well,
in space, but I don't know.

00:20:14.200 --> 00:20:16.000
We don't know
ahead of time maybe

00:20:16.000 --> 00:20:20.500
whether the space is ordinary
3D space, or whether these

00:20:20.500 --> 00:20:24.160
are sensors in a plane,
or whether we have

00:20:24.160 --> 00:20:25.610
to go to higher dimensions.

00:20:25.610 --> 00:20:30.130
I'll just put d, and
also, I'll just say then,

00:20:30.130 --> 00:20:31.820
we're also finding d.

00:20:37.570 --> 00:20:39.160
And what are these positions?

00:20:39.160 --> 00:20:44.410
These are positions x,
xi, so that the distance

00:20:44.410 --> 00:20:53.920
between xi minus xj
squared is the given dij.

00:20:56.470 --> 00:20:59.110
So we're given
distances between them,

00:20:59.110 --> 00:21:02.110
and we want to find
their positions.

00:21:02.110 --> 00:21:05.800
So we know distances, and
we want to find positions.

00:21:05.800 --> 00:21:07.510
That's the question.

00:21:07.510 --> 00:21:11.650
It's just a neat math
question that is solved,

00:21:11.650 --> 00:21:13.360
and you'll see a
complete solution.

00:21:16.120 --> 00:21:22.680
And it has lots of applications,
and it's just a nice question.

00:21:22.680 --> 00:21:24.750
So it occupies a
section of the book,

00:21:24.750 --> 00:21:28.100
but that section is
only two pages long.

00:21:28.100 --> 00:21:32.670
It's just a straightforward
solution to that question.

00:21:32.670 --> 00:21:36.900
Given the distances,
find the positions.

00:21:36.900 --> 00:21:38.965
Given the distances,
find the excess.

00:21:43.610 --> 00:21:45.500
OK.

00:21:45.500 --> 00:21:47.120
So I'm going to
speak about that.

00:21:50.350 --> 00:21:54.500
I had a suggestion, a
good suggestion, by email.

00:21:54.500 --> 00:21:58.340
Well, questions about
the projects coming in?

00:21:58.340 --> 00:22:01.400
Projects are beginning
to come in, and at least

00:22:01.400 --> 00:22:04.930
at the beginning--

00:22:04.930 --> 00:22:07.460
well, in all cases,
beginning and end,

00:22:07.460 --> 00:22:09.260
I'll read them carefully.

00:22:09.260 --> 00:22:12.290
And as long as I
can, I'll send back

00:22:12.290 --> 00:22:19.190
suggestions for a final
rewrite, and as I said,

00:22:19.190 --> 00:22:20.930
a print out is great.

00:22:20.930 --> 00:22:24.320
You could leave it in the
envelope outside my office,

00:22:24.320 --> 00:22:30.330
but of course, online is
what everybody's doing.

00:22:30.330 --> 00:22:32.240
So those are just
beginning to come in,

00:22:32.240 --> 00:22:36.230
and if we can get them
in by a week from today,

00:22:36.230 --> 00:22:38.840
I'm really, really happy.

00:22:38.840 --> 00:22:41.940
Yeah, and just feel
free to email me.

00:22:41.940 --> 00:22:49.450
I would email me about projects,
not Jonathan and not anonymous

00:22:49.450 --> 00:22:50.660
Stellar.

00:22:50.660 --> 00:22:55.280
I think you'd probably do better
just to ask me the question.

00:22:55.280 --> 00:23:00.410
That's fine, and I'll try
to answer in a useful way.

00:23:00.410 --> 00:23:04.250
Yeah, and I'm always
open to questions.

00:23:04.250 --> 00:23:11.240
So you could email me like how
long should this project be?

00:23:11.240 --> 00:23:14.030
My tutor in Oxford
said something like--

00:23:14.030 --> 00:23:19.310
when you were writing essays.

00:23:19.310 --> 00:23:22.460
That's the Oxford system
is to write an essay--

00:23:22.460 --> 00:23:25.070
and he said, just
start where it starts,

00:23:25.070 --> 00:23:26.870
and end when it finishes.

00:23:26.870 --> 00:23:33.230
So that's the idea, certainly
not enormously long.

00:23:33.230 --> 00:23:39.740
And then a question was raised--
and I can ask you if you are

00:23:39.740 --> 00:23:41.300
interested in that--

00:23:41.300 --> 00:23:45.490
the question was, what
courses after this one

00:23:45.490 --> 00:23:51.200
are natural to
take to go forward?

00:23:51.200 --> 00:23:55.730
And I don't know how many of you
are thinking to take, have time

00:23:55.730 --> 00:24:02.510
to take, other MIT courses in
this area of deep learning,

00:24:02.510 --> 00:24:08.270
machine learning, optimization,
all the topics we've had here.

00:24:08.270 --> 00:24:12.320
Anybody expecting to take more
courses, just stick up a hand.

00:24:12.320 --> 00:24:15.540
Yeah, and you already
know like what MIT offers?

00:24:18.700 --> 00:24:21.020
So that was the question
that came to me,

00:24:21.020 --> 00:24:25.710
what does MIT offer
in this direction?

00:24:25.710 --> 00:24:29.300
And I haven't looked up to see
the number of Professor Sra's

00:24:29.300 --> 00:24:32.780
course, S-R-A, in course 6.

00:24:32.780 --> 00:24:37.850
It's 6 point high number,
and after his good lecture,

00:24:37.850 --> 00:24:42.510
I think that's got
to be worthwhile.

00:24:42.510 --> 00:24:44.390
So I looked in course 6.

00:24:44.390 --> 00:24:50.450
I didn't find really
an institute-wide list.

00:24:50.450 --> 00:24:52.970
Maybe course 6 feels that
they are the Institute,

00:24:52.970 --> 00:24:55.550
but there are other
courses around.

00:25:00.570 --> 00:25:04.790
But I found in the
operations research site,

00:25:04.790 --> 00:25:11.390
ORC, the Operations Research
Center, let me just put there.

00:25:11.390 --> 00:25:14.450
This is just in
case you would like

00:25:14.450 --> 00:25:17.545
to think about any
of these things.

00:25:24.280 --> 00:25:29.740
As I write that, so I heard
the lecture by Tim Berners-Lee.

00:25:29.740 --> 00:25:33.490
Did others hear that
a week or so ago?

00:25:33.490 --> 00:25:36.710
He created the web.

00:25:36.710 --> 00:25:39.820
So that's pretty amazing--

00:25:39.820 --> 00:25:45.580
it wasn't Al Gore, after all,
and do you know his name?

00:25:45.580 --> 00:25:50.500
Well, he's now Sir
Tim Berners-Lee.

00:25:53.960 --> 00:25:57.760
So that double name makes you
suspect that he's from England,

00:25:57.760 --> 00:26:00.520
and he is.

00:26:00.520 --> 00:26:03.910
So anyway, I was
going to say, I hold

00:26:03.910 --> 00:26:07.840
him responsible for
these excessive letters

00:26:07.840 --> 00:26:14.260
in the address, in the URL.

00:26:14.260 --> 00:26:19.090
I mean, he's made us
all say W-W-W for years.

00:26:19.090 --> 00:26:23.650
Find some other way to say
it, but it's not easy to say,

00:26:23.650 --> 00:26:24.220
I think.

00:26:24.220 --> 00:26:25.690
OK, whatever.

00:26:25.690 --> 00:26:36.070
This is the OR Center
at MIT, and then it's

00:26:36.070 --> 00:26:42.750
academics or something,
and then it's something

00:26:42.750 --> 00:26:45.330
like course offerings.

00:26:45.330 --> 00:26:46.650
That's approximately right.

00:26:52.080 --> 00:26:55.920
And since they do
applied optimization,

00:26:55.920 --> 00:26:59.940
under the heading of data
analytics or statistics,

00:26:59.940 --> 00:27:05.790
there's optimization, there's
OR, Operations Research,

00:27:05.790 --> 00:27:13.020
other lists but a good list of
courses from many departments,

00:27:13.020 --> 00:27:19.200
especially course 6.

00:27:19.200 --> 00:27:22.650
Course 15 which is where
the operation and research

00:27:22.650 --> 00:27:26.190
center is, course 18,
and there are others

00:27:26.190 --> 00:27:28.780
in course 2 and elsewhere.

00:27:28.780 --> 00:27:29.280
Yeah.

00:27:34.390 --> 00:27:37.930
Would somebody like to say
what course you have in mind

00:27:37.930 --> 00:27:40.250
to take next, after this one?

00:27:40.250 --> 00:27:47.260
If you looked ahead to next
year, any suggestions of what

00:27:47.260 --> 00:27:48.490
looks like a good course?

00:27:51.340 --> 00:27:56.290
I sat in once on 6.036,
the really basic course,

00:27:56.290 --> 00:28:00.640
and you would want to go higher.

00:28:00.640 --> 00:28:01.140
OK.

00:28:04.180 --> 00:28:06.970
Maybe this is just
to say, I'd be

00:28:06.970 --> 00:28:10.810
interested to know what you do
next, what your experience is,

00:28:10.810 --> 00:28:14.270
or I'd be happy to give advice.

00:28:14.270 --> 00:28:18.250
But maybe my general
advice is that that's

00:28:18.250 --> 00:28:20.405
a useful list of courses.

00:28:23.440 --> 00:28:25.820
OK?

00:28:25.820 --> 00:28:29.000
Back to distance matrices.

00:28:29.000 --> 00:28:32.830
OK, so here's the problem.

00:28:32.830 --> 00:28:34.040
Yeah.

00:28:34.040 --> 00:28:36.620
OK, I'll probably
have to erase that,

00:28:36.620 --> 00:28:40.080
but I'll leave it for a minute.

00:28:40.080 --> 00:28:41.550
OK.

00:28:41.550 --> 00:28:46.550
So we know these distances,
and we want to find the x's, so

00:28:46.550 --> 00:28:49.730
let's call this dij maybe.

00:28:53.240 --> 00:28:58.010
So we have a D matrix, and we
want to find a position matrix,

00:28:58.010 --> 00:29:00.060
let me just see what notation.

00:29:00.060 --> 00:29:10.460
So this is section 3.9, no 4.9,
previously 3.9, but chapters 3

00:29:10.460 --> 00:29:13.130
and 4 got switched.

00:29:13.130 --> 00:29:19.250
Maybe actually, yeah, I
think it's 8 or 9 or 10,

00:29:19.250 --> 00:29:24.640
other topics are trying
to find their way in.

00:29:24.640 --> 00:29:25.220
OK.

00:29:25.220 --> 00:29:29.640
So that's the
reference on the web,

00:29:29.640 --> 00:29:32.480
and I'll get these
sections onto Stellar.

00:29:32.480 --> 00:29:33.500
OK.

00:29:33.500 --> 00:29:37.760
So the question is, can
we recover the positions

00:29:37.760 --> 00:29:40.480
from the distances?

00:29:40.480 --> 00:29:41.980
In fact, there's
also a question,

00:29:41.980 --> 00:29:48.400
are there always positions
from given distances?

00:29:48.400 --> 00:29:51.980
And I mentioned
several applications.

00:29:51.980 --> 00:29:56.810
I've already spoken about
wireless sensor networks, where

00:29:56.810 --> 00:30:00.860
you can measure travel
times between them,

00:30:00.860 --> 00:30:02.120
between the sensors.

00:30:02.120 --> 00:30:05.210
And then that gives
you the distances,

00:30:05.210 --> 00:30:09.560
and then you use this
neat little bit of math

00:30:09.560 --> 00:30:11.750
to find the positions.

00:30:11.750 --> 00:30:17.270
Well, of course, you can't
find the positions uniquely.

00:30:17.270 --> 00:30:23.780
Clearly, you could any rigid
motion of all the positions.

00:30:23.780 --> 00:30:27.920
If I have a set
of positions, what

00:30:27.920 --> 00:30:30.410
am I going to call that, x?

00:30:30.410 --> 00:30:35.375
So I'll write here, and
so I'm given the D matrix.

00:30:40.440 --> 00:30:49.450
That's distances, and the job
is to find the X matrix which

00:30:49.450 --> 00:30:50.770
gives the positions.

00:30:54.150 --> 00:30:57.780
And what I'm just going
to say, and you already

00:30:57.780 --> 00:31:02.580
saw it your mind-- that if
I have a set of positions,

00:31:02.580 --> 00:31:05.880
I could do a translation.

00:31:05.880 --> 00:31:08.130
The distances
wouldn't change, or I

00:31:08.130 --> 00:31:13.980
could do a rigid motion,
a rigid rotation.

00:31:13.980 --> 00:31:21.840
So positions are not unique,
but I can come closer by saying,

00:31:21.840 --> 00:31:26.640
put the centroid at the
origin, or something like that.

00:31:26.640 --> 00:31:29.680
That will take out the
translations at least.

00:31:29.680 --> 00:31:30.180
OK.

00:31:30.180 --> 00:31:31.500
So find the X matrix.

00:31:31.500 --> 00:31:32.550
That's the job.

00:31:32.550 --> 00:31:36.390
OK, and I was going to--
before I started on that--

00:31:36.390 --> 00:31:39.910
the shapes of molecules
is another application.

00:31:39.910 --> 00:31:45.260
Nuclear magnetic resonance
gives distances, gives d,

00:31:45.260 --> 00:31:49.280
and then we find
the positions x.

00:31:49.280 --> 00:31:51.770
And of course, there's
a noise in there,

00:31:51.770 --> 00:31:54.350
and sometimes missing entries.

00:31:54.350 --> 00:31:58.100
And machine learning
could be just described

00:31:58.100 --> 00:32:01.400
also as you're given
a whole lot of points

00:32:01.400 --> 00:32:04.760
in space, feature vectors
in a high-dimensional space.

00:32:04.760 --> 00:32:06.740
Actually, this is a big deal.

00:32:06.740 --> 00:32:09.590
You're given a
whole lot of points

00:32:09.590 --> 00:32:17.210
with in high-dimensional
space, and those are related.

00:32:17.210 --> 00:32:19.310
They sort of come
together naturally,

00:32:19.310 --> 00:32:25.620
so they tend to fit on a surface
in high-dimensional space,

00:32:25.620 --> 00:32:28.920
a low-dimensional surface
in high-dimensional space.

00:32:28.920 --> 00:32:31.290
And really, a lot
of mathematics is

00:32:31.290 --> 00:32:36.720
devoted to finding that
low-dimensional, that subspace,

00:32:36.720 --> 00:32:38.430
but it could be curved.

00:32:38.430 --> 00:32:42.630
So subspace is not
the correct word.

00:32:42.630 --> 00:32:45.600
Really, manifold,
curved manifold

00:32:45.600 --> 00:32:49.230
is what a geometer would say.

00:32:49.230 --> 00:32:51.900
That is close to all the--

00:32:51.900 --> 00:32:55.110
it's smooth and close
to all the points,

00:32:55.110 --> 00:32:58.290
and you could linearize it.

00:32:58.290 --> 00:33:01.830
You could flatten
it out, and then you

00:33:01.830 --> 00:33:04.050
have a much reduced problem.

00:33:04.050 --> 00:33:07.320
The dimension is reduced
from the original dimension

00:33:07.320 --> 00:33:13.050
of where the points
lie with a lot of data

00:33:13.050 --> 00:33:17.498
to the true dimension of the
problem which, of course,

00:33:17.498 --> 00:33:19.290
sets of points were
all on a straight line.

00:33:19.290 --> 00:33:22.630
The true dimension of
the problem would be 1.

00:33:22.630 --> 00:33:25.065
So we have to discover this.

00:33:29.230 --> 00:33:33.520
We also have to find
that dimension d.

00:33:33.520 --> 00:33:34.650
OK, so how do we do it?

00:33:38.160 --> 00:33:40.420
So it's a classical problem.

00:33:40.420 --> 00:33:42.150
It just has a neat answer.

00:33:42.150 --> 00:33:42.650
OK.

00:33:46.870 --> 00:33:53.170
All right, so let's
recognize the connection

00:33:53.170 --> 00:33:55.750
between distances and positions.

00:33:55.750 --> 00:34:04.090
So dij is the square
distance between them,

00:34:04.090 --> 00:34:19.649
so that is xi dot xi minus xi
to xj minus xj, xi plus xj, xj.

00:34:22.730 --> 00:34:23.230
OK.

00:34:26.210 --> 00:34:28.667
Is that right?

00:34:28.667 --> 00:34:30.635
Yes.

00:34:30.635 --> 00:34:33.600
OK.

00:34:33.600 --> 00:34:37.219
So those are the
dij's in a matrix,

00:34:37.219 --> 00:34:45.020
and these are entries
in the matrix D. OK.

00:34:48.070 --> 00:34:57.560
Well, these entries
depend only on i.

00:34:57.560 --> 00:34:59.820
They're the same for every j.

00:34:59.820 --> 00:35:04.200
So this is going to be-- this
will this part will produce--

00:35:04.200 --> 00:35:05.430
I'll rank one matrix.

00:35:09.690 --> 00:35:13.020
Because things depend not
only on the row but not on j,

00:35:13.020 --> 00:35:20.565
the column number,
so columns repeated.

00:35:27.970 --> 00:35:34.420
Yeah, and this
produces similarly

00:35:34.420 --> 00:35:39.550
something that depends only on
j, only on the column number.

00:35:39.550 --> 00:35:43.350
So the rows are all the
same, so this is also

00:35:43.350 --> 00:35:55.730
a rank one matrix with all
repeated, all the same rows.

00:35:55.730 --> 00:36:01.180
Because if I change i,
nothing changes in a product.

00:36:01.180 --> 00:36:06.630
So really, these
are the terms that

00:36:06.630 --> 00:36:13.900
produce most of the matrix, the
significant part of the matrix.

00:36:13.900 --> 00:36:14.890
OK.

00:36:14.890 --> 00:36:20.440
So what do we do with those?

00:36:26.710 --> 00:36:30.640
So let's see, did I give
a name for the matrix

00:36:30.640 --> 00:36:31.690
that I'm looking for?

00:36:31.690 --> 00:36:49.930
I think in the notes I call
it X. So I'm given D, find X.

00:36:49.930 --> 00:36:52.720
And what I'll actually find--

00:36:52.720 --> 00:36:54.620
you can see it coming here--

00:36:54.620 --> 00:37:06.490
is actually find X transpose X.
Because what I'm given is dot

00:37:06.490 --> 00:37:08.740
products of X's.

00:37:11.530 --> 00:37:17.430
So I would like to
discover out of all this

00:37:17.430 --> 00:37:20.870
what xi dotted with xj is.

00:37:20.870 --> 00:37:23.380
That'll be the
correct dot product.

00:37:23.380 --> 00:37:29.470
Let's call this matrix G
for the dot product matrix,

00:37:29.470 --> 00:37:40.780
and then find X from G.

00:37:40.780 --> 00:37:44.960
So this is a nice argument.

00:37:44.960 --> 00:37:50.230
So what this tells me is some
information about dot products.

00:37:50.230 --> 00:37:54.280
So this is telling me something
about the G matrix, the X

00:37:54.280 --> 00:37:56.110
transpose X matrix.

00:37:56.110 --> 00:38:01.390
And then once I know G, then
it's a separate step to find X.

00:38:01.390 --> 00:38:06.640
And of course, this is the
point at which X is not unique.

00:38:06.640 --> 00:38:11.800
If I put it in a rotation
into X, then that rotation q,

00:38:11.800 --> 00:38:15.850
I'll see a q transpose
q, and it'll disappear.

00:38:15.850 --> 00:38:19.720
So I'm free to rotate
the X's, because that

00:38:19.720 --> 00:38:22.420
doesn't change the dot product.

00:38:22.420 --> 00:38:25.990
So it's G that I want to know,
and this tells me something

00:38:25.990 --> 00:38:31.252
about G, and this tells
me something about G.

00:38:31.252 --> 00:38:36.480
And so does that, but
that's what I have to see.

00:38:36.480 --> 00:38:40.170
So what do those tell me?

00:38:40.170 --> 00:38:40.950
Let's see.

00:38:40.950 --> 00:38:43.560
Let me write down
what I have here.

00:38:50.060 --> 00:39:00.940
So let's say a diagonal matrix
with Dii as the inner product

00:39:00.940 --> 00:39:10.120
xi with xi that we're getting
partial information from here.

00:39:10.120 --> 00:39:11.530
So is that OK?

00:39:11.530 --> 00:39:15.630
I'm introducing that
notation, because this is now

00:39:15.630 --> 00:39:20.560
going to tell me
that my D matrix is--

00:39:20.560 --> 00:39:22.390
so what is that?

00:39:22.390 --> 00:39:25.660
So this is the diagonal matrix.

00:39:25.660 --> 00:39:27.704
Maybe it's just a
vector, I should say.

00:39:30.220 --> 00:39:30.720
Yeah.

00:39:39.160 --> 00:39:45.220
Yeah, so can I write down the
equation that is fundamental

00:39:45.220 --> 00:39:49.750
here, and then we'll
figure out what it means.

00:39:49.750 --> 00:39:58.600
So it's an equation for G,
for the dot product matrix.

00:39:58.600 --> 00:40:01.120
OK, let me make space
for that equation.

00:40:05.410 --> 00:40:07.660
I believe that we can
get the dot product

00:40:07.660 --> 00:40:10.054
matrix which I'm calling G as--

00:40:14.380 --> 00:40:20.680
according to this, it's
minus 1/2 of the D matrix

00:40:20.680 --> 00:40:33.070
plus 1/2 of the 1's times
the d, the diagonal d.

00:40:33.070 --> 00:40:43.410
And it's plus 1/2 of
the d times the 1's.

00:40:48.460 --> 00:40:52.420
That's a matrix
with constant rows.

00:40:55.390 --> 00:40:59.860
This here is coming from there.

00:40:59.860 --> 00:41:11.020
This is a matrix with always
the same columns, or let me see.

00:41:11.020 --> 00:41:13.990
No, I haven't got
those right yet.

00:41:13.990 --> 00:41:21.100
I mean, I want these to be rank
1 matrices, so it's this one.

00:41:21.100 --> 00:41:22.160
Let me fix that.

00:41:32.780 --> 00:41:40.090
1, 1, 1, 1 times d transpose,
so it's column times row,

00:41:40.090 --> 00:41:49.720
and this one is also column
times row with the d here.

00:41:52.310 --> 00:41:56.150
OK, now let me look
at that properly.

00:41:56.150 --> 00:42:03.350
So every row in this guy is
a multiple of 1, 1, 1, 1.

00:42:03.350 --> 00:42:04.580
So what is that telling me?

00:42:04.580 --> 00:42:09.920
That all columns are
the same, this part

00:42:09.920 --> 00:42:15.860
is reflecting these ones,
where the columns are repeated.

00:42:15.860 --> 00:42:19.280
This one is reflecting this,
where the rows are repeated.

00:42:19.280 --> 00:42:22.100
The d is just the
set of d numbers.

00:42:22.100 --> 00:42:34.455
Let's call that di, and this
is dj, and here's the D matrix.

00:42:41.090 --> 00:42:46.760
So part of the D matrix
is this bit and this bit,

00:42:46.760 --> 00:42:49.670
each giving a rank 1.

00:42:49.670 --> 00:42:52.970
Now, it's this part that
I have to understand,

00:42:52.970 --> 00:42:56.960
so while you're
checking on that,

00:42:56.960 --> 00:43:02.090
let me look again at this.

00:43:07.590 --> 00:43:08.090
Yeah.

00:43:13.590 --> 00:43:17.170
Let's just see where
we are if this is true.

00:43:17.170 --> 00:43:20.590
If this is true, I'm
given the D matrix,

00:43:20.590 --> 00:43:28.810
and then these dot
products I can find.

00:43:28.810 --> 00:43:30.710
So I can find these,
so in other words,

00:43:30.710 --> 00:43:34.164
this is the key
equation that tells me

00:43:34.164 --> 00:43:39.130
D. That's the key
equation, and it's

00:43:39.130 --> 00:43:43.300
going to come just from
that simple identity,

00:43:43.300 --> 00:43:44.930
just from checking each term.

00:43:44.930 --> 00:43:47.530
This term we identified,
that last term we

00:43:47.530 --> 00:43:54.860
identified, and now this term
is D. Well, of, course it's D.

00:43:54.860 --> 00:43:58.750
So I have two of
those, and I'm going

00:43:58.750 --> 00:44:03.460
to take half of that
to get D, I think.

00:44:10.150 --> 00:44:12.970
Yeah, and we'll look.

00:44:15.900 --> 00:44:16.400
Yeah.

00:44:22.340 --> 00:44:29.300
So I guess I'm not seeing right
away why this 1/2 is in here,

00:44:29.300 --> 00:44:32.960
but I think I had it right,
and there's a reason.

00:44:32.960 --> 00:44:37.010
You see that this matrix
this, X transpose X matrix,

00:44:37.010 --> 00:44:41.630
is coming from these rank 1
pieces and these pieces which

00:44:41.630 --> 00:44:47.430
are the cross product.

00:44:47.430 --> 00:44:48.570
Oh, I see.

00:44:48.570 --> 00:44:51.060
I see.

00:44:51.060 --> 00:44:53.060
What that equation
is really saying

00:44:53.060 --> 00:44:56.730
is that the D matrix is this--

00:44:59.340 --> 00:45:02.220
if I just read that along
and translate it and put it

00:45:02.220 --> 00:45:14.070
in matrix language-- is this 1,
1, 1, 1, d1 to d4, let's say,

00:45:14.070 --> 00:45:17.310
transpose is this rank 1 matrix.

00:45:17.310 --> 00:45:24.900
And the other one is
the d's times the 1, 1

00:45:24.900 --> 00:45:27.060
which is a transpose of that.

00:45:27.060 --> 00:45:30.650
And then the other
one was a minus 2

00:45:30.650 --> 00:45:32.590
of the cross product matrices.

00:45:32.590 --> 00:45:33.370
I see.

00:45:33.370 --> 00:45:34.530
Yeah.

00:45:34.530 --> 00:45:38.520
So when I write that
equation in matrix language,

00:45:38.520 --> 00:45:39.280
I just get that.

00:45:41.810 --> 00:45:44.611
And now, when I solve for X--

00:45:44.611 --> 00:45:53.200
oh, minus 2 X transpose X. Yeah.

00:45:53.200 --> 00:45:57.270
Sorry, cross products, the X's.

00:45:57.270 --> 00:45:59.910
So I had one set
of cross products,

00:45:59.910 --> 00:46:03.090
and then this is the same
as this, so I have minus 2

00:46:03.090 --> 00:46:03.790
of them.

00:46:03.790 --> 00:46:06.420
So now, I'm just rewriting that.

00:46:06.420 --> 00:46:08.430
When I rewrite that
equation, I have that.

00:46:08.430 --> 00:46:09.960
Do you see that?

00:46:09.960 --> 00:46:12.360
I put that on this side.

00:46:12.360 --> 00:46:15.105
I put the d over
here as a minus d.

00:46:15.105 --> 00:46:20.250
I divide by 2, and then
that's the formula.

00:46:20.250 --> 00:46:26.240
So ultimately, this
simple identity

00:46:26.240 --> 00:46:30.560
just looked at-- because
these pieces were so simple,

00:46:30.560 --> 00:46:33.860
just rank one pieces,
and these pieces

00:46:33.860 --> 00:46:39.500
were exactly what we want, the
X transpose X pieces, the G.

00:46:39.500 --> 00:46:45.110
That equation told us
the D. All this is known.

00:46:48.350 --> 00:46:53.260
Well, so what's known
is D and this and this.

00:46:53.260 --> 00:47:01.310
So now, we have the equation for
X transpose X is minus 1/2 of D

00:47:01.310 --> 00:47:05.330
minus these rank 1's.

00:47:11.530 --> 00:47:16.250
Sorry to make it look messy.

00:47:16.250 --> 00:47:19.060
I remember Raj Rao talking
about it last spring,

00:47:19.060 --> 00:47:25.330
also the algebra got flustered.

00:47:25.330 --> 00:47:26.710
So we get it.

00:47:26.710 --> 00:47:31.450
So we know X transpose
X, that matrix.

00:47:31.450 --> 00:47:35.320
Now, can we just do four
minutes of linear algebra

00:47:35.320 --> 00:47:38.620
at the end today?

00:47:41.900 --> 00:47:51.170
Given X transpose X,
find X. This is n by n.

00:47:56.590 --> 00:47:58.880
How would you do that?

00:47:58.880 --> 00:48:00.680
Could you do it?

00:48:00.680 --> 00:48:03.190
Would there be just one X?

00:48:03.190 --> 00:48:05.620
No.

00:48:05.620 --> 00:48:11.380
So if you had one X,
multiply that by a rotation,

00:48:11.380 --> 00:48:14.530
by an orthogonal matrix,
you'd have another one.

00:48:14.530 --> 00:48:19.690
So this is finding X up to
an orthogonal transformation,

00:48:19.690 --> 00:48:22.150
but how would you
actually do that?

00:48:22.150 --> 00:48:27.025
What do we know about this
matrix, X transpose X?

00:48:27.025 --> 00:48:31.030
It's symmetric, clearly,
and what we especially know

00:48:31.030 --> 00:48:32.470
is that it is also?

00:48:32.470 --> 00:48:33.490
AUDIENCE: Positive.

00:48:33.490 --> 00:48:36.970
GILBERT STRANG: Positive
or semidefinite,

00:48:36.970 --> 00:48:38.890
so this is semidefinite.

00:48:42.180 --> 00:48:44.260
So I'm given a
semidefinite matrix,

00:48:44.260 --> 00:48:48.640
and I want to find a
square root, you could say.

00:48:48.640 --> 00:48:50.890
That matrix is
the X transpose X,

00:48:50.890 --> 00:48:54.160
and I want to find X. I
think there are two leading

00:48:54.160 --> 00:48:56.920
candidates.

00:48:56.920 --> 00:49:00.730
There are many candidates,
because if you find one,

00:49:00.730 --> 00:49:09.730
then any QX is OK.

00:49:09.730 --> 00:49:13.910
Because if I put a Q transpose
Q in there, it's the identity.

00:49:13.910 --> 00:49:14.650
OK.

00:49:14.650 --> 00:49:24.580
So one way is to use
eigenvalues of X transpose X,

00:49:24.580 --> 00:49:31.320
and the other way would be to
use elimination on X transpose

00:49:31.320 --> 00:49:37.280
X. So I'll put use.

00:49:37.280 --> 00:49:40.150
So if I use
eigenvalues of X, if I

00:49:40.150 --> 00:49:43.500
find the eigenvalues
of X transpose X,

00:49:43.500 --> 00:49:45.940
then I'm writing this a--

00:49:45.940 --> 00:49:47.870
it's a symmetric,
positive definition--

00:49:47.870 --> 00:49:51.410
I'm writing it as Q
lambda Q transpose.

00:49:51.410 --> 00:49:51.910
Right?

00:49:51.910 --> 00:49:54.820
That's the fundamental
most important theorem

00:49:54.820 --> 00:49:57.130
in linear algebra,
you could say.

00:49:57.130 --> 00:50:00.940
That a symmetric, positive,
semidefinite matrix

00:50:00.940 --> 00:50:06.340
has greater eigenvalues,
greater or equal to 0,

00:50:06.340 --> 00:50:09.650
and eigenvectors
that are orthogonal.

00:50:09.650 --> 00:50:12.580
So now, if I know
that, what's a good X?

00:50:12.580 --> 00:50:16.820
Then, take X to be what?

00:50:20.300 --> 00:50:24.920
So I've got the eigenvalues and
eigenvectors of X transpose X,

00:50:24.920 --> 00:50:27.590
and I'm looking for
an X that will work.

00:50:27.590 --> 00:50:33.800
And one idea is just to
take the same eigenvectors,

00:50:33.800 --> 00:50:37.160
and take the square
roots of the eigenvalues.

00:50:40.660 --> 00:50:41.930
That's symmetric now.

00:50:41.930 --> 00:50:51.860
This is equal to X
transpose, and that's

00:50:51.860 --> 00:50:57.150
a square root symbol, or a
lambda to the 1/2, I could say.

00:50:57.150 --> 00:51:00.260
So when I multiply that--

00:51:00.260 --> 00:51:03.500
X transpose X is
just X squared here.

00:51:03.500 --> 00:51:07.610
When I square it,
the Q transpose Q

00:51:07.610 --> 00:51:10.700
multiplies itself to
give the identity.

00:51:10.700 --> 00:51:14.740
The square root of lambda times
the square root of lambda,

00:51:14.740 --> 00:51:17.750
those are diagonal
matrices that give lambda,

00:51:17.750 --> 00:51:20.010
and I get the right answer.

00:51:20.010 --> 00:51:23.510
So one way is, in
a few words, take

00:51:23.510 --> 00:51:25.550
the square roots
of the eigenvalues

00:51:25.550 --> 00:51:27.560
and keep the eigenvectors.

00:51:27.560 --> 00:51:30.440
So that's the
eigenvalue construction.

00:51:30.440 --> 00:51:34.220
So that's producing an
X that is symmetric,

00:51:34.220 --> 00:51:36.650
positive, semidefinite.

00:51:36.650 --> 00:51:38.930
That might be what you want.

00:51:38.930 --> 00:51:42.260
It's a little work, because
your computing eigenvalues

00:51:42.260 --> 00:51:46.550
and eigenvectors to do
it, but that's one choice.

00:51:46.550 --> 00:51:51.200
Now, I believe that elimination
would give us another choice.

00:51:51.200 --> 00:51:55.100
So elimination produces
what factorization of this?

00:51:55.100 --> 00:52:00.020
This is still our symmetric,
positive, definite matrix.

00:52:00.020 --> 00:52:03.020
If you do elimination
on that, you usually

00:52:03.020 --> 00:52:12.980
expect L, a lower triangular,
times D, the pivots, times U,

00:52:12.980 --> 00:52:14.060
the upper triangle.

00:52:14.060 --> 00:52:18.620
That's the usual result
of elimination, LDU.

00:52:18.620 --> 00:52:21.260
I'm factoring out the
pivots, so they're

00:52:21.260 --> 00:52:25.460
1's on the diagonals
of L and U. But now,

00:52:25.460 --> 00:52:29.980
if it's a symmetric
matrix, what's up?

00:52:29.980 --> 00:52:32.830
We zipped by
elimination, regarding

00:52:32.830 --> 00:52:40.120
that as a 18.06 trivial
bit of linear algebra,

00:52:40.120 --> 00:52:42.280
but of course, it's
highly important.

00:52:42.280 --> 00:52:47.290
So what's the situation here
when the matrix is actually

00:52:47.290 --> 00:52:48.010
symmetric?

00:52:51.020 --> 00:52:53.810
So I want something
to look symmetric.

00:52:53.810 --> 00:52:56.130
How do I make that
look symmetric?

00:52:56.130 --> 00:52:59.060
The U gets replaced
by L transpose.

00:53:03.614 --> 00:53:05.920
If I'm working on a
positive definite--

00:53:05.920 --> 00:53:08.090
say positive definite matrix--

00:53:08.090 --> 00:53:15.020
then I get positive pivots,
and L and lower triangular

00:53:15.020 --> 00:53:17.820
and upper triangular are
transposes of each other.

00:53:17.820 --> 00:53:20.960
So now, what is then the X?

00:53:23.840 --> 00:53:25.610
It's just like that.

00:53:25.610 --> 00:53:31.990
I'll use L square root
of the D L transpose.

00:53:31.990 --> 00:53:34.800
Is that right?

00:53:34.800 --> 00:53:37.260
Oh, wait a minute.

00:53:37.260 --> 00:53:38.920
What's up?

00:53:38.920 --> 00:53:41.620
No, that's not going
to work, because I

00:53:41.620 --> 00:53:45.310
don't have L transpose L.
Where I had Q transpose Q,

00:53:45.310 --> 00:53:46.490
it was good.

00:53:46.490 --> 00:53:48.250
No, sorry.

00:53:48.250 --> 00:53:52.400
Let's get that totally erased.

00:53:52.400 --> 00:53:57.560
The X part should just be the
square root of DL transpose.

00:54:00.230 --> 00:54:02.840
The X is now a
triangular matrix,

00:54:02.840 --> 00:54:07.550
the square root of the pivots,
and the L transpose part.

00:54:07.550 --> 00:54:10.220
And now, when I
do X transpose X,

00:54:10.220 --> 00:54:13.790
then you see X transpose
X coming correctly.

00:54:13.790 --> 00:54:18.950
X transpose will be L transpose.

00:54:18.950 --> 00:54:21.440
Transpose will give me
the L. Square root of D

00:54:21.440 --> 00:54:24.080
will be square root of
D. We'll give the D,

00:54:24.080 --> 00:54:26.030
and then the L
transpose is right.

00:54:26.030 --> 00:54:27.740
So this is called the--

00:54:30.290 --> 00:54:31.880
do I try to write it here?

00:54:31.880 --> 00:54:35.073
This is my last word for today--

00:54:35.073 --> 00:54:35.615
the Cholesky.

00:54:39.470 --> 00:54:42.140
This is the Cholesky
Factorization,

00:54:42.140 --> 00:54:47.430
named after a French guy,
a French soldier actually.

00:54:47.430 --> 00:54:54.470
So LDL transpose is
Cholesky, and that's

00:54:54.470 --> 00:54:56.840
easy to compute, much
faster to compute

00:54:56.840 --> 00:54:59.120
than the eigenvalue square root.

00:54:59.120 --> 00:55:01.460
But this square
root is triangular.

00:55:01.460 --> 00:55:03.710
This square root is symmetric.

00:55:03.710 --> 00:55:08.810
Those are the two pieces of
linear algebra to find things,

00:55:08.810 --> 00:55:12.020
to reduce things
to triangular form,

00:55:12.020 --> 00:55:16.350
or to reduce them to connect
them with symmetric matrices.

00:55:16.350 --> 00:55:19.130
OK, thank you for
attention today.

00:55:19.130 --> 00:55:26.180
So today, we did the
distance matrices,

00:55:26.180 --> 00:55:29.720
and this was the final
step to get the X.

00:55:29.720 --> 00:55:35.840
And also, most
important was to get

00:55:35.840 --> 00:55:39.350
the structure of a
neural net straight,

00:55:39.350 --> 00:55:42.500
separating the v's,
the sample vectors,

00:55:42.500 --> 00:55:44.780
from the x's, the weights.

00:55:44.780 --> 00:55:49.580
OK, so Friday, I've
got one volunteer

00:55:49.580 --> 00:55:54.560
to talk about a project, and I'm
desperately looking for more.

00:55:54.560 --> 00:55:56.750
Please just send me an email.

00:56:00.226 --> 00:56:02.660
It'd would be appreciated,
or I'll send you an email,

00:56:02.660 --> 00:56:03.920
if necessary.

00:56:03.920 --> 00:56:05.880
OK, thanks.