WEBVTT

00:00:00.000 --> 00:00:02.390 align:middle line:90%
[SQUEAKING]

00:00:02.390 --> 00:00:04.302 align:middle line:90%
[RUSTLING]

00:00:04.302 --> 00:00:06.692 align:middle line:90%
[CLICKING]

00:00:06.692 --> 00:00:10.585 align:middle line:90%


00:00:10.585 --> 00:00:11.960 align:middle line:84%
ALAN EDELMAN: So
a quick problem.

00:00:11.960 --> 00:00:15.050 align:middle line:90%


00:00:15.050 --> 00:00:17.540 align:middle line:84%
You might want to grab
a pen and paper, and--

00:00:17.540 --> 00:00:20.870 align:middle line:84%
I think it's not that
hard, but let's just ask.

00:00:20.870 --> 00:00:27.327 align:middle line:90%


00:00:27.327 --> 00:00:28.910 align:middle line:84%
Let's say we have
this function, which

00:00:28.910 --> 00:00:38.927 align:middle line:84%
is just the 2-norm
of x From Rn to R.

00:00:38.927 --> 00:00:40.760 align:middle line:84%
So everybody knows this
function, of course.

00:00:40.760 --> 00:00:46.720 align:middle line:84%
It's the square root of x1
squared up to xn squared.

00:00:46.720 --> 00:00:49.360 align:middle line:84%
So a vector goes in,
a scalar comes out.

00:00:49.360 --> 00:00:52.750 align:middle line:84%
And my simple
question is what is

00:00:52.750 --> 00:00:56.370 align:middle line:90%
the gradient of this function?

00:00:56.370 --> 00:00:58.730 align:middle line:84%
And the reason why I'm
putting this up in part

00:00:58.730 --> 00:01:01.430 align:middle line:84%
is because I suspect
everybody in the room

00:01:01.430 --> 00:01:04.317 align:middle line:84%
can do this the 1802
way with indices.

00:01:04.317 --> 00:01:06.650 align:middle line:84%
And if you'd like to do it
that way because that's where

00:01:06.650 --> 00:01:09.500 align:middle line:90%
your comfort is, that's fine.

00:01:09.500 --> 00:01:14.000 align:middle line:84%
But I think the real
challenge is if you're--

00:01:14.000 --> 00:01:16.160 align:middle line:84%
to exercise what
you've learned so far

00:01:16.160 --> 00:01:19.460 align:middle line:84%
is to see if you can do
this without indices,

00:01:19.460 --> 00:01:21.710 align:middle line:90%
what I call the grown up way.

00:01:21.710 --> 00:01:23.870 align:middle line:90%
There's no right and wrong.

00:01:23.870 --> 00:01:26.790 align:middle line:84%
If you get the right
answer, that's fine.

00:01:26.790 --> 00:01:32.780 align:middle line:84%
But of course, I think that
the comfortable way is, well,

00:01:32.780 --> 00:01:35.030 align:middle line:84%
comfortable for everybody,
and the grown up way

00:01:35.030 --> 00:01:36.240 align:middle line:90%
takes some practice.

00:01:36.240 --> 00:01:40.020 align:middle line:84%
So let me put this up and I'll
go through it in a moment.

00:01:40.020 --> 00:01:45.300 align:middle line:84%
So a simple question, the
gradient of this function.

00:01:45.300 --> 00:01:50.480 align:middle line:84%
And I think the
1802 way might be

00:01:50.480 --> 00:01:52.220 align:middle line:84%
to start with the
first component,

00:01:52.220 --> 00:01:54.470 align:middle line:84%
but pretty quickly, you
realize that every component's

00:01:54.470 --> 00:01:55.830 align:middle line:90%
the same.

00:01:55.830 --> 00:02:00.930 align:middle line:84%
So you might start
calculating, for example, d dx1

00:02:00.930 --> 00:02:06.660 align:middle line:84%
of the square root of x1
squared plus x1 squared.

00:02:06.660 --> 00:02:09.820 align:middle line:84%
And using all the
calculus you remember,

00:02:09.820 --> 00:02:12.030 align:middle line:84%
the derivative of
something to the one half

00:02:12.030 --> 00:02:14.445 align:middle line:84%
is the same thing to
the negative one half.

00:02:14.445 --> 00:02:17.310 align:middle line:90%


00:02:17.310 --> 00:02:23.640 align:middle line:84%
And then you'd get 2x1, so
this would be 1 times 2x1.

00:02:23.640 --> 00:02:28.680 align:middle line:84%
And you would have x1 over
the square root of x1 squared

00:02:28.680 --> 00:02:30.600 align:middle line:90%
plus x1 squared.

00:02:30.600 --> 00:02:33.810 align:middle line:84%
And it wouldn't take you long
to realize that if you did x2,

00:02:33.810 --> 00:02:34.980 align:middle line:90%
there's no real difference.

00:02:34.980 --> 00:02:37.110 align:middle line:84%
And before you knew
it, you would find out

00:02:37.110 --> 00:02:42.150 align:middle line:84%
that the gradient, of course,
would be the vector x divided

00:02:42.150 --> 00:02:43.230 align:middle line:90%
by its length.

00:02:43.230 --> 00:02:45.000 align:middle line:90%
That gives you every--

00:02:45.000 --> 00:02:47.640 align:middle line:84%
and that would be what I
would call the 1802 way.

00:02:47.640 --> 00:02:50.100 align:middle line:90%
And it's not so bad.

00:02:50.100 --> 00:02:52.950 align:middle line:84%
But in this class,
we're trying to see

00:02:52.950 --> 00:02:55.050 align:middle line:90%
if we can avoid using indices.

00:02:55.050 --> 00:02:56.280 align:middle line:90%
Sometimes, you can't.

00:02:56.280 --> 00:02:58.050 align:middle line:90%
Many times you can.

00:02:58.050 --> 00:03:02.640 align:middle line:84%
And so we want to emphasize
the ability to not use indices.

00:03:02.640 --> 00:03:06.150 align:middle line:84%
And so I don't know, has
anyone already written it

00:03:06.150 --> 00:03:08.400 align:middle line:84%
out for the-- or convinced
themselves they could do it

00:03:08.400 --> 00:03:10.380 align:middle line:90%
without indices yet?

00:03:10.380 --> 00:03:13.120 align:middle line:90%
Has anyone done it yet?

00:03:13.120 --> 00:03:16.040 align:middle line:90%
But you think you can do it?

00:03:16.040 --> 00:03:21.845 align:middle line:84%
So I'm going to write down r
is equal to the length of x.

00:03:21.845 --> 00:03:25.670 align:middle line:90%


00:03:25.670 --> 00:03:29.960 align:middle line:84%
And therefore, r squared
is x transpose x.

00:03:29.960 --> 00:03:35.040 align:middle line:84%
This is the linear algebra
way to set things up.

00:03:35.040 --> 00:03:38.650 align:middle line:84%
And then I'm just going to take
the differential of both sides

00:03:38.650 --> 00:03:43.650 align:middle line:84%
and get to 2rdr equals, and
we know this differential,

00:03:43.650 --> 00:03:45.040 align:middle line:90%
so I can just write it down.

00:03:45.040 --> 00:03:47.580 align:middle line:84%
But if you forgot it,
you can get it yourself.

00:03:47.580 --> 00:03:50.650 align:middle line:90%


00:03:50.650 --> 00:03:51.520 align:middle line:90%
Sorry, this is d r.

00:03:51.520 --> 00:03:54.030 align:middle line:90%


00:03:54.030 --> 00:03:58.560 align:middle line:90%
It's 2x transpose dx.

00:03:58.560 --> 00:04:06.190 align:middle line:84%
And so therefore, dr is equal
to x transpose over rdx,

00:04:06.190 --> 00:04:09.760 align:middle line:84%
and then the gradient
is the transpose

00:04:09.760 --> 00:04:13.660 align:middle line:90%
of this, which is x over r.

00:04:13.660 --> 00:04:15.700 align:middle line:90%
So no indices.

00:04:15.700 --> 00:04:17.740 align:middle line:84%
That was kind of
the trick here, just

00:04:17.740 --> 00:04:21.850 align:middle line:84%
to be able to do this
thing without any indices.

00:04:21.850 --> 00:04:24.430 align:middle line:84%
Kind of like-- it's
sort of like superpowers

00:04:24.430 --> 00:04:27.370 align:middle line:90%
for vectors and matrices.

00:04:27.370 --> 00:04:29.500 align:middle line:90%
Like I said, this isn't hard.

00:04:29.500 --> 00:04:31.180 align:middle line:84%
What happens,
though, is as things

00:04:31.180 --> 00:04:33.190 align:middle line:84%
get a little more
complicated, being

00:04:33.190 --> 00:04:37.525 align:middle line:84%
good at this non-index approach
gets more and more valuable.

00:04:37.525 --> 00:04:40.540 align:middle line:90%


00:04:40.540 --> 00:04:42.490 align:middle line:84%
Because there's no one
right way to do it.

00:04:42.490 --> 00:04:45.640 align:middle line:84%
You could grind it through,
no matter how complicated.

00:04:45.640 --> 00:04:46.970 align:middle line:90%
I bet you all can do it.

00:04:46.970 --> 00:04:50.500 align:middle line:84%
But this is the
fun way to do it.

00:04:50.500 --> 00:04:51.740 align:middle line:90%
there's this thing over here.

00:04:51.740 --> 00:04:54.300 align:middle line:90%
Any questions about that?

00:04:54.300 --> 00:04:55.632 align:middle line:90%
OK.

00:04:55.632 --> 00:04:57.840 align:middle line:84%
So I don't know if you've
looked at the homework yet,

00:04:57.840 --> 00:05:01.460 align:middle line:84%
but the first problem
has the transformations

00:05:01.460 --> 00:05:04.400 align:middle line:84%
of the plane with the corgis,
and we're looking for 2

00:05:04.400 --> 00:05:06.420 align:middle line:90%
by 2 Jacobian matrices.

00:05:06.420 --> 00:05:12.140 align:middle line:90%
And in one case, there's--

00:05:12.140 --> 00:05:16.040 align:middle line:84%
in part D where
there's the warp,

00:05:16.040 --> 00:05:23.150 align:middle line:84%
there's a matrix function
of x that's multiplied by x.

00:05:23.150 --> 00:05:25.580 align:middle line:84%
If the matrix
didn't depend on x,

00:05:25.580 --> 00:05:28.370 align:middle line:84%
the Jacobian would
just be the matrix.

00:05:28.370 --> 00:05:30.530 align:middle line:84%
But because the
matrix depends on x,

00:05:30.530 --> 00:05:32.780 align:middle line:90%
there's going to be two terms.

00:05:32.780 --> 00:05:41.420 align:middle line:84%
There's going to be roughly
D matrix of x times x plus--

00:05:41.420 --> 00:05:45.410 align:middle line:90%
this is kind of the template.

00:05:45.410 --> 00:05:48.240 align:middle line:84%
And somehow, a dx is going to
come out of this term as well.

00:05:48.240 --> 00:05:53.300 align:middle line:84%
And so this term is the easy
one if the matrix was constant

00:05:53.300 --> 00:05:55.380 align:middle line:84%
and this one takes a
little bit more work.

00:05:55.380 --> 00:06:01.015 align:middle line:84%
So that's problem D on your
homework in the first part.

00:06:01.015 --> 00:06:01.515 align:middle line:90%
Good.

00:06:01.515 --> 00:06:05.420 align:middle line:90%


00:06:05.420 --> 00:06:08.270 align:middle line:84%
Well, then I'm going
to go back and do what

00:06:08.270 --> 00:06:10.912 align:middle line:90%
I was starting to do last time.

00:06:10.912 --> 00:06:13.370 align:middle line:84%
And I think it just makes sense
to go back to the beginning

00:06:13.370 --> 00:06:18.020 align:middle line:84%
since we've had a whole
weekend since last time.

00:06:18.020 --> 00:06:22.370 align:middle line:90%
And so matrix Jacobian.

00:06:22.370 --> 00:06:24.122 align:middle line:90%
So this Julia notebook here--

00:06:24.122 --> 00:06:26.330 align:middle line:84%
and by the way, if anybody
has questions about Julia,

00:06:26.330 --> 00:06:28.040 align:middle line:84%
feel free to ask
about that as well.

00:06:28.040 --> 00:06:32.564 align:middle line:90%
But the goal here--

00:06:32.564 --> 00:06:36.420 align:middle line:84%
this blackboard example is a
scalar function of a vector.

00:06:36.420 --> 00:06:39.770 align:middle line:84%
And I'm really excited
about functions

00:06:39.770 --> 00:06:43.190 align:middle line:84%
where the input and output are
actually matrices themselves.

00:06:43.190 --> 00:06:46.250 align:middle line:84%
So matrix squared, matrix
inverse we've seen,

00:06:46.250 --> 00:06:48.470 align:middle line:90%
matrix cubed.

00:06:48.470 --> 00:06:51.350 align:middle line:84%
I like to think of
the LU decomposition

00:06:51.350 --> 00:06:53.990 align:middle line:90%
as a function of matrices.

00:06:53.990 --> 00:06:55.890 align:middle line:84%
I don't know if you've
ever thought about it.

00:06:55.890 --> 00:06:59.510 align:middle line:84%
But the LU decomposition, or
even the eigendecomposition

00:06:59.510 --> 00:07:05.280 align:middle line:84%
of a matrix is a function from
matrices to pieces of matrices,

00:07:05.280 --> 00:07:06.760 align:middle line:90%
in effect.

00:07:06.760 --> 00:07:10.170 align:middle line:84%
So we'll see how
that goes through.

00:07:10.170 --> 00:07:16.210 align:middle line:84%
So let me go back to
the beginning here.

00:07:16.210 --> 00:07:19.570 align:middle line:84%
Just very quickly, we're
using Julia's symbolics,

00:07:19.570 --> 00:07:21.970 align:middle line:90%
which is kind of fun to do.

00:07:21.970 --> 00:07:23.405 align:middle line:90%
And this shows up later.

00:07:23.405 --> 00:07:25.030 align:middle line:84%
I don't know why it's
at the beginning.

00:07:25.030 --> 00:07:26.690 align:middle line:84%
So we'll just skip
right past it.

00:07:26.690 --> 00:07:29.710 align:middle line:84%
But here's the matrix that
we'll work with a lot.

00:07:29.710 --> 00:07:32.825 align:middle line:84%
This is the symbolic
matrix that has p,

00:07:32.825 --> 00:07:36.130 align:middle line:84%
q, r, s going in the
column major order.

00:07:36.130 --> 00:07:38.320 align:middle line:84%
And so I just
declared p, q, and r

00:07:38.320 --> 00:07:40.760 align:middle line:84%
and s to be variables,
as well as theta.

00:07:40.760 --> 00:07:45.050 align:middle line:84%
So Julia lets me put them
in as elements of matrices.

00:07:45.050 --> 00:07:46.450 align:middle line:90%
OK, so simple thing.

00:07:46.450 --> 00:07:50.770 align:middle line:84%
And last time, I
introduced the vec command,

00:07:50.770 --> 00:07:54.910 align:middle line:84%
which basically squashes
a 2 by 2 matrix, or any--

00:07:54.910 --> 00:07:58.630 align:middle line:84%
in fact, any matrix
down to a vector.

00:07:58.630 --> 00:08:01.420 align:middle line:84%
All you do with vec, in case
you missed it last time,

00:08:01.420 --> 00:08:04.600 align:middle line:84%
is you take the first column and
stack it on the second column,

00:08:04.600 --> 00:08:07.630 align:middle line:84%
stack it on the third
column, and so forth.

00:08:07.630 --> 00:08:13.610 align:middle line:84%
And this is, again, what we show
students because it's concrete.

00:08:13.610 --> 00:08:17.240 align:middle line:84%
But as far as I
can tell, you never

00:08:17.240 --> 00:08:20.960 align:middle line:84%
really need to vec things
other than maybe in your mind.

00:08:20.960 --> 00:08:24.140 align:middle line:84%
I mean, it's another one
of these comfort things

00:08:24.140 --> 00:08:27.680 align:middle line:84%
that I don't think you really
have to do in practice.

00:08:27.680 --> 00:08:31.190 align:middle line:84%
But it does have
a way of allowing

00:08:31.190 --> 00:08:33.750 align:middle line:84%
you to use one notation
for many, many things.

00:08:33.750 --> 00:08:35.000 align:middle line:90%
So there's the vec command.

00:08:35.000 --> 00:08:36.229 align:middle line:90%
It just squashes a matrix.

00:08:36.229 --> 00:08:39.299 align:middle line:90%


00:08:39.299 --> 00:08:41.970 align:middle line:84%
And here's the matrix
square of a 2 by 2 matrix.

00:08:41.970 --> 00:08:43.080 align:middle line:90%
All of you can do this.

00:08:43.080 --> 00:08:47.250 align:middle line:84%
Just take that p, q, r, s matrix
and do that dot producty thing,

00:08:47.250 --> 00:08:51.360 align:middle line:84%
and you get the matrix
square, which we can vec.

00:08:51.360 --> 00:08:54.360 align:middle line:84%
And maybe I'll make this
just a little bit smaller

00:08:54.360 --> 00:08:56.280 align:middle line:84%
so you can get a little
more real estate.

00:08:56.280 --> 00:08:57.870 align:middle line:90%
Yeah.

00:08:57.870 --> 00:08:59.280 align:middle line:90%
OK, good.

00:08:59.280 --> 00:09:03.810 align:middle line:84%
So this is the vec
of the matrix square.

00:09:03.810 --> 00:09:07.530 align:middle line:84%
And so one way to
think of matrix square

00:09:07.530 --> 00:09:10.170 align:middle line:90%
is a function from r4 to r4.

00:09:10.170 --> 00:09:13.830 align:middle line:84%
It's the input are the
variables p, q, r, and s,

00:09:13.830 --> 00:09:16.380 align:middle line:84%
and the output are
these four functions

00:09:16.380 --> 00:09:20.430 align:middle line:84%
of p, q, r, and s, the p squared
plus qr all the way down to qr

00:09:20.430 --> 00:09:22.730 align:middle line:90%
plus s squared.

00:09:22.730 --> 00:09:27.200 align:middle line:84%
So that that's a function
that takes r4 to r4.

00:09:27.200 --> 00:09:30.080 align:middle line:84%
But really we all know it's a
function from 2 by 2 matrices

00:09:30.080 --> 00:09:32.540 align:middle line:90%
to 2 by 2 matrices.

00:09:32.540 --> 00:09:34.250 align:middle line:90%
r4 is just a crutch.

00:09:34.250 --> 00:09:37.850 align:middle line:84%
It's really 2 by 2 matrices
to 2 by 2 matrices.

00:09:37.850 --> 00:09:42.890 align:middle line:84%
And so we can calculate
the Jacobian of this thing.

00:09:42.890 --> 00:09:45.470 align:middle line:84%
And I'll just remind you
that the Jacobian is nothing

00:09:45.470 --> 00:09:48.170 align:middle line:84%
other than all the derivatives
of each of these four

00:09:48.170 --> 00:09:51.950 align:middle line:84%
expressions with respect to all
of the four input variables p,

00:09:51.950 --> 00:09:53.030 align:middle line:90%
q, r, and s.

00:09:53.030 --> 00:09:55.620 align:middle line:90%


00:09:55.620 --> 00:10:00.020 align:middle line:84%
So this output p squared plus
qr belongs to the first row.

00:10:00.020 --> 00:10:02.000 align:middle line:84%
And we'll take the
derivative with respect p,

00:10:02.000 --> 00:10:04.310 align:middle line:84%
and then q, and
then r, and then s.

00:10:04.310 --> 00:10:09.080 align:middle line:84%
And if you do that, you'll get
this first row, and so forth.

00:10:09.080 --> 00:10:13.640 align:middle line:84%
So the inputs are the columns,
the outputs are the rows,

00:10:13.640 --> 00:10:15.530 align:middle line:90%
and that's the Jacobian matrix.

00:10:15.530 --> 00:10:17.930 align:middle line:84%
And Julia is happy to
compute it for you,

00:10:17.930 --> 00:10:20.660 align:middle line:84%
but you could have all
done this yourself as well.

00:10:20.660 --> 00:10:21.990 align:middle line:90%
So there's the Jacobian.

00:10:21.990 --> 00:10:25.650 align:middle line:90%
And our goal is to--

00:10:25.650 --> 00:10:27.720 align:middle line:84%
our goal is really to
find a notation that

00:10:27.720 --> 00:10:31.260 align:middle line:84%
lets you write this matrix
without writing the matrix.

00:10:31.260 --> 00:10:32.760 align:middle line:84%
That's what we're
trying to do here.

00:10:32.760 --> 00:10:36.030 align:middle line:84%
We'd like to write this matrix,
but without writing the matrix.

00:10:36.030 --> 00:10:38.370 align:middle line:90%
That's what we're heading for.

00:10:38.370 --> 00:10:41.830 align:middle line:90%
But I love to check things over.

00:10:41.830 --> 00:10:43.668 align:middle line:84%
I like to do things
with symbols, if I can.

00:10:43.668 --> 00:10:45.460 align:middle line:84%
I like to do things
with numbers, if I can.

00:10:45.460 --> 00:10:47.600 align:middle line:84%
It makes me happy when
things come out equal.

00:10:47.600 --> 00:10:50.330 align:middle line:90%
So let's do this numerically.

00:10:50.330 --> 00:10:56.010 align:middle line:84%
So here's-- my inputs are
going to be 1, 2, 3, 4.

00:10:56.010 --> 00:11:02.220 align:middle line:84%
And I'm going to make some
perturbations of the 1, 2, 3, 4

00:11:02.220 --> 00:11:05.670 align:middle line:84%
by just taking 1, 2,
3, 4 divided by 10,000.

00:11:05.670 --> 00:11:07.200 align:middle line:90%
And I'm going to put--

00:11:07.200 --> 00:11:08.760 align:middle line:84%
into this Jacobian,
I'm just going

00:11:08.760 --> 00:11:10.464 align:middle line:84%
to put in these
numbers, this Jacobian.

00:11:10.464 --> 00:11:16.170 align:middle line:84%
So this is the Jacobian
of the square function

00:11:16.170 --> 00:11:19.380 align:middle line:90%
at the matrix 1, 2, 3, 4.

00:11:19.380 --> 00:11:23.130 align:middle line:84%
So if I perturb the matrix a
little bit, like this E here,

00:11:23.130 --> 00:11:25.125 align:middle line:84%
I should know how
the matrix perturbs.

00:11:25.125 --> 00:11:27.810 align:middle line:90%


00:11:27.810 --> 00:11:30.330 align:middle line:84%
And so all I should
have to do is

00:11:30.330 --> 00:11:32.550 align:middle line:90%
take this Jacobian, which is--

00:11:32.550 --> 00:11:34.920 align:middle line:84%
this is the same
expression as here,

00:11:34.920 --> 00:11:39.420 align:middle line:84%
and multiply that by the
vec of my perturbation.

00:11:39.420 --> 00:11:44.560 align:middle line:84%
And of course, here's
what we get as the answer.

00:11:44.560 --> 00:11:47.940 align:middle line:84%
And we can check that by just
doing a finite difference

00:11:47.940 --> 00:11:49.050 align:middle line:90%
with the matrices.

00:11:49.050 --> 00:11:52.020 align:middle line:84%
And you could see the
first order, it just works.

00:11:52.020 --> 00:11:54.090 align:middle line:90%
So when I see this I'm happy.

00:11:54.090 --> 00:11:56.460 align:middle line:90%
Other people trust mathematics.

00:11:56.460 --> 00:11:58.740 align:middle line:90%
Me, I make too many errors.

00:11:58.740 --> 00:12:03.100 align:middle line:84%
I have to double check it,
just to see these numbers,

00:12:03.100 --> 00:12:06.970 align:middle line:84%
this 0.0014, and the
0.0030, and the 0.0020,

00:12:06.970 --> 00:12:10.090 align:middle line:84%
and the 0.0044 just matching
up with the numbers above.

00:12:10.090 --> 00:12:12.340 align:middle line:84%
Again, this is just
the finite differences.

00:12:12.340 --> 00:12:14.740 align:middle line:84%
This is what the
theory tells you.

00:12:14.740 --> 00:12:17.530 align:middle line:84%
And I don't know if you guys
need to check your work,

00:12:17.530 --> 00:12:19.060 align:middle line:90%
but I love checking my work.

00:12:19.060 --> 00:12:22.220 align:middle line:84%
Invariably, I lose
factors of two or pi.

00:12:22.220 --> 00:12:24.130 align:middle line:84%
So this is the best
way to find them.

00:12:24.130 --> 00:12:31.310 align:middle line:90%


00:12:31.310 --> 00:12:35.870 align:middle line:84%
So rather than vecing
and flattening,

00:12:35.870 --> 00:12:40.623 align:middle line:84%
another way to basically express
the Jacobian of a square,

00:12:40.623 --> 00:12:42.290 align:middle line:84%
and you'll see why
it works in a minute,

00:12:42.290 --> 00:12:45.810 align:middle line:84%
is to express it as a
linear transformation.

00:12:45.810 --> 00:12:48.108 align:middle line:84%
And one of the things
that if it didn't-- if you

00:12:48.108 --> 00:12:50.400 align:middle line:84%
weren't-- if you weren't hit
with this with your linear

00:12:50.400 --> 00:12:52.680 align:middle line:84%
algebra class when you
first took linear algebra,

00:12:52.680 --> 00:12:58.920 align:middle line:84%
then this is a good time to
recognize that being able

00:12:58.920 --> 00:13:01.410 align:middle line:84%
to write down a linear
transformation doesn't mean you

00:13:01.410 --> 00:13:03.450 align:middle line:90%
have to write down a matrix.

00:13:03.450 --> 00:13:05.400 align:middle line:84%
You could write down
linear transformations

00:13:05.400 --> 00:13:07.050 align:middle line:90%
without writing down a matrix.

00:13:07.050 --> 00:13:12.660 align:middle line:84%
For example, you think
of M as a constant here.

00:13:12.660 --> 00:13:15.600 align:middle line:84%
So M is going to be our position
where we're taking derivative

00:13:15.600 --> 00:13:16.350 align:middle line:90%
of whatever it is.

00:13:16.350 --> 00:13:17.970 align:middle line:90%
It's a constant.

00:13:17.970 --> 00:13:21.750 align:middle line:84%
And I just want to
point out that this

00:13:21.750 --> 00:13:24.060 align:middle line:84%
is a linear transformation
of matrices.

00:13:24.060 --> 00:13:27.810 align:middle line:84%
So if M is a 2 by 2 matrix
and E is a 2 by 2 matrix,

00:13:27.810 --> 00:13:31.170 align:middle line:84%
this expression, without
writing down the matrix,

00:13:31.170 --> 00:13:33.343 align:middle line:90%
is a linear transformation.

00:13:33.343 --> 00:13:35.760 align:middle line:84%
What do you have to check if
it's a linear transformation?

00:13:35.760 --> 00:13:36.427 align:middle line:90%
Well, let's see.

00:13:36.427 --> 00:13:39.600 align:middle line:84%
If I multiply E by a
number like 2 or pi,

00:13:39.600 --> 00:13:41.910 align:middle line:84%
does the output get
multiplied by the same number

00:13:41.910 --> 00:13:42.960 align:middle line:90%
like 2 or pi?

00:13:42.960 --> 00:13:44.850 align:middle line:84%
Yep, the scalar
would pull right out.

00:13:44.850 --> 00:13:49.015 align:middle line:84%
And if I input E1 plus
E2, would the output

00:13:49.015 --> 00:13:53.270 align:middle line:84%
be the output of the 1
plus the output of the 2?

00:13:53.270 --> 00:13:55.815 align:middle line:84%
You check those two
things and that's

00:13:55.815 --> 00:13:57.940 align:middle line:84%
what you need to check that
the function is linear.

00:13:57.940 --> 00:14:00.130 align:middle line:90%
You don't need a matrix.

00:14:00.130 --> 00:14:01.600 align:middle line:84%
Again, matrices
are very concrete,

00:14:01.600 --> 00:14:03.850 align:middle line:84%
but you don't need them
when you're grown up.

00:14:03.850 --> 00:14:07.300 align:middle line:84%
So this is a linear
transformation on E

00:14:07.300 --> 00:14:09.910 align:middle line:84%
and it's going to
turn out to be exactly

00:14:09.910 --> 00:14:14.200 align:middle line:84%
the right linear transformation
for the very matrix

00:14:14.200 --> 00:14:15.290 align:middle line:90%
we have at hand.

00:14:15.290 --> 00:14:18.743 align:middle line:84%
And so with M being the matrix
1, 2, 3, 4 and E-- here,

00:14:18.743 --> 00:14:20.410 align:middle line:84%
let's remind you what
these matrices are

00:14:20.410 --> 00:14:21.577 align:middle line:90%
if you've already forgotten.

00:14:21.577 --> 00:14:24.550 align:middle line:84%
But if M is the
matrix 1, 2, 3, 4

00:14:24.550 --> 00:14:28.570 align:middle line:84%
and E is the perturbation
matrix, the same numbers

00:14:28.570 --> 00:14:33.190 align:middle line:84%
divided by 10,000, this linear
transformation applied to E

00:14:33.190 --> 00:14:36.250 align:middle line:84%
gives you the
derivative without--

00:14:36.250 --> 00:14:38.050 align:middle line:84%
somehow, the
information in here is

00:14:38.050 --> 00:14:40.180 align:middle line:84%
the same as the information
of the Jacobian,

00:14:40.180 --> 00:14:42.970 align:middle line:84%
and yet I never
form the Jacobian.

00:14:42.970 --> 00:14:44.860 align:middle line:90%
That's a key, key point.

00:14:44.860 --> 00:14:48.960 align:middle line:84%
Never form the Jacobian, but the
information to get the answer

00:14:48.960 --> 00:14:52.050 align:middle line:90%
is embodied in this expression.

00:14:52.050 --> 00:14:54.810 align:middle line:84%
So who needs to
build a 4 by 4 matrix

00:14:54.810 --> 00:14:56.580 align:middle line:84%
if you can just have
this expression?

00:14:56.580 --> 00:14:58.210 align:middle line:90%
Questions?

00:14:58.210 --> 00:14:59.290 align:middle line:90%
Anything?

00:14:59.290 --> 00:15:00.110 align:middle line:90%
Does it make sense?

00:15:00.110 --> 00:15:02.680 align:middle line:90%
Doesn't make sense?

00:15:02.680 --> 00:15:05.480 align:middle line:90%
You want to see why that works?

00:15:05.480 --> 00:15:08.840 align:middle line:84%
So I find the Kronecker
product notation is a good way

00:15:08.840 --> 00:15:14.240 align:middle line:84%
to be ready to
set this thing up.

00:15:14.240 --> 00:15:17.510 align:middle line:84%
So let me tell you about the
Kronecker product notation

00:15:17.510 --> 00:15:22.010 align:middle line:84%
and I'm going to set up these
variables a, b, c, and d.

00:15:22.010 --> 00:15:25.400 align:middle line:84%
And here's just a
pair of matrices.

00:15:25.400 --> 00:15:29.810 align:middle line:84%
And what I want you to
notice is that we have--

00:15:29.810 --> 00:15:31.853 align:middle line:90%
oh, am I sharing my screen?

00:15:31.853 --> 00:15:33.520 align:middle line:84%
It just occurred to
me that I'm not even

00:15:33.520 --> 00:15:34.870 align:middle line:90%
sure I'm sharing my screen.

00:15:34.870 --> 00:15:36.250 align:middle line:84%
AUDIENCE: You're not
sharing it on Zoom, though.

00:15:36.250 --> 00:15:37.270 align:middle line:90%
ALAN EDELMAN: Oh, yeah.

00:15:37.270 --> 00:15:38.970 align:middle line:90%
Sorry about that.

00:15:38.970 --> 00:15:43.120 align:middle line:84%
Just hit me that you
don't see what I'm doing.

00:15:43.120 --> 00:15:45.230 align:middle line:90%
All right, there we go.

00:15:45.230 --> 00:15:47.380 align:middle line:84%
And the reason I realized
that is because I wanted

00:15:47.380 --> 00:15:50.320 align:middle line:90%
to use this pen here anyway.

00:15:50.320 --> 00:15:53.920 align:middle line:84%
So I want everybody to
see the top half versus

00:15:53.920 --> 00:15:56.920 align:middle line:90%
the bottom half of this matrix.

00:15:56.920 --> 00:16:00.220 align:middle line:84%
So here what I'm
doing is I'm taking

00:16:00.220 --> 00:16:08.085 align:middle line:84%
the Kronecker product of this
2 by 1 matrix, Kronecker the--

00:16:08.085 --> 00:16:08.960 align:middle line:90%
see if I can do this.

00:16:08.960 --> 00:16:10.990 align:middle line:84%
It's hard to-- I'm
using just the mouse.

00:16:10.990 --> 00:16:12.475 align:middle line:90%
So p, q, r, and s.

00:16:12.475 --> 00:16:13.850 align:middle line:84%
Is this coming
out on the screen?

00:16:13.850 --> 00:16:15.220 align:middle line:90%
Can you see the green?

00:16:15.220 --> 00:16:16.990 align:middle line:90%
So p, q, r, and s.

00:16:16.990 --> 00:16:20.450 align:middle line:90%
And I just want you to see--

00:16:20.450 --> 00:16:21.690 align:middle line:90%
yes.

00:16:21.690 --> 00:16:23.220 align:middle line:90%
Oops.

00:16:23.220 --> 00:16:24.780 align:middle line:90%
It's not doing it now.

00:16:24.780 --> 00:16:32.690 align:middle line:90%


00:16:32.690 --> 00:16:35.130 align:middle line:90%
there we go.

00:16:35.130 --> 00:16:35.630 align:middle line:90%
OK.

00:16:35.630 --> 00:16:38.810 align:middle line:84%
So I want you all
to see that it's

00:16:38.810 --> 00:16:42.890 align:middle line:84%
taking the whole second
matrix p, q, r, s,

00:16:42.890 --> 00:16:45.290 align:middle line:90%
and multiplying it by a on top.

00:16:45.290 --> 00:16:48.160 align:middle line:84%
And then this is taking the
whole second matrix, pqrs,

00:16:48.160 --> 00:16:50.760 align:middle line:84%
and multiplying by
b on the bottom.

00:16:50.760 --> 00:16:53.040 align:middle line:84%
And so what it
does is, in effect.

00:16:53.040 --> 00:16:56.240 align:middle line:84%
It takes-- you'll see that every
possible product of something

00:16:56.240 --> 00:16:58.520 align:middle line:84%
over here on the left
and something over here

00:16:58.520 --> 00:17:01.040 align:middle line:84%
on the right shows
up, and it shows up

00:17:01.040 --> 00:17:02.744 align:middle line:90%
in a very consistent order.

00:17:02.744 --> 00:17:04.369 align:middle line:84%
Once you see a couple
of more examples,

00:17:04.369 --> 00:17:06.030 align:middle line:90%
you'll see how the pattern goes.

00:17:06.030 --> 00:17:08.609 align:middle line:84%
So this is the
Kronecker product.

00:17:08.609 --> 00:17:11.060 align:middle line:90%
So let's see.

00:17:11.060 --> 00:17:17.180 align:middle line:84%
So here's an example where
we take abcd with pqrs.

00:17:17.180 --> 00:17:19.640 align:middle line:84%
And I'm sure you can
all see that there's

00:17:19.640 --> 00:17:24.815 align:middle line:84%
a pqrs times the a, pqrs now
on the bottom left times the b.

00:17:24.815 --> 00:17:28.490 align:middle line:84%
In the upper right, the pqrs
is times the c and times the d.

00:17:28.490 --> 00:17:30.800 align:middle line:90%
So the pattern is--

00:17:30.800 --> 00:17:34.950 align:middle line:84%
one way to look at it is you
take every element on the left

00:17:34.950 --> 00:17:38.350 align:middle line:84%
and multiply it by the
entire matrix on the right,

00:17:38.350 --> 00:17:40.260 align:middle line:84%
and then put it
in the same order.

00:17:40.260 --> 00:17:44.700 align:middle line:84%
And so sometimes it's easier
to see it with these pictures.

00:17:44.700 --> 00:17:51.600 align:middle line:84%
And so here what we have is this
2 by 3 matrix, which is abcdef.

00:17:51.600 --> 00:17:55.420 align:middle line:84%
So I can put it over here
if you want to see it.

00:17:55.420 --> 00:17:58.950 align:middle line:84%
So I'm taking the Kronecker
product of abcdef this way

00:17:58.950 --> 00:18:04.050 align:middle line:84%
and I'm taking the
Kronecker product with--

00:18:04.050 --> 00:18:06.360 align:middle line:90%
there's the pizza.

00:18:06.360 --> 00:18:07.470 align:middle line:90%
What do we have?

00:18:07.470 --> 00:18:10.500 align:middle line:90%
The alien.

00:18:10.500 --> 00:18:17.290 align:middle line:84%
The pizza, the alien, the smiley
cat, and what do we have here?

00:18:17.290 --> 00:18:17.790 align:middle line:90%
The panda.

00:18:17.790 --> 00:18:21.010 align:middle line:90%


00:18:21.010 --> 00:18:27.610 align:middle line:84%
So Julia's perfectly comfortable
with doing that sort of thing.

00:18:27.610 --> 00:18:34.180 align:middle line:84%
And you could see that we
have a times the pizza.

00:18:34.180 --> 00:18:36.650 align:middle line:84%
Did I put the alien
in the wrong order?

00:18:36.650 --> 00:18:37.220 align:middle line:90%
Yep.

00:18:37.220 --> 00:18:39.860 align:middle line:90%
The alien's over here.

00:18:39.860 --> 00:18:42.350 align:middle line:90%
But you get the idea.

00:18:42.350 --> 00:18:45.140 align:middle line:84%
So you have a times
pizza alien panda cat,

00:18:45.140 --> 00:18:48.140 align:middle line:84%
b times pizza alien
panda cat, c times

00:18:48.140 --> 00:18:51.170 align:middle line:84%
pizza alien panda
cat, d times those,

00:18:51.170 --> 00:18:53.940 align:middle line:90%
e times those, f times those.

00:18:53.940 --> 00:18:55.160 align:middle line:90%
I think you get the pattern.

00:18:55.160 --> 00:18:56.900 align:middle line:84%
You could do it in
the other order.

00:18:56.900 --> 00:18:59.390 align:middle line:90%
Let's see, what else did we do?

00:18:59.390 --> 00:19:02.420 align:middle line:84%
Oh, here it is
with the identity.

00:19:02.420 --> 00:19:08.720 align:middle line:84%
So this one is pizza times 1,
pizza times 0, pizza times 0,

00:19:08.720 --> 00:19:09.620 align:middle line:90%
pizza times 1.

00:19:09.620 --> 00:19:12.170 align:middle line:84%
Alien times 1, alien
times 0, alien times 0,

00:19:12.170 --> 00:19:14.480 align:middle line:84%
alien times-- yeah, I
think you all get it.

00:19:14.480 --> 00:19:16.635 align:middle line:90%
But it's still fun to watch.

00:19:16.635 --> 00:19:17.885 align:middle line:90%
Here it is in the other order.

00:19:17.885 --> 00:19:20.740 align:middle line:90%


00:19:20.740 --> 00:19:25.180 align:middle line:84%
So this is 1 times pizza
alien panda cat, 0 times

00:19:25.180 --> 00:19:28.510 align:middle line:84%
pizza alien, 0 times the
2, and then again 1 times

00:19:28.510 --> 00:19:30.190 align:middle line:90%
all of these funny symbols.

00:19:30.190 --> 00:19:33.370 align:middle line:90%
OK, you get it.

00:19:33.370 --> 00:19:33.870 align:middle line:90%
Yeah.

00:19:33.870 --> 00:19:35.670 align:middle line:84%
So the Kronecker product
with the identity

00:19:35.670 --> 00:19:38.310 align:middle line:84%
is a rather important
special case.

00:19:38.310 --> 00:19:42.140 align:middle line:84%
So here it is with
just the pqrs.

00:19:42.140 --> 00:19:44.777 align:middle line:84%
Here's the Kronecker
product of the transpose

00:19:44.777 --> 00:19:47.360 align:middle line:84%
with the identity, just putting
everything in the other order.

00:19:47.360 --> 00:19:50.640 align:middle line:90%


00:19:50.640 --> 00:19:54.300 align:middle line:84%
And what I want
to demonstrate is

00:19:54.300 --> 00:19:59.940 align:middle line:84%
that if you form the sum of
these two Kronecker products

00:19:59.940 --> 00:20:03.030 align:middle line:84%
as matrices and compare
it with the Jacobian

00:20:03.030 --> 00:20:05.470 align:middle line:84%
that you've already seen
for the square function,

00:20:05.470 --> 00:20:09.870 align:middle line:84%
you'll notice that you get
exactly the same expression.

00:20:09.870 --> 00:20:13.800 align:middle line:84%
So one way to write the Jacobian
without writing the Jacobian

00:20:13.800 --> 00:20:15.885 align:middle line:84%
is to just write these
Kronecker products.

00:20:15.885 --> 00:20:19.150 align:middle line:90%


00:20:19.150 --> 00:20:22.660 align:middle line:84%
So you don't-- just because
you write down the Kronecker

00:20:22.660 --> 00:20:25.810 align:middle line:84%
product, it doesn't mean you
actually have to multiply it

00:20:25.810 --> 00:20:26.800 align:middle line:90%
all out, though.

00:20:26.800 --> 00:20:29.140 align:middle line:84%
You could write it
as an expression.

00:20:29.140 --> 00:20:34.810 align:middle line:84%
And what underlies everything,
an identity that years

00:20:34.810 --> 00:20:38.290 align:middle line:84%
ago, I remember I wrote
this in big thick letters

00:20:38.290 --> 00:20:41.260 align:middle line:84%
on an index card, and I
placed it on my bookshelf

00:20:41.260 --> 00:20:43.850 align:middle line:84%
above my desk to make sure
that I would never forget it.

00:20:43.850 --> 00:20:46.390 align:middle line:84%
So this is absolutely
worth memorizing.

00:20:46.390 --> 00:20:49.750 align:middle line:84%
That if you have the
right compatibility

00:20:49.750 --> 00:20:53.410 align:middle line:84%
so that you can multiply b
times c times a transpose

00:20:53.410 --> 00:20:56.200 align:middle line:84%
so the number of columns of has
to be the number of rows of c,

00:20:56.200 --> 00:21:00.450 align:middle line:84%
et cetera, that the
Kronecker product of a and b,

00:21:00.450 --> 00:21:04.360 align:middle line:84%
when multiplied by
the vec of c, always

00:21:04.360 --> 00:21:07.880 align:middle line:84%
is identically equal to
the vec of bca transpose.

00:21:07.880 --> 00:21:11.410 align:middle line:84%
So I would recommend memorizing
that because it's just

00:21:11.410 --> 00:21:14.380 align:middle line:90%
so handy to always know that.

00:21:14.380 --> 00:21:18.710 align:middle line:84%
And the mnemonic that I
still have in my mind,

00:21:18.710 --> 00:21:21.890 align:middle line:84%
I mean, you could just say
to yourself, bca transpose.

00:21:21.890 --> 00:21:24.300 align:middle line:84%
But the way I look at it,
and if this works for you,

00:21:24.300 --> 00:21:25.800 align:middle line:84%
that's fine, and
if it doesn't work,

00:21:25.800 --> 00:21:26.970 align:middle line:90%
you can have your own way.

00:21:26.970 --> 00:21:31.946 align:middle line:84%
But the way I look at
this is I see the b

00:21:31.946 --> 00:21:34.490 align:middle line:90%
is closest to the c on the left.

00:21:34.490 --> 00:21:36.320 align:middle line:90%
So I say bc.

00:21:36.320 --> 00:21:38.720 align:middle line:84%
And then I say, ah, the
a, then, should go--

00:21:38.720 --> 00:21:40.490 align:middle line:84%
this is, again, a
little memory thing.

00:21:40.490 --> 00:21:41.540 align:middle line:90%
It doesn't mean anything.

00:21:41.540 --> 00:21:45.020 align:middle line:84%
But the a should go
around the other way.

00:21:45.020 --> 00:21:49.160 align:middle line:84%
And since it went around, it
deserves to be transposed.

00:21:49.160 --> 00:21:51.200 align:middle line:84%
So if that works for
you, that's fine.

00:21:51.200 --> 00:21:53.240 align:middle line:84%
The b is just to
the left of the c.

00:21:53.240 --> 00:21:57.170 align:middle line:84%
The a has to come around
so it gets transposed.

00:21:57.170 --> 00:21:59.420 align:middle line:90%
v And that always works.

00:21:59.420 --> 00:22:02.620 align:middle line:90%
And so you can--

00:22:02.620 --> 00:22:05.260 align:middle line:84%
in fact, here is some random--
here you can see it with some

00:22:05.260 --> 00:22:08.560 align:middle line:90%
random numbers, where--

00:22:08.560 --> 00:22:11.290 align:middle line:84%
and in fact, I could type
this till the cows come home.

00:22:11.290 --> 00:22:13.490 align:middle line:84%
It'll always come out
to be a true identity.

00:22:13.490 --> 00:22:14.740 align:middle line:90%
So here we can--

00:22:14.740 --> 00:22:18.470 align:middle line:84%
I could hit it again,
but it'll always be true.

00:22:18.470 --> 00:22:22.040 align:middle line:90%
So you see it's bca transpose.

00:22:22.040 --> 00:22:23.390 align:middle line:90%
OK, everybody convinced?

00:22:23.390 --> 00:22:27.500 align:middle line:90%


00:22:27.500 --> 00:22:31.010 align:middle line:84%
So the Kronecker product
of two 5 by 5 matrices

00:22:31.010 --> 00:22:33.650 align:middle line:84%
would be 25 by 25,
just to show you.

00:22:33.650 --> 00:22:37.560 align:middle line:84%
And here are some useful
identities worth knowing about.

00:22:37.560 --> 00:22:41.660 align:middle line:84%
So if you take the transpose
of a Kronecker product,

00:22:41.660 --> 00:22:44.330 align:middle line:84%
it's the Kronecker
product of the transposes.

00:22:44.330 --> 00:22:47.120 align:middle line:90%
And it's not in the other order.

00:22:47.120 --> 00:22:49.570 align:middle line:84%
If you were thinking of
ordinary matrix multiply,

00:22:49.570 --> 00:22:50.570 align:middle line:90%
you'd have AB transpose.

00:22:50.570 --> 00:22:53.180 align:middle line:84%
You know that would be
B transpose A transpose.

00:22:53.180 --> 00:22:55.940 align:middle line:84%
But when you do the
Kronecker product,

00:22:55.940 --> 00:22:57.710 align:middle line:90%
the order remains the same.

00:22:57.710 --> 00:23:00.800 align:middle line:84%
So don't get confused with
matrix multiplication.

00:23:00.800 --> 00:23:03.500 align:middle line:84%
Again, with inverse, inverse,
if this wasn't a Kronecker

00:23:03.500 --> 00:23:06.260 align:middle line:84%
product, if it was a matrix
multiply product, again,

00:23:06.260 --> 00:23:08.720 align:middle line:84%
you would put this
in the other order.

00:23:08.720 --> 00:23:12.320 align:middle line:84%
But with Kronecker product,
it stays in that order.

00:23:12.320 --> 00:23:15.560 align:middle line:84%
Another useful thing to know is
if a and are square matrices,

00:23:15.560 --> 00:23:21.440 align:middle line:84%
if A's n by n and B is m by
m, then the determinant is

00:23:21.440 --> 00:23:23.570 align:middle line:90%
the determinant of a to the m.

00:23:23.570 --> 00:23:25.350 align:middle line:90%
So a is the n by n1.

00:23:25.350 --> 00:23:26.880 align:middle line:90%
It picks up the m from the b.

00:23:26.880 --> 00:23:30.060 align:middle line:90%
And then B is the m by m1.

00:23:30.060 --> 00:23:32.250 align:middle line:90%
It picks up the n from the a.

00:23:32.250 --> 00:23:33.810 align:middle line:90%
So there's a formula.

00:23:33.810 --> 00:23:37.800 align:middle line:84%
The trace of a Kronecker product
is the product of the traces.

00:23:37.800 --> 00:23:41.520 align:middle line:84%
A Kronecker B is orthogonal,
if A and B are both orthogonal.

00:23:41.520 --> 00:23:44.490 align:middle line:84%
The matrix multiply of a
Kronecker B and C Kronecker D

00:23:44.490 --> 00:23:47.760 align:middle line:84%
is the matrix multiply of A
and C with that of B and D.

00:23:47.760 --> 00:23:51.545 align:middle line:84%
And yeah, so A Kronecker
B and B Kronecker A

00:23:51.545 --> 00:23:53.420 align:middle line:84%
have the same eigenvalues,
which are actually

00:23:53.420 --> 00:23:56.630 align:middle line:84%
the products of the eigenvalues
of A and B separately,

00:23:56.630 --> 00:24:00.200 align:middle line:84%
and the eigenvectors are just
transposed of each other.

00:24:00.200 --> 00:24:06.070 align:middle line:84%
And so maybe this is worth
writing out on the board

00:24:06.070 --> 00:24:09.650 align:middle line:84%
just to see it, but this
is how I like to do this.

00:24:09.650 --> 00:24:14.200 align:middle line:84%
So if somebody tells me to
talk about the derivative--

00:24:14.200 --> 00:24:16.180 align:middle line:84%
this is my favorite
way of seeing

00:24:16.180 --> 00:24:18.490 align:middle line:90%
the derivative of x squared.

00:24:18.490 --> 00:24:22.690 align:middle line:84%
So I'm going to go over
here and I'm going to say,

00:24:22.690 --> 00:24:26.170 align:middle line:84%
the derivative of
x squared, which

00:24:26.170 --> 00:24:28.923 align:middle line:84%
we've all known to-- we've
seen this now a few times.

00:24:28.923 --> 00:24:30.340 align:middle line:84%
I'm going to use
this white chalk.

00:24:30.340 --> 00:24:32.540 align:middle line:84%
The derivative of x
squared, of course,

00:24:32.540 --> 00:24:36.940 align:middle line:84%
is going to be the
matrix x times dx plus dx

00:24:36.940 --> 00:24:40.960 align:middle line:90%
x from the product rule.

00:24:40.960 --> 00:24:42.820 align:middle line:84%
But I'm going to
look at this and I'm

00:24:42.820 --> 00:24:46.090 align:middle line:84%
going to say in my
mind, this is xdxi.

00:24:46.090 --> 00:24:48.850 align:middle line:90%


00:24:48.850 --> 00:24:52.190 align:middle line:90%
And this is idxx.

00:24:52.190 --> 00:24:55.390 align:middle line:84%
I normally wouldn't write that
out, but just for purposes

00:24:55.390 --> 00:24:58.930 align:middle line:84%
of showing you folks, I'll
put the identity matrix in.

00:24:58.930 --> 00:25:01.120 align:middle line:90%
And then I would say that--

00:25:01.120 --> 00:25:03.790 align:middle line:90%


00:25:03.790 --> 00:25:05.720 align:middle line:84%
which Kronecker
product do I need?

00:25:05.720 --> 00:25:06.910 align:middle line:90%
Well, let's see.

00:25:06.910 --> 00:25:11.260 align:middle line:84%
The thing that's on the left is
going to be closest to the dx,

00:25:11.260 --> 00:25:13.120 align:middle line:90%
so I'll put the x here.

00:25:13.120 --> 00:25:15.820 align:middle line:84%
And then this I
would be transposed.

00:25:15.820 --> 00:25:18.490 align:middle line:84%
But since it's the identity, I
don't have to write that out.

00:25:18.490 --> 00:25:21.850 align:middle line:84%
And then here, you
see the same game.

00:25:21.850 --> 00:25:23.740 align:middle line:84%
The thing that's to
the left of the x

00:25:23.740 --> 00:25:28.000 align:middle line:84%
is the I. This thing then
has to be x transpose.

00:25:28.000 --> 00:25:33.910 align:middle line:84%
And so one thing you
could write is the vec--

00:25:33.910 --> 00:25:35.530 align:middle line:90%
but I don't really like this.

00:25:35.530 --> 00:25:37.780 align:middle line:84%
But this would be
reasonable to write.

00:25:37.780 --> 00:25:43.130 align:middle line:84%
The vec of dx squared is
this times the vec of x.

00:25:43.130 --> 00:25:46.400 align:middle line:84%
And this is expressing--
it's just another notation

00:25:46.400 --> 00:25:48.290 align:middle line:90%
for what's over there.

00:25:48.290 --> 00:25:51.155 align:middle line:84%
But what I really
like to write is

00:25:51.155 --> 00:25:53.030 align:middle line:84%
I like to think of this
as a linear operator,

00:25:53.030 --> 00:25:54.613 align:middle line:84%
and maybe I'll use
the square brackets

00:25:54.613 --> 00:25:55.790 align:middle line:90%
that Stephen likes to use.

00:25:55.790 --> 00:25:59.840 align:middle line:84%
And I think I would-- the one I
would enjoy the most to write,

00:25:59.840 --> 00:26:02.060 align:middle line:84%
if I have to put everything
on the-- you see,

00:26:02.060 --> 00:26:05.520 align:middle line:84%
there's something nice about
putting the dx on the right.

00:26:05.520 --> 00:26:09.050 align:middle line:84%
So I think this is the notation
that I would like the best.

00:26:09.050 --> 00:26:14.000 align:middle line:84%
This is equal to this, as
an operator acting on dx.

00:26:14.000 --> 00:26:16.970 align:middle line:90%


00:26:16.970 --> 00:26:20.890 align:middle line:84%
And by that, what I mean by that
is if I ever have A Kronecker

00:26:20.890 --> 00:26:27.410 align:middle line:84%
B acting on anything,
I'll just call it M,

00:26:27.410 --> 00:26:31.100 align:middle line:84%
I'm going to have
that mean A M--

00:26:31.100 --> 00:26:34.520 align:middle line:90%
sorry, BMA transpose.

00:26:34.520 --> 00:26:38.120 align:middle line:84%
So that would be my definition
of a Kronecker bracket

00:26:38.120 --> 00:26:41.480 align:middle line:90%
with a square input.

00:26:41.480 --> 00:26:45.530 align:middle line:84%
So if you want to have
the dx all on one side,

00:26:45.530 --> 00:26:47.090 align:middle line:90%
this is how I would do it.

00:26:47.090 --> 00:26:47.750 align:middle line:90%
Yes?

00:26:47.750 --> 00:26:49.940 align:middle line:84%
AUDIENCE: Wouldn't it be
vec dx in the third one,

00:26:49.940 --> 00:26:50.690 align:middle line:90%
or the second one?

00:26:50.690 --> 00:26:51.690 align:middle line:90%
ALAN EDELMAN: vec of dx.

00:26:51.690 --> 00:26:52.310 align:middle line:90%
Thank you.

00:26:52.310 --> 00:26:54.350 align:middle line:90%
You are quite right.

00:26:54.350 --> 00:26:54.860 align:middle line:90%
Good.

00:26:54.860 --> 00:26:56.070 align:middle line:90%
Thanks for the catch.

00:26:56.070 --> 00:26:56.570 align:middle line:90%
Perfect.

00:26:56.570 --> 00:27:02.470 align:middle line:90%


00:27:02.470 --> 00:27:04.570 align:middle line:84%
So that's basically
what I'm writing here

00:27:04.570 --> 00:27:06.070 align:middle line:84%
in this part of the
notes over here.

00:27:06.070 --> 00:27:08.750 align:middle line:90%


00:27:08.750 --> 00:27:11.540 align:middle line:90%
And again, this is--

00:27:11.540 --> 00:27:14.210 align:middle line:84%
I often don't even put
the square bracket.

00:27:14.210 --> 00:27:17.780 align:middle line:84%
So maybe it's not a bad
idea to avoid confusion.

00:27:17.780 --> 00:27:20.952 align:middle line:84%
So following
Stephen's notation, it

00:27:20.952 --> 00:27:22.910 align:middle line:84%
would be like-- this
would be the way to do it,

00:27:22.910 --> 00:27:27.758 align:middle line:84%
as this Kronecker
operator acting on dx.

00:27:27.758 --> 00:27:30.330 align:middle line:84%
I find that there's
never any real confusion.

00:27:30.330 --> 00:27:34.580 align:middle line:84%
And so I don't write the vec,
I don't put a square bracket.

00:27:34.580 --> 00:27:35.805 align:middle line:90%
I just do this.

00:27:35.805 --> 00:27:37.430 align:middle line:84%
I think I'm the only
one who does this,

00:27:37.430 --> 00:27:40.040 align:middle line:84%
so there's no reason why
you should follow me.

00:27:40.040 --> 00:27:43.070 align:middle line:84%
But I find it quite nice
to just write it this way

00:27:43.070 --> 00:27:44.090 align:middle line:90%
and drop the vec.

00:27:44.090 --> 00:27:46.040 align:middle line:84%
I think there's
never any confusion.

00:27:46.040 --> 00:27:50.700 align:middle line:84%
This is an operator acting on
the dx in just the way we say.

00:27:50.700 --> 00:27:52.540 align:middle line:84%
So you get to make
your own choice.

00:27:52.540 --> 00:27:53.820 align:middle line:90%
That's how I like to do it.

00:27:53.820 --> 00:27:57.420 align:middle line:90%


00:27:57.420 --> 00:28:03.990 align:middle line:84%
So just to show off a
little bit in Julia,

00:28:03.990 --> 00:28:08.070 align:middle line:84%
so we can get these
Jacobians in Julia using

00:28:08.070 --> 00:28:11.280 align:middle line:90%
the automatic differentiation.

00:28:11.280 --> 00:28:13.890 align:middle line:84%
And we're going to talk about
automatic differentiation.

00:28:13.890 --> 00:28:16.530 align:middle line:84%
I would imagine-- I don't know
if it's later this week, or--

00:28:16.530 --> 00:28:18.990 align:middle line:84%
when are we talking
about add, Stephen.

00:28:18.990 --> 00:28:19.800 align:middle line:90%
Do we know?

00:28:19.800 --> 00:28:21.570 align:middle line:84%
Later this week or
maybe next week?

00:28:21.570 --> 00:28:23.910 align:middle line:84%
STEPHEN: Yeah, I think
probably by Friday or something

00:28:23.910 --> 00:28:24.940 align:middle line:90%
like that.

00:28:24.940 --> 00:28:25.940 align:middle line:90%
ALAN EDELMAN: All right.

00:28:25.940 --> 00:28:28.990 align:middle line:84%
Maybe by Friday we'll talk
about automatic differentiation.

00:28:28.990 --> 00:28:32.400 align:middle line:84%
So the key introduction,
for those of you

00:28:32.400 --> 00:28:35.550 align:middle line:84%
who've never seen
AD, is that it's not

00:28:35.550 --> 00:28:37.980 align:middle line:84%
symbolic like
Mathematica and it's not

00:28:37.980 --> 00:28:40.350 align:middle line:90%
numerical finite differences.

00:28:40.350 --> 00:28:42.100 align:middle line:90%
It's none of those things.

00:28:42.100 --> 00:28:44.410 align:middle line:84%
It's something--
I mean, of course,

00:28:44.410 --> 00:28:47.520 align:middle line:84%
it's mathematically
equivalent, but it's--

00:28:47.520 --> 00:28:49.680 align:middle line:84%
when I first saw it, I
was kind of surprised

00:28:49.680 --> 00:28:53.120 align:middle line:90%
as to exactly how it works.

00:28:53.120 --> 00:28:55.600 align:middle line:90%
So you might be surprised, too.

00:28:55.600 --> 00:28:57.650 align:middle line:84%
But right now, we're
just going to be users.

00:28:57.650 --> 00:28:59.110 align:middle line:90%
I'm just going to grab--

00:28:59.110 --> 00:29:00.700 align:middle line:90%
here's that j matrix.

00:29:00.700 --> 00:29:03.370 align:middle line:84%
And I'm just going to
point out that if--

00:29:03.370 --> 00:29:06.010 align:middle line:84%
it'll compute the Jacobian
of the square function

00:29:06.010 --> 00:29:08.590 align:middle line:84%
at the point M
and we could check

00:29:08.590 --> 00:29:10.780 align:middle line:90%
that it gives the right answer.

00:29:10.780 --> 00:29:15.130 align:middle line:84%
And in fact, it'll work
with symbols, as well as--

00:29:15.130 --> 00:29:16.630 align:middle line:84%
so in some sense,
here we're getting

00:29:16.630 --> 00:29:21.220 align:middle line:84%
the Jacobian at every
point because the elements

00:29:21.220 --> 00:29:23.290 align:middle line:90%
are variables.

00:29:23.290 --> 00:29:27.010 align:middle line:84%
Let's do everything we
just did, but let's move up

00:29:27.010 --> 00:29:28.820 align:middle line:90%
to the matrix cube function.

00:29:28.820 --> 00:29:32.170 align:middle line:84%
So here's the matrix
cube function for 2

00:29:32.170 --> 00:29:36.020 align:middle line:84%
by 2 matrices written
out in all of its glory.

00:29:36.020 --> 00:29:39.340 align:middle line:84%
This is the thing that if
I did it on the blackboard,

00:29:39.340 --> 00:29:41.620 align:middle line:84%
just writing it out would
take a little too long.

00:29:41.620 --> 00:29:43.570 align:middle line:84%
But the computer
doesn't mind, right?

00:29:43.570 --> 00:29:46.930 align:middle line:84%
So there's the
matrix cube function.

00:29:46.930 --> 00:29:50.410 align:middle line:84%
I mean, the computer wouldn't
care if I did x to the fourth,

00:29:50.410 --> 00:29:51.460 align:middle line:90%
I assume.

00:29:51.460 --> 00:29:52.060 align:middle line:90%
Let's see.

00:29:52.060 --> 00:29:53.440 align:middle line:90%
At what point does it get hairy?

00:29:53.440 --> 00:29:54.730 align:middle line:90%
Maybe at the fifth?

00:29:54.730 --> 00:29:59.320 align:middle line:84%
It's kind of-- well, you'd have
to scroll, but there it is.

00:29:59.320 --> 00:30:01.990 align:middle line:90%
So let's do the cube.

00:30:01.990 --> 00:30:04.900 align:middle line:84%
So there's the
matrix cube function.

00:30:04.900 --> 00:30:09.190 align:middle line:90%
And we can get the Jacobian.

00:30:09.190 --> 00:30:12.172 align:middle line:84%
Again, it's just a matter
of-- just to remind you that

00:30:12.172 --> 00:30:12.880 align:middle line:90%
there's nothing--

00:30:12.880 --> 00:30:14.260 align:middle line:84%
I'm just letting the
computer do the work,

00:30:14.260 --> 00:30:15.400 align:middle line:90%
but you all could do it.

00:30:15.400 --> 00:30:17.380 align:middle line:84%
If I took the derivative
of this with respect to p,

00:30:17.380 --> 00:30:19.880 align:middle line:84%
you'd get this expression, the
derivative with respect to q,

00:30:19.880 --> 00:30:22.590 align:middle line:84%
you'd get rs plus 2pr
You can all do this.

00:30:22.590 --> 00:30:25.090 align:middle line:84%
Derivative with respect to s
and the derivative with respect

00:30:25.090 --> 00:30:27.910 align:middle line:84%
to t, and then you move down
and do the same for this one,

00:30:27.910 --> 00:30:29.680 align:middle line:84%
and then this one,
then this one.

00:30:29.680 --> 00:30:31.130 align:middle line:84%
But let the computer
do the work.

00:30:31.130 --> 00:30:34.060 align:middle line:84%
And so here's all
the 16 derivatives

00:30:34.060 --> 00:30:36.700 align:middle line:90%
right here in our faces.

00:30:36.700 --> 00:30:43.037 align:middle line:90%
And here-- let's see.

00:30:43.037 --> 00:30:45.120 align:middle line:84%
So these are just two
different ways to basically,

00:30:45.120 --> 00:30:48.840 align:middle line:90%
I guess, write the same thing.

00:30:48.840 --> 00:30:51.820 align:middle line:84%
Of course, we would like
to use the product rule.

00:30:51.820 --> 00:30:53.370 align:middle line:84%
So it would be dx
times x squared

00:30:53.370 --> 00:30:56.040 align:middle line:90%
plus xdx plus dxx squared.

00:30:56.040 --> 00:30:58.710 align:middle line:84%
So Stephen showed you that,
I think, the other day.

00:30:58.710 --> 00:31:01.073 align:middle line:84%
And this cannot be
simplified, really.

00:31:01.073 --> 00:31:02.490 align:middle line:84%
I mean, you could
change notation,

00:31:02.490 --> 00:31:05.100 align:middle line:84%
but you can't combine
any of these terms.

00:31:05.100 --> 00:31:07.050 align:middle line:84%
So of course, if
it was a scalar--

00:31:07.050 --> 00:31:09.000 align:middle line:84%
if everything was a
scalar, this would

00:31:09.000 --> 00:31:13.890 align:middle line:84%
amount to 3x squared dx,
which is the ordinary 1801

00:31:13.890 --> 00:31:14.850 align:middle line:90%
kind of answer.

00:31:14.850 --> 00:31:17.050 align:middle line:84%
But for matrices,
this is all you get.

00:31:17.050 --> 00:31:20.010 align:middle line:84%
And so if you want to
write this as a function--

00:31:20.010 --> 00:31:24.480 align:middle line:84%
as a linear function where
E is the dx and M is the x,

00:31:24.480 --> 00:31:25.620 align:middle line:90%
you see that--

00:31:25.620 --> 00:31:30.450 align:middle line:84%
so here I guess I'm just
using another notation where

00:31:30.450 --> 00:31:32.100 align:middle line:90%
E and M--

00:31:32.100 --> 00:31:34.290 align:middle line:90%
oops.

00:31:34.290 --> 00:31:37.770 align:middle line:90%
E and M for dx and x.

00:31:37.770 --> 00:31:39.970 align:middle line:84%
So I'm just using
these variables.

00:31:39.970 --> 00:31:42.490 align:middle line:90%
This is the linear function.

00:31:42.490 --> 00:31:47.680 align:middle line:84%
And again, sometimes it takes
a little getting used to.

00:31:47.680 --> 00:31:49.900 align:middle line:84%
You might look in
this and say, wait,

00:31:49.900 --> 00:31:51.620 align:middle line:90%
this is not a linear function.

00:31:51.620 --> 00:31:54.280 align:middle line:90%
I mean, I've got an M squared.

00:31:54.280 --> 00:31:56.500 align:middle line:84%
Or M times E to--
this is not linear.

00:31:56.500 --> 00:31:59.680 align:middle line:84%
But why do I get to tell you
that this is a linear function?

00:31:59.680 --> 00:32:01.690 align:middle line:84%
You've got squaring of
M. That's not linear.

00:32:01.690 --> 00:32:02.890 align:middle line:90%
Squaring is not linear.

00:32:02.890 --> 00:32:03.505 align:middle line:90%
What?

00:32:03.505 --> 00:32:04.630 align:middle line:90%
AUDIENCE: It's linear in E.

00:32:04.630 --> 00:32:06.790 align:middle line:84%
ALAN EDELMAN: It's
linear in E, bingo.

00:32:06.790 --> 00:32:08.620 align:middle line:84%
So this function is
linear in E, and that's

00:32:08.620 --> 00:32:10.330 align:middle line:90%
all that ever matters.

00:32:10.330 --> 00:32:13.510 align:middle line:84%
And I'll even remind you in
ordinary freshman calculus

00:32:13.510 --> 00:32:19.240 align:middle line:84%
that the derivative of x cubed,
3x squared, is not linear in x.

00:32:19.240 --> 00:32:22.120 align:middle line:84%
The derivative actually
is very nonlinear in x.

00:32:22.120 --> 00:32:23.680 align:middle line:90%
It's quadratic in x.

00:32:23.680 --> 00:32:28.870 align:middle line:84%
But what is linear is that
if you make a perturbation

00:32:28.870 --> 00:32:34.300 align:middle line:84%
E to the x, then the output
is perturbed by 3x squared

00:32:34.300 --> 00:32:37.660 align:middle line:90%
times E. And that is linear.

00:32:37.660 --> 00:32:40.840 align:middle line:84%
So this is what's going
on in matrix land.

00:32:40.840 --> 00:32:43.510 align:middle line:84%
Here's a numerical
finite difference.

00:32:43.510 --> 00:32:46.870 align:middle line:84%
Here is this function
applied to E. And as usual,

00:32:46.870 --> 00:32:50.230 align:middle line:84%
you see that to a
reasonable number of digits,

00:32:50.230 --> 00:32:51.910 align:middle line:90%
you get the same answer.

00:32:51.910 --> 00:32:54.670 align:middle line:84%
So this is Julia's function,
this linear function right

00:32:54.670 --> 00:32:58.300 align:middle line:84%
here, applied to
the perturbation E.

00:32:58.300 --> 00:33:01.130 align:middle line:90%
So E here is a dummy variable.

00:33:01.130 --> 00:33:03.100 align:middle line:84%
It's just the dummy
variable for the function.

00:33:03.100 --> 00:33:06.500 align:middle line:84%
And then here, E is the
numerical perturbation

00:33:06.500 --> 00:33:07.880 align:middle line:90%
that I'm making.

00:33:07.880 --> 00:33:10.610 align:middle line:90%
So E's having two roles here.

00:33:10.610 --> 00:33:14.912 align:middle line:84%
And we could check against
the symbolic answer,

00:33:14.912 --> 00:33:17.120 align:middle line:84%
and of course, I don't know
how easy it is to see it.

00:33:17.120 --> 00:33:18.110 align:middle line:90%
Maybe I'll just flatten up here.

00:33:18.110 --> 00:33:20.090 align:middle line:84%
But you could see that
these numbers that

00:33:20.090 --> 00:33:24.150 align:middle line:84%
are about to go off screen
here are exactly the same here.

00:33:24.150 --> 00:33:25.520 align:middle line:90%
So everything checks out.

00:33:25.520 --> 00:33:26.480 align:middle line:90%
It all works out.

00:33:26.480 --> 00:33:27.900 align:middle line:90%
You get the right answer.

00:33:27.900 --> 00:33:30.600 align:middle line:84%
So how do we do this with
the Kronecker notation?

00:33:30.600 --> 00:33:34.580 align:middle line:84%
Well, I think-- here, let's just
do this one on the blackboard

00:33:34.580 --> 00:33:38.088 align:middle line:90%
as well, just to do it.

00:33:38.088 --> 00:33:40.130 align:middle line:84%
So I'm just going to do
the very same thing here,

00:33:40.130 --> 00:33:41.255 align:middle line:90%
but I'll do it for x cubed.

00:33:41.255 --> 00:33:44.530 align:middle line:90%


00:33:44.530 --> 00:33:48.070 align:middle line:90%
Just to get the practice.

00:33:48.070 --> 00:33:51.520 align:middle line:84%
So the x cubed
version of what's here

00:33:51.520 --> 00:33:59.170 align:middle line:84%
is going to be x squared dx
plus xdxx plus dxx squared.

00:33:59.170 --> 00:34:01.810 align:middle line:90%


00:34:01.810 --> 00:34:06.080 align:middle line:84%
And I'm going to look
at this and say, aha,

00:34:06.080 --> 00:34:08.980 align:middle line:84%
I Kronecker x squared,
because x squared

00:34:08.980 --> 00:34:11.050 align:middle line:90%
is multiplying dx on the left.

00:34:11.050 --> 00:34:13.690 align:middle line:84%
Here, I'm going to
look at this and say,

00:34:13.690 --> 00:34:17.739 align:middle line:84%
there's an x and an x transpose
because x is on the left

00:34:17.739 --> 00:34:20.199 align:middle line:84%
and x is on the right,
but it comes around

00:34:20.199 --> 00:34:22.090 align:middle line:90%
and it gets transposed.

00:34:22.090 --> 00:34:24.429 align:middle line:84%
And this one, I'm
going to say, is

00:34:24.429 --> 00:34:28.790 align:middle line:84%
x transpose squared Kronecker
I because it's on the right.

00:34:28.790 --> 00:34:31.840 align:middle line:84%
And so whatever is on the right
here goes on the left here

00:34:31.840 --> 00:34:33.610 align:middle line:90%
and gets transposed.

00:34:33.610 --> 00:34:42.710 align:middle line:90%
And what I would write is this.

00:34:42.710 --> 00:34:45.739 align:middle line:84%
But if you want to put
in the square brackets

00:34:45.739 --> 00:34:50.179 align:middle line:84%
to emphasize this is an
operator, that's fine by me.

00:34:50.179 --> 00:34:52.340 align:middle line:84%
Or you could, of course,
put everything with vecs

00:34:52.340 --> 00:34:54.860 align:middle line:90%
if you want to as well.

00:34:54.860 --> 00:34:57.515 align:middle line:84%
So your choices are
you can write this.

00:34:57.515 --> 00:34:59.160 align:middle line:90%
This is the operator notation.

00:34:59.160 --> 00:35:02.380 align:middle line:90%


00:35:02.380 --> 00:35:08.290 align:middle line:84%
You could just write
the same thing over here

00:35:08.290 --> 00:35:14.680 align:middle line:84%
with a dx, which is that
this is Stephen's notation.

00:35:14.680 --> 00:35:18.610 align:middle line:90%


00:35:18.610 --> 00:35:21.220 align:middle line:90%
I guess this is Alan's notation.

00:35:21.220 --> 00:35:23.950 align:middle line:90%
So you get to pick.

00:35:23.950 --> 00:35:27.460 align:middle line:84%
And then finally most
people's notation, which

00:35:27.460 --> 00:35:31.870 align:middle line:84%
I think is horrible, is
that the vec of the x cubed

00:35:31.870 --> 00:35:35.170 align:middle line:84%
is equal to all this
times the vec of the x.

00:35:35.170 --> 00:35:38.215 align:middle line:84%
So this is the most
standard notation.

00:35:38.215 --> 00:35:42.970 align:middle line:90%


00:35:42.970 --> 00:35:44.810 align:middle line:90%
So you get to take your pick.

00:35:44.810 --> 00:35:46.600 align:middle line:90%
There's no one right answer.

00:35:46.600 --> 00:35:47.540 align:middle line:90%
Yeah?

00:35:47.540 --> 00:35:49.790 align:middle line:84%
AUDIENCE: Maybe you have
[INAUDIBLE] looked in detail,

00:35:49.790 --> 00:35:52.910 align:middle line:84%
but it seems like further
up in the presentation,

00:35:52.910 --> 00:35:57.090 align:middle line:84%
you have dx times x
squared plus x dx x

00:35:57.090 --> 00:36:00.830 align:middle line:84%
plus dx squared instead of
having the x squared out in

00:36:00.830 --> 00:36:02.057 align:middle line:90%
[INAUDIBLE].

00:36:02.057 --> 00:36:03.140 align:middle line:90%
ALAN EDELMAN: Is it just--

00:36:03.140 --> 00:36:05.720 align:middle line:84%
did I write it in
some other order?

00:36:05.720 --> 00:36:07.737 align:middle line:84%
I mean, 1 plus 2 is 2
plus 1 kind of thing.

00:36:07.737 --> 00:36:09.320 align:middle line:84%
Is that what we're
talking about here?

00:36:09.320 --> 00:36:11.750 align:middle line:84%
Yeah, I've got the M
squared on the-- yeah.

00:36:11.750 --> 00:36:13.730 align:middle line:90%
If you look at this one--

00:36:13.730 --> 00:36:15.823 align:middle line:90%
oh yeah.

00:36:15.823 --> 00:36:17.240 align:middle line:84%
Yeah, that doesn't
matter, though.

00:36:17.240 --> 00:36:18.230 align:middle line:90%
Yeah.

00:36:18.230 --> 00:36:20.480 align:middle line:90%
Yeah.

00:36:20.480 --> 00:36:22.880 align:middle line:90%
Or even up here.

00:36:22.880 --> 00:36:23.780 align:middle line:90%
Yeah.

00:36:23.780 --> 00:36:27.180 align:middle line:90%
But of course--

00:36:27.180 --> 00:36:29.870 align:middle line:84%
STEPHEN: So you have E
times M times M twice.

00:36:29.870 --> 00:36:32.150 align:middle line:84%
Is one of those supposed
to be M times M times E?

00:36:32.150 --> 00:36:35.087 align:middle line:84%
ALAN EDELMAN: Oh,
how did that happen?

00:36:35.087 --> 00:36:36.545 align:middle line:84%
Oh, is that what
you were noticing?

00:36:36.545 --> 00:36:36.920 align:middle line:90%
AUDIENCE: Yeah.

00:36:36.920 --> 00:36:37.628 align:middle line:90%
ALAN EDELMAN: Oh.

00:36:37.628 --> 00:36:38.690 align:middle line:90%
That's not good news.

00:36:38.690 --> 00:36:42.280 align:middle line:90%
How did I get the right answer?

00:36:42.280 --> 00:36:46.060 align:middle line:84%
Oh, I happened to pick an E
that commuted with M, which

00:36:46.060 --> 00:36:48.670 align:middle line:90%
is probably not a good check.

00:36:48.670 --> 00:36:49.660 align:middle line:90%
Oh my gosh.

00:36:49.660 --> 00:36:51.430 align:middle line:84%
That's exactly why I
got the right answer.

00:36:51.430 --> 00:36:53.860 align:middle line:84%
Don't ever take--
so I didn't really

00:36:53.860 --> 00:36:56.470 align:middle line:90%
have a good general test.

00:36:56.470 --> 00:36:57.140 align:middle line:90%
I see.

00:36:57.140 --> 00:36:59.230 align:middle line:90%
So let's--

00:36:59.230 --> 00:37:01.783 align:middle line:84%
STEPHEN: Another one
below, it looks like.

00:37:01.783 --> 00:37:03.950 align:middle line:84%
ALAN EDELMAN: Yeah, I
probably copied and pasted it.

00:37:03.950 --> 00:37:04.950 align:middle line:90%
You're right.

00:37:04.950 --> 00:37:06.760 align:middle line:90%
Oh, good catch.

00:37:06.760 --> 00:37:08.350 align:middle line:90%
Thank you.

00:37:08.350 --> 00:37:10.660 align:middle line:90%
Why did I do that?

00:37:10.660 --> 00:37:12.450 align:middle line:84%
So yes, we should
fix the thing online,

00:37:12.450 --> 00:37:13.840 align:middle line:90%
not just what I'm doing here.

00:37:13.840 --> 00:37:15.960 align:middle line:90%
So where did I define E?

00:37:15.960 --> 00:37:22.720 align:middle line:84%
Or let's-- did E get
defined earlier and earlier?

00:37:22.720 --> 00:37:27.310 align:middle line:90%
There's a way in Pluto to--

00:37:27.310 --> 00:37:29.620 align:middle line:84%
if you go below
the E and I think

00:37:29.620 --> 00:37:31.690 align:middle line:90%
you hit Command or option--

00:37:31.690 --> 00:37:32.350 align:middle line:90%
oh.

00:37:32.350 --> 00:37:35.830 align:middle line:84%
Command click to jump
to the definition.

00:37:35.830 --> 00:37:38.150 align:middle line:84%
So let's see, if I command
click, there's my--

00:37:38.150 --> 00:37:38.650 align:middle line:90%
OK.

00:37:38.650 --> 00:37:39.850 align:middle line:90%
So this is where--

00:37:39.850 --> 00:37:41.110 align:middle line:90%
yeah.

00:37:41.110 --> 00:37:43.580 align:middle line:84%
Because E commuted with
M, I got away with murder.

00:37:43.580 --> 00:37:45.500 align:middle line:90%
So let's do the following.

00:37:45.500 --> 00:37:46.960 align:middle line:90%
Let's not do that.

00:37:46.960 --> 00:37:50.770 align:middle line:84%
Let's go round of
2 divided by 1--

00:37:50.770 --> 00:37:51.980 align:middle line:90%
10,000 or something.

00:37:51.980 --> 00:37:54.760 align:middle line:90%
So here's a perturbation.

00:37:54.760 --> 00:37:58.150 align:middle line:84%
And I'll point out
that with this one,

00:37:58.150 --> 00:37:59.770 align:middle line:84%
I really did have
the wrong answer,

00:37:59.770 --> 00:38:01.580 align:middle line:90%
and thank you for catching that.

00:38:01.580 --> 00:38:05.780 align:middle line:90%
So let's see.

00:38:05.780 --> 00:38:09.920 align:middle line:84%
So where were before we
had all the wrong answers?

00:38:09.920 --> 00:38:11.170 align:middle line:90%
So here.

00:38:11.170 --> 00:38:13.690 align:middle line:90%


00:38:13.690 --> 00:38:18.320 align:middle line:90%
So if I did this--

00:38:18.320 --> 00:38:20.070 align:middle line:84%
so actually, it doesn't
matter right here.

00:38:20.070 --> 00:38:22.272 align:middle line:84%
But if I did it
over here, so you

00:38:22.272 --> 00:38:23.730 align:middle line:84%
see we're getting
the right answer.

00:38:23.730 --> 00:38:29.430 align:middle line:84%
Now but if I put back what
I had at the beginning,

00:38:29.430 --> 00:38:31.710 align:middle line:84%
you'd see I'd have
the wrong answer.

00:38:31.710 --> 00:38:34.400 align:middle line:84%
So it really did matter
and I'm glad you found it.

00:38:34.400 --> 00:38:38.010 align:middle line:90%


00:38:38.010 --> 00:38:40.905 align:middle line:90%
Nobody found that last year.

00:38:40.905 --> 00:38:42.905 align:middle line:84%
And I think I even showed
this in another class.

00:38:42.905 --> 00:38:45.620 align:middle line:90%


00:38:45.620 --> 00:38:47.670 align:middle line:84%
But I had a notebook that
I showed for about four

00:38:47.670 --> 00:38:49.760 align:middle line:84%
or five years until
somebody found the mistake,

00:38:49.760 --> 00:38:51.920 align:middle line:90%
so we're doing better.

00:38:51.920 --> 00:38:53.240 align:middle line:90%
OK, good.

00:38:53.240 --> 00:38:57.890 align:middle line:90%
Any other questions about--

00:38:57.890 --> 00:38:58.390 align:middle line:90%
all right.

00:38:58.390 --> 00:39:00.880 align:middle line:84%
So let's move on
to something that

00:39:00.880 --> 00:39:04.960 align:middle line:84%
might seem a little bit
even more interesting,

00:39:04.960 --> 00:39:07.990 align:middle line:84%
something that I like to
think of as a matrix function,

00:39:07.990 --> 00:39:10.480 align:middle line:84%
but however you want to
think about it is fine,

00:39:10.480 --> 00:39:13.480 align:middle line:90%
the LU decomposition.

00:39:13.480 --> 00:39:15.610 align:middle line:84%
So let's think about
this for a minute.

00:39:15.610 --> 00:39:20.490 align:middle line:84%
Does everybody know what LU
even means, just to make sure?

00:39:20.490 --> 00:39:28.000 align:middle line:84%
So the LU decomposition
says that a matrix,

00:39:28.000 --> 00:39:31.480 align:middle line:84%
which, by the way, it
doesn't always exist,

00:39:31.480 --> 00:39:33.400 align:middle line:90%
but it often exists.

00:39:33.400 --> 00:39:39.010 align:middle line:84%
A matrix A can be written as
a unit lower triangular, which

00:39:39.010 --> 00:39:42.530 align:middle line:84%
means one's on the
diagonal, times

00:39:42.530 --> 00:39:47.000 align:middle line:84%
an upper triangular matrix, no
assumptions about the diagonal.

00:39:47.000 --> 00:39:49.820 align:middle line:84%
And so this is called the LU
decomposition of a matrix.

00:39:49.820 --> 00:39:52.610 align:middle line:90%
And you've all seen it?

00:39:52.610 --> 00:39:54.430 align:middle line:90%
You've all heard the word LU?

00:39:54.430 --> 00:39:57.380 align:middle line:84%
It's equivalent to
Gaussian elimination.

00:39:57.380 --> 00:40:01.010 align:middle line:84%
And one thing that's
worth pointing out

00:40:01.010 --> 00:40:04.550 align:middle line:84%
is that if a is
an n by n matrix,

00:40:04.550 --> 00:40:08.300 align:middle line:84%
so it has n squared
elements, if you look

00:40:08.300 --> 00:40:11.000 align:middle line:84%
at the amount of
memory you need,

00:40:11.000 --> 00:40:15.095 align:middle line:84%
the number of numbers that
you have to write out L

00:40:15.095 --> 00:40:17.630 align:middle line:90%
as a matrix, what you need is--

00:40:17.630 --> 00:40:18.590 align:middle line:90%
well, let's see.

00:40:18.590 --> 00:40:21.125 align:middle line:84%
How many numbers are there
sitting inside of a--

00:40:21.125 --> 00:40:23.630 align:middle line:84%
I mean, we don't care
about the zeroes.

00:40:23.630 --> 00:40:25.220 align:middle line:84%
Numbers with
information in them.

00:40:25.220 --> 00:40:28.295 align:middle line:84%
How many numbers are in an
upper triangular n by n?

00:40:28.295 --> 00:40:31.860 align:middle line:90%


00:40:31.860 --> 00:40:35.890 align:middle line:84%
n times n plus 1
over 2, exactly.

00:40:35.890 --> 00:40:38.120 align:middle line:84%
And then you don't even
have to store the ones.

00:40:38.120 --> 00:40:42.490 align:middle line:84%
In fact, all the
linear algebra routines

00:40:42.490 --> 00:40:45.635 align:middle line:84%
like LAPACK, which is what it's
underneath Julia, or Python,

00:40:45.635 --> 00:40:47.260 align:middle line:84%
or MATLAB, or whatever
you like to use.

00:40:47.260 --> 00:40:50.800 align:middle line:84%
Everybody's using the same
LAPACK software under the hood.

00:40:50.800 --> 00:40:53.380 align:middle line:84%
They don't even store the ones
because you don't have to.

00:40:53.380 --> 00:40:56.800 align:middle line:84%
I mean, why waste memory that
you implicitly know it's a one.

00:40:56.800 --> 00:40:57.910 align:middle line:90%
So what are the numbers?

00:40:57.910 --> 00:41:02.080 align:middle line:84%
How many are there here
that has information?

00:41:02.080 --> 00:41:05.100 align:middle line:90%
How many numbers on the outside?

00:41:05.100 --> 00:41:06.047 align:middle line:90%
Sorry?

00:41:06.047 --> 00:41:07.360 align:middle line:90%
AUDIENCE: Same one but a minus.

00:41:07.360 --> 00:41:08.733 align:middle line:90%
ALAN EDELMAN: Same, but?

00:41:08.733 --> 00:41:09.650 align:middle line:90%
AUDIENCE: But a minus.

00:41:09.650 --> 00:41:10.640 align:middle line:90%
ALAN EDELMAN: But a minus.

00:41:10.640 --> 00:41:12.015 align:middle line:84%
Oh, I like the
way you said that.

00:41:12.015 --> 00:41:13.270 align:middle line:90%
The same, but a minus.

00:41:13.270 --> 00:41:15.350 align:middle line:90%
OK, n times n minus 1 over 2.

00:41:15.350 --> 00:41:15.860 align:middle line:90%
Good.

00:41:15.860 --> 00:41:18.630 align:middle line:84%
And if you add those
two numbers up--

00:41:18.630 --> 00:41:19.670 align:middle line:90%
so if you--

00:41:19.670 --> 00:41:22.760 align:middle line:84%
I mean, you can do it
visually, if you like.

00:41:22.760 --> 00:41:26.370 align:middle line:90%
You could add them up.

00:41:26.370 --> 00:41:29.090 align:middle line:90%
This one's below the ones.

00:41:29.090 --> 00:41:30.520 align:middle line:90%
What do you get?

00:41:30.520 --> 00:41:32.980 align:middle line:84%
You can add it visually or
just do it algebraically,

00:41:32.980 --> 00:41:34.980 align:middle line:90%
whatever works for you.

00:41:34.980 --> 00:41:35.750 align:middle line:90%
What do you get?

00:41:35.750 --> 00:41:36.370 align:middle line:90%
AUDIENCE: n squared.

00:41:36.370 --> 00:41:37.400 align:middle line:90%
ALAN EDELMAN: n squared.

00:41:37.400 --> 00:41:41.260 align:middle line:84%
So I don't know if you ever
stopped to think about this,

00:41:41.260 --> 00:41:43.690 align:middle line:84%
but the LU decomposition
is nothing other

00:41:43.690 --> 00:41:46.390 align:middle line:84%
than a function from
n squared variables

00:41:46.390 --> 00:41:48.160 align:middle line:90%
to n squared variables.

00:41:48.160 --> 00:41:50.620 align:middle line:84%
It's just a function
from n squared variables

00:41:50.620 --> 00:41:51.640 align:middle line:90%
to n squared variables.

00:41:51.640 --> 00:41:54.070 align:middle line:84%
And so therefore,
it has a Jacobian.

00:41:54.070 --> 00:41:56.440 align:middle line:84%
It has a Jacobian that, if
you write it as a matrix,

00:41:56.440 --> 00:41:58.360 align:middle line:90%
will be n squared by n squared.

00:41:58.360 --> 00:42:00.190 align:middle line:84%
Just as much as
the matrix square,

00:42:00.190 --> 00:42:02.560 align:middle line:84%
or matrix cube, or the
matrix inverse function

00:42:02.560 --> 00:42:05.210 align:middle line:84%
has a Jacobian that's
n squared by n squared,

00:42:05.210 --> 00:42:07.100 align:middle line:90%
so does the LU decomposition.

00:42:07.100 --> 00:42:13.410 align:middle line:84%
And in fact, here's
the 2 by 2 version.

00:42:13.410 --> 00:42:15.840 align:middle line:84%
It gets more complicated
when you go above 2 by 2.

00:42:15.840 --> 00:42:18.900 align:middle line:90%
But here are L and U explicitly.

00:42:18.900 --> 00:42:22.560 align:middle line:84%
So there's the-- so 2 by 2
matrix has four elements,

00:42:22.560 --> 00:42:24.630 align:middle line:84%
and there's this
one element in L

00:42:24.630 --> 00:42:26.940 align:middle line:84%
and these three
elements in U. And we

00:42:26.940 --> 00:42:28.980 align:middle line:84%
could think of the
LU decomposition

00:42:28.980 --> 00:42:32.460 align:middle line:84%
as the function from
the four elements

00:42:32.460 --> 00:42:36.990 align:middle line:84%
pqrs to these four numbers,
the 1 in L and the 3 in U.

00:42:36.990 --> 00:42:39.090 align:middle line:84%
So here's the LU
decomposition in Julia,

00:42:39.090 --> 00:42:44.070 align:middle line:84%
just checking that
L times U is pqrs.

00:42:44.070 --> 00:42:49.830 align:middle line:84%
And so here are the four
entries, the L entry over here

00:42:49.830 --> 00:42:52.140 align:middle line:84%
and then the three U entries
I'm just writing out.

00:42:52.140 --> 00:42:54.870 align:middle line:84%
And of course,
Julia would happily

00:42:54.870 --> 00:42:56.400 align:middle line:84%
take the Jacobian
of that function

00:42:56.400 --> 00:42:59.220 align:middle line:84%
and give you the 4
by 4 matrix Jacobian.

00:42:59.220 --> 00:43:02.110 align:middle line:84%
And so I hope everybody
remembers how this works.

00:43:02.110 --> 00:43:03.180 align:middle line:90%
I've said it a few times.

00:43:03.180 --> 00:43:04.770 align:middle line:84%
You take the first
one, q over p,

00:43:04.770 --> 00:43:06.240 align:middle line:84%
and then take
respect to p, that's

00:43:06.240 --> 00:43:09.060 align:middle line:84%
minus q over p squared. p
respect to q would be 1 over p.

00:43:09.060 --> 00:43:13.600 align:middle line:84%
And then there's no r
or s here, and you can--

00:43:13.600 --> 00:43:15.830 align:middle line:84%
and then you can do the
same for the second, third,

00:43:15.830 --> 00:43:16.538 align:middle line:90%
and fourth entry.

00:43:16.538 --> 00:43:21.670 align:middle line:90%


00:43:21.670 --> 00:43:25.510 align:middle line:84%
So in fact, I'm writing
this as an exercise,

00:43:25.510 --> 00:43:27.760 align:middle line:84%
but hopefully you
can all see this,

00:43:27.760 --> 00:43:29.518 align:middle line:84%
you can just do this
with the product rule

00:43:29.518 --> 00:43:30.560 align:middle line:90%
and get the right answer.

00:43:30.560 --> 00:43:34.377 align:middle line:84%
So dLU is going to
be dLU plus LdU.

00:43:34.377 --> 00:43:35.710 align:middle line:90%
And did I write the answer here?

00:43:35.710 --> 00:43:36.670 align:middle line:90%
No.

00:43:36.670 --> 00:43:38.300 align:middle line:90%
I'll have you guys shout it out.

00:43:38.300 --> 00:43:41.890 align:middle line:84%
So let's see if you're getting
the hang of this Kronecker

00:43:41.890 --> 00:43:43.630 align:middle line:90%
product notation.

00:43:43.630 --> 00:43:47.290 align:middle line:84%
And so the product
rule says that dLU

00:43:47.290 --> 00:43:51.520 align:middle line:90%
is going to equal LdU plus dLU.

00:43:51.520 --> 00:43:52.900 align:middle line:90%
Or did I write it the other way?

00:43:52.900 --> 00:43:54.558 align:middle line:90%
Not that it matters.

00:43:54.558 --> 00:43:55.600 align:middle line:90%
I wrote it the other way.

00:43:55.600 --> 00:43:56.683 align:middle line:90%
It doesn't matter, though.

00:43:56.683 --> 00:43:58.120 align:middle line:90%
Plus dLU.

00:43:58.120 --> 00:43:59.710 align:middle line:84%
The thing that does
matter is the L

00:43:59.710 --> 00:44:01.450 align:middle line:84%
is always to the
left of the U. It

00:44:01.450 --> 00:44:04.090 align:middle line:84%
doesn't matter which term
you write first, though.

00:44:04.090 --> 00:44:06.250 align:middle line:90%
So anybody getting good at this?

00:44:06.250 --> 00:44:07.555 align:middle line:90%
What Kronecker product is this?

00:44:07.555 --> 00:44:10.140 align:middle line:90%


00:44:10.140 --> 00:44:12.870 align:middle line:84%
There's an I and
an L or something.

00:44:12.870 --> 00:44:18.340 align:middle line:84%
This one will have an I and a U.
You're getting good at it yet?

00:44:18.340 --> 00:44:20.865 align:middle line:84%
This one's I Kronecker
L. And then this one?

00:44:20.865 --> 00:44:24.540 align:middle line:90%


00:44:24.540 --> 00:44:25.390 align:middle line:90%
What goes here?

00:44:25.390 --> 00:44:27.980 align:middle line:90%
What goes here?

00:44:27.980 --> 00:44:30.110 align:middle line:84%
U transpose
Kronecker I. Exactly.

00:44:30.110 --> 00:44:31.430 align:middle line:90%
You guys are good.

00:44:31.430 --> 00:44:33.930 align:middle line:84%
So you could write
this like this.

00:44:33.930 --> 00:44:36.110 align:middle line:84%
And that, I think, is
a perfectly good way

00:44:36.110 --> 00:44:38.780 align:middle line:90%
to write down the Jacobian.

00:44:38.780 --> 00:44:40.358 align:middle line:84%
Of course, this
is in matrix form

00:44:40.358 --> 00:44:41.900 align:middle line:84%
and it's not in
separate form, but it

00:44:41.900 --> 00:44:43.010 align:middle line:90%
has all the right information.

00:44:43.010 --> 00:44:43.550 align:middle line:90%
Yes?

00:44:43.550 --> 00:44:46.610 align:middle line:90%
AUDIENCE: [INAUDIBLE]?

00:44:46.610 --> 00:44:48.710 align:middle line:84%
ALAN EDELMAN: Yeah, that
gets a little tricky.

00:44:48.710 --> 00:44:51.930 align:middle line:84%
But yeah, that does
get a little tricky.

00:44:51.930 --> 00:44:54.980 align:middle line:90%
So I'm just going to--

00:44:54.980 --> 00:44:58.590 align:middle line:84%
I'm just going to
say that it is--

00:44:58.590 --> 00:45:01.070 align:middle line:84%
so the underlying
vector space is--

00:45:01.070 --> 00:45:03.890 align:middle line:90%
yeah, this is a little tricky.

00:45:03.890 --> 00:45:12.200 align:middle line:84%
So I like to think of the
DL as 0 0 0 and then dL21.

00:45:12.200 --> 00:45:14.807 align:middle line:90%
After all, the ones--

00:45:14.807 --> 00:45:16.890 align:middle line:84%
the derivative of the ones
on the diagonal is zero

00:45:16.890 --> 00:45:18.940 align:middle line:84%
and then everything
else is zero anyway.

00:45:18.940 --> 00:45:28.180 align:middle line:84%
And the dU is 0
dU11 dU12 and dU22.

00:45:28.180 --> 00:45:31.410 align:middle line:84%
So in some sense,
if you really wanted

00:45:31.410 --> 00:45:33.690 align:middle line:84%
to do it the standard
notation way, the one that

00:45:33.690 --> 00:45:37.780 align:middle line:84%
would be like this, you
would have to write down--

00:45:37.780 --> 00:45:41.370 align:middle line:84%
you would have to write down
something that worked with dL21

00:45:41.370 --> 00:45:45.840 align:middle line:90%
and dU11, dU12, and dU22.

00:45:45.840 --> 00:45:53.390 align:middle line:84%
But again, I feel like that's
just following the crowd.

00:45:53.390 --> 00:45:56.090 align:middle line:84%
Me, I would rather just
say that I understand

00:45:56.090 --> 00:45:59.270 align:middle line:84%
that my vector space
consists of lower triangular

00:45:59.270 --> 00:46:03.050 align:middle line:84%
perturbations to L and upper
triangular perturbations to U.

00:46:03.050 --> 00:46:04.880 align:middle line:84%
And so let's just
look at it that way.

00:46:04.880 --> 00:46:07.520 align:middle line:90%


00:46:07.520 --> 00:46:09.620 align:middle line:90%
And that's how I like to see it.

00:46:09.620 --> 00:46:13.800 align:middle line:84%
But maybe that's-- maybe I'm
pushing the notation too far.

00:46:13.800 --> 00:46:15.980 align:middle line:90%
Maybe you would prefer--

00:46:15.980 --> 00:46:18.450 align:middle line:84%
maybe for the purposes
of this class,

00:46:18.450 --> 00:46:22.990 align:middle line:84%
I would be better off
doing it like this.

00:46:22.990 --> 00:46:25.180 align:middle line:84%
But even this is
slightly-- yeah.

00:46:25.180 --> 00:46:27.370 align:middle line:84%
Maybe that would
be better for now.

00:46:27.370 --> 00:46:30.560 align:middle line:84%
But you see, I like to put all
my perturbations on the right.

00:46:30.560 --> 00:46:33.670 align:middle line:84%
So again, nobody's fully
standardized all this notation.

00:46:33.670 --> 00:46:35.170 align:middle line:84%
So once again, you
have your choice.

00:46:35.170 --> 00:46:38.950 align:middle line:90%


00:46:38.950 --> 00:46:41.350 align:middle line:84%
So here I'll show you just
a couple of more matrix

00:46:41.350 --> 00:46:43.432 align:middle line:90%
factorizations just quickly.

00:46:43.432 --> 00:46:45.140 align:middle line:84%
I'm not going to go
into too much detail,

00:46:45.140 --> 00:46:47.530 align:middle line:84%
But I want you to just
see that you can still

00:46:47.530 --> 00:46:49.850 align:middle line:90%
do this for other situations.

00:46:49.850 --> 00:46:54.940 align:middle line:84%
So one favorite of mine is a
two-parameter example in a 2

00:46:54.940 --> 00:46:56.530 align:middle line:84%
by 2 matrices have
four parameters,

00:46:56.530 --> 00:46:58.030 align:middle line:84%
but there's only
two real parameters

00:46:58.030 --> 00:47:03.530 align:middle line:84%
here, which traceless
symmetric matrices.

00:47:03.530 --> 00:47:06.790 align:middle line:84%
So the fact that the 2
by 2 matrix is symmetric

00:47:06.790 --> 00:47:08.950 align:middle line:84%
means you have the s
over here and over here.

00:47:08.950 --> 00:47:13.120 align:middle line:84%
And the fact that it's traceless
says that the trace is zero.

00:47:13.120 --> 00:47:15.650 align:middle line:84%
So if you have a p over
here, you need a minus p

00:47:15.650 --> 00:47:18.880 align:middle line:84%
to make the sum of
the diagonals zero.

00:47:18.880 --> 00:47:21.190 align:middle line:84%
And so this is a
two-parameter family.

00:47:21.190 --> 00:47:24.670 align:middle line:84%
And this is, by the way,
the q that showed up on top.

00:47:24.670 --> 00:47:27.680 align:middle line:90%
So if you write down--

00:47:27.680 --> 00:47:29.260 align:middle line:84%
you all know that
symmetric matrices

00:47:29.260 --> 00:47:31.820 align:middle line:90%
have orthogonal eigenvectors.

00:47:31.820 --> 00:47:34.060 align:middle line:84%
So you can write it
with cosines and sines.

00:47:34.060 --> 00:47:35.800 align:middle line:84%
And as a little
bit of setup, I'm

00:47:35.800 --> 00:47:40.810 align:middle line:84%
using one half theta as
opposed to just using theta.

00:47:40.810 --> 00:47:43.390 align:middle line:90%
And here's my eigenvalues.

00:47:43.390 --> 00:47:46.762 align:middle line:84%
The eigenvalue
matrix-- the only thing

00:47:46.762 --> 00:47:48.970 align:middle line:84%
that we know about the
eigenvalue matrix for starters

00:47:48.970 --> 00:47:51.640 align:middle line:84%
is that the sum of the
eigenvalues are zero.

00:47:51.640 --> 00:47:55.390 align:middle line:84%
So let's call this the positive
1 and this is the negative 1.

00:47:55.390 --> 00:48:04.930 align:middle line:84%
And what we have to equate is
this is the eigendecomposition.

00:48:04.930 --> 00:48:08.120 align:middle line:84%
So these two are equal,
these two are opposite.

00:48:08.120 --> 00:48:09.910 align:middle line:90%
And so it all becomes--

00:48:09.910 --> 00:48:14.610 align:middle line:84%
let's see if we can
get this right here.

00:48:14.610 --> 00:48:15.420 align:middle line:90%
Yeah.

00:48:15.420 --> 00:48:18.380 align:middle line:90%
So this is the matrix.

00:48:18.380 --> 00:48:22.970 align:middle line:84%
And this is-- the matrix
has two parameters, p and s.

00:48:22.970 --> 00:48:25.760 align:middle line:84%
The eigendecomposition
has two parameters.

00:48:25.760 --> 00:48:27.590 align:middle line:84%
Maybe I should have
emphasized that.

00:48:27.590 --> 00:48:31.430 align:middle line:84%
So the eigenvectors have
one parameter, theta,

00:48:31.430 --> 00:48:35.840 align:middle line:84%
and the eigenvalue
has one parameter, r.

00:48:35.840 --> 00:48:39.050 align:middle line:84%
So the eigendecomposition
here, this little 2 by 2

00:48:39.050 --> 00:48:43.040 align:middle line:84%
eigendecomposition, is a
map from the two variables

00:48:43.040 --> 00:48:47.065 align:middle line:90%
ps to the two variables r theta.

00:48:47.065 --> 00:48:49.190 align:middle line:84%
And computing the eigenvalues
of this 2 by 2 matrix

00:48:49.190 --> 00:48:53.140 align:middle line:90%
is just a map from r2 to r2.

00:48:53.140 --> 00:48:55.750 align:middle line:90%
And we can get the Jacobian.

00:48:55.750 --> 00:48:58.300 align:middle line:84%
And here's the
Jacobian of that map.

00:48:58.300 --> 00:49:02.080 align:middle line:84%
So this is the actual 2
by 2 Jacobian of that.

00:49:02.080 --> 00:49:03.730 align:middle line:84%
And sometimes,
it's fun to compute

00:49:03.730 --> 00:49:06.670 align:middle line:84%
the Jacobian determinant,
which is r in this case.

00:49:06.670 --> 00:49:11.740 align:middle line:84%
And some of you might recognize
that this eigendecomposition

00:49:11.740 --> 00:49:15.370 align:middle line:84%
change of coordinates is exactly
the same mathematical change

00:49:15.370 --> 00:49:19.720 align:middle line:84%
of coordinates of going from
Cartesian to polar coordinates.

00:49:19.720 --> 00:49:24.010 align:middle line:84%
And this determinant,
R, is exactly the-- many

00:49:24.010 --> 00:49:26.650 align:middle line:84%
of you probably have
memorized that the area

00:49:26.650 --> 00:49:30.260 align:middle line:84%
element in Cartesian coordinates
dx dy is r dr d theta.

00:49:30.260 --> 00:49:32.590 align:middle line:84%
Well, here's-- we
just, in effect,

00:49:32.590 --> 00:49:35.800 align:middle line:90%
let Julia derive that for us.

00:49:35.800 --> 00:49:36.760 align:middle line:90%
But let's go further.

00:49:36.760 --> 00:49:39.580 align:middle line:84%
I'll do one more
example, the full 2 by 2

00:49:39.580 --> 00:49:40.720 align:middle line:90%
symmetric eigenproblem.

00:49:40.720 --> 00:49:45.140 align:middle line:84%
But any questions about this
restricted eigenproblem?

00:49:45.140 --> 00:49:48.830 align:middle line:84%
It's a map from r2 to r2
and it's exactly the map--

00:49:48.830 --> 00:49:51.470 align:middle line:84%
it's exactly the map-- that's
why I chose the theta over 2.

00:49:51.470 --> 00:49:55.670 align:middle line:84%
It's exactly the map from
Cartesian to polar coordinates

00:49:55.670 --> 00:49:56.660 align:middle line:90%
by--

00:49:56.660 --> 00:49:59.060 align:middle line:84%
I don't know whether you call
that a coincidence or what,

00:49:59.060 --> 00:50:02.150 align:middle line:84%
but I think it's intriguing
that it's the same map.

00:50:02.150 --> 00:50:04.100 align:middle line:90%
Let's do the full symmetric.

00:50:04.100 --> 00:50:09.200 align:middle line:84%
So in the full case,
a symmetric matrix

00:50:09.200 --> 00:50:11.360 align:middle line:84%
is a three-dimensional
vector space.

00:50:11.360 --> 00:50:15.920 align:middle line:84%
So you've got three parameters,
the two diagonals and then

00:50:15.920 --> 00:50:18.330 align:middle line:84%
the off-diagonals
are the same number.

00:50:18.330 --> 00:50:22.340 align:middle line:84%
So we can refer to every
2 by 2 symmetric matrix

00:50:22.340 --> 00:50:25.790 align:middle line:90%
with the variables p, r, and s.

00:50:25.790 --> 00:50:28.940 align:middle line:90%
And the three variables are--

00:50:28.940 --> 00:50:32.630 align:middle line:84%
since the symmetric matrix has
an orthogonal eigenvectors,

00:50:32.630 --> 00:50:35.610 align:middle line:84%
you still have one parameter,
theta, for the eigenvectors,

00:50:35.610 --> 00:50:38.030 align:middle line:84%
but we have two parameters,
lambda 1 and lambda 2,

00:50:38.030 --> 00:50:39.800 align:middle line:90%
for the eigenvalues.

00:50:39.800 --> 00:50:42.170 align:middle line:84%
And so the symmetric
eigenvalue problem

00:50:42.170 --> 00:50:46.733 align:middle line:84%
is, once again, mapping
this time from r3 to r3.

00:50:46.733 --> 00:50:48.150 align:middle line:84%
You can go backwards
and forwards.

00:50:48.150 --> 00:50:50.490 align:middle line:84%
You can go from the
eigenvalues and eigenvectors

00:50:50.490 --> 00:50:52.505 align:middle line:84%
to the matrix,
which, by the way,

00:50:52.505 --> 00:50:53.880 align:middle line:84%
is the more
convenient direction,

00:50:53.880 --> 00:50:56.820 align:middle line:84%
or you can go from
the matrix entries

00:50:56.820 --> 00:50:59.730 align:middle line:84%
to the eigenvalues
and eigenvectors.

00:50:59.730 --> 00:51:02.880 align:middle line:84%
So everybody in
first-year linear algebra

00:51:02.880 --> 00:51:04.215 align:middle line:90%
learns how to go this way.

00:51:04.215 --> 00:51:06.750 align:middle line:90%


00:51:06.750 --> 00:51:08.700 align:middle line:84%
You set up the
characteristic polynomial,

00:51:08.700 --> 00:51:11.880 align:middle line:84%
you write down the
quadratic, you solve it.

00:51:11.880 --> 00:51:14.370 align:middle line:84%
And in one way or another,
you can get the eigenvector

00:51:14.370 --> 00:51:16.060 align:middle line:90%
as well.

00:51:16.060 --> 00:51:19.710 align:middle line:84%
So if you look at the null
space of a minus lambda

00:51:19.710 --> 00:51:20.940 align:middle line:90%
I or something.

00:51:20.940 --> 00:51:22.980 align:middle line:84%
So this left
direction, you've all

00:51:22.980 --> 00:51:26.310 align:middle line:84%
done in your first-year
linear algebra class.

00:51:26.310 --> 00:51:29.820 align:middle line:84%
Turns out it's a little more
convenient to go the other way.

00:51:29.820 --> 00:51:32.460 align:middle line:84%
This is just a matrix multiply
where I set up the eigenvalues

00:51:32.460 --> 00:51:35.340 align:middle line:84%
and eigenvectors and then
I compute the matrix.

00:51:35.340 --> 00:51:39.670 align:middle line:84%
And so here is the
Jacobian calculation.

00:51:39.670 --> 00:51:41.310 align:middle line:90%
So it's a 3 by 3 Jacobian.

00:51:41.310 --> 00:51:43.050 align:middle line:90%
There it is in all of its glory.

00:51:43.050 --> 00:51:46.950 align:middle line:84%
A little bit painful to do by
hand, but completely doable.

00:51:46.950 --> 00:51:49.560 align:middle line:84%
Julia, of course, just
happily gives you the answer

00:51:49.560 --> 00:51:51.450 align:middle line:90%
without all the hard work.

00:51:51.450 --> 00:51:53.880 align:middle line:84%
And I'll just
mention that if you

00:51:53.880 --> 00:51:57.360 align:middle line:84%
calculate the determinant
of this matrix,

00:51:57.360 --> 00:52:00.750 align:middle line:90%
you get the volume element.

00:52:00.750 --> 00:52:02.920 align:middle line:84%
So just to make sure everybody
knows how that works,

00:52:02.920 --> 00:52:09.600 align:middle line:84%
the volume of the Jacobian is
how much a little cube scales.

00:52:09.600 --> 00:52:15.510 align:middle line:84%
If I go back to that picture
of the corgi being warped,

00:52:15.510 --> 00:52:19.200 align:middle line:84%
I could look at his nose
and then in the image,

00:52:19.200 --> 00:52:20.730 align:middle line:90%
maybe it gets squashed.

00:52:20.730 --> 00:52:23.830 align:middle line:84%
The ratio is the determinant
of the transformation.

00:52:23.830 --> 00:52:26.310 align:middle line:84%
It's actually determinant
of the Jacobian.

00:52:26.310 --> 00:52:30.000 align:middle line:84%
And if you do that, you actually
get lambda 1 minus lambda 2.

00:52:30.000 --> 00:52:33.480 align:middle line:84%
And people like to
say that symmetric--

00:52:33.480 --> 00:52:37.116 align:middle line:90%
the eigenvalues don't want to--

00:52:37.116 --> 00:52:40.170 align:middle line:84%
so if the determinant being
lambda 1 minus lambda 2

00:52:40.170 --> 00:52:42.480 align:middle line:84%
is an indication of repulsion
between eigenvalues,

00:52:42.480 --> 00:52:44.770 align:middle line:84%
but maybe that's
neither here nor there.

00:52:44.770 --> 00:52:45.270 align:middle line:90%
All right.

00:52:45.270 --> 00:52:47.820 align:middle line:84%
Well, I think my
timing is perfect.

00:52:47.820 --> 00:52:50.910 align:middle line:84%
I went through this notebook
and it's 12:01 exactly.

00:52:50.910 --> 00:52:53.700 align:middle line:84%
So should we take a five
minute break, and then Stephen,

00:52:53.700 --> 00:52:56.280 align:middle line:84%
you'll be on board for
the next set and stuff?

00:52:56.280 --> 00:52:57.250 align:middle line:90%
STEPHEN: That's right.

00:52:57.250 --> 00:52:59.650 align:middle line:84%
ALAN EDELMAN: All right,
so 12:06 come on back.

00:52:59.650 --> 00:53:01.740 align:middle line:90%
We'll start up again.

00:53:01.740 --> 00:53:05.000 align:middle line:90%