WEBVTT 00:00:00.000 --> 00:00:01.964 align:middle line:90% [SQUEAKING] 00:00:01.964 --> 00:00:03.437 align:middle line:90% [RUSTLING] 00:00:03.437 --> 00:00:05.401 align:middle line:90% [CLICKING] 00:00:05.401 --> 00:00:13.570 align:middle line:90% 00:00:13.570 --> 00:00:16.510 align:middle line:84% STEVEN JOHNSON: So I want to briefly start 00:00:16.510 --> 00:00:18.588 align:middle line:84% to talk about going beyond 18.02 derivatives. 00:00:18.588 --> 00:00:21.130 align:middle line:84% So we've already gone, I would say, beyond 18.02 in the sense 00:00:21.130 --> 00:00:22.900 align:middle line:90% that they never do the-- 00:00:22.900 --> 00:00:24.910 align:middle line:84% they have Jacobians maybe, but they never 00:00:24.910 --> 00:00:26.110 align:middle line:90% really do the chain rule. 00:00:26.110 --> 00:00:34.200 align:middle line:84% They never really write even this definition in-- 00:00:34.200 --> 00:00:35.160 align:middle line:90% where is it? 00:00:35.160 --> 00:00:39.960 align:middle line:84% Go back-- saying that the change in the function 00:00:39.960 --> 00:00:44.160 align:middle line:84% is the Jacobian times dX, which I view as almost 00:00:44.160 --> 00:00:45.865 align:middle line:90% the definition of the Jacobian. 00:00:45.865 --> 00:00:47.490 align:middle line:84% I think that they never really do that. 00:00:47.490 --> 00:00:49.950 align:middle line:84% They never really do, like, multi-dimensional 00:00:49.950 --> 00:00:51.820 align:middle line:84% Taylor expansion or something like that. 00:00:51.820 --> 00:00:55.050 align:middle line:84% But now what we want to do in this class 00:00:55.050 --> 00:00:58.710 align:middle line:84% is go really beyond 18.02 to have functions where the inputs 00:00:58.710 --> 00:01:04.500 align:middle line:84% and outputs are not just column vectors necessarily or scalars. 00:01:04.500 --> 00:01:16.360 align:middle line:84% But they could live in other vector spaces, 00:01:16.360 --> 00:01:18.880 align:middle line:84% in sort of more general vector spaces. 00:01:18.880 --> 00:01:29.650 align:middle line:90% 00:01:29.650 --> 00:01:33.590 align:middle line:90% So let me give an example. 00:01:33.590 --> 00:01:36.920 align:middle line:84% And you probably learned in 18.06 hopefully, remember, 00:01:36.920 --> 00:01:38.960 align:middle line:84% that a vector space is not just a column vector. 00:01:38.960 --> 00:01:40.790 align:middle line:84% It's really more abstractly anything 00:01:40.790 --> 00:01:43.797 align:middle line:84% you can add and subtract or multiply by scalars. 00:01:43.797 --> 00:01:45.380 align:middle line:84% And so there are other things that you 00:01:45.380 --> 00:01:47.540 align:middle line:90% can put in vector spaces-- 00:01:47.540 --> 00:01:49.070 align:middle line:84% you can think of as vector spaces. 00:01:49.070 --> 00:01:50.990 align:middle line:90% And so for example-- 00:01:50.990 --> 00:02:04.280 align:middle line:84% would be matrix inputs and/or outputs. 00:02:04.280 --> 00:02:04.780 align:middle line:90% OK? 00:02:04.780 --> 00:02:13.580 align:middle line:84% For example, suppose we have the function 00:02:13.580 --> 00:02:17.600 align:middle line:90% f that takes a matrix as input. 00:02:17.600 --> 00:02:26.130 align:middle line:84% And suppose it's an n by n matrix as input. 00:02:26.130 --> 00:02:28.190 align:middle line:90% So now there's n-squared inputs. 00:02:28.190 --> 00:02:30.170 align:middle line:84% It's an n-squared dimensional vector space, 00:02:30.170 --> 00:02:33.020 align:middle line:90% if I allow arbitrary matrices. 00:02:33.020 --> 00:02:36.260 align:middle line:84% And I could say, for example, f of A 00:02:36.260 --> 00:02:46.780 align:middle line:84% equals a inverse, right, or A-cubed, or even more 00:02:46.780 --> 00:02:47.840 align:middle line:90% complicated things. 00:02:47.840 --> 00:02:52.840 align:middle line:84% For example, I could say f of A equals-- 00:02:52.840 --> 00:02:54.490 align:middle line:90% you take a matrix as input. 00:02:54.490 --> 00:02:57.370 align:middle line:84% And you get the u, the upper triangular matrix 00:02:57.370 --> 00:02:58.720 align:middle line:90% from Gaussian elimination. 00:02:58.720 --> 00:03:04.800 align:middle line:90% 00:03:04.800 --> 00:03:05.300 align:middle line:90% Right? 00:03:05.300 --> 00:03:07.070 align:middle line:84% If you do Gaussian elimination, let's 00:03:07.070 --> 00:03:08.000 align:middle line:84% assume that then we have to restrict 00:03:08.000 --> 00:03:10.310 align:middle line:84% ourselves to maybe nonsingular matrices or something. 00:03:10.310 --> 00:03:11.840 align:middle line:90% You take a matrix in. 00:03:11.840 --> 00:03:15.106 align:middle line:84% And it gives you an upper triangular matrix out. 00:03:15.106 --> 00:03:16.900 align:middle line:90% OK? 00:03:16.900 --> 00:03:20.020 align:middle line:84% Or you could also have things-- these are things with matrix 00:03:20.020 --> 00:03:25.455 align:middle line:90% outputs and matrix inputs. 00:03:25.455 --> 00:03:31.880 align:middle line:90% 00:03:31.880 --> 00:03:39.540 align:middle line:84% You could also have scalar outputs, 00:03:39.540 --> 00:03:42.585 align:middle line:90% but you have matrix inputs. 00:03:42.585 --> 00:03:57.780 align:middle line:90% 00:03:57.780 --> 00:03:58.280 align:middle line:90% Right? 00:03:58.280 --> 00:04:00.480 align:middle line:84% And what kind of functions would those be? 00:04:00.480 --> 00:04:05.630 align:middle line:84% So for example, the determinant is a perfectly good function 00:04:05.630 --> 00:04:08.210 align:middle line:84% that takes a matrix in and gives you a scalar out-- 00:04:08.210 --> 00:04:09.350 align:middle line:90% or the trace. 00:04:09.350 --> 00:04:12.210 align:middle line:90% 00:04:12.210 --> 00:04:13.870 align:middle line:90% OK? 00:04:13.870 --> 00:04:17.620 align:middle line:84% Or you could have even more complicated things. 00:04:17.620 --> 00:04:25.205 align:middle line:84% We could have-- or the sigma 1 of A, the largest singular 00:04:25.205 --> 00:04:25.705 align:middle line:90% value. 00:04:25.705 --> 00:04:31.220 align:middle line:90% 00:04:31.220 --> 00:04:31.720 align:middle line:90% Right? 00:04:31.720 --> 00:04:32.607 align:middle line:90% You take a matrix in. 00:04:32.607 --> 00:04:33.940 align:middle line:90% You compute the singular values. 00:04:33.940 --> 00:04:35.200 align:middle line:90% Take the largest one. 00:04:35.200 --> 00:04:37.170 align:middle line:90% That's a number. 00:04:37.170 --> 00:04:42.010 align:middle line:84% That's a function that takes a matrix in and gives you 00:04:42.010 --> 00:04:45.465 align:middle line:90% a scalar out, right-- 00:04:45.465 --> 00:04:46.970 align:middle line:90% or many other examples. 00:04:46.970 --> 00:04:49.810 align:middle line:84% And so we're going to see how to take derivatives 00:04:49.810 --> 00:04:50.933 align:middle line:90% of things like this. 00:04:50.933 --> 00:04:52.600 align:middle line:84% We'd like to be able to take derivatives 00:04:52.600 --> 00:04:54.140 align:middle line:90% of things like this. 00:04:54.140 --> 00:04:55.970 align:middle line:84% And so let's do a couple of them. 00:04:55.970 --> 00:04:58.600 align:middle line:84% They're not that hard to actually analyze-- 00:04:58.600 --> 00:04:59.320 align:middle line:90% the simple ones. 00:04:59.320 --> 00:05:01.607 align:middle line:84% Determinant-- trace is actually pretty easy, 00:05:01.607 --> 00:05:02.440 align:middle line:90% but I won't do that. 00:05:02.440 --> 00:05:04.180 align:middle line:84% But determinant is a little bit more complicated. 00:05:04.180 --> 00:05:05.555 align:middle line:84% But Alan's going to-- we're going 00:05:05.555 --> 00:05:08.740 align:middle line:84% to show you some clever tricks for that pretty soon. 00:05:08.740 --> 00:05:13.000 align:middle line:84% And matrix factorizations, like SVD or Gaussian elimination, 00:05:13.000 --> 00:05:15.310 align:middle line:90% are also a little bit tricky. 00:05:15.310 --> 00:05:17.230 align:middle line:84% But once you see the trick, it's not so bad. 00:05:17.230 --> 00:05:19.600 align:middle line:84% But things like A inverse and A-cubed 00:05:19.600 --> 00:05:22.250 align:middle line:84% are actually pretty straightforward. 00:05:22.250 --> 00:05:34.360 align:middle line:84% So for example, let's do, f of A is A-cubed. 00:05:34.360 --> 00:05:35.440 align:middle line:90% Right? 00:05:35.440 --> 00:05:36.400 align:middle line:90% So if we do-- 00:05:36.400 --> 00:05:40.120 align:middle line:84% I think actually Alan already did f of A equals A-squared. 00:05:40.120 --> 00:05:41.770 align:middle line:90% But now we'll do a-cubed. 00:05:41.770 --> 00:05:42.280 align:middle line:90% Right? 00:05:42.280 --> 00:05:44.770 align:middle line:84% So now we can just do the product rule. 00:05:44.770 --> 00:05:49.008 align:middle line:90% 00:05:49.008 --> 00:05:54.110 align:middle line:90% Our df, right-- 00:05:54.110 --> 00:05:57.830 align:middle line:84% A-cubed is just A times A times A. 00:05:57.830 --> 00:06:02.050 align:middle line:84% And again, I'm going to assume these are square matrices. 00:06:02.050 --> 00:06:05.540 align:middle line:84% Otherwise, you couldn't multiply them by themselves. 00:06:05.540 --> 00:06:08.495 align:middle line:90% [INAUDIBLE] right, are blue. 00:06:08.495 --> 00:06:11.830 align:middle line:90% 00:06:11.830 --> 00:06:14.956 align:middle line:84% Should I put-- my matrix input is red. 00:06:14.956 --> 00:06:17.490 align:middle line:90% Right? 00:06:17.490 --> 00:06:19.770 align:middle line:90% And what's the product rule? 00:06:19.770 --> 00:06:22.120 align:middle line:90% Well, it's just three terms. 00:06:22.120 --> 00:06:30.950 align:middle line:84% I have a dA times A-squared plus A times dA times 00:06:30.950 --> 00:06:36.800 align:middle line:90% A plus A-squared times dA. 00:06:36.800 --> 00:06:37.773 align:middle line:90% Does everyone see that? 00:06:37.773 --> 00:06:40.190 align:middle line:84% So I can just take the d of this term, the d of that term, 00:06:40.190 --> 00:06:40.982 align:middle line:90% the d of this term. 00:06:40.982 --> 00:06:43.200 align:middle line:84% And A is no longer a constant matrix. 00:06:43.200 --> 00:06:45.590 align:middle line:90% A is not a fixed parameter. 00:06:45.590 --> 00:06:47.940 align:middle line:90% It's the input parameters. 00:06:47.940 --> 00:06:49.100 align:middle line:90% So I can-- remember? 00:06:49.100 --> 00:06:55.130 align:middle line:84% So all this is just saying, suppose I take my function f, 00:06:55.130 --> 00:07:01.150 align:middle line:84% and I take A, and I add an arbitrary change-- 00:07:01.150 --> 00:07:06.990 align:middle line:84% this is an arbitrary n-by-n matrix with little entries, 00:07:06.990 --> 00:07:08.580 align:middle line:90% infinitesimally small entries. 00:07:08.580 --> 00:07:12.780 align:middle line:84% And I ask how much does the function change, 00:07:12.780 --> 00:07:15.780 align:middle line:90% does A-cubed change-- 00:07:15.780 --> 00:07:20.660 align:middle line:84% and again, dropping terms that's proportional to dA-squared 00:07:20.660 --> 00:07:25.270 align:middle line:90% or higher, right? 00:07:25.270 --> 00:07:29.252 align:middle line:90% So it's nothing too mysterious. 00:07:29.252 --> 00:07:30.960 align:middle line:84% It's just asking how much does it change, 00:07:30.960 --> 00:07:32.520 align:middle line:90% keeping only linear terms. 00:07:32.520 --> 00:07:34.320 align:middle line:84% That's what the derivative means. 00:07:34.320 --> 00:07:35.100 align:middle line:90% OK? 00:07:35.100 --> 00:07:37.110 align:middle line:84% And if we keep only linear terms, 00:07:37.110 --> 00:07:38.280 align:middle line:90% we get these product rule. 00:07:38.280 --> 00:07:39.730 align:middle line:90% We get these three terms. 00:07:39.730 --> 00:07:50.730 align:middle line:84% So our f prime of A is a linear operator. 00:07:50.730 --> 00:07:52.500 align:middle line:84% And I think this is one of the cases where 00:07:52.500 --> 00:07:58.250 align:middle line:84% it's really useful to have the square brackets here, right? 00:07:58.250 --> 00:08:02.490 align:middle line:84% Because it's not anything multiplying dA on the left. 00:08:02.490 --> 00:08:05.940 align:middle line:84% It's the linear operator that takes any little change dA-- 00:08:05.940 --> 00:08:09.500 align:middle line:84% any arbitrary change there-- and returns this matrix. 00:08:09.500 --> 00:08:13.370 align:middle line:90% It returns A-squared, right? 00:08:13.370 --> 00:08:17.080 align:middle line:90% 00:08:17.080 --> 00:08:18.400 align:middle line:90% ALAN EDELMAN: So no square on-- 00:08:18.400 --> 00:08:19.467 align:middle line:90% STEVEN JOHNSON: OK? 00:08:19.467 --> 00:08:20.800 align:middle line:90% ALAN EDELMAN: --the middle term. 00:08:20.800 --> 00:08:21.220 align:middle line:90% STEVEN JOHNSON: Oh, yeah. 00:08:21.220 --> 00:08:22.512 align:middle line:90% The middle term is not squared. 00:08:22.512 --> 00:08:23.650 align:middle line:90% Thanks. 00:08:23.650 --> 00:08:24.860 align:middle line:90% Good that Alan is back. 00:08:24.860 --> 00:08:28.870 align:middle line:84% So I notice that this is very much not 00:08:28.870 --> 00:08:32.650 align:middle line:84% equal to what you would get for-- if this were a scalar, 00:08:32.650 --> 00:08:38.680 align:middle line:84% the derivative of A-cubed would be 3A-squared dA. 00:08:38.680 --> 00:08:41.164 align:middle line:84% ALAN EDELMAN: [INAUDIBLE] it would be equal, of course. 00:08:41.164 --> 00:08:42.860 align:middle line:90% STEVEN JOHNSON: Yeah. 00:08:42.860 --> 00:08:45.770 align:middle line:90% When would it be equal, right? 00:08:45.770 --> 00:08:46.685 align:middle line:90% Unless what? 00:08:46.685 --> 00:08:51.310 align:middle line:90% 00:08:51.310 --> 00:08:57.250 align:middle line:90% dA and-- anyone guess? 00:08:57.250 --> 00:09:00.218 align:middle line:84% For particular perturbations, dA, what kind of special-- 00:09:00.218 --> 00:09:02.260 align:middle line:84% ALAN EDELMAN: Well, you may not have picked it up 00:09:02.260 --> 00:09:04.690 align:middle line:84% on the microphone, but there was an answer that dA 00:09:04.690 --> 00:09:05.710 align:middle line:90% and A commute. 00:09:05.710 --> 00:09:07.240 align:middle line:90% STEVEN JOHNSON: Yes. 00:09:07.240 --> 00:09:07.780 align:middle line:90% Yes. 00:09:07.780 --> 00:09:13.000 align:middle line:84% So unless dA and A commute, which is not true, right-- 00:09:13.000 --> 00:09:14.950 align:middle line:84% so if you're interested in perturbations 00:09:14.950 --> 00:09:17.938 align:middle line:84% that are proportional to A, right, or something like that, 00:09:17.938 --> 00:09:19.480 align:middle line:84% right-- or some power of A, something 00:09:19.480 --> 00:09:23.048 align:middle line:84% that commutes with A-- then you can use the 3A-squared dA. 00:09:23.048 --> 00:09:25.090 align:middle line:84% They could-- basically, commuting matrices almost 00:09:25.090 --> 00:09:27.010 align:middle line:90% act like scalars, right? 00:09:27.010 --> 00:09:27.520 align:middle line:90% But-- 00:09:27.520 --> 00:09:28.937 align:middle line:84% ALAN EDELMAN: Well, I think people 00:09:28.937 --> 00:09:31.300 align:middle line:84% find it kind of surprising that this 00:09:31.300 --> 00:09:35.450 align:middle line:84% is kind of the derivative of A-cubed in, really, simplest 00:09:35.450 --> 00:09:35.950 align:middle line:90% form. 00:09:35.950 --> 00:09:38.450 align:middle line:84% I mean, you can't-- there's not-- 00:09:38.450 --> 00:09:40.905 align:middle line:84% there's nothing you can do to simplify that any further. 00:09:40.905 --> 00:09:41.780 align:middle line:90% STEVEN JOHNSON: Yeah. 00:09:41.780 --> 00:09:50.890 align:middle line:84% And then, particularly, it's not equal to any matrix times dA. 00:09:50.890 --> 00:09:51.390 align:middle line:90% Right? 00:09:51.390 --> 00:09:54.270 align:middle line:90% I can't just multiply dA-- 00:09:54.270 --> 00:09:56.820 align:middle line:84% multiplying a matrix dA-- on the left of my matrix-- 00:09:56.820 --> 00:09:59.190 align:middle line:84% it is a linear operation, but I cannot write this linear 00:09:59.190 --> 00:10:02.050 align:middle line:90% operation in this form. 00:10:02.050 --> 00:10:04.110 align:middle line:90% Right? 00:10:04.110 --> 00:10:06.915 align:middle line:90% And so writing-- think about it. 00:10:06.915 --> 00:10:08.850 align:middle line:84% The Jacobian of this is going to be-- 00:10:08.850 --> 00:10:11.490 align:middle line:84% we can't really-- this is a linear operation, 00:10:11.490 --> 00:10:16.170 align:middle line:84% but I can't really write down a Jacobian matrix that easily. 00:10:16.170 --> 00:10:19.200 align:middle line:84% And the only caveat is, pretty soon, Alan 00:10:19.200 --> 00:10:27.440 align:middle line:84% will show you actually how you can do this unless we 00:10:27.440 --> 00:10:35.140 align:middle line:90% convert dA to a column vector. 00:10:35.140 --> 00:10:38.810 align:middle line:90% 00:10:38.810 --> 00:10:40.970 align:middle line:90% And pretty soon, we will-- 00:10:40.970 --> 00:10:41.638 align:middle line:90% you can do that. 00:10:41.638 --> 00:10:44.180 align:middle line:84% You can take the dA, and take all of its n-squared variables, 00:10:44.180 --> 00:10:45.620 align:middle line:90% and just stack them up, right? 00:10:45.620 --> 00:10:49.320 align:middle line:84% And then you can write this back as a matrix times this. 00:10:49.320 --> 00:10:49.820 align:middle line:90% But-- 00:10:49.820 --> 00:10:51.170 align:middle line:84% ALAN EDELMAN: And by the way, I hate doing that 00:10:51.170 --> 00:10:52.820 align:middle line:90% as much as I hate indices. 00:10:52.820 --> 00:10:53.683 align:middle line:90% STEVEN JOHNSON: Yes. 00:10:53.683 --> 00:10:55.850 align:middle line:84% ALAN EDELMAN: Because-- and I'm saying that out loud 00:10:55.850 --> 00:10:59.510 align:middle line:84% because students are very prone to do this mechanically. 00:10:59.510 --> 00:11:02.270 align:middle line:84% And I kind of want to put pressure 00:11:02.270 --> 00:11:05.783 align:middle line:84% in 180 degrees the opposite way-- not to do it. 00:11:05.783 --> 00:11:07.700 align:middle line:84% But of course, I will do it right away anyway. 00:11:07.700 --> 00:11:09.290 align:middle line:90% But I hate it. 00:11:09.290 --> 00:11:11.685 align:middle line:84% [CHUCKLES] I'll explain further in a little bit. 00:11:11.685 --> 00:11:12.560 align:middle line:90% STEVEN JOHNSON: Yeah. 00:11:12.560 --> 00:11:15.480 align:middle line:84% And so let me do the other example. 00:11:15.480 --> 00:11:18.133 align:middle line:84% And then I'm going to stop, right? 00:11:18.133 --> 00:11:20.300 align:middle line:84% And of course, these are not the only vector spaces. 00:11:20.300 --> 00:11:22.970 align:middle line:84% But maybe later, I'll talk about if you have functions 00:11:22.970 --> 00:11:27.080 align:middle line:84% that take a function as input and give you, say, 00:11:27.080 --> 00:11:28.350 align:middle line:90% a number as output. 00:11:28.350 --> 00:11:30.520 align:middle line:84% Then you can also talk about derivatives. 00:11:30.520 --> 00:11:32.390 align:middle line:84% Because functions form a vector space. 00:11:32.390 --> 00:11:35.850 align:middle line:84% And that leads you to something called calculus of variations, 00:11:35.850 --> 00:11:38.880 align:middle line:84% which is another really cool application 00:11:38.880 --> 00:11:42.100 align:middle line:90% of the same kind of idea. 00:11:42.100 --> 00:11:46.480 align:middle line:84% So the other one that's really, really practically important 00:11:46.480 --> 00:11:48.520 align:middle line:84% is taking the derivative of matrix inverses. 00:11:48.520 --> 00:11:49.520 align:middle line:90% Why is it important? 00:11:49.520 --> 00:11:51.940 align:middle line:84% Because matrix inverses come up in solving 00:11:51.940 --> 00:11:55.960 align:middle line:84% lots of real-world problems, like solving the stress 00:11:55.960 --> 00:11:57.040 align:middle line:90% on an airplane wing. 00:11:57.040 --> 00:12:00.700 align:middle line:84% And if you want to compute the derivative 00:12:00.700 --> 00:12:03.460 align:middle line:84% of the output with respect to the parameters in the matrix, 00:12:03.460 --> 00:12:05.950 align:middle line:84% right, you need to differentiate through a matrix inverse 00:12:05.950 --> 00:12:06.830 align:middle line:90% somehow. 00:12:06.830 --> 00:12:08.372 align:middle line:84% ALAN EDELMAN: But let me quickly ask, 00:12:08.372 --> 00:12:12.520 align:middle line:84% if A were a scalar, just what's the derivative of 1 over A? 00:12:12.520 --> 00:12:15.060 align:middle line:90% Shout it out. 00:12:15.060 --> 00:12:17.710 align:middle line:90% You all know it. 00:12:17.710 --> 00:12:20.450 align:middle line:84% What's the derivative of the function that takes [INAUDIBLE] 00:12:20.450 --> 00:12:21.940 align:middle line:90% of 1 over A? 00:12:21.940 --> 00:12:23.500 align:middle line:84% AUDIENCE: Negative 1 over A-squared. 00:12:23.500 --> 00:12:23.860 align:middle line:90% ALAN EDELMAN: Right. 00:12:23.860 --> 00:12:26.193 align:middle line:84% So we're looking for something that's kind of like that. 00:12:26.193 --> 00:12:28.250 align:middle line:90% But it's more matrixy, right? 00:12:28.250 --> 00:12:28.750 align:middle line:90% So just to-- 00:12:28.750 --> 00:12:29.625 align:middle line:90% STEVEN JOHNSON: Yeah. 00:12:29.625 --> 00:12:31.067 align:middle line:90% ALAN EDELMAN: [INAUDIBLE] 00:12:31.067 --> 00:12:32.650 align:middle line:84% STEVEN JOHNSON: So here's a case where 00:12:32.650 --> 00:12:35.470 align:middle line:84% if I did it the 18.02 style, I would go mad, right? 00:12:35.470 --> 00:12:39.190 align:middle line:90% I could write down the inverse-- 00:12:39.190 --> 00:12:40.810 align:middle line:84% like, some explicit, horrible formula 00:12:40.810 --> 00:12:43.720 align:middle line:84% in terms of Cramer's rule and determinants 00:12:43.720 --> 00:12:46.743 align:middle line:84% of minors, and cofactors, and things, right; and then 00:12:46.743 --> 00:12:48.910 align:middle line:84% try to take the derivative of each component of that 00:12:48.910 --> 00:12:51.460 align:middle line:84% with respect to each entry of A. And it's just-- 00:12:51.460 --> 00:12:53.928 align:middle line:84% ALAN EDELMAN: You-- even in the 2-by-2 case, 00:12:53.928 --> 00:12:54.970 align:middle line:90% I would hate to see that. 00:12:54.970 --> 00:12:56.270 align:middle line:84% STEVEN JOHNSON: Even in the 2-by-2 case, 00:12:56.270 --> 00:12:57.812 align:middle line:84% it would be-- oh, you would go crazy. 00:12:57.812 --> 00:13:01.690 align:middle line:84% But so there's a trick here that makes it really easy, right? 00:13:01.690 --> 00:13:12.450 align:middle line:84% So we know that A inverse times A is equal to what? 00:13:12.450 --> 00:13:14.790 align:middle line:84% Yeah, hopefully somebody said "I", right? 00:13:14.790 --> 00:13:17.610 align:middle line:84% So again, these are going to be n-by-n. 00:13:17.610 --> 00:13:20.580 align:middle line:84% Obviously, this is only going to work for n-by-n nonsingular 00:13:20.580 --> 00:13:21.090 align:middle line:90% matrices. 00:13:21.090 --> 00:13:25.570 align:middle line:84% And this is the n-by-n identity matrix, right? 00:13:25.570 --> 00:13:29.260 align:middle line:84% So then what I can do is I can say, therefore, 00:13:29.260 --> 00:13:31.330 align:middle line:90% let's ask what happens-- 00:13:31.330 --> 00:13:32.710 align:middle line:90% what's d of this? 00:13:32.710 --> 00:13:36.520 align:middle line:84% And d of this, again, means I change A by a little bit. 00:13:36.520 --> 00:13:40.120 align:middle line:84% And I ask how much does this change by. 00:13:40.120 --> 00:13:42.100 align:middle line:90% Oops. 00:13:42.100 --> 00:13:46.090 align:middle line:84% And the answer is, of course, it doesn't matter how I change A. 00:13:46.090 --> 00:13:48.460 align:middle line:84% If I change A by a little bit, the product is I. 00:13:48.460 --> 00:13:52.710 align:middle line:90% So the derivative is 0. 00:13:52.710 --> 00:13:58.340 align:middle line:84% But by the product rule, this must 00:13:58.340 --> 00:14:16.260 align:middle line:84% be equal to d of A inverse times A plus A inverse times d of dA. 00:14:16.260 --> 00:14:18.760 align:middle line:90% 00:14:18.760 --> 00:14:19.260 align:middle line:90% All right? 00:14:19.260 --> 00:14:20.468 align:middle line:90% That's just the product tool. 00:14:20.468 --> 00:14:24.530 align:middle line:84% That must equal-- is this clear to everyone? 00:14:24.530 --> 00:14:26.750 align:middle line:84% So if I take this thing and I ask-- 00:14:26.750 --> 00:14:29.380 align:middle line:84% if you change A by a little bit, how does this change? 00:14:29.380 --> 00:14:30.410 align:middle line:90% I can do it two ways. 00:14:30.410 --> 00:14:31.868 align:middle line:84% I can apply the product rule, which 00:14:31.868 --> 00:14:35.440 align:middle line:84% means it's d of this, the change of A inverse times A 00:14:35.440 --> 00:14:40.610 align:middle line:84% plus A inverse times the change in A. Or I can say, 00:14:40.610 --> 00:14:43.330 align:middle line:84% well, A inverse is I. So I can ask, how much does I 00:14:43.330 --> 00:14:44.650 align:middle line:90% change if I change A? 00:14:44.650 --> 00:14:46.910 align:middle line:90% And the answer is, not at all. 00:14:46.910 --> 00:14:50.570 align:middle line:90% So this-- 0 has to equal that. 00:14:50.570 --> 00:14:54.720 align:middle line:84% So if I just put those two together, right, 00:14:54.720 --> 00:15:07.660 align:middle line:84% and I solve for-- that means that dA inverse, 00:15:07.660 --> 00:15:24.780 align:middle line:84% right, times A has to equal minus A inverse dA. 00:15:24.780 --> 00:15:41.240 align:middle line:84% But that means that df, which is, of course, dA inverse, OK-- 00:15:41.240 --> 00:15:42.308 align:middle line:90% how can I get dA inverse? 00:15:42.308 --> 00:15:44.100 align:middle line:84% I need to move this to the right-hand side. 00:15:44.100 --> 00:15:49.500 align:middle line:84% So I just multiply both sides of this 00:15:49.500 --> 00:15:51.300 align:middle line:90% by A inverse on both sides. 00:15:51.300 --> 00:15:56.050 align:middle line:90% 00:15:56.050 --> 00:15:56.705 align:middle line:90% Right? 00:15:56.705 --> 00:16:02.180 align:middle line:84% And so if I multiply this by A inverse on the right, OK, then 00:16:02.180 --> 00:16:03.950 align:middle line:90% I just get dA inverse. 00:16:03.950 --> 00:16:17.547 align:middle line:84% And on the other side, I get a minus A inverse dA A inverse. 00:16:17.547 --> 00:16:18.130 align:middle line:90% And that's it. 00:16:18.130 --> 00:16:32.377 align:middle line:84% This is-- and so this is my f prime of A acting on dA. 00:16:32.377 --> 00:16:34.210 align:middle line:84% ALAN EDELMAN: So you see when it's a scalar, 00:16:34.210 --> 00:16:38.184 align:middle line:84% it's minus 1 over A-squared times dA-- 00:16:38.184 --> 00:16:39.520 align:middle line:90% the result you remember. 00:16:39.520 --> 00:16:42.160 align:middle line:84% But in a way, I like to think of it 00:16:42.160 --> 00:16:46.270 align:middle line:84% as, when you have one-- like the scalar case, the minus 1 00:16:46.270 --> 00:16:47.470 align:middle line:90% over A-squared. 00:16:47.470 --> 00:16:49.630 align:middle line:84% Should that 1 over A-squared be on the left, 00:16:49.630 --> 00:16:50.950 align:middle line:90% or should it be on the right? 00:16:50.950 --> 00:16:53.660 align:middle line:84% Well, there's no reason to prefer one over the other. 00:16:53.660 --> 00:16:55.870 align:middle line:84% And the math, of course, makes it 00:16:55.870 --> 00:16:57.998 align:middle line:84% very fair by putting one A inverse on the left 00:16:57.998 --> 00:16:59.290 align:middle line:90% and one A inverse on the right. 00:16:59.290 --> 00:17:02.948 align:middle line:84% And that's kind of how I like to look at that answer. 00:17:02.948 --> 00:17:03.823 align:middle line:90% STEVEN JOHNSON: Yeah. 00:17:03.823 --> 00:17:03.960 align:middle line:90% And we're going to-- 00:17:03.960 --> 00:17:04.800 align:middle line:90% [INTERPOSING VOICES] 00:17:04.800 --> 00:17:05.790 align:middle line:84% STEVEN JOHNSON: And we'll see that this [INAUDIBLE] 00:17:05.790 --> 00:17:07.609 align:middle line:90% is incredibly useful. 00:17:07.609 --> 00:17:08.617 align:middle line:90% Yeah. 00:17:08.617 --> 00:17:10.200 align:middle line:84% Because differentiating through matrix 00:17:10.200 --> 00:17:14.910 align:middle line:84% inverses is something that actually happens all the time, 00:17:14.910 --> 00:17:19.680 align:middle line:84% especially if you're optimizing an engineering problem. 00:17:19.680 --> 00:17:21.329 align:middle line:84% Because engineering problems involve 00:17:21.329 --> 00:17:22.800 align:middle line:90% solving systems of equations. 00:17:22.800 --> 00:17:24.660 align:middle line:84% And then if you want to optimize the result, 00:17:24.660 --> 00:17:28.515 align:middle line:84% you need to differentiate through this. 00:17:28.515 --> 00:17:30.235 align:middle line:84% I'm going to-- should I stop here, Alan? 00:17:30.235 --> 00:17:31.860 align:middle line:84% Did you want to talk for a few minutes? 00:17:31.860 --> 00:17:32.850 align:middle line:84% ALAN EDELMAN: [INAUDIBLE] have 10 minutes. 00:17:32.850 --> 00:17:34.350 align:middle line:84% But I think it'd be fun to show some 00:17:34.350 --> 00:17:36.930 align:middle line:84% of the examples in the Pluto notebook at this point. 00:17:36.930 --> 00:17:37.250 align:middle line:90% STEVEN JOHNSON: Sure, sure. 00:17:37.250 --> 00:17:38.625 align:middle line:84% ALAN EDELMAN: So let me go ahead. 00:17:38.625 --> 00:17:39.705 align:middle line:90% And I'll do that. 00:17:39.705 --> 00:17:44.535 align:middle line:84% I'll grab the mic and put it on my shirt. 00:17:44.535 --> 00:17:48.640 align:middle line:90% 00:17:48.640 --> 00:17:50.425 align:middle line:84% And I'll grab the screen as well. 00:17:50.425 --> 00:17:57.160 align:middle line:90% 00:17:57.160 --> 00:17:57.660 align:middle line:90% OK. 00:17:57.660 --> 00:18:02.328 align:middle line:90% So I've got these-- 00:18:02.328 --> 00:18:06.310 align:middle line:84% this notebook here, which I'd like to just show you. 00:18:06.310 --> 00:18:08.500 align:middle line:84% And again, I only have about 9 or 10 minutes. 00:18:08.500 --> 00:18:12.340 align:middle line:84% So it'll just be a little bit of a taste. 00:18:12.340 --> 00:18:17.320 align:middle line:84% But in effect, what I'm really doing is showing you, again, 00:18:17.320 --> 00:18:19.233 align:middle line:90% what Steven just showed you-- 00:18:19.233 --> 00:18:21.400 align:middle line:84% maybe slightly different examples or maybe the same, 00:18:21.400 --> 00:18:22.660 align:middle line:90% I don't remember-- 00:18:22.660 --> 00:18:25.210 align:middle line:84% but kind of also a little bit of a different view. 00:18:25.210 --> 00:18:27.560 align:middle line:84% Because I think when it comes to this sort of thing, 00:18:27.560 --> 00:18:29.590 align:middle line:84% it's really helpful to see the same thing 00:18:29.590 --> 00:18:31.210 align:middle line:90% from multiple viewpoints. 00:18:31.210 --> 00:18:33.680 align:middle line:90% And so-- let's see. 00:18:33.680 --> 00:18:37.120 align:middle line:84% So first of all, just a little bit of Julia over here-- 00:18:37.120 --> 00:18:38.950 align:middle line:90% these 2-by-2 matrix Jacobians. 00:18:38.950 --> 00:18:44.260 align:middle line:84% So I'm using the Symbolics package in Julia at the moment, 00:18:44.260 --> 00:18:45.490 align:middle line:90% besides some linear algebra. 00:18:45.490 --> 00:18:47.448 align:middle line:84% I'm not even sure if I need the linear algebra. 00:18:47.448 --> 00:18:52.660 align:middle line:84% So just to see some variables, here are some variables-- p, 00:18:52.660 --> 00:18:55.120 align:middle line:90% q, r, s, and theta. 00:18:55.120 --> 00:18:58.180 align:middle line:90% And let's see. 00:18:58.180 --> 00:18:59.840 align:middle line:90% What did I do here? 00:18:59.840 --> 00:19:00.340 align:middle line:90% Yeah. 00:19:00.340 --> 00:19:05.640 align:middle line:84% So here I'm going to take a rotation matrix and multiply-- 00:19:05.640 --> 00:19:06.140 align:middle line:90% let's see. 00:19:06.140 --> 00:19:08.470 align:middle line:90% What did we do here? 00:19:08.470 --> 00:19:09.588 align:middle line:90% Where was q? 00:19:09.588 --> 00:19:10.880 align:middle line:90% Well, it doesn't really matter. 00:19:10.880 --> 00:19:13.840 align:middle line:84% The point is that I'm going to be working with 2-by-2-- 00:19:13.840 --> 00:19:14.950 align:middle line:90% let's go to this one-- 00:19:14.950 --> 00:19:17.080 align:middle line:90% 2-by-2 matrices. 00:19:17.080 --> 00:19:19.390 align:middle line:84% I like to go in column major order. 00:19:19.390 --> 00:19:20.480 align:middle line:90% This is what Julia does. 00:19:20.480 --> 00:19:23.740 align:middle line:84% And so p, q, r, s goes down the columns, right? 00:19:23.740 --> 00:19:26.720 align:middle line:84% So this is a 2-by-2 matrix over here. 00:19:26.720 --> 00:19:27.220 align:middle line:90% OK? 00:19:27.220 --> 00:19:31.600 align:middle line:84% And as Steven started to say, it's 00:19:31.600 --> 00:19:37.090 align:middle line:84% not unusual to flatten a matrix with this vec operator, right? 00:19:37.090 --> 00:19:40.420 align:middle line:84% And so here's this matrix-- p, q, r, and s-- 00:19:40.420 --> 00:19:43.660 align:middle line:84% this matrix X. And the vec of the matrix 00:19:43.660 --> 00:19:46.150 align:middle line:84% converts the matrix into a vector. 00:19:46.150 --> 00:19:48.040 align:middle line:84% This is the thing that I hate to do, 00:19:48.040 --> 00:19:52.810 align:middle line:84% but it's not bad for kind of beginning students to show you 00:19:52.810 --> 00:19:55.600 align:middle line:90% the concept of flattening. 00:19:55.600 --> 00:19:59.260 align:middle line:84% And so if we consider the matrix square function, 00:19:59.260 --> 00:20:02.020 align:middle line:84% here it is in all of its glory symbolically, right? 00:20:02.020 --> 00:20:04.450 align:middle line:84% So if I were to square that matrix 00:20:04.450 --> 00:20:07.150 align:middle line:90% X that had p, q, r, and x-- 00:20:07.150 --> 00:20:09.590 align:middle line:90% s-- p, q, r, s, right? 00:20:09.590 --> 00:20:11.330 align:middle line:84% This is what you would get if you 00:20:11.330 --> 00:20:13.760 align:middle line:90% multiplied X times X, right? 00:20:13.760 --> 00:20:18.930 align:middle line:84% And so when we talk about taking the derivative, 00:20:18.930 --> 00:20:22.140 align:middle line:84% you could think of this as a function, right? 00:20:22.140 --> 00:20:24.940 align:middle line:84% You could actually look at the vec of X-squared. 00:20:24.940 --> 00:20:26.730 align:middle line:90% And if you want to-- 00:20:26.730 --> 00:20:28.620 align:middle line:90% in a way, this is forced. 00:20:28.620 --> 00:20:31.230 align:middle line:84% But in a way, this is also a good idea. 00:20:31.230 --> 00:20:33.870 align:middle line:84% You could see that I have a vector from four input 00:20:33.870 --> 00:20:36.810 align:middle line:90% variables-- p, q, r, and s-- 00:20:36.810 --> 00:20:39.930 align:middle line:84% to these four outputs, right, these expressions-- 00:20:39.930 --> 00:20:42.010 align:middle line:90% p-squared plus qr and so forth. 00:20:42.010 --> 00:20:45.450 align:middle line:84% So if you were to work with the vec function-- 00:20:45.450 --> 00:20:48.000 align:middle line:84% if you worked the four inputs to the four outputs, 00:20:48.000 --> 00:20:51.150 align:middle line:84% what would the size of the Jacobian be? 00:20:51.150 --> 00:20:52.740 align:middle line:90% What would be the shape? 00:20:52.740 --> 00:20:56.310 align:middle line:84% If you have the four inputs and four outputs, 00:20:56.310 --> 00:20:57.252 align:middle line:90% the shape would be-- 00:20:57.252 --> 00:20:57.960 align:middle line:90% AUDIENCE: 4-by-4. 00:20:57.960 --> 00:20:58.835 align:middle line:90% ALAN EDELMAN: 4-by-4. 00:20:58.835 --> 00:21:01.230 align:middle line:84% And let's think about the general matrix squared just 00:21:01.230 --> 00:21:01.980 align:middle line:90% for a moment. 00:21:01.980 --> 00:21:04.980 align:middle line:84% If I have an n-by-n matrix that I'm squaring, 00:21:04.980 --> 00:21:08.220 align:middle line:84% right, if I vec that matrix, it'll 00:21:08.220 --> 00:21:11.880 align:middle line:84% be a column that's n-squared long. 00:21:11.880 --> 00:21:14.910 align:middle line:84% And the output will be a column that's n-squared long. 00:21:14.910 --> 00:21:17.087 align:middle line:84% And so what would the Jacobian be for general n? 00:21:17.087 --> 00:21:18.170 align:middle line:90% AUDIENCE: n-Squared matrix 00:21:18.170 --> 00:21:20.295 align:middle line:84% ALAN EDELMAN: It would be an n-squared by n-squared 00:21:20.295 --> 00:21:22.670 align:middle line:84% matrix if you write it as a Jacobian, right? 00:21:22.670 --> 00:21:24.260 align:middle line:84% And of course, you could do that. 00:21:24.260 --> 00:21:26.600 align:middle line:90% And I want to urge you to-- 00:21:26.600 --> 00:21:28.430 align:middle line:84% I don't want to say never do that, 00:21:28.430 --> 00:21:32.930 align:middle line:84% but not to do that too instinctively. 00:21:32.930 --> 00:21:35.420 align:middle line:84% But nonetheless, we can do the symbolic Jacobian. 00:21:35.420 --> 00:21:38.870 align:middle line:84% And in fact, Julia will do it for you. 00:21:38.870 --> 00:21:40.440 align:middle line:90% And here, you have it. 00:21:40.440 --> 00:21:42.980 align:middle line:90% And so let's see. 00:21:42.980 --> 00:21:48.020 align:middle line:84% So Steven made a point that if you move this way, 00:21:48.020 --> 00:21:49.160 align:middle line:90% you're changing the inputs. 00:21:49.160 --> 00:21:51.750 align:middle line:84% And if you move this way, you're changing the outputs. 00:21:51.750 --> 00:21:53.420 align:middle line:84% So let's focus on this one output 00:21:53.420 --> 00:21:56.420 align:middle line:84% over here, p-squared plus qr, right? 00:21:56.420 --> 00:21:58.430 align:middle line:84% If I take the derivative with respect to p, 00:21:58.430 --> 00:21:59.665 align:middle line:90% I'll get 2p, right? 00:21:59.665 --> 00:22:01.790 align:middle line:84% The next derivative I'll take is with respect to q. 00:22:01.790 --> 00:22:05.030 align:middle line:84% And I'll get the r, which goes this way, right? 00:22:05.030 --> 00:22:07.700 align:middle line:84% Then if I take the derivative with respect to r, 00:22:07.700 --> 00:22:09.158 align:middle line:84% I'll get the q, which is over here. 00:22:09.158 --> 00:22:10.825 align:middle line:84% And finally, the derivative with respect 00:22:10.825 --> 00:22:13.170 align:middle line:84% to s-- well, this doesn't depend on s at all. 00:22:13.170 --> 00:22:14.600 align:middle line:90% So the derivative is 0, right? 00:22:14.600 --> 00:22:18.870 align:middle line:84% And so if I focus on this one output and take all the 00:22:18.870 --> 00:22:22.030 align:middle line:84% four input derivatives, I go this way. 00:22:22.030 --> 00:22:24.930 align:middle line:84% If I did the same down here, I would get this row, right? 00:22:24.930 --> 00:22:26.820 align:middle line:84% The derivative of this with respect to p 00:22:26.820 --> 00:22:28.900 align:middle line:84% is 0, which is that 0 over there. 00:22:28.900 --> 00:22:29.400 align:middle line:90% OK? 00:22:29.400 --> 00:22:33.000 align:middle line:84% And so somehow or another, this is 00:22:33.000 --> 00:22:36.540 align:middle line:84% capturing the derivative of A-squared 00:22:36.540 --> 00:22:38.700 align:middle line:90% explicitly as a matrix. 00:22:38.700 --> 00:22:40.800 align:middle line:84% You see, I think by now you're probably 00:22:40.800 --> 00:22:43.170 align:middle line:84% getting my point of view that if you 00:22:43.170 --> 00:22:47.100 align:middle line:84% take 18.06 or any elementary linear algebra class-- 00:22:47.100 --> 00:22:50.070 align:middle line:84% in the old days-- and maybe some of you took a course like 00:22:50.070 --> 00:22:50.650 align:middle line:90% this-- 00:22:50.650 --> 00:22:52.570 align:middle line:90% you never would see a matrix. 00:22:52.570 --> 00:22:54.120 align:middle line:84% But if you take Gil Strang's course 00:22:54.120 --> 00:22:55.920 align:middle line:84% and a lot of other linear algebra classes, 00:22:55.920 --> 00:22:59.318 align:middle line:84% you think that you only see a matrix by element-by-element. 00:22:59.318 --> 00:23:01.110 align:middle line:84% You don't see the linear operators anymore. 00:23:01.110 --> 00:23:03.027 align:middle line:84% You just-- it's almost like missing the forest 00:23:03.027 --> 00:23:05.300 align:middle line:90% for the trees because-- 00:23:05.300 --> 00:23:06.120 align:middle line:90% or is it the trees? 00:23:06.120 --> 00:23:08.037 align:middle line:84% Yeah, you're missing the forest for the trees. 00:23:08.037 --> 00:23:10.470 align:middle line:84% Because in most modern linear algebra classes, 00:23:10.470 --> 00:23:12.190 align:middle line:84% you're focusing element by element. 00:23:12.190 --> 00:23:13.980 align:middle line:84% And so that's kind of how you're-- 00:23:13.980 --> 00:23:16.120 align:middle line:84% you almost psychologically learn to think that way. 00:23:16.120 --> 00:23:18.245 align:middle line:84% And you don't think about linear operators anymore. 00:23:18.245 --> 00:23:20.310 align:middle line:84% I don't know how you all learned linear algebra. 00:23:20.310 --> 00:23:22.710 align:middle line:84% But in the old days, it was no matrices, 00:23:22.710 --> 00:23:24.040 align:middle line:90% pure linear operators. 00:23:24.040 --> 00:23:24.540 align:middle line:90% OK? 00:23:24.540 --> 00:23:26.440 align:middle line:90% So here's an explicit matrix. 00:23:26.440 --> 00:23:31.900 align:middle line:84% And of course, we can do this numerically as well. 00:23:31.900 --> 00:23:37.340 align:middle line:84% So here, I'm replacing p, q, r and s 00:23:37.340 --> 00:23:40.980 align:middle line:90% with the numbers 1, 3, 2, 4. 00:23:40.980 --> 00:23:41.480 align:middle line:90% OK? 00:23:41.480 --> 00:23:43.750 align:middle line:90% And let's see. 00:23:43.750 --> 00:23:45.500 align:middle line:84% And of course, I could do this numerically 00:23:45.500 --> 00:23:48.620 align:middle line:84% by making a perturbation of-- so I have my matrix 00:23:48.620 --> 00:23:51.120 align:middle line:90% M, which is 1, 2, 3, 4, right? 00:23:51.120 --> 00:23:52.190 align:middle line:90% And I have this-- 00:23:52.190 --> 00:23:55.760 align:middle line:84% E is 0.001-- just a few small numbers. 00:23:55.760 --> 00:23:59.300 align:middle line:84% And of course, you can see M plus E-squared minus M-squared 00:23:59.300 --> 00:24:03.500 align:middle line:84% is, in fact, the change to the square, right? 00:24:03.500 --> 00:24:07.740 align:middle line:90% So here-- let's see. 00:24:07.740 --> 00:24:09.806 align:middle line:84% So maybe I'll add this right now. 00:24:09.806 --> 00:24:19.480 align:middle line:84% So in the language that Steven had just done, 00:24:19.480 --> 00:24:22.900 align:middle line:84% we would probably write this as dX-squared is-- 00:24:22.900 --> 00:24:23.680 align:middle line:90% oops, oops, oops. 00:24:23.680 --> 00:24:28.150 align:middle line:84% Where did I go? dX squared is equal to X-- 00:24:28.150 --> 00:24:32.410 align:middle line:84% he just did the cube, but the same product rule 00:24:32.410 --> 00:24:35.260 align:middle line:84% gives you the answer for the square, right? 00:24:35.260 --> 00:24:37.270 align:middle line:84% So I'll just write this right here. 00:24:37.270 --> 00:24:38.270 align:middle line:90% I won't execute that. 00:24:38.270 --> 00:24:38.770 align:middle line:90% Right? 00:24:38.770 --> 00:24:42.250 align:middle line:90% So this is how we're writing-- 00:24:42.250 --> 00:24:44.650 align:middle line:84% see, I want to make sure that you clearly 00:24:44.650 --> 00:24:47.800 align:middle line:84% understand that, one way or another, the information 00:24:47.800 --> 00:24:51.610 align:middle line:84% in this 4-by-4 matrix and, in some sense, 00:24:51.610 --> 00:24:54.460 align:middle line:84% the information in this linear operator 00:24:54.460 --> 00:24:57.000 align:middle line:84% somehow has to be the same thing mathematically, right? 00:24:57.000 --> 00:24:59.830 align:middle line:84% It's a matter of presentation, but it's not so much 00:24:59.830 --> 00:25:02.493 align:middle line:90% a matter of what's going on. 00:25:02.493 --> 00:25:04.660 align:middle line:84% I mean, some people would say that basis was chosen. 00:25:04.660 --> 00:25:09.130 align:middle line:84% But forgetting about that, from our point of view, 00:25:09.130 --> 00:25:11.750 align:middle line:84% this thing must express the same thing. 00:25:11.750 --> 00:25:12.250 align:middle line:90% OK? 00:25:12.250 --> 00:25:14.320 align:middle line:84% And so I'm going to write this operator. 00:25:14.320 --> 00:25:16.280 align:middle line:84% I don't like to write the dX's all the time. 00:25:16.280 --> 00:25:18.650 align:middle line:84% So I'm going to write this linear operator, 00:25:18.650 --> 00:25:21.380 align:middle line:84% this linear transformation of dX. 00:25:21.380 --> 00:25:22.842 align:middle line:84% I'm going to call it E. All right? 00:25:22.842 --> 00:25:24.800 align:middle line:84% And I'm not going to use X. I'm going to use M. 00:25:24.800 --> 00:25:26.810 align:middle line:84% But you'll see it's just ME plus EM. 00:25:26.810 --> 00:25:29.870 align:middle line:84% It's really the same thing here with different letters. 00:25:29.870 --> 00:25:33.020 align:middle line:84% And if I go linear transformation of E, 00:25:33.020 --> 00:25:35.960 align:middle line:84% I get the exact same answer to first order. 00:25:35.960 --> 00:25:38.210 align:middle line:84% I mean, you could see a little bit of the second order 00:25:38.210 --> 00:25:39.290 align:middle line:90% term showing up. 00:25:39.290 --> 00:25:41.570 align:middle line:84% But here, we're taking a difference. 00:25:41.570 --> 00:25:43.790 align:middle line:84% Here, we're computing a linear transformation. 00:25:43.790 --> 00:25:46.100 align:middle line:84% And you see that, to first order, 00:25:46.100 --> 00:25:48.062 align:middle line:90% we're getting the same numbers. 00:25:48.062 --> 00:25:49.520 align:middle line:84% I don't know if you've ever stopped 00:25:49.520 --> 00:25:53.180 align:middle line:84% to think how magic calculus is, where you can numerically 00:25:53.180 --> 00:25:55.000 align:middle line:84% take a little perturbation-- or you 00:25:55.000 --> 00:25:56.750 align:middle line:84% have a formula for the answer that's going 00:25:56.750 --> 00:25:57.950 align:middle line:90% to turn out to be the same. 00:25:57.950 --> 00:25:59.450 align:middle line:84% I mean, we're all so used to it, you 00:25:59.450 --> 00:26:01.790 align:middle line:84% may have never stopped to see the magic in all of that. 00:26:01.790 --> 00:26:05.450 align:middle line:84% But it's really very magical that you could numerically 00:26:05.450 --> 00:26:10.250 align:middle line:84% subtract two nearby things, and the mathematical gods above 00:26:10.250 --> 00:26:11.960 align:middle line:84% give you a formula for that difference. 00:26:11.960 --> 00:26:13.670 align:middle line:90% I think it's-- 00:26:13.670 --> 00:26:15.620 align:middle line:84% I bet you've never stopped to appreciate that, 00:26:15.620 --> 00:26:17.310 align:middle line:84% but it's really a wonderful thing. 00:26:17.310 --> 00:26:21.910 align:middle line:84% So here, the ability to compute ME plus EM 00:26:21.910 --> 00:26:24.790 align:middle line:84% gives you this exact infinitesimal answer 00:26:24.790 --> 00:26:27.310 align:middle line:84% that corresponds to this numerical difference. 00:26:27.310 --> 00:26:31.540 align:middle line:84% And I think that's just a wonderful, wonderful thing. 00:26:31.540 --> 00:26:35.110 align:middle line:84% Now, there is a notation called the Kronecker product 00:26:35.110 --> 00:26:37.840 align:middle line:84% which kind of lets you express these sorts of things 00:26:37.840 --> 00:26:40.960 align:middle line:84% quite well, quite nicely, kind of going back and forth 00:26:40.960 --> 00:26:43.810 align:middle line:84% between the explicit matrix and the linear operator 00:26:43.810 --> 00:26:44.973 align:middle line:90% point of view. 00:26:44.973 --> 00:26:46.390 align:middle line:84% And I don't know-- how many of you 00:26:46.390 --> 00:26:48.340 align:middle line:84% have said you've seen the Kronecker product somewhere, 00:26:48.340 --> 00:26:49.120 align:middle line:90% somehow? 00:26:49.120 --> 00:26:51.580 align:middle line:90% So about three or four. 00:26:51.580 --> 00:26:53.230 align:middle line:90% It's a small-- oh, five-- 00:26:53.230 --> 00:26:55.720 align:middle line:84% a small minority of the students in the room. 00:26:55.720 --> 00:27:00.310 align:middle line:84% So I like to tell the story that, in one of the courses 00:27:00.310 --> 00:27:03.680 align:middle line:84% years ago, I showed the Kronecker product to-- 00:27:03.680 --> 00:27:06.370 align:middle line:84% and I just brought it up just like I'm doing for you today. 00:27:06.370 --> 00:27:09.670 align:middle line:84% And about a month or two later, a student came back to me 00:27:09.670 --> 00:27:12.760 align:middle line:84% and said, you know, I'm so glad you showed me the Kronecker 00:27:12.760 --> 00:27:13.450 align:middle line:90% product. 00:27:13.450 --> 00:27:15.830 align:middle line:84% I actually used the Kronecker product in a paper. 00:27:15.830 --> 00:27:17.050 align:middle line:90% And I won a big prize. 00:27:17.050 --> 00:27:19.560 align:middle line:84% So thank you for showing me the Kronecker product. 00:27:19.560 --> 00:27:21.680 align:middle line:84% So I can't guarantee you'll get a prize for it, 00:27:21.680 --> 00:27:23.640 align:middle line:90% but that did happen to me. 00:27:23.640 --> 00:27:24.140 align:middle line:90% All right. 00:27:24.140 --> 00:27:28.520 align:middle line:90% So let's see. 00:27:28.520 --> 00:27:31.610 align:middle line:90% So here, I've got-- 00:27:31.610 --> 00:27:33.260 align:middle line:84% I'll add the variables a, b, c, and d. 00:27:33.260 --> 00:27:34.488 align:middle line:90% And I'll just take two-- 00:27:34.488 --> 00:27:36.530 align:middle line:84% I'll just show you the Kronecker product quickly. 00:27:36.530 --> 00:27:38.810 align:middle line:84% And I'll kind of use it on Monday. 00:27:38.810 --> 00:27:41.210 align:middle line:84% But here are two 2-by-2 matrices. 00:27:41.210 --> 00:27:48.290 align:middle line:84% And you could imagine trying to take all possible products 00:27:48.290 --> 00:27:51.230 align:middle line:84% of entries in the first matrix with entries 00:27:51.230 --> 00:27:52.830 align:middle line:90% in the second matrix. 00:27:52.830 --> 00:27:53.330 align:middle line:90% OK? 00:27:53.330 --> 00:27:58.990 align:middle line:84% And so let's-- oh, yeah, let's do that here. 00:27:58.990 --> 00:28:00.990 align:middle line:84% So let me take the Kronecker product 00:28:00.990 --> 00:28:09.140 align:middle line:84% of these two matrices just to show you what it looks like. 00:28:09.140 --> 00:28:13.270 align:middle line:84% And I think you'll notice that on the-- 00:28:13.270 --> 00:28:16.210 align:middle line:84% we have a's, b's, c's, and d's in all possible combinations 00:28:16.210 --> 00:28:18.520 align:middle line:84% on the left and p's, q's, r's, and s's 00:28:18.520 --> 00:28:21.400 align:middle line:84% in all kinds of possible combinations on the right. 00:28:21.400 --> 00:28:23.440 align:middle line:84% This gets shown a little bit better 00:28:23.440 --> 00:28:26.710 align:middle line:84% with letters and these cute characters 00:28:26.710 --> 00:28:28.330 align:middle line:90% that we have available in Julia. 00:28:28.330 --> 00:28:33.100 align:middle line:84% So here, I'm going to do it with the Kronecker product of-- 00:28:33.100 --> 00:28:33.670 align:middle line:90% let's see. 00:28:33.670 --> 00:28:36.700 align:middle line:84% Here's a, b, c, d, e, f, I guess. 00:28:36.700 --> 00:28:42.010 align:middle line:84% So that's a 2-by-3 matrix, with the pizza, the alien-- 00:28:42.010 --> 00:28:45.070 align:middle line:84% what do I have-- the panda, and the smiley cat. 00:28:45.070 --> 00:28:47.080 align:middle line:84% And again, I think with your eyeballs, 00:28:47.080 --> 00:28:49.520 align:middle line:84% you could kind of see what's happening. 00:28:49.520 --> 00:28:52.400 align:middle line:84% Here's the 2-by-2 matrix with the identity. 00:28:52.400 --> 00:28:55.040 align:middle line:84% Here's what happens when the identity is on the left. 00:28:55.040 --> 00:28:55.540 align:middle line:90% OK? 00:28:55.540 --> 00:28:59.543 align:middle line:84% And so it's just all possible products with the convention-- 00:28:59.543 --> 00:29:01.960 align:middle line:84% and I think, with the identity, you might even see it most 00:29:01.960 --> 00:29:02.570 align:middle line:90% clearly-- 00:29:02.570 --> 00:29:07.850 align:middle line:84% the convention that you take the top-left entry, 00:29:07.850 --> 00:29:09.680 align:middle line:84% say, of the first matrix with all 00:29:09.680 --> 00:29:11.870 align:middle line:84% of the entries in the second matrix, right? 00:29:11.870 --> 00:29:12.980 align:middle line:90% And then you go-- 00:29:12.980 --> 00:29:15.680 align:middle line:84% the bottom left is down here, the top right 00:29:15.680 --> 00:29:16.940 align:middle line:90% of the left entry, and the-- 00:29:16.940 --> 00:29:18.530 align:middle line:90% you see how that works, right? 00:29:18.530 --> 00:29:21.380 align:middle line:90% And so that's the order. 00:29:21.380 --> 00:29:22.820 align:middle line:90% And OK. 00:29:22.820 --> 00:29:23.960 align:middle line:90% I'm going to end now. 00:29:23.960 --> 00:29:25.370 align:middle line:90% The class is over. 00:29:25.370 --> 00:29:29.240 align:middle line:84% But I will mention that to get this-- 00:29:29.240 --> 00:29:32.810 align:middle line:84% this Jacobian matrix that you saw before from the square 00:29:32.810 --> 00:29:35.250 align:middle line:84% can be expressed using Kronecker products. 00:29:35.250 --> 00:29:36.538 align:middle line:90% This is how it's done. 00:29:36.538 --> 00:29:37.580 align:middle line:90% So I'm going to pick up-- 00:29:37.580 --> 00:29:40.310 align:middle line:84% I'll probably practically start from the beginning on Monday, 00:29:40.310 --> 00:29:42.860 align:middle line:90% I imagine, since class is over. 00:29:42.860 --> 00:29:44.683 align:middle line:84% But I just wanted you to quickly get 00:29:44.683 --> 00:29:46.100 align:middle line:84% the sense of the Kronecker product 00:29:46.100 --> 00:29:49.820 align:middle line:90% as a way of expressing the-- 00:29:49.820 --> 00:29:54.320 align:middle line:84% it's a great way of expressing some of these matrix functions 00:29:54.320 --> 00:29:56.780 align:middle line:84% and the derivatives of some of these matrix functions. 00:29:56.780 --> 00:29:57.020 align:middle line:90% All right? 00:29:57.020 --> 00:29:58.020 align:middle line:90% So I'll stop right here. 00:29:58.020 --> 00:29:58.610 align:middle line:90% And-- 00:29:58.610 --> 00:30:01.737 align:middle line:84% STEVEN JOHNSON: And we'll post pset 1 later today, 00:30:01.737 --> 00:30:02.570 align:middle line:90% by early [INAUDIBLE] 00:30:02.570 --> 00:30:04.195 align:middle line:84% ALAN EDELMAN: We're hoping for 5:00 PM. 00:30:04.195 --> 00:30:06.460 align:middle line:84% But if it doesn't happen, please don't hate us. 00:30:06.460 --> 00:30:09.430 align:middle line:84% But I think we're going to target 5:00 PM. 00:30:09.430 --> 00:30:11.440 align:middle line:90% And we'll see you all on Monday. 00:30:11.440 --> 00:30:14.440 align:middle line:84% So have a good weekend, everybody. 00:30:14.440 --> 00:30:17.000 align:middle line:90%