WEBVTT 00:00:00.000 --> 00:00:02.380 align:middle line:90% [SQUEAKING] 00:00:02.380 --> 00:00:04.284 align:middle line:90% [RUSTLING] 00:00:04.284 --> 00:00:06.664 align:middle line:90% [CLICKING] 00:00:06.664 --> 00:00:09.610 align:middle line:90% 00:00:09.610 --> 00:00:10.610 align:middle line:90% ALAN EDELMAN: All right. 00:00:10.610 --> 00:00:13.070 align:middle line:84% Well, welcome to the second lecture for this IAP class 00:00:13.070 --> 00:00:15.992 align:middle line:90% on matrix calculus. 00:00:15.992 --> 00:00:17.450 align:middle line:84% Steven, do you want to say anything 00:00:17.450 --> 00:00:21.380 align:middle line:84% before I start on nonlinear versus linear maps? 00:00:21.380 --> 00:00:24.390 align:middle line:84% So Philip's going to help out right away. 00:00:24.390 --> 00:00:27.485 align:middle line:84% So here's a picture of Philip, the corgi. 00:00:27.485 --> 00:00:31.100 align:middle line:84% He's right over there, in case you haven't seen him yet. 00:00:31.100 --> 00:00:34.250 align:middle line:84% And you folks could actually grab your favorite photo. 00:00:34.250 --> 00:00:35.540 align:middle line:90% You don't have to use a corgi. 00:00:35.540 --> 00:00:37.110 align:middle line:84% You could have any photo you like. 00:00:37.110 --> 00:00:39.500 align:middle line:84% So here in Julia I'm just uploading 00:00:39.500 --> 00:00:42.170 align:middle line:90% this picture of Philip. 00:00:42.170 --> 00:00:46.670 align:middle line:84% And I've got a few different transformations of a corgi. 00:00:46.670 --> 00:00:49.820 align:middle line:84% And the reason why I'm showing this to you, 00:00:49.820 --> 00:00:52.930 align:middle line:84% it's because it's come to my attention 00:00:52.930 --> 00:00:56.800 align:middle line:84% that not everybody has a fairly good visual intuition 00:00:56.800 --> 00:01:00.970 align:middle line:84% as to the difference between a linear and a nonlinear map, 00:01:00.970 --> 00:01:03.580 align:middle line:90% or even a map. 00:01:03.580 --> 00:01:05.200 align:middle line:90% Here, so let's go ahead. 00:01:05.200 --> 00:01:08.060 align:middle line:84% Let me just ask a real quick simple question for starters. 00:01:08.060 --> 00:01:09.560 align:middle line:90% So here I'm just rotating. 00:01:09.560 --> 00:01:10.060 align:middle line:90% Right? 00:01:10.060 --> 00:01:11.080 align:middle line:90% So I'm rotating Philip. 00:01:11.080 --> 00:01:13.160 align:middle line:90% He's going to get dizzy. 00:01:13.160 --> 00:01:13.660 align:middle line:90% OK. 00:01:13.660 --> 00:01:14.993 align:middle line:90% No, he's not going to get dizzy. 00:01:14.993 --> 00:01:15.940 align:middle line:90% No animal is harmed. 00:01:15.940 --> 00:01:19.090 align:middle line:90% 00:01:19.090 --> 00:01:20.410 align:middle line:90% So, all right. 00:01:20.410 --> 00:01:24.760 align:middle line:84% If I'm talking about maps from vectors to vectors from Rn-- 00:01:24.760 --> 00:01:28.750 align:middle line:84% it's usually Rn as in Nancy to Rm as in Mary. 00:01:28.750 --> 00:01:29.530 align:middle line:90% OK? 00:01:29.530 --> 00:01:33.340 align:middle line:84% What are m and n for this transformation? 00:01:33.340 --> 00:01:34.960 align:middle line:90% That's an easy question. 00:01:34.960 --> 00:01:37.790 align:middle line:90% Not a trick question. 00:01:37.790 --> 00:01:39.960 align:middle line:90% So what are m and n for here? 00:01:39.960 --> 00:01:41.810 align:middle line:84% So think of this as a transformation 00:01:41.810 --> 00:01:45.200 align:middle line:84% from the original upright picture to the rotated picture. 00:01:45.200 --> 00:01:48.480 align:middle line:90% What are m and n? 00:01:48.480 --> 00:01:51.860 align:middle line:90% That's an easy one. 00:01:51.860 --> 00:01:52.532 align:middle line:90% Oh, gosh. 00:01:52.532 --> 00:01:53.990 align:middle line:84% No, I'm not even thinking that way. 00:01:53.990 --> 00:01:55.160 align:middle line:90% That's too fancy. 00:01:55.160 --> 00:01:58.640 align:middle line:84% Not that you're-- in some deep sense you're probably right, 00:01:58.640 --> 00:02:02.520 align:middle line:84% but for the purposes of this class you're way off. 00:02:02.520 --> 00:02:03.930 align:middle line:90% So let's try again. 00:02:03.930 --> 00:02:06.510 align:middle line:84% So m and n are the dimensions of the vectors. 00:02:06.510 --> 00:02:11.050 align:middle line:90% 00:02:11.050 --> 00:02:12.010 align:middle line:90% Anybody? 00:02:12.010 --> 00:02:13.870 align:middle line:90% I thought this was easy. 00:02:13.870 --> 00:02:16.553 align:middle line:84% The fact that it's not easy is telling me something. 00:02:16.553 --> 00:02:18.470 align:middle line:84% STEVEN JOHNSON: I think the question is, is it 00:02:18.470 --> 00:02:19.730 align:middle line:84% the transformation of the image you're 00:02:19.730 --> 00:02:20.870 align:middle line:84% talking about or the transformation 00:02:20.870 --> 00:02:23.550 align:middle line:84% of the coordinates of the image that you're talking about? 00:02:23.550 --> 00:02:25.970 align:middle line:84% ALAN EDELMAN: Does that make a difference? 00:02:25.970 --> 00:02:27.365 align:middle line:90% Right. 00:02:27.365 --> 00:02:29.918 align:middle line:84% STEVEN JOHNSON: Is it image in and image out or coordinate in 00:02:29.918 --> 00:02:30.710 align:middle line:90% and coordinate out? 00:02:30.710 --> 00:02:32.043 align:middle line:90% Which one are you talking about? 00:02:32.043 --> 00:02:33.550 align:middle line:90% ALAN EDELMAN: Does it matter? 00:02:33.550 --> 00:02:35.415 align:middle line:84% Aren't they inverses of each other? 00:02:35.415 --> 00:02:37.290 align:middle line:84% Answer either question that Steven just said. 00:02:37.290 --> 00:02:40.850 align:middle line:90% 00:02:40.850 --> 00:02:42.690 align:middle line:90% What do you guys think? 00:02:42.690 --> 00:02:45.790 align:middle line:84% Or is the question just not clear? 00:02:45.790 --> 00:02:47.500 align:middle line:90% Yeah, it's from R2 to R2. 00:02:47.500 --> 00:02:48.970 align:middle line:90% That's whatever. 00:02:48.970 --> 00:02:50.920 align:middle line:84% I mean, don't overthink the question. 00:02:50.920 --> 00:02:52.870 align:middle line:90% Right? 00:02:52.870 --> 00:02:54.520 align:middle line:90% right I've got this Julia. 00:02:54.520 --> 00:02:56.080 align:middle line:90% I'm going to hold up my phone. 00:02:56.080 --> 00:02:57.700 align:middle line:90% Steven, maybe you can see. 00:02:57.700 --> 00:02:59.300 align:middle line:90% I've got this Julia sticker. 00:02:59.300 --> 00:02:59.800 align:middle line:90% Right? 00:02:59.800 --> 00:03:03.760 align:middle line:84% It's in the plane, this plane, this two dimensional plane 00:03:03.760 --> 00:03:05.080 align:middle line:90% of the phone. 00:03:05.080 --> 00:03:05.800 align:middle line:90% Right? 00:03:05.800 --> 00:03:12.280 align:middle line:84% And this rotation is a mapping from R2 to R2. 00:03:12.280 --> 00:03:14.660 align:middle line:84% I mean, if you want to focus on-- 00:03:14.660 --> 00:03:17.860 align:middle line:84% I suppose if I have the origin in the middle of the phone 00:03:17.860 --> 00:03:19.810 align:middle line:84% and I rotate, this is a linear map. 00:03:19.810 --> 00:03:21.605 align:middle line:90% Right? 00:03:21.605 --> 00:03:22.980 align:middle line:84% STEVEN JOHNSON: Because Alan, you 00:03:22.980 --> 00:03:24.790 align:middle line:84% could also talk about the transformation that 00:03:24.790 --> 00:03:27.280 align:middle line:84% takes the image of the corgi in and gives the rotated image 00:03:27.280 --> 00:03:28.450 align:middle line:90% of the corgi out. 00:03:28.450 --> 00:03:31.450 align:middle line:84% ALAN EDELMAN: Isn't that what I was-- that was exactly that-- 00:03:31.450 --> 00:03:33.347 align:middle line:84% STEVEN JOHNSON: But then the vector space 00:03:33.347 --> 00:03:35.680 align:middle line:84% is the number of pixels in and the number of pixels out. 00:03:35.680 --> 00:03:37.900 align:middle line:90% 00:03:37.900 --> 00:03:38.900 align:middle line:90% ALAN EDELMAN: All right. 00:03:38.900 --> 00:03:41.192 align:middle line:84% This is going down a rat hole that I don't want to go-- 00:03:41.192 --> 00:03:41.930 align:middle line:90% I don't agree. 00:03:41.930 --> 00:03:45.327 align:middle line:90% But I think-- yeah. 00:03:45.327 --> 00:03:46.910 align:middle line:84% All I really wanted to do-- all right, 00:03:46.910 --> 00:03:48.460 align:middle line:84% I'm going to go to the blackboard. 00:03:48.460 --> 00:03:51.700 align:middle line:90% 00:03:51.700 --> 00:03:54.150 align:middle line:84% I'm going to go to the blackboard then. 00:03:54.150 --> 00:03:58.155 align:middle line:84% And let's turn on the lights on the blackboard here. 00:03:58.155 --> 00:04:02.890 align:middle line:90% 00:04:02.890 --> 00:04:03.540 align:middle line:90% All right. 00:04:03.540 --> 00:04:08.740 align:middle line:84% So for years in linear algebra classes-- 00:04:08.740 --> 00:04:10.360 align:middle line:90% and I'm sure you've seen this-- 00:04:10.360 --> 00:04:15.170 align:middle line:90% people will write R2. 00:04:15.170 --> 00:04:15.670 align:middle line:90% Right? 00:04:15.670 --> 00:04:17.920 align:middle line:84% And then they would put down-- if it's linear algebra, 00:04:17.920 --> 00:04:21.355 align:middle line:84% you'd have a matrix A. And people would-- 00:04:21.355 --> 00:04:24.560 align:middle line:84% you would draw a vector here, like a vector x. 00:04:24.560 --> 00:04:25.060 align:middle line:90% OK? 00:04:25.060 --> 00:04:27.340 align:middle line:84% And then you would have the image plane, 00:04:27.340 --> 00:04:29.770 align:middle line:90% also R2 in this case. 00:04:29.770 --> 00:04:30.490 align:middle line:90% Right? 00:04:30.490 --> 00:04:33.010 align:middle line:90% And x goes to ax. 00:04:33.010 --> 00:04:33.520 align:middle line:90% Right? 00:04:33.520 --> 00:04:40.990 align:middle line:84% And if you have another vector y, that might go to ay. 00:04:40.990 --> 00:04:44.180 align:middle line:84% You've all seen this in a linear algebra class, right? 00:04:44.180 --> 00:04:44.680 align:middle line:90% OK. 00:04:44.680 --> 00:04:48.370 align:middle line:84% And if anybody is familiar with the cover 00:04:48.370 --> 00:04:49.780 align:middle line:90% of Gil Strang's book-- 00:04:49.780 --> 00:04:52.850 align:middle line:84% which maybe I should just bring up while we're at it. 00:04:52.850 --> 00:04:54.550 align:middle line:84% So if you're familiar with the cover 00:04:54.550 --> 00:04:57.790 align:middle line:84% of Strang's linear algebra book, which is the textbook that's 00:04:57.790 --> 00:05:01.850 align:middle line:84% used right here at MIT and all over the world, 00:05:01.850 --> 00:05:05.440 align:middle line:84% one of the versions of the book has these houses, 00:05:05.440 --> 00:05:07.430 align:middle line:90% this one right here. 00:05:07.430 --> 00:05:10.260 align:middle line:84% OK, so maybe you've seen this version. 00:05:10.260 --> 00:05:11.700 align:middle line:90% Right here with the-- 00:05:11.700 --> 00:05:16.000 align:middle line:84% and this is meant to indicate the transformation of a house. 00:05:16.000 --> 00:05:16.500 align:middle line:90% Right? 00:05:16.500 --> 00:05:20.968 align:middle line:84% And so you have a house and some windows and a chimney 00:05:20.968 --> 00:05:21.510 align:middle line:90% or something. 00:05:21.510 --> 00:05:22.635 align:middle line:90% I don't know what you have. 00:05:22.635 --> 00:05:24.750 align:middle line:90% And the image-- let's say-- 00:05:24.750 --> 00:05:28.500 align:middle line:84% I mean, just to be clear, if this house was a square 00:05:28.500 --> 00:05:31.650 align:middle line:84% and this was your everyday general matrix, 00:05:31.650 --> 00:05:33.780 align:middle line:84% what would the image look like over here? 00:05:33.780 --> 00:05:34.935 align:middle line:90% Just generally speaking. 00:05:34.935 --> 00:05:38.750 align:middle line:90% 00:05:38.750 --> 00:05:42.520 align:middle line:84% If you transform a square with a matrix, what does it look like? 00:05:42.520 --> 00:05:44.290 align:middle line:90% Just what shape? 00:05:44.290 --> 00:05:47.390 align:middle line:84% It's a word you learned in high school or elementary school. 00:05:47.390 --> 00:05:49.890 align:middle line:90% It's not a hard question. 00:05:49.890 --> 00:05:51.830 align:middle line:84% It's a parallelogram of some kind, right? 00:05:51.830 --> 00:05:54.680 align:middle line:84% I don't know where it'll be, but it'll be some sort of-- 00:05:54.680 --> 00:05:57.710 align:middle line:84% the square will become a parallelogram of some sort. 00:05:57.710 --> 00:05:58.880 align:middle line:90% I don't know where it'll be. 00:05:58.880 --> 00:06:01.370 align:middle line:84% The square window will also be a parallelogram 00:06:01.370 --> 00:06:04.100 align:middle line:90% with parallel parts. 00:06:04.100 --> 00:06:05.900 align:middle line:84% That little chimney or I don't know 00:06:05.900 --> 00:06:09.740 align:middle line:84% what that is will get skewed in some way as well, right? 00:06:09.740 --> 00:06:13.430 align:middle line:84% This is what happens when you take every point. 00:06:13.430 --> 00:06:15.810 align:middle line:84% I mean, I didn't want to get into pixels. 00:06:15.810 --> 00:06:18.230 align:middle line:84% I mean, just every point in the plane 00:06:18.230 --> 00:06:21.380 align:middle line:84% is being transformed to another point in the plane, right? 00:06:21.380 --> 00:06:25.265 align:middle line:84% This is what we would call a linear map from R2 to R2. 00:06:25.265 --> 00:06:29.140 align:middle line:90% 00:06:29.140 --> 00:06:32.380 align:middle line:90% Linear map from R2 to R2. 00:06:32.380 --> 00:06:34.360 align:middle line:90% It emphasizes about a matrix. 00:06:34.360 --> 00:06:35.890 align:middle line:84% Not the fact that a matrix is just 00:06:35.890 --> 00:06:37.660 align:middle line:90% a two by two four elements. 00:06:37.660 --> 00:06:40.300 align:middle line:84% It emphasizes the fact that a matrix is something 00:06:40.300 --> 00:06:43.140 align:middle line:90% that transforms space. 00:06:43.140 --> 00:06:43.640 align:middle line:90% OK? 00:06:43.640 --> 00:06:46.250 align:middle line:84% And that's all I wanted to say, and I 00:06:46.250 --> 00:06:48.597 align:middle line:84% thought that this was a basic element of linear algebra. 00:06:48.597 --> 00:06:50.180 align:middle line:84% I'm a little worried that just the way 00:06:50.180 --> 00:06:52.800 align:middle line:84% I asked the question wasn't clear to people. 00:06:52.800 --> 00:06:57.710 align:middle line:84% So if I had a picture of Philip over here-- which I could never 00:06:57.710 --> 00:06:59.240 align:middle line:90% draw a good picture of Philip. 00:06:59.240 --> 00:07:00.590 align:middle line:90% Here's a Minecraft Philip. 00:07:00.590 --> 00:07:05.690 align:middle line:90% 00:07:05.690 --> 00:07:07.550 align:middle line:84% It's a terrible Minecraft picture of Philip. 00:07:07.550 --> 00:07:08.180 align:middle line:90% Right? 00:07:08.180 --> 00:07:12.050 align:middle line:84% It would get transformed to some skewed version of Philip 00:07:12.050 --> 00:07:14.970 align:middle line:84% on the other side, and that's what a linear map is. 00:07:14.970 --> 00:07:15.470 align:middle line:90% OK. 00:07:15.470 --> 00:07:17.600 align:middle line:84% So am I saying things you don't know? 00:07:17.600 --> 00:07:19.730 align:middle line:90% You all know this, right? 00:07:19.730 --> 00:07:21.500 align:middle line:90% OK. 00:07:21.500 --> 00:07:27.750 align:middle line:84% So with that understanding, let's go back, 00:07:27.750 --> 00:07:31.570 align:middle line:90% turn the lights down low again. 00:07:31.570 --> 00:07:33.690 align:middle line:84% So with that understanding, I just 00:07:33.690 --> 00:07:36.120 align:middle line:84% wanted to show you a few linear and nonlinear maps. 00:07:36.120 --> 00:07:40.120 align:middle line:90% 00:07:40.120 --> 00:07:45.280 align:middle line:84% Not to spend too much time on this, but maybe it's worth it. 00:07:45.280 --> 00:07:45.780 align:middle line:90% Yeah. 00:07:45.780 --> 00:07:50.610 align:middle line:84% So a rotation is a linear map from R2 to R2. 00:07:50.610 --> 00:07:52.620 align:middle line:84% Is there still any confusion about pixels 00:07:52.620 --> 00:07:53.770 align:middle line:90% or any other nonsense? 00:07:53.770 --> 00:07:57.380 align:middle line:90% I just want to think of-- 00:07:57.380 --> 00:08:00.530 align:middle line:84% to be very, very clear, I'm just thinking of every point-- 00:08:00.530 --> 00:08:01.610 align:middle line:90% this is the center. 00:08:01.610 --> 00:08:04.490 align:middle line:84% His little black nose here is the center of the plane. 00:08:04.490 --> 00:08:07.002 align:middle line:84% And you can always draw a vector. 00:08:07.002 --> 00:08:08.960 align:middle line:84% I guess, oh, I could use Zoom to draw a vector. 00:08:08.960 --> 00:08:11.780 align:middle line:90% This is great. 00:08:11.780 --> 00:08:15.530 align:middle line:90% Let's draw a vector right here. 00:08:15.530 --> 00:08:18.920 align:middle line:84% So if I'm at the origin, every point, like the tip-- 00:08:18.920 --> 00:08:21.650 align:middle line:84% well, I can't draw a straight line, but that's my fault. 00:08:21.650 --> 00:08:24.920 align:middle line:84% But this is meant to be a vector right to the tip of his ear. 00:08:24.920 --> 00:08:28.070 align:middle line:84% And then that vector will stay where it is, 00:08:28.070 --> 00:08:34.419 align:middle line:90% but if I rotate the plane, say-- 00:08:34.419 --> 00:08:36.250 align:middle line:84% you know, let's not rotate that much. 00:08:36.250 --> 00:08:40.360 align:middle line:90% Let's rotate maybe 30 degrees. 00:08:40.360 --> 00:08:43.270 align:middle line:84% You see that his ear has moved 30 degrees. 00:08:43.270 --> 00:08:44.530 align:middle line:90% Oops, oops, oops. 00:08:44.530 --> 00:08:45.280 align:middle line:90% Everybody with me? 00:08:45.280 --> 00:08:46.510 align:middle line:90% Any questions? 00:08:46.510 --> 00:08:47.395 align:middle line:90% OK. 00:08:47.395 --> 00:08:49.010 align:middle line:90% Let's not belabor this point. 00:08:49.010 --> 00:08:54.710 align:middle line:84% So the next thing I want you to do was-- 00:08:54.710 --> 00:08:56.360 align:middle line:84% the next thing I wanted to do-- let's 00:08:56.360 --> 00:08:59.270 align:middle line:84% just clear this-- is show you some other linear 00:08:59.270 --> 00:09:00.570 align:middle line:90% transformations. 00:09:00.570 --> 00:09:03.320 align:middle line:84% So instead of rotating is a little bit too simple, 00:09:03.320 --> 00:09:07.190 align:middle line:84% so another linear transformation is this hyperbolic rotate. 00:09:07.190 --> 00:09:11.513 align:middle line:84% So let me just switch to hyperbolic rotate. 00:09:11.513 --> 00:09:12.430 align:middle line:90% Yet again, forgive me. 00:09:12.430 --> 00:09:15.200 align:middle line:90% I just forgot my glasses today. 00:09:15.200 --> 00:09:17.080 align:middle line:84% So I'm kind of winging this a little bit. 00:09:17.080 --> 00:09:19.480 align:middle line:84% So hyperbolic rotate is a little less familiar 00:09:19.480 --> 00:09:21.760 align:middle line:90% to a lot of people. 00:09:21.760 --> 00:09:23.290 align:middle line:90% So when you rotate-- 00:09:23.290 --> 00:09:26.240 align:middle line:84% just to say something also, I think, very obvious. 00:09:26.240 --> 00:09:26.740 align:middle line:90% Yeah. 00:09:26.740 --> 00:09:29.590 align:middle line:84% When I rotate it, you see the red circle? 00:09:29.590 --> 00:09:31.720 align:middle line:84% What happens to Philip's ears as I rotate? 00:09:31.720 --> 00:09:38.470 align:middle line:90% 00:09:38.470 --> 00:09:40.370 align:middle line:84% If I rotate, the ears follow the red circle. 00:09:40.370 --> 00:09:40.870 align:middle line:90% Right? 00:09:40.870 --> 00:09:43.900 align:middle line:84% They just move along the red circle. 00:09:43.900 --> 00:09:44.800 align:middle line:90% OK. 00:09:44.800 --> 00:09:46.060 align:middle line:90% Now I want you to now focus. 00:09:46.060 --> 00:09:47.590 align:middle line:90% I'm now doing hyperbolic rotate. 00:09:47.590 --> 00:09:49.130 align:middle line:90% I want you to focus on his eyes. 00:09:49.130 --> 00:09:51.280 align:middle line:84% You see the eyes are on a hyperbola? 00:09:51.280 --> 00:09:52.540 align:middle line:90% OK? 00:09:52.540 --> 00:09:57.055 align:middle line:84% You'll notice that as you move, his eyes always actually-- 00:09:57.055 --> 00:09:59.350 align:middle line:84% this is what's called a hyperbolic rotation. 00:09:59.350 --> 00:10:01.330 align:middle line:84% And his eyes, even as the picture 00:10:01.330 --> 00:10:04.750 align:middle line:84% gets more and more skewed, so these squares become 00:10:04.750 --> 00:10:06.460 align:middle line:84% flatter and flatter parallelograms, 00:10:06.460 --> 00:10:07.510 align:middle line:90% you could still see-- 00:10:07.510 --> 00:10:08.320 align:middle line:90% I'll go backwards. 00:10:08.320 --> 00:10:11.470 align:middle line:84% You could still see that his eyes are always 00:10:11.470 --> 00:10:13.675 align:middle line:84% following a hyperbola as opposed to a rotation, 00:10:13.675 --> 00:10:17.270 align:middle line:84% an ordinary rotation where the red ears would follow a circle. 00:10:17.270 --> 00:10:17.770 align:middle line:90% Right? 00:10:17.770 --> 00:10:23.020 align:middle line:84% So this is also a linear transformation in that 00:10:23.020 --> 00:10:26.950 align:middle line:84% literally if you draw a vector from his nose to any pixel 00:10:26.950 --> 00:10:29.470 align:middle line:84% and another vector from his nose to any pixel 00:10:29.470 --> 00:10:31.870 align:middle line:84% and you want to know, if you add those two 00:10:31.870 --> 00:10:33.310 align:middle line:84% vectors to create a third vector, 00:10:33.310 --> 00:10:35.630 align:middle line:90% then it will add in the image. 00:10:35.630 --> 00:10:36.130 align:middle line:90% OK? 00:10:36.130 --> 00:10:40.370 align:middle line:84% And now let me just show you a nonlinear-- 00:10:40.370 --> 00:10:42.840 align:middle line:90% some nonlinear transformations. 00:10:42.840 --> 00:10:46.320 align:middle line:90% So here's a nonlinear shear. 00:10:46.320 --> 00:10:51.030 align:middle line:84% So what this does is shears things, and you could-- 00:10:51.030 --> 00:10:54.200 align:middle line:84% it's already-- and so squares are no longer guaranteed 00:10:54.200 --> 00:10:55.130 align:middle line:90% to be parallelograms. 00:10:55.130 --> 00:10:58.610 align:middle line:84% In fact, they're not parallelograms. 00:10:58.610 --> 00:11:01.120 align:middle line:90% 00:11:01.120 --> 00:11:03.250 align:middle line:84% It's much more of a-- here he is. 00:11:03.250 --> 00:11:06.240 align:middle line:90% Poor Philip is being folded up. 00:11:06.240 --> 00:11:07.597 align:middle line:90% And then there's this warp. 00:11:07.597 --> 00:11:09.430 align:middle line:84% I mean, we could put in any-- this is Julia. 00:11:09.430 --> 00:11:11.680 align:middle line:84% You could put in any function you want. 00:11:11.680 --> 00:11:15.790 align:middle line:84% So here's a warping, which is also nonlinear. 00:11:15.790 --> 00:11:20.710 align:middle line:84% So here's the original picture, and you 00:11:20.710 --> 00:11:24.990 align:middle line:84% can see different parts of the picture will move differently. 00:11:24.990 --> 00:11:27.560 align:middle line:84% And so just to make it clear what's 00:11:27.560 --> 00:11:30.260 align:middle line:84% going on with matrix calculus, everybody 00:11:30.260 --> 00:11:33.175 align:middle line:84% understands that this is a mapping from two vectors to two 00:11:33.175 --> 00:11:33.800 align:middle line:90% vectors, right? 00:11:33.800 --> 00:11:35.210 align:middle line:90% From R2 to R2. 00:11:35.210 --> 00:11:35.720 align:middle line:90% Right? 00:11:35.720 --> 00:11:38.570 align:middle line:84% So if you form the Jacobian matrix, 00:11:38.570 --> 00:11:40.325 align:middle line:84% what's the size of that Jacobian matrix? 00:11:40.325 --> 00:11:43.710 align:middle line:90% 00:11:43.710 --> 00:11:45.440 align:middle line:90% A two by two Jacobian matrix. 00:11:45.440 --> 00:11:50.990 align:middle line:84% And the Jacobian matrix is different at every point 00:11:50.990 --> 00:11:52.880 align:middle line:90% in the plane, right? 00:11:52.880 --> 00:11:55.460 align:middle line:84% When you're linear, the two by two Jacobian 00:11:55.460 --> 00:11:56.910 align:middle line:84% is constant throughout the plane. 00:11:56.910 --> 00:11:59.340 align:middle line:84% It's actually-- like the picture on the board, 00:11:59.340 --> 00:12:02.090 align:middle line:90% it's just the matrix everywhere. 00:12:02.090 --> 00:12:05.000 align:middle line:84% But when you have a nonlinear map like this one, 00:12:05.000 --> 00:12:09.080 align:middle line:84% the two by two Jacobian depends on where you're starting. 00:12:09.080 --> 00:12:10.602 align:middle line:90% And so if one could zoom-- 00:12:10.602 --> 00:12:11.810 align:middle line:90% I don't know if one can zoom. 00:12:11.810 --> 00:12:12.310 align:middle line:90% Yeah. 00:12:12.310 --> 00:12:15.020 align:middle line:84% If you were to zoom in, what it's saying 00:12:15.020 --> 00:12:16.400 align:middle line:90% is that a zoomed in picture-- 00:12:16.400 --> 00:12:18.067 align:middle line:84% I don't know if you'd be able to see it, 00:12:18.067 --> 00:12:20.270 align:middle line:84% that I'm trying to Chrome zoom in. 00:12:20.270 --> 00:12:22.640 align:middle line:84% But what a Jacobian is saying is that if you 00:12:22.640 --> 00:12:25.490 align:middle line:84% look in a small little patch, squares 00:12:25.490 --> 00:12:26.900 align:middle line:90% will become parallelograms. 00:12:26.900 --> 00:12:29.510 align:middle line:90% Circles will become ellipses. 00:12:29.510 --> 00:12:30.170 align:middle line:90% Right? 00:12:30.170 --> 00:12:32.150 align:middle line:84% And they'll do it in a constant way. 00:12:32.150 --> 00:12:34.760 align:middle line:84% But as you move away from that small patch, 00:12:34.760 --> 00:12:37.190 align:middle line:84% it'll do it differently in different places. 00:12:37.190 --> 00:12:43.380 align:middle line:84% Is there any questions about what a map from R2 to R2 00:12:43.380 --> 00:12:44.580 align:middle line:90% looks like? 00:12:44.580 --> 00:12:48.720 align:middle line:84% And of course, I can move my hands around and talk 00:12:48.720 --> 00:12:50.850 align:middle line:90% about maps from R3 to R3. 00:12:50.850 --> 00:12:53.900 align:middle line:84% I mean, you could easily imagine-- 00:12:53.900 --> 00:12:57.420 align:middle line:84% if we turn our heads, in effect, what we see in our eyes 00:12:57.420 --> 00:12:58.960 align:middle line:90% is a rotation of R3. 00:12:58.960 --> 00:12:59.460 align:middle line:90% Right? 00:12:59.460 --> 00:13:03.600 align:middle line:84% That's a linear map as long as you fix on one point. 00:13:03.600 --> 00:13:07.140 align:middle line:84% But one could imagine nonlinear maps of R3. 00:13:07.140 --> 00:13:10.140 align:middle line:84% It's not something I could easily do with my hands. 00:13:10.140 --> 00:13:12.240 align:middle line:90% But you could-- right? 00:13:12.240 --> 00:13:13.890 align:middle line:90% And so that's R3 to R3. 00:13:13.890 --> 00:13:15.930 align:middle line:84% And of course, in general, we don't even 00:13:15.930 --> 00:13:18.270 align:middle line:84% have to have the same dimension going in and going out. 00:13:18.270 --> 00:13:21.210 align:middle line:90% It's Rn in, Rm out. 00:13:21.210 --> 00:13:23.440 align:middle line:90% OK? 00:13:23.440 --> 00:13:23.940 align:middle line:90% All right. 00:13:23.940 --> 00:13:26.880 align:middle line:84% That's all I wanted to say just to start things off to get 00:13:26.880 --> 00:13:29.560 align:middle line:90% people used to these maps. 00:13:29.560 --> 00:13:30.060 align:middle line:90% All right. 00:13:30.060 --> 00:13:32.275 align:middle line:84% Steven, should I turn it over to you? 00:13:32.275 --> 00:13:33.150 align:middle line:90% STEVEN JOHNSON: Yeah. 00:13:33.150 --> 00:13:34.500 align:middle line:90% Sounds good. 00:13:34.500 --> 00:13:38.550 align:middle line:84% So I want to talk again today about this generalization 00:13:38.550 --> 00:13:45.660 align:middle line:84% of the notion of a derivative or reinterpretation or revisiting 00:13:45.660 --> 00:13:47.920 align:middle line:84% of a derivative as a linear operator. 00:13:47.920 --> 00:13:49.677 align:middle line:90% So if you have any function-- 00:13:49.677 --> 00:13:51.010 align:middle line:90% remember what we said last time. 00:13:51.010 --> 00:13:53.250 align:middle line:84% So if you have any function f of x and you 00:13:53.250 --> 00:13:56.140 align:middle line:84% change the input by a little amount, of course, 00:13:56.140 --> 00:13:58.940 align:middle line:84% there's a little change in the output. 00:13:58.940 --> 00:14:02.530 align:middle line:84% And if the change in the input is very small, 00:14:02.530 --> 00:14:04.530 align:middle line:84% then we can approximate the change 00:14:04.530 --> 00:14:09.210 align:middle line:84% in the output by something that's linear in the input 00:14:09.210 --> 00:14:10.530 align:middle line:90% where it's f. 00:14:10.530 --> 00:14:12.630 align:middle line:90% And so this is going to be-- 00:14:12.630 --> 00:14:15.720 align:middle line:84% basically it's linear in the input plus higher order terms, 00:14:15.720 --> 00:14:18.270 align:middle line:84% but it'll be convenient just to drop these higher order terms 00:14:18.270 --> 00:14:21.010 align:middle line:84% and the notation will be this differential notation. 00:14:21.010 --> 00:14:24.570 align:middle line:84% So we think of this as a really small infinitesimal change 00:14:24.570 --> 00:14:29.120 align:middle line:84% in an arbitrary direction if x is a vector. 00:14:29.120 --> 00:14:33.000 align:middle line:84% And so when we take f of x plus d minus f of x, 00:14:33.000 --> 00:14:34.700 align:middle line:84% we're just going to drop implicitly 00:14:34.700 --> 00:14:37.700 align:middle line:84% any term that's higher order than linear, 00:14:37.700 --> 00:14:41.100 align:middle line:84% has a dx squared or a dx to the 1.5 or anything like that. 00:14:41.100 --> 00:14:44.540 align:middle line:84% And so the definition of the derivative f prime 00:14:44.540 --> 00:14:47.630 align:middle line:84% is going to be the linear operator acting 00:14:47.630 --> 00:14:50.480 align:middle line:84% on dx, the change in the input that gives you 00:14:50.480 --> 00:14:51.960 align:middle line:90% the change in the output. 00:14:51.960 --> 00:14:53.420 align:middle line:90% And so that gives us-- 00:14:53.420 --> 00:14:57.290 align:middle line:84% it's equivalent to the 18.01 definition of a derivative, 00:14:57.290 --> 00:14:59.780 align:middle line:90% for if f is a scalar function. 00:14:59.780 --> 00:15:02.780 align:middle line:84% A linear operator is just a number, the slope, 00:15:02.780 --> 00:15:06.960 align:middle line:84% that multiplies by the change in x to give you the change in y. 00:15:06.960 --> 00:15:11.310 align:middle line:84% But it also works for multivariable functions. 00:15:11.310 --> 00:15:16.070 align:middle line:84% So if the input x is a vector and the output is a scalar, 00:15:16.070 --> 00:15:18.530 align:middle line:90% then this is a linear operator. 00:15:18.530 --> 00:15:22.450 align:middle line:84% The derivative is a linear operator acting on a vector. 00:15:22.450 --> 00:15:26.540 align:middle line:84% dx is an arbitrary change in an arbitrary direction 00:15:26.540 --> 00:15:28.980 align:middle line:84% in this input vector that gives us 00:15:28.980 --> 00:15:31.320 align:middle line:84% the change in f, which is a scalar. 00:15:31.320 --> 00:15:36.720 align:middle line:84% And a linear operator on a vector that gives a scalar 00:15:36.720 --> 00:15:39.450 align:middle line:84% is a row vector or a one row matrix 00:15:39.450 --> 00:15:41.623 align:middle line:90% or a co-vector or linear form. 00:15:41.623 --> 00:15:43.290 align:middle line:84% There's all sorts of fancy names for it, 00:15:43.290 --> 00:15:45.810 align:middle line:84% but it's basically-- it's a dot product with something. 00:15:45.810 --> 00:15:46.350 align:middle line:90% Right? 00:15:46.350 --> 00:15:48.450 align:middle line:84% That's the only way to get a vector 00:15:48.450 --> 00:15:50.280 align:middle line:84% and do a linear operation and get a scalar. 00:15:50.280 --> 00:15:52.950 align:middle line:84% And the thing we're taking with the dot product with we 00:15:52.950 --> 00:15:54.100 align:middle line:90% call the gradient. 00:15:54.100 --> 00:15:58.470 align:middle line:84% So the gradient, this df is the gradient dot dx, 00:15:58.470 --> 00:16:02.400 align:middle line:84% or equivalently f prime is the transpose of the gradient. 00:16:02.400 --> 00:16:04.530 align:middle line:84% It's the row vector, the linear operator 00:16:04.530 --> 00:16:09.615 align:middle line:84% that you act that row vector on a column vector on a dx. 00:16:09.615 --> 00:16:13.380 align:middle line:90% It gives you the df. 00:16:13.380 --> 00:16:14.780 align:middle line:90% Right? 00:16:14.780 --> 00:16:16.680 align:middle line:84% This works for ordinary column vectors, 00:16:16.680 --> 00:16:19.530 align:middle line:84% and we're going to think of other vector spaces very soon. 00:16:19.530 --> 00:16:20.070 align:middle line:90% OK? 00:16:20.070 --> 00:16:24.450 align:middle line:84% And then the thing I just started at the very end 00:16:24.450 --> 00:16:26.700 align:middle line:90% of the lecture was suppose-- 00:16:26.700 --> 00:16:28.560 align:middle line:90% so this is not 18.06 revisited. 00:16:28.560 --> 00:16:30.870 align:middle line:90% This is 18.02 revisited. 00:16:30.870 --> 00:16:32.640 align:middle line:90% This is still 18.02. 00:16:32.640 --> 00:16:33.364 align:middle line:90% Oops. 00:16:33.364 --> 00:16:33.864 align:middle line:90% Black. 00:16:33.864 --> 00:16:36.530 align:middle line:90% 00:16:36.530 --> 00:16:40.560 align:middle line:84% The other kind of function you deal with in 18.02 00:16:40.560 --> 00:16:45.180 align:middle line:84% are functions that take vectors in and vectors out. 00:16:45.180 --> 00:16:48.960 align:middle line:84% And so for example in an integral and multi-dimensional 00:16:48.960 --> 00:16:51.840 align:middle line:84% integral, if you change coordinates, you have x and y 00:16:51.840 --> 00:16:53.550 align:middle line:84% and you get some new x and y, that's 00:16:53.550 --> 00:16:56.040 align:middle line:84% a function that takes a vector in and a vector out. 00:16:56.040 --> 00:16:58.170 align:middle line:90% May or may not be linear. 00:16:58.170 --> 00:17:00.870 align:middle line:84% So that's where the function-- the inputs, 00:17:00.870 --> 00:17:04.069 align:middle line:84% I'm going to say the inputs live in Rn 00:17:04.069 --> 00:17:06.560 align:middle line:90% and the outputs live in Rm. 00:17:06.560 --> 00:17:10.430 align:middle line:84% So maybe there's a different number of inputs and outputs. 00:17:10.430 --> 00:17:13.490 align:middle line:84% And then when we ask, same thing. 00:17:13.490 --> 00:17:16.440 align:middle line:84% If we change the input vector by a little bit, 00:17:16.440 --> 00:17:20.240 align:middle line:84% then there's some change in the output vector which we call df. 00:17:20.240 --> 00:17:23.119 align:middle line:84% And the change in the output vector has m components. 00:17:23.119 --> 00:17:26.060 align:middle line:84% The change in the input vector has n components 00:17:26.060 --> 00:17:29.120 align:middle line:84% in some arbitrary direction, and f prime 00:17:29.120 --> 00:17:33.410 align:middle line:84% is the linear operator that goes from the change in the input, n 00:17:33.410 --> 00:17:37.590 align:middle line:84% components, to a change in the output, m components. 00:17:37.590 --> 00:17:39.240 align:middle line:90% And the only kind-- 00:17:39.240 --> 00:17:42.360 align:middle line:84% and the way we typically describe such a linear operator 00:17:42.360 --> 00:17:43.900 align:middle line:90% is an m by n matrix. 00:17:43.900 --> 00:17:44.400 align:middle line:90% Right? 00:17:44.400 --> 00:17:46.710 align:middle line:84% So that's what you would multiply 00:17:46.710 --> 00:17:50.640 align:middle line:84% by a vector with n components to get a vector with components. 00:17:50.640 --> 00:17:54.660 align:middle line:84% And we call that matrix, that m by n matrix, the Jacobian J. 00:17:54.660 --> 00:17:58.890 align:middle line:84% And in fact, you can also think of it 00:17:58.890 --> 00:18:07.370 align:middle line:90% in terms of the entries of J. 00:18:07.370 --> 00:18:10.910 align:middle line:84% If we ask what the entry Jij is, this notation 00:18:10.910 --> 00:18:16.920 align:middle line:90% means row i, column j. 00:18:16.920 --> 00:18:19.470 align:middle line:90% 00:18:19.470 --> 00:18:21.465 align:middle line:90% Right? 00:18:21.465 --> 00:18:24.150 align:middle line:84% This is going to be equivalent to what you learned, hopefully, 00:18:24.150 --> 00:18:27.900 align:middle line:84% in 18.02 although not every version of 1802, apparently. 00:18:27.900 --> 00:18:32.910 align:middle line:84% It's the same thing as partial fi, 00:18:32.910 --> 00:18:36.940 align:middle line:84% the derivative of the i-th output with respect 00:18:36.940 --> 00:18:41.650 align:middle line:90% to the j-th input. 00:18:41.650 --> 00:18:43.820 align:middle line:84% I always used to get these confused. 00:18:43.820 --> 00:18:46.750 align:middle line:84% Is it, which one's the rows, which one's the columns. 00:18:46.750 --> 00:18:50.210 align:middle line:90% The dfi, dxj, dfj, dxi. 00:18:50.210 --> 00:18:54.340 align:middle line:84% And in 18.02, people just take the determinant 00:18:54.340 --> 00:18:57.225 align:middle line:84% of this matrix for integration, and then it doesn't matter. 00:18:57.225 --> 00:18:58.600 align:middle line:84% Because you transpose the matrix, 00:18:58.600 --> 00:19:00.017 align:middle line:84% it doesn't change the determinant. 00:19:00.017 --> 00:19:03.340 align:middle line:84% But as a linear operator, it really matters. 00:19:03.340 --> 00:19:07.000 align:middle line:84% The rows-- here we can even write this. 00:19:07.000 --> 00:19:09.910 align:middle line:90% Let me just do it in blue. 00:19:09.910 --> 00:19:13.480 align:middle line:84% If you think of a matrix as a linear operator, 00:19:13.480 --> 00:19:19.720 align:middle line:84% the rows are the outputs which are the df's, and the columns 00:19:19.720 --> 00:19:23.650 align:middle line:84% are the inputs, which are the dx's. 00:19:23.650 --> 00:19:27.460 align:middle line:84% So it really matters that the rows are the different 00:19:27.460 --> 00:19:30.350 align:middle line:84% f's and the columns are different x's. 00:19:30.350 --> 00:19:32.350 align:middle line:84% Let me just do an example, since we were doing-- 00:19:32.350 --> 00:19:35.300 align:middle line:90% 00:19:35.300 --> 00:19:39.160 align:middle line:90% this is to m equals n equals 2. 00:19:39.160 --> 00:19:44.650 align:middle line:84% And so m equals n equals 2, since we were just doing 00:19:44.650 --> 00:19:45.970 align:middle line:90% transformations of the plane. 00:19:45.970 --> 00:19:56.000 align:middle line:84% So we're going from inputs in R2 and then f 00:19:56.000 --> 00:20:01.055 align:middle line:84% is taking us to outputs also in R2. 00:20:01.055 --> 00:20:02.930 align:middle line:84% So we're just mapping the plane to the plane, 00:20:02.930 --> 00:20:05.300 align:middle line:84% but it maybe it's not a linear transformation. 00:20:05.300 --> 00:20:06.020 align:middle line:90% All right? 00:20:06.020 --> 00:20:13.990 align:middle line:84% So if we thought of this in 18.02 style, component 00:20:13.990 --> 00:20:22.990 align:middle line:90% by component, then we have-- 00:20:22.990 --> 00:20:28.640 align:middle line:84% f of x, we can think of it as two functions, f1 00:20:28.640 --> 00:20:31.145 align:middle line:90% of x and f2 of x. 00:20:31.145 --> 00:20:34.074 align:middle line:90% 00:20:34.074 --> 00:20:40.620 align:middle line:84% And x, we can think of it as two components, x1, x2. 00:20:40.620 --> 00:20:41.460 align:middle line:90% Right? 00:20:41.460 --> 00:20:45.290 align:middle line:84% And then if you want the change in f-- 00:20:45.290 --> 00:20:47.143 align:middle line:84% let's put a vector here to keep track 00:20:47.143 --> 00:20:49.310 align:middle line:84% of what things are vectors, what things are scalars. 00:20:49.310 --> 00:20:52.190 align:middle line:90% 00:20:52.190 --> 00:20:58.170 align:middle line:84% The change in f, my claim is going to be the Jacobian. 00:20:58.170 --> 00:21:05.650 align:middle line:84% And the Jacobian is now going to be our two by two Jacobian. 00:21:05.650 --> 00:21:10.610 align:middle line:84% And I guess maybe I should put outputs in blue here. 00:21:10.610 --> 00:21:12.160 align:middle line:90% All right? 00:21:12.160 --> 00:21:15.500 align:middle line:84% And the outputs are the different rows. 00:21:15.500 --> 00:21:24.690 align:middle line:84% So there's df1, df1, df1, df2, df2. 00:21:24.690 --> 00:21:34.370 align:middle line:84% And the inputs are the different columns, dx1, dx2, dx1, dx2. 00:21:34.370 --> 00:21:37.900 align:middle line:90% 00:21:37.900 --> 00:21:42.020 align:middle line:84% And then if we multiply this by any change-- 00:21:42.020 --> 00:21:45.710 align:middle line:84% so you change the input by any vector in dx in any direction. 00:21:45.710 --> 00:21:48.390 align:middle line:90% So this is our dx. 00:21:48.390 --> 00:21:49.250 align:middle line:90% Right? 00:21:49.250 --> 00:21:51.210 align:middle line:90% You can see that it works out. 00:21:51.210 --> 00:21:51.710 align:middle line:90% Right? 00:21:51.710 --> 00:21:54.110 align:middle line:84% So I don't know how I would really write this out, 00:21:54.110 --> 00:22:01.820 align:middle line:84% but if we multiply rows times columns, you get df1, dx1. 00:22:01.820 --> 00:22:04.280 align:middle line:90% Let me do it, actually. 00:22:04.280 --> 00:22:06.590 align:middle line:84% So that's-- I need a little more space, maybe. 00:22:06.590 --> 00:22:15.100 align:middle line:90% 00:22:15.100 --> 00:22:15.600 align:middle line:90% All right. 00:22:15.600 --> 00:22:17.070 align:middle line:90% What's the first row? 00:22:17.070 --> 00:22:19.830 align:middle line:90% Row times column. 00:22:19.830 --> 00:22:21.690 align:middle line:90% Let me keep my color scheme. 00:22:21.690 --> 00:22:28.720 align:middle line:90% df1, df1, df2, df2. 00:22:28.720 --> 00:22:45.540 align:middle line:90% 00:22:45.540 --> 00:22:46.710 align:middle line:90% I think this is it, right? 00:22:46.710 --> 00:22:50.790 align:middle line:84% If I do row times column, I get partial f1-- 00:22:50.790 --> 00:22:54.150 align:middle line:84% partial f1 partial x1 times sx1, which is this. 00:22:54.150 --> 00:22:56.767 align:middle line:84% Partial f2, partial x2 times dx2, which is this. 00:22:56.767 --> 00:22:59.100 align:middle line:84% And the same thing for the other row, it's just with f2. 00:22:59.100 --> 00:22:59.790 align:middle line:90% Right? 00:22:59.790 --> 00:23:01.150 align:middle line:90% Oops, and I forgot a-- 00:23:01.150 --> 00:23:02.430 align:middle line:90% I forgot a plus sign. 00:23:02.430 --> 00:23:07.390 align:middle line:90% 00:23:07.390 --> 00:23:08.530 align:middle line:90% I have to add them. 00:23:08.530 --> 00:23:09.030 align:middle line:90% All right? 00:23:09.030 --> 00:23:09.840 align:middle line:90% And you can see. 00:23:09.840 --> 00:23:13.521 align:middle line:84% So a component by component it makes sense, right? 00:23:13.521 --> 00:23:16.620 align:middle line:84% If you want the change in f1, well, you take-- 00:23:16.620 --> 00:23:19.860 align:middle line:84% its change, the rate of-- its slope with respect to x1. 00:23:19.860 --> 00:23:22.710 align:middle line:84% You multiply by the change in x1. 00:23:22.710 --> 00:23:26.850 align:middle line:84% And you also have to add the slope of f1 with x2 00:23:26.850 --> 00:23:29.280 align:middle line:84% and multiply by the change in x2 and so forth. 00:23:29.280 --> 00:23:31.530 align:middle line:84% So really, all the components match up, 00:23:31.530 --> 00:23:33.225 align:middle line:90% but this is a really-- 00:23:33.225 --> 00:23:35.430 align:middle line:84% this gets awkward very, very quickly 00:23:35.430 --> 00:23:38.640 align:middle line:84% to write things out component-wide like this. 00:23:38.640 --> 00:23:39.530 align:middle line:90% Right? 00:23:39.530 --> 00:23:42.030 align:middle line:84% It's much easier to just say, this is a linear-- think of it 00:23:42.030 --> 00:23:43.080 align:middle line:90% as a whole. 00:23:43.080 --> 00:23:46.110 align:middle line:84% It's a linear operator acting on the change of x, and that, 00:23:46.110 --> 00:23:49.680 align:middle line:84% if we want, we can write it down as a Jacobian matrix in this m 00:23:49.680 --> 00:23:51.610 align:middle line:90% by n matrix. 00:23:51.610 --> 00:23:53.340 align:middle line:84% So I want to do a couple of examples. 00:23:53.340 --> 00:23:55.960 align:middle line:90% We'll do another example. 00:23:55.960 --> 00:23:58.640 align:middle line:84% I think Alan did this the other day, but let me just-- 00:23:58.640 --> 00:24:02.530 align:middle line:90% it doesn't hurt to do it again. 00:24:02.530 --> 00:24:12.170 align:middle line:84% So suppose f of x is just a matrix times x. 00:24:12.170 --> 00:24:15.540 align:middle line:90% Suppose this is a linear. 00:24:15.540 --> 00:24:16.040 align:middle line:90% Right? 00:24:16.040 --> 00:24:21.620 align:middle line:84% And so here I'm just going to say, this is a constant matrix. 00:24:21.620 --> 00:24:24.710 align:middle line:84% ALAN EDELMAN: Like the ordinary rotation and the hyperbolic 00:24:24.710 --> 00:24:27.850 align:middle line:84% rotation would be two examples of what Steven's doing now. 00:24:27.850 --> 00:24:28.850 align:middle line:90% STEVEN JOHNSON: Exactly. 00:24:28.850 --> 00:24:31.363 align:middle line:84% So this could be a rotation matrix or something like that 00:24:31.363 --> 00:24:32.280 align:middle line:90% if this is two by two. 00:24:32.280 --> 00:24:33.822 align:middle line:84% But it doesn't have to be two by two. 00:24:33.822 --> 00:24:35.900 align:middle line:84% It doesn't even have to be a square, right? 00:24:35.900 --> 00:24:42.710 align:middle line:84% This could be in output's in Rm and the input's in-- 00:24:42.710 --> 00:24:49.980 align:middle line:84% so the outputs are in Rm and the inputs are in Rn. 00:24:49.980 --> 00:24:57.510 align:middle line:84% And therefore the A matrix is an m by n matrix in general. 00:24:57.510 --> 00:24:59.680 align:middle line:84% It could be, for example, two by two. 00:24:59.680 --> 00:25:00.220 align:middle line:90% Right? 00:25:00.220 --> 00:25:04.210 align:middle line:84% And again, let's just do this the slow way. 00:25:04.210 --> 00:25:06.370 align:middle line:84% But still much faster than 18.02. 00:25:06.370 --> 00:25:10.930 align:middle line:84% So the 18.02 way would be to write this out as 00:25:10.930 --> 00:25:12.810 align:middle line:90% in components. 00:25:12.810 --> 00:25:14.740 align:middle line:90% So you take ai-- 00:25:14.740 --> 00:25:23.870 align:middle line:90% this is fi is aij sum over-- 00:25:23.870 --> 00:25:27.680 align:middle line:90% let me keep my color scheme. 00:25:27.680 --> 00:25:36.050 align:middle line:84% aij xj and sum over j equals 1 to n. 00:25:36.050 --> 00:25:39.250 align:middle line:90% And then ugh. 00:25:39.250 --> 00:25:40.188 align:middle line:90% All right? 00:25:40.188 --> 00:25:42.730 align:middle line:84% I mean, you could write it out in terms of all the components 00:25:42.730 --> 00:25:43.980 align:middle line:90% and take derivatives that way. 00:25:43.980 --> 00:25:47.150 align:middle line:84% It's not so bad, but it starts to become really awkward really 00:25:47.150 --> 00:25:47.650 align:middle line:90% quickly. 00:25:47.650 --> 00:25:48.150 align:middle line:90% Right? 00:25:48.150 --> 00:25:49.270 align:middle line:90% If we just think of-- 00:25:49.270 --> 00:25:51.755 align:middle line:84% ALAN EDELMAN: --everybody to think of indices as ugh. 00:25:51.755 --> 00:25:52.630 align:middle line:90% STEVEN JOHNSON: Yeah. 00:25:52.630 --> 00:25:53.020 align:middle line:90% ALAN EDELMAN: Often. 00:25:53.020 --> 00:25:53.590 align:middle line:90% Not always. 00:25:53.590 --> 00:25:58.030 align:middle line:84% Sometimes they're useful, but usually they could be not used. 00:25:58.030 --> 00:25:59.710 align:middle line:90% STEVEN JOHNSON: Yeah. 00:25:59.710 --> 00:26:01.780 align:middle line:84% And another way of thinking about it 00:26:01.780 --> 00:26:05.210 align:middle line:84% is as soon as you start writing things in terms of indices, 00:26:05.210 --> 00:26:07.770 align:middle line:84% everything is very coordinate system dependent. 00:26:07.770 --> 00:26:09.890 align:middle line:84% So whereas if you think of it just generically, 00:26:09.890 --> 00:26:13.500 align:middle line:84% a rotation is an operator that rotates things by 90 degrees. 00:26:13.500 --> 00:26:14.000 align:middle line:90% Right? 00:26:14.000 --> 00:26:15.750 align:middle line:84% You don't have to have a coordinate system 00:26:15.750 --> 00:26:17.480 align:middle line:90% for that operator. 00:26:17.480 --> 00:26:21.330 align:middle line:84% Anyway, so what's the derivative? 00:26:21.330 --> 00:26:24.257 align:middle line:90% Well, let's just take-- 00:26:24.257 --> 00:26:25.840 align:middle line:84% pretty soon we'll have a product rule, 00:26:25.840 --> 00:26:29.320 align:middle line:84% but here you just do it the slow way again. 00:26:29.320 --> 00:26:32.800 align:middle line:84% It's not even that slow, you take f. 00:26:32.800 --> 00:26:45.550 align:middle line:84% You take your x, and you add a little change minus f of x. 00:26:45.550 --> 00:26:46.810 align:middle line:90% And what's that? 00:26:46.810 --> 00:26:49.570 align:middle line:84% We can just plug it in and drop nonlinear terms, 00:26:49.570 --> 00:26:52.180 align:middle line:84% except in this case there won't be any nonlinear terms. 00:26:52.180 --> 00:26:55.360 align:middle line:84% This will be f-- what's f of x plus delta x? 00:26:55.360 --> 00:27:01.010 align:middle line:90% It's ax plus a dx. 00:27:01.010 --> 00:27:01.510 align:middle line:90% Right? 00:27:01.510 --> 00:27:06.630 align:middle line:84% Because this is a linear operation, so I can-- 00:27:06.630 --> 00:27:08.165 align:middle line:90% I guess I skipped a step. 00:27:08.165 --> 00:27:13.710 align:middle line:84% a times x plus x is ax plus adx and minus f of x is ax, 00:27:13.710 --> 00:27:17.260 align:middle line:90% and then these two terms cancel. 00:27:17.260 --> 00:27:24.460 align:middle line:84% And so what we get is just adx, and this 00:27:24.460 --> 00:27:31.490 align:middle line:90% has to equal f prime of x dx. 00:27:31.490 --> 00:27:36.350 align:middle line:84% So you can see that if we just did a little compare of these, 00:27:36.350 --> 00:27:38.040 align:middle line:90% f prime is just a. 00:27:38.040 --> 00:27:39.290 align:middle line:90% It's just a linear operator a. 00:27:39.290 --> 00:27:47.590 align:middle line:84% So f prime, f prime of x is just a in this case, which 00:27:47.590 --> 00:27:50.430 align:middle line:90% that's our Jacobian. 00:27:50.430 --> 00:27:52.180 align:middle line:84% ALAN EDELMAN: And just to say the obvious, 00:27:52.180 --> 00:27:54.430 align:middle line:90% it doesn't depend on x at all. 00:27:54.430 --> 00:27:59.050 align:middle line:84% Everywhere in n-dimensional space it's the same a. 00:27:59.050 --> 00:27:59.950 align:middle line:90% STEVEN JOHNSON: Yeah. 00:27:59.950 --> 00:28:01.990 align:middle line:84% ALAN EDELMAN: No matter where you are in Rn. 00:28:01.990 --> 00:28:02.865 align:middle line:90% STEVEN JOHNSON: Yeah. 00:28:02.865 --> 00:28:08.530 align:middle line:84% Because in this case, it's like, again, if you're in 18.01 00:28:08.530 --> 00:28:13.420 align:middle line:84% and I have a linear function 3x plus f of x equals 3x, 00:28:13.420 --> 00:28:16.480 align:middle line:84% the slope is just 3 independent of x in that case. 00:28:16.480 --> 00:28:17.890 align:middle line:84% That's not true of all functions. 00:28:17.890 --> 00:28:18.390 align:middle line:90% Right? 00:28:18.390 --> 00:28:21.100 align:middle line:84% If the function is nonlinear, like 3x squared, 00:28:21.100 --> 00:28:23.590 align:middle line:90% the slope is 6x. 00:28:23.590 --> 00:28:24.430 align:middle line:90% It depends on x. 00:28:24.430 --> 00:28:27.790 align:middle line:84% But if it's a linear function, the slope doesn't depend on x. 00:28:27.790 --> 00:28:30.430 align:middle line:84% Same thing happens here, and in homework I'll 00:28:30.430 --> 00:28:33.070 align:middle line:84% ask you to do an even slightly more general version of that 00:28:33.070 --> 00:28:34.300 align:middle line:90% maybe. 00:28:34.300 --> 00:28:35.140 align:middle line:90% OK? 00:28:35.140 --> 00:28:38.440 align:middle line:84% But this is-- again, writing this out, 00:28:38.440 --> 00:28:41.200 align:middle line:84% I think it's still a lot easier than component by component. 00:28:41.200 --> 00:28:41.770 align:middle line:90% Right? 00:28:41.770 --> 00:28:44.260 align:middle line:84% Doing this versus doing that and taking 00:28:44.260 --> 00:28:46.990 align:middle line:84% the derivative partial fi, partial xj 00:28:46.990 --> 00:28:50.860 align:middle line:84% and seeing that's aij, and then realizing that, oh, that means 00:28:50.860 --> 00:28:53.530 align:middle line:90% that the whole thing is just a. 00:28:53.530 --> 00:28:55.792 align:middle line:84% But it's still a little bit cumbersome, 00:28:55.792 --> 00:28:58.000 align:middle line:84% especially when it gets to more complicated functions 00:28:58.000 --> 00:29:00.685 align:middle line:84% like this, so it's nice to have some rules. 00:29:00.685 --> 00:29:10.170 align:middle line:90% 00:29:10.170 --> 00:29:10.920 align:middle line:90% Right? 00:29:10.920 --> 00:29:19.482 align:middle line:84% So just like when you learned the derivatives in 18.01, 00:29:19.482 --> 00:29:21.660 align:middle line:84% you don't use the definition of a derivative 00:29:21.660 --> 00:29:24.180 align:middle line:84% at some limiting procedure for every time. 00:29:24.180 --> 00:29:27.630 align:middle line:84% You learn some product rules and power rules and sum rules 00:29:27.630 --> 00:29:31.770 align:middle line:84% and those kinds of things, so we can do the same kind of thing. 00:29:31.770 --> 00:29:33.580 align:middle line:90% Let's do an easy one. 00:29:33.580 --> 00:29:34.440 align:middle line:90% Let's do a sum rule. 00:29:34.440 --> 00:29:37.950 align:middle line:90% 00:29:37.950 --> 00:29:50.150 align:middle line:84% Suppose you have a function f of x is g of x plus h of x. 00:29:50.150 --> 00:29:52.850 align:middle line:90% And let me draw out my vectors. 00:29:52.850 --> 00:29:56.090 align:middle line:84% You can understand that everything here 00:29:56.090 --> 00:29:57.350 align:middle line:90% can be a vector. 00:29:57.350 --> 00:29:57.860 align:middle line:90% Right? 00:29:57.860 --> 00:30:00.800 align:middle line:84% The x's can be a vector and the f's and g's and h's can 00:30:00.800 --> 00:30:03.710 align:middle line:90% be vectors. 00:30:03.710 --> 00:30:07.520 align:middle line:84% In fact, in arbitrary vector spaces we'll see very soon. 00:30:07.520 --> 00:30:08.120 align:middle line:90% Right? 00:30:08.120 --> 00:30:15.930 align:middle line:84% Then if I want df, that's just going to be-- 00:30:15.930 --> 00:30:20.100 align:middle line:90% just write it as dg plus dh. 00:30:20.100 --> 00:30:24.600 align:middle line:90% Or equivalently, this is-- 00:30:24.600 --> 00:30:34.680 align:middle line:84% this is f prime of x dx equals g prime of x dx plus h prime 00:30:34.680 --> 00:30:36.240 align:middle line:90% of x dx. 00:30:36.240 --> 00:30:39.820 align:middle line:84% So f prime equals g prime plus h prime. 00:30:39.820 --> 00:30:40.320 align:middle line:90% Right? 00:30:40.320 --> 00:30:43.138 align:middle line:90% Just the obvious, some rule. 00:30:43.138 --> 00:30:43.680 align:middle line:90% I don't know. 00:30:43.680 --> 00:30:44.880 align:middle line:84% I don't know if you want me to derive this. 00:30:44.880 --> 00:30:47.040 align:middle line:84% I mean, again, it just comes from the definition. 00:30:47.040 --> 00:30:52.370 align:middle line:84% If I take f of x plus dx minus f of x, addition is linear. 00:30:52.370 --> 00:30:57.193 align:middle line:84% It's g of x plus dx minus gx plus h of x dx minus dx. 00:30:57.193 --> 00:30:59.610 align:middle line:84% ALAN EDELMAN: I don't think anybody in the room doubts it. 00:30:59.610 --> 00:31:01.280 align:middle line:84% You don't actually have to derive it. 00:31:01.280 --> 00:31:02.155 align:middle line:90% STEVEN JOHNSON: Yeah. 00:31:02.155 --> 00:31:04.910 align:middle line:90% So let me do the product rule. 00:31:04.910 --> 00:31:08.070 align:middle line:90% 00:31:08.070 --> 00:31:08.570 align:middle line:90% All right? 00:31:08.570 --> 00:31:12.390 align:middle line:84% So this one starts to be a little bit. 00:31:12.390 --> 00:31:15.620 align:middle line:84% I should do this on-- well, I'll write it out first. 00:31:15.620 --> 00:31:18.620 align:middle line:84% Suppose you have-- and I'm just going to write-- 00:31:18.620 --> 00:31:19.940 align:middle line:90% do I need to write the f? 00:31:19.940 --> 00:31:27.511 align:middle line:84% Sure, I'll write f of x equals g of x times h of x. 00:31:27.511 --> 00:31:30.870 align:middle line:84% So these have to be things you can multiply, obviously. 00:31:30.870 --> 00:31:31.988 align:middle line:90% Right? 00:31:31.988 --> 00:31:33.780 align:middle line:84% These could be two numbers or they could be 00:31:33.780 --> 00:31:35.970 align:middle line:90% two matrices of the right size. 00:31:35.970 --> 00:31:38.220 align:middle line:84% They can't be two column vectors, 00:31:38.220 --> 00:31:40.320 align:middle line:90% at least not with the ordinary. 00:31:40.320 --> 00:31:42.780 align:middle line:84% I'm just doing multiplication here. 00:31:42.780 --> 00:31:45.370 align:middle line:84% Maybe I should do element-wise multiplication in a minute, 00:31:45.370 --> 00:31:47.920 align:middle line:84% but let me just do multiplication. 00:31:47.920 --> 00:31:49.950 align:middle line:90% OK. 00:31:49.950 --> 00:32:04.485 align:middle line:84% So then df is going to be just dg times h plus g times dh. 00:32:04.485 --> 00:32:06.110 align:middle line:84% ALAN EDELMAN: And I guess the point is, 00:32:06.110 --> 00:32:10.160 align:middle line:84% whenever gh makes sense, whether it's element-wise products 00:32:10.160 --> 00:32:14.370 align:middle line:84% or matrix multiply, if it makes sense-- 00:32:14.370 --> 00:32:16.440 align:middle line:90% [INTERPOSING VOICES] 00:32:16.440 --> 00:32:17.315 align:middle line:90% STEVEN JOHNSON: Yeah. 00:32:17.315 --> 00:32:20.180 align:middle line:90% 00:32:20.180 --> 00:32:22.970 align:middle line:84% So if you think about what this means in terms of f prime, 00:32:22.970 --> 00:32:37.080 align:middle line:84% this f prime of x dx is going to equal g prime of x dx times h 00:32:37.080 --> 00:32:46.820 align:middle line:84% plus g of x plus h of x plus g of x times h prime of x dx. 00:32:46.820 --> 00:32:53.940 align:middle line:84% I cannot necessarily write f prime equals g prime h plus h 00:32:53.940 --> 00:32:59.750 align:middle line:90% prime g because I can't commute. 00:32:59.750 --> 00:33:03.895 align:middle line:90% 00:33:03.895 --> 00:33:05.145 align:middle line:90% I'm running out of space here. 00:33:05.145 --> 00:33:07.730 align:middle line:90% 00:33:07.730 --> 00:33:09.680 align:middle line:90% Why can't I do that? 00:33:09.680 --> 00:33:17.720 align:middle line:84% Because I can't necessarily move the x to the side here. 00:33:17.720 --> 00:33:18.230 align:middle line:90% All right? 00:33:18.230 --> 00:33:28.690 align:middle line:84% So g prime dx h is not equal to g prime h times d in general. 00:33:28.690 --> 00:33:31.530 align:middle line:84% If these are numbers, obviously this works. 00:33:31.530 --> 00:33:35.220 align:middle line:84% But now that these could be some other kind of object matrices 00:33:35.220 --> 00:33:38.280 align:middle line:84% or something like that, this is not going to work in general. 00:33:38.280 --> 00:33:42.480 align:middle line:84% OK, so let's-- and maybe that's-- let's derive this, 00:33:42.480 --> 00:33:43.710 align:middle line:90% just to-- 00:33:43.710 --> 00:33:47.010 align:middle line:84% we already kind of derived it, but let's do it in general. 00:33:47.010 --> 00:33:48.300 align:middle line:90% This is derivation. 00:33:48.300 --> 00:33:51.560 align:middle line:90% 00:33:51.560 --> 00:33:52.060 align:middle line:90% Right? 00:33:52.060 --> 00:33:55.120 align:middle line:90% So f is in gh. 00:33:55.120 --> 00:33:56.260 align:middle line:90% So what is df? 00:33:56.260 --> 00:34:00.220 align:middle line:84% Just use our definition. df is going 00:34:00.220 --> 00:34:08.150 align:middle line:84% to be f of x plus dx minus f of x, and f was g times h. 00:34:08.150 --> 00:34:14.300 align:middle line:84% This is g of x plus dx times h of x plus dx 00:34:14.300 --> 00:34:19.190 align:middle line:90% minus g of x h of x. 00:34:19.190 --> 00:34:22.219 align:middle line:90% 00:34:22.219 --> 00:34:25.040 align:middle line:90% And then this is g of x plus dx. 00:34:25.040 --> 00:34:28.040 align:middle line:84% We said by the definition of the derivative, that's 00:34:28.040 --> 00:34:31.639 align:middle line:84% g of x plus g prime of x, whatever 00:34:31.639 --> 00:34:37.250 align:middle line:84% that linear operator is, times dx times h of x plus dx-- 00:34:37.250 --> 00:34:38.239 align:middle line:90% h of x plus dx. 00:34:38.239 --> 00:34:40.949 align:middle line:90% Same thing. 00:34:40.949 --> 00:34:43.230 align:middle line:84% By the definition of the derivative, 00:34:43.230 --> 00:34:54.230 align:middle line:84% that's h of x plus h prime of x minus g of x h of x. 00:34:54.230 --> 00:34:56.420 align:middle line:84% And if we just write out all those terms-- 00:34:56.420 --> 00:34:58.520 align:middle line:84% and again, these might be matrices or something. 00:34:58.520 --> 00:35:00.120 align:middle line:90% I can't change the order. 00:35:00.120 --> 00:35:03.620 align:middle line:84% But I'm assuming that if this is a multiplication operator, 00:35:03.620 --> 00:35:05.490 align:middle line:84% it could be element-wise or it could 00:35:05.490 --> 00:35:06.740 align:middle line:90% be just matrix multiplication. 00:35:06.740 --> 00:35:10.990 align:middle line:84% Anything that satisfies the distributive law. 00:35:10.990 --> 00:35:14.695 align:middle line:90% I can write this as gx h of x. 00:35:14.695 --> 00:35:17.000 align:middle line:84% That's this term times this term. 00:35:17.000 --> 00:35:18.790 align:middle line:84% There's also this term times this term, 00:35:18.790 --> 00:35:21.937 align:middle line:84% this term times this term, and this term times this term. 00:35:21.937 --> 00:35:23.020 align:middle line:90% This term times this term. 00:35:23.020 --> 00:35:24.460 align:middle line:90% Sorry. 00:35:24.460 --> 00:35:34.050 align:middle line:84% So I get plus g prime x dx times-- 00:35:34.050 --> 00:35:36.750 align:middle line:84% which I could also just write as dg. 00:35:36.750 --> 00:35:39.600 align:middle line:90% This is the same thing as dg. 00:35:39.600 --> 00:35:42.180 align:middle line:90% This is the same thing as gh. 00:35:42.180 --> 00:35:43.905 align:middle line:90% Maybe I should have done that. 00:35:43.905 --> 00:35:45.600 align:middle line:90% It's a little shorter to write. 00:35:45.600 --> 00:35:53.700 align:middle line:84% This is dg times h plus I have a g times 00:35:53.700 --> 00:36:02.620 align:middle line:90% dh plus I have dg times dh. 00:36:02.620 --> 00:36:05.425 align:middle line:84% But that term we're going to ignore. 00:36:05.425 --> 00:36:06.535 align:middle line:90% This is higher order. 00:36:06.535 --> 00:36:13.130 align:middle line:90% 00:36:13.130 --> 00:36:14.105 align:middle line:90% All right? 00:36:14.105 --> 00:36:14.840 align:middle line:90% That's gone. 00:36:14.840 --> 00:36:17.720 align:middle line:84% If d is arbitrarily small, that's infinitesimal. 00:36:17.720 --> 00:36:25.950 align:middle line:84% And then we still have the minus g of x, h of x term here. 00:36:25.950 --> 00:36:26.740 align:middle line:90% And that's it. 00:36:26.740 --> 00:36:27.240 align:middle line:90% Right? 00:36:27.240 --> 00:36:30.860 align:middle line:90% 00:36:30.860 --> 00:36:32.763 align:middle line:90% So we're done. 00:36:32.763 --> 00:36:34.430 align:middle line:84% ALAN EDELMAN: And this is really nothing 00:36:34.430 --> 00:36:38.420 align:middle line:84% but a copy of what you did in the first month of calculus 00:36:38.420 --> 00:36:42.500 align:middle line:84% when you first learned it except that we are just extending it 00:36:42.500 --> 00:36:45.950 align:middle line:84% to vectors and matrices and making sure 00:36:45.950 --> 00:36:48.255 align:middle line:90% that we don't mess the order up. 00:36:48.255 --> 00:36:49.130 align:middle line:90% STEVEN JOHNSON: Yeah. 00:36:49.130 --> 00:36:50.810 align:middle line:84% ALAN EDELMAN: Otherwise it's exactly what you did 00:36:50.810 --> 00:36:52.060 align:middle line:90% in the early days of calculus. 00:36:52.060 --> 00:36:53.143 align:middle line:90% I'm sure you all remember. 00:36:53.143 --> 00:36:54.018 align:middle line:90% STEVEN JOHNSON: Yeah. 00:36:54.018 --> 00:36:55.650 align:middle line:84% And let me just do a couple examples. 00:36:55.650 --> 00:36:59.600 align:middle line:84% Suppose the one we just had. f of x equals ax. 00:36:59.600 --> 00:37:00.440 align:middle line:90% Right? 00:37:00.440 --> 00:37:09.560 align:middle line:84% Then df is equal to da times x plus a times dx. 00:37:09.560 --> 00:37:11.000 align:middle line:90% But this is zero. 00:37:11.000 --> 00:37:14.650 align:middle line:84% Since a is constant, a is independent. 00:37:14.650 --> 00:37:18.100 align:middle line:90% It is independent of x. 00:37:18.100 --> 00:37:19.850 align:middle line:84% So remember, you always have to keep track 00:37:19.850 --> 00:37:21.100 align:middle line:90% of what are the inputs, right? 00:37:21.100 --> 00:37:27.810 align:middle line:84% So here df-- f is a function whose input, in this case, 00:37:27.810 --> 00:37:28.950 align:middle line:90% is x. 00:37:28.950 --> 00:37:32.610 align:middle line:84% So df really means, f of x plus dx. 00:37:32.610 --> 00:37:37.230 align:middle line:90% 00:37:37.230 --> 00:37:40.530 align:middle line:90% f of x plus dx minus f of x. 00:37:40.530 --> 00:37:42.750 align:middle line:84% So since a doesn't change with x, that's zero, 00:37:42.750 --> 00:37:44.620 align:middle line:90% so then I'm done. 00:37:44.620 --> 00:37:45.120 align:middle line:90% Right? 00:37:45.120 --> 00:37:47.358 align:middle line:84% This is, again, the same term I got before. 00:37:47.358 --> 00:37:49.650 align:middle line:84% ALAN EDELMAN: Let me ask the class and not you, Steven. 00:37:49.650 --> 00:37:51.315 align:middle line:90% What zero is this? 00:37:51.315 --> 00:37:56.180 align:middle line:90% 00:37:56.180 --> 00:37:56.680 align:middle line:90% Right? 00:37:56.680 --> 00:38:00.040 align:middle line:84% I think that might be useful to think about. 00:38:00.040 --> 00:38:01.780 align:middle line:84% Can someone tell me exactly which zero-- 00:38:01.780 --> 00:38:03.280 align:middle line:84% STEVEN JOHNSON: Shape of zero, yeah. 00:38:03.280 --> 00:38:04.447 align:middle line:90% ALAN EDELMAN: Shape of zero. 00:38:04.447 --> 00:38:05.603 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:38:05.603 --> 00:38:06.436 align:middle line:90% ALAN EDELMAN: Sorry? 00:38:06.436 --> 00:38:08.140 align:middle line:84% AUDIENCE: M dimensional column vector. 00:38:08.140 --> 00:38:09.760 align:middle line:84% ALAN EDELMAN: Is it m dimensional column vector? 00:38:09.760 --> 00:38:10.720 align:middle line:90% No, I don't think so. 00:38:10.720 --> 00:38:12.130 align:middle line:90% AUDIENCE: It is m. 00:38:12.130 --> 00:38:13.390 align:middle line:90% ALAN EDELMAN: Yeah. 00:38:13.390 --> 00:38:14.640 align:middle line:90% So what is da? 00:38:14.640 --> 00:38:16.424 align:middle line:90% AUDIENCE: A square matrix. 00:38:16.424 --> 00:38:17.758 align:middle line:90% An m cross n matrix of zero. 00:38:17.758 --> 00:38:18.550 align:middle line:90% ALAN EDELMAN: Yeah. 00:38:18.550 --> 00:38:20.920 align:middle line:90% It's an m by n matrix of zeroes. 00:38:20.920 --> 00:38:23.890 align:middle line:84% So that infinite red zero that Steven drew 00:38:23.890 --> 00:38:28.610 align:middle line:84% is really the m by n zero matrix. 00:38:28.610 --> 00:38:29.110 align:middle line:90% Right? 00:38:29.110 --> 00:38:32.275 align:middle line:84% In fact, he could have written da equals 0dx. 00:38:32.275 --> 00:38:36.663 align:middle line:90% 00:38:36.663 --> 00:38:37.830 align:middle line:90% Might have pointed that out. 00:38:37.830 --> 00:38:40.190 align:middle line:84% But it's so easy to quickly look at that zero 00:38:40.190 --> 00:38:44.190 align:middle line:84% and lose sight of the fact that that's a m by n zero matrix. 00:38:44.190 --> 00:38:50.790 align:middle line:84% Which if you multiply by any dx that's n little perturbations, 00:38:50.790 --> 00:38:53.400 align:middle line:90% you get m0's coming back out. 00:38:53.400 --> 00:38:57.410 align:middle line:90% 00:38:57.410 --> 00:38:59.080 align:middle line:90% STEVEN JOHNSON: Yeah. 00:38:59.080 --> 00:39:01.910 align:middle line:84% So let me do another one, one that I did before. 00:39:01.910 --> 00:39:06.080 align:middle line:84% So did x transpose ax, I believe. 00:39:06.080 --> 00:39:06.640 align:middle line:90% Right? 00:39:06.640 --> 00:39:14.550 align:middle line:84% So now if I do df I get dx transpose. 00:39:14.550 --> 00:39:16.840 align:middle line:84% This confuses people, by the way, a little bit. 00:39:16.840 --> 00:39:21.780 align:middle line:90% 00:39:21.780 --> 00:39:27.880 align:middle line:84% Well, actually let me do-- why does dx 00:39:27.880 --> 00:39:33.010 align:middle line:90% transpose equal dx transpose? 00:39:33.010 --> 00:39:36.712 align:middle line:90% Because it's just dx-- 00:39:36.712 --> 00:39:47.080 align:middle line:84% dx transpose is really x plus dx transpose minus x transpose. 00:39:47.080 --> 00:39:52.090 align:middle line:84% But that's equal to x transpose plus dx transpose minus x 00:39:52.090 --> 00:39:53.060 align:middle line:90% transpose. 00:39:53.060 --> 00:39:54.760 align:middle line:90% So that's dx transpose. 00:39:54.760 --> 00:39:55.490 align:middle line:90% Right? 00:39:55.490 --> 00:39:56.810 align:middle line:84% ALAN EDELMAN: I wouldn't have thought of it that way, 00:39:56.810 --> 00:39:57.852 align:middle line:90% but of course it's right. 00:39:57.852 --> 00:39:58.840 align:middle line:90% STEVEN JOHNSON: Yeah. 00:39:58.840 --> 00:40:00.940 align:middle line:84% ALAN EDELMAN: To me it's just you just 00:40:00.940 --> 00:40:03.610 align:middle line:84% take something as a column and you make it a row 00:40:03.610 --> 00:40:04.300 align:middle line:90% and you perturb. 00:40:04.300 --> 00:40:06.970 align:middle line:90% Either way, it's the same thing. 00:40:06.970 --> 00:40:09.155 align:middle line:84% But formally you're quite right, Steven. 00:40:09.155 --> 00:40:10.030 align:middle line:90% STEVEN JOHNSON: Yeah. 00:40:10.030 --> 00:40:11.948 align:middle line:90% [INTERPOSING VOICES] 00:40:11.948 --> 00:40:13.990 align:middle line:84% And so in the homework I want to get you to think 00:40:13.990 --> 00:40:15.395 align:middle line:90% of this in terms of linearity. 00:40:15.395 --> 00:40:15.895 align:middle line:90% Right? 00:40:15.895 --> 00:40:18.463 align:middle line:84% The transpose is-- well, maybe I'm 00:40:18.463 --> 00:40:19.630 align:middle line:90% kind of giving away, though. 00:40:19.630 --> 00:40:24.850 align:middle line:84% But transpose is an example of a linear operation on a vector. 00:40:24.850 --> 00:40:27.190 align:middle line:84% And it's one that's kind of annoying to write down 00:40:27.190 --> 00:40:31.450 align:middle line:84% as a matrix because it's not really 00:40:31.450 --> 00:40:35.650 align:middle line:84% multiplying x by a matrix unless you reinterpret 00:40:35.650 --> 00:40:39.170 align:middle line:84% the output as a vector and the input as a vector in some way. 00:40:39.170 --> 00:40:40.570 align:middle line:90% So anyway. 00:40:40.570 --> 00:40:42.320 align:middle line:84% So yeah, so this is-- let me just go back. 00:40:42.320 --> 00:40:43.565 align:middle line:90% The product rule-- 00:40:43.565 --> 00:40:45.940 align:middle line:84% I guess I only showed you the product rule for two terms, 00:40:45.940 --> 00:40:48.210 align:middle line:84% but it's the same thing for three terms. 00:40:48.210 --> 00:40:49.377 align:middle line:90% So there's df. 00:40:49.377 --> 00:40:51.710 align:middle line:84% ALAN EDELMAN: You know, you could talk about the product 00:40:51.710 --> 00:40:53.840 align:middle line:84% rule for three terms, or you could 00:40:53.840 --> 00:40:56.780 align:middle line:84% think of this as two terms and use the result that's 00:40:56.780 --> 00:40:58.200 align:middle line:90% on the top of your screen. 00:40:58.200 --> 00:40:59.400 align:middle line:90% STEVEN JOHNSON: Yeah. 00:40:59.400 --> 00:40:59.900 align:middle line:90% Exactly. 00:40:59.900 --> 00:41:00.887 align:middle line:90% I could just use-- 00:41:00.887 --> 00:41:01.970 align:middle line:90% ALAN EDELMAN: --now Right. 00:41:01.970 --> 00:41:02.930 align:middle line:90% STEVEN JOHNSON: Yeah. 00:41:02.930 --> 00:41:05.510 align:middle line:84% But let me just do it do it for three terms. 00:41:05.510 --> 00:41:09.535 align:middle line:84% But yeah, I could use two terms and then you write it as-- 00:41:09.535 --> 00:41:10.800 align:middle line:90% let's do it with two terms. 00:41:10.800 --> 00:41:14.930 align:middle line:84% So there's the x transpose times dax. 00:41:14.930 --> 00:41:16.580 align:middle line:90% Right? 00:41:16.580 --> 00:41:20.383 align:middle line:84% And that's equal to dx transpose ax. 00:41:20.383 --> 00:41:21.800 align:middle line:84% ALAN EDELMAN: So in effect, you're 00:41:21.800 --> 00:41:24.890 align:middle line:84% putting a parentheses around the-- it's x transpose times 00:41:24.890 --> 00:41:28.090 align:middle line:84% left parenthesis ax right parenthesis [INAUDIBLE].. 00:41:28.090 --> 00:41:29.695 align:middle line:90% Right? 00:41:29.695 --> 00:41:32.320 align:middle line:90% You see what I'm saying? 00:41:32.320 --> 00:41:33.670 align:middle line:90% STEVEN JOHNSON: Yes. 00:41:33.670 --> 00:41:36.910 align:middle line:84% ALAN EDELMAN: The parenthesis itself is thought of as extra-- 00:41:36.910 --> 00:41:38.140 align:middle line:90% well, even-- yeah, yeah. 00:41:38.140 --> 00:41:38.950 align:middle line:90% That's what I'm thinking of. 00:41:38.950 --> 00:41:40.480 align:middle line:84% STEVEN JOHNSON: You're thinking of that, yes. 00:41:40.480 --> 00:41:41.650 align:middle line:84% And of course, I can put parentheses 00:41:41.650 --> 00:41:43.210 align:middle line:84% anywhere I want in these kinds of things. 00:41:43.210 --> 00:41:43.550 align:middle line:84% ALAN EDELMAN: Of course you could, 00:41:43.550 --> 00:41:45.550 align:middle line:84% but that's what you're focusing on this way, right? 00:41:45.550 --> 00:41:46.425 align:middle line:90% STEVEN JOHNSON: Yeah. 00:41:46.425 --> 00:41:49.000 align:middle line:84% And dax is product rule again, but I just 00:41:49.000 --> 00:41:51.620 align:middle line:90% did that, so it's adx. 00:41:51.620 --> 00:41:52.120 align:middle line:90% Right? 00:41:52.120 --> 00:41:57.560 align:middle line:84% And then we used the same trick before to-- 00:41:57.560 --> 00:42:01.680 align:middle line:84% this term here is a scalar, so I can just transpose it 00:42:01.680 --> 00:42:03.450 align:middle line:90% and get the dx on the right. 00:42:03.450 --> 00:42:06.740 align:middle line:84% And so that's equal to x transpose 00:42:06.740 --> 00:42:10.280 align:middle line:90% a plus a transpose dx. 00:42:10.280 --> 00:42:17.240 align:middle line:84% And then this was our f prime of x, 00:42:17.240 --> 00:42:20.990 align:middle line:84% which is the same thing as the gradient of f transposed. 00:42:20.990 --> 00:42:21.790 align:middle line:90% Right? 00:42:21.790 --> 00:42:24.550 align:middle line:84% Which therefore means that the gradient of f 00:42:24.550 --> 00:42:27.230 align:middle line:84% is equal to the transpose of that, 00:42:27.230 --> 00:42:32.005 align:middle line:90% which is a plus a transpose x. 00:42:32.005 --> 00:42:33.820 align:middle line:90% So that's the product rule. 00:42:33.820 --> 00:42:35.020 align:middle line:90% Pretty straightforward. 00:42:35.020 --> 00:42:35.785 align:middle line:90% Right? 00:42:35.785 --> 00:42:37.972 align:middle line:84% We could do more complicated things. 00:42:37.972 --> 00:42:39.180 align:middle line:90% So we could do, for example-- 00:42:39.180 --> 00:42:42.410 align:middle line:90% 00:42:42.410 --> 00:42:44.755 align:middle line:84% actually, let me hold off on that for a second. 00:42:44.755 --> 00:42:45.755 align:middle line:90% Let's do the chain rule. 00:42:45.755 --> 00:42:53.190 align:middle line:90% 00:42:53.190 --> 00:43:05.500 align:middle line:84% So suppose we have f of x equals g of h of x. 00:43:05.500 --> 00:43:10.270 align:middle line:84% So gh of x is a function, takes a vector in, gives you-- 00:43:10.270 --> 00:43:13.570 align:middle line:84% some x in, gives you some vector out, some vector space. 00:43:13.570 --> 00:43:14.410 align:middle line:90% Maybe numbers. 00:43:14.410 --> 00:43:15.820 align:middle line:90% Maybe column vectors. 00:43:15.820 --> 00:43:17.260 align:middle line:90% Maybe matrices pretty soon. 00:43:17.260 --> 00:43:19.870 align:middle line:84% And g takes whatever the output space of h 00:43:19.870 --> 00:43:30.780 align:middle line:84% is and gives you some other vector as output. 00:43:30.780 --> 00:43:31.280 align:middle line:90% All right? 00:43:31.280 --> 00:43:34.670 align:middle line:84% So for example, x could be a column vector, and then h of x 00:43:34.670 --> 00:43:36.590 align:middle line:84% takes a column vector and produces a matrix. 00:43:36.590 --> 00:43:38.780 align:middle line:84% And then g of x takes a matrix and gives you 00:43:38.780 --> 00:43:41.370 align:middle line:84% a scalar, for example, like a determinant. 00:43:41.370 --> 00:43:41.870 align:middle line:90% OK? 00:43:41.870 --> 00:43:44.160 align:middle line:84% So they could be all different shapes. 00:43:44.160 --> 00:43:44.660 align:middle line:90% Right? 00:43:44.660 --> 00:43:47.270 align:middle line:84% And one of the things we talk about in linear algebra 00:43:47.270 --> 00:43:49.805 align:middle line:84% is it's always good to keep track of the shapes of things. 00:43:49.805 --> 00:43:52.790 align:middle line:90% 00:43:52.790 --> 00:43:55.550 align:middle line:84% Everything's not a number anymore 00:43:55.550 --> 00:43:58.620 align:middle line:84% and you need to keep your shapes lined up. 00:43:58.620 --> 00:43:59.120 align:middle line:90% OK? 00:43:59.120 --> 00:44:02.150 align:middle line:84% So then, what's the chain rule going to be? 00:44:02.150 --> 00:44:05.240 align:middle line:90% It's going to be-- 00:44:05.240 --> 00:44:09.240 align:middle line:84% I'll just write it down first and we'll just see why. 00:44:09.240 --> 00:44:10.305 align:middle line:90% f prime of x. 00:44:10.305 --> 00:44:13.090 align:middle line:90% 00:44:13.090 --> 00:44:16.685 align:middle line:84% x is a linear operator acting on a dx. 00:44:16.685 --> 00:44:20.250 align:middle line:90% 00:44:20.250 --> 00:44:21.540 align:middle line:90% Right? 00:44:21.540 --> 00:44:24.332 align:middle line:84% And that's our-- and that's our df. 00:44:24.332 --> 00:44:25.665 align:middle line:90% We should maybe write that down. 00:44:25.665 --> 00:44:29.650 align:middle line:90% 00:44:29.650 --> 00:44:33.210 align:middle line:90% df is that. 00:44:33.210 --> 00:44:33.740 align:middle line:90% OK. 00:44:33.740 --> 00:44:36.380 align:middle line:90% And what do you do? 00:44:36.380 --> 00:44:41.540 align:middle line:84% It's going to be just like the chain rule for numbers. 00:44:41.540 --> 00:44:47.260 align:middle line:84% It's going to be g prime of h of x. 00:44:47.260 --> 00:44:50.710 align:middle line:84% That's a linear operator, and that's 00:44:50.710 --> 00:44:56.510 align:middle line:84% going to act on h prime of x, and that's 00:44:56.510 --> 00:45:01.250 align:middle line:84% a linear operator that's going to act on dx. 00:45:01.250 --> 00:45:02.510 align:middle line:90% I guess I haven't-- 00:45:02.510 --> 00:45:05.090 align:middle line:84% we're not putting vector signs in there. 00:45:05.090 --> 00:45:11.730 align:middle line:84% Very often, I'll drop the f prime 00:45:11.730 --> 00:45:19.350 align:middle line:84% of x equals g prime of h of x, h prime of x. 00:45:19.350 --> 00:45:21.000 align:middle line:84% But when I write a product like this, 00:45:21.000 --> 00:45:26.150 align:middle line:84% this really means the composition. 00:45:26.150 --> 00:45:29.210 align:middle line:84% It means it's a product of two linear operators, 00:45:29.210 --> 00:45:32.540 align:middle line:90% of g prime and h prime. 00:45:32.540 --> 00:45:34.760 align:middle line:84% It means you do h prime first and then g prime, 00:45:34.760 --> 00:45:40.260 align:middle line:84% and this is very much not equal to h prime times g prime. 00:45:40.260 --> 00:45:42.300 align:middle line:84% Think of these, for example, as matrices, right? 00:45:42.300 --> 00:45:43.425 align:middle line:90% You can't change the order. 00:45:43.425 --> 00:45:45.797 align:middle line:90% 00:45:45.797 --> 00:45:47.380 align:middle line:84% ALAN EDELMAN: Can I throw in something 00:45:47.380 --> 00:45:49.047 align:middle line:84% that's a foreshadowing of something that 00:45:49.047 --> 00:45:50.440 align:middle line:90% might come later? 00:45:50.440 --> 00:45:52.270 align:middle line:90% Just to set a stage. 00:45:52.270 --> 00:45:54.940 align:middle line:84% I want to ask the class, from ordinary calculus, 00:45:54.940 --> 00:45:57.820 align:middle line:84% just an application of the chain rule. 00:45:57.820 --> 00:46:01.390 align:middle line:84% Let's say you were taking just the derivative-- you all 00:46:01.390 --> 00:46:01.890 align:middle line:90% can do this. 00:46:01.890 --> 00:46:04.280 align:middle line:90% Derivative of sine of x squared. 00:46:04.280 --> 00:46:08.590 align:middle line:84% I want to ask, how many of you will first do the cosine? 00:46:08.590 --> 00:46:11.620 align:middle line:84% So it'll be cosine of x squared, and then in your mind 00:46:11.620 --> 00:46:13.240 align:middle line:90% will do the 2x? 00:46:13.240 --> 00:46:15.310 align:middle line:84% And as opposed to, how many of you 00:46:15.310 --> 00:46:20.200 align:middle line:84% will do the 2x on the inside first and then do the cosine? 00:46:20.200 --> 00:46:23.905 align:middle line:84% So how many of you are cosine first people? 00:46:23.905 --> 00:46:25.030 align:middle line:90% I think that was everybody. 00:46:25.030 --> 00:46:29.530 align:middle line:84% Does anybody do the x squared first going into out? 00:46:29.530 --> 00:46:31.660 align:middle line:84% There are two people in the back who do it. 00:46:31.660 --> 00:46:35.270 align:middle line:84% But you see, I mean, of course, you could do it either way. 00:46:35.270 --> 00:46:39.290 align:middle line:84% And just to say some words now that will come up later. 00:46:39.290 --> 00:46:44.290 align:middle line:84% But the first way is what eventually becomes forward 00:46:44.290 --> 00:46:47.020 align:middle line:84% mode automatic differentiation, if you've ever heard that term. 00:46:47.020 --> 00:46:50.710 align:middle line:84% And the second way is reverse mode automatic differentiation. 00:46:50.710 --> 00:46:53.330 align:middle line:84% Of course, for scalars it's all pretty simple, 00:46:53.330 --> 00:46:55.060 align:middle line:84% but when we get into this matrix calculus 00:46:55.060 --> 00:47:00.070 align:middle line:84% stuff it gets to be a whole big thing of machine learning 00:47:00.070 --> 00:47:01.360 align:middle line:90% and lots of other things-- 00:47:01.360 --> 00:47:04.780 align:middle line:84% technologies that underlie so much these days. 00:47:04.780 --> 00:47:06.040 align:middle line:90% Right? 00:47:06.040 --> 00:47:08.290 align:middle line:84% But it really-- in its simplest, simplest form, 00:47:08.290 --> 00:47:12.610 align:middle line:84% it all boils down to, as Steven writes, you have the choice, 00:47:12.610 --> 00:47:17.950 align:middle line:84% in effect, of doing the h prime first or the g prime h first. 00:47:17.950 --> 00:47:18.970 align:middle line:90% Right? 00:47:18.970 --> 00:47:21.400 align:middle line:84% If you're a computer, you'll have to do it sequentially. 00:47:21.400 --> 00:47:21.900 align:middle line:90% Right? 00:47:21.900 --> 00:47:23.630 align:middle line:84% So one of the things will come first. 00:47:23.630 --> 00:47:25.150 align:middle line:84% I guess it doesn't have to be sequentially. 00:47:25.150 --> 00:47:26.800 align:middle line:84% STEVEN JOHNSON: I'll show an example of that in a second. 00:47:26.800 --> 00:47:29.362 align:middle line:84% So I think it becomes more clear once you have three things. 00:47:29.362 --> 00:47:30.320 align:middle line:90% ALAN EDELMAN: Yeah, OK. 00:47:30.320 --> 00:47:31.838 align:middle line:90% All right. 00:47:31.838 --> 00:47:33.380 align:middle line:84% I just wanted to set that thought up. 00:47:33.380 --> 00:47:33.890 align:middle line:90% STEVEN JOHNSON: Yeah. 00:47:33.890 --> 00:47:34.310 align:middle line:90% Absolutely. 00:47:34.310 --> 00:47:36.102 align:middle line:84% It's really going to matter, left to right. 00:47:36.102 --> 00:47:37.700 align:middle line:84% When these are not scalars anymore, 00:47:37.700 --> 00:47:39.410 align:middle line:84% the left to right versus right to left 00:47:39.410 --> 00:47:41.245 align:middle line:84% is going to start to matter a lot. 00:47:41.245 --> 00:47:42.620 align:middle line:84% ALAN EDELMAN: And also the order. 00:47:42.620 --> 00:47:43.010 align:middle line:90% STEVEN JOHNSON: Yeah. 00:47:43.010 --> 00:47:43.730 align:middle line:90% ALAN EDELMAN: Right? 00:47:43.730 --> 00:47:44.810 align:middle line:90% STEVEN JOHNSON: So, yeah. 00:47:44.810 --> 00:47:46.650 align:middle line:84% I don't know if people want me to derive this again. 00:47:46.650 --> 00:47:48.350 align:middle line:84% We can just plug in the definitions 00:47:48.350 --> 00:47:56.750 align:middle line:84% and do f of x, take this plus dx minus g of h of x plus dx 00:47:56.750 --> 00:48:00.000 align:middle line:90% minus g of x is g of h of x. 00:48:00.000 --> 00:48:00.500 align:middle line:90% Sorry. 00:48:00.500 --> 00:48:01.160 align:middle line:90% And then-- 00:48:01.160 --> 00:48:02.390 align:middle line:90% [INTERPOSING VOICES] 00:48:02.390 --> 00:48:04.240 align:middle line:84% --everything and just drop the dx's. 00:48:04.240 --> 00:48:05.120 align:middle line:84% ALAN EDELMAN: Who wants the derivation 00:48:05.120 --> 00:48:07.160 align:middle line:84% and who wants to go straight to the application? 00:48:07.160 --> 00:48:08.090 align:middle line:90% Derivation? 00:48:08.090 --> 00:48:09.215 align:middle line:90% STEVEN JOHNSON: Derivation. 00:48:09.215 --> 00:48:11.696 align:middle line:84% ALAN EDELMAN: How many want to see the application? 00:48:11.696 --> 00:48:14.220 align:middle line:90% Like, five or six people. 00:48:14.220 --> 00:48:15.510 align:middle line:90% STEVEN JOHNSON: OK. 00:48:15.510 --> 00:48:18.240 align:middle line:90% OK. 00:48:18.240 --> 00:48:24.600 align:middle line:84% So let me do an example of this just to-- suppose, 00:48:24.600 --> 00:48:27.710 align:middle line:90% to start with, that-- 00:48:27.710 --> 00:48:34.090 align:middle line:90% so the x, or inputs, live in Rn. 00:48:34.090 --> 00:48:37.600 align:middle line:84% So we have n component vectors as inputs. 00:48:37.600 --> 00:48:40.600 align:middle line:90% And then suppose that h of x-- 00:48:40.600 --> 00:48:43.510 align:middle line:90% 00:48:43.510 --> 00:48:50.500 align:middle line:90% h of x are our first function. 00:48:50.500 --> 00:48:56.880 align:middle line:84% Our inner function lives in our p. 00:48:56.880 --> 00:49:00.690 align:middle line:90% p sends vectors to vectors. 00:49:00.690 --> 00:49:01.190 align:middle line:90% Oops. 00:49:01.190 --> 00:49:02.270 align:middle line:90% What happened? 00:49:02.270 --> 00:49:03.480 align:middle line:90% OK. 00:49:03.480 --> 00:49:16.750 align:middle line:84% And suppose our outer thing, our g of h, lives in our m. 00:49:16.750 --> 00:49:22.750 align:middle line:90% So if we do f of x. 00:49:22.750 --> 00:49:26.430 align:middle line:90% 00:49:26.430 --> 00:49:28.350 align:middle line:90% All right? 00:49:28.350 --> 00:49:30.900 align:middle line:90% Which is not going to be-- 00:49:30.900 --> 00:49:33.975 align:middle line:84% because remember, it's g of h of x. 00:49:33.975 --> 00:49:39.520 align:middle line:90% 00:49:39.520 --> 00:49:40.360 align:middle line:90% All right? 00:49:40.360 --> 00:49:42.020 align:middle line:90% What it does is it tend-- 00:49:42.020 --> 00:49:49.480 align:middle line:84% it sends things in Rn through to Rp via h. 00:49:49.480 --> 00:49:54.140 align:middle line:90% And then sends them via g to Rn. 00:49:54.140 --> 00:49:54.640 align:middle line:90% OK? 00:49:54.640 --> 00:49:56.280 align:middle line:90% So it sends vectors to vectors. 00:49:56.280 --> 00:49:59.140 align:middle line:84% Vectors may be all the same size, maybe not. 00:49:59.140 --> 00:50:00.340 align:middle line:90% OK. 00:50:00.340 --> 00:50:02.800 align:middle line:90% And then what is our chain rule? 00:50:02.800 --> 00:50:08.550 align:middle line:90% 00:50:08.550 --> 00:50:09.050 align:middle line:90% Right? 00:50:09.050 --> 00:50:11.770 align:middle line:90% Is that f prime-- 00:50:11.770 --> 00:50:13.660 align:middle line:90% let's e blue. 00:50:13.660 --> 00:50:18.510 align:middle line:90% f prime of x. 00:50:18.510 --> 00:50:22.200 align:middle line:84% And so now this is-- we know this is a Jacobian matrix. 00:50:22.200 --> 00:50:25.000 align:middle line:90% This takes n inputs, m outputs. 00:50:25.000 --> 00:50:28.950 align:middle line:90% This is an m by n Jacobian. 00:50:28.950 --> 00:50:34.720 align:middle line:90% 00:50:34.720 --> 00:50:38.820 align:middle line:84% This is equal to g prime times h prime. 00:50:38.820 --> 00:50:47.505 align:middle line:84% So this is going to be g prime of h of x. 00:50:47.505 --> 00:50:52.850 align:middle line:90% 00:50:52.850 --> 00:50:57.650 align:middle line:84% And g takes key inputs and m outputs. 00:50:57.650 --> 00:50:59.576 align:middle line:90% So this one is a-- 00:50:59.576 --> 00:51:02.992 align:middle line:90% 00:51:02.992 --> 00:51:08.508 align:middle line:90% is an m by p Jacobian of g. 00:51:08.508 --> 00:51:12.010 align:middle line:90% 00:51:12.010 --> 00:51:14.557 align:middle line:84% I think I might need a little bit more space. 00:51:14.557 --> 00:51:15.640 align:middle line:90% I love that I can do that. 00:51:15.640 --> 00:51:17.537 align:middle line:84% On the blackboard I always write and write 00:51:17.537 --> 00:51:19.120 align:middle line:84% and get to the edge of the blackboard, 00:51:19.120 --> 00:51:23.620 align:middle line:84% and then my writing gets tinier and tinier. 00:51:23.620 --> 00:51:25.540 align:middle line:90% And then also not equals. 00:51:25.540 --> 00:51:27.142 align:middle line:90% Times h prime. 00:51:27.142 --> 00:51:29.350 align:middle line:84% ALAN EDELMAN: I've seen people move over to the walls 00:51:29.350 --> 00:51:32.200 align:middle line:84% when the blackboard space runs out. 00:51:32.200 --> 00:51:35.810 align:middle line:84% STEVEN JOHNSON: Of x, and h prime of 00:51:35.810 --> 00:51:42.850 align:middle line:84% x takes an input- h has n inputs and p outputs. 00:51:42.850 --> 00:51:47.680 align:middle line:90% 00:51:47.680 --> 00:51:48.180 align:middle line:90% Right? 00:51:48.180 --> 00:51:49.770 align:middle line:90% So it's a p by n matrix. 00:51:49.770 --> 00:51:51.950 align:middle line:84% When you linearize it, it's a p by n. 00:51:51.950 --> 00:51:55.480 align:middle line:84% These h and g are not linear functions in general, 00:51:55.480 --> 00:51:58.545 align:middle line:84% but the derivatives are linearization. 00:51:58.545 --> 00:51:59.670 align:middle line:90% There are linear operators. 00:51:59.670 --> 00:52:02.050 align:middle line:84% They're just matrices in this case. 00:52:02.050 --> 00:52:09.480 align:middle line:84% And you can see that these match up. 00:52:09.480 --> 00:52:12.120 align:middle line:90% Right? 00:52:12.120 --> 00:52:21.350 align:middle line:84% Not only is this not equal to h prime times g prime, 00:52:21.350 --> 00:52:23.670 align:middle line:84% but that wouldn't even make sense. 00:52:23.670 --> 00:52:24.170 align:middle line:90% Right? 00:52:24.170 --> 00:52:28.720 align:middle line:90% 00:52:28.720 --> 00:52:29.240 align:middle line:90% Right? 00:52:29.240 --> 00:52:38.930 align:middle line:84% If you multiply a p by n times an m by p, 00:52:38.930 --> 00:52:39.990 align:middle line:90% it doesn't even work. 00:52:39.990 --> 00:52:40.490 align:middle line:90% Right? 00:52:40.490 --> 00:52:44.310 align:middle line:84% So you really can't change the order of these things. 00:52:44.310 --> 00:52:48.515 align:middle line:84% So this is nonsense, I guess, if n is not equal to m. 00:52:48.515 --> 00:52:51.720 align:middle line:90% 00:52:51.720 --> 00:52:56.700 align:middle line:84% And even if n equals m, then that product means something 00:52:56.700 --> 00:52:58.560 align:middle line:84% but it doesn't actually-- it's not 00:52:58.560 --> 00:53:02.400 align:middle line:84% equal to the derivative of f anymore. 00:53:02.400 --> 00:53:04.800 align:middle line:84% That would be the derivative of g of h 00:53:04.800 --> 00:53:06.960 align:middle line:90% of x if you swap the order. 00:53:06.960 --> 00:53:10.740 align:middle line:90% 00:53:10.740 --> 00:53:14.760 align:middle line:84% So for just functions that are vectors in, vectors out, 00:53:14.760 --> 00:53:18.390 align:middle line:84% the chain rule is just the ordinary product, 00:53:18.390 --> 00:53:21.690 align:middle line:84% the matrix product of Jacobian matrices of each 00:53:21.690 --> 00:53:23.580 align:middle line:90% of the terms in the order. 00:53:23.580 --> 00:53:26.580 align:middle line:84% So you always have to go keep them in the same order, 00:53:26.580 --> 00:53:29.970 align:middle line:84% otherwise it's not going to make sense. 00:53:29.970 --> 00:53:31.560 align:middle line:90% The outputs are here. 00:53:31.560 --> 00:53:33.810 align:middle line:84% Then the outputs of h are there and then 00:53:33.810 --> 00:53:35.160 align:middle line:90% the inputs are finally there. 00:53:35.160 --> 00:53:42.880 align:middle line:90% 00:53:42.880 --> 00:53:45.280 align:middle line:84% And so not only does the order matter, 00:53:45.280 --> 00:53:54.330 align:middle line:84% but the associativity matters practically. 00:53:54.330 --> 00:53:58.890 align:middle line:90% 00:53:58.890 --> 00:53:59.390 align:middle line:90% Right? 00:53:59.390 --> 00:54:01.647 align:middle line:84% And associativity is going to matter 00:54:01.647 --> 00:54:02.730 align:middle line:90% where you put parentheses. 00:54:02.730 --> 00:54:04.888 align:middle line:84% So here I'm just multiplying two things. 00:54:04.888 --> 00:54:05.930 align:middle line:90% There's only one product. 00:54:05.930 --> 00:54:06.930 align:middle line:90% There's only one choice. 00:54:06.930 --> 00:54:08.900 align:middle line:90% But suppose we have-- 00:54:08.900 --> 00:54:11.210 align:middle line:90% suppose we have three functions. 00:54:11.210 --> 00:54:15.800 align:middle line:90% 00:54:15.800 --> 00:54:16.300 align:middle line:90% Right? 00:54:16.300 --> 00:54:24.320 align:middle line:84% So now suppose we have f of x equals-- 00:54:24.320 --> 00:54:26.900 align:middle line:90% this is still Triton blue. 00:54:26.900 --> 00:54:34.110 align:middle line:90% F of x equals-- 00:54:34.110 --> 00:54:34.610 align:middle line:90% let's see. 00:54:34.610 --> 00:54:36.300 align:middle line:90% I'm running out of letters. gh. 00:54:36.300 --> 00:54:38.125 align:middle line:90% What letter should I use? 00:54:38.125 --> 00:54:41.530 align:middle line:90% 00:54:41.530 --> 00:54:42.130 align:middle line:90% a. 00:54:42.130 --> 00:54:43.010 align:middle line:90% Well, no. 00:54:43.010 --> 00:54:44.530 align:middle line:90% Let's use abc. 00:54:44.530 --> 00:54:47.440 align:middle line:90% Let's use abc. 00:54:47.440 --> 00:54:51.280 align:middle line:90% A of b of c of x. 00:54:51.280 --> 00:54:54.820 align:middle line:90% 00:54:54.820 --> 00:55:00.000 align:middle line:84% And suppose that-- suppose these all have different number 00:55:00.000 --> 00:55:01.830 align:middle line:90% of components. 00:55:01.830 --> 00:55:07.690 align:middle line:84% Suppose this has-- f has m components. 00:55:07.690 --> 00:55:16.750 align:middle line:84% Suppose that x has n components, and these have-- let's see. 00:55:16.750 --> 00:55:18.470 align:middle line:90% c has p components. 00:55:18.470 --> 00:55:25.430 align:middle line:90% 00:55:25.430 --> 00:55:28.920 align:middle line:90% p, and b has q components. 00:55:28.920 --> 00:55:31.740 align:middle line:84% And then a-- let me color code it. 00:55:31.740 --> 00:55:39.660 align:middle line:84% So a is the output so that it has to be the same as m-- as f. 00:55:39.660 --> 00:55:42.235 align:middle line:90% So then what is f prime? 00:55:42.235 --> 00:55:45.830 align:middle line:90% 00:55:45.830 --> 00:55:49.120 align:middle line:90% That's a function of x. 00:55:49.120 --> 00:55:51.610 align:middle line:84% And it's just a product of the three Jacobians. 00:55:51.610 --> 00:55:55.140 align:middle line:90% So it's a prime of x. 00:55:55.140 --> 00:55:57.010 align:middle line:90% Am I putting my x's in red? 00:55:57.010 --> 00:55:58.720 align:middle line:90% Let me just not put x's. 00:55:58.720 --> 00:56:01.190 align:middle line:90% You know that the x's are there. 00:56:01.190 --> 00:56:06.860 align:middle line:84% It's a prime times b prime times c prime. 00:56:06.860 --> 00:56:07.360 align:middle line:90% Right? 00:56:07.360 --> 00:56:12.280 align:middle line:90% And a is m by q. 00:56:12.280 --> 00:56:13.090 align:middle line:90% My m's are in blue. 00:56:13.090 --> 00:56:16.010 align:middle line:90% 00:56:16.010 --> 00:56:17.345 align:middle line:90% By q. 00:56:17.345 --> 00:56:20.600 align:middle line:84% b prime is q by p because it takes p inputs 00:56:20.600 --> 00:56:21.710 align:middle line:90% and had q outputs. 00:56:21.710 --> 00:56:28.450 align:middle line:84% And c is p by n because it has p inputs and q outputs. 00:56:28.450 --> 00:56:30.610 align:middle line:90% And so this-- 00:56:30.610 --> 00:56:34.860 align:middle line:84% I can now-- so I can't change the order. 00:56:34.860 --> 00:56:38.030 align:middle line:84% This is matrix multiplication that's not commutative, 00:56:38.030 --> 00:56:41.700 align:middle line:90% but it is associative. 00:56:41.700 --> 00:56:44.070 align:middle line:90% I can always put parentheses. 00:56:44.070 --> 00:56:49.880 align:middle line:84% I can always multiply from left to right, or right to left. 00:56:49.880 --> 00:57:03.590 align:middle line:84% So this is doing it left to right and right to left. 00:57:03.590 --> 00:57:07.455 align:middle line:90% And these are going to have-- 00:57:07.455 --> 00:57:09.840 align:middle line:84% it seems like it's who cares, right? 00:57:09.840 --> 00:57:12.870 align:middle line:84% You do it in whatever order is convenient. 00:57:12.870 --> 00:57:16.335 align:middle line:84% It turns out, it's incredibly important. 00:57:16.335 --> 00:57:26.840 align:middle line:84% This one, from left to right, is called reverse mode, 00:57:26.840 --> 00:57:31.430 align:middle line:84% and this is called forward mode differentiation 00:57:31.430 --> 00:57:33.470 align:middle line:84% for an automatic differentiation. 00:57:33.470 --> 00:57:41.020 align:middle line:90% 00:57:41.020 --> 00:57:44.240 align:middle line:84% ALAN EDELMAN: You see the reason for the names? 00:57:44.240 --> 00:57:48.830 align:middle line:84% At first glance, it looks like the names are backwards. 00:57:48.830 --> 00:57:50.420 align:middle line:84% It looks like reverse modes going 00:57:50.420 --> 00:57:54.620 align:middle line:84% left to right, which is forward in English and many languages, 00:57:54.620 --> 00:57:57.050 align:middle line:84% and forward is going the other way. 00:57:57.050 --> 00:57:59.090 align:middle line:84% But obviously forward mode refers 00:57:59.090 --> 00:58:02.840 align:middle line:84% to the first function is c, then b, then a, 00:58:02.840 --> 00:58:05.100 align:middle line:84% and it's respecting that same order. 00:58:05.100 --> 00:58:06.650 align:middle line:90% So it's forward. 00:58:06.650 --> 00:58:07.670 align:middle line:90% Right? 00:58:07.670 --> 00:58:09.650 align:middle line:84% Reverse mode is going a, b, c which 00:58:09.650 --> 00:58:11.990 align:middle line:84% is the reverse order in which the functions are applied. 00:58:11.990 --> 00:58:14.033 align:middle line:90% 00:58:14.033 --> 00:58:14.950 align:middle line:90% STEVEN JOHNSON: Right. 00:58:14.950 --> 00:58:17.600 align:middle line:84% So forward mode is going from-- inside out 00:58:17.600 --> 00:58:21.500 align:middle line:84% is like from inputs to outputs, from starting with the input c 00:58:21.500 --> 00:58:22.938 align:middle line:90% and then going outwards. 00:58:22.938 --> 00:58:24.980 align:middle line:84% And reverse mode is going from outputs to inputs. 00:58:24.980 --> 00:58:26.075 align:middle line:90% So why does this matter? 00:58:26.075 --> 00:58:28.617 align:middle line:90% 00:58:28.617 --> 00:58:29.700 align:middle line:90% I claim that this matters. 00:58:29.700 --> 00:58:36.430 align:middle line:84% And the reason it matters is that the multiply matrices, 00:58:36.430 --> 00:58:41.560 align:middle line:84% how costly it is depends on the shape a lot. 00:58:41.560 --> 00:58:46.540 align:middle line:84% So let's think about the cost of matrix multiplication. 00:58:46.540 --> 00:58:51.710 align:middle line:90% 00:58:51.710 --> 00:58:55.505 align:middle line:90% So if I do a-- 00:58:55.505 --> 00:58:56.380 align:middle line:90% what am I doing here? 00:58:56.380 --> 00:58:59.050 align:middle line:90% 00:58:59.050 --> 00:59:02.230 align:middle line:90% m by q times q times p. 00:59:02.230 --> 00:59:10.370 align:middle line:84% So if I multiply an m by q times q by p matrices. 00:59:10.370 --> 00:59:17.710 align:middle line:90% 00:59:17.710 --> 00:59:23.340 align:middle line:84% So the result of this is, of course, an m by p tricks. 00:59:23.340 --> 00:59:26.350 align:middle line:90% 00:59:26.350 --> 00:59:27.800 align:middle line:90% So how do you do it? 00:59:27.800 --> 00:59:31.180 align:middle line:84% So for each entry of the output you've 00:59:31.180 --> 00:59:34.590 align:middle line:84% probably learned row times column. 00:59:34.590 --> 00:59:39.090 align:middle line:84% For each component of the output, you do a dot product. 00:59:39.090 --> 00:59:52.740 align:middle line:84% So this is np dot products of length q. 00:59:52.740 --> 00:59:53.830 align:middle line:90% Is that clear? 00:59:53.830 --> 00:59:54.330 align:middle line:90% All right. 00:59:54.330 --> 01:00:00.270 align:middle line:84% So for each entry, if I imagine this as a matrix, 01:00:00.270 --> 01:00:03.480 align:middle line:84% for each entry here I do a row times a column. 01:00:03.480 --> 01:00:07.350 align:middle line:84% And the rows have q entries and the columns here 01:00:07.350 --> 01:00:08.190 align:middle line:90% have q entries. 01:00:08.190 --> 01:00:10.560 align:middle line:84% So it's a dot part of length q, and there 01:00:10.560 --> 01:00:13.380 align:middle line:90% are m times p outputs. 01:00:13.380 --> 01:00:14.070 align:middle line:90% Right? 01:00:14.070 --> 01:00:17.040 align:middle line:84% And so the cost, how much does a dot product cost? 01:00:17.040 --> 01:00:20.265 align:middle line:84% Well, to take two dot products of length q-- 01:00:20.265 --> 01:00:21.630 align:middle line:90% a dot product of length q. 01:00:21.630 --> 01:00:26.412 align:middle line:90% 01:00:26.412 --> 01:00:27.750 align:middle line:90% Write it in blue. 01:00:27.750 --> 01:00:32.650 align:middle line:90% 01:00:32.650 --> 01:00:39.917 align:middle line:84% So a dot product of length q costs q multiplications, 01:00:39.917 --> 01:00:41.875 align:middle line:84% because you have to multiply q things together. 01:00:41.875 --> 01:00:45.240 align:middle line:90% 01:00:45.240 --> 01:00:54.160 align:middle line:84% And q minus 1 additions of scalars. 01:00:54.160 --> 01:01:01.540 align:middle line:84% So basically it's proportional to q scalar operations, which 01:01:01.540 --> 01:01:02.620 align:middle line:90% is what the CPU is doing. 01:01:02.620 --> 01:01:08.530 align:middle line:84% So the whole thing is proportional to mpq 01:01:08.530 --> 01:01:10.030 align:middle line:90% scalar operations. 01:01:10.030 --> 01:01:14.050 align:middle line:90% 01:01:14.050 --> 01:01:14.550 align:middle line:90% Right? 01:01:14.550 --> 01:01:17.580 align:middle line:90% So that's a review. 01:01:17.580 --> 01:01:18.795 align:middle line:90% If I do, for example-- 01:01:18.795 --> 01:01:21.700 align:middle line:90% 01:01:21.700 --> 01:01:29.100 align:middle line:84% if I do m by m times m by m, two matrices, 01:01:29.100 --> 01:01:30.960 align:middle line:90% this is proportional to m cubed. 01:01:30.960 --> 01:01:33.660 align:middle line:90% 01:01:33.660 --> 01:01:38.825 align:middle line:84% In computer science you would say this is theta mpq. 01:01:38.825 --> 01:01:42.390 align:middle line:84% That notation for proportional to. 01:01:42.390 --> 01:01:45.120 align:middle line:90% That big O notation. 01:01:45.120 --> 01:01:48.240 align:middle line:84% But I'll just say proportional to m cubed. 01:01:48.240 --> 01:01:54.180 align:middle line:90% If I do m times m times m by 1-- 01:01:54.180 --> 01:02:02.890 align:middle line:84% so if I do a matrix times vector times a column vector, 01:02:02.890 --> 01:02:06.210 align:middle line:84% that's only proportional to m squared. 01:02:06.210 --> 01:02:09.510 align:middle line:90% I only do m dot products. 01:02:09.510 --> 01:02:14.520 align:middle line:90% Or if I do 1 by m times m by m. 01:02:14.520 --> 01:02:17.220 align:middle line:84% So if I do a row vector times that, 01:02:17.220 --> 01:02:21.410 align:middle line:84% that's also proportional to m squared. 01:02:21.410 --> 01:02:22.660 align:middle line:90% So why does this matter? 01:02:22.660 --> 01:02:26.740 align:middle line:90% 01:02:26.740 --> 01:02:29.770 align:middle line:84% Does the order of the chain rule matter? 01:02:29.770 --> 01:02:35.860 align:middle line:90% 01:02:35.860 --> 01:02:42.270 align:middle line:84% So suppose you have lots of inputs and only one output. 01:02:42.270 --> 01:02:44.820 align:middle line:90% 01:02:44.820 --> 01:02:46.370 align:middle line:90% So let's do-- 01:02:46.370 --> 01:02:47.930 align:middle line:84% ALAN EDELMAN: Machine learning, you-- 01:02:47.930 --> 01:02:49.888 align:middle line:84% STEVEN JOHNSON: Yeah, like in machine learning. 01:02:49.888 --> 01:02:50.930 align:middle line:90% So we have many inputs. 01:02:50.930 --> 01:02:53.070 align:middle line:90% Lots of inputs. 01:02:53.070 --> 01:02:54.830 align:middle line:84% Actually, my inputs are red, right? 01:02:54.830 --> 01:02:58.595 align:middle line:90% So lots of inputs. 01:02:58.595 --> 01:03:01.370 align:middle line:90% 01:03:01.370 --> 01:03:03.470 align:middle line:90% n is large. 01:03:03.470 --> 01:03:04.580 align:middle line:90% Right? 01:03:04.580 --> 01:03:10.310 align:middle line:90% And only one output. 01:03:10.310 --> 01:03:13.010 align:middle line:90% So m equals 1. 01:03:13.010 --> 01:03:17.450 align:middle line:84% So this would be the case, for example, 01:03:17.450 --> 01:03:19.000 align:middle line:90% in most of optimization. 01:03:19.000 --> 01:03:19.500 align:middle line:90% Right? 01:03:19.500 --> 01:03:23.420 align:middle line:90% So in large scale optimization. 01:03:23.420 --> 01:03:24.360 align:middle line:90% Optimization. 01:03:24.360 --> 01:03:27.150 align:middle line:90% 01:03:27.150 --> 01:03:30.840 align:middle line:84% Or, for example, such as machine learning. 01:03:30.840 --> 01:03:37.530 align:middle line:90% 01:03:37.530 --> 01:03:45.477 align:middle line:84% So in that case, the output is called the loss function 01:03:45.477 --> 01:03:46.560 align:middle line:90% or the objective function. 01:03:46.560 --> 01:03:54.110 align:middle line:90% 01:03:54.110 --> 01:03:56.200 align:middle line:90% And these are the parameters. 01:03:56.200 --> 01:03:59.160 align:middle line:90% 01:03:59.160 --> 01:03:59.660 align:middle line:90% Right? 01:03:59.660 --> 01:04:03.160 align:middle line:90% The optimization parameters. 01:04:03.160 --> 01:04:05.030 align:middle line:84% So machine learning, if the neural network-- 01:04:05.030 --> 01:04:06.280 align:middle line:90% you have a big neural network. 01:04:06.280 --> 01:04:08.680 align:middle line:84% It has a billion parameters in it, 01:04:08.680 --> 01:04:11.530 align:middle line:84% and you're trying to optimize those parameters to make 01:04:11.530 --> 01:04:12.720 align:middle line:90% one number better. 01:04:12.720 --> 01:04:13.220 align:middle line:90% Right? 01:04:13.220 --> 01:04:17.230 align:middle line:84% There's something called a loss, some measure of accuracy 01:04:17.230 --> 01:04:19.850 align:middle line:84% of whatever the neural network is trying to predict, 01:04:19.850 --> 01:04:20.410 align:middle line:90% for example. 01:04:20.410 --> 01:04:22.720 align:middle line:84% You're trying to make that measure of accuracy better. 01:04:22.720 --> 01:04:26.710 align:middle line:84% So you have one output and a billion inputs. 01:04:26.710 --> 01:04:28.630 align:middle line:90% OK? 01:04:28.630 --> 01:04:30.880 align:middle line:90% And let's also suppose-- 01:04:30.880 --> 01:04:40.590 align:middle line:84% let's suppose we have lots of intermediate values, which-- 01:04:40.590 --> 01:04:42.900 align:middle line:90% what was I calling those? 01:04:42.900 --> 01:04:43.740 align:middle line:90% q and p. 01:04:43.740 --> 01:04:49.420 align:middle line:90% 01:04:49.420 --> 01:04:49.920 align:middle line:90% All right. 01:04:49.920 --> 01:04:52.530 align:middle line:84% Suppose those maybe are comparable to n. 01:04:52.530 --> 01:04:54.540 align:middle line:84% So this also happens in a neural network. 01:04:54.540 --> 01:04:57.390 align:middle line:84% You have a zillion inputs, and they 01:04:57.390 --> 01:05:01.230 align:middle line:84% go through a bunch of stages of the neural network that 01:05:01.230 --> 01:05:04.060 align:middle line:84% have lots and lots of values at each stage, 01:05:04.060 --> 01:05:07.200 align:middle line:84% and then finally, at the very end, all those outputs, 01:05:07.200 --> 01:05:09.960 align:middle line:84% you have a zillion outputs, like maybe it's 01:05:09.960 --> 01:05:11.623 align:middle line:84% trying to predict an image, and then 01:05:11.623 --> 01:05:14.040 align:middle line:84% that goes through one function that's saying, how accurate 01:05:14.040 --> 01:05:16.296 align:middle line:90% is that image? 01:05:16.296 --> 01:05:18.137 align:middle line:84% How well does it match if you're trying 01:05:18.137 --> 01:05:20.220 align:middle line:84% to generate images of dogs or something like that, 01:05:20.220 --> 01:05:22.020 align:middle line:90% or how good is it? 01:05:22.020 --> 01:05:22.770 align:middle line:90% Right? 01:05:22.770 --> 01:05:25.170 align:middle line:84% Or this happens in engineering optimization, right? 01:05:25.170 --> 01:05:27.420 align:middle line:84% Which is not really data driven, but suppose 01:05:27.420 --> 01:05:31.530 align:middle line:84% you're trying to design an airplane wing that 01:05:31.530 --> 01:05:33.120 align:middle line:90% is as strong as possible. 01:05:33.120 --> 01:05:34.710 align:middle line:84% Then you have a zillion inputs which 01:05:34.710 --> 01:05:37.920 align:middle line:84% are describing the distribution of all 01:05:37.920 --> 01:05:41.940 align:middle line:84% the material over the whole shape of the airplane wing. 01:05:41.940 --> 01:05:46.080 align:middle line:84% But you only have one output at the end of the day, which 01:05:46.080 --> 01:05:49.600 align:middle line:84% is your one number which is some measure of how strong it 01:05:49.600 --> 01:05:52.420 align:middle line:84% is, how much weight can it hold, or something like that. 01:05:52.420 --> 01:05:54.130 align:middle line:90% Right? 01:05:54.130 --> 01:05:55.780 align:middle line:84% And you take those zillion inputs, 01:05:55.780 --> 01:05:58.960 align:middle line:84% get a zillion numbers which are all the stresses everywhere 01:05:58.960 --> 01:06:01.270 align:middle line:84% on the airplane wing, and then pass it 01:06:01.270 --> 01:06:03.545 align:middle line:84% through another function that takes all those stresses 01:06:03.545 --> 01:06:05.170 align:middle line:84% and works out the strength of the wing, 01:06:05.170 --> 01:06:06.920 align:middle line:84% and that's what you're trying to maximize. 01:06:06.920 --> 01:06:07.510 align:middle line:90% All right? 01:06:07.510 --> 01:06:14.550 align:middle line:84% So then think about what the shape of the chain rule is. 01:06:14.550 --> 01:06:15.080 align:middle line:90% All right? 01:06:15.080 --> 01:06:17.650 align:middle line:90% So our f prime-- 01:06:17.650 --> 01:06:18.150 align:middle line:90% right? 01:06:18.150 --> 01:06:23.050 align:middle line:84% So this is one output by n inputs, 01:06:23.050 --> 01:06:24.760 align:middle line:90% so really, this is a gradient. 01:06:24.760 --> 01:06:27.440 align:middle line:90% 01:06:27.440 --> 01:06:28.070 align:middle line:90% All right? 01:06:28.070 --> 01:06:29.660 align:middle line:84% And this is what you're going to typically going 01:06:29.660 --> 01:06:31.610 align:middle line:84% to want in optimization machine learning. 01:06:31.610 --> 01:06:33.140 align:middle line:84% We'll talk about this more later, 01:06:33.140 --> 01:06:36.380 align:middle line:84% that you want the-- the gradient gives you 01:06:36.380 --> 01:06:38.510 align:middle line:84% basically the direction, the uphill direction-- 01:06:38.510 --> 01:06:40.490 align:middle line:84% or the downhill direction is minus gradient. 01:06:40.490 --> 01:06:42.020 align:middle line:84% It tells you in what direction do 01:06:42.020 --> 01:06:45.530 align:middle line:84% you perturb the parameters to make your system better, 01:06:45.530 --> 01:06:47.300 align:middle line:84% like in what direction do you change 01:06:47.300 --> 01:06:48.650 align:middle line:84% all the parameters in the neural network 01:06:48.650 --> 01:06:50.650 align:middle line:84% to make it more accurate, to make the loss less, 01:06:50.650 --> 01:06:54.110 align:middle line:84% or what direction do you perturb all the parameters describing 01:06:54.110 --> 01:06:57.060 align:middle line:84% the shape of your airplane wing to make it stronger. 01:06:57.060 --> 01:06:57.560 align:middle line:90% Right? 01:06:57.560 --> 01:07:00.810 align:middle line:84% So you really care about getting this gradient, these n numbers. 01:07:00.810 --> 01:07:01.310 align:middle line:90% OK? 01:07:01.310 --> 01:07:04.040 align:middle line:84% But what is that in terms of the chain rule? 01:07:04.040 --> 01:07:08.530 align:middle line:90% We said it's g prime times-- 01:07:08.530 --> 01:07:09.030 align:middle line:90% sorry. 01:07:09.030 --> 01:07:12.110 align:middle line:84% It's a prime times b prime times c prime. 01:07:12.110 --> 01:07:12.740 align:middle line:90% Right? 01:07:12.740 --> 01:07:21.610 align:middle line:84% So it's a prime times b prime times c prime. 01:07:21.610 --> 01:07:31.815 align:middle line:84% And we said a is m, m by q. m is one, so one by q. 01:07:31.815 --> 01:07:33.450 align:middle line:84% Let's just suppose for simplicity 01:07:33.450 --> 01:07:34.950 align:middle line:90% that these are just equal to n. 01:07:34.950 --> 01:07:38.570 align:middle line:90% 01:07:38.570 --> 01:07:40.685 align:middle line:90% So this is 1 by n. 01:07:40.685 --> 01:07:43.790 align:middle line:90% 01:07:43.790 --> 01:07:46.235 align:middle line:90% b is n by n. 01:07:46.235 --> 01:07:49.630 align:middle line:90% 01:07:49.630 --> 01:07:54.490 align:middle line:90% c is n by n. 01:07:54.490 --> 01:07:56.320 align:middle line:90% c prime is n by n. 01:07:56.320 --> 01:07:57.430 align:middle line:90% Right? 01:07:57.430 --> 01:08:00.790 align:middle line:84% c takes n inputs, your x's, gives you 01:08:00.790 --> 01:08:04.140 align:middle line:84% n intermediate outputs, p, and then b takes those n things 01:08:04.140 --> 01:08:05.140 align:middle line:90% and gives you another n. 01:08:05.140 --> 01:08:07.370 align:middle line:84% So its Jacobian is another n by n. 01:08:07.370 --> 01:08:09.712 align:middle line:90% Then you multiply by a. 01:08:09.712 --> 01:08:13.915 align:middle line:84% And notice it really matters where 01:08:13.915 --> 01:08:15.540 align:middle line:84% you put the parentheses, because if you 01:08:15.540 --> 01:08:21.410 align:middle line:84% put the parentheses here, think what you're doing, 01:08:21.410 --> 01:08:25.890 align:middle line:84% versus if you put the parentheses in the other place. 01:08:25.890 --> 01:08:27.314 align:middle line:90% a prime. 01:08:27.314 --> 01:08:28.830 align:middle line:90% b prime. 01:08:28.830 --> 01:08:29.790 align:middle line:90% c prime. 01:08:29.790 --> 01:08:32.820 align:middle line:84% If you do forward versus reverse mode. 01:08:32.820 --> 01:08:35.540 align:middle line:90% 01:08:35.540 --> 01:08:36.040 align:middle line:90% Right? 01:08:36.040 --> 01:08:40.210 align:middle line:84% So if you put the parentheses here, this is our reverse mode. 01:08:40.210 --> 01:08:47.130 align:middle line:90% 01:08:47.130 --> 01:08:50.269 align:middle line:90% a, b. 01:08:50.269 --> 01:08:54.240 align:middle line:84% This product here is a row vector times a matrix. 01:08:54.240 --> 01:08:56.760 align:middle line:90% Vector times matrix is cheap. 01:08:56.760 --> 01:09:00.660 align:middle line:84% That's m proportional to n operations. 01:09:00.660 --> 01:09:02.590 align:middle line:84% And then what's the output of this? 01:09:02.590 --> 01:09:05.460 align:middle line:84% It's the whole thing, the product of vector times 01:09:05.460 --> 01:09:08.200 align:middle line:84% matrix-- row vector times matrix is another row vector. 01:09:08.200 --> 01:09:11.229 align:middle line:84% And so then you multiply this row vector times a matrix. 01:09:11.229 --> 01:09:19.154 align:middle line:84% So you have the whole thing costs order n cost. 01:09:19.154 --> 01:09:20.904 align:middle line:84% ALAN EDELMAN: Vector times matrix, Steven? 01:09:20.904 --> 01:09:21.399 align:middle line:90% STEVEN JOHNSON: Sorry. 01:09:21.399 --> 01:09:22.120 align:middle line:90% Order n squared. 01:09:22.120 --> 01:09:24.439 align:middle line:90% Sorry. 01:09:24.439 --> 01:09:25.340 align:middle line:90% Yes. 01:09:25.340 --> 01:09:27.060 align:middle line:90% It's proportional to n squared. 01:09:27.060 --> 01:09:27.560 align:middle line:90% Right? 01:09:27.560 --> 01:09:30.910 align:middle line:84% Because it's vector times matrix. 01:09:30.910 --> 01:09:31.470 align:middle line:90% Right? 01:09:31.470 --> 01:09:43.620 align:middle line:84% Whereas if you do forward mode, if you multiply right to left, 01:09:43.620 --> 01:09:44.760 align:middle line:90% what's the cost? 01:09:44.760 --> 01:09:47.130 align:middle line:84% Well, you're doing b times c first. 01:09:47.130 --> 01:09:48.660 align:middle line:90% This is matrix times matrix. 01:09:48.660 --> 01:09:50.370 align:middle line:90% That's much harder. 01:09:50.370 --> 01:09:54.232 align:middle line:84% Matrix times matrix is n cubed operations, 01:09:54.232 --> 01:09:56.440 align:middle line:84% and then you multiply by a vector which is n squared. 01:09:56.440 --> 01:10:01.090 align:middle line:84% So the whole thing ends up being n cubed. 01:10:01.090 --> 01:10:04.635 align:middle line:84% So you really, really want to do it in this order. 01:10:04.635 --> 01:10:07.680 align:middle line:90% 01:10:07.680 --> 01:10:09.430 align:middle line:84% ALAN EDELMAN: This is probably a good time 01:10:09.430 --> 01:10:10.650 align:middle line:90% to take a few minute break. 01:10:10.650 --> 01:10:11.380 align:middle line:90% We're a little late. 01:10:11.380 --> 01:10:12.213 align:middle line:90% STEVEN JOHNSON: Yep. 01:10:12.213 --> 01:10:14.110 align:middle line:90% And let me just say-- 01:10:14.110 --> 01:10:15.910 align:middle line:84% we're going to take a break in a second. 01:10:15.910 --> 01:10:19.300 align:middle line:90% So this order is also called-- 01:10:19.300 --> 01:10:20.440 align:middle line:90% this is reverse mode. 01:10:20.440 --> 01:10:23.430 align:middle line:90% 01:10:23.430 --> 01:10:24.520 align:middle line:90% It has a lot of names. 01:10:24.520 --> 01:10:26.550 align:middle line:84% It's also called back propagation. 01:10:26.550 --> 01:10:31.010 align:middle line:90% 01:10:31.010 --> 01:10:37.001 align:middle line:84% And it's also called an adjoint differentiation. 01:10:37.001 --> 01:10:40.260 align:middle line:90% 01:10:40.260 --> 01:10:43.950 align:middle line:84% And it's the key to optimizing functions 01:10:43.950 --> 01:10:46.290 align:middle line:90% over zillions of parameters. 01:10:46.290 --> 01:10:50.940 align:middle line:84% It means that I can compute all the derivatives with respect 01:10:50.940 --> 01:10:55.380 align:middle line:84% to all the parameters in quite a reasonable cost. 01:10:55.380 --> 01:10:58.020 align:middle line:84% And basically it ends up being in cost proportional 01:10:58.020 --> 01:11:01.050 align:middle line:84% to the cost of just evaluating f. 01:11:01.050 --> 01:11:03.915 align:middle line:84% Whereas if I did on the opposite direction, 01:11:03.915 --> 01:11:06.390 align:middle line:84% if I have a billion parameters, it becomes a billion times 01:11:06.390 --> 01:11:07.590 align:middle line:90% more expensive. 01:11:07.590 --> 01:11:10.935 align:middle line:84% ALAN EDELMAN: And I do like to stress that most students-- 01:11:10.935 --> 01:11:13.785 align:middle line:84% when I did the poll earlier in this classroom, 01:11:13.785 --> 01:11:14.910 align:middle line:90% it showed itself. 01:11:14.910 --> 01:11:17.370 align:middle line:84% Most students, even for scalar calculus, 01:11:17.370 --> 01:11:20.680 align:middle line:84% do the outputs towards the inputs 01:11:20.680 --> 01:11:21.930 align:middle line:90% working their way from output. 01:11:21.930 --> 01:11:23.292 align:middle line:90% When I ask you about sine of-- 01:11:23.292 --> 01:11:24.750 align:middle line:84% STEVEN JOHNSON: Forward mode, yeah. 01:11:24.750 --> 01:11:27.975 align:middle line:84% ALAN EDELMAN: Or sine of x squared or whatever I asked, 01:11:27.975 --> 01:11:29.268 align:middle line:90% you are all doing-- 01:11:29.268 --> 01:11:31.560 align:middle line:84% without even thinking about it, it's all second nature. 01:11:31.560 --> 01:11:33.120 align:middle line:84% You're going from outputs to inputs. 01:11:33.120 --> 01:11:37.290 align:middle line:84% And so this reverse mode, which is often 01:11:37.290 --> 01:11:40.410 align:middle line:84% more efficient for many real applications today 01:11:40.410 --> 01:11:43.050 align:middle line:84% is also exactly what you're doing instinctively 01:11:43.050 --> 01:11:44.987 align:middle line:90% without even thinking about it. 01:11:44.987 --> 01:11:45.820 align:middle line:90% STEVEN JOHNSON: Yes. 01:11:45.820 --> 01:11:47.140 align:middle line:90% Well, it depends. 01:11:47.140 --> 01:11:48.730 align:middle line:84% When it's more complicated functions, 01:11:48.730 --> 01:11:51.340 align:middle line:90% sometimes reverse mode is like-- 01:11:51.340 --> 01:11:53.740 align:middle line:84% usually when you're doing more complicated things, 01:11:53.740 --> 01:11:57.145 align:middle line:84% I would say the forward mode is often the more obvious. 01:11:57.145 --> 01:11:58.270 align:middle line:90% ALAN EDELMAN: That's right. 01:11:58.270 --> 01:11:59.645 align:middle line:84% I wasn't going to say that today. 01:11:59.645 --> 01:12:00.610 align:middle line:90% STEVEN JOHNSON: Yeah. 01:12:00.610 --> 01:12:01.870 align:middle line:90% And we'll talk about it. 01:12:01.870 --> 01:12:04.360 align:middle line:84% And we'll talk more about how these things are evaluated. 01:12:04.360 --> 01:12:08.550 align:middle line:84% How many of you have heard of back propagation, by the way? 01:12:08.550 --> 01:12:10.380 align:middle line:90% ALAN EDELMAN: Everybody but one. 01:12:10.380 --> 01:12:12.320 align:middle line:84% STEVEN JOHNSON: Everybody but one. 01:12:12.320 --> 01:12:14.690 align:middle line:84% But how many people knew that it was just 01:12:14.690 --> 01:12:17.480 align:middle line:84% multiplying Jacobians left to right instead of right to left? 01:12:17.480 --> 01:12:20.257 align:middle line:90% 01:12:20.257 --> 01:12:22.340 align:middle line:84% ALAN EDELMAN: Three, including one who never heard 01:12:22.340 --> 01:12:23.990 align:middle line:90% of the term back propagation. 01:12:23.990 --> 01:12:25.460 align:middle line:90% STEVEN JOHNSON: OK. 01:12:25.460 --> 01:12:27.620 align:middle line:90% Yeah. 01:12:27.620 --> 01:12:30.180 align:middle line:84% And we'll show more examples of this. 01:12:30.180 --> 01:12:34.910 align:middle line:84% It's one of those things that once you see it explained, 01:12:34.910 --> 01:12:36.500 align:middle line:90% it's obvious and trivial. 01:12:36.500 --> 01:12:41.870 align:middle line:84% It's just left to right versus right to left, after all. 01:12:41.870 --> 01:12:46.800 align:middle line:84% But I think that until you see it explained-- 01:12:46.800 --> 01:12:51.500 align:middle line:84% so for example, if you're computing 01:12:51.500 --> 01:12:53.510 align:middle line:84% the strength of an airplane wing as a function 01:12:53.510 --> 01:12:55.040 align:middle line:84% of 10 billion parameters describing 01:12:55.040 --> 01:12:57.890 align:middle line:84% the shape of the wing, it's a really huge 01:12:57.890 --> 01:13:00.620 align:middle line:84% mechanical stimulation, like a solid mechanics simulation, 01:13:00.620 --> 01:13:04.200 align:middle line:84% to calculate the strength of that wing given its shape. 01:13:04.200 --> 01:13:06.860 align:middle line:84% And it's really not obvious that you 01:13:06.860 --> 01:13:09.260 align:middle line:84% can get the derivative of that strength with respect 01:13:09.260 --> 01:13:11.720 align:middle line:84% to all billion parameters with basically 01:13:11.720 --> 01:13:14.900 align:middle line:90% one additional mechanics solve. 01:13:14.900 --> 01:13:17.390 align:middle line:84% And it turns out that that's another instance 01:13:17.390 --> 01:13:18.630 align:middle line:90% of this same thing. 01:13:18.630 --> 01:13:20.340 align:middle line:84% But once you see it explained-- and we'll 01:13:20.340 --> 01:13:24.120 align:middle line:84% see that kind of example more explicitly later on. 01:13:24.120 --> 01:13:28.710 align:middle line:84% I think it becomes much more straightforward. 01:13:28.710 --> 01:13:31.187 align:middle line:84% Really, it's surprisingly non-obvious 01:13:31.187 --> 01:13:33.645 align:middle line:84% when it gets to complicated functions that you can do this. 01:13:33.645 --> 01:13:36.525 align:middle line:90% 01:13:36.525 --> 01:13:38.900 align:middle line:84% ALAN EDELMAN: But I think maybe this is just a great time 01:13:38.900 --> 01:13:40.280 align:middle line:84% to just take a five minute break. 01:13:40.280 --> 01:13:41.180 align:middle line:90% STEVEN JOHNSON: Yeah. 01:13:41.180 --> 01:13:44.450 align:middle line:84% ALAN EDELMAN: So I've got 12:23 on the clock behind-- 01:13:44.450 --> 01:13:49.750 align:middle line:84% in the back of this room, so how about we reconvene at 12:28? 01:13:49.750 --> 01:13:57.000 align:middle line:90%