WEBVTT 00:00:00.000 --> 00:00:01.948 align:middle line:90% [SQUEAKING] 00:00:01.948 --> 00:00:03.409 align:middle line:90% [RUSTLING] 00:00:03.409 --> 00:00:05.357 align:middle line:90% [CLICKING] 00:00:05.357 --> 00:00:10.390 align:middle line:90% 00:00:10.390 --> 00:00:11.890 align:middle line:84% ALAN EDELMAN: So welcome, everybody, 00:00:11.890 --> 00:00:14.850 align:middle line:84% to this IAP class on matrix calculus. 00:00:14.850 --> 00:00:16.170 align:middle line:90% I'm Professor Alan Edelman. 00:00:16.170 --> 00:00:21.270 align:middle line:84% And the other professor is Steven Johnson, 00:00:21.270 --> 00:00:22.890 align:middle line:90% who's going to have-- 00:00:22.890 --> 00:00:24.120 align:middle line:90% he had to go out of town. 00:00:24.120 --> 00:00:28.020 align:middle line:84% And so we're going to take advantage of Zoom technology 00:00:28.020 --> 00:00:30.210 align:middle line:84% so that he could give lectures remotely. 00:00:30.210 --> 00:00:32.340 align:middle line:84% We did some testing of the audio and the video. 00:00:32.340 --> 00:00:35.410 align:middle line:84% So hopefully it'll work out just fine. 00:00:35.410 --> 00:00:39.460 align:middle line:84% So maybe just a quick introduction. 00:00:39.460 --> 00:00:43.060 align:middle line:90% So-- oh, good, good. 00:00:43.060 --> 00:00:44.738 align:middle line:90% Yes, Steven-- recording? 00:00:44.738 --> 00:00:46.280 align:middle line:84% STEVEN JOHNSON: I already started it. 00:00:46.280 --> 00:00:47.000 align:middle line:90% ALAN EDELMAN: All right. 00:00:47.000 --> 00:00:48.220 align:middle line:90% Thank you for the reminder. 00:00:48.220 --> 00:00:48.720 align:middle line:90% Yeah. 00:00:48.720 --> 00:00:50.860 align:middle line:84% So I even put the two reminders on the blackboard. 00:00:50.860 --> 00:00:52.570 align:middle line:84% Yeah, so that was reminder number one. 00:00:52.570 --> 00:00:53.270 align:middle line:90% Good. 00:00:53.270 --> 00:00:53.770 align:middle line:90% OK. 00:00:53.770 --> 00:00:55.640 align:middle line:90% So let's see. 00:00:55.640 --> 00:00:59.590 align:middle line:84% So both of us are in the math department. 00:00:59.590 --> 00:01:01.750 align:middle line:90% I'm also in CSAIL. 00:01:01.750 --> 00:01:05.560 align:middle line:84% And I run the Julia lab over in CSAIL. 00:01:05.560 --> 00:01:11.470 align:middle line:84% And my own research involves both mathematics as well as 00:01:11.470 --> 00:01:12.820 align:middle line:90% computing software. 00:01:12.820 --> 00:01:15.817 align:middle line:84% Steven, do you want to say a few words quickly about yourself? 00:01:15.817 --> 00:01:17.150 align:middle line:90% STEVEN JOHNSON: Hello, everyone. 00:01:17.150 --> 00:01:20.710 align:middle line:84% So some of you may remember me if you took 18.06 00:01:20.710 --> 00:01:22.030 align:middle line:90% in the spring-- 00:01:22.030 --> 00:01:23.410 align:middle line:84% ALAN EDELMAN: How many of you took 18.06 in the spring? 00:01:23.410 --> 00:01:25.330 align:middle line:84% STEVEN JOHNSON: --or in the fall with me. 00:01:25.330 --> 00:01:26.538 align:middle line:90% ALAN EDELMAN: Or in the fall. 00:01:26.538 --> 00:01:28.250 align:middle line:84% So there's two, or three, or four hands. 00:01:28.250 --> 00:01:28.750 align:middle line:90% OK. 00:01:28.750 --> 00:01:29.065 align:middle line:90% STEVEN JOHNSON: Yep. 00:01:29.065 --> 00:01:29.565 align:middle line:90% Yeah. 00:01:29.565 --> 00:01:32.410 align:middle line:84% So those people who took 18.06 with me 00:01:32.410 --> 00:01:37.000 align:middle line:84% will have gotten one hour of these 16 hours of lectures 00:01:37.000 --> 00:01:40.240 align:middle line:84% that we're going to do this IAP on. 00:01:40.240 --> 00:01:41.940 align:middle line:90% And that'll be familiar. 00:01:41.940 --> 00:01:42.440 align:middle line:90% But yeah. 00:01:42.440 --> 00:01:44.360 align:middle line:84% So I'm also in the math department. 00:01:44.360 --> 00:01:45.890 align:middle line:90% And I'm also in physics. 00:01:45.890 --> 00:01:50.750 align:middle line:84% And I come into matrix calculus through a lot 00:01:50.750 --> 00:01:53.660 align:middle line:84% of PDE-constrained optimization mainly. 00:01:53.660 --> 00:01:57.500 align:middle line:84% But I've also worked with Julia, which will also 00:01:57.500 --> 00:02:00.380 align:middle line:90% be used on the problem sets. 00:02:00.380 --> 00:02:04.130 align:middle line:84% And so I'm sorry I can't be there with you in person. 00:02:04.130 --> 00:02:05.540 align:middle line:90% I'll be missing-- 00:02:05.540 --> 00:02:07.980 align:middle line:84% I'll be doing it remotely for the first week and a half. 00:02:07.980 --> 00:02:09.313 align:middle line:90% And then I'll be back in person. 00:02:09.313 --> 00:02:12.242 align:middle line:90% 00:02:12.242 --> 00:02:12.950 align:middle line:90% ALAN EDELMAN: OK. 00:02:12.950 --> 00:02:18.020 align:middle line:84% So let me quickly show everybody the GitHub site 00:02:18.020 --> 00:02:19.670 align:middle line:90% that we're going to be using. 00:02:19.670 --> 00:02:21.560 align:middle line:90% Where-- did it disappear on me? 00:02:21.560 --> 00:02:23.120 align:middle line:90% Where did it go? 00:02:23.120 --> 00:02:23.880 align:middle line:90% Oh, there it is. 00:02:23.880 --> 00:02:24.380 align:middle line:90% OK. 00:02:24.380 --> 00:02:26.330 align:middle line:90% So-- oh, no. 00:02:26.330 --> 00:02:27.680 align:middle line:90% That's not the right one. 00:02:27.680 --> 00:02:29.790 align:middle line:90% This was another course. 00:02:29.790 --> 00:02:30.290 align:middle line:90% All right. 00:02:30.290 --> 00:02:32.490 align:middle line:84% I think we should just dive right in. 00:02:32.490 --> 00:02:34.320 align:middle line:90% So let me go ahead and do that. 00:02:34.320 --> 00:02:39.530 align:middle line:84% So yes, here are the lectures, the problem sets-- three units. 00:02:39.530 --> 00:02:43.130 align:middle line:84% And we assume people are familiar with linear algebra 00:02:43.130 --> 00:02:44.190 align:middle line:90% and not much more. 00:02:44.190 --> 00:02:44.690 align:middle line:90% OK? 00:02:44.690 --> 00:02:46.820 align:middle line:84% And there's no assumption that you've already 00:02:46.820 --> 00:02:47.900 align:middle line:90% used Julia before. 00:02:47.900 --> 00:02:49.520 align:middle line:84% But of course, if you've taken 18.06 00:02:49.520 --> 00:02:52.400 align:middle line:84% with Professor Johnson or some other courses, 00:02:52.400 --> 00:02:54.200 align:middle line:84% you may already be somewhat familiar. 00:02:54.200 --> 00:02:56.390 align:middle line:90% But you could easily learn. 00:02:56.390 --> 00:02:58.100 align:middle line:84% The kind of Julia we're using here 00:02:58.100 --> 00:03:00.500 align:middle line:84% is sort of basic calculator Julia. 00:03:00.500 --> 00:03:03.570 align:middle line:84% And so you won't need to know much. 00:03:03.570 --> 00:03:04.070 align:middle line:90% OK? 00:03:04.070 --> 00:03:09.600 align:middle line:84% So let me delve right into where does matrix calculus fit in. 00:03:09.600 --> 00:03:12.350 align:middle line:84% So if you look at MIT's course catalog, which 00:03:12.350 --> 00:03:14.390 align:middle line:90% I've copied over here-- 00:03:14.390 --> 00:03:16.760 align:middle line:84% this is replicated probably in universities 00:03:16.760 --> 00:03:17.900 align:middle line:90% all over the planet, right? 00:03:17.900 --> 00:03:19.970 align:middle line:84% There's single-variable calculus. 00:03:19.970 --> 00:03:22.010 align:middle line:84% This is, like, the first semester of calculus. 00:03:22.010 --> 00:03:25.040 align:middle line:84% And it's required at MIT for all undergraduates 00:03:25.040 --> 00:03:27.980 align:middle line:84% to take 18.01 and learn how to take 00:03:27.980 --> 00:03:31.440 align:middle line:84% the derivative or an integral of a function of one variable. 00:03:31.440 --> 00:03:31.940 align:middle line:90% OK? 00:03:31.940 --> 00:03:34.490 align:middle line:84% And then there's that second semester 00:03:34.490 --> 00:03:38.220 align:middle line:84% of the sequence, 18.02, where you learn vector calculus 00:03:38.220 --> 00:03:39.930 align:middle line:90% or multivariate calculus, right? 00:03:39.930 --> 00:03:42.390 align:middle line:84% And so you learn the basic definitions 00:03:42.390 --> 00:03:44.880 align:middle line:90% of a gradient or a Jacobian. 00:03:44.880 --> 00:03:47.040 align:middle line:84% And that's kind of this whole 18.02 thing. 00:03:47.040 --> 00:03:49.770 align:middle line:84% And like I said, this is in every university 00:03:49.770 --> 00:03:50.500 align:middle line:90% around the world. 00:03:50.500 --> 00:03:51.000 align:middle line:90% OK? 00:03:51.000 --> 00:03:52.620 align:middle line:84% And I bet everybody in this room has 00:03:52.620 --> 00:03:55.170 align:middle line:84% gone through these two classes one way or another, 00:03:55.170 --> 00:03:57.450 align:middle line:84% or learned it on the streets, or something. 00:03:57.450 --> 00:04:02.280 align:middle line:84% So it seems to me that there's actually 00:04:02.280 --> 00:04:06.040 align:middle line:84% a sequence that's sort of been cut off completely arbitrarily, 00:04:06.040 --> 00:04:06.540 align:middle line:90% right? 00:04:06.540 --> 00:04:10.260 align:middle line:84% The sequence should go scalar, vector, right? 00:04:10.260 --> 00:04:13.930 align:middle line:84% So that's one, two dimensions-- matrix. 00:04:13.930 --> 00:04:14.430 align:middle line:90% Right? 00:04:14.430 --> 00:04:15.840 align:middle line:84% Scalar is zero-dimensional; vector is one-dimensional; 00:04:15.840 --> 00:04:18.040 align:middle line:84% and matrices are two-dimensional; 00:04:18.040 --> 00:04:20.760 align:middle line:84% and then, of course, higher dimensional arrays, right? 00:04:20.760 --> 00:04:25.290 align:middle line:84% It just makes sense that these objects are-- 00:04:25.290 --> 00:04:27.030 align:middle line:84% you should be able to do calculus 00:04:27.030 --> 00:04:28.200 align:middle line:90% on any of these objects. 00:04:28.200 --> 00:04:31.680 align:middle line:84% When you talk about programming languages, for example, some 00:04:31.680 --> 00:04:32.550 align:middle line:90% but not-- 00:04:32.550 --> 00:04:35.880 align:middle line:84% every programming language has two-dimensional structures 00:04:35.880 --> 00:04:37.780 align:middle line:90% and more, right-- arrays. 00:04:37.780 --> 00:04:39.810 align:middle line:84% So if you're familiar with Matlab, 00:04:39.810 --> 00:04:41.670 align:middle line:84% it does not have a one-dimensional array. 00:04:41.670 --> 00:04:47.040 align:middle line:84% People talk about n-by-1 column vectors and 1-by-n row vectors. 00:04:47.040 --> 00:04:49.860 align:middle line:84% But they're really just matrices, right? 00:04:49.860 --> 00:04:52.050 align:middle line:84% Other languages have one-dimensional arrays. 00:04:52.050 --> 00:04:54.480 align:middle line:84% Some even have zero-dimensional arrays, right? 00:04:54.480 --> 00:04:58.514 align:middle line:84% So these are the things that you find in programming languages. 00:04:58.514 --> 00:05:00.330 align:middle line:90% And this is Julia. 00:05:00.330 --> 00:05:03.030 align:middle line:84% The size of a matrix, for example, has two components. 00:05:03.030 --> 00:05:06.220 align:middle line:84% The size of a vector has one component, et cetera. 00:05:06.220 --> 00:05:09.540 align:middle line:90% So it seems to me-- 00:05:09.540 --> 00:05:11.868 align:middle line:90% I've been teaching at MIT now-- 00:05:11.868 --> 00:05:13.410 align:middle line:84% I don't even want to count the years. 00:05:13.410 --> 00:05:15.510 align:middle line:84% We're pushing on 30 years now for me. 00:05:15.510 --> 00:05:19.230 align:middle line:84% And when I started here, linear algebra 00:05:19.230 --> 00:05:21.100 align:middle line:90% was not what it is today. 00:05:21.100 --> 00:05:22.170 align:middle line:90% I mean, maybe you have-- 00:05:22.170 --> 00:05:24.480 align:middle line:84% I don't know if you folks know, but linear algebra 00:05:24.480 --> 00:05:25.800 align:middle line:90% was this thing to avoid. 00:05:25.800 --> 00:05:27.870 align:middle line:84% It was this, like, required course maybe. 00:05:27.870 --> 00:05:30.540 align:middle line:84% And a few people needed it perhaps. 00:05:30.540 --> 00:05:35.070 align:middle line:84% But I think because of machine learning, and statistics, 00:05:35.070 --> 00:05:38.070 align:middle line:84% and lots of other reasons, linear algebra has gradually 00:05:38.070 --> 00:05:45.000 align:middle line:84% taken over a much bigger part of today's tools 00:05:45.000 --> 00:05:48.210 align:middle line:84% for lots and lots of areas compared 00:05:48.210 --> 00:05:50.140 align:middle line:90% to when I started 30 years ago. 00:05:50.140 --> 00:05:52.950 align:middle line:84% And so machine learning, statistics, engineering-- 00:05:52.950 --> 00:05:55.920 align:middle line:90% everybody needs linear algebra. 00:05:55.920 --> 00:05:59.670 align:middle line:84% And so it stands to reason that you'd want to be able to do 00:05:59.670 --> 00:06:03.940 align:middle line:84% calculus on matrices and higher-dimensional objects, 00:06:03.940 --> 00:06:06.090 align:middle line:84% especially because everybody these days-- 00:06:06.090 --> 00:06:08.110 align:middle line:90% they're doing machine learning. 00:06:08.110 --> 00:06:10.420 align:middle line:84% I don't have to tell you because everybody in MIT 00:06:10.420 --> 00:06:11.890 align:middle line:90% is already doing it, right? 00:06:11.890 --> 00:06:14.508 align:middle line:84% Machine learning-- you need to take gradients. 00:06:14.508 --> 00:06:17.050 align:middle line:84% And you need to take gradients of complicated objects, right? 00:06:17.050 --> 00:06:18.730 align:middle line:84% So you want to do gradient descent? 00:06:18.730 --> 00:06:21.056 align:middle line:84% You need to be able to do this sort of thing. 00:06:21.056 --> 00:06:23.650 align:middle line:90% 00:06:23.650 --> 00:06:25.780 align:middle line:84% But I've always been interested in matrix calculus. 00:06:25.780 --> 00:06:26.605 align:middle line:90% I always just thought it was-- 00:06:26.605 --> 00:06:28.150 align:middle line:84% I always liked calculus to begin with. 00:06:28.150 --> 00:06:30.430 align:middle line:84% And I thought the idea of doing calculus with matrices 00:06:30.430 --> 00:06:31.420 align:middle line:90% just seemed like a-- 00:06:31.420 --> 00:06:32.670 align:middle line:90% I always liked linear algebra. 00:06:32.670 --> 00:06:34.840 align:middle line:84% I always thought it would be sort of fun to marry 00:06:34.840 --> 00:06:36.730 align:middle line:90% calculus and linear algebra. 00:06:36.730 --> 00:06:38.920 align:middle line:90% And I used to go to the library. 00:06:38.920 --> 00:06:41.770 align:middle line:84% And then when Google came along, you'd Google "matrix calculus." 00:06:41.770 --> 00:06:44.560 align:middle line:84% And up until fairly recently, I was 00:06:44.560 --> 00:06:47.050 align:middle line:84% rather disappointed with what I found, right? 00:06:47.050 --> 00:06:48.270 align:middle line:90% Matrix calculus was-- 00:06:48.270 --> 00:06:50.770 align:middle line:84% I found, like, three books I think with the title of "matrix 00:06:50.770 --> 00:06:51.490 align:middle line:90% calculus." 00:06:51.490 --> 00:06:52.910 align:middle line:84% And the moment I opened the book, 00:06:52.910 --> 00:06:54.368 align:middle line:84% I realized, no, no, no, this is not 00:06:54.368 --> 00:06:55.720 align:middle line:90% what I wanted at all, right? 00:06:55.720 --> 00:06:59.740 align:middle line:84% I wanted to be able to do the kind of calculus 00:06:59.740 --> 00:07:01.660 align:middle line:90% we did in college-- 00:07:01.660 --> 00:07:05.330 align:middle line:84% 18.01, 18.02 calculus-- but on higher-dimensional objects. 00:07:05.330 --> 00:07:05.830 align:middle line:90% OK? 00:07:05.830 --> 00:07:09.850 align:middle line:84% So here's a little bit of a quick question for all of you 00:07:09.850 --> 00:07:12.590 align:middle line:84% to think about if you don't already know the answer. 00:07:12.590 --> 00:07:14.590 align:middle line:84% You might ask yourself, for example, 00:07:14.590 --> 00:07:18.520 align:middle line:84% suppose you have the function that squares a matrix, right? 00:07:18.520 --> 00:07:20.820 align:middle line:90% What should the derivative be? 00:07:20.820 --> 00:07:22.570 align:middle line:84% In fact, if you've never done this before, 00:07:22.570 --> 00:07:25.870 align:middle line:84% you might not even know what it should look like, right? 00:07:25.870 --> 00:07:29.560 align:middle line:84% Should it be 2x, like the scalar case? 00:07:29.560 --> 00:07:31.960 align:middle line:84% Well, I'll tell you right now that that's not correct 00:07:31.960 --> 00:07:34.890 align:middle line:90% when x is not a scalar. 00:07:34.890 --> 00:07:38.290 align:middle line:84% And then, of course, you could ask about 00:07:38.290 --> 00:07:40.180 align:middle line:84% what is the derivative of the matrix inverse, 00:07:40.180 --> 00:07:44.110 align:middle line:84% or what is the derivative of the matrix inverse squared, 00:07:44.110 --> 00:07:46.250 align:middle line:90% and so forth. 00:07:46.250 --> 00:07:49.520 align:middle line:84% Or yeah, is the derivative of x inverse the same as negative xn 00:07:49.520 --> 00:07:50.020 align:middle line:90% minus 2? 00:07:50.020 --> 00:07:51.310 align:middle line:90% And the answer is no. 00:07:51.310 --> 00:07:54.940 align:middle line:84% So the first point I want to make 00:07:54.940 --> 00:07:57.640 align:middle line:90% is, matrix calculus is not-- 00:07:57.640 --> 00:08:01.310 align:middle line:84% it's more complicated than scalar and vector calculus. 00:08:01.310 --> 00:08:04.630 align:middle line:84% It's not a lot more complicated, but it's 00:08:04.630 --> 00:08:07.510 align:middle line:84% unfamiliar to everybody who hasn't studied it. 00:08:07.510 --> 00:08:10.400 align:middle line:84% That's what I want to say-- that you can't just say, 00:08:10.400 --> 00:08:13.000 align:middle line:84% oh, I know vector calculus, I know scalar calculus, 00:08:13.000 --> 00:08:15.580 align:middle line:84% it's probably just some simple generalization. 00:08:15.580 --> 00:08:18.217 align:middle line:84% So my first message to you is, no, it's not. 00:08:18.217 --> 00:08:19.300 align:middle line:90% I mean, you can master it. 00:08:19.300 --> 00:08:21.260 align:middle line:84% You'll learn it during this IEP course. 00:08:21.260 --> 00:08:27.500 align:middle line:84% But it's not just a simple, obvious generalization. 00:08:27.500 --> 00:08:28.000 align:middle line:90% OK. 00:08:28.000 --> 00:08:32.799 align:middle line:84% And so as far as applications go, this is a bit of a collage 00:08:32.799 --> 00:08:36.860 align:middle line:84% that Steven put together from a year or two ago. 00:08:36.860 --> 00:08:38.919 align:middle line:84% So you could just look at this yourself. 00:08:38.919 --> 00:08:41.409 align:middle line:84% But I think you all probably-- you're 00:08:41.409 --> 00:08:42.520 align:middle line:90% all here for some reason. 00:08:42.520 --> 00:08:44.500 align:middle line:84% So you probably already all know the buzzwords. 00:08:44.500 --> 00:08:46.990 align:middle line:84% But "machine learning," "parameter optimization," 00:08:46.990 --> 00:08:49.990 align:middle line:84% "stochastic gradient descent," "automatic differentiation," 00:08:49.990 --> 00:08:52.540 align:middle line:84% "backpropagation," right-- so these are all the reasons 00:08:52.540 --> 00:08:54.183 align:middle line:90% for doing matrix calculus. 00:08:54.183 --> 00:08:55.600 align:middle line:84% And this little collage here-- you 00:08:55.600 --> 00:08:59.950 align:middle line:84% could just see the places where things are happening 00:08:59.950 --> 00:09:02.040 align:middle line:90% in various parts of the web. 00:09:02.040 --> 00:09:02.540 align:middle line:90% OK? 00:09:02.540 --> 00:09:05.340 align:middle line:84% So you can take a quick look yourselves. 00:09:05.340 --> 00:09:10.160 align:middle line:84% The part that interests some of us 00:09:10.160 --> 00:09:13.880 align:middle line:84% are the applications to physical problems 00:09:13.880 --> 00:09:16.670 align:middle line:84% as well as machine learning and statistics. 00:09:16.670 --> 00:09:19.190 align:middle line:84% And so for example, in the upper left here, 00:09:19.190 --> 00:09:22.730 align:middle line:84% you have the so-called topology-optimized aircraft 00:09:22.730 --> 00:09:26.780 align:middle line:84% wing, where in the old days, when 00:09:26.780 --> 00:09:29.502 align:middle line:84% people used to engineer an airplane wing, what 00:09:29.502 --> 00:09:31.460 align:middle line:84% they would do is they would just pick a design. 00:09:31.460 --> 00:09:34.475 align:middle line:84% And then they would just somehow figure out the aerodynamics. 00:09:34.475 --> 00:09:37.100 align:middle line:84% And if they wanted to try again, they would take another design 00:09:37.100 --> 00:09:38.540 align:middle line:90% and figure out the aerodynamics. 00:09:38.540 --> 00:09:41.060 align:middle line:84% But now, with faster computers and machine 00:09:41.060 --> 00:09:43.490 align:middle line:84% learning techniques, you could have the computer 00:09:43.490 --> 00:09:45.900 align:middle line:90% decide what is the-- 00:09:45.900 --> 00:09:47.630 align:middle line:84% you can optimize for whatever you 00:09:47.630 --> 00:09:51.240 align:middle line:84% want-- minimize fuel, or minimize metal that you use, 00:09:51.240 --> 00:09:53.240 align:middle line:84% or whatever it is that you want to optimize for. 00:09:53.240 --> 00:09:55.590 align:middle line:84% And you can let the computer figure out the shape. 00:09:55.590 --> 00:09:58.070 align:middle line:84% And so that becomes a big optimization problem. 00:09:58.070 --> 00:10:00.920 align:middle line:84% And that gets called topology optimization. 00:10:00.920 --> 00:10:02.510 align:middle line:90% It happens for an airplane wing. 00:10:02.510 --> 00:10:06.450 align:middle line:90% It happens for fluid dynamics. 00:10:06.450 --> 00:10:08.310 align:middle line:84% So it doesn't really matter what you do. 00:10:08.310 --> 00:10:10.212 align:middle line:84% The point is that, in the old days, 00:10:10.212 --> 00:10:11.670 align:middle line:84% it was hard enough to just simulate 00:10:11.670 --> 00:10:14.400 align:middle line:84% the physics around anything, like an airplane wing. 00:10:14.400 --> 00:10:18.270 align:middle line:84% But now we can actually put that physics inside an inner loop. 00:10:18.270 --> 00:10:21.240 align:middle line:84% And we can put an optimization problem around it, right? 00:10:21.240 --> 00:10:24.630 align:middle line:90% And that's happening everywhere. 00:10:24.630 --> 00:10:28.470 align:middle line:84% So data science and multivariate statistics, of course, 00:10:28.470 --> 00:10:32.030 align:middle line:84% is another big area, especially the computer scientists. 00:10:32.030 --> 00:10:34.950 align:middle line:84% They're doing this sort of stuff all over the place nowadays. 00:10:34.950 --> 00:10:35.450 align:middle line:90% OK. 00:10:35.450 --> 00:10:37.490 align:middle line:84% So there is-- you can take a look at this video. 00:10:37.490 --> 00:10:40.700 align:middle line:90% Or here's a book. 00:10:40.700 --> 00:10:42.990 align:middle line:90% And let me say some roles-- 00:10:42.990 --> 00:10:45.590 align:middle line:84% let me say a few words about the role of autodiff, 00:10:45.590 --> 00:10:47.550 align:middle line:90% or automatic differentiation. 00:10:47.550 --> 00:10:52.110 align:middle line:84% So automatic differentiation is a very exciting technology. 00:10:52.110 --> 00:10:56.693 align:middle line:84% And one of the things that I like to say-- 00:10:56.693 --> 00:10:58.110 align:middle line:84% I mean, you can look at the slide. 00:10:58.110 --> 00:11:02.540 align:middle line:84% But when I first learned about automatic differentiation, 00:11:02.540 --> 00:11:06.390 align:middle line:84% I was rather surprised as to what it is and what it isn't. 00:11:06.390 --> 00:11:10.400 align:middle line:84% So how many of you are familiar with autodiff already a little 00:11:10.400 --> 00:11:11.340 align:middle line:90% bit? 00:11:11.340 --> 00:11:11.840 align:middle line:90% OK. 00:11:11.840 --> 00:11:13.940 align:middle line:84% So about maybe five or six people 00:11:13.940 --> 00:11:16.370 align:middle line:84% have raised their hands-- a minority of the course, 00:11:16.370 --> 00:11:17.030 align:middle line:90% for sure. 00:11:17.030 --> 00:11:23.120 align:middle line:84% So everybody at MIT is good at differentiating, right? 00:11:23.120 --> 00:11:25.983 align:middle line:84% You didn't get into MIT if you can't differentiate functions. 00:11:25.983 --> 00:11:26.900 align:middle line:90% You're all good at it. 00:11:26.900 --> 00:11:28.670 align:middle line:90% I have no doubt, right, that-- 00:11:28.670 --> 00:11:30.170 align:middle line:84% certainly if you came to this class. 00:11:30.170 --> 00:11:33.170 align:middle line:84% But everybody at MIT can differentiate everything. 00:11:33.170 --> 00:11:35.610 align:middle line:84% I don't care if I write x-squared, square root, sine, 00:11:35.610 --> 00:11:36.682 align:middle line:90% cosine-- 00:11:36.682 --> 00:11:38.390 align:middle line:84% arctan maybe you have to remind yourself. 00:11:38.390 --> 00:11:40.800 align:middle line:84% But yeah, you can differentiate everything. 00:11:40.800 --> 00:11:45.090 align:middle line:84% And nowadays, automatic differentiation 00:11:45.090 --> 00:11:47.670 align:middle line:84% has become almost more of a compiler technology 00:11:47.670 --> 00:11:49.230 align:middle line:90% than a math course, right? 00:11:49.230 --> 00:11:52.510 align:middle line:84% It's kind of become this new thing. 00:11:52.510 --> 00:11:54.570 align:middle line:84% It's not numerical differentiation, 00:11:54.570 --> 00:11:56.460 align:middle line:84% like if you take a numerical analysis course. 00:11:56.460 --> 00:11:58.127 align:middle line:84% Or maybe you see a little bit at the end 00:11:58.127 --> 00:12:00.510 align:middle line:84% of your calculus course, where you learn to take 00:12:00.510 --> 00:12:02.848 align:middle line:90% a delta y over a delta x. 00:12:02.848 --> 00:12:03.390 align:middle line:90% You remember? 00:12:03.390 --> 00:12:07.030 align:middle line:84% You take a small difference over a small change. 00:12:07.030 --> 00:12:08.250 align:middle line:90% So it's not that. 00:12:08.250 --> 00:12:11.280 align:middle line:84% And it's not-- like, some of you may have used Wolfram Alpha 00:12:11.280 --> 00:12:14.340 align:middle line:84% or Mathematica, where you let the computer differentiate 00:12:14.340 --> 00:12:15.090 align:middle line:90% symbolically. 00:12:15.090 --> 00:12:16.330 align:middle line:90% And it's not that either. 00:12:16.330 --> 00:12:18.997 align:middle line:84% And I think that's what makes it interesting-- that it's neither 00:12:18.997 --> 00:12:20.760 align:middle line:90% of those two things, right? 00:12:20.760 --> 00:12:22.380 align:middle line:90% It's something different. 00:12:22.380 --> 00:12:26.670 align:middle line:84% And we'll show you a little bit about how it's done. 00:12:26.670 --> 00:12:29.550 align:middle line:84% And what you learn is that you should never-- well, 00:12:29.550 --> 00:12:32.940 align:middle line:90% we teach in 18.01 to do things-- 00:12:32.940 --> 00:12:35.640 align:middle line:84% to me, it's almost like what's happened with long division. 00:12:35.640 --> 00:12:37.470 align:middle line:84% Like, maybe-- how many of you actually 00:12:37.470 --> 00:12:39.360 align:middle line:90% learned to do long division? 00:12:39.360 --> 00:12:42.540 align:middle line:84% How many of you think you could still do it today 00:12:42.540 --> 00:12:44.400 align:middle line:90% if forced to, right? 00:12:44.400 --> 00:12:46.560 align:middle line:84% So to tell Steven-- the whole class 00:12:46.560 --> 00:12:49.260 align:middle line:84% raised their hand for the first one and the minority 00:12:49.260 --> 00:12:51.408 align:middle line:90% for the second question. 00:12:51.408 --> 00:12:52.950 align:middle line:84% But of course, one of the good things 00:12:52.950 --> 00:12:54.510 align:middle line:84% is you kind of understand division 00:12:54.510 --> 00:12:56.500 align:middle line:90% maybe by learning long division. 00:12:56.500 --> 00:12:57.270 align:middle line:90% I'm not even sure. 00:12:57.270 --> 00:12:59.700 align:middle line:84% But calculus is, I think, becoming the same way, 00:12:59.700 --> 00:13:03.428 align:middle line:84% where we teach it the old-fashioned way. 00:13:03.428 --> 00:13:05.470 align:middle line:84% And I guess that's good because you can learn it. 00:13:05.470 --> 00:13:07.350 align:middle line:90% You understand it. 00:13:07.350 --> 00:13:10.662 align:middle line:90% But as to what should-- 00:13:10.662 --> 00:13:12.870 align:middle line:84% we don't really think humans should do long division. 00:13:12.870 --> 00:13:15.870 align:middle line:84% And it's getting to the point where probably-- 00:13:15.870 --> 00:13:18.360 align:middle line:84% this may be heresy here in Building 2 00:13:18.360 --> 00:13:21.150 align:middle line:84% in the math-- this is home of mathematics at MIT. 00:13:21.150 --> 00:13:24.600 align:middle line:84% But I would say that taking derivatives may or may not 00:13:24.600 --> 00:13:26.910 align:middle line:84% be as important-- certainly complicated 00:13:26.910 --> 00:13:28.530 align:middle line:84% derivatives are beyond human ability 00:13:28.530 --> 00:13:31.473 align:middle line:84% anyway, no matter how good you are at it. 00:13:31.473 --> 00:13:33.390 align:middle line:84% The things we want to differentiate these days 00:13:33.390 --> 00:13:34.990 align:middle line:90% are just too complicated. 00:13:34.990 --> 00:13:35.490 align:middle line:90% OK. 00:13:35.490 --> 00:13:38.040 align:middle line:90% So what did I put over here? 00:13:38.040 --> 00:13:38.540 align:middle line:90% Let's see. 00:13:38.540 --> 00:13:40.150 align:middle line:90% So yeah. 00:13:40.150 --> 00:13:42.970 align:middle line:84% So today's courses are mostly symbolic. 00:13:42.970 --> 00:13:46.330 align:middle line:84% I think that's fair to say-- that 18.01 is probably 00:13:46.330 --> 00:13:48.055 align:middle line:90% a 90% symbolic course. 00:13:48.055 --> 00:13:51.582 align:middle line:90% 00:13:51.582 --> 00:13:53.540 align:middle line:84% And I'm saying, yeah, today's differentiation-- 00:13:53.540 --> 00:13:55.260 align:middle line:90% neither of these two things. 00:13:55.260 --> 00:13:56.150 align:middle line:90% The math is fun. 00:13:56.150 --> 00:13:58.020 align:middle line:84% We'll learn about it in this class. 00:13:58.020 --> 00:13:58.520 align:middle line:90% OK? 00:13:58.520 --> 00:13:58.820 align:middle line:90% So-- 00:13:58.820 --> 00:14:00.200 align:middle line:84% STEVEN JOHNSON: And it's not just fun. 00:14:00.200 --> 00:14:01.492 align:middle line:90% ALAN EDELMAN: Wait, wait, wait. 00:14:01.492 --> 00:14:02.900 align:middle line:84% Let me put the microphone on you. 00:14:02.900 --> 00:14:04.460 align:middle line:84% STEVEN JOHNSON: Even if you're using the computer 00:14:04.460 --> 00:14:07.070 align:middle line:84% to take derivatives, right-- so you write a program. 00:14:07.070 --> 00:14:10.310 align:middle line:84% You let it take the derivative for you. 00:14:10.310 --> 00:14:12.080 align:middle line:84% To use that effectively, you really 00:14:12.080 --> 00:14:16.298 align:middle line:84% have to have some idea of what's going on under the hood 00:14:16.298 --> 00:14:17.090 align:middle line:90% because there are-- 00:14:17.090 --> 00:14:19.507 align:middle line:84% and especially there are cases where it doesn't work well. 00:14:19.507 --> 00:14:20.780 align:middle line:90% You need to know about those. 00:14:20.780 --> 00:14:23.030 align:middle line:84% And you need to know something about the technology 00:14:23.030 --> 00:14:24.860 align:middle line:84% and about what its capabilities are-- 00:14:24.860 --> 00:14:27.890 align:middle line:84% to know when to use forward mode or reverse mode 00:14:27.890 --> 00:14:30.500 align:middle line:84% differentiation, and what a vector Jacobian product is 00:14:30.500 --> 00:14:32.390 align:middle line:90% and why it matters-- 00:14:32.390 --> 00:14:36.223 align:middle line:84% in order to really be an effective user 00:14:36.223 --> 00:14:37.265 align:middle line:90% of these kinds of things. 00:14:37.265 --> 00:14:39.477 align:middle line:84% ALAN EDELMAN: Yes, yes Steven's getting fancy already 00:14:39.477 --> 00:14:41.060 align:middle line:84% a little bit faster than I would have. 00:14:41.060 --> 00:14:42.598 align:middle line:90% But he's quite right that-- 00:14:42.598 --> 00:14:46.646 align:middle line:90% 00:14:46.646 --> 00:14:48.350 align:middle line:84% I mean, I would actually tell people 00:14:48.350 --> 00:14:51.455 align:middle line:84% that it's not such a bad idea to know how an engine works 00:14:51.455 --> 00:14:52.830 align:middle line:84% if you want to drive a car, but I 00:14:52.830 --> 00:14:54.450 align:middle line:90% guess you don't really have to. 00:14:54.450 --> 00:14:57.280 align:middle line:84% And these days, cars don't even have engines anymore, 00:14:57.280 --> 00:14:57.870 align:middle line:90% many of them. 00:14:57.870 --> 00:15:03.897 align:middle line:84% But so what Steven's saying is that automatic differentiation 00:15:03.897 --> 00:15:06.480 align:middle line:84% is probably not quite as easy as driving a car-- that actually 00:15:06.480 --> 00:15:08.850 align:middle line:84% understanding the technology underneath it 00:15:08.850 --> 00:15:10.420 align:middle line:90% will really help you to use it. 00:15:10.420 --> 00:15:14.170 align:middle line:90% And sometimes it's necessary. 00:15:14.170 --> 00:15:14.950 align:middle line:90% OK. 00:15:14.950 --> 00:15:20.140 align:middle line:84% So let me start with something you all know just 00:15:20.140 --> 00:15:22.070 align:middle line:90% to establish some notation. 00:15:22.070 --> 00:15:22.570 align:middle line:90% OK? 00:15:22.570 --> 00:15:25.112 align:middle line:84% And then I think Steven in the second part of today's lecture 00:15:25.112 --> 00:15:28.150 align:middle line:84% will kind of reiterate and talk a little bit more 00:15:28.150 --> 00:15:31.540 align:middle line:84% about the notation that we'll be using. 00:15:31.540 --> 00:15:37.480 align:middle line:84% But I want to emphasize the concept of linearization. 00:15:37.480 --> 00:15:38.950 align:middle line:84% I just want to get that word out. 00:15:38.950 --> 00:15:40.630 align:middle line:84% And you'll see more of that pretty soon. 00:15:40.630 --> 00:15:43.870 align:middle line:84% But instead, the idea that a derivative 00:15:43.870 --> 00:15:47.950 align:middle line:84% is taking a nonlinear function and pretending at least 00:15:47.950 --> 00:15:50.420 align:middle line:84% locally that it's a linear function-- 00:15:50.420 --> 00:15:52.610 align:middle line:84% I want to emphasize that point of view. 00:15:52.610 --> 00:15:54.080 align:middle line:84% And of course, you all know that. 00:15:54.080 --> 00:15:57.640 align:middle line:84% So looking over here at this expression, 00:15:57.640 --> 00:16:02.200 align:middle line:84% you all know that, in some sense, 00:16:02.200 --> 00:16:04.390 align:middle line:90% if you're sitting at a point-- 00:16:04.390 --> 00:16:08.500 align:middle line:84% x0, y0-- then the linear function that's 00:16:08.500 --> 00:16:11.650 align:middle line:84% tangent to the curve can be written 00:16:11.650 --> 00:16:15.650 align:middle line:84% as y minus y0 is-- well, the linear function 00:16:15.650 --> 00:16:18.290 align:middle line:84% is equal to f prime of x0 x minus x0, right? 00:16:18.290 --> 00:16:20.970 align:middle line:84% So if I put this down in the denominator, 00:16:20.970 --> 00:16:23.420 align:middle line:84% it's just saying the change in y over the change of x 00:16:23.420 --> 00:16:25.980 align:middle line:90% is the derivative, right? 00:16:25.980 --> 00:16:30.260 align:middle line:84% So in some sense, why we take derivatives-- 00:16:30.260 --> 00:16:32.150 align:middle line:84% this is all very trivial in one-dimension. 00:16:32.150 --> 00:16:34.708 align:middle line:84% But it's good to kind of keep track of this point of view 00:16:34.708 --> 00:16:36.500 align:middle line:84% so that, when we go into higher dimensions, 00:16:36.500 --> 00:16:38.900 align:middle line:84% you kind of remember that calculus is really 00:16:38.900 --> 00:16:42.470 align:middle line:84% about pretending some complicated curved surface is 00:16:42.470 --> 00:16:44.840 align:middle line:90% just locally linear. 00:16:44.840 --> 00:16:47.840 align:middle line:84% And for one variable, that's just a tangent line. 00:16:47.840 --> 00:16:50.300 align:middle line:84% For multivariables, you have planes, hyperplanes, 00:16:50.300 --> 00:16:51.030 align:middle line:90% and so forth. 00:16:51.030 --> 00:16:52.220 align:middle line:90% But they're flat. 00:16:52.220 --> 00:16:54.450 align:middle line:90% That's what a linearization is. 00:16:54.450 --> 00:16:57.676 align:middle line:90% So here are some notations. 00:16:57.676 --> 00:17:01.010 align:middle line:84% We'll use delta to indicate finite perturbations. 00:17:01.010 --> 00:17:04.109 align:middle line:84% And so I have this approximate symbol over here. 00:17:04.109 --> 00:17:04.609 align:middle line:90% OK? 00:17:04.609 --> 00:17:08.359 align:middle line:84% So delta y is approximately f prime of x delta x. 00:17:08.359 --> 00:17:10.790 align:middle line:84% Or you can take the infinitesimal limit. 00:17:10.790 --> 00:17:14.869 align:middle line:84% And many of you are familiar with dy over dx 00:17:14.869 --> 00:17:16.615 align:middle line:90% is f prime of x. 00:17:16.615 --> 00:17:17.990 align:middle line:84% But how many of you have actually 00:17:17.990 --> 00:17:20.675 align:middle line:84% seen it written this way, where-- 00:17:20.675 --> 00:17:22.550 align:middle line:84% or we're told, you're not allowed to do that, 00:17:22.550 --> 00:17:23.390 align:middle line:90% where you go-- 00:17:23.390 --> 00:17:23.890 align:middle line:90% right? 00:17:23.890 --> 00:17:30.430 align:middle line:84% So everybody knows this notation. 00:17:30.430 --> 00:17:34.340 align:middle line:90% 00:17:34.340 --> 00:17:38.160 align:middle line:84% And maybe they told you that this is really not a division. 00:17:38.160 --> 00:17:40.160 align:middle line:84% Or maybe they told you it is kind of a division. 00:17:40.160 --> 00:17:42.290 align:middle line:84% I don't know what your calculus teacher told you. 00:17:42.290 --> 00:17:44.332 align:middle line:84% But has anyone ever-- have you seen it like this? 00:17:44.332 --> 00:17:45.957 align:middle line:84% How many of you have seen it like this, 00:17:45.957 --> 00:17:47.310 align:middle line:90% where you're allowed to-- 00:17:47.310 --> 00:17:47.810 align:middle line:90% OK. 00:17:47.810 --> 00:17:48.860 align:middle line:90% So almost everybody. 00:17:48.860 --> 00:17:49.680 align:middle line:90% Good. 00:17:49.680 --> 00:17:50.180 align:middle line:90% OK. 00:17:50.180 --> 00:17:52.850 align:middle line:84% So we'll talk a little bit about what that really means. 00:17:52.850 --> 00:17:57.980 align:middle line:84% Here's the linearization that's-- instead of just using 00:17:57.980 --> 00:18:00.410 align:middle line:90% y's, I'm using x over here. 00:18:00.410 --> 00:18:05.750 align:middle line:84% And this one is my favorite, where you write, 00:18:05.750 --> 00:18:08.930 align:middle line:84% if y is equal to f of x, right, so then-- 00:18:08.930 --> 00:18:10.430 align:middle line:84% in fact, there's no real y at all. 00:18:10.430 --> 00:18:12.380 align:middle line:84% If you just have a function of x, 00:18:12.380 --> 00:18:16.280 align:middle line:84% that df is equal to f prime of x dx. 00:18:16.280 --> 00:18:17.840 align:middle line:90% And so we have this-- 00:18:17.840 --> 00:18:20.720 align:middle line:90% 00:18:20.720 --> 00:18:22.850 align:middle line:84% f prime is a linear function in the end. 00:18:22.850 --> 00:18:25.910 align:middle line:84% I mean, in one variable, it's just a constant, right? 00:18:25.910 --> 00:18:31.200 align:middle line:84% But a constant times the change of x is the change of f. 00:18:31.200 --> 00:18:31.700 align:middle line:90% OK? 00:18:31.700 --> 00:18:32.980 align:middle line:90% So you all know this. 00:18:32.980 --> 00:18:36.520 align:middle line:84% It's one-dimensional calculus-- but just to set the stage. 00:18:36.520 --> 00:18:37.020 align:middle line:90% OK. 00:18:37.020 --> 00:18:37.560 align:middle line:90% And-- 00:18:37.560 --> 00:18:39.268 align:middle line:84% STEVEN JOHNSON: Hey, can I say something? 00:18:39.268 --> 00:18:44.250 align:middle line:84% So the reason we don't want to put dx on the other side 00:18:44.250 --> 00:18:47.310 align:middle line:84% is we don't want to divide by dx. 00:18:47.310 --> 00:18:50.370 align:middle line:84% One way of thinking about it is that pretty soon this-- right 00:18:50.370 --> 00:18:51.180 align:middle line:90% now, it's a scalar. 00:18:51.180 --> 00:18:52.222 align:middle line:90% We can divide by numbers. 00:18:52.222 --> 00:18:54.570 align:middle line:84% But pretty soon, x could be a vector. 00:18:54.570 --> 00:18:56.010 align:middle line:90% Or it could be some other thing. 00:18:56.010 --> 00:18:58.210 align:middle line:90% You can't divide by a vector. 00:18:58.210 --> 00:19:01.320 align:middle line:84% So it's harder to-- it's easier to generalize 00:19:01.320 --> 00:19:04.365 align:middle line:84% this kind of notation to other kinds of objects 00:19:04.365 --> 00:19:06.600 align:middle line:84% to where you can multiply, you can operate, 00:19:06.600 --> 00:19:08.370 align:middle line:90% but you can't divide. 00:19:08.370 --> 00:19:10.000 align:middle line:90% ALAN EDELMAN: Yep. 00:19:10.000 --> 00:19:11.710 align:middle line:90% Good. 00:19:11.710 --> 00:19:13.600 align:middle line:90% Good. 00:19:13.600 --> 00:19:14.170 align:middle line:90% Yeah. 00:19:14.170 --> 00:19:16.790 align:middle line:84% Try dividing a vector by a vector. 00:19:16.790 --> 00:19:17.965 align:middle line:90% It doesn't really mean much. 00:19:17.965 --> 00:19:21.980 align:middle line:90% 00:19:21.980 --> 00:19:22.480 align:middle line:90% All right. 00:19:22.480 --> 00:19:25.870 align:middle line:84% So again, more trivialities just to set the stage, 00:19:25.870 --> 00:19:28.190 align:middle line:90% just to go baby steps. 00:19:28.190 --> 00:19:32.810 align:middle line:84% So here, what I'm doing is I'm looking at the square function. 00:19:32.810 --> 00:19:33.310 align:middle line:90% OK? 00:19:33.310 --> 00:19:35.602 align:middle line:84% And I'm looking at it at a particular point-- the point 00:19:35.602 --> 00:19:37.210 align:middle line:90% 3, 9. 00:19:37.210 --> 00:19:39.490 align:middle line:84% And you all know that the derivative is 2x. 00:19:39.490 --> 00:19:44.920 align:middle line:84% And so if x equals 3, 2x is equal to 6, right? 00:19:44.920 --> 00:19:47.420 align:middle line:90% So that's the derivative. 00:19:47.420 --> 00:19:48.970 align:middle line:84% The square of 3, of course, is 9. 00:19:48.970 --> 00:19:52.760 align:middle line:84% And even if you do this on a computer, this is easy to do. 00:19:52.760 --> 00:19:54.580 align:middle line:84% You don't even need a computer, really. 00:19:54.580 --> 00:19:57.920 align:middle line:84% It's actually easy to check this even on paper and pencil. 00:19:57.920 --> 00:20:00.280 align:middle line:84% But if you get closer and closer to 3, 00:20:00.280 --> 00:20:03.880 align:middle line:84% as I'm doing in these four lines over here, of course 00:20:03.880 --> 00:20:05.980 align:middle line:90% you get closer and closer to 9. 00:20:05.980 --> 00:20:09.140 align:middle line:84% And what we're interested in is the difference. 00:20:09.140 --> 00:20:15.160 align:middle line:84% So you see that if I add a little delta x to 3, 00:20:15.160 --> 00:20:20.080 align:middle line:84% I get a 9 plus delta y, which is 9 plus 6 delta x 00:20:20.080 --> 00:20:21.980 align:middle line:84% plus some higher-order term, which 00:20:21.980 --> 00:20:23.600 align:middle line:90% we don't care about, right? 00:20:23.600 --> 00:20:29.810 align:middle line:84% And so we see this as delta y is f prime of x0 delta x, right? 00:20:29.810 --> 00:20:31.880 align:middle line:90% delta y is 6 delta x. 00:20:31.880 --> 00:20:33.890 align:middle line:84% So the thing I want you to walk away with 00:20:33.890 --> 00:20:38.697 align:middle line:84% is really that I want you to look at these numbers, 00:20:38.697 --> 00:20:39.530 align:middle line:90% like this one here-- 00:20:39.530 --> 00:20:42.950 align:middle line:84% I think of this as the delta x and think of this red part 00:20:42.950 --> 00:20:45.360 align:middle line:84% here as the sixth delta x, right? 00:20:45.360 --> 00:20:50.780 align:middle line:84% So again, this idea that Steven also just said-- 00:20:50.780 --> 00:20:52.650 align:middle line:84% that this is a linear functional. 00:20:52.650 --> 00:20:55.340 align:middle line:84% This is the function that multiplies by 6. 00:20:55.340 --> 00:20:59.240 align:middle line:84% If I want to know the change in y, I have that change in x. 00:20:59.240 --> 00:21:00.590 align:middle line:90% And I multiply by 6. 00:21:00.590 --> 00:21:02.750 align:middle line:84% That's my-- you see, eventually this is going 00:21:02.750 --> 00:21:04.042 align:middle line:90% to be not just multiplied by 6. 00:21:04.042 --> 00:21:06.000 align:middle line:84% But it's going to be, you have a vector change, 00:21:06.000 --> 00:21:07.730 align:middle line:84% and you're going to multiply by a matrix. 00:21:07.730 --> 00:21:08.300 align:middle line:90% OK? 00:21:08.300 --> 00:21:10.340 align:middle line:84% But just going slowly, we make a little change. 00:21:10.340 --> 00:21:11.600 align:middle line:90% And you multiply by 6. 00:21:11.600 --> 00:21:15.510 align:middle line:90% And that's what's going on here. 00:21:15.510 --> 00:21:19.880 align:middle line:84% So I like to think of dx and dy as really small numbers 00:21:19.880 --> 00:21:20.660 align:middle line:90% on a computer. 00:21:20.660 --> 00:21:24.750 align:middle line:84% The mathematicians call them infinitesimals. 00:21:24.750 --> 00:21:29.690 align:middle line:84% From-- there's a lot of rigorous math that makes infinitesimals 00:21:29.690 --> 00:21:32.600 align:middle line:84% work, which I think is sort of a great human achievement 00:21:32.600 --> 00:21:36.320 align:middle line:84% but of little value to practical computations. 00:21:36.320 --> 00:21:39.845 align:middle line:84% I think for practical computations, to think of-- 00:21:39.845 --> 00:21:41.720 align:middle line:84% you could think of it any way you like, but I 00:21:41.720 --> 00:21:43.670 align:middle line:90% like to think of dx and dy-- 00:21:43.670 --> 00:21:47.570 align:middle line:84% when I'm actually working and I'm not thinking theoretically, 00:21:47.570 --> 00:21:50.690 align:middle line:84% I like to think of dx and dy as, like, the limit 00:21:50.690 --> 00:21:51.770 align:middle line:90% of very small numbers. 00:21:51.770 --> 00:21:58.130 align:middle line:84% And when I play on the computer, I type 0.0001 or 0.00001. 00:21:58.130 --> 00:22:00.920 align:middle line:90% And I say, that's small enough. 00:22:00.920 --> 00:22:02.330 align:middle line:84% It depends on context, of course. 00:22:02.330 --> 00:22:05.950 align:middle line:90% But that's usually what I do. 00:22:05.950 --> 00:22:06.450 align:middle line:90% OK? 00:22:06.450 --> 00:22:07.680 align:middle line:90% So all right. 00:22:07.680 --> 00:22:11.680 align:middle line:84% Now we get to go to where the big boys and big girls go, 00:22:11.680 --> 00:22:12.180 align:middle line:90% right? 00:22:12.180 --> 00:22:15.330 align:middle line:84% So we're leaving the world of scalar calculus 00:22:15.330 --> 00:22:20.380 align:middle line:84% and entering this new world of matrix calculus. 00:22:20.380 --> 00:22:23.280 align:middle line:84% So to get started, let me just mention 00:22:23.280 --> 00:22:24.700 align:middle line:90% a little bit of notation. 00:22:24.700 --> 00:22:28.990 align:middle line:84% So it's handy to have the element-wise-- 00:22:28.990 --> 00:22:32.650 align:middle line:84% so yeah, just a little bit of vector and matrix notation. 00:22:32.650 --> 00:22:36.000 align:middle line:84% So I like to use the Julia notation 00:22:36.000 --> 00:22:38.800 align:middle line:84% for element-wise product of vectors. 00:22:38.800 --> 00:22:43.860 align:middle line:90% So this dot times this point-- 00:22:43.860 --> 00:22:46.440 align:middle line:84% I like to call it point-wise times, right? 00:22:46.440 --> 00:22:50.880 align:middle line:84% So 2, 3 point-wise times 10, 11 is 20, 33. 00:22:50.880 --> 00:22:52.967 align:middle line:90% If you happen to use languages-- 00:22:52.967 --> 00:22:55.050 align:middle line:84% I think Python does it, too-- and certainly Julia. 00:22:55.050 --> 00:22:57.520 align:middle line:84% They pronounce it as "broadcasting." 00:22:57.520 --> 00:23:01.170 align:middle line:84% I always hated that word because I don't feel it really 00:23:01.170 --> 00:23:02.340 align:middle line:90% indicates what's going on. 00:23:02.340 --> 00:23:04.560 align:middle line:90% I think point-wise multiply-- 00:23:04.560 --> 00:23:06.450 align:middle line:84% to me, a dot is like a decimal point. 00:23:06.450 --> 00:23:07.210 align:middle line:90% It says it better. 00:23:07.210 --> 00:23:09.550 align:middle line:84% But if you're used to the term "broadcasting," 00:23:09.550 --> 00:23:11.210 align:middle line:90% that's fine as well. 00:23:11.210 --> 00:23:14.200 align:middle line:84% So you see we're doing element-wise multiplication. 00:23:14.200 --> 00:23:18.520 align:middle line:84% And in this little demo here, you'll 00:23:18.520 --> 00:23:22.210 align:middle line:84% also see some people use this dot with a circle around it 00:23:22.210 --> 00:23:24.430 align:middle line:90% to indicate point-wise multiply. 00:23:24.430 --> 00:23:27.610 align:middle line:84% I remind you, in case linear algebra was too long ago, 00:23:27.610 --> 00:23:29.230 align:middle line:84% the notion of a trace of a matrix 00:23:29.230 --> 00:23:33.250 align:middle line:84% is, if you have a big square matrix, 00:23:33.250 --> 00:23:36.280 align:middle line:84% then you look at the diagonal, and you add up all the numbers. 00:23:36.280 --> 00:23:39.080 align:middle line:90% And that's the trace. 00:23:39.080 --> 00:23:44.710 align:middle line:84% And if you go to this matrix.calculus.org, 00:23:44.710 --> 00:23:49.385 align:middle line:90% there was-- oh, yes. 00:23:49.385 --> 00:23:50.260 align:middle line:90% There's a story here. 00:23:50.260 --> 00:23:52.360 align:middle line:84% Let me step back and tell you the story. 00:23:52.360 --> 00:23:55.780 align:middle line:90% So the reason why we started-- 00:23:55.780 --> 00:23:59.230 align:middle line:84% one of the reasons we started teaching this class during IEP 00:23:59.230 --> 00:24:03.700 align:middle line:84% was because there was a question on the math Piazza-- 00:24:03.700 --> 00:24:08.980 align:middle line:84% all the undergraduates have access to a Piazza page, 00:24:08.980 --> 00:24:11.830 align:middle line:84% not for an individual class, but for the majors. 00:24:11.830 --> 00:24:14.350 align:middle line:84% And there was this question that arose, 00:24:14.350 --> 00:24:18.250 align:middle line:84% which is basically this question. 00:24:18.250 --> 00:24:21.640 align:middle line:84% If y is an n-by-m matrix-- here, I'll 00:24:21.640 --> 00:24:26.060 align:middle line:84% use the mouse so it'll get recorded properly. 00:24:26.060 --> 00:24:31.000 align:middle line:84% So if y is an n-by-m matrix, x is an n-by-k matrix, 00:24:31.000 --> 00:24:34.330 align:middle line:84% and theta is also a matrix-- a k-by-m matrix-- 00:24:34.330 --> 00:24:38.920 align:middle line:84% then this scalar, the trace, makes sense, right? 00:24:38.920 --> 00:24:41.670 align:middle line:84% The trace of this multiply makes sense. 00:24:41.670 --> 00:24:44.300 align:middle line:84% And one could try to understand what does 00:24:44.300 --> 00:24:46.290 align:middle line:90% it mean to take the derivative. 00:24:46.290 --> 00:24:46.790 align:middle line:90% OK? 00:24:46.790 --> 00:24:51.830 align:middle line:84% And this student on Piazza asked, how do you do it? 00:24:51.830 --> 00:24:53.910 align:middle line:84% And how do you learn how to do things like this? 00:24:53.910 --> 00:24:56.150 align:middle line:84% And that really is the origin of this IEP class-- 00:24:56.150 --> 00:24:58.400 align:middle line:90% that, here's the answer. 00:24:58.400 --> 00:25:00.710 align:middle line:90% The answer is itself a matrix. 00:25:00.710 --> 00:25:04.100 align:middle line:84% It's minus 2x transpose times y minus x theta. 00:25:04.100 --> 00:25:06.080 align:middle line:84% And you can actually get the answer 00:25:06.080 --> 00:25:09.217 align:middle line:84% through this matrixcalculus.org, right? 00:25:09.217 --> 00:25:11.300 align:middle line:84% So those of you who want to, you could look at it. 00:25:11.300 --> 00:25:13.280 align:middle line:90% It's kind of a nice web page. 00:25:13.280 --> 00:25:14.930 align:middle line:90% You can type in some matrices. 00:25:14.930 --> 00:25:18.050 align:middle line:84% It's got its limitations, but you can look at some of these 00:25:18.050 --> 00:25:19.500 align:middle line:90% and see what's going on. 00:25:19.500 --> 00:25:20.000 align:middle line:90% OK. 00:25:20.000 --> 00:25:20.750 align:middle line:90% But let me go back. 00:25:20.750 --> 00:25:21.890 align:middle line:90% How do I get back to my slides? 00:25:21.890 --> 00:25:22.880 align:middle line:90% I'll never find them again. 00:25:22.880 --> 00:25:23.840 align:middle line:90% Where are my slides? 00:25:23.840 --> 00:25:25.790 align:middle line:84% They're underneath here, aren't they? 00:25:25.790 --> 00:25:27.748 align:middle line:84% STEVEN JOHNSON: Ctrl-left arrow or right arrow? 00:25:27.748 --> 00:25:29.630 align:middle line:84% ALAN EDELMAN: No, I think they're underneath. 00:25:29.630 --> 00:25:31.940 align:middle line:90% Yeah, they're underneath. 00:25:31.940 --> 00:25:34.740 align:middle line:84% Oh, you think I could have gotten it that way? 00:25:34.740 --> 00:25:36.740 align:middle line:84% STEVEN JOHNSON: Full screen mode, I think, yeah. 00:25:36.740 --> 00:25:37.430 align:middle line:90% ALAN EDELMAN: Oh, OK. 00:25:37.430 --> 00:25:37.930 align:middle line:90% Let's see. 00:25:37.930 --> 00:25:39.260 align:middle line:90% Can I really get to that? 00:25:39.260 --> 00:25:41.990 align:middle line:90% Ctrl-- never mind. 00:25:41.990 --> 00:25:43.400 align:middle line:90% I don't know how I can get back. 00:25:43.400 --> 00:25:44.793 align:middle line:90% Anyway, yeah. 00:25:44.793 --> 00:25:46.460 align:middle line:84% So you might play with this a little bit 00:25:46.460 --> 00:25:48.043 align:middle line:84% and see some of the things you can do. 00:25:48.043 --> 00:25:49.260 align:middle line:90% It has limitations. 00:25:49.260 --> 00:25:51.140 align:middle line:84% For example, it doesn't do a good job 00:25:51.140 --> 00:25:55.070 align:middle line:84% when the answer is higher than two-dimensional, right? 00:25:55.070 --> 00:25:58.130 align:middle line:84% But you can take a look and see what it will do. 00:25:58.130 --> 00:26:01.760 align:middle line:84% One of the things-- you might see some matrix 00:26:01.760 --> 00:26:03.250 align:middle line:90% calculus in some classes. 00:26:03.250 --> 00:26:05.250 align:middle line:84% For example-- I think this is on the next slide, 00:26:05.250 --> 00:26:07.490 align:middle line:84% but I'll just put it on the blackboard. 00:26:07.490 --> 00:26:10.070 align:middle line:84% For example, you might ask for, how 00:26:10.070 --> 00:26:14.060 align:middle line:84% do you take the gradient of x transpose x, 00:26:14.060 --> 00:26:16.245 align:middle line:84% and things like that-- or x transpose ax. 00:26:16.245 --> 00:26:18.560 align:middle line:84% I think it's coming up on the next slide, anyway. 00:26:18.560 --> 00:26:20.393 align:middle line:84% So how do you take the gradient with respect 00:26:20.393 --> 00:26:22.100 align:middle line:90% to x of objects like these? 00:26:22.100 --> 00:26:25.110 align:middle line:84% And I've seen many classes at MIT do that. 00:26:25.110 --> 00:26:26.990 align:middle line:84% And what I would say is they would do it 00:26:26.990 --> 00:26:28.550 align:middle line:90% the old-fashioned way. 00:26:28.550 --> 00:26:30.860 align:middle line:84% They do it variable by variable by variable 00:26:30.860 --> 00:26:33.060 align:middle line:90% as opposed to the holistic way. 00:26:33.060 --> 00:26:35.870 align:middle line:84% So one of the things that I'd like to kind of invite 00:26:35.870 --> 00:26:38.430 align:middle line:84% all of you to think about as we go through this course 00:26:38.430 --> 00:26:40.950 align:middle line:84% is to think of a matrix holistically 00:26:40.950 --> 00:26:42.690 align:middle line:84% or think of a vector holistically. 00:26:42.690 --> 00:26:45.870 align:middle line:84% Stop thinking of a vector as a bunch of elements or a matrix 00:26:45.870 --> 00:26:49.950 align:middle line:84% as an m-by-n table, as you would see in an early linear algebra 00:26:49.950 --> 00:26:50.520 align:middle line:90% class. 00:26:50.520 --> 00:26:53.505 align:middle line:84% Think of a matrix as sort of having-- 00:26:53.505 --> 00:26:57.090 align:middle line:84% like, we're more than our hands, and our feet, and our noses, 00:26:57.090 --> 00:26:58.020 align:middle line:90% and our mouths, right? 00:26:58.020 --> 00:26:58.940 align:middle line:90% We're people, right? 00:26:58.940 --> 00:27:00.690 align:middle line:84% I want you to think of matrices that way-- 00:27:00.690 --> 00:27:03.070 align:middle line:90% as a holistic object. 00:27:03.070 --> 00:27:06.210 align:middle line:84% And there are ways of doing matrix calculus without going 00:27:06.210 --> 00:27:08.420 align:middle line:90% down to the element level. 00:27:08.420 --> 00:27:12.390 align:middle line:84% To kind of get that point across a little bit more, 00:27:12.390 --> 00:27:14.040 align:middle line:90% I still see this today. 00:27:14.040 --> 00:27:16.650 align:middle line:84% Though, it happened more and more. 00:27:16.650 --> 00:27:19.020 align:middle line:84% I, as a professor, would walk into a classroom 00:27:19.020 --> 00:27:20.430 align:middle line:90% to start to give a lecture. 00:27:20.430 --> 00:27:24.270 align:middle line:84% And in the old days, professors used chalk and blackboards. 00:27:24.270 --> 00:27:26.325 align:middle line:84% I guess they still do that sometimes. 00:27:26.325 --> 00:27:27.300 align:middle line:90% And I would walk in. 00:27:27.300 --> 00:27:29.940 align:middle line:84% And I would see what's on the blackboard 00:27:29.940 --> 00:27:31.470 align:middle line:90% from the previous class. 00:27:31.470 --> 00:27:34.693 align:middle line:84% And as I'm erasing it, I would actually look and see 00:27:34.693 --> 00:27:35.610 align:middle line:90% what would be written. 00:27:35.610 --> 00:27:38.340 align:middle line:90% And I would see people taking-- 00:27:38.340 --> 00:27:40.920 align:middle line:90% essentially, I would stand back. 00:27:40.920 --> 00:27:43.020 align:middle line:84% And I would look at the entire blackboard. 00:27:43.020 --> 00:27:45.420 align:middle line:84% And I realized that, this is just 00:27:45.420 --> 00:27:48.420 align:middle line:84% a couple of matrix multiplies written out element-wise. 00:27:48.420 --> 00:27:50.310 align:middle line:84% But it fills up the whole blackboard 00:27:50.310 --> 00:27:52.380 align:middle line:84% because, for whatever reason, the professor 00:27:52.380 --> 00:27:53.877 align:middle line:90% didn't use matrix notation. 00:27:53.877 --> 00:27:55.710 align:middle line:84% They just used scalar notation, with indices 00:27:55.710 --> 00:27:56.890 align:middle line:90% all over the place. 00:27:56.890 --> 00:27:59.130 align:middle line:84% And one of the things you'll see is 00:27:59.130 --> 00:28:01.740 align:middle line:84% how to kind of grow up from that point of view 00:28:01.740 --> 00:28:04.920 align:middle line:84% to actually-- sometimes it's useful to work with indices. 00:28:04.920 --> 00:28:06.300 align:middle line:90% And sometimes it's comforting. 00:28:06.300 --> 00:28:07.860 align:middle line:84% You feel like you're getting the right answer. 00:28:07.860 --> 00:28:09.418 align:middle line:84% But in some sense, it's more elegant 00:28:09.418 --> 00:28:10.710 align:middle line:90% when you don't have to do that. 00:28:10.710 --> 00:28:13.650 align:middle line:84% And we'll show you how to do that as well. 00:28:13.650 --> 00:28:14.150 align:middle line:90% OK. 00:28:14.150 --> 00:28:23.730 align:middle line:84% So let's take a look at sort of the various types of cases 00:28:23.730 --> 00:28:26.790 align:middle line:84% of what it even means to take a derivative, right? 00:28:26.790 --> 00:28:34.500 align:middle line:84% And so 18.01 is very much in the upper-left corner, right? 00:28:34.500 --> 00:28:37.680 align:middle line:84% So you have a scalar in, and a scalar comes out, right? 00:28:37.680 --> 00:28:41.460 align:middle line:84% That's everybody's first semester of calculus. 00:28:41.460 --> 00:28:42.750 align:middle line:90% OK? 00:28:42.750 --> 00:28:47.340 align:middle line:84% If you go along the first row over here, for example, 00:28:47.340 --> 00:28:49.530 align:middle line:90% probably at least in physics-- 00:28:49.530 --> 00:28:51.180 align:middle line:90% but very simple physics-- 00:28:51.180 --> 00:28:54.150 align:middle line:84% you learn the idea that you could 00:28:54.150 --> 00:28:57.870 align:middle line:84% have, say, your position in three dimensions 00:28:57.870 --> 00:28:59.220 align:middle line:90% as a function of time, right? 00:28:59.220 --> 00:29:01.050 align:middle line:90% So your input is time. 00:29:01.050 --> 00:29:02.040 align:middle line:90% It's a scalar. 00:29:02.040 --> 00:29:05.590 align:middle line:84% And your output is a position in space, like a-- 00:29:05.590 --> 00:29:08.700 align:middle line:84% for those of you who can't see what I'm doing online, 00:29:08.700 --> 00:29:12.310 align:middle line:84% I'm moving my hand to indicate a trajectory in space. 00:29:12.310 --> 00:29:12.810 align:middle line:90% OK? 00:29:12.810 --> 00:29:19.230 align:middle line:84% And of course, the derivative is now the velocity vector, right? 00:29:19.230 --> 00:29:21.480 align:middle line:84% That's what the derivative is of the function of time. 00:29:21.480 --> 00:29:23.220 align:middle line:90% It's tangent to the curve. 00:29:23.220 --> 00:29:26.580 align:middle line:84% And its magnitude tells you how fast you're going, right? 00:29:26.580 --> 00:29:30.120 align:middle line:84% And that's the derivative of a vector with a scalar. 00:29:30.120 --> 00:29:32.220 align:middle line:84% Now, this is not the sort of thing 00:29:32.220 --> 00:29:34.860 align:middle line:84% you would see much in previous classes, I wouldn't think. 00:29:34.860 --> 00:29:37.080 align:middle line:84% But it's perfectly reasonable to also talk 00:29:37.080 --> 00:29:40.050 align:middle line:84% about a trajectory in matrix space, right? 00:29:40.050 --> 00:29:41.910 align:middle line:84% And so you could have an m-by-n matrix 00:29:41.910 --> 00:29:43.605 align:middle line:90% that's a function of time. 00:29:43.605 --> 00:29:46.450 align:middle line:84% I could say it as every element as a function of time. 00:29:46.450 --> 00:29:49.050 align:middle line:84% And of course, you can take the derivative of that thing, 00:29:49.050 --> 00:29:51.810 align:middle line:84% right, which will be a fixed matrix, which, 00:29:51.810 --> 00:29:54.330 align:middle line:84% if you could imagine an m-by-n matrix space-- 00:29:54.330 --> 00:29:57.910 align:middle line:84% mathematicians love to imagine high-dimensional matrix spaces. 00:29:57.910 --> 00:29:59.590 align:middle line:84% high-dimensional spaces of all kinds. 00:29:59.590 --> 00:30:01.680 align:middle line:84% So if you can imagine an m-by-n matrix space, 00:30:01.680 --> 00:30:03.750 align:middle line:84% then the derivative would be a tangent 00:30:03.750 --> 00:30:05.890 align:middle line:84% in that big high-dimensional space. 00:30:05.890 --> 00:30:06.390 align:middle line:90% OK. 00:30:06.390 --> 00:30:08.250 align:middle line:90% So that's the first row. 00:30:08.250 --> 00:30:12.994 align:middle line:84% Maybe it's a good time to now go down the first column. 00:30:12.994 --> 00:30:21.350 align:middle line:84% So this blue-- it looks-- yeah, I guess it looks more purple 00:30:21.350 --> 00:30:23.975 align:middle line:90% on the screen, but it's blue-- 00:30:23.975 --> 00:30:26.153 align:middle line:90% is sort of the-- 00:30:26.153 --> 00:30:28.070 align:middle line:84% this is what you do in multivariate calculus-- 00:30:28.070 --> 00:30:31.400 align:middle line:84% 18.02-- and machine learning everywhere, which is, we take 00:30:31.400 --> 00:30:32.180 align:middle line:90% gradients. 00:30:32.180 --> 00:30:36.290 align:middle line:84% So it's very typical where you have a function 00:30:36.290 --> 00:30:38.840 align:middle line:90% with many variables going in. 00:30:38.840 --> 00:30:40.140 align:middle line:90% I'll just talk about vectors. 00:30:40.140 --> 00:30:41.720 align:middle line:84% But in fact, you could have lots more variables 00:30:41.720 --> 00:30:42.512 align:middle line:90% than just a vector. 00:30:42.512 --> 00:30:44.630 align:middle line:84% But let's just say, in machine learning, 00:30:44.630 --> 00:30:47.510 align:middle line:84% you have a vector going in and a scalar going out. 00:30:47.510 --> 00:30:49.700 align:middle line:84% The scalar is often called the loss function 00:30:49.700 --> 00:30:51.230 align:middle line:90% in machine learning. 00:30:51.230 --> 00:30:55.910 align:middle line:84% And how many of you have heard the word "loss function"? 00:30:55.910 --> 00:30:56.790 align:middle line:90% Everybody. 00:30:56.790 --> 00:30:57.290 align:middle line:90% OK. 00:30:57.290 --> 00:31:00.440 align:middle line:84% I think students are learning it in kindergarten these days. 00:31:00.440 --> 00:31:02.540 align:middle line:90% So yeah, everybody-- right. 00:31:02.540 --> 00:31:05.600 align:middle line:84% So you're all familiar with many, many variables coming in. 00:31:05.600 --> 00:31:08.703 align:middle line:84% That's the vector, the input, and the loss function, 00:31:08.703 --> 00:31:09.620 align:middle line:90% the scalar coming out. 00:31:09.620 --> 00:31:12.950 align:middle line:84% And you want to take a gradient of that thing, right? 00:31:12.950 --> 00:31:16.460 align:middle line:84% And so if you actually literally have a vector going in, 00:31:16.460 --> 00:31:20.410 align:middle line:90% then the gradient is-- 00:31:20.410 --> 00:31:22.800 align:middle line:84% in fact, for every scalar function, 00:31:22.800 --> 00:31:25.780 align:middle line:84% the gradient always has the same shape as the inputs. 00:31:25.780 --> 00:31:29.380 align:middle line:84% So if your input is a vector, then the gradient is a vector, 00:31:29.380 --> 00:31:29.880 align:middle line:90% right? 00:31:29.880 --> 00:31:34.950 align:middle line:84% So that's often denoted with this nabla notation. 00:31:34.950 --> 00:31:37.800 align:middle line:90% In LaTeX, it's backslash nabla-- 00:31:37.800 --> 00:31:41.940 align:middle line:84% so usually pronounced grad f, or gradient of f. 00:31:41.940 --> 00:31:45.510 align:middle line:84% And in Julia, it would be a vector. 00:31:45.510 --> 00:31:48.210 align:middle line:84% We don't have to talk about column vectors and row vectors. 00:31:48.210 --> 00:31:51.537 align:middle line:84% It's simple to just say "vectors." 00:31:51.537 --> 00:31:53.620 align:middle line:84% It gets too confused when you have column vectors, 00:31:53.620 --> 00:31:55.250 align:middle line:84% but people are used to that notation. 00:31:55.250 --> 00:31:56.350 align:middle line:90% It's just a vector. 00:31:56.350 --> 00:31:58.570 align:middle line:84% But you could call it a column vector. 00:31:58.570 --> 00:32:02.860 align:middle line:84% But the derivative is the transpose. 00:32:02.860 --> 00:32:04.960 align:middle line:90% It's a row vector, right? 00:32:04.960 --> 00:32:08.650 align:middle line:84% And Steven and I-- we actually went back and forth 00:32:08.650 --> 00:32:10.250 align:middle line:90% on this a couple of years ago. 00:32:10.250 --> 00:32:12.610 align:middle line:84% And we were absolutely convinced that this 00:32:12.610 --> 00:32:15.970 align:middle line:84% is the best way to do it-- that the gradient is 00:32:15.970 --> 00:32:20.380 align:middle line:84% a vector, or a column vector, and the f prime 00:32:20.380 --> 00:32:22.580 align:middle line:90% is a row vector. 00:32:22.580 --> 00:32:24.370 align:middle line:84% So in particular, I'll just give you 00:32:24.370 --> 00:32:26.110 align:middle line:90% the answer for x transpose x. 00:32:26.110 --> 00:32:28.450 align:middle line:90% So here's f of x, right? 00:32:28.450 --> 00:32:32.290 align:middle line:84% And we're talking about x being a vector, right? 00:32:32.290 --> 00:32:38.320 align:middle line:84% And so the gradient of f is equal to 2x. 00:32:38.320 --> 00:32:38.820 align:middle line:90% OK? 00:32:38.820 --> 00:32:44.350 align:middle line:90% But df will be 2x transpose dx. 00:32:44.350 --> 00:32:48.700 align:middle line:84% Or f prime of x will be 2x transpose. 00:32:48.700 --> 00:32:51.140 align:middle line:84% And I hope you can start to appreciate 00:32:51.140 --> 00:32:52.610 align:middle line:90% why this is a good idea. 00:32:52.610 --> 00:32:55.670 align:middle line:84% Because we want this to be a linear operator-- that we 00:32:55.670 --> 00:32:59.870 align:middle line:90% put in a little vector change. 00:32:59.870 --> 00:33:02.840 align:middle line:84% And we want the output to be a scalar, right? 00:33:02.840 --> 00:33:05.660 align:middle line:90% So it makes no sense to go-- 00:33:05.660 --> 00:33:09.050 align:middle line:84% you see, it makes no sense to go to 2x dx, 00:33:09.050 --> 00:33:11.453 align:middle line:84% where everything-- this is a little vector change. 00:33:11.453 --> 00:33:12.620 align:middle line:90% And this is a little vector. 00:33:12.620 --> 00:33:15.410 align:middle line:84% There's no such thing as multiplying vector by vector. 00:33:15.410 --> 00:33:17.600 align:middle line:90% But if we write this-- 00:33:17.600 --> 00:33:22.670 align:middle line:84% 2x transpose-- then when we multiply by dx, 00:33:22.670 --> 00:33:25.250 align:middle line:84% as we're doing here, out comes a scalar, right? 00:33:25.250 --> 00:33:28.070 align:middle line:84% So I make a little small change to x-- 00:33:28.070 --> 00:33:29.857 align:middle line:84% infinitesimal or a tiny little change, 00:33:29.857 --> 00:33:31.190 align:middle line:90% however you want to think of it. 00:33:31.190 --> 00:33:33.710 align:middle line:84% And out comes a scalar change to f. 00:33:33.710 --> 00:33:37.380 align:middle line:84% And so this actually makes perfect sense. 00:33:37.380 --> 00:33:42.020 align:middle line:84% And I don't think you would learn it that way in 18.02. 00:33:42.020 --> 00:33:46.310 align:middle line:84% And so this is-- but this seems to lead to consistent answers. 00:33:46.310 --> 00:33:50.030 align:middle line:84% And so we're going to encourage you to take that point of view. 00:33:50.030 --> 00:33:50.540 align:middle line:90% All right. 00:33:50.540 --> 00:33:54.580 align:middle line:84% So I've covered four of these boxes. 00:33:54.580 --> 00:33:57.340 align:middle line:84% Let me point out that there's a color-coding that you might 00:33:57.340 --> 00:34:00.280 align:middle line:84% have noticed to these boxes, if you haven't noticed it already. 00:34:00.280 --> 00:34:04.120 align:middle line:84% The green is when the derivative is zero-dimensional. 00:34:04.120 --> 00:34:05.980 align:middle line:84% It goes along the diagonals, right? 00:34:05.980 --> 00:34:08.949 align:middle line:84% The blue is one-dimensional answers, right-- 00:34:08.949 --> 00:34:12.370 align:middle line:84% the vector and the gradient, those two boxes. 00:34:12.370 --> 00:34:15.730 align:middle line:90% And then the next diagonal-- 00:34:15.730 --> 00:34:17.170 align:middle line:90% it's a funny diagonal, right? 00:34:17.170 --> 00:34:18.909 align:middle line:84% It's going from northeast to southwest. 00:34:18.909 --> 00:34:21.730 align:middle line:84% But this funny diagonal-- the next step 00:34:21.730 --> 00:34:25.239 align:middle line:84% is when the derivative itself is a matrix, right? 00:34:25.239 --> 00:34:29.980 align:middle line:84% And so one example was in the top right, when you just have 00:34:29.980 --> 00:34:32.920 align:middle line:90% a trajectory in matrix space. 00:34:32.920 --> 00:34:35.920 align:middle line:84% Another one is in the bottom left-- 00:34:35.920 --> 00:34:40.449 align:middle line:84% is, if your parameters are coming into a machine learning 00:34:40.449 --> 00:34:43.969 align:middle line:84% algorithm as an array and you have a loss function, 00:34:43.969 --> 00:34:46.270 align:middle line:84% then the gradient-- as I said, the gradient of a scalar 00:34:46.270 --> 00:34:49.179 align:middle line:84% function always matches the shape of the input, right? 00:34:49.179 --> 00:34:51.710 align:middle line:84% And so the gradient of a matrix is a matrix. 00:34:51.710 --> 00:34:52.210 align:middle line:90% OK? 00:34:52.210 --> 00:34:55.100 align:middle line:84% And then in the middle of these 3-by-3 arrays, 00:34:55.100 --> 00:34:58.410 align:middle line:84% I think you would have all seen in 18.02-- 00:34:58.410 --> 00:35:00.820 align:middle line:84% I don't know if you remember it anymore-- 00:35:00.820 --> 00:35:01.820 align:middle line:90% most of you probably do. 00:35:01.820 --> 00:35:02.630 align:middle line:90% Some of you may have forgotten. 00:35:02.630 --> 00:35:05.172 align:middle line:84% STEVEN JOHNSON: 18.02 doesn't always cover it, unfortunately. 00:35:05.172 --> 00:35:06.895 align:middle line:84% ALAN EDELMAN: Doesn't cover Jacobians? 00:35:06.895 --> 00:35:09.020 align:middle line:84% STEVEN JOHNSON: And certainly not as linearization. 00:35:09.020 --> 00:35:11.180 align:middle line:84% They sometimes do Jacobian matrices 00:35:11.180 --> 00:35:13.370 align:middle line:90% determinants for integrals. 00:35:13.370 --> 00:35:15.620 align:middle line:84% But they don't even always cover that, apparently. 00:35:15.620 --> 00:35:19.610 align:middle line:84% ALAN EDELMAN: But you mean students in 18.02 00:35:19.610 --> 00:35:22.925 align:middle line:84% will not see a Jacobian matrix in some form or another? 00:35:22.925 --> 00:35:24.050 align:middle line:90% STEVEN JOHNSON: Apparently. 00:35:24.050 --> 00:35:25.508 align:middle line:84% ALAN EDELMAN: What has MIT come to? 00:35:25.508 --> 00:35:28.630 align:middle line:84% How many of you have actually taken 18.02 at MIT? 00:35:28.630 --> 00:35:29.290 align:middle line:90% OK. 00:35:29.290 --> 00:35:33.210 align:middle line:84% And among those, how many of you think you once learned 00:35:33.210 --> 00:35:34.210 align:middle line:90% about a Jacobian matrix? 00:35:34.210 --> 00:35:36.190 align:middle line:84% You don't have to have remembered it. 00:35:36.190 --> 00:35:36.690 align:middle line:90% OK. 00:35:36.690 --> 00:35:38.370 align:middle line:84% Well, we've got a good set of students 00:35:38.370 --> 00:35:41.030 align:middle line:84% because apparently they learned Jacobian matrices. 00:35:41.030 --> 00:35:43.530 align:middle line:84% I would have thought that that would just go without saying, 00:35:43.530 --> 00:35:45.000 align:middle line:90% but-- 00:35:45.000 --> 00:35:46.500 align:middle line:90% STEVEN JOHNSON: But it's very-- 00:35:46.500 --> 00:35:49.523 align:middle line:84% often, it's only for multi-dimensional integration. 00:35:49.523 --> 00:35:51.940 align:middle line:84% So it's only determinants of Jacobians that they ever see. 00:35:51.940 --> 00:35:54.300 align:middle line:84% So the Jacobian and its determinant-- they 00:35:54.300 --> 00:35:57.010 align:middle line:90% kind of get mixed together. 00:35:57.010 --> 00:35:58.010 align:middle line:90% ALAN EDELMAN: All right. 00:35:58.010 --> 00:36:00.260 align:middle line:90% That sounds unfortunate. 00:36:00.260 --> 00:36:02.360 align:middle line:90% [CHUCKLES] But OK. 00:36:02.360 --> 00:36:05.990 align:middle line:90% Thanks for the update on 18.02. 00:36:05.990 --> 00:36:09.510 align:middle line:84% A few years back, I actually looked at the 18.02 ASE notes. 00:36:09.510 --> 00:36:11.420 align:middle line:90% And I know that it was there. 00:36:11.420 --> 00:36:14.270 align:middle line:84% But I don't know what changes have been made. 00:36:14.270 --> 00:36:16.340 align:middle line:90% OK. 00:36:16.340 --> 00:36:19.410 align:middle line:84% But in any event, just to kind of set the stage, 00:36:19.410 --> 00:36:22.500 align:middle line:84% this is where you have a vector input and a vector output, 00:36:22.500 --> 00:36:23.000 align:middle line:90% right? 00:36:23.000 --> 00:36:26.660 align:middle line:84% So you have a function in, say, three dimensions. 00:36:26.660 --> 00:36:30.590 align:middle line:84% And the output is also a function, say, in three-- 00:36:30.590 --> 00:36:33.000 align:middle line:84% or it could be lower or higher dimensions, right? 00:36:33.000 --> 00:36:39.860 align:middle line:84% And if you have n dimensions going in and m-- as in "Mary"-- 00:36:39.860 --> 00:36:44.390 align:middle line:84% dimensions going out, then the derivative is most naturally 00:36:44.390 --> 00:36:47.100 align:middle line:84% done if you do it as a matrix, not as a linear operator. 00:36:47.100 --> 00:36:49.920 align:middle line:84% But if you do it as a matrix, it will be an m-by-n matrix. 00:36:49.920 --> 00:36:50.420 align:middle line:90% OK? 00:36:50.420 --> 00:36:53.240 align:middle line:84% And then, of course, we get into the interesting situations 00:36:53.240 --> 00:36:54.800 align:middle line:90% as you move down this table-- 00:36:54.800 --> 00:36:56.720 align:middle line:84% the situations where that web page doesn't 00:36:56.720 --> 00:36:57.840 align:middle line:90% do a very good job at it. 00:36:57.840 --> 00:36:59.673 align:middle line:84% But we're going to show you how to do this-- 00:36:59.673 --> 00:37:03.150 align:middle line:84% how to think about just even what's a good notation when 00:37:03.150 --> 00:37:06.930 align:middle line:84% you start moving into higher-order arrays, 00:37:06.930 --> 00:37:12.960 align:middle line:84% right, where the derivative is no longer best expressed 00:37:12.960 --> 00:37:15.370 align:middle line:84% with two-dimensional matrices anymore, right? 00:37:15.370 --> 00:37:18.550 align:middle line:84% And so that's down on the bottom right. 00:37:18.550 --> 00:37:19.050 align:middle line:90% OK? 00:37:19.050 --> 00:37:25.727 align:middle line:84% So here are some answers to set the stage. 00:37:25.727 --> 00:37:27.810 align:middle line:84% And again, we're going to show you how to do this. 00:37:27.810 --> 00:37:30.840 align:middle line:84% But I think it's kind of nice to foreshadow a little bit of what 00:37:30.840 --> 00:37:31.870 align:middle line:90% you're going to see. 00:37:31.870 --> 00:37:36.565 align:middle line:84% So here, I'm going to use this sort of operator notation, 00:37:36.565 --> 00:37:38.190 align:middle line:84% starting with one you're familiar with. 00:37:38.190 --> 00:37:41.760 align:middle line:84% The derivative of x-cubed is 3x-squared dx, right? 00:37:41.760 --> 00:37:43.710 align:middle line:90% So everybody kind of knows that. 00:37:43.710 --> 00:37:45.720 align:middle line:84% And just to encourage you, just to 00:37:45.720 --> 00:37:47.370 align:middle line:90% remember what I've said before-- 00:37:47.370 --> 00:37:49.680 align:middle line:84% that I want you to think of this as, 00:37:49.680 --> 00:37:54.660 align:middle line:84% if I make a little small change to x, my change to my function 00:37:54.660 --> 00:37:58.410 align:middle line:84% will be 3x squared times that small change, right? 00:37:58.410 --> 00:38:04.470 align:middle line:84% If I'm at x equals 7, right, and I make a 0.01 change, 00:38:04.470 --> 00:38:06.300 align:middle line:84% then the result is going to be 3 7 00:38:06.300 --> 00:38:09.370 align:middle line:84% squared times 0.01 approximately, right? 00:38:09.370 --> 00:38:09.870 align:middle line:90% Right? 00:38:09.870 --> 00:38:12.930 align:middle line:84% And of course, that gets better as 0.01 gets closer and closer 00:38:12.930 --> 00:38:13.870 align:middle line:90% to 0. 00:38:13.870 --> 00:38:14.370 align:middle line:90% OK? 00:38:14.370 --> 00:38:19.720 align:middle line:90% So here's one that-- 00:38:19.720 --> 00:38:22.980 align:middle line:84% in fact, let's use the mouse again just to emphasize. 00:38:22.980 --> 00:38:24.368 align:middle line:90% But this one here-- whoops. 00:38:24.368 --> 00:38:25.410 align:middle line:90% I didn't want to do that. 00:38:25.410 --> 00:38:28.540 align:middle line:90% This one here is-- 00:38:28.540 --> 00:38:29.040 align:middle line:90% let's see. 00:38:29.040 --> 00:38:29.790 align:middle line:90% We have a-- 00:38:29.790 --> 00:38:32.530 align:middle line:84% I want this to be a vector input and a scalar output. 00:38:32.530 --> 00:38:34.120 align:middle line:90% So it's this box again. 00:38:34.120 --> 00:38:36.120 align:middle line:84% And it's the same thing I wrote over here. 00:38:36.120 --> 00:38:38.160 align:middle line:84% I'm just using prime instead of transpose. 00:38:38.160 --> 00:38:40.350 align:middle line:84% But again, I want to encourage you-- 00:38:40.350 --> 00:38:46.620 align:middle line:84% that if x is a vector and dx is a small change to that vector, 00:38:46.620 --> 00:38:48.900 align:middle line:84% then you take the dot product, right? 00:38:48.900 --> 00:38:52.650 align:middle line:84% So I hope everybody realizes that, if you ever 00:38:52.650 --> 00:38:55.660 align:middle line:84% go x transpose y, that's the same thing as a dot product, 00:38:55.660 --> 00:38:56.160 align:middle line:90% right? 00:38:56.160 --> 00:38:57.720 align:middle line:84% That means multiply all the elements 00:38:57.720 --> 00:38:59.460 align:middle line:84% and then add everything up, right? 00:38:59.460 --> 00:39:03.690 align:middle line:84% So this is a dot product between 2x and-- 00:39:03.690 --> 00:39:05.820 align:middle line:90% this is, I'm at the point x. 00:39:05.820 --> 00:39:07.860 align:middle line:90% And my change is a little dx. 00:39:07.860 --> 00:39:11.715 align:middle line:84% This dot product will be the change to x transpose x. 00:39:11.715 --> 00:39:12.547 align:middle line:90% OK? 00:39:12.547 --> 00:39:16.560 align:middle line:84% And here's your first matrix answer. 00:39:16.560 --> 00:39:20.010 align:middle line:84% I told you that, if you square a matrix, 00:39:20.010 --> 00:39:22.740 align:middle line:84% the answer is not, like, 2x, right? 00:39:22.740 --> 00:39:23.670 align:middle line:90% What is it? 00:39:23.670 --> 00:39:27.390 align:middle line:84% Well, in fact, because matrices don't commute-- 00:39:27.390 --> 00:39:31.050 align:middle line:90% the x and the dx don't commute-- 00:39:31.050 --> 00:39:33.720 align:middle line:84% in a way, you could think about it this way. 00:39:33.720 --> 00:39:38.250 align:middle line:84% Should we go x dx times 2 or dx times x times 2? 00:39:38.250 --> 00:39:41.470 align:middle line:84% Well, in fact, there's no reason to prefer 1 over the other. 00:39:41.470 --> 00:39:43.470 align:middle line:84% And in fact, the correct answer is 00:39:43.470 --> 00:39:46.810 align:middle line:84% to add the multiplications both ways. 00:39:46.810 --> 00:39:52.960 align:middle line:84% So if you make a small change to a matrix and you ask, 00:39:52.960 --> 00:39:55.440 align:middle line:84% what is the change to the square, then 00:39:55.440 --> 00:39:57.070 align:middle line:90% this is the correct answer. 00:39:57.070 --> 00:40:01.170 align:middle line:84% This is-- and you see now maybe for the first time 00:40:01.170 --> 00:40:05.850 align:middle line:84% why we need to think of it as a linear operator. 00:40:05.850 --> 00:40:10.140 align:middle line:84% We can write this out as a big matrix. 00:40:10.140 --> 00:40:11.835 align:middle line:90% But it's not a good idea, right? 00:40:11.835 --> 00:40:13.960 align:middle line:84% It might be comfortable, but we don't recommend it. 00:40:13.960 --> 00:40:19.260 align:middle line:84% So you see, you could think of x as n-squared variables. 00:40:19.260 --> 00:40:21.810 align:middle line:84% If x is an n-by-n matrix, right-- 00:40:21.810 --> 00:40:24.900 align:middle line:90% 00:40:24.900 --> 00:40:26.700 align:middle line:90% here's x. 00:40:26.700 --> 00:40:28.365 align:middle line:90% That's n-by-n. 00:40:28.365 --> 00:40:30.168 align:middle line:90% We have n-squared variables. 00:40:30.168 --> 00:40:31.710 align:middle line:84% And so we could think of the function 00:40:31.710 --> 00:40:34.380 align:middle line:90% that squares our matrix. 00:40:34.380 --> 00:40:36.480 align:middle line:84% If you want, you could, like, flatten everything 00:40:36.480 --> 00:40:39.510 align:middle line:84% and think of it as going from n-squared variables 00:40:39.510 --> 00:40:40.800 align:middle line:90% to n-squared variables. 00:40:40.800 --> 00:40:42.622 align:middle line:84% And then the derivative would have 00:40:42.622 --> 00:40:43.830 align:middle line:90% n to the fourth things in it. 00:40:43.830 --> 00:40:46.200 align:middle line:84% You could write it as an n-squared by n-squared matrix, 00:40:46.200 --> 00:40:48.400 align:middle line:90% but we're not recommending that. 00:40:48.400 --> 00:40:51.740 align:middle line:84% We're saying, just think of it as this linear operator. 00:40:51.740 --> 00:40:52.240 align:middle line:90% OK? 00:40:52.240 --> 00:40:54.670 align:middle line:84% And that's going to be the answer 00:40:54.670 --> 00:40:58.030 align:middle line:84% for the derivative of x-squared, OK, for a matrix square. 00:40:58.030 --> 00:41:01.360 align:middle line:84% Obviously, it reduces to 2x dx when it scalars, 00:41:01.360 --> 00:41:03.735 align:middle line:90% but it's much better than that. 00:41:03.735 --> 00:41:04.540 align:middle line:90% OK? 00:41:04.540 --> 00:41:09.090 align:middle line:84% So sometimes I open up a Julia to do this, 00:41:09.090 --> 00:41:11.620 align:middle line:84% but I think these slides kind of say it all. 00:41:11.620 --> 00:41:17.040 align:middle line:84% So just to kind of hit home on both the matrix square 00:41:17.040 --> 00:41:21.390 align:middle line:84% and the x transpose x, I'll just give you a numerical example 00:41:21.390 --> 00:41:23.830 align:middle line:84% just to make sure this is completely clear. 00:41:23.830 --> 00:41:26.340 align:middle line:84% So let's consider the function x transpose x, right? 00:41:26.340 --> 00:41:28.530 align:middle line:84% So I hope you all realize that's just 00:41:28.530 --> 00:41:32.080 align:middle line:84% the sum of the squares of the entries of x, right? 00:41:32.080 --> 00:41:37.440 align:middle line:84% And so if I'm at the point 3, 4, right-- 00:41:37.440 --> 00:41:40.020 align:middle line:84% so I'm at a point in space, a two-dimensional space-- 00:41:40.020 --> 00:41:41.640 align:middle line:90% 3, 4, right? 00:41:41.640 --> 00:41:47.640 align:middle line:84% So my function value is 9 plus 16 is 25, right? 00:41:47.640 --> 00:41:49.350 align:middle line:90% And let's make a little change. 00:41:49.350 --> 00:41:51.450 align:middle line:90% Let's say my dx is-- 00:41:51.450 --> 00:41:53.610 align:middle line:84% I'm going to go 0.001 in my first direction 00:41:53.610 --> 00:41:55.980 align:middle line:90% and 0.002 in the next direction. 00:41:55.980 --> 00:41:57.300 align:middle line:90% Well, you could do the math. 00:41:57.300 --> 00:42:01.960 align:middle line:84% And basically, the answer is about 25.022. 00:42:01.960 --> 00:42:02.460 align:middle line:90% OK? 00:42:02.460 --> 00:42:07.590 align:middle line:84% And so here, we're explicitly making those changes, right? 00:42:07.590 --> 00:42:10.590 align:middle line:84% But the whole point of calculus is 00:42:10.590 --> 00:42:15.111 align:middle line:84% to have, like, a formula for doing that, right? 00:42:15.111 --> 00:42:19.920 align:middle line:84% The reason why we don't just do the calculation that's 00:42:19.920 --> 00:42:23.550 align:middle line:84% on this line is because the magic of calculus 00:42:23.550 --> 00:42:25.780 align:middle line:84% is there's an exact way to do it. 00:42:25.780 --> 00:42:28.920 align:middle line:90% And this is that exact way. 00:42:28.920 --> 00:42:32.160 align:middle line:84% If you go 2x0 transpose dx, you see 00:42:32.160 --> 00:42:33.740 align:middle line:84% we're going to get that right answer. 00:42:33.740 --> 00:42:34.240 align:middle line:90% OK? 00:42:34.240 --> 00:42:36.350 align:middle line:84% And that's what we love about calculus. 00:42:36.350 --> 00:42:36.850 align:middle line:90% OK? 00:42:36.850 --> 00:42:41.780 align:middle line:84% And so the delta f is going to be that. 00:42:41.780 --> 00:42:46.060 align:middle line:90% Any questions about that so far? 00:42:46.060 --> 00:42:46.560 align:middle line:90% OK. 00:42:46.560 --> 00:42:50.878 align:middle line:90% And I could-- 00:42:50.878 --> 00:42:51.420 align:middle line:90% I don't know. 00:42:51.420 --> 00:42:52.337 align:middle line:90% Well, let me just see. 00:42:52.337 --> 00:42:54.910 align:middle line:84% Do you want to see me do this with a matrix? 00:42:54.910 --> 00:42:57.270 align:middle line:84% I don't think I put it in the slides. 00:42:57.270 --> 00:42:59.100 align:middle line:84% Should I do a matrix quickly in Julia? 00:42:59.100 --> 00:42:59.760 align:middle line:90% We could do it. 00:42:59.760 --> 00:43:02.160 align:middle line:84% Do you want-- oh, people are saying, yes, I should do it. 00:43:02.160 --> 00:43:02.660 align:middle line:90% All right. 00:43:02.660 --> 00:43:04.745 align:middle line:84% So let's open up a Julia quickly. 00:43:04.745 --> 00:43:06.120 align:middle line:84% I think this is the sort of thing 00:43:06.120 --> 00:43:08.245 align:middle line:84% that's just easy enough I could actually just do it 00:43:08.245 --> 00:43:08.910 align:middle line:90% in the repl. 00:43:08.910 --> 00:43:11.160 align:middle line:84% Should I do it with VS Code or just do it in the repl? 00:43:11.160 --> 00:43:12.370 align:middle line:90% I'll just do it in the repl. 00:43:12.370 --> 00:43:12.870 align:middle line:90% OK. 00:43:12.870 --> 00:43:15.760 align:middle line:90% So let's get a Julia. 00:43:15.760 --> 00:43:18.080 align:middle line:84% This is a bit impromptu, but why not? 00:43:18.080 --> 00:43:18.580 align:middle line:90% OK. 00:43:18.580 --> 00:43:19.955 align:middle line:84% So here, let's just grab a Julia. 00:43:19.955 --> 00:43:22.650 align:middle line:90% 00:43:22.650 --> 00:43:23.150 align:middle line:90% OK. 00:43:23.150 --> 00:43:25.410 align:middle line:84% And what I'd like to do is demonstrate-- 00:43:25.410 --> 00:43:31.770 align:middle line:84% so the plan here is to demonstrate that dx-- 00:43:31.770 --> 00:43:35.520 align:middle line:90% 00:43:35.520 --> 00:43:37.020 align:middle line:90% how do I do this? 00:43:37.020 --> 00:43:37.740 align:middle line:90% Like that. 00:43:37.740 --> 00:43:44.390 align:middle line:84% Dx-squared is going to equal dx times x plus x dx. 00:43:44.390 --> 00:43:44.890 align:middle line:90% OK. 00:43:44.890 --> 00:43:48.010 align:middle line:90% So that's going to be my goal. 00:43:48.010 --> 00:43:49.720 align:middle line:84% Any particular size you want to see? 00:43:49.720 --> 00:43:50.630 align:middle line:90% 3-by-3 good enough? 00:43:50.630 --> 00:43:51.130 align:middle line:90% I guess so. 00:43:51.130 --> 00:43:52.830 align:middle line:90% Let's just do 3-by-3. 00:43:52.830 --> 00:43:54.580 align:middle line:84% Yeah, we'll make this a little bit bigger. 00:43:54.580 --> 00:43:57.710 align:middle line:84% So I don't even think I need linear algebra. 00:43:57.710 --> 00:43:58.210 align:middle line:90% Let's see. 00:43:58.210 --> 00:43:59.835 align:middle line:84% I don't even think I need any packages. 00:43:59.835 --> 00:44:02.440 align:middle line:90% So should I take-- 00:44:02.440 --> 00:44:03.310 align:middle line:90% all right. 00:44:03.310 --> 00:44:06.560 align:middle line:84% Let me hear your nine favorite integers-- 00:44:06.560 --> 00:44:07.060 align:middle line:90% small. 00:44:07.060 --> 00:44:08.020 align:middle line:90% Don't make them too big. 00:44:08.020 --> 00:44:08.380 align:middle line:90% Come on. 00:44:08.380 --> 00:44:09.700 align:middle line:90% Just shout out nine integers. 00:44:09.700 --> 00:44:10.200 align:middle line:90% AUDIENCE: 5. 00:44:10.200 --> 00:44:11.990 align:middle line:90% ALAN EDELMAN: 5. 00:44:11.990 --> 00:44:12.490 align:middle line:90% AUDIENCE: 7. 00:44:12.490 --> 00:44:13.705 align:middle line:90% ALAN EDELMAN: 7. 00:44:13.705 --> 00:44:14.580 align:middle line:90% AUDIENCE: Negative 3. 00:44:14.580 --> 00:44:15.622 align:middle line:90% ALAN EDELMAN: Negative 3. 00:44:15.622 --> 00:44:16.163 align:middle line:90% AUDIENCE: 17. 00:44:16.163 --> 00:44:16.940 align:middle line:90% AUDIENCE: 1, 2, 3. 00:44:16.940 --> 00:44:18.200 align:middle line:90% ALAN EDELMAN: 1, 2, 3. 00:44:18.200 --> 00:44:19.190 align:middle line:90% Thank you. 00:44:19.190 --> 00:44:20.240 align:middle line:90% OK. 00:44:20.240 --> 00:44:21.170 align:middle line:90% Hurrying along here. 00:44:21.170 --> 00:44:23.290 align:middle line:90% Three more? 00:44:23.290 --> 00:44:24.076 align:middle line:90% AUDIENCE: 6, 7, 8. 00:44:24.076 --> 00:44:24.470 align:middle line:90% AUDIENCE: 0. 00:44:24.470 --> 00:44:25.070 align:middle line:90% ALAN EDELMAN: 6, 7, 8. 00:44:25.070 --> 00:44:25.700 align:middle line:90% All right. 00:44:25.700 --> 00:44:28.370 align:middle line:84% Just to show you that there's nothing up my sleeve. 00:44:28.370 --> 00:44:29.360 align:middle line:90% OK. 00:44:29.360 --> 00:44:34.750 align:middle line:84% And let's go, y is equal to x-squared maybe. 00:44:34.750 --> 00:44:36.590 align:middle line:90% So here's the matrix square. 00:44:36.590 --> 00:44:37.090 align:middle line:90% OK? 00:44:37.090 --> 00:44:38.548 align:middle line:84% So you could check that yourselves. 00:44:38.548 --> 00:44:43.390 align:middle line:90% And let's take dx to be-- 00:44:43.390 --> 00:44:47.717 align:middle line:84% oh, let's just go 0.001 times rand of 3,3. 00:44:47.717 --> 00:44:48.550 align:middle line:90% Or is that too ugly? 00:44:48.550 --> 00:44:50.750 align:middle line:90% Oh, it's fine. 00:44:50.750 --> 00:44:51.780 align:middle line:90% Let's just do it. 00:44:51.780 --> 00:44:52.280 align:middle line:90% OK? 00:44:52.280 --> 00:44:54.180 align:middle line:84% So here's a bunch of ugly numbers. 00:44:54.180 --> 00:44:54.680 align:middle line:90% OK? 00:44:54.680 --> 00:44:58.370 align:middle line:84% So everybody understands that I'm sitting at this matrix X. 00:44:58.370 --> 00:45:03.950 align:middle line:84% And I'm moving over to X plus dx, right? 00:45:03.950 --> 00:45:05.580 align:middle line:84% And I want to know the difference. 00:45:05.580 --> 00:45:07.190 align:middle line:90% So let's do that, right? 00:45:07.190 --> 00:45:10.340 align:middle line:84% So dy-- we'll do it first numerically-- 00:45:10.340 --> 00:45:19.110 align:middle line:84% is going to be X plus dx-squared minus X-squared. 00:45:19.110 --> 00:45:19.610 align:middle line:90% OK? 00:45:19.610 --> 00:45:22.030 align:middle line:90% So there's the dy. 00:45:22.030 --> 00:45:22.570 align:middle line:90% OK? 00:45:22.570 --> 00:45:27.170 align:middle line:84% And now let's compare that with this magic matrix calculus 00:45:27.170 --> 00:45:27.670 align:middle line:90% formula. 00:45:27.670 --> 00:45:33.760 align:middle line:84% The matrix calculus formula is X times dx plus dx times X. 00:45:33.760 --> 00:45:37.900 align:middle line:84% And this is where I always have that fear that I made a mistake 00:45:37.900 --> 00:45:38.620 align:middle line:90% or a typo. 00:45:38.620 --> 00:45:42.200 align:middle line:84% But let's hit Enter and see what happens. 00:45:42.200 --> 00:45:43.880 align:middle line:90% And look at that. 00:45:43.880 --> 00:45:44.880 align:middle line:90% You see? 00:45:44.880 --> 00:45:48.170 align:middle line:84% So this is the numerical way of calculating a derivative. 00:45:48.170 --> 00:45:52.190 align:middle line:84% Just see where I am over there minus where I started. 00:45:52.190 --> 00:45:54.170 align:middle line:90% Or there's this magic formula. 00:45:54.170 --> 00:45:58.130 align:middle line:84% And just to show you that these other ways you 00:45:58.130 --> 00:46:01.340 align:middle line:84% might think of doing it won't work, like 2x dx-- 00:46:01.340 --> 00:46:03.560 align:middle line:90% that's not right. 00:46:03.560 --> 00:46:06.920 align:middle line:90% Or 2 times dx times X-- 00:46:06.920 --> 00:46:08.690 align:middle line:90% I don't need that times. 00:46:08.690 --> 00:46:10.550 align:middle line:90% That's not right. 00:46:10.550 --> 00:46:16.810 align:middle line:84% But the one that is right-- here, let's put it back up-- 00:46:16.810 --> 00:46:18.560 align:middle line:90% is clearly that one. 00:46:18.560 --> 00:46:21.260 align:middle line:90% OK? 00:46:21.260 --> 00:46:21.760 align:middle line:90% OK. 00:46:21.760 --> 00:46:23.410 align:middle line:84% So I haven't showed you how to derive it yet, 00:46:23.410 --> 00:46:24.580 align:middle line:90% but that's coming soon. 00:46:24.580 --> 00:46:26.810 align:middle line:84% But I just wanted you to sort of-- 00:46:26.810 --> 00:46:29.980 align:middle line:84% really, for me-- I don't know if your minds work this way. 00:46:29.980 --> 00:46:35.530 align:middle line:84% But for me, seeing this is pretty convincing, right? 00:46:35.530 --> 00:46:37.690 align:middle line:84% I feel like I know what's going on, right? 00:46:37.690 --> 00:46:40.370 align:middle line:84% I really like looking at it this way. 00:46:40.370 --> 00:46:40.870 align:middle line:90% OK? 00:46:40.870 --> 00:46:44.290 align:middle line:84% So after all, if I were doing calculus 00:46:44.290 --> 00:46:46.900 align:middle line:84% on a computer for scalars, I would just 00:46:46.900 --> 00:46:49.493 align:middle line:84% take X plus dx-squared for scalars. 00:46:49.493 --> 00:46:50.660 align:middle line:90% And this is what I would do. 00:46:50.660 --> 00:46:51.940 align:middle line:90% And I would get 2X. 00:46:51.940 --> 00:46:54.850 align:middle line:84% And in fact, just to remind you that this really 00:46:54.850 --> 00:47:02.650 align:middle line:84% is the same thing, we could take X to be 3 and dx to be 0.001. 00:47:02.650 --> 00:47:03.150 align:middle line:90% OK? 00:47:03.150 --> 00:47:08.340 align:middle line:90% And then we can go like this. 00:47:08.340 --> 00:47:11.550 align:middle line:84% And I can compare that with what everybody 00:47:11.550 --> 00:47:16.335 align:middle line:84% knows from ordinary calculus, 2X times dx. 00:47:16.335 --> 00:47:16.835 align:middle line:90% Right? 00:47:16.835 --> 00:47:19.770 align:middle line:84% And to the right number of digits, they match. 00:47:19.770 --> 00:47:20.270 align:middle line:90% OK? 00:47:20.270 --> 00:47:22.010 align:middle line:90% Any questions about this? 00:47:22.010 --> 00:47:22.510 align:middle line:90% All right. 00:47:22.510 --> 00:47:25.150 align:middle line:84% So hopefully you now kind of believe at least 00:47:25.150 --> 00:47:30.440 align:middle line:84% s not theoretically, that matrix calculus kind of works. 00:47:30.440 --> 00:47:30.940 align:middle line:90% All right. 00:47:30.940 --> 00:47:36.010 align:middle line:84% So getting back to the slides here-- 00:47:36.010 --> 00:47:40.260 align:middle line:84% so it's helpful to know certain rules that you all 00:47:40.260 --> 00:47:43.990 align:middle line:84% learned in freshman calculus extend to matrices. 00:47:43.990 --> 00:47:47.190 align:middle line:84% So you might remember that you learned the product 00:47:47.190 --> 00:47:48.310 align:middle line:90% rule, right? 00:47:48.310 --> 00:47:50.130 align:middle line:90% So what's the product rule say? 00:47:50.130 --> 00:47:54.000 align:middle line:84% That-- people learn it with different notations. 00:47:54.000 --> 00:48:02.430 align:middle line:84% Many people learn it as duv equals udv plus vdu, right? 00:48:02.430 --> 00:48:05.190 align:middle line:84% Other people learn the product rule with other notations. 00:48:05.190 --> 00:48:06.900 align:middle line:84% But the thing that I want you to know 00:48:06.900 --> 00:48:10.830 align:middle line:84% is that the product rule just works for matrices and vectors 00:48:10.830 --> 00:48:13.140 align:middle line:90% as well as for scalars. 00:48:13.140 --> 00:48:17.160 align:middle line:84% So any time you have a compatible vector product or-- 00:48:17.160 --> 00:48:19.650 align:middle line:84% like a vector dot product or a matrix times vector, 00:48:19.650 --> 00:48:24.720 align:middle line:84% where the sizes all work out, then it's 00:48:24.720 --> 00:48:30.580 align:middle line:84% perfectly OK to take the matrix rule dab to be dab plus adb. 00:48:30.580 --> 00:48:31.080 align:middle line:90% OK? 00:48:31.080 --> 00:48:33.270 align:middle line:84% So this would-- the only thing that you 00:48:33.270 --> 00:48:36.470 align:middle line:84% have to remember is matrices don't commute. 00:48:36.470 --> 00:48:40.450 align:middle line:84% So if a is on the left and b is on the right, a must always-- 00:48:40.450 --> 00:48:43.150 align:middle line:84% so in all the formats, a must be on the left. 00:48:43.150 --> 00:48:44.950 align:middle line:84% And b must always be on the right. 00:48:44.950 --> 00:48:48.760 align:middle line:90% Like, if-- right? 00:48:48.760 --> 00:48:53.590 align:middle line:84% So for example, this was a little bit 00:48:53.590 --> 00:48:55.690 align:middle line:90% giving you the wrong idea. 00:48:55.690 --> 00:48:59.230 align:middle line:84% I mean, it might have been better to say udv plus duv, 00:48:59.230 --> 00:49:03.640 align:middle line:84% you see, because this will work for matrices, right? 00:49:03.640 --> 00:49:06.940 align:middle line:84% I must always keep the u on the left and the v on the right. 00:49:06.940 --> 00:49:09.190 align:middle line:84% And that's the only rule that makes 00:49:09.190 --> 00:49:12.470 align:middle line:84% matrices just a little bit different from scalars. 00:49:12.470 --> 00:49:14.560 align:middle line:90% OK? 00:49:14.560 --> 00:49:18.550 align:middle line:84% So let's apply that to x transpose x, right? 00:49:18.550 --> 00:49:20.770 align:middle line:84% And so if we apply it to x transpose x, 00:49:20.770 --> 00:49:25.150 align:middle line:84% it says that d x transpose x will be dx transpose x plus x 00:49:25.150 --> 00:49:26.750 align:middle line:90% transpose dx. 00:49:26.750 --> 00:49:27.250 align:middle line:90% OK? 00:49:27.250 --> 00:49:31.330 align:middle line:84% And here's where students get confused. 00:49:31.330 --> 00:49:35.440 align:middle line:84% I'm going to go ahead, and combine them, and violate 00:49:35.440 --> 00:49:36.700 align:middle line:90% every rule that I just said. 00:49:36.700 --> 00:49:39.010 align:middle line:84% I'm going to say that I can combine them and get 00:49:39.010 --> 00:49:41.230 align:middle line:90% 2x transpose dx. 00:49:41.230 --> 00:49:42.070 align:middle line:90% Wait a minute. 00:49:42.070 --> 00:49:45.283 align:middle line:84% What are-- how is it that I can combine this here 00:49:45.283 --> 00:49:47.200 align:middle line:84% when I just told you that you can't put things 00:49:47.200 --> 00:49:48.250 align:middle line:90% in other orders? 00:49:48.250 --> 00:49:50.660 align:middle line:84% Anybody want to tell me why, in this case, 00:49:50.660 --> 00:49:52.720 align:middle line:90% I can get away with it? 00:49:52.720 --> 00:49:54.670 align:middle line:84% I just said to you the a must be on the left 00:49:54.670 --> 00:49:55.962 align:middle line:90% and the b must be on the right. 00:49:55.962 --> 00:49:56.950 align:middle line:90% And here I am. 00:49:56.950 --> 00:50:00.490 align:middle line:84% I'm telling you that dx transpose x and x transpose dx 00:50:00.490 --> 00:50:02.660 align:middle line:90% are actually the same thing. 00:50:02.660 --> 00:50:07.790 align:middle line:84% How come I'm allowed to do that in this one case 00:50:07.790 --> 00:50:08.870 align:middle line:90% and combine these two? 00:50:08.870 --> 00:50:10.190 align:middle line:90% Can anybody tell me? 00:50:10.190 --> 00:50:12.070 align:middle line:90% Yes, please, in the back. 00:50:12.070 --> 00:50:13.570 align:middle line:84% AUDIENCE: A dot product is a scalar. 00:50:13.570 --> 00:50:15.070 align:middle line:84% ALAN EDELMAN: Yeah, the dot product is just a scalar. 00:50:15.070 --> 00:50:16.987 align:middle line:84% And it doesn't matter what order you do it in. 00:50:16.987 --> 00:50:18.040 align:middle line:90% Exactly right. 00:50:18.040 --> 00:50:19.780 align:middle line:84% Just to kind of show you in Julia-- 00:50:19.780 --> 00:50:20.590 align:middle line:90% oh. 00:50:20.590 --> 00:50:21.340 align:middle line:90% Oh, you know what? 00:50:21.340 --> 00:50:25.570 align:middle line:84% [CHUCKLES] There's a tiny little hole where the wire comes out. 00:50:25.570 --> 00:50:29.920 align:middle line:84% And if the chair leg falls into that hole, I go down with it. 00:50:29.920 --> 00:50:30.590 align:middle line:90% [LAUGHS] 00:50:30.590 --> 00:50:31.090 align:middle line:90% All right. 00:50:31.090 --> 00:50:32.240 align:middle line:90% So here we go. 00:50:32.240 --> 00:50:37.070 align:middle line:90% So let's say X is "1 2 3." 00:50:37.070 --> 00:50:40.440 align:middle line:84% Oh, well, actually, let's put it with commas. 00:50:40.440 --> 00:50:40.940 align:middle line:90% OK? 00:50:40.940 --> 00:50:48.060 align:middle line:84% And if dx is 0.001 times rand of three-- 00:50:48.060 --> 00:50:50.610 align:middle line:84% see, I think if I did it this way, you'd all believe it. 00:50:50.610 --> 00:50:53.070 align:middle line:90% If I go-- can I do dot in Base? 00:50:53.070 --> 00:50:53.730 align:middle line:90% Let's see. 00:50:53.730 --> 00:50:56.212 align:middle line:90% Maybe, maybe not. 00:50:56.212 --> 00:50:57.020 align:middle line:90% "not defined." 00:50:57.020 --> 00:50:57.680 align:middle line:90% OK. 00:50:57.680 --> 00:50:59.680 align:middle line:84% But anyway, I don't even want to do it that way. 00:50:59.680 --> 00:51:03.860 align:middle line:84% Let's go x transpose dx, OK, and dx transpose x. 00:51:03.860 --> 00:51:05.850 align:middle line:84% You see they're exactly the same thing. 00:51:05.850 --> 00:51:09.510 align:middle line:84% After all, the dot product of the-- 00:51:09.510 --> 00:51:10.010 align:middle line:90% right? 00:51:10.010 --> 00:51:16.700 align:middle line:84% So my point is the dot product of x and dx 00:51:16.700 --> 00:51:21.950 align:middle line:84% is, of course, the same as dx and x. 00:51:21.950 --> 00:51:22.864 align:middle line:90% OK? 00:51:22.864 --> 00:51:30.660 align:middle line:84% So just to remind you, a times b is not b times a. 00:51:30.660 --> 00:51:35.490 align:middle line:84% But x transpose y equals y transpose 00:51:35.490 --> 00:51:39.230 align:middle line:90% x if x and y are vectors. 00:51:39.230 --> 00:51:40.018 align:middle line:90% Understood? 00:51:40.018 --> 00:51:42.560 align:middle line:84% You understand the difference between the general case, where 00:51:42.560 --> 00:51:44.690 align:middle line:84% things don't commute, and the dot product, 00:51:44.690 --> 00:51:46.120 align:middle line:90% where things do commute? 00:51:46.120 --> 00:51:46.620 align:middle line:90% OK? 00:51:46.620 --> 00:51:49.620 align:middle line:84% So students seem to get confused by that sometimes. 00:51:49.620 --> 00:51:52.770 align:middle line:84% And so I just wanted to say that as clearly as I know how. 00:51:52.770 --> 00:51:53.270 align:middle line:90% OK? 00:51:53.270 --> 00:51:57.800 align:middle line:84% So one more time, using the product rule on dx 00:51:57.800 --> 00:52:00.200 align:middle line:90% transpose x-- 00:52:00.200 --> 00:52:02.660 align:middle line:84% you can use this approach to write it as-- 00:52:02.660 --> 00:52:04.620 align:middle line:90% you use the product rule. 00:52:04.620 --> 00:52:05.120 align:middle line:90% OK? 00:52:05.120 --> 00:52:06.853 align:middle line:84% And then you recognize that those two dot 00:52:06.853 --> 00:52:07.770 align:middle line:90% products are the same. 00:52:07.770 --> 00:52:11.460 align:middle line:84% And so you can combine them into 2X transpose dx. 00:52:11.460 --> 00:52:11.960 align:middle line:90% OK? 00:52:11.960 --> 00:52:14.690 align:middle line:90% And so in a way-- 00:52:14.690 --> 00:52:17.720 align:middle line:84% now, of course, you could do it with-- 00:52:17.720 --> 00:52:19.730 align:middle line:84% to take the gradient of x transpose 00:52:19.730 --> 00:52:22.520 align:middle line:84% x is not hard to do the old-fashioned way. 00:52:22.520 --> 00:52:25.040 align:middle line:84% And there's no-- you could decide for yourselves. 00:52:25.040 --> 00:52:29.870 align:middle line:84% But on the blackboard, let me just quickly remind you that-- 00:52:29.870 --> 00:52:31.820 align:middle line:84% here's the old-fashioned way to do it. 00:52:31.820 --> 00:52:33.350 align:middle line:84% The old fashioned way is to say f 00:52:33.350 --> 00:52:38.660 align:middle line:84% of x equals x transpose x is the sum of the xi-squared. 00:52:38.660 --> 00:52:40.410 align:middle line:84% And so this is like the baby way to do it, 00:52:40.410 --> 00:52:41.610 align:middle line:84% but this is the sort of thing that you'll 00:52:41.610 --> 00:52:43.350 align:middle line:90% see in a lot of classes, right? 00:52:43.350 --> 00:52:47.460 align:middle line:84% And you'll learn that the gradient of f 00:52:47.460 --> 00:52:54.000 align:middle line:84% is the vector that has df dx1 all the way down to df dxn. 00:52:54.000 --> 00:52:56.670 align:middle line:90% And so you'll look f of xi. 00:52:56.670 --> 00:53:01.980 align:middle line:84% And you'll say, oh, df of dxi is equal to 2xi. 00:53:01.980 --> 00:53:03.960 align:middle line:84% And so you'll put them all in a vector. 00:53:03.960 --> 00:53:07.708 align:middle line:84% And you will say, ha, the gradient of f is equal to 2x, 00:53:07.708 --> 00:53:08.208 align:middle line:90% right? 00:53:08.208 --> 00:53:10.470 align:middle line:84% And there's nothing wrong with that. 00:53:10.470 --> 00:53:13.450 align:middle line:84% I mean, you get good at this sort of thing. 00:53:13.450 --> 00:53:16.620 align:middle line:84% But I hope you could sort of see the advantage of doing it 00:53:16.620 --> 00:53:17.580 align:middle line:90% holistically. 00:53:17.580 --> 00:53:19.050 align:middle line:90% I don't have to use indices. 00:53:19.050 --> 00:53:21.870 align:middle line:84% I don't have to think about what this thing is made of. 00:53:21.870 --> 00:53:23.340 align:middle line:84% I just do this little trick here. 00:53:23.340 --> 00:53:25.260 align:middle line:84% And boom, I get the answer right away, right? 00:53:25.260 --> 00:53:27.990 align:middle line:84% It's kind of like a magic trick for vectors and matrices, 00:53:27.990 --> 00:53:30.030 align:middle line:84% that-- like, if you take courses where people 00:53:30.030 --> 00:53:33.453 align:middle line:84% are taking gradients, all the other students in the class 00:53:33.453 --> 00:53:34.620 align:middle line:90% are going to do it this way. 00:53:34.620 --> 00:53:36.030 align:middle line:84% And you're going to know the secret trick 00:53:36.030 --> 00:53:37.620 align:middle line:84% of being able to get the gradient this way, right? 00:53:37.620 --> 00:53:39.990 align:middle line:84% So you'll have super powers that they don't have. 00:53:39.990 --> 00:53:40.490 align:middle line:90% OK. 00:53:40.490 --> 00:53:43.670 align:middle line:90% So of course, both ways work. 00:53:43.670 --> 00:53:46.170 align:middle line:90% But OK. 00:53:46.170 --> 00:53:46.670 align:middle line:90% OK. 00:53:46.670 --> 00:53:47.570 align:middle line:90% So here, let's see. 00:53:47.570 --> 00:53:49.355 align:middle line:90% We have du transpose v is-- 00:53:49.355 --> 00:53:52.630 align:middle line:90% 00:53:52.630 --> 00:53:55.500 align:middle line:84% this is sort of the point that I was making already. 00:53:55.500 --> 00:53:56.610 align:middle line:90% So I think we have that. 00:53:56.610 --> 00:53:57.110 align:middle line:90% OK? 00:53:57.110 --> 00:53:59.030 align:middle line:90% Any questions? 00:53:59.030 --> 00:53:59.530 align:middle line:90% All right. 00:53:59.530 --> 00:54:03.240 align:middle line:90% 00:54:03.240 --> 00:54:03.750 align:middle line:90% OK. 00:54:03.750 --> 00:54:09.360 align:middle line:84% So let's-- just, again, a little warm up. 00:54:09.360 --> 00:54:16.500 align:middle line:84% And so sometimes it's good to count how many parameters are 00:54:16.500 --> 00:54:18.540 align:middle line:90% needed to express an answer. 00:54:18.540 --> 00:54:20.430 align:middle line:84% "Parameters" is sort of a vague term. 00:54:20.430 --> 00:54:22.105 align:middle line:84% But think about it as, how many numbers 00:54:22.105 --> 00:54:24.480 align:middle line:84% do I need if I'm going to put something in a matrix-- how 00:54:24.480 --> 00:54:25.330 align:middle line:90% many elements. 00:54:25.330 --> 00:54:28.950 align:middle line:84% So let me ask you quickly, if I have a function that 00:54:28.950 --> 00:54:31.740 align:middle line:84% takes n inputs to m outputs and I 00:54:31.740 --> 00:54:34.380 align:middle line:84% do want to express the derivative as a matrix, 00:54:34.380 --> 00:54:35.850 align:middle line:90% how many numbers do I need? 00:54:35.850 --> 00:54:38.960 align:middle line:90% 00:54:38.960 --> 00:54:41.320 align:middle line:84% So n inputs, m outputs-- how many 00:54:41.320 --> 00:54:43.797 align:middle line:84% numbers do I need to express it as a matrix? 00:54:43.797 --> 00:54:44.630 align:middle line:90% AUDIENCE: m times n. 00:54:44.630 --> 00:54:45.410 align:middle line:90% ALAN EDELMAN: m times n. 00:54:45.410 --> 00:54:45.650 align:middle line:90% All right. 00:54:45.650 --> 00:54:46.460 align:middle line:90% Everybody knows. 00:54:46.460 --> 00:54:48.110 align:middle line:90% OK. 00:54:48.110 --> 00:54:50.372 align:middle line:84% And I think this is my last slide. 00:54:50.372 --> 00:54:52.580 align:middle line:84% So I'll just say a few words about second derivatives 00:54:52.580 --> 00:54:53.690 align:middle line:90% quickly. 00:54:53.690 --> 00:54:57.080 align:middle line:84% And then maybe we take, like, a five-minute break or something. 00:54:57.080 --> 00:55:00.410 align:middle line:90% And then Steven will speak. 00:55:00.410 --> 00:55:03.770 align:middle line:84% So I don't know how much we're going 00:55:03.770 --> 00:55:06.650 align:middle line:84% to talk about second derivatives in this class. 00:55:06.650 --> 00:55:09.650 align:middle line:84% But there's one example that comes up so often that it's 00:55:09.650 --> 00:55:11.810 align:middle line:90% probably just worth mentioning. 00:55:11.810 --> 00:55:14.510 align:middle line:90% And in a way, it-- 00:55:14.510 --> 00:55:18.140 align:middle line:84% I think when I first learned about differentiating vectors, 00:55:18.140 --> 00:55:21.530 align:middle line:84% I always got confused between the Hessian, which 00:55:21.530 --> 00:55:22.440 align:middle line:90% is a matrix-- 00:55:22.440 --> 00:55:23.870 align:middle line:84% it's a second derivative matrix-- 00:55:23.870 --> 00:55:26.480 align:middle line:84% and the Jacobian, which is also a matrix, 00:55:26.480 --> 00:55:28.640 align:middle line:84% but it's a first derivative matrix, right? 00:55:28.640 --> 00:55:36.750 align:middle line:90% So let me be clear. 00:55:36.750 --> 00:55:39.260 align:middle line:84% If you have a function from rm to rn, 00:55:39.260 --> 00:55:41.600 align:middle line:84% you can express the answer as an m-by-n matrix. 00:55:41.600 --> 00:55:44.102 align:middle line:84% And that's what we call the Jacobian, right? 00:55:44.102 --> 00:55:45.560 align:middle line:84% So the big point I'm just saying is 00:55:45.560 --> 00:55:47.810 align:middle line:84% it's a function from vectors to vectors. 00:55:47.810 --> 00:55:50.030 align:middle line:84% The other case that comes up very 00:55:50.030 --> 00:55:57.240 align:middle line:84% often is if you have a function from vectors to scalars. 00:55:57.240 --> 00:55:57.740 align:middle line:90% OK? 00:55:57.740 --> 00:56:04.930 align:middle line:84% And so examples like x transpose ax 00:56:04.930 --> 00:56:08.080 align:middle line:84% would be vector transpose matrix times vector 00:56:08.080 --> 00:56:09.370 align:middle line:90% as a function of the vector. 00:56:09.370 --> 00:56:09.870 align:middle line:90% OK? 00:56:09.870 --> 00:56:13.300 align:middle line:90% And so now the-- 00:56:13.300 --> 00:56:15.820 align:middle line:84% so if it's going from vector to scalar, 00:56:15.820 --> 00:56:18.490 align:middle line:84% remember that the gradient is a vector. 00:56:18.490 --> 00:56:19.540 align:middle line:90% It's one-dimensional. 00:56:19.540 --> 00:56:22.660 align:middle line:84% But the second derivative is often expressed as a matrix. 00:56:22.660 --> 00:56:25.330 align:middle line:84% And that's what we call the Hessian. 00:56:25.330 --> 00:56:28.230 align:middle line:84% So you might ask yourself, if we're 00:56:28.230 --> 00:56:31.050 align:middle line:84% trying to get away from element-wise matrix 00:56:31.050 --> 00:56:33.390 align:middle line:84% representations, what's sort of the abstraction 00:56:33.390 --> 00:56:36.480 align:middle line:84% that we will need to talk about second derivatives? 00:56:36.480 --> 00:56:38.850 align:middle line:84% And the answer in advanced linear algebra classes 00:56:38.850 --> 00:56:40.480 align:middle line:90% is called a quadratic form. 00:56:40.480 --> 00:56:40.980 align:middle line:90% OK? 00:56:40.980 --> 00:56:42.105 align:middle line:90% And we'll see what that's-- 00:56:42.105 --> 00:56:46.740 align:middle line:84% So we can get past the clunkiness of matrices, 00:56:46.740 --> 00:56:49.560 align:middle line:84% like 18.06, and get into the more abstraction. 00:56:49.560 --> 00:56:51.750 align:middle line:84% And those are the quadratic forms. 00:56:51.750 --> 00:56:52.410 align:middle line:90% OK. 00:56:52.410 --> 00:56:57.160 align:middle line:84% So I think that's the end of what I'm going to say today. 00:56:57.160 --> 00:56:58.978 align:middle line:90% Last chance for questions. 00:56:58.978 --> 00:57:00.270 align:middle line:90% Otherwise, what should we take? 00:57:00.270 --> 00:57:01.740 align:middle line:84% Should we take a five-minute break, Steven? 00:57:01.740 --> 00:57:02.550 align:middle line:90% What do you think-- 00:57:02.550 --> 00:57:05.398 align:middle line:90% or a little less? 00:57:05.398 --> 00:57:07.190 align:middle line:84% STEVEN JOHNSON: Sure, five minutes is good. 00:57:07.190 --> 00:57:07.910 align:middle line:84% ALAN EDELMAN: Five minutes is good? 00:57:07.910 --> 00:57:08.410 align:middle line:90% All right. 00:57:08.410 --> 00:57:11.010 align:middle line:84% So I've got 12:03 on the clock back here. 00:57:11.010 --> 00:57:13.850 align:middle line:84% So let's just take a five-minute break-- until 12:08. 00:57:13.850 --> 00:57:16.790 align:middle line:84% And people can get some water, or ask me some questions, 00:57:16.790 --> 00:57:19.530 align:middle line:84% or read their email, or whatever you do in five minutes. 00:57:19.530 --> 00:57:20.030 align:middle line:90% All right? 00:57:20.030 --> 00:57:25.000 align:middle line:84% And then, Steven, you can grab the screen and set yourself up. 00:57:25.000 --> 00:57:41.000 align:middle line:90%