WEBVTT 00:00:00.000 --> 00:00:01.992 align:middle line:90% [SQUEAKING] 00:00:01.992 --> 00:00:04.482 align:middle line:90% [RUSTLING] 00:00:04.482 --> 00:00:05.976 align:middle line:90% [CLICKING] 00:00:05.976 --> 00:00:10.533 align:middle line:90% 00:00:10.533 --> 00:00:11.450 align:middle line:90% STEVEN G. JOHNSON: OK. 00:00:11.450 --> 00:00:12.810 align:middle line:90% So I think we can get started. 00:00:12.810 --> 00:00:15.620 align:middle line:84% So the next part, I'm going to show some slides, so talking 00:00:15.620 --> 00:00:17.450 align:middle line:90% about some applications. 00:00:17.450 --> 00:00:20.330 align:middle line:84% As usual, these slides will be posted later today 00:00:20.330 --> 00:00:21.800 align:middle line:90% to the GitHub page. 00:00:21.800 --> 00:00:26.360 align:middle line:84% And also the handwritten notes also from the first part 00:00:26.360 --> 00:00:28.040 align:middle line:90% will also be posted. 00:00:28.040 --> 00:00:31.260 align:middle line:90% They're probably already posted. 00:00:31.260 --> 00:00:36.085 align:middle line:84% So now I want to talk about some-- 00:00:36.085 --> 00:00:37.460 align:middle line:84% remind you of why we're computing 00:00:37.460 --> 00:00:38.850 align:middle line:90% derivatives in the first place. 00:00:38.850 --> 00:00:40.940 align:middle line:84% And so we talked about some things 00:00:40.940 --> 00:00:43.040 align:middle line:90% at a high level in lecture one. 00:00:43.040 --> 00:00:45.200 align:middle line:84% I want to just drill down a little bit 00:00:45.200 --> 00:00:50.450 align:middle line:84% and then also talk more about computation of derivatives. 00:00:50.450 --> 00:00:53.510 align:middle line:90% 00:00:53.510 --> 00:00:57.800 align:middle line:84% So one basic use of derivatives is 00:00:57.800 --> 00:00:59.990 align:middle line:90% to solve a nonlinear equations. 00:00:59.990 --> 00:01:02.780 align:middle line:84% And you all learned this probably in first semester 00:01:02.780 --> 00:01:04.390 align:middle line:90% calculus if you have a-- 00:01:04.390 --> 00:01:07.640 align:middle line:84% a scalar function f of x that takes a scalar in and gives 00:01:07.640 --> 00:01:08.780 align:middle line:90% a scalar out. 00:01:08.780 --> 00:01:11.760 align:middle line:90% And that's some nonlinear-- 00:01:11.760 --> 00:01:14.190 align:middle line:84% if it's linear obviously, you can solve it. 00:01:14.190 --> 00:01:16.302 align:middle line:84% If it's a quadratic, you have a quadratic formula. 00:01:16.302 --> 00:01:17.760 align:middle line:84% But if it's some arbitrary function 00:01:17.760 --> 00:01:21.030 align:middle line:84% like sine of cosine of x or something 00:01:21.030 --> 00:01:23.640 align:middle line:84% like that, like some complicated thing, 00:01:23.640 --> 00:01:26.190 align:middle line:84% you might not be able to write a closed-form solution. 00:01:26.190 --> 00:01:30.990 align:middle line:84% But there's a nice way to get the solution approximately 00:01:30.990 --> 00:01:33.240 align:middle line:84% to any accuracy you want, as many digits 00:01:33.240 --> 00:01:35.550 align:middle line:84% as you want very quickly if you know 00:01:35.550 --> 00:01:40.050 align:middle line:84% at least roughly where the root is, and that's Newton's method. 00:01:40.050 --> 00:01:43.650 align:middle line:84% And what it really is it's really a linear algebra 00:01:43.650 --> 00:01:44.310 align:middle line:90% technique. 00:01:44.310 --> 00:01:49.500 align:middle line:84% It's really taking the function, approximating it by a line, 00:01:49.500 --> 00:01:51.130 align:middle line:90% by linear function. 00:01:51.130 --> 00:01:53.380 align:middle line:84% And then once it's linear, it's easy to find the root, 00:01:53.380 --> 00:01:55.050 align:middle line:84% and that gives you an approximate root. 00:01:55.050 --> 00:01:56.400 align:middle line:90% And you use that as a guess. 00:01:56.400 --> 00:01:59.500 align:middle line:90% So that's the three steps here. 00:01:59.500 --> 00:02:01.530 align:middle line:84% So if you're solving f of x equals 0-- 00:02:01.530 --> 00:02:04.460 align:middle line:84% so f of x is this blue curve on the right-- 00:02:04.460 --> 00:02:06.540 align:middle line:84% then you linearize with your derivatives. 00:02:06.540 --> 00:02:12.410 align:middle line:84% So f of x plus dx is approximate-- for delta x, 00:02:12.410 --> 00:02:15.620 align:middle line:84% and it is not infinitesimal-- is approximately f 00:02:15.620 --> 00:02:18.860 align:middle line:90% of x plus f prime of x delta x. 00:02:18.860 --> 00:02:21.340 align:middle line:90% That's what the derivative is. 00:02:21.340 --> 00:02:24.070 align:middle line:84% And then if we treat the equation 00:02:24.070 --> 00:02:27.940 align:middle line:84% on the right, the linear problem as an approximation 00:02:27.940 --> 00:02:30.140 align:middle line:84% for a real problem, that one's easy to solve. 00:02:30.140 --> 00:02:33.250 align:middle line:84% So we can set that linear thing equal to 0. 00:02:33.250 --> 00:02:39.160 align:middle line:84% Solve for delta x, and it's just minus f over f prime. 00:02:39.160 --> 00:02:41.290 align:middle line:90% And so that gives us a new x. 00:02:41.290 --> 00:02:43.990 align:middle line:84% So that's our guess for the solution 00:02:43.990 --> 00:02:46.570 align:middle line:84% is x plus delta x, which is the same thing as x 00:02:46.570 --> 00:02:49.560 align:middle line:90% minus f divided by f prime. 00:02:49.560 --> 00:02:51.330 align:middle line:84% Of course, it's not an exact route 00:02:51.330 --> 00:02:56.220 align:middle line:84% because this linear was only a linear approximation. 00:02:56.220 --> 00:02:58.440 align:middle line:84% But it gets us closer to the root. 00:02:58.440 --> 00:03:01.410 align:middle line:84% So on the right is a nice animation from Wikipedia 00:03:01.410 --> 00:03:05.220 align:middle line:84% that shows this process, starting from a few x's where 00:03:05.220 --> 00:03:06.630 align:middle line:90% you start with an x. 00:03:06.630 --> 00:03:08.820 align:middle line:84% What you're doing is approximating a function 00:03:08.820 --> 00:03:09.450 align:middle line:90% by its tangent. 00:03:09.450 --> 00:03:11.970 align:middle line:84% That's the red line, and that gives you the next x. 00:03:11.970 --> 00:03:13.470 align:middle line:90% And so you start with this x. 00:03:13.470 --> 00:03:16.290 align:middle line:84% And the x1 on the right gives you the x2. 00:03:16.290 --> 00:03:17.850 align:middle line:90% And then you linearize it. 00:03:17.850 --> 00:03:23.940 align:middle line:84% You get another x, x3, and then another x, x4. 00:03:23.940 --> 00:03:28.600 align:middle line:84% And then very, very quickly, once you start getting close, 00:03:28.600 --> 00:03:31.210 align:middle line:90% this converges extremely fast. 00:03:31.210 --> 00:03:33.760 align:middle line:84% If you're far away, it can be a little unpredictable. 00:03:33.760 --> 00:03:37.030 align:middle line:84% But once you're close, it turns out-- it's actually amazing. 00:03:37.030 --> 00:03:39.560 align:middle line:84% It doubles the number of decimal places in every step. 00:03:39.560 --> 00:03:42.290 align:middle line:84% So if you have it to one digit, on the next step, 00:03:42.290 --> 00:03:44.350 align:middle line:84% you'll have it to two digits, then the next step 00:03:44.350 --> 00:03:47.020 align:middle line:84% four, then 8, and then 16, and then 00:03:47.020 --> 00:03:50.840 align:middle line:84% you run out of digits on your computer very quickly. 00:03:50.840 --> 00:03:54.160 align:middle line:84% So it's really, really a great algorithm. 00:03:54.160 --> 00:03:58.570 align:middle line:84% It goes back thousands of years, the Newton's algorithm. 00:03:58.570 --> 00:04:02.620 align:middle line:84% But you can see that as soon as we write it this way, 00:04:02.620 --> 00:04:06.070 align:middle line:84% then we should be able to see that the same thing works 00:04:06.070 --> 00:04:07.787 align:middle line:90% for multi-dimensional functions. 00:04:07.787 --> 00:04:09.745 align:middle line:84% So if we have a function that takes a vector in 00:04:09.745 --> 00:04:11.210 align:middle line:90% and a vector out-- 00:04:11.210 --> 00:04:14.140 align:middle line:90% so suppose x is a vector in rn. 00:04:14.140 --> 00:04:16.810 align:middle line:84% I don't know why that comes out in that weird font here. 00:04:16.810 --> 00:04:21.267 align:middle line:84% So you have n equations and n variables. 00:04:21.267 --> 00:04:22.100 align:middle line:90% So that's important. 00:04:22.100 --> 00:04:24.110 align:middle line:84% You have to have the same number of equations and variables. 00:04:24.110 --> 00:04:26.480 align:middle line:84% But now they're nonlinear for some arbitrary nonlinear 00:04:26.480 --> 00:04:29.750 align:middle line:84% function, and you're trying to find a root. 00:04:29.750 --> 00:04:33.740 align:middle line:84% And then you do exactly the same thing. 00:04:33.740 --> 00:04:37.820 align:middle line:84% You approximate f of by a linear function. f of x plus x 00:04:37.820 --> 00:04:41.690 align:middle line:84% is approximately f of x plus f prime of x delta x. 00:04:41.690 --> 00:04:43.940 align:middle line:84% And that's what we've been seeing over and over again. 00:04:43.940 --> 00:04:46.580 align:middle line:84% This is the definition of f prime. 00:04:46.580 --> 00:04:48.500 align:middle line:90% Is that linearization? 00:04:48.500 --> 00:04:51.440 align:middle line:84% Oops, and this always happens when 00:04:51.440 --> 00:04:54.650 align:middle line:90% I display it on the computer. 00:04:54.650 --> 00:04:57.800 align:middle line:90% Let me see if I can-- 00:04:57.800 --> 00:04:59.982 align:middle line:84% and for some reason, this display 00:04:59.982 --> 00:05:01.190 align:middle line:90% gets screwed up a little bit. 00:05:01.190 --> 00:05:02.930 align:middle line:84% Hold on, how do I get out of this? 00:05:02.930 --> 00:05:05.460 align:middle line:90% 00:05:05.460 --> 00:05:05.960 align:middle line:90% Yeah. 00:05:05.960 --> 00:05:09.740 align:middle line:84% So it looks fine here in non-presentation mode. 00:05:09.740 --> 00:05:12.830 align:middle line:90% 00:05:12.830 --> 00:05:15.755 align:middle line:90% So what do we do? 00:05:15.755 --> 00:05:18.620 align:middle line:84% We approximate f by a linear thing, 00:05:18.620 --> 00:05:22.170 align:middle line:84% and now we know how to do that for any vector space. 00:05:22.170 --> 00:05:25.380 align:middle line:84% And then, for example, x is a column vector; 00:05:25.380 --> 00:05:28.700 align:middle line:84% then f prime is a Jacobian matrix. 00:05:28.700 --> 00:05:32.830 align:middle line:84% You set this linear approximation equal to 0, 00:05:32.830 --> 00:05:34.510 align:middle line:84% and you solve it for delta x, and that's 00:05:34.510 --> 00:05:35.570 align:middle line:90% just a linear equation. 00:05:35.570 --> 00:05:38.240 align:middle line:84% So that's just a 1806 thing, a linear algebra thing. 00:05:38.240 --> 00:05:40.595 align:middle line:90% So now delta x is-- 00:05:40.595 --> 00:05:42.970 align:middle line:84% you move f to the other side, and you multiply both sides 00:05:42.970 --> 00:05:44.260 align:middle line:90% by f prime inverse. 00:05:44.260 --> 00:05:47.800 align:middle line:84% Delta x is minus f prime inverse times f of x. 00:05:47.800 --> 00:05:50.770 align:middle line:84% So it's just the inverse of the Jacobian matrix times 00:05:50.770 --> 00:05:52.790 align:middle line:90% f with the minus sign. 00:05:52.790 --> 00:05:54.100 align:middle line:90% And so that's our new x. 00:05:54.100 --> 00:05:56.680 align:middle line:90% Our new x is this x-- 00:05:56.680 --> 00:05:58.460 align:middle line:84% the original x plus this delta x, 00:05:58.460 --> 00:06:00.970 align:middle line:84% which should give us a root if it were linear. 00:06:00.970 --> 00:06:02.350 align:middle line:90% So it's just f-- 00:06:02.350 --> 00:06:06.970 align:middle line:84% the new x is x minus inverse Jacobian times f. 00:06:06.970 --> 00:06:09.720 align:middle line:84% So you solve one linear equation with a Jacobian, 00:06:09.720 --> 00:06:11.250 align:middle line:90% and you get your new x. 00:06:11.250 --> 00:06:13.380 align:middle line:84% And of course, this is an approximate root, 00:06:13.380 --> 00:06:15.220 align:middle line:84% but you just keep repeating the process. 00:06:15.220 --> 00:06:20.160 align:middle line:84% It's much harder to picture this in higher dimensions, 00:06:20.160 --> 00:06:22.500 align:middle line:90% compared to the 1D case. 00:06:22.500 --> 00:06:25.230 align:middle line:90% But it works amazingly fast. 00:06:25.230 --> 00:06:26.220 align:middle line:90% It says the same thing. 00:06:26.220 --> 00:06:27.870 align:middle line:84% You solve a linear system in each step. 00:06:27.870 --> 00:06:30.635 align:middle line:84% It doubles the number of digits in each step. 00:06:30.635 --> 00:06:32.010 align:middle line:84% And with the caveat that you need 00:06:32.010 --> 00:06:34.080 align:middle line:84% a starting guess close enough to the root. 00:06:34.080 --> 00:06:35.880 align:middle line:84% If you start very far away from the root, 00:06:35.880 --> 00:06:39.610 align:middle line:84% it can be kind of unpredictable what this does. 00:06:39.610 --> 00:06:42.410 align:middle line:84% In fact, it can even make these amazing fractal patterns. 00:06:42.410 --> 00:06:44.160 align:middle line:84% So if you Google Newton fractal, there's 00:06:44.160 --> 00:06:47.190 align:middle line:84% a nice Wikipedia page on this that 00:06:47.190 --> 00:06:50.670 align:middle line:84% has lots of beautiful images of what Newton's method does 00:06:50.670 --> 00:06:53.250 align:middle line:84% if you start far away from the root in higher dimensions. 00:06:53.250 --> 00:06:56.010 align:middle line:84% It has this weird fractal pattern. 00:06:56.010 --> 00:07:00.210 align:middle line:84% So you have to have some rough idea where the solution is, 00:07:00.210 --> 00:07:04.760 align:middle line:84% but then you can polish it off super fast. 00:07:04.760 --> 00:07:06.150 align:middle line:90% So that's Newton's methond. 00:07:06.150 --> 00:07:09.560 align:middle line:84% So this is a great application for Jacobians 00:07:09.560 --> 00:07:13.880 align:middle line:84% and for derivatives in general of equations that take vectors 00:07:13.880 --> 00:07:16.620 align:middle line:90% in and give vectors out. 00:07:16.620 --> 00:07:20.510 align:middle line:84% And the other even more famous application I would say 00:07:20.510 --> 00:07:23.100 align:middle line:90% is optimization. 00:07:23.100 --> 00:07:26.220 align:middle line:84% So optimization, you have a scalar function. 00:07:26.220 --> 00:07:30.400 align:middle line:84% So you have a vector in and a scalar out, 00:07:30.400 --> 00:07:33.920 align:middle line:84% and you're trying to minimize it, for example, or maximize 00:07:33.920 --> 00:07:34.420 align:middle line:90% it. 00:07:34.420 --> 00:07:39.050 align:middle line:84% Maximizing, minimizing is just flipping the sign. 00:07:39.050 --> 00:07:42.830 align:middle line:84% So for example, machine learning, you have a fx. 00:07:42.830 --> 00:07:44.310 align:middle line:90% You have a big neural network. 00:07:44.310 --> 00:07:47.145 align:middle line:84% x Is a million parameters of your neural network. 00:07:47.145 --> 00:07:48.770 align:middle line:84% And the thing you're trying to minimize 00:07:48.770 --> 00:07:51.500 align:middle line:84% is the loss function that basically you 00:07:51.500 --> 00:07:55.130 align:middle line:84% evaluate your neural network, compare its output to what 00:07:55.130 --> 00:07:56.600 align:middle line:90% you want its output to be. 00:07:56.600 --> 00:07:59.952 align:middle line:84% And you take some norm of the difference, and that's your f. 00:07:59.952 --> 00:08:02.160 align:middle line:84% It's the error in the neural network for its function 00:08:02.160 --> 00:08:02.910 align:middle line:90% of the parameters. 00:08:02.910 --> 00:08:05.330 align:middle line:84% And you want to go downhill and make that error as 00:08:05.330 --> 00:08:06.660 align:middle line:90% small as possible. 00:08:06.660 --> 00:08:10.310 align:middle line:84% And so the key point is if you can take the gradient of this-- 00:08:10.310 --> 00:08:13.160 align:middle line:84% and now we have a general definition-- for any vector 00:08:13.160 --> 00:08:17.490 align:middle line:84% space, any scalar function, if we have a dot product, 00:08:17.490 --> 00:08:19.480 align:middle line:90% then we can define a gradient. 00:08:19.480 --> 00:08:24.430 align:middle line:84% And that gradient then points in the uphill direction, 00:08:24.430 --> 00:08:27.080 align:middle line:84% or minus gradient points in the downhill direction, 00:08:27.080 --> 00:08:29.320 align:middle line:84% which is called the steepest descent direction. 00:08:29.320 --> 00:08:34.030 align:middle line:90% And the most basic idea-- 00:08:34.030 --> 00:08:36.289 align:middle line:84% and there's many complications on top of this. 00:08:36.289 --> 00:08:39.070 align:middle line:84% But the most basic idea is you just go downhill. 00:08:39.070 --> 00:08:42.309 align:middle line:84% Imagine you're in the mountains in a fog. 00:08:42.309 --> 00:08:43.730 align:middle line:90% You can't see anything. 00:08:43.730 --> 00:08:46.120 align:middle line:84% But if you want to find the bottom of a valley, 00:08:46.120 --> 00:08:48.010 align:middle line:90% you just go downhill. 00:08:48.010 --> 00:08:50.890 align:middle line:84% You might not find the deepest valley. 00:08:50.890 --> 00:08:52.900 align:middle line:84% You're going to find the closest valley. 00:08:52.900 --> 00:08:54.817 align:middle line:84% There might be a deeper valley someplace else, 00:08:54.817 --> 00:08:57.610 align:middle line:90% but you'll find a local minimum. 00:08:57.610 --> 00:09:00.880 align:middle line:84% And even if you have a million parameters, 00:09:00.880 --> 00:09:03.460 align:middle line:84% as you have very commonly in neural networks, 00:09:03.460 --> 00:09:06.670 align:middle line:84% this allows you to evolve them all simultaneously 00:09:06.670 --> 00:09:08.380 align:middle line:90% in the downhill direction right. 00:09:08.380 --> 00:09:10.270 align:middle line:90% You want to update all-- 00:09:10.270 --> 00:09:11.930 align:middle line:84% in your million dimensional space, 00:09:11.930 --> 00:09:14.380 align:middle line:84% you want to go in this direction. 00:09:14.380 --> 00:09:17.290 align:middle line:84% You don't want to optimize each variable one at a time 00:09:17.290 --> 00:09:20.950 align:middle line:84% because that'll take forever if there are a million variables. 00:09:20.950 --> 00:09:23.230 align:middle line:84% And as we're going to talk about over and over again, 00:09:23.230 --> 00:09:25.930 align:middle line:84% that it turns out that when you have a scalar 00:09:25.930 --> 00:09:28.120 align:middle line:84% function of vector inputs, you can 00:09:28.120 --> 00:09:32.440 align:middle line:84% compute all million derivatives or all n derivatives, 00:09:32.440 --> 00:09:35.620 align:middle line:84% the gradient, with about the same cost as evaluating 00:09:35.620 --> 00:09:36.815 align:middle line:90% the function once. 00:09:36.815 --> 00:09:39.190 align:middle line:84% So evaluating the function, say, is a big neural network. 00:09:39.190 --> 00:09:41.770 align:middle line:84% You have to evaluate that once to get your loss function. 00:09:41.770 --> 00:09:43.395 align:middle line:84% It turns out you can get the derivative 00:09:43.395 --> 00:09:45.370 align:middle line:84% of that loss with respect to all million inputs 00:09:45.370 --> 00:09:49.570 align:middle line:84% with essentially the equivalent of one additional function 00:09:49.570 --> 00:09:50.390 align:middle line:90% evaluation. 00:09:50.390 --> 00:09:54.040 align:middle line:84% So for neural networks, this is called backpropagation. 00:09:54.040 --> 00:09:55.570 align:middle line:84% In automatic differentiation, this 00:09:55.570 --> 00:09:57.830 align:middle line:84% is called reverse mode differentiation. 00:09:57.830 --> 00:09:59.660 align:middle line:84% This is also called adjoint methods. 00:09:59.660 --> 00:10:01.900 align:middle line:84% And as I alluded to the other day, 00:10:01.900 --> 00:10:04.960 align:middle line:84% it's equivalent to just evaluating the chain rule 00:10:04.960 --> 00:10:07.330 align:middle line:84% and multiplying left to right so that you always 00:10:07.330 --> 00:10:10.210 align:middle line:84% multiply vectors times matrices and never matrices times 00:10:10.210 --> 00:10:11.380 align:middle line:90% matrices. 00:10:11.380 --> 00:10:13.930 align:middle line:84% And this is what makes large-scale optimization 00:10:13.930 --> 00:10:14.925 align:middle line:90% practical. 00:10:14.925 --> 00:10:17.050 align:middle line:84% So when you're optimizing over a million parameters 00:10:17.050 --> 00:10:18.758 align:middle line:84% in a neural network, or you're optimizing 00:10:18.758 --> 00:10:21.170 align:middle line:84% the shape of an airplane wing characterized 00:10:21.170 --> 00:10:24.410 align:middle line:84% by zillions of parameters and so forth, portfolio optimization 00:10:24.410 --> 00:10:26.750 align:middle line:90% and in finance and so forth. 00:10:26.750 --> 00:10:28.820 align:middle line:84% So there's a lot of complications in this 00:10:28.820 --> 00:10:30.500 align:middle line:84% that I'm not going to go through at all, 00:10:30.500 --> 00:10:33.440 align:middle line:84% but I just want to mention some of the complications 00:10:33.440 --> 00:10:35.060 align:middle line:84% that if you want to do this for real, 00:10:35.060 --> 00:10:38.300 align:middle line:84% that you would cover, to some extent, 00:10:38.300 --> 00:10:40.320 align:middle line:84% in a nonlinear optimization class. 00:10:40.320 --> 00:10:45.200 align:middle line:84% So the gradient points uphill minus gradient points downhill. 00:10:45.200 --> 00:10:47.960 align:middle line:84% So, great, that tells you what direction to go, 00:10:47.960 --> 00:10:50.620 align:middle line:84% but how far do you go in that direction? 00:10:50.620 --> 00:10:53.380 align:middle line:84% And usually, you want to take kind of as big a step 00:10:53.380 --> 00:10:57.580 align:middle line:84% as you can to go downhill as far as possible. 00:10:57.580 --> 00:10:59.410 align:middle line:84% But if you take too big of a step, 00:10:59.410 --> 00:11:02.620 align:middle line:84% the gradient is like a linear approximation. 00:11:02.620 --> 00:11:05.377 align:middle line:84% So that's only going to be valid for relatively small steps. 00:11:05.377 --> 00:11:07.210 align:middle line:84% If you take too big of a steps, the gradient 00:11:07.210 --> 00:11:15.800 align:middle line:84% is not going to be an indication of the downhill direction. 00:11:15.800 --> 00:11:18.733 align:middle line:84% So there's a lot of different strategies. 00:11:18.733 --> 00:11:19.900 align:middle line:90% One is you do a line search. 00:11:19.900 --> 00:11:22.930 align:middle line:84% You do 1D minimization in this direction. 00:11:22.930 --> 00:11:24.550 align:middle line:90% And you take a step. 00:11:24.550 --> 00:11:26.780 align:middle line:84% And the simpler thing is you take a big step. 00:11:26.780 --> 00:11:28.870 align:middle line:84% And if the function gets better, great. 00:11:28.870 --> 00:11:31.300 align:middle line:84% If the function does not get better, 00:11:31.300 --> 00:11:33.490 align:middle line:84% then you kind of backtrack along that direction. 00:11:33.490 --> 00:11:36.250 align:middle line:84% So you'd divide your step by two or some geometric factor, 00:11:36.250 --> 00:11:39.850 align:middle line:84% and you keep doing that until you find you're going downhill. 00:11:39.850 --> 00:11:42.590 align:middle line:84% Another thing is you have something called a trust 00:11:42.590 --> 00:11:43.090 align:middle line:90% region. 00:11:43.090 --> 00:11:46.720 align:middle line:84% You have some notion of how big a step you're allowed to take. 00:11:46.720 --> 00:11:48.921 align:middle line:90% And you take you-- 00:11:48.921 --> 00:11:51.950 align:middle line:84% you basically optimize it within this region 00:11:51.950 --> 00:11:53.820 align:middle line:90% using the linear approximation. 00:11:53.820 --> 00:11:55.880 align:middle line:84% And then the tricky parts are like, how 00:11:55.880 --> 00:11:59.060 align:middle line:84% do you decide how big the trust region is as you go along? 00:11:59.060 --> 00:12:00.913 align:middle line:84% You might want to grow it or shrink it. 00:12:00.913 --> 00:12:03.080 align:middle line:84% And it turns out this is really tricky to get right. 00:12:03.080 --> 00:12:06.050 align:middle line:84% It's really easy to write down an algorithm that 00:12:06.050 --> 00:12:08.450 align:middle line:84% looks reasonable, but actually kind of goes 00:12:08.450 --> 00:12:11.540 align:middle line:84% in circles forever and ever and never converges. 00:12:11.540 --> 00:12:12.620 align:middle line:90% So you have to-- 00:12:12.620 --> 00:12:14.720 align:middle line:84% and the history of nonlinear optimization 00:12:14.720 --> 00:12:17.790 align:middle line:84% is full of such algorithms that look reasonable, 00:12:17.790 --> 00:12:20.810 align:middle line:84% but it turns out, don't converge in general. 00:12:20.810 --> 00:12:23.060 align:middle line:84% They might converge sometimes, but then sometimes they 00:12:23.060 --> 00:12:25.790 align:middle line:84% might just loop around forever; or worse, they 00:12:25.790 --> 00:12:27.650 align:middle line:84% might seem to converge, but they're actually 00:12:27.650 --> 00:12:31.160 align:middle line:84% stopping at a point that's not a minimum. 00:12:31.160 --> 00:12:33.260 align:middle line:84% But there's lots of algorithms for this. 00:12:33.260 --> 00:12:36.470 align:middle line:84% Usually, most people doing nonlinear optimization 00:12:36.470 --> 00:12:38.540 align:middle line:84% do not write their own algorithms. 00:12:38.540 --> 00:12:40.490 align:middle line:90% They take one off the shelf. 00:12:40.490 --> 00:12:42.980 align:middle line:84% There's a lot of trade out-- and you try different ones. 00:12:42.980 --> 00:12:44.450 align:middle line:84% Each one, you provide a function, 00:12:44.450 --> 00:12:46.580 align:middle line:84% you provide the gradient, and you 00:12:46.580 --> 00:12:50.930 align:middle line:84% let it somehow decide what to do, and you try different ones 00:12:50.930 --> 00:12:53.060 align:middle line:84% and see which one works well for your problem. 00:12:53.060 --> 00:12:55.280 align:middle line:84% Another complication is that a lot of real problems, 00:12:55.280 --> 00:12:56.780 align:middle line:84% you're not just minimizing function. 00:12:56.780 --> 00:12:58.170 align:middle line:90% You also have constraints. 00:12:58.170 --> 00:13:00.450 align:middle line:84% So you have often a set of constraints. 00:13:00.450 --> 00:13:02.300 align:middle line:84% And the general form is you're minimizing 00:13:02.300 --> 00:13:05.270 align:middle line:84% a function f of x, subject to some set of constraints, 00:13:05.270 --> 00:13:09.590 align:middle line:84% say, of some functions gk that you want to stay negative; 00:13:09.590 --> 00:13:13.070 align:middle line:90% or feasible, is the terminology. 00:13:13.070 --> 00:13:15.190 align:middle line:84% So it's more complicated, but you still 00:13:15.190 --> 00:13:17.440 align:middle line:84% need the gradient of f to find the downhill direction. 00:13:17.440 --> 00:13:20.150 align:middle line:84% And turns out you still need the gradient of your constraint 00:13:20.150 --> 00:13:20.930 align:middle line:90% functions. 00:13:20.930 --> 00:13:23.360 align:middle line:84% To approximate the bilinear things 00:13:23.360 --> 00:13:26.820 align:middle line:84% that are easy to understand and easy to take steps. 00:13:26.820 --> 00:13:32.180 align:middle line:84% Another complication is that if you just go straight downhill, 00:13:32.180 --> 00:13:36.200 align:middle line:84% if you have a function that has kind of a long narrow valley, 00:13:36.200 --> 00:13:38.210 align:middle line:90% things tend to zig-zag a lot. 00:13:38.210 --> 00:13:40.850 align:middle line:84% So you might converge very, very slowly 00:13:40.850 --> 00:13:44.580 align:middle line:84% where you zig-zag back and forth and with steepest descent. 00:13:44.580 --> 00:13:47.360 align:middle line:84% So people have lots of corrections to try and improve 00:13:47.360 --> 00:13:49.070 align:middle line:90% this using momentum terms-- 00:13:49.070 --> 00:13:51.470 align:middle line:84% you might have heard that-- conjugate gradients, 00:13:51.470 --> 00:13:53.600 align:middle line:84% other forms of memory to try and remember 00:13:53.600 --> 00:13:57.710 align:middle line:84% the previous steps and not zig-zag, 00:13:57.710 --> 00:14:00.830 align:middle line:84% backtrack in a direction you just optimized. 00:14:00.830 --> 00:14:02.270 align:middle line:84% Another fancy thing you can do is 00:14:02.270 --> 00:14:04.490 align:middle line:84% you can try and estimate second derivatives 00:14:04.490 --> 00:14:06.590 align:middle line:84% or even calculate second derivatives-- are called 00:14:06.590 --> 00:14:10.710 align:middle line:84% Hessian matrices-- and then use that information to help you. 00:14:10.710 --> 00:14:13.040 align:middle line:84% And there's the most important class 00:14:13.040 --> 00:14:14.383 align:middle line:90% are called BFGS algorithms. 00:14:14.383 --> 00:14:16.550 align:middle line:84% So I don't expect you to understand all these terms. 00:14:16.550 --> 00:14:19.110 align:middle line:84% I just want to throw out some culture. 00:14:19.110 --> 00:14:23.090 align:middle line:84% So if you hear, oh, use a momentum term or use BFGS, 00:14:23.090 --> 00:14:24.950 align:middle line:84% you're trying to avoid zig-zagging 00:14:24.950 --> 00:14:28.330 align:middle line:84% by kind of remembering what you did before. 00:14:28.330 --> 00:14:31.240 align:middle line:84% And it's one of those things where 00:14:31.240 --> 00:14:33.910 align:middle line:90% there is no one best algorithm. 00:14:33.910 --> 00:14:35.873 align:middle line:84% In certain fields, there's algorithms 00:14:35.873 --> 00:14:36.790 align:middle line:90% that are very popular. 00:14:36.790 --> 00:14:38.680 align:middle line:84% Like, in machine learning, there's 00:14:38.680 --> 00:14:40.570 align:middle line:84% an algorithm called atom that's very popular. 00:14:40.570 --> 00:14:43.600 align:middle line:84% But in general, there are a lot of algorithms out there. 00:14:43.600 --> 00:14:46.930 align:middle line:84% There's not one that's kind of one that's 00:14:46.930 --> 00:14:48.500 align:middle line:90% become the best algorithm. 00:14:48.500 --> 00:14:51.880 align:middle line:90% So in general, people-- 00:14:51.880 --> 00:14:53.860 align:middle line:84% you have to do a little bit of trial and error. 00:14:53.860 --> 00:14:56.290 align:middle line:84% For a given problem, you experiment a little bit, 00:14:56.290 --> 00:14:59.050 align:middle line:84% see what other people on similar problems use, 00:14:59.050 --> 00:15:01.390 align:middle line:84% or just try different things and see which one 00:15:01.390 --> 00:15:03.650 align:middle line:90% works best for your problem. 00:15:03.650 --> 00:15:09.170 align:middle line:84% And so and some imparting advice on this 00:15:09.170 --> 00:15:11.630 align:middle line:84% is that when you're doing large scale optimization, 00:15:11.630 --> 00:15:14.590 align:middle line:84% the main trick is not often the detail-- 00:15:14.590 --> 00:15:17.090 align:middle line:84% it's often not the details of these optimization algorithms. 00:15:17.090 --> 00:15:19.340 align:middle line:84% You're usually using off-the-shelf algorithms. 00:15:19.340 --> 00:15:23.060 align:middle line:84% So most of the practitioners doing optimization 00:15:23.060 --> 00:15:25.100 align:middle line:84% in practice, they spend most of their time 00:15:25.100 --> 00:15:28.250 align:middle line:84% not choosing the algorithm, but on the problem formulation, 00:15:28.250 --> 00:15:30.890 align:middle line:84% really thinking about what exact function 00:15:30.890 --> 00:15:33.020 align:middle line:84% you're trying to optimize, what constraints, 00:15:33.020 --> 00:15:35.150 align:middle line:84% how do you write down the parameters in order 00:15:35.150 --> 00:15:39.770 align:middle line:84% to enable you to use the best algorithms 00:15:39.770 --> 00:15:41.960 align:middle line:84% and have things behave most nicely? 00:15:41.960 --> 00:15:46.640 align:middle line:84% Because very often, there are multiple mathematically 00:15:46.640 --> 00:15:49.040 align:middle line:84% equivalent formulations of the same problem, 00:15:49.040 --> 00:15:51.200 align:middle line:84% but that have very different characteristics. 00:15:51.200 --> 00:15:55.430 align:middle line:84% And very often, someone if you're optimizing something, 00:15:55.430 --> 00:15:57.455 align:middle line:84% someone doesn't hand you a mathematical function 00:15:57.455 --> 00:15:58.580 align:middle line:90% that they want to optimize. 00:15:58.580 --> 00:16:01.430 align:middle line:84% They describe in words, "I want to make a stronger airplane 00:16:01.430 --> 00:16:03.080 align:middle line:90% wing," or whatever it is. 00:16:03.080 --> 00:16:06.870 align:middle line:84% And there are a lot of qualitatively similar ways 00:16:06.870 --> 00:16:09.940 align:middle line:84% to write down that problem mathematically. 00:16:09.940 --> 00:16:12.030 align:middle line:84% And so it really matters for optimization 00:16:12.030 --> 00:16:16.710 align:middle line:84% to choose one that well-match to optimization algorithms, 00:16:16.710 --> 00:16:19.050 align:middle line:84% and choose one where you can compute where it's 00:16:19.050 --> 00:16:21.818 align:middle line:84% differentiable, where you can compute derivatives efficiently 00:16:21.818 --> 00:16:22.360 align:middle line:90% and so forth. 00:16:22.360 --> 00:16:25.440 align:middle line:84% And as I said earlier, if you have lots of parameters, 00:16:25.440 --> 00:16:28.610 align:middle line:84% you really have to compute derivatives analytically. 00:16:28.610 --> 00:16:31.010 align:middle line:84% If you find differences, you'd have 00:16:31.010 --> 00:16:34.690 align:middle line:84% to take evaluate your function once per parameter. 00:16:34.690 --> 00:16:37.810 align:middle line:84% So in particular, I want to talk a little bit more 00:16:37.810 --> 00:16:40.240 align:middle line:84% about not machine learning, but something that's 00:16:40.240 --> 00:16:43.300 align:middle line:84% near and dear to my heart, which is engineering or physics 00:16:43.300 --> 00:16:46.210 align:middle line:84% optimization, where, for example, that you're 00:16:46.210 --> 00:16:49.560 align:middle line:84% trying to make an airplane wing that's as strong as possible. 00:16:49.560 --> 00:16:55.040 align:middle line:84% So let me try and write down the general process of this. 00:16:55.040 --> 00:16:58.227 align:middle line:84% So you have some engineering problem 00:16:58.227 --> 00:16:59.310 align:middle line:90% you're trying to optimize. 00:16:59.310 --> 00:17:00.030 align:middle line:90% What does it look like? 00:17:00.030 --> 00:17:02.072 align:middle line:84% So first of all, you have some design parameters. 00:17:02.072 --> 00:17:03.140 align:middle line:90% Let's call those p. 00:17:03.140 --> 00:17:05.089 align:middle line:90% And those may be the materials. 00:17:05.089 --> 00:17:08.869 align:middle line:84% It may be the geometry of the shape of the wing. 00:17:08.869 --> 00:17:10.550 align:middle line:90% So it might be the forces. 00:17:10.550 --> 00:17:12.060 align:middle line:90% Some way you're going to-- 00:17:12.060 --> 00:17:14.060 align:middle line:84% what your input's in the system are going to be, 00:17:14.060 --> 00:17:16.010 align:middle line:84% some other kind of unknowns these 00:17:16.010 --> 00:17:18.950 align:middle line:84% are the things you can control as an engineer. 00:17:18.950 --> 00:17:21.880 align:middle line:84% And then you might have a lot of them. 00:17:21.880 --> 00:17:23.630 align:middle line:84% To describe the shape of an airplane wing, 00:17:23.630 --> 00:17:26.282 align:middle line:84% you might need a lot of parameters. 00:17:26.282 --> 00:17:28.240 align:middle line:84% And then you're going to take those parameters, 00:17:28.240 --> 00:17:30.940 align:middle line:84% and you're going to stick them into some physical model, 00:17:30.940 --> 00:17:32.020 align:middle line:90% the physical model. 00:17:32.020 --> 00:17:35.020 align:middle line:84% If it's strength, it might be solid mechanics. 00:17:35.020 --> 00:17:36.740 align:middle line:90% It might be chemical reactions. 00:17:36.740 --> 00:17:38.050 align:middle line:90% It might be heat transport. 00:17:38.050 --> 00:17:40.180 align:middle line:84% It might be electromagnetic waves. 00:17:40.180 --> 00:17:41.440 align:middle line:90% It might be sound waves. 00:17:41.440 --> 00:17:45.400 align:middle line:84% It might be fluid flow problems and so forth. 00:17:45.400 --> 00:17:49.030 align:middle line:84% And you're going to solve that model to find some physics, 00:17:49.030 --> 00:17:51.220 align:middle line:84% like the stresses or something like that, 00:17:51.220 --> 00:17:52.700 align:middle line:90% or the electromagnetic waves. 00:17:52.700 --> 00:17:55.180 align:middle line:84% So the simplest example would be a linear model. 00:17:55.180 --> 00:17:57.840 align:middle line:90% So you're solving ax equals b. 00:17:57.840 --> 00:18:01.120 align:middle line:84% And your matrix and your right hand side 00:18:01.120 --> 00:18:02.560 align:middle line:90% depend on your parameters. 00:18:02.560 --> 00:18:04.180 align:middle line:84% Those are the things you can control. 00:18:04.180 --> 00:18:06.970 align:middle line:84% And then you find some solution, x, which is the physics. 00:18:06.970 --> 00:18:11.380 align:middle line:84% Say, the heat in your system or the electromagnetic fields 00:18:11.380 --> 00:18:14.680 align:middle line:84% in your system or the stresses in your system. 00:18:14.680 --> 00:18:18.400 align:middle line:84% And given that solution so that those X's now 00:18:18.400 --> 00:18:20.478 align:middle line:84% depend on the parameters through the matrix 00:18:20.478 --> 00:18:22.520 align:middle line:84% and through the and through the right hand sides. 00:18:22.520 --> 00:18:24.700 align:middle line:84% So these could be forces, displacements, 00:18:24.700 --> 00:18:27.790 align:middle line:84% and so forth, magnetic field, pressures, velocities. 00:18:27.790 --> 00:18:31.390 align:middle line:84% Then, you have something you're trying to do, some objective. 00:18:31.390 --> 00:18:33.640 align:middle line:84% You're trying to make it as strong as possible 00:18:33.640 --> 00:18:37.930 align:middle line:84% or as fast as possible or as efficient as possible in power, 00:18:37.930 --> 00:18:42.550 align:middle line:84% or you're trying to match something to experiment. 00:18:42.550 --> 00:18:45.243 align:middle line:84% Like, if you have some unknowns, an unknown physics, 00:18:45.243 --> 00:18:46.660 align:middle line:84% and you're trying to match your it 00:18:46.660 --> 00:18:49.450 align:middle line:84% to your measurement of chemical reactions. 00:18:49.450 --> 00:18:51.910 align:middle line:84% So that's something that takes the physical solution 00:18:51.910 --> 00:18:54.400 align:middle line:84% and gives you a number that's your objective function. 00:18:54.400 --> 00:18:56.410 align:middle line:84% Your parameters go into the physics. 00:18:56.410 --> 00:18:57.850 align:middle line:84% The physics goes into a solution. 00:18:57.850 --> 00:19:00.220 align:middle line:84% The solution goes into a design objective 00:19:00.220 --> 00:19:02.710 align:middle line:90% function, a number, a scalar. 00:19:02.710 --> 00:19:04.493 align:middle line:84% And then you want to optimize it. 00:19:04.493 --> 00:19:06.160 align:middle line:84% You want to update the design parameters 00:19:06.160 --> 00:19:09.710 align:middle line:84% to make this function better to say maximize or minimize it. 00:19:09.710 --> 00:19:11.830 align:middle line:84% And so you have to compute the gradient. 00:19:11.830 --> 00:19:13.890 align:middle line:84% You need to compute the derivative of f, 00:19:13.890 --> 00:19:15.560 align:middle line:90% not with respect to f-- 00:19:15.560 --> 00:19:17.840 align:middle line:84% I'm sorry, not with respect to x, 00:19:17.840 --> 00:19:20.160 align:middle line:84% but with respect to p, your design parameters. 00:19:20.160 --> 00:19:22.430 align:middle line:84% So you have to follow the chain rue all the way 00:19:22.430 --> 00:19:25.850 align:middle line:84% through this process from the design parameters 00:19:25.850 --> 00:19:28.280 align:middle line:84% to the objective to get all these derivatives. 00:19:28.280 --> 00:19:31.190 align:middle line:84% And then you can go update the design parameters 00:19:31.190 --> 00:19:32.990 align:middle line:84% in the downhill or uphill direction 00:19:32.990 --> 00:19:35.180 align:middle line:90% to make your function better. 00:19:35.180 --> 00:19:37.130 align:middle line:84% So this is kind of the general cartoon. 00:19:37.130 --> 00:19:39.343 align:middle line:84% And an example of that, I wanted to show-- 00:19:39.343 --> 00:19:40.760 align:middle line:84% there's lots and lots of examples, 00:19:40.760 --> 00:19:44.490 align:middle line:84% but this is kind of a fun function example. 00:19:44.490 --> 00:19:46.490 align:middle line:90% So this was found online. 00:19:46.490 --> 00:19:50.690 align:middle line:84% So they're trying to do what's called topology optimization 00:19:50.690 --> 00:19:52.430 align:middle line:90% of a chair. 00:19:52.430 --> 00:19:54.500 align:middle line:84% So what you're trying to do here is 00:19:54.500 --> 00:19:57.350 align:middle line:84% you're trying to make a chair as strong 00:19:57.350 --> 00:20:01.190 align:middle line:84% as possible with a minimal amount of material. 00:20:01.190 --> 00:20:04.280 align:middle line:84% And it has a seat and it has a seat back. 00:20:04.280 --> 00:20:05.065 align:middle line:90% Those are fixed. 00:20:05.065 --> 00:20:06.350 align:middle line:84% ALAN EDELMAN: I want to buy one of these. 00:20:06.350 --> 00:20:07.280 align:middle line:90% Where do I buy it? 00:20:07.280 --> 00:20:10.400 align:middle line:84% STEVEN G. JOHNSON: [CHUCKLES] But the shape of the chair 00:20:10.400 --> 00:20:12.140 align:middle line:90% is just completely free. 00:20:12.140 --> 00:20:14.510 align:middle line:84% So if you imagine, you're going to start 00:20:14.510 --> 00:20:16.010 align:middle line:90% with just a block of metal. 00:20:16.010 --> 00:20:20.540 align:middle line:84% And every voxel-- you're going to divide it into 3D pixels, 00:20:20.540 --> 00:20:21.530 align:middle line:90% voxels. 00:20:21.530 --> 00:20:24.980 align:middle line:84% Every voxel, you can either put metal or air. 00:20:24.980 --> 00:20:26.690 align:middle line:84% This is called topology optimization. 00:20:26.690 --> 00:20:30.470 align:middle line:84% Topology in math is how things are connected. 00:20:30.470 --> 00:20:33.770 align:middle line:84% Are their hole-- is it one big thing or are there holes? 00:20:33.770 --> 00:20:35.490 align:middle line:84% How many holes are there and so forth. 00:20:35.490 --> 00:20:37.530 align:middle line:84% And we're going to leave this completely free. 00:20:37.530 --> 00:20:39.980 align:middle line:84% So the computer is going to decide where to put the holes 00:20:39.980 --> 00:20:43.670 align:middle line:84% or if there are holes and so forth. 00:20:43.670 --> 00:20:45.170 align:middle line:84% And I think what they're doing is 00:20:45.170 --> 00:20:49.370 align:middle line:84% they're maximizing the weight with a given 00:20:49.370 --> 00:20:50.370 align:middle line:90% amount of material. 00:20:50.370 --> 00:20:53.580 align:middle line:84% Either that or they're fixing the weight that it has to hold 00:20:53.580 --> 00:20:54.630 align:middle line:90% and minimizing. 00:20:54.630 --> 00:20:57.960 align:middle line:84% Probably, they're minimizing the amount of material subject 00:20:57.960 --> 00:21:01.255 align:middle line:84% to a constraint of a certain amount of weight. 00:21:01.255 --> 00:21:02.880 align:middle line:84% So there's zillions of parameters here. 00:21:02.880 --> 00:21:05.460 align:middle line:90% And this is a movie. 00:21:05.460 --> 00:21:07.020 align:middle line:90% Let's see, does this work? 00:21:07.020 --> 00:21:08.450 align:middle line:90% Present on this device. 00:21:08.450 --> 00:21:11.200 align:middle line:90% 00:21:11.200 --> 00:21:11.700 align:middle line:90% OK. 00:21:11.700 --> 00:21:14.860 align:middle line:84% So this is a movie of that optimization you can see. 00:21:14.860 --> 00:21:18.090 align:middle line:90% So it starts with just a-- 00:21:18.090 --> 00:21:21.510 align:middle line:84% their starting thing is just a solid block of metal. 00:21:21.510 --> 00:21:23.610 align:middle line:84% So that certainly will support your weight, 00:21:23.610 --> 00:21:25.650 align:middle line:90% but it's very, very heavy. 00:21:25.650 --> 00:21:28.818 align:middle line:84% And so now they're going to try and optimize 00:21:28.818 --> 00:21:30.360 align:middle line:84% it to minimize that amount of weight, 00:21:30.360 --> 00:21:31.985 align:middle line:84% subject to the constraint that it still 00:21:31.985 --> 00:21:34.800 align:middle line:90% has to support a person right. 00:21:34.800 --> 00:21:39.780 align:middle line:84% And this thing-- even though there's a zillion parameters, 00:21:39.780 --> 00:21:42.330 align:middle line:84% going in the downhill direction allows 00:21:42.330 --> 00:21:47.240 align:middle line:84% you to continuously evolve that into this structure. 00:21:47.240 --> 00:21:52.710 align:middle line:84% And so it continuously shaves away 00:21:52.710 --> 00:21:57.510 align:middle line:84% metal, varying all million parameters at once. 00:21:57.510 --> 00:22:01.890 align:middle line:84% And it decides to put these weird this arrange of holes 00:22:01.890 --> 00:22:04.290 align:middle line:90% here to make this chair. 00:22:04.290 --> 00:22:06.250 align:middle line:84% And so this is in some art exhibit. 00:22:06.250 --> 00:22:10.410 align:middle line:84% I think they said they then use some kind of 3D printing 00:22:10.410 --> 00:22:13.980 align:middle line:84% somehow to make life-size chair out of metal with this design, 00:22:13.980 --> 00:22:15.450 align:middle line:90% and it works. 00:22:15.450 --> 00:22:16.950 align:middle line:90% And they displayed it. 00:22:16.950 --> 00:22:19.100 align:middle line:84% More practically, I think, in the first lecture, 00:22:19.100 --> 00:22:20.850 align:middle line:84% Alan showed an example of an airplane wing 00:22:20.850 --> 00:22:22.517 align:middle line:84% that was designed by this kind of thing. 00:22:22.517 --> 00:22:25.460 align:middle line:84% And there's lots of examples of this. 00:22:25.460 --> 00:22:29.770 align:middle line:84% So the key thing in this that makes this practical 00:22:29.770 --> 00:22:31.532 align:middle line:84% is your ability to calculate derivatives. 00:22:31.532 --> 00:22:33.490 align:middle line:84% You need to be able to calculate the derivative 00:22:33.490 --> 00:22:38.350 align:middle line:84% of the strength of this or some function of a solution 00:22:38.350 --> 00:22:41.860 align:middle line:84% with respect to all million parameters 00:22:41.860 --> 00:22:47.470 align:middle line:84% of every voxel, the density of the material at every voxel. 00:22:47.470 --> 00:22:50.920 align:middle line:84% So as I said, this is sometimes called backpropagation 00:22:50.920 --> 00:22:52.510 align:middle line:90% in neural networks. 00:22:52.510 --> 00:22:55.490 align:middle line:84% It really is just doing the chain rule left or right. 00:22:55.490 --> 00:22:57.670 align:middle line:84% And so I want to talk about that in this kind 00:22:57.670 --> 00:22:59.200 align:middle line:90% of specific circumstance. 00:22:59.200 --> 00:23:01.990 align:middle line:84% You can really see how it makes a huge difference. 00:23:01.990 --> 00:23:04.180 align:middle line:84% So suppose we're computing a function. 00:23:04.180 --> 00:23:07.540 align:middle line:84% We have some scalar function f of x. 00:23:07.540 --> 00:23:12.520 align:middle line:84% But our x we get by solving a system of equations, ax 00:23:12.520 --> 00:23:15.970 align:middle line:84% equals b, where the matrix depends on some parameters. 00:23:15.970 --> 00:23:18.730 align:middle line:84% And I want to compute the derivative of f with respect 00:23:18.730 --> 00:23:20.380 align:middle line:90% to the parameters in the matrix. 00:23:20.380 --> 00:23:22.360 align:middle line:90% So x is a inverse b. 00:23:22.360 --> 00:23:24.820 align:middle line:84% So I'm really computing f of A inverse 00:23:24.820 --> 00:23:27.370 align:middle line:84% b, where the parameters are inside the matrix, 00:23:27.370 --> 00:23:30.600 align:middle line:84% and I want to apply the chain rule. 00:23:30.600 --> 00:23:33.990 align:middle line:90% So we can just do the-- 00:23:33.990 --> 00:23:38.310 align:middle line:90% so this is-- yeah. 00:23:38.310 --> 00:23:41.162 align:middle line:84% So we have f of x and then x of p, 00:23:41.162 --> 00:23:43.120 align:middle line:84% and I want to propagate my chain rules to that. 00:23:43.120 --> 00:23:45.300 align:middle line:84% So I to ask what's df, the change in f, 00:23:45.300 --> 00:23:49.140 align:middle line:84% for a small change in the parameters p? 00:23:49.140 --> 00:23:50.410 align:middle line:90% So let's just do chain rule. 00:23:50.410 --> 00:23:52.140 align:middle line:84% So first of all, we have f prime of x. 00:23:52.140 --> 00:23:55.470 align:middle line:84% So this is just the derivative of f with respect to x. 00:23:55.470 --> 00:23:56.760 align:middle line:90% So that's just a row vector. 00:23:56.760 --> 00:23:59.460 align:middle line:90% So x is a column vector. 00:23:59.460 --> 00:24:00.630 align:middle line:90% f is a scalar. 00:24:00.630 --> 00:24:01.890 align:middle line:90% So f prime is a row vector. 00:24:01.890 --> 00:24:04.930 align:middle line:84% It's just the transpose of our gradient of f. 00:24:04.930 --> 00:24:08.050 align:middle line:84% But we're not done yet because that gets multiplied by dx. 00:24:08.050 --> 00:24:11.983 align:middle line:84% But that's the change in the solution. 00:24:11.983 --> 00:24:13.150 align:middle line:90% And that's not what we want. 00:24:13.150 --> 00:24:16.260 align:middle line:84% We want is what happens when you change the parameters 00:24:16.260 --> 00:24:17.850 align:middle line:90% or when you change the matrix. 00:24:17.850 --> 00:24:19.660 align:middle line:90% So we do the chain rule. 00:24:19.660 --> 00:24:23.400 align:middle line:84% And we have to take the derivative-- x is A inverse b, 00:24:23.400 --> 00:24:25.440 align:middle line:90% so we need to take dA. 00:24:25.440 --> 00:24:27.102 align:middle line:90% And b depend on the parameters. 00:24:27.102 --> 00:24:28.060 align:middle line:90% That's just a constant. 00:24:28.060 --> 00:24:29.781 align:middle line:90% So we just need dA inverse b. 00:24:29.781 --> 00:24:32.430 align:middle line:90% 00:24:32.430 --> 00:24:38.850 align:middle line:84% Then we saw I think last lecture that the d of A inverse 00:24:38.850 --> 00:24:43.345 align:middle line:84% is equal to minus A inverse dA inverse. 00:24:43.345 --> 00:24:44.220 align:middle line:90% Do you remember that? 00:24:44.220 --> 00:24:47.100 align:middle line:84% We derived that last term just by taking the product 00:24:47.100 --> 00:24:50.220 align:middle line:84% rule AA inverse equals I. d of that, 00:24:50.220 --> 00:24:52.230 align:middle line:84% we derive this rule-- the D of A inverse 00:24:52.230 --> 00:24:54.570 align:middle line:84% is minus A inverse Da A inverse-- really 00:24:54.570 --> 00:24:56.970 align:middle line:90% useful formula. 00:24:56.970 --> 00:25:00.990 align:middle line:84% And so now if we group this, let's look at this expression. 00:25:00.990 --> 00:25:04.870 align:middle line:84% You get A inverse b, which is just x. 00:25:04.870 --> 00:25:07.170 align:middle line:84% So you're really computing f prime, which 00:25:07.170 --> 00:25:11.640 align:middle line:90% is a row vec, times A inverse-- 00:25:11.640 --> 00:25:13.500 align:middle line:84% so inverse of our system matrix-- 00:25:13.500 --> 00:25:19.760 align:middle line:84% times the change in the matrix to the parameters 00:25:19.760 --> 00:25:26.060 align:middle line:90% times x times A inverse b. 00:25:26.060 --> 00:25:29.490 align:middle line:84% So the whole trick, the whole stupid trick, 00:25:29.490 --> 00:25:32.660 align:middle line:84% which becomes obvious once you see it, 00:25:32.660 --> 00:25:37.340 align:middle line:84% is just to take this product and multiply it left to right. 00:25:37.340 --> 00:25:39.820 align:middle line:84% So that means you want to put parentheses 00:25:39.820 --> 00:25:42.710 align:middle line:90% around the F prime A inverse. 00:25:42.710 --> 00:25:43.490 align:middle line:90% Why is that? 00:25:43.490 --> 00:25:49.320 align:middle line:84% Because that's independent of dA. 00:25:49.320 --> 00:25:51.000 align:middle line:90% That's one vector. 00:25:51.000 --> 00:25:54.270 align:middle line:84% That's solving one system of equations. 00:25:54.270 --> 00:25:57.870 align:middle line:84% If you do a row vector times a matrix on the left, 00:25:57.870 --> 00:25:59.850 align:middle line:84% that's one matrix vector multiply. 00:25:59.850 --> 00:26:04.380 align:middle line:84% So that's the same thing as A inverse transpose times F. 00:26:04.380 --> 00:26:07.260 align:middle line:84% Or equivalently, if you call this-- this is a row vector. 00:26:07.260 --> 00:26:08.970 align:middle line:84% If you call this v transpose, it's 00:26:08.970 --> 00:26:11.820 align:middle line:84% the same thing as solving A transpose V 00:26:11.820 --> 00:26:15.820 align:middle line:90% equals f prime transpose. 00:26:15.820 --> 00:26:18.490 align:middle line:84% So that means this is called the adjoint equation. 00:26:18.490 --> 00:26:20.220 align:middle line:84% So you solved-- think what you do. 00:26:20.220 --> 00:26:24.480 align:middle line:84% You first solve one equation Ax equals to get x. 00:26:24.480 --> 00:26:27.595 align:middle line:84% That might be really expensive because this is a big physics 00:26:27.595 --> 00:26:28.095 align:middle line:90% simulation. 00:26:28.095 --> 00:26:31.530 align:middle line:90% A is a huge matrix. 00:26:31.530 --> 00:26:34.950 align:middle line:84% But then you solve another equation with the same matrix, 00:26:34.950 --> 00:26:42.710 align:middle line:84% just transposed, to get v. And then once you have that, 00:26:42.710 --> 00:26:47.080 align:middle line:84% then the derivative with respect to any parameter 00:26:47.080 --> 00:26:50.620 align:middle line:84% is just v transpose dAx, which means if you have a parameter, 00:26:50.620 --> 00:26:53.900 align:middle line:84% this should be a pk out of that little box there. 00:26:53.900 --> 00:26:54.790 align:middle line:90% It's a font problem. 00:26:54.790 --> 00:26:56.273 align:middle line:90% It should be a p usb k. 00:26:56.273 --> 00:26:57.940 align:middle line:84% If you have a whole bunch of parameters, 00:26:57.940 --> 00:27:02.530 align:middle line:84% if you want the derivative of f with respect to pk, 00:27:02.530 --> 00:27:05.590 align:middle line:84% that's just dividing both sides by tpk. 00:27:05.590 --> 00:27:16.687 align:middle line:84% That's partial f partial pk is just df dpk is v transpose I 00:27:16.687 --> 00:27:17.770 align:middle line:90% guess with the minus sign. 00:27:17.770 --> 00:27:19.312 align:middle line:84% I should have put a minus sign there. 00:27:19.312 --> 00:27:22.870 align:middle line:90% 00:27:22.870 --> 00:27:25.210 align:middle line:90% How do I get back? 00:27:25.210 --> 00:27:28.840 align:middle line:84% And I tried this tablet presentation mode. 00:27:28.840 --> 00:27:31.177 align:middle line:90% Let me just-- 00:27:31.177 --> 00:27:32.635 align:middle line:84% I'm going to switch to my computer. 00:27:32.635 --> 00:27:35.930 align:middle line:90% 00:27:35.930 --> 00:27:38.150 align:middle line:84% That's just because this is annoying. 00:27:38.150 --> 00:27:40.080 align:middle line:84% ALAN EDELMAN: Did you develop on a Mac 00:27:40.080 --> 00:27:42.310 align:middle line:90% and go to a PC or vice versa? 00:27:42.310 --> 00:27:43.680 align:middle line:90% STEVEN G. JOHNSON: No. 00:27:43.680 --> 00:27:47.040 align:middle line:84% I developed it on Google Slides, but then I'm 00:27:47.040 --> 00:27:53.760 align:middle line:84% using the Google Slides app on my tablet, which is 00:27:53.760 --> 00:27:55.920 align:middle line:90% a little bit flaky, apparently. 00:27:55.920 --> 00:27:58.735 align:middle line:90% So let me just share my screen. 00:27:58.735 --> 00:28:00.360 align:middle line:84% ALAN EDELMAN: Yeah, sometimes the trick 00:28:00.360 --> 00:28:02.285 align:middle line:90% is to just do the PDF. 00:28:02.285 --> 00:28:03.660 align:middle line:84% STEVEN G. JOHNSON: Well, I wanted 00:28:03.660 --> 00:28:05.310 align:middle line:90% to show movies and things. 00:28:05.310 --> 00:28:09.680 align:middle line:90% So I can just share it on my-- 00:28:09.680 --> 00:28:14.112 align:middle line:84% on my laptop, it should work better. 00:28:14.112 --> 00:28:15.570 align:middle line:84% Do you see that the fonts come out? 00:28:15.570 --> 00:28:17.220 align:middle line:90% Yes, good. 00:28:17.220 --> 00:28:19.710 align:middle line:84% So there should be a minus sign here. 00:28:19.710 --> 00:28:24.620 align:middle line:90% 00:28:24.620 --> 00:28:26.570 align:middle line:90% We take this product. 00:28:26.570 --> 00:28:28.880 align:middle line:84% We're going to multiply it left to right, which 00:28:28.880 --> 00:28:31.040 align:middle line:84% means I multiply row vector-- the first thing I do 00:28:31.040 --> 00:28:35.660 align:middle line:84% is multiply row vector times a matrix, which gives you 00:28:35.660 --> 00:28:36.800 align:middle line:90% another row vector. 00:28:36.800 --> 00:28:39.980 align:middle line:84% But that's multiplying on the left by-- 00:28:39.980 --> 00:28:42.560 align:middle line:84% A inverse on the left is equivalent to solving 00:28:42.560 --> 00:28:47.090 align:middle line:84% a system of equations with A transpose for this row vector, 00:28:47.090 --> 00:28:51.650 align:middle line:84% where f prime transpose is on the right. 00:28:51.650 --> 00:28:53.400 align:middle line:84% That's why it's called an adjoint equation 00:28:53.400 --> 00:28:55.230 align:middle line:84% because an adjoint is another name 00:28:55.230 --> 00:28:59.610 align:middle line:84% for a transpose of a real vector of a real matrix. 00:28:59.610 --> 00:29:01.290 align:middle line:84% So you solved one system of equations 00:29:01.290 --> 00:29:03.870 align:middle line:84% to get your x, another system of equations 00:29:03.870 --> 00:29:06.120 align:middle line:84% with almost the same matrix just transposed to get v. 00:29:06.120 --> 00:29:09.690 align:middle line:90% So it's equally easy or hard. 00:29:09.690 --> 00:29:13.860 align:middle line:90% And then your df is just-- 00:29:13.860 --> 00:29:17.128 align:middle line:84% there should be a minus sign here. 00:29:17.128 --> 00:29:20.290 align:middle line:84% Or I can just put the minus sign inside. 00:29:20.290 --> 00:29:21.640 align:middle line:90% Well, it doesn't matter. 00:29:21.640 --> 00:29:25.720 align:middle line:90% 00:29:25.720 --> 00:29:28.780 align:middle line:84% df is just minus v transpose dAx. 00:29:28.780 --> 00:29:32.650 align:middle line:84% So now you have a whole bunch of parameters, pk. 00:29:32.650 --> 00:29:38.210 align:middle line:84% So if you want df/dpk or just divide both sides by dpk. 00:29:38.210 --> 00:29:42.950 align:middle line:84% So df/dpk is the same thing as minus v transpose 00:29:42.950 --> 00:29:46.390 align:middle line:90% partial a partial plx. 00:29:46.390 --> 00:29:49.387 align:middle line:90% So I just take for each-- 00:29:49.387 --> 00:29:50.970 align:middle line:84% if I have a million parameters, I just 00:29:50.970 --> 00:29:52.320 align:middle line:90% take a million dot products. 00:29:52.320 --> 00:29:55.048 align:middle line:84% I only solve two systems of equations, 00:29:55.048 --> 00:29:56.340 align:middle line:90% but this is the expensive part. 00:29:56.340 --> 00:29:58.830 align:middle line:84% This is solving my big structural mechanics 00:29:58.830 --> 00:30:00.210 align:middle line:90% equation or whatever it is. 00:30:00.210 --> 00:30:02.992 align:middle line:84% I solve it twice, and then I just 00:30:02.992 --> 00:30:04.200 align:middle line:90% take a bunch of dot products. 00:30:04.200 --> 00:30:07.560 align:middle line:84% And in fact, usually, this matrix is mostly 0. 00:30:07.560 --> 00:30:10.590 align:middle line:84% It's very sparse because usually each often each parameter only 00:30:10.590 --> 00:30:12.820 align:middle line:84% affects a few entries of the matrix. 00:30:12.820 --> 00:30:15.630 align:middle line:84% So this dot product is extremely cheap and because you 00:30:15.630 --> 00:30:17.700 align:middle line:90% don't need to multiply by 0's. 00:30:17.700 --> 00:30:21.120 align:middle line:84% So this is what allows you to do this efficiently. 00:30:21.120 --> 00:30:24.030 align:middle line:84% You only have to solve the equations twice, 00:30:24.030 --> 00:30:27.810 align:middle line:84% your huge systems of equations, your physics equations, twice. 00:30:27.810 --> 00:30:31.140 align:middle line:84% And then you can get both the f and its gradient 00:30:31.140 --> 00:30:34.850 align:middle line:84% with respect to all the parameters. 00:30:34.850 --> 00:30:36.230 align:middle line:90% So this is the adjoint method. 00:30:36.230 --> 00:30:39.260 align:middle line:84% It's exactly the same idea as back propagation 00:30:39.260 --> 00:30:40.100 align:middle line:90% in neural networks. 00:30:40.100 --> 00:30:43.310 align:middle line:84% It's exactly what people in automatic differentiation 00:30:43.310 --> 00:30:45.710 align:middle line:84% call reverse mode, but it's really just doing 00:30:45.710 --> 00:30:46.865 align:middle line:90% chain rule left to right. 00:30:46.865 --> 00:30:49.450 align:middle line:90% 00:30:49.450 --> 00:30:51.100 align:middle line:84% And I want to make sure you understand 00:30:51.100 --> 00:30:53.100 align:middle line:84% why this is so important to do it left or right. 00:30:53.100 --> 00:30:56.960 align:middle line:84% So suppose we did it right to left. 00:30:56.960 --> 00:31:02.260 align:middle line:84% So if I did right to left, and I wanted partial f partial pk, 00:31:02.260 --> 00:31:07.340 align:middle line:84% then I have minus f prime, a inverse, partial a, 00:31:07.340 --> 00:31:09.105 align:middle line:90% partial pkx. 00:31:09.105 --> 00:31:11.480 align:middle line:84% It's the same equation, but it's going to put parentheses 00:31:11.480 --> 00:31:12.740 align:middle line:90% in a different place. 00:31:12.740 --> 00:31:15.110 align:middle line:90% Suppose I did that. 00:31:15.110 --> 00:31:19.600 align:middle line:84% So for each parameter, this is a vector. 00:31:19.600 --> 00:31:22.670 align:middle line:84% And then multiplying by A inverse is a solve. 00:31:22.670 --> 00:31:26.420 align:middle line:84% It's a solving a huge system of equations. 00:31:26.420 --> 00:31:29.980 align:middle line:84% So this is-- and that's expensive part of the physics 00:31:29.980 --> 00:31:32.620 align:middle line:84% is usually solving this huge system of equations. 00:31:32.620 --> 00:31:34.330 align:middle line:84% So you have to-- but you have one 00:31:34.330 --> 00:31:37.700 align:middle line:84% of these vectors for every parameter. 00:31:37.700 --> 00:31:39.630 align:middle line:84% So if you just did it right to left, 00:31:39.630 --> 00:31:43.252 align:middle line:84% which is called forward mode differentiation, it's terrible. 00:31:43.252 --> 00:31:45.460 align:middle line:84% You have to solve-- if you have a million parameters, 00:31:45.460 --> 00:31:50.560 align:middle line:84% you have to solve a million mechanics problems 00:31:50.560 --> 00:31:53.960 align:middle line:84% or whatever your physics problem is for this. 00:31:53.960 --> 00:31:57.000 align:middle line:90% So this is called forward mode. 00:31:57.000 --> 00:31:58.890 align:middle line:84% And this is very bad basically when 00:31:58.890 --> 00:32:02.040 align:middle line:90% you have lots of parameters. 00:32:02.040 --> 00:32:05.340 align:middle line:84% And so it turns out this reverse mode differentiation 00:32:05.340 --> 00:32:06.015 align:middle line:90% is really good. 00:32:06.015 --> 00:32:07.890 align:middle line:84% That's what you should use when you have lots 00:32:07.890 --> 00:32:09.960 align:middle line:90% of inputs and only one output. 00:32:09.960 --> 00:32:11.340 align:middle line:84% Forward mode is going to turn out 00:32:11.340 --> 00:32:13.680 align:middle line:84% to be good when you have lots of outputs 00:32:13.680 --> 00:32:17.250 align:middle line:84% but only one input or a small number input. 00:32:17.250 --> 00:32:19.680 align:middle line:84% So yeah, actually, I wrote this down. 00:32:19.680 --> 00:32:21.180 align:middle line:84% Right-to-left forward mode is better 00:32:21.180 --> 00:32:22.888 align:middle line:84% when you have one input and many outputs. 00:32:22.888 --> 00:32:25.083 align:middle line:84% Left-to-right, which is also called backward mode, 00:32:25.083 --> 00:32:26.250 align:middle line:90% also called adjoint methods. 00:32:26.250 --> 00:32:27.690 align:middle line:84% It's also called backpropagation. 00:32:27.690 --> 00:32:28.920 align:middle line:84% It's a sort of thing that's been-- 00:32:28.920 --> 00:32:30.337 align:middle line:84% it has a lot of names because it's 00:32:30.337 --> 00:32:34.900 align:middle line:84% been I think rediscovered independently multiple times. 00:32:34.900 --> 00:32:36.642 align:middle line:84% It's just left to right chain rule. 00:32:36.642 --> 00:32:38.850 align:middle line:84% It's better when you have one output or a few outputs 00:32:38.850 --> 00:32:40.530 align:middle line:90% and lots of inputs. 00:32:40.530 --> 00:32:44.460 align:middle line:84% And so if you're using automatic differentiation, most of them 00:32:44.460 --> 00:32:48.460 align:middle line:84% will use the term reverse mode or forward mode. 00:32:48.460 --> 00:32:51.940 align:middle line:84% So in Jax, for example, which is the Python thing. 00:32:51.940 --> 00:32:53.350 align:middle line:90% They'll call Jack forward. 00:32:53.350 --> 00:32:55.210 align:middle line:84% In Julia, there's a package called forward 00:32:55.210 --> 00:32:58.425 align:middle line:84% diff, which stands for forward mode differentiation. 00:32:58.425 --> 00:33:00.550 align:middle line:84% We're going to talk pretty soon about dual numbers. 00:33:00.550 --> 00:33:03.180 align:middle line:84% That's forward mode, and so forth. 00:33:03.180 --> 00:33:04.680 align:middle line:84% Another thing you might come to mind 00:33:04.680 --> 00:33:07.890 align:middle line:90% is to use finite differences. 00:33:07.890 --> 00:33:14.470 align:middle line:84% So finite differences, if you have lots of parameters, 00:33:14.470 --> 00:33:17.618 align:middle line:84% it's also just as bad. right so if imagine you're 00:33:17.618 --> 00:33:18.660 align:middle line:90% using finite differences. 00:33:18.660 --> 00:33:20.260 align:middle line:84% So you're going to approximate your partial derivative 00:33:20.260 --> 00:33:21.910 align:middle line:90% with respect to each parameter. 00:33:21.910 --> 00:33:23.420 align:middle line:90% What do you do? 00:33:23.420 --> 00:33:27.310 align:middle line:84% You take each parameter-- you take f and you perturb 00:33:27.310 --> 00:33:30.010 align:middle line:90% each parameter by epsilon. 00:33:30.010 --> 00:33:32.170 align:middle line:84% So you think of ek, here is the unit 00:33:32.170 --> 00:33:34.790 align:middle line:90% vector in the direction of pk. 00:33:34.790 --> 00:33:36.550 align:middle line:90% So you take each parameter. 00:33:36.550 --> 00:33:39.610 align:middle line:90% You perturb it by epsilon. 00:33:39.610 --> 00:33:42.370 align:middle line:84% In the case direction is [INAUDIBLE] fp 00:33:42.370 --> 00:33:43.330 align:middle line:90% divided by epsilon. 00:33:43.330 --> 00:33:45.882 align:middle line:84% That gives you one partial derivative. 00:33:45.882 --> 00:33:47.590 align:middle line:84% But now if you have a million parameters, 00:33:47.590 --> 00:33:49.090 align:middle line:84% you need to do this a million times, 00:33:49.090 --> 00:33:52.000 align:middle line:90% each one is a separate solve. 00:33:52.000 --> 00:33:54.250 align:middle line:90% You need to solve for-- 00:33:54.250 --> 00:33:56.220 align:middle line:84% you're perturbing the parameters one by one 00:33:56.220 --> 00:33:57.970 align:middle line:84% and, then you have to solve it one by one. 00:33:57.970 --> 00:34:00.310 align:middle line:90% So again, it's a disaster. 00:34:00.310 --> 00:34:01.990 align:middle line:84% If you had to use finite differences, 00:34:01.990 --> 00:34:07.150 align:middle line:84% you would not be able to do neural networks or topology 00:34:07.150 --> 00:34:09.190 align:middle line:84% optimization of chairs or airplane wings 00:34:09.190 --> 00:34:11.090 align:middle line:90% or things like that. 00:34:11.090 --> 00:34:13.280 align:middle line:84% And you can generalize it in other ways. 00:34:13.280 --> 00:34:18.860 align:middle line:84% So that was adjoint methods for linear equations. 00:34:18.860 --> 00:34:21.520 align:middle line:84% So this is-- so you're solving f of x, where 00:34:21.520 --> 00:34:25.780 align:middle line:84% x solves a linear equation, but you can do the same thing where 00:34:25.780 --> 00:34:29.590 align:middle line:90% f solves a nonlinear equation. 00:34:29.590 --> 00:34:31.659 align:middle line:84% Suppose you're solving-- you're computing 00:34:31.659 --> 00:34:36.310 align:middle line:84% f of x, where x is your physics, p is your parameters, 00:34:36.310 --> 00:34:38.380 align:middle line:84% like your shape of your airplane wing. 00:34:38.380 --> 00:34:41.620 align:middle line:84% But now to solve for your stresses or whatever, 00:34:41.620 --> 00:34:45.710 align:middle line:84% your physical equations are not linear. 00:34:45.710 --> 00:34:47.510 align:middle line:84% So what do nonlinear equations look like? 00:34:47.510 --> 00:34:48.770 align:middle line:84% In general, nonlinear equations, you 00:34:48.770 --> 00:34:50.540 align:middle line:84% can write them as a root-finding problem. 00:34:50.540 --> 00:34:53.289 align:middle line:90% You have n parameters. 00:34:53.289 --> 00:34:58.680 align:middle line:84% You have n physics unknowns, and you have n nonlinear equations. 00:34:58.680 --> 00:35:00.290 align:middle line:84% So you have some nonlinear equations. 00:35:00.290 --> 00:35:03.260 align:middle line:84% It's called g that depend on the parameters, that 00:35:03.260 --> 00:35:04.700 align:middle line:84% depend on the physics, and you're 00:35:04.700 --> 00:35:06.470 align:middle line:90% trying to set these equal to 0. 00:35:06.470 --> 00:35:10.160 align:middle line:84% So these are-- g is not a scalar here. 00:35:10.160 --> 00:35:12.650 align:middle line:90% Its n component vector. 00:35:12.650 --> 00:35:14.510 align:middle line:84% So it's exactly like what we talked about 00:35:14.510 --> 00:35:16.100 align:middle line:90% with Newton's method. 00:35:16.100 --> 00:35:19.040 align:middle line:84% So how would you solve this in the first place? 00:35:19.040 --> 00:35:21.020 align:middle line:84% You compute the Jacobian of this thing, 00:35:21.020 --> 00:35:27.070 align:middle line:84% and you do a sequence of linear solves to converge 00:35:27.070 --> 00:35:29.570 align:middle line:90% towards the solution. 00:35:29.570 --> 00:35:33.380 align:middle line:84% But now how do we take the derivative of f with respect 00:35:33.380 --> 00:35:33.880 align:middle line:90% to p? 00:35:33.880 --> 00:35:36.910 align:middle line:84% So now, again, we want to do the chain rule. 00:35:36.910 --> 00:35:42.370 align:middle line:90% So df is f prime of x dx. 00:35:42.370 --> 00:35:44.290 align:middle line:90% That's the derivative of f of x. 00:35:44.290 --> 00:35:48.440 align:middle line:84% So f prime is still a row vector. 00:35:48.440 --> 00:35:50.240 align:middle line:90% What is dx? 00:35:50.240 --> 00:35:52.460 align:middle line:84% So now we need to think about the chain rule. 00:35:52.460 --> 00:35:54.950 align:middle line:84% So we need to go back and think about the derivative 00:35:54.950 --> 00:35:56.610 align:middle line:90% of this equation. 00:35:56.610 --> 00:35:58.070 align:middle line:84% So this is our equation that we're 00:35:58.070 --> 00:36:01.660 align:middle line:90% going to use to solve for x. 00:36:01.660 --> 00:36:04.620 align:middle line:90% So g equals 0. 00:36:04.620 --> 00:36:09.030 align:middle line:90% That means dg is also 0. 00:36:09.030 --> 00:36:10.660 align:middle line:84% The solution g is always a constant. 00:36:10.660 --> 00:36:12.960 align:middle line:90% So d of 0 is 0. 00:36:12.960 --> 00:36:16.392 align:middle line:90% But then if we do-- 00:36:16.392 --> 00:36:17.850 align:middle line:84% you can think of this as two terms. 00:36:17.850 --> 00:36:23.610 align:middle line:84% There's dg, partial g partial p times dp. 00:36:23.610 --> 00:36:26.130 align:middle line:84% So I'm sneaking back in some 18.02 notation. 00:36:26.130 --> 00:36:29.910 align:middle line:84% But think of this as like a Jacobian matrix now. 00:36:29.910 --> 00:36:34.470 align:middle line:84% It's not necessarily a scalar derivative. 00:36:34.470 --> 00:36:37.192 align:middle line:84% So this is the derivative aspect of dp, 00:36:37.192 --> 00:36:38.400 align:middle line:90% the change in the parameters. 00:36:38.400 --> 00:36:41.670 align:middle line:84% There's also the derivative with respectto x, which, again-- 00:36:41.670 --> 00:36:44.280 align:middle line:84% think of this as an n by n Jacobian matrix 00:36:44.280 --> 00:36:47.280 align:middle line:84% of the changes in all n components of g 00:36:47.280 --> 00:36:50.160 align:middle line:90% with all n components of x dx. 00:36:50.160 --> 00:36:53.160 align:middle line:84% So we're just linearizing this nonlinear function in two 00:36:53.160 --> 00:36:54.780 align:middle line:90% variables, which we haven't-- 00:36:54.780 --> 00:36:56.740 align:middle line:84% two parameters, which we haven't done so far. 00:36:56.740 --> 00:36:59.890 align:middle line:84% So I'm just kind of introducing this notation. 00:36:59.890 --> 00:37:05.805 align:middle line:84% So if we write this down, and we just solve it for dx, 00:37:05.805 --> 00:37:09.400 align:middle line:84% I move this to the right hand side, 00:37:09.400 --> 00:37:12.690 align:middle line:84% I get a minus this on the right hand side, 00:37:12.690 --> 00:37:16.822 align:middle line:84% and I get a Jacobian inverse times dp. 00:37:16.822 --> 00:37:17.780 align:middle line:90% Does everyone see this? 00:37:17.780 --> 00:37:18.930 align:middle line:90% So I took this. 00:37:18.930 --> 00:37:23.400 align:middle line:84% I linearized it with two Jacobian matrices with respect 00:37:23.400 --> 00:37:25.290 align:middle line:84% to the parameters, respects to the x's. 00:37:25.290 --> 00:37:29.430 align:middle line:84% And then I solve for dx in terms of dp. 00:37:29.430 --> 00:37:32.790 align:middle line:84% And you can see it's this equation. 00:37:32.790 --> 00:37:35.700 align:middle line:90% And then we plug that in for dx. 00:37:35.700 --> 00:37:37.920 align:middle line:84% And so what you get is you get f prime 00:37:37.920 --> 00:37:44.320 align:middle line:84% of x here with a minus sign from here 00:37:44.320 --> 00:37:49.240 align:middle line:84% times this inverse Jacobian matrix 00:37:49.240 --> 00:37:56.700 align:middle line:84% times this other Jacobian matrix of the parameters times dp. 00:37:56.700 --> 00:37:59.370 align:middle line:84% And notice that this Jacobian matrix is exactly 00:37:59.370 --> 00:38:01.680 align:middle line:84% the same Jacobian matrix you would use in the Newton 00:38:01.680 --> 00:38:04.290 align:middle line:84% solver for x So if you're solving nonlinear equations, 00:38:04.290 --> 00:38:07.140 align:middle line:90% you already have this Jacobian. 00:38:07.140 --> 00:38:10.382 align:middle line:84% And you're already solving systems with this-- 00:38:10.382 --> 00:38:12.090 align:middle line:84% you're inverting this Jacobian or solving 00:38:12.090 --> 00:38:14.430 align:middle line:90% a system of equations. 00:38:14.430 --> 00:38:16.860 align:middle line:84% And so, again, you do the same trick. 00:38:16.860 --> 00:38:20.040 align:middle line:84% The whole trick is, to compute df, you just 00:38:20.040 --> 00:38:22.910 align:middle line:90% multiply this left to right. 00:38:22.910 --> 00:38:26.210 align:middle line:90% So this is a row vector. 00:38:26.210 --> 00:38:28.280 align:middle line:90% This is an inverse Jacobian. 00:38:28.280 --> 00:38:32.600 align:middle line:84% You call this-- row vector times matrix is a row vector. 00:38:32.600 --> 00:38:34.530 align:middle line:90% Call that v transpose. 00:38:34.530 --> 00:38:37.160 align:middle line:84% So multiplying this by the inverse 00:38:37.160 --> 00:38:40.250 align:middle line:84% is equivalent to solving a system of equations 00:38:40.250 --> 00:38:42.960 align:middle line:84% with the transpose of the Jacobian matrix. 00:38:42.960 --> 00:38:45.350 align:middle line:84% So again, we call this an adjoint equation 00:38:45.350 --> 00:38:48.530 align:middle line:84% or a transpose equation for the adjoint solution 00:38:48.530 --> 00:38:52.940 align:middle line:84% v. And on the right hand side is the transpose of this row 00:38:52.940 --> 00:38:53.720 align:middle line:90% vector f prime. 00:38:53.720 --> 00:38:59.390 align:middle line:90% 00:38:59.390 --> 00:39:04.430 align:middle line:84% So we solve-- first, we solve a nonlinear system of equations 00:39:04.430 --> 00:39:06.200 align:middle line:90% to get x. 00:39:06.200 --> 00:39:08.990 align:middle line:84% And we do that by a sequence of linear solves 00:39:08.990 --> 00:39:11.710 align:middle line:84% probably, using this inverse Jacobian matrix. 00:39:11.710 --> 00:39:13.880 align:middle line:90% We solve with many times. 00:39:13.880 --> 00:39:18.410 align:middle line:84% And then we solve one more linear solve just 00:39:18.410 --> 00:39:22.030 align:middle line:84% with the transposed Jacobian matrix. 00:39:22.030 --> 00:39:24.550 align:middle line:84% And then once you have that, then you 00:39:24.550 --> 00:39:27.010 align:middle line:84% can get the derivatives because this is just-- 00:39:27.010 --> 00:39:33.630 align:middle line:84% if you want df/dp, it's just a dot product 00:39:33.630 --> 00:39:36.855 align:middle line:84% of this transpose with dg/dp times-- 00:39:36.855 --> 00:39:40.440 align:middle line:90% 00:39:40.440 --> 00:39:43.030 align:middle line:90% the dp cancels. 00:39:43.030 --> 00:39:45.910 align:middle line:84% That's it, so it's just a bunch of dot products. 00:39:45.910 --> 00:39:47.480 align:middle line:90% So this is called-- 00:39:47.480 --> 00:39:51.670 align:middle line:84% by the way, this formula where you linearize the nonlinear 00:39:51.670 --> 00:39:54.580 align:middle line:84% equations is also called the implicit function theorem. 00:39:54.580 --> 00:39:58.360 align:middle line:90% And so these are-- 00:39:58.360 --> 00:39:59.950 align:middle line:90% this is all the rage now. 00:39:59.950 --> 00:40:02.530 align:middle line:84% You have what's called a bilevel optimization problem 00:40:02.530 --> 00:40:05.080 align:middle line:84% or problems with optimizing problems 00:40:05.080 --> 00:40:08.350 align:middle line:90% with implicit equations. 00:40:08.350 --> 00:40:12.790 align:middle line:84% And it's really, really powerful to be able to do this. 00:40:12.790 --> 00:40:15.122 align:middle line:84% And actually, the automatic differentiation systems 00:40:15.122 --> 00:40:17.080 align:middle line:84% usually have to be told that you're doing this. 00:40:17.080 --> 00:40:23.320 align:middle line:84% They don't know-- if you solve this nonlinear system 00:40:23.320 --> 00:40:28.240 align:middle line:84% by a sequence of Newton's steps, the automatic differentiation 00:40:28.240 --> 00:40:29.980 align:middle line:84% doesn't know that what you're solving 00:40:29.980 --> 00:40:31.977 align:middle line:84% is g equals 0 by Newton's method. 00:40:31.977 --> 00:40:34.060 align:middle line:84% It thinks, oh, you're just doing a bunch of steps, 00:40:34.060 --> 00:40:37.390 align:middle line:84% and it tries to do the chain rule, propagate it through all 00:40:37.390 --> 00:40:39.040 align:middle line:90% those Newton's steps. 00:40:39.040 --> 00:40:41.980 align:middle line:84% So it's basically trying to exactly differentiate 00:40:41.980 --> 00:40:45.060 align:middle line:84% the error in your Newton's solver. 00:40:45.060 --> 00:40:49.800 align:middle line:84% If you tell it, if you know that you're solving a nonlinear 00:40:49.800 --> 00:40:52.390 align:middle line:84% systems, you know that this is what you should be doing. 00:40:52.390 --> 00:40:54.660 align:middle line:84% You should really only be for the adjoint problem, 00:40:54.660 --> 00:40:57.450 align:middle line:84% for the back propagation, or for reverse mode, 00:40:57.450 --> 00:41:00.690 align:middle line:84% you should really only do effectively one Newton step, 00:41:00.690 --> 00:41:02.830 align:middle line:90% one linear solve. 00:41:02.830 --> 00:41:07.510 align:middle line:84% So it's really important to be able to do this basically 00:41:07.510 --> 00:41:10.680 align:middle line:84% to know that you have to tell the automatic differentiation 00:41:10.680 --> 00:41:12.000 align:middle line:90% to do this either. 00:41:12.000 --> 00:41:14.850 align:middle line:84% You either supply a manual derivative of this part, 00:41:14.850 --> 00:41:18.390 align:middle line:84% or there are now ways to basically tell 00:41:18.390 --> 00:41:24.036 align:middle line:84% the automatic differentiation solver that one step of it, 00:41:24.036 --> 00:41:25.710 align:middle line:90% is solving a nonlinear system. 00:41:25.710 --> 00:41:27.750 align:middle line:90% And it should do this for you. 00:41:27.750 --> 00:41:30.910 align:middle line:90% 00:41:30.910 --> 00:41:32.780 align:middle line:90% Yes, that's my last slide here. 00:41:32.780 --> 00:41:35.747 align:middle line:84% So even if you're using automatic differentiation, 00:41:35.747 --> 00:41:38.080 align:middle line:84% and the computer is taking all your derivatives for you, 00:41:38.080 --> 00:41:40.987 align:middle line:84% doing all your chain rules and so forth, it's 00:41:40.987 --> 00:41:43.570 align:middle line:84% important to have some idea of what's going on under the hood. 00:41:43.570 --> 00:41:44.770 align:middle line:90% Understand adjoint methods. 00:41:44.770 --> 00:41:48.510 align:middle line:84% Understand reverse mode versus forward mode. 00:41:48.510 --> 00:41:51.160 align:middle line:84% Because, first of all, you need to know what is forward mode? 00:41:51.160 --> 00:41:52.420 align:middle line:90% What is reverse mode? 00:41:52.420 --> 00:41:54.250 align:middle line:90% When should you use forward? 00:41:54.250 --> 00:41:56.800 align:middle line:90% When should you use reverse? 00:41:56.800 --> 00:41:58.390 align:middle line:84% Another problem is that if you're 00:41:58.390 --> 00:42:01.420 align:middle line:84% doing you know physics, especially, a lot of physics 00:42:01.420 --> 00:42:04.150 align:middle line:84% methods, you're calling large software packages written 00:42:04.150 --> 00:42:06.250 align:middle line:84% over decades in various languages 00:42:06.250 --> 00:42:11.230 align:middle line:84% that can't be differentiated by automatic differentiation. 00:42:11.230 --> 00:42:12.770 align:middle line:84% Jax can understand Python programs, 00:42:12.770 --> 00:42:14.770 align:middle line:84% but it doesn't really understand Python programs 00:42:14.770 --> 00:42:17.140 align:middle line:84% that call libraries written in Fortran. 00:42:17.140 --> 00:42:20.510 align:middle line:84% So, often, you need-- some parts of it, it just can't handle. 00:42:20.510 --> 00:42:23.110 align:middle line:84% And so but it turns out you only need 00:42:23.110 --> 00:42:27.760 align:middle line:84% to write down the derivative for that part or a vector Jacobian 00:42:27.760 --> 00:42:29.840 align:middle line:84% product, what's called, for that part. 00:42:29.840 --> 00:42:32.090 align:middle line:84% And then it can do the rest of the chain rule for you. 00:42:32.090 --> 00:42:35.390 align:middle line:84% But you really need to know these adjoint methods in order 00:42:35.390 --> 00:42:37.720 align:middle line:90% to do that in a reasonable way. 00:42:37.720 --> 00:42:40.240 align:middle line:84% And then even if you have Ad, even 00:42:40.240 --> 00:42:42.370 align:middle line:84% if it works, if you have a model that 00:42:42.370 --> 00:42:44.560 align:middle line:84% involves an approximate calculation, like a Newton 00:42:44.560 --> 00:42:47.837 align:middle line:84% solve or something like that, automatic differentiation 00:42:47.837 --> 00:42:49.420 align:middle line:84% typically doesn't know this and spends 00:42:49.420 --> 00:42:52.450 align:middle line:84% a lot of effort trying to differentiate exactly the error 00:42:52.450 --> 00:42:53.680 align:middle line:90% in your approximation. 00:42:53.680 --> 00:42:54.770 align:middle line:90% It wastes a lot of time. 00:42:54.770 --> 00:42:59.260 align:middle line:84% So it's really, really useful to know that 00:42:59.260 --> 00:43:03.340 align:middle line:84% and to know, oh, I need to tell it something to help it. 00:43:03.340 --> 00:43:06.340 align:middle line:84% So examples when you're solving nonlinear systems 00:43:06.340 --> 00:43:08.710 align:middle line:84% by iterative methods like Newton equations, 00:43:08.710 --> 00:43:13.360 align:middle line:84% or even if you're approximating things like by discretizing 00:43:13.360 --> 00:43:15.880 align:middle line:84% physics, like finite element methods, 00:43:15.880 --> 00:43:18.550 align:middle line:84% often you want to differentiate the physics before you 00:43:18.550 --> 00:43:20.170 align:middle line:84% discretize, rather than have it try 00:43:20.170 --> 00:43:25.210 align:middle line:84% to do automatic differentiation on the discretization itself. 00:43:25.210 --> 00:43:27.050 align:middle line:90% Anyway, so I'll stop there. 00:43:27.050 --> 00:43:33.060 align:middle line:84% And we'll be spending more time on the nuts and bolts 00:43:33.060 --> 00:43:35.220 align:middle line:90% of automatic differentiation. 00:43:35.220 --> 00:43:37.020 align:middle line:84% And I'll also give some more links 00:43:37.020 --> 00:43:40.110 align:middle line:84% to further reading on adjoint methods and backward mode 00:43:40.110 --> 00:43:43.890 align:middle line:84% and vector Jacobian products and these kinds of-- 00:43:43.890 --> 00:43:47.640 align:middle line:90% these kinds of systems I've-- 00:43:47.640 --> 00:43:49.500 align:middle line:84% there's lots of notes written in web pages 00:43:49.500 --> 00:43:50.542 align:middle line:90% on these kinds of things. 00:43:50.542 --> 00:43:52.980 align:middle line:84% ALAN EDELMAN: And I'll be back live on Friday. 00:43:52.980 --> 00:43:54.250 align:middle line:90% STEVEN G. JOHNSON: Yes. 00:43:54.250 --> 00:43:58.120 align:middle line:84% And so maybe Friday, I don't know if you're 00:43:58.120 --> 00:43:59.488 align:middle line:90% going to do your dual numbers. 00:43:59.488 --> 00:44:01.030 align:middle line:84% ALAN EDELMAN: I could do dual numbers 00:44:01.030 --> 00:44:02.920 align:middle line:84% and even reverse and forward mode 00:44:02.920 --> 00:44:04.860 align:middle line:90% the graph theory if you want. 00:44:04.860 --> 00:44:05.860 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:44:05.860 --> 00:44:08.320 align:middle line:84% So maybe I think maybe it's time to do 00:44:08.320 --> 00:44:10.680 align:middle line:90% some automatic differentiation. 00:44:10.680 --> 00:44:12.850 align:middle line:90% ALAN EDELMAN: OK, sounds good. 00:44:12.850 --> 00:44:15.060 align:middle line:90% All right, see you then. 00:44:15.060 --> 00:44:26.000 align:middle line:90%