WEBVTT 00:00:00.000 --> 00:00:03.460 align:middle line:90% [AUDIO LOGO] 00:00:03.460 --> 00:00:03.960 align:middle line:90% 00:00:03.960 --> 00:00:10.900 align:middle line:90% [MOUSE CLICK] 00:00:10.900 --> 00:00:13.310 align:middle line:84% STEVEN JOHNSON: OK, so let's get started. 00:00:13.310 --> 00:00:17.890 align:middle line:84% So as I said, I'm going to do second derivatives, which 00:00:17.890 --> 00:00:22.550 align:middle line:84% is just going to be the derivative of the derivative. 00:00:22.550 --> 00:00:27.260 align:middle line:84% When you have functions from matrices 00:00:27.260 --> 00:00:29.000 align:middle line:84% to matrices or something like that, 00:00:29.000 --> 00:00:31.250 align:middle line:84% you have to think a little bit carefully just 00:00:31.250 --> 00:00:34.080 align:middle line:84% to make sure we understand what kind of thing 00:00:34.080 --> 00:00:36.530 align:middle line:90% the second derivative is. 00:00:36.530 --> 00:00:42.190 align:middle line:84% So remember when we take basically the first derivative, 00:00:42.190 --> 00:00:50.020 align:middle line:84% what we have is this linear operator, 00:00:50.020 --> 00:00:58.160 align:middle line:84% f primed of x, that takes in a little change 00:00:58.160 --> 00:01:05.630 align:middle line:84% and gives us a df, which is the f of x plus dx 00:01:05.630 --> 00:01:12.330 align:middle line:84% minus f of x to first order, dropping higher-order terms. 00:01:12.330 --> 00:01:23.580 align:middle line:84% And so it's really natural to define the second derivative 00:01:23.580 --> 00:01:25.240 align:middle line:90% as the derivative of that. 00:01:25.240 --> 00:01:32.640 align:middle line:84% So what should f double prime be? 00:01:32.640 --> 00:01:36.400 align:middle line:90% 00:01:36.400 --> 00:01:45.940 align:middle line:84% f double primed of x should take in-- 00:01:45.940 --> 00:01:47.680 align:middle line:90% let me call it dx prime. 00:01:47.680 --> 00:01:49.390 align:middle line:90% I mean, that's not a derivative. 00:01:49.390 --> 00:01:53.060 align:middle line:90% It just means a different-- 00:01:53.060 --> 00:01:56.390 align:middle line:90% we put it in here in red. 00:01:56.390 --> 00:02:01.055 align:middle line:84% This is going to be a different small change. 00:02:01.055 --> 00:02:03.840 align:middle line:90% 00:02:03.840 --> 00:02:06.393 align:middle line:84% So prime here is not derivative, it's just different. 00:02:06.393 --> 00:02:08.060 align:middle line:84% We call it tilde or something like that. 00:02:08.060 --> 00:02:12.500 align:middle line:84% But we overload the prime a lot in mathematics. 00:02:12.500 --> 00:02:14.190 align:middle line:90% So what should it be? 00:02:14.190 --> 00:02:16.377 align:middle line:84% Well, it should be the derivative. 00:02:16.377 --> 00:02:18.210 align:middle line:84% I mean, it should be exactly the same thing. 00:02:18.210 --> 00:02:23.090 align:middle line:90% It should be df primed. 00:02:23.090 --> 00:02:34.360 align:middle line:84% So it should be f primed of x plus dx primed minus f 00:02:34.360 --> 00:02:36.070 align:middle line:90% prime of x. 00:02:36.070 --> 00:02:39.130 align:middle line:90% 00:02:39.130 --> 00:02:41.910 align:middle line:84% So it should look exactly the same. 00:02:41.910 --> 00:02:44.530 align:middle line:84% But now let's think about what this means. 00:02:44.530 --> 00:02:49.000 align:middle line:84% So this f prime here is not a number anymore. 00:02:49.000 --> 00:02:50.790 align:middle line:84% It could be a number, but in general, it's 00:02:50.790 --> 00:02:52.830 align:middle line:90% a linear operator. 00:02:52.830 --> 00:02:59.100 align:middle line:84% So this here is the difference of two linear operators. 00:02:59.100 --> 00:03:09.380 align:middle line:90% 00:03:09.380 --> 00:03:14.540 align:middle line:84% And what does it mean to add and subtract and take differences 00:03:14.540 --> 00:03:16.500 align:middle line:90% of linear operators? 00:03:16.500 --> 00:03:18.680 align:middle line:84% And we did a little bit of this in problem set 1. 00:03:18.680 --> 00:03:20.960 align:middle line:84% You can think of linear operators themselves 00:03:20.960 --> 00:03:23.480 align:middle line:90% as a vector space. 00:03:23.480 --> 00:03:27.620 align:middle line:84% Just like we can take functions, sine x plus cosine x 00:03:27.620 --> 00:03:29.030 align:middle line:90% is another function. 00:03:29.030 --> 00:03:33.650 align:middle line:84% So we can think of linear operators as a vector space 00:03:33.650 --> 00:03:34.430 align:middle line:90% as well. 00:03:34.430 --> 00:03:36.860 align:middle line:90% So if we have linear operators-- 00:03:36.860 --> 00:03:47.390 align:middle line:90% 00:03:47.390 --> 00:03:56.680 align:middle line:84% operators L1 and L2, then what we mean by their addition, L1 00:03:56.680 --> 00:04:05.280 align:middle line:84% plus L2, or plus or minus, is the linear operator 00:04:05.280 --> 00:04:16.519 align:middle line:84% that takes a vector v and gives you L1 of V plus L2 of V. 00:04:16.519 --> 00:04:18.970 align:middle line:84% And the same thing if you multiply a linear operator 00:04:18.970 --> 00:04:24.800 align:middle line:84% by a scalar, that's the linear operator 00:04:24.800 --> 00:04:33.680 align:middle line:84% that sends a vector V to alpha times L1 of V. 00:04:33.680 --> 00:04:35.180 align:middle line:90% And this should be familiar. 00:04:35.180 --> 00:04:37.880 align:middle line:84% So if I take two matrices and I add them, 00:04:37.880 --> 00:04:39.380 align:middle line:84% you'd actually know what that means. 00:04:39.380 --> 00:04:42.920 align:middle line:84% You say, oh, that means you add the elements up element-wise. 00:04:42.920 --> 00:04:45.260 align:middle line:84% But where does that rule come from? 00:04:45.260 --> 00:04:48.110 align:middle line:84% That adding up two matrices element-wise 00:04:48.110 --> 00:04:51.470 align:middle line:84% is exactly the new matrix that, acting on a vector, 00:04:51.470 --> 00:04:53.570 align:middle line:84% gives you the first matrix times the vector 00:04:53.570 --> 00:04:55.730 align:middle line:84% plus the second matrix times the vector. 00:04:55.730 --> 00:04:58.495 align:middle line:84% If I take a matrix and multiply it by 3, 00:04:58.495 --> 00:05:00.620 align:middle line:84% you can say, oh, you multiply all the entries by 3. 00:05:00.620 --> 00:05:01.460 align:middle line:90% But why? 00:05:01.460 --> 00:05:05.580 align:middle line:84% It's because if I take-- that's the linear operator that 00:05:05.580 --> 00:05:08.240 align:middle line:84% takes a vector, it's equivalent to multiplying it 00:05:08.240 --> 00:05:11.308 align:middle line:84% by that original matrix and then multiplying it by 3. 00:05:11.308 --> 00:05:13.100 align:middle line:84% So that's where those rules come from, just 00:05:13.100 --> 00:05:15.560 align:middle line:84% like the rule for multiplying two matrices-- 00:05:15.560 --> 00:05:17.750 align:middle line:90% oh, rows times columns. 00:05:17.750 --> 00:05:18.350 align:middle line:90% Why? 00:05:18.350 --> 00:05:20.030 align:middle line:84% Because that's the new linear operator 00:05:20.030 --> 00:05:23.630 align:middle line:84% that's equivalent to multiplying the first matrix 00:05:23.630 --> 00:05:25.940 align:middle line:84% by the right matrix and then by the left matrix. 00:05:25.940 --> 00:05:27.560 align:middle line:84% That's where these rules come from. 00:05:27.560 --> 00:05:32.130 align:middle line:90% So this thing here is now-- 00:05:32.130 --> 00:05:40.730 align:middle line:84% so what we really mean by this is this is a linear operator, 00:05:40.730 --> 00:05:52.650 align:middle line:84% so that this is the difference of two linear operators. 00:05:52.650 --> 00:05:54.660 align:middle line:84% These are linear operators that take in a dx 00:05:54.660 --> 00:05:59.190 align:middle line:84% and give you something else, give you a df. 00:05:59.190 --> 00:06:01.140 align:middle line:84% So the difference of these two things 00:06:01.140 --> 00:06:03.180 align:middle line:84% is a linear operator that takes in a dx-- 00:06:03.180 --> 00:06:04.890 align:middle line:90% not a dx prime, a dx-- 00:06:04.890 --> 00:06:08.340 align:middle line:90% and gives you something else. 00:06:08.340 --> 00:06:12.610 align:middle line:84% So this is a linear operator that takes in-- 00:06:12.610 --> 00:06:19.500 align:middle line:84% we can call it f double prime of x. 00:06:19.500 --> 00:06:29.300 align:middle line:84% It takes in a dx primed that takes in another input. 00:06:29.300 --> 00:06:32.180 align:middle line:84% We can take it, write it as it takes in an input dx. 00:06:32.180 --> 00:06:35.960 align:middle line:90% 00:06:35.960 --> 00:06:46.190 align:middle line:84% And what it gives you is f primed of x plus dx primed, 00:06:46.190 --> 00:07:00.230 align:middle line:84% acting on dx, minus f prime of x acting on dx. 00:07:00.230 --> 00:07:06.260 align:middle line:90% 00:07:06.260 --> 00:07:07.430 align:middle line:90% Does everyone see? 00:07:07.430 --> 00:07:11.410 align:middle line:84% So this f double prime is going to be-- 00:07:11.410 --> 00:07:15.340 align:middle line:84% it's just copying down the formula for the derivative, 00:07:15.340 --> 00:07:16.720 align:middle line:90% except applying it to f prime. 00:07:16.720 --> 00:07:19.690 align:middle line:84% But then-- so this is the linear operator 00:07:19.690 --> 00:07:23.560 align:middle line:84% you get when you take the original linear operator 00:07:23.560 --> 00:07:27.850 align:middle line:84% at a slightly shifted x minus f prime of x. 00:07:27.850 --> 00:07:31.060 align:middle line:84% So this is a linear operator that acts on a dx. 00:07:31.060 --> 00:07:33.580 align:middle line:84% And what it is, it's just the linear operator 00:07:33.580 --> 00:07:36.370 align:middle line:84% that takes this on dx minus this on dx. 00:07:36.370 --> 00:07:38.140 align:middle line:84% But then if you think about this object, 00:07:38.140 --> 00:07:42.210 align:middle line:84% now, this object now takes two inputs. 00:07:42.210 --> 00:07:49.500 align:middle line:84% It takes the change x where dx prime is the change in where 00:07:49.500 --> 00:07:51.870 align:middle line:90% you're taking the derivative. 00:07:51.870 --> 00:07:56.970 align:middle line:84% And then the second one, dx, is the change 00:07:56.970 --> 00:08:01.010 align:middle line:84% in x that you're acting the derivative, f primed, on. 00:08:01.010 --> 00:08:03.410 align:middle line:84% So this is called a bilinear form. 00:08:03.410 --> 00:08:11.090 align:middle line:90% 00:08:11.090 --> 00:08:13.730 align:middle line:84% So we have-- we're going to have f double 00:08:13.730 --> 00:08:18.770 align:middle line:84% primed that's taking of x that takes 00:08:18.770 --> 00:08:26.620 align:middle line:90% two inputs, dx prime, and dx. 00:08:26.620 --> 00:08:29.240 align:middle line:90% 00:08:29.240 --> 00:08:31.070 align:middle line:84% But writing those two pairs of brackets 00:08:31.070 --> 00:08:32.309 align:middle line:90% is a little bit annoying. 00:08:32.309 --> 00:08:34.970 align:middle line:84% So let me just write one pair of brackets 00:08:34.970 --> 00:08:36.500 align:middle line:90% and just give it two arguments. 00:08:36.500 --> 00:08:43.020 align:middle line:90% 00:08:43.020 --> 00:08:45.135 align:middle line:90% And so it acts on two vectors. 00:08:45.135 --> 00:08:51.010 align:middle line:90% 00:08:51.010 --> 00:08:52.465 align:middle line:90% And it's linear in both. 00:08:52.465 --> 00:08:55.390 align:middle line:90% 00:08:55.390 --> 00:08:58.230 align:middle line:90% So in general, a bilinear form-- 00:08:58.230 --> 00:09:02.270 align:middle line:90% 00:09:02.270 --> 00:09:05.390 align:middle line:84% and I said let me just remind you of linearity. 00:09:05.390 --> 00:09:13.800 align:middle line:84% You have a bilinear form, call it B, takes in a vector u, 00:09:13.800 --> 00:09:20.490 align:middle line:84% and it takes in a vector v. And linearity 00:09:20.490 --> 00:09:28.430 align:middle line:84% means if you take B of u1 plus u2, comma v, that 00:09:28.430 --> 00:09:36.460 align:middle line:84% had better equal B of u1, v plus B of u2, v. 00:09:36.460 --> 00:09:39.400 align:middle line:84% But it also has to be linear in the second argument. 00:09:39.400 --> 00:09:47.320 align:middle line:84% B of u, v1 plus v2 has to equal B of u, v1 00:09:47.320 --> 00:09:53.270 align:middle line:90% plus B of u v2, et cetera. 00:09:53.270 --> 00:09:55.540 align:middle line:84% You can also scale the-- multiply one of the arguments 00:09:55.540 --> 00:09:57.230 align:middle line:90% by 2, and so forth. 00:09:57.230 --> 00:10:03.960 align:middle line:84% If I multiply both of the arguments by 2, if I do B of, 00:10:03.960 --> 00:10:13.755 align:middle line:84% like, 2u times 3v, that had better be 2 times 3 B of u, 00:10:13.755 --> 00:10:16.530 align:middle line:84% v. So if I multiply both arguments by 2, 00:10:16.530 --> 00:10:18.030 align:middle line:84% it doesn't multiply the output by 2. 00:10:18.030 --> 00:10:19.005 align:middle line:90% It multiplies it by 4. 00:10:19.005 --> 00:10:24.140 align:middle line:90% 00:10:24.140 --> 00:10:26.000 align:middle line:84% So that's what this second derivative is. 00:10:26.000 --> 00:10:28.100 align:middle line:90% It takes in a change-- 00:10:28.100 --> 00:10:29.982 align:middle line:84% two changes, a change in where you're 00:10:29.982 --> 00:10:31.940 align:middle line:84% taking the derivative and a change in the thing 00:10:31.940 --> 00:10:33.980 align:middle line:84% that you're taking the derivative on. 00:10:33.980 --> 00:10:37.820 align:middle line:84% And you might wonder if it matters, like the order-- oops, 00:10:37.820 --> 00:10:42.180 align:middle line:84% and this should not have been a prime. 00:10:42.180 --> 00:10:46.410 align:middle line:84% If it matters, does the order of these two things matter? 00:10:46.410 --> 00:10:50.040 align:middle line:84% And in general for bilinear forms, the order matters. 00:10:50.040 --> 00:11:01.960 align:middle line:84% In general, for bilinear forms, B of u, 00:11:01.960 --> 00:11:10.410 align:middle line:90% v is not equal to B of v, u. 00:11:10.410 --> 00:11:12.420 align:middle line:84% And, in fact, it may not even make sense 00:11:12.420 --> 00:11:13.613 align:middle line:90% to swap the arguments. 00:11:13.613 --> 00:11:15.030 align:middle line:84% You can have a bilinear form where 00:11:15.030 --> 00:11:17.700 align:middle line:84% u lives in one vector space, and v lives in a completely 00:11:17.700 --> 00:11:19.180 align:middle line:90% different vector space. 00:11:19.180 --> 00:11:21.990 align:middle line:84% So u is a scalar, and v is a column vector, 00:11:21.990 --> 00:11:25.690 align:middle line:84% and it doesn't even-- you can't even swap them. 00:11:25.690 --> 00:11:33.080 align:middle line:84% So I said this may not even make sense. 00:11:33.080 --> 00:11:36.870 align:middle line:84% But here, dx and dx prime, they clearly 00:11:36.870 --> 00:11:38.120 align:middle line:90% live in the same vector space. 00:11:38.120 --> 00:11:39.860 align:middle line:90% They both changes to x. 00:11:39.860 --> 00:11:44.510 align:middle line:84% And so you could swap them and ask, what's the change? 00:11:44.510 --> 00:11:48.060 align:middle line:84% And, in fact-- but here, it turns out they are the same. 00:11:48.060 --> 00:12:04.210 align:middle line:84% So here, it turns out that f double prime of dx primed-- 00:12:04.210 --> 00:12:07.300 align:middle line:84% I'm making my color scheme right. 00:12:07.300 --> 00:12:08.920 align:middle line:90% I was using black, right? 00:12:08.920 --> 00:12:12.070 align:middle line:84% f double prime of x-- there's the point at which 00:12:12.070 --> 00:12:14.110 align:middle line:84% you're evaluating the second derivative-- 00:12:14.110 --> 00:12:21.530 align:middle line:84% of dx primed comma dx always, in fact, 00:12:21.530 --> 00:12:32.970 align:middle line:84% equals f double primed of x dx comma dx primed. 00:12:32.970 --> 00:12:35.810 align:middle line:84% And so this is what's called a symmetric. 00:12:35.810 --> 00:12:46.110 align:middle line:90% 00:12:46.110 --> 00:12:50.950 align:middle line:84% And we can show it pretty easily just from the definition. 00:12:50.950 --> 00:12:54.535 align:middle line:84% So in fact, as I'll show in a minute, 00:12:54.535 --> 00:12:56.910 align:middle line:84% this is actually-- you've seen this kind of thing before, 00:12:56.910 --> 00:12:58.330 align:middle line:90% and you didn't realize it. 00:12:58.330 --> 00:13:01.080 align:middle line:84% So you learned in 1802, points of variable calculus, 00:13:01.080 --> 00:13:04.200 align:middle line:84% that when I take a partial derivative, like partial f, 00:13:04.200 --> 00:13:07.200 align:middle line:84% partial x, partial y, like two n's, I can swap them, 00:13:07.200 --> 00:13:08.540 align:middle line:90% and it doesn't matter. 00:13:08.540 --> 00:13:14.130 align:middle line:84% It turns out that is going to be a special case of this. 00:13:14.130 --> 00:13:17.140 align:middle line:84% So let me show it in general just from the definition. 00:13:17.140 --> 00:13:18.570 align:middle line:90% So why is this? 00:13:18.570 --> 00:13:22.750 align:middle line:90% 00:13:22.750 --> 00:13:24.100 align:middle line:90% So why symmetric? 00:13:24.100 --> 00:13:27.270 align:middle line:90% 00:13:27.270 --> 00:13:29.850 align:middle line:84% And so we just need to write out the definition a little bit. 00:13:29.850 --> 00:13:37.388 align:middle line:84% So just write out-- so f double primed of x of dx primed dx-- 00:13:37.388 --> 00:13:39.180 align:middle line:84% I'm not going to use colors here because it 00:13:39.180 --> 00:13:42.960 align:middle line:84% gets too annoying to swap pens back and forth. 00:13:42.960 --> 00:13:47.440 align:middle line:90% OK, so there's two terms here. 00:13:47.440 --> 00:13:51.630 align:middle line:84% So there's a term that comes from f prime 00:13:51.630 --> 00:14:03.650 align:middle line:84% of x plus dx primed acting on dx minus f of x f 00:14:03.650 --> 00:14:06.618 align:middle line:90% primed of x acting on dx. 00:14:06.618 --> 00:14:08.285 align:middle line:84% That's just the definition I did before. 00:14:08.285 --> 00:14:10.820 align:middle line:90% 00:14:10.820 --> 00:14:14.030 align:middle line:84% f double primed is the difference of these two 00:14:14.030 --> 00:14:14.780 align:middle line:90% linear operators. 00:14:14.780 --> 00:14:16.580 align:middle line:84% So acting on dx is the difference 00:14:16.580 --> 00:14:19.060 align:middle line:90% of what they do on the dx. 00:14:19.060 --> 00:14:20.780 align:middle line:84% But now let's take it one step further. 00:14:20.780 --> 00:14:21.822 align:middle line:90% I want you to expand out. 00:14:21.822 --> 00:14:23.320 align:middle line:84% What's the definition of f prime? 00:14:23.320 --> 00:14:24.670 align:middle line:90% Let's go back to that. 00:14:24.670 --> 00:14:29.037 align:middle line:84% f primed of-- f primed, and let's do the second one. 00:14:29.037 --> 00:14:29.620 align:middle line:90% That's easier. 00:14:29.620 --> 00:14:34.470 align:middle line:90% 00:14:34.470 --> 00:14:36.740 align:middle line:90% So the second term-- 00:14:36.740 --> 00:14:38.510 align:middle line:90% what's f primed of x dx? 00:14:38.510 --> 00:14:42.230 align:middle line:84% We said in the very first lecture, 00:14:42.230 --> 00:14:47.930 align:middle line:84% that's the same thing as f of x plus dx minus f of x. 00:14:47.930 --> 00:14:50.690 align:middle line:90% That's what that is. 00:14:50.690 --> 00:14:56.500 align:middle line:84% So what's f primed of x plus dx primed dx? 00:14:56.500 --> 00:15:07.020 align:middle line:84% It's f of x plus dx primed plus dx minus f of x plus dx primed. 00:15:07.020 --> 00:15:10.060 align:middle line:90% 00:15:10.060 --> 00:15:12.400 align:middle line:84% Now, let me just regroup these terms a little bit. 00:15:12.400 --> 00:15:15.310 align:middle line:84% So I have one term that looks like f 00:15:15.310 --> 00:15:18.430 align:middle line:90% of x plus dx primed plus dx. 00:15:18.430 --> 00:15:21.020 align:middle line:90% 00:15:21.020 --> 00:15:26.510 align:middle line:84% And I have another term that looks like just a plus f of x. 00:15:26.510 --> 00:15:28.530 align:middle line:90% That's not very interesting. 00:15:28.530 --> 00:15:34.340 align:middle line:84% And then I have another term that looks like f of x plus dx 00:15:34.340 --> 00:15:35.780 align:middle line:90% with a minus sign. 00:15:35.780 --> 00:15:37.370 align:middle line:84% And I have another term that looks 00:15:37.370 --> 00:15:40.570 align:middle line:90% like f of x plus dx primed. 00:15:40.570 --> 00:15:45.280 align:middle line:90% 00:15:45.280 --> 00:15:50.410 align:middle line:84% But now, if you stare at this clearly, 00:15:50.410 --> 00:15:52.540 align:middle line:84% a vector addition is commutative. 00:15:52.540 --> 00:15:53.950 align:middle line:90% I can swap those. 00:15:53.950 --> 00:16:00.380 align:middle line:90% I can swap these two terms. 00:16:00.380 --> 00:16:03.500 align:middle line:84% I can swap the addition and the input. 00:16:03.500 --> 00:16:05.510 align:middle line:84% I can also swap the addition of the output. 00:16:05.510 --> 00:16:08.780 align:middle line:84% I can just rearrange these two terms. 00:16:08.780 --> 00:16:12.050 align:middle line:84% And so this whole thing is exactly what-- 00:16:12.050 --> 00:16:14.990 align:middle line:84% the same as what you would get as if I took f double 00:16:14.990 --> 00:16:19.310 align:middle line:84% primed of x and evaluated it, and I swapped the outputs. 00:16:19.310 --> 00:16:26.750 align:middle line:90% 00:16:26.750 --> 00:16:28.780 align:middle line:84% So it's just writing out the definition. 00:16:28.780 --> 00:16:31.930 align:middle line:84% This does not look symmetric because here, the dx 00:16:31.930 --> 00:16:35.360 align:middle line:84% prime is in the argument, and dx is what f is acting on. 00:16:35.360 --> 00:16:38.090 align:middle line:84% But when you write out the definition of the derivative, 00:16:38.090 --> 00:16:41.350 align:middle line:84% then you see that both dx and dx prime appear in the same way. 00:16:41.350 --> 00:16:43.900 align:middle line:90% 00:16:43.900 --> 00:16:53.215 align:middle line:84% So let's do-- so the 1801 example is very easy. 00:16:53.215 --> 00:16:54.590 align:middle line:84% The first derivative is a number, 00:16:54.590 --> 00:16:56.382 align:middle line:84% and the second derivative is also a number. 00:16:56.382 --> 00:16:57.710 align:middle line:90% That's kind of boring. 00:16:57.710 --> 00:17:00.998 align:middle line:84% So it's the familiar-- this is a strict generalization of what 00:17:00.998 --> 00:17:01.790 align:middle line:90% you learned before. 00:17:01.790 --> 00:17:03.140 align:middle line:90% It's nothing new. 00:17:03.140 --> 00:17:07.090 align:middle line:84% But let's do an 1802, multivariable calculus. 00:17:07.090 --> 00:17:10.390 align:middle line:90% 00:17:10.390 --> 00:17:17.949 align:middle line:84% So suppose we have a scalar function f of x. 00:17:17.949 --> 00:17:24.359 align:middle line:90% 00:17:24.359 --> 00:17:26.625 align:middle line:84% All right, so the output is a scalar. 00:17:26.625 --> 00:17:29.200 align:middle line:90% 00:17:29.200 --> 00:17:34.190 align:middle line:90% But the input is going to be-- 00:17:34.190 --> 00:17:37.240 align:middle line:90% this is going to be in R n. 00:17:37.240 --> 00:17:38.800 align:middle line:84% So this is an n component vector. 00:17:38.800 --> 00:17:41.920 align:middle line:90% 00:17:41.920 --> 00:17:45.700 align:middle line:84% Now, you can think of it as a column vector if you want. 00:17:45.700 --> 00:17:51.940 align:middle line:90% 00:17:51.940 --> 00:17:53.480 align:middle line:90% So then what do we know? 00:17:53.480 --> 00:18:00.190 align:middle line:84% So we know that f primed of x is a row vector. 00:18:00.190 --> 00:18:01.780 align:middle line:84% It's the transpose of the gradient. 00:18:01.780 --> 00:18:05.160 align:middle line:90% 00:18:05.160 --> 00:18:12.920 align:middle line:84% So that way, f prime of x times dx is a scalar. 00:18:12.920 --> 00:18:14.810 align:middle line:90% That's the only way to get-- 00:18:14.810 --> 00:18:17.060 align:middle line:84% that's the only linear operator that takes a vector in 00:18:17.060 --> 00:18:18.720 align:middle line:90% and scalar out. 00:18:18.720 --> 00:18:27.440 align:middle line:84% So now let's think about what is our second derivative, 00:18:27.440 --> 00:18:29.150 align:middle line:90% our f primed of x. 00:18:29.150 --> 00:18:34.370 align:middle line:84% It has to take in two vectors, dx prime and dx. 00:18:34.370 --> 00:18:37.535 align:middle line:84% Now, it doesn't really matter in which order I put the prime. 00:18:37.535 --> 00:18:40.050 align:middle line:90% 00:18:40.050 --> 00:18:42.030 align:middle line:90% But it has to give a scalar. 00:18:42.030 --> 00:18:45.280 align:middle line:90% 00:18:45.280 --> 00:18:50.350 align:middle line:84% In the end, the output has to match. 00:18:50.350 --> 00:18:53.590 align:middle line:84% If you plug in both these things, it matches f. 00:18:53.590 --> 00:18:57.400 align:middle line:84% You can't just go back to the definition 00:18:57.400 --> 00:19:03.700 align:middle line:84% that f primed takes in a vector and gives you 00:19:03.700 --> 00:19:08.680 align:middle line:84% the same output as f because it gives you the df. 00:19:08.680 --> 00:19:12.160 align:middle line:84% So f double primed, when you plug in both vectors, 00:19:12.160 --> 00:19:17.680 align:middle line:84% or dx prime and dx, it has to give the same output as f 00:19:17.680 --> 00:19:19.480 align:middle line:90% plug in of dx. 00:19:19.480 --> 00:19:22.150 align:middle line:84% So it has to give something the same shape as f. 00:19:22.150 --> 00:19:27.290 align:middle line:84% So this takes in two vectors and gives you a scalar. 00:19:27.290 --> 00:19:30.870 align:middle line:90% 00:19:30.870 --> 00:19:37.290 align:middle line:84% And it turns out there's only one kind of way to write down 00:19:37.290 --> 00:19:40.440 align:middle line:84% an operation that takes in two column vectors 00:19:40.440 --> 00:19:44.370 align:middle line:84% and outputs a scalar and is linear in both of the vectors. 00:19:44.370 --> 00:19:54.150 align:middle line:84% And that is to put the matrix here and put a dx here 00:19:54.150 --> 00:19:59.640 align:middle line:84% and a dx plug in transposed over there. 00:19:59.640 --> 00:20:01.530 align:middle line:90% And it has to be symmetric. 00:20:01.530 --> 00:20:05.310 align:middle line:90% 00:20:05.310 --> 00:20:07.550 align:middle line:84% We know it has to be the same thing if you 00:20:07.550 --> 00:20:12.015 align:middle line:90% swap the dx and dx prime. 00:20:12.015 --> 00:20:16.580 align:middle line:90% 00:20:16.580 --> 00:20:19.760 align:middle line:84% But that means, actually, it has to be a symmetric matrix 00:20:19.760 --> 00:20:23.090 align:middle line:90% because this is also a number. 00:20:23.090 --> 00:20:25.130 align:middle line:84% So it equals the transpose of itself. 00:20:25.130 --> 00:20:30.300 align:middle line:90% 00:20:30.300 --> 00:20:31.980 align:middle line:90% This is true for any number. 00:20:31.980 --> 00:20:33.880 align:middle line:84% You can always equal the transpose itself. 00:20:33.880 --> 00:20:41.050 align:middle line:84% So that equals dx transpose h transpose dx primed. 00:20:41.050 --> 00:20:44.670 align:middle line:90% So these two have to be equal. 00:20:44.670 --> 00:20:58.810 align:middle line:84% So you have a symmetric n by n matrix H, which 00:20:58.810 --> 00:21:00.595 align:middle line:90% is called the Hessian matrix. 00:21:00.595 --> 00:21:05.060 align:middle line:90% 00:21:05.060 --> 00:21:08.270 align:middle line:84% How many of you have heard of Hessian matrices before? 00:21:08.270 --> 00:21:10.070 align:middle line:90% Yeah, it's a fair number of you. 00:21:10.070 --> 00:21:13.220 align:middle line:84% So yeah, so that's what a bilinear form looks 00:21:13.220 --> 00:21:15.140 align:middle line:90% like acting on column vectors. 00:21:15.140 --> 00:21:16.325 align:middle line:90% It's a matrix. 00:21:16.325 --> 00:21:20.020 align:middle line:90% 00:21:20.020 --> 00:21:26.440 align:middle line:84% But it's nice to write it in 1802 terms more explicitly, 00:21:26.440 --> 00:21:28.870 align:middle line:84% like component-wise, just like the gradient 00:21:28.870 --> 00:21:31.250 align:middle line:84% you need to know when you first learn it. 00:21:31.250 --> 00:21:34.090 align:middle line:84% So it's partial f partial x1, partial f partial x2, 00:21:34.090 --> 00:21:35.860 align:middle line:90% and so forth. 00:21:35.860 --> 00:21:39.700 align:middle line:84% H is going to be the same thing, so same kind of thing. 00:21:39.700 --> 00:21:52.390 align:middle line:84% So now if we do it explicitly, let's just write out slowly. 00:21:52.390 --> 00:21:54.850 align:middle line:84% So we know that the gradient of f 00:21:54.850 --> 00:21:59.695 align:middle line:84% is the vector that has partial f partial x1 00:21:59.695 --> 00:22:02.185 align:middle line:90% to partial f partial xn. 00:22:02.185 --> 00:22:06.340 align:middle line:90% 00:22:06.340 --> 00:22:11.200 align:middle line:84% So that means what we want to take 00:22:11.200 --> 00:22:25.750 align:middle line:84% is the change, d, of gradient f, say transposed, I guess. 00:22:25.750 --> 00:22:28.120 align:middle line:90% Yeah. 00:22:28.120 --> 00:22:39.320 align:middle line:84% So what's d of this, for example? 00:22:39.320 --> 00:22:52.360 align:middle line:84% So well, it's the same thing as d of partial f partial x1 to d 00:22:52.360 --> 00:22:55.300 align:middle line:90% of partial f partial xn. 00:22:55.300 --> 00:22:58.540 align:middle line:90% 00:22:58.540 --> 00:23:04.060 align:middle line:84% But partial f partial x1 is a scalar function 00:23:04.060 --> 00:23:05.197 align:middle line:90% of x, of all the-- 00:23:05.197 --> 00:23:06.780 align:middle line:84% it depends on all the components in x. 00:23:06.780 --> 00:23:09.870 align:middle line:84% And it spits out a number, which is partial f, partial x1. 00:23:09.870 --> 00:23:12.150 align:middle line:84% So we know what the d of that looks like. 00:23:12.150 --> 00:23:23.570 align:middle line:84% The d of that is a gradient of partial f partial x1 transpose 00:23:23.570 --> 00:23:26.310 align:middle line:90% dx. 00:23:26.310 --> 00:23:31.660 align:middle line:84% Say all the way to gradient of partial f partial xn transpose 00:23:31.660 --> 00:23:32.160 align:middle line:90% dx. 00:23:32.160 --> 00:23:35.100 align:middle line:90% 00:23:35.100 --> 00:23:38.280 align:middle line:90% So we can write that out as-- 00:23:38.280 --> 00:23:40.530 align:middle line:84% we can pull out the dx column vector. 00:23:40.530 --> 00:23:47.100 align:middle line:84% And that's gradient of partial f partial x1 transpose 00:23:47.100 --> 00:23:55.710 align:middle line:84% all the way to gradient of partial f partial xn transpose. 00:23:55.710 --> 00:24:00.770 align:middle line:84% But that matrix-- and with that, what is the gradient? 00:24:00.770 --> 00:24:04.850 align:middle line:84% That's partial f-- that's partial squared 00:24:04.850 --> 00:24:11.930 align:middle line:84% of f partial x1 partial x1 to partial f 00:24:11.930 --> 00:24:14.243 align:middle line:90% partial x1 partial xn. 00:24:14.243 --> 00:24:16.160 align:middle line:84% That's what the gradient transpose looks like. 00:24:16.160 --> 00:24:18.620 align:middle line:84% I take partial partial f x1 and take its derivative 00:24:18.620 --> 00:24:20.220 align:middle line:90% with respect to x1 to xn. 00:24:20.220 --> 00:24:23.660 align:middle line:84% So that's a mixed second derivative. 00:24:23.660 --> 00:24:25.640 align:middle line:84% And then in the last row is the same thing, 00:24:25.640 --> 00:24:28.108 align:middle line:84% partial squared partial f partial xn. 00:24:28.108 --> 00:24:29.900 align:middle line:84% And then I take its derivative with respect 00:24:29.900 --> 00:24:37.350 align:middle line:84% to x1 all the way to its derivative with respect to xn. 00:24:37.350 --> 00:24:39.480 align:middle line:90% Derivative's xn dx. 00:24:39.480 --> 00:24:42.240 align:middle line:90% 00:24:42.240 --> 00:24:49.680 align:middle line:84% And this is a matrix of second derivatives. 00:24:49.680 --> 00:24:53.700 align:middle line:90% This is exactly going to be-- 00:24:53.700 --> 00:24:54.660 align:middle line:90% I guess this was-- 00:24:54.660 --> 00:24:59.035 align:middle line:90% 00:24:59.035 --> 00:25:03.360 align:middle line:84% this is, I guess, H, or it's equal to-- 00:25:03.360 --> 00:25:05.700 align:middle line:84% I guess it's H transpose, because grad 00:25:05.700 --> 00:25:08.050 align:middle line:84% f was the transpose of the derivative. 00:25:08.050 --> 00:25:10.020 align:middle line:90% So this gives you the change. 00:25:10.020 --> 00:25:15.130 align:middle line:84% This is the d of f prime transpose. 00:25:15.130 --> 00:25:16.230 align:middle line:90% So it's H transpose. 00:25:16.230 --> 00:25:19.830 align:middle line:84% But we know that that equals H. So it doesn't really matter. 00:25:19.830 --> 00:25:27.920 align:middle line:84% And so then, we get that the H, the IJ element is just partial 00:25:27.920 --> 00:25:32.310 align:middle line:90% squared f partial xi partial xj. 00:25:32.310 --> 00:25:36.510 align:middle line:84% But that equals xji because we know in general 00:25:36.510 --> 00:25:39.390 align:middle line:84% that the Hessian is a symmetric matrix, 00:25:39.390 --> 00:25:41.490 align:middle line:84% is a symmetric bilinear form, which means 00:25:41.490 --> 00:25:42.970 align:middle line:90% this is a symmetric matrix. 00:25:42.970 --> 00:25:44.640 align:middle line:84% So that gives you this relationship 00:25:44.640 --> 00:25:47.028 align:middle line:84% you learned in multivariable calculus 00:25:47.028 --> 00:25:48.570 align:middle line:84% that when I take partial derivatives, 00:25:48.570 --> 00:25:50.260 align:middle line:90% I can swap the order. 00:25:50.260 --> 00:25:52.760 align:middle line:84% I mean, it just comes from the definition of the derivative. 00:25:52.760 --> 00:25:55.220 align:middle line:84% But the point is that that extends to more-- 00:25:55.220 --> 00:25:57.320 align:middle line:84% when you extend it to more general objects, 00:25:57.320 --> 00:26:01.520 align:middle line:84% it turns into this symmetric bilinear form business. 00:26:01.520 --> 00:26:02.180 align:middle line:90% Any questions? 00:26:02.180 --> 00:26:02.680 align:middle line:90% Yeah? 00:26:02.680 --> 00:26:04.670 align:middle line:84% AUDIENCE: Just a quick clarification-- 00:26:04.670 --> 00:26:06.440 align:middle line:90% this should be an n1? 00:26:06.440 --> 00:26:08.060 align:middle line:84% Like, this should be reversed, right, 00:26:08.060 --> 00:26:10.353 align:middle line:90% just in the matrix itself? 00:26:10.353 --> 00:26:11.270 align:middle line:90% STEVEN JOHNSON: Which? 00:26:11.270 --> 00:26:13.120 align:middle line:84% The partial derivatives, these should be reversed? 00:26:13.120 --> 00:26:14.287 align:middle line:90% AUDIENCE: Yeah, technically. 00:26:14.287 --> 00:26:16.580 align:middle line:84% STEVEN JOHNSON: I guess it depends on-- yes. 00:26:16.580 --> 00:26:18.020 align:middle line:90% It depends on what you mean by-- 00:26:18.020 --> 00:26:19.790 align:middle line:84% because we're so used to the fact 00:26:19.790 --> 00:26:22.460 align:middle line:84% that this-- you can take the derivative in either order. 00:26:22.460 --> 00:26:24.027 align:middle line:90% So actually, I don't know. 00:26:24.027 --> 00:26:25.610 align:middle line:84% Does it mean you take this derivative, 00:26:25.610 --> 00:26:28.980 align:middle line:84% then this derivative, or the other way around? 00:26:28.980 --> 00:26:32.548 align:middle line:84% If I put parentheses here, I think it's right. 00:26:32.548 --> 00:26:33.090 align:middle line:90% AUDIENCE: OK. 00:26:33.090 --> 00:26:34.590 align:middle line:84% STEVEN JOHNSON: Yeah, because it's-- 00:26:34.590 --> 00:26:38.377 align:middle line:84% but the notation is kind of ambiguous 00:26:38.377 --> 00:26:39.960 align:middle line:84% because it doesn't need to be explicit 00:26:39.960 --> 00:26:41.260 align:middle line:90% because you can swap the order. 00:26:41.260 --> 00:26:41.760 align:middle line:90% But yeah. 00:26:41.760 --> 00:26:45.930 align:middle line:84% So I think yeah, if I put derivative parentheses here, 00:26:45.930 --> 00:26:47.700 align:middle line:90% that's the explicit thing. 00:26:47.700 --> 00:26:49.013 align:middle line:90% I take partial f partial x1. 00:26:49.013 --> 00:26:50.430 align:middle line:84% I take its derivative with respect 00:26:50.430 --> 00:26:52.890 align:middle line:90% to x1 all the way to xn. 00:26:52.890 --> 00:26:55.470 align:middle line:84% But if you put parentheses there, then it's the opposite. 00:26:55.470 --> 00:26:57.210 align:middle line:84% But at the end of the day, it's not 00:26:57.210 --> 00:27:01.000 align:middle line:84% going to matter because this is symmetric. 00:27:01.000 --> 00:27:02.260 align:middle line:90% AUDIENCE: Yeah. 00:27:02.260 --> 00:27:05.386 align:middle line:90% STEVEN JOHNSON: OK, so that's-- 00:27:05.386 --> 00:27:07.575 align:middle line:84% how many learned the Hessian matrix this way? 00:27:07.575 --> 00:27:08.950 align:middle line:84% Basically, it's the matrix of all 00:27:08.950 --> 00:27:09.880 align:middle line:90% the mixed second derivatives. 00:27:09.880 --> 00:27:10.270 align:middle line:90% Yeah. 00:27:10.270 --> 00:27:12.103 align:middle line:84% So that's usually how it's presented, right? 00:27:12.103 --> 00:27:13.360 align:middle line:90% And that's-- yes? 00:27:13.360 --> 00:27:15.220 align:middle line:84% AUDIENCE: And just conclude that it's 00:27:15.220 --> 00:27:17.140 align:middle line:84% the Jacobian of the gradient, in a way? 00:27:17.140 --> 00:27:18.250 align:middle line:84% STEVEN JOHNSON: Yeah, it's exactly 00:27:18.250 --> 00:27:19.360 align:middle line:90% the Jacobian of the gradient. 00:27:19.360 --> 00:27:19.860 align:middle line:90% Yes. 00:27:19.860 --> 00:27:20.895 align:middle line:90% Good, good, good. 00:27:20.895 --> 00:27:22.960 align:middle line:90% And we write that down. 00:27:22.960 --> 00:27:24.355 align:middle line:90% So this is the-- 00:27:24.355 --> 00:27:32.460 align:middle line:90% 00:27:32.460 --> 00:27:34.575 align:middle line:90% Yeah, exactly. 00:27:34.575 --> 00:27:39.120 align:middle line:90% 00:27:39.120 --> 00:27:40.140 align:middle line:90% Yeah. 00:27:40.140 --> 00:27:42.810 align:middle line:84% But now we have it in a much more general setting. 00:27:42.810 --> 00:27:46.140 align:middle line:90% So I think it's nice to do-- 00:27:46.140 --> 00:27:50.200 align:middle line:84% let's do a more general example, an example 00:27:50.200 --> 00:28:00.910 align:middle line:84% that it's not so easy to do with 1801, or 1802 even. 00:28:00.910 --> 00:28:03.400 align:middle line:84% Let's take our favorite function, 00:28:03.400 --> 00:28:06.760 align:middle line:84% our new favorite function, that takes in a matrix 00:28:06.760 --> 00:28:07.780 align:middle line:90% and gives you a scalar. 00:28:07.780 --> 00:28:12.100 align:middle line:90% 00:28:12.100 --> 00:28:14.260 align:middle line:84% So from the previous lecture, we learned 00:28:14.260 --> 00:28:17.570 align:middle line:90% what the derivative of this is. 00:28:17.570 --> 00:28:22.130 align:middle line:84% So if you take f primed of A, that's 00:28:22.130 --> 00:28:25.280 align:middle line:84% the linear operator acting on dA, 00:28:25.280 --> 00:28:30.260 align:middle line:84% what it gives you is determinant A times 00:28:30.260 --> 00:28:36.270 align:middle line:90% the trace of A inverse dA. 00:28:36.270 --> 00:28:39.390 align:middle line:84% Or equivalent, we showed that the gradient of f 00:28:39.390 --> 00:28:43.080 align:middle line:84% was the determinant of A times the transpose of this, 00:28:43.080 --> 00:28:47.550 align:middle line:84% A inverse transpose, which is called the adjugate matrix. 00:28:47.550 --> 00:28:55.210 align:middle line:90% 00:28:55.210 --> 00:28:57.130 align:middle line:84% Yeah, the adjugate or the transpose 00:28:57.130 --> 00:29:01.831 align:middle line:84% of the adjugate-- sorry, the adjugate transpose. 00:29:01.831 --> 00:29:06.460 align:middle line:84% of A. I always forget which is which. 00:29:06.460 --> 00:29:10.500 align:middle line:84% It's the cofactor matrix of A. So yes, we saw this, 00:29:10.500 --> 00:29:13.938 align:middle line:84% and there's various ways you can show this. 00:29:13.938 --> 00:29:16.230 align:middle line:84% Professor Edelman looked at a couple of different ways. 00:29:16.230 --> 00:29:18.170 align:middle line:84% So now let's go one step further. 00:29:18.170 --> 00:29:21.230 align:middle line:84% And now let's take the second derivative. 00:29:21.230 --> 00:29:24.140 align:middle line:84% And whenever we're faced with something new and confusing, 00:29:24.140 --> 00:29:27.540 align:middle line:84% it's always good to fall back on the definition. 00:29:27.540 --> 00:29:31.460 align:middle line:84% So all we're doing is just going to take-- 00:29:31.460 --> 00:29:48.060 align:middle line:84% we're going to take d of this, d of this whole formula, 00:29:48.060 --> 00:29:54.600 align:middle line:84% determinant A trace A inverse dA. 00:29:54.600 --> 00:29:57.900 align:middle line:90% 00:29:57.900 --> 00:30:01.500 align:middle line:84% I'm going to put a prime here, just 00:30:01.500 --> 00:30:04.983 align:middle line:84% to make it clear that what I'm changing is A, not dA. 00:30:04.983 --> 00:30:06.150 align:middle line:90% I'm going to use a dA prime. 00:30:06.150 --> 00:30:11.070 align:middle line:84% So d primed-- so what this is going to be, is this 00:30:11.070 --> 00:30:23.340 align:middle line:84% is going to be our f primed of A plus dA primed acting on dA. 00:30:23.340 --> 00:30:24.965 align:middle line:84% Let me give myself a little more space. 00:30:24.965 --> 00:30:32.270 align:middle line:90% 00:30:32.270 --> 00:30:38.470 align:middle line:90% Minus f prime of A on dA. 00:30:38.470 --> 00:30:41.280 align:middle line:84% So the prime here is just-- it's not another derivative. 00:30:41.280 --> 00:30:43.770 align:middle line:84% I'm overloading my primes a little bit. 00:30:43.770 --> 00:30:45.510 align:middle line:90% It just means I'm using-- 00:30:45.510 --> 00:30:49.170 align:middle line:90% I'm changing A by dA primed. 00:30:49.170 --> 00:30:51.540 align:middle line:90% OK, and dA is going to be fixed. 00:30:51.540 --> 00:30:52.280 align:middle line:90% Yes? 00:30:52.280 --> 00:30:54.030 align:middle line:84% AUDIENCE: In the previous example as well, 00:30:54.030 --> 00:30:57.570 align:middle line:84% that was technically dx prime, then, just to clarify that? 00:30:57.570 --> 00:30:59.640 align:middle line:90% STEVEN JOHNSON: Yeah. 00:30:59.640 --> 00:31:01.530 align:middle line:84% Yeah, well, I didn't have any dx's here. 00:31:01.530 --> 00:31:03.840 align:middle line:90% So I didn't need a d prime. 00:31:03.840 --> 00:31:04.725 align:middle line:90% AUDIENCE: Right. 00:31:04.725 --> 00:31:05.850 align:middle line:90% STEVEN JOHNSON: Yeah, yeah. 00:31:05.850 --> 00:31:08.200 align:middle line:84% But I could have used a d prime there if I wanted. 00:31:08.200 --> 00:31:10.090 align:middle line:84% But it's really the same kind of thing. 00:31:10.090 --> 00:31:11.760 align:middle line:84% It's just I already have a dA here. 00:31:11.760 --> 00:31:14.640 align:middle line:84% And I want to be careful that I'm not changing this. 00:31:14.640 --> 00:31:16.080 align:middle line:90% This is now going to be-- 00:31:16.080 --> 00:31:18.360 align:middle line:84% this is-- actually, let me put this in blue here. 00:31:18.360 --> 00:31:28.570 align:middle line:90% This is dA, dA, dA. 00:31:28.570 --> 00:31:30.250 align:middle line:90% This is going to be fixed. 00:31:30.250 --> 00:31:33.410 align:middle line:90% 00:31:33.410 --> 00:31:35.780 align:middle line:90% So dA is not changing. 00:31:35.780 --> 00:31:39.590 align:middle line:84% So we can think of it as a constant. 00:31:39.590 --> 00:31:43.190 align:middle line:84% So when I change things, I'm changing A by dA primed. 00:31:43.190 --> 00:31:49.640 align:middle line:84% And so now I can just use our derivative rules, 00:31:49.640 --> 00:31:59.830 align:middle line:84% so our product, and our chain rules, dot, dot, dot. 00:31:59.830 --> 00:32:01.585 align:middle line:90% And what do I get? 00:32:01.585 --> 00:32:05.590 align:middle line:90% 00:32:05.590 --> 00:32:08.110 align:middle line:84% So now, I'm just treating dA as just a constant matrix 00:32:08.110 --> 00:32:09.080 align:middle line:90% I'm sticking in there. 00:32:09.080 --> 00:32:10.720 align:middle line:84% And then I'm taking the derivative the same way 00:32:10.720 --> 00:32:11.220 align:middle line:90% as before. 00:32:11.220 --> 00:32:14.080 align:middle line:84% Well, I have the derivative of this term times this 00:32:14.080 --> 00:32:16.400 align:middle line:84% plus this times the derivative of that term. 00:32:16.400 --> 00:32:18.520 align:middle line:84% And the derivative of this term, we just-- 00:32:18.520 --> 00:32:20.350 align:middle line:84% or the differential, sorry, of that term, 00:32:20.350 --> 00:32:22.120 align:middle line:90% we just saw what it was. 00:32:22.120 --> 00:32:28.420 align:middle line:90% This is our-- it's this. 00:32:28.420 --> 00:32:30.160 align:middle line:90% It's this, right? 00:32:30.160 --> 00:32:37.440 align:middle line:84% So the first term is determinant of A times-- 00:32:37.440 --> 00:32:40.020 align:middle line:90% 00:32:40.020 --> 00:32:44.000 align:middle line:84% so the d of the determinant is the determinant of A times 00:32:44.000 --> 00:32:46.730 align:middle line:90% the trace of A prime-- 00:32:46.730 --> 00:32:53.180 align:middle line:84% not A prime, A inverse dA primed. 00:32:53.180 --> 00:33:04.180 align:middle line:84% All right, so this came from that term times the other term, 00:33:04.180 --> 00:33:13.850 align:middle line:84% trace of A inverse dA plus I still 00:33:13.850 --> 00:33:19.470 align:middle line:84% have my determinant of A times the derivative 00:33:19.470 --> 00:33:20.400 align:middle line:90% of the other term. 00:33:20.400 --> 00:33:24.780 align:middle line:90% 00:33:24.780 --> 00:33:28.000 align:middle line:90% And trace is a linear operator. 00:33:28.000 --> 00:33:31.510 align:middle line:84% So I can just take the derivative inside. 00:33:31.510 --> 00:33:34.050 align:middle line:84% And so now-- or the differential inside-- 00:33:34.050 --> 00:33:37.140 align:middle line:90% and now I need dA inverse. 00:33:37.140 --> 00:33:40.260 align:middle line:84% And we know what dA inverse, that was a minus sign. 00:33:40.260 --> 00:33:48.440 align:middle line:84% So let me change this to a minus sign of A inverse dA 00:33:48.440 --> 00:33:54.803 align:middle line:84% primed, A inverse, and then there's still a dA. 00:33:54.803 --> 00:33:55.720 align:middle line:90% Can everyone see that? 00:33:55.720 --> 00:34:05.870 align:middle line:84% So this term here is exactly d primed of A inverse-- 00:34:05.870 --> 00:34:07.310 align:middle line:90% well, with the minus sign. 00:34:07.310 --> 00:34:15.199 align:middle line:90% 00:34:15.199 --> 00:34:21.070 align:middle line:84% And now the question is, is this-- 00:34:21.070 --> 00:34:21.790 align:middle line:90% this is it. 00:34:21.790 --> 00:34:24.730 align:middle line:84% This is-- it's not going to get much simpler than this. 00:34:24.730 --> 00:34:25.989 align:middle line:90% This is our bilinear form. 00:34:25.989 --> 00:34:29.770 align:middle line:90% 00:34:29.770 --> 00:34:30.624 align:middle line:90% This is bilinear. 00:34:30.624 --> 00:34:33.179 align:middle line:90% 00:34:33.179 --> 00:34:45.030 align:middle line:84% And in dA primed and dA, so I stick in any dA primed, any dA. 00:34:45.030 --> 00:34:49.409 align:middle line:84% Clearly, this is linear in each one of them individually. 00:34:49.409 --> 00:34:52.320 align:middle line:84% I never-- I can multiply dA by dA prime. 00:34:52.320 --> 00:34:53.429 align:middle line:90% It's quadratic. 00:34:53.429 --> 00:34:57.300 align:middle line:90% 00:34:57.300 --> 00:35:00.495 align:middle line:84% But I can't multiply a dA by itself. 00:35:00.495 --> 00:35:03.000 align:middle line:90% 00:35:03.000 --> 00:35:05.260 align:middle line:90% And is it symmetric? 00:35:05.260 --> 00:35:06.990 align:middle line:90% Well, let's look. 00:35:06.990 --> 00:35:10.400 align:middle line:90% This term is the same as this. 00:35:10.400 --> 00:35:13.230 align:middle line:84% So if I swap-- and these are just-- trace is just a number. 00:35:13.230 --> 00:35:15.500 align:middle line:90% So I can swap these two terms. 00:35:15.500 --> 00:35:17.360 align:middle line:90% And that's fine. 00:35:17.360 --> 00:35:19.610 align:middle line:90% And this is also a number. 00:35:19.610 --> 00:35:21.530 align:middle line:90% What about this? 00:35:21.530 --> 00:35:25.920 align:middle line:84% If I swap dA and d primed, it looks a little bit different. 00:35:25.920 --> 00:35:28.400 align:middle line:84% But remember, the trace has the cyclic property. 00:35:28.400 --> 00:35:34.270 align:middle line:84% I can move the A inverse dA over to the beginning, 00:35:34.270 --> 00:35:37.360 align:middle line:84% and then it looks like trace of A inverse dA, 00:35:37.360 --> 00:35:40.590 align:middle line:90% A inverse dA primed. 00:35:40.590 --> 00:35:51.690 align:middle line:84% So this is symmetric using this cyclic property of the trace. 00:35:51.690 --> 00:35:59.490 align:middle line:90% 00:35:59.490 --> 00:36:00.060 align:middle line:90% That's it. 00:36:00.060 --> 00:36:02.880 align:middle line:90% This doesn't simplify. 00:36:02.880 --> 00:36:05.040 align:middle line:84% We can't write it as a Hessian matrix 00:36:05.040 --> 00:36:07.350 align:middle line:90% unless I vectorize things. 00:36:07.350 --> 00:36:12.390 align:middle line:84% So if I do vec of dA, this could be some big matrix-- 00:36:12.390 --> 00:36:13.860 align:middle line:90% n squared by-- 00:36:13.860 --> 00:36:15.600 align:middle line:84% yeah, it's n squared by n squared 00:36:15.600 --> 00:36:19.500 align:middle line:84% matrix that has a vec of dA on one side 00:36:19.500 --> 00:36:22.080 align:middle line:84% and vec of dA primed on the other side. 00:36:22.080 --> 00:36:26.640 align:middle line:84% But it's not very natural to do that. 00:36:26.640 --> 00:36:28.665 align:middle line:84% And it's in some ways, it's easier to do this. 00:36:28.665 --> 00:36:33.520 align:middle line:90% 00:36:33.520 --> 00:36:42.060 align:middle line:84% So I want to talk a little bit about why second derivatives? 00:36:42.060 --> 00:36:48.310 align:middle line:90% 00:36:48.310 --> 00:36:50.350 align:middle line:84% And of course, they come up in lots of cases. 00:36:50.350 --> 00:36:53.750 align:middle line:84% But let me just mention a few salient things. 00:36:53.750 --> 00:36:55.090 align:middle line:90% So first of all, they give you-- 00:36:55.090 --> 00:36:58.390 align:middle line:84% the first derivative gives you a linear approximation 00:36:58.390 --> 00:37:00.000 align:middle line:90% of a function linearization. 00:37:00.000 --> 00:37:04.270 align:middle line:84% The second derivatives give you quadratic approximations. 00:37:04.270 --> 00:37:11.620 align:middle line:90% 00:37:11.620 --> 00:37:19.110 align:middle line:84% So if you have f of x plus a little change. 00:37:19.110 --> 00:37:21.720 align:middle line:90% Let's call it delta x. 00:37:21.720 --> 00:37:23.950 align:middle line:90% It's not infinitesimal anymore. 00:37:23.950 --> 00:37:27.350 align:middle line:84% So this is going to be an approximation. 00:37:27.350 --> 00:37:33.830 align:middle line:84% This is approximately f of x plus f primed of x delta x. 00:37:33.830 --> 00:37:37.310 align:middle line:84% That's our linear approximation from the first derivative, 00:37:37.310 --> 00:37:39.320 align:middle line:84% our finite difference approximation, 00:37:39.320 --> 00:37:40.250 align:middle line:90% if you think of it. 00:37:40.250 --> 00:37:43.950 align:middle line:84% And now there'll be a new term that'll 00:37:43.950 --> 00:37:51.647 align:middle line:84% look like f double primed of x with a delta x, a delta x, 00:37:51.647 --> 00:37:52.730 align:middle line:90% and I'm missing something. 00:37:52.730 --> 00:37:53.480 align:middle line:90% What am I missing? 00:37:53.480 --> 00:37:57.400 align:middle line:90% 00:37:57.400 --> 00:37:59.830 align:middle line:84% Just think of 1801, Taylor series. 00:37:59.830 --> 00:38:01.030 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:38:01.030 --> 00:38:03.690 align:middle line:90% STEVEN JOHNSON: Is 1/2, yeah. 00:38:03.690 --> 00:38:08.020 align:middle line:84% And then there's higher order terms, a little low 00:38:08.020 --> 00:38:11.670 align:middle line:90% of delta x squared. 00:38:11.670 --> 00:38:13.090 align:middle line:90% We're dropping terms. 00:38:13.090 --> 00:38:21.370 align:middle line:84% So we're dropping terms that are smaller than quadratic. 00:38:21.370 --> 00:38:23.065 align:middle line:84% So if it's three times differentiable, 00:38:23.065 --> 00:38:24.190 align:middle line:90% we're dropping cubic terms. 00:38:24.190 --> 00:38:26.590 align:middle line:84% But maybe the function doesn't have it, their derivative. 00:38:26.590 --> 00:38:28.180 align:middle line:84% But definitely what we're dropping 00:38:28.180 --> 00:38:30.220 align:middle line:84% are our terms that are higher than quadratic. 00:38:30.220 --> 00:38:33.460 align:middle line:90% So the 1/2. 00:38:33.460 --> 00:38:36.340 align:middle line:84% You can derive this pretty easily by just-- 00:38:36.340 --> 00:38:38.800 align:middle line:84% if you take two derivatives of this, 00:38:38.800 --> 00:38:40.360 align:middle line:84% you'd better get back to f primed-- 00:38:40.360 --> 00:38:42.340 align:middle line:90% f double primed. 00:38:42.340 --> 00:38:44.800 align:middle line:84% It's at a better match, the second derivative. 00:38:44.800 --> 00:38:49.180 align:middle line:84% But because the x appears twice in this, 00:38:49.180 --> 00:38:53.770 align:middle line:84% then if you take two derivatives back to delta x, 00:38:53.770 --> 00:38:58.030 align:middle line:84% you should get back to f double primed. 00:38:58.030 --> 00:39:00.430 align:middle line:84% But because delta x appears twice in this, 00:39:00.430 --> 00:39:03.108 align:middle line:84% you need to have a half there in order 00:39:03.108 --> 00:39:04.400 align:middle line:90% to get back to f double primed. 00:39:04.400 --> 00:39:06.260 align:middle line:90% So we just write that in. 00:39:06.260 --> 00:39:09.450 align:middle line:90% So if we take-- 00:39:09.450 --> 00:39:16.790 align:middle line:84% OK, the second derivative with a respect 00:39:16.790 --> 00:39:23.450 align:middle line:84% to delta x, you had better get back to f double primed of x. 00:39:23.450 --> 00:39:41.420 align:middle line:84% And that 1/2 factor is necessary because it appears twice. 00:39:41.420 --> 00:39:44.230 align:middle line:84% So otherwise, you'd see it get twice, just like if-- 00:39:44.230 --> 00:39:47.020 align:middle line:84% for the same reason you have it in the Taylor series. 00:39:47.020 --> 00:39:50.290 align:middle line:84% If these are just scalars, when I take the-- this is a delta 00:39:50.290 --> 00:39:51.033 align:middle line:90% x squared. 00:39:51.033 --> 00:39:52.450 align:middle line:84% When I take the second derivative, 00:39:52.450 --> 00:39:53.530 align:middle line:90% I'm going to get a 2. 00:39:53.530 --> 00:39:56.620 align:middle line:84% And I really want it to match the second derivative 00:39:56.620 --> 00:39:58.870 align:middle line:90% of my function at that point. 00:39:58.870 --> 00:40:03.550 align:middle line:84% OK, so linear-- approximating things by other things 00:40:03.550 --> 00:40:04.330 align:middle line:90% is useful. 00:40:04.330 --> 00:40:07.160 align:middle line:84% Approximating things by quadratic functions is useful. 00:40:07.160 --> 00:40:15.440 align:middle line:84% So for example, if you're doing optimization, 00:40:15.440 --> 00:40:24.780 align:middle line:84% we approximate f by approximately a quadratic 00:40:24.780 --> 00:40:33.175 align:middle line:84% and then optimize the quadratic to get a step. 00:40:33.175 --> 00:40:37.280 align:middle line:90% 00:40:37.280 --> 00:40:38.900 align:middle line:90% This is sometimes called-- 00:40:38.900 --> 00:40:40.320 align:middle line:90% this is a variety of names. 00:40:40.320 --> 00:40:55.210 align:middle line:84% It's sometimes called Sequential Quadratic Programming, or SQP. 00:40:55.210 --> 00:40:59.860 align:middle line:84% Technically, you-- also, if you have constraints, 00:40:59.860 --> 00:41:06.270 align:middle line:84% you make linear approximations of the constraints or, I guess, 00:41:06.270 --> 00:41:08.090 align:middle line:90% affine approximations. 00:41:08.090 --> 00:41:10.860 align:middle line:90% 00:41:10.860 --> 00:41:14.370 align:middle line:84% But colloquially, it's called linear approximations 00:41:14.370 --> 00:41:19.630 align:middle line:90% of products of constraints. 00:41:19.630 --> 00:41:22.470 align:middle line:84% So if you have a-- but you're minimizing an arbitrary 00:41:22.470 --> 00:41:25.380 align:middle line:84% nonlinear function, it's back to arbitrary nonlinear 00:41:25.380 --> 00:41:27.060 align:middle line:90% constraints. 00:41:27.060 --> 00:41:28.272 align:middle line:90% That's hard. 00:41:28.272 --> 00:41:29.730 align:middle line:84% But if you approximate the function 00:41:29.730 --> 00:41:32.640 align:middle line:84% by a quadratic function using a second derivative, 00:41:32.640 --> 00:41:34.380 align:middle line:84% approximate the constraints by linear, 00:41:34.380 --> 00:41:36.450 align:middle line:84% that's called a QP, or Quadratic Program. 00:41:36.450 --> 00:41:40.725 align:middle line:84% And there are good methods to solve those kinds of things. 00:41:40.725 --> 00:41:43.800 align:middle line:90% 00:41:43.800 --> 00:41:48.120 align:middle line:84% Another equivalent-- equivalently, 00:41:48.120 --> 00:41:54.060 align:middle line:84% optimizing a quadratic function is the same thing as-- 00:41:54.060 --> 00:41:57.180 align:middle line:84% so optimizing a function is equivalent to finding-- 00:41:57.180 --> 00:42:00.960 align:middle line:84% locally to finding a root of the gradient. 00:42:00.960 --> 00:42:08.773 align:middle line:84% So we're equivalently finding a root of the gradient of f. 00:42:08.773 --> 00:42:10.190 align:middle line:84% And you're going to approximate it 00:42:10.190 --> 00:42:24.070 align:middle line:84% by a linear, or by an affine, or let's say, linear-- 00:42:24.070 --> 00:42:29.920 align:middle line:84% colloquially linear, because equivalently, you're 00:42:29.920 --> 00:42:37.150 align:middle line:84% approximating it by f primed of x plus your f double primed 00:42:37.150 --> 00:42:42.460 align:middle line:90% of x delta x. 00:42:42.460 --> 00:42:48.000 align:middle line:84% And so if you're finding a root of a linear function, 00:42:48.000 --> 00:42:50.160 align:middle line:84% you have a nonlinear-- you're trying 00:42:50.160 --> 00:42:52.290 align:middle line:84% to find the root of a nonlinear function grad f, 00:42:52.290 --> 00:42:54.042 align:middle line:84% and you approximate it by a linear thing, 00:42:54.042 --> 00:42:55.500 align:middle line:84% and you find a root of that, what's 00:42:55.500 --> 00:42:57.090 align:middle line:90% the name for that method? 00:42:57.090 --> 00:42:57.870 align:middle line:90% AUDIENCE: Newton-- 00:42:57.870 --> 00:42:59.020 align:middle line:90% STEVEN JOHNSON: Newton's method. 00:42:59.020 --> 00:43:00.353 align:middle line:90% So this is just Newton's method. 00:43:00.353 --> 00:43:12.510 align:middle line:90% 00:43:12.510 --> 00:43:22.410 align:middle line:84% And so in practice, finding the Hessian, or finding the f 00:43:22.410 --> 00:43:32.110 align:middle line:84% double primed or the Hessian matrix is often expensive. 00:43:32.110 --> 00:43:37.240 align:middle line:90% 00:43:37.240 --> 00:43:44.770 align:middle line:84% If f of x again takes a Rn to a scalar, 00:43:44.770 --> 00:43:50.440 align:middle line:90% then H is an n-by-n matrix. 00:43:50.440 --> 00:43:56.320 align:middle line:90% And this is huge if n is large. 00:43:56.320 --> 00:43:59.305 align:middle line:84% So if you have a neural network where n is a billion, 00:43:59.305 --> 00:44:01.180 align:middle line:84% the Hessian is a billion by a billion matrix. 00:44:01.180 --> 00:44:05.010 align:middle line:84% You can't even store this matrix, much less compute it. 00:44:05.010 --> 00:44:08.890 align:middle line:84% So it's hard to get exactly for-- 00:44:08.890 --> 00:44:10.630 align:middle line:90% I guess it's in high dimensions. 00:44:10.630 --> 00:44:17.470 align:middle line:90% 00:44:17.470 --> 00:44:22.090 align:middle line:84% So often, what you would try and do is you try and approximate. 00:44:22.090 --> 00:44:25.390 align:middle line:84% If you don't want to give up on it entirely, 00:44:25.390 --> 00:44:34.550 align:middle line:84% you approximate the Hessian in various ways. 00:44:34.550 --> 00:44:39.470 align:middle line:90% 00:44:39.470 --> 00:44:45.980 align:middle line:84% And so these give you rise to a variety of methods called 00:44:45.980 --> 00:44:57.300 align:middle line:84% quasi-Newton methods, the most famous of which 00:44:57.300 --> 00:45:02.400 align:middle line:84% is called the BFGS method, which is Broyden, Fletcher, 00:45:02.400 --> 00:45:04.830 align:middle line:90% Goldfarb, and Shannon, I think. 00:45:04.830 --> 00:45:08.280 align:middle line:84% It's named after four people who amusingly discovered 00:45:08.280 --> 00:45:10.930 align:middle line:84% the same thing in the same year independently, 00:45:10.930 --> 00:45:15.510 align:middle line:84% like there's three or four separate papers [LAUGHS] 00:45:15.510 --> 00:45:17.200 align:middle line:90% on the same thing. 00:45:17.200 --> 00:45:19.140 align:middle line:84% There's also a closely related method 00:45:19.140 --> 00:45:29.550 align:middle line:84% called Newton-Krylov methods, and so forth. 00:45:29.550 --> 00:45:31.680 align:middle line:84% So I don't have time to explain all these things. 00:45:31.680 --> 00:45:36.480 align:middle line:84% But these are some key words if you ever need to do this. 00:45:36.480 --> 00:45:39.227 align:middle line:84% So Hessians are useful for a small n. 00:45:39.227 --> 00:45:40.560 align:middle line:90% You can compute them explicitly. 00:45:40.560 --> 00:45:42.000 align:middle line:84% By even automatic differentiation, 00:45:42.000 --> 00:45:43.350 align:middle line:90% you can get Hessians for you. 00:45:43.350 --> 00:45:46.390 align:middle line:84% But for big n's, you can't even store it, much less compute it. 00:45:46.390 --> 00:45:48.100 align:middle line:84% So then there are ways to kind of-- 00:45:48.100 --> 00:45:50.010 align:middle line:84% and it's a really intricate problem 00:45:50.010 --> 00:45:52.020 align:middle line:90% to approximate Hessians. 00:45:52.020 --> 00:45:56.100 align:middle line:84% Or you can compute the Hessian times in a particular direction 00:45:56.100 --> 00:45:59.490 align:middle line:84% quickly, like a Hessian times an operated on a dx. 00:45:59.490 --> 00:46:02.610 align:middle line:90% That you can get quickly. 00:46:02.610 --> 00:46:05.105 align:middle line:90% But yeah. 00:46:05.105 --> 00:46:09.000 align:middle line:90%