WEBVTT 00:00:00.000 --> 00:00:02.410 align:middle line:90% [SQUEAKING] 00:00:02.410 --> 00:00:03.856 align:middle line:90% [RUSTLING] 00:00:03.856 --> 00:00:07.712 align:middle line:90% [CLICKING] 00:00:07.712 --> 00:00:09.867 align:middle line:90% 00:00:09.867 --> 00:00:11.700 align:middle line:84% STEVEN JOHNSON: So I wanted to start the day 00:00:11.700 --> 00:00:15.150 align:middle line:90% just by finishing up a couple-- 00:00:15.150 --> 00:00:17.160 align:middle line:84% comment on a couple of additional things 00:00:17.160 --> 00:00:18.150 align:middle line:90% from the last lecture. 00:00:18.150 --> 00:00:20.460 align:middle line:84% So last lecture, we talked about finite difference 00:00:20.460 --> 00:00:21.150 align:middle line:90% approximations. 00:00:21.150 --> 00:00:25.803 align:middle line:84% In particular, I talked about the most obvious approximation, 00:00:25.803 --> 00:00:27.970 align:middle line:84% which is just from the definition of the derivative. 00:00:27.970 --> 00:00:35.050 align:middle line:84% We approximate df by f prime of xdx, 00:00:35.050 --> 00:00:38.020 align:middle line:84% our linear thing, the exact derivative for a infinitesimal, 00:00:38.020 --> 00:00:46.330 align:middle line:84% dx, by just f of x plus dx minus f of x for some finite delta x, 00:00:46.330 --> 00:00:48.160 align:middle line:90% noninfinitesimal. 00:00:48.160 --> 00:00:52.720 align:middle line:84% And if you divide both sides by delta x, 00:00:52.720 --> 00:00:55.280 align:middle line:84% the more familiar form is was this one. 00:00:55.280 --> 00:00:58.600 align:middle line:84% This is called the forward difference approximation . 00:00:58.600 --> 00:01:03.280 align:middle line:84% And we showed that this has an error in it that 00:01:03.280 --> 00:01:10.750 align:middle line:84% decreases linearly with dx until dx becomes really small 00:01:10.750 --> 00:01:13.780 align:middle line:84% and then becomes limited by roundoff errors. 00:01:13.780 --> 00:01:15.680 align:middle line:84% But I wanted to mention that, of course, 00:01:15.680 --> 00:01:18.010 align:middle line:84% there are other ways of approximating derivatives 00:01:18.010 --> 00:01:21.760 align:middle line:84% by differences, some of which are even more accurate. 00:01:21.760 --> 00:01:24.000 align:middle line:84% They're called higher order finite difference. 00:01:24.000 --> 00:01:25.820 align:middle line:84% So the forward differences are first order 00:01:25.820 --> 00:01:30.110 align:middle line:84% because the error goes linearly with dx. 00:01:30.110 --> 00:01:34.580 align:middle line:84% The most famous next improvement would 00:01:34.580 --> 00:01:40.510 align:middle line:84% be approximate f prime dx by instead of f of x plus dx 00:01:40.510 --> 00:01:46.600 align:middle line:84% minus f of x, we take f of x plus dx minus f of x minus dx. 00:01:46.600 --> 00:01:49.840 align:middle line:84% And since now you've basically doubled the interval, 00:01:49.840 --> 00:01:52.450 align:middle line:84% because you're going from minus dx to plus dx, 00:01:52.450 --> 00:01:54.160 align:middle line:90% you have to divide by 2. 00:01:54.160 --> 00:02:00.280 align:middle line:84% Or if x is a scalar, so then you can just divide by delta x. 00:02:00.280 --> 00:02:02.860 align:middle line:84% The second form is the more familiar form 00:02:02.860 --> 00:02:06.790 align:middle line:84% of these center difference approximations, which is you 00:02:06.790 --> 00:02:09.160 align:middle line:84% divide both sides by delta x, which only 00:02:09.160 --> 00:02:10.810 align:middle line:90% works if delta x is a scalar. 00:02:10.810 --> 00:02:13.540 align:middle line:84% And you get f prime is approximately f 00:02:13.540 --> 00:02:19.420 align:middle line:84% of x plus dx minus f of x minus dx divided by 2 dx, 00:02:19.420 --> 00:02:22.960 align:middle line:84% so again, 2 because you're going from minus dx to plus dx. 00:02:22.960 --> 00:02:26.680 align:middle line:90% So the overall interval is 2dx. 00:02:26.680 --> 00:02:29.990 align:middle line:84% So it turns out that the error in this case, 00:02:29.990 --> 00:02:34.845 align:middle line:84% instead of going linearly with delta x, is quadratic. 00:02:34.845 --> 00:02:37.790 align:middle line:84% I guess this should be order delta x squared. 00:02:37.790 --> 00:02:39.570 align:middle line:90% I left out the x. 00:02:39.570 --> 00:02:42.520 align:middle line:90% I'll fix that later. 00:02:42.520 --> 00:02:47.470 align:middle line:84% And the reason it goes quadratically-- 00:02:47.470 --> 00:02:50.480 align:middle line:84% so this is called second order accurate-- 00:02:50.480 --> 00:02:52.400 align:middle line:84% you can derive it in a variety of ways. 00:02:52.400 --> 00:02:56.300 align:middle line:84% But one way to derive it is to just plug 00:02:56.300 --> 00:03:01.730 align:middle line:84% in the Taylor expansion for f of x plus dx and f of x minus dx. 00:03:01.730 --> 00:03:04.790 align:middle line:84% And what happens is because this is symmetric-- 00:03:04.790 --> 00:03:08.270 align:middle line:84% it's f of x plus dx minus f of x minus dx, 00:03:08.270 --> 00:03:13.850 align:middle line:84% all the terms with even powers of delta x in the Taylor series 00:03:13.850 --> 00:03:17.360 align:middle line:84% all cancel because if it's an even power, like remember, 00:03:17.360 --> 00:03:24.860 align:middle line:84% the there's f plus f prime dx plus f double prime dx over 2, 00:03:24.860 --> 00:03:28.380 align:middle line:84% and so forth-- that's your Taylor series-- 00:03:28.380 --> 00:03:30.320 align:middle line:84% f double prime dx squared over 2. 00:03:30.320 --> 00:03:31.550 align:middle line:90% Sorry. 00:03:31.550 --> 00:03:33.200 align:middle line:84% Anything with an even power of delta 00:03:33.200 --> 00:03:37.740 align:middle line:84% x, It doesn't matter if it's plus dx or minus dx, 00:03:37.740 --> 00:03:39.680 align:middle line:84% it's the same thing when you square it or take 00:03:39.680 --> 00:03:41.010 align:middle line:90% the fourth power and so forth. 00:03:41.010 --> 00:03:43.550 align:middle line:84% So when you subtract these two terms, they cancel. 00:03:43.550 --> 00:03:46.820 align:middle line:84% And so the error becomes the next order term, 00:03:46.820 --> 00:03:50.600 align:middle line:84% assuming it's at least three times differentiable. 00:03:50.600 --> 00:03:54.260 align:middle line:84% So it turns out this center difference formula, 00:03:54.260 --> 00:03:57.740 align:middle line:84% the error and the derivative goes like delta x squared, 00:03:57.740 --> 00:03:59.623 align:middle line:84% or if I multiply both sides by delta x, 00:03:59.623 --> 00:04:01.790 align:middle line:84% the error in the difference goes like delta x cubed. 00:04:01.790 --> 00:04:04.670 align:middle line:90% 00:04:04.670 --> 00:04:05.780 align:middle line:90% And we can check this. 00:04:05.780 --> 00:04:07.850 align:middle line:90% We can plot it for sine x. 00:04:07.850 --> 00:04:12.260 align:middle line:84% So I plot, for sine x, we're going to approximate it 00:04:12.260 --> 00:04:17.480 align:middle line:84% at x equals 1 by either the first order difference, sine 00:04:17.480 --> 00:04:22.370 align:middle line:84% of 1 plus something minus sine of 1 over s, 00:04:22.370 --> 00:04:24.740 align:middle line:84% or the center difference, sine of 1 00:04:24.740 --> 00:04:29.090 align:middle line:84% plus s minus sine of 1 minus s over 2s. 00:04:29.090 --> 00:04:32.960 align:middle line:84% I don't know why I called it s there. 00:04:32.960 --> 00:04:36.070 align:middle line:84% And I'll plot it as a function of delta x. 00:04:36.070 --> 00:04:41.290 align:middle line:84% And the forward difference is the blue. 00:04:41.290 --> 00:04:46.150 align:middle line:90% That goes linearly with delta x. 00:04:46.150 --> 00:04:52.670 align:middle line:84% The blue line is just a plot of delta x for reference, linear. 00:04:52.670 --> 00:04:57.890 align:middle line:84% And the central difference is the red dots. 00:04:57.890 --> 00:05:00.350 align:middle line:84% They go quadratically with delta x. 00:05:00.350 --> 00:05:02.880 align:middle line:90% 00:05:02.880 --> 00:05:07.048 align:middle line:84% And the straight red line is just a power law quadratic 00:05:07.048 --> 00:05:07.590 align:middle line:90% with delta x. 00:05:07.590 --> 00:05:09.570 align:middle line:84% Of course, again, this is a log log scale. 00:05:09.570 --> 00:05:11.730 align:middle line:84% So power laws turn into straight lines 00:05:11.730 --> 00:05:18.090 align:middle line:84% until in both cases, once they hit the machine precision, 00:05:18.090 --> 00:05:22.450 align:middle line:84% the error starts getting worse, if you make delta x too small. 00:05:22.450 --> 00:05:28.810 align:middle line:84% And in both cases, you want to have a delta x that's small, 00:05:28.810 --> 00:05:32.230 align:middle line:84% but not too small, like 10 to the minus 5, in this case, 00:05:32.230 --> 00:05:36.890 align:middle line:84% for the quadratic case, or 10 to the minus-- yeah, 00:05:36.890 --> 00:05:39.920 align:middle line:90% 5 was about where this-- 00:05:39.920 --> 00:05:42.670 align:middle line:90% 00:05:42.670 --> 00:05:45.510 align:middle line:84% the error is minimum in this case. 00:05:45.510 --> 00:05:50.310 align:middle line:84% And in fact, there are even higher order differences. 00:05:50.310 --> 00:05:55.830 align:middle line:90% If you take even more points-- 00:05:55.830 --> 00:05:57.810 align:middle line:84% in some sense, what you're doing is basically 00:05:57.810 --> 00:06:01.020 align:middle line:84% taking multiple points and fitting it to a polynomial 00:06:01.020 --> 00:06:03.970 align:middle line:90% and taking the slope from that. 00:06:03.970 --> 00:06:06.420 align:middle line:84% So as you get to add even more and more points, 00:06:06.420 --> 00:06:09.510 align:middle line:84% you can fit to higher and higher order degree polynomials 00:06:09.510 --> 00:06:11.760 align:middle line:84% and get higher order difference formulas 00:06:11.760 --> 00:06:17.610 align:middle line:84% that give even smaller error for the same size of delta x. 00:06:17.610 --> 00:06:21.470 align:middle line:84% Another even fancier thing to do is 00:06:21.470 --> 00:06:25.670 align:middle line:84% you can start with just some delta x 00:06:25.670 --> 00:06:28.640 align:middle line:84% and then compute a sequence of delta x's that you keep 00:06:28.640 --> 00:06:31.070 align:middle line:90% dividing it by 2 or something. 00:06:31.070 --> 00:06:32.960 align:middle line:84% And each time you divide it by 2, 00:06:32.960 --> 00:06:35.090 align:middle line:84% you fit it to a higher degree polynomial. 00:06:35.090 --> 00:06:39.860 align:middle line:84% And you're extrapolating the slope to delta x equals 0. 00:06:39.860 --> 00:06:42.110 align:middle line:84% And it turns out there's a way to do this adaptively. 00:06:42.110 --> 00:06:45.560 align:middle line:84% So you keep decreasing delta x by, say, a factor of 2, 00:06:45.560 --> 00:06:48.530 align:middle line:84% and you go to higher and higher order polynomials. 00:06:48.530 --> 00:06:52.100 align:middle line:84% But there's a way to watch this and figure out 00:06:52.100 --> 00:06:54.290 align:middle line:84% that at some point when delta x goes too small, 00:06:54.290 --> 00:06:56.870 align:middle line:84% you can notice that the error is getting worse. 00:06:56.870 --> 00:06:58.050 align:middle line:90% And you can stop. 00:06:58.050 --> 00:07:00.440 align:middle line:84% So you can adaptively find the optimal delta 00:07:00.440 --> 00:07:05.810 align:middle line:84% x and the optimal order to extrapolate as accurately 00:07:05.810 --> 00:07:06.615 align:middle line:90% as possible. 00:07:06.615 --> 00:07:08.240 align:middle line:84% And the fancy version of this is called 00:07:08.240 --> 00:07:09.440 align:middle line:90% Richardson extrapolation. 00:07:09.440 --> 00:07:12.938 align:middle line:84% So it's a really nice technique for extrapolating functions. 00:07:12.938 --> 00:07:14.730 align:middle line:84% I'm not going to go into it in more detail. 00:07:14.730 --> 00:07:16.280 align:middle line:90% But it's a fun thing to look up. 00:07:16.280 --> 00:07:18.050 align:middle line:84% I have a package for it in Julia. 00:07:18.050 --> 00:07:21.050 align:middle line:84% And there's a Julia package called FiniteDifferences.jl 00:07:21.050 --> 00:07:23.730 align:middle line:84% that has zillions of different finite difference formulas, 00:07:23.730 --> 00:07:25.520 align:middle line:84% including Richardson extrapolation. 00:07:25.520 --> 00:07:27.740 align:middle line:84% They can get pretty high order accuracy. 00:07:27.740 --> 00:07:30.770 align:middle line:84% But anyway, so I wanted to mention those. 00:07:30.770 --> 00:07:33.690 align:middle line:84% And the other thing I wanted to mention 00:07:33.690 --> 00:07:37.590 align:middle line:84% was what happens when you go to higher dimensions, so 00:07:37.590 --> 00:07:38.890 align:middle line:90% higher dimensional inputs. 00:07:38.890 --> 00:07:42.130 align:middle line:84% So suppose instead of just x is a number, 00:07:42.130 --> 00:07:47.350 align:middle line:84% you have x is a vector or something like that. 00:07:47.350 --> 00:07:52.620 align:middle line:84% So of course, this formula still works. 00:07:52.620 --> 00:07:56.480 align:middle line:84% We can still approximate f prime dx 00:07:56.480 --> 00:07:58.640 align:middle line:90% by f of x plus dx minus f of x. 00:07:58.640 --> 00:08:01.190 align:middle line:84% And there are still higher order versions as well. 00:08:01.190 --> 00:08:03.380 align:middle line:90% We can no longer divide by dx. 00:08:03.380 --> 00:08:06.940 align:middle line:90% But that's not a big deal. 00:08:06.940 --> 00:08:11.500 align:middle line:84% But each finite difference, if x is a vector, 00:08:11.500 --> 00:08:13.480 align:middle line:84% each finite difference basically only 00:08:13.480 --> 00:08:19.340 align:middle line:84% gives you the derivative in one direction in that space. 00:08:19.340 --> 00:08:24.950 align:middle line:84% So think of, for example, if f is a scalar, a scalar valued 00:08:24.950 --> 00:08:27.230 align:middle line:84% function, and x is a vector, then what you really 00:08:27.230 --> 00:08:28.580 align:middle line:90% want is the gradient. 00:08:28.580 --> 00:08:30.440 align:middle line:84% And every component of the gradient 00:08:30.440 --> 00:08:34.474 align:middle line:84% is the derivative in one direction in that space. 00:08:34.474 --> 00:08:38.070 align:middle line:90% 00:08:38.070 --> 00:08:40.799 align:middle line:84% But every finite difference only effectively gives you 00:08:40.799 --> 00:08:43.020 align:middle line:84% one component of the-- can give you 00:08:43.020 --> 00:08:45.270 align:middle line:90% one component of the gradient. 00:08:45.270 --> 00:08:46.890 align:middle line:84% It can only give you the derivative 00:08:46.890 --> 00:08:48.520 align:middle line:90% in that direction, dx. 00:08:48.520 --> 00:08:51.600 align:middle line:84% So if you want the derivative in all possible directions, 00:08:51.600 --> 00:08:56.380 align:middle line:84% and x is n dimensional, you need n finite differences. 00:08:56.380 --> 00:09:02.370 align:middle line:84% So for example, if x is an RN and f is a scalar, 00:09:02.370 --> 00:09:04.710 align:middle line:84% and you want every component of the gradient, 00:09:04.710 --> 00:09:06.300 align:middle line:84% and it has n components, you want 00:09:06.300 --> 00:09:08.730 align:middle line:84% each one of those partial derivatives, each one of those 00:09:08.730 --> 00:09:11.860 align:middle line:90% is a separate finite difference. 00:09:11.860 --> 00:09:14.170 align:middle line:84% So you need n finite differences. 00:09:14.170 --> 00:09:16.870 align:middle line:84% And so what happens is this gets this gets very expensive. 00:09:16.870 --> 00:09:18.870 align:middle line:84% So if you have three parameters, fine. 00:09:18.870 --> 00:09:20.470 align:middle line:84% You need three finite differences. 00:09:20.470 --> 00:09:22.470 align:middle line:84% But if you have a million parameters, 00:09:22.470 --> 00:09:25.680 align:middle line:84% then you need to take a finite differences a million times. 00:09:25.680 --> 00:09:27.780 align:middle line:84% You need a million different delta x's. 00:09:27.780 --> 00:09:30.960 align:middle line:84% And if f is a very expensive function, 00:09:30.960 --> 00:09:34.638 align:middle line:84% like evaluating a neural network, this is horrible. 00:09:34.638 --> 00:09:36.430 align:middle line:84% Evaluating a neural network a million times 00:09:36.430 --> 00:09:38.170 align:middle line:90% is very expensive. 00:09:38.170 --> 00:09:41.890 align:middle line:84% So the net result is that in high dimensions, 00:09:41.890 --> 00:09:44.728 align:middle line:84% this becomes incredibly impractical to compute 00:09:44.728 --> 00:09:46.270 align:middle line:84% finite differences if you really want 00:09:46.270 --> 00:09:49.910 align:middle line:84% all the derivatives, the derivatives in all directions, 00:09:49.910 --> 00:09:52.720 align:middle line:84% which you want for things like optimization. 00:09:52.720 --> 00:09:58.180 align:middle line:84% So you really, really need to do derivatives analytically 00:09:58.180 --> 00:10:00.610 align:middle line:84% in high dimensions in order for this 00:10:00.610 --> 00:10:02.890 align:middle line:84% to be practical for things like machine 00:10:02.890 --> 00:10:05.780 align:middle line:84% learning, optimizing neural networks, to be practical. 00:10:05.780 --> 00:10:08.740 align:middle line:84% That's the only way to get the a million derivatives 00:10:08.740 --> 00:10:10.480 align:middle line:90% efficiently. 00:10:10.480 --> 00:10:13.630 align:middle line:84% On the other hand, they're still very useful as a check. 00:10:13.630 --> 00:10:18.633 align:middle line:84% So as a check, if you're developing your own derivative, 00:10:18.633 --> 00:10:20.800 align:middle line:84% for example, if you're doing your homework problems, 00:10:20.800 --> 00:10:24.280 align:middle line:84% and some of those functions, it's not so easy. 00:10:24.280 --> 00:10:27.637 align:middle line:84% Derivatives-- you might think in 1801, derivatives are easy. 00:10:27.637 --> 00:10:29.470 align:middle line:84% But when the functions get more complicated, 00:10:29.470 --> 00:10:31.600 align:middle line:84% you have vectors in, vectors out, or vectors in, 00:10:31.600 --> 00:10:35.740 align:middle line:84% matrices out, probably you're seeing it's not always 00:10:35.740 --> 00:10:38.470 align:middle line:84% so easy to get the derivative and be 00:10:38.470 --> 00:10:40.810 align:middle line:84% confident you did all the chain rules and product 00:10:40.810 --> 00:10:42.940 align:middle line:90% rules correctly. 00:10:42.940 --> 00:10:45.340 align:middle line:84% So it's really, really useful to check it. 00:10:45.340 --> 00:10:47.710 align:middle line:84% And I definitely encourage you on homework to check it. 00:10:47.710 --> 00:10:52.770 align:middle line:84% And you can very easily check it just by, basically, 00:10:52.770 --> 00:10:54.495 align:middle line:84% just computing a few random directions, 00:10:54.495 --> 00:10:58.620 align:middle line:84% so to choose delta x as a random vector that's small, 00:10:58.620 --> 00:11:00.210 align:middle line:90% do it a few times. 00:11:00.210 --> 00:11:02.760 align:middle line:84% If that matches to several decimal 00:11:02.760 --> 00:11:07.160 align:middle line:84% places your f prime dx, that's a pretty good sign 00:11:07.160 --> 00:11:09.170 align:middle line:90% that you didn't make a mistake. 00:11:09.170 --> 00:11:13.430 align:middle line:84% It's very unlikely that if you have a bug, that it will just 00:11:13.430 --> 00:11:15.980 align:middle line:84% happen to be correct to six decimal places 00:11:15.980 --> 00:11:17.915 align:middle line:84% in some random direction in space. 00:11:17.915 --> 00:11:20.960 align:middle line:90% 00:11:20.960 --> 00:11:22.770 align:middle line:84% So I just wanted to stop with that. 00:11:22.770 --> 00:11:24.770 align:middle line:84% So any questions about finite differences 00:11:24.770 --> 00:11:27.270 align:middle line:84% before I'm going to move on to a completely different topic? 00:11:27.270 --> 00:11:33.950 align:middle line:90% 00:11:33.950 --> 00:11:41.390 align:middle line:84% So what I want to do now is talk more about gradients. 00:11:41.390 --> 00:11:43.460 align:middle line:84% And I want to generalize gradients 00:11:43.460 --> 00:11:46.670 align:middle line:90% to higher dimensional problems. 00:11:46.670 --> 00:11:50.450 align:middle line:84% So the familiar gradient from 18.02 00:11:50.450 --> 00:11:56.630 align:middle line:84% is when you have a function that takes a vector in and gives you 00:11:56.630 --> 00:12:00.350 align:middle line:84% a scalar out, just where the vector is just like an n 00:12:00.350 --> 00:12:02.120 align:middle line:90% component column vector. 00:12:02.120 --> 00:12:07.370 align:middle line:84% And then the gradient is an n component column vector. 00:12:07.370 --> 00:12:10.010 align:middle line:84% It's a component vector of partial f, 00:12:10.010 --> 00:12:14.332 align:middle line:84% partial x1, partial f, partial x2, and so forth. 00:12:14.332 --> 00:12:16.040 align:middle line:84% And in this class, we try and think of it 00:12:16.040 --> 00:12:22.370 align:middle line:84% more holistically that we have the differential, df, 00:12:22.370 --> 00:12:25.640 align:middle line:90% is f of x plus dx minus f of x. 00:12:25.640 --> 00:12:26.710 align:middle line:90% So this is just review. 00:12:26.710 --> 00:12:29.620 align:middle line:90% 00:12:29.620 --> 00:12:34.780 align:middle line:84% Our derivative is the linear operator that takes in the dx 00:12:34.780 --> 00:12:36.580 align:middle line:90% and gives you df to first order. 00:12:36.580 --> 00:12:40.490 align:middle line:84% So this linear operator takes in a vector dx 00:12:40.490 --> 00:12:43.700 align:middle line:90% and gives you out a scalar. 00:12:43.700 --> 00:12:46.790 align:middle line:84% So we sometimes call that a linear form, something that 00:12:46.790 --> 00:12:48.830 align:middle line:84% takes a vector in and a scalar out. 00:12:48.830 --> 00:12:53.970 align:middle line:84% And if this is a column vector, an n component column vector, 00:12:53.970 --> 00:12:57.120 align:middle line:84% it's pretty easy to see that the only linear operator that 00:12:57.120 --> 00:13:00.360 align:middle line:84% takes a column vector in and gives a scalar out 00:13:00.360 --> 00:13:03.810 align:middle line:84% is multiplying by a row vector, or equivalently, 00:13:03.810 --> 00:13:07.590 align:middle line:84% is taking a dot product of dx with some vector. 00:13:07.590 --> 00:13:09.820 align:middle line:84% And that thing we call the gradient. 00:13:09.820 --> 00:13:12.630 align:middle line:84% So this f prime of x is then a row vector. 00:13:12.630 --> 00:13:14.850 align:middle line:84% It's the transpose of the gradient and the gradient, 00:13:14.850 --> 00:13:16.830 align:middle line:84% or equivalently, the gradient is the thing 00:13:16.830 --> 00:13:20.790 align:middle line:84% you take the dot product of dx with to get df. 00:13:20.790 --> 00:13:25.450 align:middle line:84% And this is equivalent to the 18.02 definition of gradient. 00:13:25.450 --> 00:13:29.460 align:middle line:84% But what I want to do today is talk about generalizing this 00:13:29.460 --> 00:13:32.490 align:middle line:84% to other kinds of vector spaces, to where 00:13:32.490 --> 00:13:35.340 align:middle line:84% x is not necessarily a column vector, but some other kind 00:13:35.340 --> 00:13:36.240 align:middle line:90% of vector. 00:13:36.240 --> 00:13:39.370 align:middle line:84% But we still have a scalar valued function. 00:13:39.370 --> 00:13:42.180 align:middle line:84% So we're still going to generalize to-- 00:13:42.180 --> 00:13:44.740 align:middle line:84% we're still talking about scalar value functions, 00:13:44.740 --> 00:13:47.400 align:middle line:84% for example, a function that takes a matrix in and gives you 00:13:47.400 --> 00:13:48.250 align:middle line:90% a number out. 00:13:48.250 --> 00:13:51.390 align:middle line:84% So for example, a determinant is a function 00:13:51.390 --> 00:13:54.550 align:middle line:84% that takes a matrix in, but gives you a scalar out. 00:13:54.550 --> 00:13:59.470 align:middle line:84% And can we talk about the gradient of such a function? 00:13:59.470 --> 00:14:02.830 align:middle line:84% What is the gradient of a determinant? 00:14:02.830 --> 00:14:05.710 align:middle line:84% And so we'd like to have it this same kind of thing, 00:14:05.710 --> 00:14:07.240 align:middle line:84% where it's some kind of dot product, 00:14:07.240 --> 00:14:12.520 align:middle line:84% and where the gradient is of the same shape as x. 00:14:12.520 --> 00:14:15.490 align:middle line:84% So if x is a column vector here of n components, 00:14:15.490 --> 00:14:16.870 align:middle line:90% the gradient is a column vector. 00:14:16.870 --> 00:14:18.820 align:middle line:84% And this is going to be really important soon, 00:14:18.820 --> 00:14:21.237 align:middle line:84% when you start talking about optimization because then you 00:14:21.237 --> 00:14:24.370 align:middle line:84% can think of the gradient as the uphill direction 00:14:24.370 --> 00:14:25.130 align:middle line:90% for the function. 00:14:25.130 --> 00:14:27.003 align:middle line:84% So if you want to make the function bigger, 00:14:27.003 --> 00:14:28.670 align:middle line:84% you go in the direction of the gradient. 00:14:28.670 --> 00:14:29.410 align:middle line:90% That's uphill. 00:14:29.410 --> 00:14:31.600 align:middle line:84% If you want to make a function smaller, 00:14:31.600 --> 00:14:33.980 align:middle line:84% you go in the opposite direction of the gradient. 00:14:33.980 --> 00:14:35.840 align:middle line:90% That's downhill. 00:14:35.840 --> 00:14:38.030 align:middle line:84% So how do we generalize this to other kinds 00:14:38.030 --> 00:14:41.660 align:middle line:84% of things that are not just n component column 00:14:41.660 --> 00:14:46.320 align:middle line:84% vectors and not the familiar 18.02 gradient? 00:14:46.320 --> 00:14:50.068 align:middle line:90% So I think it's pretty easy to-- 00:14:50.068 --> 00:14:52.110 align:middle line:84% what we basically want is the same kind of thing. 00:14:52.110 --> 00:14:55.900 align:middle line:84% If x lives in some arbitrary vector space, 00:14:55.900 --> 00:14:57.540 align:middle line:84% and we have a scalar function that 00:14:57.540 --> 00:15:01.410 align:middle line:84% takes x in that vector space and gives you a number out, 00:15:01.410 --> 00:15:05.670 align:middle line:84% a real number-- everything's real in our class right now. 00:15:05.670 --> 00:15:09.120 align:middle line:84% So all our scalars are real numbers. 00:15:09.120 --> 00:15:13.380 align:middle line:84% So it's a linear operator that takes a dx, a vector, 00:15:13.380 --> 00:15:15.210 align:middle line:90% in and gives you a scalar out. 00:15:15.210 --> 00:15:17.070 align:middle line:84% And we want to define a gradient. 00:15:17.070 --> 00:15:18.900 align:middle line:84% What we want is we want to define this 00:15:18.900 --> 00:15:21.220 align:middle line:90% as some kind of dot product. 00:15:21.220 --> 00:15:25.290 align:middle line:84% And so to do that, we need to define a dot product 00:15:25.290 --> 00:15:27.100 align:middle line:90% for our vector space. 00:15:27.100 --> 00:15:29.490 align:middle line:84% So in general, gradients, we're going 00:15:29.490 --> 00:15:32.490 align:middle line:84% to be able to define gradients any time we have a vector 00:15:32.490 --> 00:15:35.430 align:middle line:84% space, we have a scalar value function, 00:15:35.430 --> 00:15:37.810 align:middle line:84% and we have a dot product on the vector space. 00:15:37.810 --> 00:15:41.040 align:middle line:84% So we need to generalize our notion of a dot product 00:15:41.040 --> 00:15:43.960 align:middle line:90% to other kinds of vector spaces. 00:15:43.960 --> 00:15:48.070 align:middle line:84% So first of all, let me just give you the general definition 00:15:48.070 --> 00:15:49.210 align:middle line:90% of a dot product. 00:15:49.210 --> 00:15:51.475 align:middle line:84% So usually, once you generalize it 00:15:51.475 --> 00:15:54.100 align:middle line:84% to other kind of vector spaces, you don't call it a dot product 00:15:54.100 --> 00:15:54.460 align:middle line:90% anymore. 00:15:54.460 --> 00:15:55.930 align:middle line:90% People call it an inner product. 00:15:55.930 --> 00:15:59.350 align:middle line:84% But dot product is a perfectly good colloquial name for that. 00:15:59.350 --> 00:16:03.795 align:middle line:84% But the pure mathematicians will look at you 00:16:03.795 --> 00:16:05.920 align:middle line:84% sideways a little bit if you call it a dot product. 00:16:05.920 --> 00:16:08.080 align:middle line:90% You call it an inner product. 00:16:08.080 --> 00:16:10.570 align:middle line:84% So an inner product, or a dot product, 00:16:10.570 --> 00:16:15.220 align:middle line:84% is some rule that takes two vectors in our space. 00:16:15.220 --> 00:16:16.240 align:middle line:90% Call it x and y. 00:16:16.240 --> 00:16:18.200 align:middle line:84% And again, this is some general vector space. 00:16:18.200 --> 00:16:19.870 align:middle line:90% So it could be column vectors. 00:16:19.870 --> 00:16:22.360 align:middle line:84% But it could also be matrices or something 00:16:22.360 --> 00:16:24.310 align:middle line:90% even more complicated. 00:16:24.310 --> 00:16:27.490 align:middle line:84% And it's a rule that gives you a scalar out of them. 00:16:27.490 --> 00:16:35.170 align:middle line:84% And we can denote that in 18.02 style, or freshman physics 00:16:35.170 --> 00:16:38.710 align:middle line:90% style, as x dot y. 00:16:38.710 --> 00:16:40.000 align:middle line:90% That's there. 00:16:40.000 --> 00:16:44.170 align:middle line:84% More commonly, in pure math, they often use angle brackets. 00:16:44.170 --> 00:16:48.970 align:middle line:84% So they would denote this thing by angle x, comma, y. 00:16:48.970 --> 00:16:50.980 align:middle line:90% I just wrote it here. 00:16:50.980 --> 00:16:53.410 align:middle line:84% In quantum mechanics and physics, 00:16:53.410 --> 00:16:56.290 align:middle line:84% they like this bracket notation, where they use angle brackets, 00:16:56.290 --> 00:16:59.260 align:middle line:84% but they put a vertical bar in between. 00:16:59.260 --> 00:17:01.150 align:middle line:84% And there's some complications there. 00:17:01.150 --> 00:17:04.000 align:middle line:84% They really call the vectors, they call them x, 00:17:04.000 --> 00:17:06.369 align:middle line:90% with brackets around them. 00:17:06.369 --> 00:17:08.790 align:middle line:84% But I'm not going to worry about that too much. 00:17:08.790 --> 00:17:10.790 align:middle line:84% But you may have seen, this is an inner product. 00:17:10.790 --> 00:17:11.928 align:middle line:90% This is an inner product. 00:17:11.928 --> 00:17:12.970 align:middle line:90% This is an inner product. 00:17:12.970 --> 00:17:15.609 align:middle line:84% I'm just going to use dots in this class, 00:17:15.609 --> 00:17:19.589 align:middle line:84% just to keep it kind of elementary looking. 00:17:19.589 --> 00:17:21.569 align:middle line:84% So it's a rule that takes two vectors 00:17:21.569 --> 00:17:23.950 align:middle line:90% and gives you a scalar out. 00:17:23.950 --> 00:17:26.640 align:middle line:84% And right now, we're dealing with just real vector spaces. 00:17:26.640 --> 00:17:28.800 align:middle line:84% So our scalar is going to be real numbers. 00:17:28.800 --> 00:17:32.700 align:middle line:84% And it has to have certain properties to be a dot product. 00:17:32.700 --> 00:17:35.340 align:middle line:84% Basically, you need to obey certain rules, 00:17:35.340 --> 00:17:37.980 align:middle line:84% so that you can do the familiar rules of algebra 00:17:37.980 --> 00:17:40.570 align:middle line:84% with it that you do with your familiar dot products. 00:17:40.570 --> 00:17:43.230 align:middle line:84% You want it to work, those algebraic rules to work, 00:17:43.230 --> 00:17:46.290 align:middle line:84% with your dot products on whatever crazy vector space 00:17:46.290 --> 00:17:47.070 align:middle line:90% you have. 00:17:47.070 --> 00:17:49.870 align:middle line:84% And it turns out, you really just need three rules. 00:17:49.870 --> 00:17:52.440 align:middle line:84% So first of all, we want it to be symmetric. 00:17:52.440 --> 00:17:54.180 align:middle line:84% So the rule should be that it should 00:17:54.180 --> 00:17:58.730 align:middle line:84% satisfy that x dot y equals y dot x, for any x and y. 00:17:58.730 --> 00:18:00.590 align:middle line:84% If these are complex numbers, then you 00:18:00.590 --> 00:18:04.055 align:middle line:84% need a complex conjugate over this. 00:18:04.055 --> 00:18:07.280 align:middle line:90% 00:18:07.280 --> 00:18:11.060 align:middle line:84% But since this is real numbers, I'm 00:18:11.060 --> 00:18:13.798 align:middle line:84% only dealing with real numbers, I can just say it's symmetric, 00:18:13.798 --> 00:18:16.340 align:middle line:84% but just keep in mind, if you ever work with complex numbers, 00:18:16.340 --> 00:18:19.370 align:middle line:90% there is a conjugation here. 00:18:19.370 --> 00:18:23.360 align:middle line:84% Second, and of course, our familiar dot product certainly 00:18:23.360 --> 00:18:28.100 align:middle line:84% is symmetric, where you multiply the components and add them up. 00:18:28.100 --> 00:18:31.280 align:middle line:90% That's symmetric. 00:18:31.280 --> 00:18:34.610 align:middle line:90% And then it has to be linear. 00:18:34.610 --> 00:18:36.150 align:middle line:84% And so that means if, for example, 00:18:36.150 --> 00:18:41.560 align:middle line:84% if you take the dot product of x with alpha y plus beta z-- 00:18:41.560 --> 00:18:44.230 align:middle line:84% and as usual, Greek letters mean scalars here-- then that 00:18:44.230 --> 00:18:46.510 align:middle line:84% has to be the same thing as alpha times 00:18:46.510 --> 00:18:50.080 align:middle line:90% x dot y plus beta times x dot z. 00:18:50.080 --> 00:18:54.260 align:middle line:84% And again, the familiar dot product is certainly linear. 00:18:54.260 --> 00:18:58.740 align:middle line:84% And since it's symmetric, this also means it's linear. 00:18:58.740 --> 00:19:01.650 align:middle line:84% I wrote it down here, it is linear in the second argument. 00:19:01.650 --> 00:19:07.562 align:middle line:84% But from rule number one, I would swap the second argument 00:19:07.562 --> 00:19:08.520 align:middle line:90% for the first argument. 00:19:08.520 --> 00:19:10.520 align:middle line:84% It means it's also linear in the first argument. 00:19:10.520 --> 00:19:14.170 align:middle line:90% 00:19:14.170 --> 00:19:17.080 align:middle line:84% And the third rule that it has to satisfy 00:19:17.080 --> 00:19:18.820 align:middle line:84% if we want to call it an inner product 00:19:18.820 --> 00:19:20.560 align:middle line:84% is that it has to be non-negative. 00:19:20.560 --> 00:19:24.070 align:middle line:84% So if you take the dot product of a vector with itself, 00:19:24.070 --> 00:19:27.860 align:middle line:84% we're going to call that the norm of the vector squared. 00:19:27.860 --> 00:19:31.180 align:middle line:84% So the norm of the vector is the square root of this. 00:19:31.180 --> 00:19:33.460 align:middle line:90% So x dot itself is norm squared. 00:19:33.460 --> 00:19:34.932 align:middle line:84% Just like the familiar dot product, 00:19:34.932 --> 00:19:36.640 align:middle line:84% the dot product with itself is the length 00:19:36.640 --> 00:19:38.290 align:middle line:90% of the vector squared. 00:19:38.290 --> 00:19:43.390 align:middle line:84% That length had better not be a negative number. 00:19:43.390 --> 00:19:45.700 align:middle line:84% We're going to require that anything, if it wants 00:19:45.700 --> 00:19:47.530 align:middle line:84% to call itself an inner product, x 00:19:47.530 --> 00:19:50.230 align:middle line:84% dot itself has to be non-negative. 00:19:50.230 --> 00:19:55.330 align:middle line:84% And furthermore, it can only equal if and only 00:19:55.330 --> 00:19:56.230 align:middle line:90% if x equals 0. 00:19:56.230 --> 00:19:57.910 align:middle line:90% So the 0 vector-- 00:19:57.910 --> 00:20:01.230 align:middle line:84% every vector space must have a 0 vector. 00:20:01.230 --> 00:20:04.430 align:middle line:84% And the 0 vector, its length had better be 0. 00:20:04.430 --> 00:20:07.880 align:middle line:84% And it better be the only vector that has a length of 0. 00:20:07.880 --> 00:20:10.040 align:middle line:84% And if you have these three rules, 00:20:10.040 --> 00:20:12.650 align:middle line:84% then you can call it an inner product. 00:20:12.650 --> 00:20:14.820 align:middle line:84% And any questions on these rules yet? 00:20:14.820 --> 00:20:16.820 align:middle line:84% And I'm going to give some examples in a second. 00:20:16.820 --> 00:20:20.190 align:middle line:90% 00:20:20.190 --> 00:20:21.760 align:middle line:90% So this is an inner product. 00:20:21.760 --> 00:20:23.988 align:middle line:84% And so now, just for terminology, 00:20:23.988 --> 00:20:25.780 align:middle line:84% I'm not going to use this terminology much, 00:20:25.780 --> 00:20:30.130 align:middle line:84% but if you have a continuous vector space-- 00:20:30.130 --> 00:20:31.530 align:middle line:90% you know what a vector space is. 00:20:31.530 --> 00:20:34.200 align:middle line:84% So if you add an inner product and it's a continuous vector 00:20:34.200 --> 00:20:36.450 align:middle line:84% space, so it's the real numbers and you 00:20:36.450 --> 00:20:38.430 align:middle line:84% can vary the vector smoothly, people 00:20:38.430 --> 00:20:39.730 align:middle line:90% call this a Hilbert space. 00:20:39.730 --> 00:20:42.060 align:middle line:84% So if you ever hear that term, don't get scared. 00:20:42.060 --> 00:20:43.602 align:middle line:84% It just means you have a vector space 00:20:43.602 --> 00:20:46.170 align:middle line:90% and you have a dot product. 00:20:46.170 --> 00:20:48.490 align:middle line:84% And so now we can define a gradient. 00:20:48.490 --> 00:20:51.420 align:middle line:84% So if you have a scalar function that 00:20:51.420 --> 00:20:54.780 align:middle line:84% takes a vector in and gives you a scalar out, 00:20:54.780 --> 00:20:58.290 align:middle line:84% but vector space has a dot product, 00:20:58.290 --> 00:21:04.270 align:middle line:84% so it's a Hilbert space, then the derivative 00:21:04.270 --> 00:21:08.500 align:middle line:84% must be a linear function that takes a vector in and gives you 00:21:08.500 --> 00:21:10.410 align:middle line:90% a scalar out. 00:21:10.410 --> 00:21:14.270 align:middle line:84% And it turns out that if you're in a vector space 00:21:14.270 --> 00:21:19.250 align:middle line:84% and you have a dot product, and you have a linear function that 00:21:19.250 --> 00:21:21.740 align:middle line:84% takes-- it's called a linear form that takes a vector in 00:21:21.740 --> 00:21:27.020 align:middle line:84% and a vector out, every linear form must be of the form 00:21:27.020 --> 00:21:29.750 align:middle line:84% the dot product of some vector with the x. 00:21:29.750 --> 00:21:30.810 align:middle line:90% This is always true. 00:21:30.810 --> 00:21:33.410 align:middle line:84% So this is certainly true for column vectors. 00:21:33.410 --> 00:21:35.840 align:middle line:84% For a column vector, if it takes a column vector 00:21:35.840 --> 00:21:39.312 align:middle line:84% and gives you a number out, it must be 00:21:39.312 --> 00:21:40.520 align:middle line:90% a dot product with something. 00:21:40.520 --> 00:21:43.185 align:middle line:84% It turns out this is true in general for any Hilbert space, 00:21:43.185 --> 00:21:44.810 align:middle line:84% any vector space with an inner product. 00:21:44.810 --> 00:21:47.690 align:middle line:84% If you have a linear function, vector in, number out, 00:21:47.690 --> 00:21:50.060 align:middle line:84% it must be a dot product with something. 00:21:50.060 --> 00:21:53.900 align:middle line:84% This is called the Riesz representation theorem, 00:21:53.900 --> 00:21:55.490 align:middle line:90% by the way. 00:21:55.490 --> 00:21:59.315 align:middle line:84% I feel like I spelled Riesz wrong. 00:21:59.315 --> 00:22:01.700 align:middle line:90% I think it's C-Z, like this. 00:22:01.700 --> 00:22:08.240 align:middle line:84% Yes, I think it's C-Z. It's a Riesz representation-- 00:22:08.240 --> 00:22:09.200 align:middle line:90% I'll double check. 00:22:09.200 --> 00:22:10.240 align:middle line:90% Or is it Z-C? 00:22:10.240 --> 00:22:13.900 align:middle line:90% 00:22:13.900 --> 00:22:14.400 align:middle line:90% No. 00:22:14.400 --> 00:22:18.080 align:middle line:84% I think it's C-Z, one of those names. 00:22:18.080 --> 00:22:19.220 align:middle line:90% I can never remember it. 00:22:19.220 --> 00:22:22.980 align:middle line:84% Yeah, so this is the theroem called the Riesz representation 00:22:22.980 --> 00:22:23.480 align:middle line:90% theorem. 00:22:23.480 --> 00:22:25.010 align:middle line:84% But it's kind of an intuitive thing. 00:22:25.010 --> 00:22:27.260 align:middle line:84% If you think of column vectors, it's very intuitive. 00:22:27.260 --> 00:22:29.630 align:middle line:84% If you have a linear rule, a column vector to a number, 00:22:29.630 --> 00:22:30.672 align:middle line:90% it must be a dot product. 00:22:30.672 --> 00:22:33.290 align:middle line:84% It turns out this is always true. 00:22:33.290 --> 00:22:35.420 align:middle line:84% This must be a dot product with something. 00:22:35.420 --> 00:22:39.170 align:middle line:84% And we call that thing the gradient. 00:22:39.170 --> 00:22:42.470 align:middle line:84% So the gradient for any vector space, 00:22:42.470 --> 00:22:44.450 align:middle line:84% any scalar function of any vector space 00:22:44.450 --> 00:22:46.580 align:middle line:84% with an inner product, is the thing 00:22:46.580 --> 00:22:50.720 align:middle line:84% you take the dot product with to get your df. 00:22:50.720 --> 00:22:57.095 align:middle line:90% So this is our df. 00:22:57.095 --> 00:22:59.760 align:middle line:90% 00:22:59.760 --> 00:23:01.380 align:middle line:84% And so it's always something that has 00:23:01.380 --> 00:23:05.450 align:middle line:90% kind of the same shape as x. 00:23:05.450 --> 00:23:07.790 align:middle line:90% So let me just do some examples. 00:23:07.790 --> 00:23:11.630 align:middle line:84% So of course, the example, the familiar example, 00:23:11.630 --> 00:23:17.240 align:middle line:84% is just our vector space is just our n. 00:23:17.240 --> 00:23:20.060 align:middle line:84% So it's n component column vectors. 00:23:20.060 --> 00:23:25.030 align:middle line:90% 00:23:25.030 --> 00:23:34.180 align:middle line:84% And then our dot product is just x dot y is x transpose y. 00:23:34.180 --> 00:23:35.560 align:middle line:90% That's our familiar dot product. 00:23:35.560 --> 00:23:39.060 align:middle line:90% 00:23:39.060 --> 00:23:43.020 align:middle line:90% Well, I shouldn't say therefore. 00:23:43.020 --> 00:23:46.420 align:middle line:90% 00:23:46.420 --> 00:23:49.208 align:middle line:84% this is the familiar Euclidean dot product. 00:23:49.208 --> 00:23:50.875 align:middle line:84% It's not the only possible dot product-- 00:23:50.875 --> 00:23:53.480 align:middle line:90% 00:23:53.480 --> 00:23:54.440 align:middle line:90% Euclidean. 00:23:54.440 --> 00:24:02.860 align:middle line:90% 00:24:02.860 --> 00:24:05.500 align:middle line:84% There are actually other dot products. 00:24:05.500 --> 00:24:07.360 align:middle line:90% You could also have this. 00:24:07.360 --> 00:24:10.610 align:middle line:90% So this is-- what is this? 00:24:10.610 --> 00:24:22.520 align:middle line:84% This is basically x1y1 plus x2y2 plus dot, dot, dot. 00:24:22.520 --> 00:24:26.110 align:middle line:84% You just multiply the components and add them up. 00:24:26.110 --> 00:24:30.285 align:middle line:84% But you could also have, for example, a weighted dot 00:24:30.285 --> 00:24:30.785 align:middle line:90% product. 00:24:30.785 --> 00:24:35.680 align:middle line:90% 00:24:35.680 --> 00:24:41.115 align:middle line:84% So we have-- let's call it x dot-- 00:24:41.115 --> 00:24:42.490 align:middle line:84% I need to use a different symbol. 00:24:42.490 --> 00:24:50.820 align:middle line:90% Let's call it x dot sub w y is-- 00:24:50.820 --> 00:24:52.938 align:middle line:90% I could do x1y1-- 00:24:52.938 --> 00:24:56.730 align:middle line:90% 00:24:56.730 --> 00:25:05.860 align:middle line:84% sorry-- w1x1y1 plus w2x2y2 plus dot, dot, 00:25:05.860 --> 00:25:20.887 align:middle line:84% dot for weights w1 to wn that all have to be positive. 00:25:20.887 --> 00:25:22.220 align:middle line:90% Why do they have to be positive? 00:25:22.220 --> 00:25:25.670 align:middle line:84% Because of the non-negativity property, for us 00:25:25.670 --> 00:25:27.377 align:middle line:90% to call this an inner product. 00:25:27.377 --> 00:25:28.460 align:middle line:90% Let me put those in black. 00:25:28.460 --> 00:25:31.700 align:middle line:90% 00:25:31.700 --> 00:25:36.370 align:middle line:84% If it had property property 3, you 00:25:36.370 --> 00:25:39.430 align:middle line:84% have to have dot products be non-negative. 00:25:39.430 --> 00:25:41.800 align:middle line:84% It means the weights will have to be non-negative. 00:25:41.800 --> 00:25:43.435 align:middle line:84% This is a perfectly good inner product. 00:25:43.435 --> 00:25:44.560 align:middle line:90% It's not the Euclidean one. 00:25:44.560 --> 00:25:46.930 align:middle line:84% It weights each dimension differently. 00:25:46.930 --> 00:25:49.820 align:middle line:84% But this might be useful in statistics or something 00:25:49.820 --> 00:25:50.320 align:middle line:90% like that. 00:25:50.320 --> 00:25:55.630 align:middle line:84% For example, if you want to have a bunch of measurements, 00:25:55.630 --> 00:25:57.940 align:middle line:84% but there's a different uncertainty 00:25:57.940 --> 00:26:01.840 align:middle line:84% in each measurement, you might want to weight the numbers-- 00:26:01.840 --> 00:26:04.660 align:middle line:84% the components that have more uncertainty, 00:26:04.660 --> 00:26:06.430 align:middle line:84% you might want to weight them less. 00:26:06.430 --> 00:26:07.960 align:middle line:90% You give them less weight. 00:26:07.960 --> 00:26:10.810 align:middle line:84% And the ones that are more certain, give them more weight. 00:26:10.810 --> 00:26:15.913 align:middle line:84% Or another example, if you had a vector where 00:26:15.913 --> 00:26:17.830 align:middle line:84% the different components have different units, 00:26:17.830 --> 00:26:22.150 align:middle line:84% for example, a vector where the first component is in meters, 00:26:22.150 --> 00:26:24.700 align:middle line:84% and the second component is in seconds, 00:26:24.700 --> 00:26:28.090 align:middle line:84% and the third component is in kilograms, 00:26:28.090 --> 00:26:29.980 align:middle line:84% then the ordinary dot product formula 00:26:29.980 --> 00:26:31.360 align:middle line:90% doesn't even make sense. 00:26:31.360 --> 00:26:33.470 align:middle line:84% It'll be adding up things with different units. 00:26:33.470 --> 00:26:35.660 align:middle line:84% So then you have to weight each one. 00:26:35.660 --> 00:26:38.900 align:middle line:84% You divide-- the first weight would 00:26:38.900 --> 00:26:42.467 align:middle line:84% be 1 over some characteristic length scale in meters. 00:26:42.467 --> 00:26:45.050 align:middle line:84% The second one would be some 1 over some characteristic length 00:26:45.050 --> 00:26:45.740 align:middle line:90% scale in time. 00:26:45.740 --> 00:26:48.110 align:middle line:84% The third one would be 1 over some characteristic length 00:26:48.110 --> 00:26:49.850 align:middle line:90% scale in kilograms. 00:26:49.850 --> 00:26:51.740 align:middle line:84% And that way, each term is dimensionless. 00:26:51.740 --> 00:26:55.630 align:middle line:84% You can add them up or something like that. 00:26:55.630 --> 00:26:58.330 align:middle line:84% Another way it's also useful to write this down 00:26:58.330 --> 00:27:00.440 align:middle line:90% is linear algebra. 00:27:00.440 --> 00:27:08.950 align:middle line:84% So we could also write this down as x transpose y. 00:27:08.950 --> 00:27:14.410 align:middle line:84% And what do we put in the middle to get this? 00:27:14.410 --> 00:27:15.250 align:middle line:90% Any ideas? 00:27:15.250 --> 00:27:18.820 align:middle line:90% 00:27:18.820 --> 00:27:20.550 align:middle line:84% What should it look like in the middle? 00:27:20.550 --> 00:27:23.862 align:middle line:90% What can I put here? 00:27:23.862 --> 00:27:24.820 align:middle line:90% Let me give you a hint. 00:27:24.820 --> 00:27:25.680 align:middle line:90% It's a matrix. 00:27:25.680 --> 00:27:30.440 align:middle line:84% What's the entries of that matrix? 00:27:30.440 --> 00:27:32.460 align:middle line:84% Alex, if someone says an answer, I 00:27:32.460 --> 00:27:33.710 align:middle line:90% can't hear anyone in the room. 00:27:33.710 --> 00:27:39.130 align:middle line:90% 00:27:39.130 --> 00:27:41.290 align:middle line:84% Alex, is anyone raising their hands? 00:27:41.290 --> 00:27:43.360 align:middle line:90% AUDIENCE: Yeah. 00:27:43.360 --> 00:27:44.560 align:middle line:90% W on the diagonal. 00:27:44.560 --> 00:27:45.490 align:middle line:90% Can you hear that? 00:27:45.490 --> 00:27:46.300 align:middle line:90% STEVEN JOHNSON: Exactly. 00:27:46.300 --> 00:27:47.570 align:middle line:84% I cannot hear anything in the room. 00:27:47.570 --> 00:27:49.278 align:middle line:84% So you have to speak into the microphone. 00:27:49.278 --> 00:27:53.507 align:middle line:84% So that's your job, is to repeat any comments or questions 00:27:53.507 --> 00:27:54.340 align:middle line:90% from the microphone. 00:27:54.340 --> 00:27:54.840 align:middle line:90% Yes. 00:27:54.840 --> 00:28:02.540 align:middle line:90% So it's just a diagonal matrix. 00:28:02.540 --> 00:28:10.535 align:middle line:84% And an another example is another weighted dot product. 00:28:10.535 --> 00:28:19.560 align:middle line:90% 00:28:19.560 --> 00:28:22.110 align:middle line:90% Let's call this x-- 00:28:22.110 --> 00:28:25.580 align:middle line:90% so this was with-- 00:28:25.580 --> 00:28:35.990 align:middle line:84% let's call this x dot y sub capital W, for column vectors, 00:28:35.990 --> 00:28:46.370 align:middle line:90% would be x transpose y. 00:28:46.370 --> 00:28:50.120 align:middle line:90% And you put any matrix in here. 00:28:50.120 --> 00:28:53.840 align:middle line:84% But you need it to be symmetric for property one. 00:28:53.840 --> 00:28:59.285 align:middle line:84% So we need W to be symmetric for property one. 00:28:59.285 --> 00:29:02.470 align:middle line:90% 00:29:02.470 --> 00:29:07.090 align:middle line:84% And we need for property three, for positivity, we 00:29:07.090 --> 00:29:08.890 align:middle line:90% need W to be positive definite. 00:29:08.890 --> 00:29:17.350 align:middle line:90% 00:29:17.350 --> 00:29:23.390 align:middle line:84% And so the one we had before is just a special case of that. 00:29:23.390 --> 00:29:26.170 align:middle line:84% A diagonal matrix is certainly symmetric. 00:29:26.170 --> 00:29:28.450 align:middle line:84% If it has positive diagonals, it's 00:29:28.450 --> 00:29:30.100 align:middle line:90% certainly positive definite. 00:29:30.100 --> 00:29:33.340 align:middle line:84% But you can have some arbitrary thing that mixes them up. 00:29:33.340 --> 00:29:38.717 align:middle line:84% And these all are useful in surprisingly-- 00:29:38.717 --> 00:29:40.300 align:middle line:84% so there's a surprisingly large number 00:29:40.300 --> 00:29:42.100 align:middle line:84% of cases in which you don't want to have 00:29:42.100 --> 00:29:43.840 align:middle line:84% the traditional Euclidean dot product. 00:29:43.840 --> 00:29:46.995 align:middle line:84% You want to have something more complicated. 00:29:46.995 --> 00:29:48.370 align:middle line:84% And with all of these things, you 00:29:48.370 --> 00:29:52.487 align:middle line:84% could then define gradients differently. 00:29:52.487 --> 00:29:54.070 align:middle line:84% So if you have a weighted dot product, 00:29:54.070 --> 00:29:58.060 align:middle line:84% it's going to change your definition of the gradient 00:29:58.060 --> 00:30:01.187 align:middle line:84% because it's going to be the thing with that W in it. 00:30:01.187 --> 00:30:03.520 align:middle line:84% But most of the time when we're talking about gradients, 00:30:03.520 --> 00:30:06.310 align:middle line:84% we're talking about this is the-- 00:30:06.310 --> 00:30:08.520 align:middle line:84% you're talking about the familiar one. 00:30:08.520 --> 00:30:11.480 align:middle line:90% 00:30:11.480 --> 00:30:14.765 align:middle line:84% The Euclidean product gives you the usual gradient. 00:30:14.765 --> 00:30:20.170 align:middle line:90% 00:30:20.170 --> 00:30:23.510 align:middle line:84% But you might need a weighted dot product, for example, 00:30:23.510 --> 00:30:29.130 align:middle line:84% for grading to make sense if you have a derivative with respect 00:30:29.130 --> 00:30:30.580 align:middle line:84% to a bunch of different components 00:30:30.580 --> 00:30:32.510 align:middle line:90% that have different units. 00:30:32.510 --> 00:30:36.000 align:middle line:84% So let's do another example that's even less familiar, 00:30:36.000 --> 00:30:36.500 align:middle line:90% though. 00:30:36.500 --> 00:30:39.460 align:middle line:90% 00:30:39.460 --> 00:30:48.210 align:middle line:90% Let's say that v is R m by n. 00:30:48.210 --> 00:30:51.220 align:middle line:90% let's do m by n. 00:30:51.220 --> 00:30:53.080 align:middle line:84% So this is going to be m by n matrices. 00:30:53.080 --> 00:30:58.080 align:middle line:90% 00:30:58.080 --> 00:30:59.970 align:middle line:84% Those form a perfectly good vector space. 00:30:59.970 --> 00:31:02.438 align:middle line:84% I can take two matrices, and I can add them. 00:31:02.438 --> 00:31:03.480 align:middle line:90% They have the same shape. 00:31:03.480 --> 00:31:06.030 align:middle line:84% I can multiply them by 2 or some other scalar. 00:31:06.030 --> 00:31:07.950 align:middle line:90% I can subtract. 00:31:07.950 --> 00:31:11.130 align:middle line:84% So this is fine as a vector space. 00:31:11.130 --> 00:31:15.060 align:middle line:84% And in fact, using that vec operator that Alan talked about 00:31:15.060 --> 00:31:17.130 align:middle line:84% in the last lecture, you could convert it 00:31:17.130 --> 00:31:18.990 align:middle line:90% into column vectors-- 00:31:18.990 --> 00:31:21.520 align:middle line:84% you can stack them up into column vectors with m times n 00:31:21.520 --> 00:31:22.020 align:middle line:90% components. 00:31:22.020 --> 00:31:23.670 align:middle line:84% But I want to leave them as matrices. 00:31:23.670 --> 00:31:26.330 align:middle line:90% 00:31:26.330 --> 00:31:28.080 align:middle line:84% But yeah, let me mention that because this 00:31:28.080 --> 00:31:30.060 align:middle line:90% has come up in a second. 00:31:30.060 --> 00:31:38.440 align:middle line:84% So there's an isomorphism with this vec 00:31:38.440 --> 00:31:42.460 align:middle line:90% function to column vectors. 00:31:42.460 --> 00:31:47.800 align:middle line:90% 00:31:47.800 --> 00:32:01.830 align:middle line:84% Vec, vecA that live in R m times n because this is really 00:32:01.830 --> 00:32:03.628 align:middle line:90% an mn dimensional space. 00:32:03.628 --> 00:32:05.670 align:middle line:84% You could just take all the entries of the matrix 00:32:05.670 --> 00:32:08.310 align:middle line:90% and stack them up. 00:32:08.310 --> 00:32:10.830 align:middle line:84% That's what vec does, if you remember from last lecture. 00:32:10.830 --> 00:32:13.095 align:middle line:90% This just stacks the columns. 00:32:13.095 --> 00:32:20.500 align:middle line:90% 00:32:20.500 --> 00:32:25.230 align:middle line:84% And I don't know if people know the term isomorphism. 00:32:25.230 --> 00:32:29.580 align:middle line:84% How many people know what this means? 00:32:29.580 --> 00:32:32.040 align:middle line:84% I don't know if people are raising their hands or not. 00:32:32.040 --> 00:32:34.350 align:middle line:84% Alex, are people raising their hands. 00:32:34.350 --> 00:32:35.460 align:middle line:90% AUDIENCE: Five people. 00:32:35.460 --> 00:32:36.335 align:middle line:90% STEVEN JOHNSON: Yeah. 00:32:36.335 --> 00:32:39.150 align:middle line:84% So it basically means that if you add two matrices, 00:32:39.150 --> 00:32:41.550 align:middle line:84% it's equivalent to adding these two vectors because-- 00:32:41.550 --> 00:32:42.672 align:middle line:90% or multiply by 2. 00:32:42.672 --> 00:32:44.130 align:middle line:84% It's multiplying these vectors by 2 00:32:44.130 --> 00:32:46.680 align:middle line:84% because these vectors are just the same component 00:32:46.680 --> 00:32:48.150 align:middle line:90% entries of A rearranged. 00:32:48.150 --> 00:32:50.618 align:middle line:84% So adding matrices is adding the entries. 00:32:50.618 --> 00:32:52.410 align:middle line:84% Adding these vectors is adding the entries. 00:32:52.410 --> 00:32:54.660 align:middle line:84% Multiplying by 2 is multiplying the entries by 2. 00:32:54.660 --> 00:32:57.670 align:middle line:84% Multiplying this vector by 2 is multiplying the entries by 2. 00:32:57.670 --> 00:32:59.400 align:middle line:90% So it's the same thing. 00:32:59.400 --> 00:33:04.305 align:middle line:84% But anyways, and sometimes it's useful. 00:33:04.305 --> 00:33:06.900 align:middle line:90% 00:33:06.900 --> 00:33:10.800 align:middle line:84% It's nice to stick with-- if your problem is expressed 00:33:10.800 --> 00:33:12.300 align:middle line:84% in terms of matrices, usually you 00:33:12.300 --> 00:33:14.730 align:middle line:84% want to keep in terms of matrices if possible. 00:33:14.730 --> 00:33:16.950 align:middle line:84% That's the natural way of looking 00:33:16.950 --> 00:33:19.440 align:middle line:84% at your degree of freedom is an n by n matrix, 00:33:19.440 --> 00:33:23.020 align:middle line:84% whereas stacking it up into a column vector of mn components 00:33:23.020 --> 00:33:25.120 align:middle line:90% is kind of an unnatural way. 00:33:25.120 --> 00:33:27.340 align:middle line:84% Internally, the computer does that. 00:33:27.340 --> 00:33:30.820 align:middle line:84% But sometimes it helps you disentangle 00:33:30.820 --> 00:33:32.470 align:middle line:84% what's going on if you stack things 00:33:32.470 --> 00:33:34.360 align:middle line:90% up into a more familiar vector. 00:33:34.360 --> 00:33:37.090 align:middle line:90% 00:33:37.090 --> 00:33:40.035 align:middle line:84% What we'd like to do is define a dot product of matrices. 00:33:40.035 --> 00:33:41.410 align:middle line:84% And again, just like for vectors, 00:33:41.410 --> 00:33:43.240 align:middle line:90% there'll be multiple choices. 00:33:43.240 --> 00:33:46.150 align:middle line:84% There's never one dot product for a vector space. 00:33:46.150 --> 00:33:48.070 align:middle line:84% There's multiple possible choices. 00:33:48.070 --> 00:33:49.060 align:middle line:90% And you choose one. 00:33:49.060 --> 00:33:51.310 align:middle line:84% And then everything else follows from that. 00:33:51.310 --> 00:33:53.060 align:middle line:90% The geometry follows. 00:33:53.060 --> 00:34:07.770 align:middle line:84% So I just want to show you the most obvious one, 00:34:07.770 --> 00:34:16.504 align:middle line:84% the analog of the familiar Euclidean inner product. 00:34:16.504 --> 00:34:20.659 align:middle line:90% 00:34:20.659 --> 00:34:23.340 align:middle line:90% So what would that be? 00:34:23.340 --> 00:34:30.500 align:middle line:84% So if I have two matrices, A and B, where this is m by n, 00:34:30.500 --> 00:34:33.862 align:middle line:84% and this is m by n, And this is our dot product. 00:34:33.862 --> 00:34:35.570 align:middle line:84% This is not this is not a matrix product. 00:34:35.570 --> 00:34:37.129 align:middle line:90% This is our inner product. 00:34:37.129 --> 00:34:42.070 align:middle line:90% 00:34:42.070 --> 00:34:43.990 align:middle line:84% So we want to give this to be a scalar. 00:34:43.990 --> 00:34:53.580 align:middle line:90% 00:34:53.580 --> 00:34:54.449 align:middle line:90% what should I do? 00:34:54.449 --> 00:34:55.620 align:middle line:90% Any suggestions? 00:34:55.620 --> 00:34:58.530 align:middle line:90% 00:34:58.530 --> 00:35:00.750 align:middle line:90% Alex, you'll need to repeat. 00:35:00.750 --> 00:35:02.695 align:middle line:90% I can't-- 00:35:02.695 --> 00:35:03.320 align:middle line:90% AUDIENCE: Yeah. 00:35:03.320 --> 00:35:10.280 align:middle line:84% This one, multiply by all the components, like elementwise. 00:35:10.280 --> 00:35:13.580 align:middle line:84% STEVEN JOHNSON: Exactly, just multiply 00:35:13.580 --> 00:35:18.620 align:middle line:84% the elements, the entries of these matrices elementwise, 00:35:18.620 --> 00:35:19.890 align:middle line:90% and add them up. 00:35:19.890 --> 00:35:24.230 align:middle line:84% So this is the sum of the elementwise products. 00:35:24.230 --> 00:35:30.610 align:middle line:90% 00:35:30.610 --> 00:35:32.975 align:middle line:84% So that's it's a little bit annoying 00:35:32.975 --> 00:35:34.100 align:middle line:90% to write that out in words. 00:35:34.100 --> 00:35:34.960 align:middle line:84% So let's see if we can figure out 00:35:34.960 --> 00:35:37.150 align:middle line:84% a better way to algebraically to write that down. 00:35:37.150 --> 00:35:39.670 align:middle line:84% So in Julia, you could write this down. 00:35:39.670 --> 00:35:42.760 align:middle line:90% 00:35:42.760 --> 00:36:00.850 align:middle line:84% The one way to do this is to say sum of A dot star B. Dot star 00:36:00.850 --> 00:36:04.150 align:middle line:84% is an elementwise product of two vectors and matrices. 00:36:04.150 --> 00:36:07.000 align:middle line:84% And sum just sums then sums all the entries. 00:36:07.000 --> 00:36:07.930 align:middle line:90% That's what this is. 00:36:07.930 --> 00:36:12.520 align:middle line:90% 00:36:12.520 --> 00:36:22.075 align:middle line:84% I could write it as sum over i and j of Aij Bij. 00:36:22.075 --> 00:36:27.360 align:middle line:90% 00:36:27.360 --> 00:36:30.150 align:middle line:90% I could write it with vex. 00:36:30.150 --> 00:36:35.040 align:middle line:84% It's the same thing as if I take the elements of A 00:36:35.040 --> 00:36:37.890 align:middle line:84% and I stack them up into a vector, 00:36:37.890 --> 00:36:41.925 align:middle line:84% and I take the elements of B and I stack them up into a vector. 00:36:41.925 --> 00:36:47.820 align:middle line:90% 00:36:47.820 --> 00:36:52.770 align:middle line:84% This is just the ordinary dot product of those. 00:36:52.770 --> 00:36:54.280 align:middle line:90% VecA is just the same numbers. 00:36:54.280 --> 00:36:55.270 align:middle line:90% It's just stacked up. 00:36:55.270 --> 00:36:58.080 align:middle line:84% So if I take the ordinary dot product of vecA and vecB, 00:36:58.080 --> 00:36:59.910 align:middle line:84% that is this elementwise product. 00:36:59.910 --> 00:37:01.890 align:middle line:84% But again, that that's a little bit frustrating 00:37:01.890 --> 00:37:05.670 align:middle line:84% because if someone hands me this vector space, 00:37:05.670 --> 00:37:07.440 align:middle line:84% it's probably because they really want 00:37:07.440 --> 00:37:09.240 align:middle line:90% to leave things as matrices. 00:37:09.240 --> 00:37:12.340 align:middle line:84% That's the natural language of the problem. 00:37:12.340 --> 00:37:15.600 align:middle line:84% So I'd rather use this-- this vec notation is 00:37:15.600 --> 00:37:17.280 align:middle line:90% kind of a last resort. 00:37:17.280 --> 00:37:20.190 align:middle line:84% If you can't figure out how to deal with it as a matrix, 00:37:20.190 --> 00:37:25.920 align:middle line:84% you cram it into a vector and then go back to what you know. 00:37:25.920 --> 00:37:30.660 align:middle line:84% So it turns out that there's a really nice way 00:37:30.660 --> 00:37:33.780 align:middle line:84% to express this in linear algebra, 00:37:33.780 --> 00:37:35.370 align:middle line:90% using linear algebra operations. 00:37:35.370 --> 00:37:45.570 align:middle line:84% And that turns out to be the trace of A transpose B. 00:37:45.570 --> 00:37:48.940 align:middle line:84% I don't know if you want to if you want to see that derived. 00:37:48.940 --> 00:37:53.690 align:middle line:84% It's actually pretty straightforward to derive this. 00:37:53.690 --> 00:37:59.120 align:middle line:84% So this is also this is this is also called the Frobenius 00:37:59.120 --> 00:38:12.680 align:middle line:90% inner product of two matrices. 00:38:12.680 --> 00:38:15.430 align:middle line:84% So if you just write out the formula for trace and matrix 00:38:15.430 --> 00:38:18.490 align:middle line:84% multiplication, you can actually see that. 00:38:18.490 --> 00:38:20.650 align:middle line:84% Another way of seeing that is basically, 00:38:20.650 --> 00:38:21.970 align:middle line:90% what is A transpose B? 00:38:21.970 --> 00:38:26.560 align:middle line:84% So A times B is take rows of a times dot product 00:38:26.560 --> 00:38:33.010 align:middle line:84% with columns of B. But if you do A transpose of B, A transpose, 00:38:33.010 --> 00:38:38.110 align:middle line:84% its rows are the columns of A. So what this really is, 00:38:38.110 --> 00:38:41.380 align:middle line:84% rows times columns here, is you're 00:38:41.380 --> 00:38:43.570 align:middle line:84% taking the dot product of each column of A 00:38:43.570 --> 00:38:47.160 align:middle line:90% with each column of B. 00:38:47.160 --> 00:38:49.120 align:middle line:84% And the trace is the sum of the diagonals. 00:38:49.120 --> 00:38:52.640 align:middle line:84% So what's the first diagonal entry of this? 00:38:52.640 --> 00:38:57.970 align:middle line:84% The first diagonal entry is column 1 of A dot column 00:38:57.970 --> 00:39:03.170 align:middle line:84% 1 of B. The second diagonal entry is column 2 of A dot 00:39:03.170 --> 00:39:06.430 align:middle line:90% column 2 of B and so forth. 00:39:06.430 --> 00:39:10.170 align:middle line:84% And if you add those up, you're adding up all the dot product 00:39:10.170 --> 00:39:12.960 align:middle line:84% of each column of A with the corresponding column 00:39:12.960 --> 00:39:14.770 align:middle line:90% of B. That's exactly this. 00:39:14.770 --> 00:39:18.600 align:middle line:84% It's the sum of the product of all the entries. 00:39:18.600 --> 00:39:22.950 align:middle line:84% So this is a really useful thing to know about, 00:39:22.950 --> 00:39:24.960 align:middle line:84% that you can take a dot product of two matrices. 00:39:24.960 --> 00:39:26.460 align:middle line:84% The obvious inner product is really 00:39:26.460 --> 00:39:30.510 align:middle line:84% the trace of A transpose B. And this then 00:39:30.510 --> 00:39:39.290 align:middle line:84% gives you the Frobenius norm, which I mentioned 00:39:39.290 --> 00:39:42.740 align:middle line:90% the other day, of matrices. 00:39:42.740 --> 00:39:49.400 align:middle line:84% So the norm of a matrix of anything, 00:39:49.400 --> 00:39:54.125 align:middle line:84% we want it to be the square root of the dot product with itself. 00:39:54.125 --> 00:39:57.430 align:middle line:90% 00:39:57.430 --> 00:40:03.420 align:middle line:84% And what this is is exactly the square root 00:40:03.420 --> 00:40:13.080 align:middle line:90% of the trace of A transpose A. 00:40:13.080 --> 00:40:14.260 align:middle line:90% So it's really useful. 00:40:14.260 --> 00:40:19.320 align:middle line:84% It's kind of the obvious Euclidean norm of this. 00:40:19.320 --> 00:40:22.400 align:middle line:84% So this is exactly the same thing 00:40:22.400 --> 00:40:34.320 align:middle line:84% as the Euclidean norm of all the components, 00:40:34.320 --> 00:40:39.340 align:middle line:84% if we take and vecA and take it, to just stack up 00:40:39.340 --> 00:40:42.910 align:middle line:84% the components, it's that, or equivalently, it's 00:40:42.910 --> 00:40:46.030 align:middle line:90% the square root of the sum of-- 00:40:46.030 --> 00:40:51.867 align:middle line:84% if you just take all the entries of A and just 00:40:51.867 --> 00:40:53.825 align:middle line:84% take the square root of the sum of the squares. 00:40:53.825 --> 00:41:01.530 align:middle line:90% 00:41:01.530 --> 00:41:07.800 align:middle line:84% This is the most obvious way to measure the size of a matrix. 00:41:07.800 --> 00:41:15.570 align:middle line:84% And so now, using this, we can now 00:41:15.570 --> 00:41:34.765 align:middle line:84% take the gradient of scalar functions of matrix inputs. 00:41:34.765 --> 00:41:44.200 align:middle line:90% 00:41:44.200 --> 00:41:48.420 align:middle line:84% So for example, let's start with the-- 00:41:48.420 --> 00:41:49.690 align:middle line:90% let's do some examples. 00:41:49.690 --> 00:41:53.520 align:middle line:84% Let's take some derivatives of matrix of scalar functions 00:41:53.520 --> 00:41:55.680 align:middle line:90% of matrices. 00:41:55.680 --> 00:42:03.220 align:middle line:84% So let's do f of A, where this is 00:42:03.220 --> 00:42:06.760 align:middle line:90% going to be an m by n matrix. 00:42:06.760 --> 00:42:08.740 align:middle line:90% What's a scalar function? 00:42:08.740 --> 00:42:10.630 align:middle line:84% The simplest one is maybe the one 00:42:10.630 --> 00:42:12.205 align:middle line:90% we just learned a second ago. 00:42:12.205 --> 00:42:17.710 align:middle line:90% 00:42:17.710 --> 00:42:20.390 align:middle line:90% Let's do that. 00:42:20.390 --> 00:42:23.000 align:middle line:90% Let's take that Frobenius norm. 00:42:23.000 --> 00:42:25.340 align:middle line:90% This is the sometimes denoted-- 00:42:25.340 --> 00:42:29.060 align:middle line:90% sometimes people put an F here. 00:42:29.060 --> 00:42:32.910 align:middle line:84% I'm just going to leave out the F, but also-- 00:42:32.910 --> 00:42:35.010 align:middle line:84% let me just note that in because there 00:42:35.010 --> 00:42:38.520 align:middle line:84% are multiple ways of denoting the norm of matrix, 00:42:38.520 --> 00:42:42.320 align:middle line:84% just like there are multiple inner products. 00:42:42.320 --> 00:42:43.970 align:middle line:84% So when people want to be specific, 00:42:43.970 --> 00:42:46.190 align:middle line:90% they usually put an F there. 00:42:46.190 --> 00:42:48.020 align:middle line:84% But I'm just going to-- well, I guess 00:42:48.020 --> 00:42:52.100 align:middle line:84% I can leave this here right, just so we know what it is. 00:42:52.100 --> 00:42:55.170 align:middle line:90% 00:42:55.170 --> 00:43:10.836 align:middle line:84% So this is the square root of the trace of A transpose A. 00:43:10.836 --> 00:43:14.210 align:middle line:84% And now, I want to compute a derivative of this. 00:43:14.210 --> 00:43:16.520 align:middle line:84% So first, before I worry about the gradient, 00:43:16.520 --> 00:43:18.980 align:middle line:84% let's just make sure we can compute a derivative. 00:43:18.980 --> 00:43:21.110 align:middle line:84% We better be able to take what is df 00:43:21.110 --> 00:43:25.400 align:middle line:84% and use linear algebra, linear algebra. 00:43:25.400 --> 00:43:30.860 align:middle line:90% So how do we get df? 00:43:30.860 --> 00:43:33.320 align:middle line:90% So I need to use the chain rule. 00:43:33.320 --> 00:43:40.330 align:middle line:90% 00:43:40.330 --> 00:43:42.760 align:middle line:84% I want to use the chain rule, just to simplify life, 00:43:42.760 --> 00:43:45.650 align:middle line:84% so we don't have to rederive everything from scratch. 00:43:45.650 --> 00:43:48.163 align:middle line:84% And so first of all, we're applying the chain rule 00:43:48.163 --> 00:43:49.330 align:middle line:90% to the square root function. 00:43:49.330 --> 00:43:52.570 align:middle line:84% But it's the square root of a number. 00:43:52.570 --> 00:43:54.740 align:middle line:84% Once you take the trace, this is just a number. 00:43:54.740 --> 00:43:56.680 align:middle line:84% And it's the square root of a number. 00:43:56.680 --> 00:44:00.400 align:middle line:84% So we can use the ordinary-- once you have scalar functions, 00:44:00.400 --> 00:44:06.307 align:middle line:84% you can use all your ordinary 1801 first year calculus rules. 00:44:06.307 --> 00:44:08.140 align:middle line:84% What we're doing is not different from that. 00:44:08.140 --> 00:44:11.900 align:middle line:84% It's just a generalization of that. 00:44:11.900 --> 00:44:17.950 align:middle line:84% So you can really just use the 1801 chain rule to start with. 00:44:17.950 --> 00:44:26.440 align:middle line:84% So the derivative of a square root is 1 over 2, 00:44:26.440 --> 00:44:29.560 align:middle line:84% the square root of that thing, of trace 00:44:29.560 --> 00:44:40.010 align:middle line:84% of A transpose A times d of what you took the square root of, 00:44:40.010 --> 00:44:49.415 align:middle line:84% of trace of A transpose A. This is just the 1801 chain rule. 00:44:49.415 --> 00:44:51.990 align:middle line:84% So we don't want to throw away everything we learned. 00:44:51.990 --> 00:44:56.910 align:middle line:84% So once we hit scalars, we can use our familiar rules. 00:44:56.910 --> 00:44:59.200 align:middle line:84% What we're doing is exactly equivalent to that right. 00:44:59.200 --> 00:45:05.325 align:middle line:84% So this is just the ordinary chain rule for square roots. 00:45:05.325 --> 00:45:08.460 align:middle line:84% It's the derivative, or the d in this-- the differential, 00:45:08.460 --> 00:45:11.070 align:middle line:84% in this case, of whatever is inside the square root 00:45:11.070 --> 00:45:15.375 align:middle line:84% times 1/2 that thing to the minus 1/2 power. 00:45:15.375 --> 00:45:17.960 align:middle line:90% 00:45:17.960 --> 00:45:21.430 align:middle line:84% So let's see if we can go further. 00:45:21.430 --> 00:45:29.910 align:middle line:90% So now, so I want to do a-- 00:45:29.910 --> 00:45:38.850 align:middle line:84% so what's the d of the trace, of the trace 00:45:38.850 --> 00:45:43.740 align:middle line:84% of a function of, say, B. So this 00:45:43.740 --> 00:45:57.460 align:middle line:84% is trace of B plus dB minus trace of B. 00:45:57.460 --> 00:45:59.990 align:middle line:90% But the trace is linear. 00:45:59.990 --> 00:46:00.830 align:middle line:90% This is the same. 00:46:00.830 --> 00:46:03.710 align:middle line:90% The trace of B plus-- 00:46:03.710 --> 00:46:06.590 align:middle line:84% the trace of the sum of two matrices 00:46:06.590 --> 00:46:09.020 align:middle line:84% is the same thing as the sum of the traces 00:46:09.020 --> 00:46:12.980 align:middle line:84% because if you think about what a trace means, 00:46:12.980 --> 00:46:14.730 align:middle line:84% it's just the sum of the diagonal entries. 00:46:14.730 --> 00:46:17.510 align:middle line:84% So if I add up two matrices, their diagonal entries 00:46:17.510 --> 00:46:18.530 align:middle line:90% just adds. 00:46:18.530 --> 00:46:20.030 align:middle line:84% That means their trace is just add. 00:46:20.030 --> 00:46:24.730 align:middle line:90% 00:46:24.730 --> 00:46:26.230 align:middle line:90% So this is just trace dB. 00:46:26.230 --> 00:46:31.670 align:middle line:84% So this is very closely related to problem 1 on the homework. 00:46:31.670 --> 00:46:33.650 align:middle line:90% So this thing is-- 00:46:33.650 --> 00:46:34.350 align:middle line:90% what do we have? 00:46:34.350 --> 00:46:36.590 align:middle line:90% We have 1 over 2. 00:46:36.590 --> 00:46:40.250 align:middle line:84% That square root is really just our norm of A in the front, 00:46:40.250 --> 00:46:44.050 align:middle line:84% so I'll just simplify that there. 00:46:44.050 --> 00:46:47.140 align:middle line:90% And then we have the trace. 00:46:47.140 --> 00:46:48.970 align:middle line:84% The d of the trace is the same thing 00:46:48.970 --> 00:47:01.550 align:middle line:84% as the trace of d of A transpose A. 00:47:01.550 --> 00:47:04.400 align:middle line:84% And now, we can use the product rule. 00:47:04.400 --> 00:47:10.630 align:middle line:90% 00:47:10.630 --> 00:47:22.620 align:middle line:84% So this is 1 over 2 norm of A with a trace of-- 00:47:22.620 --> 00:47:23.890 align:middle line:90% and what do we have? 00:47:23.890 --> 00:47:29.820 align:middle line:84% We have dA-- that needs to be written-- 00:47:29.820 --> 00:47:39.185 align:middle line:84% dA transpose A plus A transpose dA. 00:47:39.185 --> 00:47:45.790 align:middle line:90% 00:47:45.790 --> 00:47:47.980 align:middle line:84% And now, I want to simplify this if I can, 00:47:47.980 --> 00:47:51.620 align:middle line:84% to try and move all the dA's to the right. 00:47:51.620 --> 00:47:54.670 align:middle line:90% So any questions so far? 00:47:54.670 --> 00:47:56.880 align:middle line:84% Again, Alex, if you have any questions, 00:47:56.880 --> 00:48:00.972 align:middle line:84% you have to repeat because I can't hear anyone in the room. 00:48:00.972 --> 00:48:01.940 align:middle line:90% AUDIENCE: No questions. 00:48:01.940 --> 00:48:04.110 align:middle line:90% STEVEN JOHNSON: Yeah? 00:48:04.110 --> 00:48:05.250 align:middle line:90% AUDIENCE: No questions. 00:48:05.250 --> 00:48:06.667 align:middle line:84% STEVEN JOHNSON: No questions, yes. 00:48:06.667 --> 00:48:09.010 align:middle line:90% 00:48:09.010 --> 00:48:17.170 align:middle line:90% So again, I want to move-- 00:48:17.170 --> 00:48:19.610 align:middle line:84% again, we're going to use linearity. 00:48:19.610 --> 00:48:25.750 align:middle line:84% So this whole thing is equal to the trace 00:48:25.750 --> 00:48:36.070 align:middle line:84% of dA transpose A plus the trace of A transpose dA. 00:48:36.070 --> 00:48:38.993 align:middle line:90% 00:48:38.993 --> 00:48:41.160 align:middle line:84% I'm going to try and make it look like a dot product 00:48:41.160 --> 00:48:44.190 align:middle line:84% with A, so I want to group my terms with dA 00:48:44.190 --> 00:48:47.350 align:middle line:84% so all the terms, the dA terms, are going to be on the right. 00:48:47.350 --> 00:48:52.630 align:middle line:84% So one of the things I can use is-- 00:48:52.630 --> 00:49:03.100 align:middle line:84% so dA transpose A is not the same thing as A transpose dA. 00:49:03.100 --> 00:49:06.990 align:middle line:90% But its trace is. 00:49:06.990 --> 00:49:09.690 align:middle line:84% So note, over on the right here, I'm 00:49:09.690 --> 00:49:14.460 align:middle line:84% just keeping a running note of my properties. 00:49:14.460 --> 00:49:18.680 align:middle line:90% That was my note one. 00:49:18.680 --> 00:49:21.770 align:middle line:84% Note two, if I have the trace of B, 00:49:21.770 --> 00:49:26.750 align:middle line:84% that's the same thing as the trace of B transpose. 00:49:26.750 --> 00:49:28.880 align:middle line:84% If I transpose a matrix, it doesn't 00:49:28.880 --> 00:49:30.860 align:middle line:90% change the diagonal entries. 00:49:30.860 --> 00:49:34.070 align:middle line:84% So this thing here is the same thing 00:49:34.070 --> 00:49:36.560 align:middle line:90% as the trace of A transpose dA. 00:49:36.560 --> 00:49:40.380 align:middle line:90% 00:49:40.380 --> 00:49:42.180 align:middle line:90% The matrix inside is different. 00:49:42.180 --> 00:49:43.830 align:middle line:90% But its trace is the same. 00:49:43.830 --> 00:49:46.110 align:middle line:84% B is not equal to B transpose, but trace B 00:49:46.110 --> 00:49:48.850 align:middle line:90% equals trace of B transpose. 00:49:48.850 --> 00:49:52.330 align:middle line:84% So that, of course, is the same as this. 00:49:52.330 --> 00:49:54.030 align:middle line:90% So I just have two of these. 00:49:54.030 --> 00:49:57.380 align:middle line:90% It cancels the two out there. 00:49:57.380 --> 00:50:06.070 align:middle line:84% And what you're left with is 1 over the norm of A times 00:50:06.070 --> 00:50:12.160 align:middle line:90% the trace of A transpose dA. 00:50:12.160 --> 00:50:17.380 align:middle line:90% 00:50:17.380 --> 00:50:20.270 align:middle line:84% So this is quite a simple formula. 00:50:20.270 --> 00:50:22.900 align:middle line:90% The 2's cancel. 00:50:22.900 --> 00:50:26.045 align:middle line:90% And so what's that? 00:50:26.045 --> 00:50:29.030 align:middle line:84% That, I claim, is the dot product-- 00:50:29.030 --> 00:50:31.860 align:middle line:84% if we just go back to our formula, 00:50:31.860 --> 00:50:37.855 align:middle line:84% trace of A transpose B is a dot product of A and B. 00:50:37.855 --> 00:50:40.790 align:middle line:84% And so this trace here is exactly 00:50:40.790 --> 00:50:51.790 align:middle line:90% the dot product of A with dA. 00:50:51.790 --> 00:50:56.030 align:middle line:90% 00:50:56.030 --> 00:50:57.480 align:middle line:90% But then have a scale factor. 00:50:57.480 --> 00:51:00.745 align:middle line:84% But dot products are linear, so I can bring that inside. 00:51:00.745 --> 00:51:02.120 align:middle line:84% Let me move this up a little bit. 00:51:02.120 --> 00:51:09.372 align:middle line:90% 00:51:09.372 --> 00:51:10.330 align:middle line:90% Does everyone see this? 00:51:10.330 --> 00:51:12.800 align:middle line:84% So this is, according to my definition of the dot product, 00:51:12.800 --> 00:51:16.640 align:middle line:84% this trace here is exactly the dot product of A over norm A, 00:51:16.640 --> 00:51:19.940 align:middle line:84% like the unit vector in the direction of, in some sense, 00:51:19.940 --> 00:51:20.560 align:middle line:90% with dA. 00:51:20.560 --> 00:51:23.862 align:middle line:90% 00:51:23.862 --> 00:51:35.315 align:middle line:84% And so this term here is exactly my gradient. 00:51:35.315 --> 00:51:39.840 align:middle line:90% 00:51:39.840 --> 00:51:50.460 align:middle line:84% This is my nabla gradient of norm of A 00:51:50.460 --> 00:52:02.945 align:middle line:90% is exactly A over norm of A. 00:52:02.945 --> 00:52:05.070 align:middle line:84% So I've just taken the gradient of a normal matrix. 00:52:05.070 --> 00:52:06.320 align:middle line:90% And if you think of a vector-- 00:52:06.320 --> 00:52:09.200 align:middle line:90% 00:52:09.200 --> 00:52:11.480 align:middle line:84% actually, Professor Edelman derived 00:52:11.480 --> 00:52:15.470 align:middle line:84% this for if you took the gradient of norm of x, 00:52:15.470 --> 00:52:17.960 align:middle line:90% that was x over norm of x. 00:52:17.960 --> 00:52:20.180 align:middle line:90% You derived that last time. 00:52:20.180 --> 00:52:23.820 align:middle line:84% This is actually literally the same formula, 00:52:23.820 --> 00:52:28.340 align:middle line:84% which makes sense, because this Frobenius norm is really, 00:52:28.340 --> 00:52:30.767 align:middle line:84% in spirit, it's the same thing as your familiar norm 00:52:30.767 --> 00:52:32.600 align:middle line:84% of a vector, just the square root of the sum 00:52:32.600 --> 00:52:33.900 align:middle line:90% of the squared components. 00:52:33.900 --> 00:52:38.240 align:middle line:84% So when you take the gradient, if you 00:52:38.240 --> 00:52:39.890 align:middle line:84% think of the gradient as the derivative 00:52:39.890 --> 00:52:43.505 align:middle line:84% of this vector, each component of A, 00:52:43.505 --> 00:52:46.440 align:middle line:90% you get exactly the same thing. 00:52:46.440 --> 00:52:48.530 align:middle line:84% And in fact, I should mention that. 00:52:48.530 --> 00:52:58.180 align:middle line:84% So in fact, in 18.02 terms, if you 00:52:58.180 --> 00:53:01.000 align:middle line:90% have a function, the gradient-- 00:53:01.000 --> 00:53:21.610 align:middle line:84% if you have a scalar function of a matrix for a scalar 00:53:21.610 --> 00:53:28.640 align:middle line:84% f of a matrix, this is exactly going 00:53:28.640 --> 00:53:38.670 align:middle line:84% to be the matrix of partial f partial A. 00:53:38.670 --> 00:53:40.670 align:middle line:84% GUEST SPEAKER: Steven, just giving a quick hello 00:53:40.670 --> 00:53:42.290 align:middle line:90% to you and the class. 00:53:42.290 --> 00:53:42.710 align:middle line:90% STEVEN JOHNSON: You're here. 00:53:42.710 --> 00:53:43.040 align:middle line:90% Great. 00:53:43.040 --> 00:53:44.915 align:middle line:84% GUEST SPEAKER: Yeah, I just finished my talk. 00:53:44.915 --> 00:53:47.870 align:middle line:84% I was just giving a talk on Julia and high performance 00:53:47.870 --> 00:53:49.010 align:middle line:90% computing. 00:53:49.010 --> 00:53:50.450 align:middle line:90% It was a Dell talk. 00:53:50.450 --> 00:53:52.220 align:middle line:90% So anyway, yeah, I'm here. 00:53:52.220 --> 00:53:54.860 align:middle line:84% But let me not let me not interrupt you 00:53:54.860 --> 00:53:56.370 align:middle line:90% any more than I just have. 00:53:56.370 --> 00:53:57.620 align:middle line:90% Is the technology all working? 00:53:57.620 --> 00:53:58.580 align:middle line:90% Maybe I'll just ask that. 00:53:58.580 --> 00:53:58.970 align:middle line:90% STEVEN JOHNSON: Yep. 00:53:58.970 --> 00:53:59.803 align:middle line:90% GUEST SPEAKER: Good. 00:53:59.803 --> 00:54:02.480 align:middle line:90% 00:54:02.480 --> 00:54:04.550 align:middle line:84% STEVEN JOHNSON: So it's exactly just the matrix 00:54:04.550 --> 00:54:05.780 align:middle line:90% of partial derivatives. 00:54:05.780 --> 00:54:08.358 align:middle line:84% So if we wanted it to do it elementwise, we could. 00:54:08.358 --> 00:54:10.400 align:middle line:84% We could take the partial derivative with respect 00:54:10.400 --> 00:54:13.250 align:middle line:84% to A11, partial derivatives with respect 00:54:13.250 --> 00:54:16.160 align:middle line:84% to A12, partial derivative with respect 00:54:16.160 --> 00:54:22.490 align:middle line:84% to A21, partial derivative of A22 and so forth. 00:54:22.490 --> 00:54:25.130 align:middle line:84% Of course, this is really becomes really awkward. 00:54:25.130 --> 00:54:29.100 align:middle line:90% 00:54:29.100 --> 00:54:31.530 align:middle line:84% But this is for matrix value function. 00:54:31.530 --> 00:54:33.810 align:middle line:84% But this is exactly equivalent to saying 00:54:33.810 --> 00:54:44.410 align:middle line:84% that df is this inner product, grad f dot dA, 00:54:44.410 --> 00:54:56.790 align:middle line:84% which is equal to the trace of grad f transposed dA 00:54:56.790 --> 00:54:59.430 align:middle line:84% because our inner product really is the elementwise product 00:54:59.430 --> 00:55:01.890 align:middle line:90% of the components. 00:55:01.890 --> 00:55:07.540 align:middle line:84% And so this gradient is what you expect a matrix gradient to be. 00:55:07.540 --> 00:55:09.540 align:middle line:84% But as I said, it's just really annoying 00:55:09.540 --> 00:55:11.650 align:middle line:90% to work with in this form. 00:55:11.650 --> 00:55:15.870 align:middle line:90% So maybe I'll do one more. 00:55:15.870 --> 00:55:22.140 align:middle line:84% So maybe I'll mention next time, or soon, maybe not next time. 00:55:22.140 --> 00:55:22.930 align:middle line:90% I don't know. 00:55:22.930 --> 00:55:31.990 align:middle line:84% Soon, we'll also do the gradient of the determinant of A 00:55:31.990 --> 00:55:36.830 align:middle line:90% for an m by m matrix. 00:55:36.830 --> 00:55:41.175 align:middle line:84% And it turns out this is not so easy to derive. 00:55:41.175 --> 00:55:42.550 align:middle line:84% And you definitely would not want 00:55:42.550 --> 00:55:44.440 align:middle line:84% to do this component by component. 00:55:44.440 --> 00:55:46.760 align:middle line:84% That's a really awkward way to do this. 00:55:46.760 --> 00:55:56.960 align:middle line:84% It turns out this is exactly equal to determinant of A times 00:55:56.960 --> 00:55:59.830 align:middle line:90% A inverse transposed. 00:55:59.830 --> 00:56:04.500 align:middle line:90% 00:56:04.500 --> 00:56:06.345 align:middle line:90% And let me do a simpler example. 00:56:06.345 --> 00:56:11.970 align:middle line:90% 00:56:11.970 --> 00:56:14.790 align:middle line:84% It also illustrates a nice property of a trace. 00:56:14.790 --> 00:56:19.360 align:middle line:84% Let's do f-- and then we'll take a break. 00:56:19.360 --> 00:56:29.400 align:middle line:84% So let's do f of A is x transpose A y. 00:56:29.400 --> 00:56:33.810 align:middle line:90% So this A is going to be m by n. 00:56:33.810 --> 00:56:45.100 align:middle line:84% And this is for some constant x in R something and y 00:56:45.100 --> 00:56:46.810 align:middle line:90% in R something. 00:56:46.810 --> 00:56:49.480 align:middle line:84% So how many components does x have? 00:56:49.480 --> 00:56:51.950 align:middle line:90% 00:56:51.950 --> 00:56:53.575 align:middle line:84% Alex, are people shouting out anything. 00:56:53.575 --> 00:56:59.400 align:middle line:90% 00:56:59.400 --> 00:57:00.838 align:middle line:90% I can't hear anything. 00:57:00.838 --> 00:57:01.380 align:middle line:90% AUDIENCE: No. 00:57:01.380 --> 00:57:02.255 align:middle line:90% No one said anything. 00:57:02.255 --> 00:57:05.460 align:middle line:90% 00:57:05.460 --> 00:57:08.330 align:middle line:84% STEVEN JOHNSON: This, I wanted to give a number. 00:57:08.330 --> 00:57:10.290 align:middle line:90% Just think in the shapes. 00:57:10.290 --> 00:57:11.840 align:middle line:90% This is n by n. 00:57:11.840 --> 00:57:14.120 align:middle line:84% Just for this product to even makes sense, 00:57:14.120 --> 00:57:17.452 align:middle line:84% how many components does x have to have? 00:57:17.452 --> 00:57:19.180 align:middle line:90% AUDIENCE: M? 00:57:19.180 --> 00:57:24.230 align:middle line:84% STEVEN JOHNSON: M. And how many components does y have? 00:57:24.230 --> 00:57:24.730 align:middle line:90% AUDIENCE: N. 00:57:24.730 --> 00:57:25.688 align:middle line:90% STEVEN JOHNSON: N, yes. 00:57:25.688 --> 00:57:28.300 align:middle line:90% 00:57:28.300 --> 00:57:30.850 align:middle line:84% So if we just take a matrix and sandwich it 00:57:30.850 --> 00:57:32.530 align:middle line:84% in between two vectors of the right-- 00:57:32.530 --> 00:57:34.280 align:middle line:84% but they have to match the number of rows, 00:57:34.280 --> 00:57:37.610 align:middle line:84% the number of columns, then you get a scalar out. 00:57:37.610 --> 00:57:38.920 align:middle line:90% And so then what's df? 00:57:38.920 --> 00:57:42.370 align:middle line:90% This one is even easier. 00:57:42.370 --> 00:57:45.750 align:middle line:90% So df is just x transpose. 00:57:45.750 --> 00:57:46.960 align:middle line:90% Everything is a constant. 00:57:46.960 --> 00:57:49.835 align:middle line:90% So I'm just going to use the-- 00:57:49.835 --> 00:57:51.150 align:middle line:90% you can go to the product rule. 00:57:51.150 --> 00:57:54.030 align:middle line:90% But there's only one. 00:57:54.030 --> 00:57:56.220 align:middle line:84% So technically, this is the product rule. 00:57:56.220 --> 00:57:59.918 align:middle line:90% But dx is 0. dy is 0. 00:57:59.918 --> 00:58:00.710 align:middle line:90% Those are constant. 00:58:00.710 --> 00:58:03.860 align:middle line:90% So this is just this. 00:58:03.860 --> 00:58:05.690 align:middle line:84% But this is a function that takes 00:58:05.690 --> 00:58:10.760 align:middle line:84% in a vector, in this case, a vector space, a matrix, 00:58:10.760 --> 00:58:12.680 align:middle line:90% and gives you a scalar. 00:58:12.680 --> 00:58:15.410 align:middle line:84% So I should be able to express this as a dot product. 00:58:15.410 --> 00:58:17.920 align:middle line:90% 00:58:17.920 --> 00:58:19.690 align:middle line:84% So I want to express this as a trace. 00:58:19.690 --> 00:58:25.490 align:middle line:90% 00:58:25.490 --> 00:58:32.290 align:middle line:84% So I'm going to expresses this somehow as grad f dot dA. 00:58:32.290 --> 00:58:36.590 align:middle line:90% 00:58:36.590 --> 00:58:40.260 align:middle line:90% So how can I do that? 00:58:40.260 --> 00:58:42.180 align:middle line:90% So there's a couple tricks. 00:58:42.180 --> 00:58:43.990 align:middle line:84% The first trick is this is a number. 00:58:43.990 --> 00:58:45.440 align:middle line:90% df, this is a scalar. 00:58:45.440 --> 00:58:48.980 align:middle line:90% 00:58:48.980 --> 00:58:53.260 align:middle line:84% A scalar, I can always write as the trace of itself. 00:58:53.260 --> 00:58:57.400 align:middle line:84% The trace of a scalar is just itself 00:58:57.400 --> 00:59:00.320 align:middle line:84% if you think of a scalar as a 1 by 1 matrix. 00:59:00.320 --> 00:59:01.900 align:middle line:84% So I can always put a trace there. 00:59:01.900 --> 00:59:03.370 align:middle line:84% But this is not a trace in the form 00:59:03.370 --> 00:59:06.040 align:middle line:84% that I want because I want the dA over on the right. 00:59:06.040 --> 00:59:10.090 align:middle line:90% 00:59:10.090 --> 00:59:14.400 align:middle line:84% So let me just put my little notes on the side. 00:59:14.400 --> 00:59:21.297 align:middle line:84% Note, if you take the trace of AB-- 00:59:21.297 --> 00:59:22.630 align:middle line:90% do people remember this formula? 00:59:22.630 --> 00:59:25.470 align:middle line:84% This is this is the same thing as the trace of BA. 00:59:25.470 --> 00:59:32.670 align:middle line:84% This is called the cyclic property of the trace. 00:59:32.670 --> 00:59:36.760 align:middle line:90% 00:59:36.760 --> 00:59:39.370 align:middle line:84% And if you don't remember it, it's pretty easy 00:59:39.370 --> 00:59:42.840 align:middle line:90% to convince yourself of it. 00:59:42.840 --> 00:59:46.010 align:middle line:84% So that does not mean I can rearrange 00:59:46.010 --> 00:59:47.450 align:middle line:90% this in any order I want. 00:59:47.450 --> 00:59:50.070 align:middle line:84% It means if I have two things, I can swap them. 00:59:50.070 --> 00:59:51.770 align:middle line:84% So it's called cyclic because if I 00:59:51.770 --> 00:59:54.410 align:middle line:84% think of this thing as the first thing and this as the second, 00:59:54.410 --> 00:59:55.130 align:middle line:90% I can swap it. 00:59:55.130 --> 00:59:59.353 align:middle line:84% I can move y over to there, or I could move the x over 00:59:59.353 --> 01:00:00.770 align:middle line:84% to the other side, I can do what's 01:00:00.770 --> 01:00:02.930 align:middle line:84% called a cyclic permutation of these things right. 01:00:02.930 --> 01:00:04.722 align:middle line:84% So I have to think of them-- to apply this, 01:00:04.722 --> 01:00:06.510 align:middle line:84% I have to group this into two pieces. 01:00:06.510 --> 01:00:13.250 align:middle line:84% So I could group this into, for example, x transpose and this. 01:00:13.250 --> 01:00:16.280 align:middle line:84% And then I should be able to swap them. 01:00:16.280 --> 01:00:18.860 align:middle line:90% 01:00:18.860 --> 01:00:20.957 align:middle line:90% And y-- sorry. 01:00:20.957 --> 01:00:22.790 align:middle line:84% I don't want to do it in that order, though. 01:00:22.790 --> 01:00:24.710 align:middle line:84% I want to swap it in the other way. 01:00:24.710 --> 01:00:32.370 align:middle line:90% 01:00:32.370 --> 01:00:36.475 align:middle line:84% So this is my cyclic permutation because I want 01:00:36.475 --> 01:00:37.600 align:middle line:90% to put the dA in the right. 01:00:37.600 --> 01:00:44.610 align:middle line:84% So this is the trace of y x transpose times dA. 01:00:44.610 --> 01:00:47.300 align:middle line:90% 01:00:47.300 --> 01:00:49.220 align:middle line:84% Now, this looks like a dot product. 01:00:49.220 --> 01:00:52.500 align:middle line:90% 01:00:52.500 --> 01:00:55.640 align:middle line:84% So this is-- remember, the dot product 01:00:55.640 --> 01:00:59.250 align:middle line:84% is always trace of something transpose dA. 01:00:59.250 --> 01:01:01.760 align:middle line:84% So this is really, to make it a dot product, 01:01:01.760 --> 01:01:03.080 align:middle line:90% I have to transpose the thing. 01:01:03.080 --> 01:01:08.660 align:middle line:90% This is xy transpose dot dA. 01:01:08.660 --> 01:01:11.510 align:middle line:90% 01:01:11.510 --> 01:01:14.580 align:middle line:90% Does everyone see this? 01:01:14.580 --> 01:01:19.470 align:middle line:84% Because remember, the trace of the dot product, again, is-- 01:01:19.470 --> 01:01:22.060 align:middle line:90% where is it? 01:01:22.060 --> 01:01:24.370 align:middle line:84% The dot product is trace A transpose B. 01:01:24.370 --> 01:01:32.980 align:middle line:84% So I have to view this term here as the transpose of that. 01:01:32.980 --> 01:01:36.410 align:middle line:90% And so that's our gradient. 01:01:36.410 --> 01:01:41.195 align:middle line:90% So this is our gradient of f. 01:01:41.195 --> 01:01:47.480 align:middle line:90% 01:01:47.480 --> 01:01:50.782 align:middle line:84% And it's whatever we take the dot product with to get dA. 01:01:50.782 --> 01:01:52.490 align:middle line:84% So as you see in the homework, basically, 01:01:52.490 --> 01:01:56.120 align:middle line:84% sometimes if you have a vector in and a scalar out, 01:01:56.120 --> 01:01:59.250 align:middle line:84% you know it has to be a dot product in some form. 01:01:59.250 --> 01:02:01.370 align:middle line:84% If you have a dot product in your vector space, 01:02:01.370 --> 01:02:04.130 align:middle line:84% you have to be able to write it in that form. 01:02:04.130 --> 01:02:08.270 align:middle line:84% But sometimes when you apply the derivative rule, 01:02:08.270 --> 01:02:10.100 align:middle line:84% it doesn't look like a dot product. 01:02:10.100 --> 01:02:12.098 align:middle line:84% You need to do a little bit of rearrangement 01:02:12.098 --> 01:02:14.390 align:middle line:84% to get it in the form where it looks like a dot product 01:02:14.390 --> 01:02:15.485 align:middle line:90% to get the gradient out. 01:02:15.485 --> 01:02:18.015 align:middle line:90% 01:02:18.015 --> 01:02:19.520 align:middle line:90% You need to learn a few rules. 01:02:19.520 --> 01:02:21.590 align:middle line:84% And they are always of this kind of form, 01:02:21.590 --> 01:02:25.010 align:middle line:84% where you need to be able to transpose things or learn which 01:02:25.010 --> 01:02:28.100 align:middle line:84% things can be permuted and so forth. 01:02:28.100 --> 01:02:31.760 align:middle line:84% But it's really useful once you get used to this. 01:02:31.760 --> 01:02:34.130 align:middle line:84% I could have done the same thing by writing this out 01:02:34.130 --> 01:02:36.320 align:middle line:84% in terms of all the components of A 01:02:36.320 --> 01:02:38.480 align:middle line:84% and then taking the partial derivative with respect 01:02:38.480 --> 01:02:39.230 align:middle line:90% to each component. 01:02:39.230 --> 01:02:41.990 align:middle line:84% If you work hard enough, you'll realize 01:02:41.990 --> 01:02:46.190 align:middle line:84% that all these partial derivatives, in fact, 01:02:46.190 --> 01:02:48.560 align:middle line:84% give you the matrix xy transpose. 01:02:48.560 --> 01:02:51.275 align:middle line:90% But it's a lot more work. 01:02:51.275 --> 01:02:55.545 align:middle line:90% 01:02:55.545 --> 01:02:58.170 align:middle line:84% This is suddenly now we can take gradients of matrix functions. 01:02:58.170 --> 01:02:59.587 align:middle line:84% And in fact, we can take gradients 01:02:59.587 --> 01:03:02.220 align:middle line:84% of any scalar function in any vector spaces 01:03:02.220 --> 01:03:03.880 align:middle line:90% with a dot product. 01:03:03.880 --> 01:03:06.960 align:middle line:84% So maybe later on, we'll talk about even other kinds 01:03:06.960 --> 01:03:09.240 align:middle line:84% of vector spaces, where you can define gradients 01:03:09.240 --> 01:03:11.910 align:middle line:84% that are even weirder looking than gradients 01:03:11.910 --> 01:03:13.170 align:middle line:90% of matrix functions. 01:03:13.170 --> 01:03:15.720 align:middle line:84% And in a subsequent lecture, Alan 01:03:15.720 --> 01:03:20.550 align:middle line:84% has a nice trick to derive this property of the gradient 01:03:20.550 --> 01:03:21.993 align:middle line:90% of a determinant. 01:03:21.993 --> 01:03:23.660 align:middle line:84% GUEST SPEAKER: I forgot that I had that. 01:03:23.660 --> 01:03:24.710 align:middle line:90% Thanks for the reminder. 01:03:24.710 --> 01:03:27.380 align:middle line:90% STEVEN JOHNSON: Yes. 01:03:27.380 --> 01:03:28.550 align:middle line:90% But any questions? 01:03:28.550 --> 01:03:32.660 align:middle line:84% We'll take maybe a three-minute break. 01:03:32.660 --> 01:03:36.070 align:middle line:90% 01:03:36.070 --> 01:03:37.240 align:middle line:90% AUDIENCE: No questions. 01:03:37.240 --> 01:03:38.448 align:middle line:90% STEVEN JOHNSON: No questions? 01:03:38.448 --> 01:03:40.660 align:middle line:84% So let's take a three-minute break. 01:03:40.660 --> 01:03:41.950 align:middle line:90% It's 12:11. 01:03:41.950 --> 01:03:44.640 align:middle line:90% We'll get back to 12:14. 01:03:44.640 --> 01:03:48.000 align:middle line:90%