WEBVTT 00:00:00.000 --> 00:00:01.968 align:middle line:90% [SQUEAKING] 00:00:01.968 --> 00:00:03.444 align:middle line:90% [RUSTLING] 00:00:03.444 --> 00:00:04.920 align:middle line:90% [CLICKING] 00:00:04.920 --> 00:00:10.650 align:middle line:90% 00:00:10.650 --> 00:00:15.420 align:middle line:84% STEVEN G. JOHNSON: So I want to revisit the things that Alan 00:00:15.420 --> 00:00:17.700 align:middle line:84% talked about, but just a little bit more 00:00:17.700 --> 00:00:20.580 align:middle line:90% slowly and a bit more-- 00:00:20.580 --> 00:00:23.213 align:middle line:84% just try and lay out the rules for you as clearly as I can. 00:00:23.213 --> 00:00:24.630 align:middle line:84% And what we're going to try and do 00:00:24.630 --> 00:00:29.460 align:middle line:84% is, again, just revisit the notion of a derivative 00:00:29.460 --> 00:00:33.660 align:middle line:84% to try and write it in a way that we can generalize 00:00:33.660 --> 00:00:34.830 align:middle line:90% to other kinds of objects. 00:00:34.830 --> 00:00:36.900 align:middle line:84% And so I'm going to start with 18.01 00:00:36.900 --> 00:00:39.220 align:middle line:84% and then go to 18.02 and so forth. 00:00:39.220 --> 00:00:43.530 align:middle line:84% So as Alan said, the key notion of a derivative, 00:00:43.530 --> 00:00:48.030 align:middle line:84% just, I think, it's easy to get so good at taking derivatives, 00:00:48.030 --> 00:00:49.980 align:middle line:84% like knowing the rule for the derivative 00:00:49.980 --> 00:00:52.770 align:middle line:90% of sine or cosine or x squared. 00:00:52.770 --> 00:00:56.020 align:middle line:84% You're so good at doing them that you forget what they are, 00:00:56.020 --> 00:00:56.520 align:middle line:90% right? 00:00:56.520 --> 00:01:00.030 align:middle line:84% And so the very first thing you learned about a derivative 00:01:00.030 --> 00:01:03.000 align:middle line:84% is that it's the slope of the tangent. 00:01:03.000 --> 00:01:08.370 align:middle line:84% But what that really is is linearization. 00:01:08.370 --> 00:01:12.730 align:middle line:84% So you have some arbitrary maybe nonlinear function f of x. 00:01:12.730 --> 00:01:16.210 align:middle line:90% And you're at a point x. 00:01:16.210 --> 00:01:19.240 align:middle line:84% And near that point, you're going 00:01:19.240 --> 00:01:21.950 align:middle line:84% to approximate the function with a straight line. 00:01:21.950 --> 00:01:23.140 align:middle line:90% That's the tangent. 00:01:23.140 --> 00:01:26.640 align:middle line:84% So it's really the linear approximation of f. 00:01:26.640 --> 00:01:30.870 align:middle line:84% And then if you move a little bit away from x-- so 00:01:30.870 --> 00:01:33.990 align:middle line:84% let me call that delta x, so not d. 00:01:33.990 --> 00:01:37.470 align:middle line:84% Delta is going to be a finite change. 00:01:37.470 --> 00:01:40.420 align:middle line:84% d is going to be infinitesimal pretty soon. 00:01:40.420 --> 00:01:43.200 align:middle line:84% But if you move it just a finite amount, a little finite amount 00:01:43.200 --> 00:01:47.350 align:middle line:84% delta x away, of course, the function value changes. 00:01:47.350 --> 00:01:52.350 align:middle line:84% But in a linear approximation, the new function value 00:01:52.350 --> 00:01:53.400 align:middle line:90% is the red dot here. 00:01:53.400 --> 00:01:57.150 align:middle line:84% So that linear approximation is, if you're 00:01:57.150 --> 00:02:04.030 align:middle line:84% taking the function f of x at x plus delta x, 00:02:04.030 --> 00:02:08.289 align:middle line:90% the new value is f of x. 00:02:08.289 --> 00:02:10.270 align:middle line:84% And then this linear thing is just 00:02:10.270 --> 00:02:13.510 align:middle line:84% the slope, which we call f prime of x times delta x. 00:02:13.510 --> 00:02:15.400 align:middle line:84% That's just the definition of the slope. 00:02:15.400 --> 00:02:20.155 align:middle line:84% It's the little change in y for a little change in x. 00:02:20.155 --> 00:02:23.440 align:middle line:84% And, of course, these two terms are not exact. 00:02:23.440 --> 00:02:26.830 align:middle line:84% This red dot doesn't exactly match where 00:02:26.830 --> 00:02:28.100 align:middle line:90% you are in the real function. 00:02:28.100 --> 00:02:29.420 align:middle line:90% So there are also corrections. 00:02:29.420 --> 00:02:32.080 align:middle line:84% But the corrections are higher order. 00:02:32.080 --> 00:02:35.365 align:middle line:84% They're terms look like delta x squared, delta x cubed maybe. 00:02:35.365 --> 00:02:38.092 align:middle line:84% Maybe if the function is not higher-- 00:02:38.092 --> 00:02:39.550 align:middle line:84% it doesn't have higher derivatives, 00:02:39.550 --> 00:02:43.600 align:middle line:84% it might have square root of delta x, so delta x to the 1.1. 00:02:43.600 --> 00:02:45.640 align:middle line:84% But these are all terms that are going 00:02:45.640 --> 00:02:48.220 align:middle line:84% to be higher powers of delta x terms 00:02:48.220 --> 00:02:51.520 align:middle line:84% that, if delta x is sufficiently small, 00:02:51.520 --> 00:02:55.000 align:middle line:84% these terms will become more and more negligible 00:02:55.000 --> 00:02:57.790 align:middle line:90% compared to this linear term. 00:02:57.790 --> 00:03:01.450 align:middle line:84% And a nice notation for this that's 00:03:01.450 --> 00:03:06.040 align:middle line:84% used a lot in computer science, less so outside of that, 00:03:06.040 --> 00:03:09.395 align:middle line:84% but it is used in calculus as well, maybe not in 18.01, 00:03:09.395 --> 00:03:11.300 align:middle line:90% is this asymptotic notation. 00:03:11.300 --> 00:03:14.600 align:middle line:84% In computer science, probably many of you 00:03:14.600 --> 00:03:16.400 align:middle line:90% will have seen big O notation. 00:03:16.400 --> 00:03:19.860 align:middle line:84% Here, I'm going to use a variant called little o notation. 00:03:19.860 --> 00:03:23.150 align:middle line:84% So this is not a capital O. This is a lowercase o. 00:03:23.150 --> 00:03:26.790 align:middle line:84% So these terms we call little o of delta x. 00:03:26.790 --> 00:03:29.090 align:middle line:90% It's a little o of delta x. 00:03:29.090 --> 00:03:34.880 align:middle line:84% It denotes any function that goes to 0 faster than linear, 00:03:34.880 --> 00:03:38.300 align:middle line:90% faster than delta x. 00:03:38.300 --> 00:03:41.840 align:middle line:84% So delta x squared goes to 0 faster than delta 00:03:41.840 --> 00:03:44.150 align:middle line:90% x as delta x goes to 0. 00:03:44.150 --> 00:03:50.690 align:middle line:84% I should say as delta x goes to 0. 00:03:50.690 --> 00:03:56.930 align:middle line:84% And so you look at this thing and this probably 00:03:56.930 --> 00:03:59.090 align:middle line:84% looks a lot like a Taylor series. 00:03:59.090 --> 00:04:00.860 align:middle line:84% You do a Taylor expansion of f of x 00:04:00.860 --> 00:04:02.750 align:middle line:90% plus delta x around f of x. 00:04:02.750 --> 00:04:03.950 align:middle line:90% This is the first term. 00:04:03.950 --> 00:04:05.570 align:middle line:90% This is the second term. 00:04:05.570 --> 00:04:10.670 align:middle line:84% The third term, remember, is 1/2 f double prime delta x squared 00:04:10.670 --> 00:04:12.710 align:middle line:90% or something like that. 00:04:12.710 --> 00:04:15.320 align:middle line:84% But that's the wrong way to look at this. 00:04:15.320 --> 00:04:19.350 align:middle line:84% A Taylor series is a much more advanced concept. 00:04:19.350 --> 00:04:21.620 align:middle line:84% It's something you can do much later in calculus 00:04:21.620 --> 00:04:22.460 align:middle line:90% and for good reason. 00:04:22.460 --> 00:04:25.700 align:middle line:84% Because not every function even has a Taylor series 00:04:25.700 --> 00:04:26.780 align:middle line:90% that converges. 00:04:26.780 --> 00:04:28.070 align:middle line:90% This is more basic. 00:04:28.070 --> 00:04:30.950 align:middle line:84% This is really the definition of a derivative. 00:04:30.950 --> 00:04:35.600 align:middle line:84% The derivative is the linearization. 00:04:35.600 --> 00:04:40.580 align:middle line:84% If you make a small change in delta x, the change in f 00:04:40.580 --> 00:04:44.810 align:middle line:84% is a linear term plus smaller stuff. 00:04:44.810 --> 00:04:48.270 align:middle line:84% And that smaller stuff only gives you a Taylor series 00:04:48.270 --> 00:04:50.210 align:middle line:90% if it's basically a polynomial. 00:04:50.210 --> 00:04:53.693 align:middle line:84% It might be smaller stuff like delta x to the 1.1 00:04:53.693 --> 00:04:55.610 align:middle line:84% in which case it doesn't have a Taylor series. 00:04:55.610 --> 00:04:57.390 align:middle line:90% But this is always true. 00:04:57.390 --> 00:04:59.150 align:middle line:84% This is just this is what it means 00:04:59.150 --> 00:05:01.190 align:middle line:90% to be the slope of a tangent. 00:05:01.190 --> 00:05:04.630 align:middle line:84% And so the nice thing about this is 00:05:04.630 --> 00:05:08.860 align:middle line:84% Alan says this notion, if we keep the delta x on the right, 00:05:08.860 --> 00:05:14.560 align:middle line:84% this is going to be much easier to generalize 00:05:14.560 --> 00:05:18.460 align:middle line:84% to other kinds of x's that are vectors or matrices or even 00:05:18.460 --> 00:05:20.995 align:middle line:84% other functions, other kinds of things. 00:05:20.995 --> 00:05:23.560 align:middle line:90% 00:05:23.560 --> 00:05:25.940 align:middle line:90% So this is the linearization. 00:05:25.940 --> 00:05:26.440 align:middle line:90% Whoops. 00:05:26.440 --> 00:05:32.490 align:middle line:90% 00:05:32.490 --> 00:05:40.630 align:middle line:84% So we're going to have a delta f, which I'm 00:05:40.630 --> 00:05:42.730 align:middle line:90% going to have to define as-- 00:05:42.730 --> 00:05:46.540 align:middle line:90% whoops, let me put it in black-- 00:05:46.540 --> 00:05:49.480 align:middle line:90% f of x plus delta x. 00:05:49.480 --> 00:05:52.270 align:middle line:84% And again, the delta, the Greek letter delta, 00:05:52.270 --> 00:05:53.590 align:middle line:90% is not infinitesimal. 00:05:53.590 --> 00:05:55.780 align:middle line:90% It's just a small number. 00:05:55.780 --> 00:05:57.475 align:middle line:90% It's just a number. 00:05:57.475 --> 00:06:01.420 align:middle line:90% 00:06:01.420 --> 00:06:04.510 align:middle line:90% This is that thing. 00:06:04.510 --> 00:06:12.215 align:middle line:84% But I'm going to drop terms, higher order terms. 00:06:12.215 --> 00:06:13.465 align:middle line:90% So there'll be an error there. 00:06:13.465 --> 00:06:16.960 align:middle line:90% 00:06:16.960 --> 00:06:19.460 align:middle line:84% Well, yeah, so actually let me put this another way. 00:06:19.460 --> 00:06:27.100 align:middle line:84% So this is going to be approximately f prime of x 00:06:27.100 --> 00:06:30.160 align:middle line:90% delta x plus higher order terms. 00:06:30.160 --> 00:06:33.980 align:middle line:90% 00:06:33.980 --> 00:06:35.675 align:middle line:90% This is the higher order. 00:06:35.675 --> 00:06:40.230 align:middle line:90% 00:06:40.230 --> 00:06:52.090 align:middle line:84% So this is the small change in the input of the function. 00:06:52.090 --> 00:06:55.720 align:middle line:90% 00:06:55.720 --> 00:07:01.875 align:middle line:84% And this is the resulting small change in the output. 00:07:01.875 --> 00:07:08.380 align:middle line:90% 00:07:08.380 --> 00:07:13.630 align:middle line:84% And this is going to be the definition of the derivative. 00:07:13.630 --> 00:07:19.900 align:middle line:84% The derivative is whatever you do to delta x to first order 00:07:19.900 --> 00:07:24.580 align:middle line:84% to give you the linear change in the output for a small change 00:07:24.580 --> 00:07:26.870 align:middle line:90% in the input. 00:07:26.870 --> 00:07:32.210 align:middle line:84% It's a little annoying, though, to keep these o's around. 00:07:32.210 --> 00:07:34.610 align:middle line:84% So we keep always having to-- whenever 00:07:34.610 --> 00:07:36.350 align:middle line:84% we have a finite change in the input 00:07:36.350 --> 00:07:39.260 align:middle line:84% and a finite change in the output, this is never exact. 00:07:39.260 --> 00:07:43.610 align:middle line:84% This is an approximate relationship. 00:07:43.610 --> 00:07:46.460 align:middle line:84% And we have to keep saying plus higher order terms, 00:07:46.460 --> 00:07:49.010 align:middle line:84% plus higher order terms, plus little o delta x. 00:07:49.010 --> 00:07:52.010 align:middle line:90% And it's annoying. 00:07:52.010 --> 00:07:59.200 align:middle line:84% So it's easier to use to switch to differential notation. 00:07:59.200 --> 00:08:07.530 align:middle line:90% 00:08:07.530 --> 00:08:13.130 align:middle line:84% So I'm just going to change my delta to d. 00:08:13.130 --> 00:08:26.000 align:middle line:84% So df is going to be f of x plus dx minus f of x. 00:08:26.000 --> 00:08:28.880 align:middle line:90% 00:08:28.880 --> 00:08:38.150 align:middle line:84% And this is going to be f prime of x dx where this is-- 00:08:38.150 --> 00:08:40.100 align:middle line:90% I'm just going to right equal. 00:08:40.100 --> 00:08:45.950 align:middle line:84% So we can think of this differential in dx 00:08:45.950 --> 00:08:48.380 align:middle line:90% as being arbitrarily small. 00:08:48.380 --> 00:08:50.940 align:middle line:90% 00:08:50.940 --> 00:08:53.285 align:middle line:84% So it's really a limit, some kind of limit, of course. 00:08:53.285 --> 00:08:57.950 align:middle line:90% 00:08:57.950 --> 00:09:01.760 align:middle line:84% And so you can also think of it as really it's just this. 00:09:01.760 --> 00:09:06.482 align:middle line:84% But I'm just implicitly going to drop any higher order terms. 00:09:06.482 --> 00:09:08.440 align:middle line:84% ALAN EDELMAN: That's how I like to think of it. 00:09:08.440 --> 00:09:09.490 align:middle line:90% STEVEN G. JOHNSON: Right? 00:09:09.490 --> 00:09:11.140 align:middle line:84% So we don't have to get too fancy 00:09:11.140 --> 00:09:13.850 align:middle line:90% with defining differentials. 00:09:13.850 --> 00:09:16.030 align:middle line:84% I mean, this is a definition, right? 00:09:16.030 --> 00:09:20.350 align:middle line:84% This is just shorthand for this where 00:09:20.350 --> 00:09:23.110 align:middle line:84% I don't have to write plus a little o over delta x 00:09:23.110 --> 00:09:25.726 align:middle line:90% all the time. 00:09:25.726 --> 00:09:26.530 align:middle line:90% Let's see. 00:09:26.530 --> 00:09:42.020 align:middle line:90% 00:09:42.020 --> 00:09:45.920 align:middle line:84% And so it's important to keep in-- so this f prime to delta 00:09:45.920 --> 00:09:50.060 align:middle line:90% x, this is the derivative. 00:09:50.060 --> 00:09:55.580 align:middle line:90% Here df is the differential. 00:09:55.580 --> 00:09:59.200 align:middle line:84% So if I ask you for the derivative, 00:09:59.200 --> 00:10:00.370 align:middle line:90% I'm asking for f prime. 00:10:00.370 --> 00:10:01.480 align:middle line:90% I'm not asking for df. 00:10:01.480 --> 00:10:03.550 align:middle line:90% Of course, they're related. 00:10:03.550 --> 00:10:06.200 align:middle line:84% And let's see if we can move this out of the way. 00:10:06.200 --> 00:10:06.700 align:middle line:90% Good. 00:10:06.700 --> 00:10:10.400 align:middle line:90% 00:10:10.400 --> 00:10:12.030 align:middle line:90% So what was I saying? 00:10:12.030 --> 00:10:12.530 align:middle line:90% Yes. 00:10:12.530 --> 00:10:15.080 align:middle line:90% 00:10:15.080 --> 00:10:18.110 align:middle line:84% So now, what I want to do is basically use 00:10:18.110 --> 00:10:21.230 align:middle line:84% this as the definition of a derivative, a more general 00:10:21.230 --> 00:10:22.740 align:middle line:90% definition of a derivative. 00:10:22.740 --> 00:10:28.940 align:middle line:84% So the linear algebra notion is that we have 00:10:28.940 --> 00:10:30.470 align:middle line:90% what's called a linear operator. 00:10:30.470 --> 00:10:37.230 align:middle line:90% 00:10:37.230 --> 00:10:49.440 align:middle line:84% So basically, the change in the output df 00:10:49.440 --> 00:10:55.300 align:middle line:84% is going to be a linear operator. 00:10:55.300 --> 00:11:07.800 align:middle line:84% Let me write that as times, which 00:11:07.800 --> 00:11:13.600 align:middle line:84% I'm going to call f prime of x times the change in the input. 00:11:13.600 --> 00:11:21.400 align:middle line:90% 00:11:21.400 --> 00:11:23.200 align:middle line:90% This is our-- whoops. 00:11:23.200 --> 00:11:26.200 align:middle line:84% Actually, but my input is in red, right? 00:11:26.200 --> 00:11:27.280 align:middle line:90% That's my color code. 00:11:27.280 --> 00:11:34.450 align:middle line:84% My color scheme is input is red and output is blue. 00:11:34.450 --> 00:11:36.280 align:middle line:90% So this is our dx. 00:11:36.280 --> 00:11:39.620 align:middle line:90% 00:11:39.620 --> 00:11:42.130 align:middle line:84% So I'm going to interpret this more generally. 00:11:42.130 --> 00:11:45.260 align:middle line:84% If x is going to be some kind of vector or matrix or whatever, 00:11:45.260 --> 00:11:48.790 align:middle line:84% this is just going to be a linear operation on this. 00:11:48.790 --> 00:11:52.120 align:middle line:84% And of course, for numbers, a linear operation, 00:11:52.120 --> 00:11:53.540 align:middle line:90% this is just a number. 00:11:53.540 --> 00:11:56.320 align:middle line:84% If dx is a number, the only linear operation you can do 00:11:56.320 --> 00:11:57.320 align:middle line:90% is multiply by a number. 00:11:57.320 --> 00:12:03.610 align:middle line:84% So let me just remind you if you haven't taken linear algebra 00:12:03.610 --> 00:12:06.170 align:middle line:90% for a while, a review. 00:12:06.170 --> 00:12:21.380 align:middle line:90% 00:12:21.380 --> 00:12:23.400 align:middle line:84% Let's talk about what a linear operator is. 00:12:23.400 --> 00:12:33.140 align:middle line:84% So suppose we have some given vectors, v, 00:12:33.140 --> 00:12:36.335 align:middle line:90% in some vector space. 00:12:36.335 --> 00:12:41.280 align:middle line:90% 00:12:41.280 --> 00:12:43.680 align:middle line:84% That would be capital V. And remember, 00:12:43.680 --> 00:12:50.460 align:middle line:84% a vector space is anything where you can basically 00:12:50.460 --> 00:12:52.470 align:middle line:84% add, subtract, and multiply by scalars. 00:12:52.470 --> 00:12:58.710 align:middle line:90% 00:12:58.710 --> 00:13:05.350 align:middle line:84% We have a plus or minus and times scalar operations 00:13:05.350 --> 00:13:07.140 align:middle line:90% that stay in our vector space. 00:13:07.140 --> 00:13:09.510 align:middle line:84% That's what the informal definition of a vector space 00:13:09.510 --> 00:13:09.630 align:middle line:90% is. 00:13:09.630 --> 00:13:11.410 align:middle line:84% You can write out axioms and so forth, 00:13:11.410 --> 00:13:14.850 align:middle line:84% but that's basically what it means. 00:13:14.850 --> 00:13:19.970 align:middle line:84% A linear operator, this is what we're 00:13:19.970 --> 00:13:22.880 align:middle line:84% going to mean by linearization in the derivative. 00:13:22.880 --> 00:13:23.990 align:middle line:90% It has to be linear. 00:13:23.990 --> 00:13:26.712 align:middle line:84% What does it mean to be linear in general? 00:13:26.712 --> 00:13:27.920 align:middle line:90% So we're going to call this-- 00:13:27.920 --> 00:13:36.810 align:middle line:84% I'm going to denote this by, let's say, L of-- 00:13:36.810 --> 00:13:47.250 align:middle line:84% let me denote it by square brackets or just by Lv. 00:13:47.250 --> 00:13:49.380 align:middle line:84% When it's clear enough, I'll just write it 00:13:49.380 --> 00:13:52.170 align:middle line:90% as if it were a multiplication. 00:13:52.170 --> 00:13:54.060 align:middle line:90% Often, that'll be clear enough. 00:13:54.060 --> 00:13:56.220 align:middle line:90% This is really acting on-- 00:13:56.220 --> 00:13:59.220 align:middle line:84% when I write Lv, it's not necessarily 00:13:59.220 --> 00:14:01.530 align:middle line:90% an ordinary multiplication. 00:14:01.530 --> 00:14:05.750 align:middle line:84% This is just going to be acting on v. 00:14:05.750 --> 00:14:13.708 align:middle line:84% So a linear operator is a rule that basically takes a vector 00:14:13.708 --> 00:14:16.250 align:middle line:84% and gives you a vector out maybe in a different vector space. 00:14:16.250 --> 00:14:27.900 align:middle line:84% So this is L takes a vector in, v in, 00:14:27.900 --> 00:14:30.980 align:middle line:90% and it gives you a vector-- 00:14:30.980 --> 00:14:36.212 align:middle line:90% 00:14:36.212 --> 00:14:37.827 align:middle line:90% that's a terrible L-- 00:14:37.827 --> 00:14:44.000 align:middle line:84% Lv out maybe in a different vector space. 00:14:44.000 --> 00:14:52.950 align:middle line:84% And linearity means what you think it means. 00:14:52.950 --> 00:14:55.540 align:middle line:84% It means if you take, for example, 00:14:55.540 --> 00:15:06.930 align:middle line:84% L of v1 plus v2, if you take the sum of the inputs, that's 00:15:06.930 --> 00:15:15.510 align:middle line:84% the same thing as L of v1 plus L of v2. 00:15:15.510 --> 00:15:17.520 align:middle line:84% So if you add inputs, that's the same thing 00:15:17.520 --> 00:15:22.630 align:middle line:84% as adding the outputs or if you multiply by a scalar. 00:15:22.630 --> 00:15:26.010 align:middle line:84% And so as usual in linear algebra, 00:15:26.010 --> 00:15:30.990 align:middle line:84% Greek letters are going to denote scalars as you would. 00:15:30.990 --> 00:15:34.330 align:middle line:90% So that's equal to-- 00:15:34.330 --> 00:15:35.970 align:middle line:90% you can pull out the scalar. 00:15:35.970 --> 00:15:42.760 align:middle line:90% 00:15:42.760 --> 00:15:44.680 align:middle line:84% And so the nice thing about linear operators 00:15:44.680 --> 00:15:48.770 align:middle line:84% is we can define them on lots of kinds of vector spaces. 00:15:48.770 --> 00:15:50.350 align:middle line:84% So let's just do a couple of examples 00:15:50.350 --> 00:15:54.265 align:middle line:84% just to make sure we're on the same page here. 00:15:54.265 --> 00:15:57.550 align:middle line:90% 00:15:57.550 --> 00:16:02.610 align:middle line:84% And so, for example, you could just 00:16:02.610 --> 00:16:07.695 align:middle line:84% have L is multiplication by a scalar. 00:16:07.695 --> 00:16:12.830 align:middle line:90% 00:16:12.830 --> 00:16:23.530 align:middle line:84% So you can have just L of v is just alpha v. 00:16:23.530 --> 00:16:26.020 align:middle line:84% That's a perfectly good linear operation. 00:16:26.020 --> 00:16:29.500 align:middle line:84% And if your vector space, if your vs are scalars, 00:16:29.500 --> 00:16:31.000 align:middle line:90% this is the only option. 00:16:31.000 --> 00:16:43.790 align:middle line:90% 00:16:43.790 --> 00:16:47.510 align:middle line:84% If vs are, say, real numbers, that's 00:16:47.510 --> 00:16:49.250 align:middle line:90% a perfectly good vector space. 00:16:49.250 --> 00:16:52.280 align:middle line:84% Another one that you're very familiar with 00:16:52.280 --> 00:16:57.653 align:middle line:84% is if L is a multiplication by a matrix. 00:16:57.653 --> 00:17:00.320 align:middle line:84% ALAN EDELMAN: Steven, maybe I'll just point out sometimes people 00:17:00.320 --> 00:17:01.070 align:middle line:90% like to ask me. 00:17:01.070 --> 00:17:07.530 align:middle line:84% Wait, I thought that the linear operators on scalars are, 00:17:07.530 --> 00:17:11.119 align:middle line:84% I think in high school notation, y equals mx plus b, right? 00:17:11.119 --> 00:17:12.410 align:middle line:90% It's scalar times-- 00:17:12.410 --> 00:17:13.660 align:middle line:90% STEVEN G. JOHNSON: Yeah, yeah. 00:17:13.660 --> 00:17:15.740 align:middle line:84% ALAN EDELMAN: --plus an offset that may not be 0. 00:17:15.740 --> 00:17:17.960 align:middle line:90% So what's going on here? 00:17:17.960 --> 00:17:19.760 align:middle line:90% Is that linear or not linear? 00:17:19.760 --> 00:17:20.990 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:17:20.990 --> 00:17:25.440 align:middle line:84% So what about-- yeah, so let's do that. 00:17:25.440 --> 00:17:31.390 align:middle line:90% 00:17:31.390 --> 00:17:34.600 align:middle line:84% Let me call it a different thing. 00:17:34.600 --> 00:17:37.180 align:middle line:90% What letter should I use? 00:17:37.180 --> 00:17:56.868 align:middle line:84% O, let's use O. Ov equals 2v plus 1 for v is real numbers. 00:17:56.868 --> 00:17:58.660 align:middle line:84% ALAN EDELMAN: Is that linear or not linear? 00:17:58.660 --> 00:17:59.535 align:middle line:90% That is the question. 00:17:59.535 --> 00:18:02.005 align:middle line:90% 00:18:02.005 --> 00:18:04.300 align:middle line:90% The graph is a line. 00:18:04.300 --> 00:18:08.840 align:middle line:84% So we all think of it as linear, but go ahead. 00:18:08.840 --> 00:18:10.650 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:18:10.650 --> 00:18:12.077 align:middle line:90% So does it satisfy the rules? 00:18:12.077 --> 00:18:12.910 align:middle line:90% That's the question. 00:18:12.910 --> 00:18:15.570 align:middle line:84% So if I multiply the input by 2, does it 00:18:15.570 --> 00:18:17.110 align:middle line:90% multiply the output by 2? 00:18:17.110 --> 00:18:17.610 align:middle line:90% No. 00:18:17.610 --> 00:18:23.650 align:middle line:90% 00:18:23.650 --> 00:18:28.240 align:middle line:90% So if we do O of 2-- 00:18:28.240 --> 00:18:29.273 align:middle line:90% no, let's do 3. 00:18:29.273 --> 00:18:38.500 align:middle line:90% 3v, that's, what, 6v plus 1. 00:18:38.500 --> 00:18:41.360 align:middle line:90% 00:18:41.360 --> 00:18:50.820 align:middle line:84% And that's very much not equal to 3Ov, 00:18:50.820 --> 00:18:58.680 align:middle line:90% which that would be 6v plus 3. 00:18:58.680 --> 00:19:04.640 align:middle line:90% 00:19:04.640 --> 00:19:06.980 align:middle line:84% So this one, it does have a name. 00:19:06.980 --> 00:19:07.730 align:middle line:90% It's related. 00:19:07.730 --> 00:19:09.350 align:middle line:84% These are sometimes called affine. 00:19:09.350 --> 00:19:16.270 align:middle line:90% 00:19:16.270 --> 00:19:18.520 align:middle line:84% ALAN EDELMAN: Affine, but not linear even if the graph 00:19:18.520 --> 00:19:21.640 align:middle line:90% is demonstrably aligned? 00:19:21.640 --> 00:19:22.690 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:19:22.690 --> 00:19:25.357 align:middle line:84% ALAN EDELMAN: They're not linear in the sense of linear algebra. 00:19:25.357 --> 00:19:29.608 align:middle line:90% 00:19:29.608 --> 00:19:30.650 align:middle line:90% STEVEN G. JOHNSON: Right? 00:19:30.650 --> 00:19:37.250 align:middle line:84% So another one is clearly multiplication by a matrix. 00:19:37.250 --> 00:19:39.710 align:middle line:84% That's why we do matrices in linear algebra. 00:19:39.710 --> 00:19:42.950 align:middle line:84% Because they're a nice way of writing down a linear operation 00:19:42.950 --> 00:19:46.383 align:middle line:90% if your vs are column vectors. 00:19:46.383 --> 00:19:48.800 align:middle line:84% They're not the only way of writing down linear operation. 00:19:48.800 --> 00:19:51.065 align:middle line:84% So for example, if you take a column vector 00:19:51.065 --> 00:19:53.810 align:middle line:84% and multiply it by 3, you could write 00:19:53.810 --> 00:19:56.270 align:middle line:84% that down as a matrix with all 3s along the diagonal. 00:19:56.270 --> 00:19:58.760 align:middle line:84% But it's a lot easier to write that down as 3, 00:19:58.760 --> 00:20:01.370 align:middle line:84% I'd say, as a scalar, than to write it down as a matrix. 00:20:01.370 --> 00:20:04.020 align:middle line:90% 00:20:04.020 --> 00:20:09.345 align:middle line:84% So another example, just to be more-- 00:20:09.345 --> 00:20:14.030 align:middle line:90% 00:20:14.030 --> 00:20:15.200 align:middle line:90% so another vector space. 00:20:15.200 --> 00:20:19.170 align:middle line:84% If you took 1806, you learned that we can have a-- whoops, 00:20:19.170 --> 00:20:20.420 align:middle line:90% I have to get my color scheme. 00:20:20.420 --> 00:20:22.010 align:middle line:90% Yeah, so my vectors are red. 00:20:22.010 --> 00:20:26.260 align:middle line:90% 00:20:26.260 --> 00:20:38.190 align:middle line:84% Suppose the vector space V is the set of functions f of x 00:20:38.190 --> 00:20:42.510 align:middle line:84% that take real numbers in and give you real numbers out. 00:20:42.510 --> 00:20:44.580 align:middle line:84% Those are a perfectly good vector space. 00:20:44.580 --> 00:20:45.750 align:middle line:90% I can take two functions. 00:20:45.750 --> 00:20:48.360 align:middle line:84% I can add them or subtract them, get another function. 00:20:48.360 --> 00:20:50.220 align:middle line:84% I can take a function and multiply by 2, 00:20:50.220 --> 00:20:52.210 align:middle line:90% get another function. 00:20:52.210 --> 00:20:54.970 align:middle line:84% ALAN EDELMAN: Wait, how do we get sine plus cosine? 00:20:54.970 --> 00:20:57.708 align:middle line:84% STEVEN G. JOHNSON: I get sine x plus cosine x. 00:20:57.708 --> 00:20:59.125 align:middle line:84% It's just got some other function. 00:20:59.125 --> 00:21:04.590 align:middle line:90% 00:21:04.590 --> 00:21:10.290 align:middle line:84% So if you take sine x plus cosine x, 00:21:10.290 --> 00:21:12.900 align:middle line:84% that's the function f of x equals sine x plus cosine x. 00:21:12.900 --> 00:21:14.640 align:middle line:84% It's another rule that gives you-- takes 00:21:14.640 --> 00:21:17.580 align:middle line:90% real numbers to real numbers. 00:21:17.580 --> 00:21:20.865 align:middle line:84% And so what would be your linear operators on this? 00:21:20.865 --> 00:21:25.490 align:middle line:90% 00:21:25.490 --> 00:21:29.500 align:middle line:84% Well, multiplication by a scalar, that, of course, works. 00:21:29.500 --> 00:21:40.330 align:middle line:84% So let's think of L on a function f of x is just 2 f 00:21:40.330 --> 00:21:40.990 align:middle line:90% of x. 00:21:40.990 --> 00:21:42.880 align:middle line:84% That takes a function in, function out. 00:21:42.880 --> 00:21:44.650 align:middle line:90% That's linear. 00:21:44.650 --> 00:21:50.440 align:middle line:84% What about a linear operator on a function of f of x? 00:21:50.440 --> 00:22:00.620 align:middle line:84% Again, that gives you the derivative, just 00:22:00.620 --> 00:22:03.320 align:middle line:90% the ordinary 18.01 derivative. 00:22:03.320 --> 00:22:08.080 align:middle line:90% This is the 18.01 derivative. 00:22:08.080 --> 00:22:11.000 align:middle line:84% Obviously, that only works if the function is differentiable. 00:22:11.000 --> 00:22:16.900 align:middle line:84% So maybe we can look at the subspace 00:22:16.900 --> 00:22:18.550 align:middle line:90% of differentiable functions. 00:22:18.550 --> 00:22:21.840 align:middle line:90% 00:22:21.840 --> 00:22:23.550 align:middle line:90% That's also a vector space. 00:22:23.550 --> 00:22:25.800 align:middle line:84% Because if I take two differentiable functions 00:22:25.800 --> 00:22:28.060 align:middle line:84% and add or subtract or multiply by constants, 00:22:28.060 --> 00:22:29.880 align:middle line:90% they're still differentiable. 00:22:29.880 --> 00:22:31.890 align:middle line:90% I could also do integration. 00:22:31.890 --> 00:22:34.470 align:middle line:90% 00:22:34.470 --> 00:22:38.610 align:middle line:84% So if f of x that takes a function f of x in 00:22:38.610 --> 00:22:41.190 align:middle line:84% and gives you the integral from, I don't know, 00:22:41.190 --> 00:22:44.355 align:middle line:90% 0 to x of f prime-- 00:22:44.355 --> 00:22:51.120 align:middle line:84% no, so f of x prime and dx prime if they're integrable. 00:22:51.120 --> 00:22:53.340 align:middle line:84% Again, we need to restrict what functions 00:22:53.340 --> 00:22:56.400 align:middle line:84% are allowed if we're taking derivatives or integrals, so 00:22:56.400 --> 00:22:58.770 align:middle line:90% things where these exist. 00:22:58.770 --> 00:23:00.610 align:middle line:90% But this is perfectly linear. 00:23:00.610 --> 00:23:01.110 align:middle line:90% Why? 00:23:01.110 --> 00:23:04.110 align:middle line:84% Because if I take the function and I double it, 00:23:04.110 --> 00:23:08.520 align:middle line:84% if I double the integrand, it doubles the integrals. 00:23:08.520 --> 00:23:11.790 align:middle line:84% If I add two integrands, you add the integrals. 00:23:11.790 --> 00:23:13.500 align:middle line:84% Integration is a linear operation. 00:23:13.500 --> 00:23:15.300 align:middle line:84% Derivative is a linear operation. 00:23:15.300 --> 00:23:28.420 align:middle line:84% Another fun one is suppose we take L of f of x. 00:23:28.420 --> 00:23:39.180 align:middle line:84% And the output is the function f of x squared. 00:23:39.180 --> 00:23:40.920 align:middle line:90% So this doesn't look linear. 00:23:40.920 --> 00:23:42.030 align:middle line:90% I have a square there. 00:23:42.030 --> 00:23:45.630 align:middle line:90% 00:23:45.630 --> 00:23:46.810 align:middle line:90% But why is this linear? 00:23:46.810 --> 00:23:47.310 align:middle line:90% Why? 00:23:47.310 --> 00:23:59.315 align:middle line:84% Because, let's see, if I take L of two functions, 00:23:59.315 --> 00:24:07.390 align:middle line:84% if I have f of x plus g of x, that should be f of x squared 00:24:07.390 --> 00:24:10.090 align:middle line:90% plus g of x squared. 00:24:10.090 --> 00:24:13.690 align:middle line:84% I'm squaring the input, not the output. 00:24:13.690 --> 00:24:22.352 align:middle line:84% So that's equal to L of f plus L of g. 00:24:22.352 --> 00:24:23.810 align:middle line:84% ALAN EDELMAN: So I'll just comment. 00:24:23.810 --> 00:24:27.080 align:middle line:84% Leave it to mathematicians to take what most people would 00:24:27.080 --> 00:24:28.830 align:middle line:84% think of as just a column of numbers 00:24:28.830 --> 00:24:35.090 align:middle line:84% and abstract it out and say that this finite dimensional column 00:24:35.090 --> 00:24:39.800 align:middle line:84% of numbers is somehow the same as continuous functions 00:24:39.800 --> 00:24:45.410 align:middle line:84% or differentiable functions, satisfies the same axioms. 00:24:45.410 --> 00:24:47.230 align:middle line:84% So we'll call it a vector space as well. 00:24:47.230 --> 00:24:48.230 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:24:48.230 --> 00:24:50.630 align:middle line:84% But it's incredibly useful, though. 00:24:50.630 --> 00:24:54.770 align:middle line:84% Because very often, especially in physical sciences, 00:24:54.770 --> 00:24:58.220 align:middle line:84% you have something where conceptually you're 00:24:58.220 --> 00:24:59.690 align:middle line:90% solving for a functions. 00:24:59.690 --> 00:25:01.940 align:middle line:84% So you're solving for the fluid flow or something 00:25:01.940 --> 00:25:04.100 align:middle line:90% around an airplane wing. 00:25:04.100 --> 00:25:08.150 align:middle line:84% And what you want is that then take that fluid flow 00:25:08.150 --> 00:25:11.930 align:middle line:84% and compute the drag on the airplane wing. 00:25:11.930 --> 00:25:14.600 align:middle line:84% And then in order to optimize it, 00:25:14.600 --> 00:25:16.790 align:middle line:84% you want the derivative of the drag with respect 00:25:16.790 --> 00:25:19.550 align:middle line:84% to that flow field, with respect to the function, 00:25:19.550 --> 00:25:22.380 align:middle line:84% or with respect to the shape of the airplane, which 00:25:22.380 --> 00:25:23.290 align:middle line:90% is a function. 00:25:23.290 --> 00:25:25.910 align:middle line:84% So it's very, very nice to be able to take derivatives 00:25:25.910 --> 00:25:29.190 align:middle line:84% connected to functions and work with vector functions 00:25:29.190 --> 00:25:30.390 align:middle line:90% as vector spaces. 00:25:30.390 --> 00:25:34.050 align:middle line:84% And very soon we're going to be able to do that 00:25:34.050 --> 00:25:36.690 align:middle line:84% with this notion of a derivative. 00:25:36.690 --> 00:25:41.730 align:middle line:84% Because we're going to be able to define linear operators, 00:25:41.730 --> 00:25:43.410 align:middle line:90% functions that act on functions. 00:25:43.410 --> 00:25:44.940 align:middle line:84% And linear operators are functions. 00:25:44.940 --> 00:25:49.690 align:middle line:84% But that's getting a bit too far ahead of ourselves. 00:25:49.690 --> 00:25:50.190 align:middle line:90% OK. 00:25:50.190 --> 00:25:52.520 align:middle line:84% So the point is that the 18.01-- so far, we 00:25:52.520 --> 00:25:54.770 align:middle line:84% haven't done any derivatives more than 18.01, at least 00:25:54.770 --> 00:25:56.070 align:middle line:90% in my half. 00:25:56.070 --> 00:25:58.890 align:middle line:90% Alan went a bit further. 00:25:58.890 --> 00:26:02.220 align:middle line:84% But already we can start to see, hopefully, how 00:26:02.220 --> 00:26:03.870 align:middle line:90% this is going to generalize. 00:26:03.870 --> 00:26:06.720 align:middle line:84% So if you have a function f of x and you 00:26:06.720 --> 00:26:09.990 align:middle line:84% make a small change in the input, delta x, 00:26:09.990 --> 00:26:12.750 align:middle line:84% and you ask for the small change in the output 00:26:12.750 --> 00:26:17.100 align:middle line:84% to first order, which we can denote with this d notation, 00:26:17.100 --> 00:26:20.490 align:middle line:84% the derivative is the linear operator 00:26:20.490 --> 00:26:22.440 align:middle line:84% that gives us that, the linearization 00:26:22.440 --> 00:26:25.530 align:middle line:84% of that function for a small change in the input. 00:26:25.530 --> 00:26:31.160 align:middle line:84% And that is exactly equivalent to what you learned in 18.01. 00:26:31.160 --> 00:26:34.340 align:middle line:84% But it's going to be easier now to generalize this 00:26:34.340 --> 00:26:37.760 align:middle line:84% to other kinds of inputs and other kinds of outputs, 00:26:37.760 --> 00:26:42.890 align:middle line:84% where, in 18.01, we move this to the side where we take df, dx. 00:26:42.890 --> 00:26:44.690 align:middle line:90% We divide them. 00:26:44.690 --> 00:26:45.920 align:middle line:90% For numbers, that's fine. 00:26:45.920 --> 00:26:49.387 align:middle line:84% For other kinds of things, that becomes a little bit weirder 00:26:49.387 --> 00:26:49.970 align:middle line:90% to talk about. 00:26:49.970 --> 00:26:53.220 align:middle line:84% Of course, you could define it as notation. 00:26:53.220 --> 00:26:56.040 align:middle line:84% But I think it's a lot clearer if you 00:26:56.040 --> 00:26:58.380 align:middle line:84% think of it in this sense once you start generalizing 00:26:58.380 --> 00:27:00.190 align:middle line:90% to other kinds of objects. 00:27:00.190 --> 00:27:03.240 align:middle line:84% So with that said, let's do that. 00:27:03.240 --> 00:27:11.430 align:middle line:90% Now, let's revisit 18.02. 00:27:11.430 --> 00:27:13.005 align:middle line:90% And let me do it in two parts. 00:27:13.005 --> 00:27:15.720 align:middle line:90% 00:27:15.720 --> 00:27:19.920 align:middle line:84% So part one is going to be functions-- 00:27:19.920 --> 00:27:22.400 align:middle line:84% the first thing you usually do in 18.02, 00:27:22.400 --> 00:27:25.890 align:middle line:84% which is functions that take a vector in or multiple variables 00:27:25.890 --> 00:27:26.390 align:middle line:90% in. 00:27:26.390 --> 00:27:30.570 align:middle line:84% But we'll think of it as a vector in and a scalar out. 00:27:30.570 --> 00:27:34.720 align:middle line:90% So we're going to have a scalar. 00:27:34.720 --> 00:27:37.150 align:middle line:90% My output is blue, right? 00:27:37.150 --> 00:27:39.070 align:middle line:90% Yes. 00:27:39.070 --> 00:27:40.800 align:middle line:84% We'll keep the same color scheme, good. 00:27:40.800 --> 00:27:55.928 align:middle line:84% So we're going to have a scalar function f of a vector input x. 00:27:55.928 --> 00:27:57.720 align:middle line:84% And I'll put a little vector sign above it. 00:27:57.720 --> 00:28:04.948 align:middle line:84% I won't always do that, but it's nice to be clear sometimes, 00:28:04.948 --> 00:28:06.490 align:middle line:84% which is a vector, which is a scalar. 00:28:06.490 --> 00:28:10.380 align:middle line:90% So x is going to live in our m. 00:28:10.380 --> 00:28:14.640 align:middle line:84% So this is going to be an m component column vector. 00:28:14.640 --> 00:28:22.130 align:middle line:90% 00:28:22.130 --> 00:28:22.700 align:middle line:90% OK. 00:28:22.700 --> 00:28:26.820 align:middle line:84% And what we want to do is imagine 00:28:26.820 --> 00:28:33.880 align:middle line:84% what happens to the output when you change 00:28:33.880 --> 00:28:37.060 align:middle line:90% the input by a little bit. 00:28:37.060 --> 00:28:49.370 align:middle line:84% So we're going to take f of x plus dx. 00:28:49.370 --> 00:28:52.420 align:middle line:84% Think of this as a really small change. 00:28:52.420 --> 00:28:53.570 align:middle line:90% It's infinitesimal. 00:28:53.570 --> 00:28:55.340 align:middle line:84% We're going to drop anything that 00:28:55.340 --> 00:28:59.040 align:middle line:84% goes like dx squared or anything like that, any higher terms. 00:28:59.040 --> 00:29:01.040 align:middle line:84% Let me just move it to [INAUDIBLE].. 00:29:01.040 --> 00:29:06.190 align:middle line:90% It's black-- minus f of x. 00:29:06.190 --> 00:29:22.830 align:middle line:84% And we're going to define this as f prime of x dx. 00:29:22.830 --> 00:29:27.360 align:middle line:84% So we wanted to note we have an arbitrary change in the inputs. 00:29:27.360 --> 00:29:31.970 align:middle line:84% dx is an arbitrary, very, very small vector. 00:29:31.970 --> 00:29:35.300 align:middle line:84% And we want to ask, what's the change in the output 00:29:35.300 --> 00:29:36.530 align:middle line:90% differential, df? 00:29:36.530 --> 00:29:38.840 align:middle line:84% And the answer is going to be that this 00:29:38.840 --> 00:29:41.480 align:middle line:84% is going to be, for a very small dx, 00:29:41.480 --> 00:29:46.460 align:middle line:84% we can approximate this by a linear operator on dx. 00:29:46.460 --> 00:29:55.110 align:middle line:84% So this is going to be a linear operator, always 00:29:55.110 --> 00:29:57.620 align:middle line:90% going to be a linear operator. 00:29:57.620 --> 00:30:00.320 align:middle line:84% And what is that linear operator do? 00:30:00.320 --> 00:30:06.380 align:middle line:84% This one takes a vector in and gives you a scalar. 00:30:06.380 --> 00:30:09.200 align:middle line:90% So this has to equal a-- 00:30:09.200 --> 00:30:15.110 align:middle line:84% df is a scalar, but dx is a vector. 00:30:15.110 --> 00:30:18.835 align:middle line:84% So what this has to be is it has to be kind of a row vector. 00:30:18.835 --> 00:30:23.340 align:middle line:90% 00:30:23.340 --> 00:30:26.740 align:middle line:84% You can think of it more as a one-row matrix. 00:30:26.740 --> 00:30:30.870 align:middle line:90% 00:30:30.870 --> 00:30:34.680 align:middle line:84% Or there's fancier names for this, like covector 00:30:34.680 --> 00:30:37.890 align:middle line:90% or dual vector. 00:30:37.890 --> 00:30:39.270 align:middle line:90% We won't really use that. 00:30:39.270 --> 00:30:40.770 align:middle line:84% I just want to throw them out there. 00:30:40.770 --> 00:30:44.160 align:middle line:90% 00:30:44.160 --> 00:30:50.458 align:middle line:84% So if you want to take a vector in and take a vector out, 00:30:50.458 --> 00:30:52.000 align:middle line:84% you need to multiply by a row vector. 00:30:52.000 --> 00:30:53.680 align:middle line:84% Another way of thinking about it is you 00:30:53.680 --> 00:30:55.030 align:middle line:90% need to take the dot product-- 00:30:55.030 --> 00:30:55.930 align:middle line:84% ALAN EDELMAN: The vector and the scalar out. 00:30:55.930 --> 00:30:58.138 align:middle line:84% STEVEN G. JOHNSON: The vector and scalar out-- sorry. 00:30:58.138 --> 00:31:00.520 align:middle line:84% You need to multiply it by a one row thing. 00:31:00.520 --> 00:31:02.200 align:middle line:84% Another way of thinking about it is 00:31:02.200 --> 00:31:06.220 align:middle line:84% that, if you have a linear operation that 00:31:06.220 --> 00:31:10.990 align:middle line:84% takes a vector in and gives you a scalar 00:31:10.990 --> 00:31:17.035 align:middle line:84% out, the only type of thing that does that is a dot product. 00:31:17.035 --> 00:31:24.253 align:middle line:90% 00:31:24.253 --> 00:31:26.670 align:middle line:84% If you take a dot product of the vector, you get a scalar. 00:31:26.670 --> 00:31:28.170 align:middle line:84% And that's the only linear operation 00:31:28.170 --> 00:31:31.070 align:middle line:84% that gives you a scalar from a vector in some sense. 00:31:31.070 --> 00:31:33.590 align:middle line:84% And so this is a dot product with some vector. 00:31:33.590 --> 00:31:37.670 align:middle line:84% That vector must be pretty special. 00:31:37.670 --> 00:31:39.720 align:middle line:90% And so we'll give it a name. 00:31:39.720 --> 00:31:44.400 align:middle line:84% And we'll call that the gradient. 00:31:44.400 --> 00:31:47.130 align:middle line:84% So I think this is going to be the gradient of f. 00:31:47.130 --> 00:31:59.580 align:middle line:84% And this is the thing we take the dot product with 00:31:59.580 --> 00:32:04.200 align:middle line:90% to get our scalar df. 00:32:04.200 --> 00:32:09.570 align:middle line:84% So you can think of this in linear algebra terms. 00:32:09.570 --> 00:32:15.470 align:middle line:84% So this dot product is the same thing 00:32:15.470 --> 00:32:18.570 align:middle line:90% as multiplying by a transpose. 00:32:18.570 --> 00:32:21.120 align:middle line:90% So this is the same thing as-- 00:32:21.120 --> 00:32:23.780 align:middle line:90% 00:32:23.780 --> 00:32:32.000 align:middle line:84% this is saying that f prime is really grad f transpose. 00:32:32.000 --> 00:32:35.060 align:middle line:84% Or equivalently, f prime dx is the operation 00:32:35.060 --> 00:32:37.140 align:middle line:84% of a dot product with a gradient. 00:32:37.140 --> 00:32:39.470 align:middle line:84% This is going to be really powerful pretty soon 00:32:39.470 --> 00:32:43.130 align:middle line:84% because it's going to also allow us to generalize gradients 00:32:43.130 --> 00:32:44.810 align:middle line:90% to other kinds of vector spaces. 00:32:44.810 --> 00:32:47.550 align:middle line:84% As long as we have a dot product and a scalar function, 00:32:47.550 --> 00:32:49.050 align:middle line:84% you'll be able to define a gradient. 00:32:49.050 --> 00:32:51.860 align:middle line:84% So if you have a scalar function of-- if this is something that 00:32:51.860 --> 00:32:55.160 align:middle line:84% takes a matrix in and a scalar out, like a determinant, 00:32:55.160 --> 00:32:57.290 align:middle line:84% pretty soon we're going to be able to take 00:32:57.290 --> 00:32:59.090 align:middle line:90% the gradient of a determinant. 00:32:59.090 --> 00:33:02.120 align:middle line:84% We'll be able to define what that means. 00:33:02.120 --> 00:33:03.620 align:middle line:84% ALAN EDELMAN: Just to be clear, what 00:33:03.620 --> 00:33:04.970 align:middle line:84% is on the other side of that equals sign 00:33:04.970 --> 00:33:05.840 align:middle line:90% that you just wrote? 00:33:05.840 --> 00:33:08.872 align:middle line:90% 00:33:08.872 --> 00:33:10.330 align:middle line:84% STEVEN G. JOHNSON: So, yes, the f-- 00:33:10.330 --> 00:33:11.880 align:middle line:90% yeah, I should write that. 00:33:11.880 --> 00:33:12.910 align:middle line:90% That's the f prime. 00:33:12.910 --> 00:33:16.760 align:middle line:90% 00:33:16.760 --> 00:33:17.260 align:middle line:90% Yeah. 00:33:17.260 --> 00:33:18.850 align:middle line:90% So this is-- yeah. 00:33:18.850 --> 00:33:21.230 align:middle line:90% I need to-- my equals-- 00:33:21.230 --> 00:33:21.730 align:middle line:90% let's see. 00:33:21.730 --> 00:33:24.310 align:middle line:84% The gradient of f is the thing we take a dot product. 00:33:24.310 --> 00:33:30.483 align:middle line:84% And here it's f prime of x is this. 00:33:30.483 --> 00:33:31.400 align:middle line:90% ALAN EDELMAN: Exactly. 00:33:31.400 --> 00:33:32.290 align:middle line:90% It's f prime of x. 00:33:32.290 --> 00:33:33.330 align:middle line:90% And it's not df. 00:33:33.330 --> 00:33:34.330 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:33:34.330 --> 00:33:35.920 align:middle line:90% It's not-- no dx. 00:33:35.920 --> 00:33:39.010 align:middle line:84% Just the f prime by itself is a row vector. 00:33:39.010 --> 00:33:42.940 align:middle line:84% That's the transpose of the gradient. 00:33:42.940 --> 00:33:44.950 align:middle line:84% So now, that's the definition of the gradient. 00:33:44.950 --> 00:33:48.130 align:middle line:84% It's only something we're usually 00:33:48.130 --> 00:33:51.728 align:middle line:84% going to define for scalar functions of vectors. 00:33:51.728 --> 00:33:53.770 align:middle line:84% And pretty soon, we'll be able to generalize that 00:33:53.770 --> 00:33:56.530 align:middle line:84% to other kinds of vectors, but will still be a scalar. 00:33:56.530 --> 00:33:58.810 align:middle line:84% And it's the thing you take the dot product 00:33:58.810 --> 00:34:02.620 align:middle line:84% with of dx, of the differential, and the change in the input 00:34:02.620 --> 00:34:06.575 align:middle line:84% with to get the change in the output, OK? 00:34:06.575 --> 00:34:19.710 align:middle line:90% 00:34:19.710 --> 00:34:20.429 align:middle line:90% Yeah. 00:34:20.429 --> 00:34:24.989 align:middle line:84% So we did-- Alan did the example of x transpose x. 00:34:24.989 --> 00:34:26.805 align:middle line:84% Let's do another example just for fun. 00:34:26.805 --> 00:34:30.929 align:middle line:90% 00:34:30.929 --> 00:34:33.405 align:middle line:90% So suppose f of x. 00:34:33.405 --> 00:34:38.980 align:middle line:90% 00:34:38.980 --> 00:34:41.570 align:middle line:90% So x is going to be a vector. 00:34:41.570 --> 00:34:53.210 align:middle line:84% Suppose that x transpose Ax where-- so this is x. 00:34:53.210 --> 00:34:56.550 align:middle line:84% x here is going to have m components. 00:34:56.550 --> 00:35:01.530 align:middle line:84% And so we're going to let A be an m by m matrix. 00:35:01.530 --> 00:35:03.928 align:middle line:84% ALAN EDELMAN: And are you assuming asymmetric or-- 00:35:03.928 --> 00:35:05.220 align:middle line:90% STEVEN G. JOHNSON: No, I'm not. 00:35:05.220 --> 00:35:11.820 align:middle line:84% I won't make it symmetric just for generality, OK? 00:35:11.820 --> 00:35:13.170 align:middle line:90% And so this is going to be-- 00:35:13.170 --> 00:35:16.565 align:middle line:90% 00:35:16.565 --> 00:35:18.038 align:middle line:90% A is not going to be an input. 00:35:18.038 --> 00:35:19.830 align:middle line:84% This is just going to be a constant matrix. 00:35:19.830 --> 00:35:22.910 align:middle line:90% 00:35:22.910 --> 00:35:26.590 align:middle line:84% So when I do my d's, when I change x, A does not change. 00:35:26.590 --> 00:35:28.990 align:middle line:90% 00:35:28.990 --> 00:35:29.490 align:middle line:90% OK. 00:35:29.490 --> 00:35:32.280 align:middle line:84% So let me do it the long way first, 00:35:32.280 --> 00:35:38.430 align:middle line:84% and then we'll try and derive some rules. 00:35:38.430 --> 00:35:40.305 align:middle line:90% So let's do the long way-- 00:35:40.305 --> 00:35:44.000 align:middle line:90% 00:35:44.000 --> 00:35:45.800 align:middle line:84% long way, but still faster than doing it 00:35:45.800 --> 00:35:56.100 align:middle line:84% component by component, faster than the 18.02 component 00:35:56.100 --> 00:35:57.060 align:middle line:90% by component. 00:35:57.060 --> 00:36:03.720 align:middle line:90% 00:36:03.720 --> 00:36:08.040 align:middle line:84% Because I could take the derivative of this with respect 00:36:08.040 --> 00:36:11.370 align:middle line:84% to x1, with respect to x2, with respect to all the components, 00:36:11.370 --> 00:36:14.040 align:middle line:90% and then build up the gradient. 00:36:14.040 --> 00:36:16.380 align:middle line:84% It starts to become really awkward really quickly. 00:36:16.380 --> 00:36:20.250 align:middle line:84% I know it can be tempting when you're faced with new problems 00:36:20.250 --> 00:36:22.080 align:middle line:84% to fall back on what you know, right? 00:36:22.080 --> 00:36:23.910 align:middle line:90% And that's not a bad strategy. 00:36:23.910 --> 00:36:27.060 align:middle line:84% But we really want to encourage you to-- 00:36:27.060 --> 00:36:28.780 align:middle line:84% even though you know calculus really, 00:36:28.780 --> 00:36:31.280 align:middle line:84% really well, you know how to take derivatives really, really 00:36:31.280 --> 00:36:34.440 align:middle line:84% well, 18.02 and 18.01 style, we're 00:36:34.440 --> 00:36:36.570 align:middle line:84% going to try and learn something new here, 00:36:36.570 --> 00:36:39.550 align:middle line:84% a new way that can be really a lot more powerful besides 00:36:39.550 --> 00:36:40.050 align:middle line:90% [INAUDIBLE]. 00:36:40.050 --> 00:36:42.600 align:middle line:84% ALAN EDELMAN: I call it the way for big boys and big girls. 00:36:42.600 --> 00:36:44.445 align:middle line:84% STEVEN G. JOHNSON: Yes, for big kids. 00:36:44.445 --> 00:36:44.760 align:middle line:90% ALAN EDELMAN: For grown-ups. 00:36:44.760 --> 00:36:45.810 align:middle line:84% STEVEN G. JOHNSON: The big-kid way. 00:36:45.810 --> 00:36:46.727 align:middle line:90% ALAN EDELMAN: Kid way. 00:36:46.727 --> 00:36:47.970 align:middle line:90% STEVEN G. JOHNSON: OK. 00:36:47.970 --> 00:36:51.900 align:middle line:90% So df, let's just do it slowly. 00:36:51.900 --> 00:36:54.295 align:middle line:84% So what we want to do is we want to take f. 00:36:54.295 --> 00:36:55.170 align:middle line:90% We're going to take-- 00:36:55.170 --> 00:36:57.075 align:middle line:84% I'm going to draw my vector symbols here. 00:36:57.075 --> 00:36:58.200 align:middle line:90% I guess I'll put them here. 00:36:58.200 --> 00:37:02.280 align:middle line:84% But I get tired of writing them all the time. 00:37:02.280 --> 00:37:06.050 align:middle line:84% All my x's, and therefore my dx's, are vectors. 00:37:06.050 --> 00:37:07.950 align:middle line:84% So think of it as an arbitrary small change 00:37:07.950 --> 00:37:09.180 align:middle line:90% in an arbitrary direction. 00:37:09.180 --> 00:37:12.222 align:middle line:84% We want it to be able to handle anything like that. 00:37:12.222 --> 00:37:13.680 align:middle line:84% ALAN EDELMAN: And in case it wasn't 00:37:13.680 --> 00:37:17.970 align:middle line:84% already obvious to everybody, what is the output of f? 00:37:17.970 --> 00:37:19.785 align:middle line:84% Is it a scalar, a vector, a matrix? 00:37:19.785 --> 00:37:23.150 align:middle line:90% 00:37:23.150 --> 00:37:24.140 align:middle line:90% It's a scalar, exactly. 00:37:24.140 --> 00:37:25.820 align:middle line:84% Just wanted to make sure everybody realized-- 00:37:25.820 --> 00:37:26.600 align:middle line:90% STEVEN G. JOHNSON: Sorry, yes. 00:37:26.600 --> 00:37:29.420 align:middle line:84% ALAN EDELMAN: --that this is a scalar function of a vector. 00:37:29.420 --> 00:37:30.440 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:37:30.440 --> 00:37:37.557 align:middle line:84% You could also write this as x dot product with ax. 00:37:37.557 --> 00:37:42.660 align:middle line:90% 00:37:42.660 --> 00:37:43.920 align:middle line:90% That's the same thing. 00:37:43.920 --> 00:37:44.640 align:middle line:90% OK. 00:37:44.640 --> 00:37:46.620 align:middle line:84% So I'm just going to do this out. 00:37:46.620 --> 00:37:49.590 align:middle line:84% I think it's still a little bit laboriously, 00:37:49.590 --> 00:37:52.662 align:middle line:84% but we'll have a better rule for-- we'll do the product 00:37:52.662 --> 00:37:53.370 align:middle line:90% rule in a minute. 00:37:53.370 --> 00:37:55.243 align:middle line:84% But let's do it without the benefit of that. 00:37:55.243 --> 00:37:56.910 align:middle line:84% ALAN EDELMAN: Because you're effectively 00:37:56.910 --> 00:37:59.237 align:middle line:84% deriving the product rule in what's about to come. 00:37:59.237 --> 00:38:00.570 align:middle line:90% STEVEN G. JOHNSON: Exactly, yes. 00:38:00.570 --> 00:38:01.900 align:middle line:90% So what do we do? 00:38:01.900 --> 00:38:02.825 align:middle line:90% So I'm going to plug-- 00:38:02.825 --> 00:38:04.200 align:middle line:84% I'm going to take f of x plus dx. 00:38:04.200 --> 00:38:05.520 align:middle line:90% I'm going to subtract fx. 00:38:05.520 --> 00:38:06.717 align:middle line:90% And I'm going to drop-- 00:38:06.717 --> 00:38:08.550 align:middle line:84% because it's d's, I'm going to drop anything 00:38:08.550 --> 00:38:12.960 align:middle line:84% that looks like a d squared, a dx squared, something 00:38:12.960 --> 00:38:14.940 align:middle line:90% that goes to 0 faster than dx. 00:38:14.940 --> 00:38:16.440 align:middle line:90% So what's f of x plus dx? 00:38:16.440 --> 00:38:18.480 align:middle line:90% Well, I take x. 00:38:18.480 --> 00:38:21.810 align:middle line:90% I add dx, and I plug it into f. 00:38:21.810 --> 00:38:25.770 align:middle line:90% So there's a transpose there. 00:38:25.770 --> 00:38:32.520 align:middle line:84% There's an A. There's an x plus dx there. 00:38:32.520 --> 00:38:39.690 align:middle line:84% And then I subtract x transpose Ax. 00:38:39.690 --> 00:38:43.230 align:middle line:84% And then I can just multiply everything out. 00:38:43.230 --> 00:38:45.480 align:middle line:84% Just think of dx as a really small vector. 00:38:45.480 --> 00:38:51.780 align:middle line:84% And just use your ordinary rules from linear algebra. 00:38:51.780 --> 00:38:54.510 align:middle line:90% 00:38:54.510 --> 00:38:56.580 align:middle line:84% I can just use the distributed whatever it's 00:38:56.580 --> 00:38:58.647 align:middle line:90% called, the distributive rule. 00:38:58.647 --> 00:39:01.230 align:middle line:84% I just have to make sure I don't change the orders of anything 00:39:01.230 --> 00:39:03.160 align:middle line:84% since these are vectors and matrices. 00:39:03.160 --> 00:39:05.460 align:middle line:90% So I can multiply. 00:39:05.460 --> 00:39:07.873 align:middle line:84% There's this term times this term times this term. 00:39:07.873 --> 00:39:08.790 align:middle line:90% That's the first term. 00:39:08.790 --> 00:39:12.932 align:middle line:90% So there's an x transpose Ax. 00:39:12.932 --> 00:39:15.900 align:middle line:90% 00:39:15.900 --> 00:39:21.960 align:middle line:84% There's also this term times this term times this term. 00:39:21.960 --> 00:39:29.720 align:middle line:90% So that's a dx transpose Ax. 00:39:29.720 --> 00:39:31.220 align:middle line:90% So dx is just a little vector. 00:39:31.220 --> 00:39:34.170 align:middle line:84% It's perfectly fine to transpose it. 00:39:34.170 --> 00:39:37.910 align:middle line:84% And then I also have this term times this term 00:39:37.910 --> 00:39:40.790 align:middle line:90% times this term. 00:39:40.790 --> 00:39:48.134 align:middle line:90% So that's x transpose Adx. 00:39:48.134 --> 00:39:52.770 align:middle line:90% 00:39:52.770 --> 00:39:58.440 align:middle line:84% And then I have this term times this term times this term. 00:39:58.440 --> 00:39:59.700 align:middle line:90% But that has a dx squared. 00:39:59.700 --> 00:40:02.270 align:middle line:90% 00:40:02.270 --> 00:40:05.290 align:middle line:90% And so that I'm going to just-- 00:40:05.290 --> 00:40:08.050 align:middle line:90% 00:40:08.050 --> 00:40:08.563 align:middle line:90% let's see. 00:40:08.563 --> 00:40:10.480 align:middle line:84% Well, I actually am going to [INAUDIBLE] here. 00:40:10.480 --> 00:40:19.180 align:middle line:84% So that term is your dx transpose Adx. 00:40:19.180 --> 00:40:21.688 align:middle line:90% This term is gone. 00:40:21.688 --> 00:40:22.480 align:middle line:90% This is high order. 00:40:22.480 --> 00:40:26.360 align:middle line:90% 00:40:26.360 --> 00:40:30.410 align:middle line:84% So in the limit as dx gets smaller and smaller 00:40:30.410 --> 00:40:32.360 align:middle line:84% and smaller, this term is negligible 00:40:32.360 --> 00:40:35.760 align:middle line:90% compared to these terms. 00:40:35.760 --> 00:40:40.800 align:middle line:84% And then I still have this term over here, can't forget that. 00:40:40.800 --> 00:40:44.850 align:middle line:84% Otherwise, these terms are negligible compared to this. 00:40:44.850 --> 00:40:47.820 align:middle line:84% But it's OK because I'm going to subtract that. 00:40:47.820 --> 00:40:54.040 align:middle line:90% 00:40:54.040 --> 00:40:55.810 align:middle line:90% And that term cancels. 00:40:55.810 --> 00:40:59.160 align:middle line:90% 00:40:59.160 --> 00:41:04.420 align:middle line:84% And what's left is these two terms. 00:41:04.420 --> 00:41:07.510 align:middle line:84% And this is a perfectly good linear operation on dx, 00:41:07.510 --> 00:41:11.460 align:middle line:84% but it's not written in a very nice form. 00:41:11.460 --> 00:41:19.270 align:middle line:84% So the trick is to-- since these are numbers, 00:41:19.270 --> 00:41:21.790 align:middle line:90% I can transpose them. 00:41:21.790 --> 00:41:27.010 align:middle line:84% And this is something I feel like in 18.06 people 00:41:27.010 --> 00:41:32.110 align:middle line:84% get very confused by, that normally you're 00:41:32.110 --> 00:41:34.330 align:middle line:84% not allowed to change the order of anything. 00:41:34.330 --> 00:41:36.970 align:middle line:84% Normally, a matrix is not equal to its transpose. 00:41:36.970 --> 00:41:41.200 align:middle line:84% But a number is always equal to its transpose. 00:41:41.200 --> 00:41:42.850 align:middle line:90% So this is since-- 00:41:42.850 --> 00:41:45.370 align:middle line:90% 00:41:45.370 --> 00:41:47.470 align:middle line:90% let me do a note over here. 00:41:47.470 --> 00:41:52.220 align:middle line:90% 00:41:52.220 --> 00:42:04.160 align:middle line:84% Since dx transpose Ax is a scalar, 00:42:04.160 --> 00:42:08.390 align:middle line:84% a scalar is always equal to its transpose. 00:42:08.390 --> 00:42:12.800 align:middle line:84% So then we can take this and set it 00:42:12.800 --> 00:42:25.490 align:middle line:84% equal to its own transpose, which is the same thing as x 00:42:25.490 --> 00:42:29.690 align:middle line:90% transpose A transpose dx. 00:42:29.690 --> 00:42:33.410 align:middle line:84% And again, make sure you understand that. 00:42:33.410 --> 00:42:37.550 align:middle line:84% This somehow is a source of endless confusion. 00:42:37.550 --> 00:42:39.770 align:middle line:84% Alan already talked about it when 00:42:39.770 --> 00:42:41.870 align:middle line:84% he said, for example, dx transpose 00:42:41.870 --> 00:42:44.090 align:middle line:90% x equals x transpose dx. 00:42:44.090 --> 00:42:46.340 align:middle line:84% That's a little easier to understand because it's just 00:42:46.340 --> 00:42:48.650 align:middle line:90% a dot product. 00:42:48.650 --> 00:42:51.380 align:middle line:84% You can swap things, a scalar product. 00:42:51.380 --> 00:42:53.545 align:middle line:84% So this is also an instance of the same rule. 00:42:53.545 --> 00:42:55.920 align:middle line:84% But you know, it's a little bit more complicated looking. 00:42:55.920 --> 00:42:57.800 align:middle line:90% It's the same idea. 00:42:57.800 --> 00:43:01.750 align:middle line:84% Because this is a number, I can transpose it. 00:43:01.750 --> 00:43:04.620 align:middle line:84% And that swaps everything around. 00:43:04.620 --> 00:43:06.720 align:middle line:90% And the transpose of a product-- 00:43:06.720 --> 00:43:08.520 align:middle line:84% hopefully, you remember from linear algebra 00:43:08.520 --> 00:43:10.260 align:middle line:84% that the transpose of a product is 00:43:10.260 --> 00:43:13.500 align:middle line:84% the product of the transpose in reverse order. 00:43:13.500 --> 00:43:21.510 align:middle line:84% So this term here, this whole term, 00:43:21.510 --> 00:43:26.240 align:middle line:84% equals x transpose A transpose dx. 00:43:26.240 --> 00:43:28.430 align:middle line:84% And what that allows me to do is it allows 00:43:28.430 --> 00:43:30.320 align:middle line:90% me to combine the two terms. 00:43:30.320 --> 00:43:32.420 align:middle line:84% Now, this term and this term, they 00:43:32.420 --> 00:43:34.055 align:middle line:90% both have a dx on the right. 00:43:34.055 --> 00:43:37.340 align:middle line:90% 00:43:37.340 --> 00:43:39.445 align:middle line:84% And so I can put that over in the right. 00:43:39.445 --> 00:43:42.180 align:middle line:90% 00:43:42.180 --> 00:43:44.550 align:middle line:90% And I can put parentheses. 00:43:44.550 --> 00:43:49.050 align:middle line:84% And I have two terms, and both of them 00:43:49.050 --> 00:43:52.420 align:middle line:90% have an x transpose on the left. 00:43:52.420 --> 00:43:54.310 align:middle line:84% And the first term, one of the terms, 00:43:54.310 --> 00:43:57.310 align:middle line:84% has an A. The other term has an A transpose. 00:43:57.310 --> 00:44:01.180 align:middle line:90% 00:44:01.180 --> 00:44:14.580 align:middle line:84% So that means this thing here is our derivative, f prime. 00:44:14.580 --> 00:44:17.220 align:middle line:84% Again, don't get that confused with the differential. 00:44:17.220 --> 00:44:19.197 align:middle line:90% f prime times dx is df. 00:44:19.197 --> 00:44:20.530 align:middle line:90% That's the change in the output. 00:44:20.530 --> 00:44:22.350 align:middle line:84% That's the little change in the output. 00:44:22.350 --> 00:44:24.930 align:middle line:90% f prime is the rate of change. 00:44:24.930 --> 00:44:28.050 align:middle line:84% It's the thing you operate on dx. 00:44:28.050 --> 00:44:30.670 align:middle line:90% This is a row vector. 00:44:30.670 --> 00:44:33.300 align:middle line:84% Notice that this is a row vector. 00:44:33.300 --> 00:44:34.800 align:middle line:84% ALAN EDELMAN: Could I ask the class? 00:44:34.800 --> 00:44:39.790 align:middle line:84% Then quickly tell me what is the gradient of x transpose dx. 00:44:39.790 --> 00:44:43.895 align:middle line:90% Anybody want to shout it out? 00:44:43.895 --> 00:44:44.770 align:middle line:90% What is the gradient? 00:44:44.770 --> 00:44:49.144 align:middle line:90% 00:44:49.144 --> 00:44:50.657 align:middle line:90% AUDIENCE: You [INAUDIBLE]? 00:44:50.657 --> 00:44:51.490 align:middle line:90% ALAN EDELMAN: Right. 00:44:51.490 --> 00:44:53.968 align:middle line:84% Do you want to say it in its full glory? 00:44:53.968 --> 00:44:54.844 align:middle line:90% AUDIENCE: OK. 00:44:54.844 --> 00:44:57.300 align:middle line:84% A plus-- A plus A transpose times x. 00:44:57.300 --> 00:44:58.300 align:middle line:90% ALAN EDELMAN: Good, yup. 00:44:58.300 --> 00:45:00.467 align:middle line:84% A plus A transpose times x is the gradient, exactly. 00:45:00.467 --> 00:45:05.018 align:middle line:90% 00:45:05.018 --> 00:45:06.060 align:middle line:90% STEVEN G. JOHNSON: Right. 00:45:06.060 --> 00:45:07.840 align:middle line:84% It's just the transpose of this thing. 00:45:07.840 --> 00:45:12.890 align:middle line:84% So we transpose this thing. dx goes over on the right. 00:45:12.890 --> 00:45:15.020 align:middle line:90% This thing gets transposed. 00:45:15.020 --> 00:45:18.807 align:middle line:84% But this is symmetric, so it equals itself. 00:45:18.807 --> 00:45:20.390 align:middle line:84% ALAN EDELMAN: So, Steven, on the clock 00:45:20.390 --> 00:45:21.980 align:middle line:90% here, we're already at 12:56. 00:45:21.980 --> 00:45:23.613 align:middle line:90% So you might want to kind of-- 00:45:23.613 --> 00:45:24.530 align:middle line:90% STEVEN G. JOHNSON: OK. 00:45:24.530 --> 00:45:27.650 align:middle line:84% ALAN EDELMAN: --come to a conclusion. 00:45:27.650 --> 00:45:29.420 align:middle line:90% STEVEN G. JOHNSON: Yeah. 00:45:29.420 --> 00:45:32.640 align:middle line:84% So this is just revisiting the notion of a gradient. 00:45:32.640 --> 00:45:38.060 align:middle line:84% And so next time, we're going to continue so that next time, 00:45:38.060 --> 00:45:49.220 align:middle line:84% basically, we're going to do 18.06 revisited part two. 00:45:49.220 --> 00:45:54.440 align:middle line:84% And we're going to have f is now going 00:45:54.440 --> 00:46:10.400 align:middle line:84% to be a vector function that takes a vector of outputs 00:46:10.400 --> 00:46:14.950 align:middle line:84% and also takes a vector of inputs. 00:46:14.950 --> 00:46:17.740 align:middle line:90% 00:46:17.740 --> 00:46:26.350 align:middle line:84% And so we have outputs in, say, Rm inputs in, say, Rn. 00:46:26.350 --> 00:46:29.220 align:middle line:90% 00:46:29.220 --> 00:46:33.990 align:middle line:84% I probably should have used n just to be consistent before. 00:46:33.990 --> 00:46:36.000 align:middle line:84% And then what we're going to find is that then-- 00:46:36.000 --> 00:46:39.270 align:middle line:90% 00:46:39.270 --> 00:46:40.880 align:middle line:90% you can almost do it right now. 00:46:40.880 --> 00:46:52.270 align:middle line:84% df has to be a linear operator that 00:46:52.270 --> 00:46:55.480 align:middle line:84% takes a small change in the input 00:46:55.480 --> 00:46:59.460 align:middle line:84% and gives you a small change in the output. 00:46:59.460 --> 00:47:04.575 align:middle line:84% And so this has to have m outputs, m components here. 00:47:04.575 --> 00:47:06.930 align:middle line:84% It has to have n components there. 00:47:06.930 --> 00:47:09.570 align:middle line:84% The only way you can get a linear operator that 00:47:09.570 --> 00:47:12.240 align:middle line:84% takes n inputs and m outputs is that this 00:47:12.240 --> 00:47:16.120 align:middle line:90% has to be an m by n matrix. 00:47:16.120 --> 00:47:20.073 align:middle line:90% 00:47:20.073 --> 00:47:21.990 align:middle line:84% ALAN EDELMAN: It has to be expressible as an m 00:47:21.990 --> 00:47:22.590 align:middle line:90% by n matrix. 00:47:22.590 --> 00:47:22.770 align:middle line:90% STEVEN G. JOHNSON: Exactly. 00:47:22.770 --> 00:47:25.020 align:middle line:84% ALAN EDELMAN: But you don't have to write down the matrix. 00:47:25.020 --> 00:47:26.562 align:middle line:84% STEVEN G. JOHNSON: Yes, that's right. 00:47:26.562 --> 00:47:36.300 align:middle line:84% So this is our f prime of x linear operator. 00:47:36.300 --> 00:47:41.340 align:middle line:84% And this, if we write it down as a matrix, 00:47:41.340 --> 00:47:43.403 align:middle line:90% we call it the Jacobian. 00:47:43.403 --> 00:47:44.820 align:middle line:84% ALAN EDELMAN: I think what I'll do 00:47:44.820 --> 00:47:47.738 align:middle line:84% next time is show my favorite nonlinear 00:47:47.738 --> 00:47:49.530 align:middle line:84% operator on two-dimensional space, which is 00:47:49.530 --> 00:47:52.260 align:middle line:90% hyperbolic operators on corgis. 00:47:52.260 --> 00:47:55.040 align:middle line:84% So you'll see hyperbolic corgis on Friday if you come. 00:47:55.040 --> 00:47:57.540 align:middle line:84% STEVEN G. JOHNSON: So probably on Friday maybe we'll switch. 00:47:57.540 --> 00:48:01.080 align:middle line:84% And I'll start with the first half and then finish this up. 00:48:01.080 --> 00:48:03.360 align:middle line:84% And then Alan can do the second half. 00:48:03.360 --> 00:48:05.190 align:middle line:84% ALAN EDELMAN: OK, we could do it that way. 00:48:05.190 --> 00:48:05.460 align:middle line:90% OK. 00:48:05.460 --> 00:48:06.345 align:middle line:90% STEVEN G. JOHNSON: Thanks, all. 00:48:06.345 --> 00:48:07.800 align:middle line:90% Any questions at this point? 00:48:07.800 --> 00:48:08.358 align:middle line:90% But let's-- 00:48:08.358 --> 00:48:09.900 align:middle line:84% ALAN EDELMAN: I'll also stick around. 00:48:09.900 --> 00:48:12.456 align:middle line:84% You can ask Steven or, you know-- 00:48:12.456 --> 00:48:14.370 align:middle line:84% STEVEN G. JOHNSON: Let me stop the recording. 00:48:14.370 --> 00:48:16.832 align:middle line:90% 00:48:16.832 --> 00:48:17.540 align:middle line:90% ALAN EDELMAN: OK. 00:48:17.540 --> 00:48:23.140 align:middle line:84% Otherwise, see you on Friday, same room, 11:00 AM. 00:48:23.140 --> 00:48:27.000 align:middle line:90%