WEBVTT 00:00:00.000 --> 00:00:01.940 align:middle line:90% [SQUEAKING] 00:00:01.940 --> 00:00:03.395 align:middle line:90% [RUSTLING] 00:00:03.395 --> 00:00:05.335 align:middle line:90% [CLICKING] 00:00:05.335 --> 00:00:11.620 align:middle line:90% 00:00:11.620 --> 00:00:13.960 align:middle line:84% STEVEN JOHNSON: So Alan mentioned fine differences 00:00:13.960 --> 00:00:17.150 align:middle line:84% as something that automatic differentiation is not doing. 00:00:17.150 --> 00:00:19.540 align:middle line:84% But I do want to talk about fine differences 00:00:19.540 --> 00:00:24.560 align:middle line:84% because they do come up a lot when 00:00:24.560 --> 00:00:26.935 align:middle line:84% you're talking about computing derivatives on a computer. 00:00:26.935 --> 00:00:29.530 align:middle line:90% 00:00:29.530 --> 00:00:31.630 align:middle line:84% And as he mentioned, we're going to spend 00:00:31.630 --> 00:00:34.040 align:middle line:84% a lot of time talking about automatic differentiation, 00:00:34.040 --> 00:00:35.800 align:middle line:84% which is this amazing technology where 00:00:35.800 --> 00:00:39.130 align:middle line:84% you can take a program that computes a function. 00:00:39.130 --> 00:00:42.760 align:middle line:84% And it will basically compute the derivative analytically 00:00:42.760 --> 00:00:43.690 align:middle line:90% in some sense for you. 00:00:43.690 --> 00:00:47.470 align:middle line:84% But how it does it takes a little bit of explanation. 00:00:47.470 --> 00:00:49.360 align:middle line:90% And this is great. 00:00:49.360 --> 00:00:51.910 align:middle line:84% If you're hand-computing derivatives, 00:00:51.910 --> 00:00:54.430 align:middle line:84% even though you might think the rules are simple, 00:00:54.430 --> 00:00:56.260 align:middle line:84% as you see from the homework, when you get 00:00:56.260 --> 00:00:59.350 align:middle line:84% to more complicated functions, especially involving vectors 00:00:59.350 --> 00:01:01.780 align:middle line:84% and matrices, it's quite error-prone. 00:01:01.780 --> 00:01:05.050 align:middle line:84% And it's a common source of bugs if you're 00:01:05.050 --> 00:01:08.230 align:middle line:84% doing numerical optimization route finding sensitivity 00:01:08.230 --> 00:01:11.123 align:middle line:84% analysis-- all of these things on computers 00:01:11.123 --> 00:01:13.040 align:middle line:84% where you might want to differentiate a really 00:01:13.040 --> 00:01:14.370 align:middle line:90% complicated calculation. 00:01:14.370 --> 00:01:17.810 align:middle line:90% So AD is a great alternative. 00:01:17.810 --> 00:01:19.550 align:middle line:90% It's extremely reliable. 00:01:19.550 --> 00:01:24.620 align:middle line:84% If it gives an answer, it's probably correct. 00:01:24.620 --> 00:01:27.110 align:middle line:84% Unfortunately, there's still a lot of cases 00:01:27.110 --> 00:01:30.800 align:middle line:84% where automatic differentiation doesn't work, 00:01:30.800 --> 00:01:33.290 align:middle line:84% where if, for example, it's code that 00:01:33.290 --> 00:01:35.360 align:middle line:84% calls external libraries and other languages 00:01:35.360 --> 00:01:37.670 align:middle line:84% that the automatic differentiation can't handle. 00:01:37.670 --> 00:01:39.650 align:middle line:84% Or even in a given language, usually they 00:01:39.650 --> 00:01:42.710 align:middle line:84% can only handle a subset of the language. 00:01:42.710 --> 00:01:44.660 align:middle line:84% And there's other cases where, even 00:01:44.660 --> 00:01:47.900 align:middle line:84% if automatic differentiation could work, 00:01:47.900 --> 00:01:53.030 align:middle line:84% you need to really give it a little help in order to use it 00:01:53.030 --> 00:01:54.380 align:middle line:90% effectively-- 00:01:54.380 --> 00:01:57.740 align:middle line:84% basically any kind of problem where 00:01:57.740 --> 00:02:00.230 align:middle line:84% you're computing an answer approximately. 00:02:00.230 --> 00:02:02.930 align:middle line:84% Like, for example, you're solving a nonlinear equation 00:02:02.930 --> 00:02:04.940 align:middle line:90% by Newton's method. 00:02:04.940 --> 00:02:07.250 align:middle line:84% If you try to use automatic differentiation, 00:02:07.250 --> 00:02:10.729 align:middle line:84% even if it works, it wastes a lot of effort 00:02:10.729 --> 00:02:13.040 align:middle line:84% basically trying to not only differentiate 00:02:13.040 --> 00:02:14.900 align:middle line:84% your function you're computing but trying 00:02:14.900 --> 00:02:17.902 align:middle line:84% to exactly differentiate the error in your approximation. 00:02:17.902 --> 00:02:20.360 align:middle line:84% So it tries to propagate the derivatives through every step 00:02:20.360 --> 00:02:22.290 align:middle line:90% of the Newton's method. 00:02:22.290 --> 00:02:26.100 align:middle line:84% And so it turns out you can usually do much, 00:02:26.100 --> 00:02:28.310 align:middle line:84% much better if you know what function you're actually 00:02:28.310 --> 00:02:31.010 align:middle line:90% trying to compute. 00:02:31.010 --> 00:02:33.960 align:middle line:84% And we'll talk about that a little bit. 00:02:33.960 --> 00:02:39.050 align:middle line:84% And so in the cases where automatic differentiation fails 00:02:39.050 --> 00:02:43.700 align:middle line:84% completely or where automatic differentiation works 00:02:43.700 --> 00:02:45.320 align:middle line:84% but it's really suboptimal-- like, 00:02:45.320 --> 00:02:49.220 align:middle line:84% you'd like to replace portions of it with a more specialized 00:02:49.220 --> 00:02:51.110 align:middle line:90% method-- 00:02:51.110 --> 00:02:53.610 align:middle line:84% often you only need to give it a little bit of help. 00:02:53.610 --> 00:02:55.040 align:middle line:84% So what you do is you take the one 00:02:55.040 --> 00:02:57.500 align:middle line:84% piece of your program, the one function, 00:02:57.500 --> 00:03:01.940 align:middle line:84% one subroutine that it fails on or where 00:03:01.940 --> 00:03:03.680 align:middle line:90% it's doing a poor job on. 00:03:03.680 --> 00:03:06.020 align:middle line:84% And you do a manual differentiation of that. 00:03:06.020 --> 00:03:09.260 align:middle line:84% But then you let it handle the rest of it, your program. 00:03:09.260 --> 00:03:13.330 align:middle line:84% And AD systems are very good at the chain rule. 00:03:13.330 --> 00:03:15.063 align:middle line:84% They're very good at basically-- if they 00:03:15.063 --> 00:03:16.480 align:middle line:84% know the derivative of each piece, 00:03:16.480 --> 00:03:17.855 align:middle line:84% they can propagate the derivative 00:03:17.855 --> 00:03:19.400 align:middle line:90% through the whole thing. 00:03:19.400 --> 00:03:22.290 align:middle line:84% So on Julia, there's a package called ChainRules for this. 00:03:22.290 --> 00:03:25.030 align:middle line:84% If you're using Python with autograd or Jax, 00:03:25.030 --> 00:03:29.500 align:middle line:84% you do it by defining a custom Jacobian vector product, 00:03:29.500 --> 00:03:31.240 align:middle line:90% or vector Jacobian product. 00:03:31.240 --> 00:03:33.850 align:middle line:84% And we'll talk about these as well. 00:03:33.850 --> 00:03:36.130 align:middle line:84% But the end result is you often find 00:03:36.130 --> 00:03:38.500 align:middle line:84% yourself doing at least some manual derivatives 00:03:38.500 --> 00:03:39.620 align:middle line:90% of complicated things. 00:03:39.620 --> 00:03:42.490 align:middle line:84% And as I said, it's very error-prone. 00:03:42.490 --> 00:03:45.910 align:middle line:84% And so you really need to check it. 00:03:45.910 --> 00:03:48.340 align:middle line:84% If you have your program to compute the derivative 00:03:48.340 --> 00:03:50.410 align:middle line:84% or you're doing homework and you think 00:03:50.410 --> 00:03:53.050 align:middle line:84% you've differentiated some complicated expression 00:03:53.050 --> 00:03:57.040 align:middle line:84% that we've given you, it's a really good idea to check it. 00:03:57.040 --> 00:04:00.970 align:middle line:84% And then the usual way to check it 00:04:00.970 --> 00:04:03.400 align:middle line:84% is using finite difference approximations. 00:04:03.400 --> 00:04:07.710 align:middle line:84% And so remember, when we look at the derivatives, 00:04:07.710 --> 00:04:11.220 align:middle line:84% right, you should think of it-- this is really 00:04:11.220 --> 00:04:15.090 align:middle line:84% a way of computing the small change in the output 00:04:15.090 --> 00:04:20.640 align:middle line:84% df for a small change in the input dx. 00:04:20.640 --> 00:04:22.380 align:middle line:84% And mathematically, we like these d's 00:04:22.380 --> 00:04:24.900 align:middle line:90% to be infinitesimal things. 00:04:24.900 --> 00:04:27.330 align:middle line:84% But they're really-- you can think 00:04:27.330 --> 00:04:31.500 align:middle line:84% of these as an approximation in the case 00:04:31.500 --> 00:04:32.837 align:middle line:90% when you have a finite delta x. 00:04:32.837 --> 00:04:34.920 align:middle line:84% And we actually started in the opposite direction. 00:04:34.920 --> 00:04:41.430 align:middle line:84% We started with imagining a finite small change delta x. 00:04:41.430 --> 00:04:46.640 align:middle line:84% And that gives a finite small change delta f, right? 00:04:46.640 --> 00:04:50.270 align:middle line:84% And in both cases, the linear approximation is f primed. 00:04:50.270 --> 00:04:53.990 align:middle line:84% So it's f-- if it's an infinitesimal change 00:04:53.990 --> 00:04:56.735 align:middle line:84% in the input, then this has exactly f primed 00:04:56.735 --> 00:04:58.940 align:middle line:90% the x, or linearization. 00:04:58.940 --> 00:05:02.330 align:middle line:84% If you have a finite small change dx, 00:05:02.330 --> 00:05:05.660 align:middle line:84% then the change in the output is approximately 00:05:05.660 --> 00:05:09.530 align:middle line:90% the derivative times this. 00:05:09.530 --> 00:05:10.030 align:middle line:90% Right? 00:05:10.030 --> 00:05:15.060 align:middle line:84% And so if you just look at the left-hand side, 00:05:15.060 --> 00:05:15.990 align:middle line:90% this is easy, right? 00:05:15.990 --> 00:05:18.480 align:middle line:84% If you have a function that computes f of x, 00:05:18.480 --> 00:05:22.050 align:middle line:84% you can just pick a very small delta x, 00:05:22.050 --> 00:05:24.810 align:middle line:84% compute f of x plus delta x minus f of x, 00:05:24.810 --> 00:05:25.920 align:middle line:90% take a difference. 00:05:25.920 --> 00:05:30.840 align:middle line:84% And it should be approximately equal to your derivative times 00:05:30.840 --> 00:05:34.110 align:middle line:84% that or operating on that change, delta x, 00:05:34.110 --> 00:05:37.350 align:middle line:84% plus higher-order terms which are negligible if you can 00:05:37.350 --> 00:05:39.060 align:middle line:90% make this delta x small enough. 00:05:39.060 --> 00:05:41.370 align:middle line:84% And we've actually-- Alan has done 00:05:41.370 --> 00:05:44.670 align:middle line:84% a whole bunch of examples of this sort in the notebooks 00:05:44.670 --> 00:05:45.340 align:middle line:90% so far. 00:05:45.340 --> 00:05:49.110 align:middle line:84% But I want to dig into this in a little more detail, right? 00:05:49.110 --> 00:05:53.190 align:middle line:84% So this is a finite difference approximation, right, 00:05:53.190 --> 00:05:58.920 align:middle line:90% for the change in the output. 00:05:58.920 --> 00:06:04.470 align:middle line:84% Very conventionally, if delta x is a scalar, 00:06:04.470 --> 00:06:08.400 align:middle line:84% people often put the delta x on the bottom 00:06:08.400 --> 00:06:12.170 align:middle line:90% there and to think of it as a-- 00:06:12.170 --> 00:06:14.990 align:middle line:90% did I write that in? 00:06:14.990 --> 00:06:15.560 align:middle line:90% Yeah. 00:06:15.560 --> 00:06:16.518 align:middle line:90% I should write that in. 00:06:16.518 --> 00:06:21.117 align:middle line:84% But so if you put the delta x in the bottom there, then-- 00:06:21.117 --> 00:06:22.700 align:middle line:84% because if it's a scalar, you can just 00:06:22.700 --> 00:06:31.610 align:middle line:84% say f prime of x is approximately f of x plus delta 00:06:31.610 --> 00:06:40.020 align:middle line:84% x minus f of x divided by delta x. 00:06:40.020 --> 00:06:43.590 align:middle line:84% So this is probably sort of the more conventional form 00:06:43.590 --> 00:06:45.450 align:middle line:84% to see this kind of approximation 00:06:45.450 --> 00:06:49.500 align:middle line:84% in as an approximation for the derivative 00:06:49.500 --> 00:06:53.130 align:middle line:84% rather than the approximation for the differential, right, 00:06:53.130 --> 00:06:54.720 align:middle line:90% the df. 00:06:54.720 --> 00:06:58.710 align:middle line:84% But we try to push this format here 00:06:58.710 --> 00:07:01.942 align:middle line:84% because this one works even if delta x is not a scalar. 00:07:01.942 --> 00:07:03.150 align:middle line:90% We can't divide by it, right? 00:07:03.150 --> 00:07:06.180 align:middle line:84% You have a vector, for example, or a matrix, or something 00:07:06.180 --> 00:07:06.730 align:middle line:90% like that. 00:07:06.730 --> 00:07:09.030 align:middle line:84% And this is a linear operator. f prime 00:07:09.030 --> 00:07:11.100 align:middle line:84% is a linear operator acting on delta x, right? 00:07:11.100 --> 00:07:13.260 align:middle line:84% So this is still accurate, right? 00:07:13.260 --> 00:07:16.140 align:middle line:84% So if you have a program that computes f, 00:07:16.140 --> 00:07:21.330 align:middle line:84% and then you can easily compute the change in f. 00:07:21.330 --> 00:07:23.540 align:middle line:90% Oops, I computed the-- 00:07:23.540 --> 00:07:28.060 align:middle line:84% the change in f for a small change in the output. 00:07:28.060 --> 00:07:29.670 align:middle line:84% And then you can compare that to what 00:07:29.670 --> 00:07:32.520 align:middle line:84% you think is your derivative that you worked out laboriously 00:07:32.520 --> 00:07:33.300 align:middle line:90% by hand maybe. 00:07:33.300 --> 00:07:36.360 align:middle line:84% Or if you're lucky, you have an automatic differentiation. 00:07:36.360 --> 00:07:37.920 align:middle line:90% You operate that on delta x. 00:07:37.920 --> 00:07:40.410 align:middle line:84% You should get approximately the same thing. 00:07:40.410 --> 00:07:43.050 align:middle line:84% So this formula here, as it says here-- 00:07:43.050 --> 00:07:44.730 align:middle line:84% this is called a forward difference. 00:07:44.730 --> 00:07:46.230 align:middle line:84% It's called "forward" because you 00:07:46.230 --> 00:07:50.010 align:middle line:84% kind of perturb x in the plus direction 00:07:50.010 --> 00:07:52.350 align:middle line:90% if delta x is positive. 00:07:52.350 --> 00:07:52.950 align:middle line:90% Right? 00:07:52.950 --> 00:07:56.400 align:middle line:84% You can also do a backwards difference-- 00:07:56.400 --> 00:07:59.410 align:middle line:84% f of x minus f of x minus delta x, 00:07:59.410 --> 00:08:04.140 align:middle line:84% which is also approximately f prime of x dx. 00:08:04.140 --> 00:08:06.530 align:middle line:84% It doesn't really matter which one you use. 00:08:06.530 --> 00:08:08.955 align:middle line:84% So that-- but if I tell you "forward difference," 00:08:08.955 --> 00:08:11.580 align:middle line:84% the immediate question is like, what's the backward difference? 00:08:11.580 --> 00:08:11.910 align:middle line:90% It's just-- 00:08:11.910 --> 00:08:13.327 align:middle line:84% ALAN EDELMAN: Can I just point out 00:08:13.327 --> 00:08:14.940 align:middle line:84% that the backwards difference is not 00:08:14.940 --> 00:08:17.490 align:middle line:84% the same sense of "backwards" as in backward mode 00:08:17.490 --> 00:08:19.050 align:middle line:84% differentiation, or reverse mode? 00:08:19.050 --> 00:08:19.770 align:middle line:90% STEVEN JOHNSON: Yes, yes. 00:08:19.770 --> 00:08:21.770 align:middle line:84% We're going to talk about-- forward and backward 00:08:21.770 --> 00:08:24.090 align:middle line:84% mode differentiation-- those are totally 00:08:24.090 --> 00:08:27.920 align:middle line:84% unrelated to forward and backward differences. 00:08:27.920 --> 00:08:30.040 align:middle line:90% So yeah. 00:08:30.040 --> 00:08:32.017 align:middle line:84% And as I said, there's a lot of forms. 00:08:32.017 --> 00:08:34.600 align:middle line:84% And we'll talk about maybe some more sophisticated forms of it 00:08:34.600 --> 00:08:37.600 align:middle line:84% than forward and backward differences. 00:08:37.600 --> 00:08:41.020 align:middle line:84% You can do them-- the ones that give you higher 00:08:41.020 --> 00:08:43.690 align:middle line:90% accuracy for a given delta x. 00:08:43.690 --> 00:08:46.010 align:middle line:84% But they're generally a last resort 00:08:46.010 --> 00:08:47.530 align:middle line:90% as a computational scheme. 00:08:47.530 --> 00:08:50.830 align:middle line:84% But they're a first resort as a quick check, right? 00:08:50.830 --> 00:08:52.570 align:middle line:84% They're really, really easy to do-- 00:08:52.570 --> 00:08:54.700 align:middle line:90% really kind of braindead. 00:08:54.700 --> 00:08:59.080 align:middle line:84% And they're hard to get too wrong. 00:08:59.080 --> 00:09:04.680 align:middle line:84% So if this is a good match for your f prime of x dx, 00:09:04.680 --> 00:09:07.000 align:middle line:90% that's a really good sign. 00:09:07.000 --> 00:09:08.140 align:middle line:90% Right? 00:09:08.140 --> 00:09:11.095 align:middle line:84% So let's just start out with an example. 00:09:11.095 --> 00:09:12.970 align:middle line:84% And this is an example we've seen a few times 00:09:12.970 --> 00:09:15.910 align:middle line:84% in the class, which is f of A. So A is now 00:09:15.910 --> 00:09:17.890 align:middle line:90% going to be a square matrix. 00:09:17.890 --> 00:09:21.500 align:middle line:90% And f of A is just A-squared OK? 00:09:21.500 --> 00:09:24.020 align:middle line:84% And we saw from the product rule several times 00:09:24.020 --> 00:09:30.940 align:middle line:84% that df is A dA plus dA A, which is not 00:09:30.940 --> 00:09:35.680 align:middle line:84% the same thing as 2AdA, the first-year calculus 00:09:35.680 --> 00:09:36.790 align:middle line:90% rule for scalars. 00:09:36.790 --> 00:09:39.970 align:middle line:84% Because A and delta A don't commute. 00:09:39.970 --> 00:09:42.000 align:middle line:90% We've seen this many times. 00:09:42.000 --> 00:09:42.500 align:middle line:90% Right? 00:09:42.500 --> 00:09:44.950 align:middle line:90% So f prime here is now-- 00:09:44.950 --> 00:09:48.370 align:middle line:84% the derivative is really a linear operator that gives 00:09:48.370 --> 00:09:52.840 align:middle line:84% a small change in the input, gives you-- to first order, 00:09:52.840 --> 00:09:56.560 align:middle line:84% to the linear part of the change in the output-- 00:09:56.560 --> 00:10:00.220 align:middle line:90% A delta A plus delta A A. Right? 00:10:00.220 --> 00:10:03.430 align:middle line:90% And what we can do is just-- 00:10:03.430 --> 00:10:04.850 align:middle line:84% suppose I've derived this formula 00:10:04.850 --> 00:10:07.300 align:middle line:84% and I want to check that I didn't screw it up. 00:10:07.300 --> 00:10:08.510 align:middle line:90% This one's pretty simple. 00:10:08.510 --> 00:10:10.052 align:middle line:84% I'm pretty sure I didn't screw it up, 00:10:10.052 --> 00:10:12.200 align:middle line:90% but we want to check this. 00:10:12.200 --> 00:10:16.710 align:middle line:84% So what you do is-- so let's define this function, f of A 00:10:16.710 --> 00:10:18.660 align:middle line:90% equals A-squared. 00:10:18.660 --> 00:10:19.160 align:middle line:90% Right? 00:10:19.160 --> 00:10:24.170 align:middle line:84% We'll try it for a random input and a random small perturbation 00:10:24.170 --> 00:10:27.290 align:middle line:84% delta A. So I'll define that function, 00:10:27.290 --> 00:10:30.410 align:middle line:90% define a random 4-by-4 matrix. 00:10:30.410 --> 00:10:32.270 align:middle line:84% randn is Gaussian random numbers. 00:10:32.270 --> 00:10:35.300 align:middle line:84% They can be plus, or minus, or kind of bell curve-distributed. 00:10:35.300 --> 00:10:36.630 align:middle line:90% So some of them are positive. 00:10:36.630 --> 00:10:37.713 align:middle line:90% Some of them are negative. 00:10:37.713 --> 00:10:41.110 align:middle line:90% But they're all of order one. 00:10:41.110 --> 00:10:41.610 align:middle line:90% All right? 00:10:41.610 --> 00:10:43.893 align:middle line:90% And then delta A, or d-- 00:10:43.893 --> 00:10:45.310 align:middle line:84% I'll just call it, right, dA here. 00:10:45.310 --> 00:10:48.400 align:middle line:84% But it's really a delta A. It's a finite change. 00:10:48.400 --> 00:10:51.028 align:middle line:84% It's also going to be a different random matrix, 00:10:51.028 --> 00:10:52.320 align:middle line:90% but I'm going to make it small. 00:10:52.320 --> 00:10:55.390 align:middle line:84% And I'm going to multiply it by a 10 to the minus 8th. 00:10:55.390 --> 00:10:57.820 align:middle line:84% And we'll talk very soon about how 00:10:57.820 --> 00:10:59.197 align:middle line:90% small do you want to make it. 00:10:59.197 --> 00:11:01.030 align:middle line:84% Do we want to make it 10 to the minus 100th? 00:11:01.030 --> 00:11:02.697 align:middle line:84% Do we want to make it 10 to the minus 1? 00:11:02.697 --> 00:11:06.110 align:middle line:84% Where does this-- how do we choose this, right? 00:11:06.110 --> 00:11:09.100 align:middle line:84% So this is going to be a small change. 00:11:09.100 --> 00:11:12.520 align:middle line:84% So it's random numbers of order one times 10 to the minus 8. 00:11:12.520 --> 00:11:15.820 align:middle line:84% So they're all kind of ordered 10 to minus 8, 10 to minus 9. 00:11:15.820 --> 00:11:18.460 align:middle line:84% It's different random numbers than in A. 00:11:18.460 --> 00:11:20.920 align:middle line:90% So these will not commute. 00:11:20.920 --> 00:11:22.930 align:middle line:90% Should we check that, actually? 00:11:22.930 --> 00:11:32.510 align:middle line:84% Let's do A times dA minus dA times A. Right? 00:11:32.510 --> 00:11:34.590 align:middle line:84% And you can see that this is not 0. 00:11:34.590 --> 00:11:36.800 align:middle line:84% And you can say, oh, these are kind of small numbers 00:11:36.800 --> 00:11:41.885 align:middle line:84% but remember that the dA was a small number of order 10 00:11:41.885 --> 00:11:42.510 align:middle line:90% to the minus 8. 00:11:42.510 --> 00:11:44.210 align:middle line:84% And so these are also small numbers. 00:11:44.210 --> 00:11:47.270 align:middle line:84% But they're small the same order as dA, right? 00:11:47.270 --> 00:11:50.820 align:middle line:84% So these are not actually small compared to dA. 00:11:50.820 --> 00:11:56.890 align:middle line:84% So this is not roundoff errors or something else. 00:11:56.890 --> 00:11:57.390 align:middle line:90% OK? 00:11:57.390 --> 00:12:00.060 align:middle line:84% So now our finite difference approximation-- 00:12:00.060 --> 00:12:03.360 align:middle line:84% we just said it's just going to be f of A plus dA minus f of A. 00:12:03.360 --> 00:12:06.060 align:middle line:84% It's just the most simple, stupid thing directly from 00:12:06.060 --> 00:12:08.310 align:middle line:84% the definition of the derivative, 00:12:08.310 --> 00:12:11.640 align:middle line:90% but plugging in instead of-- 00:12:11.640 --> 00:12:14.250 align:middle line:84% we're plugging in a non-infinitesimal change 00:12:14.250 --> 00:12:15.330 align:middle line:90% in dA-- 00:12:15.330 --> 00:12:17.370 align:middle line:84% pretty small but non-infinitesimal. 00:12:17.370 --> 00:12:19.680 align:middle line:84% So we get this approximate change. 00:12:19.680 --> 00:12:24.570 align:middle line:84% It's small, but it's of the same order as delta A, right? 00:12:24.570 --> 00:12:29.910 align:middle line:84% So this-- and so the exact derivative, again, 00:12:29.910 --> 00:12:34.200 align:middle line:84% is going to be A times dA plus dA times A. And again, 00:12:34.200 --> 00:12:37.350 align:middle line:84% this is small, but it's of the same-- 00:12:37.350 --> 00:12:39.120 align:middle line:84% it's small because delta A is small. 00:12:39.120 --> 00:12:40.825 align:middle line:90% This is the df, right? 00:12:40.825 --> 00:12:41.950 align:middle line:90% This is not the derivative. 00:12:41.950 --> 00:12:48.720 align:middle line:84% This is, I should say, maybe the directional differential. 00:12:48.720 --> 00:12:51.500 align:middle line:90% 00:12:51.500 --> 00:12:52.260 align:middle line:90% Right? 00:12:52.260 --> 00:12:52.760 align:middle line:90% Right? 00:12:52.760 --> 00:12:55.160 align:middle line:90% So it's not the derivative. 00:12:55.160 --> 00:12:58.160 align:middle line:84% It's the derivative operator acting on dA, right? 00:12:58.160 --> 00:13:02.330 align:middle line:84% And you can already see, just to the eye, this kind of matches, 00:13:02.330 --> 00:13:02.900 align:middle line:90% right? 00:13:02.900 --> 00:13:04.820 align:middle line:84% It looks like it's pretty good, right? 00:13:04.820 --> 00:13:10.280 align:middle line:84% There's the number and so forth, right? 00:13:10.280 --> 00:13:15.100 align:middle line:90% If we do 2A dA-- 00:13:15.100 --> 00:13:18.070 align:middle line:84% just a little bit smaller so I can see this. 00:13:18.070 --> 00:13:19.580 align:middle line:90% OK? 00:13:19.580 --> 00:13:21.560 align:middle line:90% If I did 2A dA-- 00:13:21.560 --> 00:13:23.870 align:middle line:84% if I pretended they commuted, right, 00:13:23.870 --> 00:13:29.390 align:middle line:84% and I wrote A times dA plus dA times A as 2A dA, 00:13:29.390 --> 00:13:33.340 align:middle line:84% you get answers that are completely different. 00:13:33.340 --> 00:13:36.970 align:middle line:84% And in fact, these look exactly the same as these. 00:13:36.970 --> 00:13:39.820 align:middle line:84% But that's because Julia's only printing out 00:13:39.820 --> 00:13:41.242 align:middle line:90% six digits of this. 00:13:41.242 --> 00:13:43.450 align:middle line:84% And actually, there are more digits in these numbers. 00:13:43.450 --> 00:13:45.650 align:middle line:84% It's just not showing them by default. 00:13:45.650 --> 00:13:56.260 align:middle line:84% So it might be useful to look at the exact answer minus-- 00:13:56.260 --> 00:14:00.720 align:middle line:84% or let's see, the approximate answer minus the exact answer. 00:14:00.720 --> 00:14:01.220 align:middle line:90% Right? 00:14:01.220 --> 00:14:04.530 align:middle line:84% So the approximate is the finite difference. 00:14:04.530 --> 00:14:08.940 align:middle line:84% The exact is this formula A dA plus dA A 00:14:08.940 --> 00:14:13.450 align:middle line:84% that we derive from our derivative operator, right? 00:14:13.450 --> 00:14:14.902 align:middle line:90% And the errors are really small. 00:14:14.902 --> 00:14:16.110 align:middle line:90% They're 10 to the minus 16th. 00:14:16.110 --> 00:14:21.720 align:middle line:84% So they're small compared to these numbers, 00:14:21.720 --> 00:14:23.040 align:middle line:90% 10 to the minus 8th. 00:14:23.040 --> 00:14:31.890 align:middle line:84% So this brings us into to a key question 00:14:31.890 --> 00:14:33.640 align:middle line:84% when you're talking about errors. 00:14:33.640 --> 00:14:37.110 align:middle line:84% And just in general in mathematics, or science, 00:14:37.110 --> 00:14:41.520 align:middle line:84% or engineering, whenever someone tells you something is small 00:14:41.520 --> 00:14:44.400 align:middle line:84% or something is large, you should always 00:14:44.400 --> 00:14:46.980 align:middle line:90% ask, compared to what, right? 00:14:46.980 --> 00:14:50.160 align:middle line:84% It's a meaningless question to say, for example, 00:14:50.160 --> 00:14:54.810 align:middle line:84% is 1 meter-- is that a small distance? 00:14:54.810 --> 00:15:00.690 align:middle line:84% Well, not compared to the radius of a hydrogen atom. 00:15:00.690 --> 00:15:05.040 align:middle line:84% That's a really long distance for an electron to move away 00:15:05.040 --> 00:15:06.420 align:middle line:90% from a hydrogen atom. 00:15:06.420 --> 00:15:09.090 align:middle line:84% But 1 meter is a tiny distance compared 00:15:09.090 --> 00:15:11.520 align:middle line:90% to the radius of the galaxy. 00:15:11.520 --> 00:15:13.980 align:middle line:84% It might as well be 0 if you're an astronomer, right? 00:15:13.980 --> 00:15:16.520 align:middle line:84% And so it really matters what you compare it to. 00:15:16.520 --> 00:15:18.000 align:middle line:90% We need a yardstick. 00:15:18.000 --> 00:15:22.903 align:middle line:84% And a typical yardstick is, we want to have some measure-- 00:15:22.903 --> 00:15:24.320 align:middle line:84% so I'm going to talk about a norm. 00:15:24.320 --> 00:15:27.200 align:middle line:84% But if these were vectors, you'd know what the norm is. 00:15:27.200 --> 00:15:28.560 align:middle line:90% It's the length of the vector. 00:15:28.560 --> 00:15:32.760 align:middle line:84% So the distance from approximate to exact 00:15:32.760 --> 00:15:36.380 align:middle line:90% is the size of this error. 00:15:36.380 --> 00:15:38.367 align:middle line:84% But I need to compare it to something 00:15:38.367 --> 00:15:39.950 align:middle line:84% to decide whether it's large or small. 00:15:39.950 --> 00:15:41.450 align:middle line:84% And a natural thing to compare it to 00:15:41.450 --> 00:15:43.490 align:middle line:90% is the size of the exact answer. 00:15:43.490 --> 00:15:48.740 align:middle line:84% Because the exact answer is that f prime of A dA, right? 00:15:48.740 --> 00:15:51.090 align:middle line:84% It's a small number because dA is small. 00:15:51.090 --> 00:15:55.130 align:middle line:84% So you really want to know whether this error is small 00:15:55.130 --> 00:15:55.898 align:middle line:90% compared to that. 00:15:55.898 --> 00:15:58.190 align:middle line:84% So for example-- and we can see by eye they are, right? 00:15:58.190 --> 00:16:02.030 align:middle line:84% These errors are on the order of 10 to the minus 16th. 00:16:02.030 --> 00:16:05.420 align:middle line:84% And the exact answer is on the order of 10 to the minus 8th. 00:16:05.420 --> 00:16:07.370 align:middle line:90% So this is good, right? 00:16:07.370 --> 00:16:10.970 align:middle line:84% But to make it quantitative, we define this ratio. 00:16:10.970 --> 00:16:12.800 align:middle line:84% And we call it the relative error, right? 00:16:12.800 --> 00:16:18.380 align:middle line:84% So it's the difference between what we want and what-- 00:16:18.380 --> 00:16:21.090 align:middle line:84% the difference between what we want, 00:16:21.090 --> 00:16:24.670 align:middle line:84% which is the exact answer, and what we get, 00:16:24.670 --> 00:16:28.200 align:middle line:84% which is the approximate answer, and divided 00:16:28.200 --> 00:16:30.870 align:middle line:90% by the size of the exact thing. 00:16:30.870 --> 00:16:31.410 align:middle line:90% OK? 00:16:31.410 --> 00:16:36.680 align:middle line:84% So this norm is defined in Julia in a package called 00:16:36.680 --> 00:16:38.210 align:middle line:90% the LinearAlgebra package. 00:16:38.210 --> 00:16:40.550 align:middle line:84% And I can define a function relative_error 00:16:40.550 --> 00:16:43.280 align:middle line:84% that takes approx, exact, and computes exactly this-- 00:16:43.280 --> 00:16:45.230 align:middle line:84% the norm of the difference divided 00:16:45.230 --> 00:16:47.760 align:middle line:90% by the norm of the exact answer. 00:16:47.760 --> 00:16:50.000 align:middle line:84% And if I just compare it here, we 00:16:50.000 --> 00:16:52.830 align:middle line:84% see actually it's what we expect. 00:16:52.830 --> 00:16:55.050 align:middle line:84% The difference, we said, was on the order of 10 00:16:55.050 --> 00:16:55.970 align:middle line:90% to the minus 16th. 00:16:55.970 --> 00:16:58.220 align:middle line:84% The exact was on the order of the 10 to the minus 8th. 00:16:58.220 --> 00:17:00.620 align:middle line:84% The ratio was on the order of 10 to the minus 8th. 00:17:00.620 --> 00:17:04.280 align:middle line:84% And indeed, what we expected by eye 00:17:04.280 --> 00:17:08.347 align:middle line:84% is what we get here-- that it's about 9 times e to the minus 9. 00:17:08.347 --> 00:17:09.680 align:middle line:90% It's about 10 to the minus 8th-- 00:17:09.680 --> 00:17:13.960 align:middle line:84% so about eight significant digits, right? 00:17:13.960 --> 00:17:19.400 align:middle line:90% So as I said, if we do the-- 00:17:19.400 --> 00:17:24.859 align:middle line:84% and let's do also the relative error of our stupid-- 00:17:24.859 --> 00:17:26.119 align:middle line:90% of our incorrect way. 00:17:26.119 --> 00:17:28.460 align:middle line:90% If I do 2A dA-- 00:17:28.460 --> 00:17:34.205 align:middle line:84% if I use that as my exact formula, 00:17:34.205 --> 00:17:36.800 align:middle line:84% and if I had done my derivative incorrectly, 00:17:36.800 --> 00:17:39.650 align:middle line:84% right-- suppose I thought my exact formula was 2 A dA. 00:17:39.650 --> 00:17:43.190 align:middle line:84% Then I'd get a relative error of 0.6 of order one. 00:17:43.190 --> 00:17:46.340 align:middle line:84% So I really would have caught it and said, oh, this-- 00:17:46.340 --> 00:17:50.000 align:middle line:84% you notice immediately that this relative error is huge. 00:17:50.000 --> 00:17:56.790 align:middle line:84% Even though the 2A dA is small because dA is small, 00:17:56.790 --> 00:18:00.630 align:middle line:84% the difference is not small compared to the correct answer. 00:18:00.630 --> 00:18:04.075 align:middle line:84% So you immediately notice something is wrong here-- 00:18:04.075 --> 00:18:06.450 align:middle line:84% either have a bug in my finite difference approximation-- 00:18:06.450 --> 00:18:08.260 align:middle line:84% but that's really hard to get wrong. 00:18:08.260 --> 00:18:12.450 align:middle line:84% It's just f of A plus dA minus f of A, right? 00:18:12.450 --> 00:18:14.820 align:middle line:90% It's usually dead simple. 00:18:14.820 --> 00:18:16.620 align:middle line:90% So you usually get it right. 00:18:16.620 --> 00:18:18.540 align:middle line:84% Or I have a bug in my derivative, which is 00:18:18.540 --> 00:18:20.950 align:middle line:90% really, really common, right? 00:18:20.950 --> 00:18:23.880 align:middle line:90% So this is good. 00:18:23.880 --> 00:18:24.880 align:middle line:90% So you get a good match. 00:18:24.880 --> 00:18:28.420 align:middle line:84% This isn't a proof that something is correct, 00:18:28.420 --> 00:18:33.970 align:middle line:84% but it's a pretty big hint that you got something 00:18:33.970 --> 00:18:35.230 align:middle line:90% basically fundamentally right. 00:18:35.230 --> 00:18:41.530 align:middle line:84% It's pretty unlikely, if you had a wrong formula, 00:18:41.530 --> 00:18:44.140 align:middle line:84% if you picked a random input and a random displacement, 00:18:44.140 --> 00:18:47.650 align:middle line:84% that's going to give you the correct answer to eight digits, 00:18:47.650 --> 00:18:48.240 align:middle line:90% right? 00:18:48.240 --> 00:18:54.040 align:middle line:84% And so it's a really good way to catch, really, screw ups. 00:18:54.040 --> 00:18:57.190 align:middle line:84% Oh, I actually wrote this relative error down there. 00:18:57.190 --> 00:18:59.300 align:middle line:90% Let me just-- yeah. 00:18:59.300 --> 00:19:06.300 align:middle line:84% So here's my-- so if I really screwed up, I'm actually-- 00:19:06.300 --> 00:19:10.470 align:middle line:84% I would be checking it against the approximate answer. 00:19:10.470 --> 00:19:12.000 align:middle line:84% But yeah, if I really screwed up, 00:19:12.000 --> 00:19:13.470 align:middle line:90% you'd notice immediately, right? 00:19:13.470 --> 00:19:17.830 align:middle line:90% So it's over 50%, right? 00:19:17.830 --> 00:19:18.815 align:middle line:90% Right? 00:19:18.815 --> 00:19:21.513 align:middle line:84% And now I need to go back here and explain something 00:19:21.513 --> 00:19:22.680 align:middle line:90% that I kind of glossed over. 00:19:22.680 --> 00:19:29.190 align:middle line:84% So you all know what these vertical bars are for a vector, 00:19:29.190 --> 00:19:29.690 align:middle line:90% right? 00:19:29.690 --> 00:19:31.405 align:middle line:84% And so the norm of a vector is just 00:19:31.405 --> 00:19:33.780 align:middle line:84% the square root of the sum of the squares of the entries. 00:19:33.780 --> 00:19:35.310 align:middle line:90% It's the length of the vector. 00:19:35.310 --> 00:19:37.227 align:middle line:84% And that's one of the first things you learned 00:19:37.227 --> 00:19:39.740 align:middle line:84% about a vector, is that it has a direction 00:19:39.740 --> 00:19:41.160 align:middle line:90% and then has a length. 00:19:41.160 --> 00:19:44.450 align:middle line:84% That's probably the first definition of a vector 00:19:44.450 --> 00:19:46.020 align:middle line:90% that you heard. 00:19:46.020 --> 00:19:49.310 align:middle line:84% But now, in linear algebra, we generalized our notion 00:19:49.310 --> 00:19:51.320 align:middle line:84% of a vector to a vector space-- things 00:19:51.320 --> 00:19:54.680 align:middle line:84% you can add, and subtract, and multiply by scalars. 00:19:54.680 --> 00:19:58.260 align:middle line:84% And so for example, in this example, 00:19:58.260 --> 00:20:01.590 align:middle line:84% we're using a matrix as a kind of vector. 00:20:01.590 --> 00:20:05.010 align:middle line:84% So we're thinking the inputs are matrices, which we can add, 00:20:05.010 --> 00:20:06.950 align:middle line:90% subtract, multiply by scalars. 00:20:06.950 --> 00:20:13.990 align:middle line:84% And you really need to not only be able to add, subtract, 00:20:13.990 --> 00:20:17.640 align:middle line:84% and multiply by scalars to talk about errors, and derivatives, 00:20:17.640 --> 00:20:18.660 align:middle line:90% and magnitudes. 00:20:18.660 --> 00:20:21.300 align:middle line:84% You really need to also-- a notion of a length. 00:20:21.300 --> 00:20:23.520 align:middle line:84% Or the fancy word in linear algebra for that 00:20:23.520 --> 00:20:25.660 align:middle line:90% is a "norm" of a vector. 00:20:25.660 --> 00:20:30.340 align:middle line:84% And so we need to be able to-- we need a norm of our matrix. 00:20:30.340 --> 00:20:32.590 align:middle line:90% And so what would that mean? 00:20:32.590 --> 00:20:35.790 align:middle line:84% What's the norm-- we know what the norm of a vector is. 00:20:35.790 --> 00:20:39.650 align:middle line:84% You probably at least learned the Euclidean norm. 00:20:39.650 --> 00:20:41.470 align:middle line:90% So if you have a matrix-- 00:20:41.470 --> 00:20:44.050 align:middle line:90% say, a 3-by-3 matrix-- 00:20:44.050 --> 00:20:45.610 align:middle line:90% or is this 4-by-4? 00:20:45.610 --> 00:20:48.400 align:middle line:90% These are 4-by-4 matrix, right? 00:20:48.400 --> 00:20:51.920 align:middle line:90% What does that norm mean? 00:20:51.920 --> 00:20:55.953 align:middle line:84% And it turns out there are multiple choices. 00:20:55.953 --> 00:20:57.370 align:middle line:84% And it turns out it doesn't really 00:20:57.370 --> 00:21:00.800 align:middle line:84% matter too much for our purposes which one we choose. 00:21:00.800 --> 00:21:05.965 align:middle line:84% So I'm going to focus on the simplest one and kind 00:21:05.965 --> 00:21:08.230 align:middle line:84% of the most familiar one, which is basically-- 00:21:08.230 --> 00:21:11.030 align:middle line:84% this is the same as your Euclidean norm of vectors, 00:21:11.030 --> 00:21:11.530 align:middle line:90% right? 00:21:11.530 --> 00:21:15.965 align:middle line:84% So for a Euclidean norm of a, right, column vector, 00:21:15.965 --> 00:21:16.590 align:middle line:90% what do you do? 00:21:16.590 --> 00:21:19.700 align:middle line:84% You take the square root of the sum 00:21:19.700 --> 00:21:21.980 align:middle line:90% of the squares of the entries. 00:21:21.980 --> 00:21:24.950 align:middle line:84% And this is also called an L2 norm, right? 00:21:24.950 --> 00:21:27.080 align:middle line:84% Or in linear algebra notation, I could write it 00:21:27.080 --> 00:21:32.340 align:middle line:84% as x transpose x is my dot product with itself. 00:21:32.340 --> 00:21:34.600 align:middle line:90% And then I take the square root. 00:21:34.600 --> 00:21:36.690 align:middle line:90% So for a matrix-- 00:21:36.690 --> 00:21:39.150 align:middle line:84% at the simplest level, you could think of it 00:21:39.150 --> 00:21:41.550 align:middle line:84% as just a big bag full of numbers, right? 00:21:41.550 --> 00:21:43.980 align:middle line:84% The most obvious choice of a norm 00:21:43.980 --> 00:21:47.700 align:middle line:84% is you do exactly the same process to the matrix entries. 00:21:47.700 --> 00:21:50.340 align:middle line:84% You think of the matrix entries as the components 00:21:50.340 --> 00:21:51.300 align:middle line:90% of the matrix. 00:21:51.300 --> 00:21:53.730 align:middle line:84% And you take the square root of the sum of the squares 00:21:53.730 --> 00:21:54.670 align:middle line:90% of those entries. 00:21:54.670 --> 00:21:58.800 align:middle line:84% And this is, in fact, called a matrix form, a very famous one 00:21:58.800 --> 00:22:00.890 align:middle line:90% called the Frobenius norm. 00:22:00.890 --> 00:22:03.450 align:middle line:90% And in Julia, you call-- 00:22:03.450 --> 00:22:04.210 align:middle line:90% oops-- norm(A). 00:22:04.210 --> 00:22:06.730 align:middle line:90% 00:22:06.730 --> 00:22:08.500 align:middle line:84% You call the norm function on a matrix. 00:22:08.500 --> 00:22:09.820 align:middle line:90% That's what it does by default. 00:22:09.820 --> 00:22:12.820 align:middle line:84% And we can use the Symbolics package to just see 00:22:12.820 --> 00:22:14.560 align:middle line:90% what it's doing symbolically. 00:22:14.560 --> 00:22:16.330 align:middle line:84% So that's what Alan has been using there. 00:22:16.330 --> 00:22:24.250 align:middle line:84% So if I define a matrix full of variables-- 00:22:24.250 --> 00:22:27.460 align:middle line:84% and maybe I should use koala bears or something like that. 00:22:27.460 --> 00:22:31.900 align:middle line:90% But this works. 00:22:31.900 --> 00:22:33.790 align:middle line:84% I think actually I can do an even nicer one. 00:22:33.790 --> 00:22:39.670 align:middle line:90% I can do m 1 to 2, 1 to 3. 00:22:39.670 --> 00:22:43.750 align:middle line:90% And M equals collect(m). 00:22:43.750 --> 00:22:44.680 align:middle line:90% I think that works. 00:22:44.680 --> 00:22:45.280 align:middle line:90% Yes. 00:22:45.280 --> 00:22:45.780 align:middle line:90% OK. 00:22:45.780 --> 00:22:50.200 align:middle line:84% So now I can have subscripts if you like subscripts better. 00:22:50.200 --> 00:22:51.490 align:middle line:90% And then I can-- 00:22:51.490 --> 00:22:52.115 align:middle line:90% to do the norm. 00:22:52.115 --> 00:22:53.490 align:middle line:84% And it will do that symbolically. 00:22:53.490 --> 00:22:55.090 align:middle line:84% And it's exactly what I just said. 00:22:55.090 --> 00:22:57.640 align:middle line:84% So the norm is the square root of the sum 00:22:57.640 --> 00:22:59.170 align:middle line:90% of the squares of the entries. 00:22:59.170 --> 00:23:00.790 align:middle line:84% And it puts absolute value signs here. 00:23:00.790 --> 00:23:03.220 align:middle line:84% Because if they're complex numbers, then you need those. 00:23:03.220 --> 00:23:06.620 align:middle line:84% But we're dealing mostly with real numbers in this class. 00:23:06.620 --> 00:23:07.120 align:middle line:90% OK? 00:23:07.120 --> 00:23:09.918 align:middle line:90% 00:23:09.918 --> 00:23:11.460 align:middle line:84% Oh, it used to have an abs2 function, 00:23:11.460 --> 00:23:12.480 align:middle line:90% but it doesn't anymore. 00:23:12.480 --> 00:23:14.800 align:middle line:84% So let me just delete that comment. 00:23:14.800 --> 00:23:15.300 align:middle line:90% Right? 00:23:15.300 --> 00:23:18.300 align:middle line:90% So one of the-- 00:23:18.300 --> 00:23:22.530 align:middle line:84% so this Frobenius norm way of measuring the size of a matrix 00:23:22.530 --> 00:23:24.810 align:middle line:90% is really intuitive. 00:23:24.810 --> 00:23:27.510 align:middle line:84% Because like I said, it's the direct analog 00:23:27.510 --> 00:23:31.470 align:middle line:84% of the first and maybe the only norm 00:23:31.470 --> 00:23:35.130 align:middle line:84% you learned for column vectors, the Euclidean norm. 00:23:35.130 --> 00:23:38.655 align:middle line:84% But it's also nice in linear algebra because-- 00:23:38.655 --> 00:23:42.120 align:middle line:84% so this Euclidean norm is really useful in linear algebra 00:23:42.120 --> 00:23:45.330 align:middle line:84% because you can express it in terms 00:23:45.330 --> 00:23:48.330 align:middle line:84% of these simple linear algebra operations, 00:23:48.330 --> 00:23:49.688 align:middle line:90% as the x transpose x. 00:23:49.688 --> 00:23:50.730 align:middle line:90% And then that's a number. 00:23:50.730 --> 00:23:52.240 align:middle line:90% And you take the square root. 00:23:52.240 --> 00:23:55.710 align:middle line:84% So this Frobenius norm of matrices-- it turns out we 00:23:55.710 --> 00:24:03.180 align:middle line:84% can also write it in terms of matrix operations. 00:24:03.180 --> 00:24:05.640 align:middle line:84% It turns out it's exactly the trace 00:24:05.640 --> 00:24:08.610 align:middle line:90% of A transpose A square root. 00:24:08.610 --> 00:24:11.010 align:middle line:84% So A transpose A-- you take the trace. 00:24:11.010 --> 00:24:12.390 align:middle line:90% And you work out what it is. 00:24:12.390 --> 00:24:14.640 align:middle line:84% And it turns out that's exactly the sum of the squares 00:24:14.640 --> 00:24:16.373 align:middle line:90% of the entries of the matrix. 00:24:16.373 --> 00:24:17.790 align:middle line:84% And then you take the square root. 00:24:17.790 --> 00:24:20.640 align:middle line:84% And we call that the Frobenius norm, or AF. 00:24:20.640 --> 00:24:23.370 align:middle line:84% And later on-- it says "recall," but I think we haven't done 00:24:23.370 --> 00:24:25.195 align:middle line:90% this yet-- 00:24:25.195 --> 00:24:27.320 align:middle line:84% we're going to generalize this to think about a dot 00:24:27.320 --> 00:24:29.900 align:middle line:90% product of matrices. 00:24:29.900 --> 00:24:37.510 align:middle line:84% We'll think of B dot A as a trace-- 00:24:37.510 --> 00:24:39.970 align:middle line:84% B transpose A. So this is like the dot product of A 00:24:39.970 --> 00:24:40.580 align:middle line:90% with itself. 00:24:40.580 --> 00:24:43.420 align:middle line:84% So this is going to become important in defining 00:24:43.420 --> 00:24:47.950 align:middle line:84% the gradient of matrix functions like determinants, and traces, 00:24:47.950 --> 00:24:49.580 align:middle line:90% and so forth. 00:24:49.580 --> 00:24:50.080 align:middle line:90% OK. 00:24:50.080 --> 00:24:53.380 align:middle line:84% So this is-- and anyway, so this is kind of an aside. 00:24:53.380 --> 00:24:55.720 align:middle line:84% We're going to spend some more time on this pretty soon. 00:24:55.720 --> 00:25:01.580 align:middle line:84% But yes, you can absolutely define the norm for a matrix. 00:25:01.580 --> 00:25:05.650 align:middle line:84% And you can-- and it turns out the simplest one 00:25:05.650 --> 00:25:07.480 align:middle line:84% is kind of what you would expect, 00:25:07.480 --> 00:25:09.190 align:middle line:84% the square root of the sum of the squares of the entries. 00:25:09.190 --> 00:25:10.660 align:middle line:84% But it has this really nice formula 00:25:10.660 --> 00:25:13.682 align:middle line:84% that lets you do linear algebra with this norm, which we're not 00:25:13.682 --> 00:25:15.140 align:middle line:84% going to do too much in this class. 00:25:15.140 --> 00:25:19.420 align:middle line:84% But that's one of the reasons why this is so useful. 00:25:19.420 --> 00:25:21.680 align:middle line:84% I can compute it in multiple ways in Julia. 00:25:21.680 --> 00:25:25.600 align:middle line:84% But I can take the square root of the trace of A transpose A. 00:25:25.600 --> 00:25:29.230 align:middle line:84% I can take norm of A. The square root of the sum of abs2 00:25:29.230 --> 00:25:32.110 align:middle line:84% is the absolute value squared of the entries. 00:25:32.110 --> 00:25:34.960 align:middle line:84% Or I can take this kind of more complicated expression, 00:25:34.960 --> 00:25:38.530 align:middle line:84% square root of the sum of A i, j squared for-- 00:25:38.530 --> 00:25:40.450 align:middle line:90% i is 1 to 4. j is 1 to 4. 00:25:40.450 --> 00:25:42.760 align:middle line:90% They all give the same answers. 00:25:42.760 --> 00:25:43.360 align:middle line:90% OK? 00:25:43.360 --> 00:25:50.200 align:middle line:84% And in fact, if we go back, this is 00:25:50.200 --> 00:25:52.330 align:middle line:84% something I kind of swept under the rug earlier. 00:25:52.330 --> 00:25:55.510 align:middle line:84% But in the whole definition of a derivative, 00:25:55.510 --> 00:25:57.730 align:middle line:84% right, when we define the derivative 00:25:57.730 --> 00:26:02.000 align:middle line:84% as the linearization of the difference, 00:26:02.000 --> 00:26:04.900 align:middle line:84% right-- so that basically the derivative 00:26:04.900 --> 00:26:08.290 align:middle line:84% is what gives you the difference to first order 00:26:08.290 --> 00:26:12.430 align:middle line:84% such that every other term is higher order. 00:26:12.430 --> 00:26:15.880 align:middle line:84% In order to define what it means to be higher order, 00:26:15.880 --> 00:26:18.250 align:middle line:84% you have to have some measure of the length, 00:26:18.250 --> 00:26:20.390 align:middle line:90% of the size of a vector. 00:26:20.390 --> 00:26:24.370 align:middle line:84% And so you actually need a norm of any vector space 00:26:24.370 --> 00:26:27.310 align:middle line:84% if you want to define a derivative in this way. 00:26:27.310 --> 00:26:30.460 align:middle line:84% But fortunately, norms are very, very easy to define. 00:26:30.460 --> 00:26:34.450 align:middle line:84% And they're available for just about any vector space 00:26:34.450 --> 00:26:37.850 align:middle line:90% you're likely to think of. 00:26:37.850 --> 00:26:38.350 align:middle line:90% OK. 00:26:38.350 --> 00:26:41.410 align:middle line:90% So now we have a norm of-- 00:26:41.410 --> 00:26:44.330 align:middle line:84% we know what it means to measure the norm of a matrix. 00:26:44.330 --> 00:26:47.160 align:middle line:84% We have-- oops, let me scroll down. 00:26:47.160 --> 00:26:54.750 align:middle line:90% We have a measure of the error. 00:26:54.750 --> 00:26:56.000 align:middle line:90% So this is the relevant error. 00:26:56.000 --> 00:26:59.810 align:middle line:84% We want the difference between the approximation 00:26:59.810 --> 00:27:05.450 align:middle line:84% and the exact answer, or a derivative formula divided 00:27:05.450 --> 00:27:10.010 align:middle line:84% by the magnitude of the exact formula. 00:27:10.010 --> 00:27:14.260 align:middle line:84% And so now I want to go back and talk about this issue 00:27:14.260 --> 00:27:18.200 align:middle line:90% of how big dA should be. 00:27:18.200 --> 00:27:21.100 align:middle line:90% So we have a small perturbation. 00:27:21.100 --> 00:27:23.770 align:middle line:84% Here, I chose them of order 10 to the minus 8th. 00:27:23.770 --> 00:27:25.720 align:middle line:84% So why did I choose 10 to the minus 8? 00:27:25.720 --> 00:27:26.590 align:middle line:90% And does it matter? 00:27:26.590 --> 00:27:29.740 align:middle line:84% Like, what happens if I make it 10 to the minus 4th, 00:27:29.740 --> 00:27:32.720 align:middle line:84% or 10 to the minus 1, and so forth? 00:27:32.720 --> 00:27:35.830 align:middle line:84% So before I do any calculations, let's 00:27:35.830 --> 00:27:38.740 align:middle line:84% think about what you expect might happen. 00:27:38.740 --> 00:27:45.090 align:middle line:90% So this formula, right-- 00:27:45.090 --> 00:27:50.530 align:middle line:84% this finite difference formula is an approximation 00:27:50.530 --> 00:27:54.610 align:middle line:84% for the derivative times delta x. 00:27:54.610 --> 00:27:56.920 align:middle line:84% But it's dropping higher-order terms, right? 00:27:56.920 --> 00:28:04.060 align:middle line:84% This only becomes exact in the limit as delta x goes to 0. 00:28:04.060 --> 00:28:04.560 align:middle line:90% Right? 00:28:04.560 --> 00:28:07.215 align:middle line:84% That's the limit in which we get-- 00:28:07.215 --> 00:28:09.840 align:middle line:84% or strictly speaking, I should maybe say, the norm of that 00:28:09.840 --> 00:28:11.220 align:middle line:90% goes to 0, right? 00:28:11.220 --> 00:28:16.470 align:middle line:84% That's the limit in which we get a derivative out 00:28:16.470 --> 00:28:18.030 align:middle line:90% of a difference, right? 00:28:18.030 --> 00:28:20.340 align:middle line:84% So you might expect that, as delta x 00:28:20.340 --> 00:28:23.650 align:middle line:84% gets smaller, and smaller, and smaller, 00:28:23.650 --> 00:28:27.920 align:middle line:84% this difference formula should get more and more accurate, 00:28:27.920 --> 00:28:28.420 align:middle line:90% right? 00:28:28.420 --> 00:28:31.690 align:middle line:84% It should match-- these terms that you're neglecting 00:28:31.690 --> 00:28:33.730 align:middle line:90% should get smaller and smaller. 00:28:33.730 --> 00:28:38.170 align:middle line:84% And this difference should match our derivative more and more 00:28:38.170 --> 00:28:39.172 align:middle line:90% closely. 00:28:39.172 --> 00:28:40.630 align:middle line:84% So what I'm going to do in a second 00:28:40.630 --> 00:28:46.430 align:middle line:84% is I'm going to plot the relative error 00:28:46.430 --> 00:28:50.360 align:middle line:84% as a function of the norm of delta 00:28:50.360 --> 00:28:54.820 align:middle line:84% x-- so in this case, delta A, right? 00:28:54.820 --> 00:28:58.390 align:middle line:84% And you might expect that the error is going to go 00:28:58.390 --> 00:29:04.480 align:middle line:84% down with delta A. And in fact, it-- 00:29:04.480 --> 00:29:06.760 align:middle line:84% and if we think about it, these are 00:29:06.760 --> 00:29:08.470 align:middle line:90% going to be small magnitudes. 00:29:08.470 --> 00:29:12.040 align:middle line:84% So it's always nice to plot them on a log-log scale. 00:29:12.040 --> 00:29:13.990 align:middle line:84% A log-log scale has the nice property 00:29:13.990 --> 00:29:16.550 align:middle line:84% that it's going to turn power laws into straight lines. 00:29:16.550 --> 00:29:18.250 align:middle line:84% So usually in all of these things, 00:29:18.250 --> 00:29:20.860 align:middle line:84% we're going to see power law dependencies. 00:29:20.860 --> 00:29:22.480 align:middle line:84% It could be proportional to delta A, 00:29:22.480 --> 00:29:24.220 align:middle line:90% delta A-squared, delta A-cubed. 00:29:24.220 --> 00:29:26.490 align:middle line:84% On a log-log scale, that turns into a straight line. 00:29:26.490 --> 00:29:28.990 align:middle line:84% And this is something that I try and drill into my students. 00:29:28.990 --> 00:29:32.900 align:middle line:84% Like, anytime you're plotting something, if at all possible, 00:29:32.900 --> 00:29:35.020 align:middle line:84% you should choose your coordinates-- 00:29:35.020 --> 00:29:37.047 align:middle line:84% either logs or some change of variables-- 00:29:37.047 --> 00:29:39.130 align:middle line:84% to make the thing you're plotting a straight line. 00:29:39.130 --> 00:29:41.170 align:middle line:90% It just makes it-- 00:29:41.170 --> 00:29:43.240 align:middle line:84% or at least, you expect it to be a straight line. 00:29:43.240 --> 00:29:45.620 align:middle line:84% That makes it much easier to understand what's going on. 00:29:45.620 --> 00:29:48.287 align:middle line:84% So if we expect it to be a power law, we expect a straight line. 00:29:48.287 --> 00:29:50.037 align:middle line:84% If it's not a straight line, you should be 00:29:50.037 --> 00:29:51.250 align:middle line:90% able to immediately see that. 00:29:51.250 --> 00:29:53.930 align:middle line:84% Whereas, if you plot it on a linear scale 00:29:53.930 --> 00:29:58.910 align:middle line:84% and it goes on a linear scale, the error 00:29:58.910 --> 00:30:02.750 align:middle line:90% might go like this, right? 00:30:02.750 --> 00:30:06.125 align:middle line:84% It's really hard to see what kind of dependence that is. 00:30:06.125 --> 00:30:08.000 align:middle line:84% And it's also really hard to see what happens 00:30:08.000 --> 00:30:10.190 align:middle line:90% when things get really small. 00:30:10.190 --> 00:30:10.690 align:middle line:90% OK. 00:30:10.690 --> 00:30:12.200 align:middle line:90% So let's do that. 00:30:12.200 --> 00:30:18.430 align:middle line:84% So I wrote down here some Julia code that is going to-- 00:30:18.430 --> 00:30:21.100 align:middle line:84% because I'm going to use the plotting package. 00:30:21.100 --> 00:30:23.845 align:middle line:84% And I'm going to make some nice labels. 00:30:23.845 --> 00:30:25.720 align:middle line:84% There's multiple plotting packages for Julia. 00:30:25.720 --> 00:30:28.820 align:middle line:84% I'm going to use one called Plots. 00:30:28.820 --> 00:30:34.280 align:middle line:84% And I'm going to compute the relative error of my f 00:30:34.280 --> 00:30:37.460 align:middle line:84% of A plus dA minus f of A. But I'm 00:30:37.460 --> 00:30:41.410 align:middle line:84% going to scale it dA by different factors. 00:30:41.410 --> 00:30:42.970 align:middle line:90% OK? 00:30:42.970 --> 00:30:47.200 align:middle line:84% And I'm going to compare that to my exact answer which 00:30:47.200 --> 00:30:54.285 align:middle line:84% I had above, my A dA plus dA A. But if I scale dA by s-- 00:30:54.285 --> 00:30:55.660 align:middle line:84% that scale factor, which is going 00:30:55.660 --> 00:30:58.705 align:middle line:84% to go from 10 to the minus 8th to 10 to the 8th-- 00:30:58.705 --> 00:31:01.620 align:middle line:90% 00:31:01.620 --> 00:31:03.910 align:middle line:90% the exact formula is linear. 00:31:03.910 --> 00:31:05.250 align:middle line:90% It's a linear operator. 00:31:05.250 --> 00:31:07.680 align:middle line:84% So scaling dA by s will just scale that by s. 00:31:07.680 --> 00:31:09.190 align:middle line:90% So this is just going to be-- 00:31:09.190 --> 00:31:11.310 align:middle line:84% so I'm going to keep the same random dA, 00:31:11.310 --> 00:31:16.170 align:middle line:84% but I'm just going to scale it up or down in magnitude by-- 00:31:16.170 --> 00:31:19.200 align:middle line:84% this gives me a range of points from minus 8 to 8. 00:31:19.200 --> 00:31:20.830 align:middle line:84% And then I'm going to take 10 to that. 00:31:20.830 --> 00:31:22.380 align:middle line:90% So it's going to go from-- 00:31:22.380 --> 00:31:36.870 align:middle line:84% this is 50 points from 10 to the minus 8th to 10 to the 8th. 00:31:36.870 --> 00:31:37.620 align:middle line:90% OK? 00:31:37.620 --> 00:31:38.940 align:middle line:90% And I'm going to plot them. 00:31:38.940 --> 00:31:43.070 align:middle line:84% And I'm going to plot also a straight line for reference. 00:31:43.070 --> 00:31:46.650 align:middle line:90% And this is what we get. 00:31:46.650 --> 00:31:49.310 align:middle line:84% So this is a plot of the relative error-- 00:31:49.310 --> 00:31:54.050 align:middle line:84% so the finite difference minus the derivative formula 00:31:54.050 --> 00:31:56.960 align:middle line:84% divided by the derivative formula, magnitude, 00:31:56.960 --> 00:32:01.100 align:middle line:84% versus the norm of A-- so for the same direction, 00:32:01.100 --> 00:32:03.230 align:middle line:90% but scaling the size. 00:32:03.230 --> 00:32:05.690 align:middle line:84% And the direction was chosen randomly. 00:32:05.690 --> 00:32:08.270 align:middle line:90% And the dots are these errors. 00:32:08.270 --> 00:32:12.050 align:middle line:84% And the line here is just a line proportional 00:32:12.050 --> 00:32:15.400 align:middle line:84% to delta A-- just for comparison. 00:32:15.400 --> 00:32:15.940 align:middle line:90% OK? 00:32:15.940 --> 00:32:19.660 align:middle line:84% So we see-- stare at this for a second. 00:32:19.660 --> 00:32:22.210 align:middle line:84% And just a bunch of things should jump out at you. 00:32:22.210 --> 00:32:25.110 align:middle line:90% So if delta A is-- 00:32:25.110 --> 00:32:27.145 align:middle line:84% so suppose we start out with a pretty big delta 00:32:27.145 --> 00:32:30.340 align:middle line:90% A of order one, right? 00:32:30.340 --> 00:32:33.217 align:middle line:84% Then you expect the finite difference is not 00:32:33.217 --> 00:32:35.050 align:middle line:84% going to be very accurate if delta A is big. 00:32:35.050 --> 00:32:35.620 align:middle line:90% And it's not. 00:32:35.620 --> 00:32:38.680 align:middle line:90% The error is, like, 100%. 00:32:38.680 --> 00:32:42.220 align:middle line:84% And then as you decrease delta A in magnitude, 00:32:42.220 --> 00:32:44.470 align:middle line:84% the error gets smaller just like we expect, right? 00:32:44.470 --> 00:32:48.190 align:middle line:84% The error-- the fine difference formula 00:32:48.190 --> 00:32:50.920 align:middle line:84% should get more accurate mathematically 00:32:50.920 --> 00:32:54.850 align:middle line:84% as delta A gets closer to a dA, it 00:32:54.850 --> 00:32:56.322 align:middle line:90% goes closer to an infinitesimal. 00:32:56.322 --> 00:32:58.030 align:middle line:84% So you can drop those higher-order terms. 00:32:58.030 --> 00:33:00.010 align:middle line:90% And it actually goes to 0. 00:33:00.010 --> 00:33:02.920 align:middle line:84% It gets smaller linearly, proportional 00:33:02.920 --> 00:33:06.040 align:middle line:84% to a straight line, proportional to delta A. 00:33:06.040 --> 00:33:11.300 align:middle line:84% But at some point, when the relative error gets about 10 00:33:11.300 --> 00:33:11.958 align:middle line:90% to the minus-- 00:33:11.958 --> 00:33:12.500 align:middle line:90% I don't know. 00:33:12.500 --> 00:33:16.390 align:middle line:84% This is about 10 to the minus 10th here, it looks like. 00:33:16.390 --> 00:33:17.050 align:middle line:90% It stops. 00:33:17.050 --> 00:33:18.820 align:middle line:90% The error stops getting smaller. 00:33:18.820 --> 00:33:22.090 align:middle line:84% And then as you make delta A even smaller, 00:33:22.090 --> 00:33:24.110 align:middle line:90% the error gets bigger. 00:33:24.110 --> 00:33:27.590 align:middle line:84% So it seems like math is broken here. 00:33:27.590 --> 00:33:30.760 align:middle line:84% So does anyone-- any guesses on why 00:33:30.760 --> 00:33:35.200 align:middle line:84% the error is getting bigger when we make delta 00:33:35.200 --> 00:33:37.310 align:middle line:84% A really small, like 10 to the minus 10th, 10 00:33:37.310 --> 00:33:38.327 align:middle line:90% to the minus 12th? 00:33:38.327 --> 00:33:39.910 align:middle line:84% Like, why isn't the error just getting 00:33:39.910 --> 00:33:41.202 align:middle line:90% better, and better, and better? 00:33:41.202 --> 00:33:42.860 align:middle line:84% That's what the equations tell us. 00:33:42.860 --> 00:33:43.900 align:middle line:84% AUDIENCE: Does it have this over there? 00:33:43.900 --> 00:33:45.233 align:middle line:90% AUDIENCE: It's a roundoff error. 00:33:45.233 --> 00:33:46.733 align:middle line:84% STEVEN JOHNSON: It's roundoff error. 00:33:46.733 --> 00:33:47.260 align:middle line:90% Yes. 00:33:47.260 --> 00:33:52.030 align:middle line:84% So what's happening-- and so there's two main features. 00:33:52.030 --> 00:33:53.760 align:middle line:84% So first of all, the relative error 00:33:53.760 --> 00:33:55.830 align:middle line:90% decreases linearly with delta. 00:33:55.830 --> 00:33:57.330 align:middle line:90% That's what we expect. 00:33:57.330 --> 00:33:59.130 align:middle line:84% This is called first-order accuracy, 00:33:59.130 --> 00:34:01.080 align:middle line:84% by the way-- this linear decrease. 00:34:01.080 --> 00:34:02.640 align:middle line:90% But then it increases. 00:34:02.640 --> 00:34:07.440 align:middle line:84% And it increases because of roundoff errors. 00:34:07.440 --> 00:34:09.000 align:middle line:84% Because the computer is only going 00:34:09.000 --> 00:34:13.679 align:middle line:84% to keep a certain number of digits when it does arithmetic. 00:34:13.679 --> 00:34:17.760 align:middle line:84% And so when delta A gets really, really small, 00:34:17.760 --> 00:34:22.699 align:middle line:84% it turns out it's going to cause the answers to be completely 00:34:22.699 --> 00:34:23.199 align:middle line:90% wrong. 00:34:23.199 --> 00:34:24.639 align:middle line:84% So I'm going to talk about both of these things. 00:34:24.639 --> 00:34:26.190 align:middle line:84% So first of all, I'm going to talk 00:34:26.190 --> 00:34:29.310 align:middle line:84% about the right portion of the plot, where it's 00:34:29.310 --> 00:34:30.630 align:middle line:90% kind of doing what we expect. 00:34:30.630 --> 00:34:34.590 align:middle line:84% The error is decreasing as you make dA smaller. 00:34:34.590 --> 00:34:36.270 align:middle line:84% But I want to drill in a little deeper. 00:34:36.270 --> 00:34:39.550 align:middle line:84% Like, why is it decreasing linearly with dA? 00:34:39.550 --> 00:34:43.469 align:middle line:84% Why are we getting first-order accuracy, as it's called-- 00:34:43.469 --> 00:34:48.510 align:middle line:84% so error that goes dA to the first power, right? 00:34:48.510 --> 00:34:52.870 align:middle line:84% And then I'm going to talk about the second portion of the plot, 00:34:52.870 --> 00:34:54.909 align:middle line:90% this left portion-- 00:34:54.909 --> 00:35:00.010 align:middle line:84% that when dA gets really small, the error actually 00:35:00.010 --> 00:35:01.480 align:middle line:90% not only stops decreasing. 00:35:01.480 --> 00:35:03.580 align:middle line:90% It actually gets worse. 00:35:03.580 --> 00:35:05.620 align:middle line:84% And as someone said, it has to do 00:35:05.620 --> 00:35:08.140 align:middle line:84% with roundoff errors because of the finite precision 00:35:08.140 --> 00:35:09.760 align:middle line:90% of computer arithmetic. 00:35:09.760 --> 00:35:13.970 align:middle line:84% But I want to drill down into that a little bit more as well. 00:35:13.970 --> 00:35:16.300 align:middle line:84% So let's start with the right portion of the plot. 00:35:16.300 --> 00:35:19.060 align:middle line:84% This is easier to understand, I think, 00:35:19.060 --> 00:35:22.270 align:middle line:84% because it's not about computer arithmetic. 00:35:22.270 --> 00:35:25.300 align:middle line:84% You can understand it just from the equations, 00:35:25.300 --> 00:35:29.290 align:middle line:84% treating everything as real numbers, as exact arithmetic. 00:35:29.290 --> 00:35:35.170 align:middle line:84% So we're computing-- so basically, 00:35:35.170 --> 00:35:40.370 align:middle line:84% the way to understand this is, you can use a Taylor series. 00:35:40.370 --> 00:35:47.830 align:middle line:84% So if you can imagine taking f of plus dx 00:35:47.830 --> 00:35:52.027 align:middle line:84% and thinking of this not just as the definition of a derivative, 00:35:52.027 --> 00:35:53.610 align:middle line:84% but thinking of it as a Taylor series. 00:35:53.610 --> 00:35:57.870 align:middle line:84% So if you take the Taylor series, you get f of x plus dx. 00:35:57.870 --> 00:35:58.950 align:middle line:90% What's the first term? 00:35:58.950 --> 00:36:00.390 align:middle line:90% It's f of x. 00:36:00.390 --> 00:36:02.520 align:middle line:90% What's the second term? 00:36:02.520 --> 00:36:03.780 align:middle line:90% It's f prime of x dx. 00:36:03.780 --> 00:36:07.050 align:middle line:90% That's our linear term. 00:36:07.050 --> 00:36:11.040 align:middle line:84% And this still works even when dx is a vector, or a matrix, 00:36:11.040 --> 00:36:13.350 align:middle line:90% or in some vector space. 00:36:13.350 --> 00:36:17.160 align:middle line:90% And think about, again, 18.01. 00:36:17.160 --> 00:36:19.320 align:middle line:84% What's the next term in the Taylor series? 00:36:19.320 --> 00:36:22.260 align:middle line:84% The next term is the second derivative term, 00:36:22.260 --> 00:36:23.490 align:middle line:90% the f double prime term. 00:36:23.490 --> 00:36:26.640 align:middle line:90% 00:36:26.640 --> 00:36:28.940 align:middle line:84% And that would give you terms that go like dx-squared. 00:36:28.940 --> 00:36:32.580 align:middle line:84% And it turns out this still works for arbitrary vector 00:36:32.580 --> 00:36:33.080 align:middle line:90% spaces. 00:36:33.080 --> 00:36:35.330 align:middle line:84% This second derivative is going to be something called 00:36:35.330 --> 00:36:37.700 align:middle line:90% a Hessian or quadratic form. 00:36:37.700 --> 00:36:39.240 align:middle line:90% We'll talk about that later. 00:36:39.240 --> 00:36:41.390 align:middle line:84% But the basic idea is this is proportional 00:36:41.390 --> 00:36:42.380 align:middle line:90% to a second derivative. 00:36:42.380 --> 00:36:46.250 align:middle line:84% And it has terms that go like the square of the change. 00:36:46.250 --> 00:36:48.260 align:middle line:84% And then there are even higher-order terms, 00:36:48.260 --> 00:36:51.080 align:middle line:84% which I can use this notation in computer science-- 00:36:51.080 --> 00:36:53.810 align:middle line:84% this little o of delta x-squared-- terms 00:36:53.810 --> 00:36:56.330 align:middle line:84% that go even smaller-- delta x-cubed, delta x 00:36:56.330 --> 00:36:59.263 align:middle line:84% to the 4th from the third derivative, 00:36:59.263 --> 00:37:00.930 align:middle line:84% and the fourth derivative, and so forth. 00:37:00.930 --> 00:37:03.723 align:middle line:84% So if the function-- and it turns out this is true. 00:37:03.723 --> 00:37:05.390 align:middle line:84% It doesn't even require a Taylor series. 00:37:05.390 --> 00:37:08.090 align:middle line:84% This is true any time you have a function 00:37:08.090 --> 00:37:09.740 align:middle line:90% that has a second derivative. 00:37:09.740 --> 00:37:12.980 align:middle line:84% It may not even be 3 times differentiable. 00:37:12.980 --> 00:37:15.030 align:middle line:84% But if it has a second derivative, 00:37:15.030 --> 00:37:19.670 align:middle line:84% then you can write these first three terms and then 00:37:19.670 --> 00:37:20.670 align:middle line:90% the higher-order term. 00:37:20.670 --> 00:37:23.060 align:middle line:84% So imagine what does this mean for our difference 00:37:23.060 --> 00:37:24.760 align:middle line:90% approximation, right? 00:37:24.760 --> 00:37:26.010 align:middle line:90% So what are we computing here? 00:37:26.010 --> 00:37:27.480 align:middle line:84% We're plotting this relative error. 00:37:27.480 --> 00:37:33.570 align:middle line:84% We're plotting f of x plus dx minus f of x. 00:37:33.570 --> 00:37:35.180 align:middle line:90% That's our finite difference. 00:37:35.180 --> 00:37:41.870 align:middle line:84% And we're subtracting our exact derivative f prime of x dx. 00:37:41.870 --> 00:37:42.370 align:middle line:90% Right? 00:37:42.370 --> 00:37:46.030 align:middle line:84% But then we're dividing by the exact derivative norm. 00:37:46.030 --> 00:37:54.140 align:middle line:84% But what happens when we plug this Taylor series into here? 00:37:54.140 --> 00:38:01.280 align:middle line:84% So you-- well, first of all, the first two terms 00:38:01.280 --> 00:38:03.830 align:middle line:84% here are just these first two terms here. 00:38:03.830 --> 00:38:04.940 align:middle line:90% It's just f of x. 00:38:04.940 --> 00:38:10.245 align:middle line:84% If we move the f of x to the left-hand side, 00:38:10.245 --> 00:38:11.120 align:middle line:90% it's gone from there. 00:38:11.120 --> 00:38:13.020 align:middle line:90% That's this. 00:38:13.020 --> 00:38:13.520 align:middle line:90% OK? 00:38:13.520 --> 00:38:14.720 align:middle line:90% So what's left? 00:38:14.720 --> 00:38:16.610 align:middle line:90% Well, there's this term. 00:38:16.610 --> 00:38:20.330 align:middle line:84% But this cancels the thing we're subtracting, 00:38:20.330 --> 00:38:23.090 align:middle line:90% the exact derivative. 00:38:23.090 --> 00:38:25.330 align:middle line:90% So what's left over? 00:38:25.330 --> 00:38:28.010 align:middle line:90% These terms. 00:38:28.010 --> 00:38:28.510 align:middle line:90% OK? 00:38:28.510 --> 00:38:32.770 align:middle line:84% It is the terms proportional to delta 00:38:32.770 --> 00:38:35.625 align:middle line:84% x-squared plus higher-order terms. 00:38:35.625 --> 00:38:38.720 align:middle line:90% 00:38:38.720 --> 00:38:39.402 align:middle line:90% Right? 00:38:39.402 --> 00:38:40.610 align:middle line:90% So does everyone follow that? 00:38:40.610 --> 00:38:45.610 align:middle line:84% So I know, doing math like this from slides, 00:38:45.610 --> 00:38:48.510 align:middle line:84% I need to pause and make sure everyone has a chance 00:38:48.510 --> 00:38:49.510 align:middle line:90% to stare at the formula. 00:38:49.510 --> 00:38:50.968 align:middle line:84% Because it's easy to kind of scroll 00:38:50.968 --> 00:38:52.780 align:middle line:84% faster than you can read compared 00:38:52.780 --> 00:38:55.460 align:middle line:84% to when I write it on the blackboard or on a notebook, 00:38:55.460 --> 00:38:55.960 align:middle line:90% right? 00:38:55.960 --> 00:38:58.067 align:middle line:84% So all I'm doing is plugging-- and you 00:38:58.067 --> 00:38:59.650 align:middle line:84% can think of this as a Taylor series-- 00:38:59.650 --> 00:39:02.860 align:middle line:84% into my finite difference formula 00:39:02.860 --> 00:39:06.820 align:middle line:84% and thinking about my error, my finite difference minus exact 00:39:06.820 --> 00:39:08.140 align:middle line:90% over exact. 00:39:08.140 --> 00:39:10.840 align:middle line:84% And this finite difference minus exact-- 00:39:10.840 --> 00:39:14.640 align:middle line:90% the first two terms cancel. 00:39:14.640 --> 00:39:17.160 align:middle line:84% And what's left over are these terms 00:39:17.160 --> 00:39:20.970 align:middle line:84% that go, like, delta x-squared and higher-order terms. 00:39:20.970 --> 00:39:26.170 align:middle line:84% But then we're dividing it by a denominator that's 00:39:26.170 --> 00:39:29.140 align:middle line:84% f of x dx is proportional to delta x. 00:39:29.140 --> 00:39:31.360 align:middle line:84% So if I take a term that's proportional to delta 00:39:31.360 --> 00:39:33.910 align:middle line:84% x-squared plus higher-order terms 00:39:33.910 --> 00:39:38.260 align:middle line:84% and divide it by a term that's proportional to delta x, what 00:39:38.260 --> 00:39:41.290 align:middle line:84% you get are terms that are proportional to delta 00:39:41.290 --> 00:39:46.150 align:middle line:84% x from the first term plus higher-order terms. 00:39:46.150 --> 00:39:46.650 align:middle line:90% Right? 00:39:46.650 --> 00:39:56.440 align:middle line:84% So what we expect is exactly this. 00:39:56.440 --> 00:40:03.980 align:middle line:84% We expect our relative error to be approximately linear 00:40:03.980 --> 00:40:07.370 align:middle line:84% in delta x plus terms that are higher-order that vanish 00:40:07.370 --> 00:40:08.390 align:middle line:90% as delta x-- 00:40:08.390 --> 00:40:13.340 align:middle line:84% that are negligible as delta X gets small. 00:40:13.340 --> 00:40:15.400 align:middle line:90% Does that make sense? 00:40:15.400 --> 00:40:20.570 align:middle line:84% So this is kind of the inherent error in the finite difference 00:40:20.570 --> 00:40:21.290 align:middle line:90% approximation. 00:40:21.290 --> 00:40:23.810 align:middle line:84% We expect this forward difference formula 00:40:23.810 --> 00:40:27.080 align:middle line:84% to have errors that decrease linearly with delta x. 00:40:27.080 --> 00:40:29.290 align:middle line:84% This is called the truncation error, 00:40:29.290 --> 00:40:35.520 align:middle line:84% the error that you get because you truncated the Taylor 00:40:35.520 --> 00:40:36.020 align:middle line:90% series. 00:40:36.020 --> 00:40:39.020 align:middle line:84% Because delta x is not infinitesimal. 00:40:39.020 --> 00:40:40.730 align:middle line:84% And that's exactly what we're seeing 00:40:40.730 --> 00:40:42.560 align:middle line:84% in this plot in the right-hand side. 00:40:42.560 --> 00:40:45.020 align:middle line:84% We're seeing it's actually almost perfectly linear 00:40:45.020 --> 00:40:53.450 align:middle line:84% as we decrease delta A. So that's what the math predicts. 00:40:53.450 --> 00:40:59.530 align:middle line:84% So this deviation here is not predicted by that math. 00:40:59.530 --> 00:41:03.820 align:middle line:84% And it would not happen if we were doing exact arithmetic 00:41:03.820 --> 00:41:04.940 align:middle line:90% with real numbers. 00:41:04.940 --> 00:41:09.940 align:middle line:84% So what we're-- this increase over here is due 00:41:09.940 --> 00:41:13.060 align:middle line:90% to our roundoff errors. 00:41:13.060 --> 00:41:15.040 align:middle line:84% As I said, it's due to the fact that we're not 00:41:15.040 --> 00:41:21.130 align:middle line:84% doing exact math, at least exact real number math on a computer. 00:41:21.130 --> 00:41:25.000 align:middle line:84% We're doing only approximate real number mathematics. 00:41:25.000 --> 00:41:29.560 align:middle line:84% And that's called floating-point arithmetic. 00:41:29.560 --> 00:41:32.270 align:middle line:84% And so the basic thing is-- you can think of it this way. 00:41:32.270 --> 00:41:36.388 align:middle line:84% So a computer, right-- it cannot store-- 00:41:36.388 --> 00:41:37.930 align:middle line:84% if you have an arbitrary real number, 00:41:37.930 --> 00:41:41.770 align:middle line:84% you would need infinitely many digits to store it, right? 00:41:41.770 --> 00:41:43.480 align:middle line:84% A computer doesn't have infinite memory. 00:41:43.480 --> 00:41:46.000 align:middle line:84% And even working with a huge number of digits 00:41:46.000 --> 00:41:48.760 align:middle line:90% would be very expensive, right? 00:41:48.760 --> 00:41:51.490 align:middle line:84% And so a computer-- when it's working with numbers, 00:41:51.490 --> 00:41:54.730 align:middle line:84% by default, it keeps only a relatively small number 00:41:54.730 --> 00:41:55.790 align:middle line:90% of significant digits-- 00:41:55.790 --> 00:41:57.880 align:middle line:84% I mean relatively small compared to infinity. 00:41:57.880 --> 00:42:01.310 align:middle line:84% It keeps about 15 decimal digits, which is a lot, right? 00:42:01.310 --> 00:42:06.030 align:middle line:84% That's a lot more than you would carry in a hand calculation, 00:42:06.030 --> 00:42:06.530 align:middle line:90% right? 00:42:06.530 --> 00:42:08.420 align:middle line:84% And with 15 decimal digits, it can do 00:42:08.420 --> 00:42:09.740 align:middle line:90% arithmetic very, very quickly. 00:42:09.740 --> 00:42:11.750 align:middle line:90% It's built into the CPU. 00:42:11.750 --> 00:42:14.570 align:middle line:84% That's why everyone uses this same number of digits. 00:42:14.570 --> 00:42:18.560 align:middle line:84% Whether you're in Matlab, or in Python, or in Julia, or C, 00:42:18.560 --> 00:42:20.900 align:middle line:84% or Fortran, this is kind of the default. 00:42:20.900 --> 00:42:23.840 align:middle line:84% It's called double-precision floating-point arithmetic. 00:42:23.840 --> 00:42:26.990 align:middle line:84% And so everyone uses the same thing most 00:42:26.990 --> 00:42:28.950 align:middle line:90% of the time, at least. 00:42:28.950 --> 00:42:32.750 align:middle line:84% And so this is the source of that increase. 00:42:32.750 --> 00:42:34.850 align:middle line:84% It's these things called roundoff errors 00:42:34.850 --> 00:42:38.160 align:middle line:84% from this finite number of significant digits. 00:42:38.160 --> 00:42:42.320 align:middle line:84% And so let's think in a little bit more detail of how 00:42:42.320 --> 00:42:44.600 align:middle line:90% that happens. 00:42:44.600 --> 00:42:48.590 align:middle line:84% So what happens if delta x is too small? 00:42:48.590 --> 00:42:53.720 align:middle line:84% So then f of x plus delta x and f of x 00:42:53.720 --> 00:42:57.470 align:middle line:84% are almost the same number, right? 00:42:57.470 --> 00:42:59.310 align:middle line:90% They're almost the same. 00:42:59.310 --> 00:43:03.610 align:middle line:84% And when you subtract them, right, then most 00:43:03.610 --> 00:43:07.150 align:middle line:84% of the significant digits cancel one another. 00:43:07.150 --> 00:43:07.650 align:middle line:90% OK? 00:43:07.650 --> 00:43:10.980 align:middle line:84% And in fact, even when you take x plus dx, 00:43:10.980 --> 00:43:15.810 align:middle line:84% if x is really small-- if x is only storing 15 digits 00:43:15.810 --> 00:43:20.640 align:middle line:84% and you add a delta x that's 10 to the minus 100th, x plus dx 00:43:20.640 --> 00:43:22.680 align:middle line:90% will just give you x. 00:43:22.680 --> 00:43:25.770 align:middle line:84% Because it'll just round off x plus dx back to x. 00:43:25.770 --> 00:43:30.340 align:middle line:84% It won't be able to store that 1 in the 100th decimal place. 00:43:30.340 --> 00:43:30.840 align:middle line:90% Right? 00:43:30.840 --> 00:43:34.260 align:middle line:84% So in general, this is called a catastrophic cancellation-- 00:43:34.260 --> 00:43:37.500 align:middle line:84% when you take two nearly equal numbers 00:43:37.500 --> 00:43:41.950 align:middle line:84% and you subtract them to finite, but they each 00:43:41.950 --> 00:43:42.835 align:middle line:90% have finite accuracy. 00:43:42.835 --> 00:43:43.690 align:middle line:90% You subtract them. 00:43:43.690 --> 00:43:45.580 align:middle line:84% Most of the significant digits cancel. 00:43:45.580 --> 00:43:49.240 align:middle line:84% You're left with only a few significant digits or maybe 00:43:49.240 --> 00:43:50.410 align:middle line:90% even no significant digits. 00:43:50.410 --> 00:43:52.327 align:middle line:84% Maybe if they're small enough, this difference 00:43:52.327 --> 00:43:53.560 align:middle line:90% might just be 0. 00:43:53.560 --> 00:43:57.340 align:middle line:84% So it's easier to see with a simpler function like-- 00:43:57.340 --> 00:44:00.470 align:middle line:84% not a matrix function, but just a scalar function. 00:44:00.470 --> 00:44:02.190 align:middle line:90% So that's just sine of x. 00:44:02.190 --> 00:44:02.690 align:middle line:90% OK? 00:44:02.690 --> 00:44:05.830 align:middle line:84% At x equals 1, all right, is it the exact derivative 00:44:05.830 --> 00:44:08.920 align:middle line:90% of just cosine of 1, right? 00:44:08.920 --> 00:44:12.340 align:middle line:84% And so we can imagine our finite difference approximation is 00:44:12.340 --> 00:44:17.500 align:middle line:84% just f of x plus dx. f is just sine minus fx over delta x 00:44:17.500 --> 00:44:20.557 align:middle line:90% plus first-order terms, right? 00:44:20.557 --> 00:44:22.390 align:middle line:84% That's what happens when I take this formula 00:44:22.390 --> 00:44:26.080 align:middle line:84% and just divide it through by delta x, right? 00:44:26.080 --> 00:44:33.010 align:middle line:84% So if dx equals 1e to the minus 5, I get 0.54 something. 00:44:33.010 --> 00:44:36.790 align:middle line:84% The cosine of 1-- the exact answer is 0.54 something. 00:44:36.790 --> 00:44:41.470 align:middle line:84% But you can see they differ in the third digit already. 00:44:41.470 --> 00:44:45.430 align:middle line:84% The relative error is actually 10 to the minus 6. 00:44:45.430 --> 00:44:47.085 align:middle line:84% So it's actually pretty good here. 00:44:47.085 --> 00:44:49.210 align:middle line:84% Why is it 10 to the minus-- oh, because it's really 00:44:49.210 --> 00:44:50.043 align:middle line:90% not the third digit. 00:44:50.043 --> 00:44:52.420 align:middle line:90% It's .540298. 00:44:52.420 --> 00:44:54.040 align:middle line:90% And this is "302." 00:44:54.040 --> 00:44:58.120 align:middle line:84% So they really differ in almost, like, the sixth digit here 00:44:58.120 --> 00:44:59.260 align:middle line:90% versus there. 00:44:59.260 --> 00:45:01.510 align:middle line:90% It's-- right? 00:45:01.510 --> 00:45:04.780 align:middle line:84% And that's differing by 4 in the sixth digit, right? 00:45:04.780 --> 00:45:08.920 align:middle line:84% And so now what happens if you do delta x equals 10 00:45:08.920 --> 00:45:10.150 align:middle line:90% to the minus 100th, right? 00:45:10.150 --> 00:45:12.580 align:middle line:84% Mathematically, that should give you a much better answer. 00:45:12.580 --> 00:45:15.880 align:middle line:84% But actually, it just gives you 0. 00:45:15.880 --> 00:45:19.078 align:middle line:84% And the computer is perfectly capable of representing 10 00:45:19.078 --> 00:45:19.870 align:middle line:90% to the minus 100th. 00:45:19.870 --> 00:45:21.550 align:middle line:84% So it's basically floating-point. 00:45:21.550 --> 00:45:26.780 align:middle line:84% It stores a fixed number of digits and then an exponent. 00:45:26.780 --> 00:45:28.923 align:middle line:84% You can think of it as like scientific notation 00:45:28.923 --> 00:45:29.590 align:middle line:90% on the computer. 00:45:29.590 --> 00:45:33.070 align:middle line:84% So it can store really big, really small numbers 00:45:33.070 --> 00:45:37.450 align:middle line:84% just with a finite number of significant digits, right? 00:45:37.450 --> 00:45:41.380 align:middle line:84% But if you do 1 plus 10 to the minus 100th, you get 1. 00:45:41.380 --> 00:45:43.810 align:middle line:84% Because what would the correct answer be? 00:45:43.810 --> 00:45:46.240 align:middle line:84% The correct answer would be 1 plus 10 to the minus 100th 00:45:46.240 --> 00:45:52.120 align:middle line:84% is 1, a decimal point, 99 0's, and then a 1, right? 00:45:52.120 --> 00:45:54.880 align:middle line:84% And if the computer is only storing 15 digits, 00:45:54.880 --> 00:45:59.710 align:middle line:84% it cannot store 1, 99 0's, and then a 1, right? 00:45:59.710 --> 00:46:03.790 align:middle line:84% And what it does is it takes this result. 00:46:03.790 --> 00:46:07.960 align:middle line:84% And effectively, it rounds it to the closest number 00:46:07.960 --> 00:46:10.690 align:middle line:84% that it can represent with 15 digits. 00:46:10.690 --> 00:46:14.020 align:middle line:90% And the closest number is 1. 00:46:14.020 --> 00:46:16.010 align:middle line:90% So that's what it gives you. 00:46:16.010 --> 00:46:19.695 align:middle line:84% So even if you do something less extreme-- so if you do-- 00:46:19.695 --> 00:46:20.990 align:middle line:90% it's storing 15 digits. 00:46:20.990 --> 00:46:23.830 align:middle line:84% So if you do 10 to the minus 13th for your dx, 00:46:23.830 --> 00:46:28.270 align:middle line:84% then 1 plus dx it can represent, right? 00:46:28.270 --> 00:46:31.420 align:middle line:84% So then the computer can store this number, right? 00:46:31.420 --> 00:46:37.140 align:middle line:84% But if you take sine of 1 plus dx and sine of 1, 00:46:37.140 --> 00:46:40.080 align:middle line:84% right, you can see that they're now-- 00:46:40.080 --> 00:46:42.180 align:middle line:84% if dx is 10 to the minus 13th, they're 00:46:42.180 --> 00:46:44.620 align:middle line:90% almost the same number-- 00:46:44.620 --> 00:46:46.810 align:middle line:90% this and this. 00:46:46.810 --> 00:46:47.310 align:middle line:90% See? 00:46:47.310 --> 00:46:53.370 align:middle line:84% They only differ here, in the last four significant digits. 00:46:53.370 --> 00:46:56.330 align:middle line:84% So when you subtract them in a finite difference approximation 00:46:56.330 --> 00:47:00.520 align:middle line:84% and take this minus this, almost all the significant digits 00:47:00.520 --> 00:47:01.020 align:middle line:90% cancel. 00:47:01.020 --> 00:47:04.570 align:middle line:90% 00:47:04.570 --> 00:47:05.800 align:middle line:90% You have 16 digits here-- 00:47:05.800 --> 00:47:06.970 align:middle line:90% 15 digits. 00:47:06.970 --> 00:47:10.000 align:middle line:90% But 11 of them cancel. 00:47:10.000 --> 00:47:15.512 align:middle line:84% And you're just left with only four digits of accuracy. 00:47:15.512 --> 00:47:17.220 align:middle line:84% And so if you compute the relative error, 00:47:17.220 --> 00:47:24.260 align:middle line:84% you get 10 to the minus 3 from those four digits of accuracy. 00:47:24.260 --> 00:47:28.370 align:middle line:84% So this is the general problem with finite difference 00:47:28.370 --> 00:47:31.490 align:middle line:84% is that they're not completely bulletproof. 00:47:31.490 --> 00:47:35.060 align:middle line:84% You need to make delta x small enough that the truncation 00:47:35.060 --> 00:47:39.500 align:middle line:84% error is small but not so small that the roundoff errors kill 00:47:39.500 --> 00:47:40.730 align:middle line:90% you, right? 00:47:40.730 --> 00:47:45.470 align:middle line:84% And kind of a rule of thumb is that it 00:47:45.470 --> 00:47:47.570 align:middle line:84% works-- it's not perfect, like any rule of thumb. 00:47:47.570 --> 00:47:50.660 align:middle line:84% No rule of thumb is perfectly reliable. 00:47:50.660 --> 00:47:52.970 align:middle line:84% A good rule of thumb is to use about half the digits 00:47:52.970 --> 00:47:54.090 align:middle line:90% for your difference. 00:47:54.090 --> 00:47:57.470 align:middle line:84% So more precisely, the number of significant digits 00:47:57.470 --> 00:47:59.780 align:middle line:84% on the computer is represented by this quantity 00:47:59.780 --> 00:48:01.800 align:middle line:90% called the machine epsilon. 00:48:01.800 --> 00:48:05.120 align:middle line:84% So basically, epsilon x is the size 00:48:05.120 --> 00:48:08.100 align:middle line:90% of the last significant digit. 00:48:08.100 --> 00:48:10.380 align:middle line:84% Or the next floating-point number 00:48:10.380 --> 00:48:14.020 align:middle line:84% out of x-- after x is x times 1 plus epsilon. 00:48:14.020 --> 00:48:17.350 align:middle line:84% So in Julia, there's a function called eps that returns this. 00:48:17.350 --> 00:48:20.640 align:middle line:90% It's about 10 to the minus 16th. 00:48:20.640 --> 00:48:24.450 align:middle line:84% So as I said, it's about 15 or 16 decimal digits. 00:48:24.450 --> 00:48:28.840 align:middle line:84% The reason this is kind of a weird-looking number-- 00:48:28.840 --> 00:48:32.220 align:middle line:84% it's not 1e to the minus 16th or 1e to the minus 15th-- 00:48:32.220 --> 00:48:34.110 align:middle line:84% is because the computer is actually not 00:48:34.110 --> 00:48:35.220 align:middle line:90% storing decimal digits. 00:48:35.220 --> 00:48:36.940 align:middle line:90% It's storing binary digits. 00:48:36.940 --> 00:48:40.755 align:middle line:90% So it's actually storing 1.000-- 00:48:40.755 --> 00:48:42.850 align:middle line:84% a whole bunch of digits times an exponent, 00:48:42.850 --> 00:48:43.830 align:middle line:90% which is a power of 2. 00:48:43.830 --> 00:48:47.010 align:middle line:84% And the digits are actually binary digits. 00:48:47.010 --> 00:48:51.210 align:middle line:84% So this is not actually 10 to the minus 15th. 00:48:51.210 --> 00:48:53.520 align:middle line:90% It's 2 to the minus 52. 00:48:53.520 --> 00:48:56.230 align:middle line:90% But it's a crude approximation. 00:48:56.230 --> 00:48:59.110 align:middle line:84% You can think of it as 15 decimal digits, right? 00:48:59.110 --> 00:49:02.620 align:middle line:84% And so a crude rule of thumb for finding differences-- 00:49:02.620 --> 00:49:05.650 align:middle line:90% it's kind of a good median-- 00:49:05.650 --> 00:49:08.860 align:middle line:84% for a small truncation error but not big roundoff errors 00:49:08.860 --> 00:49:11.050 align:middle line:84% is to choose your delta x to be about half 00:49:11.050 --> 00:49:12.610 align:middle line:90% the significant digits of x. 00:49:12.610 --> 00:49:14.560 align:middle line:84% So basically, the magnitude of delta x 00:49:14.560 --> 00:49:17.710 align:middle line:84% is on the order of square root of epsilon times x. 00:49:17.710 --> 00:49:21.700 align:middle line:84% So square root of epsilon is about 10 to the minus 8th. 00:49:21.700 --> 00:49:24.220 align:middle line:84% So that's why I chose 10 to the minus 8th. 00:49:24.220 --> 00:49:27.730 align:middle line:84% My matrix A was random numbers on order one. 00:49:27.730 --> 00:49:30.700 align:middle line:84% So my matrix dA, I made them random numbers on order 10 00:49:30.700 --> 00:49:31.930 align:middle line:90% to minus 8th. 00:49:31.930 --> 00:49:36.790 align:middle line:84% And that was-- if you go back up here, 10 to the minus 8th 00:49:36.790 --> 00:49:40.360 align:middle line:84% is about where we see this crossover point happening. 00:49:40.360 --> 00:49:42.760 align:middle line:84% It's about the point where the truncation error is 00:49:42.760 --> 00:49:44.700 align:middle line:90% balancing the roundoff error. 00:49:44.700 --> 00:49:48.400 align:middle line:84% And so these aren't perfect rules of thumb, but it's-- 00:49:48.400 --> 00:49:55.120 align:middle line:84% and if you have to pick a delta x quickly, that's-- 00:49:55.120 --> 00:49:55.630 align:middle line:90% let's see. 00:49:55.630 --> 00:50:02.090 align:middle line:84% That's a good way to choose that, right? 00:50:02.090 --> 00:50:02.590 align:middle line:90% So-- 00:50:02.590 --> 00:50:03.910 align:middle line:84% ALAN EDELMAN: Steven, let me point out that we're just about 00:50:03.910 --> 00:50:04.390 align:middle line:90% out of time. 00:50:04.390 --> 00:50:04.870 align:middle line:90% STEVEN JOHNSON: Yep. 00:50:04.870 --> 00:50:06.760 align:middle line:84% And I'll show this, too-- just the same thing 00:50:06.760 --> 00:50:07.730 align:middle line:90% for the sine plot. 00:50:07.730 --> 00:50:10.540 align:middle line:84% So this is the same thing for difference. 00:50:10.540 --> 00:50:12.580 align:middle line:84% The dots are the relative error compared 00:50:12.580 --> 00:50:16.010 align:middle line:84% to the exact derivative, which is cosine of 1. 00:50:16.010 --> 00:50:18.418 align:middle line:84% And the line is just a straight line for reference. 00:50:18.418 --> 00:50:19.710 align:middle line:90% You see exactly the same thing. 00:50:19.710 --> 00:50:23.120 align:middle line:84% It's, as delta x gets smaller, initially 00:50:23.120 --> 00:50:26.300 align:middle line:84% the error gets smaller because the finite difference is 00:50:26.300 --> 00:50:27.150 align:middle line:90% getting better. 00:50:27.150 --> 00:50:29.270 align:middle line:84% And it goes down linearly because this is 00:50:29.270 --> 00:50:32.240 align:middle line:90% a first-order-accurate formula. 00:50:32.240 --> 00:50:36.740 align:middle line:84% But eventually, when delta x gets too small, 00:50:36.740 --> 00:50:39.230 align:middle line:84% the cancellation error, the roundoff error kills you. 00:50:39.230 --> 00:50:40.730 align:middle line:90% And the error gets worse. 00:50:40.730 --> 00:50:43.070 align:middle line:84% And the crossover point-- again, that rule of thumb-- 00:50:43.070 --> 00:50:43.820 align:middle line:90% works pretty well. 00:50:43.820 --> 00:50:48.980 align:middle line:84% It's about 10 to the minus 8th in here for this, since x is 1. 00:50:48.980 --> 00:50:51.740 align:middle line:90% So I'll stop there. 00:50:51.740 --> 00:50:56.450 align:middle line:84% And we'll see you again on Wednesday. 00:50:56.450 --> 00:50:58.190 align:middle line:90% And I'll post this notebook. 00:50:58.190 --> 00:50:59.480 align:middle line:90% And we'll post-- 00:50:59.480 --> 00:51:01.640 align:middle line:84% I'll get Alan's updated notebook from him. 00:51:01.640 --> 00:51:04.980 align:middle line:84% And we'll post that on the website as well. 00:51:04.980 --> 00:51:06.670 align:middle line:90% ALAN EDELMAN: Sounds good. 00:51:06.670 --> 00:51:10.000 align:middle line:90%