WEBVTT 00:00:00.000 --> 00:00:01.992 align:middle line:90% [SQUEAKING] 00:00:01.992 --> 00:00:04.482 align:middle line:90% [RUSTLING] 00:00:04.482 --> 00:00:06.972 align:middle line:90% [CLICKING] 00:00:06.972 --> 00:00:12.970 align:middle line:90% 00:00:12.970 --> 00:00:15.580 align:middle line:90% PROFESSOR: And I just-- 00:00:15.580 --> 00:00:18.620 align:middle line:84% since I think it's been over a week, 00:00:18.620 --> 00:00:21.490 align:middle line:84% let me just kind of remind you the toy problem 00:00:21.490 --> 00:00:24.080 align:middle line:90% that I put up about a week ago. 00:00:24.080 --> 00:00:27.640 align:middle line:84% And the problem was where I was going to input x and y. 00:00:27.640 --> 00:00:29.680 align:middle line:90% Maybe you all remember this. 00:00:29.680 --> 00:00:33.460 align:middle line:84% And I had a kind of a program, if you will. 00:00:33.460 --> 00:00:36.550 align:middle line:84% I kind of think of this as sort of steps in some software 00:00:36.550 --> 00:00:44.050 align:middle line:84% if you like, where I take a was sine x, and b is-- 00:00:44.050 --> 00:00:46.960 align:middle line:84% nothing special here, just illustrating 00:00:46.960 --> 00:00:48.010 align:middle line:90% a number of things. 00:00:48.010 --> 00:00:49.720 align:middle line:90% c is b plus x. 00:00:49.720 --> 00:00:55.120 align:middle line:84% And I was going to return z as the answer, OK? 00:00:55.120 --> 00:00:58.810 align:middle line:84% And maybe you all remember that there was this graph 00:00:58.810 --> 00:01:01.720 align:middle line:84% that I drew that kind of illustrates a little bit 00:01:01.720 --> 00:01:02.890 align:middle line:90% of what's going on. 00:01:02.890 --> 00:01:06.190 align:middle line:90% I put the inputs-- 00:01:06.190 --> 00:01:09.520 align:middle line:84% in fact, I put all the variables in circles. 00:01:09.520 --> 00:01:11.076 align:middle line:90% And so there a. 00:01:11.076 --> 00:01:12.140 align:middle line:90% There was b. 00:01:12.140 --> 00:01:14.630 align:middle line:90% And I'm going to end up with z. 00:01:14.630 --> 00:01:19.790 align:middle line:84% And I just put the local derivatives. 00:01:19.790 --> 00:01:23.360 align:middle line:84% And, Steven, I want to ask you if you would like or hate 00:01:23.360 --> 00:01:26.430 align:middle line:84% this notation, a notation that I think I've seen some people 00:01:26.430 --> 00:01:26.930 align:middle line:90% use. 00:01:26.930 --> 00:01:29.540 align:middle line:84% But, again, nobody's consistent in this field right now. 00:01:29.540 --> 00:01:35.750 align:middle line:90% I kind of like the idea that-- 00:01:35.750 --> 00:01:40.060 align:middle line:84% here, let's-- and I don't want to get into placeholder 00:01:40.060 --> 00:01:40.730 align:middle line:90% notation. 00:01:40.730 --> 00:01:44.130 align:middle line:90% But if I were to talk about-- 00:01:44.130 --> 00:01:45.600 align:middle line:90% if I were to talk about-- 00:01:45.600 --> 00:01:50.310 align:middle line:84% I want to make a distinction between dz dx, 00:01:50.310 --> 00:01:54.060 align:middle line:90% and partial z and partial x. 00:01:54.060 --> 00:01:58.680 align:middle line:90% 00:01:58.680 --> 00:02:03.900 align:middle line:84% And to be clear, this one, I want to equal 1. 00:02:03.900 --> 00:02:07.230 align:middle line:84% And I want to think of it as a one-step derivative. 00:02:07.230 --> 00:02:10.440 align:middle line:84% I look at this formula alone, and I 00:02:10.440 --> 00:02:14.460 align:middle line:84% don't want to think of b as being dependent on x, which, 00:02:14.460 --> 00:02:15.900 align:middle line:90% in fact, it is. 00:02:15.900 --> 00:02:18.240 align:middle line:84% But I'm pretending b is not dependent on x. 00:02:18.240 --> 00:02:22.380 align:middle line:84% And I'm just looking at what I would call the step 3 00:02:22.380 --> 00:02:26.540 align:middle line:84% derivative or the one-step derivative 00:02:26.540 --> 00:02:32.260 align:middle line:84% as opposed to the actual derivative, the one that 00:02:32.260 --> 00:02:33.700 align:middle line:90% puts it all together. 00:02:33.700 --> 00:02:35.050 align:middle line:90% And we wrote it down last time. 00:02:35.050 --> 00:02:35.967 align:middle line:90% What did I say it was? 00:02:35.967 --> 00:02:38.980 align:middle line:90% 00:02:38.980 --> 00:02:44.850 align:middle line:84% z, if you put it all together, was sine x over y plus x. 00:02:44.850 --> 00:02:46.980 align:middle line:84% And so this thing, the thing that we actually 00:02:46.980 --> 00:02:50.880 align:middle line:84% want to calculate, is cosine x over y plus 1. 00:02:50.880 --> 00:02:53.790 align:middle line:84% So my question for Steven was, what 00:02:53.790 --> 00:02:55.080 align:middle line:90% do you think of this notation? 00:02:55.080 --> 00:02:55.860 align:middle line:90% Do you like it? 00:02:55.860 --> 00:02:56.370 align:middle line:90% OK. 00:02:56.370 --> 00:03:00.070 align:middle line:84% Anybody want to object to the notation? 00:03:00.070 --> 00:03:03.390 align:middle line:84% So I like the idea of partial being a partial-- 00:03:03.390 --> 00:03:04.970 align:middle line:90% it's not the whole set of steps. 00:03:04.970 --> 00:03:05.970 align:middle line:90% It's just like one step. 00:03:05.970 --> 00:03:09.630 align:middle line:84% It's partial truth, and then the derivative 00:03:09.630 --> 00:03:11.790 align:middle line:90% being the whole truth. 00:03:11.790 --> 00:03:14.190 align:middle line:84% And so what I'm putting-- what I'm going to put here 00:03:14.190 --> 00:03:15.990 align:middle line:90% are all the partials. 00:03:15.990 --> 00:03:18.300 align:middle line:84% So I'm going to put down the cosine 00:03:18.300 --> 00:03:21.570 align:middle line:84% x, which is the derivative of a with respect to x. 00:03:21.570 --> 00:03:24.720 align:middle line:90% And the 1 over y goes here. 00:03:24.720 --> 00:03:27.750 align:middle line:90% And there's a 1 that goes here. 00:03:27.750 --> 00:03:31.440 align:middle line:84% But there's also a 1 coming from here. 00:03:31.440 --> 00:03:34.080 align:middle line:90% And let's see. 00:03:34.080 --> 00:03:36.280 align:middle line:90% What else did we have? 00:03:36.280 --> 00:03:41.380 align:middle line:84% And we had a minus a over y squared over here. 00:03:41.380 --> 00:03:43.030 align:middle line:84% And maybe it's also useful sometimes 00:03:43.030 --> 00:03:45.800 align:middle line:84% to write down what these things are equal to. 00:03:45.800 --> 00:03:47.480 align:middle line:84% I'll just stick this below the circle. 00:03:47.480 --> 00:03:49.310 align:middle line:84% I don't know if the printing is too small. 00:03:49.310 --> 00:03:51.950 align:middle line:84% But I'll just do it anyway-- cosine y, 00:03:51.950 --> 00:03:56.078 align:middle line:90% and b plus x was here. 00:03:56.078 --> 00:03:57.620 align:middle line:84% All right, so this is where we were I 00:03:57.620 --> 00:04:01.520 align:middle line:84% think when I left off last time, just 00:04:01.520 --> 00:04:03.980 align:middle line:84% to show an example of how a program is 00:04:03.980 --> 00:04:07.970 align:middle line:84% a sequence of transformations of variables. 00:04:07.970 --> 00:04:15.390 align:middle line:84% And for a program to work, for a program to work, 00:04:15.390 --> 00:04:17.850 align:middle line:84% any time you're at a node, all of the inputs 00:04:17.850 --> 00:04:20.100 align:middle line:84% have to be available and known to you. 00:04:20.100 --> 00:04:22.950 align:middle line:84% So you may never stop to think about this property 00:04:22.950 --> 00:04:24.340 align:middle line:90% of a computer program. 00:04:24.340 --> 00:04:27.210 align:middle line:84% But it's sort of obvious, right, that I can't execute 00:04:27.210 --> 00:04:29.010 align:middle line:90% b unless I know a and y. 00:04:29.010 --> 00:04:32.040 align:middle line:84% I can't execute z unless I know b and x. 00:04:32.040 --> 00:04:36.450 align:middle line:84% And so the very nature of a computer program 00:04:36.450 --> 00:04:39.450 align:middle line:84% is that this thing is going to be a DAG-- 00:04:39.450 --> 00:04:40.170 align:middle line:90% that's the word. 00:04:40.170 --> 00:04:41.740 align:middle line:84% Maybe most of you know that word. 00:04:41.740 --> 00:04:46.020 align:middle line:84% It's a Directed Acyclic Graph, which, in simple English, 00:04:46.020 --> 00:04:49.330 align:middle line:84% means that wherever I am, all the inputs are moving left 00:04:49.330 --> 00:04:49.830 align:middle line:90% to right. 00:04:49.830 --> 00:04:53.100 align:middle line:84% They're moving from the beginning to the end. 00:04:53.100 --> 00:04:54.725 align:middle line:84% That could actually be relaxed, and you 00:04:54.725 --> 00:04:56.517 align:middle line:84% could have an implicit method or something. 00:04:56.517 --> 00:04:57.990 align:middle line:84% And most of what I'm talking about 00:04:57.990 --> 00:04:59.980 align:middle line:84% will still work under such situations. 00:04:59.980 --> 00:05:03.330 align:middle line:84% But I think, for simplicity, let's stick to DAGs and things 00:05:03.330 --> 00:05:05.020 align:middle line:90% going left to right, OK? 00:05:05.020 --> 00:05:05.520 align:middle line:90% Yep? 00:05:05.520 --> 00:05:08.232 align:middle line:84% AUDIENCE: Why does a depend on y? 00:05:08.232 --> 00:05:11.260 align:middle line:90% 00:05:11.260 --> 00:05:14.140 align:middle line:90% PROFESSOR: Let's see. 00:05:14.140 --> 00:05:15.760 align:middle line:90% Oh, of course, it doesn't. 00:05:15.760 --> 00:05:16.585 align:middle line:90% Thank you. 00:05:16.585 --> 00:05:22.600 align:middle line:90% 00:05:22.600 --> 00:05:25.525 align:middle line:84% If you want the honest reason as to why a depends on y is I 00:05:25.525 --> 00:05:26.650 align:middle line:90% left my glasses home today. 00:05:26.650 --> 00:05:30.140 align:middle line:90% 00:05:30.140 --> 00:05:31.400 align:middle line:90% Thank you. 00:05:31.400 --> 00:05:32.970 align:middle line:90% Better? 00:05:32.970 --> 00:05:33.470 align:middle line:90% Good. 00:05:33.470 --> 00:05:34.940 align:middle line:90% Good catch. 00:05:34.940 --> 00:05:36.530 align:middle line:90% All righty. 00:05:36.530 --> 00:05:38.940 align:middle line:90% That's why a depends on y. 00:05:38.940 --> 00:05:39.440 align:middle line:90% All righty. 00:05:39.440 --> 00:05:40.070 align:middle line:90% Good. 00:05:40.070 --> 00:05:42.560 align:middle line:90% So that's where we were. 00:05:42.560 --> 00:05:43.730 align:middle line:90% And let's see. 00:05:43.730 --> 00:05:45.240 align:middle line:90% How far did we get in all this? 00:05:45.240 --> 00:05:49.010 align:middle line:84% So I think we got as far as saying 00:05:49.010 --> 00:05:54.260 align:middle line:84% the bit of the general situation for the forward derivatives. 00:05:54.260 --> 00:05:57.130 align:middle line:90% Let's see. 00:05:57.130 --> 00:05:58.390 align:middle line:90% Yeah, how far did we get? 00:05:58.390 --> 00:06:05.004 align:middle line:90% 00:06:05.004 --> 00:06:10.770 align:middle line:84% So the goal, as you remember, is for every input-- 00:06:10.770 --> 00:06:12.370 align:middle line:90% so the inputs are over here. 00:06:12.370 --> 00:06:14.530 align:middle line:84% And for every output, so in this example, 00:06:14.530 --> 00:06:16.305 align:middle line:84% we have two inputs and one output. 00:06:16.305 --> 00:06:19.730 align:middle line:90% 00:06:19.730 --> 00:06:21.920 align:middle line:84% Somehow calculating the derivative 00:06:21.920 --> 00:06:25.688 align:middle line:84% can be thought of as basically multiplying the path weights. 00:06:25.688 --> 00:06:28.230 align:middle line:84% If these were matrices, you have to do it in the right order. 00:06:28.230 --> 00:06:30.830 align:middle line:84% If it's scalars, you get to ignore that a little bit. 00:06:30.830 --> 00:06:33.530 align:middle line:84% But we want to multiply the path weights 00:06:33.530 --> 00:06:36.410 align:middle line:84% and add that up over all the paths. 00:06:36.410 --> 00:06:40.280 align:middle line:84% So whether it's forward or backwards or any other way, 00:06:40.280 --> 00:06:44.240 align:middle line:84% our goal is to look at all the paths between x and z 00:06:44.240 --> 00:06:47.022 align:middle line:90% and then multiply the weights. 00:06:47.022 --> 00:06:48.230 align:middle line:90% Here there's just one weight. 00:06:48.230 --> 00:06:51.650 align:middle line:84% And then add it up over all the paths. 00:06:51.650 --> 00:07:01.910 align:middle line:84% And so it's really the sum over paths from input i 00:07:01.910 --> 00:07:11.700 align:middle line:84% to output j of the product over the path of the weights. 00:07:11.700 --> 00:07:13.370 align:middle line:84% So I'll just kind of say it like that. 00:07:13.370 --> 00:07:15.140 align:middle line:90% I hope it's clear what is meant. 00:07:15.140 --> 00:07:17.480 align:middle line:84% That's really what's going on, right? 00:07:17.480 --> 00:07:20.270 align:middle line:84% And some people would tell you it's just the chain rule, 00:07:20.270 --> 00:07:22.440 align:middle line:90% and they'd be right. 00:07:22.440 --> 00:07:27.050 align:middle line:84% I'm not a big fan of just the chain rule because-- 00:07:27.050 --> 00:07:29.153 align:middle line:84% not that there's anything wrong about it. 00:07:29.153 --> 00:07:31.820 align:middle line:84% But I feel like you don't really see what's going on if you just 00:07:31.820 --> 00:07:33.690 align:middle line:90% say just the chain rule. 00:07:33.690 --> 00:07:36.230 align:middle line:84% But if you look at it on the graph, the computational graph, 00:07:36.230 --> 00:07:39.145 align:middle line:84% I think it's much easier to see what's going on. 00:07:39.145 --> 00:07:42.880 align:middle line:90% 00:07:42.880 --> 00:07:46.660 align:middle line:90% And it really doesn't matter-- 00:07:46.660 --> 00:07:49.130 align:middle line:84% if you really have the graph in front of you-- 00:07:49.130 --> 00:07:51.020 align:middle line:84% that's sort of the human thing to do-- 00:07:51.020 --> 00:07:55.340 align:middle line:84% it really doesn't matter if you multiply 00:07:55.340 --> 00:07:59.120 align:middle line:84% this times this times this or if you did it in the other order. 00:07:59.120 --> 00:08:00.980 align:middle line:90% It just doesn't matter. 00:08:00.980 --> 00:08:03.830 align:middle line:84% The product of the weights can happen in any order, 00:08:03.830 --> 00:08:08.180 align:middle line:84% and you can use associativity to your benefit any way you like. 00:08:08.180 --> 00:08:11.030 align:middle line:84% And just so it's completely clear, 00:08:11.030 --> 00:08:14.210 align:middle line:90% I drew this symbolically. 00:08:14.210 --> 00:08:16.130 align:middle line:84% But on any computer, when you're doing 00:08:16.130 --> 00:08:19.340 align:middle line:84% automatic differentiation, everything here is going to be, 00:08:19.340 --> 00:08:20.720 align:middle line:90% most of the time, a number-- 00:08:20.720 --> 00:08:22.910 align:middle line:90% I would say 99.9% of the time. 00:08:22.910 --> 00:08:25.220 align:middle line:84% All of the variable, everything that you would see 00:08:25.220 --> 00:08:28.280 align:middle line:90% would be a numerical value. 00:08:28.280 --> 00:08:30.568 align:middle line:84% So I just don't want any confusion that-- 00:08:30.568 --> 00:08:33.110 align:middle line:84% I did it symbolically because if I put a whole lot of numbers 00:08:33.110 --> 00:08:35.750 align:middle line:84% here, I don't think I'd make the point as well. 00:08:35.750 --> 00:08:39.950 align:middle line:84% But on a computer, when you do automatic differentiation, 00:08:39.950 --> 00:08:41.360 align:middle line:90% it's all numerical. 00:08:41.360 --> 00:08:43.669 align:middle line:90% It's not finite differences. 00:08:43.669 --> 00:08:45.350 align:middle line:84% The way you get to the edge weights 00:08:45.350 --> 00:08:46.880 align:middle line:90% are never finite differences. 00:08:46.880 --> 00:08:51.350 align:middle line:84% But it is all numbers, usually floating point numbers 00:08:51.350 --> 00:08:53.330 align:middle line:84% of double precision or something, 00:08:53.330 --> 00:08:57.740 align:middle line:84% or maybe not, but floating point numbers. 00:08:57.740 --> 00:08:58.670 align:middle line:90% OK. 00:08:58.670 --> 00:09:05.490 align:middle line:84% So then we talked about how this thing really works. 00:09:05.490 --> 00:09:10.130 align:middle line:84% And which of these do I want to tell you? 00:09:10.130 --> 00:09:12.660 align:middle line:90% So-- doo-doo-doo. 00:09:12.660 --> 00:09:14.410 align:middle line:84% STEVEN JOHNSON: You did forward last time. 00:09:14.410 --> 00:09:16.118 align:middle line:84% PROFESSOR: Yeah, I did forward last time. 00:09:16.118 --> 00:09:19.060 align:middle line:84% And maybe I could some up forward. 00:09:19.060 --> 00:09:20.485 align:middle line:90% I didn't say it last time. 00:09:20.485 --> 00:09:26.070 align:middle line:90% 00:09:26.070 --> 00:09:38.800 align:middle line:84% But maybe it's worth saying that when you do forward, 00:09:38.800 --> 00:09:42.460 align:middle line:84% you might have a node coming at your next step. 00:09:42.460 --> 00:09:43.800 align:middle line:90% So you're in the middle of-- 00:09:43.800 --> 00:09:47.910 align:middle line:84% you're in the middle of doing forward mode. 00:09:47.910 --> 00:09:51.120 align:middle line:84% And I'll draw a little picture like this. 00:09:51.120 --> 00:09:54.480 align:middle line:84% Here I only had a case where a point depended on two inputs. 00:09:54.480 --> 00:09:56.915 align:middle line:84% But I'll just draw a picture kind of more generally. 00:09:56.915 --> 00:09:58.290 align:middle line:84% I'll write it, like, five inputs. 00:09:58.290 --> 00:10:03.000 align:middle line:84% But this could also be dot dot dot of some sort. 00:10:03.000 --> 00:10:07.230 align:middle line:84% And I don't know what these variable names are. 00:10:07.230 --> 00:10:08.730 align:middle line:84% Maybe I'll just call them something. 00:10:08.730 --> 00:10:11.070 align:middle line:84% I don't want to call them x's or y's or z's. 00:10:11.070 --> 00:10:12.120 align:middle line:90% Maybe I'll call them a's. 00:10:12.120 --> 00:10:12.910 align:middle line:90% Let's do that. 00:10:12.910 --> 00:10:15.795 align:middle line:84% So I'm at a step, and I've got a bunch of a's. 00:10:15.795 --> 00:10:18.360 align:middle line:90% 00:10:18.360 --> 00:10:22.530 align:middle line:84% And I've got a b over here where b depends on a. 00:10:22.530 --> 00:10:29.258 align:middle line:84% And I'm really just using the fact 00:10:29.258 --> 00:10:32.370 align:middle line:84% that here I'm going to put all these derivatives over here. 00:10:32.370 --> 00:10:34.710 align:middle line:90% I'm going to put db da1. 00:10:34.710 --> 00:10:36.660 align:middle line:84% This is what I'm going to put on the labels-- 00:10:36.660 --> 00:10:38.460 align:middle line:90% db da2. 00:10:38.460 --> 00:10:42.315 align:middle line:84% This is just the one-step derivatives over here-- 00:10:42.315 --> 00:10:45.260 align:middle line:90% 00:10:45.260 --> 00:10:51.060 align:middle line:84% db da4, and this is 5, and so forth. 00:10:51.060 --> 00:10:52.910 align:middle line:90% I'm just going to put these. 00:10:52.910 --> 00:10:56.120 align:middle line:90% And I've got the fact-- 00:10:56.120 --> 00:10:58.460 align:middle line:84% let's say I'm starting at a particular x. 00:10:58.460 --> 00:11:00.140 align:middle line:90% Let's just call it x1. 00:11:00.140 --> 00:11:02.360 align:middle line:90% And I've worked my way over. 00:11:02.360 --> 00:11:06.200 align:middle line:84% And so at this point in time, I know a1, 00:11:06.200 --> 00:11:10.400 align:middle line:84% and I know da1 dx with a-- not the partial. 00:11:10.400 --> 00:11:12.320 align:middle line:84% But I know all of these values here, you see. 00:11:12.320 --> 00:11:15.890 align:middle line:90% I know da1 up to da5 dx. 00:11:15.890 --> 00:11:21.170 align:middle line:84% So what I know is the paths that take me 00:11:21.170 --> 00:11:23.660 align:middle line:90% from this input to here. 00:11:23.660 --> 00:11:26.540 align:middle line:84% That's the information I have right now. 00:11:26.540 --> 00:11:30.050 align:middle line:84% And what I'd like to do is figure out 00:11:30.050 --> 00:11:32.570 align:middle line:84% how to move forward on the graph. 00:11:32.570 --> 00:11:35.180 align:middle line:84% And so I'm just going to say this symbolically now. 00:11:35.180 --> 00:11:38.870 align:middle line:90% I want to be able to calculate-- 00:11:38.870 --> 00:11:40.595 align:middle line:84% my computer program will give me b. 00:11:40.595 --> 00:11:44.750 align:middle line:84% But the question is, how do I get db dx from the information 00:11:44.750 --> 00:11:45.770 align:middle line:90% I have right now? 00:11:45.770 --> 00:11:50.760 align:middle line:84% And exactly what's happening is I'm going to sum-- 00:11:50.760 --> 00:11:53.560 align:middle line:90% I'm going to be summing-- 00:11:53.560 --> 00:12:00.330 align:middle line:84% let's see, what is it?-- db dai times dai dx from i 00:12:00.330 --> 00:12:03.570 align:middle line:90% equals 1 to whatever, right? 00:12:03.570 --> 00:12:07.446 align:middle line:84% And that is the multivariate chain rule. 00:12:07.446 --> 00:12:12.120 align:middle line:84% This is really what's happening, that this is equal to this. 00:12:12.120 --> 00:12:14.670 align:middle line:84% And that's kind of why it all works, 00:12:14.670 --> 00:12:17.940 align:middle line:84% that you can proceed forward and keep working your way forward 00:12:17.940 --> 00:12:21.320 align:middle line:84% to get the derivative all the way to the end. 00:12:21.320 --> 00:12:24.440 align:middle line:84% When you do reverse mode, we're going 00:12:24.440 --> 00:12:27.560 align:middle line:84% to have arrows going the other way. 00:12:27.560 --> 00:12:30.050 align:middle line:84% And so we'll see how it all happens. 00:12:30.050 --> 00:12:35.240 align:middle line:84% But the arrows are going to be-- here, it's, 00:12:35.240 --> 00:12:36.698 align:middle line:90% what do we depend on? 00:12:36.698 --> 00:12:38.240 align:middle line:84% But what it's going to turn out to be 00:12:38.240 --> 00:12:41.570 align:middle line:84% the other way is, what are we influenced by, 00:12:41.570 --> 00:12:43.325 align:middle line:90% or what do we influence later? 00:12:43.325 --> 00:12:44.450 align:middle line:90% But we'll see reverse mode. 00:12:44.450 --> 00:12:45.742 align:middle line:90% But this is all forward so far. 00:12:45.742 --> 00:12:50.930 align:middle line:90% 00:12:50.930 --> 00:12:56.890 align:middle line:84% And an interesting fact about forward mode is-- 00:12:56.890 --> 00:12:59.950 align:middle line:84% a detail about forward mode is you 00:12:59.950 --> 00:13:03.300 align:middle line:84% don't need to create the graph as a data structure. 00:13:03.300 --> 00:13:04.300 align:middle line:90% Yes, there's a question? 00:13:04.300 --> 00:13:07.820 align:middle line:84% AUDIENCE: Just compute the partials [INAUDIBLE]?? 00:13:07.820 --> 00:13:09.620 align:middle line:84% PROFESSOR: Oh, well, that's-- so, OK, 00:13:09.620 --> 00:13:12.050 align:middle line:90% maybe we should talk about that. 00:13:12.050 --> 00:13:14.460 align:middle line:84% Yeah, let's talk about computing these partials. 00:13:14.460 --> 00:13:17.270 align:middle line:84% So the first answer I'm going to give you 00:13:17.270 --> 00:13:22.345 align:middle line:84% is the most simple basic answer, the "no cheating" answer. 00:13:22.345 --> 00:13:23.720 align:middle line:84% And then, in the real world, I'll 00:13:23.720 --> 00:13:26.990 align:middle line:84% tell you what the cheating answer is. 00:13:26.990 --> 00:13:31.470 align:middle line:84% So you could write every computer program in the world. 00:13:31.470 --> 00:13:33.060 align:middle line:84% So here, let's put this over here. 00:13:33.060 --> 00:13:35.060 align:middle line:84% I'm going to need this for reverse mode as well. 00:13:35.060 --> 00:13:42.430 align:middle line:84% So every computer program in the world 00:13:42.430 --> 00:13:48.100 align:middle line:84% basically it can come down to plus, minus, times, and divide. 00:13:48.100 --> 00:13:50.170 align:middle line:84% I could probably even remove some of these. 00:13:50.170 --> 00:13:53.140 align:middle line:84% But every computer program in the world 00:13:53.140 --> 00:13:56.890 align:middle line:84% can be brought down to plus, minus, times, and divide. 00:13:56.890 --> 00:14:00.190 align:middle line:84% When you do a sine function or an exponential function, 00:14:00.190 --> 00:14:03.550 align:middle line:84% there's some power series or maybe a ratio of power series, 00:14:03.550 --> 00:14:06.007 align:middle line:84% and it's not the one you learned in calculus. 00:14:06.007 --> 00:14:07.090 align:middle line:90% So it's not Taylor series. 00:14:07.090 --> 00:14:08.890 align:middle line:84% I don't care what your calculus teacher told you. 00:14:08.890 --> 00:14:10.225 align:middle line:90% It's never done by that way. 00:14:10.225 --> 00:14:13.000 align:middle line:90% 00:14:13.000 --> 00:14:15.850 align:middle line:84% But nonetheless, there's some sort of power series or ratio, 00:14:15.850 --> 00:14:19.210 align:middle line:84% a power series, usually, somewhere in there. 00:14:19.210 --> 00:14:22.210 align:middle line:84% And in the end, everything you ever do on a computer 00:14:22.210 --> 00:14:23.860 align:middle line:90% can come down to this. 00:14:23.860 --> 00:14:27.007 align:middle line:84% There's also conditionals and other things. 00:14:27.007 --> 00:14:28.840 align:middle line:84% But as far as all the arithmetic operations, 00:14:28.840 --> 00:14:31.720 align:middle line:84% it can be reduced to plus, minus, times, and divide. 00:14:31.720 --> 00:14:36.640 align:middle line:84% So in fact, all you need are the partials for these. 00:14:36.640 --> 00:14:40.570 align:middle line:84% So, for example, all you really need, 00:14:40.570 --> 00:14:45.220 align:middle line:84% theoretically, is the ability to do-- 00:14:45.220 --> 00:14:45.730 align:middle line:90% right? 00:14:45.730 --> 00:14:51.400 align:middle line:84% So if b is a1 plus a2, the only two partials you need are this. 00:14:51.400 --> 00:14:53.290 align:middle line:84% In fact, let's put a plus or minus, 00:14:53.290 --> 00:14:55.960 align:middle line:84% and I'll cover two of these cases right there. 00:14:55.960 --> 00:14:57.460 align:middle line:90% And what else is there? 00:14:57.460 --> 00:14:59.120 align:middle line:90% There's times. 00:14:59.120 --> 00:15:04.330 align:middle line:84% So if b is equal to a1 a2, then all I need is this. 00:15:04.330 --> 00:15:06.790 align:middle line:84% So I just need to know that, right? 00:15:06.790 --> 00:15:09.970 align:middle line:84% And this is, by the way, what I did in that little notebook 00:15:09.970 --> 00:15:13.630 align:middle line:84% where I kind of did the magic of forward mode differentiation. 00:15:13.630 --> 00:15:17.560 align:middle line:84% And then, finally, there's the division one. 00:15:17.560 --> 00:15:20.530 align:middle line:90% And the partials are-- 00:15:20.530 --> 00:15:22.630 align:middle line:84% I mean, this is maybe a slightly different point 00:15:22.630 --> 00:15:27.190 align:middle line:84% of view of the quotient rule and the product rule. 00:15:27.190 --> 00:15:30.040 align:middle line:84% But if you actually step back and look at it, 00:15:30.040 --> 00:15:33.580 align:middle line:84% this is exactly the sum and difference rule, the product 00:15:33.580 --> 00:15:37.410 align:middle line:84% rule, and I'm about to write the quotient rule down. 00:15:37.410 --> 00:15:37.910 align:middle line:90% Let's see. 00:15:37.910 --> 00:15:42.350 align:middle line:84% So the partial with respect to a1 is 1 over a2. 00:15:42.350 --> 00:15:46.220 align:middle line:84% And the partial with respect to a2 is minus a1 over a2 squared. 00:15:46.220 --> 00:15:51.200 align:middle line:84% And I hope you can see that the information that you see here 00:15:51.200 --> 00:15:54.470 align:middle line:84% is exactly the same information as the product rule. 00:15:54.470 --> 00:15:59.450 align:middle line:90% It's a2 da1 plus a1 da2. 00:15:59.450 --> 00:16:01.940 align:middle line:90% It's 1 over a2. 00:16:01.940 --> 00:16:05.310 align:middle line:84% This is the denominator times the derivative of the numerator 00:16:05.310 --> 00:16:05.910 align:middle line:90% or whatever. 00:16:05.910 --> 00:16:08.880 align:middle line:84% I guess you might want to write this as a2 00:16:08.880 --> 00:16:11.667 align:middle line:84% over a2 squared just to see that it's the same thing. 00:16:11.667 --> 00:16:14.250 align:middle line:84% But it's the denominator times the derivative of the numerator 00:16:14.250 --> 00:16:16.090 align:middle line:84% minus the numerator times derivative of-- you see, 00:16:16.090 --> 00:16:17.400 align:middle line:84% so it's all the information you've known 00:16:17.400 --> 00:16:18.840 align:middle line:90% in the old days of calculus. 00:16:18.840 --> 00:16:21.640 align:middle line:84% I'm just writing it in a slightly different way. 00:16:21.640 --> 00:16:25.050 align:middle line:84% If you have these rules, and your computer program only 00:16:25.050 --> 00:16:28.710 align:middle line:84% has those things, that's all you need, right? 00:16:28.710 --> 00:16:31.050 align:middle line:84% If that's what your program breaks up to atomically, 00:16:31.050 --> 00:16:33.840 align:middle line:84% you can actually do forward mode differentiation completely. 00:16:33.840 --> 00:16:37.240 align:middle line:84% Now, of course, that's not how it really 00:16:37.240 --> 00:16:38.240 align:middle line:90% works in the real world. 00:16:38.240 --> 00:16:41.020 align:middle line:84% But I want everybody to see that in theory, this 00:16:41.020 --> 00:16:42.200 align:middle line:90% is all you need. 00:16:42.200 --> 00:16:46.280 align:middle line:84% So in practice, we know lots of derivatives. 00:16:46.280 --> 00:16:48.070 align:middle line:84% We know the derivative of sine is cosine. 00:16:48.070 --> 00:16:51.550 align:middle line:84% We know the derivative of log x is 1 over x. 00:16:51.550 --> 00:16:56.620 align:middle line:84% So we throw them in so you don't have to waste your time. 00:16:56.620 --> 00:16:59.800 align:middle line:84% And Julia-- I think Steven mentioned 00:16:59.800 --> 00:17:02.920 align:middle line:84% in the Piazza or somewhere about chainrules.jl. 00:17:02.920 --> 00:17:06.940 align:middle line:84% So lots and lots of derivatives that are known get 00:17:06.940 --> 00:17:10.810 align:middle line:84% stuck in so that you can do this at a higher level. 00:17:10.810 --> 00:17:13.990 align:middle line:84% So instead of letting the computer derive the derivative 00:17:13.990 --> 00:17:16.430 align:middle line:84% of sine x with all those pluses and minuses, 00:17:16.430 --> 00:17:19.450 align:middle line:84% it could just know the derivative of sine is cosine. 00:17:19.450 --> 00:17:22.972 align:middle line:84% STEVEN JOHNSON: And as you also find in your problem set-- 00:17:22.972 --> 00:17:24.639 align:middle line:84% PROFESSOR: Something was missing, right? 00:17:24.639 --> 00:17:27.990 align:middle line:84% STEVEN JOHNSON: And sometimes it fails because it calls out-- 00:17:27.990 --> 00:17:30.850 align:middle line:84% so if forwarddiff relies on this dual number type, 00:17:30.850 --> 00:17:32.860 align:middle line:84% it works as long as you have Julia code that 00:17:32.860 --> 00:17:34.450 align:middle line:90% works with any number type. 00:17:34.450 --> 00:17:36.290 align:middle line:84% If you call out to a FORTRAN library that 00:17:36.290 --> 00:17:39.200 align:middle line:84% only works for double precision floating point numbers, 00:17:39.200 --> 00:17:41.400 align:middle line:84% then it can't differentiate that. 00:17:41.400 --> 00:17:43.700 align:middle line:84% So then someone has to write a special rule. 00:17:43.700 --> 00:17:46.245 align:middle line:84% And if they didn't, then it fails and probably 00:17:46.245 --> 00:17:48.245 align:middle line:84% complains that you if you try to use forwarddiff 00:17:48.245 --> 00:17:50.670 align:middle line:84% on [? the matrix ?] [INAUDIBLE] focusing. 00:17:50.670 --> 00:17:54.276 align:middle line:84% PROFESSOR: So that's the problem with calling-- 00:17:54.276 --> 00:17:56.550 align:middle line:84% if only we can get everybody to consistently write 00:17:56.550 --> 00:17:59.435 align:middle line:84% all their code in Julia, you would never have that problem. 00:17:59.435 --> 00:18:01.560 align:middle line:84% AUDIENCE: I mean, one solution is if you're Google, 00:18:01.560 --> 00:18:03.420 align:middle line:84% and you just hire a small army of engineers 00:18:03.420 --> 00:18:05.587 align:middle line:84% to just re-implement all the math libraries you ever 00:18:05.587 --> 00:18:08.220 align:middle line:84% need in Jax, which is basically [INAUDIBLE] implement 00:18:08.220 --> 00:18:11.230 align:middle line:84% NumPy, [INAUDIBLE],, SciPy, and so forth. 00:18:11.230 --> 00:18:13.855 align:middle line:84% And then, as long as you stay in that closed universe of things 00:18:13.855 --> 00:18:15.772 align:middle line:84% that people have written derivative rules for, 00:18:15.772 --> 00:18:16.290 align:middle line:90% you're fine. 00:18:16.290 --> 00:18:18.750 align:middle line:84% PROFESSOR: So I'll tell a little story. 00:18:18.750 --> 00:18:20.537 align:middle line:84% Before I really grew to understand-- 00:18:20.537 --> 00:18:22.620 align:middle line:84% automatic differentiation has only gotten exciting 00:18:22.620 --> 00:18:23.787 align:middle line:90% in the last couple of years. 00:18:23.787 --> 00:18:27.290 align:middle line:84% And as I was just starting to poke my head into it, 00:18:27.290 --> 00:18:31.470 align:middle line:84% I differentiated the eigenvalue problem. 00:18:31.470 --> 00:18:34.440 align:middle line:84% Besides the FORTRAN libraries, the LAPACK that-- 00:18:34.440 --> 00:18:35.580 align:middle line:90% it's probably C now anyway. 00:18:35.580 --> 00:18:39.073 align:middle line:84% But besides the LAPACK library, there's 00:18:39.073 --> 00:18:41.490 align:middle line:84% another implementation of the symmetric eigenvalue problem 00:18:41.490 --> 00:18:43.590 align:middle line:90% in genericlinearalgebra.jl. 00:18:43.590 --> 00:18:45.630 align:middle line:90% It's written in all Julia. 00:18:45.630 --> 00:18:47.340 align:middle line:90% And it actually works beyond-- 00:18:47.340 --> 00:18:49.620 align:middle line:84% it works for quaternions and weird number systems 00:18:49.620 --> 00:18:52.510 align:middle line:84% and finite fields and not just plain old 00:18:52.510 --> 00:18:54.670 align:middle line:90% floating point numbers. 00:18:54.670 --> 00:18:57.820 align:middle line:84% And so you can differentiate through it 00:18:57.820 --> 00:19:00.040 align:middle line:84% because it's all Julia, and it got the right answer. 00:19:00.040 --> 00:19:03.142 align:middle line:84% And I was talking to a friend of mine. 00:19:03.142 --> 00:19:04.600 align:middle line:84% What was he what was the package he 00:19:04.600 --> 00:19:08.320 align:middle line:84% was excited about with its automatic differentiation? 00:19:08.320 --> 00:19:10.090 align:middle line:90% Oh, it's a statistics thing. 00:19:10.090 --> 00:19:12.560 align:middle line:90% Oh, what was it? 00:19:12.560 --> 00:19:15.610 align:middle line:84% Oh, I'm just blanking on the name. 00:19:15.610 --> 00:19:17.250 align:middle line:90% But it's a very well-known one. 00:19:17.250 --> 00:19:19.053 align:middle line:90% Anyway, it doesn't matter. 00:19:19.053 --> 00:19:20.970 align:middle line:84% I was talking to a friend of mine, and I said, 00:19:20.970 --> 00:19:23.520 align:middle line:84% can you differentiate the symmetric eigenvalue problem? 00:19:23.520 --> 00:19:26.220 align:middle line:84% And he said to me, oh, well, we can, 00:19:26.220 --> 00:19:27.900 align:middle line:90% but we have to teach the system. 00:19:27.900 --> 00:19:29.760 align:middle line:90% And I thought that was crazy. 00:19:29.760 --> 00:19:31.320 align:middle line:90% Having to teach the system-- 00:19:31.320 --> 00:19:32.920 align:middle line:90% we really can't right now. 00:19:32.920 --> 00:19:34.860 align:middle line:84% But if we threw in a new rule, then we can. 00:19:34.860 --> 00:19:39.490 align:middle line:84% But I now realize that everybody does that, right? 00:19:39.490 --> 00:19:43.770 align:middle line:84% It's actually reasonable from a kind 00:19:43.770 --> 00:19:45.040 align:middle line:90% of a complexity point of view. 00:19:45.040 --> 00:19:46.400 align:middle line:84% So you don't have to actually drill down 00:19:46.400 --> 00:19:47.460 align:middle line:90% from every operation. 00:19:47.460 --> 00:19:48.900 align:middle line:84% But I was impressed that Julia was 00:19:48.900 --> 00:19:51.990 align:middle line:84% able to just get the answer of the derivative 00:19:51.990 --> 00:19:53.460 align:middle line:84% of the symmetric eigenvalue problem 00:19:53.460 --> 00:19:56.620 align:middle line:84% without being taught how to do it. 00:19:56.620 --> 00:19:59.650 align:middle line:84% It actually just did it using just these rules. 00:19:59.650 --> 00:20:02.920 align:middle line:84% And I thought that was kind of cool when I first saw that. 00:20:02.920 --> 00:20:06.090 align:middle line:84% OK, so this is really all that you need. 00:20:06.090 --> 00:20:10.290 align:middle line:84% But as I say, in practice, in practice, 00:20:10.290 --> 00:20:12.630 align:middle line:90% we've added lots of things. 00:20:12.630 --> 00:20:16.710 align:middle line:90% 00:20:16.710 --> 00:20:20.970 align:middle line:84% For example, if b is equal to the sine of a, 00:20:20.970 --> 00:20:26.210 align:middle line:84% then this edge with cosine a is just-- 00:20:26.210 --> 00:20:29.070 align:middle line:90% it comes on the edges. 00:20:29.070 --> 00:20:34.170 align:middle line:84% OK, so-- what are you doing there? 00:20:34.170 --> 00:20:34.670 align:middle line:90% It's-- 00:20:34.670 --> 00:20:35.626 align:middle line:84% AUDIENCE: Just-- we're all right. 00:20:35.626 --> 00:20:37.293 align:middle line:84% It's too small to have the [INAUDIBLE].. 00:20:37.293 --> 00:20:39.070 align:middle line:84% PROFESSOR: Oh, yeah, that's probably true. 00:20:39.070 --> 00:20:41.080 align:middle line:84% OK, maybe the last thing I'll say 00:20:41.080 --> 00:20:43.117 align:middle line:90% before I take a break, and-- 00:20:43.117 --> 00:20:44.950 align:middle line:84% the break may be slightly longer because I'm 00:20:44.950 --> 00:20:46.870 align:middle line:84% going to put a sweater on this little guy. 00:20:46.870 --> 00:20:48.100 align:middle line:84% And that'll take me a couple of more minutes. 00:20:48.100 --> 00:20:49.642 align:middle line:84% But he's really cute in that sweater. 00:20:49.642 --> 00:20:51.830 align:middle line:90% So you'll all like that. 00:20:51.830 --> 00:20:53.620 align:middle line:84% So the other thing I wanted to say 00:20:53.620 --> 00:20:57.740 align:middle line:84% is when you do forward mode, you might ask yourself, 00:20:57.740 --> 00:21:00.602 align:middle line:84% where is this graph on the computer? 00:21:00.602 --> 00:21:02.560 align:middle line:84% I mean, you know, of course, on the blackboard, 00:21:02.560 --> 00:21:04.060 align:middle line:84% you can visually look at this graph, 00:21:04.060 --> 00:21:05.450 align:middle line:90% and you can see the paths. 00:21:05.450 --> 00:21:07.690 align:middle line:84% So where is this graph on the computer? 00:21:07.690 --> 00:21:11.780 align:middle line:84% Well, in some sense, when you do forward mode, 00:21:11.780 --> 00:21:14.210 align:middle line:84% one way to say it is you don't need the graph at all. 00:21:14.210 --> 00:21:16.700 align:middle line:84% Maybe a better way to say it is the graph 00:21:16.700 --> 00:21:21.230 align:middle line:84% is kind of being built up as software executes, in effect. 00:21:21.230 --> 00:21:25.370 align:middle line:84% As you execute, as you execute sine x, 00:21:25.370 --> 00:21:30.110 align:middle line:84% there's nothing stopping you from creating that cosine x 00:21:30.110 --> 00:21:34.400 align:middle line:84% and then getting that second part of the dual number 00:21:34.400 --> 00:21:35.180 align:middle line:90% as you go. 00:21:35.180 --> 00:21:39.620 align:middle line:84% And so you see as you execute, we can start out with x1. 00:21:39.620 --> 00:21:41.240 align:middle line:90% You get to over here. 00:21:41.240 --> 00:21:45.380 align:middle line:84% We've got a, which is sine x and then cosine x. 00:21:45.380 --> 00:21:47.700 align:middle line:84% And you just can keep moving along. 00:21:47.700 --> 00:21:51.380 align:middle line:84% So in some sense, I think this is worth 00:21:51.380 --> 00:21:52.460 align:middle line:90% thinking about because-- 00:21:52.460 --> 00:21:54.560 align:middle line:84% I don't know if it's completely obvious or not. 00:21:54.560 --> 00:21:58.580 align:middle line:84% But as the computer executes, as it actually 00:21:58.580 --> 00:22:03.620 align:middle line:84% calculates this from this, it can also 00:22:03.620 --> 00:22:07.310 align:middle line:84% keep that path, the product of the weights going forward, 00:22:07.310 --> 00:22:09.900 align:middle line:84% just by multiplying-- at this moment in time, 00:22:09.900 --> 00:22:10.880 align:middle line:90% you can get this. 00:22:10.880 --> 00:22:13.100 align:middle line:84% And just as you need it, you can multiply it 00:22:13.100 --> 00:22:14.810 align:middle line:90% by what you have here. 00:22:14.810 --> 00:22:18.200 align:middle line:84% And so that's why overloading works when you do forward mode. 00:22:18.200 --> 00:22:20.210 align:middle line:84% You take your original computer program. 00:22:20.210 --> 00:22:22.880 align:middle line:84% You let those dual numbers propagate forward. 00:22:22.880 --> 00:22:25.460 align:middle line:84% And you are literally, as you get there, 00:22:25.460 --> 00:22:28.490 align:middle line:84% you're getting the right value for-- 00:22:28.490 --> 00:22:31.820 align:middle line:84% so you got this-- you have all the values 00:22:31.820 --> 00:22:35.420 align:middle line:84% for the sum over the paths-- of the product of the paths 00:22:35.420 --> 00:22:37.070 align:middle line:90% to this point. 00:22:37.070 --> 00:22:39.980 align:middle line:84% And so you don't need to go crazy to actually build this 00:22:39.980 --> 00:22:42.440 align:middle line:90% as some weird data structure. 00:22:42.440 --> 00:22:44.910 align:middle line:84% You just go forward, and it's there. 00:22:44.910 --> 00:22:47.960 align:middle line:84% However, no such luck with reverse mode. 00:22:47.960 --> 00:22:50.600 align:middle line:84% So after the break, after I put on his little sweater 00:22:50.600 --> 00:22:52.100 align:middle line:84% and come back, what I'm going to do 00:22:52.100 --> 00:22:54.950 align:middle line:84% is discuss reverse mode in some real detail. 00:22:54.950 --> 00:22:58.910 align:middle line:84% And I find it very interesting to see 00:22:58.910 --> 00:23:02.960 align:middle line:84% in what sense reverse mode is the symmetric version 00:23:02.960 --> 00:23:03.807 align:middle line:90% of forward mode. 00:23:03.807 --> 00:23:05.390 align:middle line:84% It's just looking at it in the mirror. 00:23:05.390 --> 00:23:08.420 align:middle line:84% And what is it about computer software which kind of 00:23:08.420 --> 00:23:10.250 align:middle line:90% makes it asymmetric? 00:23:10.250 --> 00:23:13.850 align:middle line:84% From a pure math point of view, reverse mode and forward mode 00:23:13.850 --> 00:23:15.740 align:middle line:84% are just mirror images of each other. 00:23:15.740 --> 00:23:19.520 align:middle line:84% But because of the realities of how computers execute, 00:23:19.520 --> 00:23:22.670 align:middle line:84% then they are no longer exactly mirror images. 00:23:22.670 --> 00:23:25.370 align:middle line:84% And reverse mode seems a little bit harder 00:23:25.370 --> 00:23:26.360 align:middle line:90% to actually implement. 00:23:26.360 --> 00:23:30.300 align:middle line:84% It's not hard, but it feels a little bit-- 00:23:30.300 --> 00:23:34.830 align:middle line:84% it just feels a little bit less intuitive because-- 00:23:34.830 --> 00:23:37.770 align:middle line:84% it's ultimately because computer programs execute forward. 00:23:37.770 --> 00:23:39.660 align:middle line:90% What else are they going to do? 00:23:39.660 --> 00:23:42.300 align:middle line:84% If you could only run computer programs backwards, 00:23:42.300 --> 00:23:44.790 align:middle line:90% then you wouldn't have to-- 00:23:44.790 --> 00:23:46.140 align:middle line:90% it doesn't mean anything but. 00:23:46.140 --> 00:23:47.620 align:middle line:84% But I just want to make that point. 00:23:47.620 --> 00:23:48.120 align:middle line:90% All right. 00:23:48.120 --> 00:23:48.960 align:middle line:90% So we take a break. 00:23:48.960 --> 00:23:50.730 align:middle line:84% It might be slightly more than five minutes. 00:23:50.730 --> 00:23:52.605 align:middle line:84% But I'm going to take this young man outside. 00:23:52.605 --> 00:23:57.280 align:middle line:90% 00:23:57.280 --> 00:24:00.270 align:middle line:84% So I think the next thing to emphasize 00:24:00.270 --> 00:24:05.580 align:middle line:84% as I go from forward to reverse is to really focus on, perhaps, 00:24:05.580 --> 00:24:11.280 align:middle line:84% this basic picture where we talk about this variable I'm 00:24:11.280 --> 00:24:12.470 align:middle line:90% calling b. 00:24:12.470 --> 00:24:17.880 align:middle line:84% And it's all of the things that I depend on. 00:24:17.880 --> 00:24:20.797 align:middle line:84% And maybe I should actually think a little bit. 00:24:20.797 --> 00:24:22.380 align:middle line:84% I'm going to get confused in a second. 00:24:22.380 --> 00:24:25.480 align:middle line:84% Did I actually put this in the order I wanted or not? 00:24:25.480 --> 00:24:27.690 align:middle line:90% So let's see. 00:24:27.690 --> 00:24:32.130 align:middle line:84% If these were matrices, I want them to-- 00:24:32.130 --> 00:24:34.233 align:middle line:84% I would want them to be multiplied. 00:24:34.233 --> 00:24:36.150 align:middle line:84% Oh, god, I'm going to get completely confused. 00:24:36.150 --> 00:24:36.540 align:middle line:90% You know what? 00:24:36.540 --> 00:24:37.582 align:middle line:90% We're not doing matrices. 00:24:37.582 --> 00:24:39.290 align:middle line:84% I'll think about it offline and let you-- 00:24:39.290 --> 00:24:40.915 align:middle line:84% unless you know the answer off the top, 00:24:40.915 --> 00:24:42.510 align:middle line:84% but I wanted it this way or this way. 00:24:42.510 --> 00:24:44.290 align:middle line:90% But we're doing scalars today. 00:24:44.290 --> 00:24:45.360 align:middle line:90% So I won't have to think. 00:24:45.360 --> 00:24:47.400 align:middle line:84% All right, but the key point is-- 00:24:47.400 --> 00:24:49.215 align:middle line:90% AUDIENCE: Yeah, you [INAUDIBLE]. 00:24:49.215 --> 00:24:50.340 align:middle line:90% PROFESSOR: Oh, I got lucky. 00:24:50.340 --> 00:24:51.810 align:middle line:90% OK. 00:24:51.810 --> 00:24:52.410 align:middle line:90% All right. 00:24:52.410 --> 00:25:00.330 align:middle line:84% So here it's every node that b depends on. 00:25:00.330 --> 00:25:05.180 align:middle line:90% 00:25:05.180 --> 00:25:11.740 align:middle line:84% So when you're going backwards, what you need 00:25:11.740 --> 00:25:14.900 align:middle line:90% is everybody that you influence. 00:25:14.900 --> 00:25:20.215 align:middle line:84% So to draw that as a picture, I have a node a. 00:25:20.215 --> 00:25:24.290 align:middle line:84% And it's going to influence a whole bunch of nodes. 00:25:24.290 --> 00:25:26.510 align:middle line:90% Let's call these b1, b2. 00:25:26.510 --> 00:25:28.520 align:middle line:90% I'll just stick with my-- 00:25:28.520 --> 00:25:32.880 align:middle line:90% I'll stick with my running 5. 00:25:32.880 --> 00:25:35.420 align:middle line:84% But, no, there's nothing special about 5 here. 00:25:35.420 --> 00:25:44.490 align:middle line:84% So in your graph, the graph is still going left to right. 00:25:44.490 --> 00:25:47.730 align:middle line:84% But you have a node, and somewhere along the line, 00:25:47.730 --> 00:25:51.760 align:middle line:84% you can say all of the nodes that a influences-- 00:25:51.760 --> 00:25:55.810 align:middle line:84% for example, influences in one step, right? 00:25:55.810 --> 00:25:58.750 align:middle line:84% So x is influencing in one step here, 00:25:58.750 --> 00:26:01.900 align:middle line:84% and x is influencing z in one step. 00:26:01.900 --> 00:26:05.570 align:middle line:84% And so if that node over there was x, 00:26:05.570 --> 00:26:09.130 align:middle line:90% I would have two output arrows. 00:26:09.130 --> 00:26:12.320 align:middle line:90% This is every node-- 00:26:12.320 --> 00:26:16.650 align:middle line:84% so every node, I'll say, influenced 00:26:16.650 --> 00:26:27.070 align:middle line:84% by a or, maybe more simply, every node that depends on a. 00:26:27.070 --> 00:26:32.140 align:middle line:90% 00:26:32.140 --> 00:26:37.460 align:middle line:84% So that's what one has to look at there. 00:26:37.460 --> 00:26:40.390 align:middle line:84% And what we're going to do is we're 00:26:40.390 --> 00:26:46.070 align:middle line:84% going to have a final output variable z, where everything 00:26:46.070 --> 00:26:47.390 align:middle line:90% is going to start. 00:26:47.390 --> 00:26:50.443 align:middle line:84% And it wouldn't surprise you all that we're 00:26:50.443 --> 00:26:51.860 align:middle line:84% going to start-- we're still going 00:26:51.860 --> 00:26:53.902 align:middle line:84% to have something that looks like an ordered pair 00:26:53.902 --> 00:26:56.540 align:middle line:90% of numbers. 00:26:56.540 --> 00:26:59.060 align:middle line:84% But we're going to work our way this way. 00:26:59.060 --> 00:27:02.090 align:middle line:84% So we're going to do all the multiplications from right 00:27:02.090 --> 00:27:03.560 align:middle line:90% to left. 00:27:03.560 --> 00:27:09.180 align:middle line:84% And at the end, over here, we're going to have-- 00:27:09.180 --> 00:27:11.300 align:middle line:90% let's say I had one x over here. 00:27:11.300 --> 00:27:13.160 align:middle line:84% When we're finally finished, we're 00:27:13.160 --> 00:27:19.460 align:middle line:90% going to compute x and dz dx. 00:27:19.460 --> 00:27:24.800 align:middle line:84% So that's our goal is to start at this end 00:27:24.800 --> 00:27:29.510 align:middle line:84% and compute the derivative-- in other words, 00:27:29.510 --> 00:27:31.430 align:middle line:90% compute the exact same thing. 00:27:31.430 --> 00:27:34.250 align:middle line:84% But just to remind you, what we did 00:27:34.250 --> 00:27:39.230 align:middle line:84% here was we started with x, comma, 1 this way. 00:27:39.230 --> 00:27:45.890 align:middle line:84% And at the very end, we ended up with z and dz dx over here, OK? 00:27:45.890 --> 00:27:50.270 align:middle line:84% So that's your first kind of peek at how reverse mode will 00:27:50.270 --> 00:27:50.990 align:middle line:90% actually work. 00:27:50.990 --> 00:27:56.368 align:middle line:90% 00:27:56.368 --> 00:28:00.090 align:middle line:84% And the way it's going to work is here 00:28:00.090 --> 00:28:03.870 align:middle line:84% we're going to presume that we have the-- 00:28:03.870 --> 00:28:07.620 align:middle line:84% we're going to have the b's and the-- 00:28:07.620 --> 00:28:11.040 align:middle line:84% we're going to have the full derivatives-- 00:28:11.040 --> 00:28:13.710 align:middle line:90% so dz db2. 00:28:13.710 --> 00:28:16.320 align:middle line:84% So in the middle of the algorithm, 00:28:16.320 --> 00:28:19.605 align:middle line:84% we're assuming that we have all of these things. 00:28:19.605 --> 00:28:24.075 align:middle line:90% 00:28:24.075 --> 00:28:26.850 align:middle line:84% Sorry, these are not the partials. 00:28:26.850 --> 00:28:32.850 align:middle line:84% We have the derivatives all the way from z to where we are now. 00:28:32.850 --> 00:28:36.690 align:middle line:84% And our goal is to figure out what to put here. 00:28:36.690 --> 00:28:41.550 align:middle line:84% And so I hope you could see that if you can always go-- 00:28:41.550 --> 00:28:44.190 align:middle line:84% if you just know what to put here, 00:28:44.190 --> 00:28:48.170 align:middle line:84% then you can actually go from right to left. 00:28:48.170 --> 00:28:54.530 align:middle line:84% And so our goal is to figure out dz da. 00:28:54.530 --> 00:28:58.310 align:middle line:84% So if we can only have a good formula for dz da in terms 00:28:58.310 --> 00:29:01.340 align:middle line:84% of all of these pieces, we would have no trouble 00:29:01.340 --> 00:29:04.140 align:middle line:90% being able to do this. 00:29:04.140 --> 00:29:08.230 align:middle line:84% And, of course, the way this is going to work is-- 00:29:08.230 --> 00:29:13.090 align:middle line:84% what we're going to do is we're going to sum up now-- 00:29:13.090 --> 00:29:13.720 align:middle line:90% let's see. 00:29:13.720 --> 00:29:14.720 align:middle line:90% How is this going to go? 00:29:14.720 --> 00:29:24.390 align:middle line:84% Is it going to be dz dbi, dbi da-- 00:29:24.390 --> 00:29:26.082 align:middle line:84% from i equals 1 to 5 or something. 00:29:26.082 --> 00:29:28.040 align:middle line:84% AUDIENCE: Do you need a transpose [INAUDIBLE]?? 00:29:28.040 --> 00:29:29.720 align:middle line:84% PROFESSOR: No, I'm not in scalars. 00:29:29.720 --> 00:29:31.820 align:middle line:84% But, yes, I mean, ultimately, there'll 00:29:31.820 --> 00:29:34.220 align:middle line:90% be some sort of transposer. 00:29:34.220 --> 00:29:36.650 align:middle line:84% Or I have to write it in the right direction as an alt-- 00:29:36.650 --> 00:29:39.138 align:middle line:84% AUDIENCE: That's from the outputs on the left, right? 00:29:39.138 --> 00:29:40.590 align:middle line:90% [INAUDIBLE] 00:29:40.590 --> 00:29:41.460 align:middle line:90% PROFESSOR: Yeah. 00:29:41.460 --> 00:29:46.890 align:middle line:84% But I'm going to skip that for now, just to not complicate it. 00:29:46.890 --> 00:29:48.330 align:middle line:90% But Steven is just right. 00:29:48.330 --> 00:29:51.490 align:middle line:90% 00:29:51.490 --> 00:29:58.120 align:middle line:84% And you see that this is also a chain rule. 00:29:58.120 --> 00:30:01.030 align:middle line:90% 00:30:01.030 --> 00:30:02.740 align:middle line:84% But also, I want you to really see 00:30:02.740 --> 00:30:06.460 align:middle line:84% that from a graph point of view, all that's happening 00:30:06.460 --> 00:30:09.820 align:middle line:84% is we're making sure that what we put here 00:30:09.820 --> 00:30:12.280 align:middle line:84% is the path all the way to the right 00:30:12.280 --> 00:30:15.040 align:middle line:84% multiplied by what's on this edge. 00:30:15.040 --> 00:30:16.520 align:middle line:90% And then just add them all up. 00:30:16.520 --> 00:30:20.740 align:middle line:84% So you could see it the calculus way 00:30:20.740 --> 00:30:25.150 align:middle line:84% as these are two versions of the multivariate chain rule. 00:30:25.150 --> 00:30:27.010 align:middle line:84% I don't prefer that view, by the way. 00:30:27.010 --> 00:30:27.940 align:middle line:90% But some people do. 00:30:27.940 --> 00:30:29.140 align:middle line:90% And it's not wrong. 00:30:29.140 --> 00:30:32.200 align:middle line:84% Or you could see it as we're just 00:30:32.200 --> 00:30:34.300 align:middle line:84% multiplying the sums of all these weights, 00:30:34.300 --> 00:30:37.103 align:middle line:84% and you notice, huh, by the time I get here, 00:30:37.103 --> 00:30:38.770 align:middle line:84% I'm actually calculating the same thing. 00:30:38.770 --> 00:30:40.690 align:middle line:84% I'm just getting the sum over all the paths 00:30:40.690 --> 00:30:41.955 align:middle line:90% of all the weights. 00:30:41.955 --> 00:30:43.330 align:middle line:84% And, again, you get to look at it 00:30:43.330 --> 00:30:44.830 align:middle line:90% both ways, whatever you like. 00:30:44.830 --> 00:30:47.650 align:middle line:90% But both are absolutely correct. 00:30:47.650 --> 00:30:52.060 align:middle line:90% And so you have no problem. 00:30:52.060 --> 00:30:54.610 align:middle line:84% So you just keep moving your way through the graph, 00:30:54.610 --> 00:30:56.360 align:middle line:90% and you form these sums. 00:30:56.360 --> 00:31:00.560 align:middle line:90% So any questions about that? 00:31:00.560 --> 00:31:04.330 align:middle line:84% So the next thing I want to talk about is a bit of a detail 00:31:04.330 --> 00:31:11.785 align:middle line:84% as to, how do you actually, in practice, compute these sums? 00:31:11.785 --> 00:31:15.386 align:middle line:90% 00:31:15.386 --> 00:31:19.710 align:middle line:84% You see a sum like this, and perhaps your temptation 00:31:19.710 --> 00:31:24.150 align:middle line:84% is to form this entire sum at once. 00:31:24.150 --> 00:31:25.650 align:middle line:90% But you don't have to. 00:31:25.650 --> 00:31:28.840 align:middle line:84% You can traverse these arrows one at a time. 00:31:28.840 --> 00:31:34.680 align:middle line:84% So, for example, you start out with-- 00:31:34.680 --> 00:31:37.830 align:middle line:84% so in practice-- let me put this over here. 00:31:37.830 --> 00:31:40.560 align:middle line:84% In practice, at some point, you'll 00:31:40.560 --> 00:31:43.650 align:middle line:90% initialize everything with a, 0. 00:31:43.650 --> 00:31:49.740 align:middle line:84% And you traverse this arrow, and maybe you'll get a, comma, 00:31:49.740 --> 00:31:56.640 align:middle line:84% oh I don't know, db3 da times dz db3. 00:31:56.640 --> 00:32:02.610 align:middle line:84% You'll get one of your sum ends, and there's four more to go. 00:32:02.610 --> 00:32:05.010 align:middle line:84% And then, at some later time, maybe what you'll do 00:32:05.010 --> 00:32:06.510 align:middle line:90% is fold in-- 00:32:06.510 --> 00:32:08.310 align:middle line:84% there's no rule that says you have to form 00:32:08.310 --> 00:32:10.650 align:middle line:90% this entire sum at once, right? 00:32:10.650 --> 00:32:13.860 align:middle line:84% So maybe you'll add the first one later-- 00:32:13.860 --> 00:32:19.257 align:middle line:90% so db1 da dz db1. 00:32:19.257 --> 00:32:21.340 align:middle line:84% And this is close to what's happening in practice, 00:32:21.340 --> 00:32:25.390 align:middle line:84% that you traverse an arrow, and then 00:32:25.390 --> 00:32:28.840 align:middle line:84% you just add the answer to whatever is here. 00:32:28.840 --> 00:32:31.300 align:middle line:84% Once you're finally done, you'll compute this entire sum. 00:32:31.300 --> 00:32:32.980 align:middle line:84% But you don't have to do it at once. 00:32:32.980 --> 00:32:36.880 align:middle line:84% And so I think the secret to every good algorithm I've ever 00:32:36.880 --> 00:32:39.400 align:middle line:84% seen is to recognize that you don't 00:32:39.400 --> 00:32:41.940 align:middle line:84% have to implement something the way the formula looks. 00:32:41.940 --> 00:32:43.690 align:middle line:84% You see a sum, you can do it in any order. 00:32:43.690 --> 00:32:45.050 align:middle line:90% You can do it at any time. 00:32:45.050 --> 00:32:52.510 align:middle line:84% So what actually happens in practice is one way or another, 00:32:52.510 --> 00:32:56.140 align:middle line:84% if you're going to do reverse mode, you have to have-- 00:32:56.140 --> 00:32:58.850 align:middle line:84% the first step is you have to go forward to-- 00:32:58.850 --> 00:33:03.490 align:middle line:84% you're going to go forward to actually compute all the nodes. 00:33:03.490 --> 00:33:05.103 align:middle line:84% Even if you did this one backwards, 00:33:05.103 --> 00:33:07.270 align:middle line:84% you're going to have to have this forward step where 00:33:07.270 --> 00:33:09.520 align:middle line:90% you compute a, b, and z. 00:33:09.520 --> 00:33:12.460 align:middle line:84% And during those forward steps, you actually 00:33:12.460 --> 00:33:13.960 align:middle line:90% build up this graph. 00:33:13.960 --> 00:33:15.910 align:middle line:90% There's no way around it. 00:33:15.910 --> 00:33:19.730 align:middle line:84% You build up the graph, and you put the values on the edges. 00:33:19.730 --> 00:33:23.740 align:middle line:84% And so unlike forward mode, which is-- so let 00:33:23.740 --> 00:33:27.170 align:middle line:90% me say this in a better way. 00:33:27.170 --> 00:33:36.350 align:middle line:84% So in terms of implementation, for forward mode-- 00:33:36.350 --> 00:33:41.280 align:middle line:90% 00:33:41.280 --> 00:33:43.545 align:middle line:90% so you don't build the graph. 00:33:43.545 --> 00:33:46.680 align:middle line:90% 00:33:46.680 --> 00:33:48.710 align:middle line:90% It's kind of implicit. 00:33:48.710 --> 00:33:51.830 align:middle line:84% In reverse mode, there's an explicit graph 00:33:51.830 --> 00:33:53.700 align:middle line:90% that has to be built up. 00:33:53.700 --> 00:33:56.780 align:middle line:84% So in reverse mode, you need to calculate 00:33:56.780 --> 00:34:00.405 align:middle line:84% one or another-- you actually have to build the graph. 00:34:00.405 --> 00:34:02.030 align:middle line:84% So you need some sort of data structure 00:34:02.030 --> 00:34:04.850 align:middle line:84% to be able to build up this graph. 00:34:04.850 --> 00:34:08.150 align:middle line:84% Oh, I just remembered, I have a nice demo of all this. 00:34:08.150 --> 00:34:09.960 align:middle line:84% I wonder if I'll find it fast enough. 00:34:09.960 --> 00:34:11.570 align:middle line:90% I forgot I had a good demo. 00:34:11.570 --> 00:34:14.929 align:middle line:84% You've never seen it because it was since last year. 00:34:14.929 --> 00:34:17.449 align:middle line:90% I'll try to find it. 00:34:17.449 --> 00:34:19.650 align:middle line:90% But we have an explicit graph. 00:34:19.650 --> 00:34:28.659 align:middle line:84% And so what happens is you'll have a forward pass where 00:34:28.659 --> 00:34:32.020 align:middle line:84% you actually compute the primals, or the variables, 00:34:32.020 --> 00:34:33.355 align:middle line:90% and you build the graph. 00:34:33.355 --> 00:34:36.370 align:middle line:90% 00:34:36.370 --> 00:34:38.290 align:middle line:84% And then, once you have the graph, 00:34:38.290 --> 00:34:44.030 align:middle line:84% you can do a reverse pass, where you 00:34:44.030 --> 00:34:47.270 align:middle line:84% start evaluating the derivatives, where 00:34:47.270 --> 00:34:49.594 align:middle line:90% you evaluate the path products. 00:34:49.594 --> 00:34:54.050 align:middle line:90% 00:34:54.050 --> 00:34:56.989 align:middle line:84% And, again, this could all be done with just-- this, too, 00:34:56.989 --> 00:35:00.740 align:middle line:84% could also just be done with plus, minus, times, and divide. 00:35:00.740 --> 00:35:03.360 align:middle line:84% Or you could actually add pieces to it. 00:35:03.360 --> 00:35:06.110 align:middle line:84% So in a way, this is like a transpose. 00:35:06.110 --> 00:35:08.340 align:middle line:84% But I'll kind of show you how it would work. 00:35:08.340 --> 00:35:11.810 align:middle line:84% So with plus, minus, times, and divide-- where can I put that? 00:35:11.810 --> 00:35:13.880 align:middle line:90% Maybe I have enough room here. 00:35:13.880 --> 00:35:16.805 align:middle line:90% And so-- let's see. 00:35:16.805 --> 00:35:18.180 align:middle line:84% I just want to get this straight. 00:35:18.180 --> 00:35:19.566 align:middle line:90% Let's see, doo-doo-doo. 00:35:19.566 --> 00:35:22.480 align:middle line:90% 00:35:22.480 --> 00:35:30.050 align:middle line:84% So, yeah, so how should I do this? 00:35:30.050 --> 00:35:32.500 align:middle line:90% So if I had, at this point-- 00:35:32.500 --> 00:35:36.590 align:middle line:84% let's say I had b, comma-- what's a good notation? 00:35:36.590 --> 00:35:37.340 align:middle line:90% It doesn't matter. 00:35:37.340 --> 00:35:38.618 align:middle line:90% I'm going to call this p. 00:35:38.618 --> 00:35:41.490 align:middle line:90% 00:35:41.490 --> 00:35:49.470 align:middle line:84% So the gradient of plus is just 1 1. 00:35:49.470 --> 00:35:52.140 align:middle line:84% The gradient-- or plus minus is plus or minus 1. 00:35:52.140 --> 00:35:58.770 align:middle line:84% The gradient of product is, like, a2, a1. 00:35:58.770 --> 00:36:02.050 align:middle line:84% And the divide is what's over here. 00:36:02.050 --> 00:36:07.395 align:middle line:84% So in a way, what's going on is I need the transpose. 00:36:07.395 --> 00:36:09.510 align:middle line:90% This becomes 1 plus minus 1. 00:36:09.510 --> 00:36:14.940 align:middle line:84% This becomes a2, a1, if I take the gradient transpose. 00:36:14.940 --> 00:36:20.360 align:middle line:84% And so basically, what happens is if I have a plus, 00:36:20.360 --> 00:36:23.360 align:middle line:90% this just becomes a, p. 00:36:23.360 --> 00:36:29.540 align:middle line:84% Well, the way to write it is a and-- 00:36:29.540 --> 00:36:34.147 align:middle line:90% 00:36:34.147 --> 00:36:36.230 align:middle line:84% I'm getting caught up in a little bit of notation. 00:36:36.230 --> 00:36:38.410 align:middle line:90% Let me say it in words. 00:36:38.410 --> 00:36:42.960 align:middle line:84% Suppose you're moving along, and a plus operation happens. 00:36:42.960 --> 00:36:45.520 align:middle line:84% So maybe I already have some partial results. 00:36:45.520 --> 00:36:49.620 align:middle line:84% So here I have pb at the moment, and I have a, 00:36:49.620 --> 00:36:56.890 align:middle line:84% and I have a pa1 and an a2 and a pa2. 00:36:56.890 --> 00:36:58.920 align:middle line:90% And here I am on the graph. 00:36:58.920 --> 00:37:02.170 align:middle line:84% This a1 comes from other things, not this edge. 00:37:02.170 --> 00:37:05.020 align:middle line:90% I want to process this edge now. 00:37:05.020 --> 00:37:06.850 align:middle line:84% These two edges have not been processed, 00:37:06.850 --> 00:37:08.120 align:middle line:90% and I'm going to process it. 00:37:08.120 --> 00:37:09.580 align:middle line:90% So what's the operation? 00:37:09.580 --> 00:37:17.860 align:middle line:84% The operation is going to be pa1 plus equals pb. 00:37:17.860 --> 00:37:22.930 align:middle line:84% And then the other operation will be pb2 plus equals pb. 00:37:22.930 --> 00:37:27.440 align:middle line:84% And that's how you do an add backwards. 00:37:27.440 --> 00:37:29.390 align:middle line:90% You just move this along. 00:37:29.390 --> 00:37:31.010 align:middle line:84% If you were doing a multiply, which 00:37:31.010 --> 00:37:32.677 align:middle line:84% might be a little bit more informative-- 00:37:32.677 --> 00:37:34.160 align:middle line:90% let's do the multiply. 00:37:34.160 --> 00:37:40.250 align:middle line:84% So here we have a, pa, a1, pa1, a2, pa2, 00:37:40.250 --> 00:37:43.670 align:middle line:84% we're going to multiply these two. 00:37:43.670 --> 00:37:48.030 align:middle line:90% And let's call this b and pb. 00:37:48.030 --> 00:37:49.830 align:middle line:84% So what do you put on these edges? 00:37:49.830 --> 00:37:51.660 align:middle line:84% The edges are just the same derivatives. 00:37:51.660 --> 00:37:53.220 align:middle line:90% Nothing changes there. 00:37:53.220 --> 00:37:54.600 align:middle line:90% We're multiplying a1 and a2. 00:37:54.600 --> 00:37:56.910 align:middle line:84% So this is going to be a2 and a1. 00:37:56.910 --> 00:37:59.130 align:middle line:84% And now what's the implementation? 00:37:59.130 --> 00:38:08.050 align:middle line:84% The implementation is just pa1 plus equals a2 times pb 00:38:08.050 --> 00:38:14.140 align:middle line:90% and pa2 plus equals a1 times pb. 00:38:14.140 --> 00:38:16.990 align:middle line:84% So that's basically how you would do it. 00:38:16.990 --> 00:38:18.468 align:middle line:90% And division would be the same. 00:38:18.468 --> 00:38:19.510 align:middle line:90% You want to see division? 00:38:19.510 --> 00:38:21.640 align:middle line:90% It's not much different. 00:38:21.640 --> 00:38:23.290 align:middle line:90% You would just use these things. 00:38:23.290 --> 00:38:25.450 align:middle line:90% And so I would multiply-- 00:38:25.450 --> 00:38:30.220 align:middle line:84% I would add the product with a2 over a1. 00:38:30.220 --> 00:38:33.550 align:middle line:84% Well, I would just call this 1 over a2. 00:38:33.550 --> 00:38:36.790 align:middle line:84% But-- do I have time to find the demo or not? 00:38:36.790 --> 00:38:38.400 align:middle line:90% I should have thought of this. 00:38:38.400 --> 00:38:40.817 align:middle line:84% I've got a really cool demo, but it might take me a minute 00:38:40.817 --> 00:38:41.570 align:middle line:90% to find it. 00:38:41.570 --> 00:38:43.190 align:middle line:90% It's really good, though. 00:38:43.190 --> 00:38:46.040 align:middle line:84% Let me see if I can find it quickly. 00:38:46.040 --> 00:38:48.097 align:middle line:90% If I could find it quickly-- 00:38:48.097 --> 00:38:50.180 align:middle line:84% while I'm looking, does anybody have any questions 00:38:50.180 --> 00:38:51.065 align:middle line:90% about reverse mode? 00:38:51.065 --> 00:38:53.690 align:middle line:90% 00:38:53.690 --> 00:38:58.540 align:middle line:84% It really is-- let me see if I know where this thing is. 00:38:58.540 --> 00:39:00.580 align:middle line:84% STEVEN JOHNSON: The one thing to remind you of 00:39:00.580 --> 00:39:02.680 align:middle line:90% is the reason for this. 00:39:02.680 --> 00:39:05.540 align:middle line:90% So the forward mode basically-- 00:39:05.540 --> 00:39:06.860 align:middle line:90% suppose you have some function. 00:39:06.860 --> 00:39:08.443 align:middle line:84% So there's some cost of that function. 00:39:08.443 --> 00:39:10.360 align:middle line:84% Forward mode, the cost of that is 00:39:10.360 --> 00:39:12.880 align:middle line:84% proportional to the cost of the function 00:39:12.880 --> 00:39:15.780 align:middle line:90% times the number of inputs. 00:39:15.780 --> 00:39:18.870 align:middle line:84% Reverse mode is proportional to the cost of the function 00:39:18.870 --> 00:39:20.770 align:middle line:90% times the number of outputs. 00:39:20.770 --> 00:39:24.090 align:middle line:84% So if you have lots and lots of inputs and few outputs, 00:39:24.090 --> 00:39:25.380 align:middle line:90% you want to use reverse mode. 00:39:25.380 --> 00:39:30.300 align:middle line:84% If you have lots and lots of outputs and a few inputs, 00:39:30.300 --> 00:39:32.250 align:middle line:90% you want to use forward mode. 00:39:32.250 --> 00:39:36.870 align:middle line:84% If you have lots of both, then it sucks to be you. 00:39:36.870 --> 00:39:39.630 align:middle line:84% It's going to be expensive either way, right? 00:39:39.630 --> 00:39:42.930 align:middle line:84% And basically, what happens is if you 00:39:42.930 --> 00:39:45.600 align:middle line:84% have lots of inputs and a few outputs, 00:39:45.600 --> 00:39:50.200 align:middle line:84% then forward mode ends up recomputing a lot of terms 00:39:50.200 --> 00:39:52.790 align:middle line:84% as it explores all the paths, whereas in reverse mode, 00:39:52.790 --> 00:39:58.420 align:middle line:84% you share computations as long as you can until things fan 00:39:58.420 --> 00:39:58.920 align:middle line:90% out. 00:39:58.920 --> 00:40:01.410 align:middle line:90% That's the intuitive thing. 00:40:01.410 --> 00:40:04.200 align:middle line:84% The other perspective-- the explicit perspective 00:40:04.200 --> 00:40:06.630 align:middle line:84% is if you write it as a product of Jacobian matrices, 00:40:06.630 --> 00:40:08.950 align:middle line:84% then basically you want to arrange it 00:40:08.950 --> 00:40:11.530 align:middle line:84% so if you only have one output, then you 00:40:11.530 --> 00:40:12.780 align:middle line:90% have a row vector on the left. 00:40:12.780 --> 00:40:14.730 align:middle line:90% And you just want to do vector-- 00:40:14.730 --> 00:40:16.720 align:middle line:84% row vector matrix products are cheap. 00:40:16.720 --> 00:40:18.630 align:middle line:84% Matrix-matrix products are expensive. 00:40:18.630 --> 00:40:20.260 align:middle line:90% We did that perspective before. 00:40:20.260 --> 00:40:22.090 align:middle line:84% But that perspective is a little harder 00:40:22.090 --> 00:40:24.820 align:middle line:84% to write down once you start getting general graphs. 00:40:24.820 --> 00:40:27.460 align:middle line:84% But it's the same basic idea, that you 00:40:27.460 --> 00:40:29.710 align:middle line:84% want to arrange it so that you're multiplying matrices 00:40:29.710 --> 00:40:30.340 align:middle line:90% times vector-- 00:40:30.340 --> 00:40:32.770 align:middle line:84% Jacobians times vectors and not Jacobians 00:40:32.770 --> 00:40:38.380 align:middle line:90% times other matrix Jacobians. 00:40:38.380 --> 00:40:40.670 align:middle line:90% PROFESSOR: I think I found it. 00:40:40.670 --> 00:40:45.415 align:middle line:84% And there's also something else I can show you related to-- 00:40:45.415 --> 00:40:49.280 align:middle line:90% 00:40:49.280 --> 00:40:51.890 align:middle line:90% well, maybe not. 00:40:51.890 --> 00:40:56.620 align:middle line:84% So regarding the Jacobians, I think 00:40:56.620 --> 00:40:58.360 align:middle line:84% it's instructive and maybe not obvious. 00:40:58.360 --> 00:41:03.400 align:middle line:84% How do you turn this picture into that Jacobian point 00:41:03.400 --> 00:41:04.060 align:middle line:90% of view? 00:41:04.060 --> 00:41:07.810 align:middle line:84% Or even if these are all vector-valued or matrix-valued 00:41:07.810 --> 00:41:12.370 align:middle line:84% functions, it gets a little bit confusing because what you have 00:41:12.370 --> 00:41:13.840 align:middle line:90% to do is actually-- 00:41:13.840 --> 00:41:16.870 align:middle line:84% you have to put all the variables together 00:41:16.870 --> 00:41:18.010 align:middle line:90% to be able to write down. 00:41:18.010 --> 00:41:21.058 align:middle line:84% So you get a very sparse Jacobian. 00:41:21.058 --> 00:41:23.350 align:middle line:84% I mean, maybe this could have been a homework exercise. 00:41:23.350 --> 00:41:25.183 align:middle line:84% Maybe we should remember this for next year. 00:41:25.183 --> 00:41:28.240 align:middle line:84% But turn this, or something like this, 00:41:28.240 --> 00:41:31.000 align:middle line:84% into a matrix multiply of Jacobians. 00:41:31.000 --> 00:41:32.470 align:middle line:90% And see what that looks like. 00:41:32.470 --> 00:41:35.522 align:middle line:84% I think it's not so obvious the first time you see that. 00:41:35.522 --> 00:41:37.480 align:middle line:84% In fact, I remember being a little bit confused 00:41:37.480 --> 00:41:40.033 align:middle line:84% because I was trying to compare this viewpoint 00:41:40.033 --> 00:41:41.200 align:middle line:90% with the Jacobian viewpoint. 00:41:41.200 --> 00:41:43.690 align:middle line:84% And in the end, the mathematics is the same. 00:41:43.690 --> 00:41:46.345 align:middle line:84% But it's not really obvious at first. 00:41:46.345 --> 00:41:49.370 align:middle line:90% 00:41:49.370 --> 00:41:54.680 align:middle line:84% But regarding the number of outputs 00:41:54.680 --> 00:41:56.300 align:middle line:84% versus the number of inputs, I think 00:41:56.300 --> 00:41:57.540 align:middle line:90% it is sort of interesting. 00:41:57.540 --> 00:41:59.510 align:middle line:84% Let me just draw one quick example that I think 00:41:59.510 --> 00:42:02.250 align:middle line:84% is very illustrative with the graph point of view. 00:42:02.250 --> 00:42:08.270 align:middle line:84% So suppose I could do this just by pathway. 00:42:08.270 --> 00:42:10.850 align:middle line:90% Maybe this is enough right here. 00:42:10.850 --> 00:42:12.510 align:middle line:90% Maybe this is enough. 00:42:12.510 --> 00:42:13.833 align:middle line:90% So let me just put the-- 00:42:13.833 --> 00:42:15.500 align:middle line:84% I don't even need to put the nodes down. 00:42:15.500 --> 00:42:16.700 align:middle line:90% Let me just put-- 00:42:16.700 --> 00:42:22.880 align:middle line:84% I'm just going to call these things a, b, c, and d. 00:42:22.880 --> 00:42:25.700 align:middle line:84% And your goal in life, your only goal in life-- 00:42:25.700 --> 00:42:27.200 align:middle line:90% this could be done all-- 00:42:27.200 --> 00:42:29.540 align:middle line:84% no calculus need to be mentioned, purely graph theory 00:42:29.540 --> 00:42:30.260 align:middle line:90% question. 00:42:30.260 --> 00:42:32.750 align:middle line:90% Your goal is to calculate-- 00:42:32.750 --> 00:42:34.553 align:middle line:90% these are your sources. 00:42:34.553 --> 00:42:35.970 align:middle line:84% I won't even say "inputs" anymore. 00:42:35.970 --> 00:42:37.080 align:middle line:90% I'll just say "sources." 00:42:37.080 --> 00:42:40.130 align:middle line:84% This is a source, and this is a sinc. 00:42:40.130 --> 00:42:43.490 align:middle line:84% And I just want all of the products of the graph 00:42:43.490 --> 00:42:44.970 align:middle line:90% rates from source to sinc. 00:42:44.970 --> 00:42:47.540 align:middle line:84% And in other words, the answer I want you to give me 00:42:47.540 --> 00:42:52.530 align:middle line:90% is acd and bcd. 00:42:52.530 --> 00:42:54.640 align:middle line:84% Your only goal in life is to give me that. 00:42:54.640 --> 00:42:57.930 align:middle line:84% So the obvious first way, the naive way, 00:42:57.930 --> 00:43:00.180 align:middle line:84% is, all right, I'll start at this source, 00:43:00.180 --> 00:43:03.600 align:middle line:84% and I'll compute a, and then I'll multiply it by c, 00:43:03.600 --> 00:43:06.090 align:middle line:90% And then I'll multiply it by d. 00:43:06.090 --> 00:43:09.900 align:middle line:84% Or I could start over here right, and I'll start with b, 00:43:09.900 --> 00:43:13.320 align:middle line:84% and I'll multiply by c, and I'll multiply by d. 00:43:13.320 --> 00:43:16.740 align:middle line:84% And if you count, I did four multiplies. 00:43:16.740 --> 00:43:19.320 align:middle line:84% But if you want to do it the reverse way-- 00:43:19.320 --> 00:43:22.200 align:middle line:84% if you want to do it the reverse way, I could do d, 00:43:22.200 --> 00:43:25.050 align:middle line:90% and I could multiply by c. 00:43:25.050 --> 00:43:29.310 align:middle line:84% And I could take this result, and I can go in two ways on it. 00:43:29.310 --> 00:43:31.080 align:middle line:90% Now, I can multiply this by a-- 00:43:31.080 --> 00:43:35.490 align:middle line:84% so a times this, a times this result. Or I could go b-- 00:43:35.490 --> 00:43:36.630 align:middle line:90% I could reuse. 00:43:36.630 --> 00:43:40.860 align:middle line:84% And now, all of a sudden, what do I have? 00:43:40.860 --> 00:43:44.170 align:middle line:84% I have three products instead of four. 00:43:44.170 --> 00:43:49.990 align:middle line:84% So this is not a bad way to see why reverse mode could possibly 00:43:49.990 --> 00:43:52.990 align:middle line:84% be better than forward mode without thinking about matrices 00:43:52.990 --> 00:43:53.700 align:middle line:90% and whatnot. 00:43:53.700 --> 00:43:56.237 align:middle line:90% 00:43:56.237 --> 00:43:58.070 align:middle line:84% This is a little bit like-- for those of you 00:43:58.070 --> 00:43:59.528 align:middle line:84% who know about dynamic programming, 00:43:59.528 --> 00:44:02.820 align:middle line:84% where people talk about reuse of precomputed quantities, 00:44:02.820 --> 00:44:04.610 align:middle line:90% this is what's going on here. 00:44:04.610 --> 00:44:06.660 align:middle line:84% When you start moving from right to left, 00:44:06.660 --> 00:44:08.940 align:middle line:84% if you have a lot of sources and maybe only one sinc, 00:44:08.940 --> 00:44:10.500 align:middle line:84% which is typical in machine learning, 00:44:10.500 --> 00:44:13.830 align:middle line:84% there's a lot of quantities that you can reuse. 00:44:13.830 --> 00:44:19.580 align:middle line:84% So it's kind of nuts to go a times c times d and b times 00:44:19.580 --> 00:44:22.400 align:middle line:84% c times d when you can just store the result of c times 00:44:22.400 --> 00:44:24.860 align:middle line:84% d and then just do the two separate multiplies. 00:44:24.860 --> 00:44:27.350 align:middle line:84% And, maybe, I think this gives you-- 00:44:27.350 --> 00:44:33.530 align:middle line:84% to me, this gives the best view as to why reverse mode could 00:44:33.530 --> 00:44:36.590 align:middle line:84% be very, very valuable compared-- without derivatives, 00:44:36.590 --> 00:44:39.050 align:middle line:84% without matrices, just this simple example, 00:44:39.050 --> 00:44:42.350 align:middle line:84% I think, kind of does the trick very nicely. 00:44:42.350 --> 00:44:44.940 align:middle line:84% It's saying the same thing that Steven was saying. 00:44:44.940 --> 00:44:45.440 align:middle line:90% All right. 00:44:45.440 --> 00:44:46.260 align:middle line:90% I found this thing. 00:44:46.260 --> 00:44:47.840 align:middle line:90% Now let's just see if it works. 00:44:47.840 --> 00:44:49.235 align:middle line:90% Yeah, if you wouldn't mind. 00:44:49.235 --> 00:44:53.050 align:middle line:90% 00:44:53.050 --> 00:44:54.550 align:middle line:84% I haven't looked at this in a while. 00:44:54.550 --> 00:44:56.230 align:middle line:84% But it was kind of cool when it worked. 00:44:56.230 --> 00:44:57.632 align:middle line:90% So sorry I haven't checked. 00:44:57.632 --> 00:44:58.840 align:middle line:90% I should have checked before. 00:44:58.840 --> 00:45:03.950 align:middle line:90% 00:45:03.950 --> 00:45:05.075 align:middle line:90% But here, let's see. 00:45:05.075 --> 00:45:12.530 align:middle line:90% 00:45:12.530 --> 00:45:13.555 align:middle line:90% Do we have the two? 00:45:13.555 --> 00:45:14.555 align:middle line:90% Did this thing come out? 00:45:14.555 --> 00:45:18.860 align:middle line:90% 00:45:18.860 --> 00:45:19.360 align:middle line:90% OK. 00:45:19.360 --> 00:45:21.790 align:middle line:90% So this is really cute. 00:45:21.790 --> 00:45:22.930 align:middle line:90% So let's see. 00:45:22.930 --> 00:45:26.547 align:middle line:84% I think the computation that's being seen here, 00:45:26.547 --> 00:45:27.880 align:middle line:90% I think-- is this up right here? 00:45:27.880 --> 00:45:30.550 align:middle line:90% 2x-- yeah, 2x is going to get-- 00:45:30.550 --> 00:45:33.260 align:middle line:84% 2 and x are going to get multiplied, the black numbers, 00:45:33.260 --> 00:45:35.470 align:middle line:90% and then times y. 00:45:35.470 --> 00:45:37.240 align:middle line:84% And then we're going to have a plus. 00:45:37.240 --> 00:45:40.640 align:middle line:84% And here we have the minus of x and 1 squared. 00:45:40.640 --> 00:45:43.750 align:middle line:90% 00:45:43.750 --> 00:45:45.850 align:middle line:84% So the actual computation that you see here 00:45:45.850 --> 00:45:48.580 align:middle line:84% is exactly what's in these parentheses. 00:45:48.580 --> 00:45:51.955 align:middle line:90% And now let's see what. 00:45:51.955 --> 00:45:54.580 align:middle line:84% Are we going to do are we doing forward mode or backwards mode? 00:45:54.580 --> 00:45:55.970 align:middle line:90% Let's take a quick look. 00:45:55.970 --> 00:46:00.320 align:middle line:84% So let me just see how this works. 00:46:00.320 --> 00:46:03.060 align:middle line:84% Sorry, I should have looked at this, but I forgot I had it. 00:46:03.060 --> 00:46:03.560 align:middle line:90% Oh. 00:46:03.560 --> 00:46:07.400 align:middle line:84% So I think in red, we're just going 00:46:07.400 --> 00:46:10.850 align:middle line:84% to assume the variables are x equals 3-- 00:46:10.850 --> 00:46:13.040 align:middle line:84% so first we're just going to do the expression, 00:46:13.040 --> 00:46:14.750 align:middle line:90% without any derivatives. 00:46:14.750 --> 00:46:17.660 align:middle line:84% So we're going to take x to be 3 for whatever reason 00:46:17.660 --> 00:46:19.045 align:middle line:90% and y to be 5. 00:46:19.045 --> 00:46:20.420 align:middle line:84% And then we have these constants. 00:46:20.420 --> 00:46:22.273 align:middle line:90% So 2 is 2, and 1 is 1. 00:46:22.273 --> 00:46:24.440 align:middle line:84% So we're just going to take x to be 3 and y to be 5. 00:46:24.440 --> 00:46:26.690 align:middle line:84% Oh, if x equals 3 and y equals 5, 00:46:26.690 --> 00:46:31.430 align:middle line:90% then, OK, the derivatives are-- 00:46:31.430 --> 00:46:32.700 align:middle line:90% the derivatives are 14 and 6. 00:46:32.700 --> 00:46:33.950 align:middle line:90% So that's where we want to go. 00:46:33.950 --> 00:46:36.470 align:middle line:84% All right, so let's take the forward pass. 00:46:36.470 --> 00:46:38.420 align:middle line:90% So 2 times 3 is 6. 00:46:38.420 --> 00:46:41.010 align:middle line:90% 6 times 5 is 30. 00:46:41.010 --> 00:46:41.510 align:middle line:90% Let's see. 00:46:41.510 --> 00:46:43.968 align:middle line:84% I think we're going to do this minus next, I would imagine. 00:46:43.968 --> 00:46:44.840 align:middle line:90% So 3 minus 1 is 2. 00:46:44.840 --> 00:46:45.560 align:middle line:90% There it is. 00:46:45.560 --> 00:46:46.940 align:middle line:90% 2 squared is 4. 00:46:46.940 --> 00:46:50.660 align:middle line:90% And let's add 30 plus 4 is 4. 00:46:50.660 --> 00:46:52.795 align:middle line:84% OK, so now what's going to happen next? 00:46:52.795 --> 00:46:54.170 align:middle line:84% Hopefully, we're going to start-- 00:46:54.170 --> 00:46:57.500 align:middle line:84% OK, we're going to start the reverse pass. 00:46:57.500 --> 00:47:00.660 align:middle line:84% So the answer to this computation is 34. 00:47:00.660 --> 00:47:03.240 align:middle line:84% That's the z that I've been talking about, all right? 00:47:03.240 --> 00:47:06.090 align:middle line:84% And now I want you to just see the reverse pass. 00:47:06.090 --> 00:47:07.590 align:middle line:90% And so we always start out as 1. 00:47:07.590 --> 00:47:12.990 align:middle line:84% I think that's what's behind over here. 00:47:12.990 --> 00:47:15.570 align:middle line:84% The funny thing is you can get good at doing this by hand. 00:47:15.570 --> 00:47:17.520 align:middle line:84% I mean, it's kind of actually not a bad-- 00:47:17.520 --> 00:47:19.140 align:middle line:84% I'm not sure I'm completely there yet, 00:47:19.140 --> 00:47:21.030 align:middle line:84% but I have done this in classes where 00:47:21.030 --> 00:47:22.410 align:middle line:84% I would take a little computation 00:47:22.410 --> 00:47:23.243 align:middle line:90% and do reverse mode. 00:47:23.243 --> 00:47:25.210 align:middle line:84% But it's also easy to get a little confused. 00:47:25.210 --> 00:47:26.460 align:middle line:90% So it's very dangerous. 00:47:26.460 --> 00:47:27.640 align:middle line:90% OK, so let's see. 00:47:27.640 --> 00:47:30.420 align:middle line:90% So we have a plus, OK? 00:47:30.420 --> 00:47:34.500 align:middle line:90% So when you have a plus, then-- 00:47:34.500 --> 00:47:38.310 align:middle line:84% these derivatives are all starting out at 0. 00:47:38.310 --> 00:47:40.490 align:middle line:90% So what's going to happen-- 00:47:40.490 --> 00:47:42.980 align:middle line:84% what's going to happen for this plus? 00:47:42.980 --> 00:47:46.623 align:middle line:84% What's going to be the second number here and here? 00:47:46.623 --> 00:47:47.540 align:middle line:90% I haven't done it yet. 00:47:47.540 --> 00:47:50.810 align:middle line:84% But if you're following how reverse mode works, 00:47:50.810 --> 00:47:55.640 align:middle line:84% what will be the derivative here? 00:47:55.640 --> 00:47:58.210 align:middle line:84% So what should be the numbers in green as soon as I 00:47:58.210 --> 00:48:01.090 align:middle line:90% press the button here and here? 00:48:01.090 --> 00:48:03.751 align:middle line:90% Do you see what that is? 00:48:03.751 --> 00:48:05.700 align:middle line:84% STEVEN JOHNSON: I heard somebody whisper 1. 00:48:05.700 --> 00:48:07.440 align:middle line:84% PROFESSOR: Yeah, both of them are 1. 00:48:07.440 --> 00:48:11.070 align:middle line:84% And one way to say it-- if you don't remember what's 00:48:11.070 --> 00:48:17.460 align:middle line:84% going on here, you can remember that the partial derivative 00:48:17.460 --> 00:48:20.820 align:middle line:84% of the sum with respect to this number is always 1 here. 00:48:20.820 --> 00:48:22.380 align:middle line:90% There's like a little 1-- 00:48:22.380 --> 00:48:24.690 align:middle line:84% just to copy the way it was on the blackboard, 00:48:24.690 --> 00:48:26.070 align:middle line:84% there's a 1 sitting on this edge, 00:48:26.070 --> 00:48:27.570 align:middle line:84% and there's a 1 sitting on the edge. 00:48:27.570 --> 00:48:30.180 align:middle line:84% It's the partial derivative of the sum with respect 00:48:30.180 --> 00:48:31.540 align:middle line:90% to each of the two inputs. 00:48:31.540 --> 00:48:35.610 align:middle line:84% So this should just be 1 each, and there it is. 00:48:35.610 --> 00:48:39.630 align:middle line:84% OK, now let's go a little fancier. 00:48:39.630 --> 00:48:40.860 align:middle line:90% Let's do this product. 00:48:40.860 --> 00:48:43.170 align:middle line:84% If you remember how the product works, 00:48:43.170 --> 00:48:46.430 align:middle line:84% what should I imagine is sitting on this edge? 00:48:46.430 --> 00:48:48.560 align:middle line:84% What number here is on this edge? 00:48:48.560 --> 00:48:50.030 align:middle line:90% And what's on this edge? 00:48:50.030 --> 00:48:51.858 align:middle line:84% So the product rule for calculus-- 00:48:51.858 --> 00:48:53.150 align:middle line:90% it's the same old product rule. 00:48:53.150 --> 00:48:57.450 align:middle line:84% You just have to get used to it in this new circumstance. 00:48:57.450 --> 00:49:00.700 align:middle line:90% So what goes here? 00:49:00.700 --> 00:49:02.920 align:middle line:90% The 5, and here is the 6, right? 00:49:02.920 --> 00:49:04.360 align:middle line:84% So what we're going to do is we're 00:49:04.360 --> 00:49:07.510 align:middle line:84% going to go 5 times the 1, add it to the 0, 00:49:07.510 --> 00:49:08.935 align:middle line:90% and we're going to get the 5. 00:49:08.935 --> 00:49:10.810 align:middle line:84% And now here we're going to go 6 times the 1, 00:49:10.810 --> 00:49:14.440 align:middle line:84% and add it to the 0, and we're going to get a 6. 00:49:14.440 --> 00:49:16.690 align:middle line:90% And we have another multiply. 00:49:16.690 --> 00:49:18.970 align:middle line:90% So there's the 3 and the 2. 00:49:18.970 --> 00:49:24.310 align:middle line:84% So if I've got this right, we should get the 15 and the 10. 00:49:24.310 --> 00:49:27.190 align:middle line:84% Yeah, the 15 and 10 should show up, and it does. 00:49:27.190 --> 00:49:29.380 align:middle line:84% And notice the 10, by the way, showed up here 00:49:29.380 --> 00:49:31.570 align:middle line:90% because it's the same variable. 00:49:31.570 --> 00:49:35.380 align:middle line:84% And so there's really-- in storage, there's just one x. 00:49:35.380 --> 00:49:38.380 align:middle line:84% So now what do we-- we need to do the square. 00:49:38.380 --> 00:49:42.080 align:middle line:84% So the square-- we've got the 1 here, right? 00:49:42.080 --> 00:49:45.580 align:middle line:84% And the derivative of square is 2x, right? 00:49:45.580 --> 00:49:47.900 align:middle line:84% And so we would actually have a 4 over here. 00:49:47.900 --> 00:49:50.710 align:middle line:84% And so we have to take 1 times the 4 00:49:50.710 --> 00:49:52.420 align:middle line:90% and get a 4 if I did this right. 00:49:52.420 --> 00:49:53.140 align:middle line:90% Yep. 00:49:53.140 --> 00:49:56.870 align:middle line:90% And now we have a minus. 00:49:56.870 --> 00:49:59.300 align:middle line:90% So we have a 1 and a minus 1. 00:49:59.300 --> 00:50:02.960 align:middle line:84% The 4 times the 1 adds to the 10 and gives a 14. 00:50:02.960 --> 00:50:03.530 align:middle line:90% Yep. 00:50:03.530 --> 00:50:05.030 align:middle line:90% Oh, I like that, 10 plus 4. 00:50:05.030 --> 00:50:07.640 align:middle line:90% And then this is 0 minus 4. 00:50:07.640 --> 00:50:09.210 align:middle line:90% And there we go. 00:50:09.210 --> 00:50:11.730 align:middle line:90% And then we actually have-- 00:50:11.730 --> 00:50:19.410 align:middle line:84% the derivative with respect to x of this quantity z, 00:50:19.410 --> 00:50:22.770 align:middle line:90% apparently, is 14 and 6. 00:50:22.770 --> 00:50:24.840 align:middle line:84% So we've got the 14 over here because we just 00:50:24.840 --> 00:50:27.420 align:middle line:90% did it with respect to the x. 00:50:27.420 --> 00:50:28.740 align:middle line:90% Oh, the y is there, too, the 6. 00:50:28.740 --> 00:50:30.540 align:middle line:90% Yeah, the y is there, too. 00:50:30.540 --> 00:50:32.310 align:middle line:90% OK, so this is-- 00:50:32.310 --> 00:50:35.100 align:middle line:84% I mean, if you guys want this, I could give you-- 00:50:35.100 --> 00:50:36.390 align:middle line:90% we can link this. 00:50:36.390 --> 00:50:39.930 align:middle line:90% But I sort of go through here-- 00:50:39.930 --> 00:50:43.200 align:middle line:84% we're out of time now, but we actually talk about-- 00:50:43.200 --> 00:50:45.310 align:middle line:84% you can see the whole code from beginning to end. 00:50:45.310 --> 00:50:46.050 align:middle line:90% So it doesn't-- 00:50:46.050 --> 00:50:46.710 align:middle line:90% I don't know. 00:50:46.710 --> 00:50:48.570 align:middle line:84% Reverse mode seems more mysterious, I think, 00:50:48.570 --> 00:50:49.560 align:middle line:90% than forward mode. 00:50:49.560 --> 00:50:52.200 align:middle line:84% But we actually have the whole thing self-contained 00:50:52.200 --> 00:50:54.630 align:middle line:84% in one notebook here where you could actually 00:50:54.630 --> 00:50:57.820 align:middle line:84% see the building up of the tree and actually how it works. 00:50:57.820 --> 00:50:59.370 align:middle line:90% And so we will-- 00:50:59.370 --> 00:51:01.530 align:middle line:84% in fact, you could actually see this tree coming up 00:51:01.530 --> 00:51:03.300 align:middle line:90% in a data structure. 00:51:03.300 --> 00:51:09.190 align:middle line:84% Yeah, so I'll link to this so you can see how it works. 00:51:09.190 --> 00:51:11.515 align:middle line:84% But I think we're going to wrap it up now. 00:51:11.515 --> 00:51:13.890 align:middle line:84% Steven's going to make a list of things we haven't shown. 00:51:13.890 --> 00:51:18.110 align:middle line:90% 00:51:18.110 --> 00:51:19.520 align:middle line:90% Do you want me to-- 00:51:19.520 --> 00:51:23.760 align:middle line:84% do you want to start talking, and I could do that, Steven? 00:51:23.760 --> 00:51:27.090 align:middle line:90% I'll get the lights on. 00:51:27.090 --> 00:51:29.600 align:middle line:84% STEVEN JOHNSON: So, yeah, so it'd 00:51:29.600 --> 00:51:32.380 align:middle line:84% be great if we had another couple of weeks in this course. 00:51:32.380 --> 00:51:35.160 align:middle line:84% It's actually amazing how much there 00:51:35.160 --> 00:51:37.560 align:middle line:84% is to say just about taking derivatives, 00:51:37.560 --> 00:51:40.560 align:middle line:84% even though you come in thinking you 00:51:40.560 --> 00:51:44.280 align:middle line:84% know how to take derivatives, and there's so much left. 00:51:44.280 --> 00:51:47.130 align:middle line:84% PROFESSOR: The problem is linear algebra was evolved lately 00:51:47.130 --> 00:51:49.050 align:middle line:90% in math departments. 00:51:49.050 --> 00:51:50.445 align:middle line:90% People thought it was all easy. 00:51:50.445 --> 00:51:51.090 align:middle line:90% STEVEN JOHNSON: Yeah. 00:51:51.090 --> 00:51:53.132 align:middle line:84% PROFESSOR: And so they actually never pursued it, 00:51:53.132 --> 00:51:54.310 align:middle line:90% and they missed the boat. 00:51:54.310 --> 00:51:55.490 align:middle line:90% I really believe that. 00:51:55.490 --> 00:51:59.130 align:middle line:84% STEVEN JOHNSON: So, for example, one very practical thing 00:51:59.130 --> 00:52:03.090 align:middle line:84% that I was hoping to cover, but we didn't get to, 00:52:03.090 --> 00:52:06.150 align:middle line:84% is how to do these custom AD rules. 00:52:06.150 --> 00:52:08.130 align:middle line:90% So suppose you have a box. 00:52:08.130 --> 00:52:11.400 align:middle line:84% This is your program, or maybe just 00:52:11.400 --> 00:52:13.380 align:middle line:90% a subroutine in your program. 00:52:13.380 --> 00:52:19.530 align:middle line:84% Or let me just say a function that takes in an x1, 00:52:19.530 --> 00:52:23.960 align:middle line:84% and then say this is a vector, and an x2, 00:52:23.960 --> 00:52:27.920 align:middle line:90% or let's say x and y-- 00:52:27.920 --> 00:52:35.780 align:middle line:84% x and y-- and outputs, say, an f and a g. 00:52:35.780 --> 00:52:37.790 align:middle line:90% And then suppose that AD fails. 00:52:37.790 --> 00:52:40.503 align:middle line:90% 00:52:40.503 --> 00:52:42.920 align:middle line:84% PROFESSOR: As it did in the first version of the homework. 00:52:42.920 --> 00:52:44.880 align:middle line:84% STEVEN JOHNSON: As it did in the first version. 00:52:44.880 --> 00:52:46.460 align:middle line:84% You called out to some [INAUDIBLE] 00:52:46.460 --> 00:52:47.870 align:middle line:84% or something that doesn't know how to differentiate. 00:52:47.870 --> 00:52:49.703 align:middle line:84% And there are, unfortunately, lots of things 00:52:49.703 --> 00:52:51.950 align:middle line:84% that these systems don't know how to differentiate. 00:52:51.950 --> 00:52:54.840 align:middle line:84% Or maybe it differentiates, but it differentiates it very, 00:52:54.840 --> 00:52:55.410 align:middle line:90% very badly. 00:52:55.410 --> 00:52:57.488 align:middle line:84% So that happens in a very inefficient-- 00:52:57.488 --> 00:52:59.780 align:middle line:84% that can happen if this function is computing something 00:52:59.780 --> 00:53:00.860 align:middle line:90% approximately. 00:53:00.860 --> 00:53:03.710 align:middle line:84% And it spends a lot of effort trying to exactly differentiate 00:53:03.710 --> 00:53:06.660 align:middle line:84% the error in the approximations instead of realizing, 00:53:06.660 --> 00:53:09.470 align:middle line:84% oh, we're really doing an integral, 00:53:09.470 --> 00:53:11.750 align:middle line:84% and we're doing it accurate to 13 decimal places. 00:53:11.750 --> 00:53:16.580 align:middle line:84% I don't need to get the derivative of the last three 00:53:16.580 --> 00:53:18.350 align:middle line:90% digits. 00:53:18.350 --> 00:53:21.350 align:middle line:84% And so what you do is if you're doing forward 00:53:21.350 --> 00:53:29.940 align:middle line:84% mode, what you have to do is supply a thing 00:53:29.940 --> 00:53:41.220 align:middle line:84% that computes, basically, a Jacobian vector product, 00:53:41.220 --> 00:53:44.820 align:middle line:90% a product, a JVP. 00:53:44.820 --> 00:53:51.420 align:middle line:84% It's also called an frule in chain rules. 00:53:51.420 --> 00:53:53.865 align:middle line:84% It's also called a pushforward, I think this thing-- 00:53:53.865 --> 00:53:55.740 align:middle line:84% PROFESSOR: Yeah, differential geometry, yeah. 00:53:55.740 --> 00:53:58.900 align:middle line:84% STEVEN JOHNSON: So some things use this terminology. 00:53:58.900 --> 00:54:00.900 align:middle line:84% It comes from differential geometry, 00:54:00.900 --> 00:54:03.520 align:middle line:90% and it's called a pushforward. 00:54:03.520 --> 00:54:05.220 align:middle line:84% And so basically, you're supplying-- 00:54:05.220 --> 00:54:08.040 align:middle line:84% you just write a program that computes f, 00:54:08.040 --> 00:54:12.870 align:middle line:90% partial f, partial x times dx. 00:54:12.870 --> 00:54:23.710 align:middle line:84% And it computes a partial f, partial y dy, given dx dy. 00:54:23.710 --> 00:54:26.017 align:middle line:84% So you implement this yourself by-- you 00:54:26.017 --> 00:54:27.690 align:middle line:84% know what your function is computing, 00:54:27.690 --> 00:54:30.630 align:middle line:84% and you work out manually with the derivative is. 00:54:30.630 --> 00:54:38.300 align:middle line:84% And if you're doing it in reverse mode, which 00:54:38.300 --> 00:54:40.580 align:middle line:84% is more common if you're doing any kind 00:54:40.580 --> 00:54:42.650 align:middle line:84% of optimization machine learning, 00:54:42.650 --> 00:54:44.450 align:middle line:84% you have to go in the other direction. 00:54:44.450 --> 00:54:47.420 align:middle line:90% You have to provide a vector-- 00:54:47.420 --> 00:54:57.060 align:middle line:84% and I should say a row Vector Jacobian Product, 00:54:57.060 --> 00:55:05.430 align:middle line:84% which is also called a VJP or an rrule in reverse rule, 00:55:05.430 --> 00:55:10.960 align:middle line:84% or a pullback in that funny differential geometry notation. 00:55:10.960 --> 00:55:13.410 align:middle line:84% And so basically, equivalently, you're 00:55:13.410 --> 00:55:14.970 align:middle line:84% effectively providing a function that 00:55:14.970 --> 00:55:22.500 align:middle line:84% computes partial f, partial x, that Jacobian transposed, 00:55:22.500 --> 00:55:29.000 align:middle line:90% times a vector u and partial-- 00:55:29.000 --> 00:55:32.460 align:middle line:84% sorry, that should have been partial-- 00:55:32.460 --> 00:55:34.170 align:middle line:90% yeah, well same thing for-- 00:55:34.170 --> 00:55:38.190 align:middle line:84% and partial f, partial y, transposed, 00:55:38.190 --> 00:55:46.140 align:middle line:84% times u for an arbitrary u, and similarly for g. 00:55:46.140 --> 00:55:48.870 align:middle line:84% PROFESSOR: And maybe just to say that what Steven's 00:55:48.870 --> 00:55:51.600 align:middle line:84% writing right now are exactly the rules in those boxes, 00:55:51.600 --> 00:55:53.640 align:middle line:84% except he's being careful about the fact 00:55:53.640 --> 00:55:55.300 align:middle line:84% that this is the multivariate version-- 00:55:55.300 --> 00:55:55.540 align:middle line:90% STEVEN JOHNSON: Yeah. 00:55:55.540 --> 00:55:56.770 align:middle line:84% PROFESSOR: --when I was like, oh, 00:55:56.770 --> 00:55:57.850 align:middle line:84% I'm just going to do the scalars. 00:55:57.850 --> 00:55:59.680 align:middle line:84% STEVEN JOHNSON: And so, in principle, this 00:55:59.680 --> 00:56:00.700 align:middle line:90% is straightforward. 00:56:00.700 --> 00:56:05.020 align:middle line:84% In practice, you need to think a little bit about the rules 00:56:05.020 --> 00:56:06.320 align:middle line:90% for what this even means. 00:56:06.320 --> 00:56:09.880 align:middle line:84% So, for example, these systems are 00:56:09.880 --> 00:56:13.220 align:middle line:84% designed-- written in such a way as to be quite general. 00:56:13.220 --> 00:56:16.420 align:middle line:84% And so because of that, the interfaces for supplying these 00:56:16.420 --> 00:56:18.520 align:middle line:84% can sometimes be a little bit confusing 00:56:18.520 --> 00:56:19.980 align:middle line:90% when you're not used to it. 00:56:19.980 --> 00:56:27.680 align:middle line:84% Another thing that I would have liked to have time to cover 00:56:27.680 --> 00:56:30.335 align:middle line:90% is talk about complex numbers. 00:56:30.335 --> 00:56:37.710 align:middle line:90% 00:56:37.710 --> 00:56:39.560 align:middle line:84% So, for example, suppose you have 00:56:39.560 --> 00:56:48.770 align:middle line:84% f of z where z is x plus iy is a complex number. 00:56:48.770 --> 00:56:56.110 align:middle line:84% Suppose you have that equals z squared, the magnitude squared. 00:56:56.110 --> 00:57:00.650 align:middle line:84% So this is the same thing as z times its complex conjugate. 00:57:00.650 --> 00:57:05.170 align:middle line:84% So if you take complex analysis, they'll say, OK, this function, 00:57:05.170 --> 00:57:07.640 align:middle line:84% you can't differentiate at the end. 00:57:07.640 --> 00:57:09.460 align:middle line:84% It doesn't have a derivative with respect 00:57:09.460 --> 00:57:12.650 align:middle line:90% to z in the usual sense. 00:57:12.650 --> 00:57:18.110 align:middle line:84% But, of course, if you change z by a little bit, 00:57:18.110 --> 00:57:20.960 align:middle line:84% the output does change by a little bit. 00:57:20.960 --> 00:57:22.870 align:middle line:84% And if you look, for example, in terms 00:57:22.870 --> 00:57:25.220 align:middle line:84% of x and y, the real and imaginary parts, 00:57:25.220 --> 00:57:27.460 align:middle line:84% this is a perfectly differentiable function 00:57:27.460 --> 00:57:30.240 align:middle line:90% of x and y. 00:57:30.240 --> 00:57:34.002 align:middle line:90% And so suppose you're-- 00:57:34.002 --> 00:57:35.460 align:middle line:84% there are lots and lots of problems 00:57:35.460 --> 00:57:36.835 align:middle line:84% that are very naturally expressed 00:57:36.835 --> 00:57:38.520 align:middle line:84% in terms of complex numbers, especially 00:57:38.520 --> 00:57:40.830 align:middle line:84% in physics for example, physical quantities. 00:57:40.830 --> 00:57:42.660 align:middle line:84% You want to be complex, and you want 00:57:42.660 --> 00:57:44.220 align:middle line:84% to optimize some function of them, 00:57:44.220 --> 00:57:47.820 align:middle line:90% some real valued function. 00:57:47.820 --> 00:57:50.243 align:middle line:84% And so, of course, if you split everything 00:57:50.243 --> 00:57:51.660 align:middle line:84% into its real and imaginary parts, 00:57:51.660 --> 00:57:53.580 align:middle line:84% it becomes differentiable in the usual way. 00:57:53.580 --> 00:57:54.997 align:middle line:84% And then you can take derivatives. 00:57:54.997 --> 00:58:02.945 align:middle line:84% But this is a very awkward way of-- 00:58:02.945 --> 00:58:04.320 align:middle line:84% if things are naturally expressed 00:58:04.320 --> 00:58:05.220 align:middle line:90% in terms of complex numbers. 00:58:05.220 --> 00:58:06.330 align:middle line:84% PROFESSOR: At first it doesn't seem awkward right, 00:58:06.330 --> 00:58:08.515 align:middle line:84% but you only realize later that it's awkward. 00:58:08.515 --> 00:58:09.390 align:middle line:90% STEVEN JOHNSON: Yeah. 00:58:09.390 --> 00:58:11.180 align:middle line:84% PROFESSOR: I mean, it seems kind of natural at first. 00:58:11.180 --> 00:58:11.950 align:middle line:84% STEVEN JOHNSON: It seems natural at first, 00:58:11.950 --> 00:58:14.220 align:middle line:84% but then it just turns into-- because then, 00:58:14.220 --> 00:58:18.000 align:middle line:84% basically, your code does not look like your math anymore. 00:58:18.000 --> 00:58:20.010 align:middle line:84% It doesn't look like your physical equations. 00:58:20.010 --> 00:58:20.718 align:middle line:90% PROFESSOR: Right. 00:58:20.718 --> 00:58:22.800 align:middle line:84% Complex code, like z cubed, is much like-- 00:58:22.800 --> 00:58:23.880 align:middle line:90% other than all those-- 00:58:23.880 --> 00:58:26.940 align:middle line:84% STEVEN JOHNSON: It's like If you read Maxwell's original paper 00:58:26.940 --> 00:58:30.682 align:middle line:84% on Maxwell's equations in 1865, he didn't have vector notation. 00:58:30.682 --> 00:58:32.640 align:middle line:84% He just used a different letter of the alphabet 00:58:32.640 --> 00:58:34.682 align:middle line:84% for every component of every field. 00:58:34.682 --> 00:58:37.140 align:middle line:84% And you have, like, 20 variable equations and 20 variables. 00:58:37.140 --> 00:58:39.780 align:middle line:90% So this is a version of that. 00:58:39.780 --> 00:58:44.220 align:middle line:84% So now you really only have one quantity that's complex, 00:58:44.220 --> 00:58:47.465 align:middle line:84% but now you have to have two variables. 00:58:47.465 --> 00:58:51.480 align:middle line:90% So if you just ask, what is df-- 00:58:51.480 --> 00:58:59.520 align:middle line:84% what is f of z plus dz minus f of z in terms of z-- 00:58:59.520 --> 00:59:02.340 align:middle line:84% this turns out to have a perfectly-- 00:59:02.340 --> 00:59:13.120 align:middle line:84% it's basically z times z bar plus z times d z bar. 00:59:13.120 --> 00:59:19.810 align:middle line:84% And so in general, it turns out what 00:59:19.810 --> 00:59:31.000 align:middle line:84% you can do is you can write df as df dz times dz 00:59:31.000 --> 00:59:38.500 align:middle line:90% plus df d z bar times d z bar. 00:59:38.500 --> 00:59:40.773 align:middle line:90% And you treat these-- 00:59:40.773 --> 00:59:42.190 align:middle line:84% the notation is a little confusing 00:59:42.190 --> 00:59:43.300 align:middle line:84% because it looks like you're treating 00:59:43.300 --> 00:59:45.550 align:middle line:84% z and z bar as independent variables when of course 00:59:45.550 --> 00:59:46.120 align:middle line:90% they're not. 00:59:46.120 --> 00:59:48.310 align:middle line:84% You should really think of this as a function of-- 00:59:48.310 --> 00:59:51.890 align:middle line:84% this is really a function of two variables. 00:59:51.890 --> 00:59:55.630 align:middle line:84% This is like f of a, b equals a times 00:59:55.630 --> 01:00:01.720 align:middle line:84% b where you just let a equals z and b equals z bar. 01:00:01.720 --> 01:00:03.970 align:middle line:84% So you can think of it as a function of two variables, 01:00:03.970 --> 01:00:07.680 align:middle line:84% and at the end, you plug in the complex conjugates. 01:00:07.680 --> 01:00:09.600 align:middle line:90% And it's nice to-- 01:00:09.600 --> 01:00:11.920 align:middle line:84% so everything ends up being perfectly sensible. 01:00:11.920 --> 01:00:13.975 align:middle line:84% It's equivalent in some sense to differentiate 01:00:13.975 --> 01:00:15.490 align:middle line:84% in [? respect of ?] the real and imaginary parts. 01:00:15.490 --> 01:00:16.330 align:middle line:84% PROFESSOR: The way I look at it-- 01:00:16.330 --> 01:00:17.830 align:middle line:84% I don't know if this helps or hurts. 01:00:17.830 --> 01:00:23.530 align:middle line:84% But in the complex plane, dx is eastward and dy is northward. 01:00:23.530 --> 01:00:26.800 align:middle line:84% And here you have dz and d z bar are these two 01:00:26.800 --> 01:00:28.000 align:middle line:90% symmetric directions. 01:00:28.000 --> 01:00:29.830 align:middle line:84% But it's still-- you need two dimensions 01:00:29.830 --> 01:00:31.477 align:middle line:90% to describe what's going on. 01:00:31.477 --> 01:00:32.810 align:middle line:90% So that's one way to look at it. 01:00:32.810 --> 01:00:36.370 align:middle line:84% STEVEN JOHNSON: So then you can go through and work out, what 01:00:36.370 --> 01:00:39.910 align:middle line:90% does this mean for gradients? 01:00:39.910 --> 01:00:42.850 align:middle line:84% What does this mean for Newton methods? 01:00:42.850 --> 01:00:45.050 align:middle line:90% All of those kinds of things. 01:00:45.050 --> 01:00:46.860 align:middle line:90% And so the name for this is-- 01:00:46.860 --> 01:00:48.610 align:middle line:84% PROFESSOR: Does anyone in machine learning 01:00:48.610 --> 01:00:50.170 align:middle line:90% ever use complex numbers? 01:00:50.170 --> 01:00:52.000 align:middle line:84% STEVEN JOHNSON: I don't know if it's very 01:00:52.000 --> 01:00:54.255 align:middle line:90% common in machine learning. 01:00:54.255 --> 01:00:56.380 align:middle line:84% Maybe if you're doing FFTs or something like that-- 01:00:56.380 --> 01:00:57.100 align:middle line:90% PROFESSOR: Oh, maybe. 01:00:57.100 --> 01:00:58.892 align:middle line:84% STEVEN JOHNSON: --then it would be natural. 01:00:58.892 --> 01:01:04.300 align:middle line:84% But this is also called the Wirtinger calculus-- 01:01:04.300 --> 01:01:07.540 align:middle line:84% Wirting-- do you actually know what the C and the R 01:01:07.540 --> 01:01:08.680 align:middle line:90% stand for in CR calculus? 01:01:08.680 --> 01:01:09.597 align:middle line:90% PROFESSOR: Not a clue. 01:01:09.597 --> 01:01:12.700 align:middle line:90% 01:01:12.700 --> 01:01:14.440 align:middle line:84% STEVEN JOHNSON: So if you ever need 01:01:14.440 --> 01:01:17.230 align:middle line:84% to optimize a function of complex variables, 01:01:17.230 --> 01:01:19.540 align:middle line:84% it turns out there's a perfectly sensible way 01:01:19.540 --> 01:01:23.085 align:middle line:84% to do it that still works when you have 01:01:23.085 --> 01:01:24.460 align:middle line:84% complex conjugates and real parts 01:01:24.460 --> 01:01:28.720 align:middle line:84% and imaginary parts that you can't differentiate normally. 01:01:28.720 --> 01:01:30.105 align:middle line:90% So just look this up. 01:01:30.105 --> 01:01:31.480 align:middle line:84% PROFESSOR: That's what I'm doing. 01:01:31.480 --> 01:01:32.980 align:middle line:84% STEVEN JOHNSON: You do not have to drop 01:01:32.980 --> 01:01:34.855 align:middle line:84% into separate real and imaginary parts, which 01:01:34.855 --> 01:01:36.110 align:middle line:90% is always an option. 01:01:36.110 --> 01:01:38.470 align:middle line:84% But it can be a very awkward and unnatural option 01:01:38.470 --> 01:01:43.270 align:middle line:84% if that's not the language that your problem is naturally 01:01:43.270 --> 01:01:44.770 align:middle line:90% expressed in. 01:01:44.770 --> 01:01:46.530 align:middle line:90% So there's more things. 01:01:46.530 --> 01:01:52.150 align:middle line:84% I know the first time we taught this course last January, 01:01:52.150 --> 01:01:55.970 align:middle line:84% Alan had ambitions of getting into exterior calculus 01:01:55.970 --> 01:01:57.715 align:middle line:90% and some differential geometry. 01:01:57.715 --> 01:01:58.840 align:middle line:90% PROFESSOR: It's not so bad. 01:01:58.840 --> 01:02:00.173 align:middle line:90% STEVEN JOHNSON: It's not so bad. 01:02:00.173 --> 01:02:02.980 align:middle line:84% PROFESSOR: I could have done it in one half of one lecture. 01:02:02.980 --> 01:02:03.700 align:middle line:90% But I didn't. 01:02:03.700 --> 01:02:06.240 align:middle line:84% STEVEN JOHNSON: Yeah, it's one of those things-- 01:02:06.240 --> 01:02:08.300 align:middle line:84% when we first started planning this, I was like, 01:02:08.300 --> 01:02:09.400 align:middle line:84% do we really have enough material 01:02:09.400 --> 01:02:10.810 align:middle line:90% to fill 16 hours of lectures? 01:02:10.810 --> 01:02:15.670 align:middle line:84% And then you realize, oh, every little thing-- 01:02:15.670 --> 01:02:17.980 align:middle line:84% there's all these topics and sensitivity analysis, 01:02:17.980 --> 01:02:19.660 align:middle line:84% and they're all more complicated than-- 01:02:19.660 --> 01:02:22.243 align:middle line:84% PROFESSOR: So I don't think it's any of these things, Calculus 01:02:22.243 --> 01:02:24.010 align:middle line:84% of Real Values, Calculus Readiness, which 01:02:24.010 --> 01:02:25.540 align:middle line:84% is an education term for students 01:02:25.540 --> 01:02:27.310 align:middle line:84% who are ready for calculus, Cost Reduced-- 01:02:27.310 --> 01:02:28.810 align:middle line:84% STEVEN JOHNSON: Yeah, I've even seen 01:02:28.810 --> 01:02:31.060 align:middle line:90% it written as blackboard bold. 01:02:31.060 --> 01:02:34.570 align:middle line:90% So I think this is the clue. 01:02:34.570 --> 01:02:36.880 align:middle line:84% CR, I've seen it written this way. 01:02:36.880 --> 01:02:39.130 align:middle line:84% So it's like the complex numbers and the real numbers. 01:02:39.130 --> 01:02:45.500 align:middle line:84% I think it might be just a reference to those two sets 01:02:45.500 --> 01:02:46.040 align:middle line:90% somehow. 01:02:46.040 --> 01:02:50.390 align:middle line:84% But you want to map complex derivatives 01:02:50.390 --> 01:02:54.490 align:middle line:84% into real derivatives or vice versa. 01:02:54.490 --> 01:02:56.610 align:middle line:90% So, yeah, anyway, so this is-- 01:02:56.610 --> 01:02:59.460 align:middle line:84% there's a whole set of rules you can derive 01:02:59.460 --> 01:03:03.150 align:middle line:90% beyond this for CR calculus. 01:03:03.150 --> 01:03:06.540 align:middle line:84% If the function is real, for example, if the output is real, 01:03:06.540 --> 01:03:09.120 align:middle line:84% then these two terms have to be complex conjugates. 01:03:09.120 --> 01:03:12.450 align:middle line:84% And so this is twice the real part of that. 01:03:12.450 --> 01:03:15.840 align:middle line:84% If it's analytic, then this time it's 0. 01:03:15.840 --> 01:03:21.990 align:middle line:84% So if it has an ordinary derivative in the 18.04 sense, 01:03:21.990 --> 01:03:25.860 align:middle line:84% in the complex analysis sense, then one of the terms 01:03:25.860 --> 01:03:27.325 align:middle line:90% drops out. 01:03:27.325 --> 01:03:29.700 align:middle line:84% PROFESSOR: You don't think it's a chain rule of calculus? 01:03:29.700 --> 01:03:30.670 align:middle line:90% No, maybe not. 01:03:30.670 --> 01:03:31.000 align:middle line:90% STEVEN JOHNSON: Oh, yeah. 01:03:31.000 --> 01:03:32.708 align:middle line:84% PROFESSOR: Somebody threw that out there. 01:03:32.708 --> 01:03:34.150 align:middle line:90% But I don't think that's right. 01:03:34.150 --> 01:03:35.080 align:middle line:84% STEVEN JOHNSON: But anyway, if you google it, 01:03:35.080 --> 01:03:36.908 align:middle line:84% you can find notes on these things. 01:03:36.908 --> 01:03:38.950 align:middle line:84% Alan, is there anything else you wanted to come-- 01:03:38.950 --> 01:03:40.120 align:middle line:90% what else did we leave out? 01:03:40.120 --> 01:03:41.330 align:middle line:84% PROFESSOR: Oh, there's bazillions of things. 01:03:41.330 --> 01:03:43.810 align:middle line:84% But I think that's a pretty good list right there. 01:03:43.810 --> 01:03:46.060 align:middle line:84% We can call it a day with that, I think. 01:03:46.060 --> 01:03:47.200 align:middle line:90% STEVEN JOHNSON: Yeah, yeah. 01:03:47.200 --> 01:03:48.430 align:middle line:90% There's so many more things. 01:03:48.430 --> 01:03:53.448 align:middle line:84% I mean, Alan did a little bit on derivatives of eigenvalues 01:03:53.448 --> 01:03:53.990 align:middle line:90% and so forth. 01:03:53.990 --> 01:03:55.630 align:middle line:84% But there's whole papers on how do 01:03:55.630 --> 01:03:58.383 align:middle line:84% you differentiate different matrix factorizations and-- 01:03:58.383 --> 01:04:00.550 align:middle line:84% PROFESSOR: I wrote chapters about this 20 years ago, 01:04:00.550 --> 01:04:02.710 align:middle line:84% but I never did anything with those chapters. 01:04:02.710 --> 01:04:04.090 align:middle line:84% And now, yeah, now everybody's discovering it. 01:04:04.090 --> 01:04:06.298 align:middle line:84% STEVEN JOHNSON: And there's [INAUDIBLE] open research 01:04:06.298 --> 01:04:08.320 align:middle line:84% problems in how to efficiently take 01:04:08.320 --> 01:04:10.862 align:middle line:84% care of some of these things, especially if some of the terms 01:04:10.862 --> 01:04:15.220 align:middle line:84% are sparse, and how do you track that information 01:04:15.220 --> 01:04:18.760 align:middle line:84% through the process to take full advantage of the structure 01:04:18.760 --> 01:04:19.710 align:middle line:90% of the matrix. 01:04:19.710 --> 01:04:21.585 align:middle line:84% PROFESSOR: But hopefully, you now all realize 01:04:21.585 --> 01:04:24.400 align:middle line:84% that calculus is way more interesting than your freshman 01:04:24.400 --> 01:04:25.840 align:middle line:90% teacher had you believe. 01:04:25.840 --> 01:04:27.507 align:middle line:84% STEVEN JOHNSON: Just taking derivatives. 01:04:27.507 --> 01:04:29.360 align:middle line:84% You think, oh, they tell you that, oh, 01:04:29.360 --> 01:04:30.950 align:middle line:90% taking derivatives is trivial. 01:04:30.950 --> 01:04:34.560 align:middle line:84% You learn these five rules, and then you can handle everything. 01:04:34.560 --> 01:04:36.830 align:middle line:84% PROFESSOR: And I think it's safe to say that most 01:04:36.830 --> 01:04:40.280 align:middle line:84% members of our department, they don't think about these things. 01:04:40.280 --> 01:04:42.530 align:middle line:84% They probably have never come across these things, 01:04:42.530 --> 01:04:44.120 align:middle line:90% by and large, I think. 01:04:44.120 --> 01:04:46.670 align:middle line:84% In other words, it's not something that's-- it's coming 01:04:46.670 --> 01:04:49.370 align:middle line:84% up more in computer science than mathematics, I believe. 01:04:49.370 --> 01:04:51.920 align:middle line:90% I think that's the truth. 01:04:51.920 --> 01:04:53.480 align:middle line:84% So if you mention matrix calculus 01:04:53.480 --> 01:04:56.300 align:middle line:84% to many people in math, they-- it may not mean too much, 01:04:56.300 --> 01:04:56.870 align:middle line:90% I suppose. 01:04:56.870 --> 01:04:57.832 align:middle line:90% I'm not sure. 01:04:57.832 --> 01:04:59.540 align:middle line:84% STEVEN JOHNSON: I mean, computer science, 01:04:59.540 --> 01:05:00.740 align:middle line:90% computational science-- 01:05:00.740 --> 01:05:02.740 align:middle line:84% PROFESSOR: It's coming up a lot more these days. 01:05:02.740 --> 01:05:05.666 align:middle line:84% STEVEN JOHNSON: That's where you end up differentiating-- 01:05:05.666 --> 01:05:08.030 align:middle line:84% I mean, you say, I know the rules for the derivatives, 01:05:08.030 --> 01:05:11.180 align:middle line:84% and now let me differentiate the drag on an airplane wing 01:05:11.180 --> 01:05:13.130 align:middle line:84% that I solved by Navier-Stokes equation. 01:05:13.130 --> 01:05:15.690 align:middle line:84% PROFESSOR: Well, give me that machine learning gradient 01:05:15.690 --> 01:05:16.640 align:middle line:90% to move along. 01:05:16.640 --> 01:05:17.660 align:middle line:84% STEVEN JOHNSON: And then you realize it's not 01:05:17.660 --> 01:05:19.010 align:middle line:90% so easy to take derivatives. 01:05:19.010 --> 01:05:21.030 align:middle line:90% PROFESSOR: Right. 01:05:21.030 --> 01:05:21.560 align:middle line:90% All right. 01:05:21.560 --> 01:05:22.280 align:middle line:90% I think that's it. 01:05:22.280 --> 01:05:23.072 align:middle line:90% STEVEN JOHNSON: OK. 01:05:23.072 --> 01:05:23.720 align:middle line:90% All right. 01:05:23.720 --> 01:05:26.470 align:middle line:90% PROFESSOR: So thanks, everybody. 01:05:26.470 --> 01:05:29.000 align:middle line:90%