WEBVTT 00:00:00.000 --> 00:00:01.976 align:middle line:90% [SQUEAKING] 00:00:01.976 --> 00:00:03.458 align:middle line:90% [RUSTLING] 00:00:03.458 --> 00:00:05.434 align:middle line:90% [CLICKING] 00:00:05.434 --> 00:00:11.580 align:middle line:90% 00:00:11.580 --> 00:00:15.440 align:middle line:84% PROFESSOR: So today, for the first half of the lecture, 00:00:15.440 --> 00:00:18.880 align:middle line:84% we're going to have another guest lecturer, Gaurav Arya, 00:00:18.880 --> 00:00:20.740 align:middle line:84% who's actually an undergraduate at MIT. 00:00:20.740 --> 00:00:22.780 align:middle line:84% But he's done research projects with me, 00:00:22.780 --> 00:00:25.810 align:middle line:84% and also research projects with Alan Edelman's group, 00:00:25.810 --> 00:00:30.860 align:middle line:84% and he's done some really interesting recent work 00:00:30.860 --> 00:00:36.230 align:middle line:84% on differentiating random functions, like you program, 00:00:36.230 --> 00:00:40.330 align:middle line:84% you flip a coin, and 50% it gives you sine, 50% chance it 00:00:40.330 --> 00:00:41.210 align:middle line:90% gives you cosine. 00:00:41.210 --> 00:00:43.335 align:middle line:84% So what does it mean to differentiate such a thing, 00:00:43.335 --> 00:00:45.280 align:middle line:84% and how do you do it automatically? 00:00:45.280 --> 00:00:47.320 align:middle line:84% And so I thought that would be a-- 00:00:47.320 --> 00:00:51.190 align:middle line:84% we thought that would be a nice topic for half a lecture. 00:00:51.190 --> 00:00:53.890 align:middle line:84% So he's in Hong Kong right now, but he's 00:00:53.890 --> 00:00:55.850 align:middle line:90% going to give a guest lecture. 00:00:55.850 --> 00:00:58.800 align:middle line:84% So I think we can get started, Gaurav. 00:00:58.800 --> 00:01:00.420 align:middle line:90% GAURAV ARYA: OK. 00:01:00.420 --> 00:01:03.335 align:middle line:84% Yeah, so I'm-- can you hear me all right? 00:01:03.335 --> 00:01:04.720 align:middle line:90% PROFESSOR: Can anyone hear him? 00:01:04.720 --> 00:01:05.430 align:middle line:90% Good. 00:01:05.430 --> 00:01:06.353 align:middle line:90% Good, yeah. 00:01:06.353 --> 00:01:07.020 align:middle line:90% GAURAV ARYA: OK. 00:01:07.020 --> 00:01:08.630 align:middle line:90% Yeah, yeah. 00:01:08.630 --> 00:01:09.600 align:middle line:90% I see myself. 00:01:09.600 --> 00:01:10.517 align:middle line:90% PROFESSOR: Yeah, yeah. 00:01:10.517 --> 00:01:14.685 align:middle line:84% So I'll try and repeat any questions and things like that. 00:01:14.685 --> 00:01:16.080 align:middle line:90% GAURAV ARYA: Yeah. 00:01:16.080 --> 00:01:18.570 align:middle line:90% So I'm a junior at MIT. 00:01:18.570 --> 00:01:20.280 align:middle line:84% So yeah, as Professor Johnson said. 00:01:20.280 --> 00:01:23.580 align:middle line:84% I did a year with Professor Johnson previously. 00:01:23.580 --> 00:01:27.120 align:middle line:84% And recently, I've been doing a year in the Julia Lab, which 00:01:27.120 --> 00:01:29.580 align:middle line:90% is led by Professor Edelman. 00:01:29.580 --> 00:01:33.330 align:middle line:84% And today, I'm going to be talking about derivatives 00:01:33.330 --> 00:01:34.990 align:middle line:90% of random functions. 00:01:34.990 --> 00:01:41.392 align:middle line:90% So I'm going to try and do this. 00:01:41.392 --> 00:01:42.610 align:middle line:90% Push right-- oops. 00:01:42.610 --> 00:01:46.480 align:middle line:90% That's not what I wanted to do. 00:01:46.480 --> 00:01:46.980 align:middle line:90% Yeah. 00:01:46.980 --> 00:01:50.470 align:middle line:84% So I wanted to start by just giving a bit of motivation. 00:01:50.470 --> 00:01:52.260 align:middle line:90% So I'm going to just-- 00:01:52.260 --> 00:01:54.540 align:middle line:84% before I go into the random functions, 00:01:54.540 --> 00:01:58.110 align:middle line:84% I'm going to write this function, which 00:01:58.110 --> 00:02:02.100 align:middle line:84% I think you all probably know how to differentiate better 00:02:02.100 --> 00:02:03.490 align:middle line:90% than I do at this point. 00:02:03.490 --> 00:02:10.845 align:middle line:84% So here, A is a matrix, and this is a matrix-accepting function. 00:02:10.845 --> 00:02:11.970 align:middle line:90% Can you read this all fine? 00:02:11.970 --> 00:02:12.870 align:middle line:90% PROFESSOR: Yep. 00:02:12.870 --> 00:02:13.980 align:middle line:90% We can see it. 00:02:13.980 --> 00:02:14.730 align:middle line:90% GAURAV ARYA: Yeah. 00:02:14.730 --> 00:02:18.760 align:middle line:84% So this is a function with matrix input and matrix output. 00:02:18.760 --> 00:02:20.940 align:middle line:84% And the reason I want to go through this 00:02:20.940 --> 00:02:27.150 align:middle line:84% again is just to set up an analogy with how I-- 00:02:27.150 --> 00:02:28.950 align:middle line:84% with differentiating random functions, 00:02:28.950 --> 00:02:32.400 align:middle line:84% what are the broad steps that 1 takes in-- 00:02:32.400 --> 00:02:34.770 align:middle line:84% when you're faced with a function like this. 00:02:34.770 --> 00:02:38.160 align:middle line:84% The essential complication is that your input and output 00:02:38.160 --> 00:02:40.782 align:middle line:84% spaces are much more complicated, 00:02:40.782 --> 00:02:42.240 align:middle line:84% and they're not real-to-reals, like 00:02:42.240 --> 00:02:44.110 align:middle line:90% in single variable calculus. 00:02:44.110 --> 00:02:46.600 align:middle line:84% So when you want to differentiate a function 00:02:46.600 --> 00:02:49.960 align:middle line:84% like this, it becomes a lot harder 00:02:49.960 --> 00:02:53.560 align:middle line:84% to just do it super mechanically like you might do 00:02:53.560 --> 00:02:55.540 align:middle line:90% for single variable calculus. 00:02:55.540 --> 00:02:58.450 align:middle line:84% And it becomes a lot more appealing to go back 00:02:58.450 --> 00:03:01.660 align:middle line:84% to the fundamental questions that you 00:03:01.660 --> 00:03:03.520 align:middle line:90% ask when taking a derivative. 00:03:03.520 --> 00:03:06.820 align:middle line:84% So the first might be something like-- 00:03:06.820 --> 00:03:09.850 align:middle line:84% ultimately, derivatives are about when 00:03:09.850 --> 00:03:13.150 align:middle line:84% you perturb the input slightly, how does the output perturb? 00:03:13.150 --> 00:03:16.360 align:middle line:84% A map from input wiggles, the output wiggles. 00:03:16.360 --> 00:03:17.830 align:middle line:84% So the first question you might ask 00:03:17.830 --> 00:03:23.080 align:middle line:84% is just, if I perturb the input, how does the output change? 00:03:23.080 --> 00:03:25.960 align:middle line:90% So perturb the input. 00:03:25.960 --> 00:03:30.860 align:middle line:90% 00:03:30.860 --> 00:03:33.700 align:middle line:90% How does the output change? 00:03:33.700 --> 00:03:35.800 align:middle line:84% What's the sensitivity of the output 00:03:35.800 --> 00:03:39.190 align:middle line:90% with respect to the input? 00:03:39.190 --> 00:03:42.940 align:middle line:84% So for the case of this function, what you've been-- 00:03:42.940 --> 00:03:46.510 align:middle line:84% what you've done in the class is you may have written something 00:03:46.510 --> 00:03:47.740 align:middle line:90% like this. 00:03:47.740 --> 00:03:49.960 align:middle line:84% You might have used this notation df, 00:03:49.960 --> 00:03:51.880 align:middle line:90% which is the differential of f. 00:03:51.880 --> 00:03:53.087 align:middle line:90% PROFESSOR: Yep. 00:03:53.087 --> 00:03:53.920 align:middle line:90% That's the notation. 00:03:53.920 --> 00:03:54.460 align:middle line:90% Yeah. 00:03:54.460 --> 00:03:55.400 align:middle line:90% GAURAV ARYA: Yeah. 00:03:55.400 --> 00:03:55.900 align:middle line:90% Yep. 00:03:55.900 --> 00:03:56.260 align:middle line:90% Sorry? 00:03:56.260 --> 00:03:57.760 align:middle line:84% PROFESSOR: That's the notation, yes. 00:03:57.760 --> 00:03:59.440 align:middle line:90% GAURAV ARYA: Yeah, OK. 00:03:59.440 --> 00:04:03.985 align:middle line:84% And you would have replaced A with A plus dA. 00:04:03.985 --> 00:04:06.940 align:middle line:84% So the A is your wiggle and the input, 00:04:06.940 --> 00:04:08.020 align:middle line:90% this is the differential. 00:04:08.020 --> 00:04:13.000 align:middle line:84% So we subtract off the original A squared. 00:04:13.000 --> 00:04:15.050 align:middle line:90% You can expand this out. 00:04:15.050 --> 00:04:20.050 align:middle line:84% So the A squares cancel, so you have-- you've done this, 00:04:20.050 --> 00:04:22.820 align:middle line:90% I'm sure, quite a few times. 00:04:22.820 --> 00:04:25.990 align:middle line:84% Then we have this dA squared term here. 00:04:25.990 --> 00:04:27.430 align:middle line:90% And this-- 00:04:27.430 --> 00:04:30.490 align:middle line:84% now, I know that the next step you're probably eager to do 00:04:30.490 --> 00:04:32.440 align:middle line:90% is to cancel out the dA squared. 00:04:32.440 --> 00:04:37.180 align:middle line:84% But just this on its own is the answer to question 1. 00:04:37.180 --> 00:04:42.550 align:middle line:84% We're not worried about neglecting the higher order 00:04:42.550 --> 00:04:46.150 align:middle line:84% terms or forming the derivative, but just thinking 00:04:46.150 --> 00:04:50.170 align:middle line:84% about how do we write down the change in the output. 00:04:50.170 --> 00:04:52.190 align:middle line:90% And this is your function. 00:04:52.190 --> 00:04:54.310 align:middle line:84% So this is your answer to question 1. 00:04:54.310 --> 00:04:59.890 align:middle line:84% And then the second thing is you have this differential, 00:04:59.890 --> 00:05:04.150 align:middle line:84% and you want to figure out what terms can you neglect here. 00:05:04.150 --> 00:05:06.340 align:middle line:84% So what terms do I keep, what terms do I neglect? 00:05:06.340 --> 00:05:08.770 align:middle line:84% Because, really, our goal at this point 00:05:08.770 --> 00:05:11.440 align:middle line:84% is to form this derivative, which 00:05:11.440 --> 00:05:14.860 align:middle line:84% the idea of the derivative is to capture 00:05:14.860 --> 00:05:17.770 align:middle line:84% the sensitivity of f with respect 00:05:17.770 --> 00:05:20.110 align:middle line:90% to A to the first order. 00:05:20.110 --> 00:05:22.580 align:middle line:84% The most important effects to the first order, 00:05:22.580 --> 00:05:24.750 align:middle line:90% and we have to drop the rest. 00:05:24.750 --> 00:05:30.800 align:middle line:84% So the second question is what terms to neglect, like that. 00:05:30.800 --> 00:05:35.060 align:middle line:90% 00:05:35.060 --> 00:05:35.600 align:middle line:90% Yeah. 00:05:35.600 --> 00:05:39.980 align:middle line:84% And here, you can just cross off the dA squared. 00:05:39.980 --> 00:05:41.580 align:middle line:90% That's a higher order term. 00:05:41.580 --> 00:05:43.460 align:middle line:90% So you can justify it that way. 00:05:43.460 --> 00:05:50.910 align:middle line:84% And your derivative operator becomes this map from dA 00:05:50.910 --> 00:05:54.670 align:middle line:90% to what's left here. 00:05:54.670 --> 00:05:55.170 align:middle line:90% Right? 00:05:55.170 --> 00:06:00.720 align:middle line:90% The dA times A plus A times dA. 00:06:00.720 --> 00:06:05.610 align:middle line:84% So I just wanted to separate out these two steps, 00:06:05.610 --> 00:06:08.460 align:middle line:84% because both of them are kind of non-trivial 00:06:08.460 --> 00:06:10.270 align:middle line:90% in the case of random functions. 00:06:10.270 --> 00:06:13.710 align:middle line:84% So the first is just thinking about this map 00:06:13.710 --> 00:06:16.290 align:middle line:84% from input perturbations to output perturbations. 00:06:16.290 --> 00:06:18.090 align:middle line:90% And the second is, OK, great. 00:06:18.090 --> 00:06:19.200 align:middle line:90% I understand that. 00:06:19.200 --> 00:06:22.590 align:middle line:84% But now what terms should I neglect 00:06:22.590 --> 00:06:25.590 align:middle line:90% to form this derivative? 00:06:25.590 --> 00:06:26.980 align:middle line:90% Does that make sense? 00:06:26.980 --> 00:06:28.178 align:middle line:90% Are there any questions? 00:06:28.178 --> 00:06:29.220 align:middle line:90% PROFESSOR: Any questions? 00:06:29.220 --> 00:06:29.970 align:middle line:90% Nope. 00:06:29.970 --> 00:06:31.358 align:middle line:90% People are shaking their heads. 00:06:31.358 --> 00:06:32.150 align:middle line:90% GAURAV ARYA: Great. 00:06:32.150 --> 00:06:33.775 align:middle line:84% PROFESSOR: They've seen this 100 times. 00:06:33.775 --> 00:06:36.330 align:middle line:84% GAURAV ARYA: So that is a matrix-to-matrix function. 00:06:36.330 --> 00:06:39.990 align:middle line:84% But the idea there was just thinking 00:06:39.990 --> 00:06:42.270 align:middle line:84% about a function with a more complicated input 00:06:42.270 --> 00:06:44.860 align:middle line:84% space and a more complicated output space. 00:06:44.860 --> 00:06:51.100 align:middle line:84% So now we're going to shift our attention to another case 00:06:51.100 --> 00:06:53.110 align:middle line:84% where we were trying to generalize 00:06:53.110 --> 00:06:57.520 align:middle line:84% the notion of the derivative to more complicated spaces. 00:06:57.520 --> 00:07:01.520 align:middle line:84% And this time, we're considering functions which are random. 00:07:01.520 --> 00:07:07.940 align:middle line:84% So more concretely, I'm going to write a random function 00:07:07.940 --> 00:07:10.700 align:middle line:84% with a capital letter X, which already might 00:07:10.700 --> 00:07:14.360 align:middle line:84% be a bit suggestive, because capital letters are often used 00:07:14.360 --> 00:07:17.180 align:middle line:90% for denoting random variables. 00:07:17.180 --> 00:07:22.740 align:middle line:84% And indeed, this is a random variable value function. 00:07:22.740 --> 00:07:32.460 align:middle line:84% So as a map, we might write that it takes in some input P, 00:07:32.460 --> 00:07:43.300 align:middle line:84% and it spits out X of P, where X of P is a random variable. 00:07:43.300 --> 00:07:44.615 align:middle line:90% And I'm going to keep-- 00:07:44.615 --> 00:07:46.240 align:middle line:84% the output space is pretty complicated, 00:07:46.240 --> 00:07:48.450 align:middle line:84% so I'm going to keep the output space fairly simple. 00:07:48.450 --> 00:07:51.550 align:middle line:84% The input, let's assume, is just a single real number. 00:07:51.550 --> 00:07:55.280 align:middle line:90% So P belongs to the reals. 00:07:55.280 --> 00:07:55.780 align:middle line:90% So yeah. 00:07:55.780 --> 00:07:58.600 align:middle line:84% So now we want to ask the same question. 00:07:58.600 --> 00:07:59.380 align:middle line:90% How-- 00:07:59.380 --> 00:08:04.090 align:middle line:84% So for our matrix function, we started with this map from to A 00:08:04.090 --> 00:08:07.555 align:middle line:84% to A squared, and we got as our answer for the derivative 00:08:07.555 --> 00:08:12.520 align:middle line:84% another map, map from dA to dA times A plus A times dA. 00:08:12.520 --> 00:08:18.100 align:middle line:84% And now we want to ask, what is a useful notion of derivative? 00:08:18.100 --> 00:08:21.640 align:middle line:84% First of all, what is our notion of differential? 00:08:21.640 --> 00:08:25.300 align:middle line:84% How does the perturbation of the input affect-- 00:08:25.300 --> 00:08:26.950 align:middle line:84% if every time I run this function 00:08:26.950 --> 00:08:29.470 align:middle line:84% I get a different value if it's a random function, 00:08:29.470 --> 00:08:33.740 align:middle line:84% what is a useful notion of sensitivity of the output, 00:08:33.740 --> 00:08:34.340 align:middle line:90% and so forth? 00:08:34.340 --> 00:08:36.700 align:middle line:84% So we want to do the same process 00:08:36.700 --> 00:08:38.926 align:middle line:90% on this random function. 00:08:38.926 --> 00:08:41.080 align:middle line:84% PROFESSOR: So does everyone know what he 00:08:41.080 --> 00:08:42.480 align:middle line:90% means by a "random variable." 00:08:42.480 --> 00:08:47.720 align:middle line:84% Like I said, suppose you flip a coin, and if it's heads, 00:08:47.720 --> 00:08:48.490 align:middle line:90% you return 1. 00:08:48.490 --> 00:08:50.035 align:middle line:90% And if it's tails, you return 0. 00:08:50.035 --> 00:08:50.930 align:middle line:90% All right? 00:08:50.930 --> 00:08:52.790 align:middle line:84% That would be a discrete random variable. 00:08:52.790 --> 00:08:53.540 align:middle line:90% GAURAV ARYA: Yeah. 00:08:53.540 --> 00:08:55.530 align:middle line:84% Why don't I actually show some examples? 00:08:55.530 --> 00:08:57.050 align:middle line:90% So let me first write them down. 00:08:57.050 --> 00:08:57.675 align:middle line:90% PROFESSOR: Yes. 00:08:57.675 --> 00:09:03.710 align:middle line:84% GAURAV ARYA: So we could have, let's say, a random variable-- 00:09:03.710 --> 00:09:07.540 align:middle line:84% and one crucial thing is-- so this is a random variable value 00:09:07.540 --> 00:09:08.040 align:middle line:90% function. 00:09:08.040 --> 00:09:11.900 align:middle line:84% So it's actually a whole family of random variables indexed 00:09:11.900 --> 00:09:16.100 align:middle line:84% by parameter P. So let's say one example 00:09:16.100 --> 00:09:22.060 align:middle line:84% you could have is X of P could be following the Bernoulli 00:09:22.060 --> 00:09:23.980 align:middle line:84% distribution with probability P. This 00:09:23.980 --> 00:09:28.480 align:middle line:84% is your example of probability P of heads and otherwise tails. 00:09:28.480 --> 00:09:30.680 align:middle line:90% Then maybe another example-- 00:09:30.680 --> 00:09:31.810 align:middle line:90% PROFESSOR: So that yeah. 00:09:31.810 --> 00:09:34.585 align:middle line:84% With probability P, you return 1, and with probability-- 00:09:34.585 --> 00:09:35.335 align:middle line:90% GAURAV ARYA: Yeah. 00:09:35.335 --> 00:09:37.625 align:middle line:84% PROFESSOR: --1 minus P returns 0, for example. 00:09:37.625 --> 00:09:38.500 align:middle line:90% GAURAV ARYA: Exactly. 00:09:38.500 --> 00:09:39.250 align:middle line:90% Yeah. 00:09:39.250 --> 00:09:41.060 align:middle line:90% And maybe another example. 00:09:41.060 --> 00:09:44.200 align:middle line:84% So I'm going to write an exponential distribution here, 00:09:44.200 --> 00:09:47.990 align:middle line:84% and I'm not hoping to really say what it is. 00:09:47.990 --> 00:09:50.350 align:middle line:84% But the point is, this one's a bit different in nature 00:09:50.350 --> 00:09:53.017 align:middle line:84% from the Bernoulli, because it's like a continuous distribution. 00:09:53.017 --> 00:09:55.810 align:middle line:84% And I'll show you how examples from that distribution 00:09:55.810 --> 00:09:56.990 align:middle line:90% look as well. 00:09:56.990 --> 00:10:03.380 align:middle line:84% So let me just try and share my screen. 00:10:03.380 --> 00:10:04.095 align:middle line:90% Can you see-- 00:10:04.095 --> 00:10:04.720 align:middle line:90% PROFESSOR: Yep. 00:10:04.720 --> 00:10:05.110 align:middle line:90% GAURAV ARYA: You can see well? 00:10:05.110 --> 00:10:06.190 align:middle line:84% PROFESSOR: We can see your screen, yeah. 00:10:06.190 --> 00:10:06.857 align:middle line:90% GAURAV ARYA: OK. 00:10:06.857 --> 00:10:10.610 align:middle line:90% So yeah. 00:10:10.610 --> 00:10:12.410 align:middle line:84% So let's consider the Bernoulli example. 00:10:12.410 --> 00:10:15.950 align:middle line:90% 00:10:15.950 --> 00:10:18.500 align:middle line:90% So X of P-- 00:10:18.500 --> 00:10:21.230 align:middle line:84% so really, what I'm writing here is a program 00:10:21.230 --> 00:10:22.760 align:middle line:90% that samples from X of P. 00:10:22.760 --> 00:10:24.740 align:middle line:90% So I'll call it sampleX(p). 00:10:24.740 --> 00:10:30.130 align:middle line:84% And in this case, this is basically a sampling 00:10:30.130 --> 00:10:31.380 align:middle line:90% from a Bernoulli distribution. 00:10:31.380 --> 00:10:34.040 align:middle line:84% So this thing inside of the rand() is the distribution. 00:10:34.040 --> 00:10:36.150 align:middle line:84% But we're getting samples from it, 00:10:36.150 --> 00:10:38.820 align:middle line:84% so that's why there's this rand in front. 00:10:38.820 --> 00:10:42.990 align:middle line:84% OK, so now I can run this program with X of 0.6 00:10:42.990 --> 00:10:44.880 align:middle line:90% and sampling from X of 0.6. 00:10:44.880 --> 00:10:46.860 align:middle line:90% In this case, it's a 1 or 0. 00:10:46.860 --> 00:10:48.360 align:middle line:84% In this case, it's using a Boolean 00:10:48.360 --> 00:10:49.620 align:middle line:90% to represent the output. 00:10:49.620 --> 00:10:52.260 align:middle line:84% But the point is that you're going 00:10:52.260 --> 00:10:57.100 align:middle line:84% to get true 60% of the time and false 40% of the time. 00:10:57.100 --> 00:11:01.680 align:middle line:84% So you can verify this by taking a bunch of samples. 00:11:01.680 --> 00:11:04.710 align:middle line:84% PROFESSOR: In Julia, true is equivalent to the number 1, 00:11:04.710 --> 00:11:07.320 align:middle line:84% and false is equivalent to the number 0. 00:11:07.320 --> 00:11:09.060 align:middle line:90% GAURAV ARYA: Yeah. 00:11:09.060 --> 00:11:11.340 align:middle line:84% And then let's just stare at the other example 00:11:11.340 --> 00:11:14.195 align:middle line:90% of an exponential distribution. 00:11:14.195 --> 00:11:15.570 align:middle line:84% So the Bernoulli is what's called 00:11:15.570 --> 00:11:18.180 align:middle line:90% a discrete random variable. 00:11:18.180 --> 00:11:20.100 align:middle line:84% And now for an exponential distribution, 00:11:20.100 --> 00:11:23.160 align:middle line:90% maybe I'll make P 100. 00:11:23.160 --> 00:11:26.128 align:middle line:84% And now it's doing some continuous random thing. 00:11:26.128 --> 00:11:28.420 align:middle line:84% But the point is you get a different output every time. 00:11:28.420 --> 00:11:29.985 align:middle line:90% Now, the question is-- 00:11:29.985 --> 00:11:31.215 align:middle line:84% PROFESSOR: Do people know what the exponential-- 00:11:31.215 --> 00:11:31.530 align:middle line:90% GAURAV ARYA: --differentiation. 00:11:31.530 --> 00:11:32.908 align:middle line:90% So someone codes up a function-- 00:11:32.908 --> 00:11:33.700 align:middle line:90% PROFESSOR: One sec. 00:11:33.700 --> 00:11:34.125 align:middle line:90% One second. 00:11:34.125 --> 00:11:34.380 align:middle line:90% GAURAV ARYA: Yeah. 00:11:34.380 --> 00:11:36.390 align:middle line:84% PROFESSOR: Do people know what the exponential distribution 00:11:36.390 --> 00:11:36.890 align:middle line:90% means? 00:11:36.890 --> 00:11:39.330 align:middle line:90% It means the probability-- 00:11:39.330 --> 00:11:43.000 align:middle line:84% you're giving a non-negative real number. 00:11:43.000 --> 00:11:44.970 align:middle line:90% It's returning continuously out. 00:11:44.970 --> 00:11:47.730 align:middle line:84% But the probability decays exponentially 00:11:47.730 --> 00:11:48.930 align:middle line:90% as it gets larger. 00:11:48.930 --> 00:11:52.500 align:middle line:84% And so P here is just the scale of that exponential. 00:11:52.500 --> 00:11:55.800 align:middle line:84% So your-- the real number is in the order of-- 00:11:55.800 --> 00:11:58.590 align:middle line:84% sampleX(100) is real numbers in the order of 100. 00:11:58.590 --> 00:11:59.440 align:middle line:90% It can be larger. 00:11:59.440 --> 00:12:00.400 align:middle line:90% It can be smaller. 00:12:00.400 --> 00:12:02.100 align:middle line:84% But as it gets bigger than 100, it 00:12:02.100 --> 00:12:05.220 align:middle line:84% gets exponentially less likely, and then follows 00:12:05.220 --> 00:12:07.020 align:middle line:90% an exponential curve. 00:12:07.020 --> 00:12:07.920 align:middle line:90% GAURAV ARYA: Right. 00:12:07.920 --> 00:12:08.420 align:middle line:90% Yeah. 00:12:08.420 --> 00:12:11.160 align:middle line:84% PROFESSOR: But it's continuous numbers that are coming out. 00:12:11.160 --> 00:12:12.310 align:middle line:84% GAURAV ARYA: Yeah, so I should have said that. 00:12:12.310 --> 00:12:14.790 align:middle line:84% So here, I'm parameterizing exponential distribution 00:12:14.790 --> 00:12:15.840 align:middle line:90% by its scale. 00:12:15.840 --> 00:12:18.300 align:middle line:90% Yeah. 00:12:18.300 --> 00:12:22.650 align:middle line:84% So another question is, what is a useful notion of derivative 00:12:22.650 --> 00:12:23.610 align:middle line:90% for programs like this? 00:12:23.610 --> 00:12:26.100 align:middle line:84% Someone codes up a program that samples 00:12:26.100 --> 00:12:29.580 align:middle line:84% from an exponential distribution. 00:12:29.580 --> 00:12:31.480 align:middle line:84% What's a useful notion of derivatives? 00:12:31.480 --> 00:12:32.970 align:middle line:90% So there's two questions here. 00:12:32.970 --> 00:12:37.740 align:middle line:84% There's how do we find this useful notion of derivative, 00:12:37.740 --> 00:12:40.270 align:middle line:90% but there's also the why. 00:12:40.270 --> 00:12:45.940 align:middle line:84% Why would we-- for the functions you considered so far, 00:12:45.940 --> 00:12:50.950 align:middle line:84% matrix value functions, the why is pretty clear, 00:12:50.950 --> 00:12:54.910 align:middle line:84% in that gradients are useful for optimizing functions 00:12:54.910 --> 00:12:55.850 align:middle line:90% and so forth. 00:12:55.850 --> 00:12:58.760 align:middle line:84% And if you have a totally deterministic function, 00:12:58.760 --> 00:13:00.220 align:middle line:84% then it's very clear what it means 00:13:00.220 --> 00:13:01.653 align:middle line:90% to optimize that function. 00:13:01.653 --> 00:13:03.070 align:middle line:84% You have some objective and you're 00:13:03.070 --> 00:13:04.695 align:middle line:84% trying to make it as small as possible, 00:13:04.695 --> 00:13:07.540 align:middle line:84% and that's why you want to take gradients. 00:13:07.540 --> 00:13:11.050 align:middle line:84% But for these random functions, maybe it's 00:13:11.050 --> 00:13:15.550 align:middle line:84% not immediately obvious why we want any notion of derivative 00:13:15.550 --> 00:13:16.550 align:middle line:90% for these programs. 00:13:16.550 --> 00:13:20.440 align:middle line:84% So why do we want to differentiate these? 00:13:20.440 --> 00:13:24.100 align:middle line:84% And maybe this is a question I can ask to the class. 00:13:24.100 --> 00:13:26.470 align:middle line:84% Why might we want to differentiate 00:13:26.470 --> 00:13:28.150 align:middle line:90% these sorts of programs? 00:13:28.150 --> 00:13:31.060 align:middle line:84% PROFESSOR: Does anyone have any notion 00:13:31.060 --> 00:13:33.700 align:middle line:84% of why you would want to differentiate whatever 00:13:33.700 --> 00:13:35.695 align:middle line:84% deri-- we haven't defined what it means, 00:13:35.695 --> 00:13:38.410 align:middle line:84% but why would we want the sensitivity, in some sense, 00:13:38.410 --> 00:13:41.230 align:middle line:84% of that program with respect to some parameter, 00:13:41.230 --> 00:13:43.930 align:middle line:84% even though the outputs are random? 00:13:43.930 --> 00:13:44.543 align:middle line:90% Yeah. 00:13:44.543 --> 00:13:45.460 align:middle line:90% Someone raised-- yeah? 00:13:45.460 --> 00:13:47.085 align:middle line:84% AUDIENCE: Maybe maximize the likelihood 00:13:47.085 --> 00:13:48.070 align:middle line:90% of something occurring? 00:13:48.070 --> 00:13:50.403 align:middle line:84% PROFESSOR: Maybe to maximize the likelihood of something 00:13:50.403 --> 00:13:51.460 align:middle line:90% occurring, for example. 00:13:51.460 --> 00:13:52.090 align:middle line:90% That's-- 00:13:52.090 --> 00:13:52.757 align:middle line:90% GAURAV ARYA: OK. 00:13:52.757 --> 00:13:54.132 align:middle line:90% Yeah, yeah, yeah. 00:13:54.132 --> 00:13:55.090 align:middle line:90% PROFESSOR: Another one? 00:13:55.090 --> 00:13:55.590 align:middle line:90% Yeah? 00:13:55.590 --> 00:13:58.660 align:middle line:84% AUDIENCE: In physics, there are lots of systems with disorder 00:13:58.660 --> 00:14:01.400 align:middle line:90% that are random, in some sense. 00:14:01.400 --> 00:14:04.227 align:middle line:84% And if you'd like to measure properties of that system, 00:14:04.227 --> 00:14:06.310 align:middle line:84% then you need to take the randomness into account. 00:14:06.310 --> 00:14:06.977 align:middle line:90% PROFESSOR: Yeah. 00:14:06.977 --> 00:14:09.160 align:middle line:84% So if you have a physical system with randomness, 00:14:09.160 --> 00:14:14.650 align:middle line:84% and you want to measure the sensitivity of something, 00:14:14.650 --> 00:14:16.075 align:middle line:90% some statistical quantity to-- 00:14:16.075 --> 00:14:17.200 align:middle line:90% GAURAV ARYA: Exactly, yeah. 00:14:17.200 --> 00:14:18.230 align:middle line:84% PROFESSOR: --some physical parameter. 00:14:18.230 --> 00:14:18.730 align:middle line:90% Right. 00:14:18.730 --> 00:14:19.810 align:middle line:90% GAURAV ARYA: Yeah, yeah. 00:14:19.810 --> 00:14:22.300 align:middle line:84% So both those answers are, yeah, totally correct. 00:14:22.300 --> 00:14:26.590 align:middle line:84% So basically, the idea is maybe you're normally 00:14:26.590 --> 00:14:30.500 align:middle line:84% interested in some statistical quantities of your system. 00:14:30.500 --> 00:14:33.790 align:middle line:90% So X is your random variable. 00:14:33.790 --> 00:14:36.790 align:middle line:84% But ultimately, it might be computing 00:14:36.790 --> 00:14:42.190 align:middle line:84% some stochastic estimate which, on average, is 00:14:42.190 --> 00:14:45.830 align:middle line:84% going to give you something you want to minimize, for example. 00:14:45.830 --> 00:14:49.210 align:middle line:90% So I misspelled that. 00:14:49.210 --> 00:14:50.650 align:middle line:84% So the physical quantity basically 00:14:50.650 --> 00:14:54.060 align:middle line:84% mean things that are constructed using averages. 00:14:54.060 --> 00:14:55.990 align:middle line:84% So they're based on expectations. 00:14:55.990 --> 00:14:59.270 align:middle line:90% 00:14:59.270 --> 00:15:04.280 align:middle line:84% And you may want to get these expectations somewhere 00:15:04.280 --> 00:15:08.180 align:middle line:84% you want them, and you want to find a P that maybe minimizes 00:15:08.180 --> 00:15:10.300 align:middle line:90% some expectation, and so forth. 00:15:10.300 --> 00:15:11.670 align:middle line:90% So broadly, this is the why. 00:15:11.670 --> 00:15:15.790 align:middle line:84% And there's one more other point today, which might-- 00:15:15.790 --> 00:15:17.940 align:middle line:84% which is that, sure, you have this system which 00:15:17.940 --> 00:15:21.000 align:middle line:90% is, in principle, stochastic. 00:15:21.000 --> 00:15:24.190 align:middle line:84% But you're interested in maybe its average value. 00:15:24.190 --> 00:15:27.690 align:middle line:84% So why not just write a program which 00:15:27.690 --> 00:15:29.820 align:middle line:84% directly computes that average value, 00:15:29.820 --> 00:15:31.800 align:middle line:90% and is therefore deterministic? 00:15:31.800 --> 00:15:36.390 align:middle line:84% If you have X of P, then the program 00:15:36.390 --> 00:15:38.640 align:middle line:84% you might be interested in is actually the expectation 00:15:38.640 --> 00:15:41.010 align:middle line:90% of X of P's average value. 00:15:41.010 --> 00:15:44.040 align:middle line:84% And this is a deterministic function of P. 00:15:44.040 --> 00:15:48.400 align:middle line:84% So why not just code that up and differentiate that? 00:15:48.400 --> 00:15:52.470 align:middle line:84% And the answer why we don't do that sometimes-- 00:15:52.470 --> 00:15:54.300 align:middle line:90% PROFESSOR: So quick question. 00:15:54.300 --> 00:15:55.050 align:middle line:90% GAURAV ARYA: Yeah? 00:15:55.050 --> 00:15:56.258 align:middle line:90% PROFESSOR: So quick question. 00:15:56.258 --> 00:16:00.900 align:middle line:84% So for the Bernoulli function that returned 1 00:16:00.900 --> 00:16:03.930 align:middle line:84% with probability P and 0 with probability 1 minus P, 00:16:03.930 --> 00:16:08.310 align:middle line:84% what's the expectation value of that? 00:16:08.310 --> 00:16:09.375 align:middle line:90% It's just P. Right? 00:16:09.375 --> 00:16:10.640 align:middle line:90% So yeah. 00:16:10.640 --> 00:16:11.140 align:middle line:90% Yeah. 00:16:11.140 --> 00:16:12.180 align:middle line:90% GAURAV ARYA: Yeah. 00:16:12.180 --> 00:16:13.680 align:middle line:84% PROFESSOR: And P is an easy function 00:16:13.680 --> 00:16:17.315 align:middle line:84% to differentiate with respect to P. Right? 00:16:17.315 --> 00:16:18.315 align:middle line:90% GAURAV ARYA: Yeah, yeah. 00:16:18.315 --> 00:16:21.773 align:middle line:90% 00:16:21.773 --> 00:16:22.440 align:middle line:90% So that's right. 00:16:22.440 --> 00:16:24.570 align:middle line:84% But you might have, for example, a very simple-- 00:16:24.570 --> 00:16:25.710 align:middle line:90% a random walk. 00:16:25.710 --> 00:16:27.750 align:middle line:84% In maybe five lines of code with a for loop, 00:16:27.750 --> 00:16:30.780 align:middle line:84% you can write a random walk which walks left-to-right. 00:16:30.780 --> 00:16:33.105 align:middle line:84% Maybe it changes the probability of left-to-right 00:16:33.105 --> 00:16:34.230 align:middle line:90% depending on where you are. 00:16:34.230 --> 00:16:36.240 align:middle line:84% All that you can do in five lines of code. 00:16:36.240 --> 00:16:41.070 align:middle line:84% And now it becomes a lot easier to sample from this process 00:16:41.070 --> 00:16:43.625 align:middle line:84% than to analytically compute its average. 00:16:43.625 --> 00:16:45.000 align:middle line:84% To compute its average, you might 00:16:45.000 --> 00:16:47.542 align:middle line:84% have to look at the transition matrix or something like that. 00:16:47.542 --> 00:16:50.950 align:middle line:84% But it's super easy to sample from the process. 00:16:50.950 --> 00:16:52.470 align:middle line:90% So that's the point. 00:16:52.470 --> 00:16:57.600 align:middle line:84% It's often a lot easier to sample. 00:16:57.600 --> 00:17:00.060 align:middle line:84% And therefore, these samples are often 00:17:00.060 --> 00:17:03.300 align:middle line:84% called unbiased estimates, because on average, they're 00:17:03.300 --> 00:17:09.417 align:middle line:84% giving you the right thing than to compute analytically. 00:17:09.417 --> 00:17:14.780 align:middle line:90% 00:17:14.780 --> 00:17:15.710 align:middle line:90% Exactly compute. 00:17:15.710 --> 00:17:21.890 align:middle line:90% 00:17:21.890 --> 00:17:22.430 align:middle line:90% Yeah. 00:17:22.430 --> 00:17:26.760 align:middle line:84% So then maybe I'll also give some more concrete examples. 00:17:26.760 --> 00:17:32.420 align:middle line:84% So one example is in deep learning, in ML, 00:17:32.420 --> 00:17:35.480 align:middle line:84% there is this very popular architecture 00:17:35.480 --> 00:17:40.640 align:middle line:84% called the variational autoencoder, or VAE. 00:17:40.640 --> 00:17:44.360 align:middle line:84% And this architecture has randomness baked 00:17:44.360 --> 00:17:45.290 align:middle line:90% into its model. 00:17:45.290 --> 00:17:49.550 align:middle line:84% It's not just randomness from selecting a subset of examples 00:17:49.550 --> 00:17:52.130 align:middle line:84% to train on, because it's intrinsically random. 00:17:52.130 --> 00:18:00.410 align:middle line:84% So the loss function of this VAE would again be defined strictly 00:18:00.410 --> 00:18:03.110 align:middle line:90% speaking as an expectation. 00:18:03.110 --> 00:18:09.140 align:middle line:84% But computationally, the model gives you the ability to sample 00:18:09.140 --> 00:18:12.510 align:middle line:84% from this X of P. That's what super easy to do, 00:18:12.510 --> 00:18:14.680 align:middle line:84% whereas computing this analytically is hard. 00:18:14.680 --> 00:18:17.295 align:middle line:84% So the question is, if you have a process-- 00:18:17.295 --> 00:18:19.170 align:middle line:84% if you have a program from which you can only 00:18:19.170 --> 00:18:21.990 align:middle line:84% get stochastic estimates of this, 00:18:21.990 --> 00:18:28.850 align:middle line:84% we might be interested in dL of P by dP, 00:18:28.850 --> 00:18:30.920 align:middle line:84% but how are we going to get that? 00:18:30.920 --> 00:18:32.690 align:middle line:84% Because we can't-- we're no longer-- 00:18:32.690 --> 00:18:35.880 align:middle line:84% we're not going directly from here to here. 00:18:35.880 --> 00:18:42.040 align:middle line:84% Somehow, we have to start here at X of P and get dL by dP. 00:18:42.040 --> 00:18:44.830 align:middle line:90% So this is one example. 00:18:44.830 --> 00:18:50.353 align:middle line:84% Let me give you another example, which to me, is 00:18:50.353 --> 00:18:51.520 align:middle line:90% the most compelling example. 00:18:51.520 --> 00:18:53.890 align:middle line:84% Now, it's sort of already set, which 00:18:53.890 --> 00:19:00.790 align:middle line:84% is in the physical sciences, your model 00:19:00.790 --> 00:19:03.310 align:middle line:90% may be inherently stochastic. 00:19:03.310 --> 00:19:05.620 align:middle line:90% Your model is stochastic. 00:19:05.620 --> 00:19:12.250 align:middle line:84% For example, you might have two molecules or two particles, 00:19:12.250 --> 00:19:17.140 align:middle line:84% and they're interacting with a rate r. 00:19:17.140 --> 00:19:18.920 align:middle line:84% They're binding or something that. 00:19:18.920 --> 00:19:22.210 align:middle line:84% But this is the average rate of interaction. 00:19:22.210 --> 00:19:27.130 align:middle line:84% Whereas in reality, if you let these molecules roam around, 00:19:27.130 --> 00:19:30.070 align:middle line:84% they're going to interact-- the times at which they're 00:19:30.070 --> 00:19:34.060 align:middle line:84% going to interact is going to be a stochastic process in time. 00:19:34.060 --> 00:19:35.500 align:middle line:84% For example, in this case, if you 00:19:35.500 --> 00:19:40.120 align:middle line:84% have a rate r of interaction, then 00:19:40.120 --> 00:19:41.500 align:middle line:84% the times at which they interact, 00:19:41.500 --> 00:19:45.355 align:middle line:90% that's called a Poisson process. 00:19:45.355 --> 00:19:48.310 align:middle line:84% And if you say, how long am I going to wait until they first 00:19:48.310 --> 00:19:52.540 align:middle line:84% interact, that's actually distributed 00:19:52.540 --> 00:19:55.780 align:middle line:84% as an exponential distribution with rate-- with scale r, 00:19:55.780 --> 00:19:58.240 align:middle line:90% or maybe 1 over r. 00:19:58.240 --> 00:20:00.380 align:middle line:90% I'll have to think about that. 00:20:00.380 --> 00:20:05.540 align:middle line:84% It should be 1 over r rates, because the higher the rate is, 00:20:05.540 --> 00:20:06.590 align:middle line:90% smaller-- 00:20:06.590 --> 00:20:09.050 align:middle line:84% the less time you'd expect to wait. 00:20:09.050 --> 00:20:12.538 align:middle line:84% But the main point here is that your model of the system-- 00:20:12.538 --> 00:20:14.330 align:middle line:84% you're interested in some physical process, 00:20:14.330 --> 00:20:16.760 align:middle line:84% and your model of it is inherently stochastic. 00:20:16.760 --> 00:20:20.270 align:middle line:84% So here, there's no getting around the fact 00:20:20.270 --> 00:20:22.340 align:middle line:84% that you have to deal with stochasticity. 00:20:22.340 --> 00:20:24.740 align:middle line:84% You can't just switch to a different model that 00:20:24.740 --> 00:20:27.860 align:middle line:84% doesn't have stochasticity, because then you're 00:20:27.860 --> 00:20:29.760 align:middle line:90% tied to this physical process. 00:20:29.760 --> 00:20:33.215 align:middle line:84% So this is another case where you have 00:20:33.215 --> 00:20:34.340 align:middle line:90% to deal with stochasticity. 00:20:34.340 --> 00:20:36.830 align:middle line:84% And now if you want to-- let's say 00:20:36.830 --> 00:20:38.970 align:middle line:84% you have some model of what's going on, 00:20:38.970 --> 00:20:40.460 align:middle line:84% but you don't know your parameters. 00:20:40.460 --> 00:20:42.380 align:middle line:90% For example, you don't know r. 00:20:42.380 --> 00:20:46.900 align:middle line:84% Then you want to fit your parameters to data. 00:20:46.900 --> 00:20:50.040 align:middle line:84% And this is, again, some sort of optimization or inference 00:20:50.040 --> 00:20:51.280 align:middle line:90% problem. 00:20:51.280 --> 00:20:53.040 align:middle line:84% So you have some real world data and you 00:20:53.040 --> 00:20:55.050 align:middle line:84% want to optimize the parameters of your model 00:20:55.050 --> 00:20:56.560 align:middle line:90% to best fit the data. 00:20:56.560 --> 00:20:58.350 align:middle line:84% This is some optimization problem, 00:20:58.350 --> 00:21:00.660 align:middle line:84% and again, we're dealing with the same thing, which 00:21:00.660 --> 00:21:03.480 align:middle line:84% is have a system where it's very easy to simulate it 00:21:03.480 --> 00:21:05.700 align:middle line:84% in a stochastic way, but maybe it's 00:21:05.700 --> 00:21:09.390 align:middle line:84% really hard to exactly compute the statistical quantities 00:21:09.390 --> 00:21:10.300 align:middle line:90% for the system. 00:21:10.300 --> 00:21:13.500 align:middle line:84% So again, you want to solve the same problem, which 00:21:13.500 --> 00:21:21.720 align:middle line:84% is that we're interested in the derivative of the expectation 00:21:21.720 --> 00:21:25.670 align:middle line:84% of your program with respect to P. OK, 00:21:25.670 --> 00:21:29.600 align:middle line:84% so our derivative is not-- this doesn't mean-- this isn't 00:21:29.600 --> 00:21:33.140 align:middle line:84% our answer to the derivative, but it's something 00:21:33.140 --> 00:21:36.365 align:middle line:84% we'd like our derivative to help us compute, 00:21:36.365 --> 00:21:38.570 align:middle line:84% like if we're happy with many different notions 00:21:38.570 --> 00:21:39.320 align:middle line:90% of the derivative. 00:21:39.320 --> 00:21:42.620 align:middle line:84% But at least let me compute this using the notion 00:21:42.620 --> 00:21:44.360 align:middle line:90% of using the derivative. 00:21:44.360 --> 00:21:48.470 align:middle line:84% And it's also a bit ambitious to hope to get this exactly 00:21:48.470 --> 00:21:50.270 align:middle line:84% analytically, because we couldn't even 00:21:50.270 --> 00:21:53.000 align:middle line:84% get the original expectation analytically. 00:21:53.000 --> 00:21:58.530 align:middle line:84% So a more reasonable goal might be-- 00:21:58.530 --> 00:22:00.710 align:middle line:84% so this is getting into automatic differentiation. 00:22:00.710 --> 00:22:02.810 align:middle line:84% It's like you're given a program X of P, 00:22:02.810 --> 00:22:05.300 align:middle line:84% and your goal is to produce a new program, which 00:22:05.300 --> 00:22:15.220 align:middle line:84% I'll call X tilde of P, which averages to the derivative of-- 00:22:15.220 --> 00:22:15.940 align:middle line:90% sorry. 00:22:15.940 --> 00:22:18.160 align:middle line:90% My screen turned off. 00:22:18.160 --> 00:22:20.830 align:middle line:84% Which averages to the derivative of the average 00:22:20.830 --> 00:22:23.680 align:middle line:84% of your original program, which is a bit of a mouthful. 00:22:23.680 --> 00:22:26.230 align:middle line:84% So in machine learning, this is often 00:22:26.230 --> 00:22:31.060 align:middle line:84% called a gradient estimator that you're trying to find. 00:22:31.060 --> 00:22:35.680 align:middle line:84% And this isn't necessarily-- even X tilde of P, 00:22:35.680 --> 00:22:37.570 align:middle line:84% this may not be our notion of derivative 00:22:37.570 --> 00:22:38.770 align:middle line:90% that we're looking for. 00:22:38.770 --> 00:22:41.770 align:middle line:84% But it's certainly something we'd 00:22:41.770 --> 00:22:44.440 align:middle line:84% like to be able to construct from whatever notion 00:22:44.440 --> 00:22:47.270 align:middle line:90% of derivative we come up with. 00:22:47.270 --> 00:22:49.040 align:middle line:84% So this is the motivation for why 00:22:49.040 --> 00:22:51.440 align:middle line:84% we might want to characterize the sensitivity 00:22:51.440 --> 00:22:53.795 align:middle line:84% of a random program, why we might want a derivative. 00:22:53.795 --> 00:22:55.920 align:middle line:84% PROFESSOR: Is everyone clear on this at this point? 00:22:55.920 --> 00:22:57.503 align:middle line:84% So this-- you're going to try and find 00:22:57.503 --> 00:23:00.230 align:middle line:84% a random program whose average is 00:23:00.230 --> 00:23:02.930 align:middle line:84% the derivative of the average of the original program. 00:23:02.930 --> 00:23:03.840 align:middle line:90% All right? 00:23:03.840 --> 00:23:05.870 align:middle line:84% So it's kind of a little mouthful. 00:23:05.870 --> 00:23:08.492 align:middle line:90% 00:23:08.492 --> 00:23:09.950 align:middle line:84% OK, people are nodding their heads. 00:23:09.950 --> 00:23:10.450 align:middle line:90% So-- 00:23:10.450 --> 00:23:13.373 align:middle line:90% 00:23:13.373 --> 00:23:14.040 align:middle line:90% GAURAV ARYA: OK. 00:23:14.040 --> 00:23:17.370 align:middle line:84% So that is the why we want to do this. 00:23:17.370 --> 00:23:19.320 align:middle line:90% And now the question is, how? 00:23:19.320 --> 00:23:21.690 align:middle line:84% What notion of derivative should we 00:23:21.690 --> 00:23:23.770 align:middle line:90% come up with to let us do this? 00:23:23.770 --> 00:23:26.160 align:middle line:90% So yeah. 00:23:26.160 --> 00:23:31.110 align:middle line:84% So once again, the functions we're interested in look this. 00:23:31.110 --> 00:23:37.860 align:middle line:84% P maps to X of P. So now let's think back to the beginning 00:23:37.860 --> 00:23:39.730 align:middle line:84% where we had these two questions. 00:23:39.730 --> 00:23:42.510 align:middle line:84% One was, just think about the differential, 00:23:42.510 --> 00:23:45.630 align:middle line:84% or just how the output is changing with respect 00:23:45.630 --> 00:23:49.590 align:middle line:84% to the input, without worrying about turning it 00:23:49.590 --> 00:23:51.780 align:middle line:84% into a derivative that cuts out higher order things. 00:23:51.780 --> 00:23:53.880 align:middle line:90% Just, how does the program-- 00:23:53.880 --> 00:23:56.110 align:middle line:84% how do perturbations of the program look like? 00:23:56.110 --> 00:23:58.020 align:middle line:84% So let's try and answer that first. 00:23:58.020 --> 00:23:59.430 align:middle line:90% Let's answer question 1 first. 00:23:59.430 --> 00:24:02.040 align:middle line:90% 00:24:02.040 --> 00:24:04.290 align:middle line:84% So the first thing I'm going to write is-- 00:24:04.290 --> 00:24:07.530 align:middle line:90% 00:24:07.530 --> 00:24:15.630 align:middle line:84% just try to naively characterize the change in my program. 00:24:15.630 --> 00:24:20.690 align:middle line:84% So here, I'm using slightly different notation from before, 00:24:20.690 --> 00:24:23.810 align:middle line:84% and maybe Professor Johnson or Professor Edelman 00:24:23.810 --> 00:24:25.100 align:middle line:90% might object to this. 00:24:25.100 --> 00:24:29.195 align:middle line:84% But this is what I'm calling the differential 00:24:29.195 --> 00:24:31.100 align:middle line:90% for a stochastic case. 00:24:31.100 --> 00:24:36.140 align:middle line:84% This is a stochastic differential, 00:24:36.140 --> 00:24:38.150 align:middle line:90% and I'm writing it as-- 00:24:38.150 --> 00:24:40.670 align:middle line:84% So basically, epsilon is your dP. 00:24:40.670 --> 00:24:42.720 align:middle line:84% So this right-hand side should make sense. 00:24:42.720 --> 00:24:45.730 align:middle line:84% It's basically X of P plus dP minus X of P. 00:24:45.730 --> 00:24:47.600 align:middle line:84% One thing I've done here is I'm not 00:24:47.600 --> 00:24:49.990 align:middle line:84% super comfortable with dealing with differentials 00:24:49.990 --> 00:24:53.360 align:middle line:84% as infinitesimal things, so I've written this left-hand side 00:24:53.360 --> 00:24:54.710 align:middle line:90% as a function. 00:24:54.710 --> 00:25:00.230 align:middle line:84% dX of epsilon equals X of P plus epsilon minus X of P. 00:25:00.230 --> 00:25:03.860 align:middle line:84% So this is just exactly what it says. 00:25:03.860 --> 00:25:08.630 align:middle line:84% There is no dropping of higher order terms here. 00:25:08.630 --> 00:25:10.730 align:middle line:90% And epsilon, I'm thinking as-- 00:25:10.730 --> 00:25:12.120 align:middle line:90% so remember P was real. 00:25:12.120 --> 00:25:14.860 align:middle line:84% So epsilon, I'm thinking, is some small real number. 00:25:14.860 --> 00:25:19.510 align:middle line:84% So this is basically just, for a finite perturbation 00:25:19.510 --> 00:25:21.472 align:middle line:84% in the input, how does the output perturb? 00:25:21.472 --> 00:25:22.680 align:middle line:90% PROFESSOR: Yeah, so this is-- 00:25:22.680 --> 00:25:23.190 align:middle line:84% GAURAV ARYA: Although I might not 00:25:23.190 --> 00:25:25.060 align:middle line:84% have made it totally clear what this means yet. 00:25:25.060 --> 00:25:25.710 align:middle line:90% That'll come next. 00:25:25.710 --> 00:25:25.890 align:middle line:90% PROFESSOR: Yeah. 00:25:25.890 --> 00:25:26.520 align:middle line:90% So in lecture-- 00:25:26.520 --> 00:25:27.450 align:middle line:84% GAURAV ARYA: But I'm interested if-- 00:25:27.450 --> 00:25:29.580 align:middle line:84% PROFESSOR: In lecture 1, this was what we called-- 00:25:29.580 --> 00:25:32.963 align:middle line:84% in lecture 1, this is what we called delta X. So delta X-- 00:25:32.963 --> 00:25:33.880 align:middle line:90% GAURAV ARYA: Yeah, OK. 00:25:33.880 --> 00:25:35.340 align:middle line:84% PROFESSOR: --was the nondifferential change. 00:25:35.340 --> 00:25:36.007 align:middle line:90% But that's fine. 00:25:36.007 --> 00:25:38.160 align:middle line:84% You can stick with your favorite notation. 00:25:38.160 --> 00:25:40.760 align:middle line:84% Don't change notations midstream. 00:25:40.760 --> 00:25:41.760 align:middle line:90% GAURAV ARYA: Sure, yeah. 00:25:41.760 --> 00:25:43.050 align:middle line:90% So yeah. 00:25:43.050 --> 00:25:44.850 align:middle line:90% But this is-- so this idea-- 00:25:44.850 --> 00:25:48.420 align:middle line:84% but then, I guess, now you have to ask, what sort of object 00:25:48.420 --> 00:25:50.920 align:middle line:90% is this dX of epsilon? 00:25:50.920 --> 00:25:54.450 align:middle line:84% So maybe this is also something I can ask the class. 00:25:54.450 --> 00:25:56.830 align:middle line:84% X of P was a random variable and so forth, 00:25:56.830 --> 00:25:58.515 align:middle line:90% so what is the dX of epsilon? 00:25:58.515 --> 00:26:05.574 align:middle line:90% 00:26:05.574 --> 00:26:06.700 align:middle line:90% PROFESSOR: So what is it? 00:26:06.700 --> 00:26:07.200 align:middle line:90% Yeah? 00:26:07.200 --> 00:26:08.367 align:middle line:90% AUDIENCE: A random variable. 00:26:08.367 --> 00:26:10.760 align:middle line:84% PROFESSOR: It's a random variable is the answer. 00:26:10.760 --> 00:26:11.760 align:middle line:90% GAURAV ARYA: Yeah, yeah. 00:26:11.760 --> 00:26:14.220 align:middle line:84% Generally, if you subtract two random variables, 00:26:14.220 --> 00:26:16.870 align:middle line:84% what you're going to get is another random variable. 00:26:16.870 --> 00:26:20.460 align:middle line:84% However, I'd argue that I haven't really fully specified 00:26:20.460 --> 00:26:22.107 align:middle line:90% to you what dX epsilon is. 00:26:22.107 --> 00:26:23.940 align:middle line:84% You're right it should be a random variable, 00:26:23.940 --> 00:26:25.530 align:middle line:84% but I think it could be a whole lot 00:26:25.530 --> 00:26:28.830 align:middle line:84% of random variables dependent on something I haven't really 00:26:28.830 --> 00:26:31.110 align:middle line:90% told you. 00:26:31.110 --> 00:26:33.713 align:middle line:90% Does anyone see why-- 00:26:33.713 --> 00:26:35.880 align:middle line:84% is it totally unambiguous, or-- let's think of one-- 00:26:35.880 --> 00:26:37.480 align:middle line:84% let's think of a concrete example. 00:26:37.480 --> 00:26:39.210 align:middle line:84% Let's think of the case where X of P 00:26:39.210 --> 00:26:42.510 align:middle line:84% is distributed as the exponential distribution 00:26:42.510 --> 00:26:49.330 align:middle line:84% with scale P. For this case, is it 00:26:49.330 --> 00:26:53.722 align:middle line:84% unambiguous what dX of epsilon is, or is it unclear? 00:26:53.722 --> 00:26:55.930 align:middle line:84% PROFESSOR: So what's-- is something missing from this 00:26:55.930 --> 00:26:58.450 align:middle line:90% definition? 00:26:58.450 --> 00:26:58.950 align:middle line:90% Yeah? 00:26:58.950 --> 00:26:59.992 align:middle line:90% Someone's raising-- yeah? 00:26:59.992 --> 00:27:03.060 align:middle line:84% AUDIENCE: I guess by "change," do you mean a CDF change? 00:27:03.060 --> 00:27:04.650 align:middle line:84% Or, like-- I'm not really sure what 00:27:04.650 --> 00:27:06.525 align:middle line:84% you mean by "subtracting random variables." 00:27:06.525 --> 00:27:07.350 align:middle line:90% PROFESSOR: Yeah. 00:27:07.350 --> 00:27:08.640 align:middle line:90% So the question is-- 00:27:08.640 --> 00:27:10.658 align:middle line:84% the comment was he wasn't sure what 00:27:10.658 --> 00:27:12.450 align:middle line:84% you mean by "subtracting random variables." 00:27:12.450 --> 00:27:13.890 align:middle line:90% Do you mean the CDF change? 00:27:13.890 --> 00:27:15.520 align:middle line:90% Or what is this minus? 00:27:15.520 --> 00:27:17.020 align:middle line:84% GAURAV ARYA: That's a good question. 00:27:17.020 --> 00:27:20.790 align:middle line:90% So what I mean is you-- 00:27:20.790 --> 00:27:23.670 align:middle line:84% so a random variable is basically telling you 00:27:23.670 --> 00:27:24.990 align:middle line:90% how to sample from something. 00:27:24.990 --> 00:27:27.210 align:middle line:90% A random variable says how you-- 00:27:27.210 --> 00:27:29.800 align:middle line:84% it gives you the procedure for sampling from a distribution. 00:27:29.800 --> 00:27:31.547 align:middle line:90% So I'm saying, take a sample-- 00:27:31.547 --> 00:27:33.880 align:middle line:84% I'm going to construct a new random variable which says, 00:27:33.880 --> 00:27:35.520 align:middle line:90% how do you sample from dX? 00:27:35.520 --> 00:27:37.680 align:middle line:84% You sample from X of P plus epsilon, 00:27:37.680 --> 00:27:42.120 align:middle line:84% you sample from X of P, and you subtract so that is-- 00:27:42.120 --> 00:27:42.620 align:middle line:90% yeah. 00:27:42.620 --> 00:27:43.810 align:middle line:84% PROFESSOR: It's a sampling procedure. 00:27:43.810 --> 00:27:44.190 align:middle line:90% OK. 00:27:44.190 --> 00:27:46.010 align:middle line:84% GAURAV ARYA: So it's very different from doing operations 00:27:46.010 --> 00:27:46.820 align:middle line:90% with distributions. 00:27:46.820 --> 00:27:48.568 align:middle line:84% If you were to look at this-- if you're 00:27:48.568 --> 00:27:50.360 align:middle line:84% trying to figure out the distribution of dX 00:27:50.360 --> 00:27:52.813 align:middle line:84% in terms of the distribution of these two, 00:27:52.813 --> 00:27:54.230 align:middle line:84% when you add two random variables, 00:27:54.230 --> 00:27:55.970 align:middle line:90% the distribution convolves. 00:27:55.970 --> 00:27:57.800 align:middle line:90% You convolve the distribution. 00:27:57.800 --> 00:27:59.450 align:middle line:84% But if you think sample-wise, then it's 00:27:59.450 --> 00:28:00.680 align:middle line:90% just a simple subtraction. 00:28:00.680 --> 00:28:03.380 align:middle line:90% 00:28:03.380 --> 00:28:04.100 align:middle line:90% Yeah. 00:28:04.100 --> 00:28:07.590 align:middle line:84% But why might this still not be fully specified? 00:28:07.590 --> 00:28:11.910 align:middle line:84% PROFESSOR: Any other ideas what might be missing? 00:28:11.910 --> 00:28:12.410 align:middle line:90% Yeah? 00:28:12.410 --> 00:28:14.840 align:middle line:90% AUDIENCE: Is P Fixed, or-- 00:28:14.840 --> 00:28:16.190 align:middle line:90% PROFESSOR: Is P fixed? 00:28:16.190 --> 00:28:17.120 align:middle line:90% I guess, is that-- 00:28:17.120 --> 00:28:17.630 align:middle line:90% GAURAV ARYA: P is fixed. 00:28:17.630 --> 00:28:18.640 align:middle line:84% Actually, I should have said that. 00:28:18.640 --> 00:28:19.800 align:middle line:90% So that's a good point. 00:28:19.800 --> 00:28:21.710 align:middle line:84% So in some sense, there's a dependence on P 00:28:21.710 --> 00:28:23.960 align:middle line:84% here that I'm sort of suppressing. 00:28:23.960 --> 00:28:25.550 align:middle line:84% Imagining you consider a fixed P, 00:28:25.550 --> 00:28:28.280 align:middle line:84% and now you're thinking about a neighborhood of P. But yeah. 00:28:28.280 --> 00:28:30.050 align:middle line:90% So that is true. 00:28:30.050 --> 00:28:32.660 align:middle line:90% Yeah. 00:28:32.660 --> 00:28:35.350 align:middle line:90% PROFESSOR: Any other ideas? 00:28:35.350 --> 00:28:35.970 align:middle line:90% No? 00:28:35.970 --> 00:28:37.523 align:middle line:90% We're drawing a blank, Gaurav. 00:28:37.523 --> 00:28:38.190 align:middle line:90% GAURAV ARYA: OK. 00:28:38.190 --> 00:28:44.670 align:middle line:84% So we know what's so-called the marginal distributions of each 00:28:44.670 --> 00:28:45.910 align:middle line:90% of these random variables. 00:28:45.910 --> 00:28:47.670 align:middle line:84% We know how to sample from each of these. 00:28:47.670 --> 00:28:54.680 align:middle line:84% But there's a question of what their joint distribution is, 00:28:54.680 --> 00:28:57.320 align:middle line:90% which means-- 00:28:57.320 --> 00:28:59.630 align:middle line:90% for example, if you have-- 00:28:59.630 --> 00:29:04.985 align:middle line:84% Suppose you were considering X of P minus X of P for a second. 00:29:04.985 --> 00:29:08.030 align:middle line:90% 00:29:08.030 --> 00:29:11.030 align:middle line:84% Well, maybe that's a bit too confusing. 00:29:11.030 --> 00:29:14.450 align:middle line:84% Let's say we're considering-- so let me go to a new page. 00:29:14.450 --> 00:29:17.180 align:middle line:84% Let's go away from that context for a second 00:29:17.180 --> 00:29:20.490 align:middle line:84% and let's consider subtracting two random variables, A and B, 00:29:20.490 --> 00:29:23.640 align:middle line:84% where let's say A and B follow the same distribution. 00:29:23.640 --> 00:29:26.720 align:middle line:84% So A and B follow the same distribution. 00:29:26.720 --> 00:29:30.050 align:middle line:84% Let's call it some distribution d. 00:29:30.050 --> 00:29:32.850 align:middle line:90% 00:29:32.850 --> 00:29:35.690 align:middle line:90% Is this going to equal 0? 00:29:35.690 --> 00:29:44.696 align:middle line:90% Like-- If the answers are yes-- 00:29:44.696 --> 00:29:47.660 align:middle line:84% PROFESSOR: So if A is a-- if A and B are 00:29:47.660 --> 00:29:52.040 align:middle line:84% both coin flips with probability 1/2 half they're 1, 00:29:52.040 --> 00:29:56.390 align:middle line:84% and probability 1 they're 0, is the difference 0? 00:29:56.390 --> 00:29:58.490 align:middle line:84% People are shaking their heads no. 00:29:58.490 --> 00:29:59.240 align:middle line:90% GAURAV ARYA: Yeah. 00:29:59.240 --> 00:30:01.310 align:middle line:90% So it actually depends. 00:30:01.310 --> 00:30:03.960 align:middle line:84% Because if A and B are independent, 00:30:03.960 --> 00:30:06.590 align:middle line:84% like you would imagine for two coin flips, 00:30:06.590 --> 00:30:08.450 align:middle line:90% then it would not always be 0. 00:30:08.450 --> 00:30:11.060 align:middle line:84% If on the other hand, A and B could be the same thing, 00:30:11.060 --> 00:30:14.450 align:middle line:84% they could literally equal A, in which case, yes, 00:30:14.450 --> 00:30:15.680 align:middle line:90% this would be 0. 00:30:15.680 --> 00:30:20.180 align:middle line:84% So this is the idea of a joint distribution. 00:30:20.180 --> 00:30:22.970 align:middle line:84% Even if you know how to sample from a random variable 00:30:22.970 --> 00:30:27.980 align:middle line:84% marginally, you can figure out, in the whole probability space, 00:30:27.980 --> 00:30:30.650 align:middle line:84% do they-- how do they behave together as they-- 00:30:30.650 --> 00:30:33.230 align:middle line:84% so in some senses, what we haven't specified 00:30:33.230 --> 00:30:37.663 align:middle line:84% is a distribution of A comma B. Does that make sense? 00:30:37.663 --> 00:30:38.330 align:middle line:90% PROFESSOR: Yeah. 00:30:38.330 --> 00:30:40.550 align:middle line:84% So are they the same coin flip, or are they 00:30:40.550 --> 00:30:43.110 align:middle line:84% independent coin flips, or something in between? 00:30:43.110 --> 00:30:44.030 align:middle line:90% Yeah. 00:30:44.030 --> 00:30:47.030 align:middle line:84% People are shaking their heads yes, so I think-- 00:30:47.030 --> 00:30:47.840 align:middle line:90% GAURAV ARYA: OK. 00:30:47.840 --> 00:30:52.520 align:middle line:84% So let's now go back to our particular case 00:30:52.520 --> 00:30:54.770 align:middle line:84% of subtracting two random variables where 00:30:54.770 --> 00:31:05.430 align:middle line:84% we had X of P plus epsilon minus X of P. 00:31:05.430 --> 00:31:09.180 align:middle line:84% And my claim is it's not enough to say that this thing is 00:31:09.180 --> 00:31:12.840 align:middle line:84% distributed as an exponential distribution with rate P 00:31:12.840 --> 00:31:16.680 align:middle line:84% plus epsilon, and this thing is distributed 00:31:16.680 --> 00:31:19.320 align:middle line:84% as an exponential distribution with rate P. This 00:31:19.320 --> 00:31:20.380 align:middle line:90% doesn't tell you enough. 00:31:20.380 --> 00:31:24.330 align:middle line:84% You have to specify how they behave jointly. 00:31:24.330 --> 00:31:28.440 align:middle line:84% And perhaps the most simple thing to do 00:31:28.440 --> 00:31:30.840 align:middle line:84% is to say, why don't we make these two 00:31:30.840 --> 00:31:32.890 align:middle line:90% random variables independent? 00:31:32.890 --> 00:31:36.270 align:middle line:84% So basically, it means these sampling processes are not 00:31:36.270 --> 00:31:37.800 align:middle line:90% tied to each other at all. 00:31:37.800 --> 00:31:40.120 align:middle line:84% We're going to take a sample from a sample from X 00:31:40.120 --> 00:31:45.340 align:middle line:84% of P, a sample of X of P plus epsilon, and subtract them. 00:31:45.340 --> 00:31:49.060 align:middle line:84% So let's actually see how that works in code. 00:31:49.060 --> 00:31:52.540 align:middle line:90% 00:31:52.540 --> 00:31:53.170 align:middle line:90% So yeah. 00:31:53.170 --> 00:31:56.710 align:middle line:84% So my sampleX is already set up as an exponential random 00:31:56.710 --> 00:32:01.960 align:middle line:84% variable, and now I'm going to sample_dx, 00:32:01.960 --> 00:32:04.720 align:middle line:90% which takes as input epsilon. 00:32:04.720 --> 00:32:05.845 align:middle line:90% This is a little confusing. 00:32:05.845 --> 00:32:07.780 align:middle line:90% Let me put that. 00:32:07.780 --> 00:32:10.790 align:middle line:90% And I'm going to fix p at 100. 00:32:10.790 --> 00:32:15.730 align:middle line:84% So then this looks like sampleX of 100 plus 00:32:15.730 --> 00:32:20.020 align:middle line:90% epsilon minus sampleX at 100. 00:32:20.020 --> 00:32:25.200 align:middle line:84% And now let's try sampling from dX just to get a feel. 00:32:25.200 --> 00:32:27.870 align:middle line:90% So you can sample dX. 00:32:27.870 --> 00:32:31.110 align:middle line:84% Maybe let's assume a small perturbation, like 0.1. 00:32:31.110 --> 00:32:35.460 align:middle line:84% And we can see it looks like a random variable, as was said. 00:32:35.460 --> 00:32:37.170 align:middle line:84% And it takes fairly large values. 00:32:37.170 --> 00:32:41.680 align:middle line:90% Here's a negative 145. 00:32:41.680 --> 00:32:44.930 align:middle line:84% Maybe let's make epsilon really, really tiny. 00:32:44.930 --> 00:32:46.690 align:middle line:84% It still takes fairly large values. 00:32:46.690 --> 00:32:48.400 align:middle line:90% It still looks pretty large. 00:32:48.400 --> 00:32:50.710 align:middle line:84% And at this point, maybe alarm bells 00:32:50.710 --> 00:32:53.500 align:middle line:84% should be ringing because in a derivative, 00:32:53.500 --> 00:32:55.510 align:middle line:84% you sort of have this idea that if you perturb 00:32:55.510 --> 00:32:57.820 align:middle line:84% the input by a smaller and smaller amount, 00:32:57.820 --> 00:33:02.380 align:middle line:84% the output, in some sense of size, some sense of norm, 00:33:02.380 --> 00:33:05.590 align:middle line:84% should get smaller and smaller and smaller. 00:33:05.590 --> 00:33:09.310 align:middle line:84% Whereas here, this isn't happening at all. 00:33:09.310 --> 00:33:12.760 align:middle line:84% So if you were to try and do something like divide 00:33:12.760 --> 00:33:17.040 align:middle line:84% by your step size, as you often try to do with derivatives, 00:33:17.040 --> 00:33:20.550 align:middle line:84% this looks concerning because, sure, this might 00:33:20.550 --> 00:33:22.110 align:middle line:90% average to the right thing. 00:33:22.110 --> 00:33:24.357 align:middle line:84% But its variance is going to be really large 00:33:24.357 --> 00:33:26.940 align:middle line:84% because the right thing might be something like, I don't know, 00:33:26.940 --> 00:33:27.600 align:middle line:90% 1. 00:33:27.600 --> 00:33:30.900 align:middle line:84% And we're getting these numbers of order 10 00:33:30.900 --> 00:33:33.597 align:middle line:84% to the 7, which, if you trust the math, 00:33:33.597 --> 00:33:34.680 align:middle line:90% are going to average to 1. 00:33:34.680 --> 00:33:36.847 align:middle line:84% But if you actually want to do this computationally, 00:33:36.847 --> 00:33:38.160 align:middle line:90% you're in trouble. 00:33:38.160 --> 00:33:43.790 align:middle line:84% So in making all of these, making these independent 00:33:43.790 --> 00:33:46.660 align:middle line:90% is not a good idea. 00:33:46.660 --> 00:33:47.890 align:middle line:90% Are people convinced by this? 00:33:47.890 --> 00:33:48.910 align:middle line:90% Are there any questions? 00:33:48.910 --> 00:33:52.450 align:middle line:90% 00:33:52.450 --> 00:33:56.370 align:middle line:84% PROFESSOR: So I think people are shaking their heads OK. 00:33:56.370 --> 00:33:58.970 align:middle line:90% GAURAV ARYA: OK. 00:33:58.970 --> 00:33:59.570 align:middle line:90% All right. 00:33:59.570 --> 00:34:04.440 align:middle line:84% So what could be a better approach to this? 00:34:04.440 --> 00:34:07.100 align:middle line:90% Well, here we have to-- 00:34:07.100 --> 00:34:09.949 align:middle line:84% so let's again stick to our example. 00:34:09.949 --> 00:34:12.860 align:middle line:84% And all these papers, I'm going to fill up all their nooks 00:34:12.860 --> 00:34:15.860 align:middle line:90% and crannies later. 00:34:15.860 --> 00:34:17.464 align:middle line:84% I feel bad about wasting so much. 00:34:17.464 --> 00:34:20.480 align:middle line:90% 00:34:20.480 --> 00:34:22.320 align:middle line:84% Yeah, so let's go to this example. 00:34:22.320 --> 00:34:25.489 align:middle line:84% And now we have this family of random variables. 00:34:25.489 --> 00:34:29.815 align:middle line:84% And we need to figure out how they work with each other. 00:34:29.815 --> 00:34:31.190 align:middle line:84% How are they distributed jointly? 00:34:31.190 --> 00:34:34.370 align:middle line:90% 00:34:34.370 --> 00:34:38.550 align:middle line:84% And here I have to introduce some more concepts. 00:34:38.550 --> 00:34:42.857 align:middle line:84% So how is a random variable actually defined? 00:34:42.857 --> 00:34:44.690 align:middle line:84% We're trying to differentiate these objects. 00:34:44.690 --> 00:34:47.670 align:middle line:84% Maybe we ought to understand these objects a bit better. 00:34:47.670 --> 00:34:50.510 align:middle line:84% So this is getting into a bit of probability theory. 00:34:50.510 --> 00:34:55.069 align:middle line:84% But basically, there's something called a sample space. 00:34:55.069 --> 00:34:59.190 align:middle line:90% 00:34:59.190 --> 00:35:00.660 align:middle line:90% Omega. 00:35:00.660 --> 00:35:03.510 align:middle line:90% This is a space of things. 00:35:03.510 --> 00:35:17.995 align:middle line:84% And it's equipped with a probability distribution p. 00:35:17.995 --> 00:35:19.620 align:middle line:84% Some other way you can think about this 00:35:19.620 --> 00:35:22.470 align:middle line:84% is there's some random variable with output space omega. 00:35:22.470 --> 00:35:25.760 align:middle line:84% But here I've actually written the distribution. 00:35:25.760 --> 00:35:28.640 align:middle line:90% And then your random variable. 00:35:28.640 --> 00:35:34.850 align:middle line:90% Let's say any x of p is a map. 00:35:34.850 --> 00:35:37.280 align:middle line:84% And also, for simplicity, I said p was real. 00:35:37.280 --> 00:35:40.670 align:middle line:84% Let's also make x of p a real valued random variable, 00:35:40.670 --> 00:35:41.870 align:middle line:90% for simplicity. 00:35:41.870 --> 00:35:50.140 align:middle line:84% In which case, x of p is defined as a map from omega to R. 00:35:50.140 --> 00:35:53.650 align:middle line:84% So this is a bit confusing because this is 00:35:53.650 --> 00:35:55.310 align:middle line:90% like a deterministic function. 00:35:55.310 --> 00:35:57.430 align:middle line:84% So how are we defining random variables 00:35:57.430 --> 00:35:59.660 align:middle line:84% through a deterministic function? 00:35:59.660 --> 00:36:04.130 align:middle line:84% The idea is that all of the randomness is here. 00:36:04.130 --> 00:36:08.520 align:middle line:84% This is a distribution which is independent of p. 00:36:08.520 --> 00:36:10.710 align:middle line:84% This distribution doesn't depend on p. 00:36:10.710 --> 00:36:13.290 align:middle line:84% And then we have a fixed distribution, a fixed source 00:36:13.290 --> 00:36:16.410 align:middle line:84% of randomness, for all of our family, all 00:36:16.410 --> 00:36:18.970 align:middle line:84% of the random variables in our family. 00:36:18.970 --> 00:36:23.380 align:middle line:84% And what differs within our family is what the map is. 00:36:23.380 --> 00:36:26.800 align:middle line:90% This map is parametrized by p. 00:36:26.800 --> 00:36:30.550 align:middle line:84% So x, you can write x of p on the main-- 00:36:30.550 --> 00:36:33.610 align:middle line:84% PROFESSOR: It might help to be concrete here, Gaurav. 00:36:33.610 --> 00:36:35.860 align:middle line:84% So on a computer, there's a function 00:36:35.860 --> 00:36:39.890 align:middle line:84% in Julia called rand that will return a number between 0 00:36:39.890 --> 00:36:42.260 align:middle line:90% and 1 uniformly distributed. 00:36:42.260 --> 00:36:46.330 align:middle line:84% So now suppose you're given that as input, 00:36:46.330 --> 00:36:49.300 align:middle line:84% and then you want to spit out an exponentially distributed 00:36:49.300 --> 00:36:50.150 align:middle line:90% thing as output. 00:36:50.150 --> 00:36:53.330 align:middle line:84% So you only need that source of randomness, 00:36:53.330 --> 00:36:55.510 align:middle line:84% which who knows where that comes from? 00:36:55.510 --> 00:36:57.730 align:middle line:84% And then somehow, it passes through some map 00:36:57.730 --> 00:37:02.410 align:middle line:84% to produce some other distribution, 00:37:02.410 --> 00:37:04.468 align:middle line:90% other type of random number. 00:37:04.468 --> 00:37:05.260 align:middle line:90% GAURAV ARYA: Right. 00:37:05.260 --> 00:37:06.400 align:middle line:90% Yeah, yeah. 00:37:06.400 --> 00:37:08.410 align:middle line:84% So actually, yes, I'll try and show 00:37:08.410 --> 00:37:10.420 align:middle line:84% that for an exponential distribution. 00:37:10.420 --> 00:37:12.040 align:middle line:84% Are there any other questions first? 00:37:12.040 --> 00:37:16.560 align:middle line:90% 00:37:16.560 --> 00:37:20.550 align:middle line:84% PROFESSOR: OK, no other questions right now. 00:37:20.550 --> 00:37:21.882 align:middle line:90% GAURAV ARYA: Yeah. 00:37:21.882 --> 00:37:23.340 align:middle line:84% Yeah, so I'll try and show that for 00:37:23.340 --> 00:37:24.507 align:middle line:90% an exponential distribution. 00:37:24.507 --> 00:37:28.110 align:middle line:84% So again, x of p distributed as an exponential. 00:37:28.110 --> 00:37:32.370 align:middle line:84% And then for simplicity, just for consistency, 00:37:32.370 --> 00:37:34.050 align:middle line:84% this isn't actually the simplest choice 00:37:34.050 --> 00:37:37.380 align:middle line:84% for doing this for an exponential random variable, 00:37:37.380 --> 00:37:41.520 align:middle line:84% but I'm always going to make omega between 0 and 1. 00:37:41.520 --> 00:37:48.970 align:middle line:84% And p is the uniform distribution over 0 and 1. 00:37:48.970 --> 00:37:50.410 align:middle line:90% PROFESSOR: You have a coin flip. 00:37:50.410 --> 00:37:51.160 align:middle line:90% GAURAV ARYA: Yeah. 00:37:51.160 --> 00:37:52.120 align:middle line:90% Exactly. 00:37:52.120 --> 00:37:55.210 align:middle line:84% And sort of going back to this map thing, 00:37:55.210 --> 00:37:58.120 align:middle line:84% maybe I should have said also explicitly 00:37:58.120 --> 00:38:00.100 align:middle line:90% how do you sample from x of p? 00:38:00.100 --> 00:38:03.080 align:middle line:84% And the answer is sort of like a two-step process. 00:38:03.080 --> 00:38:06.880 align:middle line:84% First you pick a random omega, according to p, 00:38:06.880 --> 00:38:08.800 align:middle line:84% and then you plug it into this map. 00:38:08.800 --> 00:38:10.690 align:middle line:84% That's how you sample from x of p. 00:38:10.690 --> 00:38:13.330 align:middle line:84% So here this is like your call to rand. 00:38:13.330 --> 00:38:15.010 align:middle line:90% And this is going to give you-- 00:38:15.010 --> 00:38:17.260 align:middle line:84% if you think of sampling from p, that's 00:38:17.260 --> 00:38:21.880 align:middle line:84% going to give you an omega, which is between 0 and 1. 00:38:21.880 --> 00:38:24.040 align:middle line:84% And then it turns out that the function 00:38:24.040 --> 00:38:27.520 align:middle line:84% for an exponential distribution, think 00:38:27.520 --> 00:38:30.180 align:middle line:90% it's log, maybe negative log. 00:38:30.180 --> 00:38:31.990 align:middle line:90% It should be negative log. 00:38:31.990 --> 00:38:36.100 align:middle line:90% The negative log of a uniform. 00:38:36.100 --> 00:38:38.170 align:middle line:84% And I could also just write omega here. 00:38:38.170 --> 00:38:39.940 align:middle line:90% I'm writing 1 minus omega. 00:38:39.940 --> 00:38:47.980 align:middle line:90% But this is x of p of omega. 00:38:47.980 --> 00:38:54.290 align:middle line:84% So maybe it'll become more clear if I try and show you in code. 00:38:54.290 --> 00:38:55.370 align:middle line:90% PROFESSOR: Questions? 00:38:55.370 --> 00:38:58.430 align:middle line:84% AUDIENCE: Shouldn't that depend on p in a way? 00:38:58.430 --> 00:39:01.850 align:middle line:84% PROFESSOR: The question was, shouldn't that depend on p? 00:39:01.850 --> 00:39:02.760 align:middle line:90% Oh, yeah. 00:39:02.760 --> 00:39:05.000 align:middle line:90% Oh, your p scaling, yes. 00:39:05.000 --> 00:39:06.988 align:middle line:84% Yeah, you didn't put your scale in. 00:39:06.988 --> 00:39:08.030 align:middle line:90% GAURAV ARYA: Yeah, sorry. 00:39:08.030 --> 00:39:09.980 align:middle line:84% That should have been multiplied by p. 00:39:09.980 --> 00:39:10.580 align:middle line:90% Thanks. 00:39:10.580 --> 00:39:11.830 align:middle line:90% Thanks for the question, yeah. 00:39:11.830 --> 00:39:12.930 align:middle line:90% PROFESSOR: Yeah, yeah. 00:39:12.930 --> 00:39:14.180 align:middle line:90% GAURAV ARYA: Yeah, so we add-- 00:39:14.180 --> 00:39:15.930 align:middle line:84% PROFESSOR: So there's a p outside the log. 00:39:15.930 --> 00:39:16.895 align:middle line:90% So that's [INAUDIBLE]. 00:39:16.895 --> 00:39:18.770 align:middle line:84% GAURAV ARYA: Yeah, so if you go back to here, 00:39:18.770 --> 00:39:24.410 align:middle line:84% we had a way of sampling from an exponential distribution. 00:39:24.410 --> 00:39:27.320 align:middle line:84% Now let's write our own sampler, where 00:39:27.320 --> 00:39:33.530 align:middle line:84% we use this trick, where our fundamental primitive is 00:39:33.530 --> 00:39:34.400 align:middle line:90% going to be rand. 00:39:34.400 --> 00:39:38.110 align:middle line:84% So maybe I'll write this in multiple lines. 00:39:38.110 --> 00:39:40.920 align:middle line:84% So first, we sample our omega, which 00:39:40.920 --> 00:39:42.510 align:middle line:90% is a uniform random number. 00:39:42.510 --> 00:39:45.720 align:middle line:84% PROFESSOR: Yeah, so rand is uniform between 0 and 1, 00:39:45.720 --> 00:39:46.410 align:middle line:90% basically. 00:39:46.410 --> 00:39:47.730 align:middle line:90% GAURAV ARYA: Yes, yes. 00:39:47.730 --> 00:39:55.340 align:middle line:84% And then we return log of 1 minus omega. 00:39:55.340 --> 00:39:58.000 align:middle line:90% PROFESSOR: Times p. 00:39:58.000 --> 00:39:59.510 align:middle line:90% GAURAV ARYA: Times p, yes. 00:39:59.510 --> 00:40:04.580 align:middle line:84% So yeah, so now if I try this with 100, 00:40:04.580 --> 00:40:07.203 align:middle line:84% it does seem to look something centered around 100. 00:40:07.203 --> 00:40:08.120 align:middle line:90% We can check its mean. 00:40:08.120 --> 00:40:10.958 align:middle line:90% 00:40:10.958 --> 00:40:12.500 align:middle line:84% So yeah, so, in fact, under the hood, 00:40:12.500 --> 00:40:14.375 align:middle line:84% this is probably what your computer is doing. 00:40:14.375 --> 00:40:16.933 align:middle line:84% There's a really optimized way of generating uniforms. 00:40:16.933 --> 00:40:19.100 align:middle line:84% Actually I'm not 100% sure that's how it's doing it, 00:40:19.100 --> 00:40:21.560 align:middle line:90% but this is one possible way. 00:40:21.560 --> 00:40:28.820 align:middle line:84% And why is this relevant to our differentiation problem? 00:40:28.820 --> 00:40:35.430 align:middle line:84% It's relevant because if we go back to this phrasing, 00:40:35.430 --> 00:40:44.390 align:middle line:84% the key part is that this call to rand is independent of p. 00:40:44.390 --> 00:40:48.740 align:middle line:84% So what we can do is we can not make 00:40:48.740 --> 00:40:50.900 align:middle line:84% all of these x of p's independent, 00:40:50.900 --> 00:40:56.000 align:middle line:84% but rather make them all rely on exactly the same call to rand, 00:40:56.000 --> 00:40:58.415 align:middle line:84% which we just plug into their respective maps. 00:40:58.415 --> 00:41:01.710 align:middle line:90% 00:41:01.710 --> 00:41:02.620 align:middle line:90% Does that make sense? 00:41:02.620 --> 00:41:05.790 align:middle line:90% 00:41:05.790 --> 00:41:07.770 align:middle line:84% Since I'm a bit short on time, maybe I'll 00:41:07.770 --> 00:41:09.075 align:middle line:90% switch to some slides. 00:41:09.075 --> 00:41:12.480 align:middle line:90% 00:41:12.480 --> 00:41:15.090 align:middle line:84% PROFESSOR: There's a famous quote in computer science 00:41:15.090 --> 00:41:18.330 align:middle line:84% that random numbers are far too important to be left to chance. 00:41:18.330 --> 00:41:21.120 align:middle line:90% 00:41:21.120 --> 00:41:24.390 align:middle line:84% So it's quite difficult to get a source of randomness 00:41:24.390 --> 00:41:25.060 align:middle line:90% on a computer. 00:41:25.060 --> 00:41:28.090 align:middle line:84% So there's a lot going on in the rand function. 00:41:28.090 --> 00:41:30.310 align:middle line:84% So once you have that, you don't want to touch it. 00:41:30.310 --> 00:41:33.435 align:middle line:84% And then you kind of reuse that for everything else. 00:41:33.435 --> 00:41:38.580 align:middle line:90% 00:41:38.580 --> 00:41:41.980 align:middle line:84% GAURAV ARYA: Yeah, so this is how it looks pictorially. 00:41:41.980 --> 00:41:45.900 align:middle line:84% So we're sampling this random number from omega, 00:41:45.900 --> 00:41:50.710 align:middle line:84% and you're plugging it in to x of p. 00:41:50.710 --> 00:41:54.753 align:middle line:90% And this is your map. 00:41:54.753 --> 00:41:56.920 align:middle line:84% And then you're sort of looking it up on the y-axis. 00:41:56.920 --> 00:41:59.430 align:middle line:90% 00:41:59.430 --> 00:42:01.200 align:middle line:84% OK, so that's how x of p looks like. 00:42:01.200 --> 00:42:03.480 align:middle line:84% We sample a uniform from 0 to 1, we plug it in. 00:42:03.480 --> 00:42:05.910 align:middle line:84% And at a point is that x of p plus epsilon, 00:42:05.910 --> 00:42:09.270 align:middle line:84% for instance, is jointly distributed with x and p. 00:42:09.270 --> 00:42:13.230 align:middle line:84% So if we were to pick this omega 1 on the real line, 00:42:13.230 --> 00:42:16.083 align:middle line:84% then our sample from x of p would be this. 00:42:16.083 --> 00:42:17.250 align:middle line:90% I hope you can see my mouse. 00:42:17.250 --> 00:42:23.580 align:middle line:84% And our sample from x of p plus epsilon would be this. 00:42:23.580 --> 00:42:25.650 align:middle line:84% And then their difference would be our sample 00:42:25.650 --> 00:42:27.570 align:middle line:84% from the x of epsilon, which, as you can see, 00:42:27.570 --> 00:42:31.492 align:middle line:84% is sort of getting smaller as epsilon gets smaller. 00:42:31.492 --> 00:42:32.950 align:middle line:84% So this whole procedure, which I'll 00:42:32.950 --> 00:42:34.660 align:middle line:84% describe in a bit more detail, they're 00:42:34.660 --> 00:42:36.160 align:middle line:84% called the reparameterization trick, 00:42:36.160 --> 00:42:39.460 align:middle line:84% and it's a very old trick and used nearly everywhere. 00:42:39.460 --> 00:42:42.580 align:middle line:84% For example, in those variational autoencoders. 00:42:42.580 --> 00:42:45.670 align:middle line:84% So before the reparameterization trick, 00:42:45.670 --> 00:42:49.810 align:middle line:84% you can also imagine how our idea of sampling from them 00:42:49.810 --> 00:42:51.980 align:middle line:90% independently would look like. 00:42:51.980 --> 00:42:53.230 align:middle line:90% And it looks like this. 00:42:53.230 --> 00:42:56.050 align:middle line:84% Rather than using the same omega for both of them, 00:42:56.050 --> 00:42:57.940 align:middle line:84% you sample two independent omegas, 00:42:57.940 --> 00:42:59.320 align:middle line:90% and you take their difference. 00:42:59.320 --> 00:43:02.320 align:middle line:84% And you can see that this can be pretty huge, no matter how 00:43:02.320 --> 00:43:03.350 align:middle line:90% small I make epsilon. 00:43:03.350 --> 00:43:05.350 align:middle line:84% So it's sort of like you're trying to figure out 00:43:05.350 --> 00:43:09.460 align:middle line:84% how this curve is shifting, how fast the area under this curve 00:43:09.460 --> 00:43:10.270 align:middle line:90% is changing. 00:43:10.270 --> 00:43:13.270 align:middle line:84% And the way you decided to do it is like a random sample 00:43:13.270 --> 00:43:14.650 align:middle line:90% from each of these curves. 00:43:14.650 --> 00:43:16.720 align:middle line:84% But what's going to happen is the noise 00:43:16.720 --> 00:43:19.750 align:middle line:84% of the intrinsic randomness is going to totally wash out 00:43:19.750 --> 00:43:23.530 align:middle line:84% the signal of how much my curve is going up. 00:43:23.530 --> 00:43:27.200 align:middle line:84% And the solution to that is use the same value 00:43:27.200 --> 00:43:29.830 align:middle line:84% in the y-axis, same omega, for both curves 00:43:29.830 --> 00:43:31.610 align:middle line:84% and directly sample their difference. 00:43:31.610 --> 00:43:36.160 align:middle line:84% That's how you can estimate how fast the curve is moving. 00:43:36.160 --> 00:43:39.070 align:middle line:84% So that's what we're going to do. 00:43:39.070 --> 00:43:42.490 align:middle line:84% So this is your dX of epsilon, which sort of subtracts 00:43:42.490 --> 00:43:45.670 align:middle line:90% these at every fixed omega. 00:43:45.670 --> 00:43:50.080 align:middle line:84% And now the nice property is that at every fixed omega, 00:43:50.080 --> 00:43:54.380 align:middle line:84% dX of epsilon is kind of like order epsilon. 00:43:54.380 --> 00:43:55.550 align:middle line:90% It's of magnitude epsilon. 00:43:55.550 --> 00:43:57.770 align:middle line:90% It's not O of 1. 00:43:57.770 --> 00:44:00.840 align:middle line:84% I think you've used this big O notation in the class. 00:44:00.840 --> 00:44:03.830 align:middle line:84% PROFESSOR: Yeah, we used little O notation, but yeah. 00:44:03.830 --> 00:44:05.030 align:middle line:90% GAURAV ARYA: OK, OK. 00:44:05.030 --> 00:44:05.990 align:middle line:84% PROFESSOR: This is similar, yeah. 00:44:05.990 --> 00:44:06.740 align:middle line:90% GAURAV ARYA: Yeah. 00:44:06.740 --> 00:44:07.850 align:middle line:90% Yeah. 00:44:07.850 --> 00:44:10.385 align:middle line:84% PROFESSOR: So it's proportional to epsilon or smaller. 00:44:10.385 --> 00:44:13.090 align:middle line:90% 00:44:13.090 --> 00:44:15.980 align:middle line:84% GAURAV ARYA: Yeah, so this is your differential. 00:44:15.980 --> 00:44:17.950 align:middle line:84% So we've solved problem one, which 00:44:17.950 --> 00:44:21.100 align:middle line:84% was how do we describe a change in our program? 00:44:21.100 --> 00:44:25.330 align:middle line:84% We've decided that it's going to be itself a random variable. 00:44:25.330 --> 00:44:28.360 align:middle line:84% And it's basically the subtraction 00:44:28.360 --> 00:44:31.150 align:middle line:84% of these two random variables with a particular joint 00:44:31.150 --> 00:44:31.750 align:middle line:90% distribution. 00:44:31.750 --> 00:44:35.080 align:middle line:84% Namely they're all sharing the same source of randomness. 00:44:35.080 --> 00:44:35.740 align:middle line:90% Yeah. 00:44:35.740 --> 00:44:39.010 align:middle line:84% And the final piece of the puzzle-- 00:44:39.010 --> 00:44:40.090 align:middle line:90% that's a bit scary-- 00:44:40.090 --> 00:44:43.720 align:middle line:84% is now that we have differentials, 00:44:43.720 --> 00:44:46.990 align:middle line:84% we can just take the derivative kind of like point-wise 00:44:46.990 --> 00:44:52.090 align:middle line:90% at every point on every omega. 00:44:52.090 --> 00:44:54.400 align:middle line:90% So we have this differential. 00:44:54.400 --> 00:44:56.650 align:middle line:84% And this differential, if you fix omega 1, 00:44:56.650 --> 00:44:59.740 align:middle line:84% it's moving up in a nice well-behaved way. 00:44:59.740 --> 00:45:00.980 align:middle line:90% This change is small. 00:45:00.980 --> 00:45:03.850 align:middle line:84% So we can actually compute its derivative. 00:45:03.850 --> 00:45:07.850 align:middle line:84% And we can compute its derivative at every omega. 00:45:07.850 --> 00:45:10.380 align:middle line:84% And then we'll have a new random variable. 00:45:10.380 --> 00:45:13.280 align:middle line:84% So remember a random variable on a probability space 00:45:13.280 --> 00:45:16.410 align:middle line:84% is just a function from omega to some output. 00:45:16.410 --> 00:45:18.680 align:middle line:84% So now our function from omega to our output 00:45:18.680 --> 00:45:24.920 align:middle line:84% will be this derivative at every particular omega. 00:45:24.920 --> 00:45:27.950 align:middle line:84% So that's represented by this thing here, 00:45:27.950 --> 00:45:31.475 align:middle line:84% the limit as epsilon goes to 0 of dX of epsilon over epsilon. 00:45:31.475 --> 00:45:34.880 align:middle line:90% 00:45:34.880 --> 00:45:36.380 align:middle line:90% And I'm calling this delta. 00:45:36.380 --> 00:45:38.660 align:middle line:84% And it turns out you can maybe-- it maybe 00:45:38.660 --> 00:45:41.210 align:middle line:84% it's pretty clear intuitively staring at this, 00:45:41.210 --> 00:45:44.060 align:middle line:84% that if you're sampling the derivatives at every fixed 00:45:44.060 --> 00:45:46.588 align:middle line:84% omega, what you're going to get, on average, 00:45:46.588 --> 00:45:48.755 align:middle line:84% is a derivative of the average of the whole program. 00:45:48.755 --> 00:45:51.515 align:middle line:90% 00:45:51.515 --> 00:45:53.053 align:middle line:84% I said that our derivative might not 00:45:53.053 --> 00:45:55.220 align:middle line:84% be exactly the same thing as the gradient estimator, 00:45:55.220 --> 00:45:57.890 align:middle line:84% but it sure looks like, in this case, it is. 00:45:57.890 --> 00:46:01.760 align:middle line:84% Like our derivative, which tells you every point in the sample 00:46:01.760 --> 00:46:05.630 align:middle line:84% space how much is the random function changing, 00:46:05.630 --> 00:46:07.920 align:middle line:84% is just, if you average it out, it 00:46:07.920 --> 00:46:11.670 align:middle line:84% is giving us what we decided we wanted at the beginning. 00:46:11.670 --> 00:46:14.470 align:middle line:90% So that was quite a lot. 00:46:14.470 --> 00:46:16.105 align:middle line:90% Are there any questions? 00:46:16.105 --> 00:46:17.730 align:middle line:84% PROFESSOR: Any questions at this point? 00:46:17.730 --> 00:46:21.810 align:middle line:90% 00:46:21.810 --> 00:46:25.640 align:middle line:90% So everyone's silent so far. 00:46:25.640 --> 00:46:27.750 align:middle line:90% Oh, one question. 00:46:27.750 --> 00:46:30.557 align:middle line:84% AUDIENCE: What does it mean for the derivative to be 0? 00:46:30.557 --> 00:46:32.640 align:middle line:84% PROFESSOR: So what does it mean for the derivative 00:46:32.640 --> 00:46:37.790 align:middle line:90% to be 0 for a program like this? 00:46:37.790 --> 00:46:39.930 align:middle line:84% GAURAV ARYA: For the derivative to be 0. 00:46:39.930 --> 00:46:41.010 align:middle line:90% So there's two senses. 00:46:41.010 --> 00:46:43.650 align:middle line:84% So yeah, the derivative to be 0 everywhere. 00:46:43.650 --> 00:46:47.250 align:middle line:90% So delta is a random variable. 00:46:47.250 --> 00:46:49.950 align:middle line:84% So maybe I should also just state for you 00:46:49.950 --> 00:46:53.200 align:middle line:84% what delta is for an exponential random variable. 00:46:53.200 --> 00:46:58.110 align:middle line:90% So let me show you that. 00:46:58.110 --> 00:47:01.380 align:middle line:90% 00:47:01.380 --> 00:47:04.200 align:middle line:90% So where did I have my-- 00:47:04.200 --> 00:47:07.810 align:middle line:90% 00:47:07.810 --> 00:47:08.640 align:middle line:90% I lost it. 00:47:08.640 --> 00:47:09.480 align:middle line:90% It's over here. 00:47:09.480 --> 00:47:11.520 align:middle line:90% OK. 00:47:11.520 --> 00:47:13.470 align:middle line:90% So this was our function. 00:47:13.470 --> 00:47:15.360 align:middle line:84% There was a little p added in there. 00:47:15.360 --> 00:47:17.550 align:middle line:84% Negative p times log of 1 minus omega. 00:47:17.550 --> 00:47:21.225 align:middle line:90% So we had the dX-- 00:47:21.225 --> 00:47:22.560 align:middle line:90% I'll need a new page. 00:47:22.560 --> 00:47:25.710 align:middle line:90% So dX of epsilon. 00:47:25.710 --> 00:47:27.480 align:middle line:84% This is a random variable, which, 00:47:27.480 --> 00:47:32.010 align:middle line:84% given a probability space, is just a map, taking an omega. 00:47:32.010 --> 00:47:38.280 align:middle line:84% And it's d by dp of negative p log of 1 00:47:38.280 --> 00:47:44.980 align:middle line:84% minus omega, which is equal to negative log of 1 minus omega. 00:47:44.980 --> 00:47:45.580 align:middle line:90% Sorry. 00:47:45.580 --> 00:47:47.020 align:middle line:90% Sorry. 00:47:47.020 --> 00:47:50.050 align:middle line:84% This is delta of omega, I would say, 00:47:50.050 --> 00:47:51.650 align:middle line:84% instead of taking the derivative. 00:47:51.650 --> 00:47:57.400 align:middle line:84% So delta of omega is negative log of 1 minus omega. 00:47:57.400 --> 00:47:58.850 align:middle line:90% So this is your random variable. 00:47:58.850 --> 00:48:01.360 align:middle line:84% So in this case, it certainly looks like it's not always 0. 00:48:01.360 --> 00:48:03.670 align:middle line:84% Although interestingly, it's always constant 00:48:03.670 --> 00:48:07.150 align:middle line:84% because this p was kind of just the scale. 00:48:07.150 --> 00:48:09.430 align:middle line:84% Now your question is, what if we got a case where 00:48:09.430 --> 00:48:13.720 align:middle line:90% delta everywhere in omega is 0? 00:48:13.720 --> 00:48:16.390 align:middle line:84% Well, that would just mean that your random variable 00:48:16.390 --> 00:48:17.870 align:middle line:90% isn't changing at all. 00:48:17.870 --> 00:48:21.460 align:middle line:84% So except in very degenerate cases, 00:48:21.460 --> 00:48:23.980 align:middle line:84% I wouldn't expect that to happen, 00:48:23.980 --> 00:48:26.680 align:middle line:84% where the whole distribution of your random variable 00:48:26.680 --> 00:48:27.790 align:middle line:90% isn't changing. 00:48:27.790 --> 00:48:30.280 align:middle line:84% Perhaps slightly more common would be 00:48:30.280 --> 00:48:34.480 align:middle line:84% a case where the derivative of the loss, the derivative 00:48:34.480 --> 00:48:36.550 align:middle line:90% of the expectation, is 0. 00:48:36.550 --> 00:48:38.200 align:middle line:90% Maybe it's slightly more common. 00:48:38.200 --> 00:48:41.910 align:middle line:84% And that would happen when the expectation of delta is 0. 00:48:41.910 --> 00:48:43.165 align:middle line:90% Yeah. 00:48:43.165 --> 00:48:44.040 align:middle line:90% Does that make sense? 00:48:44.040 --> 00:48:44.700 align:middle line:90% PROFESSOR: Another question. 00:48:44.700 --> 00:48:45.325 align:middle line:90% AUDIENCE: Yeah. 00:48:45.325 --> 00:48:47.910 align:middle line:84% I was just curious if this had any connection to what it 00:48:47.910 --> 00:48:49.770 align:middle line:84% means to be a stationary distribution 00:48:49.770 --> 00:48:50.910 align:middle line:90% for the exponential. 00:48:50.910 --> 00:48:52.410 align:middle line:84% PROFESSOR: So the question was, does 00:48:52.410 --> 00:48:54.452 align:middle line:84% this have any connection with what it means to be 00:48:54.452 --> 00:48:57.060 align:middle line:90% a stationary distribution? 00:48:57.060 --> 00:48:59.087 align:middle line:84% GAURAV ARYA: To be a stationary distribution. 00:48:59.087 --> 00:49:03.220 align:middle line:90% 00:49:03.220 --> 00:49:07.390 align:middle line:84% Not that I can think of at the top of my head. 00:49:07.390 --> 00:49:09.460 align:middle line:90% Yeah. 00:49:09.460 --> 00:49:10.582 align:middle line:90% I'm not totally sure. 00:49:10.582 --> 00:49:12.040 align:middle line:84% I mean, I guess there's some notion 00:49:12.040 --> 00:49:16.120 align:middle line:90% of derivative being 0 there. 00:49:16.120 --> 00:49:22.730 align:middle line:84% I guess if you let your p be sort of like timed, 00:49:22.730 --> 00:49:29.320 align:middle line:84% like sort of like a stochastic process in time, 00:49:29.320 --> 00:49:33.010 align:middle line:84% and you're differentiating with respect to t, 00:49:33.010 --> 00:49:40.080 align:middle line:84% then I would expect even then, even if you're-- 00:49:40.080 --> 00:49:43.770 align:middle line:84% I mean, yes, so it could be the case that you get-- 00:49:43.770 --> 00:49:47.490 align:middle line:84% you would definitely get this if you're at a stationary-- 00:49:47.490 --> 00:49:49.092 align:middle line:84% if your t is approaching infinity, 00:49:49.092 --> 00:49:50.550 align:middle line:84% so instead you're short of reaching 00:49:50.550 --> 00:49:52.950 align:middle line:84% a stationary distribution, and you're doing x of t 00:49:52.950 --> 00:49:57.450 align:middle line:84% and x of t plus delta t, then any statistical quantities 00:49:57.450 --> 00:49:59.370 align:middle line:90% are going to go to 0. 00:49:59.370 --> 00:50:01.560 align:middle line:84% It's possible that this is still non-zero 00:50:01.560 --> 00:50:03.228 align:middle line:84% because the parameterization depends. 00:50:03.228 --> 00:50:04.770 align:middle line:84% I really have to think about it more. 00:50:04.770 --> 00:50:06.930 align:middle line:84% But yeah, maybe there's some connection 00:50:06.930 --> 00:50:11.250 align:middle line:84% if you think of t as a time or number of steps 00:50:11.250 --> 00:50:12.820 align:middle line:90% in some continuous sense. 00:50:12.820 --> 00:50:13.320 align:middle line:90% Yeah. 00:50:13.320 --> 00:50:14.487 align:middle line:90% PROFESSOR: Another question? 00:50:14.487 --> 00:50:16.800 align:middle line:84% AUDIENCE: Shouldn't that be minus 1 over p? 00:50:16.800 --> 00:50:20.490 align:middle line:84% PROFESSOR: Shouldn't that be minus 1 over p? 00:50:20.490 --> 00:50:21.340 align:middle line:90% Where? 00:50:21.340 --> 00:50:22.200 align:middle line:90% AUDIENCE: No, like-- 00:50:22.200 --> 00:50:22.867 align:middle line:90% GAURAV ARYA: Oh. 00:50:22.867 --> 00:50:25.100 align:middle line:90% 00:50:25.100 --> 00:50:26.680 align:middle line:90% I don't think so. 00:50:26.680 --> 00:50:29.347 align:middle line:84% I mean, actually I think there are two different conventions. 00:50:29.347 --> 00:50:30.930 align:middle line:84% That's why it's very important that we 00:50:30.930 --> 00:50:33.030 align:middle line:84% clarified at the beginning that t was a scale. 00:50:33.030 --> 00:50:34.500 align:middle line:84% Often the exponential distribution 00:50:34.500 --> 00:50:36.060 align:middle line:84% is parametrized by the rate, which 00:50:36.060 --> 00:50:37.500 align:middle line:90% is the inverse of the scale. 00:50:37.500 --> 00:50:40.710 align:middle line:84% But here this makes sense to me because it's 00:50:40.710 --> 00:50:42.168 align:middle line:84% getting larger with larger p, which 00:50:42.168 --> 00:50:43.377 align:middle line:90% would make sense [INAUDIBLE]. 00:50:43.377 --> 00:50:45.180 align:middle line:84% PROFESSOR: Yes, p is not a probability. 00:50:45.180 --> 00:50:47.850 align:middle line:84% AUDIENCE: [INAUDIBLE] 1 minus p to the power minus px 00:50:47.850 --> 00:50:49.590 align:middle line:90% or minus x by p? 00:50:49.590 --> 00:50:50.610 align:middle line:90% PROFESSOR: Yes. 00:50:50.610 --> 00:50:54.300 align:middle line:84% So the point is p is the scale of-- 00:50:54.300 --> 00:50:58.197 align:middle line:90% so the average is p of x. 00:50:58.197 --> 00:51:00.940 align:middle line:90% All right. 00:51:00.940 --> 00:51:04.030 align:middle line:84% GAURAV ARYA: Yeah, so maybe I can write down 00:51:04.030 --> 00:51:07.840 align:middle line:90% our gradient estimator program. 00:51:07.840 --> 00:51:10.360 align:middle line:90% So this one looks like delta. 00:51:10.360 --> 00:51:13.670 align:middle line:84% And actually, it's technically a random variable, 00:51:13.670 --> 00:51:14.992 align:middle line:90% but in this case-- 00:51:14.992 --> 00:51:16.450 align:middle line:84% oh, sorry, it is a random variable. 00:51:16.450 --> 00:51:19.510 align:middle line:84% Sorry, I misspoke when I said it was constant earlier because it 00:51:19.510 --> 00:51:22.870 align:middle line:90% is a function of omega. 00:51:22.870 --> 00:51:27.070 align:middle line:84% So this is our random variable as a function of omega. 00:51:27.070 --> 00:51:31.330 align:middle line:84% And then if you want to sample delta, 00:51:31.330 --> 00:51:35.740 align:middle line:84% we can do negative log of 1 minus rand. 00:51:35.740 --> 00:51:39.780 align:middle line:90% So theta and rand. 00:51:39.780 --> 00:51:45.670 align:middle line:84% And now if we sample this a bunch of times, 00:51:45.670 --> 00:51:47.680 align:middle line:84% I'm getting something close to 1. 00:51:47.680 --> 00:51:51.940 align:middle line:84% And it actually turns out-- so if we go to our-- 00:51:51.940 --> 00:51:56.620 align:middle line:84% so let's just remind ourselves what x was. 00:51:56.620 --> 00:51:57.940 align:middle line:90% x was this. 00:51:57.940 --> 00:52:06.630 align:middle line:84% If I do this, this is actually p. 00:52:06.630 --> 00:52:08.780 align:middle line:90% So the derivative of p-- 00:52:08.780 --> 00:52:11.280 align:middle line:84% the derivative of expectation, which is p with respect to p, 00:52:11.280 --> 00:52:12.780 align:middle line:90% is just going to be 1. 00:52:12.780 --> 00:52:14.160 align:middle line:90% This makes sense. 00:52:14.160 --> 00:52:16.140 align:middle line:84% But then the crucial part is if we're 00:52:16.140 --> 00:52:18.270 align:middle line:84% interested in automatic differentiation, 00:52:18.270 --> 00:52:20.400 align:middle line:90% this notion of derivative-- 00:52:20.400 --> 00:52:23.520 align:middle line:84% we could have just said our notion of derivative is 1. 00:52:23.520 --> 00:52:25.930 align:middle line:84% That's the expectation of x by the p. 00:52:25.930 --> 00:52:26.547 align:middle line:90% It's 1. 00:52:26.547 --> 00:52:28.380 align:middle line:84% But the issue with that notion of derivative 00:52:28.380 --> 00:52:29.820 align:middle line:90% is it doesn't compose. 00:52:29.820 --> 00:52:31.202 align:middle line:90% So consider this new function. 00:52:31.202 --> 00:52:32.910 align:middle line:84% And remember it's super easy to just make 00:52:32.910 --> 00:52:34.710 align:middle line:84% the distribution really complicated by just doing 00:52:34.710 --> 00:52:35.920 align:middle line:90% a little bit of extra stuff. 00:52:35.920 --> 00:52:38.170 align:middle line:84% So here I'm sampling from an exponential distribution. 00:52:38.170 --> 00:52:39.000 align:middle line:90% I'm squaring. 00:52:39.000 --> 00:52:40.410 align:middle line:90% And it has some mean. 00:52:40.410 --> 00:52:43.710 align:middle line:84% And I might be interested in its derivative. 00:52:43.710 --> 00:52:47.070 align:middle line:84% And what you can do, I guess you've 00:52:47.070 --> 00:52:48.280 align:middle line:90% learned about dual numbers. 00:52:48.280 --> 00:52:50.980 align:middle line:84% So this is going to be a bit fast. 00:52:50.980 --> 00:52:54.900 align:middle line:90% 00:52:54.900 --> 00:52:56.250 align:middle line:90% Or maybe I'll just handwrite it. 00:52:56.250 --> 00:53:00.360 align:middle line:84% So if you use the chain rule, it should 00:53:00.360 --> 00:53:05.365 align:middle line:84% be 2 times the sample from x times a sample from delta. 00:53:05.365 --> 00:53:11.470 align:middle line:90% 00:53:11.470 --> 00:53:13.750 align:middle line:84% And you're just going to have to trust me that that's 00:53:13.750 --> 00:53:14.680 align:middle line:90% the true derivative. 00:53:14.680 --> 00:53:18.520 align:middle line:84% I'm not 100% sure because I did a bit of-- 00:53:18.520 --> 00:53:20.110 align:middle line:84% I'm not 100% sure I got it right. 00:53:20.110 --> 00:53:22.540 align:middle line:84% But basically, the chain rule just 00:53:22.540 --> 00:53:24.620 align:middle line:84% works with this notion of derivative, 00:53:24.620 --> 00:53:27.010 align:middle line:84% which isn't something I've totally justified to you, 00:53:27.010 --> 00:53:29.140 align:middle line:90% but that's sort of the upshot. 00:53:29.140 --> 00:53:31.838 align:middle line:84% So that's why this is really popular. 00:53:31.838 --> 00:53:33.130 align:middle line:90% This is a really popular trick. 00:53:33.130 --> 00:53:35.485 align:middle line:84% You can also use it for reverse mode differential. 00:53:35.485 --> 00:53:38.470 align:middle line:84% PROFESSOR: Gaurav, shouldn't that 00:53:38.470 --> 00:53:44.290 align:middle line:84% have been the same omega in the x, 100, and the sample delta? 00:53:44.290 --> 00:53:47.530 align:middle line:84% Right now, you're using independent random numbers 00:53:47.530 --> 00:53:51.040 align:middle line:84% in x, 100, and sample delta in that 2 times x times 00:53:51.040 --> 00:53:52.053 align:middle line:90% sample times delta. 00:53:52.053 --> 00:53:54.220 align:middle line:84% GAURAV ARYA: Yeah, you're right, so this is probably 00:53:54.220 --> 00:53:55.272 align:middle line:90% not the right answer. 00:53:55.272 --> 00:53:56.230 align:middle line:90% So I would have to do-- 00:53:56.230 --> 00:53:56.620 align:middle line:84% PROFESSOR: So they should have really 00:53:56.620 --> 00:53:57.730 align:middle line:90% been the same random number. 00:53:57.730 --> 00:53:58.938 align:middle line:90% GAURAV ARYA: --delta of rand. 00:53:58.938 --> 00:54:04.307 align:middle line:84% And then write my x as like negative 100 times log of-- 00:54:04.307 --> 00:54:05.890 align:middle line:84% PROFESSOR: No, it has to be the same-- 00:54:05.890 --> 00:54:06.670 align:middle line:90% GAURAV ARYA: --1 minus rand. 00:54:06.670 --> 00:54:07.420 align:middle line:90% PROFESSOR: No, no. 00:54:07.420 --> 00:54:10.280 align:middle line:84% No, you're still using independent random numbers. 00:54:10.280 --> 00:54:11.540 align:middle line:90% GAURAV ARYA: Oh. 00:54:11.540 --> 00:54:12.080 align:middle line:90% OK. 00:54:12.080 --> 00:54:15.140 align:middle line:84% Well, OK, maybe I don't have time to get it right here. 00:54:15.140 --> 00:54:16.820 align:middle line:84% I guess it would be an omega and a rand. 00:54:16.820 --> 00:54:20.840 align:middle line:84% But the point is that this notion of derivative 00:54:20.840 --> 00:54:23.932 align:middle line:84% obeys the chain rule, which is really nice. 00:54:23.932 --> 00:54:25.640 align:middle line:84% PROFESSOR: Yeah, because basically you're 00:54:25.640 --> 00:54:29.210 align:middle line:84% differentiating that it's just the ordinary derivative 00:54:29.210 --> 00:54:31.250 align:middle line:84% of that completely deterministic function that 00:54:31.250 --> 00:54:37.192 align:middle line:84% takes in rand, omega, and outputs something else. 00:54:37.192 --> 00:54:38.900 align:middle line:84% Once you have the deterministic function, 00:54:38.900 --> 00:54:40.983 align:middle line:84% it's the ordinary derivative, ordinary chain rule, 00:54:40.983 --> 00:54:42.910 align:middle line:90% ordinary everything. 00:54:42.910 --> 00:54:43.910 align:middle line:90% GAURAV ARYA: Yeah, yeah. 00:54:43.910 --> 00:54:46.367 align:middle line:84% And I think if you search up reparameterization trick, 00:54:46.367 --> 00:54:48.950 align:middle line:84% because it's gotten very popular in machine learning recently, 00:54:48.950 --> 00:54:51.600 align:middle line:84% there's a lot of resources you can find online. 00:54:51.600 --> 00:54:55.100 align:middle line:84% Some of them may explain it in a more barebones, simpler way 00:54:55.100 --> 00:55:00.050 align:middle line:84% than what I have done because part of the reason I've 00:55:00.050 --> 00:55:04.610 align:middle line:84% done what I've done is also to think about how we might 00:55:04.610 --> 00:55:06.770 align:middle line:84% generalize to the discrete case, which I'll try 00:55:06.770 --> 00:55:09.787 align:middle line:84% and just maybe just do a five-minute overview of, 00:55:09.787 --> 00:55:10.620 align:middle line:90% if that sounds fine? 00:55:10.620 --> 00:55:10.920 align:middle line:90% PROFESSOR: Sure. 00:55:10.920 --> 00:55:11.670 align:middle line:90% Yeah, that's fine. 00:55:11.670 --> 00:55:15.880 align:middle line:90% 00:55:15.880 --> 00:55:19.400 align:middle line:84% GAURAV ARYA: OK, so I'll just try and briefly explain. 00:55:19.400 --> 00:55:23.140 align:middle line:84% So basically, my project has been about taking that idea 00:55:23.140 --> 00:55:25.662 align:middle line:84% and now handling discrete random variables. 00:55:25.662 --> 00:55:27.370 align:middle line:84% The other example I gave at the beginning 00:55:27.370 --> 00:55:30.190 align:middle line:90% was a Bernoulli random variable. 00:55:30.190 --> 00:55:32.650 align:middle line:84% And now we want to apply the same approach. 00:55:32.650 --> 00:55:35.440 align:middle line:84% Come up with a notion of derivative 00:55:35.440 --> 00:55:39.340 align:middle line:84% that sort of explains the sensitivity of our program 00:55:39.340 --> 00:55:40.570 align:middle line:90% with respect to p. 00:55:40.570 --> 00:55:44.420 align:middle line:84% And why is this more difficult for a discrete random variable? 00:55:44.420 --> 00:55:47.780 align:middle line:84% So does this step function here make sense? 00:55:47.780 --> 00:55:50.200 align:middle line:84% Basically I'm using the same probability space. 00:55:50.200 --> 00:55:52.960 align:middle line:84% I'm choosing a uniform omega from 0 to 1. 00:55:52.960 --> 00:55:55.660 align:middle line:84% And now this point here is 1 minus p. 00:55:55.660 --> 00:55:58.060 align:middle line:84% So if I land on the right, if I land 00:55:58.060 --> 00:56:00.910 align:middle line:84% in this region of probability p, I'm going to get a 1. 00:56:00.910 --> 00:56:02.480 align:middle line:90% Otherwise, I'm going to get a 0. 00:56:02.480 --> 00:56:03.355 align:middle line:90% Does that make sense? 00:56:03.355 --> 00:56:06.390 align:middle line:90% 00:56:06.390 --> 00:56:08.072 align:middle line:84% PROFESSOR: Yep, people are nodding yes. 00:56:08.072 --> 00:56:09.030 align:middle line:90% GAURAV ARYA: All right. 00:56:09.030 --> 00:56:13.050 align:middle line:84% So now if we perturb p, something qualitatively 00:56:13.050 --> 00:56:16.410 align:middle line:84% very different happens, which is that where 00:56:16.410 --> 00:56:19.000 align:middle line:90% the step occurs changes. 00:56:19.000 --> 00:56:24.660 align:middle line:84% So if you think about it, nearly everywhere in our sample space, 00:56:24.660 --> 00:56:26.650 align:middle line:90% nothing is changing. 00:56:26.650 --> 00:56:30.210 align:middle line:84% So as I make epsilon smaller and smaller and smaller, 00:56:30.210 --> 00:56:33.690 align:middle line:84% that object delta now that I defined before is just 00:56:33.690 --> 00:56:36.780 align:middle line:84% going to be 0 everywhere because for small enough epsilon, 00:56:36.780 --> 00:56:38.550 align:middle line:84% nothing interesting is happening. 00:56:38.550 --> 00:56:42.510 align:middle line:84% But does this mean that the derivative of your expectation 00:56:42.510 --> 00:56:44.160 align:middle line:90% is 0? 00:56:44.160 --> 00:56:44.890 align:middle line:90% No. 00:56:44.890 --> 00:56:48.390 align:middle line:84% In fact, for Bernoulli, the derivative is 1. 00:56:48.390 --> 00:56:50.100 align:middle line:90% So what are we missing? 00:56:50.100 --> 00:56:53.250 align:middle line:84% We're missing the fact that the difference, 00:56:53.250 --> 00:56:56.970 align:middle line:84% if we look at the differential between these two, 00:56:56.970 --> 00:56:59.250 align:middle line:90% it's nearly everywhere 0. 00:56:59.250 --> 00:57:02.490 align:middle line:84% But in the place where it's non-zero, it's huge. 00:57:02.490 --> 00:57:04.170 align:middle line:90% It's a flip of the whole-- 00:57:04.170 --> 00:57:05.820 align:middle line:90% it's like a flip of the coin. 00:57:05.820 --> 00:57:09.370 align:middle line:84% This 1 isn't getting smaller and smaller in epsilon. 00:57:09.370 --> 00:57:12.660 align:middle line:84% So for people who like analysis, there 00:57:12.660 --> 00:57:16.350 align:middle line:84% was this interchange of limit and expectation 00:57:16.350 --> 00:57:18.420 align:middle line:84% that occurred here when we were trying 00:57:18.420 --> 00:57:22.080 align:middle line:84% to justify that this reparameterization trick works. 00:57:22.080 --> 00:57:25.260 align:middle line:84% And that is no longer valid, in this case, 00:57:25.260 --> 00:57:27.990 align:middle line:84% because if you think about dividing this by epsilon, 00:57:27.990 --> 00:57:31.230 align:middle line:84% this magnitude of this thing is really, really large. 00:57:31.230 --> 00:57:33.340 align:middle line:90% It's 1 over epsilon. 00:57:33.340 --> 00:57:35.730 align:middle line:84% So that's the essential challenge 00:57:35.730 --> 00:57:38.280 align:middle line:90% of discrete randomness. 00:57:38.280 --> 00:57:40.980 align:middle line:84% And if you think about it, so remember 00:57:40.980 --> 00:57:45.560 align:middle line:84% the derivative is the best linear approximation 00:57:45.560 --> 00:57:47.480 align:middle line:84% of the sensitivity of your program. 00:57:47.480 --> 00:57:49.250 align:middle line:84% It's the best linear approximation 00:57:49.250 --> 00:57:52.400 align:middle line:84% of how your program gets perturbed. 00:57:52.400 --> 00:57:57.500 align:middle line:84% And here what's linear is how fast this step 00:57:57.500 --> 00:57:59.750 align:middle line:90% is moving to the left. 00:57:59.750 --> 00:58:01.810 align:middle line:84% So it kind of feels like we want to differentiate 00:58:01.810 --> 00:58:06.310 align:middle line:84% not the value, the change in the value, 00:58:06.310 --> 00:58:09.220 align:middle line:84% but rather we want to differentiate the probability 00:58:09.220 --> 00:58:10.790 align:middle line:90% that they're different. 00:58:10.790 --> 00:58:13.300 align:middle line:84% So that's sort of the idea of how 00:58:13.300 --> 00:58:16.840 align:middle line:84% to extend the usual reparameterization trick, which 00:58:16.840 --> 00:58:21.580 align:middle line:84% just has delta, into an object which can also capture 00:58:21.580 --> 00:58:25.060 align:middle line:84% these flips with tiny probability, which 00:58:25.060 --> 00:58:26.600 align:middle line:90% you see in the discrete case. 00:58:26.600 --> 00:58:30.400 align:middle line:90% So I know that's very fast. 00:58:30.400 --> 00:58:33.040 align:middle line:84% And it's just really just the essential idea. 00:58:33.040 --> 00:58:34.340 align:middle line:90% Do people have questions? 00:58:34.340 --> 00:58:36.340 align:middle line:84% PROFESSOR: And so Gaurav actually just published 00:58:36.340 --> 00:58:38.980 align:middle line:90% a paper on exactly this. 00:58:38.980 --> 00:58:41.710 align:middle line:84% Basically you write a program that has rand calls 00:58:41.710 --> 00:58:44.620 align:middle line:84% and maybe has an, if rand is greater than 0.5, 00:58:44.620 --> 00:58:46.060 align:middle line:90% return cosine. 00:58:46.060 --> 00:58:48.340 align:middle line:84% Otherwise, return sine or something like that. 00:58:48.340 --> 00:58:51.970 align:middle line:84% And it can differentiate that in this sense, 00:58:51.970 --> 00:58:54.413 align:middle line:84% give you something whose expectation value is 00:58:54.413 --> 00:58:56.080 align:middle line:84% the derivative of the expectation value. 00:58:56.080 --> 00:59:01.220 align:middle line:84% But you have to track these discontinuous things that 00:59:01.220 --> 00:59:03.530 align:middle line:90% don't have an 18.01 derivative. 00:59:03.530 --> 00:59:06.468 align:middle line:84% You have to track them separately, in some sense. 00:59:06.468 --> 00:59:07.260 align:middle line:90% GAURAV ARYA: Right. 00:59:07.260 --> 00:59:09.570 align:middle line:90% Yeah, so here's a quick demo. 00:59:09.570 --> 00:59:13.880 align:middle line:84% So here I have a Bernoulli, which its mean is around 0.5. 00:59:13.880 --> 00:59:17.360 align:middle line:84% So we'd expect its average value to be-- 00:59:17.360 --> 00:59:19.220 align:middle line:84% so we'd expect its derivative to be 1 00:59:19.220 --> 00:59:20.780 align:middle line:90% because the expectation is p. 00:59:20.780 --> 00:59:23.155 align:middle line:84% So again, I'm always doing this boring case where we just 00:59:23.155 --> 00:59:24.380 align:middle line:90% have a single distribution. 00:59:24.380 --> 00:59:27.710 align:middle line:84% Maybe I'll try something more complicated right after this. 00:59:27.710 --> 00:59:31.295 align:middle line:84% But basically, the idea of our approach, using dual numbers-- 00:59:31.295 --> 00:59:35.030 align:middle line:90% 00:59:35.030 --> 00:59:36.620 align:middle line:84% this looks like a dual member, right? 00:59:36.620 --> 00:59:38.450 align:middle line:84% There's this epsilon thing, which is 00:59:38.450 --> 00:59:40.290 align:middle line:90% like this infinitesimal change. 00:59:40.290 --> 00:59:43.708 align:middle line:84% So it's like you got a 1, and your derivative is 0. 00:59:43.708 --> 00:59:45.500 align:middle line:84% This is sort of what the reparameterization 00:59:45.500 --> 00:59:48.440 align:middle line:84% trick would always say, that the derivative contribution is 0. 00:59:48.440 --> 00:59:51.290 align:middle line:84% But sometimes, what these triples 00:59:51.290 --> 00:59:55.610 align:middle line:84% tell you is that you could flip with probability that's 00:59:55.610 --> 00:59:56.820 align:middle line:90% like something times epsilon. 00:59:56.820 --> 00:59:59.023 align:middle line:84% So it's a probability that's differentiable. 00:59:59.023 --> 01:00:01.123 align:middle line:90% 01:00:01.123 --> 01:00:02.790 align:middle line:84% And then what this would be telling you, 01:00:02.790 --> 01:00:05.410 align:middle line:84% if you collapse this into a derivative estimator, 01:00:05.410 --> 01:00:11.790 align:middle line:84% so now the derivative object, which is all of these numbers, 01:00:11.790 --> 01:00:14.580 align:middle line:84% is no longer the same thing as your gradient estimator. 01:00:14.580 --> 01:00:17.190 align:middle line:84% But it can be collapsed into your gradient estimator 01:00:17.190 --> 01:00:22.390 align:middle line:84% by doing this value plus this value multiplied by that value. 01:00:22.390 --> 01:00:27.830 align:middle line:90% So if we sort of maybe-- 01:00:27.830 --> 01:00:31.400 align:middle line:84% if we let this be our triple, you can sort of collapse it 01:00:31.400 --> 01:00:32.960 align:middle line:90% into a gradient estimator. 01:00:32.960 --> 01:00:34.130 align:middle line:90% And it's going to be 2. 01:00:34.130 --> 01:00:38.645 align:middle line:84% But it's like 2 50% of the time and 0 50% of the time. 01:00:38.645 --> 01:00:39.770 align:middle line:90% So on average, that's what? 01:00:39.770 --> 01:00:41.810 align:middle line:84% Which is cool, but it's not very impressive 01:00:41.810 --> 01:00:43.250 align:middle line:90% for a single Bernoulli. 01:00:43.250 --> 01:00:46.610 align:middle line:90% So maybe I'll try a random walk. 01:00:46.610 --> 01:00:49.850 align:middle line:84% So it has to depend on some parameter p. 01:00:49.850 --> 01:00:54.320 align:middle line:84% So I'm going to do a random walk where I start maybe at 0. 01:00:54.320 --> 01:00:56.900 align:middle line:90% I equals 0. 01:00:56.900 --> 01:00:58.220 align:middle line:90% Maybe n equals 0. 01:00:58.220 --> 01:01:00.406 align:middle line:90% And I take 100 steps. 01:01:00.406 --> 01:01:02.310 align:middle line:90% Maybe I'll zoom on it more. 01:01:02.310 --> 01:01:06.050 align:middle line:84% And each step, I'm going to either stay put or go 01:01:06.050 --> 01:01:07.140 align:middle line:90% to the right. 01:01:07.140 --> 01:01:11.510 align:middle line:84% So I'm going to do n plus equal to rand of Bernoulli. 01:01:11.510 --> 01:01:13.940 align:middle line:84% And then I'm going to make my transition function depend 01:01:13.940 --> 01:01:14.810 align:middle line:90% on p. 01:01:14.810 --> 01:01:17.180 align:middle line:84% For example, in this case, maybe I'll 01:01:17.180 --> 01:01:26.870 align:middle line:90% do something like p over 100. 01:01:26.870 --> 01:01:31.912 align:middle line:90% Yeah, p times i over 100. 01:01:31.912 --> 01:01:34.120 align:middle line:84% So this is something that depends on where-- or maybe 01:01:34.120 --> 01:01:36.250 align:middle line:90% that should be [INAUDIBLE]. 01:01:36.250 --> 01:01:39.190 align:middle line:84% This is something that depends on where you currently 01:01:39.190 --> 01:01:41.770 align:middle line:84% are in your walk and also depends on p. 01:01:41.770 --> 01:01:44.050 align:middle line:84% And if this Bernoulli turns out to be true, 01:01:44.050 --> 01:01:45.520 align:middle line:84% you're going to walk to the right. 01:01:45.520 --> 01:01:47.505 align:middle line:90% Otherwise, you'll stay put. 01:01:47.505 --> 01:01:48.880 align:middle line:84% And then we're going to return n. 01:01:48.880 --> 01:01:51.930 align:middle line:90% 01:01:51.930 --> 01:01:54.570 align:middle line:84% So now we can just check that this function runs. 01:01:54.570 --> 01:01:55.125 align:middle line:90% It might not. 01:01:55.125 --> 01:01:58.680 align:middle line:90% 01:01:58.680 --> 01:02:00.055 align:middle line:84% Or it might do something strange. 01:02:00.055 --> 01:02:02.630 align:middle line:90% 01:02:02.630 --> 01:02:03.680 align:middle line:90% Oh. 01:02:03.680 --> 01:02:04.940 align:middle line:90% Yeah. 01:02:04.940 --> 01:02:06.890 align:middle line:84% Oh, this probability becomes tidy. 01:02:06.890 --> 01:02:11.260 align:middle line:90% 01:02:11.260 --> 01:02:13.830 align:middle line:90% Let's make p a bit larger. 01:02:13.830 --> 01:02:15.530 align:middle line:90% So this is like p of 50. 01:02:15.530 --> 01:02:19.033 align:middle line:84% PROFESSOR: It's always 0 because n starts at a 0. 01:02:19.033 --> 01:02:19.700 align:middle line:90% GAURAV ARYA: Oh. 01:02:19.700 --> 01:02:21.140 align:middle line:90% OK, thank you. 01:02:21.140 --> 01:02:22.730 align:middle line:90% Maybe I'll do 100 minus-- 01:02:22.730 --> 01:02:25.580 align:middle line:90% so let's do 1 minus all of that. 01:02:25.580 --> 01:02:27.153 align:middle line:90% Does that sound reasonable? 01:02:27.153 --> 01:02:27.820 align:middle line:90% PROFESSOR: Yeah. 01:02:27.820 --> 01:02:30.590 align:middle line:90% 01:02:30.590 --> 01:02:31.460 align:middle line:90% GAURAV ARYA: No. 01:02:31.460 --> 01:02:33.463 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:02:33.463 --> 01:02:34.130 align:middle line:90% GAURAV ARYA: OK. 01:02:34.130 --> 01:02:39.340 align:middle line:90% 01:02:39.340 --> 01:02:42.130 align:middle line:84% So let's have a function which goes 01:02:42.130 --> 01:02:49.300 align:middle line:84% from 1 to 0, dependent on where you are. 01:02:49.300 --> 01:02:51.580 align:middle line:84% So it's going to start at 1, go to 0. 01:02:51.580 --> 01:02:54.520 align:middle line:90% And let's multiply it by p. 01:02:54.520 --> 01:02:57.610 align:middle line:84% And then let's make p something like 0.5. 01:02:57.610 --> 01:02:59.240 align:middle line:84% OK, something interesting is occurring. 01:02:59.240 --> 01:03:01.160 align:middle line:90% So we just sample from this. 01:03:01.160 --> 01:03:02.035 align:middle line:90% Get something random. 01:03:02.035 --> 01:03:04.535 align:middle line:84% PROFESSOR: Yeah, so other people were suggesting probability 01:03:04.535 --> 01:03:05.200 align:middle line:90% depending on i. 01:03:05.200 --> 01:03:06.742 align:middle line:84% But he's doing even more complicated. 01:03:06.742 --> 01:03:09.010 align:middle line:84% The probability depends on where you currently are. 01:03:09.010 --> 01:03:09.723 align:middle line:90% Right. 01:03:09.723 --> 01:03:10.390 align:middle line:90% GAURAV ARYA: OK. 01:03:10.390 --> 01:03:11.210 align:middle line:90% Great. 01:03:11.210 --> 01:03:11.710 align:middle line:90% All right. 01:03:11.710 --> 01:03:12.720 align:middle line:90% So I'll do that too. 01:03:12.720 --> 01:03:15.610 align:middle line:90% 01:03:15.610 --> 01:03:17.650 align:middle line:90% So how should I do this? 01:03:17.650 --> 01:03:21.370 align:middle line:84% Maybe I'll do 1 minus n plus i over 200. 01:03:21.370 --> 01:03:24.730 align:middle line:84% This should be a valid probability. 01:03:24.730 --> 01:03:25.780 align:middle line:90% 0.5. 01:03:25.780 --> 01:03:26.440 align:middle line:90% OK. 01:03:26.440 --> 01:03:30.680 align:middle line:90% So it has a mean. 01:03:30.680 --> 01:03:31.880 align:middle line:90% PROFESSOR: OK, yeah. 01:03:31.880 --> 01:03:33.140 align:middle line:90% GAURAV ARYA: OK. 01:03:33.140 --> 01:03:35.000 align:middle line:84% If you just try and differentiate it 01:03:35.000 --> 01:03:38.720 align:middle line:84% the naive way by doing something like this 01:03:38.720 --> 01:03:42.735 align:middle line:84% and then maybe dividing by this-- 01:03:42.735 --> 01:03:47.100 align:middle line:90% 01:03:47.100 --> 01:03:48.480 align:middle line:90% have I got my brackets right? 01:03:48.480 --> 01:03:50.850 align:middle line:90% There's one missing, right? 01:03:50.850 --> 01:03:52.130 align:middle line:90% That should be there. 01:03:52.130 --> 01:03:52.650 align:middle line:90% OK. 01:03:52.650 --> 01:03:54.358 align:middle line:84% Then you're going to get something that's 01:03:54.358 --> 01:03:56.400 align:middle line:84% pretty large, which you'll have to trust 01:03:56.400 --> 01:03:57.660 align:middle line:90% averages did the right thing. 01:03:57.660 --> 01:04:02.490 align:middle line:84% But now if you try and use these stochastic triples, 01:04:02.490 --> 01:04:05.300 align:middle line:84% so maybe let's just do one stochastic triple. 01:04:05.300 --> 01:04:07.050 align:middle line:84% PROFESSOR: So again, the stochastic triple 01:04:07.050 --> 01:04:09.872 align:middle line:84% is the analog of what Professor Edelman did a couple lectures 01:04:09.872 --> 01:04:11.580 align:middle line:84% ago with automatic differentiation, where 01:04:11.580 --> 01:04:14.370 align:middle line:84% you had the value of the function and the derivative. 01:04:14.370 --> 01:04:16.240 align:middle line:90% We call that a dual number. 01:04:16.240 --> 01:04:18.850 align:middle line:84% Now you can run three numbers, the function, 01:04:18.850 --> 01:04:21.870 align:middle line:84% the derivative, the change up, down of the distribution, 01:04:21.870 --> 01:04:23.980 align:middle line:84% and also the probability of a big jump. 01:04:23.980 --> 01:04:27.480 align:middle line:84% So now you need a third bit of information. 01:04:27.480 --> 01:04:29.530 align:middle line:90% You carry through. 01:04:29.530 --> 01:04:30.280 align:middle line:90% GAURAV ARYA: Yeah. 01:04:30.280 --> 01:04:32.290 align:middle line:84% So yeah, so for these stochastic triples, 01:04:32.290 --> 01:04:35.230 align:middle line:84% I'm just showing you the whole triple object first. 01:04:35.230 --> 01:04:37.710 align:middle line:84% There's never a discontinuous part. 01:04:37.710 --> 01:04:39.460 align:middle line:84% The deltas are telling you how your output 01:04:39.460 --> 01:04:40.510 align:middle line:90% is changing continuously. 01:04:40.510 --> 01:04:42.010 align:middle line:84% But your output is an integer, so it 01:04:42.010 --> 01:04:43.660 align:middle line:84% can't change continuously, but it 01:04:43.660 --> 01:04:47.920 align:middle line:84% can change to an adjacent value with some probability. 01:04:47.920 --> 01:04:49.730 align:middle line:84% And the nice thing is even though the range 01:04:49.730 --> 01:04:51.730 align:middle line:84% of your function is pretty large-- it can be 36. 01:04:51.730 --> 01:04:52.990 align:middle line:90% It can be 29-- 01:04:52.990 --> 01:04:58.480 align:middle line:84% we sort of couple the original 29 to the alternate, which 01:04:58.480 --> 01:05:03.273 align:middle line:84% is like 30, which means that your estimator is going to be 01:05:03.273 --> 01:05:04.440 align:middle line:90% lower variance than average. 01:05:04.440 --> 01:05:06.280 align:middle line:84% It's just sort of the stuff I was 01:05:06.280 --> 01:05:07.900 align:middle line:84% saying with this joint distribution 01:05:07.900 --> 01:05:09.820 align:middle line:84% and what the reparameterization trick does 01:05:09.820 --> 01:05:12.200 align:middle line:84% so well is something we're trying to capture 01:05:12.200 --> 01:05:13.450 align:middle line:90% with these stochastic triples. 01:05:13.450 --> 01:05:18.970 align:middle line:84% It's still a work in progress, and there's a lot of issues. 01:05:18.970 --> 01:05:21.620 align:middle line:90% But now we can also-- 01:05:21.620 --> 01:05:24.280 align:middle line:84% so now we can take this stochastic triple, 01:05:24.280 --> 01:05:26.140 align:middle line:84% we can propagate it through your program, 01:05:26.140 --> 01:05:29.440 align:middle line:84% we can collapse it into a gradient estimator. 01:05:29.440 --> 01:05:32.230 align:middle line:84% And then we can do this 1,000 times. 01:05:32.230 --> 01:05:35.500 align:middle line:84% And you're going to have to trust me that this is 01:05:35.500 --> 01:05:37.630 align:middle line:90% the derivative of your program. 01:05:37.630 --> 01:05:40.330 align:middle line:84% You could verify it by computing the transition matrix 01:05:40.330 --> 01:05:43.120 align:middle line:84% or using an alternative gradient estimation method, 01:05:43.120 --> 01:05:44.753 align:middle line:90% but that's the upshot. 01:05:44.753 --> 01:05:46.420 align:middle line:84% That with this new notion of derivative, 01:05:46.420 --> 01:05:49.030 align:middle line:84% we can now answer this problem we wanted 01:05:49.030 --> 01:05:50.590 align:middle line:84% to solve from the start, which is 01:05:50.590 --> 01:05:55.330 align:middle line:84% for any complicated stochastic simulation, 01:05:55.330 --> 01:05:58.810 align:middle line:84% can we get a new program which averages the derivative 01:05:58.810 --> 01:06:00.670 align:middle line:90% of the average of the original? 01:06:00.670 --> 01:06:01.740 align:middle line:90% Yeah. 01:06:01.740 --> 01:06:02.510 align:middle line:90% That's all I had. 01:06:02.510 --> 01:06:04.330 align:middle line:90% Are there any questions? 01:06:04.330 --> 01:06:07.740 align:middle line:90% PROFESSOR: Any questions? 01:06:07.740 --> 01:06:10.200 align:middle line:84% OK, so let's take a five-minute break. 01:06:10.200 --> 01:06:12.890 align:middle line:90% So we'll start again at 12:14. 01:06:12.890 --> 01:06:17.000 align:middle line:90%