WEBVTT 00:00:00.000 --> 00:00:02.380 align:middle line:90% [SQUEAKING] 00:00:02.380 --> 00:00:03.332 align:middle line:90% [RUSTLING] 00:00:03.332 --> 00:00:05.236 align:middle line:90% [CLICKING] 00:00:05.236 --> 00:00:09.520 align:middle line:90% 00:00:09.520 --> 00:00:11.440 align:middle line:84% ALAN EDELMAN: Today, the plan would be I'm 00:00:11.440 --> 00:00:13.375 align:middle line:90% going to talk a little bit-- 00:00:13.375 --> 00:00:15.250 align:middle line:84% I've got this notebook from last year talking 00:00:15.250 --> 00:00:19.720 align:middle line:84% about taking the derivative of such matrix quantities 00:00:19.720 --> 00:00:20.870 align:middle line:90% as eigenvalues. 00:00:20.870 --> 00:00:23.800 align:middle line:84% And so what's going on here is that we are-- 00:00:23.800 --> 00:00:26.170 align:middle line:84% the big story is orthogonal matrices 00:00:26.170 --> 00:00:28.910 align:middle line:84% may play a role if you have symmetric eigenvalue problems. 00:00:28.910 --> 00:00:30.650 align:middle line:84% And so we have constrained problems, 00:00:30.650 --> 00:00:32.930 align:middle line:84% which is something we haven't really talked about yet. 00:00:32.930 --> 00:00:34.390 align:middle line:84% So we'll talk about differentiating 00:00:34.390 --> 00:00:36.250 align:middle line:90% under constraints a little bit. 00:00:36.250 --> 00:00:37.960 align:middle line:90% Then I will use the blackboard. 00:00:37.960 --> 00:00:40.930 align:middle line:84% And you might remember from over a week ago, I think it was, 00:00:40.930 --> 00:00:43.420 align:middle line:84% where I was talking about forward 00:00:43.420 --> 00:00:45.820 align:middle line:84% and reverse mode differentiation on the blackboard. 00:00:45.820 --> 00:00:48.378 align:middle line:84% And we made it through forward, but we never 00:00:48.378 --> 00:00:50.170 align:middle line:84% made it through backwards, or reverse mode. 00:00:50.170 --> 00:00:52.420 align:middle line:84% And so the plan is to kind of remember where we 00:00:52.420 --> 00:00:54.910 align:middle line:90% were and go back and do that. 00:00:54.910 --> 00:00:58.120 align:middle line:84% And Steven had the idea of, the last 10 minutes-- 00:00:58.120 --> 00:01:00.790 align:middle line:84% at the end of class, we'll talk about the things 00:01:00.790 --> 00:01:02.440 align:middle line:90% that we haven't talked about. 00:01:02.440 --> 00:01:05.200 align:middle line:84% We've already figured out that, in principle, this class could 00:01:05.200 --> 00:01:07.160 align:middle line:90% go on for another week or two. 00:01:07.160 --> 00:01:11.050 align:middle line:84% But of course, IEP is coming to an end this week. 00:01:11.050 --> 00:01:12.800 align:middle line:84% But there's so much more that we could do. 00:01:12.800 --> 00:01:15.710 align:middle line:84% So here, let me just kind go through this a little bit 00:01:15.710 --> 00:01:16.790 align:middle line:90% quickly. 00:01:16.790 --> 00:01:20.610 align:middle line:84% So ultimately, I want to think about orthogonal matrices, 00:01:20.610 --> 00:01:21.110 align:middle line:90% right? 00:01:21.110 --> 00:01:22.880 align:middle line:84% You all know what orthogonal matrices are. 00:01:22.880 --> 00:01:24.710 align:middle line:90% q transpose q is the identity. 00:01:24.710 --> 00:01:28.850 align:middle line:84% But to go slowly and as a bit of a warm up, 00:01:28.850 --> 00:01:32.630 align:middle line:84% let me just talk about differentiating on the unit 00:01:32.630 --> 00:01:33.420 align:middle line:90% sphere. 00:01:33.420 --> 00:01:33.920 align:middle line:90% OK? 00:01:33.920 --> 00:01:35.060 align:middle line:90% So the unit sphere-- 00:01:35.060 --> 00:01:35.810 align:middle line:90% right? 00:01:35.810 --> 00:01:37.248 align:middle line:84% The sphere is like a baby version 00:01:37.248 --> 00:01:38.540 align:middle line:90% of an orthogonal matrix, right? 00:01:38.540 --> 00:01:40.800 align:middle line:84% It's one column of an orthogonal matrix. 00:01:40.800 --> 00:01:44.580 align:middle line:84% So let's talk about being on the sphere. 00:01:44.580 --> 00:01:45.080 align:middle line:90% OK? 00:01:45.080 --> 00:01:51.510 align:middle line:84% And just as a reminder, if you are restricted-- 00:01:51.510 --> 00:01:52.630 align:middle line:90% [BARK] 00:01:52.630 --> 00:01:53.630 align:middle line:90% Philly, what's going on? 00:01:53.630 --> 00:01:55.310 align:middle line:84% There's something about the sunbeams. 00:01:55.310 --> 00:01:58.850 align:middle line:84% He likes the-- oh, it's somebody's reflection. 00:01:58.850 --> 00:01:59.720 align:middle line:90% Is it mine? 00:01:59.720 --> 00:02:02.090 align:middle line:84% Or somebody is reflecting their phone. 00:02:02.090 --> 00:02:03.710 align:middle line:90% And Philip is going to chase it. 00:02:03.710 --> 00:02:13.360 align:middle line:84% [CHUCKLES] He-- he's really part cat somewhere 00:02:13.360 --> 00:02:14.740 align:middle line:90% genetically or something. 00:02:14.740 --> 00:02:16.570 align:middle line:90% Yeah, he will chase it. 00:02:16.570 --> 00:02:18.640 align:middle line:90% [CHUCKLING] OK. 00:02:18.640 --> 00:02:21.350 align:middle line:90% Thanks for-- all right. 00:02:21.350 --> 00:02:21.850 align:middle line:90% Where was I? 00:02:21.850 --> 00:02:22.350 align:middle line:90% Yes. 00:02:22.350 --> 00:02:26.440 align:middle line:84% So you all know that if you move along on a circle, 00:02:26.440 --> 00:02:29.470 align:middle line:84% then tangents on the circle or [? quasar ?] 00:02:29.470 --> 00:02:31.050 align:middle line:90% are orthogonal, right? 00:02:31.050 --> 00:02:33.550 align:middle line:84% If you think of the vector of where you are from the center, 00:02:33.550 --> 00:02:35.500 align:middle line:84% right, tangents of course are orthogonal. 00:02:35.500 --> 00:02:37.480 align:middle line:84% And that's true in n dimensions-- 00:02:37.480 --> 00:02:41.390 align:middle line:84% that if you move around along a sphere in n dimensions, 00:02:41.390 --> 00:02:43.858 align:middle line:84% your position vector is from the origin. 00:02:43.858 --> 00:02:45.400 align:middle line:84% And then the velocity is always going 00:02:45.400 --> 00:02:48.550 align:middle line:84% to be orthogonal or tangent to the sphere, right? 00:02:48.550 --> 00:02:51.460 align:middle line:84% And so all of a sudden now, we're talking about-- 00:02:51.460 --> 00:02:54.910 align:middle line:84% up until now, derivatives were free to be in any direction, 00:02:54.910 --> 00:02:55.780 align:middle line:90% right? 00:02:55.780 --> 00:02:58.210 align:middle line:84% All of our little dx's that we've seen-- 00:02:58.210 --> 00:02:59.920 align:middle line:84% or da's or whatever you want to call 00:02:59.920 --> 00:03:03.610 align:middle line:84% it-- have always been free to be in any direction you like. 00:03:03.610 --> 00:03:08.170 align:middle line:84% But if your point is restricted to a surface like the sphere, 00:03:08.170 --> 00:03:10.270 align:middle line:84% then somehow your dx is restricted 00:03:10.270 --> 00:03:12.880 align:middle line:84% to being tangent to the sphere now, right? 00:03:12.880 --> 00:03:14.890 align:middle line:84% It's not in any direction anymore, right? 00:03:14.890 --> 00:03:17.920 align:middle line:84% We're no longer interested in radial directions 00:03:17.920 --> 00:03:19.600 align:middle line:84% because they don't even exist, right? 00:03:19.600 --> 00:03:21.470 align:middle line:84% We're only interested in tangent directions 00:03:21.470 --> 00:03:24.300 align:middle line:84% because they describe movements on the constrained surface, 00:03:24.300 --> 00:03:24.800 align:middle line:90% right? 00:03:24.800 --> 00:03:26.758 align:middle line:84% And so we need to figure out how to accommodate 00:03:26.758 --> 00:03:29.750 align:middle line:84% these sorts of tangential directions, right? 00:03:29.750 --> 00:03:38.540 align:middle line:84% And so here, I'm kind of just showing you roughly 00:03:38.540 --> 00:03:39.742 align:middle line:90% what it would look like. 00:03:39.742 --> 00:03:41.450 align:middle line:84% By the way, is the sun drowning this out? 00:03:41.450 --> 00:03:42.680 align:middle line:90% Or is it OK? 00:03:42.680 --> 00:03:45.360 align:middle line:84% I'm wondering if this one should close-- 00:03:45.360 --> 00:03:48.470 align:middle line:90% or at least the little-- 00:03:48.470 --> 00:03:49.657 align:middle line:90% or partially close. 00:03:49.657 --> 00:03:51.740 align:middle line:84% It looks like there's a lot of glare from the sun. 00:03:51.740 --> 00:03:52.782 align:middle line:90% I'm happy to see the sun. 00:03:52.782 --> 00:03:54.080 align:middle line:90% Don't get me wrong. 00:03:54.080 --> 00:03:56.810 align:middle line:84% We'll take as much of it as we can get this. 00:03:56.810 --> 00:03:59.730 align:middle line:84% But a little-- yeah, that might be a little bit better. 00:03:59.730 --> 00:04:00.230 align:middle line:90% Right. 00:04:00.230 --> 00:04:02.390 align:middle line:84% So just to kind of show you how you 00:04:02.390 --> 00:04:06.530 align:middle line:84% construct sort of the right kind of dq here, right-- 00:04:06.530 --> 00:04:08.120 align:middle line:84% by the way, do people know the trick 00:04:08.120 --> 00:04:10.800 align:middle line:84% for generating a random vector on the sphere? 00:04:10.800 --> 00:04:17.190 align:middle line:84% So if you-- so the question is, on the sphere, 00:04:17.190 --> 00:04:19.190 align:middle line:84% you would like to pick a point uniformly, right? 00:04:19.190 --> 00:04:21.110 align:middle line:84% "Uniformly" means that, no matter 00:04:21.110 --> 00:04:23.870 align:middle line:84% how you rotate the sphere, how you're picking it 00:04:23.870 --> 00:04:25.230 align:middle line:90% doesn't matter, right? 00:04:25.230 --> 00:04:28.310 align:middle line:84% And the trick to doing that is to take 00:04:28.310 --> 00:04:29.930 align:middle line:90% x to be a standard normal. 00:04:29.930 --> 00:04:32.630 align:middle line:84% Like, here, I'm doing it in five dimensions, right? 00:04:32.630 --> 00:04:34.340 align:middle line:90% So x is a standard normal. 00:04:34.340 --> 00:04:37.070 align:middle line:84% And then all you have to do is normalize, 00:04:37.070 --> 00:04:38.570 align:middle line:90% which makes it a unit vector. 00:04:38.570 --> 00:04:41.795 align:middle line:84% And properties of the normal distribution 00:04:41.795 --> 00:04:43.670 align:middle line:84% have it that it's invariant on the rotations. 00:04:43.670 --> 00:04:46.520 align:middle line:84% And so q is invariant on the sphere, right? 00:04:46.520 --> 00:04:48.650 align:middle line:84% And so that's the simplest way to generate 00:04:48.650 --> 00:04:52.130 align:middle line:84% a random vector of unit one that actually 00:04:52.130 --> 00:04:53.900 align:middle line:90% is uniform on the sphere. 00:04:53.900 --> 00:04:56.090 align:middle line:90% If you took Rand, which is-- 00:04:56.090 --> 00:04:57.690 align:middle line:84% you wouldn't get the right answer. 00:04:57.690 --> 00:05:00.570 align:middle line:84% It's very special to take Rand n, which 00:05:00.570 --> 00:05:01.960 align:middle line:90% is the normal distribution. 00:05:01.960 --> 00:05:05.290 align:middle line:90% So q is on the sphere. 00:05:05.290 --> 00:05:08.490 align:middle line:90% And if you take a little dx-- 00:05:08.490 --> 00:05:09.700 align:middle line:90% you guys know me by now. 00:05:09.700 --> 00:05:12.850 align:middle line:84% I like to type 0.0001, something like that. 00:05:12.850 --> 00:05:15.980 align:middle line:84% If you take a random direction, of course, 00:05:15.980 --> 00:05:17.860 align:middle line:84% it won't be tangential to the sphere. 00:05:17.860 --> 00:05:23.260 align:middle line:84% So what I'm doing is I'm normalizing it 00:05:23.260 --> 00:05:27.190 align:middle line:84% and subtracting it from q to kind of see-- 00:05:27.190 --> 00:05:31.040 align:middle line:84% so this is a random nearby point from x on the sphere. 00:05:31.040 --> 00:05:32.800 align:middle line:84% And I'm going to see its distance from q. 00:05:32.800 --> 00:05:34.765 align:middle line:84% And I will observe that q transpose 00:05:34.765 --> 00:05:37.490 align:middle line:90% dq is heading towards 0. 00:05:37.490 --> 00:05:37.990 align:middle line:90% OK? 00:05:37.990 --> 00:05:40.965 align:middle line:84% So that's basically saying that-- 00:05:40.965 --> 00:05:42.590 align:middle line:84% I mean, I hope you all see the picture. 00:05:42.590 --> 00:05:43.990 align:middle line:84% But here, let's draw the picture. 00:05:43.990 --> 00:05:45.200 align:middle line:90% Maybe that'd be even better. 00:05:45.200 --> 00:05:48.100 align:middle line:90% So here's the picture, right? 00:05:48.100 --> 00:05:50.440 align:middle line:90% I've got my q over here. 00:05:50.440 --> 00:05:57.710 align:middle line:84% And I've got my q plus dq just a little bit off from here. 00:05:57.710 --> 00:06:00.040 align:middle line:90% And so this is basically the dq. 00:06:00.040 --> 00:06:02.980 align:middle line:90% And what I'm showing you is-- 00:06:02.980 --> 00:06:06.878 align:middle line:84% numerically, the things that's easy to believe in your mind 00:06:06.878 --> 00:06:09.170 align:middle line:84% is that this is a right angle, right-- that q transpose 00:06:09.170 --> 00:06:10.400 align:middle line:90% dq is 0, right? 00:06:10.400 --> 00:06:12.860 align:middle line:84% So this right angle over here is the same thing 00:06:12.860 --> 00:06:15.980 align:middle line:90% as saying q transpose dq is 0. 00:06:15.980 --> 00:06:18.530 align:middle line:90% Any questions about that? 00:06:18.530 --> 00:06:20.330 align:middle line:84% That's all pretty intuitive, right, 00:06:20.330 --> 00:06:23.710 align:middle line:84% that tangents are orthogonal to the radii 00:06:23.710 --> 00:06:25.580 align:middle line:90% no matter what the dimensions? 00:06:25.580 --> 00:06:26.080 align:middle line:90% OK? 00:06:26.080 --> 00:06:30.370 align:middle line:84% And there's a math way of describing this. 00:06:30.370 --> 00:06:32.840 align:middle line:84% You don't just need your geometric intuition to see it. 00:06:32.840 --> 00:06:34.090 align:middle line:90% You just do the obvious thing. 00:06:34.090 --> 00:06:36.490 align:middle line:84% If you're restricted to the sphere, 00:06:36.490 --> 00:06:38.890 align:middle line:84% if x transpose x equals 1, then we 00:06:38.890 --> 00:06:42.010 align:middle line:84% could take our usual rules to differentiate, right? 00:06:42.010 --> 00:06:45.400 align:middle line:84% So if we differentiate, we have dx transpose 00:06:45.400 --> 00:06:47.410 align:middle line:90% x plus x transpose dx. 00:06:47.410 --> 00:06:51.220 align:middle line:84% And Steven spent a lot of time in the first week reminding you 00:06:51.220 --> 00:06:54.260 align:middle line:84% that x transpose dx and dx transpose x are the same thing. 00:06:54.260 --> 00:06:55.400 align:middle line:90% They're just dot products. 00:06:55.400 --> 00:06:59.170 align:middle line:84% So we can combine them to get 2x transpose dx is the derivative 00:06:59.170 --> 00:07:00.850 align:middle line:90% of the constant 1, which is 0. 00:07:00.850 --> 00:07:03.505 align:middle line:90% And so here, we get the-- 00:07:03.505 --> 00:07:06.070 align:middle line:84% this is a constraint on the perturbation, right? 00:07:06.070 --> 00:07:09.400 align:middle line:84% It says that dx is orthogonal to x. 00:07:09.400 --> 00:07:12.550 align:middle line:84% Those are the only dx's we're allowed. 00:07:12.550 --> 00:07:14.560 align:middle line:84% We're not allowed to go in any direction. 00:07:14.560 --> 00:07:16.600 align:middle line:84% Only those that are orthogonal to x 00:07:16.600 --> 00:07:19.040 align:middle line:84% are what we're going to consider if we want to restrict 00:07:19.040 --> 00:07:20.460 align:middle line:90% ourselves to a sphere. 00:07:20.460 --> 00:07:20.960 align:middle line:90% OK? 00:07:20.960 --> 00:07:23.910 align:middle line:84% So that's working with constraints. 00:07:23.910 --> 00:07:27.130 align:middle line:90% And this is our first example. 00:07:27.130 --> 00:07:29.130 align:middle line:90% You can even do it-- 00:07:29.130 --> 00:07:33.450 align:middle line:84% here's just a little exercise just to kind of check. 00:07:33.450 --> 00:07:36.510 align:middle line:84% Suppose x is restricted to being on a circle, right? 00:07:36.510 --> 00:07:40.110 align:middle line:84% So x is a vector in two dimensions, 00:07:40.110 --> 00:07:43.010 align:middle line:84% but it's parameterized by theta over here. 00:07:43.010 --> 00:07:45.480 align:middle line:84% And so it's living on just a pure circle. 00:07:45.480 --> 00:07:47.610 align:middle line:84% If you wanted to calculate x transpose 00:07:47.610 --> 00:07:50.520 align:middle line:84% dx as a kind of quick check just to show that it all-- 00:07:50.520 --> 00:07:52.030 align:middle line:84% I just like seeing that it works. 00:07:52.030 --> 00:07:53.780 align:middle line:84% I mean, again, there's nothing fancy here. 00:07:53.780 --> 00:07:56.580 align:middle line:84% But x is the vector cosine theta sine theta. 00:07:56.580 --> 00:07:59.730 align:middle line:84% dx just by taking simple, ordinary derivatives 00:07:59.730 --> 00:08:02.860 align:middle line:84% is minus sine theta d theta and cosine theta d theta. 00:08:02.860 --> 00:08:04.140 align:middle line:90% We can pull out the d theta. 00:08:04.140 --> 00:08:07.650 align:middle line:84% And you can see that this dot product is obviously 00:08:07.650 --> 00:08:09.810 align:middle line:84% minus cosine sine plus sine cosine. 00:08:09.810 --> 00:08:11.230 align:middle line:90% It's obviously 0. 00:08:11.230 --> 00:08:11.730 align:middle line:90% OK? 00:08:11.730 --> 00:08:14.060 align:middle line:90% So you-- 00:08:14.060 --> 00:08:19.030 align:middle line:84% And then there's a little bit of math gossip-- 00:08:19.030 --> 00:08:20.732 align:middle line:84% maybe just as a side story, if you'd 00:08:20.732 --> 00:08:22.440 align:middle line:84% like to hear a little bit of math gossip. 00:08:22.440 --> 00:08:28.710 align:middle line:84% So the x vector with its two coordinates 00:08:28.710 --> 00:08:31.230 align:middle line:84% is sometimes called an extrinsic vector, right? 00:08:31.230 --> 00:08:34.320 align:middle line:84% It's living in this plane, right? 00:08:34.320 --> 00:08:38.400 align:middle line:84% The single theta coordinate, the one-dimensional-- 00:08:38.400 --> 00:08:41.100 align:middle line:90% 00:08:41.100 --> 00:08:45.435 align:middle line:84% the perimeter of a circle is a one-dimensional object, right? 00:08:45.435 --> 00:08:48.360 align:middle line:84% That's what's called intrinsic coordinates-- that, here, we 00:08:48.360 --> 00:08:50.940 align:middle line:84% are just parameter-- like, from the theta point of view, 00:08:50.940 --> 00:08:52.710 align:middle line:84% there's nothing but the circle, right? 00:08:52.710 --> 00:08:54.030 align:middle line:90% x has two coordinates. 00:08:54.030 --> 00:08:55.200 align:middle line:90% And we have to constrain it. 00:08:55.200 --> 00:08:59.940 align:middle line:84% And mathematicians started out doing everything extrinsically. 00:08:59.940 --> 00:09:02.220 align:middle line:84% And then they discovered that intrinsic was better. 00:09:02.220 --> 00:09:04.290 align:middle line:84% And nowadays, many pure mathematicians 00:09:04.290 --> 00:09:06.870 align:middle line:84% will tell you you've got to do things intrinsically, 00:09:06.870 --> 00:09:07.930 align:middle line:90% it's so much better. 00:09:07.930 --> 00:09:10.230 align:middle line:84% But I'll tell you, from computational viewpoints, 00:09:10.230 --> 00:09:12.505 align:middle line:84% I think extrinsic is actually better in the end. 00:09:12.505 --> 00:09:14.880 align:middle line:84% So I'm not convinced of what the pure mathematicians will 00:09:14.880 --> 00:09:15.550 align:middle line:90% tell you. 00:09:15.550 --> 00:09:17.530 align:middle line:90% So you get to decide. 00:09:17.530 --> 00:09:24.800 align:middle line:84% But whatever variables work is fine for you. 00:09:24.800 --> 00:09:25.300 align:middle line:90% OK. 00:09:25.300 --> 00:09:28.060 align:middle line:84% So now what I want to do is move into matrix land. 00:09:28.060 --> 00:09:29.020 align:middle line:90% We want to grow up. 00:09:29.020 --> 00:09:30.370 align:middle line:90% This was simple circles. 00:09:30.370 --> 00:09:36.520 align:middle line:90% Let's go into matrix land. 00:09:36.520 --> 00:09:40.802 align:middle line:90% And wait, where are we going? 00:09:40.802 --> 00:09:41.760 align:middle line:90% What are we doing here? 00:09:41.760 --> 00:09:42.635 align:middle line:90% Let me just remember. 00:09:42.635 --> 00:09:47.560 align:middle line:90% 00:09:47.560 --> 00:09:48.190 align:middle line:90% Oh, right. 00:09:48.190 --> 00:09:49.490 align:middle line:90% OK. 00:09:49.490 --> 00:09:51.620 align:middle line:90% So yeah. 00:09:51.620 --> 00:09:53.670 align:middle line:84% So we're taking another baby step 00:09:53.670 --> 00:09:55.530 align:middle line:90% into matrix land for the moment. 00:09:55.530 --> 00:09:56.030 align:middle line:90% All right. 00:09:56.030 --> 00:10:00.410 align:middle line:84% So I'm going to restrict myself to symmetric matrices-- 00:10:00.410 --> 00:10:04.310 align:middle line:84% not for any particular reason, but just for simplicity. 00:10:04.310 --> 00:10:10.400 align:middle line:84% And I'm going to look at essentially the gradient 00:10:10.400 --> 00:10:13.340 align:middle line:84% of the quadratic form x transpose ax, which we've seen 00:10:13.340 --> 00:10:15.140 align:middle line:90% enough times in this class. 00:10:15.140 --> 00:10:22.430 align:middle line:84% And as you know, the gradient is basically ax, right? 00:10:22.430 --> 00:10:24.230 align:middle line:90% And you've seen that before. 00:10:24.230 --> 00:10:27.050 align:middle line:84% But now what I'd like to do is, ask 00:10:27.050 --> 00:10:30.950 align:middle line:84% what happens if you take the gradient of that very 00:10:30.950 --> 00:10:34.220 align:middle line:84% same quadratic form, but we restrict ourselves 00:10:34.220 --> 00:10:36.350 align:middle line:90% to being on the sphere, right? 00:10:36.350 --> 00:10:40.264 align:middle line:84% So what kind of change does that happen? 00:10:40.264 --> 00:10:46.070 align:middle line:84% So to do that, here's kind of the trick. 00:10:46.070 --> 00:10:50.570 align:middle line:84% We want to use the fact that the dx's we're 00:10:50.570 --> 00:10:52.880 align:middle line:90% allowing are orthogonal to x. 00:10:52.880 --> 00:10:56.820 align:middle line:84% Those are the only dx's that we want to allow now. 00:10:56.820 --> 00:11:01.100 align:middle line:84% And one way to do that, one way to kind of say the same thing 00:11:01.100 --> 00:11:04.580 align:middle line:84% is that, if you use this projection matrix which 00:11:04.580 --> 00:11:09.840 align:middle line:84% projects orthogonal to x, then within this projection matrix 00:11:09.840 --> 00:11:11.570 align:middle line:90% times dx is still dx, right? 00:11:11.570 --> 00:11:14.103 align:middle line:90% So if you're-- right? 00:11:14.103 --> 00:11:15.020 align:middle line:90% So here's the picture. 00:11:15.020 --> 00:11:15.980 align:middle line:90% I'll just do it with my hands. 00:11:15.980 --> 00:11:17.060 align:middle line:90% Here's your radius. 00:11:17.060 --> 00:11:20.210 align:middle line:84% Here's the plane perpendicular to the radius. 00:11:20.210 --> 00:11:21.215 align:middle line:90% And I want to project-- 00:11:21.215 --> 00:11:23.090 align:middle line:84% of course, the plane goes through the origin. 00:11:23.090 --> 00:11:26.070 align:middle line:84% I want to project onto that plane. 00:11:26.070 --> 00:11:28.605 align:middle line:84% And if I'm orthogonal, that's kind of a NOOP, right? 00:11:28.605 --> 00:11:30.230 align:middle line:84% If I'm in the radial direction, it's 0. 00:11:30.230 --> 00:11:32.870 align:middle line:84% And if I'm orthogonal to the radial direction, 00:11:32.870 --> 00:11:35.160 align:middle line:90% it doesn't change anything. 00:11:35.160 --> 00:11:38.990 align:middle line:84% And so one way to describe the perturbations that 00:11:38.990 --> 00:11:43.020 align:middle line:90% are tangential is in this way. 00:11:43.020 --> 00:11:47.630 align:middle line:84% And so if you now calculate x transpose adx, 00:11:47.630 --> 00:11:51.020 align:middle line:84% I get to put this projection matrix in, right? 00:11:51.020 --> 00:11:53.990 align:middle line:84% And then I do a little bit of manipulation. 00:11:53.990 --> 00:11:58.460 align:middle line:84% And then what you see is that the gradient is now 00:11:58.460 --> 00:12:00.680 align:middle line:90% the projection of ax, right? 00:12:00.680 --> 00:12:02.990 align:middle line:84% And so if you're moving around the sphere, 00:12:02.990 --> 00:12:05.780 align:middle line:84% this is the way to actually get the gradient. 00:12:05.780 --> 00:12:09.680 align:middle line:84% Because we want the gradient to live on that tangent space 00:12:09.680 --> 00:12:11.730 align:middle line:84% to be perpendicular to the radius. 00:12:11.730 --> 00:12:16.580 align:middle line:84% So what's really going on could be said more simply perhaps, 00:12:16.580 --> 00:12:19.280 align:middle line:84% which is, in general, here you are on the sphere. 00:12:19.280 --> 00:12:21.650 align:middle line:84% And you want to figure out the best direction 00:12:21.650 --> 00:12:25.312 align:middle line:84% to go in to minimize that quadratic form. 00:12:25.312 --> 00:12:27.020 align:middle line:84% But you're only restricted to the sphere. 00:12:27.020 --> 00:12:29.900 align:middle line:84% So go ahead and project that gradient down to the sphere. 00:12:29.900 --> 00:12:31.530 align:middle line:84% That's really all that's happening. 00:12:31.530 --> 00:12:33.990 align:middle line:84% But I just thought I'd show it to you formally. 00:12:33.990 --> 00:12:34.490 align:middle line:90% OK? 00:12:34.490 --> 00:12:37.065 align:middle line:90% So what did we just do? 00:12:37.065 --> 00:12:37.940 align:middle line:90% We needed two things. 00:12:37.940 --> 00:12:39.648 align:middle line:84% We needed a linearization of the function 00:12:39.648 --> 00:12:41.630 align:middle line:84% that's correct on tangents and a direction 00:12:41.630 --> 00:12:42.800 align:middle line:90% that itself is tangent. 00:12:42.800 --> 00:12:45.020 align:middle line:84% That's kind of what we just did to be 00:12:45.020 --> 00:12:46.980 align:middle line:90% able to do this sort of thing. 00:12:46.980 --> 00:12:47.850 align:middle line:90% OK? 00:12:47.850 --> 00:12:50.610 align:middle line:90% Here's a-- in general-- 00:12:50.610 --> 00:12:52.750 align:middle line:90% this wouldn't surprise anybody. 00:12:52.750 --> 00:12:55.800 align:middle line:84% But if you had some general function on the sphere, 00:12:55.800 --> 00:12:58.080 align:middle line:84% then, again, you would end up just 00:12:58.080 --> 00:13:01.380 align:middle line:84% projecting the original gradient so 00:13:01.380 --> 00:13:05.530 align:middle line:90% that it's in the tangent space. 00:13:05.530 --> 00:13:06.030 align:middle line:90% OK. 00:13:06.030 --> 00:13:07.890 align:middle line:84% Let me get to the thing that's more fun. 00:13:07.890 --> 00:13:09.960 align:middle line:90% Let's go to real matrices. 00:13:09.960 --> 00:13:12.940 align:middle line:84% Let's differentiate orthogonal matrices. 00:13:12.940 --> 00:13:13.440 align:middle line:90% OK? 00:13:13.440 --> 00:13:14.740 align:middle line:90% That's the more fun part. 00:13:14.740 --> 00:13:17.830 align:middle line:84% And so this is where I want to go now. 00:13:17.830 --> 00:13:18.330 align:middle line:90% OK. 00:13:18.330 --> 00:13:19.990 align:middle line:90% So what am I doing here? 00:13:19.990 --> 00:13:24.480 align:middle line:90% So let's see. 00:13:24.480 --> 00:13:28.940 align:middle line:84% So what I wanted this to look like is 00:13:28.940 --> 00:13:31.700 align:middle line:84% a complete analog of what I just did on the sphere, right? 00:13:31.700 --> 00:13:35.090 align:middle line:84% I just want to do it in this bigger orthogonal matrix land. 00:13:35.090 --> 00:13:40.070 align:middle line:84% So instead of taking a vector, I'm now going to get a 5-by-5 00:13:40.070 --> 00:13:42.470 align:middle line:84% random matrix with standard normal entries 00:13:42.470 --> 00:13:47.540 align:middle line:84% and a perturbation that's also a 5-by-5 matrix with my 0, 0, 00:13:47.540 --> 00:13:48.440 align:middle line:90% 1's. 00:13:48.440 --> 00:13:50.480 align:middle line:90% But what I'd like to do is-- 00:13:50.480 --> 00:13:52.340 align:middle line:84% I don't want to work with general matrices. 00:13:52.340 --> 00:13:54.750 align:middle line:84% I want to work with orthogonal matrices. 00:13:54.750 --> 00:13:57.590 align:middle line:84% So one quick way to get your hands 00:13:57.590 --> 00:14:02.150 align:middle line:84% on a random orthogonal matrix is to do the QR factorization of a 00:14:02.150 --> 00:14:04.022 align:middle line:90% and grab the q component. 00:14:04.022 --> 00:14:05.480 align:middle line:84% AUDIENCE: That's not quite uniform. 00:14:05.480 --> 00:14:06.680 align:middle line:84% ALAN EDELMAN: Not quite uniform-- yeah, I 00:14:06.680 --> 00:14:07.340 align:middle line:90% have written that down. 00:14:07.340 --> 00:14:09.110 align:middle line:84% I've written a whole thing about that. 00:14:09.110 --> 00:14:11.723 align:middle line:84% But if you randomize the signs, then it would be. 00:14:11.723 --> 00:14:12.890 align:middle line:90% Yeah, there's a whole story. 00:14:12.890 --> 00:14:16.010 align:middle line:84% I contacted the LAPACK people years ago about that-- yeah, 00:14:16.010 --> 00:14:17.480 align:middle line:90% not quite uniform among-- 00:14:17.480 --> 00:14:19.240 align:middle line:84% so there's this thing called Haar measure 00:14:19.240 --> 00:14:21.740 align:middle line:84% on the orthogonal group, which is the uniform measure that's 00:14:21.740 --> 00:14:25.540 align:middle line:84% invariant not quite because of certain reasons. 00:14:25.540 --> 00:14:28.080 align:middle line:84% But it doesn't matter for this purpose, right? 00:14:28.080 --> 00:14:31.230 align:middle line:84% So q will be random, just not uniformly random, right? 00:14:31.230 --> 00:14:35.010 align:middle line:84% And then this dq will be a little bit of a change. 00:14:35.010 --> 00:14:35.520 align:middle line:90% OK? 00:14:35.520 --> 00:14:40.320 align:middle line:84% And what I want to do is look at the same kind of thing 00:14:40.320 --> 00:14:41.550 align:middle line:90% we looked at before. 00:14:41.550 --> 00:14:45.730 align:middle line:84% What is the relationship between dq and q? 00:14:45.730 --> 00:14:49.200 align:middle line:84% On the sphere, it was orthogonality. 00:14:49.200 --> 00:14:52.710 align:middle line:90% And if you look at-- 00:14:52.710 --> 00:14:55.530 align:middle line:84% I'm going to divide by roughly the size of my da 00:14:55.530 --> 00:14:57.600 align:middle line:90% there so that we can see. 00:14:57.600 --> 00:15:01.557 align:middle line:84% And I wonder, if you don't know what q transpose dq is-- 00:15:01.557 --> 00:15:04.140 align:middle line:84% if you've known this before, I don't want you to say anything. 00:15:04.140 --> 00:15:07.500 align:middle line:84% But can anybody look at these numbers and guess 00:15:07.500 --> 00:15:12.270 align:middle line:84% what is going to be the differential version of being 00:15:12.270 --> 00:15:14.310 align:middle line:90% tangent to the-- 00:15:14.310 --> 00:15:17.010 align:middle line:84% what does it mean to be tangent to an orthogonal matrix, right? 00:15:17.010 --> 00:15:19.930 align:middle line:84% What's a small change to an orthogonal matrix? 00:15:19.930 --> 00:15:24.240 align:middle line:84% So what do you see when you look at this 5-by-5 set of numbers? 00:15:24.240 --> 00:15:28.020 align:middle line:84% What do you think it's trying to point you towards? 00:15:28.020 --> 00:15:32.500 align:middle line:84% So it's obviously not the zero matrix. 00:15:32.500 --> 00:15:34.308 align:middle line:84% So-- yeah, what do you think you see? 00:15:34.308 --> 00:15:35.350 align:middle line:90% AUDIENCE: Skew symmetric. 00:15:35.350 --> 00:15:36.160 align:middle line:90% ALAN EDELMAN: Skew symmetric. 00:15:36.160 --> 00:15:37.240 align:middle line:90% That's exactly right. 00:15:37.240 --> 00:15:39.490 align:middle line:90% So the diagonal wants to be 0. 00:15:39.490 --> 00:15:41.890 align:middle line:84% That's all those 10 to the minus 6 and 1/7. 00:15:41.890 --> 00:15:45.100 align:middle line:84% And if you look, for example, at this entry versus this entry, 00:15:45.100 --> 00:15:48.400 align:middle line:84% you could see that they're the opposite sign 00:15:48.400 --> 00:15:49.840 align:middle line:84% with the same number, essentially. 00:15:49.840 --> 00:15:54.010 align:middle line:84% So yes, this-- and so what this little numerical experiment 00:15:54.010 --> 00:15:56.050 align:middle line:84% is showing you is that, if you want 00:15:56.050 --> 00:15:59.080 align:middle line:84% to make a small change to a matrix q, 00:15:59.080 --> 00:16:02.830 align:middle line:84% if you make a small change dq, the constraint 00:16:02.830 --> 00:16:05.950 align:middle line:84% of being orthogonal at the differential level 00:16:05.950 --> 00:16:08.620 align:middle line:84% is that q transpose dq will be skew symmetric, 00:16:08.620 --> 00:16:11.140 align:middle line:90% or antisymmetric, right? 00:16:11.140 --> 00:16:17.320 align:middle line:84% And of course, we could do this with-- here's the proof. 00:16:17.320 --> 00:16:18.910 align:middle line:84% The proof is almost the same proof 00:16:18.910 --> 00:16:22.180 align:middle line:90% as when you're on the sphere-- 00:16:22.180 --> 00:16:24.830 align:middle line:84% that, let's just differentiate the constraint. 00:16:24.830 --> 00:16:28.330 align:middle line:84% So this is the very definition of a matrix being orthogonal. 00:16:28.330 --> 00:16:29.710 align:middle line:90% q transpose q is the identity. 00:16:29.710 --> 00:16:33.980 align:middle line:84% That is the entire constraint of being orthogonal. 00:16:33.980 --> 00:16:36.590 align:middle line:84% And so if you differentiate, you get 00:16:36.590 --> 00:16:40.140 align:middle line:84% q transpose dq plus dq transpose q equals 0. 00:16:40.140 --> 00:16:40.640 align:middle line:90% OK? 00:16:40.640 --> 00:16:48.620 align:middle line:84% But now you don't really get to combine these 00:16:48.620 --> 00:16:50.540 align:middle line:90% like we did on the sphere. 00:16:50.540 --> 00:16:51.040 align:middle line:90% OK? 00:16:51.040 --> 00:16:52.840 align:middle line:84% But what we can notice is that this 00:16:52.840 --> 00:16:58.120 align:middle line:84% is the same thing as saying that this plus its transpose is 0. 00:16:58.120 --> 00:16:58.620 align:middle line:90% OK? 00:16:58.620 --> 00:17:01.500 align:middle line:84% But that's just saying the very definition 00:17:01.500 --> 00:17:03.540 align:middle line:84% of being antisymmetric, or skew symmetric, 00:17:03.540 --> 00:17:05.790 align:middle line:84% is that a matrix plus its transpose is 0. 00:17:05.790 --> 00:17:10.750 align:middle line:84% And so therefore, q transpose is an antisymmetric differential. 00:17:10.750 --> 00:17:11.250 align:middle line:90% OK? 00:17:11.250 --> 00:17:13.109 align:middle line:84% Now, if "antisymmetric differential" 00:17:13.109 --> 00:17:16.810 align:middle line:84% sounds like a weird thing to say, from my point of view, 00:17:16.810 --> 00:17:18.630 align:middle line:84% from a practical point of view, it's 00:17:18.630 --> 00:17:21.780 align:middle line:84% just like if you-- in the limit of making little perturbations, 00:17:21.780 --> 00:17:24.532 align:middle line:84% q transpose dq will be a little antisymmetric matrix, right? 00:17:24.532 --> 00:17:26.490 align:middle line:84% And then you don't have to worry about it being 00:17:26.490 --> 00:17:28.770 align:middle line:90% an antisymmetric differential. 00:17:28.770 --> 00:17:30.270 align:middle line:90% OK? 00:17:30.270 --> 00:17:33.780 align:middle line:90% Let's talk about this. 00:17:33.780 --> 00:17:36.050 align:middle line:84% You all know that if you're in n dimensions 00:17:36.050 --> 00:17:40.160 align:middle line:84% and I ask you what is the dimensionality of the-- let's 00:17:40.160 --> 00:17:41.330 align:middle line:90% talk about the sphere first. 00:17:41.330 --> 00:17:43.460 align:middle line:84% If I have a sphere in n dimensions, 00:17:43.460 --> 00:17:46.280 align:middle line:84% what is the dimensionality of the sphere itself? 00:17:46.280 --> 00:17:49.240 align:middle line:90% 00:17:49.240 --> 00:17:53.230 align:middle line:84% Like, do you understand the question? 00:17:53.230 --> 00:17:53.730 align:middle line:90% Yeah. 00:17:53.730 --> 00:17:55.800 align:middle line:84% So yeah, the surface of the sphere. 00:17:55.800 --> 00:17:59.310 align:middle line:84% And so in 3 dimensions, the surface is two, right? 00:17:59.310 --> 00:18:00.840 align:middle line:84% When you look at the world locally, 00:18:00.840 --> 00:18:02.370 align:middle line:90% it's kind of flat, right? 00:18:02.370 --> 00:18:04.710 align:middle line:84% I mean, if the world were a perfect sphere 00:18:04.710 --> 00:18:07.662 align:middle line:84% and you only could look at a small distance 00:18:07.662 --> 00:18:09.870 align:middle line:84% from where you are, right, it looks like a flat plane 00:18:09.870 --> 00:18:10.800 align:middle line:90% locally, right? 00:18:10.800 --> 00:18:13.480 align:middle line:84% I mean, it curves but from a small-- 00:18:13.480 --> 00:18:16.413 align:middle line:84% so in n dimensions, what's the dimensionality of a sphere? 00:18:16.413 --> 00:18:17.380 align:middle line:90% AUDIENCE: n minus 1. 00:18:17.380 --> 00:18:18.220 align:middle line:90% ALAN EDELMAN: n minus 1, right? 00:18:18.220 --> 00:18:19.387 align:middle line:90% There's only one constraint. 00:18:19.387 --> 00:18:22.370 align:middle line:84% And so that takes you down from n dimensions to n minus 1, 00:18:22.370 --> 00:18:22.870 align:middle line:90% right? 00:18:22.870 --> 00:18:25.240 align:middle line:84% The only constraint is that the sum of squares is 1. 00:18:25.240 --> 00:18:25.750 align:middle line:90% OK? 00:18:25.750 --> 00:18:27.730 align:middle line:90% So let's now talk about-- 00:18:27.730 --> 00:18:32.140 align:middle line:90% 00:18:32.140 --> 00:18:34.450 align:middle line:84% generally speaking, n-by-n matrices 00:18:34.450 --> 00:18:37.270 align:middle line:84% live in what dimensional space totally? 00:18:37.270 --> 00:18:39.310 align:middle line:84% Just general n-by-n matrices for starters 00:18:39.310 --> 00:18:42.160 align:middle line:90% just to get our feet wet. 00:18:42.160 --> 00:18:43.120 align:middle line:90% n-squared, right? 00:18:43.120 --> 00:18:45.710 align:middle line:84% So every matrix is a point in n-squared dimensional space, 00:18:45.710 --> 00:18:46.210 align:middle line:90% right? 00:18:46.210 --> 00:18:48.840 align:middle line:84% Mathematicians have no trouble with high dimensions. 00:18:48.840 --> 00:18:50.500 align:middle line:90% We talk about it all the time. 00:18:50.500 --> 00:18:52.660 align:middle line:84% We don't get into philosophical discussions 00:18:52.660 --> 00:18:53.920 align:middle line:90% about time or anything, right? 00:18:53.920 --> 00:18:56.390 align:middle line:84% We just write down n-squared dimensional space. 00:18:56.390 --> 00:18:56.890 align:middle line:90% OK? 00:18:56.890 --> 00:18:59.470 align:middle line:84% Now, the set of orthogonal matrices 00:18:59.470 --> 00:19:01.840 align:middle line:84% is some sort of blob in that space, right? 00:19:01.840 --> 00:19:05.270 align:middle line:84% It may be hard to imagine, but there's some constraints. 00:19:05.270 --> 00:19:08.860 align:middle line:84% And so now my question is, what is the dimension of that? 00:19:08.860 --> 00:19:13.418 align:middle line:84% Anybody want to attack that right now or want 00:19:13.418 --> 00:19:14.710 align:middle line:90% to-- maybe we should go slowly. 00:19:14.710 --> 00:19:15.320 align:middle line:90% Let's see. 00:19:15.320 --> 00:19:20.670 align:middle line:84% So the 2-by-2 orthogonal matrices-- 00:19:20.670 --> 00:19:22.690 align:middle line:90% how many parameters are there? 00:19:22.690 --> 00:19:25.480 align:middle line:84% So let's just focus on rotations for starters. 00:19:25.480 --> 00:19:27.920 align:middle line:84% So we're talking about rotations and reflections. 00:19:27.920 --> 00:19:32.560 align:middle line:84% How many parameters for rotations? 00:19:32.560 --> 00:19:35.680 align:middle line:84% So we're in four-dimensional space-- all possible matrices. 00:19:35.680 --> 00:19:39.670 align:middle line:84% And then we're looking at rotations only as 00:19:39.670 --> 00:19:41.860 align:middle line:84% some sort of blob in four-dimensional space. 00:19:41.860 --> 00:19:43.150 align:middle line:90% Dimensionality? 00:19:43.150 --> 00:19:43.750 align:middle line:90% One, right? 00:19:43.750 --> 00:19:45.640 align:middle line:84% Just the angle of rotation, right? 00:19:45.640 --> 00:19:46.300 align:middle line:90% So that's one. 00:19:46.300 --> 00:19:48.670 align:middle line:84% There's the cosine theta sine theta. 00:19:48.670 --> 00:19:52.400 align:middle line:84% Reflections also are kind of like that. 00:19:52.400 --> 00:19:57.370 align:middle line:84% So for 2-by-2 matrices, it's one. 00:19:57.370 --> 00:20:00.880 align:middle line:84% For 3-by-3 matrices-- people who fly airplanes know the answer 00:20:00.880 --> 00:20:03.890 align:middle line:90% to this, believe it or not. 00:20:03.890 --> 00:20:06.470 align:middle line:84% Do you know how many parameters describe an orthogonal matrix 00:20:06.470 --> 00:20:07.280 align:middle line:90% in-- 00:20:07.280 --> 00:20:12.920 align:middle line:84% I mean, you don't need all nine numbers of a-- right? 00:20:12.920 --> 00:20:14.490 align:middle line:90% There's constraints. 00:20:14.490 --> 00:20:15.730 align:middle line:90% Do you want to guess? 00:20:15.730 --> 00:20:16.558 align:middle line:90% AUDIENCE: 2. 00:20:16.558 --> 00:20:17.600 align:middle line:90% ALAN EDELMAN: It's not 2. 00:20:17.600 --> 00:20:20.220 align:middle line:90% 00:20:20.220 --> 00:20:22.830 align:middle line:90% So it's not 6, but 6 is-- 00:20:22.830 --> 00:20:24.870 align:middle line:84% I feel like 6 is on the right track in a sort 00:20:24.870 --> 00:20:26.280 align:middle line:90% of backwards kind of way. 00:20:26.280 --> 00:20:27.320 align:middle line:90% AUDIENCE: Is it 4? 00:20:27.320 --> 00:20:27.990 align:middle line:90% ALAN EDELMAN: 4? 00:20:27.990 --> 00:20:28.850 align:middle line:90% We're running out of numbers. 00:20:28.850 --> 00:20:29.720 align:middle line:90% No, it's not 4. 00:20:29.720 --> 00:20:30.470 align:middle line:90% AUDIENCE: Is it 3? 00:20:30.470 --> 00:20:31.370 align:middle line:90% ALAN EDELMAN: 3. 00:20:31.370 --> 00:20:32.300 align:middle line:90% A good guesser here. 00:20:32.300 --> 00:20:33.590 align:middle line:90% Give this man a prize. 00:20:33.590 --> 00:20:37.130 align:middle line:84% So roll, pitch, and yaw are what the airplane people 00:20:37.130 --> 00:20:38.030 align:middle line:90% seem to know. 00:20:38.030 --> 00:20:42.920 align:middle line:84% That's what they've named it for the 3. 00:20:42.920 --> 00:20:44.880 align:middle line:90% So the general answer-- 00:20:44.880 --> 00:20:47.193 align:middle line:84% anybody want to guess the general-- oh, yeah, 00:20:47.193 --> 00:20:48.860 align:middle line:84% anybody want to guess the general answer 00:20:48.860 --> 00:20:49.880 align:middle line:90% before I show you? 00:20:49.880 --> 00:20:52.025 align:middle line:90% You've got two data points. 00:20:52.025 --> 00:20:53.690 align:middle line:84% That may be not enough to guess with. 00:20:53.690 --> 00:20:56.030 align:middle line:90% It's one for 2-by-2. 00:20:56.030 --> 00:20:59.240 align:middle line:90% And it's 3 for 3-by-3. 00:20:59.240 --> 00:21:04.440 align:middle line:84% Anybody want to-- can you name that song in how many notes? 00:21:04.440 --> 00:21:04.940 align:middle line:90% All right. 00:21:04.940 --> 00:21:07.550 align:middle line:90% I'll just kind of tell you. 00:21:07.550 --> 00:21:09.015 align:middle line:90% So-- 00:21:09.015 --> 00:21:14.280 align:middle line:84% AUDIENCE: n-squared minus n choose 2. 00:21:14.280 --> 00:21:18.255 align:middle line:84% ALAN EDELMAN: That sounds like the right backwards answer. 00:21:18.255 --> 00:21:21.780 align:middle line:90% 00:21:21.780 --> 00:21:23.510 align:middle line:90% So it's the complement of that. 00:21:23.510 --> 00:21:26.640 align:middle line:90% It's n choose 2. 00:21:26.640 --> 00:21:28.330 align:middle line:90% So somebody had said 6. 00:21:28.330 --> 00:21:29.170 align:middle line:90% You said 6. 00:21:29.170 --> 00:21:30.450 align:middle line:90% And the answer was 3. 00:21:30.450 --> 00:21:32.200 align:middle line:84% I feel like you're on the same wavelength. 00:21:32.200 --> 00:21:34.120 align:middle line:84% You said the 6 number when n equals 3. 00:21:34.120 --> 00:21:37.990 align:middle line:84% And so there's a couple of ways to see this. 00:21:37.990 --> 00:21:41.260 align:middle line:84% But in general, the answer is n times n minus 1 over 2. 00:21:41.260 --> 00:21:44.320 align:middle line:84% One way to see it is to take the n-squared free parameters 00:21:44.320 --> 00:21:49.570 align:middle line:84% and count the constraints of q transpose q as the identity, 00:21:49.570 --> 00:21:50.210 align:middle line:90% right? 00:21:50.210 --> 00:21:57.280 align:middle line:84% And the constraints-- as a picture, 00:21:57.280 --> 00:21:59.890 align:middle line:84% you could kind of count the constraints 00:21:59.890 --> 00:22:01.330 align:middle line:90% as the ones on the diagonal. 00:22:01.330 --> 00:22:03.580 align:middle line:84% The ones on the diagonals say that your sum of squares 00:22:03.580 --> 00:22:04.870 align:middle line:90% is 1, right? 00:22:04.870 --> 00:22:07.510 align:middle line:84% And then you could pick, say, the upper triangular 00:22:07.510 --> 00:22:09.460 align:middle line:84% because the lower is the same, right? 00:22:09.460 --> 00:22:11.930 align:middle line:84% This one tells you that the i'th column is-- 00:22:11.930 --> 00:22:12.430 align:middle line:90% right? 00:22:12.430 --> 00:22:14.138 align:middle line:84% This one says the i'th column has norm 1. 00:22:14.138 --> 00:22:16.760 align:middle line:84% This one says the i and jth column are orthogonal, right? 00:22:16.760 --> 00:22:19.570 align:middle line:84% And so the total number is sort of an upper-- 00:22:19.570 --> 00:22:20.380 align:middle line:90% oh, wait. 00:22:20.380 --> 00:22:23.530 align:middle line:84% Yeah, the number of constraints is n times n plus 1 over 2. 00:22:23.530 --> 00:22:26.020 align:middle line:84% And so you have to subtract that from n-squared. 00:22:26.020 --> 00:22:32.590 align:middle line:84% And you get-- what's left is these numbers here, right? 00:22:32.590 --> 00:22:34.720 align:middle line:84% This is n times n minus 1 over 2-- or n choose 2, 00:22:34.720 --> 00:22:35.990 align:middle line:90% as this man was saying. 00:22:35.990 --> 00:22:36.490 align:middle line:90% OK? 00:22:36.490 --> 00:22:39.113 align:middle line:90% So that's one way to count it. 00:22:39.113 --> 00:22:40.780 align:middle line:84% Another way to count it-- there are lots 00:22:40.780 --> 00:22:42.130 align:middle line:90% of ways to count it, actually. 00:22:42.130 --> 00:22:44.630 align:middle line:84% You could think about the QR factorization. 00:22:44.630 --> 00:22:46.930 align:middle line:84% And the R, which is upper triangular, 00:22:46.930 --> 00:22:49.510 align:middle line:84% has the same n times n plus 1 over 2. 00:22:49.510 --> 00:22:52.600 align:middle line:84% Leaving n times n minus 1 over 2 free parameters for the q 00:22:52.600 --> 00:22:55.450 align:middle line:90% is another way to say that. 00:22:55.450 --> 00:22:58.430 align:middle line:84% You could also think about the symmetric eigenvalue problem, 00:22:58.430 --> 00:23:03.080 align:middle line:84% if you like, where a symmetric matrix has n times n plus 1 00:23:03.080 --> 00:23:03.980 align:middle line:90% over 2. 00:23:03.980 --> 00:23:06.450 align:middle line:84% And the eigenvalues eat up n of them. 00:23:06.450 --> 00:23:10.110 align:middle line:84% And so now you have n times n minus 1 over 2. 00:23:10.110 --> 00:23:12.930 align:middle line:84% So there are a lot of ways to-- you could also look at the SVD. 00:23:12.930 --> 00:23:14.550 align:middle line:90% That's another way to see it. 00:23:14.550 --> 00:23:17.220 align:middle line:84% For square matrices, this is n-squared. 00:23:17.220 --> 00:23:18.720 align:middle line:90% This one has n. 00:23:18.720 --> 00:23:20.490 align:middle line:84% And so these two orthogonal matrices 00:23:20.490 --> 00:23:22.740 align:middle line:84% each have n squared minus n over 2, right? 00:23:22.740 --> 00:23:24.240 align:middle line:84% So there's lots and lots of fun ways 00:23:24.240 --> 00:23:28.060 align:middle line:84% to check that this is the right answer. 00:23:28.060 --> 00:23:28.560 align:middle line:90% OK? 00:23:28.560 --> 00:23:30.390 align:middle line:84% So now what I want to do is sort of talk 00:23:30.390 --> 00:23:34.100 align:middle line:84% about the subject of this class, which is to differentiate 00:23:34.100 --> 00:23:35.100 align:middle line:90% matrix functions, right? 00:23:35.100 --> 00:23:39.450 align:middle line:84% So this is what the buildup has been for the last 30 minutes 00:23:39.450 --> 00:23:40.120 align:middle line:90% or so. 00:23:40.120 --> 00:23:43.590 align:middle line:84% So I want to differentiate the symmetric eigenvalue problem. 00:23:43.590 --> 00:23:45.430 align:middle line:84% And I want to show you how to do it. 00:23:45.430 --> 00:23:45.930 align:middle line:90% OK? 00:23:45.930 --> 00:23:51.570 align:middle line:84% So in fact, I guess we might somewhere derive this-- 00:23:51.570 --> 00:23:53.810 align:middle line:84% you had a name I had never heard before of something 00:23:53.810 --> 00:23:54.935 align:middle line:90% that I think is much older. 00:23:54.935 --> 00:23:57.450 align:middle line:84% But the physics-- the something Feynman theorem. 00:23:57.450 --> 00:23:57.990 align:middle line:90% AUDIENCE: Hellmann-Feynman. 00:23:57.990 --> 00:23:58.770 align:middle line:90% ALAN EDELMAN: Hellmann-Feynman. 00:23:58.770 --> 00:24:01.020 align:middle line:84% I'm sure it's much older than Hellman and Feynman. 00:24:01.020 --> 00:24:03.843 align:middle line:84% So that sounds like the-- was Hellmann a physicist? 00:24:03.843 --> 00:24:05.510 align:middle line:84% AUDIENCE: I don't know who Hellmann was. 00:24:05.510 --> 00:24:06.290 align:middle line:90% ALAN EDELMAN: OK. 00:24:06.290 --> 00:24:08.510 align:middle line:90% But we all know who Feynman was. 00:24:08.510 --> 00:24:09.080 align:middle line:90% All right. 00:24:09.080 --> 00:24:15.310 align:middle line:84% So the symmetric eigenvalue problem as, you all know, 00:24:15.310 --> 00:24:19.470 align:middle line:84% d you know that the eigenvalues of a symmetric matrix are real. 00:24:19.470 --> 00:24:23.500 align:middle line:84% And the eigenvectors can be put into an orthogonal matrix, 00:24:23.500 --> 00:24:24.000 align:middle line:90% right? 00:24:24.000 --> 00:24:27.150 align:middle line:84% And so the symmetric eigenvalue problem 00:24:27.150 --> 00:24:31.050 align:middle line:84% can be written as factoring a matrix s into q lambda q 00:24:31.050 --> 00:24:32.260 align:middle line:90% transpose. 00:24:32.260 --> 00:24:32.760 align:middle line:90% OK? 00:24:32.760 --> 00:24:35.460 align:middle line:90% And so we can differentiate. 00:24:35.460 --> 00:24:36.870 align:middle line:90% And we get the three terms. 00:24:36.870 --> 00:24:39.640 align:middle line:90% This is just the product rule. 00:24:39.640 --> 00:24:43.900 align:middle line:84% And it turns out to be handy to kind of spin around 00:24:43.900 --> 00:24:44.995 align:middle line:90% the differential. 00:24:44.995 --> 00:24:47.950 align:middle line:84% It kind of makes you look like you're at a diagonal matrix. 00:24:47.950 --> 00:24:50.320 align:middle line:90% And if you spin it around-- 00:24:50.320 --> 00:24:53.170 align:middle line:84% I'm just putting q transpose on the left and q on the right. 00:24:53.170 --> 00:24:56.200 align:middle line:84% Let me move it up to make sure everybody can see this. 00:24:56.200 --> 00:25:00.100 align:middle line:90% But yeah. 00:25:00.100 --> 00:25:01.490 align:middle line:90% Let's move everything up. 00:25:01.490 --> 00:25:01.990 align:middle line:90% Yeah. 00:25:01.990 --> 00:25:11.700 align:middle line:84% So this rotated version of the change to your symmetric matrix 00:25:11.700 --> 00:25:16.740 align:middle line:84% is q transpose dq lambda minus lambda q transpose dq plus d 00:25:16.740 --> 00:25:17.380 align:middle line:90% lambda. 00:25:17.380 --> 00:25:17.880 align:middle line:90% OK? 00:25:17.880 --> 00:25:20.730 align:middle line:90% So that's the derivative of-- 00:25:20.730 --> 00:25:24.060 align:middle line:84% so what this says is, if I perturb my symmetric matrix 00:25:24.060 --> 00:25:28.830 align:middle line:84% this much, then my eigenvectors will be perturbed by this much. 00:25:28.830 --> 00:25:31.540 align:middle line:84% And my eigenvalues will be perturbed this much. 00:25:31.540 --> 00:25:34.500 align:middle line:90% And you know me by now. 00:25:34.500 --> 00:25:36.570 align:middle line:84% I don't believe any theorem unless I can check it 00:25:36.570 --> 00:25:38.130 align:middle line:90% numerically, right? 00:25:38.130 --> 00:25:40.620 align:middle line:84% So without a computer, I wouldn't be able to do math. 00:25:40.620 --> 00:25:42.400 align:middle line:90% So let's do that. 00:25:42.400 --> 00:25:44.790 align:middle line:84% Let's create a random 5-by-5 matrix and also 00:25:44.790 --> 00:25:46.170 align:middle line:90% a random perturbation. 00:25:46.170 --> 00:25:49.200 align:middle line:84% But what I'd like to do is I want to symmetrize. 00:25:49.200 --> 00:25:50.670 align:middle line:90% So I'm going to call this s. 00:25:50.670 --> 00:25:53.628 align:middle line:84% And I'll also symmetrize my perturbation. 00:25:53.628 --> 00:25:55.170 align:middle line:84% And I'm just going to do the obvious. 00:25:55.170 --> 00:25:57.720 align:middle line:84% I'm going to take the eigendecomposition of s 00:25:57.720 --> 00:26:00.690 align:middle line:84% and the eigendecomposition of my perturbed s. 00:26:00.690 --> 00:26:02.400 align:middle line:90% And so here's my eigenvalues. 00:26:02.400 --> 00:26:04.390 align:middle line:84% And here's the perturbed eigenvalues. 00:26:04.390 --> 00:26:08.320 align:middle line:84% So let's call dq to be the perturbed eigenvectors. d 00:26:08.320 --> 00:26:11.410 align:middle line:84% lambda is the perturbed eigenvalues. 00:26:11.410 --> 00:26:15.040 align:middle line:84% And let's just do a comparison of-- 00:26:15.040 --> 00:26:15.590 align:middle line:90% let's see. 00:26:15.590 --> 00:26:17.147 align:middle line:90% So how do I see the comparison? 00:26:17.147 --> 00:26:18.730 align:middle line:84% Did I stack them on top of each other? 00:26:18.730 --> 00:26:19.300 align:middle line:90% Yes. 00:26:19.300 --> 00:26:21.065 align:middle line:84% I was stacking them on top of each other. 00:26:21.065 --> 00:26:22.690 align:middle line:84% I'm not sure why I did that, but I did. 00:26:22.690 --> 00:26:24.950 align:middle line:84% I stacked them on top of each other. 00:26:24.950 --> 00:26:28.270 align:middle line:84% So you should be able to see that, if I look at q transpose 00:26:28.270 --> 00:26:34.000 align:middle line:84% dq versus the math thing that I say it has to equal, 00:26:34.000 --> 00:26:37.150 align:middle line:84% I guess you could see that if you look at the top five rows-- 00:26:37.150 --> 00:26:38.945 align:middle line:84% what would happen if I didn't stack them 00:26:38.945 --> 00:26:39.820 align:middle line:90% on top of each other? 00:26:39.820 --> 00:26:43.620 align:middle line:90% What if I just went comma? 00:26:43.620 --> 00:26:45.150 align:middle line:90% Would that not be easy? 00:26:45.150 --> 00:26:46.620 align:middle line:90% Maybe 5-by-5 is too big. 00:26:46.620 --> 00:26:47.980 align:middle line:90% Maybe that's why I did that. 00:26:47.980 --> 00:26:51.820 align:middle line:84% I just want to see why I did that. 00:26:51.820 --> 00:26:54.220 align:middle line:90% Oh, that's a bad-- 00:26:54.220 --> 00:26:56.530 align:middle line:90% and if I do this, that's good. 00:26:56.530 --> 00:26:57.520 align:middle line:90% That's better. 00:26:57.520 --> 00:26:58.850 align:middle line:90% All right Let's do it that way. 00:26:58.850 --> 00:26:59.350 align:middle line:90% OK. 00:26:59.350 --> 00:27:00.760 align:middle line:84% I think that's easier on the eye. 00:27:00.760 --> 00:27:03.100 align:middle line:84% So yeah, you can see these two matrices 00:27:03.100 --> 00:27:06.010 align:middle line:90% are to enough digits the same. 00:27:06.010 --> 00:27:08.920 align:middle line:84% And you can believe the math right now, right? 00:27:08.920 --> 00:27:11.440 align:middle line:84% So I perturbed my symmetric matrix. 00:27:11.440 --> 00:27:12.370 align:middle line:90% And I look at this. 00:27:12.370 --> 00:27:13.870 align:middle line:84% And I have an identity that connects 00:27:13.870 --> 00:27:15.430 align:middle line:84% the perturbation of the eigenvalues 00:27:15.430 --> 00:27:18.340 align:middle line:84% and the perturbation of the eigenvectors. 00:27:18.340 --> 00:27:22.360 align:middle line:84% And that Hellmann-Feynman theorem 00:27:22.360 --> 00:27:26.330 align:middle line:84% is just the diagonal part of this equation here. 00:27:26.330 --> 00:27:29.860 align:middle line:84% So the Hellmann-Feynman theorem would have said-- 00:27:29.860 --> 00:27:32.410 align:middle line:84% and I'll just write it in this notation. 00:27:32.410 --> 00:27:37.430 align:middle line:84% But the Hellmann-Feynman theorem is the diagonal, 00:27:37.430 --> 00:27:43.270 align:middle line:84% which says that essentially qi transpose dsqi-- 00:27:43.270 --> 00:27:47.390 align:middle line:84% so qi being the eigenvector-- is the diagonal. 00:27:47.390 --> 00:27:48.430 align:middle line:90% So tell me. 00:27:48.430 --> 00:27:50.410 align:middle line:90% There's three terms there. 00:27:50.410 --> 00:27:52.180 align:middle line:90% There's three terms. 00:27:52.180 --> 00:27:53.170 align:middle line:90% What is the diagonal? 00:27:53.170 --> 00:27:55.740 align:middle line:90% 00:27:55.740 --> 00:27:58.700 align:middle line:84% So what's the diagonal of the first term? 00:27:58.700 --> 00:28:02.450 align:middle line:84% So capital lambda is a diagonal matrix, right? 00:28:02.450 --> 00:28:05.000 align:middle line:84% q transpose dq you tell me is an antisymmetric 00:28:05.000 --> 00:28:06.570 align:middle line:90% or a skew symmetric matrix. 00:28:06.570 --> 00:28:08.330 align:middle line:84% So what's the diagonal of that term 00:28:08.330 --> 00:28:09.680 align:middle line:90% there, q transpose dq lambda? 00:28:09.680 --> 00:28:13.020 align:middle line:90% 00:28:13.020 --> 00:28:14.587 align:middle line:84% Well, let me ask an easier question. 00:28:14.587 --> 00:28:16.170 align:middle line:84% What's the diagonal of q transpose dq? 00:28:16.170 --> 00:28:19.680 align:middle line:90% 00:28:19.680 --> 00:28:22.930 align:middle line:90% That's an easy question. 00:28:22.930 --> 00:28:23.430 align:middle line:90% AUDIENCE: 0. 00:28:23.430 --> 00:28:24.920 align:middle line:90% ALAN EDELMAN: It's 0, right? 00:28:24.920 --> 00:28:27.620 align:middle line:84% That's just the fact that any skew symmetric matrix 00:28:27.620 --> 00:28:28.980 align:middle line:90% has diagonal 0. 00:28:28.980 --> 00:28:29.480 align:middle line:90% OK? 00:28:29.480 --> 00:28:30.707 align:middle line:90% Now, if you take a-- 00:28:30.707 --> 00:28:32.540 align:middle line:84% what happens if you multiply it on the right 00:28:32.540 --> 00:28:33.433 align:middle line:90% by a diagonal matrix? 00:28:33.433 --> 00:28:35.225 align:middle line:84% What does that do to the diagonal elements? 00:28:35.225 --> 00:28:37.690 align:middle line:90% 00:28:37.690 --> 00:28:38.190 align:middle line:90% Right? 00:28:38.190 --> 00:28:39.900 align:middle line:84% So now I want to take that q transpose dq 00:28:39.900 --> 00:28:43.850 align:middle line:84% and multiply it by that capital lambda, which is diagonal. 00:28:43.850 --> 00:28:46.800 align:middle line:84% What does it do to the diagonals? 00:28:46.800 --> 00:28:49.910 align:middle line:84% So if I have an antisymmetric times diagonal, 00:28:49.910 --> 00:28:51.360 align:middle line:90% what's the diagonal of that? 00:28:51.360 --> 00:28:51.860 align:middle line:90% AUDIENCE: 0. 00:28:51.860 --> 00:28:52.790 align:middle line:84% ALAN EDELMAN: It's just 0, right? 00:28:52.790 --> 00:28:53.940 align:middle line:90% It doesn't change anything. 00:28:53.940 --> 00:28:56.600 align:middle line:84% So the only term that matters is the d lambda, right? 00:28:56.600 --> 00:28:59.510 align:middle line:84% And so basically, this is actually 00:28:59.510 --> 00:29:06.450 align:middle line:84% just saying that the change to the eigenvalue is exactly this. 00:29:06.450 --> 00:29:08.930 align:middle line:84% You just take the inner product of the change 00:29:08.930 --> 00:29:11.360 align:middle line:84% to your matrix in the direction of the eigenvector. 00:29:11.360 --> 00:29:12.560 align:middle line:90% And there you get it, right? 00:29:12.560 --> 00:29:15.380 align:middle line:84% So that's one quick derivation of that theorem. 00:29:15.380 --> 00:29:17.922 align:middle line:84% STEVEN JOHNSON: Fun fact is that the Hellmann-Feynman theorem 00:29:17.922 --> 00:29:21.350 align:middle line:84% does not require the perturbation to be symmetric. 00:29:21.350 --> 00:29:24.280 align:middle line:84% So even if ps is not symmetric, you 00:29:24.280 --> 00:29:25.560 align:middle line:90% can-- which is really useful. 00:29:25.560 --> 00:29:28.280 align:middle line:84% Because then you can take a symmetric matrix. 00:29:28.280 --> 00:29:30.322 align:middle line:84% But then you look at a non-symmetric perturbation 00:29:30.322 --> 00:29:31.738 align:middle line:84% and you can still find the effect. 00:29:31.738 --> 00:29:33.370 align:middle line:84% ALAN EDELMAN: That is a fun fact. 00:29:33.370 --> 00:29:34.838 align:middle line:90% Yeah, and of course, the same-- 00:29:34.838 --> 00:29:36.380 align:middle line:84% STEVEN JOHNSON: The two terms cancel. 00:29:36.380 --> 00:29:38.797 align:middle line:84% So you actually didn't need the diagonals to be 0. 00:29:38.797 --> 00:29:39.880 align:middle line:90% ALAN EDELMAN: That's true. 00:29:39.880 --> 00:29:40.880 align:middle line:84% STEVEN JOHNSON: In the Q transpose 00:29:40.880 --> 00:29:42.520 align:middle line:84% dQ Because the first term-- if you look at it, 00:29:42.520 --> 00:29:43.630 align:middle line:90% the diagonals cancel. 00:29:43.630 --> 00:29:44.755 align:middle line:90% ALAN EDELMAN: That is true. 00:29:44.755 --> 00:29:46.950 align:middle line:84% Yeah, I can see that right there, that-- 00:29:46.950 --> 00:29:51.050 align:middle line:84% and of course, you could have taken a non-symmetric matrix. 00:29:51.050 --> 00:29:53.810 align:middle line:84% But then you would need both eigenvectors to play a role. 00:29:53.810 --> 00:29:55.435 align:middle line:90% So lots of generalizations. 00:29:55.435 --> 00:29:56.560 align:middle line:90% Yeah, I never noticed that. 00:29:56.560 --> 00:29:59.512 align:middle line:84% But I never perturbed symmetric matrices-- 00:29:59.512 --> 00:30:00.220 align:middle line:90% you perturb them? 00:30:00.220 --> 00:30:00.550 align:middle line:90% STEVEN JOHNSON: 00:30:00.550 --> 00:30:02.740 align:middle line:84% ALAN EDELMAN: I feel like it's violating the-- 00:30:02.740 --> 00:30:05.515 align:middle line:90% STEVEN JOHNSON: Physics often-- 00:30:05.515 --> 00:30:07.390 align:middle line:84% a symmetric system is like a lossless system, 00:30:07.390 --> 00:30:08.920 align:middle line:84% like a lossless vibrating system. 00:30:08.920 --> 00:30:11.650 align:middle line:84% But real systems, of course, often have losses. 00:30:11.650 --> 00:30:12.940 align:middle line:90% But the losses are small. 00:30:12.940 --> 00:30:14.080 align:middle line:84% So you want to analyze the losses-- 00:30:14.080 --> 00:30:15.250 align:middle line:84% ALAN EDELMAN: You want to analyze-- that makes sense. 00:30:15.250 --> 00:30:16.010 align:middle line:90% STEVEN JOHNSON: Where it's easy. 00:30:16.010 --> 00:30:18.397 align:middle line:84% And then, you want to put in the losses perturbatively. 00:30:18.397 --> 00:30:19.730 align:middle line:90% ALAN EDELMAN: That's cool, yeah. 00:30:19.730 --> 00:30:22.010 align:middle line:84% STEVEN JOHNSON: and so it's nice. 00:30:22.010 --> 00:30:23.510 align:middle line:84% ALAN EDELMAN: The other part of this 00:30:23.510 --> 00:30:25.940 align:middle line:84% actually tells you what happens to eigenvectors, right? 00:30:25.940 --> 00:30:29.090 align:middle line:84% That's living on the off diagonal. 00:30:29.090 --> 00:30:31.610 align:middle line:90% So here's the theorem. 00:30:31.610 --> 00:30:34.400 align:middle line:84% I just wrote it on the board because I forgot it was here. 00:30:34.400 --> 00:30:35.420 align:middle line:90% But yeah. 00:30:35.420 --> 00:30:40.160 align:middle line:84% So if s depended on a parameter, one way to say it 00:30:40.160 --> 00:30:43.940 align:middle line:84% is that the derivative of the eigenvalue is this. 00:30:43.940 --> 00:30:47.130 align:middle line:84% And we can use this to, for example, 00:30:47.130 --> 00:30:52.420 align:middle line:84% get the gradient of a single eigenvalue because the-- 00:30:52.420 --> 00:30:56.372 align:middle line:84% oh, I wonder if I should mention-- 00:30:56.372 --> 00:30:58.020 align:middle line:84% is this obviously the same thing? 00:30:58.020 --> 00:30:58.590 align:middle line:90% No, it's not. 00:30:58.590 --> 00:31:01.170 align:middle line:84% Let me-- there's another theorem that I think 00:31:01.170 --> 00:31:05.230 align:middle line:84% hasn't come up yet, but it's a very useful fact 00:31:05.230 --> 00:31:06.090 align:middle line:90% in linear algebra. 00:31:06.090 --> 00:31:07.890 align:middle line:84% And maybe, Steven, you could tell me if you did mention it. 00:31:07.890 --> 00:31:08.700 align:middle line:90% But I don't think-- 00:31:08.700 --> 00:31:09.360 align:middle line:84% STEVEN JOHNSON: The cyclic property of the trace? 00:31:09.360 --> 00:31:10.830 align:middle line:84% ALAN EDELMAN: The cyclic property of the trace. 00:31:10.830 --> 00:31:10.950 align:middle line:90% STEVEN JOHNSON: Yeah. 00:31:10.950 --> 00:31:12.270 align:middle line:84% So they're actually deriving this in homework. 00:31:12.270 --> 00:31:13.200 align:middle line:90% But yeah. 00:31:13.200 --> 00:31:15.070 align:middle line:84% We've used the cyclic property quite a bit. 00:31:15.070 --> 00:31:15.510 align:middle line:90% ALAN EDELMAN: Oh, OK. 00:31:15.510 --> 00:31:16.015 align:middle line:90% All right. 00:31:16.015 --> 00:31:17.640 align:middle line:84% I didn't remember that you had done it. 00:31:17.640 --> 00:31:18.720 align:middle line:90% But all right. 00:31:18.720 --> 00:31:20.850 align:middle line:84% Then I don't have to say a word about it then. 00:31:20.850 --> 00:31:23.412 align:middle line:84% And the fact that the cyclic property kind of holds-- 00:31:23.412 --> 00:31:24.870 align:middle line:84% there's another one of these things 00:31:24.870 --> 00:31:26.370 align:middle line:84% where, when it's a scalar, you might 00:31:26.370 --> 00:31:30.270 align:middle line:84% be sort of a little bit surprised that it works, right? 00:31:30.270 --> 00:31:32.910 align:middle line:84% Because you don't have to write the word "trace." 00:31:32.910 --> 00:31:35.850 align:middle line:84% When you have a scalar, it's like a 1-by-1 matrix, right? 00:31:35.850 --> 00:31:38.650 align:middle line:84% You don't have to say the word "trace" because it's just that. 00:31:38.650 --> 00:31:40.770 align:middle line:84% But if you pretend the word "trace" is here, 00:31:40.770 --> 00:31:42.660 align:middle line:90% then you can move it around. 00:31:42.660 --> 00:31:44.642 align:middle line:84% And now it's a big matrix, right? 00:31:44.642 --> 00:31:45.600 align:middle line:90% Was that another thing? 00:31:45.600 --> 00:31:48.565 align:middle line:84% I mean, you pointed out that it's kind of like dot products 00:31:48.565 --> 00:31:49.065 align:middle line:90% commuting-- 00:31:49.065 --> 00:31:50.580 align:middle line:84% STEVEN JOHNSON: We've used that a couple of times. 00:31:50.580 --> 00:31:51.390 align:middle line:90% ALAN EDELMAN: OK. 00:31:51.390 --> 00:31:52.590 align:middle line:84% STEVEN JOHNSON: Yeah, this just kind of gives away 00:31:52.590 --> 00:31:53.340 align:middle line:90% one of the homework problems. 00:31:53.340 --> 00:31:53.940 align:middle line:90% But that's OK. 00:31:53.940 --> 00:31:55.080 align:middle line:90% ALAN EDELMAN: Oh, I see. 00:31:55.080 --> 00:31:56.430 align:middle line:84% STEVEN JOHNSON: But hey probably already looked at the homework 00:31:56.430 --> 00:31:56.930 align:middle line:90% already. 00:31:56.930 --> 00:31:57.430 align:middle line:90% So. 00:31:57.430 --> 00:31:58.430 align:middle line:90% ALAN EDELMAN: All right. 00:31:58.430 --> 00:32:00.233 align:middle line:84% Well, it doesn't hurt if it helps, I think. 00:32:00.233 --> 00:32:01.650 align:middle line:84% STEVEN JOHNSON: So physicists call 00:32:01.650 --> 00:32:04.700 align:middle line:84% this first order perturbation theory 00:32:04.700 --> 00:32:06.110 align:middle line:90% which is the same thing as-- 00:32:06.110 --> 00:32:07.700 align:middle line:90% a first order perturbation-- 00:32:07.700 --> 00:32:09.025 align:middle line:84% ALAN EDELMAN: Is the same thing as computing the derivative. 00:32:09.025 --> 00:32:09.400 align:middle line:90% STEVEN JOHNSON: 00:32:09.400 --> 00:32:10.195 align:middle line:90% ALAN EDELMAN: Right, yeah. 00:32:10.195 --> 00:32:10.510 align:middle line:90% STEVEN JOHNSON: 00:32:10.510 --> 00:32:12.885 align:middle line:84% ALAN EDELMAN: It's just physics speak for the same thing. 00:32:12.885 --> 00:32:16.810 align:middle line:90% 00:32:16.810 --> 00:32:17.410 align:middle line:90% OK. 00:32:17.410 --> 00:32:22.040 align:middle line:84% And we can also get information about the eigenvectors as well. 00:32:22.040 --> 00:32:28.720 align:middle line:84% So off the diagonal, you can write down this equation. 00:32:28.720 --> 00:32:32.860 align:middle line:84% And it basically says that this divided by this 00:32:32.860 --> 00:32:34.840 align:middle line:84% will give you the change of the eigenvectors. 00:32:34.840 --> 00:32:38.110 align:middle line:84% And all sorts of havoc happens if two eigenvalues are equal. 00:32:38.110 --> 00:32:41.140 align:middle line:84% And thing is not exactly differentiable anymore 00:32:41.140 --> 00:32:43.130 align:middle line:90% and kind of revealing itself. 00:32:43.130 --> 00:32:43.630 align:middle line:90% OK? 00:32:43.630 --> 00:32:47.910 align:middle line:90% And so I think-- 00:32:47.910 --> 00:32:48.550 align:middle line:90% let's see. 00:32:48.550 --> 00:32:52.320 align:middle line:84% So-- oh, and by the way, here's second order perturbation 00:32:52.320 --> 00:32:54.040 align:middle line:90% theory, while we're at it. 00:32:54.040 --> 00:32:55.560 align:middle line:90% So yeah. 00:32:55.560 --> 00:32:59.400 align:middle line:84% So one can follow all the way through. 00:32:59.400 --> 00:33:01.600 align:middle line:84% Maybe I won't make a big deal about it right now. 00:33:01.600 --> 00:33:03.990 align:middle line:84% But if you know the first order perturbation 00:33:03.990 --> 00:33:05.595 align:middle line:84% of the eigenvectors, then you can 00:33:05.595 --> 00:33:07.470 align:middle line:84% use that to get the second order perturbation 00:33:07.470 --> 00:33:09.030 align:middle line:90% of the eigenvalues. 00:33:09.030 --> 00:33:12.510 align:middle line:84% And here's your whole perturbation theory. 00:33:12.510 --> 00:33:14.370 align:middle line:90% Should I have have divided by 2? 00:33:14.370 --> 00:33:15.450 align:middle line:90% Oh, no, there is a 2. 00:33:15.450 --> 00:33:16.350 align:middle line:90% So this is correct. 00:33:16.350 --> 00:33:17.400 align:middle line:90% OK. 00:33:17.400 --> 00:33:18.060 align:middle line:90% All right. 00:33:18.060 --> 00:33:20.790 align:middle line:84% So I just wanted to kind of show you 00:33:20.790 --> 00:33:23.970 align:middle line:84% this far for the derivatives of eigenvalues. 00:33:23.970 --> 00:33:25.920 align:middle line:84% You could do essentially the same game 00:33:25.920 --> 00:33:28.140 align:middle line:84% and get derivatives of singular values, 00:33:28.140 --> 00:33:29.890 align:middle line:90% all sorts of other things. 00:33:29.890 --> 00:33:33.210 align:middle line:84% But I think this is kind of just enough of a peak for the time 00:33:33.210 --> 00:33:34.720 align:middle line:90% we have in this class. 00:33:34.720 --> 00:33:37.850 align:middle line:90% Any questions about all this? 00:33:37.850 --> 00:33:40.100 align:middle line:84% Anything you had always wondered about differentiating 00:33:40.100 --> 00:33:42.980 align:middle line:84% eigenvalues, or eigenvectors, and stuff like that? 00:33:42.980 --> 00:33:45.820 align:middle line:90% 00:33:45.820 --> 00:33:47.310 align:middle line:90% OK then. 00:33:47.310 --> 00:33:47.820 align:middle line:90% All right. 00:33:47.820 --> 00:33:50.970 align:middle line:84% Well, then I'm going to switch gears. 00:33:50.970 --> 00:33:52.140 align:middle line:90% And-- oh, yes. 00:33:52.140 --> 00:33:54.242 align:middle line:84% There is a question-- two questions, actually. 00:33:54.242 --> 00:33:56.530 align:middle line:84% AUDIENCE: What if-- for example, sometimes 00:33:56.530 --> 00:33:59.530 align:middle line:84% I want to take the derivative confined 00:33:59.530 --> 00:34:02.832 align:middle line:84% to something like the the positive semidefinite matrices. 00:34:02.832 --> 00:34:05.720 align:middle line:90% 00:34:05.720 --> 00:34:06.710 align:middle line:90% ALAN EDELMAN: So that-- 00:34:06.710 --> 00:34:09.130 align:middle line:84% AUDIENCE: Do I have to also check, am I inside the thing, 00:34:09.130 --> 00:34:10.233 align:middle line:90% so I can go anywhere? 00:34:10.233 --> 00:34:11.650 align:middle line:84% And then I have to on the boundary 00:34:11.650 --> 00:34:13.429 align:middle line:90% so that then I can do this? 00:34:13.429 --> 00:34:14.630 align:middle line:90% ALAN EDELMAN: I think so. 00:34:14.630 --> 00:34:16.400 align:middle line:90% That's a good question. 00:34:16.400 --> 00:34:19.586 align:middle line:90% So this is now a-- 00:34:19.586 --> 00:34:21.800 align:middle line:84% it's kind of like an inequality constraint. 00:34:21.800 --> 00:34:24.830 align:middle line:84% And I was kind of talking about an equality constraint. 00:34:24.830 --> 00:34:26.659 align:middle line:90% So yeah. 00:34:26.659 --> 00:34:27.784 align:middle line:90% I don't know any other way. 00:34:27.784 --> 00:34:30.159 align:middle line:84% STEVEN JOHNSON: Well, you could use intrinsic coordinates 00:34:30.159 --> 00:34:30.710 align:middle line:90% because-- 00:34:30.710 --> 00:34:32.037 align:middle line:90% ALAN EDELMAN: I'm not a fan. 00:34:32.037 --> 00:34:34.370 align:middle line:84% STEVEN JOHNSON: But you could write semidefinite-- any-- 00:34:34.370 --> 00:34:35.845 align:middle line:84% ALAN EDELMAN: Use Cholesky factor, yeah. 00:34:35.845 --> 00:34:37.657 align:middle line:84% STEVEN JOHNSON: well, just as b transpose b. 00:34:37.657 --> 00:34:38.490 align:middle line:90% ALAN EDELMAN: Right. 00:34:38.490 --> 00:34:41.179 align:middle line:84% STEVEN JOHNSON: And so if you parameterize it in terms 00:34:41.179 --> 00:34:43.639 align:middle line:84% of some kind of factor-- it doesn't have to be a Cholesky 00:34:43.639 --> 00:34:44.270 align:middle line:90% factor, but you can-- 00:34:44.270 --> 00:34:45.440 align:middle line:90% ALAN EDELMAN: That's true. 00:34:45.440 --> 00:34:47.690 align:middle line:84% It could be the square root, right, or any-- 00:34:47.690 --> 00:34:48.820 align:middle line:84% STEVEN JOHNSON: It's probably the easiest way. 00:34:48.820 --> 00:34:50.575 align:middle line:84% ALAN EDELMAN: Maybe for that case that's probably true. 00:34:50.575 --> 00:34:50.830 align:middle line:90% STEVEN JOHNSON: 00:34:50.830 --> 00:34:51.729 align:middle line:90% ALAN EDELMAN: But in general-- 00:34:51.729 --> 00:34:52.029 align:middle line:90% STEVEN JOHNSON: Yeah. 00:34:52.029 --> 00:34:52.540 align:middle line:90% Well, yeah. 00:34:52.540 --> 00:34:54.623 align:middle line:84% In general, I would say that semidefinite matrices 00:34:54.623 --> 00:34:56.139 align:middle line:90% don't fall down from the sky. 00:34:56.139 --> 00:34:59.410 align:middle line:84% So they usually-- if you stare at it hard enough, 00:34:59.410 --> 00:35:01.310 align:middle line:90% they came from a b transpose b. 00:35:01.310 --> 00:35:01.810 align:middle line:90% So-- 00:35:01.810 --> 00:35:02.950 align:middle line:84% ALAN EDELMAN: So maybe the b is handy. 00:35:02.950 --> 00:35:04.325 align:middle line:84% But that's actually a good point. 00:35:04.325 --> 00:35:05.040 align:middle line:90% STEVEN JOHNSON: 00:35:05.040 --> 00:35:06.498 align:middle line:84% ALAN EDELMAN: That is a good point. 00:35:06.498 --> 00:35:07.430 align:middle line:90% OK. 00:35:07.430 --> 00:35:09.690 align:middle line:84% I don't know how general that is, though-- 00:35:09.690 --> 00:35:12.995 align:middle line:84% that it's easy to find an intrinsic coordinate system. 00:35:12.995 --> 00:35:13.870 align:middle line:90% STEVEN JOHNSON: Yeah. 00:35:13.870 --> 00:35:15.100 align:middle line:84% ALAN EDELMAN: So that might be the tricky part. 00:35:15.100 --> 00:35:16.490 align:middle line:90% But that is a good answer. 00:35:16.490 --> 00:35:16.990 align:middle line:90% OK. 00:35:16.990 --> 00:35:19.073 align:middle line:84% And I think there was another question over there? 00:35:19.073 --> 00:35:19.870 align:middle line:90% OK. 00:35:19.870 --> 00:35:21.370 align:middle line:84% AUDIENCE: I had a question where you 00:35:21.370 --> 00:35:25.690 align:middle line:84% take x transpose A x and the constraint x transpose x is 1. 00:35:25.690 --> 00:35:29.821 align:middle line:84% So if you just like the df assuming 00:35:29.821 --> 00:35:33.358 align:middle line:84% that for x transpose x, you still get the dot product of Ax 00:35:33.358 --> 00:35:36.737 align:middle line:90% and x and dx, right? 00:35:36.737 --> 00:35:37.570 align:middle line:90% ALAN EDELMAN: Right. 00:35:37.570 --> 00:35:43.750 align:middle line:90% But we wouldn't be going-- 00:35:43.750 --> 00:35:45.250 align:middle line:84% like, our search direction is kind 00:35:45.250 --> 00:35:47.330 align:middle line:90% of not physical, if you will. 00:35:47.330 --> 00:35:49.900 align:middle line:90% It's not on the sphere anymore. 00:35:49.900 --> 00:35:50.642 align:middle line:90% AUDIENCE: Yeah. 00:35:50.642 --> 00:35:52.600 align:middle line:84% That's why you have to multiply the projection. 00:35:52.600 --> 00:35:53.433 align:middle line:90% ALAN EDELMAN: Right. 00:35:53.433 --> 00:35:54.370 align:middle line:90% So to put the-- 00:35:54.370 --> 00:35:56.470 align:middle line:84% one way to look at it is the projection matrix 00:35:56.470 --> 00:35:58.660 align:middle line:84% is just, like, slapping it back onto the sphere, 00:35:58.660 --> 00:36:01.270 align:middle line:84% like whack-a-mole right down back onto the tangent space 00:36:01.270 --> 00:36:02.800 align:middle line:90% of the sphere. 00:36:02.800 --> 00:36:05.050 align:middle line:84% But we also see that it gives the correct answer 00:36:05.050 --> 00:36:07.480 align:middle line:84% and that it's moving in the steepest descent 00:36:07.480 --> 00:36:09.910 align:middle line:90% direction on the sphere. 00:36:09.910 --> 00:36:11.680 align:middle line:84% Because we don't want any directions 00:36:11.680 --> 00:36:13.760 align:middle line:84% that go off the sphere at this point. 00:36:13.760 --> 00:36:15.100 align:middle line:90% So that's right. 00:36:15.100 --> 00:36:16.660 align:middle line:90% Was there another question? 00:36:16.660 --> 00:36:19.780 align:middle line:84% Did I-- was there another hand up or just those two? 00:36:19.780 --> 00:36:20.530 align:middle line:90% All right. 00:36:20.530 --> 00:36:21.610 align:middle line:84% STEVEN JOHNSON: Do you want the blackboard now? 00:36:21.610 --> 00:36:23.902 align:middle line:84% ALAN EDELMAN: Yeah, I'm going to do the blackboard now. 00:36:23.902 --> 00:36:25.630 align:middle line:90% And we're going to switch gears. 00:36:25.630 --> 00:36:27.220 align:middle line:90% Thanks, Steven. 00:36:27.220 --> 00:36:30.040 align:middle line:84% And I'm going to pull back these notes here. 00:36:30.040 --> 00:36:32.460 align:middle line:90% So I guess I can close this. 00:36:32.460 --> 00:36:37.000 align:middle line:90%