WEBVTT 00:00:01.550 --> 00:00:03.920 The following content is provided under a Creative 00:00:03.920 --> 00:00:05.310 Commons license. 00:00:05.310 --> 00:00:07.520 Your support will help MIT OpenCourseWare 00:00:07.520 --> 00:00:11.610 continue to offer high quality educational resources for free. 00:00:11.610 --> 00:00:14.180 To make a donation or to view additional materials 00:00:14.180 --> 00:00:18.140 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:18.140 --> 00:00:19.026 at ocw.mit.edu. 00:00:22.758 --> 00:00:23.550 GILBERT STRANG: OK. 00:00:23.550 --> 00:00:25.770 Why don't I start? 00:00:25.770 --> 00:00:32.130 So I was hoping that we would have the next online assignment 00:00:32.130 --> 00:00:37.110 ready, but Julia 0.6, the new version, is slowing us down, 00:00:37.110 --> 00:00:42.140 and it'll probably be next time. 00:00:42.140 --> 00:00:47.110 But my lectures, of course, are-- 00:00:47.110 --> 00:00:49.030 well, what I want to say, perhaps, 00:00:49.030 --> 00:00:52.690 is this isn't intended to be a course 00:00:52.690 --> 00:00:55.750 in numerical linear algebra, but I 00:00:55.750 --> 00:01:00.550 thought I couldn't let the whole semester go by without saying 00:01:00.550 --> 00:01:04.780 something about how to compute eigenvalues and singular 00:01:04.780 --> 00:01:07.420 values. 00:01:07.420 --> 00:01:14.350 Of course, you're going to call eig or SVD or the equivalent 00:01:14.350 --> 00:01:17.560 in Python or Julia. 00:01:17.560 --> 00:01:23.020 But actually, the QR factorization 00:01:23.020 --> 00:01:27.640 that we spoke about, that we spoke entirely about last time, 00:01:27.640 --> 00:01:29.470 is the key-- 00:01:29.470 --> 00:01:31.870 unexpectedly, unexpectedly. 00:01:31.870 --> 00:01:38.450 You have a matrix A whose eigenvalues you want. 00:01:38.450 --> 00:01:41.590 So let's start with eigenvalues. 00:01:41.590 --> 00:01:44.580 And it might be a symmetric matrix. 00:01:44.580 --> 00:01:49.060 I'll stay with A rather than S because it doesn't have to be, 00:01:49.060 --> 00:01:51.040 but you get special-- 00:01:51.040 --> 00:01:55.630 always you get something special if the matrix is symmetric. 00:01:58.240 --> 00:02:03.130 So this method of computing eigenvalues, to me at least 00:02:03.130 --> 00:02:05.950 and I think to many people, came out 00:02:05.950 --> 00:02:12.640 of the blue a while ago, but not that long ago. 00:02:12.640 --> 00:02:14.840 And it worked very well. 00:02:14.840 --> 00:02:16.420 So here's the idea. 00:02:16.420 --> 00:02:21.730 It's called the QR method because you start by factoring 00:02:21.730 --> 00:02:23.530 your matrix into QR. 00:02:23.530 --> 00:02:29.000 So here's A. Can we call it A0? 00:02:29.000 --> 00:02:33.400 That's the matrix we start with whose eigenvalues we want. 00:02:33.400 --> 00:02:37.354 And I'll call these Q0 and R0. 00:02:37.354 --> 00:02:40.940 And you remember what that means, 00:02:40.940 --> 00:02:44.620 what's hiding behind those letters that I've written? 00:02:44.620 --> 00:02:49.930 You have the columns of A, possibly symmetric as I said, 00:02:49.930 --> 00:02:54.100 but not orthogonal usually. 00:02:54.100 --> 00:02:59.380 So you find an orthogonal basis. 00:02:59.380 --> 00:03:01.840 You orthogonalize-- you line them up 00:03:01.840 --> 00:03:03.700 perpendicular to each other. 00:03:03.700 --> 00:03:07.390 And then there is a matrix R, which 00:03:07.390 --> 00:03:11.800 happens to be upper triangular, that 00:03:11.800 --> 00:03:15.400 connects the not-orthogonal basis 00:03:15.400 --> 00:03:18.070 with the orthogonality base. 00:03:18.070 --> 00:03:18.810 Right. 00:03:18.810 --> 00:03:26.310 We constructed R step-by-step. 00:03:26.310 --> 00:03:34.390 And then, the idea is write these in the reverse order, 00:03:34.390 --> 00:03:42.300 and that will be A1, the next A. And then do it again and again 00:03:42.300 --> 00:03:44.690 and again. 00:03:44.690 --> 00:03:50.670 And so we're hoping, of course, that the eigenvalues didn't 00:03:50.670 --> 00:03:51.730 change. 00:03:51.730 --> 00:03:56.190 We're hoping that we can forget A0, start again with A1, 00:03:56.190 --> 00:03:59.730 and produce A2 and continue. 00:03:59.730 --> 00:04:00.960 So we're hoping two things. 00:04:00.960 --> 00:04:02.070 One will not be a hope. 00:04:02.070 --> 00:04:08.580 We can see that the eigenvalues of A1 and A0 are the same. 00:04:08.580 --> 00:04:11.190 So we have not changed the eigenvalues. 00:04:11.190 --> 00:04:12.840 How do we see that? 00:04:12.840 --> 00:04:15.900 If you had to show that two matrices had 00:04:15.900 --> 00:04:20.490 the same eigenvalues, would you compute all the eigenvalues 00:04:20.490 --> 00:04:21.810 and compare them? 00:04:21.810 --> 00:04:23.190 Certainly not. 00:04:23.190 --> 00:04:23.940 What would you do? 00:04:23.940 --> 00:04:29.190 What's the best test, the usual test, the usual thing 00:04:29.190 --> 00:04:31.860 you would want to show to-- 00:04:31.860 --> 00:04:33.090 AUDIENCE: They're similar. 00:04:33.090 --> 00:04:35.007 GILBERT STRANG: They're similar, that's right. 00:04:35.007 --> 00:04:39.060 So the claim would be that these two matrices are similar. 00:04:39.060 --> 00:04:40.570 Right. 00:04:40.570 --> 00:04:42.270 So maybe we should show that. 00:04:42.270 --> 00:04:45.330 So we want to write A1 in a different way. 00:04:45.330 --> 00:04:51.820 So the claim is A1 is similar to A. 00:04:51.820 --> 00:04:55.180 So we just have to figure out-- 00:04:55.180 --> 00:04:58.720 we have to get A1 and turn it to A0. 00:04:58.720 --> 00:04:59.620 So here is A1. 00:05:03.250 --> 00:05:05.320 So A1. 00:05:05.320 --> 00:05:07.300 I want to show that that's right. 00:05:07.300 --> 00:05:11.830 So that's R0, Q0. 00:05:11.830 --> 00:05:14.570 But what is Q0? 00:05:14.570 --> 00:05:17.090 From here, Q0 is what we-- 00:05:20.000 --> 00:05:22.460 Q0 is-- what do I want? 00:05:22.460 --> 00:05:24.960 I want to put R0 inverse over there. 00:05:24.960 --> 00:05:27.800 So R0. 00:05:27.800 --> 00:05:31.938 Now for Q0, I'm going to substitute. 00:05:31.938 --> 00:05:35.050 So what is Q0? 00:05:35.050 --> 00:05:39.400 I multiply both sides of that equation by R0 inverse. 00:05:39.400 --> 00:05:51.550 So Q0 is A0 is-- sorry, I said A and wrote Q. A0, R0 inverse. 00:05:55.320 --> 00:05:59.570 For Q0, I've put in what it equals. 00:05:59.570 --> 00:06:02.290 And that's done it. 00:06:02.290 --> 00:06:09.670 That's exactly telling me that A1 which is this, equals this, 00:06:09.670 --> 00:06:12.410 and that's a similarity transformation. 00:06:12.410 --> 00:06:14.860 I have not changed the eigenvalues. 00:06:14.860 --> 00:06:17.010 So that's OK. 00:06:17.010 --> 00:06:20.460 The other thing, which is the, you could say, 00:06:20.460 --> 00:06:24.360 the miracle in this thing, is that when 00:06:24.360 --> 00:06:29.330 I continue to do that for almost every matrix, 00:06:29.330 --> 00:06:35.660 the matrices begin to lose stuff off the diagonal, 00:06:35.660 --> 00:06:41.180 especially below the diagonal by the ordering that QR has done. 00:06:41.180 --> 00:06:43.440 So it would tend to-- 00:06:43.440 --> 00:06:45.010 you start with a matrix A0. 00:06:49.230 --> 00:06:54.980 You got a matrix A1, which is a little smaller here. 00:06:59.080 --> 00:07:02.760 This part being especially smaller. 00:07:02.760 --> 00:07:03.690 You do it again. 00:07:03.690 --> 00:07:06.000 This is even smaller. 00:07:06.000 --> 00:07:07.240 Even smaller. 00:07:07.240 --> 00:07:08.460 A2. 00:07:08.460 --> 00:07:10.080 Even smaller. 00:07:10.080 --> 00:07:11.820 And you keep going. 00:07:11.820 --> 00:07:18.730 And for most matrices, the result is that-- 00:07:18.730 --> 00:07:22.420 I don't know how many steps we want to think of taking, 00:07:22.420 --> 00:07:28.500 but we get quite small numbers here, especially-- 00:07:28.500 --> 00:07:29.000 yeah. 00:07:29.000 --> 00:07:32.940 So once we get small numbers here, little epsilons-- 00:07:32.940 --> 00:07:37.610 that's everybody's shorthand for small numbers-- 00:07:37.610 --> 00:07:41.330 what would you expect to see on the diagonal? 00:07:41.330 --> 00:07:42.460 AUDIENCE: The eigenvalues. 00:07:42.460 --> 00:07:45.550 GILBERT STRANG: The eigenvalues, because this 00:07:45.550 --> 00:07:47.740 has the same eigenvalues as this, 00:07:47.740 --> 00:07:50.530 as this, as this, and these little epsilons 00:07:50.530 --> 00:07:53.590 are not going to change the eigenvalues too much. 00:07:53.590 --> 00:07:56.440 So these will be, on the diagonal, 00:07:56.440 --> 00:08:02.020 will be close to the eigenvalues. 00:08:02.020 --> 00:08:07.270 And actually, what happens is this one comes first. 00:08:07.270 --> 00:08:10.600 That one is quite accurate first. 00:08:10.600 --> 00:08:16.640 I guess we should probably do a simple example 00:08:16.640 --> 00:08:19.870 to see this happen. 00:08:19.870 --> 00:08:22.980 Actually, I do an example in the notes. 00:08:22.980 --> 00:08:25.570 And let me say what happens. 00:08:25.570 --> 00:08:29.860 So I do a 2 by 2 example which has something 00:08:29.860 --> 00:08:33.624 like cos theta, sine theta. 00:08:37.728 --> 00:08:38.520 I don't know what-- 00:08:38.520 --> 00:08:41.940 I've forgotten what I took, and I don't have that page of notes 00:08:41.940 --> 00:08:42.440 here. 00:08:42.440 --> 00:08:51.370 Something here, something here, something here as A0. 00:08:51.370 --> 00:08:55.810 And, then A1, after just one step, 00:08:55.810 --> 00:09:04.410 has sine cubed theta there and numbers there 00:09:04.410 --> 00:09:08.160 that are getting much closer to the eigenvalues. 00:09:08.160 --> 00:09:11.820 Sorry that this isn't a full-scale example, 00:09:11.820 --> 00:09:14.460 but the point of the example is here, 00:09:14.460 --> 00:09:22.880 that this off-diagonal entry gets cubed. 00:09:22.880 --> 00:09:27.770 And the next step will be its 9th power, and then 00:09:27.770 --> 00:09:29.330 the 27th power. 00:09:29.330 --> 00:09:32.540 So it's really quickly going to 0. 00:09:32.540 --> 00:09:39.800 And this happens-- so cubic convergence is-- 00:09:39.800 --> 00:09:45.410 that's a price in numerical linear algebra. 00:09:45.410 --> 00:09:48.880 So that happens. 00:09:48.880 --> 00:09:53.560 So this arrived on the scene and quickly 00:09:53.560 --> 00:09:57.400 blew away all other methods that were being 00:09:57.400 --> 00:10:00.560 used to compute eigenvalues. 00:10:00.560 --> 00:10:04.610 But numerical people, being what they are, 00:10:04.610 --> 00:10:06.020 they wanted to improve it-- 00:10:06.020 --> 00:10:08.780 like, is there a way to make it faster? 00:10:08.780 --> 00:10:12.050 And it turned out there is. 00:10:12.050 --> 00:10:14.585 It turned out that the better way-- 00:10:17.120 --> 00:10:21.440 basically, the same idea, but just you're 00:10:21.440 --> 00:10:27.260 always looking for a simple change. 00:10:31.060 --> 00:10:34.680 And the idea of introducing a shift 00:10:34.680 --> 00:10:42.140 was tried and turned out to work extremely well. 00:10:42.140 --> 00:10:44.960 So that's the improvement. 00:10:44.960 --> 00:10:47.230 So we take A1-- 00:10:47.230 --> 00:10:47.730 no. 00:10:47.730 --> 00:10:49.500 So how does it-- 00:10:49.500 --> 00:10:51.910 how does the shift work? 00:10:51.910 --> 00:10:54.890 So instead of A1-- 00:10:54.890 --> 00:10:57.480 have I got space here to do this? 00:10:57.480 --> 00:11:02.430 Instead of A0, I take A1 minus a shift. 00:11:02.430 --> 00:11:06.360 So a shift is some multiple of the identity. 00:11:06.360 --> 00:11:10.040 If I just move a matrix by a multiple of the identity, 00:11:10.040 --> 00:11:12.050 what happens to its eigenvectors? 00:11:12.050 --> 00:11:14.720 What happens to its eigenvalues? 00:11:14.720 --> 00:11:16.970 Something pretty simple, right, because I'm just 00:11:16.970 --> 00:11:19.610 shifting by sI. 00:11:19.610 --> 00:11:21.282 What happens to its eigenvectors? 00:11:21.282 --> 00:11:22.490 AUDIENCE: They stay the same. 00:11:22.490 --> 00:11:24.820 GILBERT STRANG: They're the same. 00:11:24.820 --> 00:11:26.605 And what happens to the eigenvalues? 00:11:26.605 --> 00:11:27.730 AUDIENCE: They change by s. 00:11:27.730 --> 00:11:30.460 GILBERT STRANG: They change by s. 00:11:30.460 --> 00:11:37.150 If A0 V equaled lambda v, then when I multiply this by V, 00:11:37.150 --> 00:11:39.010 there will be an extra-- 00:11:39.010 --> 00:11:44.910 it'll be lambda minus s because the identity times the V 00:11:44.910 --> 00:11:50.310 is just V. So it just shifts all the eigenvalues by s. 00:11:50.310 --> 00:11:52.010 And you try to shift-- 00:11:52.010 --> 00:11:54.570 you look for a shift. 00:11:54.570 --> 00:11:55.620 This would be great. 00:11:55.620 --> 00:11:59.460 If you knew lambda N, a shift there, 00:11:59.460 --> 00:12:03.450 you would be looking for 0 then if you shifted them 00:12:03.450 --> 00:12:05.250 all by that lambda. 00:12:05.250 --> 00:12:07.650 And it turns out that would speed things up. 00:12:07.650 --> 00:12:12.780 So it will work instead with this matrix as now, 00:12:12.780 --> 00:12:17.190 again, I'm factoring it in a Q0 R0. 00:12:17.190 --> 00:12:23.250 So that's the work of the method is in doing Gram-Schmidt 00:12:23.250 --> 00:12:25.020 at every step. 00:12:25.020 --> 00:12:28.740 And then there's a little work in reversing the order. 00:12:33.150 --> 00:12:37.010 And then I want to undo the shift, 00:12:37.010 --> 00:12:39.470 so I do that factorization. 00:12:39.470 --> 00:12:40.410 Now, let's see. 00:12:40.410 --> 00:12:45.240 You may have to help me to remember what I should do here. 00:12:45.240 --> 00:12:47.880 So I factor those. 00:12:47.880 --> 00:12:50.250 I reverse those. 00:12:50.250 --> 00:12:57.460 And then I think I add back the shift, and that's my A1. 00:13:00.700 --> 00:13:01.800 So I took A0. 00:13:01.800 --> 00:13:03.450 I shifted it. 00:13:03.450 --> 00:13:04.590 I worked with it. 00:13:04.590 --> 00:13:05.610 QR. 00:13:05.610 --> 00:13:06.750 Reversed the order. 00:13:06.750 --> 00:13:07.800 RQ. 00:13:07.800 --> 00:13:11.150 Added back the shift to get a matrix. 00:13:11.150 --> 00:13:16.980 And what am I, of course, hoping about the matrix A1 and A0? 00:13:16.980 --> 00:13:19.960 I'm hoping they're still similar. 00:13:19.960 --> 00:13:23.030 So I did a shift and I undid a shift. 00:13:23.030 --> 00:13:28.780 But of course, after doing our QR, I have to check, 00:13:28.780 --> 00:13:31.060 are these really still similar? 00:13:31.060 --> 00:13:34.880 So let me just try to check that one again. 00:13:34.880 --> 00:13:39.940 Maybe I'll just-- it's sitting right there, 00:13:39.940 --> 00:13:42.550 so let me do it again. 00:13:42.550 --> 00:13:49.400 I'm hoping something-- where did we-- 00:13:49.400 --> 00:13:50.570 oh yeah, here. 00:13:50.570 --> 00:13:54.730 We show that A1 was similar to A0, 00:13:54.730 --> 00:13:58.860 and I'm hoping that's probably still true even-- 00:13:58.860 --> 00:14:02.380 the shift didn't mess that up. 00:14:02.380 --> 00:14:03.620 Let's just try. 00:14:03.620 --> 00:14:08.825 So A0-- that's R0, Q0 plus sI. 00:14:12.120 --> 00:14:15.240 And now what am I going to do? 00:14:15.240 --> 00:14:19.170 I'm going to-- what did I do before? 00:14:19.170 --> 00:14:24.060 I figured out what Q0 was from this. 00:14:26.770 --> 00:14:27.570 You remember? 00:14:27.570 --> 00:14:28.850 So this R0. 00:14:31.670 --> 00:14:33.320 Now I have to put in Q0. 00:14:33.320 --> 00:14:36.860 But Q0 is this thing inverse times this. 00:14:36.860 --> 00:14:38.750 Is this going to work? 00:14:38.750 --> 00:14:42.920 I'm hoping, but I don't think I wanted to get that. 00:14:47.190 --> 00:14:50.610 No, it's not Q0 there. 00:14:50.610 --> 00:14:52.470 What do I put it here? 00:14:55.070 --> 00:15:01.666 And if it doesn't work, we'll leave it as an exercise. 00:15:01.666 --> 00:15:02.870 AUDIENCE: [INAUDIBLE] 00:15:03.370 --> 00:15:08.125 GILBERT STRANG: Yeah, because I didn't start right somehow. 00:15:11.320 --> 00:15:14.530 But let me just push along to see what happens. 00:15:14.530 --> 00:15:17.620 So I'm plugging in for Q0 here. 00:15:17.620 --> 00:15:20.970 I'm plugging in this matrix, so it's shouldn't have 00:15:20.970 --> 00:15:25.600 inverted it, times R0 inverse. 00:15:25.600 --> 00:15:26.170 Who knows? 00:15:26.170 --> 00:15:28.390 It might work. 00:15:28.390 --> 00:15:29.840 Who knows? 00:15:29.840 --> 00:15:31.990 So that's the R0, Q0. 00:15:31.990 --> 00:15:33.160 Right? 00:15:33.160 --> 00:15:34.120 Is everybody with me? 00:15:34.120 --> 00:15:37.090 Sorry about the-- 00:15:37.090 --> 00:15:39.820 R0 inverse. 00:15:39.820 --> 00:15:41.275 And then I have to add sI. 00:15:45.860 --> 00:15:47.210 So what have I done? 00:15:47.210 --> 00:15:51.840 I've just pushed on, believing that this 00:15:51.840 --> 00:15:59.340 would work because it's the method that is constantly used. 00:15:59.340 --> 00:16:01.555 Now, do I have-- 00:16:01.555 --> 00:16:02.430 what do I have there? 00:16:05.020 --> 00:16:07.760 Is it working? 00:16:07.760 --> 00:16:11.700 This is R0 A. That was, of course, A0. 00:16:11.700 --> 00:16:14.390 R0, A0, R0 inverse. 00:16:14.390 --> 00:16:19.040 R0, A0, R0 inverse. 00:16:19.040 --> 00:16:21.750 Good. 00:16:21.750 --> 00:16:23.270 Minus s. 00:16:23.270 --> 00:16:29.940 What have I got there from the R0 minus sI R0? 00:16:29.940 --> 00:16:31.332 What is that? 00:16:31.332 --> 00:16:32.337 AUDIENCE: Minus sI. 00:16:32.337 --> 00:16:33.670 GILBERT STRANG: That's minus sI. 00:16:33.670 --> 00:16:35.450 Ha, look. 00:16:35.450 --> 00:16:36.540 Success. 00:16:36.540 --> 00:16:38.850 The R0 cancels the R0 inverse. 00:16:38.850 --> 00:16:44.580 So that term from that, that, that is minus sI 00:16:44.580 --> 00:16:45.770 cancels plus sI. 00:16:45.770 --> 00:16:46.980 I'm finished. 00:16:46.980 --> 00:16:51.930 And lo and behold, we have the same similarity. 00:16:51.930 --> 00:16:56.520 So we messed around by a multiple of the identity, 00:16:56.520 --> 00:16:58.440 and it didn't-- 00:16:58.440 --> 00:17:04.839 it actually makes the thing converge faster 00:17:04.839 --> 00:17:08.950 if we choose the shifts well. 00:17:08.950 --> 00:17:12.640 But basically, the same idea is still working. 00:17:12.640 --> 00:17:16.230 So that's the QR method. 00:17:16.230 --> 00:17:17.819 Well, that's the method. 00:17:17.819 --> 00:17:22.260 We haven't shown and won't show that-- 00:17:22.260 --> 00:17:28.970 except for this half-completed example, 00:17:28.970 --> 00:17:36.310 I don't plan to prove that the lower triangular part begins 00:17:36.310 --> 00:17:40.130 to disappear, gets smaller and smaller and smaller, 00:17:40.130 --> 00:17:44.000 and then the eigenvalues pop up on the diagonal. 00:17:44.000 --> 00:17:44.755 It's amazing. 00:17:47.700 --> 00:17:50.480 Amazing. 00:17:50.480 --> 00:17:55.430 Now, is there any other improvement we can make? 00:17:55.430 --> 00:17:57.820 So that's the method. 00:17:57.820 --> 00:18:00.760 And where is the work in using that method? 00:18:00.760 --> 00:18:03.160 Because that's what we always focus on. 00:18:03.160 --> 00:18:06.430 Where are we spending computer time? 00:18:06.430 --> 00:18:08.230 Well, we're spending computer time 00:18:08.230 --> 00:18:13.880 in doing the factorization. 00:18:13.880 --> 00:18:17.380 So it didn't cost anything to shift by the identity, 00:18:17.380 --> 00:18:21.270 but then we had to factor that into Q0, R0. 00:18:21.270 --> 00:18:24.070 Then it didn't cost much to multiply them 00:18:24.070 --> 00:18:25.580 in the opposite order. 00:18:25.580 --> 00:18:29.050 So the work was in QR. 00:18:29.050 --> 00:18:36.650 So could we think of anything to improve that aspect? 00:18:36.650 --> 00:18:38.760 Can we think of anything there? 00:18:38.760 --> 00:18:43.910 And then we've got a really first class method. 00:18:43.910 --> 00:18:48.790 Well, the matrix A-- 00:18:48.790 --> 00:18:52.730 A0, the matrix we started with-- 00:18:52.730 --> 00:19:02.690 had some zeros that allowed us to skip steps 00:19:02.690 --> 00:19:05.370 in doing the QR factorization. 00:19:05.370 --> 00:19:07.610 So what am I going to say? 00:19:07.610 --> 00:19:14.570 I'm going to say if A or A0, our original matrix, 00:19:14.570 --> 00:19:16.100 has a bunch of zeros-- 00:19:16.100 --> 00:19:18.410 let's say it's got a whole lot of zeros there. 00:19:18.410 --> 00:19:18.950 Maybe it's-- 00:19:25.530 --> 00:19:26.940 Well, OK. 00:19:26.940 --> 00:19:29.460 I overdid it here. 00:19:29.460 --> 00:19:33.470 I know the eigenvalues right off. 00:19:33.470 --> 00:19:37.130 But so the truth is I can't-- 00:19:37.130 --> 00:19:41.060 saying that if that is not going to happen. 00:19:41.060 --> 00:19:48.470 But we can get zeros with one extra diagonal. 00:19:48.470 --> 00:19:51.050 That turns out-- so here is the main diagonal. 00:19:51.050 --> 00:19:55.250 Everybody's got his eye on the main diagonal. 00:19:55.250 --> 00:20:01.140 And one diagonal-- 00:20:01.140 --> 00:20:06.830 I can get a lot of zeros, but I can't by simple computations. 00:20:06.830 --> 00:20:10.740 And I'll show you how to get one. 00:20:10.740 --> 00:20:12.680 But I can't get all those to be 0, 00:20:12.680 --> 00:20:18.300 because then I would have the eigenvalues right there. 00:20:18.300 --> 00:20:20.700 Well, how do I know that I can't? 00:20:20.700 --> 00:20:24.600 In elimination, ordinary solving AX equal B, 00:20:24.600 --> 00:20:29.520 you really do get to an upper triangular form. 00:20:29.520 --> 00:20:35.670 Some operator, some elimination steps you plug away, 00:20:35.670 --> 00:20:42.030 and your matrix becomes upper triangular U and you're golden. 00:20:42.030 --> 00:20:48.000 But that's too much to expect here. 00:20:48.000 --> 00:20:52.460 In fact, we know we can't do it by simple steps 00:20:52.460 --> 00:20:55.500 because if we could do it, if we could get to a U 00:20:55.500 --> 00:20:58.860 with a whole lower triangular part 0, 00:20:58.860 --> 00:21:02.230 we would have found the eigenvalues. 00:21:02.230 --> 00:21:06.040 And we know that the eigenvalues solve a system-- 00:21:06.040 --> 00:21:09.420 solve an equation of nth degree. 00:21:09.420 --> 00:21:16.440 And we know-- somebody proved a century or more ago-- 00:21:16.440 --> 00:21:18.900 that you can't solve an nth degree equation 00:21:18.900 --> 00:21:22.100 by simple little steps. 00:21:22.100 --> 00:21:24.510 Do you know who that was and that you 00:21:24.510 --> 00:21:30.450 know that fact and what degree does it apply to? 00:21:30.450 --> 00:21:36.540 So that's an important fact that you pick up in math. 00:21:36.540 --> 00:21:37.040 Yeah? 00:21:37.040 --> 00:21:38.300 AUDIENCE: The 5th [INAUDIBLE]. 00:21:38.300 --> 00:21:39.675 GILBERT STRANG: 5th degree, yeah. 00:21:39.675 --> 00:21:44.280 So 5 by 5 and up would be-- 00:21:44.280 --> 00:21:45.870 this is impossible. 00:21:45.870 --> 00:21:48.830 Impossible. 00:21:48.830 --> 00:22:02.700 There is no formula to find a simple formula for the lambdas. 00:22:02.700 --> 00:22:06.200 And similarly for the sigmas for singular values. 00:22:06.200 --> 00:22:10.680 So the eigenvalues is definitely a level 00:22:10.680 --> 00:22:15.330 of difficulty beyond AX equal B, the inverse matrix, 00:22:15.330 --> 00:22:17.220 or something, that pivots. 00:22:17.220 --> 00:22:20.910 All that you can do exactly if you're doing exact arithmetic. 00:22:20.910 --> 00:22:27.970 We cannot find the lambdas exactly, 00:22:27.970 --> 00:22:35.840 but we can get as close as we like by continuing with the QR 00:22:35.840 --> 00:22:36.340 method. 00:22:36.340 --> 00:22:37.570 So yeah. 00:22:37.570 --> 00:22:39.040 In other words, we can't-- 00:22:39.040 --> 00:22:41.350 we have to settle for-- 00:22:41.350 --> 00:22:44.620 if we want to, like at the beginning, 00:22:44.620 --> 00:22:49.980 improve our matrix before we start doing that stuff, 00:22:49.980 --> 00:22:52.860 we can get it with one extra diagonal. 00:22:52.860 --> 00:22:56.330 And do you know what kind of a matrix, whose name-- 00:22:56.330 --> 00:22:57.675 I don't know why. 00:22:57.675 --> 00:22:58.890 AUDIENCE: Upper Hessenberg. 00:22:58.890 --> 00:22:59.190 GILBERT STRANG: Yeah. 00:22:59.190 --> 00:22:59.745 Say it again. 00:22:59.745 --> 00:23:00.870 AUDIENCE: Upper Hessenberg. 00:23:00.870 --> 00:23:02.245 GILBERT STRANG: Upper Hessenberg. 00:23:02.245 --> 00:23:04.260 So upper is just like upper triangular. 00:23:04.260 --> 00:23:05.520 It's up there. 00:23:05.520 --> 00:23:10.980 But key person's name is Hessenberg. 00:23:10.980 --> 00:23:17.120 As I say, that's a Hessenberg matrix. 00:23:17.120 --> 00:23:21.810 So Hessenberg matrix is a matrix with one triangular 00:23:21.810 --> 00:23:26.680 plus one more diagonal, but lots of zeros. 00:23:26.680 --> 00:23:28.450 Order of N squared. 00:23:28.450 --> 00:23:31.210 Something like almost like half N squared-- 00:23:31.210 --> 00:23:34.640 not quite, but close-- 00:23:34.640 --> 00:23:35.920 zeros. 00:23:35.920 --> 00:23:42.280 And you could show those zeros stay zeros in QR. 00:23:42.280 --> 00:23:44.810 So that really pays off. 00:23:44.810 --> 00:23:49.340 It cuts the work down significantly. 00:23:49.340 --> 00:23:55.750 So that's the-- full QR method is Step 1-- 00:23:55.750 --> 00:24:08.110 reduce A to Hessenberg form with these zeros. 00:24:08.110 --> 00:24:11.410 And when I say reduce, I mean find a similarity 00:24:11.410 --> 00:24:13.540 transformation, of course, because I 00:24:13.540 --> 00:24:19.030 want the eigenvalues of this to end up the same as the Hessen. 00:24:19.030 --> 00:24:21.820 I want to keep the same eigenvalues as I go. 00:24:21.820 --> 00:24:28.495 And then, Step 2 is QR on this Hessenberg matrix with shifts. 00:24:33.250 --> 00:24:37.970 So that's the code that would be programmed in eig(A). 00:24:41.060 --> 00:24:42.990 That's what Matlab and-- 00:24:42.990 --> 00:24:46.110 well really, Matlab is appealing-- 00:24:46.110 --> 00:24:49.800 like other matrix systems-- 00:24:49.800 --> 00:24:54.360 is appealing to LAPACK and LINPACK. 00:24:54.360 --> 00:24:59.160 A team of professional numerical analysts 00:24:59.160 --> 00:25:04.200 really spent a lot of effort and time. 00:25:04.200 --> 00:25:14.220 The book LAPACK has 10 authors, and you 00:25:14.220 --> 00:25:18.870 can download any of these codes, like the eigenvalue code. 00:25:18.870 --> 00:25:23.760 So that's where Matlab, naturally-- 00:25:23.760 --> 00:25:29.580 that's the Bible for code in linear algebra. 00:25:32.640 --> 00:25:35.780 I think it's interesting to know. 00:25:35.780 --> 00:25:42.030 And there's one more good thing to tell you about this method. 00:25:42.030 --> 00:25:45.910 And it applies if the matrix is symmetric. 00:25:45.910 --> 00:25:49.370 If the matrix is symmetric, then if we check all this, 00:25:49.370 --> 00:25:55.450 we could find that the matrices stayed symmetric. 00:25:55.450 --> 00:25:58.210 If A0 is symmetric, I can check-- 00:25:58.210 --> 00:26:00.430 you could easily check through this 00:26:00.430 --> 00:26:03.130 and you would discover that A1 is also symmetric. 00:26:08.630 --> 00:26:12.140 It turns out you could rewrite this 00:26:12.140 --> 00:26:17.850 with a Q0 in a Q0 inverse on the other side, 00:26:17.850 --> 00:26:21.360 but that Q0 inverse is the same as Q0 transposed 00:26:21.360 --> 00:26:23.220 because it's an orthogonal matrix, 00:26:23.220 --> 00:26:26.010 and symmetry would fall out. 00:26:26.010 --> 00:26:31.630 So if it's symmetric, and it's in his Hessenberg form 00:26:31.630 --> 00:26:34.980 and it stays symmetric at every step, 00:26:34.980 --> 00:26:39.840 what can you tell me about a symmetric Heisenberg matrix? 00:26:39.840 --> 00:26:41.058 AUDIENCE: [INAUDIBLE] 00:26:41.058 --> 00:26:42.350 GILBERT STRANG: It's only got-- 00:26:42.350 --> 00:26:42.980 yeah. 00:26:42.980 --> 00:26:45.020 You just erase all these. 00:26:45.020 --> 00:26:48.920 If they are zeros there and if the matrix is symmetric, 00:26:48.920 --> 00:26:54.990 then we can safely predict that it will only 00:26:54.990 --> 00:26:57.940 have one diagonal above, one non-zero 00:26:57.940 --> 00:27:00.120 diagonal above the main diagonal. 00:27:00.120 --> 00:27:04.200 In fact, it will say symmetric. 00:27:04.200 --> 00:27:09.540 So now I should write "symmetric Hessenberg matrix," 00:27:09.540 --> 00:27:13.380 and equals tridiagonal matrix. 00:27:19.900 --> 00:27:23.380 Three diagonals. 00:27:23.380 --> 00:27:25.540 So now you really have reduced the time 00:27:25.540 --> 00:27:32.000 to do QR because you've got a tridiagonal matrix. 00:27:32.000 --> 00:27:35.360 It'll stay tridiagonal in all these steps. 00:27:35.360 --> 00:27:38.520 So you're working with just three N numbers. 00:27:38.520 --> 00:27:42.890 Well actually, two N because the diagonal above and the diagonal 00:27:42.890 --> 00:27:44.760 below are the same. 00:27:44.760 --> 00:27:47.540 You're working with just two N numbers instead of order N 00:27:47.540 --> 00:27:50.600 squared, and it just goes like a bomb. 00:27:50.600 --> 00:27:55.460 So that's eig for symmetric matrices. 00:27:55.460 --> 00:27:57.900 And you see that it was all based-- 00:27:57.900 --> 00:28:08.480 that really the heart of the algorithm was QR. 00:28:08.480 --> 00:28:12.230 So that's my-- that took half the class 00:28:12.230 --> 00:28:20.350 to report on the favorite way, the eig way, 00:28:20.350 --> 00:28:23.650 to find eigenvalues. 00:28:23.650 --> 00:28:29.450 Oh, I should say something about singular values. 00:28:29.450 --> 00:28:31.760 So singular values. 00:28:31.760 --> 00:28:35.910 Of course, the singular values of the matrix 00:28:35.910 --> 00:28:39.870 are the eigenvalues of A transpose A-- 00:28:39.870 --> 00:28:43.140 square root of those eigenvalues. 00:28:43.140 --> 00:28:44.550 But you wouldn't do it that way. 00:28:44.550 --> 00:28:46.470 You would never form A transpose A. 00:28:46.470 --> 00:28:49.830 Oh, I didn't mention the other thing you would never, 00:28:49.830 --> 00:28:57.000 ever, ever do, so let me just put it here like in disgrace. 00:28:57.000 --> 00:29:03.870 To solve that equation is like, OK, back to first grade, 00:29:03.870 --> 00:29:06.870 because that's not-- 00:29:06.870 --> 00:29:09.140 that's very bad. 00:29:09.140 --> 00:29:14.280 A determinant-- first of all, it's extremely slow-- extremely 00:29:14.280 --> 00:29:15.090 slow. 00:29:15.090 --> 00:29:21.870 And the determinant is packing all this N squared pieces 00:29:21.870 --> 00:29:24.850 of information into N coefficient, 00:29:24.850 --> 00:29:28.110 and it's hopelessly ill conditioned. 00:29:28.110 --> 00:29:28.610 Yeah. 00:29:28.610 --> 00:29:31.320 You lose information all the time. 00:29:31.320 --> 00:29:36.100 So really, if this is going on camera, 00:29:36.100 --> 00:29:40.240 it better go on camera with an x, because you don't do it. 00:29:42.820 --> 00:29:44.410 Yeah. 00:29:44.410 --> 00:29:47.810 So where was I? 00:29:47.810 --> 00:29:49.940 Singular values. 00:29:49.940 --> 00:29:56.670 So A transpose A. So again, let's 00:29:56.670 --> 00:29:59.940 think about what you could do at the beginning 00:29:59.940 --> 00:30:05.670 before starting on QR for A transpose A 00:30:05.670 --> 00:30:08.880 or for the matrix A. What could you 00:30:08.880 --> 00:30:13.540 do with orthogonal matrices? 00:30:13.540 --> 00:30:17.610 So I guess-- what did we say about symmetric matrices? 00:30:17.610 --> 00:30:20.250 So here's what I said about symmetric matrices. 00:30:20.250 --> 00:30:22.510 If you'll give me a symmetric matrix, 00:30:22.510 --> 00:30:26.820 I can in just a simple number of simple steps 00:30:26.820 --> 00:30:28.320 make it tridiagonal. 00:30:31.230 --> 00:30:33.090 I can't make it diagonal because then I'd 00:30:33.090 --> 00:30:39.300 be finding the eigenvalues and Abel, who was the first person 00:30:39.300 --> 00:30:42.360 to see that that was impossible, forbids it. 00:30:45.290 --> 00:30:48.450 So let me let me write down what I'm saying here. 00:30:48.450 --> 00:30:51.750 If I have a symmetric matrix S, I 00:30:51.750 --> 00:30:56.720 can find a bunch of Qs and Q transposes, 00:30:56.720 --> 00:31:00.510 and I can put them all together into one big Q, 00:31:00.510 --> 00:31:02.490 and it's transposed. 00:31:02.490 --> 00:31:05.205 And what do I know about the eigenvalues of that matrix? 00:31:09.780 --> 00:31:12.000 Q is orthogonal always. 00:31:12.000 --> 00:31:13.860 So what can you tell me about the-- 00:31:13.860 --> 00:31:20.400 this is the same as QSQ inverse, and therefore the eigenvalues 00:31:20.400 --> 00:31:23.520 are the same. 00:31:23.520 --> 00:31:35.630 It's similar to S. And it becomes tridiagonal After? 00:31:35.630 --> 00:31:43.255 I find a good Q. Therefore, same lambdas. 00:31:47.007 --> 00:31:50.240 It's tridiagonal with the same lambdas. 00:31:50.240 --> 00:31:52.820 Now, what am I thinking about here? 00:31:52.820 --> 00:32:00.390 I'm thinking about-- tell me the corresponding possibility 00:32:00.390 --> 00:32:01.770 about singular values. 00:32:01.770 --> 00:32:04.470 I wanted to do something to my matrix. 00:32:04.470 --> 00:32:11.126 Now, I'm always taking a general matrix A. 00:32:11.126 --> 00:32:14.070 And I'm looking for its singular values. 00:32:14.070 --> 00:32:17.590 And I'm looking to simplify it. 00:32:17.590 --> 00:32:21.240 And what am I allowed to do? 00:32:21.240 --> 00:32:24.070 Yeah, I guess my question is-- 00:32:24.070 --> 00:32:28.210 similarity transformations left the eigenvalues alone. 00:32:28.210 --> 00:32:32.270 What can I do that leaves the singular values alone? 00:32:32.270 --> 00:32:35.590 That's a fundamental question because it was so fundamental 00:32:35.590 --> 00:32:37.240 for eigenvalues. 00:32:37.240 --> 00:32:41.110 By doing this, a matrix and its inverse, 00:32:41.110 --> 00:32:44.440 I got something similar, and I checked even 00:32:44.440 --> 00:32:49.370 in this class that the eigenvalues, same lambdas. 00:32:49.370 --> 00:32:56.200 Now I want a whole line that ends up with the same sigmas. 00:32:56.200 --> 00:33:01.930 And I want you to tell me what I'm allowed to do to the matrix 00:33:01.930 --> 00:33:03.880 without changing the sigmas. 00:33:03.880 --> 00:33:05.500 So this is a-- 00:33:05.500 --> 00:33:07.950 maybe don't shout it out immediately. 00:33:07.950 --> 00:33:10.030 Let everybody think. 00:33:10.030 --> 00:33:13.330 What am I allowed to do to a matrix? 00:33:13.330 --> 00:33:16.790 Every matrix has got these singular values, 00:33:16.790 --> 00:33:21.250 and now I want to make it a better matrix 00:33:21.250 --> 00:33:25.020 with more zeros or something. 00:33:25.020 --> 00:33:32.530 If I do that to it, does that change the sigmas? 00:33:32.530 --> 00:33:35.080 Can I do more than that to it? 00:33:35.080 --> 00:33:36.630 What can I do? 00:33:36.630 --> 00:33:42.970 What group of matrices will have the same sigmas 00:33:42.970 --> 00:33:45.710 as my starting matrix A? 00:33:45.710 --> 00:33:50.500 So that's a basic, basic question about singular values 00:33:50.500 --> 00:33:52.300 and the SVD. 00:33:52.300 --> 00:33:57.390 So let's think of the answer together. 00:33:57.390 --> 00:34:02.490 So it's connected to the SVD, so let me remember the SVD. 00:34:02.490 --> 00:34:06.250 The SVD-- I have some orthogonal matrix. 00:34:06.250 --> 00:34:08.310 Then the singular value of the matrix-- 00:34:08.310 --> 00:34:10.650 SV for singular values. 00:34:10.650 --> 00:34:12.930 And then another orthogonal matrix. 00:34:16.670 --> 00:34:20.560 What could I do to that equation that would not touch this guy? 00:34:25.330 --> 00:34:28.870 So I'm asking, what invariants? 00:34:28.870 --> 00:34:32.909 Because not touching it means leaving it not varying. 00:34:32.909 --> 00:34:39.060 So I'm looking for under what operations are the singular 00:34:39.060 --> 00:34:41.040 values invariant? 00:34:41.040 --> 00:34:43.570 When I was looking at eigenvalues, 00:34:43.570 --> 00:34:45.040 this was the operation. 00:34:45.040 --> 00:34:47.940 Well, it didn't have to be orthogonal-- 00:34:47.940 --> 00:34:50.159 something, then its inverse. 00:34:50.159 --> 00:34:52.420 But now, what is it up there? 00:34:52.420 --> 00:34:54.780 What could I do to that matrix A? 00:34:57.850 --> 00:34:59.900 Could I multiply it by Q? 00:34:59.900 --> 00:35:04.690 Could I throw in a Q maybe not even on the other end? 00:35:04.690 --> 00:35:10.760 If I throw in an orthogonal Q, do I change the singular values 00:35:10.760 --> 00:35:12.470 or do I not change them? 00:35:12.470 --> 00:35:15.170 Fundamental question. 00:35:15.170 --> 00:35:18.770 The answer is no, I don't change them. 00:35:18.770 --> 00:35:19.670 I'm allowed to do. 00:35:19.670 --> 00:35:23.540 That because here's an orthogonal matrix, 00:35:23.540 --> 00:35:26.870 a Q times U. If both of those are orthogonal, 00:35:26.870 --> 00:35:29.060 then the product is. 00:35:29.060 --> 00:35:33.020 Everybody knows that a product of two orthogonal matrix 00:35:33.020 --> 00:35:35.020 is still orthogonal. 00:35:35.020 --> 00:35:36.650 Better know that. 00:35:36.650 --> 00:35:37.400 Better know that. 00:35:39.970 --> 00:35:42.670 So if I have an orthogonal matrix Q 00:35:42.670 --> 00:35:45.370 and an orthogonal matrix U, I claim 00:35:45.370 --> 00:35:51.650 that this is still orthogonal. 00:35:51.650 --> 00:35:53.330 And how do I check it? 00:35:53.330 --> 00:35:56.450 Well, I use some test for orthogonality. 00:35:56.450 --> 00:35:57.790 What would be the-- 00:35:57.790 --> 00:36:01.180 what test do you like to use? 00:36:01.180 --> 00:36:03.115 The inverse is the same as that transpose, 00:36:03.115 --> 00:36:05.800 do you like that test? 00:36:05.800 --> 00:36:08.010 So I'll invert it. 00:36:08.010 --> 00:36:10.660 QU inverse. 00:36:10.660 --> 00:36:13.705 Of course, for any matrix that's U inverse, Q inverse. 00:36:16.220 --> 00:36:18.470 But these were separately orthogonal, 00:36:18.470 --> 00:36:20.990 so that's U transpose Q transpose. 00:36:23.660 --> 00:36:28.550 And that is the same as QU transpose. 00:36:31.950 --> 00:36:36.920 So I used the orthogonality of U and the orthogonality of Q 00:36:36.920 --> 00:36:40.770 to conclude that the inverse is the transpose. 00:36:40.770 --> 00:36:44.060 So the answer is yes, I could do that. 00:36:44.060 --> 00:36:45.620 Now, with singular value-- 00:36:45.620 --> 00:36:50.420 with eigenvalues, I had to multiply on the other side 00:36:50.420 --> 00:36:53.960 by Q inverse or Q transpose. 00:36:53.960 --> 00:36:55.160 Do I have to do that now? 00:36:55.160 --> 00:36:56.960 No. 00:36:56.960 --> 00:36:58.680 What can I do on the right-hand side? 00:37:01.600 --> 00:37:03.240 I can multiply by-- 00:37:03.240 --> 00:37:06.050 I can leave it alone. 00:37:06.050 --> 00:37:10.040 Then it has the same singular values because it's 00:37:10.040 --> 00:37:11.580 the same sigma in there. 00:37:11.580 --> 00:37:14.910 If I have a orthogonal matrix times a diagonal 00:37:14.910 --> 00:37:18.090 times an orthogonal, that diagonal is-- 00:37:18.090 --> 00:37:22.290 positive diagonal is going to be sigma. 00:37:22.290 --> 00:37:26.620 So what can I do on this side? 00:37:26.620 --> 00:37:32.380 I can multiply by any orthogonal matrix on that side too. 00:37:32.380 --> 00:37:37.330 So let's call this guy Q1 and this guy Q2. 00:37:37.330 --> 00:37:40.090 I still have an orthogonal matrix there, 00:37:40.090 --> 00:37:43.450 orthogonal matrix there, and the same sigma 00:37:43.450 --> 00:37:45.070 popped in the middle. 00:37:45.070 --> 00:37:48.610 So that's what you're allowed to do. 00:37:48.610 --> 00:37:51.250 That gives us more freedom. 00:37:51.250 --> 00:37:55.660 Before we got-- when we had to do similarity transformations 00:37:55.660 --> 00:38:00.640 with the same guy, we got it to be tridiagonal. 00:38:00.640 --> 00:38:04.960 But now, we're allowed to do more stuff. 00:38:04.960 --> 00:38:09.880 We're allowed to use different orthogonal matrices on the left 00:38:09.880 --> 00:38:11.290 and right. 00:38:11.290 --> 00:38:13.420 And we can reduce it even further 00:38:13.420 --> 00:38:17.110 from tridiagonal to bidiagonal. 00:38:17.110 --> 00:38:21.250 So the first step is getting zero. 00:38:21.250 --> 00:38:25.510 The step of getting zeros reduces it all the way 00:38:25.510 --> 00:38:28.480 to that, with all zeros there. 00:38:31.290 --> 00:38:37.690 So it's easier. 00:38:37.690 --> 00:38:40.570 Then I work on this. 00:38:40.570 --> 00:38:44.230 This is the matrix I work on using a QR type 00:38:44.230 --> 00:38:46.810 idea, some method like that. 00:38:46.810 --> 00:38:53.360 So everybody's seeing that our algorithm has got two stages. 00:38:53.360 --> 00:38:56.960 One is get a lot of zeros and get them 00:38:56.960 --> 00:39:02.450 in places that will stay zero as Part 2 of the algorithm 00:39:02.450 --> 00:39:03.860 gets going. 00:39:03.860 --> 00:39:06.640 And then, run Part 2 of the algorithm. 00:39:06.640 --> 00:39:10.910 You staying with-- each step is very fast now 00:39:10.910 --> 00:39:17.570 because doing a QR factorization is fast. 00:39:17.570 --> 00:39:18.750 Was there a question? 00:39:18.750 --> 00:39:20.420 Yeah. 00:39:20.420 --> 00:39:23.710 So I would call this bidiagonal, of course. 00:39:26.760 --> 00:39:31.250 And everybody recognizes that if I have a bidiagonal matrix-- 00:39:31.250 --> 00:39:35.530 call it A or A0 or whatever-- 00:39:35.530 --> 00:39:39.820 then what do you think about A transpose A? 00:39:39.820 --> 00:39:43.970 What would A transpose A-- if that was A, 00:39:43.970 --> 00:39:46.460 what could you tell me about A transpose A? 00:39:50.710 --> 00:39:57.570 Could you multiply matrices knowing where the non-zeros are 00:39:57.570 --> 00:40:00.210 in your head and get an idea of where-- 00:40:00.210 --> 00:40:06.000 and so if I have a bidiagonal matrix A, then implicitly 00:40:06.000 --> 00:40:10.590 in the SVD, I'm looking at A transpose A. 00:40:10.590 --> 00:40:15.950 And what would be true about A transpose A? 00:40:15.950 --> 00:40:18.750 It would be tridiagonal. 00:40:18.750 --> 00:40:25.410 So what I've done here and what I've done there just match up. 00:40:25.410 --> 00:40:29.700 You can operate-- if you don't want to change singular values, 00:40:29.700 --> 00:40:32.220 you can get all the way to here. 00:40:32.220 --> 00:40:35.100 But then, to find those singular values, that 00:40:35.100 --> 00:40:37.590 would involve A transpose A. It would 00:40:37.590 --> 00:40:44.010 be symmetric and tridiagonal, and then you'd be in that game. 00:40:44.010 --> 00:40:50.950 So those are the basic facts of eig and SVD 00:40:50.950 --> 00:40:56.720 for matrices of order up to 1,000, say. 00:40:56.720 --> 00:40:59.265 I'm not enough of an expert to know where-- 00:41:03.130 --> 00:41:09.670 Maybe higher, because in perfect math, 00:41:09.670 --> 00:41:12.930 it's going to take infinitely many steps 00:41:12.930 --> 00:41:16.050 or Abel would be very surprised. 00:41:16.050 --> 00:41:19.670 He would see you solving for eigenvalues an nth degree 00:41:19.670 --> 00:41:21.960 equation by a whole lot of little steps 00:41:21.960 --> 00:41:23.310 and getting them exactly right. 00:41:23.310 --> 00:41:24.480 That won't happen. 00:41:24.480 --> 00:41:29.160 But you get them within epsilon in a number 00:41:29.160 --> 00:41:32.170 of steps that's like N cubed. 00:41:32.170 --> 00:41:35.040 So that's pretty impressive. 00:41:35.040 --> 00:41:38.340 The eigenvalue problem is being "solved," 00:41:38.340 --> 00:41:47.020 in quotes, by a fast method that gets 00:41:47.020 --> 00:41:52.360 you a good answer within a tolerance in N cube steps. 00:41:55.290 --> 00:41:58.810 So that's great as long as N isn't too big. 00:41:58.810 --> 00:42:03.400 And then, when N is too big-- which, of course, happens-- 00:42:03.400 --> 00:42:04.910 you have to think again. 00:42:04.910 --> 00:42:12.940 So this method is a giant success up to large matrices, 00:42:12.940 --> 00:42:15.780 but then you have to think again. 00:42:15.780 --> 00:42:19.270 And what is involved in thinking again? 00:42:19.270 --> 00:42:24.030 Well, I guess more thinking. 00:42:24.030 --> 00:42:26.490 So what do you do if the matrix is bigger? 00:42:29.710 --> 00:42:35.960 I guess that Krylov would say, use my method. 00:42:35.960 --> 00:42:39.550 So Krylov would say-- especially if your matrix was sparse. 00:42:39.550 --> 00:42:43.190 Can we just remember what Krylov was? 00:42:43.190 --> 00:42:47.150 Krylov started with a vector b, multiplied it 00:42:47.150 --> 00:42:53.210 by A, multiplied that by A, and got up to, 00:42:53.210 --> 00:42:55.182 let's say, A to the 999b. 00:42:59.870 --> 00:43:01.460 So now he's got-- 00:43:01.460 --> 00:43:05.300 Krylov has got 1,000-dimensional space. 00:43:05.300 --> 00:43:06.950 He's got a basis for it-- 00:43:06.950 --> 00:43:13.320 1,000 vectors that span 1,000-dimensional space. 00:43:13.320 --> 00:43:17.580 And he'll look at the matrix A only on that space. 00:43:17.580 --> 00:43:20.120 In other words-- 00:43:20.120 --> 00:43:22.850 I won't go into detail about that. 00:43:22.850 --> 00:43:28.130 He restricts the matrix to this 1,000-dimensional space, 00:43:28.130 --> 00:43:29.990 and he hopes that it's captured. 00:43:29.990 --> 00:43:33.510 We hope that the eigenvector is almost-- 00:43:33.510 --> 00:43:35.720 is virtually in that space. 00:43:35.720 --> 00:43:38.450 And actually, I wouldn't go to-- 00:43:38.450 --> 00:43:39.910 let me take a 9 out of that. 00:43:42.510 --> 00:43:46.170 100-dimensional would probably catch the eigenvector. 00:43:46.170 --> 00:43:50.290 So if the eigenvector is virtually in this space, 00:43:50.290 --> 00:43:53.590 then we can look at a matrix of order 100. 00:43:53.590 --> 00:43:59.700 We can bring A down to just see its action on that space. 00:43:59.700 --> 00:44:02.350 And any piece of-- 00:44:02.350 --> 00:44:06.190 so here is a-- so I look at vectors 00:44:06.190 --> 00:44:14.980 v, which are some combination-- c1 b plus c2 Ab plus c3 A 00:44:14.980 --> 00:44:20.710 squared b, and c100 A to the 99th b. 00:44:25.540 --> 00:44:26.410 Plus an error. 00:44:29.710 --> 00:44:36.280 And I'm going to ignore that error because I've 00:44:36.280 --> 00:44:38.640 gone up to dimension 100. 00:44:38.640 --> 00:44:42.010 I'd probably say it's pretty safe to ignore that error. 00:44:42.010 --> 00:44:47.050 And then, in this space, just looking at the matrix 00:44:47.050 --> 00:44:51.460 A-- so wherever A to the 100th comes in, forget it. 00:44:51.460 --> 00:44:56.650 Just think about the matrix A as multiplying vectors 00:44:56.650 --> 00:44:58.810 of this kind in this space. 00:44:58.810 --> 00:45:03.940 Then I have 100 by 100 eigenvalue problem. 00:45:03.940 --> 00:45:12.370 And so the big matrix A is reduced to a matrix of size 100 00:45:12.370 --> 00:45:13.060 by-- 00:45:13.060 --> 00:45:15.130 do you see what I'm saying even? 00:45:15.130 --> 00:45:17.260 So I'm not giving the details. 00:45:20.010 --> 00:45:25.050 Think of a matrix A of size a million. 00:45:27.810 --> 00:45:30.760 And you apply it to Krylov vectors-- 00:45:30.760 --> 00:45:33.750 so I call them little k for a Krylov vector-- 00:45:36.540 --> 00:45:38.190 in this 100-dimensional space. 00:45:41.310 --> 00:45:47.740 So they have a million minus 100 0-components, 00:45:47.740 --> 00:45:49.330 you could say, this k. 00:45:49.330 --> 00:45:51.350 This is in the Krylov space. 00:45:51.350 --> 00:45:53.770 This is A, a million-- 00:45:53.770 --> 00:45:58.720 k, 100. 00:45:58.720 --> 00:46:02.030 It's a full-- so it's got a million components, 00:46:02.030 --> 00:46:04.650 but it's out of just 100-dimensional space. 00:46:04.650 --> 00:46:10.880 So when I multiply by A, it'll be mostly in-- 00:46:10.880 --> 00:46:13.910 partly in the Krylov space-- 00:46:13.910 --> 00:46:21.880 k100-- and a piece out of k100. 00:46:21.880 --> 00:46:25.740 And I just ignore that part of the matrix. 00:46:25.740 --> 00:46:27.770 So I have 100 by 100 problem, and I 00:46:27.770 --> 00:46:29.570 solve to find the eigenvalues. 00:46:29.570 --> 00:46:31.970 And they're a pretty good approximation 00:46:31.970 --> 00:46:35.840 to the eigenvalues, to the, hopefully, 00:46:35.840 --> 00:46:38.210 like the lowest 100 eigenvalues. 00:46:38.210 --> 00:46:40.940 I'd like to know that, but I might not 00:46:40.940 --> 00:46:43.910 be sure that this idea would give me 00:46:43.910 --> 00:46:49.060 the lowest 100-- the first 100 eigenvalues of the million, 00:46:49.060 --> 00:46:51.140 of the matrix of size a million. 00:46:51.140 --> 00:46:58.780 I'm just taking a few minutes here to wave hands 00:46:58.780 --> 00:47:03.730 about what Krylov idea would do. 00:47:03.730 --> 00:47:07.550 And I probably won't mention Krylov again in this semester. 00:47:07.550 --> 00:47:14.630 So what it can do is look at this particular type of space 00:47:14.630 --> 00:47:17.660 because we can get a basis for it quickly. 00:47:17.660 --> 00:47:21.320 Just multiply again and again by A. Then we 00:47:21.320 --> 00:47:24.700 can orthogonalize that basis. 00:47:24.700 --> 00:47:27.160 That's Gram-Schmidt in some form. 00:47:27.160 --> 00:47:29.620 We're always going back to Gram-Schmidt. 00:47:29.620 --> 00:47:31.570 Then I have 100 by 100-- 00:47:31.570 --> 00:47:34.900 I have a subspace of size 100. 00:47:34.900 --> 00:47:38.440 I look at what the matrix does in that space, 00:47:38.440 --> 00:47:41.800 and that I could look for-- 00:47:41.800 --> 00:47:45.190 I could find eigenvalues restricted to that space. 00:47:45.190 --> 00:47:47.620 They wouldn't be the perfect eigenvalues, 00:47:47.620 --> 00:47:50.820 but they would be accurate. 00:47:50.820 --> 00:47:54.630 So I didn't know it would take one class time to talk 00:47:54.630 --> 00:47:57.330 about finding eigenvalues and singular values, 00:47:57.330 --> 00:48:00.330 but we did some important things. 00:48:00.330 --> 00:48:05.250 We remembered that similarity is the thing to check, 00:48:05.250 --> 00:48:09.360 thing to preserve, because it doesn't change the eigenvalues. 00:48:09.360 --> 00:48:14.760 And then we-- for singular values, what was the thing? 00:48:14.760 --> 00:48:16.740 You could multiply left and right 00:48:16.740 --> 00:48:20.100 by different orthogonal matrices. 00:48:20.100 --> 00:48:24.150 And somehow, maybe that doesn't have an established name, 00:48:24.150 --> 00:48:31.070 multiplying left and right by a Q1 and a Q2 transpose. 00:48:31.070 --> 00:48:34.520 But the idea is clear, and that doesn't 00:48:34.520 --> 00:48:37.680 change the singular values. 00:48:37.680 --> 00:48:42.630 We're ready to move now into-- 00:48:42.630 --> 00:48:47.550 maybe our next step, which we don't spend a long time on, 00:48:47.550 --> 00:48:49.890 will be random sampling. 00:48:49.890 --> 00:48:53.700 What if your matrix is just way too big? 00:48:53.700 --> 00:49:00.000 So that's a very new idea, very different idea 00:49:00.000 --> 00:49:05.160 in numerical linear algebra, is just to sample the matrix. 00:49:05.160 --> 00:49:06.780 Could you believe that the answer 00:49:06.780 --> 00:49:10.300 is going to come out right just for a random sample? 00:49:10.300 --> 00:49:13.110 Well, the odds are in your favor. 00:49:13.110 --> 00:49:15.810 So that will be Wednesday, and then we 00:49:15.810 --> 00:49:23.190 have lots of new-- we'll move onward after that. 00:49:23.190 --> 00:49:24.360 See you Wednesday. 00:49:24.360 --> 00:49:25.910 Thanks.