WEBVTT 00:00:14.160 --> 00:00:14.860 Okay. 00:00:14.860 --> 00:00:21.760 This is the lecture on the singular value decomposition. 00:00:21.760 --> 00:00:27.030 But everybody calls it the SVD. 00:00:27.030 --> 00:00:34.350 So this is the final and best factorization of a matrix. 00:00:34.350 --> 00:00:37.270 Let me tell you what's coming. 00:00:37.270 --> 00:00:43.330 The factors will be, orthogonal matrix, diagonal matrix, 00:00:43.330 --> 00:00:45.610 orthogonal matrix. 00:00:45.610 --> 00:00:48.610 So it's things that we've seen before, 00:00:48.610 --> 00:00:53.180 these special good matrices, orthogonal diagonal. 00:00:53.180 --> 00:00:58.810 The new point is that we need two orthogonal matrices. 00:00:58.810 --> 00:01:02.720 A can be any matrix whatsoever. 00:01:02.720 --> 00:01:07.140 Any matrix whatsoever has this singular value decomposition, 00:01:07.140 --> 00:01:11.870 so a diagonal one in the middle, but I need two different -- 00:01:11.870 --> 00:01:17.030 probably different orthogonal matrices to be able to do this. 00:01:17.030 --> 00:01:17.560 Okay. 00:01:17.560 --> 00:01:22.030 And this factorization has jumped into importance 00:01:22.030 --> 00:01:27.240 and is properly, I think, maybe the bringing together 00:01:27.240 --> 00:01:29.620 of everything in this course. 00:01:29.620 --> 00:01:36.970 One thing we'll bring together is the very good family 00:01:36.970 --> 00:01:39.070 of matrices that we just studied, 00:01:39.070 --> 00:01:41.120 symmetric, positive, definite. 00:01:41.120 --> 00:01:43.730 Do you remember the stories with those guys? 00:01:43.730 --> 00:01:49.160 Because they were symmetric, their eigenvectors were 00:01:49.160 --> 00:01:53.880 orthogonal, so I could produce an orthogonal matrix -- 00:01:53.880 --> 00:01:56.070 this is my usual one. 00:01:56.070 --> 00:01:59.630 My usual one is the eigenvectors and eigenvalues In 00:01:59.630 --> 00:02:05.540 the symmetric case, the eigenvectors are orthogonal, 00:02:05.540 --> 00:02:09.430 so I've got the good -- my ordinary s has become 00:02:09.430 --> 00:02:12.180 an especially good Q. 00:02:12.180 --> 00:02:15.610 And positive definite, my ordinary lambda 00:02:15.610 --> 00:02:18.260 has become a positive lambda. 00:02:18.260 --> 00:02:24.420 So that's the singular value decomposition in case 00:02:24.420 --> 00:02:28.240 our matrix is symmetric positive definite -- 00:02:28.240 --> 00:02:30.650 in that case, I don't need two -- 00:02:30.650 --> 00:02:35.795 U and a V -- one orthogonal matrix will do for both sides. 00:02:38.970 --> 00:02:41.260 So this would be no good in general, 00:02:41.260 --> 00:02:45.190 because usually the eigenvector matrix isn't orthogonal. 00:02:45.190 --> 00:02:49.900 So that's not what I'm after. 00:02:49.900 --> 00:02:56.790 I'm looking for orthogonal times diagonal times orthogonal. 00:02:56.790 --> 00:03:00.250 And let me show you what that means and where it comes from. 00:03:00.250 --> 00:03:01.880 Okay. 00:03:01.880 --> 00:03:02.680 What does it mean? 00:03:05.440 --> 00:03:10.310 You remember the picture of any linear transformation. 00:03:10.310 --> 00:03:16.670 This was, like, the most important figure in 00:03:16.670 --> 00:03:19.810 And what I looking for now? 00:03:19.810 --> 00:03:26.020 A typical vector in the row space -- 00:03:26.020 --> 00:03:28.940 typical vector, let me call it v1, 00:03:28.940 --> 00:03:35.470 gets taken over to some vector in the column space, say u1. 00:03:35.470 --> 00:03:37.850 So u1 is Av1. 00:03:41.350 --> 00:03:43.320 Okay. 00:03:43.320 --> 00:03:48.800 Now, another vector gets taken over here somewhere. 00:03:48.800 --> 00:03:50.290 What I looking for? 00:03:50.290 --> 00:03:54.590 In this SVD, this singular value decomposition, 00:03:54.590 --> 00:03:59.950 what I'm looking for is an orthogonal basis here 00:03:59.950 --> 00:04:06.130 that gets knocked over into an orthogonal basis over there. 00:04:06.130 --> 00:04:11.810 See that's pretty special, to have an orthogonal basis 00:04:11.810 --> 00:04:18.180 in the row space that goes over into an orthogonal basis -- 00:04:18.180 --> 00:04:21.490 so this is like a right angle and this is a right angle -- 00:04:21.490 --> 00:04:25.700 into an orthogonal basis in the column space. 00:04:25.700 --> 00:04:30.940 So that's our goal, is to find -- 00:04:30.940 --> 00:04:34.230 do you see how things are coming together? 00:04:34.230 --> 00:04:37.590 First of all, can I find an orthogonal basis 00:04:37.590 --> 00:04:39.960 for this row space? 00:04:39.960 --> 00:04:41.100 Of course. 00:04:41.100 --> 00:04:44.210 No big deal to find an orthogonal basis. 00:04:44.210 --> 00:04:46.870 Graham Schmidt tells me how to do it. 00:04:46.870 --> 00:04:50.710 Start with any old basis and grind through Graham Schmidt, 00:04:50.710 --> 00:04:54.100 out comes an orthogonal basis. 00:04:54.100 --> 00:04:58.280 But then, if I just take any old orthogonal basis, then 00:04:58.280 --> 00:05:00.990 when I multiply by A, there's no reason 00:05:00.990 --> 00:05:04.770 why it should be orthogonal over here. 00:05:04.770 --> 00:05:06.800 So I'm looking for this special set 00:05:06.800 --> 00:05:13.850 up where A takes these basis vectors into orthogonal vectors 00:05:13.850 --> 00:05:14.780 over there. 00:05:14.780 --> 00:05:18.570 Now, you might have noticed that the null space 00:05:18.570 --> 00:05:19.400 I didn't include. 00:05:19.400 --> 00:05:22.400 Why don't I stick that in? 00:05:22.400 --> 00:05:25.770 You remember our usual figure had a little null space 00:05:25.770 --> 00:05:28.730 and a little null space. 00:05:28.730 --> 00:05:31.650 And those are no problems. 00:05:31.650 --> 00:05:34.000 Those null spaces are going to show up 00:05:34.000 --> 00:05:37.510 as zeroes on the diagonal of sigma, 00:05:37.510 --> 00:05:43.730 so that doesn't present any difficulty. 00:05:43.730 --> 00:05:46.250 Our difficulty is to find these. 00:05:46.250 --> 00:05:48.220 So do you see what this will mean? 00:05:48.220 --> 00:05:57.950 This will mean that A times these v-s, v1, v2, up to -- 00:05:57.950 --> 00:06:01.460 what's the dimension of this row space? 00:06:01.460 --> 00:06:03.180 Vr. 00:06:03.180 --> 00:06:07.390 Sorry, make that V a little smaller -- up to vr. 00:06:10.650 --> 00:06:12.990 So that's -- 00:06:12.990 --> 00:06:16.680 Av1 is going to be the first column, 00:06:16.680 --> 00:06:20.250 so here's what I'm achieving. 00:06:20.250 --> 00:06:24.390 Oh, I'm not only going to make these orthogonal, 00:06:24.390 --> 00:06:27.030 but why not make them orthonormal? 00:06:27.030 --> 00:06:28.880 Make them unit vectors. 00:06:28.880 --> 00:06:34.380 So maybe the unit vector is here, is the u1, 00:06:34.380 --> 00:06:38.510 and this might be a multiple of it. 00:06:38.510 --> 00:06:45.560 So really, what's happening is Av1 is some multiple of u1, 00:06:45.560 --> 00:06:46.660 right? 00:06:46.660 --> 00:06:50.040 These guys will be unit vectors and they'll 00:06:50.040 --> 00:06:52.940 go over into multiples of unit vectors 00:06:52.940 --> 00:06:56.820 and the multiple I'm not going to call lambda anymore. 00:06:56.820 --> 00:06:58.090 I'm calling it sigma. 00:06:58.090 --> 00:07:01.480 So that's the number -- the stretching number. 00:07:01.480 --> 00:07:06.000 And similarly, Av2 is sigma two u2. 00:07:06.000 --> 00:07:09.000 This is my goal. 00:07:09.000 --> 00:07:13.730 And now I want to express that goal in matrix language. 00:07:13.730 --> 00:07:15.340 That's the usual step. 00:07:15.340 --> 00:07:18.400 Think of what you want and then express it 00:07:18.400 --> 00:07:20.420 as a matrix multiplication. 00:07:20.420 --> 00:07:25.800 So Av1 is sigma one u1 -- 00:07:25.800 --> 00:07:27.960 actually, here we go. 00:07:27.960 --> 00:07:30.710 Let me pull out these -- 00:07:30.710 --> 00:07:36.765 u1, u2 to ur and then a matrix with the sigmas. 00:07:40.460 --> 00:07:45.970 Everything now is going to be in that little part 00:07:45.970 --> 00:07:46.720 of the blackboard. 00:07:46.720 --> 00:07:52.880 Do you see that this equation says what I'm 00:07:52.880 --> 00:07:55.740 trying to do with my figure. 00:07:55.740 --> 00:08:00.770 A times the first basis vector should be sigma one times 00:08:00.770 --> 00:08:04.580 the other basis -- the other first basis vector. 00:08:04.580 --> 00:08:08.150 These are the basis vectors in the row space, 00:08:08.150 --> 00:08:11.470 these are the basis vectors in the column space 00:08:11.470 --> 00:08:14.680 and these are the multiplying factors. 00:08:14.680 --> 00:08:23.660 So Av2 is sigma two times u2, Avr is sigma r times ur. 00:08:23.660 --> 00:08:27.170 And then we've got a whole lot of zeroes and maybe some zeroes 00:08:27.170 --> 00:08:31.480 at the end, but that's the heart of it. 00:08:31.480 --> 00:08:34.064 And now if I express that in -- 00:08:40.000 --> 00:08:43.350 as matrices, because you knew that was coming -- 00:08:43.350 --> 00:08:45.140 that's what I have. 00:08:45.140 --> 00:08:51.520 So, this is my goal, to find an orthogonal basis 00:08:51.520 --> 00:08:55.500 in the orthonormal, even -- basis in the row space 00:08:55.500 --> 00:09:00.420 and an orthonormal basis in the column space so that I've sort 00:09:00.420 --> 00:09:02.590 of diagonalized the matrix. 00:09:02.590 --> 00:09:05.530 The matrix A is, like, getting converted 00:09:05.530 --> 00:09:08.660 to this diagonal matrix sigma. 00:09:08.660 --> 00:09:13.140 And you notice that usually I have to allow myself 00:09:13.140 --> 00:09:15.980 two different bases. 00:09:15.980 --> 00:09:19.230 My little comment about symmetric positive definite 00:09:19.230 --> 00:09:24.860 was the one case where it's A Q equal Q sigma, 00:09:24.860 --> 00:09:27.430 where V and U are the same Q. 00:09:27.430 --> 00:09:32.310 But mostly, you know, I'm going to take a matrix like -- oh, 00:09:32.310 --> 00:09:38.240 let me take a matrix like four four minus three three. 00:09:38.240 --> 00:09:39.540 Okay. 00:09:39.540 --> 00:09:42.420 There's a two by two matrix. 00:09:42.420 --> 00:09:48.430 It's invertible, so it has rank two. 00:09:48.430 --> 00:09:51.210 So I'm going to look for two vectors, 00:09:51.210 --> 00:09:55.230 v1 and v2 in the row space, and U -- 00:09:55.230 --> 00:10:00.770 so I'm going to look for v1, v2 in the row space, 00:10:00.770 --> 00:10:05.350 which of course is R^2. 00:10:05.350 --> 00:10:10.280 And I'm going to look for u1, u2 in the column space, 00:10:10.280 --> 00:10:15.620 which of course is also R^2, and I'm going to look for numbers 00:10:15.620 --> 00:10:21.160 sigma one and sigma two so that it all comes out right. 00:10:21.160 --> 00:10:26.060 So these guys are orthonormal, these guys are orthonormal 00:10:26.060 --> 00:10:28.820 and these are the scaling factors. 00:10:28.820 --> 00:10:33.510 So I'll do that example as soon as I get the matrix 00:10:33.510 --> 00:10:35.030 picture straight. 00:10:35.030 --> 00:10:36.280 Okay. 00:10:36.280 --> 00:10:39.380 Do you see that this expresses what I want? 00:10:39.380 --> 00:10:45.360 Can I just say two words about null spaces? 00:10:45.360 --> 00:10:52.010 If there's some null space, then we 00:10:52.010 --> 00:10:56.080 want to stick in a basis for those, for that. 00:10:56.080 --> 00:11:02.700 So here comes a basis for the null space, v(r+1) down to vm. 00:11:02.700 --> 00:11:06.590 So if we only had an r dimensional row space 00:11:06.590 --> 00:11:11.160 and the other n-r dimensions were in the null space -- okay, 00:11:11.160 --> 00:11:14.360 we'll take an orthogonal -- orthonormal basis there. 00:11:14.360 --> 00:11:15.430 No problem. 00:11:15.430 --> 00:11:18.500 And then we'll just get zeroes. 00:11:18.500 --> 00:11:21.700 So, actually, w- those zeroes will come out 00:11:21.700 --> 00:11:25.040 on the diagonal matrix. 00:11:25.040 --> 00:11:33.250 So I'll complete that to an orthonormal basis for the whole 00:11:33.250 --> 00:11:35.420 space, R^m. 00:11:35.420 --> 00:11:38.080 I complete this to an orthonormal basis for the whole 00:11:38.080 --> 00:11:42.420 space R^n and I complete that with zeroes. 00:11:42.420 --> 00:11:46.870 Null spaces are no problem here. 00:11:46.870 --> 00:11:52.570 So really the true problem is in a matrix like that, 00:11:52.570 --> 00:11:56.170 which isn't symmetric, so I can't use its eigenvectors, 00:11:56.170 --> 00:11:58.110 they're not orthogonal -- 00:11:58.110 --> 00:12:02.740 but somehow I have to get these orthogonal -- in fact, 00:12:02.740 --> 00:12:09.470 orthonormal guys that make it work. 00:12:09.470 --> 00:12:13.990 I have to find these orthonormal guys, these orthonormal guys 00:12:13.990 --> 00:12:22.960 and I want Av1 to be sigma one u1 and Av2 to be sigma two u2. 00:12:22.960 --> 00:12:23.460 Okay. 00:12:26.790 --> 00:12:28.260 That's my goal. 00:12:28.260 --> 00:12:34.310 Here's the matrices that are going to get me there. 00:12:34.310 --> 00:12:36.500 Now these are orthogonal matrices. 00:12:36.500 --> 00:12:41.610 I can put that -- if I multiply on both sides by V inverse, 00:12:41.610 --> 00:12:47.740 I have A equals U sigma V inverse, 00:12:47.740 --> 00:12:53.230 and of course you know the other way I can write V inverse. 00:12:53.230 --> 00:12:57.200 This is one of those square orthogonal matrices, 00:12:57.200 --> 00:13:02.731 so it's the same as U sigma V transpose. 00:13:02.731 --> 00:13:03.230 Okay. 00:13:06.020 --> 00:13:08.900 Here's my problem. 00:13:08.900 --> 00:13:14.580 I've got two orthogonal matrices here. 00:13:14.580 --> 00:13:17.330 And I don't want to find them both at once. 00:13:17.330 --> 00:13:21.120 So I want to cook up some expression that 00:13:21.120 --> 00:13:26.630 will make the Us disappear. 00:13:26.630 --> 00:13:29.010 I would like to make the Us disappear and leave me 00:13:29.010 --> 00:13:31.070 only with the Vs. 00:13:31.070 --> 00:13:34.200 And here's how to do it. 00:13:34.200 --> 00:13:37.520 It's the same combination that keeps showing up 00:13:37.520 --> 00:13:41.040 whenever we have a general rectangular matrix, 00:13:41.040 --> 00:13:46.640 then it's A transpose A, that's the great matrix. 00:13:46.640 --> 00:13:48.760 That's the great matrix. 00:13:48.760 --> 00:13:50.530 That's the matrix that's symmetric, 00:13:50.530 --> 00:13:53.910 and in fact positive definite or at least 00:13:53.910 --> 00:13:55.240 positive semi-definite. 00:13:55.240 --> 00:13:58.300 This is the matrix with nice properties, so let's see what 00:13:58.300 --> 00:13:59.230 will it be? 00:13:59.230 --> 00:14:03.100 So if I took the transpose, then, I would have -- 00:14:03.100 --> 00:14:06.140 A transpose A will be what? 00:14:06.140 --> 00:14:06.920 What do I have? 00:14:06.920 --> 00:14:12.580 If I transpose that I have V sigma transpose U transpose, 00:14:12.580 --> 00:14:14.900 that's the A transpose. 00:14:14.900 --> 00:14:18.350 Now the A -- 00:14:18.350 --> 00:14:19.290 and what have I got? 00:14:21.930 --> 00:14:25.960 Looks like worse, because it's got six things now together, 00:14:25.960 --> 00:14:30.660 but it's going to collapse into something good. 00:14:30.660 --> 00:14:34.300 What does U transpose U collapse into? 00:14:34.300 --> 00:14:35.770 I, the identity. 00:14:35.770 --> 00:14:37.330 So that's the key point. 00:14:37.330 --> 00:14:42.050 This is the identity and we don't have U anymore. 00:14:42.050 --> 00:14:44.380 And sigma transpose times sigma, those 00:14:44.380 --> 00:14:48.300 are diagonal matrixes, so their product is just 00:14:48.300 --> 00:14:50.590 going to have sigma squareds on the diagonal. 00:14:50.590 --> 00:14:52.650 So do you see what we've got here? 00:14:52.650 --> 00:14:57.440 This is V times this easy matrix sigma 00:14:57.440 --> 00:15:03.080 one squared sigma two squared times V transpose. 00:15:03.080 --> 00:15:05.840 This is the A transpose A. 00:15:05.840 --> 00:15:07.590 This is -- let me copy down -- 00:15:07.590 --> 00:15:09.185 A transpose A is that. 00:15:12.760 --> 00:15:14.170 Us are out of the picture, now. 00:15:14.170 --> 00:15:17.900 I'm only having to choose the Vs, and what are these Vs? 00:15:17.900 --> 00:15:19.370 And what are these sigmas? 00:15:19.370 --> 00:15:26.750 Do you know what the Vs are? 00:15:26.750 --> 00:15:28.650 They're the eigenvectors that -- see, 00:15:28.650 --> 00:15:34.120 this is a perfect eigenvector, eigenvalue, 00:15:34.120 --> 00:15:41.920 Q lambda Q transpose for the matrix A transpose A. 00:15:41.920 --> 00:15:45.020 A itself is nothing special. 00:15:45.020 --> 00:15:48.620 But A transpose A will be special. 00:15:48.620 --> 00:15:50.700 It'll be symmetric positive definite, 00:15:50.700 --> 00:15:55.340 so this will be its eigenvectors and this'll be its eigenvalues. 00:15:55.340 --> 00:15:59.100 And the eigenvalues'll be positive because this thing's 00:15:59.100 --> 00:16:02.400 positive definite. 00:16:02.400 --> 00:16:04.650 So this is my method. 00:16:04.650 --> 00:16:07.310 This tells me what the Vs are. 00:16:07.310 --> 00:16:09.210 And how I going to find the Us? 00:16:11.730 --> 00:16:18.570 Well, one way would be to look at A A transpose. 00:16:18.570 --> 00:16:21.650 Multiply A by A transpose in the opposite order. 00:16:21.650 --> 00:16:24.340 That will stick the Vs in the middle, 00:16:24.340 --> 00:16:27.130 knock them out, and leave me with the Us. 00:16:27.130 --> 00:16:29.620 So here's the overall picture, then. 00:16:29.620 --> 00:16:36.270 The Vs are the eigenvectors of A transpose A. 00:16:36.270 --> 00:16:38.180 The Us are the eigenvectors of A A 00:16:38.180 --> 00:16:39.860 transpose, which are different. 00:16:39.860 --> 00:16:43.940 And the sigmas are the square roots 00:16:43.940 --> 00:16:48.730 of these and the positive square roots, 00:16:48.730 --> 00:16:50.580 so we have positive sigmas. 00:16:50.580 --> 00:16:52.780 Let me do it for that example. 00:16:52.780 --> 00:16:56.570 This is really what you should know 00:16:56.570 --> 00:17:01.460 and be able to do for the SVD. 00:17:01.460 --> 00:17:02.520 Okay. 00:17:02.520 --> 00:17:03.740 Let me take that matrix. 00:17:03.740 --> 00:17:06.550 So what's my first step? 00:17:06.550 --> 00:17:12.191 Compute A transpose A, because I want its eigenvectors. 00:17:12.191 --> 00:17:12.690 Okay. 00:17:12.690 --> 00:17:16.690 So I have to compute A transpose A. 00:17:16.690 --> 00:17:22.240 So A transpose is four four minus three three, 00:17:22.240 --> 00:17:26.970 and A is four four minus three three, 00:17:26.970 --> 00:17:30.810 and I do that multiplication and I get sixteen -- 00:17:30.810 --> 00:17:32.930 I get twenty five -- 00:17:32.930 --> 00:17:36.300 I get sixteen minus nine -- 00:17:36.300 --> 00:17:37.870 is that seven? 00:17:37.870 --> 00:17:40.260 And it better come out symmetric. 00:17:40.260 --> 00:17:43.190 And -- oh, okay, and then it comes out 25. 00:17:43.190 --> 00:17:43.690 Okay. 00:17:47.360 --> 00:17:52.030 So, I want its eigenvectors and its eigenvalues. 00:17:52.030 --> 00:17:55.680 Its eigenvectors will be the Vs, its eigenvalues 00:17:55.680 --> 00:17:58.990 will be the squares of the sigmas. 00:17:58.990 --> 00:17:59.570 Okay. 00:17:59.570 --> 00:18:04.900 What are the eigenvalues and eigenvectors of this guy? 00:18:04.900 --> 00:18:10.690 Have you seen that two by two example enough to recognize 00:18:10.690 --> 00:18:16.970 that the eigenvectors are -- that one one is an eigenvector? 00:18:16.970 --> 00:18:19.210 So this here is A transpose A. 00:18:19.210 --> 00:18:22.300 I'm looking for its eigenvectors. 00:18:22.300 --> 00:18:27.620 So its eigenvectors, I think, are one one and one minus one, 00:18:27.620 --> 00:18:29.800 because if I multiply that matrix 00:18:29.800 --> 00:18:32.390 by one one, what do I get? 00:18:32.390 --> 00:18:37.720 If I multiply that matrix by one one, I get 32 32, 00:18:37.720 --> 00:18:41.530 which is 32 of one one. 00:18:41.530 --> 00:18:45.240 So there's the first eigenvector, 00:18:45.240 --> 00:18:49.360 and there's the eigenvalue for A transpose A. 00:18:49.360 --> 00:18:59.120 So I'm going to take its square root for sigma. 00:18:59.120 --> 00:18:59.620 Okay. 00:18:59.620 --> 00:19:01.310 What's the eigenvector that goes -- 00:19:01.310 --> 00:19:03.410 eigenvalue that goes with this one? 00:19:03.410 --> 00:19:05.730 If I do that multiplication, what do I get? 00:19:05.730 --> 00:19:12.420 I get some multiple of one minus one, and what is that multiple? 00:19:12.420 --> 00:19:13.250 Looks like 18. 00:19:16.470 --> 00:19:17.340 Okay. 00:19:17.340 --> 00:19:20.880 So those are the two eigenvectors, but -- oh, 00:19:20.880 --> 00:19:24.300 just a moment, I didn't normalize them. 00:19:24.300 --> 00:19:27.310 To make everything absolutely right, 00:19:27.310 --> 00:19:30.180 I ought to normalize these eigenvectors, 00:19:30.180 --> 00:19:33.370 divide by their length, square root of two. 00:19:33.370 --> 00:19:42.890 So all these guys should be true unit vectors and, of course, 00:19:42.890 --> 00:19:46.900 that normalization didn't change the 32 and the 18. 00:19:46.900 --> 00:19:48.500 Okay. 00:19:48.500 --> 00:19:51.980 So I'm happy with the Vs. 00:19:51.980 --> 00:19:53.380 Here are the Vs. 00:19:53.380 --> 00:19:57.490 So now let me put together the pieces here. 00:19:57.490 --> 00:19:59.340 Here's my A. 00:19:59.340 --> 00:20:01.060 Here's my A. 00:20:01.060 --> 00:20:03.550 Let me write down A again. 00:20:07.860 --> 00:20:16.150 If life is right, we should get U, which I don't yet know -- 00:20:16.150 --> 00:20:19.870 U I don't yet know, sigma I do now know. 00:20:19.870 --> 00:20:21.040 What's sigma? 00:20:21.040 --> 00:20:24.080 So I'm looking for a U sigma V transpose. 00:20:24.080 --> 00:20:28.115 U, the diagonal guy and V transpose. 00:20:31.500 --> 00:20:32.030 Okay. 00:20:32.030 --> 00:20:33.740 Let's just see that come out right. 00:20:33.740 --> 00:20:36.780 So what are the sigmas? 00:20:36.780 --> 00:20:39.130 They're the square roots of these things. 00:20:39.130 --> 00:20:43.820 So square root of 32 and square root of 18. 00:20:47.900 --> 00:20:49.120 Zero zero. 00:20:49.120 --> 00:20:50.210 Okay. 00:20:50.210 --> 00:20:52.040 What are the Vs? 00:20:52.040 --> 00:20:53.810 They're these two. 00:20:53.810 --> 00:20:56.900 And I have to transpose -- 00:20:56.900 --> 00:20:59.450 maybe that just leaves me with ones -- 00:20:59.450 --> 00:21:03.790 with one over square root of two in that row and the other one 00:21:03.790 --> 00:21:06.870 is one over square root of two minus one 00:21:06.870 --> 00:21:08.010 over square root of two. 00:21:11.590 --> 00:21:15.030 Now finally, I've got to know the Us. 00:21:15.030 --> 00:21:18.570 Well, actually, one way to do -- since I now know all the other 00:21:18.570 --> 00:21:21.100 pieces, I could put those together and figure out what 00:21:21.100 --> 00:21:22.110 the Us are. 00:21:22.110 --> 00:21:25.780 But let me do it the A A transpose way. 00:21:25.780 --> 00:21:26.280 Okay. 00:21:26.280 --> 00:21:27.380 Find the Us now. 00:21:31.390 --> 00:21:33.070 u1 and u2. 00:21:33.070 --> 00:21:34.650 And what are they? 00:21:37.220 --> 00:21:41.240 I look at A A transpose -- 00:21:41.240 --> 00:21:47.230 so A is supposed to be U sigma V transpose, and then 00:21:47.230 --> 00:21:52.652 when I transpose that I get V sigma transpose U transpose. 00:21:57.210 --> 00:21:59.160 So I'm just doing it in the opposite order, 00:21:59.160 --> 00:22:03.470 A times A transpose, and what's the good part here? 00:22:03.470 --> 00:22:10.480 That in the middle, V transpose V is going to be the identity. 00:22:10.480 --> 00:22:14.990 So this is just U sigma sigma transpose, 00:22:14.990 --> 00:22:22.340 that's some diagonal matrix with sigma squareds and U transpose. 00:22:22.340 --> 00:22:24.480 So what I seeing here? 00:22:24.480 --> 00:22:29.280 I'm seeing here, again, a symmetric positive definite 00:22:29.280 --> 00:22:32.000 or at least semi-definite matrix and I'm 00:22:32.000 --> 00:22:36.600 seeing its eigenvectors and its eigenvalues. 00:22:36.600 --> 00:22:41.590 So if I compute A A transpose, its eigenvectors 00:22:41.590 --> 00:22:44.060 will be the things that go into U. 00:22:44.060 --> 00:22:47.470 Okay, so I need to compute A A transpose. 00:22:47.470 --> 00:22:50.970 I guess I'm going to have to go -- 00:22:50.970 --> 00:22:53.450 can I move that up just a little? 00:22:53.450 --> 00:22:56.580 Maybe a little more and do A A transpose. 00:22:59.880 --> 00:23:01.820 So what's A? 00:23:01.820 --> 00:23:05.930 Four four minus three and three. 00:23:05.930 --> 00:23:07.550 And what's A transpose? 00:23:07.550 --> 00:23:10.310 Four four minus three and three. 00:23:10.310 --> 00:23:15.530 And when I do that multiplication, what do I get? 00:23:15.530 --> 00:23:18.750 Sixteen and sixteen, thirty two. 00:23:18.750 --> 00:23:21.630 Uh, that one comes out zero. 00:23:21.630 --> 00:23:26.940 Oh, so this is a lucky case and that one comes out 18. 00:23:26.940 --> 00:23:31.560 So this is an accident that A A transpose 00:23:31.560 --> 00:23:38.030 happens to come out diagonal, so we know easily its eigenvectors 00:23:38.030 --> 00:23:38.990 and eigenvalues. 00:23:38.990 --> 00:23:43.130 So its eigenvectors -- what's the first eigenvector for this 00:23:43.130 --> 00:23:45.150 A A transpose matrix? 00:23:45.150 --> 00:23:49.940 It's just one zero, and when I do that multiplication, 00:23:49.940 --> 00:23:54.020 I get 32 times one zero. 00:23:54.020 --> 00:23:57.380 And the other eigenvector is just zero one 00:23:57.380 --> 00:24:00.350 and when I multiply by that I get 18. 00:24:00.350 --> 00:24:04.720 So this is A A transpose. 00:24:04.720 --> 00:24:08.910 Multiplying that gives me the 32 A A transpose. 00:24:08.910 --> 00:24:14.860 Multiplying this guy gives me First of all, 00:24:14.860 --> 00:24:18.590 I got 32 and 18 again. 00:24:18.590 --> 00:24:19.740 Am I surprised? 00:24:19.740 --> 00:24:24.420 You know, it's clearly not an accident. 00:24:24.420 --> 00:24:29.190 The eigenvalues of A A transpose were exactly the same 00:24:29.190 --> 00:24:37.820 as the eigenvalues of -- this one was A transpose A. 00:24:37.820 --> 00:24:40.140 That's no surprise at all. 00:24:40.140 --> 00:24:47.140 The eigenvalues of A B are the same as the eigenvalues of B A. 00:24:47.140 --> 00:24:50.530 That's a very nice fact, that eigenvalues 00:24:50.530 --> 00:24:55.310 stay the same if I switch the order of multiplication. 00:24:55.310 --> 00:25:01.650 So no surprise to see 32 and What I learned -- 00:25:01.650 --> 00:25:05.550 first the check that things were numerically correct, 00:25:05.550 --> 00:25:07.950 but now I've learned these eigenvectors, 00:25:07.950 --> 00:25:11.980 and actually they're about as nice as can be. 00:25:11.980 --> 00:25:16.045 They're the best orthogonal matrix, just the identity. 00:25:18.511 --> 00:25:19.010 Okay. 00:25:21.940 --> 00:25:26.400 So my claim is that it ought to all fit together, 00:25:26.400 --> 00:25:31.560 that these numbers should come out right. 00:25:31.560 --> 00:25:33.440 The numbers should come out right 00:25:33.440 --> 00:25:41.070 because the matrix multiplications use 00:25:41.070 --> 00:25:42.450 the properties that we want. 00:25:42.450 --> 00:25:42.970 Okay. 00:25:42.970 --> 00:25:44.370 Shall we just check that? 00:25:44.370 --> 00:25:47.020 Here's the identity, so not doing anything -- 00:25:47.020 --> 00:25:50.530 square root of 32 is multiplying that row, 00:25:50.530 --> 00:25:53.670 so that square root of 32 divided by square root of two 00:25:53.670 --> 00:25:58.150 means square root of 16, four, correct? 00:25:58.150 --> 00:26:01.680 And square root of 18 is divided by square root of two, 00:26:01.680 --> 00:26:07.570 so that leaves me square root of 9, which is three, but -- 00:26:07.570 --> 00:26:11.100 well, Professor Strang, you see the problem? 00:26:11.100 --> 00:26:12.740 Why is that -- 00:26:12.740 --> 00:26:13.240 okay. 00:26:13.240 --> 00:26:16.790 Why I getting minus three three here 00:26:16.790 --> 00:26:19.965 and here I'm getting three minus three? 00:26:24.640 --> 00:26:26.240 Phooey. 00:26:26.240 --> 00:26:27.105 I don't know why. 00:26:30.980 --> 00:26:34.650 It shouldn't have happened, but it did. 00:26:34.650 --> 00:26:38.402 Now, okay, you could say, well, just -- 00:26:41.300 --> 00:26:43.560 the eigenvector there could have -- 00:26:43.560 --> 00:26:47.410 I could have had the minus sign here for that eigenvector, 00:26:47.410 --> 00:26:49.710 but I'm not happy about that. 00:26:49.710 --> 00:26:50.210 Hmm. 00:26:50.210 --> 00:26:50.710 Okay. 00:26:55.740 --> 00:26:58.480 So I realize there's a little catch here somewhere 00:26:58.480 --> 00:27:02.630 and I may not see it until Wednesday. 00:27:02.630 --> 00:27:04.830 Which then gives you a very important reason 00:27:04.830 --> 00:27:09.930 to come back on Wednesday, to catch that sine difference. 00:27:09.930 --> 00:27:14.310 So what did I do illegally? 00:27:14.310 --> 00:27:22.550 I think I put the eigenvectors in that matrix V transpose -- 00:27:22.550 --> 00:27:24.160 okay, I'm going to have to think. 00:27:24.160 --> 00:27:29.460 Why did that come out with with the opposite sines? 00:27:29.460 --> 00:27:30.510 So you see -- 00:27:30.510 --> 00:27:35.000 I mean, if I had a minus there, I would be all right, 00:27:35.000 --> 00:27:36.490 but I don't want that. 00:27:36.490 --> 00:27:45.331 I want positive entries down the diagonal of sigma squared. 00:27:45.331 --> 00:27:45.830 Okay. 00:27:45.830 --> 00:27:51.590 It'll come to me, but, I'm going to leave this example 00:27:51.590 --> 00:27:57.600 to finish. 00:27:57.600 --> 00:27:58.920 Okay. 00:27:58.920 --> 00:28:02.560 And the beauty of, these sliding boards 00:28:02.560 --> 00:28:05.780 is I can make that go away. 00:28:05.780 --> 00:28:10.910 Can I,-- let me not do it, though, yet. 00:28:10.910 --> 00:28:15.090 Let me take a second example. 00:28:15.090 --> 00:28:18.720 Let me take a second example where the matrix is singular. 00:28:21.940 --> 00:28:24.090 So rank one. 00:28:24.090 --> 00:28:32.390 Okay, so let me take as an example two, 00:28:32.390 --> 00:28:38.770 where my matrix A is going to be rectangular again -- 00:28:38.770 --> 00:28:43.220 let me just make it four three eight six. 00:28:47.590 --> 00:28:48.090 Okay. 00:28:48.090 --> 00:28:50.830 That's a rank one matrix. 00:28:50.830 --> 00:28:57.430 So that has a null space and only a one dimensional row 00:28:57.430 --> 00:28:59.370 space and column space. 00:28:59.370 --> 00:29:05.510 So actually, my picture becomes easy for this matrix, 00:29:05.510 --> 00:29:09.510 because what's my row space for this one? 00:29:09.510 --> 00:29:12.450 So this is two by two. 00:29:12.450 --> 00:29:17.000 So my pictures are both two dimensional. 00:29:17.000 --> 00:29:22.390 My row space is all multiples of that vector four three. 00:29:22.390 --> 00:29:24.760 So the whole -- the row space is just a line, right? 00:29:27.770 --> 00:29:29.380 That's the row space. 00:29:29.380 --> 00:29:33.050 And the null space, of course, is the perpendicular line. 00:29:33.050 --> 00:29:46.480 So the row space for this matrix is multiples of four three. 00:29:46.480 --> 00:29:47.680 Typical row. 00:29:47.680 --> 00:29:48.520 Okay. 00:29:48.520 --> 00:29:50.060 What's the column space? 00:29:50.060 --> 00:29:55.980 The columns are all multiples of four eight, three six, one two. 00:29:55.980 --> 00:30:00.315 The column space, then, goes in, like, this direction. 00:30:03.040 --> 00:30:07.490 So the column space -- 00:30:07.490 --> 00:30:09.650 when I look at those columns, the column space -- 00:30:09.650 --> 00:30:12.660 so it's only one dimensional, because the rank is one. 00:30:12.660 --> 00:30:21.480 It's multiples of four eight. 00:30:21.480 --> 00:30:22.370 Okay. 00:30:22.370 --> 00:30:26.360 And what's the null space of A transpose? 00:30:26.360 --> 00:30:30.270 It's the perpendicular guy. 00:30:30.270 --> 00:30:35.150 So this was the null space of A and this is 00:30:35.150 --> 00:30:38.921 the null space of A transpose. 00:30:38.921 --> 00:30:39.420 Okay. 00:30:42.260 --> 00:30:48.650 What I want to say here is that choosing these orthogonal bases 00:30:48.650 --> 00:30:53.750 for the row space and the column space is, like, no problem. 00:30:53.750 --> 00:30:55.800 They're only one dimensional. 00:30:55.800 --> 00:30:58.020 So what should V be? 00:30:58.020 --> 00:31:02.620 V should be -- v1, but -- yes, v1, rather -- 00:31:02.620 --> 00:31:05.540 v1 is supposed to be a unit vector. 00:31:05.540 --> 00:31:08.640 There's only one v1 to choose here, 00:31:08.640 --> 00:31:11.210 only one dimension in the row space. 00:31:11.210 --> 00:31:13.800 I just want to make it a unit vector. 00:31:13.800 --> 00:31:17.180 So v1 will be -- 00:31:17.180 --> 00:31:24.570 it'll be this vector, but made into a unit vector, so four -- 00:31:24.570 --> 00:31:25.970 point eight point six. 00:31:28.660 --> 00:31:30.670 Four fifths, three fifths. 00:31:30.670 --> 00:31:33.040 And what will be u1? 00:31:33.040 --> 00:31:35.740 u1 will be the unit vector there. 00:31:35.740 --> 00:31:40.820 So I want to turn four eight or one two into a unit vector, 00:31:40.820 --> 00:31:44.150 so u1 will be -- 00:31:44.150 --> 00:31:47.760 let's see, if it's one two, then what multiple of one two 00:31:47.760 --> 00:31:49.230 do I want? 00:31:49.230 --> 00:31:51.240 That has length square root of five, 00:31:51.240 --> 00:31:53.460 so I have to divide by square root of five. 00:31:56.140 --> 00:31:58.470 Let me complete the singular value 00:31:58.470 --> 00:32:01.660 decomposition for this matrix. 00:32:01.660 --> 00:32:09.540 So this matrix, four three eight six, is -- 00:32:09.540 --> 00:32:11.520 so I know what u1 -- 00:32:11.520 --> 00:32:19.570 here's A and I want to get U the basis in the column space. 00:32:19.570 --> 00:32:24.020 And it has to start with this guy, one 00:32:24.020 --> 00:32:27.290 over square root of five two over square root of five. 00:32:30.520 --> 00:32:36.350 Then I want the sigma. 00:32:36.350 --> 00:32:37.540 Okay. 00:32:37.540 --> 00:32:40.370 What are we expecting now for sigma? 00:32:44.400 --> 00:32:47.010 This is only a rank one matrix. 00:32:47.010 --> 00:32:51.720 We're only expecting a sigma one, which I have to find, 00:32:51.720 --> 00:32:55.040 but zeroes here. 00:32:55.040 --> 00:32:55.540 Okay. 00:32:55.540 --> 00:32:57.420 So what's sigma one? 00:32:57.420 --> 00:33:02.950 It should be the -- 00:33:02.950 --> 00:33:05.480 where did these sigmas come from? 00:33:05.480 --> 00:33:08.370 They came from A transpose A, so I -- 00:33:08.370 --> 00:33:10.750 can I do that little calculation over here? 00:33:10.750 --> 00:33:19.840 A transpose A is four three -- four three eight six times four 00:33:19.840 --> 00:33:23.340 three eight six. 00:33:23.340 --> 00:33:26.240 This had better -- this is a rank one matrix, 00:33:26.240 --> 00:33:29.170 this is going to be -- the whole thing will have rank one, 00:33:29.170 --> 00:33:40.240 that's 16 and 64 is 80, 12 and 48 is 60, 12 and 48 is 60, 00:33:40.240 --> 00:33:43.880 9 and 36 is 45. 00:33:43.880 --> 00:33:45.520 Okay. 00:33:45.520 --> 00:33:47.340 It's a rank one matrix. 00:33:47.340 --> 00:33:48.230 Of course. 00:33:48.230 --> 00:33:52.450 Every row is a multiple of four three. 00:33:52.450 --> 00:33:56.820 And what's the eigen -- what are the eigenvalues of that matrix? 00:33:56.820 --> 00:33:59.730 So this is the calculation -- this is like practicing, 00:33:59.730 --> 00:34:00.230 now. 00:34:00.230 --> 00:34:04.510 What are the eigenvalues of this rank one matrix? 00:34:04.510 --> 00:34:08.389 Well, tell me one eigenvalue, since the rank is only one, 00:34:08.389 --> 00:34:11.920 one eigenvalue is going to be zero. 00:34:11.920 --> 00:34:14.320 And then you know that the other eigenvalue 00:34:14.320 --> 00:34:19.330 is going to be a hundred and twenty five. 00:34:19.330 --> 00:34:22.760 So that's sigma squared, right, in A transpose A. 00:34:22.760 --> 00:34:28.780 So this will be the square root of a hundred and twenty five. 00:34:28.780 --> 00:34:34.900 And then finally, the V transpose -- 00:34:34.900 --> 00:34:37.750 the Vs will be -- 00:34:37.750 --> 00:34:41.830 there's v1, and what's v2? 00:34:41.830 --> 00:34:45.170 What's v2 in the -- 00:34:45.170 --> 00:34:50.739 how do I make this into an orthonormal basis? 00:34:50.739 --> 00:34:55.480 Well, v2 is, in the null space direction. 00:34:55.480 --> 00:34:59.810 It's perpendicular to that, so point six and minus point 00:34:59.810 --> 00:35:00.790 eight. 00:35:00.790 --> 00:35:04.400 So those are the Vs that go in here. 00:35:04.400 --> 00:35:11.920 Point eight, point six and point six minus point eight. 00:35:11.920 --> 00:35:12.420 Okay. 00:35:14.960 --> 00:35:17.410 And I guess I better finish this guy. 00:35:17.410 --> 00:35:21.580 So this guy, all I want is to complete the orthonormal basis 00:35:21.580 --> 00:35:23.670 -- it'll be coming from there. 00:35:23.670 --> 00:35:28.770 It'll be a two over square root of five and a minus one 00:35:28.770 --> 00:35:31.330 over square root of five. 00:35:31.330 --> 00:35:35.100 Let me take square root of five out of that matrix 00:35:35.100 --> 00:35:38.080 to make it look better. 00:35:38.080 --> 00:35:44.850 So one over square root of five times one two two minus one. 00:35:47.560 --> 00:35:49.920 Okay. 00:35:49.920 --> 00:35:53.160 So there I have -- including the square root of five -- 00:35:53.160 --> 00:35:56.870 that's an orthogonal matrix, that's an orthogonal matrix, 00:35:56.870 --> 00:36:01.190 that's a diagonal matrix and its rank is only one. 00:36:01.190 --> 00:36:03.800 And now if I do that multiplication, 00:36:03.800 --> 00:36:07.550 I pray that it comes out right. 00:36:10.200 --> 00:36:12.022 The square root of five will cancel 00:36:12.022 --> 00:36:13.480 into that square root of one twenty 00:36:13.480 --> 00:36:16.390 five and leave me with the square root of 25, which 00:36:16.390 --> 00:36:20.190 is five, and five will multiply these numbers 00:36:20.190 --> 00:36:24.410 and I'll get whole numbers and out will come A. 00:36:24.410 --> 00:36:25.230 Okay. 00:36:25.230 --> 00:36:31.110 That's like a second example showing how the null space guy 00:36:31.110 --> 00:36:39.680 -- so this -- this vector and this one were multiplied 00:36:39.680 --> 00:36:40.620 by this zero. 00:36:40.620 --> 00:36:46.370 So they were easy to deal with. 00:36:46.370 --> 00:36:50.960 Tthe key ones are the ones in the column space and the row 00:36:50.960 --> 00:36:51.480 space. 00:36:51.480 --> 00:36:57.920 Do you see how I'm getting columns here, diagonal here, 00:36:57.920 --> 00:37:02.330 rows here, coming together to produce A. 00:37:02.330 --> 00:37:06.790 Okay, that's the singular value decomposition. 00:37:06.790 --> 00:37:13.370 So, let me think what I want to add to complete this topic. 00:37:18.130 --> 00:37:22.970 So that's two examples. 00:37:22.970 --> 00:37:25.740 And now let's think what we're really doing. 00:37:25.740 --> 00:37:33.770 We're choosing the right basis for the four subspaces 00:37:33.770 --> 00:37:35.010 of linear algebra. 00:37:35.010 --> 00:37:39.640 Let me write this down. 00:37:39.640 --> 00:37:52.785 So v1 up to vr is an orthonormal basis for the row space. 00:37:58.220 --> 00:38:05.570 u1 up to ur is an orthonormal basis for the column space. 00:38:09.420 --> 00:38:14.930 And then I just finish those out by v(r+1), 00:38:14.930 --> 00:38:20.350 the rest up to vn is an orthonormal basis for the null 00:38:20.350 --> 00:38:20.850 space. 00:38:24.510 --> 00:38:35.220 And finally, u(r+1) up to is an orthonormal basis for the null 00:38:35.220 --> 00:38:36.320 space of A transpose. 00:38:39.670 --> 00:38:45.360 Do you see that we finally got the bases right? 00:38:45.360 --> 00:38:51.170 They're right because they're orthonormal, and also -- 00:38:51.170 --> 00:38:55.100 again, Graham Schmidt would have done this in chapter four. 00:38:55.100 --> 00:39:00.920 Here we needed eigenvalues, because these bases 00:39:00.920 --> 00:39:03.070 make the matrix diagonal. 00:39:03.070 --> 00:39:09.400 A times V I is a multiple of U I. 00:39:09.400 --> 00:39:11.250 So I'll put "and" -- 00:39:14.540 --> 00:39:16.640 the matrix has been made diagonal. 00:39:16.640 --> 00:39:24.730 When we choose these bases, there's no coupling between Vs 00:39:24.730 --> 00:39:26.740 and no coupling between Us. 00:39:26.740 --> 00:39:30.550 Each A -- A times each V is in the direction 00:39:30.550 --> 00:39:31.980 of the corresponding U. 00:39:31.980 --> 00:39:36.700 So it's exactly the right basis for the four 00:39:36.700 --> 00:39:38.130 fundamental subspaces. 00:39:38.130 --> 00:39:41.570 And of course, their dimensions are what we know. 00:39:41.570 --> 00:39:44.850 The dimension of the row space is 00:39:44.850 --> 00:39:49.450 the rank r, and so is the dimension of the column space. 00:39:49.450 --> 00:39:51.190 The dimension of the null space is 00:39:51.190 --> 00:39:54.620 n-r, that's how many vectors we need, 00:39:54.620 --> 00:39:59.790 and m-r basis vectors for the left null space, the null space 00:39:59.790 --> 00:40:01.960 of A transpose. 00:40:01.960 --> 00:40:04.620 Okay. 00:40:04.620 --> 00:40:05.850 I'm going to stop there. 00:40:05.850 --> 00:40:10.720 I could develop further from the SVD, 00:40:10.720 --> 00:40:14.140 but we'll see it again in the very last lectures 00:40:14.140 --> 00:40:14.949 of the course. 00:40:14.949 --> 00:40:15.740 So there's the SVD. 00:40:15.740 --> 00:40:17.290 Thanks.