1 00:00:13,665 --> 00:00:15,540 GILBERT STRANG: Moving now to the second half 2 00:00:15,540 --> 00:00:17,220 of linear algebra. 3 00:00:17,220 --> 00:00:20,160 It's about eigenvalues and eigenvectors. 4 00:00:20,160 --> 00:00:22,320 The first half, I just had a matrix. 5 00:00:22,320 --> 00:00:24,180 I solved equations. 6 00:00:24,180 --> 00:00:27,150 The second half, you'll see the point 7 00:00:27,150 --> 00:00:29,820 of eigenvalues and eigenvectors as a new way 8 00:00:29,820 --> 00:00:34,500 to look deeper into the matrix to see what's important there. 9 00:00:34,500 --> 00:00:36,660 OK, so what are they? 10 00:00:36,660 --> 00:00:40,510 This is a big equation, S time x. 11 00:00:40,510 --> 00:00:42,590 So S is our matrix. 12 00:00:42,590 --> 00:00:45,060 And I've called it S because I'm taking 13 00:00:45,060 --> 00:00:47,400 it to be a symmetric matrix. 14 00:00:47,400 --> 00:00:49,920 What's on one side of the diagonal 15 00:00:49,920 --> 00:00:52,530 is also on the other side of the diagonal. 16 00:00:52,530 --> 00:00:54,800 So those have the beautiful properties. 17 00:00:54,800 --> 00:00:57,270 Those are the kings of linear algebra. 18 00:00:57,270 --> 00:01:02,280 Now, about eigenvectors x and eigenvalues lambda. 19 00:01:02,280 --> 00:01:06,810 So what does that equation, Sx equal lambda x, tell me? 20 00:01:06,810 --> 00:01:10,550 That says that I have a special vector x. 21 00:01:10,550 --> 00:01:14,400 When I multiply it by S, my matrix, 22 00:01:14,400 --> 00:01:17,730 I stay in the same direction as the original x. 23 00:01:17,730 --> 00:01:19,810 It might get multiplied by 2. 24 00:01:19,810 --> 00:01:21,630 Lambda could be 2. 25 00:01:21,630 --> 00:01:23,670 It might get multiplied by 0. 26 00:01:23,670 --> 00:01:25,590 Lambda there could even be 0. 27 00:01:25,590 --> 00:01:28,740 It might get multiplied by minus 2, whatever. 28 00:01:28,740 --> 00:01:30,680 But it's along the same line. 29 00:01:30,680 --> 00:01:35,760 So that's like taking a matrix and discovering inside it 30 00:01:35,760 --> 00:01:39,740 something that stays on a line. 31 00:01:39,740 --> 00:01:43,640 That means that it's really a sort of one dimensional problem 32 00:01:43,640 --> 00:01:47,060 if we're looking along that eigenvector. 33 00:01:47,060 --> 00:01:51,380 And that makes computations infinitely easier. 34 00:01:51,380 --> 00:01:55,070 The hard part of a matrix is all the connections 35 00:01:55,070 --> 00:01:57,830 between different rows and columns. 36 00:01:57,830 --> 00:02:00,050 So eigenvectors are the guys that 37 00:02:00,050 --> 00:02:02,222 stay in that same direction. 38 00:02:04,830 --> 00:02:08,330 And y is another eigenvector. 39 00:02:08,330 --> 00:02:10,169 It has its own eigenvalue. 40 00:02:10,169 --> 00:02:14,200 It got multiplied by alpha where Sx multiplied the x 41 00:02:14,200 --> 00:02:16,540 by some other number lambda. 42 00:02:16,540 --> 00:02:18,400 So there's our couple of eigenvectors. 43 00:02:18,400 --> 00:02:23,600 And the beautiful fact is that because S is symmetric, 44 00:02:23,600 --> 00:02:26,300 those two eigenvectors are perpendicular. 45 00:02:26,300 --> 00:02:29,450 They are orthogonal, as it says up there. 46 00:02:29,450 --> 00:02:32,450 So symmetric matrices are really the best 47 00:02:32,450 --> 00:02:35,330 because their eigenvectors are perpendicular. 48 00:02:35,330 --> 00:02:38,600 And we have a bunch of one dimensional problems. 49 00:02:38,600 --> 00:02:41,780 And here, I've included a proof. 50 00:02:41,780 --> 00:02:47,460 You want a proof that the eigenvectors are perpendicular? 51 00:02:47,460 --> 00:02:48,980 So what does perpendicular mean? 52 00:02:48,980 --> 00:02:55,980 It means that x transpose times y, the dot product is 0. 53 00:02:55,980 --> 00:02:58,940 The angle is 90 degrees. 54 00:02:58,940 --> 00:03:01,640 The cosine is 1. 55 00:03:01,640 --> 00:03:03,050 OK. 56 00:03:03,050 --> 00:03:07,670 How to show the cosine might be there. 57 00:03:07,670 --> 00:03:09,560 How to show that? 58 00:03:09,560 --> 00:03:10,500 Yeah, proof. 59 00:03:10,500 --> 00:03:13,730 This is just you can tune out for two minutes 60 00:03:13,730 --> 00:03:15,410 if you hate proofs. 61 00:03:15,410 --> 00:03:17,990 OK, I start with what I know. 62 00:03:17,990 --> 00:03:20,200 What I know is in that box. 63 00:03:20,200 --> 00:03:21,640 Sx is lambda x. 64 00:03:21,640 --> 00:03:23,260 That's one eigenvector. 65 00:03:23,260 --> 00:03:25,570 That tells me the eigenvector y. 66 00:03:25,570 --> 00:03:27,940 This tells me the eigenvalues are different. 67 00:03:27,940 --> 00:03:30,430 And that tells me the matrix is symmetric. 68 00:03:30,430 --> 00:03:33,640 I'm just going to juggle those four facts. 69 00:03:33,640 --> 00:03:39,040 And I'll end up with x transpose y equals 0. 70 00:03:39,040 --> 00:03:41,350 That's orthogonality. 71 00:03:41,350 --> 00:03:42,400 OK. 72 00:03:42,400 --> 00:03:46,000 So I'll just do it quickly, too quickly. 73 00:03:46,000 --> 00:03:49,900 So I take this first thing, and I transpose 74 00:03:49,900 --> 00:03:52,740 it, turn it into row vectors. 75 00:03:52,740 --> 00:03:57,450 And then when I transpose it, that transpose 76 00:03:57,450 --> 00:03:59,220 means I flip rows and columns. 77 00:03:59,220 --> 00:04:02,670 But for as symmetric matrix, no different. 78 00:04:02,670 --> 00:04:05,390 So S transpose is the same as S. 79 00:04:05,390 --> 00:04:08,840 And then I look at this one, and I multiply that 80 00:04:08,840 --> 00:04:13,130 by x transpose, both sides by x transpose. 81 00:04:13,130 --> 00:04:16,700 And what I end up with is recognizing 82 00:04:16,700 --> 00:04:19,250 that lambda times that dot product 83 00:04:19,250 --> 00:04:22,140 equals alpha times that dot product. 84 00:04:22,140 --> 00:04:24,690 But lambda is different from alpha. 85 00:04:24,690 --> 00:04:26,760 So the only way lambda times that number 86 00:04:26,760 --> 00:04:28,470 could equal alpha times that number 87 00:04:28,470 --> 00:04:31,050 is that number has to be 0. 88 00:04:31,050 --> 00:04:32,460 And that's the answer. 89 00:04:32,460 --> 00:04:34,740 OK, so that's the proof that used 90 00:04:34,740 --> 00:04:37,980 exactly every fact we knew. 91 00:04:37,980 --> 00:04:39,930 End of proof. 92 00:04:39,930 --> 00:04:42,720 Main point to remember, eigenvectors 93 00:04:42,720 --> 00:04:47,130 are perpendicular when the matrix is symmetric. 94 00:04:47,130 --> 00:04:49,710 OK. 95 00:04:49,710 --> 00:04:54,180 In that case, now, you always want to express these facts 96 00:04:54,180 --> 00:04:59,030 as from multiplying matrices. 97 00:04:59,030 --> 00:05:02,180 That says everything in a few symbols 98 00:05:02,180 --> 00:05:06,000 where I had to use all those words on the previous slide. 99 00:05:06,000 --> 00:05:10,950 So that's the result that I'm shooting for, 100 00:05:10,950 --> 00:05:14,450 that a symmetric matrix-- 101 00:05:14,450 --> 00:05:19,150 just focus on that box. 102 00:05:19,150 --> 00:05:24,740 A symmetric matrix can be broken up into its eigenvectors. 103 00:05:24,740 --> 00:05:27,740 Those are in Q. Its eigenvalues. 104 00:05:27,740 --> 00:05:28,910 Those are the lambdas. 105 00:05:28,910 --> 00:05:32,120 Those are the numbers lambda 1 to lambda n 106 00:05:32,120 --> 00:05:34,070 on the diagonal of lambda. 107 00:05:34,070 --> 00:05:36,890 And then the transpose, so the eigenvectors are now 108 00:05:36,890 --> 00:05:39,440 rows in Q transpose. 109 00:05:39,440 --> 00:05:42,160 That's just perfect. 110 00:05:42,160 --> 00:05:43,480 Perfect. 111 00:05:43,480 --> 00:05:47,020 Every symmetric matrix is an orthogonal matrix 112 00:05:47,020 --> 00:05:51,490 times a diagonal matrix times the transpose 113 00:05:51,490 --> 00:05:53,680 of the orthogonal matrix. 114 00:05:53,680 --> 00:05:55,810 Yeah, that's called the spectral theorem. 115 00:05:55,810 --> 00:06:00,670 And you could say it's up there with the most important facts 116 00:06:00,670 --> 00:06:04,410 in linear algebra and in wider mathematics. 117 00:06:04,410 --> 00:06:12,580 Yeah, so that's the fact that controls what we do here. 118 00:06:12,580 --> 00:06:18,160 Oh, now I have to say what's the situation if the matrix is not 119 00:06:18,160 --> 00:06:19,860 symmetric. 120 00:06:19,860 --> 00:06:24,170 Now I am not going to get perpendicular eigenvectors. 121 00:06:24,170 --> 00:06:27,470 That was a symmetric thing mostly. 122 00:06:27,470 --> 00:06:30,260 But I'll get eigenvectors. 123 00:06:30,260 --> 00:06:35,140 So I'll get Ax equal lambda x. 124 00:06:35,140 --> 00:06:37,060 The first one won't be perpendicular 125 00:06:37,060 --> 00:06:37,870 to the second one. 126 00:06:37,870 --> 00:06:40,520 The matrix A, it has to be square, 127 00:06:40,520 --> 00:06:41,710 or this doesn't make sense. 128 00:06:41,710 --> 00:06:45,070 So eigenvalues and eigenvectors are the way 129 00:06:45,070 --> 00:06:50,590 to break up a square matrix and find this diagonal matrix 130 00:06:50,590 --> 00:06:54,640 lambda with the eigenvalues, lambda 1, lambda 2, to lambda 131 00:06:54,640 --> 00:06:55,270 n. 132 00:06:55,270 --> 00:06:58,900 That's the purpose. 133 00:06:58,900 --> 00:07:01,510 And eigenvectors are perpendicular when 134 00:07:01,510 --> 00:07:03,250 it's a symmetric matrix. 135 00:07:03,250 --> 00:07:08,710 Otherwise, I just have x and its inverse matrix but no symmetry. 136 00:07:08,710 --> 00:07:09,490 OK. 137 00:07:09,490 --> 00:07:14,470 So that's the quick expression, another factorization 138 00:07:14,470 --> 00:07:17,530 of eigenvalues in lambda. 139 00:07:17,530 --> 00:07:19,670 Diagonal, just numbers. 140 00:07:19,670 --> 00:07:23,370 And eigenvectors in the columns of x. 141 00:07:23,370 --> 00:07:27,610 And now I'm not going to-- oh, I was 142 00:07:27,610 --> 00:07:29,530 going to say I'm not going to solve all 143 00:07:29,530 --> 00:07:31,030 the problems of applied math. 144 00:07:31,030 --> 00:07:33,850 But that's what these are for. 145 00:07:33,850 --> 00:07:38,400 Let's just see what's special here about these eigenvectors. 146 00:07:38,400 --> 00:07:46,800 Suppose I multiply again by A. I Start with Ax equal lambda x. 147 00:07:46,800 --> 00:07:49,650 Now I'm going to multiply both sides by A. 148 00:07:49,650 --> 00:07:53,550 That'll tell me something about eigenvalues of A squared. 149 00:07:53,550 --> 00:07:55,970 Because when I multiply by A-- 150 00:07:55,970 --> 00:07:58,230 so let me start with A squared now 151 00:07:58,230 --> 00:08:02,810 times x, which means A times Ax. 152 00:08:02,810 --> 00:08:04,400 A times Ax. 153 00:08:04,400 --> 00:08:06,440 But Ax is lambda x. 154 00:08:06,440 --> 00:08:09,320 So I have A times lambda x. 155 00:08:09,320 --> 00:08:12,180 And I pull out that number lambda. 156 00:08:12,180 --> 00:08:15,040 And I still have a 1Ax. 157 00:08:15,040 --> 00:08:17,450 And that's also still lambda x. 158 00:08:17,450 --> 00:08:20,180 You see I'm just talking around in a little circle 159 00:08:20,180 --> 00:08:24,080 here, just using Ax equal lambda x a couple of times. 160 00:08:24,080 --> 00:08:25,860 And the result is-- 161 00:08:25,860 --> 00:08:28,400 do you see what that means, that result? 162 00:08:28,400 --> 00:08:33,080 That means that the eigenvalue for A squared, same eigenvector 163 00:08:33,080 --> 00:08:33,830 x. 164 00:08:33,830 --> 00:08:37,230 The eigenvalue is lambda squared. 165 00:08:37,230 --> 00:08:39,500 And if I add A cubed, the eigenvalue 166 00:08:39,500 --> 00:08:41,299 would come out lambda cubed. 167 00:08:41,299 --> 00:08:44,990 And if I have a to the-- yeah, yeah. 168 00:08:44,990 --> 00:08:49,640 So if I had A to the n times, n multiplies-- so when 169 00:08:49,640 --> 00:08:53,090 would you have A to a high power? 170 00:08:53,090 --> 00:08:56,480 That's a interesting matrix. 171 00:08:56,480 --> 00:08:58,910 Take a matrix and square it, cube it, 172 00:08:58,910 --> 00:09:02,450 take high powers of it. 173 00:09:02,450 --> 00:09:04,520 The eigenvectors don't change. 174 00:09:04,520 --> 00:09:05,690 That's the great thing. 175 00:09:05,690 --> 00:09:07,850 That's the whole point of eigenvectors. 176 00:09:07,850 --> 00:09:09,120 They don't change. 177 00:09:09,120 --> 00:09:12,540 And the eigenvalues just get taken to the high power. 178 00:09:12,540 --> 00:09:16,010 So for example, we could ask the question, when, 179 00:09:16,010 --> 00:09:19,160 if I multiply a matrix by itself over and over and over again, 180 00:09:19,160 --> 00:09:20,840 when do I approach 0? 181 00:09:20,840 --> 00:09:24,920 Well, if these numbers are below 1. 182 00:09:24,920 --> 00:09:27,260 So eigenvectors, eigenvalues gives you something 183 00:09:27,260 --> 00:09:33,330 that you just could not see by those column operations or L 184 00:09:33,330 --> 00:09:36,980 times U. This is looking deeper. 185 00:09:36,980 --> 00:09:38,120 OK. 186 00:09:38,120 --> 00:09:45,040 And OK, and then you'll see we have almost already seen 187 00:09:45,040 --> 00:09:49,600 with least squares, this combination A transpose A. So 188 00:09:49,600 --> 00:09:53,750 remember A is a rectangular matrix, m by n. 189 00:09:53,750 --> 00:09:56,220 I multiply it by its transpose. 190 00:09:56,220 --> 00:10:02,750 When I transpose it, I have n by m. 191 00:10:02,750 --> 00:10:05,450 And when I multiply them together, I get n by n. 192 00:10:05,450 --> 00:10:10,880 So A transpose A is, for theory, is a great matrix, 193 00:10:10,880 --> 00:10:14,030 A transpose times A. It's symmetric. 194 00:10:14,030 --> 00:10:16,430 Yeah, let's just see what we have about A. 195 00:10:16,430 --> 00:10:19,460 It's square for sure. 196 00:10:19,460 --> 00:10:20,000 Oh, yeah. 197 00:10:20,000 --> 00:10:22,125 This tells me that it's symmetric. 198 00:10:22,125 --> 00:10:23,000 And you remember why. 199 00:10:23,000 --> 00:10:26,020 I'm always looking for symmetric matrices 200 00:10:26,020 --> 00:10:29,530 because they have those orthogonal eigenvectors. 201 00:10:29,530 --> 00:10:32,930 They're the beautiful ones for eigenvectors. 202 00:10:32,930 --> 00:10:36,070 And A transpose A, automatically symmetric. 203 00:10:36,070 --> 00:10:40,660 You just you're multiplying something 204 00:10:40,660 --> 00:10:45,940 by its adjoint, its transpose, and the result 205 00:10:45,940 --> 00:10:51,010 is that this matrix is symmetric. 206 00:10:51,010 --> 00:10:55,920 And maybe there's even more about A transpose A. Yes. 207 00:10:55,920 --> 00:10:56,580 What is that? 208 00:11:00,610 --> 00:11:03,260 Here is a final-- 209 00:11:03,260 --> 00:11:07,020 I always say certain matrices are important, 210 00:11:07,020 --> 00:11:10,330 but these are the winners. 211 00:11:10,330 --> 00:11:12,510 They are symmetric matrices. 212 00:11:12,510 --> 00:11:16,260 If I want beautiful matrices, make them symmetric 213 00:11:16,260 --> 00:11:18,950 and make the eigenvalues positive. 214 00:11:21,540 --> 00:11:26,160 Or non-negative allows 0. 215 00:11:26,160 --> 00:11:28,890 So I can either say positive definite 216 00:11:28,890 --> 00:11:31,320 when the eigenvalues are positive, 217 00:11:31,320 --> 00:11:34,880 or I can say non-negative, which allows 0. 218 00:11:34,880 --> 00:11:38,700 And so I have greater than or equal to 0. 219 00:11:38,700 --> 00:11:41,820 I just want to say that bringing all 220 00:11:41,820 --> 00:11:44,520 the pieces of linear algebra come together 221 00:11:44,520 --> 00:11:46,320 in these matrices. 222 00:11:46,320 --> 00:11:49,770 And we're seeing the eigenvalue part of it. 223 00:11:49,770 --> 00:11:53,020 And here, I've mentioned something called the energy. 224 00:11:53,020 --> 00:11:55,380 So that's a physical quantity that 225 00:11:55,380 --> 00:11:58,230 also is greater or equal to 0. 226 00:11:58,230 --> 00:12:02,970 So that's A transpose A is the matrix 227 00:12:02,970 --> 00:12:10,290 that I'm going to use in the final part of this video 228 00:12:10,290 --> 00:12:15,060 to achieve the greatest factorization. 229 00:12:15,060 --> 00:12:18,780 Q lambda, Q transpose was fantastic. 230 00:12:18,780 --> 00:12:22,830 But for a non-square matrix, it's not. 231 00:12:22,830 --> 00:12:25,260 For a non-square matrix, they don't even 232 00:12:25,260 --> 00:12:27,840 have eigenvalues and eigenvectors. 233 00:12:27,840 --> 00:12:31,650 But data comes in non-square matrices. 234 00:12:31,650 --> 00:12:34,230 Data is about like we have a bunch of diseases 235 00:12:34,230 --> 00:12:37,940 and a bunch of patients or a bunch of medicines. 236 00:12:37,940 --> 00:12:39,470 And the number of medicines is not 237 00:12:39,470 --> 00:12:42,090 equal the number of patients or diseases. 238 00:12:42,090 --> 00:12:43,740 Those are different numbers. 239 00:12:43,740 --> 00:12:48,560 So the matrices that we see in data are rectangular. 240 00:12:48,560 --> 00:12:52,800 And eigenvalues don't make sense for those. 241 00:12:52,800 --> 00:12:56,650 And singular values take the place of eigenvalues. 242 00:12:56,650 --> 00:12:59,280 So singular values, and my hope is 243 00:12:59,280 --> 00:13:04,550 that linear algebra courses, 18.06 for sure, 244 00:13:04,550 --> 00:13:08,690 will always reach, after you explain 245 00:13:08,690 --> 00:13:12,170 eigenvalues that everybody agrees is important, 246 00:13:12,170 --> 00:13:14,600 get singular values into the course 247 00:13:14,600 --> 00:13:18,740 because they really have come on as the big things 248 00:13:18,740 --> 00:13:20,600 to do in data. 249 00:13:20,600 --> 00:13:27,370 So that would be the last part of this summary video 250 00:13:27,370 --> 00:13:30,780 for 2020 vision of linear algebra 251 00:13:30,780 --> 00:13:33,520 is to get singular values in there. 252 00:13:33,520 --> 00:13:36,120 OK, that's coming next.