WEBVTT 00:00:01.161 --> 00:00:03.920 NARRATOR: The following content is provided under a Creative 00:00:03.920 --> 00:00:05.310 Commons license. 00:00:05.310 --> 00:00:07.520 Your support will help MIT OpenCourseWare 00:00:07.520 --> 00:00:11.610 continue to offer high-quality educational resources for free. 00:00:11.610 --> 00:00:14.180 To make a donation or to view additional materials 00:00:14.180 --> 00:00:18.140 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:18.140 --> 00:00:19.026 at ocw.mit.edu. 00:00:23.528 --> 00:00:24.900 GILBERT STRANG: OK. 00:00:24.900 --> 00:00:31.740 So, I'd like to pick up again on this neat family of matrices, 00:00:31.740 --> 00:00:32.960 circulant matrices. 00:00:32.960 --> 00:00:37.020 But first, let me say here and then put it 00:00:37.020 --> 00:00:40.140 on the web, my thought about the projects. 00:00:40.140 --> 00:00:45.690 So, I think the last deadline I can give is the final class. 00:00:45.690 --> 00:00:49.830 So, I think that's not next week but Wednesday 00:00:49.830 --> 00:00:53.550 of the following week, I think, is our last class meeting. 00:00:53.550 --> 00:00:57.730 So, be great to get them then or earlier. 00:00:57.730 --> 00:01:00.570 And if anybody or everybody would 00:01:00.570 --> 00:01:07.420 like to tell the class a little bit about their project, 00:01:07.420 --> 00:01:12.480 you know it's a friendly audience 00:01:12.480 --> 00:01:17.310 and I'd be happy to make space and time for that. 00:01:17.310 --> 00:01:22.320 So, send me an email and give me the project earlier 00:01:22.320 --> 00:01:25.620 if you would like to just say a few words in class. 00:01:25.620 --> 00:01:29.370 Or even if you are willing to say a few words in class, 00:01:29.370 --> 00:01:30.320 I'll say. 00:01:30.320 --> 00:01:30.820 Yeah. 00:01:30.820 --> 00:01:34.650 Because I realize-- yeah, OK. 00:01:34.650 --> 00:01:36.970 So, other questions about-- 00:01:36.970 --> 00:01:40.660 so, we're finished with all psets and so on. 00:01:40.660 --> 00:01:43.040 So, it's really just a project, and yeah. 00:01:43.040 --> 00:01:44.498 STUDENT: How is the project graded? 00:01:44.498 --> 00:01:45.785 Like, on what basis? 00:01:45.785 --> 00:01:47.160 GILBERT STRANG: How is it graded? 00:01:47.160 --> 00:01:48.840 Good question. 00:01:48.840 --> 00:01:51.600 But it's going to be me, I guess. 00:01:51.600 --> 00:01:58.270 So I'll read all the projects and come up with a grade 00:01:58.270 --> 00:02:01.320 somehow, you know. 00:02:01.320 --> 00:02:05.970 I hope you guys have understood that my feeling is 00:02:05.970 --> 00:02:07.950 that the grades in this course are 00:02:07.950 --> 00:02:11.890 going to be on the high side because they should be. 00:02:11.890 --> 00:02:12.390 Yeah. 00:02:12.390 --> 00:02:14.760 I think it's that kind of a course 00:02:14.760 --> 00:02:19.860 and I've asked you to do a fair amount, and-- 00:02:19.860 --> 00:02:22.875 anyway, that's my starting basis. 00:02:25.590 --> 00:02:28.560 And there's a lot of topics like circulant matrices 00:02:28.560 --> 00:02:31.920 that I'm not going to be able to give you a pset about. 00:02:31.920 --> 00:02:36.720 But of course, these are closely connected 00:02:36.720 --> 00:02:41.070 to the discrete Fourier transform. 00:02:41.070 --> 00:02:49.350 So, let me just write the name of the great man Fourier. 00:02:49.350 --> 00:02:51.806 So, the discrete Fourier transform is, as you know, 00:02:51.806 --> 00:02:58.740 a very, very important algorithm in engineering 00:02:58.740 --> 00:02:59.970 and in mathematics. 00:02:59.970 --> 00:03:01.080 Everywhere. 00:03:01.080 --> 00:03:08.955 Fourier is just a key idea and so 00:03:08.955 --> 00:03:10.830 I think it's just good to know about, though. 00:03:10.830 --> 00:03:13.860 So, circulant matrices are connected 00:03:13.860 --> 00:03:20.160 with finite size matrices. 00:03:20.160 --> 00:03:22.230 Matrices of size n. 00:03:22.230 --> 00:03:28.460 So our circulant matrices will be N by N. 00:03:28.460 --> 00:03:30.320 And you remember this special form. 00:03:37.340 --> 00:03:40.490 So, this is a key point about these matrices, 00:03:40.490 --> 00:03:48.050 C. That they're defined by not n squared entries, only n. 00:03:48.050 --> 00:03:51.970 If you tell me just the first row of the matrix, 00:03:51.970 --> 00:03:57.160 and that's all you would tell Matlab, say, c0, c1, c2 00:03:57.160 --> 00:04:01.208 to c N minus 1. 00:04:01.208 --> 00:04:03.200 Then for a circulant, that's all I 00:04:03.200 --> 00:04:07.120 need to know because these diagonals are constant. 00:04:07.120 --> 00:04:09.590 This diagonal is constant-- c1-- 00:04:09.590 --> 00:04:12.020 and then gets completed here. 00:04:12.020 --> 00:04:19.459 c2 diagonal come to c2 and then gets completed cyclically here. 00:04:19.459 --> 00:04:24.050 So, n numbers and not n squared. 00:04:24.050 --> 00:04:29.780 The reason I mention that, or a reason is, 00:04:29.780 --> 00:04:33.460 that's a big selling point when you go to applications, 00:04:33.460 --> 00:04:39.010 say machine learning, for images. 00:04:39.010 --> 00:04:45.130 So, you remember the big picture of machine learning, 00:04:45.130 --> 00:04:48.070 deep learning, was that you had samples. 00:04:51.670 --> 00:04:57.220 A lot of samples, let's say N samples, maybe. 00:04:57.220 --> 00:05:03.710 And then each sample in this image part will be an image. 00:05:03.710 --> 00:05:09.590 So, the thing is that an image is described by its pixels 00:05:09.590 --> 00:05:13.550 and if I have 1,000 by 1,000 pixel-- 00:05:13.550 --> 00:05:18.460 so, that's a million pixels. 00:05:18.460 --> 00:05:20.300 The feature vector, the vector that's 00:05:20.300 --> 00:05:24.290 associated with 1 sample, is enormous. 00:05:24.290 --> 00:05:26.500 Is enormous. 00:05:26.500 --> 00:05:32.180 So I have N samples but maybe-- 00:05:32.180 --> 00:05:35.150 well, if they were in color that million 00:05:35.150 --> 00:05:36.740 suddenly becomes 3 million. 00:05:36.740 --> 00:05:40.680 So say 3 million features. 00:05:46.020 --> 00:05:50.030 So, our vectors are a vector of-- 00:05:50.030 --> 00:05:55.650 the whole computation of deep learning works with our vectors 00:05:55.650 --> 00:05:58.740 with 3 million components. 00:05:58.740 --> 00:06:01.350 And that means that in the ordinary way, if we didn't 00:06:01.350 --> 00:06:04.710 do anything special we would be multiplying those 00:06:04.710 --> 00:06:10.380 by matrices of size like 3 million times 3 million. 00:06:10.380 --> 00:06:12.910 We would be computing that many weights. 00:06:12.910 --> 00:06:14.530 That's like, impossible. 00:06:17.410 --> 00:06:21.430 And we would be computing that for each layer 00:06:21.430 --> 00:06:24.670 in the deep network so it would go up-- 00:06:29.900 --> 00:06:32.230 so 3 million by 3 million is just-- 00:06:32.230 --> 00:06:34.090 we can't compute. 00:06:34.090 --> 00:06:39.040 We can't use gradient descent to optimize that many weights. 00:06:39.040 --> 00:06:45.190 So, the point is that the matrices in deep learning 00:06:45.190 --> 00:06:47.140 are special. 00:06:47.140 --> 00:06:48.580 And they don't depend-- 00:06:48.580 --> 00:06:51.730 they're like circulant matrices. 00:06:51.730 --> 00:06:54.160 They might not loop around. 00:06:54.160 --> 00:06:58.060 So, circulant matrices have this cyclic feature 00:06:58.060 --> 00:07:02.560 that makes the theory extremely nice. 00:07:02.560 --> 00:07:10.165 But of course, in general we have matrices, let's say t0-- 00:07:12.670 --> 00:07:18.310 constant diagonals and maybe a bunch of diagonals. 00:07:18.310 --> 00:07:22.330 And here not necessarily symmetric, or they 00:07:22.330 --> 00:07:23.710 might be symmetric. 00:07:23.710 --> 00:07:25.450 But they're not cyclic. 00:07:25.450 --> 00:07:29.530 So, what are these matrices called? 00:07:29.530 --> 00:07:32.290 Well, they have a bunch of names because they're so important. 00:07:32.290 --> 00:07:37.450 They're linear shift invariant. 00:07:37.450 --> 00:07:40.630 Or linear time invariant, whatever is 00:07:40.630 --> 00:07:44.620 the right word in your context. 00:07:44.620 --> 00:07:48.430 So, they're convolutions. 00:07:48.430 --> 00:07:50.365 You could call it a convolution matrix. 00:07:54.810 --> 00:07:57.880 When you multiply by one of these matrices, 00:07:57.880 --> 00:08:05.210 I guess I'm going to call it t, you're doing our convolution. 00:08:05.210 --> 00:08:09.090 And I'll better write down the formula for convolution. 00:08:09.090 --> 00:08:11.190 You're not doing a cyclic convolution 00:08:11.190 --> 00:08:14.670 unless the matrix cycles round. 00:08:14.670 --> 00:08:18.120 When you multiply by C, this would give you 00:08:18.120 --> 00:08:20.820 cyclic convolution. 00:08:23.920 --> 00:08:27.300 Say if I multiply C by some vector v, 00:08:27.300 --> 00:08:31.290 the result is the cyclic convolution 00:08:31.290 --> 00:08:35.909 of the c vector with the v vector. 00:08:35.909 --> 00:08:39.270 So, big C is a matrix but it's completely 00:08:39.270 --> 00:08:42.659 defined by its first row or first column. 00:08:42.659 --> 00:08:47.130 So I just have a vector operation in there 00:08:47.130 --> 00:08:48.810 and it's a cyclic one. 00:08:48.810 --> 00:08:53.160 And over here, t times a vector v 00:08:53.160 --> 00:08:59.960 will be the convolution of a t vector with v, but not cyclic. 00:09:05.280 --> 00:09:07.830 And probably these are the ones that would actually 00:09:07.830 --> 00:09:10.980 come into machine learning. 00:09:10.980 --> 00:09:16.620 So, linear shift invariant, linear time invariant. 00:09:16.620 --> 00:09:17.970 I would call it-- 00:09:17.970 --> 00:09:22.470 so, math people would call it a Toeplitz matrix. 00:09:25.650 --> 00:09:29.840 That's why I used the letter t. 00:09:29.840 --> 00:09:37.180 In engineering it would be a filter or a convolution 00:09:37.180 --> 00:09:40.240 or a constant diagonal matrix. 00:09:44.860 --> 00:09:47.420 These come up in all sorts of places 00:09:47.420 --> 00:09:51.230 and they come up in machine learning 00:09:51.230 --> 00:09:55.080 and with image processing. 00:09:55.080 --> 00:09:57.320 But basically, because what you're 00:09:57.320 --> 00:10:00.380 doing at one point in an image is pretty much 00:10:00.380 --> 00:10:02.450 what you're going to do at the other points, 00:10:02.450 --> 00:10:05.240 you're not going to figure out special weights 00:10:05.240 --> 00:10:08.053 for each little pixel in the image. 00:10:08.053 --> 00:10:09.470 You're going to take-- if you have 00:10:09.470 --> 00:10:15.350 an image, say you have an image with zillions of pixels. 00:10:15.350 --> 00:10:19.340 Well, you might want to cut down. 00:10:19.340 --> 00:10:21.830 I mean, it would be very sensible to do 00:10:21.830 --> 00:10:30.160 some max pooling, some pooling operation to make it smaller. 00:10:37.220 --> 00:10:42.260 So, that's really like, OK, we don't want this large a system. 00:10:42.260 --> 00:10:43.940 Let's just reduce it. 00:10:43.940 --> 00:10:46.820 So, max pooling. 00:10:46.820 --> 00:10:49.100 That operation would be-- 00:10:49.100 --> 00:10:55.640 say, take them 3 at a time, some 9 pixels. 00:10:55.640 --> 00:10:59.130 And replace that 9 pixels by 1 pixel. 00:10:59.130 --> 00:11:02.000 So the max of those 9 numbers. 00:11:02.000 --> 00:11:05.720 That would be a very simple operation that just reduces 00:11:05.720 --> 00:11:08.330 the dimension, make it smaller. 00:11:08.330 --> 00:11:09.680 Reduce the dimension. 00:11:16.770 --> 00:11:22.020 OK, so that's a cheap way to make an image 4 times 00:11:22.020 --> 00:11:25.170 or 9 times or 64 times smaller. 00:11:25.170 --> 00:11:29.120 But the convolution part now-- 00:11:29.120 --> 00:11:31.500 so, that's not involving convolution. 00:11:31.500 --> 00:11:35.100 That's a different operation here. 00:11:35.100 --> 00:11:40.300 Not even linear if I take the max in each box. 00:11:40.300 --> 00:11:45.060 That's not a linear operation but it's a fast one. 00:11:45.060 --> 00:11:51.140 OK, so where do circulants or convolution or Toeplitz 00:11:51.140 --> 00:11:54.500 matrices or filters come into it? 00:11:54.500 --> 00:11:56.510 So, I'll forget about the max pooling. 00:11:56.510 --> 00:11:58.910 Suppose that's happened and I still 00:11:58.910 --> 00:12:07.880 have a very big system with n squared pixels, 00:12:07.880 --> 00:12:12.920 n squared features for each sample. 00:12:12.920 --> 00:12:17.700 So, I want to operate on that by matrices, as usual. 00:12:17.700 --> 00:12:24.200 I want to choose the weights to bring out the important points. 00:12:24.200 --> 00:12:26.720 So, the whole idea is-- 00:12:26.720 --> 00:12:32.150 on a image like that I'll use a convolution. 00:12:35.420 --> 00:12:40.010 The same operation is happening at each point. 00:12:40.010 --> 00:12:41.510 So, forget the max part. 00:12:41.510 --> 00:12:45.430 Let me erase, if I can find an eraser here. 00:12:45.430 --> 00:12:49.200 OK, so I'm not going to-- 00:12:49.200 --> 00:12:49.970 we've done this. 00:12:49.970 --> 00:12:51.890 So, that's done. 00:12:51.890 --> 00:12:56.090 Now, I want to multiply it by weights. 00:12:56.090 --> 00:12:59.710 So, that's already done. 00:12:59.710 --> 00:13:02.410 OK. 00:13:02.410 --> 00:13:03.940 So, what am I looking to do? 00:13:07.000 --> 00:13:11.650 What kind of a job would a filter do? 00:13:11.650 --> 00:13:17.290 A low-pass filter would kill, or nearly kill, 00:13:17.290 --> 00:13:21.160 the high frequencies, the noise. 00:13:21.160 --> 00:13:28.060 So, if I wanted to get a simpler image there, 00:13:28.060 --> 00:13:32.010 I would use a low-pass filter, which might just-- 00:13:32.010 --> 00:13:39.550 it might be this filter here. 00:13:39.550 --> 00:13:42.250 Let me just put in some numbers that would-- 00:13:45.760 --> 00:13:46.780 say 1/2 and 1/2. 00:13:50.470 --> 00:13:55.360 So, I'm averaging each pixel with its neighbor 00:13:55.360 --> 00:13:59.770 just to take out some of the high frequencies. 00:13:59.770 --> 00:14:02.350 The low frequencies are constant. 00:14:02.350 --> 00:14:06.460 An all-black image would come out not changed 00:14:06.460 --> 00:14:14.560 but a very highly speckled image would get largely removed 00:14:14.560 --> 00:14:15.700 by that averaging. 00:14:15.700 --> 00:14:19.330 So, it's the same idea that comes up 00:14:19.330 --> 00:14:24.400 in all of signal processing, filtering. 00:14:24.400 --> 00:14:34.060 So, just to complete this thought of, 00:14:34.060 --> 00:14:41.250 why do neural nets-- so, I'm answering this question. 00:14:41.250 --> 00:14:43.710 How do they come in machine learning? 00:14:43.710 --> 00:14:48.810 So, they come when the samples are images 00:14:48.810 --> 00:14:55.920 and then it's natural to use a constant diagonal matrix, 00:14:55.920 --> 00:15:00.930 a shift invariant matrix and not an arbitrary matrix. 00:15:00.930 --> 00:15:08.300 So, we only have to compute n weights and not n squared. 00:15:08.300 --> 00:15:10.470 Yeah, so that's the point. 00:15:10.470 --> 00:15:13.710 So, that's one reason for talking 00:15:13.710 --> 00:15:22.620 about convolution and circulant matrices in this course. 00:15:22.620 --> 00:15:26.940 I guess I feel another reason is that everything 00:15:26.940 --> 00:15:33.690 to do with the DFT, with Fourier and Fourier transforms 00:15:33.690 --> 00:15:39.600 and Fourier matrices, that's just stuff you gotta know. 00:15:39.600 --> 00:15:46.590 Every time you're dealing with vectors where shifting 00:15:46.590 --> 00:15:49.230 the vectors comes into it, that's-- 00:15:49.230 --> 00:15:51.510 Fourier is going to come in. 00:15:51.510 --> 00:15:54.120 So, it's just we should see Fourier. 00:15:54.120 --> 00:15:54.900 OK. 00:15:54.900 --> 00:16:04.260 So now I'll go back to this specially nice case 00:16:04.260 --> 00:16:11.010 where the matrix loops around. 00:16:11.010 --> 00:16:14.100 Where I have this cyclic convolution. 00:16:14.100 --> 00:16:22.950 So, this would be cyclic because of the looping around stuff. 00:16:26.340 --> 00:16:29.910 So, what was the point of last time? 00:16:29.910 --> 00:16:32.145 I started with this permutation matrix. 00:16:35.430 --> 00:16:43.340 And the permutation matrix has c0 equals 0, c1 equal 1, 00:16:43.340 --> 00:16:46.820 and the rest of the c's are 0. 00:16:46.820 --> 00:16:51.350 So, it's just the effect of multiplying by this-- 00:16:53.920 --> 00:16:56.210 get a box around it here-- 00:16:56.210 --> 00:16:59.390 the effect of multiplying by this permutation matrix 00:16:59.390 --> 00:17:03.380 is to shift everything and then bring the last one up 00:17:03.380 --> 00:17:05.250 to the top. 00:17:05.250 --> 00:17:07.010 So, it's a cyclic shift. 00:17:11.910 --> 00:17:16.250 And I guess at the very end of last time 00:17:16.250 --> 00:17:18.950 I was asking about its eigenvalues 00:17:18.950 --> 00:17:21.599 and its eigenvectors, so can we come to that question? 00:17:21.599 --> 00:17:25.640 So, that's the starting question for everything here. 00:17:25.640 --> 00:17:30.500 I guess we've understood that to get deeper 00:17:30.500 --> 00:17:35.180 into a matrix, its eigenvalues, eigenvectors, 00:17:35.180 --> 00:17:40.880 or singular value, singular vectors, are the way to go. 00:17:40.880 --> 00:17:44.645 Actually, what would be the singular values of that matrix? 00:17:50.080 --> 00:17:53.070 Let's just think about singular values 00:17:53.070 --> 00:17:57.650 and then we'll see why it's eigenvalues we want. 00:17:57.650 --> 00:18:02.100 What are the singular values of a permutation matrix? 00:18:02.100 --> 00:18:04.960 They're all 1. 00:18:04.960 --> 00:18:06.190 All 1. 00:18:06.190 --> 00:18:10.200 That matrix is a orthogonal matrix, 00:18:10.200 --> 00:18:18.270 so the SVD of the matrix just has the permutation 00:18:18.270 --> 00:18:21.660 and then the identity is there for the sigma. 00:18:21.660 --> 00:18:25.860 So, sigma is I for this for this matrix. 00:18:31.320 --> 00:18:33.300 So, the singular values don't-- 00:18:36.140 --> 00:18:39.270 that's because P transpose P is the identity matrix. 00:18:41.980 --> 00:18:44.280 Any time I have-- 00:18:44.280 --> 00:18:48.000 that's an orthogonal matrix, and anytime P transpose P 00:18:48.000 --> 00:18:50.850 is the identity, the singular values 00:18:50.850 --> 00:18:53.020 will be the eigenvalues of the identity. 00:18:53.020 --> 00:18:56.010 And they're all just 1's. 00:18:56.010 --> 00:18:59.400 The eigenvalues of P, that's what we want to find, so let's 00:18:59.400 --> 00:19:00.380 do that. 00:19:00.380 --> 00:19:06.030 OK, eigenvalues of P. So, one way 00:19:06.030 --> 00:19:15.200 is to take P minus lambda I. That's just the way we teach 00:19:15.200 --> 00:19:18.860 in 18.06 and never use again. 00:19:18.860 --> 00:19:23.450 So, it puts minus lambda on the diagonal, 00:19:23.450 --> 00:19:26.480 and of course P is sitting up here. 00:19:26.480 --> 00:19:31.100 And then the rest is 0. 00:19:31.100 --> 00:19:35.930 OK, so now following the 18.06 rule, 00:19:35.930 --> 00:19:39.350 I should take that determinant, right? 00:19:39.350 --> 00:19:42.870 And set it to 0. 00:19:42.870 --> 00:19:45.770 This is one of the very few occasions we can actually 00:19:45.770 --> 00:19:49.220 do it, so allow me to do it. 00:19:49.220 --> 00:19:51.440 So, what is the determinant of this? 00:19:51.440 --> 00:19:58.800 Well, there's that lambda to the fourth, 00:19:58.800 --> 00:20:04.485 and I guess I think it's lambda to the fourth minus 1. 00:20:04.485 --> 00:20:08.670 I think that's the right determinant. 00:20:08.670 --> 00:20:15.590 That certainly has property-- so, I would set that to 0, 00:20:15.590 --> 00:20:21.360 then I would find that the eigenvalues for that 00:20:21.360 --> 00:20:27.780 will be 1 and minus 1, and I and minus I. 00:20:27.780 --> 00:20:37.610 And they're the fourth roots of 1. 00:20:37.610 --> 00:20:39.230 Lambda to the fourth equal 1. 00:20:42.410 --> 00:20:44.390 That's our eigenvalue equation. 00:20:44.390 --> 00:20:48.840 Lambda to the fourth equal 1 or lambda to the n-th equal 1. 00:20:48.840 --> 00:20:54.560 So, what would be the eigenvalues for the P 8 by 8? 00:21:00.420 --> 00:21:03.320 This is the complex plane, of course. 00:21:03.320 --> 00:21:08.120 Real and imaginary. 00:21:08.120 --> 00:21:12.520 So, that's got 8 eigenvalues. 00:21:12.520 --> 00:21:17.420 P to the eighth power would be the identity. 00:21:17.420 --> 00:21:19.910 And that means that lambda to the eighth power 00:21:19.910 --> 00:21:23.000 is 1 for the eigenvalues. 00:21:23.000 --> 00:21:25.970 And what are the 8 solutions? 00:21:25.970 --> 00:21:29.480 Every polynomial equation of degree 8 00:21:29.480 --> 00:21:32.160 has got to have 8 solutions. 00:21:32.160 --> 00:21:37.510 That's Gauss's fundamental theorem of algebra. 00:21:37.510 --> 00:21:40.158 8 solutions, so what are they? 00:21:40.158 --> 00:21:45.510 What are the 8 numbers whose eighth power gives 1? 00:21:48.630 --> 00:21:51.430 You all probably know them. 00:21:51.430 --> 00:21:54.460 So, they're 1, of course the eighth power of 1, 00:21:54.460 --> 00:21:58.420 the eighth power of minus 1, the eighth power of minus I, 00:21:58.420 --> 00:22:00.340 and the other guys are just here. 00:22:03.820 --> 00:22:07.780 The roots of 1 are equally spaced around the circle. 00:22:07.780 --> 00:22:09.060 So, Fourier has come in. 00:22:09.060 --> 00:22:13.710 You know, Fourier wakes up when he sees that picture. 00:22:13.710 --> 00:22:19.030 Fourier is going to be here and it'll be in the eigenvectors. 00:22:19.030 --> 00:22:23.030 So, you're OK with the eigenvalues? 00:22:23.030 --> 00:22:25.080 The eigenvalues of P will be-- 00:22:27.660 --> 00:22:31.470 we better give a name to this number. 00:22:31.470 --> 00:22:32.090 Let's see. 00:22:32.090 --> 00:22:36.240 I'm going to call that number w and it will be 00:22:36.240 --> 00:22:41.070 e to the 2 pi i over 8, right? 00:22:41.070 --> 00:22:49.590 Because the whole angle is 2 pi divided in 8 pieces. 00:22:49.590 --> 00:22:52.650 So that's 2 pi i over 8. 00:22:52.650 --> 00:22:57.840 2 pi i over N for a matrix of-- 00:22:57.840 --> 00:23:00.850 for the n by n permutation. 00:23:00.850 --> 00:23:03.720 Yeah, so that's number w. 00:23:03.720 --> 00:23:07.920 And of course, this guy is w squared. 00:23:07.920 --> 00:23:15.840 This one is w cubed, w fourth, w fifth, sixth, seventh, 00:23:15.840 --> 00:23:19.480 and w to the eighth is the same as 1. 00:23:19.480 --> 00:23:19.980 Right. 00:23:27.685 --> 00:23:30.160 The reason I put those numbers up there 00:23:30.160 --> 00:23:32.740 is that they come into the eigenvectors as well 00:23:32.740 --> 00:23:34.120 as the eigenvalues. 00:23:34.120 --> 00:23:37.570 They are the eigenvalues, these 8 numbers. 00:23:37.570 --> 00:23:45.220 1, 2, 3, 4, 5, 6, 7, 8 are the 8 eigenvalues of the matrix. 00:23:45.220 --> 00:23:48.160 Here's the 4 by 4 case. 00:23:48.160 --> 00:23:51.340 The matrix is an orthogonal matrix. 00:23:51.340 --> 00:23:54.880 Oh, what does that tell us about the eigenvectors? 00:23:54.880 --> 00:23:57.760 The eigenvectors of an orthogonal matrix 00:23:57.760 --> 00:24:05.180 are orthogonal just like symmetric matrices. 00:24:05.180 --> 00:24:12.820 So, do you know that little list of matrices 00:24:12.820 --> 00:24:20.650 with orthogonal eigenvectors? 00:24:25.570 --> 00:24:27.900 I'm going to call them q. 00:24:27.900 --> 00:24:36.520 So qi dotted qj, the inner product, is 1 or 0. 00:24:36.520 --> 00:24:42.280 1 if i equal j, 0 if i is not j. 00:24:42.280 --> 00:24:43.660 Orthogonal eigenvectors. 00:24:43.660 --> 00:24:46.480 Now, what matrices have orthogonal eigenvectors? 00:24:46.480 --> 00:24:49.120 We're going back to linear algebra 00:24:49.120 --> 00:24:53.110 because this is a fundamental fact to know, 00:24:53.110 --> 00:24:57.370 this family of wonderful matrices. 00:24:57.370 --> 00:25:00.340 Matrices with orthogonal eigenvectors. 00:25:00.340 --> 00:25:03.296 Or tell me one bunch of matrices that you know 00:25:03.296 --> 00:25:05.470 has orthogonal eigenvectors. 00:25:05.470 --> 00:25:06.420 STUDENT: Symmetric. 00:25:06.420 --> 00:25:07.642 GILBERT STRANG: Symmetric. 00:25:12.370 --> 00:25:15.150 And what is special about the eigenvalues? 00:25:15.150 --> 00:25:17.730 They're real. 00:25:17.730 --> 00:25:21.330 But there are other matrices that 00:25:21.330 --> 00:25:25.720 have orthogonal eigenvectors and we really 00:25:25.720 --> 00:25:28.670 should know the whole story about those guys. 00:25:28.670 --> 00:25:31.090 They're too important not to know. 00:25:31.090 --> 00:25:34.550 So, what's another bunch of matrices? 00:25:34.550 --> 00:25:39.010 So, these symmetric matrices have orthogonal eigenvectors 00:25:39.010 --> 00:25:41.230 and-- 00:25:41.230 --> 00:25:44.590 real symmetrics and the eigenvalues will be real. 00:25:44.590 --> 00:25:47.980 Well, what other kind of matrices 00:25:47.980 --> 00:25:51.220 have orthogonal eigenvectors? 00:25:51.220 --> 00:25:56.200 But they might be complex and the eigenvalues 00:25:56.200 --> 00:25:58.240 might be complex. 00:25:58.240 --> 00:26:03.640 And you can't know Fourier without saying, OK, I can 00:26:03.640 --> 00:26:07.600 deal with this complex number. 00:26:07.600 --> 00:26:10.750 OK, so what's another family of matrices that 00:26:10.750 --> 00:26:13.960 has orthogonal eigenvectors? 00:26:13.960 --> 00:26:14.460 Yes. 00:26:14.460 --> 00:26:15.585 STUDENT: Diagonal matrices. 00:26:15.585 --> 00:26:19.270 GILBERT STRANG: Diagonal for sure, right? 00:26:19.270 --> 00:26:29.860 And then we know that we have the eigenvectors go 00:26:29.860 --> 00:26:32.710 into the identity matrix, right. 00:26:32.710 --> 00:26:35.740 Yeah, so we know everything about diagonal ones. 00:26:35.740 --> 00:26:38.070 You could say those are included in symmetric. 00:26:38.070 --> 00:26:40.480 Now, let's get some new ones. 00:26:40.480 --> 00:26:41.500 What else? 00:26:41.500 --> 00:26:42.940 STUDENT: [INAUDIBLE] 00:26:42.940 --> 00:26:45.580 GILBERT STRANG: Orthogonal matrices count. 00:26:45.580 --> 00:26:51.760 Orthogonal matrices, like permutations or like rotations 00:26:51.760 --> 00:26:54.180 or like reflections. 00:26:54.180 --> 00:26:55.160 Orthogonal matrices. 00:26:58.990 --> 00:27:02.970 And what's special about their eigenvalues? 00:27:02.970 --> 00:27:06.684 The eigenvalues of an orthogonal matrix? 00:27:06.684 --> 00:27:07.752 STUDENT: [INAUDIBLE] 00:27:07.752 --> 00:27:09.585 GILBERT STRANG: The magnitude is 1, exactly. 00:27:12.110 --> 00:27:16.110 It has to be 1 because an orthogonal matrix doesn't 00:27:16.110 --> 00:27:18.420 change the length of the vector. 00:27:18.420 --> 00:27:26.090 Q times x has the same length as x for all vectors. 00:27:26.090 --> 00:27:29.130 And in particular, for eigenvectors. 00:27:29.130 --> 00:27:36.410 So, if this was an eigenvector, Q x would equal lambda x. 00:27:36.410 --> 00:27:39.510 And now if that equals that, then lambda has to be 1. 00:27:39.510 --> 00:27:42.010 The magnitude of lambda has to be 1. 00:27:42.010 --> 00:27:44.540 Of course. 00:27:44.540 --> 00:27:46.700 Complex numbers are expected here 00:27:46.700 --> 00:27:49.610 and that's exactly what we're seeing here. 00:27:49.610 --> 00:27:53.480 All the eigenvalues of permutations 00:27:53.480 --> 00:27:56.460 are very special orthogonal matrices. 00:27:56.460 --> 00:27:59.720 I won't add permutations separately to the list 00:27:59.720 --> 00:28:00.590 but they count. 00:28:07.900 --> 00:28:10.660 The fact that this is on the list 00:28:10.660 --> 00:28:13.150 tells us that the eigenvectors that we're going to find 00:28:13.150 --> 00:28:14.200 are orthogonal. 00:28:14.200 --> 00:28:16.450 We don't have to do a separate check 00:28:16.450 --> 00:28:19.930 to see that they are once we compute them. 00:28:19.930 --> 00:28:21.520 They have to be. 00:28:21.520 --> 00:28:24.400 They're the eigenvectors of an orthogonal matrix. 00:28:24.400 --> 00:28:29.320 Now, I could ask you-- let's keep going with this 00:28:29.320 --> 00:28:32.890 and get the whole list here. 00:28:32.890 --> 00:28:38.320 Along with symmetric there is another bunch of guys. 00:28:38.320 --> 00:28:39.670 Antisymmetric. 00:28:39.670 --> 00:28:42.070 Big deal, but those are important. 00:28:42.070 --> 00:28:48.220 So, symmetric means A transpose equals A. Diagonal you know. 00:28:48.220 --> 00:28:54.110 A transpose equals A inverse for orthogonal matrices. 00:28:54.110 --> 00:28:57.730 Now, I'm going to put in antisymmetric matrices 00:28:57.730 --> 00:29:05.810 where A transpose is minus A. What 00:29:05.810 --> 00:29:08.840 do you think you know about the eigenvalues 00:29:08.840 --> 00:29:11.180 for antisymmetric matrices? 00:29:14.220 --> 00:29:17.190 Shall we take a example? 00:29:17.190 --> 00:29:18.600 Anti symmetric matrix. 00:29:21.940 --> 00:29:26.440 Say 0, 0, 1, and minus 1. 00:29:26.440 --> 00:29:28.540 What are the eigenvalues of that? 00:29:28.540 --> 00:29:36.180 Well, if I subtract lambda from the diagonal 00:29:36.180 --> 00:29:43.460 and take the determinant, I get lambda squared plus 1 equals 0. 00:29:43.460 --> 00:29:46.330 So lambda is i or minus i. 00:29:50.740 --> 00:29:51.995 That's a rotation matrix. 00:29:55.380 --> 00:29:58.070 It's a rotation through 90 degrees. 00:29:58.070 --> 00:30:00.320 So there could not be a real eigenvalue. 00:30:00.320 --> 00:30:04.160 Have you thought about that? 00:30:04.160 --> 00:30:05.710 Or a real eigenvector. 00:30:05.710 --> 00:30:09.170 If I rotate every vector, how could a vector 00:30:09.170 --> 00:30:11.810 come out a multiple of itself? 00:30:11.810 --> 00:30:19.120 How could I have A transpose times the vector equal lambda 00:30:19.120 --> 00:30:20.600 times a vector? 00:30:20.600 --> 00:30:24.170 I've rotated it and yet it's in the same direction. 00:30:24.170 --> 00:30:32.030 Well, somehow that's possible in imaginary space 00:30:32.030 --> 00:30:34.820 and not possible in real space. 00:30:34.820 --> 00:30:41.940 OK, so here the lambdas are imaginary. 00:30:41.940 --> 00:30:47.060 And now finally, tell me if you know 00:30:47.060 --> 00:30:52.370 the name of the whole family of matrices that 00:30:52.370 --> 00:30:55.010 includes all of those and more. 00:30:55.010 --> 00:30:59.810 Of matrices with orthogonal eigenvectors. 00:30:59.810 --> 00:31:02.990 So, what are the special properties then? 00:31:02.990 --> 00:31:04.620 These would be matrices. 00:31:04.620 --> 00:31:09.470 Shall I call them M for matrix? 00:31:09.470 --> 00:31:12.740 So, it has orthogonal eigenvectors. 00:31:12.740 --> 00:31:17.210 So it's Q times the diagonal times Q transpose. 00:31:21.910 --> 00:31:24.010 I've really written down somehow-- 00:31:24.010 --> 00:31:25.840 I haven't written a name down for them 00:31:25.840 --> 00:31:30.160 but that's the way to get them. 00:31:30.160 --> 00:31:35.870 I'm allowing any orthogonal eigenvectors. 00:31:35.870 --> 00:31:37.330 So, this is diagonalized. 00:31:37.330 --> 00:31:41.510 I've diagonalized the matrix. 00:31:41.510 --> 00:31:43.660 And here are any eigenvalues. 00:31:43.660 --> 00:31:49.000 So, the final guy on this list allows any eigenvalues, 00:31:49.000 --> 00:31:51.220 any complex numbers. 00:31:51.220 --> 00:31:56.680 But the eigenvectors, I want to be orthogonal. 00:31:56.680 --> 00:32:00.730 So that's why I have the Q. 00:32:00.730 --> 00:32:03.620 So, how would you recognize such a matrix 00:32:03.620 --> 00:32:07.980 and what is the name for them? 00:32:07.980 --> 00:32:11.550 We're going beyond 18.06, because probably I 00:32:11.550 --> 00:32:20.200 don't mention the name for these matrices in 18.06, but I could. 00:32:20.200 --> 00:32:23.440 Anybody know it? 00:32:23.440 --> 00:32:29.860 A matrix of that form is a normal matrix. 00:32:29.860 --> 00:32:31.060 Normal. 00:32:31.060 --> 00:32:34.790 So, that's the total list, is a normal matrix. 00:32:41.870 --> 00:32:43.920 So, normal matrices look like that. 00:32:47.760 --> 00:32:51.910 I have to apologize for whoever thought up that name, normal. 00:32:51.910 --> 00:32:55.155 I mean that's like, OK. 00:32:55.155 --> 00:32:57.030 A little more thought, you could have come up 00:32:57.030 --> 00:33:01.950 with something more meaningful than just, say, normal. 00:33:01.950 --> 00:33:05.490 [INAUDIBLE] that's the absolute opposite of normal. 00:33:05.490 --> 00:33:08.670 Almost all matrices are not normal. 00:33:08.670 --> 00:33:10.890 So anyway, but that's what they're called. 00:33:10.890 --> 00:33:12.360 Normal matrices. 00:33:12.360 --> 00:33:17.070 And finally, how do you recognize a normal matrix? 00:33:17.070 --> 00:33:20.010 Everybody knows how to recognize a symmetric matrix 00:33:20.010 --> 00:33:22.410 or a diagonal matrix, and we even 00:33:22.410 --> 00:33:26.610 know how to recognize an orthogonal matrix or skew 00:33:26.610 --> 00:33:27.900 or antisymmetric. 00:33:27.900 --> 00:33:31.200 But what's the quick test for a normal matrix? 00:33:33.800 --> 00:33:40.240 Well, I'll just tell you that a normal matrix has M transpose 00:33:40.240 --> 00:33:42.490 M equal M M transpose. 00:33:45.970 --> 00:33:48.250 I'm talking here about real matrices 00:33:48.250 --> 00:33:50.860 and I really should move to complex. 00:33:50.860 --> 00:33:54.445 But let me just think of them as real. 00:33:57.310 --> 00:34:01.000 Well, the trouble is that the matrices might be real 00:34:01.000 --> 00:34:04.910 but the eigenvectors are not going to be real 00:34:04.910 --> 00:34:07.130 and the eigenvalues are not going to be real. 00:34:07.130 --> 00:34:08.489 So, really I-- 00:34:08.489 --> 00:34:10.719 I'm sorry to say really again-- 00:34:10.719 --> 00:34:17.090 I should get out of the limitation to real. 00:34:17.090 --> 00:34:18.100 Yeah. 00:34:18.100 --> 00:34:22.570 And how do I get out of the limitation to real? 00:34:22.570 --> 00:34:26.980 What do I change here if M is a complex matrix instead 00:34:26.980 --> 00:34:28.520 of a real matrix? 00:34:28.520 --> 00:34:30.790 Then whenever you transpose it you 00:34:30.790 --> 00:34:33.880 should take its complex conjugate. 00:34:33.880 --> 00:34:35.920 So now that that's the real thing. 00:34:35.920 --> 00:34:39.750 That's the normal thing, that's the right thing. 00:34:39.750 --> 00:34:41.920 Yeah, right thing. 00:34:41.920 --> 00:34:42.750 Better. 00:34:42.750 --> 00:34:46.010 OK, so that's a normal matrix. 00:34:46.010 --> 00:34:48.790 And you can check that if you took 00:34:48.790 --> 00:34:55.239 that M and you figured out M transpose and did that, 00:34:55.239 --> 00:34:56.980 it would work. 00:34:56.980 --> 00:34:59.350 Because in the end the Q's cancel 00:34:59.350 --> 00:35:05.890 and you just have 2 diagonal matrices there 00:35:05.890 --> 00:35:11.200 and that's sort of automatic, that diagonal matrices commute. 00:35:11.200 --> 00:35:16.360 So, a normal matrix is one that commutes with its transpose. 00:35:16.360 --> 00:35:19.390 Commutes with its transpose or its conjugate 00:35:19.390 --> 00:35:21.850 transpose in the complex case. 00:35:21.850 --> 00:35:25.990 OK, why did I say all that? 00:35:25.990 --> 00:35:30.450 Simply because-- oh, I guess that-- 00:35:30.450 --> 00:35:39.280 so the permutation P is orthogonal 00:35:39.280 --> 00:35:42.070 so its eigenvectors, which we're going to write down 00:35:42.070 --> 00:35:45.100 in a minute, are orthogonal. 00:35:45.100 --> 00:35:51.865 But actually, this matrix C will be a normal matrix. 00:35:58.670 --> 00:36:01.100 I didn't see that coming as I started 00:36:01.100 --> 00:36:03.170 talking about these guys. 00:36:03.170 --> 00:36:06.380 Yeah, so that's a normal matrix. 00:36:06.380 --> 00:36:08.780 Because circulant matrices commute. 00:36:08.780 --> 00:36:11.410 Any 2 circulant matrices commute. 00:36:11.410 --> 00:36:13.900 C1 C2 equals C2 C1. 00:36:18.160 --> 00:36:21.220 And now if C2 is the transpose of-- 00:36:21.220 --> 00:36:24.190 so, here's a matrix. 00:36:24.190 --> 00:36:25.870 Yeah, so these are matrices here. 00:36:29.710 --> 00:36:30.860 Circulants all commute. 00:36:30.860 --> 00:36:33.730 It's a little family of matrices. 00:36:33.730 --> 00:36:36.660 When you multiply them together you get more of them. 00:36:36.660 --> 00:36:39.100 You're just staying in that little circulant 00:36:39.100 --> 00:36:43.060 world with n parameters. 00:36:43.060 --> 00:36:46.570 And once you know the first row, you know all the other rows. 00:36:50.080 --> 00:36:54.790 So in fact, they all have the same eigenvectors. 00:36:54.790 --> 00:37:00.890 So, now let me be sure we get the eigenvectors straight. 00:37:00.890 --> 00:37:01.390 OK. 00:37:08.820 --> 00:37:28.320 OK, eigenvectors of P will also be eigenvectors of C 00:37:28.320 --> 00:37:42.100 because it's a combination of powers of P. 00:37:42.100 --> 00:37:44.770 So once I find the eigenvectors of P, 00:37:44.770 --> 00:37:48.430 I've found the eigenvectors of any circulant matrix. 00:37:48.430 --> 00:37:50.980 And these eigenvectors are very special, 00:37:50.980 --> 00:37:53.110 and that's the connection to Fourier. 00:37:53.110 --> 00:37:56.860 That's why-- we expect a connection to Fourier 00:37:56.860 --> 00:37:59.560 because we have something periodic. 00:37:59.560 --> 00:38:02.650 And that's what Fourier is entirely about. 00:38:02.650 --> 00:38:05.530 OK, so what are these eigenvectors? 00:38:05.530 --> 00:38:11.800 Let's take P to be 4 by 4. 00:38:20.110 --> 00:38:23.810 OK, so the eigenvectors are-- 00:38:23.810 --> 00:38:27.040 so we remember, the eigenvalues are lambda equal 1, 00:38:27.040 --> 00:38:30.910 lambda equal minus 1, lambda equal I, 00:38:30.910 --> 00:38:35.730 and lambda equal minus I. We've got 4 eigenvectors to find 00:38:35.730 --> 00:38:38.980 and when we find those, you'll have the picture. 00:38:38.980 --> 00:38:43.876 OK, what's the eigenvector for lambda equal 1? 00:38:43.876 --> 00:38:44.820 STUDENT: 1, 1, 1, 1. 00:38:44.820 --> 00:38:47.220 GILBERT STRANG: 1, 1, 1, 1. 00:38:47.220 --> 00:38:50.340 So, let me make it into a vector. 00:38:50.340 --> 00:38:54.690 And the eigenvector for lambda equal minus 1 is? 00:38:54.690 --> 00:38:58.870 So, I want this shift to change every sign. 00:38:58.870 --> 00:39:06.240 So I better alternate those signs so that if I shift it, 00:39:06.240 --> 00:39:08.460 the 1 goes to the minus 1. 00:39:08.460 --> 00:39:10.020 Minus 1 goes to the 1. 00:39:10.020 --> 00:39:12.880 So the eigenvalue is minus 1. 00:39:12.880 --> 00:39:16.090 Now, what about the eigenvalues of i? 00:39:16.090 --> 00:39:20.190 Sorry, the eigenvector that goes with eigenvalue i? 00:39:23.790 --> 00:39:30.640 If I start it with 1 and I do the permutation, 00:39:30.640 --> 00:39:35.330 I think I just want i, i squared, i cubed there. 00:39:35.330 --> 00:39:37.970 And I think with this guy, with minus i, 00:39:37.970 --> 00:39:42.100 I think I want the vector 1, minus i, 00:39:42.100 --> 00:39:44.980 minus i squared, minus i cubed. 00:39:52.810 --> 00:39:57.100 So without stopping to check, let's 00:39:57.100 --> 00:40:00.880 just see the nice point here. 00:40:00.880 --> 00:40:03.160 All the components of eigenvectors 00:40:03.160 --> 00:40:08.410 are in this picture. 00:40:08.410 --> 00:40:10.630 Here we've got 8 eigenvectors. 00:40:10.630 --> 00:40:12.910 8 eigenvalues, 8 eigenvectors. 00:40:12.910 --> 00:40:15.250 The eigenvectors have 8 components 00:40:15.250 --> 00:40:20.210 and every component is one of these 8 numbers. 00:40:20.210 --> 00:40:23.300 The whole thing is constructed from the same 8 numbers. 00:40:23.300 --> 00:40:26.840 The eigenvalues and the eigenvectors. 00:40:26.840 --> 00:40:29.990 And really the key point is, what 00:40:29.990 --> 00:40:33.200 is the matrix of eigenvectors? 00:40:33.200 --> 00:40:35.210 So, let's just write that down. 00:40:39.220 --> 00:40:56.320 So, the eigenvector matrix for all circulants of size 00:40:56.320 --> 00:41:06.520 N. They all have the same eigenvectors, including 00:41:06.520 --> 00:41:18.710 P. All circulants C of size N including P of size N. 00:41:18.710 --> 00:41:20.690 So, what's the eigenvector matrix? 00:41:20.690 --> 00:41:24.320 What are the eigenvectors? 00:41:24.320 --> 00:41:31.490 Well, the first vector is all 1's. 00:41:31.490 --> 00:41:33.840 Just as there. 00:41:33.840 --> 00:41:36.950 So, that's an eigenvector of P, right? 00:41:36.950 --> 00:41:44.330 Because if I multiply by P, I do a shift, a cyclic shift, 00:41:44.330 --> 00:41:46.870 and I've got all 1's. 00:41:46.870 --> 00:41:51.860 The next eigenvector is powers of w. 00:41:59.760 --> 00:42:01.970 And let me remind you, everything 00:42:01.970 --> 00:42:03.845 is going to be powers of w. 00:42:03.845 --> 00:42:10.260 e to the 2 pi i over N. It's that complex number 00:42:10.260 --> 00:42:13.880 that's 1/n of the way around. 00:42:13.880 --> 00:42:16.950 So, what happens if I multiply that by P? 00:42:16.950 --> 00:42:24.920 It shift it and it multiplies by w or 1/w, 00:42:24.920 --> 00:42:27.200 which is another eigenvector. 00:42:27.200 --> 00:42:32.960 OK, and then the next one in this list will be going with w 00:42:32.960 --> 00:42:33.590 squared. 00:42:33.590 --> 00:42:39.570 So it will be w fourth, w to the sixth, w to the eighth. 00:42:39.570 --> 00:42:43.480 Wait a minute, did I get these lined up all right? 00:42:43.480 --> 00:42:45.410 w goes with w squared. 00:42:45.410 --> 00:42:45.910 Whoops. 00:42:51.610 --> 00:42:52.520 w squared. 00:42:52.520 --> 00:42:57.550 Now it's w to the fourth, w the sixth, w to the eighth, 00:42:57.550 --> 00:43:01.965 w to the 10th, w to the 12th, and w to the 14th. 00:43:05.470 --> 00:43:06.310 And they keep going. 00:43:09.940 --> 00:43:13.780 So that's the eigenvector with eigenvalue 1. 00:43:13.780 --> 00:43:18.460 This will have the eigenvalue-- it's either w or the conjugate, 00:43:18.460 --> 00:43:21.280 might be the conjugate, w bar. 00:43:21.280 --> 00:43:24.610 And you see this matrix. 00:43:24.610 --> 00:43:27.300 So, what would be the last eigenvector? 00:43:27.300 --> 00:43:29.740 It would be w-- 00:43:29.740 --> 00:43:33.100 so this is 8 by 8. 00:43:33.100 --> 00:43:36.530 I'm going to call that the Fourier matrix of size 8. 00:43:39.370 --> 00:43:41.350 And it's the eigenvector matrix. 00:43:41.350 --> 00:43:50.710 So Fourier matrix equals eigenvector matrix. 00:43:58.490 --> 00:44:02.680 So, what I'm saying is that the linear algebra 00:44:02.680 --> 00:44:06.970 for these circulants is fantastic. 00:44:06.970 --> 00:44:09.880 They all have the same eigenvector matrix. 00:44:09.880 --> 00:44:12.790 It happens to be the most important complex matrix 00:44:12.790 --> 00:44:19.090 in the world and its properties are golden. 00:44:19.090 --> 00:44:23.420 And it allows the fast Fourier transform, 00:44:23.420 --> 00:44:26.980 which we could write in matrix language next time. 00:44:26.980 --> 00:44:30.010 And all the entries are powers of w. 00:44:30.010 --> 00:44:36.130 All the entries are on the unit circle 00:44:36.130 --> 00:44:38.830 at one of those 8 points. 00:44:38.830 --> 00:44:43.780 And the last guy would be w to the seventh, w to the 14th, w 00:44:43.780 --> 00:44:52.345 to the 21st, 28th, 35th, 42nd, and 49th. 00:44:56.970 --> 00:45:00.960 So, w to the 49th would be the last. 00:45:00.960 --> 00:45:02.460 7 squared. 00:45:02.460 --> 00:45:06.350 It starts out with w to the 0 times 0. 00:45:11.880 --> 00:45:15.510 You see that picture. 00:45:15.510 --> 00:45:16.890 w to the 49th. 00:45:16.890 --> 00:45:19.530 What is actually w to the 49th? 00:45:19.530 --> 00:45:26.600 If w is the eighth root of 1, so we have w to the eighth, 00:45:26.600 --> 00:45:31.170 it's 1 because I'm doing 8 by 8. 00:45:31.170 --> 00:45:33.450 What is w to the 49th power? 00:45:36.306 --> 00:45:37.260 STUDENT: [INAUDIBLE] 00:45:37.260 --> 00:45:38.020 GILBERT STRANG: w? 00:45:38.020 --> 00:45:39.450 It's the same as w. 00:45:39.450 --> 00:45:47.260 OK, because w to the 48th is 1, right? 00:45:47.260 --> 00:45:51.380 I take the sixth power of this and I get that w to the 48th 00:45:51.380 --> 00:45:52.170 is 1. 00:45:52.170 --> 00:45:54.862 So w to the 49th is the same as w. 00:45:59.520 --> 00:46:04.700 Every column, every entry, in the matrix is a power of w. 00:46:04.700 --> 00:46:08.310 And in fact, that power is just the column number 00:46:08.310 --> 00:46:10.010 times the row number. 00:46:10.010 --> 00:46:12.800 Yeah, so those are the good matrices. 00:46:19.860 --> 00:46:22.660 So, that is an orthogonal matrix. 00:46:22.660 --> 00:46:25.800 Well, almost. 00:46:25.800 --> 00:46:28.770 It has orthogonal columns but it doesn't 00:46:28.770 --> 00:46:30.570 have orthonormal columns. 00:46:33.090 --> 00:46:37.404 What's the length of that column vector? 00:46:37.404 --> 00:46:38.400 STUDENT: [INAUDIBLE] 00:46:38.400 --> 00:46:41.060 GILBERT STRANG: The square root of 8, right. 00:46:41.060 --> 00:46:44.270 I add up 1 squared 8 times and I take the square root, 00:46:44.270 --> 00:46:46.950 I get to the square root of 8. 00:46:46.950 --> 00:46:49.880 So, this is really-- 00:46:49.880 --> 00:46:54.860 it's the square root of 8 times an orthogonal matrix. 00:46:58.670 --> 00:46:59.170 Of course. 00:46:59.170 --> 00:47:02.780 The square root of 8 is just a number 00:47:02.780 --> 00:47:09.180 to divide out to make the columns orthonormal instead 00:47:09.180 --> 00:47:11.060 of just orthogonal. 00:47:11.060 --> 00:47:15.080 But how do I know that those are orthogonal? 00:47:15.080 --> 00:47:18.890 Well, I know they have to be but I'd like to see it clearly. 00:47:18.890 --> 00:47:24.050 Why is that vector orthogonal to that vector? 00:47:24.050 --> 00:47:25.850 First of all, they have to be. 00:47:25.850 --> 00:47:31.220 Because the matrix is a normal matrix. 00:47:31.220 --> 00:47:34.700 Normal matrices have orthogonal-- 00:47:34.700 --> 00:47:38.370 oh yeah, how do I know it's a normal matrix? 00:47:38.370 --> 00:47:39.725 So, I guess I can do the test. 00:47:44.840 --> 00:47:49.290 If I have the permutation P, I know that P transpose P 00:47:49.290 --> 00:47:50.790 equals P P transpose. 00:47:50.790 --> 00:47:54.440 The permutations commute. 00:47:54.440 --> 00:47:57.080 So, it's a normal matrix. 00:47:57.080 --> 00:48:01.640 But I'd like to see directly why is the dot 00:48:01.640 --> 00:48:05.060 product of the first or the 0-th eigenvector 00:48:05.060 --> 00:48:08.810 and the eigenvector equals 0? 00:48:08.810 --> 00:48:10.400 Let me take that dot product. 00:48:10.400 --> 00:48:12.430 1 times 1 is 1. 00:48:12.430 --> 00:48:14.798 1 times w is w. 00:48:14.798 --> 00:48:17.960 1 times w squared is w squared. 00:48:17.960 --> 00:48:23.200 Up to w to the seventh, I guess I'm going to finish at, 00:48:23.200 --> 00:48:24.050 equals 0. 00:48:29.610 --> 00:48:32.460 Well, what's that saying? 00:48:32.460 --> 00:48:44.340 Those numbers are these points in my picture, those 8 points. 00:48:44.340 --> 00:48:50.940 So, those are the 8 numbers that go into that column of-- 00:48:50.940 --> 00:48:52.590 that eigenvector. 00:48:52.590 --> 00:48:55.140 Why do they add to 0? 00:48:55.140 --> 00:49:01.600 How do you see that the sum of those 8 numbers is 0? 00:49:01.600 --> 00:49:03.040 STUDENT: There's symmetry. 00:49:03.040 --> 00:49:05.060 GILBERT STRANG: Yeah, the symmetry would do it. 00:49:05.060 --> 00:49:12.110 When I add that guy to that guy, w to the 0, or w to the eighth, 00:49:12.110 --> 00:49:14.870 or w to the 0. 00:49:14.870 --> 00:49:17.750 Yeah, when I add 1 and minus 1, I get 0. 00:49:17.750 --> 00:49:19.560 When I add these guys I get 0. 00:49:19.560 --> 00:49:21.280 When I add these-- 00:49:21.280 --> 00:49:23.420 by pairs. 00:49:23.420 --> 00:49:27.140 But what about a 3 by 3? 00:49:34.820 --> 00:49:35.815 So, 3 by 3. 00:49:38.420 --> 00:49:43.220 This would be e to the 2 pi i over 3. 00:49:43.220 --> 00:49:47.820 And then this would be w to the 4 pi-- 00:49:47.820 --> 00:49:52.250 this would be w squared, e to the 4 pi i over 3. 00:49:52.250 --> 00:49:56.345 And I believe that those 3 vectors add to 0. 00:49:59.200 --> 00:50:01.350 And therefore they are orthogonal to the 1, 00:50:01.350 --> 00:50:05.460 1, 1 eigenvector because the dot product will just 00:50:05.460 --> 00:50:07.480 want to add those 3 numbers. 00:50:07.480 --> 00:50:09.090 So why is that true? 00:50:09.090 --> 00:50:16.830 1 plus e the 2 pi i over 3 plus e to the 4 pi over 3 equals 0. 00:50:22.260 --> 00:50:27.420 Last minute of class today, we can figure out how to do that. 00:50:27.420 --> 00:50:29.590 Well, I could get a formula for-- 00:50:29.590 --> 00:50:33.240 that sum is 1 and I could get a closed form 00:50:33.240 --> 00:50:35.260 and check that I get the answer 0. 00:50:35.260 --> 00:50:44.780 The quick way to see it is maybe suppose I multiply by e 00:50:44.780 --> 00:50:48.530 to the 2 pi i over 3. 00:50:48.530 --> 00:50:52.730 So, I multiply every term, so that's e to the 2 pi i over 3. 00:50:52.730 --> 00:50:56.450 e to the 4 pi i over 3. 00:50:56.450 --> 00:51:00.530 And e to the 6 pi i over 3. 00:51:00.530 --> 00:51:03.060 OK, what do I learn from this? 00:51:03.060 --> 00:51:04.760 STUDENT: [INAUDIBLE] 00:51:04.760 --> 00:51:07.570 GILBERT STRANG: It's the same because e to the 6 pi i over 3 00:51:07.570 --> 00:51:08.340 is? 00:51:08.340 --> 00:51:09.110 STUDENT: 1. 00:51:09.110 --> 00:51:11.480 GILBERT STRANG: Is 1. 00:51:11.480 --> 00:51:14.210 That's 2 pi i, so that's 1. 00:51:14.210 --> 00:51:17.440 So I got the same sum, 1 plus this plus this. 00:51:17.440 --> 00:51:19.370 This plus this plus 1. 00:51:19.370 --> 00:51:24.710 So I got the same sum when I multiplied by that number. 00:51:24.710 --> 00:51:27.110 And that sum has to be 0. 00:51:27.110 --> 00:51:29.570 I can't get the same sum-- 00:51:29.570 --> 00:51:32.790 I can't multiply by this and get the same answer 00:51:32.790 --> 00:51:35.450 unless I'm multiplying 0. 00:51:35.450 --> 00:51:43.520 So that shows me that when n is odd I also have 00:51:43.520 --> 00:51:45.740 those n numbers adding to 0. 00:51:45.740 --> 00:51:48.740 OK, those are the basic-- 00:51:48.740 --> 00:51:54.410 the beautiful picture of the eigenvalues, 00:51:54.410 --> 00:51:57.620 the eigenvectors being orthogonal. 00:51:57.620 --> 00:52:04.460 And then the actual details here of what those eigenvectors are. 00:52:04.460 --> 00:52:05.750 OK, good. 00:52:05.750 --> 00:52:10.460 Hope you have a good weekend, and we've just got a week 00:52:10.460 --> 00:52:14.060 and a half left of class. 00:52:14.060 --> 00:52:17.310 I may probably have one more thing to do about Fourier 00:52:17.310 --> 00:52:19.760 and then we'll come back to other topics. 00:52:19.760 --> 00:52:25.640 But ask any questions, topics that you'd like to see 00:52:25.640 --> 00:52:27.590 included here. 00:52:27.590 --> 00:52:33.280 We're closing out 18.065 while you guys do the projects. 00:52:33.280 --> 00:52:35.800 OK, thank you.