1 00:00:06 --> 00:00:06 OK. 2 00:00:06 --> 00:00:14 Here's lecture sixteen and if you remember I ended up the last 3 00:00:14 --> 00:00:21.5 lecture with this formula for what I called a projection 4 00:00:21.5 --> 00:00:22 matrix. 5 00:00:22 --> 00:00:29 And maybe I could just recap for a minute what is that magic 6 00:00:29 --> 00:00:31 formula doing? 7 00:00:31 --> 00:00:37 For example, it's supposed to be -- 8 00:00:37 --> 00:00:42 it's supposed to produce a projection, if I multiply by a 9 00:00:42 --> 00:00:47 b, so I take P times b, I'm supposed to project that 10 00:00:47 --> 00:00:52 vector b to the nearest point in the column space. 11 00:00:52 --> 00:00:53 OK. 12 00:00:53 --> 00:00:58.7 Can I just -- one way to recap is to take the two extreme 13 00:00:58.7 --> 00:00:59 cases. 14 00:00:59 --> 00:01:03 Suppose a vector b is in the column space? 15 00:01:03 --> 00:01:09 Then what do I get when I apply the projection P? 16 00:01:09 --> 00:01:13.76 So I'm projecting into the column space but I'm starting 17 00:01:13.76 --> 00:01:17 with a vector in this case that's already in the column 18 00:01:17 --> 00:01:21 space, so of course when I project it I get B again, 19 00:01:21 --> 00:01:22 right. 20 00:01:22 --> 00:01:26 And I want to show you how that comes out of this formula. 21 00:01:26 --> 00:01:28 Let me do the other extreme. 22 00:01:28 --> 00:01:32 Suppose that vector is perpendicular to the column 23 00:01:32 --> 00:01:34 space. 24 00:01:34 --> 00:01:39 So imagine this column space as a plane and imagine b as 25 00:01:39 --> 00:01:43 sticking straight up perpendicular to it. 26 00:01:43 --> 00:01:48 What's the nearest point in the column space to b in that case? 27 00:01:48 --> 00:01:54 So what's the projection onto the plane, the nearest point in 28 00:01:54 --> 00:02:00.28 the plane, if the vector b that I'm looking at is -- 29 00:02:00.28 --> 00:02:04 got no component in the column space, it's sticking completely 30 00:02:04 --> 00:02:08 -- ninety degrees with it, then Pb should be zero, 31 00:02:08 --> 00:02:09 right. 32 00:02:09 --> 00:02:11 So those are the two extreme cases. 33 00:02:11 --> 00:02:15.96 The average vector has a component P in the column space 34 00:02:15.96 --> 00:02:20 and a component perpendicular to it, and what the projection does 35 00:02:20 --> 00:02:25 is it kills this part and it preserves this part. 36 00:02:25 --> 00:02:25 OK. 37 00:02:25 --> 00:02:29 Can we just see why that's true? 38 00:02:29 --> 00:02:33 Just -- that formula ought to work. 39 00:02:33 --> 00:02:36 So let me start with this one. 40 00:02:36 --> 00:02:43 What vectors are in the -- are perpendicular to the column 41 00:02:43 --> 00:02:43 space? 42 00:02:43 --> 00:02:49.06 How do I see that I really get zero? 43 00:02:49.06 --> 00:02:52 I have to think, what does it mean for a vector 44 00:02:52 --> 00:02:56 b to be perpendicular to the column space? 45 00:02:56 --> 00:03:00.98 So if it's perpendicular to all the columns, then it's in some 46 00:03:00.98 --> 00:03:01 other space. 47 00:03:01 --> 00:03:06 We've got our four spaces so the reason I do this is it's 48 00:03:06 --> 00:03:11 perfectly using what we know about our four spaces. 49 00:03:11 --> 00:03:15 What vectors are perpendicular to the column space? 50 00:03:15 --> 00:03:19 Those are the guys in the null space of A transpose, 51 00:03:19 --> 00:03:19 right? 52 00:03:19 --> 00:03:22 That's the first section of this chapter, 53 00:03:22 --> 00:03:25 that's the key geometry of these spaces. 54 00:03:25 --> 00:03:28 If I'm perpendicular to the column space, 55 00:03:28 --> 00:03:32 I'm in the null space of A transpose. 56 00:03:32 --> 00:03:32 OK. 57 00:03:32 --> 00:03:37 So if I'm in the null space of A transpose, and I multiply this 58 00:03:37 --> 00:03:41 big formula times b, so now I'm getting Pb, 59 00:03:41 --> 00:03:46 this is now the projection, Pb, do you see that I get zero? 60 00:03:46 --> 00:03:47 Of course I get zero. 61 00:03:47 --> 00:03:52 Right at the end there, A transpose b will give me zero 62 00:03:52 --> 00:03:53 right away. 63 00:03:53 --> 00:03:56 So that's why that zero's here. 64 00:03:56 --> 00:04:00 Because if I'm perpendicular to the column space, 65 00:04:00 --> 00:04:05.46 then I'm in the null space of A transpose and A transpose b is 66 00:04:05.46 --> 00:04:05 zilch. 67 00:04:05 --> 00:04:08 OK, what about the other possibility. 68 00:04:08 --> 00:04:13 How do I see that this formula gives me the right answer if b 69 00:04:13 --> 00:04:16 is in the column space? 70 00:04:16 --> 00:04:20 So what's a typical vector in the column space? 71 00:04:20 --> 00:04:23 It's a combination of the columns. 72 00:04:23 --> 00:04:26 How do I write a combination of the columns? 73 00:04:26 --> 00:04:31 So tell me, how would I write, you know, your everyday vector 74 00:04:31 --> 00:04:33 that's in the column space? 75 00:04:33 --> 00:04:38 It would have the form A times some x, right? 76 00:04:38 --> 00:04:42 That's what's in the column space, A times something. 77 00:04:42 --> 00:04:45.34 That makes it a combination of the columns. 78 00:04:45.34 --> 00:04:49 So these b's were in the null space of A transpose. 79 00:04:49 --> 00:04:52 These guys in the column space, those b's are Ax-s. 80 00:04:52 --> 00:04:53 Right? 81 00:04:53 --> 00:04:57.09 If b is in the column space then it has the form Ax. 82 00:04:57.09 --> 00:05:02 I'm going to stick that on the quiz or the final for sure. 83 00:05:02 --> 00:05:06 That you have to realize -- because we've said it like a 84 00:05:06 --> 00:05:12 thousand times that the things in the column space are vectors 85 00:05:12 --> 00:05:13 A times x. 86 00:05:13 --> 00:05:13 OK. 87 00:05:13 --> 00:05:17 And do you see what happens now if we use our formula? 88 00:05:17 --> 00:05:19 There's an A transpose A. 89 00:05:19 --> 00:05:23 Gets canceled by its inverse. 90 00:05:23 --> 00:05:26 We're left with an A times x. 91 00:05:26 --> 00:05:28 So the result was Ax. 92 00:05:28 --> 00:05:29 Which was b. 93 00:05:29 --> 00:05:32 Do you see that it works? 94 00:05:32 --> 00:05:35 This is that whole business. 95 00:05:35 --> 00:05:37 Cancel, cancel, leaving Ax. 96 00:05:37 --> 00:05:39 And Ax was b. 97 00:05:39 --> 00:05:43 So that turned out to be b, in this case. 98 00:05:43 --> 00:05:49 OK, so geometrically what we're seeing is we're taking a vector 99 00:05:49 --> 00:05:54.71 -- we've got the column space and 100 00:05:54.71 --> 00:06:01 perpendicular to that is the null space of A transpose. 101 00:06:01 --> 00:06:05.22 And our typical vector b is out here. 102 00:06:05.22 --> 00:06:10 There's zero, so there's our typical vector 103 00:06:10 --> 00:06:16 b, and what we're doing is we're projecting it to P. 104 00:06:16 --> 00:06:21 And the -- and of course at the same time 105 00:06:21 --> 00:06:26 we're finding the other part of it which is e. 106 00:06:26 --> 00:06:31 So the two pieces, the projection piece and the 107 00:06:31 --> 00:06:35 error piece, add up to the original b. 108 00:06:35 --> 00:06:36 OK. 109 00:06:36 --> 00:06:39 That's like what our matrix does. 110 00:06:39 --> 00:06:44 So this is P -- P is -- this P is Ab, 111 00:06:44 --> 00:06:48 is sorry -- is Pb, it's the projection, 112 00:06:48 --> 00:06:51 applied to b, and this one is -- OK, 113 00:06:51 --> 00:06:53 that's a projection too. 114 00:06:53 --> 00:06:57 That's a projection down onto that space. 115 00:06:57 --> 00:06:59 What's a good formula for it? 116 00:06:59 --> 00:07:04 Suppose I ask you for the projection of the projection 117 00:07:04 --> 00:07:10 matrix onto the -- this space, this perpendicular 118 00:07:10 --> 00:07:11 space? 119 00:07:11 --> 00:07:17 So if this projection was P, what's the projection that 120 00:07:17 --> 00:07:18 gives me e? 121 00:07:18 --> 00:07:24 It's the -- what I want is to get the rest of the vector, 122 00:07:24 --> 00:07:30 so it'll be just I minus P times b, that's a projection 123 00:07:30 --> 00:07:31 too. 124 00:07:31 --> 00:07:37 That's the projection onto the perpendicular space. 125 00:07:37 --> 00:07:38 OK. 126 00:07:38 --> 00:07:43 So if P's a projection, I minus P is a projection. 127 00:07:43 --> 00:07:47 If P is symmetric, I minus P is symmetric. 128 00:07:47 --> 00:07:52.96 If P squared equals P, then I minus P squared equals I 129 00:07:52.96 --> 00:07:53 minus P. 130 00:07:53 --> 00:08:00 It's just -- the algebra -- is only doing what your -- 131 00:08:00 --> 00:08:04 picture is completely telling you. 132 00:08:04 --> 00:08:08 But the algebra leads to this expression. 133 00:08:08 --> 00:08:16 That expression for P given -- given a basis for the subspace, 134 00:08:16 --> 00:08:22 given the matrix A whose columns are a basis for our 135 00:08:22 --> 00:08:23 column space. 136 00:08:23 --> 00:08:31 OK, that's recap because you -- you need to see that formula 137 00:08:31 --> 00:08:32 more than once. 138 00:08:32 --> 00:08:35 And now can I pick up on using it? 139 00:08:35 --> 00:08:40 So now -- and the -- it's like, let me do that again, 140 00:08:40 --> 00:08:46 I'll go right through a problem that I started at the end, 141 00:08:46 --> 00:08:49 which is find a best straight line. 142 00:08:49 --> 00:08:53 You remember that problem, I -- 143 00:08:53 --> 00:08:58 I picked a particular set of points, they weren't specially 144 00:08:58 --> 00:09:02 brilliant, t equal one, two, three, the heights were 145 00:09:02 --> 00:09:05 one, two, and then two again. 146 00:09:05 --> 00:09:09 So they were -- heights were that point, that point, 147 00:09:09 --> 00:09:14 which makes it look like I've got a nice forty-five-degree 148 00:09:14 --> 00:09:18 line -- but then the third point didn't 149 00:09:18 --> 00:09:19 lie on the line. 150 00:09:19 --> 00:09:23.82 And I wanted to find the best straight line. 151 00:09:23.82 --> 00:09:27 So I'm looking for the -- this line, y=C+Dt. 152 00:09:27 --> 00:09:32 And it's not going to go through all three points, 153 00:09:32 --> 00:09:36 because no line goes through all three points. 154 00:09:36 --> 00:09:40 So I'm going to pick the best line, the -- 155 00:09:40 --> 00:09:46 the best being the one that makes the overall error as small 156 00:09:46 --> 00:09:47 as I can make it. 157 00:09:47 --> 00:09:52.47 Now I have to tell you, what is that overall error? 158 00:09:52.47 --> 00:09:57 And -- because that determines what's the winning line. 159 00:09:57 --> 00:10:02 If we don't know -- I mean we have to decide what we mean by 160 00:10:02 --> 00:10:07 the error -- and then we minimize and we 161 00:10:07 --> 00:10:11.2 find the right -- the best C and D. 162 00:10:11.2 --> 00:10:11 OK. 163 00:10:11 --> 00:10:17 So if I went through this -- if I went through that point, 164 00:10:17 --> 00:10:20 I would solve the equation C+D=1. 165 00:10:20 --> 00:10:25 Because at t equal to one -- I'd have C plus D, 166 00:10:25 --> 00:10:28 and it would come out right. 167 00:10:28 --> 00:10:33 If it went through this point, I'd have C plus two D equal to 168 00:10:33 --> 00:10:33 two. 169 00:10:33 --> 00:10:38 Because at t equal to two, I would like to get the answer 170 00:10:38 --> 00:10:38 two. 171 00:10:38 --> 00:10:42.78 At the third point, I have C plus three D because t 172 00:10:42.78 --> 00:10:48 is three, but the -- the answer I'm shooting for is 173 00:10:48 --> 00:10:48 two again. 174 00:10:48 --> 00:10:51 So those are my three equations. 175 00:10:51 --> 00:10:54 And they don't have a solution. 176 00:10:54 --> 00:10:57 But they've got a best solution. 177 00:10:57 --> 00:11:00 What do I mean by best solution? 178 00:11:00 --> 00:11:05 So let me take time out to remember what I'm talking about 179 00:11:05 --> 00:11:06.79 for best solution. 180 00:11:06.79 --> 00:11:10 So this is my equation Ax=b. 181 00:11:10 --> 00:11:14.36 A is this matrix, one, one, one, 182 00:11:14.36 --> 00:11:19.39 one, two, three. x is my -- only have two 183 00:11:19.39 --> 00:11:25 unknowns, C and D, and b is my right-hand side, 184 00:11:25 --> 00:11:27 one, two, three. 185 00:11:27 --> 00:11:27 OK. 186 00:11:27 --> 00:11:29 No solution. 187 00:11:29 --> 00:11:35.61 Three eq- I have a three by two matrix, I do have two 188 00:11:35.61 --> 00:11:42 independent columns -- so I do have a basis for the 189 00:11:42 --> 00:11:45 column space, those two columns are 190 00:11:45 --> 00:11:50 independent, they're a basis for the column space, 191 00:11:50 --> 00:11:54 but the column space doesn't include that vector. 192 00:11:54 --> 00:12:00 So best possible in this -- what would best possible mean? 193 00:12:00 --> 00:12:06 The way that comes out to linear equations is I -- 194 00:12:06 --> 00:12:11 I want to minimize the sum of these -- I'm going to make an 195 00:12:11 --> 00:12:13 error here. 196 00:12:13 --> 00:12:16 I'm going to make an error here. 197 00:12:16 --> 00:12:19 I'm going to make an error there. 198 00:12:19 --> 00:12:25.02 And I'm going to sum and square and add up those errors. 199 00:12:25.02 --> 00:12:27 So it's a sum of squares. 200 00:12:27 --> 00:12:32.13 It's a least squares solution I'm looking for. 201 00:12:32.13 --> 00:12:37 So if I -- those errors are the difference 202 00:12:37 --> 00:12:38 between Ax and b. 203 00:12:38 --> 00:12:41 That's what I want to make small. 204 00:12:41 --> 00:12:45 And the way I'm measuring this -- this is a vector, 205 00:12:45 --> 00:12:45 right? 206 00:12:45 --> 00:12:47 This is e1,e2 ,e3. 207 00:12:47 --> 00:12:49 The Ax-b, this is the e. 208 00:12:49 --> 00:12:50 The error vector. 209 00:12:50 --> 00:12:52.86 And small means its length. 210 00:12:52.86 --> 00:12:56 The length of that vector. 211 00:12:56 --> 00:13:00 That's what I'm going to try to minimize. 212 00:13:00 --> 00:13:03 And it's convenient to square. 213 00:13:03 --> 00:13:09 If I make something small, I make -- this is a never 214 00:13:09 --> 00:13:11 negative quantity, right? 215 00:13:11 --> 00:13:14 The length of that vector. 216 00:13:14 --> 00:13:20 The length will be zero exactly when the -- 217 00:13:20 --> 00:13:22 when I have the zero vector here. 218 00:13:22 --> 00:13:26 That's exactly the case when I can solve exactly, 219 00:13:26 --> 00:13:29 b is in the column space, all great. 220 00:13:29 --> 00:13:31 But I'm not in that case now. 221 00:13:31 --> 00:13:33 I'm going to have an error vector, e. 222 00:13:33 --> 00:13:36 What's this error vector in my picture? 223 00:13:36 --> 00:13:40 I guess what I'm trying to say is there's -- 224 00:13:40 --> 00:13:44 there's two pictures of what's going on. 225 00:13:44 --> 00:13:47 There's two pictures of what's going on. 226 00:13:47 --> 00:13:52 One picture is -- in this is the three points and the line. 227 00:13:52 --> 00:13:56 And in that picture, what are the three errors? 228 00:13:56 --> 00:14:01 The three errors are what I miss by in this equation. 229 00:14:01 --> 00:14:05 So it's this -- this little bit here. 230 00:14:05 --> 00:14:08 That vertical distance up to the line. 231 00:14:08 --> 00:14:13 There's one -- sorry there's one, and there's C plus D. 232 00:14:13 --> 00:14:15 And it's that difference. 233 00:14:15 --> 00:14:17 Here's two and here's C+2D. 234 00:14:17 --> 00:14:21 So vertically it's that distance -- that little error 235 00:14:21 --> 00:14:22 there is e1. 236 00:14:22 --> 00:14:25 This little error here is e2. 237 00:14:25 --> 00:14:28 This little error coming up is e3. 238 00:14:28 --> 00:14:29.24 e3. 239 00:14:29.24 --> 00:14:32 And what's my overall error? 240 00:14:32 --> 00:14:36 Is e1 square plus e2 squared plus e3 squared. 241 00:14:36 --> 00:14:40 That's what I'm trying to make small. 242 00:14:40 --> 00:14:46 I -- some statisticians -- this is a big part of statistics, 243 00:14:46 --> 00:14:52 fitting straight lines is a big part of science -- 244 00:14:52 --> 00:14:58 and specifically statistics, where the right word to use 245 00:14:58 --> 00:15:01 would be regression. 246 00:15:01 --> 00:15:04.22 I'm doing regression here. 247 00:15:04.22 --> 00:15:06 Linear regression. 248 00:15:06 --> 00:15:12 And I'm using this sum of squares as the measure of error. 249 00:15:12 --> 00:15:18 Again, some statisticians would be -- 250 00:15:18 --> 00:15:21.42 they would say, OK, I'll solve that problem 251 00:15:21.42 --> 00:15:23 because it's the clean problem. 252 00:15:23 --> 00:15:26 It leads to a beautiful linear system. 253 00:15:26 --> 00:15:30 But they would be a little careful about these squares, 254 00:15:30 --> 00:15:32 for -- in this case. 255 00:15:32 --> 00:15:34 If one of these points was way off. 256 00:15:34 --> 00:15:40 Suppose I had a measurement at t equal zero that was way off. 257 00:15:40 --> 00:15:45 Well, would the straight line, would the best line be the same 258 00:15:45 --> 00:15:47 if I had this fourth point? 259 00:15:47 --> 00:15:50 Suppose I have this fourth data point. 260 00:15:50 --> 00:15:54 No, certainly the line would -- it wouldn't be the -- that 261 00:15:54 --> 00:15:56 wouldn't be the best line. 262 00:15:56 --> 00:16:00.96 Because that line would have a giant error -- 263 00:16:00.96 --> 00:16:05 and when I squared it it would be like way out of sight 264 00:16:05 --> 00:16:06 compared to the others. 265 00:16:06 --> 00:16:10.74 So this would be called by statisticians an outlier, 266 00:16:10.74 --> 00:16:15 and they would not be happy to see the whole problem turned 267 00:16:15 --> 00:16:19 topsy-turvy by this one outlier, which could be a mistake, 268 00:16:19 --> 00:16:21 after all. 269 00:16:21 --> 00:16:26 So they wouldn't -- so they wouldn't like maybe squaring, 270 00:16:26 --> 00:16:31 if there were outliers, they would want to identify 271 00:16:31 --> 00:16:32 them. 272 00:16:32 --> 00:16:32 OK. 273 00:16:32 --> 00:16:37 I'm not going to -- I don't want to suggest that least 274 00:16:37 --> 00:16:42 squares isn't used, it's the most used, 275 00:16:42 --> 00:16:46 but it's not exclusively used because it's a little -- 276 00:16:46 --> 00:16:48 overcompensates for outliers. 277 00:16:48 --> 00:16:50 Because of that squaring. 278 00:16:50 --> 00:16:50 OK. 279 00:16:50 --> 00:16:54 So suppose we don't have this guy, we just have these three 280 00:16:54 --> 00:16:55 equations. 281 00:16:55 --> 00:16:58 And I want to make -- minimize this error. 282 00:16:58 --> 00:16:58 OK. 283 00:16:58 --> 00:17:03 Now, what I said is there's two pictures to look at. 284 00:17:03 --> 00:17:06 One picture is this one. 285 00:17:06 --> 00:17:09 The three points, the best line. 286 00:17:09 --> 00:17:11 And the errors. 287 00:17:11 --> 00:17:16 Now, on this picture, what are these points on the 288 00:17:16 --> 00:17:20 line, the points that are really on the line? 289 00:17:20 --> 00:17:24 So they're -- points, let me call them P1, 290 00:17:24 --> 00:17:30 P2, and P3, those are three numbers, so this -- 291 00:17:30 --> 00:17:34 this height is P1, this height is P2, 292 00:17:34 --> 00:17:39 this height is P3, and what are those guys? 293 00:17:39 --> 00:17:46 Suppose those were the three values instead of -- there's b1, 294 00:17:46 --> 00:17:52 ev- everybody's seen all these -- sorry, my art is as usual not 295 00:17:52 --> 00:17:57 the greatest, but there's the given b1, 296 00:17:57 --> 00:18:00 the given b2, and the given b3. 297 00:18:00 --> 00:18:06 I promise not to put a single letter more on that picture. 298 00:18:06 --> 00:18:06 OK. 299 00:18:06 --> 00:18:12 There's b1, P1 is the one on the line, and e1 is the distance 300 00:18:12 --> 00:18:12 between. 301 00:18:12 --> 00:18:18 And same at points two and same at points three. 302 00:18:18 --> 00:18:20 OK, so what's up? 303 00:18:20 --> 00:18:22 What's up with those Ps? 304 00:18:22 --> 00:18:24 P1, P2, P3, what are they? 305 00:18:24 --> 00:18:29 They're the components, they lie on the line, 306 00:18:29 --> 00:18:29 right? 307 00:18:29 --> 00:18:33.76 They're the points which if instead of one, 308 00:18:33.76 --> 00:18:38 two, two, which were the b's, suppose I put P1, 309 00:18:38 --> 00:18:40 P2, P3 in here. 310 00:18:40 --> 00:18:45 I'll figure out in a minute what those numbers are. 311 00:18:45 --> 00:18:51 But I just want to get the picture of what I'm doing. 312 00:18:51 --> 00:18:54 If I put P1, P2, P3 in those three 313 00:18:54 --> 00:19:00 equations, what would be good about the three equations? 314 00:19:00 --> 00:19:02 I could solve them. 315 00:19:02 --> 00:19:04 A line goes through the Ps. 316 00:19:04 --> 00:19:08 So the P1, P2, P3 vector, that's in the column 317 00:19:08 --> 00:19:08 space. 318 00:19:08 --> 00:19:11.72 That is a combination of these columns. 319 00:19:11.72 --> 00:19:14 It's the closest combination. 320 00:19:14 --> 00:19:15 It's this picture. 321 00:19:15 --> 00:19:20 See, I've got the two pictures like here's the picture that 322 00:19:20 --> 00:19:25.15 shows the points, this is a picture in a 323 00:19:25.15 --> 00:19:30 blackboard plane, here's a picture that's showing 324 00:19:30 --> 00:19:32 the vectors. 325 00:19:32 --> 00:19:36.22 The vector b, which is in this case, 326 00:19:36.22 --> 00:19:41.23 in this example is the vector one, two, two. 327 00:19:41.23 --> 00:19:47 The column space is in this case spanned by the -- well, 328 00:19:47 --> 00:19:50 you see A there. 329 00:19:50 --> 00:19:55 The column space of the matrix one, one, one, 330 00:19:55 --> 00:19:56 one, two, three. 331 00:19:56 --> 00:20:00 And this picture shows the nearest point. 332 00:20:00 --> 00:20:06 There's the -- that point P1, P2, P3, which I'm going to 333 00:20:06 --> 00:20:12 compute before the end of this hour, is the closest point in 334 00:20:12 --> 00:20:14 the column space. 335 00:20:14 --> 00:20:14 OK. 336 00:20:14 --> 00:20:19 Let me -- t I don't dare leave it any 337 00:20:19 --> 00:20:23 longer -- can I just compute it now. 338 00:20:23 --> 00:20:27 So I want to compute -- find P. 339 00:20:27 --> 00:20:28 All right. 340 00:20:28 --> 00:20:29 Find P. 341 00:20:29 --> 00:20:33 Find x, which is CD, find P and P. 342 00:20:33 --> 00:20:34 OK. 343 00:20:34 --> 00:20:41 And I really should put these little hats on to remind myself 344 00:20:41 --> 00:20:50 that they're the estimated the best line, not the perfect line. 345 00:20:50 --> 00:20:50 OK. 346 00:20:50 --> 00:20:51 OK. 347 00:20:51 --> 00:20:53 How do I proceed? 348 00:20:53 --> 00:20:57.41 Let's just run through the mechanics. 349 00:20:57.41 --> 00:21:00 What's the equation for x? 350 00:21:00 --> 00:21:02 The -- or x hat. 351 00:21:02 --> 00:21:08 The equation for that is A transpose A x hat equals A 352 00:21:08 --> 00:21:11 transpose x -- A transpose b. 353 00:21:11 --> 00:21:18 The most -- I'm -- will venture to call that the most important 354 00:21:18 --> 00:21:22 equation in statistics. 355 00:21:22 --> 00:21:24 And in estimation. 356 00:21:24 --> 00:21:30 And whatever you're -- wherever you've got error and noise this 357 00:21:30 --> 00:21:34 is the estimate that you use first. 358 00:21:34 --> 00:21:34 OK. 359 00:21:34 --> 00:21:39 Whenever you're fitting things by a few parameters, 360 00:21:39 --> 00:21:42 that's the equation to use. 361 00:21:42 --> 00:21:43.93 OK, let's solve it. 362 00:21:43.93 --> 00:21:47 What is A transpose A? 363 00:21:47 --> 00:21:51 So I have to figure out what these matrices are. 364 00:21:51 --> 00:21:55 One, one, one, one, two, three and one, 365 00:21:55 --> 00:21:59 one, one, one, two, three, that gives me some 366 00:21:59 --> 00:22:04 matrix, that gives me a matrix, what do I get out of that, 367 00:22:04 --> 00:22:08 three, six, six, and one and four and nine, 368 00:22:08 --> 00:22:10 fourteen. 369 00:22:10 --> 00:22:11 OK. 370 00:22:11 --> 00:22:18 And what do I expect to see in that matrix and I do see it, 371 00:22:18 --> 00:22:24 just before I keep going with the calculation? 372 00:22:24 --> 00:22:29 I expect that matrix to be symmetric. 373 00:22:29 --> 00:22:34.59 I expect it to be invertible. 374 00:22:34.59 --> 00:22:40 And near the end of the course I'm going to say I expect it to 375 00:22:40 --> 00:22:44 be positive definite, but that's a future fact about 376 00:22:44 --> 00:22:48 this crucial matrix, A transpose A. 377 00:22:48 --> 00:22:48.43 OK. 378 00:22:48.43 --> 00:22:51 And now let me figure A transpose b. 379 00:22:51 --> 00:22:56 So let me -- can I tack on b as an extra column here, 380 00:22:56 --> 00:22:58 one, two, two? 381 00:22:58 --> 00:23:04 And tack on the extra A transpose b is -- looks like 382 00:23:04 --> 00:23:09.18 five and one and four and six, eleven. 383 00:23:09.18 --> 00:23:15 I think my equations are three C plus six D equals five, 384 00:23:15 --> 00:23:21 and six D plus fourt-six C plus fourteen D is eleven. 385 00:23:21 --> 00:23:26 Can I just for safety see if I did that right? 386 00:23:26 --> 00:23:33 One, one, one times one, two, two is five. 387 00:23:33 --> 00:23:39 One, two, three, that's one, four and six, 388 00:23:39 --> 00:23:40 eleven. 389 00:23:40 --> 00:23:42 Looks good. 390 00:23:42 --> 00:23:45 These are my equations. 391 00:23:45 --> 00:23:53 That's my -- they're called the normal equations. 392 00:23:53 --> 00:24:04 I'll just write that word down because it -- so I solve them. 393 00:24:04 --> 00:24:06 I solve that for C and D. 394 00:24:06 --> 00:24:11.48 I would like to -- before I solve them could I do one thing 395 00:24:11.48 --> 00:24:14 that's on the -- that's just above here? 396 00:24:14 --> 00:24:19 I would like to -- I'd like to find these equations from 397 00:24:19 --> 00:24:20 calculus. 398 00:24:20 --> 00:24:24 I'd like to find them from this minimizing thing. 399 00:24:24 --> 00:24:28 So what's the first error? 400 00:24:28 --> 00:24:34 The first error is what I missed by in the first equation. 401 00:24:34 --> 00:24:37 C plus D minus one squared. 402 00:24:37 --> 00:24:43.27 And the second error is what I miss in the second equation. 403 00:24:43.27 --> 00:24:46 C plus two D minus two squared. 404 00:24:46 --> 00:24:52 And the third error squared is C plus three D minus two 405 00:24:52 --> 00:24:53 squared. 406 00:24:53 --> 00:24:58 That's my -- overall squared error that I'm trying to 407 00:24:58 --> 00:25:01 minimize. 408 00:25:01 --> 00:25:01 OK. 409 00:25:01 --> 00:25:05 So how would you minimize that? 410 00:25:05 --> 00:25:12 OK, linear algebra has given us the equations for the minimum. 411 00:25:12 --> 00:25:15 But we could use calculus too. 412 00:25:15 --> 00:25:21 That's a function of two variables, C and D, 413 00:25:21 --> 00:25:24 and we're looking for the minimum. 414 00:25:24 --> 00:25:27.48 So how do we find it? 415 00:25:27.48 --> 00:25:33 Directly from calculus, we take partial derivatives, 416 00:25:33 --> 00:25:38 right, we've got two variables, C and D, so take the partial 417 00:25:38 --> 00:25:41 derivative with respect to C and set it to zero, 418 00:25:41 --> 00:25:43 and you'll get that equation. 419 00:25:43 --> 00:25:48 Take the partial derivative with respect -- I'm not going to 420 00:25:48 --> 00:25:51 write it all out, just -- you will. 421 00:25:51 --> 00:25:54.33 The partial derivative with respect to D, 422 00:25:54.33 --> 00:25:56 it -- you know, it's going to be linear, 423 00:25:56 --> 00:26:00 that's the beauty of these squares,that if I have the 424 00:26:00 --> 00:26:04.42 square of something and I take its derivative I get something 425 00:26:04.42 --> 00:26:04 linear. 426 00:26:04 --> 00:26:06 And this is what I get. 427 00:26:06 --> 00:26:10 So this is the derivative of the error with respect to C 428 00:26:10 --> 00:26:13 being zero, and this is the derivative of 429 00:26:13 --> 00:26:16 the error with respect to D being zero. 430 00:26:16 --> 00:26:20 Wherever you look, these equations keep coming. 431 00:26:20 --> 00:26:24 So now I guess I'm going to solve it, what will I do, 432 00:26:24 --> 00:26:27 I'll subtract, I'll do elimination of course, 433 00:26:27 --> 00:26:31 because that's the only thing I know how to do. 434 00:26:31 --> 00:26:36 Two of these away from this would give me -- let's see, 435 00:26:36 --> 00:26:39 six, so would that be two Ds equals one? 436 00:26:39 --> 00:26:39 Ha. 437 00:26:39 --> 00:26:44 So it wasn't -- I was afraid these numbers were going to come 438 00:26:44 --> 00:26:45 out awful. 439 00:26:45 --> 00:26:49.8 But if I take two of those away from that, the equation I get 440 00:26:49.8 --> 00:26:55.56 left is two D equals one, so I think D is a half and C is 441 00:26:55.56 --> 00:27:00 whatever back substitution gives, six D is three, 442 00:27:00 --> 00:27:05 so three C plus three is five, I'm doing back substitution 443 00:27:05 --> 00:27:09 now, right, three, can I do it in light letters, 444 00:27:09 --> 00:27:13 three C plus that six D is three equals five, 445 00:27:13 --> 00:27:19 so three C is two, so I think C is two-thirds. 446 00:27:19 --> 00:27:23 One-half and two-thirds. 447 00:27:23 --> 00:27:30 So the best line, the best line is the constant 448 00:27:30 --> 00:27:35 two-thirds plus one-half t. 449 00:27:35 --> 00:27:41 And I -- is my picture more or less right? 450 00:27:41 --> 00:27:48.98 Let me write, let me copy that best line down 451 00:27:48.98 --> 00:27:55 again, two-thirds and a half. 452 00:27:55 --> 00:27:58 Let me -- I'll put in the two-thirds and the half. 453 00:27:58 --> 00:27:59 OK. 454 00:27:59 --> 00:28:02 So what's this P1, that's the value at t equal to 455 00:28:02 --> 00:28:03.11 one. 456 00:28:03.11 --> 00:28:06 At t equal to one, I have two-thirds plus a half, 457 00:28:06 --> 00:28:10 which is -- what's that, four-sixths and three-sixths, 458 00:28:10 --> 00:28:16 so P1, oh, I promised not to write another thing on this -- 459 00:28:16 --> 00:28:19 I'll erase P1 and I'll put seven-sixths. 460 00:28:19 --> 00:28:20 OK. 461 00:28:20 --> 00:28:24 And yeah, it's above one, and e1 is one-sixth, 462 00:28:24 --> 00:28:24 right. 463 00:28:24 --> 00:28:26 You see it all. 464 00:28:26 --> 00:28:26 Right? 465 00:28:26 --> 00:28:27 What's P2? 466 00:28:27 --> 00:28:28 OK. 467 00:28:28 --> 00:28:32 At point t equal to two, where's my line here? 468 00:28:32 --> 00:28:36 At t equal to two, it's two-thirds plus one, 469 00:28:36 --> 00:28:38 right? 470 00:28:38 --> 00:28:40 That's five-thirds. 471 00:28:40 --> 00:28:45 Two-thirds and t is two, so that's two-thirds and one 472 00:28:45 --> 00:28:46 make five-thirds. 473 00:28:46 --> 00:28:51 And that's -- sure enough, that's smaller than the exact 474 00:28:51 --> 00:28:52 two. 475 00:28:52 --> 00:28:55 And then final P3, when t is three, 476 00:28:55 --> 00:29:00 oh, what's two-thirds plus three-halves? 477 00:29:00 --> 00:29:04 It's the same as three-halves plus two-thirds. 478 00:29:04 --> 00:29:08 It's -- so maybe four-sixths and nine-sixths, 479 00:29:08 --> 00:29:10 maybe thirteen-sixths. 480 00:29:10 --> 00:29:13 OK, and again, look, oh, look at this, 481 00:29:13 --> 00:29:14 OK. 482 00:29:14 --> 00:29:18 You have to admire the beauty of this answer. 483 00:29:18 --> 00:29:20 What's this first error? 484 00:29:20 --> 00:29:24 So here are the errors. e1, e2 and e3. 485 00:29:24 --> 00:29:27 OK, what was that first error, e1? 486 00:29:27 --> 00:29:32 Well, if we decide the errors counting up, then it's 487 00:29:32 --> 00:29:33 one-sixth. 488 00:29:33 --> 00:29:38 And the last error, thirteen-sixths minus the 489 00:29:38 --> 00:29:41 correct two is one-sixth again. 490 00:29:41 --> 00:29:44 And what's this error in the middle? 491 00:29:44 --> 00:29:49 Let's see, the correct answer was two, two. 492 00:29:49 --> 00:29:56 And we got five-thirds and it's the other direction, 493 00:29:56 --> 00:30:00 minus one-third, minus two-sixths. 494 00:30:00 --> 00:30:03 That's our error vector. 495 00:30:03 --> 00:30:08 In our picture, in our other picture, 496 00:30:08 --> 00:30:09 here it is. 497 00:30:09 --> 00:30:14 We just found P and e. e is this vector, 498 00:30:14 --> 00:30:23 one-sixth, minus two-sixths, one-sixth, and P is this guy. 499 00:30:23 --> 00:30:29 Well, maybe I have the signs of e wrong, I think I have, 500 00:30:29 --> 00:30:30 let me fix it. 501 00:30:30 --> 00:30:36 Because I would like this one-sixth -- I would like this 502 00:30:36 --> 00:30:39 plus the P to give the original b. 503 00:30:39 --> 00:30:42 I want P plus e to match b. 504 00:30:42 --> 00:30:48 So I want minus a sixth, plus seven-sixths to give the 505 00:30:48 --> 00:30:51 correct b equal one. 506 00:30:51 --> 00:30:52 OK. 507 00:30:52 --> 00:30:58.84 Now -- I'm going to take a deep breath here, and ask what do we 508 00:30:58.84 --> 00:31:02.24 know about this error vector e? 509 00:31:02.24 --> 00:31:07 You've seen now this whole problem worked completely 510 00:31:07 --> 00:31:12 through, and I even think the numbers are right. 511 00:31:12 --> 00:31:18 So there's P, so let me -- I'll write -- 512 00:31:18 --> 00:31:23 if I can put it down here, B is P plus e. 513 00:31:23 --> 00:31:27 b I believe was one, two, two. 514 00:31:27 --> 00:31:32.06 The nearest point had seven-sixths, 515 00:31:32.06 --> 00:31:34 what were the others? 516 00:31:34 --> 00:31:38 Five-thirds and thirteen-sixths. 517 00:31:38 --> 00:31:45 And the e vector was minus a sixth, two-sixths, 518 00:31:45 --> 00:31:52 one-third in other words, and minus a sixth. 519 00:31:52 --> 00:31:52 OK. 520 00:31:52 --> 00:31:57 Tell me some stuff about these two vectors. 521 00:31:57 --> 00:32:02 Tell me something about those two vectors, well, 522 00:32:02 --> 00:32:05 they add to b, right, great. 523 00:32:05 --> 00:32:06 OK. 524 00:32:06 --> 00:32:07 What else? 525 00:32:07 --> 00:32:11 What else about those two vectors, the P, 526 00:32:11 --> 00:32:18.4 the projection vector P, and the error vector e. 527 00:32:18.4 --> 00:32:21.78 What else do you know about them? 528 00:32:21.78 --> 00:32:24 They're perpendicular, right. 529 00:32:24 --> 00:32:27 Do we dare verify that? 530 00:32:27 --> 00:32:31 Can you take the dot product of those vectors? 531 00:32:31 --> 00:32:37 I'm like getting like minus seven over thirty-six, 532 00:32:37 --> 00:32:40 can I change that to ten-sixths? 533 00:32:40 --> 00:32:43 Oh, God, come out right here. 534 00:32:43 --> 00:32:50 Minus seven over thirty-six, plus twenty over thirty-six, 535 00:32:50 --> 00:32:54 minus thirteen over thirty-six. 536 00:32:54 --> 00:32:55 Thank you, God. 537 00:32:55 --> 00:32:56 OK. 538 00:32:56 --> 00:33:01 And what else should we know about that vector? 539 00:33:01 --> 00:33:07 Actually we know -- I've got to say we know even a little more. 540 00:33:07 --> 00:33:12 This vector, e, is perpendicular to P, 541 00:33:12 --> 00:33:17 but it's perpendicular to other stuff too. 542 00:33:17 --> 00:33:22 It's perpendicular not just to this guy in the column space, 543 00:33:22 --> 00:33:24 this is in the column space for sure. 544 00:33:24 --> 00:33:27 This is perpendicular to the column space. 545 00:33:27 --> 00:33:31 So like give me another vector it's perpendicular to. 546 00:33:31 --> 00:33:34 Another because it's perpendicular to the whole 547 00:33:34 --> 00:33:37 column space, not just to this -- this 548 00:33:37 --> 00:33:42 particular projection that's -- that is in the column space, 549 00:33:42 --> 00:33:46 but it's perpendicular to other stuff, whatever's in the column 550 00:33:46 --> 00:33:49 space, so tell me another vector in the -- oh, 551 00:33:49 --> 00:33:53 well, I've written down the matrix, so tell me another 552 00:33:53 --> 00:33:55 vector in the column space. 553 00:33:55 --> 00:33:57 Pick a nice one. 554 00:33:57 --> 00:33:58 One, one, one. 555 00:33:58 --> 00:34:00 That's what everybody's thinking. 556 00:34:00 --> 00:34:03 OK, one, one, one is in the column space. 557 00:34:03 --> 00:34:06 And this guy is supposed to be perpendicular to one, 558 00:34:06 --> 00:34:06 one, one. 559 00:34:06 --> 00:34:07 Is it? 560 00:34:07 --> 00:34:07 Sure. 561 00:34:07 --> 00:34:11.37 If I take the dot product with one, one, one I get minus a 562 00:34:11.37 --> 00:34:13 sixth, plus two-sixths, minus a sixth, 563 00:34:13 --> 00:34:15 zero. 564 00:34:15 --> 00:34:19 And it's perpendicular to one, two, three. 565 00:34:19 --> 00:34:23 Because if I take the dot product with one, 566 00:34:23 --> 00:34:28 two, three I get minus one, plus four, minus three, 567 00:34:28 --> 00:34:29 zero again. 568 00:34:29 --> 00:34:34 OK, do you see the -- I hope you see the two pictures. 569 00:34:34 --> 00:34:38 The picture here for vectors and, 570 00:34:38 --> 00:34:44.74 the picture here for the best line, and it's the same picture, 571 00:34:44.74 --> 00:34:50 just -- this one's in the plane and it's showing the line, 572 00:34:50 --> 00:34:56 this one never did show the line, this -- in this picture, 573 00:34:56 --> 00:34:59 C and D never showed up. 574 00:34:59 --> 00:35:03 In this picture, C and D were -- 575 00:35:03 --> 00:35:06.61 you know, they determined that line. 576 00:35:06.61 --> 00:35:09 But the two are exactly the same. 577 00:35:09 --> 00:35:15 C and D is the combination of the two columns that gives P. 578 00:35:15 --> 00:35:15 OK. 579 00:35:15 --> 00:35:18 So that's these squares. 580 00:35:18 --> 00:35:23 And the special but most important example of fitting by 581 00:35:23 --> 00:35:27 straight line, so the homework that's coming 582 00:35:27 --> 00:35:33 then Wednesday asks you to fit by straight lines. 583 00:35:33 --> 00:35:41.2 So you're just going to end up solving the key equation. 584 00:35:41.2 --> 00:35:48 You're going to end up solving that key equation and then P 585 00:35:48 --> 00:35:50 will be Ax hat. 586 00:35:50 --> 00:35:52 That's it. 587 00:35:52 --> 00:35:52 OK. 588 00:35:52 --> 00:35:59 Now, can I put in a little piece of linear algebra that I 589 00:35:59 --> 00:36:04 mentioned earlier, mentioned again, 590 00:36:04 --> 00:36:07 but I never did write? 591 00:36:07 --> 00:36:12 And I've -- I should do it right. 592 00:36:12 --> 00:36:17 It's about this matrix A transpose A. 593 00:36:17 --> 00:36:18 There. 594 00:36:18 --> 00:36:23 I was sure that that matrix would be invertible. 595 00:36:23 --> 00:36:29 And of course I wanted to be sure it was invertible, 596 00:36:29 --> 00:36:38 because I planned to solve this system with with that matrix. 597 00:36:38 --> 00:36:46 So and I announced like before -- as the chapter was just 598 00:36:46 --> 00:36:54 starting, I announced that it would be invertible. 599 00:36:54 --> 00:37:00 But now I -- can I come back to that? 600 00:37:00 --> 00:37:00 OK. 601 00:37:00 --> 00:37:09 So what I said was -- that if A has independent columns, 602 00:37:09 --> 00:37:15 then A transpose A is invertible. 603 00:37:15 --> 00:37:23 And I would like to -- first to repeat that important fact, 604 00:37:23 --> 00:37:30 that that's the requirement that makes everything go here. 605 00:37:30 --> 00:37:37 It's this independent columns of A that guarantees everything 606 00:37:37 --> 00:37:39 goes through. 607 00:37:39 --> 00:37:41 And think why. 608 00:37:41 --> 00:37:46.48 Why does this matrix A transpose A, 609 00:37:46.48 --> 00:37:53 why is it invertible if the columns of A are independent? 610 00:37:53 --> 00:38:00 OK, there's -- so if it wasn't invertible, I'm -- so I want to 611 00:38:00 --> 00:38:02 prove that. 612 00:38:02 --> 00:38:06 If it isn't invertible, then what? 613 00:38:06 --> 00:38:13 I want to reach -- I want to follow that -- follow that line 614 00:38:13 --> 00:38:18.72 -- of thinking and see what I come 615 00:38:18.72 --> 00:38:19 to. 616 00:38:19 --> 00:38:21 Suppose, so proof. 617 00:38:21 --> 00:38:24 Suppose A transpose Ax is zero. 618 00:38:24 --> 00:38:27 I'm trying to prove this. 619 00:38:27 --> 00:38:29 This is now to prove. 620 00:38:29 --> 00:38:36 I don't like hammer away at too many proofs in this course. 621 00:38:36 --> 00:38:42 But this is like the central fact and it brings in all the 622 00:38:42 --> 00:38:45 stuff we know. 623 00:38:45 --> 00:38:46 OK. 624 00:38:46 --> 00:38:48 So I'll start the proof. 625 00:38:48 --> 00:38:51 Suppose A transpose Ax is zero. 626 00:38:51 --> 00:38:56 What -- and I'm aiming to prove A transpose A is invertible. 627 00:38:56 --> 00:38:59 So what do I want to prove now? 628 00:38:59 --> 00:39:02 So I'm aiming to prove this fact. 629 00:39:02 --> 00:39:06 I'll use this, and I'm aiming to prove that 630 00:39:06 --> 00:39:12 this matrix is invertible, OK, so if I suppose A transpose 631 00:39:12 --> 00:39:17 Ax is zero, then what conclusion do I want to reach? 632 00:39:17 --> 00:39:21 I'd like to know that x must be zero. 633 00:39:21 --> 00:39:23 I want to show x must be zero. 634 00:39:23 --> 00:39:28 To show now -- to prove x must be the zero vector. 635 00:39:28 --> 00:39:32.75 Is that right, that's what we worked in the 636 00:39:32.75 --> 00:39:40 previous chapter to understand, that a matrix was invertible 637 00:39:40 --> 00:39:46 when its null space is only the zero vector. 638 00:39:46 --> 00:39:51 So that's what I want to show. 639 00:39:51 --> 00:39:59 How come if A transpose Ax is zero, how come x must be zero? 640 00:39:59 --> 00:40:03 What's going to be the reason? 641 00:40:03 --> 00:40:08 Actually I have two ways to do it. 642 00:40:08 --> 00:40:11 Let me show you one way. 643 00:40:11 --> 00:40:16.67 This is -- here, trick. 644 00:40:16.67 --> 00:40:21 Take the dot product of both sides with x. 645 00:40:21 --> 00:40:25.92 So I'll multiply both sides by x transpose. 646 00:40:25.92 --> 00:40:30 x transpose A transpose Ax equals zero. 647 00:40:30 --> 00:40:33.61 I shouldn't have written trick. 648 00:40:33.61 --> 00:40:38 That makes it sound like just a dumb idea. 649 00:40:38 --> 00:40:41 Brilliant idea, I should have put. 650 00:40:41 --> 00:40:42 OK. 651 00:40:42 --> 00:40:45 I'll just put idea. 652 00:40:45 --> 00:40:46 OK. 653 00:40:46 --> 00:40:53 Now, I got to that equation, x transpose A transpose Ax=0, 654 00:40:53 --> 00:41:01 and I'm hoping you can see the right way to -- to look at that 655 00:41:01 --> 00:41:02 equation. 656 00:41:02 --> 00:41:09 What can I conclude from that equation, that if I have x 657 00:41:09 --> 00:41:16 transpose A -- well, what is x transpose A transpose 658 00:41:16 --> 00:41:18 Ax? 659 00:41:18 --> 00:41:22 Does that -- what it's giving you? 660 00:41:22 --> 00:41:27 It's again going to be putting in parentheses, 661 00:41:27 --> 00:41:32 I'm looking at Ax and what I seeing here? 662 00:41:32 --> 00:41:34 Its transpose. 663 00:41:34 --> 00:41:40 So I'm seeing here this is Ax transpose Ax. 664 00:41:40 --> 00:41:42 Equaling zero. 665 00:41:42 --> 00:41:49 Now if Ax transpose Ax, so like let's call it y or 666 00:41:49 --> 00:41:57 something, if y transpose y is zero, what does that tell me? 667 00:41:57 --> 00:42:02 That the vector has to be zero, right? 668 00:42:02 --> 00:42:10.38 This is the length squared, that's the length of the vector 669 00:42:10.38 --> 00:42:15 Ax squared, that's Ax times Ax. 670 00:42:15 --> 00:42:21 So I conclude that Ax has to be zero. 671 00:42:21 --> 00:42:25 Well, I'm getting somewhere. 672 00:42:25 --> 00:42:33 Now that I know Ax is zero, now I'm going to use my little 673 00:42:33 --> 00:42:34 hypothesis. 674 00:42:34 --> 00:42:43.9 Somewhere every mathematician has to use the hypothesis. 675 00:42:43.9 --> 00:42:44 Right? 676 00:42:44 --> 00:42:49.09 Now, if A has independent columns and we've -- we're at 677 00:42:49.09 --> 00:42:53 the point where Ax is zero, what does that tell us? 678 00:42:53 --> 00:42:58 I could -- I mean that could be like a fill-in question on the 679 00:42:58 --> 00:42:59 final exam. 680 00:42:59 --> 00:43:04 If A has independent columns and if Ax equals zero then what? 681 00:43:04 --> 00:43:07 Please say it. x is zero, right. 682 00:43:07 --> 00:43:12 Which was just what we wanted to prove. 683 00:43:12 --> 00:43:14 That -- do you see why that is? 684 00:43:14 --> 00:43:20 If Ax eq- equals zero, now we're using -- here we used 685 00:43:20 --> 00:43:25.29 this was the square of something, so I'll put in little 686 00:43:25.29 --> 00:43:31 parentheses the observation we made, that was a square which is 687 00:43:31 --> 00:43:35 zero, so the thing has to be zero. 688 00:43:35 --> 00:43:41.09 Now we're using the hypothesis of independent columns at the A 689 00:43:41.09 --> 00:43:43.37 has independent columns. 690 00:43:43.37 --> 00:43:48 If A has independent columns, this is telling me x is in its 691 00:43:48 --> 00:43:54.39 null space, and the only thing in the null space of such a 692 00:43:54.39 --> 00:43:56.86 matrix is the zero vector. 693 00:43:56.86 --> 00:43:57 OK. 694 00:43:57 --> 00:44:02 So that's the argument and you see how it really used our 695 00:44:02 --> 00:44:08 understanding of the -- of the null space. 696 00:44:08 --> 00:44:09 OK. 697 00:44:09 --> 00:44:11.05 That's great. 698 00:44:11.05 --> 00:44:12 All right. 699 00:44:12 --> 00:44:15 So where are we then? 700 00:44:15 --> 00:44:24 That board is like the backup theory that tells me that this 701 00:44:24 --> 00:44:33 matrix had to be invertible because these columns were 702 00:44:33 --> 00:44:35.03 independent. 703 00:44:35.03 --> 00:44:40.29 OK. there's one case of independent 704 00:44:40.29 --> 00:44:45 -- there's one case where the 705 00:44:45 --> 00:44:48 geometry gets even better. 706 00:44:48 --> 00:44:54 When the -- there's one case when columns are sure to be 707 00:44:54 --> 00:44:55 independent. 708 00:44:55 --> 00:45:01 And let me put that -- let me write that down and that'll be 709 00:45:01 --> 00:45:04.79 the subject for next time. 710 00:45:04.79 --> 00:45:12 Columns are sure -- are certainly independent, 711 00:45:12 --> 00:45:20 definitely independent, if they're perpendicular. 712 00:45:20 --> 00:45:30 Oh, I've got to rule out the zero column, let me give them 713 00:45:30 --> 00:45:38 all length one, so they can't be zero if they 714 00:45:38 --> 00:45:45 are perpendicular unit vectors. 715 00:45:45 --> 00:45:49 Like the vectors one, zero, zero, zero, 716 00:45:49 --> 00:45:52 one, zero and zero, zero, one. 717 00:45:52 --> 00:45:57.28 Those vectors are unit vectors, they're perpendicular, 718 00:45:57.28 --> 00:46:00 and they certainly are independent. 719 00:46:00 --> 00:46:04 And what's more, suppose they're -- oh, 720 00:46:04 --> 00:46:08 that's so nice, I mean what is A transpose A 721 00:46:08 --> 00:46:10 for that matrix? 722 00:46:10 --> 00:46:15 For the matrix with these three columns? 723 00:46:15 --> 00:46:17 It's the identity. 724 00:46:17 --> 00:46:21.52 So here's the key to the lecture that's coming. 725 00:46:21.52 --> 00:46:26 If we're dealing with perpendicular unit vectors and 726 00:46:26 --> 00:46:32 the word for that will be -- see I could have said orthogonal, 727 00:46:32 --> 00:46:38 but I said perpendicular -- and this unit vectors gets put in as 728 00:46:38 --> 00:46:40 the word normal. 729 00:46:40 --> 00:46:42 Orthonormal vectors. 730 00:46:42 --> 00:46:46 Those are the best columns you could ask for. 731 00:46:46 --> 00:46:50 Matrices with -- whose columns are orthonormal, 732 00:46:50 --> 00:46:55 they're perpendicular to each other, and they're unit vectors, 733 00:46:55 --> 00:47:00 well, they don't have to be those three, let me do a final 734 00:47:00 --> 00:47:04 example over here, how about one at an angle like 735 00:47:04 --> 00:47:11 that and one at ninety degrees, that vector would be cos theta, 736 00:47:11 --> 00:47:17 sine theta, a unit vector, and this vector would be minus 737 00:47:17 --> 00:47:20 sine theta cos theta. 738 00:47:20 --> 00:47:26 That is our absolute favorite pair of orthonormal vectors. 739 00:47:26 --> 00:47:32.61 They're both unit vectors and they're perpendicular. 740 00:47:32.61 --> 00:47:36 That angle is ninety degrees. 741 00:47:36 --> 00:47:43 So like our job next time is first to see why orthonormal 742 00:47:43 --> 00:47:48 vectors are great, and then to make vectors 743 00:47:48 --> 00:47:52 orthonormal by picking the right basis. 744 00:47:52 --> 00:47:53 OK, see you. 745 00:47:53 --> 00:47:56 Thanks.