Part 6: Finding the Nullspace: Solving Ax = 0 by Elimination

Flash and JavaScript are required for this feature.

Download the video from Internet Archive.

Description

This new video computes the nullspace of any matrix A. The equation Ax = 0 is reduced to row echelon form Rx = 0. Then all solution vectors are combinations of "special solutions". We understand the pieces of R from the factorization A = CR.

Slides Used in this Video: Part 6: Finding the Nullspace: Solving Ax = 0 by Elimination (PDF)

See also Lecture 7 from 18.06 Linear Algebra for more on this topic.

Instructor: Gilbert Strang

[SQUEAKING]

[RUSTLING]

[CLICKING]

GILBERT STRANG: OK, this is about finding the null space of a matrix A-- any matrix, square or rectangular. And what does that mean? That means, well, in algebra, we're solving the equation Ax equals 0.

So A is our matrix, x is a vector that we're looking for, and Ax is a combination of the columns of A. So we're looking for combinations of the columns that give the zero vector-- dependent columns, we'll be saying. So that's the goal.

And the new start for linear algebra that I've suggested solves that problem for small matrices. It kind of just does it for a small matrix. But for any matrix, or a big one, we need a system. So this kind of completes the idea by giving the system, which uses the algorithm of linear algebra, which is elimination. So elimination is going to be the key to solving this problem-- to finding the null space.

So that's my picture. Oh, these are the-- I'll just mention-- the three books which discuss this. The Introduction to Linear Algebra is the main textbook. Then the Learning from Data book-- actually, that's where this new idea got started. And the Linear Algebra for Everyone has got the idea more completely. So I'm sort of speaking about a section out of that third book. OK.

And they all have websites. Math.mit.edu, the addresses are there to see more. Actually, I should say, so these are the key ideas of this lecture. And yesterday, I looked at my lecture from a few years ago-- quite a few years ago-- lecture seven in the OpenCourseWare series for Math 18.06. So that was on the same topic, elimination and Ax equals 0.

So a lot of good stuff is there, but there's a little more to say now. And that's what this is about today. So OK, here are key ideas of the lecture.

So the null space-- so now you see it in writing-- the null space is all the solutions x to Ax equals 0. Remember, x is a vector. And elimination is the key and it keeps the same null space. And the Matlab command is rref of A. That's the command that does elimination.

And we'll see the identity matrix, we'll see a matrix F, and the new idea was to factor the matrix into a column matrix C times a row matrix R. So this is really putting all the ideas together. And we learn that if a matrix is short and wide, then there are certainly-- if we've got lots of columns, then some of them will be dependent and that means that there'll be solutions to Ax equals 0.

So here we go. Oh OK, I'm going to start with an example. And elimination has already done its job. So what elimination did was to get to that-- you see that identity matrix in the first two columns? 1, 0 and 0, 1. So you can't ask for a simpler matrix than that. So once we've got that, we can't mess with it, so 3, 5, 4, 6, we're stuck with-- those are our other columns.

So the first two columns are independent. Columns of the identity matrix are very independent. But then 3, 4 is a combination of those two columns, right? 3, 4 is 3 times the first column plus 4 times the second. So there is an x in the null space. That's one of the vectors we're after. I'll call it a special solution because it just came specially from one column.

And you see it down below as s1. So if we took minus 3 of the first column, minus 4 of the second column, and then plus 1 of the third column, we would have 0. Ax equals 0-- what we're looking for. And the second special solution would come from the last column of the matrix, 5, 6. So again, that's 5 of the 1, 0 column and 6 of the 0, 1 column. And if we put that together into a special solution s2, we want minus 5 of column 1, minus 6 of column 2, plus nothing of column 3, plus column 4, giving 0.

So in this case, elimination has produced a simple matrix R-- simple because it's got the identity there. So can I show you one more example of R before I begin to talk about how do we get to R? OK, so here's another R. A little different, though.

It's different because it's got a row of 0's. Well that won't pose any problem, but we just have to think what to do with it. We always move 0 rows to the bottom. And there it is. And it does have an identity matrix as the first example did, but you see that the 0, 1 is often column 3. The identity is in columns 1 and 3 here and that makes a small change.

But the idea is still the same. Those two columns are the independent ones. They're very simple. The 1, 0, 0 and 0, 1, 0, those are totally different directions. Then the 7, 0, 0 is 7 of the first column. So that spotted a special solution-- an x that I call s1. At the bottom line, you see the minus 7 of the first column plus 1 of the second column produces 0. If you just look at those columns, minus 7 of the first plus 1 of the second, everything cancels.

And the other one is going to come from the 8, 9, 0 column. That'll be 8 of the first column and 9 of the third-- the two bits of the identity. So you see that when we get to this reduced row echelon form-- horrible set of words, but what it means is elimination has made it as simple as it can be. OK, I might have something more to say.

Yeah, so this is summarizing what you've just seen. So we have the simplest case where the identity just sits there at the left or the more general case where the identity is mixed in with the other two columns. We can live with both. And I'm using F for the other columns-- the columns that are not part of the identity. And then it has that extra 0 row. OK.

And now, I want to write it-- a key part of this lecture is to see the result this matrix R-- see it in matrix form instead of a bunch of numbers. Often, in the computing, you're just pages full of numbers and you don't see what's happening. So what we're looking for is the identity matrix and the non-zero matrix F.

And now, in that R0 case, that second example, there's a P. What is that matrix P doing there? Well, it's a permutation matrix-- an exchange matrix. Because the identity in this second example is not in columns 1 and 2. It's in 1 and 3. So that P has to put it there. So that P is-- and because P is on the right, it'll move columns.

And there I wrote down what that P is. That exchanges column 2 and 3 and puts the identity in where we want it. OK. So that's the linear algebra-- the matrix notation part.

Oh, and this is-- here, now you're seeing the new start, this C times R. So that's what I'm suggesting that if you give me a matrix, the first thing I would do would be to find independent columns-- those would go into C. And then the matrix R would-- well, we've seen the matrix R and that would tell me the combinations of those C columns to get A. Yeah, you'll see it.

Well, this box is around all the matrix formulas. So we're sorting every matrix into independent columns followed by dependent columns and then this permutation-- this exchange matrix P-- if they don't really come in that order. If we need a new order. So all this is about A equals CR, the new-- well, not new. Not completely new, I'm sure, but it's the factorization of any matrix.

OK. And now how do we actually do it if the matrix is big and we need the computer to help? Well, we do it by elimination. OK, this is what elimination is about. What are we allowed to do in elimination? We don't want to mess up the equation Ax equals 0. I'm operating on A, but I'm not creating or losing any of those solutions.

So if I subtract one row from another row, or any multiple of one row from another row, that's the main operation of elimination. And obviously, the equations are still true. If I take 3 of equation one away from equation two, I still have a correct equation. And I could multiply a row by 15 or any non-zero number. No change. And I can switch any two rows. That third one of switching rows, exchanging rows, is just to move the 0 row down to the bottom.

This is the example. Can you see those numbers? So I'm giving an example of a matrix that leads to that very same R that we worked out. So you see what I'm doing now. I'm backing up. I know the R I want, but I'm given some matrix A, and it's these elimination steps that will lead me to R. OK.

So you see A over on the left? So it's a full matrix A. No identity there. But I'm allowed to do elimination. I'm allowed to do these three steps when I want to.

So what shall I do? Probably, at this point, you've seen elimination as much as you want to, but allow me to say it once or twice more. So I would take 3 of row 1 away from row 2. That would produce-- that 3 that's in the original matrix would turn into a 0. So when you take 3 of that first row, that would be 3, 6, 33, 51. I subtract it and I get just that 0, 1, 4, 6. So I've got a much better second row.

Now, I'm going to use the second row going upwards. So I'll take 2 of the second row away from the first row. That'll knock out the 2 that I don't want and it'll produce the identity that I do want. So do you see that? If I take 2 of that last row, that 0, 2, 8, 12, when I take 0, 2, 8, 12 away from the top row, I'm left with 1 and 0. And 3 and 5 are the numbers that happen to be left. OK.

So really, what has elimination done? This is an important idea that I think I never caught onto when I was learning linear algebra. Elimination has found the inverse of this 1, 2, 3, 7-- of this 2 by 2 matrix. Because it's started with 1, 2, 3, 7 and it's ended with the identity. So that must have inverted that matrix.

And then it had to apply the inverse to the second half of the matrix and that produced the 3, 4, 5, 6. So you see, this is-- oh, I think there's probably an example that leads to number two. Here, I must have thought ahead. So it says on the screen what I just said, that elimination has inverted the leading matrix. And then it's written out the step H equal WF that connects the original dependent columns to the final dependent columns in F, 3, 4, 5, 6.

OK. I think elimination is just a matter of practice and you probably have done it. And here is an important point about elimination, that you could do things in different orders. I spoke and I did them in a sort of natural order, but you could do it other ways. But you wouldn't get a different outcome. You'd get the same matrix R.

Well, because the equations are still true at every point. The x's, the null space of a matrix, is not changing by my elimination steps. Yeah. So let's just repeat this.

When you finally get to R, when you've done the elimination and you get to R with its identity matrix, that identity matrix in R tells you which columns at the start were independent. So you need to have a way to do that and now we have a way. In the other opening talks in this series-- in this series about the new start-- the matrices were nice and small so I didn't need much of a system. I could practically wing it.

But now, we need a system. And of course, if we want to use the Matlab or other code, it has to have a system. So anyway, we now have a system of row elimination steps and we know what it ends with. It ends with an identity matrix of the right size, a matrix F-- that's the other columns-- and then possibly a permutation to tell us the order. That's elimination.

OK and here's the second example. So we're moving along. You've got to pay attention because this is all there is is two examples and then put the ideas together. So here is an example of a matrix A with numbers as big as 97. I don't know how I got there. OK.

So there are no 0's in the original matrix A but elimination aims to get 0's. Oh, here. It got 0's very quickly because the second row of that matrix, 2, 14, 6, 70, that's just twice the first row. So when I subtract 2 times the first row from the second row, I get that 0 row. So I'm now one step along with a row of 0's. That's great.

And the third row improved too because I took 2 of the first row away from the third row and that gave me a 0, 0 at the start and then a 3, 27. So are we OK? We're at the second matrix now. It's not finished. It's not in its R form, in its echelon form, but it's a lot closer. OK.

So what do we still have to do? Well I look at that 3, and I'm looking to get the identity matrix, so I'll divide that-- sooner or later, I'm going to divide that last row by 3 and get 0, 0, 1, 9. But another thing I want to do is clean up the first row. So I subtract that last row from the first row. You see that? See, I'm much closer to the identity.

So I'm now moved on to the third matrix. And if you've done a lot of these, you know what I'm doing. I'm working a column at a time. And notice, column 2, I can't do anything with. Column 2 is a dependent column. It's seven times column 1. I can't improve column 2.

But I can improve column 3 and that's what we do. Divide that third, that 0, 0, 3, 27, by 3 and exchange rows. Then you see that the result is the R0. So it's the R0 with the 0 row. Yeah.

So just to repeat, this is a matrix A that leads through the steps of elimination, which we're remembering to R0. So we're really finished with this matrix A because this shows how to get it to R0, and then we've already seen with R0, we've figured out which were the special solutions-- the vectors in the null space. So we're good. We really have done the job.

And it remains to just see a little bit, what did we do and what's going on with these other two columns? The F columns that are not the identity. And you see, what we've done is-- well, we started with columns 1 and 3. Yeah, so that's our matrix C. Those are the two independent columns.

Well how do I know? Where did C come from? Because that's part of this search, where are the independent columns. They're the ones that end up with the identity because the identity is the way to go to the right independent columns. So they're columns 1 and 3 of the original matrix.

So you see C there? 1, 2, 2 and 3, 6, 9. Then F is the part of R from the dependent column. So that's the 7, 8, 0, 9. And then I'm seeing the dependent columns. Yeah. You have to do this a few times, but all the ideas now are-- and there I've put it together.

So the idea is, with a small matrix, like only three columns and maybe only two rows, we could find C and R almost by sight. But now, even where we're up to four columns and three rows, we need a method. And that's the point of this talk and it was also the point of lecture seven in the original 18.06-- Math 18.06 on OpenCourseWare. OK, so we've found C and R. Good. Good. I think it's just congratulating ourselves from here in.

These are remembering the special solutions. Oh, and then why not-- you remember those? That second column was 7 of the first column, so that gave that special solution, that combination, to give 0. And then the other combination gave the 0 vector. So we've got two special solutions.

Oh, and here is the general principle. Yeah. Yeah. It's very simple. After you've got this matrix down to identity and F, then the solutions x should have a minus FI. You remember the minus signs that we saw? So all I'm saying here on the first line is that the matrix IF times minus FI gives the 0 matrix. Of course it does.

And then the second line is the little bit of special trick that you have to do if there's a P, a permutation, an exchange involved here. When the identity matrix is not sitting where you want it, you have to have a P to put it there. Then P transpose has to come into the solutions. Yeah. Yeah, because P times P transpose. So remember about permutation matrices.

Linear algebra is about learning different special character of different matrices. And permutations just take the identity matrix and mess around with the rows or mess around with the columns, same thing. And then P transpose unmesses it. Brings it back to the identity. So P times P transpose is the identity.

OK, so we're just about there. Oh, then this is the little point I made-- well, it's an important point. Suppose I have five columns and only three rows. That means I've got five unknowns and only three equations. So there are going to be solutions to that. There are going to be solutions other than the 0 solution. The 0 combination will surely give 0, but if I have five unknowns, five columns in there, and I have only three equations, three rows-- so five greater than three, n greater than m-- then there will be some columns in F. There'll be some non-zero solutions. And these examples show.

So take that matrix M at the end there. You see there's a matrix with four columns and only two rows. So I've got four vectors in the plane. If I have four vectors in the plane, there are combinations that give 0. They can complete triangles. So in that case, rank would be 2.

So we're really seeing this part of linear algebra, to simplify the matrix by elimination so that all the main facts are clear. Yeah. And oh, if you really want to see it in shorthand-- I'm not necessarily pushing this last idea, but if you want to see it in a shorthand, we could think of the matrix as having just four blocks. So it's a 2 by 2 matrix, but unfortunately, each of those guys is a block.

So W is the one in the corner that we talked about earlier that gets inverted. So if we look at the last line, that W in the corner, that ended up as the identity. The second block row, the J, K, ended up as 0's. And the one remaining guy, it tells us the combinations that we are looking for-- is this W inverse H.

So let me look at that equation in the box W, H, J, K, that's the matrix I'm starting with. I'm going to invert W and I know that J, K is some combination of W, H. So when I invert W, I get-- the first elimination produces the first row, starting with the identity. And when I apply that to the second, the J, K row, those go to 0.

So that's what elimination is doing in real shorthand. It's taking a 2 by 2 block matrix to that matrix R at the end, with a 0 block row and an identity block in the upper corner. So that is elimination and the solution of-- and finding the null space. So really, this completes the job of the first topic in linear algebra, is identifying-- understanding these four fundamental subspaces.

The null space being one of them. The column space being another. We understand now which columns are independent. And the row space being another. And so we're really understanding the idea of these four fundamental subspaces that go into the big picture of linear algebra. So this completes the first major stage of a linear algebra course.

And then what's to follow will come. Eigenvalues, singular values, applications of all kinds. Least squares. Good. Good. Good math. And thank you very much. So that's my summary of finding the null space. Good.