The following content is provided
by MIT OpenCourseWare under a Creative Commons license.
Additional information about our license
and MIT OpenCourseWare in general,
is available at ocw.mit.edu.
PROFESSOR: For me, this is the third and last major topic
of the course.
The first one was initial value problems -- stability,
accuracy.
Topic two was solving large linear systems
by iterative methods and also by direct methods
like re-ordering the equations.
Now, topic three is a whole world of optimization.
In reality it means you're minimizing or possibly
maximizing some expression.
That expression could be a function of several variables.
We could be in the discrete case, so we've got --
maybe I just emphasize that we have discrete optimization.
So that's in R^n, discrete in n dimensions,
and that will include some famous areas as single,
as small subsets, really, of the big picture.
For example, one subset would be linear programming.
So that's a very special but important case of problems
where the cost is linear, the constraints
are linear, and has its own special methods.
So I think that's worth considering on its own
at a later point in a lecture.
Another bigger picture -- for us, actually,
bigger will be quadratic programming where the quantity
that's being minimized is a quadratic function.
Now what's good about a quadratic function?
Its derivatives are linear.
So that leads us to linear equations,
but always with constraints.
We don't have a free choice of any vector in R^n.
We have constraints on the vectors, they have to --
maybe they solve a linear system of their own.
We might be in 100 dimensions and we
might have 10 linear equations as the unknowns have to solve.
So in some way we're really in 90 dimensions, but it might --
you know, how should we treat those constraints?
Well, you know that Lagrange multipliers play a role.
So there's a big area, very big area, of discrete optimization.
Then, also, there is the continuous problems
where the unknown is a function, is a function u of x I'll say,
or u of x and y.
It's a function.
That's why I refer to that area as continuous optimization.
Well, first, you always want to get
an equation, which is in some way going to be
derivative equals zero, right.
When we learn about minimization in elementary calculus,
somewhere along the line is going
to be an equation that has something
like derivative equals zero.
But, of course, we have to account for the constraints.
We have to ask, what is -- the derivative of what when
our unknown is a function.
I'm just going to write down a topic within mathematics
that this is often expressed as.
Calculus -- that would be derivative.
But in the case for functions it's
often called the calculus of variations.
So that's just so you see that word.
There are books with that title.
The idea of what is that derivative when our unknown is
a function and the objective that we're trying to minimize
is the integral of the function and its derivatives --
all sorts of possibilities there.
So that that's a very quick overview of a field
that we'll soon know a lot about.
I was trying to think where to start.
I think maybe it better be discrete.
I want to get to the system of equations
that you constantly see.
So let me use, as an example the, most basic problem which
comes for -- maybe I'll start over here --
the problem of least squares, which I'll express this way.
I'm given a matrix A, and a right-hand side b,
and I want to minimize the length squared of A*u minus b.
So that would be a first problem to which
we could apply calculus, because it's a straight minimization
and I haven't got any constraints in there yet.
We could also apply linear algebra,
and actually linear algebra's going
to throw a little extra light.
So, just what am I thinking of here?
I'm thinking of A as being m by n, with m larger than n.
If A is a square matrix, then this problem is the same
as solving A*u equal b.
And, of course, we'll always reduce to that case
if m equals n.
But to focus on the problems that I'm really thinking about,
I'm thinking about the case where this is the --
n is the number of unknowns.
It's the size of u.
m, the larger number, is the number of measurements,
the number of the data, so it's the number of equations,
and it's the size of b.
So we have more equations than unknowns.
You've met least squares before.
I hope that maybe even in these few minutes, a little new light
will be shed on least squares.
So here's our problem, and calculus could lead us
to the equation for the best u.
So, u stands for u_1, u_2, up to u_n.
There are n components in that vector, u.
Maybe you know the equation -- I guess I hope you know
the equation, because it's such a key to so many applications.
If I just write it, it will sound as if problem over.
Let me write it though.
So the key equation -- and then this comes up in statistics,
for example.
Well, that's one of 100 examples.
But in statistics, this is the topic of linear regression
in statistics.
Let me write down -- they gave the name normal equation
to the equation that gives u directly.
Do you remember what it is?
The normal equation?
The equation for the minimizing u, which
we could find by calculus?
It involves the key matrix A transpose A. Let me call u hat
the minimizer, the winner in this competition.
The right-hand side of the equation is A transpose b.
So I won't directly go back to derive it,
though probably I'm going to end up deriving it,
because you can't help approaching that equation
from one side or another.
As I say, one way to approach it would just
be to write out what that sum of squares is, take its derivative
and you would get linear equation.
So again, u hat stands for the u that gives the minimum.
This A transpose A, of course -- so I'm putting in a little bit
of linear algebra.
This matrix A transpose A is obviously symmetric.
Its important property, beyond the symmetry,
is that it's positive definite.
Well, I have to say positive definite --
there's always a proviso, of course.
I haven't eliminated the degenerate case yet.
A has m, many, many rows, a smaller number of columns,
and let's assume that those columns
are linearly independent so that we really do have n unknowns.
If those columns were, say if all the columns were the same,
then A*u would just be multiplying that same column
and there would really be only one unknown.
So I'm going to say that A has rank n, by which I
mean n independent columns.
In that case, that's what guarantees
that this is positive definite.
Let me try to draw an arrow there --
this is the same statement.
If I say about A that the columns are independent,
then I'm saying about A transpose A
that it is positive definite.
That means all its eigenvalues are positive;
it's invertible, certainly; all its pivots are positive.
It's the great class of matrices.
But I don't really want to start with that equation.
Here's my point.
Optimization -- a key word that I better get on the board,
maybe up here, to show that it's really important,
is plus the idea of duality.
The effect of duality, if I just give a first mention
to that word, is that very often optimization problems,
there are really two problems.
Two problems that don't look identical,
but in some important way they both,
each problem is a statement of the task ahead of us.
What are the two problems, the two dual problems
in this basic example of least squares.
All right, here's a good picture.
Here's a good picture.
Let me put it on this board so I can recover it.
So, minimize A*u minus b.
So I think of the vector b as being --
where it's in m dimensions.
So it's a picture -- I'm in m dimensions here.
Now, what about A*u?
A*u -- where will A*u go in this picture?
So, A*u is -- all the candidates A*u,
multiply A by any vector u, so that means A*u is a combination
of the columns of A, the possible vectors A*u lie
in a subspace.
So this is the subspace of all possible vectors A*u.
And it's only n-dimensional.
This is an n-dimensional subspace because I have only n
parameters in u, only n columns in A.
So the set of all A*u's I think of as a --
you could say a plane, an n-dimensional plane within
the bigger space R^m.
Another name for that subspace, that plane, is the --
in 18.06 I would call it the column space of A,
or the range of A is another expression that you--.
All the possible A*u's and here's b,
which isn't one of the possible A*u's.
So where is u hat?
Where is the best A*u -- the best A*u now?
The one that's closest to b is --
now comes another central word in this subject.
If I draw it, I'm going to draw it here.
That will be my best A*u, which I'm calling A u hat.
That's the -- if the picture seems reasonable to your eye,
this is the vector that's in the plane closest to b.
What's the geometry here?
See, that's what I wanted to see --
a little geometry and a little algebra, not just calculus.
So the geometry is that this vector b,
what's the connection between b and that vector --
that's the closest vector, right?
We're minimizing the distance.
This distance here, I might call that the error vector e.
This is as small as possible.
That's being minimized.
That's the difference between -- this is Pythagoras here.
Of course, when I say it's Pythagoras,
I'm already saying the most important point,
that this a right angle here.
That's a right angle.
The closest A*u, which is A u hat, which is this vector,
the way geometrically we know it's closest is that the line
from b to the plane, that's where the line from b,
perpendicular to the plane.
This line, this error vector e is perpendicular to the plane.
There's a good word that everybody uses for this vector.
Take a vector b that's not on a plane, what's
the word to look for the nearest vector in the plane?
AUDIENCE: Projection.
PROFESSOR: Projection.
So this is the projection.
Orthogonal projection, if I wanted
to really emphasize the fact that that's a right angle.
So that would give me a geometric way
to see the least squares problem.
Now comes the point to see the dual problem.
The dual problem will be, here, if I draw the perpendicular
subspace.
So that's a subspace of what dimension?
This contains all the vectors perpendicular to the plane.
So I have it as -- if m is 3, so we're in three dimensions,
and our plane is an ordinary two-dimensional plane,
then the dimension is one -- that's the perpendicular line.
But thinking bigger, if we're in m dimensions and this plane is
n-dimensional, than this is going to have --
the true dimension of this is m minus n,
which could be pretty substantial.
But anyway, that's the perpendicular subspace.
If this is the column space of A,
I can figure out what vectors are
perpendicular to the columns of A. That's really what I mean.
This contains -- I've drawn it as a line,
but I've written up there its dimension so that you see --
I just don't know how to draw, like, a bigger subspace.
Yet you would have to see that all the vectors in it
were perpendicular to all of these vectors.
Do you see what I'm saying?
If we were stuck in thinking in three dimensions,
if I make this a plane I can't make that a plane.
If I make m equal 3, and I make n equal to two,
I'm only got a line left to be perpendicular.
But in higher dimensions there are lots of dimensions left.
So what's my dual problem?
My dual problem is find the vector e in this plane
closest to b.
In other words, by the same reasoning,
what I'm saying is take the vector b,
project it over to this plane, project it orthogonally --
that same right angle is going to be there.
This plane -- I haven't said what's in this --
I've said what's in this plane but I haven't written it yet.
But you could tell me already.
What is this?
What's that vector?
One answer would be, it's the projection
of b onto this perpendicular.
So you see we're really taking the vector b
and we're separating it into two components.
One in the column space, the other perpendicular
to the column space.
So just tell me what that vector is.
It is e.
Same guy.
In other words, e is the solution to the dual problem --
maybe I call this projection p for the best vector
in the plane. e is the best vector in this subspace,
and they add up to--.
So, we're really taking the vector b,
and we're splitting it into a part
p in this space, and a part e in the perpendicular space.
If I just write down the equations for that,
I'll see what's cooking.
Well, I guess what I have to do is remember what
are the equations to be in this subspace,
to be perpendicular to the column.
So I can't go further without remembering
what's in that subspace.
So everything in that subspace is
perpendicular to the columns of A.
Let me just write down what that means.
Let me use the letter maybe y for the vectors
in that subspace, and e for the winning vector, the projection.
So y will be the vectors in that subspace.
So those vectors are perpendicular --
so this is the subspace of y, all the y's in here.
Now, what's the condition?
So y -- that y in this perpendicular subspace.
What do I mean?
I mean that y is perpendicular to the columns of A.
How shall I write that?
Perpendicular means inner product zero.
So I want to change the columns into rows,
and take the inner product with y, and get zeros.
Zero, zero, zero.
So, this is column 1 transposed, to be a row.
I'm trying to express this requirement
in terms of the matrix.
So to be perpendicular to the first column,
I know that means that the inner product of the first column
with y should be zero.
The inner product with the second column --
the second column with y should be zero.
The n-th column, its inner product with y should be zero.
So what matrix have I got here?
What's the condition on y's, simple and beautiful?
What matrix is that?
It's A transpose.
So that perpendicular thing -- this is completely expressed
by the equation A transpose y equals zero.
That tells me the y's, and, of course,
e is going to be one of the y's.
It's going to be, I could say y hat,
but I've already named it e.
It's the particular one that's closest to b --
the y's are everybody all along here --
this is the null space of A transpose.
So in words, I would call it that perpendicular thing is
the null space of A transpose.
So when you did linear algebra, you'll remember that.
That the null space of A transpose -- let me write it --
is perpendicular to the column space of A.
The fundamentals theorem of linear algebra right there.
Now we're just using it again to see what
are the two dual problems here.
So the primal problem, the one that we stated first,
was this one.
So I'll call this the primal -- P for primal.
What is the dual problem?
The dual problem is the problem about the y's.
Not about the u's at all.
That's the beauty of this duality.
One problem is about u's, and it's
a problem that ends up projecting on the column space.
The second problem is about y's, it's
the problem that ends up projecting
onto this perpendicular space, and it was a projection.
So the dual problem is just minimize the distance from b
to y, but with the constraint, with --
and now I get to use that word constraint --
with A transpose y equals zero.
So there is the other problem.
So I hope your eye can travel between the -- well,
let me write underneath it the primal again.
Minimize A*u minus b square.
This is the one whose solution is e,
and this is the one whose solution is, well, u,
and the projection -- u hat, sorry,
and the projection p is A u hat, and I guess what I'm trying
to say is that somehow there's a very important connection
between the two problems.
First of all, the two problems use the same data.
They use the same vector b, they use the same matrix A. Notice
that in one problem it's A, and in the other problem
it's the transpose appears.
That's very common -- we'll see that always.
But there's something a little different
about the two problems.
This problem was unconstrained, any u was allowed.
This problem was constrained, only a subset of y's, only
that subspace of y's was allowed.
This is a problem with n unknowns.
This is a problem with m minus n unknowns.
m minus n unknown variables, once we've
accounted for the constraints.
This is one thing I'm thinking about.
Often, the problem will come with a constraint.
Maybe I'll do a physical example right away.
The problem comes to us with a constraint.
In other words, suppose you were given this problem.
How would you deal with it?
That's like the first question in optimization,
or one of the central questions.
How do you deal with a constraint?
If we minimize this, of course, the minimum
would be when y equaled b.
But that's failing to take into account the constraint on y.
So how do you take constraints into account
and end up with an equation?
We can see, in this picture, that somehow or other we
ended up with this normal equation,
but actually I would rather end up with a primal dual equation.
I'd like to end up with an equation for the best
u and the best y.
So what will that be?
So I need now two equations that will connect the best
u and the best y, and probably this is going to be the key.
This b is A u hat, right.
So this will be one of my equations,
and this will be the other.
Let me see if I -- well, OK.
I don't know what to do now.
Here I've called it y.
Over here it's e.
I've got myself in a corner.
Maybe I should call e y hat, would you like that?
We have in mind that it's e, the error in the primal problem.
But just to make the notation for the two
problems consistent, let me call the winner here y hat,
the winner here u hat.
What's the relation?
So let me just -- here I'll write down the relation between
the two.
Well, it's over there.
Let's see, is that right?
Yes?
y hat plus A u hat is b, and A transpose y hat is zero.
That's it.
Here we have -- that's the pair of equations that solves,
that connects the primal and the dual, solves them both,
solves each one, and is really, it's a system --
you could say it's a block equation.
The block matrix being identity A, A transpose, zero;
the unknown being the y and the u.
The right-hand side being the data, which in this case
was the b.
I guess what I want to do is emphasize,
in what's coming for the month of April,
the importance of this class of problems.
It's dealing with two -- it's dealing with the primal
and the dual at the same time.
It's important for so many reasons I can't say them
all on the first day.
That would be a mistake, to try to say everything
the first day.
But let me just say something -- that linear programming,
which is just one example, and it doesn't fit this because it
has inequality constraints.
But you maybe know that the number one
method to solve linear program is called the simplex method.
Well, it was the number one method for many years.
For many problems it's still the right way to do it.
But a new method called the primal-dual --
at least that's part of its name, primal-dual.
It is essentially solving the primal and the dual problems
at once, and there are inequality constraints,
of course.
I'm going to stop there with linear programming
and give it its turn later.
In this perfect example here, we have only equations.
How many do we have?
We have m plus n equations, because y is m unknowns,
u is n unknowns.
I have altogether m plus n equations --
m y's, and n u's, and they come together.
Now, could you, just to connect back
with what we absolutely know, that it's a normal equation,
where is this normal equation coming from?
So here's the normal equation.
We know that that's gotta come, right, out of the thing.
How does it come?
Suppose I have a block system, two by two.
How do I solve it?
Well actually, that's, in a way, the big question.
But one way to solve it, the natural way to solve it,
would be elimination.
Multiply this first row by a suitable matrix.
Subtract from the second row to produce a zero there.
In other words, eliminate y and get an equation for u hat.
So what do I do?
How do I do it?
I multiply -- would you rather look at equations or matrices?
I've tried to keep the two absolutely together.
Let me look at the equation.
What shall I multiply that equation by and subtract from
this -- I just want to eliminate.
I want to get y hat out of there and leave just an equation
for u hat that we will totally recognize.
So what do I do?
I multiplying that first equation by?
A transpose, thanks.
So I multiply this first equation by A transpose.
Let me just do it this way.
Now what?
A transpose y is zero.
Now I use the second -- well, this is one way to do it.
A transpose y is zero.
What am I left with?
The normal equation.
Well, it shouldn't be a surprise.
We had to end up with the normal equation.
Maybe you would rather -- and actually,
what I intended was to make it sound more like Gaussian
elimination.
Multiply this row by A transpose,
subtract from this row.
I still have identity A up here -- when I do that subtraction,
I get the zero, that was the point.
Here I get A transpose A subtracted from zero,
it's minus A transpose A, y hat, u hat.
Of course, I had to do the same thing to the right-hand side,
and when I subtracted this the b was still there,
but it was minus A transpose b.
So, now I've got in the second equation --
the second equation only involves u hat, and, of course,
when I changed the signs, it's our friend --
A transpose A u hat equal A transpose b.
So this is -- maybe you would say the natural way to solve
this type of system, but I want to emphasize -- throw it away.
I really want to emphasize the importance of these --
let me clean it back up again to what it was --
of these block systems.
Now, they need a name.
We have to give some name to this type of two-field problem.
I guess then, in the next month, I'm
going to find examples of it everywhere.
So here I've found the first example of it
in ordinary old-fashioned least square.
So what am I going to call this?
I'll call it -- I'll give it a couple of names.
Saddle point equation, saddle point system maybe
I should say.
I have to explain, why do I call it saddle point system.
In optimization, I could call it the optimality equation --
just meaning it's the equations for the winners.
In the world of optimization, the names of Kuhn and Tucker
are associated with these equations --
the Kuhn-Tucker equations, and there are other names
we'll see.
But let me just say for a moment why saddle point.
Why do I think of this as a saddle point problem?
See, the point about A transpose A
was that it was positive definite.
This is A transpose A.
Now what's the corresponding issue for this matrix?
So this is my matrix that I'm constantly gonna look at.
Matrices of that form are going to show up all the time.
I've said probably in 18.085 where these appear
but then we don't do much with them.
Now we're ready to do something.
I didn't appreciate their importance
until I realized, in going to lectures
on applied mathematics, that if I waited a little
while that matrix would appear.
That block matrix.
It just shows up in all these applications.
One of our issues will be how to solve it.
Another issue that comes first is what's
the general form of this?
Can I jump to that issue?
Just so we see something more than this single problem here.
Let me put in the matrix as it comes in applications.
Some matrix A, rectangular, right?
Its transpose.
A zero.
Often a zero.
But what's up here is not always the identity.
I want to allow something more general.
I want to allow, for example, weighted least squares.
So weighted least squares -- if you've met least squares,
it's very important to meet its extension to weighted least
squares.
When the equations A*u equal b are not given the same weight,
there's a weighting matrix, often it's a covariance matrix.
I'm going to call the matrix that goes in here C inverse.
So this will be then an important class of application.
This is pretty important already when the identity is there,
but many, many applications produce some other matrix that
is usually -- it very, very often is symmetric positive
definite in that corner, like the identity is.
But key point, which I have to make today.
Is the whole matrix -- either this one or this one.
It is symmetric, right?
That matrix is symmetric.
Is it or isn't it positive definite?
If I do elimination do I get all positive pivots?
It's a matrix of size m plus n.
So I'm asking, are all its eigenvalues positive,
but I don't want to really compute eigenvalues.
Also, in a lot of cases I would.
Finding the eigenvalues of this matrix
would lead me to the singular value decomposition,
absolutely crucial topic in linear algebra that we'll see.
But let me just take it as a linear system.
If I do elimination, what are the first m pivots?
Let me not be abstract here.
Let me be quite concrete.
Let me put the identity here.
Let me put some matrix -- oh, I want the matrix here to be --
I better put a bigger identity just so we see the picture.
Here I'm going to put to an A transpose, which might be 2, 3,
4; 5, 6, 7 -- when I write numbers like that
you'll realize that I've just pick them out of the hat.
Here is the transpose, 2, 3, 4; 5, 6, 7.
Here's the zero block.
I'd like to know about that matrix.
It's symmetric.
And it's full rank.
It's invertible.
How do I know that?
It's the invertibility -- of course,
the identity part is great.
I guess I see that this is invertible because I've ended
up with A transpose A, and here my A has rank two --
those two columns are independent.
They're not in the same direction -- [2, 3,
4] is not a multiple of [5, 6, 7].
That's an invertible matrix, and this process finds the inverse.
Elimination finds the inverse.
It's the pivots I want to ask you about.
What are the pivots in this matrix?
So what do I do?
The first pivot is a 1 -- I use it to clean out that column.
The second pivot is a 1 -- I use it to clean out that column.
The third pivot is a 1 -- I use it to clean out that column.
What's the next pivot?
What do I have -- of course, now some stuff has filled in here.
What is actually filled in there?
So this is the identity, if I can write it fast.
This guy is now the zero.
This guy didn't move.
What matrix filled in here?
Well, just what I was doing there.
Elimination is exactly what I'm repeating with numbers,
what I did there with letters.
What's in here is minus A transpose A.
I could figure out what that -- I could do --
if I was fast enough, I could do 2, 3, 4; 5, 6, 7 times 2, 3, 4;
5, 6, 7 and I'd get this little two by two matrix that sits
there.
With a minus sign, and that's the point.
That the pivots -- of course, what's this first number going
to be?
4 plus 9 plus 16.
29 is that number.
So a minus 29 sits right there.
That's the next pivot.
The next pivot is a negative number, minus 29,
and the fifth pivot is negative.
So what I'm seeing is a matrix --
this matrix has three positive pivots,
and two negative pivots.
I sort of say that was a saddle point.
Positive pivots describe for me a surface that's going upwards.
This surface is going upward.
I'm a surface in five dimensions here.
It's going upwards in three directions,
but it's going downwards in two.
The point at the heart of it, the saddle point,
is the solution to our system, is the y hat, u hat.
Well, one conclusion is that I wouldn't
be able to use conjugate gradient methods,
for example, which we've just learned how powerful they
are, on the big matrix because it's not positive definite.
It's symmetric.
I could use some other available methods.
I couldn't use conjugate gradient.
So if I wanted to use conjugate gradient,
I better do this reduction to the definite system.
That's longer than I intended to spend on the simple example.
But if you see that example, then we'll
be ready to move it to the wide variety of applications.
So let me just note that one section, already up on the web,
called saddle point systems, solves differential equations
that are of this kind.
So we'll come to that, and we'll come to matrix problems too.
It's a very, very central question,
how to solve linear systems with matrices of that form.
In fact, I guess I could say it's
almost the fundamental problem of numerical linear algebra,
is to solve systems that fall into that saddle point
description.
I'll try to justify that by the importance
I'm assigning to this problem in the next weeks.
OK.
Thanks for today and I'll turn off volume.