The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
SRINIVAS DEVADAS: Good morning, everyone.
So we have a singleton lecture today on linear programming,
which is general purpose optimization technique that you
can use to solve a whole bunch of problems,
including ones that we've seen in 6.046 and previously in 006.
And most recently, we looked at max flow.
We wouldn't have had to go through all
of that pain we went through to derive a max flow algorithm
if you had a linear programming package handy
and all you wanted to do was find the optimum solution.
You could have just run the linear program
with an appropriate input, of course, that is
derived from the flow network.
And you'd get your optimal solution.
And we'll spend a couple of minutes on that
as we look at the power of linear programming in today's
lecture.
But it's not just max flow.
You could do shortest paths.
You could do multi commodity max flow,
which is more complicated than max flow and a variety
of other problems.
So that's that good news.
The bad news is that the algorithms
for linear programming are a heck of a lot more complicated
than max flow.
And you can imagine that that would be the case, because it's
a more general purpose and more powerful technique.
The history really is that it was an open problem.
Up until 1979, people did not know if linear programming was
polynomial-time solvable until Khachiyan came up
with this ellipsoid method, and then there's
been progress sense.
But the algorithm we're going to describe today and execute
on a example input is a simplex algorithm--
the simplex algorithm-- that runs in worst case
exponential time.
But it's a very efficient in practice.
And it's held its ground, even with the advent
of more efficient, from a theoretical sense,
polynomial-time algorithms, namely the ellipsoid method,
which actually is not that efficient in practice
and new interior point methods.
So a little bit of context, let's
just dive into an example of optimization
in the context of politics and see
how you could formulate this particular problem
as a linear program.
So how does politics work?
You buy elections, right?
So you don't want to spend a lot of money,
so you want to minimize the amount of money that
is required to buy an election.
And the way you buy an election is, well, campaign,
but you advertise.
And it doesn't matter facts that are relevant.
As long as you get to the right demographic with the right
message, let's assume that you're
going to win the election.
So that's our mathematical abstraction pf campaigning
and politics all in 30 seconds.
So how to campaign to win an election.
And as I said, we're going to advertise.
But you do have a little bit of work to do here.
That's why you need your campaign manager.
And this manager is going to estimate votes
obtained per dollar spent.
But that dollar is spent advertising in support
of a particular case, or a particular issue.
And contradictions are allowed.
As long as you're sending different messages
to different demographics, you're all good, right?
You're assuming that people don't
watch more than one channel.
So you're a Fox News guy or you're a MSNB guy.
You don't do both.
So now, we get at this estimate.
And it turns into a table.
And so you have your policy, and you've got your demographic.
So you got urban, now think Detroit, suburban,
I guess, you could Lexington where you live,
and rural-- I really don't have any idea what that means.
But presumably, there's places.
And here's our policy.
You want to build roads-- kind of boring,
but some people are interested in roads-- gun control,
of very sensitive, farm subsidies--
you know who's interested in that-- and gasoline tax--
kind of more, this hits your pocket,
so more or less everybody's interested in that.
So you tell the urban guys you want to build roads,
and they don't like you.
So you get a minus 2 there.
So this can go, you advertise, and it hurts you.
You lose votes.
Tell the suburban people-- well, typically,
it's a situation where you have these nice cars,
and you don't like potholes, so you like roads
if you live in suburbia.
Rural people have 4-by-4's.
They don't particularly care.
They don't care as much.
Gun control-- well, you can imagine that urban people
like that.
Suburban people are, hm, OK, meh.
And the rural people hate it, right?
You do not want to advertise on gun control in the rural areas.
Farm subsidies-- so like, don't want to deal with that.
What is a farm?
And the rural people love it, right?
And then gasoline tax, well, urban people are commuting.
And, well, they typically don't have a lot of money.
So there you go.
And those are the numbers.
I'm not going to justify every number here.
But you could put whatever numbers you what, I mean.
So let's move on.
This is just a table.
It could have positive numbers.
It could have negative numbers.
And you still want to win this election.
Regardless of how crazy the demographics
are, how crazy your electorate is,
you want to win the election.
So as long as you have a great campaign
manager, who can get you this table,
it's all mathematical here on out.
You've just got to figure out how
you're going to win a majority.
And you could argue that all you want is to win the election.
We're going to do something slightly different, which
is something that's obviously going to guarantee victory.
But you want to win a majority in every demographic.
As the tables may be off by little, you want to be careful.
So the last thing, of course, in order
to estimate how much money you need is the population here.
So that's votes obtained per dollar spent.
So you're getting-- it's $10 a vote, it's $5 a vote,
et cetera.
And so we need to translate that to votes,
because that's the dollars.
And you got your population here corresponding
to each of these areas.
And that's what you got here.
Majority, we'll assume you win in the case of ties,
just to keep these numbers easy, so that's just divided by 2.
So that's what you got so far.
And you want to win by spending the minimum amount of money.
So that's our optimization problem.
So we can take this, and we can convert it
to a set of linear equations.
And that's going to create our linear program--
our first linear program.
And this is our algebraic setup.
And so let-- we need some variables here,
so let x1, x2, x3, x4 denote the dollars spent per issue.
So you got those four issues up there.
So let me write that out.
It's important to make sure you know
what I'm talking about with respect to a particular issue.
So those are our four issues.
And those are our four variables.
So this linear program has four variables.
You're trying to discover the values of these variables
to optimize, minimize your cost function.
The second thing that a linear program has--
and pretty much the only other thing a linear program has--
are constraints.
And these constraints are also linear.
It gets much more complicated if you have quadratic constraints,
and we won't go there.
These constraints that I'm going to write correspond
to this statement here that says you want to majority
in each demographic.
So you can imagine that because you have three demographics,
you're going to have three constraints.
You could have written this differently.
There's just any number of variants here.
And you'll get the sense of that as we go to other examples.
But we'll just stick to one variant here.
So now, I want to translate everything
that I've written in English over there into algebra.
And so I got my minimization criterion-- minimize x1 plus x2
plus x3 plus x4-- subject to minus 2x1 plus 8x2 plus 0x3
plus 10x4 greater than or equal to 50,000.
And this represents the requirement
that I want a majority in the first demographic, namely
the urban demographic.
And so I want at least 50,000 votes there.
And I need to spend the money corresponding
to the values of x1 through x4 in such a way
that I get those 50,000 votes.
And that represents that, and it's just
reading off the minus 2, 8, 0, and 10 from the urban column.
So those numbers that you see here correspond to the column,
because I'm talking about the urban demographic.
And you can imagine that the next constraint
is going to correspond to the middle column, and the third
to the third.
So I'll just write that out.
I will call this constraint the constraint number
1-- I might refer to it later-- 5x1, plus 2x2 plus 0x3,
plus 0x4 greater than or equal to 100,000.
Call this number two.
And then finally, 3x1 minus 5x2 plus 10x3.
And that's our set of constraints,
but there's one more little issue
that we have to be careful about, if you're being precise,
and that is that there's no notion of un-advertising.
And so you're going to spend positive dollars.
And so x1 through x4 is greater than or equal to 0.
So that's our first linear program.
And it came from this particular problem.
It'd be wonderful-- and that's exactly what
we're going to do for the rest of the lecture--
if we could solve this linear program
and any possible linear program in an efficient way.
And so the number of variables is small n.
And you can imagine that the number of constraints here,
just talking about these constraints, are m constraints.
And you certainly want at run time that is polynomial in n.
That's our goal here.
And as I mentioned early on, it was unclear
for the longest time-- well, at least not until 1979,
but people had been thinking about it
for a long time before that-- as to whether there
was a general algorithm that would solve any linear program
in time-polynomial in n.
And that was resolved in '79 by Khachiyan.
We'll look at a algorithm as simplex that in the worst case
runs exponentially in n but is simpler to describe
and is very efficient in practice.
So in our particular problem, this one, it turns out you--
and I'm actually going to come back to this in a second--
but I will just tell you that the optimum
for this particular linear program
with these particular numbers correspond to these numbers
here.
So you want to spend something of the order of $20,000
for-- so there's 100 here, so take away the two
0's-- so spend something of the order of $20,000 for the first
issue, building roads, spend a bit of money for the second
issue, ignore the third corresponding to farm
subsidies, and spend a bit of money for the gasoline tax
issue.
And these numbers aren't important, other than the fact
that they happen to be optimum.
So if you add up these numbers, then x1 plus x2 plus x3 plus x4
is something of the order of $21,000--
$27,000, excuse me, though I'm writing it out
as this fraction.
So important consideration here is that these values xi
are real numbers.
That's it.
It's not that they have to be integral.
Clearly, there were fractions here
for the optimum, some of them anyway.
But in general, linear programming
says the variable values are real.
There's also integer linear programming,
which is NP complete, which adds the additional constraint
that the xi values are integral.
So it turns into a harder problem.
You got polynomial-time solvable if the xi are real.
You got NP complete, which Eric is
going to talk about on Thursday, if the values are
forced to be integral.
So this extra constraint makes things worse from a complexity
standpoint.
We won't talk about ILP anymore for the rest of this lecture.
So I will come back to this.
And I'll talk about how we can show
that this is optimum without actually going
into a deep algorithmic dive.
But what I want to do just before that
is to give you the general formulation
of a linear program.
It's called the standard form in CLRS, also called
the general form.
In some cases, we'll look at the standard form for LP.
And I want to pop up a level about this example
and give you the general setting.
And we'll focus in on the general setting for the most
part.
But what I have here is I can either minimize
or maximize-- we had a minimization
problem-- for the political problem,
minimize the linear objective function
subject to linear inequalities or equations.
And the variables, think of x as a vector,
it's a column vector, or x1, x2, all the way to xn.
And the objective function is c times x, so that's c1x1 dot,
dot, dot, cnxn.
And we just had all the coefficients being a 1
over there.
And the inequalities, they're the fun part,
you can represent them as a matrix A,
so A times x less than or equal to b.
And notice that this is the standard form
that I'm talking about.
And now, I have diverged from what
I had here, because I had greater
than or equal to over here.
So it turns out, you'll see linear programs
in different settings.
Sometimes, you'll have minimization.
Sometimes, you'll have maximization.
Sometimes, you'll have less than constraints,
less than or equal to constraints.
Sometimes, you'll have greater than or equal to constraints.
Sometimes, you'll have equality constraints.
We'll spend a little bit of time talking
about how you can transform any given linear program
into a standard form.
So our standard form is going to be something that maximizes
the objective function.
So these are our inequalities, and they're
represented as less than or equal to.
That's the standard form.
And you want to maximize c times x-- again,
max for standard-- such that these set of inequalities
told Ax less than or equal to b and x greater than
or equal to 0.
So for each of the values that correspond to the variables,
you want these variables to be non-negative
in the standard form.
And you want less than or equal to corresponding
to each of the inequalities-- not equal to,
not greater than or equal to, but less than or equal to.
And you have this linear cost function,
where you could have arbitrary coefficients,
but you're maximizing it.
So that's it.
So it's all about polarities, not much more than that.
It's just about polarities.
And if you get a linear program, a specific linear program that
doesn't conform to this-- we'll spend a few minutes talking
about conversions-- and it's going
to be fairly straightforward.
May not be immediately obvious, but we'll get to that.
Any questions so far?
So I want to go back to this claim
here, where I said this is optimum.
Now without actually describing an algorithm to you,
I'm going to be able to show you, convince
you, that this value here corresponding
to whatever it is, 3 million, 3.1 million,
is in fact optimum.
And this is something I could do,
because linear programming has this powerful notion
of duality.
So what is that?
Well, let's just first look at our specific example here.
And I'll give you a very specific observation.
I'm going to give you what you can
think of as a certificate of optimality.
I'm going to give you a certificate of optimality
for that set of numbers.
And here's how I'm going to do it.
So is there a short certificate?
I can imagine giving you a long proof
that a particular linear programming algorithm always
gives you the minimum answer, the optimum answer,
walk through that, executive that algorithm
on this particular example, and then you're
convinced, of course, that the solution is
going to be optimum.
But for this specific example, I want to give you a certificate.
This certificate isn't going to work for other examples.
It's going to short, because it only works with this example.
But it won't work for other ones.
And so how do I do that?
So the answer is, in fact, there is
a certificate that shows that the LP solution is optimal.
And consider that I compute this particular algebraic quantity,
where all I've done is I've taken these three equations
and I've multiplied them by these magical constants.
And so I'm not going to tell you how we get
this certificate of optimality.
But I'm going to give you the certificate.
And it's going to be clear that it's
a certificate of optimality.
And if I take these three equations here, 1, 2, and 3--
actually, I refer to 1, 2, and 3, they refer to the equations.
These are equations or constraints.
And so I take that.
And obviously, if I have a bunch of equations
and I multiply them out, I can certainly add them up,
and I get another equation at the end of it.
And it's all going to be linear.
And if I do that, I get x1 plus x2
plus 140 divided by 222 times x3 plus x4.
So that's what happens to the left hand side.
And the right hand side is-- you'll
recognize this quantity-- five 0's divided by 111.
That's what you get.
So now, can someone tell me why in the last step, why
this is a certificate of optimality--
the fact that obviously, this is all algebra, once I've
discovered the coefficients-- so now that I've done this,
why have I just shown you that 3.1 million divided by 111
is, in fact, optimum?
Can someone tell me this?
Look at what you have on the left hand side.
No?
Yeah.
AUDIENCE: Any other solution would cost more
than the amount put in spent.
SRINIVAS DEVADAS: Any other solution,
but I want you to relate that to-- what am I spending?
AUDIENCE: You're spending 3,100,000--
like, it's the same thing.
SRINIVAS DEVADAS: Yeah, but I mean, this was a claim.
This was a claim-- and at this point, an unproven claim.
It was an unproven claim.
Yeah, go ahead.
AUDIENCE: You know that the left hand side of that inequality
is less than or equal to x1 plus x2 plus x3 plus x4.
SRINIVAS DEVADAS: Correct.
And what is x1 plus x2 plus x3 plus x4, to be clear?
AUDIENCE: The thing you're trying to minimize?
SRINIVAS DEVADAS: Yeah, exactly, the thing
you're trying to minimize.
Exactly.
You're almost there.
but the key observation here is that x1 plus x2 plus x3 plus x4
is greater than or equal to x1 plus x2-- because all of these
are positive quantities, remember-- 140 divided by 222,
that's less than 1, x3 plus x4, correct?
So given that, I can say that this is greater than our equal
to 3,100,000 111.
It's because of this observation that it's
a certificate of optimality.
She has her head down, OK.
Great.
So that's pretty cool.
Just cooked up these coefficients from somewhere,
pulled them out of a hat, you're all
convinced now that the value we got was optimum.
Did not run an algorithm.
Maybe I ran an algorithm-- of course,
you ran an algorithm to get those coefficients, right?
Well, how did those coefficients appear?
So we're not going to spend a whole lot of time on this.
You'll see this likely in a problem
set or perhaps in section.
But in general, I won't worry too much about duality,
other than knowing the concept.
And this notion of LP duality essentially
says that what just happened wasn't a coincidence.
You can do this all the time.
There will always be, for a linear program,
a short certificate of optimality
that corresponds to some set of coefficients
that you can do this particular math with by taking
these linear constraints, multiplying them out, adding
them up together, and showing that you have a lower bound
on-- in the case of this problem-- you can't get lower
than this.
And therefore, for a minimization problem,
when you reach that, you clearly found the optimum.
And that's the notion of LP duality.
And the basic theorem-- and this is really more as an FYI,
we won't prove this theorem-- is that if you
had the standard form for the LP, which I'm just writing down
again here, where you had Ax less than b, x vector
greater than or equal to 0.
So that's identically what I had up here, or done here,
corresponding to the standard LP form.
Well, there's a dual-- this is what's called the primal form.
Usually, if you don't say it, you
think of it as the primal form.
And if it's dual, you call it a dual.
And this is primal form of LP.
This is a dual form of LP, or dual LP.
And the dual LP flips everything.
And it's not just negation, but transposed,
and the actual variables also swap functionalities.
So it's really pretty cool.
So your max turns into a min.
The c gets replaced by the b, which is on the right hand
side of the inequalities.
And your constraints are A transpose,
y greater than or equal to c.
So there's a flip there as well.
And y is greater than or equal to 0.
So there's a bunch of things that's going on here.
And these two problems end up being equivalent--
the primal and the dual, you can always do this.
And essentially, what is happening here
is that you're solving these two problems simultaneously.
And there's lots of algorithms that
keep flipping between these two forms for efficiency.
But ultimately, what ends up happening is
you see that the actual constraints that you had here
corresponding to the b constraints turn into-- the b
ends up in the cost function here.
And that's essentially what's happening out here
with respect to multiplying these equations out
with particular coefficients.
So as I said, this is really more as an FYI.
This is an obviously interesting and an interesting proof
of optimality, which is a different kind of proof
from proving an algorithm correct
and applying that proof to a particular instance.
That's the kind of thing that happens
in LP, especially when you flip from primal and dual forms.
So I'll leave it at that.
What I'd like to do is give you a sense
of how we can convert to standard form,
so you can apply an algorithm that-- for example, you
have a program and it only requires standard form.
It runs on standard form.
Let's go over it really quickly.
This is not going to take very long-- a translation
from different kinds of LPs-- and we had a slightly different
here for our political problem that had a minimization--
and how would we convert that to standard form.
So it's probably just one conversion here that's tricky.
So suppose I want to minimize minus 2x1 plus 3x2,
and I want to convert it to standard formal.
All I have is a standard LP solver.
What do I do?
It should be easy.
What do I do if I had a solver that was maximizing,
but I want to minimize a quantity?
Just switch the signs, right?
So negate to 2x1 minus 3x2 and maximize.
So that was easy.
This is a tricky one.
Suppose xj does not have a non-negativity constraint.
So it just happens to be the case that it's not dollars
but it's some other quantity that can go negative.
It might be profit or loss.
So that quantity represents profit and loss,
so it could go negative if it's loss.
So I don't have this constraint in my original problem
specification.
But my standard form and my LP solver
requires this entire vector to be non-negative.
So I got a problem here.
I can't use my standard solver, because
of this non-negativity constraint.
So how do I fix that?
How do I turn it into a problem that allows the standard solver
to be used?
Yeah, go ahead.
AUDIENCE: You can break it up into two variables,
like xj1 and xj2, so xj1 minus xj2 equals xj,
and both could have [INAUDIBLE].
SRINIVAS DEVADAS: Perfect, great, that's good.
Here you go.
So what you do here is take xj and replace it with,
let's say, xj prime minus xj double prime.
And you have xj prime greater than or equal to 0,
xj double prime greater than or equal to 0.
But depending on the particular values
in whatever solution you're exploring
are the final solution, you may have an actual xj value
that's negative or positive.
So you added an extra variable here to your linear program.
And a couple more real quick, suppose
that I have an equality constraint corresponding
to x1 plus x2 equals 7.
What do I do with an equality constraint
where I have x1 plus x2 equals 7?
Yeah, go ahead.
AUDIENCE: You can say x1 plus x2 is greater than or equal to 7,
and x1 plus x2 is less than or equal to 7.
SRINIVAS DEVADAS: No, you can't do less than or equal to.
AUDIENCE: But then you can flip the signs to--
SRINIVAS DEVADAS: Ah, then you could flip the signs.
So you have two steps there, good.
So your less than or equal to needs another multiplier.
So what you end up doing is something like x1 plus x2
greater than or equal to 7.
And then you need-- if you do what the gentleman just
said-- and flip the sign, you get minus x1 minus x2
greater than or equal to minus 7.
Is that right?
No?
I messed up?
Oh, I want less than or equal to.
You're right, you're right.
So I need less than or equal to-- that's right, of course,
thank you.
So I need less than or equal to in both places.
So that's the standard form.
I needed less than or equal to.
Good.
What you've done is increased the number of constraints
by one.
Did I get this right the second time?
All right.
So that's pretty much it.
The last thing, which I won't really write out,
is, which we've done here already, greater than
or equal to constraint translated-- I won't give you
an example of this; we have this already--
translates to less than or equal to by minus 1 multiplied.
So we have to invoke that in order
to do the equality anyway.
So you're in business.
If you have a standard LP solver,
you can take pretty much any optimization problem that
is linear in terms of its objective function
and has linear constraints, and you can transform it into LP.
If you had non-linear constraints,
there's lots of work that goes on
in linearizing those constraints and using LP solvers.
It's a very practical thing to do.
It may be something you'll end up doing, invoking
these powerful LP solvers-- typically, they're commercially
available; the best ones are commercial--
and use it to solve your particular problem.
It turns your algorithm design problem into a reduction.
And so you'll spend really the next couple of weeks
thinking about reductions.
We'll start that up right now, where
we'll take existing combinatorial
problems, for which we already know algorithms for,
and you're going to reduce them to LP.
Just to give you a sense of what the power of LP
is, but this notion of reduction is very powerful,
you can use it to do complexity proofs.
Here, we're just using it as a convenience in today's lecture
to use our LP hammer.
So let's say that I have our favorite problem
of the week, namely max flow.
And I want to convert that to LP.
So go back a week ago, and right about this time a week ago,
and we'd set up the max flow problem.
And let's assume that we went back there.
And we didn't talk about augmenting paths,
and we didn't talk about residual capacities or min-cut
or anything like that, but we knew LP already.
And we just want to solve max flow using LP.
So let's do that.
So this is maximum flow.
I'm not going to bother with converting to standard form.
We know how to do that, given what we just
did here, over there.
So I'll just do whatever I want to keep things simple.
Max flow is obviously a maximization problem.
And using the same notation we've used,
it's not going to look like Ax and b,
just because I want you to recall what max flow is.
And we're going to translate that.
And the values of these variables--
or the names of these variables, whether they're x or f,
it should really matter.
We know how to do LP at this point-- we
know how to formulate LP, I should say, at this point.
And we're assuming that we have an LP solver.
So what I want to do here with the maximum flow problem
is maximizing the flow value.
And it's simply, you grab the source,
you have a variable associated with the flow from the source
to every other vertex of v. And you have to maximize that.
So that was the setup for max flow.
I'm not changing that.
What do you think the three constraints, or whatever
set of constraints that we have here,
are going to correspond to the LP?
You spent a week on max flow, looked
at the problem set, what constraints am I
going to have to put up there?
I'm going to have to put up capacity constraints.
That's an obvious one.
What is another one?
Conservation constraints.
All flow entering a node that is not the source or the sink
has to leave it.
In the original network, is there
a concept of negative flow?
No, you will define it going in the other direction.
So we did talk about negative numbers, et cetera,
but you're going to have positive quantities, especially
if you look at net flow, the version of the flow
that we zoomed in on in the Tuesday lecture from last week.
And you also have-- in the general setting,
you're going to have these skew symmetry constraints as well.
So the three things that you need here
are skew symmetry, conservation, and capacity.
So you have such that f u, v equals minus f v, u for all u,
v belonging to V.
And depending on the kind of network that you have,
if you constrain it to a sudden type
that you don't have these two-way edges,
you could certainly remove some, if not all,
of these skew symmetry constraints.
Important ones are conservation and capacity.
And this should seem pretty familiar to you.
But the key-- the reason I'm writing these all out
is primarily to ensure that you understand that these are all
linear constraints.
So that's pretty much the only thing
that you need to observe here.
Obviously, these constraints you've
seen many a time from the two lectures last week.
But notice that they're all linear.
And finally, this one is f u, v less than or equal to c of u,
v for all u, v belonging to cap V.
So this is f.
That's a variable that's less than another constant,
clearly linear.
Doing a bunch of sums here.
I could obviously have multipliers, scalar
multipliers.
In this case, for conservation, I
don't have scalar multipliers, but clearly linear.
Skew symmetry, got two variables in here.
One of them is a negation of the other, clearly linear,
so that's why this is an LP.
And so you might say, well, I know better,
max flow is much more efficient than any LP
solver that's out there.
And you would be right.
If you have a max flow problem of this variety,
it's difficult to imagine that you
would get performance, empirical performance from running an LP
solver.
But this generalization of max flow
that's a multi-commodity max flow, where you just don't have
one commodity flowing through.
You may be counting cars and trucks on a road,
or there's two different kinds of liquid
flowing through the same pipe, whatever, gas or liquid.
And so when you have multiple commodities,
you may have a linear but more complicated cost function
that's a function of the flow of each of the commodities.
And they may have a certain weight associated with them.
So there's a lot of things that could
be more general-- there could be more general settings
corresponding to max flow.
And I'll just leave you with the thought
that you could simply have two commodities.
And we'll just call them 1 and 2.
And so now, you have the f1's and the c1's and the f2's and
the c2's.
Each commodity has to be conserved.
But what about the capacity?
What do you think happens with the capacity?
Let's just assume these are two different kinds of cars.
So what would the capacity constraint look like?
Yeah.
AUDIENCE: You can say either c1 or f1
plus f2 is-- for each edge, you can add them together
or you might take the linear [INAUDIBLE].
SRINIVAS DEVADAS: Exactly, that's right.
So good point.
It may be the case that I have distinct capacities.
And in fact, if you have completely disjoint problems,
you're right in that you can solve them separately.
But actually, the more interesting case
would be that you have a single capacity c,
so you'll have-- let me just write this out here.
If in fact you had two distinct things,
so if you had f1, c1, f1, c2, the question
is, do you have two distinct, disjoint optimizations,
in which case you just use max flow twice.
On other hand, what's more interesting really--
and I should've used this example for starters--
but here's a better one.
You have two commodities and a single capacity.
So the road is a good example.
Both the cars and the trucks share the same road.
It has a certain capacity.
And now, your capacity constraint
is looking like f1 plus f2 over here.
And that's the flow through the particular edge uv.
So you have something like f1 u, v plus f2 u,
v is less than or equal to c u, v for this total capacity.
And that's pretty much it.
So that is linear.
The nice thing is that it's linear.
You could put weights on it.
If you wanted to claim that a particular commodity
1 uses up-- because it's a truck,
it uses up more space on the road.
And you can accommodate fewer of them.
You could put a multiplier in there.
Still say it's linear.
So that's the power of by having an LP engine.
You could translate problems that
are not exactly max flow, that are multi-commodity flow.
You may have additional linear constraints that you could add,
and you could still use your LP package.
So that's the reason why this is interesting and powerful.
So that is kind of an obvious, corresponding to max flow.
Let's look at something that's a little less obvious.
And it's going to be a little tricky to convert the shortest
path problem to LP, not a lot of work
but one little observation that's
going to be important to make in order for the whole thing
to flow through or actually work out.
So we all know the shortest path problem.
We want to find-- let's just call it the single source
shortest path problem.
You have a specific source.
That's going to turn into the point from which you're
going to start computing the distance.
That's what Dijkstra does, and that's Bellman-Ford does.
And so this is from vertex x-- s, excuse me, s.
And what I want to do here is obviously set it up
as a set of linear constraints.
If I have dv corresponding to the distance from the source--
so dv represents the distance from the source--
and eventually I want dv to be the shortest
distance from the source.
That's our notation for shortest paths.
dv represents an existing path-- it may not
be the shortest path-- from s to v, the value of that.
But dv monotonically decreases as you run through.
It's initially infinity in Dijkstra
going back to Dijkstra.
And then we shrink it through a process of relaxation.
Now I want to try and model that-- I want to try and model
all of this as an LP.
So it's not immediately obvious--
the thing that the flow networks had,
where we had these constraints.
We have capacity constraints and conservation constraints.
And we could turn that constraint into an inequality.
And it was pretty smooth.
It's pretty easy.
So what I need to do here with shortest paths
is something that's a little more subtle.
So what basic constraint do I have in a shortest path
algorithm?
What's an inequality-- you remember an inequality
from shortest paths that we kept talking about?
The triangle inequality.
So we're going to have to go with the triangle inequality
and take the triangle inequality and use that to create an LP
formulation of shortest paths.
In particular, what we have here is
that I could write dv minus du is
less than or equal to w u, v for all u, v belonging to E.
And that's the triangle inequality.
And I'm going to have d of s equals 0.
That's the only thing that I start with.
And so what's happening out here is simply
that there's different ways of getting
to v. And my shortest path is going to be the best
way of getting to v. So in particular,
the way you want to think about this is that if I have a v
and I can get to it from, let's just say, u1 and u2.
And maybe the source is over here.
And these are the only two edges that can get to v.
So I'm just looking at a fairly limited setting.
u1 and u2 are going to have to be the two vertices.
One of these two is going to get me to v.
And I got w u1, v here.
And I got w u2, v over here.
And so what this is saying is, I'm
going to have to write this out for each of these edges.
For each of these edges, I'm going to have this constraint.
And that's says that the dv value, if I want the shortest
path, should obey both of these constraints.
And if I want to obey both of these constraints, one of them
is going to be my limiting constraint.
And I'm going to get the min of those two.
Correct?
So in effect what this translates to
is that it's an AND, right?
So dv have minus du1 is less than or equal to w u1,
v. dv minus du2 is less than or equal to w u2, v. That's
an AND, because I'm putting both of those constraints in here.
And that essentially means that dv
is going to be the min of the two quantities--
the du1 quantity plus the w u1, v,
and the du2 quantity plus the w u2, v. That make sense?
Ask me questions if this is unclear.
So that simply corresponds to the fact
that I'm doing an AND over here.
I'm adding all of these constraints in there.
So I'm applying the triangle inequality to every edge,
to every relationship between a vertex that
has a path ending at it.
And you're pushing it forward to this vertex v,
all the different ways that you can get to v. In this case,
there is two sets of ways-- one from u1 and one from u2.
And the last step is a minimization step.
So you think you're done or we're done,
but we're not quite done, because what's
missing here in terms of my formulation of LP?
What else do I have to do here?
Well, sure, non-negative-- let's do that.
Sorry?
AUDIENCE: Objective.
SRINIVAS DEVADAS: Objective, who said objective?
You again?
So we are missing an objective function.
Now shortest path is what kind of problem again?
Short means minimum?
Minimum, height whatever.
So do I put a-- what happens if I put a minimum in here?
And let's say that I do something like sigma
over V, cap V, dv, because I want to minimize--
or I could pick a particular one.
I could pick a particular single source, single destination,
and I put a minimum there.
What happens?
Does this work?
What's the solution to this problem,
if I minimize the distance?
0, because the zero value is going to work.
So there's something-- I haven't put in the constraint
that I do want a path.
I do want a path from s to v for any v that
matches one of the quantities in the min,
because the min says I have equality.
The big issue here is, this is a less than or equal to,
and that's why the min doesn't work in the objective function.
It's a less than or equal to.
But this min over here, which is the definition of a shortest
path, is saying that it's either equal to this quantity
or equal to that quantity.
There's an equality over here that is missing from this side.
And that's the key observation.
Once you observe that, that need equality
for one of these constituent quantities of the min,
then you'll see that what you have to do
is simply change this min to a max.
So you say, well, how the heck did
a min get changed into a max?
And I'm not sure I'm going to convince every on of you
in the next minute or so.
But the bottom line is, it comes down
to I do have a min already in my inequalities,
because I'm ANDing each of these inequalities,
and I'm putting down each of those inequalities in there.
So each of them is going to force me to find the best
solution, because they're going to constrain me to not
go via u2 if u1 is better, because the other constraint
corresponding to u1 is going to force me down.
So there's an additional min in there,
because the ANDing of the less than or equal tos.
And then in order to actually force the equality for one
of those, I need to push up as hard as I can,
or as high as I can.
So think about it.
Play around with a couple of examples.
Choose a simple example for starters.
And you'll see that this is the correct formulation.
So you can see that it's not completely clear in some cases
how to transform problems to LP.
But even in those cases, sometimes you can.
So there's just a ton of different problems,
a good skill to have to be able to take
combinatorial optimization engines,
like LP or even max flow, and be able to translate problems
to them.
It's something that you'll probably
do if you stick to algorithms in your careers
or exploiting available algorithm packages.
So the last thing I want to do here for the rest of the time
is to give you some sense for how an LP program is actually
optimized.
How can you possibly take the standard LP formulation,
which is a general setting.
You know nothing about shortest paths,
let's assume, nothing about max flow.
It's not about a specific problem.
It's about the general setting.
How can we solve the general setting,
because that was the theme here anyway.
You had this engine, and you want to use this engine.
But now, how do you build this engine?
So what we're going to do is look at a fairly simple example
of the simplex algorithm.
And this algorithm is in the textbook.
And it'll be in my notes.
So I'll get as far as I can.
It's not that complicated to describe, especially
from an example standpoint.
But I may not get through all of the steps
to get you the optimum for this particular example
given how much time we have.
The most important concept in simplex
is yet another form of representation
for simplex, which says that you can represent the LP, not
in standard form, but in slack form.
So I'm going to tell you what slack form is.
And then what we're going to do is,
the flow of simplex, algorithmic flow,
is to convert one slack form into and equivalent.
Obviously, you don't want to do something that's incorrect,
but it has to be an equivalent slack
form, whose objective value has not decreased and has likely
increased.
So you're guaranteed that the objective value has not
decreased.
You're not guaranteed that it's increased.
And then we're going to keep going till the optimal solution
becomes obvious.
And you might say, how is this obvious?
That's the reason why I talked about the short certificate
of optimality.
It's definitely a relationship between the termination
of simplex and the fact that you can now say,
hey, I know I'm done here, it's kind of obvious
that I can't do any better.
And hopefully, you'll see that by the end of this lecture
in this simple example.
So that's it.
It's an iterative algorithm.
It's exponential, unfortunately, because this
takes m plus n, choose n iterations in the worst case,
where n is the number of variables
and m is the number of constraints.
Most of the time, it does a lot better,
but that's the only bound that you can actually
prove in the worst case.
And so you're stuck with an exponential algorithm
if you're using simplex worst case.
We won't actually do much analysis on simplex.
It's really out of scope for 046 in terms of the analysis.
The actual algorithm is certainly within scope.
So what I want to do is give you some sense for what
the slack form looks like.
And we'll do a couple of iterations of simplex.
And we'll get as far as we can before the end the lecture.
So we'll take a different example
from our political example.
It's similar in size.
And I want to explain to you what the slack form is
and why it's interesting.
So what I want to do is maximize 3x1 plus x2 plus x3 subject
to the constraints that x1 plus x2 plus 3x2
is less than or equal to 30, 2x1 plus 2x2 plus 5x3 is
less than or equal to 24, 4x1 plus x2 plus 2x3 less
than or equal to 36, and then non-negativity constraints,
x1, x2, x3 greater than or equal to 0.
So that's our example problem.
I'll leave it up there.
You're going to convert this to slack form.
And so what is the slack form?
We're going to introduce an additional number of variables
that correspond to the number of equations that we have.
So we're going to introduce, in this case, three new labels,
because I have three equations.
And the slack for this problem looks like this.
I'm going to have z equals 3x1 plus x2
plus x3, same as before.
And then I'm going to have variables
that represent-- these are called basic variables.
And the original variables are called non-basic variables.
So I'm going to add three basic labels, x4, x5,
and x6, corresponding to these three constraints.
And they're going to represent slack in the sense
that they're going to correspond to how much slack you
have in the inequalities that you
have in the original problem.
So if x4 happens to be 0, then you're jammed up.
You have no slack, because x1 plus x2 plus 3x3 equals 30.
And increasing any one of them will violate the constraint.
So that's just simply the notion of slack.
It's how much room do you have.
x5 is 24 minus 2x1 minus 2x2 minus 5x3.
And the last one is 36 minus 4x1.
This is very mechanical up to this point.
And so I'll call this set of equations, equation I.
And what I'm going to do is I'm going
to now work on a space that corresponds to x1, x2, x3, x4,
x5, x6.
So I'm going to have these solutions that
now have six values associated with them,
as opposed to just three values, because I've
added three variables, the basic variables, to my non-basic set.
So the original variables are non-basic,
just to differentiate.
So so far it's just set up its definitions.
We can think about up running through iterations of simplex.
It takes about three iterations here
in order to get to the point where the optimum is obvious.
So you're going to convert through three slack forms.
And then finally, when you get to this fourth slack form,
you see that you have an optimal solution.
And how does that work?
You're going to have the notion of a basic solution, where
we set all non-basic variables to 0.
So in this case, what we're going to have--
and then we're going to compute values of the basic variables.
So our objective function here is going to be 3 times
0 plus 1 times 0 plus 2 times 0, which is obviously 0.
And the values x4, of course, is going
to be-- because all of these are 0-- is going to be 30.
x5 is going to be 24.
And x6 is going to be 36.
So this is kind of a trivial starting point.
So you can think of this as 0, 0, 0,
the solution that you're looking at, 30 24, and 36.
So that's our starting point, which doesn't really
tell you much.
But now comes the key step, where
we're going to do something that's called pivoting.
And in pivoting, you're going to swap
a basic able with a non-basic variable.
It is a step that requires some intelligence.
But the basic step is a swap.
So one of the basic variables is going
to become a non-basic variable and vice versa.
And how do we select this?
Initially, you can kind of do this in an arbitrary way.
It gets a little more subtle as you go along.
You don't want to do things in a random way.
But let's just start with what pivoting actually
does in a more generic setting.
The basic step is select a non-basic variable-- let's
call it x sub e-- who's coefficient in the objective
function is positive.
You can always redefine things if you can't find something
like that.
But we won't go there.
And then what we're going to do is
we're going to increase the value of x as much as possible.
And we always have constraints, of course.
So we can do that without violating
any of the constraints.
And then at this point, variable xe becomes basic.
So it's going to turn into the left hand side.
It's going to move over.
You're going to swap-- the x1 might be over here, over here,
and you got to rewrite these equations so the x1 becomes
basic, for example, over to the left hand side,
and the variable that it replaced
goes over to the right hand side.
So you can think of this as Gaussian elimination,
except with inequalities.
There's definitely relationships there,
if you recall your Gaussian elimination.
If you don't, don't worry about it.
So xe becomes basic, and some other variable
becomes non-basic.
The values of other basic variables
and the objective function may change.
So we'll do one pivot step at least,
so you get a sense for the algebra involved.
And it becomes a little more concrete.
To motivate you further, you'll be
doing this in problem set 8 on a different example.
So what did I do here is we're going
to select a non-basic variable.
So let's just select x1, lexicographic order
or numeric order, let's select x1 is selected.
That's the non-basic variable.
And what I want to do is increase the value of x1.
So I want to increase it without violating constraints.
Now, which of these constraints do you think
is going to cause trouble first with respect
to increasing the value of x1?
x1 is now 0.
We're at ground level, we have all 0, things are feasible.
Now, as I start increasing x1, remember,
you have non-negativity constraints
associated with each of the x4 values as well.
That's what these basic variables represent.
So don't forget the fact that the constraints are violated.
You need all the xi's to be greater than or equal to 0.
That was true for the non-basic variables.
It's also true for the basic variables.
So a violation of a constraint implies
that one of the currently basic variables goes negative.
That's exactly equivalent to the original inequality
not being satisfied.
So which of these constraints, do you think,
is going to cause trouble here?
Just look at it, and should be able to look at the three
equations and tell me, as I increase the value of x1,
where am I going to hit my limit?
x6, yes, absolutely correct.
So that's because of the minus 4 here.
This is a big multiplier.
I got a minus 1 here, a minus 2 here, and a minus 4 here.
And if we just look at 4, the magnitudes, 4 is bigger than 2
and is bigger than 1, so it's going
to be that third constraint.
So third constraint-- you can obviously
compute this numerically or mechanically.
It was just easier to do this in this example by eyeballing it.
And so third constraint is the tightest one.
And it limits how much we can increase x1.
So I'm going to do my second step
up there, which corresponds to rewriting
x1 as these other variables.
And now, I got x1 on the left hand side and x6
on the right hand side.
And now, it's just merely a matter of substitution.
Once I've done this, I'm just going to jam through and go in
and I'm going to rewrite the other equations with x6
on the right hand side.
And that is, I'm going to replace x1
with the eyeball equation.
And that's really a simple substitution.
So at this point, what's happened
is, because the third constraint representing x6
was the one that was chosen, what's happened
is that x1 and x6 are going to interchange roles.
x1 was non-basic, it's going to become basic.
x6 was basic and is going to become non-basic.
And that's basically the essence of the simplex algorithm.
The iteration and then the convergence and all of that
is, as I mentioned, going to require getting to a point
where the optimality is obvious, but we
won't be doing any proofs corresponding
to conversions for simplex or any other specific LP
techniques in this class.
Maybe some constraint techniques,
I take back what I said, but certainly not for simplex.
But I just want to give you a sense for the flow here.
And so let's just go through this last thing here in terms
of finishing off the pivot xi's.
And I want to show you what the equations look like.
And if you just keep doing that, at some point, you'll converge.
And so what happens here is you have z
equals 27 plus x2 divided by 4 plus x3 divided
by 2 minus 3x6 divided by 4.
x1 equals-- so there's a bunch of algebra
here that I'm obviously skipping over, but it's simple algebra.
And I have x4 over here, 21 minus 3x2
divided by 4 minus 5x3 divided by 2 plus x6 divided by 4,
and then one last thing here.
So that's my pivot step, that I've flipped x1 and x6.
So now you ask, what was the point?
What was the point of this?
Well, the point of this was that you actually increased
the objective function value and, in this particular case,
quite significantly, while maintaining correctness.
And so let me just make these observations and conclude here,
because then that gets us to the point where you've seen
the details of one pivot step.
And you can imagine applying it over and over to a convergence.
And let's just look at the original basic solution,
which as you recall was 0, 0, 0, 30, 24, and 36.
And this is simply x1 through x6.
This original basic solution suddenly
satisfies these equations-- equations II,
if you just put them in there.
And it makes sense that it will have the objective value of 0,
given equivalents, but you can verify
that by saying that you have 27--
the original had the objective value of 0,
because all the x sub 1, x2, x3 were 0,
so that was an easy check in the set of equations corresponding
to the first set, which I've erased at this point.
But no matter.
And if you look at what I have here, I have 36 equals 0.
So the objective value here is 0.
It matches what you had before.
But the basic solution for II, I'm
going to set the non-basic values.
So what is a solution here?
The non-basic values are 0.
So the solution is going to be 9, because 9 is non-basic.
x1 is now non-basic.
x2 and x3 are still basic.
And then I have 21 and 6.
And x6 now has become basic.
So the way I get this solution is simply by plugging in 0's
on this aside.
And I get 9, 21, and 6, because I just plugged
in 0's on the right hand side.
So that's how I got a new solution.
And if you look at the objective value for this,
the objective value, you can look at this objective value
simply by looking at the original problem.
And the original problem had 3x1 plus x2 plus x3
as the objective value.
And so if you go off and you see, well, that you had 0's
for the other ones, but you have 3 times 9,
so we have an objective value of 27.
So this flip of our pivot basically
got you from an objective value of 0,
while maintaining correctness, to an objective value of 27.
And you can look at this in the notes or in CLRS,
but you have to do two more pivots corresponding
to two other variables-- the same grungy
stuff that I went through here, substitution after selection.
And the objective value is going to increase.
And you might ask, how do I know that I'm done?
And so that was the last thing here,
which is increase the value until it becomes obvious-- no,
this was pivoting, I'm sorry-- increase the value pivot
until it becomes obvious what the optimum is.
And what ends up happening is you end up
with an objective function.
In this case, the objective function mind you,
this thing over here is the objective function.
And notice that it has a negative coefficient
on the variable that was actually
the part of the first pivot.
So x1 and x6 were a part of the first pivot.
And this got a negative coefficient.
So what ends up happening here is you end up getting something
like 30 minus something minus something
minus something, where you have xi values over here.
And when you set these to be 0, that's the best you can do,
because these are all negative quantities.
So I'll just leave you with that.
Hopefully, you understood how we do the pivoting--
sub through it three times, and then
you get that objective function, and the optimum value is 30.
See you next time.