SRINI DEVADAS: Good morning, everyone.
Thanks for making it here through the snow and sleet.
You will be, quote unquote, rewarded
with a cool little puzzle that has both recreational value as
well as a deep connection to the most popular sorting algorithm,
or at least the most commonly used sorting algorithm,
called quicksort.
And so what we have up here is what
I call the disorganized handyman puzzle.
This is not a puzzle of my invention,
but I called it this because I think at some point
when I read this, it was a carpenter who
had a bunch of nuts and bolts in his bag,
and they were all mixed up.
So rather than having these nuts attached to the bolts,
he was disorganized and the nuts and bolts were separate.
And then they all got mixed up together in a bag,
OK?
Now, obviously, all puzzles are a little bit contrived.
And so we're going to assume here
that there's 100 different nuts and 100 associated bolts.
And these 200 objects are all mixed up into this one bag
that the carpenter is carrying.
And not only that, each of the bolts
is unique in terms of its size.
And as I mentioned, there is an associated nut associated
with each unique bolt. And so, as you can imagine,
the goal of this puzzle is going to be finding
the most efficient way of creating
some organization in this bag by attaching
each nut to the corresponding bolt, or vice versa.
And you can assume that there's no ambiguity because
of the uniqueness of the nuts and the bolts.
And there's 100 pairs waiting to be discovered.
And you can try to make that process as
efficient as possible.
As with all of the puzzles that we look at here,
there's going to be naive slash straightforward way of doing
this.
You're going to go ahead and figure that out pretty quickly.
We're going to analyze the complexity of that,
or how long it takes for a specific example of 100
nuts and 100 bolts, and then we're
going to scratch our heads and try and do better, all right?
And as I mentioned, obviously, there's
going to be some programming associated with this.
And we're not going to represent nuts and bolts
in programs or codes.
We're going to switch gears and talk about sorting once we're
done with this puzzle.
Good.
The only way that you or the carpenter,
if you're the carpenter's helper,
have of checking to see that a nut attaches to a bolt
is to try it out.
And the nuts and bolts are different enough in size
that even if you had your eyes closed or it
was a dark room, if the nut attaches to the bolt,
it would screw on perfectly.
If the bolt is smaller than the nut, it would just go through
and it would be obvious that the bolt is smaller than the nut.
And if the nut is smaller than the bolt,
it would also be obvious because the bolt won't
go through it, which makes perfect sense
from a physical standpoint, right?
And you all tried this before.
We've had-- maybe not all of you,
but I've certainly had occasion to discover
pairs of nuts and bolts, though it was never 100 of them.
So that's kind of a little bit contrived, as I said.
All right, good.
So the setup is clear?
We're good on the setup?
Excellent.
So what is the straightforward way of doing this?
Someone.
Yeah, Fadi back there.
AUDIENCE: So like choose a nut by random
and try every different bolt in turn.
SRINI DEVADAS: Right, we could choose a nut at random,
try every different bolt, or choose a bolt at random
and try every different nut.
And we're guaranteed, in this case, that there is a pairing.
And so after we do that, we can put this paired nut and bolt
aside and we've shrunk the problem down to one
less than the original problem.
If I started out with 100 pairs that are not paired together,
but there's 100 nuts and 100 bolts,
then I get one pair which is a correct pair,
and I have 99 nuts left and 99 bolts left.
And I keep going.
Perfectly reasonable way of doing things.
And let me show you how that would work.
These slides were made by my daughters many years ago
you know, back when they listened to me.
I have a fat chance of getting them to do any work for me
anymore.
But they are busier than they were, I guess, years ago.
At least they pretend to be.
And there you go.
You end up, in this particular example, which obviously
has many fewer than 100 nuts and bolts,
you end up checking this.
And in the worst case, how many comparisons
would I have had to do if I had 100 nuts and 100 bolts?
In the worst case, I would have to--
AUDIENCE: Do all 100.
SRINI DEVADAS: I have to do all 100,
because I might just have gotten unlucky.
Like I said, we're not eyeballing this at all.
You could probably prune the search a little bit
with respect to putting aside--
not having to do an exact comparison.
But let's just say that even looking
at a nut, as opposed to even touching it
with the bolt that you've chosen,
is, in fact, a comparison.
You're just eyeballing it as opposed to physically
touching the bolt and nut together,
and we're going to call that a comparison.
And if you do that, and if you make that assumption, then
obviously, you're going to have to look at all 100 nuts
if you have a random bolt, in the worst case, OK?
You might get lucky.
On an average, it might be only 50.
We're not going to do that type of analysis here.
So that's good.
And what is the complexity in terms of the growth rate
of this particular algorithm?
If I had N nuts and bolts, then I
do N comparisons in the worst case, to find the first pair.
And when I say first pair, I mean the correct pairing
that's associated with the nut and the bolt.
And what happens-- yeah, Kevin, you have a question?
AUDIENCE: Wouldn't it be less than N squared,
because once you find each pair, you have one less?
SRINI DEVADAS: That's exactly right.
It's absolutely less than N square in terms of numerics.
The growth rate is a slightly different question, right?
It's related, obviously.
And so if you do this and you get the first pair,
you now have a problem that is of size N minus 1, right?
And so in this case, how many comparisons
am I going to make when I have N minus 1 nuts
and N minus 1 bolts, in the worst case?
I'm going to do N minus 1 comparisons, right?
And then I keep going down.
And you could argue that at the very end,
when I have two nuts and two bolts, one comparison,
if I assume that this was a perfect set of nuts and bolts,
that we had all pairs right at the beginning,
you could argue that that small problem corresponding
to two nuts and bolts can be solved using one comparison.
And you immediately know which nut pairs with which bolt,
and the other one as well.
But let's call it a confirmation comparison,
and essentially say you need two comparisons here and one here.
You can obviously shave a number a little bit out of this.
But this goes back to--
Kevin is right.
It's less than N square if you look at it numerically.
But the growth rate is N square, because we
know that N plus N minus 1 plus N minus 2,
dot dot dot, 2 plus 1 is N times N plus 1 divided by 2.
And sure, you could take this 1 off
and this would become N times N minus 1 divided by 2,
but that growth rate is N square.
Grows as N square.
So if you had 100 nuts and hundreds bolts,
you're talking about something of the order of 5,000
comparisons if you want to do it numerically.
And it grows as N squared.
If you had 1,000, then it would be
1,000 times 1,000, which is 1 million,
divided by 2, that's 500,000.
So that is astronomical in terms of its growth.
You don't want to do that.
Obviously a contrived problem, but N square in general,
when you talk about manual labor,
is generally not very good.
So we'd like to improve this.
And we've been talking about recursion.
We've been talking about divide and conquer.
This is not divide and conquer in the true sense
in that you're going to a smaller problem, admittedly.
But recall I said, divide and conquer
is usually used when you break the problem
up into fractional pieces, which means in mergesort,
for example, we took--
or the tiling puzzle, we took courtyard and we broke it up
into four courtyards.
So essentially, we had four quarters
in the case of the tiling puzzle.
We had two halves in the case of mergesort.
Here we are solving, basically, one nut and bolt
in the original puzzle.
And then we're going from N to N minus 1,
and so there's many more steps, right?
The one thing to remember when you think about complexity
is that when you go N to N divided by 2 to N divided by 4,
and then you keep going, and you go all the way down
to 1, this, of course, when you go like this,
there's a linear number of steps.
But when you do that, how many steps
do you have to get all the way to 1 in relation--
if you start with N, how many steps do you have?
If this were 64--
or let's just take a simpler one.
If this were 4, how many steps would I have?
I would have two steps.
If this were 8, I'd have three steps.
And so what's that formula?
AUDIENCE: Logarithm.
SRINI DEVADAS: Logarithm.
Log to the base 2, right?
So that's the power of fractional sizes.
And this is actually a very fundamental notion
that is going to appear over and over if you
do any algorithms work or take any algorithms classes.
That divide and conquer is very efficient,
because the number of steps in order to get down
to small problems is relatively small.
It's a logarithm.
Now, if you broke this up into three parts
and you went to N/3, et cetera, then
this would be log to the base something else,
log to the base 3.
So log to the base 2 came because we broke it up
and we went down to half the size.
Now remember, of course, I mean, it's
not that the complexity here is just logarithmic.
It's that you do have two problems.
This is a function of the particular specific algorithm
that corresponds to divide and conquer.
But in the case of mergesort, it's
not that we just went from N to N/2--
well, we went from N to N/2, but there were two N/2 size
problems.
But the beauty of divide and conquer
is if I magically broke up the N sized problem--
and let's go back to nuts and bolts--
into two problems of size N/2, so
each problem has N/2 nuts and N/2 bolts, each problem--
I'll repeat that-- N/2 nuts and N/2 bolts, then if I used--
and this is magical, right?
I don't quite know how to do that yet.
But if I did that, then think about what happens with respect
to these comparisons.
So I said when N was 100, I needed 5,000 comparisons
for this naive algorithm.
But now if I did this and I got two N/2's, then I have 50,
so I call that M equals 50.
And then I have another one which is M equals 50.
And so roughly how many comparisons would
I need if I had, as problem of size 50, 50 nuts and 50 bolts,
using our original naive strategy?
Roughly?
AUDIENCE: 1200
SRINI DEVADAS: 1,225, right?
Roughly.
And this would be 1,225.
And so, so that is 2,500.
So of course, I haven't quite told you how to do this.
This is still magic.
But I was upfront about it.
But clearly, I've gotten an improvement
if I have this magic, OK?
And so let's talk about that.
Let's turn this nuts and bolts puzzle, or the solution
to this puzzle, and try and figure out a divide
and conquer strategy which is distinctly
different from the brute force strategy that
just reduced by one, all right?
Now, we'll-- this is just going on.
So the comparisons in the worst case is what I just said.
And it grows as N square.
So big O N square means it just grows as N square.
That's asymptotic notation that we won't go into,
but you understand what that means now.
So if I do a straightforward divide
and conquer, like I did with mergesort,
where I just took the array and I split it
in half, if I take these nuts and bolts
and I separate the nuts out, and I put 50 on this side, 50
on the other side, take the bolts,
put 50 on this side and 50 on the other side,
is that going to work?
No, that's not going to work.
And the reason is if I do this arbitrary partition, then--
and let's say that I take this, and the two of you
get together, you're friends and partners,
and you say, let's do this in parallel and save some time,
you're still going to be counting
the number of comparisons.
And so, obviously, you also would
like to reduce the number of comparisons.
But you can't even use a helper here in the sense
that if you split this up-- obviously,
this was not 50 and 50, but three nuts and three
bolts on one side and four and four on the other side.
As you can see from this example,
you had a situation where the matching nut was in one pile
on the left for a bolt that was on the right-hand side.
And that could happen not just for one,
which would kill the process, but could
happen for many nut-bolt pairs.
So we can't do that.
We cannot use the straightforward divide
and conquer approach.
So this comes to the first interesting question
that we have here, which is, how do we exploit the fact that all
of you are going to help me?
So I'm the disorganized carpenter or handyman.
And I'd like to break this up, first into two piles, such
that I can go off and send one or more of you off
and say, OK, I can guarantee that this is a subproblem,
in the sense that the original problem had
all of the bolts that matched all of the nuts that
were in my pile.
I want to break it up into two problems in--
this is the magic that we have.
We need to figure out this magic--
such that if I give you 50 nuts and 50 bolts,
and I keep 50 nuts and 50 bolts, that you
can solve that problem.
It is a problem.
I mean, there's a matching there, right?
For those 50 nuts, you have the 50 bolts in your pile.
Same thing with me.
So how do I do that?
Yeah, go ahead, Josh.
AUDIENCE: I have a question.
Can you compare the size of nuts to each other?
SRINI DEVADAS: No, you cannot.
Yeah, good question.
So the only thing you can do is you have a nut
and you have a bolt, and if it goes through,
then the nut is bigger.
If it doesn't, then the bolt is bigger.
And then if it fits exactly, you have a match, all right?
Great, so yeah, go ahead, Ganatra.
AUDIENCE: So once you get one to fit,
you can use that to sort of like order the other ones.
Because it fits in perfectly, right, it fits,
so all the ones that-- you just go through all of them,
and if it goes through just the hole,
then you know those are smaller--
well, they're bigger.
And then if it doesn't fit through,
then you know that those--
SRINI DEVADAS: Right, great.
Excellent.
So Ganatra has discovered this notion of pivoting.
And pivoting is essentially something
that is best described here in the animation that gives you
a divide and conquer strategy.
Now, I answered Josh's question, and that was a key question.
And I'm not going to violate the answer to that question
by using some other strategy that does not correspond
to this nut-bolt check, right?
So the only thing I can do in this puzzle
is a nut-bolt check.
But I do get three potential possibilities
whenever I do a nut-bolt check.
I do get the information about a perfect match,
whether the nut is smaller, or whether the nut is bigger.
And that's all I need in order to do pivoting, right?
And so what's going to happen here is I go ahead
and I choose an arbitrary bolt. And I don't even
actually need to find the match before I start this process.
So it's a small variant on what Ganatra said,
but it's really a pretty small variant.
And when I see that this is not a match,
and in fact, the nut is bigger, then I put the nut
on the right-hand side pile.
When I see that the nut is smaller,
then I put the nut in the left-hand side pile.
And I see a match, I just put that aside.
I don't put it in either of the piles,
because I'm going to actually-- it turns out
I'm not going to get a pile of 50 and 50, or in this case,
I'm actually going to get something like three and three,
because I'm going to get one match out of it.
So in general, I'm not necessarily
going to get equal size piles.
So that's actually a little bit of an issue,
and we'll get back to that maybe later.
But I will get a matching, and I'll
get a pile on the left that is the perfect problem,
perfect subproblem.
There I can hand it off to any of you
and you can go off and solve it and it will work.
And I'll get a pile on the right, same thing for that.
So this was a match, but I don't stop.
I don't put the nut in either of the two piles.
It's just going to stay up there.
And then I keep going.
And I make up my right pile and my left pile.
Now, I'm not quite done yet.
What do I do now?
Yeah, back there.
AUDIENCE: You would test every screw into that nut
to see which one--
SRINI DEVADAS: That's right.
So remember, I kept the--
I guess you used a different term here.
You used-- I said bolt, you said screw.
That's fine.
But we're talking about a nut here.
So let's go with the nut.
And this nut that I put aside, I'm
going to now compare that, not with this pivot bolt that I
picked, which is now set aside and never have used that again,
but I'm going to put that aside, and I'm
going to compare each of the bolts with this pivot nut
that I have.
So the pivot bolt was picked and I discover
the associated pivot nut.
And now I'm going to use that pivot nut, which
is the nut that I have up here, the light green one,
and then I'm going to do the same thing.
And all of this does not violate the answer to Josh's question.
And it's going to do exactly the right thing
in terms of giving me two piles that
are beautiful problems that are going to be solvable.
There's nothing that's stopping me from repeating this process.
When we talk about divide and conquer,
usually we talk about recursion, and we
talk about repeatedly doing divide and conquer.
Then I wrote this up here in terms of the N size problem
turned into two N/2 problems.
Well, each of those N/2 problems I
could turn into two N/4 problems,
so I could have four N/4's, and so on and so forth.
And so I could clearly do that.
And especially if N is large, then I
want to get a reduction in comparisons.
And get this to grow, by the way.
And we won't do this analysis.
But rather than growing as N square,
you'd like it to grow as N log N.
And that's kind of the asymptotic analysis
that you'll have to do if you take a class like 6006.
But the general sense that you should take away from this
is that you're going to have a logarithmic number of steps.
And obviously, that doesn't imply a logarithmic number
of comparisons, because the number of subproblems in each
of those steps is doubling.
Initially, you had a problem, then you have two subproblems,
and then you have four subproblems.
Each of them are small in size, but together,
assuming these problems are N/2 and N/2, then
you can get something that's substantially smaller than N
square, namely N log N, OK?
And you can do that numerically.
And I don't want to get into that too much,
but be happy to talk to you about this after lecture
or during office hours.
There's a couple of caveats.
One of them is there's no guarantee here
if I pick a random pivot bolt that I'm actually
going to get N/2 and N/2.
It's going to be one less than that.
Maybe it'd be in over N/2 and N over 2 minus 1.
I could get something-- if I picked a large pivot bolt,
I might get a very skewed pair of piles, right?
One of them could have 80 in them--
80 nuts and bolts in them, and the other one
could have 20 nuts and bolts in them, OK?
So that's something to worry about.
We won't worry about that.
But we'll talk about it when we move
to sorting, which is what we're going to do
in just a couple of minutes.
So people buy the solution to the nuts and bolts puzzle.
Clearly, it's going to give you some efficiency.
If you pick a middling size bolt,
maybe you can eyeball that.
It's a little bit of a violation of what we've talked about,
but you could clearly--
I mean, let's assume that you can make out
the difference between something that's as
thick as this and that finger.
And then you pick something in the middle.
And then you can get two large piles
that are roughly equal in size from the original really large
pile.
And at least at the beginning, you
could probably get, in the context
of this puzzle, piles that are roughly similar in size.
And after that, it doesn't really
matter once you've broken things up.
So N is not 100, but N is more like 5 or 10.
At that point, it doesn't really matter what strategy you use,
because the numbers aren't really different, regardless
of the strategy.
So good, all right.
So I promised you a relationship to sorting.
And obviously, we want to do some programming here.
And it turns out that just like we
had mergesort, which was a divide
and conquer out algorithm, it turns out
this pivoting strategy turns into a strategy for divide
and conquer that is quite different from mergesort,
but is equally applicable to sorting numbers.
And so let me remind you what mergesort was.
And then I'm going to contrast mergesort
with quicksort, which is a pivot-based sorting algorithm.
So forget about nuts and bolts.
We're now down to boring numbers, integers.
And we want to just sort these numbers in ascending order.
So if I have a bunch of numbers and I
want to sort these in ascending order, mergesort would say,
I'm going to go ahead and break it up into halves.
And let's just do one level of recursion.
With all of these things you can always do more,
but these problems aren't large enough
that you really want to do that in this example.
And the important thing is you just broke it into half.
But it was easy.
The split was easy.
You slice the list, splice the list, what have you.
And you assume that somehow you can sort this
in ascending order.
And in this case, you need to go to that.
And then you sort this in ascending order.
So you go like that.
And then you need to do a merge.
And we used what we call the two finger algorithm to do
the merge that essentially says, I'm
going to assume that this array is sorted
and this array is sorted, and I'm
going to put pointers up at the beginning of these two arrays.
And I'm going to do comparisons.
And I'm essentially going to assume
that I have a blank array of size six, blank list of size
six.
And I'm going to be writing into this blank array
the result of the comparison that is, which one is
less in the comparisons.
So I'm going to get minus 31.
And then I compare 0 with minus 4,
and I'm going to get minus 4, 0, et cetera, et cetera.
So that was our merge.
And the important thing to remember
is that our merge algorithm had an easy divide step,
and it had a more difficult step that
corresponded to taking the subarrays that were sorted,
and the work was all in the merge,
putting the things together.
Because the divide, obviously, is just like chop.
Now, as you can see from the nuts and bolts,
division was non-trivial, right?
Division required pivoting.
But there was no real merge after that.
And one side handed off, let's say, to Ganatra
these 50 nuts and bolts after I did the work in pivoting.
And I kept 50 nuts and bolts as well.
We were done.
I mean, maybe I wanted the nuts and bolts back.
I don't want him to run away with it.
But it wasn't like I had to process anything
that he gave me back, right?
If he paired them up, then I didn't
have to process that, right?
So this is mergesort.
The quicksort algorithm, which I'll now describe to you,
is divide and conquer, two-way divide and conquer,
same as mergesort, but it kind of flips the work.
And it does more work up front before the division,
and then this little or no work at the end.
So what happens here is--
the way we're going to think about quicksort
is we're going to choose a pivot.
You can have some array here.
And I can go ahead and just call these a, b, c, d, e, f, g.
And I'm going to choose a pivot.
And in the code that I'm going to show you,
we're just going to go ahead and choose the last element.
We assume that this is in random order.
We're going to choose the last element of this array
as the pivot.
And what did we do with the pivot back in our nuts
and bolts example?
We-- someone, what did we do with the pivot?
Yeah, go ahead.
AUDIENCE: We spliced around it.
SRINI DEVADAS: Yeah, you just basically spliced around it.
You compared it with--
we took the pivot bolt, and in this case of the puzzle,
you compared it with the nuts.
Here we don't have nuts and bolts, we just have numbers.
And so you're going to take that pivot
and you're to start comparing it with the other numbers.
And you're exactly right in that we're going to get to the point
where we have to splice round it,
which isn't your standard Python splicing,
because that requires contiguous locations.
But what you want to do is you want to get to a situation
where you have something like this, where you have
g somewhere in the middle and all elements
less than g are to the left.
And let's assume they're all unique elements.
All elements greater than g are to the right, OK?
So this is exactly pivoting around g.
So it's referred to as pivoting around g.
Now, the nice thing here, when you do this,
is that you can go off and you can fix the location of g.
In fact, g's location, just like the pair,
the nut-bolt pair was corresponding to the pivot nut,
and the pivot bolt was determined during the pivoting
step.
And you never had to check whether--
you never had to discover that pair again.
You can put that pair aside.
The location of g, the pivot chosen,
in the final sorted array, is determined
by this pivoting step, OK?
That makes sense?
And so this location, whatever that index is,
it might be right in the middle.
It might be a little bit skewed.
But that location is determined.
And if you did this with roughly equal piles, if you will,
or equal sides, if there were 100 elements here,
you might see 50-odd here and 40-odd over there.
Now, of course, if g happened to be the largest number,
then g would be all the way to the right.
So there's that to worry about.
But this is the divide step.
So the divide step is the pivoting step.
And then you can go off and sort each of those sublists.
Because those less than g are not
necessarily in ascending order.
You just dumped them.
There's still work to be done.
And then the greater than g, likewise.
They're not necessarily in ascending order.
But you know that there's 42 elements corresponding
to less than g, and you can go sort that array.
And those are going to be the first 42
elements of your final array.
And the g is going to be at the 43rd position.
And then the ones that are greater than g are going
to be to the right of that.
All make sense?
Yeah, go ahead, Fadi.
AUDIENCE: So like if we were very unlucky and at each step,
we just picked the largest number, the largest
number, then it's
got to be N squared, correct?
SRINI DEVADAS: That's right.
So that's pathological.
Now, it turns out that, amazingly, in order
to avoid that situation when people
use quicksort in practice, which is essentially this algorithm
that I describe, they take the array that they're given
and they actually randomize it.
They actually make it--
they shake it up.
It's like you get these nuts and bolts,
and you want to close your eyes and you
want to pick up a nut that's middling in size,
and maybe all the small ones are up at the top
and the big ones are down at the bottom.
You want things in the middle, so you
go, you shake, shake, shake.
You randomize.
And then go stick your hand in About.
Halfway through the pile and pick up a nut,
or pick up a bolt, right?
That's kind of what happens here.
So quicksort, it turns out--
and I'll just say this, and if this doesn't make sense,
ask me later.
But it should give you some intuition.
The randomized quicksort, where you have this random input,
and you have some probabilistic guarantee that you're
picking a pivot that is middling in size in terms of an integer
size, is going to be N log N in complexity.
But the worst case complexity of quicksort
is exactly as you said, it's N square.
But that's really not something that you
need to understand deeply to understand, well,
either the puzzle or the rest of this lecture.
You do get a sense of--
as long as you have a sense of if you pick something
in the middle and these piles are roughly equal in size,
I'm going to get some improvement.
The guarantee that the piles are roughly
equal in size, all the way down to the depths of recursion,
is a difficult one to achieve in a deterministic way.
But it's not hard to achieve in a probablistic way
by doing this randomization.
You had another question?
Yeah.
AUDIENCE: So then why do we use this more than mergesort,
which is like--
SRINI DEVADAS: Ah, that's exactly-- so
that is the rest of the lecture, which is only 10 minutes left.
But OK, so that's the rest of the lecture.
So one of the things that I said here,
I said a blank list of size six.
So what mergesort requires, in order for mergesort
to be efficient, you needed auxiliary storage
that corresponded to the size of the list.
Because right at the first level of recursion,
when you got two arrays that were N/2 and N/2,
and they were both sorted, and you had to merge the two
together to create the final result corresponding
to the sorted array, you ended up
requiring storage that was N in size
in order to actually do this two finger algorithm
and compare the minus 31's with the zeros
and then write them into this array.
Now, the obvious way of taking this and going to here
is code that I'm going to show you that is easy to write.
And it requires a blank list as well.
I mean, the easy way to do that--
and I'll just show you the code, because it's easier
to show you the code as opposed to waving my hands here
and potentially confusing people.
But let me show you first the quicksort divide and conquer,
which is absolutely trivial.
I mean, just like mergesort was.
But pivot partition is this step here.
This is the pivoting step.
And this is what that procedure does.
So we'll get to that in just a second.
But the rest of it is simply I'm going to go ahead,
and once I do the pivot partition,
I have something that looks like this.
And then I just go off and run quicksort on this
and quicksort on that.
And I'm doing that on the array itself.
It's just like I have this array,
and I'm just saying I'm going to call quicksort
with the indices that are associated
with the beginning and the end of this array.
And then I'm going to call quicksort
with the indices that begin with the beginning and the end
of this array.
And that same array is going to be my final output, right?
So everything is done--
the term that's used here is "in place."
So in place sorting and insertion
sort, which is also N square, and a few other things
are in place sorting algorithms, which essentially say, look,
I need to sort all of you in some way,
perhaps by age or something.
And I don't want another room, right?
I don't want to say, who's the youngest here,
and then go over to that room, and then
who's the next youngest and go over to the room, et cetera.
I just want you to start swapping positions here.
And we're all going to be in this room,
and somehow we're going to get sorted.
So that's in place.
And when I have an array that corresponds to integers,
I don't want another blank list or blank array
and use that storage.
And so that answers your question
at the top level as to why quicksort is used.
And it's used because the memory requirements of quicksort
are substantially less than the memory
requirements of mergesort.
And in fact, you can do this pivoting step
using one integer as storage.
You need the original array, of course,
but you needed that to store all of the numbers.
And the auxiliary storage, the extra storage that you need,
is exactly the storage for one integer
where you store the pivot, OK?
And I'm going to show you the naive way, which
is the regular quicksort.
And that is a trivial pivot partition.
It's not the clever one, which essentially says,
look, I'm going to go ahead and I'm
going to create a new array.
In fact, I have two arrays that are less and more,
or two lists.
Together the sizes of less and more
are going to be the size of the original, maybe minus 1.
I'm adding to less and adding to more,
depending on whether the element is less than the pivot
or greater than the pivot.
And then I'm good.
So there's that algorithm, which would just be, oh yeah,
I'm going to create a new list here.
And I'm going to compare a to g.
And then I'm going to compare b to g, et cetera, et cetera.
So this does not give you anything.
This is an uninteresting algorithm.
I'm sorry.
Implementation-- this is an uninteresting implementation,
OK?
It is much more--
you need to be much more clever--
I don't need this anymore--
to do this in place.
So what I want to do is I'm going to take an example here.
I want to take this array.
And I'm going to not talk about the recursive
sorting or anything like that, because that's easy.
We kind of know how to do that.
What I want to do is I want to choose this as the pivot.
And I want to translate that somehow
into the final pivoted output, which
is going to be 0, minus 31, 1, 2, 65, 99.
Oops, I have something wrong here.
So this should be the other way around.
83, right?
OK.
Mm, bug.
Oh, I'm sorry, I'm sorry.
This is fine.
I just need it--
I don't need it to be sorted.
OK, good.
Whew.
All right.
I just thought I found the first bug in my book.
So there's probably many bugs in the book,
but I don't want to know.
So I don't need this to be sorted.
Yeah, so there's a 0, a minus 31.
So the key is that I've discovered this.
And everything to the left of that is less than 1.
And everything to the right of that is greater than 1.
And that's the only requirement that I have.
And as I said, I mean, I said this to begin with
and then I forgot, I said we're not going
to worry about the recursion.
So clearly, I have to do more work here
with respect to taking this set of numbers
and sorting them in ascending order, et cetera.
But what I've done here is I know that 1 is fixed in place,
and 1 will never move.
And I just have to turn this into minus 31 and 0.
And I have to do some reordering here.
But 1 is fixed in place.
Now, the algorithm that I'm going to show you,
I'm going to show you the code.
And it's probably code that you won't
be able to parse, at least in minutes or seconds.
And this code is in place pivoting.
It's clever code.
And what it does is it has one additional variable
worth of storage that we're going to call the pivot.
So you see the variable pivot there.
And just using that one additional integer storage,
it manages to transform this array using a linear number
of steps into this array.
So you can see that this is non-trivial.
And I need to move from here to there without having
this extra storage.
If you had extra storage, it would be easy.
We know how to do that.
But if you didn't, you need this code
that has an outer while loop.
You see while not done.
And then it's got two inner while loops in it.
And it's got a couple of counters, couple of pointers
I should say, to indices.
And you go left and right, and you magically get this answer.
But there's no magic here, because we have the code up
and we can run it.
And computers aren't smart, so clearly,
that code is doing something clever,
and we just need to understand that, all right?
So I'm going to--
in the couple of minutes I have left,
this is the last thing I want to do
is I want to tell you how this code works.
And it's really pretty code.
And it's clever code.
So what I have is I have two pointers, top and bottom.
So I'm going to, essentially, say that top is--
bottom is 0.
Initially, it's minus 1, but we go ahead and increment it.
And top is 8.
And so how many--
I have 9-- 1, 2, 3, 4, 5, 6, 7, 8, 9.
So top is pointing to the pivot.
So this is pointing to pivot, which top is 8.
And bottom is pointing to the first element.
So these two while loops are essentially
going from the left of the array and going this way
from the right of the array.
So what we're going to do--
and don't worry about exactly how the code does this.
I'm going to show you the way this array is transformed
and the way these pointers are changed.
And then you'll get a sense of how this code works.
And obviously, this transformation
happens in one of those while loops.
So we have the pivot corresponding to 1.
And bottom equals 0 and top equals 8.
What I'm going to do is I'm going
to start moving leftward from--
this would be the second inner while loop.
I'm going to start moving--
oh, I'm sorry.
I'll start with bottom.
I'm going to increment-- bottom is initially minus 1.
I increment it to 0.
And I start moving rightward from the left of the list
and try and find an element that is greater than 1.
And that is immediate.
I realize that 4 is greater than 1, OK?
So when I realize that 4 is greater than 1,
I'm going to copy over 4.
And this is doesn't mean that I'm copying the entire array.
I'm just going to copy over 4.
But I'm writing what the array looks like, because that's
important, over to here.
And you might say--
pivot equals 1.
You might say, oh, but I overwrote the location 1.
And I'm like, don't worry about it.
I do have 1 stored in my pivot.
So my pivot has 1 in it.
So I haven't lost anything here.
It's not like I threw away any locations.
I do have that extra location.
So I did this rightward move going from the left.
Now I'm going to go leftward from the right.
And I'm going to look for something that is less than 1
and I'm going to try and move it.
So basically, what this algorithm does
is it tries to find things that are--
if a is greater than g, it moves a to the rightmost part
of the array.
And the same thing, if d is less than g,
then it moves it to the left part of the array.
So depending on whether the comparison to g
is greater or less, you want to end up
in the edges of the array.
And you're going to try and get the edges of the array
to be correct in the sense that they have elements
on the left that are less than the pivot,
and the elements on the right that
are greater than the pivot.
And you're trying to get to the middle.
And when your two pointers converge,
your top and bottom pointers converge, you're done.
So let's do a couple more steps and close this lecture.
So I did the right step.
Now I'm going rightward.
And now I'm going to go leftward.
What is the first element that is less than 1--
no, less, than 1.
0, right?
Yes, but I'm going this way.
AUDIENCE: oh!
SRINI DEVADAS: Yeah, I'm going--
I'm glad you pointed that out.
I needed to make sure.
So I went rightward the first time,
and I want to go leftward this time.
So this would be-- so I come here and I see 0.
When I see 0, what I'm going to do
is I'm going to transform the array.
I'll write this out to make sure I get this right.
And so right now I have 4, 65 to minus 31, 0, et cetera.
And I'm going to go this way, and I'm going to put--
when I see the 0, I'm going to copy over 0 over here
and leave the 0 in here for a second.
And there's no problem here, because 4 was copied over here.
And so now I overwrote 4 with 0.
OK, now you kind of see, maybe get some sense
of how this algorithm works.
This is the first interesting step where I took 0,
and because 0 is less than 1, as I mentioned,
I want to jam it all the way to the left.
Just put it all the way to the left.
And I'm cool with losing 4, because 4 is being put
all the way to the right, OK?
Now, at this point, my bottom and top
are going to get modified.
My bottom is still 0, but the top is 4,
because I moved-- top was 8, and I decremented top all the way
to the point where it became 4.
And I realized that 0 was less than 1.
And then I copied over that value over to the bottom.
You can kind of see the code up there.
This is actually an output of that code.
And I put a 0 in here.
So that's what my counters look like.
And it's only a couple of more steps.
We're going to go now go right.
And I'm going to go do this again, except that 0 is already
taken care of, so now I see 65.
Clearly I'm going to have a situation where
if I go this way, 65 is greater than the pivot,
so I'm going to go 0, 65, 2, minus 31.
And because 65 is greater than the pivot,
I'm going to write it into what the top was, which obviously
got copied over that way.
So I'm going to have 65 in here, 99, 83, 782, and 4.
So now after this step, bottom equals 1.
And the reason for that was 65 was in the location on 1,
and top equals 4.
And then the next one, someone want
to tell me what the next one is going to be?
If I go leftward.
I'm going to start from here, top was 4.
So I was pointing at 65.
And I'm looking for something that when I go this way,
I saw 65 greater than 1.
Now I'm looking for something that is less than 1.
And minus 31 is less than 1, correct?
When I go this way, I'm looking for something that's less than.
So given that it's minus 31, I'm going to copy over minus 31
to what top was pointing to.
So I have minus 31, 2, minus 31, 65, 99, 28, 782, and 4.
And at this point, I'm going to have bottom
equals 1 and top equals 3.
Keep going one more, and I get to the point where I see 2.
2 is greater than 1.
0 and minus 31 are less than 1.
And so I take 2 and I copy it over here.
And now at this point, when I do an increment,
I realize that bottom is 2 and top is 3.
And in the very next step, bottom will equal top.
And I'm sitting up here, and so I copy--
I make two--
I'd write the pivot into the location that
corresponds to the bottom after top got decremented.
And in that case, both bottom and top are 2.
And that index, 0, 1, and 2, the pivot gets written into it.
And so that's exactly what I had,
which I wrote right at the beginning.
So I would say this is probably the most complicated code
from a control flow standpoint that I've ever shown you.
This algorithm is an in place algorithm that does not
require any extra storage.
And that's exactly why it's so popular.
People are sorting billions of elements in lists,
and you can't use gigabytes of storage
during the sorting process.
And what this clever strategy tells you is you
don't need all of that storage.
You can do things in place.
And as long as you choose the pivot reasonably well,
you get your average case N log N complexity,
which is the same as the worst case complexity of mergesort.
So you've got to be a little bit careful with that.