1
00:00:08,000 --> 00:00:11,000
OK.
Today we are going to talk
2
00:00:11,000 --> 00:00:18,000
about a very interesting
algorithm called Quicksort --
3
00:00:23,000 --> 00:00:30,000
-- which was invented by Tony
Hoare in 1962.
4
00:00:30,000 --> 00:00:35,000
And it has ended up being a
really interesting algorithm
5
00:00:35,000 --> 00:00:40,000
from many points of view.
And because of that,
6
00:00:40,000 --> 00:00:47,000
it turns out today's lecture is
going to be both hard and fast.
7
00:00:47,000 --> 00:00:52,000
If you see the person next to
you sleeping,
8
00:00:52,000 --> 00:00:56,000
you will want to say let's get
going.
9
00:00:56,000 --> 00:01:01,000
It's a divide-and-conquer
algorithm.
10
00:01:06,000 --> 00:01:08,000
And it sorts,
as they say,
11
00:01:08,000 --> 00:01:14,000
in place, meaning that it just
rearranged the elements where
12
00:01:14,000 --> 00:01:17,000
they are.
That is like insertion sort
13
00:01:17,000 --> 00:01:20,000
rearranges elements where they
are.
14
00:01:20,000 --> 00:01:24,000
Mergesort does not.
Mergesort requires extra
15
00:01:24,000 --> 00:01:30,000
storage in order to do the merge
operation.
16
00:01:30,000 --> 00:01:34,000
To merge in linear time and
place, it doesn't merge in place
17
00:01:34,000 --> 00:01:37,000
in linear time.
It doesn't do it just by
18
00:01:37,000 --> 00:01:41,000
rearranging.
It is nice because it is in
19
00:01:41,000 --> 00:01:45,000
place, so that means that it is
fairly efficient in its use of
20
00:01:45,000 --> 00:01:48,000
storage.
And it also happens to be very
21
00:01:48,000 --> 00:01:53,000
practical if you tune it a bit.
The basic algorithm turns out,
22
00:01:53,000 --> 00:01:58,000
if you just implement that,
it's not necessarily that
23
00:01:58,000 --> 00:02:02,000
efficient.
But if what you do was then do
24
00:02:02,000 --> 00:02:07,000
the standard kinds of things you
do to goose up the runtime of
25
00:02:07,000 --> 00:02:12,000
something, and we'll talk a
little about what those things
26
00:02:12,000 --> 00:02:16,000
are, then it can be very,
very practical.
27
00:02:16,000 --> 00:02:20,000
So, it uses divide-and-conquer
paradigm.
28
00:02:33,000 --> 00:02:40,000
First step is divide.
And to do this basically it
29
00:02:40,000 --> 00:02:48,000
does it by partitioning.
So, it partitions the input
30
00:02:48,000 --> 00:02:59,000
array into two subarrays around
an element we call the pivot --
31
00:03:04,000 --> 00:03:14,000
-- such that elements in the
lower subarray are less than or
32
00:03:14,000 --> 00:03:25,000
equal to x, are less than or
equal to elements in the upper
33
00:03:25,000 --> 00:03:31,000
subarray.
If we draw a picture of the
34
00:03:31,000 --> 00:03:38,000
input array, this partition step
basically takes some element x
35
00:03:38,000 --> 00:03:45,000
and everything over here is less
than or equal to x after the
36
00:03:45,000 --> 00:03:52,000
partition step and everything
over here is greater than or
37
00:03:52,000 --> 00:03:57,000
equal to x.
And so now the conquer step is
38
00:03:57,000 --> 00:04:04,000
pretty easy.
You just recursively sort the
39
00:04:04,000 --> 00:04:10,000
two subarrays.
So, I recursively sort the
40
00:04:10,000 --> 00:04:19,000
elements less than or equal to
x, I recursively sort the
41
00:04:19,000 --> 00:04:24,000
elements greater than or equal
to x.
42
00:04:24,000 --> 00:04:32,000
And then combine is then just
trivial.
43
00:04:32,000 --> 00:04:36,000
Because once I have sorted the
things less than or equal to x,
44
00:04:36,000 --> 00:04:40,000
then sorted the things greater
than or equal to x,
45
00:04:40,000 --> 00:04:44,000
the whole thing is sorted.
So, there is nothing to do
46
00:04:44,000 --> 00:04:48,000
really for the combine.
The key step in quicksort is
47
00:04:48,000 --> 00:04:52,000
this partition step.
That is the thing that does all
48
00:04:52,000 --> 00:04:55,000
of the work.
And so you can view quicksort
49
00:04:55,000 --> 00:04:57,000
of just as recursive
partitioning.
50
00:04:57,000 --> 00:05:06,000
That's all it is.
Just as mergesort was recursive
51
00:05:06,000 --> 00:05:18,000
merging, quicksort sort of goes
the other way around and does
52
00:05:18,000 --> 00:05:28,000
recursive partitioning.
The key is the linear time,
53
00:05:28,000 --> 00:05:40,000
by which I mean theta n,
partitioning subroutine.
54
00:05:40,000 --> 00:05:43,000
And here are some pseudocode
for it.
55
00:05:43,000 --> 00:05:47,000
This is actually slightly
different from the book.
56
00:05:47,000 --> 00:05:51,000
The book has one.
In fact, there is a nice
57
00:05:51,000 --> 00:05:55,000
problem in the book that has
even a different one,
58
00:05:55,000 --> 00:06:00,000
but they are all basically the
same idea.
59
00:06:00,000 --> 00:06:01,000
Partition (A,
p, q).
60
00:06:01,000 --> 00:06:06,000
And what we are looking at,
at this step of the recursion,
61
00:06:06,000 --> 00:06:11,000
is the subarray A from p to q.
And basically we pick a pivot,
62
00:06:11,000 --> 00:06:17,000
which is we are going to just
pick as the first element of the
63
00:06:17,000 --> 00:06:19,000
array A of p.
64
00:06:26,000 --> 00:06:29,000
And the book,
just for your information,
65
00:06:29,000 --> 00:06:32,000
uses A of q.
I use A of p.
66
00:06:32,000 --> 00:06:37,000
It doesn't really matter.
And then we set an index to p
67
00:06:37,000 --> 00:06:40,000
and then we have a loop.
68
00:07:35,000 --> 00:07:40,000
This is the code.
Basically the structure of it
69
00:07:40,000 --> 00:07:47,000
is a for loop with an "if"
statement in the middle.
70
00:07:47,000 --> 00:07:54,000
And so the structure of the
algorithm of this partitioning
71
00:07:54,000 --> 00:08:00,000
step looks as follows.
We set the pivot to be the
72
00:08:00,000 --> 00:08:05,000
first element.
Here is p and here is q.
73
00:08:05,000 --> 00:08:10,000
This is going to be our
invariant for the loop.
74
00:08:10,000 --> 00:08:15,000
And, at any time during the
execution of a loop,
75
00:08:15,000 --> 00:08:22,000
I essentially have some values
up to i which are already less
76
00:08:22,000 --> 00:08:28,000
than or equal to x and then some
values that end at j minus 1
77
00:08:28,000 --> 00:08:34,000
that are greater than or equal
to x.
78
00:08:34,000 --> 00:08:38,000
And then I don't know about the
rest.
79
00:08:38,000 --> 00:08:44,000
And so we start out with i
equal to p and j equal to p plus
80
00:00:01,000 --> 00:08:47,000
It starts out at p plus 1 so
This is called,
So, once again,
81
00:08:47,000 --> 00:08:52,000
that everything is unknown
except for x here.
82
00:08:52,000 --> 00:08:58,000
And then the idea is that it is
going to preserve this
83
00:08:58,000 --> 00:09:02,000
invariant.
And the way it does it is,
84
00:09:02,000 --> 00:09:06,000
as we go through the loop,
it looks at a of j and says is
85
00:09:06,000 --> 00:09:10,000
it greater than or equal to x?
Sorry, is it less than or equal
86
00:09:10,000 --> 00:09:12,000
to x?
If it is greater than or equal
87
00:09:12,000 --> 00:09:15,000
to x it does nothing,
because what can happen?
88
00:09:15,000 --> 00:09:19,000
If this is greater than or
equal to x, essentially it just
89
00:09:19,000 --> 00:09:23,000
goes to the next iterational
loop which moves this boundary
90
00:09:23,000 --> 00:09:27,000
and the invariant is satisfied.
Does everybody see that?
91
00:09:27,000 --> 00:09:31,000
Yeah, OK.
But if it is less than or equal
92
00:09:31,000 --> 00:09:35,000
to x, I have got a problem if I
want to maintain the invariant
93
00:09:35,000 --> 00:09:38,000
if this next element is less
than or equal to x.
94
00:09:38,000 --> 00:09:43,000
And so what it does then is it
says oh, let me just move this
95
00:09:43,000 --> 00:09:47,000
boundary and swap this element
here, which is greater than or
96
00:09:47,000 --> 00:09:51,000
equal to x, with this one here
that is less than or equal to x,
97
00:09:51,000 --> 00:09:55,000
thereby increasing the size of
this subarray and then the
98
00:09:55,000 --> 00:09:59,000
invariant is satisfied again.
It is a fairly simple
99
00:09:59,000 --> 00:10:03,000
algorithm.
And it is actually a very tight
100
00:10:03,000 --> 00:10:08,000
and easy algorithm.
That is one reason that this is
101
00:10:08,000 --> 00:10:13,000
such a great piece of code
because it is very efficient.
102
00:10:13,000 --> 00:10:18,000
Now, in principle,
the running time for this on n
103
00:10:18,000 --> 00:10:21,000
elements is order n.
104
00:10:28,000 --> 00:10:31,000
Because I am basically just
going through the n elements and
105
00:10:31,000 --> 00:10:35,000
just doing a constant amount of
work and then just a constant
106
00:10:35,000 --> 00:10:39,000
amount of work outside.
This is a clever piece of code.
107
00:10:39,000 --> 00:10:41,000
In fact, in principle partition
is easy, right?
108
00:10:41,000 --> 00:10:44,000
If I weren't worrying about
doing it in place,
109
00:10:44,000 --> 00:10:47,000
it is really a pretty easy
thing to do.
110
00:10:47,000 --> 00:10:50,000
I take an element and just
compare every other element with
111
00:10:50,000 --> 00:10:52,000
it.
I throw one into one bin and
112
00:10:52,000 --> 00:10:57,000
one into the other.
That is clearly linear time.
113
00:10:57,000 --> 00:11:01,000
But often what you find is that
just because you can do it that
114
00:11:01,000 --> 00:11:04,000
way theoretically doesn't mean
that that is going to end up
115
00:11:04,000 --> 00:11:08,000
giving you good code.
And this is a nice piece of
116
00:11:08,000 --> 00:11:10,000
code that allows you to do it in
place.
117
00:11:10,000 --> 00:11:14,000
And that is one reason why this
is a particularly good
118
00:11:14,000 --> 00:11:16,000
algorithm, because the constants
are good.
119
00:11:16,000 --> 00:11:20,000
So, yes, when we do asymptotic
analysis we tend to ignore the
120
00:11:20,000 --> 00:11:24,000
constants, but when you're
actually building code you care
121
00:11:24,000 --> 00:11:30,000
about the constants.
But first you care much more
122
00:11:30,000 --> 00:11:37,000
than just about the constants,
is whether overall it is going
123
00:11:37,000 --> 00:11:43,000
to be a fast algorithm.
Let's go through an example of
124
00:11:43,000 --> 00:11:50,000
this, I guess I will do it over
here, just so we get the gist.
125
00:11:50,000 --> 00:11:57,000
Here is a sample array that I
have created out of hallcloth.
126
00:11:57,000 --> 00:12:05,000
And here we are going to set x,
the pivot, to be 6.
127
00:12:05,000 --> 00:12:08,000
Let's look to see how this
algorithm works.
128
00:12:08,000 --> 00:12:12,000
So, i starts out here and j
starts out here if we
129
00:12:12,000 --> 00:12:15,000
initialize.
And what we do is start
130
00:12:15,000 --> 00:12:18,000
scanning right,
essentially that code is
131
00:12:18,000 --> 00:12:23,000
scanning right until it gets
something which is less than or
132
00:12:23,000 --> 00:12:27,000
equal to the pivot.
It keeps going here until it
133
00:12:27,000 --> 00:12:32,000
finds, j keeps incrementing
until it finds something that is
134
00:12:32,000 --> 00:12:38,000
less than or equal to the pivot.
And, in that case,
135
00:12:38,000 --> 00:12:44,000
it is the number 5.
Then it says we will swap these
136
00:12:44,000 --> 00:12:49,000
two things.
And it does that and we get 6,
137
00:12:49,000 --> 00:12:52,000
5, 13, 10, 8,
3, 2, 11.
138
00:12:52,000 --> 00:12:58,000
And meanwhile now i gets
incremented and j continues
139
00:12:58,000 --> 00:13:05,000
where it left off.
And so now we keep scanning
140
00:13:05,000 --> 00:13:12,000
right until we get to something
that is less than or equal to
141
00:13:12,000 --> 00:13:16,000
the pivot.
In this case it is 3.
142
00:13:16,000 --> 00:13:20,000
We swap 3 and 5 and get 6,
3, etc.
143
00:13:20,000 --> 00:13:27,000
And now, at this step we
increment i, we start j out
144
00:13:27,000 --> 00:13:30,000
here.
And in this case,
145
00:13:30,000 --> 00:13:35,000
right off the bat,
we have something which is less
146
00:13:35,000 --> 00:13:39,000
than or equal to x,
so we swap these two.
147
00:13:39,000 --> 00:13:42,000
I blew it, didn't I?
Oops.
148
00:13:42,000 --> 00:13:46,000
What did I do?
I swapped the wrong thing,
149
00:13:46,000 --> 00:13:48,000
didn't I, here?
Ah-ha.
150
00:13:48,000 --> 00:13:51,000
That is why I am not a
computer.
151
00:13:51,000 --> 00:13:54,000
Good.
We should have swapped this
152
00:13:54,000 --> 00:13:57,000
guy, right?
Swapped i plus 1,
153
00:13:57,000 --> 00:14:01,000
right?
This was i.
154
00:14:01,000 --> 00:14:03,000
We swap i plus 1,
good.
155
00:14:03,000 --> 00:14:09,000
So, that's all wrong.
Let's swap the right things.
156
00:14:09,000 --> 00:14:12,000
Now we have 6,
5, 3, 10, 8,
157
00:14:12,000 --> 00:14:16,000
13, 2, 11.
That even corresponds to my
158
00:14:16,000 --> 00:14:22,000
notes for some strange reason.
This is i and now this is j.
159
00:14:22,000 --> 00:14:28,000
And now when I look,
I immediately have something
160
00:14:28,000 --> 00:14:34,000
that is less than or equal to
the pivot.
161
00:14:34,000 --> 00:14:41,000
We swap this and i plus 1,
so now we have 6,
162
00:14:41,000 --> 00:14:45,000
5, 3, 2, 8, 13,
10, 11.
163
00:14:45,000 --> 00:14:52,000
And we, at that point,
increment i to here.
164
00:14:52,000 --> 00:15:03,000
And we have j now going here
and j runs to the end.
165
00:15:03,000 --> 00:15:08,000
And the loop terminates.
When the loop terminates there
166
00:15:08,000 --> 00:15:14,000
is one less swap that we do,
which is to put our pivot
167
00:15:14,000 --> 00:15:18,000
element in the middle between
the two subarrays.
168
00:15:18,000 --> 00:15:25,000
Here we swap this one and this
one, and so that gives us then
169
00:15:25,000 --> 00:15:27,000
2, 5, 3, 6, 8,
13, 10, 11.
170
00:15:27,000 --> 00:15:33,000
And this is the pivot.
And everything over here is
171
00:15:33,000 --> 00:15:40,000
less than or equal to the pivot.
And everything over here is
172
00:15:40,000 --> 00:15:44,000
greater than or equal to the
pivot.
173
00:15:56,000 --> 00:15:59,000
OK, so the quicksort routine.
Once we have this partition
174
00:15:59,000 --> 00:16:03,000
routine, quicksort is a pretty
easy piece of code to write.
175
00:16:18,000 --> 00:16:21,000
I should have said return here
i, right?
176
00:16:21,000 --> 00:16:24,000
You have got to return with the
pivot.
177
00:16:24,000 --> 00:16:29,000
Here I have got to return i
because we want to know where
178
00:16:29,000 --> 00:16:33,000
the pivot element is.
Sorry.
179
00:16:33,000 --> 00:16:46,000
I will plug in my code.
r gets partition of (A,
180
00:16:46,000 --> 00:17:03,000
p, q) and then we quicksort (A,
p, r-1) and quicksort of (A,
181
00:17:03,000 --> 00:17:09,000
r+1, q).
And that is it.
182
00:17:09,000 --> 00:17:17,000
That's the code.
The initial call is quicksort
183
00:17:17,000 --> 00:17:24,000
of (A, 1, n).
Because once we partitioned,
184
00:17:24,000 --> 00:17:31,000
we just have to quicksort the
two portions,
185
00:17:31,000 --> 00:17:39,000
the left and right portions.
Just the boundary case is
186
00:17:39,000 --> 00:17:41,000
probably worth mentioning for a
second.
187
00:17:41,000 --> 00:17:44,000
If there are zero or one
elements, that is basically what
188
00:17:44,000 --> 00:17:47,000
can possibly happen here,
is that I get zero or one
189
00:17:47,000 --> 00:17:49,000
elements here.
Then the point is there is
190
00:17:49,000 --> 00:17:52,000
nothing to do because the array
is sorted, either because it is
191
00:17:52,000 --> 00:17:56,000
an empty array or because it
only has one element.
192
00:17:56,000 --> 00:17:59,000
One of the tricks to making
quicksort go fast,
193
00:17:59,000 --> 00:18:01,000
as one tunes this,
is to, in fact,
194
00:18:01,000 --> 00:18:04,000
look at having a special
purpose sorting routine for
195
00:18:04,000 --> 00:18:07,000
small numbers of elements.
For example,
196
00:18:07,000 --> 00:18:10,000
if you get down to five
elements having some straight
197
00:18:10,000 --> 00:18:14,000
line piece of code that knows
how to sort five elements
198
00:18:14,000 --> 00:18:17,000
sufficiently as opposed to
continuing to go through
199
00:18:17,000 --> 00:18:20,000
recursion in order to accomplish
that.
200
00:18:20,000 --> 00:18:22,000
And there are a variety of
other things.
201
00:18:22,000 --> 00:18:27,000
This is a tail recursive code,
and so you can use certain tail
202
00:18:27,000 --> 00:18:31,000
recursion optimizations.
And there are a variety of
203
00:18:31,000 --> 00:18:35,000
other kinds of optimizations
that you can use to make this
204
00:18:35,000 --> 00:18:37,000
code go fast.
So, yeah, you can tune it up a
205
00:18:37,000 --> 00:18:40,000
bit beyond what is there,
but the core of it is this
206
00:18:40,000 --> 00:18:43,000
efficient partitioning routine.
207
00:18:53,000 --> 00:18:58,000
That is the algorithm.
It turns out that looking at
208
00:18:58,000 --> 00:19:05,000
how fast it runs is actually a
little bit challenging.
209
00:19:05,000 --> 00:19:09,000
In the analysis,
we are going to assume that all
210
00:19:09,000 --> 00:19:13,000
elements are distinct.
It turns out that this
211
00:19:13,000 --> 00:19:19,000
particular code does not work
very well when you have repeated
212
00:19:19,000 --> 00:19:25,000
elements, but Hoare's original
partitioning routine is actually
213
00:19:25,000 --> 00:19:31,000
more efficient in that case if
there are duplicates in what you
214
00:19:31,000 --> 00:19:36,000
are sorting.
And I encourage you to look at
215
00:19:36,000 --> 00:19:38,000
that.
It has a much more complicated
216
00:19:38,000 --> 00:19:42,000
invariant for partitioning
routine, but it does a similar
217
00:19:42,000 --> 00:19:45,000
kind of thing.
It's just a bit more
218
00:19:45,000 --> 00:19:48,000
complicated.
If they weren't all distinct,
219
00:19:48,000 --> 00:19:52,000
there are things you can do to
make them distinct or you can
220
00:19:52,000 --> 00:19:56,000
just use this code.
The easiest thing to do is just
221
00:19:56,000 --> 00:20:00,000
use Hoare's original code
because that works pretty well
222
00:20:00,000 --> 00:20:07,000
when they are nondistinct.
But this is a little bit easier
223
00:20:07,000 --> 00:20:12,000
to understand.
Let's let T(n) be the
224
00:20:12,000 --> 00:20:18,000
worst-case running time on n
elements.
225
00:20:18,000 --> 00:20:27,000
And so what is the worse-case?
What is the worse-case going to
226
00:20:27,000 --> 00:20:31,000
be for quicksort?
227
00:20:40,000 --> 00:20:43,000
That's right.
If you always pick the pivot
228
00:20:43,000 --> 00:20:47,000
and everything is greater than
or everything is less than,
229
00:20:47,000 --> 00:20:50,000
you are not going to partition
the array very well.
230
00:20:50,000 --> 00:20:54,000
And when does that happen?
What does the original input
231
00:20:54,000 --> 00:20:57,000
look like that makes that
happen?
232
00:20:57,000 --> 00:21:01,000
If it is already sorted or
reverse sorted.
233
00:21:01,000 --> 00:21:05,000
So, if the input is sorted or
reverse sorted.
234
00:21:05,000 --> 00:21:09,000
That is actually kind of
important to understand,
235
00:21:09,000 --> 00:21:14,000
because it turns out the most
common thing to sort is
236
00:21:14,000 --> 00:21:19,000
something that is already
sorted, surprisingly,
237
00:21:19,000 --> 00:21:22,000
or things that are nearly
sorted.
238
00:21:22,000 --> 00:21:27,000
But often it is just sorted and
somebody wants to make sure it
239
00:21:27,000 --> 00:21:33,000
is sorted.
Well, let's just sort it again
240
00:21:33,000 --> 00:21:38,000
rather than checking to see if
it is sorted.
241
00:21:38,000 --> 00:21:43,000
And, in those cases,
one side of the partition of
242
00:21:43,000 --> 00:21:50,000
each partition has no elements.
Then we can write out what the
243
00:21:50,000 --> 00:21:54,000
recursion is for that.
We have T(n).
244
00:21:54,000 --> 00:22:00,000
If one side has no elements,
we are going to have T(0) on
245
00:22:00,000 --> 00:22:06,000
that side.
And on the other side we are
246
00:22:06,000 --> 00:22:11,000
going to have T(n-1).
We are just writing out the
247
00:22:11,000 --> 00:22:16,000
recursion for this.
One side has no elements.
248
00:22:16,000 --> 00:22:19,000
The other side has n-1
elements.
249
00:22:19,000 --> 00:22:26,000
And then partitioning and all
the bookkeeping and so forth is
250
00:22:26,000 --> 00:22:30,000
order n.
What is T(0)?
251
00:22:30,000 --> 00:22:35,000
What is T(0)?
What is that asymptotically?
252
00:22:35,000 --> 00:22:38,000
It's a constant,
order 1.
253
00:22:38,000 --> 00:22:44,000
That is just order 1 + T(n-1) +
order n.
254
00:22:44,000 --> 00:22:50,000
Well, the order 1 can be
absorbed into the order n,
255
00:22:50,000 --> 00:22:59,000
so this is really just saying
it is T(n-1) + order n.
256
00:22:59,000 --> 00:23:08,000
And what is that equal to?
That is order n^2.
257
00:23:08,000 --> 00:23:19,000
Why is that order n^2?
It is an arithmetic series.
258
00:23:19,000 --> 00:23:30,000
Actually, just like we got for
insertion sort.
259
00:23:30,000 --> 00:23:35,000
Just like for insertion sort it
is an arithmetic series.
260
00:23:35,000 --> 00:23:42,000
Going through all that work and
we have an algorithm called
261
00:23:42,000 --> 00:23:47,000
quicksort, and it is no faster
than insertion sort.
262
00:23:47,000 --> 00:23:51,000
Nevertheless,
I said it was a good algorithm.
263
00:23:51,000 --> 00:23:57,000
The reason it is a good
algorithm is because its average
264
00:23:57,000 --> 00:24:04,000
case time, as we are going to
see, is very good.
265
00:24:04,000 --> 00:24:09,000
But let's try to understand
this a little bit more just so
266
00:24:09,000 --> 00:24:15,000
that we understand the
difference between what is going
267
00:24:15,000 --> 00:24:20,000
to happen in the average case
and what is going to happen in
268
00:24:20,000 --> 00:24:25,000
the worse-case.
Let's draw a recursion tree for
269
00:24:25,000 --> 00:24:31,000
this for T(n) = T(0) + T(n-1) +
and I will make the constant
270
00:24:31,000 --> 00:24:36,000
explicit for cn.
So, we get an intuition of what
271
00:24:36,000 --> 00:24:39,000
is going on.
Some constant times n.
272
00:24:39,000 --> 00:24:44,000
And then we have T(n) is equal
to, and we write it with the
273
00:24:44,000 --> 00:24:47,000
constant part here,
cn, and then T(0) here,
274
00:24:47,000 --> 00:24:51,000
and then T(n-1) here.
Now, I know that all you folks
275
00:24:51,000 --> 00:24:56,000
are really fast and want to jump
immediately to the full-blown
276
00:24:56,000 --> 00:24:59,000
tree.
But, let me tell you,
277
00:24:59,000 --> 00:25:04,000
my advice is that you spend
just a couple of minutes writing
278
00:25:04,000 --> 00:25:06,000
it out.
Since the tree grows
279
00:25:06,000 --> 00:25:10,000
exponentially,
it only costs you a constant
280
00:25:10,000 --> 00:25:14,000
overhead to write out the small
cases and make sure that you
281
00:25:14,000 --> 00:25:18,000
have got the pattern that you
are developing.
282
00:25:18,000 --> 00:25:21,000
So, I am going to go one more
step.
283
00:25:21,000 --> 00:25:26,000
Here we have T(0) and now this
becomes c(n-1) and now we have
284
00:25:26,000 --> 00:25:30,000
another T(0) over here and
T(n-2).
285
00:25:30,000 --> 00:25:35,000
And we continue that,
dot, dot, dot.
286
00:25:35,000 --> 00:25:45,000
That is all equal to cn with a
T(0) here, c(n-1) with a T(0),
287
00:25:45,000 --> 00:25:53,000
c(n-2), T(0) here,
and that goes all the way down
288
00:25:53,000 --> 00:26:01,000
until we end up with T(1) down
here.
289
00:26:01,000 --> 00:26:07,000
What is the height of this tree?
290
00:26:17,000 --> 00:26:20,000
What is the height of the tree
here?
291
00:26:20,000 --> 00:26:22,000
Yeah, n.
Good.
292
00:26:22,000 --> 00:26:30,000
Because every step we are just
decrementing the argument by 1.
293
00:26:30,000 --> 00:26:35,000
So, the height is n.
To analyze this,
294
00:26:35,000 --> 00:26:40,000
let's first add up everything
that is here.
295
00:26:40,000 --> 00:26:48,000
Just so we understand where
these things are coming from,
296
00:26:48,000 --> 00:26:56,000
this is just theta of the
summation of k equals 1 to n of
297
00:26:56,000 --> 00:27:02,000
k, actually of ck.
That is what is in there.
298
00:27:02,000 --> 00:27:08,000
And that is equal to order n^2.
That is where our algorithmatic
299
00:27:08,000 --> 00:27:12,000
series is coming from.
So, that is Theta(n^2).
300
00:27:12,000 --> 00:27:18,000
And then all of these things
here are all Theta(1).
301
00:27:23,000 --> 00:27:33,000
And how many of them are there?
There are n Theta(1)'s.
302
00:27:33,000 --> 00:27:54,000
So, the total amount is T(n) =
Theta(n) + Theta(n^2) =
303
00:27:54,000 --> 00:28:03,000
Theta(n^2).
Just to see what the structure
304
00:28:03,000 --> 00:28:09,000
is in terms of the recursion
tree, it is a highly unbalanced
305
00:28:09,000 --> 00:28:14,000
recursion tree.
Now I am going to do something
306
00:28:14,000 --> 00:28:20,000
that I told you should never do,
which is we are going to be do
307
00:28:20,000 --> 00:28:25,000
a best-case analysis.
This is for intuition only.
308
00:28:25,000 --> 00:28:32,000
And, in general,
we don't do best-case analyses.
309
00:28:32,000 --> 00:28:35,000
It doesn't mean anything,
unless we get some intuition
310
00:28:35,000 --> 00:28:38,000
for it maybe.
But basically it means nothing
311
00:28:38,000 --> 00:28:43,000
mathematically because it's
providing no guarantee.
312
00:28:55,000 --> 00:29:00,000
And so this is intuition only.
313
00:29:05,000 --> 00:29:13,000
If we are really lucky what
happens for partition?
314
00:29:13,000 --> 00:29:20,000
What is going to be the lucky
case?
315
00:29:26,000 --> 00:29:28,000
Yeah, it splits right in the
middle.
316
00:29:28,000 --> 00:29:31,000
Which is essentially --
317
00:29:43,000 --> 00:29:46,000
-- n/2 : n/2.
It is really (n-1)/2 :
318
00:29:46,000 --> 00:29:51,000
(n-1)/2, but we're not going to
worry about the details because
319
00:29:51,000 --> 00:29:56,000
we're only doing intuition for
the best-case because best-case
320
00:29:56,000 --> 00:30:01,000
is not what we want.
If that happened,
321
00:30:01,000 --> 00:30:11,000
what is the recurrence I get?
Imagine it split it exactly in
322
00:30:11,000 --> 00:30:18,000
the middle every time,
then what happens?
323
00:30:18,000 --> 00:30:26,000
You get T(n) = 2T(n/2) + order
n for partitioning and
324
00:30:26,000 --> 00:30:33,000
bookkeeping.
And what is the solution of
325
00:30:33,000 --> 00:30:36,000
that recurrence?
That is n log n.
326
00:30:36,000 --> 00:30:42,000
That is the same as the merge
sort recurrence.
327
00:30:42,000 --> 00:30:46,000
It is which case of the master
theorem?
328
00:30:46,000 --> 00:30:51,000
Case 2, right?
Because n to the log base 2 of
329
00:30:51,000 --> 00:30:55,000
2 is n to the 1,
it is the same,
330
00:30:55,000 --> 00:31:05,000
so we tack on the extra log n.
Case 2 of the master theorem.
331
00:31:05,000 --> 00:31:14,000
That is pretty good.
That says that in the best-case
332
00:31:14,000 --> 00:31:24,000
quicksort is going to do well.
How about let's suppose the
333
00:31:24,000 --> 00:31:32,000
split is always let's say 1/10 :
9/10, 1/10n :
334
00:31:32,000 --> 00:31:35,000
9/10n.
In that case,
335
00:31:35,000 --> 00:31:43,000
are we lucky or are we unlucky?
336
00:31:52,000 --> 00:31:57,000
I mean, if the split is really
skewed, we clearly are going to
337
00:31:57,000 --> 00:32:00,000
be unlucky, right,
because then it's,
338
00:32:00,000 --> 00:32:04,000
say, 1 to n.
If it is really in the middle
339
00:32:04,000 --> 00:32:08,000
it is n log n.
What do you suppose it is if it
340
00:32:08,000 --> 00:32:11,000
is 1/10 : 9/10?
Is that lucky or unlucky?
341
00:32:11,000 --> 00:32:14,000
We will have a little democracy
here.
342
00:32:14,000 --> 00:32:17,000
Who thinks that that is a lucky
case?
343
00:32:17,000 --> 00:32:20,000
It is going to be fast running
time.
344
00:32:20,000 --> 00:32:23,000
And who thinks it is an unlucky
case?
345
00:32:23,000 --> 00:32:26,000
OK, so we have some brave
souls.
346
00:32:26,000 --> 00:32:30,000
And who didn't vote?
Oh, come on.
347
00:32:30,000 --> 00:32:32,000
Come on.
It is always better,
348
00:32:32,000 --> 00:32:35,000
by the way, to say yes or no
and be right or wrong,
349
00:32:35,000 --> 00:32:40,000
because then you have some
emotional commitment to it and
350
00:32:40,000 --> 00:32:43,000
we will remember better,
rather than just sitting and
351
00:32:43,000 --> 00:32:47,000
being quiet.
You don't manipulate your own
352
00:32:47,000 --> 00:32:50,000
emotions well enough to remember
things well.
353
00:32:50,000 --> 00:32:54,000
Those people who voted win over
the people who don't vote,
354
00:32:54,000 --> 00:32:56,000
whether they are right or
wrong.
355
00:32:56,000 --> 00:33:02,000
Well, let's take a look.
Here is the recurrence.
356
00:33:02,000 --> 00:33:07,000
T(n) = T(1/10n) + T(9/10n) +
Theta(n).
357
00:33:07,000 --> 00:33:15,000
And we will assume that this
part here is less than or equal
358
00:33:15,000 --> 00:33:19,000
to some cn in order to analyze
it.
359
00:33:19,000 --> 00:33:25,000
We will just do a recursion
tree for this and see.
360
00:33:25,000 --> 00:33:30,000
Here is a recursion tree.
361
00:33:35,000 --> 00:33:41,000
We have T(n) = cn,
T(1/10n), T(9/10n).
362
00:33:41,000 --> 00:33:46,000
Now we have again cn at the
top.
363
00:33:46,000 --> 00:33:51,000
This gets complicated,
right?
364
00:33:51,000 --> 00:34:00,000
This is 1/10cn.
Now, over here we have 1/10.
365
00:34:00,000 --> 00:34:09,000
And then we are plugging it
into the recursion again,
366
00:34:09,000 --> 00:34:20,000
so we now get T(1/100n) and
over here we get T(9/100n).
367
00:34:20,000 --> 00:34:26,000
And over here we have now
9/10cn.
368
00:34:26,000 --> 00:34:34,000
And that gives us T(9/100n)
again.
369
00:34:34,000 --> 00:34:48,000
And here we get T(81/100n).
And we keep going on.
370
00:34:48,000 --> 00:34:59,000
That is equal to cn,
1/10cn here.
371
00:34:59,000 --> 00:35:08,000
Down this way we have 1/100cn.
And that keeps going down until
372
00:35:08,000 --> 00:35:16,000
we get to order 1 down here.
And over here we have 9/10cn.
373
00:35:16,000 --> 00:35:24,000
And here, let's see,
this is 9/100cn and this is now
374
00:35:24,000 --> 00:35:32,000
9/100cn and this is 81/100cn.
And these things keep going
375
00:35:32,000 --> 00:35:36,000
down until they get down to
order 1.
376
00:35:36,000 --> 00:35:41,000
But the leaves are not all at
uniform depth here,
377
00:35:41,000 --> 00:35:45,000
right?
This side is way further up
378
00:35:45,000 --> 00:35:47,000
than this side,
right?
379
00:35:47,000 --> 00:35:53,000
Because here we are only going
down by 9/10 each time.
380
00:35:53,000 --> 00:35:58,000
So, in fact,
what is the length of this path
381
00:35:58,000 --> 00:36:02,000
here?
What is the length of this path
382
00:36:02,000 --> 00:36:06,000
down to this,
if I take the left most spine?
383
00:36:15,000 --> 00:36:23,000
Somebody raise there hand.
Yeah?
384
00:36:23,000 --> 00:36:30,000
Log base 10 of n.
Because I am basically cutting
385
00:36:30,000 --> 00:36:33,000
down by a factor of 10 each
time.
386
00:36:33,000 --> 00:36:35,000
And how long does it take me to
reduce it to 1?
387
00:36:35,000 --> 00:36:38,000
That is the definition,
if you will,
388
00:36:38,000 --> 00:36:40,000
of what a log is,
log base 10.
389
00:36:40,000 --> 00:36:43,000
What is this one?
What is this path going that
390
00:36:43,000 --> 00:36:44,000
way?
391
00:36:53,000 --> 00:37:02,000
Log of n. Log base 10/9 of n.
Because we're going down by
392
00:37:02,000 --> 00:37:05,000
9/10 each time.
Once again, essentially the
393
00:37:05,000 --> 00:37:09,000
definition of n.
And everything in between there
394
00:37:09,000 --> 00:37:14,000
is somewhere between log base 10
of n and log base 10/9 of n.
395
00:37:14,000 --> 00:37:17,000
So, everything is in between
there.
396
00:37:17,000 --> 00:37:22,000
Now what I can do is do the
trick that we did for mergesort
397
00:37:22,000 --> 00:37:26,000
in looking at what the
evaluation of this is by adding
398
00:37:26,000 --> 00:37:31,000
up what is the cost of the total
level.
399
00:37:31,000 --> 00:37:36,000
That is just cn.
What is the cost of the next
400
00:37:36,000 --> 00:37:38,000
level?
cn.
401
00:37:38,000 --> 00:37:43,000
And what is the cost of the
next level?
402
00:37:43,000 --> 00:37:47,000
cn.
Every level we are still doing
403
00:37:47,000 --> 00:37:54,000
the same amount of work.
And we take that all the way
404
00:37:54,000 --> 00:38:00,000
down.
And the last levels --
405
00:38:00,000 --> 00:38:04,000
Eventually we hit some point
where it is not equal to cn
406
00:38:04,000 --> 00:38:09,000
where we start getting things
that are less than or equal to
407
00:38:09,000 --> 00:38:14,000
cn because some of the leaves
start dropping out starting at
408
00:38:14,000 --> 00:38:17,000
this level.
Basically this part is going to
409
00:38:17,000 --> 00:38:21,000
be log base 10 of n,
and then we start getting
410
00:38:21,000 --> 00:38:25,000
things that are less than or
equal to cn, and so forth,
411
00:38:25,000 --> 00:38:30,000
until finally we get to add it
all up.
412
00:38:30,000 --> 00:38:36,000
T(n) is going to be less than
or equal to cn times,
413
00:38:36,000 --> 00:38:43,000
well, what is the longest that
this could possibly be?
414
00:38:43,000 --> 00:38:49,000
Log base 10/9 of n.
Plus we have all of the leaves
415
00:38:49,000 --> 00:38:56,000
that we have to add in,
but all the leaves together add
416
00:38:56,000 --> 00:39:03,000
up to just order n.
All the leaves add up to order
417
00:39:03,000 --> 00:39:08,000
n, so we have + Theta(n).
And so this is how much?
418
00:39:08,000 --> 00:39:14,000
If I add all of this together,
what is this asymptotically?
419
00:39:14,000 --> 00:39:18,000
That is n log n.
So, T(n) is actually bounded by
420
00:39:18,000 --> 00:39:21,000
n log n.
We are lucky.
421
00:39:21,000 --> 00:39:25,000
Those people who guessed lucky
were right.
422
00:39:25,000 --> 00:39:30,000
A 1/10 : 9/10 split is
asymptotically as good as a 50 :
423
00:39:30,000 --> 00:39:34,000
50 split.
And, in fact,
424
00:39:34,000 --> 00:39:41,000
we can lower bound this by just
looking at these things here and
425
00:39:41,000 --> 00:39:45,000
discover that,
in fact, T(n) is lower bounded
426
00:39:45,000 --> 00:39:49,000
by cn log base 10 of n + order
n.
427
00:39:49,000 --> 00:39:55,000
And so T(n) is lower bounded by
also asymptotically n log n.
428
00:39:55,000 --> 00:40:00,000
So, T(n) is actually Theta(n lg
n).
429
00:40:00,000 --> 00:40:04,000
Now, this is not really proof.
I generally recommend that you
430
00:40:04,000 --> 00:40:07,000
don't do this kind of thing to
do a proof.
431
00:40:07,000 --> 00:40:10,000
This is a good intuition of a
recursion tree.
432
00:40:10,000 --> 00:40:14,000
The way you prove this is what?
Substitution method.
433
00:40:14,000 --> 00:40:17,000
Good.
What you do is use this to get
434
00:40:17,000 --> 00:40:20,000
your guess and then use
substitution method to prove
435
00:40:20,000 --> 00:40:25,000
that your guess is right.
It is too easy to make mistakes
436
00:40:25,000 --> 00:40:28,000
with this method.
It is very easy to make
437
00:40:28,000 --> 00:40:32,000
mistakes.
With the substitution method it
438
00:40:32,000 --> 00:40:37,000
is harder to make mistakes
because there is just algebra
439
00:40:37,000 --> 00:40:40,000
there that you are cranking
through.
440
00:40:40,000 --> 00:40:43,000
It is easier to verify rather
than dot, dot,
441
00:40:43,000 --> 00:40:48,000
dots and trees that you drew
improperly and wrote in wrong
442
00:40:48,000 --> 00:40:50,000
amounts and so forth.
OK?
443
00:40:50,000 --> 00:40:53,000
So, this is n log n.
That's pretty good.
444
00:40:53,000 --> 00:40:56,000
It is order n log n.
And we are lucky.
445
00:40:56,000 --> 00:41:01,000
Now let's try another one.
This is all for intuition
446
00:41:01,000 --> 00:41:05,000
because, I will tell you,
by the time we get to the end
447
00:41:05,000 --> 00:41:10,000
of this class you folks are
going to bolting for the door
448
00:41:10,000 --> 00:41:13,000
because we are going to do some
good math today,
449
00:41:13,000 --> 00:41:16,000
actually.
It is actually fun math,
450
00:41:16,000 --> 00:41:20,000
I think, but it is challenging.
If you are not awake,
451
00:41:20,000 --> 00:41:23,000
you can still sleep now,
but I will tell you when to
452
00:41:23,000 --> 00:41:26,000
wake up.
One more bit of intuition.
453
00:41:26,000 --> 00:41:30,000
Suppose that we alternate
steps.
454
00:41:30,000 --> 00:41:33,000
Suppose we do the partitioning
thing.
455
00:41:33,000 --> 00:41:38,000
And it happens that we start
out lucky and then we have a
456
00:41:38,000 --> 00:41:43,000
partitioning step that is
unlucky and then we have a step
457
00:41:43,000 --> 00:41:49,000
that is lucky and a step that is
unlucky and we do that all the
458
00:41:49,000 --> 00:41:54,000
way down the tree.
Suppose we alternate.
459
00:42:07,000 --> 00:42:09,000
Are we lucky or unlucky if we
do that?
460
00:42:09,000 --> 00:42:11,000
This time I want everybody
voting.
461
00:42:11,000 --> 00:42:13,000
It doesn't matter what your
answer is.
462
00:42:13,000 --> 00:42:16,000
Everybody has to have a stake
in the game.
463
00:42:16,000 --> 00:42:19,000
It is sort of like horseracing.
If ever you have watched
464
00:42:19,000 --> 00:42:23,000
horseracing, it is really
boring, but if you put a little
465
00:42:23,000 --> 00:42:26,000
bit of money down,
a little skin in the game
466
00:42:26,000 --> 00:42:30,000
suddenly it is interesting.
The same thing here.
467
00:42:30,000 --> 00:42:33,000
I want everybody to put some
skin in the game.
468
00:42:33,000 --> 00:42:37,000
Who thinks that this is going
to be lucky?
469
00:42:37,000 --> 00:42:40,000
Who thinks it is going to be
unlucky?
470
00:42:40,000 --> 00:42:42,000
OK.
Who didn't vote?
471
00:42:42,000 --> 00:42:46,000
[LAUGHTER] You guys.
No skin in the game,
472
00:42:46,000 --> 00:42:48,000
ha?
Let's analyze this so we can
473
00:42:48,000 --> 00:42:53,000
once again write a recurrence.
On the lucky step,
474
00:42:53,000 --> 00:42:57,000
we will have L(n) be the
running time on a lucky step of
475
00:42:57,000 --> 00:43:02,000
size n.
And that is going to be twice.
476
00:43:02,000 --> 00:43:07,000
While the next step is going to
be unlucky.
477
00:43:07,000 --> 00:43:10,000
It is two unluckies over 2 plus
order n.
478
00:43:10,000 --> 00:43:15,000
That is our lucky step.
And then for the unlucky step
479
00:43:15,000 --> 00:43:20,000
it is essentially going to be L
of n minus 1,
480
00:43:20,000 --> 00:43:25,000
it is going to be lucky on the
next step, plus order n.
481
00:43:25,000 --> 00:43:30,000
That is unlucky.
See how I have described this
482
00:43:30,000 --> 00:43:36,000
behavior with a system now of
recurrences that are dependent
483
00:43:36,000 --> 00:43:41,000
where the boundary cases,
once again which are unstated,
484
00:43:41,000 --> 00:43:47,000
is that the recurrences have a
constant solution with constant
485
00:43:47,000 --> 00:43:50,000
input.
Now we just do a little bit of
486
00:43:50,000 --> 00:43:54,000
algebra using substitution.
L(n) is then equal to,
487
00:43:54,000 --> 00:44:00,000
well, I can just plug in,
for U(n/2) plug in the value of
488
00:44:00,000 --> 00:44:06,000
U(n/2).
And that gives me 2[L(n/2-1) +
489
00:44:06,000 --> 00:44:11,000
Theta(n) + Theta(n)].
See what I did here?
490
00:44:11,000 --> 00:44:18,000
I simply plugged in,
for U(n/2), this recurrence.
491
00:44:18,000 --> 00:44:27,000
In fact, technically I guess I
should have said Theta(n/2) just
492
00:44:27,000 --> 00:44:35,000
to make this substitution more
straightforward.
493
00:44:35,000 --> 00:44:41,000
It is the same thing,
but just to not skip a step.
494
00:44:41,000 --> 00:44:49,000
That we can now crank through.
And that is 2L(n/2 - 1) +,
495
00:44:49,000 --> 00:44:57,000
and now I have two T(n/2) plus
another one, so all of that is
496
00:44:57,000 --> 00:45:04,000
just order n.
And what is the solution to
497
00:45:04,000 --> 00:45:07,000
that recurrence?
n log n.
498
00:45:07,000 --> 00:45:13,000
Theta(n lg n).
Does everybody see that?
499
00:45:13,000 --> 00:45:15,000
OK?
Theta(n lg n).
500
00:45:15,000 --> 00:45:24,000
This is basically just,
once again, master theorem with
501
00:45:24,000 --> 00:45:33,000
a little bit of jiggering here.
That minus one is only going to
502
00:45:33,000 --> 00:45:38,000
help us, actually,
in the solution of the master
503
00:45:38,000 --> 00:45:41,000
theorem.
So, it is order n lg n.
504
00:45:41,000 --> 00:45:45,000
We are lucky.
If we alternate lucky and
505
00:45:45,000 --> 00:45:50,000
unlucky, we are lucky.
How can we insure that we are
506
00:45:50,000 --> 00:45:55,000
usually lucky?
If I have the input already
507
00:45:55,000 --> 00:46:00,000
sorted, I am going to be
unlucky.
508
00:46:00,000 --> 00:46:03,000
Excuse me?
You could randomly arrange the
509
00:46:03,000 --> 00:46:07,000
elements, that is one way.
What is another way?
510
00:46:07,000 --> 00:46:10,000
That is a perfectly good way,
actually.
511
00:46:10,000 --> 00:46:13,000
In fact, it is a common thing
to do.
512
00:46:13,000 --> 00:46:16,000
Randomly choose the pivot,
OK.
513
00:46:16,000 --> 00:46:20,000
It turns out those are
effectively equivalent,
514
00:46:20,000 --> 00:46:24,000
but we are going to do the
randomly choose the pivot
515
00:46:24,000 --> 00:46:30,000
because it is a little bit
easier to analyze.
516
00:46:30,000 --> 00:46:33,000
But they are effectively
equivalent.
517
00:46:33,000 --> 00:46:39,000
That gives us the algorithm
called randomized quicksort.
518
00:46:46,000 --> 00:46:53,000
And the nice thing about
randomized quicksort is that the
519
00:46:53,000 --> 00:47:00,000
running time is independent of
the input ordering.
520
00:47:00,000 --> 00:47:04,000
Very much for the same reason
that if I just scramble the
521
00:47:04,000 --> 00:47:08,000
input, it would be independent
of the input ordering.
522
00:47:08,000 --> 00:47:12,000
If I randomly scramble the
input then it doesn't matter
523
00:47:12,000 --> 00:47:17,000
what the order of the input was.
Whereas, original quicksort has
524
00:47:17,000 --> 00:47:21,000
some slow cases,
input sorted or reverse sorted,
525
00:47:21,000 --> 00:47:24,000
and some fast cases.
In particular,
526
00:47:24,000 --> 00:47:28,000
it turns out that if it is
random it is going to be pretty
527
00:47:28,000 --> 00:47:31,000
fast.
If I actually randomly scramble
528
00:47:31,000 --> 00:47:35,000
the input or pivot on a random
element, it doesn't matter what
529
00:47:35,000 --> 00:47:38,000
the input was.
One way of thinking about this
530
00:47:38,000 --> 00:47:40,000
is with an adversary.
Imagine your adversary,
531
00:47:40,000 --> 00:47:44,000
you are saying I have a good
sorting algorithm and he says I
532
00:47:44,000 --> 00:47:47,000
have a good sorting algorithm
and you're trying to sell to a
533
00:47:47,000 --> 00:47:50,000
single customer.
And the customer says OK,
534
00:47:50,000 --> 00:47:53,000
you guys come up with
benchmarks for each of your
535
00:47:53,000 --> 00:47:55,000
algorithms.
And you get to look at his
536
00:47:55,000 --> 00:47:59,000
algorithm.
Well, you look and you say oh,
537
00:47:59,000 --> 00:48:02,000
he is using quicksort.
I will just give him something
538
00:48:02,000 --> 00:48:06,000
that is already sorted.
That is what you could do to
539
00:48:06,000 --> 00:48:08,000
him.
If you had quicksort,
540
00:48:08,000 --> 00:48:10,000
he would do the same thing to
you.
541
00:48:10,000 --> 00:48:14,000
So, how can you defeat him?
Well, one way is with
542
00:48:14,000 --> 00:48:17,000
randomization.
Big idea in computer science,
543
00:48:17,000 --> 00:48:20,000
use random numbers.
The idea here is if I permute
544
00:48:20,000 --> 00:48:23,000
the ordering at random,
as one suggestion,
545
00:48:23,000 --> 00:48:28,000
or I pivot at random places,
then the input ordering didn't
546
00:48:28,000 --> 00:48:32,000
matter.
And so there is no bad ordering
547
00:48:32,000 --> 00:48:36,000
that he can provide that is
going to make my code run
548
00:48:36,000 --> 00:48:39,000
slowly.
Now, I might get unlucky.
549
00:48:39,000 --> 00:48:43,000
But that is just unlucky in my
use of my random-number
550
00:48:43,000 --> 00:48:46,000
generator.
It is not unlucky with respect
551
00:48:46,000 --> 00:48:50,000
to what the input was.
What the input was doesn't
552
00:48:50,000 --> 00:48:52,000
matter.
Everybody follow that?
553
00:48:52,000 --> 00:48:55,000
OK.
The nice thing about randomized
554
00:48:55,000 --> 00:48:59,000
quicksort is that it makes no
assumptions about the input
555
00:48:59,000 --> 00:49:05,000
distribution.
You don't have to assume that
556
00:49:05,000 --> 00:49:13,000
all inputs are equally likely
because either you can make it
557
00:49:13,000 --> 00:49:20,000
that way or you pivot in a way
that makes that effectively
558
00:49:20,000 --> 00:49:23,000
whole.
And, in particular,
559
00:49:23,000 --> 00:49:30,000
there is no specific input that
can elicit the worst-case
560
00:49:30,000 --> 00:49:45,000
behavior.
The worst-case is determined
561
00:49:45,000 --> 00:50:00,000
only by a random-number
generator.
562
00:50:00,000 --> 00:50:03,000
And, therefore,
since it is only determined by
563
00:50:03,000 --> 00:50:08,000
a random-number generator,
we can essentially bound the
564
00:50:08,000 --> 00:50:13,000
unluckiness mathematically.
We can say what are the odds?
565
00:50:13,000 --> 00:50:16,000
So, we are going to analyze
this.
566
00:50:16,000 --> 00:50:21,000
And this is where you know if
you belong in this course or
567
00:50:21,000 --> 00:50:24,000
not.
If you have skipped 6.042 or
568
00:50:24,000 --> 00:50:30,000
whatever, this is a good place
to do the comparison.
569
00:50:30,000 --> 00:50:34,000
Since it is going to be a
little bit, why don't people
570
00:50:34,000 --> 00:50:37,000
just stand up for a moment and
take a stretch break.
571
00:50:37,000 --> 00:50:42,000
Since this is going to be a
nice piece of mathematics we are
572
00:50:42,000 --> 00:50:47,000
going to do, you are going to
want to feel fresh for it.
573
00:51:02,000 --> 00:51:04,000
Stretch break is over.
574
00:51:10,000 --> 00:51:11,000
Analysis.
Good.
575
00:51:11,000 --> 00:51:16,000
I think we are going to make
this.
576
00:51:16,000 --> 00:51:22,000
I am sort of racing.
There is a lot of stuff to
577
00:51:22,000 --> 00:51:24,000
cover today.
Good.
578
00:51:24,000 --> 00:51:32,000
Let's let T(n) now be the
random variable for the running
579
00:51:32,000 --> 00:51:36,000
time assuming --
580
00:51:46,000 --> 00:51:47,000
Wow.
I didn't even write here what
581
00:51:47,000 --> 00:51:49,000
we did here.
So, we are going to pivot on a
582
00:51:49,000 --> 00:51:51,000
random element.
583
00:51:59,000 --> 00:52:01,000
That is the basic scheme we are
going to do.
584
00:52:01,000 --> 00:52:05,000
And the way I do that,
by the way, is just in the code
585
00:52:05,000 --> 00:52:07,000
for partition,
rather than partitioning on the
586
00:52:07,000 --> 00:52:10,000
first element,
before I do the partition,
587
00:52:10,000 --> 00:52:14,000
I just swap the first element
with some other element in the
588
00:52:14,000 --> 00:52:16,000
array chosen at random,
perhaps itself.
589
00:52:16,000 --> 00:52:19,000
So, they are all equally likely
to be pivoted on.
590
00:52:19,000 --> 00:52:23,000
And then just run the ordinary
partition.
591
00:52:23,000 --> 00:52:27,000
This is a random variable for
running in time assuming,
592
00:52:27,000 --> 00:52:32,000
we have to make an assumption
for doing probability,
593
00:52:32,000 --> 00:52:36,000
the random numbers are
independent.
594
00:52:41,000 --> 00:52:44,000
So that when I pivot in one
place, it is independent of how
595
00:52:44,000 --> 00:52:48,000
I pivoted in some other place as
I am running this algorithm.
596
00:52:48,000 --> 00:52:52,000
Then, to analyze this,
what I am going to do is I want
597
00:52:52,000 --> 00:52:57,000
to know where we pivoted.
For k = 0, 1,
598
00:52:57,000 --> 00:53:08,000
..., n-1, let's let,
for a particular partition,
599
00:53:08,000 --> 00:53:21,000
the random variable X_k = 1 if
partition generates a k :
600
00:53:21,000 --> 00:53:30,000
n-k-1 split,
and 0 otherwise.
601
00:53:35,000 --> 00:53:39,000
In the partition routine,
I am picking a random element
602
00:53:39,000 --> 00:53:42,000
to pivot on.
And X_k is going to be my
603
00:53:42,000 --> 00:53:47,000
random variable that is 1 if it
generates a split that has k
604
00:53:47,000 --> 00:53:52,000
elements on the left side and
n-k-1 elements on the right side
605
00:53:52,000 --> 00:53:54,000
of the pivot.
Some of those,
606
00:53:54,000 --> 00:54:00,000
too, of course are n-1 because
I also have the pivot.
607
00:54:00,000 --> 00:54:03,000
And 0 otherwise.
So, I now have n random
608
00:54:03,000 --> 00:54:09,000
variables that I have defined
associated with a single
609
00:54:09,000 --> 00:54:15,000
partition where all of them are
going to be zero except one of
610
00:54:15,000 --> 00:54:21,000
them, whichever one happens to
occur is going to have the value
611
00:54:22,000 --> 00:54:26,000
by the way.
What is the name of this type
612
00:54:26,000 --> 00:54:30,000
of random variable?
613
00:54:37,000 --> 00:54:40,000
Bernoulli.
Well, Bernoulli has other
614
00:54:40,000 --> 00:54:43,000
assumptions.
It is an indicator random
615
00:54:43,000 --> 00:54:46,000
variable.
It turns out it is Bernoulli,
616
00:54:46,000 --> 00:54:50,000
but that's OK.
It is an indicator random
617
00:54:50,000 --> 00:54:53,000
variable.
It just takes on the value of
618
00:54:53,000 --> 00:54:56,000
0, 1.
And Bernoulli random variables
619
00:54:56,000 --> 00:55:02,000
are a particular type of
indicator random variable.
620
00:55:02,000 --> 00:55:06,000
Which it turns out these are.
That is an indicator random
621
00:55:06,000 --> 00:55:09,000
variable.
Indicator random variables are
622
00:55:09,000 --> 00:55:14,000
a good way when you are trying
to understand what the sum of a
623
00:55:14,000 --> 00:55:18,000
bunch of things is.
It is a good way to break apart
624
00:55:18,000 --> 00:55:22,000
your big random variables into
smaller ones that can be
625
00:55:22,000 --> 00:55:25,000
analyzed.
Let's just take a look at this
626
00:55:25,000 --> 00:55:30,000
indicator random variable.
What is the expectation of X_k
627
00:55:30,000 --> 00:55:32,000
equal to?
628
00:55:42,000 --> 00:55:45,000
In other words,
what is the probability that I
629
00:55:45,000 --> 00:55:48,000
generate a k :
n-k-1 split?
630
00:56:00,000 --> 00:56:05,000
X_k is, let's just write out
what that means,
631
00:56:05,000 --> 00:56:08,000
just to refresh people's
memory.
632
00:56:08,000 --> 00:56:15,000
That is 0 times the probability
that X_k equals 0 plus 1 times
633
00:56:15,000 --> 00:56:21,000
the probability that X_k equals
1, which is equal,
634
00:56:21,000 --> 00:56:27,000
well, that is all zero.
That is just equal to the
635
00:56:27,000 --> 00:56:34,000
probability that X_k equals 1.
And that is a general property
636
00:56:34,000 --> 00:56:39,000
of indicator random variables,
is that their expectation is
637
00:56:39,000 --> 00:56:45,000
the probability that they are 1.
The nice thing about indicator
638
00:56:45,000 --> 00:56:51,000
random variables is it directly
connects the probability to the
639
00:56:51,000 --> 00:56:55,000
expectation without any other
terms going on.
640
00:56:55,000 --> 00:56:58,000
What is the probability of X_k
equals 1?
641
00:56:58,000 --> 00:57:02,000
1/n.
So, all splits are equally
642
00:57:02,000 --> 00:57:04,000
likely.
And I have n elements,
643
00:57:04,000 --> 00:57:10,000
so each ones has a 1/n chance
of being picked as the pivot.
644
00:57:10,000 --> 00:57:15,000
And, once you pick the pivot,
that determines what is on the
645
00:57:15,000 --> 00:57:19,000
left and the right and so forth.
So, it is 1/n.
646
00:57:19,000 --> 00:57:23,000
Everybody with me so far?
More or less?
647
00:57:23,000 --> 00:57:26,000
OK.
As I say, this is going to test
648
00:57:26,000 --> 00:57:32,000
whether you're in the class.
If you go home and you study
649
00:57:32,000 --> 00:57:37,000
this and you cannot get it,
and you have a deficiency in
650
00:57:37,000 --> 00:57:41,000
your math background in trying
to take the course,
651
00:57:41,000 --> 00:57:46,000
this is a good indication that
probably you have taken
652
00:57:46,000 --> 00:57:49,000
something a little over your
head.
653
00:57:49,000 --> 00:57:54,000
Let's write out what T(n) is
equal to here.
654
00:58:00,000 --> 00:58:11,000
T(n) is going to be equal to
T(0) + T(n-1) + Theta(n) if we
655
00:58:11,000 --> 00:58:24,000
get a 0 : n-1 split and is equal
to T(1) + T(n-2) + order n if we
656
00:58:24,000 --> 00:58:30,000
have a 1 : n-2 split.
657
00:58:35,000 --> 00:58:46,000
And now down here it is going
to be T(n-1) + T(0) + Theta(n)
658
00:58:46,000 --> 00:58:52,000
if we end up with an n-1 :
0 split.
659
00:58:52,000 --> 00:59:00,000
So, this is our recurrence for
T(n).
660
00:59:00,000 --> 00:59:03,000
And, unfortunately,
the recurrence is kind of hairy
661
00:59:03,000 --> 00:59:06,000
because it has got n cases.
And this is,
662
00:59:06,000 --> 00:59:11,000
once again, where the
brilliance of being able to use
663
00:59:11,000 --> 00:59:13,000
indicator random variables comes
in.
664
00:59:13,000 --> 00:59:18,000
Because we will be able to take
this case analysis and reduce it
665
00:59:18,000 --> 00:59:23,000
to mathematics so we don't have
cases using indicator random
666
00:59:23,000 --> 00:59:29,000
variables.
And the way we do that is using
667
00:59:29,000 --> 00:59:37,000
the following trick of
converting the cases into a
668
00:59:37,000 --> 00:59:39,000
summation.
669
00:59:55,000 --> 01:00:00,666
Let's just take a look at why
these two things are the same.
670
01:00:00,666 --> 01:00:06,333
The indicator random variable
is zero, except if you get the
671
01:00:06,333 --> 01:00:10,655
particular split.
Therefore, this summation is
672
01:00:10,655 --> 01:00:14,497
going to be zero,
except for that k which
673
01:00:14,497 --> 01:00:20,355
actually appeared in which case
it is the value that we say it
674
01:00:20,355 --> 01:00:22,468
is.
See the trick using
675
01:00:22,468 --> 01:00:27,079
multiplication by 0,
1 variable to handle all the
676
01:00:27,079 --> 01:00:31,404
cases?
I think that is damn clever.
677
01:00:31,404 --> 01:00:36,213
I think that is damn clever.
And this is like the classic
678
01:00:36,213 --> 01:00:40,421
thing that you do with indicator
random variables.
679
01:00:40,421 --> 01:00:45,144
It's one of the reasons they
are a very powerful method.
680
01:00:45,144 --> 01:00:49,781
Because now we actually have a
mathematical expression,
681
01:00:49,781 --> 01:00:53,559
hairy although it may be,
for our recurrence.
682
01:00:53,559 --> 01:00:58,454
Now, what we are going to
analyze is the expected value of
683
01:00:58,454 --> 01:01:02,977
T(n).
That is what we want to do.
684
01:01:02,977 --> 01:01:06,727
What is the expected value of
T(n)?
685
01:01:06,727 --> 01:01:13,235
To do that, I just write the
expected value of T(n) is equal
686
01:01:13,235 --> 01:01:17,977
to the expected value of this
big summation.
687
01:01:17,977 --> 01:01:24,264
And now we can go ahead and
start to evaluate the expected
688
01:01:24,264 --> 01:01:30,000
value of that summation.
Everybody with me?
689
01:01:30,000 --> 01:01:32,486
Yes?
Any questions at this point?
690
01:01:32,486 --> 01:01:35,360
I see a thumbs up.
That's nice to see.
691
01:01:35,360 --> 01:01:39,944
But I generally believe that
what I want to see is no thumbs
692
01:01:39,944 --> 01:01:42,508
down.
It is good to see the thumbs
693
01:01:42,508 --> 01:01:46,704
up, but that means one person
understands, or thinks he
694
01:01:46,704 --> 01:01:48,724
understands.
[LAUGHTER] So,
695
01:01:48,724 --> 01:01:51,832
this is, I claim,
equal to the following.
696
01:01:51,832 --> 01:01:56,571
Actually, I am going to need a
little space here so I am going
697
01:01:56,571 --> 01:02:01,000
to move the equal sign over a
little bit.
698
01:02:25,000 --> 01:02:27,702
I claim that summation is equal
to that.
699
01:02:27,702 --> 01:02:32,000
This expectation is equal to
that summation of expectations.
700
01:02:32,000 --> 01:02:36,042
Why is that?
What are the magic words that
701
01:02:36,042 --> 01:02:40,281
justify this step?
Linearity of expectation.
702
01:02:40,281 --> 01:02:45,704
The expectation of a sum is the
sum of the expectations.
703
01:02:45,704 --> 01:02:49,253
So, that is linearity of
expectation.
704
01:02:49,253 --> 01:02:52,605
I don't need independence for
that.
705
01:02:52,605 --> 01:02:57,830
That is just always true for
expectation of any random
706
01:02:57,830 --> 01:03:04,591
variables.
The sum of the expectations is
707
01:03:04,591 --> 01:03:10,867
the expectation of the sum and
vice versa.
708
01:03:10,867 --> 01:03:19,744
Here we did the vice versa.
That is equal to now the sum of
709
01:03:19,744 --> 01:03:30,000
k=0 to n-1 of expectation of X_k
[T(k) + T(n-k-1) + Theta(n)].
710
01:03:36,000 --> 01:03:42,272
Why is that true?
What I have done is I've said
711
01:03:42,272 --> 01:03:49,227
the expectation of the product
is the product of the
712
01:03:49,227 --> 01:03:53,454
expectations.
That is because of
713
01:03:53,454 --> 01:04:00,000
independence.
What is independent of what?
714
01:04:00,000 --> 01:04:02,492
The X_k here,
random variable,
715
01:04:02,492 --> 01:04:06,962
are independent of any of the
other partitionings in,
716
01:04:06,962 --> 01:04:10,401
if you will,
the X_k that would exist for
717
01:04:10,401 --> 01:04:13,151
any of the other recursive
calls.
718
01:04:13,151 --> 01:04:18,223
So, whatever happens in here is
independent of what happened
719
01:04:18,223 --> 01:04:20,716
there.
We are actually hiding.
720
01:04:20,716 --> 01:04:25,358
Since we have a recurrence,
we are not partitioning the
721
01:04:25,358 --> 01:04:30,000
same wage time.
We have a different one.
722
01:04:30,000 --> 01:04:33,168
We actually have something
going on underneath the
723
01:04:33,168 --> 01:04:36,271
mathematics you have to pay
attention to that the
724
01:04:36,271 --> 01:04:40,150
mathematics alone isn't really
showing, which is that in T(k)
725
01:04:40,150 --> 01:04:43,706
there is actually a set of
random choices that are being
726
01:04:43,706 --> 01:04:46,745
made, if you will.
And so you have to understand
727
01:04:46,745 --> 01:04:50,366
that those are independent of
those, in which case we can
728
01:04:50,366 --> 01:04:53,469
multiple the probabilities of
their expectations.
729
01:04:53,469 --> 01:04:55,991
Is everybody with me?
That is a big one,
730
01:04:55,991 --> 01:05:00,000
independence of X_k from other
random choices.
731
01:05:00,000 --> 01:05:04,624
That is equal to now,
well, first of all,
732
01:05:04,624 --> 01:05:09,710
this is nice.
What is the expectation of X_k?
733
01:05:09,710 --> 01:05:13,179
1/n.
That actually doesn't even
734
01:05:13,179 --> 01:05:20,000
belong in the summation.
We will just pop it outside.
735
01:05:20,000 --> 01:05:35,541
I get 1/n times the sum of k=0
to n-1 of expectation of T(k) +
736
01:05:35,541 --> 01:05:49,808
1/n summation k=0 to n-1 of
expectation of T(n-k-1) + 1/n
737
01:05:49,808 --> 01:06:00,000
summation k=0 to n-1 up to
Theta(n).
738
01:06:00,000 --> 01:06:04,838
That is, again,
using linearity of expectation
739
01:06:04,838 --> 01:06:11,075
there this time to split up
these pieces and just factoring
740
01:06:11,075 --> 01:06:15,053
out the expectation of k as
being 1/n.
741
01:06:15,053 --> 01:06:20,430
Everybody with me still?
All of this is elementary.
742
01:06:20,430 --> 01:06:26,989
It is just one of these things
that is hard just because there
743
01:06:26,989 --> 01:06:33,174
are so many steps.
And it takes that you have seen
744
01:06:33,174 --> 01:06:37,986
some of this before.
Now the next observation is
745
01:06:37,986 --> 01:06:43,003
that these two summations are,
in fact, identical.
746
01:06:43,003 --> 01:06:48,532
They are the same summation,
just in a different order.
747
01:06:48,532 --> 01:06:53,549
This is going T(0),
T(1), T(2), T(3) up to T(n-1).
748
01:06:53,549 --> 01:07:00,000
This one is going T(n-1),
T(n-2), T(n-3) down to T(0).
749
01:07:00,000 --> 01:07:02,049
These are, in fact,
equal.
750
01:07:02,049 --> 01:07:05,000
So, therefore,
I have two of them.
751
01:07:19,000 --> 01:07:22,000
And then what is this term
equal to?
752
01:07:37,000 --> 01:07:40,991
What is that one equal to?
Theta(n).
753
01:07:40,991 --> 01:07:45,438
Let's just see why.
The summation of 0 :
754
01:07:45,438 --> 01:07:50,000
n of Theta(n) is Theta(n^2)
divided by n.
755
01:07:50,000 --> 01:07:54,447
Or, if I want to bring the
Theta(n) out,
756
01:07:54,447 --> 01:08:01,631
I have 1 times the summation of
k equals1 to n of Theta(1) or of
757
01:08:04,527 --> 01:08:07,888
you get n, either way of doing
it.
758
01:08:07,888 --> 01:08:12,981
This is, in some sense,
because the summations have
759
01:08:12,981 --> 01:08:17,157
identical terms,
and this is just algebra.
760
01:08:17,157 --> 01:08:22,861
Now what we are going to do is
do something for technical
761
01:08:22,861 --> 01:08:30,216
convenience.
Because we are going to absorb
762
01:08:30,216 --> 01:08:38,390
the k=0, 1 terms into the
Theta(n) for technical
763
01:08:38,390 --> 01:08:41,000
convenience.
764
01:08:46,000 --> 01:08:50,432
We have a recurrence here where
I have an order n.
765
01:08:50,432 --> 01:08:54,412
And, if I look at the cases
where k=0 or k=1,
766
01:08:54,412 --> 01:08:59,929
I know what the expectation is.
For 0, 1, the expected cost is
767
01:08:59,929 --> 01:09:04,000
the worst case cost,
which is constant.
768
01:09:04,000 --> 01:09:07,920
Because I am only solving the
problem for a constant size.
769
01:09:07,920 --> 01:09:12,252
And we know that for any of the
boundary cases that our solution
770
01:09:12,252 --> 01:09:15,142
of recurrence,
our assumption is that it is
771
01:09:15,142 --> 01:09:18,100
constant time.
So, I basically can just take
772
01:09:18,100 --> 01:09:21,608
those two terms out.
And all that does it accumulate
773
01:09:21,608 --> 01:09:24,291
some more constant here in the
Theta(n).
774
01:09:24,291 --> 01:09:27,798
It is going to make the
solution of the recurrence a
775
01:09:27,798 --> 01:09:33,395
little bit easier.
And, if I do that,
776
01:09:33,395 --> 01:09:43,584
I get expectation of T(n) = 2/n
summation k=2 to n-1 of
777
01:09:43,584 --> 01:09:50,000
expectation of T(k) + Theta(n).
778
01:10:00,000 --> 01:10:06,864
So, all of that work was to
derive the recurrence.
779
01:10:06,864 --> 01:10:14,570
And now we have to solve it.
Just to review what we did,
780
01:10:14,570 --> 01:10:21,434
we started out with a
recurrence which was for the
781
01:10:21,434 --> 01:10:29,000
random variable which involved a
case statement.
782
01:10:29,000 --> 01:10:33,128
We converted that into some
mathematics without the case
783
01:10:33,128 --> 01:10:37,031
statement, just with a product,
and then we derived a
784
01:10:37,031 --> 01:10:41,535
recurrence for the expectation.
And now we are in the process
785
01:10:41,535 --> 01:10:44,087
of trying to solve that
recurrence.
786
01:10:44,087 --> 01:10:47,765
We have done some
simplification of the recurrence
787
01:10:47,765 --> 01:10:52,568
so that we understand what it is
that we are going to solve here.
788
01:10:52,568 --> 01:10:56,472
By the way, I don't give things
like this on quizzes.
789
01:10:56,472 --> 01:11:00,000
I do expect you to understand
it.
790
01:11:00,000 --> 01:11:03,120
The elements of this you will
find on a quiz.
791
01:11:03,120 --> 01:11:05,602
This is a lot of work to figure
out.
792
01:11:05,602 --> 01:11:09,148
This took smart people to do.
Even though it is all
793
01:11:09,148 --> 01:11:12,907
elementary, but working out
something like this at the
794
01:11:12,907 --> 01:11:17,375
elementary level is still a bit
of work even for somebody who is
795
01:11:17,375 --> 01:11:23,465
knowledgeable in this area.
Now we are going to solve that
796
01:11:23,465 --> 01:11:30,395
last recurrence over there and
we are going to prove that the
797
01:11:30,395 --> 01:11:37,556
expectation of T(n) is less than
or equal to (an lg n) for some
798
01:11:37,556 --> 01:11:44,139
constant a greater than 0.
That is going to be what we are
799
01:11:44,139 --> 01:11:48,759
going to do.
And so what technique do you
800
01:11:48,759 --> 01:11:52,571
think we should use to prove
this?
801
01:11:52,571 --> 01:11:58,000
Does this look like a master
method?
802
01:12:03,000 --> 01:12:08,600
It is nothing like the master
method.
803
01:12:08,600 --> 01:12:13,733
So, when in doubt do
substitution.
804
01:12:13,733 --> 01:12:19,333
It is the grand-daddy of all
methods.
805
01:12:19,333 --> 01:12:28,355
What we will do is solve the
base case by simply choosing a
806
01:12:28,355 --> 01:12:37,844
big enough so that (an lg n) is
bigger than the expectation of
807
01:12:37,844 --> 01:12:45,000
T(n) for sufficiently large
small n.
808
01:12:45,000 --> 01:12:49,212
So, I just pick a to be big
enough.
809
01:12:49,212 --> 01:12:54,044
And this is,
by the way, why I wanted to
810
01:12:54,044 --> 01:12:59,000
exclude 0 and 1 from the
recurrence.
811
01:12:59,000 --> 01:13:03,338
Because, for example,
when n=0, log of 0 is,
812
01:13:03,338 --> 01:13:08,181
it's like dividing by 0,
right, you cannot do it.
813
01:13:08,181 --> 01:13:12,519
Log of 1 is 0.
So here, even if I restricted
814
01:13:12,519 --> 01:13:18,370
it to 1, here I would have a 0,
and I can't ever pick a big
815
01:13:18,370 --> 01:13:24,424
enough to dominate those cases.
What I do is I just say look,
816
01:13:24,424 --> 01:13:30,577
I just absorb whatever the cost
is into the T(n) for technical
817
01:13:30,577 --> 01:13:37,169
convenience.
And that lets me address it as
818
01:13:37,169 --> 01:13:44,119
(an lg n) to be big enough to
handle the base case.
819
01:13:44,119 --> 01:13:50,930
So, that is why we made that
technical assumption.
820
01:13:50,930 --> 01:13:58,714
We are going to use a fact
which is that the summation of
821
01:13:58,714 --> 01:14:06,776
k=2 to n-1 of k lg k is less
than or equal to 1/2n^2 lg n -
822
01:14:06,776 --> 01:14:11,242
1/8n^2.
I am going to leave that as an
823
01:14:11,242 --> 01:14:15,429
exercise for you to workout.
I think it is an exercise in
824
01:14:15,429 --> 01:14:18,644
the book, too.
I want you to go and evaluate
825
01:14:18,644 --> 01:14:21,261
this.
There are two ways to evaluate
826
01:14:21,261 --> 01:14:23,130
it.
One is by using purely
827
01:14:23,130 --> 01:14:27,093
summations and facts about
summations by splitting the
828
01:14:27,093 --> 01:14:31,579
summation into two pieces and
reconstituting it to prove this
829
01:14:31,579 --> 01:14:35,338
bound.
The other way is to use the
830
01:14:35,338 --> 01:14:38,629
integral method for solving
summations.
831
01:14:38,629 --> 01:14:43,220
Either way you can prove.
The integral method actually
832
01:14:43,220 --> 01:14:46,165
gets you a tighter bound than
this.
833
01:14:46,165 --> 01:14:50,582
This is a basic fact,
and you should go off and know
834
01:14:50,582 --> 01:14:55,000
how to do that.
Now let's do substitution.
835
01:15:06,000 --> 01:15:18,821
The expectation of T(n) is less
than or equal to 2/n times the
836
01:15:18,821 --> 01:15:29,331
summation k=2 to n-1,
now we do the substitution of
837
01:15:29,331 --> 01:15:39,000
ak lg k, the smaller values plus
Theta(n).
838
01:15:39,000 --> 01:15:42,416
I might mentioned,
by the way, that the hard part
839
01:15:42,416 --> 01:15:45,334
of doing this,
it is easy to get the bound
840
01:15:45,334 --> 01:15:48,679
without this term,
it is easy to get this bound,
841
01:15:48,679 --> 01:15:51,669
1/2n^2 lg n,
it is harder to get the second
842
01:15:51,669 --> 01:15:54,231
order term.
It turns out you need the
843
01:15:54,231 --> 01:15:59,000
second order term in order to do
what we are going to do.
844
01:15:59,000 --> 01:16:05,621
You have to be able to subtract
a quadratic amount of the n^2 lg
845
01:16:05,621 --> 01:16:09,195
n in order to make this proof
work.
846
01:16:09,195 --> 01:16:15,291
And that is the trickier part
in evaluating that summation.
847
01:16:15,291 --> 01:16:20,126
So, we get this.
That is less than or equal to?
848
01:16:20,126 --> 01:16:26,537
Well, I happen to know how much
this is by using that formula.
849
01:16:26,537 --> 01:16:31,792
I use my fact and get 2a/n
(1/2n^2 lg n - 1/8n^2) +
850
01:16:31,792 --> 01:16:43,558
Theta(n).
Did I do something wrong?
851
01:16:43,558 --> 01:16:51,970
There we go.
Very good.
852
01:16:51,970 --> 01:17:05,272
That is equal to -
If I multiply this first part
853
01:17:05,272 --> 01:17:14,363
through that is an lg n.
And now, so I don't make a
854
01:17:14,363 --> 01:17:22,545
mistake, I want to express this
as my desired,
855
01:17:22,545 --> 01:17:32,000
this is what I want it to be,
minus a residual.
856
01:17:32,000 --> 01:17:38,702
I am going to write the
residual as this part.
857
01:17:38,702 --> 01:17:47,489
And so, the way to write that
is, that is going to be minus.
858
01:17:47,489 --> 01:17:56,723
And then it is going to be this
term here, which is going to be
859
01:17:56,723 --> 01:18:00,000
an/4 - Theta(n).
860
01:18:05,000 --> 01:18:12,309
And that is going to be less
than or equal to an lg n if this
861
01:18:12,309 --> 01:18:17,304
part is positive.
And I can make that part
862
01:18:17,304 --> 01:18:24,614
positive by picking a big enough
such that an/4 dominates the
863
01:18:24,614 --> 01:18:34,590
constant in the Theta(n) here.
Whatever the constant is here,
864
01:18:34,590 --> 01:18:45,211
I can find an a that is big
enough so that this term makes
865
01:18:45,211 --> 01:18:54,527
this part positive.
If a is big enough so that an/4
866
01:18:54,527 --> 01:19:01,217
dominates Theta(n).
And so the running time of
867
01:19:01,217 --> 01:19:04,288
randomized quicksort is order n
lg n.
868
01:19:04,288 --> 01:19:09,064
That is what we just proved,
the expected running time is
869
01:19:09,064 --> 01:19:11,623
order n lg n.
Now, in practice,
870
01:19:11,623 --> 01:19:16,741
quicksort is a great algorithm.
It is typically three or more
871
01:19:16,741 --> 01:19:21,688
times faster than mergesort.
It doesn't give you the strong
872
01:19:21,688 --> 01:19:26,464
guarantee necessarily of
mergesort and being worst-case n
873
01:19:26,464 --> 01:19:29,245
lg n.
But in practice,
874
01:19:29,245 --> 01:19:33,136
if you use randomized
quicksort, it is generally as
875
01:19:33,136 --> 01:19:37,573
much as three times faster.
It does require code tuning in
876
01:19:37,573 --> 01:19:40,219
order to get it up to be that
fast.
877
01:19:40,219 --> 01:19:44,966
You do have to go and coarsen
the base cases and do some other
878
01:19:44,966 --> 01:19:47,613
tricks there,
but most good sorting
879
01:19:47,613 --> 01:19:51,660
algorithms that you will find
are based on quicksort.
880
01:19:51,660 --> 01:19:56,018
Also one of the other reasons
it works well is because it
881
01:19:56,018 --> 01:20:01,000
tends to work well with caches
in virtual memory.
882
01:20:01,000 --> 01:20:05,431
We are not really talking much
about caching models and so
883
01:20:05,431 --> 01:20:09,941
forth, big topic these days in
algorithms, but it does work
884
01:20:09,941 --> 01:20:12,973
very well with caches in virtual
memory.
885
01:20:12,973 --> 01:20:17,405
It is another reason that this
is a good algorithm to use.
886
01:20:17,405 --> 01:20:20,359
Good recitation,
by the way, on Friday.
887
01:20:20,359 --> 01:20:24,247
We are going to see another n
log n time algorithm,
888
01:20:24,247 --> 01:20:27,000
a very important one in
recitation on Friday.