1
00:00:10 --> 00:00:13
My name is Erik Demaine.
You should call me Erik.
2
00:00:13 --> 00:00:16
Welcome back to 6.046.
This is Lecture 2.
3
00:00:16 --> 00:00:20
And today we are going to
essentially fill in some of the
4
00:00:20 --> 00:00:23
more mathematical underpinnings
of Lecture 1.
5
00:00:23 --> 00:00:26
So, Lecture 1,
we just sort of barely got our
6
00:00:26 --> 00:00:31
feet wet with some analysis of
algorithms, insertion sort and
7
00:00:31 --> 00:00:34
mergesort.
And we needed a couple of
8
00:00:34 --> 00:00:36
tools.
We had this big idea of
9
00:00:36 --> 00:00:40
asymptotics and forgetting about
constants, just looking at the
10
00:00:40 --> 00:00:41
lead term.
And so, today,
11
00:00:41 --> 00:00:44
we're going to develop
asymptotic notation so that we
12
00:00:44 --> 00:00:47
know that mathematically.
And we also ended up with a
13
00:00:47 --> 00:00:51
recurrence with mergesort,
the running time of mergesort,
14
00:00:51 --> 00:00:53
so we need to see how to solve
recurrences.
15
00:00:53 --> 00:00:55
And we will do those two things
today.
16
00:00:55 --> 00:00:58
Question?
Yes, I will speak louder.
17
00:00:58 --> 00:01:00
Thanks.
Good.
18
00:01:00 --> 00:01:03
Even though I have a
microphone, I am not amplified.
19
00:01:03 --> 00:01:07
OK, so let's start with
asymptotic notation.
20
00:01:07 --> 00:01:16
21
00:01:16 --> 00:01:19
We have seen some basic
asymptotic notation.
22
00:01:19 --> 00:01:22
I am sure you have seen it in
other classes before,
23
00:01:22 --> 00:01:26
things like big O-notation.
And today we are going to
24
00:01:26 --> 00:01:30
really define this rigorously so
we know what is true and what is
25
00:01:30 --> 00:01:34
not, what is valid and what is
not.
26
00:01:34 --> 00:01:40
27
00:01:40 --> 00:01:43
We are going to define,
and unfortunately today is
28
00:01:43 --> 00:01:48
going to be really mathematical
and really no algorithms today,
29
00:01:48 --> 00:01:52
which is sort of an anticlimax.
But next lecture we will talk
30
00:01:52 --> 00:01:57
about real algorithms and will
apply all the things we learned
31
00:01:57 --> 00:02:04
today to real algorithms.
This is big O-notation,
32
00:02:04 --> 00:02:12
capital O-notation.
We have f(n)=O[g(n)].
33
00:02:12 --> 00:02:21
This means that there are some
suitable constants,
34
00:02:21 --> 00:02:30
c and n_o, such that f is
bounded by cg(n) for all
35
00:02:30 --> 00:02:38
sufficiently large n.
So, this is pretty intuitive
36
00:02:38 --> 00:02:42
notion.
We have seen it before.
37
00:02:42 --> 00:02:47
We are going to assume that
f(n) is non-negative here.
38
00:02:47 --> 00:02:52
And I just want f(n) to be
bounded above by g(n).
39
00:02:52 --> 00:02:57
We have seen a bunch of
examples, but something like
40
00:02:57 --> 00:03:02
2n^2=O(n^3) defined.
And roughly this means if you
41
00:03:02 --> 00:03:07
drop leading constants and low
order terms then this is less
42
00:03:07 --> 00:03:10
than or equal to that.
So, big O corresponds roughly
43
00:03:10 --> 00:03:14
to less than or equal to.
But this is the formalization.
44
00:03:14 --> 00:03:18
Another way to think of it
formally, a funny thing about
45
00:03:18 --> 00:03:20
this notation is it is
asymmetric.
46
00:03:20 --> 00:03:23
Normally, you think of
equality being symmetric.
47
00:03:23 --> 00:03:26
If A=B then B=A.
But it's not true here.
48
00:03:26 --> 00:03:30
We do not have n^3 being big O
of n^2.
49
00:03:30 --> 00:03:33
We don't even have big O of n^3
equaling n^2.
50
00:03:33 --> 00:03:38
So, we will see exactly what
that means in a second.
51
00:03:38 --> 00:03:43
But before we get there,
this is a bit bizarre notation
52
00:03:43 --> 00:03:47
and you should always think
about what it really means.
53
00:03:47 --> 00:03:53
Another way to think about what
it really means is that f(n) is
54
00:03:53 --> 00:03:58
in some set of functions that
are like g.
55
00:03:58 --> 00:04:04
You could define big O[g(n)] to
be a set of functions,
56
00:04:04 --> 00:04:10
let's call it f(n),
such that there exist
57
00:04:10 --> 00:04:15
constants.
They are the same definition,
58
00:04:15 --> 00:04:21
I think, fancy here,
c and n_o, such that we have
59
00:04:21 --> 00:04:28
the bound f(n) is between zero
and cg(n).
60
00:04:28 --> 00:04:34
61
00:04:34 --> 00:04:37
It is a bit of a long
definition, and that is why we
62
00:04:37 --> 00:04:41
use the notation,
to avoid having to write this
63
00:04:41 --> 00:04:44
over and over.
You can think of instead of n^2
64
00:04:44 --> 00:04:48
being equal to big O of n^3,
what we really mean is that
65
00:04:48 --> 00:04:52
2n^2 is in the set big O(n^3).
When we write equal sign,
66
00:04:52 --> 00:04:56
we in some sense mean this in
the set, but we are going to use
67
00:04:56 --> 00:05:00
equal sign.
You could write this.
68
00:05:00 --> 00:05:03
And occasionally you see papers
that write this,
69
00:05:03 --> 00:05:08
but this is the notation that
we are going to use.
70
00:05:08 --> 00:05:12
That has the consequence the
equal sign is asymmetric,
71
00:05:12 --> 00:05:17
just like this operator.
We have some nifty ways that we
72
00:05:17 --> 00:05:20
actually use big O-notation.
73
00:05:20 --> 00:05:29
74
00:05:29 --> 00:05:33
And it is using it as a macro.
By the way, we have a lot to
75
00:05:33 --> 00:05:37
cover today, so I am going to go
relatively fast.
76
00:05:37 --> 00:05:41
If anything is unclear,
just stop, ask questions,
77
00:05:41 --> 00:05:45
then I will slow down.
Otherwise, I will take this as
78
00:05:45 --> 00:05:50
all completely obvious and I can
keep going at full speed.
79
00:05:50 --> 00:05:52
The convention,
this is intuitive,
80
00:05:52 --> 00:05:57
I guess, if you do some macro
programming or something,
81
00:05:57 --> 00:06:01
but it's a bit more
mathematical.
82
00:06:01 --> 00:06:09
83
00:06:09 --> 00:06:13
We have defined big O-notation
and it equals big O of
84
00:06:13 --> 00:06:16
something.
And so we have only defined big
85
00:06:16 --> 00:06:20
O when on the equal sign we have
big O of some function.
86
00:06:20 --> 00:06:24
But it is useful to have some
general expression on the
87
00:06:24 --> 00:06:28
right-hand side that involves
big O.
88
00:06:28 --> 00:06:34
For example,
let's say we have f(n) = n^3 +
89
00:06:34 --> 00:06:39
O(n^2).
This is attempting to get an
90
00:06:39 --> 00:06:45
error bound.
This is saying f(n) is
91
00:06:45 --> 00:06:54
basically n^3 but there are
these lower order terms that are
92
00:06:54 --> 00:07:00
O(n^2).
And so this means that there is
93
00:07:00 --> 00:07:08
a function, shorthand for a
function, h(n) which is in
94
00:07:08 --> 00:07:18
O(n^2) or equals O(n^2) such
that f(n) = n^3 + h(n).
95
00:07:18 --> 00:07:21
It is saying that there are
some lower order terms that are
96
00:07:21 --> 00:07:25
bounded above by some constant
times n^2 for sufficiently large
97
00:07:25 --> 00:07:28
n, and that is what is here.
And then f(n) equals,
98
00:07:28 --> 00:07:32
now this is a true equality,
n^3 plus that error term.
99
00:07:32 --> 00:07:34
This is very useful here.
Essentially,
100
00:07:34 --> 00:07:37
I am expressing what the lead
constant is and then saying
101
00:07:37 --> 00:07:40
well, there is other stuff and
it's all at most n^2.
102
00:07:40 --> 00:07:42
Saying that f(n) therefore is
also order n^3,
103
00:07:42 --> 00:07:44
but that is a bit weaker of a
statement.
104
00:07:44 --> 00:07:47
This is a bit more refined.
We won't need to use this too
105
00:07:47 --> 00:07:50
often, but it is useful.
Sometimes we will see,
106
00:07:50 --> 00:07:53
like in last class we even had
a big O inside a summation.
107
00:07:53 --> 00:07:55
So, you can use them all over
the place.
108
00:07:55 --> 00:08:00
The point is they represent
some function in that set.
109
00:08:00 --> 00:08:04
A bit less intuitive,
and this is more subtle,
110
00:08:04 --> 00:08:10
is what it means to have big O
on the left-hand side.
111
00:08:10 --> 00:08:15
It means the same thing,
but there is some convention
112
00:08:15 --> 00:08:20
what equality means.
And this is why equal sign is
113
00:08:20 --> 00:08:24
asymmetric.
You should read equals like
114
00:08:24 --> 00:08:27
"is".
Is means that everything over
115
00:08:27 --> 00:08:34
here is something over here.
So, there is an implicit for
116
00:08:34 --> 00:08:39
all on the left-hand side and
there exists on the right-hand
117
00:08:39 --> 00:08:41
side.
This is a true statement.
118
00:08:41 --> 00:08:47
Anything that is n^2 + O(n) is
also O(n^2), but not the other
119
00:08:47 --> 00:08:50
way around.
So, this is a bit asymmetric.
120
00:08:50 --> 00:08:55
If you think about it,
this is pretty intuitive but it
121
00:08:55 --> 00:08:59
is subtle so you should be
careful.
122
00:08:59 --> 00:09:15
123
00:09:15 --> 00:09:20
This says for any expansion of
the macro on the left-hand side,
124
00:09:20 --> 00:09:24
which should be f(n),
there is an expansion of the
125
00:09:24 --> 00:09:30
macro on the right-hand side
such that we get equality.
126
00:09:30 --> 00:09:34
And what this allows you to do
is if you have a chain of equal
127
00:09:34 --> 00:09:36
signs relations,
a chain of "is"s,
128
00:09:36 --> 00:09:40
then the very first one is
equal to or bounded by the very
129
00:09:40 --> 00:09:42
last one.
So, you can chain equal signs
130
00:09:42 --> 00:09:45
the way you normally would.
You just cannot flip them
131
00:09:45 --> 00:09:46
around.
Good.
132
00:09:46 --> 00:09:51
So, that's big O-notation.
Any questions about that?
133
00:09:51 --> 00:09:58
134
00:09:58 --> 00:10:01
So, big O is great for
expressing upper bounds.
135
00:10:01 --> 00:10:04
But we also want to talk about
lower bounds.
136
00:10:04 --> 00:10:06
For algorithms,
we usually care about upper
137
00:10:06 --> 00:10:11
bounds on their running time.
Running times at most n^2 is at
138
00:10:11 --> 00:10:14
most n log n up to big O,
but sometimes we need to
139
00:10:14 --> 00:10:17
express functions that are at
least some quantity.
140
00:10:17 --> 00:10:20
For example,
we will show that sorting
141
00:10:20 --> 00:10:23
requires at least n log n time
in some model.
142
00:10:23 --> 00:10:26
So, we need some other notation
for that.
143
00:10:26 --> 00:10:30
And the notation is big
Omega-notation.
144
00:10:30 --> 00:10:35
And it is pretty symmetric.
I will just write out the set
145
00:10:35 --> 00:10:40
definition here.
And we are going to write f(n)=
146
00:10:40 --> 00:10:46
big Omega[g(n)] to mean f(n) is
at least some constant times
147
00:10:46 --> 00:10:48
g(n) --
148
00:10:48 --> 00:10:55
149
00:10:55 --> 00:10:57
-- for sufficiently large n.
150
00:10:57 --> 00:11:09
151
00:11:09 --> 00:11:12
So, I am basically just
reversing the inequality
152
00:11:12 --> 00:11:16
relation between f and g,
nothing surprising,
153
00:11:16 --> 00:11:19
just to have it there.
A random example,
154
00:11:19 --> 00:11:24
and now we will get a little
bit more sophisticated,
155
00:11:24 --> 00:11:29
root n= big Omega(lg n).
And you should read this that
156
00:11:29 --> 00:11:34
up to constant factors root n is
at least log n for sufficiently
157
00:11:34 --> 00:11:38
large n.
So, omega sort of corresponds
158
00:11:38 --> 00:11:43
to greater than or equal to.
Let me give you some analogies.
159
00:11:43 --> 00:11:46
We have big O,
we have big omega,
160
00:11:46 --> 00:11:51
this is less than or equal to,
this is greater than or equal
161
00:11:51 --> 00:11:53
to.
And I am going to fill in some
162
00:11:53 --> 00:11:57
more here in a moment.
163
00:11:57 --> 00:12:10
164
00:12:10 --> 00:12:12
It's nice to have all the usual
operators we have.
165
00:12:12 --> 00:12:16
Normally we have strict less
than, strict greater than and
166
00:12:16 --> 00:12:18
equal sign.
And we want those sort of
167
00:12:18 --> 00:12:22
analogs in the asymptotic world
where we ignore constant factors
168
00:12:22 --> 00:12:25
and ignore lower order terms.
We have, for example,
169
00:12:25 --> 00:12:28
big Theta[g(n)].
This is a capital theta which
170
00:12:28 --> 00:12:31
means you write the horizontal
bar in the middle as opposed to
171
00:12:31 --> 00:12:35
all the way through.
I didn't invent Greek,
172
00:12:35 --> 00:12:40
so that is the way it is.
Theta means that you are less
173
00:12:40 --> 00:12:44
than or equal to and you are
greater than or equal to up to
174
00:12:44 --> 00:12:48
constant factors,
so it is the inner section of
175
00:12:48 --> 00:12:51
these two sets,
big O and big Omega.
176
00:12:51 --> 00:12:55
That is sort of like equal sign
but, of course,
177
00:12:55 --> 00:13:00
this is very different.
You have things like n^2 is big
178
00:13:00 --> 00:13:04
Theta of 2(n^2) because you
ignore constant factors,
179
00:13:04 --> 00:13:07
but all of these other
relations, OK,
180
00:13:07 --> 00:13:11
n^2 + O(n) = Theta(n^2),
but this does not hold with
181
00:13:11 --> 00:13:16
theta because square root of n
is really asymptotically bigger
182
00:13:16 --> 00:13:19
than log n.
And some of the other examples
183
00:13:19 --> 00:13:25
we saw like n^2 versus n^3,
those don't hold with T.
184
00:13:25 --> 00:13:29
And we have some strict
notation which are the little
185
00:13:29 --> 00:13:32
o-notation and little
omega-notation.
186
00:13:32 --> 00:13:37
There is no little theta
because there is not notion of
187
00:13:37 --> 00:13:41
strict equality versus unstrict
equality.
188
00:13:41 --> 00:13:47
Little o is going to correspond
roughly to less than and little
189
00:13:47 --> 00:13:51
omega is going to correspond to
greater than.
190
00:13:51 --> 00:13:57
This is a notation you will
just have to get used to.
191
00:13:57 --> 00:14:03
And I am not going to define it
precisely here because it is
192
00:14:03 --> 00:14:08
almost exactly the same.
The difference is that instead
193
00:14:08 --> 00:14:15
of saying there exists constant
c and n_o, you have to say for
194
00:14:15 --> 00:14:19
every constant c there exists a
constant n_o.
195
00:14:19 --> 00:14:26
The relationship between f and
g, this inequality must hold for
196
00:14:26 --> 00:14:32
all c instead of just for 1.
And so n_o can now depend on c.
197
00:14:32 --> 00:14:37
You can assume that really n is
sufficiently large,
198
00:14:37 --> 00:14:40
but this gives you a strict
inequality.
199
00:14:40 --> 00:14:45
No matter what constant you put
here, in front of g,
200
00:14:45 --> 00:14:50
let's say we are doing little
o, f will be still less than c
201
00:14:50 --> 00:14:53
times g for sufficiently large
n.
202
00:14:53 --> 00:14:57
We have some random examples.
203
00:14:57 --> 00:15:04
204
00:15:04 --> 00:15:06
We are again ignoring
constants.
205
00:15:06 --> 00:15:11
n^2 is always less than n^3 for
sufficiently large n.
206
00:15:11 --> 00:15:15
And it is a bit subtle here.
I mean in order to prove
207
00:15:15 --> 00:15:19
something like this,
it will become intuitive after
208
00:15:19 --> 00:15:25
you manipulate it a little bit.
You have to figure out what n_o
209
00:15:25 --> 00:15:30
is in terms of c.
I think it something like 2/c.
210
00:15:30 --> 00:15:34
If we have less than or equal
to, that should be right.
211
00:15:34 --> 00:15:38
As long n is at least this big,
no matter how small of a c,
212
00:15:38 --> 00:15:42
you should think of c here as
being epsilon now,
213
00:15:42 --> 00:15:44
in the usual epsilon and
deltas.
214
00:15:44 --> 00:15:49
No matter how small c gets,
still I can bound n^2 in terms
215
00:15:49 --> 00:15:53
of n^3, upper bound,
but whenever you have theta you
216
00:15:53 --> 00:15:57
do not have either of these
relations.
217
00:15:57 --> 00:16:00
For example,
½n^2 = Theta(n^2) and it is not
218
00:16:00 --> 00:16:06
little o(n^2) and it not little
omega(n^2) because it is exactly
219
00:16:06 --> 00:16:08
n^2.
You will get some sense in
220
00:16:08 --> 00:16:13
order relation out of this,
although there are some messy
221
00:16:13 --> 00:16:17
behaviors as you will see in
your problem set.
222
00:16:17 --> 00:16:21
Any questions about asymptotic
notation?
223
00:16:21 --> 00:16:25
That is the quick rundown.
Now we are going to use it to
224
00:16:25 --> 00:16:31
solve some recurrences.
Although we won't use it that
225
00:16:31 --> 00:16:35
much today, we will use it a lot
more on Wednesday.
226
00:16:35 --> 00:16:36
OK.
227
00:16:36 --> 00:16:53
228
00:16:53 --> 00:16:57
We will move onto the second
topic of today,
229
00:16:57 --> 00:17:02
which is solving recurrences.
You have probably solved some
230
00:17:02 --> 00:17:06
recurrences before in 6.042 or
whatever discrete math class you
231
00:17:06 --> 00:17:09
have taken.
We are going to do more and
232
00:17:09 --> 00:17:14
have some techniques here that
are particularly useful for
233
00:17:14 --> 00:17:18
analyzing recursive algorithms,
and we will see that mostly on
234
00:17:18 --> 00:17:21
Wednesday.
There are three main methods
235
00:17:21 --> 00:17:25
that we are going to use here
for solving recurrences.
236
00:17:25 --> 00:17:30
The first one is the
substitution method.
237
00:17:30 --> 00:17:32
There is no general procedure
for solving a recurrence.
238
00:17:32 --> 00:17:35
There is no good algorithm for
solving recurrences,
239
00:17:35 --> 00:17:37
unfortunately.
We just have a bunch of
240
00:17:37 --> 00:17:39
techniques.
Some of them work some of the
241
00:17:39 --> 00:17:42
time, and if you are lucky yours
will work for your recurrence,
242
00:17:42 --> 00:17:44
but it is sort of like solving
an integral.
243
00:17:44 --> 00:17:47
You have to just know some of
them, you have to know various
244
00:17:47 --> 00:17:50
methods for solving them.
It is usually easy to check if
245
00:17:50 --> 00:17:53
you have the right answer.
Just like with integrals,
246
00:17:53 --> 00:17:56
you just differentiate and say
oh, I got the right answer.
247
00:17:56 --> 00:18:00
And that is essentially the
idea of substitution method.
248
00:18:00 --> 00:18:04
Substitution method will always
work, but unfortunately Step 1
249
00:18:04 --> 00:18:07
is guess the answer.
And you have to guess it
250
00:18:07 --> 00:18:09
correctly.
That makes it a big difficult.
251
00:18:09 --> 00:18:12
You don't have to guess it
completely.
252
00:18:12 --> 00:18:15
You can usually get away with
not knowing the constant
253
00:18:15 --> 00:18:19
factors, which is a good thing
because we don't really care
254
00:18:19 --> 00:18:22
about the constant factors.
You guess the form.
255
00:18:22 --> 00:18:26
You say oh, it is going to be
roughly n^2, and so it's some
256
00:18:26 --> 00:18:31
constant times n^2 presumably.
So, you guess that.
257
00:18:31 --> 00:18:34
We are going to figure out the
constants.
258
00:18:34 --> 00:18:38
You try to verify whether the
recurrence satisfies this bound
259
00:18:38 --> 00:18:40
by induction,
and that is the key.
260
00:18:40 --> 00:18:44
Substitution uses induction.
And from that you usually get
261
00:18:44 --> 00:18:48
the constants for free.
You figure out what the
262
00:18:48 --> 00:18:51
constants have to be in order to
make this work.
263
00:18:51 --> 00:18:55
So, that is the general idea.
You will see a few examples of
264
00:18:55 --> 00:18:57
this.
Actually, the same example
265
00:18:57 --> 00:19:01
several times.
Unfortunately,
266
00:19:01 --> 00:19:04
this is what you might call,
I don't know.
267
00:19:04 --> 00:19:08
This is an algorithm,
but it uses an oracle which is
268
00:19:08 --> 00:19:12
knowing the right answer.
But sometimes it is not too
269
00:19:12 --> 00:19:14
hard to guess the answer.
It depends.
270
00:19:14 --> 00:19:18
If you look at this recurrence,
T(n) = 4T(n/2) + n,
271
00:19:18 --> 00:19:23
we should implicitly always
have some base case of T of some
272
00:19:23 --> 00:19:26
constant, usually 1 is a
constant, so we don't really
273
00:19:26 --> 00:19:32
care about the base case.
For algorithms that is always
274
00:19:32 --> 00:19:34
the case.
And we want to solve this
275
00:19:34 --> 00:19:37
thing.
Does anyone have a guess to
276
00:19:37 --> 00:19:40
what the solution is?
Ideally someone who doesn't
277
00:19:40 --> 00:19:43
already know how to solve this
recurrence.
278
00:19:43 --> 00:19:45
OK.
How many people know how to
279
00:19:45 --> 00:19:48
solve this recurrence?
A few, OK.
280
00:19:48 --> 00:19:50
And, of the rest,
any guesses?
281
00:19:50 --> 00:19:55
If you look at what is going on
here, here you have T(n/2) and
282
00:19:55 --> 00:19:59
let's ignore this term more or
less.
283
00:19:59 --> 00:20:02
We have n/2 here.
If we double n and get T(n)
284
00:20:02 --> 00:20:06
then we multiply the value by 4.
And then there is this additive
285
00:20:06 --> 00:20:08
end, but that doesn't matter so
much.
286
00:20:08 --> 00:20:13
What function do you know that
when you double the argument the
287
00:20:13 --> 00:20:15
output goes up by a factor of 4?
Sorry?
288
00:20:15 --> 00:20:18
n^2,yeah.
You should think n^2 and you
289
00:20:18 --> 00:20:21
would be right.
But we won't prove n^2 yet.
290
00:20:21 --> 00:20:25
Let's prove something simpler,
because it turns out proving
291
00:20:25 --> 00:20:29
that it is at most n^2 is a bit
of a pain.
292
00:20:29 --> 00:20:31
We will see that in just a few
minutes.
293
00:20:31 --> 00:20:36
But let's guess that T(n) =
O(n^3) first because that will
294
00:20:36 --> 00:20:41
be easier to prove by induction.
You sort of see how it is done
295
00:20:41 --> 00:20:44
in the easy case,
and then we will actually get
296
00:20:44 --> 00:20:47
the right answer,
n^2, later.
297
00:20:47 --> 00:20:50
I need to prove.
What I am going to do is guess
298
00:20:50 --> 00:20:55
that T(n) is some constant times
n^3 at most, so I will be a
299
00:20:55 --> 00:20:59
little more precise.
I cannot use the big O-notation
300
00:20:59 --> 00:21:03
in the substitution method so I
have to expand it out to use
301
00:21:03 --> 00:21:08
constants.
I will show you why in a little
302
00:21:08 --> 00:21:13
bit, but let me just tell you at
a high level what is important
303
00:21:13 --> 00:21:17
in not using big O-notation.
Big O-notation is great if you
304
00:21:17 --> 00:21:21
have a finite chain of big O
relations, you know,
305
00:21:21 --> 00:21:25
n^2 is big O(n^3) is big O(n^4)
is big O(n^4) is big O(n^4).
306
00:21:25 --> 00:21:29
That is all true.
And so you get that n^2 is big
307
00:21:29 --> 00:21:33
O(n^4).
But if you have an infinite
308
00:21:33 --> 00:21:37
chain of those relations then
the first thing is not big O of
309
00:21:37 --> 00:21:41
the last thing.
You have to be very careful.
310
00:21:41 --> 00:21:44
For example,
this is a total aside on the
311
00:21:44 --> 00:21:47
lecture notes.
Suppose you want to prove that
312
00:21:47 --> 00:21:50
n = O(1).
This is a great relation.
313
00:21:50 --> 00:21:53
If it were true,
every algorithm would have
314
00:21:53 --> 00:21:56
constant running time.
This is not true.
315
00:21:56 --> 00:22:02
Not in Wayne's World notation.
You could "prove this by
316
00:22:02 --> 00:22:07
induction" by saying well,
base case is 1 = O(1).
317
00:22:07 --> 00:22:12
OK, that is true.
And then the induction step as
318
00:22:12 --> 00:22:18
well, if I know that n-1,
so let's suppose that n-1 =
319
00:22:18 --> 00:22:23
O(1), well, that implies that n,
which is (n-1) +1,
320
00:22:23 --> 00:22:30
if this is O(1) and 1 = O(1),
the whole thing is O(1).
321
00:22:30 --> 00:22:32
And that is true.
If you knew that (n-1) = O(1)
322
00:22:32 --> 00:22:35
and 1 = O(1) then their sum is
also O(1), but this is a false
323
00:22:35 --> 00:22:37
proof.
You cannot induct over big Os.
324
00:22:37 --> 00:22:40
What is going on here is that
the constants that are implicit
325
00:22:40 --> 00:22:43
in here are changing.
Here you have some big O of 1,
326
00:22:43 --> 00:22:46
here you have some big O of 1.
You are probably doubling the
327
00:22:46 --> 00:22:49
constant in there every time you
do this relation.
328
00:22:49 --> 00:22:52
If you have a finite number of
doubling of constants,
329
00:22:52 --> 00:22:54
no big deal,
it is just a constant,
330
00:22:54 --> 00:22:55
two the power number of
doublings.
331
00:22:55 --> 00:23:00
But here you are doing n
doublings and that is no good.
332
00:23:00 --> 00:23:02
The constant is now depending
on n.
333
00:23:02 --> 00:23:06
So, we are avoiding this kind
of problem by writing out the
334
00:23:06 --> 00:23:08
constant.
We have to make sure that
335
00:23:08 --> 00:23:11
constant doesn't change.
Good.
336
00:23:11 --> 00:23:13
Now I have written out the
constant.
337
00:23:13 --> 00:23:16
I should be safe.
I am assuming it for all k less
338
00:23:16 --> 00:23:20
than n, now I have to prove it
for k equal to n.
339
00:23:20 --> 00:23:23
I am going to take T(n) and
just expand it.
340
00:23:23 --> 00:23:25
I am going to do the obvious
thing.
341
00:23:25 --> 00:23:30
I have this recurrence how to
expand T(n).
342
00:23:30 --> 00:23:35
Then it involves T(n/2).
And I know some fact about
343
00:23:35 --> 00:23:39
T(n/2) because n/2 is less than
n.
344
00:23:39 --> 00:23:43
So, let's expand.
T(n) = 4T(n/2) + n.
345
00:23:43 --> 00:23:50
And now I have an upper bound
on this thing from the induction
346
00:23:50 --> 00:23:55
hypothesis.
This is at most 4 times c times
347
00:23:55 --> 00:24:00
the argument cubed plus n.
348
00:24:00 --> 00:24:40
349
00:24:40 --> 00:24:48
Continuing on here.
Let's expand this a little bit.
350
00:24:48 --> 00:24:56
We have n cubed over 2 cubed.
Two cubed is 8,
351
00:24:56 --> 00:25:05
so 4 over 8 is a half.
So, we have ½cn^3 + n.
352
00:25:05 --> 00:25:10
And what I would like this to
be is, so at the bottom where I
353
00:25:10 --> 00:25:13
would like to go is that this is
at most cn3.
354
00:25:13 --> 00:25:18
That is what I would like to
prove to reestablish the
355
00:25:18 --> 00:25:21
induction hypothesis for n.
What I will do,
356
00:25:21 --> 00:25:26
in order to see when that is
case, is just write this as what
357
00:25:26 --> 00:25:30
I want, so this is sort of the
desired value,
358
00:25:30 --> 00:25:34
cn3, minus whatever I don't
want.
359
00:25:34 --> 00:25:39
This is called the residual.
Now I have to actually figure
360
00:25:39 --> 00:25:41
this out.
Let's see.
361
00:25:41 --> 00:25:44
We have cn^3,
but only ½cn^3 here,
362
00:25:44 --> 00:25:49
so I need to subtract off ½cn^3
to get that lead term correct.
363
00:25:49 --> 00:25:54
And then I have plus n and
there is a minus here,
364
00:25:54 --> 00:25:59
so it is minus n.
And that is the residual.
365
00:25:59 --> 00:26:04
In order for this to be at most
this, I need that the residual
366
00:26:04 --> 00:26:07
is non-negative.
This is if the residual part is
367
00:26:07 --> 00:26:12
greater than or equal to zero,
which is pretty easy to do
368
00:26:12 --> 00:26:15
because here I have control over
c.
369
00:26:15 --> 00:26:18
I get to pick c to be whatever
I want.
370
00:26:18 --> 00:26:22
And, as long as c is at least,
oh, I don't know,
371
00:26:22 --> 00:26:26
2, then this is a 1 at least.
Then I have n^3 should be
372
00:26:26 --> 00:26:33
greater than or equal to n.
And that is always the case.
373
00:26:33 --> 00:26:37
For example,
this is true if c is at least
374
00:26:37 --> 1.
375
1. --> 00:26:41
And I don't think it matters
376
00:26:41 --> 00:26:47
what n is, but let's say n is at
least 1 just for kicks.
377
00:26:47 --> 00:26:53
So, what we have done is proved
that T(n) is at most some
378
00:26:53 --> 00:27:00
constant times n^3.
And the constant is like 1.
379
00:27:00 --> 00:27:03
So, that is an upper bound.
It is not a tight upper bound.
380
00:27:03 --> 00:27:06
We actually believed that it is
n^2, and it is,
381
00:27:06 --> 00:27:09
but you have to be a little
careful.
382
00:27:09 --> 00:27:11
This does not mean that the
answer is n^3.
383
00:27:11 --> 00:27:14
It just means that at most n^3
is big O(n^3).
384
00:27:14 --> 00:27:16
And this is a proof by
induction.
385
00:27:16 --> 00:27:20
Now, technically I should have
put a base case in this
386
00:27:20 --> 00:27:22
induction, so there is a little
bit missing.
387
00:27:22 --> 00:27:26
The base case is pretty easy
because T(1) is some constant,
388
00:27:26 --> 00:27:29
but it will sort of influence
things.
389
00:27:29 --> 00:27:33
If the base case T(1) is some
constant.
390
00:27:33 --> 00:27:37
And what we need is that it is
at most c times one cubed,
391
00:27:37 --> 00:27:40
which is c.
And that will be true as long
392
00:27:40 --> 00:27:43
as you choose c to be
sufficiently large.
393
00:27:43 --> 00:27:47
So, this is true if c is chosen
sufficiently large.
394
00:27:47 --> 00:27:52
Now, we don't care about
constants, but the point is just
395
00:27:52 --> 00:27:56
to be a little bit careful.
It is not true that T(n) is at
396
00:27:56 --> 00:28:00
most 1 times n^2,
even though here all we need is
397
00:28:00 --> 00:28:05
that c is at least 1.
For the base case to work,
398
00:28:05 --> 00:28:10
c actually might have to be a
hundred or whatever T(1) is.
399
00:28:10 --> 00:28:14
So, be a little bit careful
there.
400
00:28:14 --> 00:28:19
It doesn't really affect the
answer, usually it won't because
401
00:28:19 --> 00:28:23
we have very simple base cases
here.
402
00:28:23 --> 00:28:29
OK, so let's try to prove the
tight bound of O(n^2).
403
00:28:29 --> 00:28:33
I am not going to prove an
omega bound, but you can prove
404
00:28:33 --> 00:28:38
an omega n squared bound as well
using substitution method.
405
00:28:38 --> 00:28:43
I will just be satisfied for
now proving an upper bound of n
406
00:28:43 --> 00:28:46
squared.
Let's try to prove that T(n),
407
00:28:46 --> 00:28:51
this is the same recurrence,
I want to prove that it is
408
00:28:51 --> 00:28:53
O(n^2).
I am going to do the same
409
00:28:53 --> 00:28:56
thing.
And I will write a bit faster
410
00:28:56 --> 00:29:01
because this is basically
copying.
411
00:29:01 --> 00:29:06
412
00:29:06 --> 00:29:10
Except now, instead of three,
I have two.
413
00:29:10 --> 00:29:17
Then I have T(n) = 4T(n/2) + n.
I expand this T(n/2).
414
00:29:17 --> 00:29:24
This is at most 4c(n/2)^2 + n.
And now, instead of have 2
415
00:29:24 --> 00:29:30
cubed, I have 2 squared,
which is only 4.
416
00:29:30 --> 00:29:32
The fours cancel.
I get cn^2 + n.
417
00:29:32 --> 00:29:37
And if you prefer to write it
as desired minus residual,
418
00:29:37 --> 00:29:42
then I have cn^2 - (-n).
And I want this to be
419
00:29:42 --> 00:29:46
non-negative.
And it is damn hard for minus n
420
00:29:46 --> 00:29:50
to be non-negative.
If n is zero we are happy,
421
00:29:50 --> 00:29:54
but unfortunately this is an
induction on n.
422
00:29:54 --> 00:30:00
It's got to hold for all n
greater than or equal to 1.
423
00:30:00 --> 00:30:02
This is not less than or equal
to cn^2.
424
00:30:02 --> 00:30:06
Notice the temptation is to
write that this equals O(n^2),
425
00:30:06 --> 00:30:09
which is true for this one
step.
426
00:30:09 --> 00:30:12
cn^2 - (-n),
well, these are both order n,
427
00:30:12 --> 00:30:15
or this is order n,
this is order n squared.
428
00:30:15 --> 00:30:18
Certainly this thing is O(n^2),
that is true,
429
00:30:18 --> 00:30:21
but it is not completing the
induction.
430
00:30:21 --> 00:30:25
To complete the induction,
you have to prove the induction
431
00:30:25 --> 00:30:29
hypothesis for n with this
constant c.
432
00:30:29 --> 00:30:32
Here you are getting a constant
c of like c + 1,
433
00:30:32 --> 00:30:36
which is not good.
This is true but useless.
434
00:30:36 --> 00:30:40
It does not finish the
induction, so you can sort of
435
00:30:40 --> 00:30:42
ignore that.
This proof doesn't work,
436
00:30:42 --> 00:30:46
which is kind of annoying
because we feel,
437
00:30:46 --> 00:30:49
in our heart of hearts,
that T(n) = n^2.
438
00:30:49 --> 00:30:53
It turns out to fix this you
need to express T(n) in a
439
00:30:53 --> 00:30:56
slightly different form.
This is, again,
440
00:30:56 --> 00:31:00
divine inspiration.
And, if you have a good
441
00:31:00 --> 00:31:03
connection to some divinity,
you are all set.
442
00:31:03 --> 00:31:06
[LAUGHTER] But it is a little
bit harder for the rest of us
443
00:31:06 --> 00:31:08
mere mortals.
It turns out,
444
00:31:08 --> 00:31:12
and maybe you could guess this,
that the idea is we want to
445
00:31:12 --> 00:31:14
strengthen the induction
hypothesis.
446
00:31:14 --> 00:31:18
We assumed this relatively weak
thing, T(k) is less than or
447
00:31:18 --> 00:31:20
equal to some constant times
k^2.
448
00:31:20 --> 00:31:22
We didn't know what the
constant was,
449
00:31:22 --> 00:31:25
that is fine,
but we assumed that there were
450
00:31:25 --> 00:31:28
no lower order terms.
I want to look at lower order
451
00:31:28 --> 00:31:31
terms.
Maybe they play a role.
452
00:31:31 --> 00:31:33
And if you look at this
progression you say,
453
00:31:33 --> 00:31:36
oh, well, I am getting
something like n^2 and the
454
00:31:36 --> 00:31:40
constants are pretty damn tight.
I mean the fours are canceling
455
00:31:40 --> 00:31:43
and the c just is preserved.
How am I going to get rid of
456
00:31:43 --> 00:31:46
this lower order term plus n?
Well, maybe I could subtract
457
00:31:46 --> 00:31:49
off a linear term in here and,
if I am lucky,
458
00:31:49 --> 00:31:52
it will cancel with this one.
That is all the intuition we
459
00:31:52 --> 00:31:56
have at this point.
It turns out it works.
460
00:31:56 --> 00:32:01
We look at T(n) and this is
4T(n/2) + n as usual.
461
00:32:01 --> 00:32:05
Now we expand a slightly
messier form.
462
00:32:05 --> 00:32:09
We have 4[c_1*(n/2)^2 -
c_2*(n/2)] + n.
463
00:32:09 --> 00:32:14
This part is the same because
the fours cancel again.
464
00:32:14 --> 00:32:18
So, we get c_1*n^2,
which is good.
465
00:32:18 --> 00:32:22
I mean that is sort of the form
we want.
466
00:32:22 --> 00:32:30
Then we have something times n,
so let's figure it out.
467
00:32:30 --> 00:32:34
We have a plus 1 times n,
so let's write it 1 minus c_2
468
00:32:34 --> 00:32:38
over 2 times n.
Oops, got that wrong.
469
00:32:38 --> 00:32:43
There is four times a two so,
in fact, the two is upstairs.
470
00:32:43 --> 00:32:46
Let me double check.
Right.
471
00:32:46 --> 00:32:48
OK.
Now we can write this as
472
00:32:48 --> 00:32:53
desired minus residual.
And we have to be a little
473
00:32:53 --> 00:32:58
careful here because now we have
a stronger induction hypothesis
474
00:32:58 --> 00:33:03
to prove.
We don't just need it is at
475
00:33:03 --> 00:33:07
most c_1*n^2,
which would be fine here
476
00:33:07 --> 00:33:12
because we could choose c_2 to
be large, but what we really
477
00:33:12 --> 00:33:17
need is c_1*n^2 - c_2*n,
and then minus some other
478
00:33:17 --> 00:33:19
stuff.
This is, again,
479
00:33:19 --> 00:33:23
desired minus residual.
And minus residual,
480
00:33:23 --> 00:33:30
let's see, we have a minus 1
and we have a minus c_2.
481
00:33:30 --> 00:33:35
That doesn't look so happy.
Plus c_2, thank you,
482
00:33:35 --> 00:33:40
because that again looked
awfully negative.
483
00:33:40 --> 00:33:44
It is plus c_2.
I am getting my signs,
484
00:33:44 --> 00:33:50
there is a minus here and there
is one minus here,
485
00:33:50 --> 00:33:55
so there we go.
Again, I want my residual to be
486
00:33:55 --> 00:34:03
greater than or equal to zero.
And if I have that I will be
487
00:34:03 --> 00:34:07
all set in making this inductive
argument.
488
00:34:07 --> 00:34:14
Office hours start this week,
in case you are eager to go.
489
00:34:14 --> 00:34:19
They are all held in some room
in Building 24,
490
00:34:19 --> 00:34:25
which is roughly the midpoint
between here and Stata,
491
00:34:25 --> 00:34:30
I think, for no particular
reason.
492
00:34:30 --> 00:34:34
And you can look at the Web
page for details on the office
493
00:34:34 --> 00:34:35
hours.
Continuing along,
494
00:34:35 --> 00:34:39
when is c_2 - 1 going to be
greater than or equal to zero?
495
00:34:39 --> 00:34:44
Well, that is true if c_2 is at
least 1, which is no big deal.
496
00:34:44 --> 00:34:47
Again, we get to choose the
constants however we want.
497
00:34:47 --> 00:34:51
It only has to hold for some
choice of constants.
498
00:34:51 --> 00:34:54
So, we can set c_2 greater than
or equal to 1.
499
00:34:54 --> 00:34:59
And then we are happy.
That means this whole thing is
500
00:34:59 --> 00:35:03
less than or equal to c_1*n^2 -
c_2*n if c_2 is greater than or
501
00:35:03 --> 00:35:06
equal to 1.
It is kind of funny here.
502
00:35:06 --> 00:35:10
This finishes the induction,
at least the induction step.
503
00:35:10 --> 00:35:13
We proved now that for any
value of c_1,
504
00:35:13 --> 00:35:16
and provided c_2 is at least
one.
505
00:35:16 --> 00:35:20
We have to be a little more
careful that c_1 does actually
506
00:35:20 --> 00:35:26
have to be sufficiently large.
Any particular reason why?
507
00:35:26 --> 00:35:32
508
00:35:32 --> 00:35:34
c_1 better not be negative,
indeed.
509
00:35:34 --> 00:35:39
c_1 has to be positive for this
to work, but it even has to be
510
00:35:39 --> 00:35:43
larger than positive depending.
Sorry.
511
00:35:43 --> 00:35:47
I have been going so fast,
I haven't asked you questions.
512
00:35:47 --> 00:35:50
Now you are caught off guard.
Yeah?
513
00:35:50 --> 00:35:53
Because of the base case,
exactly.
514
00:35:53 --> 00:35:58
So, the base case will have
T(1) is c_1 time 1 squared minus
515
00:35:58 --> 00:36:02
c_2, we want to prove that it is
at most this,
516
00:36:02 --> 00:36:07
and T(1) is some constant we
have assumed.
517
00:36:07 --> 00:36:11
We need to choose c_1 to be
sufficiently larger than c_2,
518
00:36:11 --> 00:36:14
in fact, so c_2 has to be at
least 1.
519
00:36:14 --> 00:36:20
c_1 may have to be at least a
hundred more than one if this is
520
00:36:20 --> 100.
521
100. --> 00:36:22
This will be true if c_1 is
522
00:36:22 --> 00:36:26
sufficiently large.
And sufficiently large now
523
00:36:26 --> 00:36:31
means with respect to c_2.
You have to be a little bit
524
00:36:31 --> 00:36:34
careful, but in this case it
doesn't matter.
525
00:36:34 --> 00:36:37
Any questions about the
substitution method?
526
00:36:37 --> 00:36:40
That was the same example three
times.
527
00:36:40 --> 00:36:43
In the end, it turned out we
got the right answer.
528
00:36:43 --> 00:36:47
But we sort of had to know the
answer in order to find it,
529
00:36:47 --> 00:36:51
which is a bit of a pain.
It would certainly be nicer to
530
00:36:51 --> 00:36:54
just figure out the answer by
some procedure,
531
00:36:54 --> 00:36:58
and that will be the next two
techniques we talk about.
532
00:36:58 --> 00:37:02
Sorry?
How would you prove a lower
533
00:37:02 --> 00:37:04
bound?
I haven't tried it for this
534
00:37:04 --> 00:37:09
recurrence, but you should be
able to do exactly the same
535
00:37:09 --> 00:37:12
form.
Argue that T(n) is greater than
536
00:37:12 --> 00:37:16
or equal to c_1*n^2 - c_2*n.
I didn't check whether that
537
00:37:16 --> 00:37:20
particular form will work,
but I think it does.
538
00:37:20 --> 00:37:22
Try it.
These other methods will give
539
00:37:22 --> 00:37:26
you, in some sense,
upper and lower bounds if you
540
00:37:26 --> 00:37:31
are a little bit careful.
But, to really check things,
541
00:37:31 --> 00:37:33
you pretty much have to do the
substitution method.
542
00:37:33 --> 00:37:35
And you will get some practice
with that.
543
00:37:35 --> 00:37:37
Usually we only care about
upper bounds.
544
00:37:37 --> 00:37:39
Proving upper bounds like this
is what we will focus on,
545
00:37:39 --> 00:37:41
but occasionally we need lower
bounds.
546
00:37:41 --> 00:37:43
It is always nice to know that
you have the right answer by
547
00:37:43 --> 00:37:46
proving a matching lower bound.
548
00:37:46 --> 00:37:51
549
00:37:51 --> 00:37:54
The next method we will talk
about is the recursion-tree
550
00:37:54 --> 00:37:57
method.
And it is a particular way of
551
00:37:57 --> 00:38:00
adding up a recurrence,
and it is my favorite way.
552
00:38:00 --> 00:38:04
It usually just works.
That's the great thing about
553
00:38:04 --> 00:38:06
it.
It provides you intuition for
554
00:38:06 --> 00:38:08
free.
It tells you what the answer is
555
00:38:08 --> 00:38:11
pretty much.
It is slightly nonrigorous,
556
00:38:11 --> 00:38:14
this is a bit of a pain,
so you have to be really
557
00:38:14 --> 00:38:18
careful when you apply it.
Otherwise, you might get the
558
00:38:18 --> 00:38:20
wrong answer.
Because it involves dot,
559
00:38:20 --> 00:38:24
dot, dots, our favorite three
characters, but dot,
560
00:38:24 --> 00:38:30
dot, dots are always a little
bit nonrigorous so be careful.
561
00:38:30 --> 00:38:32
Technically,
what you should do is find out
562
00:38:32 --> 00:38:34
what the answer is with
recursion-tree method.
563
00:38:34 --> 00:38:37
Then prove that it is actually
right with the substitution
564
00:38:37 --> 00:38:39
method.
Usually that is not necessary,
565
00:38:39 --> 00:38:42
but you should at least have in
your mind that that is required
566
00:38:42 --> 00:38:43
rigorously.
And probably the first few
567
00:38:43 --> 00:38:46
recurrences you solve,
you should do it that way.
568
00:38:46 --> 00:38:48
When you really understand the
recursion-tree method,
569
00:38:48 --> 00:38:51
you can be a little bit more
sloppy if you are really sure
570
00:38:51 --> 00:38:55
you have the right answer.
Let's do an example.
571
00:38:55 --> 00:38:59
We saw recursion trees very
briefly last time with mergesort
572
00:38:59 --> 00:39:01
as the intuition why it was n
log n.
573
00:39:01 --> 00:39:05
And, if you took an example
like the one we just did with
574
00:39:05 --> 00:39:08
the recursion-tree method,
it is dead simple.
575
00:39:08 --> 00:39:12
Just to make our life harder,
let's do a more complicated
576
00:39:12 --> 00:39:15
recursion.
Here we imagine we have some
577
00:39:15 --> 00:39:17
algorithm.
It starts with a problem size
578
00:39:17 --> 00:39:21
n, it recursively solves a
problem of size n/4,
579
00:39:21 --> 00:39:24
it then recursively solves a
problem of size n/2,
580
00:39:24 --> 00:39:30
and it does n^2 work on the
side without nonrecursive work.
581
00:39:30 --> 00:39:33
What is that?
I mean that is a bit less
582
00:39:33 --> 00:39:38
obvious, I would say.
What we are going to do is draw
583
00:39:38 --> 00:39:45
a picture, and we are just going
to expand out that recursion in
584
00:39:45 --> 00:39:47
tree form --
585
00:39:47 --> 00:39:56
586
00:39:56 --> 00:40:00
-- and then just add everything
up.
587
00:40:00 --> 00:40:05
We want the general picture,
and the general principle in
588
00:40:05 --> 00:40:12
the recursion-tree method is we
just draw this as a picture.
589
00:40:12 --> 00:40:16
We say well,
T(n) equals the sum of n^2,
590
00:40:16 --> 00:40:21
T(n/4) and T(n/2).
This is a weird way of writing
591
00:40:21 --> 00:40:25
a sum but why not write it that
way.
592
00:40:25 --> 00:40:31
This is going to be a tree.
And it is going to be a tree by
593
00:40:31 --> 00:40:35
recursively expanding each of
these two leaves.
594
00:40:35 --> 00:40:40
I start by expanding T(n) to
this, then I keep expanding,
595
00:40:40 --> 00:40:42
expanding, expanding
everything.
596
00:40:42 --> 00:40:46
Let's go one more step.
We have this n^2,
597
00:40:46 --> 00:40:49
T(n/4), T(n/2).
If we expand one more time,
598
00:40:49 --> 00:40:53
this is going to be n^2 plus
two things.
599
00:40:53 --> 00:40:58
The first thing is going to be
(n/4)^2, the second thing is
600
00:40:58 --> 00:41:03
going to be (n/2)^2.
Plus their recursive branches.
601
00:41:03 --> 00:41:08
We have T(n/16) and T(n/8).
Here my arithmetic shows thin.
602
00:41:08 --> 00:41:12
This better be the same,
T(n/8), and this should be
603
00:41:12 --> 00:41:15
T(n/4), I believe.
You just keep going forever,
604
00:41:15 --> 00:41:20
I mean, until you get down to
the base case where T is a
605
00:41:20 --> 00:41:23
constant.
So, I am now going to skip some
606
00:41:23 --> 00:41:25
steps and say dot,
dot, dot.
607
00:41:25 --> 00:41:30
This is where you have to be
careful.
608
00:41:30 --> 00:41:33
We have n^2,
(n/4)^2, (n/2)^2.
609
00:41:33 --> 00:41:39
Now this is easy because I have
already done them all.
610
00:41:39 --> 00:41:43
(n/16)^2, (n/8)^2,
(n/8)^2 again,
611
00:41:43 --> 00:41:47
(n/4)^2 and et cetera,
dot, dot, dot,
612
00:41:47 --> 00:41:52
of various levels of recursion
here.
613
00:41:52 --> 00:41:57
At the bottom,
we are going to get a bunch of
614
00:41:57 --> 00:42:01
constants.
These are the leaves.
615
00:42:01 --> 00:42:04
I would like to know how many
leaves there are.
616
00:42:04 --> 00:42:07
One challenge is how many
leaves in this tree could there
617
00:42:07 --> 00:42:09
be?
This is a bit subtle,
618
00:42:09 --> 00:42:13
unlike mergesort or unlike the
previous recurrence we solved,
619
00:42:13 --> 00:42:16
the number of leaves here is a
bit funny because we are
620
00:42:16 --> 00:42:20
recursing at different speeds.
This tree is going to be much
621
00:42:20 --> 00:42:23
smaller than this tree.
It is going to have smaller
622
00:42:23 --> 00:42:26
depth because it has already
done down to (n/16).
623
00:42:26 --> 00:42:30
Here it has only gone down to
(n/4).
624
00:42:30 --> 00:42:35
But how many leaves are there
in this recursion tree?
625
00:42:35 --> 00:42:41
All I need is an upper bound,
some reasonable upper bound.
626
00:42:41 --> 00:42:47
I can tell you it is at most
T(n^10), but that is a bit
627
00:42:47 --> 00:42:51
unreasonable.
It should be less than n,
628
00:42:51 --> 00:42:54
good.
Why is it less than n?
629
00:42:54 --> 00:42:58
Exactly.
I start with a problem of size
630
00:42:58 --> 00:43:02
n.
And I recurse into a problem
631
00:43:02 --> 00:43:04
that n/4 and a problem that says
n/2.
632
00:43:04 --> 00:43:08
When I get down to one I stop.
So, n/4 + n/2 = ¾n,
633
00:43:08 --> 00:43:13
which is strictly less than n.
So, definitely the total number
634
00:43:13 --> 00:43:17
of leaves has to be at most n.
If I start out with n sort of
635
00:43:17 --> 00:43:21
stuff and get rid of a quarter
of it and then recurse,
636
00:43:21 --> 00:43:26
it is definitely going to be
less than n stuff at the bottom.
637
00:43:26 --> 00:43:30
So, strictly less than n
leaves.
638
00:43:30 --> 00:43:32
At this point,
I have done nothing
639
00:43:32 --> 00:43:34
interesting.
And then the second cool idea
640
00:43:34 --> 00:43:38
in recursion trees is you don't
just expand this tree and see
641
00:43:38 --> 00:43:42
what it looks like and then say,
well, God, how the hell am I
642
00:43:42 --> 00:43:45
going to sum that?
You sum it level by level.
643
00:43:45 --> 00:43:48
That is the only other idea.
It usually works really,
644
00:43:48 --> 00:43:50
really well.
Here it is a bit complicated
645
00:43:50 --> 00:43:54
and I have to think a bit to
figure out n^2 is n^2.
646
00:43:54 --> 00:43:55
That is the first level.
Easy.
647
00:43:55 --> 00:44:00
The second level,
I have to think a lot harder.
648
00:44:00 --> 00:44:03
There are three kinds of
mathematicians,
649
00:44:03 --> 00:44:08
those who can add and those who
cannot, and I am the latter kind
650
00:44:08 --> 00:44:12
so I need your help.
Can you add these things
651
00:44:12 --> 00:44:15
together?
It's n^2 over something.
652
00:44:15 --> 00:44:16
Please?
(5/16)n^2.
653
00:44:16 --> 00:44:21
Now I really need your help.
I think that one I could have
654
00:44:21 --> 00:44:24
done, but this one is a little
bit harder.
655
00:44:24 --> 00:44:30
I will go look at my notes
while you compute that.
656
00:44:30 --> 00:44:35
657
00:44:35 --> 00:44:37
Any answers?
73/256.
658
00:44:37 --> 00:44:44
Anyone else confirm that?
It seems a bit high to me.
659
00:44:44 --> 00:44:49
73 does not sound right to me.
64?
660
00:44:49 --> 00:44:54
Closer.
It is actually important that
661
00:44:54 --> 00:44:59
we get this right.
The 256 is correct.
662
00:44:59 --> 00:45:04
I can tell.
Everyone should know that 16^2
663
00:45:04 --> 00:45:06
= 256.
We are computer scientists.
664
00:45:06 --> 00:45:09
25, good.
We have two people saying 25,
665
00:45:09 --> 00:45:12
therefore it is correct by
democracy.
666
00:45:12 --> 00:45:17
[LAUGHTER] 25 is also what my
notes say, and I computed it at
667
00:45:17 --> 00:45:19
home.
(25/256)n^2 is the right
668
00:45:19 --> 00:45:21
answer.
Now, did anyone notice
669
00:45:21 --> 00:45:24
something magical about this
progression?
670
00:45:24 --> 00:45:28
It squares each time,
good.
671
00:45:28 --> 00:45:32
And, if we were going to add
these up, you might call it?
672
00:45:32 --> 00:45:34
A geometric series,
very good.
673
00:45:34 --> 00:45:37
So, it turns out this is
geometric.
674
00:45:37 --> 00:45:40
And we know how to sum
geometric series,
675
00:45:40 --> 00:45:43
at least you should.
676
00:45:43 --> 00:45:59
677
00:45:59 --> 00:46:01
We started n^2.
We know that at the bottom,
678
00:46:01 --> 00:46:05
well, this is not quite a
level, we get something like n,
679
00:46:05 --> 00:46:07
but we are decreasing
geometrically.
680
00:46:07 --> 00:46:10
So, the total,
I mean the solution to the
681
00:46:10 --> 00:46:13
recurrence is the sum of all the
numbers in this tree.
682
00:46:13 --> 00:46:17
If we added it up level by
level and then add up all the
683
00:46:17 --> 00:46:20
levels that is going to give us
the answer.
684
00:46:20 --> 00:46:22
This is the total computed
level by level.
685
00:46:22 --> 00:46:25
It is just a cute way to
compute it.
686
00:46:25 --> 00:46:30
It usually gives you nice
answers like geometric answers.
687
00:46:30 --> 00:46:32
We have n^2(1 + 5/16 + 25/256 +
...).
688
00:46:32 --> 00:46:37
And, if we believe in fate and
we see this three number
689
00:46:37 --> 00:46:41
recurrence, we know that we have
the right answer.
690
00:46:41 --> 00:46:45
In general, it is going to be
(5/16)k, at least we hope,
691
00:46:45 --> 00:46:47
and so on.
And it keeps going.
692
00:46:47 --> 00:46:52
It doesn't go on infinitely,
but let's just assume it goes
693
00:46:52 --> 00:46:55
on infinitely.
That will be an upper bound
694
00:46:55 --> 00:47:00
that goes on forever.
This is all times n^2.
695
00:47:00 --> 00:47:05
Now, if you are going to know
one thing about geometric
696
00:47:05 --> 00:47:10
series, you should know that 1 +
½ + ¼, if you sum all the powers
697
00:47:10 --> 00:47:14
of 2 you get 2.
We are computer scientists.
698
00:47:14 --> 00:47:19
We have got to know at least
the binary case.
699
00:47:19 --> 00:47:23
This is like writing 0.1111111
in binary, actually,
700
00:47:23 --> 1.11111.
701
1.11111. --> 00:47:27
And 11111 forever is the same
702
00:47:27 --> 00:47:31
as 1, so this is 2.
This is even smaller.
703
00:47:31 --> 00:47:34
We have 5/16,
that is less than a half and
704
00:47:34 --> 00:47:39
then we are squaring each time,
so this is even less than 2.
705
00:47:39 --> 00:47:42
If you want,
there is a nifty formula for
706
00:47:42 --> 00:47:47
solving the general geometric
series, but all we need is that
707
00:47:47 --> 00:47:49
it is a constant.
This is O(n^2).
708
00:47:49 --> 00:47:53
It is also O(n^2).
It is pretty obvious that it is
709
00:47:53 --> 00:47:56
O(n^2) because the top thing is
n^2.
710
00:47:56 --> 00:48:00
So, there is our lower bound of
n^2.
711
00:48:00 --> 00:48:03
And we have it within a factor
of 2, which is pretty good.
712
00:48:03 --> 00:48:05
You actually get a better
factor here.
713
00:48:05 --> 00:48:07
So, that is recursion-tree
method.
714
00:48:07 --> 00:48:10
It is a little shaky here
because we have these dot,
715
00:48:10 --> 00:48:13
dot, dots, and we just believe
that it is geometric.
716
00:48:13 --> 00:48:16
It turns out most of the time
it is geometric.
717
00:48:16 --> 00:48:18
No problem here.
I would definitely check it
718
00:48:18 --> 00:48:21
with the substitution method
because this is not obvious to
719
00:48:21 --> 00:48:23
me that it is going to be
geometric.
720
00:48:23 --> 00:48:27
In the cases we will look at in
a moment, it will be much
721
00:48:27 --> 00:48:30
clearer, so clear that we can
state a theorem that everything
722
00:48:30 --> 00:48:34
is working fine.
And still time,
723
00:48:34 --> 00:48:38
good.
So, that was recursion-trees.
724
00:48:38 --> 00:48:43
There is one more method we are
going to talk about,
725
00:48:43 --> 00:48:49
and you could essentially think
of it as an application of the
726
00:48:49 --> 00:48:55
recursion-tree method but it is
made more precise.
727
00:48:55 --> 00:49:00
And it is an actual theorem,
whereas recursion trees,
728
00:49:00 --> 00:49:04
if the dot, dot,
dots aren't obvious,
729
00:49:04 --> 00:49:10
you better check them.
The sad part about the master
730
00:49:10 --> 00:49:13
method is it is pretty
restrictive.
731
00:49:13 --> 00:49:18
It only applies to a particular
family of recurrences.
732
00:49:18 --> 00:49:27
733
00:49:27 --> 00:49:29
It should be T(n) = aT(n/b) +
f(n).
734
00:49:29 --> 00:49:32
Am I going to call it f?
Yes, I will call it f.
735
00:49:32 --> 00:49:35
In particular,
it will not cover the
736
00:49:35 --> 00:49:40
recurrence I just solved because
I was recursing on two different
737
00:49:40 --> 00:49:44
problems of different sizes.
Here, every problem you recurse
738
00:49:44 --> 00:49:48
on should be of the same size.
There are a subproblems.
739
00:49:48 --> 00:49:51
A way to think of this is a
recursive algorithm.
740
00:49:51 --> 00:49:55
You have a subproblems.
Each of them is of size n/b,
741
00:49:55 --> 00:49:57
so the total costs will be
this.
742
00:49:57 --> 00:50:02
Then you are doing f(n)
nonrecursive work.
743
00:50:02 --> 00:50:05
A few constraints.
a should be at least 1,
744
00:50:05 --> 00:50:08
should have at least 1
recursion.
745
00:50:08 --> 00:50:12
b should be strictly greater
than 1.
746
00:50:12 --> 00:50:17
You better make the problem
smaller or else it is going to
747
00:50:17 --> 00:50:21
be infinity.
And f should have some nice
748
00:50:21 --> 00:50:25
property.
f(n) should be asymptotically
749
00:50:25 --> 00:50:27
positive.
750
00:50:27 --> 00:50:32
751
00:50:32 --> 00:50:37
How many people know what
asymptotically positive means?
752
00:50:37 --> 00:50:40
No one.
OK, you haven't read the
753
00:50:40 --> 00:50:41
textbook.
That's OK.
754
00:50:41 --> 00:50:46
I haven't read it either,
although don't tell Charles.
755
00:50:46 --> 00:50:50
And he'd notice.
And what might you think
756
00:50:50 --> 00:50:55
asymptotically positive means?
That we can do a little bit
757
00:50:55 --> 00:50:56
better.
Sorry?
758
00:50:56 --> 00:51:03
Yes, it means for large enough
n, f(n) is positive.
759
00:51:03 --> 00:51:07
This means f(n) is greater than
zero for n, at least some n_o,
760
00:51:07 --> 00:51:10
so for some constant n_o.
Eventually it should be
761
00:51:10 --> 00:51:12
positive.
I mean, we don't care about
762
00:51:12 --> 00:51:16
whether it's negative 1 for n=1,
not a big deal.
763
00:51:16 --> 00:51:20
It won't affect the answer
because we only care about the
764
00:51:20 --> 00:51:22
asympotics within.
765
00:51:22 --> 00:51:28
766
00:51:28 --> 00:51:30
The master method,
you gave it a recurrence of
767
00:51:30 --> 00:51:33
this form, it tells you the
answer.
768
00:51:33 --> 00:51:36
That is the great thing about
the master method.
769
00:51:36 --> 00:51:39
The annoying thing about the
master method is that it has
770
00:51:39 --> 00:51:41
three cases.
It is a big long.
771
00:51:41 --> 00:51:45
It takes a little bit longer to
memorize than all the others
772
00:51:45 --> 00:51:47
because the others are just
ideas.
773
00:51:47 --> 00:51:50
Here we need to actually
remember a few things.
774
00:51:50 --> 00:51:53
Let me state the theorem.
Well, not quite yet.
775
00:51:53 --> 00:51:57
There is one very simple idea,
which is we are going to
776
00:51:57 --> 00:52:01
compare this nonrecursive work
f(n) with a very particular
777
00:52:01 --> 00:52:05
function n^(log_b(a)).
Why n^(log_b(a))?
778
00:52:05 --> 00:52:08
You will see later.
It turns out it is the number
779
00:52:08 --> 00:52:13
of leaves in the recursion tree,
but that is foreshadowing.
780
00:52:13 --> 00:52:16
So, it is either less,
equal or bigger.
781
00:52:16 --> 00:52:18
And here we care about
asymptotics.
782
00:52:18 --> 00:52:22
And we have to be a little bit
more precious about less,
783
00:52:22 --> 00:52:25
equal or bigger.
You might think well,
784
00:52:25 --> 00:52:30
it means little o,
big Theta, or little omega.
785
00:52:30 --> 00:52:34
It would be nice if the theorem
held for all of those cases,
786
00:52:34 --> 00:52:38
but it leaves some gaps.
Let's start with Case 1.
787
00:52:38 --> 00:52:42
Case 1 is when f is smaller.
And not just that it is little
788
00:52:42 --> 00:52:46
o, but it is actually quite a
bit smaller.
789
00:52:46 --> 00:52:51
It has got to be polynomially
smaller than n^(log_b(a)).
790
00:52:51 --> 00:53:00
791
00:53:00 --> 00:53:04
For some positive epsilon,
the running time should be this
792
00:53:04 --> 00:53:09
n to this constant log base b of
a minus that epsilon,
793
00:53:09 --> 00:53:13
so it is really polynomially
smaller than n^(log_b(a)).
794
00:53:13 --> 00:53:18
We cannot handle the little o
case, that's a little bit too
795
00:53:18 --> 00:53:21
strong.
This is saying it is really
796
00:53:21 --> 00:53:25
quite a bit smaller.
But the answer then is really
797
00:53:25 --> 00:53:28
simple, T(n) =
Theta(n^(log_b(a))).
798
00:53:28 --> 00:53:31
Great.
That is Case 1.
799
00:53:31 --> 00:53:39
Case 2 is when f(n) is pretty
much equal to n^(log_b(a)).
800
00:53:39 --> 00:53:46
And by pretty much equal I mean
up to poly log factors.
801
00:53:46 --> 00:53:51
This is log base 2 of n to the
power k.
802
00:53:51 --> 00:53:56
You should know this notation.
For example,
803
00:53:56 --> 00:54:02
k could be zero.
And then they are equal up to
804
00:54:02 --> 00:54:06
constant factors,
for some k greater than or
805
00:54:06 --> 00:54:09
equal to zero.
Less than will not work,
806
00:54:09 --> 00:54:13
so it is really important that
k is non-negative.
807
00:54:13 --> 00:54:16
It should probably be an
integer.
808
00:54:16 --> 00:54:20
It doesn't actually matter
whether there is an integer,
809
00:54:20 --> 00:54:24
but there it is.
It could n^(log_b(a)) times log
810
00:54:24 --> 00:54:27
n or just times nothing,
whatever.
811
00:54:27 --> 00:54:32
Again, the solution is easy
here, T(n) = Theta(n^(log_b(a))*
812
00:54:32 --> 00:54:38
lg^(k+1)(n)).
Presumably it has to be at
813
00:54:38 --> 00:54:44
least times log k.
It turns out it is log to the k
814
00:54:44 --> 00:54:47
plus 1 of n.
That is Case 2.
815
00:54:47 --> 00:54:53
We have one more case which is
slightly more complicated.
816
00:54:53 --> 00:55:00
We need to assume slightly more
for Case 3.
817
00:55:00 --> 00:55:05
But Case 3 is roughly when f(n)
grows bigger than n^(log_b(a)).
818
00:55:05 --> 00:55:10
So, it should be capital Omega,
here is one place where we get
819
00:55:10 --> 00:55:14
to use omega,
(n^(log_b(a)) + epsilon) for
820
00:55:14 --> 00:55:19
some positive epsilon.
It should grow not just bigger
821
00:55:19 --> 00:55:23
but polynomially bigger.
Here it was growing just a log
822
00:55:23 --> 00:55:27
factor bigger,
poly log, and here it is a
823
00:55:27 --> 00:55:31
polynomial factor.
In this case,
824
00:55:31 --> 00:55:36
we need another assumption
about f because we worry a
825
00:55:36 --> 00:55:40
little bit about how quickly f
grows.
826
00:55:40 --> 00:55:46
We want to make sure that as
you go down the recursion f gets
827
00:55:46 --> 00:55:49
smaller.
It would be kind of nice if f
828
00:55:49 --> 00:55:54
gets smaller as you go down,
otherwise you are,
829
00:55:54 --> 00:55:58
again, trying to sum to
infinity or whatever.
830
00:55:58 --> 00:56:06
I see why this is for some
epsilon prime greater than zero.
831
00:56:06 --> 00:56:09
What I would like is that if I
just sort of take the
832
00:56:09 --> 00:56:13
recurrence, this T(n) and just
throw in fs instead,
833
00:56:13 --> 00:56:16
f(n) should be somehow related
to af(n/b).
834
00:56:16 --> 00:56:20
What I would like is that f(n),
which is at the top of the
835
00:56:20 --> 00:56:23
recursion tree,
should be bigger than the thing
836
00:56:23 --> 00:56:27
at the next level down.
The sum of all the values at
837
00:56:27 --> 00:56:33
the next level down should be
bigger by some constant factor.
838
00:56:33 --> 00:56:37
Here I have the next level down
is at most some 1 - e,
839
00:56:37 --> 00:56:42
something strictly less than 1,
some constant strictly less
840
00:56:42 --> 00:56:45
than 1 times the thing at the
top level.
841
00:56:45 --> 00:56:49
I need that to make sure things
are getting smaller as I go
842
00:56:49 --> 00:56:52
down.
Then T(n) = Theta[f(n)].
843
00:56:52 --> 00:56:56
And that is the theorem.
This is the master theorem or
844
00:56:56 --> 00:57:02
whatever you want to call it.
It is not named after some guy
845
00:57:02 --> 00:57:05
name Master.
It is just the master of all
846
00:57:05 --> 00:57:09
methods because it is very easy
to apply.
847
00:57:09 --> 00:57:14
Let's apply it a few times.
It is a bit much to take in all
848
00:57:14 --> 00:57:16
at once.
And then I will give you a
849
00:57:16 --> 00:57:22
sketch of the proof to see that
it is really not that surprising
850
00:57:22 --> 00:57:26
this is true if you look at the
recursion-tree.
851
00:57:26 --> 00:57:30
But first let's just try using
it.
852
00:57:30 --> 00:57:35
For example,
we could take T(n) = 4T(n/2) +
853
00:57:35 --> 00:57:38
n.
This is a, this is b,
854
00:57:38 --> 00:57:44
this is f(n).
The first thing we should
855
00:57:44 --> 00:57:51
compute is n^(log_b(a)).
This I think even I can do.
856
00:57:51 --> 00:57:56
Log base 2 of 4.
Yeah, log base 2 I can do.
857
00:57:56 --> 00:58:04
This is n^2.
OK, so is f(n) smaller or
858
00:58:04 --> 00:58:10
bigger than n^2?
Well, f(n) = n.
859
00:58:10 --> 00:58:19
n^2 is clearly bigger by a
polynomial factor.
860
00:58:19 --> 00:58:26
So, we are in Case 1.
What is the answer?
861
00:58:26 --> 00:58:32
n^2, yeah.
It is T(n^(log_b(a))),
862
00:58:32 --> 00:58:40
which here it is just n^2.
Let's do some slight variation.
863
00:58:40 --> 00:58:46
I am going to keep a and b the
same and just change f.
864
00:58:46 --> 00:58:54
Let's say T(n) = 4T(n/2) + n^2.
This is like drill spelling.
865
00:58:54 --> 00:59:03
n^2 is asymptotically the same
as n^2 even up to constants.
866
00:59:03 --> 00:59:06
What is the answer?
This is Case 2.
867
00:59:06 --> 00:59:12
868
00:59:12 --> 00:59:14
It is slightly harder.
869
00:59:14 --> 00:59:22
870
00:59:22 --> 00:59:31
What is k in this example? Zero.
The answer is?
871
00:59:31 --> 00:59:35
Survey says?
n^2 log n.
872
00:59:35 --> 00:59:44
873
00:59:44 --> 00:59:49
Good.
And a couple more.
874
00:59:49 --> 00:59:58
T(n) = 4T(n/2) + n^3.
What is the answer?
875
00:59:58 --> 1:00:01.948
n^3.
This is Case 3.
876
1:00:01.948 --> 1:00:08.961
I know this is pretty boring.
At this point we are just
877
1:00:08.961 --> 1:00:15.194
applying this stupid theorem.
How about n^2/lg n?
878
1:00:15.194 --> 1:00:18.311
What is the answer?
Good.
879
1:00:18.311 --> 1:00:22.597
In this case no one should
answer.
880
1:00:22.597 --> 1:00:30
It is a big tricky.
I forget exactly the answer.
881
1:00:30 --> 1:00:31.539
I think it is like n^2 log log
n over log n,
882
1:00:31.539 --> 1:00:31.897
no?
Oh, no.
883
1:00:31.897 --> 1:00:32.864
n^2 log log n,
that's right.
884
1:00:32.864 --> 1:00:34.046
Yeah.
But you shouldn't know that,
885
1:00:34.046 --> 1:00:35.693
and this doesn't follow from
the master method.
886
1:00:35.693 --> 1:00:37.161
This is something you would
have to solve,
887
1:00:37.161 --> 1:00:38.843
probably with the
recursion-tree would be a good
888
1:00:38.843 --> 1:00:40.419
way to do this one,
and you need to know some
889
1:00:40.419 --> 1:00:41.851
properties of logs to know how
that goes.
890
1:00:41.851 --> 1:00:44
But here the master method does
not apply.
891
1:00:44 --> 1:01:05
892
1:01:05 --> 1:01:08.094
And so you have to use a
different method.
893
1:01:08.094 --> 1:01:10.584
OK.
The last thing I want to do is
894
1:01:10.584 --> 1:01:15.188
tell you why the master method
is true, and that makes it much
895
1:01:15.188 --> 1:01:17.528
more intuitive,
especially using
896
1:01:17.528 --> 1:01:21
recursion-trees,
why everything works.
897
1:01:21 --> 1:01:35
898
1:01:35 --> 1:01:38.77
This is a sketch of a proof,
not the full thing.
899
1:01:38.77 --> 1:01:42.06
You should read the proof in
the textbook.
900
1:01:42.06 --> 1:01:45.911
It is not that much harder than
what I will show,
901
1:01:45.911 --> 1:01:49.842
but it is good for you to know
the formal details.
902
1:01:49.842 --> 1:01:53.613
I don't have time here to do
all of the details.
903
1:01:53.613 --> 1:01:56.661
I will just tell you the
salient parts.
904
1:01:56.661 --> 1:02:01.315
This is the proof sketch or the
intuition behind the master
905
1:02:01.315 --> 1:02:06.099
method.
What we are going to do is just
906
1:02:06.099 --> 1:02:12.5
take the recursion-tree for this
recurrence and add up each level
907
1:02:12.5 --> 1:02:17.5
and then add up all the levels
and see what we get.
908
1:02:17.5 --> 1:02:23
We start with f(n) at the top
after we have expanded one
909
1:02:23 --> 1:02:25.9
level.
Then we get a different
910
1:02:25.9 --> 1:02:31.498
problems, each of n/b.
And after we expand them it
911
1:02:31.498 --> 1:02:36.31
will f(n/b) for each one.
They are all the same size.
912
1:02:36.31 --> 1:02:41.677
Then we expand all of those and
so on, and we get another a
913
1:02:41.677 --> 1:02:46.026
subproblems from there.
We are going to get like
914
1:02:46.026 --> 1:02:49.45
f((n/b)^2).
That is sort of decreasing
915
1:02:49.45 --> 1:02:54.355
geometrically the size,
and so on and so on and so on,
916
1:02:54.355 --> 1:03:00
until at the bottom we get
constant size problems.
917
1:03:00 --> 1:03:03.904
This is a bit special because
this is the base case,
918
1:03:03.904 --> 1:03:07.349
but we have some other constant
at the bottom.
919
1:03:07.349 --> 1:03:10.947
We would like to know how many
leaves there are,
920
1:03:10.947 --> 1:03:14.392
but that is a little bit tricky
at the moment.
921
1:03:14.392 --> 1:03:17.684
Let's first compute the height
of this tree.
922
1:03:17.684 --> 1:03:21.588
Let me draw it over here.
What is the height of this
923
1:03:21.588 --> 1:03:24.267
tree?
I start with a problem of size
924
1:03:24.267 --> 1:03:26.794
n.
I want to get down to a problem
925
1:03:26.794 --> 1:03:29.397
of size 1.
How long does that take?
926
1:03:29.397 --> 1:03:32
How many levels?
927
1:03:32 --> 1:03:38
928
1:03:38 --> 1:03:44.346
This is probably too easy for
some and not at your fingertips
929
1:03:44.346 --> 1:03:47.201
for others.
Log base b of n,
930
1:03:47.201 --> 1:03:50.48
good.
The height of this tree is
931
1:03:50.48 --> 1:03:54.711
n^(log_b(a)),
because it is just how many
932
1:03:54.711 --> 1:04:00
times I divide by b until I get
down to 1.
933
1:04:00 --> 1:04:04.189
That is great.
Now I should be able to compute
934
1:04:04.189 --> 1:04:09.216
the number of leaves because I
have branching factor a,
935
1:04:09.216 --> 1:04:13.312
I have height h.
The number of leaves is a^h,
936
1:04:13.312 --> 1:04:16.849
a^log_b(n).
Let me expand that a little
937
1:04:16.849 --> 1:04:20.108
bit.
a^log_b(n), properties of logs,
938
1:04:20.108 --> 1:04:24.855
we can take the n downstairs
and put the a upstairs,
939
1:04:24.855 --> 1:04:31
and we get n^(log_b(a)).
Our good friend n^(log_b(a)).
940
1:04:31 --> 1:04:35.246
So, that is why Our good friend
n^(log_b(a)) is so important in
941
1:04:35.246 --> 1:04:38.534
the master method.
What we are doing is comparing
942
1:04:38.534 --> 1:04:41.41
f, which is the top level,
to n^(log_b(a)),
943
1:04:41.41 --> 1:04:43.945
which up to theta is the bottom
level.
944
1:04:43.945 --> 1:04:47.643
Now the leaves are all at the
same level because we are
945
1:04:47.643 --> 1:04:50.589
decreasing at the same rate in
every branch.
946
1:04:50.589 --> 1:04:53.328
If I add up the cost at the
bottom level,
947
1:04:53.328 --> 1:04:57.164
it is Theta(n^(log_b(a))).
I add up the things at the top
948
1:04:57.164 --> 1:05:01
level it is f(n),
not terribly exciting.
949
1:05:01 --> 1:05:04.529
But the next level,
this is a little bit more
950
1:05:04.529 --> 1:05:08.78
interesting, is af(n/b),
which should look familiar if
951
1:05:08.78 --> 1:05:12.229
you had the master method
already memorized,
952
1:05:12.229 --> 1:05:15.358
it is that.
So, we know that af(n/b) has
953
1:05:15.358 --> 1:05:19.368
decreased by some constant
factor, 1-epsilon prime.
954
1:05:19.368 --> 1:05:22.818
We have gone down.
This is a constant factor
955
1:05:22.818 --> 1:05:26.508
smaller than this.
And then you sum up the next
956
1:05:26.508 --> 1:05:28.754
level.
It is going to be like
957
1:05:28.754 --> 1:05:33.551
a^2f(n/b^2).
I see that I actually wrote
958
1:05:33.551 --> 1:05:37.804
this wrong, the parentheses.
Sorry about that.
959
1:05:37.804 --> 1:05:40.829
It is not (n/b)^2.
It is (n/b^2).
960
1:05:40.829 --> 1:05:44.326
So, this sequence,
in Case 3 at least,
961
1:05:44.326 --> 1:05:48.768
is decreasing geometrically.
If it is decreasing
962
1:05:48.768 --> 1:05:54.344
geometrically up to constant
factors, it is dominated by the
963
1:05:54.344 --> 1:05:56.896
biggest term,
which is f(n).
964
1:05:56.896 --> 1:06:02
Therefore, in Case 3,
we get Theta[f(n)].
965
1:06:02 --> 1:06:07.613
Let's look at the other cases,
and let me adapt those cases to
966
1:06:07.613 --> 1:06:11.846
how much time we have left.
Wow, lot's of time.
967
1:06:11.846 --> 1:06:14.239
Five minutes.
Tons of time.
968
1:06:14.239 --> 1:06:17.368
What to do?
Let me write that down.
969
1:06:17.368 --> 1:06:22.429
Case 3, the costs decrease.
Now, this is a place I would
970
1:06:22.429 --> 1:06:26.754
argue where the dot,
dot, dot is pretty obvious.
971
1:06:26.754 --> 1:06:32
Here, this is damn simple,
it is a^kf(n/b^k).
972
1:06:32 --> 1:06:37.187
And, in Case 3,
we assume that the costs
973
1:06:37.187 --> 1:06:43.172
decrease geometrically as we go
down the tree.
974
1:06:43.172 --> 1:06:49.423
That was sort of backwards to
start with Case 3.
975
1:06:49.423 --> 1:06:55.009
Let's do Case 1,
which is sort of the other
976
1:06:55.009 --> 1:07:01.079
intuitively easy case.
In Case 1, we know that f(n) is
977
1:07:01.079 --> 1:07:03.494
polynomially smaller than this
thing.
978
1:07:03.494 --> 1:07:07.452
And we are sort of changing by
this very simple procedure in
979
1:07:07.452 --> 1:07:10.203
the middle.
I am going to wave my hands if
980
1:07:10.203 --> 1:07:13.222
this is where you need a more
formal argument.
981
1:07:13.222 --> 1:07:16.241
I claim that this will increase
geometrically.
982
1:07:16.241 --> 1:07:19.528
It has to increase
geometrically because this f(n)
983
1:07:19.528 --> 1:07:23.419
is polynomially smaller than
this one, you are going to get
984
1:07:23.419 --> 1:07:26.639
various polynomials in the
middle which interpret
985
1:07:26.639 --> 1:07:31
geometrically from the small one
to the big one.
986
1:07:31 --> 1:07:34.444
Therefore, the big one
dominates because it is,
987
1:07:34.444 --> 1:07:38.413
again, geometric series.
As I said, this is intuition,
988
1:07:38.413 --> 1:07:42.007
not a formal argument.
This one was pretty formal
989
1:07:42.007 --> 1:07:45.751
because we assumed it,
but here you need a bit more
990
1:07:45.751 --> 1:07:47.997
argument.
They may not increase
991
1:07:47.997 --> 1:07:51.292
geometrically but they could
increase faster,
992
1:07:51.292 --> 1:07:53.987
and that is also fine.
So, in Case 3,
993
1:07:53.987 --> 1:07:57.657
you are dominated,
I mean you are always dominated
994
1:07:57.657 --> 1:08:02
by the biggest term in a
geometric series.
995
1:08:02 --> 1:08:08.416
Here it happens to be f(n) and
here you are dominated by
996
1:08:08.416 --> 1:08:13.316
n^(log_b(a)) with a bottom term,
oh, Theta.
997
1:08:13.316 --> 1:08:19.733
Case 2, here it is pretty easy
but you need to know some
998
1:08:19.733 --> 1:08:25.333
properties of logs.
In Case 2, we assume that all
999
1:08:25.333 --> 1:08:31.905
of these are basically the same.
I mean, we assume that the top
1000
1:08:31.905 --> 1:08:35.145
is equal to the bottom.
And this is changing in this
1001
1:08:35.145 --> 1:08:38.258
very procedural way.
Therefore, all of the ones in
1002
1:08:38.258 --> 1:08:40.927
the middle have to be pretty
much the same.
1003
1:08:40.927 --> 1:08:44.167
Not quite because here we don't
have the log factor.
1004
1:08:44.167 --> 1:08:47.851
Here we have a log to the k.
We have n^(log_b(a)) times log
1005
1:08:47.851 --> 1:08:50.329
to the kn.
Here we don't have the log to
1006
1:08:50.329 --> 1:08:52.68
the k.
So, the logs do disappear here.
1007
1:08:52.68 --> 1:08:57
It turns out the way they
disappear is pretty slowly.
1008
1:08:57 --> 1:09:02.255
If you look at the top half of
these terms, they will all have
1009
1:09:02.255 --> 1:09:06.046
log to the k.
The bottom half they will start
1010
1:09:06.046 --> 1:09:09.493
to disappear.
I am giving you some oracle
1011
1:09:09.493 --> 1:09:13.112
information.
If you take logs and you don't
1012
1:09:13.112 --> 1:09:17.248
change the argument by too much,
the logs remain.
1013
1:09:17.248 --> 1:09:22.073
Maybe halfway is too far.
The claim is that each level is
1014
1:09:22.073 --> 1:09:25.691
roughly the same,
especially the upper most
1015
1:09:25.691 --> 1:09:30
levels are all asymptotically
equal.
1016
1:09:30 --> 1:09:34.659
Roughly the same.
And, therefore,
1017
1:09:34.659 --> 1:09:42.375
the cost is one level,
here like f(n) times the number
1018
1:09:42.375 --> 1:09:47.908
of levels, h.
And h is log base b of n.
1019
1:09:47.908 --> 1:09:52.567
B is a constant so we don't
care.
1020
1:09:52.567 --> 1:09:57.662
This is Theta(lg n).
And, therefore,
1021
1:09:57.662 --> 1:10:05.961
we get T(n) = (n^(log_b(a))
lg^(k+1)(n)) times another log
1022
1:10:05.961 --> 1:10:11.232
n.
So, we get [f(n)lg n].
1023
1:10:11.232 --> 1:10:20.046
That is the very quick sketch.
Sorry, I am being pretty fuzzy
1024
1:10:20.046 --> 1:10:27.098
on Cases 1 and 2.
Read the proof because you will
1025
1:10:27.098 --> 1:10:34.59
have to, at some point,
manipulate logs in that way.
1026
1:10:34.59 --> 1:10:38.998
And that is all.
Any questions?
1027
1:10:38.998 --> 1:10:43.552
Or, you are all eager to go.
OK.
1028
1:10:43.552 --> 1:10:46
Thanks.
See you Wednesday.