WEBVTT
00:00:00.060 --> 00:00:01.770
The following
content is provided
00:00:01.770 --> 00:00:04.010
under a Creative
Commons license.
00:00:04.010 --> 00:00:06.860
Your support will help MIT
OpenCourseWare continue
00:00:06.860 --> 00:00:10.720
to offer high quality
educational resources for free.
00:00:10.720 --> 00:00:13.320
To make a donation or
view additional materials
00:00:13.320 --> 00:00:17.207
from hundreds of MIT courses,
visit MIT OpenCourseWare
00:00:17.207 --> 00:00:17.832
at ocw.mit.edu.
00:00:21.896 --> 00:00:24.520
PROFESSOR: Good
morning everyone.
00:00:24.520 --> 00:00:26.470
Morning.
00:00:26.470 --> 00:00:28.910
Let's get started.
00:00:28.910 --> 00:00:33.420
So the second of two
lectures on numerics.
00:00:33.420 --> 00:00:37.210
Last time we had this
motivating question
00:00:37.210 --> 00:00:41.280
of finding the millionth
digit of the square root of 2,
00:00:41.280 --> 00:00:44.530
or the square root
of quantities that
00:00:44.530 --> 00:00:47.510
end up becoming irrational.
00:00:47.510 --> 00:00:51.060
And we talked about
high-precision arithmetic,
00:00:51.060 --> 00:00:55.960
and we use Newton's method
to compute the square roots.
00:00:55.960 --> 00:00:59.010
You saw a demo of
computing square roots,
00:00:59.010 --> 00:01:02.100
but there's a few
things missing.
00:01:02.100 --> 00:01:05.840
We don't quite know how
to do division, which
00:01:05.840 --> 00:01:08.040
is required for the
Newton's method,
00:01:08.040 --> 00:01:13.680
and we didn't really talk at
all about algorithmic complexity
00:01:13.680 --> 00:01:17.520
beyond talking about the
complexity of multiplication.
00:01:17.520 --> 00:01:20.370
So multiplication is a
primitive that at this point
00:01:20.370 --> 00:01:24.180
we know how to do in a
couple of different ways,
00:01:24.180 --> 00:01:26.050
including the naive
order n squared
00:01:26.050 --> 00:01:27.930
algorithm and the
Karatsuba algorithm,
00:01:27.930 --> 00:01:31.370
which is something
like n raised to 1.58.
00:01:31.370 --> 00:01:34.040
But how many times
is multiplication
00:01:34.040 --> 00:01:37.500
called when you
compute square roots?
00:01:37.500 --> 00:01:40.310
In fact, multiplication
is called
00:01:40.310 --> 00:01:43.141
when you call the
division operator
00:01:43.141 --> 00:01:44.390
when you compute square roots.
00:01:44.390 --> 00:01:48.510
So there's really two levels
off a computation going on here
00:01:48.510 --> 00:01:51.850
and we need to open this
up, and look at in detail,
00:01:51.850 --> 00:01:55.500
and figure out what our overall
algorithmic complexity is.
00:01:55.500 --> 00:02:00.800
So that's really the
meat of today's lecture.
00:02:00.800 --> 00:02:04.020
Getting to the
point where we know
00:02:04.020 --> 00:02:07.580
what we have with respect
to asymptotic complexity
00:02:07.580 --> 00:02:10.979
of computing the square
root of a number.
00:02:10.979 --> 00:02:15.800
So let me start with a review
of what we covered last time.
00:02:18.460 --> 00:02:24.230
We decided that we wanted
the millionth digit
00:02:24.230 --> 00:02:26.350
of square root of 2.
00:02:26.350 --> 00:02:27.870
And the way we're
going to do this
00:02:27.870 --> 00:02:35.280
is by working with integers
and computing the floor,
00:02:35.280 --> 00:02:41.810
since we needed to be an
integer, of 2 times 10 raised
00:02:41.810 --> 00:02:46.115
to 2d, where d is the number
of digits of precision.
00:02:53.940 --> 00:02:56.570
N over there.
00:02:56.570 --> 00:03:01.080
So we'll take a look
at an example or two
00:03:01.080 --> 00:03:04.510
here as to how this
works with integers.
00:03:04.510 --> 00:03:12.180
But what we do is compute
essentially the floor
00:03:12.180 --> 00:03:15.930
of some quantity a, the square
root of some quantity a,
00:03:15.930 --> 00:03:17.270
via Newton's method.
00:03:27.170 --> 00:03:29.220
And the way Newton's
method works
00:03:29.220 --> 00:03:32.280
is you go through an iteration.
00:03:32.280 --> 00:03:39.460
You start with x0 being one,
which is your initial guess,
00:03:39.460 --> 00:03:50.949
and compute xi plus 1 equals
xi plus a divided by xi over 2.
00:03:50.949 --> 00:03:52.740
And as you can see,
this requires division,
00:03:52.740 --> 00:03:55.130
because we're computing
a divided by xi.
00:03:55.130 --> 00:03:58.380
That's the outer
Newton iteration.
00:03:58.380 --> 00:04:05.870
And I said a couple
of things that's said
00:04:05.870 --> 00:04:11.260
you are going to have a
quadratic rate of convergence.
00:04:11.260 --> 00:04:17.000
The precision with respect
to the number of digits
00:04:17.000 --> 00:04:22.680
is going to increase by a
factor of 2 every iteration.
00:04:22.680 --> 00:04:24.960
And so if you started out
with one digit of precision,
00:04:24.960 --> 00:04:28.320
you go to two, then
four, eight, et cetera.
00:04:28.320 --> 00:04:30.290
And so that's a
geometric progression.
00:04:30.290 --> 00:04:35.250
And that means that
we're going to have
00:04:35.250 --> 00:04:39.550
a logarithmic number of
iterations, which is nice.
00:04:39.550 --> 00:04:45.340
And we were all happy about
that, and you believed me.
00:04:45.340 --> 00:04:47.740
I gave you an example and
it looked pretty good,
00:04:47.740 --> 00:04:52.890
but didn't really prove anything
about the rate of convergence.
00:04:52.890 --> 00:04:59.480
What I'd like to do now is
take a look at this particular
00:04:59.480 --> 00:05:02.450
iterative computation, where
we're computing xi plus 1 given
00:05:02.450 --> 00:05:05.990
xi , and argue
that this, in fact,
00:05:05.990 --> 00:05:07.495
has a quadratic
rate of convergence.
00:05:10.060 --> 00:05:13.110
So you can think
of this as doing
00:05:13.110 --> 00:05:18.790
an error analysis
of Newton's method.
00:05:26.310 --> 00:05:34.960
And let's say that xn equals
square root of a 1 plus epsilon
00:05:34.960 --> 00:05:42.370
n, where epsilon may be
positive or negative.
00:05:42.370 --> 00:05:48.080
So we have an error
associated with xn
00:05:48.080 --> 00:05:50.710
in the n-th iteration with
respect to what we want,
00:05:50.710 --> 00:05:52.800
which is the square root of a.
00:05:52.800 --> 00:05:54.570
And it's off by something.
00:05:54.570 --> 00:05:56.720
It may be a large
quantity in the beginning.
00:05:56.720 --> 00:05:58.740
We want to show
convergence, so obviously we
00:05:58.740 --> 00:06:04.210
want epsilon n, as n
becomes large, do tend to 0.
00:06:04.210 --> 00:06:06.620
How fast does this approach 0?
00:06:06.620 --> 00:06:08.900
That's the question.
00:06:08.900 --> 00:06:14.130
And so if you take this equation
and plug this into that,
00:06:14.130 --> 00:06:17.410
and say, what is xn plus 1?
00:06:17.410 --> 00:06:29.040
xn plus 1 would be square root
of a times 1 plus epsilon n
00:06:29.040 --> 00:06:33.760
plus a divided by square
root of a 1 plus epsilon
00:06:33.760 --> 00:06:39.990
n divided by 2.
00:06:39.990 --> 00:06:43.130
Just plugging it
in, the value of xn.
00:06:43.130 --> 00:06:47.990
And then some a couple of steps
of algebraic simplification,
00:06:47.990 --> 00:06:50.830
you can pull out the
square root of a here,
00:06:50.830 --> 00:06:56.290
then you have 1
plus epsilon n, 1
00:06:56.290 --> 00:07:01.170
divided by 1 plus
epsilon n over here.
00:07:01.170 --> 00:07:04.650
The whole thing divided by 2.
00:07:04.650 --> 00:07:12.730
And if you keep going-- there's
one step that I'm skipping here
00:07:12.730 --> 00:07:16.750
in terms of
simplification, but let
00:07:16.750 --> 00:07:19.340
me write this last result out.
00:07:24.590 --> 00:07:32.560
Which is xn plus 1 is
square root of a times 1
00:07:32.560 --> 00:07:39.760
plus epsilon n squared
divided by 2 times 1
00:07:39.760 --> 00:07:43.370
plus epsilon n
down at the bottom.
00:07:43.370 --> 00:07:51.990
So what do we have here in
terms of the overall observation
00:07:51.990 --> 00:07:55.980
for epsilon n plus 1,
which is the error in the n
00:07:55.980 --> 00:07:59.700
plus 1-th iteration given
that you have an epsilon n
00:07:59.700 --> 00:08:02.100
error in the n-th iteration?
00:08:02.100 --> 00:08:09.920
You have a relationship like
so where epsilon n plus 1
00:08:09.920 --> 00:08:13.630
is related to epsilon
n whole square.
00:08:13.630 --> 00:08:17.920
And this part here,
as n becomes large,
00:08:17.920 --> 00:08:21.720
epsilon n is going
to go to 0 assuming
00:08:21.720 --> 00:08:24.140
a decent initial guess.
00:08:24.140 --> 00:08:27.020
And so you can say that
this is essentially
00:08:27.020 --> 00:08:32.789
1, which means you have this
quadratic rate of convergence
00:08:32.789 --> 00:08:37.340
where the error, which
is a small quantity,
00:08:37.340 --> 00:08:40.159
is getting squared
at every iteration.
00:08:40.159 --> 00:08:43.530
And so if you have
something like a 0.01 error
00:08:43.530 --> 00:08:46.640
at the beginning for
epsilon n, epsilon n squared
00:08:46.640 --> 00:08:55.040
is going to be 0.0001.
00:08:55.040 --> 00:08:58.840
So that's where you get the
quadratic rate of convergence.
00:08:58.840 --> 00:09:02.560
So it really comes from this
relationship, the relationship
00:09:02.560 --> 00:09:06.449
epsilon n squared
to epsilon n plus 1,
00:09:06.449 --> 00:09:07.490
Any questions about this?
00:09:12.010 --> 00:09:12.780
Great.
00:09:12.780 --> 00:09:17.160
So if you have the quadratic
rate of convergence,
00:09:17.160 --> 00:09:27.610
if you want to go to d digits
of precision like I have here,
00:09:27.610 --> 00:09:31.805
you can argue that you
need to log d iterations.
00:09:35.612 --> 00:09:37.820
So that's kind of nice, you
have a logarithmic number
00:09:37.820 --> 00:09:38.650
of iterations.
00:09:38.650 --> 00:09:40.540
I'm going to get back to that.
00:09:40.540 --> 00:09:45.030
There's one little
subtlety that is associated
00:09:45.030 --> 00:09:48.610
with asymptotic analysis
that goes beyond simply
00:09:48.610 --> 00:09:51.510
the number of
iterations that you have
00:09:51.510 --> 00:09:53.300
and the digits of precision.
00:09:53.300 --> 00:09:55.370
But so far so good.
00:09:55.370 --> 00:09:57.730
We're happy with this
logarithmic number
00:09:57.730 --> 00:09:58.490
of iterations.
00:09:58.490 --> 00:10:08.020
And if we can now compute the
complexity of the division,
00:10:08.020 --> 00:10:10.930
then obviously we need
an algorithm for that.
00:10:10.930 --> 00:10:13.130
But if you have an
algorithm and we figure out
00:10:13.130 --> 00:10:15.990
what the complexity of
the division algorithm is,
00:10:15.990 --> 00:10:20.520
then we have complexity
for the square root of 2
00:10:20.520 --> 00:10:22.800
or square root of a
using Newton's method.
00:10:25.540 --> 00:10:30.450
So just justified what I
said last time with respect
00:10:30.450 --> 00:10:33.000
to quadratic rate
of convergence.
00:10:33.000 --> 00:10:36.140
And then we talked about
multiplication last time.
00:10:36.140 --> 00:10:39.460
I want to revisit that.
00:10:39.460 --> 00:10:50.090
You have multiplication
algorithms,
00:10:50.090 --> 00:10:55.205
and we want to be able to
multiply d digit numbers.
00:10:59.090 --> 00:11:01.560
And the naive algorithm.
00:11:01.560 --> 00:11:05.870
And you could imagine
doing divide and conquer.
00:11:10.690 --> 00:11:17.870
So you take x1,
x0; y1, y0 where x1
00:11:17.870 --> 00:11:21.170
is the most
significant half of x.
00:11:21.170 --> 00:11:23.055
You're trying to
multiply x times y.
00:11:27.880 --> 00:11:30.970
And same thing for y1 and y0.
00:11:30.970 --> 00:11:36.100
So each of these will have
d by 2, digits of precision.
00:11:36.100 --> 00:11:40.870
And if you implement
the naive algorithm that
00:11:40.870 --> 00:11:48.730
looks like tn equals 4
tn by 2 plus theta n,
00:11:48.730 --> 00:11:52.940
you end up with theta n
squared complexity out
00:11:52.940 --> 00:11:55.690
so you have to do four
multiplications corresponding
00:11:55.690 --> 00:11:59.960
to x1 times Y1 x1
times y0, et cetera.
00:11:59.960 --> 00:12:02.790
And at each level in
the recursive tree,
00:12:02.790 --> 00:12:06.870
you're breaking things down
by a factor of 2 respect
00:12:06.870 --> 00:12:08.850
to the digits of
precision that you
00:12:08.850 --> 00:12:12.140
need to multiply on as
you're going down the tree.
00:12:12.140 --> 00:12:14.200
And this is the four
multiplications.
00:12:14.200 --> 00:12:16.740
You get your theta n
squared complexity.
00:12:16.740 --> 00:12:19.760
This gentleman by
the name of Karatsuba
00:12:19.760 --> 00:12:25.070
recognized that you could play
a few mathematical tricks, which
00:12:25.070 --> 00:12:29.730
I won't go over
again, but reduce
00:12:29.730 --> 00:12:32.340
to three multiplications.
00:12:32.340 --> 00:12:37.510
And you do a few more
additions, but given
00:12:37.510 --> 00:12:40.820
that the additions have
theta n complexity,
00:12:40.820 --> 00:12:49.610
the recurrence relationship
turns into tn equals 3t of n
00:12:49.610 --> 00:12:52.920
over 2 plus theta n.
00:12:52.920 --> 00:13:03.030
And this ends up having
1.58 dot dot dot complexity.
00:13:03.030 --> 00:13:09.470
No reason to stop with breaking
things up into two parts.
00:13:09.470 --> 00:13:12.960
You could imagine
generalizing Karatsuba
00:13:12.960 --> 00:13:14.235
and people have done this.
00:13:18.200 --> 00:13:22.860
Two different researchers,
Toom and Cook,
00:13:22.860 --> 00:13:26.230
generalized Karatsuba
for the case
00:13:26.230 --> 00:13:29.280
where k is greater than
or equal to 2, where
00:13:29.280 --> 00:13:32.020
you're breaking it into k parts.
00:13:32.020 --> 00:13:36.800
So the Toom-Cook 2 algorithm
is basically Karatsuba,
00:13:36.800 --> 00:13:39.720
but you have Toom 3,
Toom 4, and so on.
00:13:39.720 --> 00:13:44.520
And I'm not going to give
you a lot of details on this.
00:13:44.520 --> 00:13:50.190
We don't expect you to work
on this, at least in 6006.
00:13:50.190 --> 00:13:55.500
But just to give you a
sense of what happens,
00:13:55.500 --> 00:14:00.750
the Toom 3 method, or
the Toom-Cook 3 method,
00:14:00.750 --> 00:14:04.190
breaks and number
up into three parts.
00:14:04.190 --> 00:14:10.290
So each of these would have
d by 3 digits of precision.
00:14:10.290 --> 00:14:12.230
So this is what you're
starting out with.
00:14:12.230 --> 00:14:13.980
You're starting out
with a d digit number.
00:14:13.980 --> 00:14:16.070
But the very first level
of recursion, you're
00:14:16.070 --> 00:14:20.810
going to break things up
into three xi numbers that
00:14:20.810 --> 00:14:22.370
are d by 3 digits long.
00:14:22.370 --> 00:14:23.980
Same thing for y.
00:14:23.980 --> 00:14:27.820
And if you did a naive
multiplication of this,
00:14:27.820 --> 00:14:31.170
how many multiplications
do I need?
00:14:31.170 --> 00:14:34.000
If I just forget about
any mathematical tricks,
00:14:34.000 --> 00:14:38.740
if I just tried to
multiply these things out,
00:14:38.740 --> 00:14:43.680
how many d by 3 by d by 3
multiplications do I need?
00:14:43.680 --> 00:14:44.740
AUDIENCE: Nine.
00:14:44.740 --> 00:14:46.520
PROFESSOR: Nine.
00:14:46.520 --> 00:14:50.390
So if you can beat nine
using mathematical tricks,
00:14:50.390 --> 00:14:54.730
you have a better divide
and conquer algorithm.
00:14:54.730 --> 00:15:02.970
And it turns out that Toom 3
plays some arithmetic games
00:15:02.970 --> 00:15:13.780
and ends up with a
recurrence relationship that
00:15:13.780 --> 00:15:15.790
looks like this.
00:15:15.790 --> 00:15:21.020
Where you reduce the nine
multiplications down to five.
00:15:21.020 --> 00:15:24.110
So that's a win.
00:15:24.110 --> 00:15:33.070
And that ends up being
theta of n raised to what?
00:15:33.070 --> 00:15:35.030
Someone?
00:15:35.030 --> 00:15:37.640
Someone loudly.
00:15:37.640 --> 00:15:38.660
Log--
00:15:38.660 --> 00:15:40.780
AUDIENCE: Base 3.
00:15:40.780 --> 00:15:43.550
PROFESSOR: Log
with a base 3 of 5.
00:15:43.550 --> 00:15:46.440
Another irrational number.
00:15:46.440 --> 00:15:51.940
And this ends up being
n raised to 1.465.
00:15:51.940 --> 00:15:53.400
So you won.
00:15:53.400 --> 00:15:56.850
If you use Toom 3, assuming
the constants worked out--
00:15:56.850 --> 00:15:58.990
and Victor can say
a little bit more
00:15:58.990 --> 00:16:04.290
about that, because we're having
a little trouble justifying
00:16:04.290 --> 00:16:05.810
this particular
problem set question
00:16:05.810 --> 00:16:09.670
that we want to give you, given
the constant factors involved.
00:16:09.670 --> 00:16:14.400
So the issue really
here is this is correct.
00:16:14.400 --> 00:16:16.550
It's n raised to 1.46.
00:16:16.550 --> 00:16:18.640
That's n raised to 1.5.
00:16:18.640 --> 00:16:20.980
And then the naive
algorithm is n square.
00:16:20.980 --> 00:16:24.470
But how big does n
have to be in order
00:16:24.470 --> 00:16:27.440
for the n raised
to 1.58 algorithm
00:16:27.440 --> 00:16:30.850
to beat the n square
algorithm, and for the n raised
00:16:30.850 --> 00:16:33.130
to 1.46 algorithm
to beat the n raised
00:16:33.130 --> 00:16:35.450
to 1.58 algorithm, et cetera.
00:16:35.450 --> 00:16:38.010
And it turns out n needs
to be really, really large
00:16:38.010 --> 00:16:39.980
if you implement
these in Python.
00:16:39.980 --> 00:16:42.960
So if you're having a
little trouble here,
00:16:42.960 --> 00:16:45.900
giving you this
pristine problem set
00:16:45.900 --> 00:16:50.550
that you can go off and
learn about multiplication,
00:16:50.550 --> 00:16:52.785
and also appreciate
asymptotic complexity.
00:16:55.310 --> 00:16:58.010
So that's a bit of a catch-22.
00:16:58.010 --> 00:17:02.600
Anyway, for the purposes
of theory, this is great.
00:17:02.600 --> 00:17:04.980
It turns people have
done even better.
00:17:04.980 --> 00:17:10.359
Multiplication is just this
obviously incredibly important
00:17:10.359 --> 00:17:14.220
primitive that you
would need for doing
00:17:14.220 --> 00:17:16.240
any reasonable computation.
00:17:16.240 --> 00:17:22.619
And so people have worked on
using things like fast Fourier
00:17:22.619 --> 00:17:26.900
transforms and other
techniques improve
00:17:26.900 --> 00:17:29.010
the complexity of
multiplication.
00:17:29.010 --> 00:17:37.480
And best scheme
until a few years
00:17:37.480 --> 00:17:42.420
ago was this scheme called
Schonhage-Strassen scheme,
00:17:42.420 --> 00:17:44.920
which is almost
linear in complexity.
00:17:44.920 --> 00:17:53.480
It's n log n log log n time.
00:17:53.480 --> 00:17:58.940
And this uses the fast
Fourier transform, FFT.
00:17:58.940 --> 00:18:00.690
And you can play with
all of these things.
00:18:00.690 --> 00:18:06.210
You can play with Karatsuba
the naive algorithm, Toom 3,
00:18:06.210 --> 00:18:13.220
et cetera in the gmpy
package in Python.
00:18:13.220 --> 00:18:17.541
And you can see as to
what the value of n
00:18:17.541 --> 00:18:19.540
needs to be in order for
one of these algorithms
00:18:19.540 --> 00:18:21.510
to beat the other.
00:18:21.510 --> 00:18:23.007
This is not
something that you're
00:18:23.007 --> 00:18:24.840
going to do specifically
in the problem set,
00:18:24.840 --> 00:18:26.840
but I say that as an aside.
00:18:26.840 --> 00:18:28.230
These algorithms
are implemented,
00:18:28.230 --> 00:18:30.300
and they're used in real life.
00:18:30.300 --> 00:18:30.800
Eric?
00:18:30.800 --> 00:18:32.925
ERIC: It may be worth
mentioning that Python itself
00:18:32.925 --> 00:18:35.660
for long integers
uses Karatsuba.
00:18:35.660 --> 00:18:40.070
PROFESSOR: Yeah, so Python
uses-- beyond a certain n,
00:18:40.070 --> 00:18:42.480
you are going to
have decisions that
00:18:42.480 --> 00:18:44.790
are made within the package.
00:18:44.790 --> 00:18:50.210
And Python shifts to Karatsuba
after n becomes large.
00:18:50.210 --> 00:18:52.370
But if n is small,
then it's going
00:18:52.370 --> 00:18:53.670
to run the naive algorithm.
00:18:53.670 --> 00:18:55.378
Now if you write your
own multiplication,
00:18:55.378 --> 00:18:56.600
you can do whatever you want.
00:18:56.600 --> 00:18:58.910
You can have your own
adaptive scheme, assuming you
00:18:58.910 --> 00:19:01.380
have many of these
algorithms implemented,
00:19:01.380 --> 00:19:04.065
or you're calling them
using the gmpy package.
00:19:06.600 --> 00:19:10.600
So lastly, this looked
pretty good for a while.
00:19:10.600 --> 00:19:14.660
And from a
theoretical standpoint
00:19:14.660 --> 00:19:17.102
there was a breakthrough.
00:19:17.102 --> 00:19:24.620
Guy by the name of Furer came
up with this algorithm that
00:19:24.620 --> 00:19:32.140
is n log n-- and let me write
this carefully-- 2 raised
00:19:32.140 --> 00:19:41.620
big O-- that's an upper
bound-- of log star n.
00:19:41.620 --> 00:19:42.730
That makes sense?
00:19:42.730 --> 00:19:43.563
No.
00:19:43.563 --> 00:19:45.690
I'll have to explain it.
00:19:45.690 --> 00:19:47.310
OK, so what does this mean?
00:19:47.310 --> 00:19:48.790
This part is clear.
00:19:48.790 --> 00:19:50.260
This is like sorting.
00:19:50.260 --> 00:19:53.280
It doesn't need to really use
sorting, but that's n log n.
00:19:53.280 --> 00:19:56.940
And then you have this 2
raised to big O log star n.
00:19:56.940 --> 00:19:58.930
I need to define
what log star n is.
00:19:58.930 --> 00:20:06.460
And log star n is what's called
the iterative algorithm--
00:20:06.460 --> 00:20:07.230
logarithm, rather.
00:20:10.080 --> 00:20:11.740
I guess it's an
iterative algorithm,
00:20:11.740 --> 00:20:14.060
but it computes logs.
00:20:14.060 --> 00:20:17.640
And the iterative
logarithm is the number
00:20:17.640 --> 00:20:37.420
of times log needs to be
applied to get a result that
00:20:37.420 --> 00:20:41.170
is less than or equal to 1.
00:20:41.170 --> 00:20:46.790
So this thing really cuts
you down to size really fast.
00:20:46.790 --> 00:20:48.110
So it doesn't matter.
00:20:48.110 --> 00:20:52.520
You could be a 10 raised
to 24, or 2 raised to 50,
00:20:52.520 --> 00:20:56.000
let's say, if you were
doing binary logs.
00:20:56.000 --> 00:20:59.690
And in the very first iteration
you go down to 50, right?
00:20:59.690 --> 00:21:03.645
And then you take a log of
50 and you go down to about 7
00:21:03.645 --> 00:21:04.990
or something.
00:21:04.990 --> 00:21:07.210
And then you take the log of 7.
00:21:07.210 --> 00:21:11.470
And if you're talking
about base 2, like we were,
00:21:11.470 --> 00:21:14.110
you're down to less than 3.
00:21:14.110 --> 00:21:17.160
And so four or five
iterations, you're
00:21:17.160 --> 00:21:20.370
down to less than or equal to 1.
00:21:20.370 --> 00:21:24.280
And that's what log
star n computes.
00:21:24.280 --> 00:21:28.720
It's not the logarithm as
much as the number of times
00:21:28.720 --> 00:21:32.290
so you have to apply log to
get the result that's less than
00:21:32.290 --> 00:21:33.690
or equal to 1.
00:21:33.690 --> 00:21:36.710
So you have these giant numbers,
and it's only like five,
00:21:36.710 --> 00:21:41.310
six, eight times do you apply
log and you're down to one.
00:21:41.310 --> 00:21:44.060
So for all practical
purposes, you can think of--
00:21:44.060 --> 00:21:46.680
and this is upper bound--
you can think of this,
00:21:46.680 --> 00:21:48.550
even though this is 2
raised to something,
00:21:48.550 --> 00:21:51.320
it's 2 raised to a
pretty small number.
00:21:51.320 --> 00:21:53.470
2 raised to 10,
that would be 1,000.
00:21:53.470 --> 00:21:56.390
And so from an asymptotic
complexity standpoint,
00:21:56.390 --> 00:21:58.340
this is the winner.
00:21:58.340 --> 00:22:02.490
From a practical standpoint,
Schonhage-Strassen
00:22:02.490 --> 00:22:05.480
is really what you
probably want to use
00:22:05.480 --> 00:22:08.220
when n becomes very
large, to the billions
00:22:08.220 --> 00:22:09.780
and so on and so forth.
00:22:09.780 --> 00:22:13.020
And as of now, to the
best of my knowledge
00:22:13.020 --> 00:22:16.310
this hasn't been implemented
in the gmpy package.
00:22:16.310 --> 00:22:23.110
So if you actually want to use
gmpy, this is where you stop.
00:22:23.110 --> 00:22:24.820
So that's multiplication.
00:22:24.820 --> 00:22:26.480
So we have a bunch
of different ways
00:22:26.480 --> 00:22:29.000
that you could do
multiplication.
00:22:29.000 --> 00:22:34.860
What I'd like to do is give
you a sense of assuming a given
00:22:34.860 --> 00:22:40.270
complexity of multiplication,
how long would division take?
00:22:40.270 --> 00:22:47.150
So we are 1 and 1/2 lectures
in, and I haven't really
00:22:47.150 --> 00:22:50.020
told you how we're going
to do division, which
00:22:50.020 --> 00:22:55.140
is what we have to do when we
compute a divided by xi, which
00:22:55.140 --> 00:22:58.236
is the basic integration
in the Newton method.
00:22:58.236 --> 00:22:59.110
So let's get to that.
00:23:19.430 --> 00:23:25.325
So finally
high-precision division.
00:23:30.630 --> 00:23:42.600
So we want a high-precision
rep off a divided by b.
00:23:42.600 --> 00:23:47.500
And we're going to compute
a high-precision rep
00:23:47.500 --> 00:23:52.430
off 1 divided by b first.
00:23:52.430 --> 00:23:59.440
And what we mean by
that is that we'll
00:23:59.440 --> 00:24:10.640
compute r divided
by b floor where
00:24:10.640 --> 00:24:14.645
r is a really large value.
00:24:18.030 --> 00:24:28.250
And more importantly,
it's easy to divide
00:24:28.250 --> 00:24:31.310
by r in a particular base.
00:24:31.310 --> 00:24:34.820
So for example, r
equals 2 raised to k,
00:24:34.820 --> 00:24:38.650
when we use base
2, you can easily
00:24:38.650 --> 00:24:41.240
divide through a shift operator.
00:24:41.240 --> 00:24:44.060
So if I give you r divided
by b and I give you
00:24:44.060 --> 00:24:49.230
this long computer word that's
in base 2, which typically
00:24:49.230 --> 00:24:53.110
could have millions of
digits in its representation,
00:24:53.110 --> 00:24:56.107
I can shift that by
the appropriate amount
00:24:56.107 --> 00:24:58.900
to a given r divided by b.
00:24:58.900 --> 00:25:02.440
I can get 1 over b by
shifting that quantity.
00:25:02.440 --> 00:25:03.970
So it feels like,
hey wait a minute.
00:25:03.970 --> 00:25:05.800
Why are we dividing by r?
00:25:05.800 --> 00:25:08.740
Well remember that
you want 1 over b.
00:25:08.740 --> 00:25:11.980
And if you're computing
r divided by b floor,
00:25:11.980 --> 00:25:15.400
and you actually want 1
over b, which then you
00:25:15.400 --> 00:25:18.740
could use to multiply by a
so you can run your Newton
00:25:18.740 --> 00:25:22.370
iteration, then you
want to divide by r.
00:25:22.370 --> 00:25:24.750
And that division
is essentially going
00:25:24.750 --> 00:25:28.740
to be something that
shifts things to the right.
00:25:28.740 --> 00:25:31.400
So the most significant
bits move to the right,
00:25:31.400 --> 00:25:33.950
and you get a smaller number.
00:25:33.950 --> 00:25:35.810
That make sense?
00:25:35.810 --> 00:25:38.680
So we all know how
to divide by using
00:25:38.680 --> 00:25:41.730
shifting assuming the
bases work out right.
00:25:41.730 --> 00:25:44.390
And if you had a representation
that was decimal,
00:25:44.390 --> 00:25:48.830
suddenly you could certainly
divide by 10 raised to k.
00:25:48.830 --> 00:25:50.330
That's easy.
00:25:50.330 --> 00:25:51.770
You've done this many times.
00:25:51.770 --> 00:25:53.660
But you just changed
the decimal point
00:25:53.660 --> 00:25:55.493
when you're working
with decimal arithmetic.
00:25:55.493 --> 00:25:59.740
When you divide 72 by
100 and you get 0.72.
00:25:59.740 --> 00:26:02.550
And that's a very
similar notion here.
00:26:02.550 --> 00:26:06.310
It doesn't really matter what
base you're talking about.
00:26:06.310 --> 00:26:08.270
So that's the setup.
00:26:08.270 --> 00:26:10.290
That's how are we
going to try and tackle
00:26:10.290 --> 00:26:12.350
this division problem.
00:26:12.350 --> 00:26:18.460
But we still have this problem
of computing r divided by b.
00:26:18.460 --> 00:26:21.700
So how are we going to
compute r divided by b?
00:26:25.430 --> 00:26:29.740
And we want this to be a large
number of digits of precision.
00:26:29.740 --> 00:26:32.070
So we're going to use
Newton's method again.
00:26:36.160 --> 00:26:42.850
You've got some non-linearity
here with respect to 1 over x.
00:26:42.850 --> 00:26:46.380
And we're gonna use
Newton's method again.
00:26:46.380 --> 00:26:48.830
And we'll have to hope
that this works out,
00:26:48.830 --> 00:26:53.830
that we can get Newton's
method, it'll converge,
00:26:53.830 --> 00:26:59.830
and it'll require operations
that we know how to do.
00:26:59.830 --> 00:27:02.420
And all of this is going
to work out really well.
00:27:02.420 --> 00:27:04.460
I'm going to set
up a function, f
00:27:04.460 --> 00:27:14.550
of x equals 1 divided by
x minus b divided by r.
00:27:14.550 --> 00:27:17.230
So what this means is
that this function has
00:27:17.230 --> 00:27:23.790
a 0 at x equals r divided by b.
00:27:23.790 --> 00:27:28.470
So if I try and find
the 0 of this function,
00:27:28.470 --> 00:27:31.230
and I start out with a
decent initial guess,
00:27:31.230 --> 00:27:33.100
I'm going to end up
with r divided by b.
00:27:33.100 --> 00:27:35.000
And if I'm working
with integers,
00:27:35.000 --> 00:27:38.710
effectively that's the floor
that I have for r divided by b.
00:27:38.710 --> 00:27:43.800
And then I do my shift and
I end up with 1 over b.
00:27:43.800 --> 00:27:49.812
So someone who remembers
differentiation,
00:27:49.812 --> 00:27:52.260
if you're gonna apply
Newton's method,
00:27:52.260 --> 00:27:56.630
tell me what the
derivative of f of x is.
00:27:59.454 --> 00:28:00.870
Somebody's stretching
at the back,
00:28:00.870 --> 00:28:03.720
but I don't think
that was an answer.
00:28:03.720 --> 00:28:06.870
Someone at the back?
00:28:06.870 --> 00:28:08.580
Too easy a question?
00:28:08.580 --> 00:28:10.446
For the cushion.
00:28:10.446 --> 00:28:11.945
AUDIENCE: 1 over
negative x squared.
00:28:11.945 --> 00:28:13.486
PROFESSOR: 1 over
negative x squared.
00:28:13.486 --> 00:28:14.710
Who's that?
00:28:14.710 --> 00:28:15.370
All right.
00:28:15.370 --> 00:28:17.650
You can come pick this up.
00:28:17.650 --> 00:28:19.264
Whatever.
00:28:19.264 --> 00:28:20.180
Cut the monotony here.
00:28:22.570 --> 00:28:23.570
Just veered to the left.
00:28:23.570 --> 00:28:26.536
I think next time I'm going
to weight them or something.
00:28:26.536 --> 00:28:27.910
Let's just do
frisbees next time.
00:28:27.910 --> 00:28:30.230
Let's just do
frisbees next time.
00:28:30.230 --> 00:28:31.370
It makes it easy.
00:28:31.370 --> 00:28:33.450
Forget cushions.
00:28:33.450 --> 00:28:34.870
No?
00:28:34.870 --> 00:28:37.050
Frisbees or cushions?
00:28:37.050 --> 00:28:39.300
How many want frisbees?
00:28:39.300 --> 00:28:41.530
How many want cushions?
00:28:41.530 --> 00:28:44.470
Frisbees win.
00:28:44.470 --> 00:28:49.870
So you got derivative of x is
minus 1 divided by x squared.
00:28:49.870 --> 00:28:54.480
And then if you go off and
apply Newton's method--
00:28:54.480 --> 00:28:58.400
and I'm not going to go through
the symbolic equations here
00:28:58.400 --> 00:28:59.950
associated with
Newton's method--
00:28:59.950 --> 00:29:02.780
but that's basically the
same as we did before.
00:29:02.780 --> 00:29:09.380
You are computing a tangent,
and the new value of xi plus 1
00:29:09.380 --> 00:29:12.860
given the value of xi
is the x-intercept.
00:29:12.860 --> 00:29:16.740
And we needed the
derivative to compute that.
00:29:16.740 --> 00:29:21.590
But bottom line, you
have xi plus 1 equals
00:29:21.590 --> 00:29:30.830
xi minus f of xi divided
by f prime of xi.
00:29:30.830 --> 00:29:34.000
So that's the Newton iteration.
00:29:34.000 --> 00:29:42.130
And it's worth plugging in
the various values here.
00:29:42.130 --> 00:29:45.740
1 divided by xi
minus b divided by r.
00:29:45.740 --> 00:29:52.810
That's f of x on top divided by
minus 1 divided by xi square.
00:29:52.810 --> 00:29:54.610
So that's the
derivative over here.
00:29:54.610 --> 00:29:56.260
So all I'm doing is
plugging things in.
00:29:56.260 --> 00:30:00.140
But you want to visualize
this because this is really
00:30:00.140 --> 00:30:01.620
what we need to compute.
00:30:01.620 --> 00:30:10.820
And we have xi plus 1 equals
xi plus xi square times
00:30:10.820 --> 00:30:16.050
1 over xi minus b divided by r.
00:30:16.050 --> 00:30:24.800
And finally I get 2xi minus
b xi square divided by r.
00:30:24.800 --> 00:30:26.800
That is key.
00:30:26.800 --> 00:30:29.690
This is pretty important.
00:30:29.690 --> 00:30:32.380
So let's us look all the
way to the left, which
00:30:32.380 --> 00:30:37.620
is xi plus 1, all the way
to the right, 2 times xi.
00:30:37.620 --> 00:30:40.610
That doesn't scare
us, 2 times something.
00:30:40.610 --> 00:30:43.070
Especially base 2, pretty easy.
00:30:43.070 --> 00:30:43.840
That's a multiply.
00:30:43.840 --> 00:30:45.340
Multiplies don't
scare us because we
00:30:45.340 --> 00:30:47.090
know how to do
multiplies anyway.
00:30:47.090 --> 00:30:49.810
This is a simple multiply.
00:30:49.810 --> 00:30:52.061
And then I got a square here.
00:30:52.061 --> 00:30:52.560
Square.
00:30:52.560 --> 00:30:54.130
Not a square root.
00:30:54.130 --> 00:30:57.240
Squares don't scare us
because that's a multiply,
00:30:57.240 --> 00:30:59.700
just multiplying the
same number to itself.
00:30:59.700 --> 00:31:01.230
And this doesn't
scare us because we
00:31:01.230 --> 00:31:06.620
know that we've chosen r
to be an easy division.
00:31:06.620 --> 00:31:12.140
So all of the operations
here are either easy,
00:31:12.140 --> 00:31:15.720
or they require a multiply.
00:31:15.720 --> 00:31:19.290
So remember I'm going to put a
picture up towards the end here
00:31:19.290 --> 00:31:23.280
that tells you the overall
structure for computing
00:31:23.280 --> 00:31:25.400
square root of a or
square root of 2.
00:31:25.400 --> 00:31:29.010
But we've just sort of sold
out to Newton, if you will.
00:31:29.010 --> 00:31:32.360
Because we said that we're
going to use Newton's method
00:31:32.360 --> 00:31:39.960
to compute essentially,
iteratively, square root of a.
00:31:39.960 --> 00:31:43.350
And within the Newton
method, the first iteration,
00:31:43.350 --> 00:31:44.890
if you will, of
the Newton method,
00:31:44.890 --> 00:31:47.750
we had to compute a reciprocal.
00:31:47.750 --> 00:31:49.730
We had to compute 1 over xi.
00:31:49.730 --> 00:31:52.070
And in order to
compute 1 over xi,
00:31:52.070 --> 00:31:56.320
we're going to apply Newton's
method again like I showed over
00:31:56.320 --> 00:31:58.270
here and over there.
00:31:58.270 --> 00:32:03.920
And so that division is
going to require iteration.
00:32:03.920 --> 00:32:09.150
But the iteration at the second
level is one of multiplication.
00:32:09.150 --> 00:32:11.180
You're gonna repeatedly
apply multiplication
00:32:11.180 --> 00:32:13.480
because you're going
to go xi plus 1
00:32:13.480 --> 00:32:17.890
based on xi using multiplication
and some easy operations.
00:32:17.890 --> 00:32:22.070
And then you go xi plus 2, xi
plus 3, and so on and so forth.
00:32:22.070 --> 00:32:24.062
That make sense?
00:32:24.062 --> 00:32:27.590
I'll try and put this up to
give you the complete picture
00:32:27.590 --> 00:32:32.250
once we're done talking
about the division
00:32:32.250 --> 00:32:35.736
algorithm and its complexity.
00:32:35.736 --> 00:32:37.110
But before I do
that, I just want
00:32:37.110 --> 00:32:41.440
to give you a sense of the
convergence of this scheme.
00:32:41.440 --> 00:32:43.820
Again, I want to give
you an example first,
00:32:43.820 --> 00:32:46.000
and then I'll argue
about the convergence.
00:32:50.330 --> 00:32:52.230
You have to run
this iteratively.
00:32:52.230 --> 00:32:54.820
You've got to make i
to get to the point
00:32:54.820 --> 00:32:59.230
where it's large enough that you
have your digits of precision.
00:32:59.230 --> 00:33:01.340
And just as an
example, let's say
00:33:01.340 --> 00:33:08.387
we want r divided by b equals
2 raised to 16 divided by 5.
00:33:08.387 --> 00:33:10.220
So this is a fairly
straightforward example.
00:33:10.220 --> 00:33:14.660
But when you get up to integers,
it turns out it's evocative.
00:33:14.660 --> 00:33:21.760
So r was selected to be 2 raised
to k to make for easy division.
00:33:21.760 --> 00:33:26.780
And what I really want is that.
00:33:26.780 --> 00:33:31.540
And I want to see how I get
to that using Newton's method.
00:33:31.540 --> 00:33:42.180
And our initial
guess, let's say we
00:33:42.180 --> 00:33:45.110
try 2 raised to 16
divided by 4, because we
00:33:45.110 --> 00:33:49.010
know how to divide
by a power of two.
00:33:49.010 --> 00:33:50.600
And so that's 2 raised to 14.
00:33:50.600 --> 00:33:51.890
And that's our initial guess.
00:33:51.890 --> 00:33:56.400
So think of that as being x0.
00:33:56.400 --> 00:33:58.160
That is x0.
00:33:58.160 --> 00:34:02.834
And that 16384.
00:34:02.834 --> 00:34:10.679
x1 is going to be 2 times
16384, which is exactly that,
00:34:10.679 --> 00:34:16.850
minus 5 times
16384 whole square.
00:34:16.850 --> 00:34:19.610
So now you're starting to
square a fairly big number.
00:34:19.610 --> 00:34:22.159
And obviously if you'd
started with an even bigger r,
00:34:22.159 --> 00:34:23.820
this would be a large number.
00:34:26.920 --> 00:34:35.661
You go 65536 equals--
and this is 12288.
00:34:35.661 --> 00:34:38.989
So you really have one
digit of precision there.
00:34:38.989 --> 00:34:46.440
But the next time around,
you get 2 times 12288 minus 5
00:34:46.440 --> 00:34:53.159
times 12288 square
divided by 65536.
00:34:53.159 --> 00:34:55.239
And this division is easy.
00:34:55.239 --> 00:34:55.780
It's a shift.
00:34:55.780 --> 00:34:59.810
You get to 13056.
00:34:59.810 --> 00:35:01.640
And I won't write
this whole thing out,
00:35:01.640 --> 00:35:07.080
but if you take that, the next
thing you'll get is 13107.
00:35:07.080 --> 00:35:11.710
So as you can see, there's
rapid convergence here.
00:35:11.710 --> 00:35:16.660
And you can actually do a very
similar analysis to the epsilon
00:35:16.660 --> 00:35:18.620
analysis-- and I'll
put it in the notes,
00:35:18.620 --> 00:35:22.220
but I won't do it here-- that
I did for the square root
00:35:22.220 --> 00:35:24.860
iteration to show that you
have a quadratic the rate
00:35:24.860 --> 00:35:31.800
of convergence when you apply
Newton's method to division as
00:35:31.800 --> 00:35:33.700
well.
00:35:33.700 --> 00:35:38.190
So you can prove that using
the symbolic analysis than we
00:35:38.190 --> 00:35:41.420
did very similar to the
epsilon n relationship
00:35:41.420 --> 00:35:42.949
to epsilon n plus 1.
00:35:42.949 --> 00:35:44.740
I'd suggest that it's
a difference equation
00:35:44.740 --> 00:35:48.230
here so that analysis
is not exactly the same.
00:35:48.230 --> 00:35:50.330
But you can run
through that, and you
00:35:50.330 --> 00:35:53.200
can read that in the notes.
00:35:53.200 --> 00:35:54.700
So we're in business.
00:35:54.700 --> 00:35:57.250
Finally things are
looking up with respect
00:35:57.250 --> 00:36:00.850
to being able to actually
implement this in practice.
00:36:00.850 --> 00:36:02.850
I want to talk about complexity.
00:36:02.850 --> 00:36:05.720
And I promise that there
was a subtlety associated
00:36:05.720 --> 00:36:11.400
with the complexity of division
in relation to multiplication,
00:36:11.400 --> 00:36:16.490
but let me just go over and
write down what I just told you
00:36:16.490 --> 00:36:19.220
with respect to the
number of iterations
00:36:19.220 --> 00:36:23.150
that division requires.
00:36:23.150 --> 00:36:30.880
So division,
quadratic convergence.
00:36:35.020 --> 00:36:43.470
So number of digits
doubles at each step.
00:36:43.470 --> 00:36:44.730
Good news.
00:36:44.730 --> 00:36:56.770
So d digits of precision,
log d iterations.
00:37:00.130 --> 00:37:05.320
Now let's say that we have
a particular algorithm
00:37:05.320 --> 00:37:08.820
for multiplication that
I'm just going to say,
00:37:08.820 --> 00:37:13.330
since we have so many
different algorithms,
00:37:13.330 --> 00:37:18.660
I'm going to say multiplication
in theta n raised
00:37:18.660 --> 00:37:24.290
to alpha time, where alpha is
greater than or equal to 1.
00:37:24.290 --> 00:37:26.750
I just want to be
general about it.
00:37:26.750 --> 00:37:32.640
And so assuming that I have a
multiplication algorithm, that
00:37:32.640 --> 00:37:35.150
can run in theta
n raised to alpha,
00:37:35.150 --> 00:37:40.950
where clearly you know alpha can
be 1.46 for Toom 3, et cetera.
00:37:40.950 --> 00:37:45.360
And it's not quite that
for Schonhage-Strassen,
00:37:45.360 --> 00:37:50.450
but I just want to be working
with one particular complexity.
00:37:50.450 --> 00:37:52.340
So I'll parameterize
it in this fashion.
00:37:52.340 --> 00:37:56.620
And everything I say is going to
be true for Schonhage-Strassen
00:37:56.620 --> 00:37:58.090
and Furer as well.
00:37:58.090 --> 00:38:00.890
But first, easy question.
00:38:00.890 --> 00:38:03.890
What is the
complexity of division
00:38:03.890 --> 00:38:09.610
using the analysis that I've
put on the board so far?
00:38:09.610 --> 00:38:14.064
n digit numbers
it's going to be?
00:38:14.064 --> 00:38:14.980
I wanna hear from you.
00:38:17.630 --> 00:38:21.640
How many hard
multipliers do I have?
00:38:24.520 --> 00:38:25.476
Log of?
00:38:25.476 --> 00:38:26.270
AUDIENCE: n.
00:38:26.270 --> 00:38:27.530
PROFESSOR: Log of n, right?
00:38:27.530 --> 00:38:30.400
It wasn't a hard question.
00:38:30.400 --> 00:38:37.240
So division would be theta
log n times n raised to alpha.
00:38:40.059 --> 00:38:40.850
Everybody buy that?
00:38:44.651 --> 00:38:45.150
No?
00:38:50.144 --> 00:38:51.560
Ask a question if
you're confused.
00:38:55.610 --> 00:39:00.620
Maybe I should say
everybody buy that?
00:39:04.797 --> 00:39:06.130
How many people agree with that?
00:39:06.130 --> 00:39:07.737
Big O?
00:39:07.737 --> 00:39:09.070
How many people agree with that?
00:39:12.099 --> 00:39:12.890
Yeah, that's right.
00:39:12.890 --> 00:39:16.170
Big O. I'm hedging my bets here.
00:39:16.170 --> 00:39:19.680
I'm just saying big O. I
could say big O of n cubed
00:39:19.680 --> 00:39:21.370
and you should
all agree with me.
00:39:21.370 --> 00:39:22.780
Or big O of whatever.
00:39:22.780 --> 00:39:23.816
You had a question?
00:39:23.816 --> 00:39:25.482
AUDIENCE: What's the
longest [INAUDIBLE]
00:39:25.482 --> 00:39:27.565
number of [INAUDIBLE]
we need to get
00:39:27.565 --> 00:39:29.310
a certain level of [INAUDIBLE]?
00:39:29.310 --> 00:39:30.310
PROFESSOR: That's right.
00:39:30.310 --> 00:39:35.990
So if you want d
digits of precision,
00:39:35.990 --> 00:39:41.002
then according to this
argument-- and I think you
00:39:41.002 --> 00:39:42.710
guys are a little
doubtful here because I
00:39:42.710 --> 00:39:44.990
kept talking about subtleties,
and in fact there's
00:39:44.990 --> 00:39:48.910
a subtlety here, which I want
to get to-- but this big O
00:39:48.910 --> 00:39:50.260
thing is perfectly correct.
00:39:50.260 --> 00:39:52.010
But to answer your
question, yes.
00:39:52.010 --> 00:39:53.980
Let's assume that it's
n digits of precision.
00:39:53.980 --> 00:39:56.530
That's what we assume
whether it's n or d.
00:39:56.530 --> 00:39:58.520
You can plug in the
appropriate symbol here.
00:39:58.520 --> 00:40:02.130
And we're saying that, look,
every iteration is bounded
00:40:02.130 --> 00:40:07.050
by n raised to alpha
complexity for the multiply.
00:40:07.050 --> 00:40:08.860
And I'm going to do
a logarithmic number
00:40:08.860 --> 00:40:09.770
of iterations.
00:40:09.770 --> 00:40:13.150
So I end up getting log n
times n raised to alpha.
00:40:13.150 --> 00:40:15.220
So that is correct, in fact.
00:40:15.220 --> 00:40:16.450
Big O is correct.
00:40:16.450 --> 00:40:19.810
So now it comes to the
interesting question,
00:40:19.810 --> 00:40:23.050
which is can you do
a better analysis?
00:40:23.050 --> 00:40:26.250
So this sort of hearkens
back to three weeks
00:40:26.250 --> 00:40:27.750
ago, maybe you've forgotten.
00:40:27.750 --> 00:40:29.560
Maybe you've blanked
it out of your memory,
00:40:29.560 --> 00:40:34.670
but I thought I described
to you build max-heap.
00:40:34.670 --> 00:40:36.540
And we had this
straightforward analysis
00:40:36.540 --> 00:40:39.379
of build max-heap that
was n log n complexity.
00:40:39.379 --> 00:40:41.420
And then we looked at it
a little more carefully,
00:40:41.420 --> 00:40:44.160
and we started adding things
up much more carefully.
00:40:44.160 --> 00:40:45.940
We turned into bank accountants.
00:40:45.940 --> 00:40:49.370
And then we decided that
it was theta n complexity.
00:40:49.370 --> 00:40:50.590
People remember that?
00:40:50.590 --> 00:40:51.090
Right?
00:40:51.090 --> 00:40:52.881
So I want you to turn
into bank accountants
00:40:52.881 --> 00:40:57.920
again, and then tell me first,
there's a nice observation
00:40:57.920 --> 00:41:02.110
that you can make here
that we haven't made yet
00:41:02.110 --> 00:41:05.542
with respect to the
size of these numbers.
00:41:05.542 --> 00:41:07.000
We know what we
want to eventually,
00:41:07.000 --> 00:41:09.416
but there's a nice observation
we can make it with respect
00:41:09.416 --> 00:41:10.930
to the size of these numbers.
00:41:10.930 --> 00:41:14.290
And then we want to
exploit that observation
00:41:14.290 --> 00:41:19.830
to do a better analysis of the
theta complexity of division.
00:41:19.830 --> 00:41:22.962
So who wants to tell me
what the observation is.
00:41:22.962 --> 00:41:25.260
This is definitely
worth a cushion.
00:41:25.260 --> 00:41:26.680
What's the observation?
00:41:26.680 --> 00:41:29.270
I want to end up with
d digits of precision.
00:41:32.930 --> 00:41:35.440
If I give you another hint,
I'm gonna give it away.
00:41:35.440 --> 00:41:38.690
Someone tell me.
00:41:38.690 --> 00:41:42.630
This is a dynamic process, OK?
00:41:42.630 --> 00:41:46.160
So what do I start with?
00:41:46.160 --> 00:41:49.230
What do I start with?
00:41:49.230 --> 00:41:51.154
If I want to compute
something and you
00:41:51.154 --> 00:41:53.320
want to use Newton's method,
what do you start with?
00:41:53.320 --> 00:41:54.242
Yeah?
00:41:54.242 --> 00:41:55.749
AUDIENCE: [INAUDIBLE]
00:41:55.749 --> 00:41:57.790
PROFESSOR: You start with
one digit of precision.
00:41:57.790 --> 00:41:59.507
That's fantastic.
00:41:59.507 --> 00:42:01.590
I don't know if you already
have a cushion or not,
00:42:01.590 --> 00:42:03.140
but here's the second one.
00:42:03.140 --> 00:42:07.830
So you start with a small
number of digits of precision.
00:42:07.830 --> 00:42:12.550
And then you end up with a
large million, whatever, number,
00:42:12.550 --> 00:42:15.100
which is your d.
00:42:15.100 --> 00:42:17.340
So what does that mean?
00:42:17.340 --> 00:42:19.780
So now somebody take
that and run with it.
00:42:19.780 --> 00:42:22.770
Somebody take that
and run with it.
00:42:22.770 --> 00:42:24.380
You already have a cushion.
00:42:24.380 --> 00:42:25.050
Like many?
00:42:28.510 --> 00:42:31.720
You guys, usual suspects.
00:42:31.720 --> 00:42:33.620
So someone take that
and run with it.
00:42:33.620 --> 00:42:34.614
What can I do now?
00:42:34.614 --> 00:42:37.030
What does it mean if I start
with a small number of digits
00:42:37.030 --> 00:42:38.520
of precision?
00:42:38.520 --> 00:42:40.230
My initial guess was one, right?
00:42:40.230 --> 00:42:42.270
I mean, that had one
digit of precision.
00:42:42.270 --> 00:42:46.330
And then the number of digits
doubles with each step.
00:42:46.330 --> 00:42:50.270
So is there any
reason why I'm doing,
00:42:50.270 --> 00:42:52.450
if I had d digits of
precision, eventually
00:42:52.450 --> 00:43:00.550
that I'll have to do d digit
multiplies in each iteration?
00:43:00.550 --> 00:43:01.740
Any reason why?
00:43:01.740 --> 00:43:02.526
Yeah.
00:43:02.526 --> 00:43:04.984
AUDIENCE: You don't have to,
because [INAUDIBLE] multiplies
00:43:04.984 --> 00:43:05.468
are going to be trivial.
00:43:05.468 --> 00:43:07.525
And [INAUDIBLE] then you're
going to eventually approach
00:43:07.525 --> 00:43:08.856
the d to the alpha iteration.
00:43:08.856 --> 00:43:09.910
PROFESSOR: That's exactly right.
00:43:09.910 --> 00:43:10.620
Exactly right.
00:43:10.620 --> 00:43:12.400
That's worth a cushion.
00:43:12.400 --> 00:43:15.490
But now I want you
or someone else,
00:43:15.490 --> 00:43:19.540
tell me what the
iteration looks like.
00:43:19.540 --> 00:43:21.250
So this is the key observation.
00:43:21.250 --> 00:43:40.390
The key observation is that if
I want d digits of precision,
00:43:40.390 --> 00:43:42.960
I'm going to start with
maybe one digit of precision.
00:43:42.960 --> 00:43:49.120
So this is d of p, or dig
of p, not to be confused.
00:43:49.120 --> 00:43:52.810
I start with 1, 2, 4,
and I end up with d.
00:43:52.810 --> 00:43:57.460
And our claim was that this
was log d iterations, right?
00:43:57.460 --> 00:44:05.460
So the initial
multiplies are easy.
00:44:05.460 --> 00:44:07.580
Initially you're
doing constant work
00:44:07.580 --> 00:44:10.530
if you have really
small numbers associated
00:44:10.530 --> 00:44:11.840
with these multiplies.
00:44:11.840 --> 00:44:13.940
It's only towards the
end that you end up
00:44:13.940 --> 00:44:16.720
doing a lot more work, right?
00:44:16.720 --> 00:44:24.510
So someone tell me if I
have n raised to alpha,
00:44:24.510 --> 00:44:30.440
and if I say I want
to write an equation.
00:44:30.440 --> 00:44:32.841
And I don't want
to use theta here.
00:44:32.841 --> 00:44:34.340
I'm going to use
constants because I
00:44:34.340 --> 00:44:38.200
want to add up constants,
and it's a little iffy then
00:44:38.200 --> 00:44:40.400
you add up thetas.
00:44:40.400 --> 00:44:43.630
You need to be
looking at constants.
00:44:43.630 --> 00:44:52.330
Now I can imagine that for this
iteration, the very first one,
00:44:52.330 --> 00:44:55.850
that I have something like
c times 1 raised to alpha,
00:44:55.850 --> 00:44:58.070
because it's just a
single digit of precision.
00:44:58.070 --> 00:45:02.210
OK And the next one, I'm
using the same algorithm.
00:45:02.210 --> 00:45:05.646
This is c times 2 raised
to alpha, c times 4
00:45:05.646 --> 00:45:06.330
raised to alpha.
00:45:09.910 --> 00:45:12.890
And then out here
I'm going to have
00:45:12.890 --> 00:45:19.000
c times d by 4 raised to
alpha plus c times d by 2
00:45:19.000 --> 00:45:24.120
raised to alpha plus finally
c times d raised to alpha.
00:45:24.120 --> 00:45:29.022
And someone give me a bound.
00:45:29.022 --> 00:45:30.755
Who wants to give
me a bound on this?
00:45:33.882 --> 00:45:37.370
Who wants to give
me a bound on this?
00:45:37.370 --> 00:45:40.320
Less than or equal to.
00:45:40.320 --> 00:45:42.130
Let's just make it less than.
00:45:42.130 --> 00:45:43.030
What?
00:45:43.030 --> 00:45:43.530
Someone?
00:45:47.500 --> 00:45:49.650
Just plug in a value of alpha.
00:45:49.650 --> 00:45:54.020
And remember your convergent
geometric series and things
00:45:54.020 --> 00:45:55.084
like that.
00:45:55.084 --> 00:45:55.625
What is that?
00:45:58.300 --> 00:45:59.890
Someone?
00:45:59.890 --> 00:46:00.550
Yeah.
00:46:00.550 --> 00:46:03.005
AUDIENCE: Just some constant
times d to the alpha?
00:46:03.005 --> 00:46:04.338
PROFESSOR: That's exactly right.
00:46:04.338 --> 00:46:07.710
Just some constant
times d to the alpha.
00:46:07.710 --> 00:46:12.320
And in fact, you can say,
it's 2c d to the alpha.
00:46:16.100 --> 00:46:17.840
Keep a question for you aside.
00:46:17.840 --> 00:46:18.500
So that' sit.
00:46:18.500 --> 00:46:22.280
That's the little careful
analysis that we had to do,
00:46:22.280 --> 00:46:26.610
which basically without
changing your code, really,
00:46:26.610 --> 00:46:28.810
suddenly gave you a
better complexity.
00:46:28.810 --> 00:46:30.000
Isn't that fun?
00:46:30.000 --> 00:46:31.270
That's always fun.
00:46:31.270 --> 00:46:34.760
You had this neat
algorithm to begin with.
00:46:34.760 --> 00:46:37.840
And bottom line is you're
just computing things
00:46:37.840 --> 00:46:41.130
a little more accurately,
than essentially saying
00:46:41.130 --> 00:46:44.540
that you had to do
all of this work
00:46:44.540 --> 00:46:48.360
with large number of digits of
precision at every iteration.
00:46:48.360 --> 00:46:51.250
The number of digits
actually increases.
00:46:51.250 --> 00:46:52.840
So what does this mean?
00:46:52.840 --> 00:46:56.000
I guess ultimately, the
complexity of division
00:46:56.000 --> 00:46:57.930
is now what?
00:46:57.930 --> 00:47:03.650
It's the same as the complexity
of multiplication, right?
00:47:03.650 --> 00:47:08.570
So regardless of whether we
did a Newton iteration or not,
00:47:08.570 --> 00:47:12.905
the complexity of division.
00:47:24.940 --> 00:47:27.270
You are doing a logarithmic
number of iterations,
00:47:27.270 --> 00:47:29.870
but since eventually
all of the work
00:47:29.870 --> 00:47:32.766
is going to get done
at the end here.
00:47:32.766 --> 00:47:34.765
Most of the work is getting
done at the end when
00:47:34.765 --> 00:47:36.740
you have these long numbers.
00:47:36.740 --> 00:47:40.250
That's basically the
essence of the argument.
00:47:40.250 --> 00:47:44.430
So let me finish up and
talk about the complexity
00:47:44.430 --> 00:47:45.785
of computing square roots.
00:47:51.000 --> 00:47:56.030
And as you can imagine,
even though you
00:47:56.030 --> 00:47:58.830
have two nested Newton
iterations here,
00:47:58.830 --> 00:48:01.850
you can make basically
the same argument.
00:48:01.850 --> 00:48:04.690
So let's recall what
we're doing in terms
00:48:04.690 --> 00:48:06.400
of computing square roots.
00:48:06.400 --> 00:48:09.320
We want to compute
square root of a.
00:48:09.320 --> 00:48:12.270
And we said, well we don't
quite know how to do this.
00:48:12.270 --> 00:48:18.370
We're going to end up doing
10 raised to 2d times a,
00:48:18.370 --> 00:48:20.710
and we're going to run
Newton's method on it.
00:48:20.710 --> 00:48:24.000
So you've got one level
of Newton's method.
00:48:26.590 --> 00:48:29.730
And the iteration here with
respect to Newton's method
00:48:29.730 --> 00:48:40.100
is something like xi plus 1
equals xi plus a divided by xi.
00:48:40.100 --> 00:48:44.970
Now every time you do
that for a particular xi,
00:48:44.970 --> 00:48:49.320
you're going to end up
having to call a division.
00:48:49.320 --> 00:48:52.940
So you're going to
call a division here,
00:48:52.940 --> 00:48:56.050
and then you're going
to call a division here.
00:48:56.050 --> 00:48:58.430
For each iteration you
have to call a division.
00:48:58.430 --> 00:49:00.300
And what we're
saying is, well we're
00:49:00.300 --> 00:49:03.747
going to end up having to call
for each of these division
00:49:03.747 --> 00:49:05.580
methods we're going to
call Newton's method.
00:49:09.260 --> 00:49:16.830
And what that is
something like 2xi
00:49:16.830 --> 00:49:22.520
minus b xi square divided by r.
00:49:22.520 --> 00:49:25.975
And that's going to be a
bunch of multiplications.
00:49:28.600 --> 00:49:30.970
And what we argued up
until this point was
00:49:30.970 --> 00:49:33.460
that the complexity
of the division,
00:49:33.460 --> 00:49:35.640
even though we had a
bunch of iterations here,
00:49:35.640 --> 00:49:37.920
a logarithmic number of
iterations, the complexity
00:49:37.920 --> 00:49:40.000
of the division was the
same as the complexity
00:49:40.000 --> 00:49:42.240
of the multiplication
because the numbers
00:49:42.240 --> 00:49:44.690
started out small and grew big.
00:49:44.690 --> 00:49:45.200
All right?
00:49:45.200 --> 00:49:46.960
Everybody buy that?
00:49:46.960 --> 00:49:49.250
I'm going to use exactly
the same argument
00:49:49.250 --> 00:49:52.500
for this level of
iteration as well.
00:49:52.500 --> 00:49:57.010
And again, when you start out
with the digits of precision
00:49:57.010 --> 00:49:59.180
corresponding to
square root of 2,
00:49:59.180 --> 00:50:01.630
you're going to start
out guessing 1.5,
00:50:01.630 --> 00:50:04.435
which is your initial guess
for the square root of 2,
00:50:04.435 --> 00:50:07.270
and it's going to be a small
number of digits of precision.
00:50:07.270 --> 00:50:09.560
And eventually you'll
get to a million digits.
00:50:09.560 --> 00:50:14.840
So using essentially the
same equation summing,
00:50:14.840 --> 00:50:17.610
you can argue that the
complexity of computing
00:50:17.610 --> 00:50:25.060
square roots is the complexity
of division, which of course is
00:50:25.060 --> 00:50:28.200
the complexity of
multiplication.
00:50:32.500 --> 00:50:34.880
And that's the story.
00:50:34.880 --> 00:50:37.990
So obviously the code would
be a little more complicated
00:50:37.990 --> 00:50:39.890
than a multiplication
code, because you
00:50:39.890 --> 00:50:42.360
have all this control
structure outside of it.
00:50:42.360 --> 00:50:44.920
It's really two nested loops.
00:50:44.920 --> 00:50:47.320
The multiply is getting
called a bunch of times
00:50:47.320 --> 00:50:49.100
to do the divide,
and the divide is
00:50:49.100 --> 00:50:51.840
getting called a bunch of times
to compute the square root.
00:50:51.840 --> 00:50:54.440
But ultimately, because
the numbers are growing
00:50:54.440 --> 00:50:56.690
and you start out with small
numbers, most of the work
00:50:56.690 --> 00:50:58.820
is done when you get to
the millions of digits
00:50:58.820 --> 00:50:59.760
of precision.
00:50:59.760 --> 00:51:03.760
And you basically
have theta n raised
00:51:03.760 --> 00:51:06.940
to alpha complexity for
computing square roots.
00:51:06.940 --> 00:51:09.940
If you have n raised
to alpha multiply,
00:51:09.940 --> 00:51:13.060
and you want n
digits of precision.
00:51:13.060 --> 00:51:13.990
All right?
00:51:13.990 --> 00:51:14.970
See you next time.
00:51:14.970 --> 00:51:16.980
Stick around for questions.