WEBVTT

00:00:00.000 --> 00:00:02.445 align:middle line:90%
[SQUEAKING]

00:00:02.445 --> 00:00:03.912 align:middle line:90%
[RUSTLING]

00:00:03.912 --> 00:00:06.846 align:middle line:90%
[CLICKING]

00:00:06.846 --> 00:00:21.820 align:middle line:90%


00:00:21.820 --> 00:00:24.100 align:middle line:84%
RAMA RAMAKRISHNAN: OK,
so let's get going.

00:00:24.100 --> 00:00:26.880 align:middle line:84%
Today, we're going to talk
about how do you actually

00:00:26.880 --> 00:00:30.400 align:middle line:84%
train a neural network, because
that is the heart of the game

00:00:30.400 --> 00:00:31.560 align:middle line:90%
here.

00:00:31.560 --> 00:00:35.000 align:middle line:84%
So just to recap, we
looked, last class,

00:00:35.000 --> 00:00:37.260 align:middle line:84%
at what it takes to
design a neural network,

00:00:37.260 --> 00:00:40.600 align:middle line:84%
and we made this very important
distinction between the things

00:00:40.600 --> 00:00:43.200 align:middle line:84%
that you are handed by
your problem and the things

00:00:43.200 --> 00:00:45.960 align:middle line:84%
that you have agency over,
that you have control over.

00:00:45.960 --> 00:00:48.800 align:middle line:84%
And we noticed that
the input layer

00:00:48.800 --> 00:00:51.400 align:middle line:84%
for your problem, the
input is the input,

00:00:51.400 --> 00:00:52.442 align:middle line:90%
the output is the output.

00:00:52.442 --> 00:00:54.650 align:middle line:84%
You've got to do something
with the output, something

00:00:54.650 --> 00:00:55.620 align:middle line:90%
that's expected.

00:00:55.620 --> 00:00:57.520 align:middle line:84%
But everything that
happens in the middle

00:00:57.520 --> 00:00:59.140 align:middle line:90%
is actually in your hands.

00:00:59.140 --> 00:01:01.660 align:middle line:84%
And in particular,
we noticed that we

00:01:01.660 --> 00:01:05.019 align:middle line:84%
have to decide how many
hidden layers we want;

00:01:05.019 --> 00:01:08.580 align:middle line:84%
we have to decide, in each
layer, how many neurons to have;

00:01:08.580 --> 00:01:12.475 align:middle line:84%
and then we have to decide
what activation to use,

00:01:12.475 --> 00:01:15.100 align:middle line:84%
even though I'm kind of cheating
when I say that because I told

00:01:15.100 --> 00:01:17.820 align:middle line:84%
you very clearly on Monday
that for the hidden layer

00:01:17.820 --> 00:01:20.640 align:middle line:84%
activations, just go with
the ReLU activation function.

00:01:20.640 --> 00:01:23.230 align:middle line:84%
You don't have to think
deep thoughts about this.

00:01:23.230 --> 00:01:25.480 align:middle line:84%
But the other things are all
choices you have to make,

00:01:25.480 --> 00:01:28.100 align:middle line:84%
and we will talk a bit later
about how do you actually

00:01:28.100 --> 00:01:29.900 align:middle line:90%
make those choices?

00:01:29.900 --> 00:01:30.400 align:middle line:90%
OK.

00:01:30.400 --> 00:01:33.580 align:middle line:84%
Now The rule of thumb,
the rule of thumb

00:01:33.580 --> 00:01:35.820 align:middle line:84%
always is to start
with the simplest

00:01:35.820 --> 00:01:39.560 align:middle line:84%
network you can think of,
and if it gets the job done,

00:01:39.560 --> 00:01:41.500 align:middle line:90%
stop working on it.

00:01:41.500 --> 00:01:45.180 align:middle line:84%
If it's not good enough, make
it slightly more complicated.

00:01:45.180 --> 00:01:48.340 align:middle line:84%
So that's the meta thing you
have to remember always when

00:01:48.340 --> 00:01:49.940 align:middle line:90%
you're designing these things

00:01:49.940 --> 00:01:54.120 align:middle line:84%
OK, so that's what it takes to
design a deep neural network.

00:01:54.120 --> 00:01:56.100 align:middle line:84%
So what we will do
in this class is

00:01:56.100 --> 00:01:59.060 align:middle line:84%
we'll actually take a real
example with real data,

00:01:59.060 --> 00:02:01.760 align:middle line:84%
and then we'll think through
how we would design a network

00:02:01.760 --> 00:02:03.320 align:middle line:90%
to solve this problem.

00:02:03.320 --> 00:02:05.880 align:middle line:84%
And while doing so, we
will cover a whole bunch

00:02:05.880 --> 00:02:09.132 align:middle line:84%
of conceptual foundations
such as optimization,

00:02:09.132 --> 00:02:10.840 align:middle line:84%
loss functions, gradient
descent, and all

00:02:10.840 --> 00:02:12.960 align:middle line:90%
that good stuff.

00:02:12.960 --> 00:02:13.620 align:middle line:90%
All right.

00:02:13.620 --> 00:02:17.640 align:middle line:84%
So the case study,
or the scenario here,

00:02:17.640 --> 00:02:20.720 align:middle line:84%
is we have a data set of
patients made available

00:02:20.720 --> 00:02:23.680 align:middle line:84%
by the Cleveland Clinic,
and essentially, we

00:02:23.680 --> 00:02:25.180 align:middle line:90%
have a bunch of patients.

00:02:25.180 --> 00:02:26.920 align:middle line:84%
And for all these
patients, the setting

00:02:26.920 --> 00:02:29.280 align:middle line:84%
is that they have come
into the Cleveland Clinic,

00:02:29.280 --> 00:02:31.517 align:middle line:84%
and they have not come
in with a heart problem,

00:02:31.517 --> 00:02:33.100 align:middle line:84%
they have come in
with something else.

00:02:33.100 --> 00:02:34.840 align:middle line:84%
Maybe they just came
in for a physical.

00:02:34.840 --> 00:02:37.240 align:middle line:84%
And we measured a whole
bunch of things about them.

00:02:37.240 --> 00:02:39.560 align:middle line:84%
And the kinds of
things we measured

00:02:39.560 --> 00:02:43.878 align:middle line:84%
are demographic information,
like what's their age, gender,

00:02:43.878 --> 00:02:45.920 align:middle line:84%
whether they have any
chest pain at all when they

00:02:45.920 --> 00:02:49.420 align:middle line:84%
came in, blood pressure,
cholesterol, sugar,

00:02:49.420 --> 00:02:50.520 align:middle line:90%
so on and so forth.

00:02:50.520 --> 00:02:51.780 align:middle line:90%
You get the idea.

00:02:51.780 --> 00:02:55.140 align:middle line:84%
Demographic information and a
bunch of biomarker information.

00:02:55.140 --> 00:02:59.580 align:middle line:84%
And then, what the Cleveland
Clinic did was they actually

00:02:59.580 --> 00:03:04.100 align:middle line:84%
tracked these people and
figured out, in the next year,

00:03:04.100 --> 00:03:07.780 align:middle line:84%
did they get diagnosed
with heart disease or not?

00:03:07.780 --> 00:03:09.020 align:middle line:90%
In the next year.

00:03:09.020 --> 00:03:11.330 align:middle line:84%
Which means that maybe
you can build a model,

00:03:11.330 --> 00:03:13.580 align:middle line:84%
when someone comes in, even
though they didn't come in

00:03:13.580 --> 00:03:15.313 align:middle line:84%
for a chest problem,
maybe you can

00:03:15.313 --> 00:03:16.980 align:middle line:84%
predict that something's
going to happen

00:03:16.980 --> 00:03:18.040 align:middle line:90%
to them in the next year.

00:03:18.040 --> 00:03:23.020 align:middle line:84%
It's a nice classic
machine learning setup.

00:03:23.020 --> 00:03:23.620 align:middle line:90%
All right.

00:03:23.620 --> 00:03:26.500 align:middle line:84%
So this is the thing-- so what
we want to do is we can totally

00:03:26.500 --> 00:03:28.880 align:middle line:84%
solve this problem using
decision trees neural net--

00:03:28.880 --> 00:03:31.220 align:middle line:84%
I mean, sorry, random
forest and gradient boosting

00:03:31.220 --> 00:03:33.740 align:middle line:84%
and all that good stuff you
folks may have already learned

00:03:33.740 --> 00:03:35.460 align:middle line:84%
from machine
learning, but we will

00:03:35.460 --> 00:03:37.240 align:middle line:84%
try to solve it using
neural networks.

00:03:37.240 --> 00:03:39.522 align:middle line:84%
This is an example,
of course, of what's

00:03:39.522 --> 00:03:40.980 align:middle line:84%
called structured
data because this

00:03:40.980 --> 00:03:43.940 align:middle line:84%
is all data sitting in the
columns of a spreadsheet.

00:03:43.940 --> 00:03:46.140 align:middle line:84%
So working with
structured data is

00:03:46.140 --> 00:03:49.160 align:middle line:84%
the way we warm up our
knowledge of neural networks.

00:03:49.160 --> 00:03:51.900 align:middle line:84%
And then we will do things like
working with unstructured data,

00:03:51.900 --> 00:03:53.800 align:middle line:84%
starting next week,
with images, and then

00:03:53.800 --> 00:03:55.840 align:middle line:84%
later on with text and
so on and so forth.

00:03:55.840 --> 00:03:57.340 align:middle line:90%
OK, any questions on this?

00:03:57.340 --> 00:04:00.560 align:middle line:90%


00:04:00.560 --> 00:04:01.320 align:middle line:90%
OK.

00:04:01.320 --> 00:04:02.438 align:middle line:90%
Yes?

00:04:02.438 --> 00:04:04.480 align:middle line:84%
STUDENT: Just connected
even to last time's class

00:04:04.480 --> 00:04:07.880 align:middle line:84%
where we took the same example,
and first it was a logistic,

00:04:07.880 --> 00:04:09.540 align:middle line:84%
and then we did
a neural network.

00:04:09.540 --> 00:04:13.380 align:middle line:84%
So the probability in case of
one was 0.85, then was 0.22,

00:04:13.380 --> 00:04:14.560 align:middle line:90%
and here as well.

00:04:14.560 --> 00:04:18.240 align:middle line:84%
How do you know
when to use what?

00:04:18.240 --> 00:04:20.200 align:middle line:84%
Usually in textbooks,
when to use logistic

00:04:20.200 --> 00:04:25.600 align:middle line:84%
or when to use something
else, but in this case,

00:04:25.600 --> 00:04:28.440 align:middle line:84%
when do I complicate it to
neural networks vis-a-vis,

00:04:28.440 --> 00:04:30.220 align:middle line:84%
in this case, maybe
just doing a random?

00:04:30.220 --> 00:04:31.320 align:middle line:84%
RAMA RAMAKRISHNAN:
It's a great question.

00:04:31.320 --> 00:04:32.540 align:middle line:90%
When do you use what?

00:04:32.540 --> 00:04:34.332 align:middle line:84%
So I think there are
two broad dimensions

00:04:34.332 --> 00:04:35.540 align:middle line:90%
that you have to think about.

00:04:35.540 --> 00:04:38.560 align:middle line:84%
One broad dimension
is, how important

00:04:38.560 --> 00:04:41.120 align:middle line:84%
is it that you need to
explain or interpret

00:04:41.120 --> 00:04:43.760 align:middle line:84%
what's going on inside
the model to perhaps

00:04:43.760 --> 00:04:46.320 align:middle line:90%
a non-technical consumer?

00:04:46.320 --> 00:04:48.160 align:middle line:84%
The other dimension
is, how important

00:04:48.160 --> 00:04:50.800 align:middle line:90%
is sheer predictive accuracy?

00:04:50.800 --> 00:04:53.860 align:middle line:84%
In some situations, predictive
accuracy trumps everything else,

00:04:53.860 --> 00:04:55.458 align:middle line:90%
in which case, just go with it.

00:04:55.458 --> 00:04:57.500 align:middle line:84%
In other cases, explainability
becomes a big deal

00:04:57.500 --> 00:05:00.273 align:middle line:84%
because if they can't understand
it, they won't use it.

00:05:00.273 --> 00:05:01.940 align:middle line:84%
And in those cases,
it's probably better

00:05:01.940 --> 00:05:05.320 align:middle line:84%
to go with simpler models, such
as decision trees and neural--

00:05:05.320 --> 00:05:08.720 align:middle line:84%
I mean, not-- decision trees,
maybe even random forests,

00:05:08.720 --> 00:05:10.220 align:middle line:84%
certainly logistic
regression, those

00:05:10.220 --> 00:05:12.340 align:middle line:90%
are all a little more amenable.

00:05:12.340 --> 00:05:15.500 align:middle line:84%
But that said, even
complex black box

00:05:15.500 --> 00:05:17.380 align:middle line:84%
methods like neural
networks, there

00:05:17.380 --> 00:05:20.820 align:middle line:84%
is a whole field called
mechanistic interpretability,

00:05:20.820 --> 00:05:23.820 align:middle line:84%
which seeks to try to get
insight into what's going on

00:05:23.820 --> 00:05:25.700 align:middle line:90%
inside these big black boxes.

00:05:25.700 --> 00:05:30.280 align:middle line:84%
So the story isn't over, but
that's just the first-cut way

00:05:30.280 --> 00:05:33.170 align:middle line:90%
you analyze the problem.

00:05:33.170 --> 00:05:33.670 align:middle line:90%
OK.

00:05:33.670 --> 00:05:36.820 align:middle line:90%
So, let's get going.

00:05:36.820 --> 00:05:39.740 align:middle line:84%
So if you want to
design a network--

00:05:39.740 --> 00:05:40.280 align:middle line:90%
all right.

00:05:40.280 --> 00:05:41.940 align:middle line:90%
So we design the network.

00:05:41.940 --> 00:05:43.940 align:middle line:84%
So we have to choose the
number of hidden layers

00:05:43.940 --> 00:05:45.787 align:middle line:84%
and the number of
neurons in each layer.

00:05:45.787 --> 00:05:47.620 align:middle line:84%
Then we have to pick
the right output layer.

00:05:47.620 --> 00:05:50.307 align:middle line:90%
So here, what I did is--

00:05:50.307 --> 00:05:52.140 align:middle line:84%
the simplest thing you
can do is, of course,

00:05:52.140 --> 00:05:53.760 align:middle line:90%
just have no hidden layer.

00:05:53.760 --> 00:05:58.520 align:middle line:84%
So if you have no hidden layers,
what is that model called?

00:05:58.520 --> 00:06:00.945 align:middle line:90%
Yes, logistic regression.

00:06:00.945 --> 00:06:02.820 align:middle line:84%
So, of course, we want
to do neural networks,

00:06:02.820 --> 00:06:04.720 align:middle line:84%
so I'm going to have one
hidden layer because that's

00:06:04.720 --> 00:06:06.120 align:middle line:90%
the simplest thing I can do.

00:06:06.120 --> 00:06:09.800 align:middle line:84%
And then, I'll confess, I
tried a few different numbers

00:06:09.800 --> 00:06:12.320 align:middle line:84%
of neurons in this thing,
and when I had 16 neurons,

00:06:12.320 --> 00:06:14.185 align:middle line:90%
it actually did pretty well.

00:06:14.185 --> 00:06:15.560 align:middle line:84%
So there was some
trial and error

00:06:15.560 --> 00:06:18.535 align:middle line:84%
that went on before I
landed on the number 16.

00:06:18.535 --> 00:06:20.660 align:middle line:84%
And for some reason, people
always use powers of 2,

00:06:20.660 --> 00:06:22.640 align:middle line:90%
so may as well do that.

00:06:22.640 --> 00:06:25.520 align:middle line:84%
So I tried 4, 8, 16,
and 16 was really good.

00:06:25.520 --> 00:06:28.440 align:middle line:84%
And it turns out,
when I went above 16,

00:06:28.440 --> 00:06:30.322 align:middle line:90%
it started to do badly.

00:06:30.322 --> 00:06:32.280 align:middle line:84%
And it started to do
badly because of something

00:06:32.280 --> 00:06:35.520 align:middle line:84%
called overfitting, which we're
going to talk about later.

00:06:35.520 --> 00:06:37.680 align:middle line:90%
So, yeah, 16.

00:06:37.680 --> 00:06:40.280 align:middle line:84%
And then by default,
I use ReLUs.

00:06:40.280 --> 00:06:41.860 align:middle line:90%
So 16 ReLU neurons.

00:06:41.860 --> 00:06:45.980 align:middle line:84%
And then here, the output
is a categorical output.

00:06:45.980 --> 00:06:47.280 align:middle line:90%
Heart disease, yes or no?

00:06:47.280 --> 00:06:48.320 align:middle line:90%
1 or 0.

00:06:48.320 --> 00:06:49.780 align:middle line:90%
Classification problem.

00:06:49.780 --> 00:06:52.760 align:middle line:84%
Which means that we want to emit
a probability at the very end,

00:06:52.760 --> 00:06:54.940 align:middle line:84%
therefore, we will
use a sigmoid.

00:06:54.940 --> 00:06:57.160 align:middle line:90%
So, so far, so good, right?

00:06:57.160 --> 00:06:59.180 align:middle line:90%
Any questions?

00:06:59.180 --> 00:07:00.420 align:middle line:90%
All right.

00:07:00.420 --> 00:07:04.640 align:middle line:84%
So, we're going to lay
out this network visually.

00:07:04.640 --> 00:07:06.180 align:middle line:90%
So we have an input.

00:07:06.180 --> 00:07:08.580 align:middle line:90%
And so I just have an input.

00:07:08.580 --> 00:07:11.760 align:middle line:84%
And as you will see
here, x1 through x29,

00:07:11.760 --> 00:07:13.100 align:middle line:90%
that's our input layer.

00:07:13.100 --> 00:07:15.120 align:middle line:90%
And you may be wondering, 29?

00:07:15.120 --> 00:07:17.380 align:middle line:90%
Where did he get that from?

00:07:17.380 --> 00:07:20.460 align:middle line:84%
Because that doesn't
seem to be 29 rows here

00:07:20.460 --> 00:07:21.900 align:middle line:90%
of independent variables.

00:07:21.900 --> 00:07:26.420 align:middle line:84%
So it turns out, there are
only 13 input variables here,

00:07:26.420 --> 00:07:29.260 align:middle line:84%
but some of them
are categorical.

00:07:29.260 --> 00:07:32.380 align:middle line:84%
So what I ended up doing is to
take each categorical variable

00:07:32.380 --> 00:07:34.100 align:middle line:90%
and one-hot-encode it.

00:07:34.100 --> 00:07:37.340 align:middle line:84%
And when you do
that, you get to 39--

00:07:37.340 --> 00:07:39.300 align:middle line:90%
I'm sorry, 29.

00:07:39.300 --> 00:07:39.880 align:middle line:90%
All right.

00:07:39.880 --> 00:07:42.300 align:middle line:84%
And when we actually
do the Colab later on,

00:07:42.300 --> 00:07:45.380 align:middle line:84%
I'll show you exactly
how I one-hot-encoded it,

00:07:45.380 --> 00:07:46.920 align:middle line:90%
but that's what I'm doing here.

00:07:46.920 --> 00:07:49.280 align:middle line:90%
That's why you have 29, not 13.

00:07:49.280 --> 00:07:50.040 align:middle line:90%
OK.

00:07:50.040 --> 00:07:54.160 align:middle line:84%
Now, obviously we have decided
on these hidden units, 16 units,

00:07:54.160 --> 00:07:56.240 align:middle line:90%
with nice ReLUs here.

00:07:56.240 --> 00:07:59.520 align:middle line:84%
And then we have an output
layer with a sigmoid.

00:07:59.520 --> 00:08:01.065 align:middle line:84%
And I got bored
of trying to draw

00:08:01.065 --> 00:08:02.940 align:middle line:84%
all these arrows, so I
just gave up and said,

00:08:02.940 --> 00:08:07.880 align:middle line:84%
assume there are arrows
between all these things.

00:08:07.880 --> 00:08:09.880 align:middle line:90%
Good?

00:08:09.880 --> 00:08:10.628 align:middle line:90%
Yeah?

00:08:10.628 --> 00:08:12.920 align:middle line:84%
STUDENT: Yeah, I'm sorry, I
think you already mentioned

00:08:12.920 --> 00:08:15.200 align:middle line:90%
this, but why 16 units?

00:08:15.200 --> 00:08:16.480 align:middle line:90%
RAMA RAMAKRISHNAN: Why 16?

00:08:16.480 --> 00:08:19.360 align:middle line:84%
I tried a bunch of
different numbers of units,

00:08:19.360 --> 00:08:22.060 align:middle line:84%
and at 16, the resulting
model did well,

00:08:22.060 --> 00:08:23.280 align:middle line:90%
so I just went with that.

00:08:23.280 --> 00:08:25.443 align:middle line:84%
STUDENT: And the logic
of why it's a ReLU?

00:08:25.443 --> 00:08:26.860 align:middle line:84%
RAMA RAMAKRISHNAN:
Oh, why a ReLU?

00:08:26.860 --> 00:08:27.360 align:middle line:90%
Yeah.

00:08:27.360 --> 00:08:29.960 align:middle line:84%
So there is just a mountain
of empirical evidence

00:08:29.960 --> 00:08:33.080 align:middle line:84%
that suggests that ReLU
is a really good default

00:08:33.080 --> 00:08:36.880 align:middle line:84%
option for using as
activations in hidden layers.

00:08:36.880 --> 00:08:40.220 align:middle line:84%
There is also a really great
set of theoretical results,

00:08:40.220 --> 00:08:42.240 align:middle line:84%
and I'll allude to some
of them when we actually

00:08:42.240 --> 00:08:45.270 align:middle line:90%
talk about gradient descent.

00:08:45.270 --> 00:08:47.682 align:middle line:90%
Yeah?

00:08:47.682 --> 00:08:48.890 align:middle line:90%
STUDENT: So a quick question.

00:08:48.890 --> 00:08:51.250 align:middle line:84%
You mentioned, in
the input layer,

00:08:51.250 --> 00:08:54.310 align:middle line:84%
how did you get to 29 again
when you had 13 variables?

00:08:54.310 --> 00:08:56.310 align:middle line:84%
RAMA RAMAKRISHNAN: So
some of those 13 variables

00:08:56.310 --> 00:09:00.230 align:middle line:84%
are categorical variables, like
cholesterol, low, medium, high.

00:09:00.230 --> 00:09:02.790 align:middle line:84%
And so I took them and
one-hot-encoded them.

00:09:02.790 --> 00:09:08.950 align:middle line:84%
So if it had five levels, I
would get five columns now.

00:09:08.950 --> 00:09:09.870 align:middle line:90%
Yeah?

00:09:09.870 --> 00:09:13.950 align:middle line:84%
And by the way,
folks, just like--

00:09:13.950 --> 00:09:16.402 align:middle line:84%
yeah, just like-- please
use a microphone so

00:09:16.402 --> 00:09:18.610 align:middle line:84%
that people on the livestream
can hear your question.

00:09:18.610 --> 00:09:19.673 align:middle line:90%
Yeah, go ahead.

00:09:19.673 --> 00:09:21.090 align:middle line:84%
STUDENT: Sorry,
just one question.

00:09:21.090 --> 00:09:23.290 align:middle line:84%
So the vectors, since we
didn't represent them,

00:09:23.290 --> 00:09:25.875 align:middle line:84%
are we assuming every x is
connected to all the units?

00:09:25.875 --> 00:09:27.000 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:09:27.000 --> 00:09:28.542 align:middle line:84%
STUDENT: And this
is also a parameter

00:09:28.542 --> 00:09:30.187 align:middle line:90%
that we have to decide or--

00:09:30.187 --> 00:09:32.270 align:middle line:84%
RAMA RAMAKRISHNAN: That
ends up being the default.

00:09:32.270 --> 00:09:35.590 align:middle line:84%
And we will see deviations
from that assumption

00:09:35.590 --> 00:09:38.537 align:middle line:84%
when we go to Image processing
and language processing and so

00:09:38.537 --> 00:09:40.870 align:middle line:84%
on, but when you're working
with structured data like we

00:09:40.870 --> 00:09:44.070 align:middle line:84%
are doing now, that's
the default. OK,

00:09:44.070 --> 00:09:46.130 align:middle line:90%
so let's keep going.

00:09:46.130 --> 00:09:47.850 align:middle line:90%
So this is what we have.

00:09:47.850 --> 00:09:49.930 align:middle line:84%
So remember what I told
you in the last class.

00:09:49.930 --> 00:09:51.763 align:middle line:84%
Whenever you're working
with these networks,

00:09:51.763 --> 00:09:54.170 align:middle line:84%
get into the habit of
very quickly calculating

00:09:54.170 --> 00:09:55.970 align:middle line:90%
the number of parameters.

00:09:55.970 --> 00:09:58.410 align:middle line:84%
Just do it a few times
the first few times so

00:09:58.410 --> 00:10:01.290 align:middle line:84%
that you really know cold
exactly what's going on.

00:10:01.290 --> 00:10:04.570 align:middle line:84%
So, yeah, how many
parameters do we have here?

00:10:04.570 --> 00:10:05.990 align:middle line:90%
How many weights and biases?

00:10:05.990 --> 00:10:07.358 align:middle line:90%
You can work through it.

00:10:07.358 --> 00:10:09.150 align:middle line:84%
You don't have to tell
me the final number.

00:10:09.150 --> 00:10:14.410 align:middle line:84%
You can say x times y
plus z, stuff like that.

00:10:14.410 --> 00:10:15.810 align:middle line:90%
Yeah?

00:10:15.810 --> 00:10:16.870 align:middle line:90%
STUDENT: 65.

00:10:16.870 --> 00:10:19.495 align:middle line:84%
You have 48 weights
and 17 biases.

00:10:19.495 --> 00:10:21.870 align:middle line:84%
RAMA RAMAKRISHNAN: OK, and
how did you come up with that?

00:10:21.870 --> 00:10:24.370 align:middle line:84%
STUDENT: So for the weights,
you have-- for the first layer,

00:10:24.370 --> 00:10:25.390 align:middle line:90%
it's 2 times 16.

00:10:25.390 --> 00:10:28.210 align:middle line:84%
Then for the second
connection, it's 1 times 16.

00:10:28.210 --> 00:10:32.250 align:middle line:84%
And then the biases, it's
16 hidden plus the outputs.

00:10:32.250 --> 00:10:33.770 align:middle line:90%
RAMA RAMAKRISHNAN: OK.

00:10:33.770 --> 00:10:36.090 align:middle line:90%
Any other views on this?

00:10:36.090 --> 00:10:38.670 align:middle line:84%
STUDENT: I think
it's 29 into 16.

00:10:38.670 --> 00:10:39.690 align:middle line:90%
RAMA RAMAKRISHNAN: 29--

00:10:39.690 --> 00:10:41.550 align:middle line:90%
OK, 29 into 16.

00:10:41.550 --> 00:10:44.110 align:middle line:90%
STUDENT: And then 16 into plus--

00:10:44.110 --> 00:10:45.290 align:middle line:90%
I mean 16 there.

00:10:45.290 --> 00:10:46.290 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

00:10:46.290 --> 00:10:48.838 align:middle line:84%
STUDENT: And then biases,
16 biases and one bias.

00:10:48.838 --> 00:10:49.880 align:middle line:90%
RAMA RAMAKRISHNAN: Right.

00:10:49.880 --> 00:10:53.670 align:middle line:84%
So, the way it's going to
work is we have 29 things

00:10:53.670 --> 00:10:55.370 align:middle line:90%
here, 16 in the middle.

00:10:55.370 --> 00:10:58.390 align:middle line:90%
So 29 into 16 arrows.

00:10:58.390 --> 00:11:00.270 align:middle line:84%
And then for each
of these fellows,

00:11:00.270 --> 00:11:04.990 align:middle line:84%
there is a bias coming
in, so that's another 16.

00:11:04.990 --> 00:11:09.590 align:middle line:84%
Plus, you have 16
times 1, which is here,

00:11:09.590 --> 00:11:12.070 align:middle line:84%
plus there is one
bias for this one.

00:11:12.070 --> 00:11:16.750 align:middle line:90%
So the total is 497.

00:11:16.750 --> 00:11:19.830 align:middle line:84%
So, you can see here, there
is something very interesting

00:11:19.830 --> 00:11:22.350 align:middle line:84%
going on, which is that
when you go from one layer

00:11:22.350 --> 00:11:25.390 align:middle line:84%
to another layer,
the number of weights

00:11:25.390 --> 00:11:29.610 align:middle line:84%
is roughly on the order of A
times B. The number of units.

00:11:29.610 --> 00:11:31.270 align:middle line:84%
And so that's a
dramatic explosion

00:11:31.270 --> 00:11:33.470 align:middle line:90%
in the number of parameters.

00:11:33.470 --> 00:11:35.750 align:middle line:84%
And that's something we
have to watch for later on

00:11:35.750 --> 00:11:38.090 align:middle line:90%
to prevent overfitting.

00:11:38.090 --> 00:11:40.250 align:middle line:84%
OK, that's where the
explosion of parameters

00:11:40.250 --> 00:11:42.730 align:middle line:84%
comes from the fact that
each layer is fully connected

00:11:42.730 --> 00:11:44.590 align:middle line:90%
to the next layer.

00:11:44.590 --> 00:11:46.210 align:middle line:90%
But we'll revisit this later on.

00:11:46.210 --> 00:11:47.142 align:middle line:90%
OK.

00:11:47.142 --> 00:11:49.890 align:middle line:84%
So what I'm going
to do now is I'm

00:11:49.890 --> 00:11:52.930 align:middle line:84%
going to actually translate this
network, the one that we have

00:11:52.930 --> 00:11:56.010 align:middle line:84%
laid out graphically,
into Keras code

00:11:56.010 --> 00:11:58.810 align:middle line:90%
to demonstrate how easy it is.

00:11:58.810 --> 00:12:02.890 align:middle line:84%
So I will give a fuller intro to
Keras and TensorFlow later on,

00:12:02.890 --> 00:12:06.250 align:middle line:84%
but for now, just
suspend your disbelief,

00:12:06.250 --> 00:12:09.850 align:middle line:84%
we will just try to do it in
Keras as if we know Keras.

00:12:09.850 --> 00:12:10.730 align:middle line:90%
So let's try that.

00:12:10.730 --> 00:12:12.610 align:middle line:84%
Later on, we'll get into
all the gory details

00:12:12.610 --> 00:12:15.130 align:middle line:84%
and train it in Colab
and so on and so forth.

00:12:15.130 --> 00:12:16.210 align:middle line:90%
OK, all right.

00:12:16.210 --> 00:12:19.610 align:middle line:84%
So, the way we
typically do it is

00:12:19.610 --> 00:12:21.950 align:middle line:84%
that once we have a
network like this,

00:12:21.950 --> 00:12:25.290 align:middle line:84%
we typically start from the left
and start defining each layer

00:12:25.290 --> 00:12:26.710 align:middle line:90%
in Keras, one after the other.

00:12:26.710 --> 00:12:28.210 align:middle line:90%
So we flow left to right.

00:12:28.210 --> 00:12:30.930 align:middle line:90%
So let's take the input layer.

00:12:30.930 --> 00:12:34.730 align:middle line:84%
The way you define an input
layer in Keras is really easy.

00:12:34.730 --> 00:12:38.470 align:middle line:90%
You literally say keras.Input.

00:12:38.470 --> 00:12:42.510 align:middle line:84%
And then you tell Keras how
many nodes you have in the input

00:12:42.510 --> 00:12:43.350 align:middle line:90%
coming in.

00:12:43.350 --> 00:12:45.950 align:middle line:84%
In this case, it happens to be
29, so you tell it the shape,

00:12:45.950 --> 00:12:47.210 align:middle line:90%
shape equals 29.

00:12:47.210 --> 00:12:49.870 align:middle line:84%
And the reason why we say
shape as opposed to length

00:12:49.870 --> 00:12:52.110 align:middle line:84%
is because, as you
will see later on,

00:12:52.110 --> 00:12:54.190 align:middle line:84%
we don't have to
just send vectors in.

00:12:54.190 --> 00:12:57.310 align:middle line:84%
We can send complicated
things into Keras.

00:12:57.310 --> 00:13:00.010 align:middle line:84%
And those complicated
objects could be matrices,

00:13:00.010 --> 00:13:02.590 align:middle line:84%
it could be 3D cubes,
it could be 4D tensors,

00:13:02.590 --> 00:13:06.497 align:middle line:84%
and so on and so forth,
so it's expecting a shape.

00:13:06.497 --> 00:13:08.830 align:middle line:84%
What is the shape of this
thing you're going to send me?

00:13:08.830 --> 00:13:10.288 align:middle line:84%
In this particular
case, it happens

00:13:10.288 --> 00:13:12.570 align:middle line:84%
to be a nice list or
a vector, so it's 29.

00:13:12.570 --> 00:13:13.590 align:middle line:90%
OK, that's it.

00:13:13.590 --> 00:13:15.650 align:middle line:90%
So we write this down.

00:13:15.650 --> 00:13:19.150 align:middle line:84%
This creates the input
layer, and we give it a name.

00:13:19.150 --> 00:13:22.910 align:middle line:84%
And the name here means this
layer, whatever comes out

00:13:22.910 --> 00:13:24.370 align:middle line:90%
of this layer has a name input.

00:13:24.370 --> 00:13:27.790 align:middle line:90%


00:13:27.790 --> 00:13:28.570 align:middle line:90%
Good.

00:13:28.570 --> 00:13:30.012 align:middle line:90%
Next.

00:13:30.012 --> 00:13:31.470 align:middle line:84%
You specify the
shape of the input,

00:13:31.470 --> 00:13:34.330 align:middle line:90%
as I mentioned, right there.

00:13:34.330 --> 00:13:35.730 align:middle line:90%
Then we go to the next one.

00:13:35.730 --> 00:13:37.790 align:middle line:84%
And here-- and we
will unpack this,

00:13:37.790 --> 00:13:41.370 align:middle line:84%
the way you define a
layer is typically--

00:13:41.370 --> 00:13:43.490 align:middle line:84%
a hidden layer,
keras.layers.Dense

00:13:43.490 --> 00:13:44.650 align:middle line:90%
and all this stuff.

00:13:44.650 --> 00:13:47.270 align:middle line:90%
So what this is is--

00:13:47.270 --> 00:13:50.070 align:middle line:84%
first of all, it says,
I want a dense layer.

00:13:50.070 --> 00:13:51.930 align:middle line:84%
By dense layer, I
mean a layer that's

00:13:51.930 --> 00:13:55.090 align:middle line:84%
going to fully connect to the
prior and the later layers.

00:13:55.090 --> 00:13:55.750 align:middle line:90%
Fully connect.

00:13:55.750 --> 00:13:58.130 align:middle line:84%
That's what the
word "dense" means.

00:13:58.130 --> 00:14:03.410 align:middle line:84%
Number two, I want 16
nodes here in this layer.

00:14:03.410 --> 00:14:06.890 align:middle line:90%
Finally, I want to use a ReLU.

00:14:06.890 --> 00:14:09.977 align:middle line:84%
See how compact and
parsimonious it is.

00:14:09.977 --> 00:14:11.310 align:middle line:90%
And that is the appeal of Keras.

00:14:11.310 --> 00:14:13.620 align:middle line:90%
It's very easy to get going.

00:14:13.620 --> 00:14:15.370 align:middle line:84%
So the moment you do
that, you've actually

00:14:15.370 --> 00:14:18.570 align:middle line:90%
defined this layer.

00:14:18.570 --> 00:14:21.770 align:middle line:84%
But what you have not
done is you have not

00:14:21.770 --> 00:14:25.370 align:middle line:84%
told this layer what
input is going to get.

00:14:25.370 --> 00:14:27.350 align:middle line:84%
Because as far as this
layer is concerned,

00:14:27.350 --> 00:14:30.330 align:middle line:84%
it doesn't know that
this other layer exists.

00:14:30.330 --> 00:14:31.630 align:middle line:90%
So you need to connect them.

00:14:31.630 --> 00:14:33.670 align:middle line:90%
Yes?

00:14:33.670 --> 00:14:35.910 align:middle line:84%
STUDENT: Do we need
to define for the ReLU

00:14:35.910 --> 00:14:37.510 align:middle line:90%
where the bends are?

00:14:37.510 --> 00:14:38.975 align:middle line:90%
Like, where you take the max?

00:14:38.975 --> 00:14:41.350 align:middle line:84%
RAMA RAMAKRISHNAN: No, the
ReLU, the bend is always at 0.

00:14:41.350 --> 00:14:41.850 align:middle line:90%
STUDENT: OK.

00:14:41.850 --> 00:14:42.410 align:middle line:90%
Thank you.

00:14:42.410 --> 00:14:45.630 align:middle line:90%


00:14:45.630 --> 00:14:47.350 align:middle line:90%
RAMA RAMAKRISHNAN: OK.

00:14:47.350 --> 00:14:48.790 align:middle line:90%
All right.

00:14:48.790 --> 00:14:51.230 align:middle line:90%
So that's what we have here.

00:14:51.230 --> 00:14:54.470 align:middle line:84%
And then, what we do
is we have to tell it--

00:14:54.470 --> 00:14:57.670 align:middle line:84%
you want to feed this layer, the
output of the previous layer.

00:14:57.670 --> 00:15:00.270 align:middle line:84%
So you feed it by taking
whatever is coming out

00:15:00.270 --> 00:15:02.090 align:middle line:84%
of this thing, which
is called input,

00:15:02.090 --> 00:15:05.470 align:middle line:84%
and you basically
stick it in here.

00:15:05.470 --> 00:15:07.790 align:middle line:84%
So the moment you
do that, boom, it's

00:15:07.790 --> 00:15:10.470 align:middle line:84%
going to receive the input
from the previous layer.

00:15:10.470 --> 00:15:14.470 align:middle line:84%
And because this one's output
needs to go to the final layer,

00:15:14.470 --> 00:15:16.510 align:middle line:84%
you need to give a
name to that output.

00:15:16.510 --> 00:15:17.490 align:middle line:90%
So you give it a name.

00:15:17.490 --> 00:15:18.990 align:middle line:84%
I'm just calling
it h4 because it's

00:15:18.990 --> 00:15:20.550 align:middle line:90%
coming out of the hidden layer.

00:15:20.550 --> 00:15:22.883 align:middle line:84%
It's just a variable, you can
call it anything you want.

00:15:22.883 --> 00:15:24.990 align:middle line:90%


00:15:24.990 --> 00:15:28.950 align:middle line:84%
Now what we do, we go to
the final output layer,

00:15:28.950 --> 00:15:30.070 align:middle line:90%
and this is what we use.

00:15:30.070 --> 00:15:32.810 align:middle line:84%
The output layer is just
another dense layer.

00:15:32.810 --> 00:15:34.410 align:middle line:90%
That's why I use the word dense.

00:15:34.410 --> 00:15:36.810 align:middle line:84%
But we say, hey, give me
just one thing because I just

00:15:36.810 --> 00:15:39.410 align:middle line:84%
literally just need one
unit here because I need

00:15:39.410 --> 00:15:41.930 align:middle line:90%
to emit just one probability.

00:15:41.930 --> 00:15:46.950 align:middle line:84%
And the activation I
want to use is a sigmoid.

00:15:46.950 --> 00:15:47.450 align:middle line:90%
Done.

00:15:47.450 --> 00:15:50.410 align:middle line:90%


00:15:50.410 --> 00:15:53.490 align:middle line:84%
And once you do that,
you have to feed

00:15:53.490 --> 00:15:57.810 align:middle line:84%
it the input from the second
layer, so you stick an h here.

00:15:57.810 --> 00:16:01.370 align:middle line:84%
Now we have connected the
third and the second layers.

00:16:01.370 --> 00:16:04.050 align:middle line:84%
And after you do that, you give
a name to the output coming out

00:16:04.050 --> 00:16:05.790 align:middle line:84%
of that, we'll just
call it output.

00:16:05.790 --> 00:16:07.540 align:middle line:84%
You can call it y, you
can call it output,

00:16:07.540 --> 00:16:09.730 align:middle line:84%
you can call it
whatever you want.

00:16:09.730 --> 00:16:12.010 align:middle line:84%
So at this point,
what we have done

00:16:12.010 --> 00:16:16.210 align:middle line:84%
is we have mapped that picture
into those three lines.

00:16:16.210 --> 00:16:16.810 align:middle line:90%
That's it.

00:16:16.810 --> 00:16:19.370 align:middle line:90%


00:16:19.370 --> 00:16:20.710 align:middle line:90%
But we aren't quite done yet.

00:16:20.710 --> 00:16:22.490 align:middle line:84%
There's one little
thing we have to do.

00:16:22.490 --> 00:16:26.990 align:middle line:84%
So what we have to do is we
have to formally define a model

00:16:26.990 --> 00:16:30.130 align:middle line:84%
so that Keras can just work
with this model object--

00:16:30.130 --> 00:16:31.850 align:middle line:84%
it can train it,
it can evaluate it,

00:16:31.850 --> 00:16:34.017 align:middle line:84%
it can use it for prediction
and so on and so forth.

00:16:34.017 --> 00:16:36.810 align:middle line:84%
So we tell Keras, hey,
create a model for me,

00:16:36.810 --> 00:16:40.770 align:middle line:84%
keras.Model, and basically where
the input is this thing here

00:16:40.770 --> 00:16:42.470 align:middle line:84%
and the output is
that thing there.

00:16:42.470 --> 00:16:45.950 align:middle line:84%
And then the whole thing,
we'll just call it model.

00:16:45.950 --> 00:16:48.550 align:middle line:90%
So that's it.

00:16:48.550 --> 00:16:49.250 align:middle line:90%
We are done.

00:16:49.250 --> 00:16:50.403 align:middle line:90%
That is the whole model.

00:16:50.403 --> 00:16:52.070 align:middle line:84%
That is-- it sounds
really fancy, right?

00:16:52.070 --> 00:16:54.650 align:middle line:84%
A neural model for heart
disease prediction.

00:16:54.650 --> 00:16:56.670 align:middle line:90%
That's pretty cool.

00:16:56.670 --> 00:16:58.630 align:middle line:90%
Four lines.

00:16:58.630 --> 00:17:01.690 align:middle line:84%
And we will show how to train
this model with real data,

00:17:01.690 --> 00:17:05.270 align:middle line:84%
and so on and so forth, and
use it for prediction after we

00:17:05.270 --> 00:17:08.609 align:middle line:84%
switch gears and really get into
some conceptual building blocks.

00:17:08.609 --> 00:17:09.359 align:middle line:90%
We had a question.

00:17:09.359 --> 00:17:13.869 align:middle line:90%


00:17:13.869 --> 00:17:16.430 align:middle line:84%
STUDENT: Can you define
a custom activation

00:17:16.430 --> 00:17:20.151 align:middle line:84%
function that is not in the
list of characters library?

00:17:20.151 --> 00:17:21.109 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:17:21.109 --> 00:17:21.369 align:middle line:90%
Yeah.

00:17:21.369 --> 00:17:22.490 align:middle line:84%
You can define--
the question was,

00:17:22.490 --> 00:17:24.323 align:middle line:84%
can you define a custom
activation function?

00:17:24.323 --> 00:17:25.650 align:middle line:90%
You totally can.

00:17:25.650 --> 00:17:28.490 align:middle line:84%
And in fact, I mean, the
kind of flexibility you have

00:17:28.490 --> 00:17:30.330 align:middle line:90%
here is incredible.

00:17:30.330 --> 00:17:33.010 align:middle line:84%
And these innocent four
lines, unfortunately,

00:17:33.010 --> 00:17:36.310 align:middle line:84%
hide the potential
that's possible here,

00:17:36.310 --> 00:17:38.490 align:middle line:84%
but I guarantee you,
in two to three weeks,

00:17:38.490 --> 00:17:41.410 align:middle line:84%
you folks will be thinking in
building blocks like LEGOs.

00:17:41.410 --> 00:17:44.422 align:middle line:90%
So I'm so happy when it happens.

00:17:44.422 --> 00:17:46.130 align:middle line:84%
Students come to my
office hours and say,

00:17:46.130 --> 00:17:48.850 align:middle line:84%
I want to create a network where
I have a little network going up

00:17:48.850 --> 00:17:51.470 align:middle line:84%
on top, one going in the bottom,
then they meet in the middle,

00:17:51.470 --> 00:17:52.845 align:middle line:84%
then they fork
again, they split.

00:17:52.845 --> 00:17:54.090 align:middle line:90%
I'm like, unbelievable.

00:17:54.090 --> 00:17:54.950 align:middle line:90%
It's fantastic.

00:17:54.950 --> 00:17:56.908 align:middle line:84%
And you're going to be
doing this in two weeks,

00:17:56.908 --> 00:17:58.513 align:middle line:90%
I guarantee you.

00:17:58.513 --> 00:18:00.930 align:middle line:84%
STUDENT: Yeah, in the case of
a multi-class classification

00:18:00.930 --> 00:18:03.710 align:middle line:84%
problem, are the output nodes
equal to the number of classes?

00:18:03.710 --> 00:18:05.330 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:18:05.330 --> 00:18:08.543 align:middle line:84%
So we will come to-- so this
is binary classification.

00:18:08.543 --> 00:18:10.710 align:middle line:84%
And the question is, for
multi-class classification,

00:18:10.710 --> 00:18:13.370 align:middle line:84%
let's say you're trying to
classify some input into one

00:18:13.370 --> 00:18:14.830 align:middle line:90%
of 10 possibilities.

00:18:14.830 --> 00:18:16.690 align:middle line:90%
We will have 10 outputs.

00:18:16.690 --> 00:18:19.490 align:middle line:84%
But the way we define it is
going to be using something

00:18:19.490 --> 00:18:24.070 align:middle line:84%
called a softmax function, which
we're going to cover on Monday.

00:18:24.070 --> 00:18:29.270 align:middle line:84%
So for now, we just live
with binary classification.

00:18:29.270 --> 00:18:32.230 align:middle line:84%
STUDENT: Is there a default
activation method in Keras

00:18:32.230 --> 00:18:33.870 align:middle line:90%
or you have to put something?

00:18:33.870 --> 00:18:35.620 align:middle line:84%
RAMA RAMAKRISHNAN:
That's a good question.

00:18:35.620 --> 00:18:37.930 align:middle line:84%
I believe the default might
be ReLUs for hidden layers,

00:18:37.930 --> 00:18:39.170 align:middle line:90%
but I'm not 100% sure.

00:18:39.170 --> 00:18:40.510 align:middle line:90%
Let's double-check that, yeah.

00:18:40.510 --> 00:18:43.147 align:middle line:90%


00:18:43.147 --> 00:18:45.230 align:middle line:84%
STUDENT: Just to get a
clearer understanding, when

00:18:45.230 --> 00:18:47.390 align:middle line:84%
you said that
beyond 16, when you

00:18:47.390 --> 00:18:52.140 align:middle line:84%
tried working on those neurons,
the performance worsened.

00:18:52.140 --> 00:18:54.390 align:middle line:84%
That is where you are playing
around with initially 2,

00:18:54.390 --> 00:18:56.170 align:middle line:90%
and then maybe 4 and 6 and 8?

00:18:56.170 --> 00:18:58.442 align:middle line:84%
RAMA RAMAKRISHNAN:
Exactly, right.

00:18:58.442 --> 00:18:59.650 align:middle line:90%
Could you use the microphone?

00:18:59.650 --> 00:19:02.140 align:middle line:90%
Yeah.

00:19:02.140 --> 00:19:04.390 align:middle line:84%
STUDENT: Do we need to define
each of the hidden layer

00:19:04.390 --> 00:19:06.110 align:middle line:84%
when the model gets
more complex, when

00:19:06.110 --> 00:19:07.768 align:middle line:90%
we have more than one layer?

00:19:07.768 --> 00:19:08.310 align:middle line:90%
Or is there--

00:19:08.310 --> 00:19:10.047 align:middle line:84%
RAMA RAMAKRISHNAN: If
you have 25 layers?

00:19:10.047 --> 00:19:10.630 align:middle line:90%
STUDENT: Yeah.

00:19:10.630 --> 00:19:11.130 align:middle line:84%
RAMA RAMAKRISHNAN:
Yeah, yeah, yeah.

00:19:11.130 --> 00:19:12.672 align:middle line:84%
So what we typically--
good question.

00:19:12.672 --> 00:19:14.790 align:middle line:84%
If you have, let's
say, 100 layers.

00:19:14.790 --> 00:19:17.690 align:middle line:84%
Do you actually have to type in
each by hand and cut and paste?

00:19:17.690 --> 00:19:18.190 align:middle line:90%
No.

00:19:18.190 --> 00:19:20.773 align:middle line:84%
You can actually write a little
loop, which just automatically

00:19:20.773 --> 00:19:22.810 align:middle line:90%
create them for you.

00:19:22.810 --> 00:19:25.570 align:middle line:84%
And so, basically what's going
on is that this little output

00:19:25.570 --> 00:19:27.650 align:middle line:84%
thing you see here,
this variable,

00:19:27.650 --> 00:19:32.213 align:middle line:84%
this output could be the result
of a 1,000-layer network with

00:19:32.213 --> 00:19:34.130 align:middle line:84%
all sorts of complicated
transformations going

00:19:34.130 --> 00:19:36.630 align:middle line:84%
on, and then finally, it pops
up as this little thing called

00:19:36.630 --> 00:19:37.490 align:middle line:90%
the output.

00:19:37.490 --> 00:19:39.390 align:middle line:84%
And what Keras will
do is it will be like,

00:19:39.390 --> 00:19:42.390 align:middle line:84%
OK, this model has this
input and has this output,

00:19:42.390 --> 00:19:44.890 align:middle line:84%
but boy, this output came from
incredible transformations

00:19:44.890 --> 00:19:46.448 align:middle line:84%
applied to the
input, and Keras will

00:19:46.448 --> 00:19:47.990 align:middle line:84%
process all that
very easily for you,

00:19:47.990 --> 00:19:50.070 align:middle line:84%
you don't have to
worry about it.

00:19:50.070 --> 00:19:53.330 align:middle line:84%
It's really a beautiful example
of the power of abstraction.

00:19:53.330 --> 00:19:55.210 align:middle line:84%
And you will see
that as we go along.

00:19:55.210 --> 00:19:58.510 align:middle line:84%
OK, so, now let's
switch gears and say,

00:19:58.510 --> 00:20:00.550 align:middle line:84%
once you've written a
model like that in Keras,

00:20:00.550 --> 00:20:01.890 align:middle line:90%
how do you actually train it?

00:20:01.890 --> 00:20:05.570 align:middle line:84%
Now, training is something
you've been doing a lot.

00:20:05.570 --> 00:20:08.290 align:middle line:84%
So for example, when you have
something like linear regression

00:20:08.290 --> 00:20:10.207 align:middle line:84%
where you have all these
coefficients you need

00:20:10.207 --> 00:20:11.870 align:middle line:84%
to estimate, you
have this model,

00:20:11.870 --> 00:20:13.370 align:middle line:90%
then you have a bunch of data.

00:20:13.370 --> 00:20:16.650 align:middle line:84%
Then you run it through
something like LLM if you use R.

00:20:16.650 --> 00:20:19.630 align:middle line:84%
And what it gives you is actual
values for the coefficients.

00:20:19.630 --> 00:20:21.630 align:middle line:84%
2.8, 2.9, and so
on and so forth.

00:20:21.630 --> 00:20:25.570 align:middle line:84%
So the role of the data is
to give you the coefficients.

00:20:25.570 --> 00:20:27.070 align:middle line:84%
Or you can think
of the coefficients

00:20:27.070 --> 00:20:30.350 align:middle line:84%
as really a compressed
version of the data.

00:20:30.350 --> 00:20:32.630 align:middle line:84%
Similarly, if you do
logistic regression,

00:20:32.630 --> 00:20:34.850 align:middle line:84%
you have a model like
that, you add some data,

00:20:34.850 --> 00:20:36.600 align:middle line:84%
you run it through
some estimation routine

00:20:36.600 --> 00:20:40.190 align:middle line:84%
like GNN or scikit-learn
or statsmodels,

00:20:40.190 --> 00:20:42.030 align:middle line:84%
pick your favorite tool,
then you'll come up

00:20:42.030 --> 00:20:43.110 align:middle line:90%
with something like that.

00:20:43.110 --> 00:20:44.750 align:middle line:84%
So basically what's
going on here

00:20:44.750 --> 00:20:48.790 align:middle line:84%
is training simply means find
the values of the coefficients

00:20:48.790 --> 00:20:50.750 align:middle line:84%
so that the model's
predictions are

00:20:50.750 --> 00:20:52.970 align:middle line:84%
as close to the actual
values as possible.

00:20:52.970 --> 00:20:55.160 align:middle line:90%
That's it.

00:20:55.160 --> 00:20:58.310 align:middle line:84%
And to find the one that is
as close to the actual value

00:20:58.310 --> 00:21:00.830 align:middle line:84%
as possible, a whole bunch
of optimization involved.

00:21:00.830 --> 00:21:02.830 align:middle line:84%
You didn't have to worry
about optimization when

00:21:02.830 --> 00:21:04.990 align:middle line:84%
you did regression,
linear or logistic,

00:21:04.990 --> 00:21:07.050 align:middle line:84%
because it's all done
under the hood for you,

00:21:07.050 --> 00:21:08.670 align:middle line:84%
but for neural
networks, we actually

00:21:08.670 --> 00:21:12.910 align:middle line:84%
get to know how it's done
because it's important.

00:21:12.910 --> 00:21:16.770 align:middle line:84%
OK, so training a neural
network, a deep neural network,

00:21:16.770 --> 00:21:19.450 align:middle line:84%
even GPT-4, it's
basically the same process

00:21:19.450 --> 00:21:21.610 align:middle line:90%
as what you do for regression.

00:21:21.610 --> 00:21:24.210 align:middle line:84%
You basically-- you're just
a very complicated function

00:21:24.210 --> 00:21:26.355 align:middle line:84%
with lots of parameters,
but ultimately, you

00:21:26.355 --> 00:21:28.230 align:middle line:84%
have a network with all
these question marks,

00:21:28.230 --> 00:21:30.350 align:middle line:84%
you add some data, you do
some training, and boom,

00:21:30.350 --> 00:21:31.225 align:middle line:90%
you get some numbers.

00:21:31.225 --> 00:21:36.290 align:middle line:90%


00:21:36.290 --> 00:21:38.330 align:middle line:84%
STUDENT: You may get
into this, but are we

00:21:38.330 --> 00:21:43.472 align:middle line:84%
determining the architecture of
the network before we train it?

00:21:43.472 --> 00:21:44.430 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:21:44.430 --> 00:21:47.130 align:middle line:84%
Because if you don't
define the architecture,

00:21:47.130 --> 00:21:49.730 align:middle line:84%
Keras doesn't know how
to actually calculate

00:21:49.730 --> 00:21:52.270 align:middle line:90%
the output given an input.

00:21:52.270 --> 00:21:55.130 align:middle line:84%
And unless it knows
input/output pairs,

00:21:55.130 --> 00:21:58.130 align:middle line:84%
it can't do anything
more with it.

00:21:58.130 --> 00:22:00.410 align:middle line:90%
OK.

00:22:00.410 --> 00:22:02.850 align:middle line:84%
So the essence of training
is to find the best values

00:22:02.850 --> 00:22:05.610 align:middle line:90%
for the weights and biases.

00:22:05.610 --> 00:22:07.610 align:middle line:84%
And the way we think
of the best values

00:22:07.610 --> 00:22:10.490 align:middle line:84%
is that we basically set
up a little function,

00:22:10.490 --> 00:22:14.170 align:middle line:84%
and this function measures the
discrepancy between the actual

00:22:14.170 --> 00:22:16.370 align:middle line:90%
and the predicted values.

00:22:16.370 --> 00:22:19.590 align:middle line:84%
And I use the word
discrepancy because the way

00:22:19.590 --> 00:22:21.710 align:middle line:84%
you define discrepancy,
there's incredible amounts

00:22:21.710 --> 00:22:23.270 align:middle line:90%
of creativity in the field.

00:22:23.270 --> 00:22:25.750 align:middle line:84%
In fact, a lot of
breakthroughs in deep learning

00:22:25.750 --> 00:22:28.150 align:middle line:84%
come because people define
a very clever measure

00:22:28.150 --> 00:22:30.037 align:middle line:84%
of discrepancy, and
then it turns out,

00:22:30.037 --> 00:22:32.370 align:middle line:84%
it actually gives you all
sorts of interesting behavior.

00:22:32.370 --> 00:22:34.590 align:middle line:84%
That's why I use the word
discrepancy as opposed

00:22:34.590 --> 00:22:36.690 align:middle line:84%
to the word error
because when I say error,

00:22:36.690 --> 00:22:38.315 align:middle line:84%
you might be just
thinking of something

00:22:38.315 --> 00:22:40.070 align:middle line:90%
like predicted minus actual.

00:22:40.070 --> 00:22:42.023 align:middle line:90%
That's still limiting.

00:22:42.023 --> 00:22:43.690 align:middle line:84%
Prediction minus
actual is too limiting,

00:22:43.690 --> 00:22:45.357 align:middle line:84%
that's why you use
the word discrepancy.

00:22:45.357 --> 00:22:48.110 align:middle line:84%
So we basically
define a function

00:22:48.110 --> 00:22:50.190 align:middle line:84%
that captures the discrepancy
between the actual

00:22:50.190 --> 00:22:52.550 align:middle line:84%
and the predicted values,
and these functions

00:22:52.550 --> 00:22:55.830 align:middle line:84%
are called loss functions
in the deep learning world.

00:22:55.830 --> 00:22:57.950 align:middle line:84%
And every paper
that you read, you

00:22:57.950 --> 00:23:00.170 align:middle line:84%
will find interesting
loss functions.

00:23:00.170 --> 00:23:01.890 align:middle line:84%
There are hundreds
of loss functions.

00:23:01.890 --> 00:23:04.350 align:middle line:84%
Enormous research creativity
goes into defining these loss

00:23:04.350 --> 00:23:05.910 align:middle line:90%
functions.

00:23:05.910 --> 00:23:06.550 align:middle line:90%
All right.

00:23:06.550 --> 00:23:08.590 align:middle line:90%
So, these are loss functions.

00:23:08.590 --> 00:23:10.470 align:middle line:84%
And so a loss function
is a function that

00:23:10.470 --> 00:23:12.110 align:middle line:90%
quantifies the discrepancy.

00:23:12.110 --> 00:23:13.930 align:middle line:84%
So let's say the
predictions are really

00:23:13.930 --> 00:23:15.690 align:middle line:90%
close to the actual values.

00:23:15.690 --> 00:23:19.042 align:middle line:90%
The loss would be what?

00:23:19.042 --> 00:23:20.440 align:middle line:90%
STUDENT: Close to 0.

00:23:20.440 --> 00:23:21.690 align:middle line:90%
RAMA RAMAKRISHNAN: Close to 0.

00:23:21.690 --> 00:23:23.370 align:middle line:90%
Very small.

00:23:23.370 --> 00:23:26.970 align:middle line:84%
And if you have a perfect
model, perfect crystal

00:23:26.970 --> 00:23:28.850 align:middle line:90%
ball, what would the loss be?

00:23:28.850 --> 00:23:30.090 align:middle line:90%
Exactly 0.

00:23:30.090 --> 00:23:31.710 align:middle line:90%
Exactly 0.

00:23:31.710 --> 00:23:34.890 align:middle line:84%
So, in linear
regression, the loss

00:23:34.890 --> 00:23:37.445 align:middle line:84%
function we use is called
sum of squared errors.

00:23:37.445 --> 00:23:39.570 align:middle line:84%
We didn't call it loss
function because we were not

00:23:39.570 --> 00:23:41.670 align:middle line:84%
doing deep learning,
just linear regression,

00:23:41.670 --> 00:23:43.810 align:middle line:84%
but that's basically
the loss function.

00:23:43.810 --> 00:23:48.450 align:middle line:84%
So the loss function we use must
be very matched very properly

00:23:48.450 --> 00:23:51.290 align:middle line:90%
with the kind of output we have.

00:23:51.290 --> 00:23:54.395 align:middle line:84%
So if your output
is a number like 23,

00:23:54.395 --> 00:23:55.770 align:middle line:84%
you're trying to
predict demand--

00:23:55.770 --> 00:23:59.330 align:middle line:84%
like a product demand for next
week for a particular product,

00:23:59.330 --> 00:24:01.630 align:middle line:90%
and the predicted value is 23.

00:24:01.630 --> 00:24:05.490 align:middle line:84%
the actual value is 21,
it's OK to do 23 minus 21,

00:24:05.490 --> 00:24:07.270 align:middle line:90%
too, as a discrepancy.

00:24:07.270 --> 00:24:08.650 align:middle line:90%
The error.

00:24:08.650 --> 00:24:12.320 align:middle line:84%
But for other kinds of
outputs, it's not so obvious

00:24:12.320 --> 00:24:14.060 align:middle line:84%
what the correct
loss function is,

00:24:14.060 --> 00:24:16.160 align:middle line:84%
what the correct measure
of discrepancy is.

00:24:16.160 --> 00:24:22.680 align:middle line:84%
And so here, for the simple
case of regression, the yi--

00:24:22.680 --> 00:24:25.040 align:middle line:84%
the i here, by the way,
is a superscript which

00:24:25.040 --> 00:24:27.360 align:middle line:90%
stands for the i-th data point.

00:24:27.360 --> 00:24:28.700 align:middle line:90%
The i-th data point.

00:24:28.700 --> 00:24:31.520 align:middle line:84%
So what I'm saying is that
for the i-th data point,

00:24:31.520 --> 00:24:33.880 align:middle line:84%
this is the actual
value y, and this

00:24:33.880 --> 00:24:36.760 align:middle line:90%
is what the model predicted.

00:24:36.760 --> 00:24:39.100 align:middle line:84%
I take the
difference, square it,

00:24:39.100 --> 00:24:41.440 align:middle line:84%
and I square it for
each point I just

00:24:41.440 --> 00:24:43.840 align:middle line:84%
average all these
numbers to get an average

00:24:43.840 --> 00:24:47.120 align:middle line:84%
squared error-- i.e.,
Mean Squared Error, MSE.

00:24:47.120 --> 00:24:52.280 align:middle line:84%
So this is the
easiest loss function

00:24:52.280 --> 00:24:55.040 align:middle line:90%
Now, let's crank it up a notch.

00:24:55.040 --> 00:24:57.800 align:middle line:84%
In the heart disease
example, the heart disease--

00:24:57.800 --> 00:25:00.520 align:middle line:84%
the neural prediction
model, the prediction

00:25:00.520 --> 00:25:03.240 align:middle line:84%
is a number between
0 and 1 because it's

00:25:03.240 --> 00:25:04.760 align:middle line:90%
coming out of the sigmoid.

00:25:04.760 --> 00:25:06.000 align:middle line:90%
It's a fraction.

00:25:06.000 --> 00:25:09.020 align:middle line:90%
The actual output is a 0 or 1.

00:25:09.020 --> 00:25:11.180 align:middle line:90%
One of the two, it's binary.

00:25:11.180 --> 00:25:13.560 align:middle line:84%
So how would we compare
the discrepancy?

00:25:13.560 --> 00:25:16.420 align:middle line:84%
How would we measure the
discrepancy between a fraction

00:25:16.420 --> 00:25:18.180 align:middle line:90%
and the numbers 0 and 1?

00:25:18.180 --> 00:25:21.100 align:middle line:84%
What is a good loss
function in this situation

00:25:21.100 --> 00:25:22.640 align:middle line:90%
is the key question.

00:25:22.640 --> 00:25:26.060 align:middle line:84%
So let's build some
intuition around this.

00:25:26.060 --> 00:25:30.852 align:middle line:84%
And let's see if my little
daisy chain iPad thing works.

00:25:30.852 --> 00:25:33.060 align:middle line:84%
I'm doing it on iPad so that
people on the livestream

00:25:33.060 --> 00:25:34.620 align:middle line:84%
can see it, otherwise
the blackboard

00:25:34.620 --> 00:25:37.100 align:middle line:90%
is a little tough for them.

00:25:37.100 --> 00:25:41.106 align:middle line:84%
OK, so let's have
a situation here--

00:25:41.106 --> 00:25:41.606 align:middle line:90%
OK.

00:25:41.606 --> 00:25:44.740 align:middle line:84%
So let's say that you have
a patient who comes in,

00:25:44.740 --> 00:25:47.000 align:middle line:84%
and let's say they
have heart disease.

00:25:47.000 --> 00:25:50.520 align:middle line:90%
So for that patient, y equals 1.

00:25:50.520 --> 00:25:53.020 align:middle line:84%
The true value is
1 for that patient.

00:25:53.020 --> 00:25:56.560 align:middle line:90%
And now, you have this model.

00:25:56.560 --> 00:26:01.410 align:middle line:84%
And this is the predicted
probability from this model.

00:26:01.410 --> 00:26:04.540 align:middle line:90%


00:26:04.540 --> 00:26:07.560 align:middle line:84%
Can people see my
handwriting OK?

00:26:07.560 --> 00:26:08.320 align:middle line:90%
Good.

00:26:08.320 --> 00:26:11.320 align:middle line:84%
I could have never
been a doctor, right?

00:26:11.320 --> 00:26:13.100 align:middle line:90%
So 0, 1.

00:26:13.100 --> 00:26:15.480 align:middle line:84%
It's going to be between 0
and 1 because of probability.

00:26:15.480 --> 00:26:17.900 align:middle line:84%
And then this is the
loss we want to have.

00:26:17.900 --> 00:26:19.160 align:middle line:90%
This is the loss.

00:26:19.160 --> 00:26:23.403 align:middle line:84%
So, this patient actually had
heart disease, y equals 1.

00:26:23.403 --> 00:26:25.320 align:middle line:84%
So let's say that the
predicted probability is

00:26:25.320 --> 00:26:27.440 align:middle line:90%
pretty close to 1.

00:26:27.440 --> 00:26:29.960 align:middle line:84%
What do you think
the loss should be?

00:26:29.960 --> 00:26:31.320 align:middle line:90%
STUDENT: Small.

00:26:31.320 --> 00:26:32.920 align:middle line:90%
STUDENT: Small.

00:26:32.920 --> 00:26:34.240 align:middle line:90%
RAMA RAMAKRISHNAN: Sorry?

00:26:34.240 --> 00:26:35.160 align:middle line:90%
Close to 0, exactly.

00:26:35.160 --> 00:26:37.700 align:middle line:84%
So here, if the
prediction comes here,

00:26:37.700 --> 00:26:39.400 align:middle line:84%
you want the loss
to be-- you want

00:26:39.400 --> 00:26:42.320 align:middle line:90%
the loss to be somewhere here.

00:26:42.320 --> 00:26:44.747 align:middle line:84%
But if the predicted probability
is pretty close to 0,

00:26:44.747 --> 00:26:47.080 align:middle line:84%
even though the patient
actually has heart disease, what

00:26:47.080 --> 00:26:49.480 align:middle line:90%
do you want the loss to be?

00:26:49.480 --> 00:26:52.640 align:middle line:84%
Really high because
it's screwing up badly.

00:26:52.640 --> 00:26:55.360 align:middle line:84%
So you want the loss
to be somewhere here.

00:26:55.360 --> 00:26:57.200 align:middle line:84%
So basically, you
want a function

00:26:57.200 --> 00:26:58.310 align:middle line:90%
that's kind of like that.

00:26:58.310 --> 00:27:00.840 align:middle line:90%


00:27:00.840 --> 00:27:04.165 align:middle line:84%
You want the loss function
shape to be like that.

00:27:04.165 --> 00:27:05.540 align:middle line:84%
High values of
probability should

00:27:05.540 --> 00:27:06.880 align:middle line:90%
have low losses, low values.

00:27:06.880 --> 00:27:08.380 align:middle line:84%
Probability should
have high losses.

00:27:08.380 --> 00:27:08.980 align:middle line:90%
Yeah?

00:27:08.980 --> 00:27:11.810 align:middle line:84%
STUDENT: I understand why it has
to be increasing or decreasing,

00:27:11.810 --> 00:27:13.268 align:middle line:84%
but then your
statement has to be--

00:27:13.268 --> 00:27:14.518 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah, yeah.

00:27:14.518 --> 00:27:15.600 align:middle line:90%
So it can be linear.

00:27:15.600 --> 00:27:17.160 align:middle line:90%
It can certainly be linear.

00:27:17.160 --> 00:27:18.540 align:middle line:84%
But basically what
you want to do

00:27:18.540 --> 00:27:22.540 align:middle line:84%
is, the more it makes a
mistake, the more harshly you

00:27:22.540 --> 00:27:24.560 align:middle line:90%
want to penalize it.

00:27:24.560 --> 00:27:26.740 align:middle line:84%
So basically what
you really want

00:27:26.740 --> 00:27:28.340 align:middle line:84%
is something where
if it basically

00:27:28.340 --> 00:27:29.960 align:middle line:84%
says this person's
probability is,

00:27:29.960 --> 00:27:32.780 align:middle line:84%
say, the probability--
the predicted probability,

00:27:32.780 --> 00:27:35.640 align:middle line:84%
say, 1 over a million,
basically to 0,

00:27:35.640 --> 00:27:38.860 align:middle line:84%
you want the loss to be super
high so that the model is

00:27:38.860 --> 00:27:41.320 align:middle line:84%
like-- it's like a huge rap
on the knuckles for the model.

00:27:41.320 --> 00:27:42.383 align:middle line:90%
Don't do that!

00:27:42.383 --> 00:27:43.800 align:middle line:84%
That's basically
what we're doing.

00:27:43.800 --> 00:27:45.540 align:middle line:84%
And I'm demonstrating
that dynamic

00:27:45.540 --> 00:27:49.565 align:middle line:84%
by using a very curved
and steep loss function.

00:27:49.565 --> 00:27:51.440 align:middle line:84%
But you can absolutely
use a linear function.

00:27:51.440 --> 00:27:52.340 align:middle line:90%
It's totally fine.

00:27:52.340 --> 00:27:54.540 align:middle line:84%
It won't be as effective
for gradient descent

00:27:54.540 --> 00:27:57.820 align:middle line:84%
later on, but a bunch
of technical details.

00:27:57.820 --> 00:27:59.380 align:middle line:90%
Are we good with this?

00:27:59.380 --> 00:28:00.060 align:middle line:90%
All right.

00:28:00.060 --> 00:28:03.720 align:middle line:84%
So, now let's look at the
case where a patient does not

00:28:03.720 --> 00:28:06.840 align:middle line:90%
have heart disease, y equals 0.

00:28:06.840 --> 00:28:09.080 align:middle line:90%
Same setup.

00:28:09.080 --> 00:28:15.320 align:middle line:84%
Predicted probability,
0, 1, loss.

00:28:15.320 --> 00:28:20.640 align:middle line:84%
So for this patient, they
don't have whatever--

00:28:20.640 --> 00:28:22.000 align:middle line:90%
they don't have heart disease.

00:28:22.000 --> 00:28:26.280 align:middle line:84%
If the probability is close
to 0, what should the loss be?

00:28:26.280 --> 00:28:26.860 align:middle line:90%
Close to 0.

00:28:26.860 --> 00:28:28.027 align:middle line:90%
It should be somewhere here.

00:28:28.027 --> 00:28:30.978 align:middle line:84%
And the more and
more the probability

00:28:30.978 --> 00:28:32.520 align:middle line:84%
gets closer and
closer to 1, you want

00:28:32.520 --> 00:28:34.160 align:middle line:84%
to penalize it
very heavily, which

00:28:34.160 --> 00:28:36.120 align:middle line:84%
means you want the loss
to be somewhere here.

00:28:36.120 --> 00:28:37.703 align:middle line:84%
So you basically
want a loss, ideally,

00:28:37.703 --> 00:28:42.200 align:middle line:84%
that's going up like that and
climbing higher and higher.

00:28:42.200 --> 00:28:43.240 align:middle line:90%
Are we good?

00:28:43.240 --> 00:28:44.640 align:middle line:90%
OK, perfect.

00:28:44.640 --> 00:28:49.000 align:middle line:84%
Because we have a perfect
loss function for that.

00:28:49.000 --> 00:28:53.080 align:middle line:84%
So, just to recap,
this is what we want.

00:28:53.080 --> 00:28:56.260 align:middle line:84%
People with for points with
y equals 1, lower prediction,

00:28:56.260 --> 00:28:57.880 align:middle line:84%
predictions should
have higher loss.

00:28:57.880 --> 00:28:59.260 align:middle line:90%
We want something like that.

00:28:59.260 --> 00:29:04.740 align:middle line:84%
And then turns out, there's a
very little simple loss function

00:29:04.740 --> 00:29:06.873 align:middle line:84%
which just literally
just uses the logarithm,

00:29:06.873 --> 00:29:08.040 align:middle line:90%
which will get the job done.

00:29:08.040 --> 00:29:11.220 align:middle line:84%
So what you do is you
literally do minus log

00:29:11.220 --> 00:29:13.180 align:middle line:90%
of that predicted probability.

00:29:13.180 --> 00:29:14.060 align:middle line:90%
That's it.

00:29:14.060 --> 00:29:17.120 align:middle line:84%
And that thing has
exactly that shape.

00:29:17.120 --> 00:29:18.960 align:middle line:84%
And in fact, you can
see it numerically.

00:29:18.960 --> 00:29:20.680 align:middle line:90%
So if the loss is 1, it's 0.

00:29:20.680 --> 00:29:22.760 align:middle line:90%
If it's half, it's 1.0.

00:29:22.760 --> 00:29:25.340 align:middle line:84%
And if it's 1 over
1,000, it's almost 10.

00:29:25.340 --> 00:29:28.480 align:middle line:84%
If it's 1 over 10,000, it's
going to be much higher,

00:29:28.480 --> 00:29:30.760 align:middle line:90%
very high losses.

00:29:30.760 --> 00:29:34.540 align:middle line:84%
So minus log
probability, boom, done.

00:29:34.540 --> 00:29:37.140 align:middle line:84%
Similarly, this is what
we want for patients

00:29:37.140 --> 00:29:38.560 align:middle line:90%
for whom y equals 0.

00:29:38.560 --> 00:29:42.780 align:middle line:84%
And it turns out, if
you do minus log 1 minus

00:29:42.780 --> 00:29:45.240 align:middle line:84%
predicted probability,
it does the same thing.

00:29:45.240 --> 00:29:47.800 align:middle line:90%


00:29:47.800 --> 00:29:48.300 align:middle line:90%
OK.

00:29:48.300 --> 00:29:50.840 align:middle line:90%


00:29:50.840 --> 00:29:54.700 align:middle line:84%
Mathematicians, once again,
saved by the logarithm.

00:29:54.700 --> 00:29:58.892 align:middle line:84%
So in summary, this
is what we have.

00:29:58.892 --> 00:30:01.600 align:middle line:84%
For data points with y equals 1,
we have this; data points with y

00:30:01.600 --> 00:30:02.980 align:middle line:90%
equals 0, we have this.

00:30:02.980 --> 00:30:07.240 align:middle line:84%
But it feels a little inelegant
to say, well if it's y equals 1,

00:30:07.240 --> 00:30:09.560 align:middle line:84%
I want to use this; if y
equals 0, I want to use that.

00:30:09.560 --> 00:30:12.205 align:middle line:84%
There's an if-then
thing going on here,

00:30:12.205 --> 00:30:13.580 align:middle line:84%
and I don't know
about you folks,

00:30:13.580 --> 00:30:16.440 align:middle line:84%
but if-then really
irks me mathematically

00:30:16.440 --> 00:30:20.280 align:middle line:84%
because you can't do derivatives
and so on very easily.

00:30:20.280 --> 00:30:21.600 align:middle line:90%
But no worries.

00:30:21.600 --> 00:30:22.520 align:middle line:90%
This is MIT.

00:30:22.520 --> 00:30:26.680 align:middle line:84%
We know-- we have a bag of
math tricks, so what we do is,

00:30:26.680 --> 00:30:30.880 align:middle line:84%
we can actually combine them
both into a single expression

00:30:30.880 --> 00:30:32.600 align:middle line:90%
like this.

00:30:32.600 --> 00:30:35.740 align:middle line:84%
And here, the y i, again,
is the i-th data point.

00:30:35.740 --> 00:30:38.360 align:middle line:84%
Remember, y i is
either 1 or 0 always.

00:30:38.360 --> 00:30:42.160 align:middle line:84%
And this model of x i is
the predicted probability.

00:30:42.160 --> 00:30:46.060 align:middle line:84%
And I've just taken the
minus log, the minus,

00:30:46.060 --> 00:30:48.860 align:middle line:90%
and I've just moved it here.

00:30:48.860 --> 00:30:52.740 align:middle line:84%
And I've taken the minus that
was here and just moved it here.

00:30:52.740 --> 00:30:54.600 align:middle line:90%
That's why you see it like this.

00:30:54.600 --> 00:30:58.420 align:middle line:84%
So this one is basically,
you can convince yourself

00:30:58.420 --> 00:30:59.240 align:middle line:90%
what happens.

00:30:59.240 --> 00:31:01.073 align:middle line:84%
The single expression
will get the job done.

00:31:01.073 --> 00:31:04.020 align:middle line:84%
So let's say there is a
patient for whom y equals 1.

00:31:04.020 --> 00:31:07.080 align:middle line:84%
What's going to happen is that
when you plug in y equals 1,

00:31:07.080 --> 00:31:10.540 align:middle line:84%
this becomes 0, the whole
thing will collapse to 0.

00:31:10.540 --> 00:31:12.460 align:middle line:84%
While here, y
equals 1 just means

00:31:12.460 --> 00:31:14.260 align:middle line:84%
it becomes minus log
probability, which

00:31:14.260 --> 00:31:15.120 align:middle line:90%
is what we want.

00:31:15.120 --> 00:31:17.660 align:middle line:90%


00:31:17.660 --> 00:31:20.340 align:middle line:84%
Conversely, if y equals
0, this whole thing

00:31:20.340 --> 00:31:22.060 align:middle line:90%
is going to disappear.

00:31:22.060 --> 00:31:24.680 align:middle line:84%
And this thing becomes 1
minus 0, which is just 1,

00:31:24.680 --> 00:31:27.180 align:middle line:84%
and so it becomes minus log
1 minus probability, which

00:31:27.180 --> 00:31:29.700 align:middle line:90%
is, again, what we want.

00:31:29.700 --> 00:31:31.980 align:middle line:90%
Simple and neat, right?

00:31:31.980 --> 00:31:35.880 align:middle line:84%
So in one expression, we have
defined the perfect loss.

00:31:35.880 --> 00:31:39.500 align:middle line:90%
No if-thens, none of that crap.

00:31:39.500 --> 00:31:40.220 align:middle line:90%
Good.

00:31:40.220 --> 00:31:44.020 align:middle line:84%
So now what we do is, that
was true for every data point,

00:31:44.020 --> 00:31:45.800 align:middle line:84%
but we obviously have
lots of data points,

00:31:45.800 --> 00:31:50.460 align:middle line:84%
so we just add them all
up and take the average.

00:31:50.460 --> 00:31:50.960 align:middle line:90%
That's it.

00:31:50.960 --> 00:31:52.877 align:middle line:84%
We average across all
the data points we have.

00:31:52.877 --> 00:31:55.520 align:middle line:90%
So that we get an average loss.

00:31:55.520 --> 00:31:57.352 align:middle line:90%
OK.

00:31:57.352 --> 00:31:59.310 align:middle line:84%
This is the binary cross
entropy loss function.

00:31:59.310 --> 00:32:06.680 align:middle line:90%


00:32:06.680 --> 00:32:09.620 align:middle line:84%
STUDENT: Is there a way you
can edit the loss function

00:32:09.620 --> 00:32:12.602 align:middle line:84%
so that you penalize false
negatives more strongly than--

00:32:12.602 --> 00:32:13.560 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:32:13.560 --> 00:32:14.518 align:middle line:90%
You can do all of them.

00:32:14.518 --> 00:32:15.680 align:middle line:90%
Great question.

00:32:15.680 --> 00:32:20.160 align:middle line:84%
I'm just looking at the basic
case where it's symmetric loss.

00:32:20.160 --> 00:32:23.040 align:middle line:84%
You can actually penalize
overestimates much more

00:32:23.040 --> 00:32:25.382 align:middle line:84%
than underestimates
and things like that.

00:32:25.382 --> 00:32:26.840 align:middle line:84%
And if you're
curious, you can just

00:32:26.840 --> 00:32:28.570 align:middle line:84%
Google something called
the pinball loss.

00:32:28.570 --> 00:32:31.480 align:middle line:90%


00:32:31.480 --> 00:32:32.700 align:middle line:90%
OK.

00:32:32.700 --> 00:32:34.440 align:middle line:90%
Any other questions on this?

00:32:34.440 --> 00:32:37.240 align:middle line:84%
So when you see this massive
deep neural network built

00:32:37.240 --> 00:32:39.340 align:middle line:84%
by Google for doing
something or the other,

00:32:39.340 --> 00:32:41.380 align:middle line:84%
if it's a binary
classification problem,

00:32:41.380 --> 00:32:45.080 align:middle line:84%
chances are they're
using this thing.

00:32:45.080 --> 00:32:45.960 align:middle line:90%
All right.

00:32:45.960 --> 00:32:49.380 align:middle line:84%
So, now let's figure out how to
minimize these loss functions

00:32:49.380 --> 00:32:51.880 align:middle line:84%
because the name of the game
is to find a way to minimize

00:32:51.880 --> 00:32:53.160 align:middle line:90%
these loss functions.

00:32:53.160 --> 00:32:56.820 align:middle line:84%
So now loss functions are just
a particular kind of function.

00:32:56.820 --> 00:33:00.020 align:middle line:84%
So we'll first consider the
general problem of minimizing

00:33:00.020 --> 00:33:02.060 align:middle line:90%
some arbitrary function.

00:33:02.060 --> 00:33:04.580 align:middle line:84%
And once we develop a little
bit of intuition about that,

00:33:04.580 --> 00:33:06.380 align:middle line:84%
we'll return to
the specific task

00:33:06.380 --> 00:33:08.240 align:middle line:90%
of minimizing loss functions.

00:33:08.240 --> 00:33:12.180 align:middle line:90%


00:33:12.180 --> 00:33:15.260 align:middle line:90%
How's everyone doing?

00:33:15.260 --> 00:33:18.860 align:middle line:90%
Yes, no, good, bad?

00:33:18.860 --> 00:33:23.220 align:middle line:84%
You have a bit of a
tough-to-interpret head shake.

00:33:23.220 --> 00:33:25.038 align:middle line:84%
STUDENT: It's more
like I kind of lost you

00:33:25.038 --> 00:33:26.580 align:middle line:84%
where you said that
the loss function

00:33:26.580 --> 00:33:29.132 align:middle line:84%
and the predicted
probability, how were they

00:33:29.132 --> 00:33:31.340 align:middle line:84%
inversely because the
understanding was that the loss

00:33:31.340 --> 00:33:33.820 align:middle line:84%
function is supposed to
be the sum of errors,

00:33:33.820 --> 00:33:35.000 align:middle line:90%
we are averaging the errors.

00:33:35.000 --> 00:33:35.660 align:middle line:90%
And when you said that--

00:33:35.660 --> 00:33:36.100 align:middle line:90%
RAMA RAMAKRISHNAN: No, no, no.

00:33:36.100 --> 00:33:36.400 align:middle line:90%
No, no.

00:33:36.400 --> 00:33:36.942 align:middle line:90%
Sorry, sorry.

00:33:36.942 --> 00:33:38.900 align:middle line:90%
Let us stop there for a second.

00:33:38.900 --> 00:33:41.142 align:middle line:84%
For each point, you
define the loss.

00:33:41.142 --> 00:33:42.600 align:middle line:84%
That's the whole
point of the game.

00:33:42.600 --> 00:33:44.980 align:middle line:84%
And once you define it, you
calculate it for every point

00:33:44.980 --> 00:33:45.760 align:middle line:90%
and average it.

00:33:45.760 --> 00:33:49.460 align:middle line:84%
So just focus on a
single data point.

00:33:49.460 --> 00:33:50.980 align:middle line:90%
And so now continue.

00:33:50.980 --> 00:33:53.020 align:middle line:84%
STUDENT: So now when
the heart patient has,

00:33:53.020 --> 00:33:56.140 align:middle line:84%
there is more probability
that they know.

00:33:56.140 --> 00:33:59.440 align:middle line:84%
So when there is a person
who has the heart disease,

00:33:59.440 --> 00:34:02.600 align:middle line:84%
you said that you want the
loss function to be high.

00:34:02.600 --> 00:34:04.100 align:middle line:84%
I think I'm going
back to the graph.

00:34:04.100 --> 00:34:05.975 align:middle line:84%
RAMA RAMAKRISHNAN: You
want the loss function

00:34:05.975 --> 00:34:08.040 align:middle line:84%
to be high if I'm predicting
that they basically

00:34:08.040 --> 00:34:09.920 align:middle line:90%
don't have heart disease.

00:34:09.920 --> 00:34:12.840 align:middle line:84%
If the prediction is close to
0, the predicted probability

00:34:12.840 --> 00:34:18.040 align:middle line:84%
is close to 0, then I'm badly
wrong because in reality, they

00:34:18.040 --> 00:34:19.958 align:middle line:90%
do have heart disease.

00:34:19.958 --> 00:34:22.000 align:middle line:84%
And that's why I want the
loss to be really high.

00:34:22.000 --> 00:34:23.375 align:middle line:84%
STUDENT: OK, so
effectively, loss

00:34:23.375 --> 00:34:25.594 align:middle line:84%
is my way of finding out
how good my model is.

00:34:25.594 --> 00:34:26.719 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:34:26.719 --> 00:34:28.440 align:middle line:90%
STUDENT: Instead of saying, OK--

00:34:28.440 --> 00:34:30.656 align:middle line:84%
RAMA RAMAKRISHNAN: Or rather,
how bad your model is.

00:34:30.656 --> 00:34:31.239 align:middle line:90%
STUDENT: Yeah.

00:34:31.239 --> 00:34:31.739 align:middle line:90%
RAMA RAMAKRISHNAN: Right?

00:34:31.739 --> 00:34:32.552 align:middle line:90%
How bad is it?

00:34:32.552 --> 00:34:34.219 align:middle line:84%
That's really what
the loss function is.

00:34:34.219 --> 00:34:34.886 align:middle line:90%
STUDENT: Got it.

00:34:34.886 --> 00:34:37.360 align:middle line:84%
RAMA RAMAKRISHNAN: And you
want to minimize badness.

00:34:37.360 --> 00:34:40.120 align:middle line:84%
That's the whole
point of optimization.

00:34:40.120 --> 00:34:41.588 align:middle line:90%
STUDENT: OK.

00:34:41.588 --> 00:34:43.880 align:middle line:84%
RAMA RAMAKRISHNAN: I guess
I don't have a fully-- like,

00:34:43.880 --> 00:34:45.880 align:middle line:84%
similar to the point
raised before, I don't have

00:34:45.880 --> 00:34:48.440 align:middle line:84%
a fully clear intuition of
why exactly a log function

00:34:48.440 --> 00:34:52.659 align:middle line:84%
rather than something that,
say, flatter for small and then

00:34:52.659 --> 00:34:53.556 align:middle line:90%
really steep later.

00:34:53.556 --> 00:34:55.639 align:middle line:84%
RAMA RAMAKRISHNAN: Those
are all fantastic things,

00:34:55.639 --> 00:34:57.020 align:middle line:90%
you can totally do it.

00:34:57.020 --> 00:35:00.540 align:middle line:84%
The reason we pick the
loss-- this function because,

00:35:00.540 --> 00:35:03.440 align:middle line:84%
A, is easy to work with,
it has good gradients,

00:35:03.440 --> 00:35:05.220 align:middle line:84%
it's well-behaved
mathematically,

00:35:05.220 --> 00:35:07.160 align:middle line:84%
but there are many
alternatives to it.

00:35:07.160 --> 00:35:10.100 align:middle line:84%
I don't want you to think that
this is the only game in town

00:35:10.100 --> 00:35:11.720 align:middle line:90%
or it's the only choice for us.

00:35:11.720 --> 00:35:13.000 align:middle line:90%
We have many choices.

00:35:13.000 --> 00:35:15.980 align:middle line:84%
This is really-- this happens
to be a very easy choice, which

00:35:15.980 --> 00:35:18.900 align:middle line:84%
also happens to be
empirically very effective.

00:35:18.900 --> 00:35:20.380 align:middle line:84%
And I'm happy to
give you pointers

00:35:20.380 --> 00:35:23.300 align:middle line:84%
to other crazy loss
functions which can actually

00:35:23.300 --> 00:35:24.400 align:middle line:90%
do all these things, too.

00:35:24.400 --> 00:35:26.720 align:middle line:90%


00:35:26.720 --> 00:35:27.220 align:middle line:90%
OK.

00:35:27.220 --> 00:35:30.340 align:middle line:90%


00:35:30.340 --> 00:35:33.280 align:middle line:84%
All right, so, minimizing
a single variable function,

00:35:33.280 --> 00:35:37.040 align:middle line:84%
we will warm up by looking
at this little function here,

00:35:37.040 --> 00:35:38.277 align:middle line:90%
which is a--

00:35:38.277 --> 00:35:39.610 align:middle line:90%
what do you call a fourth power?

00:35:39.610 --> 00:35:42.180 align:middle line:90%


00:35:42.180 --> 00:35:43.093 align:middle line:90%
Quartic, right?

00:35:43.093 --> 00:35:43.760 align:middle line:90%
Yeah, Thank you.

00:35:43.760 --> 00:35:44.540 align:middle line:90%
Quartic.

00:35:44.540 --> 00:35:47.640 align:middle line:84%
So yeah, it's a
quartic function.

00:35:47.640 --> 00:35:50.000 align:middle line:90%
And this is how it looks like.

00:35:50.000 --> 00:35:52.480 align:middle line:84%
And you can see, there is
a minimum somewhere here,

00:35:52.480 --> 00:35:54.560 align:middle line:90%
between minus 1 and minus 2.

00:35:54.560 --> 00:35:56.000 align:middle line:90%
Maybe minus 1.5.

00:35:56.000 --> 00:35:58.560 align:middle line:84%
So we want to minimize
this function.

00:35:58.560 --> 00:36:00.760 align:middle line:84%
It's obviously a toy
function, a little function

00:36:00.760 --> 00:36:03.400 align:middle line:84%
with one variable, but
the intuition we use here

00:36:03.400 --> 00:36:08.240 align:middle line:84%
is going to be exactly what we
use for GPT-4, so pay attention.

00:36:08.240 --> 00:36:10.960 align:middle line:84%
So, how can we go about
minimizing this function?

00:36:10.960 --> 00:36:11.820 align:middle line:90%
What will we do?

00:36:11.820 --> 00:36:15.080 align:middle line:90%


00:36:15.080 --> 00:36:16.755 align:middle line:90%
Yeah?

00:36:16.755 --> 00:36:18.880 align:middle line:84%
STUDENT: Take the derivative
and set it equal to 0?

00:36:18.880 --> 00:36:19.860 align:middle line:84%
RAMA RAMAKRISHNAN: You
take the derivative.

00:36:19.860 --> 00:36:20.360 align:middle line:90%
Exactly.

00:36:20.360 --> 00:36:22.320 align:middle line:90%
So you take the derivative.

00:36:22.320 --> 00:36:26.640 align:middle line:84%
So let's look at what the
derivative does for us, but then

00:36:26.640 --> 00:36:29.900 align:middle line:90%
the second part of what I said--

00:36:29.900 --> 00:36:30.400 align:middle line:90%
yeah.

00:36:30.400 --> 00:36:33.760 align:middle line:84%
Second part of what I said was
set it to 0, setting it to 0

00:36:33.760 --> 00:36:37.860 align:middle line:84%
becomes problematic when you
have very complicated functions.

00:36:37.860 --> 00:36:40.140 align:middle line:84%
It's not clear at all
what's going to make them 0,

00:36:40.140 --> 00:36:41.000 align:middle line:90%
unfortunately.

00:36:41.000 --> 00:36:43.880 align:middle line:84%
But the idea of derivative
is, in fact, the right idea.

00:36:43.880 --> 00:36:46.123 align:middle line:84%
So we can go about this, we
can calculate derivative,

00:36:46.123 --> 00:36:48.040 align:middle line:84%
and that actually happens
with the derivative,

00:36:48.040 --> 00:36:49.460 align:middle line:90%
you can convince yourself.

00:36:49.460 --> 00:36:53.460 align:middle line:84%
And if you plot the
derivative, it looks like that.

00:36:53.460 --> 00:36:56.240 align:middle line:84%
And, as you would hope, wherever
the minimum is, in fact,

00:36:56.240 --> 00:36:58.700 align:middle line:90%
the derivative is crossing--

00:36:58.700 --> 00:37:01.300 align:middle line:84%
the derivative at 0 here
is crossing the x-axis,

00:37:01.300 --> 00:37:03.740 align:middle line:84%
in this case we can
actually do that.

00:37:03.740 --> 00:37:06.580 align:middle line:84%
So let's say you have the
derivative, how can you use it?

00:37:06.580 --> 00:37:08.380 align:middle line:84%
What is the value
of a derivative?

00:37:08.380 --> 00:37:09.780 align:middle line:90%
What does it tell you?

00:37:09.780 --> 00:37:11.300 align:middle line:90%
Yeah?

00:37:11.300 --> 00:37:13.682 align:middle line:84%
STUDENT: You use a
gradient descent algorithm?

00:37:13.682 --> 00:37:16.140 align:middle line:84%
RAMA RAMAKRISHNAN: You are 10
steps ahead of me, my friend.

00:37:16.140 --> 00:37:18.300 align:middle line:90%
I just want a basic answer.

00:37:18.300 --> 00:37:20.233 align:middle line:90%
Like, what good is a derivative?

00:37:20.233 --> 00:37:21.400 align:middle line:90%
Like, what does it tell you?

00:37:21.400 --> 00:37:23.020 align:middle line:84%
When you calculate the
derivative of something

00:37:23.020 --> 00:37:23.890 align:middle line:90%
at a particular point--

00:37:23.890 --> 00:37:25.980 align:middle line:84%
STUDENT: It tells you the
rate of change of the function

00:37:25.980 --> 00:37:26.720 align:middle line:90%
at the place you are.

00:37:26.720 --> 00:37:28.470 align:middle line:84%
RAMA RAMAKRISHNAN:
Correct, exactly right.

00:37:28.470 --> 00:37:30.260 align:middle line:84%
So here, what the
derivative tells

00:37:30.260 --> 00:37:33.780 align:middle line:84%
us is that the slope tells
us a change in the function

00:37:33.780 --> 00:37:36.060 align:middle line:90%
for a very small increase in w.

00:37:36.060 --> 00:37:38.420 align:middle line:84%
And this just high
school calculus, I'm

00:37:38.420 --> 00:37:41.940 align:middle line:90%
just doing a quick refresher.

00:37:41.940 --> 00:37:47.680 align:middle line:84%
So, what that means is that
if the derivative is positive,

00:37:47.680 --> 00:37:50.200 align:middle line:84%
what that means is that
increasing w slightly

00:37:50.200 --> 00:37:52.080 align:middle line:90%
will increase the function.

00:37:52.080 --> 00:37:55.035 align:middle line:84%
So if you're here, you
calculate the slope is positive,

00:37:55.035 --> 00:37:57.160 align:middle line:84%
it means that if you go
slightly in this direction,

00:37:57.160 --> 00:38:00.200 align:middle line:84%
the function is
going to get higher.

00:38:00.200 --> 00:38:02.720 align:middle line:84%
Similarly, if it's
negative, let's

00:38:02.720 --> 00:38:05.340 align:middle line:84%
say here, you calculate
the slope is like this,

00:38:05.340 --> 00:38:08.265 align:middle line:84%
it's negative, which means
that if you increase w,

00:38:08.265 --> 00:38:09.640 align:middle line:84%
if you go in this
direction, it's

00:38:09.640 --> 00:38:10.932 align:middle line:90%
going to decrease the function.

00:38:10.932 --> 00:38:13.720 align:middle line:90%


00:38:13.720 --> 00:38:15.000 align:middle line:90%
All right.

00:38:15.000 --> 00:38:17.960 align:middle line:84%
And if it's kind
of close to 0, it

00:38:17.960 --> 00:38:22.320 align:middle line:84%
means that changing w slightly
won't change anything.

00:38:22.320 --> 00:38:26.880 align:middle line:84%
So if you're here, changing it
slightly won't change anything.

00:38:26.880 --> 00:38:28.080 align:middle line:90%
That's it.

00:38:28.080 --> 00:38:31.680 align:middle line:84%
So what we do is
this immediately

00:38:31.680 --> 00:38:33.760 align:middle line:84%
suggests an algorithm
for minimizing

00:38:33.760 --> 00:38:38.360 align:middle line:84%
g w, which is let's start
with some random point w,

00:38:38.360 --> 00:38:41.400 align:middle line:84%
and then let's calculate the
derivative at that point.

00:38:41.400 --> 00:38:44.980 align:middle line:84%
And once we do that, there
are three possibilities.

00:38:44.980 --> 00:38:48.300 align:middle line:84%
It could be positive,
negative, or close to 0.

00:38:48.300 --> 00:38:50.540 align:middle line:84%
And if it's positive, we
know that increasing w,

00:38:50.540 --> 00:38:52.820 align:middle line:84%
will increase the
function, but we

00:38:52.820 --> 00:38:55.440 align:middle line:84%
want to decrease the function,
we want to minimize it,

00:38:55.440 --> 00:38:57.460 align:middle line:84%
which means that we should
not be increasing w.

00:38:57.460 --> 00:38:59.420 align:middle line:90%
We should be doing what here?

00:38:59.420 --> 00:39:01.340 align:middle line:90%
STUDENTS: Decrease.

00:39:01.340 --> 00:39:02.300 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

00:39:02.300 --> 00:39:04.790 align:middle line:84%
And similarly, if it's negative,
what should we do here?

00:39:04.790 --> 00:39:07.315 align:middle line:90%
STUDENT: Increase.

00:39:07.315 --> 00:39:08.440 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:39:08.440 --> 00:39:10.660 align:middle line:84%
So in the first case,
you reduce w slightly.

00:39:10.660 --> 00:39:12.633 align:middle line:84%
In the second case, you
increase w slightly.

00:39:12.633 --> 00:39:14.300 align:middle line:84%
And if the thing is
close to 0, you just

00:39:14.300 --> 00:39:16.175 align:middle line:84%
stop because there's
nothing else you can do.

00:39:16.175 --> 00:39:21.380 align:middle line:90%


00:39:21.380 --> 00:39:26.700 align:middle line:84%
This is the basic intuition
behind how GPT-4 was built,

00:39:26.700 --> 00:39:29.220 align:middle line:84%
which is kind of shocking
if you think about it.

00:39:29.220 --> 00:39:32.780 align:middle line:84%
Which means that all the
heavy-duty optimization stuff

00:39:32.780 --> 00:39:35.060 align:middle line:84%
that people have figured
out over the decades

00:39:35.060 --> 00:39:38.020 align:middle line:90%
is kind of not used.

00:39:38.020 --> 00:39:39.850 align:middle line:84%
This algorithm is
what's being used

00:39:39.850 --> 00:39:42.170 align:middle line:90%
with some flavors on top of it.

00:39:42.170 --> 00:39:44.490 align:middle line:90%
So, yeah, so back to this.

00:39:44.490 --> 00:39:46.070 align:middle line:90%
And you do that.

00:39:46.070 --> 00:39:49.890 align:middle line:84%
And then, if you have run
out of time or compute,

00:39:49.890 --> 00:39:53.950 align:middle line:84%
or if you're run out of
time and so on, just stop.

00:39:53.950 --> 00:39:55.950 align:middle line:84%
Otherwise, just go back
to step 2 and try again.

00:39:55.950 --> 00:40:00.090 align:middle line:84%
Of course, if it's close to
0, you've got to stop anyway.

00:40:00.090 --> 00:40:02.330 align:middle line:90%
Yeah?

00:40:02.330 --> 00:40:06.250 align:middle line:84%
STUDENT: Is there the concern
of a potentially local minimum

00:40:06.250 --> 00:40:06.750 align:middle line:90%
there?

00:40:06.750 --> 00:40:08.042 align:middle line:90%
RAMA RAMAKRISHNAN: It's coming.

00:40:08.042 --> 00:40:10.010 align:middle line:90%


00:40:10.010 --> 00:40:11.230 align:middle line:90%
So that's a function.

00:40:11.230 --> 00:40:13.570 align:middle line:84%
It's going to find
you some point where

00:40:13.570 --> 00:40:15.090 align:middle line:90%
the data is kind of close to 0.

00:40:15.090 --> 00:40:19.950 align:middle line:84%
So this is called
gradient descent.

00:40:19.950 --> 00:40:24.810 align:middle line:84%
This is gradient descent,
this little algorithm.

00:40:24.810 --> 00:40:29.090 align:middle line:84%
And this very
PowerPointy MBA table

00:40:29.090 --> 00:40:32.650 align:middle line:84%
can be collapsed into
this little expression.

00:40:32.650 --> 00:40:35.930 align:middle line:84%
Basically says, calculate
the derivative multiplied

00:40:35.930 --> 00:40:38.410 align:middle line:84%
by a small number, which
we'll get to in a second,

00:40:38.410 --> 00:40:40.990 align:middle line:90%
and then change the old w to--

00:40:40.990 --> 00:40:43.870 align:middle line:84%
the new w is the old w
minus the little number

00:40:43.870 --> 00:40:45.670 align:middle line:90%
times the gradient.

00:40:45.670 --> 00:40:48.790 align:middle line:84%
So this little one-line formula
is basically gradient descent.

00:40:48.790 --> 00:40:51.550 align:middle line:90%


00:40:51.550 --> 00:40:54.750 align:middle line:84%
And what you should do, just
to build your intuition,

00:40:54.750 --> 00:40:58.270 align:middle line:84%
is to make sure that these
three possibilities here

00:40:58.270 --> 00:40:59.105 align:middle line:90%
map nicely to this.

00:40:59.105 --> 00:41:01.230 align:middle line:84%
Like, this thing will
actually captures these three

00:41:01.230 --> 00:41:02.595 align:middle line:90%
possibilities.

00:41:02.595 --> 00:41:05.020 align:middle line:84%
Can you guess when gradient
descent was invented?

00:41:05.020 --> 00:41:07.237 align:middle line:90%


00:41:07.237 --> 00:41:09.070 align:middle line:84%
You need to have some
historical fun, right?

00:41:09.070 --> 00:41:12.910 align:middle line:90%


00:41:12.910 --> 00:41:14.990 align:middle line:90%
STUDENT: The 19th century?

00:41:14.990 --> 00:41:16.350 align:middle line:90%
RAMA RAMAKRISHNAN: 19th century.

00:41:16.350 --> 00:41:17.110 align:middle line:90%
Yeah, OK.

00:41:17.110 --> 00:41:17.610 align:middle line:90%
Good.

00:41:17.610 --> 00:41:18.210 align:middle line:90%
Very good.

00:41:18.210 --> 00:41:20.350 align:middle line:90%
Excellent guess.

00:41:20.350 --> 00:41:22.710 align:middle line:90%
1847.

00:41:22.710 --> 00:41:27.190 align:middle line:84%
It was invented in 1847 by
Cauchy, the great mathematician.

00:41:27.190 --> 00:41:31.410 align:middle line:84%
And in fact, if you're curious,
you can check out the paper.

00:41:31.410 --> 00:41:35.650 align:middle line:84%
I gave you give you the paper
here for handy reference.

00:41:35.650 --> 00:41:36.610 align:middle line:90%
OK.

00:41:36.610 --> 00:41:38.170 align:middle line:90%
So 1847.

00:41:38.170 --> 00:41:44.250 align:middle line:84%
So GPT-4 is built using an
algorithm invented in 1847.

00:41:44.250 --> 00:41:47.890 align:middle line:84%
Which I find
astonishing, frankly.

00:41:47.890 --> 00:41:51.610 align:middle line:90%
This little thing is so capable.

00:41:51.610 --> 00:41:52.970 align:middle line:90%
OK.

00:41:52.970 --> 00:41:54.470 align:middle line:90%
So that's gradient descent.

00:41:54.470 --> 00:41:58.130 align:middle line:84%
And this little number alpha
is called the learning rate,

00:41:58.130 --> 00:42:00.610 align:middle line:84%
and it's our way of
essentially quantifying

00:42:00.610 --> 00:42:04.790 align:middle line:84%
the idea of, let's not increase
or decrease w massively,

00:42:04.790 --> 00:42:07.370 align:middle line:84%
let's do it slightly
because the gradient is

00:42:07.370 --> 00:42:11.270 align:middle line:84%
only valid for small
movements around your point.

00:42:11.270 --> 00:42:14.810 align:middle line:84%
If you take a big
step, all bets are off.

00:42:14.810 --> 00:42:20.850 align:middle line:84%
So this alpha tells you how
small a step should you take.

00:42:20.850 --> 00:42:25.712 align:middle line:84%
And typically, it's set to very
small values, like 0.1, 0.01,

00:42:25.712 --> 00:42:26.670 align:middle line:90%
and so on and so forth.

00:42:26.670 --> 00:42:28.045 align:middle line:84%
And in fact, if
you read any deep

00:42:28.045 --> 00:42:30.610 align:middle line:84%
learning academic papers
where they have trained

00:42:30.610 --> 00:42:33.990 align:middle line:84%
like a big model to do
something, a lot of researchers

00:42:33.990 --> 00:42:35.693 align:middle line:84%
will very quickly
go to the appendix

00:42:35.693 --> 00:42:38.110 align:middle line:84%
where they have described
exactly what learning rates were

00:42:38.110 --> 00:42:41.030 align:middle line:84%
used because the
learning rate is

00:42:41.030 --> 00:42:43.813 align:middle line:84%
part of the IP for
how it's built.

00:42:43.813 --> 00:42:45.230 align:middle line:84%
It's a lot of trial
and error that

00:42:45.230 --> 00:42:47.510 align:middle line:90%
goes into these learning rates.

00:42:47.510 --> 00:42:51.230 align:middle line:90%
OK, so that is gradient descent.

00:42:51.230 --> 00:42:55.590 align:middle line:84%
So if we apply this algorithm to
g w, our original function, we

00:42:55.590 --> 00:42:59.910 align:middle line:84%
just keep on doing this thing
a few times, what you will find

00:42:59.910 --> 00:43:03.110 align:middle line:84%
is that if-- let's say we start
with two points-- the point

00:43:03.110 --> 00:43:05.390 align:middle line:90%
we randomly pick is 2.5.

00:43:05.390 --> 00:43:07.250 align:middle line:90%
We set the alpha to 1.

00:43:07.250 --> 00:43:09.667 align:middle line:84%
We run this algorithm, it
starts here, then it goes there,

00:43:09.667 --> 00:43:11.500 align:middle line:84%
it goes there, boom,
boom, boom, boom, boom,

00:43:11.500 --> 00:43:12.910 align:middle line:84%
and then it finally
ends up here.

00:43:12.910 --> 00:43:14.327 align:middle line:84%
In, like, four or
five iterations,

00:43:14.327 --> 00:43:16.430 align:middle line:90%
it finds the minimum.

00:43:16.430 --> 00:43:18.910 align:middle line:84%
This is obviously a very
simple, well-behaved, nice

00:43:18.910 --> 00:43:22.670 align:middle line:84%
little function, so you
can easily optimize it.

00:43:22.670 --> 00:43:24.490 align:middle line:84%
If you want, you can
just go to this thing.

00:43:24.490 --> 00:43:27.810 align:middle line:84%
There's a nice animation
of this thing as well.

00:43:27.810 --> 00:43:28.310 align:middle line:90%
OK.

00:43:28.310 --> 00:43:31.150 align:middle line:84%
So now-- all right,
before we actually

00:43:31.150 --> 00:43:32.868 align:middle line:84%
go to the
multivariable function,

00:43:32.868 --> 00:43:34.410 align:middle line:84%
I want to go to the
question that you

00:43:34.410 --> 00:43:36.180 align:middle line:90%
posed about local minima.

00:43:36.180 --> 00:43:37.310 align:middle line:90%
Actually, you know what?

00:43:37.310 --> 00:43:38.852 align:middle line:84%
I think I may have
some slides on it,

00:43:38.852 --> 00:43:40.290 align:middle line:84%
so, sorry, I'll
come back to this.

00:43:40.290 --> 00:43:41.930 align:middle line:90%
So let's actually see what--

00:43:41.930 --> 00:43:44.850 align:middle line:84%
we looked at a toy example where
there was only one variable.

00:43:44.850 --> 00:43:46.570 align:middle line:90%
What if you have--

00:43:46.570 --> 00:43:47.990 align:middle line:90%
what if it was GPT-3?

00:43:47.990 --> 00:43:51.570 align:middle line:84%
GPT-3 has 175
billion parameters.

00:43:51.570 --> 00:43:52.910 align:middle line:90%
175 billion.

00:43:52.910 --> 00:43:55.310 align:middle line:84%
And GPT-4, they haven't
published it, so we don't know.

00:43:55.310 --> 00:43:57.730 align:middle line:84%
It's supposed to
be 8 times as much.

00:43:57.730 --> 00:44:01.150 align:middle line:84%
So, I mean, the number
of parameters is massive.

00:44:01.150 --> 00:44:06.850 align:middle line:84%
So basically, our loss function
has billions of variables,

00:44:06.850 --> 00:44:10.970 align:middle line:84%
billions of w's that we need to
optimize over, minimize over.

00:44:10.970 --> 00:44:13.830 align:middle line:84%
So we need to use this notion
of a partial derivative.

00:44:13.830 --> 00:44:16.090 align:middle line:84%
So let's take baby steps
and say, OK, what if you

00:44:16.090 --> 00:44:18.270 align:middle line:90%
have a two-variable function?

00:44:18.270 --> 00:44:19.970 align:middle line:84%
Something like
this, very simple.

00:44:19.970 --> 00:44:22.610 align:middle line:84%
So what we can do is, we can
calculate the partial derivative

00:44:22.610 --> 00:44:26.570 align:middle line:84%
of g with respect to
each of these w's, and

00:44:26.570 --> 00:44:29.830 align:middle line:84%
the partial derivative, just to
quickly refresh your memories,

00:44:29.830 --> 00:44:32.590 align:middle line:84%
is you take a
function, you pretend

00:44:32.590 --> 00:44:36.510 align:middle line:84%
that everything other
than w is a constant.

00:44:36.510 --> 00:44:38.790 align:middle line:84%
Then the function
becomes-- a function

00:44:38.790 --> 00:44:40.245 align:middle line:90%
of just one variable w, w1.

00:44:40.245 --> 00:44:42.870 align:middle line:84%
And then you just differentiate
it like you do everything else,

00:44:42.870 --> 00:44:48.550 align:middle line:84%
and you get something, and
that is this thing here.

00:44:48.550 --> 00:44:51.230 align:middle line:84%
And then you do the same thing
for w2, you get this thing here.

00:44:51.230 --> 00:44:55.110 align:middle line:84%
And then you just stack
them up in a nice list.

00:44:55.110 --> 00:44:58.017 align:middle line:84%
This is the vector of
partial derivatives.

00:44:58.017 --> 00:44:59.350 align:middle line:90%
So how should we interpret this?

00:44:59.350 --> 00:45:00.370 align:middle line:90%
The same way as before.

00:45:00.370 --> 00:45:03.350 align:middle line:84%
Basically, for a
small change in w1,

00:45:03.350 --> 00:45:05.550 align:middle line:84%
keeping w2 and
everything else fixed,

00:45:05.550 --> 00:45:09.210 align:middle line:84%
how does the function change
if we change just w1 slightly?

00:45:09.210 --> 00:45:14.070 align:middle line:84%
And similarly for w2 and all
the way to w1, 75 billion.

00:45:14.070 --> 00:45:15.750 align:middle line:90%
It's the same thing.

00:45:15.750 --> 00:45:20.690 align:middle line:84%
So, now, when you have these
functions with many variables,

00:45:20.690 --> 00:45:23.910 align:middle line:84%
many w's, since
we have a gradient

00:45:23.910 --> 00:45:26.390 align:middle line:84%
for each one of those
w's, we stack them

00:45:26.390 --> 00:45:29.470 align:middle line:84%
up into a nice vector
of derivatives,

00:45:29.470 --> 00:45:32.250 align:middle line:84%
and this vector is
called the gradient.

00:45:32.250 --> 00:45:36.410 align:middle line:90%
And it's denoted using this--

00:45:36.410 --> 00:45:38.834 align:middle line:84%
anyone know what the
symbol is called?

00:45:38.834 --> 00:45:39.798 align:middle line:90%
STUDENT: Laplacian?

00:45:39.798 --> 00:45:41.730 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah?

00:45:41.730 --> 00:45:43.348 align:middle line:90%
STUDENT: Laplacian?

00:45:43.348 --> 00:45:44.390 align:middle line:90%
RAMA RAMAKRISHNAN: Maybe.

00:45:44.390 --> 00:45:48.930 align:middle line:84%
Maybe it's a synonym, but the
one I'm familiar with is nabla.

00:45:48.930 --> 00:45:51.122 align:middle line:84%
Delta is the one that's
upside-down triangle,

00:45:51.122 --> 00:45:53.330 align:middle line:84%
but I think the upside-down
triangle is called nabla,

00:45:53.330 --> 00:45:54.810 align:middle line:90%
if I recall.

00:45:54.810 --> 00:45:55.930 align:middle line:90%
Am I right?

00:45:55.930 --> 00:45:58.147 align:middle line:90%
Thank you.

00:45:58.147 --> 00:45:58.730 align:middle line:90%
He's my go-to.

00:45:58.730 --> 00:46:02.530 align:middle line:90%


00:46:02.530 --> 00:46:03.030 align:middle line:90%
So yeah.

00:46:03.030 --> 00:46:05.990 align:middle line:84%
So the gradient, we just
call it the gradient,

00:46:05.990 --> 00:46:08.490 align:middle line:90%
and it's written as this.

00:46:08.490 --> 00:46:10.970 align:middle line:84%
All right, so what
we do is we simply

00:46:10.970 --> 00:46:13.570 align:middle line:84%
do gradient descent on
every one of the w's

00:46:13.570 --> 00:46:16.990 align:middle line:90%
using its partial derivative.

00:46:16.990 --> 00:46:21.010 align:middle line:84%
So in a gradient step, we
update w1 using this formula,

00:46:21.010 --> 00:46:23.010 align:middle line:90%
w2 using this formula.

00:46:23.010 --> 00:46:25.485 align:middle line:90%
Finished.

00:46:25.485 --> 00:46:27.110 align:middle line:84%
We've just generalized
gradient descent

00:46:27.110 --> 00:46:30.870 align:middle line:84%
to an arbitrary
number of variables.

00:46:30.870 --> 00:46:33.710 align:middle line:84%
And of course, as before, this
can be summarized compactly

00:46:33.710 --> 00:46:37.030 align:middle line:90%
as this vector formula.

00:46:37.030 --> 00:46:38.260 align:middle line:90%
We'll just do this.

00:46:38.260 --> 00:46:42.950 align:middle line:90%


00:46:42.950 --> 00:46:46.830 align:middle line:84%
So what's going
on here is that we

00:46:46.830 --> 00:46:59.310 align:middle line:84%
have w1 or w1 minus alpha times
the function g of w1 and w2,

00:46:59.310 --> 00:47:03.578 align:middle line:90%
w2 minus alpha g by w2.

00:47:03.578 --> 00:47:05.870 align:middle line:84%
And then all we're doing is
we're just stacking them up

00:47:05.870 --> 00:47:08.990 align:middle line:90%
into a vector like that.

00:47:08.990 --> 00:47:14.970 align:middle line:90%


00:47:14.970 --> 00:47:16.630 align:middle line:90%
Minus alpha.

00:47:16.630 --> 00:47:22.100 align:middle line:90%
And this vector is like that.

00:47:22.100 --> 00:47:27.690 align:middle line:90%


00:47:27.690 --> 00:47:31.760 align:middle line:84%
So this can be written as just
this vector w, the new vector,

00:47:31.760 --> 00:47:37.610 align:middle line:84%
old vector minus alpha, and
the gradient is finished.

00:47:37.610 --> 00:47:40.410 align:middle line:90%
And you can see, if it's GPT-3.

00:47:40.410 --> 00:47:45.210 align:middle line:84%
This vector is going
to be 175 billion long.

00:47:45.210 --> 00:47:47.390 align:middle line:84%
Whether it's 2 or 175
billion, who cares?

00:47:47.390 --> 00:47:50.370 align:middle line:90%
It's the same thing, right?

00:47:50.370 --> 00:47:51.810 align:middle line:90%
OK.

00:47:51.810 --> 00:47:53.755 align:middle line:84%
So yeah, so that's
what we have here.

00:47:53.755 --> 00:47:56.130 align:middle line:84%
I'm really thrilled, by the
way, this whole iPad business

00:47:56.130 --> 00:47:56.990 align:middle line:90%
is working out.

00:47:56.990 --> 00:47:59.350 align:middle line:84%
So I was a little
worried about it.

00:47:59.350 --> 00:48:00.180 align:middle line:90%
OK.

00:48:00.180 --> 00:48:03.218 align:middle line:84%
So, if you look at two
dimensions, this function--

00:48:03.218 --> 00:48:05.510 align:middle line:84%
and if you actually look at--
if you plot the function,

00:48:05.510 --> 00:48:08.292 align:middle line:84%
this is w, the first w,
the second w, and then

00:48:08.292 --> 00:48:09.750 align:middle line:84%
this is actually
the loss function,

00:48:09.750 --> 00:48:12.250 align:middle line:84%
match that function,
g w, and so you

00:48:12.250 --> 00:48:14.290 align:middle line:84%
want to find the minimum
here, and so this

00:48:14.290 --> 00:48:16.950 align:middle line:84%
is how gradient descent
will progress if you're

00:48:16.950 --> 00:48:18.410 align:middle line:90%
starting from this point.

00:48:18.410 --> 00:48:21.887 align:middle line:84%
Or you can also look at it from
up top-down into the function.

00:48:21.887 --> 00:48:24.470 align:middle line:84%
And that's what this picture is,
and it shows gradient descent

00:48:24.470 --> 00:48:28.310 align:middle line:84%
starting from there, and working
its way down from here all

00:48:28.310 --> 00:48:30.630 align:middle line:90%
the way to the center.

00:48:30.630 --> 00:48:34.590 align:middle line:90%
OK, so all right, local minima.

00:48:34.590 --> 00:48:37.070 align:middle line:84%
So now gradient
descent will just

00:48:37.070 --> 00:48:41.930 align:middle line:90%
stop near hopefully a minimum.

00:48:41.930 --> 00:48:44.690 align:middle line:84%
But the problem is, it may
not be a global minimum.

00:48:44.690 --> 00:48:47.390 align:middle line:90%
It may not even be a minimum.

00:48:47.390 --> 00:48:51.070 align:middle line:84%
So let's see what I'm
talking about here.

00:48:51.070 --> 00:48:53.870 align:middle line:90%
Here are some possibilities.

00:48:53.870 --> 00:48:57.590 align:middle line:90%
So let's take a simple function.

00:48:57.590 --> 00:49:00.990 align:middle line:84%
Let's say this is
g w, this is w.

00:49:00.990 --> 00:49:03.900 align:middle line:84%
And turns out, this function
actually looks like this.

00:49:03.900 --> 00:49:12.230 align:middle line:90%


00:49:12.230 --> 00:49:14.620 align:middle line:90%
So you can see here--

00:49:14.620 --> 00:49:17.790 align:middle line:90%


00:49:17.790 --> 00:49:27.130 align:middle line:84%
well, this point, this point
here is a local minimum.

00:49:27.130 --> 00:49:28.810 align:middle line:90%
This is a local minimum.

00:49:28.810 --> 00:49:30.890 align:middle line:90%
This is a local minimum.

00:49:30.890 --> 00:49:35.130 align:middle line:84%
These are all lots
of local minima here.

00:49:35.130 --> 00:49:39.370 align:middle line:84%
And, yeah, there's a lot
of local minima here, too.

00:49:39.370 --> 00:49:43.930 align:middle line:84%
So these are all places in which
the derivative is going to be 0.

00:49:43.930 --> 00:49:46.250 align:middle line:84%
So if you run gradient
descent and it

00:49:46.250 --> 00:49:49.290 align:middle line:84%
stops because the
gradient is 0, you

00:49:49.290 --> 00:49:52.630 align:middle line:90%
could be in any of these places.

00:49:52.630 --> 00:49:55.610 align:middle line:84%
So there is no guarantee,
so this in this picture

00:49:55.610 --> 00:49:59.450 align:middle line:84%
happens to be maybe the
global minimum because it's

00:49:59.450 --> 00:50:02.450 align:middle line:84%
the lowest of the
lot, But there's

00:50:02.450 --> 00:50:04.950 align:middle line:84%
no guarantee you're
actually going to get there.

00:50:04.950 --> 00:50:06.450 align:middle line:84%
There's not even
a guarantee you're

00:50:06.450 --> 00:50:08.070 align:middle line:84%
going to be in any
of these places

00:50:08.070 --> 00:50:10.010 align:middle line:84%
because you could
literally be in this thing

00:50:10.010 --> 00:50:14.650 align:middle line:84%
here where it's taking a break
and then continuing on down.

00:50:14.650 --> 00:50:16.750 align:middle line:84%
That, by the way, is
called a saddle point.

00:50:16.750 --> 00:50:19.630 align:middle line:84%
I drew it badly, but the source
of coming in-- taking a break

00:50:19.630 --> 00:50:22.150 align:middle line:84%
and going down again, it's
called a saddle point.

00:50:22.150 --> 00:50:25.010 align:middle line:84%
So gradient descent can
stop at a saddle point,

00:50:25.010 --> 00:50:26.155 align:middle line:90%
it can stop at some minima.

00:50:26.155 --> 00:50:28.030 align:middle line:84%
There's no guarantee
it's going to be global.

00:50:28.030 --> 00:50:33.030 align:middle line:90%


00:50:33.030 --> 00:50:37.310 align:middle line:84%
But it turns out,
it has not mattered.

00:50:37.310 --> 00:50:38.810 align:middle line:90%
So it has not mattered.

00:50:38.810 --> 00:50:41.987 align:middle line:84%
And there are a whole bunch of
reasons why it has not mattered.

00:50:41.987 --> 00:50:44.570 align:middle line:84%
Because when you have these very
complicated neural networks--

00:50:44.570 --> 00:50:49.042 align:middle line:84%
they're very complex functions,
even finding a decent solution

00:50:49.042 --> 00:50:50.750 align:middle line:84%
to these complicated
networks is actually

00:50:50.750 --> 00:50:52.990 align:middle line:84%
really good for
solving the problem.

00:50:52.990 --> 00:50:56.010 align:middle line:84%
You don't have to go to the
best, best possible solution.

00:50:56.010 --> 00:50:58.690 align:middle line:84%
And in fact, if you go to
the best possible solution,

00:50:58.690 --> 00:51:02.110 align:middle line:84%
you actually run the
risk of overfitting.

00:51:02.110 --> 00:51:03.330 align:middle line:90%
So that's one reason.

00:51:03.330 --> 00:51:05.190 align:middle line:90%
The other interesting reason--

00:51:05.190 --> 00:51:07.950 align:middle line:84%
and by the way, this is a
very hot area of research,

00:51:07.950 --> 00:51:09.610 align:middle line:90%
to figure out exactly--

00:51:09.610 --> 00:51:10.505 align:middle line:90%
so it's like this.

00:51:10.505 --> 00:51:12.630 align:middle line:84%
Empirically, what we have
seen is that not worrying

00:51:12.630 --> 00:51:14.090 align:middle line:84%
about local minima,
global minima,

00:51:14.090 --> 00:51:16.930 align:middle line:84%
all that stuff has not hurt
us because these things

00:51:16.930 --> 00:51:18.670 align:middle line:90%
are amazing.

00:51:18.670 --> 00:51:20.573 align:middle line:84%
GPT-4, probably they
just stopped somewhere.

00:51:20.573 --> 00:51:22.490 align:middle line:84%
They probably-- it wasn't
even a local minima.

00:51:22.490 --> 00:51:24.698 align:middle line:84%
They're like, oh God, it's
been running for six days,

00:51:24.698 --> 00:51:27.090 align:middle line:84%
it's spent $2
million, let's stop.

00:51:27.090 --> 00:51:29.050 align:middle line:84%
Because these are
very expensive.

00:51:29.050 --> 00:51:31.777 align:middle line:90%
But that's still so magical.

00:51:31.777 --> 00:51:34.110 align:middle line:84%
You don't need to get to
anywhere close to local minima.

00:51:34.110 --> 00:51:37.650 align:middle line:84%
But there's another interesting
point, which I read about.

00:51:37.650 --> 00:51:41.650 align:middle line:84%
People basically
hypothesize that for you

00:51:41.650 --> 00:51:44.350 align:middle line:84%
to be at a local minimum, just
think about what it means.

00:51:44.350 --> 00:51:47.050 align:middle line:84%
It means that you're standing
at a particular point.

00:51:47.050 --> 00:51:49.450 align:middle line:84%
In every direction
that you look,

00:51:49.450 --> 00:51:52.850 align:middle line:90%
things are just sloping upward.

00:51:52.850 --> 00:51:54.650 align:middle line:90%
Everything is sloping upward.

00:51:54.650 --> 00:51:56.870 align:middle line:84%
If everything is sloping
upward all around you,

00:51:56.870 --> 00:51:59.690 align:middle line:84%
could you be at a local
minima, by definition?

00:51:59.690 --> 00:52:03.698 align:middle line:84%
But if you have a billion
dimensions, what are the odds

00:52:03.698 --> 00:52:05.490 align:middle line:84%
that you're going to
be standing at a point

00:52:05.490 --> 00:52:08.730 align:middle line:84%
where every one of those
billion is going upward?

00:52:08.730 --> 00:52:10.145 align:middle line:90%
The odds are really low.

00:52:10.145 --> 00:52:12.270 align:middle line:84%
Chances are, some of them
are going to be going up,

00:52:12.270 --> 00:52:14.395 align:middle line:84%
some of them are going
down, others are coming down

00:52:14.395 --> 00:52:16.870 align:middle line:84%
and going the other way,
it's going to be crazy.

00:52:16.870 --> 00:52:18.310 align:middle line:84%
So in some sense,
the best you can

00:52:18.310 --> 00:52:20.550 align:middle line:84%
hope for in these very
high-dimensional situations

00:52:20.550 --> 00:52:23.150 align:middle line:90%
is probably a saddle point.

00:52:23.150 --> 00:52:25.870 align:middle line:84%
And it turns out,
it's good enough.

00:52:25.870 --> 00:52:29.843 align:middle line:84%
So for those reasons, we are
content with just running

00:52:29.843 --> 00:52:31.510 align:middle line:84%
gradient descent with
some tweaks, which

00:52:31.510 --> 00:52:35.910 align:middle line:84%
I'll get to in a second, and it
just performs really admirably.

00:52:35.910 --> 00:52:37.470 align:middle line:90%
Yeah?

00:52:37.470 --> 00:52:41.290 align:middle line:84%
STUDENT: How does alpha depend
on how much compute you have?

00:52:41.290 --> 00:52:42.990 align:middle line:84%
Like, would you set
the learning rate

00:52:42.990 --> 00:52:45.430 align:middle line:90%
based on that, or not really?

00:52:45.430 --> 00:52:48.030 align:middle line:84%
RAMA RAMAKRISHNAN: No, the
learning rate is really--

00:52:48.030 --> 00:52:50.710 align:middle line:84%
is a measure of--
it's like this.

00:52:50.710 --> 00:52:53.750 align:middle line:84%
When you're at a point where
you think that the gradient is

00:52:53.750 --> 00:52:54.870 align:middle line:90%
looking nice and--

00:52:54.870 --> 00:52:56.370 align:middle line:84%
if you take a step
in the direction,

00:52:56.370 --> 00:52:57.410 align:middle line:90%
it's going to go down.

00:52:57.410 --> 00:52:59.790 align:middle line:84%
And if you further
believe that it's

00:52:59.790 --> 00:53:02.550 align:middle line:84%
going to keep going down in
the direction for a while,

00:53:02.550 --> 00:53:05.975 align:middle line:84%
then you're very confident
about taking a big step.

00:53:05.975 --> 00:53:07.350 align:middle line:84%
But if you're
like, I don't know,

00:53:07.350 --> 00:53:09.250 align:middle line:84%
because maybe I
take a little step,

00:53:09.250 --> 00:53:10.455 align:middle line:90%
maybe I have to go this way.

00:53:10.455 --> 00:53:11.830 align:middle line:84%
I can't go straight
anymore, then

00:53:11.830 --> 00:53:13.913 align:middle line:84%
you don't want to take a
big step because then you

00:53:13.913 --> 00:53:14.730 align:middle line:90%
have to backtrack.

00:53:14.730 --> 00:53:17.930 align:middle line:84%
So those kinds of considerations
go into the learning rate.

00:53:17.930 --> 00:53:21.630 align:middle line:84%
And so that's of the rough
answer to your question.

00:53:21.630 --> 00:53:24.850 align:middle line:84%
It's not so much determined by
compute and bandwidth and things

00:53:24.850 --> 00:53:25.850 align:middle line:90%
like that.

00:53:25.850 --> 00:53:29.250 align:middle line:84%
But, again, it's a complicated
thing because sometimes

00:53:29.250 --> 00:53:31.370 align:middle line:84%
with a given amount
of compute, if you

00:53:31.370 --> 00:53:33.130 align:middle line:84%
have a particular
kind of data, you

00:53:33.130 --> 00:53:35.290 align:middle line:84%
can have very aggressive
learning rates.

00:53:35.290 --> 00:53:39.230 align:middle line:84%
So it tends to be a bit
jumbled up, complicated,

00:53:39.230 --> 00:53:41.930 align:middle line:84%
but that's the quick
surface-level idea of what's

00:53:41.930 --> 00:53:43.830 align:middle line:90%
going on.

00:53:43.830 --> 00:53:44.330 align:middle line:90%
OK.

00:53:44.330 --> 00:53:47.010 align:middle line:90%


00:53:47.010 --> 00:53:47.510 align:middle line:90%
9:31.

00:53:47.510 --> 00:53:50.938 align:middle line:90%


00:53:50.938 --> 00:53:52.730 align:middle line:84%
By the way, folks, this
lecture is probably

00:53:52.730 --> 00:53:54.570 align:middle line:84%
one of the driest
in the semester

00:53:54.570 --> 00:53:57.410 align:middle line:84%
because I have to go
through all the concepts.

00:53:57.410 --> 00:54:00.930 align:middle line:84%
Once we start doing Colabs,
things get a lot more lively.

00:54:00.930 --> 00:54:01.890 align:middle line:90%
OK.

00:54:01.890 --> 00:54:02.510 align:middle line:90%
All right.

00:54:02.510 --> 00:54:05.650 align:middle line:84%
So, now let's talk about
minimizing a loss function

00:54:05.650 --> 00:54:06.650 align:middle line:90%
gradient descent.

00:54:06.650 --> 00:54:09.170 align:middle line:84%
So here's our little binary
cross entropy loss function

00:54:09.170 --> 00:54:11.230 align:middle line:90%
that we saw from before.

00:54:11.230 --> 00:54:12.750 align:middle line:84%
This is what we
want to minimize.

00:54:12.750 --> 00:54:15.190 align:middle line:84%
So if you look at
this thing, where

00:54:15.190 --> 00:54:17.150 align:middle line:84%
are the variables
we need to change

00:54:17.150 --> 00:54:19.310 align:middle line:90%
to minimize this function?

00:54:19.310 --> 00:54:21.310 align:middle line:84%
Folks, don't look
at your phones.

00:54:21.310 --> 00:54:24.540 align:middle line:84%
Even with laptop and iPad use,
don't look at your phones.

00:54:24.540 --> 00:54:27.670 align:middle line:90%


00:54:27.670 --> 00:54:31.970 align:middle line:84%
STUDENT: Sorry, we've kind
of abstracted the variable w,

00:54:31.970 --> 00:54:34.630 align:middle line:84%
but just to bring it back,
those are actually the weights

00:54:34.630 --> 00:54:36.073 align:middle line:90%
in the neural network, right?

00:54:36.073 --> 00:54:37.990 align:middle line:84%
RAMA RAMAKRISHNAN: The
weights and the biases.

00:54:37.990 --> 00:54:39.450 align:middle line:84%
I'm just calling
them as weights.

00:54:39.450 --> 00:54:44.870 align:middle line:84%
STUDENT: So the output of
these minimization functions

00:54:44.870 --> 00:54:47.505 align:middle line:84%
are going to be the actual
weights in our model?

00:54:47.505 --> 00:54:48.630 align:middle line:90%
RAMA RAMAKRISHNAN: Exactly.

00:54:48.630 --> 00:54:50.030 align:middle line:90%
Exactly right.

00:54:50.030 --> 00:54:52.315 align:middle line:84%
The whole name of the game
is to find the weights.

00:54:52.315 --> 00:54:54.190 align:middle line:84%
And so for example, when
you see in the press

00:54:54.190 --> 00:54:58.950 align:middle line:84%
that Meta has essentially
made the weights of Llama

00:54:58.950 --> 00:55:03.030 align:middle line:84%
2 or something available, that's
basically what they have done.

00:55:03.030 --> 00:55:04.830 align:middle line:90%
Basically published the weights.

00:55:04.830 --> 00:55:06.030 align:middle line:84%
STUDENT: The reason
that's so valuable is--

00:55:06.030 --> 00:55:06.990 align:middle line:84%
RAMA RAMAKRISHNAN:
Microphone, please.

00:55:06.990 --> 00:55:07.490 align:middle line:90%
STUDENT: Oh.

00:55:07.490 --> 00:55:09.560 align:middle line:84%
Because if you have
a billion parameters,

00:55:09.560 --> 00:55:12.720 align:middle line:84%
the compute time on that is
horrendous and expensive.

00:55:12.720 --> 00:55:14.720 align:middle line:84%
That's why those
weights are so valuable?

00:55:14.720 --> 00:55:15.140 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

00:55:15.140 --> 00:55:17.140 align:middle line:84%
The weights are the
crown jewel because they

00:55:17.140 --> 00:55:19.140 align:middle line:84%
are the result of a
lot of money and time

00:55:19.140 --> 00:55:21.762 align:middle line:90%
and smartness being spent.

00:55:21.762 --> 00:55:23.220 align:middle line:84%
There is a separate
question of why

00:55:23.220 --> 00:55:26.020 align:middle line:90%
are they making it open source?

00:55:26.020 --> 00:55:28.740 align:middle line:84%
I'm happy to chat
about it offline.

00:55:28.740 --> 00:55:29.240 align:middle line:90%
All right.

00:55:29.240 --> 00:55:29.540 align:middle line:90%
Cool.

00:55:29.540 --> 00:55:31.550 align:middle line:84%
So what are the variables
we need to minimize?

00:55:31.550 --> 00:55:33.300 align:middle line:84%
It's basically the
parameters, and they're

00:55:33.300 --> 00:55:36.460 align:middle line:90%
hiding inside the model term.

00:55:36.460 --> 00:55:38.258 align:middle line:90%
Because what is the model?

00:55:38.258 --> 00:55:39.800 align:middle line:84%
The model is some
function like that.

00:55:39.800 --> 00:55:43.220 align:middle line:84%
If you look at the
simple GPA and experience

00:55:43.220 --> 00:55:45.820 align:middle line:84%
thing we looked at
on Monday, we finally

00:55:45.820 --> 00:55:48.660 align:middle line:84%
figured out that the actual
thing that comes out here

00:55:48.660 --> 00:55:51.260 align:middle line:84%
is going to be this complicated
function of all the x's

00:55:51.260 --> 00:55:52.840 align:middle line:84%
and the w's and so
on and so forth,

00:55:52.840 --> 00:55:56.440 align:middle line:84%
and that complicated thing is
showing up inside this thing.

00:55:56.440 --> 00:55:59.364 align:middle line:90%


00:55:59.364 --> 00:56:02.860 align:middle line:84%
And the w's here are the
variables we need to change

00:56:02.860 --> 00:56:04.580 align:middle line:90%
to minimize the loss function.

00:56:04.580 --> 00:56:07.840 align:middle line:84%
And it's important for
you to note and understand

00:56:07.840 --> 00:56:13.240 align:middle line:84%
that the values of x and
y and so on are just data.

00:56:13.240 --> 00:56:15.800 align:middle line:84%
You're not optimizing
anything, they're just data.

00:56:15.800 --> 00:56:17.910 align:middle line:84%
What you're
optimizing is the w's.

00:56:17.910 --> 00:56:18.410 align:middle line:90%
The weights.

00:56:18.410 --> 00:56:22.480 align:middle line:90%


00:56:22.480 --> 00:56:23.320 align:middle line:90%
OK.

00:56:23.320 --> 00:56:26.258 align:middle line:84%
So you imagine
replacing the model

00:56:26.258 --> 00:56:27.800 align:middle line:84%
here with the
mathematical expression

00:56:27.800 --> 00:56:30.480 align:middle line:84%
above, whenever this
appears the loss function.

00:56:30.480 --> 00:56:33.480 align:middle line:84%
And once you do that, your loss
function is just a good old

00:56:33.480 --> 00:56:35.880 align:middle line:90%
function of the w's.

00:56:35.880 --> 00:56:39.200 align:middle line:84%
The fact that it's a loss
function is kind of irrelevant.

00:56:39.200 --> 00:56:40.788 align:middle line:90%
It's just a function.

00:56:40.788 --> 00:56:43.080 align:middle line:84%
And since it's just a good
old function of the w's, you

00:56:43.080 --> 00:56:45.960 align:middle line:84%
can apply gradient descent
to it as we normally would.

00:56:45.960 --> 00:56:46.820 align:middle line:90%
It's no big deal.

00:56:46.820 --> 00:56:49.480 align:middle line:90%


00:56:49.480 --> 00:56:52.140 align:middle line:84%
Which brings us to something
called backpropagation.

00:56:52.140 --> 00:56:56.375 align:middle line:90%


00:56:56.375 --> 00:56:58.500 align:middle line:84%
If you remember nothing
else about backpropagation,

00:56:58.500 --> 00:56:59.960 align:middle line:90%
just remember this--

00:56:59.960 --> 00:57:01.860 align:middle line:84%
never used the word
backpropagation again.

00:57:01.860 --> 00:57:05.140 align:middle line:90%
Only use the word backprop.

00:57:05.140 --> 00:57:07.900 align:middle line:84%
You're hip and cool to the
deep learning community.

00:57:07.900 --> 00:57:09.880 align:middle line:90%
Backprop.

00:57:09.880 --> 00:57:12.340 align:middle line:90%
All right, so what is backprop?

00:57:12.340 --> 00:57:15.660 align:middle line:84%
Backprop is a very efficient
way to compute the gradient

00:57:15.660 --> 00:57:17.540 align:middle line:90%
of the loss function.

00:57:17.540 --> 00:57:19.620 align:middle line:84%
So when you have this
loss function, and let's

00:57:19.620 --> 00:57:22.260 align:middle line:84%
say you have a
billion w's, and you

00:57:22.260 --> 00:57:24.440 align:middle line:90%
have 10 million data points--

00:57:24.440 --> 00:57:27.620 align:middle line:84%
so, the little n we
saw was 10 million.

00:57:27.620 --> 00:57:30.580 align:middle line:90%
That is a lot of computation.

00:57:30.580 --> 00:57:34.340 align:middle line:84%
And that is just for one
step of gradient descent.

00:57:34.340 --> 00:57:36.940 align:middle line:90%
So backprop is a way--

00:57:36.940 --> 00:57:38.980 align:middle line:84%
is a very efficient
and clever way

00:57:38.980 --> 00:57:41.700 align:middle line:84%
to compute the gradient of
the loss function, which

00:57:41.700 --> 00:57:45.580 align:middle line:84%
takes advantage of the fact
that what we have here is not

00:57:45.580 --> 00:57:47.980 align:middle line:84%
some arbitrary model,
it's a model that

00:57:47.980 --> 00:57:51.962 align:middle line:84%
came from a particular kind of
neural network, which has layers

00:57:51.962 --> 00:57:53.420 align:middle line:84%
one after the other,
and then there

00:57:53.420 --> 00:57:55.220 align:middle line:90%
was an output at the very end.

00:57:55.220 --> 00:57:58.153 align:middle line:84%
So what backprop
does is it organizes

00:57:58.153 --> 00:57:59.820 align:middle line:84%
the computation in
the form of something

00:57:59.820 --> 00:58:01.740 align:middle line:84%
called a computational
graph, and the book has

00:58:01.740 --> 00:58:03.100 align:middle line:90%
a good discussion about it.

00:58:03.100 --> 00:58:06.040 align:middle line:84%
And so what we do is we
start at the very end,

00:58:06.040 --> 00:58:10.160 align:middle line:84%
we calculate the gradient of the
loss with respect to the output.

00:58:10.160 --> 00:58:12.692 align:middle line:84%
Then we move left, we
calculate the gradient

00:58:12.692 --> 00:58:14.400 align:middle line:84%
of that output with
respect to the output

00:58:14.400 --> 00:58:17.200 align:middle line:90%
of just the prior hidden layer.

00:58:17.200 --> 00:58:19.962 align:middle line:84%
Step to the left, calculate
the gradient of the current

00:58:19.962 --> 00:58:21.420 align:middle line:84%
with respect to
the previous layer.

00:58:21.420 --> 00:58:22.760 align:middle line:90%
You get the idea.

00:58:22.760 --> 00:58:25.520 align:middle line:84%
It's iterative and it
moves backwards, and by

00:58:25.520 --> 00:58:28.400 align:middle line:84%
doing so, you never repeat
the same computation twice

00:58:28.400 --> 00:58:30.920 align:middle line:90%
wastefully.

00:58:30.920 --> 00:58:32.320 align:middle line:90%
That's the big advantage.

00:58:32.320 --> 00:58:35.720 align:middle line:84%
You calculate once and reuse it
many, many, many, many times.

00:58:35.720 --> 00:58:38.880 align:middle line:84%
The second advantage is that
if you organize it this way,

00:58:38.880 --> 00:58:43.000 align:middle line:84%
it just becomes a sequence
of matrix multiplications.

00:58:43.000 --> 00:58:46.320 align:middle line:84%
And [AUDIO OUT]
sequence of matrix

00:58:46.320 --> 00:58:49.480 align:middle line:84%
multiplications and eliminates
redundant calculations.

00:58:49.480 --> 00:58:52.760 align:middle line:84%
And, best of all, there
are these things called

00:58:52.760 --> 00:58:55.160 align:middle line:84%
GPUs, Graphics Processing
Units, originally invented

00:58:55.160 --> 00:58:57.400 align:middle line:84%
to accelerate video
game rendering,

00:58:57.400 --> 00:58:59.880 align:middle line:84%
and as it turns out, to
accelerate video game rendering,

00:58:59.880 --> 00:59:01.980 align:middle line:84%
the core math operation
you do is basically

00:59:01.980 --> 00:59:03.580 align:middle line:90%
matrix multiplication.

00:59:03.580 --> 00:59:06.660 align:middle line:90%
Some linear algebra operations.

00:59:06.660 --> 00:59:08.460 align:middle line:84%
And so someone
really, at some point,

00:59:08.460 --> 00:59:11.940 align:middle line:84%
had the bright idea for deep
learning, calculating gradients,

00:59:11.940 --> 00:59:14.480 align:middle line:84%
and so on, we need to do
matrix multiplications,

00:59:14.480 --> 00:59:17.660 align:middle line:84%
and here are some specialized
hardware that does really--

00:59:17.660 --> 00:59:20.160 align:middle line:84%
that does a fast job of
matrix multiplications.

00:59:20.160 --> 00:59:22.340 align:middle line:90%
Can we use this for that?

00:59:22.340 --> 00:59:26.260 align:middle line:84%
And they did it, and
all hell broke loose.

00:59:26.260 --> 00:59:28.060 align:middle line:90%
That's literally what happened.

00:59:28.060 --> 00:59:30.900 align:middle line:84%
And that's why NVIDIA is
valued at about $1.5 trillion

00:59:30.900 --> 00:59:32.420 align:middle line:90%
or something?

00:59:32.420 --> 00:59:33.460 align:middle line:90%
So, yeah.

00:59:33.460 --> 00:59:35.340 align:middle line:90%
So they are really good.

00:59:35.340 --> 00:59:40.980 align:middle line:84%
And so backprop, the way you do
backprop, plus using it on GPUs,

00:59:40.980 --> 00:59:44.420 align:middle line:84%
leads to fast calculation
of loss function gradients.

00:59:44.420 --> 00:59:47.140 align:middle line:84%
If this thing were
not true, this class

00:59:47.140 --> 00:59:50.300 align:middle line:84%
would not exist because there
wouldn't be any deep learning

00:59:50.300 --> 00:59:52.860 align:middle line:90%
revolution.

00:59:52.860 --> 00:59:54.850 align:middle line:84%
This is a fundamental,
seminal reason.

00:59:54.850 --> 00:59:57.860 align:middle line:90%


00:59:57.860 --> 01:00:01.920 align:middle line:84%
All right, so the book
has a bunch of detail.

01:00:01.920 --> 01:00:05.080 align:middle line:90%
And I actually did--

01:00:05.080 --> 01:00:08.600 align:middle line:84%
I hand-worked-out an
example of calculating

01:00:08.600 --> 01:00:10.880 align:middle line:84%
a gradient the
old-fashioned way,

01:00:10.880 --> 01:00:13.400 align:middle line:84%
and calculating
it using backprop.

01:00:13.400 --> 01:00:15.680 align:middle line:84%
So take a look at it,
I'll post it on Canvas,

01:00:15.680 --> 01:00:18.420 align:middle line:84%
and you will understand exactly
where the savings come from,

01:00:18.420 --> 01:00:21.617 align:middle line:84%
where the efficiency
gains come from.

01:00:21.617 --> 01:00:23.700 align:middle line:84%
Because of time, I'm not
going to get into it now.

01:00:23.700 --> 01:00:26.440 align:middle line:90%


01:00:26.440 --> 01:00:27.020 align:middle line:90%
All right.

01:00:27.020 --> 01:00:28.880 align:middle line:90%
Any questions so far?

01:00:28.880 --> 01:00:30.175 align:middle line:90%
Yep?

01:00:30.175 --> 01:00:30.800 align:middle line:90%
STUDENT: Sorry.

01:00:30.800 --> 01:00:31.980 align:middle line:90%
I followed up to--

01:00:31.980 --> 01:00:34.380 align:middle line:84%
and so we've done
gradient descent,

01:00:34.380 --> 01:00:36.038 align:middle line:84%
which is different
than calculation

01:00:36.038 --> 01:00:37.580 align:middle line:84%
of the gradient of
the loss function.

01:00:37.580 --> 01:00:39.830 align:middle line:84%
What is the purpose of the
calculation of the gradient

01:00:39.830 --> 01:00:41.020 align:middle line:90%
of the loss function?

01:00:41.020 --> 01:00:42.895 align:middle line:84%
RAMA RAMAKRISHNAN: You
calculate the gradient

01:00:42.895 --> 01:00:45.120 align:middle line:84%
because the fundamental
operation of gradient descent

01:00:45.120 --> 01:00:50.060 align:middle line:84%
is to take your current value
of w and modify it slightly,

01:00:50.060 --> 01:00:52.960 align:middle line:84%
and the modification is
old value minus learning

01:00:52.960 --> 01:00:53.830 align:middle line:90%
rate times gradient.

01:00:53.830 --> 01:01:02.964 align:middle line:90%


01:01:02.964 --> 01:01:06.220 align:middle line:84%
It would be cool if I say go
back five slides to this thing

01:01:06.220 --> 01:01:07.340 align:middle line:90%
and it just goes back.

01:01:07.340 --> 01:01:08.040 align:middle line:90%
Product idea.

01:01:08.040 --> 01:01:08.540 align:middle line:90%
Anyone?

01:01:08.540 --> 01:01:09.040 align:middle line:90%
Startups?

01:01:09.040 --> 01:01:11.460 align:middle line:90%


01:01:11.460 --> 01:01:14.300 align:middle line:90%
So this one.

01:01:14.300 --> 01:01:16.900 align:middle line:84%
So this is the fundamental
step of gradient descent.

01:01:16.900 --> 01:01:19.260 align:middle line:84%
So this is the
current value of w.

01:01:19.260 --> 01:01:22.420 align:middle line:84%
You calculate the gradient at
that current value multiplied

01:01:22.420 --> 01:01:26.140 align:middle line:84%
by alpha, do this thing,
and you get the new value.

01:01:26.140 --> 01:01:27.460 align:middle line:90%
And you keep repeating.

01:01:27.460 --> 01:01:31.572 align:middle line:84%
STUDENT: Right, but g w,
that's not the loss function?

01:01:31.572 --> 01:01:33.280 align:middle line:84%
RAMA RAMAKRISHNAN: It
is a loss function.

01:01:33.280 --> 01:01:34.240 align:middle line:84%
STUDENT: That is
the loss function?

01:01:34.240 --> 01:01:35.240 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

01:01:35.240 --> 01:01:37.860 align:middle line:84%
Here, I'm just using g
as an arbitrary function

01:01:37.860 --> 01:01:39.720 align:middle line:84%
to just to
demonstrate the point,

01:01:39.720 --> 01:01:41.900 align:middle line:84%
but when you're optimizing--
when you're training

01:01:41.900 --> 01:01:44.260 align:middle line:84%
a neural network, what
you're actually doing

01:01:44.260 --> 01:01:47.697 align:middle line:84%
is minimizing a loss
function, loss of w.

01:01:47.697 --> 01:01:49.280 align:middle line:84%
STUDENT: Sorry, I
got things mixed up.

01:01:49.280 --> 01:01:51.420 align:middle line:90%
RAMA RAMAKRISHNAN: No worries.

01:01:51.420 --> 01:01:52.885 align:middle line:90%
Yeah?

01:01:52.885 --> 01:01:54.760 align:middle line:84%
STUDENT: How do we define
the initial weights

01:01:54.760 --> 01:01:56.947 align:middle line:90%
for the neural network?

01:01:56.947 --> 01:01:59.030 align:middle line:84%
RAMA RAMAKRISHNAN: So,
yeah, the initial weights--

01:01:59.030 --> 01:02:02.240 align:middle line:90%


01:02:02.240 --> 01:02:04.500 align:middle line:84%
so there are many ways
to-- so first of all,

01:02:04.500 --> 01:02:07.560 align:middle line:84%
they are initialized
randomly, but randomly

01:02:07.560 --> 01:02:09.720 align:middle line:84%
doesn't mean you can just
pick any random weight.

01:02:09.720 --> 01:02:11.120 align:middle line:84%
There are actually
some good ways

01:02:11.120 --> 01:02:13.360 align:middle line:84%
to randomly pick the
weights, and those are

01:02:13.360 --> 01:02:15.760 align:middle line:90%
called initialization schemes.

01:02:15.760 --> 01:02:18.480 align:middle line:84%
And there are a bunch of very
effective initialization schemes

01:02:18.480 --> 01:02:20.140 align:middle line:84%
people have figured
out over the years,

01:02:20.140 --> 01:02:22.880 align:middle line:84%
and those things are baked
into Keras as the default.

01:02:22.880 --> 01:02:24.920 align:middle line:84%
So Keras, I believe,
uses something

01:02:24.920 --> 01:02:29.160 align:middle line:84%
called the HE initialization,
H-E initialization,

01:02:29.160 --> 01:02:31.795 align:middle line:84%
or the Xavier Glorot
initialization.

01:02:31.795 --> 01:02:33.920 align:middle line:84%
I wouldn't worry about it,
just go with the default

01:02:33.920 --> 01:02:35.825 align:middle line:90%
initialization.

01:02:35.825 --> 01:02:38.200 align:middle line:84%
And the reason why they have
to be very careful about how

01:02:38.200 --> 01:02:40.640 align:middle line:84%
these weights are
initialized is because if you

01:02:40.640 --> 01:02:44.200 align:middle line:84%
have a very big network and
if you initialize badly,

01:02:44.200 --> 01:02:49.040 align:middle line:84%
then the gradient will just
explode as you calculate it.

01:02:49.040 --> 01:02:52.040 align:middle line:84%
The earlier layers, the weights
will have massive gradients

01:02:52.040 --> 01:02:54.253 align:middle line:84%
or gradients will
vanish, so therefore,

01:02:54.253 --> 01:02:56.420 align:middle line:84%
call the exploding gradient
problem or the vanishing

01:02:56.420 --> 01:02:57.420 align:middle line:90%
gradient problem.

01:02:57.420 --> 01:02:59.020 align:middle line:84%
To avoid all those
things, researchers

01:02:59.020 --> 01:03:01.270 align:middle line:84%
figured out some clever way
to initialize so that it's

01:03:01.270 --> 01:03:03.740 align:middle line:90%
well-behaved throughout.

01:03:03.740 --> 01:03:05.380 align:middle line:90%
Yeah?

01:03:05.380 --> 01:03:09.240 align:middle line:84%
STUDENT: If using backprops
and GPUs was so critical,

01:03:09.240 --> 01:03:11.620 align:middle line:84%
I'm just curious, who
first did it and when?

01:03:11.620 --> 01:03:13.560 align:middle line:90%
Was this a couple years ago?

01:03:13.560 --> 01:03:14.620 align:middle line:90%
Was it a company?

01:03:14.620 --> 01:03:15.120 align:middle line:90%
Was it--

01:03:15.120 --> 01:03:16.120 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

01:03:16.120 --> 01:03:20.140 align:middle line:84%
Well, GPUs have been used for
deep learning, I want to say--

01:03:20.140 --> 01:03:24.745 align:middle line:84%
I think the first case may have
been in the mid-2005, 2006 sort

01:03:24.745 --> 01:03:26.220 align:middle line:90%
of thing.

01:03:26.220 --> 01:03:29.660 align:middle line:84%
But I would say that it burst
out onto the world stage

01:03:29.660 --> 01:03:33.140 align:middle line:84%
and made everyone take
notice when a deep learning

01:03:33.140 --> 01:03:39.180 align:middle line:84%
model called AlexNet in 2012 won
a very famous computer vision

01:03:39.180 --> 01:03:40.700 align:middle line:90%
competition.

01:03:40.700 --> 01:03:42.380 align:middle line:90%
And it beat the--

01:03:42.380 --> 01:03:45.302 align:middle line:84%
and it set a world record
for how good it was,

01:03:45.302 --> 01:03:47.760 align:middle line:84%
and that's when everyone was
like, hey, what is this thing?

01:03:47.760 --> 01:03:49.720 align:middle line:84%
And that's really when it
burst onto the world stage.

01:03:49.720 --> 01:03:51.137 align:middle line:84%
I'll talk a bit
more about it when

01:03:51.137 --> 01:03:54.098 align:middle line:84%
I get into the computer
vision segment of the class.

01:03:54.098 --> 01:03:55.640 align:middle line:84%
But you can Google
AlexNet and you'll

01:03:55.640 --> 01:03:59.412 align:middle line:84%
find a whole bunch
of history around it.

01:03:59.412 --> 01:04:01.120 align:middle line:84%
STUDENT: Hypothetically,
then, is it true

01:04:01.120 --> 01:04:04.960 align:middle line:84%
that if we could get
to a global minima,

01:04:04.960 --> 01:04:07.340 align:middle line:84%
that would mean there
would be no hallucinations?

01:04:07.340 --> 01:04:09.840 align:middle line:84%
RAMA RAMAKRISHNAN:
Aha, good question.

01:04:09.840 --> 01:04:13.120 align:middle line:84%
So if it is perfect, if you
get to a global minimum-- first

01:04:13.120 --> 01:04:15.580 align:middle line:84%
of all, global minima doesn't
mean the model is perfect.

01:04:15.580 --> 01:04:18.440 align:middle line:90%
It may still have some loss.

01:04:18.440 --> 01:04:24.320 align:middle line:84%
But global minima is going
to be on the training data.

01:04:24.320 --> 01:04:26.960 align:middle line:84%
You can imagine that the
test data, future data

01:04:26.960 --> 01:04:29.520 align:middle line:90%
has its own loss function.

01:04:29.520 --> 01:04:31.818 align:middle line:84%
So what is minimum here
may not be minimum there.

01:04:31.818 --> 01:04:32.610 align:middle line:90%
That's the problem.

01:04:32.610 --> 01:04:36.440 align:middle line:90%


01:04:36.440 --> 01:04:37.460 align:middle line:90%
Is it a comment?

01:04:37.460 --> 01:04:37.960 align:middle line:90%
No?

01:04:37.960 --> 01:04:38.880 align:middle line:90%
OK.

01:04:38.880 --> 01:04:42.080 align:middle line:84%
STUDENT: Just saying that
that would mean that also you

01:04:42.080 --> 01:04:42.965 align:middle line:90%
can be overfitting--

01:04:42.965 --> 01:04:44.840 align:middle line:84%
RAMA RAMAKRISHNAN:
Correct, exactly, exactly.

01:04:44.840 --> 01:04:47.600 align:middle line:84%
So if you overdo, if you find
the best thing in the training

01:04:47.600 --> 01:04:50.180 align:middle line:84%
function, chances are,
it doesn't match the best

01:04:50.180 --> 01:04:52.180 align:middle line:90%
thing of the test data.

01:04:52.180 --> 01:04:56.420 align:middle line:84%
So on the test data, you're
actually doing badly.

01:04:56.420 --> 01:04:57.060 align:middle line:90%
OK.

01:04:57.060 --> 01:04:59.120 align:middle line:90%
So, come back to this.

01:04:59.120 --> 01:05:03.820 align:middle line:90%


01:05:03.820 --> 01:05:04.580 align:middle line:90%
OK.

01:05:04.580 --> 01:05:07.620 align:middle line:84%
Now the final twist
to the tale here,

01:05:07.620 --> 01:05:09.780 align:middle line:84%
we're going to go from
something, gradient descent,

01:05:09.780 --> 01:05:11.800 align:middle line:84%
to something called
stochastic gradient descent.

01:05:11.800 --> 01:05:14.460 align:middle line:84%
And stochastic gradient
descent, or SGD,

01:05:14.460 --> 01:05:17.500 align:middle line:84%
is the workhorse for
all deep learning.

01:05:17.500 --> 01:05:21.140 align:middle line:84%
And funnily enough,
SGD is simpler than GD.

01:05:21.140 --> 01:05:23.820 align:middle line:84%
Just when you thought
it couldn't get simpler.

01:05:23.820 --> 01:05:25.300 align:middle line:90%
OK.

01:05:25.300 --> 01:05:27.340 align:middle line:84%
So for large data
sets, computing

01:05:27.340 --> 01:05:30.300 align:middle line:84%
the gradient of the loss
function can be very expensive,

01:05:30.300 --> 01:05:33.040 align:middle line:84%
needless to say, because it
has to be done at every step,

01:05:33.040 --> 01:05:35.860 align:middle line:84%
and the cardinality of the
data set is really big,

01:05:35.860 --> 01:05:38.080 align:middle line:84%
and you may have
billions of parameters,

01:05:38.080 --> 01:05:43.180 align:middle line:84%
it's just very, very tough to
compute it even with backprop.

01:05:43.180 --> 01:05:45.700 align:middle line:84%
So the solution is,
at each iteration--

01:05:45.700 --> 01:05:48.000 align:middle line:84%
when I say iteration, I'm
talking about this step

01:05:48.000 --> 01:05:50.160 align:middle line:90%
of gradient descent.

01:05:50.160 --> 01:05:54.280 align:middle line:84%
Instead of using all the data,
instead of calculating the loss

01:05:54.280 --> 01:05:58.145 align:middle line:84%
function by averaging the
loss across all n data points,

01:05:58.145 --> 01:05:59.520 align:middle line:84%
and then calculating
the gradient

01:05:59.520 --> 01:06:01.920 align:middle line:84%
of that thing, what
you do is you just

01:06:01.920 --> 01:06:04.360 align:middle line:90%
choose a small sample randomly.

01:06:04.360 --> 01:06:06.520 align:middle line:84%
You choose just a few
of the n observations,

01:06:06.520 --> 01:06:08.400 align:middle line:90%
and we call it a batch.

01:06:08.400 --> 01:06:10.200 align:middle line:84%
So for example,
the number of data

01:06:10.200 --> 01:06:12.560 align:middle line:84%
points-- you may have
10 billion data points,

01:06:12.560 --> 01:06:14.360 align:middle line:84%
but in every iteration,
you may literally

01:06:14.360 --> 01:06:18.300 align:middle line:84%
grab just 32 or 64,
something really small.

01:06:18.300 --> 01:06:20.280 align:middle line:90%
Like absurdly small.

01:06:20.280 --> 01:06:24.240 align:middle line:84%
And then you pretend that, OK,
that's all the data I have,

01:06:24.240 --> 01:06:26.700 align:middle line:84%
you calculate the loss,
find the gradient,

01:06:26.700 --> 01:06:28.390 align:middle line:90%
and just use that here instead.

01:06:28.390 --> 01:06:31.200 align:middle line:90%


01:06:31.200 --> 01:06:34.320 align:middle line:84%
So this is called
stochastic gradient descent.

01:06:34.320 --> 01:06:38.520 align:middle line:84%
So strictly speaking,
theoretically, SGD uses just

01:06:38.520 --> 01:06:41.480 align:middle line:84%
one data point,
but in practice, we

01:06:41.480 --> 01:06:42.880 align:middle line:90%
use what's called a minibatch.

01:06:42.880 --> 01:06:45.060 align:middle line:90%
32, 64, whatever.

01:06:45.060 --> 01:06:47.780 align:middle line:84%
And so minibatch gradient
descent is just loosely

01:06:47.780 --> 01:06:49.570 align:middle line:84%
called stochastic
gradient descent, SGD.

01:06:49.570 --> 01:06:52.740 align:middle line:90%


01:06:52.740 --> 01:06:56.140 align:middle line:84%
So an SGD, as it turns
out, you can see,

01:06:56.140 --> 01:06:59.180 align:middle line:84%
it's clearly very
efficient because it's just

01:06:59.180 --> 01:07:01.180 align:middle line:90%
processing a few at a time.

01:07:01.180 --> 01:07:03.540 align:middle line:84%
And in fact, if you
have a lot of data,

01:07:03.540 --> 01:07:06.320 align:middle line:84%
and you calculate the
gradient of the loss function,

01:07:06.320 --> 01:07:09.600 align:middle line:90%
it may not even fit into memory.

01:07:09.600 --> 01:07:11.020 align:middle line:90%
It's really problematic.

01:07:11.020 --> 01:07:12.620 align:middle line:84%
But with SGD, it
says, I don't care

01:07:12.620 --> 01:07:14.995 align:middle line:84%
whether you have a billion
data points or a trillion data

01:07:14.995 --> 01:07:17.680 align:middle line:84%
points, just give
me 32 at a time.

01:07:17.680 --> 01:07:20.660 align:middle line:90%
And you just keep on doing it.

01:07:20.660 --> 01:07:22.740 align:middle line:84%
It turns out, because
not all the points are

01:07:22.740 --> 01:07:24.780 align:middle line:84%
used in the
calculation, this only

01:07:24.780 --> 01:07:26.595 align:middle line:90%
approximates the true gradient.

01:07:26.595 --> 01:07:27.720 align:middle line:90%
It's only an approximation.

01:07:27.720 --> 01:07:29.887 align:middle line:84%
It's not the real thing,
it's only an approximation,

01:07:29.887 --> 01:07:32.120 align:middle line:84%
but it works extremely
well in practice.

01:07:32.120 --> 01:07:33.438 align:middle line:90%
Extremely well in practice.

01:07:33.438 --> 01:07:34.980 align:middle line:84%
And there's a whole
bunch of research

01:07:34.980 --> 01:07:37.340 align:middle line:84%
that goes into why
is it so effective.

01:07:37.340 --> 01:07:40.580 align:middle line:84%
And people are discovering
interesting things about SGD,

01:07:40.580 --> 01:07:42.560 align:middle line:84%
but we don't have
a definitive theory

01:07:42.560 --> 01:07:44.360 align:middle line:90%
as to why it's so good yet.

01:07:44.360 --> 01:07:46.400 align:middle line:84%
We have some interesting
research threads

01:07:46.400 --> 01:07:47.840 align:middle line:90%
that have happened.

01:07:47.840 --> 01:07:51.840 align:middle line:84%
And, very tantalizingly,
very tantalizingly,

01:07:51.840 --> 01:07:53.840 align:middle line:84%
because it's only
an approximation

01:07:53.840 --> 01:07:59.520 align:middle line:84%
of the true gradient, SGD can
actually escape local minima.

01:07:59.520 --> 01:08:04.200 align:middle line:84%
So in the true loss function,
you're at a local minimum,

01:08:04.200 --> 01:08:07.580 align:middle line:84%
but in SGD loss function,
when you're doing SGD,

01:08:07.580 --> 01:08:10.400 align:middle line:84%
you're reaching the
minimum of the SGD loss

01:08:10.400 --> 01:08:14.240 align:middle line:84%
function, which actually may
not be the actual loss function.

01:08:14.240 --> 01:08:15.942 align:middle line:84%
So as you're moving
around, you're

01:08:15.942 --> 01:08:17.359 align:middle line:84%
actually jumping
from local minima

01:08:17.359 --> 01:08:20.399 align:middle line:84%
to local minima of the
actual loss function.

01:08:20.399 --> 01:08:23.080 align:middle line:84%
I know, that's a mouthful,
I'm happy to tell you more.

01:08:23.080 --> 01:08:26.516 align:middle line:84%
It's just a side thing that I
just want you to be aware of.

01:08:26.516 --> 01:08:28.819 align:middle line:84%
One of the reasons why
SGD is actually effective,

01:08:28.819 --> 01:08:34.097 align:middle line:84%
it's almost like you work
less and you do better.

01:08:34.097 --> 01:08:35.680 align:middle line:84%
How many times does
it happen in life?

01:08:35.680 --> 01:08:36.580 align:middle line:90%
This is one of them.

01:08:36.580 --> 01:08:40.030 align:middle line:90%


01:08:40.030 --> 01:08:43.145 align:middle line:90%
Now SGD comes in many flavors.

01:08:43.145 --> 01:08:45.520 align:middle line:84%
Many siblings, it's got a lot
of siblings and variations.

01:08:45.520 --> 01:08:46.819 align:middle line:90%
It's a big family.

01:08:46.819 --> 01:08:49.899 align:middle line:84%
And we're going to use a
particular flavor called Adam

01:08:49.899 --> 01:08:52.140 align:middle line:84%
as our default in
this course, and I'll

01:08:52.140 --> 01:08:55.979 align:middle line:84%
get back to it when you get into
the Colabs and things like that.

01:08:55.979 --> 01:08:58.140 align:middle line:90%
All right.

01:08:58.140 --> 01:09:01.053 align:middle line:84%
By the way you
know how all these

01:09:01.053 --> 01:09:03.720 align:middle line:84%
pictures I've been showing you a
nice little function like that,

01:09:03.720 --> 01:09:05.420 align:middle line:90%
a little bowl and so on?

01:09:05.420 --> 01:09:09.100 align:middle line:84%
This is a visualization of
an actual neural network loss

01:09:09.100 --> 01:09:11.380 align:middle line:90%
function.

01:09:11.380 --> 01:09:13.439 align:middle line:84%
You can see the hills and
valleys and the cracks

01:09:13.439 --> 01:09:14.939 align:middle line:90%
and so on and so forth.

01:09:14.939 --> 01:09:17.859 align:middle line:84%
And you can check out that
paper to get more insight

01:09:17.859 --> 01:09:20.439 align:middle line:84%
into how they actually came
up with this visualization.

01:09:20.439 --> 01:09:21.660 align:middle line:90%
It's crazy.

01:09:21.660 --> 01:09:24.340 align:middle line:90%
It's complicated.

01:09:24.340 --> 01:09:25.580 align:middle line:90%
Yeah?

01:09:25.580 --> 01:09:29.580 align:middle line:84%
STUDENT: So for SGD, do you
perform the iterations until you

01:09:29.580 --> 01:09:32.340 align:middle line:84%
minimize the loss function
for each minibatch

01:09:32.340 --> 01:09:34.200 align:middle line:84%
and then move to
another minibatch?

01:09:34.200 --> 01:09:34.300 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

01:09:34.300 --> 01:09:36.500 align:middle line:84%
So what you do is you
take each minibatch,

01:09:36.500 --> 01:09:40.100 align:middle line:84%
and then you calculate the
loss for that minibatch,

01:09:40.100 --> 01:09:41.560 align:middle line:90%
you find the gradient.

01:09:41.560 --> 01:09:43.740 align:middle line:84%
And use the gradient
and update the w.

01:09:43.740 --> 01:09:45.279 align:middle line:84%
Then you pick up
the next minibatch.

01:09:45.279 --> 01:09:47.200 align:middle line:84%
STUDENT: So you don't
pick a minibatch

01:09:47.200 --> 01:09:49.660 align:middle line:84%
and try to perform the
iterations on that minibatch

01:09:49.660 --> 01:09:50.160 align:middle line:90%
until you--

01:09:50.160 --> 01:09:51.399 align:middle line:90%
RAMA RAMAKRISHNAN: Correct.

01:09:51.399 --> 01:09:52.779 align:middle line:90%
Each minibatch, one iteration.

01:09:52.779 --> 01:09:54.200 align:middle line:90%
Each minibatch, one iteration.

01:09:54.200 --> 01:09:55.840 align:middle line:84%
Because if you do
a lot of iterations

01:09:55.840 --> 01:09:57.790 align:middle line:84%
on one minibatch,
first of all, you never

01:09:57.790 --> 01:10:00.040 align:middle line:84%
be sure that you're going
to find any optimal solution

01:10:00.040 --> 01:10:02.260 align:middle line:84%
because you're not guaranteed
of any global minima.

01:10:02.260 --> 01:10:03.843 align:middle line:84%
And secondly, it's
much better for you

01:10:03.843 --> 01:10:05.420 align:middle line:84%
to get new
information constantly

01:10:05.420 --> 01:10:08.440 align:middle line:84%
because what you can do is you
can revisit the minibatch later

01:10:08.440 --> 01:10:09.640 align:middle line:90%
on.

01:10:09.640 --> 01:10:12.400 align:middle line:84%
And that gets us to these things
called epochs and batch size

01:10:12.400 --> 01:10:14.920 align:middle line:84%
and so on, which we'll get
into a lot of gory detail

01:10:14.920 --> 01:10:16.320 align:middle line:90%
when we do the Colab.

01:10:16.320 --> 01:10:17.860 align:middle line:90%
So let's revisit that question.

01:10:17.860 --> 01:10:20.440 align:middle line:90%
It's a good question.

01:10:20.440 --> 01:10:22.640 align:middle line:90%
Yeah?

01:10:22.640 --> 01:10:25.220 align:middle line:84%
STUDENT: When you do the
backprop process, it's--

01:10:25.220 --> 01:10:25.680 align:middle line:90%
RAMA RAMAKRISHNAN: Very good.

01:10:25.680 --> 01:10:26.940 align:middle line:90%
Backprop, no backpropagation.

01:10:26.940 --> 01:10:27.280 align:middle line:90%
Nice.

01:10:27.280 --> 01:10:28.155 align:middle line:90%
STUDENT: I made sure.

01:10:28.155 --> 01:10:29.240 align:middle line:90%
RAMA RAMAKRISHNAN: Yes.

01:10:29.240 --> 01:10:32.040 align:middle line:84%
It sounded like you started
from the layers that

01:10:32.040 --> 01:10:34.100 align:middle line:84%
were closest to the output
and you went backward.

01:10:34.100 --> 01:10:38.030 align:middle line:84%
And my question is, are you
doing that once or is it

01:10:38.030 --> 01:10:39.090 align:middle line:90%
looping multiple times?

01:10:39.090 --> 01:10:39.590 align:middle line:90%
And then--

01:10:39.590 --> 01:10:40.350 align:middle line:84%
RAMA RAMAKRISHNAN:
You just do it once.

01:10:40.350 --> 01:10:41.110 align:middle line:90%
STUDENT: Just once?

01:10:41.110 --> 01:10:42.110 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

01:10:42.110 --> 01:10:44.588 align:middle line:84%
So for each gradient
calculation, you do it once.

01:10:44.588 --> 01:10:46.630 align:middle line:84%
STUDENT: Why does it want
to start from the layer

01:10:46.630 --> 01:10:47.410 align:middle line:90%
that's closest--

01:10:47.410 --> 01:10:48.830 align:middle line:84%
or why do you want to start
it from the layer that's

01:10:48.830 --> 01:10:49.890 align:middle line:90%
closest to the output?

01:10:49.890 --> 01:10:50.030 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

01:10:50.030 --> 01:10:52.113 align:middle line:84%
So basically what happens
is, let's say that, just

01:10:52.113 --> 01:10:54.830 align:middle line:84%
for argument, that you go
in the reverse direction,

01:10:54.830 --> 01:10:57.390 align:middle line:84%
you will discover that a lot
of paths to go from the left

01:10:57.390 --> 01:10:59.070 align:middle line:84%
to the right will
end up calculating

01:10:59.070 --> 01:11:01.390 align:middle line:84%
certain intermediate
quantities, including

01:11:01.390 --> 01:11:05.977 align:middle line:84%
the very final gradient item,
again and again and again.

01:11:05.977 --> 01:11:08.310 align:middle line:84%
The same thing is going to
be calculated again and again

01:11:08.310 --> 01:11:08.990 align:middle line:90%
and again.

01:11:08.990 --> 01:11:11.170 align:middle line:84%
So by starting from the
end and working backwards,

01:11:11.170 --> 01:11:14.270 align:middle line:84%
you just reuse stuff
you've already calculated.

01:11:14.270 --> 01:11:15.850 align:middle line:90%
So that is the rough idea.

01:11:15.850 --> 01:11:18.648 align:middle line:84%
But if you see my PDF, I've
actually worked out the example,

01:11:18.648 --> 01:11:20.690 align:middle line:84%
and that will demonstrate
what I'm talking about.

01:11:20.690 --> 01:11:23.350 align:middle line:90%


01:11:23.350 --> 01:11:26.120 align:middle line:84%
By the way, the
backprop is just a--

01:11:26.120 --> 01:11:28.950 align:middle line:90%


01:11:28.950 --> 01:11:31.640 align:middle line:84%
in calculus, we have something
called the chain rule.

01:11:31.640 --> 01:11:33.890 align:middle line:84%
To calculate the derivative
of a complicated function,

01:11:33.890 --> 01:11:36.550 align:middle line:84%
you calculate the derivative
of the outer function,

01:11:36.550 --> 01:11:38.610 align:middle line:84%
then the inner function,
and so on and so forth.

01:11:38.610 --> 01:11:39.985 align:middle line:84%
The backdrop is
essentially a way

01:11:39.985 --> 01:11:41.730 align:middle line:84%
to organize the
chain rule to work

01:11:41.730 --> 01:11:44.070 align:middle line:84%
with the neural network,
layer-by-layer architecture,

01:11:44.070 --> 01:11:44.590 align:middle line:90%
that's all.

01:11:44.590 --> 01:11:48.250 align:middle line:90%


01:11:48.250 --> 01:11:49.530 align:middle line:90%
Yeah?

01:11:49.530 --> 01:11:52.610 align:middle line:84%
STUDENT: So is it fair to
say that once we are finding

01:11:52.610 --> 01:11:55.490 align:middle line:84%
the local minimum, we
are not optimizing to all

01:11:55.490 --> 01:11:57.853 align:middle line:84%
the g w's because
this local minimum

01:11:57.853 --> 01:12:00.270 align:middle line:84%
is coming like from different
curves from different lines,

01:12:00.270 --> 01:12:01.437 align:middle line:90%
so is that fair to say--

01:12:01.437 --> 01:12:03.770 align:middle line:84%
RAMA RAMAKRISHNAN: When we
are using stochastic gradient

01:12:03.770 --> 01:12:04.530 align:middle line:90%
descent, yes.

01:12:04.530 --> 01:12:06.430 align:middle line:84%
So in stochastic
gradient descent,

01:12:06.430 --> 01:12:09.290 align:middle line:84%
when you take, say, 32
data points from a million

01:12:09.290 --> 01:12:12.230 align:middle line:84%
and you're calculating the
loss for that 32 data points,

01:12:12.230 --> 01:12:15.010 align:middle line:84%
you're basically trying
to do a gradient step.

01:12:15.010 --> 01:12:18.130 align:middle line:84%
The w equals w minus
alpha gradient thing.

01:12:18.130 --> 01:12:21.490 align:middle line:84%
You're doing it
for that 32 points

01:12:21.490 --> 01:12:25.770 align:middle line:84%
loss function, which is not the
1 million points loss function.

01:12:25.770 --> 01:12:27.610 align:middle line:90%
That's why it's approximate.

01:12:27.610 --> 01:12:30.010 align:middle line:84%
But the approximation,
instead of hurting you,

01:12:30.010 --> 01:12:32.170 align:middle line:84%
actually helps you
because it helps

01:12:32.170 --> 01:12:35.710 align:middle line:84%
you escape the local minima
of the global loss function.

01:12:35.710 --> 01:12:38.002 align:middle line:84%
So it's an interesting and
somewhat technically subtle

01:12:38.002 --> 01:12:40.210 align:middle line:84%
point, which is why I'm not
getting into it too much,

01:12:40.210 --> 01:12:42.830 align:middle line:84%
but I'm happy to give pointers
if people are interested.

01:12:42.830 --> 01:12:44.288 align:middle line:90%
Yeah?

01:12:44.288 --> 01:12:46.330 align:middle line:84%
STUDENT: When you say you
initialize the weights,

01:12:46.330 --> 01:12:49.750 align:middle line:84%
you initialize for the whole
network or just the end layer

01:12:49.750 --> 01:12:51.110 align:middle line:90%
and then go backwards--

01:12:51.110 --> 01:12:53.830 align:middle line:84%
RAMA RAMAKRISHNAN: So usually
it's everything in one shot.

01:12:53.830 --> 01:12:55.110 align:middle line:84%
Because if you don't
initialize everything

01:12:55.110 --> 01:12:56.568 align:middle line:84%
in one shot, what's
going to happen

01:12:56.568 --> 01:12:58.590 align:middle line:84%
is that you can't do
the forward computation

01:12:58.590 --> 01:13:01.110 align:middle line:90%
to find the prediction.

01:13:01.110 --> 01:13:02.850 align:middle line:84%
And so they are
done independently,

01:13:02.850 --> 01:13:05.730 align:middle line:84%
and the initialization schemes
will take into account,

01:13:05.730 --> 01:13:08.310 align:middle line:84%
OK, I'm initializing the
weights between a layer which

01:13:08.310 --> 01:13:11.870 align:middle line:84%
has 10 nodes on one side and 32
on the other side, and the 10

01:13:11.870 --> 01:13:15.950 align:middle line:84%
and the 32 actually play a
role in how you initialize.

01:13:15.950 --> 01:13:16.830 align:middle line:90%
OK.

01:13:16.830 --> 01:13:19.910 align:middle line:84%
So, the summary of the
overall training flow

01:13:19.910 --> 01:13:22.830 align:middle line:90%
is that you have an input.

01:13:22.830 --> 01:13:24.370 align:middle line:84%
It goes through a
bunch of layers.

01:13:24.370 --> 01:13:25.650 align:middle line:90%
You come up with a prediction.

01:13:25.650 --> 01:13:27.590 align:middle line:84%
You compare it to
the true values.

01:13:27.590 --> 01:13:30.250 align:middle line:84%
And these two things go into
the loss function calculation,

01:13:30.250 --> 01:13:32.170 align:middle line:90%
you get a loss number.

01:13:32.170 --> 01:13:34.250 align:middle line:84%
And you do it for, say,
10 points or 32 points

01:13:34.250 --> 01:13:35.390 align:middle line:90%
or a million points.

01:13:35.390 --> 01:13:38.110 align:middle line:84%
And this loss thing
goes into the optimizer,

01:13:38.110 --> 01:13:39.650 align:middle line:90%
which calculates the gradient.

01:13:39.650 --> 01:13:41.190 align:middle line:84%
And once it calculates
the gradient,

01:13:41.190 --> 01:13:44.210 align:middle line:84%
it updates the weights of
every layer using the w

01:13:44.210 --> 01:13:46.470 align:middle line:84%
equals w minus alpha
times gradient formula,

01:13:46.470 --> 01:13:47.710 align:middle line:90%
gradient descent formula.

01:13:47.710 --> 01:13:50.370 align:middle line:84%
And then you keep doing this
again and again and again.

01:13:50.370 --> 01:13:52.955 align:middle line:90%
This is the overall flow.

01:13:52.955 --> 01:13:54.330 align:middle line:84%
This is how our
little network is

01:13:54.330 --> 01:13:57.070 align:middle line:84%
going to get built for
heart disease prediction,

01:13:57.070 --> 01:14:02.690 align:middle line:84%
this is how GPT was built. And
this is how AlphaFold was built.

01:14:02.690 --> 01:14:07.330 align:middle line:84%
And AlphaGo was built.
You get the idea.

01:14:07.330 --> 01:14:09.010 align:middle line:84%
I mean, it's
astonishing, frankly.

01:14:09.010 --> 01:14:11.130 align:middle line:84%
If you're not getting
goosebumps at the thought

01:14:11.130 --> 01:14:13.808 align:middle line:84%
that this simple thing can do
all these complicated things,

01:14:13.808 --> 01:14:15.100 align:middle line:90%
we really need to talk offline.

01:14:15.100 --> 01:14:17.487 align:middle line:90%


01:14:17.487 --> 01:14:19.070 align:middle line:84%
STUDENT: There was
a hand raised here.

01:14:19.070 --> 01:14:20.410 align:middle line:90%
RAMA RAMAKRISHNAN: Yeah.

01:14:20.410 --> 01:14:21.290 align:middle line:90%
STUDENT: Sorry.

01:14:21.290 --> 01:14:24.836 align:middle line:84%
Just quickly, this is for
each minibatch, right?

01:14:24.836 --> 01:14:27.850 align:middle line:84%
so my question is, if you
came with different weights

01:14:27.850 --> 01:14:31.460 align:middle line:84%
for each minibatch,
how do you add it up?

01:14:31.460 --> 01:14:34.710 align:middle line:84%
Like, OK, this weight is
the perfect combination

01:14:34.710 --> 01:14:38.230 align:middle line:84%
for this minibatch, but you
have way different weight

01:14:38.230 --> 01:14:40.670 align:middle line:84%
for another minibatch, how
do you combine those two?

01:14:40.670 --> 01:14:41.590 align:middle line:90%
RAMA RAMAKRISHNAN: No.

01:14:41.590 --> 01:14:44.570 align:middle line:84%
At each point, what you
do is you find you find--

01:14:44.570 --> 01:14:47.835 align:middle line:84%
you start with a weight, you
run it through for a minibatch,

01:14:47.835 --> 01:14:49.210 align:middle line:84%
you come up with
a loss function,

01:14:49.210 --> 01:14:50.565 align:middle line:90%
you calculate the gradient.

01:14:50.565 --> 01:14:52.690 align:middle line:84%
And now using the gradient,
you updated the weight,

01:14:52.690 --> 01:14:53.870 align:middle line:84%
now you have a new
set of weights,

01:14:53.870 --> 01:14:55.078 align:middle line:90%
which is the updated weights.

01:14:55.078 --> 01:14:57.550 align:middle line:90%
Call it w2 instead of w1.

01:14:57.550 --> 01:14:59.470 align:middle line:84%
Now w2 is using your
network, and when

01:14:59.470 --> 01:15:00.845 align:middle line:84%
you take the next
minibatch, it's

01:15:00.845 --> 01:15:03.550 align:middle line:84%
going to use w2 to
calculate the prediction.

01:15:03.550 --> 01:15:06.150 align:middle line:84%
And this whole flow will
become a lot clearer

01:15:06.150 --> 01:15:08.790 align:middle line:90%
when we do the Colabs.

01:15:08.790 --> 01:15:11.790 align:middle line:90%
OK, so we have three minutes.

01:15:11.790 --> 01:15:14.070 align:middle line:84%
I don't want to go
into regularization

01:15:14.070 --> 01:15:19.710 align:middle line:84%
overfitting in three minutes, so
let's have some more questions.

01:15:19.710 --> 01:15:20.670 align:middle line:90%
Yeah?

01:15:20.670 --> 01:15:22.550 align:middle line:84%
STUDENT: Can you use
any activation function

01:15:22.550 --> 01:15:26.110 align:middle line:84%
as long as it gives positive
values for x squared

01:15:26.110 --> 01:15:28.570 align:middle line:90%
or mod x or something?

01:15:28.570 --> 01:15:34.170 align:middle line:84%
RAMA RAMAKRISHNAN: You can use a
variety of activation functions,

01:15:34.170 --> 01:15:35.130 align:middle line:90%
but--

01:15:35.130 --> 01:15:37.610 align:middle line:84%
there's a whole literature
on the pros and cons

01:15:37.610 --> 01:15:39.810 align:middle line:84%
of various activation
functions that you could use,

01:15:39.810 --> 01:15:43.430 align:middle line:84%
but in general, you have to
make sure of a couple of things.

01:15:43.430 --> 01:15:46.683 align:middle line:84%
One is that when
you do backprop,

01:15:46.683 --> 01:15:48.850 align:middle line:84%
the gradient is going to
flow through the activation

01:15:48.850 --> 01:15:50.610 align:middle line:84%
function in the
reverse direction,

01:15:50.610 --> 01:15:52.610 align:middle line:84%
and the activation
function should actually

01:15:52.610 --> 01:15:55.410 align:middle line:84%
make sure the gradient
doesn't get squished.

01:15:55.410 --> 01:15:58.570 align:middle line:84%
It shouldn't get squished,
it shouldn't get exploded.

01:15:58.570 --> 01:15:59.987 align:middle line:84%
So those are some
considerations--

01:15:59.987 --> 01:16:01.445 align:middle line:84%
these are technical
considerations,

01:16:01.445 --> 01:16:03.790 align:middle line:84%
but all those conditions have
to be taken into account.

01:16:03.790 --> 01:16:07.050 align:middle line:84%
If you can take those into
account, then you're OK.

01:16:07.050 --> 01:16:09.010 align:middle line:84%
That's the key thing
to keep in mind.

01:16:09.010 --> 01:16:11.890 align:middle line:84%
And that's, in fact, why the
ReLU is actually very popular,

01:16:11.890 --> 01:16:13.990 align:middle line:84%
because as long as
the value is positive,

01:16:13.990 --> 01:16:17.130 align:middle line:84%
the gradient of
the ReLU is just 1

01:16:17.130 --> 01:16:24.330 align:middle line:84%
because if you
look at something--

01:16:24.330 --> 01:16:24.830 align:middle line:90%
oops.

01:16:24.830 --> 01:16:28.750 align:middle line:90%


01:16:28.750 --> 01:16:30.430 align:middle line:90%
Because it's frozen.

01:16:30.430 --> 01:16:31.950 align:middle line:90%
I jinxed it.

01:16:31.950 --> 01:16:34.870 align:middle line:90%
So, sorry, livestream.

01:16:34.870 --> 01:16:39.870 align:middle line:84%
If you have something like
this, the ReLU is like that.

01:16:39.870 --> 01:16:42.190 align:middle line:84%
So the gradient
here is always going

01:16:42.190 --> 01:16:45.330 align:middle line:84%
to be 1, which means that as
long as the value is positive,

01:16:45.330 --> 01:16:46.890 align:middle line:84%
whatever gradient
comes in like this,

01:16:46.890 --> 01:16:49.650 align:middle line:84%
it just gets multiplied by 1 and
gets pushed out the other side.

01:16:49.650 --> 01:16:53.870 align:middle line:84%
So it doesn't get harmed or
squished or anything like that.

01:16:53.870 --> 01:16:57.190 align:middle line:84%
So that's one reason why the
ReLU is very popular, because it

01:16:57.190 --> 01:16:59.270 align:middle line:84%
preserves the gradient
while injecting

01:16:59.270 --> 01:17:01.590 align:middle line:84%
almost the minimum
amount of non-linearity

01:17:01.590 --> 01:17:02.770 align:middle line:90%
to do interesting things.

01:17:02.770 --> 01:17:05.350 align:middle line:90%


01:17:05.350 --> 01:17:07.630 align:middle line:90%
Yeah?

01:17:07.630 --> 01:17:10.450 align:middle line:84%
STUDENT: If you have a
high number of dimensions,

01:17:10.450 --> 01:17:14.190 align:middle line:84%
can you do minibatching
on features, dimensions

01:17:14.190 --> 01:17:15.930 align:middle line:90%
instead of just observations?

01:17:15.930 --> 01:17:18.090 align:middle line:84%
Keep the same number
of observations,

01:17:18.090 --> 01:17:21.630 align:middle line:84%
but just take a small sample
of the number of features

01:17:21.630 --> 01:17:22.930 align:middle line:90%
that you're actually using?

01:17:22.930 --> 01:17:24.430 align:middle line:84%
RAMA RAMAKRISHNAN:
Oh, I see, I see.

01:17:24.430 --> 01:17:27.090 align:middle line:84%
So you're saying-- let's
say you have 10 features.

01:17:27.090 --> 01:17:29.830 align:middle line:84%
Instead of taking all data
points or 10 features,

01:17:29.830 --> 01:17:33.330 align:middle line:84%
what if you choose five
features and just use them

01:17:33.330 --> 01:17:34.930 align:middle line:90%
and do this thing?

01:17:34.930 --> 01:17:38.190 align:middle line:84%
As long as you can actually
compute the prediction.

01:17:38.190 --> 01:17:41.930 align:middle line:84%
To compute the prediction,
you may need all 10 features.

01:17:41.930 --> 01:17:45.210 align:middle line:84%
Or you need to have some
defaults for those features.

01:17:45.210 --> 01:17:48.230 align:middle line:84%
But if you define defaults
for those other five features,

01:17:48.230 --> 01:17:50.310 align:middle line:84%
you're basically
using all features.

01:17:50.310 --> 01:17:51.310 align:middle line:90%
So that's the key thing.

01:17:51.310 --> 01:17:54.390 align:middle line:84%
Can you actually calculate the
prediction by manipulating?

01:17:54.390 --> 01:17:57.810 align:middle line:90%
And typically you can't.

01:17:57.810 --> 01:17:58.950 align:middle line:90%
All right.

01:17:58.950 --> 01:17:59.450 align:middle line:90%
OK, folks.

01:17:59.450 --> 01:18:00.790 align:middle line:90%
9:55, I'm done.

01:18:00.790 --> 01:18:04.080 align:middle line:84%
Have a great rest of your
week, I'll see you on Monday.

01:18:04.080 --> 01:18:22.000 align:middle line:90%